In my youth, while in the U.S. Navy, I spent a great deal of time implementing and tuning one particular sorting algorithm: merge sort. We had log data captured during the Vietnam war, produced by the NTDS system I maintained, that was (of course!) in linear time order. We needed that data to be ordered by various other things, such as by target. The data was far too large to fit into the computer's very limited memory; it was stored on tape. I had the job of figuring out how to sort that data, using the four tape drives attached to the computer. The best answer was a merge sort. I remember that my first implementation took over a day to run, which was dangerously close to the average up-time of the system (computers back in those days were not very reliable!). After weeks of tuning, I managed to get that down to “just” a couple of hours. Most of my performance gain came from figuring out how to use multiple computers to increase the amount of (relatively fast) computer memory I had available. That turned out to be a career-changer, as I was sorting the data faster on a deployed ship than the big data center in the Philippines could do it. I ended up being helicoptered from my ship to the USS Enterprise, and from their flown on a COD to Subic Bay, where I taught the data center folks how to do it. They had lots more computers than my ship had, so their implementation ended up being much faster. After that incident my job shifted from primarily being a hardware repair guy to being a programmer...
Thursday, November 21, 2013
Geek: the sounds of sorting...
In my youth, while in the U.S. Navy, I spent a great deal of time implementing and tuning one particular sorting algorithm: merge sort. We had log data captured during the Vietnam war, produced by the NTDS system I maintained, that was (of course!) in linear time order. We needed that data to be ordered by various other things, such as by target. The data was far too large to fit into the computer's very limited memory; it was stored on tape. I had the job of figuring out how to sort that data, using the four tape drives attached to the computer. The best answer was a merge sort. I remember that my first implementation took over a day to run, which was dangerously close to the average up-time of the system (computers back in those days were not very reliable!). After weeks of tuning, I managed to get that down to “just” a couple of hours. Most of my performance gain came from figuring out how to use multiple computers to increase the amount of (relatively fast) computer memory I had available. That turned out to be a career-changer, as I was sorting the data faster on a deployed ship than the big data center in the Philippines could do it. I ended up being helicoptered from my ship to the USS Enterprise, and from their flown on a COD to Subic Bay, where I taught the data center folks how to do it. They had lots more computers than my ship had, so their implementation ended up being much faster. After that incident my job shifted from primarily being a hardware repair guy to being a programmer...
No comments:
Post a Comment