Storing and transmitting monetary values...

Storing and transmitting monetary values...  Most financial applications these days are not standalone applications, but rather components of much larger and more complicated systems.  In a past life I was the CTO of a firm that made stock and option trading software, and I once made a map of all the systems that our servers communicated with.  There were about a dozen separate machines in our own server room that talked to each other.  There were also over two hundred other applications (in over thirty other organizations) that our servers talked to.  Those communications paths used a dozen or so different protocols, not counting minor variations.  In addition to all these connections, our servers also stored financial information: in memory, on disk, and in logs.  That system was not an outlier – I've worked on dozens of other systems just as complex, or even more so.

This complexity has lots of implications when it comes to representing monetary values.  What jumps right out to anyone examining all these needs is that there is no “one way” to represent a monetary value that is correct, best, or optimal.

Often the representation has to match an existing standard.  For instance, perhaps the monetary value must be in the form of an ASCII string, like this: $4532.97.  The standard might specify that it be in a fixed-length field, with spaces padding to the left as required.  Or the standard might specify that the value be in XML, encoded in UTF-16, with a tag of “AMT” for the amount, and “CUR” for the three-character ISO currency code.  Some older protocols have specified binary formats, with a numeric code for the currency and packed BCD for the value.  I could go on and on with these, because many clever and/or crazy people have devoted a great amount of time to inventing all these schemes.

When storing or transmitting large numbers of values, often a compact (or compressed) encoding is essential.  I once worked on a server that did algorithmic trading (that is, the program decided what stocks to buy and sell).  This server needed to store (both on disk and in memory) huge tables of stock pricing data.  On busy days this could amount to several hundred million entries.  The original design of the server represented the monetary values in a fixed length binary structure that occupied 22 bytes, so on busy days we were storing a couple of gigabytes of data – and moving that data between disk, memory, and CPU cache.  We observed that this was a bottleneck in our code, so we came up with a variable-length encoding that averaged just 7.5 bytes per value.  We rolled our own on this one, as the numeric representation we were using didn't have a compact encoding.  The performance impact was stunning, much more than the 3x you might naively expect.  The biggest impact came from fewer CPU cache misses, the second biggest from faster encoding/decoding time.  The latter we didn't expect at all.  The real lesson for me: compact encodings are very valuable!

Databases present another challenge for representing monetary values.  The numeric column types available often can be specified with the precision required – but the API will require converting the value to or from a string or an unlimited precision value (such as BigDecimal).  Sometimes the values don't need to be queried, but rather simply stored, as key-accessed blobs.  In those cases, a compact encoding is valuable, especially for large tables of values accessed as a whole.

From all this we can derive another requirement for a monetary value representation: flexibility of encodings.  There should be a variety of supported, standard encodings including (especially) a compact, variable-length binary encoding, a fixed-length binary encoding.  There should be support for conversions to and from unlimited precision numbers as well as native types.  There should be support for conversions to and from strings.  Also very important, for flexibility: support for adding encodings – because there's no way to anticipate all the strange things the world will require of you...