Not many people see Windows development as a high-octane, nerve-fraying, roller-coaster ride. It’s fair to say we software developers are also not known for our thrill-seeking behavior, often leaning towards basking in the warm, safe, comfortable glow of an array of screens, and clicky keyboards. Occasionally the bugs we inadvertently create can lead to awkward conversations at best and yes, sometimes, we can cost our employers money. I can honestly say though that today’s Enterprise Article from Delphi developer David Bolton is in a whole different league of expensive errors. David tells us of the time he was called upon to fix an $800 million dollar bug. Oh boy…no pressure!
Table of Contents
How Delphi smooths the flow of Oil Trading software
Oil is the most traded commodity in the world, and I worked for a top US bank in their commodities software department in Canary Wharf in London. The bank had 60,000 employees worldwide. The job of our team of four was to keep the oil trading business running, patching software, troubleshooting bugs and adding new features.
In particular, we looked after two trading applications both written in Delphi. The trader’s system was for entering trades and was about 1.6 million lines of code. These were mainly physical oil trades but also included some derivatives such as futures and options.
This was a two-tier application with the main trade entry application talking to a server running Microsoft transaction Server (MTS). There were 30 COM server programs all running under MTS, each invoked for one type of trade when a trader saved a trade. All were written in Delphi and typically 20,000-30,000 lines long.
Running under MTS made it highly resilient. If a bug happened, then MTS would kill off the offending COM object and another would be restarted when needed. Plus, the architecture made it possible to save several trades all at once, saving in parallel. There were typically 20-30 traders entering and saving trades simultaneously.
Is XML really running the World’s finances?
Trades were saved from the main application as large XML files that could be several hundred lines long. These were passed to the relevant COM server which would read the XML file then generate a large SQL transaction based on multiple inserts and updates. This was run against a Sybase ASE 12 database (very similar to SQL Server which is based on Sybase), and the main application would be updated when the trade saved.
What did you use for database management?
One benefit of being an enterprise was we were provided with enterprise tools; we used DBArtisan for doing all the database management. I wrote a lot of SQL in this job and being able to log DBArtisan operations and capture the underlying SQL made it a lot easier.
I was asked to merge two code bases that originally been one and had diverged over a ten-year period. This meant doing a diff between two folder trees with about 800 files in each to identify where the differences were. I wrote a Delphi application to use Araxis Merge (an enterprise file comparison utility that can be used as a COM server) to compare all folders and files and saved myself weeks of effort.
Oil trades can be quite complicated. Tankers can take several days unloading fuel at refineries and the value of the oil varies during this period. It is priced at an agreed market price at the end of each day until fully unloaded and then the title changes hands. After that the contract price is used to value it.
What other applications did you work on?
The 2nd Delphi application we supported was a PLServer and was about 1.2 million lines of Delphi code. Each of the different oil desks ran a PLServer and it went through all the day’s trades and calculated a profit and loss figure after doing all valuations. These were run by traders every day between 6.00 PM and 8.00 PM and some could take an hour to run due to the volumes of trades involved.
The end-of-day P & L figures were passed onto controllers and made available next day to the bank’s board. So not producing these figures was out of the question. All went well for several years until we found “the $800 million bug”.
The $800 Million Bug!
I maintained the PlServers. They had all sorts of logging built in and I could usually find and fix bugs very quickly; it ran perfectly 99.9% of the time. But one day we got a message that a particular desk’s PLServer had produced $800 million more profit than was expected the night before. This was noticed very quickly at the bank’s board level and there was a lot of pressure to get it sorted fast. All that profit sounded great, but it was a totally false figure. The controllers had to make a manual adjustment and expected me to find and fix it.
For the next few evenings, I ran that desk’s PLServer on my desktop computer, directly in the Delphi IDE as soon as the trader Ok’d it. It was a bit naughty running in production that way but the urgency required it. It took a few days to find because I could only run it once each day. I was using a divide and conquer technique to home in on it.
Did you track down the bug?
Eventually I caught it; it was a divide by zero exception that skipped over a very important piece of code. This particular trade it was valuing was using Japanese Yen and it needed a division to convert to dollars. The FX rate was something like 140 Yen to the dollar.
E.g. Say the Trade was valued at 10,000 YEN. In dollars the value := Value_in_Yen/Yen_rate; That should have been $71.25 dollars. Instead, the exception wasn’t caught until further on; it missed the FX division and the value was treated as dollars i.e., $10,000 so it was 140 times too big.
The divide by zero is the kind of bug that shouldn’t ever happen. It turned out that a new type of trade had been introduced by traders without telling us and this trade had a trade length of zero days and that caused the exception. No trade before that had ever had a day length of zero, so there was no divide by zero check on a division that divided by the day length. I added checks to divisions, tested the fix and released it to production next day.
How did Delphi help with tracking down an $800 million dollar bug?
Delphi simplified the task of debugging code. Even a large application like these could have a code change, compile and be running again within a minute. A colleague of mine programmed in C++ and it would have taken him at least five-ten minutes to edit, compile and rerun. He was envious of the faster compile time.
COM Servers and using Microsoft’s Dynamic Virtual Channels with Delphi
Traditionally, developing any application that used COM (Windows Component Object Model) was something you did in C++ but it’s a fairly advanced subject and the syntax is horrible. Delphi is the only other programming language that lets you create complex applications using COM such as COM Servers, automation servers etc. Apart from the Delphi type library editor being a little bit tricky at times, I’ve found it to be a lot simpler to work with than in C++.
I successfully implemented Microsoft’s Dynamic Virtual Channels which involves creating a multi-threaded COM server that implements three Interfaces; see Virtual channels. This lets programs running in an RDP session talk to programs on your computer. I wrote it all in Delphi and it has been successfully deployed.
This article was submitted as part of our Enterprise Article Showcase. If you have a success story to tell about a project which makes use of RAD Studio with Delphi or C++ Builder, or any of our other great enterprise products please get in touch, you could be featured too and win a prize! Read all about it here: Enterprise Article Showcase