Challenge: Perform all testing on a new “screen-based trading” system designed to process millions of transactions per hour to make markets for hundreds of thousands of complex regulated derivatives. The exchange’s legacy test process averaged about one test case per hour. This was a non-starter: even with 10 times more budget, it couldn’t have covered 1/1000th of the complex derivative scenarios the new system offered or evaluate its performance under realistic and varying loads.
Solution: Devised and developed a completely original model-based testing system using discrete-event simulation and AI business rule evaluation. Identified ten load profiles (graph at right shows the “typical day” profile.) This multi-dimensional approach generated over 200 completely realistic six hour trading sessions (each with hundreds of thousands of transactions) as tests. Developed an adapter framework with instances for all interfaces including trader screens, a new CORBA API for high-volume program trading, and several legacy feeds. Developed rule-based oracle to automatically check the results of every test trade, allowing evaluation of millions of test orders and quotes.
Results: This multi-dimensional approach achieved highly realistic performance and stress tests and a huge improvement in test productivity and effectiveness. Revealed about 1,500 bugs over two years, of which 5% were showstoppers. The last pre-release test run applied and evaluated 500,000 fresh test cases in two hours, with no failures detected. Post-release, no failures were reported in the first six months of operation. The system was later used to prove that integration with another exchange was feasible and scalable, leading to a multi-billion dollar venture for joint operations.