Edward John Pring

Implementing a high-volume, low-latency market data processing system on commodity hardware using IBM middleware

A stock market data processing system that can handle high data volumes at low latencies is critical to market makers. Such systems play a critical role in algorithmic trading, risk analysis, market surveillance, and many other related areas. We show that such a system can be built with general-purpose middleware and run on commodity hardware. The middleware we use is IBM System S, which has been augmented with transport technology from IBM WebSphere MQ Low Latency Messaging. Using eight commodity x86 blades connected with Ethernet and Infiniband, this system can achieve 80 μsec average latency at 3 times the February 2008 options market data rate and 206 μsec average latency at 15 times the February 2008 rate.

The X-flex cross-platform scheduler: who's the fairest of them all?

We introduce the X-Flex cross-platform scheduler. X-Flex is intended as an alternative to the Dominant Resource Fairness (DRF) scheduler currently employed by both YARN and Mesos. There are multiple design differences between X-Flex and DRF. For one thing, DRF is based on an instantaneous notion of fairness, while X-Flex monitors instantaneous fairness in order to take a long-term view. The definition of instantaneous fairness itself is different among the two schedulers. Furthermore, the packing of containers into processing nodes in DRF is done online, while in X-Flex it is performed offline in order to improve packing quality. Finally, DRF is essentially an extension to multiple dimensions of the Fair MapReduce scheduler. As such it makes scheduling decisions at a very low level. X-Flex, on the other hand, takes the perspective that some frameworks have sufficient structure to make higher level scheduling decisions. So X-Flex allows this, and also gives platforms a great deal of autonomy over the degree of sharing they will permit with other platforms. We describe the technical details of X-Flex and provide experiments to show its excellent performance.