CoinTossX: An open-source low-latency high-throughput matching engine
CCoinTossX: An open-source low-latency high-throughput matching engine
Ivan Jericevich a , Dharmesh Sing b , Tim Gebbie a,b a Department of Statistical Sciences, University of Cape Town, Rondebosch 7700, South Africa b Department of Computer Science and Applied Mathematics, University of the Witwatersrand, Johannesburg 2000, South Africa
Abstract
We deploy and demonstrate the CoinTossX low-latency, high-throughput, open-source matching engine with orderssent using the Julia and Python languages. We show how this can be deployed for small-scale local desk-top testingand discuss a larger scale, but local hosting, with multiple traded instruments managed concurrently and managed bymultiple clients. We then demonstrate a cloud based deployment using Microsoft Azure, with large-scale industrialand simulation research use cases in mind. The system is exposed and interacted with via sockets using UDP SBEmessage protocols and can be monitored using a simple web browser interface using HTTP. We give examples showinghow orders can be be sent to the system and market data feeds monitored using the Julia and Python languages. Thesystem is developed in Java with orders submitted as binary encodings (SBE) via UDP protocols using the AeronMedia Driver as the low-latency, high throughput message transport. The system separates the order-generationand simulation environments e.g. agent-based model simulation, from the matching of orders, data-feeds and variousmodularised components of the order-book system. This ensures a more natural and realistic asynchronicity betweenevents generating orders, and the events associated with order-book dynamics and market data-feeds. We promotethe use of Julia as the preferred order submission and simulation environment.
Keywords:
Market matching-engine, Open-source application, Microsoft Azure deployment, Java web application,Julia, Python, Agent-based modeling, JSE
Contents1 Introduction 12 Market simulation for testing 23 Realistically simulating high-concurrency 34 CoinTossX 4
Email addresses: [email protected] (Ivan Jericevich), [email protected] (Dharmesh Sing), [email protected] (Tim Gebbie)
Appendix B.1 Java . . . . . . . . . . . . . 18Appendix B.2 Julia . . . . . . . . . . . . . 19Appendix B.3 Python . . . . . . . . . . . 20
1. Introduction
A complete study of the market microstructure of the Jo-hannesburg Stock Exchange (JSE) is not possible with-out access to their matching engine . Studying marketmicrostructure is challenging due to the various changesin the market, regulation and technology. However, mostof the current literature focuses on analyzing existing ex-changes and the building of agent based models. The im-portance of order matching engines in the trading infras-tructure makes these systems of interest not only to com-puter scientists but also to computational finance and riskmanagement, while the non-linear impact of event-drivenprocesses relating to order matching may provided an im-penetrable calibration boundary for agent-based modelsattempting to empirically relate temporal dynamics withspecific agent behaviours [1, 2, 3].A trade matching engine is the core software and hard-ware component of an electronic exchange. It matchesup bids and offers to complete trades. Electronic ordermatching was introduced in the early 1980s in the United A matching engine is component of an exchange that matchesbuy and sell orders according to the rules of the exchange.
Preprint submitted to SoftwareX February 23, 2021 a r X i v : . [ c s . D C ] F e b MARKET SIMULATION FOR TESTING
States to supplement open outcry trading . Before this,stocks where traded on exchange floors and transactioncosts where high. Failures in these systems increased asthe frequency and volume on the electronic networks in-creased.Modern matching engines are fully automated and useone or several algorithms to allocate trades among com-peting bids and offers at the same price. They typicallysupport different order types and have unique APIs oruse standard ones . As it pertains to trading, latency directly influences the amount of time it takes for a traderto interact with the market.Traditionally, to achieve low latency, high-frequency trad-ing has required powerful server hardware appropriatelynetworked in a data center, scaled to accommodate worst-case network traffic scenarios on the busiest trading days.These trading systems must be resilient in the face ofnetwork or power failures, requiring expensive redundanthardware as well as offsite data retention [4]. For ex-ample, firms would use co-location, fibre-optic networkcables, optimized hardware architectures and other tech-nology to get as close to zero latency as possible [5]. Onthe other hand, cloud-based software solutions benefit bybeing resilient and offering cost-effective, easy scaling —an advantage that is not offered by traditional tradingsystems. However, the biggest challenge for latency ina cloud-based environment, and one of the greatest bar-riers to building high-frequency trading systems in thisenvironment, is the fact that hardware is not co-locatedwithin a data center [4]. That said, a combination ofthe low latency of traditional matching engines and theresiliency, scalability and availability of cloud-based en-vironments is something that is yet to come about .A low latency high throughput matching engine doesnot exist for academics to further their understanding inthis field [6]. CoinTossX provides an environment for theapplication of agent-based modelling experiments whichwould otherwise be prohibitive to undertake in real finan-cial markets due to cost, complexity and other factors. Itwas for this reason that CoinTossX was developed andwhy it’s applications are studied further here. Here we A method of communication between professionals on a stockexchange or futures exchange typically on a trading floor Vendors include Connamara Systems, Cinnober (acquired byNasdaq),
Aquis Technologies (A2X) , SIA S.p.A., Nasdaq,Match-Trade,
MillenniumIT (JSE) , GATElab Ltd (acquired byLondon Stock Exchange), Eurex, LIST, Stellar Trading Systems,Quodd, Baymarkets, Market Grid, ARQA Technologies, Kappsoft,Thesys Technologies Latency refers to the ability of a system to handle data messageswith minimal delay. There is however a strong argument to place the Order Manage-ment System (OMS), that decides what to trade - the selection ofparent orders and execution strategies, in the Cloud; while retrain-ing the Execution Management System (EMS), that implementschild orders, in close proximity to the matching engine. Here weare moving the matching engine into the Cloud. High throughput refers to the ability of a system to process avery high volume of data messages. hope to provide an easy-to-use, openly available, realis-tic, real-time, simulated trading system that is straight-forward to set-up on multiple platforms. Hence, this pa-per benefits institutions, academics and others seekingto take advantage of an open-source, resilient, scalablematching engine simulator that may avoid a variety ofconflicts of interests related research or insights derivedfrom commercial alternatives.This paper is structured as follows: Section 2 and Sec-tion 3 outline the existing literature on the topic. Sec-tion 4 is dedicated to an outline of CoinTossX. Morespecifically, Section 4.1 gives a detailed description of thestructure and software construction. Section 4.3 presentsthe capabilities and features of CoinTossX. Section 4.4gives the details of the extensive tests conducted to ensurethe system meets the stringent latency and throughputrequirements. Supplementary to the testing framework,Section 5 shows the results of the simulation of a simpleHawkes process for generating large volumes of marketand limit orders. Section 6 ends with some concludingremarks and, lastly, Appendix A provides instructionfor users to deploy and use the application on their localmachine or on a remote server.
2. Market simulation for testing
Simulating the entire financial market ecosystem for tradestrategy testing and risk management is appealing be-cause of the mechanistic complexity of the market struc-ture and costs and the nonlinear feed-backs and interac-tions that multiple interacting agents bring to the marketecosystem. This type of simulation can be both a finan-cially costly, as well as computational expensive exercise;yet it appears tractable, and subsets of the ecosystem areused for system verification e.g. in vendor provided testmarket venues.The rapid evolution of software and hardware and the in-creasing need to reduce transaction costs, system failuresand transaction errors has led to important changes inmarket structure and market architecture. These changeshave supported the rise of electronic, algorithmic, andhigh frequency trading. The specifics are unique to eachand every market and regulatory environment. For ex-ample, in the South African context the JSE uses theMillenniumIT trading systems following the approach ofthe London Stock Exchange with the BDA broker-dealerclearing system [7] with rules and regulations set by theFinancial Markets Act (2012), the JSE rules and direc-tives, and the Financial Intelligence Centre (FIC) Act(2001). However, CoinTossX is fully configurable. Al-though implemented and tested here using the publishedand publicly available JSE market rules and test-cases, itcan be configured for a rich variety of market structures.Many researchers developing trading strategies do nothave access to vendor provided testing environments,while those that do often do not want to expose theirstrategies to competitors in shared testing environments.2
REALISTICALLY SIMULATING HIGH-CONCURRENCY
However, sufficiently realistic multi-agents simulation en-vironments remain illusive, particularly for low liquidity,and collective behaviour based risk-event scenarios [8] be-cause a key component remains the realism of the un-derlying trading agents and their interactions within testmarkets.Most agent-based simulation environments and modelsare over simplified to the extent of not being relevant forrealistic science and risk management. Some examplesinclude using a global calendar time to sequentially ordertrading events, or providing intentional (and sometimesunintentional) market clearing events that synchroniseprice discovery and information flow with agent interac-tions in terms of a unique global time – the loss of high-concurrency and cohersion in favour of tight-coupling forcomputational convenience.There are many examples, in the context of South Africanmarkets. Nair [9] prototyped a simple matching engineusing a single stock and adopting the standard ordertypes and mechanisms associated with a continuous dou-ble auction market as specified in the JSE. Althoughworks like this seem to provide a simplified but realisticframework demonstrating the over-all principles of cou-pling a matching engine with agent-based modeling ofa stock market; such frameworks are unlikely to recoverrealistic dynamics or lead to useful insights when interro-gated at the level of asynchronous but high concurrencyorder-book events and dynamics. High-concurrency anda careful management of the concept of time [1] seem aprudent requirement related to both the stylised facts ofthe market, as argued for in many South African marketexamples [8], but more importantly, for realistic strategytesting and risk management.Here like-for-like message delays, order-book rules, andboth asynchronous order matching as well as reactive,asynchronous, and high-concurrency agent (for the rulesand behaviours) and actor (for the computational rep-resentations and causation models) generation can affectmodel outcomes and estimation. It is the asynchronyand order-splitting at the agent level that can dominatethe emergence of auto-correlations and cross-correlationsin various traded assets because there is no single calen-dar time related mechanisms that generates equilibriumprices, can synchronise information and order-flow, or canco-ordinate machine time events, with sequential calen-dar time [1].
3. Realistically simulating high-concurrency
The LMAX Exchange (London Multi Asset Exchange)[10] is an FX exchange with an ultra low latency match-ing engine. Thompson et al. [10] developed the Disruptorring buffer for inter-thread concurrency as an alternativeto storing events in queues or linked lists. This was inresponse to the problem that linked-lists could grow andincrease the garbage in the system — causing significant costs to latency and jitter . At the heart of the disrup-tor mechanism sits a pre-allocated bounded data struc-ture in the form of a ring-buffer. The Disruptor preallo-cates memory in a cache-friendly manner which is sharedwith the consumer. The system was also developed onthe JVM and can process up to 25 million messages persecond on a single thread with latencies lower than 50nanoseconds [10].Recently, Addison et al. [4] implement a simple foreignexchange (FX) trading system and deploy it to cloud en-vironments from multiple cloud providers (Amazon WebService, Microsoft Azure and Oracle Cloud Infrastruc-ture), recording network latency and overall system la-tency in order to assess the capability of public cloudinfrastructure in performing low-latency trade executionunder various configurations and scenarios. They con-clude that sufficiently low latency and controlled jittercan be achieved in a public cloud environment to supportsecurity trading in the public cloud [4]. More specifically,they demonstrate the ability to achieve sub-500 microsec-ond roundtrip latency — therefore concluding that it iscurrently feasible to build a production low-latency, high-frequency trading system in the cloud.For research in finance, economics and computer science,the importance of having a realistic, flexible, high perfor-mance matching engine deployable to different environ-ments can be found in a wide range of fields. In par-ticular, given the whole new range of realism that sucha software provides, agent-based computational financeand financial models in general are some of those thatmay show the greatest potential in terms of the insightsto complex systems that can be gained from their applica-tion. This is not to mention the additional class/layer ofcausation that is introduced in the modelling frameworkby having the complex set rules of the environment/sys-tem/architecture be separate/independent from the mod-elling and decision making processes (at the agent level).In this way more emphasis can be placed on top-down ac-tions and states — a potential step towards hierarchicalcausality [11].On this note, CoinTossX is a simulation environment andso the task of producing realistic simulated market dy-namics, comparable to those observed in empirical inves-tigations, is left to the user(s). For this purpose the twopopular methods usually adopted are mutually excitingHawkes processes (see Section 4.4) and agent-based mod-els (ABMs) [12].ABMs provide a bottom-up approach to modelling theactions and interactions of autonomous agents with theaim of assessing their effect on a complex system. Propo-nents of agent-based models argue that financial markets Jitter is the deviation from true periodicity of a presumablyperiodic signal, often in relation to a reference clock signal. Thisvariation in the time between data packets arriving is caused bynetwork congestion. COINTOSSX exhibit many emergent phenomena, and that such phe-nomena are usually attributed to the interactions and re-lationships between the agents that make up the system.Successfully calibrating ABMs to financial time series [13]can allow for inference about the factors determining theprice behaviour observed in the real world, provided thatparameter estimates are sufficiently robust.In recent years, ABMs became popular as a tool tostudy macroeconomics — specifically, the impact of trad-ing taxes, market regulatory policies, quantitative eas-ing, and the general role of central banks. ABMs canalso play an important role in analysis of the impact ofthe cross-market structure [12]. Of particular interest isthe class of models described by Lussange et al. [12] whooutline a computational research study of collective eco-nomic behavior via agent-based models, where each agentwould be endowed with specific cognitive and behavioralbiases known to the field of neuroeconomics, and at thesame time autonomously implement rational quantitativefinancial strategies updated by machine learning (and re-inforcement learning (RL)).ABMs are not without their pitfalls [14, 12, 13]: computa-tional cost, validation and calibration challenges, a biastowards inductive decision making, endemic parameterdegeneracies, and their tendency to focus on mechanisticbehaviours and rules, rather than interaction dynamicsand learning . Despite these challenges, their potentialto link the micro-level rules of investors behaviour withthe macro-behaviour of asset prices in real market is com-pelling, in part due to the apparent expansive amountof data that is collected from financial markets. How-ever, a key concern remains the scientific costs, or impact,of losing the realism of high-concurrency adaptive inter-actions between strategic agents in a sufficient reactiveframework, for the computational convenience of time-synchronised bottom-up rule based approaches. Here wehave decided to first focus on the matching engine frame-work, and separate it entirely from the agent generationframework; this may ensure that high-concurrency andlow latency features of real markets are not lost.
4. CoinTossX
CoinTossX is a an open-source, high-frequency, low la-tency, high throughput matching engine for simulatingthe JSE [6, 5, 15] (or any market that has well establishedrules and test cases). The software was developed withJava and open-source libraries and is designed to max-imize throughput, minimize latency, and accommodaterapid development of additional functionality. It can beconfigured for multiple clients, stocks and trading sessions(continuous trading, opening auction, closing auction, in-traday auction and volatility auction). The software hasbeen configured to replicate the rules and processes ofthe JSE. It may allow traders, organizations and aca-demic institutions to test market structure, fragility anddynamics without the cost of live test trading. The soft-ware can provide a platform to study price formations in stock exchanges and the interplay between regulators,market structure and dynamics [6]. This work also ad-dresses some aspects of the unavailability of data and di-rect data-feed access from industry, by providing a frame-work that can be compared to recorded transaction dataarising from the actual market system interfaces.The system requirements we implemented were obtainedfrom JSE’s publicly available technical documentationand test cases [16, 17, 18] .The eight main components of the simulator are [6]:1. Stocks are objects for which clients can send ordersand limit order books can be constructed and kepttrack of.2.
Clients are computer algorithms that send orderevents to the simulator. Clients will send orderevents to the trading gateway (refer to Section 4.2for more detail).3. The trading gateway receives the client request,validates the request and then sends it to the match-ing component to be processed. It sends updates tothe client to indicate the status of the event.4. The matching engine processes the events fromthe Trading Gateway. It manages one or more limitorder books. If there is an update to the LOB, itsends updates to the market data gateway.5. The market data gateway receives updates fromthe matching engine component and sends marketdata updates to all connected clients.6. The website receives updates from the market datagateway. It displays the LOBs for each security andallows the user to configure the stocks and clients.7. The web-event listener was specifically incorpo-rated for the purpose of reducing pauses to the web-site due to garbage collection. This single, dedicatedwriter principle would improve performance of thewebsite as well (refer to the discussion on the web-site in Section 4.1). This component also saves eachevent to the database8. The database stores the LOBs of each stock and isdesigned to have low memory overhead whilst beingefficient in searching, updating and deleting orders(refer to the discussion on off-heap data structuresin Section 4.1). COINTOSSX 4.1 Architecture
Changetradingsession Login Logout OrderrequestTrading gateway ClientUserWebsiteManageclients Managestocks ViewLOBs Runtestingframework Orderrequest Changetradingsession MarketdataupdateMatching engineMarketdataupdateMarket data gateway
Figure 1: A High level diagram of the relationship between thecomponents of CoinTossX in terms of end-user functionality [6].
A website is required to monitor and configure the stocksand clients. The relationships between these componentsare also summarised in Figure 1. The client sends a loginrequest message to the trading gateway which then vali-dates the client. If the client is already logged in or theusername or password is invalid it responds with a rejectcode message. The client can also send a log out requestto the trading gateway. In this case the trading gatewayremoves the client from the list of connected clients andsends a response to the client to indicate if the log out wassuccessful. The trading gateway logs out clients when itshuts down. When there is a change in the trading ses-sion, the website sends a message to the trading gatewaywhich then sends the message to the matching engine.Clients can send an order request or order message tothe trading gateway which then validates the message.If the message is valid, it then sends the message to getmatched. Valid order types are market, limit, hidden,stop and stop limit. Each order has a time in force (TIF)value, which cannot be amended, that determines howlong an order is active until it is executed, deleted or ex-pired (which ever comes first). The engine matches theorders using the matching algorithms based on the trad-ing session. Market data updates are sent to the marketdata gateway to indicate the changes in the LOB of eachstock. Additionally, the website and all clients receivemarket data updates and internal messages (to monitorthe application) from the gateway. A user can create, up-date and delete clients and stocks. A user can view thelimit order book for each stock as a bar chart of all activeorders. Tables will show the details of these active orders.A user can start, stop and configure the parameters of thetesting framework.
This software is open source and was built to run on dif-ferent operating systems as well as on a single server oron multiple servers. Implementation in Java means thatCoinTossX can be deployed on different hardware con-figurations (due to the JVM). The goal was to build amatching engine that could achieve low-latencies usingindustry standard technology. This ensures that the soft-ware is portable and can be deployed confidently. The high level architecture of CoinTossX is summarised inFigure 2.A single market event goes through the following pro-cess. A simple binary encoding (SBE) message will besent to the trading gateway. The trading gateway willforward the message to the matching engine which willthen process it. It will send a message back to the tradinggateway to indicate the status of the message. The trad-ing gateway will forward the message back to the client.The matching engine will send an update to the marketdata gateway if there is a change in the limit order book.The market data gateway will forward the updates to allconnected clients and will forward some of the updatesto the web event listener. The web event listener saveseach event to the file system. The website then reads anddisplays the data from the file system.The communication between the website and the file sys-tem is done by transmitting data over a network usingonly User Datagram Protocol (UDP) as opposed toTransmission Control Protocol (TCP). TCP is a heavy-weight protocol because it requires three packets to setup a connection and requires a connection to be set upbetween two applications before data can be transmit-ted . UDP, on the other hand, was chosen for beinglightweight protocol as it does not check the connectionor the order of the messages and does not require a con-nection to be setup before data is transmitted. Data istransmitted irrespective of whether the receiver is readyto receive the message or not. UDP may, however, beunreliable because the sender does not know if the mes-sage was delivered. The messages that are received (readas a byte stream) will always be in the order that it issent. Modularisation is achieved by designing each of theabove components independently of one other while al-lowing communication between them to occur only viaexposed ports using these high speed SBE message pro-tocols. This functionality was introduced by assigningeach component an IP address and port on which to lis-ten. These ports are specified in the “properties” files inthe root project directory. Furthermore, each client/usermay submit orders from a remote server. So if Coin-TossX is deployed to the cloud, one should ensure thatthe virtual machine allows for these types of inbound portcommunications.After an extensive comparison of message transport soft-ware, the Aeron media driver library was chosen for sup-porting the above protocols being an efficient and reliable JSE uses text messages between the clients and their matchingengine which is inefficient since characters take up more memoryand are slower to transmit across the network. From the compar-isons made in [6], the SBE message protocol was found to be thefastest way to encode and decode messages; and had the greatestthroughput. JSE uses TCP for their trading gateway and UDP for theirmarket data gateway. Nonetheless, TCP can be reliable because if a message is notreceived, it will try multiple times to deliver the message. TCP willdrop the connection if there are multiple timeouts. COINTOSSX 4.1 Architecture
UDP unicast, UDP multicast, and IPC message transportfor communicating between all the components. Eachcomponent has its own media driver to shovel events toand from the component. Aeron is designed to be thehighest throughput with the lowest latency possible ofany messaging system [19] (see Figure 3).Given this modular design, each component and clientmay be started and run on seperate, independent servers.As mentioned, components communicate with each otherthrough the media driver via UDP SBE message packets.Therefore. it is essentially possible to have an industrialmatching engine by providing each component with itsown high performance server.
Client { ,...,N } (cid:143) Media driver J Media driver J Media driver J Trading (cid:4) gateway (cid:5)
Matching engine Market data gateway ‘ Media driver J Web event listener (cid:211)
File systemWebsite … Figure 2: A High level architecture diagram visualising the technicalrelationship between the software components [6].Figure 3: A High Dynamic Range (HDR) Histogram graph showingthe latency percentiles of Aeron’s PingPong test. This provides alower-bound for the achievable latency of CoinTossX as well as foropen-source message transport libraries in general [19].
The matching engine is the component which relies heav-iest on the latency and throughput of the system andis therefore designed with these features in mind whilststicking to the JSE matching engine’s architecture. Thesoftware was designed such that different matching logicalgorithms are in separate Java classes. This allows thelogic to be changed to test any variations of the matchinglogic. The matching engine adopts the price-time priorityalgorithm during the continuous trading session. That is,for multiple orders occurring at the same price, a marketorder will be matched with the order having the earli-est submission time. The trading session execution times and monitoring was not implemented in the componentsas this would reduce the throughput and increase the la-tency.Limit and market orders submitted during a call auctionare not matched immediately. These orders are matchedat the end of the call auction at a single price using aprice discovery process. The Volume Maximizing Auc-tion Algorithm finds the price that will match the mostnumber of buy and sell orders. More specifically, dur-ing both the continuous trading and auction call sessions,the matching engine processes orders as follows. The fil-ter and uncross algorithms run each time the Best Bid orOffer (BBO) changes or every 30 seconds. The filter algo-rithm uses the heuristic Hill Climber search/optimizationalgorithm to find the optimal volume of hidden limitorders that can be executed. The search will filter outhidden limit orders with MES constraints that are not el-igible. After the filtering, specific rules are used to selectthe orders and price to executed in the crossed region.As in the JSE, the matching engine uses native messageprotocols to communicate with clients as opposed toFIX (Financial Information eXchange), FAST (FIXAdapted for STreaming), ITCH or OUTCH . Thematching engine component stores the active orders inmemory since storing the active orders on disk would in-crease the I/O and reduce performance. The data struc-ture for storing the LOB is designed to have a low mem-ory overhead (lists, hash tables, trees, B-tree, B+tree)and be efficient in searching, updating and deleting or-ders [6]. More importantly, the low latency is achievedthrough the efficient use of CPU cache. The simulatoronly uses main memory without the use of virtual mem-ory . The time taken for data to move from the CPUto main memory is reduced by using caches that containcopies of frequently used data from main memory.The website was developed using Spring Boot and ApacheWicket and only has the permission to read from the filesystem. The original design had the event listener andwebsite in the same Java process. When the website wasused or paused because of garbage collection, it affected The Hill Climber algorithm is an optimization technique thatiteratively searches for the solution to a problem by changing oneelement in each iteration Message protocols are the methods by which the exchange com-municates with market participants. Native protocols are custombuilt protocols. FIX is an open standard non-propriety protocol that is usedby buy and sell firms, trading platforms and regulators to transmittrade data. The protocol allows organizations to easily communi-cate domestically and internationally with each other. Greater numbers of market participants and therefore volumeincreased the network latency and lead to market participants notreceiving updates in an acceptable time. The FAST protocol re-duces latency by encoding and compressing the data before trans-mission. ITCH is used to publish market data only. OUCH is used to place, amend and cancel orders. Virtual memory is space on the hard disk which is used by anoperating system when it requires more memory. The time to fetchdata from virtual memory is very slow and is therefore not used. COINTOSSX 4.2 Clients the receiving and saving of events. Therefore this logicwas split into a separate component — the web eventlistener. The listener could keep the received events inmemory or save it to the file system. Saving the datain memory would be fast, but would require an unknownmaximum memory setting. Therefore the data needs tobe saved to the file system. Using the MapDB library ,an off heap hashmap is used to save the events to memorymapped files. An off heap hashmap stores data outsidethe Java heap space and is not affected by garbage col-lection. Off heap data is suited for storing data largerthan the current memory and allows sharing of data be-tween JVMs. Memory mapped files allow Java programsto read and write files using only memory while the op-erating system reads and writes to the file system. Thissignificantly improves performance. The entire file or apart of the file can be loaded into memory. The valuesin memory will still be written to the file system even ifthe JVM crashes. The web event listener has read andwrite permissions. It saves events to the file system butreceives events faster than it can save it to file. This issueis solved with the Disruptor by having two threads: oneto receive and store events, and the other to save eventsto the file system [10]. Important to the design of this trading system are theclients who send orders to the trading gateway and re-ceive updates from the website and market data gateway.Each client is assigned input and output URLs for boththe Native Gateway and Market Data Gateway. TheseURLs specify the IP addresses (in this case localhost -the user’s local machine) to/from which messages will besent/received as well as the ports on which these com-ponents are listening. Therefore each client takes up anumber of threads on the machine it runs on — whichcan be the same or different from the machine runningthe other components of the matching engine. So, hav-ing clients send orders to the server remotely would freeup hardware and improve performance on the matchingengine server.It is also important to note that each client here is aunique trader having its own unique ID, password, portsand login credentials. However, as was done in the testingframework, a single client may still be used to simulatemultiple traders. This would be preferable in the casewhere the client(s) and other components run on the samemachine, thereby limiting the computational resources re-quired for simulations. In that case, the client is simply atool/object for submitting orders to the gateway. In thisway one can have a self-contained simulation frameworkbe independent from the actual implementation or send-ing of orders by, for example, defining an ABM in julia MapDB is an open-source, embedded Java database engineand collection framework. It provides Maps, Sets, Lists, Queues,Bitmaps with range queries, expiration, compression, off-heap stor-age and streaming. with each agent holding a reference to the client objectthat sends orders.The number of client-stock pairs that can be supported intotal is only limited to the performance capabilities of theserver on which the matching engine is running (see sec-tion 4.4 for hardware recommendations and performanceresults with differing number of client-stock pairs). Thelimits of the number of clients that can be supported byeach stock, however, still needs to be explored and mea-sured with respect to different hardware configurations.
When the user starts CoinTossX, through the website,three main screens can be switched between and dis-played. The stock screen shows the stocks configured,the trading session that is active for each stock and abutton to view the limit order book of the stock. In thelimit order books for each stock a bar chart will be gener-ated to display the active orders. The market data shownon the LOB pages for each stock are snapshots represent-ing the current state of the limit order book. Tables willshow the details of the bids, offers, trades and all sub-mitted orders. All data on these pages can be exported.The Hawkes configuration page (see Section 4.4) allowsthe user to change the values of the Hawkes input databefore running the simulation. The simulation page al-lows the user to stop and start the warmup process andthe Hawkes simulation. It shows the status of each clientand the active trading session. The clients screen showsthe clients that are configured and allows the user to cre-ate, update and delete clients. Clients not configured willnot be able to log in to the trading gateway. In order fora client to submit orders, a login request, with the rel-evant username and password, must first be sent to thetrading gateway. Similarly, log-out request must be sentwhen the client logs out. Thereafter the client can sub-mit orders by publishing order messages to the tradinggateway via UDP ports. Traders cannot submit ordersthat are smaller than the LOB’s tick size (controls thesmallest order size). The allowable order types that arenot conditioned on time are [16]:1.
Market orders (MO) – contains the quantity ofshares to trade, but not the price, and executesagainst each orders in the LOB on the contra sideuntil it is fully filled. Orders submitted during theauction call will remain in the LOB until uncrossingis done. All market orders that are not filled willexpire.2.
Limit order (LO) – displays the quantity and pricewhich may execute against a trade or expire basedon it’s time-in-force (TIF).3.
Hidden orders (HO) – allows traders to submitorders which are completely hidden from the market(price and volume). These have a lower priority thanall visible active orders. These orders can executeagainst other visible and hidden orders. Hidden limit7
COINTOSSX 4.3 Functionality orders that are submitted during an auction call arerejected. The quantity of a hidden order must meetthe minimum reserve size (MRS) . Each hidden or-der also has a minimum execution size (MES) . Af-ter a trade executes against a hidden order, if theremaining quantity is less than the MES (¿ MRS)then the order will expire. If the remaining quantityis greater than the MRS and MES then the orderwill only expire when it executes or based on it’s TIF(whichever comes first). Hidden orders may only besubmitted during the continuous trading session.4. Stop order and stop limit order (SO & SL) – a stop order is a market order with a stop price.These orders do not enter the order book (remainunelected) until their stop price is reached. Whenthe stop price is reached, the stop order becomes amarket order. Similarly, a stop limit order is a limitorder with a stop price. These orders do not enter theorder book until their stop price is reached. Whenthe stop price is reached, the stop order becomes alimit order. Stop and stop limit buy orders will beelected if the last traded price is equal to or greaterthan the stop price. Similarly, stop and stop limitsell orders will be elected if the last traded price isequal to or less than the stop price. An incomingstop or stop limit order may be immediately electedon receipt if the stop price has already been reached.These orders will also only be elected at the end ofthe execution of an order.Some of the above orders can also have a time-in-force(TIF) which cannot be amended after the order is placed.The following time in force options are available [16]:1.
At the opening (OPG) – only accepted duringthe opening auction and are used to direct ordersonly towards this session. OPG orders that are notfilled during the uncrossing will expire at the end ofthe opening auction. If there is no opening auctionscheduled, OPG orders will be rejected.2.
Good for auction (GFA) – orders of this typethat are submitted will be parked until the nextauction and are injected at the start of the auction.GFA orders that are not filled during the uncross-ing will be parked for the next auction and are onlyremoved when they are filled or cancelled. If thereis no auction session scheduled, GFA orders will berejected.3. Good for intraday auction (GFX) – These or-ders are parked until the next intraday auction andare injected at the start of the auction. As opposedto GFA orders, GFX orders that are not filled dur-ing uncrossing will expire at the end of the intraday The minimum order quantity for orders to qualify as hiddenlimit orders. The minimum quantity of the hidden limit order which is per-mitted to execute While orders are parked they cannot be executed. auction. If there is no intraday auction scheduled,GFX orders will be rejected.4.
At the close (ATC) – submitted orders of this typeare parked until the next closing auction. They areinjected at the start of the auction and if they are notfilled, will expire at the end of the closing auction.If there is no closing auction scheduled, ATC orderswill be rejected.5.
Day (DAY) – orders that expire at the end of thetrading day. If an order does not specify a TIF, itwill default to DAY.6.
Immediate or cancel (IOC) – can be partially orfully filled. An IOC that is only partially filled willcancel immediately after execution. IOC orders arerejected during auction calls.7.
Fill or kill (FOK) – this order type differs fromIOC orders in that they are either fully filled or ex-pired. Partially filled orders are not allowed.8.
Good till cancel (GTC) – can remain in the orderbook for a maximum of 90 calendar days or until theorder is filled or canceled.9.
Good till date (GTD) – as opposed to GTC or-ders, these orders remain in the order book until theorder is filled, cancelled or a specified expiration dateis reached (maximum active time is 90 days). GTDorders do not allow for the specification of an expi-ration time (only date).10.
Good till time (GTT) – remain in the order bookuntil the specified expiry time is reached in the trad-ing day. GTT orders that have an expiry time duringan auction will not expire until uncrossing is finished.11.
Closing price cross (CPX) – these orders areparked until the start of the closing price cross ses-sion. Unexecuted CPX orders are expired at the endof the closing price cross session. Stop and stop limitorders are not allowed to have a TIF of type CPX.Valid combinations of order types and TIF are shown inTable 1 below.8
COINTOSSX 4.4 Testing framework
TIF Market Limit Hidden Stop Stoplimit limitIOCFOKDAYGFAGFXOPGATCGTCGTDGTTCPX Accepted Rejected
Table 1: Valid order types and time-in-force (TIF) combinationsfor the submission of new orders to the trading gateway.
Lastly, CoinTossX also allows for multiple trading sessiontypes which correspond to that of the JSE. The tradingsessions below and the rules adopted by each are config-urable in the data/tradingSessionsCron.properties file. The times shown are those of the JSE, provided forcontext [16]. All trading session start and end times canbe specified by the user through the cron expressions inthe data directory. During the auction call sessions ordersare not matched immediately, rather, they are matchedat the end of the call auction at a single price using avolume maximizing price-discovery process. This processfinds the price that will match the most number or buyand sell orders.1.
Start of trading ( ) – no orders can besubmitted or executed executed during this session.Traders will be able to cancel, but not submit, ordersduring this session.2. Opening auction call sessions ( )3. Continuous trading session ( ) –the system will continuously match incoming ordersagainst those in the order book according to theprice-time priority execution rule.4. Volatility auction call session ( triggered ) – thissession will only trigger when a stock’s circuitbreaker tolerance level has been breached. volatil-ity auction call sessions last for a scheduled periodof 5 minutes. The orders accumulated during thissession will be executed at the uncrossing based onthe volume maximizing algorithm.5. Intraday auction call session ( )6. Closing auction call session ( )7. Closing price publication session ( )– no orders can be submitted or executed executedduring this session. Traders will be able to cancel,but not submit, orders during this session.8.
Closing price cross session ( ) – trading will only take place at the closing price thatwas published during the closing price publicationsession.9. Post close session ( ) – traders willbe able to cancel, but not submit, orders during thissession.10. Halt ( manually envoked ) – A halt session may beactivated by the user during which time no orderequests may be executed. Traders will however beable to cancel orders.11. Halt and close ( manually envoked ) – The be-haviour is the same as the halt session except closingprice calculations will be performed.12. Pause ( manually envoked ) – No executions will takeplace during the pause session. Traders will be ableto submit, amend or cancel orders during this ses-sion, however, market orders will expire at the endof the session.13. Re-opening auction call ( manually envoked ) –The user may manually invoke the re-opening auc-tion call session when resuming from a manual trad-ing halt or a trading pause.14. Trade reporting ( ) – trades andstatistics, where applicable, will be publishedthrough the market data gateways. CoinTossX has been successfully deployed locally as wellas to remote servers such as Microsoft Azure, CHPC andTW Kambule Mathematical Sciences Laboratories pro-vided servers (using 4 re-purposed legacy TACC Rangerblades, see Table 4). Deployment to high-performancecompute solutions such as UCT HPC was found to beinfeasible for a number of reasons: First, facilities suchas UCT HPC often perform computations by relying onan MPI approach using one of a variety of different jobmanagement systems where the submission of “jobs” to“worker nodes” is highly constrained by the preferencesof systems administrators managing many different usecases; the worker nodes are not equivalent to virtual ma-chines, rather they receive tasks from the “head node” tobe executed in parallel. Simplistically this means that, forexample, web interfaces will not be easily accessible fromworker nodes due to the system and network architecture- but this more generally impacts any client worker in-teractions when agents interact via clients through somecentralised architecture - here the matching engine; butthis can be any interaction landscape. In general, suchsystems are poorly suited for real-time and reactive usecases. This is because, by design, many high-performancecomputing facilities, such as UCT HPC or CHPC, are nothigh-throughput facilities. They are typically not wellsuited for large scale high-concurrency, low latency mar-ket and agent-based simulation problems that necessarily9
COINTOSSX 4.4 Testing framework
Order type/time-in-force Order type/time-in-forceSession OPG ATC IOC FOK GTC GTD GTT GFA GFX DAY CPX MO LO SO &SL HLStart oftradingOpeningauction callContinuoustradingVolatilityauction callIntradayauction callClosingauction callClosing pricepublicationClosing pricecross sessionPost closeHaltHalt andclosePauseRe-openingauction callFCO auctioncall sessionAccepted Rejected Accepted and parkeduntil injected Accepted and expired immediately ifthey do not execute upon aggression Carried forward fromthe previous day
Table 2: Valid trading session and order type/TIF combinations for the submission of new orders to the trading gateway. need to be highly reactive in nature. We believe thatthis is an important design perspective that is often notwell considered when designing advanced market (or so-cial science) simulators. The functionality of the software was tested using thetest cases made available online from the JSE. The re-quirements were taken from [16]. Testing of the trad-ing sessions were restricted to the continuous and intra-day auction trading sessions using only the DAY TIF.Unit tests were implemented to cover the testing of thefunctional requirements of the software while individualthroughput and latency performance tests were imple-mented in conjunction with the Java MicrobenchmarkHarness (JMH) to test methods whose performancewere critical. These tests, covering the majority of theapplication, provide a safety net to allow changes to bemade to the code without breaking existing functionality.The outputs of these tests were not compared to the JSEor another exchange as the data is not available by the Reactive is used here in the broad sense of system architecturesthat require the fast propagation of data changes and relationshipswithin a system with many clients, but require high coherence withlow coupling. JMH is a Java harness for building, running, and analysingnano/micro/milli/macro benchmarks written in Java and other lan-guages targetting the JVM industry. The JSE’s test environment is also closed anddoes not provide realistic order-book dynamics. This pa-per focuses mainly on the results of latency and through-put tests, however, the types and results of the extensivelist of tests performed can be found by referring to Singand Gebbie [6].
Matching engine integrity was evaluated using unit testsand, instead of an agent-based approach, simulations andtests aim to understand throughput and latency were car-ried out with a 8-variate marked Hawkes process [20, 21].This provides a flexible framework to simulate a marketdata feed with varying throughput, with full control overthe trade and quote conditional intensities. The soft-ware and Hawkes client processes were deployed on oneserver which affected the performance of all components.The mutually exciting processes correspond to 8 differ-ent order types that are considered in the testing frame-work. The testing framework only considers basic aggres-sive and passive market and limit orders as in Large [22]:10
COINTOSSX 4.4 Testing framework
Type Event Ask or Immediate Moves Descritpion (cid:51) (MO) (cid:51)
Market buy thatmoves the ask2 Trade Bid (cid:51) (MO) (cid:51)
Market sell thatmoves the bid3 Add Bid (cid:55) (LO) (cid:51)
Bid between quotes4 Add Ask (cid:55) (LO) (cid:51)
Ask between quotes5 Trade Ask (cid:51) (MO) (cid:55)
Market buy doesnot move ask6 Trade Bid (cid:51) (MO) (cid:55)
Market sell doesnot move bid7 Add Bid (cid:55) (LO) (cid:55)
Bid at or belowbest bid8 Add Ask (cid:55) (LO) (cid:55)
Ask at or abovebest ask
Table 3: Order event types used for the 8-variate Hawkes process inthe testing framework simulation following the approach of Large[22].
The simulation is conducted using the intensity-basedthinning algorithm as introduced by Lewis and Shedler[23] and modified by Ogata [24] and is used to definethe time at which orders arrive. Each thinning algorithmsubmitting large numbers of orders may be thought of asclients. So, in the case of the testing framework, a sin-gle client is associated with each stock and is meant torepresent a large group of traders investing in that stock.The prices and volumes are generated based on the ordertype in a fairly random manner.First, a maximum LOB depth M = 10 is specified. Alower limit for the price at which buy limit orders are gen-erated is set to L b = 25000. Similarly, an upper limit forthe price at which sell limit orders are generated is set to H s = 25057. The LOB is initialised with an initial limitand buy I b = 25034 and sell I s = 25057 order. Thereafterprices and volumes are generated according to a randomnormal distribution in these bounds. According to thisbids and asks can cross each other - which is not real-istic. Nonetheless it’s purpose is merely to demonstratethe applications ability. The VWAP is used to calculatethe price for an aggressive buy/sell trade (not the same asthe execution price) that executes against the LOB. Thatis, if an aggressive trade affects the first k highest/lowestlevels of the order book then the price is calculated as: (cid:80) ki =1 Price i × Volume i (cid:80) ki =1 Volume i For the next section, test scenarios were created to testthe performance of the software by evaluating the impactof multiple clients-stock pairs. That is, each stock is as-signed a unique client who sends high volumes of orders.By increasing the number of stocks and clients after eachsubsequent run, the volume of messages being processedincreases as well - during which time the throughput persecond and latency are automatically recorded. Theseperformance tests have been conducted on multiple com-puters, all with different operating systems and hardware specifications. Here the performance results are docu-mented by running the application on the two machineslisted in Table 4. The average start-up time of the webapplication is approximately 50 seconds.
Machine OperatingSystem Memory ProcessorsWITSMSSServer 64-bit LinuxUbuntu16.04-LTS 32GB 4 × Quadcore AMDOpteron 8356@ 2.3GHz (16 cores)StandardA4m V2Azure VM 64-bit LinuxUbuntu18.04-LTS 32GB 4 × Intel XeonCPU E5-2673 v3@ 2.4GHz (32 cores)
Table 4: Hardware specifications of machines used for throughputand latency testing used to compare the cloud solution (Azure) tothe dedicated physical hardware solution (WITS MSS Server blade).
Although not presented here, limit order book storagetesting was also conducted on the WITS MathematicalScience Support provided server hardware and demon-strated the ability to store thousands of orders at eachprice point. The design of the LOB also allows orders tobe added and removed easily
The latency was tested and visualized using the HdrHis-togram library . With every run, each client submitsapproximately 110 000 orders. The time it takes to pro-cess all these orders is then measured and compared be-tween machines. To reproduce Figure 4 simply re-run theHawkes simulations for the system to write latency andthroughput results to file.Figure 4 shows the latency results, in nanoseconds, forruns with differing numbers of active client-stock pairson each machine. The latency increases as the number ofclients-stock pairs increase - since each client is associatedwith multiple threads running processes in parallel. Dueto there being only 4 CPUs on the Azure VM, runningmore than 6 clients simultaneously was found to be in-feasible. For this reason the user should ensure that thehardware on the server is capable of supporting the de-sired number of clients. For the high spec Wits Server ma-chine the minimum and maximum latency (measured upto 10 clients) at the 90th percentile is 106ns and 248ns, re-spectively. Similarly, for the medium spec machine (witha maximum of 6 clients) the minimum and maximumlatency at the 90th percentile is 123ns and 393ns, respec-tively. Therefore, on a high end machine one can expectsub 250ns latency (with 10 clients), while with a mediumspec machine one can expect sub 400ns latency (with 6clinets) . HdrHistogram is designed for recording histograms of valuemeasurements in latency and performance sensitive applications.Measurements show value recording times as low as 3-6 nanosec-onds on modern (circa 2012) Intel CPUs JSE’s average round-trip colocation network latency is sub 100 SIMULATION RESULTS
Figure 5 considers the scenario where high volumes of or-ders are submitted to the trading gateway. One millionorders are submitted from a single client and comparedacross the two machines. For the Wits server, the la-tency is 735ns at the 90th percentile but maintains a sig-nificantly lower latency on average. On the other hand,the Azure server has higher latency’s on average with alatency of 964ns at the 90th percentile.
Figure 5: A comparison of latencies during a high volume scenario.One million orders were submitted by a single client on the Witsserver and Azure machines.
Table 5 shows the throughput per second as the numberof clients-stock pairs increase on each machine. To repro-duce the data in Table 5 simply re-run the Hawkes sim-ulations for the system to write latency and throughputresults to file. It should be noted that each client waitsfor market data updates before calculating the next or-der to send. The client also waits a few nanoseconds aspart of the Hawkes simulation. These delays reduced thethroughput of orders sent. The results show that as thenumber of clients-stock pairs increase, the throughput de-creases. The most significant decrease in throughput iswhen more than 4 clients and stocks are used. Beyond 6clients the hardware configuration of the Azure VM wasfound to be insufficient. In total, the high spec machinewas shown to be capable of processing more than onemillion orders in less than a 19 minute simulation periodwith significantly low latencies. On the other hand, themedium spec machine was capable of processing 450000and 560000 orders in approximately 44 minute and 2 hoursimulation periods for 4 and 6 client-stock pairs, respec-tively.
5. Simulation results
The results in this section relate to the simulation for asingle stock using a simple Hawkes process as a proof of microseconds and it’s matching engine has a latency of 50 microsec-onds [25]. concept. Simulation results are contained in three seper-ate .csv files for each security: one for market ordersonly, another for limit orders and the last for a snapshotof the limit order book at then end of the simulation.This data is only printed to file at the end of the sim-ulation once all logged-in clients have logged out. Theformat of these files are shown in snippets 6 and 7 below.The submission times are given by the UTC standard.
TradId ’s correspond to the
OrderId ’s of the limit ordersorder against which they were executed.
SecurityId,"OrderId","SubmittedTime","Price","Volume","Side"1,"1","2020-11-22 07:43:08.231","25034","1200","Buy"1,"2","2020-11-22 07:43:08.696","25057","1000","Sell"1,"3","2020-11-22 07:43:11.683","25056","3600","Buy"1,"4","2020-11-22 07:43:11.763","25050","2600","Buy"1,"5","2020-11-22 07:43:11.915","25048","1200","Sell"
Figure 6: File format for the output of simulated limit orders sent tothe trading gateway. This data is written to file after the simulationis complete and the client ends their session.
TradeId,"Price","Quantity","CreationTime"1,"25056","1200","2020-11-22 07:43:12.352"2,"25056","1700","2020-11-22 07:43:12.905"3,"25056","700","2020-11-22 07:43:12.906"5,"25057","100","2020-11-22 07:43:12.984"6,"25057","300","2020-11-22 07:43:13.115"
Figure 7: File format for the output of simulated market orderssent to the trading gateway.Figure 8: A Graph of the intensities of the 8-variate Hawkes processfor 50 seconds of the Hawkes simulation. CONCLUSION (a) Wits Server high specification machine (b) Azure medium specification virtual machineFigure 4: A High Dynamic Range (HDR) Histogram graph showing the latency percentiles of the matching engine for increasing numbersof stocks and clients. In the testing framework each client submits approximately 110000 market/limit orders.
Wits Server Microsoft Azureclient-stockpairs Duration Number oforders Throughput persecond Duration Number oforders Throughput persecond1 00:00:48.820 111646 2287 00:06:44.917 110825 2742 00:04:23.981 224562 850 00:19:38.576 222390 1894 00:12:02.204 448774 621 00:43:55.734 447878 1706 00:15:12.386 669331 733 02:01:27.494 564070 778 00:20:27.010 895080 729 - - -10 00:18:52.289 1120514 989 - - -
Table 5: Tabulated measurements of throughput per second with increasing numbers of client-stock pairs. Based on the Hawkes processparameters each client submits approximately 110000 orders. After each simulation the start and end times are published and used tocalculate the throughput per second. The hardware configuration on the Azure VM was found to be insufficient when defining more that6 clients. All data generated in this table can be acquired at the end of each simulation in the data directory of the start-up folder.Figure 9: A Graph of the limit and market orders submitted to thetrading gateway. As a proof of concept, the Hawkes simulation as-signs volumes and prices in a fairly random and unrealistic manneras the bid and ask are allowed to cross (negative spread). The sizeof the points are proportional to the volume of the order.
6. Conclusion
CoinTossX is a low latency high throughput stock ex-change. It is configurable and allows users to view the limit order book in real time as well as for multiple clientsto connect and send orders to the exchange. The ex-change supports multiple stocks and a variety of differenttrading session. Here Hawkes processes are used to pro-vide a simple but robust simulation of order arrivals forinfrastructure testing. The CoinTossX website can beenhanced to analyze the data that is processed. The ex-change provides a realistic platform for agent based mod-els exploration. The software was designed so that differ-ent matching logic algorithms are in separate Java classes.This allows the logic to be changed to test any variationsof the matching logic or market rules. Hence, additionalwork can be done to change the matching rules on the en-gine to test the impact of rules and regulation changes onthe limit order book and its dynamics. Since the compo-nents are de-coupled, their implementation language canbe continually and easily changed to support the latestframeworks [5]. Future work will be aimed at better un-derstanding the interaction dynamics from increasinglyrealistic approaches to point-process based simulations e.g. with trading clients based on Hawkes processes us-ing models such as those of Bacry and Muzy [26] andZheng et al. [27] to investigate the interactions of limit-order and market-order trading agents through a realistic13
EFERENCES order matching process. Ultimately this type of work isaimed at better understanding high-concurrency agent-based perspectives for market modeling and their impacton model calibration [2] and causation.
7. Acknowledgement
We thank Turgay Celik and Brian Maistry WITS Math-ematical Science Support and the TW Kambule Math-ematical Sciences Laboratories for allowing us to use 4former TACC Ranger blades for some of the simulationwork. We thank Patrick Chang, Dieter Hendricks andDiane Wilcox for various discussions and advice with re-gards to the project.
References [1] P. Chang, E. Pienaar, T. Gebbie, The epps effect under al-ternative sampling schemes, 2020. URL: https://arxiv.org/abs/2011.11281 . arXiv:2011.11281 .[2] D. Platt, T. Gebbie, Can agent-based models probemarket microstructure?, Physica A: Statistical Mechan-ics and its Applications 503 (2018) 1092–1106. URL: . doi: .[3] K. Goosen, T. Gebbie, Calibrating high-frequency tradingdata to agent-based models using approximate bayesian com-putation, Master’s thesis, University of cape town, 2020.URL: https://github.com/KellyGoosen1/hft-abm-smc-abc .doi: .[4] A. Addison, C. Andrews, N. Azad, D. Bardsley, J. Bauman,J. Diaz, T. Didik, K. Fazliddin, M. Gromoa, A. Krish, R. Prins,L. Ryan, N. Villette, Low-latency trading in the cloud environ-ment, in: 2019 IEEE International Conference on Computa-tional Science and Engineering (CSE) and IEEE InternationalConference on Embedded and Ubiquitous Computing (EUC),IEEE, 2019, pp. 272–282. URL: https://ieeexplore.ieee.org/document/8919600 . doi: .[5] D. Sing, Cointossx software, 2017. URL: https://github.com/dharmeshsing/CoinTossX . doi: .[6] D. Sing, T. Gebbie, JSE Matching Engine Simulator, Master’sthesis, University of the Witwatersrand, 2017. URL: http://wiredspace.wits.ac.za/handle/10539/25136 .[7] J. S. Exchange, Broker Deal Accounting System, 2020.URL: .[8] I. Jericevich, P. Chang, T. Gebbie, Comparing the market mi-crostructure between two south african exchanges, 2020. URL: https://arxiv.org/abs/2011.04367 . arXiv:2011.04367 .[9] P. Nair, Agent based modelling of a single-stock market on theJSE, Master’s thesis, University of the Witwatersrand, 2015.URL: http://wiredspace.wits.ac.za/handle/10539/16840 .[10] M. Thompson, D. Farley, M. Barker, P. Gee, A. Stewart, Dis-ruptor: High performance alternative to bounded queues forexchanging data between concurrent threads, Technical Re-port, LMAX, 2011. URL: https://lmax-exchange.github.io/disruptor/ .[11] D. Wilcox, T. Gebbie, Hierarchical causality in financial eco-nomics, SSRN (2014). URL: https://ssrn.com/abstract=2544327 . doi: .[12] J. Lussange, A. Belianin, S. Bourgeois-Gironde, B. Gutkin,A bright future for financial agent-based models, SSRN(2018). URL: https://ssrn.com/abstract=3109904 . doi: .[13] D. Platt, A comparison of economic agent-based model cal-ibration methods, Journal of Economic Dynamics and Con-trol 113 (2020) 103859. URL: . doi: . [14] B. LeBaron, Agent-based computational finance, Handbookof Computational Economics 2 (2006) 1187–1233.[15] I. Jericevich, D. Sing, T. Gebbie, Cointossx, GitHub (2021).URL: https://zivahub.uct.ac.za/articles/software/CoinTossX/14069552 . doi: .[16] J. S. Exchange, Equity Market Trading and Information Solu-tion JSE Specification Document Volume 00E — Trading andInformation Overview, 4 ed., 2020. URL: .[17] J. S. Exchange, Equity Market Trading and Informa-tion Solution JSE Specification Document Volume 01 —Native Trading Gateway, 3.05 ed., 2018. URL: .[18] J. S. Exchange, Equity Market Trading and InformationSolution JSE Specification Document Volume 05 — MarketData Gateway, 3.09 ed., 2019. URL: .[19] R. Logic, Aeron media driver, 2021. URL: https://github.com/real-logic/Aeron/wiki .[20] A. G. Hawkes, Spectra of some self-exciting and mutu-ally exciting point processes, Biometrika 58 (1971) 83–90.URL: . doi: .[21] Y. Ogata, Statistical models for earthquake occurrencesand residual analysis for point processes, Journal of theAmerican Statistical association 83 (1988) 9–27. URL: . doi: .[22] J. Large, Measuring the resiliency of an electronic limitorder book, Journal of Financial Markets 10 (2007) 1–25. URL: . doi: .[23] P. W. Lewis, G. S. Shedler, Simulation of nonhomo-geneous poisson processes by thinning, Naval researchlogistics quarterly 26 (1979) 403–413. URL: https://onlinelibrary.wiley.com/doi/10.1002/nav.3800260304 .doi: .[24] Y. Ogata, On lewis’ simulation method for point pro-cesses, IEEE transactions on information theory 27 (1981) 23–31. URL: https://ieeexplore.ieee.org/document/1056305 .doi: .[25] J. S. Exchange, The lowest-latency connection to JSEmarkets, 2015. URL: .[26] E. Bacry, J.-F. Muzy, Hawkes model for price and trades high-frequency dynamics, Quantitative Finance 14 (2014) 1147–1166. doi: .[27] B. Zheng, F. Roueff, F. Abergel, Modelling bid and ask pricesusing constrained hawkes processes: Ergodicity and scalinglimit, SIAM Journal on Financial Mathematics 5 (2014) 99–136. doi: . PPENDIX A DEPLOYMENT AND USAGE
Appendix A. Deployment and usage
CoinTossXClientSimulatorbuildsrcmainjavaclientexamplehawkesresourcestest/java/hawkesbuild.gradleCommonLimitOrderBookMarketDataGatewayMatchingEngineMatchingEngineClientMessagesNativeGatewaySocketWebWebEventListenerdataClientData.csvhawkesData.propertiesStock.csvTrader.csvtradingSessionsCron.propertiesbuild.gradlegradle/wrappergradlewgradlew.batinstall.gradlelibscriptslocal.propertiesdeploy local.gradleremote.propertiesdeploy remote.gradlewindows.propertiessettings.gradle
Below are the instructions for deploying CoinTossX to a user’s local machine or virtual ma-chine in a cloud environment. These instructions apply to Windows, Linux and OSX operatingsystems. CoinTossX is a Java web application and is built using the Gradle build tool. Theuser need not install Gradle or change the version of Gradle installed on their system as theproject uses the Gradle wrapper to download and run the required Gradle version a . CurrentlyCoinTossX is compatible with Java version 8 and Gradle version 6.7.1. The application can bestarted from the command line, however, it is recommended that the user make use of a JavaIDE such as Eclipse or IntelliJ IDEA to automate the start-up process and simplify deployment. Prerequisites
1. JDK version 8 (see Installation section for more detail)2. A computer with 4 or more CPU cores and a sufficient amount of RAM (ideally 4 cores 32GB but can be less depending on the number of clients and stocks used).3. A Java IDE such as Eclipse or IntelliJ IDEA.4. The user may be required to run commands using Command Prompt (Windows) or Bash(Linux and OSX).
Additional resources • Tutorial for building a java project with gradle: fi . • Tutorial for building a simple Java web application: fi • Introduction to Spring Boot fi • Introduction to Apache Wicket fi • Tutorial for setting up a student Azure account: fi • Instructions for the installation of Oracle JDK 8 and configuration of the system environ-ment variables on Linux: fi • Calling Java in Julia fi , Python fi and R fi Installation
If the correct version of Java is already installed and configured correctly, the user can skip step1 below.1. CoinTossX is currently only compatible with version 8 of Java. Therefore, the user shouldinstall version 8 of either the Oracle ( fi ) or Open JDK (Java Development Kit) b . Forsimplicity, it is assumed that Windows and OSX users will install the Oracle JDK whileLinux users will install the Open JDK. • Windows — After installing Oracle JDK 8 using the link above, if not done so au-tomatically by the java install wizard, ensure that
JAVA_HOME is set and that the javaexecutable is set in the path environment variable. To set/add java to the system’s
JAVA_HOME and path environment variables, go to
Settings > Advanced System Settings (cid:44) → > Environment Variables > System Variables . Then add the location of the java instal-lation to a new variable called JAVA_HOME which points to the relevant java distribution(e.g.
C:\ Program Files\Java\jdk1 .8.0 _271 ). Thereafter append a new pointer in the path environment variable by adding \% JAVA_HOME \%\ bin . • Linux — Installation of Open JDK 8 is done by executing the commands below: sudo apt -get updatesudo apt -get install openjdk -8-jdk
After this, if the correct version of Java is still not being used, the user can switch tothe correct version using sudo update - alternatives --set java (cid:44) → /usr/lib/jvm/jdk1 .8.0_[ version ]/ bin/java • OSX — To set
JAVA_HOME run export (cid:44) → JAVA_HOME =/ Library /Java/ JavaVirtualMachines /jdk1 .8.0_[ version ]. jdk/ Contents /Home • Clone the CoinTossX repository (using either git in the terminal or Github Desktop).The user can clone the repository to any location of their choosing. cd [ directory to clone into]git clone https :// github .com/ dharmeshsing / CoinTossX .git a The Wrapper is a script that invokes a declared version of Gradle, downloading it beforehand if necessary.This standardizes a project on a given Gradle version and simplifies execution (especially when using an IDE) b Note that the user should install the JDK (which includes the JRE), not only the JRE. PPENDIX A DEPLOYMENT AND USAGE
The next set of instructions provides guidelines for the deployment of CoinTossX. By this point the user will have clonedthe repository to a location of their choosing for either local or remote deployment. Depending on the operating system,different deployment configurations would need to be employed - hence the multiple configuration and deploymentfiles. The files with the ”.properties” extensions define the required ports for each component as well as user specificpath definitions. To match the location to which the repository was cloned, the user would have to configure the local. properties file (for Linux) or the windows . properties file (for Windows) to correspond with the user’s directories.For remote deployment the user would have to configure the remote . properties with the corresponding paths on theremote server. The path variables which need to be configured are: • MEDIA_DRIVER_DIR pointing to the Aeron Media Driver. This folder will only be created after the application isstarted. For Linux users it is recommended that the default path ( /dev/shm ) is not deviated from in order toachieve optimal performance. Otherwise the path can be amended as follows:
MEDIA_DRIVER_DIR =[...]/ aeron
Note that if the user deviates from the default media driver path, they would have to make the same change inthe build. gradle file. • SOFTWARE_PATH pointing to the start-up folder that will be created upon deployment. This path can be set to anyvalid path on the user’s machine and may be given any name. For example:
SOFTWARE_PATH =[...]/[ folder name] • DATA_PATH points to the data folder within the software path. For example:
DATA_PATH =[ SOFTWARE_PATH ]/ data
Before the application can be started, we are required to change a few system settings to ensure that network per-formance and system memory are utilised correctly. Firstly, the receive and send UDP buffer sizes/limits need to beconfigured as follows. sudo sysctl net.core. rmem_max =2097152sudo sysctl net.core. wmem_max =2097152
Secondly, this Java application requires large pages to be enabled. By default Linux does not enable any HugePages(portions of memory set aside to be solely used by the application). To determine the current HugePage usage, run grep Huge /proc/ meminfo . Furthermore, the default HugePage size is 2MB (2048kB). So if the user wishes to enableapproximately 20GB of HugePages (dedicated memory), this would require 10000 HugePages - so the command torun would be sudo sysctl vm. nr_hugepages =10000
Last is the running of the application. Users deploying the application to Microsoft Azure, CHPC, Wits Server orany other server may choose to do so locally or remotely. Remote deployment will require that the user specify theabove paths to correspond with that of the remote server. The instructions below demonstrate both local and remotedeployment. For users deploying remotely, one must first ensure that SSH is enabled on the server and that port 22 isopen for the transfer of files. Additionally, the username, IP address and password fields in the deploy_remote . gradle file should match that of the server.1. Set the current directory to the location of CoinTossX: cd [path to]/ CoinTossX
2. To build the project, use the provided gradle wrapper to run: ./ gradlew -Penv=local build -x test
3. To deploy the project to a start-up folder, execute one of the following. ./ gradlew -Penv=local clean installDist bootWar copyResourcesToInstallDir copyToDeploy deployLocal (cid:44) → local deployment./ gradlew -Penv= remote clean installDist bootWar copyResourcesToInstallDir copyToDeploy deployRemote (cid:44) → For remote deployment./ gradlew .bat -Penv= windows clean installDist bootWar copyResourcesToInstallDir copyToDeploy deployLocal (cid:44) →
4. Finally, to run the program, execute the shell script: 16
PPENDIX A DEPLOYMENT AND USAGE sh [ SOFTWARE_PATH ]/ scripts / startAll .sh
5. Access the web app by typing localhost :8080 (if deployed locally) or [ server IP ]:8080 (if deployed remotely) intothe URL search bar of a browser.
Usage
To configure the trading sessions to be fired during the simulationrefer to tradingSessionsCron . properties file found in the data direc-tory. The usage/syntax of the cron expressions within that file areas follows. A cron expression is a string consisting of six or sevenfields, separated by white space, that describe individual details ofthe schedule: [ seconds ] [ minutes ] [hours] [day of month] [month] [day of week] (cid:44) → [year] A few examples, provided in tradingSessionsCron . properties , areshown on the right.
TRADING_SESSIONS = ContinuousTrading2
Each component of the system as well as other actions can be started independently via the shell scripts. The listof all the runnable shell scripts can be found in the deploy / scripts directory. Each shell script simply executes theJava byte code of a “main” method whose class is specified in the build. gradle file of the corresponding module. Forexample, the startAll .sh script starts each component consecutively (equivalent to clicking the “Start” button on theHawkes simulation web page). Similarly, stopAll .sh stops all the components (equivalent to clicking the “Shut Down”button on the Hawkes simulation webpage). Furthermore, any other actions on the website can also be done from thecommand line. To start the Hawkes simulation for a single client and stock simply run ./ startHawkesSimulation .sh 1 1 .This starts the simulation for client 1 and stock 1 by submitting the client ID and stock ID as the first and secondargument, respectively.The list of clients that can be activated can be found in the data/ ClientData .csv file. Consider, for example, the firstclient shown below.
CompID ,Password ,NGInputURL , NGInputStreamId ,NGOutputURL , NGOutputStreamId ,MDGInputURL , MDGInputStreamId , MDGOutputURL , (cid:44) → MDGOutputStreamId , SecurityId1,test111111 ,udp :// localhost :5000 ,10 , udp :// localhost :5001 ,10 , udp :// localhost :5002 ,10 , udp :// localhost :5003 ,10 ,1
Each client is assigned input and output URLs for both the Native Gateway and Market Data Gateway. These URLsspecify the IP addresses (in this case localhost - the user’s local machine) to/from which messages will be sent/receivedas well as the ports on which these components are listening.From the website the user can: view/add/edit/delete clients, view the simulation status of clients and stocks, start theHawkes simulation, edit the Hawkes simulation parameters, view/extract Hawkes simulation data as well as view asnapshot of the limit order books of each stock. With regards to the output of results, after all clients have submittedan end of trading session message, the orders along with their submission times are written to file and stored in the deploy /data directory. At the end of each Hawkes simulation the HdrHistogram latency results are written to a textfile in the same directory. 17
PPENDIX B SUBMITTING ORDERS (a) Splash screen (b) All stocks and clients page snapshot (c) Simulation page snapshot(d) Hawkes configuration page snap-shot (e) Client page snapshotFigure A.10: CoinTossX website
Appendix B. Submitting orders
The subsections below demonstrate the submission of orders to the trading gateway (on a user’s local machine)in three programming languages: Java, Julia and Python. Note that currently CoinTossX only allows for Javaimplementations. Nonetheless, most programming languages include packages/libraries which provide interfaces forcalling Java. Although it may be considered sub-optimal, this solution allows us to bypass some problems with aJava-only implementation without the need for a software shim. Even though the message interface is port based, andhence it should not matter what language or interface is used, to submit orders the choices made here are convenientas they still allow the separation of the matching engine for the components that may generate orders.The below programming languages are only a few of those that provide these types of libraries. The code snippets belowcan be found in the CoinTossX project under the directory
ClientSimulator /src/main/java/ example . More specifically,all necessary static Java methods are defined in the
Utilities class. Clients have the following functionality that willbe demonstrated in the code snippets:1. Submit the order combinations shown in table 2.2. Cancel limit orders.3. Market data updates(a) VWAP of buy/sell side of LOB.(b) Current best bid/ask price/quantity.(c) Active trading session.(d) Whether or not the LOB has updated.
Appendix B.1. Java
As opposed to the other implementations below, the Java implementation is shown in full without reference to the
Utilities class to provide more detail for how the other implementations function. /* Example : - Java version : 1.8.0 _271 - Authors : Ivan Jericevich , Dharmesh Sing , Tim Gebbie - Structure :
1. Dependencies
2. Supplementary methods
3. Implementation - Login and start session - Submit orders - Market data updates PPENDIX B SUBMITTING ORDERS Appendix B.2 Julia - End session and logout */ // --------------------------------------------------------------------------------------------------- // ----- Import the necessary dependencies -----// package example ; import client .*; import sbe.msg .*; import java.util. Properties ; import java.io. IOException ; import java.io. InputStream ; import java.util. Properties ; import java.time. LocalDateTime ; import com. carrotsearch .hppc. IntObjectMap ; // --------------------------------------------------------------------------------------------------- public class Example { private static final String PROPERTIES_FILE = " simulation . properties "; // The name of the file specifying (cid:44) → the simulation configuration private static Properties properties = new Properties (); // ----- Supplementary method for extracting the simulation settings -----// private static void loadProperties ( Properties properties , String propertiesFile ) throws IOException { try ( InputStream inputStream = Example .class. getClassLoader (). getResourceAsStream ( propertiesFile )) { if ( inputStream != null) { properties .load( inputStream ); } else { throw new IOException (" Unable to load properties file " + propertiesFile ); } } } // --------------------------------------------------------------------------------------------------- // ----- Implementation -----// public static void main ( String [] args) throws Exception { // ----- Login and start session -----// int clientId = 1; int securityId = 1; // Define the client ID corresponding the client to be logged in as (cid:44) → well as the security ID corresponding to the security in which they will trade // Load the simulation settings as well as all client data (ports , passwords and IDs) loadProperties (properties , PROPERTIES_FILE ); String dataPath = properties .get(" DATA_PATH "). toString (); IntObjectMap
System .exit (0); } // ------------------------------------------------------------------------------------------------------------------ } Appendix B.2. Julia
The Julia-Java interface is provided in the JavaCall.jl package. This package works by specifying the paths to thecompiled Java byte code (.jar files) as well as the path to the required dependencies. Thereafter the JVM is started19
PPENDIX B SUBMITTING ORDERS Appendix B.3 Python and the utilities for logging a client in, starting the trading session, submitting an order and logging out are imported.With the execution of each command, the Java stacktrace is printed in the REPL as well. Example : - Julia version : 1.5.3 - Java version : 1.8.0 _271 - Authors : Ivan Jericevich , Dharmesh Sing , Tim Gebbie - Structure :
1. Preliminaries
2. Login and start session
3. Submit orders
4. Market data updates
5. End session and logout = using JavaCall cd( @__DIR__ ); clearconsole () JavaCall . addClassPath ("/home/ ivanjericevich / CoinTossX / ClientSimulator /build/ classes /main") (cid:44) → JavaCall . getClassPath () JavaCall . addClassPath ("/home/ ivanjericevich / CoinTossX / ClientSimulator /build/ install / ClientSimulator /lib /*. jar") JavaCall .init () utilities = @jimport example . Utilities (cid:44) → security in which they will trade clientId = 1; securityId = 1 client = jcall(utilities , " loadClientData ", JavaObject { Symbol (" client . Client ")}, (jint , jint), clientId , (cid:44) → securityId ) jcall(client , " sendStartMessage ", Nothing , ()) (cid:44) → size , stop price jcall(client , " submitOrder ", Nothing , (jlong , jlong , JString , JString , JString , jlong , jlong , jlong), 1000 , 99, (cid:44) → "Buy", "Limit", "Day", 1000 , 0, 0) jcall(client , " submitOrder ", Nothing , (jlong , jlong , JString , JString , JString , jlong , jlong , jlong), 1000 , 101, (cid:44) → "Sell", "Limit", "Day", 1000 , 0, 0) jcall(client , " submitOrder ", Nothing , (jlong , jlong , JString , JString , JString , jlong , jlong , jlong), 1000 , 0, (cid:44) → "Buy", " Market ", "Day", 1000 , 0, 0) jcall(client , " submitOrder ", Nothing , (jlong , jlong , JString , JString , JString , jlong , jlong , jlong), 1000 , 0, (cid:44) → "Buy", " StopLimit ", "Day", 1000 , 0, 0) jcall(client , " submitOrder ", Nothing , (jlong , jlong , JString , JString , JString , jlong , jlong , jlong), 1000 , 0, (cid:44) → "Buy", "Stop", "Day", 1000 , 0, 0) jcall(client , " cancelOrder ", Nothing , (jlong , jlong), "1", "Buy") jcall(client , " calcVWAP ", jlong , (JString ,), "Buy") jcall(client , " getBid ", jlong , ()) jcall(client , " getBidQuantity ", jlong , ()) jcall(client , " getOffer ", jlong , ()) jcall(client , " getOfferQuantity ", jlong , ()) jcall(client , " waitForMarketDataUpdate ", Nothing , ()) jcall(client , " isAuction ", jboolean , ()) jcall(client , " sendEndMessage ", Nothing , ()); jcall(client , "close", Nothing , ()) Appendix B.3. Python
The Python Java interface is provided in the JPype module. This code snippet follows a similar process as above butwith different syntax. ’’’ Example : PPENDIX B SUBMITTING ORDERS Appendix B.3 Python - Python version : 3.8.5 - Java version : 1.8.0 _271 - Authors : Ivan Jericevich , Dharmesh Sing , Tim Gebbie - Structure :
1. Preliminaries
2. Login and start session
3. Submit orders
4. Market data updates