Kaiwen Zhang

Total Order in Content-Based Publish/Subscribe Systems

Total ordering is a messaging guarantee increasingly required of content-based pub/sub systems, which are traditionally focused on performance. The main challenge is the uniform ordering of streams of publications from multiple publishers within an overlay broker network to be delivered to multiple subscribers. Our solution integrates total ordering into the pub/sub logic instead of offloading it as an external service. We show that our solution is fully distributed and relies only on local broker knowledge and overlay links. We can identify and isolate specific publications and subscribers where synchronization is required: the overhead is therefore contained to the affected subscribers. Our solution remains safe under the presence of failure, where we show total order to be impossible to maintain. Our experiments demonstrate that our solution scales with the number of subscriptions and has limited overhead for the non-conflicting cases. A holistic comparison with group communication systems is offered to evaluate their relative scalability.

Persistence in massively multiplayer online games

The most important asset of a Massively Multiplayer Online Game is its world state, as it represents the combined efforts and progress of all its participants. Thus, it is extremely important that this state is not lost in case of server failures. Survival of the world state is typically achieved by making it persistent, e.g., by storing it in a relational database. The main challenge of this approach is to track the large volume of modifications applied to the world in real time. This paper compares a variety of strategies to persist changes of the game world. While critical events must be written synchronously to the persistent storage, a set of approximation strategies are discussed and compared that are suitable for events with low consistency requirements, such as player movements. An analysis to better understand the possible limitations and bottlenecks of these strategies is presented using experimental data from an MMOG research framework. Our analysis shows that a distance-based solution offers the scalability and efficiency required for large-scale games as well as offering error bounds and eliminating unnecessary updates associated with localized movement.

Publish/subscribe network designs for multiplayer games

Massively multiplayer online games (MMOGs), which are typically supported by large distributed systems, require a scalable, low latency messaging middleware that supports the location-based semantics and the loosely coupled interaction of multiplayer games components. In this paper, we present three different pub/sub-driven designs for a MMOG networking engine that account for the highly interactive and massive nature of these games. Each design uses not only different pub/sub approaches (from topic-based to content-based) but also serves varying degrees of responsibilities. In particular, some of them integrate game functionality, such as interest management, into the network engine. We implement, evaluate, and compare our proposed designs in the MMOG prototype Mammoth. Our real-world results show the viability of pub/sub while at the same time highlighting clear trade-offs between the different designs used, especially in the number and frequency of the various message types, such as subscriptions.

Distributed Ranked Data Dissemination in Social Networks

The amount of content served on social networks can overwhelm users, who must sift through the data for relevant information. To facilitate users, we develop and implement dissemination of ranked data in social networks. Although top-k computation can be performed centrally at the user, the size of the event stream can constitute a significant bottleneck. Our approach distributes the top-k computation on an overlay network to reduce the number of events flowing through. Experiments performed using real Twitter and Facebook datasets with 5K and 30K query subscriptions demonstrate that social workloads exhibit properties that are advantageous for our solution.

Distributed event aggregation for content-based publish/subscribe systems

Modern data-intensive applications handling massive event streams such as real-time traffic monitoring require support for both rich data filtering and aggregation. While the pub/sub communication paradigm provides an effective solution for the sought semantic diversity of event filtering, the event processing capabilities of existing pub/sub systems are restricted to singular event matching without support for stream aggregation, which so far can be accommodated only at the subscriber edge brokers. In this paper, we propose the first systematic solution for supporting distributed aggregation over a range of time-based aggregation window semantics in a content-based pub/sub system. In order to eschew the need to disseminate a large number of publications to subscribers, we strive to distribute the aggregation computation within the pub/sub overlay network. By enriching the pub/sub language with aggregation semantics, we allow pub/sub brokers to aggregate incoming publications and forward only results to the next broker downstream. We show that our baseline solutions, one which aggregates early (at the publisher edge) and another which aggregates late (at the subscriber edge), are not optimal strategies for minimizing bandwidth consumption. We then propose an adaptive rate-based heuristic solution which determines which brokers should aggregate publications. Using real datasets extracted from our traffic monitoring use case, we show that this adaptive solution leads to improved performance compared to that of our baseline solutions.

Transaction Models for Massively Multiplayer Online Games

Massively Multiplayer Online Games are considered large distributed systems where the game state is partially replicated across the server and thousands of clients. Given the scale, game engines typically offer only relaxed consistency without well-defined guarantees. In this paper, we leverage the concept of transactions to define consistency models that are suitable for gaming environments. We define game specific levels of consistency that differ in the degree of isolation and atomicity they provide, and demonstrate the costs associated with their execution. Each action type within a game can then be assigned the appropriate consistency level, choosing the right trade-off between consistency and performance. The issue of durability and fault-tolerance of game actions is also discussed.

Solving manufacturing equipment monitoring through efficient complex event processing: DEBS grand challenge

In this paper, we present an efficient complex event processing system tailored toward monitoring a large-scale setup of manufacturing equipment. In particular, the key challenge in the equipment monitoring is to develop an event-based system for computing complex manufacturing queries coupled with event notifications and event and query result visualization components. Furthermore, we present an experimental evaluation to validate the effectiveness of the proposed solution with respect to both query latency and throughput.

Publish/Subscribe for Mobile Applications Using Shared Dictionary Compression

Publish/Subscribe is known as a scalable and efficient data dissemination mechanism. In a mobile environment, there is an added challenge for the pub/sub system to economizemobile bandwidth, which is especially precious in areas not wellcovered by mobile providers. While well-known compressionmethods such as GZip or Deflate are generally useful in suchsituations, we propose using Shared Dictionary Compression(SDC) to achieve a greater level of bandwidth efficiency. SDCrequires a dictionary, generated upfront, to be shared betweentwo communicating peers before it can be used. We proposea design where brokers forming the pub/sub overlay can be incharge of generating and propagating the shared dictionary. Oursolution employs an adaptive algorithm, executed at the brokers, which creates and maintains the dictionaries over time. Withthis approach, it is possible to reduce the required bandwidth byup to 88% including the introduced dictionary overhead. Ourdemo shows this approach applied to a smartphone applicationcommunicating with a publish/subscribe broker using the MQTTprotocol.

Smart phone application for connected vehicles and smart transportation

Real-time traffic analysis is an interesting research topic across different disciplines and areas. In order to provide a better and practical way of representing and gathering traffic data, we need to have a suitable media with high mobility and qualitative performance such as smartphones. This paper proposes a smartphone application which presents and collects traffic information data to improve the travelling experience. The application is based on a server-client model, developed in Java and deployed on the Android platform.

Hyperpubsub: a decentralized, permissioned, publish/subscribe service using blockchains: demo

Since the introduction of Bitcoin in 2008, blockchain systems have evolved immensely in terms of performance and usability. There is a massive focus on building enterprise blockchain solutions, with providers such as IBM and Microsoft already providing Blockchain-as-a Service (BaaS). To facilitate the adoption of blockchain technologies across various business verticals, we argue that middleware plays an integral role in accelerating the development of automated business processes (i.e., smart contracts). We argue that decentralized messaging is a key requirement of many distributed applications and should be provided as a reusable blockchain middleware. Our system, called HyperPubSub, provides decentralized publish/subscribe messaging for a multi-federated, permissioned, environment. HyperPubSub provides secure and privacy-preserving messaging, which is audited using blockchains for validation and monetization purposes. We demonstrate our implementation using Kafka and Hyperledger.