Weilong Ding

Feature-based high-availability mechanism for quantile tasks in real-time data stream processing

Under distributed Cloud environment, the real‐time and continuous data stream makes the availability during processing essential but expensive. For aggregation tasks of data stream processing systems, traditional replica‐based high‐availability mechanisms require large overheads at run‐time and long recovery latency at fail‐time, because of specific nature of aggregations. In this paper, we focus on the typical quantile tasks and propose a feature‐based high‐availability mechanism to reduce related overhead and the latency. With the help of monitor module, quantile feature is maintained incrementally through histogram synopsis over time‐based sliding window, and the failed quantile tasks can be recovered precisely with high probability in an efficient way. The effectiveness has been analyzed theoretically, and meanwhile, the acceptable tradeoff between overheads and performance has been demonstrated by comprehensive experiments on both synthetic and real data. Copyright

A Consistency-Preserving Mechanism for Web Services Response Caching

Web services are rapidly emerging as a popular standard technology for sharing data and functionality among heterogeneous systems. Service providers and consumers are loosely coupled and distributed across the network, either within an organization or across organizational boundaries, and therefore, performance becomes a major concern in such a distributed environment. Furthermore, XML is widely used as message format for service providers and consumers in Web services environment. XML message packaging and parsing brings extra overhead to both ends. Web services response latency, as well as throughput, is becoming a bottleneck problem. In this paper, We propose a consistency-preserving mechanism for Web services response caching, which reduces the volume of data transmitted without semantic interpretation of service requests or responses, and accelerates the services response finally. It achieves this reduction through the use of cryptographic hashing to detect similarities with previous results. Experiments with an initial prototype called SigsitAcclerator indicate that our mechanism can lead to significant performance improvement over more straightforward techniques.

A Hybrid Processing System for Large-Scale Traffic Sensor Data

In recent years, with the further adoption of the Internet of Things and sensor technology, all kinds of intelligent transportation system (ITS) applications based on a wide range of traffic sensor data have had rapid development. Traffic sensor data gathered by large amounts of sensors show some new features, such as massiveness, continuity, streaming, and spatio-temporality. ITS applications utilizing traffic sensor data can be divided into three main types: 1) offline processing of historical data; 2) online processing of streaming data; and 3) hybrid processing of both. Current research tends to solve these problems in separate solutions, such as stream computing and batch processing. In this paper, we propose a hybrid processing approach and present corresponding system implementation for both streaming and historical traffic sensor data, which combines spatio-temporal data partitioning, pipelined parallel processing, and stream computing techniques to support hybrid processing of traffic sensor data in real-time. Three types of real-world applications are explained in detail to show the usability and generality of our approach and system. Our experiments show that the system can achieve better performance than a popular open-source streaming system called Storm.

An Integrated Processing Platform for Traffic Sensor Data and Its Applications in Intelligent Transportation Systems

With the continuous expansion of the scope of traffic sensor networks, traffic sensor data becomes widely available and large in amount. Traffic sensor data gathered by large amounts of sensors shows the massive, continuous, streaming and spatio-temporal characteristics compared to traditional traffic data. In order to satisfy the requirements of different applications in Intelligent Transportation Systems (ITS), we need to have the capability of real-time processing over both streaming and historical traffic sensor data. In this paper, we present DeCloud4SD, an integrated processing platform for traffic sensor data, which is designed to provide services for receiving, storing, acquiring and computing traffic sensor data in a scalable architecture with real-time guarantee. Three types of applications using DeCloud4SD in a real ITS project are also described in detail. Through the analysis of these applications, we can see that DeCloud4SD can ensure: 1) scalable and customizable traffic sensor data gathering and computing, 2) rapid application development and deployment using a MapReduce-like model, 3) seamless integration with existing relational data sources and applications.

Feature-Based High Availability Mechanism for Extreme Aggregation Tasks in Real-Time Data Stream Processing

Data stream under Cloud or IoT (Internet of Things) implies real-time and continuous processing, during which availability guarantee is essential but expensive. For extreme aggregation tasks, traditional HA mechanisms require vast space at run-time and much longer recovery latency at fail-time especially under worse input. In this paper, feature-based high availability mechanism is proposed for extreme aggregation tasks, in which space-bounded feature is maintained through random sampling over time-based sliding window and failed tasks can be recovered precisely with high probability in an efficient way. The probabilistic effectiveness has been proved theoretically, and meanwhile the acceptable tradeoff between related overheads and performance has been demonstrated by comprehensive experiments on both synthetic and real data.

A Pattern-Oriented Impact Analysis Approach for Mashups

Mashups have gained popularity among the end users for its convenient building and sharing fashions. Inevitably, a mashup is vulnerable to break down caused by its integrated components’ change, such as upgrade and inaccessibility. This paper focuses on the change impact analysis for mashups. We distill the data dependencies of mashups from pattern-oriented perspective, and abstract the mashup creation to a dependency graph. With the graph and the influence probability we defined, the impact of the components on the mashup can be calculated quantitatively. The approach is implemented in our mashup environment –Mashroom. The case study is also discussed.

A Domain-Specific Query Language for Information Services Mash-up

With the prevalence of Web Services technology, more and more information in the web are provided through Web Services. In certain domains, information can be queried by web services, but building a complete and exact query from these services is always time-consuming and less convenient for end users, especially domain users. Meanwhile, mash-up brings a new way for end-users to build personal view of data and many mash-up tools like Yahoo Pipes, Popfly etc are provided to construct mash-up application. Many of these tools rely on a graphical user interface for ease of use. However, for domain users, they can not help to get the desired outcome quickly. Domain users are familiar with the domain knowledge, and yearn for a dialect using this available domain knowledge to express their query requirements precisely and concisely.In this paper, a domain-specific query language (DSQL) for services mash-up is proposed. We first abstract the domain knowledge model as components of the DSQL, and propose the definition of DSQL to express advanced query requirements. Meanwhile, correlated services could be recommended via business process in domain after the execution of the DSQL. We build a portal site as a case study, featuring the central idea of domain-specific query language (DSQL), according to the application of National Scientific Information System (NSIS) in scientific information domain. The portal site is an interactive platform to receive and respond to userspsila requests by the DSQL.

ViPen: A Model Supporting Knowledge Provenance for Exploratory Service Composition

In current service composition efforts, the intermediate products such as data and related process fragments are neglected in many cases when the deviation occurs from predefined composition. The lack of adequate provenance support makes it not convenient when building the composition in an exploratory fashion. In this paper, we present ViPen model, and illustrate its operations to enhance the provenance for flexible deviation in service composition. Meanwhile, a partially ordered relation “derivation” is formed which guarantees the new derived processes can reuse previous knowledge effectively. A case study of one bioinformatics experiment is explained to verify our work.

A collaborative calculation on real-time stream in smart cities

Abstract In the smart cities, the travel-time is a typical business calculation to monitor and control the traffic congestions. But it still faces challenges on real-time stream due to the limitation of latency and accuracy. In this paper, we propose a collaborative approach for travel-time calculation on stream of recognized data of vehicles. Compared with other types of sensory data in urban roads, the recognized data of vehicles has wider coverage, finer interval and more exact locality. Our approach continuously achieves both factual and predictive values, and consists of two-step spatio-temporal parallelism on real-time data and Bayes prior rules mining on historical data. It can be analyzed theoretically for its low latency with high accuracy, and has been implemented on Apache Storm correlated with Hadoop MapReduce. Through exhaustive experiments on simulated and real data, our approach holds millisecond-level latencies steadily on high speed stream with nearly linear scalability, and keeps the accuracy above 80% for prediction.

A Framework to Improve the Availability of Stream Computing

In Big Data era, continuous data with low latency and high throughput makes high-availability essential for stream computing. Traditional availability guarantee is tightly-coupled and inefficient for customization and reuse. In this paper, a framework is proposed to improve the availability of stream computing, in which basic functions are provided as general services like reliable point-to-point communication and distributed status management. With its help, high-level patterns can be achieved effectively. Comprehensive experiments have been designed and evaluated to show the availability improvement with acceptable extra overheads.