Itaru Nishizawa

Adaptive ordering of pipelined stream filters

We consider the problem of pipelined filters, where a continuous stream of tuples is processed by a set of commutative filters. Pipelined filters are common in stream applications and capture a large class of multiway stream joins. We focus on the problem of ordering the filters adaptively to minimize processing cost in an environment where stream and filter characteristics vary unpredictably over time. Our core algorithm, A-Greedy (for Adaptive Greedy), has strong theoretical guarantees: If stream and filter characteristics were to stabilize, A-Greedy would converge to an ordering within a small constant factor of optimal. (In experiments A-Greedy usually converges to the optimal ordering.) One very important feature of A-Greedy is that it monitors and responds to selectivities that are correlated across filters (i.e., that are nonindependent), which provides the strong quality guarantee but incurs run-time overhead. We identify a three-way tradeoff among provable convergence to good orderings, run-time overhead, and speed of adaptivity. We develop a suite of variants of A-Greedy that lie at different points on this tradeoff spectrum. We have implemented all our algorithms in the STREAM prototype Data Stream Management System and a thorough performance evaluation is presented.

Real-time log analysis using hitachi ucosminexus stream data platform

In this demo, we present real-time log analysis using Hitachi uCosminexus Stream Data Platform, uCSDP for short. Real-time log analysis is one of the key applications that offers preventive measures to detect irregular manipulations and human mistakes in system management, and reduces the risk and loss caused by such operations to the minimum in advance. uCSDP is the stream data processing system featuring its declarative query processing language, flexible time management, RAS support for high-available processing, and eager scheduling for ultra low latency processing. This demo highlights the uCSDP features for realizing real-time log analysis very easily and effectively.

SUPRA: a sampling-query optimization method for large-scale OLAP

Relational online analytical processing (ROLAP) reduces the amount of storage required for maintaining various sizes of data cubes by materializing only parts of them in a lazy evaluation manner. In ROLAP however, cube creation queries need to be issued repeatedly in order to search for useful features (i.e. rules or patterns) within large scale databases. The cube creation cost can be a bottleneck in the whole ROLAP processing. The cost of the queries can be effectively reduced by estimating the query results using samples. To maintain the accuracy of ROLAP even when using samples, the samples need to be extracted in an appropriate unit. However, conventional query optimization methods only support record based sampling and cannot be applied for complex queries that have other sampling units, such as the ones that include grouping aggregate operations. We develop a query optimization method named SUPRA that preserves the sampling unit used in random data extraction. The method is designed to preserve both the sampling unit and the randomness of the sampling operation. Using this method, typical ROLAP queries can be transformed into more efficient ones than those obtained through conventional methods.