Yousuke Watanabe

Integrating a stream processing engine and databases for persistent streaming data management

Because of increased stream data, managing stream data has become quite important. This paper describes our data stream management system, which employs an architecture combining a stream processing engine and DBMS. Based on the architecture, the system processes both continuous queries and traditional one-shot queries. Our proposed query language supports not only filtering, join, and projection over data streams, but also continuous persistence requirements for stream data. Users can also specify continuous queries that integrate streaming data and historical data stored in DBMS. Another contribution of this paper is feasibility validation of queries. Processing queries on streams with frequent inputs may cause the system to overflow its capacity. Specifically, the maximum writing rate to DBMS is a significant bottleneck when we try to store stream data into DBMS. Our system detects infeasible queries in advance.

A Multiple Continuous Query Optimization Method Based on Query Execution Pattern Analysis

Many data streams are provided through the network today, and continuous queries are often used to extract useful information from data streams. When a system must process many queries continuously, query optimization is quite important for their efficient execution. In this paper, we propose a novel multiple query optimization method for continuous queries based on query execution pattern analysis. In the advanced stream processing environment assumed in the paper, we use window operators to specify time intervals to select information of interest and the execution time specification to designate when the query should be evaluated. Queries having the same operators may share many intermediate results when they are executed at close instants, but may involve only disjoint data when executed at completely different instants. Thus, query execution timing as well as common subexpressions is a key to deciding an efficient query execution plan. The basic idea of the proposed method is to identify query execution patterns from data arrival logs of data streams and to make the most of the information in deciding an efficient query execution plan. The proposed query optimization scheme first analyzes data arrival logs and extracts query execution patterns. It then forms clusters of continuous queries such that queries in the same cluster are likely to be executed at close instants. Finally, it extracts common subexpressions from among queries in each cluster and decides the query execution plan. We also show experiment results using the prototype implementation, and discuss effectiveness of the proposed approach.

A file search method based on intertask relationships derived from access frequency and RMC operations on files

The tremendous growth in the number of files stored in filesystems makes it increasingly difficult to find desired files. Traditional keyword-based search engines are incapable of retrieving files that do not include keywords. To tackle this problem, we use file-access logs to derive intertask relationships for file search. Our observations are that 1) files related to the same task are frequently used together, and 2) a set of Rename, Move, and Copy (RMC) operations tends to initiate a new task. We have implemented a system named SUGOI, which detects two types of task, FI tasks and RMC tasks, from file-access logs. An FI task corresponds to a group of files frequently accessed together. An RMC task is generated by RMC operations and then constructs a graph of intertask relationships based on the influence of RMC operations and the similarity between tasks. In utilizing detected tasks and intertask relationships, our system expands the search results of a keyword-based search engine. Experiments using actual file-access logs indicate that the proposed approach significantly improves search results.

AEDSMS: Automotive Embedded Data Stream Management System

Data stream management systems (DSMSs) are useful for the management and processing of continuous data at a high input rate with low latency. In the automotive domain, embedded systems use a variety of sensor data and communications from outside the vehicle to promote autonomous and safe driving. Thus, the software developed for these systems must be capable of handling large volumes of data and complex processing. At present, we are developing a platform for the integration and management of data in an automotive embedded system using a DSMS. However, compared with conventional DSMS fields, we have encountered new challenges such as precompiling queries when designing automotive systems (which demands time predictability), distributed stream processing in in-vehicle networks, and real-time scheduling and sensor data fusion by stream processing. Therefore, we developed an automotive embedded DSMS (AEDSMS) to address these challenges. The main contributions of the present study are: (1) a clear understanding of the challenges faced when introducing DSMSs into the automotive field; (2) the development of AEDSMS to tackle these challenges; and (3) an evaluation of AEDSMS during runtime using a driving assistance application.

Real-Time-Aware Embedded DSMS Applicable to Advanced Driver Assistance Systems

Automotive embedded systems are composed of many kinds of sensors, and it is becoming difficult to manage these data. Therefore, we define a vehicle data processing by a Stream Processing Description (SPD), which is the data processing description used in the Data Stream Management Systems (DSMS). However, because existing DSMSs run on generalpurpose systems, it is difficult to deploy DSMSs in automotive embedded systems and fulfill these performance requirements. The DSMS for automotive embedded systems (eDSMS) generates an optimized runtime from the queries defined by an SPD not dynamically (in execution) but statically (in development), which reduces the ROM/RAM usage. In this study, we extend eDSMS for real-time processing and show the extended eDSMS is applicable to the Advanced Driver Assistance Systems (ADAS).

Similarity search for office XML documents based on style and structure data

Purpose – Office documents are widely used in our daily activities, so the number of them has been increasing. A demand for sophisticated search for office documents becomes more important. The recent file format of office documents is based on a package of multiple XML files. These XML files include not only body text but also page structure data and style data. The purpose of this paper is to utilize them to find similar office documents.Design/methodology/approach – The authors propose SOS, a similarity search method based on structures and styles of office documents. SOS needs to compute similarity values between multiple pairs of XML files included in the office documents. We also propose LAX+, which is an algorithm to calculate a similarity value for a pair of XML files, by extending existing XML leaf node clustering algorithm.Findings – SOS and LAX+ are evaluated by using three types of office documents (docx, xlsx and pptx) in our experiments. The results of LAX+ and SOS are better than ones of th...

Query result caching for multiple event-driven continuous queries

With the increasing demands for advanced use of streaming data, efficient execution of continuous queries is an important research issue. This paper focuses on event-driven continuous queries that are activated by foreign events such as data arrival and the progression of time. Existing approaches to multiple continuous query optimization decide the optimal query plan by extracting common subexpressions from the given queries. Event-driven queries containing the common subexpressions may produce many common intermediate results when they are activated within a small interval, but may produce only disjoint data when activated at completely different timings. This paper proposes an efficient data stream processing scheme for multiple event-driven continuous queries. In the proposed approach, we introduce query result caching to achieve a flexible way to share common operators among queries activated by unpredictable events. When a query is activated, an intermediate result generated for the query is stored into the cache area if it is expected to be reused by other queries. When other queries including the same operator are activated, they reuse the cached result if the cache includes reusable data. Efficiency of the proposed scheme is validated by intensive experimental evaluations.

A video stream management system for heterogeneous information integration environments

Massive data streams are obtained from various types of sources such as sensors, GPS and live cameras. Since demand for applications integrating multiple data streams and databases is increasing, we must consider an integration framework for heterogeneous information sources. Integrating numeric and text streams is comparatively easy, but integrating video streams and other sources is difficult because of the properties of video streams: highly-frequent, largevolume and complex binary data. Based on this background, we propose a video stream management system. The system provides a SQL-like query interface for heterogeneous information sources including video streams. To integrate video streams, we employ an abstract data type to hold a subsequence of video frames and functions to extract metadata from video data. Beyond that, we also propose a dynamic source selection scheme for some applications, like moving object tracking with video streams. The scheme is used when information sources to be accessed may change according to changes in user interest. Our system dynamically accesses necessary information sources and saves system resources by closing connections to unnecessary information sources.

Adaptive Query Optimization Method for Multiple Continuous Queries

Continuous query is widely recognized as a scheme for processing queries over data streams, and efficient methods for processing multiple continuous queries are needed. Our research group has proposed a multiple query optimization method for continuous queries. In our method, the system forms clusters of queries with similar execution patterns, and derives query plans sharing the result of common operators. Our previous experiments have shown that a parameter value in the clustering phase controls divisions of clusters and has a great impact on query processing effi- ciency. However, the optimal parameter value must be decided by trial and error. This paper extends our previous work. The proposed method automatically estimates the optimal value and iteratively adjusts it even if properties of underlying data streams dramatically change.

Relationship extraction methods based on co-occurrence in web pages and files

Every day, information on the Web becomes increasingly enriched. Web access is now very useful in many aspects of daily life, particularly for writing documents and programs. In fact, it has become quite usual to edit files while referring to information on the Web. During the file-editing process, we usually visit so many Web pages that we cannot remember all of the relevant ones. Later, if we want to revisit the same Web pages to modify some part of a file, it can be very hard to track down the Web pages originally referred to. In this paper, we propose methods for finding relationships between files and Web pages based on the co-occurrence of data in Web-access logs and file-access logs. These relationships are very useful for revisiting Web pages related to target files. To analyze co-occurrence in these two types of access logs, there are two approaches for merging the logs, involving a trade-off between accuracy and execution time. We call them the Pre-Merge and Post-Merge methods, and we have evaluated these two methods using actual access logs.