Wang Shan

A Parallel Recovery Scheme for Update Intensive Main Memory Database Systems

In update intensive applications, main memory database systems produce large volume of log records, it is critical to write out the log records efficiently to speedup transaction processing. We propose a parallel recovery scheme based on XOR differential logging for main memory database systems in such environments. Some NVRAM is used to temporarily hold log records and decouple transaction committing from disk writes, inherited parallelism properties of differential logging are exploited to accelerate log flushing by using multiple log disks. During recovery, log records are loaded from multiple log disks and applied to data partition in time without the need of reordering according to serialization order, total recovery time is cut down. The scheme employs a data partition based consistent checkpointing method. The log records are classified according to IDs of data partitions accessed. Data partitions are recovered according to loading priorities computed from update frequencies and transaction waiting times, data access demands of new transactions coming after failure recovery are given attention immediately, thus the scheme provides system availability during recovery, which is of importance for large scale main memory database systems.

Parallel Techniques for Large Data Analysis in a Futures Trading Evaluation Service System

Futures trading evaluation system is used to analyze trading history of individuals, to find out the root cause of profit and loss, so that investors can learn from their past and make better decisions in the future. To analyze trading history of investors, the system processes a large volume of transaction data, to calculate key performance indicators, as well as time series behavior patterns, finally concludes recommendations with the help of an expert knowledge base. The paper firstly presents the working logic of the evaluation system, then it focuses on parallel data processing techniques that the system is based on. Parallel processing architecture, data distribution scheme, key performance indicators calculating algorithms and distributed time series analysis algorithms are elaborated in details. The system is highly scalable, and by exploiting the power of parallel processing, the generation time of an evaluation report is cut down from 1 to 3 minute, to 30 to 45 seconds.