Johannes Wust

Efficient logging for enterprise workloads on column-oriented in-memory databases

The introduction of a 64 bit address space in commodity operating systems and the constant drop in hardware prices made large capacities of main memory in the order of terabytes technically feasible and economically viable. Especially column-oriented in-memory databases are a promising platform to improve data management for enterprise applications. As in-memory databases hold the primary persistence in volatile memory, some form of recovery mechanism is required to prevent potential data loss in case of failures. Two desirable characteristics of any recovery mechanism are (1) that it has a minimal impact on the running system, and (2) that the system recovers quickly and without any data loss after a failure. This paper introduces an efficient logging mechanism for dictionary-compressed column structures that addresses these two characteristics by (1) reducing the overall log size by writing dictionary-compressed values and (2) allowing for parallel writing and reading of log files. We demonstrate the efficiency of our logging approach by comparing the resulting log-file size with traditional logical logging on a workload produced by a productive enterprise system.

Main memory databases for enterprise applications

Enterprise applications are traditionally divided in transactional and analytical processing. This separation was essential as growing data volume and more complex requests were no longer performing feasibly on conventional relational databases.

Dynamic task prioritization for in-memory databases

In-memory database management systems have the potential to reduce the execution time of complex operational analytical queries to the order of seconds while executing business transactions in parallel. The main reasons for this increase of performance are massive intra-query parallelism on many-core CPUs and primary data storage in main memory instead of disks or SSDs. However, database management systems in enterprise scenarios typically run a mix of different applications and users, of varying importance, concurrently. As an example, interactive applications have a much higher response-time objective compared to periodic jobs producing daily reports and should be run with priority. In addition to strict prioritization, enforcing a fair share of database resources is desirable, if several users work on applications that share a database. Solutions for resource management based on priorities have been proposed for disk-based database management systems. They typically rely on multiplexing threads on a number of processing units, which is unfavorable for in-memory databases on multi-cores, as single queries are executed in parallel and numerous context switches disrupt cache-conscious algorithms. Consequently, we propose an approach towards resource management based on a task-based query execution that avoids thread multiplexing. The basic idea is to calculate the allowed share of execution time for each user based on the priorities of all users and adjust priorities of tasks of incoming queries to converge to this share.

Efficient Transaction Processing for Hyrise in Mixed Workload Environments

Hyrise is an in-memory storage engine designed for mixed enterprise workloads that originally started as a research prototype for hybrid table layouts and basic transaction processing capabilities. This paper presents our incremental improvements and learnings to better support transactional consistency in mixed workloads.

Concurrent Execution of Mixed Enterprise Workloads on In-Memory Databases

In the world of enterprise computing, single applications are often classified either as transactional or analytical. From a data management perspective, both application classes issue a database workload with commonly agreed characteristics. However, traditional database management systems (DBMS) are typically optimized for one or the other. Today, we see two trends in enterprise applications that require bridging these two workload categories: (1) enterprise applications of both classes access a single database instance and (2) longer-running, analytical-style queries issued by transactional applications. As a reaction to this change, in-memory DBMS on multi-core CPUs have been proposed to handle the mix of transactional and analytical queries in a single database instance. However, running heterogeneous queries potentially causes situations where longer running queries block shorter running queries from execution. A task-based query execution model with priority-based scheduling allows for an effective prioritization of query classes. This paper discusses the impact of task granularity on responsiveness and throughput of an in-memory DBMS. We show that a larger task size for long running operators negatively affects the response time of short running queries. Based on this observation, we propose a solution to limit the maximum task size with the objective of controlling the mutual performance impact of query classes.

An Integrated Data Management for Enterprise Systems

Over past decades, higher demands on performance for enterprise systems have led to an increased architectural complexity. Demands as real-time analytics or graph computation add further complexity to the technology stack by adding redundancy and distributing business data over multiple components. We argue that enterprises need to simplify data management and reduce complexity as well as data redundancy. We propose a structured approach using the shearing layer concept with a unified data management to improve adaptability as well as maintainability.

Split Dictionaries for In-memory Column Stores in Mixed Workload Environments

Columnar in-memory databases use dictionary encoding as a compression technique, replacing long and frequently occurring values with short integers. Sorted dictionaries allow for more efficient query processing as comparisons can be performed directly on the compressed data whereas unsorted dictionaries are faster when inserting new values.