Adam J. Storm

DB2 with BLU acceleration: so much more than just a column store

DB2 with BLU Acceleration deeply integrates innovative new techniques for defining and processing column-organized tables that speed read-mostly Business Intelligence queries by 10 to 50 times and improve compression by 3 to 10 times, compared to traditional row-organized tables, without the complexity of defining indexes or materialized views on those tables. But DB2 BLU is much more than just a column store. Exploiting frequency-based dictionary compression and main-memory query processing technology from the Blink project at IBM Research - Almaden, DB2 BLU performs most SQL operations - predicate application (even range predicates and IN-lists), joins, and grouping - on the compressed values, which can be packed bit-aligned so densely that multiple values fit in a register and can be processed simultaneously via SIMD (single-instruction, multipledata) instructions. Designed and built from the ground up to exploit modern multi-core processors, DB2 BLUs hardware-conscious algorithms are carefully engineered to maximize parallelism by using novel data structures that need little latching, and to minimize data-cache and instruction-cache misses. Though DB2 BLU is optimized for in-memory processing, database size is not limited by the size of main memory. Fine-grained synopses, late materialization, and a new probabilistic buffer pool protocol for scans minimize disk I/Os, while aggressive prefetching reduces I/O stalls. Full integration with DB2 ensures that DB2 with BLU Acceleration benefits from the full functionality and robust utilities of a mature product, while still enjoying order-of-magnitude performance gains from revolutionary technology without even having to change the SQL, and can mix column-organized and row-organized tables in the same tablespace and even within the same query.

Incorporating cost of control into the design of a load balancing controller

Load balancing is widely used in computing systems as a way to optimize performance by reducing bottleneck utilizations, such as adjusting the size of buffer pools to balance resource demands in a database management system. Load balancing is generally approached as a constrained optimization problem in which only the benefits of load balancing are considered. However, the costs of control are important as well. Herein, we study the value of including in controller design the trade-off between the cost of transient imbalances in resource utilizations and the cost of changing resource allocations. An example of the latter are actions such as resizing buffer pools that can reduce throughputs. This is because requests for data in pools whose memory is reduced immediately have longer access times whereas requests for data in pools whose memory is increased must fill this memory with data from disk before accessed times are reduced. We frame our study of control costs in terms of the widely used linear quadratic regulator (LQR). We develop a cost model that allows us to specify the LQR Q and R matrices based on the impact on system performance of changing resource allocations and transient load imbalances. Our studies of a DB2 universal database server using benchmarks for online transaction processing and decision support workloads show that incorporating our cost model into the MIMO LQR controller results in a 14% improvement in performance beyond that achieved by dynamically allocating the size of buffers without properly considering the cost of control.

Comparative studies of load balancing with control and optimization techniques

Load balancing is a widely used technique to optimizing distributed computing system performance. System response delays are reduced by equalizing the loads, such as adjusting memory pool sizes to balance disk access demands in a database management system. In this paper we formulate load balancing as a constrained optimization problem and investigate two load balancing controllers based on feedback control theory and optimization theory. We show the difference and equivalence between their design methods and criteria. Furthermore, our studies on a DB2 universal database server reveal their performance difference regarding to system noise and workload variations.

Control Theory: a Foundational Technique for Self Managing Databases

Control theory is a well established discipline that has emerged from aeronautical, electrical, and mechanical engineering to provide a formal approach to building robust systems. While similar robustness concerns exist in database management systems, control theory is rarely used due to the lack of canonical control models and a dearth of control theory expertise among database researchers. We discuss our experience with using control theory to build self managing databases, showing experimental results, discussing pitfalls and limitations, and contrasting formal models against with feedback loops. While our experience indicates that control theory is a good paradigm for database self management, control theory should be used Judiciously since its techniques are not suited to all problems in database administration.

Autonomic features of the IBM DB2 universal database for linux, UNIX, and windows

Thevast majority of the worlds structured data are now stored and managed by relational database management systems (RDBMSs). These systems provide powerful data management capabilities. However, as the power and functionality of these systems has grown, so has the complexity of their administration. In this paper, we show how the IBM DB2 Universal Database for Linux, UNIX, and Windows (DB2 UDB) product has exploited autonomic computing to reduce this administration complexity and become more self-managing. We survey the major autonomic computing features in the DB2 UDB product and describe the benefits, with experimental data in some cases

Towards a Hybrid Design for Fast Query Processing in DB2 with BLU Acceleration Using Graphical Processing Units: A Technology Demonstration

In this paper, we show how we use Nvidia GPUs and host CPU cores for faster query processing in a DB2 database using BLU Acceleration (DB2s column store technology). Moreover, we show the benefits and problems of using hardware accelerators (more specifically GPUs) in a real commercial Relational Database Management System(RDBMS).We investigate the effect of off-loading specific database operations to a GPU, and show how doing so results in a significant performance improvement. We then demonstrate that for some queries, using just CPU to perform the entire operation is more beneficial. While we use some of Nvidias fast kernels for operations like sort, we have also developed our own high performance kernels for operations such as group by and aggregation. Finally, we show how we use a dynamic design that can make use of optimizer metadata to intelligently choose a GPU kernel to run. For the first time in the literature, we use benchmarks representative of customer environments to gauge the performance of our prototype, the results of which show that we can get a speed increase upwards of 2x, using a realistic set of queries.

Wildfire: Concurrent Blazing Data Ingest and Analytics

We demonstrate Hybrid Transactional and Analytics Processing (HTAP) on the Spark platform by the Wildfire prototype, which can ingest up to ~6 million inserts per second per node and simultaneously perform complex SQL analytics queries. Here, a simplified mobile application uses Wildfire to recommend advertising to mobile customers based upon their distance from stores and their interest in products sold by these stores, while continuously graphing analytics results as those customers move and respond to the ads with purchases.

Optimizing Concurrency Through Automated Lock Memory Tuning in DB2

Lock memory consumption can be difficult to project and can vary rapidly in short amounts of time. This volatility makes lock memory tuning difficult and can result in either significant memory waste if systems are configured for peak requirements, or lock escalation and lock wait if under configured; either of which can cause significant performance penalties. This paper describes an algorithm for adaptive tuning of database lock memory. The DB2 technique adapts the locking memory in real time to mitigate the occurrence of lock escalations. The technique uses a combination of synchronous and asynchronous modification to the locking structures so that it can respond well to rapid immediate growth in locking requirements. The adaptive algorithm also relaxes the locking memory over time so that peak requirements in lock memory will not result in a permanently large allocation of memory to locks. Experimental tests have shown this technique to work well in a number of benchmark and adaptive workloads, converging almost immediately to optimal settings which avoid lock escalations and achieve optimal throughput. The solution has been implemented in DB2 9.