Hasso Plattner

A common database approach for OLTP and OLAP using an in-memory column database

When SQL and the relational data model were introduced 25 years ago as a general data management concept, enterprise software migrated quickly to this new technology. It is fair to say that SQL and the various implementations of RDBMSs became the backbone of enterprise systems. In those days. we believed that business planning, transaction processing and analytics should reside in one single system. Despite the incredible improvements in computer hardware, high-speed networks, display devices and the associated software, speed and flexibility remained an issue. The nature of RDBMSs, being organized along rows, prohibited us from providing instant analytical insight and finally led to the introduction of so-called data warehouses. This paper will question some of the fundamentals of the OLAP and OLTP separation. Based on the analysis of real customer environments and experience in some prototype implementations, a new proposal for an enterprise data management concept will be presented. In our proposal, the participants in enterprise applications, customers, orders, accounting documents, products, employees etc. will be modeled as objects and also stored and maintained as such. Despite that, the vast majority of business functions will operate on an in memory representation of their objects. Using the relational algebra and a column-based organization of data storage will allow us to revolutionize transactional applications while providing an optimal platform for analytical data processing. The unification of OLTP and OLAP workloads on a shared architecture and the reintegration of planning activities promise significant gains in application development while simplifying enterprise systems drastically. The latest trends in computer technology -- e.g. blade architecture, multiple CPUs per blade with multiple cores per CPU allow for a significant parallelization of application processes. The organization of data in columns supports the parallel use of cores for filtering and aggregation. Elements of application logic can be implemented as highly efficient stored procedures operating on columns. The vast increase in main memory combined with improvements in L1--, L2--, L3--caching, together with the high data compression rate column storage will allow us to support substantial data volumes on one single blade. Distributing data across multiple blades using a shared nothing approach provides further scalability.

HYRISE: a main memory hybrid storage engine

In this paper, we describe a main memory hybrid database system called HYRISE, which automatically partitions tables into vertical partitions of varying widths depending on how the columns of the table are accessed. For columns accessed as a part of analytical queries (e.g., via sequential scans), narrow partitions perform better, because, when scanning a single column, cache locality is improved if the values of that column are stored contiguously. In contrast, for columns accessed as a part of OLTP-style queries, wider partitions perform better, because such transactions frequently insert, delete, update, or access many of the fields of a row, and co-locating those fields leads to better cache locality. Using a highly accurate model of cache misses, HYRISE is able to predict the performance of different partitionings, and to automatically select the best partitioning using an automated database design algorithm. We show that, on a realistic workload derived from customer applications, HYRISE can achieve a 20% to 400% performance improvement over pure all-column or all-row designs, and that it is both more scalable and produces better designs than previous vertical partitioning approaches for main memory systems.

SIMD-scan: ultra fast in-memory table scan using on-chip vector processing units

The availability of huge system memory, even on standard servers, generated a lot of interest in main memory database engines. In data warehouse systems, highly compressed column-oriented data structures are quite prominent. In order to scale with the data volume and the system load, many of these systems are highly distributed with a shared-nothing approach. The fundamental principle of all systems is a full table scan over one or multiple compressed columns. Recent research proposed different techniques to speedup table scans like intelligent compression or using an additional hardware such as graphic cards or FPGAs. In this paper, we show that utilizing the embedded Vector Processing Units (VPUs) found in standard superscalar processors can speed up the performance of mainmemory full table scan by factors. This is achieved without changing the hardware architecture and thereby without additional power consumption. Moreover, as on-chip VPUs directly access the systems RAM, no additional costly copy operations are needed for using the new SIMD-scan approach in standard main memory database engines. Therefore, we propose this scan approach to be used as the standard scan operator for compressed column-oriented main memory storage. We then discuss how well our solution scales with the number of processor cores; consequently, to what degree it can be applied in multi-threaded environments. To verify the feasibility of our approach, we implemented the proposed techniques on a modern Intel multi-core processor using Intel® Streaming SIMD Extensions (Intel® SSE). In addition, we integrated the new SIMD-scan approach into SAP® Netweaver® Business Warehouse Accelerator. We conclude with describing the performance benefits of using our approach for processing and scanning compressed data using VPUs in column-oriented main memory database systems.

Fast updates on read-optimized databases using multi-core CPUs

Read-optimized columnar databases use differential updates to handle writes by maintaining a separate write-optimized delta partition which is periodically merged with the read-optimized and compressed main partition. This merge process introduces significant overheads and unacceptable downtimes in update intensive systems, aspiring to combine transactional and analytical workloads into one system. In the first part of the paper, we report data analyses of 12 SAP Business Suite customer systems. In the second half, we present an optimized merge process reducing the merge overhead of current systems by a factor of 30. Our linear-time merge algorithm exploits the underlying high compute and bandwidth resources of modern multi-core CPUs with architecture-aware optimizations and efficient parallelization. This enables compressed in-memory column stores to handle the transactional update rate required by enterprise applications, while keeping properties of read-optimized databases for analytic-style queries.

Optimizing write performance for read optimized databases

Compression in column-oriented databases has been proven to offer both performance enhancements and reductions in storage consumption. This is especially true for read access as compressed data can directly be processed for query execution.Nevertheless, compression happens to be disadvantageous when it comes to write access due to unavoidable re-compression: write-access requires significantly more data to be read than involved in the particular operation, more tuples may have to be modified depending on the compression algorithm, and table-level locks have to be acquired instead of row-level locks as long as no second version of the data is stored. As an effect the duration of a single modification — both insert and update — limits both throughput and response time significantly. In this paper, we propose to use an additional write-optimized buffer to maintain the delta that in conjunction with the compressed main store represents the current state of the data. This buffer facilitates an uncompressed, column-oriented data structure. To address the mentioned disadvantages of data compression, we trade write-performance for query-performance and memory consumption by using the buffer as an intermediate storage for several modifications which are then populated as a bulk in a merge operation. Hereby, the overhead created by one single re-compression is shared among all recent modifications. We evaluated our implementation inside SAP’s in memory column store. We then analyze the different parameters influencing the merge process, and make a complexity analysis. Finally, we show optimizations regarding resource consumption and merge duration.

Predicting in-memory database performance for automating cluster management tasks

In Software-as-a-Service, multiple tenants are typically consolidated into the same database instance to reduce costs. For analytics-as-a-service, in-memory column databases are especially suitable because they offer very short response times. This paper studies the automation of operational tasks in multi-tenant in-memory column database clusters. As a prerequisite, we develop a model for predicting whether the assignment of a particular tenant to a server in the cluster will lead to violations of response time goals. This model is then extended to capture drops in capacity incurred by migrating tenants between servers. We present an algorithm for moving tenants around the cluster to ensure that response time goals are met. In so doing, the number of servers in the cluster may be dynamically increased or decreased. The model is also extended to manage multiple copies of a tenants data for scalability and availability. We validated the model with an implementation of a multi-tenant clustering framework for SAPs in-memory column database TREX.

A Simulation of the Pharmaceutical Supply Chain to Provide Realistic Test Data

The amount of counterfeit pharmaceuticals in the Europeanpharmaceutical supply chain increased enormously inthe past years. Thus, the European Commission introducedan amendment which will lead to new information systems.No realistic test data for these information systems isavailable yet which hinders the progress in designing andimplementing appropriate information systems. We make afirst step to close this gap by providing realistic test datawhich respects the upcoming legislative changes. For thispurpose, we provide four different scenarios which differ insupply chain size and an aspect of the legal requirementswhich is still subject to consideration. The test data isavailable on the web, so researchers and software engineerscan use it to evaluate their information systems.

User, Use & Utility Research The Digital User as New Design Perspective in Business and Information Systems Engineering

Business and Information Systems Engineering (BISE) is at a turning point. Planning, designing, developing and operating IT used to be a management task of a few elites in public ad-ministrations and corporations. But the continuous digitization of nearly all areas of life changes the IT landscape fundamentally. Success in this new era requires putting the human perspective – the digital user – at the very heart of the new digitized service-led economy.BISE faces not just a temporary trend but a complex socio-technical phenomenon with far-reaching implications. The challenges are manifold and have major consequences for all stakeholders, both in information systems and management research as well as in practice. Corporate processes have to be re-designed from the ground up, starting with the user’s perspective, thus putting usage experience and utility of the individual center stage.The digital service economy leads to highly personalized application systems while organizational functions are being fragmented. Entirely new ways of interacting with information systems, in particular beyond desktop IT, are being invented and established. These fundamental challenges require novel approaches with regards to innovation and development methods as well as adequate concepts for enterprise or service system architectures. Gigantic amounts of data are being generated at an accelerating rate by an increasing number of devices – data that need to be managed.In order to tackle these extraordinary challenges we introduce ‘user, use & utility’ as a new field of BISE that focuses primarily on the digital user, his or her usage behavior and the utility associated with system usage in the digitized service-led economy.The research objectives encompass the development of theories, methods and tools for systematic requirement elicitation, systems design, and business development for successful Business and Information Systems Engineering in a digitized economy – information systems that digital users enjoy using. This challenge calls for leveraging insights from various scientific disciplines such as Design, Engineering, Computer Science, Psychology and Sociology. BISE can provide an integrated perspective, thereby assuming a pivotal role within the digitized service led economy.

A Course in In-Memory Data Management

Click the title of book to access its contents. NOTE: Books are accessible in all university campuses via internet facility provided by CUK and also through remote access facility provided by Central Library, CUK. Please feel free to contact the library staff in case of any clarification or assitance. We are here to better serve your information needs. S.No Title Author(s)/Editor(s) Subject Volume Edition Year

The impact of columnar in-memory databases on enterprise systems: implications of eliminating transaction-maintained aggregates

Five years ago I proposed a common database approach for transaction processing and analytical systems using a columnar in-memory database, disputing the common belief that column stores are not suitable for transactional workloads. Today, the concept has been widely adopted in academia and industry and it is proven that it is feasible to run analytical queries on large data sets directly on a redundancy-free schema, eliminating the need to maintain pre-built aggregate tables during data entry transactions. The resulting reduction in transaction complexity leads to a dramatic simplification of data models and applications, redefining the way we build enterprise systems. First analyses of productive applications adopting this concept confirm that system architectures enabled by in-memory column stores are conceptually superior for business transaction processing compared to row-based approaches. Additionally, our analyses show a shift of enterprise workloads to even more read-oriented processing due to the elimination of updates of transaction-maintained aggregates.