Stefan Deßloch

Towards generating ETL processes for incremental loading

Extract, Transform, and Load (ETL) processes physically integrate data from multiple, heterogeneous sources in a central repository referred to as data warehouse. Physically integrated data gets stale when source data is changed, hence periodic refreshes are required. For efficiency reasons data warehouses are typically refreshed incrementally, i.e. changes are captured at the sources and propagated to the data warehouse on a regular basis. Dedicated ETL processes referred to as incremental load processes are employed to extract changes from the sources, propagate the changes, and refresh the data warehouse incrementally. Changes required in the data warehouse are inferred from changes captured at the sources during change propagation. The creation of incremental load processes is a complex task reserved to trained ETL programmers. In this paper we review existing Change Data Capture (CDC) techniques and discuss limitations of different approaches. We further review existing techniques for refreshing data warehouses. We then present an approach for generating incremental load processes from abstract schema mappings.

Handling Integrity in a KBMS Architecture for Workstation/Server Environments

This paper presents the concepts and mechanisms for maintaining semantic integrity of complex applications, which are provided by the knowledge base management system KRISYS. We discuss the integration of integrity maintenance into the global architecture of the system which is based on a workstation/server environment. As a result of our elaborations, we argue that semantic integrity should be handled in the knowledge model (at the workstation), leaving only basic consistency tasks for the data model (at the server).

NotaQL Is Not a Query Language! It’s for Data Transformation on Wide-Column Stores

It is simple to query a relational database because all columns of the tables are known and the language SQL is easily applicable. In NoSQL, there usually is no fixed schema and no query language. In this article, we present NotaQL, a data-transformation language for wide-column stores. NotaQL is easy to use and powerful. Many MapReduce algorithms like filtering, grouping, aggregation and even breadth-first-search, PageRank and other graph and text algorithms can be expressed in two or three short lines of code.

Enhancing knowledge processing in client/server environments

A great variety of techniques has been developed to optimize and enhance query processing for relational, client/server, distributed, parallel, and heterogeneous database systems (DBS). Based on that work and experience, we investigate how far those techniques are applicable to query processing in Knowledge Base Management Systems (KBMS). Our reference system is the KRISYS KBMS that consists of a knowledge-processing system at the client (clientbased processing) and a data-processing system at the server (database backend). We describe a unifying framework for query processing incorporating both processing systems (as realized in KRISYS). This allows to distribute and balance the amount of work done in the client and in the server. Based on an evaluation of that framework, several approaches to further enhance knowledge processing are reported.

Cross-system NoSQL data transformations with NotaQL

The rising adoption of NoSQL technology in enterprises causes a heterogeneous landscape of different data stores. Different stores provide distinct advantages and disadvantages, making it necessary for enterprises to facilitate multiple systems for specific purposes. This resulting polyglot persistence is difficult to handle for developers since some data needs to be replicated and aggregated between different and within the same stores. Currently, there are no uniform tools to perform these data transformations since all stores feature different APIs and data models. In this paper, we present the transformation language NotaQL that allows cross-system data transformations. These transformations are output-oriented, meaning that the structure of a transformation script is similar to that of the output. Besides, we provide an aggregation-centric approach, which makes aggregation operations as easy as possible.

The KRISYS Project: a Summary of What We Have Learned so far

KRISYS is a prototype of a Knowledge Base Management System whose first implementation was completed at the University of Kaiserslautern in 1989. Since then, the system has been used for the development of various applications which allowed us to perform a well-founded evaluation of the system. In this paper, we summarize our evaluation by describing the major lessons we have learned from the design and implementation of KRISYS and, above all, from its use in the development of these applications. We address issues related with the concepts available for application modeling and processing, the support of designing an application, as well as the overall means for efficient processing in a workstation/server environment. Additionally, we point out in how far these experiences validate our approach or stimulate improvements and future research.

Design and implementation of advanced knowledge processing in the KBMS KRISYS

Advanced data models and knowledge models together with their powerful query and manipulation languages have already proven to be essential for systems that support non-standard applications such as engineering and knowledge-based application systems. In order to raise their usability and acceptability, it is overly important to provide adequate implementation techniques that guarantee extensible and efficient processing for this advanced DBMS scenario. In this paper we present design alternatives and implementation techniques for such kinds of advanced DBMS, strongly focussing on query and knowledge processing in client/server architectures. To discuss our considerations and implementation technologies, we refer to the knowledge-processing framework of the KBMS KRISYS, although our ideas are generally applicable to (advanced) DBMS.

Inkrementelle Neuberechnungen in MapReduce

ZusammenfassungDas MapReduce-Programmiermodell ermöglicht die skalierbare Analyse und Transformation großer Datenmengen. Wir stellen das auf MapReduce basierende Marimba-Framework zur einfachen Entwicklung von inkrementellen, selbstwartbaren Programmen vor, welche bei Änderung von Quelldaten eine vollständige Wiederholung des MapReduce-Jobs vermeiden. Marimba wird anhand mehrerer Anwendungen illustriert und durch Leistungsmessungen evaluiert.

COCOON and KRISYS — a comparison —

The design of Object Based Systems, which map object-oriented, semantic or knowledge representation models to an underlying database kernel system, in order to close the gap between application and database system, are currently considered as candidates for future database architectures. KRISYS (University of Kaiserslautern) and COCOON (ETH Zurich), as two prototype systems following this kernel architecture approach, will be compared in this report. We give a very short overview of the underlying models and query languages as well as their mapping approaches and their processing models, compare the corresponding features and discuss differences in both systems.

COCOON and KRISYS - a survey and comparison

The design of Object Based Systems, which map object oriented–, semantic– or knowledge representation models to an underlying database kernel system, in order to close the gap between application and database systems, are currently considered as candidates for future database architectures. KRISYS (University of Kaiserslautern) and COCOON (ETH Zurich), as two prototype systems following this kernel architecture approach, will be compared in this report. We give an overview of the underlying models and query languages as well as their mapping approaches and their processing models, compare the corresponding features, and discuss differences in both systems.