Robin Dhananjay Dhamankar

Transaction log based application error recovery and point in-time query

Database backups have traditionally been used as the primary mechanism to recover from hardware and user errors. High availability solutions maintain redundant copies of data that can be used to recover from most failures except user or application errors. Database backups are neither space nor time efficient for recovering from user errors which typically occur in the recent past and affect a small portion of the database. Moreover periodic full backups impact user workload and increase storage costs. In this paper we present a scheme that can be used for both user and application error recovery starting from the current state and rewinding the database back in time using the transaction log. While we provide a consistent view of the entire database as of a point in time in the past, the actual prior versions are produced only for data that is accessed. We make the as of data accessible to arbitrary point in time queries by integrating with the database snapshot feature in Microsoft SQL Server.

InProcDiskSim : testing database recovery on commodity disk drives

Database recovery mechanisms depend on various guarantees provided by the underlying storage media. With the changing industrial landscape, database systems are deployed on storage media that may not provide all the guarantees that traditional implementations may have relied on. In the absence of these stricter guarantees, enhancements may have to be made to the database engine to support newer storage media. Testing database recoverability with the new implementation requires simulation of predictable failure conditions which may be difficult, costly and error-prone. In this paper we present our experience in testing the enhancements made to enforce recoverability in the absence of write-through guarantees in commodity disks. We propose InProcDiskSim, a test framework that simulates the relevant features of the storage media running in the same process as the database engine, thereby providing a cheap, predictable and programmatic method to test context specific failure conditions. We also discuss how our methods apply beyond the specific feature that they were originally used to test.

DistributedLog: A High Performance Replicated Log Service

DistributedLog is a high performance, strictly ordered, durably replicated log. It is multi-tenant, designed with a layered architecture that allows reads and writes to be scaled independently and supports OLTP, stream processing and batch workloads. It also supports a globally synchronous consistent replicated log spanning multiple geographically separated regions. This paper describes how DistributedLog is structured, its components and the rationale underlying various design decisions. We have been using DistributedLog in production for several years, supporting applications ranging from transactional database journaling, real-time data ingestion, and analytics to general publish-subscribe messaging.