Towards application-specific query processing systems
TTowards application-specific query processingsystems
Dimitrios Vasilas
ScalitySorbonne Université - LIP6 & [email protected]
Marc Shapiro
Sorbonne Université - LIP6 & [email protected]
Bradley King
Sara S. Hamouda
Sorbonne Université - LIP6 & [email protected]
ABSTRACT
Database systems use query processing sub-systems for en-abling efficient query-based data retrieval. An essential as-pect of designing any query-intensive application is tuningthe query system to fit the application’s requirements andworkload characteristics. However, the configuration param-eters provided by traditional database systems do not coverthe design decisions and trade-offs that arise from the geo-distribution of users and data. In this paper, we present avision towards a new type of query system architecture thataddresses this challenge by enabling query systems to bedesigned and deployed in a per use case basis. We proposea distributed abstraction called Query Processing Unit thatencapsulates primitive query processing tasks, and showhow it can be used as a building block for assembling querysystems. Using this approach, application architects can con-struct query systems specialized to their use cases, by con-trolling the query system’s architecture and the placementof its state. We demonstrate the expressiveness of this ap-proach by applying it to the design of a query system thatcan flexibly place its state in the data center or at the edge,and show that state placement decisions affect the trade-offbetween query response time and query result freshness.
A major requirement of user-facing services is providing fastresponses to user requests [4, 5]. Serving user requests in-volves communication from a user’s device to an applicationserver, and then potentially querying data from a backendstore.It is well known that achieving good query processingperformance requires tunning the query processing systemto the needs of different use cases. To achieve that, querysystems expose configuration parameters and knobs such asselecting which indexes to materialize and choosing betweenconsistent and lazy index maintenance. In addition, query systems are able to generate query execution plans adaptedto each individual query (query optimization).However, the users and data of most web services nowa-days are geographically distributed across the globe. In thecontext of geo-distribution, the objectives of fast query re-sponse, consistent results, and low operational cost are in-herently conflicting and create trade-offs.
Query response time.
In a query processing system inwhich data are distributed across multiple data centers, queryexecution may involve communication round-trips acrossdata centers, incurring overhead to query response time.
Query result consistency.
Query systems aim at return-ing up-to-date results, which requires keeping their derivedstate (e.g. indexes, materialized views) in sync with the basedata. In the context of geo-distribution this is often not prac-tical due to the resulting overhead to write operations. Thealternative of updating derived state asynchronously canlead to returning stale query results.
Operational cost.
Network resources across geographi-cally distributed sites are often limited and costly, thereforethe amount of data transferred by the query system acrosssites affects network consumption and hence the system’soperational cost.While techniques for reducing query response time havebeen studied extensively and applied in commercial data-base systems, the aspects of query result consistency andoperational cost have not been sufficiently explored.A crucial design choice that affects the above trade-offsis the placement of the query processing system’s state andcomputations. Existing database systems provide little flexi-bility for configuring query processing state and computa-tion placement [2].We argue that, achieving the right balance between lowresponse time, up-to-date query results, and low operationalcost, requires additional tunning mechanisms. Moreover, thisbalance is different across geo-distributed application due totheir diverse characteristics and requirements. a r X i v : . [ c s . D B ] S e p imitrios Vasilas, Marc Shapiro, Bradley King, and Sara S. Hamouda To address this need, we propose an architectural designpattern that enables application designers to navigate thedesign space of geo-distributed query processing by givingthem control over the placement of the query processingsystem’s state and computations. The key idea is that queryprocessing can be decomposed into basic tasks that can beencapsulated by independent components. We present anarchitecture component abstraction, called Query ProcessingUnit (QPU), that embodies this principle. The QPU abstrac-tion defines a set of interfaces and a communication pro-tocol. Different instantiations of the abstraction implementdifferent functionalities while conforming to the commonspecification. Query processing systems are constructed byinterconnecting QPU instances in a modular microservice-like architecture.We demonstrate the expressiveness of the proposed ap-proach by applying it to the design of a middleware systemthat maintains materialized views in order to speed up queryprocessing. We show how the proposed approach enablesmaterialized views to be flexibly placed in the cloud or atthe edge according to the application’s access patterns andrequirements.This work includes the following contributions: • We propose an architectural design pattern for con-structing and deploying geo-distributed query process-ing systems tailored to the characteristics and require-ments of individual applications. • We realize this design pattern by introducing an ab-straction termed Query Processing Unit aimed to en-able the design of modular query processing systems.We show how a geo-distributed query system can beconstructed using the QPU abstraction (Section 3). • We demonstrate the expressiveness of the QPU-basedarchitectural design and the flexibility that can be ob-tained from its use by applying it to the design of anapplication that maintains materialized views at theedge (Section 4).
Internet services typically rely on a two-tiered backend ar-chitecture to serve user requests. Data is stored persistentlyin a database (data storage tier). A query processing tier isused on top of the database to provide the required queryprocessing capabilities to the service.The query processing tier can serve two roles. In caseswhere the database does not support querying, the query pro-cessing tier is responsible for providing querying capabilitiesto the service, for example by building secondary indexes. Al-ternatively, the query processing tier’s responsibility may beto improve query scalability and performance by maintainingpre-computed or cached query results. Typically, the query processing layer maintains derived state (indexes, caches,materialized views) for serving queries. Changes to the basedata need to be propagated and applied to the derived state(state maintenance). State maintenance is often asynchro-nous, and therefore derived state is eventually consistentrelative to the base data.We model the system as a collection of sites . We define asite as a group of servers located in the same geographic loca-tion. Network communication latency among servers withina site is significantly lower than latency among servers ondifferent sites (a few milliseconds compared to tens to hun-dreds of milliseconds). Moreover, network resources withina site cost significantly less compared to network resourcesacross sites. This is in accordance with cloud service pricingmodels where cross-site data transfer is more expensive thandata transfer within a site. [1]. A site can correspond eitherto a data center, or to a collection of edge nodes that form atier along with the user devices located geographically closeto them.In this paper, we examine the challenges and design deci-sions involved in the design of the query processing tier. Weassume that the query processing tier is deployed on top ofan already existing data storage tier architecture.
In this section we present our main contribution, an archi-tectural design pattern for constructing and deploying geo-distributed query processing systems. This pattern decouplesthe query system from the storage architecture, and shiftsthe task of designing and implementing it from the databasedesigner to the application architect.In particular, our design is based on the following objec-tives: • Declaratively defined architecture.
The query sys-tem architecture should not be predefined. Rather,system architects should be able to construct queryprocessing architectures in a per use case basis, andhave control over the query processing techniquesthat the system employs. The query system designprocess should include decisions such as selecting thequery processing state partitioning and replicationschemes, and whether to use caching. In addition, thearchitecture should not make assumptions about thedistribution scheme of the base data. • Flexible component placement.
The system designershould have fine-grained control over the placementof the query system’s state and computations.
We enable these objectives using an assembly-based designstrategy. The key idea is that query processing systems can beconstructed by assembling composable building blocks that owards application-specific query processing systems encapsulate primitive query processing tasks. In that way,complex query processing tasks can be expressed throughcomposition of simple building blocks. To enable this archi-tecture design pattern, we introduce an architecture compo-nent abstraction, termed Query Processing Unit (QPU).The QPU abstraction has the role of an architecture com-ponent template: it defines a set of properties including inter-faces, functionalities, and communication patterns. Differentinstantiations (classes) of the QPU abstraction can be definedand implemented, but all need to conform to the propertiesdefined by the abstraction. Instances of these QPU classesare the concrete building blocks that can be composed toconstruct query processing systems. In the rest of this sec-tion we use the terms query processing unit , QPU , and unit interchangeably.We define the query processing unit as a long-runningprocess with the following properties: • Query processing state (optional).
A QPU can ei-ther maintain state, or be stateless. Stateful QPUs canexpress query processing tasks such as indexing (inwhich case the state is an indexing data structure),caching, and aggregation computations. Stateless QPUscan express tasks such as filtering and projections. • Query API.
Each QPU exposes an API for receivingand serving queries. When this API is called, a streamconnection is established between the QPU and thecaller. Query result entries are sent though the streamas stream records.The unit implements a query processing computation that is invoked when the query API is called. The im-plementation of this computation is different for eachQPU class. However, any implementation can use twofunctionalities: reading from the unit’s state and per-forming downstream queries. • Downstream queries.
Each query processing unitcan invoke the query API of other units. • Callback computation.
When a QPU invokes thequery API of another, a stream connection is estab-lished between them, as described above. The unitimplements a callback computation that is invoked inresponse to receiving a record through that stream. • Configuration state.
Each QPU maintains additionalconfiguration that specify its query processing capa-bilities. For example, in the case of an index QPU classthis may include configuration parameters specify-ing which attribute it is responsible for indexing (andhence which queries it can process). In addition, it in-cludes information that enables it to send downstreamqueries to other units, such as their endpoints andquery processing capabilities.
Client QPU 1
Query processing Q A QPU State Q A.1 R A R A.1 Q A.2 R A.2
Callback Callback
QPU 0 QPU 2
Figure 1: A conceptual depiction the QPU abstraction.
A conceptual depiction of the query processing unit ab-straction is shown in Figure 1. When the QPU’s query API iscalled, a response stream ( R A ) is established between the unitand the client, and the unit’s query processing computationis invoked. The query processing computation can read theQPU’s state, and can perform downstream queries to otherunits. For each downstream query, a corresponding streamis established ( Q A . and Q A . ). When a record is receivedfrom one of the streams, the QPU’s callback computation isinvoked. Each callback computation processes the receivedrecord, and returns the result to the query processing compu-tation. Upon receiving a result from the callback, the queryprocessing computation can write to the QPU’s state and po-tentially send a computed query result through the responsestream. A query processing system is a directed acyclic graph (DAG)with QPUs as its nodes. Edges are connections betweenQPUs, which represent potential paths of communicationamong them: the QPU at a parent node can perform a down-stream query to the QPU at the child node. Leaf nodes com-municate with the data storage tier, while clients performqueries by invoking the query API of root nodes. The globalcapabilities of a QPU-based DAG are synthesized from theindividual functionalities of each of the QPU classes at itsnodes, as well as the graph topology.
The QPU graph runs a distributed bidirectional data-flowcomputation.A client performs a query Q c by invoking the query API ofa query processing unit at the root of the graph. As described imitrios Vasilas, Marc Shapiro, Bradley King, and Sara S. Hamouda in Section 3.1, the QPUs query processing computation canread from the unit’s state, or perform downstream queriesto QPUs at its child nodes.When a downstream query is performed, this process isrecursively executed at each unit whose query API is invoked.Though this mechanism, Q c is incrementally transformedto sub-queries which flow downwards through the QPUgraph, invoking computations at different nodes. Sub-queryresults are returned through the QPU streams establishedfrom query API invocations, and flow upwards through thegraph. These results are incrementally processed, potentiallyupdating the state of different QPUs, and eventually producethe initial query results, which are returned to the client. In this section we describe the process of constructing anddeploying a QPU-based query processing system, and showhow this design pattern achieves our objectives.The process of constructing a query processing systemconsists of four steps. The first step is selecting the QPUclasses to be used for implementing the query processingfunctionalities needed by the application. The second step isdesigning the QPU DAG topology by defining the instancesof QPU classes to be used as nodes, and the connectionsamong them. The third step is defining the placement of eachgraph node across the system infrastructure. The final step isdeploying the QPU graph. This step involves deploying eachprocess that implements a QPU class instance, and passingto it configuration that defines each functionality and itschildren in the QPU graph.The properties of the query processing unit abstraction,and the properties that emerge from the composition ofQPUs realize our design objectives.
Declaratively defined architecture.
The properties ofthe QPU abstraction (common query API, downstream queries)make query processing units composable. This enables aquery processing system to be expressed as a composition ofbuilding blocks (QPUs) that provide basic query processingfunctionalities. Different query system architectures can beexpressed by selecting the QPU classes to be used and thetopology of the connections among them.In addition, the common query API across any QPU classenables the separation between interface and implementa-tion. For example, different implementations of an indexQPU class can use different index data structures. More gen-erally, a graph node is agnostic of the sub-graph below eachof its children and only requires local information aboutthe query capabilities of its children. For example, a QPUclass that implements query result caching can be transpar-ently connected to the root of any QPU sub-graph, such as an individual index QPU or a sub-graph that implements apartitioned index.
Flexible component placement.
Query processing unitsact as microservices: each unit manages its internal state,and communication is performed through API invocations.Therefore, each individual QPU of the query system DAG canbe independently placed across the system, without impact-ing the system’s functionality. This property decouples thearchitecture design from the placement of its components,and gives the system designer control over the placementthe query system’s state and computations.
We consider an application that provides an in-game ad-vertisement service for mobile game development. Gameapplications use this service to query a database for avail-able advertisement videos (ads), select an ad based on somecriteria, and then request the ad from the service in orderto display it to the user. We are interested in the part of theapplication’s architecture responsible for serving queries tothe ad database, and consider the video serving architectureout of the scope of this study.Each ad is associated with a set of tags describing its con-tent, and a price. The ad-serving application and game de-velopers are paid per-click according to the ad price. Tagsare used to select ads with content that users are likely toclick on.Advertisers perform writes to the database to add or re-move ads, and to adjust ad prices. Because the available adsand their prices are modified frequently, ad selection is per-formed at the point of the game when an ad needs to bedisplayed. Therefore, queries require low latency as theyimpact user experience. Additionally, queries require freshresults, as having the latest information about available adsand prices enables ad selection to make optimal choices formaximizing profit.
The ad-serving application’s storage tier consists of a data-base that stores the available ads, their associated tags (tableAds), and their prices (table Prices).We are interested in the design of the application’s queryprocessing tier. Our goal is to achieve low query responsetime while also providing fresh query results. The common-case query performed by games has the following character-istics: (1) it selects ads that contain tags among a given set,(2) joins the selected ads with the associated prices, and (3)selects the ads with the K highest prices. owards application-specific query processing systems A technique for providing low query response time is touse materialized views [5]. However, existing approachesexecute queries in the data center. The ad-serving applica-tion’s users are distributed worldwide, and therefore, thecommunication latency between user devices and the datacenter may be significant. Placing data geographically closerto end users is a common technique for reducing the largeaccess latencies resulting from geo-distribution [3].
We assume a system composed of two types of sites : datacenters and edge sites. The storage tier is placed in the datacenter, while the query processing tier can be distributedacross data center and edge. For simplicity, for the rest ofthis section, we focus on a single data center and edge site.The query system is queried by clients located at the edge;it keeps its derived state up to date by asynchronously re-ceiving updates from the database, which is located in thedata center. As a result, query performance is affected by thelatency between the query system’s state and clients, whilethe consistency between base data and the query system’sstate is affected by the latency between them.There exists thus an inherent trade-off between query per-formance and freshness which is affected by the placementof the query system’s state relative to the clients and the basedata. Placement at the edge can improve query performanceat the expense of query result freshness. Therefore, it is moresuitable for parts of the state that are heavily queried. Onthe other hand, placement in the data center is better suitedfor parts of the state that are more frequently updated.Additionally, placement decisions affect network consump-tion costs. When the query system’s state is placed in thedata center, costly inter-site communication is required forsending queries and responses between the query systemand the clients. When it is placed at the edge, inter-site com-munication is required for propagating updates from basedata to the query system.
In this section, we present the design of a QPU-based queryprocessing system for the ad-serving application describedabove. We demonstrate how the proposed architectural de-sign pattern enables flexible placement of the query system’sstate across data center and edge.
QPU Classes.
Following the steps defined in Section 3.4,we first define the QPU classes that will be used for con-structing the query system. • Data store driver QPU (DSD-QPU).
The DSD-QPUis responsible for connecting the query processing sys-tem DAG with the underlying database. Its query APIprovides the following functionality. Given a query, the
Ads Prices
Write Query D a t a c en t e r E dge DS-QPU DS-QPUI-QPU J-QPUTK-QPU (a)
Ads Prices
Write Query D a t a c en t e r E dge DS-QPU DS-QPUI-QPU J-QPUC-QPUTK-QPU (b)
Figure 2: Alternative query system designs and place-ment schemes for the in-game advertising application,addressing different workload types.
DSD-QPU initially returns the query result (by send-ing each result entry through the response stream). Itthen continues publishing (as stream records) changesto the database state that affect the result of the givenquery. • Index QPU (I-QPU).
The I-QPU is responsible formaintaining an index for a database table column.Upon deployment, the index QPU performs a down-stream query to a DSD-QPU that is responsible forthe corresponding column. It initially receives the en-tire column and builds its index. It then incrementallyupdates the index by receiving updates for databasewrites from the DSD-QPU. Its query processing com-putation processes a given query by performing anindex lookup. • Join QPU (J-QPU).
The join QPU is responsible forperforming the SQL join operation on two or moredatabase tables or materialized views. Similarly to theindex QPU, it performs downstream queries to otherQPUs in order to build the join operation result, andkeep up to date by incrementally applying updates. • Top-K QPU (TK-QPU).
The top-K QPU is responsi-ble for maintaining a materialized view with the K records of a database table or materialized view withthe highest values in a given column. Similarly to theindex and join QPUs, it uses downstream queries toimplement this functionality. • Cache QPU (C-QPU).
The cache QPU is responsiblefor storing recent query results in a cache. When acache miss occurs, it forwards the query further downthe graph through a downstream query.Figure 2 depicts the two query processing system architec-tures for the application. The index QPU maintains an index imitrios Vasilas, Marc Shapiro, Bradley King, and Sara S. Hamouda for the ad tags. The join QPU performs a join between theindex and the
Prices table. As a result of the join, index en-tries contain ads and corresponding prices. The top-K QPUmaintains, for each index entry, the K ads with the highestprices.We present two alternative query systems, designed toaddress different workloads. In Figure 2a, the top-K QPUis placed at the edge. This design is better suited to query-heavy workloads, as it favors query performance but sacri-fices query result freshness. In Figure 2b, the top-K QPU isplaced in the data center, and a cache QPU is placed at theedge. This design is more suitable for update-heavy work-loads. Placing the top-K QPU in the data center achievesbetter freshness, while the cache QPU placed at the edge canpotentially speed up query processing. CockroachDB, a globally-distributed SQL database, providesdifferent choices for distributing and placing data acrossmultiple data centers, termed topology-patterns, aimed atreducing read and write latency and improving resiliency[2]. However, these placement options are primarily focusedon the data storage tier, rather that the query processingsystem.The indexing systems in Azure DocumentDB [6] and Diff-index [7] allow database administrators to choose betweenmultiple index update modes. While this enables a trade-offbetween index consistency and the overhead incurred toupdate operations, these works do not consider queryingprocessing across multiple data centers.Noria [5] is a middleware system aimed at improvingperformance of read-heavy web applications. Similarly to ourapproach, Noria uses data-flow to incrementally apply writesto materialized views. However, Noria does not consider geo-distributed data, and is not focused on flexible materializedview placement.Our previous work [8] addressed the problem of buildingmodular query processing systems and introduced the termQuery Processing Unit. This paper advances this work bypresenting a detailed specification of the QPU abstractionand demonstrating how it can be used to enable flexibilityin the design of query processing systems.
In the context of geo-distribution, the placement of queryprocessing state affects query response time, query resultfreshness, and operational cost. Different design choices caterto different use cases due to applications’ diverse require-ments and characteristics.To address this challenge, we proposed an architecturaldesign pattern for constructing query processing systems in a per use case basis. To realize this, we introduced theQuery Processing Unit abstraction and showed how it canbe used as a building block to enable the construction ofapplication-specific query systems.In this work-in-progress paper, we presented the QPUabstraction and the modular system design pattern that itenables. We have implemented the proposed approach inthe form of a framework that includes (1) a library of QPUimplementations and (2) deployment, orchestration, and self-configuration mechanisms aimed to facilitate the construc-tion of query processing systems. Our prototype implemen-tation is available at github.com/dvasilas/proteus. We arecurrently in the process of experimentally evaluating ourprototype.In addition, we aim to extend this work with mechanismsfor reducing the complexity exposed to the application de-veloper. In particular, we are developing a cost model basedapproach for generating and deploying query engine archi-tectures based on use case descriptions that include the topol-ogy of the underlying storage system, the expected work-load characteristics, and the application’s requirements inthe form of a target metric (query performance, freshness orcost) to optimize for.
REFERENCES
Presented as part of the 2013 USENIXAnnual Technical Conference (USENIX ATC 13) , pages 49–60, San Jose,CA, 2013. USENIX.[5] J. Gjengset, M. Schwarzkopf, J. Behrens, L. T. Araújo, M. Ek, E. Kohler,M. F. Kaashoek, and R. Morris. Noria: dynamic, partially-stateful data-flow for high-performance web applications. In , pages 213–231, Carlsbad, CA, Oct. 2018. USENIX Association.[6] D. Shukla, S. Thota, K. Raman, M. Gajendran, A. Shah, S. Ziuzin, K. Sun-daram, M. G. Guajardo, A. Wawrzyniak, S. Boshra, R. Ferreira, M. Nas-sar, M. Koltachev, J. Huang, S. Sengupta, J. Levandoski, and D. Lomet.Schema-agnostic indexing with azure documentdb.
Proc. VLDB Endow. ,8(12):1668–1679, Aug. 2015.[7] W. Tan, S. Tata, Y. Tang, and L. L. Fong. Diff-index: Differentiated indexin distributed log-structured data stores. In
Proceedings of the 17thInternational Conference on Extending Database Technology, EDBT 2014,Athens, Greece, March 24-28, 2014 , pages 700–711. OpenProceedings.org,2014.[8] D. Vasilas, M. Shapiro, and B. King. A modular design for geo-distributedquerying: Work in progress report. In