In-Order Sliding-Window Aggregation in Worst-Case Constant Time
IIn-Order Sliding-Window Aggregation in Worst-Case Constant Time
Kanat Tangwongsan · Martin Hirzel · Scott SchneiderAbstract
Sliding-window aggregation is a widely-used ap-proach for extracting insights from the most recent portionof a data stream. The aggregations of interest can usually beexpressed as binary operators that are associative but not nec-essarily commutative nor invertible. Non-invertible operators,however, are difficult to support efficiently. In a 2017 con-ference paper, we introduced DABA, the first algorithm forsliding-window aggregation with worst-case constant time.Before DABA, if a window had size n , the best publishedalgorithms would require O ( log n ) aggregation steps per win-dow operation—and while for strictly in-order streams, thisbound could be improved to O ( ) aggregation steps on aver-age, it was not known how to achieve an O ( ) bound for theworst-case, which is critical for latency-sensitive applications.This article is an extended version of our 2017 paper. Besidesdescribing DABA in more detail, this article introduces a newvariant, DABA Lite, which achieves the same time boundsin less memory. Whereas DABA requires space for storing2 n partial aggregates, DABA Lite only requires space for n + Keywords
Real-time, aggregation, continuous analytics,(de-)amortization
Stream processing is a now-standard paradigm for handlinghigh-speed continuous data, spurring the development ofmany stream-processing engines [2,5,6,9,13,16,23,25,37,42]. Since stream processing is often subject to strict quality-of-service or real-time requirements, it requires low-latency
K. Tangwongsan, Mahidol University, E-mail: [email protected]. Hirzel, IBM Research, E-mail: [email protected]. Schneider, IBM Research, E-mail: [email protected]
Table 1
Aggregation operators with their algebraic properties.Invertible Associative Commutative
Sum-like : sum, count, mean,geomean, stddev, ... (cid:88) (cid:88) (cid:88)
Max-like : max, min, argMax,maxCount, M4 [20], ... × (cid:88) × Mergeable sketch [4] : Bloom,CountMin, HyperLogLog,algebraic classifiers [19], ... × (cid:88) (cid:88) responses. As a mainstay of stream processing, aggregation(e.g., computing the maximum, geometric mean, or moreelaborate summaries such as Bloom filters [8]) is one of themost common computations in streaming applications, usedboth standalone and as a building block for other analyt-ics. Unfortunately, existing techniques for sliding-windowaggregation cannot consistently guarantee low latency.Because the newest data is often deemed more pertinentor valuable than older data, streaming aggregation is typi-cally performed on a sliding window (e.g., the last hour’sworth of data). This not only provides intuitive semantics tothe end users but also helps bound the amount of data thesystem has to keep around. Following Boykin et al. [9], weuse the term aggregation broadly, to include both classical re-lational aggregation operators such as sum, geometric mean,and maximum, as well as a more general class of associativeoperators. Table 1 lists several such operators and character-izes them by their algebraic properties. While some operatorsare invertible or commutative, many are not. This paper fo-cuses on algorithms that work with all associative operators,including non-invertible and non-commutative ones.An algorithm for sliding-window aggregation supportsthree operations (formally described in Section 2): insert for a data item’s arrival, query for requesting the currentaggregation outcome, and evict for a data item’s depar- a r X i v : . [ c s . D B ] S e p Kanat Tangwongsan, Martin Hirzel, and Scott Schneider ture [18]. This paper presents the
De-Amortized Banker’sAggregator (DABA) , a novel general-purpose sliding-windowaggregation algorithm that guarantees low-latency responseon every operation—in the worst case, not just on average.The algorithm is simple and supports both fixed-sized andvariable-sized windows. It works as long as (i) the aggre-gation operator, denoted by ⊗ in this paper, is an associa-tive binary operator and (ii) the window has first-in first-out(FIFO) semantics. DABA does not require any other proper-ties from the ⊗ operator. In particular, DABA works equallywell whether ⊗ is invertible or non-invertible, commuta-tive or non-commutative. DABA supports each of the query , insert , and evict operations by making at most a constantnumber of calls to the ⊗ operator in the worst case. This isindependent of the window size, denoted by n in this paper.If each invocation of ⊗ takes constant time, then the DABAalgorithm takes worst-case constant time.We first published the DABA algorithm in a 2015 tech-nical report [31] and later in a 2017 conference paper [32].Prior to the publication of DABA, the algorithms with thebest algorithmic complexity for this problem in the publishedliterature took O ( log n ) time [7,34], i.e., not O ( ) time likeDABA. After the publication of DABA, there have beenother papers with algorithms that take amortized O ( ) timefor FIFO sliding-window aggregation [28,33,35,40]. How-ever, these algorithms maintain the O ( ) time complexityonly in the amortized sense, i.e., not in the worst case likeDABA. In terms of space complexity, for a window of size n ,DABA stores 2 n partial aggregates. This journal version ofthe DABA paper also introduces a new, previously unpub-lished algorithm called DABA Lite that reduces the memoryrequirements to n + O ( ) time complexity and to de-amortizeit. The algorithm that serves as a starting point is Two-Stacks.It uses an old trick from functional programming for repre-senting a FIFO queue with two stacks, which we augmentwith aggregation. A special case of this algorithm was firstmentioned in a Stack Overflow thread [3]. The Two-Stacksalgorithm has one rare—but expensive—operation called flip that transfers all data from one stack to the other. Theflip operation causes a latency spike, which can be undesir-able for low-latency streaming. De-amortization turns theaverage-case O ( ) behavior of Two-Stacks into the worst-case O ( ) behavior of DABA by spreading out the expensiveflip operation, thus eliminating the latency spike.Two-Stacks, DABA, and DABA Lite only work for FIFOwindows, i.e., for sliding windows over in-order streams.Handling out-of-order streams is beyond the scope of thispaper. Indeed, it has been shown that the lower bound onthe time complexity for aggregating out-of-order streams is worse than O ( ) unless disorder in a stream is bounded bya constant [33]. Before the emergence of constant-time slid-ing window aggregation algorithms, a popular approach forachieving low latency was to use coarse-grained windowswhere evictions occur in batches. This approach reduces theeffective window size n , thus making algorithms whose timecomplexity depends on n feasible. Pre-aggregating insideeach batch reduces the cost of aggregating across batches [10,22,24]. But coarse-grained windows are an approximationthat does not always satisfy application requirements. An-other popular research topic in sliding-window aggregationis window sharing , where aggregations for multiple differentwindow sizes are computed on a single data structure. Whilesome sliding window aggregation algorithms support windowsharing in amortized O ( ) time, none of them achieve worst-case O ( ) time [28,33]. DABA and DABA Lite achieveworst-case O ( ) time but do not support window sharing.Experiments show that DABA and DABA Lite performwell in practice. We have implemented our new algorithmsin C++ and benchmarked them against average-case O ( ) algorithms. True to being worst-case O ( ) , our results showthat DABA and DABA Lite have lower latency and com-petitive throughput as we increase the window size. Whenthe aggregation operation is cheap, the low latency and highthroughput are due to constant-time updates to a lightweightdata structure. When the aggregation operation is expensive,they are due to a low-constant number of calls to the costlyaggregation operator.Our implementations of DABA, DABA Lite, and all otheralgorithms used in this paper are available on GitHub fromthe open source project Sliding Window Aggregators . This section formalizes the problem of maintaining aggrega-tion in a first-in first-out (FIFO) sliding window and discussesthe kinds of aggregations supported in this work.2.1 Sliding-Window Aggregation Data TypeThis paper is concerned with sliding-window aggregation onin-order streams with a first-in first-out (FIFO) window. Inthis type of window, the earliest data item to arrive is alsothe earliest data item to leave the window. Hence, the slidingwindow is a queue that supports aggregation of its data. Thefront of the queue contains the earliest data, the back of thequeue holds the latest data, and the aggregation is from theearliest to the latest. As a queue, the window is only affectedby two kinds of changes: https://github.com/IBM/sliding-window-aggregators n-Order Sliding-Window Aggregation in Worst-Case Constant Time 3 Data Arrival:
The arrival of a window data item results ina new data item at the end of the window. This is oftentriggered by the arrival of a data item in a relevant stream.
Data Eviction:
An eviction causes the data item at the frontof the window to be removed from the window. The choiceof when this happens is typically controlled by the win-dow policy (e.g., a time-based window evicts the earliestdata item when it falls out of the time frame of interestand a count-based window evicts the earliest data itemto keep the size fixed [15]). Window eviction policies areorthogonal to the algorithms in this paper.We will model the problem of maintaining aggregation in aFIFO sliding window as an abstract data type (ADT) with aninterface similar to that of a queue. To begin, we review analgebraic structure called a monoid:
Definition: A monoid is a triple M = ( S , ⊗ , ) with a binaryoperator ⊗ : S × S → S on S such that– Associativity:
For a , b , c ∈ S , a ⊗ ( b ⊗ c ) = ( a ⊗ b ) ⊗ c ; and– Identity: ∈ S is the identity: ⊗ a = a = a ⊗ for all a ∈ S .In comparison to real-number arithmetic, the ⊗ operatorcan be seen as a generalization of arithmetic multiplicationwhere the identity element is a generalization of the num-ber 1. Some of our earlier papers instead used an analogy toarithmetic addition with an identity element of zero. Whilethat works equally well, here we adopt the multiplicationanalogy, because it makes it natural to adopt a shorthandnotation of abc for a ⊗ b ⊗ c . That shorthand makes it easierto write detailed examples.A monoid is commutative if a ⊗ b = b ⊗ a for all a , b ∈ S .A monoid has a left inverse if there exists a (known andreasonably cheap) function inv ( · ) such that a ⊗ b ⊗ inv ( a ) = b for all a , b ∈ S . In general, a monoid may not be commutativenor invertible.In the context of aggregation, monoids strike a good bal-ance between generality and efficiency as was demonstratedbefore [9,34,41]. For this reason, we focus our attention onsupporting monoidal aggregation, formulating the abstractdata type as follows: Definition:
The first-in first-out sliding-window aggregation (SWAG) abstract data type maintains a collection of windowdata and supports the following operations: – query () returns the ordered monoidal product of the win-dow data. That is, if the sliding window contains val-ues v , v , . . . , v n − in their arrival order, query returns v ⊗ v ⊗ · · · ⊗ v n − . If the window is empty, it returns . – insert ( v ) adds v to the end of the sliding window. That is,if the sliding window contains values v , v , . . . , v n − intheir arrival order, then insert ( v ) updates the collectionto v (cid:48) , v (cid:48) , . . . , v (cid:48) n , where v (cid:48) i = v i for i = , , . . . , n − v (cid:48) n = v . – evict () removes the oldest item from the front of the slid-ing window. That is to say, if the sliding window containsvalues v , v , . . . , v n − in their arrival order, then evict () updates the collection to v (cid:48) , v (cid:48) , . . . , v (cid:48) n − , where v (cid:48) i = v i + for i = , , , . . . , n − n will denote the size of the current slidingwindow and v , v , . . . , v n − will denote the contents of thesliding window in their arrival order, where v is the old-est element. SWAG itself is not a concrete algorithm; it ismerely an abstract data type, defining a set of operations withtheir expected behavior. The algorithms introduced in thispaper (including Two-Stacks, DABA, and DABA Lite) areall concrete instantiations for the SWAG abstract data type.2.2 Aggregation on MonoidsDespite their simplicity, monoids are expressive enough tocapture most basic aggregations [9,34], as well as more so-phisticated aggregations such as maintaining approximatemembership via a Bloom filter [8], maintaining an approxi-mate count of distinct elements [14], maintaining the versatilecount-min sketch [12], and indeed all operators in Table 1.However, many aggregations (e.g., standard deviation)are not themselves monoids but can be couched as operationson a monoid with the help of two extra steps. To accomplishthis, prior work [34] gives a framework for the developer toprovide three types In , Agg , and
Out , and write three functionsas follows: – lift ( e : In ) : Agg takes an element of the input type and“lifts” it to an aggregation type that will be monoid opera-ble. – combine ( v : Agg , v : Agg ) : Agg is a binary operator oper-ating on the aggregation type. In our paper’s terminology, combine is the monoidal binary operator ⊗ . – lower ( v : Agg ) : Out turns an element of the aggregationtype into an element of the output type.Consider the example of maintaining the maxcount , whichyields the number of times the maximal value occurs in thewindow. Define the type
Agg as a pair (cid:104) m , c (cid:105) comprising themaximum m and its count c . Then, define the three functions lift , combine , and lower as: lift ( e ) = (cid:104) m (cid:55)→ e , c (cid:55)→ (cid:105) combine ( v , v ) = v if v . m > v . mv if v . m < v . m (cid:28) m (cid:55)→ v . m , c (cid:55)→ v . c + v . c (cid:29) if v . m = v . m lower ( v ) = v . c It is easy to show that the combine function is an associa-tive binary operator with identify element = (cid:104)− ∞ , (cid:105) . Con-sequently, M maxcount = ( Agg , combine , (cid:104)− ∞ , (cid:105) ) is a monoid. Kanat Tangwongsan, Martin Hirzel, and Scott Schneider
In this framework, a query is conceptually answered asfollows. If the sliding window currently contains the elements e , e , . . . , e n − , from the earliest to the latest, then lift de-rives v i = lift ( e i ) for i = , , , . . . , n −
1. Then, combine ,rendered as infix ⊗ , computes v = v ⊗ v ⊗ · · · ⊗ v n − . Fi-nally, lower produces the final answer as lower ( v ) .Note that lift only needs to be applied to each elementwhen it first arrives and lower to query results at the end.Therefore, the present paper focuses exclusively on the issueof maintaining the monoidal product—i.e., how to make asfew invocations of combine as possible.2.3 Example TraceUsing the maxcount monoid mentioned previously as a run-ning example for the following sections, we will look at atrace of window operations and their effects on aggregations.Consider a window with the following contents, with theoldest element on the left and the youngest on the right.4, 5, 3, 4, 0, 4, 4, max =5, maxcount =1The largest number in the window is 5, and it occurs onlyonce, so the maxcount is 1. The oldest element on the left is4, which is smaller than the current maximum 5, so evictingit does not affect the maximum or the maxcount.5, 3, 4, 0, 4, 4, max =5, maxcount =1If we again evict the oldest element from the left, themaximum remaining window element becomes 4. Since thenumber 4 occurs thrice in the window, the maxcount is 3.The monoid is not invertible: this update could not havebeen accomplished by “subtracting out” information fromthe previous partial aggregate.3, 4, 0, 4, 4, max =4, maxcount =3Inserting 2 does not affect the maximum, and hence, italso does not affect the maxcount.3, 4, 0, 4, 4, 2, max =4, maxcount =3Finally, inserting 6 changes the maximum. Since thenewly inserted element is the only 6 in the window, themaxcount becomes 1.3, 4, 0, 4, 4, 2, 6, max =6, maxcount =1Notice that in this trace, insert and evict do not strictlyalternate. In general, the SWAG data type, as well as all ouralgorithms, places no restrictions on how insert and evict may be called. They can be arbitrarily interleaved, allowingfor dynamically-sized windows. Two-Stacks is a simple algorithm for in-order sliding windowaggregation (SWAG). For a window size n , it stores a total of2 n partial aggregates and implements each SWAG operationwith amortized O ( ) and worst-case O ( n ) invocations of ⊗ . The text of this section embeds several data-structure vi-sualizations, which are all taken from concrete and completeexample traces shown in Figs. 1 and 2. Two-Stacks Data Structure.
As the name implies, the datastructure for Two-Stacks comprises two stacks. We refer tothem as the front stack F and the back stack B . Here is anexample of these two stacks for maxcount aggregation: valagg F34x3 44x3 04x2 44x2 44x1 B22x1 66x1 56x1 66x2 16x2 66x3 56x3 E The front stack is shown in blue rotated 90 ◦ left, with its topmarked F and its bottom marked B . The back stack is shownin green rotated 90 ◦ right, with its bottom marked B and itstop marked E . To avoid clutter, the visualization only shows B once to mark the bottoms of both the front stack and theback stack. As the window slides, evictions pop elementsfrom the front stack on the left (at F ) and insertions pushelements on the back stack on the right (at E ). Each stackelement is a struct with two partial aggregates, val shownon top and agg shown on the bottom. The notation 4 × max= maxcount= O ( ) if we canshow that by billing the user a constant amount for everyoperation invoked, there is enough money at all time, withouttaking out a loan, to pay for the the actual work being done.We visualize the savings as small golden “coins” above theelements; they are not actual manifest in the data structure. Two-Stacks Invariants.
An invariant for a data structure thatimplements in-order SWAG is a property that holds beforeand after every query , insert , or evict . Let v , . . . , v n − bethe lifted partial aggregates of the current window contents.Each val field stores the corresponding v i . Each agg fieldstores the partial aggregate of the corresponding v i and allother values below it in the same stack. Formally, if F [ i ] and B [ i ] denote the i th element of F and B indexed from the leftstarting at index 0: ∀ i ∈ . . . | F | − F [ i ] . val = v i and ∀ i ∈ . . . | F | − F [ i ] . agg = v i ⊗ . . . ⊗ v | F |− and ∀ i ∈ | F | . . . | F | + | B | − B [ i − | F | ] . val = v i and ∀ i ∈ | F | . . . | F | + | B | − B [ i − | F | ] . agg = v | F | ⊗ . . . ⊗ v i The front stack aggregates to the right (easy eviction fromthe left). The back stack aggregates to the left (easy insertionfrom the right). Here is a visual example of the invariants:
F[0] B[0]F[1] B[1] B[2]0 21 3 4ccd..g ddefg eefg ffg gg hh ihi jhij khijk lhi..l mhi..m nhi..nvalagg 5 76 8 9 10 11F[2] F[3] F[4] B[3] B[4] B[5] B[6]
The notation cd..g is shorthand for c ⊗ d ⊗ . . . ⊗ g . To workcorrectly regardless of commutativity, the aggregation in bothstacks is ordered from the older elements on the left to newerelements on the right. For example, we take care to aggregate f ⊗ g instead of g ⊗ f because f is older than g . n-Order Sliding-Window Aggregation in Worst-Case Constant Time 5 (A)max=0, maxcount=0 FBEinsert 4(B)max=4, maxcount=1 FB44x1 Einsert 5(C)max=5, maxcount=1 FB44x1 55x1 Einsert 3(D)max=5, maxcount=1 FB44x1 55x1 35x1 Einsert 4(E)max=5, maxcount=1 FB44x1 55x1 35x1 45x1 Einsert 0(F)max=5, maxcount=1 FB44x1 55x1 35x1 45x1 05x1 Einsert 4(G)max=5, maxcount=1 FB44x1 55x1 35x1 45x1 05x1 45x1 Einsert 4(H)max=5, maxcount=1 FB44x1 55x1 35x1 45x1 05x1 45x1 45x1 Eevict 4: flip 7, pop(I)max=5, maxcount=1 F55x1 34x3 44x3 04x2 44x2 44x1 BEevict 5(J)max=4, maxcount=3 F34x3 44x3 04x2 44x2 44x1 BEinsert 2(K)max=4, maxcount=3 F34x3 44x3 04x2 44x2 44x1 B22x1 Einsert 6(L)max=6, maxcount=1 F34x3 44x3 04x2 44x2 44x1 B22x1 66x1 Einsert 5(M)max=6, maxcount=1 F34x3 44x3 04x2 44x2 44x1 B22x1 66x1 56x1 Einsert 6(N)max=6, maxcount=2 F34x3 44x3 04x2 44x2 44x1 B22x1 66x1 56x1 66x2 Einsert 1(O)max=6, maxcount=2 F34x3 44x3 04x2 44x2 44x1 B22x1 66x1 56x1 66x2 16x2 Einsert 6(P)max=6, maxcount=3 F34x3 44x3 04x2 44x2 44x1 B22x1 66x1 56x1 66x2 16x2 66x3 Einsert 5(Q)max=6, maxcount=3 F34x3 44x3 04x2 44x2 44x1 B22x1 66x1 56x1 66x2 16x2 66x3 56x3 Eevict 3(R)max=6, maxcount=3 F44x3 04x2 44x2 44x1 B22x1 66x1 56x1 66x2 16x2 66x3 56x3 Eevict 4(S)max=6, maxcount=3 F04x2 44x2 44x1 B22x1 66x1 56x1 66x2 16x2 66x3 56x3 E Fig. 1
Two-Stacks example trace for maxcount aggregation. The nota-tion m × c is shorthand for max= m , maxcount= c . (A)agg=1 FBEinsert a(B)agg=a FBaa Einsert b(C)agg=ab FBaa bab Einsert c(D)agg=abc FBaa bab cabc Einsert d(E)agg=abcd FBaa bab cabc dabcd Einsert e(F)agg=ab..e FBaa bab cabc dabcd eab..e Einsert f(G)agg=ab..f FBaa bab cabc dabcd eab..e fab..f Einsert g(H)agg=ab..g FBaa bab cabc dabcd eab..e fab..f gab..g Eevict a: flip 7, pop(I)agg=bc..g Fbbc..g ccd..g ddefg eefg ffg gg BEevict b(J)agg=cd..g Fccd..g ddefg eefg ffg gg BEinsert h(K)agg=cd..h Fccd..g ddefg eefg ffg gg Bhh Einsert i(L)agg=cd..i Fccd..g ddefg eefg ffg gg Bhh ihi Einsert j(M)agg=cd..j Fccd..g ddefg eefg ffg gg Bhh ihi jhij Einsert k(N)agg=cd..k Fccd..g ddefg eefg ffg gg Bhh ihi jhij khijk Einsert l(O)agg=cd..l Fccd..g ddefg eefg ffg gg Bhh ihi jhij khijk lhi..l Einsert m(P)agg=cd..m Fccd..g ddefg eefg ffg gg Bhh ihi jhij khijk lhi..l mhi..m Einsert n(Q)agg=cd..n Fccd..g ddefg eefg ffg gg Bhh ihi jhij khijk lhi..l mhi..m nhi..n Eevict c(R)agg=de..n Fddefg eefg ffg gg Bhh ihi jhij khijk lhi..l mhi..m nhi..n Eevict d(S)agg=ef..n Feefg ffg gg Bhh ihi jhij khijk lhi..l mhi..m nhi..n E Fig. 2
Two-Stacks example trace for any aggregation. The notation foraggregates omits ⊗ , e.g., bc..g is shorthand for b ⊗ c ⊗ .. ⊗ g . Two-Stacks Algorithm.
Being an algorithm that implementsin-order SWAG, Two-Stacks needs to define the functions query , insert , and evict . But first, we will define two privatehelper functions that retrieve the partial aggregate of theentire front stack F and back stack B , respectively. fun Π ⊗ F if F . isEmpty () return 1 else return F . top (). agg fun Π ⊗ B if B . isEmpty () return 1 else return B . top (). agg Recall that is the identity element of the monoid. Thesehelpers return the correct values in constant time, thanks to the invariants discussed previously. Function query combinesthe results of both helpers, using a single invocation of ⊗ . fun query () return Π ⊗ F ⊗ Π ⊗ B As an example, given the following data structure state, Π ⊗ F is 4 × Π ⊗ B is 6 ×
2, so query returns the maximum 6 × (O)max=6, maxcount=2 F34x3 44x3 04x2 44x2 44x1 B22x1 66x1 56x1 66x2 16x2 E Next, to define insert and evict , we can assume that theinvariants hold before function calls and must guarantee that
Kanat Tangwongsan, Martin Hirzel, and Scott Schneider they hold afterwards. The insert function just pushes onto B ,taking constant time. Assuming the invariants hold for theold top of B , insert guarantees the invariants for the new topof B by setting its partial aggregate agg to Π ⊗ B ⊗ v . fun insert ( v ) B . push ( v , Π ⊗ B ⊗ v ) The following example illustrates insert . (O)max=6, maxcount=2 F34x3 44x3 04x2 44x2 44x1 B22x1 66x1 56x1 66x2 16x2 Einsert 6(P)max=6, maxcount=3 F34x3 44x3 04x2 44x2 44x1 B22x1 66x1 56x1 66x2 16x2 66x3 E Finally, evict pops from F after first ensuring that F isnonempty. fun evict () if F . isEmpty () while not B . isEmpty () F . push ( B . top (). val , B . top (). val ⊗ Π ⊗ F ) B . pop () F . pop () If F is nonempty, then evict is trivial, for example: (Q)max=6, maxcount=3evict 3(R)max=6, maxcount=3 F34x3 44x3 04x2 44x2 44x1 B22x1 66x1 56x1 66x2 16x2 66x3 56x3 EF44x3 04x2 44x2 44x1 B22x1 66x1 56x1 66x2 16x2 66x3 56x3 E On the other hand, if F is empty, then evict must first do a flip . The flip operation pushes all values from B onto F andreverses the direction of the aggregation. In other words, itestablishes that the agg fields satisfy the invariant for F . Afterthe flip, evict simply does a pop as before. (H)max=5, maxcount=1 FB44x1 55x1 35x1 45x1 05x1 45x1 45x1 Eevict 4(I)max=5, maxcount=1 F55x1 34x3 44x3 04x2 44x2 44x1 BE flippopF 55x1 34x3 44x3 04x2 44x2 44x145x1 BE Because of the reversal loop, flip takes O ( n ) time, where n is the current number of elements. Nevertheless, evict onlytakes amortized O ( ) time, as we will see below. Two-Stacks Theorems.
Lemma 1
Two-Stacks maintains the invariants listed above.Proof
The query function does not modify the stacks andthus does not change the invariants. The insert functionmaintains the invariants by correctly setting agg for the newlypushed element. The evict function maintains the invariantsby correctly setting agg for all elements of F during flip. (cid:117)(cid:116) Theorem 2
If the window currently contains v , . . . , v n − ,then query returns v ⊗ . . . ⊗ v n − .Proof Using Lemma 1, query ()= Π ⊗ F ⊗ Π ⊗ B = v ⊗ . . . ⊗ v | F |− ⊗ v | F | ⊗ . . . ⊗ v | F | + | B |− = v ⊗ . . . ⊗ v n − (cid:117)(cid:116) Theorem 3
Two-Stacks requires space to store n partialaggregates. Each call to query and insert invokes ⊗ exactlyone time. Each call to evict invokes ⊗ at most n times andamortized time.Proof The only part of the theorem that is not immediatelyobvious is the amortized complexity of evict . To see this, webill each call to insert two imaginary coins: one for pushingan element onto B and one to go into the savings. Hence,every element in B , as visualized, has a golden coin on top ofit. When flip happens, it invokes ⊗ once for every elementof B , which is completely paid for by spending the coin onthat element. Because billing a constant amount per operationcovers the total cost, each operation is amortized O ( ) . (cid:117)(cid:116) To summarize the workings of Two-Stacks, we will haveanother look at Figs. 1 and 2, which show complete exampletraces. Insertions push to the right of the back stack, visual-ized in green. Evictions pop from the left of the front stack,visualized in blue. Given an empty front stack, evict firstperforms a flip, as show in in Step (H) → (I). The flip keepsvalues unchanged but reverses the associated partial aggre-gates. In this example, there are 7 partial aggregates to flip,coming from the preceding 7 insert operations, which havedeposited 7 coins to the savings to pay for the flip. Two-Stacks Lite improves upon Two-Stacks by reducingthe space complexity from 2 n down to n + val fields of the frontstack and reads only the last agg field of the back stack. Thisidea comes from the Hammer Slide paper by Theodorakis etal. [35]. Another improvement is that instead of physicallymaintaining two separate stacks, Two-Stacks Lite maintains asingle double-ended queue with an internal pointer B to trackthe virtual stack boundary. The time complexity is unchanged,at amortized O ( ) and worst-case O ( n ) invocations of ⊗ perSWAG operation. The data-structure visualizations in thissection are taken from the concrete example traces shown inFigs. 3 and 4. Two-Stacks Lite Data Structure.
The data structure for Two-Stacks Lite comprises a double-ended queue deque of partialaggregates, one additional partial aggregate aggB , and threepointers F , B , and E . Pointer F points to the start of deque , B points to a location between start and end, and E points to n-Order Sliding-Window Aggregation in Worst-Case Constant Time 7 (A)max=0, maxcount=0 FBEinsert 4(B)max=4, maxcount=1 FB4x14x1 Einsert 5(C)max=5, maxcount=1 FB4x15x1 5x1 Einsert 3(D)max=5, maxcount=1 FB4x15x1 5x1 3x1 Einsert 4(E)max=5, maxcount=1 FB4x15x1 5x1 3x1 4x1 Einsert 0(F)max=5, maxcount=1 FB4x15x1 5x1 3x1 4x1 0x1 Einsert 4(G)max=5, maxcount=1 FB4x15x1 5x1 3x1 4x1 0x1 4x1 Einsert 4(H)max=5, maxcount=1 FB4x15x1 5x1 3x1 4x1 0x1 4x1 4x1 Eevict 4: flip 7, pop(I)max=5, maxcount=1 F5x1 4x3 4x3 4x2 4x2 4x1 BEevict 5(J)max=4, maxcount=3 F4x3 4x3 4x2 4x2 4x1 BEinsert 2(K)max=4, maxcount=3 F4x3 4x3 4x2 4x2 4x1 B2x12x1 Einsert 6(L)max=6, maxcount=1 F4x3 4x3 4x2 4x2 4x1 B2x16x1 6x1 Einsert 5(M)max=6, maxcount=1 F4x3 4x3 4x2 4x2 4x1 B2x16x1 6x1 5x1 Einsert 6(N)max=6, maxcount=2 F4x3 4x3 4x2 4x2 4x1 B2x16x2 6x1 5x1 6x1 Einsert 1(O)max=6, maxcount=2 F4x3 4x3 4x2 4x2 4x1 B2x16x2 6x1 5x1 6x1 1x1 Einsert 6(P)max=6, maxcount=3 F4x3 4x3 4x2 4x2 4x1 B2x16x3 6x1 5x1 6x1 1x1 6x1 Einsert 5(Q)max=6, maxcount=3 F4x3 4x3 4x2 4x2 4x1 B2x16x3 6x1 5x1 6x1 1x1 6x1 5x1 Eevict 3(R)max=6, maxcount=3 F4x3 4x2 4x2 4x1 B2x16x3 6x1 5x1 6x1 1x1 6x1 5x1 Eevict 4(S)max=6, maxcount=3 F4x2 4x2 4x1 B2x16x3 6x1 5x1 6x1 1x1 6x1 5x1 E Fig. 3
Two-Stacks Lite example trace for maxcount aggregation. Thenotation m × c is shorthand for max= m , maxcount= c . (A)agg=1 FBEinsert a(B)agg=a FBaa Einsert b(C)agg=ab FBaab b Einsert c(D)agg=abc FBaabc b c Einsert d(E)agg=abcd FBaabcd b c d Einsert e(F)agg=ab..e FBaab..e b c d e Einsert f(G)agg=ab..f FBaab..f b c d e f Einsert g(H)agg=ab..g FBaab..g b c d e f g Eevict a: flip 7, pop(I)agg=bc..g Fbc..g cd..g defg efg fg g BEevict b(J)agg=cd..g Fcd..g defg efg fg g BEinsert h(K)agg=cd..h Fcd..g defg efg fg g Bhh Einsert i(L)agg=cd..i Fcd..g defg efg fg g Bhhi i Einsert j(M)agg=cd..j Fcd..g defg efg fg g Bhhij i j Einsert k(N)agg=cd..k Fcd..g defg efg fg g Bhhijk i j k Einsert l(O)agg=cd..l Fcd..g defg efg fg g Bhhi..l i j k l Einsert m(P)agg=cd..m Fcd..g defg efg fg g Bhhi..m i j k l m Einsert n(Q)agg=cd..n Fcd..g defg efg fg g Bhhi..n i j k l m n Eevict c(R)agg=de..n Fdefg efg fg g Bhhi..n i j k l m n Eevict d(S)agg=ef..n Fefg fg g Bhhi..n i j k l m n E Fig. 4
Two-Stacks Lite example trace for any aggregation. The notationfor aggregates omits ⊗ , e.g., bc..f is shorthand for b ⊗ c ⊗ .. ⊗ f . the end. Here is an example with a max-count aggregation(the golden “coins” visualizing the savings serve the samepurpose as in Two-Stacks): F B6x3 EdequeaggB l B l F Definition 4 (Pointers)
In this paper, a pointer is an iter-ator into a resizable double-ended queue that supports thefollowing basic data structure operations: – dereference and read or write contents: v ← * p , * p ← v – pointer comparison: p = q , p (cid:54) = q – pointer increment and decrement: p + p − – pointer assignment: p ← q Our algorithms only use the above-listed pointer operations.These pointer operations can be implemented in worst-case O ( ) time over a variable-sized double-ended queue by im-plementing that queue using chunked arrays [31]. For statinginvariants, we will also use a few additional pointer opera-tions such as p + i , p − q , or p < q . These additional oper-ations do not need to be O ( ) since they are used only byinvariants and not directly by our algorithms. (cid:117)(cid:116) Kanat Tangwongsan, Martin Hirzel, and Scott Schneider
Though physically the data structure uses a single deque, thepointers demarcate two virtual sublists, l F and l B . Two-Stacks Lite Invariants.
Let v , . . . , v n − be the currentwindow contents from the oldest to the youngest. Then, eachdeque element in the front sublist l F ( F ≤ p < B ) stores thepartial aggregate starting from the corresponding v i up to theelement before B . Each deque element in the back sublist l B ( B ≤ p < E ) stores the corresponding v i . In addition, aggB stores the partial aggregate of all elements in l B . Formally: ∀ i ∈ . . . B − F − * ( F + i ) = v i ⊗ . . . ⊗ v B − F − and ∀ i ∈ B − F . . . E − F − * ( F + i ) = v i and aggB = * B ⊗ . . . ⊗ * ( E − ) As an example, assume that the current window contains thevalues v = c , v = d , v = e , etc. up to v = m and v = n .Then, if there are five elements in l F and seven in l B , the datastructure looks as follows: Fcd..g defg efg fg g Bhhi..n i j k l m n EdequeaggB 0 21 3 4 5 76 8 9 10 11 l B l F The notation cd..g is shorthand for c ⊗ d ⊗ . . . ⊗ g . Two-Stacks Lite Algorithm.
The private helper function Π ⊗ F retrieves the partial aggregate of the entire front sublist l F ,where is the identity element of the monoid. fun Π ⊗ F if ( F = B ) return 1 else return * F Function query obtains the partial aggregates of l F (by call-ing Π ⊗ F ) and of l B (by reading aggB ) and combines them inconstant time. fun query () return Π ⊗ F ⊗ aggB As an example, given the following data structure state, Π ⊗ F is4 × aggB is 6 ×
2, so query returns the maximum 6 × (O)max=6, maxcount=2 F B6x2 E4x3 4x3 4x2 4x2 4x1 2x1 6x1 5x1 6x1 1x1 Both insert and evict can assume that the invariants holdbefore they are called and must guarantee that they holdafterwards. Function insert ( v ) accomplishes this by pushing v and updating aggB accordingly. fun insert ( v ) deque . pushBack ( v ) E ← E + 1 aggB ← aggB ⊗ v The following example illustrates insert . (O)max=6, maxcount=2 F B6x2 Einsert 6(P)max=6, maxcount=3 F B6x3 E4x3 4x3 4x2 4x2 4x1 2x1 6x1 5x1 6x1 1x14x3 4x3 4x2 4x2 4x1 2x1 6x1 5x1 6x1 1x1 6x1 Finally, evict pops from the front, after first ensuring thatthe front sublist l F is nonempty. fun evict () if F = B and B (cid:54) = E I ← E - 1 while I (cid:54) = F I ← I - 1 * I ← * I ⊗ *( I + 1) B ← E aggB ← F ← F + 1 deque . popFront () If l F is nonempty, then evict is trivial, for example: (Q)max=6, maxcount=3 F B6x3 Eevict 3(R)max=6, maxcount=3 F B6x3 E4x3 4x3 4x2 4x2 4x1 2x1 6x1 5x1 6x1 1x1 6x1 5x14x3 4x2 4x2 4x1 2x1 6x1 5x1 6x1 1x1 6x1 5x1 If l F is empty, then evict does a flip . Lines 11–14 rewritethe contents of deque in-place to contain partial aggregatesfrom the corresponding element to the end. Line 15 updatespointer B to indicate that the front sublist l F now occupies theentire data structure and the back sublist l B is empty. Then,Line 16 resets aggB to the monoid identity element. Here is avisualization of flip followed by pop: (H)max=5, maxcount=1 FB5 Eevict 4(I)max=5, maxcount=1 F BE flippopF4x1 5x1 4x3 4x3 4x2 4x2 4x1 BE4x1 5x1 3x1 4x1 0x1 4x1 4x15x1 4x3 4x3 4x2 4x2 4x1 The loop takes time O ( n ) but can be amortized over thepreceding n insertions, where n is the current window size. Two-Stacks Lite Theorems.
Lemma 5
The Two-Stacks Lite algorithm maintains the Two-Stacks Lite invariants.Proof
By inspection, function query preserves the data andthus the invariants. Function insert maintains the invariantsby pushing v and updating aggB . Function evict optionallydoes a flip, which reestablishes the invariants, then alwaysdoes a pop, which also maintains the invariants. (cid:117)(cid:116) Theorem 6
If the window currently contains v , . . . , v n − ,then query returns v ⊗ . . . ⊗ v n − .Proof Using Lemma 5, query ()= Π ⊗ F ⊗ aggB = v ⊗ . . . ⊗ v B − F − ⊗ v B − F ⊗ . . . ⊗ v E − F − = v ⊗ . . . ⊗ v n − (cid:117)(cid:116) Theorem 7
Two-Stacks Lite requires space to store n + partial aggregates. Each call to query and insert invokes ⊗ exactly one time. Each call to evict invokes ⊗ at most ntimes and amortized one time. n-Order Sliding-Window Aggregation in Worst-Case Constant Time 9 Proof
Most of the theorem is obvious. To prove the amor-tized complexity of evict , bill each call to insert two imag-inary coins for pushing an element onto the back and forthe savings (visualized as small golden “coin” above the el-ements). Hence, every element in l B has a golden coin ontop of it. When flip happens, it invokes ⊗ once for everyelement of l B , which is completely paid for by spending thecoin on that element. (cid:117)(cid:116) DABA is a low-latency algorithm for in-order sliding windowaggregation (SWAG). When the window has size n , DABArequires space to store 2 n partial aggregates and supportseach SWAG operation using worst-case O ( ) invocationsof ⊗ . For a brief explanation of the name, DABA stands forDe-Amortized Banker’s Aggregator: Amortization looks atthe average cost of an operation over a long period of time.The banker’s method conceptualizes amortization as mov-ing imaginary coins between the algorithm and a fictitiousbank. Deamortization is a method that turns the average-casebehavior into the worst-case behavior, usually by carefullyspreading out expensive operations. In this spirit, notice thatthe expensive operation in the Two-Stacks algorithm is theloop for reversing the direction of aggregation during flip,paid for by imaginary coins deposited on preceding inser-tions. Whereas Two-Stacks does the flip late when the frontstack becomes empty, DABA does the flip earlier, when thefront and back stack reach the same length. Furthermore,instead of doing a reversal loop at the time of the flip, DABAspreads out the steps for reversing the direction of aggrega-tion. The text of this section embeds several data-structurevisualizations taken from concrete example traces shown inFigs. 5 and 6. DABA Data Structure.
The DABA data structure comprisesa double-ended queue, deque , and six pointers, F , L , R , A , B ,and E , into that queue. Each queue element is a struct withtwo partial aggregates: val (top row) and agg (bottom row).The basic pointer operations are the same as in Definition 4and are easy to implement in O ( ) time. The pointers arealways ordered as follows: F ≤ L ≤ R ≤ A ≤ B ≤ E Here is an example with a max-count aggregation:
F L R A B Evalagg l L l R l A l B l F The val fields store the window contents, with the i th old-est value in FIFO order stored at v i = ∗ ( F + i ) . val . The agg fields store partial aggregations over subranges of the win-dow. Conceptually, each pointer p corresponds to a sublist l p . For example, pointer F corresponds to sublist l F . Each sublistis either aggregated to the left or to the right. The directionis carefully chosen to enable the DABA operations. The left-most portion of the front list l F is aggregated to the left tofacilitate eviction. The back list l B is aggregated to the rightto facilitate insertion. The inner sublists l L , l R , and l A aredesigned to facilitate incremental reversal. Incremental re-versal happens by adjusting the pointers demarcating sublistboundaries one step at a time. When a pointer moves, a dequeelement changes membership from one sublist to another andits agg field may need to be updated accordingly. DABA Invariants.
DABA maintains three groups of invari-ants: values invariants, partial aggregate invariants, and sizeinvariants. DABA’s values invariants specify that the val field of each element stores a singleton partial aggregate v i obtained by lifting the corresponding single stream element. ∀ i ∈ . . . E − F − ∗ ( F + i ) . val = v i DABA’s partial aggregate invariants specify the contents ofthe agg fields before and after each SWAG operation, basedon sublists. In the visualizations, blue indicates aggregationto the right and green indicates aggregation to the left. Inthe left-most portion of sublist l F (the front sublist, in darkblue), each agg field holds an aggregate starting from thatelement to the right end of l F . In sublist l L (the left sublist,in light blue), each agg field holds an aggregate starting fromthat element to the right end of l L . In sublist l R (the rightsublist, in light green), each agg field holds an aggregatestarting from the left end of l R to that element. In sublist l A (the accumulator sublist, in dark blue), each agg field holdsan aggregate starting from that element to the right end of l A ,which coincides with the right end of l F . Finally, in sublist l B (the back sublist, in dark green), each agg field holds anaggregate starting from the left end of l B to that element.Formally: ∀ i ∈ . . . L − F − ∗ ( F + i ) . agg = v i ⊗ . . . ⊗ v B − F − and ∀ i ∈ L − F . . . R − F − ∗ ( F + i ) . agg = v i ⊗ . . . ⊗ v R − F − and ∀ i ∈ R − F . . . A − F − ∗ ( F + i ) . agg = v R − F ⊗ . . . ⊗ v i and ∀ i ∈ A − F . . . B − F − ∗ ( F + i ) . agg = v i ⊗ . . . ⊗ v B − F − and ∀ i ∈ B − F . . . E − F − ∗ ( F + i ) . agg = v B − F ⊗ . . . ⊗ v i Here is a visual example of the invariants:
Fccd..l dde..l eef..l L ffg gg Rhh ihi Ajjkl kkl ll Bmm nmn0 21 3 4 5 76 8 9 10 11 E l L l R l A l B l F valagg The notation cd..l is shorthand for c ⊗ d ⊗ . . . ⊗ l . To workcorrectly irrespective of whether the monoid is commutativeor not, in all sublists, the operands of ⊗ are always orderedfrom older on the left to newer on the right.DABA’s size invariants specify constraints on the sizesof sublists. Given a pointer p , we use the notation | l p | toindicate the size of sublist l p . For example, the size of sublist (A)max=0, maxcount=0 FLRABEinsert 4: singleton(B)max=4, maxcount=1 F44x1 LRABEinsert 5: flip, shrink(C)max=5, maxcount=1 F45x1 LRA55x1 BEinsert 3: shift(D)max=5, maxcount=1 F45x1 55x1 LRAB33x1 Einsert 4: flip, shrink(E)max=5, maxcount=1 F45x1 L55x1 R33x1 A44x1 BEinsert 0: shrink(F)max=5, maxcount=1 F45x1 55x1 LRA34x1 44x1 B00x1 Einsert 4: shift(G)max=5, maxcount=1 F45x1 55x1 34x1 LRA44x1 B00x1 44x1 Einsert 4: shift(H)max=5, maxcount=1 F45x1 55x1 34x1 44x1 LRAB00x1 44x1 44x2 Eevict 4: flip, shrink(I)max=5, maxcount=1 F55x1 L34x1 44x1 R00x1 44x1 A44x1 BEevict 5: shrink(J)max=4, maxcount=3 F34x3 L44x1 R00x1 A44x2 44x1 BEinsert 2: shrink(K)max=4, maxcount=3 F34x3 44x3 LRA04x2 44x2 44x1 B22x1 Einsert 6: shift(L)max=6, maxcount=1 F34x3 44x3 04x2 LRA44x2 44x1 B22x1 66x1 Einsert 5: shift(M)max=6, maxcount=1 F34x3 44x3 04x2 44x2 LRA44x1 B22x1 66x1 56x1 Einsert 6: shift(N)max=6, maxcount=2 F34x3 44x3 04x2 44x2 44x1 LRAB22x1 66x1 56x1 66x2 Einsert 1: flip, shrink(O)max=6, maxcount=2 F36x2 L44x3 04x2 44x2 44x1 R22x1 66x1 56x1 66x2 A11x1 BEinsert 6: shrink(P)max=6, maxcount=3 F36x2 46x2 L04x2 44x2 44x1 R22x1 66x1 56x1 A66x1 11x1 B66x1 Einsert 5: shrink(Q)max=6, maxcount=3 F36x2 46x2 06x2 L44x2 44x1 R22x1 66x1 A56x1 66x1 11x1 B66x1 56x1 Eevict 3: shrink(R)max=6, maxcount=3 F46x2 06x2 46x2 L44x1 R22x1 A66x2 56x1 66x1 11x1 B66x1 56x1 Eevict 4: shrink(S)max=6, maxcount=3 F06x2 46x2 46x2 LRA26x2 66x2 56x1 66x1 11x1 B66x1 56x1 E Fig. 5
DABA example trace for maxcount aggregation. The notation m × c is shorthand for max= m , maxcount= c . (A)agg=1 FLRABEinsert a: singleton(B)agg=a Faa LRABEinsert b: flip, shrink(C)agg=ab Faab LRAbb BEinsert c: shift(D)agg=abc Faab bb LRABcc Einsert d: flip, shrink(E)agg=abcd Faabcd Lbb Rcc Add BEinsert e: shrink(F)agg=ab..e Faabcd bbcd LRAccd dd Bee Einsert f: shift(G)agg=ab..f Faabcd bbcd ccd LRAdd Bee fef Einsert g: shift(H)agg=ab..g Faabcd bbcd ccd dd LRABee fef gefg Eevict a: flip, shrink(I)agg=bc..g Fbbc..g Lccd dd Ree fef Agg BEevict b: shrink(J)agg=cd..g Fccd..g Ldd Ree Affg gg BEinsert h: shrink(K)agg=cd..h Fccd..g ddefg LRAeefg ffg gg Bhh Einsert i: shift(L)agg=cd..i Fccd..g ddefg eefg LRAffg gg Bhh ihi Einsert j: shift(M)agg=cd..j Fccd..g ddefg eefg ffg LRAgg Bhh ihi jhij Einsert k: shift(N)agg=cd..k Fccd..g ddefg eefg ffg gg LRABhh ihi jhij khijk Einsert l: flip, shrink(O)agg=cd..l Fccd..l Lddefg eefg ffg gg Rhh ihi jhij khijk All BEinsert m: shrink(P)agg=cd..m Fccd..l dde..l Leefg ffg gg Rhh ihi jhij Akkl ll Bmm Einsert n: shrink(Q)agg=cd..n Fccd..l dde..l eef..l L ffg gg Rhh ihi Ajjkl kkl ll Bmm nmn Eevict c: shrink(R)agg=de..n Fdde..l eef..l ffg..l Lgg Rhh Aiijkl jjkl kkl ll Bmm nmn Eevict d: shrink(S)agg=ef..n Feef..l ffg..l ggh..l LRAhhi..l iijkl jjkl kkl ll Bmm nmn E Fig. 6
DABA example trace for any aggregation. The notation foraggregates omits ⊗ , e.g., bc..f is shorthand for b ⊗ c ⊗ .. ⊗ f . l F is | l F | = B − F and the size of sublist l L is | l L | = R − L .Formally, the size invariants are (cid:16) | l F | = ∧ | l B | = (cid:17) ∨ (cid:16) | l L | + | l R | + | l A | + = | l F | − | l B | ∧ | l L | = | l R | (cid:17) This says that the window is either empty ( | l F | = | l B | =
0) or the following two conditions hold:-
First , | l L | + | l R | + | l A | + = | l F | − | l B | . The size of thefront list l F exceeds the size of the back list l B by thetotal size of the sublists l L , l R , and l A plus one. The sublists l L , l R , and l A are used for incremental reversal, and thealgorithm, shown below, shrinks their total size by one on each insertion or eviction. When the sublists l L , l R , and l A are empty, the algorithm can make just one more insertionor eviction before l F and l B reach the same size. At thatpoint, the algorithm does a flip , which relabels l F and l B into l L and l R , respectively.- Second , | l L | = | l R | . After each flip , l L and l R start out withthe same size and then shrink at the same pace.Below, we will see explanations for how the algorithm main-tains these invariants, using color-coding to recognize corre-sponding subequations. n-Order Sliding-Window Aggregation in Worst-Case Constant Time 11 DABA Algorithm.
For each sublist l p , a private helper func-tion Π ⊗ p retrieves the corresponding partial aggregate or re-turns the monoid’s identity element if the sublist is empty.Note that for a given sublist, we retrieve the partial aggregatein the left-most element if that sublist aggregates to the right,and the partial aggregate in the right-most element if thatsublist aggregates to the left. fun Π ⊗ F if ( F = B ) return 1 else return * F . agg fun Π ⊗ B if ( B = E ) return 1 else return *( E -1). agg fun Π ⊗ L if ( L = R ) return 1 else return * L . agg fun Π ⊗ R if ( R = A ) return 1 else return *( A -1). agg fun Π ⊗ A if ( A = B ) return 1 else return * A . agg These helpers return the correct values in constant time,thanks to the invariants defined previously. Function query combines the aggregate of l F and l B , taking only a singleinvocation of ⊗ . fun query () return Π ⊗ F ⊗ Π ⊗ B Function insert pushes a value v with corresponding partialaggregate to the back of the deque, then calls a function fixup ,defined below, for doing one step of incremental reversal. fun insert ( v ) deque . pushBack ( v , Π ⊗ B ⊗ v ) E ← E + 1 fixup () In our running maxcount example, if Π ⊗ B = v =
4, thenthe newly pushed deque element has val = agg = (F)max=5, maxcount=1 F45x1 55x1 LRA34x1 44x1 B00x1 Einsert 4(G) F 45x1 55x1 34x1LRA 44x1 B00x1 44x1 Epushfixup… Similarly, evict pops an element from the front of the deque,then calls fixup for one step of incremental reversal. fun evict () F ← F + 1 deque . popFront () fixup () In our running maxcount example, the following pictureillustrates eviction: (I)max=5, maxcount=1 F55x1 L34x1 44x1 R00x1 44x1 A44x1 BEevict 5(J) popFL 34x1 44x1 R00x1 44x1 A44x1 BEfixup…
The fixup function is responsible for restoring the invariants.Recall that we can assume that the following size invariantshold before each call to insert or evict : (cid:16) | l F | = ∧ | l B | = (cid:17) ∨ (cid:16) | l L | + | l R | + | l A | + = | l F | − | l B | ∧ | l L | = | l R | (cid:17) Function insert grows l B by one element and function evict shrinks l F by one element. The impact on the invariants is the same in both cases: they both decrease the difference | l F | − | l B | by one. Thanks to the extra + l F , nei-ther insert nor evict affects the inner sublists l L , l R , and l A .This means that upon entry to fixup , the following is true: (cid:16) | l F | = ∧ | l B | = (cid:17) ∨ (cid:16) | l L | + | l R | + | l A | = | l F | − | l B | ∧ | l L | = | l R | (cid:17) Using the above as a precondition, the postcondition of fixup is to reestablish the original size invariants. The fixup func-tion does this via four cases singleton , flip , shift , and shrink . fun fixup () if F = B B ← E , A ← E , R ← E , L ← E else if L = B L ← F , A ← E , B ← E if L = R A ← A + 1, R ← R + 1, L ← L + 1 else * L . agg ← Π ⊗ L ⊗ Π ⊗ R ⊗ Π ⊗ A L ← L + 1 *( A -1). agg ← *( A -1). val ⊗ Π ⊗ A A ← A - 1 The singleton case happens when | l F | =
0. Given theprecondition, this can only hold when | l B | =
1. Then, withouthaving to modify the deque, the pointer assignments B ← E , A ← E , R ← E , L ← E change the size of l F to | l F | = l L , l R , l A , and l B empty, as illustrated below. (A)max=0, maxcount=0 FLRABEinsert 4(B)max=4, maxcount=1 F44x1 LRABEpushfixup (singleton case)44x1 EFLRAB The singleton case code thus establishes (cid:16) | l L | + | l R | + | l A | + = | l F | − | l B | = ∧ | l L | = | l R | = (cid:17) which implies the original size invariants.The flip case happens when | l F | > | l L | + | l R | + | l A | =
0. Togetherwith the precondition, this implies that (cid:16) | l L | + | l R | + | l A | = ∧ | l F | = | l B | ∧ | l L | = ∧ | l R | = ∧ | l A | = (cid:17) Then, the pointer assignments L ← F , A ← E , B ← E turn the old outer sublists l F and l B into the new inner sublists l L and l R , respectively. No updates to agg fields are requiredbecause the corresponding sublists already have the correctaggregation direction. (H)max=5, maxcount=1 F45x1 55x1 34x1 44x1 LRAB00x1 44x1 44x2 Eevict 4(I)max=5, maxcount=1 pop flipF55x1 34x1 44x1 LRAB00x1 44x1 44x2 EFL55x1 34x1 44x1 R00x1 44x1 44x2 ABEshrink… After flip, the following holds: (cid:16) | l L | + | l R | + | l A | = | l F | ∧ | l B | = ∧ | l L | = | l R | > (cid:17) At this point, we still need to execute the shrink case to repairthe size invariants.
The shift case happens when | l F | > | l L | =
0. To-gether with the precondition, this implies that (cid:16) | l L | + | l R | + | l A | = | l F | − | l B | ∧ | l L | = ∧ | l R | = ∧ | l A | > (cid:17) Then, the pointer assignments A ← A + 1, R ← R + 1, L ← L + 1 increment the pointers separating the left-most portion of l F from l A by one. No updates to agg fields are required becauseboth the left-most portion of l F and l A are governed by thesame aggs invariants. (F)max=5, maxcount=1 F LRA B E F LRA B Epushshiftinsert 4(G)max=5, maxcount=1 F45x1 55x1 34x1 LRA44x1 B00x1 44x1 E45x1 55x1 34x1 44x1 00x1 45x1 55x1 34x1 44x1 00x1 44x1 After shift, the following holds: (cid:16) | l L | + | l R | + | l A | + = | l F | − | l B | ∧ | l L | = ∧ | l R | = (cid:17) which implies the original size invariants.The shrink case happens when | l F | > | l L | >
0. Thereare two scenarios: with or without a flip from the same fixup .Either way, shrink starts with the following precondition: (cid:16) | l L | + | l R | + | l A | = | l F | − | l B | ∧ | l L | = | l R | ∧ | l L | > (cid:17) The shrink case is the only part of fixup that modifies notjust pointers but also agg fields. It reduces the sizes of both l L and l R by one each. The top element of l L becomes part ofthe left-most portion of l F , so its agg field must be updatedto v L − F ⊗ . . . ⊗ v B − F = Π ⊗ L ⊗ Π ⊗ R ⊗ Π ⊗ A . The top element of l R becomes part of the accumulator sublist l A , so its agg fieldmust be updated to v A − F − ⊗ . . . ⊗ v B − F = v A − F − ⊗ Π ⊗ A . * L . agg ← Π ⊗ L ⊗ Π ⊗ R ⊗ Π ⊗ A L ← L + 1*( A -1). agg ← *( A -1). val ⊗ Π ⊗ A A ← A - 1 The following is a typical example of shrink, reducing thesizes of l L and l R from 2 to 1. (I)max=5, maxcount=1 F L R A BEevict 5 popFL R A BEshrink(J)max=4, maxcount=3 F34x3 L44x1 R00x1 A44x2 44x1 BE55x1 34x1 44x1 00x1 44x1 44x1 34x1 44x1 00x1 44x1 44x1 The following is an example of flip followed by shrink. Afterthe flip, l L and l R both have size 3. After the shrink, l L and l R both have size 2. (H)max=5, maxcount=1 F LRAB Eevict 4(I)max=5, maxcount=1 F55x1 L34x1 44x1 R00x1 44x1 A44x1 BEpop F LRABFL R ABEshrink E45x1 55x1 34x1 44x1 00x1 44x1 44x2 55x1 34x1 44x1 00x1 44x1 44x2 flip55x1 34x1 44x1 00x1 44x1 44x2 Given the precondition of shrink, it establishes the followingpostcondition: (cid:16) | l L | + | l R | + | l A | + = | l F | − | l B | ∧ | l L | = | l R | (cid:17) which implies the original size invariants. DABA Intuitive View.
Now that we have seen all the smallsteps that make up DABA, let us look at their interplay to help understand the algorithm more holistically. Figs. 5 and 6show two variants of the same example trace, differing onlyin their aggregation monoid. The sequence of cases startingat (D) comprises [ flip shrink shrink shift shift ] . A similarpattern starts at (H), comprising [ flip shrink shrink shrinkshift shift shift ] . More generally, each flip is followed byan equal number of shrink m and shift m cases. For instance,the sequence starting at (B) is [ flip shrink shift ] , which is aspecial case where m =
1. Visually, during the shrink m phaseof the algorithm, the light blue and light green sublists narrowto a point, looking like an upside-down step pyramid. Thiscorresponds to the incremental reversal of l R , which of coursewas l B before the flip. During the shrink m phase, pointer R does not change. Afterwards, during the shift m phase, the LRA pointers (which are now all the same) shift to the rightone element at a time until they hit pointer B . When theyreach B , the next insert or evict would cause l F and l B to havethe same length, triggering the next flip and thus the nextcycle. Within each cycle, pointer B always stays the same; itonly moves when a flip happens. DABA Theorems.
Lemma 8
DABA maintains the invariants listed above, in-cluding the values invariants, the partial aggregate invari-ants, and the size invariants.Proof
The query function does not modify the data structureand thus does not change the invariants. Functions insert and evict both establish the same precondition for fixup , asstated above. Finally, given that precondition, all cases of fixup reestablish the original invariants as a postcondition,as shown above. (cid:117)(cid:116)
Theorem 9
If the window currently contains v , . . . , v n − ,then query returns v ⊗ . . . ⊗ v n − .Proof Using Lemma 8, query ()= Π ⊗ F ⊗ Π ⊗ B = v ⊗ . . . ⊗ v B − F − ⊗ v B − F ⊗ . . . ⊗ v E − F − = v ⊗ . . . ⊗ v n − (cid:117)(cid:116) Theorem 10
DABA requires space to store n partial ag-gregates. DABA invokes ⊗ at most one time per query , fourtimes per insert , and three times per evict . Furthermore, fornonempty windows, DABA invokes ⊗ on average 2.5 timesper insert and 1.5 times per evict .Proof The worst-case numbers can be seen directly fromthe code and by noting that the algorithm contains no loopsor recursion. To see the average-case numbers, consider thesequence of fixup cases from a flip to the next. Immediatelyfollowing flip, l R is nonempty and l A is empty. As long as n-Order Sliding-Window Aggregation in Worst-Case Constant Time 13 l R is nonempty, each subsequent insert or evict executes ashrink, invoking ⊗ three times. When l R becomes empty, l A has exactly the size that l R had at the previous flip. As longas l A is nonempty, each subsequent insert or evict executesa shift, without invoking ⊗ . The next flip happens when l A isempty. That means that there was an equal number of shrinksteps as shift steps, and thus, an equal number of fixup callswith three invocations of ⊗ and with zero invocations of ⊗ .This averages out to 1.5 ⊗ -invocations per fixup , and thus,2.5 ⊗ -invocations per insert and 1.5 per evict . (cid:117)(cid:116) A corollary of Theorem 10 is that DABA implements allSWAG operations with worst-case O ( ) invocations of ⊗ .Unlike previous algorithms in the paper, DABA involves nocostly steps that would require an amortization argument. DABA Lite improves upon the space complexity of DABAwithout increasing its running time, storing only n + n in DABA. It saves space byexploiting the insights that the DABA algorithm reads noneof the val fields of the sublists that are aggregated to the leftand only the last agg fields of sublists that are aggregatedto the right. The time complexity is still worst-case O ( ) invocations of ⊗ per SWAG operation. The data-structurevisualizations in this section are all taken from concrete ex-ample traces shown in Figs. 7 and 8. DABA Lite Data Structure.
The DABA Lite data structurecomprises a double-ended queue deque of partial aggregates,two additional partial aggregates aggRA and aggB , and sixpointers F , L , R , A , B , and E into the queue, see Definition 4.The pointers are always ordered as follows: F ≤ L ≤ R ≤ A ≤ B ≤ E Here is an example with a max-count aggregation:
F L R A B Edeque aggRA l L l R l A l B l F aggB6x2 6x2 6x2 4x2 4x1 2x16x2 6x1 6x1 6x1 1x1 6x16x1 5x1 Conceptually, each pointer p corresponds to a sublist l p . Bluesublists are aggregated to the left to facilitate eviction, witheach element containing the partial aggregate starting fromthat element to the right end of its sublist. The elementsof green sublists simply contain the corresponding windowelements. The aggregates for the green sublists are includedin aggRA and aggB . DABA Lite Invariants.
The contents invariants specify thecontents of the deque and of aggRA and aggR . Let v , . . . , v n − be the current window contents. In the leftmost portion of sublist l F (the front sublist, in dark blue), each element holdsan aggregate starting from that element to the right end of l F .In sublist l L (the left sublist, in light blue), each element holdsan aggregate starting from that element to the right end of l L .In sublist l R (the right sublist, in light green), each elementholds the corresponding window element, and if L (cid:54) = R then aggRA holds the combined partial aggregate of l R and l A . Insublist l A (the accumulator sublist, in dark blue), each elementholds an aggregate starting from that element to the right endof l A . In sublist l B (the back sublist, in dark green), eachelement holds the corresponding window element, and aggB holds the aggregate of l B . Formally: ∀ i ∈ . . . L − F − * ( F + i ) = v i ⊗ . . . ⊗ v B − F − and ∀ i ∈ L − F . . . R − F − * ( F + i ) = v i ⊗ . . . ⊗ v R − F − and ∀ i ∈ R − F . . . A − F − * ( F + i ) = v i and ( L = R ) ∨ ( aggRA = v R − F ⊗ . . . ⊗ v B − F − ) and ∀ i ∈ A − F . . . B − F − * ( F + i ) = v i ⊗ . . . ⊗ v B − F − and ∀ i ∈ B − F . . . E − F − * ( F + i ) = v i and aggB = v B − F ⊗ . . . ⊗ v E − F − Here is a visual example of the invariants for a window withcontents v = c , v = d , . . . , v = m , v = n : cd..l de..l ef..l fg g hhi..l i jkl kl l mmn nF L R A B Edeque aggRA l L l R l A l B l F aggB0 21 3 4 5 76 8 9 10 11 The notation cd..l is shorthand for c ⊗ d ⊗ . . . ⊗ l .The size invariants specify constraints on the sizes ofsublists. The size invariants of DABA Lite are the same asthose of DABA: (cid:16) | l F | = ∧ | l B | = (cid:17) ∨ (cid:16) | l L | + | l R | + | l A | + = | l F | − | l B | ∧ | l L | = | l R | (cid:17) DABA Lite Algorithm.
For each sublist l p that is aggregatedto the left, a private helper function Π ⊗ p retrieves the corre-sponding partial aggregate or returns the monoid’s identityelement if the sublist is empty. fun Π ⊗ F if ( F = B ) return 1 else return * F fun Π ⊗ L if ( L = R ) return 1 else return * L fun Π ⊗ A if ( A = B ) return 1 else return * A These helpers return the correct values in constant timethanks to the invariants defined previously. Function query combines the aggregate of l F and l B , taking only a singleinvocation of ⊗ . fun query () return Π ⊗ F ⊗ aggB Function insert pushes a value v onto l B and updates aggB accordingly, then calls a function fixup , defined below, fordoing one step of incremental reversal. (A)max=0, maxcount=0 FLRABEinsert 4: singleton(B)max=4, maxcount=1 F4x1 LRABEinsert 5: flip, shrink(C)max=5, maxcount=1 F5x1 LRA5x1 BEinsert 3: shift(D)max=5, maxcount=1 F5x1 5x1 LRAB3x13x1 Einsert 4: flip, shrink(E)max=5, maxcount=1 F5x1 L5x1 R3x14x1 A4x1 BEinsert 0: shrink(F)max=5, maxcount=1 F5x1 5x1 LRA4x1 4x1 B0x10x1 Einsert 4: shift(G)max=5, maxcount=1 F5x1 5x1 4x1 LRA4x1 B0x14x1 4x1 Einsert 4: shift(H)max=5, maxcount=1 F5x1 5x1 4x1 4x1 LRAB0x14x2 4x1 4x1 Eevict 4: flip, shrink(I)max=5, maxcount=1 F5x1 L4x1 4x1 R0x14x2 4x1 A4x1 BEevict 5: shrink(J)max=4, maxcount=3 F4x3 L4x1 R0x14x2 A4x2 4x1 BEinsert 2: shrink(K)max=4, maxcount=3 F4x3 4x3 LRA4x2 4x2 4x1 B2x12x1 Einsert 6: shift(L)max=6, maxcount=1 F4x3 4x3 4x2 LRA4x2 4x1 B2x16x1 6x1 Einsert 5: shift(M)max=6, maxcount=1 F4x3 4x3 4x2 4x2 LRA4x1 B2x16x1 6x1 5x1 Einsert 6: shift(N)max=6, maxcount=2 F4x3 4x3 4x2 4x2 4x1 LRAB2x16x2 6x1 5x1 6x1 Einsert 1: flip, shrink(O)max=6, maxcount=2 F6x2 L4x3 4x2 4x2 4x1 R2x16x2 6x1 5x1 6x1 A1x1 BEinsert 6: shrink(P)max=6, maxcount=3 F6x2 6x2 L4x2 4x2 4x1 R2x16x2 6x1 5x1 A6x1 1x1 B6x16x1 Einsert 5: shrink(Q)max=6, maxcount=3 F6x2 6x2 6x2 L4x2 4x1 R2x16x2 6x1 A6x1 6x1 1x1 B6x16x1 5x1 Eevict 3: shrink(R)max=6, maxcount=3 F6x2 6x2 6x2 L4x1 R2x16x2 A6x2 6x1 6x1 1x1 B6x16x1 5x1 Eevict 4: shrink(S)max=6, maxcount=3 F6x2 6x2 6x2 LRA6x2 6x2 6x1 6x1 1x1 B6x16x1 5x1 E Fig. 7
DABA Lite example trace for maxcount aggregation. The nota-tion m × c is shorthand for max= m , maxcount= c . (A)agg=1 FLRABEinsert a: singleton(B)agg=a Fa LRABEinsert b: flip, shrink(C)agg=ab Fab LRAb BEinsert c: shift(D)agg=abc Fab b LRABcc Einsert d: flip, shrink(E)agg=abcd Fabcd Lb Rccd Ad BEinsert e: shrink(F)agg=ab..e Fabcd bcd LRAcd d Bee Einsert f: shift(G)agg=ab..f Fabcd bcd cd LRAd Beef f Einsert g: shift(H)agg=ab..g Fabcd bcd cd d LRABeefg f g Eevict a: flip, shrink(I)agg=bc..g Fbc..g Lcd d Reefg f Ag BEevict b: shrink(J)agg=cd..g Fcd..g Ld Reefg Afg g BEinsert h: shrink(K)agg=cd..h Fcd..g defg LRAefg fg g Bhh Einsert i: shift(L)agg=cd..i Fcd..g defg efg LRAfg g Bhhi i Einsert j: shift(M)agg=cd..j Fcd..g defg efg fg LRAg Bhhij i j Einsert k: shift(N)agg=cd..k Fcd..g defg efg fg g LRABhhijk i j k Einsert l: flip, shrink(O)agg=cd..l Fcd..l Ldefg efg fg g Rhhi..l i j k Al BEinsert m: shrink(P)agg=cd..m Fcd..l de..l Lefg fg g Rhhi..l i j Akl l Bmm Einsert n: shrink(Q)agg=cd..n Fcd..l de..l ef..l Lfg g Rhhi..l i Ajkl kl l Bmmn n Eevict c: shrink(R)agg=de..n Fde..l ef..l fg..l Lg Rhhi..l Aijkl jkl kl l Bmmn n Eevict d: shrink(S)agg=ef..n Fef..l fg..l gh..l LRAhi..l ijkl jkl kl l Bmmn n E Fig. 8
DABA Lite example trace for any aggregation. The notation foraggregates omits ⊗ , e.g., bc..f is shorthand for b ⊗ c ⊗ .. ⊗ f . fun insert ( v ) deque . pushBack ( v ) E ← E + 1 aggB ← aggB ⊗ v fixup () In our running example, if procNameaggB = ⊗ v = ⊗ aggB is also 4 ⊗ (F)max=5, maxcount=1 F5x1 5x1 LRA4x1 4x1 B0x10x1 Einsert 4(G) push fixupF5x1 5x1 4x1 4x1 B0x14x1 4x1LRA… E Similarly, evict pops an element from the front of thedeque, then calls fixup for one step of incremental reversal. fun evict () F ← F + 1 deque . popFront () fixup () For our running example, eviction is illustrated below: (I)max=5, maxcount=1 F5x1 L4x1 4x1 R0x14x2 4x1 A4x1 BEevict 5(J) popfixupFL4x1 4x1 R0x14x2 4x1 A4x1 BE… n-Order Sliding-Window Aggregation in Worst-Case Constant Time 15
The fixup function repairs the invariants. The effect of fixup on the size invariants is the same for DABA and for DABA Lite.Since Section 5 has a formal analysis, here we only have aninformal discussion. As before, the fixup function has fourcases singleton , flip , shift , and shrink . fun fixup () if F = B B ← E , A ← E , R ← E , L ← E aggRA ← aggB ← else if L = B L ← F , A ← E , B ← E aggRA ← aggB aggB ← if L = R A ← A + 1, R ← R + 1, L ← L + 1 else * L ← Π ⊗ L ⊗ aggRA L ← L + 1 *( A -1) ← *( A -1) ⊗ Π ⊗ A A ← A - 1 The singleton case happens when | l F | = | l B | = B ← E , A ← E , R ← E , L ← E The pointer assignments change this to | l F | = | l B | = (A)max=0, maxcount=0 FLRABEinsert 4(B)max=4, maxcount=1 F4x1 LRABEpushfixup (singleton case)4x14x1 EFLRAB The flip case happens when | l F | > | l L | = | l R | = | l A | =
0. That implies that | l F | = | l B | , whichmeans we can simply turn the old outer sublists l F and l B into the new inner sublists l L and l R . L ← F , A ← E , B ← EaggRA ← aggBaggB ← The corresponding updates to aggRA and aggB do not requireany invocations of ⊗ . (H)max=5, maxcount=1 F5x1 5x1 4x1 4x1 LRAB0x14x2 4x1 4x1 Eevict 4(I) pop F5x1 4x1 4x1 LRAB0x14x2 4x1 4x1 EFL5x1 3x1 4x1 R0x1 4x1 4x14x2 ABE… flipshrink The shift case happens when | l F | > | l L | =
0. Thatmeans the pointers L = R = A are equal, and can simply bemoved one element to the right. A ← A + 1, R ← R + 1, L ← L + 1 There is no need to update aggRA , since it will not be readanymore until after the next flip. The invariant for aggRA remains satisfied, thanks to L = R . (F)max=5, maxcount=1 F LRA B Einsert 4(G)max=5, maxcount=1 F5x1 5x1 4x1 LRA4x1 B0x14x1 4x1 EpushshiftF BLRA E5x1 5x1 4x1 4x1 0x10x1 5x1 5x1 4x1 4x1 0x14x1 4x1 The shrink case happens when | l F | > | l L | >
0. Thesizes of | l L | = | l R | are the same, and shrink reduces themby one each. This requires setting agg fields of blue sublistsappropriately for their contents invariants. * L ← Π ⊗ L ⊗ aggRAL ← L + 1*( A -1) ← *( A -1) ⊗ Π ⊗ A A ← A - 1 Even though the internal boundary A between l R and l A moves, taken together, these two sublists still occupy thesame elements, and thus, aggRA does not change. Conse-quently, the shrink case of DABA Lite requires one less ⊗ -invocation than the shrink case of DABA. (I)max=5, maxcount=1 F L R A BEevict 5(J)max=4, maxcount=3 F L R A BE popshrinkFL R A BE5x1 4x1 4x1 0x14x2 4x1 4x1 4x1 4x1 0x14x2 4x1 4x14x3 4x1 0x14x2 4x2 4x1 The shrink case also gets triggered right after a flip. (H)max=5, maxcount=1 F LRAB Eevict 4(I)max=5, maxcount=1 F L R A BE F LRAB EFL R ABEpop flipshrink5x1 5x1 4x1 4x1 0x14x2 4x1 4x1 5x1 4x1 4x1 0x14x2 4x1 4x15x1 3x1 4x1 0x1 4x1 4x14x25x1 4x1 4x1 0x14x2 4x1 4x1
DABA Lite Theorems.
Lemma 11
DABA Lite maintains both the contents invari-ants and the size invariants defined above.Proof
The query function does not modify the data struc-ture and thus does not change the invariants. The effect of insert , evict , and fixup on the size invariants is the sameas for DABA. Whenever sublist boundaries change, the codeupdates the contents of deque , aggRA , and aggB , if necessary,to uphold the contents invariants. (cid:117)(cid:116) Theorem 12
If the window currently contains v , . . . , v n − ,then query returns v ⊗ . . . ⊗ v n − .Proof Using Lemma 11, query ()= Π ⊗ F ⊗ aggB = v ⊗ . . . ⊗ v B − F − ⊗ v B − F ⊗ . . . ⊗ v E − F − = v ⊗ . . . ⊗ v n − (cid:117)(cid:116) Theorem 13
DABA Lite requires space to store n + partialaggregates. DABA Lite invokes ⊗ at most one time per query ,three times per insert , and two times per evict . Furthermore,for nonempty windows, DABA Lite invokes ⊗ on average twotimes per insert and one time per evict .Proof The algorithm contains no loops or recursion, so wecan directly see the worst-case numbers from the code. Theaverage-case numbers are based on the observation that everysequence of shrink steps is followed by an equal numberof shift steps. Shrink requires two ⊗ -invocations and shiftrequires zero ⊗ -invocations, so the average fixup call hasone ⊗ -invocation. (cid:117)(cid:116) The purpose of our experimental evaluation is to test whetherDABA’s worst-case constant algorithmic complexity yieldslow latency and high throughput in practice, and to testwhether DABA Lite is always more efficient than DABA.Our experiments use six different SWAGs: Two-Stacks,Two-Stacks Lite, and FlatFIT [28] are all amortized O ( ) ,worst-case O ( n ) algorithms designed for FIFO data. As orig-inally published, FlatFIT does not support dynamic windows.We use a modified version that resizes FlatFIT’s circularbuffer using the standard array doubling/shrinking techniquewithout disturbing FlatFIT’s internal pointer structure. Thisresults in an additional amortized O ( ) time per operation butsupports dynamic windows and guarantees that the memoryfootprint is within a constant factor of the window size. Wealso adapted the published FlatFIT algorithm to our SWAGframework. DABA and DABA Lite are both worst-case O ( ) algorithms designed for FIFO data. FiBA [33] is designed forout-of-order data and reduces to amortized O ( ) , worst-case O ( log n ) in the FIFO case. All of our experiments with FiBAuse a min-arity of 4.We chose three representative aggregation operators tospan the execution cost spectrum. The operator sum is thesum of all items in the window, and it represents aggrega-tion operations so cheap that the traversal and changes tothe underlying data structure should dominate performance.The operator bloom applies a Bloom filter to all items in thewindow, and it represents aggregation operations where theoperator itself dominates performance. The operator is so ex-pensive that minimizing calls to it matters more than changesto the underlying data structure. Finally, geomean computesthe geometric mean of all items in the window. It representsa middle ground of operator cost.We implemented all algorithms in C++11, using the g++compiler version 7.5.0 with optimization level - O3 . Our sys-tem runs Red Hat 7.3, with Linux kernel version 3.10.0. Theprocessor is an Intel Platinum 8168 at 2.7 GHz. All imple-mentations, experiments, and post-processing scripts usedin this section are available from the open-source projectSliding Window Aggregators .7.1 Static WindowsThe experiments in Figs 9 and 10 use a static count-basedwindow with synthetic data. In each experiment, we first insert n data items, where n is the size of the window. Thetimed part of the experiment consists of rounds of evict , insert , and query . For the latency experiments, we record Available at https://github.com/IBM/sliding-window-aggregators .Our experiments use the C++ implementations and benchmarks, as wellas the Python scripts from commit ee775 . all times for 10 million rounds with a fixed window size of2 data items. For the throughput experiments, we time howlong it takes to complete 200 million rounds, and we vary n from 1 to 2 .The practical reason to choose a worst-case O ( ) aggrega-tor is to minimize latency. Both Two-Stacks and Two-StacksLite in Fig 9 tend to have lower minimum latency than bothDABA and DABA Lite. But, true to their linear worst-case,Two-Stacks and Two-Stacks Lite regularly suffer from anorder-of-magnitude higher latency. This trend becomes morepronounced as the cost of the aggregation function increases.Unlike the other aggregators, FiBA is tree based. As main-taining the tree is more up-front work, it tends to have highminimum and median latency. But, also being tree based,its worst-case behavior is bounded by O ( log n ) ; it has lowerworst-case latency than the worst-case O ( n ) aggregators. Flat-FIT is not a tree-based structure, but the access pattern duringqueries ends up having similar properties: successive indirectaccesses to different parts of the window. Each query pushesindices onto a stack, and then pops indices from the stackto indirectly access the window. This is a large amount ofwork, and when the aggregation operation is cheap, this workdominates performance and yields a high latency floor.All of the aggregators are able to maintain close to con-stant behavior in Fig 10, although there are large differencesbetween them. Surprisingly, DABA’s throughput is morecompetitive with Two-Stacks than in prior work [31,32]. Weattribute this difference to a more modern compiler with moreaggressive inlining and dead-code elimination. Both DABALite and Two-Stacks Lite always outperform the correspond-ing non-Lite versions. FlatFIT becomes more competitivewith expensive operations as its indirect accesses becomeless important compared to the total number of calls to theaggregation operation.7.2 Dynamic WindowsThe experiments in Fig. 11 use a dynamic count-based win-dow with synthetic data. The experiments time a fill-and-drain pattern for a total of 200 million data items. It performs insert and query until reaching the window size n , and thencalls evict until the window is down to 0, then repeats. Wevary n from 1 to 2 .The throughput trends are largely the same as with staticwindows, which is the point of these experiments: even withdynamically changing window sizes, the fundamental proper-ties of these streaming aggregation algorithms remain mostly Our implementation performs an optimization where the same stackis reused across queries. This is safe because the stack is always emptyat the end of a query. For dynamic windows, the number of indicesinvolved can be non-constant. Avoiding the recreation of the stack andreusing the same memory makes about a 20% difference in throughput,but does not change FlatFIT’s overall comparative performance.n-Order Sliding-Window Aggregation in Worst-Case Constant Time 17 d a b a d a b a _ li t e t w o _ s t a c k s t w o _ s t a c k s _ li t e f l a t f i t f i b a p r o c e ss o r c y c l e s sum, window 2 d a b a d a b a _ li t e t w o _ s t a c k s t w o _ s t a c k s _ li t e f l a t f i t f i b a p r o c e ss o r c y c l e s geomean, window 2 d a b a d a b a _ li t e t w o _ s t a c k s t w o _ s t a c k s _ li t e f l a t f i t f i b a p r o c e ss o r c y c l e s bloom, window 2 Fig. 9
Latency, shown as violin plots, for static count-based windows with synthetic data. window size in data items t h r o u g h p u t [ m illi o n i t e m s / s ] FIFO sum dabadaba_lite two_stackstwo_stacks_lite flatfitfiba4 window size in data items t h r o u g h p u t [ m illi o n i t e m s / s ] FIFO geomean dabadaba_lite two_stackstwo_stacks_lite flatfitfiba4 window size in data items t h r o u g h p u t [ m illi o n i t e m s / s ] FIFO bloom dabadaba_lite two_stackstwo_stacks_lite flatfitfiba4
Fig. 10
Throughput for static count-based windows with synthetic data. max window size in data items t h r o u g h p u t [ m illi o n i t e m s / s ] Dynamic sum dabadaba_lite two_stackstwo_stacks_lite flatfitfiba4 max window size in data items t h r o u g h p u t [ m illi o n i t e m s / s ] Dynamic geomean dabadaba_lite two_stackstwo_stacks_lite flatfitfiba4 max window size in data items t h r o u g h p u t [ m illi o n i t e m s / s ] Dynamic bloom dabadaba_lite two_stackstwo_stacks_lite flatfitfiba4
Fig. 11
Throughput for dynamic count-based windows with synthetic data. the same. The one major difference is FlatFIT, whose through-put is consistently the best for bloom , which is the most expen-sive aggregation operation. This experimental design happensto be close to a best-case for FlatFIT, as it does not call theaggregation operation on evictions. FlatFIT only calls theaggregation operation on queries. The other algorithms re-quire calling the aggregation operation on evictions in orderto maintain their various properties of their partial aggregates.But, since the experiment performs no queries when it drainsthe window, such work is “wasted” in this case.7.3 Real DataThe experiments in Fig. 12 use dynamic event-based win-dows based on real data. We use the dataset from the DEBS2012 Grand Challenge [1], which recorded data from manu-facturing equipment at approximately 100 Hz. We removedabout 1.5% of the 32.3 million events to enforce in-order data to make it suitable for FIFO aggregation algorithms.Our experiments maintain an event-time-based window of τ seconds, which means that the actual number of data itemsin that window will fluctuate over time. We do not start mea-suring until the window has evicted its first data item. In thethroughput experiments, we vary τ from 10 milliseconds to 6hours. In the latency experiments, we choose a window of 10minutes. For both experiments, we use an aggregation opera-tion inspired by Query 2 of the DEBS 2012 Grand Challenge:relative variation.Because FiBA was designed for out-of-order data, it na-tively has a concept of timestamps. But the other algorithmsare FIFO aggregators and do not natively support timestamps.These sets of experiments use modified versions of all of theother aggregators that add support to query the youngest andoldest timestamps in the window. We use those queries tomaintain the event-time window of τ seconds.The results are consistent with the previous experiments,with two exceptions. First, DABA, DABA Lite, Two-Stacks, d a b a d a b a _ li t e t w o _ s t a c k s t w o _ s t a c k s _ li t e f l a t f i t f i b a p r o c e ss o r c y c l e s relvar, window 600.0 seconds −2 −1 window size in seconds t h r o u g h p u t [ m illi o n i t e m s / s ] DEBS 2012 relvar dabadaba_lite two_stackstwo_stacks_lite flatfitfiba4
Fig. 12
Throughput and latency for dynamic event-based windows withthe manufacturing equipment data set.
Two-Stacks Lite, and FlatFIT experience similar maximumlatency, unlike with the static experiments. This latency sim-ilarity is caused by rare bulk evictions: rounds where morethan 100 items are evicted experienced latencies more than100 × greater than the previous round. Since maintaining thesame window is a shared property across all experiments,all algorithms have similar maximum latency. Second, thethroughput of all algorithms experiences some degradationafter 10 seconds. This is because the actual window size isbecoming a large enough fraction of the total data that rareevents are not amortized.7.4 DiscussionOur experiments demonstrate several consistent behaviors.The throughput of the Lite variants of both DABA and Two-Stacks are always significantly better than the original ver-sion, but the overall difference in latency is less dramatic.Two-Stacks Lite tends to have the best throughput. Since welacked access to an implementation of FlatFIT by its authors,we re-implemented it based on their paper and extended itto support variable-sized windows. Our implementation ofFlatFIT tends to have a high latency floor, but a median thatis close to that floor. However, its maximum latency is con-sistent with being worst-case O ( n ) , and its throughput is onlycompetitive with expensive operators or when the experimenthappens to align with its design.The theoretical analysis established that both DABA andDABA Lite are worst-case constant time algorithms. Butlow latency and competitive throughputs are sensitive towhat the actual constant is. Our experiments demonstrate that DABA and DABA Lite are able to realize low latency andcompetitive throughput across a large range of n with bothstatic and dynamic windows and with real data. This section discusses the literature on sliding window aggre-gation algorithms with an emphasis on in-order streams andassociative aggregation operators ⊗ . As before, let n be thewindow size.8.1 Solutions to the Same ProblemSection 2 formalized the problem statement as an abstractdata type called SWAG for first-in-first-out sliding windowaggregation. This section discusses concrete algorithms thatimplement the abstract data type. Section 2 phrased associa-tive aggregation operators as monoids. While some monoidsare invertible or commutative, that is not true for all monoids,and thus, this section only lists algorithms that work withoutsuch additional algebraic properties. This section presentsalgorithms in chronological order by publication date. Recalculate-from-scratch implements SWAG by main-taining a FIFO queue of all data items and calculating theaggregation of the entire queue for each query. Each queryrequires O ( n ) invocations of ⊗ . The queue takes up space for n data stream values.The B-Int algorithm from 2004 [7] implements SWAGusing base intervals, which are similar to a balanced binarytree. The time complexity is O ( log n ) invocations of ⊗ andthe space complexity is around 2 n partial aggregates: n forleaves plus ∼ n for internal nodes.The Two-Stacks idea was mentioned in a Stack Overflowpost from 2011 [3], which described the idea for one ag-gregation operator, minimum. Even though Two-Stacks isa SWAG algorithm, it was not immediately noticed as suchby the academic community. As discussed in Section 3, thetime complexity is amortized O ( ) invocations of ⊗ with aworst-case of O ( n ) , and the space is 2 n partial aggregates.The Reactive Aggregator from 2015 implements SWAGusing a perfect binary tree. This algorithm uses a data struc-ture called FlatFAT, which stands for flat fixed-sized aggre-gator and represents a perfect binary tree without storingexplicit pointers and without needing any rebalancing. Thetime complexity is amortized O ( log n ) with a worst-caseof O ( n ) invocations of ⊗ . If n is a power of two, FlatFATrequires space for ∼ n partial aggregates: n leaves plus n interior nodes. If n is slightly above a power of two, the spacecan be up to ∼ n .The DABA algorithm was first published in 2015 [31].DABA was inspired by Okasaki’s purely functional queuesand deques [26]. However, the two differ substantially: DABA n-Order Sliding-Window Aggregation in Worst-Case Constant Time 19 is not a purely functional data structure, and Okasaki’s datastructures do not implement sliding window aggregation. Asdiscussed in Section 5, DABA requires worst-case O ( ) in-vocations of ⊗ and the space is 2 n partial aggregates.The FlatFIT algorithm from 2017 implements SWAGvia a flat and fast index traverser [28]. The time complexity isamortized O ( ) invocations of ⊗ with a worst-case of O ( n ) .The algorithm stores n partial aggregates as well as n pointers,which are indices into the window for stitching together thepartial aggregates of subranges. The algorithm requires anadditional stack of indices for pointer updates, and the authorsreport the total space requirements as up to 2 . n .The Hammer Slide paper from 2018 [35] starts from Two-Stacks and optimizes it further. The time complexity remainsamortized O ( ) invocations of ⊗ with a worst-case of O ( n ) .One of the optimizations from Hammer Slide is to only store n + AMTA algorithm from 2019 [40] implements SWAGvia an amortized monoid tree aggregator. AMTA adds so-phisticated tree representations that optimize FIFO insertand evict. Its amortized algorithmic time complexity is O ( ) invocations of ⊗ , with a worst-case of O ( log n ) . Like othertree-based SWAGs, AMTA requires ∼ n space for n leavesand ∼ n inner nodes. KVS-AMTA is an out-of-memory vari-ant that externalizes most of this space into a key-value store.The FiBA algorithm from 2019 [33] implements SWAGvia a finger B-tree aggregator. FiBA uses finger pointers,position-aware partial aggregates, and a suitable rebalancingstrategy to optimize insert and evict near the start and endof the window. For the FIFO case, its amortized algorithmictime complexity is O ( ) invocations of ⊗ , with a worst-caseof O ( log n ) . Its space complexity depends on the arity ofthe B-tree. Since the minimum arity of B-trees is more thanbinary, B-trees store fewer than ∼ n partial aggregates.The DABA Lite algorithm has not been published before,making it an original contribution of this paper. As discussedin Section 6, the time complexity is worst-case O ( ) invoca-tions of ⊗ and the space is n + Coarse-grained sliding reduces the effective window size n by storing only a single partial aggregate for values that willbe evicted together. A state-of-the-art algorithm for coarse-grained sliding is Scotty [38]. Reducing n reduces the time complexity of any algorithms whose time complexity de-pends upon n . Being worst-case O ( ) , the time complexityof DABA and DABA Lite does not depend on n . Reducing n also reduces the space complexity, which is somewhere be-tween n and ∼ n for all SWAG algorithms from Section 8.1. Bounded disorder handling tolerates out-of-order arrivalsof data stream items as long as the disorder is not too large.Srivastava and Widom described how to handle boundeddisorder by buffering incoming data items [30]. Later, whendata items are released from the buffer, they are orderedby their nominal timestamps. That makes it possible to useinorder SWAG algorithms from Section 8.1.
Partition parallelism is a way to parallelize stateful stream-ing applications as long as the computation for each parti-tion key is independent from the computation for the otherkeys [27]. Sliding-window aggregation is often used in away that satisfies this requirement, by aggregating separatelywithin each key. In that case, parallelization can just maintainseparate instances of a given SWAG. For this to work well, itis best not to conservatively preallocate too much memory,lest the data structures for rare keys take up too much space.Given various algorithms and techniques, how can wepick and combine the right ones for a given problem? Arecent paper by Traub et al. [39] presents decision trees fordispatching to the right combination given the stream order,window kinds, aggregation operators, window sharing, etc.We argue that DABA Lite should be used for the inorder casewith associative aggregation operators.8.3 Solutions to Other ProblemsOf course there are also problems around sliding-windowaggregations where it does not suffice to just combine aSWAG algorithm from Section 8.1 with a complementarytechnique from Section 8.2. This section highlight a few suchproblems with solutions; for more details see [18].
Window sharing serves sliding window aggregation que-ries for multiple window sizes from a single data structure.Not all data structures are suitable for this. SWAG algo-rithms from Section 8.1 that support window sharing includeB-Int [7], FlatFIT [28], and FiBA [33]. The SlideSlide algo-rithm implements SWAG for fixed-sized windows [36].
Unbounded disorder handling tolerates out-of-order ar-rivals that are arbitrarily late, incorporating them into thedata structure whenever they arrive. Truviso accomplishesthis for the case where multiple input streams have driftedarbitrarily far from each other, as long as each of the inputstreams is internally in-order [21]. FiBA supports generalout-of-order sliding window aggregation without restrictionson the degree of disorder [33]. The algorithmic complexity ofFiBA matches the theoretical lower bound for this problem.When it comes to aggregation operators , some algo-rithms are more restrictive than our problem statement from
Section 2. For instance, subtract-on-evict is a simple al-gorithm that only works when subtraction is well-defined,in other words, when the ⊗ operator is invertible. Simi-larly, SlickDeque [29] only works for aggregation opera-tors that are either invertible or that satisfy the property that x ⊗ y ∈ { x , y } . On the other hand, there are also some ag-gregation operators for which our problem statement fromSection 2 is a poor fit. One of the most prominent ones ismedian, or more generally, percentile aggregation. An effi-cient solution for sliding-window median and percentiles isan order statistics tree [17]. This paper is a journal version of our earlier conference pa-per [32] about DABA, the first algorithm for in-order slidingwindow aggregation in worst-case constant time. Besidesproviding a more comprehensive description of DABA, thispaper also introduces a new algorithm called DABA Lite thatimproves over DABA. Where DABA requires space to store2 n partial aggregates, DABA Lite only stores n + References
1. DEBS 2012 Grand Challenge: Manufacturing equipment.https://debs.org/grand-challenges/2012. Retrieved June 20202. Apache Flink: Scalable batch and stream data processing.https://flink.apache.org (2016). Retrieved Aug. 20163. adamax: Re: Implement a queue in which push rear(),pop front() and get min() are all constant time operations.http://stackoverflow.com/questions/4802038/ (2011). RetrievedAug., 20164. Agarwal, P.K., Cormode, G., Huang, Z., Phillips, J., Wei, Z., Yi, K.:Mergeable summaries. In: Symposium on Principles of DatabaseSystems (PODS), pp. 23–34 (2012)5. Akidau, T., Balikov, A., Bekiroglu, K., Chernyak, S., Haberman,J., Lax, R., McVeety, S., Mills, D., Nordstrom, P., Whittle, S.:MillWheel: Fault-tolerant stream processing at internet scale. In:Conference on Very Large Data Bases (VLDB) Industrial Track,pp. 734–746 (2013) 6. Ali, M., Chandramouli, B., Goldstein, J., Schindlauer, R.: The ex-tensibility framework in Microsoft StreamInsight. In: InternationalConference on Data Engineering (ICDE), pp. 1242–1253 (2011)7. Arasu, A., Widom, J.: Resource sharing in continuous sliding win-dow aggregates. In: Conference on Very Large Data Bases (VLDB),pp. 336–347 (2004)8. Bloom, B.H.: Space/time trade-offs in hash coding with allowableerrors. Communications of the ACM (CACM) (7), 422–426(1970)9. Boykin, O., Ritchie, S., O’Connell, I., Lin, J.: Summingbird: Aframework for integrating batch and online MapReduce compu-tations. In: Conference on Very Large Data Bases (VLDB), pp.1441–1451 (2014)10. Carbone, P., Traub, J., Katsifodimos, A., Haridi, S., Markl, V.:Cutty: Aggregate sharing for user-defined windows. In: Conferenceon Information and Knowledge Management (CIKM), pp. 1201–1210 (2016)11. Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introductionto Algorithms, 3rd Edition. MIT Press (2009). URL http://mitpress.mit.edu/books/introduction-algorithms
12. Cormode, G., Muthukrishnan, S.: An improved data stream sum-mary: The count-min sketch and its applications. Journal of Algo-rithms (1), 58–75 (2005)13. Cranor, C., Johnson, T., Spataschek, O., Shkapenyuk, V.: Gigas-cope: A stream database for network applications. In: InternationalConference on Management of Data (SIGMOD) Industrial Track,pp. 647–651 (2003)14. Flajolet, P., Fusy, E., Gandouet, O., Meunier, F.: HyperLogLog:The analysis of a near-optimal cardinality estimation algorithm.In: Conference on Analysis of Algorithms (AofA), pp. 127–146(2007)15. Gedik, B.: Generic windowing support for extensible stream pro-cessing systems. Software Practice and Experience (SP&E) pp.1105–1128 (2013)16. Hirzel, M., Andrade, H., Gedik, B., Jacques-Silva, G., Khandekar,R., Kumar, V., Mendell, M., Nasgaard, H., Schneider, S., Soul´e,R., Wu, K.L.: IBM Streams Processing Language: Analyzing bigdata in motion. IBM Journal of Research and Development (3/4)(2013)17. Hirzel, M., Rabbah, R., Suter, P., Tardieu, O., Vaziri, M.: Spread-sheets for stream processing with unbounded windows and parti-tions. In: Conference on Distributed Event-Based Systems (DEBS),pp. 49–60 (2016)18. Hirzel, M., Schneider, S., Tangwongsan, K.: Tutorial: Sliding-window aggregation algorithms. In: Conference on DistributedEvent-Based Systems (DEBS), pp. 11–14 (2017)19. Izbicki, M.: Algebraic classifiers: A generic approach to fast cross-validation, online training, and parallel training. In: InternationalConference on Machine Learning (ICML), pp. 648–656 (2013)20. Jugel, U., Jerzak, Z., Hackenbroich, G., Markl, V.: M4: Avisualization-oriented time series data aggregation. In: Confer-ence on Very Large Data Bases (VLDB), pp. 797–808 (2014)21. Krishnamurthy, S., Franklin, M.J., Davis, J., Farina, D., Golovko,P., Li, A., Thombre, N.: Continuous analytics over discontinuousstreams. In: International Conference on Management of Data(SIGMOD), pp. 1081–1092 (2010)22. Krishnamurthy, S., Wu, C., Franklin, M.: On-the-fly sharing forstreamed aggregation. In: International Conference on Manage-ment of Data (SIGMOD), pp. 623–634 (2006)23. Kulkarni, S., Bhagat, N., Fu, M., Kedigehalli, V., Kellogg, C., Mit-tal, S., Patel, J.M., Ramasamy, K., Taneja, S.: Twitter Heron: Streamprocessing at scale. In: International Conference on Managementof Data (SIGMOD), pp. 239–250 (2015)24. Li, J., Maier, D., Tufte, K., Papadimos, V., Tucker, P.A.: No pane,no gain: efficient evaluation of sliding-window aggregates overdata streams. ACM SIGMOD Record (1), 39–44 (2005)n-Order Sliding-Window Aggregation in Worst-Case Constant Time 2125. Murray, D.G., McSherry, F., Isaacs, R., Isard, M., Barham, P.,Abadi, M.: Naiad: A timely dataflow system. In: Symposium onOperating Systems Principles (SOSP) (2013)26. Okasaki, C.: Simple and efficient purely functional queues anddeques. Journal of Functional Programming (JFP) (4), 583–592(1995)27. Schneider, S., Hirzel, M., Gedik, B., Wu, K.L.: Safe data paral-lelism for general streaming. IEEE Transactions on Computers(TC) (2), 504–517 (2015)28. Shein, A.U., Chrysanthis, P.K., Labrinidis, A.: FlatFIT: Acceleratedincremental sliding-window aggregation for real-time analytics. In:Conference on Scientific and Statistical Database Management(SSDBM), pp. 5.1–5.12 (2017)29. Shein, A.U., Chrysanthis, P.K., Labrinidis, A.: SlickDeque: Highthroughput and low latency incremental sliding-window aggrega-tion. In: Conference on Extending Database Technology (EDBT),pp. 397–408 (2018)30. Srivastava, U., Widom, J.: Flexible time management in data streamsystems. In: Principles of Database Systems (PODS), pp. 263–274(2004)31. Tangwongsan, K., Hirzel, M., Schneider, S.: Constant-time slidingwindow aggregation. Tech. Rep. RC25574, IBM Research (2015)32. Tangwongsan, K., Hirzel, M., Schneider, S.: Low-latency sliding-window aggregation in worst-case constant time. In: Conferenceon Distributed Event-Based Systems (DEBS), pp. 66–77 (2017)33. Tangwongsan, K., Hirzel, M., Schneider, S.: Optimal and generalout-of-order sliding-window aggregation. In: Conference on VeryLarge Data Bases (VLDB), pp. 1167–1180 (2019)34. Tangwongsan, K., Hirzel, M., Schneider, S., Wu, K.L.: Generalincremental sliding-window aggregation. In: Conference on VeryLarge Data Bases (VLDB), pp. 702–713 (2015)35. Theodorakis, G., Koliousis, A., Pietzuch, P.R., Pirk, H.: HammerSlide: Work- and CPU-efficient streaming window aggregation. In: Workshop on Accelerating Analytics and Data Management Sys-tems Using Modern Processor and Storage Architectures (ADMS),pp. 34–41 (2018)36. Theodorakis, G., Pietzuch, P.R., Pirk, H.: SlideSlide: A fast in-cremental stream processing algorithm for multiple queries. In:Conference on Extending Database Technology (EDBT), pp. 435–438 (2020)37. Toshniwal, A., Taneja, S., Shukla, A., Ramasamy, K., Patel, J.M.,Kulkarni, S., Jackson, J., Gade, K., Fu, M., Donham, J., Bhagat,N., Mittal, S., Ryaboy, D.: Storm @Twitter. In: InternationalConference on Management of Data (SIGMOD), pp. 147–156(2014)38. Traub, J., Grulich, P., Cuellar, A.R., Bre¨s, S., Katsifodimos, A.,Rabl, T., Markl, V.: Scotty: Efficient window aggregation for out-of-order stream processing. In: Poster at the International Conferenceon Data Engineering (ICDE-Poster) (2018)39. Traub, J., Grulich, P., Cuellar, A.R., Bre¨s, S., Katsifodimos, A.,Rabl, T., Markl, V.: Efficient window aggregation with generalstream slicing. In: Conference on Extending Database Technology(EDBT) (2019)40. Villalba, A., Berral, J.L., Carrera, D.: Constant-time sliding win-dow framework with reduced memory footprint and efficient bulkevictions. Transactions on Parallel and Distributed Systems (TPDS)30