HyperBench: A Benchmark and Tool for Hypergraphs and Empirical Findings
Wolfgang Fischl, Georg Gottlob, Davide Mario Longo, Reinhard Pichler
HHyperBench: A Benchmark and Tool for Hypergraphs andEmpirical Findings ∗ WOLFGANG FISCHL,
TU Wien
GEORG GOTTLOB,
University of Oxford
DAVIDE MARIO LONGO,
TU Wien
REINHARD PICHLER,
TU WienTo cope with the intractability of answering Conjunctive Queries (CQs) and solving Constraint SatisfactionProblems (CSPs), several notions of hypergraph decompositions have been proposed – giving rise to differentnotions of width, noticeably, plain, generalized, and fractional hypertree width (hw, ghw, and fhw). Giventhe increasing interest in using such decomposition methods in practice, a publicly accessible repository ofdecomposition software, as well as a large set of benchmarks, and a web-accessible workbench for inserting,analyzing, and retrieving hypergraphs are called for.We address this need by providing (i) concrete implementations of hypergraph decompositions (includingnew practical algorithms), (ii) a new, comprehensive benchmark of hypergraphs stemming from disparateCQ and CSP collections, and (iii) HyperBench, our new web-interface for accessing the benchmark and theresults of our analyses. In addition, we describe a number of actual experiments we carried out with this newinfrastructure.
In this work we study computational problems on hypergraph decompositions which are designedto speed up the evaluation of Conjunctive Queries (CQs) and the solution of Constraint SatisfactionProblems (CSPs). Hypergraph decompositions have meanwhile found their way into commercialdatabase systems such as LogicBlox [5, 8, 36, 37, 46] and advanced research prototypes such asEmptyHeaded [1, 2, 47, 55]. Hypergraph decompositions have also been successfully used in theCSP area [4, 33, 39]. In theory, the pros and cons of various notions of decompositions and widthsare well understood (see [24] for a survey). However, from a practical point of view, many questionshave remained open.We want to collect hypergraphs from different application contexts, analyze their structuralproperties and, in particular, their (generalized) hypertree width ( ghw and hw , respectively) andmake this hypergraph collection together with the results of our analyses publicly available. Theinvestigation of millions of CQs [12, 48] posed at various SPARQL endpoints suggests that thesereal-world CQs with atoms of arity ≤ hw : the overwhelming majority is acyclic;almost all of the rest has hw =
2. It is, however, not clear if CQs with arbitrary arity and CSPs alsohave low hypertree width, say, hw ≤
5. Ghionna et al. [22] gave a positive answer to this questionfor a small set of TPC-H benchmark queries. We significantly extend their collection of CQs.Answering CQs and solving CSPs are fundamental tasks in Computer Science. Formally, theyare the same problem, since both correspond to the evaluation of first-order formulae over afinite structure, such that the formulae only use { ∃ , ∧} as connectives but not { ∀ , ∨ , ¬} . Both ∗ This is an extended and enhanced version of the paper “HyperBench: A Benchmark and Tool for Hypergraphs and EmpiricalFindings" presented at PODS 2019 [18]Authors’ addresses: Wolfgang Fischl, TU Wien, [email protected]; Georg Gottlob, University of Oxford, [email protected]; Davide Mario Longo, TU Wien, [email protected]; Reinhard Pichler, TU Wien, [email protected]. This is the author’s version of the work. It is posted here for your personal use. Not for redistribution. The definitiveVersion of Record was published in , https://doi.org/10.1145/nnnnnnn.nnnnnnn., Vol. 1, No. 1, Article . Publication date: September 2020. a r X i v : . [ c s . D B ] S e p Wolfgang Fischl, Georg Gottlob, Davide Mario Longo, and Reinhard Pichler problems, answering CQs and solving CSPs, are NP-complete [15]. Consequently, the search fortractable fragments of these problems has been an active research area in the database and artificialintelligence communities for several decades.The most powerful methods known to date for defining tractable fragments are based on variousdecompositions of the hypergraph structure underlying a given CQ or CSP. The most importantforms of decompositions are hypertree decompositions (HDs) [27], generalized hypertree decomposi-tions (GHDs) [27], and fractional hypertree decompositions (FHDs) [31]. These decomposition methodsgive rise to three notions of width of a hypergraph H : the hypertree width hw ( H ) , generalized hy-pertree width ghw ( H ) , and fractional hypertree width fhw ( H ) , where, fhw ( H ) ≤ ghw ( H ) ≤ hw ( H ) holds for every hypergraph H . For definitions, see Section 3.Both, answering CQs and solving CSPs, become tractable if the underlying hypergraphs havebounded hw , ghw , or, fhw and an appropriate decomposition is given. This gives rise to the problemof recognizing if a given CQ or CSP has hw , ghw , or, fhw bounded by some constant k . Formally,for decomposition ∈ { HD, GHD, FHD } and k ≥
1, we consider the following family of problems:Check ( decomposition , k ) Input hypergraph H = ( V , E ) ; Output decomposition of H of width ≤ k if it exists and answer ‘no’ otherwise.Clearly, bounded fhw defines the largest tractable class while bounded hw defines the smallestone. On the other hand, the problem Check ( HD , k ) is feasible in polynomial time [27] while theCheck ( GHD , k ) [28] and Check ( FHD , k ) [19] problems are NP-complete even for k = ( HD , k ) problem exist [30, 50]. In contrast, for Check ( GHD , k ) andCheck ( FHD , k ) , apart from exhaustive search over possible decomposition trees (which only worksfor small hypergraphs), no implementations have been reported yet [1] – with one exception:very recently, an interesting approach is presented in [17], where SMT-solving is applied to theCheck ( FHD , k ) problem. The same approach has been later extended to solve the Check ( HD , k ) problem [52]. In [30], tests of the Check ( HD , k ) system are presented. However, a benchmark forsystematically evaluating systems for the Check ( decomposition , k ) problem with decomposition ∈ { HD, GHD, FHD } and k ≥ Goal 1:
Create a comprehensive, easily extensible benchmark of hypergraphs corresponding toCQs or CSPs for the analysis of hypergraph decomposition algorithms.
Goal 2:
Use the benchmark from Goal 1 to find out if the hypertree width is, in general, smallenough (say ≤
5) to allow for efficient evaluation of CQs of arbitrary arity and of CSPs.Recently, in [19], the authors have identified classes of CQs for which the Check ( GHD , k ) andCheck ( FHD , k ) problems become tractable (from now on, we only speak about CQs; of course, allresults apply equally to CSPs). To this end, the Bounded Intersection Property (BIP) and, moregenerally, the Bounded Multi-Intersection Property (BMIP) have been introduced. The maximumnumber d of attributes shared by two (resp. c ) atoms is referred to as the intersection size (resp. c -multi-intersection size) of the CQ, which is similar to the notion of cutset width from the CSPliterature [16]. We say that a class of CQs satisfies the BIP (resp. BMIP) if the number of attributesshared by two (resp. by a constant number c of) query atoms is bounded by some constant d .A related property is that of bounded degree, i.e., each attribute only occurs in a constant numberof query atoms. Clearly, the BMIP is an immediate consequence of bounded degree. It has beenshown in [19] that Check ( GHD , k ) is solvable in polynomial time for CQs whose underlyinghypergraphs satisfy the BMIP. For CQs, the BMIP and bounded degree seem natural restrictions.For CSPs, the situation is not so clear. This yields the following research goals. Goal 3:
Use the hypergraph benchmark from Goal 1 to analyze how realistic the restrictions tolow (multi-)intersection size, or low degree of CQs and CSPs are. , Vol. 1, No. 1, Article . Publication date: September 2020. yperBench: A Benchmark and Tool for Hypergraphs and Empirical Findings 3
Goal 4:
Verify that for hypergraphs of low intersection size, the Check ( GHD , k ) problem indeedallows for efficient algorithms that work well in practice.The tractability results for Check ( FHD , k ) [19] are significantly weaker than for Check ( GHD , k ) :they involve a factor which is at least double-exponential in some “constant” (namely k , the bound δ on the degree and/or the bound d on the intersection size). Hence, we want to investigate if(generalized) hypertree decompositions could be “fractionally improved” by taking the integraledge cover at each node in the HD or GHD and replacing it by a fractional edge cover. We will thusintroduce the notion of fractionally improved HD which checks if there exists an HD of width ≤ k ,such that replacing each integral cover by a fractional cover yields an FHD of width ≤ k ′ for givenbounds k , k ′ with 0 < k ′ < k . Goal 5:
Explore the potential of fractionally improved HDs, i.e., investigate if the improvementsachieved are significant.In cases where Check ( GHD , k ) and Check ( FHD , k ) are intractable, we may have to settle for goodapproximations of ghw and fhw . For GHDs, we may thus use the inequality ghw ( H ) ≤ · hw ( H ) + H [3]. In contrast, for FHDs, the best known general, polynomial-time approximation is cubic. More precisely, in [43], a polynomial-time algorithm is presentedwhich, given a hypergraph H with fhw ( H ) = k , computes an FHD of width O( k ) . In [19], it isshown that a polynomial-time approximation up to a logarithmic factor is possible for any class ofhypergraphs with bounded Vapnik–Chervonenkis dimension (VC-dimension; see Section 3 for aprecise definition). The problem of efficiently approximating the ghw and/or fhw leads us to thefollowing goals. Goal 6:
Use the benchmark from Goal 1 to analyze if, in practice, hw and ghw indeed differ byfactor 3 or, if hw is typically much closer to ghw than this worst-case bound. Goal 7:
Use the benchmark from Goal 1 to analyze how realistic the restriction to small VC-dimension of CQs and CSPs is.
Results.
Our main results are as follows: • We provide
HyperBench , a comprehensive hypergraph benchmark of initially over 3,000hypergraphs ( see Section 5). This benchmark is exposed by a web interface, which allows the userto retrieve the hypergraphs or groups of hypergraphs together with a broad spectrum of propertiesof these hypergraphs, such as lower/upper bounds on hw and ghw , (multi-)intersection size, degree,etc. • We extend the software for HD computation from [30] to also solve the Check ( GHD , k ) problem. For a given hypergraph H , our system first computes the intersection size of H andthen applies the ghw -algorithm from [19], which is parameterized by the intersection size. Weimplement several improvements and we further extend the system to compute also “fractionallyimproved” HDs. • We carry out an empirical analysis of the hypergraphs in the HyperBench benchmark. This anal-ysis demonstrates, especially for real-world instances, that the restrictions to BIP, BMIP, boundeddegree, and bounded VC-dimension are astonishingly realistic. Moreover, on all hypergraphs in theHyperBench benchmark, we run our hw - and ghw -programs to identify (or at least bound) their hw and ghw . An interesting observation of our empirical study is that apart from the CQs also asignificant portion of CSPs in our benchmark has small hypertree width (all non-random CQs have hw ≤ hw ≤ hw ≤
5, inall of the cases where the ghw -computation terminates, hw and ghw have identical values. • In our study of the ghw of the hypergraphs in the HyperBench benchmark, we observe that astraightforward implementation of the algorithm from [19] for hypergraphs of low intersection , Vol. 1, No. 1, Article . Publication date: September 2020.
Wolfgang Fischl, Georg Gottlob, Davide Mario Longo, and Reinhard Pichler size is too slow in many cases. We therefore present a new approach (based on so-called “balancedseparators”) with promising experimental results. It is interesting to note that the new approachworks particularly well in those situations which are particularly hard for the straightforwardimplementation, namely hypergraphs H where the test if ghw ≤ k for given k gives a “no”-answer.Hence, combining the different approaches is very effective. Structure.
This paper is structured as follows: In Section 2, we summarize related work. In Section 3,we recall some basic notions. In Section 4, we describe our algorithms for solving the Check ( GHD , k ) problem. In Section 5, we present our system and test environment as well as our HyperBenchbenchmark. This section also contains the first results of our empirical study of the hypergraphsin this benchmark. In Section 6 we report on the performance of the implementations of ourGHD-algorithms as well as a further extension of the system to allow for the computation offractionally improved HDs. We conclude in Section 7 by highlighting the most important lessonslearned from our empirical study and by identifying some appealing directions for future work. We distinguish several types of works that are highly relevant to ours. The works most closelyrelated are the descriptions of HD, GHD and FHD algorithms in [19, 27] and the implementation ofHD computation by the
DetKDecomp program reported in [30]. We have extended these works inseveral ways. Above all, we have incorporated our analysis tool (reported in Sections 5 and 6) andthe GHD and FHD computations (reported in Sections 4 and 6.5) into the
DetKDecomp program –resulting in our
NewDetKDecomp library, which is openly available on GitHub at https://github.com/dmlongo/newdetkdecomp. For the GHD computation, we have added heuristics to speed up thebasic algorithm from [19]. Moreover, we have proposed a novel approach via balanced separators,which allowed us to significantly extend the range of instances for which the GHD computationterminates in reasonable time. We have also introduced a new form of decomposition method: thefractionally improved decompositions (see Section 6.5), which allow for a practical, lightweightform of FHDs.The second important input to our work comes from the various sources [6, 9–11, 21, 30, 34, 40, 54]which we took our CQs and CSPs from. Note that our main goal was not to add further CQs and/orCSPs to these benchmarks. Instead, we have aimed at taking and combining existing, openlyaccessible benchmarks of CQs and CSPs, and convert them into hypergraphs, which are thenthoroughly analyzed. Finally, the hypergraphs and the analysis results are made openly accessibleagain.The third kind of works highly relevant to ours are previous analyses of CQs and CSPs. To thebest of our knowledge, Ghionna et al. [22] presented the first systematic study of HDs of benchmarkCQs from TPC-H. However, Ghionna et al. pursued a research goal different from ours in that theyprimarily wanted to find out to what extent HDs can actually speed up query evaluation. Theyachieved very positive results in this respect, which have recently been confirmed by the workof Perelman et al. [47], Tu et al. [55] and Aberger et al. [1] on query evaluation using FHDs. Asa side result, Ghionna et al. also detected that CQs tend to have low hypertree width (a findingwhich was later confirmed in [12, 13, 48] and also in our study). In a pioneering effort, Bonifati,Martens, and Timm [12] have recently analyzed an unprecedented, massive amount of queries: theyinvestigated 180,653,910 queries from (not openly available) query logs of several popular SPARQLendpoints. After elimination of duplicate queries, there were still 56,164,661 queries left, out ofwhich 26,157,880 queries were in fact CQs. The authors thus significantly extend previous work byPicalausa and Vansummeren [48], who analyzed 3,130,177 SPARQL queries posed by humans andsoftware robots at the DBPedia SPARQL endpoint. The focus in [48] is on structural properties of , Vol. 1, No. 1, Article . Publication date: September 2020. yperBench: A Benchmark and Tool for Hypergraphs and Empirical Findings 5
SPARQL queries such as keywords used and variable structure in optional patterns. There is oneparagraph devoted to CQs, where it is noted that 99.99% of ca. 2 million CQs considered in [48] areacyclic.Many of the CQs (over 15 million) analyzed in [12] have arity 2 (here we consider the maximumarity of all atoms in a CQ as the arity of the query), which means that all triples in such a SPARQLquery have a constant at the predicate-position. Bonifati et al. made several interesting observationsconcerning the shape of these graph-like queries. For instance, they detected that exactly oneof these queries has tw =
3, while all others have tw ≤ hw ≤ DetKDecomp program from [30]. Out of 6,959,510 CQs of arity 3, only 86 (i.e. 0.01%(cid:24)) turnedout to have hw = hw = >
3, the hypertree width indeed tends tobe low, with the majority of queries being even acyclic.Bonifati et al. continued on this line of work and analyzed in [13] a yet bigger collection ofSPARQL queries coming from Wikidata. This repository of 208,215,209 queries was first madeavailable by Malyshev et al. for the work in [42]. Bonifati et al. divided this dataset into four disjointsets: queries for which the HTTP request was successful, further partitioned into organic androbotic queries; and timeout queries, again further partitioned into organic and robotic queries.Organic queries are the ones classified by Malyshev et al. as posed by humans, while robotic querieshave been classified as produced by synthetic algorithms. While Malyshev et al. analyzed onlythe sets of successful queries, Bonifati et al. extended their study to timeout queries. The latterturned out to be the most interesting w.r.t. structural analysis. As in [12], the focus of [13] ison examining property paths of SPARQL queries and, in particular, having a clear picture of thestructural characteristics of recursive properties.In [13], structural analysis builds upon conjunctive queries and variants thereof. In total, Bonifatiet al. identify 176,679,495 robotic and 342,576 organic queries for the largest fragment of CQs(namely C2RPQ+), which constitutes circa 85% of the whole dataset. Structural analysis showedthat these queries are mildly cyclic , i.e., their treewidth is bounded by a small constant. In particular,for all these queries tw ≤ OF + ), a structuralanalysis based on hypergraphs is more suitable, thus they computed hw for a total of 1,915,550CQ OF + queries. It turns out that 590,005 queries have hw =
2, while the rest has hw =
1, i.e., theyare acyclic.For the analysis of CSPs, much less work has been done. Although it has been shown thatexploiting (hyper-)tree decompositions may significantly improve the performance of CSP solving[4, 33, 35, 39], a systematic study on the (generalized) hypertree width of CSP instances has onlybeen carried out by few works [30, 39, 51]. To the best of our knowledge, we are the first to analyzethe hw , ghw , and fhw of ca. 2,000 CSP instances, where most of these instances have not beenstudied in this respect before.It should be noted that the focus of our work is different from the above mentioned previousworks: above all, we wanted to test the practical feasibility of various algorithms for HD, GHD,and FHD computation (including both, previously presented algorithms and new ones developedas part of this work). As far as our repository of hypergraphs (obtained from CQs and CSPs) isconcerned, we emphasize open accessibility. Thus, users can analyze their CQs and CSPs (withour implementations of HD, GHD, and FHD algorithms) or they can analyze new decompositionalgorithms (with our hypergraphs, which cover quite a broad range of characteristics). In fact, in therecent works on HD and FHD computation via SMT solving [17, 52], the HyperBench benchmarkhas already been used for the experimental evaluation. In [17] a novel approach to fhw computation , Vol. 1, No. 1, Article . Publication date: September 2020. Wolfgang Fischl, Georg Gottlob, Davide Mario Longo, and Reinhard Pichler via an efficient encoding of the check-problem for FHDs to SMT (SAT modulo Theory) is presented.The tests were carried out with 2,191 hypergraphs from the initial version of the HyperBench. Forall of these hypergraphs we have established at least some upper bound on the fhw either by our hw -computation or by one of our new algorithms presented in Sections 4 and 6.5. In contrast, theexact algorithm in [17] found FHDs only for 1.449 instances (66%). In 852 cases, both our algorithmsand the algorithm in [17] found FHDs of the same width; in 560 cases, an FHD of lower widthwas found in [17]. By using the same benchmark for the tests, the results in [17] and ours arecomparable and have thus provided valuable input for future improvements of the algorithms bycombining the different strengths and weaknesses of the two approaches.The use of the same benchmark has also allowed us to provide feedback to the authors of [17] fordebugging their system: in 9 out of 2,191 cases, the “optimal” value for the fhw computed in [17]was apparently erroneous, since it was higher than the hw found out by our analysis; note thatupper bounds on the width are, in general, more reliable than lower bounds since it is easy to verifyif a given decomposition indeed has the desired properties, whereas ruling out the existence of adecomposition of a certain width is a complex and error-prone task.The work in [29] represents a follow-up of this paper and the conference version [18]. Themain goal of [29] consists in providing major improvements in computing GHDs. To this aimthe authors present a parallel algorithm for computing GHDs based on the balanced separatormethod described in this paper, and a hybrid approach that combines the parallel and sequentialdecomposition algorithms. More specifically, the hybrid approach uses a parallel version of thebalanced separator algorithm to split a large hypergraph into smaller components and then usesthe sequential version of DetKDecomp from [30] to quickly decompose them. Moreover, the authorsof [29] propose new methods to simplify the input hypergraph and apply new heuristics to reducethe search space. Our HyperBench benchmark was fundamental for the experimental evaluation ofall these techniques. Indeed, the version of HyperBench presented in [18] was used as a baseline toshow that the newly proposed algorithms can be used to efficiently compute GHDs on modernmachines for a wide range of CSP instances.Similarly to our analysis of properties that make Check ( decomposition , k ) tractable or easy toapproximate, HyperBench has been used to empirically test the validity of theoretical hypotheses.In [38], the edge clique cover size of a graph is identified as a parameter allowing fixed-parameter-tractable algorithms for enumerating potential maximal cliques. The latter can be used to computeexact ghw and fhw . An edge clique cover of a graph is a set of cliques of the graph that covers allof its edges. In case of a CSP with n variables and m constraints, the set of constraints is an edgeclique cover of the underlying (hyper)graph. Thus, this property can be exploited for CSPs having n > m and HyperBench has been used to verify that it happens in circa 23% of the instances. We treat conjunctive queries (CQs) and constraint satisfaction problems (CSPs) as first-order formulaeusing only connectives in { ∃ , ∧} and disallowing { ∀ , ∨ , ¬} .A hypergraph H = ( V ( H ) , E ( H )) is a pair consisting of a set of vertices V ( H ) and a set of non-empty (hyper)edges E ( H ) ⊆ V ( H ) . We assume w.l.o.g. that there are no isolated vertices, i.e.,for each v ∈ V ( H ) , there is at least one edge e ∈ E ( H ) such that v ∈ e . We can thus identify ahypergraph H with its set of edges E ( H ) with the understanding that V ( H ) = { v ∈ e | e ∈ E ( H )} .A subhypergraph H ′ of H is then simply a subset of (the edges of) H .Given a formula ϕ corresponding to either a CQ or a CSP, the hypergraph H ϕ corresponding to ϕ has V ( H ϕ ) = vars ( ϕ ) and, for each atom a of ϕ , vars ( a ) ∈ E ( H ϕ ) . , Vol. 1, No. 1, Article . Publication date: September 2020. yperBench: A Benchmark and Tool for Hypergraphs and Empirical Findings 7 We are frequently dealing with sets of sets of vertices (e.g., sets of edges). For S ⊆ V ( H ) , wewrite (cid:208) S and (cid:209) S as a short-hand for taking the union or intersection, respectively, of this set ofsets of vertices, i.e., for S = { s , . . . , s ℓ } , we have (cid:208) S = (cid:208) ℓ i = s i and (cid:209) S = (cid:209) ℓ i = s i . We consider here three notions of hypergraph decompositions with their associated notions ofwidth. To this end, we first introduce the notion of (fractional) edge covers.Let H = ( V ( H ) , E ( H )) be a hypergraph and consider an edge weight function γ : E ( H ) → [ , ] .We define the set B ( γ ) of all vertices covered by γ and the weight of γ as B ( γ ) = v ∈ V ( H ) | (cid:213) e ∈ E ( H ) , v ∈ e γ ( e ) ≥ , weight ( γ ) = (cid:213) e ∈ E ( H ) γ ( e ) . We call γ a fractional edge cover of a set X ∈ V ( H ) by edges in E ( H ) , if X ⊆ B ( γ ) . We also consideran integral edge cover as a function λ : E ( H ) → { , } , i.e., a fractional edge cover whose valuesare restricted to { , } values. Following [27], we can also treat λ as a set with λ ⊆ E ( H ) (namely,the set of edges e with λ ( e ) =
1) and the weight as the cardinality of such a set of edges. In thefollowing, to emphasize the nature of the function we are dealing with, we will use γ for fractionaledge covers and λ for integral edge covers.We now introduce some relevant notions of decompositions.A tuple ⟨ T , ( B u ) u ∈ T ⟩ is a tree decomposition (TD) of hypergraph H = ( V ( H ) , E ( H )) , if T = ( N ( T ) , E ( T )) is a tree, every B u is a subset of V ( H ) , and the following conditions are satisfied:(1) For every edge e ∈ E ( H ) , there is a node u in T , such that e ⊆ B u , and(2) for every vertex v ∈ V ( H ) , { u ∈ T | v ∈ B u } is connected in T .The vertex sets B u are usually referred to as the bags of the TD. By slight abuse of notation, wewrite u ∈ T to express that u is a node in N ( T ) . Condition (2) is also called the “connectednesscondition".We use the following notational conventions throughout this paper. To avoid confusion, we willconsequently refer to the elements in V ( H ) as vertices of the hypergraph and to the elements in N ( T ) as the nodes of the decomposition. For a node u ∈ T , we write T u to denote the subtree of T rooted at u . By slight abuse of notation, we will often write u ′ ∈ T u to denote that u ′ is a node inthe subtree T u of T . Finally, we define V ( T u ) = (cid:208) u ′ ∈ T u B u ′ .A fractional hypertree decomposition (FHD) of a hypergraph H = ( V ( H ) , E ( H )) is a tuple ⟨ T , ( B u ) u ∈ T , ( γ u ) u ∈ T ⟩ , such that ⟨ T , ( B u ) u ∈ T ⟩ is a TD of H and the following condition holds:(3) For each u ∈ T , B u ⊆ B ( γ u ) holds, i.e., γ u is a fractional edge cover of B u .Following our notational convention, a generalized hypertree decomposition (GHD) is an FHD, where λ u is an integral edge weight function for every u ∈ T . Hence, by condition (3), λ u is an integraledge cover of B u . A hypertree decomposition (HD) of H is a GHD with the following additionalcondition (referred to as the “special condition” in [27]):(4) For each u ∈ T , V ( T u ) ∩ B ( γ u ) ⊆ B u , where V ( T u ) denotes the union of all bags in the subtreeof T rooted at u .Because of condition (4), it is important to consider T as a rooted tree in case of HDs. For TDs, FHDs,and GHDs, the root of T can be arbitrarily chosen or simply ignored. The width of an FHD, GHD,or HD is defined as the maximum weight of the functions γ u over all nodes u ∈ T . The fractional , Vol. 1, No. 1, Article . Publication date: September 2020. Wolfgang Fischl, Georg Gottlob, Davide Mario Longo, and Reinhard Pichler hypertree width, generalized hypertree width, and hypertree width of H (denoted fhw ( H ) , ghw ( H ) ,and hw ( H ) ) is the minimum width over all FHDs, GHDs, and HDs of H . For a set U ⊆ V ( H ) of vertices, we define [ U ] -components of a hypergraph H as follows: • We define [ U ] -adjacency as a binary relation on E ( H ) as follows: two edges e , e ∈ E ( H ) are [ U ] -adjacent , if ( e ∩ e ) \ U (cid:44) ∅ holds. • We define [ U ] -connectedness as the transitive closure of the [ U ] -adjacency relation. • A [ U ] -component of H is a maximally [ U ] -connected subset C ⊆ E ( H ) .For a set of edges S ⊆ E ( H ) , we say that C is “ [ S ] -connected ” or an “ [ S ] -component ” as a short-cutfor C is “ [ W ] -connected” or a “ [ W ] -component”, respectively, with W = (cid:208) e ∈ S e . We also call S a separator in this context. The size of an [ S ] -component C is defined as the number of edges inthe component. For a hypergraph H and a set of edges S ⊆ E ( H ) , we say that S is a balancedseparator if all [ S ] -components of H have a size ≤ | E ( H )| . It was shown in [3] that, for everyGHD ⟨ T , ( B u ) u ∈ T , ( λ u ) u ∈ T ⟩ of a hypergraph H , there exists a node u ∈ T such that λ u is a balancedseparator of H . This property can be made use of when searching for a GHD of size k of a hypergraph H as we shall show in Section 4.4 below. We briefly recall the basic principles of the
DetKDecomp program from [30] for computing HDs. It isrelevant for our work in that it is the first implementation of the original top-down nondeterministicHD algorithm presented in [27]. Even though the first implementation of a deterministic HDalgorithm is
OptKDecomp [26], it is based on a different characterization of hw , which is not suitablefor our purposes.For fixed k ≥ DetKDecomp tries to construct an HD of a hypergraph H in a top-down manner.It thus maintains a set C of edges, which is initialized to C = E ( H ) . For a node u in the HD (initially,this is the root of the HD), it “guesses” an edge cover λ u , i.e., λ u ⊆ E ( H ) and | λ u | ≤ k . For fixed k ,there are only polynomially many possible values λ u . DetKDecomp then proceeds by determiningall [ λ u ] -components C i with C i ⊆ C . The special condition of HDs restricts the possible choicesfor B u and thus guarantees that the [ λ u ] -components inside C and the [ B u ] -components inside C coincide. This is the crucial property for ensuring polynomial time complexity of HD-computation– at the price of possibly missing GHDs with a lower width.Now let C , . . . , C ℓ denote the [ λ u ] -components with C i ⊆ C . By the maximality of components,these sets C i are pairwise disjoint. Moreover, it was shown in [27] that if H has an HD of width ≤ k ,then it also has an HD of width ≤ k such that the edges in each C i are “covered” in different subtreesbelow u . More precisely, this means that u has ℓ child nodes u , . . . , u ℓ , such that for every i andevery e ∈ C i , there exists a node u e in the subtree rooted at u i with e ⊆ B u e . Hence, DetKDecomp recursively searches for an HD of the hypergraphs H i with E ( H i ) = C i and V ( H i ) = (cid:208) C i with theslight extra feature that also edges from E ( H ) \ C i are allowed to be used in the λ -labels of theseHDs. We are interested in certain structural properties of hypergraphs that make the Check ( GHD , k ) and Check ( FHD , k ) problems tractable or efficient to approximate for large classes of hypergraphs.We refer to the terminology of [25] as it makes uniform the one originally introduced in [19]. , Vol. 1, No. 1, Article . Publication date: September 2020. yperBench: A Benchmark and Tool for Hypergraphs and Empirical Findings 9 Definition 1.
For c ≥ , d ≥ , a hypergraph H = ( V ( H ) , E ( H )) is a ( c , d ) -hypergraph if theintersection of any c edges in E ( H ) has at most d elements, i.e., for every subset E ′ ⊆ E ( H ) with | E ′ | = c ,we have | (cid:209) E ′ | ≤ d . Definition 2.
A hypergraph H = ( V ( H ) , E ( H )) has c -multi-intersection size d if H is a ( c , d ) -hypergraph. In the special case of c = , we speak of intersection size of H , while if we do not have aparticular c in mind, we simply speak of multi-intersection size of H . Definition 3.
A class C of hypergraphs satisfies the bounded multi-intersection property (BMIP) ,if there exist c ≥ and d ≥ , such that every H in C is a ( c , d ) -hypergraph. As a special case, C satisfies the bounded intersection property (BIP) , if there exists d ≥ , such that every H in C is a ( , d ) -hypergraph. There are further relevant properties of (classes of) hypergraphs: bounded degree and boundedVapnik–Chervonenkis dimension (VC-dimension).Definition 4.
The degree deg ( H ) of a hypergraph H is defined as he maximum number δ ofhyperedges in which a vertex occurs, i.e., δ = max v ∈ V ( H ) |{ e ∈ E ( H ) | v ∈ E ( H )}| . A class C ofhypergraphs has bounded degree, if there exists δ ≥ , such that every hypergraph H in C has degree ≤ δ . Definition 5 ([56]).
Let H = ( V ( H ) , E ( H )) be a hypergraph, and X ⊆ V ( H ) a set of vertices. Denoteby E ( H )| X = { X ∩ e | e ∈ E ( H )} . X is called shattered if E ( H )| X = X . The Vapnik-Chervonenkisdimension (VC dimension) of H is the maximum cardinality of a shattered subset of V . We say that aclass C of hypergraphs has bounded VC-dimension, if there exists v ≥ , such that every hypergraph H ∈ C has VC-dimension ≤ v . Note that a hypergraph H with degree bounded by δ is a ( δ + , ) -hypergraph. Thus, boundeddegree implies the BMIP, which in turn implies bounded VC-dimension [19].The aforementioned properties help to solve or approximate the problems Check ( GHD , k ) andCheck ( FHD , k ) :Theorem 1 ([19, 25]). Let C be a class of hypergraphs. • If C has the BMIP, then the Check ( GHD , k ) problem is solvable in polynomial time for arbitrary k ≥ . Consequently, this tractability holds if C has bounded degree or the BIP (which eachimply the BMIP) [19]. • If C has bounded degree or the BIP, then the Check ( FHD , k ) problem is solvable in polynomialtime for arbitrary k ≥ [19, 25]. • If C has bounded VC-dimension, then the fhw can be approximated in polynomial time up to alog-factor [19]. In this section, we present two different ways to implement the tractable algorithm for computingGHDs in case of BIP [19]. We shall refer to these two implementations as
GlobalBIP and
LocalBIP .Moreover, we shall present a completely new approach to computing GHDs, which will be referredto as
BalSep . These variants exploit low intersection size to compute a GHD of width ≤ k of a hy-pergraph in polynomial time. While GlobalBIP and
LocalBIP differ in the time of the computationof certain sets of subedges, the
BalSep algorithm is a novel approach in computing decompositionsby means of balanced separators . , Vol. 1, No. 1, Article . Publication date: September 2020. In [19], the Check ( GHD , k ) problem was proved to be intractable for k ≥
2, but the authors alsoidentified tractable fragments corresponding to classes of hypergraphs having bounded (multi-)intersection size. In the following we focus on the intersection size, i.e., the maximum intersectionsize of any two edges in a hypergraph and explain how this property can be used for tractable ghw computation. Since the three variants presented later rely on the same central idea, we refer to thetheoretical algorithm as the дhw -algorithm .For a given hypergraph H = ( V ( H ) , E ( H )) , the ghw -algorithm adds a polynomial-time computableset f ( H , k ) of subedges of edges in E ( H ) to the hypergraph H . The set f ( H , k ) has the propertythat ghw ( H ) = k if and only if hw ( H ′ ) = k , where H ′ = ( V ( H ) , E ( H ) ∪ f ( H , k )) . Thus, it is possibleto use Check ( HD , k ) to solve Check ( GHD , k ) and achieve tractability. The particular set f ( H , k ) = (cid:216) e ∈ E ( H ) (cid:32) (cid:216) e ,..., e j ∈( E ( H )\{ e }) , j ≤ k ( e ∩( e ∪···∪ e j )) (cid:33) (1)contains, for each e ∈ E ( H ) , all subsets of intersections of e with up to k edges of H different from e . Although f ( H , k ) could in general contain an exponential number of elements, for fixed k andintersection size of H bounded by d , the set e ∩ ( e ∪ · · · ∪ e j ) contains at most d · k elements and,therefore, | f ( H , k )| is polynomially bounded.Since the ghw -algorithm relies on the computation of HDs, we implemented a new version of the hw -algorithm in [30] and called it NewDetKDecomp . Our program contains also the implementationsof all the algorithms in this section, but we defer a detailed discussion of the complete library toSection 6.
GlobalBIP
Algorithm
Given H and k ≥
1, a straightforward implementation of the ghw -algorithm consists in computingthe set f ( H , k ) , creating the hypergraph H ′ = ( V ( H ) , E ( H ) ∪ f ( H , k )) , and finally computing anHD of H of width ≤ k , if it exists. Since here we compute the set f ( H , k ) a priori globally for thewhole hypergraph, we call this algorithm GlobalBIP .Algorithm 1 is a detailed description of
GlobalBIP . The input consists of a hypergraph H and aconstant k , while the output is a GHD of H of width ≤ k , if it exists, and null otherwise. In line 2,we first compute f ( H , k ) as in Equation 1 and then, in line 3, we create the hypergraph H ′ which isobtained by adding the subedges in f ( H , k ) to H . In line 4, we call NewDetKDecomp on H ′ and k as ablack box and store its output in the variable HD . If HD is null, a GHD of width ≤ k of H does notexist, therefore we return null. Otherwise, we need to “fix" the decomposition as described in lines6-10. In particular, for each node u of HD , and for each edge of f ( H , k ) in λ u , i.e., e ′ ∈ ( λ u ∩ f ( H , k )) that is not an edge in the original hypergraph H , we substitute e with an edge e ′ ∈ E ( H ) suchthat e ⊆ e ′ . In this way we obtain a new edge cover λ ′ u such that B u ⊆ B ( λ u ) ⊆ B ( λ ′ u ) , but also | λ u | = | λ ′ u | . Thus, the new decomposition still satisfies all the properties of a GHD and its width isstill ≤ k . Eventually, in line 11, we return HD . LocalBIP
Algorithm
The main drawback of
GlobalBIP is that the size of f ( H , k ) , though polynomial, could be hugefor practical purposes. Therefore we looked at ways to reduce the number of edges to add to H byrestricting the computation only to those edges that might be actually necessary. The approach weused follows from an observation about the role played by f ( H , k ) in the tractability proof in [19].Recall that the proof uses Check ( HD , k ) on the hypergraph H ′ to answer Check ( GHD , k ) forthe hypergraph H . To do this in a sound way, the set f ( H , k ) has to contain all the edges that could , Vol. 1, No. 1, Article . Publication date: September 2020. yperBench: A Benchmark and Tool for Hypergraphs and Empirical Findings 11 Algorithm 1:
GlobalBIP
Input:
A hypergraph H . Parameter :
An integer k ≥ Output:
A GHD of H of width ≤ k if it exists, null otherwise. begin f ( H , k ) ← compute as in Equation 1 H ′ ← ( V ( H ) , E ( H ) ∪ f ( H , k )) HD ← NewDetKDecomp( H ′ , k ) if HD (cid:44) null then foreach u ∈ HD do foreach e ∈ ( λ u ∩ f ( H , k )) do e ′ ← e ′ ∈ E ( H ) such that e ⊆ e ′ λ u ( e ) ← λ u ( e ′ ) ← return HD be used to cover possible bags of H ′ in an HD without changing the width. Consider a top-downconstruction of a GHD of H . At some point we might want to choose, for some node u , a bag B u such that x (cid:60) B u for some variable x ∈ B ( λ u ) ∩ V ( T u ) . This choice would violate condition (4) ofHDs and would not be allowed for the computation of an HD. In particular, there is an edge e with x ∈ e and λ u ( e ) =
1. For this reason, the set f ( H , k ) contains an edge e ′ such that e ′ ⊂ e and x (cid:60) e ′ .Hence, we can substitute e with e ′ in the cover λ u (i.e., λ u ( e ) = , λ u ( e ′ ) =
1) to eliminate theviolation of condition (4). Moreover, because of the connectedness condition, there is no need tolook at the intersection of e with arbitrary edges in E ( H ) , instead we consider only the intersectionsof e with unions of edges that may possibly occur in bags of T u . In other words, for each node u ofthe decomposition, we consider only an appropriate subset f u ( H , k ) ⊆ f ( H , k ) . More specifically,for the current node u , let H u ⊆ H be the component we want to decompose. Then, we define f u ( H , k ) as follows: f u ( H , k ) = (cid:216) e ∈ E ( H ) (cid:32) (cid:216) e ,..., e j ∈( E ( H u )\{ e }) , j ≤ k ( e ∩( e ∪···∪ e j )) (cid:33) (2)We call the resulting algorithm LocalBIP because the set of edges f u ( H , k ) is computed locally for each node u during the construction of the decomposition. It follows NewDetKDecomp closely,but it differs in the search of the separators. In particular, while decomposing H , the algorithmfirst tries all possible ℓ -combinations of edges in E ( H ) (with ℓ ≤ k ) and only if the search does notsucceed, it tries ℓ -combinations of subedges in f u ( H , k ) . BalSep
Algorithm
So far we have presented two adaptations of the theoretical ghw -algorithm from [19]. On the onehand, they extend
NewDetKDecomp to compute GHDs and exploit bounded intersection size fortractability. On the other hand, they do not introduce any significant algorithmic innovation. In thefollowing we describe a novel approach to compute GHDs that makes use of sets of edges called balanced separators . We first extend the terminology of Section 3, then give a detailed descriptionof the algorithm, and finally prove that our algorithm is sound and complete. , Vol. 1, No. 1, Article . Publication date: September 2020.
Recall that a hypergraph is a pair H = ( V ( H ) , E ( H )) ,consisting of a set V ( H ) of vertices and a set E ( H ) of hyperedges (or, simply edges ), which arenon-empty subsets of V ( H ) . Since we assume that hypergraphs do not have isolated vertices, wecan identify a hypergraph H with its set of edges E ( H ) . Then, a subhypergraph H ′ of H is a subsetof (the edges of) H .Starting off with a hypergraph H , the BalSep algorithm has to deal with subhypergraphs H ′ ⊆ H augmented by a set S p of special edges. A special edge is simply a set of vertices from H . Intuitively,special edges correspond to bags B u in a GHD of H . Thus, an extended subhypergraph of H is of theform H ′ ∪ S p , where H ′ ⊆ H is a subhypergraph and S p is a set of special edges.We now extend three crucial definitions from hypergraphs to extended subhypergraphs, namelycomponents, balanced separators and GHDs. We recall that, even though a separator is a setof vertices, it can be defined as a set of edges. Then, for S ⊆ E ( H ) , an [ S ] -component is a [ W ] -component with W = (cid:208) e ∈ S e . In our algorithm we use the fact that, because of the edges in f ( H , K ) ,it is always possible to choose a separator λ u such that B ( λ u ) = B u [19]. Hence, there will not beany need to distinguish between vertex and edge separators. We start with components.Definition 6 (components of extended subhypergraphs). For a set U ⊆ V ( H ) of vertices, wedefine [ U ] -components of an extended subhypergraph H ′ ∪ S p of H as follows: • We define [ U ] -adjacency as a binary relation on H ′ ∪ S p as follows: two (possibly special) edges f , f ∈ H ′ ∪ S p are [ U ] -adjacent , if ( f ∩ f ) \ U (cid:44) ∅ holds. • We define [ U ] -connectedness as the transitive closure of the [ U ] -adjacency relation. • A [ U ] -component of H ′ ∪ S p is a maximally [ U ] -connected subset C ⊆ H ′ ∪ S p . Hence, if C , . . . , C ℓ are the [ U ] -components of H ′ ∪ S p , then H ′ ∪ S p is partitioned into C ∪ C ∪ · · · ∪ C ℓ , such that C = { f ∈ H ′ ∪ S p | f ⊆ U } .We next define balanced separators. While we give a definition w.r.t. sets of vertices U ⊆ V ( H ) ,they can be alternatively defined in terms of sets of edges S ⊆ E ( H ) .Definition 7 (balanced separators). Let H ′ ∪ S p be an extended subhypergraph of a hypergraph H and let U ⊆ V ( H ) be a set of vertices of H . The set U is a balanced separator of H ′ ∪ S p if for each [ U ] -component C i of H ′ ∪ S p , | C i | ≤ | H ′ ∪ S p | holds. In other words, no [ U ] -component must containmore than half the edges of H ′ ∪ S p . Finally, we extend GHDs to extended subhypergraphs.Definition 8 (GHDs of extended subhypergraphs).
Let H be a hypergraph and H ′ ∪ S p an extended subhypergraph of H . A GHD of H ′ ∪ S p is a tuple ⟨ T , ( B u ) u ∈ T , ( λ u ) u ∈ T ⟩ , where T = ( N ( T ) , E ( T )) is a tree, and B u and λ u are labeling functions, which map to each node u ∈ T two sets, B u ⊆ V ( H ) and λ u ⊆ E ( H ) ∪ S p . For a node u , we call B u the bag and λ u the edge cover of u . The set B ( λ u ) of vertices “covered” by λ u is defined as B ( λ u ) = { v ∈ V ( H ) | v ∈ f , f ∈ λ u } . The functions λ u and B u have to satisfy the following conditions: (1) For each node u ∈ T , eithera) λ u ⊆ E ( H ) and B u ⊆ B ( λ u ) , orb) λ u = { s } for some s ∈ S p and B u = s . (2) If, for some u ∈ T , λ u = { s } for some s ∈ S p , then u is a leaf node. (3) For each e ∈ H ′ ∪ S p , there is a node u ∈ T s.t. e ⊆ B u . (4) For each vertex v ∈ V ( H ) , { u ∈ T | v ∈ B u } is a connected subtree of T .The width of a GHD is defined as max {| λ u | : u ∈ T } . Clearly, also H itself is an extended subhypergraph of H with H ′ = H and S p = ∅ . It is readilyverified that the above definition of GHD of an extended subhypergraph H ′ ∪ S p and the definition , Vol. 1, No. 1, Article . Publication date: September 2020. yperBench: A Benchmark and Tool for Hypergraphs and Empirical Findings 13 of GHD of a hypergraph H coincide for the special case of taking H as an extended subhypergraphof itself.In [27], a normal form of hypertree decompositions was introduced. We will carry the notion ofnormal form over to GHDs of extended subhypergraph. To this end, it is convenient to first definethe set of edges exclusively covered by some subtree of a GHD:Definition 9. Let H ′ ∪ S p be an extended subhypergraph of some hypergraph H and D = ⟨ T , ( B u ) u ∈ T , ( λ u ) u ∈ T ⟩ a GHD for H ′ ∪ S p . For a node u ∈ T , we write T u to denote the subtree of T rooted at u . Moreover, we define the set of edges exclusively covered by T u as exCov ( T u ) = { f ∈ H ′ ∪ S p | ∃ v ∈ T u : f ⊆ B v ∧ f ⊈ B u } . Our normal form of GHDs is then defined as follows:Definition 10 (GHD of extended subhypergraphs normal form).
We say that a GHD ⟨ T , ( B u ) u ∈ T , ( λ u ) u ∈ T ⟩ of an extended subhypergraph H ′ ∪ S p is in normal form, if for the root node r of T , the following property holds: let u , . . . , u ℓ be the child nodes of r in T and let T u , . . . , T u ℓ denotethe subtrees in T rooted at u , . . . , u ℓ , respectively. Then exCov ( T u ) , . . . , exCov ( T u ℓ ) are precisely the [ V ( λ r )] -components of H ′ ∪ S p . Intuitively, each subtree T i below the root “covers” the edges of preciselyone [ B r ] -component of H ′ ∪ S p . The following lemma is an immediate extensions of previous results for hypergraphs to extendedsubhypergraphs.Lemma 1.
Let H ′ ∪ S p be an extended subhypergraph of some hypergraph H and suppose that thereexists a GHD D of width ≤ k for H ′ ∪ S p . Then there also exists a GHD D ′ in normal form of width ≤ k for H ′ ∪ S p , such that B r is a balanced separator of H ′ ∪ S p for the root node r of D ′ . Proof. The lemma combines two results from [27] and [3], respectively.Our normal form relaxes the normal form of HDs introduced in Definition 5.1 in [27]. Thetransformation into normal form can be taken over almost literally from the proof of Theorem 5.4in [27] for establishing the normal form of HDs.The existence of a balanced separator as the root of a GHD is implicit in the definition of“hyperlinkedness” and Theorem 19 in [3]. Again, it can be easily taken over to our case of anextended subhypergraph (i.e., to take also special edges into account). Actually, it can also beeasily proved directly by starting off at the root r of an arbitrary GHD of H ′ ∪ S p in normal formand checking if the components covered by the subtrees below all have size at most | H ′ ∪ S p | . If so,we already have the desired form. If not, there must be one subtree T r ′ rooted at a child r ′ of r ,such that exCov ( T r ′ ) is greater than | H ′ ∪ S p | . We then apply the normal form transformation alsoto T r ′ and check recursively if all the components covered by the subtrees below all have size atmost | H ′ ∪ S p | . By repeating this recursive step, we will eventually reach a node u , such that B u is a balanced separator. Then we simply take this node as the root and again apply the normalform transformation of the proof of Theorem 5.4 in [27] to this new root node and the subtreesimmediately below it. □ Here we describe Algorithm 2, which we call
BalSep . For a fixedinteger k ≥
1, it takes as input a hypergraph H and computes a GHD of H of width ≤ k if it exists,or returns null otherwise. The main of Algorithm 2 consists of a call to the Function Decomposewith parameters H and an empty set of special edges.The recursive Function Decompose constitutes the core of the algorithm and, given as input ahypergraph H ′ and a set of special edges S p , computes a GHD of H ′ ∪ S p of width ≤ k if it exists.Lines 5-12 deal with the two base cases of the algorithm. If H ′ ∪ S p has only one edge, we create a , Vol. 1, No. 1, Article . Publication date: September 2020. decomposition made of a single node u whose label λ u contains the only edge of H ′ ∪ S p and thebag B u = V ( H ′ ∪ S p ) , i.e., all the vertices of the extended subhypergraph, which are also the onescovered by λ u . In a similar way, we deal with the case | E ( H ′ ∪ S p )| =
2. We simply create two nodes u , v , one for each edge of H ′ ∪ S p , and then attach v as a child of u and return the decomposition.If the extended subhypergraph has at least 3 edges, we have to decompose it until we reachone of the two base cases. In line 13, we initialize the object BalSepIt , which is an iterator over thebalanced separators of size ≤ k of H ′ ∪ S p with edges in H . The iterator BalSepIt produces, one byone, all the ℓ -combinations of edges in H , for each ℓ ≤ k , to find a balanced separator for H ′ ∪ S p .Moreover, if all the combinations of full edges fail, the function uses subedges of H to generateseparators corresponding to elements of the set f ( H , k ) of Equation 1.In the while loop in lines 14-27, we recursively decompose H ′ ∪ S p . We are now creating thecurrent node u of the GHD and we have to compute λ u , B u and the children of u . In line 15 we set λ u as the next balanced separator and, in line 16, we fix the bag B u = B ( λ u ) as discussed above. Wewant to compute a GHD for each [ B u ] -component of H ′ ∪ S p , in such a way that it will be possibleto attach each of them to the current node u without violating any condition of Definition 8.Function ComputeSubhypergraphs computes the set of extended subhypergraphs correspondingto [ B u ] -components of H ′ ∪ S p and introduces, in each of them, a new special edge B u for connect-edness. We assume here the existence of a function ConnectedComponents which computes theconnected components of a hypergraph in a standard way. First, Function ComputeSubhypergraphscomputes the hypergraph H u = ( V u , E u ) resulting from the removal of all vertices in B u from H ′ ∪ S p .Then, in lines 5-11, it creates a new subhypergraph of H ′ ∪ S p for each connected component of H u . The new subhypergraph is stored in a variable c , which is a pair consisting of a hypergraph H and a set of special edges S p . For a single subhypergraph, the new set of special edges c . S p iscomposed of the edges of S p intersecting the current component comp plus a new special edge s = B u corresponding to the separator B u . We can then compute the hypergraph c . H . Its set ofedges E contains the edges of H ′ intersecting the current component comp and its set of vertices isthe union of E and c . S p . We then add c to the set of results res , which we finally return in line 12.Back in Function Decompose, in lines 18-24, we recursively compute a GHD for each extendedsubhypergraph returned by Function ComputeSubhypergraphs. If the decomposition D returned inline 19 is not null, we add it to the set subDecomps of the children of the current node u , otherwise,we set subDecomps to null and break the loop. At the end of the loop, we check whether subDecomps is null. If this is the case, it means that one of the recursive calls of Function Decompose wasunsuccessful. We then have to continue the while loop of lines 14-27 and try the next balancedseparator. In case all the recursive calls of Function Decompose were successful, Function BuildGHDbuilds the resulting GHD and returns it in line 27. If the algorithm exhausts all the choices ofbalanced separators for H ′ ∪ S p and exits the while loop of lines 14-27, it means that it is impossibleto create a GHD of width ≤ k for H ′ ∪ S p and we return null in line 28.Finally, we describe the Function BuildGHD that, given a bag B u , an edge cover λ u , and a set children of GHDs, returns a GHD with root u and children children . We start off creating the node u with labels B u and λ u . Then, for each child D ∈ children , we find the node r in T (the tree of D )having B r = B u and reroot T to r . Now, for each child c r of r , we attach c r as a child of u . In otherwords, we attach every subtree rooted at a child node of r to u . Finally, we return the resultingdecomposition. Here we prove that Algorithm 2 is sound and complete.Theorem 2.
Let H be a hypergraph and k ≥ an integer. Algorithm 2 called on H with parameter k returns a GHD of H of width ≤ k if and only if ghw ( H ) ≤ k . , Vol. 1, No. 1, Article . Publication date: September 2020. yperBench: A Benchmark and Tool for Hypergraphs and Empirical Findings 15 Algorithm 2:
BalSep
Input:
A hypergraph H . Parameter :
An integer k ≥ Output:
A GHD of H of width ≤ k if it exists, null otherwise. Main Make H globally visible return Decompose( H , ∅ ) Function
Decompose( H ′ : hypergraph, S p : set of special edges ) if | E ( H ′ ∪ S p )| == then return node u with B u ← V ( H ′ ∪ S p ) and λ u ← E ( H ′ ∪ S p ) if | E ( H ′ ∪ S p )| == then Let e , e be the two edges of H ′ ∪ S p Create node u with B u ← e and λ u ← { e } Create node v with B v ← e and λ v ← { e } AttachChild( u , v ) return u ; BalSepIt ← InitBalSepIterator( H , H ′ , S p , k ) while HasNext(
BalSepIt ) do λ u ← Next(
BalSepIt ) B u ← B ( λ u ) subDecomps ← {} foreach c ∈ ComputeSubhypergraphs( H ′ , S p , B u ) do D ←
Decompose( c . H , c . S p ) if D (cid:44) null then subDecomps ← subDecomps ∪ {D} else subDecomps ← null break if subDecomps == null then continue return BuildGHD( B u , λ u , subDecomps ) return nullWe prove the soundness and the completeness of Algorithm 2 separately. Nevertheless, we wantto point out that the main procedure of the algorithm consists of a call to Function Decomposewith input H and an empty set of special edges. Thus, in the next proofs, we will actually provethat Function Decompose called on ( H ′ , S p ) with parameter k returns a GHD of H ′ ∪ S p of width ≤ k if and only if ghw ( H ′ ∪ S p ) ≤ k w.r.t. Definition 8. Note that in case of a hypergraph H and aset of special edges S p = ∅ , Definition 8 coincides with the usual definition of GHD.Proof. (Soundness) We show that if Function Decompose called on ( H ′ , S p ) with parameter k returns a GHD of H ′ ∪ S p of width ≤ k , then such a decomposition actually exists and ghw ( H ′ ∪ S p ) ≤ k . We proceed by induction over the size of H ′ ∪ S p , i.e., | E ( H ′ ∪ S p )| . For the base case, we assume , Vol. 1, No. 1, Article . Publication date: September 2020. Function
BuildGHD( B u , λ u , children ) Input:
A set of vertices B u , a set of λ u , a set of GHDs children . Output:
A GHD with root u and children children . begin Create node u with B u and λ u foreach D ∈ children do Let T be the tree structure of D ˆ r ← Reroot( T , B u ) foreach c ˆ r ∈ Children( ˆ r ) do AttachChild( u , c ˆ r ) return u Function
ComputeSubhypergraphs( H ′ , S p , B u ) Input:
A hypergraph H ′ , a set of special edges S p , a set of vertices B u . Output:
The set of subhypergraphs of H ′ ∪ S p w.r.t. B u . begin V u ← V ( H ′ ) \ B u E u ← { e ∩ V u | e ∈ E ( H ′ ∪ S p )} res ← {} foreach comp ∈ ConnectedComponents( V u , E u ) do c ← initialize pair ( H = null , S p = null ) c . S p ← { s ∈ S p | s ∩ comp (cid:44) ∅} ∪ { B u } E ← { e | e ∈ E ( H ′ ) ∧ e ∩ comp (cid:44) ∅} V ← V ( E ) ∪ V ( c . S p ) c . H ← ( V , E ) res ← res ∪ { c } return res | H ′ ∪ S p | ≤
2. In case | H ′ ∪ S p | =
1, we return a GHD made of a single node whose λ -label consists ofthe only edge H ′ ∪ S p , which also cover all of the vertices of the hypergraph. Such a decompositionhas width 1 and clearly satisfies all the conditions of Definition 8. In case | H ′ ∪ S p | =
2, we createtwo nodes u , v , each one corresponding to an edge of H ′ ∪ S p and we attach v as a child of u . Notethat both u and v are leaf nodes. It is easy to verify that also in this case we return a valid GHD of H ′ ∪ S p .For the induction step, suppose that the recursive function Decompose correctly returns a GHDof H ′ ∪ S p of width ≤ k for each H ′ ∪ S p such that | H ′ ∪ S p | ≤ j , for some j ≥
2. Now suppose that | H ′ ∪ S p | = j + ≥ H ′ , S p ) returns a GHD of H ′ ∪ S p of width ≤ k .We have to show that then there indeed exists such a GHD.Algorithm 2 only returns a GHD in line 27. The program successfully reaches this line only ifthe following happens: • In line 15, a balanced separator λ u of H ′ ∪ S p is chosen. , Vol. 1, No. 1, Article . Publication date: September 2020. yperBench: A Benchmark and Tool for Hypergraphs and Empirical Findings 17 • In line 18, the extended subhypergraphs of H ′ ∪ S p w.r.t. B u = B ( λ u ) are computed; inparticular, each extended subhypergraph corresponds to a [ B u ] -component of H ′ ∪ S p plus anew special edge s = B u . • For each extended subhypergraph c . H ∪ c . S p , the call to Function Decompose in line 19 issuccessful, i.e., it returns a GHD of c . H ∪ c . S p of width ≤ k .We are assuming | H ′ ∪ S p | ≥ λ u is a balanced separator of H ′ ∪ S p . Let C , . . . , C ℓ bethe ℓ [ B u ] -components of H ′ ∪ S p . All extended subhypergraphs C i ∪ { B u } are strictly smallerthan j . Hence, by the induction hypothesis, for each i ∈ { , . . . , ℓ } , there indeed exists a GHD D i = ⟨ T i , ( B i , u ) u ∈ T , ( λ i , u ) u ∈ T ⟩ of width ≤ k for the extended subhypergraph C i ∪ { B u } .It is left to show that Function BuildGHD correctly constructs a GHD of width ≤ k of H ′ ∪ S p .For each D i , let ˆ r i be the node of T i with B i , ˆ r i = B u and λ i , ˆ r i = { B u } . By construction the nodeˆ r i exists in every D i , it is always a leaf and it has the same bag and λ -label everywhere. Let T i = ( N ( T i ) , E ( T i )) be the tree structure of D i and, w.l.o.g., assume that the node sets N ( T i ) arepairwise disjoint. We define the tree structure T = ( N ( T ) , E ( T )) and the functions B u and λ u of D as follows: • N = ( N ( T ) \ { ˆ r }) ∪ · · · ∪ ( N ( T ℓ ) \ { ˆ r ℓ }) ∪ { r } , where r is a new (root) node. • For the definition of E ( T ) recall that each ˆ r i is a leaf node in its decomposition. Let e i denotethe edge between ˆ r i and its parent. Then we define E ( T ) as E ( T ) = ( E ( T ) \ { e }) ∪ · · · ∪( E ( T ℓ ) \ { e ℓ }) ∪ R with R = {[ r , ˆ r ] , . . . , [ r , ˆ r ℓ ]} . • λ r = λ u and B r = B u . • For every v ∈ N \ { r } , there exists exactly one i , such that v ∈ N ( T i ) . We set λ v = λ i , v and B v = B i , v .Intuitively, the GHD D is obtained by taking in each GHD D i the node ˆ r i as the root node andcombining all GHDs D i to a single GHD by merging their root nodes to a single node r . This ispossible since all nodes ˆ r i have the same λ -labels and bags. It is easy to verify that the resultingGHD is indeed a GHD of width ≤ k of the extended subhypergraph H ′ ∪ Sp . □ Proof. (Completeness) For any hypergraph H ′ and set of special edges S p , we prove that if ghw ( H ′ ∪ S p ) ≤ k , then Function Decompose on input ( H ′ , S p ) returns a GHD of width ≤ k of H ′ ∪ S p . Again we proceed by induction on | H ′ ∪ S p | .The base case of | H ′ ∪ S p | ≤ | H ′ ∪ S p | and return it.For the induction step, suppose | H ′ ∪ S p | ≤ j , for some j ≥
2, and ghw ( H ′ ∪ S p ) ≤ k . Then,Function Decompose on input ( H ′ , S p ) returns a GHD of width ≤ k of H ′ ∪ S p . Now assume that | H ′ ∪ S p | = j + ≥ ghw ( H ′ ∪ S p ) ≤ k . We have to show that Function Decompose oninput ( H ′ , S p ) returns a GHD of width ≤ k of H ′ ∪ S p .By Lemma 1, we may assume w.l.o.g., that GHD D = ⟨ T , ( B u ) u ∈ T , ( λ u ) u ∈ T ⟩ is in normal formand that B r is a balanced separator of H ′ ∪ S p for the root node r of D . Let S ⊆ E ( H ) denote the λ -label of r , i.e., λ r = S .When Function Decompose is called on input ( H ′ , S p ) , the while loop in lines 14-27 eventuallygenerates all possible balanced separators of size ≤ k of H ′ ∪ S p , unless it returns on line 27 beforethe end of the loop. Remember that the object BalSepIt not only generates edge separators withedges in E ( H ) , i.e., the original hypergraph on which Algorithm 2 is called, but it also uses edges in f ( H , k ) , i.e., subedges of edges in E ( H ) . Thus, at some point, in line 15, we will choose the separator λ u = S , equivalently, B u = B r .Let C , . . . , C ℓ denote the [ B u ] -components of H ′ ∪ S . Since D is in normal form, we know thatthe root node r has ℓ child nodes such that C i = exCov ( T i ) , where T i is the subtree in T rooted at , Vol. 1, No. 1, Article . Publication date: September 2020. n i for i ∈ { , . . . , ℓ } . Recall from Definition 9 that we write exCov ( T i ) to denote the set of edgesexclusively covered by T i .Now consider the extended subhypergraph C i ∪ { B u } for arbitrary i ∈ { , . . . , ℓ } . Since B r is abalanced separator of H ′ ∪ S p , we have | C i ∪{ B u }| ≤ j . Moreover, there exists a GHD D i of C i ∪{ B u } ,namely the subtree of D induced by the nodes in T i plus r . Hence, by the induction hypothesis,calling Function Decompose with the input corresponding to the extended subhypergraph C i ∪ { B u } returns a valid GHD. This means that, in our call of Function Decompose with input ( H ′ , Sp ) , wehave the following behavior: • On line 19, Function Decompose is called recursively for all extended subhypergraphs C i ∪{ B u } . • Each call of the Function Decompose returns a GHD for the respective extended subhyper-graph. • The results of these recursive calls are collected in line 21 in the variable subDecomps . • Hence, after exiting the loop in lines 18-26, the return statement in line 27 is executed.The call to Function BuildGHD correctly produces the desired GHD as discussed in the soundnessproof. Finally, Function Decompose indeed returns a GHD of width ≤ k of H ′ ∪ S p . □ In this section, we describe
HyperBench - our new benchmark and web tool. We first introduce oursystem and test environment used for the experiments, then we present a new method to extractsimple conjunctive queries from complex SQL queries. Finally, we describe the CQs and CSPs wehave collected.
Our system is composed of two libraries: a C++ library with implementations of the algorithmsand a Java library for processing of SQL queries and hypergraphs. We have extended the algo-rithm
DetKDecomp from [30] and based our program on a new implementation, which we call
NewDetKDecomp . The resulting C++ library contains the implementations of all the algorithmspresented in Sections 4 and 6.5 and comprises around 8500 lines of code. We designed our Javalibrary hg-tools for preprocessing of SQL queries and collecting hypergraph statistics. It uses theopen source libraries
JSqlParser [57] for SQL processing and
JGraphT [44] to deal with graph datastructures. Our two libraries are available at https://github.com/dmlongo/newdetkdecomp andhttps://github.com/dmlongo/hgtools.All the experiments reported in this paper were performed on a cluster of 10 workstations eachrunning Ubuntu 16.04. Every workstation has the same specification and is equipped with two IntelXeon E5-2650 (v4) processors each having 12 cores and 256-GB main memory. Since all algorithmsare single-threaded, we could run several experiments in parallel. For all upcoming runs of ouralgorithms we set a timeout of 3600s.
Since the applicability of structural decomposition methods is limited to conjunctive queries, wehave devised a strategy to transform also more complex queries into (a collection of) hypergraphs.Our program hg-tools takes as input a complex SQL query and produces a collection of simpler SQLqueries. When facing a complex query, e.g., one containing nested queries, the idea is to identifysimple queries, extract them and transform them separately.A conjunctive query can be expressed in SQL as a SELECT-FROM-WHERE statement in whichthe WHERE clause is only allowed to be a conjunction of equality conditions. Moreover, such a , Vol. 1, No. 1, Article . Publication date: September 2020. yperBench: A Benchmark and Tool for Hypergraphs and Empirical Findings 19 query must not contain negation, disjunction and subqueries, i.e., nested SELECT statements. Forour purposes, we neglect the SELECT clause because in our experiments we focus on computingdecompositions instead of answering the queries. Hence, only the hypergraph structure determinedby the FROM and WHERE clauses is important.Query 1 is an example of an SQL query. The FROM clause uses two instances of the relation tab ( a , b , c ) and in the WHERE clause there are some conditions on the two relation instances. Thecondition on line 3 is a join and falls in the scope of conjunctive queries. Though the condition online 4 is not conjunctive, it is just a comparison with a constant value and it does not influence thequery structure. Nevertheless, the condition on line 5 involves a negation, thus Query 1 is not aconjunctive query. This kind of queries is not allowed by our framework. Listing 1. A simple SQL query SELECT ∗2 FROM t a b t 1 , t a b t 23
WHERE t 1 . a = t 2 . a4
AND t 1 . b > 55
AND t 1 . c <> t 2 . c ;Nevertheless, what really matters in defining the query structure are the relationships betweenthe different variables involved in the query, in particular join conditions. For this reason, whenfaced with a query that is not conjunctive, we consider a simplified version that contains theconjunctive core of the original query. In case of Query 1, we drop lines 4-5 and perform ourexperiments on the rest of the query.Real world SQL queries can be rather complicated and, in particular, they can contain nestedSELECT statements, which we simply call subqueries . The presence of subqueries automaticallymakes the query non conjunctive, but we have decided to extract the single queries and analyzetheir conjunctive cores separately. Query 2 is a query with nested SELECT statements. The subqueryon lines 4-6 can be examined separately. The subquery on lines 7-9 cannot be examined separatelybecause it contains a reference to a table defined in an external query and, more precisely, it requiresthe evaluation of the subquery for the value of the current row of that table.
Listing 2. A complex SQL query SELECT ∗2 FROM t a b t 1 , t a b t 23
WHERE t 1 . a = t 2 . a4
AND t 1 . b IN ( SELECT t a b . b5
FROM t a b6
WHERE t a b . c == ' ok ' )7
AND EXISTS ( SELECT ∗8 FROM d i f f e r e n t T a b l e d t9
WHERE d t . a = t 1 . a ) ;In SQL, a subquery can appear in different places in a query. Depending in which statement orcondition it appears, we treat it differently: • If a query is of the form q ◦ · · · ◦ q n , where each q i is a query and ◦ ∈ {∪ , ∩ , \} , we extractthe single queries q i and process them separately. • If a subquery appears in the FROM clause, we convert it into a view (see Section 5.4). , Vol. 1, No. 1, Article . Publication date: September 2020. qs s Fig. 1. Dependency graph of Query 2 • If a subquery contains a reference to an external query, as in lines 7-9 of Query 2, it must bediscarded.
In general, an SQL query can contain subqueries. We want to extract them and either integratethem into the main query, when possible, or analyze them separately. In order to do that in sucha way that the end result closely resembles the original query as much as possible, we build agraph representing the dependencies between the subqueries. At the end of the process, we extractqueries which are independent and eliminate those which are mutually dependent.We say that a subquery s depends on a subquery s if the result of q can be computed only aftercomputing the result of q . The dependency graph of a query Q is a graph G = ( S , D ) where the set S contains the nodes corresponding to subqueries of Q and ( s , s ) ∈ D is an arrow, for s , s ∈ S , if s depends on s . Given a query Q , we create its dependency graph G as follows:(1) Create a node q ∈ S representing the outer query.(2) For each nested query s i of q , create a new node s i ∈ S and an edge ( q , s i ) ∈ D .(3) If s i contains a reference to a table defined in any ancestor s j , create an edge ( s i , s j ) .(4) Recursively examine s i for nested queries.Once we have built the graph, we identify the nodes which are involved in cycles and eliminatethem. In particular, we consider q as a root and navigate the graph. Whenever we find a node havingan edge pointing at an ancestor, we eliminate it together with all of its incoming and outgoingedges. Eventually, we end up with a forest in which we extract a query from each node.The dependency graph G of Query 2 is shown in Figure 1. The node q corresponds to the outerquery and it is the root of the graph. The nodes s and s represent the subqueries in lines 4-6 andlines 7-9 of Query 2, respectively. As the result of Query 2 can be computed only after computingthe subqueries, the two edges ( q , s ) and ( q , s ) are present in the graph. Since s refers to the table t defined in q (line 9), G contains also the edge ( s , q ) . After the creation, we look for cycles inthe graph. In this case, we see that there is no way to evaluate s independently from q . Then, weremove s and all of its incident edges. Finally, we extract a simple query from each node of theremaining graph. Once we have extracted and simplified subqueries, we are left with simple SQL queries of the typeSELECT r i . A j , . . . , r i z . A j z FROM r , . . . , r m WHERE cond (3)such that cond is a conjunction of conditions of the form r i . A = r j . B or r i . A = c , where c is aconstant. Such queries are equivalent to conjunctive queries, thus it is easy to draw a connectionto a CQ and transform it into a hypergraph. Nevertheless, in our case it makes more sense to godirectly from an SQL query to the hypergraph. , Vol. 1, No. 1, Article . Publication date: September 2020. yperBench: A Benchmark and Tool for Hypergraphs and Empirical Findings 21 Let Q be an SQL query of the form (3), then the hypergraph H Q = ( V ( H Q ) , E ( H Q )) correspondingto Q is obtained as follows. We first build the hypergraph induced by the FROM clause. Consider arelation r ( A , . . . , A ℓ ) in the FROM clause. For each attribute A i of r , we create a vertex v A i ∈ V ( H Q ) .Then, we create the edge r = { v A , . . . , v A ℓ } ∈ E ( H Q ) . Now, we modify the hypergraph accordingto the conditions in the WHERE clause. Let cond be such a condition. It can be of two forms: • If cond is of the form r i . A = r j . B , we merge vertices v A and v b and modify their incident edges.W.l.o.g. assume v A itself becomes the merged vertex. For each edge r ∈ { e ∈ E ( H Q ) | v B ∈ e } ,we remove r from E ( H Q ) and add a new edge r ′ = ( r \ { v B }) ∪ { v A } . • If cond is of the form r i . A = c , with c constant, we remove v A from V ( H Q ) and, for each edge r ∈ { e ∈ E ( H Q ) | v A ∈ e } , we remove r from E ( H Q ) and add a new edge r ′ = r \ { v A } .At the end of this procedure, we eliminate empty edges and multiple edges. Also, in our settingSELECT clauses do not contribute to the query structure, thus we simply ignore them.We also take into account logical views. SQL views are virtual tables which are recreated everytime the view is called, thus we have decided to expand the main query by adding the view insideit. Consider Query 3. The view crossView has a cross structure composed by the two relations t , t intersecting in the node b in the middle, see Figure 2 (a). The main query adds two relations t , t which are distinct from the ones in the view crossView and intersect it in four points, thus creatingtwo cycles. The end result is depicted in Figure 2 (b). The resulting query can finally be convertedinto a hypergraph with the algorithm described above. Listing 3. An SQL query with a view AS (2 SELECT t 1 . a a1 , t 1 . c c1 , t 2 . a a2 , t 2 . c c 23
FROM t a b t 1 , t a b t 24
WHERE t 1 . b = t 2 . b5 )6
SELECT ∗7 FROM t a b t 1 , t a b t 2 , c r o s s V i e w c r8
WHERE t 1 . a = c r . a19
AND t 1 . c = c r . a210
AND t 2 . a = c r . c 111
AND t 2 . c = c r . c 2 ;
An important part of our benchmark consists of instances of Constraint Satisfaction Problems.The set we have collected presents different characteristics w.r.t. the ones found in CQs, thus theiranalysis offers a more varied picture of the hypergraphs encountered in applications. For thisreason we have retrieved CSP instances which have a significant practical aspect.The source of most of our CSPs is the website XCSP [7]. XCSP3 is an XML-based format usedto represent constraint satisfaction problems. The language offers a wide variety of options torepresent the most common constraints and frameworks, making it a solid intermediate formatbetween different solvers. They also organize solver competitions for which they use their instancesas a benchmark.From XCSP we have selected a total of 1,953 instances with less than 100 extensional constraintssuch that all constraints are extensional. These instances are divided into CSPs from concrete , Vol. 1, No. 1, Article . Publication date: September 2020. t t (a) Hypergraph of view t t (b) Combined hypergraph Fig. 2. Hypergraphs for Query 3 applications, called
CSP Application in the sequel (1090 instances), and randomly generated CSPs,called
CSP Random below (863 instances).The instances we have fetched from the website are written in well-structured XML files inwhich variables and constraints are explicitly defined through the use of specific XML tags. Thetransformation of these instances into hypergraphs did not require a specific methodology sincethe authors of the XCSP3 format provide an extensive library for parsing the instances where mostof the process is already automatized. Obviously, we still had to convert the object in memoryinto a hypergraph. To this end, we have reimplemented the behavior of some callback methodsin such a way that, whenever the program reads a variable, it adds a vertex to the hypergraph,and, whenever it reads a constraint, it adds an edge containing the vertices corresponding to thevariables affected by the constraint.Our collection of CSPs also includes a third class, which we call
CSP Other
Our benchmark contains 3,648 hypergraphs, which have been converted from CQs and CSPscollected from various sources. Out of these 3,648 hypergraphs, 3,142 hypergraphs have never beenused in a hypertree width analysis before. The hypertree width of 424 CQs and of 82 CSPs has beenanalyzed in [30], [11], and/or [12, 13]. An overview of all instances of CQs and CSPs is given inTable 1. They have been collected from various publicly available benchmarks and repositories ofCQs and CSPs. In the first column, the names of each collection of CQs and CSPs are given togetherwith references where they were first published. In the second column we display the number ofhypergraphs extracted from each collection. The hw of the CQs and CSPs in our benchmark will bediscussed in detail in Section 6.1. To get a first feeling of the hw of the various sources, we mention , Vol. 1, No. 1, Article . Publication date: September 2020. yperBench: A Benchmark and Tool for Hypergraphs and Empirical Findings 23 Table 1. Overview of benchmark instances
Benchmark No. instances hw ≥ C Q s SPARQL [12] 70 (out of 26,157,880) 70Wikidata [13] 354 (out of 273,947) 354LUBM [10, 32] 14 2iBench [6, 10] 40 0Doctors [10, 21] 14 0Deep [10] 41 0JOB (IMDB) [41] 33 7TPC-H [9, 54] 29 1TPC-DS [53] 228 5SQLShare [34] 290 (out of 15,170) 1Random [49] 500 464 C S P s Application [7] 1,090 1,090Random [7] 863 863Other [11, 30] 82 82
Total: 3,648 2,939 the number of cyclic hypergraphs (i.e., those with hw ≥
2) in the last column. When gatheringthe CQs, we proceeded as follows: of the huge benchmark reported in [12], we have only includedCQs, which were detected as having hw ≥ hw ≥ hw ≥
2. Of the big repository reported in [34], we have included those CQs, which are not triviallyacyclic (i.e., they have at least 3 atoms). Of all the small collections of queries, we have included all.It follows a detailed description of the different benchmarks.Our benchmark contains 1113 CQs from five main sources [9, 10, 12, 13, 34] and a set of 500randomly generated queries using the query generator of [49]. In the sequel, we shall refer tothe former queries as
CQ Application , and to the latter as
CQ Random . The CQs analyzed in [12]constitute a big repository of CQs – namely 26,157,880 CQs stemming from SPARQL queries.The queries come from real-users of SPARQL endpoints and their hypertree width was alreadydetermined in [12]. Almost all of these CQs were shown to be acyclic. Our analysis comprises 70CQs from [12], which (apart from few exceptions) are essentially the ones in [12] with hw ≥
2. Inparticular, we have analyzed all 8 CQs with highest hw among the CQs analyzed in [12] (namely, hw = hw ≥ hw = , Vol. 1, No. 1, Article . Publication date: September 2020. the iBench is a tool for generating schemas, constraints, and mappings for data integration tasks.However, in [10], 40 queries were created for tests with the iBench. We therefore refer to thesequeries as iBench-CQs here. In summary, we have incorporated all queries that were eithercontained in the original benchmarks or created/adapted for the tests in [10].The goal of the Join Order Benchmark (JOB) [41] was to evaluate the impact of a good join orderon the performance of query evaluation in standard RDBMSs. Those queries were formulated overthe real-world dataset Internet Movie Database (IMDB). All of the queries have between 3 and 16joins. Clearly, as the goal was to measure the impact of a good join order, those 33 queries are ofhigher complexity, hence 7 out of the 33 queries have hw ≥ ≤ CQ Application hypergraphs. However, even though these size values have been chosensimilarly, the structural properties of the hypergraphs in the two groups
CQ Application and
CQRandom differ significantly, as will become clear from our analysis in Section 6.1.As was detailed in Section 5.5, our benchmark currently contains 2035 hypergraphs from CSPinstances, out of which 1,953 instances were obtained from xcsp.org (see also [7]). These instances,in turn, are divided into CSPs from concrete applications, called
CSP Application in the sequel (1090instances), and randomly generated CSPs, called
CSP Random
CSP Other .Our HyperBench benchmark consists of the CQ and CSP instances converted to hypergraphs. InFigure 3, we show the number of vertices, the number of edges and the arity (i.e., the maximumsize of the edges) as three important metrics of the size of each hypergraph. The smallest are those , Vol. 1, No. 1, Article . Publication date: September 2020. yperBench: A Benchmark and Tool for Hypergraphs and Empirical Findings 25 coming from
CQ Application (most of them have up to 10 edges), while the hypergraphs comingfrom CSPs can be significantly larger (up to 2993 edges). Although some hypergraphs are verybig, more than 50% of all hypergraphs have maximum arity less than 5. In Figure 3 we can easilycompare the different types of hypergraphs, e.g. hypergraphs of arity greater than 20 only exist inthe application classes; the
CSP Other class contains the highest portion of hypergraphs with a bignumber of vertices and edges, etc.
CSP OtherCSP RandomCSP ApplicationCQ RandomCQ Application 0% 25% 50% 75% 100% 1−1011−2021−3031−4041−50>50
Vertices
CSP OtherCSP RandomCSP ApplicationCQ RandomCQ Application 0% 25% 50% 75% 100% 1−1011−2021−3031−4041−50>50
Edges
CSP OtherCSP RandomCSP ApplicationCQ RandomCQ Application 0% 25% 50% 75% 100% 1−56−1011−1516−20>20
Arity
Fig. 3. Hypergraph Sizes
The hypergraphs and the results of our analysis can be accessed through our web tool, availableat http://hyperbench.dbai.tuwien.ac.at.
In this section, we present the empirical results obtained with the HyperBench benchmark. On theone hand, we want to get an overview of the hypertree width of the various types of hypergraphsin our benchmark (cf. Goal 2 in Section 1). On the other hand, we want to find out how realisticthe restriction to low values for certain hypergraph invariants is (cf. Goal 3 stated in Section 1).After this first analysis of the structural properties of the hypergraphs we collected, we perform anevaluation of the different ghw algorithms presented in Section 4. Finally, we propose and evaluatetwo algorithms for computing approximated FHDs. , Vol. 1, No. 1, Article . Publication date: September 2020.
In [19, 25], several invariants of hypergraphs were used to make the problems Check ( GHD , k ) and Check ( FHD , k ) tractable or, at least, easier to approximate. We thus investigate the followingproperties (cf. Definitions 3 – 5): • Deg : the degree of the underlying hypergraph • BIP : the intersection size • c - BMIP : the c -multi-intersection size for c ∈ { , }• VC-dim : the VC-dimensionThe results obtained from computing
Deg , BIP , 3-
BMIP , 4-
BMIP , and
VC-dim for the hypergraphsin the HyperBench benchmark are shown in Table 2.Table 2 has to be read as follows: In the first column, we distinguish different values of thevarious hypergraph metrics. In the columns labeled “Deg“, “BIP“, etc., we indicate for how manyinstances each metric has a particular value. For instance, by the last row in the second column,only 212 non-random CQs have degree >
5. Actually, for most CQs, the degree is less than 10.Moreover, for the BMIP, already with intersections of 3 edges, we get 3-multi-intersection size ≤ ≤ CSP Application and
CSP Other , 543 (46%) hypergraphs have a high degree ( > > ≤
4. For 7 instances the computation of the VC-dimensiontimed out. For all others, the VC-dimension is ≤ > We have systematically applied the hw -computation from [30] to all hypergraphs in the benchmark.The results are summarized in Figure 4. In our experiments, we proceeded as follows. We usedthe same classification of instances we used in the previous experiments, i.e., we distinguish thefollowing classes: CQ Application , CQ Random , CSP Application , CSP Random , and
CSP Other . Forevery hypergraph H , we first tried to solve the Check ( HD , k ) problem for k =
1. In case of
CQApplication , we thus got 673 yes-answers and 440 no-answers. The number in each bar indicates theaverage runtime to find these yes- and no-instances, respectively. Here, the average runtime was“0” (i.e., less than 1 second). For
CQ Random we got 36 yes- and 464 no-instances with an averageruntime below 1 second. For all CSP-instances, we only got no-answers.In the second round, we tried to solve the Check ( HD , k ) problem for k = k =
1. Now the picture is a bit more diverse: 432 of the remaining 440CQs from
CQ Application yielded a yes-answer in less than 1 second. For the hypergraphs stemmingfrom
CQ Random , only 68 instances yielded a yes-answer (in less than 1 second on average), while396 instances yielded a no-answer in less than 7 seconds on average. The hypergraphs relative toCSP offer a different picture. The classes
CSP Application , CSP Random and
CSP Other have 29, 47 , Vol. 1, No. 1, Article . Publication date: September 2020. yperBench: A Benchmark and Tool for Hypergraphs and Empirical Findings 27 k N o . o f i n s t a n ces yes no timeout CQ Application k N o . o f i n s t a n ces yes no timeout CQ Random
0s 0s0s 0s736s 18s707s 13s 0s 0s 0s 0s k N o . o f i n s t a n ces yes no timeout CSP Application
0s 0s1s 0s1319s 42s1332s 59s 0s k N o . o f i n s t a n ces yes no timeout CSP Random
1s 0s219s 0s1257s 0s943s 0s0s 0s 1s 10s 0s 0s k N o . o f i n s t a n ces yes no timeout CSP Other
Fig. 4. HW analysis (labels are avg. runtimes in s) , Vol. 1, No. 1, Article . Publication date: September 2020.
Table 2. Properties of all benchmark instances
CQ Application i Deg BIP 3-BMIP 4-BMIP VC-dim0 0 74 394 647 721 74 721 673 456 4842 320 286 45 10 5573 253 17 1 0 04 181 6 0 0 05 73 9 0 0 0 > CQ Random i Deg BIP 3-BMIP 4-BMIP VC-dim0 0 1 16 49 01 1 17 77 125 202 15 53 90 120 1333 38 62 103 74 2404 31 63 62 42 1065 33 71 47 28 1 > CSP Application i Deg BIP 3-BMIP 4-BMIP VC-dim0 0 0 596 597 01 0 1030 459 486 02 596 59 34 7 10643 1 0 1 0 264 1 0 0 0 05 2 0 0 0 0 > CSP Random i Deg BIP 3-BMIP 4-BMIP VC-dim0 0 0 0 0 01 0 200 200 238 02 0 224 312 407 2203 0 76 147 95 5154 12 181 161 97 575 8 99 14 1 71 > CSP Other i Deg BIP 3-BMIP 4-BMIP VC-dim0 0 0 1 6 01 0 7 36 39 02 1 36 23 16 503 5 29 20 21 254 19 10 2 0 05 4 0 0 0 0 > CSP Other give rise to a timeout (i.e., theprogram did not terminate within 3,600 seconds), while all the other instances give a no-answerwithin the timeout. Interestingly, the our hw -algorithm gives a no-answer for 1877 instances of CSP Application and
CSP Random in less than 1 second, while it took the algorithm 219 seconds on , Vol. 1, No. 1, Article . Publication date: September 2020. yperBench: A Benchmark and Tool for Hypergraphs and Empirical Findings 29 average to answer “no" for 55 instances of
CSP Other . This shows that the class
CSP Other containsinstances which are difficult to decompose.This procedure is iterated by incrementing k and running the hw -computation for all instances,that either yielded a no-answer or a timeout in the previous round. For instance, for queries from CQ Application , one further round is needed after the second round. In other words, we confirmthe observation of low hw , which was already made for CQs of arity ≤ CQ Random (resp. CSPs), 396 (resp. 1940) instances are left in the thirdround, of which 70 (resp. 232) yield a yes-answer in less than 1 second on average, 326 (resp. 1415)instances yield a no-answer in 32 (resp. 988) seconds on average and no (resp. 293) instances yielda timeout. Note that, as we increase k , the average runtime and the percentage of timeouts firstincrease up to a certain point and then they decrease. This is due to the fact that, as we increase k ,the number of combinations of edges to be considered in each λ -label (i.e., the function λ u at eachnode u of the decomposition) increases. In principle, we have to test O( n k ) combinations, where n is the number of edges. However, if k increases beyond a certain point, then it gets easier to “guess”a λ -label since an increasing portion of the O( n k ) possible combinations leads to a solution (i.e., anHD of desired width).To answer the question in Goal 2 , it is indeed the case that for a big number of instances, thehypertree width is small enough to allow for efficient evaluation of CQs or CSPs: all instancesof non-random CQs have hw ≤ hw ≤
5. In total, including random CQs, 2,427 (66.5%) out of 3,648 instances have hw ≤ hw . Finally, we have analyzed the pairwise correlation between all properties. Of course, the differentintersection sizes (BIP, 3-BMIP, 4-BMIP) are highly correlated. Other than that, we observe quitea strong correlation of the arity with the number of vertices and the hypertree width. Moreover,there is a significant correlation between number of vertices and arity and between number ofvertices and hypertree width. Clearly, the correlation between arity and hypertree width is mainlydue to the CSP instances and the random CQs since, for non-random CQs, the hw never increasesbeyond 3, independently of the arity.A graphical presentation of all pairwise correlations is given in Figure 5. Here, large, darkcircles indicate a high correlation, while small, light circles stand for low correlation. Blue circlesindicate a positive correlation while red circles stand for a negative correlation. In [19], it hasbeen argued that Deg, BIP, 3-BMIP, 4-BMIP and VC-dim are non-trivial restrictions to achievetractability. It is interesting to note that, according to the correlations shown in Figure 5, theseproperties have almost no impact on the hypertree width of our hypergraphs. This underlines theusefulness of these restrictions in the sense that (a) they make the GHD computation and FHDapproximation easier [19] but (b) low values of degree, (multi-)intersection-size, or VC-dimensiondo not pre-determine low values of the widths. ghw Algorithms
Here we report on empirical results for the three ghw -algorithms described in Section 4. We haverun the programs on each hypergraph from the HyperBench up to hypertree width 6, trying to geta smaller ghw than hw . We have thus run the ghw -algorithms with the following parameters: forall hypergraphs H with hw ( H ) = k (or hw ≤ k and, due to timeouts, we do not know if hw ≤ k − k ∈ { , , , } , try to solve the Check ( GHD , k − ) problem. In other words, we just , Vol. 1, No. 1, Article . Publication date: September 2020. l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l −1−0.8−0.6−0.4−0.200.20.40.60.81 v e r t i c e s edge s a r i t y deg r ee b i p 3 − B M I P − B M I P V C − D i m H W verticesedgesaritydegreebip3−BMIP4−BMIPVC−DimHW Fig. 5. Correlation analysisTable 3. GHW algorithms with avg. runtimes in s hw → ghw Total GlobalBIP LocalBIP BalSep yes (s) no (s) yes (s) no (s) yes (s) no (s) → → → → hw ( H ) ∈ { , } , no improvement is possible since, inthis case, hw ( H ) = ghw ( H ) holds.In Table 3, for each algorithm, we report on the number of “successful” attempts to solve theCheck ( GHD , k − ) problem for hypergraphs with hw = k . Here “successful” means that theprogram terminated within 1 hour. For instance, for the 310 hypergraphs with hw = GlobalBIP terminated in 128 cases (i.e., 41%) when trying to solve Check ( GHD , ) .The average runtime of these “successful” runs was 537 seconds. For the 386 hypergraphs with hw = GlobalBIP terminated in 137 cases (i.e., 35%) with average runtime 2809 when trying tosolve the Check ( GHD , ) problem. For the 886 hypergraphs with hw ∈ { , } , GlobalBIP onlyterminated in 13 cases (i.e., 1.4%). Overall, it turns out that the set f ( H , k ) may be very big (eventhough it is polynomial if k and i are constants). Hence, H ′ can become considerably bigger than H . This explains the frequent timeouts in the GlobalBIP column in Table 3.The results obtained with
LocalBIP are shown in the corresponding column. Interestingly, forthe hypergraphs with hw =
3, the “local” computation performs significantly better (namely 63%solved with average runtime 162 seconds rather than 41% with average runtime 537 seconds). Incontrast, for the hypergraphs with hw =
4, the “global” computation is significantly more successful.For hw ∈ { , } , the “global” and “local” computations are equally bad. A possible explanation for , Vol. 1, No. 1, Article . Publication date: September 2020. yperBench: A Benchmark and Tool for Hypergraphs and Empirical Findings 31 Table 4. GHW of instances with average runtime in s hw → ghw yes no timeout3 → → → → hw = hw = f ( H , k ) of subedges to the “local” set f u ( H , k ) at each node u seems to be quite effective for the hypergraphs with hw =
3. In contrast, the additional cost ofhaving to compute f u ( H , k ) at each node u becomes counter-productive, when the set of subedgesthus eliminated is not significant. It is interesting to note that the sets of solved instances of theglobal computation and the local computation are incomparable, i.e., in some cases one method isbetter, while in other cases the other method is better.If we look at the number of solved instances in Table 3, we see that the recursive algorithm viabalanced separators (reported in the last column labeled BalSep ) has the least number of timeoutsdue to the fast identification of negative instances (i.e., those with no-answer), where it oftendetects quite fast that a given hypergraph does not have a balanced separator of desired width. As k increases, the performance of the balanced separators approach deteriorates. This is due to k inthe exponent of the running time of our algorithm, i.e. we need to check for each of the possible O( n k + ) combinations of ≤ k edges if it constitutes a balanced separator. Note that the balancedseparators approach only terminated in case of no-answers.We now look at Table 4, where we report for all hypergraphs with hw ≤ k and k ∈ { , , , } ,whether ghw ≤ k − GlobalBIP , LocalBIP and
BalSep ) in parallel and stop the computation, as soon as one terminates (with answer“yes” or “no”). The number in parentheses refers to the average runtime needed by the fastest ofthe three algorithms in each case. A timeout occurs if none of the three algorithms terminateswithin 3,600 seconds. It is interesting to note that in the vast majority of cases, no improvementof the width is possible when we switch from hw to ghw : in 97% of the solved cases with hw ≤ hw and ghw have identical values. Actually, we think that thehigh percentage of the solved cases gives a more realistic picture than the percentage of all cases forthe following reason: our algorithms (in particular, the “global” and “local” computations) needparticularly long time for negative instances. This is due to the fact that in a negative case, “all”possible choices of λ -labels for a node u in the GHD have to be tested before we can be sure that noGHD of H (or, equivalently, no HD of H ′ ) of desired width exists. Hence, it seems plausible that thetimeouts are mainly due to negative instances. This also explains why our new BalSep algorithm,which is particularly well suited for negative instances, has the least number of timeouts.We conclude this section with a final observation: in Figure 4, we had many cases, for which onlysome upper bound k on the hw could be determined, namely those cases, where the attempt tosolve Check ( HD , k ) yields a yes-answer and the attempt to solve Check ( HD , k − ) gives a timeout.In several such cases, we could get (with the balanced separator approach) a no-answer for theCheck ( GHD , k − ) problem, which implicitly gives a no-answer for the problem Check ( HD , k − ) .In this way, our new ghw -algorithm is also profitable for the hw -computation: for 827 instances with hw ≤
6, we were not able to determine the exact hypertree width. Using our new ghw -algorithm,we closed this gap for 297 instances; for these instances hw = ghw holds. , Vol. 1, No. 1, Article . Publication date: September 2020. To sum up, we now have a total of 2,356 (64.5%) instances for which we determined the exact hw and a total of 1,984 instances (54.4%) for which we determined the exact ghw . Out of these, 1,968instances had identical values for hw and ghw . In 16 cases, we found an improvement of the widthby 1 when moving from hw to ghw , namely from hw = ghw =
5. In 2 further cases, we couldshow hw ≤ ghw ≤
5, but the attempt to check hw = ghw = Goal 6 , hw is equal to ghw in 54.4% of the cases if we consider all instances and in68.2% of the cases (1,968 of 2,886) with small width ( hw ≤ hw and ghw ), then hw and ghw coincide in99.2% of the cases (1,968 of 1,984). The algorithms for computing FHDs in the literature are very expensive and even the tractability re-sult presented in [19] involves a double exponential “constant”. Here we propose two algorithms forcomputing an FHD of a hypergraph when we already have a GHD:
ImproveHD , and
FracImproveHD .They differ in the compromise between computational cost and quality of the approximation.The first algorithm we present is based on a simple observation: given an (G)HD, we couldsubstitute its integral edge covers with fractional edge covers and obtain an FHD. Formally, let D = ⟨ T , ( B u ) u ∈ T , ( λ u ) u ∈ T ⟩ be either a GHD or an HD. Our algorithm ImproveHD computes an FHD D ′ = ⟨ T ′ , ( B ′ u ) u ∈ T ′ , ( γ u ) u ∈ T ′ ⟩ where: • T ′ = T . • ∀ u ∈ T ′ , B ′ u = B u . • ∀ u ∈ T ′ , γ u is a fractional edge cover of B ′ u .To obtain such D ′ , it is sufficient to iterate over the nodes of D and substitute λ u with γ u . Sincecomputing a fractional edge cover is polynomial and we assume to have already computed an HDto start with, the whole algorithm is very efficient. Nevertheless, it is clear that there is a strongdependence on the starting HD. This is unsatisfactory and so we devised a more sophisticatedalgorithm.The algorithm we describe here gets rid of the dependence on a particular HD and computesa fractionally improved (G)HD with a fixed improvement threshold. given a hypergraph H andtwo numbers k , k ′ ≥
1, where k is an upper bound on the hw and k ′ the desired fractionallyimproved hw . We search for an FHD D ′ with D ′ = SimpleImproveHD (D) for some HD D of H with width (D) ≤ k and width (D ′ ) ≤ k ′ . In other words, this algorithm searches for the bestfractionally improved HD over all HDs of width ≤ k . Hence, the result is independent of anyconcrete HD.The experimental results with these algorithms for computing fractionally improved HDs aresummarized in Tables 5 and 6. We have applied these algorithms to all hypergraphs for which hw ≤ k with k ∈ { , , , , } is known from Figure 4. The various columns of the Tables 5 and 6are as follows: the first column (labeled hw ) refers to the (upper bound on the) hw according toFigure 4. The next 3 columns, labeled ≥ [ . , ) , and [ . , . ) tell us, by how much the width canbe improved (if at all) if we compute an FHD by one of the two algorithms. We thus distinguish the3 cases if, for a hypergraph of hw ≤ k , we manage to construct an FHD of width k − c for c ≥ c ∈ [ . , ) , or c ∈ [ . , . ) . The column with label “no” refers to the cases where no improvementat all or at least no improvement by c ≥ . ImproveHD algorithm and startingfrom the HD obtained by the hw -computation of Figure 4) out of 595 hypergraphs with hw =
2, noimprovement was possible in 419 cases. In the remaining 176 cases, an improvement to a width of , Vol. 1, No. 1, Article . Publication date: September 2020. yperBench: A Benchmark and Tool for Hypergraphs and Empirical Findings 33
Table 5. Instances solved with
ImproveHD hw ≥ [ . , ) [ . , . ) no timeout2 0 136 40 419 03 12 104 25 169 04 9 55 11 311 05 20 14 11 382 06 12 60 80 307 0 Table 6. Instances solved with
FracImproveHD hw ≥ [ . , ) [ . , . ) no timeout2 0 194 46 353 23 14 116 21 135 244 11 81 2 8 2845 18 126 59 2 2226 28 149 95 4 183at most 2 − . k − c with c ∈ [ . , . ) was possiblein 136 cases. For the hypergraphs with hw = c ∈ [ . , ) and 12 even by at least1. The improvements achieved for the hypergraphs with hw ≤ hw ≤ FracImproveHD implementation are displayed in Table 6. We seethat the number of hypergraphs which allow for a fractional improvement of the width by at least0.5 or even by 1 is often bigger than with
ImproveHD – in particular in the cases where k ′ ≤ k with k ∈ { , } holds. In the other cases, the results obtained with the naive ImproveHD algorithm arenot much worse than with the more sophisticated
FracImproveHD algorithm.
In this work, we have presented HyperBench, a new and comprehensive benchmark of hypergraphsderived from CQs and CSPs from various areas, together with the results of extensive empiricalanalyses with this benchmark.
Lessons learned.
The empirical study has brought many insights. Below, we summarize the mostimportant lessons learned from our studies. • The finding of [12, 13, 48] that non-random CQs have low hypertree width has been confirmedby our analysis, even if (in contrast to SPARQL queries) the arity of the CQs is not bounded by 3.For random CQs and CSPs, we have detected a correlation between the arity and the hypertreewidth, although also in this case, the increase of the hw with increased arity is not dramatic. • In [19], several hypergraph invariants were identified, which make the computation of GHDsand the approximation of FHDs tractable. We have seen that, at least for non-random instances,these invariants indeed tend to have low values. • The reduction of the ghw -computation problem to the hw -computation problem in case of lowintersection size turned out to be more problematical than the theoretical tractability results from[19] had suggested. Even the improvement by “local” computation of the additional subedges didnot help much. However, we were able to improve this significantly by presenting a new algorithm , Vol. 1, No. 1, Article . Publication date: September 2020. based on “balanced separators”. In particular for negative instances (i.e., those with a no-answer),this approach proved very effective. • An additional benefit of the new ghw -algorithm based on “balanced separators” is that itallowed us to also fill gaps in the hw -computation. Indeed, in several cases, we managed to verify hw ≤ k for some k but we could not show hw ≰ k −
1, due to a timeout for Check ( HD , k − ) . Byestablishing ghw ≰ k − hw ≰ k − hw of many further hypergraphs. • Most surprisingly, the discrepancy between hw and ghw is much lower than expected. Theo-retically, only the upper bound hw ≤ · ghw + hw ≤
6, we could show that in circa 54% of all cases, hw and ghw are simplyidentical. Moreover, in all cases when one of our implementations of ghw -computation terminatedon instances with hw ≤
5, we got identical values for hw and ghw . Future work.
Our empirical study has also given us many hints for future directions of research.We find the following tasks particularly urgent and/or rewarding. • So far, we have only implemented the ghw -computation in case of low intersection size.In [19], tractability of the Check ( GHD , k ) problem was also proved for the more relaxed boundedmulti-intersection size. Our empirical results in Table 2 show that, apart from the random CQs andrandom CSPs, the 3-multi-intersection size is ≤ • The three approaches for ghw -computation presented here turned out to have complementarystrengths and weaknesses. This was profitable when running all three algorithms in parallel andtaking the result of the first one that terminates (see Table 4). In the future, we also want toimplement a more sophisticated combination of the various approaches: for instance, one couldtry to apply our new “balanced separator” algorithm recursively only down to a certain recursiondepth (say depth 2 or 3) to split a big given hypergraph into smaller subhypergraphs and thencontinue with the “global” or “local” computation from Section 4. First promising results in thisdirection have recently been obtained in [29]. • Our new approach to ghw -computation via “balanced separators” proved quite effective in ourexperiments. However, further theoretical underpinning of this approach is missing. The empiricalresults obtained for our new GHD algorithm via balanced separators suggest that the number ofbalanced separators is often drastically smaller than the number of arbitrary separators. We want todetermine a realistic upper bound on the number of balanced separators in terms of n (the numberof edges) and k (an upper bound on the width). This will then allow us to compute also a realisticupper bound on the runtime of this new algorithm. • We want to further extend the HyperBench benchmark and tool in several directions. We willthus incorporate further implementations of decomposition algorithms from the literature such asthe GHD- and FHD computation in [45] or the polynomial-time FHD computation for hypergraphsof bounded multi-intersection size in [25]. Moreover, we will continue to fill in hypergraphsfrom further sources of CSPs and CQs. For instance, in [1, 14, 22, 23] a collection of CQs for theexperimental evaluations in those papers is mentioned. We will invite the authors to disclose theseCQs and incorporate them into the HyperBench benchmark. • Finally, we want to make use of HyperBench to test the practical feasibility of using decompo-sitions to evaluate CQs and solving CSPs. To this end, we will extend our collection of hypergraphswith the data of these problems. In other words, we want to include in our benchmark the relationscorresponding to the database for CQs and constraints for CSPs. Such a study would have as a , Vol. 1, No. 1, Article . Publication date: September 2020. yperBench: A Benchmark and Tool for Hypergraphs and Empirical Findings 35 primary goal to assess the usefulness of decompositions in solving related problems, but it wouldalso help identify which characteristics should decompositions have to serve this purpose.
ACKNOWLEDGMENTS
This work was supported by the Austrian Science Fund (FWF):P30930-N35 in the context of theproject “HyperTrac”. Georg Gottlob is a Royal Society Research Professor and acknowledges supportby the Royal Society for the present work in the context of the project "RAISON DATA" (Projectreference: RP\R1\201074). Davide Mario Longo’s work was also supported by the FWF projectW1255-N23.We would like to thank Angela Bonifati, Wim Martens, and Thomas Timm for sharing most ofthe hypergraphs with hw ≥ REFERENCES [1] Christopher R. Aberger, Andrew Lamb, Susan Tu, Andres Nötzli, Kunle Olukotun, and Christopher Ré. 2017.EmptyHeaded: A Relational Engine for Graph Processing.
ACM Trans. Database Syst.
42, 4 (2017), 20:1–20:44.https://doi.org/10.1145/3129246[2] Christopher R. Aberger, Susan Tu, Kunle Olukotun, and Christopher Ré. 2016. Old Techniques for New Join Algorithms:A Case Study in RDF Processing.
CoRR abs/1602.03557 (2016). arXiv:1602.03557 http://arxiv.org/abs/1602.03557[3] Isolde Adler, Georg Gottlob, and Martin Grohe. 2007. Hypertree width and related hypergraph invariants.
Eur. J. Comb.
28, 8 (2007), 2167–2181. https://doi.org/10.1016/j.ejc.2007.04.013[4] Kamal Amroun, Zineb Habbas, and Wassila Aggoune-Mtalaa. 2016. A compressed Generalized HypertreeDecomposition-based solving technique for non-binary Constraint Satisfaction Problems.
AI Commun.
29, 2 (2016),371–392. https://doi.org/10.3233/AIC-150694[5] Molham Aref, Balder ten Cate, Todd J. Green, Benny Kimelfeld, Dan Olteanu, Emir Pasalic, Todd L. Veldhuizen, andGeoffrey Washburn. 2015. Design and Implementation of the LogicBlox System. In
Proceedings of the 2015 ACMSIGMOD International Conference on Management of Data, Melbourne, Victoria, Australia, May 31 - June 4, 2015 , Timos K.Sellis, Susan B. Davidson, and Zachary G. Ives (Eds.). ACM, 1371–1382. https://doi.org/10.1145/2723372.2742796[6] Patricia C. Arocena, Boris Glavic, Radu Ciucanu, and Renée J. Miller. 2015. The iBench Integration Metadata Generator.
PVLDB
PVLDB
6, 14 (2013), 1990–2001. https://doi.org/10.14778/2556549.2556579[9] Michael Benedikt. 2017. CQ benchmarks. Personal Communication.[10] Michael Benedikt, George Konstantinidis, Giansalvatore Mecca, Boris Motik, Paolo Papotti, Donatello Santoro, andEfthymia Tsamoura. 2017. Benchmarking the Chase. In
Proceedings of the 36th ACM SIGMOD-SIGACT-SIGAI Symposiumon Principles of Database Systems, PODS 2017, Chicago, IL, USA, May 14-19, 2017 , Emanuel Sallinger, Jan Van den Bussche,and Floris Geerts (Eds.). ACM, 37–52. https://doi.org/10.1145/3034786.3034796[11] Jeremias Berg, Neha Lodha, Matti Järvisalo, and Stefan Szeider. 2017. MaxSAT Benchmarks based on DeterminingGeneralized Hypertree-width.
MaxSAT Evaluation 2017: Solver and Benchmark Descriptions
B-2017-2 (2017), 22.[12] Angela Bonifati, Wim Martens, and Thomas Timm. 2017. An Analytical Study of Large SPARQL Query Logs.
PVLDB
11, 2 (2017), 149–161. https://doi.org/10.14778/3149193.3149196[13] Angela Bonifati, Wim Martens, and Thomas Timm. 2019. Navigating the Maze of Wikidata Query Logs. In
TheWorld Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019 , Ling Liu, Ryen W. White, AminMantrach, Fabrizio Silvestri, Julian J. McAuley, Ricardo Baeza-Yates, and Leila Zia (Eds.). ACM, 127–138. https://doi.org/10.1145/3308558.3313472[14] Nofar Carmeli, Batya Kenig, and Benny Kimelfeld. 2017. Efficiently Enumerating Minimal Triangulations. In
Proceedingsof the 36th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems, PODS 2017, Chicago, IL, USA,May 14-19, 2017 , Emanuel Sallinger, Jan Van den Bussche, and Floris Geerts (Eds.). ACM, 273–287. https://doi.org/10.1145/3034786.3056109[15] Ashok K. Chandra and Philip M. Merlin. 1977. Optimal Implementation of Conjunctive Queries in Relational Data Bases.In
Proceedings of the 9th Annual ACM Symposium on Theory of Computing, May 4-6, 1977, Boulder, Colorado, USA , John E.Hopcroft, Emily P. Friedman, and Michael A. Harrison (Eds.). ACM, 77–90. https://doi.org/10.1145/800105.803397, Vol. 1, No. 1, Article . Publication date: September 2020. [16] Rina Dechter. 2003.
Constraint Processing . Elsevier. https://doi.org/10.1016/b978-1-55860-890-0.x5000-2[17] Johannes Klaus Fichte, Markus Hecher, Neha Lodha, and Stefan Szeider. 2018. An SMT Approach to FractionalHypertree Width. In
Principles and Practice of Constraint Programming - 24th International Conference, CP 2018, Lille,France, August 27-31, 2018, Proceedings (Lecture Notes in Computer Science, Vol. 11008) , John N. Hooker (Ed.). Springer,109–127. https://doi.org/10.1007/978-3-319-98334-9_8[18] Wolfgang Fischl, Georg Gottlob, Davide Mario Longo, and Reinhard Pichler. 2019. HyperBench: A Benchmark andTool for Hypergraphs and Empirical Findings. In
Proceedings of the 38th ACM SIGMOD-SIGACT-SIGAI Symposium onPrinciples of Database Systems, PODS 2019, Amsterdam, The Netherlands, June 30 - July 5, 2019 , Dan Suciu, SebastianSkritek, and Christoph Koch (Eds.). ACM, 464–480. https://doi.org/10.1145/3294052.3319683[19] Wolfgang Fischl, Georg Gottlob, and Reinhard Pichler. 2018. General and Fractional Hypertree Decompositions:Hard and Easy Cases. In
Proceedings of the 37th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of DatabaseSystems, Houston, TX, USA, June 10-15, 2018 , Jan Van den Bussche and Marcelo Arenas (Eds.). ACM, 17–32. https://doi.org/10.1145/3196959.3196962[20] Tobias Ganzow, Georg Gottlob, Nysret Musliu, and Marko Samer. 2005. A CSP hypergraph library. In
Institute ofInformation Systems (DBAI) .[21] Floris Geerts, Giansalvatore Mecca, Paolo Papotti, and Donatello Santoro. 2014. Mapping and cleaning. In
IEEE30th International Conference on Data Engineering, Chicago, ICDE 2014, IL, USA, March 31 - April 4, 2014 , Isabel F.Cruz, Elena Ferrari, Yufei Tao, Elisa Bertino, and Goce Trajcevski (Eds.). IEEE Computer Society, 232–243. https://doi.org/10.1109/ICDE.2014.6816654[22] Lucantonio Ghionna, Luigi Granata, Gianluigi Greco, and Francesco Scarcello. 2007. Hypertree Decompositions forQuery Optimization. In
Proceedings of the 23rd International Conference on Data Engineering, ICDE 2007, The MarmaraHotel, Istanbul, Turkey, April 15-20, 2007 , Rada Chirkova, Asuman Dogac, M. Tamer Özsu, and Timos K. Sellis (Eds.).IEEE Computer Society, 36–45. https://doi.org/10.1109/ICDE.2007.367849[23] Lucantonio Ghionna, Gianluigi Greco, and Francesco Scarcello. 2011. H-DB: a hybrid quantitative-structural sqloptimizer. In
Proceedings of the 20th ACM Conference on Information and Knowledge Management, CIKM 2011, Glasgow,United Kingdom, October 24-28, 2011 , Craig Macdonald, Iadh Ounis, and Ian Ruthven (Eds.). ACM, 2573–2576. https://doi.org/10.1145/2063576.2064023[24] Georg Gottlob, Gianluigi Greco, Nicola Leone, and Francesco Scarcello. 2016. Hypertree Decompositions: Questionsand Answers. In
Proceedings of the 35th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems,PODS 2016, San Francisco, CA, USA, June 26 - July 01, 2016 , Tova Milo and Wang-Chiew Tan (Eds.). ACM, 57–74.https://doi.org/10.1145/2902251.2902309[25] Georg Gottlob, Matthias Lanzinger, Reinhard Pichler, and Igor Razgon. 2020. Complexity Analysis of Generalized andFractional Hypertree Decompositions.
CoRR abs/2002.05239 (2020). arXiv:2002.05239 https://arxiv.org/abs/2002.05239[26] Georg Gottlob, Nicola Leone, and Francesco Scarcello. 1999. On Tractable Queries and Constraints. In
Database andExpert Systems Applications, 10th International Conference, DEXA ’99, Florence, Italy, August 30 - September 3, 1999,Proceedings (Lecture Notes in Computer Science, Vol. 1677) , Trevor J. M. Bench-Capon, Giovanni Soda, and A Min Tjoa(Eds.). Springer, 1–15. https://doi.org/10.1007/3-540-48309-8_1[27] Georg Gottlob, Nicola Leone, and Francesco Scarcello. 2002. Hypertree Decompositions and Tractable Queries.
J.Comput. Syst. Sci.
64, 3 (2002), 579–627. https://doi.org/10.1006/jcss.2001.1809[28] Georg Gottlob, Zoltán Miklós, and Thomas Schwentick. 2009. Generalized hypertree decompositions: NP-hardnessand tractable variants.
J. ACM
56, 6 (2009), 30:1–30:32. https://doi.org/10.1145/1568318.1568320[29] Georg Gottlob, Cem Okulmus, and Reinhard Pichler. 2020. Fast and Parallel Decomposition of Constraint SatisfactionProblems. In
Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020[scheduled for July 2020, Yokohama, Japan, postponed due to the Corona pandemic] , Christian Bessiere (Ed.). ijcai.org,1155–1162. https://doi.org/10.24963/ijcai.2020/161[30] Georg Gottlob and Marko Samer. 2008. A backtracking-based algorithm for hypertree decomposition.
ACM Journal ofExperimental Algorithmics
13 (2008), 1:1.1–1:1.19. https://doi.org/10.1145/1412228.1412229[31] Martin Grohe and Dániel Marx. 2014. Constraint Solving via Fractional Edge Covers.
ACM Trans. Algorithms
11, 1(2014), 4:1–4:20. https://doi.org/10.1145/2636918[32] Yuanbo Guo, Zhengxiang Pan, and Jeff Heflin. 2005. LUBM: A benchmark for OWL knowledge base systems.
J. WebSemant.
3, 2-3 (2005), 158–182. https://doi.org/10.1016/j.websem.2005.06.005[33] Zineb Habbas, Kamal Amroun, and Daniel Singer. 2015. A Forward-Checking algorithm based on a GeneralisedHypertree Decomposition for solving non-binary constraint satisfaction problems.
J. Exp. Theor. Artif. Intell.
27, 5(2015), 649–671. https://doi.org/10.1080/0952813X.2014.993507[34] Shrainik Jain, Dominik Moritz, Daniel Halperin, Bill Howe, and Ed Lazowska. 2016. SQLShare: Results from a Multi-YearSQL-as-a-Service Experiment. In
Proceedings of the 2016 International Conference on Management of Data, SIGMODConference 2016, San Francisco, CA, USA, June 26 - July 01, 2016 , Fatma Özcan, Georgia Koutrika, and Sam Madden, Vol. 1, No. 1, Article . Publication date: September 2020. yperBench: A Benchmark and Tool for Hypergraphs and Empirical Findings 37 (Eds.). ACM, 281–293. https://doi.org/10.1145/2882903.2882957[35] Shant Karakashian, Robert J. Woodward, and Berthe Y. Choueiry. 2011. Reformulating R(*, m)C with Tree Decomposition.In
Proceedings of the Ninth Symposium on Abstraction, Reformulation, and Approximation, SARA 2011, Parador deCardona, Cardona, Catalonia, Spain, July 17-18, 2011.
ACM Trans. Database Syst.
41, 4 (2016), 22:1–22:45. https://doi.org/10.1145/2967101[37] Mahmoud Abo Khamis, Hung Q. Ngo, and Atri Rudra. 2016. FAQ: Questions Asked Frequently. In
Proceedings of the35th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems, PODS 2016, San Francisco, CA, USA,June 26 - July 01, 2016 , Tova Milo and Wang-Chiew Tan (Eds.). ACM, 13–28. https://doi.org/10.1145/2902251.2902280[38] Tuukka Korhonen. 2019. Potential Maximal Cliques Parameterized by Edge Clique Cover. arXiv:1912.10989 [cs.DS][39] Mohammed Lalou, Zineb Habbas, and Kamal Amroun. 2009. Solving Hypertree Structured CSP: Sequential andParallel Approaches. In
Proceedings of the 16th RCRA workshop on Experimental Evaluation of Algorithms for SolvingProblems with Combinatorial Explosion, RCRA@AI*IA 2009, Reggio Emilia, Italy, December 11-12, 2009 (CEUR WorkshopProceedings, Vol. 589) , Marco Gavanelli and Toni Mancini (Eds.). CEUR-WS.org. http://ceur-ws.org/Vol-589/paper11.pdf[40] Viktor Leis, Andrey Gubichev, Atanas Mirchev, Peter A. Boncz, Alfons Kemper, and Thomas Neumann. 2015. HowGood Are Query Optimizers, Really?
PVLDB
9, 3 (2015), 204–215. https://doi.org/10.14778/2850583.2850594[41] Viktor Leis, Bernhard Radke, Andrey Gubichev, Atanas Mirchev, Peter A. Boncz, Alfons Kemper, and Thomas Neumann.2018. Query optimization through the looking glass, and what we found running the Join Order Benchmark.
VLDB J.
27, 5 (2018), 643–668. https://doi.org/10.1007/s00778-017-0480-7[42] Stanislav Malyshev, Markus Krötzsch, Larry González, Julius Gonsior, and Adrian Bielefeldt. 2018. Getting the MostOut of Wikidata: Semantic Technology Usage in Wikipedia’s Knowledge Graph. In
The Semantic Web - ISWC 2018 -17th International Semantic Web Conference, Monterey, CA, USA, October 8-12, 2018, Proceedings, Part II (Lecture Notes inComputer Science, Vol. 11137) , Denny Vrandecic, Kalina Bontcheva, Mari Carmen Suárez-Figueroa, Valentina Presutti,Irene Celino, Marta Sabou, Lucie-Aimée Kaffee, and Elena Simperl (Eds.). Springer, 376–394. https://doi.org/10.1007/978-3-030-00668-6_23[43] Dániel Marx. 2010. Approximating fractional hypertree width.
ACM Trans. Algorithms
6, 2 (2010), 29:1–29:17.https://doi.org/10.1145/1721837.1721845[44] Dimitrios Michail, Joris Kinable, Barak Naveh, and John V. Sichi. 2020. JGraphT—A Java Library for Graph DataStructures and Algorithms.
ACM Trans. Math. Softw.
46, 2, Article 16 (May 2020), 29 pages.[45] Lukas Moll, Siamak Tazari, and Marc Thurley. 2012. Computing hypergraph width measures exactly.
Inf. Process. Lett.
ACM Trans.Database Syst.
40, 1 (2015), 2:1–2:44. https://doi.org/10.1145/2656335[47] Adam Perelman and Christopher Ré. 2015. DunceCap: Compiling Worst-Case Optimal Query Plans. In
Proceedingsof the 2015 ACM SIGMOD International Conference on Management of Data, Melbourne, Victoria, Australia, May 31 -June 4, 2015 , Timos K. Sellis, Susan B. Davidson, and Zachary G. Ives (Eds.). ACM, 2075–2076. https://doi.org/10.1145/2723372.2764945[48] François Picalausa and Stijn Vansummeren. 2011. What are real SPARQL queries like?. In
Proceedings of the InternationalWorkshop on Semantic Web Information Management, SWIM 2011, Athens, Greece, June 12, 2011 , Roberto De Virgilio,Fausto Giunchiglia, and Letizia Tanca (Eds.). ACM, 7. https://doi.org/10.1145/1999299.1999306[49] Rachel Pottinger and Alon Y. Halevy. 2001. MiniCon: A scalable algorithm for answering queries using views.
VLDB J.
10, 2-3 (2001), 182–198. https://doi.org/10.1007/s007780100048[50] Francesco Scarcello, Gianluigi Greco, and Nicola Leone. 2007. Weighted hypertree decompositions and optimal queryplans.
J. Comput. Syst. Sci.
73, 3 (2007), 475–506. https://doi.org/10.1016/j.jcss.2006.10.010[51] Werner Schafhauser. 2006.
New heuristic methods for tree decompositions and generalized hypertree decompositions .Master’s thesis. Technische Universität Wien.[52] André Schidler and Stefan Szeider. 2020. Computing Optimal Hypertree Decompositions. In
Proceedings of theSymposium on Algorithm Engineering and Experiments, ALENEX 2020, Salt Lake City, UT, USA, January 5-6, 2020
Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, Melbourne, Victoria, Australia,May 31 - June 4, 2015 , Timos K. Sellis, Susan B. Davidson, and Zachary G. Ives (Eds.). ACM, 2077–2078. https:, Vol. 1, No. 1, Article . Publication date: September 2020. //doi.org/10.1145/2723372.2764946[56] V. N. Vapnik and A. Ya. Chervonenkis. 1971. On the Uniform Convergence of Relative Frequencies of Events to TheirProbabilities.