Experience with Heuristics, Benchmarks & Standards for Cylindrical Algebraic Decomposition
EExperience with Heuristics, Benchmarks &Standards for Cylindrical Algebraic Decomposition
Matthew England ∗ , and James H. Davenport † , ∗ Coventry University Faculty of Engineering, Environment and Computing, Coventry, CV1 2JH, U.K.Email:
[email protected] † University of Bath Department of Computer Science, Bath, BA2 7AY, U.K.Email:
Abstract —In the paper which inspired the SC project,[E. ´Abr´aham, Building Bridges between Symbolic Computation andSatisfiability Checking , Proc. ISSAC ’15, pp. 1–6, ACM, 2015] theauthor identified the use of sophisticated heuristics as a techniquethat the Satisfiability Checking community excels in and fromwhich it is likely the Symbolic Computation community couldlearn and prosper. To start this learning process we summariseour experience with heuristic development for the computeralgebra algorithm Cylindrical Algebraic Decomposition. We alsopropose and discuss standards and benchmarks as another areawhere Symbolic Computation could prosper from SatisfiabilityChecking expertise, noting that these have been identified asinitial actions for the new SC community in the CSA project,as described in [E. ´Abr´aham et al., SC : Satisfiability Checkingmeets Symbolic Computation (Project Paper) , Intelligent ComputerMathematics (LNCS 9761), pp. 28–43, Springer, 2015]. I. I
NTRODUCTION
This article is inspired by the SC project , a new initiativeto forge a joint community from the existing fields of S ymbolic C omputation and S atisfiability C hecking. For further detailson the project we refer the reader to: • [2] which introduced the two fields, describes someof the challenges and opportunities from workingtogether, and outlines planned project actions; and • [1] the accompanying paper to an invited talk atISSAC 2015 which inspired the creation of the newproject and community.Within [1] the author outlines the strengths and weaknesses ofthe two communities, writing in the introduction:“Symbolic Computation is strong in providing pow-erful procedures for sets (conjunctions) of arithmeticconstraints, but it does not exploit the achievementsin SMT solving for efficiently handling logical frag-ments, using heuristics and learning to speed-up thesearch for satisfying solutions.”By heuristic the we mean a practical method to make a choicewhich is not guaranteed to be optimal. Although ComputerAlgebra Systems prize correctness and exact solutions thereis still much scope for the use of heuristics and statisticalmethods in symbolic computation algorithms: both for tuninghow individual algorithm are run and for selecting a particularalgorithm to use in the first place. In regards to the latter point, we note that the solve procedures in Computer Algebra Sys-tems are really meta-algorithms: algorithms to select specificprocedures to use based on problem parameters. Althoughthe individual procedures are usually well documented withinthe scientific literature we are not aware of any publicationsdescribing these meta-algorithms.Another topic where Symbolic Computation might benefitfrom experience in Satisfiability Checking is standards andbenchmarks. Competitions based on these form an integralpart of the Satisfiability Checking community, and may havecontributed to the remarkable progress made in practical al-gorithms. The lack of a comparable focus for the SymbolicComputation community was acknowledged in [2]. However,recent experiments have suggested the benchmarks for non-linear real arithmetic are insufficient and the development ofnew standards and benchmarks for the joint community hasbeen identified as a key SC project action in [2, Section 3.3].In the present paper we outline our experience of theseissues for a single Symbolic Computation algorithm, Cylindri-cal Algebraic Decomposition (CAD). The aim of the paper isto instigate the learning process from the Satisfiability Check-ing community by illustrating the current use of heuristics,benchmarks and standards in (at least one area of) SymbolicComputation and posing some questions. We start with asummary of the necessary background on CAD in Section II,then survey work with heuristics in CAD in Section III andour experience with standards and benchmarks in Section IV.We finish with conclusions and questions in Section V.II. C YLINDRICAL ALGEBRAIC DECOMPOSITION
A. Definition A Cylindrical Algebraic Decomposition (CAD) is a decom-position of R n into cells (connected subsets). By algebraic we mean semi-algebraic: i.e. each cell can be described witha finite sequence of polynomial constraints. Finally, the cellsare arranged cylindrically , meaning the projections of anypair, with respect to the variable ordering in which the CADwas created, are either equal or disjoint. We assume variableslabelled according to their ordering (so the projections con-sidered are ( x , . . . , x (cid:96) ) → ( x , . . . , x k ) for k < (cid:96) ) with thehighest ordered variable present said to be the main variable .Hence CADs can be represented in a tree like format branchingon the semi-algebraic conditions involving increasing mainvariable, as in the example below (with the branching fromright to left; all √ indicating the positive root; and the tuples a r X i v : . [ c s . S C ] S e p n the left sample points of the cells). ( − , x < − ( − , − y < − , y = 0( − ,
1) 0 < y x = − (0 , − y < −√− x + 1(0 , − y = −√− x + 1(0 , −√− x + 1 < y < √− x + 1(0 , y = + √− x + 1(0 , √− x + 1 < y − < x < (1 , − y < , y = 0(1 ,
1) 0 < y x = 1(2 ,
0) 1 < x
A CAD is produced to be invariant for input; originally sign-invariant for a set of input polynomials (so on each cell eachpolynomial is positive, zero or negative). The example aboveis a sign invariant CAD for the polynomial x + y − definingthe unit circle. More recently CADs have been produced truth-invariant for input Boolean-valued polynomial formulae. Asign-invariant CAD for the polynomials in a formula is alsotruth-invariant for the formula; but we can often achieve truth-invariance with far less cells. For example suppose we need aCAD truth-invariant for the formula x + y − ∧ ( x − + y − . A sign-invariant CAD would require 55 cells (with the fulldimensional ones shown on the left of Figure 1). However, atruth-invariant CAD would need only 7 cells: 2 of which arefull dimension (as on the right of Figure 1) and 5 more todecompose the line x = at the points of intersection. B. Computation
CAD construction usually involves two phases. The first projection , applies operators recursively on polynomials, start-ing with the input. Each time the operator produces a setwith one less variable which together define the projectionpolynomials . These are used in the second phase, lifting , to
Fig. 1. Example visualising sign and truth-invariant CADs build CADs of increasing dimension. First a CAD of R isbuilt by splitting on the real roots of the univariate polynomials(those in x only). Next, a CAD of R is built by repeatingthe process over each cell in R with the bivariate polynomialsin ( x , x ) evaluated at a sample point of the cell in R ; andthe process is repeated until a CAD of R n is produced. Wecall the cells where a polynomial vanishes sections and thoseregions in-between sectors , which together form the stack overthe cell. In each lift we extrapolate the conclusions drawn fromworking at a sample point to the whole cell requiring validitytheorems for the projection operator used.CAD cells are represented by at least a sample point (asin the left of the example above), and an index : a list ofpositive integers with each integer indicating the section orstack each variable is within (in reference to the ordered rootsof the projection polynomials). Some implementations willalso encode the full algebraic description within each cell.CAD was originally introduced by Collins for quantifierelimination (QE) in real closed fields [4]. Although CAD con-struction has complexity doubly exponential in the number ofvariables [26], applications range from parametric optimisation[34] and epidemic modelling [16], to reasoning with multi-valued functions [24] and the derivation of optimal numericalschemes [33]. There have been many improvements to Collins’original approach most notably refinements to the projectionoperators [50] [10], [53]; early termination of lifting [22][62]; and symbolic-numeric schemes [58], [42]. Some recentadvances include dealing with multiple formulae [7], [8]; localprojection [14], [60]; and decompositions via complex space[21], [6]. For a more detailed introduction to CAD see e.g. [8].III. H EURISTIC USE FOR
CAD
A. Choosing the variable ordering
The most well known choice required for CAD is thatof the variable ordering, which the cylindricity is definedwith respect to. This determines the order in which variablesare eliminated during projection and the subspaces throughwhich CADs are built incrementally during lifting. When usingCAD for QE we must project variables in the order they arequantified, but we are free to project the other variables in anyorder (and to change the order within quantifier blocks).The variable ordering used can have a great effect on theoutput produced. For example, let f := ( x − y + 1) − and consider the minimal sign-invariant CAD in each variableordering, as visualised in Figure 2. In each case we projectdown with the left figure projecting x first and the right y .In this toy example the “wrong” choice more than doublesthe number of cells, while numerous experiments have shownthat for larger examples the choice can determine whether aproblem is tractable (see for example the experimental resultsin [8]). At the extreme end of this observation, [15] defineda class of examples where changing variable ordering wouldchange the number of cells required from constant to doublyexponential in the number of variables.Several heuristics have been developed to choose the variableordering, including: Brown : Use the following criteria, starting with the firstand breaking ties with successive ones:1) Eliminate variable if lowest overall degree.(2) Eliminate variable if lowest (maximum) totaldegree in terms in which it occurs.(3) Eliminate variable if smallest number of termscontains it.Suggested by Brown in [13, Section 5.2]. sotd : For all admissible orderings, calculate the projec-tion set and choose the one with smallest sum oftotal degrees for each of the monomials in eachof the polynomials [27]. Performs well but morecostly than
Brown . A greedy alternative is toallocate one variable of the ordering at a time byprojecting each unallocated variable and choosingthe one which increases the sotd least. ndrr : As with sotd construct the full projection setand choose the ordering whose set has the least number of distinct real roots of the univariatepolynomials within [9]. Even more costly than sotd , but sensitive to the real geometry andshown to assist with examples where the sotd heuristic failed.The papers cited each contain experimental results demonstrat-ing their worth. In [39] the results of an experiment comparingthe 3 on a data set of over 7000 examples were reported. Theexperiments showed
Brown selecting the best ordering (asmeasured by lowest cell count) more often than the others.However a key finding was that there were substantial subsetsof examples for which each heuristic did best. Further, whencalculating the saving made by the heuristics (compared to theaverage cell count of the different orderings) the authors of [39]found that sotd actually made a greater saving for quantifiedproblems on average (i.e. while
Brown was superior onmore examples, sotd was superior on examples with greatersavings on offer). Together, this meant that recommending oneheuristic at the expense of all others was not possible.If the minimal cell count is a priority then one furtherapproach, suggested in [64] is to compute the full dimensionalcells for each possible ordering and pick the ordering with theminimum to derive a full CAD. Computing full dimensionalcells avoids any work with algebraic numbers and so is notas costly as may be thought, although it does require morecomputation than ndrr . It was noted in [64] that the full-dimensional cell calculations could be done in parallel with thefirst to finish extended to a full CAD and the rest discarded.
B. Choices with equational constraints1) Equational constraints:
There are several ways in whichwe can modify CAD constriction to achieve truth-invariance(rather than sign-invariance) including refining sign-invariantCAD and truncating lifting once the truth-value is determined.A particularly fruitful approach is to take advantage of the
Fig. 2. CADs under different variable orderings
Booloean structure of a formula through the identification of
Equational Constraints (ECs): polynomial equations logicallyimplied by a formula. Reduced projection operators (using asubset of the usual polynomials) have been proven valid for usewhen an EC is present with corresponding main variable [51],[52]. Results in [29], [31] suggest that the double exponentin the complexity bound decreases by 1 for each EC used,although this is restricted to primitive ECs meaning the classiclower-bound examples of [26], [15] are not violated [25].These reduced operators can only use one EC for eachprojection, so when there are multiple we must make a designation . Note that ECs need not appear explicitly as atoms(formula with no logical connectives) in the input formula, butcould instead be implicit. For example, the resultant of any twoECs in the same main variable is itself an EC (not containingthat variable) [52]. Propagating ECs in this way allows for themaximal use of the reduced operators. However, it can requiremultiple choices of which EC to designate at each projection.Section 4 of [29] described such an example where the wrong designation could make add tens of thousands to the cell countof the final output, making it more than 15 times bigger.
2) Making the designation:
In [9] the authors experimentedin using the sotd and ndrr measures on this question (the
Brown heuristic was not applicable since it acted only on theinput polynomials). In general they were useful in identify-ing the optimal designation, although both could be misled.As described above these heuristics essentially complete theprojection stage of the algorithm for each ordering, whichalthough minimal in comparison to the lifting stage, is likelyfar more computation than would normally be undertaken by aheuristic. This becomes an issue when the number of choicesgrows. Further, for these experiments a fixed variable orderingwas used, and the question of addressing the two choicestogether (when the number of possibilities multiplies) has notbeen addressed.
3) Designation in TTICAD:
In [7], [8] a truth-table invari-ant CAD (TTICAD) is defined as a CAD on whose cellsthe truth-table for a set of formulae is invariant. A newoperator was presented which takes advantage of ECs presentin the separate formulae (with [7] developing the theory in thecase where all had an EC and [8] extending to the generalcase). The operator essentially recognises when to considerthe interaction of polynomials from different formulae. If anindividual formula has multiple ECs then, as above, we mustchoose just one to designate for each projection.
C. Choices for CAD by Regular Chains
Recently an alternative CAD computational scheme hasbeen proposed where, instead of projection and lifting, we: firstcylindrically decompose complex space according to whetherpolynomials are zero or not using the theory of triangulardecomposition by regular chains; and then refine to a CAD ofreal space. This was first proposed in [21] with an incrementalversion described in [20] and an extension to TTICAD in [6].All versions are implemented within the
RegularChains
Library with a summary in [19].Most of the heuristics outlined earlier in this section are notdirectly applicable to the CAD by Regular Chains computation cheme (as there is no “cheap” projection phase to deriveinformation from). We outline some of the new heuristicsdeveloped for the choices this scheme involves.
1) Variable order in TTICAD by Regular Chains:
Thisproblem was considered in [30]. Two existing heuristics werecompared: that of Brown introduced in Section III-A andanother, denoted Triangular, already in use for other algorithmsin the
RegularChains
Library. Triangular chooses first thevariable with lowest degree occurring in the input; then breaksties by choosing variables for which leading coefficients havelowest total degree; and finally sum of degrees in input. Inaddition the heuristics sotd and ndrr discussed above wereused (even though the sets of projection polynomials built werenot explicitly used later). The experiments found sotd tomake the best choices, but due to its higher costs the Triangularheuristic was the most efficient choice overall. However, aswith the experiments discussed in Section III-A, the exampleset could be subdivided into groups where different heuris-tics were dominant. Further experimentation and illustrativeexamples in [30] led to the development of a new heuristic(composed from parts of the others) tailored to the variableordering choice for TTICAD by Regular Chains [6].
2) Constraint order in TTICAD by Regular Chains:
Thelatest CAD algorithm within the
RegularChains
Library[20] processes constraints incrementally when building thecomplex cylindrical decomposition and thus are sensitive tothe order in which constraints are considered. Further, in thecase of TTICAD we have the extra question of what order toconsider the formulae in. These issues were studied in [28]which considered the following example.Assume the ordering x ≺ y and consider f := x + y − ,f := 2 y − x,f := ( x − + ( y − − ,φ := f = 0 ∧ f = 0 ,φ := f = 0 . The polynomials are graphed within the plots of Figure 3. Ifwe want to study the truth of φ and φ (or say a parentformula φ ∨ φ ) we need a TTICAD to take advantage of theECs. There are two possible orders for the formulae and twopossible to consider the constraints within φ . Hence 4 possibleways we could calculate a TTICAD by Regular Chains. Belowwe show how many cells are produced by proceeding in theorders indicated, with the two dimensional cells shown inFigure 3. • φ → φ and f → f :
37 cells. • φ → φ and f → f :
81 cells. • φ → φ and f → f :
25 cells. • φ → φ and f → f :
43 cells.No previously discussed heuristic was applicable to thisproblem. For choosing which EC to process first in a givenformula an argument could be made for measuring a setof polynomials shown to be rendered sign-invariant by thealgorithm (leading to Heuristic 1 in [28]). The only heuristicderived for the other choices was to measure the sum of
Fig. 3. Visualisations of the four TTICADs which can be built using theRegular Chains Library for the example in this section. The figures on the tophave φ → φ and those on the bottom φ → φ . The figures on the lefthave f → f and those on the right f → f . degrees of the polynomials in the complex cylindrical decom-position created. As with the ndrr and full dimensional cellsheuristics above; this requires more computation than is ideal(although it is the real root refinement that makes up the bulkof the CAD by Regular Chains computation time). D. Gr¨obner Basis preconditioning A Gr¨obner Basis G is a particular generating set of anideal I defined with respect to a monomial ordering [17]. Onedefinition is that the ideal generated by the leading terms of I is generated by the leading terms of G . Gr¨obner Bases (GB)are used extensively to study ideals and the polynomials thatdefine them as they allow properties such as dimension andnumber of zeros to be easily deduced. Although like CADthe calculation of GB is doubly exponential in the worst case[48], GB computation is now mostly trivial for any problemon which CAD construction is tractable.It was first observed in [18] that replacing a conjunction ofpolynomial equalities in a CAD problem by their GB (logicallyequivalent) could be useful for the CAD computation. Ofthe ten test problems studied: 6 were improved by the GBpreconditioning (speed-up varying from 2- to 1700-fold); 1problem resulted in a 10-fold slow-down; 1 timed out whenGB preconditioning was applied, but would complete without;and 2 were intractable both for CAD construction alone and theGB preconditioning step. The problem was revisited in [66].As expected, there had been a big decrease in the computationtimings, especially for the GB. However, it was still the casethat 2 of the problems were hindered by GB preconditioning.The key conclusion is that GB preconditioning will onaverage benefit CAD (sometimes very significantly) but couldon occasion hinder it (to the point of making a tractable CADproblem intractable). We are yet to understand why this occurs,but the authors of [66] did develop a metric to predict when itwill. They defined the Total Number of Indeterminates ( TNoI )of a set of polynomials A as TNoI ( A ) = (cid:88) a ∈ A NoI ( a ) where NoI ( a ) is the number of indeterminates in a polynomial a . The heuristic is to build a CAD for the preconditionedpolynomials only if the TNoI decreased. For most of theirtest problems the heuristic made the correct choice, but therewere examples to the contrary. . Use of machine learning
Finally, we note recent experiments using machine learn-ing, specifically support vector machines (see for example[56]), to make choices for CAD construction: • In [40] the authors used an SVM to choose be-tween the three heuristics for CAD variable orderingoutlined in Section III-A. Simple problem featureswere selected (e.g. degrees, proportion of monomialscontaining each variable) and parameter optimisationwas applied to maximise Matthews’ Correlation Co-efficient [47]. Over 7000 examples were studied, andover the 1721 reserved for the test set the machinelearned choice was found to outperform each heuristicindividually on average. • In [38] the authors used an SVM to predict whenit will be useful to precondition a CAD problemwith GB (see Section III-D). The features used werefrom both the original input and the GB: so thestudy was answering the question should we use thisGB rather than should we compute it (relevant sincethe GB computation was trivial for the problem setinvolved). The machine learned choice outperformedboth using GB universally, and the human defined
TNoI heuristic.We also note that a recent paper [45] applied a support vectormachine (seeded with the problem features from [40]) tosuggest the order in which QE should be performed on sub-formulae of a non-prenex formula. Experimental results onmore than 2,000 non-trivial examples showed that machinelearning was doing better than the human derived heuristics,following appropriate parameter optimisation.IV. S
TANDARDS AND B ENCHMARKS
A. History of benchmarking in computer algebra
The Computer Algebra community has occasionally recog-nised the importance of benchmarks. The PoSSo and FRISCOprojects aimed to do this for polynomials systems andsymbolic-numeric problems respectively in the 1990s. PoSSO,with which the second author was involved, collected a seriesof benchmark examples for GB, and a descendant of thesecan still be found online . However, this does not appearto be maintained; and the polynomials are not stored in amachine-readable form. Polynomials from this list still cropup in various papers, but there is no systematic reference,and it is not clear whether people are really referring to thesame example. Several of the examples are families, whichis good but means that a benchmark has to contain specificinstances. The PoSSo project did its best to do “level playingfield” comparisons, but at the time different implementationsran on different hardware / operating systems meaning thiswas not really achievable. The environment is much simplerthese days, and it would be feasible to organise true contests.The topic of benchmarking in computer algebra has mostrecently been taken up by the SymbolicData Project [35]which is beginning to build a database of examples in XML format (although currently not with any suitable for CAD).The software described in [36] was built to translate problemsin that database into executable code for various computeralgebra systems. The authors of [36] discuss the peculiarities ofcomputer algebra that make benchmarking particularly difficultincluding the fact that results of computations need not beunique and that the evaluation of the correctness of an outputmay not be trivial (or may be the subject of research itself).A final point to note is that while SAT / SMT-solvers haveonly a few clear possible answers (e.g. sat, unsat, unknown)in computer algebra there is also the quality of the result toconsider (e.g., size of quantifier-free formula produced). WithCAD the output size is usually correlated to computation time,but this is not always the case with other algorithms. B. The present authors’ recent experience with CAD
In our work we have taken various approaches to experi-menting with CAD:1 (a) Test new results on examples previously used to evaluateCAD algorithms.For example, the experiments in [66] started with the 10examples in [18] while [24] and [63] focussed on classicexamples from [44] and [23]. The latter were derived fromapplications of CAD while the former seem to be a collectionof geometric problems invented by the authors. Other papersthat have contributed such test problems include [5], [49][11], [27]. We wonder whether historic repetition within theliterature is alone a strong enough reason to be benchmark?In [65], [61] an attempt was made to gather together allthose test examples in the literature for CAD, along with ref-erences of their first appearance in the literature and encodingsfor some computer algebra systems.1 (b) Supplement existing examples with modified versionssuitable for demonstrating the feature in question.In [7], [8] the new TTICAD algorithm that was the subject ofthe paper offered an improvement on the state of the art forexamples consisting of multiple formulae; or a single formulaein disjunctive normal form. Such examples had not been thetopic of any CAD papers before and no existing examples werecapable of demonstrating the savings on offer. The experimentsproduced in these papers were made of up two sets: • Formulae produced to describe the branch cuts ofmultivalued functions in a proposed simplificationformula [30], with CAD to be applied so that thecomplex domain could be decomposed into regionswhere the functions were univariate, and thus theformula applicable or not. • Formulae produced by adapting the logical connectorsin previous examples from the literature in [65] so thatconjunctions became disjunctions.Clearly the former set is of great interest as they representa real example for the algorithms; but they all conform to asingle structure and so are arguably too uniform to alone drawbroad conclusions from. The second set were produced to besomehow close to the accepted test examples of the literature,ut whether this is any better than inventing a new examplesfrom scratch is debatable.2 Derive new sets of random examplesA recent experiment using machine learning [40], [38] (seeSection III-E) exposed a shortcoming in the above techniques.To train the SVMs hundreds of examples are required (withhundreds more then needed for validation and testing). Thedataset from the literature in [65] contained not nearly enoughexamples and while the datasets discussed in the next sectionwere sufficient for the first experiment in [40] they provedtoo uniform for [38]. We were left with no choice but togenerate large quantities of new examples, which we did usingthe random polynomial generator in M
APLE . We had appliedthis technique also in [28], [30] receiving positive feedbackfrom reviewers for the technique; but the initial reviews of[38] were all negative on the use of random data. It seems theappropriateness of this technique varies with the community(conference) in question. We opine that had we used data fromthe example bank of MetiTarski examples discussed in the nextsection then reviewers may have praised the focus on examplesfrom a real application; even though MetiTarski themselvesderive examples for benchmarks using random polynomials.
C. Sources of large benchmarks sets
We note some other sources of large sets of benchmarkproblems that represent real applications of CAD: • MetiTarski [3], [54] is an automatic theorem proverdesigned to prove theorems involving real-valued spe-cial functions (such as log, exp, sin, cos and sqrt). Ingeneral this theory is undeciadable but MetiTarski isable to solve many problems by applying real polyno-mial bounds and then using real quantifier eliminationtools like CAD. Applications of MetiTarski in turnderive examples for CAD. • The NRA (non-linear real arithmetic) category of theSMT-LIB library which according to [43] consistsmostly of problems originating from attempts to provetermination of term-rewrite systems.These two data sets where included in the nlsat Bench-mark Set produced to evaluate the work in [43]. This alsoincluded verification conditions from the Keymaera [55] andparametrized generalizations of the problem from [37]. To-gether this dataset had many thousands of problems. However,we note that the problems come from a small number of classesand may have some hidden uniformity.As mentioned above, the nlsat dataset was unsuitable touse for our machine learning experiment in [38]. Every singleproblem within that had more than one equality was aidedby GB preconditioning, in fact a great many simply had aGB containing only 1 indicating the problem had no solution.Previous experiments on small example sets suggested GBpreconditioning sometimes harms CAD computation and thiswas verified by analysis of a large randomly generated dataset ∼ lp15/papers/Arith http://smtlib.cs.uiowa.edu/ http://cs.nyu.edu/ ∼ dejan/nonlinear/ in [38]. Thus while the nlsat dataset is an excellent startingpoint it needs to be expanded to be less uniform.We finish this section by noting one possible source ofexamples for the future. • The Todai Robot Project [46] is a Japanese AI projectthat aims to have an artificial intelligence pass theentrance examination for the University of Tokyo by2021. A majority of questions on the Mathematicsexam can be resolved by real quantifier eliminationwith a variety of techniques employed [41]. A keydifficulty is that the natural language processing ofthe question derives a formula of far greater complex-ity than the human derived equivalent. This processderives a large bank of examples of CAD problems,as discussed in [45] for example. The authors of [45]told us there are plans to make this data set public.V. C ONCLUSIONS AND QUESTIONS
After surveying the work in Section III we see that severalapproaches to the creation of heuristics have been taken:ranging from human identified algebraic features, justified bymathematical arguments and observations to different extents;to machine learned choices using a support vector machine. Weare interested to hear how these compare with the heuristicsused in SAT-solvers and what lessons can be learned.We can identify at least two areas where CAD is in needof further heuristic development. • How best to take decisions in tandem:
The worksurveyed all considered choices to be made for CAD in isolation (assuming other choices had already beenfixed). Of course, in reality this may not be the case.We must decide which decisions to prioritise; howheuristics can be combined; and how the combinatorialblow-up of decisions can be contained. Does the SAT-solving community have experience in similar issues?There are many implementations of CAD including: the dedi-cated command line program Qepcad [12]; Mathematica [58],[59], where CAD is not available directly but is used asa subroutine for quantifier elimination; the
Redlog pack-age for R
EDUCE [57]; and 3 different M
APLE libraries -the
RegularChains
Library (see Section III-C;
SyNRAC [67] (now part of the Todai Robot project) and our own
ProjectionCAD module [32]. • How to choose between different implementations?
Each implementation includes unique pieces of theoryand features and excels on different examples. Ide-ally, we would have a single implementation whichencompasses all recent advances. A more manageablestep may be an overarching M
APLE command tochoose between the 3 packages there. SMT-solversare designed to use a variety of different theory solversand how they choose between these may offer valuablelessons here.Surveying Section IV raises a number of questions abouthow benchmark sets should be produced: http://21robot.org How best to generate large numbers of exampleswhich are not internally uniform? • How important is it that the benchmarks come fromcurrent applications? • How important is it that the benchmarks have histor-ically been used in the literature? • Are randomly generated examples a fairer way toevaluate the software, or irrelevant as too far removedfrom applications?The SAT / SMT community posses a unified, large andgrowing set of benchmarks in the SMT-LIB library and so wemay be able to extrapolate lessons from this. However, as notedabove, this library may be too uniform [38], and commentsfrom the anonymous referees suggest that this and other criticsare already a topic of discussion in the SMT community.
Acknowledgements
Thanks to our collaborators on the work surveyed here:Russell Bradford, James Bridge, Changbo Chen, ZongyanHuang, Scott McCallum, Marc Moreno Maza, Lawrence Paul-son, and David Wilson. Thanks also to the anonymous refereesfor their comments which improved the paper.Most of the work surveyed here was supported by EPSRCgrant EP/J003247/1. The authors are now supported by EUH2020-FETOPEN-2016-2017-CSA project SC (712689).R EFERENCES[1] E. ´Abr´aham. Building bridges between symbolic computation andsatisfiability checking. In
Proceedings of the 2015 InternationalSymposium on Symbolic and Algebraic Computation , ISSAC ’15, pages1–6. ACM, 2015.[2] E. ´Abrah´am, J. Abbott, B. Becker, A.M. Bigatti, M. Brain, B. Buch-berger, A. Cimatti, J.H. Davenport, M. England, P. Fontaine, S. Forrest,A. Griggio, D. Kroening, W.M. Seiler, and T. Sturm. SC : Satisfiabilitychecking meets symbolic computation. In M. Kohlhase, M. Johansson,B. Miller, L. de Moura, and F. Tompa, editors, Intelligent ComputerMathematics: Proceedings CICM 2016 , LNCS 9791, pages 28–43.Springer International Publishing, 2016.[3] B. Akbarpour and L.C. Paulson. MetiTarski: An automatic theoremprover for real-valued special functions.
Journal of Automated Reason-ing , 44(3):175–205, 2010.[4] D. Arnon, G.E. Collins, and S. McCallum. Cylindrical algebraicdecomposition I: The basic algorithm.
SIAM Journal of Computing ,13:865–877, 1984.[5] D.S. Arnon. A cluster-based cylindrical algebraic decompositionalgorithm.
Journal of Symbolic Computation , 5(1-2):189–212, 1988.[6] R. Bradford, C. Chen, J.H. Davenport, M. England, M. Moreno Maza,and D. Wilson. Truth table invariant cylindrical algebraic decompositionby regular chains. In V.P. Gerdt, W. Koepf, W.M. Seiler, and E.V.Vorozhtsov, editors,
Computer Algebra in Scientific Computing , LNCS8660, pages 44–58. Springer International Publishing, 2014.[7] R. Bradford, J.H. Davenport, M. England, S. McCallum, and D. Wilson.Cylindrical algebraic decompositions for boolean combinations. In
Proceedings of the 38th International Symposium on Symbolic andAlgebraic Computation , ISSAC ’13, pages 125–132. ACM, 2013.[8] R. Bradford, J.H. Davenport, M. England, S. McCallum, and D. Wilson.Truth table invariant cylindrical algebraic decomposition.
Journal ofSymbolic Computation , 76:1–35, 2015.[9] R. Bradford, J.H. Davenport, M. England, and D. Wilson. Optimisingproblem formulations for cylindrical algebraic decomposition. InJ. Carette, D. Aspinall, C. Lange, P. Sojka, and W. Windsteiger, editors,
Intelligent Computer Mathematics , LNCS 7961, pages 19–34. SpringerBerlin Heidelberg, 2013. [10] C.W. Brown. Improved projection for cylindrical algebraic decomposi-tion.
Journal of Symbolic Computation , 32(5):447–465, 2001.[11] C.W. Brown. Simple CAD construction and its applications.
Journalof Symbolic Computation , 31(5):521–547, 2001.[12] C.W. Brown. An overview of QEPCAD B: a tool for real quantifierelimination and formula simplification.
Journal of Japan Society forSymbolic and Algebraic Computation
Proceedings of the 38th International Symposiumon Symbolic and Algebraic Computation , ISSAC ’13, pages 133–140.ACM, 2013.[15] C.W. Brown and J.H. Davenport. The complexity of quantifier elimina-tion and cylindrical algebraic decomposition. In
Proceedings of the 2007International Symposium on Symbolic and Algebraic Computation ,ISSAC ’07, pages 54–60. ACM, 2007.[16] C.W. Brown, M. El Kahoui, D. Novotni, and A. Weber. Algorithmicmethods for investigating equilibria in epidemic modelling.
Journal ofSymbolic Computation , 41:1157–1173, 2006.[17] B. Buchberger. Bruno Buchberger’s PhD thesis (1965): An algorithm forfinding the basis elements of the residue class ring of a zero dimensionalpolynomial ideal.
Journal of Symbolic Computation , 41(3-4):475–511,2006.[18] B. Buchberger and H. Hong. Speeding up quantifier eliminationby Gr¨obner bases. Technical report, 91-06. RISC, Johannes KeplerUniversity, 1991.[19] C. Chen and M. Moreno Maza. Cylindrical algebraic decompositionin the RegularChains library. In H. Hong and C. Yap, editors,
Mathematical Software – ICMS 2014 , LNCS 8592, pages 425–433.Springer Heidelberg, 2014.[20] C. Chen and M. Moreno Maza. An incremental algorithm for computingcylindrical algebraic decompositions. In R. Feng, W. Lee, and Y. Sato,editors,
Computer Mathematics , pages 199—221. Springer Berlin Hei-delberg, 2014.[21] C. Chen, M. Moreno Maza, B. Xia, and L. Yang. Computing cylindricalalgebraic decomposition via triangular decomposition. In
Proceedingsof the 2009 International Symposium on Symbolic and AlgebraicComputation , ISSAC ’09, pages 95–102. ACM, 2009.[22] G.E. Collins and H. Hong. Partial cylindrical algebraic decompositionfor quantifier elimination.
Journal of Symbolic Computation , 12:299–328, 1991.[23] J.H. Davenport. A “Piano-Movers” Problem.
SIGSAM Bull. , 20(1-2):15–17, 1986.[24] J.H. Davenport, R. Bradford, M. England, and D. Wilson. Programverification in the presence of complex numbers, functions with branchcuts etc. In , SYNASC ’12, pages 83–88. IEEE,2012.[25] J.H. Davenport and M. England. Need polynomial systems be doublyexponential? In G.-M. Greuel, T. Koch, P. Paule, and A. Sommese,editors,
Mathematical Software – Proceedings of ICMS 2016 , LNCS9725, pages 157–164. Springer International Publishing, 2016.[26] J.H. Davenport and J. Heintz. Real quantifier elimination is doublyexponential.
Journal of Symbolic Computation , 5(1-2):29–35, 1988.[27] A. Dolzmann, A. Seidl, and T. Sturm. Efficient projection orders forCAD. In
Proceedings of the 2004 International Symposium on Symbolicand Algebraic Computation , ISSAC ’04, pages 111–118. ACM, 2004.[28] M. England, R. Bradford, C. Chen, J.H. Davenport, M. Moreno Maza,and D. Wilson. Problem formulation for truth-table invariant cylindricalalgebraic decomposition by incremental triangular decomposition. InS.M. Watt, J.H. Davenport, A.P. Sexton, P. Sojka, and J. Urban, editors,
Intelligent Computer Mathematics , LNCS 8543, pages 45–60. SpringerInternational, 2014.[29] M. England, R. Bradford, and J.H. Davenport. Improving the useof equational constraints in cylindrical algebraic decomposition. In
Proceedings of the 2015 International Symposium on Symbolic andAlgebraic Computation , ISSAC ’15, pages 165–172. ACM, 2015.30] M. England, R. Bradford, J.H. Davenport, and D. Wilson. Choosinga variable ordering for truth-table invariant cylindrical algebraic de-composition by incremental triangular decomposition. In H. Hong andC. Yap, editors,
Mathematical Software – ICMS 2014 , LNCS 8592,pages 450–457. Springer Heidelberg, 2014.[31] M. England and J.H. Davenport. The complexity of cylindrical algebraicdecomposition with respect to polynomial degree.
To appear In:Proceedings CASC 2016 (Springer LNCS) , 2016.[32] M. England, D. Wilson, R. Bradford, and J.H. Davenport. Using theRegular Chains Library to build cylindrical algebraic decompositionsby projecting and lifting. In H. Hong and C. Yap, editors,
Mathe-matical Software – ICMS 2014 , LNCS 8592, pages 458–465. SpringerHeidelberg, 2014.[33] M. Erascu and H. Hong. Synthesis of optimal numerical algorithmsusing real quantifier elimination (Case Study: Square root computation).In
Proceedings of the 39th International Symposium on Symbolic andAlgebraic Computation , ISSAC ’14, pages 162–169. ACM, 2014.[34] I.A. Fotiou, P.A. Parrilo, and M. Morari. Nonlinear parametric opti-mization using cylindrical algebraic decomposition. In
Decision andControl, 2005 European Control Conference. CDC-ECC ’05. , pages3735–3740, 2005.[35] H.G. Graebe, A. Nareike, and S. Johanning. The SymbolicDataproject: Towards a computer algebra social network. In M. England,J.H. Davenport, A. Kohlhase, M. Kohlhase, P. Libbrecht, W. Neuper,P. Quaresma, A.P. Sexton, P. Sojka, J. Urban, and S.M. Watt, editors,
Joint Proceedings of the MathUI, OpenMath and ThEdu Workshops andWork in Progress track at CICM , number 1186 in CEUR WorkshopProceedings, 2014.[36] A. Heinle and V. Levandovskyy. The SDEval benchmarking toolkit.
ACM Communications in Computer Algebra , 49(1):1–9, 2015.[37] H. Hong. Comparison of several decision algorithms for the existentialtheory of the reals. Technical report, RISC, Linz, 1991.[38] Z. Huang, M. England, J.H. Davenport, and L. Paulson. Using machinelearning to decide when to precondition cylindrical algebraic decom-position with Groebner bases. In , SYNASC’16. Preprint: Arxiv 1608.04219. IEEE, 2016.[39] Z. Huang, M. England, D. Wilson, J.H. Davenport, and L. Paulson. Acomparison of three heuristics to choose the variable ordering for cad.
ACM Communications in Computer Algebra , 48(3):121–123, 2014.[40] Z. Huang, M. England, D. Wilson, J.H. Davenport, L. Paulson, andJ. Bridge. Applying machine learning to the problem of choosinga heuristic to select the variable ordering for cylindrical algebraicdecomposition. In S.M. Watt, J.H. Davenport, A.P. Sexton, P. Sojka,and J. Urban, editors,
Intelligent Computer Mathematics , LNAI 8543,pages 92–107. Springer International, 2014.[41] H. Iwane, T. Matsuzaki, N.H. Arai, and H. Anai. Automated naturallanguage geometry math problem solving by real quantifier elimination.In
Proceedings of the 10th International Workshop on AutomatedDeduction in Geometry , ADG ’14, pages 75–84, 2014.[42] H. Iwane, H. Yanami, H. Anai, and K. Yokoyama. An effective imple-mentation of a symbolic-numeric cylindrical algebraic decompositionfor quantifier elimination. In
Proceedings of the 2009 conference onSymbolic Numeric Computation , SNC ’09, pages 55–64, 2009.[43] D. Jovanovic and L. de Moura. Solving non-linear arithmetic. InB. Gramlich, D. Miller, and U. Sattler, editors,
Automated Reasoning:6th International Joint Conference (IJCAR) , LNCS 7364, pages 339–354. Springer, 2012.[44] W. Kahan. Branch cuts for complex elementary functions. In A. Iserlesand M.J.D. Powell, editors,
Proceedings The State of Art in NumericalAnalysis , pages 165–211. Clarendon Press, 1987.[45] M. Kobayashi, H. Iwane, T. Matsuzaki, and H. Anai. Efficient subfor-mula orders for real quantifier elimination of non-prenex formulas. InS.I. Kotsireas, M.S. Rump, and K.C. Yap, editors,
Mathematical Aspectsof Computer and Information Sciences (MACIS ’15) , LNCS 9582, pages236–251. Springer International Publishing, 2016.[46] T. Matsuzaki, H. Iwane, H. Anai, and N. Arai. The most uncreativeexaminee: A first step toward wide coverage natural language mathproblem solving. In C.E. Brodley and P. Stone, editors,
Proceedings of28th Conference on Artificial Intelligence , AAAI ’14, pages 1098–1104.AAAI Press, 2014. [47] B.W. Matthews. Comparison of the predicted and observed secondarystructure of T4 phage lysozyme.
Biochimica et Biophysica Acta (BBA)-Protein Structure , 405(2):442–451, 1975.[48] E.W. Mayr and A.R. Meyer. The complexity of the word problemsfor commutative semigroups and polynomial ideals.
Advances inMathematics , 46(3):305–329, 1982.[49] S. McCallum. An improved projection operation for cylindrical alge-braic decomposition of three-dimensional space.
Journal of SymbolicComputation , 5(1-2):141–161, 1988.[50] S. McCallum. An improved projection operation for cylindricalalgebraic decomposition. In B. Caviness and J. Johnson, editors,
Quantifier Elimination and Cylindrical Algebraic Decomposition , Texts& Monographs in Symbolic Computation, pages 242–268. Springer-Verlag, 1998.[51] S. McCallum. On projection in CAD-based quantifier eliminationwith equational constraint. In
Proceedings of the 1999 InternationalSymposium on Symbolic and Algebraic Computation , ISSAC ’99, pages145–149. ACM, 1999.[52] S. McCallum. On propagation of equational constraints in CAD-based quantifier elimination. In
Proceedings of the 2001 InternationalSymposium on Symbolic and Algebraic Computation , ISSAC ’01, pages223–231. ACM, 2001.[53] S. McCallum, A. Parusi´niski, and L. Paunescu. Validity proof ofLazard’s method for CAD construction.
Preprint: Arxiv 1607:00264 ,2016.[54] L.C. Paulson. Metitarski: Past and future. In L. Beringer andA. Felty, editors,
Interactive Theorem Proving , LNCS 7406, pages 1–10.Springer, 2012.[55] A. Platzer, J.D. Quesel, and P. R¨ummer. Real world verification. InR.A. Schmidt, editor,
Automated Deduction (CADE-22) , LNCS 5663,pages 485–501. Springer Berlin Heidelberg, 2009.[56] B. Sch¨olkopf, K. Tsuda, and J.-P. Vert.
Kernel methods in computationalbiology . MIT Press, 2004.[57] A. Seidl and T. Sturm. A generic projection operator for partial cylindri-cal algebraic decomposition. In
Proceedings of the 2003 InternationalSymposium on Symbolic and Algebraic Computation , ISSAC ’03, pages240–247. ACM, 2003.[58] A. Strzebo´nski. Cylindrical algebraic decomposition using validatednumerics.
Journal of Symbolic Computation , 41(9):1021–1038, 2006.[59] A. Strzebo´nski. Computation with semialgebraic sets represented bycylindrical algebraic formulas. In
Proceedings of the 2010 InternationalSymposium on Symbolic and Algebraic Computation , ISSAC ’10, pages61–68. ACM, 2010.[60] A. Strzebo´nski. Cylindrical algebraic decomposition using local projec-tions. In
Proceedings of the 39th International Symposium on Symbolicand Algebraic Computation , ISSAC ’14, pages 389–396. ACM, 2014.[61] D. Wilson. Real geometry and connectedness via triangular description:Cad example bank, 2013.[62] D. Wilson, R. Bradford, J.H. Davenport, and M. England. Cylindricalalgebraic sub-decompositions.
Mathematics in Computer Science ,8:263–288, 2014.[63] D. Wilson, J.H. Davenport, M. England, and R. Bradford. A “pianomovers” problem reformulated. In , SYNASC’13, pages 53–60. IEEE, 2013.[64] D. Wilson, M. England, J.H. Davenport, and R. Bradford. Using the dis-tribution of cells by dimension in a cylindrical algebraic decomposition.In , SYNASC ’14, pages 53–60. IEEE, 2014.[65] D.J. Wilson, R.J. Bradford, and J.H. Davenport. A repository for CADexamples.
ACM Communications in Computer Algebra , 46(3):67–69,2012.[66] D.J. Wilson, R.J. Bradford, and J.H. Davenport. Speeding up cylindricalalgebraic decomposition by Gr¨obner bases. In J. Jeuring, J.A. Campbell,J. Carette, G. Reis, P. Sojka, M. Wenzel, and V. Sorge, editors,
Intel-ligent Computer Mathematics , LNCS 7362, pages 280–294. Springer,2012.[67] H. Yanami and H. Anai. Development of SyNRAC. In