Towards Dynamic Algorithm Selection for Numerical Black-Box Optimization: Investigating BBOB as a Use Case
TTowards Dynamic Algorithm Selection for NumericalBlack-Box Optimization: Investigating BBOB as a Use Case
Diederick Vermetten
Leiden Institute for Advanced Computer ScienceLeiden, The Netherlands
Hao Wang
Sorbonne Universit´e, CNRS, LIP6Paris, France
Thomas B¨ack
Leiden Institute for Advanced Computer ScienceLeiden, The Netherlands
Carola Doerr
Sorbonne Universit´e, CNRS, LIP6Paris, France
ABSTRACT
One of the most challenging problems in evolutionary computationis to select from its family of diverse solvers one that performs wellon a given problem. This algorithm selection problem is compli-cated by the fact that different phases of the optimization processrequire different search behavior. While this can partly be con-trolled by the algorithm itself, there exist large differences betweenalgorithm performance. It can therefore be beneficial to swap theconfiguration or even the entire algorithm during the run. Longdeemed impractical, recent advances in Machine Learning and in ex-ploratory landscape analysis give hope that this dynamic algorithmconfiguration (dynAC) can eventually be solved by automaticallytrained configuration schedules. With this work we aim at pro-moting research on dynAC, by introducing a simpler variant thatfocuses only on switching between different algorithms, not con-figurations. Using the rich data from the Black Box OptimizationBenchmark (BBOB) platform, we show that even single-switch dy-namic Algorithm selection (dynAS) can potentially result in signifi-cant performance gains. We also discuss key challenges in dynAS,and argue that the BBOB-framework can become a useful tool inovercoming these.
CCS CONCEPTS • Theory of computation → Bio-inspired optimization; Online al-gorithms;
It is well known that, when solving an optimization problem, dif-ferent stages of the process require different search behavior. Forexample, while exploration is needed in the initial phases, the al-gorithm needs to eventually converge to a solution (exploitation).State-of-the-art optimization algorithms therefore often incorpo-rate mechanisms to adjust their search behavior while optimizing ,by taking into account the information obtained during the run.These techniques are studied under many different umbrellas, suchas parameter control [10], meta-heuristics [5], adaptive operatorselection [24], or hyper-heuristics [6]. The probably best-knownand most widely used techniques for achieving a dynamic searchbehavior are the one-fifth success rule [7, 32, 33] and the covarianceadaptation technique that the family of CMA-ES algorithms [14, 15]
GECCO ’20, Cancn, Mexico is build upon. While each of these two control mechanisms tacklesthe problem of balancing performance in different phases of thesearch in its own way, they are mostly working with a specificalgorithm, aiming to tune its performance by changing internalparameters or algorithm modules. This inherently limits the poten-tial of these methods, since different algorithms can have widelyvarying performances during different phases of the optimizationprocess. By switching between these algorithms during the search,these differences could potentially be exploited to get even betterperformance. We coin the problem of choosing which algorithmsto switch between, and under which circumstances, the
DynamicAlgorithm Selection (dynAS) problem.Solving the dynAS problem would be an important milestone to-wards tackling the more general dynamic Algorithm Configuration(dynAC) problem, which also addresses the problem of selecting(and possibly adjusting) suitable algorithm configurations. Specif-ically, dynAS is limited to switching between algorithms from adiscrete portfolio of pre-configured heuristics, whereas for dynAC,the algorithms come with (possibly several) parameters whose set-tings can have significant influence on the performance.We do not solve dynAS here, but aim to show its potential fornumerical optimization. We then aim to develop suitable environ-ments to encourage and enable future research into achieving theidentified potential of dynAS and, in the longer run, to extend thisto the dynAC problem. As a first step, we need to identify a mean-ingful collection of algorithms and benchmark problems, whichtogether cover the main characteristics and challenges of the dynASproblem, without imposing too many additional challenges. TheBlack-Box Optimization Benchmarking (BBOB) environment [12]with its rich data sets available at [1] suggests itself as a naturalstarting point for such considerations, since the community hasalready acquired a quite solid understanding of the problems andsolvers in this test-bed over the last decade.We perform a first assessment of the performance that one couldexpect to see when applying dynAS to the algorithms in the BBOBdata sets, to understand whether the gains would justify furtherexploration of the dynAS paradigm on this test-bed. We find that– even when restricting the dynAS problem further to allowingonly a single switch between algorithms in the portfolio – promis-ing improvements over the best static solvers can be expected, inparticular for the more complex problems (functions 19-24).Our considerations are purely based on a theoretical investiga-tion of the potential, which might be too optimistic for the single-switch dynAS case – most importantly, because of the problem a r X i v : . [ c s . N E ] J un ECCO ’20, July 8–12, 2020, Cancn, Mexico D. Vermetten, H. Wang, T. B¨ack, and C. Doerr of warm-staring the algorithms: since the heuristics are adaptivethemselves, their states need to be initialized appropriately at theswitch. This may be a difficult problem when changing betweenalgorithms of very different structure. We do not consider, on theother hand, the possibility to switch more than once, so that ourbounds may be too too pessimistic for the full dynAS setting, inwhich an arbitrary number of switches is allowed.Given the above limitations, we therefore also provide a criticalassessment of our approach, and highlight ideas for addressing themain challenges in dynAS. The idea that a dynamic configurations and/or selection of algo-rithms can be beneficial in the context of iterative optimizationheuristics is almost as old as evolutionary computation itself, inparticular in the context of solving numerical optimization prob-lems, see [20] for an entire book focusing mostly on dynamic al-gorithm configuration techniques. However, as mentioned above,existing works almost exclusively focus on changing parametersof selected components of an otherwise stable algorithmic frame-work. This includes most works on hyper-heuristics [6] and relatedconcepts such as adaptive operator selection [24], and parametercontrol [10].To the best of our knowledge, the full dynAC problem as de-scribed above was only recently formalized [4]. Biedenkamp etal. introduce dynAC as a Contextual Markov Decision Process(CMDP), where a policy can be learned to switch hyperparametersof a meta-algorithm, with some of these hyperparameters possi-bly encoding the choice between different algorithms. They alsoshow that artificial CMDPs can be solved effectively by using rein-forcement learning techniques, providing a promising direction forfuture research on dynAC.In the context of evolutionary computation, the concept of switch-ing between different algorithms during the optimization processwas recently investigated in [35], by a similar theoretical assess-ment as in this work. The approach was then tested in [37], whereit was shown that the predicted gains can indeed materialize, withthe caveat that one has to ensure a sufficiently accurate estimatefor the median anytime performances of each algorithm. These twoworks, however, focus on a single family of numerical black-box op-timization techniques, the modular CMA-ES framework suggestedin [36]. Here in this work, in contrast, we explicitly want to go onestep further, and study combinations of heuristics that are poten-tially of very different structure, such as, for example combininga Differential Evolution (DE) algorithm for the global explorationwith a CMA-ES for the final convergence.While the dynAC problem is solved by an unsupervised rein-forcement learning approach in [4], we observe that dynAC in evo-lutionary computation is more frequently based on on supervisedlearning approaches, see [16, 23, 27] for examples. These techniques Note here that there is a long-standing debate about the classification of algorithmconfiguration vs. algorithm selection. That is, while some consider a parametrizedalgorithm framework an algorithm with different configurations, others argue thateach such configuration is an algorithm by itself. We omit this discussion here, anduse the convention that an algorithm can have possibly different configurations. Note,though, that – in the context of this work – this only makes a difference in theterminology. All concepts and ideas can be equivalently described using the other,possibly mathematically more stringent, convention. combine exploratory landscape analysis [25] and/or fitness land-scape analysis [31] with supervised learning techniques, such asrandom forests, support vector machines, etc. While still in its in-fancy, even in the static algorithm configuration case [3, 17, 18, 28],these works may pave an interesting alternative to reinforcementlearning, as they may more directly provide insight into (and makeuse of) the correlation between fitness landscapes and algorithms’performance.
Classically, algorithm selection attempts to find the best algorithm A from a portfolio A to solve a specific function f from a set offunctions F . Specifically, this static version of algorithm selectioncan be defined as follows: Definition 2.1 (Static Algorithm Selection).
Given an algorithmportfolio A and a function f ∈ F , we aim to find:arg min A ∈A PERF ( A , f ) , where PERF is a performance measure (which assigns lower valuesto better performing algorithms).To extend algorithm selection to the dynamical case, we needto define a function which switches between algorithms. We usetechniques from [4] to represent this as a policy function, andmodify it as follows: Definition 2.2 (Dynamic Algorithm Selection (dynAS)).
Given analgorithm portfolio A , a f ∈ F and a state description s t ∈ S attime step t of an algorithm run. We want to find a policy π : S −→ A which minimizes PERF ( A π , f ) Note that this definition can be extended to dynamic algorithmconfiguration by changing the policy to be π : S −→ (A × Θ A ) ,where Θ A is the configuration space of algorithm A . The Black Box Optimization Benchmark (BBOB) is widely acceptedas the go-to benchmarking framework within the field of optimiza-tion. While BBOB has grown a lot over the years, the functionswithin their noiseless suite have remained stable. This suite contains24 noiseless optimization functions, each of which being theoreti-cally defined for any number of dimensions. In practice however,the commonly used dimension set is D = { , , , , , } . Foreach function, several transformation methods are defined, bothfor the variable as the objective spaces. These transformations arefixed, and different combinations lead to different versions of thefunction, called instances. Since these functions are defined mathe-matically, the optimal values are known in advance. Because of this,we can define target values we wish to reach in terms of closenessto this optimal value, instead of an abstract value. This gives theadvantage of comparability between instances, which would notbe possible when using raw target values.The 24 noiseless functions have been studied in detail, not justfrom a performance perspective. Especially within the landscapeanalysis community, a lot of analysis of the BBOB-functions has owards Dynamic Algorithm Selection GECCO ’20, July 8–12, 2020, Cancn, Mexico been performed, leading to a lot of useful insights about their prop-erties. These properties are ideal to use when implementing dynASin practice, as they are very influential on the local performance ofalgorithms. Generally, it is agreed that the 24 BBOB functions covera broad range of potential challenges for different optimizationalgorithms [25], even though certain aspects, i.e., discontinuities orplateaus, are not very well represented [19].The popularity of BBOB means that many researchers havebenchmarked their algorithms on the BBOB-functions. Most ofthese have then submitted versions of their algorithms to compe-titions or workshops organized by the BBOB-team. Between thefirst competition in 2009 [13] and the latest workshop in 2019, atotal of 226 algorithms have been submitted and their data madeavailable to the public [1]. Because of this large amount of availabledata, there are plenty of baselines to compare algorithms againstand gain inspiration from. These algorithms have often been welljustified and rigorously tested. However, the implementations usedare generally not freely available, and even if they are, they mightbe hard to combine into a single dynAS framework, since BBOBis available in many different languages. However, the majorityof the algorithms is either directly available online or has beenwell-documented, making the challenge of implementing themdoable.Additionally, the large amount of algorithms which have beenrun on BBOB provide a good way to select sets of algorithms fromwhich to build initial dynAS portfolios. However, since the BBOB-repository is largely the result from running competitions, manyof the used algorithms are highly tuned, making them hard to beatand giving rise to the question of generalizability of dynAS resultsto other functions. Eventually, a move to true dynAC would resolvethis issue, but these techniques will require a lot of further study toimplement.Since the BBOB-framework provides the functions, algorithmsand performance baselines, it is an ideal candidate for initial exper-iments related to dynAS. To measure the performance of the algorithms on the BBOB-dataset,several approaches are possible. These usually fall into two cate-gories: fixed-budget and fixed-target. The fixed-budget approachasks the question: ”What target value is reached after x functionevaluations?”, while the fixed-target question can be phrased as:”How many function evaluations are needed to reach target y ?”.In this paper, we will use the fixed-target approach. Since mostalgorithms in our data set are stochastic in nature, the questionof how many function evaluations are needed to reach a certaintarget is dealing with random variables. For a certain functioninstance f i ∈ F and dimension d ∈ D , we let t j ( A , f ( d ) i , ϕ ) denotethe number of evaluations that algorithm A ∈ A needed in the j -thrun to evaluate for the first time a point of target precision at least ϕ . Note that t j ( A , f ( d ) i , ϕ ) is a random variable, which is commonlyreferred to as the Hitting Time (HT) . If run j did not manage to hittarget ϕ within its allocated budget, we say that t j ( A , f ( d ) i , ϕ ) = ∞ .While just taking the average of the observed hitting time givessome estimate of the true mean, previous work [2] has shownthat it is not a consistent, unbiased estimator of the mean of the distribution of hitting times. Instead, the Expected Running Time(ERT) is used. This is defined as follows: Definition 2.3 (Expected Running Time (ERT)).
ERT ( A , f ( d ) , ϕ ) = (cid:205) ni = (cid:205) Kj = min { t i ( A , f ( d ) j , ϕ ) , B } (cid:205) ni = (cid:205) Kj = { t i ( A , f ( d ) j , ϕ ) < ∞} . Here, n is the number of runs of the algorithm, K the number ofinstances of function f and B the maximum budget for algorithm A on function f ( d ) j .To allow for a fair comparison between instances, the BBOB-benchmark uses target ’precisions’ for their analysis, instead of theraw target values seen by the algorithm. The precision is simplydefined as the difference between the best-so-far- f ( x ) and the globaloptimum. This is done to make runtime comparisons betweendifferent instances and even different functions possible. Since the set of available algorithms from the BBOB-competitions isquite large, several issues in terms of data consistency arise. Whenprocessing the algorithms, we found that a small subset have issuessuch as incomplete files or missing data. We decided to ignorethese algorithms, and work only with the ones which were madeavailable within the IOHanalyzer tool [9]. This leaves us with a setof 182 out of 226 possible algorithms to do our analysis.There are some caveats to this data, mostly related to the lack of aconsistent policy for submission to the competitions over the years.For example, the 2009 competition required submission of 3 runson 5 instances each, while the 2010 version changed this to 1 run on15 instances. In theory, the instances should have very little impacton the performance of the algorithms, as they are selected in sucha way to preserve the characteristics of the functions. However, inpractice there has been some debate about the impact of instanceson algorithm performance, claiming that the landscapes of differentinstances of the same function can look significantly different to analgorithm [18, 26, 29]. In the following, we ignore this discussionand assume that performance is not significantly impacted by theinstances.Another issue with the dataset are the widely inconsistent bud-gets for the different algorithms. These can be as low as 50 D and aslarge as 10 D . However, since we use a fixed-target perspective tostudy the performance of the algorithms, these differences are notvery impactful.Since the BBOB-competitions see an optimizer as having ’solved’an optimization problem when reaching a target precision of 10 − ,many of the algorithms will stop their runs after reaching this pointto avoid unnecessary computation. Because of this, we will usethe same target value in our computations. However, for someof the more difficult functions, this target can be challenging toreach within their budget. To avoid the problem of dealing withalgorithms without any finished runs, we only consider an algo-rithm in our analysis when it has at least 15 runs on the function,of which at least one managed to reach the target 10 − . Figure 1plots the number of algorithms per each function/dimension pair ECCO ’20, July 8–12, 2020, Cancn, Mexico D. Vermetten, H. Wang, T. B¨ack, and C. Doerr F F F F F F F F F F F F F F F F F F F F F F F F
02D 03D 05D 10D 20D 40D
Function ID N u m b e r o f a l go r it h m s Figure 1: Number of algorithms with at least 15 independentruns and at least one them reaching the target ϕ = − . that satisfy all the requirements mentioned above. We observelarge discrepancies between functions and dimensions, with thenumber of admissible algorithms ranging from 4 to 155, and notethat there are no algorithms which are admissible on all functionsin all dimensions. In this work, we will restrict the dynAS problem on BBOB-functionsto using policies which switch algorithms based on the target pre-cisions hit. To get an indication for the amount of improvementwhich can be gained by dynAC over static algorithm configuration,we use the BBOB-data to theoretically simulate a simple policywhich only implements a single switch of algorithm. We can definethis as follows:
Definition 3.1 (Single-Switch dynAS).
Let f ( d ) be a BBOB-functionin dimension d and A the corresponding portfolio of admissiblealgorithms. A single-split policy is defined as the triple ( A , A , τ ) ∈A × A × Φ , where Φ = (cid:110) − . i ) | i ∈ { , . . . , } (cid:111) is the set of ad-missible splitpoints. This corresponds to the policy which startsthe optimization procedure with algorithm A , and run this untiltarget τ is reached, after which the algorithm is changed to A .The performance of this single switch method can then be calcu-lated as follows: T ( f ( d ) , A , A , τ , ϕ ) = ERT ( A , f ( d ) , τ ) + ERT ( A , f ( d ) , ϕ ) − ERT ( A , f ( d ) , τ ) Where ϕ is the final target precision we want to reach. For theBBOB-functions, we set ϕ = − , as noted in Section 3.1.Generally, to assess the performance of an algorithm selection method, its performance can be compared to the Single Best Solver(SBS) , which can be defined as follows:
Definition 3.2 (Single Best Solver).
For each dimension d ∈ D , wehave: SBS static (F ( d ) ) = arg min A ∈A (cid:213) f ∈F PERF ( A , f ( d ) , ϕ ) Often, ERT is used as the performance function, but this value candiffer widely between functions, leading to a biased weighting. Toavoid this, we can instead use the ranking of ERT per function, togive equal importance to every function. Note that we have finaltarget precision ϕ = − .While this SBS has a good average performance, it can easily bebeaten by a decent algorithm selection technique. As such, a betterbaseline for performance is needed. This is the theoretically bestalgorithm selection method, which is called the Virtual Best Solver.This can defined as follows: Definition 3.3 (Static Virtual Best Solver (VBS static )).
For eachfunction f ∈ F and dimension d ∈ D , we have:VBS static ( f ( d ) ) = arg min A ∈A PERF ( A , f ( d ) ) For the BBOB functions, we use PERF ( A , f ( d ) ) = ERT ( A , f ( d ) , ϕ ) with ϕ = − .Note that the VBS static will always perform at least as good as theSBS, and theoretically gives an upper bound for the performance ofany real implementation of algorithm selection techniques. Thus,the difference between SBS and VBS static gives an indication ofthe maximal possible performance gained by algorithm selection.For the BBOB-data, the relative ERT between these two methodsis visualized in Figure 2. From this, we see that the differencescan be extremely large, highlighting the importance of algorithmselection.Similar to the way we defined VBS static , we can define a DynamicVirtual Best Solver, VBS dyn , as follows: Definition 3.4 (Dynamic Virtual Best Solver).
For each BBOB-function f ∈ F and dimension d ∈ D , we have:VBS dyn ( f ( d ) ) = arg min ( A , A , τ )∈(A×A× Φ ) T ( f ( d ) , A , A , τ , ϕ ) Since the number of algorithms considered in this paper is relativelylarge, many of the results are only shown for a subset of functions,dimensions or algorithms. The complete data is made availableat [38]. An example of the available data is also shown in Table 1.
Before investigating the possible improvements to be gained bydynamic algorithm selection, we investigate the performance ofthe static algorithms from the BBOB-dataset. To achieve this, welook at the distribution of ERTs among the BBOB-functions. Fordimension 5, this is visualized in Figure 3. This figure shows thelarge differences in performance, both between the algorithms aswell as between the different functions. We marked the performanceof the VBS static and VBS dyn , and see that their differences also varylargely between functions.To zoom in on the differences between the VBS static and VBS dyn we see in Figure 3, we can compute for each function, dimensionand corresponding algorithm portfolio the relative ERT of a the Note that for function F05, the linear slope, most algorithms simply move outside thesearch-space to find an optimal solution, which is accepted by the BBOB-competitions,but leads to a disadvantage to those algorithms which respect the bounds. owards Dynamic Algorithm Selection GECCO ’20, July 8–12, 2020, Cancn, Mexico
FID VBS static
ERT of VBS static A A ( τ ) ERT of VBS dyn speedup1 fminunc 13.0 HMLSL HCMA 1.2 6.6 1.972 LSfminbnd 94.7 BrentSTEPrr LSfminbnd 2.0 52.4 1.813 BrentSTEPrr 315.5 STEPrr BrentSTEPif -0.2 246.8 1.284 BrentSTEPif 763.9 STEPrr BrentSTEPif -0.2 578.1 1.325 MCS 10.8 ALPS MCS 1.8 6.0 1.806 MLSL 1050.9 fmincon GLOBAL -7.0 928.2 1.137 PSA-CMA-ES 1129.8 GP5-CMAES PSA-CMA-ES 0.0 792.3 1.438 fminunc 399.1 OQNLP DE-BFGS 0.6 304.7 1.319 fminunc 188.3 fminunc DE-AUTO 0.0 152.3 1.2410 DTS-CMA-ES 262.4 fmincon DTS-CMA-ES -2.0 199.8 1.3111 DTS-CMA-ES 268.3 HMLSL DTS-CMA-ES -2.2 153.6 1.7512 NELDERDOERR 1909.7 HMLSL BFGS-P-StPt -3.2 1041.5 1.8313 IPOPsaACM 835.1 DE-AUTO IPOPsaACM -3.6 661.7 1.2614 DTS-CMA-ES 546.6 DE-BFGS DE-SIMPLEX -6.0 348.6 1.5715 PSA-CMA-ES 10029.7 LHD-10xDefault-MATSuMoTo PSA-CMA-ES 0.4 6982.4 1.4416 IPOPsaACM 6767.1 GLOBAL CMA-ES-TPA -0.4 5115.0 1.3217 PSA-CMA-ES 4862.3 PSA-CMA-ES IPOP400D -5.8 4201.8 1.1618 PSA-CMA-ES 6717.4 PSA-CMA-ES CMA-ES multistart -5.2 5687.3 1.1819 DTS-CMA-ES 18768.0 OQNLP DTS-CMA-ES -1.6 463.0 40.5420 DEctpb 10670.3 DEctpb OQNLP -0.4 3360.7 3.1821 GLOBAL 2095.5 MLSL NELDERDOERR 0.0 1209.8 1.7322 GLOBAL 1079.9 RAND-2xDefault-MATSuMoTo GLOBAL 0.4 844.1 1.2823 CMA-ES-MSR 18971.4 DTS-CMA-ES SSEABC -2.6 10295.0 1.8424 OQNLP 285173.0 GP5-CMAES CMAES-APOP-Var2 0.0 52387.0 5.44
Table 1: Relative gain of optimal single-switch dynamic algorithm combination VBS dyn over the best static algorithm VBS static for all 24 BBOB functions in dimension 5. ERT values are computed from data available at https://coco.gforge.inria.fr/doku.php?id=algorithms-bbob . We only consider algorithms with at least 15 runs, one of which reaching target precision ϕ = − , whichis also the target used for the ERT calculations. The full version of this table, also for other dimensions, is available at [38].Abbreviations: FID = function ID (as in [12], τ = splitpoint target, speedup = ERT stat/ERT dyn. We also shortened DTS-CMA-ES 005-2pop v26 1model to DTS-CMA-ES for readability
2D 3D 5D 10D 20DF24F23F22F21F20F19F18F17F16F15F14F13F12F11F10F9F8F7F6F5F4F3F2F1 11.522.539.1 1 15 1.2 1.23.1 5 3.2 3.2 8.37.4 6.7 2.5 2.3 2.68.8 1650 5.4 5 4.72 1.8 1.2 1.9 1.61 5.9 1.8 2.3 1.86.6 1.4 1.7 1.1 11.7 4.1 5.7 2 1.52.3 5.7 12 3.9 4.31.3 2.7 7 1.2 1.11.2 3 6.1 1.4 11.3 47 2 1 1.11 1.6 3.4 1 11 2 3.5 1.2 1.12.9 1.6 2.3 2.4 31 3 1.3 1.4 3.15.2 2.4 1.8 4.6 329 2.2 1.9 3.8 2.321 2.8 8.9 2.7 32.7 2.6 10 6.6 32 6.3 6.8 14 271.2 22 20 30 561.2 3.7 1.6 1.3 25.8 58 59 1.1 16
Figure 2: Relative ERT of the SBS over the VBS static . Theselected SBS are: Nelder-Doerr (2D), HCMA(3, 10 and 20D)and BIPOP-aCMA-STEP (5D). Dimension 40 was removed be-cause no algorithm hit the final target on all functions inthis dimension.
Single-Switch VBS dyn over VBS static . Specifically, this is calcu-lated as
ERT ( VBS dynamic ( f ( d ) )) ERT ( VBS static ( f ( d ) )) . This value is shown for each (function,dimension)-pair in Figure 4. From this figure, we can see that formost functions, the improvements when using a single configura-tion change are quite large. Especially for the functions which aretraditionally considered more difficult for a black-box optimization F01 F02 F03 F04 F05 F06 F07 F08 F09 F10 F11 F12 F13 F14 F15 F16 F17 F18 F19 F20 F21 F22 F23 F24101001e+31e+41e+51e+61e+71e+8
VBS_StaticVBS_Dynamic
Function ID E R T Figure 3: Distribution of ERTs among all algorithms for all24 BBOB-functions in dimension 5. Please recall from Fig. 1that the number of data points varies between functions.Also shown are the ERTs of the VBS static and VBS dyn . algorithm to solve, the possible improvement is massive. In termsof the median over all (function, dimension)-pairs, the VBS dyn is1 .
49 faster than the VBS static . Since the VBS dyn shows a lot of potential improvement over the clas-sical VBS static , it makes sense to study its behaviour in more detail.To achieve this, we can zoom in on a single (function, dimension)-pair and study the behaviour of the VBS dyn and split algorithmconfigurations in general. In Figure 5, we show the ERT of thebest possible switch between any combination of algorithms in ourportfolio A , on function 21 in dimension 10. This figure showssome clear patterns in the horizontal and vertical lines. A horizontalline, such as the one for the MLSL-algorithm [21], indicates that analgorithm adds to the performance of most algorithms by being the ECCO ’20, July 8–12, 2020, Cancn, Mexico D. Vermetten, H. Wang, T. B¨ack, and C. Doerr
2D 3D 5D 10D 20D 40DF24F23F22F21F20F19F18F17F16F15F14F13F12F11F10F9F8F7F6F5F4F3F2F1 11.522.533 2.5 2 1.9 5.3 1.71.1 1.5 1.8 1.9 1.9 1.31.3 1.5 1.3 1.3 1.5 3.11.5 1.6 1.3 1.4 1.6 1.73.4 3.2 1.8 1.5 1.8 11 1 1.1 1.1 1.1 11.9 1.5 1.4 1.4 1.2 1.31.6 1.2 1.3 1.2 1.2 1.21.2 1.3 1.2 1.2 1.2 1.31.4 1.2 1.3 1.2 1.5 1.41.4 1.7 1.7 2.5 3 5.71.5 1.7 1.8 1.3 1.3 1.11.2 1.3 1.3 1.2 1.5 1.91.1 1.5 1.6 1.8 1.7 1.51.3 4.1 1.4 2.9 4.1 5.11.6 1.3 1.3 1.4 1.3 1.31.3 1.2 1.2 1.2 1.2 1.51.5 1.4 1.2 1.1 1.3 1.31.3 13 41 8.8 3.7 4.71.1 2.5 3.2 1.9 17 1.93.4 2.6 1.7 1.9 6.7 5.83.7 1.5 1.3 3.1 7.9 1.91.5 1.5 1.8 2.1 2.3 2.33.9 3.4 5.4 5.4 1.3 2.5
Figure 4: Heatmap of the ratio of ERTs between the VirtualBest Static Solver and the Virtual Best Dynamic Solver, foreach (function, dimension)-pair. A L P S B I P O P - s a A C M - k B I P O P s a A C M C M A C M A - E S m u l t i s t a r t C M A E S - A P O P - V a r f m i n c o n F U LL N E W U O A G P C X G L O B A L I P O P - S E P - C M A - E S I P O P D J A D E J A D E c t pb M L S L N B I P O P a C M A N E L D E R D O E RR R - S H A D E - e R o s e n b r o c k u BB D E - tt b ALPSBIPOP-saACM-kBIPOPsaACMCMACMA-ES multistartCMAES-APOP-Var3fminconFULLNEWUOAG3PCXGLOBALIPOP-SEP-CMA-ESIPOP400DJADEJADEctpbMLSLNBIPOPaCMANELDERDOERRR-SHADE-10e5RosenbrockuBBDE-ttb 00.511.52 A2 A Figure 5: Relative ERT of configuration switches relative toVBS static , for function 21 in 10 dimensions. The X- and Y-axes indicate algorithms selected as A and A respectively.Larger values (red) indicate better algorithm combinations. A -algorithm. This can be interpreted as having a good exploratorysearch behaviour, but poor exploitation. There are also verticallines present, which indicate the algorithms which perform wellas A -algorithms. These are less pronounced than the horizontallines, which might indicate that the choice of A algorithms hasless impact on the performance than the choice of A . We see that there are different algorithms which perform wellas either the first or second part of the search. This gives rise to thequestion of how to quantify these differences, and more generally,how to quantify the benefit which can be gained by selecting analgorithm as A or A . This can be done by executing the followingsteps to compute a quantitative value for the benefit gained byselecting an algorithm for a part of the search: Definition 4.1 (Improvement-values).
The initial performance value I and finishing performance value I of algorithm A on function f ( d ) can be defined as: I ( A ) = min A ∈A , τ ∈ Φ T ( A , A , τ , ϕ ) min A , A ∈A , τ ∈ Φ T ( A , A , τ , ϕ ) I ( A ) = min A ∈A , τ ∈ Φ T ( A , A , τ , ϕ ) min A , A ∈A , τ ∈ Φ T ( A , A , τ , ϕ ) Note that for the VBS dyn = ( A , A , τ ) , we always have I ( A ) = = I ( A ) , and values can not be below 1. Intuitively, the largerthe value of I , the worse the algorithm can perform as the firstpart of the search, and similarly for I .The values of I and I for dimension 5 are shown in Figures 6and 7 respectively. To ensure the readability of the figures, only asubset of algorithms is chosen. This is done by selecting the algo-rithm with the best value for each function, and then adding to it theset of algorithms which have the best average value over all func-tions . From these figures, we see clear differences, both betweenfunctions and between algorithms. While some algorithms occur inboth Figures 6 and 7, many are included only once, indicating thatthey are relatively good choices for one part of the search, but notthe remainder. The clearest example of this is HMLSL [30], whichperforms very well as A , but has relatively high I -values. This iscaused by the fact that this algorithm typically converges quickly toa value close to the optimum, but has issues in the final exploitationphase, thus only being beneficial to use at the start of the search.We also notice that in general, the I -values are much lower acrossall algorithms, indicating that the choice of starting algorithm isthe most important for dynAS, while most good algorithms canprovide similar benefits to the final part of the search. Since the algorithm space we consider is quite large, it can be chal-lenging to gain insights into the individual algorithms. To show thatdynamic algorithm selection is also applicable to smaller portfolio’s,we limit ourselves to 5 algorithms. These are representative of somewidely used algorithm families: Nelder-Doerr [8], DE-Auto [40],Bipop-aCMA-Step [22], HMLSL [30] and PSO-BFGS [41].With thisreduced algorithm portfolio, we can study the improvements overtheir respective VBS static in more detail, and find interesting algo-rithms combinations to explore further.In Figure 8, we show the relative improvement in ERT overVBS static of the best combination of two algorithms. In each subplot,all 24 functions are represented. Note that the diagonal representsthe static algorithms, which can never lead to an improvement overthe VBS static . We notice some clear trends in this figure. Specifically, Missing values and values larger than are set to to reduce the large impact ofoutliers on the average. owards Dynamic Algorithm Selection GECCO ’20, July 8–12, 2020, Cancn, Mexico BIPOP-aCMA-STEPBIPOP-saACM-kBIPOPsaACMDE-AUTOHCMAHMLSLIPIP-10DDrIP-500IPOPsaACMKL-BIPOP-CMA-ESNBIPOPaCMAR-SHADE-10e5tanytexp
Function ID
Figure 6: I -values for a group of 15 selected algorithms indimension . Darker colors correspond to better values. BIPOP-aCMA-STEPBIPOP-saACM-kBIPOPsaACMCMA_mhDEDE-AUTODEAE CCOVFAC studyDEbHCMAIPOPsaACMKL-BIPOP-CMA-ESMEMPSODENBIPOPaCMAR-SHADE-10e5VNS
Function ID
Figure 7: I -values for a group of 15 selected algorithms indimension . Darker colors correspond to better values. we notice that using HMSLS as A is rarely effective, while it pro-vides large benefits when used in the initial part of the search. Wealso note that Nelder-Doerr has the reverse behaviour, seeminglyperforming much better in the final exploitation phase.To illustrate the configuration switches which can be consideredin this algorithm portfolio, we can zoom in on function 12 in di-mension 3 and look at the fixed-target curve showing ERT. This isdone in Figure 9, where we also indicate the best switching pointsbetween algorithms. This figure highlights the different behaviorsof the algorithms in the portfolio, and thus indicates where switch-ing algorithms would be beneficial. The best possible switch in thisfunction would occur from PSO-BFGS to Nelder-Doerr, at target10 − . , leading to a relative speedup of 1 .
76 over VBS static .To decide which algorithms to use in an algorithm portfolio suchas the one used here, two main ways of selecting the algorithms arepossible. The first is to use some knowledge about the algorithms todetermine which are important. This is useful for initial exploration,but might lead to useful algorithms being ignored. Instead, onecan use performance information, such as the I and I -values, toprovide some initial representation of the usefulness of algorithmsto the portfolio. This approach is much more generic, however the NELDERDOERR DE-AUTO BIPOP-aCMA-STEP HMLSL PSO-BFGS P S O - B F G S H M L S L B I P O P - a C M A - S T E P D E - A U T O N E L D E R D O E RR Figure 8: Overview of the best possible ERTs of the combi-nation of algorithms A and A over VBS static . Each plot rep-resents a single A (X-axis), A (Y-axis) combination, whereeach bar represents a single function, in dimension 3. Valuesare capped at 2. choice of measures can be challenging. For example, the I and I measures are hard to extend to more general k -switch dynASmethods. Instead, an extension of marginal contributions [42] andrelated concepts such as measures building on Shapley values (likethose suggested in [11]) would capture algorithm contribution to aportfolio in a much more robust sense, and thus be useful additionsto the dynAS setting. Summary.
The previous results have shown that there is still alarge amount of improvement possible over the VBS static by usingdynamic algorithm selection. We have shown several methodsto gain insights into the differences between different algorithmsand functions. However, the results shown in the previous sec-tions rely on an underlying assumption of feasibility of algorithmswitching. For many algorithms, this switching mechanism can beimplemented in a relatively straightforward manner, i.e. betweendifferent population-based algorithms, such as different CMA-ESvariants, for which the algorithm switching methods have alreadybeen implemented [37].
Warm-start.
For other algorithms combinations, a dynamic switchduring the optimization procedure might be more challenging. Forexample, a switch from a single-solution algorithm to a population-based one gives rise to an information deficit, which needs to bedealt with to properly initialize the new population. Because ofthis, the gains indicated by simply combining the ERT values mightbe tough to achieve in practice.
ECCO ’20, July 8–12, 2020, Cancn, Mexico D. Vermetten, H. Wang, T. B¨ack, and C. Doerr
100 1 0.01 1e−4 1e−6 1e−8 BIPOP-aCMA-STEP DE-AUTO HMLSLNELDERDOERR PSO-BFGS
Target E R T Figure 9: ERT-curves for a selected algorithm portfolioof size 5 on F12 in 3D. Markers indicate optimal switchpoints between algorithms. Their color and symbol indi-cate the starting and finishing algorithms respectively.(star= Nelder-Doerr, triangle = DE-AUTO, cross = BIPOP-aCMA-STEP, square = HMLSL and pentagon = PSO-BFGS).
More generally, internal parameters are different between al-gorithms. So the first challenge to overcome is that one needs todecide how to “warm-start” the algorithms, to assure an optimalinternal state for the required phase of the optimization process.To be able to achieve the performance of the VBS dyn , such warm-start techniques will need to be implemented without the need ofadditional function evaluations, which could be a big challenge.We would considering to use reinforcement learning approachesto be a promising first step for this task, but since those are quiteexpensive in terms of computational cost, we hope to see otherapproaches evolve in the near future.
Stochasticity.
Assuming such warm-start mechanisms are imple-mented, as was previously done for example within CMA-ES, it hasbeen shown that the theoretical improvements can still be toughto achieve in practice [37]. This is largely caused by the fact thathitting times are stochastic with relatively large variances, whichcan cause ERT to be unstable. When selecting the ( A , A , τ ) -triple,differences in ERT might be obscured by the variance of the hittingtimes, leading to a worse performance than expected. These effectsmight become even more important when dealing with larger algo-rithm spaces, or when incorporating hyperparameters in the search(see paragraph Hyperparameter tuning ). Analyzing the robustnessof common solvers therefore seems to be an essential building blockfor the development of reliable dynAC approaches.
Switch point.
Another challenge which needs to be overcometo achieve effective dynamic algorithm selection is the questionhow to identify suitable switching points. In this work we usedtarget precision, which is usually not applicable in practice, sincethe algorithm has no knowledge about the precise value of theoptimum. Because of this, we would need to find some other wayto use the knowledge of the algorithm to determine when to switch,i.e., the state of internal parameters, landscape features computed from additionally or previously evaluated points, the evolution offitness values, population diversity, etc.
True dynamic switching.
While improving the way a switchingpoint is detected is a big challenge to overcome, it also providesnew opportunities to improve performance. The estimates shownin this paper consider only a single algorithm switch, whereas atruly dynamic approach could benefit from switching more often,to fully exploit the differences in search behaviour of the differentalgorithms.
Hyperparameter tuning.
A second factor of improvement cancome from adding hyperparameter tuning into the dynamic pro-cess; i.e., when moving from the algorithm selection setting to adynamic variant of
Combined Algorithm Selection and Hyperpa-rameter optimization (CASH [34, 39]). A dynamic CASH approachwould allow the algorithms to specialize even more, so they canfocus even more on performing as good as possible on their specificpart of the optimization process.
Extensions.
As any benchmark study, our results are – for thetime being – limited to the 24 noiseless BBOB functions. Extendingthem to other classes of numerical black-box optimization prob-lems forms another important avenue for future research. In thiscontext, we consider supervised learning approaches building onexploratory landscape analysis [25] as particularly promising. It haspreviously been shown to yield promising results for the task of con-figuring the hyper-parameters of CMA-ES [3]. Note, though, thatall existing studies concentrate on static algorithm configurationand/or selection. We would therefore need to extend exploratorylandscape analysis to the dynamic setting. First steps into this di-rection have been made in [16], where it is shown that the fitnesslandscapes, as seen by the algorithm, can change quite drasticallyduring the run.
Short-term.
All the objectives listed above are quite ambitious.We therefore also formulate a few short-term goals for our research.Building on the techniques used to select interesting algorithmsin Section 4.3, we aim to create smaller algorithm portfolio’s ofalgorithms for intial implementations of dynAS. This could be donebased on techniques studied in this paper, or using measures likethe Shapley value [11], allowing for much smaller portfolios whichnonetheless capture the different performances of the algorithms.With such a portfolio we can then more efficiently carry out re-search on the problems mentioned above, i.e., how to warm-startthe algorithms and how to decide when to switch from one algo-rithm to another.
ACKNOWLEDGMENTS
This work has been supported by the Paris Ile-de-France region.
REFERENCES [1] Anne Auger, Dimo Brockhoff, Nikolaus Hansen, Tea Tuar, and KonstantinosVarelas. 2020. Data from BBOB-workshops and competitions on 24 noiselessfunctions. https://coco.gforge.inria.fr/doku.php?id=algorithms-bbob.[2] Anne Auger and Nikolaus Hansen. 2005. A restart CMA evolution strategy withincreasing population size. In
Proc. of Congress on Evolutionary Computation(CEC’05) . 1769–1776. https://doi.org/10.1109/CEC.2005.1554902[3] Nacim Belkhir, Johann Dr´eo, Pierre Sav´eant, and Marc Schoenauer. 2017. Perinstance algorithm configuration of CMA-ES with limited budget. In
Proc. of owards Dynamic Algorithm Selection GECCO ’20, July 8–12, 2020, Cancn, Mexico
Genetic and Evolutionary Computation (GECCO’17) . ACM, 681–688. https://doi.org/10.1145/3071178.3071343[4] Andr´e Biedenkapp, H. Furkan Bozkurt, Frank Hutter, and Marius Lindauer. 2019.Towards White-box Benchmarks for Algorithm Control.
CoRR abs/1906.07644(2019). arXiv:1906.07644 http://arxiv.org/abs/1906.07644[5] Ilhem Boussa¨ıd, Julien Lepagnot, and Patrick Siarry. 2013. A survey on optimiza-tion metaheuristics.
Information Sciences
237 (2013), 82–117.[6] Edmund K. Burke, Michel Gendreau, Matthew R. Hyde, Graham Kendall, GabrielaOchoa, Ender ¨Ozcan, and Rong Qu. 2013. Hyper-heuristics: a survey of the stateof the art.
JORS
64, 12 (2013), 1695–1724. https://doi.org/10.1057/jors.2013.71[7] Luc Devroye. 1972.
The compound random search . Ph.D. dissertation, PurdueUniv., West Lafayette, IN.[8] Benjamin Doerr, Mahmoud Fouz, Martin Schmidt, and Magnus Wahlstr¨om.2009. BBOB: Nelder-Mead with resize and halfruns. In
Proc. of Genetic andEvolutionary Computation (GECCO’09) , Franz Rothlauf (Ed.). ACM, 2239–2246.https://doi.org/10.1145/1570256.1570312[9] Carola Doerr, Hao Wang, Furong Ye, Sander van Rijn, and Thomas B¨ack. 2018.IOHprofiler: A Benchmarking and Profiling Tool for Iterative OptimizationHeuristics. arXiv e-prints:1810.05281 (Oct. 2018). arXiv:1810.05281 https://arxiv.org/abs/1810.05281 The BBOB datasets from [1] are available in the web-basedinterface of IOHanalyzer at http://iohprofiler.liacs.nl/.[10] Agoston Endre Eiben, Robert Hinterding, and Zbigniew Michalewicz. 1999.Parameter control in evolutionary algorithms.
IEEE Transactions on EvolutionaryComputation
Proc. of the AAAI Conference on Artificial Intelligence arXiv:1605.03560 [cs] (May 2016). http://arxiv.org/abs/1605.03560 arXiv: 1605.03560.[13] Nikolaus Hansen, Anne Auger, Raymond Ros, Steffen Finck, and Petr Poˇs´ık.2010. Comparing results of 31 algorithms from the black-box optimizationbenchmarking BBOB-2009. In
Proc. of Genetic and Evolutionary Computation(GECCO’10) . ACM, 1689–1696.[14] Nikolaus Hansen and Andreas Ostermeier. 1996. Adapting arbitrary normalmutation distributions in evolution strategies: the covariance matrix adaptation.In
CEC . 312–317. https://doi.org/10.1109/ICEC.1996.542381[15] Nikolaus Hansen and Andreas Ostermeier. 2001. Completely DerandomizedSelf-Adaptation in Evolution Strategies.
Evolutionary Computation
9, 2 (2001),159–195. https://doi.org/10.1162/106365601750190398[16] Anja Jankovi´c and Carola Doerr. 2019. Adaptive landscape analysis. In
Proc. ofGenetic and Evolutionary Computation (GECCO’19) . ACM, 2032–2035. https://doi.org/10.1145/3319619.3326905[17] Pascal Kerschke and Heike Trautmann. 2017. Automated Algorithm Selection onContinuous Black-Box Problems By Combining Exploratory Landscape Analysisand Machine Learning. arXiv preprint arXiv:1711.08921 (2017).[18] P. Kerschke and H. Trautmann. 2019. Automated Algorithm Selection onContinuous Black-Box Problems by Combining Exploratory Landscape Anal-ysis and Machine Learning.
Evolutionary Computation
27, 1 (2019), 99–127.https://doi.org/10.1162/evco a 00236[19] Benjamin Lacroix and John McCall. 2019. Limitations of Benchmark Sets andLandscape Features for Algorithm Selection and Performance Prediction. In
Proc.of Genetic and Evolutionary Computation (GECCO’19) . Association for ComputingMachinery, New York, NY, USA, 261fi?!262. https://doi.org/10.1145/3319619.3322051[20] Fernando G. Lobo, Cl´audio F. Lima, and Zbigniew Michalewicz (Eds.). 2007.
Pa-rameter Setting in Evolutionary Algorithms . Studies in Computational Intelligence,Vol. 54. Springer.[21] Marco Locatelli and Fabio Schoen. 1999. Random Linkage: a family of accep-tance/rejection algorithms for global optimisation.
Mathematical Programming
85, 2 (1999).[22] Ilya Loshchilov, Marc Schoenauer, and Michele S`ebag. 2013. Bi-Population CMA-ES Agorithms with Surrogate Models and Line Searches. In
Proc. of Genetic andEvolutionary Computation (GECCO’13) . Association for Computing Machinery,New York, NY, USA, 1177fi?!1184. https://doi.org/10.1145/2464576.2482696[23] Katherine Mary Malan and Irene Moser. 2019. Constraint Handling Guided byLandscape Analysis in Combinatorial and Continuous Search Spaces.
Evolution-ary Computation
27, 2 (2019), 267–289. https://doi.org/10.1162/evco a 00222[24] Jorge Maturana, ´Alvaro Fialho, Fr´ed´eric Saubion, Marc Schoenauer, Fr´ed´ericLardeux, and Mich`ele Sebag. 2012. Adaptive Operator Selection and Manage-ment in Evolutionary Algorithms. In
Autonomous Search , Youssef Hamadi, EricMonfroy, and Fr´ed´eric Saubion (Eds.). Springer, 161–189. https://doi.org/10.1007/978-3-642-21434-9 7[25] Olaf Mersmann, Bernd Bischl, Heike Trautmann, Mike Preuss, Claus Weihs, andG¨unter Rudolph. 2011. Exploratory landscape analysis. In
Proc. of Genetic and Evolutionary Computation (GECCO’11) . ACM, 829–836.[26] Mario A. Mu˜noz, Michael Kirley, and Saman K. Halgamuge. 2015. ExploratoryLandscape Analysis of Continuous Space Optimization Problems Using Infor-mation Content.
IEEE Trans. Evolutionary Computation
19, 1 (2015), 74–87.https://doi.org/10.1109/TEVC.2014.2302006[27] Mario A. Mu˜noz and Kate Amanda Smith-Miles. 2017. Performance Analysis ofContinuous Black-Box Optimization Algorithms via Footprints in Instance Space.
Evolutionary Computation
25, 4 (2017). https://doi.org/10.1162/evco a 00194[28] Mario A Mu˜noz, Yuan Sun, Michael Kirley, and Saman K Halgamuge. 2015.Algorithm selection for black-box continuous optimization problems: A surveyon methods and challenges.
Information Sciences
317 (2015), 224–245.[29] Mario Andrs Muoz Acosta, Michael Kirley, and Kate Smith-Miles. 2018. Reliabilityof Exploratory Landscape Analysis. (06 2018). https://doi.org/10.13140/RG.2.2.23838.64327[30] L´aszl´o P´al. 2013. Benchmarking a hybrid multi level single linkagealgorithmon the bbob noiseless testbed. In
Proc. of Genetic and Evolutionary Computation(GECCO’15) . 1145–1152.[31] Erik Pitzer and Michael Affenzeller. 2012.
A Comprehensive Survey on FitnessLandscape Analysis . Springer Berlin Heidelberg, Berlin, Heidelberg, 161–191.https://doi.org/10.1007/978-3-642-23229-9 8[32] Ingo Rechenberg. 1973.
Evolutionsstrategie . Friedrich Fromman Verlag (G¨untherHolzboog KG), Stuttgart.[33] Michael A. Schumer and Kenneth Steiglitz. 1968. Adaptive step size randomsearch.
IEEE Trans. Automat. Control
13 (1968), 270–276.[34] Chris Thornton, Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. 2013.Auto-WEKA: Combined selection and hyperparameter optimization of classifica-tion algorithms. In
ACM SIGKDD . ACM, 847–855.[35] Sander van Rijn, Carola Doerr, and Thomas B¨ack. 2018. Towards an AdaptiveCMA-ES Configurator. In
PPSN (Lecture Notes in Computer Science) , Vol. 11101.Springer, 54–65. https://doi.org/10.1007/978-3-319-99253-2 5[36] Sander van Rijn, Hao Wang, Matthijs van Leeuwen, and Thomas B¨ack. 2016.Evolving the structure of Evolution Strategies. In
SSCI . 1–8. https://doi.org/10.1109/SSCI.2016.7850138[37] Diederick Vermetten, Sander van Rijn, Thomas B¨ack, and Carola Doerr. 2019.Online selection of CMA-ES variants. In
Proc. of Genetic and Evolutionary Com-putation (GECCO’19) . ACM, 951–959. https://doi.org/10.1145/3321707.3321803[38] Diederick Vermetten, Hao Wang, Thomas B¨ack, and Carola Doerr. 2020. Githubrepository with Project Data. (2020). https://github.com/Dvermetten/BBOBDynAS.[39] Diederick Vermetten, Hao Wang, Carola Doerr, and Thomas B¨ack. 2020. Inte-grated vs. Sequential Approaches for Selecting and Tuning CMA-ES Variants.In
Proc. of Genetic and Evolutionary Computation Conference (GECCO’20) . ACM.https://doi.org/10.1145/3377930.3389831[40] Costas Voglis, Grigoris S. Piperagkas, Konstantinos E. Parsopoulos, Dimitris G.Papageorgiou, and Isaac E. Lagaris. 2012. MEMPSODE: an empirical assessmentof local search algorithm impact on a memetic algorithm using noiseless testbed.In
Proc. of Genetic and Evolutionary Computation (GECCO’12) , Terence Soule andJason H. Moore (Eds.). ACM, 245–252. https://doi.org/10.1145/2330784.2330820[41] Costas Voglis, Grigoris S. Piperagkas, Konstantinos E. Parsopoulos, Dimitris G.Papageorgiou, and Isaac E. Lagaris. 2012. MEMPSODE: comparing parti-cle swarm optimization and differential evolution within a hybrid memeticglobal optimization framework. In
Proc. of Genetic and Evolutionary Compu-tation (GECCO’12) , Terence Soule and Jason H. Moore (Eds.). ACM, 253–260.https://doi.org/10.1145/2330784.2330821[42] Lin Xu, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. 2012. EvaluatingComponent Solver Contributions to Portfolio-Based Algorithm Selectors. In