Codynamic Fitness Landscapes of Coevolutionary Minimal Substrates
CCodynamic Fitness Landscapes of Coevolutionary MinimalSubstrates
Hendrik RichterHTWK Leipzig University of Applied SciencesFaculty of Electrical Engineering and Information TechnologyPostfach 301166, D–04251 Leipzig, Germany.Email: [email protected] 8, 2018
Abstract
Coevolutionary minimal substrates are simple and abstract models that allow studying the relation-ships and codynamics between objective and subjective fitness. Using these models an approach is pre-sented for defining and analyzing fitness landscapes of coevolutionary problems. We devise similaritymeasures of codynamic fitness landscapes and experimentally study minimal substrates of test–based andcompositional problems for both cooperative and competitive interaction.
Coevolutionary scenarios are interesting for at least two reasons. A first is that in natural evolution ofbiological entities the evolutionary development of one species is almost always accompanied by evolutionaryadaption of and changes in other species. Traits, features and abilities in one species do not exist forthemselves, but can only be understood by the coupling with and response to other species’ evolution.Hence, studying natural evolution most likely means analyzing coevolutionary processes. A second reasonis that in artificial evolution used to solve optimization problems by evolutionary search algorithms, designsemploying ideas from coevolution appear to be as intriguing as promising. The advantages of coevolutionarydesigns are particularly seen for solving competitive problems such as in games [6, 18], or cooperative tasksthat require the coordination of several agents such as in some problems related to evolutionary robotics [7,11], or for situations where the fitness function cannot be designed straightforwardly. Early experimentalresults [5, 17] have produced considerable optimism about coevolutionary designs, while more recent works[10, 12] were rather to cast some doubt regarding easily understandable (and applicable) coevolutionaryproblem solvers. The main difficulty appears to be the complex notion of (co–) evolutionary progress andgenetically inheritable superiority. Several concepts have been proposed to entangle this complexity andremedy its effects, see the discussion in Sec. 2.A central issue in evolutionary computation is to have a theoretical framework for describing and under-standing the evolutionary dynamics underlying evolutionary search. One fundamental way for addressingthis issue is the concept of fitness landscapes, which has been successfully applied to gain insight into theevolutionary search processes solving static [8, 20] and dynamic [15] optimization problems. Consequently,fitness landscapes have also been proposed to understand coevolutionary processes [13, 14], while most re-cently it has been suggested to employ dynamic landscapes [16]. In this paper, the concept of dynamiclandscapes is applied and extended. The aim of this approach is twofold. A first is that dynamic landscapesoffer the possibility of studying the dynamics of two major characterizing quantities in coevolutionary algo-rithms, subjective fitness and objective fitness. A second is that the landscapes obtained enable an analytic1 a r X i v : . [ c s . N E ] A p r reatment valid for all possible individuals of a population (for instance using landscape measures, see e.g. [9]for a recent review). These analytic results may establish a quantification for the differences between theobjective landscape describing the problem to be solved and the subjective landscape describing how thecoevolutionary algorithm perceives the problem.In pursuing these aims, Sec. 2 first reviews some of the issues in coevolution and highlights the complexand possibly even pathological behavior that can sometimes be observed in coevolutionary runs. Also,ideas to explain and predict these behavioral features are discussed, namely solution concepts, interactivedomains and objective as well as subjective fitness. In Sec. 3 we consider simple models to be employedin the numerical experiments studying fitness landscapes in coevolution. Such abstract and convenientlyexperimentable coevolutionary models have been named minimal substrates by Watson and Pollack [22];this term is adopted here. The fitness landscapes of these models are reported in Sec. 4. As the respectivelandscapes of the interacting species are coupled and dynamically deforming each other, we refer to suchlandscapes as codynamic. In addition, similarity measures of codynamic fitness landscapes are introducedand experimentally studied. The paper ends with discussing results and drawing conclusions. Coevolutionary algorithms (CEAs) differ fundamentally from evolutionary algorithms (EAs) about the wayfitness is assigned to individuals. The individuals of an EA may inhabit the search space S . For staticoptimization problems, each of its points x ∈ S possesses uniquely a fitness value f ( x ), which is assigned tothe individual if the move operators of the algorithm bring the individual to that point in a given generation k ∈ N . For the search space being metric (or otherwise equipped with a neighborhood structure n ( x )),these elements cast the (static) fitness landscape Λ s = ( S, f ( x ) , n ( x )). Dynamic optimization problemsdeviate from the static view by the fitness of an individual depending on time, which can be linked togenerational time k , that is f ( x, k ). Such a dynamic landscape still consists of search space, fitness functionand neighborhood structure, but additionally includes generational time and rules for changing fitness withtime. Anyway, fitness is always objective in that the search space point (and possibly generational time)alone defines it. In other words, fitness is a property of a search space point, every individual has thesame fitness value if it is situated at the same search space point (and as long as fitness does not changedynamically), and the fitness of one individual does not depend on the fitness of other individuals.Contrary to the objective fitness of EAs, CEAs assign fitness that is subjective. In coevolution thefitness of an individual is obtained with respect to the fitness (and possibly the search space points) ofother individuals. These other individuals, which are called evaluators, do usually not belong to the samepopulation as the individual for which fitness is to be evaluated, but to a coevolving population. As aconsequence, the fitness of an individual at a given generation depends on which evaluators are taken, andon the fitness these evaluators have. As the individuals that form the pool of possible evaluators undergoevolutionary development themselves, the fitness value of a search space point (and hence of the individualsituated at this point) may vary with the selection, which makes the fitness subjective. For describingthe process of obtaining subjective fitness, the framework of interactive domains and solution concepts hasbeen proposed [12]. This framework replaces the fitness function and sets out the rules for assigning fitnessvalues to individuals. The interactive domain defines how the reciprocal actions between individuals ofone population with evaluators from the other are organized and how the solution of the interaction iscalculated. The solution concepts formalizes how the solution translates to (personal or collective) fitnessof the individuals, how these fitnesses can be compared and interpreted over the entire coevolutionary runand whether or not the comparison indicates coevolutionary progress. To establish coevolutionary progress,however, is sometimes difficult. Solving a (maximization) problem means finding the search space pointswith highest fitness – the peaks in the fitness landscape. The objective fitness of EAs allows deducing(evolutionary) progress by a simple comparison of the fitness values – the higher the value the more likelya peak is detected. Also CEAs aim at finding individuals with highest objective fitness. However, thesubjective fitness used to drive the CEA is the result of specific interactions with other coevolving individuals.2t may hence be incomplete and inconsistent with respect to the objective fitness, which is obtainable, atleast in principle, by the combination of all possible interactions. Therefore, numerical experiments withCEAs sometimes show pathological features of behavior devoid of stable progress, for instance cycling,overspecialization and disengagement [10, 12, 22]. All these coevolutionary failures are a direct consequenceof the uncertainty connected to the question whether progress in subjective fitness also implies progress inobjective fitness. To study essential features of coevolution in numerical experiments requires appropriate models. Particularlyfor studying the connection between subjective and objective fitness, it is desirable that both quantities canbe determined in a fast and easy way. Therefore, problem description from application domains such ascoevolutionary games [6, 18] and robotics [7, 11] are less suitable because they may need considerablenumerical setup and the objective fitness is difficult (if at all) calculable. Following this line of thought, itis interesting to ask what minimal structural and behavioral requirements are needed to exhibit complexand relevant coevolutionary dynamics. Such models have been named minimal substrates by Watson andPollack [22]. Accordingly, a coevolutionary minimal substrate is a simple and abstract model of coevolutionwhich defines an interactive domain and a solution concept, exhibits relevant codynamic features and allowsexperimental studies of the relationships between subjective and objective fitness. In the following, and inaddition to the initial model [22], we recall and interpret other coevolutionary models proposed earlier [13, 14]as minimal substrates and introduce some modifications to these models.The optimization problems solvable by CEAs can be classified into two groups: compositional problems(in which fitness of an individual is assigned by an interaction that forms a composite or team) and test–based problems (where the interaction involves a challenge or test) [12]. Next, simple models for bothgroups of coevolutionary problems are discussed. For the group of test–based problems, we consider numbergames [4, 22]. The game studied here has two populations P and P that inhabit the search spaces S x and S y , respectively. Both search spaces are one–dimensional and real–valued. At each generation k = 0 , , , . . . ,the individuals of population P ( k ) can take possible values x ∈ S x and the population P ( k ) may havevalues y ∈ S y . We define identical objective fitness functions over both search spaces, that is f obj ( x ) over S x and f obj ( y ) over S y , which consequently cast objective fitness landscapes. The subjective fitness for bothpopulations is the result of an interactive number game. Therefore, for each calculation of the subjectivefitness f sub ( x ) for an individual from P , a sample s ( P ) of individuals from P is randomly selected. Thissample is statistically independent from the sample for the next calculation. Denote µ the size of the sample s ( P ) out of λ individuals in P in total, with µ ≤ λ . Assigning fitness f sub ( y ) for the individuals of P islikewise but reversed with using statistically independent samples s ( P ) from P .The interactive domain of the number game considered defines that the fitness f sub ( x ) with respect tothe sample s ( P ) is calculated by counting the (mean) number of members in s ( P ) that have a smallerobjective fitness f obj ( s i ( P )), i = 1 , , . . . , µ , than the objective fitness f obj ( x ): f sub ( x ) = 1 µ µ (cid:88) i =1 score( x, s i ( P )) (1)with score( x, s i ) = (cid:26) f obj ( x ) > f obj ( s i )0 otherwise . The fitness f sub ( y ) is calculated accordingly from (1) where y and s i ( P ) replace x and s i ( P ). The subjective fitness (1) has some interesting properties. It is a unitaryfunction f ( x ) = R → [0 ,
1] for every f obj , which eases comparing variants of f sub based on different f obj .The subjective fitness f sub converges to the objective fitness f obj for f obj also being a unitary function, thesample s ( P ) being large, and the distribution of f sub over s i ( P ) matching the distribution of f obj over S x ,where the x ∈ S x should be taken to resemble a uniform distribution of f obj on the interval [0 , f obj ( x ) = x , we obtain the number game introduced by Watson and Pollack [22]. The objective fitnessfunction f obj ( x ) = x has two optima, one minimum and one maximum. These optima, however, are for thesmallest and the largest element in the search space S x , that is, on the boundary of any admissible domain.To numerically obtain these optima in experiments, the locations of the optima require to define (andalgorithmically enforce) a bounded search space. This, in turn, ultimately entails a constrained optimizationproblem and somehow makes the problem setting more complicated than desirable. Therefore, a modificationis considered with the piece-wise linear function f obj ( x ) = (cid:26) x for 0 ≤ x ≤ . . (2)This objective fitness has a minimum at x = 0 with f obj (0) = 0 and a maximum at x = 1 with f obj (1) = 1,and levels off to a mid–level value of f obj ( x ) = 0 . x → ±∞ . As a second example of objective fitnessthe smooth function f obj ( x ) = 12 + x x (3)is taken. It also has two optima, a minimum at x = − f obj ( −
1) = 0 and a maximum at x = 1 with f obj (1) = 1, and also tends to f obj ( x ) = 0 . x large in absolute value. Fig. 1 shows the objective fitnessfunction (the solid line in the graph) for the test–based coevolutionary problems considered. Whereas thesubjective fitness (1) may converge to the objective fitness for the conditions given above, in a coevolutionaryrun both quantities will almost certainly be different. This is because the sample s ( P ) is most likely smallcompared to the population size of P , and even smaller compared to the amount of samples needed to coverthe entire domain of the search space. Fig. 1 gives a realization of the subjective fitness (the dotted line inthe graph). This realization is obtained by drawing a medium size sample ( µ = 100) from a given population( λ = 400) which is uniformly distributed on S x . It can be seen that the subjective fitness resembles theobjective fitness. −0.5 0 0.5 1 1.500.20.40.60.81 x f ( x ) (a) −2 −1.5 −1 −0.5 0 0.5 1 1.5 200.20.40.60.81 x f ( x ) (b)Figure 1: Objective (solid line) and subjective (dotted line) fitness functions of test–based problems. (a)The piece–wise linear function (2). (b) The smooth function (3).For the group of compositional problems, the search spaces S x and S y of the coevolving populationsmay be combined into one shared landscape S = { S x , S y } . This might result in a unique (static) objective4andscape for simple coevolutionary scenarios. The compositional examples considered in this paper alsowork with coevolving populations that are one–dimensional and real–valued. Therefore, combining the twoone–dimensional landscapes leads to a shared two–dimensional objective landscape. This approach canbe found in previous research [13, 14] on understanding coevolutionary phenomena by fitness landscapes.We interpret these examples as compositional minimal substrates and employ ridge functions as suggestedin [13, 14]. The simplest function has one ridge: yx f ( x , y ) (a) −4 −2 0 2 4 −5 0 5−1−0.500.51 yx f ( x , y ) (b)Figure 2: Shared objective fitness functions of compositional problems. (a) The ridge function (4) for n = 8.(b) The sinusoid function (5). f obj ( x, y ) = n + 2 min ( x, y ) − max ( x, y )for 0 ≤ ( x, y ) ≤ nn otherwise , (4)with x, y ∈ R and n is a parameter that sets the size and the hight of the landscape (see Fig. 2a). Thelandscape has a single maximum at f obj ( n, n ) = 2 n and a ridge diagonally from f obj (0 ,
0) = n to f obj ( n, n )that separates two planar surfaces. There are two minima at f obj (0 , n ) = f obj ( n,
0) = 0. Outside the square0 ≤ ( x, y ) ≤ n , the landscape has the mid–level value f obj ( x, y ) = n that ensures that the optima do not lieon the boundary of the admissible domain. Equation (4) is the fitness function for both populations P and P and can be interpreted as the static shared objective fitness landscape defined over S = { S x , S y } . As asecond example the smooth shared objective landscape f obj ( x, y ) = sin( x + y )1 + x + y (5)is analyzed; see Fig. 2b. It has a global minimum at f obj ( − . , − . − . f obj (0 . , . . f obj ( x, y ) = 0 for ( x, y ) large in absolute value.The subjective fitness for each population is calculated by using the shared objective fitness functions(4) or (5) and inserting a value obtained by a metric on the population of the respective other populationinstead of the required second variable. Thus, for calculating the subjective fitness of P , a metric m ( P )on P is taken: f sub ( x ) = f obj ( x, m ( P )) . (6)5eplacing m ( P ) and y for x and m ( P ) in (6) yields the subjective fitness f sub ( y ). The metric m ( P ) used inthe numerical experiments reported here is to identify and employ the individual with maximal or minimalfitness at a given generation. In some sense this means that all individuals of the other population act asevaluators by rating and presenting its best member.Another significant issue in coevolutionary scenarios is the character of the interaction. A main classifi-cation is cooperative or competitive interaction [1, 12, 19]. Cooperative means that the individual and theevaluators interact and collaborate to solve a problem that is harder or impossible to solve by each of themalone. The better they support each other and perform together, the higher the reward and hence the fitness.In other words, both populations work towards the same aim. In competitive interaction the individual isrewarded for out–performing the evaluators, which sometimes means that the fitness of one individual isincreased at the expense of the others. The main feature here is that the aims of the populations involvedare conflicting.Interestingly, for the simple examples of coevolutionary models considered here, either cooperative orcompetitive interactions can be imposed in an abstract way. For the compositional minimal substratesgiven by (6), the question of cooperative or competitive interaction can be decided by the properties ofthe metrics m ( P ) and m ( P ). These metrics answer which individual of either population is the best, x best ( k ) = m ( P ( k )) and y best ( k ) = m ( P ( k )), respectively. As shown in [13, 14], a cooperative interactionis imposed if the task for both populations is the same, that is, both are to find the maximum or minimumof the objective fitness function (4) and (5). A competitive interaction takes place if one population is tosearch for the maximum of (4) and (5), while the other is to find the minimum of (4) and (5). In a similarmanner, for the test–based minimal substrates given by (1), a cooperative interaction can be observed ifboth populations mean to find the maximum (or minimum) of the objective fitness. A competitive scenariooccurs if one population searches for the maximum while the other intends to find the minimum.So far, the interactive domains and the character of interaction of the minimal substrates were laidout. As evaluation and subsequent updating of fitness in one population requires evaluators from the otherpopulation, the question of timely order and sequence becomes an issue. The models we consider here havea synchronous mode of evolutionary flow. We consider synchronization that takes place after a generationof both populations (shared synchronization). Such synchronization points mean that both populationsevolve along the conventional EA’s generational process (fitness evaluation followed by selection, possiblyrecombination and mutation) and communicate via delivering evaluators to the respective other population.This implies that the fitness of P ( k + 1) and P ( k + 1) is calculated using evaluators from P ( k ) and P ( k ),respectively. As the populations take turns in evolving, this creates a coupling via the (time–dependent)fitness values of the respective population. As an effect, both populations coevolve, and the landscapes showcodynamics.How this codynamics is reflected in the subjective fitness landscapes is analyzed using numerical ex-periments with a CEA. The experimental results reported are obtained for an algorithm with separatedpopulations that undergo selection and mutation independent from each other. The coevolutionary inter-actions are carried out as described above. Unless otherwise stated the population size is λ = λ = 24 andfor the test–based minimal substrate there are µ = µ = 12 evaluators. Selection is tournament with size 2and mutation is Gaussian with mutation probability 0 . .
1. In accordance to otherstudies [22], no recombination is used. 6
Codynamic fitness landscapes
For the test–based problem, the subjective fitness of population P ( k + 1) at generation k + 1 is calculatedaccording to (1) using a sample s ( P ( k )) from the population P ( k ) and yields the landscape: f sub ( x, k + 1) = 1 µ µ (cid:88) i =1 score( x, s i ( P ( k ))) . (7)The samples s ( P ( k )) are statistically independent over the individuals for which fitness is to be assigned.This means for each individual in every generation (and every coevolutionary run), there is a specific(subjective) fitness landscape. Stated like that it seems hopeless to draw any useful information fromanalyzing such landscapes. However, while the samples are statistically independent, the possible membersdrawn and used as evaluators are not as they belong to the coevolving population. This implies that thesubjective fitness landscape may follow certain patterns, and that these patterns reveal the general topologyof the (subjective) landscape, at least as the result of averaging or another analyzing method. The subjectivefitness of P ( k + 1) is calculated likewise by (7), but by using a sample s ( P ( k )). Note that the character ofthe interaction (cooperative or competitive) may influence the average composition of the population andconsequently the (average) selection of evaluators. For instance, if in competitive interaction the population P searches the minimum and P the maximum, then the evaluators drawn from P will on average be largerand generally may have other statistical properties as if both populations were to find the minimum. Thisalso affects the subjective landscape (7), albeit in an implicit way only. Further note that the subjectivelandscape of P and the landscape of P are coupled via the evaluators from the respective other populationwhich implies that codynamics occurs between these landscapes.For the compositional problem we get for the objective fitness function (4) the subjective fitness landscapeof population P f sub ( x, k + 1) = n + 2 min ( x, y best ( k )) − max ( x, y best ( k )) , (8)while for population P we obtain f sub ( y, k + 1) and replace x and y best ( k ) by y and x best ( k ) in (8). Fromthe perspective of the populations alone it appears that fitness is calculated on–the–fly while the CEA isrunning. For the objective fitness function (5), the landscapes read accordingly. Also these landscapesare coupled and codynamic. In difference to the test–based landscapes, we have the same landscape forall individuals of each population, but the landscapes still vary over generations and coevolutionary runs.Furthermore, as the landscapes depend explicitly on x best and y best , cooperative and competitive interactionmay explicitly yield landscapes with different shapes.Due to the simplicity of the examples, the codynamic fitness landscapes can be depicted as a function ofcoevolutionary run–time. Fig. 3 shows realizations of the codynamic landscapes of the test–based problemspecified by the smooth objective fitness function (3) and in Fig. 4 the codynamic landscapes of thecompositional problem specified by the shared objective fitness function (5) can be seen. As an illustrationof codynamics, both figures show the subjective landscapes f sub ( x, k ) of population P as red lines and f sub ( y, k ) of population P as blue lines. The landscapes are given for three points in coevolutionary run–time, k = 0 , ,
6. They are realizations of codynamic fitness landscapes because they are the result of a singlecoevolutionary run. Another run with another initial population may produce slightly different curves. Thenumerical experiments, however, have shown that certain pattern are preserved over the runs.As illustrated in Fig. 3, we obtain an ensemble of landscapes for each point in time for coevolutionarytest–based problems. This ensemble is built by the possibly different landscapes for each sample s ( P ) or s ( P ). Thus, at the utmost there are as many landscapes as individuals in the population, which is λ = 24 forthe example. However, the scoring function (1) that renders subjective fitness from objective fitness impliesa discretization, which means that only a finite number of different landscape shapes are possible. The effectof discretization is clearly visible in Fig. 3. Discretization also contributes to the deviation between thecurves of the subjective fitness landscape and the objective landscape. The subjective landscape frequently7 x,yk f s ub ( x , k ) ,f s ub ( y , k ) (a) x,yk f s ub ( x , k ) ,f s ub ( y , k ) (b)Figure 3: Realizations of codynamic fitness landscapes of the test–based problem specified by the smoothobjective fitness function (3). (a) Competitive interaction. (b) Cooperative interaction.overestimates or underestimates the objective landscape (compare the curves in Fig. 3 with the curve inFig. 1b), which goes along with coevolutionary intransitivity. Another interesting feature of codynamiclandscapes can be seen for coevolutionary run–time going by. For the initial population, which most likelyhas a large diversity, the shape of the subjective landscapes still somehow resembles the shape of the objectivelandscape, compare to Fig. 1b. For time going on the shape of the subjective landscape changes dramatically.In Fig. 3a competitive interaction is shown where population P searches for the minimum and P intends tofind the maximum. It is visible that the landscape f sub ( x, k ) contracts around the solution peak, while thelandscape f sub ( y, k ) does the same around the solution valley. Other topological features of the landscape (forinstance the respective valley and peak) are blanked out. It appears as if the coevolutionary process createsthe fitness landscape in which the algorithm performs the search. Interestingly, this blanking out effect isalso noticeable for cooperative interaction, see Fig. 3b. However, here both subjective landscapes evolvealong similar pattern. Comparing the figures also reveals that the degree of contraction varies from run torun, which emphasize that each subjective landscape in itself is a realization. The subjective landscapesof the compositional problem specified by the shared objective fitness function (5) are given in Fig. 4 andshow a slightly different behavior. Again, the landscapes f sub ( x, k ) of population P are shown as red linesand f sub ( y, k ) of population P as blue lines. We have a single landscape for each generation as there is justone landscape for all individuals in compositional coevolution. All subjective landscapes are slices throughthe shared objective landscape. Thus, the landscapes could also be directly derived from the shared fitnesslandscape in Fig. 2b by looking from the x –axis (or y –axis) and considering the value for y = y best ( k )(or x = x best ( k )) as slices of the S x (or S y ) space. Again, a difference in cooperative and competitivecoevolution can be observed by the landscapes in cooperation (Fig. 4b) converging while the landscapesin competition diverging. Also, it might be that the subjective landscape at a given generation does notinclude the maximum (or minimum) of the objective landscape, thus making it impossible to search for it.The codynamic landscapes for the other problems show similar characteristics, but are not depicted herefor sake of brevity. 8 x,yk f s ub ( x , k ) ,f s ub ( y , k ) (a) x,yk f s ub ( x , k ) ,f s ub ( y , k ) (b)Figure 4: Realizations of codynamic fitness landscapes of the compositional problem specified by the sharedobjective fitness function (5). (a) Competitive interaction. (b) Cooperative interaction. As instructive as these pictorial descriptions of landscapes might be, they also clearly show the limitationsof geometrical conceptualization. The pictures are widely open to interpretation and bound to maximallytwo–dimensional search spaces. Therefore, we next study properties of codynamic landscapes based onanalytic quantities. In doing so we define landscape measures of codynamic landscapes. It has been arguedthat coevolutionary failure, intransitivity and pathological behavior is a consequence of subjective fitnessdissociating from objective fitness [4, 21]. Hence, it appears to be interesting to analyze how measures ofsimilarity between the objective and subjective landscape behave over coevolutionary run–time. As theminimal substrates allow to analytically describe both the subjective and objective landscape, a calculable(geometric) similarity measure is Euclidean distance (dist), which we define asdist( k ) = 1dist max (cid:107) f obj ( x j ) − f sub ( x j , k ) (cid:107) , (9)where x j are a countable number of search space points in S x , and dist max is the maximal fitness differencein the landscape. For the compositional landscape f obj ( x, y ), the component y is set to the global y max or y min , respectively. For test–based problems, the subjective landscape f sub ( x j , k ) is built by averagingover the samples. We further test two statistical difference measures. A first is Kullback–Leibler divergence(kld), e.g., see [3], p. 19: kld( k ) = (cid:88) j ¯ f obj ( x j ) log (cid:18) ¯ f obj ( x j )¯ f sub ( x j , k ) (cid:19) , (10)which is calculated with a countable number of search space points x j and normalized subjective and ob-jective fitness values ¯ f obj and ¯ f sub ( x j , k ). This normalization allows to view subjective and objective fitnessas quantities similar to distributions. Hence, the kld in (10) measures the entropic distance from objectivefitness to subjective fitness. As a third similarity measure of landscapes we consider the Bhattacharyyacoefficient (bhatt) [2] which assesses the similarity of two probability distributions. It is obtained by parti-tioning the objective and subjective fitness landscapes into normalized histograms h obj ( x j ) and h sub ( x j , k )9ith bin centers specified by x j and calculatingbhatt( k ) = (cid:115) − (cid:88) j h obj ( x j ) · h sub ( x j , k ) . (11)Hence, the bhatt in (11) measures the amount of overlap between objective and subjective fitness. Theequations (9), (10) and (11) are for calculating the measures of the landscape over S x , For the measuresover S y , replace y for x .Fig. 5 shows the result for the codynamic landscape measures (9), (10) and (11) for all minimal substratesconsidered. For the test–based problem specified by the smooth objective fitness function (3), refer to Fig.5a-c and for the piece-wise linear function (2), see Fig. 5d-f. The results of the compositional problemspecified by the sinusoid shared objective fitness function (5) are given in Fig. 5g-i and the landscapemeasures for the ridge function (4) are shown in Fig. 5j-l. As the landscapes measures depend on theoutcome of coevolutionary runs, the averages for 100 runs and the 95% confidence intervals are given forcoevolutionary generations k = 0 to k = 10. Again the red curves indicate the results for the subjectivelandscape f sub ( x, k ) of population P , while the blue curves are for f sub ( y, k ) of population P . A firstinteresting feature of the two test–based problems (see Fig. 5a-f) is that the measures for the landscapes ofcooperating populations are almost indistinguishable. This indicates that the cooperation leads to landscapesthat become very similar. This similarity, however, is only between the two subjective landscapes, but notbetween subjective and objective landscape. Here, the distance for cooperative interaction is very oftenlarger than for competitive interaction. This is particularly visible for the similarity measure Euclideandistance, see Fig. 5a,d. The Euclidean distance being larger for cooperative interaction than for competitiveinteraction becomes plausible considering the dynamics of the subjective landscapes for the problem (3).As can be seen in Fig. 3b, the subjective landscapes contract around the solution peak. This contractionis much stronger for cooperative than for competitive interaction. The contraction, on the other hand, alsoimplies a strong deviation from the objective landscape, which in turn means a stronger differences betweenobjective and subjective landscape. This effect is clearly visible in Fig. 5a. For the two compositionalproblems (see Fig. 5g-l), the closeness of the measures for cooperative interaction is also observable, albeitthe similarity between the codynamic landscapes is not as strong as for the test–based cases. Anothergeneral features is that for the two compositional problems, the confidence intervals of the measures aregenerally much larger than for the test–based problems, which implies that the subjective landscapes havea larger variety. Particularly, for the problem modeled by the ridge function (4) (Fig. 5j-l) we can seethat the landscape over S x varies much weaker than the codynamic cooperating landscape over S y . Twofurther observations are that the measures do stop to change with run–time after a certain number ofcoevolutionary generations (indicating that a kind of steady state has been reached), and that the statisticalsimilarity measures (kld and bhatt) largely reflect the geometric measure Euclidean distance (and hencemight be usable as an substitute if the geometric measure cannot be calculated), but may also add furtherclues for discriminating the codynamics between objective and subjective fitness. In this paper an approach has been presented for analyzing coevolution using the theoretical framework offitness landscapes. It has been shown that the approach can be applied for test–based as well as compositionalproblems. For these two classes of coevolutionary problems, simple and abstract models, called minimalsubstrates, were studied for both cooperative as well as competitive interaction. An important designquestion in coevolution is whether and under what circumstances subjective fitness implies objective fitness.The dynamic fitness landscape approach aims specifically at addressing this question. Therefore, objectiveand subjective landscapes for the minimal substrates were defined and analyzed. The results have shown thatbetween these landscapes there emerges codynamics where the evolutionary development of one populationhas effect upon the other population and therefore deforms its subjective fitness landscape. As this process10orks in both ways the landscapes are coupled and codynamic. We further defined three different landscapemeasures that are designed to account for differences between objective and subjective fitness. The numericalresults suggest that these similarity measures are suitable for quantifying and discriminating the codynamicsbetween objective and subjective fitness.The results have also shown that the coevolutionary process in one population generates the landscapeof the other population and vice versa. In this sense the coevolutionary process creates the landscape inwhich the process of optimization takes place. As a consequence a strict separation between problem andproblem solving algorithm ceases to exist. The fitness landscape of the problem (objective fitness landscape)still sets the background and framework for (co–)evolutionary dynamics, but how the coevolutionary algo-rithm perceives the problem is governed by the subjective landscape and how strongly the latter deviatescodynamically from the former. Therefore, an important question in analyzing coevolutionary algorithmsis what features in the codynamic fitness landscapes the algorithm produces. The paper has shown howcodynamic landscape measures can be helpful for addressing this question. Another value of landscapeanalysis is that it permits posing questions of how the properties of the landscape reflect, explain and allowpredicting expectable behavior (and possibly performance) of evolutionary search algorithms. More specifi-cally, the topology of the landscape can be seen as a predictor of algorithmic behavior. It may be reasonableto assume that these relationships also extend to coevolution. Thus, for a landscape analysis capable ofassessing coevolutionary performance, the similarity measure introduced in this paper should be amendedwith topological landscape measures, see e.g. [9] for a recent review. In this context, it might be temptingto assume that the expectable algorithmic performance is best if the average subjective landscape reflectsmaximally the objective landscape. If this is indeed the case could be studied experimentally using theproposed minimal substrates and recorded performance data, for instance average objective fitness, averagesubjective fitness, and how both quantities correlate [4]. These studies could also go along with consideringthe influence of coevolutionary design parameters, e.g. population size, sample size, mutation strength andrate, and so on.
References [1] W. Chen and K. Tang, Impact of problem decomposition on cooperative coevolution. In: C. A. CoelloCoello (ed.),
Proc. Congress on Evolutionary Computation, IEEE CEC 2013 , IEEE Press, Piscataway,NJ, 733–740.[2] D. Comaniciu, V. Ramesh, and P. Meer, Kernel–based object tracking.
IEEE Trans. Pattern Analysisand Machine Intelligence : 564–577, 2003.[3] T. M. Cover and J. A. Thomas. Elements of Information Theory , Wiley, Hoboken, NJ, 2006.[4] E. D. de Jong, Objective fitness correlation. In: H. Lipson (ed.),
Proc. Genetic and Evolutionary Com-putation Conference, GECCO 2007 , ACM, New York, 440–447, 2007.[5] W. D. Hillis, Co-evolving parasites improve simulated evolution as an optimization procedure.
PhysicaD : 228–234, 1990.[6] B. Inden, Open–ended coevolution and the emergence of complex irreducible functional units in iteratednumber sequence games. In: T. Soule, J. H. Moore (eds.), Proc. Genetic and Evolutionary ComputationConference, GECCO 2012 , ACM, New York, 113–120, 2012.[7] R. Kala, Multi-robot path planning using co–evolutionary genetic programming.
Expert Systems withApplications : 3817–3831, 2012.[8] L. Kallel, B. Naudts, and C. R. Reeves, Properties of fitness functions and search landscapes. In: L. Kallel,B. Naudts, A. Rogers (eds.), Theoretical Aspects of Evolutionary Computing , Springer–Verlag, Berlin,Heidelberg, New York, 177–208, 2001. 119] K. M. Malan and A. P. Engelbrecht, A survey of techniques for characterising fitness landscapes andsome possible ways forward.
Information Science : 148–163, 2013.[10] T. Miconi, Why coevolution doesn’t “work”: Superiority and progress in coevolution. In: L. Vanneschi,et al. (eds.),
EuroGP 2009 , Springer–Verlag, Berlin Heidelberg New York, 49–60, 2009.[11] L. Panait, S. Luke, Cooperative multi–agent learning: the state of the art.
Autonomous Agents andMulti-Agent Systems : 387–434, 2005.[12] E. Popovici, A. Bucci, R. P. Wiegand, and E. D. de Jong, Coevolutionary principles. In: G. Rozenberg,T. B¨ack, J. N. Kok (eds.), Handbook of Natural Computing , Springer–Verlag, Berlin Heidelberg NewYork, 987–1033, 2010.[13] E. Popovici and K. A. de Jong, Understanding competitive co–evolutionary dynamics via fitness land-scapes. In: S. Luke (ed.), , AAAI Press,Palo Alto, CA, 2005.[14] E. Popovici and K. A. de Jong, Understanding cooperative co–evolutionary dynamics via simple fit-ness landscapes. In: H. G. Beyer, U. M. O’Reilly (eds.),
Proc. Genetic and Evolutionary ComputationConference, GECCO 2005 , Morgan Kaufmann, San Francisco, CA, 507–514, 2005.[15] H. Richter, Coupled map lattices as spatio–temporal fitness functions: Landscape measures and evolu-tionary optimization.
Physica D : 167–186, 2008.[16] H. Richter, Fitness landscapes that depend on time. In: H. Richter, A. P. Engelbrecht (eds.),
RecentAdvances in the Theory and Application of Fitness Landscapes , Springer–Verlag, Berlin Heidelberg NewYork, 265–299, 2014.[17] C. D. Rosin and R. K. Belew, New methods for competitive coevolution.
Evolutionary Computation :1–29, 1997.[18] S. Samothrakis, S. M. Lucas, T. P. Runarsson, and D. Robles, Coevolving game–playing agents:Measuring performance and intransitivities. IEEE Trans. Evolut. Comp. : 213–226, 2013.[19] M. Shi, Natural vs. unnatural decomposition in cooperative coevolution. In: D. S. Huang, et al.(eds.), Advanced Intelligent Computing Theories and Applications. With Aspects of Artificial Intelli-gence , Springer–Verlag, Berlin Heidelberg New York, 138–147, 2012.[20] T. Smith, P. Husbands, P. Layzell, and M. O’Shea, Fitness landscapes and evolvability.
Evolut. Comput. : 1–34, 2002.[21] R. P. T. van Wijngaarden and E. D. de Jong, Evaluation and diversity in co–evolution. In: G. Rudolph,et al. (eds.), Parallel Problem Solving from Nature–PPSN X , Springer–Verlag, Berlin Heidelberg NewYork, 631–640, 2008.[22] R. A. Watson and J. B. Pollack, Coevolutionary dynamics in a minimal substrate. In: L. Spector, etal. (eds.),
Proc. Genetic and Evolutionary Computation Conference, GECCO 2001 , Morgan Kaufmann,San Francisco, CA, 702–709, 2001. 12 xy compcoopxy k d i s t xy compcoopxy k k l d xy compcoopxy k bha tt (a) (b) (c) k d i s t k k l d k bha tt (d) (e) (f) k d i s t k k l d k bha tt (g) (h) (i) k d i s t k k l d k bha tt j) k) l)Figure 5: Landscape measures for the minimal substrates: the test–based problems specified by the objec-tive fitness (3), (a-c), and (2), (d-f). The compositional problem specified by the shared objective fitnessfunction (5), (g-h), and (4), (j-l). The red curves indicate measures over S x , blue curves over S yy