Anytime Diagnosis for Reconfiguration
Alexander Felfernig, Rouven Walter, Jose A. Galindo, David Benavides, Seda Polat-Erdeniz, Muesluem Atas, Stefan Reiterer
AAnytime Diagnosis for Reconfiguration
Alexander Felfernig · Rouven Walter · Jos´e A. Galindo · DavidBenavides · Seda Polat Erdeniz · M ¨usl ¨um Atas · Stefan Reiterer
Preprint, cite as: A. Felfernig. R. Walter, J. Galindo, D. Benavides, M. Atas, S. Polat-Erdeniz, and S. Reiterer. Anytime Diagnosis for Recon-figuration. Journal of Intelligent Information Systems, vol. 51, pp. 161-182, 2018.
Abstract
Many domains require scalable algorithms that help to determine diagnoses efficiently and often within pre-defined time limits.
Anytime diagnosis is able to determine solutions in such a way and thus is especially useful inreal-time scenarios such as production scheduling, robot control, and communication networks management where di-agnosis and corresponding reconfiguration capabilities play a major role. Anytime diagnosis in many cases comes alongwith a trade-off between diagnosis quality and the efficiency of diagnostic reasoning. In this paper we introduce and an- alyze F
LEX D IAG which is an anytime direct diagnosis approach. We evaluate the algorithm with regard to performanceand diagnosis quality using a configuration benchmark from the domain of feature models and an industrial configu-ration knowledge base from the automotive domain. Results show that F
LEX D IAG helps to significantly increase theperformance of direct diagnosis search with corresponding quality tradeoffs in terms of minimality and accuracy.
Keywords:
Anytime Diagnosis, Reconfiguration, Direct Diagnosis, Constraint Solving, Configuration, SoftwareProduct Lines.
Knowledge-based configuration is one of the most successful application areas of Artificial Intelligence [13,18,23,29,37,38,45]. There exist many application domains ranging from telecommunication systems [17,46], railway interlockingsystems [11], the automotive domain [43,50,53], software product lines [4] to the configuration of services [51].Configuration technologies must be able to deal with inconsistencies which can occur in different contexts.
First , a configuration knowledge base can be inconsistent, i.e., no solution can be determined. In this context, the task of knowl-edge engineers is to figure out which constraints are responsible for the unintended behavior of the knowledge base.
Alexander FelfernigApplied Software Engineering Group, Institute for Software Technology, TU Graz, AustriaE-mail: [email protected] WalterSymbolic Computation Group, WSI Informatics, Universit¨at T¨ubingen, GermanyE-mail: [email protected]´e A. GalindoComputer Languages and Systems Department, University of Sevilla, Spain E-mail: [email protected] BenavidesComputer Languages and Systems Department, University of Sevilla, SpainE-mail: [email protected] Polat-ErdenizApplied Software Engineering Group, Institute for Software Technology, TU Graz, AustriaE-mail: [email protected]¨usl¨um AtasApplied Software Engineering Group, Institute for Software Technology, TU Graz, AustriaE-mail: [email protected] ReitererSelectionArts, AustriaE-mail: [email protected] a r X i v : . [ c s . A I] F e b akker et al. [3] show how to apply model-based diagnosis [36] to determine minimal sets of constraints in a knowl-edge base that are responsible for a given inconsistency. A variant thereof is documented in Felfernig et al. [12] wherean approach to the automated debugging of knowledge bases with test cases is introduced. Test cases are interpretedas positive or negative examples that describe the intended behavior of a knowledge base. If some positive examplesinduce conflicts in the configuration knowledge base, some of the constraints in the knowledge base are faulty and haveto be adapted or deleted. If some negative examples are accepted (i.e., not rejected) by the configuration knowledgebase, further constraints have to be included in order to take these examples into account (in Felfernig et al. [12] thisissue is solved by simply including negative examples in negated form into the configuration knowledge base). A relatedapproach in the area of software product lines is proposed in [56]. Second , customer requirements can be inconsistentwith the underlying knowledge base. Felfernig et al. [12] also show how to diagnose customer requirements that areinconsistent with a configuration knowledge base. The underlying assumption is that the configuration knowledge baseitself is consistent but combined with a set of requirements is inconsistent.The so far mentioned configuration-related diagnosis approaches are based on conflict-directed hitting set determi-nation where conflicts have to be calculated in order to be able to derive one or more corresponding diagnoses [8,27,28,36,41]. These approaches often determine diagnoses in a breadth-first search manner which allows the identifica-tion of minimal cardinality diagnoses . The major disadvantage of applying these approaches is the need of determiningminimal conflicts which is inefficient especially in cases where only the leading diagnoses (the most relevant ones) aresought. Furthermore, in many application domains it is not necessarily the case that minimal cardinality diagnoses arethe preferred ones – Felfernig et al. [14] show how recommendation technologies [26] can be exploited for guiding thesearch for preferred (minimal but not necessarily minimal cardinality) diagnoses.Algorithms based on the idea of anytime diagnosis are useful in scenarios where diagnoses have to be provided inreal-time, i.e., within given time limits. Efficient diagnosis and reconfiguration of communication networks is crucial to retain the quality of service, i.e., if some components/nodes in a network fail, corresponding substitutes and extensionshave to be determined immediately [34,47]. In today’s production scenarios which are characterized by small batch sizesand high product variability, it is increasingly important to develop algorithms that support the efficient reconfigurationof schedules . Such functionalities support the paradigm of smart production , i.e., the flexible and efficient production ofhighly variant products. Further applications are the diagnosis and repair of robot control software [44], sensor networks[35], feature models [27,56], the reconfiguration of cars [54], and the reconfiguration of buildings [19]. In the diagnosisapproach presented in this paper, we assure diagnosis determination within certain time limits by systematically reduc-ing the number of solver calls needed. This specific interpretation of anytime diagnosis requires a trade-off betweendiagnosis quality (evaluated, e.g., in terms of minimality) and the time needed for diagnosis determination.Algorithmic approaches to provide efficient solutions for diagnosis problems are manyfold. Some approaches focuson improvements of Reiter’s original hitting set directed acyclic graph (HSDAG) [36] in terms of a personalized compu-tation of leading diagnoses [9] or other extensions that make the basic approach [36] more efficient [57]. Wang et al. [55]introduce an approach to derive binary decision diagrams (BDDs) [1,6] on the basis of a pre-determined set of conflicts– diagnoses can then be determined by finding paths in the BDD that include given variable settings (e.g., requirementsdefined by the user). A predefined set of conflicts can also be compiled into a corresponding linear optimization problem[16]; diagnoses can then be determined by solving the given problem. In knowledge-based recommendation scenarios,diagnoses for user requirements can be pre-compiled in such a way that for a given set of customer requirements, thediagnosis search task can be reduced to querying a relational table (see, for example, [25,39]). All of the mentioned ap-proaches either extend the approach of Reiter [36] or improve efficiency by exploiting pre-generated information aboutconflicts or diagnoses.An alternative to conflict-directed diagnosis [36] are direct diagnosis algorithms that determine minimal diagnoseswithout the need of predetermining minimal conflict sets [15,42]. The F
AST D IAG algorithm [15] is a divide-and-conquerbased algorithm that supports the determination of diagnoses without a preceding conflict detection. Such direct diag-nosis approaches are especially useful in situations where not the complete set of diagnoses has to be determined butusers are interested in the leading diagnoses , i.e., diagnoses with a high probability of being relevant for the user. Alsoin the context of SAT solving, algorithms have been developed that allow the efficient determination of diagnoses (alsodenoted as minimal correction subsets) in an efficient fashion [2,22,32,33]. Beside efficiency, prediction quality of adiagnosis algorithm is a major issue in interactive configuration settings, i.e., those diagnoses have to be identified thatare relevant for the user. A corresponding comparison of approaches to determine preferred minimal diagnoses andunsatisfied clauses with minimum total weights is provided in [52]. The authors point out theoretical commonalities andprove the reducibility of both concepts to each other. Requirements are additional constraints not part of the configuration knowledge base itself – these constraints represent specific prefer-ences regarding product properties.
2n this paper we show how the F
AST D IAG approach can be converted into an anytime diagnosis algorithm(F
LEX D IAG ) that allows tradeoffs between diagnosis quality (minimality and accuracy) and performance. In this paperwe focus on reconfiguration scenarios [19,34,47,53], i.e., we show how F
LEX D IAG can be applied in situations wherea given configuration (solution) has to be adapted conform to a changed set of customer requirements. Our contributionsin this paper are the following. First, based on previous work on the diagnosis of inconsistent knowledge bases, we showhow to solve reconfiguration tasks with direct diagnosis. Second, we make direct diagnosis anytime-aware by includinga parametrization that helps to systematically limit the number of consistency checks and thus make diagnosis searchmore efficient. Finally, we report the results of a F
LEX D IAG -related evaluation conducted on the basis of real-worldconfiguration knowledge bases (feature models and configuration knowledge bases from the automotive industry) anddiscuss quality properties of related diagnoses not only in terms of minimality but also in terms of accuracy.The remainder of this paper is organized as follows. In Section 2 we introduce an example configuration knowledgebase from the domain of resource allocation. This knowledge base will serve as a working example throughout the paper.Thereafter (Section 3) we introduce a definition of a reconfiguration task. In Section 4 we discuss basic principles ofdirect diagnosis on the basis of F
LEX D IAG and show how this algorithm can be applied in reconfiguration scenarios.In Section 5 we present the results of an analysis of algorithm performance and the quality of determined diagnoses.A simple example of the application of F
LEX D IAG in production environments is given in Section 6. In Section 7 wediscuss issues for future work. With Section 8 we conclude the paper.
A configuration system determines configurations (solutions) on the basis of a given set of customer requirements [24].In many cases, constraint satisfaction problem (CSP) representations are used for the definition of a configuration task. A configuration task and a corresponding configuration (solution) can be defined as follows:
Definition 1 (Configuration Task and Configuration) . A configuration task can be defined as a CSP (
V, D, C ∪ R )where V = { v , v , ..., v n } is a set of variables, D = (cid:83) v i ∈ V { dom ( v i ) } represents domain definitions, and C = { c , c , ..., c m } is a set of constraints (the configuration knowledge base). Additionally, user requirements arerepresented by a set of constraints R = { r , r , ..., r k } where R and C are disjoint. A configuration (solution) for aconfiguration task is a complete set of assignments (constraints) S = { s : v = a , s : v = a , ..., s n : v n = a n } where a i ∈ dom ( v i ) which is consistent with C ∪ R .An example of a configuration task represented as a constraint satisfaction problem (CSP) is the following. Example (Configuration Task) . In this resource allocation problem example, items (a barrel of fuel , a stack of paper ,a pallet of fireworks , a pallet of personal computers , a pallet of computer games , a barrel of oil , a pallet of roof tiles ,and a pallet of rain pipes ) have to be assigned to three different containers. There are a couple of constraints ( c i ) to betaken into account, for example, fireworks must not be combined with fuel ( c ). Furthermore, there is one requirement( r ) which indicates that the pallet of fireworks has to be assigned to container 1. On the basis of this configuration taskdefinition, a configurator can determine a configuration (solution) S . – V = { f uel, paper, f ireworks, pc, games, oil, roof, pipes } – dom ( f uel ) = dom ( paper ) = dom ( f ireworks ) = dom ( pc ) = dom ( games ) = dom ( oil ) = dom ( roof ) = dom ( pipes ) = { , , } – C = { c : f ireworks (cid:54) = f uel, c : f ireworks (cid:54) = paper,c : f ireworks (cid:54) = oil, c : pipes = roof, c : paper (cid:54) = f uel } – R = { r : f ireworks = 1 } – S = { s : pc = 3 , s : games = 1 , s : paper = 2 , s : f uel = 3 ,s : f ireworks = 1 , s : oil = 2 , s : roof = 1 , s : pipes = 1 } On the basis of the given definition of a configuration task, we now introduce the concept of reconfiguration (seealso [19,34,47,53]).
It can be the case that an existing configuration S has to be adapted due to new customer requirements. Examplesthereof are changing requirements in production schedules, failing components or overloaded network infrastructures ina mobile phone network, and changes in the internal model of the environment of a robot. In the following we assume A Constraint Satisfaction Problem (CSP) is typically defined by a set of variables, corresponding finite domains, and a set of constraints[31]. For the representation of constraints we use a notation typically used in the context of CSP solving – for details see, for example, [13]. pallet of paper should be reassigned to container 3 and the personal computer and games pallets should beassigned to the same container. Formally, the set of new requirements is represented by R ρ : { r (cid:48) : pc = games, r (cid:48) : paper = 3 } . In order to determine reconfigurations, we have to calculate a corresponding diagnosis ∆ (see Definition2). Definition 2 (Diagnosis) . A diagnosis ∆ (correction subset) is a subset of S = { s : v = a , s : v = a , ..., s n : v n = a n } such that S − ∆ ∪ C ∪ R ρ is consistent. ∆ is minimal if there does not exist a diagnosis ∆ (cid:48) with ∆ (cid:48) ⊂ ∆ .On the basis of the definition of a minimal diagnosis, we can introduce a formal definition of a reconfiguration task. Definition 3 (Reconfiguration Task and Reconfiguration) . A reconfiguration task can be defined as a CSP(
V, D, C, S, R ρ ) where V is a set of variables, D represents variable domain definitions, C is a set of constraints, S rep-resents an existing configuration, and R ρ = { r (cid:48) , r (cid:48) , ..., r (cid:48) q } ( R ρ consistent with C ) represents a set of reconfigurationrequirements. Furthermore, let ∆ be a minimal diagnosis for the reconfiguration task. A reconfiguration is a variableassignment S ∆ = { s : v = a (cid:48) , s : v = a (cid:48) , ..., s l : v l = a (cid:48) l } where s i ∈ ∆ , a (cid:48) i (cid:54) = a i , and S − ∆ ∪ S ∆ ∪ C ∪ R ρ isconsistent.If R ρ is inconsistent with C , the new requirements have to be analyzed and changed before a corresponding re-configuration task can be triggered [10,14]. An example of a reconfiguration task in the context of our configurationknowledge base is the following. Example (Reconfiguration Task) . In the resource allocation problem, the original customer requirements R are sub-stituted by the requirements R ρ = { r (cid:48) : pc = games, r (cid:48) : paper = 3 } . The resulting reconfiguration task instance isthe following. – V = { f uel, paper, f ireworks, pc, games, oil, roof, pipes } – dom ( f uel ) = dom ( paper ) = dom ( f ireworks ) = dom ( pc ) = dom ( games ) = dom ( oil ) = dom ( roof ) = dom ( pipes ) = { , , } – C = { c : f ireworks (cid:54) = f uel, c : f ireworks (cid:54) = paper,c : f ireworks (cid:54) = oil, c : pipes = roof, c : paper (cid:54) = f uel } – S = { s : pc = 3 , s : games = 1 , s : paper = 2 , s : f uel = 3 ,s : f ireworks = 1 , s : oil = 2 , s : roof = 1 , s : pipes = 1 } – R ρ = { r (cid:48) : pc = games, r (cid:48) : paper = 3 } To solve a reconfiguration task (see Definition 3), conflict-directed diagnosis approaches [36] would determine aset of minimal conflicts and then determine a hitting set that resolves each of the identified conflicts. In this context, aminimal conflict set CS ⊆ S is a minimal set of variable assignments that trigger an inconsistency with C ∪ R ρ , i.e., CS ∪ C ∪ R ρ is inconsistent and there does not exist a conflict set CS (cid:48) with CS (cid:48) ⊂ CS . In our working example, theminimal conflict sets are CS : { s : pc = 3 , s : games = 1 } , CS : { s : paper = 2 } , and CS : { s : f uel = 3 } .The corresponding minimal diagnoses are ∆ : { s , s , s } and ∆ : { s , s , s } .The elements in a diagnosis indicate which variable assignments have to be adapted such that a reconfigurationcan be determined that takes into account the new requirements in R ρ . Consequently, a reconfiguration represents aminimal set of changes to the original configuration ( S ) such that the new requirements R ρ are taken into account. Ifwe choose ∆ , a reconfiguration S ∆ (reassignments for the variable assignments in ∆ ) can be determined by a CSPsolver call C ∪ R ρ ∪ ( S − ∆ ) . The resulting configuration S (cid:48) can be { s : pc = 1 , s : games = 1 , s : paper =3 , s : f uel = 2 , s : f ireworks = 1 , s : oil = 2 , s : roof = 1 , s : pipes = 1 } . For a detailed discussionof conflict-based diagnosis we refer to Reiter [36]. In the following we introduce an approach to the determination ofminimal reconfigurations which is based on a direct diagnosis algorithm, i.e., diagnoses are determined without the needof determining related minimal conflict sets. LEX D IAG
In the following discussions, the set AC = C ∪ R ρ ∪ S represents the union of all constraints that restrict the set ofpossible solutions for a given reconfiguration task. Furthermore, S represents a set of constraints that are considered ascandidates for being included in a diagnosis ∆ . The idea of F LEX D IAG (Algorithm 1) is to systematically filter out theconstraints that become part of a minimal diagnosis using a divide-and-conquer based approach.
Sketch of Algorithm . In our example reconfiguration task, the original configuration S = { s , s , s ,s , s , s , s , s } and the new set of customer requirements is R ρ = { r (cid:48) , r (cid:48) } . Since S ∪ R ρ ∪ C is inconsistent,we are in need of a minimal diagnosis ∆ and a reconfiguration S ∆ such that S − ∆ ∪ S ∆ ∪ R ρ ∪ C is consistent. In the following we will show how the F LEX D IAG (Algorithm 1) can be applied to determine such a minimal diagnosis ∆ .F LEX D IAG is assumed to be activated under the assumption that AC = C ∪ R ρ ∪ S is inconsistent, i.e., theconsistency of AC is not checked by the algorithm. If AC is inconsistent but AC − S is also inconsistent, F LEX D IAG lgorithm 1 procedure F LEX D IAG ( S, AC = C ∪ R ρ ∪ S, m ): ∆ if isEmpty( S ) or inconsistent( AC − S ) then
3: return ∅ ;4: else
5: return F
LEX D( ∅ , S , AC , m );6: end if end procedure procedure F LEX D( D, S = { s ..s q } , AC, m ): ∆ if D (cid:54) = ∅ and consistent ( AC ) then
10: return ∅ ;11: else if size( S ) ≤ m then
12: return S ;13: end if k = (cid:100) q (cid:101) ;15: S = { s ..s k } ; S = { s k +1 ..s q } ;16: D = F LEX D ( S , S , AC − S , m ) ;17: D = F LEX D ( D , S , AC − D , m ) ;18: return ( D ∪ D ) ;19: end procedure will not be able to identify a diagnosis in S ; therefore ∅ is returned. Otherwise, a recursive function F LEX
D is activatedwhich is in charge of determining one minimal diagnosis ∆ ⊆ S . In each recursive step, the constraints in S aredivided into two different subsets ( S and S ) in order to figure out if already one of these subsets includes a diagnosis. If this is the case, the second set must not be inspected for diagnosis elements anymore. If we assume, for example, S = { s , s , s , s , s , s , s , s } is inconsistent and we divide S into the two subsets S = { s , s , s , s } and S = { s , s , s , s } and S is already consistent with C ∪ R ρ then diagnosis elements are searched in S (since S isalready consistent). The complete related walkthrough is depicted in Figures 1 and 2.F LEX D IAG is based on the concepts of F
AST D IAG [15], i.e., it returns one diagnosis ( ∆ ) at a time and is completein the sense that if a diagnosis is contained in S , then the algorithm will find it. A corresponding reconfiguration can bedetermined by a CSP solver call C ∪ R ρ ∪ ( S − ∆ ) . The determination of multiple diagnoses at a time can be realizedon the basis of the construction of a HSDAG [36]. In F LEX D IAG , the parameter m is used to control diagnosis quality in terms of minimality , accuracy , and the performance of diagnostic search (see Section 5). The higher the value of m the higher the performance of F LEX D IAG and the lower the degree of diagnosis quality. The inclusion of m to controlquality and performance is the major difference between F LEX D IAG and F
AST D IAG . If m = 1 (see Algorithm 1), thenumber of consistency checks needed for determining one minimal diagnosis is δ × log ( nδ ) + 2 δ (in the worst case)[15]. In this context, δ represents the set size of the minimal diagnosis ∆ and n represents the number of constraints insolution S .If m > , the number of needed consistency checks can be systematically reduced if we accept the tradeoff ofpossibly loosing the property of diagnosis minimality (see Definition 2). If we allow settings with m > , we canreduce the upper bound of the number of consistency checks to δ × log ( nδ × m ) (in the worst case). These upperbounds regarding the number of needed consistency checks allow to estimate the worst case runtime performance ofthe diagnosis algorithm which is extremely important for real-time scenarios . Consequently, if we are able to estimatethe upper limit of the time needed for completing one consistency check (e.g., on the basis of simulations with anunderlying constraint solver), we are also able to figure out lower bounds for m that must be chosen in order to guaranteea F LEX D IAG runtime within predefined time limits.Table 1 depicts an overview of consistency checks needed depending on the setting of the parameter m and thediagnosis size δ for | S | = 16 . For example, if m = 2 and the size of a minimal diagnosis is δ = 4 , then the upperbound for the number of needed consistency checks is 16. If the size of δ further increases, the number of correspondingconsistency checks does not increase anymore. Figures 1 and 2 depict F LEX D IAG search trees depending on the settingof granularity parameter m . The upper bound for the number of consistency checks helps us to determine the maximumamount of time that will be needed to determine a diagnosis on the basis of F LEX D IAG . For example, if the maximumtime needed for one consistency check is ms , the maximum time needed for determining a diagnosis with m = 2 (given δ = 8 ) is ≈ milliseconds.F LEX D IAG determines one diagnosis at a time which indicates variable assignments of the original configurationthat have to be changed such that a reconfiguration conform to the new requirements ( R ρ ) is possible. The algorithm supports the determination of leading diagnoses , i.e., diagnoses that are preferred with regard to given user preferences[15,52]. F LEX D IAG is based on a strict lexicographical ordering of the constraints in S : the lower the importance ofa constraint s i ∈ S the lower the index of the constraint in S . For example, s : pc = 3 has the lowest ranking. The5 m=1 m=2 m=4 m=81 10 8 6 42 16 12 8 44 24 16 8 -8 32 16 - -16 32 - - - Table 1
Worst-case estimates for the number of needed consistency checks depending on the granularity parameter m and the diagnosis size δ for | S | = 16 . Fig. 1 F LEX D IAG walkthrough: determining one minimal diagnosis with m = 1 ( ∆ = { s , s , s } ). Fig. 2 F LEX D IAG walkthrough: determining a minimal diagnosis with m = 2 ( ∆ = { s , s , s , s } ). lower the ranking, the higher the probability that the constraint will be part of a reconfiguration S ∆ . Since s has thelowest priority and it is part of a conflict, it is element of the diagnosis returned by F LEX D IAG . For a discussion of theproperties of lexicographical orderings we refer to [15,28].
In this section, we present the evaluation we executed to verify the performance of F
LEX D IAG . We first analyze howF
LEX D IAG performs in front of real and randomly generated models and then, compare it with an evolutionary approach.5.1 Evaluation aspectsTo evaluate F
LEX D IAG , we analyzed the two aspects of (1) algorithm performance (in terms of milliseconds needed todetermine one minimal diagnosis) and (2) diagnosis quality (in terms of minimality and accuracy – see Formulae 1 and m . Our hypothesis in this context was that the higher thevalue of m , the lower the number of needed consistency checks (the higher the efficiency of diagnosis search) and the6ower diagnosis quality in terms of the share of diagnosis-relevant constraints returned by F LEX D IAG . Diagnosis qualitycan, for example, be measured in terms of the degree of minimality of the constraints in a diagnosis ∆ (see Formula 1),i.e., the cardinality of ∆ compared to the cardinality of ∆ min . | ∆ min | represents the cardinality of a minimal diagnosisidentified with m = 1 . minimality ( ∆ ) = | ∆ min || ∆ | (1)If m > , there is no guarantee that the diagnosis ∆ determined for S is a superset of the diagnosis ∆ min determinedfor S in the case m = 1 . Besides minimality, we introduce accuracy as an additional quality indicator (see Formula2). The higher the share of elements of ∆ min in ∆ , the higher the corresponding accuracy (the algorithm is able toreproduce the elements of the minimal diagnosis for m = 1 ). accuracy ( ∆ ) = | ∆ ∩ ∆ min || ∆ min | (2)5.2 Datasets and ResultsWe evaluated F LEX D IAG with regard to both metrics (algorithm performance, and diagnosis quality) by applying thealgorithm to different benchmarks. First, using random feature models generated with the Betty tool [40]. Second, withthe set of models hosted in the S.P.L.O.T repository . Third, in a real-world model extracted from the last UbuntuLinux distribution [20]. Finally, a real-world automotive dataset. The configuration models are feature models which include requirement constraints, compatibility constraints, and different types of structural constraints such as mandatoryrelationships and alternatives.For all the different datasets we report on averaged values. For that, we first, calculate the acurracy, execution time,and minimality for all the executions. Then, we aggregate the data and calculate the mean for the metrics. The experiments were conducted using a version of F
LEX D IAG implemented in Java and integrated in the FaMa ToolSuite [5]. All the models were translated to a Constraint Satisfaction Problem (CSP) and used the Choco library forconsistency checking. Further, our F
LEX D IAG implementation was running in a grid of computers running on four-CPU Dell Blades with Intel Xeon X5560 CPUs running at 2.8GHz, with 8 threads per CPU, and CentOS v6. The totalRAM memory was 8GB. To parallelize the executions we used GNU Parallel [48].
The first dataset used to evaluate F
LEX D IAG was randomly generated. We used BeTTy [40] to generate a dataset thatranges from 50 to 2000 features and 10% to 30% of cross-tree constraints. The generation approach is based on Th¨um et al. [49] that imitates realistic topologies.For each model combining a given number of features and a percentage of cross-tree constraints, we randomlygenerated different sizes of reconfiguration requirements that involved the 10%, 30%, 50% and 100% of features ofthe model. Then we randomly reordered each of the reconfiguration requirements 10 times (to prevent ordering biases).Moreover, we executed F
LEX D IAG on each combination of parameters three times to get average execution times tryingto avoid third party threads.In the following, we present the results showing a comparison between the different values of m and how the valuesevolved depending on the size of the models. Note that to generate the plots we aggregated the data and therefore thevalues shown are averaged results.Figure 3 shows how the diagnosis performance can be increased depending on the setting of the m parameter. Alsowe observe how the minimality deteriorates when increasing m .Table 2 shows the averaged data we obtained. It is worth mentioning that the minimality decreases when m increasesand that accuracy still provides acceptable results with m = 10. Also, the execution time (in milliseconds) is less thanfive minutes in the worst case.As we can observe in Table 2 and Figure 3, while the execution time decreases when incrementing m , qualitydeteriorates. However, minimality is clearly affected while accuracy stays with minor variations. Also, we observe that l l l l Value of m T i m e i n m illi s e c ond s ( l og sc a l e ) Number of features l
50 100 500 1000 2000 l l l l l −0.2 −0.15 −0.1 −0.05 Value of m M i n i m a li t y ( l og sc a l e ) Number of features l
50 100 500 1000 2000
Fig. 3
Random evolution based on features, time and m the time improvement depends on m and the number of features. For example, if we compare the time between m = 1 and m = 10 , we can increase in runtime of 2.26 × for 50 features, 5.21 × for 500 features, 9.14 × for 1000 and 16.5 × for 2000 features. We extracted a total of 387 models from the SPLOT repository. For each model, we randomly generated different sizes of reconfiguration requirements that involved the 10%, 30%, 50% and 100% of features of the model. Thenwe randomly reordered each of the reconfiguration requirements 10 times (to prevent ordering biases). Moreover, we8 able 2
Random evaluation depending on m value and model size. | V | represents the number of variables in the CSP. The second columnshows the m value used in F LEX D IAG . | C | refers to the number of constraints in the model, | ∆ | is the average size of a diagnosis, and averagetime (in milliseconds), average minimality , and average accuracy represent the means of the calculated values for those metrics. | V | m | C | | ∆ | Average time Average minimality Average accuracy50 1 58.40 14.62 501.63 1.00 1.002 15.72 410.92 0.85 0.924 17.22 296.14 0.75 0.876 18.71 240.52 0.71 0.8710 19.07 222.15 0.70 0.86100 1 109.80 27.51 1081.24 1.00 0.992 32.80 896.99 0.79 0.904 38.19 651.20 0.68 0.886 40.99 477.64 0.63 0.8710 42.29 425.16 0.61 0.88500 1 566.40 182.62 11808.41 1.00 1.002 203.77 6647.27 0.88 0.964 219.41 4372.98 0.82 0.956 221.82 3643.81 0.80 0.9610 231.49 2265.53 0.78 0.971000 1 1141.00 347.25 47956.27 1.00 1.002 398.62 20227.16 0.84 0.974 434.76 10051.52 0.77 0.966 440.64 8825.81 0.75 0.9610 461.53 5242.34 0.74 0.982000 1 2274.80 663.36 264084.35 1.00 1.002 743.21 95467.42 0.84 0.974 818.12 37911.86 0.76 0.976 828.39 35271.86 0.74 0.9610 888.38 16000.95 0.71 0.97 executed F
LEX D IAG on each combination of parameters three times to get average execution times trying to avoid thirdparty threads.Table 3 shows the data of those models categorized as realistic in the repository. We again see that F
LEX D IAG scales with no problem offering a good trade-off between accuracy and minimality while keeping the average runtime(in milliseconds) below a second.As we can observe in Table 3, while the execution time decreases when incrementing m , quality deteriorates again.However minimality is clearly affected while accuracy stays with minor variations, although, we can observe somespecial cases (”REAL-FM-5” with m = 2 ) were it deteriorates a bit more. In order to test F
LEX D IAG with large-scale real models, we encoded the variability existing in the
Debian packagingsystem for the
Ubuntu distribution and generated a set of configurations representing Ubuntu user installations withwrong package selections. Concretely, we modelled the Ubuntu Xenial distribution containing 58,107 packages and52,721 constraints. This model was extracted using the mapping presented in [20,21]. We executed F LEX D IAG withdifferent m values. We randomly generated different sizes of reconfiguration requirements that involved the 10%, 30%,50% and 100% of features of the model. Then we randomly reordered each of the reconfiguration requirements 10 times(to prevent ordering biases). Moreover, we executed F LEX D IAG on each combination of parameters three times to getaverage execution times trying to avoid third party threads.Table 4 shows that F
LEX D IAG is able to provide a good accuracy even with m set to 10. Also, it shows as expected,the negative impact of m regarding minimality. We observe that the execution time with m = 1 was . hours whilewith m = 10 it was . hours. This represents an improvement of runtime in 1.47 × . The benchmark used in this experiment includes three automotive configuration models from a German car manufac-turer. For each model, we randomly generated different sizes of reconfiguration requirements that involved the 10%,30%, 50% and 100% of features of the model. Then we randomly reordered each of the reconfiguration requirements 10 http://releases.ubuntu.com/16.04/ able 3 F LEX D IAG results in front of SPLOT realistic models. | V | represents the number of variables in the CSP. The second column showsthe m value used in F LEX D IAG . Model Name is the model name in SPLOT. | C | refers to the number of constraints in the model, | ∆ | isthe average size of a diagnosis, and average time (in milliseconds), average minimality , and average accuracy represent the means of thecalculated values for those metrics. | V | m Model Name | C | | ∆ | Average time Average minimality Average accuracy290 1 REAL-FM-4 61.00 1.00 527.45 1.00 1.002 1.75 527.83 0.62 1.004 2.75 464.33 0.48 1.006 4.00 444.65 0.40 1.0010 6.25 432.65 0.35 1.0088 1 REAL-FM-1 26.00 6.93 566.92 1.00 1.002 10.48 514.35 0.62 0.864 15.02 411.55 0.42 0.766 21.17 316.52 0.32 0.7010 22.58 317.92 0.28 0.7044 1 REAL-FM-20 8.00 3.50 225.32 1.00 0.992 5.35 201.23 0.60 0.864 8.07 171.22 0.38 0.846 10.15 155.45 0.31 0.8010 10.90 152.78 0.28 0.8023 1 REAL-FM-2 9.00 2.78 177.55 1.00 1.002 4.07 161.72 0.68 0.864 4.93 144.05 0.54 0.826 6.60 131.78 0.41 0.7610 7.62 216.98 0.36 0.7643 1 REAL-FM-3 13.00 1.75 211.97 1.00 1.002 3.03 199.62 0.60 0.954 4.78 178.35 0.36 0.936 7.38 161.82 0.24 0.9210 9.15 147.82 0.21 0.88
Table 4
Results obtained after executing F
LEX D IAG with the Ubuntu Xenial variability model. The seconds column shows the m value usedin F LEX D IAG . | V | represents the number of variables in the CSP. | C | refers to the number of constraints in the CSP, | ∆ | is the average sizeof a diagnosis, and average time (in milliseconds), average minimality , and average accuracy represent the means of the calculated values forthose metrics. | V | m | C | | ∆ | Average time Average minimality Average accuracy58107 1 105459 1.75 13523986.27 1.00 1.002 3.40 12245231.77 0.51 0.754 6.12 10800288.33 0.31 0.716 7.15 10546545.60 0.24 0.7110 12.48 9208147.25 0.14 0.71 times (to prevent ordering biases). Moreover, we executed F
LEX D IAG on each combination of parameters three timesto get average execution times trying to avoid third party threads.Table 5 shows that F
LEX D IAG is able to provide a good accuracy even with m set to 10. Again, it shows as expected,the negative impact of m regarding minimality. Also, the execution time for model with id = 1 and m = 1 was . hours while with m = 10 it was . minutes. This represents an improvement of 26.9 × .5.3 Comparing F LEX D IAG with Evolutionary AlgorithmsIn this research we do not compare F
LEX D IAG with more traditional diagnosis approaches – for related evaluationswe refer the reader to [15] were detailed analyses can be found. These analyses clearly indicate that direct diagnosisapproaches outperform standard diagnosis approaches based on the resolution of minimal conflicts [36] (if the searchgoal is to identify not all minimal but the so-called leading diagnoses which should, for example, be shown to users ininteractive settings).However, in this Section we compare F
LEX D IAG with an evolutionary algorithm inspired by [7,30]. The evolution-ary algorithm has been build with the jenetics framework for Java leaving all parameters default and fixing the processto 500 generations. Also we compare the performance of F LEX D IAG with m set to 1 to have a fair comparison. Notethat with higher values of m , we would even perform better in terms of runtime. http://jenetics.io/ able 5 F LEX D IAG evaluated with benchmarks from the automotive industry (calculation of the first diagnosis: ids m value used in F LEX D IAG . | V | represents the numberof variables in the CSP. | C | refers to the number of constraints in the CSP, | ∆ | is the average size of a diagnosis, and average time (inmilliseconds), average minimality , and average accuracy represent the means of the calculated values for those metrics.id m | V | | C | | ∆ | Average time Average minimality Average accuracy01 1 1888 7404 623.78 22812981.47 1.00 1.002 787.43 6720901.40 0.80 0.994 870.25 1785352.90 0.71 0.996 876.97 1716644.85 0.70 0.9910 892.85 473017.90 0.69 1.0002 1 1828 5451 611.38 8730834.77 1.00 1.002 760.85 3176414.95 0.81 0.994 845.47 914621.30 0.72 0.996 850.28 878283.40 0.71 1.0010 865.22 237203.47 0.70 1.0003 1 1843 8056 369.00 22147386.47 1.00 1.002 538.50 11901657.38 0.80 0.984 605.00 3381818.66 0.70 0.996 613.31 3104191.16 0.68 0.9910 626.59 823533.78 0.67 0.99
The first observation is that the evolutionary approach was not capable of dealing with very large realistic mod-els (Ubuntu, automotive) when setting a time-out of 24 hours. Therefore, we report this comparison only relying onrandomly generated models.
Figure 4 shows that the required time for F
LEX D IAG is usually higher for models having less than 500 features.Therefore, there is a point when F
LEX D IAG pays off and scales much better. Also, it is worth mentioning that theevolutionary algorithm was not capable of obtaining a complete and minimal explanation and returned only partialdiagnoses. This is, in 500 generations it only found partial explanations. l l l l l
50 100 500 1000 2000
Features T i m e i n m illi s e c ond s ( l og sc a l e ) Search l Flexdiag(m=1) Evolutionary
Fig. 4
Comparison between F
LEX D IAG and the evolutionary approach regarding time
Table 6 shows that F
LEX D IAG returns minimal diagnoses while we observe that the evolutionary approach was notcapable of detecting minimal diagnoses. Also, we do see that F
LEX D IAG offered a much better accuracy.11 able 6
Comparison results of the evolutionary approach and F
LEX D IAG . | V | represents the number of variables in the CSP. Approach refers to the used approach, | C | refers to the number of constraints in the CSP, | ∆ | is the average size of a diagnosis, and average time (inmilliseconds), average minimality , and average accuracy represent the means of the calculated values for those metrics. | V | Approach | C | | ∆ | Average time Averageminimality Averageaccuracy50 Evolutionary 58.40 1.53 3237.87 8.84 0.3350 F
LEX D IAG
LEX D IAG
LEX D IAG
LEX D IAG
LEX D IAG
External validity.
The inputs used for the experiments presented in this paper were either realistic or designed tomimic realistic feature models. The Debian feature model and the Automotive are realistic since numerous experts were involved in the design. However, since they were developed using a manual design process, it may have errors andnot encode all configurations. Also, the random feature models may not accurately reflect the structure of real featuremodels used in industry. The major threats to the external validity are: – Population validity , the real feature models that we used may not represent all valid configurations in the domainsdue it manual construction. Also, random models might not have the same structure as real models (e.g. mathemat-ical operators used in the complex constraints). To reduce these threats to validity, we generated the models usingpreviously published techniques [49] and using existing implementations of these techniques in Betty [40]. – Ecological validity : While external validity, in general, is focused on the generalization of the results to other con-texts (e.g. using other models), the ecological validity ii focused on possible errors in the experiment materials andtools used. To prevent ecological validity threats, such as third party threads running in the virtual machines andimpacting performance, the F
LEX D IAG analyses were executed three times and then averaged.
Internal validity
The CPU resources required to analyse a feature model depend on the number of features andpercentage of cross-tree constraints. However, there may be other variables that affect performance, such as the natureof the constraints used. To minimize these other possible effects, we introduced a variety of models to ensure that wecovered a large part of the constraint space.5.5 Final RemarksWe observed that F
LEX D IAG scales up with random and real-world feature models. Observing that, generally, diagnosisquality in terms of minimality and accuracy deteriorates with an increasing size of parameter m .Minimality and accuracy depend on the configuration domain and are not necessarily monotonous. For example,since a diagnosis determined by F LEX D IAG is not necessarily a superset of a diagnosis determined with m = 1 , it canbe the case that the minimality of a diagnosis determined with m > is greater than 1 (if F LEX D IAG determines adiagnosis with lower cardinality than the minimal diagnosis determined with m = 1 ). For simplicity, let us assume that AC = S = { c , c , c , c , c , c , c , c } and the following conflict sets CS i exist between the constraints c i ∈ S : CS : { c , c } , CS : { c , c } , and CS : { c , c } . Given m = 1 , F LEX D IAG would determine the diagnosis { c , c , c } whereas in the case of m = 2 , { c , c } is returned by the algorithm. The following simplified reconfiguration task is related to scheduling in production where it is often the case that, for example, schedules and corresponding production equipment has to be reconfigured. In this example setting, we donot take into account configurable production equipment (configurable machines) and limit the reconfiguration to the12ssignment of orders to corresponding machines. The assignment of an order o i to a certain machine m j is representedby the corresponding variable o i m j . The domain of each such variable represents the different possible slots in whichan order can be processed, for example, o m = 1 denotes the fact that the processing of order o on machine m isperformed during and finished after time slot 1.Further constraints restrict the way in which orders are allowed to be assigned to machines, for example, o m LEX D IAG . If we keep the ordering of the constraints as defined in S , F LEX D IAG (with m = 1 ) returns the diagnosis ∆ : { s , s , s , s , s , s } which can be used to determine the newsolution S (cid:48) = { s : o m = 3 , s : o m = 4 , s : o m = 5 , s : o m = 2 , s : o m = 3 , s : o m = 4 , s : o m = 1 , s : o m = 2 , s : o m = 3 } (see Table 7). If we change the parametrization to m = 2 , F LEX D IAG returns the same diagnosis but in approximately half of the time (with 10 iterations, milliseconds were needed on anaverage for m = 2 whereas milliseconds were needed for m = 1 ). This is consistent with the estimates in Table 1.Possible ordering criteria for constraints in such rescheduling scenarios can be, for example, customer value (changesrelated to orders of important customers should occur with a significantly lower probability) and the importance ofindividual orders. If some orders in a schedule should not be changed, this can be achieved by simply defining suchrequests as requirements ( R ρ ), i.e., change requests as well as stability requests can be included as constraints r (cid:48) i in R ρ . S S (cid:48) s : o m = 1 s : o m = 3 s : o m = 2 s : o m = 2 s : o m = 3 s : o m = 1 s : o m = 2 s : o m = 4 s : o m = 3 s : o m = 3 s : o m = 4 s : o m = 2 s : o m = 3 s : o m = 5 s : o m = 4 s : o m = 4 s : o m = 5 s : o m = 3 Table 7 Reconfiguration determined for rescheduling task – S represents the original configuration and S (cid:48) represents a configuration resultingfrom a reconfiguration task. In our work, we focused on the evaluation of reconfiguration scenarios where the knowledge base itself is assumed to beconsistent. In future work, we will extend the F LEX D IAG algorithm to make it applicable in scenarios where knowledge13ases are tested [12]. An example issue is to take into account situations where unintended configurations are acceptedby the knowledge base. In this context, we will extend the work of [12] by not only taking into account negative testcases but also automatically generate relevant test cases, for example, on the basis of mutation testing approaches. Weplan to extend our empirical evaluation to further industrial configuration knowledge bases. Furthermore, we want toanalyze in which way we are able to further improve the output quality (e.g., in terms of minimality and accuracy) ofF LEX D IAG , for example, by applying different constraint orderings depending on the observed interaction patterns (ofusers) and probability estimates for diagnosis membership derived thereof. The better potentially relevant constraintsare predicted the better the diagnosis quality in terms of the mentioned metrics of minimality and accuracy. Note that,for example, counting the number of elements already identified as partial diagnosis elements in F LEX D IAG does nothelp to keep diagnosis determination within certain time limits, however, this mechanism could be used when determingmore than one diagnosis to include diagnosis size as a relevance criterion. Also, in this context will analyze furtheralternatives to evaluate the quality of diagnoses which go beyond the metrics used in this article. Efficient reconfiguration functionalities are needed in various scenarios such as the reconfiguration of production sched-ules, the reconfiguration of the settings in mobile phone networks, and the reconfiguration of robot context information.We analyzed the F LEX D IAG algorithm with regard to potentials of improving existing direct diagnosis algorithms.When using F LEX D IAG , there is a clear trade-off between performance of diagnosis calculation and diagnosis quality(measured, for example, in terms of minimality and accuracy). Acknowledgements You can find the source code and material in https://jagalindo.github.io/FlexDiag/ References 1. H. Andersen, T. Hadzic, and D. Pisinger, ‘Interactive Cost Configuration Over Decision Diagrams’, Journal of Artificial IntelligenceResearch , , 99–139, (2010).2. F. Bacchus, J. Davies, M. Tsimpoukell, and G. Katsirelos, ‘Relaxation Search: A Simple Way of Managing Optional Clauses’, in AAAI’2014 , pp. 835–841, Quebec, Canada, (2014).3. R. Bakker, F. Dikker, F. Tempelman, and P. Wogmim, ‘Diagnosing and Solving Over-determined Constraint Satisfaction Problems’, in , pp. 276–281, Chambery, France, (1993).4. D. Benavides, S. Segura, and A. Ruiz-Cortes, ‘Automated Analysis of Feature Models 20 years Later: a Literature Review’, InformationSystems , (6), 615–636, (2010).5. D. Benavides, P. Trinidad, A. Ruiz-Cort´es, and S. Segura, ‘Fama’, in Systems and Software Variability Management , 163–171, Springer,(2013).6. R. Bryant, ‘Symbolic Boolean Manipulation with Ordered Binary-Decision Diagrams’, ACM Computing Surveys , (3), 293–318, (1992).7. Bojana ´Cendi´c-Lazovi´c, ‘A Genetic Algorithm for the Minimum Hitting Set’, Scientific Publications of the State University of Novi PazarSeries A: Applied Mathematics, Informatics and mechanics , (2), 107–117, (2014).8. J. Crow and J. Rushby, ‘Model-Based Reconfiguration: Toward an Integration with Diagnosis’, in , pp. 836–841, Anaheim, California, (1991). The MIT Press.9. J. DeKleer, ‘Using Crude Probability Estimates to Guide Diagnosis’, AI Journal , (3), 381–391, (1990).10. A. Falkner, A. Felfernig, and A. Haag, ‘Recommendation Technologies for Configurable Products’, AI Magazine , (3), 99–108, (2011).11. A. Falkner and H. Schreiner, ‘SIEMENS: Configuration and Reconfiguration in Industry’, in Knowledge-based Configuration – FromResearch to Business Cases , eds., A. Felfernig, L. Hotz, C. Bagley, and J. Tiihonen, chapter 16, 251–264, Morgan Kaufmann Publishers,(2014).12. A. Felfernig, G. Friedrich, D. Jannach, and M. Stumptner, ‘Consistency-based Diagnosis of Configuration Knowledge Bases’, ArtificialIntelligence , (2), 213–234, (2004).13. A. Felfernig, L. Hotz, C. Bagley, and J. Tiihonen, Knowledge-based Configuration: From Research to Business Cases , Elsevier/MorganKaufmann, 1st edn., 2014.14. A. Felfernig, M. Schubert, G. Friedrich, M. Mandl, M. Mairitsch, and E. Teppan, ‘Plausible Repairs for Inconsistent Requirements’, in , pp. 791–796, Pasadena, CA, USA, (2009).15. A. Felfernig, M. Schubert, and C. Zehentner, ‘An Efficient Diagnosis Algorithm for Inconsistent Constraint Sets’, Artificial Intelligencefor Engineering Design, Analysis and Manufacturing (AI EDAM) , (1), 53–62, (2012).16. A. Fijany and F. Vatan, ‘New Approaches for Efficient Solution of Hitting Set Problem’, in International Symposium on Information andCommunication Technologies , pp. 1–6, Cancun, Mexico, (2004).17. G. Fleischanderl, G. Friedrich, A. Haselb¨ock, H. Schreiner, and M. Stumptner, ‘Configuring Large Systems Using Generative ConstraintSatisfaction’, IEEE Intelligent Systems , (4), 59–68, (1998).18. F. Frayman and S. Mittal, ‘COSSACK: A Constraint-Based Expert System for Configuration Tasks’, in Knowledge Based Expert Systemsin Engineering: Planning and Design , eds., D. Sriram and R. Adey, 143–166, Computational Mechanics Publications, Woburn, MA,USA, (1987). 9. G. Friedrich, A. Ryabokon, A. Falkner, A. Haselb¨ock, G. Schenner, and H. Schreiner, ‘ (Re)configuration using Answer Set Programming’,in IJCAI 2011 Workshop on Configuration , pp. 17–24, (2011).20. J. Galindo, D. Benavides, and S. Segura, ‘Debian Packages Repositories as Software Product Line Models. Towards Automated Analysis’,in ACoTA , pp. 29–34, (2010).21. J. Galindo, F. Roos-Frantz, J. Garc´ıa-Gal´an, and A. Ruiz-Cort´es, ‘Extracting Orthogonal Variability Models from Debian Repositories’, in Proc. of 2nd International Workshop on Formal Methods and Analysis in Software Product Line Engineering (FMSPLE 2011), co-locatedwith Software Product Line Conference 2011 (SPLC 2011) , pp. 8–8, Munich, (08/2011 2011). Fraunhofer, Fraunhofer.22. E. Gregoire, J. Lagniez, and B. Mazure, ‘An Experimentally Efficient Method for (MSS, CoMSS) Partitioning’, in , pp. 2666–2673, Quebec, Canada, (2014).23. A. Haag, ‘Product Configuration in SAP - A Personal Retrospective’, in Knowledge-based Configuration – From Research to BusinessCases , eds., A. Felfernig, L. Hotz, C. Bagley, and J. Tiihonen, chapter 27, 389–411, Morgan Kaufmann Publishers, (2014).24. L. Hotz, A. Felfernig, M. Stumptner, A. Ryabokon, C. Bagley, and K. Wolter, ‘Configuration Knowledge Representation & Reasoning’,in Knowledge-based Configuration – From Research to Business Cases , eds., A. Felfernig, L. Hotz, C. Bagley, and J. Tiihonen, chapter 6,59–96, Morgan Kaufmann Publishers, (2014).25. D. Jannach, ‘Finding Preferred Query Relaxations in Content-based Recommenders’, in ,pp. 355–360, London, UK, (2006).26. D. Jannach, M. Zanker, A. Felfernig, and G. Friedrich, Recommender Systems – An Introduction , Cambridge University Press, 2010.27. M. Janota, G. Botterweck, and J. Marques-Silva, ‘On Lazy and Eager Interactive Reconfiguration’, in , pp. 1–8, San Antonio, TX, USA, (2014).28. U. Junker, ‘QuickXPlain: Preferred Explanations and Relaxations for Over-constrained problems’, in , pp. 167–172. AAAI Press, (2004).29. L. Hvam and N. Mortensen and H. Riis, Product Customization , Springer, 2008.30. L. Li and J. Yunfei, ‘Computing Minimal Hitting Sets with Genetic Algorithm’, Technical report, Zhongshan (Sun Yatsen) University,China, (2002).31. A. Mackworth, ‘Consistency in Networks of Relations’, Artificial Intelligence , (1), 99–118, (1977).32. J. Marques-Silva, F. Heras, M. Janota, A. Previti, and A. Belov, ‘On Computing Minimal Correction Subsets’, in IJCAI 2013 , pp. 615–622,Peking, China, (2013).33. C. Mencia, A. Previti, and J. Marques-Silva, ‘Efficient Relaxations of Over-constrained CSPs’, in ICTAI’2014 , pp. 725–732, (2014).34. I. Nica, F. Wotawa, R. Ochenbauer, C. Schober, H. Hofbauer, and S. Boltek, ‘Kapsch: Reconfiguration of Mobile Phone Networks’, in Knowledge-based Configuration – From Research to Business Cases , eds., A. Felfernig, L. Hotz, C. Bagley, and J. Tiihonen, chapter 19,287–300, Morgan Kaufmann, (2014).35. G. Provan and Y. Chen, ‘Model-based Diagnosis and Control Reconfiguration for Discrete Event Systems: an Integrated Approach’, in Proceedings of the 38th IEEE Conference on Decision and Control , volume 2, pp. 1762–1768, Phoenix, AZ, USA, (1999).36. R. Reiter, ‘A Theory of Diagnosis From First Principles’, Artificial Intelligence , (1), 57–95, (1987).37. D. Sabin and R. Weigel, ‘Product Configuration Frameworks - A Survey’, IEEE Intelligent Systems , (4), 42–49, (1998).38. F. Salvador and C. Forza, ‘Principles for Efficient and Effective Sales Configuration Design’, International Journal of Mass Customization , (1–2), 114–127, (2007).39. M. Schubert and A. Felfernig, ‘BFX: Diagnosing Conflicting Requirements in Constraint-based Recommendation’, International Journalon Artificial Intelligence Tools , (2), 297–312, (2011).40. S. Segura, J. Galindo, D. Benavides, J. Parejo, and A. Ruiz-Cort´es, ‘BeTTy: Benchmarking and Testing on the Automated Analysis ofFeature Models’, in Proceedings of the Sixth International Workshop on Variability Modeling of Software-Intensive Systems , VaMoS’12,pp. 63–71, New York, NY, USA, (2012). ACM.41. I. Shah, ‘Direct Algorithms for Finding Minimal Unsatisfiable Subsets In Over-Constrained CSPs’, International Journal on ArtificialIntelligence Tools , (1), 53–91, (2011).42. K. Shchekotykhin, G. Friedrich, P. Rodler, and P. Fleiss, ‘Sequential Diagnosis of High-cardinality Faults in Knowledge Bases by DirectDiagnosis Generation’, in ECAI 2014 , pp. 813–818, Prague, Czech Republic, (2014).43. C. Sinz, A. Kaiser, and W. K¨uchlin, ‘Formal Methods for the Validation of Automotive Product Configuration Data’, Artificial Intelligencefor Engineering Design, Analysis and Manufacturing , (1), 75–97, (2003).44. G. Steinbauer, M. M¨orth, and F. Wotawa, ‘Real-Time Diagnosis and Repair of Faults of Robot Control Software’, in RoboCup 2005 ,LNAI, pp. 13–23. Springer, (2005).45. M. Stumptner, ‘An Overview of Knowledge-based Configuration’, AI Communications , (2), 111–126, (1997).46. M. Stumptner, A. Haselb¨ock, and G. Friedrich, ‘COCOS - A Tool for Constraint-based, Dynamic Configuration’, in , pp. 373–380, San Antonio, TX, USA, (1994).47. M. Stumptner and F. Wotawa, ‘Reconfiguration Using Model-based Diagnosis’, in , pp. 266–271, (1999).48. O. Tange, ‘GNU Parallel – The Command-Line Power Tool’, ;login: The USENIX Magazine , (1), 42–47, (February 2011).49. T. Thum, D. Batory, and C. Kastne, ‘Reasoning about edits to feature models’, in Software Engineering, 2009. ICSE 2009. IEEE 31stInternational Conference on , pp. 254–264. IEEE, (2009).50. J. Tiihonen and A. Anderson, ‘VariSales’, in Knowledge-based Configuration – From Research to Business Cases , eds., A. Felfernig,L. Hotz, C. Bagley, and J. Tiihonen, chapter 26, 377–388, Morgan Kaufmann Publishers, (2014).51. J. Tiihonen, W. Mayer, M. Stumptner, and M. Heiskala, ‘Configuring Services and Processes’, in Knowledge-based Configuration – FromResearch to Business Cases , eds., A. Felfernig, L. Hotz, C. Bagley, and J. Tiihonen, chapter 21, 313–324, Morgan Kaufmann Publishers,(2013).52. R. Walter, A. Felfernig, and W. K¨uchlin, ‘Constraint-Based and SAT-Based Diagnosis of Automotive Configuration Problems’, Journalof Intelligent Information Systems (JIIS) , 1–32, (2016).53. R. Walter and W. K¨uchlin, ‘ReMax - A MaxSAT aided Product (Re-) Configurator’, in Workshop on Configuration 2014 , pp. 55–66,(2014).54. R. Walter, C. Zengler, and W. K¨uchlin, ‘Applications of MaxSAT in Automotive Configuration’, in Workshop on Configuration 2013 , pp.21–28, (2013).55. K. Wang, Z. Li, Y. Ai, and Y. Zhang, ‘Computing Minimal Diagnosis with Binary Decision Diagrams Algorithm’, in , pp. 145–149, (2009).56. J. White, D. Benavides, D. Schmidt, P. Trinidad, B. Dougherty, and A. Ruiz-Cortez, ‘Automated Diagnosis of Feature Model Configura-tions’, Systems and Software , (7), 1094–1107, (2010).57. F. Wotawa, ‘A Variant of Reiter’s Hitting-Set Algorithm’, Inf. Processing Letters , (1), 45–51, (2001).(1), 45–51, (2001).