Towards Utility-based Prioritization of Requirements in Open Source Environments
Alexander Felfernig, Martin Stettinger, Müslüm Atas, Ralph Samer, Jennifer Nerlich, Simon Scholz, Juha Tiihonen, Mikko Raatikainen
TTowards Utility-based Prioritization ofRequirements in Open Source Environments
Alexander Felfernig ∗ , Martin Stettinger ∗ , M¨usl¨um Atas ∗ , Ralph Samer ∗ ,Jennifer Nerlich † , Simon Scholz † , Juha Tiihonen ‡ , and Mikko Raatikainen ‡∗ Graz University of Technology, Austria, Email: { afelfern,mstettinger,muatas,rsamer } @ist.tugraz.at, † vogella GmbH, Hamburg, Germany, Email: { jennifer.nerlich,simon.scholz } @vogella.com, ‡ University of Helsinki, Finland, Email: { juha.tiihonen,mikko.raatikainen } @helsinki.fi Abstract —Requirements Engineering in open source projectssuch as E
CLIPSE faces the challenge of having to prioritizerequirements for individual contributors in a more or less un-obtrusive fashion. In contrast to conventional industrial softwaredevelopment projects, contributors in open source platforms candecide on their own which requirements to implement next.In this context, the main role of prioritization is to supportcontributors in figuring out the most relevant and interestingrequirements to be implemented next and thus avoid time-consuming and inefficient search processes. In this paper, weshow how utility-based prioritization approaches can be used tosupport contributors in conventional as well as in open sourceRequirements Engineering scenarios. As an example of an opensource environment, we use B
UGZILLA . In this context, we alsoshow how dependencies can be taken into account in utility-basedprioritization processes.
I. I
NTRODUCTION
In software projects, resources are typically limited whichrequires the prioritization of requirements [16]. Prioritizationis often interpreted as a part of strategic planning where thefocus is to select and prioritize requirements that should beincluded in releases (long-term release planning) [4], [19].Decision support in prioritization is extremely important sinceespecially when dealing with large assortments of require-ments, manual prioritization processes tend to become verycostly [3], [28]. In this context, suboptimal prioritizationstrigger time wasting due to the implementation of unimportantrequirements.There are two basic approaches to prioritize requirements– for an in-depth related analysis we refer to Achimugu et al.[1].
First , requirements prioritization can be interpreted as an optimization task where the overall objective is to identify the middle ground , i.e., an aggregation of individual prioritizationsinto a global prioritization that reflects the least possible levelof dissimilarity from all stakeholder-individual prioritizations[14].
Second , in contrast to approximating individual priori-tizations on the basis of optimization functions, utility-basedapproaches focus on (1) establishing agreement with regardto the evaluation of individual requirements and (2) thereafterdetermining prioritizations [2], [11], [26].Prioritizations following the optimization approach are de-termined on the basis of individual prioritizations of stake-holders. When following a utility-based approach , preferencesof stakeholders are first aggregated and a prioritization is determined thereafter. In the line of basic approaches to deter-mine group recommendations [9], the first approach is basedon aggregated prioritizations where stakeholder-individualprioritizations are known and a recommendation minimizesdissimilarities between the given prioritizations (preferences).The second approach is based on aggregated models wherestakeholder requirement evaluations are aggregated first anda prioritization is determined on the basis of a group profile(model) derived from requirements evaluations.Aggregated models have the advantage that stakeholdersare encouraged to focus their evaluations on specific relevantaspects of a requirement (e.g., dimensions such as profit, risk,and effort) and thus contribute to stable preferences and ahigher degree of consensus [22]. Aggregated prioritizationstrigger scalability issues since each stakeholder has to provide,for example, a ranked list of requirements as input for theoptimization process. Furthermore, due to the computationalcomplexity of the underlying problem, an optimal solutioncannot be guaranteed and is often only approximated on thebasis of local search algorithms [14], [24], [29]. Utility-basedapproaches as discussed in this paper focus on evaluations ofindividual requirements on the basis of different evaluation di-mensions (e.g., profit, effort, and risk). This way, stakeholderscan focus on evaluating requirements they have knowledgeabout and the focus of prioritization is first on establishing consensus and thereafter on figuring out the most relevantprioritizations [22].Different algorithmic approaches can be used to supportrequirements prioritization – for an overview, see, for exam-ple, Achimugu et al. [1]. Examples thereof are constraint-based reasoning [25], incremental preference learning [17],evolutionary algorithms [14], machine learning [3], [23], andpairwise preference-based decision making [20].
Optimization-based approaches focus on minimizing the distance betweenthe preferences of individual stakeholders (e.g., in terms ofthe distance between individual prioritizations and the priori-tization determined by the optimization approach). A similarproblem also solved on the basis of optimization approachesis the next release problem [6], [27] where a subset of agiven set of requirements has to be selected in such a waythat predefined cost limits are taken into account and thechosen set of requirements represents the optimum choicein terms of criteria such as market value. The focus in this a r X i v : . [ c s . S E ] F e b ontext is more to identify subsets of requirements but notto prioritize a given list of requirements. In contrast to nextrelease problems, prioritization tasks in open source scenariosdo not necessarily require (and often do not allow) a globaloptimum but more focus on relevant recommendations forindividual stakeholders. Also in contrast to existing releaseplanning tasks, developers in open source scenarios in mostof the cases do not explicitly define their preferences, i.e.,preferences have to derived from given interaction data (inour case, interaction data collected by B UGZILLA ). Utility-based prioritization based on multi-attribute utilitytheory [8] can be implemented in different variants. First,requirements are simply evaluated with regard to a set ofpredefined interest dimensions and the overall utility of arequirement is determined as a sum of interest dimensionspecific utilities. Second, weights can be introduced to em-phasize on specific interest dimensions (e.g., a lower risk ismore important than high profits). Third, stakeholders can beenabled to define their personal evaluations and utility-basedapproaches should then be able to aggregate these evalua-tions and take into account stakeholder weights. Stakeholderweights can be interpreted as ”global”, i.e., there is a globalweighting of stakeholders independent of a specific dimensionor requirement. If weights are interpreted as ”local”, the impor-tance of a stakeholder can be defined on the level of individualrequirements or dimensions. Utility-based prioritization canalso be implemented on the basis of analytic hierarchy process(AHP) [13]. A major disadvantage of this approach is thatrequirements have to evaluated pairwise which does not scalewell when the number of requirements increases.Prioritization criteria differ depending on the requirementsengineering scenario. The criteria effort , risk , and profit areoften used in settings where a group of stakeholders engagedin the same project is in charge of completing a prioritizationtask [1]. In contrast, in open-source settings, developers arein most of the cases engaged in different projects and alsowork for different companies. In such scenarios, prioritizationis less focusing on establishing consensus between individualstakeholders but more on supporting stakeholders in iden-tifying requirements of relevance to them and to prioritizethe important ones by also taking into account global cri-teria. Examples of criteria in such scenarios are personalexpertise of a developer and importance of a requirementfor the community of the stakeholder and the open sourcecommunity as a whole. Thus, open source platform relatedprioritization processes completely differ from conventionalsoftware projects. A major focus of this paper is to introduceprioritization concepts especially applicable in open sourcedevelopment contexts.B UGZILLA is an open-source based issue tracking systemwhich supports users from different geographical locations toreport their findings with regard to a given set of softwarecomponents. Users can submit textual descriptions of issuesand corresponding meta-information, for example, associated components, keywords, and dependencies. Reported issues canbe selected by contributors to work on. In B UGZILLA , issuescan be requirements but also reported bugs. Distinguishing be-tween these can be performed on the basis of a meta-attribute(issue type) that can be specified for B
UGZILLA issues. Thereare different related approaches to support machine learningbased requirements prioritization. The approaches operate ondatasets including historical data of previous requirement (bugreport in B
UGZILLA ) selections and try to predict futurerequirement selections thereof. Utility-based prioritization canbe used in interactive scenarios (stakeholders are engaged inan interactive prioritization process) as well as scenarios whererequirements are recommended but no further stakeholderinteraction is needed for determining a prioritization.The contributions of this paper are the following. We pro-vide an overview of different application scenarios of utility-based requirements prioritization and discuss specific aspectsof requirements prioritization in open source projects. Forscenarios that include dependencies between requirements, weshow how such dependencies can be taken into account on thebasis of the concepts of model-based diagnosis [10], [18]. Withthis approach, we tackle the following research gaps . In con-trast to existing prioritization and release planning approaches,we introduce model-based diagnosis concepts that also supportre-prioritization and re-planning while not completely omittingalready existing stakeholder preferences which is still anopen issue in most of the existing prioritization and releaseplanning approaches (these approaches focus on taking intoaccount dependencies but do not support the aforementionedre-prioritization and re-planning scenarios). Furthermore, wepresent a first version of a user interface developed to supportprioritization tasks in open source environments – in our case,for B
UGZILLA users. This approach has the potential to reducethe workload of developers in open source platforms by auto-matically proposing requirements that should be implementednext instead of forcing users to analyze in detail a largenumber of requirements. Furthermore, the introduced utility-based approach does not require the ”manual” evaluationof individual requirements but automatically derives utilitymodels by analyzing given interaction logs taking into accountinteraction data such as number of comments related to arequirement. This is a major difference compared to existingrequirements prioritization approaches which do not supportautomated prioritization based on background data. Finally,we discuss issues for future work to further advance the stateof the art in utility-based prioritization.The remainder of this paper is organized as follows. InSection II, we introduce variants of implementing utility-based prioritization . Thereafter, we introduce our variant ofutility-based prioritization implemented for the B
UGZILLA environment and provide a sketch of a related B
UGZILLA userinterface (Section III). In Section IV, we show how to extendutility-based prioritization in such a way that dependenciesbetween requirements can be taken into account. The paper isconcluded with a discussion of future work in Section V.I. U
TILITY - BASED P RIORITIZATION
Utility-based prioritization allows stakeholders to prioritizea requirement with regard to different interest dimensions D = { d , d , ..., d n } . Examples of such interest dimensionsare profit , risk , and effort . Utility-based prioritization is basedon the idea to first evaluate each requirement with regardto the set of interest dimensions (see Table I) and thereaftercalculate the individual utility of each requirement (see For-mula 1). In general, the priority is associated with the utility of a requirement r which results from its total contributionsto all of each individual interest dimensions d (denoted as contribution ( r, d ) ) combined with the corresponding impor-tance weights of individual interest dimensions (denoted as weight ( d ) ). TABLE IC
ONTRIBUTION OF REQUIREMENTS R = { r , r , r } TO THE INTERESTDIMENSIONS D = { profit, risk, effort } .interest dimension r r r profit 10 5 4risk 7 2 8effort 2 3 7TABLE IIP REDEFINED WEIGHTS FOR THE INTEREST DIMENSIONS D = { profit, risk, effort } .interest dimension weightsprofit 0.3risk 0.5effort 0.2 utility ( r, D ) = Σ d ∈ D ( contribution ( r, d ) × weight ( d )) (1)Applying Formula 1 to the entries in Tables I and II resultsin the ranking depicted in Table III (the higher the utilitywith regard to the given interest dimensions, the higher thecorresponding priority of the requirement). TABLE IIIR
ANKING OF REQUIREMENTS WITH STATIC WEIGHTS .requirement r r r utility 6.9 3.1 6.6priority (ranking) 1 3 2 In the previous example, the evaluation of requirements withregard to interest dimensions and the weighting of interestdimensions are assumed to be predefined (e.g., by a singlestakeholder). However, requirements prioritization is often agroup decision process [9] where different stakeholders areevaluating requirements (see, e.g., Table IV) and define im-portance weights with regard to interest dimensions (see, e.g.,Table V). Both, stakeholder-individual evaluations of interest dimensions and importance weights have to be aggregated.Formula 2 shows the aggregation of stakeholder-individualevaluations of requirements where S refers to the set whichincludes all m stakeholders (i.e., S = { s , s , ..., s m } ). contribution ( r, d, S ) = Σ s ∈ S eval ( d, r, s ) | S | (2)Formula 3 shows how to aggregate the stakeholder-specificimportance weights (denoted as w ( d, s ) ) which are related toindividual interest dimensions d . Previous calculations did nottake into account potential different degrees of stakeholderexpertise, for example, a stakeholder s a could have moreexpertise with regard to estimating the market potential of arequirement in terms of profit as estimating the correspondingdevelopment efforts. To take into account this aspect, Formula3 includes a factor that represents the expertise of a stakeholder s with regard to a specific interest dimension d . weight ( d, S ) = Σ s ∈ S w ( d, s ) × expertise ( d, s ) | S | (3)Similar to the basic approach, the utility of a requirement(Formula 4) is determined as a combination of the contribu-tions of a requirement to the given interest dimensions andrelated interest dimension importance evaluations of stake-holders. utility ( r, D, S ) = Σ d ∈ D ( contribution ( r, d, S ) × weight ( d, S )) (4)The result of applying Formulae 2–4 to the evaluation datacontained in Tables IV and V is depicted in Table VI.III. U TILITY - BASED P RIORITIZATION IN B UGZILLA
In Section II, we took a look at different variants of utility-based prioritization. These variants were discussed on the basisof interest dimensions (evaluation criteria) typically occurringin software projects where a group of stakeholders is in chargeof jointly defining and prioritizing requirements. In this sec-tion, we focus on open source scenarios where individual users(e.g., contributors in an open source platform) follow theirindividual interests regarding requirements without necessarilytaking into account the preferences of other users. This canbe considered a major difference compared to conventionalsoftware projects where stakeholders commonly develop a”global” prioritization (see Section II). We now show howutility-based prioritization can be applied in such contexts.Table VII represents a B
UGZILLA -specific evaluation ofrequirements (bugs) with regard to the set of interest dimen-sions { cc, geritt, blocker, comments } . In this context, cc isthe number of contributors who are in the :cc list of bug-related emails, geritt is the number of bug-related G ERITT changes, blocker is the number of dependent bugs (dependent A code reviewing tool – gerritcodereview.com.ABLE IVC
ONTRIBUTION OF REQUIREMENTS R = { r , r , r } TO DIMENSIONS D = { profit, risk, effort } ( DEFINED BY STAKEHOLDERS S = { s , s , s } ).interest r r r dimension s s s AV G s s s AV G s s s AV G profit 5 2 2 3.0 5 1 2 2.7 2 2 6 3.3risk 3 3 4 3.3 2 5 6 4.3 3 2 2 2.3effort 2 3 2 2.3 3 4 2 3.0 5 6 2 4.3TABLE VP
REFERENCES OF STAKEHOLDERS S = { s , s , s } WITH REGARD TOTHE INTEREST DIMENSIONS D = { profit, risk, effort } .stakeholder s s s weightsprofit 0.5 0.3 0.6 0.47risk 0.3 0.6 0.3 0.4effort 0.2 0.1 0.1 0.13TABLE VIR ANKING OF REQUIREMENTS WITH GROUP WEIGHTS .requirement r r r utility 3.03 3.57 3.03priority (ranking) 2 1 2 requirements), and comments refers to the number of bug-related comments. These interest dimensions do need to beevaluated manually as it is often the case in other scenarios[15], [21] but can directly be determined from correspondinguser interaction data.Formula 5 supports the calculation of the contribution of arequirement r to a specific interest dimension d . In sharp con-trast to the previous scenarios, the contribution is not directlyspecified by stakeholders but derived from B UGZILLA specificinformation (e.g., contribution ( r, d ) = eval ( r, d ) (5)Formula 6 supports the determination of the expertise ofa stakeholder which is represented in terms of the similaritybetween the keywords stored in the stakeholder profile andthose extracted from the requirement description and corre-sponding meta-information (e.g., name of associated compo-nent/system). The similarity between requirement-related key-words, meta-information, and contributor profile informationis interpreted as expertise (see Formula 6). weight ( r, s ) = expertise ( r, s ) (6)In the line of the previously discussed utility functions, theoverall utility of a requirement is interpreted as a combinationof (1) the contributions of a requirement to a set of interestdimensions and (2) the expertise level of a stakeholder (inthis context interpreted ”globally”, i.e., not on the level ofindividual interest dimensions). utility ( r, s ) = Σ d ∈ D contribution ( r, d ) × weight ( r, s ) (7) TABLE VIIC
ONTRIBUTION OF REQUIREMENTS ( BUGS ) R = { r , r , r } TO THEINTEREST DIMENSIONS D = { cc, geritt, blocker, comments } .interest dimension r r r cc 5 2 2geritt 3 3 4blocker 2 3 2comments 2 3 2TABLE VIIIE XPERTISE OF STAKEHOLDER s WITH REGARD TO THE REQUIREMENTS { r , r , r } DETERMINED , FOR EXAMPLE , ON THE BASIS OF THESIMILARITY BETWEEN THE STAKEHOLDER PROFILE AND INFORMATIONASSOCIATED WITH A REQUIREMENT .stakeholder s r r r ANKING OF B UGZILLA BUGS WITH STATIC WEIGHTS .requirement r r r utility 6.0 3.3 2.0priority 1 2 3 IV. T
AKING INTO A CCOUNT D EPENDENCIES
Blocking Factor . Utility-based recommendation approachesper se do not explicitly take into account dependencies be-tween requirements (e.g., requirement A must be implementedbefore requirement B ). In the discussed open source prioriti-zation scenario, this aspect is taken into account by prioritizingrequirements on the basis of the number of related dependen-cies, i.e., the higher the number of requirements dependentfrom a requirement x , the higher the ”global” relevanceof x . In this context, the blocking factor (i.e., how manyrequirements depend on the implementation of requirement x ) can be considered as interest dimension that has an impacton prioritization. In other words, this requirement should beimplemented as soon as possible since it otherwise blocksthe implementation of other requirements. This approach canalso be applied in software development scenarios where agroup of stakeholders (e.g., an in-house software developmentproject) is in charge of prioritizing requirements. Such anapproach helps to avoid situations where prioritizations violatedependency constraints. Automated Repair . An alternative approach is to applyrepair mechanisms from model-based diagnosis [10] that ig. 1. B
UGZILLA view on bugs (requirements). Based on the presented utility-based prioritization approach, bugs are presented to B
UGZILLA contributorsin the order of the determined priority. help to adapt already determined prioritizations in such away that all defined dependencies are taken into account.In the following, we will shortly sketch our approach. Inorder to trigger a diagnosis process, we are in the needof a pre-defined set of dependencies between requirements(denoted as
DEP = { dep , dep , .., dep n } ). Furthermore,we assume that a prioritization (represented as sequence) P = [ p , p , .., p m ] determined by a utility-based prioritizationapproach is inconsistent with the given set of dependencies.In order to apply model-based diagnosis, we assume thatboth, the pre-defined set of dependencies and the requirementprioritization is represented in terms of constraints [25], forexample, DEP = { dep : r < r , dep : r < r } and P = { p : r < r , p : r < r , p : r Definition: Conflict Set (CS) . A conflict set CS ⊆ P is aset of individual prioritization elements that are inconsistentwith the elements of DEP , i.e., inconsistent ( CS ∪ DEP ). Aconflict set CS is minimal if ¬∃ CS (cid:48) : CS (cid:48) ⊂ CS .On the basis of a set of identified minimal conflict sets,a corresponding diagnosis includes a set of prioritizationelements in P that have to be adapted such that the consistencyof DEP ∪ P is restored (see the following definition). Definition: Diagnosis ( ∆ ) . A diagnosis ∆ ⊆ P is a set ofindividual prioritization elements that have to be deleted from P such that consistent ( P − ∆ ∪ DEP ).In our example, CS : { p } is the only conflict induced in P by the dependencies defined in DEP . CS is minimal, i.e., weneed to adapt only one of the prioritization elements in CS such that a global prioritization can be found that is consistentwith the elements in DEP [12]. A corresponding diagnosis ∆ is { p } . In our example, we could decide to replace p : r < r with the corresponding repair r > r . This is a repairaction that helps to restore the consistency of DEP ∪ P . Our approach to the repair of inconsistent prioritizations canbe used for both, interactive prioritization where stakeholdersreceive feedback on the consistency of prioritizations, and automated prioritization where repairs for inconsistent prior-itizations are determined in an automated fashion. Importantissues to improve our approach are discussed in the following.V. C ONCLUSION AND F UTURE W ORK Conclusion . In this paper, we showed how to support utility-based requirements prioritization. These scenarios range from single user prioritization where one stakeholder is in chargeof completing prioritization tasks to group-based prioritization where the preferences/evaluations of different group membershave to be taken into account. On the basis of these scenarios,we showed how utility-based prioritization can be appliedin the context of open source development projects. In thiscontext, we sketched our initial implementation currentlyprovided in the B UGZILLA environment. This implementationserves as a first version to support prioritization in B UGZILLA . Future Work . Since prioritization is a repetitive process,we will include mechanisms that are capable of learningstakeholder weights and also the weights of individual re-quirements. This approach will help to further increase theprediction quality of prioritizations in terms of the probabilitythat stakeholders accept the proposed prioritizations. In thiscontext, we will also compare the predictive quality of utility-based approaches (i.e., approaches based on aggregated mod-els) with machine learning based approaches and approachesthat determine rankings on the basis of aggregated prioriti-zations. Furthermore, we will analyze which further features(interest dimensions) are useful to improve prediction quality.For example, the number of redundant bugs (issues) can be afurther important relevance indicator.A major challenge in requirements prioritization is theprovision of persuasive user interfaces that increase the pre-paredness of stakeholders to actively engage in requirementsengineering processes [5]. Consequently we will focus ona further extension/improvement of the existing B UGZILLA requirements prioritization user interface. Furthermore, we willanalyze in which way recommended prioritizations have toe explained to support specific group decision goals such as consensus , fairness , and decision quality [7], [9]. Finally, wewill conduct a detailed empirical study regarding the impactof our prioritization approaches in the E CLIPSE community.A CKNOWLEDGMENT The work presented in this paper has been conducted withinthe scope of the W E W ANT Project (funded by the AustrianResearch Promotion Agency) and the Horizon 2020 ProjectO PEN R EQ . R EFERENCES[1] P. Achimugu, A. Selamat, R. Ibrahim, and M. Mahrin. A systematicliterature review of software requirements prioritization research. Infor-mation and Software Technology , 56(6):568–585, 2014.[2] G. Adomavicius, N. Manouselis, and Y. Kwon. Recommender SystemsHandbook , chapter Multi-Criteria Recommender Systems, pages 769–803. Springer, 1st edition, 2010.[3] M. Alenezi and S. Banitaan. Bug reports prioritization: Which featuresand classifier to use? In , pages 112–116, 2013.[4] D. Ameller, C. Farre, X. Franch, D. Valerio, and A. Cassarino. Towardscontinuous software release planning. In ,pages 402–406, 2017.[5] M. Atas, A. Felfernig, M. Stettinger, and TNT. Tran. Beyond itemrecommendation: Using recommendations to stimulate knowledge shar-ing in group decisions. In , pages 1–10, Oxford, UK, 2017.[6] A. Bagnall, V. Rayward-Smith, and I. Whittley. The next releaseproblem. Information and Software Technology , 43:883–890, 2001.[7] G. Du and G. Ruhe. Does explanation improve the acceptance ofdecision support for product release planning? In , pages56–68. IEEE, 2009.[8] J. Dyer. Multi attribute utility theory. International Series in OperationsResearch and Management Science , 78:265–292, 1997.[9] A. Felfernig, L. Boratto, M. Stettinger, and M. Tkalcic. Group Recom-mender Systems – An Introduction . Springer, 2018.[10] A. Felfernig, M. Schubert, and C. Zehentner. An Efficient DiagnosisAlgorithm for Inconsistent Constraint Sets. Artificial Intelligence for En-gineering Design, Analysis, and Manufacturing (AIEDAM) , 26(1):175–184, 2012.[11] S. Huang. Designing utility-based recommender systems for e-commerce: Evaluation of preference elicitation methods. ElectronicCommerce Research and Applications , 10(4):398–407, 2011. [12] U. Junker. Q UICK X PLAIN : Preferred Explanations and Relaxationsfor Over-Constrained Problems. In , pages 167–172, San Jose, CA, 2004.[13] J. Karlsson and K. Ryan. A Cost-Value Approach for PrioritizingRequirements. IEEE Software , 14(5):67–74, 1997.[14] F. Kifetew, A. Susi, D. Mutante, A. Perini, A. Siena, and P. Busetta.Towards multi-decision-maker requirements prioritisation via multi-objective optimisation. In Forum and Doctoral Consortium PapersPresented at the 29th International Conference on Advanced InformationSystems Engineering (CAiSE’17) , pages 137–144, Essen, Germany,2017.[15] P. Laurent, J. Cleland-Huang, and C. Duan. Towards automatedrequirements triage. In , pages 131–140, Delhi, India, 2007.[16] L. Lehtola, M. Kauppinen, and S. Kujala. Requirements prioritizationchallenges in practice. pages 497–508, Kansai Science City, Japan, 2004.[17] A. Perini, A. Susi, and P. Avesani. A machine learning approach tosoftware requirements prioritization. IEEE Transactions on SoftwareEngineering , 39(4):445–461, 2013.[18] R. Reiter. A theory of diagnosis from first principles. AI Journal ,23(1):57–95, 1987.[19] G. Ruhe and M. Saliu. The art and science of software release planning. IEEE Software , 22(6):47–53, 2005.[20] T. Saaty and L. Vargas. Models, Methods, Concepts & Applications ofthe Analytic Hierarchy Process . Springer, 2000.[21] F. Shao, R. Peng, H. Lai, and B. Wang. DRank: A semi-automatedrequirements prioritization method based on preferences and dependen-cies. The Journal of Systems and Software , 126:141–156, 2017.[22] M. Stettinger, A. Felfernig, G. Leitner, S. Reiterer, and M. Jeran.Counteracting Serial Position Effects in the CHOICLA Group DecisionSupport Environment. In , pages 148–157, Atlanta, Georgia, USA, 2015.[23] Y. Tian, D. Lo, X. Xia, and C. Sun. Automated prediction of bug reportpriority using multi-factor analysis. Empirical Software Engineering ,20(5):1354–1383, 2015.[24] P. Tonella, A. Susi, and F. Palma. Interactive requirements prioritizationusing a genetic algorithm. Information and Software Technology ,55(1):173–187, 2013.[25] E. Tsang. Foundations of Constraint Satisfaction . Academic Press,London, 1993.[26] K. Wiegers. Software Requirements . Microsoft Press, 2003.[27] J. Xuan, H. Jiang, Z. Ren, and Z. Luo. Solving the large scale nextrelease problem with a backbone-based multilevel algorithm. IEEETransactions on Software Engineering , 38(5):1195–1212, 2012.[28] J. Xuan, H. Jiang, Z. Ren, and W. Zou. Developer prioritization in bugrepositories. In , pages 25–35, Z¨urich, Switzerland, 2012.[29] Y. Zhang, M. Harman, and S. Mansouri. The multi-objective next releaseproblem. In