Exploring Factors and Measures to Select Open Source Software
EExploring Factors and Measures to Select Open SourceSoftware
Xiaozhou Li* a , Sergio Moreschini* a , Zheying Zhang a , Davide Taibi a a Tampere University, Tampere (Finland) ∗ the two authors equally contributed to the paper Abstract [Context] Open Source Software (OSS) is nowadays used and integratedin most of the commercial products. However, the selection of OSS projectsfor integration is not a simple process, mainly due to a of lack of clearselection models and lack of information from the OSS portals.[Objective] We investigated the current factors and measures that prac-titioners are currently considering when selecting OSS, the source of infor-mation and portals that can be used to assess the factors, and the possibilityto automatically get this information with APIs. [Method] We elicited thefactors and the measures adopted to assess and compare OSS performinga survey among 23 experienced developers who often integrate OSS in thesoftware they develop. Moreover, we investigated the APIs of the portalsadopted to assess OSS extracting information for the most starred 100Kprojects in GitHub.[Result] We identified a set consisting of 8 main factors and 74 sub-factors, together with 170 related metrics that companies can use to selectOSS to be integrated in their software projects. Unexpectedly, only a smallpart of the factors can be evaluated automatically, and out of 170 metrics,only 40 are available, of which only 22 returned information for all the 100Kprojects.[Conclusion.] OSS selection can be partially automated, by extractingthe information needed for the selection from portal APIs. OSS producerscan benefit from our results by checking if they are providing all the infor-mation commonly required by potential adopters. Developers can benefitfrom our results, using the list of factors we selected as a checklist during the
Email addresses: [email protected] (Xiaozhou Li*), [email protected] (Sergio Moreschini*), [email protected] (ZheyingZhang), [email protected] (Davide Taibi)
Preprint submitted to Information and Software Technology February 22, 2021 a r X i v : . [ c s . S E ] F e b election of OSS, or using the APIs we developed to automatically extractthe data from OSS projects. Keywords:
Open Source, Software Selection, Open Source Adoption
1. Introduction
Open Source Software (OSS) has become mainstream in the softwareindustry, and different OSS projects are now considered as good as closedsource ones [1][2]. However, selecting a new OSS project requires specialattention, and companies are still struggling to understand how to betterselect them [3].One of the main issues during the selection of OSS projects, is the lackof clear information provided by OSS providers, and in particular the lackof automated tools that help the selection [3].A local company hired our research group to ease and standardize theOSS selection process and to automate it as much as possible, to reduce thesubjectivity and the effort needed for the evaluation phase. Currently, thecompany does not prescribe any selection model, and reported us that theirdevelopers commonly struggle to understand what they need to considerwhen comparing OSS projects.In this paper, we investigate the first steps towards the definition of asemi-automated OSS evaluation model. Therefore, we extend our previouswork [3] by conducting a survey investigating the factors commonly consid-ered by the companies when selecting OSS, the source of information thatcan be used to analyze these factors, and the availability of such informationon the portals.The research community has been studying OSS selection and evaluationfrom different perspectives.Researchers developed methods to evaluate, compare, and select OSSprojects. Such methods and tools exploit different types of approaches, in-cluding manual extraction of data from OSS portals (e.g. OMM [4], Opeb-BQR [5], PSS-PAL [6]).Researchers also proposed platforms for mining data from OSS reposi-tories, that can also be used as the sources of information for the evalua-tion and comparison of OSS (e.g., The SourceForge Research Data Archive(SRDA) [7], FLOSSmole [8], FLOSSMetrics [9], tools to provide dump ofexisting OSS portals (e.g. GHArchive [10], GitTorrent [11], ...), and tools toextract information from OSS portals (e.g. PyDriller [12], CVSAnaly [13]...).2oreover, different approaches to evaluate software quality, often appliedto OSS, have been proposed in research, including software metrics (e.g.Chidamber and Kemerer’s metrics suite [14], Cyclomatic Complexity [15]),tools to detect technical debt [16] or to measure other quality aspects (e.g.Software Quality Index [17], Architectural Debt Index [18]).Though the previous work provided a significant amount of results onOSS quality evaluation, OSS development data crawling, and OSS selectionand adoption models, such works are still limited and not easy to apply inindustry for selecting OSS because of various reasons: • OSS selection models – Limited application in industry of the previous OSS selectionmodels [3]. The vast majority of models have never been adoptedmassively by industries with neither case studies nor success sto-ries on the usage of these models therein. One of the potential rea-sons is that it is nearly not possible to have a generally acceptedset of OSS selection criteria to use. The companies must adoptthe criteria for their specific needs and constraints to achieve theirbusiness objectives. Another reason for such limited adoption ofthe selection models can be related to the lack of maturity of themodels. The models lack clarity and guides about which metricswould offer the most relevant insights into the selection criteria. • OSS Mining Platforms such as The SourceForge Research Data Archive(SRDA) [7], GHArchive, GitTorrent [19] – They are designed for research purposes and are complex to useand often have different dependencies for developers that simplyneed to get data for an OSS project. As an example, GitArchive[10] does not allow to directly access the data with a web interfacebut needs to be accessed through Google Big Query. – They are often not maintained in the long term. As an exam-ple, The SourceForge Research Data Archive (SRDA) [7] is notavailable anymore. The GHTorrent [19] was created in 2013 withthe last activity reported in 2019. More information on theseplatforms is available in Table 1. • Tools to evaluate software quality – They are complex to use and often require effort for manual con-figuration and analysis on the target software.3
Most tools require expertise to understand which measures shouldbe used in which context, and how to interpret the evaluation re-sults. They often provide an overload of information, but not al-ways useful for every context. As an example, tools for assessingthe quality and technical debt of software, such as SonarQube ,include more than 500 different rules to validate the source code,but only a limited amount of the rules that are commonly asso-ciated to specific qualities. – When existing tools focus on a specific set of quality metrics todo the evaluation, there is a lack of a tool that can aggregate thefactors commonly considered during the selection of OSS. – The existence of an OSS project community and of a healthecosystem is an informative indicator of the maturity of a soft-ware and its propensity for growth. Even though there are manycommunity-related factors and metrics identified in the OSS se-lection model, some metrics can not be accessed directly from theproject’s repository and existing tools provide very limited sup-port to analyze the data associated with the OSS project com-munity and its support in OSS evaluation. • Existing Software Quality Models and Metrics such as the Architec-tural Quality index [18], but also metrics such as the Cyclomatic com-plexity [15] or the presence of Code Smells [20] or anti-patterns [21]. – Are usually targeting mainly on quality, while companies mightbe interested in other aspects while selecting OSS (e.g. Costs,licenses, features, ...) – Lack of comparison of the magnitude of the observed effect: dif-ferent models return different outputs. As an example, previousworks indicated that high levels of cyclomatic complexity mightresult in less readable source code. While besides that, the pres-ence of some smells can be more harmful than others, but theanalysis did not take into account the magnitude of such ob-served phenomenon. As an example, it is not clear if a pieceof code with a cyclomatic complexity equals to 10 is twice morecomplex to read than the same piece of code with that of 5. Thesame applies to the comparison of several other metrics, includ- SonarQube – Lack of complex and historical analysis. A complete comparisonof an OSS might require not only the analysis of the latest snap-shots of a project, but a historical analysis, thus increasing thecomplexity and the effort required to perform an analysis.In addition, lack of expertise in companies, in particular on software quality,hinders practitioners to select the most suitable quality models for compar-ing the projects.To cope with the aforementioned issues, this paper aims at corroboratingand extending previous empirical research on OSS selection and adoption,so as to enable, not only our target company to assess and compare OSS butalso other companies. More specifically, our contribution aims at extendingour previous work [3] by investigating • the factors that practitioners are currently considering when selectingOSS. • the sources (portals) that can be used to assess the factors • the public APIs that can be accessed to automatically assess the factorsfrom the aforementioned portalsThe remainder of this paper is structured as follows. Section 2 presentsrelated works. Section 3 describes the research method we adopted toachieve our goals. Section 4 reports the results while Section 5 discussthem. Section 6 finally draws conclusions and future works.
2. Related Work
To cope with the need of selecting valuable OSS projects, several eval-uation models have been proposed (e.g. [22], [23], [24] and [25]). At thesame time, different research groups proposed project aggregators to easethe access of different information on OSS, measures and other information.Last, but not least, research in mining software repositories also evolved inparallel, and different researchers provided datasets of OSS projects, portalsand tools to extract information from OSS projects.In the remainder of this Section, we summarize related work on the fac-tors adopted to evaluate OSS, OSS evaluation and selection models, OSSaggregator portals, tools for OSS repository mining and tools for OSS anal-ysis and audit. 5 .1. The factors considered during the adoption of OSS
In our previous systematic literature review on OSS selection and adop-tion models [3], we analyzed 60 empirical studies, including 20 surveys, 5lessons learned on OSS adoption motivation and 35 OSS evaluation models.Regarding the common factors of OSS selection and adoption, eight maincategories were reported by the selected studies, including,
Community andAdoption , Development process , Economic , Functionality , License , Opera-tional software characteristics , Quality , Support and Service . For each cate-gory, sub-factors or metrics are reported. Results show that not all factorswere considered equally important according to evaluation models and tosurveys and lessons learned. For example, factor cost is considered muchmore important by the surveys than by the models when, on the contrary,the importance of factor maturity is seen oppositely. Furthermore, certainfactors are considered important by both groups, such as,
Support and Ser-vice , Code Quality , Reliability , etc.Table 1 lists sources of information mentioned in the related works to as-sess the common factors considered during the adoption of OSS. The tableonly reports the indirect sources of information. Direct sources of infor-mation such as the official portal, or the versioning system (e.g. GitHub,GitLab) are not mentioned in the table..
Within the 35 OSS models identified in our previous literature review [3],21 (60%) were built via case study, with 5 via interview, 5 via experience andthe other 4 via the combination of interview and case study. All proposedmodels provide either checklist (13 models) or measurement (8 models) orboth (14 models) as their working approaches. On the other hand, regard-ing the studies with surveys and lesson learned, the majority (13) targetat adoption motivation identification with the remainders on other scopes.Furthermore, 12 tools are introduced in 22 of the given studies; however,only two out of the 12 are properly maintained.All the models propose to evaluate OSS with a similar approach: • Identification of OSS candidates . In this step, companies need to iden-tify a set of possible candidates based on their needs. • Factors evaluation
A list of factors are then assessed, by extracting theinformation or measures from the OSS portals, or by measuring/run-ning the project candidates 6
Project scoring
The final score is then normalized based on the impor-tance of each factor and the final evaluation is computed.The Open Source Maturity Model (OSMM), was the first model pro-posed [22][23] in the literature. OSMM is an open standard that aims atfacilitating the evaluation and adoption of OSS. The evaluation is based onthe assumption that the overall quality of the software is proportional to itsmaturity. The evaluation is performed in three steps:1. Evaluation of the maturity of each aspect. The considered aspects are:the software product, the documentation, the support and trainingprovided, the integration, the availability of professional services.2. Every aspect is weighted for importance. The default is: 4 for software,2 for the documentation, 1 for the other factors.3. The overall maturity index is computed as the weighted sum of theaspects’ maturity.The OSMM has the advantage of being quite simple. It allows fast (sub-jective) evaluations. However, the simplicity of the approach is also a limit:several potentially interesting characteristics of the products are not consid-ered. For instance, one could be interested in the availability of professionalservices and training, in details of the license, etc. All these factors have tobe ‘squeezed’ into the five aspects defined in the model. However, we doubtthat using ‘maturity’ as unique criteria for the selection of OSS is enough,since factors might also be relevant.The Open Business Readiness Rating (OpenBRR) [26] is an OSS evalu-ation method aiming at providing software professionals with an index ap-plicable to all the current OSS development initiatives, reflecting the pointsof view of large organizations, SMEs, universities, private users, etc. TheOpen BRR is a relevant step forward with respect to the OSMM, since itincludes more indicators, the idea of the target usage, and the possibilityto customize evaluations performed by other, just by providing personal-ized weights. With respect to the latter characteristics, the Open BRR hashowever some limits: one is that for many products it is difficult to choosea “reference application” that reflects the needs of all the users; another isthat there are lots of possible target usages, each with its own requirements;finally, every subjective evaluation performed by a user could be not appli-cable to other users. In any case, the final score is probably a too syntheticindicator to represent the complex set of qualities of a software product. Onthe official Open BRR site several evaluations were available, and originally7rovided as spreadsheet. However the OpenBRR website and tools are notavailable anymore.Qualification and Selection of Open Source Software (QSOS) [25] workssimilarly as OpenBRR, but requires first to create an Identity Card (IC)of each project, reporting general information (name of the product, re-lease date, type of application, description, type of license, project URL,compatible OS, . . . ), then to evaluate the available services, functional andtechnical specifications and grade them (in the 0..2 range). Then, evaluatorscan specify the importance of the criteria and their constraints. Finally anormalized score is computed to compare the selected project candidates.Although in principle the method is effectively applicable to most OSS, theQSOS approach does not represent a relevant step forward with respect toother evaluation methods. Its main contribution is probably the set of char-acteristics explicitly stated which compose the IC, and the provision of aguideline for the consistent evaluation of these characteristics. Nevertheless,in our opinion, the evaluation procedure is too rigid. For instance, it QSOSrequires to define the IC of each OSS under evaluation, even if they are notcompletely matching the requirements. Such a procedure is justified whenthe ICs of products are available from the OS community before a user be-gins the evaluation. However even in this case it may happen that the userneeds to consider aspects not included in the IC: this greatly decreases theutility of ready-to-use ICs. The strict guidelines for the evaluation of theIC, necessary to make other users’ scoring reusable, can be ill suited for aspecific product or user. Finally, even though in the selection criteria it ispossible to classify requirements as needed or optional, there is no properweighting of features with respect to the intended usage of the software.OpenBQR [5] works in a similar way as OpenBRR, but requires theevaluators to first specify the importance of the factors, and then to assessthe projects, so as to avoid to invest time evaluating factors that are notrelevant for the specific context. OpenBQR is an important step forward interms of effort required to evaluate the projects.OSS-PAL [6] works similarly as QSOS, but proposed to introduce a semi-automated evaluation, supported by an online portal. Unfortunately, theportal seems to be only a research prototype, and does not collect any dataautomatically.All the aforementioned models have some drawbacks: • Existing methods usually focus on specific aspects of OSS. • Some methods proceed to evaluate indicators before they are weighted,so some factors may be measured or assessed even if they are later given8 very low weight or even a null one. This results in unnecessary wasteof time and effort. • No OSS evaluation method adequately deals with internal and externalproduct qualities, even though the source code is available. • The dependence of the users of OSS is not adequately assessed, espe-cially the availability of support over time and the cost of proprietarymodules developed by third parties.
Many platforms have been developed to collect and share OSS-relateddata, enabling a quick extraction of the information on different OSS projects.Ohloh was one of the first project aggregators [27][28] on the market(2004) aimed at indexing several projects from different platforms (GitHub,SurceForge, ...). In 2009, Ohloh was acquired by Geeknet, owners of Source-Forge [29] that then sold it to Black Duck Software in 2010. Black Duck, wasalready developing a product for OSS audit, with a particular focus on theanalysis of the license compatibility, and integrated Ohloh’s functionalitywith their products. In 2014, Ohloh became ”Black Duck Open Hub” [30].Finally, Synopsys acquired Black Duck and renamed the Black Duck OpenHub into ”Synopsys Open Hub”. Synopsis Open Hub is currently the onlycontinuously updated OSS aggregator that include information of differentOSS projects from different sources (versioning control systems, issue track-ing systems, vulnerability databases). On January 2021, OpenHub indexednearly 500K projects, and more than 30 billions of lines of code. It providesflexibility for users to select the metrics to compare project statistics, lan-guages, repositories, etc. However, it lacks the OSS evaluation facilities thatallow to adjust the importance of selected metrics according to users’ needsfor automatically scoring the candidate software. In addition, it lacks infor-mation related to the community popularity, documentation, availability ofquestions and answers and other information.Other OSS aggregators have been proposed so far. FlossHub [31] andFLOSSMole [8] had similar goal of OpenHub. However, they have not beenupdated in the last years. FlossMetrics [9] had the goal of providing softwaremetrics on a set of OSS. However, it has also been abandoned in 2010.The Software Heritage [32], differently than the previously mentionedplatforms, has the goal of collecting and preserving the history of softwareprojects, and is not meant to enable the comparison or to provide supportfor selecting OSS. The project is sponsored by different companies and foun-dations, including the UNESCO foundation. The Software Heritage could9e used as a source of information to analyze the activity of a project. How-ever, its access is not immediate, and users need to use APIs to get detaileddata on the projects.Other platforms, designed for supporting mining software activities, mightalso be used for obtaining relevant information from OSS. In particular, theSourceforge research data archives [7] shared the SourceForge.net data withacademic researchers studying the FOSS software phenomenon; GH Archive[10] records the public GitHub timeline and makes it accessible for furtheranalysis; and the GHTorrent project [33] creates a mirror of data offeredthrough the Github REST API to monitor the event timeline, the eventcontents, and the dependencies.
Different researchers proposed tools to mine information from givenrepositories. In particular, BOA [34, 35] provide support to mine sourcecode and development history from project repositories using the domain-specific language. Candoia [36] also provided a platform for mining andsharing information from OSS projects. RepoGrams [37][38] allows to vi-sually compare projects based on the history of the activity of their gitrepositories.Other groups developed tools not aimed at supporting the selection ofOSS, but that can be used as valuable sources of information. As an exam-ple, PyDriller [12] can be used to obtain detailed information from commits.
Companies like Synopsys (formerly BlackDuck) [39] and WhiteSource[40] provide solutions to software composition analysis and offer servicesof the assessment of OSS quality and code security. Synopsis focuses ontheir professional services of the license compatibility while WhiteSourceemphasizes the open source management to offer services such as viewing thestate of OSS components, their license compliance, and the dependencies;prioritizing components’ vulnerabilities based on how the proprietary codeis utilizing them; analyzing the impact of the vulnerabilities, etc.Different tools to assess specific qualities are also available on the mar-ket. As an example, companies can use tools such as SonarQube [41] orSonatype [42] to evaluate different code-related qualities such as the stan-dard compliance or the technical debt. Or Security-specific tools such aWhiteHat Security [43], Kiuwan [44] or others to evaluate the security vul-nerabilities. 10 .6. Gaps of the current OSS assessment models
The different OSS assessment models, tools and platforms provide a pos-sibility to assess OSS projects mainly from the perspectives of license obliga-tion, application security, code quality, etc. They are about the state of soft-ware and its quality and comprise an essential part of the assessment modelfor OSS selection and adoption [45][3]. Besides, activities, supports, or otherprojects surrounding a project form an important perspective demonstrat-ing if a project exists in a lively ecosystem [46]. In particular, metrics suchas response times in Q&A forums and bug trackers, the active contributorsand their satisfaction, the user’s usage and their satisfaction, the numberof downloads, the number of forks, bug-fix time, etc. are informative refer-ences indicating the productivity and a propensity for growth of the OSSproject community. Some of the measures can be cross-referenced from dif-ferent data sources, while some need further analysis based on the collecteddata. To the best of our knowledge, no portal has effectively taken thesecommunity-related factors into account when providing service to evaluateand compare OSS projects.Furthermore, companies have their distinct needs and constraints toadopt OSS projects in software development [3]. They also value the soft-ware selection criteria differently. However, as highlighted in our previoussystematic literature review [3], it is impractical for companies to study everysoftware assessment model to select the one that fits their needs best. There-fore, there is a need to call for the OSS evaluation and selection tools thatnot only guides developers to adapt OSS assessment criteria by identifyingand weighing the ones fitting in a specific scenario, but also automates theprocess of assessing and comparing among a set of selected software basedon information which can be extracted from the public APIs of availableportals.
3. Research Method
Our work is to investigate and determine the factors that practitionersare currently considering when selecting OSS, to identify the sources (por-tals) that can be used to evaluate such factors mentioned by the practition-ers, and to validate the public APIs that can be accessed to automaticallyevaluate those factors from the sources and portals.To achieve the aforementioned goals, we defined four main research ques-tions(RQs). 11 ortal Createdon Lastactivity Information StoredOSS Aggregators [47] Software Heritage 2018 2020 Source Code[30] OpenHub 2004 2020 OSS tracker[31] FlossHub 2008 2018 Research portal[29] SourceForge Research Data 2005 2008 Statistics[10] GH Archive 2012 2020 Timeline Record[11] GH Torrent 2013 2019 GH event monitoring[8] FLOSSMole 2004 2017 Project data[48] PROMISE 2005 2006 Donated SE data[9] FLOSSMetrics 2006 2010 Metrics and Benchmarking
Audit and Analysis Tools [40] WhiteSource 2011 2020 Security[49] FossID 2016 2020 OS Compliance and Security[39] Synopsys (formerly BlackDuck) 2012 2020 legal, security, and quality risks[41] SonarQube 2006 2020 Code quality and security[43] WhiteHat 2001 2020 Software composition analysis[41] SonarCloud 2008 2020 Software quality analysis
OSS Mining Data Tools [50] BOA 2015 2019 Source code mining[36] Candoia 2016 2017 Software repository mining[38] RepoGrams 2016 2020 OSS Comparison
Questions and Answers portal [51] Stack Exchange 2009 2020 Q&A[52] Reddit 2009 2020 Q&A
Table 1: OSS Source of information and portals reported in the literature
RQ1.
What factors are practitioners considering when selecting OSS projectsto be integrated in the software they develop?In this RQ we aim at collecting the information adopted by practi-tioners when selecting projects to be integrated in the software theydevelop. We are not considering OSS product used for productivityor other purposes such as IDEs, Office Suites, but software libraries,frameworks or any other tool that will be integrated and packaged aspart of the product developed by the company.
RQ2
Which measures are used by practitioners to evaluate the factors adoptedduring the selection of OSS?In this RQ we aim at understanding how practitioners are measuringthe factors they are interested to assess. As an example, practition-ers might assess the size of the community checking the number ofcommitter in the repository, or might check the size of the project bychecking the number of commits in the repository or even downloadingthe software and measuring its size in lines of code.
RQ3
Which source of information and portals are used to assess OSS?12n this RQ we aim at understanding which portals or other sources areused by practitioners to evaluate the factors identified in RQ1, basedon the metrics reported in RQ2.
RQ4
Which factor can be extracted automatically from OSS portals?In this RQ, we aim to systematically analyze the common portalshosting OSS, to identify the information that can be extracted viaAPIs.In order to answer our RQs, we conducted our work in three main steps:Step 1: Interviews among experienced software developers and project man-agers to elicit the factors affecting the OSS selection (RQ1), the mea-sures (RQ2) and the sources of information they adopt (RQ3).Step 2: Analysis of the APIs of the source of information (portals) identifiedin RQ2.Step 3: Analysis of the availability of the measures collected in the previousstep (RQ2) in the public API of the sources of information adoptedby practitioners (RQ3) among 100k projects (RQ4).Figure 1 depicts the process adopted in this work. The detailed processis reported in the remainder of this section.
In order to elicit the factors adopted by practitioners when selecting anOSS in the software product development process, we designed and con-ducted a semi-structured interview based on a questionnaire.
The interviews were based on the same questionnaire adopted in ourprevious works to elicit the factors considered important for evaluatingOSS [53][54]. We organized the questions in the questionnaire adopted forthe interviews two sections, according to the types of information we soughtto collect: • Demographic information : In order to define the respondents’ pro-file, we collected demographic background information in relation toOSS. This information considered predominant roles and relative ex-perience. We also collected company information such as applicationdomain, organization’s size via number of employees, and number ofemployees in the respondents’ own team.13 ctivity
Input/ Output
Contribution
SEAA SLR
Models Surveys L.L.
Factors
Preliminary Questionnaire
Questionnaire Validation F2F
Questionnaire
Survey 23 Participants
Updated Factors, Measures & Sources
Identi f ication of measures and proxy to evaluate the features Info available from API
Manual API Eval.
Factors w/ Automated Eval.Factors w/o Automated Eval.
Data Extraction
Step 1 (RQ1 - RQ2 - RQ3)Questionnaire preparation and validation Step 2 (RQ4)
SLR factors and models
Step 3 Legend
Figure 1: The Study Process • Factors considered during the adoption of OSS : Here we askedto list and rank the factors considered during the adoption of OSSsoftware to be integrated in the products they develop, based on theirimportance, on a 0-to-5 scale, where 0 meant “totally irrelevant” and5 meant “fundamental”. – we first asked to list the factors the respondents consider whenadopting an OSS in the software product they develop, and torank the them on the 0-to-5 scale. This open question is toencourage respondents to identify the important factors whichmight not be clarified in our Euromicro/SEAA systematic liter-ature review [3]. – Then, we asked to rank the high-level factors identified in ourEuromicro/SEAA systematic literature review [3]: ∗ Community & Support ∗ Documentation 14
Economic ∗ License ∗ Operational SW Characteristics ∗ Maturity ∗ Quality ∗ Risk ∗ Trustworthiness – Then we asked to list any other high-level relevant factor theyconsider, and to rank them with the same 0-to-5 scale. – For each factor ranked higher or equal than 3, we asked to: ∗ Report and rank the importance of any sub-factor they con-sider ∗ Report the source they commonly use to evaluate them (e.g.GitHub, Jira, manual inspection, ...) ∗ Report the metrics they adopt to measure the factor – We finally asked if they think the factors they reported enable areasoned selection of OSS or if they would still need some pieceof information to have a complete picture of the assessment.The complete questionnaire adopted in the interviews is reported in thereplication package [55].
The interviews were conducted online, using different videoconferencingtools (Zoom, Skype and Microsoft Teams), based on the tool preferred bythe interviewed participant. Interviews were carried out from September2020 to December 2020.Because of time constraint, and of the impossibility to conduct face-to-face interviews during public events, interviewees were selected using aconvenience sampling approach (also known as Haphazard Sampling or Ac-cidental Sampling) [56]. However, we tried to maximize the diversity of theinterviewees, inviting an equal number of developers from large and mediumcompanies, and from companies in different domains. They are experienceddevelopers or project managers, and have been involved in the OSS selectionprocess or the software integration and configuration management process.We did not consider any profiles coming from academia, such as researchersor students, nor any inexperienced or junior profiles.15 .2.3. Interviews Data Analysis
Nominal data on the factor importance was analyzed by determiningthe proportion of responses in each category. Ordinal data, such as 10-pointLikert scales, was not converted into numerical equivalents since using aconversion from ordinal to numerical data entails the risk that any subse-quent analysis will yield misleading results if equidistant values cannot beguaranteed. Moreover, analyzing each value of the scale allowed us to betteridentify the potential distribution of the answers.Open questions (application domain, other factors reported, platformsadopted to extract the information and metrics adopted to evaluate thefactors) were analyzed via open and selective coding [57]. The answerswere interpreted by extracting concrete sets of similar answers and groupingthem based on their perceived similarity. Two authors manually provideda hierarchical set of codes from all the transcribed answers, applying theopen coding methodology [57]. The authors discussed and resolved codingdiscrepancies and then applied the axial coding methodology [57].
We manually analyzed the APIs of the portals identified in RQ3, lookingfor APIs that allowed to assess the information needed to measure the factorsreported by the interviewees (RQ1 and RQ2). The first two authors inde-pendently analyzed all the portals seeking for these pieces of information,and then compared the results obtained. In case of discrepancies, all theincongruities were discussed by all the authors, reaching a 100% consensus.Some factors were not directly analyzable. Therefore, the first two au-thors proposed a list of proxy measures, considering both the measuresadopted by the interviewees and measures available in the literature. Then,all measures were discussed by all the authors until we reach a consensus.However, as expected, not all the measures can be automatically ex-tracted, and some of them require a manual assessment. An example of afactor that cannot be automatically extracted is the availability of completeand updated architectural documentation.
This step was based on three sub-steps: • Project selection . We selected the top 100K GitHub projects, basedon the number of stars. The number of projects was limited to the16ime available. In particular, the different APIs limit the number ofqueries that can be executed in one hour, and therefore we limited thestudy to 100k most starred projects to ensure that the data can beextracted in 2 months. • Information extraction . We extracted the selected information fromthe APIs. We decided to extract only the information needed to eval-uate the factors. Other information are available, but requires to runa higher number of queries, and therefore would have reduced thenumber of projects that we can extract. As an example, it would bepossible to extract all the details on project issues (issue title, author,date, comments, ...), but this would have required to run a numberof additional queries, without providing any information consideredvaluable by our interviewees. • Analysis of the information available . In this step we analyzed whichinformation is actually available for each project. As an example, notall the projects might use different issue trackers instead of using theone provided by GitHub, or some projects might not be listed by theNIST NVD database, or more, some project might not have questionsand answers available in StackOverflow or Reddit.
In order to ease the replication of this work we provide the completereplication package including the questionnaire adopted for the interviews,the results obtained in the interviews, the data crawling script and the resultsof the data analysis [55].
4. Results
Here we first provide information about the sample of respondents, whichcan be used to better interpret the results and then, we show the collectedresults with a concise analysis of the responses obtained, with insights gainedby statistical analysis.We collected 23 interviews from experienced practitioners. Figure 2 con-tains the distribution of company sizes where our interviewees belong, whileFigure 3 shows the percentage for organizational roles identified in the ques-tionnaire. 17ompany Size( > (a) (b) Distribution of company size Figure 2: Distribution of company sizes of our interviewees
Roles (a) (b) The roles of our interviewees
Figure 3: Figure caption
Our interviewees consider 8 main factors and 46 sub-factors when theyselect OSS, reporting an average of 2.35 factors per interviewee, a minimumof 1 and a maximum of 21 factors.The factor that is mentioned more frequently from the interviewees is
License which has received a median importance of 4 out of 5. Surprisingly,this is not the value which has received the highest median value of impor-tance as
Community Support and Adoption, Performances and
PerceivedRisk received a median value of importance of 4.5 out of 5. It is interest-ing to note that no participant mentioned economic-related factors such aslicense costs, or cost for training.In Table 2, we report the list of factors and sub-factors together withthe number of participants who mentioned them (column RQ1-
When we asked practitioners to report the measures they use to evaluatethe factors they mentioned in RQ1, and to rank their usefulness, practition-ers mentioned 110 different measures.18
Q1 RQ2
Factor
Table 2: High-level Factors considered during the adoption of OSS ≥
3, where 0 means ”Thismeasure is useless to evaluate this factor” and 5 means that the measure isextremely useful).Surprisingly, the factor where practitioners provided the highest numberof measures to assess it, is
Maturity , which has been mentioned only 6 timescompared to the
License , mentioned 21 times, where practitioners provided7 measures instead. This indicates a wide variety of interpretations on
Ma-turity , and practitioners use different measures to evaluate this factor. Thecareful reader can also observe that for some factors considered as relevantin RQ1 such as
Perceived risks , no measures have been mentioned. Thisresult proves that in some cases, some of the most important factors in anOSS can not be objectively measured and the interviewees do not know howto retrieve such information appropriately.
Our interviewees mentioned 9 different source of information and portalsthey commonly consider when they select OSS.In Table 3 we list the sources of information adopted by the practitionersto evaluate OSS, together with the number of participants who mentioned it(columns version control systems , issuetracking systems , Question and Answer portals (Q&A), forum and blogs and security related platforms.The 5 most reported sources of information are GitHub (and GitHub Is-sue tracker), StackOverflow, Reddit, and NIST Security Vulnerability (NVD)with respectively: 23, 19, 12, 12 and 14 mentions. All of these are mentionedfrom more than 20% of the interviewees and are therefore those which proveto be the most useful when retrieving information related to OSS. Otherplatforms such as Bitbucket or Jira were rarely mentioned.
We focused our attention to the four most used sources of informationreported in RQ3: GitHub [58], StackOverflow [59], Reddit [52] and the NISTNational Vulnerability Database [60]. For such purpose, we first identifiedthe APIs that can be adopted to extract the information needed to measurethe factors, and then we extracted the data from the 100k with more starsin GitHub. 20 D Source of Information % of mentions
Version Control Systems:R GitHub [58] 23 100R GitLab [59] 1 4.3R SourceForge [60] 1 4.3R Bitbucket [61] 1 4.3Issue Tracking Systems:I GitHub Issues [58] 19 82.6I Jira [62] 1 4.3Question and Answer Portals:Q StackOverflow [63] 12 51.2Q Reddit [52] 12 51.2Forum and Blogs:F Medium [64] 5 5F Hackernews [65] 5 5Security:S CVE [66] 1 4.3S CVSS [67] 2 8.7S CWE [68] 1 4.3S NVD [69] 14 21.7
Table 3: The source of information reported by the interviewees (RQ3) ID Source of Information % of mentions
Version Control Systems:R GitHub [58] 23 100R GitLab [59] 1 4.3R SourceForge [60] 1 4.3R Bitbucket [61] 1 4.3Issue Tracking Systems:I GitHub Issues [58] 19 82.6I Jira [62] 1 4.3Question and Answer Portals:Q StackOverflow [63] 12 51.2Q Reddit [52] 12 51.2Forum and Blogs:F Medium [64] 5 5F Hackernews [65] 5 5Security:S CVE [66] 1 4.3S CVSS [67] 2 8.7S CWE [68] 1 4.3S NVD [69] 14 21.7
Table 4: The source of information reported by the interviewees (RQ3)
We focused our attention to the four most used sources of informationreported in RQ3: GitHub [58], StackOverflow [63], Reddit [52] and the NISTNational Vulnerability Database [69]. For such purpose, we first identifiedthe APIs that can be adopted to extract the information needed to measure21
Table 3: The source of information reported by the interviewees (RQ3)
The extraction of the information for 100K projects took a total of 5days for GitHub, 53 days for StackOverflow, 4 days for Reddit and 2 Daysfor the NIST National Vulnerability Database. The long processing time isdue to the limit of queries that can be performed on the APIs for differentIP addresses.Considering the projects extracted (Figure 5a), more than half of theseprojects (53.3%) have been active for 2 - 6 years. Around 12.1% of theseprojects are active for one year or less when only 3.9% of them are activefor more than 10 years. Majority of these projects (87.0%) have less than500 issues during their life cycle when around 3.4% of them have morethan 1k issues (Figure 5b). Furthermore, there are 1791 projects beingvery popular having more than 10k stars when 25.9% projects having starsranging from 1k to 10k (Figure 5c) with 46.7% having less than 500 stars.On the other hand, regarding project size, more than half of them (52.8%)have lines of code (LOC) ranging from 20k to 500k. 6.1% projects containmore than 5m LOC when only 0.9% of them have less than 1k LOC (Figure5d). Regarding developing languages, Javascript is the most popular beingthe primary language of OSS projects (17k projects) with Python and Javaat 2nd and 3rd (Figure 5e). They are the primary languages for 40.9% of theprojects. However, regarding LOC by languages, C language (57.6b) andJavascript (57.2b) rank at the top. Both have almost doubled the amount21 a) OSS Project Stats by Reddit Post Number(b) OSS Project Stats by StackOverflow Ques-tions
Figure 4: Stats for Top 100k Github Projects in Reddit and StackOverflow of C++ language (31.7b) which ranks the 3rd in terms of total LOC (Figure5f). Regarding release numbers, 58.1% projects do not have any specificrelease recorded. 37.6% have less than 50 releases when only the rest 4.3%have more than 50 releases. All the previously mentioned data is alwaysavailable from GitHub, and queries to the GitHub APIs will always returna valid information,Regarding the adopted open source licenses, 23.7% projects did not spec-ify the licenses they adopt. As for the other projects, MIT, Apache 2.0 andGNU GPL v3.0 are the most popular licenses with 53.2% projects adoptingone of them (Figure 5h). Therein, 13.6% projects adopted non-mainstreamlicense (identified as ’Other’).We also validate the APIs of Reddit and StackOverflow by finding theamount of discussion threads on each of the 100k OSS projects. As shownin Figure 4, 14.5% projects are generally discussed in Reddit with only 5.8%projects having more than 100 posts (Figure 4a). On the other hand, 13.0%projects are discussed (raised technical questions) in StackOverflow (Figure4b). Therein, 3.6% projects have more than 100 questions raised on. Ingeneral, such results show that it is hard to find sufficient generic or technicaldiscussion regarding specific OS projects from Reddit and StackOverflow.Based on the interview results, especially the obtained factors that areconsidered important by the practitioners (shown in Table 2, we furthervalidate whether such factors can be analyzed via the automatically ob-tained data from the previously mentioned portals with the results shownin Table ?? .According to the investigation on automatic OSS data extraction on theTop 100k OSS projects on Github via the APIs of the five public portals,we find that the majority of the measures towards the Community Support a) OSS Project Stats by Age (Years) (b) OSS Project Stats by Issues(c) OSS Project Stats by Stars (d) OSS Project Stats by LOC(e) OSS Project Stats by Top 20 Primary Lan-guages (f) Stats for Top 20 Language by LOC(g) OSS Project Stats by Releases (h) OSS Project Stats by Licenses Figure 5: General Stats for Top 100k Github Projects nd Adoption factor can be automatically done via data extractions fromsuch APIs. To be noted, regarding Communication , the question and an-swer sources (i.e., StackOverflow) contain information on limited number ofOSS projects, which limits the availability towards the evaluation on suchcategory. In addition, we also find, despite the availability of automatic dataextraction and measuring via APIs, many of the measures require furthercalculation and learning, as well as multiple queries to obtain. For example,in order to measure the
Number of Independent Developers , we must getthe list of contributors of a particular project via multiple queries first (maxitems per page for Github API is 100), and check the “Independentness” ofeach contributor via further investigating his/her organization status. Thus,such process shall be, to some extent, time-consuming, when the limit rateof the API usage shall be also taken into account.Another category can be automatically measured is
License , as shownin the data, 76.6% of the projects contain specific License information. Onthe other hand, for the other categories, automatic data extraction and full-grained evaluation is hindered by the limited availability of the accordingdata, as only very limited percentage of the measures can be automaticallydone via data extraction (shown in Table 2). And amongst these categories,the evaluation of
Maturity category depends on the availability of the re-lease data from repository dataset, when for the obtained 100k projects only42.2% provide such information. Measuring the availability of
Documenta-tion shall also depend on the information extracted from project descriptionand homepage, while only around 40% of projects provide those.As for the security vulnerability evaluation, the NVD Dataset provide in-formation for 12838 projects (12.8%). However, it is not clear if the projectsnot reported in NVD do not contain security vulnerabilities at all, or sim-ply are not indexed by the NVD dataset. However, it is important to notethat the NVD performs analysis on CVEs published to the CVE Dictionary.Every CVE has a CVE ID which is used by cybersecurity product/servicevendors and researchers for identifying vulnerabilities and for cross-linkingwith other repositories that also use the CVE ID. However, it is possiblethat some security vulnerabilities are not publicly reported with assignedCVE IDs.
5. Discussion
In this Section, we discuss the results of our RQs and we present thethreats to validity of our work. 24he factors and the measures adopted to evaluate and select OSS (RQ1 -RQ2) evolved over time. While in 2015 [54][3] factors such as
Customizationeasiness and
Ethic were the most important, nowadays we cannot statethe same. Already in 2020 [3] such factors have been incorporated insideother more valuable factors such as
Quality , while today are not mentionedanymore. On the other side
License and
Documentation , which are themost mentioned factors nowadays were side factors in 2020. As a matter offact the latter was a sub-section of
Development Process , while both werenot even considered in 2015. Moreover, ethical principles, that were veryrelevant in 2015, are not even mentioned in 2020.Nowadays the trend is to search for OSSs which are ready to be inte-grated as is. In order to incorporate OSSs without falling into lawsuit par-ticular attention needs to be put into the
License type , while to guaranteethe correct functioning the focus needs to be put in the documentation. Aclear example is the necessity of a clear definition of the system requirements when incorporating libraries.Another factor which gained a lot of importance over time is
Security .While in 2015 was a factor with medium relevance, nowadays it is a keypointfor measuring quality of an OSS. The growing number of portals dedicatedto ensure absence of vulnerabilities proves that people are concerned of theuse of OSSs when embedded in their system. In particular they stronglyrely on such portals to check the history of the OSSs to incorporate andin some cases also to ensure that proper reports are delivered when a newvulnerability is discovered.The source of information adopted to evaluate OSS (RQ3) did not changedcompletely from the previous years. Users are still adopting project reposi-tories and issue trackers as main source of information. Moreover, an effectof the newly introduced factor security, is that now the selection also re-quire security related information, that are commonly fetched from securitydatabases such as the NIST NVD [60], CVE [61], and CWE [62]. In ad-dition, many vendors like Synopsys[39] and WhiteSource [40] offer softwarecomposition analysis solutions that facilitate licence risk management, vul-nerability identification and management, risk reporting, etc.Unexpectedly, even nowadays, not all the portals can provide completeinformation for evaluating the measures needed by the practitioners (RQ4).The analysis of the 100K most starred projects in GitHub showed that onlythe information coming from the project repositories (e.g GitHub) are al-ways available, except for the license information (76.6%). Considering otherfactors such as the communication, the situation does not improve, and onlyin 14.5% of projects we were able to automatically extract the relevant in-25ormation from their APIs. Due to the diversity of application domains,organizational needs, and constraints, practitioners in different organiza-tions may explain the factors from their own perspective and may adoptdifferent measures in the OSS evaluation. A good example is the measuresassociated with the factor of
Maturity . Measures such as the number ofreleases and the system growth in the roadmap were commonly concernedin the evaluation of software maturity; besides, some practitioners also tookthe number of commits and the number of forks as important measures inthe evaluation. The total number of commits itself might not be enoughwhen evaluating software maturity. The prevalence of commits over timeand the types of commits could be additional and useful information in theevaluation. Therefore, how the measures help with the evaluation of thefactors could have been clarified.The results of this work show a discrepancy between the informationrequired by the practitioners to evaluate OSS and the information actuallyavailable on the portals, confirming that the collection of the factors re-quired to evaluate an OSS project is very time-consuming, mainly becausemost of the information required is not commonly available on the OSSproject [63][3][64].The automation of the data extraction, using portal APIs might helppractitioners reducing the collection time and the subjectivity. The resultof this work could be highly beneficial for OSS producers, since they couldcheck if they are providing all the information commonly required by whois evaluating their products, and maximize the likelihood of being selected.The result can also be useful to potential OSS adopters, who will speed-upthe collection of the information needed for the evaluation of the product.Even in case OSS producers do not enhance their portals by providingthe information required by the practitioners to assess OSS, the results ofthis work could be useful for practitioners that need to evaluate an OSSproduct. The list of factors can be effectively used as checklist to verify if allthe potentially important characteristics of OSS have been duly evaluated.For instance, a practitioner could have forgotten to evaluate the trend ofthe community activity and he/she could adopt an OSS product that has a“dissolving” community: this could create problems in the future becauseof the lack of maintenance and updates. The usage of checklist would allowpractitioners to double check if they considered all factors, thus reducingthe potential unexpected issues that could come up after the adoption.26 .1. Future Research Directions.
As a result of our findings, we propose the following directions for futureresearch in this area.Focus on the definition of a common tool to automatically extract infor-mation needed for the evaluation of OSS, investigating proxy measures incase direct measures are not available.Definition of refined and customizable models (which may be obtainedby merging multiple available approaches) and favor its adoption throughrigorous and extensive validation in industrial settings. This could increasethe validity of the model and thus its dissemination in industry, where OSSis still not widely adopted. Several models already exist but, according tothe results of our previous literature review [3], they have not been stronglyvalidated and, as a consequence, adoption has been limited.Try to target the models at quality factors that are of real interest forstakeholders. Most of the available models focus on the overall quality of theproduct, but few of them are able to adequately assess each single factor thatcomposes the overall quality of the OSS product. This can complicate theassessment of OSS products by stakeholders, who are interested in specificquality factors: e.g., developers are likely more interested in reliability ortestability aspects, while business people may be more interested in cost ormaintenance factors, etc..In the studies, we identified 170 metrics to measure the factors for OSSevaluation and selection, based on which we shall conduct an in-depth anal-ysis to gain a better understanding of the rationale for the metrics. Therationale explains why a measure helps gain insights into the factors, andthe assumption or other information useful in evaluating an OSS. It helpsto identify the needed data to extract from the available portals and to au-tomate the OSS analysis and assessment process. With the explanation ofwhy the measures are needed, practitioners can also better understand thefactors and their assessment, which further eases the process to adopt theOSS selection models and tools in the software development practice.Develop tools that support the research directions listed above (i.e., toolsable to support and simplify the applicability of the proposed models duringthe evaluation of OSS products).Disseminate the information that should be provided on OSS portals, soas to enable OSS producers to consider them as part of their marketing andcommunication strategies [65]. 27 .2. Threats to Validity
We applied the structure suggested by Yin [66] to report threats to thevalidity of this study and measures for mitigating them. We report internalvalidity, external validity, construct validity, and reliability.
Internal Validity . One limitation that is always a part of survey re-search is that surveys can only reveal the perceptions of the respondentswhich might not fully represent reality. However, our analysis was per-formed by means of semi-structured interviews, which gave the interviewersthe possibility to request additional information regarding unclear or im-precise statements by the respondents. The responses were analyzed andquality-checked by a team of four researchers.
External Validity . Overall, a total of 23 practitioners were inter-viewed. We considered only experienced respondents and did not acceptany interviewees with an academic background. However, we are aware thatthe convenience sampling approach we adopted could be biased, even if wetried to maximize the diversity, As for the projects we selected to validatethe presence of information in OSS portals, we are aware that the 100K moststarred GitHub projects might not represent the whole OSS ecosystem, butwe believe they might be a good representative of them. We also think thatless popular projects, might only perform worst than the selected ones.We therefore think that threats to external validity are reasonable. How-ever, additional responses and additional projects should be analyzed in thefuture.
Construct Validity . The interview guidelines were developed on thebasis of the previously performed surveys [53][54]. Therefore, The questionsare aligned with standard terminology and cover the most relevant charac-teristics and metrics. In addition, the survey was conducted in interviews,which allowed both the interviewees and the interviewer to ask questions ifsomething was unclear.
Reliability . The survey design, its execution, and the analysis followeda strict protocol, which allows replication of the survey. However, the openquestions were analyzed qualitatively, which is always subjective to someextent, but the resulting codes were documented.
6. Conclusion
In this paper, we investigated the factors considered by companies whenselecting OSS to be integrated in the software they develop, and we analyzedtheir availability in the OSS portals, in particular using OSS portal APIs.28e identified a set 8 factors and 74 sub-factors, together with 170 metricsthat companies commonly use to evaluate and select OSS. Unexpectedly,only a small part of the factors can be evaluated automatically, and out of170 measures, only 40 are available from project portals APIs.The automated extraction of the information from the 100K most starredGitHub projects showed that only 22 measures out of 40 returned informa-tion for all the 100K projects. 2 measures returned information for around80% of the projects while another 7 for around 40%. The other 4 measuresreturned information for below 15%.It is important to note that the extraction consider some of the mostfamous OSS projects. Therefore, we can speculate that the vast majority ofless common and less used projects might provide even less information.The result of this work enable us to create a list of updated factorsand measures, together with the list of automatically collectable ones, thatpractitioners can use to select OSS.Results can be used also by researchers to further validate the factorsand measures, or providing frameworks or tools to ease the selection of OSS.Future work include the validation of the factors and measures in in-dustrial settings, reducing the subjectivity of the decisions. Moreover, weare planning to develop a tool and portal to automatically collect the infor-mation and enable the comparison of OSS projects, so as to ease the OSSselection phase.
CRediT author statementXiaozhou Li:
Software, Formal Analysis, Investigation, Writing - Orig-inal Draft, Visualization.
Sergio Moreschini:
Software, Formal Analysis,Investigation, Writing - Original Draft, Visualization.
Zheying Zhang:
Conceptualization, Investigation, Writing - Review and Editing.
DavideTaibi:
Conceptualization, Methodology, Funding Acquisition, Review andEditing, Supervision. 29 eferences [1] G. Robles, I. Steinmacher, P. Adams, C. Treude, Twenty years of opensource software: From skepticism to mainstream, IEEE Software 36(2019) 12–15.[2] T. Kilamo, V. Lenarduzzi, T. Ahoniemi, A. Jaaksi, J. Rahikkala,T. Mikkonen, How the cathedral embraced the bazaar, and the bazaarbecame a cathedral, in: Open Source Systems, Springer InternationalPublishing, Cham, 2020, pp. 141–147.[3] V. Lenarduzzi, D. Taibi, D. Tosi, L. Lavazza, S. Morasca, Open sourcesoftware evaluation, selection, and adoption: a systematic literaturereview, in: 2020 46th Euromicro Conference on Software Engineeringand Advanced Applications (SEAA), pp. 437–444.[4] E. Petrinja, R. Nambakam, A. Sillitti, Introducing the opensource ma-turity model, in: Proceedings of the 2009 ICSE Workshop on Emerg-ing Trends in Free/Libre/Open Source Software Research and Devel-opment, FLOSS ’09, IEEE Computer Society, USA, 2009, p. 37–41.[5] D. Taibi, L. Lavazza, S. Morasca, Openbqr: a framework for the assess-ment of oss, in: Open Source Development, Adoption and Innovation,Springer US, Boston, MA, 2007, pp. 173–186.[6] A. I. Wasserman, X. Guo, B. McMillian, K. Qian, M.-Y. Wei, Q. Xu,Osspal: Finding and evaluating open source software, in: F. Balaguer,R. Di Cosmo, A. Garrido, F. Kon, G. Robles, S. Zacchiroli (Eds.),Open Source Systems: Towards Robust Practices, Springer Interna-tional Publishing, Cham, 2017, pp. 193–203.[7] G. Madey, The sourceforge research data archive (srda), http://zerlot.cse. nd. edu/ (2008).[8] Flossmole, https://flossmole.org , Accessed: 2020-12-30.[9] Free/libre open source software metrics., , Accessed: 2020-12-30.[10] Gh archive, , Accessed: 2020-12-30.[11] Gh torrent portal, https://ghtorrent.org , Accessed: 2020-12-30.3012] D. Spadini, M. Aniche, A. Bacchelli, Pydriller: Python framework formining software repositories, in: Proceedings of the 2018 26th ACMJoint Meeting on European Software Engineering Conference and Sym-posium on the Foundations of Software Engineering, pp. 908–911.[13] G. Robles, S. Koch, J. M. GonZ ´AlEZ-BARAHonA, J. Carlos, Remoteanalysis and measurement of libre software systems by means of thecvsanaly tool, in: Proceedings of the 2nd ICSE Workshop on RemoteAnalysis and Measurement of Software Systems (RAMSS), IET, pp.51–56.[14] S. R. Chidamber, C. F. Kemerer, A metrics suite for object orienteddesign, IEEE Transactions on Software Engineering 20 (1994) 476–493.[15] T. J. McCabe, A complexity measure, IEEE Trans. Softw. Eng. 2(1976) 308–320.[16] P. Avgeriou, D. Taibi, A. Ampatzoglou, F. Arcelli Fontana, T. Besker,A. Chatzigeorgiou, V. Lenarduzzi, A. Martini, N. Moschou, I. Pigazzini,N. Saarim¨aki, D. D. Sas, S. S. de Toledo, A. A. Tsintzira, An overviewand comparison of technical debt measurement tools, IEEE Software(2020).[17] M. Dixon, Methods and systems for generating software quality index, https://patents.google.com/patent/WO2009089294A3 , 2016.[18] R. Roveda, F. Arcelli Fontana, I. Pigazzini, M. Zanoni, Towards anarchitectural debt index, in: 2018 44th Euromicro Conference on Soft-ware Engineering and Advanced Applications (SEAA), pp. 408–416.[19] G. Gousios, The ghtorrent dataset and tool suite, in: Proceedings ofthe 10th Working Conference on Mining Software Repositories, MSR’13, IEEE Press, Piscataway, NJ, USA, 2013, pp. 233–236.[20] M. Fowler, Refactoring: Improving the Design of Existing Code,Addison-Wesley, Boston, MA, USA, 1999.[21] W. J. Brown, R. C. Malveau, H. W. McCormick, T. J. Mowbray, An-tiPatterns: Refactoring Software, Architectures, and Projects in Crisis,John Wiley and Sons, New York, 1998.[22] F. Duijnhouwer, C. Widdows, Open source maturity model, CapgeminiExpert Letter (2003). 3123] B. Golden, Open source maturity model, The Open Source BusinessResource (2008) 4–9. Copyright - Copyright Talent First Network May2008; Document feature - Tables; Last updated - 2020-11-18; Subject-sTermNotLitGenreText - United States–US.[24] D. Taibi, L. Lavazza, S. Morasca, Openbqr: a framework for the as-sessment of oss, in: J. Feller, B. Fitzgerald, W. Scacchi, A. Sillitti(Eds.), Open Source Development, Adoption and Innovation, SpringerUS, Boston, MA, 2007, pp. 173–186.[25] R. Semeteys, Method for qualification and selection of open sourcesoftware, Open Source Business Resource (2008).[26] A. I. Wasserman, M. Pal, C. Chan, The business readiness rating: aframework for evaluating open source technical report (2006).[27] M. Bruntink, An initial quality analysis of the ohloh software evolutiondata, Electron. Commun. Eur. Assoc. Softw. Sci. Technol. 65 (2014).[28] J. Allen, S. Collison, R. Luckey, Ohloh web site api, 2009.[29] Sourge forge research data, , Accessed: 2020-12-30.[30] Openhub, , Accessed: 2020-12-30.[31] Flosshub, https://flosshub.org , Accessed: 2020-12-30.[32] R. Di Cosmo, S. Zacchiroli, Software Heritage: Why and How to Pre-serve Software Source Code, in: iPRES 2017 - 14th International Con-ference on Digital Preservation, Kyoto, Japan, pp. 1–10.[33] G. Gousios, The ghtorrent dataset and tool suite, in: 2013 10th Work-ing Conference on Mining Software Repositories (MSR), pp. 233–236.[34] R. Dyer, H. A. Nguyen, H. Rajan, T. N. Nguyen, Boa: A language andinfrastructure for analyzing ultra-large-scale software repositories, in:2013 35th International Conference on Software Engineering (ICSE),pp. 422–431.[35] R. Dyer, H. A. Nguyen, H. Rajan, T. N. Nguyen, Boa: Ultra-large-scalesoftware repository and source-code mining, ACM Trans. Softw. Eng.Methodol. 25 (2015).[36] Candoia, http://candoia.github.io , Accessed: 2020-12-30.3237] D. Rozenberg, I. Beschastnikh, F. Kosmale, V. Poser, H. Becker,M. Palyart, G. C. Murphy, Comparing Repositories Visually with Re-pograms, in: Proceedings of the 13th International Conference on Min-ing Software Repositories (MSR), p. 109–120.[38] Repograms, https://github.com/RepoGrams/RepoGrams , Accessed:2020-12-30.[39] Synopsys - eda tools, semiconductor ip and application security solu-tions, , Accessed: 2020-12-30.[40] Whitesource, , Accessed:2020-12-30.[41] Sonarcloud, https://sonarcloud.io , Accessed: 2020-12-30.[42] Sonartype, , Accessed: 2020-12-30.[43] Whitehat security—application security platform, ,Accessed: 2020-12-30.[44] Kiuwan - end-to-end application security, ,Accessed: 2020-12-30.[45] N. Sbai, V. Lenarduzzi, D. Taibi, S. B. Sassi, H. H. B. Ghezala, Ex-ploring information from oss repositories and platforms to support ossselection decisions, Information and Software Technology 104 (2018)104 – 108.[46] S. Jansen, Measuring the health of open source software ecosystems:Beyond the scope of project health, Information and Software Technol-ogy 56 (2014) 1508 – 1519. Special issue on Software Ecosystems.[47] Software heritage, , Accessed:2020-12-30.[48] Promise, http://promise.site.uottawa.ca/SERepository/ , Ac-cessed: 2020-12-30.[49] Fossid, https://fossid.com , Accessed: 2020-12-30.[50] Boa, http://boa.cs.iastate.edu/boa// , Accessed: 2020-12-30.[51] Stack exchange, https://stackexchange.com , Accessed: 2020-12-30.3352] Reddit, http://reddit.com , Accessed: 2020-12-30.[53] V. Del Bianco, L. Lavazza, S. Morasca, D. Taibi, Quality of opensource software: The qualipso trustworthiness model, in: Open SourceEcosystems: Diverse Communities Interacting, Springer Berlin Heidel-berg, Berlin, Heidelberg, 2009, pp. 199–212.[54] D. Taibi, An empirical investigation on the motivations for the adoptionof open source software.[55] Replication package, https://github.com/clowee/Exploring-Factors-and-Measures-to-Select-Open-Source-Software ,Accessed: 2021-01-07.[56] M. Battaglia, Convenience sampling. encyclopedia of survey researchmethods, 2008.[57] B. Wuetherick, Basics of qualitative research: Techniques and proce-dures for developing grounded theory, Canadian Journal of UniversityContinuing Education 36 (2010).[58] Github, , Accessed: 2020-12-30.[59] Stack overflow, https://stackoverflow.com , Accessed: 2020-12-30.[60] National vulnerability database (nvd), https://nvd.nist.gov/general , Accessed: 2020-12-30.[61] Common vulnerabilities and exposures (cve), https://ossindex.sonatype.org/doc/cve , Accessed: 2020-12-30.[62] Common weakness enumeration (cwe), https://cwe.mitre.org , Ac-cessed: 2020-12-30.[63] Y. Kamei, T. Matsumoto, K. Yamashita, N. Ubayashi, T. Iwasaki,T. Shuichi, Studying the Cost and Effectiveness of OSS Quality As-sessment Models: An Experience Report of Fujitsu QNET, IEICETransactions on Information and Systems (2018) 2744–2753.[64] V. Del Bianco, L. Lavazza, S. Morasca, D. Taibi, D. Tosi, The qualispoapproach to oss product quality evaluation, in: Proceedings of the3rd International Workshop on Emerging Trends in Free/Libre/OpenSource Software Research and Development, FLOSS2010, New York,NY, USA, p. 23–28. 3465] V. Del Bianco, L. Lavazza, V. Lenarduzzi, S. Morasca, D. Taibi, D. Tosi,A study on oss marketing and communication strategies, in: OpenSource Systems: Long-Term Sustainability, Springer Berlin Heidelberg,Berlin, Heidelberg, 2012, pp. 338–343.[66] R. Yin, Case Study Research: Design and Methods, 4th Edition (Ap-plied Social Research Methods, Vol. 5), SAGE Publications, Inc, 4thedition, 2009. 35 ppendix A: Results from the interviews