OSS PESTO: An Open Source Software Project Evaluation and Selection TOol
OOSS PESTO: An Open Source Software ProjectEvaluation and Selection TOol
Xiaozhou Li and Sergio Moreschini
Tampere UniversityKalevantie 4, 33100, Tampere, Finland { xiaozhou.li, sergio.moreschini } @tuni.fi Abstract.
Open source software (OSS), playing an increasingly criti-cal role nowadays, has been commonly adopted and integrated in var-ious software products. For many practitioners, selecting and adoptingsuitable OSS can help them greatly. Though many studies have beenconducted on proposing OSS evaluation and selection models, a limitednumber are followed and used in the industry. Meanwhile, many existingOSS evaluation tools, though providing valuable details, fall short on of-fering intuitive suggestions in terms of framework-supported evaluationfactors. Towards filling the gap, we propose an Open Source SoftwareProject Evaluation and Selection TOol (OSS PESTO). Targeting OSSon Github, the largest OSS source code host, it facilitates the evalua-tion practice by enabling practitioners to compare candidates therein interms of selected OSS evaluation models. It also allows in-time Githubdata collection and customized evaluation that enriches its effectivenessand ease of use.
Keywords:
Open source Software · Open Source Evaluation · GithubMining
During the last two decades, open source software (OSS) has been flourishingwith such trend continuing [10] and nowadays, OSS is adopted by the vast ma-jority of IT companies [13][12]. On Github, the largest OSS source code host,more than 60 million users have participated in over 100 million open sourceprojects , among which many have been widely adopted by users and companies.However, due to such a large number of candidates, for many practitioners, se-lecting a suitable OSS product or library is difficult, especially when the relevantinformation are not explicitly provided [12].To support the OSS evaluation and selection practice, many studies providemodels and frameworks as guidance [7,18,21,24,23]. During the last two decades,35 models are proposed with checklists, measures or both provided [12]. Theyall work similarly with a process of “candidate software identification - factor https://github.com/search?q=type:user&type=Users https://github.com/search?q=is:public a r X i v : . [ c s . S E ] F e b X. Li and S. Moreschini evaluation - scoring”. In addition, many tools are designed and proposed tofacilitate such practice [18,21,4,23]. However, many tools are rigidly designedand allow limited customization. Meanwhile, among those proposed, only twoare properly maintained with the majority being not available any more [12].As the largest OSS host and platform, Github is a valuable channel restor-ing and presenting OSS related information. Retrospective analysis of Githubrepositories, based on the abundant information on code, developers, organiza-tions and activities within, can yield valuable insights into the evolution andgrowth of OSS and facilitate decision making processes [17]. Many studies haveused such data source and conduct research on various OSS related perspectives[9,15,22]. However, the approaches towards using Github data for OSS evaluationare limited, let alone tools to support such practice.Herein, we propose OSS PESTO, an open source tool facilitating OSS eval-uation and selection. Comparatively, besides being fully open sourced and freeto use, OSS PESTO has the following advantages: 1) it allows users to updatein-time Github data; 2) it allows users to customize evaluation with models orpreference; 3) it allows users to save data locally and to use it without networkconnection; 4) it is always accessible as maintained in Github repository. It shalllargely help the practitioners to compare and evaluate OSS candidates freely,timely and efficiently.The remainder of this paper is organized as follows. Section 2 introducesthe related work on OSS evaluation tools. Section 3 presents OSS PESTO withdetails. Section 4 presents an experiment validating its applicability. Section 5concludes the article.
The Open Source Maturity Model (OSMM) is the first proposed model and openstandard that aims for such purpose [7]. Guided by OSMM, the practitioners willevaluate OSS by its maturity of each aspect, weight each aspect with importance,and compute its overall maturity by the weighted sum. Compared to OSMM,the Open Business Readiness Rating (OpenBRR) is an OSS evaluation methodwith more indicators, the idea of target uses and the customized evaluation [24].The method provides an index applicable to all OSS development initiatives.Its main limits are related to the incompatibility of the requirements betweendifferent targets and to the difficulty of choosing the proper reference applica-tion for some projects. Similarly, a number of evaluation models are proposed,for example, Qualification and Selection of Open Source Software (QSOS) [18],OpenBQR [21], OSSPAL [23], which provide enhanced guidance and method-ological support.QSOS tool is designed to support the QSOS model which aims to qualify,select and compare OSS products [18]. However, the rigidness of compulsoryIdentity Card setting and all criteria inclusion is commonly seen as its limitation.OpenQBR [21] requires specification on factor importance before the assessment
SS PESTO 3 of the project. Compared to the QSOS tool, OpenQBR is more elastics as notrequire to evaluate factors which are not relevant to the specific project.OSS-PAL [23], though similar to QSOS, aims to partially automate the evalu-ation of the projects. Despite the appealing goal of the project, it fails to providethe automated data collection function. Other works investigated the availabilityof the information on online portals [20][14], but they did not provide tools forcollecting or aggregating data.In addition, many other tools are available over time but have been dis-continued, including real-time OpenSSL execution monitoring system (ROSEN)[4], RAP TOOL [8], SQO-OSS [19], OMM Tool [5], T-Doc Tool [16], QualiPSoTrustworthiness Checklist [3], MOSST [6] and OP2A Checklist [1] and otherchecklist included in marketing models for OSS [11][2].With OSS PESTO, we aim to overcome some of the most common drawbacksof all of these tools, such as, the focus on specific factors, the evaluation offactors before adding a weight function or the lack of control for both internaland external product quality.
We implement OSS PESTO by following the commonly acknowledged OSSevaluation process summarized from previous studies [12]. It shall contain thefollowing main activities: 1) identify the OSS candidates; 2) elicit a list of factorsthat need to be evaluated and the according metrics that measure such factors;3) provide scores or selection recommendation as evaluation output.In addition, in order to use the latest Github data to evaluate OSS, we in-tegrate a data crawler module in OSS PESTO. It enables the users to crawlthe required repository and activities information of any existing OSS projects.Additionally, it also allows them to crawl the data of a list of projects basedon the selected range of stars. Furthermore, OSS PESTO allows users to cus-tomize evaluation factors based on the selection of models and/or their personalpreferences. Fig. 1.
OSS PESTO Framework
Shown in Fig. 1, OSS PESTO contains three individual modules as follows. Source code: https://github.com/clowee/OSS-PESTO X. Li and S. Moreschini – Data Crawler : The data crawler module contains a set of Python scriptsthat extract Github repository data via Github APIs . It enables the usersto select the candidate OSS and extract the according data. – Server : The server side is implemented by ReactJS while database withMongoDB . The evalaution model is described with the config.json file, whichcan be altered with users’ preference of evaluation factors. – Client : The client side is also implemented by ReactJS. It mainly displaysthe candidate OSS projects with the selected attributes/factors shown. Italso shows the results of candidate comparison which facilitates OSS evalu-ation and selection.Fig. 1 also shows the activities of utilizing OSS PESTO to evaluate candidateOSS projects as follows. – Step 1 . identifies the OSS candidates by running the data crawler moduleto extract the according dataset. – Step 2 . select the evaluation model, configure evaluation preference, andrun the server module. – Step 3 . run the client module and compare the OSS candidates by theselected factors.The crawled data is saved locally in A comma-separated values (CSV) filewith each row containing the values of an individual OSS candidate. To be noted,the required data can be selectively crawled according to the users, who deter-mine which metrics are the important ones when evaluating particular aspectsof OSS. Such selection of data can be guided by the evaluation model chosenby the evaluator. For example, when selecting only the most popular OSS, thenumbers of stars, watches, and download are the ones to be crawled. Fig. 2.
An example of Configuration File
Furthermore, the configuration file is a Javascript file mapping the categorytabs displayed by the client and the data features/metrics that are selected toevaluate the according categories. Shown in 2 is an example of how a configura-tion file works. By editing the configuration file, the users can customize their https://docs.github.com/en/graphql; https://docs.github.com/en/rest https://reactjs.org/ selection of metrics, the evaluation categories and the links in between. For ex-ample, if the user chooses to focus on the popularity of OSS and uses the numberof watches as the metric for it, the according piece of code { Header: “ } shall be added to the “Popularity” tab block. In order to validate the applicability of OSS PESTO, we conduct a series of ex-periments, including the testing of all three modules. The testing scenario is toevaluate and compare three JavaScript frameworks, i.e., Angular , Redux andVue using the OSSPAL model [23]. The evaluation categories include “Com-munity”, “Support”, “Operational Software Characteristics”, “Documentation”,“Software Technology Attributes”, “Functionality” and “Development Process”.Herein, we focus on the “Community”, “Support” and “Software Technology At-tributes” aspects, which can be well demonstrated by the obtained data.To start crawling the Github data, given the user’s Github personal tokenand the target OSS candidates as input, the data crawler module can be ranindividually and continuously. Towards the stated objective, the crawling processtakes within two minutes. When the data is ready, we prepare the config.jsonby selecting the target metrics that are valuable towards evaluating each factorsof the candidates. For each of the selected factors, the according metrics are asfollows. – Community: number of watches, number of stars, age, average issue activetime, average issue comments, number of pull requests, and number of issueraiser. – Support: average issue closed time, number of contributor, organization issueraiser. – Software Technology Attributes: number of open issues, number of depen-dence.Thereafter, when running both the server and the client, the comparisonresult is shown in Fig. 3.Based on such comparison, we can easily observe that despite not being theoldest community, Vue is more popular than the other two candidates in terms ofwatches and stars. However, these three communities are active in different ways,as Angular has more comments on issues, pull requests, and different issue raiserswhile the others are more responsive to issues (shown in Fig.3 (a)). Regardingsupport, Angular has a much larger contributor group and organizational issueraiser for support, while on software technology attribute aspect, Redux hasmuch less dependence and open issues (shown in Fig.3 (b) and (c)).When adopting a different evaluation model, it is possible that by taking intoaccount particular overseen metrics, the user obtains new insights regarding the https://angular.io/ https://redux.js.org/ https://vuejs.org/ X. Li and S. Moreschini Fig. 3.
Experiment Results Demonstration selected candidates. For example, the SQO-OSS model [19] sees “Growth inactive developers” as a metric to evaluate the “Developer base” category, whenOSSPAL has not such category. However, due to the fact that same dataset isused for all potential models, it is hardly possible to have opposite comparativeevaluation result for the same category from different models.
This paper presents OSS PESTO, an open source software project evaluation andselection tool, to support the practitioners’ need towards OSS evaluation and se-lection. This tool provides a Github-repository-data-oriented, easy-to-maintain,customization-friendly solution. It shall benefit the practitioners in both indus-try and academia in terms of the different focuses on either the OSS projects orthe OSS evaluation models respectively.However, the current version of this tool can certainly be improved in thefollowing ways. Firstly, OSS PESTO has not yet supported the practitioners’selection of OSS candidates at the identification phase in terms of their targetfunctionalities. As the accessible data obtained from Github does not provideexplicit information regarding the main features of the OSS, such candidateselection cannot be automated via direct identification. A potential solution is toapply natural language processing (NLP) techniques to identify and summarizesuch main features from the description and Readme text of the projects. Sucha feature shall be implemented in our future work.Furthermore, the current version only utilizes limited amount of the at-tributes provided by the Github API. For many such attributes, the explicitmappings towards particular OSS evaluation categories are not verified. For ex-ample, the number of OSS downloads can be seen as a metric for its popularity.However, unless a particular user insists it being a critical evaluation criterion forhis/her customized evaluation model, such value can be ignored when it does notcontribute to any pre-defined evaluation categories. Nonetheless, the inclusionof more data features shall be taken into account in the future work. However,it should be noted such work can result in the exhaustion of Github API querylimit, as some values (e.g., issues) can only be obtained via looping enumeratedresults.
SS PESTO 7
In addition, more features, in terms of the ease of use perspective of thetool, shall be also considered. For example, a graphic user interface is neededfor the data crawler module which can also be integrated to the server side.Furthermore, the data from Github has its limitation on reflecting certain as-pects of OSS. For example, the development process of the projects cannot beeasily accessed externally, except for the number of releases and the release pace.Thus, in order to improve the potential scope of this tool, more data sources arerequired with more techniques required to process possibly unstructured dataas well. Meanwhile, more practical features, such as, exporting the evaluationresults, adding weight to different factors, editor of configure files, and modelcustomization interface, are also required.Our future work shall focus on integrating the modules and enhancing theoverall quality of the tool according to the above mentioned limitation. It is alsoimportant to investigate the ways of evaluating individual OSS by providingunified quantified results. In addition, we shall systematically investigate theavailability of data from multiple sources that could be used to support OSSevaluation.
References
1. Benlian, A., Hess, T.: Comparing the relative importance of evaluation criteriain proprietary and open-source enterprise application software selection–a conjointstudy of erp and office systems. Information Systems Journal (6), 503–525 (2011)2. del Bianco, V., Lavazza, L., Lenarduzzi, V., Morasca, S., Taibi, D., Tosi, D.: Astudy on oss marketing and communication strategies. In: Open Source Systems:Long-Term Sustainability. pp. 338–343. Springer Berlin Heidelberg, Berlin, Heidel-berg (2012)3. del Bianco, V., Lavazza, L., Morasca, S., Taibi, D., Tosi, D.: The qualispo ap-proach to oss product quality evaluation. In: Proceedings of the 3rd InternationalWorkshop on Emerging Trends in Free/Libre/Open Source Software Research andDevelopment. pp. 23–28 (2010)4. Choi, S.j., Kang, Y.h., Lee, G.s.: A security evaluation and testing methodologyfor open source software embedded information security system. In: InternationalConference on Computational Science and Its Applications. pp. 215–224. Springer(2005)5. Del Bianco, V., Lavazza, L., Morasca, S., Taibi, D.: Quality of open source software:the qualipso trustworthiness model. In: IFIP International Conference on OpenSource Systems. pp. 199–212. Springer (2009)6. Del Bianco, V., Lavazza, L., Morasca, S., Taibi, D., Tosi, D.: A survey on theimportance of some economic factors in the adoption of open source software. In:Software Engineering Research, Management and Applications 2010, pp. 151–162.Springer (2010)7. Duijnhouwer, F.W., Widdows, C.: Capgemini expert letter open source maturitymodel. Capgemini pp. 1–18 (2003)8. Immonen, A., Palviainen, M.: Trustworthiness evaluation and testing of opensource components. In: Seventh International Conference on Quality Software(QSIC 2007). pp. 316–321. IEEE (2007) X. Li and S. Moreschini9. Kalliamvakou, E., Gousios, G., Blincoe, K., Singer, L., German, D.M., Damian,D.: The promises and perils of mining github. In: Proceedings of the 11th workingconference on mining software repositories. pp. 92–101 (2014)10. Kilamo, T., Lenarduzzi, V., Ahoniemi, T., Jaaksi, A., Rahikkala, J., Mikkonen, T.:How the cathedral embraced the bazaar, and the bazaar became a cathedral. In:Ivanov, V., Kruglov, A., Masyagin, S., Sillitti, A., Succi, G. (eds.) Open SourceSystems. pp. 141–147. Springer International Publishing, Cham (2020)11. Lenarduzzi, V.: Towards a marketing strategy for open source software. In: Pro-ceedings of the 12th International Conference on Product Focused Software Devel-opment and Process Improvement. p. 31–33. Profes ’11, Association for ComputingMachinery, New York, NY, USA (2011). https://doi.org/10.1145/2181101.218110912. Lenarduzzi, V., Taibi, D., Tosi, D., Lavazza, L., Morasca, S.: Open source soft-ware evaluation, selection, and adoption: a systematic literature review. In: 202046th Euromicro Conference on Software Engineering and Advanced Applications(SEAA). pp. 437–444 (2020). https://doi.org/10.1109/SEAA51224.2020.0007613. Lenarduzzi, V., Tosi, D., Lavazza, L., Morasca, S.: Why do developers adopt opensource software? past, present and future. In: Open Source Systems. pp. 104–115.Springer International Publishing, Cham (2019)14. Li, X., Moreschini, S., Zhang, Z., Taibi, D.: Exploring factors and measures toselect open source software. In: Arxiv (2021)15. Lima, A., Rossi, L., Musolesi, M.: Coding together at scale: Github as a collabora-tive social network. In: Proceedings of the International AAAI Conference on Weband Social Media. vol. 8 (2014)16. Morasca, S., Taibi, D., Tosi, D.: T-doc: A tool for the automatic generation oftesting documentation for oss products. In: IFIP International Conference on OpenSource Systems. pp. 200–213. Springer (2010)17. Munaiah, N., Kroh, S., Cabrey, C., Nagappan, M.: Curating github for engineeredsoftware projects. Empirical Software Engineering (6), 3219–3253 (2017)18. Origin, A.: Method for qualification and selection of open source software (qsos). (Accessed: 2021-01-22)19. Samoladas, I., Gousios, G., Spinellis, D., Stamelos, I.: The sqo-oss quality model:measurement based open source software evaluation. In: IFIP international con-ference on open source systems. pp. 237–248. Springer (2008)20. Sbai, N., Lenarduzzi, V., Taibi, D., Sassi, S.B., Ghezala, H.H.B.: Ex-ploring information from oss repositories and platforms to support ossselection decisions. Information and Software Technology , 104–108 (2018). https://doi.org/https://doi.org/10.1016/j.infsof.2018.07.009,