Sustainable Research Software Hand-Over
PP r e p r i n t Sustainable Research Software Hand-Over
J. Fehr ∗ C. Himpe † S. Rave ‡ J. Saak § Abstract
Scientific software projects evolve rapidly in their initial development phase, yetat the end of a funding period, the completion of a research project, thesis, orpublication, further engagement in the project may slow down or cease com-pletely. To retain the invested effort for the sciences, this software needs to bepreserved or handed over to a succeeding developer or team, such as the nextgeneration of (PhD) students.Comparable guides provide top-down recommendations for project leads.This paper intends to be a bottom-up approach for sustainable hand-over pro-cesses from a developer’s perspective. An important characteristic in this regardis the project’s size, by which this guideline is structured. Furthermore, check-lists are provided, which can serve as a practical guide for implementing theproposed measures.
Research software, software artifacts as research products, or computer-basedexperiments are drivers of modern science. Yet, while computerization has mas-sively accelerated science, the intangible and volatile nature of software has alsoinhibited scientific progress: Once-developed-software is often not usable in sub-sequent development of algorithms, for example, due to technical incompatibil-ities, insufficient documentation, or plain unavailability. Even though advancesin supplying source codes together with published results are achieved [22], thereusability of such scientific codes remains unsatisfactory [17], and of limitedreach when tied to a publication. So, instead of building on top of “shouldersof giants”, the “wheel is reinvented” regularly in many branches of sciences andnot least in computational mathematics. A frequently occurring symptom of ∗ Institute of Engineering and Computational Mechanics, University of Stuttgart, Pfaffen-waldring 9, 70569 Stuttgart, GermanyORCID: 0000-0003-2850-1440, [email protected] † Computational Methods in Systems and Control Theory, Max Planck Institute for Dy-namics of Complex Technical Systems, Sandtorstr. 1, 39106 Magdeburg, GermanyORCID: 0000-0003-2194-6754, [email protected] ‡ Institute for Computational and Applied Mathematics, University of Münster, Einstein-strasse 62, 48149 Münster, GermanyORCID: 0000-0003-0439-7212, [email protected] § Computational Methods in Systems and Control Theory, Max Planck Institute for Dy-namics of Complex Technical Systems, Sandtorstr. 1, 39106 Magdeburg, GermanyORCID: 0000-0001-5567-9637, [email protected] a r X i v : . [ c s . G L ] S e p r e p r i n t this deficiency is the inadequate treatment of software developed for, or overthe course of a PhD thesis, which may be disregarded either by the original orsubsequent developing PhD candidate.As scientists, scientific organizations, and funding agencies are becomingmore aware of these issues, guidelines and best practices for good scientific soft-ware conduct are in demand. Examples for such academically driven effortsare the guides published by the alliance of German research associations [18],the DFG “guidelines for safeguarding good scientific practice” [7], the DLR(Deutsche Zentrum für Luft- und Raumfahrt) guideline [24], or the software sus-tainability institute guideline [16]. These guides present top-down approachesaimed at principal investigators, decision-makers and coordinators. Our con-tribution, on the other hand, intends to be a bottom-up approach presentingrequirements and recommendations for academic software developers, such asundergraduate students, PhD students, postdoctoral researchers, or researchsoftware engineers. Furthermore, instead of focusing on the development pro-cess of scientific software, as in [12, 13, 15, 9] and references therein, we focuson the continuation of a project, when the developer (or a maintainer) leaves,e.g. after completing her PhD project.We note that industry has already adapted robust collaborative softwaredevelopment practices, see for example [10]. Yet, given that developers of sci-entific codes may have no formal training in software engineering, and scientificsoftware development processes can differ, in academia only certain ideas canbe transferred to support researchers or departments.While the issues addressed in this work apply to all branches of science, weemphasize that mathematical software projects hold particular responsibilities.An example are the numerical libraries BLAS [21] and
LAPACK [1], which consti-tute the basis for numerical computations in many sciences. Hence, authors ofthis foundational layer in scientific software stacks need to take into account thecontinued use and possibly further development outside the field of mathematics.Best practices for mathematical software [6] and numerical software [20, 3] arelong known (yet still not established), and properties such as reliability, robust-ness or transportability [5], the numerical experiment attributes replicability,reproducibility and reusability [9], code as a form of scientific notation [11], aswell as basic guidelines for research software [23] have been discussed in the liter-ature, yet, sustainable hand-over strategies for (mathematical) research softwareprojects have not been documented to the best knowledge of the authors.The core of this work aims at the hand-over of general scientific softwareprojects, illustrated in Fig. 1, which is discussed in detail in the followingsections. We consider two classes of research software projects: First, smallprojects, see Section 2.1, which are implemented by a single developer, for ex-ample over the course of a PhD program or a funding period; Second, largeprojects, see Section 2.2, which have multiple developers. Since these two projectcategories serve different purposes, the proposed requirements and recommen-dations differ. Minimal requirements, as well as optional recommendations, aregiven for both project categories. Finally in Section 3, a brief conclusion is givenalongside two checklists, which summarizes the proposed measures for a practi-cal hand-over process, followed by a brief comment on minimal documentationof numerical software in the Appendix.2 r e p r i n t Figure 1: Project hand-over illustrative summary.
In the following, we lay out minimal and optional measures for a sustainableproject hand-over distinguished by the size of the project. From our experi-ence, we recommend the distinction of software projects into the two categories“small” and “large”. A more fine grained categorization is surmisable too, seee.g. [14], still, we think that two categories are sufficient in covering the essentialaspects of sustainable software hand-over, with the rationale that more straight-forward guidelines may have a higher chance of general acceptance compared tomore complicated rule sets.As a general remark: When a project is handed over, a time period frombefore the previous developer leaves, till after the next developer enters theproject is considered the hand-over time, which should be allocated in a mannerto suitably prepare the hand-over, and allow for a training phase. To this end,it can be worth the extra cost of having the previous and next developer(s)overlap for some time, depending on the project size and complexity. We alsonote that if a project is not continued in direct succession, it can be conserved,see for example [25], for information on archiving.
We consider a small project to be code developed and maintained by a singleauthor, which means, for example, a project written from scratch, or a fork ofan existing project that throughout the development is not merged back into theparent project. This is often the case for tools developed as part of a publication,thesis or with a tight focus. Such projects have their developer as the sole user,or at least a limited user basis.Following, we will lay out minimal requirements, which ensure the project’ssustainability, as well as optional recommendations that facilitate long-termusability, such as, when a new student takes over, after a previous studentfinishes her work, or if an abandoned project is revived.3 r e p r i n t The most important requirement for continuation or atleast conservation is the availability of the project contents — utilized specifichardware components may need to be kept available physically, if no virtual-ization is possible — including the source code, configuration and data files.Therefore, the project location should be discoverable, i.e.: not solely on thedeveloper’s personal computer hard-drive, but rather in a central repository ofthe associated institute at a known and accessible storage location.
Code ownership
If the code is available, the next important question is:Who owns the code? Potential owners could be the associated institute or uni-versity, the superior or supervisor of the developer, or the original developerherself. Additionally, if there is third-party funding involved, the funding entitymay have regulations about the funded project’s ownership. Besides owner-ship, third-party rights need to be considered, originating from prior developers,third-party projects, or parts thereof included in the project. These ownershipquestion can be resolved by documentation of stakeholders alongside the codeand with a license statement, which can be as easy as the project’s developerself-licensing her work or following the respective guidelines applicable to them.For further information on software licensing see [26].
Execution environment
Given all legal prerequisites are resolved, a mini-mal description of the required runtime environment, such as operating system,dependencies, and compiler or interpreter is needed, together with a short de-scription on how to compile, if necessary, and run the project. A tested uponoperating system needs to be stated (with compute architecture and endianess ifapplicable). We also recommend listing all depending software libraries, tools ortoolboxes, which are not part of the default installation of the compatible oper-ating systems. Furthermore, all components of the required software stack needto be given with a version number. We caution that even in case of high-levelcross-platform runtime environments, certain behavior may depend voluntarily,accidentally, or due to restrictions, on the underlying operating system (for aminimal report, in this case, see the Appendix). In view of increasingly complexscientific computing software stacks (Fig 2), providing a reproducible executionenvironment (see below) is highly recommended.
Working example
An essential requirement for a small project hand-over, issample code (In [9] such a file is suggested to be named
RUNME .), which can runand demonstrate the core feature(s) of the project. Such an example is essential,to test if the code is executable and also serves as a starting point to understandthe structure of the code, since the workflow can be traced for a known workingexample, e.g. by a debugging program. Moreover, the results can be used toverify that future changes do not (unintentionally) affect computational results.To these ends, the execution of such an example code should sufficiently coverthe complete functionality of the software project.4 r e p r i n t Operating System B L A S L A P A C K C U D A myold library M P I P D E S o l v e r G U I V i s u a li z a t i o n M o d e l R e d u c t i o n O p t i m i z a t i o n Figure 2: Software stack dependencies: “Tower of Doom”.
Minimal documentation
Typically the information of the previous require-ments is gathered in a
README file (
README is a widely used file name for a plaintext file, holding a minimal documentation; see: [9]). Further information thatshould be included in the
README is:• Is the code functioning, and if, on what hardware (see Appendix)?• Is the available project state current (latest use in a thesis or publication)?• New algorithms from which publications are implemented by this project?• Existing algorithms from which publications are utilized by this project?• What publications use this project?• What are the known limitations or issues?Referencing all associated publications helps to put a small research softwareproject in the appropriate scientific context, and has also educational functionfor the subsequent developer(s).
As the availability of the project is crucial, for the documen-tation of the scientific findings, the best measure is a public release under an,ideally, open license on a stable service [8]. If legal or other reasons prevent suchline of action, the reasons should be stated near the top of the aforementioned
README file, so this important information is not lost in transition.5 r e p r i n t Version control
We strongly recommend to use a version control softwareto track the changes during the development of the project in a repository.Besides documenting the history of a project, modern version control systemsallow to tag (mark) states of the repository. This is useful for associating ex-periments, for example in publications, during the development process. Hence,all experiments can refer to a specific revision of the source code, in order toensure replicability and reproducibility, in particular for future developers. Atthe very least a version control repository serves as a (very sophisticated) backup method. An introduction to generic version control workflows can be foundin [27].
Basic code cleanup
Furthermore, some software development anti-patterns [4]are more common (in our experience) in small projects, and impede projectcontinuation by another than the original developer. First, undocumented con-stants used in the source code hinder the interpretation in the absence of theoriginal developer. Second, comments containing code, so called dead code, in-troduce the uncertainty which code has been used for what experiments, andif the commented out code is still needed or not. Third, the use of hard-codedfile paths may prevent the project from functioning in a different environment,such as another developer’s computer. All these issues can, if not fixable, beeasily resolved by a few additional source code comments.
Reproducible execution environment
In addition to the minimally re-quired documentation, we recommend to report if the project was tested inother compute environments than the developer’s. To ensure long term com-patibility and conservation, it is relevant if the project can run on a simulatedcomputer, i.e. a virtual machine. This allows conserving an image file, treated asa hard drive by such a virtual machine, containing the complete software stack(including the operating system). Thus, the image file completely defines thesoftware aspect of the compute environment, and the virtual machine softwarepresents an abstraction from the hardware.As an alternative to a virtual machine image, a step-by-step guide can beincluded, which explains the preparation, i.e. correct sequence of installationof dependencies, starting from the base installation of a compatible operatingsystem. Such a guide can be easily distributed with the software, whereas,due to their size, virtual machine images often need to be archived separately.Moreover, the guide can serve as a starting point for installing the software inother execution environments.
Integration into larger project
A possible path for small projects is the in-clusion into a larger project, which, for example, provides a collection of topicallyrelated functionality. Such a large project mitigates some of the aforementionedproblems due to development guidelines. To be included into such a super-project, it is essential for the small project to be modular as well as compatiblewith the including project’s principle design, interfaces, style and contributionguidelines, as well as possibly a build and test systems. Furthermore, planned orunsuccessful directions of development should be included into the documenta-tion, to support the future (third-party) development of the incorporated smallproject. 6 r e p r i n t Figure 3: Project hand-over illustrative summary for a larger project.Practically, there are three paths to include a smaller project into an overar-ching project: First, the continuous development, for example, as a feature of theinfrastructure of the large project. This approach naturally requires adherenceto project guidelines, yet often entails slower progress due to this overhead. Sec-ond, after completion, requesting inclusion of the finished “small project”. Whilequick progress can be made this way during development, integration may behard due to independent design. Lastly, a fork of the super-project with subse-quent independent development and a final merge, allows efficient developmentwithout giving up the frame of the super-project.
We define a large project as a software package that is developed by mul-tiple authors, possibly located at different institutions. An example setting isa project consortium developing a joint tool driven by their research that alsoshould be made available, e.g. to their peers. While the developing researchersmay be a significant subgroup of the software’s users, in this case the communitycan be far larger and the users might even be unrelated to this community.In our experience it is advisable that large projects have a hierarchy ofcontributors, see Fig. 3, which follows de-facto standards. Unprivileged usersserve as reporter s, who file feature requests or bug reports (These can jointly becalled issues.). Contributors that work on closing bugs or contributing featuresare called developer s. They have limited, or no write access to the maindevelopment line of the software. The maintainers have extended permissionson the repository and oversee the progress of the software project. They alsomerge the contributions of the developers into the main development line. Whilereporters and developers may change frequently, maintainers ensure consistencyof the development, at most superseded by a rights holding entity, depicted inFig. 3 as a roof of the project. 7 r e p r i n t In the following sections, we propose hand-over guidelines for large projects,subdivided into bare minimum requirements and optional, but desired, recom-mendations. While for developers the guidelines for small projects (Sec. 2.1)apply to their branches (a branch is a copy of the development resources underversion control which can evolve in separate, but is still part of the overall sourcecode repository.), the presentation, here, focuses on maintainers.
The chosen project license is important, even crucial forpublicly available projects. While for a small project only few entities are eligibleto act as the rights holder, for large projects the situation can be, and often is,more complex. This, in turn, leads to additional difficulties that need furtherattention: Project funding can end after a certain period, and maintainers maychange their employers or even fields of interest. Thus, to ensure continuedavailability of the project, the developers need to come to a formal agreement,i.e. a software license, under which terms the project should be available. Foran open-source license hierarchy, see [28].
Code ownership of contributions
Compared to small projects, the ques-tion of contributed code ownership is more relevant for large projects. In par-ticular, developers need to consider that a later change of license requires theconsent of all copyright holders, which may have long left academia. Therefore,if a license change shall remain feasible, all code contributors could transfer theircopyright to a single entity, for example, a society or association as copyrightholder. It should also be noted, that there are important differences in copyrightlaws over the world and obtaining proper legal advice is desirable.
Access to project resources
Similarly important as legal rights are theaccess permissions in the software repository and further project resources, suchas servers, websites, domain names or mailing lists. As a minimal requirement,there should always be at least two persons with administrator access to allproject resources. In case of a smaller development team with only one activemaintainer, it is sufficient if these rights are held by a second person who isassociated with the project but is not an active developer (like a research groupleader). This measure prevents a project depending on the health and goodwillof a single individual.
Management of development branches
Modern version control systemspermit ways to continue developing a version of the software independentlyfrom a given state of the main development stream, e.g., for development ofnew features. These are called branch es, and it is good practice to use onebranch per user, or issue. Each branch has to be documented with respect toits purpose and status; furthermore, it should be clear which developers areresponsible for the branch. If the withdrawal of a developer from the projectleads to an unmaintained branch, the branch should either be merged into themain development branch, a new developer for the branch should be found, orin case either is not feasible, a detailed description of the open and completedtasks should be added to the documentation to allow continuation after a stalephase. 8 r e p r i n t Stable main branch
To ensure that a leaving maintainer cannot cause anunknown or unusable state of the project, it is essential to make sure thatthe main branch of the software can be (if applicable compiled and) executedby more than a single person (the main developer) and runs on all targetedplatforms at any time during the development process. This also means thatthe installation is flexible enough to at least specify user-specific paths duringthe build process.
All measures that improve the overall quality of thecode and its maintainability are also beneficial in a hand-over process as theyfacilitate the familiarization of a new developer with the project. More impor-tantly, after the withdrawal of a developer, old code that has been written by thisdeveloper will be much easier to understand if standard software developmentbest practices are followed. In particular, we mention usage of continuous inte-gration (CI). In software engineering, continuous integration is the practice ofmerging all developers’ working copies into the main development line regularly.This is often followed by a test-phase to ensure that none of the recent changesbreak other functionality (see also [2]). An optional add-on, which is especiallyrelevant for scientific computing software, is the more recent technique of con-tinuous benchmarking that additionally tries to ensure optimal performance ofthe implementation at all times. Furthermore, if applicable, we recommend theusage of build systems that automatically resolve dependencies, especially toother projects, during the compilation process.
Changelog
As soon as a software is developed and used by more than oneperson, keeping track of important changes in the software compared to earlierversions becomes consequential. While the history of version control systemsallows inspecting every change of the software, this information is usually toofine grained to for the “big picture”. Therefore, the most relevant changes shouldbe documented in a
CHANGELOG file [19] or the release notes. This document notonly informs users about new features, the removal of faulty code or changesin the interfaces, but also helps developers of other software projects relyingon the function interfaces, to keep track of changes and necessary updates totheir own projects. More importantly in the scope of a project hand-over it ishelpful for the new maintainer to comprehend changes and note dependenciesas well as compatibilities, especially if legacy versions of a project need to bemaintained, e.g. due to hardware restrictions, in parallel to the evolution in themain development branch.
Code of conduct
A document defining rules for the introduction and retire-ment of project maintainers as well as handling project administration questionscan have an essential role in project hand-over. In particular, when a maintainerno longer actively works on the project but is hesitant to step down, a code ofconduct document can prevent an entailing gridlock in the project.9 r e p r i n t Small Software Project Handover (cid:4)
Minimal Requirements (cid:3)
Code availability Where are source code, data and configurationfiles? (cid:3)
Code ownership Who owns the software and who holds rights? (cid:3)
Execution environment What hardware and software stack is required? (cid:3)
Working example How are the features of the code producingwhat results? (cid:3)
Minimal documentation What does a new developer need to know at theleast? (cid:4)
Optional Recommendations (cid:3)
Public release Is a public open-source release possible? (cid:3)
Version control Are revision of the software automaticallytracked? Where? (cid:3)
Basic code cleanup Are constants, dead code and hard paths re-moved? (cid:3)
Reproducible execution environment Is a (virtual) machine back up available? (cid:3)
Integration into larger project Is inclusion into a larger project possible orplanned?
Table 1: Checklist for sustainable research software hand-over of small projects.
Contribution policy
Besides the legal status of contributions discussed above,a contribution policy defines the practical requirements for the contributed code.Typical requirements regard the general workflow of the project. For example,requirements state whether single or multiple pull/merge requests, with whatlevel of documentation and tests, are expected. The code should be merge-able with the main development branch. Also, (passing) tests for all includedfeatures can be expected in the project’s favored test suite. The licensing andcopyright of the contributed code as well as the form of attribution of the contri-bution should be clear. Oftentimes also restrictions on the code’s general layoutand naming schemes are prescribed, in order to improve readability and thusaccessibility of the implemented ideas.As discussed above, a case of project hand-over is the inclusion of a smallerinto larger project. Such a policy can simplify this process, in particular, if theserequirements are known during the development of the small project.10 r e p r i n t Large Software Project Handover (cid:4)
Minimal Requirements (cid:3)
Software license Has a suitable (and compatible) software li-cense been chosen? (cid:3)
Code ownership of contributions Who owns which parts of the code? (cid:3)
Access to project resources Are full permissions to all project resourcesgranted to at least two persons? (cid:3)
Management of development branches Are there unmaintained developmentbranches? (cid:3)
Stable main branch How is stability of the main branch ensured? (cid:4)
Optional Recommendations (cid:3)
Code maintainability Is continuous integration / testing / bench-marking utilized? (cid:3)
Changelog Are the core changes of the releases trackedin a changelog or release notes? (cid:3)
Code of conduct What are the central points of the code ofconduct and why? (cid:3)
Contribution policy How are contribution policies communicated?
Table 2: Checklist for sustainable research software hand-over of large projects.
In this work we presented measures for the sustainable hand-over of researchsoftware, by differentiating between small and large software projects and propos-ing minimal requirements and optional recommendation for both categories.With this, we aim to spark a discussion in the sciences on sustainability of re-search software development and appreciate feedback. Furthermore, we hopethat this document, and especially the checklists in Table 1 and Table 2 helpsoftware sustainability (maybe even beyond science) or at least serve as a tem-plate prototype.Alternative strategies to academic development, which can also ensure sus-tainable development, such as commercialization were not discussed, as therequirements for small and large projects alike, first and foremost involve legalissues. Nonetheless, also in case of academic research software hand-overs, it isalways advisable to consult the involved entity’s legal department(s), due to thecomplex situation with copyright, licensing and ownership.11 r e p r i n t Appendix
Due to background of the authors, we give some specific documentation hints fornumerical software; this automatically includes code written in the languages
MATLAB / Octave , Python ( NumPy / SciPy ), R , and Julia , as well as most researchsoftware depending numerical computations. The bare minimum informationon the computation environment for these non-compiled numerical software isgiven by:• Runtime interpreter name and version.• Operating system name, version and architecture / word-width.• Processor name and exact identifier.• Required amount of random access memory.• BLAS library implementation name and version.• LAPACK library implementation name and version.Obviously, in other sciences additional minimal information may be necessary.For example in lab-sciences hardware and protocols for access to lab equipmentproviding the processed data would be essential information.
Acknowledgments
Supported by the German Federal Ministry for Economic Affairs and Energy,in the joint project: “
MathEnergy — Mathematical Key Technologies forEvolving Energy Grids”, sub-project: Model Order Reduction (Grant number:0324019 B ).Funded by the Deutsche Forschungsgemeinschaft (DFG, German ResearchFoundation) under Germany’s Excellence Strategy EXC 2044 –390685587, Math-ematics Münster: Dynamics–Geometry–Structure.Supported by the German Federal Ministry of Education and Research(BMBF) under contract 05M18PMA.The authors would like to thank Arnim Kargl for his help in preparingthe hand-over illustrations.
References [1] E. Anderson, Z. Bai, C. Bischof, J. Demmel, J. Dongarra, J. Du Croz,A. Greenbaum, S. Hammarling, A. McKenney, and D. Sorensen.
LAPACKUsers’ Guide . SIAM, Philadelphia, PA, third edition, 1999. doi:10.1137/1.9780898719604 .[2] K. Beck.
Test Driven Development: By Example . Addison-Wesley Profes-sional, Boston, MA, USA, 2003.[3] R. F. Boisvert, editor.
Quality of Numerical Software . IFIP Advances inInformation and Communication Technology. Springer, Boston, MA, 1997. doi:10.1007/978-1-5041-2940-4 .12 r e p r i n t [4] W. J. Brown, R. C. Malveau, H. W. McCormick, and T. J. Mowbray. AntiPatterns: Refactoring Software, Architectures, and Projects in Crisis .Wiley, 1998.URL: http://antipatterns.com .[5] W. J. Cody. Basic concepts for computational software. In P. C. Messinaand A. Murli, editors,
Problems and Methodologies in Mathematical Soft-ware Production , volume 142 of
Lecture Notes in Computer Science , pages1–23. Springer, Berlin, 1982.URL: https://doi.org/10.1007/3-540-11603-6_1 .[6] H. Crowder, R. S. Dembo, and J. M. Mulvey. On reporting computationalexperiments with mathematical software.
ACM Trans. Math. Software ,5(2):193–203, 1979. doi:10.1145/355826.355833 .[7] Deutsche Forschungsgemeinschaft. Guidelines for safeguarding good scien-tific practice. , 2019.[8] R. Di Cosmo and S. Zacchiroli. Software heritage: Why and how to pre-serve software source code. In iPRES 2017: 14th International Conferenceon Digital Preservation , 2017.URL: https://ipres2017.jp/wp-content/uploads/19Roberto-Di-Cosmo.pdf .[9] J. Fehr, J. Heiland, C. Himpe, and J. Saak. Best practices for replicability,reproducibility and reusability of computer-based experiments exemplifiedby model reduction software.
AIMS Mathematics , 1(3):261–281, 2016. doi:10.3934/Math.2016.3.261 .[10] F. Henderson. Software engineering at Google. Technical report, Google,2017.URL: https://arxiv.org/pdf/1702.01715.pdf .[11] K. Hinsen. The roles of code in computational science.
Computing inScience & Engineering , 19(1):78–82, 2017. doi:10.1109/MCSE.2017.18 .[12] N. C. Hong. Minimal information for reusable scientific software. In
Pro-ceedings of the 2nd Workshop on Sustainable Software for Science: Practiceand Experiences (WSSSPE2.1) , 2014.URL: https://doi.org/10.6084/m9.figshare.1112528 .[13] N. C. Hong. Why do we need to compare research software, and how shouldwe do it? In
Proceedings of the 4th Workshop on Sustainable Software forScience: Practice and Experiences (WSSSPE4.1) , volume 1686 of
CEURWorkshop Proceedings , 2016.URL: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_29.pdf .[14] INRIA Evaluation Committee. Criteria for software self-assessment. .[15] D. Irving. A minimum standard for publishing computational results inthe weather and climate sciences.
Bull. Amer. Meteor. Soc. , 97:1149–1158,2015. doi:10.1175/BAMS-D-15-00010.1 .13 r e p r i n t [16] M. Jackson. Software deposit guidance (version 1.0). Technical report,Software sustainability Institute, 2018. see also https://softwaresaved.github.io/software-deposit-guidance/ .URL: https://doi.org/10.5281/zenodo.1327310 .[17] A. Johanson and W. Hasselbring. Software engineering for computationalscience: Past, present, future. Computing in Science & Engineering ,20(2):90–109, 2018. doi:10.1109/MCSE.2018.021651343 .[18] M. Katerbow and G. Feulner. Handreichung zum Umgang mitForschungssoftware, 2018. Herausgegeben von der ArbeitsgruppeForschungssoftware im Rahmen der Schwerpunktinitiative Digitale Infor-mation der Allianz der deutschen Wissenschaftsorganisationen. Unter Mi-tarbeit von M. Bornschein, B. Brembs, M. Erben-Russ, K. Förstner, M.Franke, B. Fritzsch, J. Fuhrmann, M. Goedicke, S. Janosch, U. Konrad, D.Zielke.URL: https://doi.org/10.5281/zenodo.1172970 .[19] Olivier Lacan.
Keep a Changelog .URL: https://keepachangelog.com .[20] A. J. Laub. Numerical linear algebra aspects of control design com-putations.
IEEE Trans. Automat. Control , 30(2):97–108, 1985. doi:10.1109/TAC.1985.1103900 .[21] C. Lawson, R. Hanson, D. Kincaid, and F. Krogh. Basic linear algebrasubprograms for FORTRAN usage.
ACM Trans. Math. Software , 5:303–323, 1979. doi:10.1145/355841.355847 .[22] Nature Publishing Group. Code check.
Nature , 555:142, 2018. doi:10.1038/d41586-018-02741-4 .[23] F. Queiroz, R. Silva, J. Miller, S. Brockhauser, and H. Fangohr. Goodusability practices in scientific software development. In
Proceedings of the5th Workshop on Sustainable Software for Science: Practice and Experi-ences (WSSSPE5.1) , 2017.URL: https://doi.org/10.6084/m9.figshare.5331814.v3 .[24] T. Schlauch, M. Meinel, and C. Haupt. DLR software engineering guide-lines, 2018.URL: http://doi.org/10.5281/zenodo.1344612 .[25] Software Heritage. How to use software heritage for archiving andreferencing your source code: guidelines and walkthrough, 2019.URL: https://annex.softwareheritage.org/public/guidelines/archive-research-software.pdf .[26] V. Stodden. Enabling reproducible research: Open licensing for scientificinnovation.
International Journal of Communications Law and Policy ,pages 1–55, 2009.URL: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1362040 . 14 r e p r i n t [27] E. J. H. Westby. Git for Teams . O’Reilly Media, 2015.URL: http://gitforteams.com .[28] D. A. Wheeler. The free-libre / open source software (floss) license slide,2007.URL: https://dwheeler.com/essays/floss-license-slide.pdfhttps://dwheeler.com/essays/floss-license-slide.pdf