Creating optimal conditions for reproducible data analysis in R with 'fertile'
aa r X i v : . [ c s . C Y ] A ug Creating optimal conditions for reproducible dataanalysis in R with ‘fertile’ ∗ Audrey M. BertinStatistical and Data SciencesSmith College Benjamin S. BaumerStatistical and Data SciencesSmith CollegeAugust 28, 2020
Abstract
The advancement of scientific knowledge increasingly depends on ensuring thatdata-driven research is reproducible: that two people with the same data obtain thesame results. However, while the necessity of reproducibility is clear, there are signifi-cant behavioral and technical challenges that impede its widespread implementation,and no clear consensus on standards of what constitutes reproducibility in publishedresearch. We present fertile , an R package that focuses on a series of commonmistakes programmers make while conducting data science projects in R, primarilythrough the RStudio integrated development environment. fertile operates in twomodes: proactively (to prevent reproducibility mistakes from happening in the firstplace), and retroactively (analyzing code that is already written for potential prob-lems). Furthermore, fertile is designed to educate users on why their mistakes areproblematic and how to fix them.
Data-based research is considered fully reproducible when the requisite code and datafiles produce identical results when run by another analyst. As research is becomingincreasingly data-driven, and because knowledge can be shared worldwide so rapidly,reproducibility is critical to the advancement of scientific knowledge. Academicsaround the world have recognized this, and publications and discussions addressingreproducibility appear to have increased in the last several years (Eisner (2018);Fidler & Wilcox (2018); Gosselin (2020); McArthur (2019); Wallach et al. (2018)).Reproducible research has a wide variety of benefits. When researchers provide thecode and data used for their work in a well-organized and reproducible format, readersare more easily able to determine the veracity of any findings by following the stepsfrom raw data to conclusions. The creators of reproducible research can also more ∗ The authors gratefully acknowledge contributions from Hadley Wickham, Jenny Bryan, Greg Wilson,Edgar Ruiz, and other members of the tidyverse team. asily receive more specific feedback (including bug fixes) on their work. Moreover,others interested in the research topic can use the code to apply the methods andideas used in one project to their own work with minimal effort.However, while the necessity of reproducibility is clear, there are significant be-havioral and technical challenges that impede its widespread implementation and noclear consensus on standards of what constitutes reproducibility in published research(Peng (2009)). Not only are the components of reproducible research up for discussion(e.g., need the software be open source?), but the corresponding recommendationsfor ensuring reproducibility also vary (e.g., should raw and processed data files be inseparate directories?). Existing attempts to address reproducibility in data scienceare often either too generalized—resulting in shallow and vague recommendationsthat are challenging to implement—or too specific—approaching one aspect of repro-ducibility well but doing so in a highly technical way that fails to capture the biggerpicture and creates challenges for inexperienced users.In order to address these challenges, we present fertile (Baumer & Bertin (2020)),a low barrier-to-entry package focusing on common reproducibility mistakes program-mers make while conducting data science research in R. Publications on reproducibility can be found in all areas of scientific research. How-ever, as Goodman et al. (2016) argue, the language and conceptual framework ofresearch reproducibility varies significantly across the sciences, and there are no clearstandards on reproducibility agreed upon by the scientific community as a whole. Weconsider recommendations from a variety of fields and determine the key aspects ofreproducibility faced by scientists in different disciplines.Kitzes et al. (2017) present a collection of case studies on reproducibility practicesfrom across the data-intensive sciences, illustrating a variety of recommendations andtechniques for achieving reproducibility. Although their work does not come to aconsensus on the exact standards of reproducibility that should be followed, severalcommon trends and principles emerge from their case studies: 1) use clear separation,labeling, and documentation, 2) automate processes when possible, and 3) design thedata analysis workflow as a sequence of small steps glued together, with outputs fromone step serving as inputs into the next. This is a common suggestion within thecomputing community, originating as part of the Unix philosophy (Gancarz (2003)).Cooper et al. (2017) focus on data analysis in R and identify a similar list ofimportant reproducibility components, reinforcing the need for clearly labeled, well-documented, and well-separated files. In addition, they recommend publishing a listof dependencies and using version control. Broman (2019) reiterates the need forclear naming and file separation while sharing several additional suggestions: keepthe project contained in one directory, use relative paths, and include a README .The reproducibility recommendations from R OpenSci, a non-profit initiativefounded in 2011 to make scientific data retrieval reproducible, share similar prin-ciples to those discussed previously. They focus on a need for a well-developed file ystem, with no extraneous files and clear labeling. They also reiterate the need tonote dependencies and use automation when possible, while making clear a suggestionnot present in the previously-discussed literature: the need to use seeds, which allowfor the saving and restoring of the random number generator state, when runningcode involving randomness (Martinez et al. (2018)).When considered in combination, these sources provide a well-rounded pictureof the components important to research reproducibility. Using this literature as aguideline, we identify several key features of reproducible work. These recommenda-tions are a matter of opinion—due to the lack of agreement on which components ofreproducibility are most important, we select those that are mentioned most often,as well as some that are mentioned less but that we view as important.1. A well-designed file structure: • Separate folders for different file types. • No extraneous files. • Minimal clutter.2. Good documentation: • Files are clearly named, preferably in a way where the order in which theyshould be run is clear. • A README is present. • Dependencies are noted.3. Reproducible file paths: • No absolute paths, or paths leading to locations outside of a project’s di-rectory, are used in code—only portable (relative) paths.4. Randomness is accounted for: • If randomness is used in code, a seed must also be set.5. Readable, styled code: • Code should be written in a coherent style. Code that conforms to a styleguide or is written in a consistent dialect is easier to read (Hermans & Aldewereld(2017)). We believe that the tidyverse provides the most accessible dialectof R .Much of the available literature focuses on file structure, organization, and nam-ing, and fertile ’s features are consistent with this. Marwick et al. (2018) provide theframework for file structure that fertile is based on: a structure similar to that ofan R package (R-Core-Team (2020), Wickham (2015)), with an R folder, as well as data , data-raw , inst , and vignettes . Much of the work discussed in Section 2.1 is highly generalized, written to be appli-cable to users working with a variety of statistical software programs. Because allstatistical software programs operate differently, these recommendations are inher-ently vague and difficult to implement, particularly to new analysts who are relativelyunfamiliar with their software. Focused attempts to address reproducibility in spe-cific certain software programs are more likely to be successful. We focus on R , due toits open-source nature, accessibility, and popularity as a tool for statistical analysis. small body of R packages focuses on research reproducibility. rrtools (Marwick(2019)) addresses some of the issues discussed in Marwick et al. (2018) by creatinga basic R package structure for a data analysis project and implementing a basic testthat::check () functionality. The orderly (FitzJohn et al. (2020)) package alsofocuses on file structure, requiring the user to declare a desired project structure(typically a step-by-step structure, where outputs from one step are inputs into thenext) at the beginning and then creating the files necessary to achieve that structure. workflowr ’s (Blischak et al. (2019)) functionality is based around version controland making code easily available online. It works to generate a website contain-ing time-stamped, versioned, and documented results. checkers (Ross et al. (2018))allows you to create custom checks that examine different aspects of reproducibil-ity. packrat (Ushey et al. (2018)) is focused on dependencies, creating a packagedfolder containing a project as well as all of its dependencies so that projects depen-dent on lesser-used packages can be easily shared across computers. drake (OpenSci(2020)) analyzes workflows, skips steps where results are up to date, and providesevidence that results match the underlying code and data. Lastly, the reproducible (McIntire & Chubaty (2020)) package focuses on the concept of caching: saving in-formation so that projects can be run faster each time they are re-completed fromthe start.Many of these packages are narrow, with each effectively addressing a small com-ponent of reproducibility: file structure, modularization of code, version control, etc.These packages often succeed in their area of focus, but at the cost of accessibilityto a wider audience. Their functions are often quite complex to use, and many stepsmust be completed to achieve the required reproducibility goal. This cumbersomenature means that most reproducibility packages currently available are not easilyaccessible to users near the beginning of their R journey, nor particularly useful tothose looking for quick and easy reproducibility checks. A more effective way of re-alizing widespread reproducibility is to make the process for doing so simple enoughthat it takes little to no conscious effort to implement. You want users to ”fall intoa hole” of good practice. Continuous integration tools provide more general approaches to automated check-ing, which can enhance reproducibility with minimal code. For example, wercker —acommand line tool that leverages Docker—enables users to test whether their projectswill successfully compile when run on a variety of operating systems without accessto the user’s local hard drive (Oracle Corporation (2019)).
GitHub Actions is in-tegrated into GitHub and can be configured to do similar checks on projects hostedin repositories.
Travis CI and
Circle CI are popular continuous integration toolsthat can also be used to check R code.However, while these tools can be useful, they are generalized so as to be useful tothe widest audience. As a result, their checks are not designed to be R -specific, whichmakes them sub-optimal for users looking to address reproducibility issues involvingfeatures specific to the R programming language, such as package installation andseed setting. We paraphrase Hadley Wickham. .3 Our contribution fertile attempts to address these gaps in existing software by providing a simple, easy-to-learn reproducibility package that, rather than focusing intensely on a specific area,provides some information about a wide variety of aspects influencing reproducibility. fertile is flexible, offering benefits to users at any stage in the data analysis workflow,and provides R -specific features, which address certain aspects of reproducibility thatcan be missed by external project development software. fertile is designed to be used on data analyses organized as R Projects (i.e. di-rectories containing an .Rproj file). Once an R Project is created, fertile providesbenefits throughout the data analysis process, both during development as well asafter the fact. fertile achieves this by operating in two modes: proactively (to pre-vent reproducibility mistakes from happening in the first place), and retroactively(analyzing code that has already been written for potential problems).
Proactively, the package identifies potential mistakes as they are made by the userand outputs an informative message as well as a recommended solution. For example, fertile catches when a user passes a potentially problematic file path—such as anabsolute path, or a path that points to a location outside of the project directory—toa variety of common input/output functions operating on many different file types. library(fertile)file.exists("~/Desktop/my_data.csv") fertile is even more aggressive with functions (like setwd ()) that are almost certainto break reproducibility, causing them to throw errors that prevent their executionand providing recommendations for better alternatives. setwd("~/Desktop")
These proactive warning features are activated immediately after attaching the fertile package and require no additional effort by the user. .2 Retroactive Use Retroactively, fertile analyzes potential obstacles to reproducibility in an RStudioProject (i.e., a directory that contains an .Rproj file). The package considers severaldifferent aspects of the project which may influence reproducibility, including thedirectory structure, file paths, and whether randomness is used thoughtfully. Theend products of these analyses are reproducibility reports summarizing a project’sadherence to reproducibility standards and recommending remedies for where theproject falls short. For example, fertile might identify the use of randomness in codeand recommend setting a seed if one is not present.Users can access the majority of fertile ’s retroactive features through two primaryfunctions, proj check () and proj analyze ().The proj check () function runs fifteen different reproducibility tests, noting whichones passed, which ones failed, the reason for failure, a recommended solution, and aguide to where to look for help. These tests include: looking for a clear build chain,checking to make sure the root level of the project is clear of clutter, confirming thatthere are no files present that are not being directly used by or created by the code,and looking for uses of randomness that do not have a call to set.seed () present. Afull list is provided below: list_checks()
Subsets of the fifteen tests can be invoked using the tidyselect helper functions(Henry & Wickham (2020)) in combination with the more limited proj check some ()function. proj_dir <- "project_miceps"proj_check_some(proj_dir, contains("paths")) Each test can also be run individually by calling the function matching its checkname.The proj analyze () function creates a report documenting the structure of a dataanalysis project. This report contains information about all packages referenced incode, the files present in the directory and their types, suggestions for moving filesto create a more organized structure, and a list of reproducibility-breaking file pathsused in code. proj_analyze(proj_dir) fertile also contains logging functionality, which records commands run in the consolethat have the potential to affect reproducibility, enabling users to look at their pasthistory at any time. The package focuses mostly on package loading and file opening,noting which function was used, the path or package it referenced, and the timestampat which that event happened. Users can access the log recording their commands atany time via the log report () function: log_report() The log, if not managed, can grow very long over time. For users who do notdesire such functionality, log clear () provides a way to erase the log and start over.
Much of the functionality in fertile is achieved by writing shims. fertile ’s shimmedfunctions intercept the user’s commands and perform various logging and checkingtasks before executing the desired function. Our process is:1. Identify an R function that is likely to be involved in operations that maybreak reproducibility. Popular functions associated with only one package (e.g., read csv () from readr ) are ideal candidates. . Create a function in fertile with the same name that takes the same arguments(and always the dots ... ).3. Write this new function so that it: a) captures any arguments, b) logs the nameof the function called, c) performs any checks on these arguments, and d) callsthe original function with the original arguments. Except where warranted, theexecution looks the same to the user as if they were calling the original function.Most shims are quite simple and look something like what is shown below for read csv (). fertile::read_csv fertile shims many common functions, including those that read in a variety ofdata types, write data, and load packages. This works both proactively and retroac-tively, as the shimmed functions written in fertile are activated both when the useris coding interactively and when a file containing code is rendered.In order to ensure that the fertile versions of functions (”shims”) always supersede(”mask”) their original namesakes when called, fertile uses its own shims of the library () and require () functions to manipulate the R search () path so that it is alwayslocated in the first position. In the fertile version of library (), we detach fertile fromthe search path, load the requested package, and then re-attach fertile . This ensuresthat when a user executes a command, R will check fertile for a matching functionbefore considering other packages. While it is possible that this shifty behavior couldlead to unintended consequences, our goal is to catch a good deal of problems beforethey become problematic. Users can easily disable fertile by detaching it, or notloading it in the first place. fertile also provides several useful utility functions that may assist with the processof data analysis. The check path () function analyzes a vector of paths (or a single path) to determinewhether there are any absolute paths or paths that lead outside the project directory. Path inside the directory check_path("project_miceps") check_path(getwd()) check_path("../fertile.Rmd")
There are several functions that can be used to check the type of a file: is_data_file(fs::path(proj_dir, "mice.csv"))
The sandbox () function allows the user to make a copy of their project in a temporarydirectory. This can be useful for ensuring that projects run properly when access tothe local file system is removed. proj_dir One of the challenges with ensuring that work is reproducible is the issue of depen-dencies. Many data analysis projects reference a variety of R packages in their code.When such projects are shared with other users who may not have the required pack-ages downloaded, it can cause errors that prevent the project from running properly.The proj pkg script () function assists with this issue by making it simple and fastto download dependencies. When run on an R project directory, the function createsa .R script file that contains the code needed to install all of the packages referencedin the project, differentiating between packages located on CRAN and those locatedon GitHub. install_script <- proj_pkg_script(proj_dir)cat(readChar(install_script, 1e5)) fertile ’s simplicity enables users of any background to take advantage of its featuresand its big-picture design gives fertile the potential to provide benefits across avariety of disciplines.For example, professors could integrate fertile into their data science curricula,giving students an understanding and awareness of reproducibility early in their ca-reers that can positively impact the reproducibility of their future work. It could alsobe used by experienced analysts working collaboratively who are looking to promotea smoother exchange of feedback and ideas. Journal reviewers may also find the pack-age beneficial, allowing them to gain a fast overview of whether paper submissionsmeet reproducibility guidelines. he sample use cases in this section consider fertile ’s applicability to some ofthese scenarios in detail. Susan is taking an introductory data science course. This is her first time learninghow to code and she has not yet been exposed to ideas of research reproducibility. Herprofessor has assigned a data analysis project that must be completed in R Markdown.The project requires her to read in a data file located on her computer and use it toproduce a graph.She reads in the data, makes the graph, and knits her .Rmd file. It compilessuccessfully, so she submits the assignment. The next day, she receives an emailfrom her professor saying that her assignment failed to compile and that she needsto make changes and try again. Susan does not understand why it did not work onthe professor’s computer when it did on her own. The professor recommends thatshe install fertile and run proj check () on her assignment. She does this and getsa message informing her that she used an absolute path to open her dataset whenshe should have use a relative path instead. She looks up what this means and thenuses the new information to update her assignment. Her second submission compilessuccessfuly.On future projects, she always loads and runs fertile before submitting. R User
Emma is a post-doc with several years of R experience. She is familiar with somebasic rules of reproducibility—file paths should always be relative and randomnessshould always be associated with a seed—but has never needed to pass any sort ofreproducibility check before because her professors never emphasized that.She has just finished a research project and is looking to submit her work toa journal. When researching the journal to which she is interested in submitting,she discovers that it has high standards for research reproducibility and a dedicatededitor focusing on that aspect of submission. She goes online and finds the journal’sguidelines for reproducibility. They are more complete than any guidelines to whichshe has previously been required to conform. In addition to notes about file pathsand randomness, the journal requires a clean, well-organized folder structure, brokendown by file category and stripped of files that do not serve a purpose. In orderto be approved, submissions must also have a clear build chain and an informative README file.Unsure of the best way to achieve this structure, Emma goes online to find help.In her search, she comes across fertile . She downloads the package, and in onlya handful of commands, she identifies and removes excess files in her directory andautomatically organizes her files into a structure reminiscent of an R package. Shenow meets the guidelines for the journal and can submit her research. Results fertile is designed to: 1) be simple enough that users with minimal R experience canuse the package without issue, 2) increase the reproducibility of work produced byits users, and 3) educate its users on why their work is or is not reproducible andprovide guidance on how to address any problems.To test fertile ’s effectiveness, we began an initial randomized control trial of thepackage on an introductory undergraduate data science course at Smith College inSpring 2020 .The experiment was structured as follows: • Students are given a form at the start of the semester asking whether theyconsent to participate in a study on data science education. In order to success-fully consent, they must provide their system username, collected through thecommand
Sys.getenv(”LOGNAME”) . To maintain privacy the results are thentransformed into a hexadecimal string via the md5 () hashing function. • These hexadecimal strings are then randomly assigned into equally sized groups,one experimental group that receives the features of fertile and one group thatreceives a control. • The students are then asked to download a package called sds192 (the coursenumber and prefix), which was created for the purpose of this trial. It leveragesan .onAttach () function to scan the R environment and collect the usernameof the user who is loading the package and run it through the same hashingalgorithm as used previously. It then identifies whether that user belongs to theexperimental or the control group. Depending on the group they are in, theyreceive a different version of the package. • The experimental group receives the basic sds192 package, which consists ofsome data sets and R Markdown templates necessary for completing homeworkassignments and projects in the class, but also has fertile installed and loadedsilently in the background. The package’s proactive features are enabled, andtherefore users will receive warning messages when they use absolute or non-portable paths or attempt to change their working directory. The control groupreceives only the basic sds192 package, including its data sets and R Markdowntemplates. All students from both groups then use their version of the packagethroughout the semester on a variety of projects. • Both groups are given a short quiz on different components of reproducibilitythat are intended to be taught by fertile at both the beginning and end of thesemester. Their scores are then compared to see whether one group learnedmore than the other group or whether their scores were essentially equivalent.Additionally, for every homework assignment submitted, the professor takesnote of whether or not the project compiles successfully.Based on the results, we hope to determine whether fertile was successful atachieving its intended goals. A lack of notable difference between the experimental and control groups in terms of the number of code-related questions asked through-out the semester would indicate that fertile achieved its goal of simplicity. A higheraverage for the experimental group in terms of the number of homework assignments This study was approved by Smith College IRB, Protocol hat compiled successfully would indicate that fertile was successful in increasing re-producibility. A greater increase over the semester in the reproducibility quiz scoresfor students in the experimental group compared with the control group would in-dicate that fertile achieved its goal of educating users on reproducibility. Successaccording to these metrics would provide evidence showing fertile ’s benefit as toolto help educators introduce reproducibility concepts in the classroom.Unfortunately, we were unable to complete the analysis as intended as the trial hadto be postponed after the COVID-19 pandemic significantly altered the experimentalconditions at the midpoint of testing. Although the experiment was unsuccessful inits first attempt, we hope to run the same trial again and gather data on fertile ’seffectiveness. fertile is an R package that lowers barriers to reproducible data analysis projects in R , providing a wide array of checks and suggestions addressing many different aspectsof project reproducibility, including file organization, file path usage, documentation,and dependencies. fertile is meant to be educational, providing informative errormessages that indicate why users’ mistakes are problematic and sharing recommen-dations on how to fix them. The package is designed in this way so as to promote agreater understanding of reproducibility concepts in its users, with the goal of increas-ing the overall awareness and understanding of reproducibility in the R community.The package has very low barriers to entry, making it accessible to users withvarious levels of background knowledge. Unlike many other R packages focused onreproducibility that are currently available, the features of fertile can be accessedalmost effortlessly. Many of the retroactive features can be accessed in only twolines of code requiring minimal arguments and some of the proactive features can beaccessed with no additional effort beyond loading the package. This, in combinationwith the fact that fertile does not focus on one specific area of reproducibility, insteadcovering (albeit in less detail) a wide variety of topics, means that fertile makes iteasy for data analysts of all skill levels to quickly gain a better understanding of thereproducibility of the work.In the moment, it often feels easiest to take a shortcut—to use an absolute pathor change a working directory. However, when considering the long term path of aproject, spending the extra time to improve reproducibility is worthwhile. fertile ’suser-friendly features can help data analysts avoid these harmful shortcuts with min-imal effort. fertile , in its current version, addresses the vast majority of the aspects of repro-ducibility identified in Section 2.1 in some way. However, there are several areaswhere further development to extend the available features of the package would bebeneficial. These include the following: • Expanding dependency management features to include R session informationand package version numbers in addition to package names. Expanding code and documentation style features to analyze whether code hasbeen properly commented in addition to checking for a
README and tidy codestyle. • Adding make -like functionality that can analyze an R project structure andfiles and use this information to generate a Makefile. This Makefile would haveinformation about target files and their prerequisites and would assist with mak-ing sure that re-running an analysis is done as quickly as possible by ensuringthat only the necessary code and files that have been updated are run whenrebuilding and re-running code. Data Availability Statement
The sample project project miceps and package code associated with this paper canbe found in the R and tests folders at https://github.com/baumer-lab/fertile . References
Baumer, B. S. & Bertin, A. M. (2020), ‘fertile: creating optimal conditions for repro-ducibility’. R package version 0.0.0.9027.
URL: https://github.com/baumer-lab/fertile
Blischak, J., Carbonetto, P. & Stephens, M. (2019), ‘workflowr: A framework forreproducible and collaborative data science’. R package version 1.6.0.
URL: https://CRAN.R-project.org/package=workflowr
Broman, K. (2019), ‘initial steps toward reproducible research: organize your dataand code’.
URL: https://kbroman.org/steps2rr/pages/organize.html
Cooper, N., Hsing, P.-Y., Croucher, M., Graham, L., James, T., Krystalli, A. &Michonneau, F. (2017), ‘A guide to reproducible code in ecology and evolution’.
URL:
Eisner, D. A. (2018), ‘Reproducibility of science: Fraud, impact factors and careless-ness’,
Journal of Molecular and Cellular Cardiology , 364–368.
URL:
Fidler, F. & Wilcox, J. (2018), Reproducibility of scientific results, in E. N. Zalta, ed.,‘The Stanford Encyclopedia of Philosophy’, winter 2018 edn, Metaphysics ResearchLab, Stanford University.FitzJohn, R., Ashton, R., Hill, A., Eden, M., Hinsley, W., Russell, E. & Thompson,J. (2020), ‘orderly: Lightweight reproducible reporting’. R package version 1.0.4.
URL: https://CRAN.R-project.org/package=orderly
Gancarz, M. (2003),
Linux and the Unix Philosophy , 2nd edn, Digital Press, Woburn,MA. oodman, S. N., Fanelli, D. & Ioannidis, J. P. A. (2016), ‘What does research repro-ducibility mean?’, Science Translational Medicine (341), 1–6. URL: https://stm.sciencemag.org/content/8/341/341ps12
Gosselin, R.-D. (2020), ‘Statistical analysis must improve to address the reproducibil-ity crisis: The access to transparent statistics (acts) call to action’,
BioEssays (1), 1900189. URL: https://onlinelibrary.wiley.com/doi/abs/10.1002/bies.201900189
Henry, L. & Wickham, H. (2020), ‘tidyselect: Select from a set of strings’. R packageversion 1.1.0.
URL: https://CRAN.R-project.org/package=tidyselect
Hermans, F. & Aldewereld, M. (2017), Programming is writing is programming, in ‘Companion to the first International Conference on the Art, Science and Engi-neering of Programming’, pp. 1–8.Kitzes, J., Turek, D. & Deniz, F. (2017), The Practice of Reproducible Research: CaseStudies and Lessons from the Data-Intensive Sciences , University of CaliforniaPress, Berkeley, CA.
URL:
Martinez, C., Hollister, J., Marwick, B., Sz¨ocs, E., Zeitlin, S., Kinoshita, B. P.,Wykstra, S., Leek, J., Reich, N. & Meinke, B. (2018), ‘Reproducibility in Science:A Guide to enhancing reproducibility in scientific results and writing’.
URL: http://ropensci.github.io/reproducibility-guide/
Marwick, B. (2019), ‘rrtools: Creates a reproducible research compendium’. R pack-age version 0.1.0.
URL: https://github.com/benmarwick/rrtools
Marwick, B., Boettiger, C. & Mullen, L. (2018), ‘Packaging data analytical workreproducibly using R (and friends)’,
The American Statistician (1), 80–88. URL: https://peerj.com/preprints/3192.pdf
McArthur, S. L. (2019), ‘Repeatability, reproducibility, and replicability: Tacklingthe 3r challenge in biointerface science and engineering’,
Biointerphases (2), 1–2. URL: https://doi.org/10.1116/1.5093621
McIntire, E. J. B. & Chubaty, A. M. (2020), ‘reproducible: A set of tools that enhancereproducibility beyond package management’. R package version 1.0.0.
URL: https://CRAN.R-project.org/package=reproducible
OpenSci, R. (2020), ‘drake: A pipeline toolkit for reproducible computation at scale’.R package version 7.11.0.
URL: https://cran.r-project.org/package=drake
Oracle Corporation (2019), ‘Wercker’.
URL: https://github.com/wercker/wercker eng, R. D. (2009), ‘Reproducible research and Biostatistics’, Biostatistics (3), 405–408. URL: https://doi.org/10.1093/biostatistics/kxp014
R-Core-Team (2020), ‘Writing r extensions’,
R Foundation for Statistical Computing . URL: http://cran.stat.unipd.it/doc/manuals/r-release/R-exts.pdf
Ross, N., DeCicco, L. & Randhawa, N. (2018), ‘checkers: Automated checking of bestpractices for research compendia’. R package version 0.1.0.
URL: https://github.com/ropenscilabs/checkers/blob/master/DESCRIPTIONr
Ushey, K., McPherson, J., Cheng, J., Atkins, A. & Allaire, J. (2018), ‘packrat: Adependency management system for projects and their r package dependencies’. Rpackage version 0.5.0.
URL: https://CRAN.R-project.org/package=packrat
Wallach, J. D., Boyack, K. W. & Ioannidis, J. P. A. (2018), ‘Reproducible researchpractices, transparency, and open access data in the biomedical literature, 2015-2017’,
PLOS Biology (11), 1–20. URL: https://doi.org/10.1371/journal.pbio.2006930
Wickham, H. (2015),
R Packages , 1st edn, O’Reilly Media, Inc., Sebastopol, CA., 1st edn, O’Reilly Media, Inc., Sebastopol, CA.