Six Years of Shiny in Research -- Collaborative Development of Web Tools in R
Peter Kasprzak, Lachlan Mitchell, Olena Kravchuk, Andy Timmins
SSix Years of
Shiny in Research - CollaborativeDevelopment of Web Tools in R by Peter Kasprzak ∗† , Lachlan Mitchell ∗‡ ,Olena Kravchuk ∗§ , Andy Timmins ∗¶ January 27, 2021
Abstract
The use of
Shiny in research publications is investigated. From the appearance ofthis popular web application framework for R through to 2018, it has been utilisedin many diverse research areas. While it can be shown that the complexity of
Shiny applications is limited by the background architecture, and real security concernsexist for novice app developers, the collaborative benefits are worth attention fromthe wider research community.
Shiny simplifies the use of complex methodologiesfor users of different specialities, at the level of proficiency appropriate for the enduser. This enables a diverse community of users to interact efficiently, utilisingcutting-edge methodologies. The literature reviewed demonstrates that complexmethodologies can be put into practice without the necessity for investment in ∗ University of Adelaide, School of Agriculture Food and Wine, SA 5064, Australia. † [email protected] ‡ [email protected] § [email protected] ¶ [email protected] a r X i v : . [ s t a t . C O ] J a n rofessional training. It would appear that Shiny opens up concurrent benefitsin communication between those who analyse data and those in other disciplines,thereby potentially enriching research through this technology.
Data is the backbone of research. With the rise of automated data gathering tools,data size and the complexity of analysis have driven a growing gap between researchdisciplines and the required data analysis. Another issue is the fact that differentapproaches to the same data can compromise validity, as seen in an analysis of ef-fect sizes in observational studies, with varied methodological work-flows having thepotential to reverse conclusions regarding the studied intervention (Donoho, 2017).Collaborative learning employing common task frameworks can help interpret, quan-tify, and possibly cap methodological variations across disciplines (Donoho, 2017).Software such as Matlab (Moler and Mathworks, 2012), Minitab (Arend, 2010),Genstat (Payne et al., 2007) and SPSS (Landau and Everitt, 2004) have attemptedto bridge this gap by creating more user friendly interfaces that either make codingmore intuitive and easier to learn, or use drop down menus and radio button selectionto bypass the command line. Current analytical software, such as those mentionedabove, each have their own limitations that include non-publication ready qualitygraphics, unintuitive drop down menus, restrictive interfacing with other software,price point (including the cost of licensing the proprietary software) and the diffi-culties that inevitably occur when colleagues attempt to run code originating fromother software on their preferred platform. Despite its own weaknesses, which in-clude a very steep learning curve and unintuitive programming language, R hasgrown to become the most popular programming language for statistics and biolog-ical data analysis, spawning over 14,000 free-to-use packages over a wide range of ubject material (Li et al., 2018).While code of any language can easily be shared between users, general use re-quires a level of familiarity with the specific program. Until recently, transforminga piece of R code into an interactive app capable of use by a broad audience re-quired either knowledge of other coding languages, or consultation/collaborationwith a computer scientist/app developer, requiring justification of the additionaltime and cost (Gunuganti, 2018). Specialist apps that benefit only a small numberof people often do not meet cost/benefit benchmarks. This means that many usefuladvancements have stalled due to the requirement for experience with data analysissoftware, or an understanding of the underlying theory for day to day use, consti-tuting a complexity barrier (DePalma, 2013). Shiny can generalise R code for alllevels of users, bringing the latest advancements in methodology measurably closerto everyone. This does create new issues relating to data security, as novice appdevelopers will now require a knowledge of web internet protocols for secure datatransfers to be assured.The increased use of technology, sensors and other data capture devices hashighlighted a new and interesting issue: researchers and practitioners without abackground in data analysis now have the ability to gather large amounts of data(LaZerte et al., 2017), yet few options exist for those without data analysis trainingto analyse the ‘big’ data gathered from the field and experiments correctly. Thishas arguably led to issues impacting experimental reproducibility. A
Nature sur-vey of 1,576 researchers from the disciplines of chemistry, physics and engineering,earth and environment, biology, medicine and other, found more than 50% of thosesurveyed believed that low statistical power or poor analysis was a strong contrib-utor to irreproducibility (Baker, 2016). In the same survey, more than 90% ofrespondents believed that a better understanding of statistics was required to drive eproducibility of research. Learning analytical methodologies and programs is anon-trivial task, and subcontracted analysis, even in-house, generally comes with await for results. Purchasing proprietary software can be inflexible and often expen-sive, taking resources away from research, while open source software is dependenton a minimum level of computer literacy, along with the ability to test the softwareto ensure correct results (LaZerte et al., 2017).Open source and free, Shiny has grown in popularity with the first
Shiny
Devel-oper Conference held in January 2016 and growing use in peer reviewed academicpapers. While the number of papers has steadily increased each year,
Shiny remainsan incompletely explored topic, with the potential for
Shiny to make a significantpositive contribution to the general field of science not yet properly examined. Tothe best of our knowledge this is the first
Shiny review.The rest of this review is organised as follows. Section 2 details the literaturesearch, the keywords and findings. Section 3 presents the technical aspects of
Shiny ,including hosting costs and security, along with its restrictions. Section 4 discussesthe use of
Shiny in research with relevant examples from the literature, and finallysection 5 gives the conclusion along with the authors’ evaluations.
A thorough search for
Shiny results in the academic literature was undertaken toinvestigate the growth in research from 2012 - 2018, and the types of publica-tions and subject areas represented. The focus of this paper is the use of
Shiny tobridge specialist academic and theoretical innovations, and their role in disseminat-ing knowledge to governments, industry and the general community. Therefore, it isacknowledged that this is a non-exhaustive list of
Shiny case uses. We acknowledgethat the literature search is not fully comprehensive, as newspaper articles, blogs nd other non-academic areas were filtered. This makes this review biased towardsan academic standpoint. The search was conducted with four major data bases.Web of Science and Scopus were used due to their reputation as multi-disciplinarydatabases, with Google Scholar and the University of Adelaide (UofA) utilised astheir algorithms search the entire document and all fields for keywords. The key-words used in all searches were of the form, ” Shiny
Web Application” OR ”
Shiny
Web App”, with an exact search not suitable in this case, and ”R” not includedto avoid the inevitable non-related hits. The search was then filtered by year tospan 2012 - 2018 and the document type was limited to Dissertations, Articles,Conference proceedings and Reviews (which were allowed), to investigate the useof
Shiny in the research literature only. Books were excluded from the search dueto the small number of published materials. A separate search conducted for booksshowed that as of 2016, only two books were written on the use of
Shiny , with bothbeing structured as instructional manuals. Beeley (2013) takes the beginner fromtheir first application and walks them through the major concepts to more com-plicated applications, while Moon (2016) uses
Shiny to teach ggplot2 (Wickhamet al., 2018) graphics. As of 2018, a Google search for ”
Shiny
Web ApplicationBooks” yielded seven results, including one second edition release.The UofA search engine is powered by ExLibris Primo and includes all resourcesowned or subscribed to by the library and selected free and open access resources.It includes 345 databases, and links to the major collections of articles and eBookstotaling over 50 million items, which can expand out to 100s of millions. The UofAsearch was conducted with the terms, ”
Shiny web app OR
Shiny web application”,and returned 5,251 results with 1,391 peer reviewed articles, 3,456 dissertations, 18reviews and 119 conference proceedings. These are broken into results by journaltitle, subject tag, language published and are displayed in Figure 1. > he search in Scopus used TITLE-ABS-KEY( Shiny
AND web AND app*) ANDPUBYEAR > < Shiny
Web App*) and filtered to the same time frame. Choices of documentcriteria included Articles and Proceedings papers which resulted in 110 Articles and34 Proceedings papers, with dissertations not returned in this search. These areonce again broken into publications per year, journal title and subject tag displayedin Figure 3. Again the predominate language was English with 142 records, oneSpanish, and one Portuguese record found.The Google Scholar search terms used first were [Shiny web app — application]and returned 16,400 results. A range of additional terms were used to narrowdown results including, ”security OR complexity OR architecture OR hosting”, with”Shiny” being a required keyword, and this search returned approximately 10,400results. Unfortunately, 135 irrelevant results were found that did not contain therequired term ”Shiny”. This appeared to be an error in the algorithm. Given thatthe search returned over 100% more results than the UofA search, it was decidedto not use these results to create this paper, as the UofA search utilised the GoogleScholar databases.
Shiny is a relatively novel tool with the total number of papers found being > > > elatively small. Assuming the UofA library search completely covers the otherthree databases (which it is advertised to do), there is an approximate total of 5,000unique peer reviewed pieces of work utilising Shiny since 2012; an average of over700 papers per year. All searches showed that Bioinformatics journals publishedthe largest number of
Shiny papers; however, the vast majority of papers werepublished by a diverse range of titles, in a diverse range of fields. This indicatesthat
Shiny is a flexible tool and not area specific. Only the Scopus search returnedslightly different information with Computer Science, Biochemistry, Mathematics,and Other subject tags registering the largest number of relevant hits. On closerinspection, while Bioinformatics did not register as a subject heading, the journalthat published the greatest number of papers was still Bioinformatics, followed byBMC Bioinformatics. This suggests that there is simply a difference in subjectlabelling. Far more papers were found by keyword searches in the body of thedocument, as evidenced by the total numbers of papers found by the library searchfrom the UofA. This suggests that
Shiny has been utilised as a general tool, and notas a new discovery in the later years. The vast majority of all papers were written inEnglish, with some European countries represented, and very few Chinese papers.The results of the search algorithms are reasonably reproducible, with somefluctuations occurring depending on the sources of publications, and performanceof the search engine. In our experience, the fluctuation is less than 10%. GoogleScholar significantly alters the number of found papers depending on sorting. Ifsorting by relevance is checked, then 127,000 results are found. Sorting by datereduces this number to what is stated above.A subset of 600 papers was chosen for thorough reading to inform this report.These were the top 600 results returned by the UofA records search when sortedvia relevance. The relevance ranking employed by ExLibris Primo is comprised of our main criteria:1. Degree of match: Fields such as title, author and subject field are given ahigher ranking, along with the order of the query terms and completeness ofphrases.2. Academic significance: Citations and journal impact factor.3. Type of search: Primo infers if the search is broad-topic or specific-topic, withbroad topic searches amplifying overview material such as reference articles.4. Publication date: Newer material is given preference.The papers that discussed Shiny generally had ”
Shiny ” in the title and/or thesubject fields, increasing their relevance score. Earlier papers were more likely todiscuss
Shiny , with newer papers more likely to mention
Shiny in the text only. Therelevance search yielded a high number of the older papers as high relevance, alongwith a very broad range of use cases. The limit of 600 papers was an empirical cutoff point, as this was the stage at which papers had ceased discussing
Shiny , andwere only stating its use. It was decided that enough use cases had been examined tomake comments regarding
Shiny ’s relatively widespread use in the body of academicwork. 445 original applications were introduced in these papers, which utilised373 unique R packages. 229 unique peer reviewed journals were represented, with55 published in Bioinformatics, 31 published in PLoS ONE, and 21 published inBMC Bioinformatics. The final subset of papers that most thoroughly discussedthe implementation of
Shiny were chosen to create this report and are given asreferences. Technical aspects
A Web application framework for R,
Shiny was conceptualised by RStudio’s CTOJoe Cheng and announced at the Joint Statistical Meeting conference in July of2012 as a tool designed to help R programmers create interactive web applications,reports and analysis without the need to know HTML, CSS, or JavaScript (Changet al., 2018).The power of
Shiny comes from its ability to enable an R user to quickly andsimply code a reactive framework. A reactive framework allows objects to be up-dated when a source is changed, along with all connected objects. For example, inan imperative programming paradigm such as R language, setting the line: c = a + b means that c is assigned the sum of previously defined terms a, b and will notchange when the values of a, b are changed without the variable c being re-evaluated.Reactive programming allows the value of c to be updated almost instantaneously,including all other variables and outputs dependent on c , whenever a or b is changed.R completes this task with information travelling from input to output in a pullfashion. A pull fashion is when c learns of the new value of a or b when c is called. Shiny creates a system of alerts that flag changed expressions and the server re-evaluates all flags in an event known as a flush (Grolemund, 2015). Using twoobject classes called reactive values, such as a = reactive(), and observers, such asb = plot(),
Shiny creates a reactive context between the two objects known as a call-back which is a command to re-evaluate the observer. Multiple observers canbe linked to the same reactive value and the server will queue up all call-backs andrun each call-back in the event of a flush (Grolemund, 2015). his reactive framework allows user inputs to be evaluated via a user interface(UI) with a series of easily-coded widgets, such as text boxes, radio buttons anddrop down menus, from pre-programmed R code. Shiny then seamlessly updatesoutputs of tables, plots and summaries. A non-R user can change the values of a and b via the user interface and explore the pre-coded results dependent on c .A Shiny application has two main parts. A user interface object and a serverfunction. The user interface contains code for the layout and appearance of theapp, with default choices restricted in appearance. Layouts can be customised andchanges to the appearance can be made if the programmer has some knowledge ofHTML or CSS. For standard applications, simple commands suffice and a knowledgeof HTML or CSS languages is not required for tweaks. The server function housesall the code that drives the functionality of the application and can utilise all thebuilt-in programs available to R and RStudio users.
For a small number of applications and limited run hours, the cost of hosting a
Shiny application is free, but it can become expensive quickly. Hosting on shinyapps.iorequires no system administration knowledge, and comes with layers of securityand is supported by
Shiny ’s IT team. According to the RStudio pricing website(Core Team, 2012), the platform is free for 5 applications and 25 active hours, thenincreases to $
39 AUD a month for unlimited applications and 500 active hours. Thetop tier of $
299 AUD a month allows for unlimited applications and 10,000 activehours.
Shiny also has the option of
Shiny server,
Shiny
Server Pro or RStudioConnect. These require a level of system administration knowledge, and also requirethe apps to be hosted on a physical or virtual machine. RStudio Server Pro costs $ oftware on a server behind an existing firewall and costs between $ $
62 AUD per user/month) and $ $ Shiny server prices were not available at the time of going to print. For those with anin-depth knowledge of internet security, it is possible, and more economical, to hostthe application independently by their own means. As Shiny is primarily a web technology, a very strong focus on application securitymust be adhered to, with novices in computer science more likely to make criti-cal mistakes (Charpentier, 2013). A well-known concept in cryptography and websecurity is unknown unknowns (Charpentier, 2013). Put simply, this refers to thefact that a developer cannot build defences for attack vectors with which they areunfamiliar. For this reason, it is generally wise to leave the specifics of data securityto experts in the field, with the end-developer instead relying on the vetted workthat has been done for them.For users of
Shiny who elect to use shinyapps.io by RStudio, this is essentiallywhat happens. Once uploaded, the application is secured behind best practices(Core Team, 2012). Unfortunately, this service is prohibitively expensive whencompared with hosting the server on a cloud platform like Amazon Web Services(AWS) (Amazon, 2019) or Microsoft Azure (Microsoft, 2019). This requires appli-cation security to be taken into the app creator’s hands. Certificates need to becreated and kept up-to-date, and servers need to be configured for HTTPS amongstother security protocols (Charpentier, 2013). Due to the local nature of R, thisis likely to be a new issue, requiring a new set of skills, for many data analystsoperating on the platform. .4 Architectural issues Curiously, there are only a small number of papers that explicitly mention concernsand limitations with respect to the use of
Shiny to develop research-focused apps. Apaper by Dwivedi and Kowalski (2018) was the first to include a limitations section,emphasising the requirement for a fast internet connection when dealing with largedata sets. This could be mitigated with the use of cloud-based resources to storethe data and host the app, with potentially faster network and processing speedsavailable with respect to local connections. Guo (2018) found R package updates alegitimate concern, as updates can occur without warning and crash an application.A less serious issue is the lack of flexibility of the dashboard, which derives fromthe simplification of its creation, with
Shiny ’s dashboard not being as flexible asone created in Java (Ge et al., 2018). There are, however, challenges with the useof
Shiny , with one of them being the background architecture.While
Shiny has many benefits, the architecture of
Shiny is a limiting factorwhen building complex applications. Previously this concern has been dismissed,with Joe Cheng stating more recently:
In the past, we’ve responded rather glibly to these requests. Just usefunctions! (Cheung et al., 2017)
As of 2017,
Shiny has made moves to address this issue with the creation of mod-ulisation (Cheng, 2017); however, more involved use cases would be handled betterby other computing languages, for the reasons detailed below. An analogous wayof conceptualising this would be in the difference between applets and applications.Applets are generally small, discrete, and of low complexity, and are developed toperform a small number of functions for a highly specific purpose: most
Shiny prod-ucts would fit this description quite well. Meanwhile, applications are generally morecomplex (Fayram, 2011). They are built for a number of different use cases, and end to have relatively large codebases. Well-established and popular web applica-tion frameworks such as Angular and React exist to fit these situations, containingmuch more general functionality than Shiny , with much less specific functionality(such as functions related to data visualization) (Mitchell, 2018). None of this is tosay that complex applications cannot be created with
Shiny , but it may not be themost mature solution for the task.While
Shiny will undoubtedly continue to evolve in much the same way as R,and many issues today will be overcome tomorrow, a number of well-establishedsoftware development paradigms must be diverged from: • Shiny actively encourages the use of single-file applications, generally referringto this singular file as app.R (Core Team, 2017). Defining everything in asingular file works well for prototypes, but quickly falls apart as an applicationgrows and increases in complexity. In general, code is compartmentalised intofiles which contain the logic for a single component. By allowing a singlefile to grow monolithic in size, code readability and re-usability is challenged,consequently making it harder to add additional components in the future(Fayram, 2011). • Shiny insists on a reactive data-driven model over the more traditional andcommon event-driven model. While not necessarily a flaw in and of itself,many novice developers consider reactivity in programming to be a non-trivialconcept (Fayram, 2011). Considering that
Shiny , by nature, is aimed towardsdata analysts rather than computer scientists, it can increase the initial diffi-culty hurdle that beginners have to overcome. For example, a bug has existedin RStudio since at least February 2018, which prevents automatic reloadingfrom working with sourced files. When using multiple files like this, the serverneeds to be stopped manually and restarted between every change, making for tedious development cycle. Concerns on the subject have not been addressedby either the RStudio or Shiny core developers (Hansen, 2018).Both of the above points begin to cause major issues when put together. Encour-agement of singular source files results in code quickly becoming unruly, threateningflexibility. This heightened complexity of source code will invariably be replicatedwithin the reactive dependency graph,
Shiny ’s internal mapping of reactive nodesand their relationships. In the event that something is not working as expected,RStudio provides little to no internal tools for debugging this graph. A new ad-dition to CRAN in the form of reactlog (Schloerke and Cheng, 2019) is a firstattempt to address this issue, which usually forces the developer to painstakinglydebug the graph by hand. As the application becomes increasingly complex, thisprocess gets closer and closer to impossible. Many of these cases remain undocu-mented as
Shiny is a burgeoning technology and, to the best of our knowledge, thispaper is the most in-depth look at the challenges in the peer-reviewed literature.A package,
ShinyTester (Kohli, 2017), was added to CRAN https://cran.r-project.org/ early in 2017. While it provides a promising first approach todebugging tools for
Shiny (such as the inclusion of a dependency graph visualiser),it unfortunately seems to have been abandoned. Tools like this would likely alleviatethe above-outlined concerns.
Shiny is designed foremost as a server technology, with applications intended to beused remotely with a stable internet connection (Core Team, 2017).
Shiny appli-cations must be built with upload and download requirements at the fore. While
Shiny applications can be run locally, doing so requires a base level of knowledge ofR that may make it a sub-optimal approach, and limits accessibility. This is compa- able with how mobile applications are generally shipped as pre-compiled binaries,rather than as raw sources that the user would need to compile and install manually.One of the largest issues with this inherent reliance on connectivity is the need fordata to be uploaded and downloaded. Since Shiny has no in-built data streamingfunctionality, it is not possible to work with parts of the data while waiting for therest to upload (Core Team, 2017). An entire transfer must be completed before thedataset is made available to the application. This forces the application to requirepre-partitioned uploads, which may not be possible for all types of datasets.It is quite common to see a dataset approaching gigabytes in size, especiallyprevalent in areas such as genomic sequencing. It is generally technically unrealisticfor datasets of this size to be worked with remotely, and would include extra datacosts. If a large amount of bandwidth were made available to a single user, thiscould open up the user’s service to potential denial-of-service attacks by maliciousentities (Cloudflare, 2019). Furthermore, it may be legally unrealistic in terms ofdata ownership. Users are often uncomfortable about providing sensitive data tounknown receivers, as there is no way for a
Shiny app to prove that it is not storinguploaded information permanently for the developer’s own academic or financialgain (Kacha and Zitouni, 2018).
The pattern of peer reviewed work, as shown in Figure 1, Figure 2, and Figure 3,shows that Bioinformatics is a popular and growing area for
Shiny apps. Areasthat traditionally have a lower focus on data analysis skills for researchers, suchas Biological Sciences, Education and Index Medicus, appear to have higher usage evels. In the current literature, Shiny is primarily used as a delivery/visualisationtool and so it is not the focus; with many papers referencing the use of
Shiny butnot discussing its merits. This trend becomes obvious in more recent papers, withmuch of the best discussion occurring in earlier papers.To investigate the uptake of
Shiny , we must first understand some of the factorsthat determine the uptake of innovation. These are stated by Rogers (2001) as:(a) relative advantage, (b) compatibility, (c) complexity, (d) trialability and (e)observability. Rogers (2001) defined complexity as: ...the degree to which an innovation is perceived as difficult to understandand use.
The first peer-reviewed
Shiny publications appeared in 2013, with the first two dis-sertations contributing the most to this discussion, as they give a glimpses of
Shiny ’svast potential. The first dissertation using
Shiny was published by (DePalma et al.,2017). To drive innovation and uptake, tools must be accessible and usable by allinterested parties (Jahanshiri and Shariff, 2014, Klein et al., 2017). Moraga (2017)noted that the area of public health, while there had been progress in methodologyand analysis: ...these methods are still inaccessible for many researchers lacking theadequate programming skills to effectively use the required software.
The first peer-reviewed
Shiny publications appeared in 2013, with the first two dis-sertations contributing the most to this discussion, as they give a glimpses of
Shiny ’svast potential. The first dissertation using
Shiny was published by DePalma (2013).The
Shiny app allowed non-computer literate clinicians the ability to harness pow-erful statistical methodologies in a robust framework, and to conduct antimicrobialsusceptibility tests to determine an unknown pathogen’s susceptibility to variousantibioticss. DePalma (2013) noted that previously new methods have not been dopted due to: ...various computational difficulties and an absence of easy to use soft-ware for clinicians. Complex methodologies were able to be immediately used by end users without anassumption of computational skills, to inform important medical checks. This di-rect transfer of method is a concrete example of how
Shiny is able to make complexresearch available to all interested parties, regardless of knowledge level. Specialisedapplications such as this would be difficult and costly to create without
Shiny and,without general use software, any advanced methodology could stall in uptake due tocomplexity barriers. This was later followed up with dBETS (diffusion BreakpointEstimation Testing Software) by DePalma et al. (2017), who once again acknowl-edged: ...the computational complexities associated with these new approacheshas been a significant barrier for clinicians.Shiny is a potential solution to the barrier of complexity for the uptake of newmethodologies.
Cross-collaboration between researchers and the easy dissemination of results is keyto external validity (Munaf`o et al., 2017).
Shiny promotes collaboration by allowingpeople with varying skill levels access to more complex methodologies. This hasa flow-on benefit to promote the collaboration of practitioners with researchers,or field researchers with theorists, in order to create specialised, fit for purposeapplications. This is illustrated in a paper by Wages and Petroni (2018), whichdesigns and conducts Phase 1 dose-finding trials using the continual reassessmentmethod, and was noted to: ..facilitate more efficient collaborations within study teams. Klein et al. (2017) underscores the requirement that ...facilitating the deployment of web applications for data analysis is im-portant to promote collaboration within the scientific community and be-tween scientists and stakeholders.
Further examples of
Shiny being used to open discussion by using apps to bringrelevant parties with differing skills sets into collaboration include D´ıaz-Gay et al.(2018), who stated: ...analysis of somatic mutational signatures remains currently inaccessi-ble for a substantial proportion of the scientific community. as well as Whateley et al. (2015), who noted the knowledge gap between relevantparties: ...demonstrates the use of the Shiny web framework to bridge that gap,allowing for collaborative development of web tools that can be coded inthe widely-used and free R statistical computing language.
The ability to bridge the gap between researchers and the tools required for theirdata analysis was mentioned by Chen et al. (2018) in the context of environmentalDNA. eDNA is becoming an essential tool in ecology and conservation biology andis utilised by a range of people with varying skill levels, with Kandlikar et al. (2018)stating:
Results from eDNA analyses can engage and educate natural resourcemanagers, students, community scientists and naturalists, but withoutsignificant training in bioinformatics, it can be difficult for this diverseaudience to interact with eDNA results. hiny allows discipline specialists outside of computer science to code their ownapps, bridging the skill gap for other researchers (Niu, 2017). This was demon-strated by an app called Armadillo Mapper (Feng et al., 2017), which was designedspecifically to decrease the time between synthesising distributional knowledge on acomputer and carrying out conservation efforts in the field. This encourages thosewithout the resources to conduct their own analysis to collaborate closely with ana-lysts to create specialist applications. Rather than sending final data to an analystfor analysis, discussion and collaboration is encouraged at the beginning of an ex-periment. This sets up low quality data due to issues such as pseudo-replication,low power and confounding variables being avoided at the design stage rather thanthe analysis stage. Shiny has the flexibility to bridge the gap between specialised data gathering toolsand available software. A
Shiny app accompanying the R package rHyperSpec (Laney, 2013) was created to take complex data generated by hyperspectral camerasand link the data to available software packages in response to the problem of therebeing: ...few free, open-source software packages that enable researchers to eas-ily process and analyse such data in a manner that maximizes inter-comparison between studies.
This showcases the flexibility of
Shiny applications being able to upload informationin various formats, make appropriate changes, and output the data in a form usableby another, completely independent piece of equipment/software. Previously, therewere precious few options to link independent software/equipment, especially with-out breaching warranty restrictions.
Shiny shows tremendous flexibility in working ith existing infrastructure to help decrease costs, especially when technologies arein their infancy. Shiny gains flexibility and customisability directly from R. One of
Shiny ’s mostuseful abilities is to wrap both existing and new R packages for general consumption.Beck (2014) created an app called
Seed , which bundled several R packages togetherand used
Shiny to host them on the web, allowing: ...user’s access to powerful R based functions and libraries through a sim-ple user interface.
In the Precision Agriculture (PA) space, farmers have access to a multitude ofproprietary sensors, few of which can be linked directly to analysis tools (Jayaramanet al., 2016).
Shiny ’s highly customisable framework facilitates the linking of severalpieces of independent software, and can avoid manual data wrangling and transfer.As an example, Jahanshiri and Shariff (2014) took data from existing PA sensors andutilised R functions for its analysis and visualisation of results.
Shiny has provedmore than useful in the results visualising area, with packages such as
Shiny
Stan (Gabry et al., 2018) created in order to visualise modelling parameters and resultsfrom MCMC simulations.
R packages can be thought of as a level of abstraction down from mathematical the-ory, as the packages can be used without the need to have full understanding of themethodology.
Shiny can be thought of as another level of abstraction down again,as R packages can be utilised without needing an understanding of R itself. Thisability to generalise analysis methodologies makes
Shiny available to any interestedparty, driving flexibility, dialogue and cross-collaboration.There is an overarching requirement when making tools available to a broader udience to ensure correct methodology. The first example of using Shiny to guideand educate the user came from Assaad et al. (2014) who created two
Shiny appsintended to allow Microsoft Word users access to One Way Anova analysis and posthoc tests. The app gave instructions to guide users through the process, whichgreatly simplified the common statistical test, whilst promoting proper statisticalmethodology. A real-world example of protecting the end user comes from Hsu et al.(2018), who created an app for proper randomisation when allocating participantsto a three-armed, double-blinded, randomized controlled trial (RCT) for depression.One critical characteristic of the app was to ensure mistakes were not made whenproperly balancing strata. A fail-safe against experimental error was employed bynot allowing participants to have their experimental ID overwritten. This meansthat any accidental changes after treatment has begun would not impact on thetreatment received.Generalised applications must be flexible to differing individual parameters.
Shiny makes it a simple task to allow parameters of a methodology to be changed,depending on individual circumstances.
Shiny -wrapped simulations were used toexplore humanitarian responses and financial institution resiliency for earthquakerisks in Indonesia, with Hartell (2014) allowing the simulation to be tweaked byindividuals so that adjustments to calibration parameters could be made, basedon specific interests or circumstances. Other apps that allowed the user to specifyparameters were created by Zhou et al. (2014) for detecting differential expressionsin RNA sequencing and Yin (2014), who utilised Bayesian statistical modelling toinvestigate the networks of epidemic transmission.
Shiny makes complex methodologies accessible to those who would previouslynot be part of the conversation; most likely due to a lack of theoretical study, orlack of familiarity with coding or analysis programs.
Shiny was explicitly noted to elp increase engagement by LaZerte et al. (2017), who created FeedR in order torecord and visualise RFID data from ecological studies. The huge amount of datafrom RFID quickly becomes overwhelming and requires specialist methods to cope.The FeedR Shiny app was created to wrap the paired R package in order that: ...this framework will become a meeting point for science, education andcommunity awareness...we aim to inspire citizen engagement while si-multaneously enabling robust scientific analysis (LaZerte et al., 2017).
Reproducibility of research is a critical cornerstone of responsible research practices.Studies, such as Munaf`o et al. (2017), have indicated that reproducibility is notat high enough levels, while results of a survey conducted by Baker (2016) andpublished in Nature found: ...more than 70% of researchers have tried and failed to reproduce anotherscientist’s experiments and more than half have failed to reproduce theirown experiments.
Eight practices are argued for by Munaf`o et al. (2017), which include promotingtransparency and open science to increase reproducibility. Open source softwaresuch as
Shiny can aid these objectives by creating a vessel to preserve code andallow a greater number of interested parties to critically evaluate methodologiesand results.One benefit of
Shiny -wrapped code is that methodology comparisons becomemuch easier to conduct. Methodologies wrapped in
Shiny applications can be com-pared on a known data set under various conditions by the end user. This is apowerful tool in the advancement of reproducible research.
Shiny was explicitlyused in a dissertation by Parvandeh (2018) as a vessel to show this particular strat- gy and to create reproducibility of results, enhancing responsible and reproducibleresearch goals.A Shiny app, or at least the code behind it, is enduring. A paper from Sieriebri-ennikov et al. (2014) included a
Shiny application named Nematode Indicator JointAnalysis (NINJA) 2.0, to automate manual calculations previously carried out us-ing spreadsheet software, which is time consuming and prone to errors. The aimfor NINJA to remain freely accessible was validated when it was later used byBurkhardt et al. (2019) to aid nematode calculations in semi-arid wheat systems,five years after its release. This suggests that maintaining a
Shiny application is notoverly difficult. The benefit of
Shiny ’s easy maintenance and updating was men-tioned for the first time in a dissertation by Niu (2017), which highlighted the factthat only the source code requires changing, without having to download patchesor modify individual applications.
Shiny also appeared in conjunction with machine learning to explore early phasedrug discovery processes (Korkmaz et al., 2015), with Wojciechowski et al. (2015)noticing the power of
Shiny to disseminate the results of research, stating:
Interactive applications, developed using Shiny for the R programminglanguage, have the potential to revolutionize the sharing and communi-cation of pharmacometric model simulations.
Free and open source software is ideally suited to disseminating the products ofresearch (LaZerte et al., 2017). This also drives collaboration and was noted toencourage local and direct monitoring of environmental data in Kenya (Mose et al.,2017). LaZerte et al. (2017) also found that
Shiny ’s open source nature has anotherimportant benefit, which is that it: ...reduces financial barriers to its use and the open-source aspect permitsand encourages collaboration which can result in better, more powerful oftware. Cross-collaboration and use of open source
Shiny will hopefully also help to drivedata sharing. Yi et al. (2017) noted the utility and importance of data sharingpromoted by
Shiny applications. This outcome is one of the key recommendationsby Munaf`o et al. (2017) in order to drive transparency and openness and is currentlya policy for
Science and
Springer Nature journals.
The strengths shown by
Shiny seem to fit very well in the educational sector. Thus,
Shiny has frequently been used as a teaching aid to convey complex ideas to students(Williams and Williams, 2018). Educational tools, such as those by Arnholt (2018),help teach the concept of power in hypothesis tests, with Williams and Williams(2018) creating a similar application for confidence intervals. An app by Courtneyand Chang (2018) normalises large datasets and enables students to explore theresults of differing transformations. There are other benefits to using
Shiny in theeducation sector. Kandlikar et al. (2018) created the
Shiny app ranacapa and foundthat:
A key benefit of using ranacapa was that, despite having no prior bioin-formatics experience, students could begin exploring the biodiversity intheir samples in a matter of minutes by using the online instance of theShiny app.
This had the flow-on effect of enabling teachers to focus more on theory, instead ofthe inevitable problems when teaching new, more complex software, and provideda useful aid for self-learning (Kandlikar et al., 2018). Conclusion
This review examined
Shiny in peer reviewed publications from 2012 to 2018 andmapped the growth in its use through various research fields. A subset of 600 paperswere used to inform the bulk of the paper, with the authors’ personal experiences of
Shiny included. While
Shiny is not a silver bullet solution to issues in the researchfield, it confers the ability for specialised applications to be created cheaply andeasily, such that any level of end user maybe included, no matter the complexitylevel of the methodology. This primary benefit creates a direct pathway for newfindings to be rapidly incorporated into established workflows. The flexibility of
Shiny means that apps can be tailored to exact specifications in all regards, withchanges and maintenance of the app made relatively easy as an ongoing productof consultation, thereby further promoting collaboration. If an app is consideredworthwhile adopting for an existing workflow, widespread adoption across an entireworkplace is as simple as sharing the web address. This will have the inevitableeffect of allowing fewer people to achieve more. This does necessarily mean existingjobs risk becoming obsolete: as new, potentially variant jobs will be created as aresult of the new work undertaken. As we progress further into this technologicalworld, this argument will require mature debate and nuancing to be resolved.In the current literature,
Shiny has been used primarily as a visualisation anddissemination tool, with few papers exploring the concurrent benefits and challengesmentioned in this review. One benefit identified in the literature is the opportunityto increase high value dialogue between people with different skill sets. For example,field researchers, primary producers or marketers are able to work alongside theoret-ical researchers/consultants to create highly customised applications for up-comingexperiments or daily work. Code published as a
Shiny application has the usefulattribute of making methodology comparisons easy, which promotes reproducible esearch and best practice standards.With the ability to accelerate access to data analysis techniques comes theparamount issue of data security for those unfamiliar with web protocols. It isessential for those who host web-based applications to become knowledgeable inthis area. Web security protocols are likely to be a new skill set for many R pro-grammers: a crucial task, potentially preventing widespread uptake of Shiny .While the use of
Shiny apps requires minimal experience with computers, thecreation of a
Shiny application is a different story. The lack of debugging tools,the encouragement of single file applications, and the current implementation of thereactive data-driven model will limit the complexity of future applications. Otheropen source and proprietary options are currently available, however,
Shiny ’s flexi-bility, customisability, and low cost is highly desirable. Open source software comeswith a minimum knowledge requirement barrier to entry, and proprietary softwarecan be expensive and inflexible to changing situations and circumstances. Main-tenance is required with
Shiny , although it is limited to updating code when Rpackages or dependencies change. This updating can be achieved via the sourcecode for all users.
Shiny is one of the better tools available if one is an existing R programmer,given its inherited scope from R. It helps promotes open and reproducible researchand offers a real pathway for making complicated methodologies usable to thoseoutside universities and supported research facilities. The ability to provide anavenue to increase high value collaboration and dialogue between interested partieswith differing skills sets makes
Shiny a tool well worth exploring. Acknowledgements
We would like to gratefully acknowledge the first author’s scholarship for the M.Philprogram of the first author from the Grains Research and Development Corporation(GRDC) Australia.
Bibliography
Amazon (2019). Amazon EC2 Pricing. https://aws.amazon.com/ec2/pricing/.
Arend, D. (2010). Minitab 17 Statistical Software. Minitab, Inc.
Arnholt, A. T. (2018). Using a Shiny app to teach the concept of power.
TeachingStatistics . Assaad, H. I., Zhou, L., Carroll, R. J., and Wu, G. (2014). Rapidpublication-ready MS-Word tables for one-way ANOVA.
SpringerPlus , 474. Baker, M. (2016). 1,500 scientists lift the lid on reproducibility.
Nature News , 452.
Beck, D. (2014).
Investigating the Use of Classification Models to Study MicrobialCommunity Associations with Bacterial Vaginosis . Ph.D., University of Idaho,United States – Idaho.
Beeley, C. (2013).
Web Application Development with R Using Shiny . Olton:Packt Publishing Ltd, first edition.
Burkhardt, A., Briar, S. S., Martin, J. M., Carr, P. M., Lachowiec, J.,Zabinski, C., Roberts, D. W., Miller, P., and Sherman, J. (2019). Peren-nial crop legacy effects on nematode community structure in semi-arid wheatsystems.
Applied Soil Ecology . hang, W., Cheng, J., Allaire, J. J., Xie, Y., and McPherson, J. (2018). Shiny : Web Application Framework for R.
Charpentier, J. (2013). Web application Security. Technical Report NetworkProject, 7.5 hp, Halmstad University, Halmstad, Sweeden.
Chen, Z., Zheng, Y., Wang, Z., Kutner, M., Curran, W. J., and Kowal-ski, J. (2018). Interactive calculator for operating characteristics of phase Icancer clinical trials using standard 3+3 designs.
Contemporary Clinical TrialsCommunications , 145–153. Cheng, J. (2017). Shiny - Modularizing Shiny app code.https://shiny.rstudio.com/articles/modules.html.
Cheung, F., Fantoni, G., Conner, M., Sellers, B. A., Kotliarov, Y.,Candia, J., Stagliano, K., and Biancotto, A. (2017). Web Tool for Navi-gating and Plotting SomaLogic ADAT Files.
Journal of Open Research Software . Cloudflare
Core Team, R.
Core Team, R. (2017). Shiny Welcome to Shiny.https://shiny.rstudio.com/tutorial/written-tutorial/lesson1/.
Courtney, M. G. R. and Chang, K. C. (2018). Dealing with non-normality:An introduction and step-by-step guide using R.
Teaching Statistics , 51–59. DePalma, G. (2013).
Disk Diffusion Breakpoint Determination Using a BayesianNonparametric Variation of the Errors-in-Variables Model . Ph.D., Purdue Uni-versity, United States – Indiana. ePalma, G., Turnidge, J., and Craig, B. A. (2017). Determination of diskdiffusion susceptibility testing interpretive criteria using model-based analysis:Development and implementation. Diagnostic Microbiology and Infectious Dis-ease , 143–149. D´ıaz-Gay, M., Vila-Casades´us, M., Franch-Exp´osito, S., Hern´andez-Ill´an, E., Lozano, J. J., and Castellv´ı-Bel, S. (2018). Mutational Sig-natures in Cancer (
MuSiCa ): A web application to implement mutational sig-natures analysis in cancer samples.
BMC Bioinformatics , 224. Donoho, D. (2017). 50 Years of Data Science.
Journal of Computational andGraphical Statistics , 745–766. Dwivedi, B. and Kowalski, J. (2018). shinyGISPA : A web application forcharacterizing phenotype by gene sets using multiple omics data combinations.
PLoS ONE . Fayram, D. (2011). Functional Programming Is Hard, That’s Why It’s Good.
Feng, X., Castro, M. C., Linde, E., and Papes¸, M. (2017).
ArmadilloMapper : A Case Study of an Online Application to Update Estimates of Species’Potential Distributions.
Tropical Conservation Science . Gabry, J., Team, S. D., Andreae, M., Betancourt, M., Carpenter, B.,Gao, Y., Gelman, A., Goodrich, B., Lee, D., Song, D., and Trangucci,R. (2018). Shinystan: Interactive Visual and Numerical Diagnostics and PosteriorAnalysis for Bayesian Models.
Ge, S. X., Son, E. W., and Yao, R. (2018). iDEP : An integrated web appli-cation for differential expression and pathway analysis of RNA-Seq data.
BMCBioinformatics . rolemund, G. (2015). Shiny - How to understand reactivity in R.https://shiny.rstudio.com/articles/understanding-reactivity.html. Gunuganti, A. (2018). Application Development Framework for R/Shiny. In
PharmaSUG 2018 Conference Proceedings , volume AD-24, page 9, Seattle. Phar-maSUG.
Guo, J. (2018).
Developing a Visualization Tool for Unsupervised Machine LearningTechniques on *Omics Data . Master’s, University of Washington, United States– Washington.
Hansen, K. (2018). Rstudio - Reload Shiny App when using source’ed moduleswithout restart. https://stackoverflow.com/questions/50169896/reload-shiny-app-when-using-sourceed-modules-without-restart.
Hartell, J. (2014).
Earthquake Risk in Indonesia: Parametric Contingent Claimsfor Humanitarian Response and Financial Institution Resiliency . Ph.D., Univer-sity of Kentucky, United States – Kentucky.
Hsu, K. J., Caffey, K., Pisner, D., Shumake, J., Risom, S., Ray, K. L.,Smits, J. A. J., Schnyer, D. M., and Beevers, C. G. (2018). Attentionalbias modification treatment for depression: Study protocol for a randomizedcontrolled trial.
Contemporary Clinical Trials , 59–66. Jahanshiri, E. and Shariff, A. R. M. (2014). Developing web-based dataanalysis tools for precision farming using R and Shiny.
IOP Conference Series:Earth and Environmental Science . Jayaraman, P. P., Yavari, A., Georgakopoulos, D., Morshed, A., andZaslavsky, A. (2016). Internet of Things Platform for Smart Farming: Experi-ences and Lessons Learnt.
Sensors , 1884. acha, L. and Zitouni, A. (2018). An Overview on Data Security in CloudComputing. pages 250–261. Kandlikar, G. S., Gold, Z. J., Cowen, M. C., Meyer, R. S., Freise, A. C.,Kraft, N. J., Moberg-Parker, J., Sprague, J., Kushner, D. J., andCurd, E. E. (2018).
Ranacapa : An R package and Shiny web app to exploreenvironmental DNA data with exploratory statistics and interactive visualiza-tions.
F1000Research . Klein, T., Samourkasidis, A., Athanasiadis, I. N., Bellocchi, G., andCalanca, P. (2017). webXTREME : R-based web tool for calculating agrocli-matic indices of extreme events.
Computers and Electronics in Agriculture ,111–116.
Kohli, A. (2017).
ShinyTester :Functions to Minimize Bonehead Moves WhileWorking with ’shiny’.
Korkmaz, S., Zararsiz, G., and Goksuluk, D. (2015).
MLViS : A Web Toolfor Machine Learning-Based Virtual Screening in Early-Phase of Drug Discoveryand Development.
PLoS One; San Francisco . Landau, S. and Everitt, B. (2004).
A Handbook of Statistical Analyses UsingSPSS . Chapman & Hall/CRC, Boca Raton.
Laney, C. M. (2013).
Toward New Data and Information Management Solutionsfor Data-Intensive Ecological Research . Ph.D., The University of Texas at ElPaso, United States – Texas.
LaZerte, S. E., Reudink, M. W., Otter, K. A., Kusack, J., Bailey, J. M.,Woolverton, A., Paetkau, M., de Jong, A., and Hill, D. J. (2017). Feedrand animalnexus.ca: A paired R package and user-friendly Web application for ransforming and visualizing animal movement data from static stations. Ecologyand Evolution , 7884–7896. Li, J., Cui, B., Dai, Y., Bai, L., and Huang, J. (2018). BioInstaller: Acomprehensive R package to construct interactive and reproducible biologicaldata analysis applications based on the R platform.
PeerJ . Microsoft (2019). Pricing – Linux Virtual Machines — Microsoft Azure.https://azure.microsoft.com/en-au/pricing/details/virtual-machines/linux/.
Mitchell, E. (2018). Shiny applications without Shiny.
Moler, C. and Mathworks (2012). MATLAB 8.0 and Statistics Toolbox 8.1.The MathWorks, Inc.
Moon, K.-M. (2016).
Learn Ggplot2 Using Shiny App . Number 2197-5736 in UseR! Springer.
Moraga, P. (2017). SpatialEpiApp: A Shiny web application for the analysis ofspatial and spatio-temporal disease data.
Spatial and Spatio-temporal Epidemi-ology , 47–57. Mose, V. N., Western, D., and Tyrrell, P. (2017). Application of opensource tools for biodiversity conservation and natural resource management inEast Africa.
Ecological Informatics . Munaf`o, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers,C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware,J. J., and Ioannidis, J. P. A. (2017). A manifesto for reproducible science.
Nature Human Behaviour . iu, B. (2017). Mass Spectrometry-Based Structural Proteomics: Methodology andApplication of Fast Photochemical Oxidation of Proteins (FPOP) . Ph.D., Wash-ington University in St. Louis, United States – Missouri.
Parvandeh, S. (2018).
Epistasis Network and Machine Learning Methods for theAnalysis of Biological Large Data . Ph.D., The University of Tulsa, United States– Oklahoma.
Payne, R., Murray, D., Harding, S., Baird, D., and Soutar, D. (2007).GenStat. VSN International.
Rogers, M. (2001). Evolution: Diffusion of Innovations. In Smelser, N. J. andBaltes, P. B., editors,
International Encyclopedia of the Social & Behavioral Sci-ences , pages 4982–4986. Pergamon, Oxford.
Schloerke, B. and Cheng, J. (2019).
Reactlog : Reactivity Visualizer for’shiny’.
Sieriebriennikov, B., Ferris, H., and de Goede, R. G. M. (2014). NINJA:An automated calculation system for nematode-based biological monitoring.
Eu-ropean Journal of Soil Biology , 90–93. Wages, N. A. and Petroni, G. R. (2018). A web tool for designing and con-ducting phase I trials using the continual reassessment method.
BMC Cancer . Whateley, S., Walker, J. D., and Brown, C. (2015). A web-based screeningmodel for climate risk to water supply systems in the northeastern United States.
Environmental Modelling & Software , 64–75. ickham, H., Chang, W., Henry, L., Pedersen, T. L., Takahashi, K.,Wilke, C., and Woo, K. (2018). Ggplot2 : Create Elegant Data VisualisationsUsing the Grammar of Graphics.
Williams, I. J. and Williams, K. K. (2018). Using an R shiny to enhance thelearning experience of confidence intervals.
Teaching Statistics , 24–28. Wojciechowski, J., Hopkins, A. M., and Upton, R. N. (2015). InteractivePharmacometric Applications Using R and the Shiny Package.
CPT: Pharmaco-metrics & Systems Pharmacology , 146–159. Yi, L., Pimentel, H., and Pachter, L. (2017). Zika infection of neural progeni-tor cells perturbs transcription in neurodevelopmental pathways.
PLoS One; SanFrancisco . Yin, J. (2014).
Bayesian Statistical Modeling in Epidemics and the Contact Net-works That Transmit Them . Ph.D., The University of Iowa, United States –Iowa.
Zhou, X., Lindsay, H., and Robinson, M. D. (2014). Robustly detecting dif-ferential expression in RNA sequencing data using observation weights.
NucleicAcids Research . Appendix
This appendix contains the table of abbreviations for journal titles and subjectheadings. ibliography
Amazon (2019). Amazon EC2 Pricing. https://aws.amazon.com/ec2/pricing/.
Arend, D. (2010). Minitab 17 Statistical Software. Minitab, Inc.
Arnholt, A. T. (2018). Using a Shiny app to teach the concept of power.
TeachingStatistics . Assaad, H. I., Zhou, L., Carroll, R. J., and Wu, G. (2014). Rapidpublication-ready MS-Word tables for one-way ANOVA.
SpringerPlus , 474. Baker, M. (2016). 1,500 scientists lift the lid on reproducibility.
Nature News , 452.
Beck, D. (2014).
Investigating the Use of Classification Models to Study MicrobialCommunity Associations with Bacterial Vaginosis . Ph.D., University of Idaho,United States – Idaho.
Beeley, C. (2013).
Web Application Development with R Using Shiny . Olton:Packt Publishing Ltd, first edition.
Burkhardt, A., Briar, S. S., Martin, J. M., Carr, P. M., Lachowiec, J.,Zabinski, C., Roberts, D. W., Miller, P., and Sherman, J. (2019). Peren-nial crop legacy effects on nematode community structure in semi-arid wheatsystems.
Applied Soil Ecology . Chang, W., Cheng, J., Allaire, J. J., Xie, Y., and McPherson, J. (2018).
Shiny : Web Application Framework for R.
Charpentier, J. (2013). Web application Security. Technical Report NetworkProject, 7.5 hp, Halmstad University, Halmstad, Sweeden. hen, Z., Zheng, Y., Wang, Z., Kutner, M., Curran, W. J., and Kowal-ski, J. (2018). Interactive calculator for operating characteristics of phase Icancer clinical trials using standard 3+3 designs. Contemporary Clinical TrialsCommunications , 145–153. Cheng, J. (2017). Shiny - Modularizing Shiny app code.https://shiny.rstudio.com/articles/modules.html.
Cheung, F., Fantoni, G., Conner, M., Sellers, B. A., Kotliarov, Y.,Candia, J., Stagliano, K., and Biancotto, A. (2017). Web Tool for Navi-gating and Plotting SomaLogic ADAT Files.
Journal of Open Research Software . Cloudflare
Core Team, R.
Core Team, R. (2017). Shiny Welcome to Shiny.https://shiny.rstudio.com/tutorial/written-tutorial/lesson1/.
Courtney, M. G. R. and Chang, K. C. (2018). Dealing with non-normality:An introduction and step-by-step guide using R.
Teaching Statistics , 51–59. DePalma, G. (2013).
Disk Diffusion Breakpoint Determination Using a BayesianNonparametric Variation of the Errors-in-Variables Model . Ph.D., Purdue Uni-versity, United States – Indiana.
DePalma, G., Turnidge, J., and Craig, B. A. (2017). Determination of diskdiffusion susceptibility testing interpretive criteria using model-based analysis:Development and implementation.
Diagnostic Microbiology and Infectious Dis-ease , 143–149. ´ıaz-Gay, M., Vila-Casades´us, M., Franch-Exp´osito, S., Hern´andez-Ill´an, E., Lozano, J. J., and Castellv´ı-Bel, S. (2018). Mutational Sig-natures in Cancer ( MuSiCa ): A web application to implement mutational sig-natures analysis in cancer samples.
BMC Bioinformatics , 224. Donoho, D. (2017). 50 Years of Data Science.
Journal of Computational andGraphical Statistics , 745–766. Dwivedi, B. and Kowalski, J. (2018). shinyGISPA : A web application forcharacterizing phenotype by gene sets using multiple omics data combinations.
PLoS ONE . Fayram, D. (2011). Functional Programming Is Hard, That’s Why It’s Good.
Feng, X., Castro, M. C., Linde, E., and Papes¸, M. (2017).
ArmadilloMapper : A Case Study of an Online Application to Update Estimates of Species’Potential Distributions.
Tropical Conservation Science . Gabry, J., Team, S. D., Andreae, M., Betancourt, M., Carpenter, B.,Gao, Y., Gelman, A., Goodrich, B., Lee, D., Song, D., and Trangucci,R. (2018). Shinystan: Interactive Visual and Numerical Diagnostics and PosteriorAnalysis for Bayesian Models.
Ge, S. X., Son, E. W., and Yao, R. (2018). iDEP : An integrated web appli-cation for differential expression and pathway analysis of RNA-Seq data.
BMCBioinformatics . Grolemund, G. (2015). Shiny - How to understand reactivity in R.https://shiny.rstudio.com/articles/understanding-reactivity.html. unuganti, A. (2018). Application Development Framework for R/Shiny. In PharmaSUG 2018 Conference Proceedings , volume AD-24, page 9, Seattle. Phar-maSUG.
Guo, J. (2018).
Developing a Visualization Tool for Unsupervised Machine LearningTechniques on *Omics Data . Master’s, University of Washington, United States– Washington.
Hansen, K. (2018). Rstudio - Reload Shiny App when using source’ed moduleswithout restart. https://stackoverflow.com/questions/50169896/reload-shiny-app-when-using-sourceed-modules-without-restart.
Hartell, J. (2014).
Earthquake Risk in Indonesia: Parametric Contingent Claimsfor Humanitarian Response and Financial Institution Resiliency . Ph.D., Univer-sity of Kentucky, United States – Kentucky.
Hsu, K. J., Caffey, K., Pisner, D., Shumake, J., Risom, S., Ray, K. L.,Smits, J. A. J., Schnyer, D. M., and Beevers, C. G. (2018). Attentionalbias modification treatment for depression: Study protocol for a randomizedcontrolled trial.
Contemporary Clinical Trials , 59–66. Jahanshiri, E. and Shariff, A. R. M. (2014). Developing web-based dataanalysis tools for precision farming using R and Shiny.
IOP Conference Series:Earth and Environmental Science . Jayaraman, P. P., Yavari, A., Georgakopoulos, D., Morshed, A., andZaslavsky, A. (2016). Internet of Things Platform for Smart Farming: Experi-ences and Lessons Learnt.
Sensors , 1884. Kacha, L. and Zitouni, A. (2018). An Overview on Data Security in CloudComputing. pages 250–261. andlikar, G. S., Gold, Z. J., Cowen, M. C., Meyer, R. S., Freise, A. C.,Kraft, N. J., Moberg-Parker, J., Sprague, J., Kushner, D. J., andCurd, E. E. (2018). Ranacapa : An R package and Shiny web app to exploreenvironmental DNA data with exploratory statistics and interactive visualiza-tions.
F1000Research . Klein, T., Samourkasidis, A., Athanasiadis, I. N., Bellocchi, G., andCalanca, P. (2017). webXTREME : R-based web tool for calculating agrocli-matic indices of extreme events.
Computers and Electronics in Agriculture ,111–116.
Kohli, A. (2017).
ShinyTester :Functions to Minimize Bonehead Moves WhileWorking with ’shiny’.
Korkmaz, S., Zararsiz, G., and Goksuluk, D. (2015).
MLViS : A Web Toolfor Machine Learning-Based Virtual Screening in Early-Phase of Drug Discoveryand Development.
PLoS One; San Francisco . Landau, S. and Everitt, B. (2004).
A Handbook of Statistical Analyses UsingSPSS . Chapman & Hall/CRC, Boca Raton.
Laney, C. M. (2013).
Toward New Data and Information Management Solutionsfor Data-Intensive Ecological Research . Ph.D., The University of Texas at ElPaso, United States – Texas.
LaZerte, S. E., Reudink, M. W., Otter, K. A., Kusack, J., Bailey, J. M.,Woolverton, A., Paetkau, M., de Jong, A., and Hill, D. J. (2017). Feedrand animalnexus.ca: A paired R package and user-friendly Web application fortransforming and visualizing animal movement data from static stations.
Ecologyand Evolution , 7884–7896. i, J., Cui, B., Dai, Y., Bai, L., and Huang, J. (2018). BioInstaller: Acomprehensive R package to construct interactive and reproducible biologicaldata analysis applications based on the R platform. PeerJ . Microsoft (2019). Pricing – Linux Virtual Machines — Microsoft Azure.https://azure.microsoft.com/en-au/pricing/details/virtual-machines/linux/.
Mitchell, E. (2018). Shiny applications without Shiny.
Moler, C. and Mathworks (2012). MATLAB 8.0 and Statistics Toolbox 8.1.The MathWorks, Inc.
Moon, K.-M. (2016).
Learn Ggplot2 Using Shiny App . Number 2197-5736 in UseR! Springer.
Moraga, P. (2017). SpatialEpiApp: A Shiny web application for the analysis ofspatial and spatio-temporal disease data.
Spatial and Spatio-temporal Epidemi-ology , 47–57. Mose, V. N., Western, D., and Tyrrell, P. (2017). Application of opensource tools for biodiversity conservation and natural resource management inEast Africa.
Ecological Informatics . Munaf`o, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers,C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware,J. J., and Ioannidis, J. P. A. (2017). A manifesto for reproducible science.
Nature Human Behaviour . Niu, B. (2017).
Mass Spectrometry-Based Structural Proteomics: Methodology andApplication of Fast Photochemical Oxidation of Proteins (FPOP) . Ph.D., Wash-ington University in St. Louis, United States – Missouri. arvandeh, S. (2018). Epistasis Network and Machine Learning Methods for theAnalysis of Biological Large Data . Ph.D., The University of Tulsa, United States– Oklahoma.
Payne, R., Murray, D., Harding, S., Baird, D., and Soutar, D. (2007).GenStat. VSN International.
Rogers, M. (2001). Evolution: Diffusion of Innovations. In Smelser, N. J. andBaltes, P. B., editors,
International Encyclopedia of the Social & Behavioral Sci-ences , pages 4982–4986. Pergamon, Oxford.
Schloerke, B. and Cheng, J. (2019).
Reactlog : Reactivity Visualizer for’shiny’.
Sieriebriennikov, B., Ferris, H., and de Goede, R. G. M. (2014). NINJA:An automated calculation system for nematode-based biological monitoring.
Eu-ropean Journal of Soil Biology , 90–93. Wages, N. A. and Petroni, G. R. (2018). A web tool for designing and con-ducting phase I trials using the continual reassessment method.
BMC Cancer . Whateley, S., Walker, J. D., and Brown, C. (2015). A web-based screeningmodel for climate risk to water supply systems in the northeastern United States.
Environmental Modelling & Software , 64–75. Wickham, H., Chang, W., Henry, L., Pedersen, T. L., Takahashi, K.,Wilke, C., and Woo, K. (2018).
Ggplot2 : Create Elegant Data VisualisationsUsing the Grammar of Graphics.
Williams, I. J. and Williams, K. K. (2018). Using an R shiny to enhance thelearning experience of confidence intervals.
Teaching Statistics , 24–28. ojciechowski, J., Hopkins, A. M., and Upton, R. N. (2015). InteractivePharmacometric Applications Using R and the Shiny Package. CPT: Pharmaco-metrics & Systems Pharmacology , 146–159. Yi, L., Pimentel, H., and Pachter, L. (2017). Zika infection of neural progeni-tor cells perturbs transcription in neurodevelopmental pathways.
PLoS One; SanFrancisco . Yin, J. (2014).
Bayesian Statistical Modeling in Epidemics and the Contact Net-works That Transmit Them . Ph.D., The University of Iowa, United States –Iowa.
Zhou, X., Lindsay, H., and Robinson, M. D. (2014). Robustly detecting dif-ferential expression in RNA sequencing data using observation weights.
NucleicAcids Research ..