69\u2005Automating the process of systematic reviews in healthcare research – a methodological systematic review

Abstract

Objectives Systematic Reviews (SRs) are the cornerstone of evidence-informed healthcare decision making. However, they are extremely resource-intensive and commonly take 2 to 3 years to complete. One of the solutions put forward to support reviewers and reduce the time required to conduct SRs is automation. With recent advancements in artificial intelligence and machine learning, many tools have been, and are currently being, developed to support different stages of SR process. To date, the range of automated tools available, and their effectiveness, is unclear. To remedy this gap, we have undertaken a methodological systematic review. The specific objectives of this methodological SR are to: Classify existing tools according to the automation approach, the stage of SR supported and the level of automation. Compare the available tools according to each stage that they support and identify their strengths and limitations. Determine the effectiveness of the tools. Present and appraise evaluations of automated tools. Method This review is based on the Preferred Reporting Items for Systematic review and Meta-Analyses (PRISMA-P) Guidelines. We reviewed all published articles, grey literature, reports and software manuals that evaluate automated and semi-automated tools that support healthcare-related SRs, from screening to write-up. Only tools that are fully developed were eligible for inclusion. The systematic review toolbox (SR ToolBox), PubMed, Google and Google were systematically searched. Titles, abstracts and full articles were reviewed for inclusion independently by two reviewers. Data extraction and quality appraisal were undertaken independently by two reviewers, with disagreements resolved by consensus or by arbitration by a third reviewer if necessary. Tools’ characteristics and performance metrics reported in the included studies were extracted and tabulated. To enable comparisons, tools were grouped according to stages of SR they support, and the type of algorithms deployed in them. Results This review is currently in the data extraction stage and will be completed by the first week of June 2019. This presentation will focus on the evidence available on automated tools that support the screening, data extraction, quality appraisal and write-up phases of SRs. For each phase, we will present data on the number of tools that have been developed and the number of studies that have evaluated them. We will discuss the strengths and limitations of the methods and standards used to evaluate existing tools, and implications for future research and SR practice. Conclusions This review constitutes an important step in easing the transition of SR production from a primarily manual process to a semi-automated one. It will inform current collaborative efforts aimed at the development of evidence-informed integrative automated systems for conducting high quality SRs in healthcare research.