A New Approach for Timely Systematic Reviews of Rapidly Evolving Research: Identifying Evidence in Which We Can Place Our Confidence

Abstract

Coronavirus disease of 2019 (COVID-19) has impacted the world in unprecedented ways since first emerging in December 2019. In the last two years, the scientific community has made an enormous effort to understand COVID-19 and potential interventions. As of June 15, 2021, there were more than 140,000 COVID-19 focused manuscripts on PubMed and preprint servers, such as medRxiv and BioRxiv. Preprints, which constitute more than 15% of all manuscripts, may contain more up-to-date research findings compared to published papers, due to the sometimes lengthy timeline between manuscript submission and publication. Including preprints in systematic reviews and meta-analyses thus has the potential to improve the timeliness of reviews. However, there is no clear guideline on whether preprints should be included in systematic reviews and meta-analyses. Using a prototypical example of a rapid systematic review examining the comparative effectiveness of COVID-19 therapeutics, we propose including all preprints in the systematic review by assigning them a weight we term the confidence score . Motivated by our observation that, unlike the traditional journal submission process which is unobserved, the timeline from submission to publication for a preprint can be observed and can be modeled as a time-to-event outcome. This observation provides a unique opportunity to model and quantify the probability that a preprint will be published, which can be used as a confidence score to weight preprints in systematic reviews and meta-analyses. To obtain the confidence score, we propose a novel survival cure model, which incorporates both the time from posting to publication for a preprint, and key characteristics of the study described in the content of the preprint. Using meta data from 158 preprints on evaluating therapeutic options for COVID-19 posted through 09/03/2020, we demonstrate the utility of the confidence score in weighting of preprints in a systematic review. Our proposed method has the potential to advance timely systematic reviews of the evidence examining COVID-19 and other clinical conditions with rapidly evolving evidence bases by providing an approach for inclusion of unpublished manuscripts.