6\u2005Unexplained discrepancies between planned and conducted statistical analyses in randomised trials

Abstract

Objectives To evaluate how often pre-specified statistical analysis approaches were publicly available for randomised trials, and the frequency of unexplained discrepancies between planned and conducted analyses. Method We reviewed randomised trials published in six general medical journals (BMJ, JAMA, Lancet, NEJM, PLOS Medicine, and Annals of Internal Medicine) from January-April 2018. Main outcome measures were (i) the number of trials with a publicly available pre-specified analysis approach for the primary outcome (in a protocol or statistical analysis plan [SAP]); (ii) the number of trials with no unexplained discrepancies between the trial publication and the pre-specified approach; and (iii) the types of unexplained discrepancies. Discrepancies were classified as unexplained unless the change was specified in a subsequent version of the protocol or SAP or the discrepancy was discussed in the trial publication. Data extraction was performed independently by two reviewers. Results Overall, 89 of 101 eligible trials (88%) had a publicly available pre-specified analysis approach (83 in a protocol, 6 in a SAP); this document was dated after recruitment began for 27/89 trials (30%), and for 21 trials (24%) no date was available. Only 22/89 trials (25%) did not have any unexplained discrepancies (n=5 no discrepancies, n=17 explained discrepancies only). Fifty-four trials (61%) had one or more unexplained discrepancies, and in 13 trials (15%) it was impossible to ascertain whether any unexplained discrepancies occurred due to incomplete reporting of the statistical methods in the trial publication. Unexplained discrepancies were most common for the analysis model (n=30, 34%) and analysis population (n=28, 31%), followed by the use of covariates (n=23, 26%) and handling of missing data (n=16, 18%). Most trials did not report the blinding status of the statistician in relation to database access or final sign-off of the SAP. Conclusions Discrepancies in the statistical analysis approach were common. We identified several barriers preventing an evaluation of whether changes may have introduced bias: (i) many protocols and analysis plans were from after recruitment began, preventing a comparison with the pre-trial analysis approach; (ii) discrepancies were rarely explained or justified in the trial publication; (iii) the blinding status of statisticians in relation to modifications of analysis methods was rarely reported; and (iv) some descriptions of the analysis methods used in the final publication were inadequate, preventing a comparison with the pre-specified approach. Resolution of these barriers is likely to require a multi-faceted approach targeting investigators, journals, and trial registry websites.