Neil D. Dattani

The statistical significance of randomized controlled trial results is frequently fragile: a case for a Fragility Index

OBJECTIVES A P-value <0.05 is one metric used to evaluate the results of a randomized controlled trial (RCT). We wondered how often statistically significant results in RCTs may be lost with small changes in the numbers of outcomes. STUDY DESIGN AND SETTING A review of RCTs in high-impact medical journals that reported a statistically significant result for at least one dichotomous or time-to-event outcome in the abstract. In the group with the smallest number of events, we changed the status of patients without an event to an event until the P-value exceeded 0.05. We labeled this number the Fragility Index; smaller numbers indicated a more fragile result. RESULTS The 399 eligible trials had a median sample size of 682 patients (range: 15-112,604) and a median of 112 events (range: 8-5,142); 53% reported a P-value <0.01. The median Fragility Index was 8 (range: 0-109); 25% had a Fragility Index of 3 or less. In 53% of trials, the Fragility Index was less than the number of patients lost to follow-up. CONCLUSION The statistically significant results of many RCTs hinge on small numbers of events. The Fragility Index complements the P-value and helps identify less robust results.

The influence of study characteristics on reporting of subgroup analyses in randomised controlled trials: systematic review

Objective To investigate the impact of industry funding on reporting of subgroup analyses in randomised controlled trials. Design Systematic review. Data sources Medline. Study selection Randomised controlled trials published in 118 core clinical journals (defined by the National Library of Medicine) in 2007. 1140 study reports in a 1:1 ratio by high (five general medicine journals with largest number of total citations in 2007) versus lower impact journals, were randomly sampled. Two reviewers, independently and in duplicate, used standardised, piloted forms to screen study reports for eligibility and to extract data. They also used explicit criteria to determine whether a randomised controlled trial reported subgroup analyses. Logistic regression was used to examine the association of prespecified study characteristics with reporting versus not reporting of subgroup analyses. Results 469 randomised controlled trials were included, of which 207 (44%) reported subgroup analyses. High impact journals (adjusted odds ratio 2.64, 95% confidence interval 1.62 to 4.33), non-surgical (versus surgical) trials (2.10, 1.26 to 3.50), and larger sample size (3.38, 1.64 to 6.99) were associated with more frequent reporting of subgroup analyses. The strength of association between trial funding and reporting of subgroups differed in trials with and without statistically significant primary outcomes (interaction P=0.02). In trials without statistically significant results for the primary outcome, industry funded trials were more likely to report subgroup analyses (2.29, 1.30 to 4.72) than non-industry funded trials. This was not true for trials with a statistically significant primary outcome (0.79, 0.46 to 1.36). Industry funded trials were associated with less frequent prespecification of subgroup hypotheses (31.3% v 38.0%, adjusted odds ratio 0.49, 0.26 to 0.94), and less use of the interaction test for analyses of subgroup effects (41.4% v 49.1%, 0.52, 0.28 to 0.97) than non-industry funded trials. Conclusion Industry funded randomised controlled trials, in the absence of statistically significant primary outcomes, are more likely to report subgroup analyses than non-industry funded trials. Industry funded trials less frequently prespecify subgroup hypotheses and less frequently test for interaction than non-industry funded trials. Subgroup analyses from industry funded trials with negative results for the primary outcome should be viewed with caution.