Scott K. Aberegg

Empirical Consequences of Current Recommendations for the Design and Interpretation of Noninferiority Trials

BackgroundNoninferiority trials are increasingly common, though they have less standardized designs and their interpretation is less familiar to clinicians than superiority trials.ObjectiveTo empirically evaluate a cohort of noninferiority trials to determine 1) their interpretation as recommended by CONSORT, 2) choice of alpha threshold and its sidedness, and 3) differences between methods of analysis such as intention-to-treat and per-protocol.DesignWe searched MEDLINE for parallel-group randomized controlled noninferiority trials published in the five highest-impact general medical journals between 2011 and 2016.Main MeasuresData abstracted included trial design parameters, results, and interpretation of results based on CONSORT recommendations.Key ResultsOne hundred sixty-three trials and 182 noninferiority comparisons were included in our analysis. Based on CONSORT-recommended interpretation, 79% of experimental therapies met criteria for noninferiority, 13% met criteria for superiority, 20% were declared inconclusive, and 2% met criteria for inferiority. However, for 12% of trials, the experimental therapy was statistically significantly worse than the active control, but CONSORT recommended an interpretation of inconclusive or noninferior. A two-sided alpha equivalent of greater than 0.05 was used in 34% of the trials, and in five of these trials, the use of a standard two-sided alpha of 0.05 led to changes in the interpretation of results that disfavored the experimental therapy. In four of the five comparisons where different methods of analysis (e.g., intention-to-treat and per-protocol) yielded different results, the intention-to-treat analysis was the more conservative. In 11% of trials, a secondary advantage of the new therapy was neither reported nor could it be inferred by reviewers.ConclusionsIn this cohort, the design and interpretation of noninferiority trials led to significant and systematic bias in favor of the experimental therapy. Clinicians should exercise caution when interpreting these trials. Future trials may be more reliable if design parameters are standardized.

Effect sizes in ongoing randomized controlled critical care trials

BackgroundAn important limitation of many critical care trial designs is that they hypothesize large, and potentially implausible, reductions in mortality. Interpretation of trial results could be improved by systematic assessment of the plausibility of trial hypotheses; however, such assessment has not been attempted in the field of critical care medicine. The purpose of this study was to determine clinicians’ views about prior probabilities and plausible effect sizes for ongoing critical care trials where the primary endpoint is landmark mortality.MethodsWe conducted a systematic review of clinical trial registries in September 2015 to identify ongoing critical care medicine trials where landmark mortality was the primary outcome, followed by a clinician survey to obtain opinions about ten large trials. Clinicians were asked to estimate the probability that each trial would demonstrate a mortality effect equal to or larger than that used in its sample size calculations.ResultsEstimates provided by individual clinicians varied from 0% to 100% for most trials, with a median estimate of 15% (IQR 10–20%). The median largest absolute mortality reduction considered plausible was 4.5% (IQR 3.5–5%), compared with a median absolute mortality reduction used in sample size calculations of 5% (IQR 3.6–10%) (P = 0.27).ConclusionsFor some of the largest ongoing critical care trials, many clinicians regard prior probabilities as low and consider that plausible effects on absolute mortality are less than 5%. Further work is needed to determine whether pooled estimates obtained by surveying clinicians are replicable and accurate or whether other methods of estimating prior probability are preferred.

Design and Interpretation of Noninferiority Trials

A uthors’ reply: We are grateful for the correspondents’ interest in our descriptive study of noninferiority trials.1 Turgeon et al. have retrospectively extended the part of our analysis comparing ITT (intention-to-treat) and PP (perprotocol) methodologies to 2004. While we found that 64% of trials in our cohort from 2011 to 2016 reported both ITTand PP analyses, they found that only 45% of trials in the same journals between 2004 and 2014 reported both analyses. These results are consistent with those found in other reports, with modest increases over time in the reporting of both analyses. 2–4 Regarding discordant results between analyses, both of our results reinforce the observation that the theoretical claim that PP analyses are more conservative than ITT analyses is empirically ungrounded. Ideally, both PP and ITT methodologies should be carefully described and reported with an a priori plan for handling discordant results when they occur. Vach et al. highlight our finding that a fundamental ethical tenet of noninferiority trials, namely that the NT (new treatment) have some secondary benefit, is frequently unmet. We regret that we neglected to reference their systematic review of this specific issue, which showed similar results to ours. 5 In their study, 51% of trials explicitly reported the purported secondary advantages of the NT, and an additional 25% vaguely or indirectly mentioned secondary advantages. The corresponding percentages in our trial were 70 and 20%. In their article, Vach et al. propose the integration of measures of benefits and harms of NT versus active control in a manner consistent with expected utility theory. This approach, which can be adapted at the level of the patient or the population, holds promise for shared decision-making using the results of randomized trials of all types. Application of these methods will be facilitated by reliable estimates of effects from well designed and executed trials, and from a focus on actual confidence intervals of beneficial and harmful effects, rather than arbitrary and potentially misleading conclusions such as Bnoninferior^ or Binconclusive.^

Do non-inferiority trials of reduced intensity therapies show reduced effects? A descriptive analysis

Objectives To identify non-inferiority trials within a cohort where the experimental therapy is the same as the active control comparator but at a reduced intensity and determine if these non-inferiority trials of reduced intensity therapies have less favourable results than other non-inferiority trials in the cohort. Such a finding would provide suggestive evidence of biocreep in these trials. Design This metaresearch study used a cohort of non-inferiority trials published in the five highest impact general medical journals during a 5-year period. Data relating to the characteristics and results of the trials were abstracted. Primary outcome measures Proportions of trials with a declaration of superiority, non-inferiority and point estimates favouring the experimental therapy and mean absolute risk differences for trials with outcomes expressed as a proportion. Results Our search yielded 163 trials reporting 182 non-inferiority comparisons; 36 comparisons from 31 trials were between the same therapy at reduced and full intensity. Compared with trials not evaluating reduced intensity therapies, fewer comparisons of reduced intensity therapies demonstrated a favourable result (non-inferiority or superiority) (58.3%vs82.2%; P=0.002) and fewer demonstrated superiority (2.8%vs18.5%; P=0.019). Likewise, point estimates for reduced intensity therapies more often favoured active control than those for other trials (77.8%vs39.7%; P<0.001) as did mean absolute risk differences (+2.5% vs −0.7%; P=0.018). Conclusions Non-inferiority trials comparing a therapy at reduced intensity to the same therapy at full intensity showed reduced effects compared with other non-inferiority trials. This suggests these trials may have a high rate of type 1 errors and biocreep, with significant implications for the design and interpretation of future non-inferiority trials.

We Did This, and the Patient Improved: True, True, and Unrelated?

Anthony P Edwards, MD (1), Scott K Aberegg, MD, MPH (1) 1. Division of Pulmonary and Critical Care Medicine, University of Utah School of Medicine, Salt Lake City, Utah Corresponding Author: Scott K Aberegg, MD, MPH 1321 South 600 East SLC, UT 84105 scottaberegg@gmail.com Sources of Support: There was no support for this letter to the editor. Disclaimers: There are no disclaimers or conflicts of interest. Running Head: True, true, unrelated?

Cognitive Biases or Inadequate Searching

Affi liations: From the Hospital Provincial de la Misericordia (Dr S. Rodríguez Villar); Virgen de la Salud Hospital (Drs Sánchez Casado, Hortigüela Martín, Marco Schulke, and Barrientos Vega and Mss Mesa García and M. Rodríguez Villar); University Hospital La Paz (Dr Quintana); and Dana-Farber Cancer Institute and Harvard MedicalSchool (Dr Prigerson), Center for Psychooncology and Palliative Care Research, Harvard Medical School. Financial/nonfi nancial disclosures: The authors have reported to CHEST that no potential confl icts of interest exist with any com panies/organizations whose products or services may be discussed in this article . Correspondence to: Sancho Rodríguez Villar, LMS, Intensive Care Department, Hospital Ntra. Sña. Del Prado, Talavera de la Reina, Ctra, Madrid Km116, 45600, Toledo, Spain; e-mail: sancho.rodvil@hotmail.com