Joshua D. Wallach

Reproducible Research Practices and Transparency across the Biomedical Literature.

There is a growing movement to encourage reproducibility and transparency practices in the scientific community, including public access to raw data and protocols, the conduct of replication studies, systematic integration of evidence in systematic reviews, and the documentation of funding and potential conflicts of interest. In this survey, we assessed the current status of reproducibility and transparency addressing these indicators in a random sample of 441 biomedical journal articles published in 2000–2014. Only one study provided a full protocol and none made all raw data directly available. Replication studies were rare (n = 4), and only 16 studies had their data included in a subsequent systematic review or meta-analysis. The majority of studies did not mention anything about funding or conflicts of interest. The percentage of articles with no statement of conflict decreased substantially between 2000 and 2014 (94.4% in 2000 to 34.6% in 2014); the percentage of articles reporting statements of conflicts (0% in 2000, 15.4% in 2014) or no conflicts (5.6% in 2000, 50.0% in 2014) increased. Articles published in journals in the clinical medicine category versus other fields were almost twice as likely to not include any information on funding and to have private funding. This study provides baseline data to compare future progress in improving these indicators in the scientific literature.

Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials

Importance Many published randomized clinical trials (RCTs) make claims for subgroup differences. Objective To evaluate how often subgroup claims reported in the abstracts of RCTs are actually supported by statistical evidence (P < .05 from an interaction test) and corroborated by subsequent RCTs and meta-analyses. Data Sources This meta-epidemiological survey examines data sets of trials with at least 1 subgroup claim, including Subgroup Analysis of Trials Is Rarely Easy (SATIRE) articles and Discontinuation of Randomized Trials (DISCO) articles. We used Scopus (updated July 2016) to search for English-language articles citing each of the eligible index articles with at least 1 subgroup finding in the abstract. Study Selection Articles with a subgroup claim in the abstract with or without evidence of statistical heterogeneity (P < .05 from an interaction test) in the text and articles attempting to corroborate the subgroup findings. Data Extraction and Synthesis Study characteristics of trials with at least 1 subgroup claim in the abstract were recorded. Two reviewers extracted the data necessary to calculate subgroup-level effect sizes, standard errors, and the P values for interaction. For individual RCTs and meta-analyses that attempted to corroborate the subgroup findings from the index articles, trial characteristics were extracted. Cochran Q test was used to reevaluate heterogeneity with the data from all available trials. Main Outcomes and Measures The number of subgroup claims in the abstracts of RCTs, the number of subgroup claims in the abstracts of RCTs with statistical support (subgroup findings), and the number of subgroup findings corroborated by subsequent RCTs and meta-analyses. Results Sixty-four eligible RCTs made a total of 117 subgroup claims in their abstracts. Of these 117 claims, only 46 (39.3%) in 33 articles had evidence of statistically significant heterogeneity from a test for interaction. In addition, out of these 46 subgroup findings, only 16 (34.8%) ensured balance between randomization groups within the subgroups (eg, through stratified randomization), 13 (28.3%) entailed a prespecified subgroup analysis, and 1 (2.2%) was adjusted for multiple testing. Only 5 (10.9%) of the 46 subgroup findings had at least 1 subsequent pure corroboration attempt by a meta-analysis or an RCT. In all 5 cases, the corroboration attempts found no evidence of a statistically significant subgroup effect. In addition, all effect sizes from meta-analyses were attenuated toward the null. Conclusions and Relevance A minority of subgroup claims made in the abstracts of RCTs are supported by their own data (ie, a significant interaction effect). For those that have statistical support (P < .05 from an interaction test), most fail to meet other best practices for subgroup tests, including prespecification, stratified randomization, and adjustment for multiple testing. Attempts to corroborate statistically significant subgroup differences are rare; when done, the initially observed subgroup differences are not reproduced.

Sex based subgroup differences in randomized controlled trials: empirical evidence from Cochrane meta-analyses

Objective To evaluate the frequency, validity, and relevance of statistically significant (P<0.05) sex-treatment interactions in randomized controlled trials in Cochrane meta-analyses. Design Meta-epidemiological study. Data sources Cochrane Database of Systematic Reviews (CDSR) and PubMed. Eligibility criteria for study selection Reviews published in the CDSR with sex-treatment subgroup analyses in the forest plots, using data from randomized controlled trials. Data extraction Information on the study design and sex subgroup data were extracted from reviews and forest plots that met inclusion criteria. For each statistically significant sex-treatment interaction, the potential for biological plausibility and clinical significance was considered. Results Among the 41 reviews with relevant data, there were 109 separate treatment-outcome analyses (“topics”). Among the 109 topics, eight (7%) had a statistically significant sex-treatment interaction. The 109 topics included 311 randomized controlled trials (162 with both sexes, 46 with males only, 103 with females only). Of the 162 individual randomized controlled trials that included both sexes, 15 (9%) had a statistically significant sex-treatment interaction. Of four topics where the first published randomized controlled trial had a statistically significant sex-treatment interaction, no meta-analyses that included other randomized controlled trials retained the statistical significance and no meta-analyses showed statistical significance when data from the first published randomized controlled trial were excluded. Of the eight statistically significant sex-treatment interactions from the overall analyses, only three were discussed by the CDSR reviewers for a potential impact on different clinical management for males compared with females. None of these topics had a sex-treatment interaction that influenced treatment recommendations in recent guidelines. UpToDate, an online physician-authored clinical decision support resource, suggested differential management of men and women for one of these sex-treatment interactions. Conclusion Statistically significant sex-treatment interactions are only slightly more frequent than what would be expected by chance and there is little evidence of subsequent corroboration or clinical relevance of sex-treatment interactions.

Research, Regulatory and Clinical Decision-Making: The Importance of Scientific Integrity

• There have been growing concerns about the reliability and validity of the underlying research that supports regulatory and clinical decision-making

Postmarket studies required by the US Food and Drug Administration for new drugs and biologics approved between 2009 and 2012: cross sectional analysis

Abstract Objectives To characterize postmarketing requirements for new drugs and biologics approved by the US Food and Drug Administration (FDA), and to examine rates and timeliness of registration, results reporting, and publication of required prospective cohort studies, registries, and clinical trials. Design Cross sectional analysis. Setting Postmarketing requirements for all new drugs and biologics approved by the FDA between 1 January 2009 and 31 December 2012, with follow-up up to 15 November 2017. Main outcome measures Postmarketing requirements and their characteristics known at the time of FDA approval, including FDA authority, study design, and study characteristics. Rates and timeliness of registration and results reporting on ClinicalTrials.gov and publication in peer reviewed journals of required prospective cohort studies, registries, and clinical trials. Results Between 2009 and 12, the FDA approved 97 new drugs and biologics for 106 indications with at least one postmarketing requirement at the time of first approval, for a total of 437 postmarketing requirements. Postmarket study descriptions were short (median word count 44 (interquartile range 29-71)) and often lacked information to determine an up to date progress (131 (30%)). 220 (50.3%) postmarketing requirements were for new animal or other studies (including pharmacokinetic studies); 134 (30.7%) were for prospective cohort studies, registries, and clinical trials; and 83 (19.0%) were for secondary analyses or follow-up studies. Of 110 clinical trials, 38 (34.5%), 44 (40.0%), 62 (56.4%), 66 (60.0%), and 98 (89.1%) did not report enough information to establish use of randomization, comparator type, allocation, outcome, and number of patients to be enrolled, respectively. Of 134 required prospective cohort studies, registries, and clinical trials, 102 (76.1%) were registered on ClinicalTrials.gov; of 50 registered and completed studies, 36 (72.0%) had reported results on ClinicalTrials.gov. Among 65 completed studies, 47 (72.3%) had either reported results or were published a median of 47 months (interquartile range 32-67) after FDA approval. 32 (68.1%) of these 47 studies did not report results publicly by the time of their original FDA report submission deadline. Conclusions Postmarketing requirements for new drugs and biologics were often briefly described and did not contain enough information to characterize study designs. Approximately three quarters of postmarketing requirements for prospective cohort studies, registries, and clinical trials were registered on ClinicalTrials.gov, and nearly three quarters of completed studies reported results or were published, suggesting that at least a quarter of these required studies are not being publicly disseminated.

Clinical Trial Evidence Supporting FDA Approval of Drugs Granted Breakthrough Therapy Designation

Clinical Trial Evidence Supporting FDA Approval of Drugs Granted Breakthrough Therapy Designation The US Food and Drug Administration (FDA) Safety and Innovation Act of 2012 created the Breakthrough Therapy designation to expedite development and review of drugs and biologics intended to treat serious or life-threatening conditions for which preliminary clinical evidence may demonstrate substantial improvement over existing therapies, allowing the FDA to provide “intensive guidance on efficient drug development” and “rolling review.”1 Although physicians and patients often perceive that breakthrough approvals are based on rigorous clinical evidence,2,3 no systematic evaluation of the evidence supporting breakthrough approvals has occurred. We reviewed all new FDA approvals granted Breakthrough Therapy designation, characterizing the pivotal clinical trials that serve as the basis of FDA approval,4 and premarket development and review times.

The US Food and Drug Administration’s expedited approval programs: Addressing premarket flexibility with enhanced postmarket evidence generation:

When the editors of Clinical Trials solicited our review on the U.S. Food and Drug Administration’s (FDA) expedited development and review programs, we anticipated there would be accompanying commentaries from other academics with differing opinions, or perhaps from the perspectives of industry or venture capital. We hardly expected to initiate a discussion among the former Commissioner of the FDA, the current Director of the Center for Drug Evaluation and Research (CDER), and the Chief Medical Officer of the American Society of Clinical Oncology on the advantages and disadvantages to patients and clinicians of FDA’s expedited approval programs. But we appreciate the opportunity to have done so. To be clear, the goal of our review was to engage in constructive dialogue, discussing the implications of expedited approval programs on premarket and postmarket evidence generation, highlighting some specific concerns, and offering our recommendations for robust medical product evaluations that ensure high-quality clinical evidence is available to inform patient care and clinical decision-making. The FDA faces the challenging task of striking the right balance between ensuring that novel therapeutics are safe and effective and allowing promising new drugs to enter the market as quickly as possible. We agree with Dr. Califf that FDA’s regulatory approach should not ‘revert back to the strategy of the 1970s’.2 Likely in response to the desires frequently expressed by patients and clinicians, the U.S. Congress has enacted laws requiring the FDA to develop expedited development and review pathways to accelerate the availability of novel therapeutics. Some of these pathways necessarily offer potential flexibility with respect to the evidentiary standards that are required to demonstrate medical product safety and effectiveness and secure approval. Accordingly, we believe there is a need for corresponding efforts to strengthen the clinical evidence that is generated after market approval.

Commentary: The challenge of evaluating and improving evidence when research lacks reproducibility

doi: 10.1093/ije/dyw338 Advance Access Publication Date: 6 February 2017 Commentary: The challenge of evaluating and improving evidence when research lacks reproducibility Joshua D Wallach and Sanjay Basu* Department of Health Research and Policy, Stanford University School of Medicine, Meta-Research Innovation Center at Stanford, Stanford University School of Medicine, Department of Medicine, Stanford Prevention Research Center, Stanford University, Stanford, CA, USA and Center for Primary Care, Harvard Medical School, Boston, MA, USA

Adherence to the International Committee of Medical Journal Editors’ (ICMJE) prospective registration policy and implications for outcome integrity: a cross-sectional analysis of trials published in high-impact specialty society journals

BackgroundRegistration of clinical trials is critical for promoting transparency and integrity in medical research; however, trials must be registered in a prospective fashion to deter unaccounted protocol modifications or selection of alternate outcomes that may enhance favorability of reported findings. We assessed adherence to the International Committee of Medical Journal Editors’ (ICMJE) prospective registration policy and identified the frequency of registrations occurring after potential observation of primary outcome data among trials published in the highest-impact journals associated with US professional medical societies. Additionally, we examined whether trials that are unregistered or registered after potential observation of primary outcome data were more likely to report favorable findings.MethodsWe conducted a retrospective, cross-sectional analysis of the 50 most recently published clinical trials that reported primary results in each of the ten highest-impact US medical specialty society journals between 1 January 2010 and 31 December 2015. We used descriptive statistics to characterize the proportions of trials that were: registered; registered retrospectively; registered retrospectively potentially after initial ascertainment of primary outcomes; and reporting favorable findings, overall and stratified by journal and trial characteristics. Chi-squared analyses were performed to assess differences in registration by journal and trial characteristics.ResultsWe reviewed 6869 original research reports published between 1 January 2010 and 31 December 2015 to identify a total of 486 trials across 472 publications. Of these 486 trials, 47 (10%) were unregistered. Among 439 registered trials, 340 (77%) were registered prospectively and 99 (23%) retrospectively. Sixty-seven (68%) of these 99 retrospectively registered trials, or 15% of all 439 registered trials, were registered after potential observation of primary outcome data ascertained among participants enrolled at inception. Industry-funded trials, those with enrollment sites in the US, as well as those assessing FDA-regulated interventions each had lower rates of retrospective registration. Unregistered trials were more likely to report favorable findings than were registered trials (89% vs. 64%; relative risk (RR) = 1.38, 95% confidence interval (CI) = 1.20–1.58; p = 0.004), irrespective of registration timing.ConclusionsAdherence to the ICMJE’s prospective registration policy remains sub-standard, even in the highest-impact journals associated with US professional medical societies. These journals frequently published unregistered trials and trials registered after potential observation of primary outcome data.

The US Food and Drug Administration’s expedited approval programs: Evidentiary standards, regulatory trade-offs, and potential improvements:

The US Food and Drug Administration has several regulatory programs and pathways to expedite the development and approval of therapeutic agents aimed at treating serious or life-debilitating conditions. A common feature of these programs is the regulatory flexibility, which allows for a customized approval approach that enables market authorization on the basis of less rigorous evidence, in exchange for requiring postmarket evidence generation. An increasing share of therapeutic agents approved by the Food and Drug Administration in recent years are associated with expedited programs. In this article, we provide an overview of the evidentiary standards required by the Food and Drug Administration’s expedited development and review programs, summarize the findings of the recent academic literature demonstrating some of the limitations of these programs, and outline potential opportunities to address these limitations. Recent evidence suggests that therapeutic agents in the Food and Drug Administration’s expedited programs are approved on the basis of fewer and smaller studies that may lack comparator groups and random allocation, and rather than focusing on clinical outcomes for study endpoints, rely instead on surrogate markers of disease. Once on the market, agents receiving expedited approvals are often quickly incorporated into clinical practice, and evidence generated in the postmarket period may not necessarily address the evidentiary limitations at the time of market entry. Furthermore, not all pathways require additional postmarket studies. Evidence suggests that drugs in expedited approval programs are associated with a greater likelihood that the Food and Drug Administration will take a safety action following market entry. There are several opportunities to improve the timeliness, information value, and validity of the pre- and postmarket studies of therapeutic agents receiving expedited approvals. When use of nonrandomized and uncontrolled studies cannot be avoided prior to market entry, randomized trials should be mandatory in the postmarket period, unless there are strong justifications for not carrying out such studies. In the premarket period, validity of the surrogate markers can be improved by more rigorously evaluating their correlation with patient-relevant clinical outcomes. Opportunities to reduce the duration, complexity, and cost of postmarket randomized trials should not compromise their validity and instead incorporate pragmatic “real-world” design elements. Despite recent enthusiasm for widely using real-world evidence, adaptive designs, and pragmatic trials in the regulatory setting, caution is warranted until large-scale empirical evaluations demonstrate their validity compared to more traditional trial designs.