Jason Rho

A Decade of Reversal: An Analysis of 146 Contradicted Medical Practices

OBJECTIVE To identify medical practices that offer no net benefits. METHODS We reviewed all original articles published in 10 years (2001-2010) in one high-impact journal. Articles were classified on the basis of whether they addressed a medical practice, whether they tested a new or existing therapy, and whether results were positive or negative. Articles were then classified as 1 of 4 types: replacement, when a new practice surpasses standard of care; back to the drawing board, when a new practice is no better than current practice; reaffirmation, when an existing practice is found to be better than a lesser standard; and reversal, when an existing practice is found to be no better than a lesser therapy. This study was conducted from August 1, 2011, through October 31, 2012. RESULTS We reviewed 2044 original articles, 1344 of which concerned a medical practice. Of these, 981 articles (73.0%) examined a new medical practice, whereas 363 (27.0%) tested an established practice. A total of 947 studies (70.5%) had positive findings, whereas 397 (29.5%) reached a negative conclusion. A total of 756 articles addressing a medical practice constituted replacement, 165 were back to the drawing board, 146 were medical reversals, 138 were reaffirmations, and 139 were inconclusive. Of the 363 articles testing standard of care, 146 (40.2%) reversed that practice, whereas 138 (38.0%) reaffirmed it. CONCLUSION The reversal of established medical practice is common and occurs across all classes of medical practice. This investigation sheds light on low-value practices and patterns of medical research.

A Clinical Trial Comparing Physician Prompting With an Unprompted Automated Electronic Checklist to Reduce Empirical Antibiotic Utilization

Objectives:To determine whether face-to-face prompting of critical care physicians reduces empirical antibiotic utilization compared to an unprompted electronic checklist embedded within the electronic health record. Design:Random allocation design. Setting:Medical ICU with high-intensity intensivist coverage at a tertiary care urban medical center. Patients:Two hundred ninety-six critically ill patients treated with at least 1 day of empirical antibiotics. Interventions:For one medical ICU team, face-to-face prompting of critical care physicians if they did not address empirical antibiotic utilization during a patient’s daily rounds. On a separate medical ICU team, attendings and fellows were trained once to complete an electronic health record-embedded checklist daily for each patient, including a question asking whether listed empirical antibiotics could be discontinued. Measurements and Main Results:Prompting led to a more than four-fold increase in discontinuing or narrowing of empirical antibiotics compared to use of the electronic checklist. Prompted group patients had a lower proportion of patient-days on which empirical antibiotics were administered compared to electronic checklist group patients (63.1% vs 70.0%, p = 0.002). Mean proportion of antibiotic-days on which empirical antibiotics were used was also lower in the prompted group, although not statistically significant (0.78 [0.27] vs 0.83 [0.27], p = 0.093). Each additional day of empirical antibiotics predicted higher risk-adjusted mortality (odds ratio, 1.14; 95% CI, 1.05–1.23). Risk-adjusted ICU length of stay and hospital mortality were not significantly different between the two groups. Conclusions:Face-to-face prompting was superior to an unprompted electronic health record-based checklist at reducing empirical antibiotic utilization. Sustained culture change may have contributed to the electronic checklist having similar empirical antibiotic utilization to a prompted group in the same medical ICU 2 years prior. Future studies should investigate the integration of an automated prompting mechanism with a more generalizable electronic health record-based checklist.

The Diagnosis and Treatment of Pulmonary Embolism: A Metaphor for Medicine in the Evidence-Based Medicine Era

BACKGROUND The history of pulmonary embolism (PE) provides a fascinating portrait of a well-established diagnosis and standard of care treatment moving into the age of evidence-based medicine. METHODS We examined the history of PE and the practice of treating PE with anticoagulation. RESULTS Pulmonary embolism is a diagnostic category whose definition and treatment have both changed in the past century. Initially, PE was recognizable only when massive, with the signs and symptoms of right heart failure. Anticoagulants were established as the cornerstone of PE management with a single randomized controlled trial of 35 patients in 1960 and based on commonsense pathophysiologic reasoning. Since then, the diagnostic category of PE has been broadened, and the advent of computed tomography pulmonary angiography has yielded nearly a doubling of the incidence of the disease, without a concordant decrease in mortality. Although anticoagulation remains the cornerstone of management, open questions remain: what end points are altered by anticoagulation? What is the number needed to treat? CONCLUSIONS Trials of newer anticoagulants and longer durations of anticoagulation have not yielded real improvements over heparin, inviting doubts regarding its efficacy. Thus, PE is the quintessential diagnosis of medicine not because it represents our greatest success, but because it captures all the complexity of medicine in the evidence-based era. It may serve as a metaphor for many other conditions in medicine, including coronary artery disease. New trials in the field continue to test trivialities, whereas fundamental questions are unanswered.

Counterpoint: Were Industry-Sponsored Roflumilast Trials Appropriate? No

An appropriate study design is always contextual. Small, uncontrolled studies are useful in ascertaining the toxicity of a novel agent (phase 1) or even whether a drug has clinical activity (phase 2). Randomized trials are needed to confirm the efficacy of therapies, and mega-trials are required to determine mortality effects for blockbuster medications, whose efficacy data only concern subjective end points.1 When it comes to the clinical care of patients, however, an appropriate study must answer only one question: Does this drug improve quality or quantity of life beyond the best available therapy? Industry-sponsored studies of roflumilast do not meet this mark. Roflumilast, a novel phosphodiesterase-4 inhibitor, gained US Food and Drug Administration (FDA) approval for severe COPD in 2012.2 After just 4 months of promotional activity, roflumilast (Daliresp) had been prescribed approximately 70,000 times, with primary care doctors and other health-care providers writing 78% of scripts (specialists wrote 22%).3 The surge in the popularity of roflumilast, however, cannot be attributed to its evidence base. Three papers from The Lancet (containing five randomized controlled trials) are the best studies to date on roflumilast, but all, unfortunately, fail to answer meaningful clinical questions. The earliest study, by Rabe and colleagues,4 published in 2005, assigned 1,411 patients with a diagnosis of COPD for at least 1 year to roflumilast 250 μg, roflumilast 500 μg, or placebo. Besides short-acting β-agonists and short-acting anticholinergics, all other respiratory medicines were stopped, including long-acting β-agonists (LABAs) and inhaled corticosteroids (ICSs). The trial’s authors conclude that the study showed improvements in FEV1 and the number of patients with exacerbations, but no change in quality-of-life scores. Others, however, have questioned the basic statistical analysis5 used by the authors and show that simple (χ2 test) comparisons of the number of patients with exacerbations do not yield significant results. Additionally, new COPD medications should show benefit over current and maximal therapy,6 and the 2005 study did not attempt to do this. Finally, nearly 20% of patients discontinued the study medication, almost double that of the control group, with diarrhea being the most common side effect.7 Thus, given real side effects, a questionable benefit, and a starting point that was unfair (withholding existing therapy), the case for roflumilast was not made in 2005. Accordingly, at this time, the FDA failed to approve the drug.8 By the time the next two pivotal papers were published on roflumilast, there were significant advances in COPD research. In 2007, the Towards a Revolution in COPD Health (TORCH9) study compared salmeterol plus an ICS (fluticasone) against each component alone and placebo in a trial whose primary end point was mortality. In the TORCH trial, patients were allowed to use all other drugs besides the classes of drug being intervened upon (ICSs and LABAs). The TORCH trial found that the combination of salmeterol and fluticasone decreased exacerbations, improved spirometric function, and exhibited a strong trend toward overall mortality benefit (17.5% relative risk reduction) compared with patients taking placebo. In 2008, the Understanding Potential Long-Term Impacts on Function with Tiotropium10 (UPLIFT) trial tested tiotropium against placebo in patients with COPD who were permitted to use all respiratory medications except inhaled anticholinergics. The majority (60%) of these patients used LABAs at baseline. The UPLIFT trial found improvements in lung function, quality of life, exacerbations, and a trend toward improved survival. Against this backdrop, the next two roflumilast papers emerge. The Calverley et al11 study contains two trials and is listed as pivotal in the FDA application of the drug.12 Trial sponsors randomized > 3,000 patients with COPD, bronchitis symptoms, a history of exacerbations, and severe airflow limitation to roflumilast or placebo. Although β-agonists and short-acting anticholinergics were allowed, long-acting muscarinic antagonists (such as tiotropium) and inhaled corticosteroids were not. The trial showed a 17% decrease in the rate of moderate to severe COPD exacerbations (from 1.37 to 1.14 per patient per year), but that was largely driven by a decrease in moderate exacerbations, defined as requiring oral or IV steroids. Severe exacerbations, those leading to admission or death, were not statistically different. In summary, the pivotal roflumilast studies prevented patients from using one class of drugs that is of benefit (long-acting muscarinic antagonists) and halted another that is standard of care (ICSs). Regardless of the results, the trial does not inform real-world practice. If the authors wanted to enroll patients who could not tolerate tiotropium, that would be a separate hypothesis, but requiring patients to stop this drug and ICSs raises serious doubts about the generalizability of the trial. The third high-impact paper on roflumilast13 again contained two randomized trials, which examined > 1,500 patients with moderate to severe COPD based on spirometry. In one study, roflumilast was added to salmeterol. In the other, it was added to tiotropium. The authors note, “no inhaled corticosteroids, short acting anticholinergic drugs, other long acting bronchodilator drugs, theophylline, or other respiratory drugs were allowed after study enrollment.”13 Out of the many end points examined, there were some improvements, but with rules like this, do the results even matter? Whose clinical practice does this trial inform? Thus far, we have ignored the implications of withholding ICSs in the roflumilast trials. Although the data for the use of ICSs have been disputed,14 it is worth highlighting that regardless of its purported efficacy, analysis of randomized controlled trial data suggests that the abrupt discontinuation of ICSs worsens exacerbation.15 Thus, in the roflumilast studies that show some marginal improvement in exacerbations, we might ask whether the drug offers benefit or merely temporizes harm. Altogether, the roflumilast studies enrolled 9,394 adult patients.16 Together, nearly 10,000 patients gave their time and energy to take part in inconclusive medical research—and not just because the results were ambiguous, but because the trials were flawed from the start. COPD has been called a “neglected epidemic,”17 but, if the trend we have examined continues, it may be transformed into a manipulated one. The impact of industry-sponsored studies has been investigated in countless publications. Two systematic reviews18,19 find industry-sponsored studies are more likely to reach pro-industry conclusions than those by nonconflicted bodies. Industry-sponsored studies are more likely to use inactive controls.18 Thus, such analyses use straw man opponents. However, regarding study quality, industry-sponsored studies have not been seriously faulted.18 At least four analyses have found industry-sponsored studies to be of comparable quality to non-industry-sponsored ones.20-23 The roflumilast trials we have discussed may make sense of previous negative findings. Industry-sponsored studies have typically been compared against nonprofit studies20 using validated rating systems. However, such scales are insensitive tests of study quality. The Jadad scale,24 for instance, consists of three questions: Was the study described as randomized? Was the study described as double-blind? Was there a description of withdrawals and dropouts? All of the roflumilast studies we examined achieve top marks by this metric. The studies are nevertheless of limited usefulness, not because they defy basic rules governing randomized controlled trials but by virtue of asking questions of little value and restricting the use of alternative standards of care. The FDA’s approval letter16 for roflumilast reminds the manufacturers of their post-marketing commitment to “conduct a controlled clinical trial to evaluate the efficacy of roflumilast as an add-on therapy to a LABA and inhaled corticosteroid…in the population of COPD patients for which Roflumilast is indicated…[and] appropriate to demonstrate a clinically relevant beneficial effect…”. In a way, this statement is a commentary on the poor evidence base for roflumilast. Although randomized trials examined > 9,000 patients, a basic request to show efficacy is relegated to a post-marketing commitment. The roflumilast studies can be thought of as seeding trials, studies that “appear as if they answer a scientific question but primarily fulfill marketing objectives.”25 We have yet to see a single appropriate trial of this novel agent.

Publication Trends Among Internal Medicine Residents and Graduates

Internal medicine residency programs must teach research principles and residents must engage in scholarship, in accordance with current Accreditation Council for Graduate Medical Education (ACGME) program requirements, enacted in 1994. Inadequate demonstraion of this requirement is among the most frequent easons programs are cited by ACGME. Despite the strength, duration, and enforcement of this requirement, little is known regarding to what degree residents participate in research and whether the principles taught in residency continue through fellowship and beyond. One survey of 390 program directors found that 25% of residents at university programs completed case reports and that 10% published articles in peer-reviewed journals during residency. Another survey of residents, presenting scholarly work at a conference, found that 39 of 73 residents (53%) presented a clinical vignette and that 34 of 73 residents (47%) displayed a research abstract. An earlier survey of program directors estimated that 62% of their residents would complete an acceptable research project in 1 academic year. Each of these previous studies has limitations. All 3 are small cross-sectional surveys with variable response rates. Some required reporting of research endeavors y program directors, which may be inaccurate or

Can a resident's publication record predict fellowship publications?

Background Internal medicine fellowship programs have an incentive to select fellows who will ultimately publish. Whether an applicants publication record predicts long term publishing remains unknown. Methods Using records of fellowship bound internal medicine residents, we analyzed whether publications at time of fellowship application predict publications more than 3 years (2 years into fellowship) and up to 7 years after fellowship match. We calculate the sensitivity, specificity, positive and negative predictive values and likelihood ratios for every cutoff number of application publications, and plot a receiver operator characteristic curve of this test. Results Of 307 fellowship bound residents, 126 (41%) published at least one article 3 to 7 years after matching, and 181 (59%) of residents do not publish in this time period. The area under the receiver operator characteristic curve is 0.59. No cutoff value for application publications possessed adequate test characteristics. Conclusion The number of publications an applicant has at time of fellowship application is a poor predictor of who publishes in the long term. These findings do not validate the practice of using application publications as a tool for selecting fellows.

Rebuttal from Dr Rho et al.

We appreciate the thoughtful comments of Drs Suissa and Rabe.1 In many respects, regarding the specifics of the roflumilast trials, there is little disagreement. We concur with the authors that roflumilast has “rather frequent side effects, which have led to significant premature discontinuations in clinical trials.” A Cochrane meta-analysis confirms that patients receiving roflumilast are more likely to experience diarrhea, nausea, and headache and more likely to discontinue medication because of adverse events than patients receiving placebo.2 Nevertheless, there are some contrasts and clarifications worth noting. First, we do not fault the roflumilast studies because they are industry sponsored. We fault them because of their heavily restricted inclusion and exclusion criteria and, in certain cases, reliance on end points of dubious clinical significance. That they are industry sponsored may provide some explanation for why large, costly clinical trials would be conducted that avoid pragmatic questions. Second, one point of disagreement is the role of cessation of inhaled corticosteroids (ICSs). Drs Suissa and Rabe1 argue that, “cessation of ICSs or other drugs at randomization should be expected to have a similar effect in both the roflumilast and placebo groups.” However, it is entirely possible that the roflumilast temporizes the harm of ICS discontinuation. This biologic effect would be distinct from the question of whether the drug provides benefit beyond ICS. Third, the Cochrane analysis demonstrates that roflumilast has little impact on quality of life or symptoms. No improvement in mortality was appreciated, though the authors confirm the drug decreases exacerbations.2 Additionally, in that report, roflumilast was linked to an increase in psychiatric disturbances, which another analysis confirms.2 Other groups have also found an increase in atrial fibrillation among roflumilast users.3 Finally, it appears future roflumilast trials may be better but remain far from good. The forthcoming Roflumilast in the Prevention of COPD Exacerbations While Taking Appropriate Combination Treatment (REACT) trial will randomize patients already on a long-acting β-agonist/ICS combination to roflumilast or placebo. Patients are allowed to use long-acting muscarinic antagonist medication, with short-acting β-agonist as rescue. However, the protocol is unclear as to whether short-acting antimuscarinics, such as ipratropium, will be permitted. Notably, the inclusion criteria of the REACT study mandate that patients must have had at least two COPD exacerbations in the past 12 months. Over the same period, these patients must have remained on stable doses of long-acting β-agonist/ICS and long-acting muscarinic antagonist medication (despite those two exacerbations).4 Additionally, the REACT study uses a 4-week placebo run-in period. If during this time a patient has an exacerbation, he/she cannot be randomized until symptoms cease. If medication compliance is < 80%, the patient cannot be randomized. All of these choices in trial design undermine generalizability. The results of the REACT study will be applicable only to patients who have had two more COPD exacerbations in the past year, but none in the past month, with no changes to the dose of key therapies as a result. The question will again arise if this trial is appropriate, or instead whether inclusion and exclusion criteria are so limiting that the trial fails to answer a meaningful question.