Thanh H. Neville

Validity and Reliability of Value Assessment Frameworks for New Cancer Drugs

BACKGROUND Several organizations have developed frameworks to systematically assess the value of new drugs. These organizations include the American Society of Clinical Oncology (ASCO), the European Society for Medical Oncology (ESMO), the Institute for Clinical and Economic Review (ICER), and the National Comprehensive Cancer Network (NCCN). OBJECTIVES To understand the extent to which these four tools can facilitate value-based treatment decisions in oncology. METHODS In this pilot study, eight panelists conducted value assessments of five advanced lung cancer drugs using the ASCO, ESMO, and ICER frameworks. The panelists received instructions and published clinical data required to complete the assessments. Published NCCN framework scores were abstracted. The Kendalls W coefficient was used to measure convergent validity among the four frameworks. Intraclass correlation coefficients were used to measure inter-rater reliability among the ASCO, ESMO, and ICER frameworks. Sensitivity analyses were conducted. RESULTS Drugs were ranked similarly by the four frameworks, with Kendalls W of 0.703 (P = 0.006) across all the four frameworks. Pairwise, Kendalls W was the highest for ESMO-ICER (W = 0.974; P = 0.007) and ASCO-NCCN (W = 0.944; P = 0.022) and the lowest for ICER-NCCN (W = 0.647; P = 0.315) and ESMO-NCCN (W = 0.611; P = 0.360). Intraclass correlation coefficients (confidence interval [CI]) for the ASCO, ESMO, and ICER frameworks were 0.786 (95% CI 0.517-0.970), 0.804 (95% CI 0.545-0.973), and 0.281 (95% CI 0.055-0.799), respectively. When scores were rescaled to 0 to 100, the ICER framework provided the narrowest band of scores. CONCLUSIONS The ASCO, ESMO, ICER, and NCCN frameworks demonstrated convergent validity, despite differences in conceptual approaches used. The ASCO inter-rater reliability was high, although potentially at the cost of user burden. The ICER inter-rater reliability was poor, possibly because of its failure to distinguish differential value among the sample of drugs tested. Refinements of all frameworks should continue on the basis of further testing and stakeholder feedback.

Measuring the Value of New Drugs: Validity and Reliability of 4 Value Assessment Frameworks in the Oncology Setting

BACKGROUND Several organizations have developed frameworks to systematically assess the value of new drugs. OBJECTIVE To evaluate the convergent validity and interrater reliability of 4 value frameworks to understand the extent to which these tools can facilitate value-based treatment decisions in oncology. METHODS Eight panelists used the American Society of Clinical Oncology (ASCO), European Society for Medical Oncology (ESMO), Institute for Clinical and Economic Review (ICER), and National Comprehensive Cancer Network (NCCN) frameworks to conduct value assessments of 15 drugs for advanced lung and breast cancers and castration-refractory prostate cancer. Panelists received instructions and published clinical data required to complete the assessments, assigning each drug a numeric or letter score. Kendalls Coefficient of Concordance for Ranks (Kendalls W) was used to measure convergent validity by cancer type among the 4 frameworks. Intraclass correlation coefficients (ICCs) were used to measure interrater reliability for each framework across cancers. Panelists were surveyed on their experiences. RESULTS Kendalls W across all 4 frameworks for breast, lung, and prostate cancer drugs was 0.560 (P= 0.010), 0.562 (P = 0.010), and 0.920 (P < 0.001), respectively. Pairwise, Kendalls W for breast cancer drugs was highest for ESMO-ICER and ICER-NCCN (W = 0.950, P = 0.019 for both pairs) and lowest for ASCO-NCCN (W = 0.300, P = 0.748). For lung cancer drugs, W was highest pairwise for ESMO-ICER (W = 0.974, P = 0.007) and lowest for ASCO-NCCN (W = 0.218, P = 0.839); for prostate cancer drugs, pairwise W was highest for ICER-NCCN (W = 1.000, P < 0.001) and lowest for ESMO-ICER and ESMO-NCCN (W = 0.900, P = 0.052 for both pairs). When ranking drugs on distinct framework subdomains, Kendalls W among breast cancer drugs was highest for certainty (ICER, NCCN: W = 0.908, P = 0.046) and lowest for clinical benefit (ASCO, ESMO, NCCN: W = 0.345, P = 0.436). Among lung cancer drugs, W was highest for toxicity (ASCO, ESMO, NCCN: W = 0. 944, P < 0.001) and lowest for certainty (ICER, NCCN: W = 0.230, P = 0.827); and among prostate cancer drugs, it was highest for quality of life (ASCO, ESMO: W = 0.986, P = 0.003) and lowest for toxicity (ASCO, ESMO, NCCN: W = 0.200, P = 0.711). ICC (95% CI) for ASCO, ESMO, ICER, and NCCN were 0.800 (0.660-0.913), 0.818 (0.686-0.921), 0.652 (0.466-0.834), and 0.153 (0.045-0.371), respectively. When scores were rescaled to 0-100, NCCN provided the narrowest band of scores. When asked about their experiences using the ASCO, ESMO, ICER, and NCCN frameworks, panelists generally agreed that the frameworks were logically organized and reasonably easy to use, with NCCN rated somewhat easier. CONCLUSIONS Convergent validity among the ASCO, ESMO, ICER, and NCCN frameworks was fair to excellent, increasing with clinical benefit subdomain concordance and simplicity of drug trial data. Interrater reliability, highest for ASCO and ESMO, improved with clarity of instructions and specificity of score definitions. Continued use, analyses, and refinements of these frameworks will bring us closer to the ultimate goal of using value-based treatment decisions to improve patient care and outcomes. DISCLOSURES This work was funded by Eisai Inc. Copher and Knoth are employees of Eisai Inc. Bentley, Lee, Zambrano, and Broder are employees of Partnership for Health Analytic Research, a health services research company paid by Eisai Inc. to conduct this research. For this study, Cohen, Huynh, and Neville report fees from Partnership for Health Analytic Research. Outside of this study, Cohen receives grants and direct consulting fees from various companies that manufacture and market pharmaceuticals. Mei reports a grant from Eisai Inc. during this study. The other authors have no disclosures to report. Study concept and design were contributed by Bentley and Broder, with assistance from Elkin and Cohen. Bentley took the lead in data collection, along with Elkin, Huynh, Mukherjea, Neville, Mei, Popescu, Lee, and Zambrano. Data interpretation was performed by Bentley and Broder, along with Elkin, Cohen, Copher, and Knoth. The manuscript was written primarily by Bentley, along with Elkin and Broder, and revised by Bentley, Broder, Elkin, Cohen, Copher, and Knoth. Select components of this works methods were presented at ISPOR 19th Annual European Congress held in Vienna, Austria, October 29-November 2, 2016, and Society for Medical Decision Making 38th Annual North American Meeting held in Vancouver, Canada, October 23-26, 2016.

Differences between Attendings’ and Fellows’ Perceptions of Futile Treatment in the Intensive Care Unit at One Academic Health Center: Implications for Training

Purpose Knowing when patients are too ill to benefit from intensive care is essential for clinicians to recommend aggressive or palliative care as appropriate. To explore prognostic ability among critical care fellows, the authors compared fellows’ and attendings’ assessments of futile critical care and evaluated factors associated with assessments. Method Thirty-six attendings and 14 fellows in intensive care units at the University of California, Los Angeles, were surveyed daily for three months (December 2011–March 2012) to identify patients perceived as receiving futile treatment. Frequency of futile treatment assessments and reasons listed by attendings versus fellows were compared. Predictors of futile treatment assessments by provider type were assessed using multivariate probit models. Results Attendings made 6,897 assessments on 1,125 patients; fellows made 4,407 assessments on 773 patients. Fellows assessed 161 (20.8%) patients as receiving futile treatment, compared with attendings (123 [10.9%] patients, P < .001), and listed fewer reasons that treatment was futile (P < .001). Fellows were more likely to assess a patient as receiving futile treatment by the second day, whereas attendings took four days. Patients assessed as receiving futile treatment by fellows were less likely than patients so assessed by attendings to die in the hospital (51% versus 68%, P = .003) and within six months (62% versus 85%, P< .001). Conclusions Fellows made earlier assessments and judged more patients to be receiving futile treatment than attendings, and their assessments were less predictive of mortality, suggesting that assessment of treatment appropriateness develops with experience.

Blood Products Provided to Patients Receiving Futile Critical Care

&NA; The number of hospitalized patients receiving treatment perceived to be futile is not insignificant. Blood products are valuable resources that are donated to help others in need. We aimed to quantify the amount of blood transfused into patients who were receiving treatment that the critical care physician treating them perceived to be futile. During a 3‐month period, critical care physicians in 5 adult intensive care units completed a daily questionnaire to identify patients perceived as receiving futile treatment. Of 1136 critically ill patients, physicians assessed 123 patients (11%) as receiving futile treatment. Fifty‐nine (48%) of the 123 patients received blood products after they were assessed to be receiving futile treatment: 242 units of packed red blood cells (PRBCs) (7.6% of all PRBC units transfused into critical care patients during the 3‐month study period); 161 (9.9%) units of plasma, 137 (12.1%) units of platelets, and 21 (10.5%) units of cryoprecipitate. Explicit guidelines on the use of blood products should be developed to ensure that the use of this precious resource achieves meaningful goals.

Reply to: “Judgment of Futile Care in the ICU”

We thank Drs. Reddy, O’Horo, Hoskote, and Moua (1) for eloquently expressing their perspective regarding our publication on the opportunity costs of futile treatment in the ICU. Their primary concern is that the judgment of futility was made solely by the clinician and did not include the opinions of the patients and their families. We agree that the ideal paradigm of medicine is one of shared decision-making, where medical recommendations are discussed within the context of patients’ values and preferences. Far from advocating that patients’ and families’ opinions should be excluded, we believe that early frank discussion of prognosis and treatment options will lead to an improved match of treatments with patient values. However, this research recognizes what doctors in the foundational focus group told us: that there are some health states that critical care should not be used to perpetuate. Should critical care physicians not take responsibility for deciding when life-sustaining treatment is appropriate? A major part of a clinician’s training is learning which treatments are indicated—they decide when a patient should be hospitalized, what imaging studies are needed, when antibiotics should be started and stopped, whether a patient would benefit from surgery. Just as a family physician knows not to prescribe antibiotics for the common cold, the intensivist should avoid initiating dialysis on an imminently dying patient regardless of a family’s insistence. Clinicians have the responsibility to apply healthcare resources where they are indicated and to avoid the treatments that are unnecessary, non-beneficial, or harmful to the patient. The importance of critical care physicians carefully choosing which advanced, highly technical life-saving treatments are appropriate for which patients is perhaps best emphasized by considering the contrary circumstance: Families, imbued with no knowledge of medicine and encumbered by grief, choose whether patients receive mechanical ventilation, continuous veno-venous hemofiltration, and ventricular assist devices. Such a plan would not only be bad for patients, but would be an absurdly unfair use of health care resources. These decisions are inextricably linked with the physician’s responsibility for stewardship, and inappropriate critical care should not be provided simply because a patient or family “wants everything” (2). Thus, we argue that a clinician’s assessment and recognition of futility is critically important and necessary in order to re-direct some families and relieve suffering. Drs. Reddy et al also point out that the term “futile” is problematic because “many activities done in the context of critical illness serve quite well initial physiologic goals” without affecting survival. We agree that perhaps the term “inappropriate treatment” should replace “futile treatment” to avoid confusion with the concept of physiologic futility (3). The suggestion that the term “futile” should be avoided because Medicine is probabilistic at best is well taken. However, just as every patient has the right to hear the risks and benefits of any procedure, the critically ill patient and their family should be engaged in conversations regarding the likelihood that aggressive critical care would lead to a meaningful recovery. It is only then that a fully informed decision can be made.

Reply to “Of Course Futile Care Is Wasteful—Are WeReady to Act on this Knowledge?”

We thank Drs. Eiferman and Jones (1) for their thoughtful comments on our recent publication on the opportunity costs of futile treatment in the ICU. Providing futile treatment is wasteful, and we are pleased that our work has stimulated discussion about this difficult, uncomfortable, but exceedingly important issue. Regarding the question, “Is it time to take a harsher stance on futile care and tell families their loved ones is being transitioned to comfort care rather than asking?”, we must first acknowledge that the ideal paradigm in Medicine is one of shared decision-making(2) and many family conflicts can be resolved with improved communication(3). We are not able, at this time, to identify whether these processes were fully exhausted in the cases we studied. Additionally, further work is needed to develop tools that can prospectively identify patients who are at risk of receiving non-beneficial care, such that automatic interventions (such as palliative care involvement and scheduled family meetings) can be triggered to prevent futile treatment from occurring and therefore, improve the match of care with prognosis. Studies suggest that these measures can be effective(4,5). We agree that when such mechanisms fail, society at large and physicians in particular need to recognize that Medicine must be practiced responsibly and critical care is a resource that needs to be justly stewarded. The U.S. desperately needs an open, informed conversation about the goals of medicine and the societal trade-offs of using critical care to maintain people in severely compromised health states from which they cannot recover. Pragmatic, policy-level mechanisms that will support the cessation of futile treatment after thorough attempts at discussion, conflict resolution and due process are needed. Several institutions have long had such policies in place(6), and clinicians have the responsibility to activate these policies when necessary to avoid the treatments that are non-beneficial or harmful to the patient. A critical aspect of a doctor’s moral responsibility is to relieve suffering and to be able to recognize when Medicine and technology is overmastered by a patient’s condition. Avoiding these difficult conversations and conflict is a disservice: a disservice to the patient whose suffering is prolonged and dignity is compromised in the dying process, a disservice to the family whose unrealistic expectations are fueled by passivity and inadequate communication, a disservice to the medical profession where clinical acumen is cast aside and the tools we were trained with are no longer used to make a patient well, and a disservice to other patients who may be unable to access the health care they need. When faced with a family who “wants everything done,” declining to provide futile treatment is often an uphill climb, but abdicating the responsibility to use the tools of critical care medicine appropriately has steep consequences.