Donald M. Berwick

Performance of a five-item mental health screening test

We compared the screening accuracy of a short, five-item version of the Mental Health Inventory (MHI-5) with that of the 18-item MHI, the 30-item version of the General Health Questionnaire (GHQ-30), and a 28-item Somatic Symptom Inventory (SSI-28). Subjects were newly enrolled members of a health maintenance organization (HMO), and the criterion diagnoses were those found through use of the Diagnostic Interview Schedule (DIS) in a stratified sample of respondents to an initial, mailed GHQ. To compare questionnaires, we used receiver operating characteristic analysis, comparing areas under curves through the method of Hanley and McNeil. The MHI-5 was as good as the MHI-18 and the GHQ-30, and better than the SSI-28, for detecting most significant DIS disorders, including major depression, affective disorders generally, and anxiety disorders. Areas under curve for the MHI-5 ranged from 0.739 (for anxiety disorders) to 0.892 (for major depression). Single items from the MHI also performed well. In this population, short screening questionnaires, and even single items, may detect the majority of people with DIS disorders while incurring acceptably low false-positive rates. Perhaps such extremely short questionnaires could more commonly reach use in actual practice than the longer versions have so far, permitting earlier assessment and more appropriate treatment of psychiatrically troubled patients in primary care settings.

Eliminating waste in US health care.

The need is urgent to bring US health care costs into a sustainable range for both public and private payers. Commonly, programs to contain costs use cuts, such as reductions in payment levels, benefit structures, and eligibility. A less harmful strategy would reduce waste, not value-added care. The opportunity is immense. In just 6 categories of waste--overtreatment, failures of care coordination, failures in execution of care processes, administrative complexity, pricing failures, and fraud and abuse--the sum of the lowest available estimates exceeds 20% of total health care expenditures. The actual total may be far greater. The savings potentially achievable from systematic, comprehensive, and cooperative pursuit of even a fractional reduction in waste are far higher than from more direct and blunter cuts in care and coverage. The potential economic dislocations, however, are severe and require mitigation through careful transition strategies.

The Science of Improvement

IN THE EARLY 1890S, DR WILLIAM HALSTED DEVELOPED radical mastectomy for breast cancer. Surgeons performed the Halsted procedure for more than 80 years even though there was little systematic evidence for its success. Then a new breed of scholars subjected the procedure to formal methods of evaluation unknown to Halsted. The methods—randomized controlled trials (RCTs) principal among them—led to a surprise: radical mastectomy had no advantage over simpler forms of treatment. This is but 1 example of the hard-won victory of evidence over belief in medicine. The pioneers of the formal evaluation of medical practices raised questions that traditional practitioners did not welcome. But in time, formal evaluation prevailed. The pioneers developed a hierarchy of evidentiary rigor relating the design of a study to the confidence that could be placed in the findings, from the lowly, nearly valueless anecdote to the royalty of evidence, the RCT. Concurrently, a similar story of hard-won learning unfolded in the so-called quality movement. Scholars illuminated the scale and types of defects in the processes of care and the outcomes, including high rates of unscientific care, inappropriate care, geographic variations in practice, latent disagreements among specialists, and oftenunrecognized medical injury to patients. Like the pioneers of evidence-based medicine, students of medical quality were at first largely ignored, but no longer. In 1999 and 2001, the Institute of Medicine published 2 landmark reports on the evidence for quality failures and called urgently for redesign of care systems to achieve improvements. The story could end here happily with 2 great streams of endeavor merging into a framework for conjoint action: improving clinical evidence and improving the process of care. Instead, the 2 endeavors are often in unhappy tension. Neither disputes that progress toward health care’s main goal, the relief of illness and pain, requires research of many kinds: basic, clinical, systems, epidemiologic. The disagreement centers on epistemology—ways to get at “truth” and how those ways should vary depending on the knowledge sought. Individuals most involved in day-to-day improvement work fear that if “evidence” is too narrowly defined and the approach to gathering evidence too severely constrained, progress may be the victim. For example, the RCT is a powerful, perhaps unequaled, research design to explore the efficacy of conceptually neat components of clinical practice—tests, drugs, and procedures. For other crucially important learning purposes, however, it serves less well. Recent controversies about the evaluation of rapid response teams provide a case in point. These controversies show the importance of adjusting research methods to fit research questions. Although only 10% to 15% of inpatients resuscitated outside intensive care units survive to hospital discharge, early warning signs are present in a large percentage of patients who ultimately experience cardiac arrest. Rapid response team systems bring expert clinicians to the bedsides of deteriorating patients before arrest occurs. In the mid 1990s, based largely on reports from Australian investigators, the Institute for Healthcare Improvement and others began introducing the concept to willing hospitals. Local experience strongly suggested that these systems often, although not always, were associated with improved outcomes, including reduced anxiety among nursing staff; increased interdisciplinary teamwork; decreased cardiac arrests outside of intensive care units; and, in some cases, declines in mortality. The evidence base took a turn in June 2005 with the publication of the Medical Early Response Intervention and Therapy (MERIT) Study, a cluster randomized prospective trial that claimed to find no beneficial effect of these teams on several primary outcomes. Controversy has continued since then regarding the scientific evidence for rapid response systems. In fact, the MERIT trial was not negative; it was inconclusive. The study team encountered an array of serious problems in execution, common in social science. For example, although the study’s power calculation assumed a baseline rate of 30 events per 1000 admissions, the actual rate proved to be fewer than 7 events per 1000 admissions; thus, the study was ef fect ively underpowered by 500%. Crosscontamination abounded; some control hospitals implemented rapid response protocols, and several study hospi-

A primer on leading the improvement of systems.

The nurse called me urgently into the room. The child, she said, was in acute respiratory distress. I had never met either Jimmy (the 6 year old boy) or his mother (an inner city single teenage parent) before. His asthma attack was severe, his peak expiratory flow rate only 35% of normal. Twenty years ago my next steps would have been to begin bronchodilator treatment, call an ambulance, and send the boy to hospital. That also would have been the story 10 years ago, or five, or two. But today, when I entered the room, the mother handed me her up to date list of treatments, including nebuliser treatment with β2 agonists, that she had administered with equipment that had been installed in her home. It continued with her graph of Jimmys slowly improving peak flow levels, which she had measured and charted at home, having been trained by the asthma outreach nurse. She then gave me the nurses cellular telephone number, along with a specific recommendation on the next medication to try for her son, one that had worked in the past but was not yet available for her to use at home. My reply was interrupted by a knock on my door. It was the chief of the allergy department in my health maintenance organisation. He worked one floor above me in the health centre and, having been phoned by the outreach nurse, had decided to “pop down” to see if he could help. He also handed me a phial of the same new medication that the mother had just mentioned, suggesting that we try it. Two hours later Jimmy was not in a hospital bed; he was at home breathing comfortably. Just to be safe the allergy nurse would be paying him a visit later that afternoon. …

Five System Barriers to Achieving Ultrasafe Health Care

Key Summary Points In health care, the premium placed on autonomy, the drive for productivity, and the economics of the system may lead to severe safety constraints and adverse medical events. Several key building blocks must be addressed before other solutions to the problem of unsafe medical care can be considered. Among these building blocks are the need to control maximum production, use of the equivalent actor principle, and the need for standardization of practices. Safety in health care depends more on dynamic harmony among actors than on reaching an optimum level of excellence at each separate organizational level. Open dialogue and explicit team training among health care professionals are key factors in establishing a shared culture of safety in health care. The notion of a 2-tiered system of medicine may evolve logically by distinguishing between health care sectors in which ultrasafety is achievable and sectors that are characterized by ambition, audacity, and aggressive efforts to rescue patients, in which greater risk is inherent in the goals. More than 5 years ago, the Institute of Medicine report To Err Is Human: Building a Safer Health System highlighted the need to make patient safety a major priority for health care authorities (1). Since then, the pressure to increase patient safety has continuously grown in western countries. Priority has focused on identifying and reducing preventable events. Important changes have already been made to the accident and incident reporting system, and the associated techniques of analysis (2-6). However, the upper limit of harm prevention is unclear (7). Many investigators have proposed that adapting the success strategies and tools of ultrasafe systems, such as those used in the aviation and nuclear power industries, will lead to comparable successes and safety outcomes in health care (8, 9). The reality is probably more complicated. Many complex industriesfor example, the chemical industry or road safetyhave adapted the safety tools of advanced systems and made important gains in the past 2 decades. However, the safety results from most of these efforts top out well before the level reached by the civil aviation and nuclear power industries (10). This limit does not seem to be due to insufficient tools, low competence among workers, or naive safety strategies. For the most part, it seems to be the consequence of a conscious tradeoff among safety goals, performance goals, and the organization of the specific profession. Becoming ultrasafe may require health care to abandon traditions and autonomy that some professionals erroneously believe are necessary to make their work effective, profitable, and pleasant. A comparative analysis of industry behavior demonstrates that becoming an ultrasafe provider requires acceptance of 5 overall types of constraints on activity. This analysis is based on the screening of various socio-technical professions, such as the aviation, nuclear power, chemical, and food industries; road transportation; and health care. The benchmark analysis aims to associate specific traits of these industries with their safety performance. We then describe 5 high-level organizational dimensions derived from the general literature on risk and safety (11-13), each of which is associated with a range of values: type of expected performance (from daily routine work to highly innovative, and standardized or repetitive), interface of health care providers with patients (from full autonomy to full supervision), type of regulations (from few recommendations to full specification of regulations at an international level), pressure for justice after an accident (from little judicial scrutiny to routine lawsuits against people and systems), and supervision and transparency by media and people in the street of the activity (from little concern to high demand for national supervision). We consider the value of a given dimension to become a barrier when it is present for all work situations that entail equal or less safety and it is absent for all work situations that entail greater safety. The barriers can be ranged along a safety axis by considering the average safety level of work situations that cannot cross each of these barriers. A barrier may be under partial control and therefore overlap other barriers that are also under partial control, making the relative effect of each barrier on the observed final level of safety more complex. We consider one barrier to be more constraining than another barrier when the maximum safety performance associated with no control at all is lower than that of another barrier. The barriers to safety that we discuss are fundamental, or root, barriers. Addressing each root barrier demands a substantial change in practice that entails considerable economic, political, and performance tradeoffs. Risk Assessment and Communication in Industrial Activities The overall safety profile of an industrial system is measured by reporting on the number of adverse events over a time interval (for example, an annual rate). The figures are generally weighted according to the volume of activity (such as number of miles traveled per year). The variable that is best for specifying the volume of activitythe denominator in a safety calculationis industry specific and is therefore poorly standardized across industries. For example, civil aviation uses 1 million departures as the relevant value to calculate volume of activity, whereas military aviation uses the number of flight hours. Reliable measurement of health care and patient safety outcomes is the first challenge for health care benchmarking (14, 15). In health care, the ethically compelling numerator is preventable harm. In many industries, the weighting process reflects how comfortable the organization or industry is with its risk exposure. For example, the risk for fatal accidents in road traffic, which is 1 of the top 3 causes of death in western countries, is often weighted by convenience of travel and the mileage traveled (16). Use of this denominator may lead to the perception that road transportation is a very safe domain compared with the alternatives. The unwary consumer of such risk reports may therefore erroneously interpret road travel as much safer than modes of travel for which risk is calculated on the basis of a much larger denominator, such as that used in aviation. In fact, air travel is far safer than road travel. We use the rate of catastrophic events per exposure among industrial and human endeavors as an anchor to allow comparison of accident rates across industries with those in health care (Figure 1). Viewed through this lens, accident rates in health care currently range from 101 to 107 events per exposure. This ratio is the most accessible and allows easier comparisons across industries. Figure 1. Average rate per exposure of catastrophes and associated deaths in various industries and human activities. In the civil aviation, nuclear power, and railway transport industries in Europe, the rate of catastrophic accidents per exposure, such as complete failure of an airplane engine leading to loss of aircraft, is better than 1 106. That is, the rate of death in these industries is less than 1 per million exposures. The rate of fatal adverse events among hospital patients is much greater but also varies by domain (1). In obstetrics, anesthesiology, or blood transfusion, the risk for fatal adverse events per exposure is less than 105 (17). Conversely, surgery has a total rate of fatal adverse events of almost 104 (14). Numerous investigators present this 104 risk for accident as an extrapolated average value in health care (18, 19). However, not all statistics have the same validity, because of differences in definitions and comprehensiveness in monitoring methods (20). Some statistics derive from large databases with objective assessment, whereas others derive from local estimates. The latter is particularly true for health care. The rates of adverse events are most likely reasonably convergent in the published literature, but some investigators have pointed out the importance of an accurate numerator and denominator in the calculation (21, 22). For our purposes, however, we believe that these variations do not deeply alter the proposed safety framework. We aim to reason more in terms of relative ranking rather than precise safety values. Moreover, our working hypothesis on the stability of the relative ranking is all the more reasonable because the industries from which we are inferring were chosen on the basis of separation by many logs of safety amplitude. What Are the Limits and Barriers to Achieving Safety in Medicine? Bearing in mind the caveats regarding calculation of risk, the risk for catastrophic events across industries differs greatly. Some sectors continue to have a low safety level (for example, transport by microlight aircraft or helicopter, and emergency surgery), some are stuck at an average safety level (for example, road safety and occupational accidents in trade or fishing), some are at good levels (for example, the petrochemical industry and anesthesiology), and the best have achieved safety levels beyond 106 (for example, the nuclear power and civil aviation industries). Five systemic successive barriers seem to characterize limitations in safety improvement. Barrier 1: Acceptance of Limitations on Maximum Performance The first barrier involves regulations that limit the level of risk allowed. This level is dictated in situations in which high levels of production and performance are also sought. When limits do not existthat is, the prevailing attitude is attain a specified high level of production, no matter what it takesthe system in question is very unsafe. When the maximum performance is unlimited and individuals or systems are allowed to make autonomous decisions without regulation or constraints, the risk for fatal events nears 1 102 p

Connections Between Quality Measurement and Improvement

Background.Measurement is necessary but not sufficient for quality improvement. Because the purpose of the national quality measurement and reporting system (NQMRS) is to improve quality, a discussion of the link between measurement and improvement is critical for ensuring an appropriate system design. Objectives.To classify approaches to the use of measurement in improvement into two different—although linked and potentially synergistic—agendas, or “pathways.” To discuss the barriers encountered in each of these pathways and identify steps needed to motivate improvement in both pathways. Research Design.Descriptive, conceptual discussion. Findings.The barriers to the use of information to motivate change include, in Pathway I (selection), the lack of skill, knowledge, and motivation on the part of those who could drive change by using data to choose from among competing providers, and, in Pathway II (change in care delivery), the deficiencies in organizational and professional capacity in health care to lead change and improvement itself. Conclusions.Neither the dynamics of selection nor the dynamics of improvement work reliably today. The barriers are not just in the lack of uniform, simple, and reliable measurements, they also include a lack of capacity among the organizations and individuals acting on both pathways.

Preferences for Health Outcomes: Comparison of Assessment Methods

This study compared standard gamble (SG), time trade-off (TTO), and category scaling (CS) methods for assessing preferences among hypothetical outcomes of coronary artery bypass surgery. High correlations among assessment methods, as found in some previous studies, do not assure the absence of systematic differences in rating obtained by different methods. This study used analysis of variance to test for differences among the three assessment methods. Questionnaire responses were obtained from 67 of 109 physicians participating in a postgraduate course on clinical decision making, following a lecture and workshop on utility theory. SG and CS were used to rate multivariate combinations of angina (none, moderate, and severe) and survival (0, 5, and 10 years); and SG, TTO, and CS were used to rate univariate outcomes with angina (none, moderate, and severe) for the remainder of their life expectancy. SG ratings were higher than TTO ratings, which were higher than CS ratings (p less than 0.001 for all comparisons). Multivariate responses revealed a significant interaction between angina and survival dimensions using CS, but not using SG. We conclude that these methods are not interchangeable and that differences between SG and CS require a more complex explanation than differences in attitude toward risk.

Transforming healthcare: a safety imperative

Ten years ago, the Institute of Medicine reported alarming data on the scope and impact of medical errors in the US and called for national efforts to address this problem. While efforts to improve patient safety have proliferated during the past decade, progress toward improvement has been frustratingly slow. Some of this lack of progress may be attributable to the persistence of a medical ethos, institutionalized in the hierarchical structure of academic medicine and healthcare organizations, that discourages teamwork and transparency and undermines the establishment of clear systems of accountability for safe care. The Lucian Leape Institute, established by the US National Patient Safety Foundation to provide vision and strategic direction for the patient safety work, has identified five concepts as fundamental to the endeavor of achieving meaningful improvement in healthcare system safety. These five concepts are transparency, care integration, patient/consumer engagement, restoration of joy and meaning in work, and medical education reform. This paper introduces the five concepts and illustrates the meaning and implications of each as a component of a vision for healthcare safety improvement. In future roundtable sessions, the Institute will further elaborate on the meaning of each concept, identify the challenges to implementation, and issue recommendations for policy makers, organizations, and healthcare professionals.

CONTROLLING VARIATION IN HEALTH CARE : A CONSULTATION FROM WALTER SHEWHART

The control of unintended variation is an objective central to modern industrial quality management methods, based largely on the theoretical work of Walter A. Shewhart. As industrial quality management techniques find their place in health care, professionals may feel threatened by the effort to reduce variation. Understanding may reduce this fear. Variation of the types addressed in quality control efforts erodes quality and reliability, and adds unnecessarily to costs. Such undesirable variation derives, for example, from misinterpretation of random noise in clinical data, from unreliability in the performance of clinical and support systems intended to support care, from habitual differences in practice style that are not grounded in knowledge or reason, and from the failure to integrate care across the boundaries of components of the health care system. Quality management efforts can successfully reduce each of these forms of variation without insult to the professional autonomy, dignity, or purpose of health care professionals. Professionals need to embrace the scientific control of variation in the service of their patients and themselves.

Variations in Rates of Hospitalization of Children in Three Urban Communities

Hospitalization accounts for a large portion of the expenditures for child health care, and differences in the rate of hospitalization may produce important variations in the cost of that care. We studied the rates of hospitalization in Boston, Rochester (N.Y.), and New Haven (Conn.) in 1982. We assigned the risk of hospitalization in Rochester a score of 1.00. Boston children were hospitalized at more than twice the rate of Rochester children for most medical diagnostic categories (relative risk, 2.65; 95 percent confidence interval, 2.53 to 2.78), and the rate for the New Haven group was intermediate (relative risk, 1.80; 95 percent confidence interval, 1.68 to 1.93). Rates of inpatient surgery differed less (Boston relative risk, 1.12; New Haven relative risk, 0.93). The relative risks of hospitalization (as compared with Rochester children) for Boston and New Haven children, respectively, were 3.8 and 2.3 for asthma, 6.1 and 2.9 for toxic ingestions, and 2.6 and 2.7 for head injuries. Fractures of the femur, appendicitis, and bacterial meningitis (conditions uniformly treated in the hospital) had similar rates of hospitalization across the three cities, but the relative risk of hospitalization for aseptic meningitis was 3.7 in Boston. The rates of hospitalization of children in all three communities were below the national averages in 1982. Although this study does not define the reasons for the variation in rates of hospitalization, it is possible that they were related in part to differences in socioeconomic status or access to primary care. The implications of these data for the cost and quality of pediatric care therefore remain to be determined.