Mark R. Chassin

Is Volume Related to Outcome in Health Care? A Systematic Review and Methodologic Critique of the Literature

Measuring and understanding the association between volume and outcome in the delivery of health services has been the focus of much research since the 1980s (1, 2). Recently, purchasers and consumer groups have taken an interest in the wider public dissemination of data on hospital volume for specific surgical procedures and medical conditions (3-5). Because high volume has been associated with better health outcomes for a variety of procedures and conditions, these groups recommend that consumers or health plans use data on volume to choose high-quality hospitals and physicians. Because reliable, valid, and timely information about health outcomes is lacking, data on volume are even more attractive. The wisdom of this policy, however, depends critically on the strength, magnitude, and meaning of the association between volume and outcome in health care. During the past 20 years, a large body of research has focused on measuring and explaining the association between patient outcomes and the volume of health services provided by hospitals and physicians (1, 2, 6-11). Many [but not all] studies have documented that higher volume is associated with better outcomes. Two principal hypotheses have been advanced to explain these relationships: 1) Physicians (and hospitals) develop more effective skills if they treat more patients [practice makes perfect] or 2) physicians (and hospitals) achieving better outcomes receive more referrals and thus accrue larger volumes (selective referral) (2, 6, 9, 10). Past reviews of this subject have focused only on specific procedures or conditions (12-14) or on effects of hospital volume (12, 15), have included nonpopulation-based studies (14-16), or were done before many recent primary studies (12, 16). In addition, to our knowledge, no comprehensive review has systematically analyzed and described the methodologic rigor of this literature. This project was undertaken to address some of these limitations as part of a 2000 joint workshop of the U.S. Institute of Medicine and the National Cancer Policy Board (17). We set out to conduct a systematic review of the research evidence linking volume and outcome in health care, to summarize and describe the methodologic rigor of the existing literature, and to highlight the research and policy implications of these findings. Methods Study Identification and Selection We searched MEDLINE for English-language articles published between 1 January 1980 and 31 December 2000 that assessed the relationship between volume and outcome in health care. Our search algorithms combined the Medical Subject Headings (MeSH terms) and text words outcome, outcome assessment, outcome and process assessment, volume, utilization, frequency, statistics, and regionalization with names of clinical conditions and procedures (Appendix). Bibliographies were reviewed to identify other articles of interest, and experts were contacted about missed or unpublished studies. We also searched the Cochrane Collaboration database. For our systematic review, we selected English-language articles in which hospital or physician volume was an independent variable, health outcomes (for example, death, stroke, or clinical complications) were the dependent variable, samples were community- or population-based, and patients had primarily been treated since 1980. When multiple articles used the same data, we included the most recent or complete article. We excluded studies published before 1980 because clinical and surgical care and outcomes have changed considerably since then for nearly all of the topics we reviewed. We excluded studies from single institutions, voluntary registries, or other convenience samples because of concerns about reporting bias and generalizability. Our initial search also identified articles about volume and outcome in trauma, newborn intensive care, and organ transplantation. We did not include these topics in our review because these services have already been extensively regionalized or are otherwise highly regulated by government or other administrative bodies. Review Methods Two of the authors reviewed each article independently, using a standardized abstraction form to record study characteristics, methodologic attributes, and research results. Reviewers were not blinded to journal, authors, or findings. The degree of initial agreement between reviewers was very high ( = 0.82 to 1.0). Any discrepancies were resolved by discussion. A third reviewer was used in rare instances when the original two reviewers did not reach full agreement. We critically read each study with an eye toward assessing its methodologic rigor as well as describing its results. Our methodologic review focused on key design attributes that influence the validity and generalizability of systematic reviews and the studies on which they are based (18, 19). Some preexisting patient factors, such as disease severity and burden of comorbid illness, are clear determinants of outcome independent of volume. Because it is necessary to account for differences in such factors to make valid comparisons between high- and low-volume providers, we assessed the quality of risk-adjustment techniques (12, 14, 20). We considered four levels of risk adjustment (none, models based on administrative data alone, models that included some clinical data, and clinical models with good calibration [HosmerLemeshow test]) and discrimination (c-statistic 0.75) (20). In addition, we noted whether the investigators controlled for any specific processes of care (medical therapies, surgical techniques, or adjunctive procedures) that are known to result in better outcomes. Increasing evidence shows that high-volume providers may more consistently use proven effective therapies, such as aspirin or -blockers in myocardial infarction (21) or adjuvant therapy in breast cancer (22). We also noted whether the investigators had explicitly measured the appropriateness of patient selection for procedural conditions. Although there has been little investigation of this issue, one study of carotid endarterectomy found that high-volume surgeons had a higher proportion of operations for inappropriate indications compared with low-volume surgeons (23). We ascertained four other analytic attributes: the type of outcomes assessed, the number of hospitals and physicians in each study (an indicator of the potential for variation at the two provider levels of interest), the number of volume strata examined, and the unit of analysis (whether effects of hospital and physician volume were assessed). Finally, samples that included all patients undergoing a specific procedure were considered more generalizable to the overall population than studies restricted to patients with one type of insurance. Statistical Analysis Studies that found a statistically significant mortality advantage for high-volume providers (P < 0.05) were considered positive. For each study, we recorded the definition of high and low volume (cases per year) and the overall mortality rate of the sample. All summary death rates reflect in-hospital mortality unless otherwise specified. We calculated the absolute difference in mortality rate between the highest and lowest volume strata reported and expressed this as the number of excess deaths per 100 cases attributable to low volume. We report medians and ranges for high- and low-volume thresholds, mean mortality rates, and absolute differences in mortality rates. A few studies assessed volume as a continuous variable, making it impossible to calculate a mortality difference between high- and low-volume providers. We included data on whether these studies demonstrated statistically significant associations in our summary. We used chi-square tests to assess associations between positive findings and methodologic attributes. All analyses were done by using SAS software, version 6.12 (SAS Institute, Inc., Cary, North Carolina). Results Methodologic Characteristics of Included Studies Our systematic review identified 254 articles. Eighteen articles reported on more than one procedure or condition. Because we summarized the literature by topic, we evaluated each article that studied more than one procedure as more than one study. We reviewed a total of 272 studies, of which 135 (53%) met all inclusion criteria. These 135 studies covered 27 clinical topics. Studies were most commonly excluded because volume was not an independent variable (n = 43), the sample was not population based (n = 40), or the study data were obtained before 1980 (n = 29). Studies were also excluded if no primary health outcomes were assessed, if mixed outcomes were assessed (for example, death plus long length of stay), or if data were duplicative. We found the literature to be extremely heterogeneous, even for a given procedure, making formal meta-analysis impossible. The methodologic characteristics of the 135 studies we reviewed are described in Table 1. Overall, 124 studies were from the United States, 6 were from Canada, 3 were from Europe, and 2 were from the United Kingdom. Half of all the studies we identified were published since 1998, underscoring the growing interest in this topic. The vast majority of reports were based on state or national hospital-discharge databases and had sample sizes that exceeded 1000 and included 20 or more hospitals and 50 or more physicians. For 79% of studies, the primary outcome was death, usually inpatient death. Many studies, however, reported 30-day mortality rates, and several articles on cancer reported long-term survival. Twenty-one percent of studies measured more than just death. For example, all 12 studies of coronary angioplasty also reported rates of emergency coronary artery bypass surgery, and half of the investigations of carotid endarterectomy reported stroke or other neurologic outcomes. Ninety percent of studies analyzed at least three categories of volume, usually tertiles or quartiles, and a

Consensus methods: characteristics and guidelines for use.

Consensus methods are being used increasingly to solve problems in medicine and health. Their main purpose is to define levels of agreement on controversial subjects. Advocates suggest that, when properly employed, consensus strategies can create structured environments in which experts are given the best available information, allowing their solutions to problems to be more justifiable and credible than otherwise. This paper surveys the characteristics of several major methods (Delphi, Nominal Group, and models developed by the National Institutes of Health and Glaser) and provides guidelines for those who want to use the techniques. Among the concerns these guidelines address are selecting problems, choosing members for consensus panels, specifying acceptable levels of agreement, properly using empirical data, obtaining professional and political support, and disseminating results.

A method for the detailed assessment of the appropriateness of medical technologies.

The standard way to assess medical technologies is to conduct a randomized clinical trial. Patients are randomly assigned to groups receiving alternative treatments, and outcomes are monitored over a long period of time. For example, some victims of left main coronary artery disease may undergo coronary artery bypass surgery, and others may receive medical treatment with nitroglycerine and beta blockers. Comparison of five-year mortality and morbidity in the two groups helps to determine the relative appropriateness of the two procedures. In addition, information about quality of life and cost can also be collected and compared.

Variations in the Use of Medical and Surgical Services By the Medicare Population

We measured geographic differences in the use of medical and surgical services during 1981 by Medicare beneficiaries (age greater than or equal to 65) in 13 large areas of the United States. The average number of Medicare beneficiaries per site was 340,000. We found large and significant differences in the use of services provided by all medical and surgical specialties. Of 123 procedures studied, 67 showed at least threefold differences between sites with the highest and lowest rates of use. Use rates were not consistently high in one site, but rates for procedures used to diagnose and treat a specific disease varied together, as did alternative treatments for the same condition. These results cannot be explained by the actions of a small number of physicians. We do not know whether physicians in high-use areas performed too many procedures, whether physicians in low-use areas performed too few, or whether neither or both of these explanations are accurate. However, we do know that the differences are too large to ignore and that unless they are understood at a clinical level, uninformed policy decisions that have adverse effects on the health of the elderly may be made.

The Appropriateness of Carotid Endarterectomy

Abstract Carotid endarterectomy is a commonly performed but controversial procedure. We developed from the literature a list of 864 possible reasons for performing carotid endarterectomy, and asked a panel of nationally known experts to rate the appropriateness of each indication using a modified Delphi technique. On the basis of the panels ratings, we determined the appropriateness of carotid endarterectomy in a random sample of 1302 Medicare patients in three geographic areas who had had the procedure in 1981. Thirty-five percent of the patients in our sample had carotid endarterectomy for appropriate reasons, 32 percent for equivocal reasons, and 32 percent for inappropriate reasons. Of the patients having inappropriate surgery, 48 percent had less than 50 percent stenosis of the carotid artery that was operated on. Fifty-four percent of all the procedures were performed in patients without transient ischemic attacks in the carotid distribution. Of these procedures, 18 percent were judged appropriate,...

Accountability Measures — Using Measurement to Promote Quality Improvement

Measuring the quality of health care and using those measurements to promote improvements in the delivery of care, to influence payment for services, and to increase transparency are now commonplace. These activities, which now involve virtually all U.S. hospitals, are migrating to ambulatory and other care settings and are increasingly evident in health care systems worldwide. Many constituencies are pressing for continued expansion of programs that rely on quality measurement and reporting. In this article, we review the origins of contemporary standardized quality measurement, with a focus on hospitals, where such programs have reached their most highly developed state. We discuss some lessons learned from recent experience and propose a conceptual framework to guide future developments in this fast-moving field. Although many of the points we make are relevant to all kinds of quality measurement, including outcome measures, we focus our comments on process measures, both because these account for most of the measures in current use and because outcome measures have additional scientific challenges surrounding the need for case-mix adjustment. We write not as representatives of the Joint Commission articulating a specific new position of that group, but rather as individuals who have worked in the fields of quality measurement and improvement in a variety of roles and settings over many years.

Physician Ratings of Appropriate Indications for Six Medical and Surgical Procedures

We convened three panels of physicians to rate the appropriateness of a large number of indications for performing a total of six medical and surgical procedures. The panels followed a modified Delphi process. Panelists separately assigned initial ratings, then met in Santa Monica, California where they received reports showing their initial ratings and the distribution of the other panelists ratings. They discussed the indications and revised the indications lists, then individually assigned final ratings. There was generally better agreement on the final ratings than on the initial ratings. Based on reasonable criteria for agreement and disagreement, and excluding one outlying procedure, the panelists agreed on ratings for 42 to 56 per cent of the indications, and disagreed on 11 to 29 per cent.

Is Health Care Ready for Six Sigma Quality

Serious, widespread problems exist in the quality of U.S. health care: too many patients are exposed to the risks of unnecessary services; opportunities to use effective care are missed; and preventable errors lead to injuries. Advanced practitioners of industrial quality management, like Motorola and General Electric, have committed themselves to reducing the frequency of defects in their business processes to fewer than 3.4 per million, a strategy known as Six Sigma Quality. In health care, quality problems frequently occur at rates of 20 to 50 percent, or 200,000 to 500,000 per million. In order to approach Six Sigma levels of quality, the health care sector must address the underlying causes of error and make important changes: adopting new educational models; devising strategies to increase consumer awareness; and encouraging public and private investment in quality improvement.

High-Reliability Health Care: Getting There from Here

Context Despite serious and widespread efforts to improve the quality of health care, many patients still suffer preventable harm every day. Hospitals find improvement difficult to sustain, and they suffer “project fatigue” because so many problems need attention. No hospitals or health systems have achieved consistent excellence throughout their institutions. High-reliability science is the study of organizations in industries like commercial aviation and nuclear power that operate under hazardous conditions while maintaining safety levels that are far better than those of health care. Adapting and applying the lessons of this science to health care offer the promise of enabling hospitals to reach levels of quality and safety that are comparable to those of the best high-reliability organizations. Methods We combined the Joint Commissions knowledge of health care organizations with knowledge from the published literature and from experts in high-reliability industries and leading safety scholars outside health care. We developed a conceptual and practical framework for assessing hospitals’ readiness for and progress toward high reliability. By iterative testing with hospital leaders, we refined the framework and, for each of its fourteen components, defined stages of maturity through which we believe hospitals must pass to reach high reliability. Findings We discovered that the ways that high-reliability organizations generate and maintain high levels of safety cannot be directly applied to todays hospitals. We defined a series of incremental changes that hospitals should undertake to progress toward high reliability. These changes involve the leaderships commitment to achieving zero patient harm, a fully functional culture of safety throughout the organization, and the widespread deployment of highly effective process improvement tools. Conclusions Hospitals can make substantial progress toward high reliability by undertaking several specific organizational change initiatives. Further research and practical experience will be necessary to determine the validity and effectiveness of this framework for high-reliability health care.

The Ongoing Quality Improvement Journey: Next Stop, High Reliability

Quality improvement in health care has a long history that includes such epic figures as Ignaz Semmelweis, the nineteenth-century obstetrician who introduced hand washing to medical care, and Florence Nightingale, the English nurse who determined that poor living conditions were a leading cause of the deaths of soldiers at army hospitals. Systematic and sustained improvement in clinical quality in particular has a more brief and less heroic trajectory. Over the past fifty years, a variety of approaches have been tried, with only limited success. More recently, some health care organizations began to adopt the lessons of high-reliability science, which studies organizations such as those in the commercial aviation industry, which manage great hazard extremely well. We review the evolution of quality improvement in US health care and propose a framework that hospitals and other organizations can use to move toward high reliability.