Elizabeth R. DeLong

Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach.

Methods of evaluating and comparing the performance of diagnostic tests are of increasing importance as new tests are developed and marketed. When a test is based on an observed variable that lies on a continuous or graded scale, an assessment of the overall value of the test can be made through the use of a receiver operating characteristic (ROC) curve. The curve is constructed by varying the cutpoint used to determine which values of the observed variable will be considered abnormal and then plotting the resulting sensitivities against the corresponding false positive rates. When two or more empirical curves are constructed based on tests performed on the same individuals, statistical analysis on differences between curves must take into account the correlated nature of the data. This paper presents a nonparametric approach to the analysis of areas under correlated ROC curves, by using the theory on generalized U-statistics to generate an estimated covariance matrix.

The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 1--coronary artery bypass grafting surgery.

BACKGROUND The first version of The Society of Thoracic Surgeons National Adult Cardiac Surgery Database (STS NCD) was developed nearly 2 decades ago. Since its inception, the number of participants has grown dramatically, patient acuity has increased, and overall outcomes have consistently improved. To adjust for these and other changes, all STS risk models have undergone periodic revisions. This report provides a detailed description of the 2008 STS risk model for coronary artery bypass grafting surgery (CABG). METHODS The study population consisted of 774,881 isolated CABG procedures performed on adult patients aged 20 to 100 years between January 1, 2002, and December 31, 2006, at 819 STS NCD participating centers. This cohort was randomly divided into a 60% training (development) sample and a 40% test (validation) sample. The development sample was used to identify predictor variables and estimate model coefficients. The validation sample was used to assess model calibration and discrimination. Model outcomes included operative mortality, renal failure, stroke, reoperation for any cause, prolonged ventilation, deep sternal wound infection, composite major morbidity or mortality, prolonged length of stay (> 14 days), and short length of stay (< 6 days and alive). Candidate predictor variables were selected based on their availability in versions 2.35, 2.41, and 2.52.1 of the STS NCD and their presence in (or ability to be mapped to) version 2.61. Potential predictor variables were screened for overall prevalence in the study population, missing data frequency, coding concerns, bivariate relationships with outcomes, and their presence in previous STS or other CABG risk models. Supervised backwards selection was then performed with input from an expert panel of cardiac surgeons and biostatisticians. After successfully validating the fit of the models, the development and validation samples were subsequently combined, and the final regression coefficients were estimated using the overall combined (development plus validation) sample. RESULTS The c-index for the mortality model was 0.812, and the c-indices for other endpoints ranged from 0.653 for reoperation to 0.793 for renal failure in the validation sample. Plots of observed versus predicted event rates revealed acceptable calibration in the overall population and in numerous subgroups. When patients were grouped into categories of predicted risk, the absolute difference between the observed and expected event rates was less than 1.5% for each endpoint. The final model intercept and coefficients are provided. CONCLUSIONS New STS risk models have been developed for CABG mortality and eight other endpoints. Detailed descriptions of model development and testing are provided, together with the final algorithm. Overall model performance is excellent.

The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 2--isolated valve surgery.

BACKGROUND Adjustment for case-mix is essential when using observational data to compare surgical techniques or providers. That is most often accomplished through the use of risk models that account for preoperative patient factors that may impact outcomes. The Society of Thoracic Surgeons (STS) uses such risk models to create risk-adjusted performance reports for participants in the STS National Adult Cardiac Surgery Database (NCD). Although risk models were initially developed for coronary artery bypass surgery, similar models have now been developed for use with heart valve surgery, particularly as the proportion of such procedures has increased. The last published STS model for isolated valve surgery was based on data from 1994 to 1997 and did not include patients undergoing mitral valve repair. STS has developed new valve surgery models using contemporary data that include both valve repair as well as replacement. Expanding upon existing valve models, the new STS models include several nonfatal complications in addition to mortality. METHODS Using STS data from 2002 to 2006, isolated valve surgery risk models were developed for operative mortality, permanent stroke, renal failure, prolonged ventilation (> 24 hours), deep sternal wound infection, reoperation for any reason, a major morbidity or mortality composite endpoint, prolonged postoperative length of stay, and short postoperative length of stay. The study population consisted of adult patients who underwent one of three types of valve surgery: isolated aortic valve replacement (n = 67,292), isolated mitral valve replacement (n = 21,229), or isolated mitral valve repair (n = 21,238). The population was divided into a 60% development sample and a 40% validation sample. After an initial empirical investigation, the three surgery groups were combined into a single logistic regression model with numerous interactions to allow the covariate effects to differ across these groups. Variables were selected based on a combination of automated stepwise selection and expert panel review. RESULTS Unadjusted operative mortality (in-hospital regardless of timing, and 30-day regardless of venue) for all isolated valve procedures was 3.4%, and unadjusted in-hospital morbidity rates ranged from 0.3% for deep sternal wound infection to 11.8% for prolonged ventilation. The number of predictors in each model ranged from 10 covariates in the sternal infection model to 24 covariates in the composite mortality plus morbidity model. Discrimination as measured by the c-index ranged from 0.639 for reoperation to 0.799 for mortality. When patients in the validation sample were grouped into 10 categories based on deciles of predicted risk, the average absolute difference between observed versus predicted events within these groups ranged from 0.06% for deep sternal wound infection to 1.06% for prolonged postoperative stay. CONCLUSIONS The new STS risk models for valve surgery include mitral valve repair as well as multiple endpoints other than mortality. Model coefficients are provided and an online risk calculator is publicly available from The Society of Thoracic Surgeons website.

Discordance of Databases Designed for Claims Payment versus Clinical Information Systems: Implications for Outcomes Research

Insurance claims data are being used increasingly to study clinical outcomes and quality of care [1-4]. Each year, hospital-specific mortality rates, adjusted by clinically modified International Classification of Diseases (ICD-9-CM) codes from Medicare bills, are released by the Health Care Financing Administration (HCFA) [1-3]. Using ICD-9-CM data to adjust for illness severity, threefold differences for surgeon-specific mortality in Philadelphia were found by Williams and colleagues [4]. Many of the Patient Outcomes Research Teams (PORTs), supported by the Agency for Health Care Policy Research, are using ICD-9-CM coded Medicare discharge abstracts to examine the process of medical care, including physician- and hospital-specific performance [5-7]. The potential advantages of using insurance claims data sets for clinical research have been described in many previous publications [8]. They include 1) large samples of geographically dispersed patients; 2) longitudinal records; 3) data already collected and available; and 4) defined sampling frames. The question remains: Are data collected to obtain insurance reimbursement a valid proxy for data collected for clinical care and research purposes? Such validity is essential to identify clinically relevant populations and to adjust for illness severity and differences in outcomes [9]. Six reabstracting studies have attempted to answer this question with respect to analysis of patients discharged after acute myocardial infarction [10-15]. These studies selected patients with the ICD-9-CM code 410, the code for acute myocardial infarction. By examining medical records, they found that clinical criteria for an acute myocardial infarction were met in 43% to 87% of records where the code was used at discharge. Errors resulted when the physician listed the acute myocardial infarction incorrectly, when a myocardial infarction occurred in a previous admission, or when myocardial infarction was ruled out (if it was the admitting diagnosis). A substantial limitation of five of these studies was that they selected patients based on claims data. Thus, the groups selected for review were only those patients with an ICD-9-CM code for myocardial infarction. Using this design, it was only possible to obtain estimates of disagreement in one direction; patients who had a condition coded in the clinical data set, but not in the claims data set, could not be examined. A second limitation of the previous studies is that their comparison gold standard was based on retrospective review of information recorded in the discharge summary or medical chart. Medical record data are limited by the unstructured way in which they are collected. Inaccuracies in these sources cannot be identified in such a study, and it is possible that in some disagreements with ICD-9-CM codes, the medical record is incorrect. Our study examined the suitability of billing data compared with clinical data (prospectively collected for cardiology research and patient care) for use in clinical outcomes research. The descriptors for coronary artery disease that we examined were those listed as important determinants of prognosis by an expert panel from the American College of Cardiology [16]. Methods Insurance Claims Data The administrative or insurance claims information comprised all discharge abstracts from Duke University Medical Center between July 1985 and May 1990 containing any procedure code for coronary arteriography. All discharged patients, regardless of insurance status or age, were routinely classified by ICD-9-CM codes recorded by trained medical record technicians based on the attending physicians listed discharge diagnoses, the discharge summary, and selected information from the progress notes and from the test result sections of the hospital chart [17]. These records contained up to 30 diagnostic codes and 9 procedure codes. After the technician had assembled the ICD-9-CM codes, the discharge abstract and the chart were returned to the attending physician for final approval by signature; ICD-9-CM codes were not generated for patients having outpatient cardiac catheterization unless they were subsequently admitted for further evaluation or treatment. The records for the subgroup of Medicare patients in this study were sent by Duke Hospital to the North Carolina Medicare intermediary and, thus, reflect the Duke Hospital data contained in the Health Care Financing Administration data sets. Clinical Database Data The clinical information consisted of important diagnostic and prognostic information about coronary artery disease routinely collected on standardized data forms by the cardiology fellow doing the cardiac catheterization for suspected ischemic heart disease. Information collected included details from the patient history, physical examination, laboratory studies, and cardiac catheterization, as previously described [18]. Each new fellow entering the catheterization laboratory was given a 3-hour training session on variable definitions and use of the data forms and was given an operations manual covering these details. In addition, all data were reviewed for accuracy by the attending angiographer associated with the case; additional consistency, range check, and other quality control measures were done during the data entry process by trained research technicians. This information was stored in the Duke Databank for Cardiovascular Disease, a completely separate and independent system from the hospital administrative records described above. Records Matching and Variable Definitions Records from the administrative and clinical files were matched by unique, patient hospital identification numbers and hospitalization dates. Only the first matching clinical record for each patient was included in the analysis. Twelve clinical variables were mapped to ICD-9-CM codes according to an algorithm developed by the Patient Outcomes Research Team for chronic ischemic heart disease (Table 1) (Romano PS, Roos LL. Unpublished observations). The variables studied were selected if they met two criteria: 1) They were considered to be determinants of prognosis for coronary artery disease according to an expert panel from the American College of Cardiology; 2) they could be mapped to diagnoses contained in the ICD-9-CM coding system [16, 17]. The definitions of the clinically identified conditions appear in the Appendix. Table 1. International Classification of Diseases-9-CM and Clinical Detail Map Appendix.Glossary of Terms Data Analysis Based on the clinical condition and the ICD-9-CM map described above, two-by-two tables were constructed to assess the agreement between the data sources. For the claims data, a condition was considered to be absent if it was not coded. For the clinical data, patients with missing data were excluded from the analysis for the specific missing condition. Kappa statistics were generated for each condition to measure agreement while controlling for chance agreement [19]. Confidence intervals and test statistics for proportions were calculated by the normal approximation. For the diagnoses of acute myocardial infarction, congestive heart failure, angina, and unstable angina, we reviewed a random sample of 15 clinical-positive and claims-negative charts as well as 15 claims-positive and clinical-negative charts for each diagnosis to illustrate the major reasons for disagreement. In addition to the comparisons made in the overall data sets, subsets defined by age, fiscal year, and sex were compared to determine if the coding accuracy varied according to these factors. Results The study group consisted of 12 937 consecutive patients having inpatient cardiac catheterization between July 1985 and May 1990. Although each record represented the first cardiac catheterization in the claims records, from the perspective of the clinical records, 89% involved the first catheterization, 8% involved the second catheterization, and the remaining 3% involved the third or subsequent catheterization. The patients had a mean age of 58.8 years, 34% were women, and the racial composition was 88% white, 10% black, and 2% other. At cardiac catheterization, the mean left ventricular ejection fraction was 52%. The distribution of the number of diseased major epicardial vessels (zero, one, two, or three) was 23%, 26%, 23%, and 28%, respectively. Overall, the study group characteristics were similar to those of other large angiographic registries except for the greater proportion of women and the higher mean age [20, 21]. Measures of Agreement Specific measures of agreement between clinical database and ICD-9-CM variables are listed in Table 2 in descending order of value (the agreement rate adjusted for chance agreement). Kappas ranged from 0.83 for diabetes mellitus to 0.09 for unstable angina. Of the 12 conditions, only 3 (diabetes, acute myocardial infarction, and hypertension) were identified by the claims data more than 50% of the time that they were identified by the clinical data. Table 2. Comparison of Agreement by Condition Ranked by Kappa Value In the clinical data set, two conditions were graded according to severity, congestive heart failure, and mitral regurgitation. With increasing severity levels, the claims data were more likely to identify the presence of these conditions. Claims data identified 31% of clinically identified congestive heart failure that was New York Heart Association class I and II and identified 45% of class III and IV heart failure (P < 0.0001) [22]. Similarly, claims data identified 40% of grades I and II mitral regurgitation and identified 69% of grades III and IV mitral regurgitation (P < 0.0001). When all diagnoses were considered together, the overall agreement of ICD-9-CM codes with clinical data was 0.75 (99% CI, 0.75 to 0.76). The proportion of conditions in the clinical data set identified by claims data was 0.39 (99% CI, 0.38 to 0.39) (Table 3). Stratified by fisc

The society of thoracic surgeons: 30-day operative mortality and morbidity risk models

BACKGROUND Although 30 day risk-adjusted operative mortality (ROM) has been used for quality assessment, it is not sufficient to describe the outcomes after coronary artery bypass grafting (CABG) surgery. Risk-adjusted major morbidity may differentially impact quality of care (as complications occur more frequently than death) and enhance a surgical teams ability to assess their quality. This study identified the preoperative risk factors associated with several complications and a composite outcome (the presence of any major morbidity or 30-day operative mortality or both). METHODS For CABG procedures, the 1997 to 1999 Society of Thoracic Surgeons (STS) National Adult Cardiac Surgery Database was used to develop ROM and risk-adjusted morbidity (ROMB) models. Risk factors were selected using standard STS univariate screening and multivariate logistic regression approaches. Risk model performance was assessed. Across STS participating sites, the association of observed-to-expected (O/E) ratios for ROM and ROMB was evaluated. RESULTS The 30-day operative death and major complication rates for STS CABG procedures were 3.05% and 13.40%, respectively (503,478 CABG procedures), including stroke (1.63%), renal failure (3.53%), reoperation (5.17%), prolonged ventilation (5.96%), and sternal infection (0.63%). Risk models were developed (c-indexes for stroke [0.72], renal failure [0.76], reoperation [0.64], prolonged ventilation [0.75], sternal infection [0.66], and the composite endpoint [0.71]). Only a slight correlation was found, however, between ROMB and ROM indicators. CONCLUSIONS Used in combination, ROMB and ROM may provide the surgical team with additional information to evaluate the quality of their care as well as valuable insights to allow them to focus on areas for improvement.

Racial variation in the use of coronary-revascularization procedures. Are the differences real? Do they matter?

BACKGROUND Studies have reported that blacks undergo fewer coronary-revascularization procedures than whites, but it is not clear whether the clinical characteristics of the patients account for these differences or whether they indicate underuse of the procedures in blacks or overuse in whites. METHODS In a study at Duke University of 12,402 patients (10.3 percent of whom were black) with coronary disease, we calculated unadjusted and adjusted rates of angioplasty and bypass surgery in blacks and whites after cardiac catheterization. We also examined patterns of treatment after stratifying the patients according to the severity of disease, angina status, and estimated survival benefit due to revascularization. Finally, we compared five-year survival rates in blacks and whites. RESULTS After adjustment for the severity of disease and other characteristics, blacks were 13 percent less likely than whites to undergo angioplasty and 32 percent less likely to undergo bypass surgery. The adjusted black:white odds ratios for receiving these procedures were 0.87 (95 percent confidence interval, 0.73 to 1.03) and 0.68 (95 percent confidence interval, 0.56 to 0.82), respectively. The racial differences in rates of bypass surgery persisted among those with severe anginal symptoms (31 percent of blacks underwent surgery, vs. 45 percent of whites, P<0.001) and among those predicted to have the greatest survival benefit from revascularization (42 percent vs. 61 percent, P<0.001). Finally, unadjusted and adjusted rates of survival for five years were significantly lower in blacks than in whites. CONCLUSIONS Blacks with coronary disease were significantly less likely than whites to undergo coronary revascularization, particularly bypass surgery - a difference that could not be explained by the clinical features of their disease. The differences in treatment were most pronounced among those predicted to benefit the most from revascularization. Since these differences also correlated with a lower survival rate in blacks, we conclude that coronary revascularization appears to be underused in blacks.

Long-Term Adherence to Evidence-Based Secondary Prevention Therapies in Coronary Artery Disease

Background— Studies have examined the use of evidence-based therapies for coronary artery disease (CAD) in the short term and at hospital discharge, but few have evaluated long-term use. Methods and Results— Using the Duke Databank for Cardiovascular Disease for the years 1995 to 2002, we determined the annual prevalence and consistency of self-reported use of aspirin, &bgr;-blockers, lipid-lowering agents, and their combinations in all CAD patients and of angiotensin-converting enzyme inhibitors (ACEIs) in those with and without heart failure. Logistic-regression models identified characteristics associated with consistent use (reported on ≥2 consecutive follow-up surveys and then through death, withdrawal, or study end), and Cox proportional-hazards models explored the association of consistent use with mortality. Use of all agents and combinations thereof increased yearly. In 2002, 83% reported aspirin use; 61%, &bgr;-blocker use; 63%, lipid-lowering therapy use; 54%, aspirin and &bgr;-blocker use; and 39%, use of all 3. Consistent use was as follows: For aspirin, 71%; &bgr;-blockers, 46%; lipid-lowering therapy, 44%; aspirin and &bgr;-blockers, 36%; and all 3, 21%. Among patients without heart failure, 39% reported ACEI use in 2002; consistent use was 20%. Among heart failure patients, ACEI use was 51% in 2002 and consistent use, 39%. Except for ACEIs among patients without heart failure, consistent use was associated with lower adjusted mortality: Aspirin hazard ratio (HR), 0.58 and 95% confidence interval (CI), 0.54 to 0.62; &bgr;-blockers, HR, 0.63 and 95% CI, 0.59 to 0.67; lipid-lowering therapy, HR, 0.52 and 95% CI, 0.42 to 0.65; all 3, HR, 0.67 and 95% CI, 0.59 to 0.77; aspirin and &bgr;-blockers, HR, 0.61 and 95% CI, 0.57 to 0.65; and ACEIs among heart failure patients, HR, 0.75 and 95% CI, 0.67 to 0.84. Conclusions— Use of evidence-based therapies for CAD has improved but remains suboptimal. Although improved discharge prescription of these agents is needed, considerable attention must also be focused on understanding and improving long-term adherence.

Outcome of Acute Myocardial Infarction According to the Specialty of the Admitting Physician

BACKGROUND In order to limit costs, health care organizations in the United States are shifting medical care from specialists to primary care physicians. Although primary care physicians provide less resource-intensive care, there is little information concerning the effects of this strategy on outcomes. METHODS We examined mortality according to the specialty of the admitting physician among 8241 Medicare patients who were hospitalized for acute myocardial infarction in four states during a seven-month period in 1992. Proportional-hazards regression models were used to examine survival up to one year after the myocardial infarction. To determine the generalizability of our findings, we also examined insurance claims and survival data for all 220,535 patients for whom there were Medicare claims for hospital care for acute myocardial infarction in 1992. RESULTS After adjustment for characteristics of the patients and hospitals, patients who were admitted to the hospital by a cardiologist were 12 percent less likely to die within one year than those admitted by a primary care physician (P<0.001). Cardiologists also had the highest rate of use of cardiac procedures and medications, including medications (such as thrombolytic agents and beta-blockers) that are associated with improved survival. CONCLUSIONS Health care strategies that shift the care of elderly patients with myocardial infarction from cardiologists to primary care physicians lower rates of use of resources (and potentially lower costs), but they may also cause decreased survival. Additional information is needed to elucidate how primary care physicians and specialists should interact in the care of severely ill patients.

Prediction of operative mortality after valve replacement surgery

OBJECTIVES We sought to develop national benchmarks for valve replacement surgery by developing statistical risk models of operative mortality. BACKGROUND National risk models for coronary artery bypass graft surgery (CABG) have gained widespread acceptance, but there are no similar models for valve replacement surgery. METHODS The Society of Thoracic Surgeons National Cardiac Surgery Database was used to identify risk factors associated with valve surgery from 1994 through 1997. The population was drawn from 49,073 patients undergoing isolated aortic valve replacement (AVR) or mitral valve replacement (MVR) and from 43,463 patients undergoing CABG combined with AVR or MVR. Two multivariable risk models were developed: one for isolated AVR or MVR and one for CABG plus AVR or CABG plus MVR. RESULTS Operative mortality rates for AVR, MVR, combined CABG/AVR and combined CABG/ MVR were 4.00%, 6.04%, 6.80% and 13.29%, respectively. The strongest independent risk factors were emergency/salvage procedures, recent infarction, reoperations and renal failure. The c-indexes were 0.77 and 0.74 for the isolated valve replacement and combined CABG/valve replacement models, respectively. These models retained their predictive accuracy when applied to a prospective patient population undergoing operation from 1998 to 1999. The Hosmer-Lemeshow goodness-of-fit statistic was 10.6 (p = 0.225) for the isolated valve replacement model and 12.2 (p = 0.141) for the CABG/valve replacement model. CONCLUSIONS Statistical models have been developed to accurately predict operative mortality after valve replacement surgery. These models can be used to enhance quality by providing a national benchmark for valve replacement surgery.

Contemporary mortality risk prediction for percutaneous coronary intervention: results from 588,398 procedures in the National Cardiovascular Data Registry.

OBJECTIVES We sought to create contemporary models for predicting mortality risk following percutaneous coronary intervention (PCI). BACKGROUND There is a need to identify PCI risk factors and accurately quantify procedural risks to facilitate comparative effectiveness research, provider comparisons, and informed patient decision making. METHODS Data from 181,775 procedures performed from January 2004 to March 2006 were used to develop risk models based on pre-procedural and/or angiographic factors using logistic regression. These models were independently evaluated in 2 validation cohorts: contemporary (n = 121,183, January 2004 to March 2006) and prospective (n = 285,440, March 2006 to March 2007). RESULTS Overall, PCI in-hospital mortality was 1.27%, ranging from 0.65% in elective PCI to 4.81% in ST-segment elevation myocardial infarction patients. Multiple pre-procedural clinical factors were significantly associated with in-hospital mortality. Angiographic variables provided only modest incremental information to pre-procedural risk assessments. The overall National Cardiovascular Data Registry (NCDR) model, as well as a simplified NCDR risk score (based on 8 key pre-procedure factors), had excellent discrimination (c-index: 0.93 and 0.91, respectively). Discrimination and calibration of both risk tools were retained among specific patient subgroups, in the validation samples, and when used to estimate 30-day mortality rates among Medicare patients. CONCLUSIONS Risks for early mortality following PCI can be accurately predicted in contemporary practice. Incorporation of such risk tools should facilitate research, clinical decisions, and policy applications.