Deborah J. Rennie

Variation in ICU Risk-Adjusted Mortality: Impact of Methods of Assessment and Potential Confounders

BACKGROUND Federal and state agencies are considering ICU performance assessment and public reporting; however, an accurate method for measuring performance must be selected. In this study, we determine whether a substantial variation in ICU mortality performance still exists in modern ICUs, and compare the predictive accuracy, reliability, and data burden of existing ICU risk-adjustment models. METHODS A retrospective chart review of 11,300 ICU patients from 35 California hospitals from 2001 to 2004 was performed. We calculated standardized mortality ratios (SMRs) for each hospital using the mortality probability model III (MPM(0) III), the simplified acute physiology score (SAPS) II, and the acute physiology and chronic health evaluation (APACHE) IV risk-adjustment models. We compared discrimination, calibration, data reliability, and abstraction time for the models. RESULTS Regardless of the model used, there was a large variation in SMRs among the ICUs studied. The discrimination and calibration were adequate for all risk-adjustment models. APACHE IV had the best discrimination (area under the receiver operating characteristic curve [AUC], 0.892) compared to MPM(0) III (AUC, 0.809), and SAPS II (AUC, 0.873; p < 0.001). The models differed substantially in data abstraction times, as follows: MPM(0)III, 11.1 min (95% confidence interval [CI], 8.7 to 13.4); SAPS II, 19.6 min (95% CI, 17.0 to 22.2); and APACHE IV, 37.3 min (95% CI, 28.0 to 46.6). CONCLUSIONS We found substantial variation in the ICU risk-adjusted mortality rates that persisted regardless of the risk-adjustment model. With unlimited resources, the APACHE IV model offers the best predictive accuracy. If constrained by cost and manual data collection, the MPM(0) III model offers a viable alternative without a substantial loss in accuracy.

Relationship Between Discharge Practices and Intensive Care Unit In-hospital Mortality Performance: Evidence of a Discharge Bias

Context:Current intensive care unit performance measures include in-hospital mortality after intensive care unit admission. This measure does not account for deaths occurring after transfer to another hospital or soon after discharge and therefore, may be biased. Objective:Determine how transfer rates to other acute care hospitals and early post-discharge mortality rates impact hospital performance assessments using an in-hospital mortality model. Design, Setting, and Participants:Data were retrospectively collected on 10,502 eligible intensive care unit patients across 35 California hospitals between 2001 and 2004. Measures:We calculated the rates of acute care hospital transfers and early post-discharge mortality (30-day overall mortality—30-day in-hospital mortality) for each hospital. We assessed hospital performance with standardized mortality ratios (SMRs) using the Mortality Probability Model III. Using regression models, we explored the relationship between in-hospital SMRs and the rates of hospital transfers or early post-discharge mortality. We explored the same relationship using a 30-day SMR. Results:In multivariable models, for each 1% increase in patients transferred to another acute care hospital, there was an in-hospital SMR reduction of −0.021 (−0.040−0.001). Additionally, a 1% increase in early post-discharge mortality was associated with an in-hospital SMR reduction of −0.049 (−0.142–0.045). Assessing hospital performance based upon 30-day mortality end point resulted in SMRs closer to 1.0 for hospitals at high and low ends of in-hospital mortality performance. Conclusions:Variations in transfer rates and potentially discharge timing appear to bias in-hospital SMR calculations. A 30-day mortality model is a potential alternative that may limit this bias.

Mortality Probability Model III and Simplified Acute Physiology Score II: Assessing Their Value in Predicting Length of Stay and Comparison to APACHE IV

BACKGROUND To develop and compare ICU length-of-stay (LOS) risk-adjustment models using three commonly used mortality or LOS prediction models. METHODS Between 2001 and 2004, we performed a retrospective, observational study of 11,295 ICU patients from 35 hospitals in the California Intensive Care Outcomes Project. We compared the accuracy of the following three LOS models: a recalibrated acute physiology and chronic health evaluation (APACHE) IV-LOS model; and models developed using risk factors in the mortality probability model III at zero hours (MPM(0)) and the simplified acute physiology score (SAPS) II mortality prediction model. We evaluated models by calculating the following: (1) grouped coefficients of determination; (2) differences between observed and predicted LOS across subgroups; and (3) intraclass correlations of observed/expected LOS ratios between models. RESULTS The grouped coefficients of determination were APACHE IV with coefficients recalibrated to the LOS values of the study cohort (APACHE IVrecal) [R(2) = 0.422], mortality probability model III at zero hours (MPM(0) III) [R(2) = 0.279], and simplified acute physiology score (SAPS II) [R(2) = 0.008]. For each decile of predicted ICU LOS, the mean predicted LOS vs the observed LOS was significantly different (p <or= 0.05) for three, two, and six deciles using APACHE IVrecal, MPM(0) III, and SAPS II, respectively. Plots of the predicted vs the observed LOS ratios of the hospitals revealed a threefold variation in LOS among hospitals with high model correlations. CONCLUSIONS APACHE IV and MPM(0) III were more accurate than SAPS II for the prediction of ICU LOS. APACHE IV is the most accurate and best calibrated model. Although it is less accurate, MPM(0) III may be a reasonable option if the data collection burden or the treatment effect bias is a consideration.

Original ResearchCritical Care MedicineMortality Probability Model III and Simplified Acute Physiology Score II: Assessing Their Value in Predicting Length of Stay and Comparison to APACHE IV

BACKGROUND To develop and compare ICU length-of-stay (LOS) risk-adjustment models using three commonly used mortality or LOS prediction models. METHODS Between 2001 and 2004, we performed a retrospective, observational study of 11,295 ICU patients from 35 hospitals in the California Intensive Care Outcomes Project. We compared the accuracy of the following three LOS models: a recalibrated acute physiology and chronic health evaluation (APACHE) IV-LOS model; and models developed using risk factors in the mortality probability model III at zero hours (MPM(0)) and the simplified acute physiology score (SAPS) II mortality prediction model. We evaluated models by calculating the following: (1) grouped coefficients of determination; (2) differences between observed and predicted LOS across subgroups; and (3) intraclass correlations of observed/expected LOS ratios between models. RESULTS The grouped coefficients of determination were APACHE IV with coefficients recalibrated to the LOS values of the study cohort (APACHE IVrecal) [R(2) = 0.422], mortality probability model III at zero hours (MPM(0) III) [R(2) = 0.279], and simplified acute physiology score (SAPS II) [R(2) = 0.008]. For each decile of predicted ICU LOS, the mean predicted LOS vs the observed LOS was significantly different (p <or= 0.05) for three, two, and six deciles using APACHE IVrecal, MPM(0) III, and SAPS II, respectively. Plots of the predicted vs the observed LOS ratios of the hospitals revealed a threefold variation in LOS among hospitals with high model correlations. CONCLUSIONS APACHE IV and MPM(0) III were more accurate than SAPS II for the prediction of ICU LOS. APACHE IV is the most accurate and best calibrated model. Although it is less accurate, MPM(0) III may be a reasonable option if the data collection burden or the treatment effect bias is a consideration.

Differences in mortality among patients with community-acquired pneumonia in California by ethnicity and hospital characteristics

PURPOSE To determine ethnic disparities in mortality for patients with community-acquired pneumonia, and the potential effects of hospital characteristics on disparities, we compared the risk-adjusted mortality of white, African American, Hispanic, and Asian American patients hospitalized for community-acquired pneumonia. METHODS We studied patients discharged with community-acquired pneumonia in 1996 from an acute care hospital in California (n = 54,874). Logistic regression models were used to examine the association between ethnicity and hospital characteristics and 30-day mortality after adjusting for clinical characteristics. RESULTS The overall 30-day mortality was 12.2%. After adjustment for demographic, clinical, and hospital characteristics, Hispanic (odds ratio [OR] = 0.81; 95% confidence interval [CI]: 0.73 to 0.90) and Asian American patients (OR = 0.88; 95% CI: 0.77 to 1.00) had lower mortality than did white patients, whereas African Americans had a similar mortality to whites (OR = 0.93; 95% CI: 0.83 to 1.06). There were no overall differences in mortality by hospital characteristics (i.e., teaching status, rural location, and public or district hospital). CONCLUSION Hispanics and Asian Americans have a lower risk of death from community-acquired pneumonia than whites in California. No overall differences in mortality were observed by hospital characteristics.

Predictors of early postdischarge mortality in critically ill patients: a retrospective cohort study from the California Intensive Care Outcomes project.

PURPOSE Existing intensive care unit (ICU) mortality measurement systems address in-hospital mortality only. However, early postdischarge mortality contributes significantly to overall 30-day mortality. Factors associated with early postdischarge mortality are unknown. METHODS We performed a retrospective study of 8484 ICU patients. Our primary outcome was early postdischarge mortality: death after hospital discharge and 30 days or less from ICU admission. Cox regression models assessed the association between patient, hospital, and utilization factors and the primary outcome. RESULTS In multivariate analyses, the hazard for early postdischarge mortality increased with rising severity of illness and decreased with full-code status (hazard ratio [HR], 0.33; 95% confidence interval [CI], 0.21-0.49). Compared with discharges home, early postdischarge mortality was highest for acute care transfers (HR, 3.18; 95% CI, 2.45-4.12). Finally, patients with very short ICU length of stay (<1 day) had greater early postdischarge mortality (HR, 1.86; 95% CI; 1.32-2.61) than those with longest stays (≥7 days). CONCLUSIONS Early postdischarge mortality is associated with patient preferences (full-code status) and decisions regarding timing and location of discharge. These findings have important implications for anyone attempting to measure or improve ICU performance and who rely on in-hospital mortality measures to do so.