Yaoming Liu

Development and Evaluation of the Universal ACS NSQIP Surgical Risk Calculator: A Decision Aid and Informed Consent Tool for Patients and Surgeons

BACKGROUND Accurately estimating surgical risks is critical for shared decision making and informed consent. The Centers for Medicare and Medicaid Services may soon put forth a measure requiring surgeons to provide patients with patient-specific, empirically derived estimates of postoperative complications. Our objectives were to develop a universal surgical risk estimation tool, to compare performance of the universal vs previous procedure-specific surgical risk calculators, and to allow surgeons to empirically adjust the estimates of risk. STUDY DESIGN Using standardized clinical data from 393 ACS NSQIP hospitals, a web-based tool was developed to allow surgeons to easily enter 21 preoperative factors (demographics, comorbidities, procedure). Regression models were developed to predict 8 outcomes based on the preoperative risk factors. The universal model was compared with procedure-specific models. To incorporate surgeon input, a subjective surgeon adjustment score, allowing risk estimates to vary within the estimates confidence interval, was introduced and tested with 80 surgeons using 10 case scenarios. RESULTS Based on 1,414,006 patients encompassing 1,557 unique CPT codes, a universal surgical risk calculator model was developed that had excellent performance for mortality (c-statistic = 0.944; Brier score = 0.011 [where scores approaching 0 are better]), morbidity (c-statistic = 0.816, Brier score = 0.069), and 6 additional complications (c-statistics > 0.8). Predictions were similarly robust for the universal calculator vs procedure-specific calculators (eg, colorectal). Surgeons demonstrated considerable agreement on the case scenario scoring (80% to 100% agreement), suggesting reliable score assignment between surgeons. CONCLUSIONS The ACS NSQIP surgical risk calculator is a decision-support tool based on reliable multi-institutional clinical data, which can be used to estimate the risks of most operations. The ACS NSQIP surgical risk calculator will allow clinicians and patients to make decisions using empirically derived, patient-specific postoperative risks.

Improved Surgical Outcomes for ACS NSQIP Hospitals Over Time: Evaluation of Hospital Cohorts With up to 8 Years of Participation.

Background:The American College of Surgeons, National Surgical Quality Improvement Program (ACS NSQIP) surgical quality feedback models are recalibrated every 6 months, and each hospital is given risk-adjusted, hierarchical model, odds ratios that permit comparison to an estimated average NSQIP hospital at a particular point in time. This approach is appropriate for “relative” benchmarking, and for targeting quality improvement efforts, but does not permit evaluation of hospital or program-wide changes in quality over time. We report on long-term improvement in surgical outcomes associated with participation in ACS NSQIP. Study Design:ACS NSQIP data (2006–2013) were used to create prediction models for mortality, morbidity (any of several distinct adverse outcomes), and surgical site infection (SSI). For each model, for each hospital, and for year of first participation (hospital cohort), hierarchical model observed/expected (O/E) ratios were computed. The primary performance metric was the within-hospital trend in logged O/E ratios over time (slope) for mortality, morbidity, and SSI. Results:Hospital-averaged log O/E ratio slopes were generally negative, indicating improving performance over time. For all hospitals, 62%, 70%, and 65% of hospitals had negative slopes for mortality, morbidity, and any SSI, respectively. For hospitals currently in the program for at least 3 years, 69%, 79%, and 71% showed improvement in mortality, morbidity, and SSI, respectively. For these hospitals, we estimate 0.8%, 3.1%, and 2.6% annual reductions (with respect to prior years rates) for mortality, morbidity, and SSI, respectively. Conclusions:Participation in ACS NSQIP is associated with reductions in adverse events after surgery. The magnitude of quality improvement increases with time in the program.

Risk of Discharge to Postacute Care A Patient-Centered Outcome for the American College of Surgeons National Surgical Quality Improvement Program Surgical Risk Calculator

IMPORTANCE Individualized risk prediction tools have an important role as decision aids for use by patients and surgeons before surgery. Patient-centered outcomes should be incorporated into such tools to widen their appeal and improve their usability. OBJECTIVE To develop a patient-centered outcome for the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) Surgical Risk Calculator, a web-based, individualized risk prediction tool. DESIGN, SETTING, AND PARTICIPANTS Retrospective cohort study using data from the ACS NSQIP, a national clinical data registry. A total of 973 211 patients from July 2010 to June 2012, encompassing 392 hospitals, were used in this analysis. MAIN OUTCOMES AND MEASURES Risk of discharge to a postacute care setting. RESULTS The overall rate of discharge to postacute care was 8.8%. Significant predictors of discharge to postacute care included being 85 years or older (odds ratio [OR] = 9.17; 95% CI, 8.84-9.50), the presence of septic shock (OR = 2.43; 95% CI, 2.20-2.69) or ventilator dependence (OR = 2.81; 95% CI, 2.56-3.09) preoperatively, American Society of Anesthesiologists class of 4 or 5 (OR = 3.59; 95% CI, 3.46-3.71), and totally dependent functional status (OR = 2.27; 95% CI, 2.11-2.44). The final model predicted risk of discharge to postacute care with excellent accuracy (C statistic = 0.924) and calibration (Brier score = 0.05). CONCLUSIONS AND RELEVANCE Individualized risk of discharge to postacute care can be predicted with excellent accuracy. This outcome will be incorporated into the ACS NSQIP Surgical Risk Calculator.

An Examination of American College of Surgeons NSQIP Surgical Risk Calculator Accuracy

BACKGROUND The American College of Surgeons NSQIP offers a Surgical Risk Calculator (SRC) that provides detailed, patient-level, risk assessments for many adverse outcomes to surgeons, patients, and the general public. The SRC calculator was designed to help guide discussion and decisions by providing generally applicable (not hospital-specific) information about surgical risk using easily understood and broadly available preoperative variables. Although large, internal evaluations have shown that the SRC has good accuracy (model discrimination and calibration), external validations have been inconsistent and tend to favor a conclusion of inadequate performance. STUDY DESIGN External studies, attempting to validate the SRC, were examined with respect to 3 design features: sample size (small samples reduce reliability), case-mix homogeneity (homogeneity reduces discrimination); and number of institutions providing data (few institutions reduces generalizability). The impact of each feature was then examined in several sets of simulation studies. RESULTS Each of the 3 design features has the potential to act as an artifactual cause for apparent SRC predictive failure. In addition, demonstrations that SRC estimates are inferior to those from models that use additional (sometimes operation-specific) predictor variables were seen as not relevant with respect to the SRCs intended scope. CONCLUSIONS The SRC predictive failures, reported by studies with the described design limitations, should not be misunderstood as disqualifying the SRC as an accurate and appropriate tool for its intended purpose of providing a general purpose risk calculator, applicable across many surgical domains, using easily understood and generally available predictive information.

On-demand Reporting of Risk-adjusted and Smoothed Rates for Quality Profiling in Acs Nsqip

Background: Surgical quality improvement depends on hospitals having accurate and timely information about comparative performance. Profiling accuracy is improved by risk adjustment and shrinkage adjustment to stabilize estimates. These adjustments are included in ACS NSQIP reports, where hospital odds ratios (OR) are estimated using hierarchical models built on contemporaneous data. However, the timeliness of feedback remains an issue. Study Design: We describe an alternative, nonhierarchical approach, which yields risk- and shrinkage-adjusted rates. In contrast to our “Traditional” NSQIP method, this approach uses preexisting equations, built on historical data, which permits hospitals to have near immediate access to profiling results. We compared our traditional method to this new “on-demand” approach with respect to outlier determinations, kappa statistics, and correlations between logged OR and standardized rates, for 12 models (4 surgical groups by 3 outcomes). Results: When both methods used the same contemporaneous data, there were similar numbers of hospital outliers and correlations between logged OR and standardized rates were high. However, larger differences were observed when the effect of contemporaneous versus historical data was added to differences in statistical methodology. Conclusions: The on-demand, nonhierarchical approach provides results similar to the traditional hierarchical method and offers immediacy, an “over-time” perspective, application to a broader range of models and data subsets, and reporting of more easily understood rates. Although the nonhierarchical method results are now available “on-demand” in a web-based application, the hierarchical approach has advantages, which support its continued periodic publication as the gold standard for hospital profiling in the program.

Outcomes of Concurrent Operations: Results From the American College of Surgeons’ National Surgical Quality Improvement Program

Objective: To determine whether concurrently performed operations are associated with an increased risk for adverse events. Background: Concurrent operations occur when a surgeon is simultaneously responsible for critical portions of 2 or more operations. How this practice affects patient outcomes is unknown. Methods: Using American College of Surgeons’ National Surgical Quality Improvement Program data from 2014 to 2015, operations were considered concurrent if they overlapped by ≥60 minutes or in their entirety. Propensity-score-matched cohorts were constructed to compare death or serious morbidity (DSM), unplanned reoperation, and unplanned readmission in concurrent versus non-concurrent operations. Multilevel hierarchical regression was used to account for the clustered nature of the data while controlling for procedure and case mix. Results: There were 1430 (32.3%) surgeons from 390 (77.7%) hospitals who performed 12,010 (2.3%) concurrent operations. Plastic surgery (n = 393 [13.7%]), otolaryngology (n = 470 [11.2%]), and neurosurgery (n = 2067 [8.4%]) were specialties with the highest proportion of concurrent operations. Spine procedures were the most frequent concurrent procedures overall (n = 2059/12,010 [17.1%]). Unadjusted rates of DSM (9.0% vs 7.1%; P < 0.001), reoperation (3.6% vs 2.7%; P < 0.001), and readmission (6.9% vs 5.1%; P < 0.001) were greater in the concurrent operation cohort versus the non-concurrent. After propensity score matching and risk-adjustment, there was no significant association of concurrence with DSM (odds ratio [OR] 1.08; 95% confidence interval [CI] 0.96–1.21), reoperation (OR 1.16; 95% CI 0.96–1.40), or readmission (OR 1.14; 95% CI 0.99–1.29). Conclusions: In these analyses, concurrent operations were not detected to increase the risk for adverse outcomes. These results do not lessen the need for further studies, continuous self-regulation and proactive disclosure to patients.

Defining the Intrinsic Cardiac Risks of Operations to Improve Preoperative Cardiac Risk Assessments

Background: Current preoperative cardiac risk stratification practices group operations into broad categories, which might inadequately consider the intrinsic cardiac risks of individual operations. We sought to define the intrinsic cardiac risks of individual operations and to demonstrate how grouping operations might lead to imprecise estimates of perioperative cardiac risk. Methods: Elective operations (based on Common Procedural Terminology codes) performed from January 1, 2010 to December 31, 2015 at hospitals participating in the American College of Surgeons National Surgical Quality Improvement Program were studied. A composite measure of perioperative adverse cardiac events was defined as either cardiac arrest requiring cardiopulmonary resuscitation or acute myocardial infarction. Operations’ intrinsic cardiac risks were derived from mixed-effects models while controlling for patient mix. Resultant risks were sorted into low-, intermediate-, and high-risk categories, and the most commonly performed operations within each category were identified. Intrinsic operative risks were also examined using a representative grouping of operations to portray within-group variation. Results: Sixty-six low, 30 intermediate, and 106 high intrinsic cardiac risk operations were identified. Excisional breast biopsy had the lowest intrinsic cardiac risk (overall rate, 0.01%; odds ratio, 0.11; 95% CI, 0.02 to 0.25) relative to the average, whereas aorto-bifemoral bypass grafting had the highest (overall rate, 4.1%; odds ratio, 6.61; 95% CI, 5.54 to 7.90). There was wide variation in the intrinsic cardiac risks of operations within the representative grouping (median odds ratio, 1.40; interquartile range, 0.88 to 2.17). Conclusions: A continuum of intrinsic cardiac risk exists among operations. Grouping operations into broad categories inadequately accounts for the intrinsic cardiac risk of individual operations.

Procedure-Specific Trends in Surgical Outcomes

BACKGROUND Quality improvement efforts have generally focused on hospital benchmarking, and processes and outcomes shared among all operations. However, quality improvement could be inconsistent across different types of operations. The objective of this study was to identify operations needing additional concerted quality improvement efforts by examining their outcomes trends. STUDY DESIGN Ten procedures (colectomy, esophagectomy, hepatectomy, hysterectomy, pancreatectomy, proctectomy, total hip arthroplasty, total knee arthroplasty, thyroidectomy, and ventral hernia repair) commonly accrued into the American College of Surgeons NSQIP between 2008 and 2015 were included. Trends in risk-adjusted, standardized, smoothed rates were constructed for each procedure across 6 outcomes (mortality, pneumonia, renal failure, surgical site infection, unplanned intubation, and urinary tract infection [UTI]). RESULTS Of 1,255,575 operations analyzed, the overall unadjusted rate for mortality across all 10 procedures was 1.08%, for pneumonia 1.44%, for renal failure 0.67%, for surgical site infection 5.28%, for unplanned intubation 1.11%, and for UTI 1.86%. Hepatectomy demonstrated the greatest improvement across outcomes (4 of 6 outcomes; 362 adverse events avoided out of 10,000 procedures), and UTI demonstrated the greatest improvement across procedures (8 of 10 procedures; 989 adverse events avoided out of 10,000). For pancreatectomy, rates of mortality, unplanned intubation, and UTI improved, but surgical site infection rates were detected to have significantly increased (p < 0.05). CONCLUSIONS Hepatectomy was detected to have improved across the greatest number of outcomes, and UTI rates improved significantly across the greatest number of procedures. Surgical site infection rates after pancreatectomy, however, were detected to have increased, identifying an urgent need for additional concerted quality improvement efforts.

Variation of Thyroidectomy-Specific Outcomes Among Hospitals and Their Association With Risk Adjustment and Hospital Performance

Importance Current surgical quality metrics might be insufficient to fully judge the quality of certain operations because they are not procedure specific. Hypocalcemia, recurrent laryngeal nerve (RLN) injury, and hematoma are considered to be the most relevant outcomes to measure after thyroidectomy. Whether these outcomes can be used as hospital quality metrics is unknown. Objectives To evaluate whether thyroidectomy-specific outcomes vary among hospitals, whether the addition of thyroidectomy-specific variables affects risk adjustment, and whether differences in hospital performance are associated with thyroidectomy-specific care processes. Design, Setting, and Participants In this retrospective cohort study, patients undergoing thyroidectomies from January 1, 2013, through December 31, 2015, at hospitals participating in the American College of Surgeons’ National Surgical Quality Improvement Program were studied. Exposure Thyroidectomy-related care. Main Outcomes and Measures Clinically severe hypocalcemia, RLN injury, and clinically significant hematoma within 30 days of thyroid surgery and hospital-level performance variation, change in risk adjustment, and association with processes. Results Overall, 14 540 patients (mean [SD] age, 52.1 [15.0] years; 11 499 [79.1%] female) underwent operations at 98 hospitals. Because operations missing thyroidectomy-specific outcomes were excluded, the numbers of operations and hospitals analyzed differed by outcome. Of 14 540 operations included, clinically severe hypocalcemia occurred in 450 patients (3.3% overall, 0.6% after partial, and 4.7% after subtotal or total thyroidectomy), RLN injury in 755 patients (5.7% overall, 4.2% after partial, and 6.6% after subtotal or total thyroidectomy), and hematoma in 175 patients (1.3%). Hospital performance varied for hypocalcemia and RLN injury but not for hematoma. Hospital performance rankings were largely unaffected by the inclusion of thyroidectomy-specific data in risk adjustment. With regard to processes, patients undergoing thyroidectomies at the best-performing vs worst-performing hospitals less frequently had their postoperative parathyroid hormone level measured (593 [19.9%] vs 457 [31.7%], P < .001) and more often were prescribed oral calcium, vitamin D, or both (2281 [76.6%] vs 962 [66.8%], P < .001). When profiled by RLN injury, use of energy devices (1517 [69.1%] vs 507 [55.2%], P < .001) and intraoperative nerve monitoring (1223 [55.7%] vs 346 [37.7%], P < .001) were more prevalent at the best- compared with the worst-performing hospitals. Conclusions and Relevance Postoperative hypocalcemia and RLN injury, but not hematoma, potentially could be used as thyroidectomy-specific national hospital quality improvement metrics. Strategies aimed at reducing these complications after thyroidectomy may improve the care of these patients.

Evaluating the Benefits of ACS-NSQIP.

Reply:We appreciate the opportunity to respond to the issues Osborne and Etzioni1 have raised about our recent article. In that article, we demonstrated improvement in surgical quality among most hospitals participating in the American College of Surgeons-National Surgical Quality Improvement Progra