Karen Richards

Toward Robust Information: Data Quality and Inter-Rater Reliability in the American College of Surgeons National Surgical Quality Improvement Program

BACKGROUND Data used for evaluating quality of medical care need to be of high reliability to ensure valid quality assessment and benchmarking. The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) has continually emphasized the collection of highly reliable clinical data through its program infrastructure. STUDY DESIGN We provide a detailed description of the various mechanisms used in ACS NSQIP to assure collection of high quality data, including training of data collectors (surgical clinical reviewers) and ongoing audits of data reliability. For the 2005 through 2008 calendar years, inter-rater reliability was calculated overall and for individual variables using percentages of agreement between the data collector and the auditor. Variables with > 5% disagreement are flagged for educational efforts to improve accurate collection. Cohens kappa was estimated for selected variables from the 2007 audit year. RESULTS Inter-rater reliability audits show that overall disagreement rates on variables have fallen from 3.15% in 2005 (the first year of public enrollment in ACS NSQIP) to 1.56% in 2008. In addition, disagreement levels for individual variables have continually improved, with 26 individual variables demonstrating > 5% disagreement in 2005, to only 2 such variables in 2008. Estimated kappa values suggest substantial or almost perfect agreement for most variables. CONCLUSIONS The ACS NSQIP has implemented training and audit procedures for its hospital participants that are highly effective in collecting robust data. Audit results show that data have been reliable since the programs inception and that reliability has improved every year.

Does Surgical Quality Improve in the American College of Surgeons National Surgical Quality Improvement Program: An Evaluation of All Participating Hospitals

Background/Objective:The National Surgical Quality Improvement Program (NSQIP) has demonstrated quality improvement in the VA and pilot study of 14 academic institutions. The objective was to show that American College of Surgeons (ACS)-NSQIP helps all enrolled hospitals. Methods:ACS-NSQIP data was used to evaluate improvement in hospitals longitudinally over 3 years (2005–2007). Improvement was defined as reduction in risk-adjusted “Observed/Expected” (O/E) ratios between periods with risk adjustment held constant. Multivariable logistic regression-based adjustment was performed and included indicators for procedure groups. Additionally, morbidity counts were modeled using a negative binomial model, to estimate the number of avoided complications. Results:Multiple perspectives reflected improvement over time. In the analysis of 118 hospitals (2006–2007), 66% of hospitals improved risk-adjusted mortality (mean O/E improvement: 0.174; P < 0.05) and 82% improved risk adjusted complication rates (mean improvement: 0.114; P < 0.05). Correlations between starting O/E and improvement (0.834 for mortality, 0.652 for morbidity), as well as relative risk, revealed that initially worse-performing hospitals had more likelihood of improvement. Nonetheless, well-performing hospitals also improved. Modeling morbidity counts, 183 hospitals (2007), avoided ∼9598 potential complications: ∼52/hospital. Due to sampling this may represent only 1 of 5 to 1of 10 of the true total. Improvement reflected aggregate performance across all types of hospitals (academic/community, urban/rural). Changes in patient risk over time had important contributions to the effect. Conclusions:ACS-NSQIP indicates that surgical outcomes improve across all participating hospitals in the private sector. Improvement is reflected for both poor- and well-performing facilities. NSQIP hospitals appear to be avoiding substantial numbers of complications- improving care, and reducing costs. Changes in risk over time merit further study.

Association of surgical care improvement project infection-related process measure compliance with risk-adjusted outcomes: Implications for quality measurement

BACKGROUND Facility-level process measure adherence is being publicly reported. However, the association between measure adherence and surgical outcomes is not well-established. Our objective was to determine the degree to which Surgical Care Improvement Project (SCIP) process measures are associated with American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) risk-adjusted outcomes. STUDY DESIGN This cross-sectional study included hospitals participating in the ACS NSQIP and SCIP (n = 200). ACS NSQIP outcomes (30-day overall morbidity, serious morbidity, surgical site infections [SSI], and mortality) and adherence to SCIP SSI-related process measures (from the Hospital Compare database) were collected from January 1, 2008, through December 31, 2008. Hospital-level correlation coefficients between compliance with 4 process measures (ie, antibiotic administration within 1 hour before incision [SCIP-1]; appropriate antibiotic prophylaxis [SCIP-2]; antibiotic discontinuation within 24 hours after surgery [SCIP-3]; and appropriate hair removal [SCIP 6]) and 4 risk-adjusted outcomes were calculated. Regression analyses estimated the contribution of process measure adherence to risk-adjusted outcomes. RESULTS Of 211 ACS NSQIP hospitals, 95% had data reported by Hospital Compare. Depending on the measure, hospital-level compliance ranged from 60% to 100%. Of the 16 correlations, 15 demonstrated nonsignificant associations with risk-adjusted outcomes. The exception was the relationship between SCIP-2 and SSI (p = 0.004). SCIP-1 demonstrated an intriguing but nonsignificant relationship with SSI (p = 0.08) and overall morbidity (p = 0.08). Although adherence to SCIP-2 was a significant predictor of risk-adjusted SSI (p < 0.0001) and overall morbidity (p < 0.0001), inclusion of compliance for SCIP-1 and SCIP-2 caused only slight improvement in model quality. CONCLUSIONS Better adherence to infection-related process measures over the observed range was not significantly associated with better outcomes with one exception. Different measures of quality might be needed for surgical infection.

Effect of Postdischarge Morbidity and Mortality on Comparisons of Hospital Surgical Quality

Background:Hospitals increasingly rely on surgical quality assessment programs that require considerable resources to capture outcomes after hospital discharge. However, it is unclear whether capturing postdischarge complications and deaths is important. Our objectives were (1) to determine the frequency of postdischarge complications and deaths and (2) to determine whether hospital rankings change with inclusion of postdischarge outcomes. Methods:From 181 hospitals participating in the American College of Surgeons National Surgical Quality Improvement Program, 329,951 patients were identified (2006–2007). Mortality and 19 complications within 30 days of the index operation were categorized as occurring before or after discharge. Risk-adjusted hospital rankings were compared based on whether only predischarge (inpatient) versus both pre- and postdischarge (inpatient and outpatient within 30 days of operation) morbidity and mortality were included. Results:Postdischarge complications accounted for 32.9% of all complications. Certain complications occurred frequently after discharge: surgical site infections (66.0%), urinary tract infections (39.4%), pulmonary embolisms (42.2%), and deep venous thromboses (34.5%). Of all patients experiencing complications, 39.7% had only postdischarge complications. Of 5827 postoperative deaths, 23.6% occurred after discharge. Hospital quality rankings changed when postdischarge outcomes were excluded versus included for morbidity (median hospital rank change: 16 ranks; interquartile range, 7–36) and mortality (median hospital rank change: 14 ranks; interquartile range, 6–29), and there was disagreement in outlier status designations depending on whether postdischarge events were included (morbidity: &kgr; = 0.546; mortality: &kgr; = 0.507). Conclusions:A substantial proportion of postoperative complications and deaths occur after hospital discharge. Inclusion of postdischarge events considerably affects hospital quality rankings and outlier status designations. Quality improvement programs and research that do not consider postdischarge outcomes may offer incomplete information to hospitals, payers, providers, and patients.

Pediatric American College of Surgeons National Surgical Quality Improvement Program: feasibility of a novel, prospective assessment of surgical outcomes

PURPOSE The American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) provides validated assessment of surgical outcomes. This study reports initiation of an ACS NSQIP Pediatric at 4 childrens hospitals. METHODS From October 2008 to June 2009, 121 data variables were prospectively collected for 3315 patients, including 30-day outcomes and tailoring the ACS NSQIP methodology to childrens surgical specialties. RESULTS Three hundred seven postoperative complications/occurrences were detected in 231 patients representing 7.0% of the study population. Of the patients with complications, 175 (75.7%) had 1, 39 (16.9%) had 2, and 17 (7.4%) had 3 or more complications. There were 13 deaths (0.39%) and 14 intraoperative occurrences (0.42%) detected. The most common complications were infection, 105 (34%) (SSI, 54; sepsis, 31; pneumonia, 13; urinary tract infection, 7); airway/respiratory events, 27 (9%); wound disruption, 18 (6%); neurologic events, 8 (3%) (nerve injury, 4; stroke/vascular event, 2; hemorrhage, 2); deep vein thrombosis, 3 (<1%); renal failure, 3 (<1%); and cardiac events, 3 (<1%). Current sampling captures 17.5% of cases across institutions with unadjusted complication rates ranging from 6.8% to 10.2%. Completeness of data collection for all variables exceeded 95% with 98% interrater reliability and 87% of patients having full 30-day follow-up. CONCLUSION These data represent the first multiinstitutional prospective assessment of specialty-specific surgical outcomes in children. The ACS NSQIP Pediatric is poised for institutional expansion and future development of risk-adjusted models.

American College of Surgeons National Surgical Quality Improvement Program Pediatric: A Phase 1 Report

BACKGROUND There has been a long-standing desire to implement a multi-institutional, multispecialty program to address surgical quality improvement for children. This report documents results of the initial phase of the American College of Surgeons National Surgical Quality Improvement Program Pediatric. STUDY DESIGN From October 2008 to December 2009, patients from 4 pediatric referral centers were sampled using American College of Surgeons National Surgical Quality Improvement Program methodology tailored to children. RESULTS A total of 7,287 patients were sampled, representing general/thoracic surgery (n = 2,237; 30.7%), otolaryngology (n = 1,687; 23.2%), orthopaedic surgery (n = 1,367; 18.8%), urology (n = 893; 12.3%), neurosurgery (n = 697; 9.6%), and plastic surgery (n = 406; 5.6%). Overall mortality rate detected was 0.3% and 287 (3.9%) patients had postoperative occurrences. After accounting for demographic, preoperative, and operative factors, occurrences were 4 times more likely in those undergoing inpatient versus outpatient procedures (odds ratio [OR] = 4.71; 95% CI, 3.01-7.35). Other factors associated with higher likelihood of postoperative occurrences included nutritional/immune history, such as preoperative weight loss/chronic steroid use (OR = 1.49; 95% CI, 1.03-2.15), as well as physiologic compromise, such as sepsis/inotrope use before surgery (OR = 1.68; 95% CI, 1.10-1.95). Operative factors associated with occurrences included multiple procedures under the same anesthetic (OR = 1.58; 95% CI, 1.21-2.06) and American Society of Anesthesiologists classification category 4/5 versus 1 (OR = 5.74; 95% CI, 2.94-11.24). Specialty complication rates varied from 1.5% for otolaryngology to 9.0% for neurosurgery (p < 0.001), with specific procedural groupings within each specialty accounting for the majority of complications. Although infectious complications were the predominant outcomes identified across all specialties, distribution of complications varied by specialty. CONCLUSIONS Based on this initial phase of development, the highly anticipated American College of Surgeons National Surgical Quality Improvement Program Pediatric has the potential to identify outcomes of childrens surgical care that can be targeted for quality improvement efforts.

American College of Surgeons National Surgical Quality Improvement Program Pediatric: A beta phase report

PURPOSE The American College of Surgeons (ACS) National Surgical Quality Improvement Program Pediatric (NSQIP-P) expanded to beta phase testing with the enrollment of 29 institutions. Data collection and analysis were aimed at program refinement and development of risk-adjusted models for inter-institutional comparisons. METHODS Data from the first full year of beta-phase NSQIP-P were analyzed. Patient accrual used ACS-NSQIP methodology tailored to pediatric specialties. Preliminary risk adjusted modeling for all pediatric and neonatal operations and pediatric (excluding neonatal) abdominal operations was performed for all cause morbidity (other than death) and surgical site infections (SSI) using hierarchical logistic regression methodology and eight predictor variables. Results were expressed as odds ratios with 95% confidence intervals. RESULTS During calendar year 2010, 29 institutions enrolled 37,141 patients. 1644 total CPT codes were entered, of which 456 accounted for 90% of the cases. 450 codes were entered only once (1.2% of cases). For all cases, overall mortality was 0.25%, overall morbidity 7.9%, and the SSI rate 1.8%. For neonatal cases, mortality was 2.39%, morbidity 18.7%, and the SSI rate 3%. For the all operations model, risk-adjusted morbidity institutional odds ratios ranged 0.48-2.63, with 9/29 hospitals categorized as low outliers and 9/29 high outliers, while risk-adjusted SSI institutional odds ratios ranged 0.36-2.04, with 2/29 hospitals low outliers and 7/29 high outliers. CONCLUSION This report represents the first risk-adjusted hospital-level comparison of surgical outcomes in infants and children using NSQIP-P data. Programmatic and analytic modifications will improve the impact of this program as it moves into full implementation. These results indicate that NSQIP-P has the potential to serve as a model for determining risk-adjusted outcomes in the neonatal and pediatric population with the goal of developing quality improvement initiatives for the surgical care of children.

Comparison of Hospital Performance in Nonemergency Versus Emergency Colorectal Operations at 142 Hospitals

BACKGROUND Quality improvement efforts have demonstrated considerable hospital-to-hospital variation in surgical outcomes. However, information about the quality of emergency surgical care is lacking. The objective of this study was to assess whether hospitals have comparable outcomes for emergency and nonemergency operations. STUDY DESIGN Patients undergoing colorectal resections were identified from the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) 2005 to 2007 dataset. Logistic regression models for 30-day morbidity and mortality after emergency and nonemergency colorectal resections were constructed. Hospital risk-adjusted outcomes as measured by observed to expected (O/E) ratios, outlier status, and rank-order differences were compared. RESULTS Of 25,710 nonemergency colorectal resections performed at 142 ACS NSQIP hospitals, 6,138 (23.9%) patients experienced at least 1 complication, and 492 (1.9%) patients died. There were 5,083 emergency colorectal resections; 2,442 (48%) patients experienced at least 1 complication, and 780 (15.3%) patients died. Outcomes for nonemergency versus emergency operations were weakly correlated for morbidity and mortality (Pearson correlation coefficient: 0.28 versus 0.13). Median differences in hospital rankings based on O/E ratios between nonemergency and emergency performance were 30.5 ranks (interquartile range [IQR] 13 to 59) for morbidity and 34 ranks (interquartile ratio 17 to 61) for mortality. CONCLUSIONS Hospitals with favorable outcomes after nonemergency colorectal resections do not necessarily have similar outcomes for emergency operations. Hospitals should specifically examine their performance on emergency surgical procedures to identify quality improvement opportunities and focus quality improvement efforts appropriately.

Missing Data in the American College of Surgeons National Surgical Quality Improvement Program Are Not Missing at Random: Implications and Potential Impact on Quality Assessments

BACKGROUND Studying risk-adjusted outcomes in health care relies on statistical approaches to handling missing data. The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) provides risk-adjusted assessments of surgical programs, traditionally imputing certain missing data points using a single round of multivariable imputation. Such imputation assumes that data are missing at random-without systematic bias-and does not incorporate estimation uncertainty. Alternative approaches, including using multiple imputation to incorporate uncertainty or using an indicator of missingness, can enhance robustness of evaluations. STUDY DESIGN One year of de-identified data from the ACS NSQIP, representing 117 institutions and 106,113 patients, was analyzed. Using albumin variables as the missing data modeled, several imputation/adjustment models were compared, including the traditional NSQIP imputation, a new single imputation, a multiple imputation, and use of a missing indicator. RESULTS Coefficients for albumin values changed under new single imputation and multiple imputation approaches. Multiple imputation resulted in increased standard errors, as expected. An indicator of missingness was highly explanatory, disproving the missing-at-random assumption. The effects of changes in approach differed for different outcomes, such as mortality and morbidity, and effects were greatest in smaller datasets. However, ultimate changes in patient risk assessment and institutional assessment were minimal. CONCLUSIONS Newer statistical approaches to modeling missing (albumin) values result in noticeable statistical distinctions, including improved incorporation of imputation uncertainty. In addition, the missing-at-random assumption is incorrect for albumin. Despite these findings, effects on institutional assessments are small. Although effects can be most important with smaller data-sets, the current approach to imputing missing values in the ACS NSQIP appears reasonably robust.

Risk-Adjusted Hospital Outcomes for Children’s Surgery

BACKGROUND The American College of Surgeons National Surgical Quality Improvement Program-Pediatric was initiated in 2008 to drive quality improvement in children’s surgery. Low mortality and morbidity in previous analyses limited differentiation of hospital performance. METHODS: Participating institutions included children’s units within general hospitals and free-standing children’s hospitals. Cases selected by Current Procedural Terminology codes encompassed procedures within pediatric general, otolaryngologic, orthopedic, urologic, plastic, neurologic, thoracic, and gynecologic surgery. Trained personnel abstracted demographic, surgical profile, preoperative, intraoperative, and postoperative variables. Incorporating procedure-specific risk, hierarchical models for 30-day mortality and morbidities were developed with significant predictors identified by stepwise logistic regression. Reliability was estimated to assess the balance of information versus error within models. RESULTS: In 2011, 46 281 patients from 43 hospitals were accrued; 1467 codes were aggregated into 226 groupings. Overall mortality was 0.3%, composite morbidity 5.8%, and surgical site infection (SSI) 1.8%. Hierarchical models revealed outlier hospitals with above or below expected performance for composite morbidity in the entire cohort, pediatric abdominal subgroup, and spine subgroup; SSI in the entire cohort and pediatric abdominal subgroup; and urinary tract infection in the entire cohort. Based on reliability estimates, mortality discriminates performance poorly due to very low event rate; however, reliable model construction for composite morbidity and SSI that differentiate institutions is feasible. CONCLUSIONS: The National Surgical Quality Improvement Program-Pediatric expansion has yielded risk-adjusted models to differentiate hospital performance in composite and specific morbidities. However, mortality has low utility as a children’s surgery performance indicator. Programmatic improvements have resulted in actionable data.