Hillary J. Mull

Validity of Selected Patient Safety Indicators: Opportunities and Concerns

BACKGROUND The Agency for Healthcare Research and Quality (AHRQ) recently designed the Patient Safety Indicators (PSIs) to detect potential safety-related adverse events. The National Quality Forum has endorsed several of these ICD-9-CM-based indicators as quality-of-care measures. We examined the positive predictive value (PPV) of 3 surgical PSIs: postoperative pulmonary embolus and deep vein thrombosis (pPE/DVT), iatrogenic pneumothorax (iPTX), and accidental puncture and laceration (APL). STUDY DESIGN We applied the AHRQ PSI software (v.3.1a) to fiscal year 2003 to 2007 Veterans Health Administration (VA) administrative data to identify (flag) patients suspected of having a pPE/DVT, iPTX, or APL. Two trained nurse abstractors reviewed a sample of 336 flagged medical records (112 records per PSI) using a standardized instrument. Inter-rater reliability was assessed. RESULTS Of 2,343,088 admissions, 6,080 were flagged for pPE/DVT (0.26%), 1,402 for iPTX (0.06%), and 7,203 for APL (0.31%). For pPE/DVT, the PPV was 43% (95% CI, 34% to 53%); 21% of cases had inaccurate coding (eg, arterial not venous thrombosis); and 36% featured thromboembolism present on admission or preoperatively. For iPTX, the PPV was 73% (95% CI, 64% to 81%); 18% had inaccurate coding (eg, spontaneous pneumothorax), and 9% were pneumothoraces present on admission. For APL, the PPV was 85% (95% CI, 77% to 91%); 10% of cases had coding inaccuracies and 5% indicated injuries present on admission. However, 27% of true APLs were minor injuries requiring no surgical repair (eg, small serosal bowel tear). Inter-rater reliability was >90% for all 3 PSIs. CONCLUSIONS Until coding revisions are implemented, these PSIs, especially pPE/DVT, should be used primarily for screening and case-finding. Their utility for public reporting and pay-for-performance needs to be reassessed.

Development of trigger tools for surveillance of adverse events in ambulatory surgery

Background The trigger tool methodology uses clinical algorithms applied electronically to ‘flag’ medical records where adverse events (AEs) have most likely occurred. The authors sought to create surgical triggers to detect AEs in the ambulatory care setting. Methods Four consecutive steps were used to develop ambulatory surgery triggers. First, the authors conducted a comprehensive literature review for surgical triggers. Second, a series of multidisciplinary focus groups (physicians, nurses, pharmacists and information technology specialists) provided user input on trigger selection. Third, a clinical advisory panel designed an initial set of 10 triggers. Finally, a three-phase Delphi process (surgical and trigger tool experts) evaluated and rated the suggested triggers. Results The authors designed an initial set of 10 surgical triggers including five global triggers (flagging medical records for the suspicion of any AE) and five AE-specific triggers (flagging medical records for the suspicion of specific AEs). Based on the Delphi rating of the triggers utility for system-level interventions, the final triggers were: (1) emergency room visit(s) within 21 days from surgery; (2) unscheduled readmission within 30 days from surgery; (3) unscheduled procedure (interventional radiological, urological, dental, cardiac or gastroenterological) or reoperation within 30 days from surgery; (4) unplanned initial hospital length of stay more than 24 h; and (5) lower-extremity Doppler ultrasound order entry and ICD code for deep vein thrombosis or pulmonary embolus within 30 days from surgery. Conclusion The authors therefore propose a systematic methodology to develop trigger tools that takes into consideration previously published work, end-user preferences and expert opinion.

Medical and surgical readmissions in the Veterans Health Administration: what proportion are related to the index hospitalization?

Background:Readmissions are an attractive quality measure because they offer a broad view of quality beyond the index hospitalization. However, the extent to which medical or surgical readmissions reflect quality of care is largely unknown, because of the complexity of factors related to readmission. Identifying those readmissions that are clinically related to the index hospitalization is an important first step in closing this knowledge gap. Objectives:The aims of this study were to examine unplanned readmissions in the Veterans Health Administration, identify clinically related versus unrelated unplanned readmissions, and compare the leading reasons for unplanned readmission between medical and surgical discharges. Methods:We classified 2,069,804 Veterans Health Administration hospital discharges (Fiscal Years 2003–2007) into medical/surgical index discharges with/without readmissions per their diagnosis-related groups. Our outcome variable was “all-cause” 30-day unplanned readmission. We compared medical and surgical unplanned readmissions (n=217,767) on demographics, clinical characteristics, and readmission reasons using descriptive statistics. Results:Among all unplanned readmissions, 41.5% were identified as clinically related. Not surprisingly, heart failure (10.2%) and chronic obstructive pulmonary disease (6.5%) were the top 2 reasons for clinically related readmissions among medical discharges; postoperative complications (ie, complications of surgical procedures and medical care or complications of devices) accounted for 70.5% of clinically related readmissions among surgical discharges. Conclusions:Although almost 42% of unplanned readmissions were identified as clinically related, the majority of unplanned readmissions were unrelated to the index hospitalization. Quality improvement interventions targeted at processes of care associated with the index hospitalization are likely to be most effective in reducing clinically related readmissions. It is less clear how to reduce nonclinically related readmissions; these may involve broader factors than inpatient care.

Postoperative 30-day Readmission: Time to Focus on What Happens Outside the Hospital.

Objective: The aim of this study is to understand the relative contribution of preoperative patient factors, operative characteristics, and postoperative hospital course on 30-day postoperative readmissions. Background: Determining the risk of readmission after surgery is difficult. Understanding the most important contributing factors is important to improving prediction of and reducing postoperative readmission risk. Methods: National Veterans Affairs Surgical Quality Improvement Program data on inpatient general, vascular, and orthopedic surgery from 2008 to 2014 were merged with laboratory, vital signs, prior healthcare utilization, and postoperative complications data. Variables were categorized as preoperative, operative, postoperative/predischarge, and postdischarge. Logistic models predicting 30-day readmission were compared using adjusted R2 and c-statistics with cross-validation to estimate predictive discrimination. Results: Our study sample included 237,441 surgeries: 43% orthopedic, 39% general, and 18% vascular. Overall 30-day unplanned readmission rate was 11.1%, differing by surgical specialty (vascular 15.4%, general 12.9%, and orthopedic 7.6%, P < 0.001). Most common readmission reasons were wound complications (30.7%), gastrointestinal (16.1%), bleeding (4.9%), and fluid/electrolyte (7.5%) complications. Models using information available at the time of discharge explained 10.4% of the variability in readmissions. Of these, preoperative patient-level factors contributed the most to predictive models (R2 7.0% [c-statistic 0.67]); prediction was improved by inclusion of intraoperative (R2 9.0%, c-statistic 0.69) and postoperative variables (R2 10.4%, c-statistic 0.71). Including postdischarge complications improved predictive ability, explaining 19.6% of the variation (R2 19.6%, c-statistic 0.76). Conclusions: Postoperative readmissions are difficult to predict at the time of discharge, and of information available at that time, preoperative factors are the most important.

Detecting adverse events in surgery: comparing events detected by the Veterans Health Administration Surgical Quality Improvement Program and the Patient Safety Indicators

BACKGROUND The Patient Safety Indicators (PSIs) use administrative data to screen for select adverse events (AEs). In this study, VA Surgical Quality Improvement Program (VASQIP) chart review data were used as the gold standard to measure the criterion validity of 5 surgical PSIs. Independent chart review was also used to determine reasons for PSI errors. METHODS The sensitivity, specificity, and positive predictive value of PSI software version 4.1a were calculated among Veterans Health Administration hospitalizations (2003-2007) reviewed by VASQIP (n = 268,771). Nurses re-reviewed a sample of hospitalizations for which PSI and VASQIP AE detection disagreed. RESULTS Sensitivities ranged from 31% to 68%, specificities from 99.1% to 99.8%, and positive predictive values from 31% to 72%. Reviewers found that coding errors accounted for some PSI-VASQIP disagreement; some disagreement was also the result of differences in AE definitions. CONCLUSIONS These results suggest that the PSIs have moderate criterion validity; however, some surgical PSIs detect different AEs than VASQIP. Future research should explore using both methods to evaluate surgical quality.

Comparing 2 methods of assessing 30-day readmissions: what is the impact on hospital profiling in the veterans health administration?

Background: The Centers for Medicare and Medicaid Services’ (CMS) all-cause readmission measure and the 3M Health Information System Division Potentially Preventable Readmissions (PPR) measure are both used for public reporting. These 2 methods have not been directly compared in terms of how they identify high-performing and low-performing hospitals. Objectives: To examine how consistently the CMS and PPR methods identify performance outliers, and explore how the PPR preventability component impacts hospital readmission rates, public reporting on CMS’ Hospital Compare website, and pay-for-performance under CMS’ Hospital Readmission Reduction Program for 3 conditions (acute myocardial infarction, heart failure, and pneumonia). Methods: We applied the CMS all-cause model and the PPR software to VA administrative data to calculate 30-day observed FY08-10 VA hospital readmission rates and hospital profiles. We then tested the effect of preventability on hospital readmission rates and outlier identification for reporting and pay-for-performance by replacing the dependent variable in the CMS all-cause model (Yes/No readmission) with the dichotomous PPR outcome (Yes/No preventable readmission). Results: The CMS and PPR methods had moderate correlations in readmission rates for each condition. After controlling for all methodological differences but preventability, correlations increased to >90%. The assessment of preventability yielded different outlier results for public reporting in 7% of hospitals; for 30% of hospitals there would be an impact on Hospital Readmission Reduction Program reimbursement rates. Conclusions: Despite uncertainty over which readmission measure is superior in evaluating hospital performance, we confirmed that there are differences in CMS-generated and PPR-generated hospital profiles for reporting and pay-for-performance, because of methodological differences and the PPR’s preventability component.

Does Use of a Hospital-wide Readmission Measure Versus Condition-specific Readmission Measures Make a Difference for Hospital Profiling and Payment Penalties?

Background:The Centers for Medicare and Medicaid Services (CMS) use public reporting and payment penalties as incentives for hospitals to reduce readmission rates. In contrast to the current condition-specific readmission measures, CMS recently developed an all-condition, 30-day all-cause hospital-wide readmission measure (HWR) to provide a more comprehensive view of hospital performance. Objectives:We examined whether assessment of hospital performance and payment penalties depends on the readmission measure used. Research Design:We used inpatient data to examine readmissions for patients discharged from VA acute-care hospitals from Fiscal Years 2007–2010. We calculated risk-standardized 30-day readmission rates for 3 condition-specific measures (heart failure, acute myocardial infarction, and pneumonia) and the HWR measure, and examined agreement between the HWR measure and each of the condition-specific measures on hospital performance. We also assessed the effect of using different readmission measures on hospitals’ payment penalties. Results:We found poor agreement between the condition-specific measures and the HWR measure on those hospitals identified as low or high performers (eg, among those hospitals classified as poor performers by the heart failure readmission measure, only 28.6% were similarly classified by the HWR measure). We also found differences in whether a hospital would experience payment penalties. The HWR measure penalized only 60% of those hospitals that would have received penalties based on at least 1 of the condition-specific measures. Conclusions:The condition-specific measures and the HWR measure provide a different picture of hospital performance. Future research is needed to determine which measure aligns best with CMS’s overall goals to reduce hospital readmissions and improve quality.

Do pneumonia readmissions flagged as potentially preventable by the 3M PPR software have more process of care problems? A cross-sectional observational study

Background In the USA, administrative data-based readmission rates such as the Centers for Medicare and Medicaid Services’ all-cause readmission measures are used for public reporting and hospital payment penalties. To improve this measure and identify better quality improvement targets, 3M developed the Potentially Preventable Readmissions (PPRs) measure. It matches clinically related index admission and readmission diagnoses that may indicate readmissions resulting from admission- or post-discharge-related quality problems. Objective To examine whether PPR software-flagged pneumonia readmissions are associated with poorer quality of care. Methods Using a retrospective observational study design and Veterans Health Administration (VA) data, we identified pneumonia discharges associated with 30-day readmissions, and then flagged cases as PPR–yes or PPR–no using the PPR software. To assess quality of care, we abstracted electronic medical records of 100 random readmissions using a tool containing explicit care processes organised into admission work-up, in-hospital evaluation/treatment, discharge readiness and post-discharge period. We derived quality scores, scaled to a maximum of 25 per section (maximum total score=100) and compared cases by total and section-specific mean scores using t tests and effect size (ES) to characterise the clinical significance of findings. Results Our abstraction sample was selected from 11 278 pneumonia readmissions (readmission rate=16.5%) during 1 October 2005–30 September 2010; 77% were flagged as PPR–yes. Contrary to expectations, total and section mean quality scores were slightly higher, although non-significantly, among PPR–yes (N=77) versus PPR–no (N=23) cases (respective total scores, 71.2±8.7 vs 65.8±11.5, p=0.14); differences demonstrated ES >0.30 overall and for admission work-up and post-discharge period sections. Conclusions Among VA pneumonia readmissions, PPR categorisation did not produce the expected quality of care findings. Either PPR–yes cases are not more preventable, or preventability assessment requires other data collection methods to capture poorly documented processes (eg, direct observation).

Development and testing of tools to detect ambulatory surgical adverse events.

Objectives Numerous health-care systems in the United States, including the Veterans Health Administration (VA), use the National Surgical Quality Improvement Program (NSQIP) to detect surgical adverse events (AEs). VASQIP sampling methodology excludes many routine ambulatory surgeries from review. Triggers, algorithms derived from clinical logic to flag cases where AEs have most likely occurred, could complement VASQIP by detecting a higher yield of ambulatory surgeries with a true surgical AE. Methods We developed and tested a set of ambulatory surgical AE trigger algorithms using a sample of fiscal year 2008 ambulatory surgeries from the VA Boston Healthcare System. We used VA Boston VASQIP-assessed cases to refine triggers and VASQIP-excluded cases to test how many trigger-flagged surgeries had a nurse chart review-detected surgical AE. Chart review was performed using the VA electronic medical record. We calculated the ratio of cases with a true surgical AE over flagged cases (i.e., the positive predictive value [PPV]), and the 95% confidence interval for each trigger. Results Compared with the VASQIP rate (9 AEs, or 2.8%, of the 322 charts assessed), nurse chart review of the 198 trigger-flagged surgeries yielded more cases with at least 1 AE (47 surgeries with an AE, or 6.0%, of the 782 VASQIP-excluded ambulatory surgeries). Individual trigger PPVs ranged from 12.4% to 58.3%. Conclusions In comparison with VASQIP, our set of triggers identified a higher rate of surgeries with AEs in fewer chart-reviewed cases. Because our results are based on a relatively small sample, further research is necessary to confirm these findings.

Using AHRQ Patient Safety Indicators to Detect Postdischarge Adverse Events in the Veterans Health Administration

Patient safety indicators (PSIs) use inpatient administrative data to flag cases with potentially preventable adverse events (AEs) attributable to hospital care. This study explored how many AEs the PSIs identified in the 30 days post discharge. PSI software was run on Veterans Health Administration 2003-2007 administrative data for 10 recently validated PSIs. Among PSI-eligible index hospitalizations not flagged with an AE, this study evaluated how many AEs occurred within 1 to 14 and 15 to 30 days post discharge using inpatient and outpatient administrative data. Considering all PSI-eligible index hospitalizations, 11 141 postdischarge AEs were identified, compared with 40 578 inpatient-flagged AEs. More than 60% of postdischarge AEs were detected within 14 days of discharge. The majority of postdischarge AEs were decubitus ulcers and postoperative pulmonary embolisms or deep vein thromboses. Extending PSI algorithms to the postdischarge period may provide a more complete picture of hospital quality. Future work should use chart review to validate postdischarge PSI events.