Marlena H. Shin

Validity of Selected Patient Safety Indicators: Opportunities and Concerns

BACKGROUND The Agency for Healthcare Research and Quality (AHRQ) recently designed the Patient Safety Indicators (PSIs) to detect potential safety-related adverse events. The National Quality Forum has endorsed several of these ICD-9-CM-based indicators as quality-of-care measures. We examined the positive predictive value (PPV) of 3 surgical PSIs: postoperative pulmonary embolus and deep vein thrombosis (pPE/DVT), iatrogenic pneumothorax (iPTX), and accidental puncture and laceration (APL). STUDY DESIGN We applied the AHRQ PSI software (v.3.1a) to fiscal year 2003 to 2007 Veterans Health Administration (VA) administrative data to identify (flag) patients suspected of having a pPE/DVT, iPTX, or APL. Two trained nurse abstractors reviewed a sample of 336 flagged medical records (112 records per PSI) using a standardized instrument. Inter-rater reliability was assessed. RESULTS Of 2,343,088 admissions, 6,080 were flagged for pPE/DVT (0.26%), 1,402 for iPTX (0.06%), and 7,203 for APL (0.31%). For pPE/DVT, the PPV was 43% (95% CI, 34% to 53%); 21% of cases had inaccurate coding (eg, arterial not venous thrombosis); and 36% featured thromboembolism present on admission or preoperatively. For iPTX, the PPV was 73% (95% CI, 64% to 81%); 18% had inaccurate coding (eg, spontaneous pneumothorax), and 9% were pneumothoraces present on admission. For APL, the PPV was 85% (95% CI, 77% to 91%); 10% of cases had coding inaccuracies and 5% indicated injuries present on admission. However, 27% of true APLs were minor injuries requiring no surgical repair (eg, small serosal bowel tear). Inter-rater reliability was >90% for all 3 PSIs. CONCLUSIONS Until coding revisions are implemented, these PSIs, especially pPE/DVT, should be used primarily for screening and case-finding. Their utility for public reporting and pay-for-performance needs to be reassessed.

Validating the patient safety indicators in the Veterans Health Administration: do they accurately identify true safety events?

Background:The Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicators (PSIs) use administrative data to detect potentially preventable in-hospital adverse events. However, few studies have determined how accurately the PSIs identify true safety events. Objectives:We examined the criterion validity, specifically the positive predictive value (PPV), of 12 selected PSIs using clinical data abstracted from the Veterans Health Administration (VA) electronic medical record as the gold standard. Methods:We identified PSI-flagged cases from 28 representative hospitals by applying the AHRQ PSI software (v.3.1a) to VA fiscal year 2003 to 2007 administrative data. Trained nurse-abstractors used standardized abstraction tools to review a random sample of flagged medical records (112 records per PSI) for the presence of true adverse events. Interrater reliability was assessed. We evaluated PPVs and associated 95% confidence intervals of each PSI and examined false positive (FP) cases to determine why they were incorrectly flagged and gain insight into how each PSI might be improved. Results:PPVs ranged from 28% (95% CI, 15%−43%) for Postoperative Hip Fracture to 87% (95% CI, 79%−92%) for Postoperative Wound Dehiscence. Common reasons for FPs included conditions that were present on admission (POA), coding errors, and lack of coding specificity. PSIs with the lowest PPVs had the highest proportion of FPs owing to POA. Conclusions:Overall, PPVs were moderate for most of the PSIs. Implementing POA codes and using more specific ICD-9-CM codes would improve their validity. Our results suggest that additional coding improvements are needed before the PSIs evaluated herein are used for hospital reporting or pay for performance.

Examining the impact of the AHRQ Patient Safety Indicators (PSIs) on the Veterans Health Administration: the case of readmissions.

Background:By focusing primarily on outcomes in the inpatient setting one may overlook serious adverse events that may occur after discharge (eg, readmissions, mortality) as well as opportunities for improving outpatient care. Objective:Our overall objective was to examine whether experiencing an Agency for Healthcare Research and Quality Patient Safety Indicator (PSI) event in an index medical or surgical hospitalization increased the likelihood of readmission. Methods:We applied the Agency for Healthcare Research and Quality PSI software (version 4.1.a) to 2003–2007 Veterans Health Administration inpatient discharge data to generate risk-adjusted PSI rates for 9 individual PSIs and 4 aggregate PSI measures: any PSI event and composite PSIs reflecting “Technical Care,” “Continuity of Care,” and both surgical and medical care (Mixed). We estimated separate logistic regression models to predict the likelihood of 30-day readmission for individual PSIs, any PSI event, and the 3 composites, adjusting for age, sex, comorbidities, and the occurrence of other PSI(s). Results:The odds of readmission were 23% higher for index hospitalizations with any PSI event compared with those with no event [confidence interval (CI), 1.19–1.26], and ranged from 22% higher for Iatrogenic Pneumothorax (CI, 1.03–1.45) to 61% higher for Postoperative Wound Dehiscence (CI, 1.27–2.05). For the composites, the odds of readmission ranged from 15% higher for the Technical Care composite (CI, 1.08–1.22) to 37% higher for the Continuity of Care composite (CI, 1.26–1.50). Conclusions:Our results suggest that interventions that focus on minimizing preventable inpatient safety events as well as improving coordination of care between and across settings may decrease the likelihood of readmission.

Medical and surgical readmissions in the Veterans Health Administration: what proportion are related to the index hospitalization?

Background:Readmissions are an attractive quality measure because they offer a broad view of quality beyond the index hospitalization. However, the extent to which medical or surgical readmissions reflect quality of care is largely unknown, because of the complexity of factors related to readmission. Identifying those readmissions that are clinically related to the index hospitalization is an important first step in closing this knowledge gap. Objectives:The aims of this study were to examine unplanned readmissions in the Veterans Health Administration, identify clinically related versus unrelated unplanned readmissions, and compare the leading reasons for unplanned readmission between medical and surgical discharges. Methods:We classified 2,069,804 Veterans Health Administration hospital discharges (Fiscal Years 2003–2007) into medical/surgical index discharges with/without readmissions per their diagnosis-related groups. Our outcome variable was “all-cause” 30-day unplanned readmission. We compared medical and surgical unplanned readmissions (n=217,767) on demographics, clinical characteristics, and readmission reasons using descriptive statistics. Results:Among all unplanned readmissions, 41.5% were identified as clinically related. Not surprisingly, heart failure (10.2%) and chronic obstructive pulmonary disease (6.5%) were the top 2 reasons for clinically related readmissions among medical discharges; postoperative complications (ie, complications of surgical procedures and medical care or complications of devices) accounted for 70.5% of clinically related readmissions among surgical discharges. Conclusions:Although almost 42% of unplanned readmissions were identified as clinically related, the majority of unplanned readmissions were unrelated to the index hospitalization. Quality improvement interventions targeted at processes of care associated with the index hospitalization are likely to be most effective in reducing clinically related readmissions. It is less clear how to reduce nonclinically related readmissions; these may involve broader factors than inpatient care.

How Valid is the AHRQ Patient Safety Indicator “Postoperative Respiratory Failure”?

BACKGROUND The Agency for Healthcare Research and Quality Patient Safety Indicator postoperative respiratory failure (PRF) uses administrative data to screen for potentially preventable respiratory failure after elective surgery based on a respiratory failure diagnosis or an intubation or ventilation procedure code. Data on PRF accuracy in identifying true events is scant; a recent study using University HealthSystem Consortium data found a positive predictive value (PPV) of 83%. We examined the indicators PPV in the Veterans Health Administration. STUDY DESIGN We applied the Patient Safety Indicator software (v.3.1a) to fiscal year 2003-2007 VA discharge data. Trained abstractors reviewed medical records of 112 software-flagged PRF cases. We calculated the PPV and examined false positives to determine reasons for incorrect identification and true positives to determine clinical consequences and potential risk factors of PRF. RESULTS Seventy-five cases were true positive (PPV 67%; 95% CI, 57-76%); 13% were identified by a diagnosis code, 53% by a procedure code, 33% by both. Of false positives, 19% represented coding errors, 76% represented nonelective admissions. Of true positives, 28% of patients died, 56% had an American Society of Anesthesiologists level higher than II. Of associated index procedures, 53% were abdominal/pelvic, and 56% lasted >3 hours. CONCLUSIONS Based on our and University HealthSystem Consortiums findings, PRF should continue to be used as a screen for potential patient-safety events. Its PPV could be substantially improved in the Veterans Health Administration through introduction of an admission status code. Many PRF-identified cases appeared to be at high risk, based on patient and procedure-related factors. The degree to which such cases are truly preventable events requires additional assessment.

How Valid is the AHRQ Patient Safety Indicator “Postoperative Hemorrhage or Hematoma”?

BACKGROUND Postoperative hemorrhage or hematoma (PHH), an Agency for Healthcare Research and Quality Patient Safety Indicator, uses administrative data to detect cases of potentially preventable postsurgical bleeding requiring a reparative procedure. How accurately it identifies true events is unknown. We therefore determined PHHs positive predictive value. STUDY DESIGN Using Patient Safety Indicator software (v.3.1a) and fiscal year 2003-2007 discharge data from 28 Veterans Health Administration hospitals, we identified 112 possible cases of PHH. Based on medical record abstraction, we characterized cases as true (TPs) or false positives (FPs), calculated positive predictive value, and analyzed FPs to ascertain reasons for incorrect identification and TPs to determine PHH-associated clinical consequences and risk factors. RESULTS Eighty-four cases were TPs (positive predictive value, 75%; 95% CI, 66-83%); 63% had a hematoma diagnosis, 30% had a hemorrhage diagnosis, 7% had both. Reasons for FPs included events present on admission (29%); hemorrhage/hematoma identified and controlled during the original procedure rather than postoperatively (21%); or postoperative hemorrhage/hematoma that did not require a procedure (18%). Most TPs (82%) returned to the operating room for hemorrhage/hematoma management; 64% required blood products and 7% died in-hospital. The most common index procedures resulting in postoperative hemorrhage/hematoma were vascular (38%); 56% were performed by a physician-in-training (under supervision). We found no substantial association between physician training status or perioperative anticoagulant use and bleeding risk. CONCLUSIONS PHHs accuracy could be improved by coding enhancements, such as adopting present on admission codes or associating a timing factor with codes dealing with bleeding control. The ability of PHH to identify events representing quality of care problems requires additional evaluation.

Detecting adverse events in surgery: comparing events detected by the Veterans Health Administration Surgical Quality Improvement Program and the Patient Safety Indicators

BACKGROUND The Patient Safety Indicators (PSIs) use administrative data to screen for select adverse events (AEs). In this study, VA Surgical Quality Improvement Program (VASQIP) chart review data were used as the gold standard to measure the criterion validity of 5 surgical PSIs. Independent chart review was also used to determine reasons for PSI errors. METHODS The sensitivity, specificity, and positive predictive value of PSI software version 4.1a were calculated among Veterans Health Administration hospitalizations (2003-2007) reviewed by VASQIP (n = 268,771). Nurses re-reviewed a sample of hospitalizations for which PSI and VASQIP AE detection disagreed. RESULTS Sensitivities ranged from 31% to 68%, specificities from 99.1% to 99.8%, and positive predictive values from 31% to 72%. Reviewers found that coding errors accounted for some PSI-VASQIP disagreement; some disagreement was also the result of differences in AE definitions. CONCLUSIONS These results suggest that the PSIs have moderate criterion validity; however, some surgical PSIs detect different AEs than VASQIP. Future research should explore using both methods to evaluate surgical quality.

How Valid is the AHRQ Patient Safety Indicator “Postoperative Physiologic and Metabolic Derangement”?

BACKGROUND The Agency for Healthcare Research and Quality Patient Safety Indicator postoperative physiologic and metabolic derangement (PMD) uses ICD-9-CM codes to screen for potentially preventable acute kidney injury (AKI) requiring dialysis plus diabetes-related complications after elective surgery. Data on PMDs accuracy in identifying true events are limited. We examined the indicators positive predictive value (PPV) in the Veterans Health Administration (VA). STUDY DESIGN Trained abstractors reviewed medical records of 119 PSI software-flagged PMD cases. We calculated PPVs overall and separately for renal- and diabetes-related complications. We also examined false positives to determine reasons for incorrect identification, and true positives to determine PMD-related outcomes and risk factors. RESULTS Overall 75 cases were true positives (PPV 63%, 95% CI 54% to 72%); 73 of 104 AKI cases were true positives (PPV 70%, 60% to 79%); only 2 of 15 diabetes cases were true positives (PPV 13%, 2% to 40%). Of all false positives, 70% represented nonelective admissions and 23% had the complication present on admission. Of AKI true positives, 37% died and 26% were discharged on dialysis; 55% had chronic kidney disease (≥ stage 3) present on admission. Cardiac surgery represented the largest category of AKI-associated index procedures (30%). AKI was most commonly attributed to perioperative renal hypoperfusion (84% of true positives), followed by nephrotoxins (33%) including contrast (11%). CONCLUSIONS Due to its low PPV, we recommend removing diabetes complications from the indicator and focusing on AKI. PMDs PPV could be significantly improved by using present-on-admission codes, and specific to the VA, by introduction of admission status codes. Many PMD-identified cases appeared to be at high risk based on patient- and procedure-related factors. The degree to which such cases are truly preventable events requires further assessment.

Using a Virtual Breakthrough Series Collaborative to Reduce Postoperative Respiratory Failure in 16 Veterans Health Administration Hospitals

BACKGROUND The Institute for Healthcare Improvement (IHI) Virtual Breakthrough Series (VBTS) process was used in an eight-month (June 2011-January 2012) quality improvement (QI) project to improve care related to reducing postoperative respiratory failure. The VBTS collaborative drew on Patient Safety Indicator 11: Postoperative Respiratory Failure Rate to guide changes in care at the bedside. METHODS Sixteen Veterans Health Administration hospitals, each representing a regional Veterans Integrated Service Network, participated in the QI project. During the prework phase (initial two months), hospitals formed multidisciplinary teams, selected measures related to their goals, and collected baseline data. The six-month action phase included group conference calls in which the faculty presented clinical background on the topic, discussed evidence-based processes of care, and/or presented content regarding reducing postoperative respiratory failure. During a final, six-month continuous improvement and spread phase, teams were to continue implementing changes as part of their usual processes. RESULTS The six most commonly reported interventions to reduce postoperative respiratory failure focused on improving incentive spirometer use, documenting implementation of targeted interventions, oral care, standardized orders, early ambulation, and provider education. A few teams reported reduced ICU readmissions for respiratory failure. CONCLUSIONS The VBTS collaborative helped teams implement process changes to help reduce postoperative respiratory complications. Teams reported initial success at implementing site-specific improvements using real-time data. The VBTS model shows promise for knowledge sharing and efficient multifacility improvement efforts, although long-term sustainability and testing in these and other settings need to be examined.

Examining the validity of AHRQ's patient safety indicators (PSIs): is variation in PSI composite score related to hospital organizational factors?

Increasing use of Agency for Healthcare Research and Quality’s Patient Safety Indicators (PSIs) for hospital performance measurement intensifies the need to critically assess their validity. Our study examined the extent to which variation in PSI composite score is related to differences in hospital organizational structures or processes (i.e., criterion validity). In site visits to three Veterans Health Administration hospitals with high and three with low PSI composite scores (“low performers” and “high performers,” respectively), we interviewed a cross-section of hospital staff. We then coded interview transcripts for evidence in 13 safety-related domains and assessed variation across high and low performers. Evidence of leadership and coordination of work/communication (organizational process domains) was predominantly favorable for high performers only. Evidence in the other domains was either mixed, or there were insufficient data to rate the domains. While we found some evidence of criterion validity, the extent to which variation in PSI rates is related to differences in hospitals’ organizational structures/processes needs further study.