J. Carlson

Metrics of success: Measuring impact of a departmental near-miss incident learning system

PURPOSE There is a growing interest in the application of incident learning systems (ILS) to radiation oncology. The purpose of the present study is to define statistical metrics that may serve as benchmarks for successful operation of an incident learning system. METHODS AND MATERIALS A departmental safety and quality ILS was developed to monitor errors, near-miss events, and process improvement suggestions. Event reports were reviewed by a multiprofessional quality improvement committee. Events were scored by a near-miss risk index (NMRI) and categorized by event point of origination and discovery. Reporting trends were analyzed over a 2-year period, including total number and rates of events reported, users reporting, NMRI, and event origination and discovery. RESULTS A total of 1897 reports were evaluated (1.0 reports/patient, 0.9 reports/unique treatment course). Participation in the ILS increased as demonstrated by total events (2.1 additional reports/month) and unique users (0.5 new users/month). Sixteen percent of reports had an NMRI of 0 (none), 42% had an NMRI of 1 (mild), 25% had an NMRI of 2 (moderate), 12% had an NMRI of 3 (severe), and 5% had an NMRI of 4 (critical). Event NMRI showed a significant decrease in the first 6 months (1.68-1.42, P < .001). Trends in origination and discovery of reports were broadly distributed between radiation therapy process steps and staff groups. The highest risk events originated in imaging for treatment planning (NMRI = 2.0 ± 1.1; P < .0001) and were detected in on-treatment quality management (NMRI = 1.7 ± 1.1; P = .003). CONCLUSIONS Over the initial 2-year period of ILS operation, rates of reporting increased, staff participation increased, and NMRI of reported events declined. These data mirror previously reported findings of improvement in safety culture endpoints. These metrics may be useful for other institutions seeking to create or evaluate their own ILS.

Measurable improvement in patient safety culture: A departmental experience with incident learning

PURPOSE Rigorous use of departmental incident learning is integral to improving patient safety and quality of care. The goal of this study was to quantify the impact of a high-volume, departmental incident learning system on patient safety culture. METHODS AND MATERIALS A prospective, voluntary, electronic incident learning system was implemented in February 2012 with the intent of tracking near-miss/no-harm incidents. All incident reports were reviewed weekly by a multiprofessional team with regular department-wide feedback. Patient safety culture was measured at baseline with validated patient safety culture survey questions. A repeat survey was conducted after 1 and 2 years of departmental incident learning. Proportional changes were compared by χ(2) or Fisher exact test, where appropriate. RESULTS Between 2012 and 2014, a total of 1897 error/near-miss incidents were reported, representing an average of 1 near-miss report per patient treated. Reports were filed by a cross section of staff, with the majority of incidents reported by therapists, dosimetrists, and physicists. Survey response rates at baseline and 1 and 2 years were 78%, 80%, and 80%, respectively. Statistically significant and sustained improvements were noted in several safety metrics, including belief that the department was openly discussing ways to improve safety, the sense that reports were being used for safety improvement, and the sense that changes were being evaluated for effectiveness. None of the surveyed dimensions of patient safety culture worsened. Fewer punitive concerns were noted, with statistically significant decreases in the worry of embarrassment in front of colleagues and fear of getting colleagues in trouble. CONCLUSIONS A comprehensive incident learning system can identify many areas for improvement and is associated with significant and sustained improvements in patient safety culture. These data provide valuable guidance as incident learning systems become more widely used in radiation oncology.

Interrater reliability of a near-miss risk index for incident learning systems in radiation oncology

PURPOSE Tools for assessing the severity and risk of near-miss events in radiation oncology are few and needed. Recent work has described guidelines for the use of a 5-tier near-miss risk index (NMRI) for the classification of near-miss events. The purpose of this study was to assess the reliability of the NMRI among users in a radiation oncology department. METHODS AND MATERIALS Reliability of the NMRI was assessed using an online survey distributed to members of a radiation oncology department. The survey contained 70 events extracted from the departments incident learning system (ILS). Survey participants rated each event using the NMRI guidelines, reported their attendance to weekly ILS meetings (used as a surrogate for familiarity with the ILS), and indicated their familiarity with the radiation oncology workflow. Interrater reliability was determined using Krippendorffs alpha. Use of the NMRI to rate actual events during 5 weekly ILS meetings was also assessed and interrater reliability determined. RESULTS Twenty-eight survey respondents represented a wide variety of care providers. Krippendorffs alpha was calculated for the whole respondent cohort to be 0.376, indicating fair agreement among raters. Respondents who had the most participation at ILS meetings (n = 4) had moderate agreement with an alpha of 0.501. Interestingly, there were significant differences in reliability and median NMRI scores between professions. NMRI use during weekly NMRI meetings (80 events rated), participants showed moderate reliability (alpha = 0.607). CONCLUSIONS Using the NMRI guidelines, raters from a wide variety of professions were able to assess the severity of near-miss incidents with fair agreement. Those experienced with the ILS showed better agreement, and higher agreement was seen during multidisciplinary ILS meetings. These data support the use the indices such as the NMRI for near-miss risk assessment in patient safety and prioritization of process improvements in radiation oncology.

Can emergent treatments result in more severe errors?: An analysis of a large institutional near-miss incident reporting database.

PURPOSE Emergent radiation treatments may be subject to more errors because of the compressed time frame. Few data exist on the magnitude of this problem or how to guide safety improvement interventions. The purpose of this study is to examine patterns of near-miss events in emergent treatments using a large institutional incident reporting system. METHODS AND MATERIALS Events in the incident reporting database from February 2012 to October 2013 were reviewed prospectively by a multidisciplinary team to identify emergent treatments. Reports were scored for potential near-miss risk index (NMRI) on a 0 to 4 scale. Workflow steps of where events originated and were detected were analyzed. Events were categorized by use of the causal factor system from the Radiation Oncology Incident Learning System. Mann-Whitney U tests were used to compare mean NMRI score, and Fisher exact tests were performed to compare the proportion of high-risk events between emergent and nonemergent treatments and between emergent treatments on weekdays and weekends or holidays. RESULTS Over the study period, approximately 1600 patients were treated, 190 of them emergently. Seventy-one incident reports were submitted for 55 unique patients. Fewer events were reported for emergent treatments than for nonemergent treatments (0.37 events per new treatment vs 0.86; P < .01). Mean risk index for emergent reports was 1.90 versus 1.48 for nonemergent reports (P < .01). Rate of NMRI 4 was 10% for emergent treatments versus 4% for nonemergent treatments (P < .01). Emergent treatments started on a weekend or holiday had a higher proportion of critical near-miss events than emergent treatments started during the week (37% vs 7.9%, P = .034). CONCLUSIONS In this study, fewer near-miss incidents were reported per treatment course for emergent treatments. This may be attributable to reporting bias. More importantly, when emergent near misses occur, they are of greater severity.

SU-E-T-310: Targeting Safety Improvements Through Analysis of Near-Miss Error Detection Points in An Incident Learning Database

PURPOSE Radiation treatment planning involves a complex workflow that can make safety improvement efforts challenging. This study utilizes an incident reporting system to identify detection points of near-miss errors, in order to guide our departmental safety improvement efforts. Previous studies have examined where errors arise, but not where they are detected or their patterns. METHODS 1377 incidents were analyzed from a departmental nearmiss error reporting system from 3/2012-10/2013. All incidents were prospectively reviewed weekly by a multi-disciplinary team, and assigned a near-miss severity score ranging from 0-4 reflecting potential harm (no harm to critical). A 98-step consensus workflow was used to determine origination and detection points of near-miss errors, categorized into 7 major steps (patient assessment/orders, simulation, contouring/treatment planning, pre-treatment plan checks, therapist/on-treatment review, post-treatment checks, and equipment issues). Categories were compared using ANOVA. RESULTS In the 7-step workflow, 23% of near-miss errors were detected within the same step in the workflow, while an additional 37% were detected by the next step in the workflow, and 23% were detected two steps downstream. Errors detected further from origination were more severe (p<.001; Figure 1). The most common source of near-miss errors was treatment planning/contouring, with 476 near misses (35%). Of those 476, only 72(15%) were found before leaving treatment planning, 213(45%) were found at physics plan checks, and 191(40%) were caught at the therapist pre-treatment chart review or on portal imaging. Errors that passed through physics plan checks and were detected by therapists were more severe than other errors originating in contouring/treatment planning (1.81 vs 1.33, p<0.001). CONCLUSION Errors caught by radiation treatment therapists tend to be more severe than errors caught earlier in the workflow, highlighting the importance of safety checks in dosimetry and physics. We are utilizing our findings to improve manual and automated checklists for dosimetry and physics.

MO‐G‐BRE‐06: Metrics of Success: Measuring Participation and Attitudes Related to Near‐Miss Incident Learning Systems

PURPOSE Interest in incident learning systems (ILS) for improving safety and quality in radiation oncology is growing, as evidenced by the upcoming release of the national ILS. However, an institution implementing such a system would benefit from quantitative metrics to evaluate performance and impact. We developed metrics to measure volume of reporting, severity of reported incidents, and changes in staff attitudes over time from implementation of our institutional ILS. METHODS We analyzed 2023 incidents from our departmental ILS from 2/2012-2/2014. Incidents were prospectively assigned a near-miss severity index (NMSI) at multidisciplinary review to evaluate the potential for error ranging from 0 to 4 (no harm to critical). Total incidents reported, unique users reporting, and average NMSI were evaluated over time. Additionally, departmental safety attitudes were assessed through a 26 point survey adapted from the AHRQ Hospital Survey on Patient Safety Culture before, 12 months, and 24 months after implementation of the incident learning system. RESULTS Participation in the ILS increased as demonstrated by total reports (approximately 2.12 additional reports/month) and unique users reporting (0.51 additional users reporting/month). Also, the average NMSI of reports trended lower over time, significantly decreasing after 12 months of reporting (p<0.001) but with no significant change at months 18 or 24. In survey data significant improvements were noted in many dimensions, including perceived barriers to reporting incidents such as concern of embarrassment (37% to 18%; p=0.02) as well as knowledge of what incidents to report, how to report them, and confidence that these reports were used to improve safety processes. CONCLUSION Over a two-year period, our departmental ILS was used more frequently, incidents became less severe, and staff confidence in the system improved. The metrics used here may be useful for other institutions seeking to create or evaluate their own incident learning systems.

SU‐E‐T‐237: Patient Safety Improvement with a Software Tool to Prevent Isocenter Errors

Purpose: Based on one year of departmental experience with near‐miss incident learning, issues with isocenter placement were identified as a significant potential risk. Motivated by this, we initiated a patient safety improvement intervention which employs modern concepts in human factors engineering, error prevention, and software design. Methods: In February 2012 a departmental electronic incident learning system was launched that is unique in three respects: the high volume of near‐miss/no‐harm reports (20 per week or ∼1 per patient), the ability to tag each event by category type, and ranking of events by potential severity (0‐to‐4 point scale). Data over ten months (774 reports) indicate that near‐miss events related to isocenter placement have a significantly higher potential severity compared to other events: 3.0+−1.3 vs. 1.5+−1.0 (p<0.001). We therefore developed a custom software interface to manage the placement of isocenters through the clinical workflow. The approach draws from concepts in the psychology of human factors design and usability, including forcing functions and automatic error checking. Results: The custom software interface is designed to accommodate all common workflow scenarios for isocenter placement. The user is presented with a single screen containing all relevant information and is guided with a color‐coded status scheme. The software checks for common error pathways (e.g. isocenter point inadvertently moved during planning) and changes the interface status accordingly. Further evaluation is ongoing, after which wider distribution may be possible. Conclusion: Incident learning is a valuable method for objectively identifying areas for safety improvement. The critical safety issue identified here, isocenter placement, was addressed with a custom software interface that utilizes approaches not commonly employed by the vendor community such as user‐centered design. The design bridges the gap of execution and evaluation which lies at the root of most errors.

SU‐E‐T‐230: Patient Safety Improvement Related to Changes in Ongoing Radiation Treatment Plan Identified with Near‐Miss Incidents Reporting

Purpose: A significant portion of patients undergoing radiation experience a change in the treatment plan during the treatment course for a variety of reasons, including adaptive planning due to tumor response and change in fractionation due to patient clinical status. This study tests whether mid‐course changes to treatment plans increase risk of errors through the use of a large institutional near‐miss incident reporting system. Methods: We analyzed incidents from a departmental near‐miss incident reporting system launched in 2/2012. All incidents were prospectively reviewed weekly by a multi‐disciplinary team including physicians, therapists, dosimetrists, physicists, nurses, and administrative staff. Incidents were assigned a near‐miss severity score ranging from 0 to 4 (no impact/mild/moderate/severe/critical), reflecting the potential harm of the incident if it had reached the patient. Monthly root‐case‐analysis is performed on incidents with the highest severity. Incidents related to change‐in‐plan were flagged, and severity score for these were compared to other incidents via t‐test. Results: From 2/2012 through 12/2012, 662 incidents were submitted through the departmental near‐miss incident reporting system. On multi‐disciplinary review, 59(9%) incidents were directly attributable to a change‐in‐plan. Average severity score for the 59 change‐in‐plan incidents was 1.9, significantly higher than score of 1.5 for other incidents not related to change‐in‐plan (p=0.02). Three(5%) near‐miss incidents were assigned the highest severity score of 4, with issues including: wrong isocenter, wrong fields assigned to new plan, and wrong information in new treatment plan due to existing plan. Conclusion: Changes in treatment plan are sometimes required to provide highest quality of care for patients receiving radiation. Although near‐miss incidents related to change‐in‐plan are relatively uncommon, when they do occur they are more severe than other types of incidents that are observed in the course of clinical operations. Development is underway for new safety process specifically tailored to this identified high‐risk patient group.