Patricia A. Sponseller

Metrics of success: Measuring impact of a departmental near-miss incident learning system

PURPOSE There is a growing interest in the application of incident learning systems (ILS) to radiation oncology. The purpose of the present study is to define statistical metrics that may serve as benchmarks for successful operation of an incident learning system. METHODS AND MATERIALS A departmental safety and quality ILS was developed to monitor errors, near-miss events, and process improvement suggestions. Event reports were reviewed by a multiprofessional quality improvement committee. Events were scored by a near-miss risk index (NMRI) and categorized by event point of origination and discovery. Reporting trends were analyzed over a 2-year period, including total number and rates of events reported, users reporting, NMRI, and event origination and discovery. RESULTS A total of 1897 reports were evaluated (1.0 reports/patient, 0.9 reports/unique treatment course). Participation in the ILS increased as demonstrated by total events (2.1 additional reports/month) and unique users (0.5 new users/month). Sixteen percent of reports had an NMRI of 0 (none), 42% had an NMRI of 1 (mild), 25% had an NMRI of 2 (moderate), 12% had an NMRI of 3 (severe), and 5% had an NMRI of 4 (critical). Event NMRI showed a significant decrease in the first 6 months (1.68-1.42, P < .001). Trends in origination and discovery of reports were broadly distributed between radiation therapy process steps and staff groups. The highest risk events originated in imaging for treatment planning (NMRI = 2.0 ± 1.1; P < .0001) and were detected in on-treatment quality management (NMRI = 1.7 ± 1.1; P = .003). CONCLUSIONS Over the initial 2-year period of ILS operation, rates of reporting increased, staff participation increased, and NMRI of reported events declined. These data mirror previously reported findings of improvement in safety culture endpoints. These metrics may be useful for other institutions seeking to create or evaluate their own ILS.

Measurable improvement in patient safety culture: A departmental experience with incident learning

PURPOSE Rigorous use of departmental incident learning is integral to improving patient safety and quality of care. The goal of this study was to quantify the impact of a high-volume, departmental incident learning system on patient safety culture. METHODS AND MATERIALS A prospective, voluntary, electronic incident learning system was implemented in February 2012 with the intent of tracking near-miss/no-harm incidents. All incident reports were reviewed weekly by a multiprofessional team with regular department-wide feedback. Patient safety culture was measured at baseline with validated patient safety culture survey questions. A repeat survey was conducted after 1 and 2 years of departmental incident learning. Proportional changes were compared by χ(2) or Fisher exact test, where appropriate. RESULTS Between 2012 and 2014, a total of 1897 error/near-miss incidents were reported, representing an average of 1 near-miss report per patient treated. Reports were filed by a cross section of staff, with the majority of incidents reported by therapists, dosimetrists, and physicists. Survey response rates at baseline and 1 and 2 years were 78%, 80%, and 80%, respectively. Statistically significant and sustained improvements were noted in several safety metrics, including belief that the department was openly discussing ways to improve safety, the sense that reports were being used for safety improvement, and the sense that changes were being evaluated for effectiveness. None of the surveyed dimensions of patient safety culture worsened. Fewer punitive concerns were noted, with statistically significant decreases in the worry of embarrassment in front of colleagues and fear of getting colleagues in trouble. CONCLUSIONS A comprehensive incident learning system can identify many areas for improvement and is associated with significant and sustained improvements in patient safety culture. These data provide valuable guidance as incident learning systems become more widely used in radiation oncology.

Validating FMEA output against incident learning data: A study in stereotactic body radiation therapy

PURPOSE Though failure mode and effects analysis (FMEA) is becoming more widely adopted for risk assessment in radiation therapy, to our knowledge, its output has never been validated against data on errors that actually occur. The objective of this study was to perform FMEA of a stereotactic body radiation therapy (SBRT) treatment planning process and validate the results against data recorded within an incident learning system. METHODS FMEA on the SBRT treatment planning process was carried out by a multidisciplinary group including radiation oncologists, medical physicists, dosimetrists, and IT technologists. Potential failure modes were identified through a systematic review of the process map. Failure modes were rated for severity, occurrence, and detectability on a scale of one to ten and risk priority number (RPN) was computed. Failure modes were then compared with historical reports identified as relevant to SBRT planning within a departmental incident learning system that has been active for two and a half years. Differences between FMEA anticipated failure modes and existing incidents were identified. RESULTS FMEA identified 63 failure modes. RPN values for the top 25% of failure modes ranged from 60 to 336. Analysis of the incident learning database identified 33 reported near-miss events related to SBRT planning. Combining both methods yielded a total of 76 possible process failures, of which 13 (17%) were missed by FMEA while 43 (57%) identified by FMEA only. When scored for RPN, the 13 events missed by FMEA ranked within the lower half of all failure modes and exhibited significantly lower severity relative to those identified by FMEA (p = 0.02). CONCLUSIONS FMEA, though valuable, is subject to certain limitations. In this study, FMEA failed to identify 17% of actual failure modes, though these were of lower risk. Similarly, an incident learning system alone fails to identify a large number of potentially high-severity process errors. Using FMEA in combination with incident learning may render an improved overview of risks within a process.

State of dose prescription and compliance to international standard (ICRU-83) in intensity modulated radiation therapy among academic institutions

PURPOSE The purpose of this study was to evaluate dose prescription and recording compliance to international standard (International Commission on Radiation Units & Measurements [ICRU]-83) in patients treated with intensity modulated radiation therapy (IMRT) among academic institutions. METHODS AND MATERIALS Ten institutions participated in this study to collect IMRT data to evaluate compliance to ICRU-83. Under institutional review board clearance, data from 5094 patients-including treatment site, technique, planner, physician, prescribed dose, target volume, monitor units, planning system, and dose calculation algorithm-were collected anonymously. The dose-volume histogram of each patient, as well as dose points, doses delivered to 100% (D100), 98% (D98), 95% (D95), 50% (D50), and 2% (D2), of sites was collected and sent to a central location for analysis. Homogeneity index (HI) as a measure of the steepness of target and is a measure of the shape of the dose-volume histogram was calculated for every patient and analyzed. RESULTS In general, ICRU recommendations for naming the target, reporting dose prescription, and achieving desired levels of dose to target were relatively poor. The nomenclature for the target in the dose prescription had large variations, having every permutation of name and number contrary to ICRU recommendations. There was statistically significant variability in D95, D50, and HI among institutions, tumor site, and technique with P values < .01. Nearly 95% of patients had D50 higher than 100% (103.5 ± 6.9) of prescribed dose and varied among institutions. On the other hand, D95 was close to 100% (97.1 ± 9.4) of prescribed dose. Liver and lung sites had a higher D50 compared with other sites. Pelvic sites had a lower variability indicated by HI (0.13 ± 1.21). Variability in D50 is 101.2 ± 8.5, 103.4 ± 6.8, 103.4 ± 8.2, and 109.5 ± 11.5 for IMRT, tomotherapy, volume modulated arc therapy, and stereotactic body radiation therapy with IMRT, respectively. CONCLUSIONS Nearly 95% of patient treatments deviated from the ICRU-83 recommended D50 prescription dose delivery. This variability is significant (P < .01) in terms of treatment site, technique, and institution. To reduce dosimetric and associated radiation outcome variability, dose prescription in every clinical trial should be unified with international guidelines.

Targeting safety improvements through identification of incident origination and detection in a near-miss incident learning system.

PURPOSE Radiation treatment planning involves a complex workflow that has multiple potential points of vulnerability. This study utilizes an incident reporting system to identify the origination and detection points of near-miss errors, in order to guide their departmental safety improvement efforts. Previous studies have examined where errors arise, but not where they are detected or applied a near-miss risk index (NMRI) to gauge severity. METHODS From 3/2012 to 3/2014, 1897 incidents were analyzed from a departmental incident learning system. All incidents were prospectively reviewed weekly by a multidisciplinary team and assigned a NMRI score ranging from 0 to 4 reflecting potential harm to the patient (no potential harm to potential critical harm). Incidents were classified by point of incident origination and detection based on a 103-step workflow. The individual steps were divided among nine broad workflow categories (patient assessment, imaging for radiation therapy (RT) planning, treatment planning, pretreatment plan review, treatment delivery, on-treatment quality management, post-treatment completion, equipment/software quality management, and other). The average NMRI scores of incidents originating or detected within each broad workflow area were calculated. Additionally, out of 103 individual process steps, 35 were classified as safety barriers, the process steps whose primary function is to catch errors. The safety barriers which most frequently detected incidents were identified and analyzed. Finally, the distance between event origination and detection was explored by grouping events by the number of broad workflow area events passed through before detection, and average NMRI scores were compared. RESULTS Near-miss incidents most commonly originated within treatment planning (33%). However, the incidents with the highest average NMRI scores originated during imaging for RT planning (NMRI = 2.0, average NMRI of all events = 1.5), specifically during the documentation of patient positioning and localization of the patient. Incidents were most frequently detected during treatment delivery (30%), and incidents identified at this point also had higher severity scores than other workflow areas (NMRI = 1.6). Incidents identified during on-treatment quality management were also more severe (NMRI = 1.7), and the specific process steps of reviewing portal and CBCT images tended to catch highest-severity incidents. On average, safety barriers caught 46% of all incidents, most frequently at physics chart review, therapists chart check, and the review of portal images; however, most of the incidents that pass through a particular safety barrier are not designed to be capable of being captured at that barrier. CONCLUSIONS Incident learning systems can be used to assess the most common points of error origination and detection in radiation oncology. This can help tailor safety improvement efforts and target the highest impact portions of the workflow. The most severe near-miss events tend to originate during simulation, with the most severe near-miss events detected at the time of patient treatment. Safety barriers can be improved to allow earlier detection of near-miss events.

Interrater reliability of a near-miss risk index for incident learning systems in radiation oncology

PURPOSE Tools for assessing the severity and risk of near-miss events in radiation oncology are few and needed. Recent work has described guidelines for the use of a 5-tier near-miss risk index (NMRI) for the classification of near-miss events. The purpose of this study was to assess the reliability of the NMRI among users in a radiation oncology department. METHODS AND MATERIALS Reliability of the NMRI was assessed using an online survey distributed to members of a radiation oncology department. The survey contained 70 events extracted from the departments incident learning system (ILS). Survey participants rated each event using the NMRI guidelines, reported their attendance to weekly ILS meetings (used as a surrogate for familiarity with the ILS), and indicated their familiarity with the radiation oncology workflow. Interrater reliability was determined using Krippendorffs alpha. Use of the NMRI to rate actual events during 5 weekly ILS meetings was also assessed and interrater reliability determined. RESULTS Twenty-eight survey respondents represented a wide variety of care providers. Krippendorffs alpha was calculated for the whole respondent cohort to be 0.376, indicating fair agreement among raters. Respondents who had the most participation at ILS meetings (n = 4) had moderate agreement with an alpha of 0.501. Interestingly, there were significant differences in reliability and median NMRI scores between professions. NMRI use during weekly NMRI meetings (80 events rated), participants showed moderate reliability (alpha = 0.607). CONCLUSIONS Using the NMRI guidelines, raters from a wide variety of professions were able to assess the severity of near-miss incidents with fair agreement. Those experienced with the ILS showed better agreement, and higher agreement was seen during multidisciplinary ILS meetings. These data support the use the indices such as the NMRI for near-miss risk assessment in patient safety and prioritization of process improvements in radiation oncology.

Can emergent treatments result in more severe errors?: An analysis of a large institutional near-miss incident reporting database.

PURPOSE Emergent radiation treatments may be subject to more errors because of the compressed time frame. Few data exist on the magnitude of this problem or how to guide safety improvement interventions. The purpose of this study is to examine patterns of near-miss events in emergent treatments using a large institutional incident reporting system. METHODS AND MATERIALS Events in the incident reporting database from February 2012 to October 2013 were reviewed prospectively by a multidisciplinary team to identify emergent treatments. Reports were scored for potential near-miss risk index (NMRI) on a 0 to 4 scale. Workflow steps of where events originated and were detected were analyzed. Events were categorized by use of the causal factor system from the Radiation Oncology Incident Learning System. Mann-Whitney U tests were used to compare mean NMRI score, and Fisher exact tests were performed to compare the proportion of high-risk events between emergent and nonemergent treatments and between emergent treatments on weekdays and weekends or holidays. RESULTS Over the study period, approximately 1600 patients were treated, 190 of them emergently. Seventy-one incident reports were submitted for 55 unique patients. Fewer events were reported for emergent treatments than for nonemergent treatments (0.37 events per new treatment vs 0.86; P < .01). Mean risk index for emergent reports was 1.90 versus 1.48 for nonemergent reports (P < .01). Rate of NMRI 4 was 10% for emergent treatments versus 4% for nonemergent treatments (P < .01). Emergent treatments started on a weekend or holiday had a higher proportion of critical near-miss events than emergent treatments started during the week (37% vs 7.9%, P = .034). CONCLUSIONS In this study, fewer near-miss incidents were reported per treatment course for emergent treatments. This may be attributable to reporting bias. More importantly, when emergent near misses occur, they are of greater severity.

A ring‐based compensator IMRT system optimized for low‐ and middle‐income countries: Design and treatment planning study

Purpose We propose a novel compensator‐based IMRT system designed to provide a simple, reliable, and cost‐effective adjunct technology, with the goal of expanding global access to advanced radiotherapy techniques. The system would employ easily reusable tungsten bead compensators that operate independent of a gantry (e.g., mounted in a ring around the patient). Thereby the system can be retrofitted to existing linac and cobalt teletherapy units. This study explores the quality of treatment plans from the proposed system and the dependence on associated design parameters. Methods We considered 60Co‐based plans as the most challenging scenario for dosimetry and benchmarked them against clinical MLC‐based plans delivered on a linac. Treatment planning was performed in the Pinnacle treatment planning system with commissioning based on Monte Carlo simulations of compensated beams. 60Co‐compensator IMRT plans were generated for five patients with head‐and‐neck cancer and five with gynecological cancer and compared to respective IMRT plans using a 6 MV linac beam with an MLC. The dependence of dosimetric endpoints on compensator resolution, thickness, position, and number of beams was assessed. Dosimetric accuracy was validated by Monte Carlo simulations of dose distribution in a water phantom from beams with the IMRT plan compensators. Results The 60Co‐compensator plans had on average equivalent PTV coverage and somewhat inferior OAR sparing compared to the 6 MV‐MLC plans, but the differences in dosimetric endpoints were clinically acceptable. Calculated treatment times for head‐and‐neck plans were 7.6 ± 2.0 min vs 3.9 ± 0.8 min (6 MV‐MLC vs 60Co‐compensator) and for gynecological plans were 8.7 ± 3.1 min vs 4.3 ± 0.4 min. Plan quality was insensitive to most design parameters over much of the ranges studied, with no degradation found when the compensator resolution was finer than 6 mm, maximum thickness at least 2 tenth‐value‐layers, and more than five beams were used. Source‐to‐compensator distances of 53 and 63 cm resulted in very similar plan quality. Monte Carlo simulations suggest no increase in surface dose for the geometries considered here. Simulated dosimetric validation tests had median gamma pass rates of 97.6% for criteria of 3% (global)/3 mm with a 10% threshold. Conclusions The novel ring‐compensator IMRT system can produce plans of comparable quality to standard 6 MV‐MLC systems. Even when 60Co beams are used the plan quality is acceptable and treatment times are substantially reduced. 60Co‐compensator IMRT plans are adequately modeled in an existing commercial treatment planning system. These results motivate further development of this low‐cost adaptable technology with translation through clinical trials and deployment to expand the reach of IMRT in low‐ and middle‐income countries.

Validating FMEA output against incident learning data: A study in stereotactic body radiation therapy: Validating FMEA output against incident learning data

PURPOSE Though failure mode and effects analysis (FMEA) is becoming more widely adopted for risk assessment in radiation therapy, to our knowledge, its output has never been validated against data on errors that actually occur. The objective of this study was to perform FMEA of a stereotactic body radiation therapy (SBRT) treatment planning process and validate the results against data recorded within an incident learning system. METHODS FMEA on the SBRT treatment planning process was carried out by a multidisciplinary group including radiation oncologists, medical physicists, dosimetrists, and IT technologists. Potential failure modes were identified through a systematic review of the process map. Failure modes were rated for severity, occurrence, and detectability on a scale of one to ten and risk priority number (RPN) was computed. Failure modes were then compared with historical reports identified as relevant to SBRT planning within a departmental incident learning system that has been active for two and a half years. Differences between FMEA anticipated failure modes and existing incidents were identified. RESULTS FMEA identified 63 failure modes. RPN values for the top 25% of failure modes ranged from 60 to 336. Analysis of the incident learning database identified 33 reported near-miss events related to SBRT planning. Combining both methods yielded a total of 76 possible process failures, of which 13 (17%) were missed by FMEA while 43 (57%) identified by FMEA only. When scored for RPN, the 13 events missed by FMEA ranked within the lower half of all failure modes and exhibited significantly lower severity relative to those identified by FMEA (p = 0.02). CONCLUSIONS FMEA, though valuable, is subject to certain limitations. In this study, FMEA failed to identify 17% of actual failure modes, though these were of lower risk. Similarly, an incident learning system alone fails to identify a large number of potentially high-severity process errors. Using FMEA in combination with incident learning may render an improved overview of risks within a process.

Validating FMEA output against incident learning data

PURPOSE Though failure mode and effects analysis (FMEA) is becoming more widely adopted for risk assessment in radiation therapy, to our knowledge, its output has never been validated against data on errors that actually occur. The objective of this study was to perform FMEA of a stereotactic body radiation therapy (SBRT) treatment planning process and validate the results against data recorded within an incident learning system. METHODS FMEA on the SBRT treatment planning process was carried out by a multidisciplinary group including radiation oncologists, medical physicists, dosimetrists, and IT technologists. Potential failure modes were identified through a systematic review of the process map. Failure modes were rated for severity, occurrence, and detectability on a scale of one to ten and risk priority number (RPN) was computed. Failure modes were then compared with historical reports identified as relevant to SBRT planning within a departmental incident learning system that has been active for two and a half years. Differences between FMEA anticipated failure modes and existing incidents were identified. RESULTS FMEA identified 63 failure modes. RPN values for the top 25% of failure modes ranged from 60 to 336. Analysis of the incident learning database identified 33 reported near-miss events related to SBRT planning. Combining both methods yielded a total of 76 possible process failures, of which 13 (17%) were missed by FMEA while 43 (57%) identified by FMEA only. When scored for RPN, the 13 events missed by FMEA ranked within the lower half of all failure modes and exhibited significantly lower severity relative to those identified by FMEA (p = 0.02). CONCLUSIONS FMEA, though valuable, is subject to certain limitations. In this study, FMEA failed to identify 17% of actual failure modes, though these were of lower risk. Similarly, an incident learning system alone fails to identify a large number of potentially high-severity process errors. Using FMEA in combination with incident learning may render an improved overview of risks within a process.