Margaret Hart

ReCAP: Improving the Quality of Radiation Treatment for Patients in Ontario: Increasing Peer Review Activities on a Jurisdictional Level Using a Change Management Approach

PURPOSE Peer review of radiation treatment (RT) plans is a key component of quality assurance programs in radiation medicine. A 2011 current state assessment identified considerable variation in the percentage of RT plans peer reviewed across Ontarios 14 cancer centers.In response, Cancer Care Ontario launched an initiative to increase peer review of plans for patients receiving radical intent RT. METHODS The initiative was designed consistent with the Kotter eight-step process for organizational transformation. A multidisciplinary team conducted site visits to promote and guide peer review and to develop education and implementation processes in collaboration with the centers. A centralized reporting infrastructure enabled the monitoring of the percentage of RT courses peer reviewed and the timing of peer review (before completion of 25%of treatment visits, after completion of >25%treatment visits). RESULTS The initiative is ongoing, but early results indicate that the proportion of radical intent RT courses peer reviewed province wide increased from 43.5% (April 2013) to 68.0%(March 2015). This proportion is now a quality metric in Ontario and is publicly reported through the Cancer System Quality Index. The performance target for this metric was initially set at 50%(cases treated with radical intent) and revised to 60% in 2014. Provincial performance exceeded targets in both years (58.2% and 68.2%, respectively). Considerable variation was observed, however, in rates and timing of peer review among Cancer Care Ontario centers. CONCLUSION This initiative demonstrates that a change management framework can be useful for planning and achieving substantial increases in jurisdictional peer review activities.

Directly Improving the Quality of Radiation Treatment Through Peer Review: A Cross-sectional Analysis of Cancer Centers Across a Provincial Cancer Program

PURPOSE To describe the outcomes of peer review across all 14 cancer centers in Ontario. METHODS AND MATERIALS We identified all peer-reviewed, curative treatment plans delivered in Ontario within a 3-month study period from 2013 to 2014 using a provincial cancer treatment database and collected additional data on the peer-review outcomes. RESULTS Considerable variation was found in the proportion of peer-reviewed plans across the centers (average 70.2%, range 40.8%-99.2%). During the study period, 5561 curative plans underwent peer review. Of those, 184 plans (3.3%) had changes recommended. Of the 184 plans, the changes were major (defined as requiring repeat planning or having a major effect on planning or clinical outcomes, or both) in 40.2% and minor in 47.8%. For the remaining 12.0%, data were missing. The proportions of recommended changes varied among disease sites (0.0%-7.0%). The disease sites with the most recommended changes to treatment plans after peer review and with the greatest potential for benefit were the esophagus (7.0%), uterus (6.7%), upper limb (6.3%), cervix and lower limb (both 6.0%), head and neck and bilateral lung (both 5.9%), right supraclavicular lymph nodes (5.7%), rectum (5.3%), and spine (5.0%). Although the heart is an organ at risk in left-sided breast treatment plans, the proportions of recommended changes did not significantly differ between the left breast treatment plans (3.0%, 95% confidence interval 2.0%-4.5%) and right breast treatment plans (2.4%, 95% confidence interval 1.5%-3.8%). The recommended changes were more frequently made when peer review occurred before radiation therapy (3.8%) than during treatment (1.4%-2.8%; P=.0048). The proportion of plans with recommended changes was not significantly associated with patient volume (P=.23), peer-review performance (P=.36), or center academic status (P=.75). CONCLUSIONS Peer review of treatment plans directly affects the quality of care by identifying important clinical and planning changes. Provincial strategies are underway to optimize its conduct in radiation oncology.

“I sleep better at night:” How peer review of radiation treatment plans indirectly improves quality of care across radiation treatment programs

PURPOSE Peer review of radiation oncology treatment plans is increasingly recognized as an important component of quality assurance in radiation treatment planning and delivery. Peer review of treatment plans can directly improve the quality of those plans and can also have indirect effects on radiation treatment programs. We undertook a systematic, qualitative approach to describing the indirect benefits of peer review, factors that were seen to facilitate or act as barriers to the implementation of peer review, and strategies to address these barriers across a provincial jurisdiction of radiation oncology programs (ROPs). METHODS AND MATERIALS Semistructured qualitative interviews were held with radiation oncology department heads and radiation therapy managers (or delegates) in all 14 ROPs in Ontario, Canada. We used a theoretically guided phenomenological qualitative approach to design and analyze the interview content. Themes were recorded by 2 independent reviewers, and any discordance was resolved by consensus. RESULTS A total of 28 interviews were completed with 32 interviewees. Twenty-two unique themes addressed perceived benefits of peer review, relating to either peer review structure (n = 3), process (n = 9), or outcome (n = 10). Of these 22 themes, 19 related to indirect benefits to ROPs. In addition, 18 themes related to factors that facilitated peer review activities and 30 themes related to key barriers to implementing peer review were identified. Findings were consistent with, and enhanced the understanding of, previous survey-based assessments of the benefits and challenges of implementing peer review programs. CONCLUSIONS Although challenges and concerns regarding the implementation of peer review were evident, the indirect benefits to radiation programs are numerous, far outweigh the implementation challenges, and strongly complement the direct individual-patient benefits that result from peer review quality assurance of radiation treatment plans.

“It looks good”: How a provincial radiation oncology program benefits from implementing peer review across cancer centers.

238 Background: Peer review (PR) in radiation oncology (RO) has been endorsed as an indicator of treatment quality in North America and internationally. The direct benefits of PR include quality assurance (QA) on individual treatment plans. The indirect benefits for radiation oncology departments or programs (ROPs) have been postulated but not systematically evaluated. We used a rigorous and comprehensive qualitative approach to explore the indirect benefits of PR across a jurisdiction of cancer care, and to identify factors that facilitate PR, barriers to implementation, and strategies for expansion of PR across centers. METHODS Semi-structured qualitative interviews were held with all RO heads and Radiation therapy Managers (or delegate) in all 14 Radiation Oncology Programs (ROPs) in Ontario, Canada. The interview questions were developed using a Knowledge Translation Theoretical Domains Framework (TDF), guided by the results of a previous survey of Ontario cancer centers and by expert consensus. Interviews were audiotaped and abstracted for relevant themes by two independent analysts. RESULTS All interviewees endorsed numerous direct and indirect benefits of PR, and identified multiple facilitators and barriers to the implementation of PR. Thematic saturation was achieved. The structure-process-outcome model was used to categorize the results. Key findings included the identification of 34 independent benefits (structure n = 4, process n = 17, outcome n = 13), 40 key barriers (structure n = 9, process n = 26, outcome n = 5), and 22 facilitators (structure n = 4, process n = 15, outcome n = 3). Beyond QA, commonly endorsed benefits included enhanced knowledge sharing, efficiency, standardization, and education. Multiple potential strategies for the expansion of PR activities were revealed. CONCLUSIONS The qualitative exploration of Ontario ROPs acknowledges that multiple barriers and facilitators to PR exist while clearly establishing the indirect benefit of PR on ROPs. Understanding reported barriers and facilitators and exploration of suggested strategies will inform continued implementation and expansion of PR activities, and seem generalizable to other jurisdictions.

Toward quality peer review: Outcomes of peer review across provincial radiation oncology programs.

205 Background: Review of treatment plans by a second radiation oncologist is an important quality indicator in radiation oncology. Peer review (PR) can improve quality of care in individual patients by detecting clinical and planning issues and recommending plan changes. This study reports the frequency and nature of these changes across all 14 radiation oncology programs (ROPs) in Ontario, Canada. METHODS We identified all peer-reviewed curative treatment plans delivered in Ontario within a 3-month study period between Dec 2013-Nov 2014 using Cancer Care Ontarios Activity Level Reporting System, where data on treatment intent and date, disease site treated, PR status, timing of PR, and nature of recommended changes were available. RESULTS There was considerable variation in the proportion of plans peer-reviewed across ROPs (70.2%, range: 40.8-99.2%). Over the study period, 5,561 curative treatment plans were peer-reviewed and 3.3% had changes recommended. Of those, 21.0% had major clinical and re-planning implications. Recommended changes most often involved minor (63.1%) vs major (36.9%) re-planning implications. Highest proportions of changes were recommended for the treatment of the esophagus, uterus, upper limb, cervix, lower limb, H&N, bilateral lung, right supraclavicular nodes, rectum, and spine (5.0%-7.0%). Plans involving the left breast had slightly more changes recommended (3.0% [95%CI:2.0%-4.5%]) vs right breast (2.4% [95%CI:1.5%-3.8%]). Recommendations were more frequently made when PR was conducted pre-radiotherapy (3.8%) vs during (1.4%-2.8%; p = 0.005), however the nature and implementation of changes were not statistically associated with the timing of PR (p = 0.91; p = 0.23, respectively). Proportion of recommended changes to treatment plans was not statistically associated with ROP patient volume (p = 0.08), proportion of plans peer-reviewed (p = 0.36) or academic status (p = 0.75). CONCLUSIONS Significant variation exists in the proportion of recommended changes across all disease sites and ROPs. PR seems effective in detecting treatment plans with important clinical and planning issues; strategies should be developed to optimize its conduct in radiation oncology.

Communities of practice: A jurisdictional approach to improving the quality of care in radiation medicine in Ontario.

122 Background: The Radiation Treatment Program (RTP) at Cancer Care Ontario (CCO) established several Communities of Practice (CoPs), with the goal of improving radiation treatment (RT) quality and safety. The RTP identifies variation in practice and quality improvement (QI) opportunities in the 14 Regional Cancer Centres (RCCs) and facilitates the development of CoPs to share best practices and standardize care. METHODS Since 2010, the RTP has formed 7 CoPs ( > 185 members in total): 4 intra-disciplinary (Radiation Therapy, Medical Physics, Advanced Practice Radiation Therapy, Radiation Safety) and 3 inter-disciplinary (Head and Neck (HN), Gynecological (GYNE) and Lung Cancer). Members are recruited with the aim of securing engagement from all RCCs to ensure representation of regional diversity and to facilitate adoption of best practices. CoPs are supported with nominal funding and resources provided by CCO, but are led and driven by members, who identify and prioritize key quality issues and select corresponding QI projects to pursue. The RTP performs regular evaluation activities to assess initiative engagement and impact. RESULTS RTP CoPs have enhanced the quality and safety of RT delivery in Ontario through QI initiatives, advice documents and tools that have enabled: Improved RT safety (use of safety straps in RT delivery); Adoption of best practices (RT plan evaluation guidance); Education and knowledge transfer - (stereotactic body RT implementation and training framework); and Support for infrastructure improvements (recommendation for additional Magnetic Resonance-guided brachytherapy units) ( https://www.cancercare.on.ca/ocs/clinicalprogs/radiationtreatment/ ). Advice documents have improved alignment with recommended practice (40% and 50% absolute increases in two HN initiatives). Evaluation surveys indicate that members believe the CoPs have enhanced inter-regional communication and collaboration (89%), knowledge transfer/exchange (91%), and professional networking between RCCs (92%). CONCLUSIONS CoPs can be a highly effective model for improving quality of care. The establishment of CoPs should be considered for QI in other areas of the healthcare system.

An approach to implementing meaningful change in a provincial health care system.

120 Background: Radiation treatment (RT) is essential to cancer management, contributing to cure and symptom control. With increasing cancer incidence and treatment complexity, health systems must adapt to ensure patients (pts) receive the highest quality of care. METHODS With the objective of ensuring equitable access to high-quality, safe care, Cancer Care Ontario (CCO), a provincial government agency, identified provincial variability in RT activities. As a result, CCO prioritized 3 quality initiatives over the past 7 years: 1) Access to Intensity Modulated RT (IMRT) (2008-2013); 2) Peer Review of RT plans due to increasing RT planning complexity and the existence of high-profile RT errors (2012-present); and 3) Ensuring equitable access to RT (RT Utilization) (2014-present). Strategic plans were developed using change management framework adapted from the Kotter process for leading change (Kotter, JP. Harvard Bus Rev 73:59-67, 1995). In each initiative, CCO created a climate for change, engaged the provincial RT community to move priorities forward, and worked to sustain achieved gains. RESULTS CCO found that building a project team, communicating a clear understanding of goals and objectives, providing sufficient resources to cancer centres, and public reporting of results were key contributing success factors. IMRT project: Currently in sustainability phase. IMRT rates increased from 20% in 2008/09 - full implementation and target attainment in 2012/13. Public reporting continues. Peer Review of RT plans: Currently moving from engagement to implementation phase. Increase from 44% of RT cases undergoing peer review in 2013/14 to 68% in 2014/15. RT Utilization Project: Currently in engagement phase. Provincial shortfall of 11% in annual RT rates correlates to roughly 2500 pts who do not receive RT as needed. Engaging data experts and consulting with regional administrators, RT utilization is the current change priority for CCOs RT program. CONCLUSIONS These projects demonstrate the possibility of using change management practices to achieve quality improvement in healthcare. Ongoing work continues to ensure that pts in Ontario receive the highest quality cancer care.

Improving the quality of care for patients receiving radiation therapy: Increasing the proportion of radiation treatment plans undergoing peer review in Ontario.

136 Background: Peer review of radiation treatment (RT) plans is recognized as an essential component of quality assurance programs in radiation medicine (Marks et al., 2013). The benefits of peer review include: (1) identifying errors that may compromise treatment outcomes, (2) enhancing safety by promoting standardization, and (3) promoting greater attention to detail in RT staff. Current state analysis conducted in 2011 identified considerable variation in the proportion of cases undergoing peer review across Ontarios 14 cancer centres (Brundage et al., 2013). In 2012, Cancer Care Ontario launched an initiative to ensure all patients receiving radical/adjuvant radiotherapy have the benefit of peer review of their RT plans. METHODS A multi-professional project team was established to conduct site visits to promote peer review at the cancer centres. They also provided guidance on the organization of peer review rounds so that the activity could be incorporated into local workflows. The education, training, methods, and a centralized reporting infrastructure were developed in collaboration with centres over a one year ramp-up phase and patient-level data was available to the centres for audit purposes. The reporting infrastructure enabled reporting of (1) the proportion of cases peer reviewed and (2) the timing of peer review - prior to treatment, <25% dose delivered, >25% dose delivered. RESULTS Data for each centre is now a key quality metric and is publicly reported (see Cancer System Quality Index at http://www.csqi.on.ca/). The target for year-one of the project (2013-14) - the proportion of cases to be peer reviewed - was set at 50% with the intent that 100% of cases will be peer reviewed within the next two years. In the ramp-up year, the proportion of cases peer reviewed increased across all centres, though high variation still exists between centres. CONCLUSIONS This initiative demonstrates that it is possible to substantially increase peer review activities on a jurisdictional basis. Key success factors include: a dedicated project team, buy-in and confidence in data quality from centres, investment in education and training, and commitment to public reporting.