Marjo Maas

Effectiveness of Peer Assessment for Implementing a Dutch Physical Therapy Low Back Pain Guideline: Cluster Randomized Controlled Trial

Background Clinical practice guidelines are considered important instruments to improve quality of care. However, success is dependent on adherence, which may be improved using peer assessment, a strategy in which professionals assess performance of their peers in a simulated setting. Objective The aim of this study was to determine whether peer assessment is more effective than case-based discussions to improve knowledge and guideline-consistent clinical reasoning in the Dutch physical therapy guideline for low back pain (LBP). Design A cluster randomized controlled trial was conducted. Setting and Participants Ten communities of practice (CoPs) of physical therapists were cluster randomized (N=90): 6 CoPs in the peer-assessment group (n=49) and 4 CoPs in the case-based discussion group (control group) (n=41). Intervention Both groups participated in 4 educational sessions and used clinical patient cases. The peer-assessment group reflected on performed LBP management in different roles. The control group used structured discussions. Measurements Outcomes were assessed at baseline and at 6 months. The primary outcome measure was knowledge and guideline-consistent reasoning, measured with 12 performance indicators using 4 vignettes with specific guideline-related patient profiles. For each participant, the total score was calculated by adding up the percentage scores (0–100) per vignette, divided by 4. The secondary outcome measure was reflective practice, as measured by the Self-Reflection and Insight Scale (20–100). Results Vignettes were completed by 78 participants (87%). Multilevel analysis showed an increase in guideline-consistent clinical reasoning of 8.4% in the peer-assessment group, whereas the control group showed a decline of 0.1% (estimated group difference=8.7%, 95% confidence interval=3.9 to 13.4). No group differences were found on self-reflection. Limitations The small sample size, a short-term follow-up, and the use of vignettes as a proxy for behavior were limitations of the study. Conclusions Peer assessment leads to an increase in knowledge and guideline-consistent clinical reasoning.

An Innovative Peer Assessment Approach to Enhance Guideline Adherence in Physical Therapy: Single-Masked, Cluster-Randomized Controlled Trial

Background Clinical practice guidelines (CPGs) are not readily implemented in clinical practice. One of the impeding factors is that physical therapists do not hold realistic perceptions of their adherence to CPGs. Peer assessment (PA) is an implementation strategy that aims at improving guideline adherence by enhancing reflective practice, awareness of professional performance, and attainment of personal goals. Objective The purpose of this study was to compare the effectiveness of PA with the usual case discussion (CD) strategy on adherence to CPGs for physical therapist management of upper extremity complaints. Design A single-masked, cluster-randomized controlled trial with pretest-posttest design was conducted. Intervention Twenty communities of practice (n=149 physical therapists) were randomly assigned to groups receiving PA or CD, with both interventions consisting of 4 sessions over 6 months. Both PA and CD groups worked on identical clinical cases relevant to the guidelines. Peer assessment focused on individual performance observed and evaluated by peers; CD focused on discussion. Outcomes Guideline adherence was measured with clinical vignettes, reflective practice was measured with the Self-Reflection and Insight Scale (SRIS), awareness of performance was measured via the correlation between perceived and assessed improvement, and attainment of personal goals was measured with written commitments to change. Results The PA groups improved more on guideline adherence compared with the CD groups (effect=22.52; 95% confidence interval [95% CI]=2.38, 42.66; P=.03). The SRIS scores did not differ between PA and CD groups. Awareness of performance was greater for the PA groups (r=.36) than for the CD groups (r=.08) (effect=14.73; 95% CI=2.78, 26.68; P=.01). The PA strategy was more effective than the CD strategy in attaining personal goals (effect=0.50; 95% CI=0.04, 0.96; P=.03). Limitations Limited validity of clinical vignettes as a proxy measure of clinical practice was a limitation of the study. Conclusions Peer assessment was more effective than CD in improving adherence to CPGs. Personal feedback may have contributed to its effectiveness. Future research should address the role of the group coach.

Critical features of peer assessment of clinical performance to enhance adherence to a low back pain guideline for physical therapists: a mixed methods design

BackgroundClinical practice guidelines are intended to improve the process and outcomes of patient care. However, their implementation remains a challenge. We designed an implementation strategy, based on peer assessment (PA) focusing on barriers to change in physical therapy care. A previously published randomized controlled trial showed that PA was more effective than the usual strategy “case discussion” in improving adherence to a low back pain guideline. Peer assessment aims to enhance knowledge, communication, and hands-on clinical skills consistent with guideline recommendations. Participants observed and evaluated clinical performance on the spot in a role-play simulating clinical practice. Participants performed three roles: physical therapist, assessor, and patient. This study explored the critical features of the PA program that contributed to improved guideline adherence in the perception of participants.MethodsDutch physical therapists working in primary care (n = 49) organized in communities of practice (n = 6) participated in the PA program. By unpacking the program we identified three main tasks and eleven subtasks. After the program was finished, a questionnaire was administered in which participants were asked to rank the program tasks from high to low learning value and to describe their impact on performance improvement. Overall ranking results were calculated. Additional semi-structured interviews were conducted to elaborate on the questionnaires results and were transcribed verbatim. Questionnaires comments and interview transcripts were analyzed using template analysis.ResultsProgram tasks related to performance in the therapist role were perceived to have the highest impact on learning, although task perceptions varied from challenging to threatening. Perceptions were affected by the role-play format and the time schedule. Learning outcomes were awareness of performance, improved attitudes towards the guideline, and increased self-efficacy beliefs in managing patients with low back pain. Learning was facilitated by psychological safety and the quality of feedback.ConclusionThe effectiveness of PA can be attributed to the structured and performance-based design of the program. Participants showed a strong cognitive and emotional commitment to performing the physical therapist role. That might have contributed to an increased awareness of strength and weakness in clinical performance and a motivation to change routine practice.

Why peer assessment helps to improve clinical performance in undergraduate physical therapy education : A mixed methods design

BackgroundPeer Assessment (PA) in health professions education encourages students to develop a critical attitude towards their own and their peers’ performance. We designed a PA task to assess students’ clinical skills (including reasoning, communication, physical examination and treatment skills) in a role-play that simulated physical therapy (PT) practice. Students alternately performed in the role of PT, assessor, and patient. Oral face-to-face feedback was provided as well as written feedback and scores.This study aims to explore the impact of PA on the improvement of clinical performance of undergraduate PT students.MethodsThe PA task was analyzed and decomposed into task elements. A qualitative approach was used to explore students’ perceptions of the task and the task elements. Semi-structured interviews with second year students were conducted to explore the perceived impact of these task elements on performance improvement. Students were asked to select the elements perceived valuable, to rank them from highest to lowest learning value, and to motivate their choices. Interviews were transcribed verbatim and analyzed, using a phenomenographical approach and following template analysis guidelines. A quantitative approach was used to describe the ranking results.ResultsQuantitative analyses showed that the perceived impact on learning varied widely. Performing the clinical task in the PT role, was assigned to the first place (1), followed by receiving expert feedback (2), and observing peer performance (3). Receiving peer feedback was not perceived the most powerful task element.Qualitative analyses resulted in three emerging themes: pre-performance, true-performance, and post-performance triggers for improvement. Each theme contained three categories: learning activities, outcomes, and conditions for learning.Intended learning activities were reported, such as transferring prior learning to a new application context and unintended learning activities, such as modelling a peer’s performance. Outcomes related to increased self-confidence, insight in performance standards and awareness of improvement areas. Conditions for learning referred to the quality of peer feedback.ConclusionsPA may be a powerful tool to improve clinical performance, although peer feedback is not perceived the most powerful element. Peer assessors in undergraduate PT education use idiosyncratic strategies to assess their peers’ performance.

Development and Evaluation of an Implementation Strategy for Collecting Data in a National Registry and the Use of Patient-Reported Outcome Measures in Physical Therapist Practices: Quality Improvement Study

Background. In 2013, the Royal Dutch Society for Physical Therapy launched the program “Quality in Motion.” This program aims to collect data from electronic health record systems in a registry that is fed back to physical therapists, facilitating quality improvement. Purpose. The purpose of this study was to describe the development of an implementation strategy for the program and to evaluate the feasibility of building a registry and implementing patient‐reported outcome measures (PROMs) in physical therapist practices. Methods. A stepwise approach using mixed methods was established in 3 consecutive pilots with 355 physical therapists from 66 practices. Interim results were evaluated using quantitative data from a self‐assessment questionnaire and the registry and qualitative data from 21 semistructured interviews with physical therapists. Descriptive statistics and McNemars symmetry chi‐squared test were used to summarize the feasibility of implementing PROMs. Results. PROMs were selected for the 5 most prevalent musculoskeletal conditions in Dutch physical therapist practices. A core component of the implementation strategy was the introduction of knowledge brokers to support physical therapists in establishing the routine use of PROMs in clinical practice and to assist in executing peer assessment workshops. In February 2013, 30.3% of the physical therapist practices delivered 4.4 completed treatment episodes per physical therapist to the registry; this increased to 92.4% in November 2014, delivering 54.1 completed patient episodes per physical therapist. Pre‐ and posttreatment PROM use increased from 12.2% to 39.5%. Limitations. It is unclear if the participating physical therapists reflect a representative sample of Dutch therapists. Conclusion. Building a registry and implementing PROMs in physical therapist practices are feasible. The routine use of PROMs needs to increase to ensure valid feedback of outcomes. Using knowledge brokers is promising for implementing the program via peer assessment workshops.

Feasibility of peer assessment and clinical audit to self-regulate the quality of physiotherapy services: a mixed methods study

Objectives To evaluate the feasibility of a quality improvement programme aimed to enhance the client-centeredness, effectiveness and transparency of physiotherapy services by addressing three feasibility domains: (1) acceptability of the programme design, (2) appropriateness of the implementation strategy and (3) impact on quality improvement. Design Mixed methods study. Participants and setting 64 physiotherapists working in primary care, organised in a network of communities of practice in the Netherlands. Methods The programme contained: (1) two cycles of online self-assessment and peer assessment (PA) of clinical performance using client records and video-recordings of client communication followed by face-to-face group discussions, and (2) clinical audit assessing organisational performance. Assessment was based on predefined performance indicators which could be scored on a 5-point Likert scale. Discussions addressed performance standards and scoring differences. All feasibility domains were evaluated qualitatively with two focus groups and 10 in-depth interviews. In addition, we evaluated the impact on quality improvement quantitatively by comparing self-assessment and PA scores in cycles 1 and 2. Results We identified critical success features relevant to programme development and implementation, such as clarifying expectations at baseline, training in PA skills, prolonged engagement with video-assessment and competent group coaches. Self-reported impact on quality improvement included awareness of clinical and organisational performance, improved evidence-based practice and client-centeredness and increased motivation to self-direct quality improvement. Differences between self-scores and peer scores on performance indicators were not significant. Between cycles 1 and 2, scores for record keeping showed significant improvement, however not for client communication. Conclusions This study demonstrated that bottom-up initiatives to improve healthcare quality can be effective. The results justify ongoing evaluation to inform nationwide implementation when the critical success features are addressed. Further research is necessary to explore the sustainability of the results and the impact on client outcomes in a full-scale study.

Impact of Self- and Peer Assessment on the Clinical Performance of Physiotherapists in Primary Care: A Cohort Study

Purpose: This study evaluated the impact of a quality improvement programme based on self- and peer assessment to justify nationwide implementation. Method: Four professional networks of physiotherapists in The Netherlands (n = 379) participated in the programme, which consisted of two cycles of online self-assessment and peer assessment using video recordings of client communication and clinical records. Assessment was based on performance indicators that could be scored on a 5-point Likert scale, and online assessment was followed by face-to-face feedback discussions. After cycle 1, participants developed personal learning goals. These goals were analyzed thematically, and goal attainment was measured using a questionnaire. Improvement in performance was tested with multilevel regression analyses, comparing the self-assessment and peer-assessment scores in cycles 1 and 2. Results: In total, 364 (96%) of the participants were active in online self-assessment and peer assessment. However, online activities varied between cycle 1 and cycle 2 and between client communication and recordkeeping. Personal goals addressed client-centred communication (54%), recordkeeping (24%), performance and outcome measurement (15%), and other (7%). Goals were completely attained (29%), partly attained (64%), or not attained at all (7%). Self-assessment and peer-assessment scores improved significantly for both client communication (self-assessment = 11%; peer assessment = 8%) and recordkeeping (self-assessment = 7%; peer assessment = 4%). Conclusions: Self-assessment and peer assessment are effective in enhancing commitment to change and improving clinical performance. Nationwide implementation of the programme is justified. Future studies should address the impact on client outcomes.

Serious gaming voor het vergroten van de adherentie van fysiotherapeuten en manueel therapeuten aan de richtlijn lage rugpijn : Een gerandomiseerde gecontroleerde studie

SamenvattingFysiotherapie en/of manuele therapie spelen een belangrijke rol in de zorg voor mensen met lage rugpijn. Om de kwaliteit van deze zorg te verbeteren is een richtlijn ontwikkeld die vervolgens geïmplementeerd dient te worden. Technologische innovaties zoals serious games kunnen een rol spelen bij de implementatie van deze richtlijn. In een gerandomiseerde gecontroleerde studie is onderzocht in hoeverre een serious game de implementatie van een richtlijn lage rugpijn bevorderd. Achtenveertig fysiotherapeuten/manueel therapeuten zijn at random toegewezen aan een serious game groep of een groep die voorlichting kreeg over de richtlijn. Uitkomstmaten in deze studie zijn de mate van adherentie aan de richtlijn (vignettentoets), ervaren knelpunten van implementatie, de mening van de deelnemer over de wijze van implementeren, en de mate waarin de richtlijn is gelezen en wordt toegepast in de praktijk naar eigen inschatting van de deelnemer. Na zes weken follow-up is er geen significant effect gevonden van de serious game op de mate van adherentie aan de richtlijn (0,4 punten op 100 puntsschaal; 95% betrouwbaarheidsinterval -4,0 tot 4,8) ten opzichte van de voorlichtingsgroep. Ook voor de andere uitkomsten werden geen statistisch significante effecten gevonden. Het introduceren van een serious game had derhalve geen groter effect op de mate van adherentie aan de richtlijn lage rugpijn dan voorlichting.AbstractSerious gaming to improve guideline adherence of physiotherapists in low back pain care: a randomized controlled trial Physiotherapy and manual therapy are important treatment options for low back pain. To improve quality of care a clinical guideline has been developed which needs to be implemented in clinical practice. Technological innovations such as serious games may be helpful to increase the implementation of the guideline. We studied in a randomized controlled trial the effects of a serious game on guideline adherence of physiotherapists and manual therapists. Forty-eight physiotherapists/manual therapists were randomly allocated to a serious game group or a group which received education about the guideline. Outcomes in this study were guideline adherence (scored with patient vignettes), perceived barriers for guideline use, opinion regarding the implementation method used, and the degree to which the guideline has been read and used in clinical practice. After 6 weeks no significant effects were found for guideline adherence (0,4 points on 0-100 points scale; 95% confidence interval -4.0 to 4.8) and the other outcomes. The serious game therefore had no beneficial effect on guideline adherence when compared to education about the guideline. Keywords: low back pain, physiotherapy, manual therapy, guideline, implementation, serious game