Nancy L. Dudek

The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE): A Tool to Assess Surgical Competence

Purpose Most assessment of surgical trainees is based on measures of knowledge, with limited evaluation of their competence to actually perform various surgical procedures. In this study, the authors evaluated a tool they designed to assess a trainee’s competence to perform an entire surgical procedure independently, regardless of procedure type or postgraduate year (PGY). Method In phase 1, the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) was piloted in the University of Ottawa’s Division of Orthopaedic Surgery. In phase 2, the refined 11-item tool (8 items rated on a 5-point competency scale, 1 item assessing procedural competence, 2 feedback items) was used in the Divisions of Orthopaedic Surgery and General Surgery to assess residents’ performance on 11 common procedures. Quantitative and qualitative analyses were conducted. Results In phase 2, 34 orthopaedic and general surgeons assessed the performance of 37 residents in 163 procedures. ANOVA demonstrated an effect of PGY. Post hoc analysis found that total procedure scores for PGYs 1 and 2 were lower than those for PGY 3 (P < .001), and PGY 3 scores were lower than those for PGYs 4 and 5 (P < .02). Analysis of qualitative data indicated that the rating scale was practical and useful for surgeons and residents. Conclusions This novel evaluation tool successfully discriminated between junior and senior residents and identified surgical competency across various PGY levels regardless of procedure type. Multiple sources of evidence support the O-SCORE as a valid tool for the assessment of trainee operative competency.

Entrustability Scales: Outlining Their Usefulness for Competency-Based Clinical Assessment.

Meaningful residency education occurs at the bedside, along with opportunities for situated in-training assessment. A necessary component of workplace-based assessment (WBA) is the clinical supervisor, whose subjective judgments of residents’ performance can yield rich and nuanced ratings but may also occasionally reflect bias. How to improve the validity of WBA instruments while simultaneously capturing meaningful subjective judgment is currently not clear. This Perspective outlines how “entrustability scales” may help bridge the gap between the assessment judgments of clinical supervisors and WBA instruments. Entrustment-based assessment evaluates trainees against what they will actually do when independent; thus, “entrustability scales”—defined as behaviorally anchored ordinal scales based on progression to competence—reflect a judgment that has clinical meaning for assessors. Rather than asking raters to assess trainees against abstract scales, entrustability scales provide raters with an assessment measure structured around the way evaluators already make day-to-day clinical entrustment decisions, which results in increased reliability. Entrustability scales help raters make assessments based on narrative descriptors that reflect real-world judgments, drawing attention to a trainee’s readiness for independent practice rather than his/her deficiencies. These scales fit into milestone measurement both by allowing an individual resident to strive for independence in entrustable professional activities across the entire training period and by allowing residency directors to identify residents experiencing difficulty. Some WBA tools that have begun to use variations of entrustability scales show potential for allowing raters to produce valid judgments. This type of anchor scale should be brought into wider circulation.

Skin problems in an amputee clinic.

Dudek NL, Marks MB, Marshall SC: Skin problems in an amputee clinic. Am J Phys Med Rehabil 2006;85:424–429. Objective:To document the type and frequency of individual residual limb skin problems among patients using a lower extremity prosthesis, including the suggested etiology and management of each type of skin problem. Design:This is a 6-yr retrospective chart review of skin lesions diagnosed in patients examined in an outpatient amputee clinic at a regional, referral rehabilitation hospital in Ottawa, Canada. Skin lesions were included if they were on a lower extremity residual limb for a patient who functionally used a prosthesis. Descriptive statistics were used to analyze data. Results:A total of 528 skin problems were documented in 337 lower extremity residual limbs. Ulcers, irritations, inclusion cysts, calluses, and verrucous hyperplasia were the five most common skin problems representing 79.5% of all documented skin disorders. Conclusions:This study demonstrated that a wide variety of dermatologic conditions occurred frequently in the lower extremity amputee who functionally used a prosthesis. Five types of skin problems accounted for nearly 80% of the skin lesions identified. Future studies are required to evaluate prevention and management of the most frequent skin problems.

Indicators of dynamic stability in transtibial prosthesis users

An improved understanding of factors related to dynamic stability in lower-limb prosthesis users is important, given the high occurrence of falls in this population. Current methods of assessing stability are unable to adequately characterize dynamic stability over a variety of walking conditions. F-Scan Mobile has been used to collect plantar pressure data and six extracted parameters were useful measures of dynamic stability. The aim of this study was to investigate dynamic stability in individuals with unilateral transtibial amputation based on these six parameters. Twenty community ambulators with a unilateral transtibial amputation walked over level ground, uneven ground, stairs, and a ramp while plantar pressure data were collected. For each limb (intact and prosthetic) and condition, six stability parameters related to plantar center-of-pressure perturbations and gait temporal parameters, were computed from the plantar pressure data. Parameter values were compared between limbs, walking condition, and groups (unilateral transtibial prosthesis users and able-bodied subjects). Differences in parameters were found between limbs and conditions, and between prosthesis users and able-bodied individuals. Further research could investigate optimizing parameter calculations for unilateral transtibial prosthesis users and define relationships between potential for falls and the dynamic stability measures.

Bone overgrowth in the adult traumatic amputee.

Dudek NL, DeHaan MN, Marks MB: Bone overgrowth in the adult traumatic amputee. Am J Phys Med Rehabil 2003;82:897–900. Bone overgrowth of the residual limb after an amputation is a well documented complication in the pediatric amputee population. Bone overgrowth can cause pain, problems with skin breakdown, and poor prosthetic fit. There have been few reports of bone overgrowth in the adult amputee. Two cases of traumatic transfemoral amputations after extensive tissue damage are presented. Both patients successfully completed an in-patient amputee rehabilitation program and achieved functional ambulation with their prostheses. However, each developed distal residual limb pain within a year after their amputations that significantly limited the amount of time they could wear their prostheses and the distance they could walk. Radiographs demonstrated additional bone growth from the residual femur into adjacent soft tissues in both patients. These case examples demonstrate that bone overgrowth should be considered in the differential diagnosis of residual limb pain in the adult amputee.

Assessing the quality of supervisors’ completed clinical evaluation reports

Context  Although concern has been raised about the value of clinical evaluation reports for discriminating among trainees, there have been few efforts to formalise the dimensions and qualities that distinguish effective versus less useful styles of form completion.

Ambulation monitoring of transtibial amputation subjects with patient activity monitor versus pedometer

Our study aimed to compare the accuracy of step count and ambulation distance determined with the Yamax Digi-Walker SW-700 pedometer (DW) and the Ossur patient activity monitor (PAM) in 20 transtibial amputation subjects who were functioning at the K3 Medicare Functional Classification Level. Subjects completed four simulated household tasks in an apartment setup and a gymnasium walking course designed to simulate outdoor walking without the presence of environmental barriers or varied terrain. The mean step count accuracy of the DW and the PAM was equivalent for both the household activity (75.3% vs 70.6%) and the walking course (93.8% vs 94.0%). The mean distance measurement accuracy was better with the DW than with the PAM (household activity: 72.8% vs 0%, walking course: 92.5% vs 86.3%; p < 0.05). With acceptable step count accuracy, both devices are appropriate for assessing relatively continuous ambulation. The DW may be preferred for its more accurate distance measurements. Neither device is ideal for monitoring in-home ambulation.

Going places: Does the two-minute walk test predict the six-minute walk test in lower extremity amputees?

OBJECTIVE Assessing a patients ability to walk the distance required for community ambulation (at least 300 m) is important in amputee rehabilitation. During the 2-min walk test, most amputees cannot walk 300 m. Thus, the 6-min walk test may be preferred, but it has not been fully validated in this population. This study examined the convergent and discriminative validity of the 6-min walk test and assessed whether the 2-min test could predict the results of the 6-min test. METHODS A total of 86 patients with unilateral or bilateral amputations at the Syme, transtibial, knee disarticulation or transfemoral level completed the 6-min walk test, 2-min walk test, Timed Up and Go test, Locomotor Capabilities Index version 5, Houghton Scale of Prosthetic Use, and Activity-Specific Balance Confidence scale. RESULTS The 6-min walk test correlated with the other tests (R = 0.57-0.95), demonstrating convergent validity. It demonstrated discriminative validity with respect to age, aetiology of amputation, and K-level (p < 0.0001). The 2-min walk test was highly predictive of the 6-min walk test distance (R2 = 0.91). CONCLUSION The 6-min walk test is a valid measure of amputee ambulation. However, the results suggest that it may not be necessary, since the 2-min walk test strongly predicts the 6-min walk test. Clinicians could therefore save time by using the shorter test.

Quality in-training evaluation reports--does feedback drive faculty performance?

Purpose Clinical faculty often complete in-training evaluation reports (ITERs) poorly. Faculty development (FD) strategies should address this problem. An FD workshop was shown to improve ITER quality, but few physicians attend traditional FD workshops. To reach more faculty, the authors developed an “at-home” FD program offering participants various types of feedback on their ITER quality based on the workshop content. Program impact is evaluated here. Method Ninety-eight participants from four medical schools, all clinical supervisors, were recruited in 2009–2010; 37 participants completed the study. These were randomized into five groups: a control group and four other groups with different feedback conditions. ITER quality was assessed by two raters using a validated tool: the completed clinical evaluation report rating (CCERR). Participants were given feedback on their ITER quality based on group assignment. Six months later, participants submitted new ITERs. These ITERs were assessed using the CCERR, and feedback was sent to participants on the basis of their group assignment. This process was repeated two more times, ending in 2012. Results CCERR scores from the participants in all feedback groups were collapsed (n=27) and compared with scores from the control group (n = 10). Mean CCERR scores significantly increased over time forthe feedback group but not the controlgroup. Conclusions The results suggest that faculty are able to improve ITER quality following a minimal “at-home” FD intervention. This also adds to the growing literature that has found success with improving the quality of trainee assessments following rater training.

Quality evaluation reports: Can a faculty development program make a difference?

Background: The quality of medical student and resident clinical evaluation reports submitted by rotation supervisors is a concern. The effectiveness of faculty development (FD) interventions in changing report quality is uncertain. Aims: This study assessed whether faculty could be trained to complete higher quality reports. Method: A 3-h interactive program designed to improve evaluation report quality, previously developed and tested locally, was offered at three different Canadian medical schools. To assess for a change in report quality, three reports completed by each supervisor prior to the workshop and all reports completed for 6 months following the workshop were evaluated by three blinded, independent raters using the Completed Clinical Evaluation Report Rating (CCERR): a validated scale that assesses report quality. Results: A total of 22 supervisors from multiple specialties participated. The mean CCERR score for reports completed after the workshop was significantly higher (21.74 ± 4.91 versus 18.90 ± 5.00, p = 0.02). Conclusions: This study demonstrates that this FD workshop had a positive impact upon the quality of the participants’ evaluation reports suggesting that faculty have the potential to be trained with regards to trainee assessment. This adds to the literature which suggests that FD is an important component in improving assessment quality.