Kathleen W. Bartlett

Milestone-Based Assessments Are Superior to Likert-Type Assessments in Illustrating Trainee Progression

BACKGROUND The Pediatrics Milestone Project uses behavioral anchors, narrative descriptions of observable behaviors, to describe learner progression through the Accreditation Council for Graduate Medical Education competencies. Starting June 2014, pediatrics programs were required to submit milestone reports for their trainees semiannually. Likert-type scale assessment tools were not designed to inform milestone reporting, creating a challenge for Clinical Competency Committees. OBJECTIVE To determine if milestone-based assessments better stratify trainees by training level compared to Likert-type assessments. METHODS We compared assessment results for 3 subcompetencies after changing from a 5-point Likert scale to milestone-based behavioral anchors in July 2013. Program leadership evaluated the new system by (1) comparing PGY-1 mean scores on Likert-type versus milestone-based assessments; and (2) comparing mean scores on the Likert-type versus milestone-based assessments across PGY levels. RESULTS Mean scores for PGY-1 residents were significantly higher on the prior years Likert-type assessments than milestone-based assessments for all 3 subcompetencies (P < .01). Stratification by PGY level was not observed with Likert-type assessments (eg, interpersonal and communication skills 1 [ICS1] mean score for PGY-1, 3.99 versus PGY-3, 3.98; P  =  .98). In contrast, milestone-based assessments demonstrated stratification by PGY level (eg, the ICS1 mean score was 3.06 for PGY-1, 3.83 for PGY-2, and 3.99 for PGY-3; P < .01 for PGY-1 versus PGY-3). Significantly different means by trainee level were noted across 21 subcompetencies on milestone-based assessments (P < .01 for PGY-1 versus PGY-3). CONCLUSIONS Initial results indicate milestone-based assessments stratify trainee performance by level better than Likert-type assessments. Average PGY-level scores from milestone-based assessments may ultimately provide guidance for determining whether trainees are progressing at the expected pace.

Using an Innovative Curriculum Evaluation Tool to Inform Program Improvement: The Clinical Skills Fair

BACKGROUND Program evaluation is important for assessing the effectiveness of the residency curriculum. Limited resources are available, however, and curriculum evaluation processes must be sustainable and well integrated into program improvement efforts. INTERVENTION We describe the pediatric Clinical Skills Fair, an innovative method for evaluating the effectiveness of residency curriculum through assessment of trainees in 2 domains: medical knowledge/patient care and procedure. Each year from 2008 to 2011, interns completed the Clinical Skills Fair as rising interns in postgraduate year (PGY)-1 (R1s) and again at the end of the year, as rising residents in PGY-2 (R2s). Trainees completed the Clinical Skills Fair at the beginning and end of the intern year for each cohort to assess how well the curriculum prepared them to meet the intern goals and objectives. RESULTS Participants were 48 R1s and 47 R2s. In the medical knowledge/patient care domain, intern scores improved from 48% to 65% correct (P < .001). Significant improvement was demonstrated in the following subdomains: jaundice (41% to 65% correct; P < .001), fever (67% to 94% correct; P < .001), and asthma (43% to 62% correct; P  =  .002). No significant change was noted within the arrhythmia subdomain. There was significant improvement in the procedure domain for all interns (χ(2)  =  32.82, P < .001). CONCLUSIONS The Clinical Skills Fair is a readily implemented and sustainable method for our residency program curriculum assessment. Its feasibility may allow other programs to assess their curriculum and track the impact of programmatic changes; it may be particularly useful for program evaluation committees.

Case 3: Fever, Vomiting, and Altered Mental Status in a 17-year-old Boy

1. Jonathan Pelletier, MD* 2. Kathleen Bartlett, MD* 1. *Duke Childrens Hospital and Health Center, Durham NC A 17-year-old boy with a history of restrictive eating disorder and orthostatic hypotension presents via emergency medical services from an eating disorder facility with 1 week of dizziness and confusion and 1 day of fever. He also reports months of daily first-morning, nonbloody, and nonbilious vomiting. On presentation to the emergency department he is febrile (temperature, 102.9°F [39.4°C]), tachycardic (heart rate, 112 beats/min), and hypotensive (blood pressure, 91/46 mm Hg) and requires oxygen via nasal cannula. His weight is 114.2 lb (51.8 kg) (3rd percentile), height is 64.6 in (164 cm) (5th percentile), and BMI is 19.3 (15th percentile). General physical examination findings are normal. Shortly after arrival, the patient develops a brief episode of upper extremity tonic extension, unresponsiveness, and desaturation. He recovers spontaneously without an apparent postictal period. Repeated neurologic examination reveals slowed but appropriate responses to questioning. Cranial nerves, strength, sensation, and reflexes are normal. Upper extremity coordination is normal, but he has right greater than left ataxia with heel-to-shin testing. The patient is unable to stand for gait and Romberg testing due to dizziness. Laboratory tests before fluid therapy are remarkable for a serum sodium concentration of 149 mEq/L (149 mmol/L) (reference range, 135–145 mEq/L [135–145 mmol/L]), a serum chloride concentration of 118 mEq/L (118 mmol/L) (reference range, 96–108 mEq/L [96–108 mmol/L]), leukocytosis, and normocytic anemia. Urinalysis results are normal, …

Thresholds and interpretations: How clinical competency committees identify pediatric residents with performance concerns

Abstract Background: Clinical competency committee (CCC) identification of residents with performance concerns is critical for early intervention. Methods: Program directors and 94 CCC members at 14 pediatric residency programs responded to a written survey prompt asking them to describe how they identify residents with performance concerns. Data was analyzed using thematic analysis. Results: Six themes emerged from analysis and were grouped into two domains. The first domain included four themes, each describing a path through which residents could meet or exceed a concern threshold:1) written comments from rotation assessments are foundational in identifying residents with performance concerns, 2) concerning performance extremes stand out, 3) isolated data points may accumulate to raise concern, and 4) developmental trajectory matters. The second domain focused on how CCC members and program directors interpret data to make decisions about residents with concerns and contained 2 themes: 1) using norm- and/or criterion-referenced interpretation, and 2) assessing the quality of the data that is reviewed. Conclusions: Identifying residents with performance concerns is important for their education and the care they provide. This study delineates strategies used by CCC members across several programs for identifying these residents, which may be helpful for other CCCs to consider in their efforts.

Milestone Ratings and Supervisory Role Categorizations Swim Together, but is the Water Muddy?

OBJECTIVE This single-specialty, multi-institutional study aimed to determine 1) the association between milestone ratings for individual competencies and average milestone ratings (AMRs) and 2) the association between AMRs and recommended supervisory role categorizations made by individual clinical competency committee (CCC) members. METHODS During the 2015-16 academic year, CCC members at 14 pediatric residencies reported milestone ratings for 21 competencies and recommended supervisory role categories (may not supervise, may supervise in some settings, may supervise in all settings) for residents they reviewed. An exploratory factor analysis of competencies was conducted. The associations among individual competencies, the AMR, and supervisory role categorizations were determined by computing bivariate correlations. The relationship between AMRs and recommended supervisory role categorizations was examined using an ordinal mixed logistic regression model. RESULTS Of the 155 CCC members, 68 completed both milestone assignments and supervision categorizations for 451 residents. Factor analysis of individual competencies controlling for clustering of residents in raters and sites resulted in a single-factor solution (cumulative variance: 0.75). All individual competencies had large positive correlations with the AMR (correlation coefficient: 0.84-0.93), except for two professionalism competencies (Prof1: 0.63 and Prof4: 0.65). When combined across training year and time points, the AMR and supervisory role categorization had a moderately positive correlation (0.56). CONCLUSIONS This exploratory study identified a modest correlation between average milestone ratings and supervisory role categorization. Convergence of competencies on a single factor deserves further exploration, with possible rater effects warranting attention.