Janice L. Hanson

Narrative descriptions should replace grades and numerical ratings for clinical performance in medical education in the United States

Background: In medical education, evaluation of clinical performance is based almost universally on rating scales for defined aspects of performance and scores on examinations and checklists. Unfortunately, scores and grades do not capture progress and competence among learners in the complex tasks and roles required to practice medicine. While the literature suggests serious problems with the validity and reliability of ratings of clinical performance based on numerical scores, the critical issue is not that judgments about what is observed vary from rater to rater but that these judgments are lost when translated into numbers on a scale. As the Next Accreditation System of the Accreditation Council on Graduate Medical Education (ACGME) takes effect, medical educators have an opportunity to create new processes of evaluation to document and facilitate progress of medical learners in the required areas of competence. Proposal and initial experience: Narrative descriptions of learner performance in the clinical environment, gathered using a framework for observation that builds a shared understanding of competence among the faculty, promise to provide meaningful qualitative data closely linked to the work of physicians. With descriptions grouped in categories and matched to milestones, core faculty can place each learner along the milestones continua of progress. This provides the foundation for meaningful feedback to facilitate the progress of each learner as well as documentation of progress toward competence. Implications: This narrative evaluation system addresses educational needs as well as the goals of the Next Accreditation System for explicitly documented progress. Educators at other levels of education and in other professions experience similar needs for authentic assessment and, with meaningful frameworks that describe roles and tasks, may also find useful a system built on descriptions of learner performance in actual work settings. Conclusions: We must place medical learning and assessment in the contexts and domains in which learners do clinical work. The approach proposed here for gathering qualitative performance data in different contexts and domains is one step along the road to moving learners toward competence and mastery.

Writing medical student and resident performance evaluations: beyond "performed as expected".

* Abbreviations:n PRIME+ — : professionalism, reporter, interpreter, manager, educator, plus suggesting an area for focused improvement and development.n RIME — : reporter, interpreter, manager, educatornnThis article continues the Council on Medical Student Education in Pediatrics’ series on the skills of, and strategies used by, excellent clinical teachers. Here, we provide a practical framework and helpful tips for writing student evaluations that will inform both students and their medical schools. nn—Susan Bannister, Editor, Council on Medical Student Education in Pediatrics Monthly FeaturennIn the present age of competency-based evaluations, faculty complete more forced-choice performance rating scales. Nonetheless, the narrative evaluation that tells the “story” of the learner remains critical, providing context that helps students understand the feedback they receive and clerkship directors understand the “big picture” of each student’s performance. For medical students, narrative evaluations can be included in the Medical Student Performance Evaluation that summarizes their performance in the first 3 years of medical school and becomes part of their residency applications. In this article, we review a framework that can help preceptors write narratives that more fully reflect observed performance.nnDespite many clinical observations, intelligent and accomplished faculty can suffer writer’s block when trying to write meaningful descriptions of students.1 Consider the following examples that we have received:“Needs to work on follow-through of plans and communicating with staff. Will refer to clerkship director.”“Very pleasant. Fun to work with. Seemed to enjoy Ped ED setting. Overall, performed as expected.”“Exceeded all expectations. Very bright and organized.”“Although a likable person, at times he appeared to be confused during the rotation.”“Development of treatment plans will improve with experience. Continue reading to improve fund of knowledge”nnAlthough truthful and honest, these written narratives do not … nnAddress correspondence to Alison V. Holmes, MD, MPH, Dartmouth-Hitchcock Medical Center, Pediatrics, Rubin 552, One Medical Center Dr, Lebanon, NH 03756. E-mail: Alison.V.Holmes{at}hitchcock.org

Assessing residents' written learning goals and goal writing skill: Validity evidence for the learning goal scoring rubric

Purpose To provide validity evidence for use of the Learning Goal Scoring Rubric to assess the quality of written learning goals and residents’ goal writing skills. Method This two-part study used the rubric to assess University of Colorado third-year pediatric residents’ written learning goals to obtain validity evidence. In study 1, five raters independently scored 48 goals written in 2010–2011 and 2011–2012 by 48 residents, who also responded to the Jefferson Scale of Physician Lifelong Learning (JeffSPLL). In study 2, two raters independently scored 48 goals written in 2011–2012 by 12 residents. Intraclass correlation coefficients (ICCs) assessed rater agreement to provide evidence for response process. Generalizability theory assessed internal structure. Independent-samples Mann–Whitney U tests and correlations assessed relationship to other variables. Content was matched to published literature and instructional methods. Results The ICC was 0.71 for the overall rubric. In study 1, where the generalizability study’s (G study’s) object of measurement was learning goals, the phi coefficient was 0.867. In study 2, where the G study’s object of measurement was the resident (goal writing skill), the phi coefficient was 0.751. The total mean score of residents with goal writing training was significantly higher than that of those without (7.54 versus 4.98, P < .001). Correlation between goal quality and JeffSPLL score was not significant. Investigators agreed that the content matched the published literature and instructional methods. Conclusions Preliminary validity evidence indicates that this scoring rubric can assess learning goal quality and goal writing skill.

The Referral and Consultation Entrustable Professional Activity: Defining the Components in Order to Develop a Curriculum for Pediatric Residents

From the Department of Pediatrics, George Washington University, Children’s National Health System, Office of Medical Education, Washington, DC (Dr Hamburger, Dr Agrawal, Dr Boogaard, Dr Weisz, and Dr Ottolini); Departments of Pediatrics (Dr Lane and Dr Hanson), and Family Medicine (Dr Hanson), University of Colorado School of Medicine, Children’s Hospital Colorado, Aurora, Colo The authors declare that they have no conflict of interest. Address correspondence to Ellen K. Hamburger, MD, Department of Pediatrics, George Washington University, Children’s National Medical System, Office of Medical Education, 111 Michigan Ave NW, Washington, DC 20037 (e-mail: ehamburg@childrensnational.org).

Practical Suggestions for the Creation and Use of Meaningful Learning Goals in Graduate Medical Education.

From the Department of Pediatrics, Nationwide Children’s Hospital, Ohio State University School of Medicine, Columbus, Ohio (Drs Reed and Mahan); Department of Pediatrics, University of Colorado School of Medicine, Aurora, Colo (Drs Lockspeiser, Hanson, and Rosenberg); Department of Pediatrics, Wright State University Boonshoft School of Medicine, Dayton Children’s Hospital, Dayton, Ohio (Dr Burke); Department of Pediatrics, Children’s Hospital at Dartmouth, Geisel School of Medicine at Dartmouth, Hanover, NH (Dr Gifford); Department of Pediatrics, Riley Hospital for Children and IU School of Medicine, Indianapolis, Ind (Dr McKenna); and Department of Pediatrics, University of California Davis School of Medicine, Sacramento, Calif (Dr Li) The authors declare that they have no conflict of interest. Address correspondence to Suzanne Reed, MD, Division of Hematology/Oncology/BMT, Nationwide Children’s Hospital, 700 Children’s Dr, Columbus, OH 43205 (e-mail: suzanne.reed@nationwidechildrens.org).

A longitudinal career-focused block for third-year pediatrics residents.

BACKGROUNDnThe traditional 1-month training blocks in pediatrics may fail to provide sufficient exposure to develop the knowledge, skills, and attitudes residents need for practice and may not be conducive to mentoring relationships with faculty and continuity with patients.nnnINTERVENTIONnWe created a 4-month career-focused experience (CFE) for third-year residents. The CFE included block time and longitudinal experiences in different content areas related to residents choice of urban and rural primary care, hospitalist medicine, or subspecialty care (prefellowship). Content was informed by graduate surveys, focus groups with primary care pediatricians and hospitalists, and interviews with fellowship directors. Outcomes were assessed via before and after surveys of residents attitudes and skills, assessment of skills with an objective structured clinical examination (OSCE), and interviews with residents and mentors.nnnRESULTSnTwenty-three of 49 third-year residents took part in the first 2xa0years of CFE. Two residents dropped out, leaving 21 who completed the 4-month experience (9 in primary care, 2 in hospitalist medicine, and 10 in a subspecialty). Residents reported improvement in their clinical skills, increased satisfaction with faculty mentoring and evaluation, and the ability to focus on what was important to their careers. OSCE performance did not differ between residents who completed the CFE and those who did not. Administrative burden was high.nnnCONCLUSIONSnFour-month career-focused training for pediatrics residents is feasible and may be effective in meeting part of the new requirement for 6xa0months of career-focused training during pediatrics residency.

Parenting Needs of Urban, African American Fathers

Fathers play a critical role in children’s development; similarly, fatherhood positively affects men’s health. Among the larger population of fathers relatively little is known about the parenting knowledge of urban, African American fathers. Focusing on urban, African American fathers, the objectives of this study were to (1) understand the primary sources from which fathers learn about parenting, (2) determine where and how fathers prefer to receive future parenting education, and (3) explore the information perceived as most valuable to fathers and how this compares with the recommended anticipatory guidance (Bright Futures-based) delivered during well visits. Five focus groups, with a total of 21 participants, were conducted with urban fathers at a community-based organization. Study eligibility included being more than18 years old, English speaking, and having at least one child 0 to 5 years old. During the focus groups, fathers were asked where they received parenting information, how and where they preferred to receive parenting information, and what they thought about Bright Futures parenting guidelines. Fathers most commonly described receiving parenting information from their own relatives rather than from their child’s health care provider. Most fathers preferred to learn parenting from a person rather than a technology-based source and expressed interest in learning more about parenting at community-based locations. Although fathers viewed health care providers’ role as primarily teaching about physical health, they valued Bright Futures anticipatory guidance about parenting. Fathers valued learning about child rearing, health, and development. Augmenting physician counseling about Bright Futures with community-based parenting education may be beneficial for fathers.

Medical Students’ Experiences Working With Frequently Rotating Pediatric Inpatient Attending Physicians

OBJECTIVEnTo explore medical students experiences working with frequently rotating pediatric inpatient attending physicians.nnnMETHODSnWe performed a qualitative study using focus groups and individual interviews of medical students who rotated on the general pediatric inpatient service at Childrens Hospital Colorado. The majority of inpatient pediatric attending physicians worked 1-week blocks. We used a semistructured interview guide and analyzed data using the constant comparative method. In accordance with the grounded theory method, codes were developed using an iterative approach, and major themes were identified. Analysis indicated theoretical saturation was achieved. We created a theory that arose from analysis of the data.nnnRESULTSnTwenty-seven medical students participated. Data analysis yielded 6 themes: learning climate, continuity, student resilience, opportunity to progress, growth into a physician, and evaluation. In the learning climate, the emotional environment was often stressful, although students valued exposure to different patient care and teaching styles. Senior resident continuity promoted student function; lack of continuity with attending physicians inhibited relationship development. Students were resilient in adjusting to changing faculty with different expectations. In the context of frequently rotating faculty, students had difficulty showing improvement to a single attending physician after feedback, which limited students opportunities to progress. Students perceived summative evaluation as less meaningful in the absence of having a relationship with their attending physicians.nnnCONCLUSIONSnMedical students valued exposure to different patient care and teaching styles. However, frequently changing attending physicians caused students stress and limited students perceived ability to achieve and show professional growth.

Ward Rounds With or Without an Attending Physician: How Interns Learn Most Successfully

OBJECTIVEnTo explore pediatric interns perspectives on the educational value of general pediatric ward rounds, in particular their rounding experiences with and without an attending physician.nnnMETHODSnQualitative study using individual interviews of pediatric interns (2013-2014) rotating on 2 general pediatric inpatient services at different institutions with different rounding team structures. In accordance with grounded theory methodology, data were analyzed using the constant comparative method. Codes were built using an iterative approach and organized into themes.nnnRESULTSnTwenty pediatric interns participated in 25 interviews. Data analysis yielded 4 themes: what is being learned; learning environment on rounds; learning and work; and ways of learning. Senior residents generally taught practical aspects of patient care and attending physicians taught broader concepts with references to the medical literature. Rounds without an attending physician were perceived as less formal and promoted collaborative discussions with senior residents. Interns were more uncomfortable during rounds with an attending physician but appreciated how that facilitated their learning. Although patient care tasks provided opportunities for experiential learning, interns frequently perceived them to impede learning during rounds. Intern learning during ward rounds occurred via self-directed learning, interactive learning, and through caring for patients. Brief, clinically relevant teaching pearls and questioning clinical reasoning in a respectful manner were helpful.nnnCONCLUSIONSnInterns learn different content and learn in different ways depending on the presence or absence of an attending physician at rounds. There might be educational value from rounding with teams that include and do not include an attending physician.

Medical Students' Acquisition of Adolescent Interview Skills after Coached Role Play

STUDY OBJECTIVEnTo develop and evaluate an educational activity designed to teach the adolescent Home, Education and employment, Eating, Activities, Drugs, Sexuality, Suicide/depression, and Safety (HEADS) examination. DESIGN, SETTING, PARTICIPANTS, INTERVENTIONS, AND MAIN OUTCOME MEASURES: Participants were third-year medical students in their pediatric clerkships. Students received an article on the HEADS interview and attended an adolescent medicine educational session. The session included individualized goal-setting and coached role play. Students skills in doing a HEADS interview were evaluated through a standardized patient encounter (SPE) with a checklist and a retrospective pre- and post-test survey. The SPE checklist was used to assess whether the students included questions in 6 key areas of a HEADS interview.nnnRESULTSnOne hundred fifty-two students participated. During the SPE, 90% of students queried the adolescents home life, 91% education, 82% activities, 84% drug/substance abuse, 95% sexual history, and 61% symptoms of depression. Pre- and postintervention data were compared using the Kruskal-Wallis Test and showed a statistically significant difference in the students ability to list key topic areas of the HEADS exam (Pxa0<xa0.001) and to use the skills needed for an adolescent interview using the HEADS exam (Pxa0<xa0.001).nnnCONCLUSIONnAfter an introduction to the HEADS examination, most students covered almost all of the topic areas of this screening interview during a SPE. Only three-fifths of the students, however, included questions about symptoms of depression. Coached role play with goal-setting facilitated effective learning of this approach to adolescent interviewing.