Teri L. Turner

Evaluating the Performance of Medical Educators: A Novel Analysis Tool to Demonstrate the Quality and Impact of Educational Activities

Purpose Traditional promotion standards rely heavily on quantification of research grants and publications in the curriculum vitae. The promotion and retention of educators is challenged by the lack of accepted standards to evaluate the depth, breadth, quality, and impact of educational activities. The authors sought to develop a practical analysis tool for the evaluation of educator portfolios (EPs), based on measurable outcomes that allow reproducible analysis of the quality and impact of educational activities. Method The authors, 10 veteran educators and an external expert evaluator, used a scholarly, iterative consensus-building process to develop the tool and test it using real EPs from educational scholars who followed an EP template. They revised the template in parallel with the analysis tool to ensure that EP data enabled valid and reliable evaluation. The authors created the EP template and analysis tool for scholar and program evaluation in the Educational Scholars Program, a three-year national certification program of the Academic Pediatric Association. Results The analysis tool combines 18 quantitative and 25 qualitative items, with specifications, for objective evaluation of educational activities and scholarship. Conclusions The authors offer this comprehensive, yet practical tool as a method to enhance opportunities for faculty promotions and advancement, based on well-defined and documented educational outcome measures. It is relevant for clinical educators across disciplines and across institutions. Future studies will test the interrater reliability of the tool, using data from EPs written using the revised template.

Use of an expert concept map as an advance organizer to improve understanding of respiratory failure

Background: Helping novices transition toward expertise requires “meaningful” learning. Advance organizers are educational tools which help connect prior knowledge with new information, a critical step in making learning meaningful. Concept maps visually represent knowledge organization and can serve as advance organizers enabling deeper and more meaningful learning while enhancing knowledge integration. Aim: To compare respiratory failure understanding of resident physicians instructed, using an expert concept map advance organizer with learners receiving traditional didactic teaching. Methods: Residents were randomized by month of service to receive either a control lecture or a session using an expert concept map as an advanced organizer. Participants completed three concept maps; pre-education (CM1), immediately post-education (CM2), and 1 week later (CM3). Concept maps were scored using a standardized structural scoring method. Results: Forty-six pediatric residents (23 control and 23 experimental) participated. To account for repeated measures within subjects, the generalized estimating equations method compared concept map improvement between groups. The experimental group improved significantly more than controls (CM1–CM2–CM3 p = 0.001; CM1–CM2 p = 0.001; and CM1–CM3 p = 0.017). Conclusions: Using an expert concept map as an advance organizer improves knowledge organization and integration while offering a tool to enhance deeper understanding of medical knowledge among resident physicians.

What is a clinical pearl and what is its role in medical education

Background: Despite the advent of evidence-based medicine, clinical pearls, verbal and published, remain a popular and important part of medical education. Aims: The purpose of this study was to establish a definition of a clinical pearl and to determine criteria for an educationally sound clinical pearl. Methods: The authors searched the Medline database for material dealing with clinical pearls, examined and discussed the information found, and formulated a consensus opinion regarding the definition and criteria. Results: Clinical pearls are best defined as small bits of free standing, clinically relevant information based on experience or observation. They are part of the vast domain of experience-based medicine, and can be helpful in dealing with clinical problems for which controlled data do not exist. Conclusions: While there are no universally accepted criteria for preparing or evaluating a clinical pearl, we propose some rational guidelines for both.

Balancing Service and Education in Residency Training: A Logical Fallacy.

The system of residency training in the United States has at its core a perpetual tension between service and education. Graduate medical education (GME) is financed through prospective payments to teaching hospitals, where house officers compose a significant portion of the health care workforce. Numerous changes have occurred during the past 20 years that have put increasing pressure on this delicate balance. Radical transformation in the structure and process of GME training, most notably duty hour limitations, has led to compression of work, coupled with increasing emphasis on patient safety and quality. Hospitals have simultaneously undergone equally radical changes resulting from economic forces that have intensified pressure on faculty to increase clinical productivity to generate revenue. As a result, less time is available for faculty to teach and for trainees to learn.

Methodologies and study designs relevant to medical education research

Abstract Research is an important part of educational scholarship. Knowledge of research methodologies is essential for both conducting research as well as determining the soundness of the findings from published studies. Our goals for this paper therefore are to inform medical education researchers of the range and key components of educational research designs. We will discuss both qualitative and quantitative approaches to educational research. Qualitative methods will be presented according to traditions that have a distinguished history in particular disciplines. Quantitative methods will be presented according to an evidence-based hierarchy akin to that of evidence-based medicine with the stronger designs (systematic reviews and well conducted educational randomized controlled trials) at the top, and weaker designs (descriptive studies without comparison groups, or single case studies) at the bottom. It should be appreciated, however, that the research question determines the study design. Therefore, the onus is on the researcher to choose a design that is appropriate to answering the question. We conclude with an overview of how educational researchers should describe the study design and methods in order to provide transparency and clarity.

Transforming teaching into scholarship

Background:  Traditionally, scholarship has been defined very narrowly as the number of one’s publications and grant awards. Recently this definition has broadened to include dissemination of knowledge, experience or a tangible product shared with the educational community.

Investment in Faculty as Educational Scholars: Outcomes from the National Educational Scholars Program

From the Department of Pediatrics, Cincinnati Children’s Hospital Medical Center/ University of Cincinnati College of Medicine, Cincinnati, OH; Department of Pediatrics, Icahn School of Medicine at Mount Sinai, New York, NY; Department of Pediatrics, Baylor College of Medicine, Texas Children’s Hospital, Houston, TX; Department of Pediatrics, Stony Brook University School of Medicine, Stony Brook, NY; and Department of Pediatrics, University of Rochester Medical Center, A cademic institutions have strong incentives to develop faculty who are productive researchers, master clinicians, and educators. Forward-thinking institutions are encouraging their educators to become not only stellar teachers, but also productive scholars through peerreviewed dissemination of rigorously developed and evaluated curricular innovations, evaluation tools, and teaching methodologies. This emphasis on evaluation is imperative to ensure that educational programs effectively and efficiently meet the needs of the learners and the healthcare system. Even though many educator development programs train faculty to be good teachers, relatively few emphasize educational scholarship and research. Moreover, few of these are national programs offering participants the benefits of national mentoring and networking. The Educational Scholars Program (ESP) is a competitive national faculty development program created by the Academic Pediatric Association in 2006. The ESP is designed to cultivate skills in developing and evaluating curricula, conducting methodologically sound educational research, and effectively disseminating scholarly work. Mentorship and networking are key components of the ESP; each scholar is paired with a local project mentor and a national advisor with proven educational scholarship skills. Cohorts of scholars complete a 3-year longitudinal curriculum, including day-long sessions at 3 Pediatric Academic Society meetings and 6 online learning modules, each focused on specific educational scholarship skills. The cohorts form learning communities and serve as peer mentors. Scholars create an educator portfolio documenting their teaching activities and scholarly accomplishments. They also complete an educational project that they disseminate via peerreviewed publication or national presentation. The goal of this study was to describe the professional impact and value of the ESP on graduates and their institutions.

Value Placed on Formal Training in Education by Pediatric Department Chairs and Residency Program Directors

BACKGROUND While much is known about how educational leaders at the medical school level (eg, deans) view the importance of formal training in education for medical school teachers, little is known about how leaders at the clinical level (eg, department chairs) view such training. We sought to determine how pediatric department chairs and residency program directors view the value of formal training in education, such as that at a Master of Education (MEd) level, and to estimate the number of clinical pediatric faculty with or pursuing such training. METHODS A survey designed to assess the value placed on formal training in education and to estimate the number of clinical faculty with or pursuing such training was mailed to pediatric department chair persons and residency program directors at all 131 allopathic medical schools in the United States and Puerto Rico. RESULTS Eighty department chairs (61%) responded, and most indicated that when hiring new faculty, they view an applicant with an MEd as having an advantage. Both chairs and residency directors considered an MEd to be advantageous for a residency director by a ranking of 4.5 and 4.2, respectively, on a scale of 1 to 5 (P  =  .008). Of the 80 chairs who responded, 58.8% of respondents reported one or more faculty in their department had or was pursuing an MEd. Of the 72 responding residency directors (55%), 11 respondents (15.3%) indicated that they had an MEd. CONCLUSION More than half the medical school pediatric chairs responding to the survey had one or more clinical faculty with or pursuing an MEd in their departments. Survey results indicated that such training is valued by both department chairs and residency directors. Given the time and expense involved in obtaining an MEd, awareness of these data 5 be helpful to those considering pursuing, offering, or requiring such training.

Entrustable professional activities (EPAs) for teachers in medical education: Has the time come?

Abstract On a daily basis, patients put their trust in the healthcare system for safe and high-quality healthcare. However, what evidence do we have as an educational community that our supervising faculty members are competent to fulfill this responsibility? Few, if any, requirements exist for faculty members to have continuous professional development in the field of medical education. Many faculty “love to teach”, however, this love of teaching does not make them competent to teach or assess the competence of trainees whom they supervise. Faculty members who have a significant role as a teacher in the clinical setting should be assessed with regards to their baseline competence in applicable teaching EPAs. When competence is reached, an entrustment decision can be made. Once proficient or expert, a statement of awarded responsibility (STAR) may be granted. The time has come to reach beyond the “standards” of the old adage “see one, do one, teach one” in medical education. In this personal view, the authors outline an argument for and list the potential benefits for teachers, learners, and patients when we assess clinical teachers using EPAs within a competency-based medical education framework.

The pediatrics milestones assessment pilot: Development of workplace-based assessment content, instruments, and processes

Purpose To report on the development of content and user feedback regarding the assessment process and utility of the workplace-based assessment instruments of the Pediatrics Milestones Assessment Pilot (PMAP). Method One multisource feedback instrument and two structured clinical observation instruments were developed and refined by experts in pediatrics and assessment to provide evidence for nine competencies based on the Pediatrics Milestones (PMs) and chosen to inform residency program faculty decisions about learners’ readiness to serve as pediatric interns in the inpatient setting. During the 2012–2013 PMAP study, 18 U.S. pediatric residency programs enrolled interns and subinterns. Faculty, residents, nurses, and other observers used the instruments to assess learner performance through direct observation during a one-month rotation. At the end of the rotation, data were aggregated for each learner, milestone levels were assigned using a milestone classification form, and feedback was provided to learners. Learners and site leads were surveyed and/or interviewed about their experience as participants. Results Across the sites, 2,338 instruments assessing 239 learners were completed by 630 unique observers. Regarding end-of-rotation feedback, 93% of learners (128/137) agreed the assessments and feedback “helped me understand how those with whom I work perceive my performance,” and 85% (117/137) agreed they were “useful for constructing future goals or identifying a developmental path.” Site leads identified several benefits and challenges to the assessment process. Conclusions PM-based instruments used in workplace-based assessment provide a meaningful and acceptable approach to collecting evidence of learner competency development. Learners valued feedback provided by PM-based assessment.