Paul A. Hemmer

Program evaluation models and related theories: AMEE Guide No. 67

This Guide reviews theories of science that have influenced the development of common educational evaluation models. Educators can be more confident when choosing an appropriate evaluation model if they first consider the models theoretical basis against their programs complexity and their own evaluation needs. Reductionism, system theory, and (most recently) complexity theory have inspired the development of models commonly applied in evaluation studies today. This Guide describes experimental and quasi-experimental models, Kirkpatricks four-level model, the Logic Model, and the CIPP (Context/Input/Process/Product) model in the context of the theories that influenced their development and that limit or support their ability to do what educators need. The goal of this Guide is for educators to become more competent and confident in being able to design educational program evaluations that support intentional program improvement while adequately documenting or describing the changes and outcomes—intended and unintended—associated with their programs.

The development of entrustable professional activities for internal medicine residency training: A report from the Education Redesign Committee of the Alliance for Academic Internal Medicine

Purpose The Alliance for Academic Internal Medicine charged its Education Redesign Committee with the task of assisting internal medicine residency program directors in meeting the challenges of competency-based assessment that were part of the Accreditation Council for Graduate Medical Education’s (ACGME’s) Next Accreditation System. Method Recognizing the limitations of the ACGME general competencies as an organizing framework for assessment and the inability of the milestones to provide the needed context for faculty to assess residents’ competence, the Education Redesign Committee in 2011 adopted the work-based assessment framework of entrustable professional activities (EPAs). The committee selected the EPA framework after reviewing the literature on competency-based education and EPAs and consulting with experts in evaluation and assessment. The committee used an iterative approach with broad-based feedback from multiple sources, including program directors, training institutions, medical organizations, and specialty societies, to develop a set of EPAs that together define the core of the internal medicine profession. Results The resulting 16 EPAs are those activities expected of a resident who is ready to enter unsupervised practice, and they provide a starting point from which training programs could develop assessments and curricula. The committee also provided a strategy for the use of these EPAs in competency-based evaluation. Conclusions These EPAs are intended to serve as a starting point or guide for program directors to begin developing meaningful, work-based assessments that inform the evaluation of residents’ competence.

LITERATURE REVIEWS: Narrative Review: Use of Student-Generated Logbooks in Undergraduate Medical Education

Background: Logbooks are used by clinical clerkships in undergraduate medical education as tools for individual student guidance, programmatic evaluation, and Liaison Committee on Medical Education (LCME) accreditation. The purpose of this narrative review was to summarize the published literature on the form and function of logbooks and to review logbook validity and reliability. We performed a literature search from 1980 through 2004 and reviewed 50 articles on logbook use during clinical clerkships. Summary: Articles were categorized into 5 themes: description and feasibility of logbooks (27 articles), accuracy and completeness of logbook entries (14 articles), utility to student education (11 articles), utility to program evaluation (26 articles), and connecting logbook process measures to clerkship outcomes (2 articles). Conclusions: A feasible and acceptable logbook system is an attainable goal, although students usually did not complete logbooks unless required. The available literature does not establish that logbooks currently in use have sufficient reliability or validity to allow for the routine use of the information for program change or accreditation purposes. The ideal logbook should be inexpensive, feasible, and acceptable to students and should allow rapid collation of accurate, relevant data for timely analysis and feedback to the student and clerkship director.

The Structure of Program Evaluation: An Approach for Evaluating a Course, Clerkship, or Components of a Residency or Fellowship Training Program

Background: Directors of courses, clerkships, residencies, and fellowships are responsible not only for determining whether individual trainees have met educational goals but also for ensuring the quality of the training program itself. The purpose of this article is to discuss a framework for program evaluation that has sufficient rigor to satisfy accreditation requirements yet is flexible and responsive to the uniqueness of individual educational programs. Summary: We discuss key aspects of program evaluation to include cardinal definitions, measurements, needed resources, and analyses of qualitative and quantitative data. We propose a three-phase framework for data collection (Before, During, and After) that can be used across undergraduate, graduate, and continuing medical education. Conclusions: This Before, During, and After model is a feasible and practical approach that is sufficiently rigorous to allow for conclusions that can lead to action. It can be readily implemented for new and existing medical education programs.

Identifying and overcoming the barriers to bedside rounds: a multicenter qualitative study.

Purpose The use of bedside rounds in teaching hospitals has declined, despite recommendations from educational leaders to promote this effective teaching strategy. The authors sought to identify reasons for the decrease in bedside rounds, actual barriers to bedside rounds, methods to overcome trainee apprehensions, and proposed strategies to educate faculty. Method A qualitative inductive thematic analysis using transcripts from audio-recorded, semistructured telephone interviews with a purposive sampling of 34 inpatient attending physicians from 10 academic U.S. institutions who met specific inclusion criteria for “bedside rounds” was performed in 2010. Main outcomes were themes pertaining to barriers, methods to overcome trainee apprehensions, and strategies to educate faculty. Quotations highlighting themes are reported. Results Half of respondents (50%) were associate or full professors, averaging 14 years in academic medicine. Primary reasons for the perceived decline in bedside rounds were physician- and systems related, although actual barriers encountered related to systems, time, and physician-specific issues. To address resident apprehensions, six themes were identified: build partnerships, create safe learning environments, overcome with experience, make bedside rounds educationally worthwhile, respect trainee time, and highlight positive impact on patient care. Potential strategies for educating faculty were identified, most commonly faculty development initiatives, divisional/departmental culture change, and one-on-one shadowing opportunities. Conclusions Bedside teachers encountered primarily systems- and time-related barriers and overcame resident apprehensions by creating a learner-oriented environment. Strategies used by experienced bedside teachers can be used for faculty development aimed at promoting bedside rounds.

Internal Medicine Clerkship Directors' Use of and Opinions About Clerkship Examinations

Background: For years, quantifiable examinations have been a core component of assessing medical student competence during the internal medicine clerkship. Purpose: To determine how internal medicine clerkship directors use and view examinations and how uses of examinations have changed. Methods: In 1999, the Clerkship Directors in Internal Medicine conducted a confidential survey of its 123 institutional members. Results: Survey response rate was 89% (109/123). The National Board of Medical Examiners subject examination was used by 83%, alone (49%) or in combination with a faculty developed examination or a standardized patient examination (34%). Minimum passing scores were required for the subject exam by 80%, for faculty-developed examinations by 65%, and for the standardized patient exam by 63%. Examinations contribute approximately 25% toward a students final grade. Students with acceptable clerkship performances but who fail an exam typically retest after self-study. Students who fail a retest receive unsatisfactory grades and require additional medicine experience. Of the clerkship directors who reported using the National Board of Medical Examiners subject examination, 45 (50%) provided comments on ways to improve the examination. Comments focused on examination content, reporting results, basing the exam on a published core curriculum, and general administrative issues. Over the past decade, use of the National Board of Medical Examiners subject examination has increased (66% to 83%), use of faculty-developed examinations has declined (46% to 27%), and the use of a clerkship standardized patient examination increased sharply (2% to 27%). Conclusions: Internal medicine clerkship directors commonly require students to pass standardized or locally developed exams and use test results to make academic decisions. The use of standardized patient examinations has increased significantly and likely reflects a broadening of competency assessment. Our results can serve as a basis for individual programmatic evaluation, for internal medicine and other clerkship directors.

Using Formal Evaluation Sessions for Case-based Faculty Development during Clinical Clerkships

Developing housestaff and faculty in their roles as medical educators is a dynamic process. The rigorous clinical evaluation method used during the third-year internal medicine clerkship at the Uniformed Services University uniquely incorporates faculty development into the process of evaluation and generating feedback for students. Formal evaluation sessions are held monthly at all clerkship sites throughout the 12-week clerkship and are moderated by either the internal medicine clerkship director or the on-site clerkship directors. Although designed to provide an opportunity for faculty to evaluate student performance and prepare formative feedback, the sessions also function as formal, planned, and longitudinal forums of “real-time,” “case-based” faculty development that address professional, instructional, and leadership development. The evaluation sessions are used as a means to model and teach the key concepts of the Stanford Faculty Development Program. Providing a unifying form of evaluation across multiple teaching sites and settings makes formal evaluation sessions a powerful, state-of-the-art tool for faculty development.

Exploring clinical reasoning in novices: a self-regulated learning microanalytic assessment approach.

The primary objectives of this study were to examine the regulatory processes of medical students as they completed a diagnostic reasoning task and to examine whether the strategic quality of these regulatory processes were related to short‐term and longer‐term medical education outcomes.

A Prospective Controlled Trial of the Influence of a Geriatrics Home Visit Program on Medical Student Knowledge, Skills, and Attitudes Towards Care of the Elderly

ABSTRACTPURPOSETo determine the impact of a geriatrics home visit program for third-year medical students on attitudes, skills, and knowledge.METHODSUsing a mixed methods, prospective, controlled trial, volunteer control group students (n = 17) at two sites and intervention group students (n = 16) at two different sites within the same internal medicine clerkship were given Internet and CDROM-based geriatric self-study materials. Intervention group students identified a geriatrics patient from their clinical experience, performed one “home” visit (home, nursing home, or rehabilitation facility) to practice geriatric assessment skills, wrote a structured, reflective paper, and presented their findings in small-group teaching settings. Papers were qualitatively analyzed using the constant comparative method for themes. All students took a pre-test and post-test to measure changes in geriatrics knowledge and attitudes.RESULTSGeneral attitudes towards caring for the elderly improved more in the intervention group than in the control group (9.8 vs 0.5%; p = 0.04, effect size 0.78). Medical student attitudes towards their home care training in medical school (21.7 vs 3.2%; p = 0.02, effect size 0.94) improved, as did attitudes towards time and reimbursement issues surrounding home visits (10.1 vs −0.2%; p = 0.02, effect size 0.89). Knowledge of geriatrics improved in both groups (13.4 vs 15.2% improvement; p = 0.73). Students described performing a mean of seven separate geriatric assessments (range 4–13) during the home visit. Themes that emerged from the qualitative analysis of the reflective papers added depth and understanding to the quantitative data and supported results concerning attitudinal change.CONCLUSIONSWhile all participants gained geriatrics knowledge during their internal medicine clerkship, students who performed a home visit had improved attitudes towards the elderly and described performing geriatric assessment skills. Requiring little faculty time, a geriatrics home visit program like this one may be a useful clerkship addition to foster medical students’ professional growth.

The Value of Bedside Rounds: A Multicenter Qualitative Study

Background: Bedside rounds have decreased on teaching services, raising concern about trainees’ clinical skills and patient–physician relationships. Purpose: We sought to identify recognized bedside teachers’ perceived value of bedside rounds to assist in the promotion of bedside rounds on teaching services. Methods: Authors used a grounded theory, qualitative study design of telephone semistructured interviews with bedside teachers (n = 34) from 10 U.S. institutions (2010–2011). Main outcomes were characteristics of participants, themes pertaining to the perceived value of bedside rounds, and quotations highlighting each respective theme. Results: The mean years in academic medicine was 13.7, and 51% were associate or full professors. Six main themes emerged: (a) skill development for learners (e.g., physical examination, communication, and clinical decision-making skills); (b) observation and feedback; (c) role-modeling; (d) team building among trainees, attending, and patient; (e) improved patient care delivery through combined clinical decision-making and team consensus; and (f) the culture of medicine as patient-centered care, which was embodied in all themes. Conclusions: Bedside teachers identify potential benefits of bedside rounds, many of which align with national calls to change our approach to medical education. The practice of bedside rounds enables activities essential to high-quality patient care and education.