Susan Hingle

Systems-based practice assessed with a performance-based examination simulated and scored by standardized participants in the health care system: feasibility and psychometric properties.

Background: Systems-based practice is one of the six general competencies proposed by the Accreditation Council for Graduate Medical Education in their Outcome Project. However, little has been published on its assessment—possibly because the systems-based practice competency has been viewed as difficult to define and measure. Purpose: The purpose of this study was to determine whether a full performance-based examination of systems-based practice cases simulated and scored by standardized participants in the health care system could feasibly be constructed and implemented that would provide reliable and valid measurements. Methods: In the 1st year of the project (2008), four systems-based practice cases were developed and pilot tested with 13 residents. Videotapes of residents were studied to develop an instrument for subsequent assessment of performance by standardized participants. In the 2nd year (2009), the examination was expanded to a full 12 cases, which were completed by 11 second-year residents, and psychometric analyses were performed on the scores. Results: The generalizability coefficient for the full 12-case examination based on scoring by standardized participants was .71, which is nearly equal to that based on scoring by faculty physician observers, which was .78. The correlation between total scores obtained with standardized participants and physician observers was .78. Conclusions: A performance-based examination can provide a feasible and reliable assessment of systems-based practice. However, attempts to evaluate convergent validity and discriminant validity—by correlating systems-based practice performance assessments with mean global ratings of residents on the 6 competencies by faculty throughout training—were unsuccessful, due to a lack of independence between the rated dimensions.

Financing U.S. Graduate Medical Education: A Policy Position Paper of the Alliance for Academic Internal Medicine and the American College of Physicians

In this position paper, the Alliance for Academic Internal Medicine and the American College of Physicians examine the state of graduate medical education (GME) financing in the United States and recent proposals to reform GME funding. They make a series of recommendations to reform the current funding system to better align GME with the needs of the nations health care workforce. These recommendations include using Medicare GME funds to meet policy goals and to ensure an adequate supply of physicians, a proper specialty mix, and appropriate training sites; spreading the costs of financing GME across the health care system; evaluating the true cost of training a resident and establishing a single per-resident amount; increasing transparency and innovation; and ensuring that primary care residents receive training in well-functioning ambulatory settings that are financially supported for their training roles.

How we developed and piloted an electronic key features examination for the internal medicine clerkship based on a US national curriculum

Abstract Background: Key features examinations (KFEs) have been used to assess clinical decision making in medical education, yet there are no reports of an online KFE-based on a national curriculum for the internal medicine clerkship. What we did: The authors developed and pilot tested an electronic KFE based on the US Clerkship Directors in Internal Medicine core curriculum. Teams, with expert oversight and peer review, developed key features (KFs) and cases. Evaluation: The exam was pilot tested at eight medical schools with 162 third and fourth year medical students, of whom 96 (59.3%) responded to a survey. While most students reported that the exam was more difficult than a multiple choice question exam, 61 (83.3%) students agreed that it reflected problems seen in clinical practice and 51 (69.9%) students reported that it more accurately assessed the ability to make clinical decisions. Conclusions: The development of an electronic KFs exam is a time-intensive process. A team approach offers built-in peer review and accountability. Students, although not familiar with this format in the US, recognized it as authentically assessing clinical decision-making for problems commonly seen in the clerkship.

The Most Common Feedback Themes in Communication Skills Training in an Internal Medicine Residency Program: Lessons from the Resident Audio-Recording Project

ABSTRACT Individualized structured feedback is an integral part of a resident’s learning in communication skills. However, it is not clear what feedback residents receive for their communication skills development in real patient care. We will identify the most common feedback topics given to residents regarding communication skills during Internal Medicine residency training. We analyzed Resident Audio-recording Project feedback data from 2008 to 2013 by using a content analysis approach. Using open coding and an iterative categorization process, we identified 15 emerging themes for both positive and negative feedback. The most recurrent feedback topics were Patient education, Thoroughness, Organization, Questioning strategy, and Management. The residents were guided to improve their communication skills regarding Patient education, Thoroughness, Management, and Holistic exploration of patient’s problem. Thoroughness and Communication intelligibility were newly identified themes that were rarely discussed in existing frameworks. Assessment rubrics serve as a lens through which we assess the adequacy of the residents’ communication skills. Rather than sticking to a specific rubric, we chose to let the rubric evolve through our experience.