Jennifer R. Kogan

Opening the black box of clinical skills assessment via observation: a conceptual model

Medical Education 2011: 45: 1048–1060

Faculty staff perceptions of feedback to residents after direct observation of clinical skills.

Medical Education 2012: 46 : 201–215

Seeing the ‘black box’ differently: assessor cognition from three research perspectives

Performance assessments, such as workplace‐based assessments (WBAs), represent a crucial component of assessment strategy in medical education. Persistent concerns about rater variability in performance assessments have resulted in a new field of study focusing on the cognitive processes used by raters, or more inclusively, by assessors.

Twelve tips for implementing tools for direct observation of medical trainees’ clinical skills during patient encounters

Background: Direct observation of medical trainees by their supervisors with actual patients is essential for trainees to develop clinical skills competence. Despite the many available tools for direct observation of trainees by supervisors, it is unclear how educators should identify an appropriate tool for a particular clinical setting and implement the tool to maximize educational benefits for trainees in a manner that is feasible for faculty. Aims and methods: Based on our previous systematic review of the literature, we provide 12 tips for selecting and incorporating a tool for direct observation into a medical training program. We focus specifically on direct observation that occurs in clinical settings with actual patients. Results: Educators should focus on the existing tools for direct observation that have evidence of validity. Tool implementation must be a component of an educational program that includes faculty development about rating performance, providing meaningful feedback, and developing action plans collaboratively with learners. Conclusions: Educators can enhance clinical skills education with strategic incorporation of tools for direct observation into medical training programs. Identification of a psychometrically sound instrument and attention to faculty development and the feedback process are critical to the success of a program of direct observation.

Are discharge summaries teachable? The effects of a discharge summary curriculum on the quality of discharge summaries in an internal medicine residency program.

Background Interns are often required to dictate discharge summaries without formal training. We investigated the impact of a curriculum aimed at improving the quality (i.e., complete, organized, succinct, internally consistent, and readable) of interns’ discharge summaries. Method Fifty-nine medicine interns were randomized to a: (1) control group; (2) discharge summary curriculum; or (3) curriculum plus individualized feedback. Pre- and post-intervention, seven discharge summaries were graded using a 9-item instrument. T-tests, analysis of covariance, and effect sizes assessed group differences. Results There were multiple, significant within-group improvements for the intervention groups and between group differences post-intervention. The average effect size was large when the curriculum plus feedback group was compared to the control group (.70) and moderate when compared to the curriculum only group (.36). Conclusions Interns who received instruction on discharge summary skills improved the quality and of their discharge summaries. Adding feedback to the curriculum provided more benefit.

Identifying and overcoming the barriers to bedside rounds: a multicenter qualitative study.

Purpose The use of bedside rounds in teaching hospitals has declined, despite recommendations from educational leaders to promote this effective teaching strategy. The authors sought to identify reasons for the decrease in bedside rounds, actual barriers to bedside rounds, methods to overcome trainee apprehensions, and proposed strategies to educate faculty. Method A qualitative inductive thematic analysis using transcripts from audio-recorded, semistructured telephone interviews with a purposive sampling of 34 inpatient attending physicians from 10 academic U.S. institutions who met specific inclusion criteria for “bedside rounds” was performed in 2010. Main outcomes were themes pertaining to barriers, methods to overcome trainee apprehensions, and strategies to educate faculty. Quotations highlighting themes are reported. Results Half of respondents (50%) were associate or full professors, averaging 14 years in academic medicine. Primary reasons for the perceived decline in bedside rounds were physician- and systems related, although actual barriers encountered related to systems, time, and physician-specific issues. To address resident apprehensions, six themes were identified: build partnerships, create safe learning environments, overcome with experience, make bedside rounds educationally worthwhile, respect trainee time, and highlight positive impact on patient care. Potential strategies for educating faculty were identified, most commonly faculty development initiatives, divisional/departmental culture change, and one-on-one shadowing opportunities. Conclusions Bedside teachers encountered primarily systems- and time-related barriers and overcame resident apprehensions by creating a learner-oriented environment. Strategies used by experienced bedside teachers can be used for faculty development aimed at promoting bedside rounds.

BRIEF REPORT: Use of the Mini-Clinical Evaluation Exercise in Internal Medicine Core Clerkships

INTRODUCTION: Direct observation of medical students’ clinical skills is important, but occurs infrequently. The mini-clinical evaluation exercise (mCEX) is a tool developed for use with internal medicine (IM) residents that can be used to promote direct observation of medical students’ clinical skills. It is unknown how many IM core clerkships in the United States use the mCEX or how it has been implemented.METHODS: Questions about use of the mCEX were incorporated into an online annual survey distributed to the 114 IM clerkships belonging to Clerkship Directors in Internal Medicine, a national organization of individuals responsible for teaching IM to medical students.RESULTS: The survey response rate was 83%. Twenty-eight percent (N=27) of respondents use the mCEX in their clerkship. The mean number of required mCEX encounters is 2.3 (SD 1.6). The mCEX is used for formative assessment (68%) more than summative assessment (11%). Ward attendings are the most common mCEX evaluators (72%).DISCUSSION: The mCEX is being used to promote direct observation of medical students’ clinical skills in a significant minority of IM core clerkships. The mCEX is 1 tool for facilitating feedback from both faculty and residents on trainees’ developing skills.

A randomized-controlled study of encounter cards to improve oral case presentation skills of medical students

OBJECTIVE: To determine the feasibility of oral case presentation (OCP) encounter cards as a tool for formative evaluation, to estimate the reliability and validity of the ratings when used in a medicine clerkship, and to examine whether the use of OCP encounter cards improves students’ OCP skills.DESIGN: Randomized controlled study.SETTING: Medicine core clerkship at a U.S. medical school.PARTICIPANTS/INTERVENTION: Students enrolled in the medicine core clerkship (n=164) from January to December of 2003 were randomly assigned to receive weekly feedback using OCP encounter cards rating nine presentation compentencies or receive usual feedback. Mean OCP ratings were correlated with multiple summative assessments. Performance on an end-of-clerkship OCP was compared between intervention and control groups.MAIN RESULTS: Eighty percent of cards were completed. The mean OCP rating averaged over 9 competencies was 7.7 (SD=0.8) on a 9-point scale. Standard error of ratings was 0.3. OCP ratings were correlated with inpatient evaluations (r=.58), inpatient ratings of presentation skills (r=.43), and final grades (r=.40). Final OCP performance was similar for the intervention and control groups (7.0 vs 7.2, P=.09).CONCLUSION: OCP encounter cards are a novel and feasible tool to assess clerkship students’ oral case presentation skills. OCP card ratings are reproducible, and validity is suggested by their correlation with multiple markers of performance. However, encounter cards did not improve performance on summative oral presentations.

How clinical supervisors develop trust in their trainees: a qualitative study

Clinical supervisors oversee trainees’ performance while granting them increasing opportunities to work independently. Although the factors contributing to supervisors’ trust in their trainees to conduct clinical work have been identified, how the development of trust is shaped by these factors remains less clear.

Proposed Standards for Medical Education Submissions to the Journal of General Internal Medicine

To help authors design rigorous studies and prepare clear and informative manuscripts, improve the transparency of editorial decisions, and raise the bar on educational scholarship, the Deputy Editors of the Journal of General Internal Medicine articulate standards for medical education submissions to the Journal. General standards include: (1) quality questions, (2) quality methods to match the questions, (3) insightful interpretation of findings, (4) transparent, unbiased reporting, and (5) attention to human subjects’ protection and ethical research conduct. Additional standards for specific study types are described. We hope these proposed standards will generate discussion that will foster their continued evolution.