Richard E. Hawkins

Construct validity of the miniclinical evaluation exercise (miniCEX).

Purpose To investigate the construct validity of the miniclinical evaluation exercise (miniCEX). Method Forty faculty participants from 16 internal medicine residency programs enrolled in a randomized, controlled trial of faculty development. Using a standard nine-point miniCEX rating form, participants watched and rated performances of standardized residents on nine scripted clinical videotapes depicting three levels of performance (unsatisfactory, marginal/satisfactory, and high satisfactory/superior). The nine-point rating scale was 1–3 = unsatisfactory, 4–6 = marginal/satisfactory, and 7–9 = superior. The performances were rated for three clinical skills, history taking, physical examination, and counseling. Results For each of the three clinical skills, the faculty participants were able to successfully discriminate among the three levels of performance using the miniCEX scale. Differences among ratings of the three performance levels were statistically significant; however, the range in ratings among the participants for each videotape was wide. Conclusion The authors believe this to be the first study to document the construct validity of the miniCEX. Although the miniCEX appears to have reliability and construct validity, further research is needed to improve individual faculty observation skills and reduce interrater variability.

Syphilis and neurosyphilis in a human immunodeficiency virus type-1 seropositive population: Evidence for frequent serologic relapse after therapy

OBJECTIVE To describe clinical and treatment aspects of syphilis infection among patients seropositive for the human immunodeficiency virus (HIV). PATIENTS AND METHODS Results of serologic tests for syphilis, CD4+ T-lymphocyte counts, and clinical response to therapy were retrospectively monitored in 100 HIV-infected adults with syphilis from a tertiary-care military HIV program. RESULTS Of the 1,206 HIV-infected patients, 100 (8.3%) in the cohort had syphilis; 61 patients were treated for active syphilis. Serologic or clinical relapse eventually occurred in 10 of the 56 treated patients (17.9%) with follow-up available; 7 of the 10 who relapsed had previously received high-dose intravenous or procaine penicillin therapy. Relapse occurred more than 12 months after initial therapy in 6 of 10 patients (60%) who experienced relapse; 5 patients experienced multiple relapses. The mean CD4+ T-lymphocyte count was not predictive of relapse. Patients with reactive cerebrospinal fluid (CSF) Venereal Disease Research Laboratory (VDRL) test titers (4 of 7 patients [57%]) or the rash of secondary syphilis (4 of 14 patients [29%]) were at highest risk of subsequent relapse or treatment failure when monitored for an average of 2 years. CONCLUSION Standard penicillin regimens, including high-dose intravenous penicillin, transiently lowered serum VDRL titers in nearly all cases, but were sometimes inadequate in preventing serologic and clinical relapse in patients infected with HIV type-1, especially among those with secondary syphilis and reactive CSF VDRL titers. Careful long-term follow-up is essential, and repeated courses of therapy may be needed for patients infected with HIV type-1 who have syphilis.

Use of the mini-clinical evaluation exercise to rate examinee performance on a multiple-station clinical skills examination: a validity study.

Background Multivariate generalizability analysis was used to investigate the performance of a commonly used clinical evaluation tool. Method Practicing physicians were trained to use the mini-Clinical Skills Examination (CEX) rating form to rate performances from the United States Medical Licensing Examination Step 2 Clinical Skills examination. Results Differences in rater stringency made the greatest contribution to measurement error; more raters rating each examinee, even on fewer occasions, could enhance score stability. Substantial correlated error across the competencies suggests that decisions about one scale unduly influence those on others. Conclusions Given the appearance of a halo effect across competencies, score interpretations that assume assessment of distinct dimensions of clinical performance should be made with caution. If the intention is to produce a single composite score by combining results across competencies, the presence of these effects may be less critical.

Constructing a validity argument for the mini-Clinical Evaluation Exercise: a review of the research.

Purpose The mini-Clinical Evaluation Exercise (mCEX) is increasingly being used to assess the clinical skills of medical trainees. Existing mCEX research has typically focused on isolated aspects of the instruments reliability and validity. A more thorough validity analysis is necessary to inform use of the mCEX, particularly in light of increased interest in high-stakes applications of the methodology. Method Kanes (2006) validity framework, in which a structured argument is developed to support the intended interpretation(s) of assessment results, was used to evaluate mCEX research published from 1995 to 2009. In this framework, evidence to support the argument is divided into four components (scoring, generalization, extrapolation, and interpretation/decision), each of which relates to different features of the assessment or resulting scores. The strength and limitations of the reviewed research were identified in relation to these components, and the findings were synthesized to highlight overall strengths and weaknesses of existing mCEX research. Results The scoring component yielded the most concerns relating to the validity of mCEX score interpretations. More research is needed to determine whether scoring-related issues, such as leniency error and high interitem correlations, limit the utility of the mCEX for providing feedback to trainees. Evidence within the generalization and extrapolation components is generally supportive of the validity of mCEX score interpretations. Conclusions Careful evaluation of the circumstances of mCEX assessment will help to improve the quality of the resulting information. Future research should address issues of rater selection, training, and monitoring which can impact rating accuracy.

Assessment of patient management skills and clinical skills of practising doctors using computer‐based case simulations and standardised patients

Context  Standardised assessments of practising doctors are receiving growing support, but theoretical and logistical issues pose serious obstacles.

Changes in clinical skills education resulting from the introduction of the USMLE™ step 2 clinical skills (CS) examination

Background: Step 2 Clinical Skills (CS) was recently introduced into the United States Medical Licensing Examination (USMLE™) to ensure that successful candidates for licensure possess the clinical skills to provide safe and effective patient care. Aims: To explore if medical schools had changed the objectives, content, or emphasis in their pre-clinical curriculum in response to its implementation. Methods: In April 2005, the Clerkship Directors in Internal Medicine (CDIM) sent an electronic survey to a single member from each medical school with a CDIM member. The survey instrument included 26 pre-clinical curricular questions with nine questions specifically addressing changes in response to implementation of the Step 2 CS. Results: Forty-five percent of respondents reported changes to the Introduction to Clinical Medicine (ICM) course objectives while 39% and 40% reported changes in content and emphasis. Seventy-four percent felt their students were adequately prepared for the Step 2 CS and 18% were unsure. Conclusions: Over a third of medical schools are implementing changes to the objectives, content, and/or emphasis of their curriculum, at least partially in response to the institution of Step 2 CS.

How Can Medical Students Add Value? Identifying Roles, Barriers, and Strategies to Advance the Value of Undergraduate Medical Education to Patient Care and the Health System

Purpose As health systems evolve, the education community is seeking to reimagine student roles that combine learning with meaningful contributions to patient care. The authors sought to identify potential stakeholders regarding the value of student work, and roles and tasks students could perform to add value to the health system, including key barriers and associated strategies to promote value-added roles in undergraduate medical education. Method In 2016, 32 U.S. medical schools in the American Medical Association’s (AMA’s) Accelerating Change in Education Consortium met for a two-day national meeting to explore value-added medical education; 121 educators, systems leaders, clinical mentors, AMA staff leadership and advisory board members, and medical students were included. A thematic qualitative analysis of workshop discussions and written responses was performed, which extracted key themes. Results In current clinical roles, students can enhance value by performing detailed patient histories to identify social determinants of health and care barriers, providing evidence-based medicine contributions at the point-of-care, and undertaking health system research projects. Novel value-added roles include students serving as patient navigators/health coaches, care transition facilitators, population health managers, and quality improvement team extenders. Six priority areas for advancing value-added roles are student engagement, skills, and assessments; balance of service versus learning; resources, logistics, and supervision; productivity/billing pressures; current health systems design and culture; and faculty factors. Conclusions These findings provide a starting point for collaborative work to positively impact clinical care and medical education through the enhanced integration of value-added medical student roles into care delivery systems.

Priority Areas and Potential Solutions for Successful Integration and Sustainment of Health Systems Science in Undergraduate Medical Education

Educators, policy makers, and health systems leaders are calling for significant reform of undergraduate medical education (UME) and graduate medical education (GME) programs to meet the evolving needs of the health care system. Nationally, several schools have initiated innovative curricula in both classroom and workplace learning experiences to promote education in health systems science (HSS), which includes topics such as value-based care, health system improvement, and population and public health. However, the successful implementation of HSS curricula across schools is challenged by issues of curriculum design, assessment, culture, and accreditation, among others. In this report of a working conference using thematic analysis of workshop recommendations and experiences from 11 U.S. medical schools, the authors describe seven priority areas for the successful integration and sustainment of HSS in educational programs, and associated challenges and potential solutions. In 2015, following regular HSS workgroup phone calls and an Accelerating Change in Medical Education consortium-wide meeting, the authors identified the priority areas: partner with licensing, certifying, and accrediting bodies; develop comprehensive, standardized, and integrated curricula; develop, standardize, and align assessments; improve the UME to GME transition; enhance teachers’ knowledge and skills, and incentives for teachers; demonstrate value added to the health system; and address the hidden curriculum. These priority areas and their potential solutions can be used by individual schools and HSS education collaboratives to further outline and delineate the steps needed to create, deliver, study, and sustain effective HSS curricula with an eye toward integration with the basic and clinical sciences curricula.

The Impact of Postgraduate Training and Timing on USMLE Step 3 Performance

Purpose. This study examined the extent to which differences in clinical experience, gained in postgraduate training programs, affect performance on Step 3 of the United States Medical Licensing Examination (USMLE). Method. Subjects in the study were 36,805 U.S. and Canadian medical school graduates who took USMLE Step 3 for the first time between November 1999 and December 2002. Regression analyses examined the relation between length and type of postgraduate training and Step 3 score after controlling for prior performance on previous USMLE examinations. Results. Results indicate that postgraduate training in programs that provide exposure to a broad range of patient problems, and continued training in such areas, improves performance on Step 3. Conclusions. Study data reaffirm the validity of the USMLE Step 3 examination, and the information found in the pattern of results across specialties may be useful to residents and program directors.

The association of faculty and residents' gender on faculty evaluations of internal medicine residents in 16 residencies.

Purpose Previous studies have found gender bias in the global evaluations of trainees. The purpose of this study was to investigate the association of faculty and residents’ gender on the evaluation of residents’ specific clinical skills, using direct observation. Method In 2001–2002, 40 clinician–educators from 16 internal medicine residency programs viewed a series of nine scripted videotapes depicting varying levels of residents’ clinical performance in medical interviewing, physical examination, and counseling. Differences in the ratings of women versus men faculty, in relation to differences in the residents’ gender, were compared using random-effects regression analysis. Results There were no statistically or educationally significant differences in the rating of clinical skills attributable to faculty or residents’ gender for medical interviewing, physical examination, or counseling. Conclusions This study suggests that gender bias may be less prevalent in the current era of evaluation of clinical skills, particularly when specific skills are directly observed by faculty. Further work is needed to examine whether the findings of this study translate to the actual training setting.