Monica M. Cuddy

Use of the mini-clinical evaluation exercise to rate examinee performance on a multiple-station clinical skills examination: a validity study.

Background Multivariate generalizability analysis was used to investigate the performance of a commonly used clinical evaluation tool. Method Practicing physicians were trained to use the mini-Clinical Skills Examination (CEX) rating form to rate performances from the United States Medical Licensing Examination Step 2 Clinical Skills examination. Results Differences in rater stringency made the greatest contribution to measurement error; more raters rating each examinee, even on fewer occasions, could enhance score stability. Substantial correlated error across the competencies suggests that decisions about one scale unduly influence those on others. Conclusions Given the appearance of a halo effect across competencies, score interpretations that assume assessment of distinct dimensions of clinical performance should be made with caution. If the intention is to produce a single composite score by combining results across competencies, the presence of these effects may be less critical.

Neighborhood ecology and drug dependence mortality: an analysis of New York City census tracts.

Drug dependence mortality appears to be highly concentrated in certain disadvantaged populations and in certain disadvantaged areas. Using a relatively large sample of census tract data for New York City, 1991–1995 (N = 2,037), the present study examines the structural covariates of drug dependence mortality rates. Spatially lagged negative binomial regression analyses indicated considerable support for previous findings regarding the importance of poverty as a predictor of drug mortality. Furthermore, two variables especially relevant for the social disorganization and deviant opportunity perspectives in criminology exhibited significant independent effects: the neighborhood homeownership rate and the prevalence of boarded-up housing. The results support various policy initiatives concerned with the relationship between neighborhood environment and public health.

A Multilevel Analysis of Examinee Gender and USMLE Step 1 Performance

Background To examine the effects of (1) examinee gender on United States Medical Licensing Examination (USMLE) Step 1 performance, (2) examinee gender on the relationships between prematriculation measures and Step 1 performance, and (3) medical school characteristics on the relationships between examinee characteristics and Step 1 performance. Method A series of hierarchical linear models (examinees-nested-in-schools) was conducted predicting Step 1 scores. The sample included 66,412 examinees from 133 U.S. Liaison Committee on Medical Education-accredited medical schools/campuses. Results Controlling for prematriculation measures, men outperformed women slightly on Step 1. Undergraduate science grade point averages were more associated with Step 1 performance for women than men. Schools with higher mean Medical College Admission Test (MCAT) science scores had higher mean Step 1 scores, and MCAT science scores were slightly more associated with Step 1 performance for students from schools with higher percentages of female students. Conclusions Patterns of gender-related performance differences on Step 1 generally mirrored those reported in earlier research. School-level variables provided some additional insight into these relationships, but additional research is needed to fully understand why men and women perform differently on the USMLE.

A multilevel analysis of the relationships between selected examinee characteristics and United States Medical Licensing Examination Step 2 Clinical Knowledge performance: revisiting old findings and asking new questions.

Background This study examines: (1) the relationships between examinee characteristics and United States Medical Licensing Examination Step 2 Clinical Knowledge (CK) performance; (2) the effect of gender and examination timing (time per item) on the relationship between Steps 1 and 2 CK; and (3) the effect of school characteristics on the relationships between examinee characteristics and Step 2 CK performance. Method A series of hierarchical linear models (examinees-nested-in-schools) predicting Step 2 CK scores was fit to the data set. The sample included 54,487 examinees from 114 U.S. Liaison Committee on Medical Education–accredited medical schools. Results Consistent with past examinee-level research, women generally outperformed men on Step 2 CK, and examinees who received more time per item generally outperformed examinees who received less time per item. Step 1 score was generally more strongly associated with Step 2 CK performance for men and for examinees who received less time per item. School-level characteristics (size, average Step 1 performance) influenced the relationship between Steps 1 and 2 CK. Conclusion Both examinee-level and school-level characteristics are important for understanding Step 2 CK performance.

Changes in Anatomy Instruction and USMLE Performance: Empirical Evidence on the Absence of a Relationship.

Anatomy instruction has evolved over the past two decades as many medical schools have undergone various types of curricular reform. To provide empirical evidence about whether or not curricular changes impact the acquisition and retention of anatomy knowledge, this study investigated the effect of variation in gross anatomy course hours, curricular approach (stand‐alone versus integrated), and laboratory experience (dissection versus dissection and prosection) on USMLE Steps 1 and 2 Clinical Knowledge (CK) scores. Gross anatomy course directors at 54 United States schools provided information about their gross anatomy courses via an online survey (response rate of 42%). Survey responses were matched with USMLE scores for 6,411 examinees entering LCME‐accredited schools in 2007 and taking Step 1 for the first time in 2009. Regression analyses were conducted to examine relationships between gross anatomy instructional characteristics and USMLE performance. Step 1 total scores, Step 1 gross anatomy sub‐scores, and Step 2 CK scores were unrelated to instructional hours, controlling for MCAT scores. Examinees from schools with integrated curricula scored slightly lower on Steps 1 and 2 CK than those from stand‐alone courses (effect sizes of 2.1 and 1.9 on score scales with SDs of 22 and 20, respectively). Examinees with dissection and prosection experience performed slightly better on Step 2 CK than examinees in courses with dissection only laboratories (effect size of 1.2). Results suggest variation in course hours is unrelated to performance on Steps 1 and 2 CK. Although differences were observed in relation to curricular approach and laboratory experience, effect sizes were small. Anat Sci Educ 6: 3–10.

A multilevel analysis of examinee gender, standardized patient gender, and United States medical licensing examination step 2 clinical skills communication and interpersonal skills scores.

Background Women typically demonstrate stronger communication skills on performance-based assessments using human raters in medical education settings. This study examines the effects of examinee and rater gender on communication and interpersonal skills (CIS) scores from the performance-based component of the United States Medical Licensing Examination, the Step 2 Clinical Skills (CS) examination. Method Data included demographic and performance information for examinees that took Step 2 CS for the first time in 2009. The sample contained 27,910 examinees, 625 standardized patient/case combinations, and 278,776 scored patient encounters. Hierarchical linear modeling techniques were employed with CIS scores as the outcome measure. Results Females tend to slightly outperform males on CIS, when other variables related to performance are taken into account. No evidence of an examinee and rater gender interaction effect was found. Conclusions Results provide validity evidence supporting the interpretation and use of Step 2 CS CIS scores.

Progress testing in clinical science education: results of a pilot project between the National Board of Medical Examiners and a US Medical School.

Background: Though progress tests have been used for several decades in various medical education settings, a few studies have offered analytic frameworks that could be used by practitioners to model growth of knowledge as a function of curricular and other variables of interest. Aim: To explore the use of one form of progress testing in clinical education by modeling growth of knowledge in various disciplines as well as by assessing the impact of recent training (core rotation order) on performance using hierarchical linear modeling (HLM) and analysis of variance (ANOVA) frameworks. Methods: This study included performances across four test administrations occurring between July 2006 and July 2007 for 130 students from a US medical school who graduated in 2008. Measures-nested-in-examinees HLM growth curve analyses were run to estimate clinical science knowledge growth over time and repeated measures ANOVAs were run to assess the effect of recent training on performance. Results: Core rotation order was related to growth rates for total and pediatrics scores only. Additionally, scores were higher in a given discipline if training had occurred immediately prior to the test administration. Conclusions: This study provides a useful progress testing framework for assessing medical students’ growth of knowledge across their clinical science education and the related impact of training.

A Multilevel analysis of the relationships between examinee gender and United States Medical Licensing Exam (USMLE) step 2 CK content area performance.

Purpose To examine the effect of (1) gender on Step 2 Clinical Knowledge (CK) content area performance, (2) gender on the relationships between Step 1 scores and Step 2 CK content area performance, and (3) medical school characteristics on the relationships between examinee characteristics and Step 2 CK content area performance. Method The sample included 23,538 examinees from 136 Liaison Committee on Medical Education–accredited medical schools/campuses. Descriptive statistics were computed, and a series of examinees-nested-in-schools hierarchical linear models was conducted. Results Observed differences indicated that women generally outperformed men in most content areas and that controlling for Step 1 scores increased these gender-related score differences. Step 1 scores were more associated with Step 2 CK content area performance for men. School characteristics were generally unrelated to the relationships between examinee characteristics and Step 2 CK content area performance. Conclusions While past research indicated that women outperformed men in some content areas, and men outperformed women in others, the current study revealed a somewhat different pattern, with women outperforming men in most content areas.

Performance in physical examination on the USMLE Step 2 Clinical Skills examination.

Purpose To provide descriptive information about history-taking (HX) and physical examination (PE) performance for U.S. medical students as documented by standardized patients (SPs) during the Step 2 Clinical Skills (CS) component of the United States Medical Licensing Examination. The authors examined two hypotheses: (1) Students perform worse in PE compared with HX, and (2) for PE, students perform worse in the musculoskeletal system and neurology compared with other clinical domains. Method The sample included 121,767 student–SP encounters based on 29,442 examinees from U.S. medical schools who took Step 2 CS for the first time in 2011. The encounters comprised 107 clinical presentations, each categorized into one of five clinical domains: cardiovascular, gastrointestinal, musculoskeletal, neurological, and respiratory. The authors compared mean percent-correct scores for HX and PE via a one-tailed paired-samples t test and examined mean score differences by clinical domain using analysis of variance techniques. Results Average PE scores (59.6%) were significantly lower than average HX scores (78.1%). The range of scores for PE (51.4%–72.7%) was larger than for HX (74.4%–81.0%), and the standard deviation for PE scores (28.3) was twice as large as the HX standard deviation (14.7). PE performance was significantly weaker for musculoskeletal and neurological encounters compared with other encounters. Conclusions U.S. medical students perform worse on PE than HX; PE performance was weakest in musculoskeletal and neurology clinical domains. Findings may reflect imbalances in U.S. medical education, but more research is needed to fully understand the relationships among PE instruction, assessment, and proficiency.

The relationship between communication scores from the USMLE Step 2 Clinical Skills examination and communication ratings for first-year internal medicine residents.

Purpose This study extends available evidence about the relationship between scores on the Step 2 Clinical Skills (CS) component of the United States Medical Licensing Examination and subsequent performance in residency. It focuses on the relationship between Step 2 CS communication and interpersonal skills scores and communication skills ratings that residency directors assign to residents in their first postgraduate year of internal medicine training. It represents the first large-scale evaluation of the extent to which Step 2 CS communication and interpersonal skills scores can be extrapolated to examinee performance in supervised practice. Method Hierarchical linear modeling techniques were used to examine the relationships among examinee characteristics, residency program characteristics, and residency-director-provided ratings. The sample comprised 6,306 examinees from 238 internal medicine residency programs who completed Step 2 CS for the first time in 2005 and received ratings during their first year of internal medicine residency training. Results Although the relationship is modest, Step 2 CS communication and interpersonal skills scores predict communication skills ratings for first-year internal medicine residents after accounting for other factors. Conclusions The results of this study make a reasonable case that Step 2 CS communication and interpersonal skills scores provide useful information for predicting the level of communication skill that examinees will display in their first year of internal medicine residency training. This finding demonstrates some level of extrapolation from the testing context to behavior in supervised practice, thus providing validity-related evidence for using Step 2 CS communication and interpersonal skills scores in high-stakes decisions.