Christopher J. Mooney

Measurement of the general competencies of the accreditation council for graduate medical education: a systematic review.

Purpose To evaluate published evidence that the Accreditation Council for Graduate Medical Educations six general competencies can each be measured in a valid and reliable way. Method In March 2008, the authors conducted searches of Medline and ERIC using combinations of search terms “ACGME,” “Accreditation Council for Graduate Medical Education,” “core competencies,” “general competencies,” and the specific competencies “systems-based practice” (SBP) and “practice based learning and improvement (PBLI).” Included were all publications presenting new qualitative or quantitative data about specific assessment modalities related to the general competencies since 1999; opinion pieces, review articles, and reports of consensus conferences were excluded. The search yielded 127 articles, of which 56 met inclusion criteria. Articles were subdivided into four categories: (1) quantitative/psychometric evaluations, (2) preliminary studies, (3) studies of SBP and PBLI, and (4) surveys. Results Quantitative/psychometric studies of evaluation tools failed to develop measures reflecting the six competencies in a reliable or valid way. Few preliminary studies led to published quantitative data regarding reliability or validity. Only two published surveys met quality criteria. Studies of SBP and PBLI generally operationalized these competencies as properties of systems, not of individual trainees. Conclusions The peer-reviewed literature provides no evidence that current measurement tools can assess the competencies independently of one another. Because further efforts are unlikely to be successful, the authors recommend using the competencies to guide and coordinate specific evaluation efforts, rather than attempting to develop instruments to measure the competencies directly.

The impact of a program in mindful communication on primary care physicians.

Purpose In addition to structural transformations, deeper changes are needed to enhance physicians’ sense of meaning and satisfaction with their work and their ability to respond creatively to a dynamically changing practice environment. The purpose of this research was to understand what aspects of a successful continuing education program in mindful communication contributed to physicians’ well-being and the care they provide. Method In 2008, the authors conducted in-depth, semistructured interviews with primary care physicians who had recently completed a 52-hour mindful communication program demonstrated to reduce psychological distress and burnout while improving empathy. Interviews with a random sample of 20 of the 46 physicians in the Rochester, New York, area who attended at least four of eight weekly sessions and four of eight monthly sessions were audio-recorded, transcribed, and analyzed qualitatively. The authors identified salient themes from the interviews. Results Participants reported three main themes: (1) sharing personal experiences from medical practice with colleagues reduced professional isolation, (2) mindfulness skills improved the participants’ ability to be attentive and listen deeply to patients’ concerns, respond to patients more effectively, and develop adaptive reserve, and (3) developing greater self-awareness was positive and transformative, yet participants struggled to give themselves permission to attend to their own personal growth. Conclusions Interventions to improve the quality of primary care practice and practitioner well-being should promote a sense of community, specific mindfulness skills, and permission and time devoted to personal growth.

Impact of peer assessment on the professional development of medical students: a qualitative study.

Purpose Peer assessment can predict future academic performance and provide medical students with reliable feedback about professionalism. It is unclear whether peer assessment fosters personal growth or transformations in attitudes or behaviors. The authors investigated what types of peer feedback students remember and what reactions or transformations students experience as a result of peer assessment. Method In May 2005, the authors invited medical students from the second-year (n = 101) and fourth-year (n = 83) classes to provide narratives about how peer assessment affected their personal and professional development. All students had participated in peer assessment during required, formative comprehensive assessments. The authors analyzed responses using mixed qualitative–quantitative methods. Results The 138 responses represented 82% and 69% of students from the fourth-year and second-year classes, respectively. Students recalled peer assessment content about both positive (e.g., teaching skills) and negative (e.g., overconfidence) qualities. Both positive (e.g., appreciation) and negative (e.g., anger) emotional reactions were reported. Many (67%) found peer assessment helpful, reassuring, or confirming of something they knew; 65% reported important transformations in awareness, attitudes, or behaviors because of peer assessment. Change was more likely when feedback was specific and described an area for improvement. Wholly negative responses to the peer assessment process were rare. Conclusions Peer assessment can be a powerful tool to assess and encourage formation of professional behaviors, particularly the interpersonal dimensions. Participants should receive training to provide specific, constructive feedback. The institutional culture should emphasize safety around feedback, while committing to rewarding excellence and addressing concerning behaviors.

Temporal and group-related trends in peer assessment amongst medical students

Context  Peer assessment has been increasingly recommended as a way to evaluate the professional competencies of medical trainees. Prior studies have only assessed single groups measured at a single timepoint. Thus, neither the longitudinal stability of such ratings nor differences between groups using the same peer‐assessment instrument have been reported previously.

Commentary: pitfalls in assessment of competency-based educational objectives.

Requirements for accreditation of medical professionals are increasingly cast in the language of general competencies. Because the language of these competencies is generally shaped by negotiations among stakeholders, however, it has proven difficult to attain consensus on precise definitions. This lack of clarity is amplified when attempting to measure these essentially political constructs in individual learners. The authors of this commentary frame these difficulties within modern views of test validity. The most significant obstacle to valid measurement is not necessarily a lack of useful tools but, rather, a general unwillingness to question whether the competencies themselves represent valid measurement constructs. Although competencies may prove useful in defining an overall social mission for organizations, such competencies should not be mistaken for measurable and distinct attributes that people can demonstrate in the context of their actual work.

Effects of rater selection on peer assessment among medical students

Context  Although peer‐assessment appears promising as a method to assess interpersonal skills among medical students, results may be biased by method of peer selection, particularly if different kinds of classmates are assigned systematically by different methods. It is also unclear whether students with lower interpersonal skills may be more negative towards their classmates than students with higher levels of interpersonal skills and, if so, how much bias this may introduce into the results of peer assessment. It is also unclear whether low‐rated students are more likely to ask to rate one another.

Association Between Hand-off Patients and Subject Exam Performance in Medicine Clerkship Students

ABSTRACTBACKGROUNDTeaching hospitals increasingly rely on transfers of patient care to another physician (hand-offs) to comply with duty hour restrictions. Little is known about the impact of hand-offs on medical students.OBJECTIVETo evaluate the impact of hand-offs on the types of patients students see and the association with their subsequent Medicine Subject Exam performance.DESIGNObservational study over 1 year.PARTICIPANTSThird-year medical students in an Inpatient Medicine Clerkship at five hospitals with night float systems.MEASUREMENTSPrimary outcome: Medicine Subject Exam at the end of the clerkship; explanatory variables: number of fresh (without prior evaluation) and hand-off patients, diagnoses, subspecialty patients, and full evaluations performed during the clerkship, and United Stated Medical Licensing Examination (USMLE) Step I scores.MAIN RESULTSOf the 2,288 patients followed by 89 students, 990 (43.3%) were hand-offs. In a linear regression model, the only variables significantly associated with students’ Subject Exam percentile rankings were USMLE Step I scores (B = 0.26, P < 0.001) and the number of full evaluations completed on fresh patients (B =0.20,  P = 0.048; model r2 = 0.58). In other words, for each additional fresh patient evaluated, Subject Exam percentile rankings increased 0.2 points. For students in the highest quartile of Subject Exam percentile rankings, only Step I scores showed a significant association (B = 0.22, P = 0.002; r2 = 0.5). For students in the lowest quartile, only fresh patient evaluations demonstrated a significant association (B = 0.27, P = 0.03; r2 = 0.34).CONCLUSIONSHand-offs constitute a substantial portion of students’ patients and may have less educational value than “fresh” patients, especially for lower performing students.

Changes in self-perceived abilities among male and female medical students after the first year of clinical training

Background: It is known that male and female medical students have different experiences in their clinical training. Aims: To assess whether male and female medical students change in their self-rated work habits and interpersonal habits during the first year of clinical training. Method: Longitudinal study of self- and peer-assessment among 224 medical students in 3 consecutive classes at a private US medical school. Students rated themselves on global work habits (WH) and interpersonal attributes (IA). Students also rated and were rated by 6–12 peers on the same scale. Results: In the second year of medical school, there were no differences between men and women in quartiles of self-assessed WH or IA. At the end of the third year, however, women were more likely to be in the lower quartiles of self-assessed WH (X2 = 6.77; p = 0.03), as well as the highest quartiles of self-assessed IA (X2 = 11.36; p = 0.003). In both years, women rated their own WH skills significantly lower than they rated their peers, while men rated themselves similarly to peers. There were no sex differences in self-assessed IA. Conclusions: Although second-year male and female medical students appear similar to one another in terms of self-assessed WH and IA, by the end of the third year women rate themselves relatively lower in WH, while men rate themselves relatively lower in IA.

Perceived Control Mediates Effects of Socioeconomic Status and Chronic Stress on Physical Frailty: Findings From the Health and Retirement Study

Objective To investigate the psychosocial etiology of physical frailty by examining the influence of chronic stress and perceived control. Method Using population-based samples of older adults from the Health and Retirement Study, this study employed structural equation modeling in cross-sectional (N = 5,250) and longitudinal (N = 2,013) samples to estimate the effects of chronic stress and socioeconomic status (SES) on baseline frailty and change in frailty status over 4 years and the extent to which perceived control mediates or moderates effects of chronic stress. Results Perceived control fully mediated effects of chronic stress and partially mediated effects of SES on both baseline frailty and change in frailty. Multigroup analyses revealed that the mediating role of perceived control was consistent across age, gender, and racial/ethnic subgroups. There was no evidence to support a moderating role of perceived control in the chronic stress and frailty relationship. Discussion Findings provide novel evidence for a mediating role of perceived control in pathways linking SES and chronic stress to frailty, further underscoring the importance of psychosocial constructs to the development and progression of frailty in older adults.

Maturation of medical student musculoskeletal medicine knowledge and clinical confidence

Purpose : Despite the prevalence of musculoskeletal (MSK) disorders, the degree to which medical schools are providing students the knowledge and confidence to treat these problems is unclear. This study evaluated MSK knowledge in second and fourth year medical students using a newly developed written assessment tool and examined the maturation of clinical confidence in treating core MSK disorders. Methods : Over a 3-year period, the National Board of Medical Examiners (NBME) MSK subject examination consisting of 75 items was administered to 568 second and fourth year students at a single institution. Students were also asked to rate their confidence in treating a selection of medicine/pediatric and MSK clinical scenarios on a 5-point Likert scale. Results : Participation rate was 98%. The NBME MSK assessment score was 59.2±10.6 for all second year medical students and 69.7±9.6 for all fourth year medical students. There was a significant increase in NBME scores between the second and fourth years (p<0.0001). Students were more confident in treating internal medicine/pediatric conditions than MSK medicine conditions (p=0.001). Confidence in treating MSK medicine conditions did not improve between the second and fourth years (p=0.41). Conclusions : To our knowledge, this is the first study to report increased MSK medicine knowledge as measured by a standardized examination after completing medical school core clinical rotations. Despite increased MSK knowledge, low levels of MSK clinical confidence among graduating students were noted. Further research is needed to determine the factors that influence MSK knowledge and clinical confidence in medical students.