Larry D. Gruppen

Measuring Self-assessment: Current State of the Art

The competent physician pursues lifelong learning through the recognition of deficiencies and the formulation of appropriate learning goals. Despite the accepted theoretical value of self-assessment, studies have consistently shown that the accuracy of self-assessment is poor. This paper examines the methodological issues that plague the measurement of self-assessment ability and presents several strategies that address these methodological problems within the current paradigm. In addition, the article proposes an alternative conceptualization of self-assessment and describes its associated methods. The conclusions of prior research in this domain must be re-examined in light of the common pitfalls encountered in the design of the studies and the analyses of the data. Future efforts to elucidate self-assessment phenomena need to consider the implications of this review.

Defining characteristics of educational competencies

Context  Doctor competencies have become an increasing focus of medical education at all levels. However, confusion exists regarding what constitutes a competency versus a goal, objective or outcome.

Teaching the one-minute preceptor. A randomized controlled trial.

OBJECTIVE: The One-Minute Preceptor (OMP) model of faculty development is used widely to improve teaching, but its effect on teaching behavior has not been assessed. We aim to evaluate the effect of this intervention on residents’ teaching skills.DESIGN: Randomized controlled trial.SETTING: Inpatient teaching services at both a tertiary care hospital and a Veterans Administration Medical Center affiliated with a University Medical Center.PARTICIPANTS: Participants included 57 second- and third-year internal medicine residents that were randomized to the intervention group (n=28) or to the control group (n=29).INTERVENTION: The intervention was a 1-hour session incorporating lecture, group discussion, and role-play.MEASUREMENTS AND MAIN RESULTS: Primary outcome measures were resident self-report and learner ratings of resident performance of the OMP teaching behaviors. Residents assigned to the intervention group reported statistically significant changes in all behaviors (P<.05). Eighty-seven percent of residents rated the intervention as “useful or very useful” on a 1–5 point scale with a mean of 4.28. Student ratings of teacher performance showed improvements in all skills except “Teaching General Rules.” Learners of the residents in the intervention group reported increased motivation to do outside reading when compared to learners of the control residents. Ratings of overall teaching effectiveness were not significantly different between the 2 groups.CONCLUSIONS: The OMP model is a brief and easy-to-administer intervention that provides modest improvements in residents’ teaching skills.

Twelve tips for developing effective mentors

Mentoring is often identified as a crucial step in achieving career success. However, not all medical trainees or educators recognize the value of a mentoring relationship. Since medical educators rarely receive training on the mentoring process, they are often ill equipped to face challenges when taking on major mentoring responsibilities. This article is based on half-day workshops presented at the 11th Ottawa International Conference on Medical Education in Barcelona on 5 July 2004 and the annual meeting of the Association of American Medical Colleges in Boston on 10 November 2004 as well as a review of literature. Thirteen medical faculty participated in the former and 30 in the latter. Most participants held leadership positions at their institutions and mentored trainees as well as supervised mentoring programs. The workshops reviewed skills of mentoring and strategies for designing effective mentoring programs. Participants engaged in brainstorming and interactive discussions to: (a) review different types of mentoring programs; (b) discuss measures of success and failure of mentoring relationships and programs; and (c) examine the influence of gender and cultural differences on mentoring. Participants were also asked to develop an implementation plan for a mentoring program for medical students and faculty. They had to identify student and faculty mentoring needs, and describe methods to recruit mentors as well as institutional reward systems to encourage and support mentoring.

The prediction of professional behaviour

Objective  The purpose of this study was to establish outcome measures for professionalism in medical students and to identify predictors of these outcomes.

Faculty development for educational leadership and scholarship.

The Medical Education Scholars Program (MESP) at the University of Michigan Medical School is designed to develop leaders in medical education. The programs goals are to enable faculty to provide curriculum direction, improved teaching, educational research, and development, and institutional leadership at all levels of medical education. This one-year program uses a variety of educational methods and provides a broad curriculum in educational theory, assessment and evaluation, research design and methods, teaching-skills development, and educational leadership. Faculty are admitted on a competitive basis and one half-day per week of release time is funded as part of the program. Salient outcomes of the program (promotions, educational research and development, curriculum leadership, and educational scholarship) were measured in a pre- and post-program design in which each scholar acted as his or her own control. There were major increases in promotions and educational awards, new educational responsibilities, and new educational programs. A particularly important outcome was the emergence of educational scholarship in the professional portfolios of the program scholars in the form of peer-reviewed presentations and publications and educational grant funding. A cost–outcome analysis indicates that these multi-year outcomes were obtained from a one-time investment of approximately

Educational Fellowship Programs: Common Themes and Overarching Issues

21,000 per graduating scholar. This evaluation indicates that intensive faculty development programs can have measurable impacts on the careers of the participants and the institutional environment.

The Third Version of the Diabetes Attitude Scale

The trend toward intensive faculty development programs has been driven by a variety of factors, including institutional needs for educational expertise and leadership, as well as individual faculty members’ motivation to augment their educational expertise, teaching skills, and leadership skills. The nine programs described in this issue possess several common features that can be ascribed to shared perceptions of pervasive needs coupled with feasible educational resources and strategies to meet these needs. All programs identify a clear set of goals and objectives for their respective curricula. Curriculum domains include not only teaching skills but also educational research, curriculum development, and educational leadership. In spite of many similarities, each program reflects the unique character of its home institution, the faculty, educational resources, and the specific goals of the program. Each program has documented gains in such key outcomes as participant promotions, new leadership positions both locally and nationally, and scholarly productivity in the form of peer-reviewed papers and presentations. Evidence of institutional benefits includes the production of innovative curricula and a pool of educational leaders. The programs have also developed a community of knowledgeable scholars who interact with each other and serve as a catalyst for continuing change and educational improvement. Although each program was developed largely independently of the others, the common elements in their design provide opportunities to evaluate collaboratively the successful aspects of such programs and to share ideas and resources for program curricula between existing programs and with institutions considering implementing new programs.

Assessing residents' competencies at baseline: identifying the gaps.

OBJECTIVE The objective of this study was to develop a third version of the Diabetes Attitude Scale (DAS-3) that is congruent with current scientific knowledge about diabetes, has improved subscale internal reliability scores, and is shorter than the earlier versions of this instrument. RESEARCH DESIGN AND METHODS The second DAS was revised and rewritten by a panel of diabetes experts, including patients, associated with the University of Michigan Diabetes Research and Training Center. The revised version of the instrument was sent to physicians, nurses, dietitians, and patients with diabetes. Completed and usable questionnaires were obtained from 384 patients with diabetes, 321 physicians, 540 nurses, and 569 dietitians. The total number of surveys used for these analyses was 1,814. RESULTS The study resulted in a revised DAS with 33 items and five discrete subscales. The subscales were attitudes toward the following: 1) need for special training to provide diabetes care, 2) seriousness of type 2 diabetes, 3) value of tight glucose control, 4) pyschosocial impact of diabetes, and 5) attitude toward patient autonomy. Overall, the subscale reliabilities of the DAS-3 were superior to the earlier versions of the scale. CONCLUSIONS The DAS-3 is a valid and reliable general measure of diabetes-related attitudes and is most suitable for comparisons across different groups of health care professionals and/or patients. The DAS-3 is also suitable for the evaluation of patient and/or professional education programs if those programs focus on the specific topic areas measured by the five DAS-3 subscales.

The promise of competency-based education in the health professions for improving global health

Purpose. Entering residents have variable medical school experiences and differing knowledge and skill levels. To structure curricula, enhance patient safety, and begin to meet accreditation requirements, baseline assessment of individual residents knowledge and skills is needed. To this end, in 2001 the University of Michigan Health System created the Postgraduate Orientation Assessment (POA), an eight-station, objective structured clinical examination for incoming residents. Method. The POA, administered at orientation, included items addressing critical laboratory values, cross-cultural communication, evidence-based medicine, radiographic image interpretation, informed consent, pain assessment and management, aseptic technique, and system compliance such as fire safety. The POA assessed many of the skills needed by interns in their initial months of training when supervision by senior physicians might not be present. Results. In 2002, 132 interns from 14 different specialties and 59 different schools participated in the POA. The mean score was 74.8% (SD = 5.8). When scores were controlled for U.S. Medical Licensing Examination scores, there were no significant differences in performance across specialties. There were differences between University of Michigan Medical School graduates and those from other institutions (p < .001). Eighty-one percent of the residents would recommend the POA. Conclusions. The POA provides a feasible format to measure initial knowledge and skills and identify learning needs. Orientation is an effective time to identify important gaps in learning between medical school and residency. This is the first step in a continuing evaluation of the Accreditation Council for Graduate Medical Educations general competencies.