Louise Arnold

Assessing Professional Behavior: Yesterday, Today, and Tomorrow

Purpose The author interprets the state of the art of assessing professional behavior. She defines the concept of professionalism, reviews the psychometric properties of key approaches to assessing professionalism, conveys major findings that these approaches produced, and discusses recommendations to improve the assessment of professionalism. Method The author reviewed professionalism literature from the last 30 years that had been identified through database searches; included in conference proceedings, bibliographies, and reference lists; and suggested by experts. The cited literature largely came from peer-reviewed journals, represented themes or novel approaches, reported qualitative or quantitative data about measurement instruments, or described pragmatic or theoretical approaches to assessing professionalism. Results A circumscribed concept of professionalism is available to serve as a foundation for next steps in assessing professional behavior. The current array of assessment tools is rich. However, their measurement properties should be strengthened. Accordingly, future research should explore rigorous qualitative techniques; refine quantitative assessments of competence, for example, through OSCEs; and evaluate separate elements of professionalism. It should test the hypothesis that assessment tools will be better if they define professionalism as behaviors expressive of value conflicts, investigate the resolution of these conflicts, and recognize the contextual nature of professional behaviors. Whether measurement tools should be tailored to the stage of a medical career and how the environment can support or sabotage the assessment of professional behavior are central issues. Final thought Without solid assessment tools, questions about the efficacy of approaches to educating learners about professional behavior will not be effectively answered.

Medical students' views on peer assessment of professionalism.

BACKGROUND: Although peer assessment holds promise for assessing professionalism, reluctance and refusal to participate have been noted among learners and practicing physicians. Understanding the perspectives of potential participants may therefore be important in designing and implementing effective peer assessment.OBJECTIVE: To identify factors that, according to students themselves, will encourage or discourage participation in peer assessment.DESIGN: A qualitative study using grounded theory to interpret views shared during 16 focus groups that were conducted by leaders using a semi-structured guide.PARTICIPANTS: Sixty-one students in Years 1, 3, and 4 in 2 mid-western public medical schools.RESULTS: Three themes students say would promote or discourage peer assessment emerged: personal struggles with peer assessment, characteristics of the assessment system itself, and the environment in which the system operates. Students struggle with reporting an unprofessional peer lest they bring harm to the peer, themselves, or their clinic team or work group. Who receives the assessment and gives the peer feedback and whether it is formative or summative and anonymous, signed, or confidential are important system characteristics. Students’ views of characteristics promoting peer assessment were not unanimous. Receptivity to peer reports and close positive relationships among students and between students and faculty mark an environment conducive to peer assessment, students say.CONCLUSIONS: The study lays a foundation for creating acceptable peer assessment systems among students by soliciting their views. Merely introducing an assessment tool will not result in students’ willingness to assess each other.

Efficacy of Cognitive/Noncognitive Measures in Predicting Resident-Physician Performance.

There has been relatively little research on the relationship between the clinical performance of physicians and the criteria used by medical school admissions and promotions committees. The studies which have been done primarily relate clinical performance to cognitive criteria. They have usually found only negligible relationships. By considering both cognitive and noncognitive variables, several investigators recently have improved the prediction of clinical performance in medical school. The present authors attempt to determine the increase in predictive efficiency attained by adding noncognitive variables to cognitive variables in predicting clinical performance of residents. The results of this investigation indicate that a combination of cognitive and noncognitive predictor variables functions much better than any individual variable or even any specific class of variables in predicting the postgraduate clinical performance of physicians.

A RIME perspective on the quality and relevance of current and future medical education research.

In this article, the authors consider the quality and relevance of current and future medical education research by (1) presenting a framework for medical education research and reviewing basic principles of “good” empirical work, (2) extending the discussion of principles to “best practices,” (3) considering the distinctive features of medical education that present challenges to the researcher, and (4) discussing opportunities for expanding the scope and influence of medical education research. Their audience is intended to be clinicians involved in education, deans and associate deans who create and direct educational curricula and processes, and those from offices critical to the educational mission such as admissions, student services and financial aid, as well as medical education researchers. The authors argue that the quality and relevance of current work can be enhanced when research is situated within a general framework and questions are asked that are based on literature and theory and push the field toward new knowledge. Obviously methods and designs must be appropriate and well-executed and sufficient data must be gathered. Multiple studies are highlighted that showcase the rigor and creativity associated with excellent quality work. However, good research is not without its challenges, most notably short timelines and the need to work within an ever- changing real-life educational environment. Most important, the field of medical education research has many opportunities to increase its impact and advance its quest to study important learners’ behaviors and patients’ outcomes. Programs to train and collaborate with clinical and administrative colleagues, as well as researchers in other fields, have great potential to improve the quality of research in the field.

Can There Be a Single System for Peer Assessment of Professionalism among Medical Students? A Multi-Institutional Study

Purpose Peer assessment is a valuable source of information about medical students’ professionalism. How best to facilitate peer assessment of students’ professional behavior remains to be answered, however. This report extends previous research through a multi-institutional study of students’ perspectives about system characteristics for peer assessment of professionalism. It examines whether students from different schools and year levels prefer different characteristics of peer assessment to assess each other candidly, or whether a single system can be designed. It then identifies the characteristics of the resulting preferred system(s). Method At the beginning of academic year 2004–2005, students (1,661 of 2,115; 78%) in years one through four at four schools replied to a survey about which peer assessment characteristics—related to, for example, who receives the assessment, its anonymity, and timing—would prevent or encourage their participation. Multivariate analysis of variance was used to detect differences among institutions and students from each year level. Results Students across year levels and schools generally agreed about the characteristics of peer assessment. They prefer a system that is 100% anonymous, provides immediate feedback, focuses on both unprofessional and professional behaviors, and uses peer assessment formatively while rewarding exemplary behavior and addressing serious repetitive professional lapses. The system, they emphasize, must be embedded in a supportive environment. Conclusions Students’ agreement about peer-assessment characteristics suggests that one system can be created to meet the majority of students’ preferences. Once implemented, the system should be monitored for student acceptability to maximize participation and to determine the formative and summative value of the process.

Maximizing participation in peer assessment of professionalism: the students speak.

Background Medical students have unique information about peers’ professionalism but are reluctant to share it through peer assessment. Method Students (231 of 375; 62%) in one school replied to a survey about whether various characteristics of peer assessment (e.g., who receives the assessment, its anonymity, implications for the classmate) would prevent or encourage their participation. Results Sixty-six percent of the students agreed that there should be peer assessment of professionalism as long as the assessment reflected their preferences for how the assessment should take place. Some of their preferences included reporting unprofessional behavior to an impartial counselor, a 100% anonymous process, and having the classmate receive corrective instruction. Students across year levels generally agreed about the characteristics of peer assessment. Men and women disagreed about some characteristics. Conclusion Most students are willing to participate in peer assessment as long as their preferences are taken into consideration.

Changing Medical Students’ Attitudes about Older Adults and Future Older Patients

Background An important step in educating future physicians to care for older patients is to diminish the influence of negative stereotypes they often hold about older people. This study presents outcomes of a mentors-on-aging program designed to promote positive attitudes toward older adults and older patients among medical students. Method A pretest, posttest, control group design, with survey data, was used. Results Analysis of students’ preprogram responses compared to postprogram responses indicated that their attitudes did improve. Furthermore, the students’ postprogram attitudes toward older adults were more positive than the control group’s; however, their attitudes toward older patients did not differ from the control group. Conclusions A mentors-on-aging program offering students interaction with healthy, active older adults can help students to acquire positive attitudes toward older people. More targeted experiences with generally healthy older people as patients may be needed to significantly improve students’ attitudes toward older patients.

Development of an instrument to measure the climate of professionalism in a clinical teaching environment.

Background This report describes an instrument that measures professionalism in clinical environments, reports its psychometric properties, and discusses its potential uses. Method The survey asked students (n = 371) to report the frequency of peers’, residents’, and faculty’s professionalism behaviors and faculty’s professionalism teaching, and it asked faculty (n = 28) to self-assess their teaching of professionalism. The authors investigated the instrument’s reliability, convergent validity, and ability to detect differences between groups. Results Coefficient alphas were .75 or higher. Correlations showed positive relationships between students’ perceptions of professionalism behaviors and faculty’s professionalism teaching. t-tests indicated that preclinical students rated faculty’s professionalism behaviors higher than did clinical students, and students rated faculty’s professionalism teaching higher than the faculty rated themselves. Conclusions The psychometrics of the instrument’s scores are sound. The instrument has potential to meet the Liaison Committee on Medical Education’s mandate to measure professional standards within learning environments and to track effects of interventions promoting the professionalism of learners and faculty.

Peer Nominations as Related to Academic Attainment, Empathy, Personality, and Specialty Interest

Purpose To test the hypotheses that peer nomination is associated with measures of (1) academic performance, (2) empathy, (3) personality, and (4) specialty interest. Method In 2007–2008, 255 third-year medical students at Jefferson Medical College were asked to nominate classmates they considered the best in six areas of clinical and humanistic excellence. The authors compared students who received nominations with those who did not, analyzing differences in academic performance, personality factors (empathy as measured by the Jefferson Scale of Empathy and personality qualities as measured by the Zuckerman–Kuhlman Personality Questionnaire), and specialty interests. Results A comparison of the 155 students who received at least one peer nomination with the 100 students who received none found no significant difference in scores on objective examinations; nominated students, however, were rated significantly higher in clinical competence by faculty in six core third-year clerkships. Nominated students were also significantly more empathic and “active.” In addition, a larger proportion of nominated students choose “people-oriented” (rather than “technology- or procedure-oriented”) specialties. Conclusions These results confirmed the hypotheses that peer nomination can predict clinical competence, empathy and other positive personal qualities, and interest in people-oriented specialties. Thus, in the assessment of medical students, peer nomination holds promise as a valid indicator of positive dimensions of professionalism.

Preface: case studies of medical education research groups.

The case studies in this special theme issue of Academic Medicine depict eight highly productive research in medical education groups. They document each group’s productivity, albeit variously measured, and review each group’s history, size, organizational structure, and funding. They discuss perceived reasons for their research success. Finally, they reflect upon past and future challenges and describe responses to them. Staff of Academic Medicine selected the groups included in this issue. After they compiled a list of medical education research groups and informally noted highly productive ones, they compared their informal choices with the groups’ publication rates in medical education journals over the last five years. Adjusting for geographic diversity, the staff invited the medical education research groups at Dartmouth Medical School; Johns Hopkins University School of Medicine; University of Kentucky College of Medicine; University of Maastricht Faculty of Medicine; University of Michigan Medical School; University of California, San Francisco, School of Medicine; University of Toronto Faculty of Medicine; and University of Washington School of Medicine to write case studies. Although the characteristics of individual researchers influence productivity, the quality of research environments is even more important for generating scientific work. Characteristics of these productive research environments, ranging from the age and size of research groups to the nature of their leadership, have been identified, primarily in the natural sciences. This body of work on productive research environments raises a series of important questions about the case studies in this issue for readers to consider. Do the case studies contain themes about environmental characteristics that help us to account for the productivity of these eight groups in medical education? Specifically, is there a diversity among these groups that suggests their productivity is the result of adaptation to unique local conditions? At the same time, are there similarities pointing to more universal forces at work in each group that explain their output? Have the groups created innovations to stimulate productivity? What lessons for the future do these case studies convey? This preface endeavors to begin to answer these questions.