Matthew C. Holtman

Assessment of professionalism: Recommendations from the Ottawa 2010 Conference

Over the past 25 years, professionalism has emerged as a substantive and sustained theme, the operationalization and measurement of which has become a major concern for those involved in medical education. However, how to go about establishing the elements that constitute appropriate professionalism in order to assess them is difficult. Using a discourse analysis approach, the International Ottawa Conference Working Group on Professionalism studied some of the dominant notions of professionalism, and in particular the implications for its assessment. The results presented here reveal different ways of thinking about professionalism that can lead towards a multi-dimensional, multi-paradigmatic approach to assessing professionalism at different levels: individual, inter-personal, societal–institutional. Recommendations for research about professionalism assessment are also presented.

Lost in Transition: The Experience and Impact of Frequent Changes in the Inpatient Learning Environment

Purpose The traditional “rotating” model of inpatient training remains the gold standard of residency, moving residents through different systems every two to four weeks. The authors studied the experience and impact of frequent transitions on residents. Method This was a qualitative study. Ninety-seven individuals participated in 12 focus groups at three academic medical centers purposefully chosen to represent a range of geographic locations and structural characteristics. Four groups were held at each site: residents only, faculty only, nurses and ancillary staff only, and a mixed group. Grounded theory was used to analyze data. Results Perceived benefits of transitions included the ability to adapt to new environments and practice styles, improved organization and triage skills, increased comfort with stressful situations, and flexibility. Residents primarily relied on each other to cope with and prepare for transitions, with little support from the program or faculty level. Several potentially problematic workarounds were described within the context of transitions, including shortened progress notes, avoiding pages, hiding information, and sidestepping critical situations. Nearly all residents acknowledged that frequent transitions contributed to a lack of ownership and other potentially harmful effects for patient care. Conclusions These findings challenge the value of the traditional “rotating” model in residency. As residents adapt to frequent transitioning, they implicitly learn to value flexibility and efficiency over relationship building and deep system knowledge. These findings raise significant implications for professional development and patient care and highlight an important element of the hidden curriculum embedded within the current training model.

Which experiences in the hidden curriculum teach students about professionalism

Purpose To examine the relationship between learner experience in the “hidden curriculum” and student attribution of such experiences to professionalism categories. Method Using the output of a thematic analysis of 272 consecutive narratives recorded by 135 students on a medical clerkship from June through November 2007, the authors describe the frequency of these experiences within and across student-designated Association of American Medical Colleges–National Board of Medical Examiners professionalism categories and employ logistic regression to link varieties of experience to specific professionalism categories. Results Thematic analysis uncovered two main domains of student experience: medical–clinical interaction and teaching-and-learning experiences. From a student perspective the critical incident stories evoked all professionalism categories. Most frequently checked off categories were caring/compassion/communication (77%) and respect (69%). Logistic regression suggested that student experiences within the teaching-and-learning environment were associated with professionalism categories of excellence, leadership, and knowledge and skills, whereas those involving medical–clinical interactions were associated with respect, responsibility and accountability, altruism, and honor and integrity. Experiences of communicating and working within teams had the broadest association with learning about professionalism. Conclusions Student narratives touched on all major professionalism categories as well as illuminating the contexts in which critical experiences emerged. Linked qualitative and quantitative analysis identified those experiences that were associated with learning about particular aspects of professionalism. Experiences of teamwork were especially relevant to student learning about professionalism in action.

Assessment of medical professionalism: Who, what, when, where, how, and ... why?

Medical professionalism is increasingly recognized as a core competence of medical trainees and practitioners. Although the general and specific domains of professionalism are thoroughly characterized, procedures for assessing them are not well-developed. This article outlines an approach to designing and implementing an assessment program for medical professionalism that begins and ends with asking and answering a series of critical questions about the purpose and nature of the program. The process of exposing an assessment program to a series of interrogatives that comprise an integrated and iterative framework for thinking about the assessment process should lead to continued improvement in the quality and defensibility of that program.

The quality of written comments on professional behaviors in a developmental multisource feedback program.

Background Written feedback on professional behaviors is an important part of medical training, but little attention has been paid to the quality of written feedback and its expected impact on learning. A large body of research on feedback suggests that feedback is most beneficial when it is specific, clear, and behavioral. Analysis of feedback comments may reveal opportunities to improve the value of feedback. Method Using a directed content analysis, the authors coded and analyzed feedback phrases collected as part of a pilot of a developmental multisource feedback program. The authors coded feedback on various dimensions, including valence (positive or negative) and whether feedback was directed at the level of the self or behavioral performance. Results Most feedback comments were positive, self-oriented, and lacked actionable information that would make them useful to learners. Conclusions Comments often lack effective feedback characteristics. Opportunities exist to improve the quality of comments provided in multisource feedback.

Validity Considerations in the Assessment of Professionalism.

During the last decade, interest in assessing professionalism in medical education has increased exponentially and has led to the development of many new assessment tools. Efforts to validate the scores produced by tools designed to assess professionalism have lagged well behind the development of these tools. This paper provides a structured framework for collecting evidence to support the validity of assessments of professionalism. The paper begins with a short history of the concept of validity in the context of psychological assessment. It then describes Michael Kane’s approach to validity as a structured argument. The majority of the paper then focuses on how Kane’s framework can be applied to assessments of professionalism. Examples are provided from the literature, and recommendations for future investigation are made in areas where the literature is deficient.

A multilevel analysis of the relationships between selected examinee characteristics and United States Medical Licensing Examination Step 2 Clinical Knowledge performance: revisiting old findings and asking new questions.

Background This study examines: (1) the relationships between examinee characteristics and United States Medical Licensing Examination Step 2 Clinical Knowledge (CK) performance; (2) the effect of gender and examination timing (time per item) on the relationship between Steps 1 and 2 CK; and (3) the effect of school characteristics on the relationships between examinee characteristics and Step 2 CK performance. Method A series of hierarchical linear models (examinees-nested-in-schools) predicting Step 2 CK scores was fit to the data set. The sample included 54,487 examinees from 114 U.S. Liaison Committee on Medical Education–accredited medical schools. Results Consistent with past examinee-level research, women generally outperformed men on Step 2 CK, and examinees who received more time per item generally outperformed examinees who received less time per item. Step 1 score was generally more strongly associated with Step 2 CK performance for men and for examinees who received less time per item. School-level characteristics (size, average Step 1 performance) influenced the relationship between Steps 1 and 2 CK. Conclusion Both examinee-level and school-level characteristics are important for understanding Step 2 CK performance.

A theoretical sketch of medical professionalism as a normative complex.

Validity arguments for assessment tools intended to measure medical professionalism suffer for lack of a clear theoretical statement of what professionalism is and how it should behave. Drawing on several decades of field research addressing deviance and informal social control among physicians, a theoretical sketch of professionalism is presented that can be used to predict how individual adjustment to professional norms should co-vary with other social and psychological phenomena. Physicians may understand and value professional norms but fail to enact them in practice because of conflicting normative demands. Physicians’ social networks are predicted to act as conduits of social learning and social pressure, driving the resolution of normative conflicts in specific directions. A valid assessment of professionalism requires an adequate accounting of the social reaction to an individual’s professional conduct, because the reaction and the conduct itself are inseparable.

Evaluation of missing data in an assessment of professional behaviors

Background The National Board of Medical Examiners is currently developing the Assessment of Professional Behaviors, a multisource feedback (MSF) tool intended for formative use with medical students and residents. This study investigated whether missing responses on this tool can be considered random; evidence that missing values are not random would suggest response bias, a significant threat to score validity. Method Correlational analyses of pilot data (N = 2,149) investigated whether missing values were systematically related to global evaluations of observees. Results The percentage of missing items was correlated with global evaluations of observees; observers answered more items for preferred observees compared with nonpreferred observees. Conclusions Missing responses on this MSF tool seem to be nonrandom and are instead systematically related to global perceptions of observees. Further research is needed to determine whether modifications to the items, the instructions, or other components of the assessment process can reduce this effect.

Relationships among item characteristics, examine characteristics, and response times on USMLE Step 1.

In 1999, computer-based test (CBT) administration was introduced for the United States Medical Licensing Examination (USMLE). Currently, all three Steps use ‘‘fixed forms’’: large numbers of tests are constructed to match detailed content and statistical specifications. However, use of sequential (adaptive) testing procedures is under consideration. When these procedures are used, examinees performing well are tested with progressively more difficult items; the reverse is true for examinees performing poorly. To the extent that the time required to respond to an item covaries with item difficulty, more proficient examinees may, as a result, be disadvantaged by adaptive testing procedures, and there is evidence in the literature that this occurs for some kinds of test material. The present study was undertaken as part of the effort to evaluate the desirability of introducing adaptive testing for USMLE. Specifically, the study was designed to gain a better understanding of the relationship between response times and item characteristics (word count, presence of pictures, difficulty), and the extent to which these relationships are influenced by an examinee’s proficiency and English-language background (native speaker vs English as a second language).