Shiphra Ginsburg

Assessment of professionalism: Recommendations from the Ottawa 2010 Conference

Over the past 25 years, professionalism has emerged as a substantive and sustained theme, the operationalization and measurement of which has become a major concern for those involved in medical education. However, how to go about establishing the elements that constitute appropriate professionalism in order to assess them is difficult. Using a discourse analysis approach, the International Ottawa Conference Working Group on Professionalism studied some of the dominant notions of professionalism, and in particular the implications for its assessment. The results presented here reveal different ways of thinking about professionalism that can lead towards a multi-dimensional, multi-paradigmatic approach to assessing professionalism at different levels: individual, inter-personal, societal–institutional. Recommendations for research about professionalism assessment are also presented.

The Professionalism Mini-evaluation Exercise: a preliminary investigation.

Background As the evaluation of professional behaviors has been identified as an area for development, the Professionalism Mini-Evaluation Exercise (P-MEX) was developed using the mini-Clinical Examination Exercise (mini-CEX) format. Method From a set of 142 observable behaviors reflective of professionalism identified at a McGill workshop, 24 were converted into an evaluation instrument modeled on the mini-CEX. This instrument, designed for use in multiple settings, was tested on clinical clerks in medicine, surgery, obstetrics and gynecology, psychiatry, and pediatrics. In all, 211 forms were completed on 74 students by 47 evaluators. Results Results indicate content and construct validity. Exploratory factor analysis yielded 4 factors: doctor–patient relationship skills, reflective skills, time management, and interprofessional relationship skills. A decision study showed confidence intervals sufficiently narrow for many measurement purposes with as few as 8 observations. Four items frequently marked below expectations may be identifiers for “problem” students. Conclusion This preliminary study suggests that the P-MEX is a feasible format for evaluating professionalism in clinical training.

The Disavowed Curriculum: Understanding Student's Reasoning in Professionally Challenging Situations

CONTEXT: Understanding students’ perceptions of and responses to lapses in professionalism is important to shaping students’ professional development.OBJECTIVE: Utilize realistic, standardized professional dilemmas to obtain insight into students’ reasoning and motivations in “real time.”DESIGN: Qualitative study using 5 videotaped scenarios (each depicting a student placed in a situation which requires action in response to a professional dilemma) and individual interviews, in which students were questioned about what they would do next and why.SETTING: University of Toronto.PARTICIPANTS: Eighteen fourth-year medical students; participation voluntary and anonymous.MAIN OUTCOME MEASURE: A model to explain students’ reasoning in the face of professional dilemmas.RESULTS: Grounded theory analysis of interview transcripts revealed that students were motivated to consider specific actions by referencing a Principle (an abstract or idealized concept), an Affect (a feeling or emotion), or an Implication (a potential consequence of suggested actions). Principles were classified as “avowed” as ideals of four profession (e.g., honesty or disclosure), or “unavowed” (unacknowledged or undeclared, e.g., obedience or allegiance). Implications could also be avowed (e.g., concerning patients) or unavowed (e.g., concerning others); but students were predominantly motivated by considering “disavowed” implications: those pertaining to themselves (e.g., concern for grades, evaluations, or reputation), which are actively denied by the profession and discouraged as being inconsistent with altruism.CONCLUSIONS: This “disavowed curriculum” has implications for education, feedback, and evaluation. Instead of denying their existence, we should teach students how to negotiate and balance these unavowed and disavowed implications and principles, in order to help them develop their own professional stance.

Medical students' views on peer assessment of professionalism.

BACKGROUND: Although peer assessment holds promise for assessing professionalism, reluctance and refusal to participate have been noted among learners and practicing physicians. Understanding the perspectives of potential participants may therefore be important in designing and implementing effective peer assessment.OBJECTIVE: To identify factors that, according to students themselves, will encourage or discourage participation in peer assessment.DESIGN: A qualitative study using grounded theory to interpret views shared during 16 focus groups that were conducted by leaders using a semi-structured guide.PARTICIPANTS: Sixty-one students in Years 1, 3, and 4 in 2 mid-western public medical schools.RESULTS: Three themes students say would promote or discourage peer assessment emerged: personal struggles with peer assessment, characteristics of the assessment system itself, and the environment in which the system operates. Students struggle with reporting an unprofessional peer lest they bring harm to the peer, themselves, or their clinic team or work group. Who receives the assessment and gives the peer feedback and whether it is formative or summative and anonymous, signed, or confidential are important system characteristics. Students’ views of characteristics promoting peer assessment were not unanimous. Receptivity to peer reports and close positive relationships among students and between students and faculty mark an environment conducive to peer assessment, students say.CONCLUSIONS: The study lays a foundation for creating acceptable peer assessment systems among students by soliciting their views. Merely introducing an assessment tool will not result in students’ willingness to assess each other.

A contemporary approach to validity arguments: a practical guide to Kane's framework

Assessment is central to medical education and the validation of assessments is vital to their use. Earlier validity frameworks suffer from a multiplicity of types of validity or failure to prioritise among sources of validity evidence. Kanes framework addresses both concerns by emphasising key inferences as the assessment progresses from a single observation to a final decision. Evidence evaluating these inferences is planned and presented as a validity argument.

Lost in Transition: The Experience and Impact of Frequent Changes in the Inpatient Learning Environment

Purpose The traditional “rotating” model of inpatient training remains the gold standard of residency, moving residents through different systems every two to four weeks. The authors studied the experience and impact of frequent transitions on residents. Method This was a qualitative study. Ninety-seven individuals participated in 12 focus groups at three academic medical centers purposefully chosen to represent a range of geographic locations and structural characteristics. Four groups were held at each site: residents only, faculty only, nurses and ancillary staff only, and a mixed group. Grounded theory was used to analyze data. Results Perceived benefits of transitions included the ability to adapt to new environments and practice styles, improved organization and triage skills, increased comfort with stressful situations, and flexibility. Residents primarily relied on each other to cope with and prepare for transitions, with little support from the program or faculty level. Several potentially problematic workarounds were described within the context of transitions, including shortened progress notes, avoiding pages, hiding information, and sidestepping critical situations. Nearly all residents acknowledged that frequent transitions contributed to a lack of ownership and other potentially harmful effects for patient care. Conclusions These findings challenge the value of the traditional “rotating” model in residency. As residents adapt to frequent transitioning, they implicitly learn to value flexibility and efficiency over relationship building and deep system knowledge. These findings raise significant implications for professional development and patient care and highlight an important element of the hidden curriculum embedded within the current training model.

Towards a program of assessment for health professionals: from training into practice

Despite multifaceted attempts to “protect the public,” including the implementation of various assessment practices designed to identify individuals at all stages of training and practice who underperform, profound deficiencies in quality and safety continue to plague the healthcare system. The purpose of this reflections paper is to cast a critical lens on current assessment practices and to offer insights into ways in which they might be adapted to ensure alignment with modern conceptions of health professional education for the ultimate goal of improved healthcare. Three dominant themes will be addressed: (1) The need to redress unintended consequences of competency-based assessment; (2) The potential to design assessment systems that facilitate performance improvement; and (3) The importance of ensuring authentic linkage between assessment and practice. Several principles cut across each of these themes and represent the foundational goals we would put forward as signposts for decision making about the continued evolution of assessment practices in the health professions: (1) Increasing opportunities to promote learning rather than simply measuring performance; (2) Enabling integration across stages of training and practice; and (3) Reinforcing point-in-time assessments with continuous professional development in a way that enhances shared responsibility and accountability between practitioners, educational programs, and testing organizations. Many of the ideas generated represent suggestions for strategies to pilot test, for infrastructure to build, and for harmonization across groups to be enabled. These include novel strategies for OSCE station development, formative (diagnostic) assessment protocols tailored to shed light on the practices of individual clinicians, the use of continuous workplace-based assessment, and broadening the focus of high-stakes decision making beyond determining who passes and who fails. We conclude with reflections on systemic (i.e., cultural) barriers that may need to be overcome to move towards a more integrated, efficient, and effective system of assessment.

Using "Standardized Narratives" to Explore New Ways to Represent Faculty Opinions of Resident Performance

Purpose Most efforts to develop reliable evaluations of clinical competence have been oriented toward deconstructing the requisite competencies into separate scales. However, many are questioning the value of this approach on theoretical and empirical bases. This study uses “standardized narratives” to explore a different approach to assessing resident performance. Method In 2009, based on interviews with 19 experienced clinical faculty from two institutions, 16 narrative profiles were created to represent the range of resident competence that clinical faculty might encounter during supervision. Fourteen clinicians from three institutions independently grouped the profiles into as many categories as necessary to reflect various levels of performance, described their categories, then ranked the individual profiles within each category. Then, in groups of three or four, participants negotiated a final ranking and grouping of the 16 profiles. Results Despite interesting idiosyncracies in the factors some participants identified as guiding their rankings, there was strong consistency across the 14 clinicians regarding the rankings (single-rater intraclass correlation [ICC] = 0.86) and groupings (single-rater ICC = 0.81) of the profiles. Similarly, across institutions, the four groups were highly consistent in their final negotiated rankings (single-group ICC = 0.91) and groupings (single-group ICC = 0.87) of the profiles. Conclusions Faculty showed more consistency in their decisions of what constitutes excellent, competent, and problematic performance in residents than implied by current assessment techniques that require deconstruction of resident competencies. This use of standardized narratives points to interesting opportunities for more authentically codifying faculty opinions of residents.

Reading between the lines: faculty interpretations of narrative evaluation comments

Narrative comments are used routinely in many forms of rater‐based assessment. Interpretation can be difficult as a result of idiosyncratic writing styles and disconnects between literal and intended meanings. Our purpose was to explore how faculty attendings interpret and make sense of the narrative comments on residents’ in‐training evaluation reports (ITERs) and to determine the language cues that appear to be influential in generating and justifying their interpretations.

A cross-cultural study of students’ approaches to professional dilemmas: sticks or ripples

Medical Education 2012: 46 : 245–256