Saad Chahine

In the minds of OSCE examiners: uncovering hidden assumptions

Abstract The Objective Structured Clinical Exam (OSCE) is a widely used method of assessment in medical education. Rater cognition has become an important area of inquiry in the medical education assessment literature generally, and in the OSCE literature specifically, because of concerns about potential compromises of validity. In this study, a novel approach to mixed methods that combined Ordinal Logistic Hierarchical Linear Modeling and cognitive interviews was used to gain insights about what examiners were thinking during an OSCE. This study is based on data from the 2010 to 2014 administrations of the Clinician Assessment for Practice Program OSCE for International Medical Graduates (IMGs) in Nova Scotia. An IMG is a physician trained outside of Canada who was a licensed practitioner in a different country. The quantitative data were examined alongside four follow-up cognitive interviews of examiners conducted after the 2014 administration. The quantitative results show that competencies of (1) Investigation and Management and (2) Counseling were highly predictive of the Overall Global score. These competencies were also described in the cognitive interviews as the most salient parts of OSCE. Examiners also found Communication Skills and Professional Behavior to be relevant but the quantitative results revealed these to be less predictive of the Overall Global score. The interviews also reveal that there is a tacit sequence by which IMGs are expected to proceed in an OSCE, starting with more basic competencies such as History Taking and building up to Investigation Management and Counseling. The combined results confirm that a hidden pattern exists with respect to how examiners rate candidates. This study has potential implications for research into rater cognition, and the design and scoring of practice-ready OSCEs.

What trainees grapple with: a study of threshold concepts on the medicine ward

Socialisation theories of professional identity formation (PIF) consider clinical rotations to be critically intense transformative experiences. However, few studies have explored what trainees grapple with during these transformative experiences or their influence on performance. Applying a threshold concepts (TCs) lens, this study investigates and documents ‘troublesome’ and ‘transformative’ concepts that junior trainees may encounter during a clinical rotation. Insights gained are essential for supporting trainee development.

On the value of the ‘subjective’ in studies of human behavior and cognition

Letter in response to: What people say ≠ what people do We write this letter as a response to the letter ‘What people say ≠ what people do’, in which Dr. van Merrienboer acknowledges the generic value of subjective data but argues that they are unreliable, misleading, and best combined with objective data in the study of behaviour and cognitive processes [1]. As a research group studying teaching and learning in naturalistic clinical settings, we would like to offer a rejoinder. First, we question the dichotomous characterization of data as either subjective or objective. We argue that a spectrum, rather than a dichotomy, exists between ‘subjectivity’ and ‘objectivity’ in research data. At one end of the spectrum are the research approaches Van Merrienboer refers to that ‘[ask] people their opinions’ using interviews, while at the other end are research approaches that attempt to purge all external human influence. Between these poles, degrees of ‘subjectivity’ and ‘objectivity’ exist. Van Merrienboer’s focus on interview techniques that seek opinions belies the richness and diversity of naturalistic data collection methods, which can employ photography, critical incident interview techniques and video-recording of human experiences. Such data are not straightforwardly ‘subjective’: they combine, often in nuanced ways, both more subjective (influenced by human interpretation) and more objective (unfiltered representation) dimensions. Furthermore, the most ‘objective’ research – such as eye tracking analyses or double-blind randomized controlled trials – has subjective dimensions: it is necessarily influenced by human interpretation, from the wording of the question asked, to the inclusion/exclusion criteria of the sampling and the selection of statistical tests [2]. Second, we respectfully disagree with Van Merrienboer’s marginalization of ‘subjective data’ in the study of cognition and behaviour. Calls for studying cognition and behaviour outside of the laboratory [3, 4] suggest that naturalistic research generally, and interview techniques in particular [5, 6] can offer meaningful insights into cognition and behaviour. Furthermore, many techniques exist for enhancing the rigour and authenticity of interview data regarding human cognition and behaviour [5, 6]. Interviews may be framed around clinical case presentations to elicit valuable insights into how clinicians work, [7] as in a recent study exploring faculty supervisory practices [8]. The ‘guided walk’ technique enriches interviews with authentic contextual details, as in a recent study of the lived experience of medical students in remote rural communities [9]. And interview protocols that incorporate workplace observations and visual methods can elicit tacit aspects of expert practice [10, 11]. Importantly, these techniques do not reduce subjectivity in the interview. Rather, they enrich interview data with more perspectives, more interpretive resources, more glimpses of the human participants’ implicit and explicit understandings of their work processes. In conclusion, we suggest putting aside the dichotomy between subjective and objective data. We advocate that medical education researchers draw from the full spectrum of approaches in the exploration of human cognition and behaviour. From our perspective, each methodology and the data it produces have something to contribute; none is intrinsically more valuable.

In Search of Black Swans: Identifying Students at Risk of Failing Licensing Examinations.

Purpose To determine which admissions variables and curricular outcomes are predictive of being at risk of failing the Medical Council of Canada Qualifying Examination Part 1 (MCCQE1), how quickly student risk of failure can be predicted, and to what extent predictive modeling is possible and accurate in estimating future student risk. Method Data from five graduating cohorts (2011–2015), Schulich School of Medicine & Dentistry, Western University, were collected and analyzed using hierarchical generalized linear models (HGLMs). Area under the receiver operating characteristic curve (AUC) was used to evaluate the accuracy of predictive models and determine whether they could be used to predict future risk, using the 2016 graduating cohort. Four predictive models were developed to predict student risk of failure at admissions, year 1, year 2, and pre-MCCQE1. Results The HGLM analyses identified gender, MCAT verbal reasoning score, two preclerkship course mean grades, and the year 4 summative objective structured clinical examination score as significant predictors of student risk. The predictive accuracy of the models varied. The pre-MCCQE1 model was the most accurate at predicting a student’s risk of failing (AUC 0.66–0.93), while the admissions model was not predictive (AUC 0.25–0.47). Conclusions Key variables predictive of students at risk were found. The predictive models developed suggest, while it is not possible to identify student risk at admission, we can begin to identify and monitor students within the first year. Using such models, programs may be able to identify and monitor students at risk quantitatively and develop tailored intervention strategies.

Considering the interdependence of clinical performance: implications for assessment and entrustment

Our ability to assess independent trainee performance is a key element of competency‐based medical education (CBME). In workplace‐based clinical settings, however, the performance of a trainee can be deeply entangled with others on the team. This presents a fundamental challenge, given the need to assess and entrust trainees based on the evolution of their independent clinical performance. The purpose of this study, therefore, was to understand what faculty members and senior postgraduate trainees believe constitutes independent performance in a variety of clinical specialty contexts.

Unpacking the Literature on Stress and Resiliency: A Narrative Review Focused on Learners in the Operating Room

PURPOSE The operating room is a high pressure environment for surgical trainees as they attempt to reach a high level of performance in the midst of a multitude of stressors. The purpose of this work was to examine the relationships between stress, coping, and psychological resilience and their effects on performance and learning in surgical training. METHODS A narrative review was carried out of the existing literature on stress, coping, and resilience in surgeons and surgical trainees. Multiple fields of study were examined including medical education, surgery, surgical safety, anesthesia, workplace ergonomics, and psychology. RESULTS Sources of intraoperative stress include fatigue, disruptions, interpersonal conflicts, time pressure, a complex case or high risk patient, surgical errors, and surgeon temperament. These stressors can negatively impact the performance of surgeons and trainees and may inhibit learning. How a learner responds to stress in the operating room is highly variable and influenced by the context of the stress, the coping mechanisms available, and individual psychological resilience. Stress management techniques, such as mental rehearsal, are beneficial for reducing stress. Resilience is protective against stress and burnout, and resilience training is useful for reducing stress and improving mental health in physicians and medical students. CONCLUSIONS Surgical trainees experience significant stress in the operating room and their experience of stress is modulated by cognitive and behavioral factors. Further research is required on the development of effective interventions to help trainees manage intraoperative stress, with the potential to improve surgical performance, learning, and patient safety.

What Matters from Admissions? Identifying Success and Risk Among Canadian Dental Students

The aims of this study were to determine whether different student profiles would emerge in terms of high and low GPA performance in each year of dental school and to investigate the utility of preadmissions variables in predicting performance and performance stability throughout each year of dental school. Data from 11 graduating cohorts (2004-14) at the Schulich School of Medicine & Dentistry, University of Western Ontario, Canada, were collected and analyzed using bivariate correlations, latent profile analysis, and hierarchical generalized linear models (HGLMs). The data analyzed were for 616 students in total (332 males and 284 females). Four models were developed to predict adequate and poor performance throughout each of four dental school years. An additional model was developed to predict student performance stability across time. Two separate student profiles reflecting high and low GPA performance across each year of dental school were identified, and scores on cognitive preadmissions variables differentially predicted the probability of grouping into high and low performance profiles. Students with higher pre-dental GPAs and DAT chemistry were most likely to remain stable in a high-performance group across each year of dental school. Overall, the findings suggest that selection committees should consider pre-dental GPA and DAT chemistry scores as important tools for predicting dental school performance and stability across time. This research is important in determining how to better predict success and failure in various areas of preclinical dentistry courses and to provide low-performing students with adequate academic assistance.

From good to excellent: Improving clinical departments’ learning climate in residency training

Abstract Introduction: The improvement of clinical departments’ learning climate is central to achieving high-quality residency training and patient care. However, improving the learning climate can be challenging given its complexity as a multi-dimensional construct. Distinct representations of the dimensions might create different learning climate groups across departments and may require varying efforts to achieve improvement. Therefore, this study investigated: (1) whether distinct learning climate groups could be identified and (2) whether contextual factors could explain variation in departments’ learning climate performance. Methods: This study included departments that used the Dutch Residency Educational Climate Test (D-RECT) through a web-based system in 2014–2015. Latent profile analysis was used to identify learning climate groups and multilevel modeling to predict clinical departments’ learning climate performance. Results: The study included 1730 resident evaluations. Departments were classified into one of the four learning climate groups: substandard, adequate, good and excellent performers. The teaching status of the hospital, departments’ average teaching performance and percentage of time spent on educational activities by faculty-predicted departments’ learning climate performance. Discussion: Clinical departments can be successfully classified into informative learning climate groups. Ideally, given informative climate grouping with potential for cross learning, the departments could embark on targeted performance improvement.

Peer assessment in the objective structured clinical examination: A scoping review

Abstract Background: The objective structured clinical examination (OSCE), originally designed with experts assessing trainees’ competence, is more frequently employed with an element of peer assessment and feedback. Although peer assessment in higher education has been studied, its role in OSCEs has not reviewed. Aims: The aim of this study is to conduct a scoping review and explore the role of peer assessment and feedback in the OSCE. Methods: Electronic database and hand searching yielded 507 articles. Twenty-one full records were screened, of which 13 were included in the review. Two independent reviewers completed each step of the review. Results: Peer-based OSCEs are used to assess students’ accuracy in assessing OSCE performance and to promote learning. Peer examiners (PE) tend to award better global ratings and variable checklist ratings compared to faculty and provide high-quality feedback. Participating in these OSCEs is perceived as beneficial for learning. Conclusions: Peer assessment and feedback can be used to gauge PE reliability and promote learning. Teachers using these OSCEs must use methodology which fits their purpose. Competency-based education calls for diversification of assessment practices and asks how assessment impacts learning; the peer-based OSCE responds to these demands and will become an important practice in health professions education.