Laura Naismith

Faculty development initiatives designed to promote leadership in medical education. A BEME systematic review: BEME Guide No. 19.

Background: Due to the increasing complexity of medical education and practice, the preparation of healthcare professionals for leadership roles and responsibilities has become increasingly important. To date, the literature on faculty development designed to promote leadership in medical education has not been reviewed in a systematic fashion. Aim: The objective of this review is to synthesize the existing evidence that addresses the following question: ‘What are the effects of faculty development interventions designed to improve leadership abilities on the knowledge, attitudes, and skills of faculty members in medicine and on the institutions in which they work?’ Methods: Search strategy: The search, which covered the period 1980–2009, included six databases (Medline, EMBASE, CINAHL, Web of Science, ERIC, and ABI/Inform) and used the following keywords: faculty development; in-service training; doctor; medic; physician; faculty; leadership; management; administration; executive; and change agent. Hand searches were also conducted, and expert recommendations were solicited. Inclusion and exclusion criteria: Articles with a focus on faculty development to improve leadership, targeting basic science and clinical faculty members, were reviewed. All study designs that included outcome data beyond participant satisfaction were examined. From an initial 687 unique records, 48 articles met the review criteria in three broad categories: (1) reports in which leadership was the primary focus of the intervention; (2) reports in which leadership was a component of a broader focus on educational development; and (3) reports in which leadership was a component of a broader focus on academic career development. Data extraction: Data were extracted by three coders using the standardized Best Evidence Medical Education coding sheet adapted for our use. One reviewer coded all of the articles, and two reviewers each coded half of the dataset. Coding differences were resolved through discussion. Data synthesis: Data were synthesized using Kirkpatricks four levels of educational outcomes. Findings were grouped by intervention type and level of outcome. Results: Forty-eight articles described 41 studies of 35 different interventions. The majority of the interventions targeted clinical faculty members and included workshops, short courses, fellowships, and other longitudinal programs. The majority of studies were quantitative in nature, though five studies used a qualitative design, and 12 studies used mixed methods. All quantitative studies were quasi-experimental and most employed a single group design; only two studies had a comparison group. Qualitative study designs were typically not specified. The majority of evaluation data, primarily collected post-intervention, consisted of participants’ responses to questionnaires and interviews. Key points and summary of outcomes: Despite methodological limitations, the faculty development literature tends to support the following outcomes: ▪ High satisfaction with faculty development programs. Participants consistently found programs to be useful and of both personal and professional benefit. They also valued the practical relevance and applicability of the instructional methods used. ▪ A change in attitudes toward organizational contexts and leadership roles. Participants reported positive changes in attitudes toward their own organizations as well as their leadership capabilities. Some reported an increased awareness of – and commitment to – their institutions vision and challenges, whereas others reported greater self-awareness of personal strengths and limitations, increased motivation, and confidence in their leadership roles. A greater sense of community and appreciation of the benefits of networking were also identified. ▪ Gains in knowledge and skills. Participants reported increased knowledge of leadership concepts, principles, and strategies (e.g., leadership styles and strategic planning), gains in specific leadership skills (e.g., personal effectiveness and conflict resolution), and increased awareness of leadership roles in academic settings. ▪ Changes in leadership behavior. Self-perceived changes in leadership behavior were consistently reported and included a change in leadership styles, the application of new skills to the workplace (e.g., departmental reorganization and team building), the adoption of new leadership roles and responsibilities, and the creation of new collaborations and networks. Observed changes primarily suggested new leadership positions. ▪ Limited changes in organizational practice. Although not frequently examined, changes in organizational practice included the implementation of specific educational innovations, an increased emphasis on educational scholarship, and the establishment of collegial networks. ▪ Key features of faculty development. Features contributing to positive outcomes included the use of: multiple instructional methods within single interventions; experiential learning and reflective practice; individual and group projects; peer support and the development of communities of practice; mentorship; and institutional support. ▪ Avenues for future development: Moving forward, faculty development programs should: ground their work in a theoretical framework; articulate their definition of leadership; consider the role of context; explore the value of extended programs and follow-up sessions; and promote the use of alternative practices including narrative approaches, peer coaching, and team development. Methodological issues: More rigorous and diverse research designs are needed to capture the complexity of interventions in this area. Varied methods of assessment, utilizing multiple data sources to tap changes at the interpersonal and organizational level should be explored, as should the maintenance of change over time. Process-oriented studies, comparing different faculty development strategies and clarifying the process of change through faculty development, should also become a priority. Conclusion: Participants value leadership development activities and report changes in attitudes, knowledge, skills and behavior. Moreover, despite methodological limitations, certain program characteristics seem to be associated with positive outcomes. Further research is required to explore these associations and document changes at both the individual and organizational level.

A systematic review of faculty development initiatives designed to enhance teaching effectiveness: A 10-year update: BEME Guide No. 40

Abstract Background: This review, which focused on faculty development initiatives designed to improve teaching effectiveness, synthesized findings related to intervention types, study characteristics, individual and organizational outcomes, key features, and community building. Methods: This review included 111 studies (between 2002 and 2012) that met the review criteria. Findings: Overall satisfaction with faculty development programs was high. Participants reported increased confidence, enthusiasm, and awareness of effective educational practices. Gains in knowledge and skills, and self-reported changes in teaching behaviors, were frequently noted. Observed behavior changes included enhanced teaching practices, new educational initiatives, new leadership positions, and increased academic output. Organizational changes were infrequently explored. Key features included evidence-informed educational design, relevant content, experiential learning, feedback and reflection, educational projects, intentional community building, longitudinal program design, and institutional support. Conclusion: This review holds implications for practice and research. Moving forward, we should build on current success, broaden the focus beyond individual teaching effectiveness, develop programs that extend over time, promote workplace learning, foster community development, and secure institutional support. We should also embed studies in a theoretical framework, conduct more qualitative and mixed methods studies, assess behavioral and organizational change, evaluate transfer to practice, analyse key features, and explore the role of faculty development within the larger organizational context.

Limitations of subjective cognitive load measures in simulation‐based procedural training

The effective implementation of cognitive load theory (CLT) to optimise the instructional design of simulation‐based training requires sensitive and reliable measures of cognitive load. This mixed‐methods study assessed relationships between commonly used measures of total cognitive load and the extent to which these measures reflected participants’ experiences of cognitive load in simulation‐based procedural skills training.

Technology-Rich Tools to Support Self-Regulated Learning and Performance in Medicine

Medical students’ metacognitive and self-regulatory behaviors are examined as they diagnose patient cases using BioWorld, a technology rich learning environment. BioWorld offers an authentic problem-based environment where students solve clinical cases and receive expert feedback. We evaluate the effectiveness of key features in BioWorld (the evidence table and visualization maps) to see whether they promote metacognitive monitoring and evaluation. Learning outcomes were assessed through novice/expert comparisons in relation to diagnostic accuracy, confidence, and case summaries. More specifically we examined how diagnostic processes and learning outcomes were refined or improved through practice at solving a series of patient cases. The results suggest that, with practice, medical students became more expert-like in the processes involved in making crucial clinical decisions. The implications of these findings for the design of features embedded within BioWorld that foster key metacognitive and self-regulatory processes are discussed.

Validity of Cognitive Load Measures in Simulation-Based Training: A Systematic Review.

Background Cognitive load theory (CLT) provides a rich framework to inform instructional design. Despite the applicability of CLT to simulation-based medical training, findings from multimedia learning have not been consistently replicated in this context. This lack of transferability may be related to issues in measuring cognitive load (CL) during simulation. The authors conducted a review of CLT studies across simulation training contexts to assess the validity evidence for different CL measures. Method PRISMA standards were followed. For 48 studies selected from a search of MEDLINE, EMBASE, PsycInfo, CINAHL, and ERIC databases, information was extracted about study aims, methods, validity evidence of measures, and findings. Studies were categorized on the basis of findings and prevalence of validity evidence collected, and statistical comparisons between measurement types and research domains were pursued. Results CL during simulation training has been measured in diverse populations including medical trainees, pilots, and university students. Most studies (71%; 34) used self-report measures; others included secondary task performance, physiological indices, and observer ratings. Correlations between CL and learning varied from positive to negative. Overall validity evidence for CL measures was low (mean score 1.55/5). Studies reporting greater validity evidence were more likely to report that high CL impaired learning. Conclusions The authors found evidence that inconsistent correlations between CL and learning may be related to issues of validity in CL measures. Further research would benefit from rigorous documentation of validity and from triangulating measures of CL. This can better inform CLT instructional design for simulation-based medical training.

Pain catastrophizing correlates with early mild traumatic brain injury outcome.

Background. Identifying which patients are most likely to be at risk of chronic pain and other postconcussion symptoms following mild traumatic brain injury (MTBI) is a difficult clinical challenge. Objectives. To examine the relationship between pain catastrophizing, defined as the exaggerated negative appraisal of a pain experience, and early MTBI outcome. Methods. This cross-sectional design included 58 patients diagnosed with a MTBI. In addition to medical chart review, postconcussion symptoms were assessed by self-report at 1 month (Time 1) and 8 weeks (Time 2) after MTBI. Pain severity, psychological distress, level of functionality, and pain catastrophizing were measured by self-report at Time 2. Results. The pain catastrophizing subscales of rumination, magnification, and helplessness were significantly correlated with pain severity (r = .31 to .44), number of postconcussion symptoms reported (r = .35 to .45), psychological distress (r = .57 to .67), and level of functionality (r = −.43 to −.29). Pain catastrophizing scores were significantly higher for patients deemed to be at high risk of postconcussion syndrome (6 or more symptoms reported at both Time 1 and Time 2). Conclusions. Higher levels of pain catastrophizing were related to adverse early MTBI outcomes. The early detection of pain catastrophizing may facilitate goal-oriented interventions to prevent or minimize the development of chronic pain and other postconcussion symptoms.

Using expert models to provide feedback on clinical reasoning skills

Effective feedback is necessary to support expertise development in clinical reasoning Technology-rich environments (TREs) often use expert models as one means of providing this feedback A review of empirical studies showed 3 different types of expert models in TREs: outcome models, process models, and dynamic models This paper presents examples of each of these models and discusses their implications for the future design of feedback mechanisms to support clinical reasoning development through self-assessment.

Practising what we preach: using cognitive load theory for workshop design and evaluation

Theory-based instructional design is a top priority in medical education. The goal of this Show and Tell article is to present our theory-driven approach to the design of instruction for clinical educators. We adopted cognitive load theory as a framework to design and evaluate a series of professional development workshops that were delivered at local, national and international academic conferences in 2014. We used two rating scales to measure participantsʼ cognitive load. Participants also provided narrative comments as to how the workshops could be improved. Cognitive load ratings from 59 participants suggested that the workshop design optimized learning by managing complexity for different levels of learners (intrinsic load), stimulating cognitive processing for long-term memory storage (germane load), and minimizing irrelevant distracters (extraneous load). Narrative comments could also be classified as representing intrinsic, extraneous, or germane load, which provided specific directions for ongoing quality improvement. These results demonstrate that a cognitive load theory approach to workshop design and evaluation is feasible and useful in the context of medical education.

Person-Oriented Approaches to Profiling Learners in Technology-Rich Learning Environments for Ecological Learner Modeling

Technology-rich learning environments (TREs) provide opportunities for learners to engage in complex interactions involving a multitude of cognitive, metacognitive, and affective states. Understanding learners’ distinct learning progressions in TREs demand inquiry approaches that employ well-conceived theoretical accounts of these multiple facets. The present study investigated learners’ interactions with BioWorld, a TRE developed to guide students’ clinical reasoning through diagnoses of simulated patients. We applied person-oriented analytic methods to multimodal data including verbal protocols, questionnaires, and computer logs from 78 task solutions. Latent class analysis, clustering methods, and latent profile analysis followed by logistic regression analyses revealed that students’ clinical diagnosis ability was positively correlated with advanced self-regulated learning behaviors, high confidence and cognitive strategy use, critical attention to experts’ feedback, and their positive emotional responses to feedback. The study results have the potential to contribute to a theory-guided approach to designing TREs with a data-driven assessment of multidimensional growth. Building on the study results, we introduce and discuss an ecological learner model for assessing multidimensional learner traits which can be used to design a TRE for adaptive scaffolding.

Examining the Relationship Between Performance Feedback and Emotions in Diagnostic Reasoning: Toward a Predictive Framework for Emotional Support

The purpose of this research is to understand achievement emotions resulting from performance feedback in a medical education context where 30 first and second year medical students learned to diagnose virtual patients in an intelligent tutoring system (ITS), BioWorld. We found that students could be organized into groups using cluster analyses based on the emotions they reported after receiving performance feedback: a positive emotion cluster, negative emotion cluster, and low overall emotion cluster. Medical students in the positive achievement emotion cluster had the highest performance on the diagnostic reasoning cases; those in the negative achievement emotion cluster had the lowest performance; and students categorized as belonging to the low overall achievement emotion cluster had mean performance levels that fell between the two. From the results we propose critical performance thresholds that can be used to predict emotions following performance feedback.