Jocelyn Lockyer

A meta‐analysis of continuing medical education effectiveness

Introduction: We undertook a meta‐analysis of the Continuing Medical Education (CME) outcome literature to examine the effect of moderator variables on physician knowledge, performance, and patient outcomes. Methods: A literature search of MEDLINE and ERIC was conducted for randomized controlled trials and experimental design studies of CME outcomes in which physicians were a major group. CME moderator variables included the types of intervention, the types and number of participants, time, and the number of intervention sessions held over time. Results: Thirty‐one studies met the eligibility criteria, generating 61 interventions. The overall sample‐size weighted effect size for all 61 interventions was r = 0.28 (0.18). The analysis of CME moderator variables showed that active and mixed methods had medium effect sizes (r = 0.33 [0.33], r = 0.33 [0.26], respectively), and passive methods had a small effect size (r = 0.20 [0.16], confidence interval 0.15, 0.26). There was a positive correlation between the effect size and the length of the interventions (r = 0.33) and between multiple interventions over time (r = 0.36). There was a negative correlation between the effect size and programs that involved multiple disciplines (r = −0.18) and the number of participants (r = −0.13). The correlation between the effect size and the length of time for outcome assessment was negative (r = −0.31). Discussion: The meta‐analysis suggests that the effect size of CME on physician knowledge is a medium one; however, the effect size is small for physician performance and patient outcome. The examination of moderator variables shows there is a larger effect size when the interventions are interactive, use multiple methods, and are designed for a small group of physicians from a single discipline.

Factors influencing responsiveness to feedback: on the interplay between fear, confidence, and reasoning processes

Self-appraisal has repeatedly been shown to be inadequate as a mechanism for performance improvement. This has placed greater emphasis on understanding the processes through which self-perception and external feedback interact to influence professional development. As feedback is inevitably interpreted through the lens of one’s self-perceptions it is important to understand how learners interpret, accept, and use feedback (or not) and the factors that influence those interpretations. 134 participants from 8 health professional training/continuing competence programs were recruited to participate in focus groups. Analyses were designed to (a) elicit understandings of the processes used by learners and physicians to interpret, accept and use (or not) data to inform their perceptions of their clinical performance, and (b) further understand the factors (internal and external) believed to influence interpretation of feedback. Multiple influences appear to impact upon the interpretation and uptake of feedback. These include confidence, experience, and fear of not appearing knowledgeable. Importantly, however, each could have a paradoxical effect of both increasing and decreasing receptivity. Less prevalent but nonetheless important themes suggested mechanisms through which cognitive reasoning processes might impede growth from formative feedback. Many studies have examined the effectiveness of feedback through variable interventions focused on feedback delivery. This study suggests that it is equally important to consider feedback from the perspective of how it is received. The interplay observed between fear, confidence, and reasoning processes reinforces the notion that there is no simple recipe for the delivery of effective feedback. These factors should be taken into account when trying to understand (a) why self-appraisal can be flawed, (b) why appropriate external feedback is vital (yet can be ineffective), and (c) why we may need to disentangle the goals of performance improvement from the goals of improving self-assessment.

The processes and dimensions of informed self-assessment: a conceptual model.

Purpose To determine how learners and physicians engaged in various structured interventions to inform self-assessment, how they perceived and used self-assessment in clinical learning and practice, and the components and processes comprising informed self-assessment and factors that influence these. Method This was a qualitative study guided by principles of grounded theory. Using purposive sampling, eight programs were selected in Canada, the United States, the United Kingdom, the Netherlands, and Belgium, representing low, medium, and high degrees of structure/rigor in self-assessment activities. In 2008, 17 focus groups were conducted with 134 participants (53 undergraduate learners, 32 postgraduate learners, 49 physicians). Focus-group transcripts were analyzed interactively and iteratively by the research team to identify themes and compare and confirm findings. Results Informed self-assessment appeared as a flexible, dynamic process of accessing, interpreting, and responding to varied external and internal data. It was characterized by multiple tensions arising from complex interactions among competing internal and external data and multiple influencing conditions. The complex process was evident across the continuum of medical education and practice. A conceptual model of informed self-assessment emerged. Conclusions Central challenges to informing self-assessment are the dynamic interrelationships and underlying tensions among the components comprising self-assessment. Realizing this increases understanding of why self-assessment accuracy seems frequently unreliable. Findings suggest the need for attention to the varied influencing conditions and inherent tensions to progress in understanding self-assessment, how it is informed, and its role in self-directed learning and professional self-regulation. Informed self-assessment is a multidimensional, complex construct requiring further research.

Analysis of stress levels among medical students, residents, and graduate students at four Canadian schools of medicine

PURPOSE: To assess stress in medical students, residents, and graduate science students at four Canadian schools of medicine. METHOD: Four schools with different curricula in three different parts of Canada participated in the study: the University of Calgary Faculty of Medicine, the University of Alberta Faculty of Medicine, the Dalhousie University Faculty of Medicine, and the McMaster University Faculty of Health Sciences. All the medical students, residents, and graduate science students at each school were surveyed in 1994-95. The three instruments used were the University of Calgary Stress Questionnaire, the Social Readjustment Rating Scale (SRRS), and the Symptom Checklist-90. Demographic data were compared across all four schools. Analysis of variance was calculated for all test-item scores, utilizing a four (school) by three (program) by two (gender) design, which were all between subject factors. Significant main effects were followed up by using planned comparisons (Newman-Keuls, with a probability level of p < .05). Significant interaction effects were followed up by using an analysis of simple effects. RESULTS: A total of 1,681 questionnaires were returned as follows: 621 of 1,304 (48%) from the medical students, 645 of 1,495 (43%) from the residents, and 415 of 829 (50%) from the graduate science students. There were significant differences between the three groups in the natures and degrees of stress, with the graduate students reporting higher levels of stress. There were significant gender differences as well, with the women reporting higher levels of stress. Overall, stress levels were found to be mild, based on the University of Calgary Stress Questionnaire and the SRRS. CONCLUSION: This study suggests that medical students and residents experience stress at levels that appear acceptable, but ongoing monitoring and the provision of appropriate support systems will continue to be important.

Helping Babies Breathe: global neonatal resuscitation program development and formative educational evaluation.

OBJECTIVES To develop an educational program designed to train health care providers in resource limited settings to carry out neonatal resuscitation. We analyzed facilitator and learner perceptions about the course, examined skill performance, and assessed the quality of instruments used for learner evaluation as part of the formative evaluation of the educational program Helping Babies Breathe. METHODS Multiple stakeholders and a Delphi panel contributed to program development. Training of facilitators and learners occurred in global field test sites. Course evaluations and focus groups provided data on facilitator and learner perceptions. Knowledge and skill assessments included pre/post scores from multiple choice questions (MCQ) and post-training assessment of bag and mask skills, as well as 2 objective structured clinical evaluations (OSCE). RESULTS Two sites (Kenya and Pakistan) trained 31 facilitators and 102 learners. Participants expressed high satisfaction with the program and high self-efficacy with respect to neonatal resuscitation. Assessment of participant knowledge and skills pre/post-program demonstrated significant gains; however, the majority of participants could not demonstrate mastery of bag and mask ventilation on the post-training assessment without additional practice. CONCLUSIONS Participants in a program for neonatal resuscitation in resource-limited settings demonstrated high satisfaction, high self-efficacy and gains in knowledge and skills. Mastery of ventilation skills and integration of skills into case management may not be achievable in the classroom setting without additional practice, continued learning, and active mentoring in the workplace. These findings were used to revise program structure, materials and assessment tools.

Stillbirth and Newborn Mortality in India After Helping Babies Breathe Training

OBJECTIVE: This study evaluated the effectiveness of Helping Babies Breathe (HBB) newborn care and resuscitation training for birth attendants in reducing stillbirth (SB), and predischarge and neonatal mortality (NMR). India contributes to a large proportion of the worlds annual 3.1 million neonatal deaths and 2.6 million SBs. METHODS: This prospective study included 4187 births at >28 weeks’ gestation before and 5411 births after HBB training in Karnataka. A total of 599 birth attendants from rural primary health centers and district and urban hospitals received HBB training developed by the American Academy of Pediatrics, using a train-the-trainer cascade. Pre-post written trainee knowledge, posttraining provider performance and skills, SB, predischarge mortality, and NMR before and after HBB training were assessed by using χ2 and t-tests for categorical and continuous variables, respectively. Backward stepwise logistic regression analysis adjusted for potential confounding. RESULTS: Provider knowledge and performance systematically improved with HBB training. HBB training reduced resuscitation but increased assisted bag and mask ventilation incidence. SB declined from 3.0% to 2.3% (odds ratio [OR] 0.76, 95% confidence interval [CI] 0.59–0.98) and fresh SB from 1.7% to 0.9% (OR 0.54, 95% CI 0.37–0.78) after HBB training. Predischarge mortality was 0.1% in both periods. NMR was 1.8% before and 1.9% after HBB training (OR 1.09, 95% CI 0.80–1.47, P = .59) but unknown status at 28 days was 2% greater after HBB training (P = .007). CONCLUSIONS: HBB training reduced SB without increasing NMR, indicating that resuscitated infants survived the neonatal period. Monitoring and community-based assessment are recommended.

Changing physicians' practices: the effect of individual feedback.

OBJECTIVE To determine whether physicians who received feedback from six peers, six referring/referral physicians, six co-workers, and 25 patients about 55 aspects of their medical practices (e.g., able to reach doctor by phone after office hours) would make changes to their practices based on that feedback. METHOD In an earlier study, 308 physicians were given feedback about 106 aspects of their practices in the form of mean Likert-scale ratings that (1) the peers made on 26 aspects; (2) the referring/referral physicians made on 23 aspects; (3) the co-workers made on 17 aspects; and (4) the patients made on 40 aspects. Three months later 255 of these physicians responded when asked to indicate whether they had contemplated or initiated changes, or whether no change had been necessary, regarding 31 practice aspects, each of which was a summary of one or more of 55 of the original 106 aspects on which they had received ratings. These 55 were considered the aspects most amenable to change over a short period. The physicians were also asked about the educational interventions that they felt would help them make changes. Multivariate analysis of variance was used to see whether the types of changes reported for the specific aspects of practice were associated with the feedback ratings received for those aspects. RESULTS An examination of the responses showed that 83% of the 255 physicians reported having contemplated a change, and 66% reported having initiated a change for at least one aspect of practice. Changes were contemplated most frequently for aspects of practice associated with clinical skills and resource use. Changes were initiated most frequently for aspects of practice associated with communication with patients and support of patients. Physicians who contemplated or initiated changes had lower (i.e., more negative) mean ratings than did physicians who reported that no change was necessary, which suggests that the physicians did use their feedback ratings to decide about changes, although their qualitative comments indicated other sources as well. Printed material was chosen most often as a method of receiving continuing medical education related to making changes in the practice areas examined.

Assessment of non-cognitive traits through the admissions multiple mini-interview.

Context  Contemporary studies have shown that traditional medical school admissions interviews have strong face validity but provide evidence for only low reliability and validity. As a result, they do not provide a standardised, defensible and fair process for all applicants.

Making interprofessional education work: the strategic roles of the academy.

Faculties (i.e., schools) of medicine along with their sister health discipline faculties can be important organizational vehicles to promote, cultivate, and direct interprofessional education (IPE). The authors present information they gathered in 2007 about five Canadian IPE programs to identify key factors facilitating transformational change within institutional settings toward successful IPE, including (1) how successful programs start, (2) the ways successful programs influence academia to bias toward change, and (3) the ways academia supports and perpetuates the success of programs. Initially, they examine evidence regarding key factors that facilitate IPE implementation, which include (1) common vision, values, and goal sharing, (2) opportunities for collaborative work in practice and learning, (3) professional development of faculty members, (4) individuals who are champions of IPE in practice and in organizational leadership, and (5) attention to sustainability. Subsequently, they review literature-based insights regarding barriers and challenges in IPE that must be addressed for success, including barriers and challenges (1) between professional practices, (2) between academia and the professions, and (3) between individuals and faculty members; they also discuss the social context of the participants and institutions. The authors conclude by recommending what is needed for institutions to entrench IPE into core education at three levels: micro (what individuals in the faculty can do); meso (what a faculty can promote); and macro (how academic institutions can exert its influence in the health education and practice system).

Tensions in informed self-assessment: how the desire for feedback and reticence to collect and use it can conflict.

Purpose Informed self-assessment describes the set of processes through which individuals use external and internal data to generate an appraisal of their own abilities. The purpose of this project was to explore the tensions described by learners and professionals when informing their self-assessments of clinical performance. Method This 2008 qualitative study was guided by principles of grounded theory. Eight programs in five countries across undergraduate, postgraduate, and continuing medical education were purposively sampled. Seventeen focus groups were held (134 participants). Detailed analyses were conducted iteratively to understand themes and relationships. Results Participants experienced multiple tensions in informed self-assessment. Three categories of tensions emerged: within people (e.g., wanting feedback, yet fearing disconfirming feedback), between people (e.g., providing genuine feedback yet wanting to preserve relationships), and in the learning/practice environment (e.g., engaging in authentic self-assessment activities versus “playing the evaluation game”). Tensions were ongoing, contextual, and dynamic; they prevailed across participant groups, infusing all components of informed self-assessment. They also were present in varied contexts and at all levels of learners and practicing physicians. Conclusions Multiple tensions, requiring ongoing negotiation and renegotiation, are inherent in informed self-assessment. Tensions are both intraindividual and interindividual and they are culturally situated, reflecting both professional and institutional influences. Social learning theories (social cognitive theory) and sociocultural theories of learning (situated learning and communities of practice) may inform our understanding and interpretation of the study findings. The findings suggest that educational interventions should be directed at individual, collective, and institutional cultural levels. Implications for practice are presented.