Jan Illing

The effectiveness of self-assessment on the identification of learner needs, learner activity, and impact on clinical practice: BEME Guide no. 10

Review date: Literature search January 1990 to February 2005 (with update February 2006). Analysis completed January 2007. Background: Health professionals are increasingly expected to identify their own learning needs through a process of ongoing self-assessment. Self-assessment is integral to many appraisal systems and has been espoused as an important aspect of personal professional behaviour by several regulatory bodies and those developing learning outcomes for clinical students. In this review we considered the evidence base on self-assessment since Gordons comprehensive review in 1991. The overall aim of the present review was to determine whether specific methods of self-assessment lead to change in learning behaviour or clinical practice. Specific objectives sought evidence for effectiveness of self-assessment interventions to: a. improve perception of learning needs; b. promote change in learning activity; c. improve clinical practice; d. improve patient outcomes. Methods: The methods for this review were developed and refined in a series of workshops with input from an expert BEME systematic reviewer, and followed BEME guidance. Databases searched included Medline, CINAHL, BNI, Embase, EBM Collection, Psychlit, HMIC, ERIC, BEI, TIMElit and RDRB. Papers addressing self-assessment in all professions in clinical practice were included, covering under- and post-graduate education, with outcomes classified using an extended version of Kirkpatricks hierarchy. In addition we included outcome measures of accuracy of self-assessment and factors influencing it. 5,798 papers were retrieved, 194 abstracts were identified as potentially relevant and 103 papers coded independently by pairs using an electronic coding sheet adapted from the standard BEME form. This total included 12 papers identified by hand-searches, grey literature, cited references and updating. The identification of a further 12 papers during the writing-up process resulted in a total of 77 papers for final analysis. Results: Although a large number of papers resulted from our original search only a small proportion of these were of sufficient academic rigour to be included in our review. The majority of these focused on judging the accuracy of self-assessment against some external standard, which raises questions about assumed reliability and validity of this ‘gold standard’. No papers were found which satisfied Kirkpatricks hierarchy above level 2, or which looked at the association between self-assessment and resulting changes in either clinical practice or patient outcomes. Thus our review was largely unable to answer the specific research questions and provide a solid evidence base for effective self-assessment. Despite this, there was some evidence that the accuracy of self-assessment can be enhanced by feedback, particularly video and verbal, and by providing explicit assessment criteria and benchmarking guidance. There was also some evidence that the least competent are also the least able to self-assess accurately. Our review recommends that these areas merit future systematic research to further our understanding of self-assessment. Conclusion: As in other BEME reviews, the methodological issues emerging from this review indicate a need for more rigorous study designs. In addition, it highlights the need to consider the potential for combining qualitative and quantitative data to further our understanding of how self-assessment can improve learning and professional clinical practice. Practice points•There is no solid evidence base within the health professions’ literature which establishes the effectiveness of self-assessment in: identifying learner needs; influencing learning activity; changing clinical practice.•The accuracy of self-assessment in clinical training may be improved by increasing the learners awareness of the standard to be achieved.•There is some indication that practical skills in clinical training may be better self-assessed than knowledge-based activities.•Self-assessment needs to be used as one tool amongst other sources of feedback to provide a more complete appraisal of competence in health care practice.•Future research should address the role that self-assessment plays in the everyday practice of health care decision-making.

Workplace bullying in the UK NHS: a questionnaire and interview study on prevalence, impact and barriers to reporting

Objectives To examine the prevalence and impact of bullying behaviours between staff in the National Health Service (NHS) workplace, and to explore the barriers to reporting bullying. Design Cross-sectional questionnaire and semi-structured interview. Setting 7 NHS trusts in the North East of England. Participants 2950 NHS staff, of whom 43 took part in a telephone interview. Main outcome measures Prevalence of bullying was measured by the revised Negative Acts Questionnaire (NAQ-R) and the impact of bullying was measured using indicators of psychological distress (General Health Questionnaire, GHQ-12), intentions to leave work, job satisfaction and self-reported sickness absence. Barriers to reporting bullying and sources of bullying were also examined. Results Overall, 20% of staff reported having been bullied by other staff to some degree and 43% reported having witnessed bullying in the last 6 months. Male staff and staff with disabilities reported higher levels of bullying. There were no overall differences due to ethnicity, but some differences were detected on several negative behaviours. Bullying and witnessing bullying were associated with lower levels of psychological health and job satisfaction, and higher levels of intention to leave work. Managers were the most common source of bullying. Main barriers to reporting bullying were the perception that nothing would change, not wanting to be seen as a trouble-maker, the seniority of the bully and uncertainty over how policies would be implemented and bullying cases managed. Data from qualitative interviews supported these findings and identified workload pressures and organisational culture as factors contributing to workplace bullying. Conclusions Bullying is a persistent problem in healthcare organisations which has significant negative outcomes for individuals and organisations.

Preparedness for practice : the perceptions of medical graduates and clinical teams

Background: Earlier research indicated that medical graduates feel unprepared to start work, and that this varies with medical school. Aims: To examine the extent to which graduates from different UK medical schools differed in their perceptions of preparedness for practice, and compare their perceptions with those of clinical team members. Method: An anonymous questionnaire assessing perceptions of 53 aspects of preparedness was devised, and administered to the graduating cohorts of three medical schools: Newcastle (systems-based, integrated curriculum); Warwick (graduate-entry) and Glasgow (problem-based learning). In addition, a triangulating questionnaire was cascaded via ward managers to doctors, nurses and pharmacists who worked with new graduates in their first posts. Results: The response rate for the cohort questionnaire was 69% (479/698). The overall mean preparedness score was 3.5 (on a five-point scale), with no significant difference between schools. On individual items, there were large differences within each site, but smaller differences between sites. Graduates felt most prepared for aspects of working with patients and colleagues, history taking and examination. They felt least prepared for completing a cremation form, some aspects of prescribing, complex practical procedures and for applying knowledge of alternative and complementary therapies, and of the NHS. A total of 80 clinical team questionnaires were completed, similarly showing substantial variation within each site, but smaller differences between sites. Conclusions: New doctors feel relatively unprepared for a number of aspects of practice, a perception shared by their colleagues. Although medical school has some effect on preparedness, greater differences are common across sites. Differences may reflect hidden influences common to all the schools, unintended consequences of national curriculum guidance or common traits in the graduate populations sampled. Further research is needed to identify the causes.

User perceptions of multi-source feedback tools for junior doctors

Medical Education 2010: 44 : 165–176

Annual Review of Competence Progression (ARCP) performance of doctors who passed Professional and Linguistic Assessments Board (PLAB) tests compared with UK medical graduates: national data linkage study.

Objective To determine whether use of the Professional and Linguistic Assessments Board (PLAB) examination system used to grant registration for international medical graduates results in equivalent postgraduate medical performance, as evaluated at Annual Review of Competence Progression (ARCP), between UK based doctors who qualified overseas and those who obtained their primary medical qualification from UK universities. Design Observational study linking ARCP outcome data from the UK deaneries with PLAB test performance and demographic data held by the UK General Medical Council (GMC). Setting Doctors in postgraduate training for a medical specialty or general practice in the UK and doctors obtaining GMC registration via the PLAB system. Participants 53 436 UK based trainee doctors with at least one competency related ARCP outcome reported during the study period, of whom 42 017 were UK medical graduates and 11 419 were international medical graduates who were registered following a pass from the PLAB route. Main outcome measure Probability of obtaining a poorer versus a more satisfactory category of outcome at ARCP following successful registration as a doctor in the UK. Results International medical graduates were more likely to obtain a less satisfactory outcome at ARCP compared with UK graduates. This finding persisted even after adjustment for the potential influence of sex, age, years of UK based practice, and ethnicity and exclusion of outcomes associated with postgraduate examination failure (odds ratio 1.63, 95% confidence interval 1.30 to 2.06). However, international medical graduates who scored in the highest twelfth at part 1 of the PLAB (at least 32 points above the pass mark) had ARCP outcomes that did not differ significantly from those of UK graduates. Conclusions These findings suggest that the PLAB test used for registration of international medical graduates is not generally equivalent to the requirements for UK graduates. The differences in postgraduate performance, as captured at ARCP, following the two routes to registration might be levelled out by raising the standards of English language competency required as well as the pass marks for the two parts of the PLAB test. An alternative might be to introduce a different testing system.

Newly qualified doctors’ perceptions of informal learning from nurses: implications for interprofessional education and practice

Abstract Newly qualified doctors spend much of their time with nurses, but little research has considered informal learning during that formative contact. This article reports findings from a multiple case study that explored what newly qualified doctors felt they learned from nurses in the workplace. Analysis of interviews conducted with UK doctors in their first year of practice identified four overarching themes: attitudes towards working with nurses, learning about roles, professional hierarchies and learning skills. Informal learning was found to contribute to the newly qualified doctors’ knowledge of their own and others’ roles. A dynamic hierarchy was identified: one in which a “pragmatic hierarchy” recognising nurses’ expertise was superseded by a “normative structural hierarchy” that reinforced the notion of medical dominance. Alongside the implicit learning of roles, nurses contributed to the explicit learning of skills and captured doctors’ errors, with implications for patient safety. The findings are discussed in relation to professional socialisation. Issues of power between the professions are also considered. It is concluded that increasing both medical and nursing professions’ awareness of informal workplace learning may improve the efficiency of education in restricted working hours. A culture in which informal learning is embedded may also have benefits for patient safety.

A systematic literature review of undergraduate clinical placements in underserved areas

The delivery of undergraduate clinical education in underserved areas is increasing in various contexts across the world in response to local workforce needs. A collective understanding of the impact of these placements is lacking. Previous reviews have often taken a positivist approach by only looking at outcome measures. This review addresses the question: What are the strengths and weaknesses for medical students and supervisors of community placements in underserved areas?

Are specialist registrars fully prepared for the role of consultant

The step-up from specialist registrar (SpR) to consultant has been acknowledged by doctors as being large. It can involve relatively sudden change, and can be both stressful and demanding.1,2 There is increasing pressure on available time for training, with shortened training programmes and fewer hours spent at work as a result of the European Working Time Directive.2,3 Medical education research has not fully addressed this transition or explored ways of improving it for the benefit of patients and doctors. Newly appointed consultants are more prepared for some aspects of their work than others. The quality of training in clinical skills is rated most positively, although even this has room for improvement.2,4 New consultants feel less well prepared for their management responsibilities than they do for clinical work,2,4,5 including self-management.6 Training and experience in handling complaints, dealing with difficult professional relationships, recruitment and disciplinary proceedings have also been identified as weaker areas of specialty training.5,6 Feeling inadequately trained in communication and management skills can impact on stress, burnout and the mental health of consultants.4,7 It is notable that two-thirds of cases referred to the National Clinical Assessment Service (NCAS) involved behavioural issues (including difficulties with colleagues), either on their own or in conjunction with other concerns.8 In the light of these issues, a research project was developed to determine the extent to which specialty training provides doctors with the skills they require when they become consultants.

The difficulty of professional continuation among female doctors in Japan: a qualitative study of alumnae of 13 medical schools in Japan

Objectives To investigate the difficulties Japanese female doctors face in continuing professional practice. Design A qualitative study using the Kawakita Jiro method. Setting A survey conducted in 2011 of 13 private Japanese medical school alumni associations. Participants 359 female doctors. Primary outcome measures Barriers of balancing work and gender role. Results The female doctors reported that professional practice was a struggle with long working hours due to a current shortage of doctors in Japan. There was also a severe shortage of childcare facilities in the workplace. Some women appeared to have low confidence in balancing the physicians job and personal life, resulting in low levels of professional pursuit. There appeared to be two types of stereotypical gender roles, including one expected from society, stating that “child rearing is a womans job”, and the other perceived by the women themselves, that some women had a very strong desire to raise their own children. Male doctors and some female doctors who were single or older were perceived to be less enthusiastic about supporting women who worked while raising children because these coworkers feared that they would have to perform additional work as a result of the women taking long periods of leave. Conclusions Important factors identified for promoting the continuation of professional practice among female doctors in Japan were the need to improve working conditions, including cutting back on long working hours, a solution to the shortage of nurseries, a need for the introduction of educational interventions to clarify professional responsibilities, and redefinition of the gender division of labour for male and female doctors. In addition, we identified a need to modernise current employment practices by introducing temporary posts to cover maternity leave and introducing flexible working hours during specialist training, thus supporting and encouraging more women to continue their medical careers.

Without proper research funding, how can medical education be evidence based?

We can no longer leave research into medical education to chance