David Wall

Measuring effectiveness for best evidence medical education: a discussion.

This paper considers how educational interventions should be evaluated for their effectiveness. Five levels of effectiveness are clarified and illustrated: outcomes, behaviour, learning, reaction and participation. These levels are then discussed, within the context of research evidence for education within the medical profession. Methodological and practical research conclusions are then drawn. From an analysis of over 300 abstracts the evidence shows that only limited research on healthcare outcomes has been undertaken, and there is considerable ambiguity over what constitutes an effectiveness measure. The benefits of a consistent set of terms are then considered. Finally, five key messages are distilled from the discussion and summarized.

Teaching the consultant teachers: identifying the core content.

To determine the key themes for teaching hospital consultants how to teach.

A literature review of multi-source feedback systems within and without health services, leading to 10 tips for their successful design

Multi-source feedback (MSF) has become the accepted mechanism of ensuring the appropriate professional behaviour of doctors. It is part of the mandatory assessment of doctors in training and is to be utilized as part of the revalidation of trained doctors. There is significant variation in the models of MSF currently used within the National Health Service and new models of MSF are being designed by various specialties. No single model has been recognized as the ‘gold standard’. However, there is a large published literature concerning MSF, both in the context of health systems and, more extensively, within industry. This published literature is reviewed, drawing attention to aspects of MSF systems in which there is consensus on effective approaches as well as other aspects in which there is doubt about the optimum approach. In the light of the review 10 principles key in the development of effective MSF models have been produced. Practice points•Multi-source feedback has been used extensively in industry and the health sector.•Multi-source feedback is a valid way to measure professional behaviours in doctors.•Multi-source feedback can enhance team-working, productivity, good communication and trust.•Any desired behaviours you wish to assess should be described very clearly.•The assessment tool should be kept simple, with few items, and fit for purpose.

Trainee doctors’ views on workplace-based assessments: Are they just a tick box exercise?

Background: In August 2007, run-through specialty training and workplace-based assessments (WBAs) were introduced for UK junior doctors. Aim: This study explores whether WBAs have been successfully integrated into paediatric training programmes. Method: A 20-item questionnaire was distributed to regional paediatric trainees. Results: Eighty seven per cent (130/150) trainees responded. Reliability of the study was good (Cronbachs alpha 0.813); 92% (119/130) trainees knew how to conduct WBAs (scoring ≥4 on the six-point Likert scale). However, they were ambivalent about WBAs being a true reflection of their capabilities (mean Likert score 3.46). A common problem was finding assessors with 40% of trainees stating that staff had refused to do assessments. Almost half of the assessments were done retrospectively. Trainees did receive feedback but advice on future improvement was not always given (highest mean Likert score 4.00). Free text comments analysis mostly centred on difficulties in getting WBAs done. Conclusions: A cultural change is needed for trainees to feel that WBAs are not just a tick box exercise, but a useful educational tool for learning. Ongoing work on implementation needs to include additional training, especially on the value of WBAs for formative assessment and consultants having protected time in their job plans for training.

Diaper Dermatitis-Frequency and Contributory Factors in Hospital Attending Children

Abstract:  The incidence and prevalence of diaper dermatitis varies widely between published studies. It is a condition which causes considerable parental anxiety. To better understand the frequency of diaper dermatitis, treatment practices, and the current importance of previously identified etiologic factors, a questionnaire survey of parents who had children wearing diapers (n = 532) attending a large United Kingdom district general hospital was undertaken. At the time of survey, only 16% of the study population had diaper dermatitis. Forty‐eight percent of the study population had never had an episode of diaper dermatitis. In a multivariate analysis, current diaper dermatitis was independently associated with four factors: presence of oral thrush, number of previous episodes, frequency of diaper changes, and diarrhea. Recurrent episodes of diaper dermatitis were associated with increasing age, lack of barrier cream use, current diaper rash, and frequency of diaper changes. In the majority of children with diaper dermatitis at the time of survey, treatment had been instituted in the community. Diaper dermatitis usually presents and is treated successfully outside the hospital setting and is not a common clinical problem in secondary care.

Multiple mini-interviews: opinions of candidates and interviewers.

Objectives  To assess candidates’ and interviewers’ perceptions of the use of a multiple mini‐interview (MMI) for selection of senior house officers (SHOs) to a UK regional paediatric training programme.

From undergraduate medical education to pre-registration house officer year: how prepared are students?

Studies in the United Kingdom show that the first year of being a new doctor is a difficult transition from being an undergraduate medical. These doctors feel ill-prepared by their undergraduate medical student education and ill-equipped for their new role. This is a questionnaire study looking at 193 pre-registration house officers (PRHOs) and 212 consultant educational supervisors within the West Midlands Deanery in England, UK. The study asked how well prepared these new doctors were in seventeen basic subject areas (using a General Medical Council based template), and also, using the same template, how well prepared their consultant educational supervisors felt they were. Statistical testing included ranking of subject areas for each group, reliability testing and comparisons of views of house officers and consultants. Both groups ranked communication skills areas highest (best prepared) and ranked basic doctoring skills (such as prescribing, treatment, decision making and emergencies) lowest. House officers rated themselves significantly higher than did their consultant supervisors in thirteen out of the seventeen areas tested. In discussion, we ask if we have gone too far in teaching effective communication at the expense of basic doctoring skills. We also consider whether indeed young doctors have an inflated opinion of their own competencies, or are the older generation seeing the world through rose tinted spectacles?

Curriculum for the foundation years in postgraduate education and training

*The new amended curriculum has now been published in April 2005. Some improvements have taken place, including the loss of the nine point scales, the new emphasis on competencies achieved, the spiral curriculum has been corrected, and the West Midlands TAB (360 degree assessment) has now been included. However, it is still a big document of 96 pages, without references, and further work will still be needed on this important area of medical education.

How ratings vary by staff group in multi-source feedback assessment of junior doctors.

Context  UK doctors‐in‐training undergo assessments of their professional behaviours. From an analysis of multi‐source feedback (MSF) data, we report how ratings of junior doctors (Foundation Programme [FP] doctors and senior house officers [SHOs]) differed by staff group.

360 degree assessment (multisource feedback) of UK trainee doctors: Field testing of team assessment of behaviours (TAB)

Background: This study was to see if the team assessment of behaviours (TAB) 360 degree assessment tool was able to identify interpersonal behaviour problems in doctors in training, to see if feedback was useful, to gauge the value of the process by those involved, and to learn lessons about implementing the process for the future. Methods: TAB was administered to assess interpersonal behaviours of senior house officers in four hospitals in the West Midlands, UK. In addition, questionnaires were sent to all participants, some were interviewed about the whole process, and records kept of the time involved. Results: One hundred and seventyone SHO volunteers received 1378 assessments. The median number of ratings per SHO was 8 (mode 9). Sixtyfour percent of SHOs received ‘no concern’ ratings in all four behaviours (domains) assessed. Twentyone percent received one ‘some concern’ rating. Fifteen percent received more than one ‘concern’ rating. Conclusion: Assessors and trainees found the process practical, valuable and fair. Educational supervisors found it valuable, although only 23% learned something new about their trainees. Clinical tutors valued the system. Administrative staff found it time consuming. The TAB four-domain rating form with its single pass category identified specific concern about volunteer trainees’ professional behaviour. Not all trainees received skilled feedback.