Iain Colthart

The effectiveness of self-assessment on the identification of learner needs, learner activity, and impact on clinical practice: BEME Guide no. 10

Review date: Literature search January 1990 to February 2005 (with update February 2006). Analysis completed January 2007. Background: Health professionals are increasingly expected to identify their own learning needs through a process of ongoing self-assessment. Self-assessment is integral to many appraisal systems and has been espoused as an important aspect of personal professional behaviour by several regulatory bodies and those developing learning outcomes for clinical students. In this review we considered the evidence base on self-assessment since Gordons comprehensive review in 1991. The overall aim of the present review was to determine whether specific methods of self-assessment lead to change in learning behaviour or clinical practice. Specific objectives sought evidence for effectiveness of self-assessment interventions to: a. improve perception of learning needs; b. promote change in learning activity; c. improve clinical practice; d. improve patient outcomes. Methods: The methods for this review were developed and refined in a series of workshops with input from an expert BEME systematic reviewer, and followed BEME guidance. Databases searched included Medline, CINAHL, BNI, Embase, EBM Collection, Psychlit, HMIC, ERIC, BEI, TIMElit and RDRB. Papers addressing self-assessment in all professions in clinical practice were included, covering under- and post-graduate education, with outcomes classified using an extended version of Kirkpatricks hierarchy. In addition we included outcome measures of accuracy of self-assessment and factors influencing it. 5,798 papers were retrieved, 194 abstracts were identified as potentially relevant and 103 papers coded independently by pairs using an electronic coding sheet adapted from the standard BEME form. This total included 12 papers identified by hand-searches, grey literature, cited references and updating. The identification of a further 12 papers during the writing-up process resulted in a total of 77 papers for final analysis. Results: Although a large number of papers resulted from our original search only a small proportion of these were of sufficient academic rigour to be included in our review. The majority of these focused on judging the accuracy of self-assessment against some external standard, which raises questions about assumed reliability and validity of this ‘gold standard’. No papers were found which satisfied Kirkpatricks hierarchy above level 2, or which looked at the association between self-assessment and resulting changes in either clinical practice or patient outcomes. Thus our review was largely unable to answer the specific research questions and provide a solid evidence base for effective self-assessment. Despite this, there was some evidence that the accuracy of self-assessment can be enhanced by feedback, particularly video and verbal, and by providing explicit assessment criteria and benchmarking guidance. There was also some evidence that the least competent are also the least able to self-assess accurately. Our review recommends that these areas merit future systematic research to further our understanding of self-assessment. Conclusion: As in other BEME reviews, the methodological issues emerging from this review indicate a need for more rigorous study designs. In addition, it highlights the need to consider the potential for combining qualitative and quantitative data to further our understanding of how self-assessment can improve learning and professional clinical practice. Practice points•There is no solid evidence base within the health professions’ literature which establishes the effectiveness of self-assessment in: identifying learner needs; influencing learning activity; changing clinical practice.•The accuracy of self-assessment in clinical training may be improved by increasing the learners awareness of the standard to be achieved.•There is some indication that practical skills in clinical training may be better self-assessed than knowledge-based activities.•Self-assessment needs to be used as one tool amongst other sources of feedback to provide a more complete appraisal of competence in health care practice.•Future research should address the role that self-assessment plays in the everyday practice of health care decision-making.

The effectiveness of portfolios for post-graduate assessment and education: BEME Guide No 12

Background: Portfolios in post-graduate healthcare education are used to support reflective practice, deliver summative assessment, aid knowledge management processes and are seen as a key connection between learning at organisational and individual levels. This systematic review draws together the evidence on the effectiveness of portfolios across postgraduate healthcare and examines the implications of portfolios migrating from paper to an electronic medium across all professional settings. Methods: A literature search was conducted for articles describing the use of a portfolio for learning in a work or professional study environment. It was designed for high sensitivity and conducted across a wide range of published and unpublished sources relevant to professional education. No limits for study design or outcomes, country of origin or language were set. Blinded, paired quality rating was carried out, and detailed appraisal of and data extraction from included articles was managed using an online tool developed specifically for the review. Findings were discussed in-depth by the team, to identify and group pertinent themes when answering the research questions. Results: Fifty six articles from 10 countries involving seven healthcare professions met our inclusion criteria and minimum quality threshold; mostly uncontrolled observational studies. Portfolios encouraged reflection in some groups, and facilitated engagement with learning. There was limited evidence of the influence of a number of factors on portfolio use, including ongoing support from mentors or peers, implementation method, user attitude and level of initial training. Confounding variables underlying these issues, however have not been fully investigated. A number of authors explored the reliability and validity of portfolios for summative assessment but reports of accuracy across the disparate evidence base varied. Links to competency and Quality Assurance frameworks have been demonstrated. There were conflicting reports about whether the different purposes of portfolios can be combined without compromising the meaningfulness of the contents. There was good evidence that the flexibility of the electronic format brought additional benefits to users, assessors and organisations, and encouraged more enthusiastic use. Security of data remained a high priority issue at all levels, and there was emerging evidence of successful transfer between electronic portfolio systems. Conclusion: The evidence base is extensive, but contains few high quality studies with generalisable messages about the effectiveness of portfolios. There is, however, good evidence that if well implemented, portfolios are effective and practical in a number of ways including increasing personal responsibility for learning and supporting professional development. Electronic versions are better at encouraging reflection and users voluntarily spend longer on them. Regular feedback from a mentor enhances this success, despite competing demands on users’ time and occasional scepticism about the purpose of a portfolio. Reports of inter-rater reliability for summative assessments of portfolio data are varied and there is benefit to be gained from triangulating with other assessment methods. There was insufficient evidence to draw conclusions on how portfolios work in interdisciplinary settings.

The feminization of the medical work force, implications for Scottish primary care: a survey of Scottish general practitioners

BackgroundThe number of women working in general practice internationally has been steadily rising. In Scotland there have been concerns that such a change may lead to increased part-time working and subsequently to a fall in available general practice manpower despite an apparently rising overall number of general practitioners. However, there is very little information on the actual hours worked by men and women general practitioners or on the types of work they are undertaking.MethodsAnonymous workload questionnaires of all Scottish general practitioner principals and non-principalsResultsResponse rates for general practice principals and non-principals were 67.2% and 65.2% respectively. Male principals spent on average 18% more time on general medical services (GMS) and 50% more time on non-GMS activities (such as teaching, specialist sessions, administration and research) than women (both p <0.01). This difference was similar for non-principals. In no age group did the hours worked by women doctors approach that of male doctors.ConclusionWomen doctors in primary care in Scotland work fewer hours in all age groups than their male counterparts. The rapidly increasing proportion of women in general practice may lead to an increasing shortfall of medical availability in the future if current work patterns are maintained. Further longitudinal research is required to establish this and man-power planning is required now to address this. More worryingly auxiliary activities such as teaching and administrative duties are not being taken up by women. This may have serious implications for the future development of the specialty in Scotland.

What do doctors really think about the relevance and impact of GP appraisal 3 years on? A survey of Scottish GPs

BACKGROUND The aim of appraisal is to provide an opportunity for individuals to reflect on their work to facilitate learning and development. Appraisal for GPs has been a contractual requirement since 2004 in Scotland, and is seen as an integral part of revalidation. AIM To investigate the outcomes of GP appraisal in terms of whether it has prompted change in medical practice, education and learning, career development, attitudes to health and probity, how GPs organise their work, and their perception of the overall value of the process. DESIGN OF STUDY A cross-sectional postal questionnaire. SETTING GP performers in Scotland who had undertaken appraisal. METHOD The questionnaire was based on the seven principles outlined in Good Medical Practice, a literature review, and previous local research. The survey was conducted on a strictly anonymous basis with a random, representative sample of GPs. RESULTS Fifty-three per cent (671/1278) responded. Forty-seven per cent (308/661) thought that appraisal had altered their educational activity, 33% (217/660) reported undertaking further education or training as a result of appraisal, and 13% (89/660) felt that appraisal had influenced their career development. Opinion was evenly split on the overall value of appraisal. CONCLUSION Appraisal can have a significant impact on all aspects of a GPs professional life, and those who value the process report continuing benefit in how they manage their education and professional development. However, many perceive limited or no benefit. The renewed emphasis on appraisal requires examination of these findings and discussion of how appraisal can become more relevant.

Asking the right questions and getting meaningful responses: 12 tips on developing and administering a questionnaire survey for healthcare professionals

Questionnaires provide a useful and versatile tool for new and occasional researchers, and can be applied to a wide range of topics. This paper provides simple guidance on some of the potential pitfalls in developing and running a questionnaire study, and how to avoid them. Each tip is illustrated with a real-life example from the development of a UK-wide questionnaire survey of trainee doctors and their educational supervisors.

Understanding supervision in health and social care through the experiences of practitioners in Scotland

Purpose The purpose of this paper is to describe a collaborative study on supervision with health and social care practitioners in Scotland. The study attempted to gain a better understanding about the use and benefit of supervision from a multiprofessional perspective. Design/methodology/approach Consultation events with health and social care staff and 12 informant interviews were undertaken. Data analysis was via the Framework Method. Findings Managers were more likely to conceive of supervision as a positive intervention than those in lower pay bands. The practice of supervision was variable. Not all staff appeared to take part in regular supervisory activities even when it was mandated. A lack of professional, organisational or local commitment to implement robust supervisory structures and processes was seen as the major barrier to effective supervision. Research limitations/implications This was a small study, thus findings would need to be confirmed by health and social care staff working across a wider spectrum of disciplines and regions across Scotland. Practical implications A combination of factors would seem to determine effective supervisory practice. Supervision was perceived to be of benefit when individuals were willing to participate fully, when there was reflection and planned action, constructive challenge, respectful relationships, regular and protected sessions and processes were appropriate to an employee’s circumstances. Originality/value This study situates supervision in the current context of health and social care and finds it to be an irregular practice. The findings confirm the existing literature about the importance of supervisor-supervisee relationships but explain differing perceptions of supervision in terms of staff seniority.