Alex Haig

The effectiveness of self-assessment on the identification of learner needs, learner activity, and impact on clinical practice: BEME Guide no. 10

Review date: Literature search January 1990 to February 2005 (with update February 2006). Analysis completed January 2007. Background: Health professionals are increasingly expected to identify their own learning needs through a process of ongoing self-assessment. Self-assessment is integral to many appraisal systems and has been espoused as an important aspect of personal professional behaviour by several regulatory bodies and those developing learning outcomes for clinical students. In this review we considered the evidence base on self-assessment since Gordons comprehensive review in 1991. The overall aim of the present review was to determine whether specific methods of self-assessment lead to change in learning behaviour or clinical practice. Specific objectives sought evidence for effectiveness of self-assessment interventions to: a. improve perception of learning needs; b. promote change in learning activity; c. improve clinical practice; d. improve patient outcomes. Methods: The methods for this review were developed and refined in a series of workshops with input from an expert BEME systematic reviewer, and followed BEME guidance. Databases searched included Medline, CINAHL, BNI, Embase, EBM Collection, Psychlit, HMIC, ERIC, BEI, TIMElit and RDRB. Papers addressing self-assessment in all professions in clinical practice were included, covering under- and post-graduate education, with outcomes classified using an extended version of Kirkpatricks hierarchy. In addition we included outcome measures of accuracy of self-assessment and factors influencing it. 5,798 papers were retrieved, 194 abstracts were identified as potentially relevant and 103 papers coded independently by pairs using an electronic coding sheet adapted from the standard BEME form. This total included 12 papers identified by hand-searches, grey literature, cited references and updating. The identification of a further 12 papers during the writing-up process resulted in a total of 77 papers for final analysis. Results: Although a large number of papers resulted from our original search only a small proportion of these were of sufficient academic rigour to be included in our review. The majority of these focused on judging the accuracy of self-assessment against some external standard, which raises questions about assumed reliability and validity of this ‘gold standard’. No papers were found which satisfied Kirkpatricks hierarchy above level 2, or which looked at the association between self-assessment and resulting changes in either clinical practice or patient outcomes. Thus our review was largely unable to answer the specific research questions and provide a solid evidence base for effective self-assessment. Despite this, there was some evidence that the accuracy of self-assessment can be enhanced by feedback, particularly video and verbal, and by providing explicit assessment criteria and benchmarking guidance. There was also some evidence that the least competent are also the least able to self-assess accurately. Our review recommends that these areas merit future systematic research to further our understanding of self-assessment. Conclusion: As in other BEME reviews, the methodological issues emerging from this review indicate a need for more rigorous study designs. In addition, it highlights the need to consider the potential for combining qualitative and quantitative data to further our understanding of how self-assessment can improve learning and professional clinical practice. Practice points•There is no solid evidence base within the health professions’ literature which establishes the effectiveness of self-assessment in: identifying learner needs; influencing learning activity; changing clinical practice.•The accuracy of self-assessment in clinical training may be improved by increasing the learners awareness of the standard to be achieved.•There is some indication that practical skills in clinical training may be better self-assessed than knowledge-based activities.•Self-assessment needs to be used as one tool amongst other sources of feedback to provide a more complete appraisal of competence in health care practice.•Future research should address the role that self-assessment plays in the everyday practice of health care decision-making.

The effectiveness of portfolios for post-graduate assessment and education: BEME Guide No 12

Background: Portfolios in post-graduate healthcare education are used to support reflective practice, deliver summative assessment, aid knowledge management processes and are seen as a key connection between learning at organisational and individual levels. This systematic review draws together the evidence on the effectiveness of portfolios across postgraduate healthcare and examines the implications of portfolios migrating from paper to an electronic medium across all professional settings. Methods: A literature search was conducted for articles describing the use of a portfolio for learning in a work or professional study environment. It was designed for high sensitivity and conducted across a wide range of published and unpublished sources relevant to professional education. No limits for study design or outcomes, country of origin or language were set. Blinded, paired quality rating was carried out, and detailed appraisal of and data extraction from included articles was managed using an online tool developed specifically for the review. Findings were discussed in-depth by the team, to identify and group pertinent themes when answering the research questions. Results: Fifty six articles from 10 countries involving seven healthcare professions met our inclusion criteria and minimum quality threshold; mostly uncontrolled observational studies. Portfolios encouraged reflection in some groups, and facilitated engagement with learning. There was limited evidence of the influence of a number of factors on portfolio use, including ongoing support from mentors or peers, implementation method, user attitude and level of initial training. Confounding variables underlying these issues, however have not been fully investigated. A number of authors explored the reliability and validity of portfolios for summative assessment but reports of accuracy across the disparate evidence base varied. Links to competency and Quality Assurance frameworks have been demonstrated. There were conflicting reports about whether the different purposes of portfolios can be combined without compromising the meaningfulness of the contents. There was good evidence that the flexibility of the electronic format brought additional benefits to users, assessors and organisations, and encouraged more enthusiastic use. Security of data remained a high priority issue at all levels, and there was emerging evidence of successful transfer between electronic portfolio systems. Conclusion: The evidence base is extensive, but contains few high quality studies with generalisable messages about the effectiveness of portfolios. There is, however, good evidence that if well implemented, portfolios are effective and practical in a number of ways including increasing personal responsibility for learning and supporting professional development. Electronic versions are better at encouraging reflection and users voluntarily spend longer on them. Regular feedback from a mentor enhances this success, despite competing demands on users’ time and occasional scepticism about the purpose of a portfolio. Reports of inter-rater reliability for summative assessments of portfolio data are varied and there is benefit to be gained from triangulating with other assessment methods. There was insufficient evidence to draw conclusions on how portfolios work in interdisciplinary settings.

The Best Evidence Medical Education Collaboration : processes, products and principles

I n this paper we describe the development of the Best Evidence Medical Education Collaboration (BEME), an organisation committed to evidence-informed policy and practice in health care sciences education. BEME supports moves from opinion-based to evidenceinformed education, advocating that more critical thought be given to the practice of education. Several BEME systematic reviews have now been published; and there are others in progress (see Table 1). All BEME reviews are available separately as BEME Guides and can be accessed at http://www.bemecollaboration.org. This production and dissemination of reviews supports the BEME ideology of evidenceinformed practice and policy, and its belief that this is key to enhancing student learning and practitioner capability. BEME supports moves from opinion-based to evidenceinformed education Evidencebased education

METRO taxonomy - progress report on assessment.

The Medical Education Taxonomy Research Organization (METRO) was formed to map and create terms to comprehensively describe the processes, procedures and concepts of medical education. The terms will be used in a variety of purposes, such as describing content for retrieval from e-learning environments, indexing literature in databases, and extracting and exchanging data for research, such as for BEME systematic reviews. METRO has now completed an extensive set of descriptors for assessment, which are freely available at http://metro2.blogspot.com. The organization welcomes the response and participation of all medical education professionals.

Successful collaboration in education: the UMeP.

As the health care education landscape in the UK changes rapidly and dramatically, collaboration across institutions bridging undergraduate and postgraduate fields is increasingly necessary. Collaboration entails both risks and benefits. There is a paucity of advice on how to ensure collaborative projects in medical education are effective.