Richard Fuller

How to measure the quality of the OSCE: A review of metrics - AMEE guide no. 49

With an increasing use of criterion-based assessment techniques in both undergraduate and postgraduate healthcare programmes, there is a consequent need to ensure the quality and rigour of these assessments. The obvious question for those responsible for delivering assessment is how is this ‘quality’ measured, and what mechanisms might there be that allow improvements in assessment quality over time to be demonstrated? Whilst a small base of literature exists, few papers give more than one or two metrics as measures of quality in Objective Structured Clinical Examinations (OSCEs). In this guide, aimed at assessment practitioners, the authors aim to review the metrics that are available for measuring quality and indicate how a rounded picture of OSCE assessment quality may be constructed by using a variety of such measures, and also to consider which characteristics of the OSCE are appropriately judged by which measure(s). The authors will discuss the quality issues both at the individual station level and across the complete clinical assessment as a whole, using a series of ‘worked examples’ drawn from OSCE data sets from the authors’ institution.

The Use of Mobile Technology for Work-Based Assessment: The Student Experience.

This paper outlines a research project conducted at Leeds University School of Medicine with Assessment & Learning in Practice Settings Centre for Excellence in Teaching and Learning, collaboration between the Universities of Leeds, Huddersfield, Bradford, Leeds Metropolitan University and the University of York St John. The research conducted is a proof of concept, examining the impact of delivering competency based assessment via personal digital assistants (PDAs) amongst a group of final year undergraduate medical students. This evaluation reports the student experience of mobile technology for assessment with positive effects; concluding that overall the students found completing assessments using a PDA straight forward and that the structured format of the assessment resulted in an increased, improved level of feedback, allowing students to improve their skills during the placement. A relationship between using the PDA for learning and setting goals for achievement was clearly demonstrated. [ABSTRACT FROM AUTHOR]

Is short-term remediation after OSCE failure sustained? A retrospective analysis of the longitudinal attainment of underperforming students in OSCE assessments

Background: Significant improvements in the delivery of criterion-based assessment techniques have improved confidence in standard setting and assessment quality. However, for underperforming students, a lack of evidence about longitudinal performance of this group poses dilemmas to educators when making decisions about the timing and nature of remediation. Aim: To investigate the longitudinal performance of the UK undergraduate medical degree students, with a particular focus on comparing the poorly performing students (i.e. those with borderline or failing grades) with the main cohort of students. Method: Over a 5-year period, 3200-student objective structured clinical examination (OSCE) assessments from a single medical school were investigated. A poorly performing subgroup of 125 students was identified and their longitudinal performance in the final 3 years of the undergraduate medical degree analysed. Result: The relative performance of this student group declines across serial OSCEs, despite current methods of ‘remediation and retest’. Conclusions: This analysis demonstrates that typically students in the poorly performing subgroup achieve only short-term success with traditional remediation and retest models, and critically show an absence of longitudinal improvement. There is a clear need for institutions to develop profiling models that can help identify this student group and develop effective, research led models of remediation.

Advancing the objective structured clinical examination: sequential testing in theory and practice

Models of short‐term remediation for failing students are typically associated with improvements in candidate performance at retest. However, the process is costly to deliver, particularly for performance retests with objective structured clinical examinations (OSCEs), and there is increasing evidence that these traditional models are associated with the longitudinal underperformance of candidates.

Collaboration across the pond: The multi-school progress testing project

This collaborative project between the National Board of Medical Examiners and four schools in the UK is investigating the feasibility and utility of a cross-school progress testing program drawing on test material recently retired from the United States Medical Licensing Examination (USMLE) Step 2 Clinical Knowledge (CK) examination. This article describes the design of the progress test; the process used to build, translate (localize), review, and finalize test forms; the approach taken to (web-based) test administration; and the procedure used to calculate and report scores. Results to date have demonstrated that it is feasible to use test items written for the US licensing examination as a base for developing progress test forms for use in the UK. Some content areas can be localized more readily than others, and care is clearly needed in review and revision of test materials to ensure that it is clinically appropriate and suitably phrased for use in the UK. Involvement of content experts in review and vetting of the test material is essential, and it is clearly desirable to supplement expert review with the use of quality control procedures based on the item statistics as a final check on the appropriateness of individual test items.

Can the tools of activity theory help us in advancing understanding and organisational change in undergraduate medical education

Continued changes to healthcare delivery in the UK, and an increasing focus on patient safety and quality improvement, require a radical rethink on how we enable graduates to begin work in challenging, complex environments. Professional regulatory bodies now require undergraduate medical schools to implement an ‘assistantship’ period in the final year of study, where senior medical students ‘shadow’ the work of junior doctors, with an expectation that they will be better ‘prepared’ for work. However, there is little guidance about what an ‘assistantship’ entails and the current emphasis on preparedness of students arguably underplays the importance of contextualised learning within the workplace environment. This paper will describe a modified Development Work Research (DWR) (Engeström in Developmental work research: activity theory in practice. Lehmanns Media, Berlin, 2005) approach to organisational change, enabling academic, clinical and administrative partners to develop assistantship placements in different hospitals. Our findings indicate that a modified DWR approach can reveal factors indicating organisational readiness to support change within a locally contextualised framework. The process has significant practical applications across a range of healthcare disciplines, as all professions seek to engage with the challenge of enabling successful transitions of graduates to the workplace.

Longitudinal interrelationships of OSCE station level analyses, quality improvement and overall reliability.

Objective Structured Clinical Examinations (OSCEs) are a key component within many healthcare assessment programmes. Quality assurance is designed to ensure rigour and credibility in decision making for both candidates and institutions, and most commonly expressed by a single measure of reliability. How overall reliability interrelates with OSCE station level analyses is less well established, especially with respect to the impact of quality improvements. This work examined longitudinal interrelationships of reliability and station level metrics alongside interventions to improve the OSCE, revealing an anticipated relationship between poor reliability and poor station level analyses. However, longitudinal profiling revealed that overall reliability proved relatively unresponsive to continued improvements across stations – highlighting the importance of station level analyses as a routine part of any assessment quality assurance

Legitimisation, personalisation and maturation: Using the experiences of a compulsory mobile curriculum to reconceptualise mobile learning

Abstract Background: Smartphone use is well established in society, with increasing acceptance in many professional workplaces. Despite the growth in mobile resources, how students and teachers benefit from these devices remains under-researched. Aims: An exploration of student and educators views on the impact of mobile learning re-sources on placement learning experiences as part of the Leeds ‘MBChB Mobile’ programme. Methods: Focus groups incorporating visual methodologies were undertaken with students from each year group; semi-structured interviews were undertaken with clinical teaching staff, including those who experienced the mobile programme as students themselves. Results: Four key themes emerged. ‘Maturity of learning’, related to the way in which senior students use resources in a more nuanced way than junior colleagues. ‘Learning differently’, identified ‘personalisation’ and ‘just in time’ opportunities that mobile resources afforded. ‘Learning legitimately’ was identified by students as key to ensuring they adopted appropriate behaviours in relation to mobile learning. Using mobile resources at undergraduate level was found to ‘change learning patterns for life’. Conclusions: There is a need to further develop the educational theory behind using mobile resources for learning. The results here suggest that mobile technologies are shaping learning behaviours, and are an indicator of learning maturity, reflecting the wider context of societal enculturation.

Investigating disparity between global grades and checklist scores in OSCEs

Abstract Background: When measuring assessment quality, increasing focus is placed on the value of station-level metrics in the detection and remediation of problems in the assessment. Aims: This article investigates how disparity between checklist scores and global grades in an Objective Structured Clinical Examination (OSCE) can provide powerful new insights at the station level whenever such disparities occur and develops metrics to indicate when this is a problem. Method: This retrospective study uses OSCE data from multiple examinations to investigate the extent to which these new measurements of disparity complement existing station-level metrics. Results: In stations where existing metrics are poor, the new metrics provide greater understanding of the underlying sources of error. Equally importantly, stations of apparently satisfactory “quality” based on traditional metrics are shown to sometimes have problems of their own – with a tendency for checklist score “performance” to be judged stronger than would be expected from the global grades awarded. Conclusions: There is an ongoing tension in OSCE assessment between global holistic judgements and the necessarily more reductionist, but arguably more objective, checklist scores. This article develops methods to quantify the disparity between these judgements and illustrates how such analyses can inform ongoing improvement in station quality.

Managing extremes of assessor judgment within the OSCE.

Abstract Context: There is a growing body of research investigating assessor judgments in complex performance environments such as OSCE examinations. Post hoc analysis can be employed to identify some elements of “unwanted” assessor variance. However, the impact of individual, apparently “extreme” assessors on OSCE quality, assessment outcomes and pass/fail decisions has not been previously explored. This paper uses a range of “case studies” as examples to illustrate the impact that “extreme” examiners can have in OSCEs, and gives pragmatic suggestions to successfully alleviating problems. Method and results: We used real OSCE assessment data from a number of examinations where at station level, a single examiner assesses student performance using a global grade and a key features checklist. Three exemplar case studies where initial post hoc analysis has indicated problematic individual assessor behavior are considered and discussed in detail, highlighting both the impact of individual examiner behavior and station design on subsequent judgments. Conclusions: In complex assessment environments, institutions have a duty to maximize the defensibility, quality and validity of the assessment process. A key element of this involves critical analysis, through a range of approaches, of assessor judgments. However, care must be taken when assuming that apparent aberrant examiner behavior is automatically just that.