Godfrey Pell

Use and misuse of Likert scales

Editor – I read with some interest Dr Jamieson’s article on the uses and misuses of Likert scales. The issues of the appropriate statistical models for parametric (i.e. minimally interval) data and non-parametric (i.e. ordinal or categorical) data have been around for at least 50 years (Lord 1953). The real issue is not between analytical techniques, but in properly understanding the nature of the analyses, and the resulting inferences.

Web 2.0 and social software: the medical student way of e-learning

Background: A wide range of social software has become readily available to young people. There is increasing interest in the exciting possibilities of using social software for undergraduate medical education. Aims: To identify the nature and extent of the use of social software by first year medical students. Method: A structured self-administered questionnaire survey of 212 students. Results: Over 90 percent used instant messaging and social networking sites were highly used (70 percent). There was no significant difference between males and females. Blogs were read by about a fifth of students and a small number (8%) wrote their own blogs. A fifth of males stated that they were users of media sharing and contributed to wikis. Social bookmarking was rarely used by either sex. Conclusion: Medical educators need to recognise the potential of social software in undergraduate medical education but it is essential that students maintain the informality and privacy of these sites. The challenge for all medical educators is how to integrate social software into current curricula and institutional Virtual Learning Environments (VLEs).

How to measure the quality of the OSCE: A review of metrics - AMEE guide no. 49

With an increasing use of criterion-based assessment techniques in both undergraduate and postgraduate healthcare programmes, there is a consequent need to ensure the quality and rigour of these assessments. The obvious question for those responsible for delivering assessment is how is this ‘quality’ measured, and what mechanisms might there be that allow improvements in assessment quality over time to be demonstrated? Whilst a small base of literature exists, few papers give more than one or two metrics as measures of quality in Objective Structured Clinical Examinations (OSCEs). In this guide, aimed at assessment practitioners, the authors aim to review the metrics that are available for measuring quality and indicate how a rounded picture of OSCE assessment quality may be constructed by using a variety of such measures, and also to consider which characteristics of the OSCE are appropriately judged by which measure(s). The authors will discuss the quality issues both at the individual station level and across the complete clinical assessment as a whole, using a series of ‘worked examples’ drawn from OSCE data sets from the authors’ institution.

Student teachers’ experiences of initial teacher preparation in England: core themes and variation

Drawing on data generated via large‐scale survey and in‐depth interview methods, this article reports findings which show that being a student teacher in early‐twenty‐first‐century England is a demanding personal experience which requires considerable engagement and commitment in the face of built‐in challenges and risks, and which engenders, for many, highly charged affective responses. Student teachers are centrally concerned during this time with their (changing) identities, their relationships with others and the relevance of course provision. Findings also indicate that, in some respects, student teachers’ accounts of their experiences are systematically differentiated according to a number of factors, notably the initial teacher preparation route being followed, their age, and their prior conceptions and expectations of teaching and of learning to teach. These findings are situated in the broader literature on teacher development and some implications for teacher educators are discussed.

Using borderline methods to compare passing standards for OSCEs at graduation across three medical schools

Context  Medical schools in the UK set their own graduating examinations and pass marks. In a previous study we examined the equivalence of passing standards using the Angoff standard‐setting method. To address the limitation this imposed on that work, we undertook further research using a standard‐setting method specifically designed for objective structured clinical examinations (OSCEs).

Predicting performance of first year engineering students and the importance of assessment tools therein

Abstract In recent years, the increase in the number of people entering university has contributed to a greater variability in the background of those beginning programmes. Consequently, it has become even more important to understand a student’s prior knowledge of a given subject. Two main reasons for this are to produce a suitable first year curriculum and to ascertain whether a student would benefit from additional support. Therefore, in order that any necessary steps can be taken to improve a student’s performance, the ultimate goal would be the ability to predict future performance. A continuing change in students’ prior mathematics (and mechanics) knowledge is being seen in engineering, a subject that requires a significant amount of mathematics knowledge. This paper describes statistical regression models used for predicting students’ first year performance. Results from these models highlight that a mathematics diagnostic test is not only useful for gaining information on a student’s prior knowledge but is also one of the best predictors of future performance. In the models, it was also found that students’ marks could be improved by seeking help in the university’s mathematics learning support centre. Tools and methodologies (e.g. surveys and diagnostic tests) suitable for obtaining data used in the regression models are also discussed.

Is short-term remediation after OSCE failure sustained? A retrospective analysis of the longitudinal attainment of underperforming students in OSCE assessments

Background: Significant improvements in the delivery of criterion-based assessment techniques have improved confidence in standard setting and assessment quality. However, for underperforming students, a lack of evidence about longitudinal performance of this group poses dilemmas to educators when making decisions about the timing and nature of remediation. Aim: To investigate the longitudinal performance of the UK undergraduate medical degree students, with a particular focus on comparing the poorly performing students (i.e. those with borderline or failing grades) with the main cohort of students. Method: Over a 5-year period, 3200-student objective structured clinical examination (OSCE) assessments from a single medical school were investigated. A poorly performing subgroup of 125 students was identified and their longitudinal performance in the final 3 years of the undergraduate medical degree analysed. Result: The relative performance of this student group declines across serial OSCEs, despite current methods of ‘remediation and retest’. Conclusions: This analysis demonstrates that typically students in the poorly performing subgroup achieve only short-term success with traditional remediation and retest models, and critically show an absence of longitudinal improvement. There is a clear need for institutions to develop profiling models that can help identify this student group and develop effective, research led models of remediation.

Setting standards for student assessment

The increase in litigation by students who are dissatisfied with their assessment, and to a lesser extent the time and monetary costs of student appeals makes it imperative that institutions adopt a robust assessment strategy. The concern of consumers with respect to professional services offered by students after graduation is also an issue. This paper examines issues round the ‘borderline method’ of standard setting, using regression analysis. The paper will present data from two cohorts of students, and will examine the benefits and problems associated with this method. Increasingly academic institutions are being required to improve the validity of assessment processes; often this is at the expense of reliability. The new assessment procedures often have different assessors, have practical aspects which cannot be replicated across the cohort, and therefore raise issues with respect to the robustness of the comparative student grading mechanism. This issue has been particularly important in the field of medical education for a number of years; with medical students in the latter stages of their courses being required to demonstrate competence in a variety of different simulated clinical activities with different patients, in front of different assessors in different hospitals and on different days. This poses serious questions in the robustness of setting the pass–fail boundary and to a lesser extent the honours boundaries.

Advancing the objective structured clinical examination: sequential testing in theory and practice

Models of short‐term remediation for failing students are typically associated with improvements in candidate performance at retest. However, the process is costly to deliver, particularly for performance retests with objective structured clinical examinations (OSCEs), and there is increasing evidence that these traditional models are associated with the longitudinal underperformance of candidates.

The impact of the inclusion of simulated patient ratings on the reliability of OSCE assessments under the borderline regression method

Background: In Objective Structured Clinical Examinations (OSCEs), the use of simulated patients (SPs) at many stations is a key aspect of the assessment. Often the SPs are asked to provide formal feedback (ratings) of their experience with the students under examination. Aims: This study analyses whether and how exactly SP data can be best used to enhance the robustness of the formal standard setting process. Methods: A retrospective statistical investigation into the relationship between SP ratings and those provided by the clinical assessors (criterion-based checklist scores and overall grades for each station) is presented. In addition, the paper also includes a study into the impact of the inclusion of the SP ratings in the formal standard setting process for OSCEs, particularly when pass marks are calculated using the borderline regression method. Results: The general results of the analysis, including discussion of two distinct methods for the combining of the SP ratings and assessor judgements, are presented, and demonstrate that the inclusion of this additional data can have important effects on individual student results. Conclusion: It is possible for the overall quality of the OSCE assessment process to be improved, with increased reliability, by combining assessor checklist scores and SP ratings.