Bill Irish

Using computers for assessment in medicine

Computer based testing can offer many advantages over traditional paper based methods of assessment. The authors look at what it means and its potential uses for assessment in medicine

Evaluating cognitive ability, knowledge tests and situational judgement tests for postgraduate selection

Medical Education 2012: 46: 399–408

The predictive validity of selection for entry into postgraduate training in general practice: evidence from three longitudinal studies

BACKGROUND The selection methodology for UK general practice is designed to accommodate several thousand applicants per year and targets six core attributes identified in a multi-method job-analysis study AIM To evaluate the predictive validity of selection methods for entry into postgraduate training, comprising a clinical problem-solving test, a situational judgement test, and a selection centre. DESIGN AND SETTING A three-part longitudinal predictive validity study of selection into training for UK general practice. METHOD In sample 1, participants were junior doctors applying for training in general practice (n = 6824). In sample 2, participants were GP registrars 1 year into training (n = 196). In sample 3, participants were GP registrars sitting the licensing examination after 3 years, at the end of training (n = 2292). The outcome measures include: assessor ratings of performance in a selection centre comprising job simulation exercises (sample 1); supervisor ratings of trainee job performance 1 year into training (sample 2); and licensing examination results, including an applied knowledge examination and a 12-station clinical skills objective structured clinical examination (OSCE; sample 3). RESULTS Performance ratings at selection predicted subsequent supervisor ratings of job performance 1 year later. Selection results also significantly predicted performance on both the clinical skills OSCE and applied knowledge examination for licensing at the end of training. CONCLUSION In combination, these longitudinal findings provide good evidence of the predictive validity of the selection methods, and are the first reported for entry into postgraduate training. Results show that the best predictor of work performance and training outcomes is a combination of a clinical problem-solving test, a situational judgement test, and a selection centre. Implications for selection methods for all postgraduate specialties are considered.

Evaluating candidate reactions to selection practices using organisational justice theory.

Medical Education 2011: 45: 289–297

Selecting general practice specialty trainees: where next?

Selection into GP speciality training is based on results of a multi-method job analysis study. Six key competency domains were identified as priorities to assess through the current national selection process, including empathy, communication, integrity, clinical expertise, problem-solving, and resilience. Each applicant is assessed using clinical problem-solving and situational judgement machine marked tests, followed by high fidelity exercises at regional selection centres. These show good internal reliability and predictive validity, with high correlations with subsequent job performance and outcomes in the MRCGP examinations. Candidate feedback is generally positive, where candidates prefer multiple opportunities to demonstrate their aptitude. When comparing selection methods, candidates perceive high fidelity assessments (for example, a consultation exercise with a simulated patient) as the most job relevant and fair. Emerging data provides opportunities to both streamline and re-engineer the process, so as to optimise efficiency (including cost-effectiveness) while further developing its robustness. Logistic considerations favour delivery of multiple equated versions of machine marked tests in invigilated test centres. As with other assessments in medical education, concerns remain about differential performance between UK and international medical graduates. There is an urgent need to review the job analysis and selection criteria given profound changes in UK general practice taking place over the last decade.

UK academic general practice and primary care

Visible? Viable? Invaluable

Consultants' attitudes to the assessment of GP specialty trainees during hospital placements.

This study explored the views and experiences of a sample of hospital consultants using the RCGP e-portfolio, within the wider context of WPBA. Whilst the use of e-portfolios and WPBA in a hospital setting was broadly valued and accepted, issues of e-portfolio functionality often hindered the professional judgement of competencies. Issues relating to training in the use of the RCGP e-portfolio and the need for adequate protected time were identified, together with a reluctance to give negative formative feedback through this format.

Does computer-based testing (CBA) have a future in the assessment of general practitioners in the united kingdom?

The training and assessment of all doctors in the UK is about to undergo a number of significant changes. The assessment of general practitioner registrars (GPRs) has come under the scrutiny of the Postgraduate Medical Education and Training Board (PMETB), which went ‘live’ in September 2005 and which has published the principles and standards for assessment of postgraduate medical training. Their guidance advises that assessment methods should be chosen on the basis of validity, reliability, feasibility, cost-effectiveness, opportunities for feedback, and impact on learning. The National Summative Assessment Board (NSAB) and Royal College of General Practitioners (RCGP) have established an assessment group with a view to developing a single future fit-for-purpose licensing assessment package to supersede the current dual-track summative assessment and Membership of the Royal College of General Practitioners (MRCGP) qualifications. The recently published Fifth Report of the Shipman Inquiry has recommended that all doctors should pass a knowledge test every five years as part of the process of their revalidation and that there should be a ‘summative evaluation of each individual doctor’s fitness to practise’, and in addition it has been proposed that such testing should be administered online. The public has a right to proficient and safe doctors and some form of knowledge testing is seen as a pre-requisite for ensuring that an adequate standard has been achieved. Traditional medical undergraduate and postgraduate examinations are underpinned by the established assessment methodologies of essay, multiple-choice paper (MCP), viva, short and long cases, but each of these have been criticised in terms of both validity and reliability. There has, therefore, been considerable recent research interest in new assessment techniques that reflect an international trend towards outcome-based assessment. Examples include: Education for Primary Care (2006) 17: 1–9 # 2006 Radcliffe Publishing Limited

The Potential Use of Computer-Based Assessment for Knowledge Testing of General Practice Registrars

Computer-based assessment (CBA) is widely used in UK medical education for the formative assessment of medical students, and to a lesser extent at a postgraduate level. During the last decade its international use for high-stakes summative assessments (for example the United States Medical Licensing Examination) has become far more common, due to a number of advantages intrinsic to CBA. During 2002 a 47-item CBA was produced using questions of an equivalent standard and content to the current MRCGP multiple-choice paper (MCP). Using a commercial CBA software package, a self-running CD-ROM was develEducation for Primary Care (2006) 17: 24–31 # 2006 Radcliffe Publishing Limited

Developing the high-flying registrar: a qualitative evaluation of the Severn Deanery education scholar programme.

The introduction of the Education Scholarship and Education Fellowship programmes in the Severn Deanery in 2008 was in response to an emergent need for a more formalised career structure for the most able GPSTs and specifically for identified education scholars to progress over time from a scholar to temporary and substantive training programme director (TPD) posts. As a result, two scholars have progressed to being appointed as fellows (one-year fixed-term TPDs) over the last three years and one of these is now a TPD. This qualitative research study sought to identify the value and acceptance of the scholar scheme within the Severn Deanery and in particular assess the impact of the scholar scheme in terms of educational benefits, impact on professional practice and future career opportunities. It also undertook to examine the key factors affecting the quality of experience of the scholar scheme (e.g.induction, mentorship, autonomous vs. prescribed education opportunities etc.). Findings showed unequivocal and universal support for the scheme.