Fiona Patterson

Assessment for selection for the health care professions and specialty training : consensus statement and recommendations from the Ottawa 2010 Conference

Assessment for selection in medicine and the health professions should follow the same quality assurance processes as in-course assessment. The literature on selection is limited and is not strongly theoretical or conceptual. For written testing, there is evidence of the predictive validity of Medical College Admission Test (MCAT) for medical school and licensing examination performance. There is also evidence for the predictive validity of grade point average, particularly in combination with MCAT for graduate entry but little evidence about the predictive validity of school leaver scores. Interviews have not been shown to be robust selection measures. Studies of multiple mini-interviews have indicated good predictive validity and reliability. Of other measures used in selection, only the growing interest in personality testing appears to warrant future work. Widening access to medical and health professional programmes is an increasing priority and relates to the social accountability mandate of medical and health professional schools. While traditional selection measures do discriminate against various population groups, there is little evidence on the effect of non-traditional measures in widening access. Preparation and outreach programmes show most promise. In summary, the areas of consensus for assessment for selection are small in number. Recommendations for future action focus on the adoption of principles of good assessment and curriculum alignment, use of multi-method programmatic approaches, development of interdisciplinary frameworks and utilisation of sophisticated measurement models. The social accountability mandate of medical and health professional schools demands that social inclusion, workforce issues and widening of access are embedded in the principles of good assessment for selection.

How effective are selection methods in medical education? A systematic review

Selection methods used by medical schools should reliably identify whether candidates are likely to be successful in medical training and ultimately become competent clinicians. However, there is little consensus regarding methods that reliably evaluate non‐academic attributes, and longitudinal studies examining predictors of success after qualification are insufficient. This systematic review synthesises the extant research evidence on the relative strengths of various selection methods. We offer a research agenda and identify key considerations to inform policy and practice in the next 50 years.

The Validity and Incremental Validity of Knowledge Tests, Low-Fidelity Simulations, and High-Fidelity Simulations for Predicting Job Performance in Advanced-Level High-Stakes Selection

In high-stakes selection among candidates with considerable domain-specific knowledge and experience, investigations of whether high-fidelity simulations (assessment centers; ACs) have incremental validity over low-fidelity simulations (situational judgment tests; SJTs) are lacking. Therefore, this article integrates research on the validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations in advanced-level high-stakes settings. A model and hypotheses of how these 3 predictors work in combination to predict job performance were developed. In a sample of 196 applicants, all 3 predictors were significantly related to job performance. Both the SJT and the AC had incremental validity over the knowledge test. Moreover, the AC had incremental validity over the SJT. Model tests showed that the SJT fully mediated the effects of declarative knowledge on job performance, whereas the AC partially mediated the effects of the SJT.

Organizational culture change: An inter-group attributional analysis

Attempts by researchers to quantify organizational culture and culture change have been severely hindered by a lack of theoretical models to aid empirical investigation. This paper has two aims. First, a socio-cognitive model of organizational culture is discussed which dee nes ‘ culture’ as the product of a dynamic and collective process of ‘ sense-making’ undertaken by members of a group or organization. Second, an empirical case study is presented which uses attributional analysis to quantify the beliefs held by key stakeholder groups involved in a culture change programme within a multi-national manufacturing organization. A total of 1230 attributional statements were coded using a modie ed version of the Leeds Attributional Coding System (LACS), which included four main dimensions: stable‐ unstable, global‐ specie c, internal‐ external, controllable‐ uncontrollable. Results indicated considerable inter-group di Verences between managers, trainers and trainees in their cognitive maps as sense-making heuristics of this planned change process. The implications of this model and the usefulness of attributional analysis as a method for evaluating dynamic aspects of organizational culture and culture change are discussed.

Evaluations of situational judgement tests to assess non‐academic attributes in selection

Medical Education 2012: 46: 850–868

Evaluation of three short-listing methodologies for selection into postgraduate training in general practice

Objective  This study aimed to evaluate the effectiveness and efficiency of three short‐listing methodologies for use in selecting trainees into postgraduate training in general practice in the UK.

Situational judgement tests in medical education and training: Research, theory and practice: AMEE Guide No. 100

Abstract Why use SJTs? Traditionally, selection into medical education professions has focused primarily upon academic ability alone. This approach has been questioned more recently, as although academic attainment predicts performance early in training, research shows it has less predictive power for demonstrating competence in postgraduate clinical practice. Such evidence, coupled with an increasing focus on individuals working in healthcare roles displaying the core values of compassionate care, benevolence and respect, illustrates that individuals should be selected on attributes other than academic ability alone. Moreover, there are mounting calls to widen access to medicine, to ensure that selection methods do not unfairly disadvantage individuals from specific groups (e.g. regarding ethnicity or socio-economic status), so that the future workforce adequately represents society as a whole. These drivers necessitate a method of assessment that allows individuals to be selected on important non-academic attributes that are desirable in healthcare professionals, in a fair, reliable and valid way. What are SJTs? Situational judgement tests (SJTs) are tests used to assess individuals’ reactions to a number of hypothetical role-relevant scenarios, which reflect situations candidates are likely to encounter in the target role. These scenarios are based on a detailed analysis of the role and should be developed in collaboration with subject matter experts, in order to accurately assess the key attributes that are associated with competent performance. From a theoretical perspective, SJTs are believed to measure prosocial Implicit Trait Policies (ITPs), which are shaped by socialisation processes that teach the utility of expressing certain traits in different settings such as agreeable expressions (e.g. helping others in need), or disagreeable actions (e.g. advancing ones own interest at others, expense). Are SJTs reliable, valid and fair? Several studies, including good quality meta-analytic and longitudinal research, consistently show that SJTs used in many different occupational groups are reliable and valid. Although there is over 40 years of research evidence available on SJTs, it is only within the past 10 years that SJTs have been used for recruitment into medicine. Specifically, evidence consistently shows that SJTs used in medical selection have good reliability, and predict performance across a range of medical professions, including performance in general practice, in early years (foundation training as a junior doctor) and for medical school admissions. In addition, SJTs have been found to have significant added value (incremental validity) over and above other selection methods such as knowledge tests, measures of cognitive ability, personality tests and application forms. Regarding differential attainment, generally SJTs have been found to have lower adverse impact compared to other selection methods, such as cognitive ability tests. SJTs have the benefit of being appropriate both for use in selection where candidates are novices (i.e. have no prior role experience or knowledge such as in medical school admissions) as well as settings where candidates have substantial job knowledge and specific experience (as in postgraduate recruitment for more senior roles). An SJT specification (e.g. scenario content, response instructions and format) may differ depending on the level of job knowledge required. Research consistently shows that SJTs are usually found to be positively received by candidates compared to other selection tests such as cognitive ability and personality tests. Practically, SJTs are difficult to design effectively, and significant expertise is required to build a reliable and valid SJT. Once designed however, SJTs are cost efficient to administer to large numbers of candidates compared to other tests of non-academic attributes (e.g. personal statements, structured interviews), as they are standardised and can be computer-delivered and machine-marked.

Trust me...: psychological and behavioral predictors of perceived physician empathy.

A sociocognitive model of distal and proximal predictors of empathic judgments was tested among 100 physicians. The authors hypothesized that physician perceived control would affect empathy ratings via physician communication style. Specifically, physicians with high perceived control would use more open communication and be rated as more empathic. Physicians with low perceived control would use a controlling communication style and be rated as less empathic. Physicians completed a medical attribution questionnaire prior to a structured patient consultation exercise, during which patients and assessors rated physician empathy. The exercise was audiotaped, transcribed, and content analyzed for verbal behaviors. Support was found for the hypotheses; however, patients, but not medical assessors, associated empathy with reassurance and provision of medical information.

Evaluating cognitive ability, knowledge tests and situational judgement tests for postgraduate selection

Medical Education 2012: 46: 399–408

Using job analysis to identify core and specific competencies: implications for selection and recruitment.

Objective  Modern postgraduate medical training requires both accurate and reliable selection procedures. An essential first step is to conduct detailed job analysis studies. This paper reports data on a series of job analyses to develop a competency model for three secondary care specialties (anaesthesia, obstetrics and gynaecology, and paediatrics).