Lara Zibarras

Evaluations of situational judgement tests to assess non‐academic attributes in selection

Medical Education 2012: 46: 850–868

Resilience Through the Lens of Interactionism: A Systematic Review

This systematic review presents findings from a conceptual and methodological review of resilience measures using an interactionist theoretical framework. The review is also intended to update findings from previous systematic reviews. Two databases (EBSCOHost and Scopus) were searched to retrieve empirical studies published up until 2013, with no lower time limit. All articles had to meet specific inclusion criteria, which resulted in 17 resilience measures selected for full review. Measures were conceptually evaluated against an interactionist framework and methodologically reviewed using Skinners (1981) validity evidence framework. We conclude that inconsistencies associated with the definition and operationalization of resilience warrant further conceptual development to explain resilience as a dynamic and interactive phenomenon. In particular, measures of resilience may benefit from a greater focus on within-person variance typically associated with behavioral consistency across situations. The use of alternative measurement modalities to self-report scales, such as situational judgment tests, is proposed as a way of advancing knowledge in this area.

Situational judgement tests in medical education and training: Research, theory and practice: AMEE Guide No. 100

Abstract Why use SJTs? Traditionally, selection into medical education professions has focused primarily upon academic ability alone. This approach has been questioned more recently, as although academic attainment predicts performance early in training, research shows it has less predictive power for demonstrating competence in postgraduate clinical practice. Such evidence, coupled with an increasing focus on individuals working in healthcare roles displaying the core values of compassionate care, benevolence and respect, illustrates that individuals should be selected on attributes other than academic ability alone. Moreover, there are mounting calls to widen access to medicine, to ensure that selection methods do not unfairly disadvantage individuals from specific groups (e.g. regarding ethnicity or socio-economic status), so that the future workforce adequately represents society as a whole. These drivers necessitate a method of assessment that allows individuals to be selected on important non-academic attributes that are desirable in healthcare professionals, in a fair, reliable and valid way. What are SJTs? Situational judgement tests (SJTs) are tests used to assess individuals’ reactions to a number of hypothetical role-relevant scenarios, which reflect situations candidates are likely to encounter in the target role. These scenarios are based on a detailed analysis of the role and should be developed in collaboration with subject matter experts, in order to accurately assess the key attributes that are associated with competent performance. From a theoretical perspective, SJTs are believed to measure prosocial Implicit Trait Policies (ITPs), which are shaped by socialisation processes that teach the utility of expressing certain traits in different settings such as agreeable expressions (e.g. helping others in need), or disagreeable actions (e.g. advancing ones own interest at others, expense). Are SJTs reliable, valid and fair? Several studies, including good quality meta-analytic and longitudinal research, consistently show that SJTs used in many different occupational groups are reliable and valid. Although there is over 40 years of research evidence available on SJTs, it is only within the past 10 years that SJTs have been used for recruitment into medicine. Specifically, evidence consistently shows that SJTs used in medical selection have good reliability, and predict performance across a range of medical professions, including performance in general practice, in early years (foundation training as a junior doctor) and for medical school admissions. In addition, SJTs have been found to have significant added value (incremental validity) over and above other selection methods such as knowledge tests, measures of cognitive ability, personality tests and application forms. Regarding differential attainment, generally SJTs have been found to have lower adverse impact compared to other selection methods, such as cognitive ability tests. SJTs have the benefit of being appropriate both for use in selection where candidates are novices (i.e. have no prior role experience or knowledge such as in medical school admissions) as well as settings where candidates have substantial job knowledge and specific experience (as in postgraduate recruitment for more senior roles). An SJT specification (e.g. scenario content, response instructions and format) may differ depending on the level of job knowledge required. Research consistently shows that SJTs are usually found to be positively received by candidates compared to other selection tests such as cognitive ability and personality tests. Practically, SJTs are difficult to design effectively, and significant expertise is required to build a reliable and valid SJT. Once designed however, SJTs are cost efficient to administer to large numbers of candidates compared to other tests of non-academic attributes (e.g. personal statements, structured interviews), as they are standardised and can be computer-delivered and machine-marked.

Evaluating cognitive ability, knowledge tests and situational judgement tests for postgraduate selection

Medical Education 2012: 46: 399–408

A survey of UK selection practices across different organization sizes and industry sectors

This paper presents results of a study examining the methods used to select employees in 579 UK organizations representing a range of different organization sizes and industry sectors. Overall, a smaller proportion of organizations in this sample reported using formalized methods (e.g., assessment centres) than informal methods (e.g., unstructured interviews). The curriculum vitae (CVs) was the most commonly used selection method, followed by the traditional triad of application form, interviews, and references. Findings also indicated that the use of different selection methods was similar in both large organizations and small-to-medium-sized enterprises. Differences were found across industry sector with public and voluntary sectors being more likely to use formalized techniques (e.g., application forms rather than CVs and structured rather than unstructured interviews). The results are discussed in relation to their implications, both in terms of practice and future research.

Evaluating candidate reactions to selection practices using organisational justice theory.

Medical Education 2011: 45: 289–297

Designing Selection Systems for Medicine: The importance of balancing predictive and political validity in high-stakes selection contexts

Selection into medical education and training is a high‐stakes process. A key unanswered issue is the effectiveness of measuring noncognitive predictors via both low‐fidelity and high‐fidelity selection approaches in this high‐stakes context. We review studies investigating the effectiveness of multiple selection instruments in terms of predictive validity, incremental validity, and applicant reactions in both entry‐level and advanced‐level medical selection. Our results show that the situational judgment test (SJT) is the best single predictor of performance, operationalized in multiple ways. In addition, the low‐fidelity SJT has incremental predictive power over cognitively oriented tests, and high‐fidelity assessment center (AC) exercises add incremental validity over the low‐fidelity (and less costly) selection methods. Concerning applicant reactions, results show that overall, the selection system is positively received. However, the method with the highest predictive validity – the SJT – received comparatively lower face validity ratings which may present a ‘justice dilemma’ for employers. Furthermore, various other stakeholders have a political interest in the selection methods used (e.g., government, the regulators and trade unions).

Recruiting for values in healthcare: a preliminary review of the evidence

Displaying compassion, benevolence and respect, and preserving the dignity of patients are important for any healthcare professional to ensure the provision of high quality care and patient outcomes. This paper presents a structured search and thematic review of the research evidence relating to values-based recruitment within healthcare. Several different databases, journals and government reports were searched to retrieve studies relating to values-based recruitment published between 1998 and 2013, both in healthcare settings and other occupational contexts. There is limited published research related to values-based recruitment directly, so the available theoretical context of values is explored alongside an analysis of the impact of value congruence. The implications for the design of selection methods to measure values is explored beyond the scope of the initial literature search. Research suggests some selection methods may be appropriate for values-based recruitment, such as situational judgment tests (SJTs), structured interviews and multiple-mini interviews (MMIs). Personality tests were also identified as having the potential to compliment other methods (e.g. structured interviews), as part of a values-based recruitment agenda. Methods including personal statements, references and unstructured/‘traditional’ interviews were identified as inappropriate for values-based recruitment. Practical implications are discussed in the context of values-based recruitment in the healthcare context. Theoretical implications of our findings imply that prosocial implicit trait policies, which could be measured by selection tools such as SJTs and MMIs, may be linked to individuals’ values via the behaviours individuals consider to be effective in given situations. Further research is required to state this conclusively however, and methods for values-based recruitment represent an exciting and relatively unchartered territory for further research.

HRM practices used to promote pro-environmental behavior: a UK survey

To date, the extent to which UK organizations use human resource management (HRM) practices to promote pro-environmental behavior through workplace HRM policies and initiatives is under-researched within the literature. Therefore, this paper presents results of a survey investigating current HRM practices used to promote pro-environmental behavior in a sample of 214 UK organizations representing different sizes and industry sectors. Overall, findings indicated that HRM practices are not used to a great extent to encourage employees to become more pro-environmental. The most prevalent practices used within organizations incorporated elements of management involvement supporting the idea that managers are the gatekeepers to environmental performance. Although organizations indicated that some HRM practices were more effective than others at encouraging pro-environmental behavior in their staff, only a very small percentage of organizations actually conducted any form of evaluation; organizations consequently lack clear evidence as to whether their HRM practices actually result in employee behavior change. Practical implications and future research directions are discussed.

Advancing selection in an SME: Is best practice methodology applicable?

Effective selection tools are important for identifying high caliber employees in SMEs, yet few SMEs use tools created using ‘best practice’ methodology. Selection literature tends to focus on large organizations and is conceptual rather than empirical; which may make it difficult for SMEs to use a best practice approach. This article addresses this by providing an empirical account of the design and validation of two selection tools in a medium-sized recruitment consultancy using a best practice selection methodology. Two work sample tests were developed using critical incident technique interviews, and validated using a concurrent design with existing recruitment consultants who were ‘high’ or ‘low’ performers according to sales output. Results indicated that the tools significantly differentiated between high and low performers, and there was a significant correlation between test performance and individual sales output. Findings are discussed in relation to implications for research and practice in SMEs and selection.