Alec W. Serlie

Webcam testing: Validation of an innovative open-ended multimedia test

A modern test that takes advantage of the opportunities provided by advancements in computer technology is the multimedia test. The purpose of this study was to investigate the criterion-related validity of a specific open-ended multimedia test, namely a webcam test, by means of a concurrent validity study. In a webcam test a number of work-related situations are presented and participants have to respond as if these were real work situations. The responses are recorded with a webcam. The aim of the webcam test which we investigated is to measure the effectiveness of social work behaviour. This first field study on a webcam test was conducted in an employment agency in The Netherlands. The sample consisted of 188 consultants who participated in a certification process. For the webcam test, good interrater reliabilities and internal consistencies were found. The results showed the webcam test to be significantly correlated with job placement success. The webcam test scores were also found to be related to job knowledge. Hierarchical regression analysis demonstrated that the webcam test has incremental validity up to and beyond job knowledge in predicting job placement success. The webcam test, therefore, seems a promising type of instrument for personnel selection.

Effects of Individual Differences on the Perceived Job Relatedness of a Cognitive Ability Test and a Multimedia Situational Judgment Test

Although there is a growing number of publications concerning applicant reactions to different selection instruments, the relationships between individual differences and applicant reactions have largely remained unexplored. The aim of the present study was to examine the effects of several testing-related and general individual differences (anxiety, self-evaluations, and personality) on the most commonly studied dimension of applicant reactions, namely the perceived job relatedness of selection instruments. Participants were 153 psychology students, who completed a cognitive ability test and a multimedia situational judgment test as part of their educational program. Our results indicated that computer anxiety negatively affected perceived job relatedness and core self-evaluations, subjective well-being, agreeableness, emotional stability, and openness to experience positively affected perceived job relatedness. Openness to experience was the most consistent predictor of perceived job relatedness. The results of our study suggest that certain individuals may be more predisposed to react positively to selection instruments. Therefore, we concluded that the nature of the applicant pool should be carefully considered when designing interventions to improve applicant reactions.

Implicit trait policies in multimedia situational judgment tests of leadership skills: Can they predict leadership behavior?

To explain why situational judgment tests are often correlated with personality measures, Motowidlo, Hooper, and Jackson (2006a, 2006b) developed the implicit trait policy theory. Implicit trait policies are beliefs about causal relationships between personality traits and behavioral effectiveness. Among 180 employees, this field study examined whether a multimedia situational judgment test that was intended to assess leadership skills can capture individual differences in such policies. Furthermore, it was examined whether these implicit trait policies were able to predict leadership behavior. Results confirmed that the situational judgment test was able to capture individual differences in implicit trait policies for Extraversion and Conscientiousness. Furthermore, results showed that implicit trait policies for Extraversion can predict leadership behavior over and above leadership experience and the associated personality trait.

Assessing Leadership Competencies Through Social Network Analysis

Leadership competencies are regularly identified as some of the most in demand workplace competencies. However, the development of these competencies requires appropriate assessments that are often either highly subjective (e.g. manager appraisals) or prohibitively expensive (e.g. roleplays with trained actors). The increasing usage of workplace social networks and increasing prevalence of digital collaboration tools presents a continuous stream of social interactions that can contain evidence of leadership occurring in situ. In this paper we present initial research on the feasibility of Social Network Analysis in the workplace to assess leadership competencies. We examine the assessment in terms of content, construct, and criterion validity. We then present our hypotheses on how the assessment can be conducted including the algorithms necessary to extract relevant features from a social network graph model. Our initial research, to our surprise, shows a weak correlation between an individual’s degree centrality and betweenness centrality and the leadership competency that is self-reported. However, experiments indicated a strong positive correlation between network structure based and social collaboration activities based features and the characteristics of the leadership competencies. Our initial machine learning experiments achieved an Area Under the Curve (AUC) score of 0.899 when social network and collaboration activity based features were leveraged to distinguish individuals with self-reported leadership competencies from others. Finally we discuss our findings on the practicality of the approach, and future work on validating and improving the results obtained using parallel conventional assessments for leadership competencies.