Paul J. Taylor

Asking applicants what they would do versus what they did do: A meta‐analytic comparison of situational and past behaviour employment interview questions

Criterion-related validities and inter-rater reliabilities for structured employment interview studies using situational questions (e.g. “Assume that you were faced with the following situation … what would you do?”) were compared meta-analytically with studies using past behaviour questions (e.g. “Can you think of a time when … what did you do?”). Validities and reliabilities were further analysed in terms of whether descriptively-anchored rating scales were used to judge interviewees answers, and validities for each question type were also assessed across three levels of job complexity. n nWhile both question formats yielded high validity estimates, studies using past behaviour questions, when used with discriptively anchored answer rating scales, yielded a substantially higher mean validity estimate than studies using the situational question format with descriptively-anchored answer rating scales (.63 versus .47). Question type (situational versus past behaviour) was found to moderate interview validity, after controlling for whether studies used answer rating scales. No support was found for the hypothesis that situational questions are less valid for predicting job performance in high-complexity jobs. n nSample-weighted mean inter-rater reliabilities were similar for both situational and past behaviour questions, provided that descriptively-anchored rating scales were used (.79 and .77, respectively), although they were slightly lower (.73) for past behaviour question studies lacking such rating scales.

Effects of Introducing a Performance Management System on Employees' Subsequent Attitudes and Effort:

A longitudinal evaluation was conducted on the effects of introducing a performance management system (PMS), which featured merit-based bonus pay, on subsequent employee attitudes and self-reported work effort in a small, government organization. Additionally, employees targets of blame for receiving lower-than-expected ratings were explored. A significant change in employees organizational commitment occurred over the time that the PMS was implemented, with a substantial increase occurring within the performance planning/goal-setting phase, followed by a slight decay over the following year, but still ending higher than the pre-PMS baseline level. Substantial increases in ratings of satisfaction and cooperation with ones supervisor were found with the introduction of the PMS for low performers (particularly following the performance planning/goal-setting phase). In contrast, however, high performers had high base-line levels of these attitudes toward supervision, followed by substantial drops immediately after receiving appraisal and bonus pay distributions. As anticipated, most participants in this study had expected a performance rating higher than they actually received, and most of these individuals made external attributions for the rating discrepancy, blaming either their supervisor, the organization, or the PMS itself. However, neither having received a lower-than-expected appraisal rating nor having made external attributions for a lower-than-expected rating were related to changes in attitudes or self-reported effort.

Transfer of Management Training From Alternative Perspectives

One hundred seven management training evaluations were meta-analyzed to compare effect sizes for the transfer of managerial training derived from different rating sources (self, superior, peer, and subordinate) and broken down by both study- and training-related variables. For studies as a whole, and interpersonal management skills training studies in particular, transfer effects based on trainees self-ratings, and to a lesser extent ratings from their superiors, were largest and most varied across studies. In contrast, transfer effects based on peer ratings, and particularly subordinate ratings, were substantially smaller and more homogeneous. This pattern was consistent across different sources of studies, features of evaluation design, and within a subset of 14 studies that each included all 4 rating sources. Across most rating sources, transfer of training was greatest for studies conducted in nonmilitary settings, when raters were likely to have known whether the manager being rated had attended training, when criteria were targeted to training content, when training content was derived from an analysis of tasks and skill requirements, and when training included opportunities for practice. (PsycINFO Database Record (c) 2009 APA, all rights reserved).

Gilding the Outcome by Tarnishing the Past Inflationary Biases in Retrospective Pretests

We tested for inflationary bias introduced through retrospective pretests by analyzing traditional pretest, retrospective pretest, and posttest evaluation data collected on a first-line supervisory leadership training program, involving 196 supervisors and their subordinates, across 17 organizational settings. Retrospective pretest ratings by both trained (supervisors) and untrained (subordinates) respondents were significantly lower than traditional pretest ratings, resulting in substantially inflated training effect sizes when posttests were compared with retrospective pretests rather than with traditional pretests. Further analysis revealed evidence of both respondents application of an implicit theory of change (i.e., assumption that posttraining scores should generally be higher than pretraining scores) and a tendency to rate their own improvement as greater than that of others. Implications for program evaluation are discussed.

Transparency of Teacher Expectancies across Language, Cultural Boundaries.

Abstract This study investigated the detection of teacher expectancy effects from extremely brief samples of their nonverbal behavior. Four groups of judges, ranging from 10-year-old students to experienced teachers, rated 10-s samples of teacher behavior when the teachers were talking about and talking to a high-expectancy student and a low-expectancy student. Three types of clips were used for each condition: audio only, video only, and audio plus video. The study was focused exclusively on nonverbal communication: all comprehension of verbal content was eliminated, because the video-taped teachers spoke Hebrew, and the New Zealand judges spoke English. All groups of judges showed considerable detection from the “talking to student” clips, rating the (unseen) high-expectancy student as a better learner and as better liked by the teacher than the (also unseen) low-expectancy student. The judges based their guesses on teachers facial expressions and body language. As hypothesized, no detection was made b...

Supervisory support for the transfer of team-building training

Relationships between specific supervisory support behaviours and training transfer among subordinates were investigated in the context of a team-building training programme. Ninety-nine employees who attended outdoor team-building training and their supervisors rated the extent to which trainees transferred each of twenty-one trained skills to the work-place. Additionally, trainees responded to an open-ended question regarding what, if anything, they applied from the training to the work-place. Ratings of transfer, provided by both trainees and their supervisors, and coded responses to the open-ended question indicated that: (1) most trainees transferred only a small to moderate amount of skills from the outdoor training; (2) the amount of training transfer varied across trainees; and (3) skills reportedly transferred the most concerned seeking and listening to the views and ideas of other team members. Trainees also rated whether their supervisors displayed each of twenty-seven specific supervisory supp...

Congruence between theory and practice in management training needs analysis

Interviews were conducted with the human resource directors of ninety–nine randomly selected medium–large enterprises in New Zealand, to explore the gap between the principal framework of training needs analysis found in the academic literature and the methods actually used by organizations to make management training decisions. Content analyses revealed a lack of congruence between techniques recommended by training theorists and procedures used by organizations to identify their training needs. Most organizations adopted informal, relatively ad hoc approaches to making training decisions. Exploration of the methods used to make training decisions sugested that it would be beneficial to expand the training heeds analysis framework to take into account previously ignored aspects of the decision–making process. Implications for the development and application of prescriptive formulations of training needs analysis are highlighted.

Past Experience versus Situational Employment: Interview Questions in a New Zealand Social Service Agency

This study compared the validities of the past-experience and situational structured employment interview questions using a two-group, experimental design. Subjects were 130 social workers, purpose...

Industrial and organisational psychology training in Australia and New Zealand

The purpose of this article is to provide an overview and comparison of the professional training of industrial and organisational (I/O) psychologists offered in Australia and New Zealand. First, the context in which the programs operate and the training pathways for an I/O psychologist in each country are described. Second, the results of a survey of program staff concerning the types of programs offered, program content, the perceived balance in programs between scientist and practitioner, program staffing, student enrolments and student employment, are presented. Data were obtained from the complete population of I/O postgraduate course coordinators in Australia and New Zealand (nxa0=xa012 Australian, nxa0=xa04 New Zealanders). Using descriptive statistics the current status of I/O training in the two countries is reported. Conclusions are drawn about the similarities and differences between the two countries.

Functions and Implementation of Performance Appraisal Systems in New Zealand Organizations

We conducted a survey of 89 randomly selected New Zealand organizations with at least 300 employees to determine 1) what purposes performance appraisal systems serve, 2) characteristics of those systems, and 3) any difficulties organizations have had in implementing them. Human resource directors indicated that appraisal information was used most often to give individual performance feedback. Links between appraisals and pay were widespread, but generally indirect and informal. In contrast to the emphasis of past research on rating errors and cognitive processes in performance evaluations , primary difficulties identified by these human resource directors were 1) managers failing to discuss and improve performance problems with subordinates, and 2) the setting of vague performance objectives. The most frequently cited change which organizations made and were considering making was simplifying appraisal forms and procedures. Ways in which organizations can improve how appraisal systems meet their primary o...