Mary Ann Hanson

An examination of the comparative reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales

This laboratory research compared the reliability, validity, and accuracy of a computerized adaptive rating scale (CARS) format and 2 relatively common and representative rating formats. The CARS is a paired-comparison rating task that uses adaptive testing principles to present pairs of scaled behavioral statements to the rater to iteratively estimate a ratees effectiveness on 3 dimensions of contextual performance. Videotaped vignettes of 6 office workers were prepared, depicting prescripted levels of contextual performance, and 112 subjects rated these vignettes using the CARS format and one or the other competing format. Results showed 23%-37% lower standard errors of measurement for the CARS format. In addition, validity was significantly higher for the CARS format (d = .18), and Cronbachs accuracy coefficients showed significantly higher accuracy, with a median effect size of .08. The discussion focuses on possible reasons for the results.

Selecting Pilots With Crew Resource Management Skills

For years, pilot selection has focused primarily on the identification of individuals with superior flying skills and abilities. More recently, the aviation community has become increasingly aware that successful completion of a flight or mission requires not only flying skills but the ability to work well in a crew situation. This project involved development and validation of a crew resource management (CRM) skills test for Air Force transport pilots. A significant relation was found between the CRM skills test and behavior-based ratings of aircraft commander CRM performance, and the implications of these findings for CRM-based selection and training are discussed.

Progress Toward Understanding the Structure and Determinants of Job Performance: A Focus on Task and Citizenship Performance

Job performance criterion models have become more popular and more often investigated. The current research adds to this literature by testing performance models focusing on ability and personality predictors of task and citizenship job performance. Air traffic controllers (N = 928) completed cognitive ability and personality predictor measures, were administered a performance test assessing two possible mediators, and were evaluated using supervisor and peer ratings of two task performance, one citizenship performance, and an overall performance dimension. Path analyses largely confirmed that ability predicts primarily task performance and personality predicts primarily citizenship performance. The mediation related to task knowledge and skill on the task performance side was confirmed, and the overall fit of the performance models was quite good. Discussion focuses on how the present research furthers progress toward mapping relations among individual differences and individual performance constructs, beyond using overall performance as the endogenous performance variable.