Thomas R. Carretta

Development of a Cyber/Information Technology Knowledge Test for Military Enlisted Technical Training Qualification

An Armed Services Vocational Aptitude Battery (ASVAB) Review Panel, with expertise in personnel selection, job classification, psychometrics, and cognitive psychology developed recommendations for changes to the military enlistment test battery. One recommendation was to develop and evaluate a test of cyber/information and communications technology literacy to supplement current ASVAB content. This article summarizes a multiphased Cyber Test development process: (a) a review of information/computer technology literacy definitions and measures, (b) development and pilot testing of a cyber knowledge measure, (c) validation of test scores against final school grades (FSGs) for selected technical training courses, (d) development of an operational reporting metric and subgroup norms, and (e) examination of construct validity. Results indicate the Cyber Test has predictive validity versus technical training school grades and incremental validity comparable to the ASVAB technical knowledge tests when used with the ASVAB Armed Forces Qualification Test (AFQT) verbal/math composite as a baseline.

Consistency of the Relations of Cognitive Ability and Personality Traits to Pilot Training Performance

The predictive validity of cognitive ability and personality traits was examined in large samples of U.S. Air Force pilot trainees. Criterion data were collected between 1995 and 2008 from 4 training bases across 3 training tracks. Analyses also examined consistency in pilot aptitude and training outcomes. Results were consistent with previous research indicating cognitive ability is the best predictor of pilot training performance. There were few differences across training tracks, bases, and years, and none was large. Overall, results illustrated the consistency of the quality of pilot trainees as assessed by cognitive ability and personality trait measures, and the consistency of these measures in predicting training performance over time. This consistency results in a more stable training system, enabling greater efficiency and effectiveness.

Early Identification of Unmanned Aircraft Pilots Using Measures of Personality and Aptitude

This study evaluated the extent to which personality measures (Self-Description Inventory+) could improve prediction of remotely piloted aircraft (RPA) training outcomes, beyond currently used cognitive, psychomotor, and aviation knowledge and experience predictors included in the U.S. Air Force (USAF) Pilot Candidate Selection Method 2.0. Participants were 170 USAF officers with RPA Initial Flight Screening (RFS) outcomes and 110 USAF officers with RPA Instrument Qualification (RIQ) outcomes. Both RFS and RIQ emphasize development of skills traditionally required for manned aircraft pilots, also considered essential for USAF RPA pilots. Results showed significant negative relationships between the Big Five personality trait of Openness and several RPA training outcomes, and significant incremental variance explained by Openness beyond other predictors. Additionally, RFS and RIQ training outcomes were generally predicted by measures traditionally used for selection into manned aircraft pilot training courses and careers. Together, these findings suggest that personality, specifically Openness scores, and scores on traditional pilot selection measures, can serve as useful indicators in the early identification of RPA pilot talent.

Evaluation of Tests of Perceptual Speed/Accuracy and Spatial Ability for Use in Military Occupational Classification

With the exception of Assembling Objects (AO), a spatial ability test used only by the Navy in enlisted occupational classification, the Armed Services Vocational Aptitude Battery (ASVAB) is academic and knowledge-based, somewhat limiting its utility for occupational classification. This article presents the case for integrating the AO test into military classification composites and for expanding the breadth of ASVAB content by including a former ASVAB speed/accuracy test, Coding Speed (CS). Empirical evidence is presented that shows AO and CS (a) increment the validity of the ASVAB in predicting training grades for a broad array of occupations, (b) reduce adverse impact defined as test score barriers for women and minorities, and (c) improve classification in terms of matching recruits to occupations. Some cognitive theory is presented to support AO and CS, as well as nonverbal reasoning and working memory tests for inclusion in or adjuncts to the ASVAB.

Basic Attributes Test (BAT) Retest Performance

The Basic Attributes Test (BAT) contributes to a U.S. Air Force pilot selection composite known as the Pilot Candidate Selection Method (PCSM). When PCSM was operationally implemented in 1993, no retests were permitted on the BAT. To determine the effects of retesting on mean score change and reliability, the BAT was administered to 477 college students who were then retested after 2 weeks, 3 months, or 6 months. Several important findings were observed. First, about 70% of the students exhibited score improvement on retest, regardless of length of retest interval. Those who performed poorly on the 1st test generally exhibited larger improvements than those who performed well on the 1st test. Second, practice effects diminished as the length of the retest interval increased. For a 6-month retest interval, it was expected that the mean increase in PCSM scores would be about 6 percentile points. The results suggest that BAT retests could be permitted no less than 6 months after initial testing. Third, and very important, BAT scores demonstrated acceptable reliability. The reliability of the psychomotor composite ranged from .775 to .800, and the reliabilities for the other subtests ranged from .474 to .871.

Predictive Validity of UAS/RPA Sensor Operator Training Qualification Measures

U.S. Air Force sensor operators (SOs) are enlisted aviators who work side-by-side with unmanned aerial systems/remotely piloted aircraft (UAS/RPA) pilots, providing assistance with all aspects of aircraft employment and sensor management. SO training qualification includes medical, citizenship, and security standards and aptitude requirements. The current study examined the validity of the Armed Services Vocational Aptitude Battery (ASVAB) for predicting grades of students in three SO courses. The ASVAB composites used for SO training qualification (General and Electronics) demonstrated good predictive validity for all three courses (corrected for range restriction and criterion unreliability): Basic Sensor Operator Course, n = 461, r = .541 and .535; MQ-1 Initial Qualification/Requalification Training, n = 430, r = .583 and .553; MQ-9 Initial Qualification/Requalification Training, n = 249, r = .357 and .334). Although current selection methods are effective, based on results of UAS/RPA job/task analyses, the Air Force is examining the utility of other measures to supplement the ASVAB.

Aptitude and Trait Predictors of Manned and Unmanned Aircraft Pilot Job Performance

Previous research has shown that the same aptitude and trait measures that predict success in U.S. Air Force (USAF) manned aircraft pilot training predict remotely piloted aircraft (RPA) pilot training outcomes with generally similar levels of validity ( Carretta, 2013; Rose, Barron, Carretta, Arnold, & Howse, 2014). However, because USAF RPA pilots initially train in manned aircraft, validation of aptitude and traits predictive of RPA pilot success has thus far been limited to RPA pilot training outcomes that actually require manned flight. Hence, thus far, there has been no basis for determining the aptitudes and traits predictive of success in environments in which pilots actually fly RPAs. To address this gap, the present study evaluated preaccession trait (Big Five personality domains) and aptitude (spatial, quantitative, and aviation knowledge) measures as predictors of manned and unmanned aircraft pilot performance on-the-job, as measured by supervisor and senior rater stratification on 3 years of Officer Performance Reports (OPRs). Results were generally consistent in showing that the same aptitudes, knowledge, and personality traits that predict successful job performance for manned aircraft pilots also predict successful job performance for RPA pilots. However, results also showed preaccession aviation knowledge to be a stronger predictor of job performance for RPA pilots than for manned aircraft pilots. These findings and their implications for attracting and selecting RPA pilots are discussed.

Predictive Validity of Spatial Ability and Perceptual Speed Tests for Aviator Training

ABSTRACT Objective: To present a multistudy examination of spatial ability and perceptual speed tests as predictors of Air Force aviator academic and flying training outcomes. Background: Whereas verbal and math tests are commonly used in academic and pre-employment selection, spatial ability and perceptual speed testing are less common. Spatial ability and perceptual speed tests have been a mainstay in U.S. Air Force aviation training selection batteries for decades, as part of the process for selecting aviation training candidates. Method and Results: Factor analysis (Study 1) of spatial, perceptual speed, and academic aptitude tests shows a 2-factor solution. Meta-analysis results (Study 2) of the predictive validity of spatial and perceptual speed tests relative to verbal, math, and technical knowledge tests show spatial ability, aviation knowledge, and perceptual speed tests are better significant predictors of aviator flying performance, whereas academic aptitude best predicts classroom performance. Incremental validity analyses (Study 3) of spatial and perceptual speed above academic and technical knowledge for academic and pilot trainee flying performance were performed. Although spatial ability lacks incremental validity over academic and technical aptitude, perceptual speed has incremental validity above academic and technical aptitude for predicting pilot trainee flying performance. Conclusion: Perceptual speed, an aspect of spatial ability, can provide incremental predictive validity beyond academic and technical knowledge measures alone for applied science, technology, engineering, and mathematics fields like military aviation.

Validity of Spatial Ability Tests for Selection into STEM (Science, Technology, Engineering, and Math) Career Fields: The Example of Military Aviation

Quantitative and verbal aptitude tests are widely used in the context of student admissions and pre-employment screening. In contrast, there has been “contemporary neglect” of the potential for organizations to use spatial abilities testing to make informed decisions on candidates’ success in educational settings (Wai J, Lubinski D, Benbow CP, J Educ Psychol 101:817–835, 2009) and the workplace. We begin with a review of the research literature on the validity of spatial ability tests for predicting performance in STEM fields (e.g., engineering, surgery, mathematics, aviation). We address the controversy regarding the extent to which spatial abilities provide incremental validity beyond traditional measures of academic aptitude. We then present results from over a decade of U.S. Air Force research that has examined the validity of spatial ability tests relative to verbal and quantitative measures for predicting aircrew and pilot training outcomes. Finally, consistent with meta-analyses examining pilot training outcomes across several countries (e.g., Martinussen M, Int J Aviat Psychol 6:1–20, 1996), we present results showing spatial ability tests add substantive incremental validity to measures of numerical and verbal ability for predicting pilot training outcomes. Hence, organizations that fail to include spatial testing in screening may be overlooking many individuals most likely to excel in STEM fields. We conclude with a discussion of potential challenges associated with spatial ability testing and provide practical recommendations for organizations considering implementing spatial ability testing in student admissions or personnel selection.