Hall P. Beck

The role of trust in automation reliance

A recent and dramatic increase in the use of automation has not yielded comparable improvements in performance. Researchers have found human operators often underutilize (disuse) and overly rely on (misuse) automated aids (Parasuraman and Riley, 1997). Three studies were performed with Cameron University students to explore the relationship among automation reliability, trust, and reliance. With the assistance of an automated decision aid, participants viewed slides of Fort Sill terrain and indicated the presence or absence of a camouflaged soldier. Results from the three studies indicate that trust is an important factor in understanding automation reliance decisions. Participants initially considered the automated decision aid trustworthy and reliable. After observing the automated aid make errors, participants distrusted even reliable aids, unless an explanation was provided regarding why the aid might err. Knowing why the aid might err increased trust in the decision aid and increased automation reliance, even when the trust was unwarranted. Our studies suggest a need for future research focused on understanding automation use, examining individual differences in automation reliance, and developing valid and reliable self-report measures of trust in automation.

Predicting Misuse and Disuse of Combat Identification Systems

Two combat identification systems have been designed to reduce fratricide by providing soldiers with the ability to “interrogate” a potential target by sending a microwave or laser signal that, if returned, identifies the target as a “friend.” Ideally, gunners will appropriately rely on these automated aids, which will reduce fratricide rates. However, past research has found that human operators underutilize (disuse) and overly rely on (misuse) automated systems (cf. Parasuraman & Riley, 1997). The purpose of this laboratory study was to simultaneously examine misuse and disuse of an automated decision-making aid at varying levels of reliability. With or without the aid of an automated system that is correct about 90%, 75%, or 60% of the time, 91 college students viewed 226 slides of Fort Sill terrain and indicated the presence or absence of camouflaged soldiers. Regardless of the reliability of the automated aid, misuse was more prevalent than disuse, F(1, 65) = 31.43, p < .01; p = .27 for misuse, p = .13 for disuse. Results are interpreted within a general framework of automation use (Dzindolet, Beck, Pierce, & Dawe, 2001).

Establishing an Early Warning System: Predicting Low Grades in College Students from Survey of Academic Orientations Scores

Counselors, faculty, and student personnel specialists are often unaware that college students are experiencing serious academic or adjustment difficulties until it is too late to rectify the problem. Most universities would benefit from an early warning system that detects at-risk students before performance or social problems jeopardize their college careers. This investigation demonstrated that scores from the Survey of Academic Orientations (SAO) were predictive of first-semester freshmen grades. Subsequent analysis showed that the SAO significantly improved the prediction of grade point averages, after taking the effects of Scholastic Assessment Test scores and high school percentage rank into consideration. The SAO gives educators a new early warning device, a way to identify those undergraduates most at risk of receiving poor grades. The next steps in the research process are to: (1) assess the relationship of SAO scores to other important academic indexes, such as retention and student stress, and (2) determine if furnishing counselors and other college personnel with SAO scores is of therapeutic value.

The College Persistence Questionnaire: Development and Validation of an Instrument That Predicts Student Attrition

The investigators reviewed the retention literature and developed a 53-item questionnaire and tested its validity. Component analysis of the responses of 2,022 students at four schools yielded six reliable factors: Institutional Commitment, Degree Commitment, Academic Integration, Social Integration, Support Services Satisfaction, and Academic Conscientiousness. A second study on 283 first-semester freshmen examined whether factor scores predicted which students returned for their sophomore year. Logistic regression found that three factors were statistically significant predictors of enrollment status, after controlling for high school class rank and standardized test scores: Institutional Commitment, Academic Integration, and Academic Conscientiousness. Strategies are provided for making use of scores based on differences between institutions and between individual students.

Automation usage decisions: controlling intent and appraisal errors in a target detection task

Background: It was proposed that misuse and disuse often occur because operators (a) cannot determine if automation or a nonautomated alternative maximizes the likelihood of task success (appraisal errors) or (b) know the utilities of the options but disregard this information when deciding to use automation (intent errors). Objective: This investigation assessed the effectiveness of performance feedback, a procedure developed to attenuate appraisal errors, and scenario training, an intervention designed to decrease intent errors. Methods: Operators given feedback were told how many errors they and an automated device made on a target detection task. Scenario training took operators through the thought processes of optimal decision makers after the utilities of the automated and nonautomated alternatives had been determined. Following 200 training trials, participants chose between relying on their observations or an automated device. Results: There was little misuse, but disuse rates were high (84%) among operators receiving neither feedback nor scenario training. Operators paired with a more accurate machine and given feedback made approximately twice as many errors as the automated device. Nevertheless, intent errors were commonplace; 55% of the operators who received feedback without scenario training did not rely on automation. Feedback effectiveness was enhanced when used in conjunction with scenario training; the disuse rate decreased to 29%. Conclusion: A combination of feedback and scenario training was more effective in mitigating disuse than either intervention used in isolation. Application: An important application of these findings is that operator training programs should incorporate techniques to control intent and appraisal errors.

The Relations of Learning and Grade Orientations to Academic Performance

This investigation assessed the roles of learning orientation (LO) and grade orientation (GO) in academic performance. Most important, we found that GO was negatively correlated with grade point average (GPA) and General Psychology test scores. Correlations of LO scores with the academic performance measures were not significant. The poor academic performance of students with high GO scores can be partially attributed to lower Scholastic Aptitude Tests (SAT) scores. Even after the effects of SAT were controlled, GO scores were negatively related to GPA and psychology test scores. These findings demonstrate the need for investigations to determine the effects of grading practices on students with high and low grade orientations.

Misuse and Disuse of Automated AIDS

A general model of automation use, which proposes that cognitive, social, and motivational processes may lead to productivity loss of human-computer teams is briefly described and data from four experiments testing aspects of the model are presented. Automation use is determined, in part, by the perceived utility of the aid, which is the outcome of a comparison process between the human operators ability and the reliability of the aid. Perceived utility, however, is affected by a bias towards automation and by self-serving biases.

The Relationship of High School Psychology and Natural Science Courses to Performance in a College Introductory Psychology Class

This study assessed the relationship between high school psychology and natural science classes and subsequent performance in a college introductory psychology course. After removing the variability due to SAT scores, both high school psychology and high school natural science courses were significantly related to precourse knowledge of psychology. High school psychology was not significantly related to final grades. However, students with strong backgrounds in high school natural sciences obtained higher final grades than students with less preparation.

Development and Validation of Scores on a Measure of Six Academic Orientations in College Students

This article describes the development and score validation of a 36-item measure of six academic orientations in college students: structure dependence, creative expression, reading for pleasure, academic efficacy, academic apathy, and mistrust of instructors. Results from three studies indicate that the measuring instrument, the Survey of Academic Orientations (SAO), has six factorially distinct scales (Study 1) whose scores are stable across different semesters, yielding test-retest coefficients that range from .63 to .86 (Study 2). Also, each of the six scales relates in expected ways to basic personality traits, yielding validity coefficients of .30 to .69 (Study 3). Scores on the six scales are internally consistent, yielding coefficients alpha that range from .59 to .85 (Studies 1-3). Scale scores and a summative score of all 36 items, called the Adaptiveness index, are examined for their potential in predicting a variety of important student outcomes.

Strict and Lenient Grading Scales: How Do They Affect the Performance of College Students with High and Low SAT Scores?.

This study examined the relationship between strict and lenient grading scales, Scholastic Aptitude Test (SAT) scores, and performance on tests administered in an undergraduate Educational Psychology class. Analysis of variance (ANOVA) revealed that students with relatively higher SATs obtained better test scores than students with lower SATs and that students evaluated with the strict scale earned higher test scores than students graded with the lenient scale. The Grading Scale × SAT interaction was also significant. The lenient scale impaired the performance of students with low SAT scores but did not significantly affect the performance of students with moderate or high SAT scores. Other Educational Psychology students were administered the LOGO II Scale to determine why the grading scale variable had its greatest impact on the achievement of students with low SAT scores. A significant negative correlation was found, indicating that students with lower SAT scores are more grade-oriented than students with higher SAT scores.