Aaron C. Elkins

Embodied Conversational Agent-Based Kiosk for Automated Interviewing

We have created an automated kiosk that uses embodied intelligent agents to interview individuals and detect changes in arousal, behavior, and cognitive effort by using psychophysiological information systems. In this paper, we describe the system and propose a unique class of intelligent agents, which are described as Special Purpose Embodied Conversational Intelligence with Environmental Sensors (SPECIES). SPECIES agents use heterogeneous sensors to detect human physiology and behavior during interactions, and they affect their environment by influencing human behavior using various embodied states (i.e., gender and demeanor), messages, and recommendations. Based on the SPECIES paradigm, we present three studies that evaluate different portions of the model, and these studies are used as foundational research for the development of the automated kiosk. The first study evaluates human-computer interaction and how SPECIES agents can change perceptions of information systems by varying appearance and demeanor. Instantiations that had the agents embodied as males were perceived as more powerful, while female embodied agents were perceived as more likable. Similarly, smiling agents were perceived as more likable than neutral demeanor agents. The second study demonstrated that a single sensor measuring vocal pitch provides SPECIES with environmental awareness of human stress and deception. The final study ties the first two studies together and demonstrates an avatar-based kiosk that asks questions and measures the responses using vocalic measurements.

Border Security Credibility Assessments via Heterogeneous Sensor Fusion

A heterogeneous network of sensors that measure physiological and behavioral indicators of arousal, cognitive effort, and stress can facilitate deception detection, while limiting subjective judgments and improper profiling.

The INTERSPEECH 2016 Computational Paralinguistics Challenge: Deception, Sincerity & Native Language

The INTERSPEECH 2016 Computational Paralinguistics Challenge addresses three different problems for the first time in research competition under well-defined conditions: classification of deceptive vs. non-deceptive speech, the estimation of the degree of sincerity, and the identification of the native language out of 11 L1 classes of English L2 speakers. In this paper, we describe these sub-challenges, their conditions, and the baseline feature extraction and classifiers, as provided to the participants.

Are users threatened by credibility assessment systems

Despite the improving accuracy of agent-based expert systems, human expert users aided by these systems have not improved their accuracy. Self-affirmation theory suggests that human expert users could be experiencing threat, causing them to act defensively and ignore the systems conflicting recommendations. Previous research has demonstrated that affirming an individual in an unrelated area reduces defensiveness and increases objectivity to conflicting information. Using an affirmation manipulation prior to a credibility assessment task, this study investigated if experts are threatened by counterattitudinal expert system recommendations. For our study, 178 credibility assessment experts from the American Polygraph Association (n = 134) and the European Unions border security agency Frontex (n = 44) interacted with a deception detection expert system to make a deception judgment that was immediately contradicted. Reducing the threat prior to making their judgments did not improve accuracy, but did improve objectivity toward the system. This study demonstrates that human experts are threatened by advanced expert systems that contradict their expertise. As more and more systems increase integration of artificial intelligence and inadvertently assail the expertise and abilities of users, threat and self-evaluative concerns will become an impediment to technology acceptance.

A Rigidity Detection System for Automated Credibility Assessment

Credibility assessment is an area in which information systems research can make a major impact. This paper reports on two studies investigating a system solution for automatic, noninvasive detection of rigidity for automated interviewing. Kinesic rigidity has long been a phenomenon of interest in the credibility assessment literature, but until now was infeasible as a veracity indicator in practical use cases. An initial study unexpectedly revealed the occurrence of rigidity in a highly controlled concealed information test setting, prompting the design and implementation of an automated rigidity detection system for interviewing. A unique experimental evaluation supported the system concept. The results of the second study confirmed the kinesic rigidity found in the first, and provided further theoretical insights explaining the rigidity phenomenon. Although additional research is needed, the evidence from this investigation suggests that credibility assessment can benefit from a rigidity detection system.

Which Spoken Language Markers Identify Deception in High-Stakes Settings? Evidence From Earnings Conference Calls

Quarterly conference calls where corporate executives discuss earnings that are later found to be misreported offer an excellent test bed for determining if automated linguistic and vocalic analysis tools can identify potentially fraudulent utterances in prepared versus unscripted remarks. Earnings conference calls from one company that restated their financial reports and were accused of making misleading statements were annotated as restatement-relevant (or not) and as prepared (presentation) or unprepared (Q&A) responses. We submitted more than 1,000 utterances to automated analysis to identify distinct linguistic and vocalic features that characterize various types of utterances. Restatement-related utterances differed significantly on many vocal and linguistic dimensions. These results support the value of language and vocal features in identifying potentially fraudulent utterances and suggest important interplay between utterances that are unscripted responses rather than rehearsed statements.

Potential noncontact tools for rapid credibility assessment from physiological and behavioral cues

Credibility assessment is a perennial and increasingly urgent problem in light of escalating international security threats. New tools such are needed for rapid, noninvasive and possibly unobtrusive detection of deception and hostile intent. This paper reports five novel instrumented approaches to credibility assessment being investigated in a multi-institution research program. These instruments do not require physical contact with humans and can reliably measure veracity from physiological and behavioral indicators. Data were collected via an experiment, which required participants to commit a mock crime and then be interviewed by a trained interviewer. During and following the interviews, multiple instruments measured physiological, cognitive and behavioral responses of interviewees to determine which automatable features accurately differentiate truthtellers from deceivers. Details concerning the instruments and the experimental method used to test them are shared.

Predicting Users' Perceived Trust in Embodied Conversational Agents Using Vocal Dynamics

One of the major challenges facing neurophysiological HCI design is to determine the systems and sensors that accurately and noninvasively measure human cognitive processes. Specifically, it is a significant undertaking to integrate sensors and measurements into an information system and accurately measure and interpret the human state. Using an experimental design this study explores the use of unobtrusive sensors based on behavioral and neurophysiological responses to predict human trust using the voice. Participants (N=88) completed a face-to-face interview with an Embodied Conversational Agent (ECA) and reported their perceptions of the ECA. They reported three dimensions consistent with the Mayer model of perceived trustworthiness. During the interaction, the demeanor and gender of the avatar was manipulated and these manipulations affected the reported measures of trustworthiness. Using growth modeling and multilevel analysis of covariance methods, a model was developed that could predict human trust during the interaction using the voice, time, and demographics.

Sociocultural intelligence and intelligent agents

The use of predictive analytics to model terrorist rhetoric is highly instrumental in developing a strategy to deter terrorism. Traditional (e.g. Cold-War) deterrence methods are ineffective with terrorist groups such as al Qaida. Terrorists typically regard the prospect of death or loss of property as acceptable consequences of their struggle. Deterrence by threat of punishment is therefore fruitless. On the other hand, isolating terrorists from the community that may sympathize with their cause can have a decisive deterring outcome. Without the moral backing of a supportive audience, terrorism cannot be successfully framed as a justifiable political strategy and recruiting is curtailed. Ultimately, terrorism deterrence is more effectively enforced by exerting influence to neutralize the communicative reach of terrorists.

Pitch detection algorithms modifications and implementations towards automated vocal analysis

Discriminating between deceit and truth is a significant security challenge in a variety of situations, including border crossings, job interviews, flight passenger screenings, and police interviews. Previous research indicates that some features of vocal speech, e.g., fundamental frequency, are related to human emotion and stress levels making them applicable deception detection. This paper focuses on voice and speech feature extraction using advanced signal processing methodology. These generated speech features are used to submit data mining algorithms for classifying deception. The result of this paper is expected to be directly applied to the deception detection system.