Matthew Marge

Using the Amazon Mechanical Turk for transcription of spoken language

We investigate whether Amazons Mechanical Turk (MTurk) service can be used as a reliable method for transcription of spoken language data. Utterances with varying speaker demographics (native and non-native English, male and female) were posted on the MTurk marketplace together with standard transcription guidelines. Transcriptions were compared against transcriptions carefully prepared in-house through conventional (manual) means. We found that transcriptions from MTurk workers were generally quite accurate. Further, when transcripts for the same utterance produced by multiple workers were combined using the ROVER voting scheme, the accuracy of the combined transcript rivaled that observed for conventional transcription methods. We also found that accuracy is not particularly sensitive to payment amount, implying that high quality results can be obtained at a fraction of the cost and turnaround time of conventional methods.

Miscommunication Recovery in Physically Situated Dialogue

We describe an empirical study that crowdsourced human-authored recovery strategies for various problems encountered in physically situated dialogue. The purpose was to investigate the strategies that people use in response to requests that are referentially ambiguous or impossible to execute. Results suggest a general preference for including specific kinds of visual information when disambiguating referents, and for volunteering alternative plans when the original instruction was not possible to carry out.

Comparing Heads-up, Hands-free Operation of Ground Robots to Teleoperation

Today, most commercially available UGVs use teleoperation for control. Under teleoperation, users’ hands are occupied holding a handheld controller to operate the UGV, and their attention is focused on what the robot is doing. In this paper, we propose an alternative called Heads-up, Hands-free Operation, which allows an operator to control a UGV using operator following behaviors and a gesture interface. We explore whether Heads-up, Hands-free Operation is an improvement over teleoperation. In a study of 30 participants, we found that when operators used these modes of interaction, they performed missions faster, they could recall their surroundings better, and they had a lower cognitive load than they did when they teleoperated the robot. Keywords-human-robot interaction; person detection; person following; gesture interaction; teleoperation

The Future of Human Robot Teams in the Army: Factors Affecting a Model of Human-System Dialogue Towards Greater Team Collaboration

Understanding of intent is one of the most complex traits of highly efficient teams. Combining elements of verbal and non-verbal communication along with shared mental models about mission goals and team member capabilities, intent requires knowledge about both task and teammate. Beginning with the traditional models of communication, accounting for teaming factors, such as situation awareness, and incorporating the sensing, reasoning, and tactical capabilities available via autonomous systems, a revised model of team communication is needed to accurately describe the unique interactions and understanding of intent which will occur in human-robot teams. This paper focuses on examining the issue from a system capability viewpoint, identifying which system capabilities can mirror the abilities of humans through the sensor and computing strengths of autonomous systems, thus creating a team environment which is robust and adaptable while maintaining focus on mission goals.

Towards evaluating recovery strategies for situated grounding problems in human-robot dialogue

Robots can use information from their surroundings to improve spoken language communication with people. Even when speech recognition is correct, robots face challenges when interpreting human instructions. These situated grounding problems include referential ambiguities and impossible-to-execute instructions. We present an approach to resolving situated grounding problems through spoken dialogue recovery strategies that robots can invoke to repair these problems. We describe a method for evaluating these strategies in human-robot navigation scenarios.

Exploring Variation of Natural Human Commands to a Robot in a Collaborative Navigation Task

Robot-directed communication is variable, and may change based on human perception of robot capabilities. To collect training data for a dialogue system and to investigate possible communication changes over time, we developed a Wizard-of-Oz study that (a) simulates a robot’s limited understanding, and (b) collects dialogues where human participants build a progressively better mental model of the robot’s understanding. With ten participants, we collected ten hours of human-robot dialogue. We analyzed the structure of instructions that participants gave to a remote robot before it responded. Our findings show a general initial preference for including metric information (e.g., move forward 3 feet) over landmarks (e.g., move to the desk) in motion commands, but this decreased over time, suggesting changes in perception.

Assessing Agreement in Human-Robot Dialogue Strategies: A Tale of Two Wizards

The Wizard-of-Oz (WOz) method is a common experimental technique in virtual agent and human-robot dialogue research for eliciting natural communicative behavior from human partners when full autonomy is not yet possible. For the first phase of our research reported here, wizards play the role of dialogue manager, acting as a robot’s dialogue processing. We describe a novel step within WOz methodology that incorporates two wizards and control sessions: the wizards function much like corpus annotators, being asked to make independent judgments on how the robot should respond when receiving the same verbal commands in separate trials. We show that inter-wizard discussion after the control sessions and the resolution with a reconciled protocol for the follow-on pilot sessions successfully impacts wizard behaviors and significantly aligns their strategies. We conclude that, without control sessions, we would have been unlikely to achieve both the natural diversity of expression that comes with multiple wizards and a better protocol for modeling an automated system.

Introduction to the Special Issue on Machine Learning for Multiple Modalities in Interactive Systems and Robots

This special issue highlights research articles that apply machine learning to robots and other systems that interact with users through more than one modality, such as speech, gestures, and vision. For example, a robot may coordinate its speech with its actions, taking into account (audio-)visual feedback during their execution. Machine learning provides interactive systems with opportunities to improve performance not only of individual components but also of the system as a whole. However, machine learning methods that encompass multiple modalities of an interactive system are still relatively hard to find. The articles in this special issue represent examples that contribute to filling this gap.

Evaluating Robot Behavior in Response to Natural Language

Human-robot teaming can be improved if a robot»s actions meet human users» expectations. The goal of this research is to determine what variations of robot actions in response to natural language match human judges» expectations in a series of tasks. We conducted a study with 21 volunteers that analyzed how a virtual robot behaved when executing eight navigation instructions from a corpus of human-robot dialogue. Initial findings suggest that movement more accurately meets human expectation when the robot (1) navigates with an awareness of its environment and (2) demonstrates a sense of self-safety.

How We Talk with Robots: Eliciting Minimally-Constrained Speech to Build Natural Language Interfaces and Capabilities

Industry, military, and academia are showing increasing interest in collaborative human-robot teaming in a variety of task contexts. Designing effective user interfaces for human-robot interaction is an ongoing challenge, and a variety of single and multiple-modality interfaces have been explored. Our work is to develop a bi-directional natural language interface for remote human-robot collaboration in physically situated tasks. When combined with a visual interface and audio cueing, we intend for the natural language interface to provide a naturalistic user experience that requires little training. Building the language portion of this interface requires first understanding how potential users would speak to the robot. In this paper, we describe our elicitation of minimally-constrained robot-directed language, observations about the users’ language behavior, and future directions for constructing an automated robotic system that can accommodate these language needs.