Elaine Marsh

Building a multimodal human-robot interface

When we begin to build and interact with machines or robots that either look like humans or have human functionalities and capabilities, then people may well interact with their human-like machines in ways that mimic human-human communication. For example, if a robot has a face, a human might interact with it similarly to how humans interact with other creatures with faces, Specifically, a human might talk to it, gesture to it, smile at it, and so on. If a human interacts with a computer or a machine that understands spoken commands, the human might converse with the machine, expecting it to have competence in spoken language. In our research on a multimodal interface to mobile robots, we have assumed a model of communication and interaction that, in a sense, mimics how people communicate. Our interface therefore incorporates both natural language understanding and gesture recognition as communication modes. We limited the interface to these two modes to simplify integrating them in the interface and to make our research more tractable. We believe that with an integrated system, the user is less concerned with how to communicate (which interactive mode to employ for a task), and is therefore free to concentrate on the tasks and goals at hand. Because we integrate all our systems components, users can choose any combination of our interfaces modalities. The onus is on our interface to integrate the input, process it, and produce the desired results.

Goal tracking in a natural language interface: towards achieving adjustable autonomy

Intelligent mobile robots that interact with humans must exhibit adjustable autonomy; that is, the ability to dynamically adjust the level of self-sufficiency of an agent depending on the situation. When intelligent robots require close interactions with humans, they will require modes of communication that enhance the ability for humans to communicate naturally and that allow greater interaction, as well as adapt as a team member or sole agent in achieving various goals. Our previous work examined the use of multiple modes of communication, specifically natural language and gestures, to disambiguate the communication between a human and a robot. In this paper, we propose using context predicates to keep track of various goals during human-robot interactions. These context predicates allow the robot to maintain multiple goals, each with possibly different levels of required autonomy. They permit direct human interruption of the robot, while allowing the robot to smoothly return to a high level of autonomy.

Communicating with Teams of Cooperative Robots

We are designing and implementing a multi-modal interface to a team of dynamically autonomous robots. For this interface, we have elected to use natural language and gesture. Gestures can be either natural gestures perceived by a vision system installed on the robot, or they can be made by using a stylus on a Personal Digital Assistant. In this paper we describe the integrated modes of input and one of the theoretical constructs that we use to facilitate cooperation and collaboration among members of a team of robots. An integrated context and dialog processing component that incorporates knowledge of spatial relations enables cooperative activity between the multiple agents, both human and robotic.

UTILIZING DOMAIN-SPECIFIC INFORMATION FOR PROCESSING COMPACT TEXT

This paper identifies the types of sentence fragments found in the text of two domains: medical records and Navy equipment status messages. The fragment types are related to full sentence forms on the basis of the elements which were regularly deleted. A breakdown of the fragment types and their distributions in the two domains is presented. An approach to reconstructing the semantic class of deleted elements in the medical records is proposed which is based on the semantic patterns recognized in the domain.

Analysis and processing of compact text

This paper describes the characteristics of compact text as revealed in computer analysis of a set of physician notes. Computer processing of the documents was performed using the LSP system for natural language analysis. A numerical breakdown of syntactic and semantic patterns found in the texts is presented. It is found that four major properties of compact text make it possible to process the content of the documents with syntactic procedures that operate on full free text.

An experiment in automated health care evaluation from narrative medical records.

Abstract This paper describes an experiment in the automatic application of health care evaluation criteria to narrative hospital discharge summaries. The processing of the documents was done in two steps. First a three-stage system for natural language analysis accepted the sentences of each discharge summary as input and produced as output a tabular representation of the information in the document. Second, the health care evaluation criteria were implemented as a set of retrieval routines on this tabular representation. A comparison of the computer-generated evaluation results with those obtained by a physician reviewer showed good agreement, indicating that it is possible to process medical narrative automatically and to perform complex, medically significant fact retrieval on narrative input.

Natural Language in Four Spatial Interfaces

We describe our experiences building spoken language interfaces to four demonstration applications all involving 2- or 3-D spatial displays or gestural interactions: an air combat command and control simulation, an immersive VR tactical scenario viewer, a map-based air strike simulation tool with cartographic database, and a speech/gesture controller for mobile robots.

Automatic encoding of clinical narrative

An experiment in the automatic encoding of English-language medical data is described. The encoding program has two stages. First, the free-text input is parsed and the information is arranged in a tabular format by a general-purpose natural language processor developed at New York University. Then a simple code-dependent subprogram assigns numerical values to the entries on the basis of the positions the input words occupy in the information format. Results of a blind test of the encoding program using the code employed at Roswell Park Memorial Institute for earliest symptoms of head-neck cancer are presented.

Choosing Frames of Referenece: Perspective-Taking in a 2D and 3D Navigational Task:

This study investigates how frames of reference are chosen in a dynamic navigational task. Participants issued verbal instructions to an animated robot and were provided with one of three views for navigating the animated robot around a virtual world. The different views included a flat two-dimensional (2D) North-up map, a three-dimensional (3D) robots eye view of the world, and a 3D view from behind the robot (3D-Camera) in which depth cues were manipulated. Our results show people adopt an egocentric frame of reference when depth cues are salient and an exocentric reference frame when depth cues are absent. The results suggest the absence or presence of depth cues is a critical component in choosing a reference frame. We discuss the extension of Bryant and Tverskys (1999) theoretical framework to a dynamic environment, such as navigation.

Computer Structuring of Free-Text Patient Data

This paper will describe ongoing work on developing a computer representation of the information in patient narrative for mapping into a database. Previous papers reported on the design of the database and pilot applications of the system. This paper describes the steps in the text-processing, with particular emphasis on the modules that introduce semantic (i.e. medical) information into the analysis.