Mary D. Swift

Deep Linguistic Processing for Spoken Dialogue Systems

We describe a framework for deep linguistic processing for natural language understanding in task-oriented spoken dialogue systems. The goal is to create domaingeneral processing techniques that can be shared across all domains and dialogue tasks, combined with domain-specific optimization based on an ontology mapping from the generic LF to the application ontology. This framework has been tested in six domains that involve tasks such as interactive planning, coordination operations, tutoring, and learning.

Linking Semantic and Knowledge Representations in a Multi-Domain Dialogue System

We describe a two-layer architecture for supporting semantic interpretation and domain reasoning in dialogue systems. Building system that supports both semantic interpretation and domain reasoning in a transparent and well-integrated manner is an unresolved problem because of the diverging requirements of the semantic representations used in contextual interpretation versus the knowledge representations used in domain reasoning. We propose an architecture that provides both portability and efficiency in natural language interpretation by maintaining separate semantic and domain knowledge representations, and integrating them via an ontology mapping procedure. The ontology mapping is used to obtain representations of utterances in a form most suitable for domain reasoners and to automatically specialize the lexicon. The use of a linguistically motivated parser for producing semantic representations for complex natural language sentences facilitates building portable semantic interpretation components as well as connections with domain reasoners. Two evaluations demonstrate the effectiveness of our approach: we show that a small number of mapping rules are sufficient for customizing the generic semantic representation to a new domain, and that our automatic lexicon specialization technique improves parser speed and accuracy.

Mobile phone-based asthma self-management aid for adolescents (mASMAA): a feasibility study

Purpose Adolescents report high asthma-related morbidity that can be prevented by adequate self-management of the disease. Therefore, there is a need for a developmentally appropriate strategy to promote effective asthma self-management. Mobile phone-based technology is portable, commonly accessible, and well received by adolescents. The purpose of this study was to develop and evaluate the feasibility and acceptability of a comprehensive mobile phone-based asthma self-management aid for adolescents (mASMAA) that was designed to facilitate symptom monitoring, treatment adherence, and adolescent–parent partnership. The system used state-of-the-art natural language-understanding technology that allowed teens to use unconstrained English in their texts, and to self-initiate interactions with the system. Materials and methods mASMAA was developed based on an existing natural dialogue system that supports broad coverage of everyday natural conversation in English. Fifteen adolescent–parent dyads participated in a 2-week trial that involved adolescents’ daily scheduled and unscheduled interactions with mASMAA and parents responding to daily reports on adolescents’ asthma condition automatically generated by mASMAA. Subsequently, four focus groups were conducted to systematically obtain user feedback on the system. Frequency data on the daily usage of mASMAA over the 2-week period were tabulated, and content analysis was conducted for focus group interview data. Results Response rates for daily text messages were 81%–97% in adolescents. The average number of self-initiated messages to mASMAA was 19 per adolescent. Symptoms were the most common topic of teen-initiated messages. Participants concurred that use of mASMAA improved awareness of symptoms and triggers, promoted treatment adherence and sense of control, and facilitated adolescent–parent partnerships. Conclusion This study demonstrates the utility and user acceptability of mASMAA as a potential asthma self-management tool in a selective group of adolescents. Further research is needed to replicate the findings in a large group of adolescents from sociodemographically diverse backgrounds to validate the findings.

Towards a Personal Health Management Assistant

We describe design and prototyping efforts for a Personal Health Management Assistant for heart failure patients as part of Project HealthDesign. An assistant is more than simply an application. An assistant understands what its users need to do, interacts naturally with them, reacts to what they say and do, and is proactive in helping them manage their health. In this project, we focused on heart failure, which is not only a prevalent and economically significant disease, but also one that is very amenable to self-care. Working with patients, and building on our prior experience with conversational assistants, we designed and developed a prototype system that helps heart failure patients record objective and subjective observations using spoken natural language conversation. Our experience suggests that it is feasible to build such systems and that patients would use them. The system is designed to support rapid application to other self-care settings.

Generic Parsing for Multi-Domain Semantic Interpretation

Producing detailed syntactic and semantic representations of natural language is essential for practical dialog systems such as plan-based assistants and tutorial systems. Development of such systems is time-consuming and costly as they are typically hand-crafted for each application, and dialog corpus data is more difficult to obtain than text. The TRIPS parser and grammar addresses these issues by providing broad coverage of common constructions in practical dialog and producing semantic representations suitable for dialog processing across domains. Our system bootstraps dialog system development in new domains and helps build parsed corpora.

A Markov logic framework for recognizing complex events from multimodal data

We present a general framework for complex event recognition that is well-suited for integrating information that varies widely in detail and granularity. Consider the scenario of an agent in an instrumented space performing a complex task while describing what he is doing in a natural manner. The system takes in a variety of information, including objects and gestures recognized by RGB-D and descriptions of events extracted from recognized and parsed speech. The system outputs a complete reconstruction of the agents plan, explaining actions in terms of more complex activities and filling in unobserved but necessary events. We show how to use Markov Logic (a probabilistic extension of first-order logic) to create a model in which observations can be partial, noisy, and refer to future or temporally ambiguous events; complex events are composed from simpler events in a manner that exposes their structure for inference and learning; and uncertainty is handled in a sound probabilistic manner. We demonstrate the effectiveness of the approach for tracking kitchen activities in the presence of noisy and incomplete observations.

Skeletons in the parser: using a shallow parser to improve deep parsing

We describe a simple approach for integrating shallow and deep parsing. We use phrase structure bracketing obtained from the Collins parser as filters to guide deep parsing. Our experiments demonstrate that our technique yields substantial gains in speed along with modest improvements in accuracy.

Two Diverse Systems Built using Generic Components for Spoken Dialogue (Recent Progress on TRIPS)

This paper describes recent progress on the TRIPS architecture for developing spoken-language dialogue systems. The interactive poster session will include demonstrations of two systems built using TRIPS: a computer purchasing assistant, and an object placement (and manipulation) task.

Discourse annotation in the Monroe corpus

We describe a method for annotating spoken dialog corpora using both automatic and manual annotation. Our semi-automated method for corpus development results in a corpus combining rich semantics, discourse information and reference annotation, and allows us to explore issues relating these.

Utilizing Natural Language for One-Shot Task Learning

Learning tasks from a single demonstration presents a significant challenge because the observed sequence is specific to the current situation and is inherently an incomplete representation of the procedure. Observation-based machine-learning techniques are not effective without multiple examples. However, when a demonstration is accompanied by natural language explanation, the language provides a rich source of information about the relationships between the steps in the procedure and the decision-making processes that led to them. In this article, we present a one-shot task learning system built on TRIPS, a dialogue-based collaborative problem solving system, and show how natural language understanding can be used for effective one-shot task learning.