Charles B. Callaway

Adaptive, intelligent presentation of information for the museum visitor in PEACH

The study of intelligent user interfaces and user modeling and adaptation is well suited for augmenting educational visits to museums. We have defined a novel integrated framework for museum visits and claim that such a framework is essential in such a vast domain that inherently implies complex interactivity. We found that it requires a significant investment in software and hardware infrastructure, design and implementation of intelligent interfaces, and a systematic and iterative evaluation of the design and functionality of user interfaces, involving actual visitors at every stage. We defined and built a suite of interactive and user-adaptive technologies for museum visitors, which was then evaluated at the Buonconsiglio Castle in Trento, Italy: (1) animated agents that help motivate visitors and focus their attention when necessary, (2) automatically generated, adaptive video documentaries on mobile devices, and (3) automatically generated post-visit summaries that reflect the individual interests of visitors as determined by their behavior and choices during their visit. These components are supported by underlying user modeling and inference mechanisms that allow for adaptivity and personalization. Novel software infrastructure allows for agent connectivity and fusion of multiple positioning data streams in the museum space. We conducted several experiments, focusing on various aspects of PEACH. In one, conducted with 110 visitors, we found evidence that even older users are comfortable interacting with a major component of the system.

Coherent gestures, locomotion, and speech in life-like pedagogical agents

Life-like animated interface agents for knowledge-based learning environments can provide timely, customized advice to support students’ problem solving. Because of their strong visual presence, they hold significant promise for substantially increasing students’ enjoyment of their learning experiences. A key problem posed by life-like agents that inhabit artificial worlds is deictic believability. In the same manner that humans refer to objects in their environment through judicious combinations of speech, locomotion, and gesture, animated agents should be able to move through their environment, and point to and refer to objects appropriately as they provide problem-solving advice. In this paper we describe a framework for achieving deictic believabilityin animated agents. A deictic behavior planner exploits a world model and the evolving explanation plan as it selects and coordinates locomotive, gestural, and speech behaviors. The resulting behaviors and utterances are believable, and the references are unambiguous. This approach to spatial deixis has been implemented in a life-like animated agent, Cosmo, who inhabits a learning environment for the domain of Internet packet routing. The product of a large multidisciplinary team of computer scientists, 3D modelers, graphic artists, and animators, Cosmo provides realtime advice to students as they escort packets through a virtual world of interconnected routers.

Automatic cinematography and multilingual NLG for generating video documentaries

Automatically constructing a complete documentary or educational film from scattered pieces of images and knowledge is a significant challenge. Even when this information is provided in an annotated format, the problems of ordering, structuring and animating sequences of images, and producing natural language descriptions that correspond to those images within multiple constraints, are each individually difficult tasks. This paper describes an approach for tackling these problems through a combination of rhetorical structures with narrative and film theory to produce movie-like visual animations from still images along with natural language generation techniques needed to produce text descriptions of what is being seen in the animations. The use of rhetorical structures from NLG is used to integrate separate components for video creation and script generation. We further describe an implementation, named Glamour, that produces actual, short video documentaries, focusing on a cultural heritage domain, and that have been evaluated by professional filmmakers.

Pronominalization in Generated Discourse and Dialogue

Previous approaches to pronominalization have largely been theoretical rather than applied in nature. Frequently, such methods are based on Centering Theory, which deals with the resolution of anaphoric pronouns. But it is not clear that complex theoretical mechanisms, while having satisfying explanatory power, are necessary for the actual generation of pronouns. We first illustrate examples of pronouns from various domains, describe a simple method for generating pronouns in an implemented multi-page generation system, and present an evaluation of its performance.

Intelligent tutoring with natural language support in the BEETLE II system

We present Beetle II, a tutorial dialogue system designed to accept unrestricted language input and support experimentation with different tutorial planning and dialogue strategies. Our first system evaluation used two different tutoring policies and demonstrated that BEETLE II can be successfully used as a platform to study the impact of different approaches to tutoring. In the future, the system can also be used to experiment with a variety of parameters that may affect learning in intelligent tutoring systems.

Mobile drama in an instrumented museum: inducing group conversation via coordinated narratives

Museum visits can be more enjoyable to small groups if they can be both social and educational experiences. One very rewarding aspect of a visit, especially those involving small groups such as families, is the unmediated group discussion that can ensue during a shared cultural experience. We present a situated, mobile museum system that delivers an hour-long drama to museum visitors. It perceives and analyzes group behavior, uses the result to dynamically deliver coordinated dramatic narrative presentations about the nearby museum exhibit, with the expected result of stimulating group discussion. To accomplish this, our drama-based presentations contain small, complementary differences in the content delivered to each participant, leveraging the narrative tension/release cycle of drama to naturally lead visitors to fill in missing pieces by interacting with friends, thus initiating a conversation. We present two evaluations for these story variations, one in a closed, non-mobile environment, and the other a formative evaluation to gauge how well the methodology used in the non-mobile evaluation performs in evaluating the fully implemented system in a real museum environment.

Integrating discourse and domain knowledge for document drafting

Document drafting is a key component of legal expertise. Effective legal document drafting requires knowledge both of legal domain knowledge and of the structure of legal discourse. Automating the task of legal document drafting therefore requires explicit representation of both these types of knowledge. This paper proposes an architecture that integrates these two disparate knowledge sources in a modular architecture under which representation and control are optimized for each task. This architecture is being implemented in DOCUPLANNER 2.0, a system for interactive document drafting.

Personal reporting of a museum visit as an entrypoint to future cultural experience

Museum visitors can continue interacting with museum exhibits even after they have left the museum. We can help them do this by creating a report that includes a basic, personalized narration of their visit, the items and relationships they found most interesting, pointers to additional related online information, and suggestions for future visits to the current and other museums. In this work we describe the automatic generation of personalized natural language reports to help create one episode in an ongoing coherent sequence of cultural activities.

Multiple Coordinated Mobile Narratives as a Catalyst for Face-to-Face Group Conversation

Museum visits can be more useful to small groups if they can be the centerpiece of a social experience as well as an educational one. One of the most rewarding aspects of a visit, especially those involving families, is the unmediated group discussion that can ensue during a shared cultural experience. However, if current methods were to be applied to stimulate such discussions, their intrusive nature would be more likely detrimental than assistive. We present a non-intrusive method based on dramatic narrative presentations and automatic group behavior perception that we believe stimulates a group to engage in conversation as a natural part of making sense of partial narratives that require some mutual integration.

Automating Judicial Document Drafting: A Discourse-Based Approach

Document drafting is a central judicial problem-solving activity. Development of automated systems to assist judicial document drafting has been impeded by the absence of an explicit model of (1) the connection between the document drafter’s goals and the text intended to achieve those goals, and (2) the rhetorical constraints expressing the stylistic and discourse conventions of the document’s genre. This paper proposes a model in which the drafter’s goals and the stylistic and discourse conventions are represented in a discourse structure consisting of a tree of illocutionary and rhetorical operators with document text as leaves. A document grammar based on the discourse structures of a representative set of documents can be used to synthesize a wide range of additional documents from sets of case facts. The applicability of this model to a representative class of judicial orders — jurisdictional show-cause orders — is demonstrated by illustrating (1) the analysis of show-cause orders in terms of discourse structures, (2) the derivation of a document grammar from discourse structures of two typical show-cause orders, and (3) the synthesis of a new show-cause order from the document grammar.