Stephanie Elzer

Information graphics: an untapped resource for digital libraries

Information graphics are non-pictorial graphics such as bar charts and line graphs that depict attributes of entities and relations among entities. Most information graphics appearing in popular media have a communicative goal or intended message; consequently, information graphics constitute a form of language. This paper argues that information graphics are a valuable knowledge resource that should be retrievable from a digital library and that such graphics should be taken into account when summarizing a multimodal document for subsequent indexing and retrieval. But to accomplish this, the information graphic must be understood and its message recognized. The paper presents our Bayesian system for recognizing the primary message of one kind of information graphic (simple bar charts) and discusses the potential role of an information graphics message in indexing graphics and summarizing multimodal documents.

Recognizing the intended message of line graphs

Information graphics (line graphs, bar charts, etc.) that appear in popular media, such as newspapers and magazines, generally have a message that they are intended to convey. We contend that this message captures the high-level knowledge conveyed by the graphic and can serve as a brief summary of the graphics content. This paper presents a system for recognizing the intended message of a line graph. Our methodology relies on 1)segmenting the line graph into visually distinguishable trends which are used to suggest possible messages, and 2)extracting communicative signals from the graphic and using them as evidence in a Bayesian Network to identify the best hypothesis about the graphics intended message. Our system has been implemented and its performance has been evaluated on a corpus of line graphs.

Constructing and Utilizing a Model of User Preferences in Collaborative Consultation Dialogues

A natural language collaborative consultation system must take user preferences into account. A model of user preferences allows a system to appropriately evaluate alternatives using criteria of importance to the user. Additionally, decision research suggests both that an accurate model of user preferences could enable the system to improve a users decision‐making by ensuring that all important alternatives are considered, and that such a model of user preferences must be built dynamically by observing the users actions during the decision‐making process. This paper presents two strategies: one for dynamically recognizing user preferences during the course of a collaborative planning dialogue and the other for exploiting the model of user preferences to detect suboptimal solutions and suggest better alternatives. Our recognition strategy utilizes not only the utterances themselves but also characteristics of the dialogue in developing a model of user preferences. Our generation strategy takes into account both the strength of a preference and the closeness of a potential match in evaluating actions in the users plan and suggesting better alternatives. By modeling and utilizing user preferences, our system is able to fulfill its role as a collaborative agent.

Getting computers to see information graphics so users do not have to

Information graphics such as bar, line and pie charts appear frequently in electronic media and often contain information that is not found elsewhere in documents. Unfortunately, sight-impaired users have difficulty accessing and assimilating information graphics. Our goal is an interactive natural language system that provides effective access to information graphics for sight-impaired individuals. This paper describes how image processing has been applied to transform an information graphic into an XML representation that captures all aspects of the graphic that might be relevant to extracting knowledge from it. It discusses the problems that were encountered in analyzing and categorizing components of the graphic, and the algorithms and heuristics that were successfully applied. The resulting XML representation serves as input to an evidential reasoning component that hypothesizes the message that the graphic was intended to convey.

Exploring and Exploiting the Limited Utility of Captions in Recognizing Intention in Information Graphics

This paper presents a corpus study that explores the extent to which captions contribute to recognizing the intended message of an information graphic. It then presents an implemented graphic interpretation system that takes into account a variety of communicative signals, and an evaluation study showing that evidence obtained from shallow processing of the graphics caption has a significant impact on the systems success. This work is part of a larger project whose goal is to provide sight-impaired users with effective access to information graphics.

Interactive SIGHT into information graphics

Information graphics (such as bar charts and line graphs) play a vital role in many multimodal documents. Unfortunately, visually impaired individuals who use screen reader programs to navigate through such documents have limited access to the graphics. This paper presents the Interactive SIGHT (Summarizing Information GrapHics Textually) system that provides visually impaired individuals with the high-level knowledge that one would gain from viewing graphics in electronic documents. The current system, which is implemented as a browser extension, works on simple bar charts. Once launched by a keystroke combination, Interactive SIGHT first provides a brief initial summary that conveys the underlying message of the bar chart along with the charts most significant features. The system is then able to generate history-aware follow-up responses that provide further information upon request from the user. User evaluations with sighted and visually impaired users showed that the initial summary and follow-up responses are very effective in conveying the informational content of graphics and that the system interface is easy to use.

A Model of Perceptual Task Effort for Bar Charts and its Role in Recognizing Intention

This paper presents a model of perceptual task effort for use in hypothesizing the message that a bar chart is intended to convey. It presents our rules, based on research by cognitive psychologists, for estimating perceptual task effort, and discusses the results of an eye tracking experiment that demonstrates the validity of our model. These rules comprise a model that captures the relative difficulty that a viewer would have performing one perceptual task versus another on a specific bar chart. The paper outlines the role of our model of relative perceptual task effort in recognizing the intended message of an information graphic. Potential applications of this work include using this message to provide (1) a more complete representation of the content of the document to be used for searching and indexing in digital libraries, and (2) alternative access to the information graphic for visually impaired users or users of low-bandwidth environments.

Automatically recognizing intended messages in grouped bar charts

Information graphics (bar charts, line graphs, grouped bar charts, etc.) often appear in popular media such as newspapers and magazines. In most cases, the information graphic is intended to convey a high-level message; this message plays a role in understanding the document but is seldom repeated in the documents text. This paper presents our methodology for recognizing the intended message of a grouped bar chart. We discuss the types of messages communicated in grouped bar charts, the communicative signals that serve as evidence for the message, and the design and evaluation of our implemented system.

Extending plan inference techniques to recognize intentions in information graphics

Plan inference techniques have been used extensively to understand natural language dialogue. But as noted by Clark[5], language and communication are more than just utterances. This paper presents the problems that we have had to address and the solutions that we have devised in designing a system to recognize intentions from information graphics. Our work is part of a larger project to develop an interactive natural language system that provides an alternative means for individuals with sight-impairments to access the content of information graphics.

Incorporating perceptual task effort into the recognition of intention in information graphics

The rapidly increasing availability of electronic publications containing information graphics poses some interesting challenges in terms of information access. For example, visually impaired individuals should ideally be provided with access to the knowledge that would be gleaned from viewing the information graphic. Similarly, digital libraries must take into account the content of information graphics when constructing indices. This paper outlines our approach to recognizing the intended message of an information graphic, focusing on the concept of perceptual task effort, its role in the inference process, our rules for estimating effort, and the results of an eye tracking experiment conducted in order to evaluate and modify those rules.