Howard D. Wactlar

Intelligent access to digital video: Informedia project

Carnegie Mellons Informedia Digital Video Library project will establish a large, on-line digital video library featuring full-content and knowledge-based search and retrieval. Intelligent, automatic mechanisms will be developed to populate the library. Search and retrieval from digital video, audio, and text libraries will take place via desktop computer over local, metropolitan, and wide-area networks. The projects approach applies several techniques for content-based searching and video-sequence retrieval. Content is conveyed in both the narrative (speech and language) and the image. Only by the collaborative interaction of image, speech, and natural language understanding technology is it possible to successfully populate, segment, index, and search diverse video collections with satisfactory recall and precision. This collaborative interaction approach uniquely compensates for problems of interpretation and search in error-ridden and ambiguous data sets. The authors have focused the work on two corpuses. One is science documentaries and lectures, the other is broadcast news content with partial closed-captions. Further work will continue to improve the accuracy and performance of the underlying processing as well as explore performance issues related to Web-based access and interoperability with other digital video resources.

Intelligent assistive technology applications to dementia care: current capabilities, limitations, and future challenges.

The number of older Americans afflicted by Alzheimer disease and related dementias will triple to 13 million persons by 2050, thus greatly increasing healthcare needs. An approach to this emerging crisis is the development and deployment of intelligent assistive technologies that compensate for the specific physical and cognitive deficits of older adults with dementia, and thereby also reduce caregiver burden. The authors conducted an extensive search of the computer science, engineering, and medical databases to review intelligent cognitive devices, physiologic and environmental sensors, and advanced integrated sensor networks that may find future applications in dementia care. Review of the extant literature reveals an overwhelming focus on the physical disability of younger persons with typically nonprogressive anoxic and traumatic brain injuries, with few clinical studies specifically involving persons with dementia. A discussion of the specific capabilities, strengths, and limitations of each technology is followed by an overview of research methodological challenges that must be addressed to achieve measurable progress to meet the healthcare needs of an aging America.

Lessons learned from building a terabyte digital video library

The Informedia Project at Carnegie Mellon University has created a terabyte digital video library in which automatically derived descriptors for the video are used for indexing, segmenting and accessing the library contents. Begun in 1994, the project presented numerous challenges for library creation and deployment, valuable information covered in this article. The authors, developers of the project at Carnegie Mellon University, addressed these challenges by: automatically extracting information from digitized video; creating interfaces that allowed users to search for and retrieve videos based on extracted information; and validating the system through user testbeds. Through speech, image, and natural language processing, the Informedia Project has demonstrated that previously inaccessible data can be derived automatically and used to describe and index video segments.

Collages as dynamic summaries for news video

This paper introduces the video collage, a novel effective interface for browsing and interpreting video collections. The paper discusses how collages are automatically produced, illustrates their use, and evaluates their effectiveness as summaries across news stories. Collages are presentations of text and images derived from multiple video sources, which provide an interactive visualization for a set of video documents, summarizing their contents and providing a navigation aid for further exploration. The dynamic creation of collages is based on user context, e.g., an originating query, coupled with automatic processing to refine the candidate imagery. Named entity identification and common phrase extraction provides descriptive text. The dynamic manipulation of collages allows user-directed browsing and reveals additional detail. The utility of collages as summaries is examined with respect to other published news summaries.

Automated analysis of nursing home observations

Pervasive activity monitoring in a skilled-nursing facility helps capture a continuous audio and video record. The CareMedia project analyzes this video information by automatically tracking people, helping to efficiently label individuals, and characterizing selected activities and actions.

Towards automatic analysis of social interaction patterns in a nursing home environment from video

In this paper, we propose an ontology-based approach for analyzing social interaction patterns in a nursing home from video. Social interaction patterns are broken into individual activities and behavior events using a multi-level context hierarchy ontology framework. To take advantage of an ontology in representing how social interactions evolve, we design and refine the ontology based on knowledge gained from 80 hours of video recorded in the public spaces of a nursing home. The ontology is implemented using a dynamic Bayesian network to statistically model the multi-level concepts defined in the ontology. We have developed a prototype system to illustrate the proposed concept. Experiment results have demonstrated feasibility of the proposed approach. The objective of this research is to automatically create concise and comprehensive reports of activities and behaviors of patients to support physicians and caregivers in a nursing facility

Digital imagery for significant cultural and historical materials

Digital imagery for significant cultural and historical materials is an emerging research field that bridges people, culture, and technologies. In this paper, we first discuss the great importance of this field. Then we focus on its four interrelated subareas: (1) creation and preservation, (2) retrieval, (3) presentation and usability, and (4) applications and use. We propose several mechanisms to encourage collaboration and argue that the field has high potential impact on our digital society. Finally, we make specific recommendations on what to pursue in this field.

Indexing and search of multimodal information

The Informedia Digital Library Project allows full content indexing and retrieval of text, audio and video material. The integration of speech recognition, image processing, natural language processing and information retrieval overcomes limits in each technology to create a useful system. In order to answer the question how good speech recognition has to be in order to be useful and usable for indexing and retrieving speech recognizer generated transcripts, some empirical evidence is presented that illustrates the degradation of information retrieval at different levels of speech accuracy. In our experiments, word error rates up to 25% did not significantly impact information retrieval and error rates of 50% still provided 85 to 95% of the recall and precision relative to fully accurate transcripts in the same retrieval system.

Dining activity analysis using a hidden Markov model

We describe an algorithm for dining activity analysis in a nursing home. Based on several features, including motion vectors and distance between moving regions in the subspace of an individual person, a hidden Markov model is proposed to characterize different stages in dining activities with certain temporal order. Using HMM model, we are able to identify the start (and ending) of individual dining events with high accuracy and low false positive rate. This approach could be successful in assisting caregivers in assessments of residents activity levels over time.

Techniques for the Creation and Exploration of Digital Video Libraries

The Information Age is fully upon us. A recent article noted that there are perhaps 50 million people using the Internet on a regular basis, and That “the current growth rate is about 15% per month (!) and this could well continue until almost all of those in the ‘developed world’ are connected” [FM94, p. 30]. In addition, the digital domain consists not only of text but increasingly of other media representations, from graphics images to audio to motion video. As the amount of information and number of users exponentially escalate, more attention focuses on the basic problems of information management: How do you digitize information? How can you then visualize it and find what you need? How do you use and manipulate it effectively? How is it stored and managed? The proliferation of technical articles and special issues addressing these questions underscore their importance; see for example the special issue on Content-based retrieval [Nar95] or digital libraries [F+95]. This chapter will survey some of that work, especially that which relates to the treatment of video and the use of digital video libraries For education.