Jamey Graham

Visualizing multimedia content on paper documents: components of key frame selection for Video Paper

The components of a key frame selection algorithm for a paper-based multimedia browsing interface called Video Paper are described. Analysis of video image frames is combined with the results of processing the closed caption to select key frames that are printed on a paper document together with the closed caption. Bar codes positioned near the key frames allow a user to play the video from the corresponding times. This paper describes several component techniques that are being investigated for key frame selection in the Video Paper system, including face detection and text recognition. The Video Paper system implementation is also discussed.

Prescient paper: multimedia document creation with document image matching

A system is described for creating paper documents that show images of presentation slides and bar codes that point to a multimedia recording of a presentation that has not yet occurred. An image-matching algorithm applied after a presentation determines when each slide was displayed. These time-stamps map the bar codes onto commands that control a multimedia player. We describe the system infrastructure that allows us to prepare such prescient documents and the document image-matching algorithm that enables the mapping of bar codes onto the times when slides were displayed. This provides a multimedia annotation tool that requires no electronic device at capture time.

The video paper multimedia playback system

Video Paper is a prototype system for multimedia browsing, analysis, and replay. Key frames extracted from a video recording are printed on paper together with bar codes that allow for random access and replay. A transcript for the audio track can also be shown so that users can read what was said, thus making the document a stand-alone representation for the contents of the multimedia recording. The Video Paper system has been used for several applications, including the analysis of recorded meetings, broadcast news, oral histories and personal recordings. This demonstration will show how the Video Paper system was applied to these domains and the various replay systems that were developed, including a self-contained portable implementation on a PDA and a fixed implementation on desktop PC.

Portable meeting recorder

The design and implementation of a portable meeting recorder is presented. Composed of an omni-directional video camera with four-channel audio capture, the system saves a view of all the activity in a meeting and the directions from which people spoke. Subsequent analysis computes metadata that includes video activity analysis of the compressed data stream and audio processing that helps locate events that occurred during the meeting. Automatic calculation of the room in which the meeting occurred allows for efficient navigation of a collection of recorded meetings. A user interface is populated from the metadata description to allow for simple browsing and location of significant events.

Books with voices: paper transcripts as a physical interface to oral histories

Our contextual inquiry into the practices of oral historians unearthed a curious incongruity. While oral historians consider interview recordings a central historical artifact, these recordings sit unused after a written transcript is produced. We hypothesized that this is largely because books are more usable than recordings. Therefore, we created Books with Voices: bar-code augmented paper transcripts enabling fast, random access to digital video interviews on a PDA. We present quantitative results of an evaluation of this tangible interface with 13 participants. They found this lightweight, structured access to original recordings to offer substantial benefits with minimal overhead. Oral historians found a level of emotion in the video not available in the printed transcript. The video also helped readers clarify the text and observe nonverbal cues.

The reader's helper: a personalized document reading environment

Over the last two centuries, reading styles have shifted awayfrom the reading of documents from beginning to end and toward theskimming of documents in search of relevant information. This trendcontinues today where readers, often confronted with aninsurmountable amount of text, seek more efficient methods ofextracting relevant information from documents. In this paper, anew document reading environment is introduced called the ReadersHelperTM, which supports the reading of electronic and paperdocuments. The Readers Helper analyzes documents and produces arelevance score for each of the readers topics of interest, therebyhelping the reader decide whether the document is actually worthskimming or reading. Moreover, during the analysis process, topicof interest phrases are automatically annotated to help the readerquickly locate relevant information. A new informationvisualization tool, called the ThumbarTM, is used in conjunctionwith relevancy scoring and automatic annotation to portray acontinuous, dynamic thumb-nail representation of the document. Thisfurther supports rapid navigation of the text.

Query-free information retrieval

To liberate users from burdensome information-retrieval activities while incurring minimal system-development and runtime costs, the authors present query-free information retrieval. Their system, Fixit, integrates an expert diagnostic system with a preexisting full-text database of maintenance manuals.

A paper-based interface for video browsing and retrieval

A paper-based interface for browsing video is proposed. A paper document shows key frames selected from a video, a transcript for the parallel audio track, and bar codes that, when scanned, invoke a multimedia player. The paper document provides a stand-alone representation for a video recording that lets a user both understand the content of the file and replay only selected parts of the multimedia that are necessary to gain a better understanding. This approach applies the two-dimensional display characteristics of a newspaper to multimedia retrieval. By so doing, the users browsing and search efficiency is greatly improved. This poster describes an implementation of the video paper system using a pocket PC with a bar code reader as the remote control device and an archive of video recordings on the pocket PC or an external server.

Mobile image recognition: architectures and tradeoffs

We argue that the most desirable architecture for mobile image recognition runs the complete algorithm on the mobile device. Alternative solutions that run the recognizer on a remote server will not be as desirable because of the delay between image capture and receipt of a result that can cause users to abandon the technique. We present a method for mobile recognition of paper documents and an application to newspapers that lets readers retrieve electronic data linked to articles, photos, and advertisements. We show that the index for a reasonable collection of daily newspapers can be downloaded in less than a minute and will fit in the memory of todays mid-range smart phones. Experimental results show that the recognition system has an overall error rate of less than 1%. We achieved a run time of 1.2 secs. per image with a collection of 140 newspaper pages on an HTC-8282 Windows Mobile phone.

HOTPAPER demonstration: multimedia interaction with paper using mobile phones

HotPaper enables users to electronically interact with paper documents by pointing their camera phones at a document. Electronic interaction is in the form of reading and/or writing images, audio and video clips, urls, notes, etc to the paper. HotPaper does not require any machine readable code, such as a barcode or watermark, to be present on the document. It utilizes our Brick Wall Coding document recognition algorithm [1] that uses a small document patch image to retrieve electronic versions of documents without performing OCR. The HotPaper demonstration runs on a Treo 700w 312 MHz mobile phone and performs document recognition at 4 frames per second using 176x144 video input.