Katashi Nagao

The world through the computer: computer augmented interaction with real world environments

Current user interface techniques such as WIMP or the desktop metaphor do not support real world tasks, because the focus of these user interfaces is only on human–computer interactions, not on human–real world interactions. In this paper, we propose a method of building computer augmented environments using a situation-aware portable device. This device, calledNaviCam, has the ability to recognize the user’s situation by detecting color-code IDs in real world environments. It displays situation sensitive information by superimposing messages on its video see-through screen. Combination of ID-awareness and portable video-see-through display solves several problems with current ubiquitous computers systems and augmented reality systems.

Semantic annotation and transcoding: making Web content more accessible

We propose a method for constructing superstructure on the Web using XML and external annotations to Web documents. We have three approaches for annotating documents: linguistic, commentary, and multimedia. The result is annotated documents that computers can understand and process more easily, allowing content to reach a wider audience with minimal overhead.

SPEECH DIALOGUE WITH FACIAL DISPLAYS: MULTIMODAL HUMAN-COMPUTER CONVERSATION

Human face-to-face conversation is an ideal model for human-computer dialogue. One of the major features of face-to-face communication is its multiplicity of communication channels that act on multiple modalities. To realize a natural multimodal dialogue, it is necessary to study how humans perceive information and determine the information to which humans are sensitive. A face is an independent communication channel that conveys emotional and conversational signals, encoded as facial expressions. We have developed an experimental system that integrates speech dialogue and facial animation, to investigate the effect of introducing communicative facial expressions as a new modality in human-computer conversation. Our experiments have showen that facial expressions are helpful, especially upon first contact with the system. We have also discovered that featuring facial expressions at an early stage improves subsequent interaction.

Video Scene Annotation Based on Web Social Activities

This article describes a mechanism to acquire the semantics of video content from the activities of Web communities that use a bulletin-board system and Weblog tools to discuss video scenes.

A music recommendation system based on annotations about listeners' preferences and situations

In this paper, we present a playlist generation scheme that uses lyrics and annotations to discover similarity between kinds of music and user tastes. It generates a playlist according to user preferences and situations. Additionally, users can provide some feedbacks to the system such as whether each tune is suitable for the preference and the situation. The system transforms the feature values concerning preferences and situations and adapts them to each user. The playlists are generated through three phases. First, an initial playlist is found from databases by content-based retrieval. Second, transcoding improves the playlist according to the users preference and situation. Finally, by interaction between the system and the user, the playlist becomes more suitable for the user.

Annotation-based multimedia summarization and translation

This paper presents techniques for multimedia annotation and their application to video summarization and translation. Our tool for annotation allows users to easily create annotation including voice transcripts, video scene descriptions, and visual/auditory object descriptions. The module for voice transcription is capable of multilingual spoken language identification and recognition. A video scene description consists of semi-automatically detected keyframes of each scene in a video clip and time codes of scenes. A visual object description is created by tracking and interactive naming of people and objects in video scenes. The text data in the multimedia annotation are syntactically and semantically structured using linguistic annotation. The proposed multimedia summarization works upon a multimodal document that consists of a video, keyframes of scenes, and transcripts of the scenes. The multimedia translation automatically generates several versions of multimedia content in different languages.

Discussion mining: annotation-based knowledge discovery from real world activities

We present discussion mining as a preliminary study of knowledge discovery from discussion content of offline meetings. Our system generates minutes for such meetings semi-automatically and links them with audio-visual data of discussion scenes. Then, not only retrieval of the discussion content, but also we are pursuing the method of searching for a similar discussion to an ongoing discussion from the past ones, and the method of generation of an answer to a certain question based on the accumulated discussion content. In terms of mailing lists and online discussion systems such as bulletin board systems, various studies have been done. However, what we think is greatly different from the previous works is that ours includes face-to-face offline meetings. We analyze meetings from diversified perspectives using audio and visual information. We also developed a tool for semantic annotation on discussion content. We consider this research not just data mining but a kind of real-world human activity mining.

Discussion ontology: knowledge discovery from human activities in meetings

Discussion mining is a preliminary study on gathering knowledge based on the content of face-to-face discussion meetings. To extract knowledge from discussion content, we have to analyze not only the surface arguments, but also semantic information such as a statements intention and the discussion flow during meetings. We require a discussion ontology for this information. This discussion ontology forms the basis of our discussion methodology and requires semantic relations between elements in meetings. We must clarify these semantic relations to build the discussion ontology. We therefore generated discussion content and analyzed meeting metadata to build the ontology.

Collaborative Video Scene Annotation Based on Tag Cloud

In this paper, we propose a video scene annotation method based on tag clouds. First, user comments associated with a video are collected from existing video sharing services. Next, a tag cloud is generated from these user comments. The tag cloud is displayed on the video window of the Web browser. When users click on a tag included in the tag cloud while watching the video, the tag gets associated with the time point of the video. Users can share the information on the tags that have already been clicked. We confirmed that the coverage of annotations generated by this method is higher than that of the existing methods, and users are motivated to add tags by using tag-sharing and tag-cloud methods. This method assists in establishing highly accurate advanced video applications.

Speech dialogue with facial displays

The human face is an independent communication channel that conveys emotional and conversational signals encoded as facial displays. We are attempting to introduce facial displays into multimodal human computer interaction as a new modality to make computer more communicative and social. As a first step, we developed a multimodal human computer interaction system integrating speech dialogue and faciat animation. The video gives an overview of the major research topics involved in this project. They are to understand and manage speech dialogue, to design and animate communicative facial displays, and to combine multiple modalities, that is, speech and facial displays.