Arun Katkere

Reality modeling and visualization from multiple video sequences

A reality modeling and visualization system called Immersive Video uses multiple videos of an event from different perspectives to generate a 3D digital sequence of object movement. Given appropriate camera coverage, full 3D digital videos can be generated using todays technology. The potential for photorealistic videos from arbitrary perspectives exists, although the coverage and degree of realism in virtual views will ultimately be determined by the application. Many issues demand resolution before one will see practical and realistic 3D digital video, including more robust shape recovery methods, broadcast-quality virtual views, and real-time interactive rendering of long 3D sequences.

Interactive video on WWW: beyond VCR-like interfaces

Abstract The WWW is evolving into a predominantly visual medium. The demand for access to images and video has been increasing rapidly. Interactive Video systems, which provide access to the content in video archives, are starting to emerge on the www. Partly due to the two-dimensional nature of the web, and partly due to the fact that images that comprise the video are two dimensional, most of these systems provide a VCR-like interface (play, fast-forward, reverse, etc., with additions like object selection, motion specification in the image space, and viewpoint selection). The basis of this paper is the realization that the video streams represent projections of a three-dimensional world, and the user is interested in this three-dimensional content and not the actual configuration of pixels in the image space. In this paper, we justify this intuition by enumerating the information-bearing entities that the user is interested in, and the information specification mechanisms that allow the user to query upon these entities. We will describe how such a intuitive system could be implemented using WWW technologies — VRML, HTML, and HTTP — and present our current WWW prototype which is based on extensions to some of these standards. This system is built on top of our multiple perspective interactive video (MPI Video) paradigm which provides a framework for the management of and interactive access to multiple streams of video data capturing different perspectives of related events.

Towards video-based immersive environments

Abstract.Video provides a comprehensive visual record of environment activity over time. Thus, video data is an attractive source of information for the creation of virtual worlds which require some real-world fidelity. This paper describes the use of multiple streams of video data for the creation of immersive virtual environments. We outline our multiple perspective interactive video (MPI-Video) architecture which provides the infrastructure for the processing and analysis of multiple streams of video data. Our MPI-Video system performs automated analysis of the raw video and constructs a model of the environment and object activity within this environment. This model provides a comprehensive representation of the world monitored by the cameras which, in turn, can be used in the construction of a virtual world. In addition, using the information produced and maintained by the MPI-Video system, our immersive video system generates virtual video sequences. These are sequences of the dynamic environment from an arbitrary view point generated using the real camera data. Such sequences allow a user to navigate through the environment and provide a sense of immersion in the scene. We discuss results from our MPI-Video prototype, outline algorithms for the construction of virtual views and provide examples of a variety of such immersive video sequences.

An emerging medium: interactive three-dimensional digital video

This paper describes a new visual medium called interactive 3D digital video. 3D digital video displays motion pictures of real-world events from the view of a virtual camera controlled by the viewer during playback. For 3D video to become of practical use, sophisticated data manipulation, management and processing capabilities are required. These tasks are daunting, given the amount and complexity of data involved. Furthermore, due to the hybrid nature of the 3D video data, no standardized representation and coding schemes are available. We explore these issues and present an overview of a functional system called immersive video. For real events such as basketball games, immersive video analyzes and composites recorded multiviewpoint videos to create a full 3D version of the event which is then encoded and stored for immersive playback. While replaying this 3D digital movie, viewers are able to explore the scene continuously from any perspective. We describe the steps we have taken toward using the Internet infrastructure and a client-server configuration to allow World Wide Web users to interactively view any of our experimental 3D videos including a one minute staged karate demonstration captured by six video cameras. Applications of this new medium include telepresence, interactive video and television, video-based virtual environments, and immersive feature films.

VRML-based WWW interface to MPI Video

The Virtual Reality Modeling Language, VRML, is fast becoming a de facto standard for representation, interaction with, and exchange of three dimensional scenes. The purpose of this paper is threefold: First, we want to emphasize the role of multiple perspective video streams in automatic creation of dynamic three dimensional scenes and in automatic creation of hypermedia markups on objects in the scene. Second, we want to present VRML as a interaction metaphor for interactive television and video systems. The three-dimensional nature of the VRML representation allows a user the ability to view the real scene from virtually any one of an infinite number of viewpoints. The ability to markup arbitrary three dimensional objects allows the user to interact with the real world and query about parts of it. Thirdly, we want to drive the future VRML specification and development towards onthe-fly updating of three dimensional models, a mode that will be amenable to our approach: modeling the 3D world as a combination of a detailed static world, and the changes computed from per-frame integration of motion information from video data.

Content-centric interactive video on the World Wide Web

Abstract The World Wide Web (WWW) as a mechanism to provide access to “real world” information in the form of live or recorded video and audio data is becoming more common. However, access to this information is limited to simple playback. In this paper, we present an architecture and a WWW implementation of Multiple Perspective Interactive Video, MPI Video , an infrastructure to access these new forms of information in more useful ways. Using an information system to merge the sensory and virtual data into a coherent, accessible, dynamic database, MPI Video provides content-centric interactivity . Multiple users can access this database to retrieve disparate information at the same time. Most interactions occur in a three-dimensional interface (a natural medium for interacting with real world data) which combines relevant real and virtual components. This approach is useful in several applications including teleconferencing, remote monitoring, and interactive entertainment. In this paper, we present the concepts of MPI Video and describe the latest implementation, a Web-based Remote Access, Monitoring, and Presence (RAMP) system.