Hideki Mitsumine

High-definition real-time depth-mapping TV camera: HDTV Axi-Vision Camera.

We have developed a field-worthy, high-definition, real-time depth-mapping television camera called the HDTV Axi-Vision Camera. The camera can simultaneously capture both an ordinary HDTV color image and a depth image of objects on more than 1280 x 720 pixels at a frame rate of 29.97 Hz, or on 853 x 480 pixels at a frame rate of 59.94 Hz. The number of detectable pixels per unit time was increased by about 5 times that of the prototype camera by improving the sensitivity and resolution of the depthmapping camera. Short video clips demonstrate how depth information from the camera can be used to create a virtual image in actual television program production.

Immersive virtual studio

(a) (b) (c) (d) Figure 1: (a) System view, (b) Inside view, (c) and (d) Real-time composite images.

Real space-based virtual studio seamless synthesis of a real set image with a virtual set image

When making a TV program in a studio, care must be taken that the camera does not shoot beyond the boundary of the studio set. In addition, limitations in cost and space for the set must be taken into account. In a virtual studio, on the other hand, we can solve this cost and space problem, but in turn, actors are requested to perform in front of a blue background screen, which is not always an easy task for them. To solve these problems associated with real studios and virtual studios, we have developed a new type of virtual studio called Real Space-based Virtual Studio in which a real space image and virtual space image are combined naturally with no boundary seam. There are two major advantages in using this new virtual-real hybrid system. One is that the actors can concentrate on their role in the real studio sets, and the other is that camera work can be done without worrying about off-screen areas of the set. In the present study, we constructed an omnidirectional image with ultra high-definition features and combined it as a virtual studio image with a real studio image. We have developed an integration system and from the experimentation we have shown that the omnidirectional images and the real studio images can combine smoothly and naturally.

Virtual Museum-3-D fine art appreciation system

We have developed an interactive multimedia system for the appreciation of three-dimensional art objects, such as porcelain and sculptures. With the system, the viewer is able to see a representation of the object from various directions as if holding it in his/her hands. He/she can enjoy the object images on high-quality HDTV (Hi-Vision) display, with supplementary audio and written information about the object. This system consists of (1) a video server, to store the HDTV images of the fine arts, which have been shot from various paths around the fine arts at 0.2 degree intervals; (2) a specially designed manipulator to select a desired image from the server; and (3) an HDTV or stereoscopic HDTV display, and accompanying spectacles. The HDTV video server can randomly access data at a video rate (1/60 sec), so as to quickly respond to the operations of the manipulator. The experimental system has been tested by many artists and scientists for operability and effectiveness, and has been confirmed as useful for art galleries and other educational institutions.

Construction of omnidirectional images for image-based virtual studio

To generate highly realistic scenes of a virtual studio, we are developing technology of a new virtual studio based on image components from real videos instead of CG. And we call this system an Image-Based Virtual Studio. Two types of image components are now being developed for the system. These are an environmental image component for long distant view and a 3D image component for short distant view.

Camera Parameter Estimation Method Using Infrared Markers for Live TV Production

We have developed a robust method for estimating camera parameters for live TV production based on infrared markers whose feature-point extraction is simple and two-dimensional color histogram matching that considers the effects of specular reflection due to lighting. We first explain the principle of the proposed technique. We then present the results of a basic experiment for evaluating the accuracy of marker identification and the results of an image-compositing experiment using estimated camera parameters. We show that the proposed technique is effective based on those results.

Accurate camera calibration method specialized for virtual studios

Virtual studio is a popular technology for TV programs, that makes possible to synchronize computer graphics (CG) to realshot image in camera motion. Normally, the geometrical matching accuracy between CG and realshot image is not expected so much on real-time system, we sometimes compromise on directions, not to come out the problem. So we developed the hybrid camera calibration method and CG generating system to achieve the accurate geometrical matching of CG and realshot on virtual studio. Our calibration method is intended for the camera system on platform and tripod with rotary encoder, that can measure pan/tilt angles. To solve the camera model and initial pose, we enhanced the bundle adjustment algorithm to fit the camera model, using pan/tilt data as known parameters, and optimizing all other parameters invariant against pan/tilt value. This initialization yields high accurate camera position and orientation consistent with any pan/tilt values. Also we created CG generator implemented the lens distortion function with GPU programming. By applying the lens distortion parameters obtained by camera calibration process, we could get fair compositing results.

Real-time ball-position measurement using multi-view cameras for live football broadcast

We developed a robust ball-position-measurement system that can be used for actual football games. It measures the positions of a ball by integrating tracking results from multi-view cameras in real time. The positions can be used for calculating various statistical data and the production of live football broadcasts. Although several systems for visual object tracking have been proposed, tracking a ball during entire football games is still challenging due to many difficulties such as occlusion and illumination changes. We focused on the motion features of a ball object in addition to its appearance features. Tracking results from multi-view cameras are integrated in consideration of the reliability of these results, and the final ball position is robustly measured. The system was evaluated with videos shot during four professional football games held at multiple stadiums during the day and night. The evaluation results and an experimental demonstration indicate that the proposed system can robustly measure the position of a football in real time and is effective for producing live football broadcasts.

A handy system for natural composition of CG and real scene with real-time reflection of lighting changes

A handy system is presented for creating in real time a natural television image composed of a real scene and a computer graphics object. The light positions and colors (RGB values) in the real scene are estimated and then applied to the object by using simple equipment. Objective and subjective experiments demonstrated the effectiveness of this system. An enhanced algorithm is also presented that improves the accuracy of light estimation.

A new real-time video synthesis method for virtual studio environments using GPU and projected screens

This paper describes a new supportive information projecting technique for virtual TV studio environments with a back projected screen which is not visible for the audience, but visible for actors. Traditional TV studios use blue or green back chroma-key for video composition, thus the actors cannot see the final composite without the preview monitor. When the actors point or watch a point on the background image, they need an extensive experience or rehearsals. In our system, the actors can see and point the supportive information displays; such as computer generated background, virtual actors, scripts and/or final composites around them.