Masaaki Iiyama

Graphic and movie illustrations of human prenatal development and their application to embryological education based on the human embryo specimens in the Kyoto collection

Morphogenesis in the developing embryo takes place in three dimensions, and in addition, the dimension of time is another important factor in development. Therefore, the presentation of sequential morphological changes occurring in the embryo (4D visualization) is essential for understanding the complex morphogenetic events and the underlying mechanisms. Until recently, 3D visualization of embryonic structures was possible only by reconstruction from serial histological sections, which was tedious and time‐consuming. During the past two decades, 3D imaging techniques have made significant advances thanks to the progress in imaging and computer technologies, computer graphics, and other related techniques. Such novel tools have enabled precise visualization of the 3D topology of embryonic structures and to demonstrate spatiotemporal 4D sequences of organogenesis. Here, we describe a project in which staged human embryos are imaged by the magnetic resonance (MR) microscope, and 3D images of embryos and their organs at each developmental stage were reconstructed based on the MR data, with the aid of computer graphics techniques. On the basis of the 3D models of staged human embryos, we constructed a data set of 3D images of human embryos and made movies to illustrate the sequential process of human morphogenesis. Furthermore, a computer‐based self‐learning program of human embryology is being developed for educational purposes, using the photographs, histological sections, MR images, and 3D models of staged human embryos. Developmental Dynamics 235:468–477, 2006.

Estimation of the location of joint points of human body from successive volume data

Recognizing structure of the human body is important for modeling human motion. The human body is usually represented as an articulate model, which consists of the rigid parts and the joint points between them. The structure of the human body is specified by the joint points. We propose a method for estimating the location of joint points from successive volume data. Our joint point estimation method consists of three steps. In the first step, rigid parts are extracted from two successive volume data under the constraint of the rigid transformation. In the second step, the joint points are estimated based on the rigid parts. As the last step, false joint points are eliminated by using more successive data. Applying the method to the simulated data, the locations of joint points of the human are correctly estimated.

Silhouette Extraction with Random Pattern Backgrounds for the Volume Intersection Method

In this paper, we present a novel approach for extracting silhouettes by using a particular pattern that we call the random pattern. The volume intersection method reconstructs the shapes of 3D objects from their silhouettes obtained with multiple cameras. With the method, if some parts of the silhouettes are missed, the corresponding parts of the reconstructed shapes are also missed. When colors of the objects and the backgrounds are similar, many parts of the silhouettes are missed. We adopt random pattern backgrounds to extract correct silhouettes. The random pattern has many small regions with randomly-selected colors. By using the random pattern backgrounds, we can keep the rate of missing parts below a specified percentage, even for objects of unknown color. To refine the silhouettes, we detect and fill in the missing parts by integrating multiple images. From the images captured by multiple cameras used to observe the object, the objects colors can be estimated. The missing parts can be detected by comparing the objects color with its corresponding backgrounds color. In our experiments, we confirmed that this method effectively extracts silhouettes and reconstructs 3D shapes.

Super-Resolution Texture Mapping from Multiple View Images

This paper presents an artifact-free super resolution texture mapping from multiple-view images. The multiple-view images are upscaled with a learning-based super resolution technique and are mapped onto a 3D mesh model. However, mapping multiple-view images onto a 3D model is not an easy task, because artifacts may appear when different upscaled images are mapped onto neighboring meshes. We define a cost function that becomes large when artifacts appear on neighboring meshes, and our method seeks the image-and mesh assignment that minimizes the cost function. Experimental results with real images demonstrate the effectiveness of our method.

Usage of needle maps and shadows to overcome depth edges in depth map reconstruction

Photometric stereo is a method of recovering surface normals (needle map) from images. The surface integral of surface normals is used to reconstruct a depth map; however, the depth edges, which are discontinuous boundaries of the depth map, pose a problem for photometric stereo. When the surface of objects includes depth edges, the reconstructed depth map may contain errors. To solve this problem, we detect depth edges using shadows and compute a relative depth between two distant points using the widths of the corresponding shadows. We define an error function and reconstruct the depth map by minimizing the error function. Experimental results with synthetic and with real image data demonstrate the effectiveness of our approach.

Reconstructing the arbitrary view of an object using the multiple camera system

A new approach for reconstructing the arbitrary views of an object is proposed. The images are grabbed by the multiple camera system. The corresponding pairs among the images are determined from the relation of the camera positions. The exact depth measurement of each pixel of an object has been determined from the multiple image pair. The synthesized images observed from new viewing points are generated by the disparities between two different viewing point images. A panoramic image of a scene is reconstructed by the images of a real scene have been done to confirm the effectiveness of the proposed method.

Detection of social interaction from observation of daily living environments

In this article, we discuss how to detect occasional social interaction by a group of people in an open space such as a hall by observing the environment by cameras. Since it is known in the field of social psychology that some characteristic arrangement is maintained by each group of people during interaction, previous works have tried to detect social interaction based on the arrangement. However, these methods could confuse different groups especially when those groups are located close to each other, because the methods only consider direct relationship among the positions or orientations of the people for finding the characteristic arrangement. We propose a new region-based approach, which focuses on the spatial region to be occupied exclusively by each group of the people, introducing a technique for region extraction used in the field of image processing.

3D shape reconstruction from incomplete silhouettes in multiple frames

3D shapes are reconstructed from silhouettes obtained by multiple cameras with the volume intersection method. In recent work, methods of integrating silhouettes in time sequences have been proposed. The number of silhouettes can be increased by integrating silhouettes in multiple frames. The silhouettes of a rigid object in multiple frames are integrated with its rigid motion. This motion is often estimated with 3D feature points extracted from silhouettes. When the estimated motion has large error, shapes are reconstructed with missing parts. This error is given by the incomplete extraction of 3D feature points, which is caused by additional and missing regions of extracted silhouettes. We cannot prevent silhouettes from being extracted with the additional and missing regions in real environments. Here, we propose an intelligent method of integrating incomplete silhouettes where outcrop points, which are 3D feature points for estimating motion, play an important role. The reconstructed shape can be evaluated referring to how many outcrop points have been included in the reconstructed shape of another frame. Although the evaluation does not represent the accuracy of estimated motion directly, it does guarantee that outstanding parts will be preserved in the reconstructed shape. Silhouettes in multiple frames can be integrated with fewer missing and additional parts based on this evaluation.

Variational Bayesian Approach to Multiframe Image Restoration

Image restoration is a fundamental problem in the field of image processing. The key objective of image restoration is to recover clean images from images degraded by noise and blur. Recently, a family of new statistical techniques called variational Bayes (VB) has been introduced to image restoration, which enables us to automatically tune parameters that control restoration. While information from one image is often insufficient for high-quality restoration, however, current state-of-the-art methods of image restoration via VB approaches use only a single-degraded image to recover a clean image. In this paper, we propose a novel method of multiframe image restoration via a VB approach, which can achieve higher image quality while tuning parameters automatically. Given multiple degraded images, this method jointly estimates a clean image and other parameters, including an image warping parameter introduced for the use of multiple images, through Bayesian inference that we enable by making full use of VB techniques. Through various experiments, we demonstrate the effectiveness of our multiframe method by comparing it with single-frame one, and also show the advantages of our VB approach over non-VB approaches.

Recognizing conversation groups in an open space by estimating placement of lower bodies

This article discusses the problem of recognizing groups of people in conversation with each other in an open space. Previous work on this problem takes an approach based on the knowledge that people in the same conversation group often makes a circular formation called an F-formation, by referring to the work in social psychology. Since the F-formation describes spatial and orientational characteristics of the lower bodies of people, their positions and orientations need to be obtained for employing the approach. However, it is difficult to observe lower bodies by cameras, especially for people in a circular formation, due to occlusions between their bodies. We propose to recognize conversation groups while estimating the positions and orientations of the lower bodies of the people only from the information of their head positions and facial orientations observable from a single camera, by considering that the lower bodies of the people are arranged in a circular pattern if they are in the same conversation group.