Norihiko Kawai

Image Inpainting Considering Brightness Change and Spatial Locality of Textures and Its Evaluation

Image inpainting is a tequnique for removing undesired visual objects in images and filling the missing regions with plausible textures. Conventionally, the missing parts of an image are completed by optimizing the objective function, which is defined based on pattern similarity between the missing region and the rest of the image (data region). However, unnatural textures are easily generated due to two factors: (1) available samples in the data region are quite limited, and (2) pattern similarity is one of the required conditions but is not sufficient for reproducing natural textures. In this paper, in order to improve the image quality of completed texture, the objective function is extended by allowing brightness changes of sample textures (for (1)) and introducing spatial locality as an additional constraint (for (2)). The effectiveness of these extensions is successfully demonstrated by applying the proposed method to one hundred images and comparing the results with those obtained by the conventional methods.

Diminished reality considering background structures

This paper proposes a new diminished reality method for 3D scenes considering background structures. Most conventional methods using image inpainting assumes that the background around a target object is almost planar. In this study, approximating the background structure by the combination of local planes, perspective distortion of texture is corrected and searching area is limited for improving the quality of image inpainting. The temporal coherence of texture is preserved using the estimated structures and camera pose estimated by Visual-SLAM.

Diminished Reality Based on Image Inpainting Considering Background Geometry

Diminished reality aims to remove real objects from video images and fill in the missing regions with plausible background textures in realtime. Most conventional methods based on image inpainting achieve diminished reality by assuming that the background around a target object is almost planar. This paper proposes a new diminished reality method that considers background geometries with less constraints than the conventional ones. In this study, we approximate the background geometry by combining local planes, and improve the quality of image inpainting by correcting the perspective distortion of texture and limiting the search area for finding similar textures as exemplars. The temporal coherence of texture is preserved using the geometries and camera pose estimated by visual-simultaneous localization and mapping (SLAM). The mask region that includes a target object is robustly set in each frame by projecting a 3D region, rather than tracking the object in 2D image space. The effectiveness of the proposed method is successfully demonstrated using several experimental environments.

Video Completion for Generating Omnidirectional Video without Invisible Areas

Omnidirectional multi-camera systems cannot capture entire fields of view because of their inability to view areas directly below them. Such invisible areas in omnidirectional video decrease the resulting realistic sensation experienced when using a telepresence system. In this study, we generate omnidirectional video without invisible areas using an image completion technique. The proposed method compensates for the change in appearance of textures caused by camera motion and searches for appropriate exemplars considering three-dimensional geometric information. In our experiments, the effectiveness of our proposed method has been demonstrated by successfully filling in missing regions in real video sequences captured using an omnidirectional multi-camera system.

Efficient surface completion using principal curvature and its evaluation

Surface completion is a technique for filling missing regions in 3D models measured by range scanners and videos. Conventionally, although missing regions were filled with the similar shape in a model, the completion process was fairly inefficient because the whole region in the model was searched for the similar shape. In this paper, the completion is efficiently performed using principal curvatures of local shape. In experiments, the effectiveness of the proposed method is successfully verified with subjective evaluation. In addition, the quantitative evaluation which has not been in the literature is newly performed.

Surface completion by minimizing energy based on similarity of shape

3D mesh models generated with range scanner or video images often have holes due to many occlusions by other objects and the object itself. This paper proposes a novel method to fill the missing parts in the incomplete models. The missing parts are filled by minimizing the energy function, which is defined based on similarity of local shape between the missing region and the rest of the object. The proposed method can generate complex and consistent shapes in the missing region. In the experiment, the effectiveness of the method is successfully demonstrated by applying it to complex shape objects with missing parts.

AR marker hiding based on image inpainting and reflection of illumination changes

This paper proposes a new method of diminished reality which removes AR markers from a users view image in real time. To achieve natural marker hiding, assuming that an area around a marker is locally planar, the marker area in the first frame is inpainted using the rectified image to achieve high-quality inpainting. The unique inpainted texture is overlaid on the marker region in each frame according to camera motion for geometric consistency. Both global and local luminance changes around the marker are separately detected and reflected to the inpainted texture for photometric consistency.

Background Estimation for a Single Omnidirectional Image Sequence Captured with a Moving Camera

This paper proposes a background estimation method from a single omnidirectional image sequence for removing undesired regions such as moving objects, specular regions, and uncaptured regions caused by the cam- eras blind spot without manual specification. The proposed method aligns multiple frames using a reconstructed 3D model of the environment and generates background images by minimizing an energy function for selecting a frame for each pixel. In the energy function, we introduce patch similarity and camera positions to remove undesired regions more correctly and generate high-resolution images. In experiments, we demonstrate the effectiveness of the proposed method by comparing the result given by the proposed method with those from conventional approaches.

Surface completion of shape and texture based on energy minimization

In this paper, we propose a novel surface completion method to generate plausible shapes and textures for missing regions of 3D models. The missing regions are filled in by minimizing two energy functions for shape and texture, which are both based on similarities between the missing region and the rest of the object; in doing so, we take into account the positive correlation between shape and texture. We demonstrate the effectiveness of the proposed method experimentally by applying it to two models.

Pano UMECHIKA: A crowded underground city panoramic view system

Toward a really useful navigation system, utilizing spherical panoramic photos with maps like Google Street View is efficient. Users expect the system to be available in all areas they go. Conventional shooting methods obtain the shot position from GPS sensor. However, indoor areas are out of GPS range. Furthermore, most urban public indoor areas are crowded with pedestrians. Even if we blur the pedestrians in a photo, the photos with blurring are not useful for scenic information. Thus, we propose a method which simultaneously subtracts pedestrians based on background subtraction method and generates location metadata by manually input from maps. Using these methods, we achieved an underground panoramic view system which displays no pedestrians.