Noboru Nakajima

High-resolution video mosaicing for documents and photos by estimating camera motion

Recently, document and photograph digitization from a paper is very important for digital archiving and personal data transmission through the internet. Though many people wish to digitize documents on a paper easily, now heavy and large image scanners are required to obtain high quality digitization. To realize easy and high quality digitization of documents and photographs, we propose a novel digitization method that uses a movie captured by a hand-held camera. In our method, first, 6-DOF(Degree Of Freedom) position and posture parameters of the mobile camera are estimated in each frame by tracking image features automatically. Next, re-appearing feature points in the image sequence are detected and stitched for minimizing accumulated estimation errors. Finally, all the images are merged as a high-resolution mosaic image using the optimized parameters. Experiments have successfully demonstrated the feasibility of the proposed method. Our prototype system can acquire initial estimates of extrinsic camera parameters in real-time with capturing images.

Video mosaicing based on structure from motion for distortion-free document digitization

This paper presents a novel video mosaicing method capable of generating a geometric distortion-free mosaic image using a hand-held camera. For a document composed of curved pages, mosaic images of virtually flattened pages are generated. The process of our method is composed of two stages : real-time stage and off-line stage. In the realtime stage, image features are automatically tracked on the input images, and the viewpoint of each image as well as the 3-D position of each image feature are estimated by a structure-from-motion technique. In the offline stage, the estimated viewpoint and 3-D position of each feature are refined and utilized to generate a geometric distortion-free mosaic image. We demonstrate our prototype system on curved documents to show the feasibility of our approach.

Video Mosaicing for Curved Documents Based on Structure from Motion

Various methods for video mosaicing have been already investigated by many researchers. Most of these methods, however, assume that the target object is flat or very far from the camera to avoid the disparity problem. This paper describes a novel video mosaicing method for curved documents based on 3-D reconstruction. With the proposed method, the mosaic image of the geometrically restored target document is generated, even if the document has a curved surface. Experiments on curved documents have shown the feasibility of the proposed method.

Document reconstruction and recognition from an image sequence

A new method for reconstructing a document from an image sequence and recognizing contents of the document is described. In this method frame images in the image sequence are acquired as partial sub-images by a moving camera. The reconstruction includes a rather expensive registration process. To make its use practical, the proposed method registers sub-images hierarchically by sub-tree matching of the layout structure. Images of a corresponding character pair are piled using sub-pixel precision displacement and interpolated to restore image quality and resolution. The proposed method increased the character recognition rate by 6.8 points to 93.1% when resolution was 133 dpi. The computational cost was reduced to 1/23 that of registration by conventional matching.

Analysis of address layout on Japanese handwritten mail-a hierarchical process of hypothesis verification

This paper describes a new method for analyzing Japanese mail layout. We first classify the many kinds of variations in Japanese mail layout into format variations, extraneous object variations, and arrangement variations of destination address characters. These variations are cyclically dependent on one another, but have insufficient information to fix other variations. To solve these problems, we developed an address layout analysis method using a hierarchical hypothesis verification process, which consists of a format determination process and a destination address character line detection process, hierarchically. This method increased the correct address format determination rate by 5.8% to 86.6% and the correct destination address character line detection rate by 10.2% to 92.8% for 1482 test pieces of mail. This method is practically applied to Japanese mail sorting machine.

Document Layout and Reading Sequence Analysis by Extended Split Detection Method

This paper describes an Extended Split Detection Method that can hierarchically segment a machine-printed page image with a complex layout into smaller layout elements. The method performs piecewise-linear segmentation using many kinds of separator elements such as field separators, lines, edges of figures, and edges of white background areas. Furthermore, this method represents an analyzed layout of a hierarchical structure in a tree data structure, in which all nodes are traversed according to the simple rules for generating the reading sequence. We demonstrated that the new method increases the correct character line segmentation rate by 15.5%, to 95.5%, and we achieved a correct reading sequence generation of 88.1%.

A method of analyzing the handling of paper documents in motion images

In this paper, we describe a new system for analyzing the motions involved in handling documents on a desk. Video images are captured by a camera mounted above the desk and the motion which occurs when someone handles the document is classified as movement of the document, turning of the pages, or finger pointing. Furthermore, the system can detect the appearance of fresh regions as a result of movement of the document and the turning of its pages and it can also detect the positions of fingers in the image to realize a natural pointing function. We evaluated implementations of some components of the system and found that they provide sufficient performance to act as parts of a user interface for an interactive document management system.

A Method for Determining Address Format in the Automated Sorting of Japanese Mail

This paper presents a new method for determining the address format used on mail that has been hand-written in Japanese, a task made particularly difficult by the variety of formats possible in Japanese. We classify possible formats into six types, and identify the distinguishing features of each. In the proposed method, features characterizing any of the six types are identified for a given address, and from this a list of format-candidates is generated. Character lines are then determined for any format candidate, and one candidate is subsequently selected on the basis of the statistical likelihood of any address being written in that particular way, given the location and size of the character lines. We also demonstrate the effectiveness of the new method experimentally.

Super-Resolved video mosaicing for documents based on extrinsic camera parameter estimation

This paper describes a novel video mosaicing method based on extrinsic camera parameter estimation. With our method, a mosaic image without perspective distortion can be generated, even if none of the input image plane is parallel to the target document. Thus, users no longer have to take special care in holding the camera so that the image plane in the reference frame is parallel to the target. First, extrinsic camera parameters are estimated by tracking image features. Next, by utilizing re-appearing features, estimated extrinsic camera parameters are globally optimized to minimize the estimation error in the whole input sequence. Finally, all the images are projected onto the mosaic image plane, and a super-resolved mosaic image is generated by applying an iterative back projection algorithm. Experiments have successfully demonstrated the feasibility of the proposed method.