Mohamad Ivan Fanany

A neural network for recovering 3D shape from erroneous and few depth maps of shaded images

In this paper, we present a new neural network (NN) for three-dimensional (3D) shape reconstruction. This NN provides an analytic mapping of an initial 3D polyhedral model into its projection depth images. Through this analytic mapping, the NN can analytically refine vertices position of the model using error back-propagation learning. This learning is based on shape-from-shading (SFS) depth maps taken from multiple views. The depth maps are obtained by Tsai-Shah SFS algorithm. They are considered as partial 3D shapes of the object to be reconstructed. The task is to reconstruct an accurate and complete representation of a given object relying only on a limited number of views and erroneous SFS depth maps. Through hierarchical reconstruction and annealing reinforcement strategies, our reconstruction system gives more exact and stable results. In addition, it corrects and smoothly fuses the erroneous SFS depth maps. The implementation of this neural network algorithm used in this paper is available at http://kumazawa-www.cs.titech.ac.jp/~fanany/MV-SPRNN/mv-sprnn.html.

A scheme for reconstructing face from shading using smooth projected polygon representation NN

In this paper, we present a neural-network learning scheme for face reconstruction. This scheme, which we called the smooth projected polygon representation neural network (SPPRNN), is able to successively refine the polygons vertices parameter of an initial 3D shape based on depth-maps of several calibrated images taken from multiple views. The depth-maps, which are obtained by deploying the Tsai-Shah shape-from-shading (SFS) algorithm, can be considered as partial 3D shapes of the face to be reconstructed. The reconstruction is finalized by mapping the texture of face images to the initial 3D shape. There are three interesting issues investigated in this paper concerning the effectiveness of this scheme. First, how effective the SFS provides partial 3D shapes compared to if we simply used 2D images. Secondly, how essential a smooth projected polygonal model is in order to approximate the face structure and enhance the convergence rate of this scheme. Thirdly, how an appropriate initial 3D shape should be selected and used in order to improve model resolution and learning stability. By carefully addressing these three issues, it was shown from our experiment that a compact and realistic 3D model of a human (mannequin) face could be obtained.

A neural network scheme for transparent surface modelling

This paper presents a new neural network (NN) scheme for recovering three dimensional (3D) transparent surface. We view the transparent surface modeling, not as a separate problem, but as an extension of opaque surface modeling. The main insight of this work is we simulate transparency not only for generating visually realistic images, but for recovering the object shape. We construct a formulation of transparent surface modeling using ray tracing framework into our NN. We compared this ray tracing method, with a texture mapping method that simultaneously map the silhouette images and smooth shaded images (obtained form our NN), and textured images (obtained from the teacher image) to an initial 3D model. By minimizing the images error between the output images of our NN and the teacher images, observed in multiple views, we refine vertices position of the initial 3D model. We show that our NN can refine the initial 3D model obtained by polarization images and converge into more accurate surface.

A combinatorial transparent surface modeling from polarization images

This paper presents a combinatorial (decision tree induction) technique for transparent surface modeling from polarization images. This technique simultaneously uses the object’s symmetry, brewster angle, and degree of polarization to select accurate reference points. The reference points contain information about surface’s normals position and direction at near occluding boundary. We reconstruct rotationally symmetric objects by rotating these reference points.

Analysis of shape from shading algorithms for fast and realistic 3D face reconstruction APCCAS2002

The aim of this paper is to attain a fast and realistic 3D face reconstruction, when only a limited set of images taken from multiple views, is available. Instead of relying merely on 2D images as suggested by image-based modeling approaches, we investigate several shape-from-shading (SFS) techniques, which have been studied extensively in computer vision research. This is because we believe that a nearly complete set of images required by image-based modeling techniques is rarely available in real world applications. In this paper, we investigate the effectiveness of three different SFS algorithms to provide partial 3D shapes of the face to be reconstructed. Each algorithm is selected from three different classes of SFS technique, i.e., linear, propagation, and minimization approaches. The reconstruction process is performed by our novel neural network learning scheme, which is able to successively refine the polygon vertices parameter of an initial 3D shape, based on depth maps of several calibrated images. To evaluate the reconstruction result based on those SFS techniques, we measure average vertex-error and pixel-error compared to actual 3D data obtained by 3D scanner device. We also compared these result with those obtained by using only 2D images. In addition, we also measure total computational time needed in the reconstruction process.

SA-Optimized Multiple View Smooth Polyhedron Representation NN

Simulated Annealing (SA) is a powerful stochastic search method that can produce very high quality solutions for hard combinatorial optimization problem. In this paper, we applied this SA method to optimize our 3D hierarchical reconstruction neural network (NN). This NN deals with complicated task to reconstruct a complete representation of a given object relying only on a limited number of views and erroneous depth maps of shaded images. The depth maps are obtained by Tsai-Shah shape-from-shading (SFS) algorithm. The experimental results show that the SA optimization enable our reconstruction system to escape from a local minima. Hence, it gives more exact and stable results with small additional computation time.

Analytic Reconstruction of Transparent and Opaque Surfaces from Texture Images

This paper addresses the problem of reconstructing non- overlapping transparent and opaque surfaces from multiple view images. The reconstruction is attained through progressive refinement of an initial 3D shape by minimizing the error between the images of the object and the initial 3D shape. The challenge is to simultaneously reconstruct both the transparent and opaque surfaces given only a limited number of images. Any refinement methods can theoretically be applied if analytic relation between pixel value in the training images and vertices position of the initial 3D shape is known. This paper investigates such analytic relations for reconstructing opaque and transparent surfaces. The analytic relation for opaque surface follows diffuse reflection model, whereas for transparent surface follows ray tracing model. However, both relations can be converged for reconstruction both surfaces into texture mapping model. To improve the reconstruction results several strategies including regularization, hierarchical learning, and simulated annealing are investigated.

A neural network for simultaneously reconstructing transparent and opaque surfaces

This paper presents a neural network (NN) to recover three-dimensional (3D) shape of an object from its multiple view images. The object may contain non-overlapping transparent and opaque surfaces. The challenge is to simultaneously reconstruct the transparent and opaque surfaces given only a limited number of views. By minimizing the pixel error between the output images of this NN and teacher images, we want to refine vertices position of an initial 3D polyhedron model to approximate the true shape of the object. For that purpose, we incorporate a ray tracing formulation into our NN’s mapping and learning. At the implementation stage, we develop a practical regularization learning method using texture mapping instead of ray tracing. By choosing an appropriate regularization parameter and optimizing using hierarchical learning and annealing strategies, our NN gives more approximate shape.