Appu Shaji

Monocular 3D Reconstruction of Locally Textured Surfaces

Most recent approaches to monocular nonrigid 3D shape recovery rely on exploiting point correspondences and work best when the whole surface is well textured. The alternative is to rely on either contours or shading information, which has only been demonstrated in very restrictive settings. Here, we propose a novel approach to monocular deformable shape recovery that can operate under complex lighting and handle partially textured surfaces. At the heart of our algorithm are a learned mapping from intensity patterns to the shape of local surface patches and a principled approach to piecing together the resulting local shape estimates. We validate our approach quantitatively and qualitatively using both synthetic and real data.

Simultaneous point matching and 3D deformable surface reconstruction

It has been shown that the 3D shape of a deformable surface in an image can be recovered by establishing correspondences between that image and a reference one in which the shape is known. These matches can then be used to set-up a convex optimization problem in terms of the shape parameters, which is easily solved. However, in many cases, the correspondences are hard to establish reliably. In this paper, we show that we can solve simultaneously for both 3D shape and correspondences, thereby using 3D shape constraints to guide the image matching and increasing robustness, for example when the textures are repetitive. This involves solving a mixed integer quadratic problem. While optimizing this problem is NP-hard in general, we show that its solution can nevertheless be approximated effectively by a branch-and-bound algorithm.

Riemannian manifold optimisation for non-rigid structure from motion

This paper address the problem of automatically extracting the 3D configurations of deformable objects from 2D features. Our focus in this work is to build on the observation that the subspace spanned by the motion parameters is a subset of a smooth manifold, and therefore we hunt for the solution in this space, rather than use heuristics (as previously attempted earlier). We succeed in this by attaching a canonical Riemannian metric, and using a variant of the non-rigid factorisation algorithm for structure from motion. We qualitatively and quantitatively show that our algorithm produces better results when compared to the state of art.

Joint statistical analysis of images and keywords with applications in semantic image enhancement

With the advent of social image-sharing communities, millions of images with associated semantic tags are now available online for free and allow us to exploit this abundant data in new ways. We present a fast non-parametric statistical framework designed to analyze a large data corpus of images and semantic tag pairs and find correspondences between image characteristics and semantic concepts. We learn the relevance of different image characteristics for thousands of keywords from one million annotated images. We demonstrate the frameworks effectiveness with three different examples of semantic image enhancement: we adapt the gray-level tone-mapping, emphasize semantically relevant colors, and perform a defocus magnification for an image based on its semantic context. The performance of our algorithms is validated with psychophysical experiments.

Image summaries using database saliency

It is useful to have a small set of representative images from a database of thousands of images to summarize its content. There are two key aspects of such image summaries: how to generate them and how to present them. We address both issues. We extend the idea of image saliency to databases and introduce the notion of database saliency. We argue that in image databases, there are certain images that are more uncommon or salient than others and therefore are more interesting. We compute the database saliency value of an image as its total distance from all the pre-defined cluster centers of the database. We demonstrate the use of database saliency in two visualization applications: creating image collages and mosaics using automatically chosen salient images.

Manifold optimisation for motion factorisation

This paper presents a novel formulation for the popular factorisation based solution for Structure from Motion. Since our measurement matrices are populated with incomplete and inaccurate data, SVD based total least squares solution are less than appropriate. Instead, we approach the problem as a non-linear unconstrained minimisation problem on the product manifold of the Special Euclidean Group (SE3). The restriction of the domain of optimisation to the SE3 product manifold not only implies that each intermediate solution is a plausible object motion, but also ensures better intrinsic stability for the minimisation algorithm. We compare our method with existing state of art, and show that our algorithm exhibits superior performance.

Saliency Detection using regression trees on hierarchical image segments

The currently best performing state-of-the-art saliency detection algorithms incorporate heuristic functions to evaluate saliency. They require parameter tuning, and the relationship between the parameter value and visual saliency is often not well understood. Instead of using parametric methods we follow a machine learning approach, which is parameter free, to estimate saliency. Our method learns data-driven saliency-estimation functions and exploits the contributions of visual properties on saliency. First, we over-segment the image into superpixels and iteratively connect them to form hierarchical image segments. Second, from these segments, we extract biologically-plausible visual features. Finally, we use regression trees to learn the relationship between the feature values and visual saliency. We show that our algorithm outperforms the most recent state-of-the-art methods on three public databases.

Resolving occlusion in multiframe reconstruction of deformable surfaces

Occlusion is troublesome for almost all computer vision algorithms. To a certain extent, the difficulty is alleviated when multiple frames are given. On the other hand, when we consider the recovery of shapes of moving deformable objects, observed using a monocular camera, the problem appears difficult again. In this paper, we show a method that outperforms previous approaches to reconstruction when feature data is unavailable, perhaps due to occlusion. Our key intuition is that portions of the surface that are visible in some frame can be reliably reconstructed in that frame; further, the reliable portions can be stitched together to find even missing portions, much the way a human eye would hallucinate. Our techniques are based on optimization in Riemannian shape spaces, and is demonstrated on isometric surfaces without involving any kind of machine learning methods.

Vision-Based posing of 3d virtual actors

Construction of key poses is one of the most tedious and time consuming steps in synthesizing of 3D virtual actors. Recent alternate schemes expect the user to specify two inputs. Along with a neutral 3D reference model, more intuitive 2D inputs such as sketches, photographs or video frames are provided. Using these, of all the possible configurations, the “best” 3D virtual actor is posed. In this paper, we provide a solution to this ill-posed problem. We first give a solution to the problem of finding an approximate view consistent with the 2D sketch. Elements of this rigid-body solution are novel. Next, we provide a new solution to the process of extending or retracting limbs to more accurately suit the sketch. This posing algorithm, is based on a search based scheme inspired by anthropometric evidence. Less physical work is required by the actor to reach the desired pose from the base position. We also show that our algorithm converges to an acceptable solution much faster compared to the previous methods.

Creating personalized jigsaw puzzles

Designing aesthetically pleasing and challenging jigsaw puzzles is considered an art that requires considerable skill and expertise. We propose a tool that allows novice users to create customized jigsaw puzzles based on the image content and a user-defined curve. A popular design choice among puzzle makers, called color line cutting, is to cut the puzzle along the main contours in an image, making the puzzle both aesthetically interesting and challenging to solve. At the same time, the puzzle maker has to make sure that puzzle pieces interlock so that they do not disassemble easily. Our method automatically optimizes for puzzle cuts that follow the main contours in the image and match the user-defined curve. We handle the tradeoff between color line cutting and interlocking, and we introduce a linear formulation for the interlocking constraint. We propose a novel method for eliminating self-intersections and ensuring a minimum width in our output curves. Our method satisfies these necessary fabrication constraints in order to make valid puzzles that can be easily realized with present-day laser cutters.