Benjamin Kunsberg

The differential geometry of shape from shading: Biology reveals curvature structure

Shape from shading is a classical inverse problem in computer vision. We introduce a novel mathematical formulation for calculating local surface shape based on covariant derivatives, rather than the customary integral minimization or P.D.E approaches. Motivated by neurobiology, we introduce the shading flow field (the tangent map to the image isophotes) between the image and the surface levels. Just as in the perceptual organization of texture, we use the parallel transport of our shading flow field to move the isophote field at different points on the unknown surface to a single point, amassing restrictions on our surface curvatures. Under simplifying assumptions we solve exactly for the light source/surface pairs needed for a local image patch to have a given shading flow. The magnitude of the brightness gradient then restricts this family to a single light source and surface estimate pair, up to the concave/convex ambiguity and an additional elliptical/saddle ambiguity. Example calculations illustrate our approach.

How Shading Constrains Surface Patches without Knowledge of Light Sources

Shape-from-shading (SFS) is a classical inverse problem in computer and human vision. This shape reconstruction problem is inherently ill-posed. We show that the isophotes on smooth surfaces with Lambertian reflectance can be directly related to surface properties without consideration of the light source. Using techniques from modern differential geometry, we derive relationships between the curvature of the isophotes and the shape operator for the surface. Neurobiology motivates the geometric approach, and our calculations allow us to characterize the matching local family of surfaces that can result from any given shading patch. We illustrate the local ambiguity in several examples.

Why Shading Matters along Contours

Shape from shading is a classical inverse problem in computer vision. It is inherently ill-defined and, in different formulations, it depends on the assumed light source direction. In contrast to these mathematical difficulties, we introduce a novel mathematical formulation for calculating local surface shape based on covariant derivatives of the shading flow field, rather than the customary integral minimization or P.D.E approaches.Working with the shading flow field rather than the original image intensities is important to both neurogeometry and neurophysiology. To make the calculations concrete we assume a Lambertian model for image formation, butwe do not make global light source positional assumptions. In particular, for smooth surfaces in generic position, we show that second derivatives of brightness are independent of the light sources and can be directly related to surface properties. We use these measurements to define the matching local family of surfaces that could result from a given shading patch. In total our results change the emphasis from seeking a single, well-define solution to an ill-posed problem to characterizing the ambiguity in possible solutions to this problem. The result is relevant both mathematically and perceptually, because we then show how the equations simplify and the ambiguity reduces are certain critical points of intensity. We conclude with a discussion of image reconstruction at these critical points.

Global constraints on integral curves of shaded surfaces

The problem of shape from shading has been studied for many decades. Due to its ill-posed nature, almost every approach has attempted to reduce the ambiguity at the start. By making local assumptions regarding the light sources and surface, the problem can be reduced in complexity. However, due to the many ad hoc constraints, the resulting algorithms are often brittle and uninformative in describing possible biological structure. Rather, we use shape from shading as a test model to understand how the brain resolves geometric ambiguity. Our entire approach (VSS work 2012 - 2015) has been to mathematically represent ambiguity until it is at a scale sufficient to resolve itself. To this end, we have considered increasingly larger-scale features: local orientation flows (2012), critical contours (2013), and ridges (2013). In this work, we consider the global scale. There are global themes in an image that tie together the local patches. The difficulty is quantitatively describing these constraints without prior knowledge of the local patches. What does the boundary shape of a balloon animal tell you about the total air inside the balloon? We derive two theorems on the global geometry of a Lambertian surface. These are applicable regardless of light source direction or local geometry. Thus, they can be applied directly to constrain the local patches. One theorem restricts the total Gaussian curvature of the surface. The second is a relationship between the geodesic curvature of an isophote and its level value. To our knowledge, this is the first time that global constraints on isophotes have been derived. We prove these theorems and show experimental evidence illustrating their accuracy and use. Meeting abstract presented at VSS 2015.