Guy Ben-Yosef

A Tangent Bundle Theory for Visual Curve Completion

Visual curve completion is a fundamental perceptual mechanism that completes the missing parts (e.g., due to occlusion) between observed contour fragments. Previous research into the shape of completed curves has generally followed an “axiomatic” approach, where desired perceptual/geometrical properties are first defined as axioms, followed by mathematical investigation into curves that satisfy them. However, determining psychophysically such desired properties is difficult and researchers still debate what they should be in the first place. Instead, here we exploit the observation that curve completion is an early visual process to formalize the problem in the unit tangent bundle R2 × S1, which abstracts the primary visual cortex (V1) and facilitates exploration of basic principles from which perceptual properties are later derived rather than imposed. Exploring here the elementary principle of least action in V1, we show how the problem becomes one of finding minimum-length admissible curves in R2 × S1. We formalize the problem in variational terms, we analyze it theoretically, and we formulate practical algorithms for the reconstruction of these completed curves. We then explore their induced visual properties vis-à-vis popular perceptual axioms and show how our theory predicts many perceptual properties reported in the corresponding perceptual literature. Finally, we demonstrate a variety of curve completions and report comparisons to psychophysical data and other completion models.

Minimum length in the tangent bundle as a model for curve completion

The phenomenon of visual curve completion, where the visual system completes the missing part (e.g., due to occlusion) between two contour fragments, is a major problem in perceptual organization research. Previous computational approaches for the shape of the completed curve typically follow formal descriptions of desired, image-based perceptual properties (e.g, minimum total curvature, roundedness, etc.). Unfortunately, however, it is difficult to determine such desired properties psychophysically and indeed there is no consensus in the literature for what they should be. Instead, in this paper we suggest to exploit the fact that curve completion occurs in early vision in order to formalize the problem in a space that explicitly abstracts the primary visual cortex. We first argue that a suitable abstraction is the unit tangent bundle R2 × S1 and then we show that a basic principle of “minimum energy consumption” in this space, namely a minimum length completion, entails desired perceptual properties for the completion in the image plane. We present formal theoretical analysis and numerical solution methods, we show results on natural images and their advantage over existing popular approaches, and we discuss how our theory explains recent findings from the perceptual literature using basic principles only.

Tangent bundle curve completion with locally connected parallel networks

We propose a theory for cortical representation and computation of visually completed curves that are generated by the visual system to fill in missing visual information (e.g., due to occlusions). Recent computational theories and physiological evidence suggest that although such curves do not correspond to explicit image evidence along their length, their construction emerges from corresponding activation patterns of orientation-selective cells in the primary visual cortex. Previous theoretical work modeled these patterns as least energetic 3D curves in the mathematical continuous space , which abstracts the mammalian striate cortex. Here we discuss the biological plausibility of this theory and present a neural architecture that implements it with locally connected parallel networks. Part of this contribution is also a first attempt to bridge the physiological literature on curve completion with the shape problem and a shape theory. We present completion simulations of our model in natural and synthetic scenes and discuss various observations and predictions that emerge from this theory in the context of curve completion.

Tangent Bundle Elastica and Computer Vision

Visual curve completion, an early visual process that completes the occluded parts between observed boundary fragments (a.k.a. inducers), is a major problem in perceptual organization and a critical step toward higher level visual tasks in both biological and machine vision. Most computational contributions to solving this problem suggest desired perceptual properties that the completed contour should satisfy in the image plane, and then seek the mathematical curves that provide them. Alternatively, few studies (including by the authors) have suggested to frame the problem not in the image plane but rather in the unit tangent bundleR 2 × S1, the space that abstracts the primary visual cortex, where curve completion allegedly occurs. Combining both schools, here we propose and develop a biologically plausible theory of elastica in the tangent bundle that provides not only perceptually superior completion results but also a rigorous computational prediction that inducer curvatures greatly affects the shape of the completed curve, as indeed indicated by human perception.

Curvature-based perceptual singularities and texture saliency with early vision mechanisms

Recent work has shown that salient perceptual singularities occur in visual textures even in the absence of feature gradients. In smoothly varying orientation-defined textures, these striking non-smooth percepts can be predicted from two texture curvatures, one tangential and one normal [Proc. Natl. Acad. Sci. USA103, 15704 (2006)]. We address the issue of detecting these perceptual singularities in a biologically plausible manner and present three different models to compute the tangential and normal curvatures using early cortical mechanisms. The first model relies on the response summation of similarly scaled even-symmetric simple cells at different positions by utilizing intercolumnar interactions in the primary visual cortex (V1). The second model is based on intracolumnar interactions in a two-layer mechanism of simple cells having the same orientation tuning but significantly different scales. Our third model uses a three-layer circuit in which both even-symmetric and odd-symmetric receptive fields (RFs) are used to compute all possible directional derivatives of the dominant orientation, from which the tangential and normal curvatures at each spatial position are selected using nonlinear shunting inhibition. We show experimental results of all three models, we outline an extension to oriented textures with multiple dominant orientations at each point, and we discuss how our results may be relevant to the processing of general textures.

Object interpretation: extending and validating object recognition

1) Extract image measurements for candidate primitives of type points, contours, and regions 2) Score combinations of primitive candidates by their comparability with learned relations. 3) Select the maximum-score combination as the final interpretation of the object structure. • local intensity extrema • parallelism and continuity between two contours • containment of point feature in region • ‘ends-in’ relation between contour and point/region • Cover of point feature by contour • Segmentation and texture support along contours and between contours The Interpretation process:

A biologically-plausible model for curvature-based texture segregation

A superposition of two different ODTs reveals perceptual singularities in multi-oriented textures. This happens in both classical (piecewise constant) and smoothly-varying ODTs. Hence, we hypothesize that any multioriented texture (i.e., oriented pattern with more than one dominant orientation at a point), can be decompose to n different piecewise-smooth oriented manifold, each of which gives rise to its own perceptual singularities. Main idea: The computation of the orientation derivatives (i.e, curvatures) is done without separating and grouping the orientation measurements to smooth oriented manifolds. Instead, this goal is achieved By transforming the problem to a dual space, where derivatives can be computed as a solution of a differential equation.