Maria Klodt

Continuous Global Optimization in Multiview 3D Reconstruction

In this article, we introduce a new global optimization method to the field of multiview 3D reconstruction. While global minimization has been proposed in a discrete formulation in form of the maxflow-mincut framework, we suggest the use of a continuous convex relaxation scheme. Specifically, we propose to cast the problem of 3D shape reconstruction as one of minimizing a spatially continuous convex functional. In qualitative and quantitative evaluation we demonstrate several advantages of the proposed continuous formulation over the discrete graph cut solution. Firstly, geometric properties such as weighted boundary length and surface area are represented in a numerically consistent manner: The continuous convex relaxation assures that the algorithm does not suffer from metrication errors in the sense that the reconstruction converges to the continuous solution as the spatial resolution is increased. Moreover, memory requirements are reduced, allowing for globally optimal reconstructions at higher resolutions.We study three different energy models for multiview reconstruction, which are based on a common variational template unifying regional volumetric terms and on-surface photoconsistency. The three models use data measurements at increasing levels of sophistication. While the first two approaches are based on a classical silhouette-based volume subdivision, the third one relies on stereo information to define regional costs. Furthermore, this scheme is exploited to compute a precise photoconsistency measure as opposed to the classical estimation. All three models are compared on standard data sets demonstrating their advantages and shortcomings. For the third one, which gives the most accurate results, a more exhaustive qualitative and quantitative evaluation is presented.

An Experimental Comparison of Discrete and Continuous Shape Optimization Methods

Shape optimization is a problem which arises in numerous computer vision problems such as image segmentation and multiview reconstruction. In this paper, we focus on a certain class of binary labeling problems which can be globally optimized both in a spatially discrete setting and in a spatially continuous setting. The main contribution of this paper is to present a quantitative comparison of the reconstruction accuracy and computation times which allows to assess some of the strengths and limitations of both approaches. We also present a novel method to approximate length regularity in a graph cut based framework: Instead of using pairwise terms we introduce higher order terms. These allow to represent a more accurate discretization of the L 2 -norm in the length term.

A convex framework for image segmentation with moment constraints

Convex relaxation techniques have become a popular approach to image segmentation as they allow to compute solutions independent of initialization to a variety of image segmentation problems. In this paper, we will show that shape priors in terms of moment constraints can be imposed within the convex optimization framework, since they give rise to convex constraints. In particular, the lower-order moments correspond to the overall volume, the centroid, and the variance or covariance of the shape and can be easily imposed in interactive segmentation methods. Respective constraints can be imposed as hard constraints or soft constraints. Quantitative segmentation studies on a variety of images demonstrate that the user can easily impose such constraints with a few mouse clicks, giving rise to substantial improvements of the resulting segmentation, and reducing the average segmentation error from 12% to 0:35%. GPU-based computation times of around 1 second allow for interactive segmentation.

Field phenotyping of grapevine growth using dense stereo reconstruction.

BackgroundThe demand for high-throughput and objective phenotyping in plant research has been increasing during the last years due to large experimental sites. Sensor-based, non-invasive and automated processes are needed to overcome the phenotypic bottleneck, which limits data volumes on account of manual evaluations. A major challenge for sensor-based phenotyping in vineyards is the distinction between the grapevine in the foreground and the field in the background – this is especially the case for red-green-blue (RGB) images, where similar color distributions occur both in the foreground plant and in the field and background plants. However, RGB cameras are a suitable tool in the field because they provide high-resolution data at fast acquisition rates with robustness to outdoor illumination.ResultsThis study presents a method to segment the phenotypic classes ‘leaf’, ‘stem’, ‘grape’ and ‘background’ in RGB images that were taken with a standard consumer camera in vineyards. Background subtraction is achieved by taking two images of each plant for depth reconstruction. The color information is furthermore used to distinguish the leaves from stem and grapes in the foreground. The presented approach allows for objective computation of phenotypic traits like 3D leaf surface areas and fruit-to-leaf ratios. The method has been successfully applied to objective assessment of growth habits of new breeding lines. To this end, leaf areas of two breeding lines were monitored and compared with traditional cultivars. A statistical analysis of the method shows a significant (p <0.001) determination coefficient R 2= 0.93 and root-mean-square error of 3.0%.ConclusionsThe presented approach allows for non-invasive, fast and objective assessment of plant growth. The main contributions of this study are 1) the robust segmentation of RGB images taken from a standard consumer camera directly in the field, 2) in particular, the robust background subtraction via reconstruction of dense depth maps, and 3) phenotypic applications to monitoring of plant growth and computation of fruit-to-leaf ratios in 3D. This advance provides a promising tool for high-throughput, automated image acquisition, e.g., for field robots.

Continuous global optimization in multiview 3D reconstruction

In this work, we introduce a robust energy model for multiview 3D reconstruction that fuses silhouette- and stereo-based image information. It allows to cope with significant amounts of noise without manual pre-segmentation of the input images. Moreover, we suggest a method that can globally optimize this energy up to the visibility constraint. While similar global optimization has been presented in the discrete context in form of the maxflow-mincut framework, we suggest the use of a continuous counterpart. In contrast to graph cut methods, discretizations of the continuous optimization technique are consistent and independent of the choice of the grid connectivity. Our experiments demonstrate that this leads to visible improvements. Moreover, memory requirements are reduced, allowing for global reconstructions at higher resolutions.

High-Resolution Plant Shape Measurements from Multi-view Stereo Reconstruction

Accurate high-resolution 3D models are essential for a non-invasive analysis of phenotypic characteristics of plants. Leaf surface areas, fruit volumes and leaf inclination angles are typically of interest. This work presents a globally optimal 3D geometry reconstruction method that is specialized to high-resolutions and is thus suitable to reconstruct thin structures typically occuring in the geometry of plants. Volumetric 3D models are computed in a convex optimization framework from a set of RGB input images depicting the plant from different view points. The method uses the memory and run-time efficient octree data structure for fast computations of high-resolution 3D models. Results show accurate 3D reconstructions of barley, while an increase in resolution of a factor of up to 2000 is achieved in comparison to the use of a uniform voxel based data structure, making the choice of data structure crucial for feasible resolutions.

Scale-Aware Object Tracking with Convex Shape Constraints on RGB-D Images

Convex relaxation techniques have become a popular approach to a variety of image segmentation problems as they allow to compute solutions independent of the initialization. In this paper, we propose a novel technique for the segmentation of RGB-D images using convex function optimization. The function that we propose to minimize considers both the color image and the depth map for finding the optimal segmentation. We extend the objective function by moment constraints, which allow to include prior knowledge on the 3D center, surface area or volume of the object in a principled way. As we show in this paper, the relaxed optimization problem is convex, and thus can be minimized in a globally optimal way leading to high-quality solutions independent of the initialization. We validated our approach experimentally on four different datasets, and show that using both color and depth substantially improves segmentation compared to color or depth only. Further, 3D moment constraints significantly robustify segmentation which proves in particular useful for object tracking.

Moment Constraints in Convex Optimization for Segmentation and Tracking

Convex relaxation techniques have become a popular approach to shape optimization as they allow to compute solutions independent of initialization to a variety of problems. In this chapter, we will show that shape priors in terms of moment constraints can be imposed within the convex optimization framework, since they give rise to convex constraints. In particular, the lower-order moments correspond to the overall area, the centroid, and the variance or covariance of the shape and can be easily imposed in interactive segmentation methods. Respective constraints can be imposed as hard constraints or soft constraints. Quantitative segmentation studies on a variety of images demonstrate that the user can impose such constraints with a few mouse clicks, leading to substantial improvements of the resulting segmentation, and reducing the average segmentation error from 12 % to 0.35 %. GPU-based computation times of around 1 second allow for interactive segmentation.

Automatic image-based determination of pruning mass as a determinant for yield potential in grapevine management and breeding

Background and Aims Vine balance is defined as a relation between vegetative (mass of dormant pruning wood) and generative (yield) growth. For grapevine breeding, emphasis is usually placed on the evaluation of individual seedlings. In this study, we calculated the mass of dormant pruning wood with the assistance of an automated image-based method for estimating the pixel area of dormant pruning wood. The evaluation of digital images in combination with depth map calculation and image segmentation is a new and non-invasive tool for objective data acquisition. Methods and Results The proposed method was tested on a set of seedlings planted at the Institute for Grapevine Breeding Geilweilerhof, Germany. All images taken in the field were geo-referenced, and the automated method was validated by manual segmentation. Together with additional yield parameters, the vine balance indices can be used to classify seedlings for breeding purposes. Conclusion The computed pruning mass obtained using image-based methods is an accurate, inexpensive and easy method to estimate pruning mass compared with the manual time-consuming measurements. Together with the yield parameters, it is a suitable method for seedling evaluation and can also be used in precision viticulture. Significance of the Study This study demonstrates an image-based evaluation of the pruning mass to be a highly valuable tool for grapevine research and grapevine breeding. Moreover, the tool might be used by industry to monitor vine balance. The key findings reported have the potential to increase grapevine breeding efficiency by using an accurate and objective phenotyping method.