Iuri Frosio

Loss Functions for Image Restoration With Neural Networks

Neural networks are becoming central in several areas of computer vision and image processing and different architectures have been proposed to solve specific problems. The impact of the loss layer of neural networks, however, has not received much attention in the context of image processing: the default and virtually only choice is

Robust Model-Based 3D Head Pose Estimation

\ell _2

A patch memory system for image processing and computer vision

. In this paper, we bring attention to alternative choices for image restoration. In particular, we show the importance of perceptually-motivated losses when the resulting image is to be evaluated by a human observer. We compare the performance of several losses, and propose a novel, differentiable error function. We show that the quality of the results improves significantly with better loss functions, even when the network architecture is left unchanged.

Retrieving gray-level information from a Binary Sensor and its application to gesture detection

We introduce a method for accurate three dimensional head pose estimation using a commodity depth camera. We perform pose estimation by registering a morphable face model to the measured depth data, using a combination of particle swarm optimization (PSO) and the iterative closest point (ICP) algorithm, which minimizes a cost function that includes a 3D registration and a 2D overlap term. The pose is estimated on the fly without requiring an explicit initialization or training phase. Our method handles large pose angles and partial occlusions by dynamically adapting to the reliable visible parts of the face. It is robust and generalizes to different depth sensors without modification. On the Biwi Kinect dataset, we achieve best-in-class performance, with average angular errors of 2.1, 2.1 and 2.4 degrees for yaw, pitch, and roll, respectively, and an average translational error of 5.9 mm, while running at 6 fps on a graphics processing unit.

Machine learning for adaptive bilateral filtering

From self-driving cars to high dynamic range (HDR) imaging, the demand for image-based applications is growing quickly. In mobile systems, these applications place particular strain on performance and energy efficiency. As traditional memory systems are optimized for 1D memory access, they are unable to efficiently exploit the multi-dimensional locality characteristics of image-based applications which often operate on sub-regions of 2D and 3D image data. We have developed a new Patch Memory System (PMEM) tailored to application domains that process 2D and 3D data streams. PMEM supports efficient multidimensional addressing, automatic handling of image boundaries, and efficient caching and prefetching of image data. In addition to an optimized cache, PMEM includes hardware for offloading structured address calculations from processing units. We improve average energy-delay by 26% compared to EVA, a memory system for computer vision applications. Compared to a traditional cache, our results show that PMEM can reduce processor energy by 34% for a selection of CV and IP applications, leading to system performance improvement of up to 32% and energy-delay product improvement of 48-86% on the applications in this study.

A real-time energy-efficient superpixel hardware accelerator for mobile computer vision applications

We report on the use of a CMOS Contrast-based Binary Vision Sensor (CBVS), with embedded contrast extraction, for gesture detection applications. The first advantage of using this sensor over commercial imagers is a dynamic range of 120dB, made possible by a pixel design that effectively performs auto-exposure control. Another benefit is that, by only delivering the pixels detecting a contrast, the sensor requires a very limited bandwidth. We leverage the sensors fast 150μs readout speed, to perform multiple reads during a single exposure; this allows us to estimate gray-level information from the otherwise binary pixels. As a use case for this novel readout strategy, we selected in-car gesture detection, for which we carried out preliminary tests showing encouraging results.

Accelerometer-based correction of skewed horizon and keystone distortion in digital photography

We describe a supervised learning procedure for estimating the relation between a set of local image features and the local optimal parameters of an adaptive bilateral filter. A set of two entropy-based features is used to represent the properties of the image at a local scale. Experimental results show that our entropy-based adaptive bilateral filter outperforms other extensions of the bilateral filter where parameter tuning is based on empirical rules. Beyond bilateral filter, our learning procedure represents a general framework that can be used to develop a wide class of adaptive filters.

Camera re-calibration after zooming based on sets of conics

Superpixel generation is a common preprocessing step in vision processing aimed at dividing an image into non-overlapping regions. Simple Linear Iterative Clustering (SLIC) is a commonly used superpixel algorithm that offers a good balance between performance and accuracy. However, the algorithms high computational and memory bandwidth requirements result in performance and energy efficiency that do not meet the requirements of realtime embedded applications. In this work, we explore the design of an energy-efficient superpixel accelerator for real-time computer vision applications. We propose a novel algorithm, Subsampled SLIC (S-SLIC), that uses pixel subsampling to reduce the memory bandwidth by 1.8x. We integrate S-SLIC into an energy-efficient superpixel accelerator and perform an in-depth design space exploration to optimize the design. We completed a detailed design in a 16nm FinFET technology using commercially-available EDA tools for high-level synthesis to map the design automatically from a C-based representation to a gate-level implementation. The proposed S-SLIC accelerator achieves real-time performance (30 frames per second) with 250 x better energy efficiency than an optimized SLIC software implementation running on a mobile GPU.

Is L2 a Good Loss Function for Neural Networks for Image Processing

Abstract Improper camera orientation produces convergent vertical lines (keystone distortion) and skewed horizon lines (horizon distortion) in digital pictures; an a-posteriori processing is then necessary to obtain appealing pictures. We show here that, after accurate calibration, the camera on-board accelerometer can be used to automatically generate an alternative perspective view from a virtual camera, leading to images with residual keystone and horizon distortions that are essentially imperceptible at visual inspection. Furthermore, we describe the uncertainty on the position of each pixel in the corrected image with respect to the accelerometer noise. Experimental results show a similar accuracy for a smartphone and for a digital reflex camera. The method can find application in customer imaging devices as well as in the computer vision field, especially when reference vertical and horizontal features are not easily detectable in the image.

Loss Functions for Neural Networks for Image Processing.

We describe a method to compute the internal parameters (focal and principal point) of a camera with known position and orientation, based on the observation of two or more conics on a known plane. The conics can even be degenerate (e.g., pairs of lines). The proposed method can be used to re-estimate the internal parameters of a fully calibrated camera after zooming to a new, unknown, focal length. It also allows estimating the internal parameters when a second, fully calibrated camera observes the same conics. The parameters estimated through the proposed method are coherent with the output of more traditional procedures that require a higher number of calibration images. A deep analysis of the geometrical configurations that influence the proposed method is also reported.