Margarita Osadchy

SCiFI - A System for Secure Face Identification

We introduce SCiFI, a system for Secure Computation of Face Identification. The system performs face identification which compares faces of subjects with a database of registered faces. The identification is done in a secure way which protects both the privacy of the subjects and the confidentiality of the database. A specific application of SCiFI is reducing the privacy impact of camera based surveillance. In that scenario, SCiFI would be used in a setting which contains a server which has a set of faces of suspects, and client machines which might be cameras acquiring images in public places. The system runs a secure computation of a face recognition algorithm, which identifies if an image acquired by a client matches one of the suspects, but otherwise reveals no information to neither of the parties. Our work includes multiple contributions in different areas: A new face identification algorithm which is unique in having been specifically designed for usage in secure computation. Nonetheless, the algorithm has face recognition performance comparable to that of state of the art algorithms. We ran experiments which show the algorithm to be robust to different viewing conditions, such as illumination, occlusions, and changes in appearance (like wearing glasses). A secure protocol for computing the new face recognition algorithm. In addition, since our goal is to run an actual system, considerable effort was made to optimize the protocol and minimize its online latency. A system - SCiFI, which implements a secure computation of the face identification protocol. Experiments which show that the entire system can run in near real-time: The secure computation protocol performs a preprocessing of all public-key cryptographic operations. Its online performance therefore mainly depends on the speed of data communication, and our experiments show it to be extremely efficient.

Restoring subsampled color images

Abstract. In some capturing devices, such as digital cameras, there is only one color sensor at each pixel. Usually, 50% of the pixels have only a green sensor, 25% only a red sensor, and 25% only a blue sensor. The problem is then to restore the two missing colors at each pixel – this is called “demosaicing”, because the original samples are usually arranged in a mosaic pattern. In this short paper, a few demosaicing algorithms are developed and compared. They all incorporate a notion of “smoothness in chroma space”, by imposing conditions not only on the behavior of each color channel separately, but also on the correlation between the three channels.

Surface Dependent Representations for Illumination Insensitive Image Comparison

We consider the problem of matching images to tell whether they come from the same scene viewed under different lighting conditions. We show that the surface characteristics determine the type of image comparison method that should be used. Previous work has shown the effectiveness of comparing the image gradient direction for surfaces with material properties that change rapidly in one direction. We show analytically that two other widely used methods, normalized correlation of small windows and comparison of multiscale oriented filters, essentially compute the same thing. Then, we show that for surfaces whose properties change more slowly, comparison of the output of whitening filters is most effective. This suggests that a combination of these strategies should be employed to compare general objects. We discuss indications that Gabor jets use such a mixed strategy effectively, and we propose a new mixed strategy. We validate our results on synthetic and real images

Illumination invariant representation for privacy preserving face identification

Most effective face recognition methods store biometric information in the clear. Doing so exposes those systems to the risk of identity theft and violation of privacy. This problem significantly narrows the practical use of face recognition technology. Recent methods for privacy preserving face recognition address face verification task. Most of them are unable to generalize to unseen conditions and require a large number of images of every user for training. We address the problem of face identification, which is more useful in security applications, and propose a binary, illumination invariant representation that can be easily integrated with various efficient cryptographic tools for protection. We propose several privacy preserving applications for our representation and test it on a number of benchmark databases to show its robustness to severe illumination changes, occlusions, and some other appearance variations.

Incorporating the Boltzmann Prior in Object Detection Using SVM

In this paper we discuss object detection when only a small number of training examples are given. Specifically, we show how to incorporate a simple prior on the distribution of natural images into support vector machines. SVMs are known to be robust to overfitting; however, a few training examples usually do not represent well the structure of the class. Thus the resulting detectors are not robust and highly depend on the choice of the training examples. We incorporate the prior on natural images by requiring that the separating hyperplane will not only yield a wide margin, but also that the corresponding positive half space will have a low probability to contain natural images (the background). Our experiments on real data sets show that the resulting detector is more robust to the choice of training examples, and substantially improves both linear and kernel SVMwhen trained on 10 positive and 10 negative examples.

Using specularities in comparing 3D models and 2D images

We aim to create systems that identify and locate objects by comparing known, 3D shapes to intensity images that they have produced. To do this we focus on verification methods that determine whether a known model in a specific pose is consistent with an image. We build on prior work that has done this successfully for Lambertian objects, to handle a much broader class of shiny objects that produce specular highlights. Our core contribution is a novel method for determining whether a known 3D shape is consistent with the 2D shape of a possible highlight found in an image. We do this using only a qualitative description of highlight formation that is consistent with most models of specular reflection, so no specific knowledge of an objects specular reflectance properties is needed. This allows us to treat non-Lambertian image effects as a positive source of information about object identity, rather than treating them as a potential source of noise. We then show how to integrate information about highlights into a system that also checks the consistency of Lambertian reflectance effects. Also, we show how to model Lambertian reflectance using a reference image, rather than albedos, which can be difficult to measure in shiny objects. We test each aspect of our approach using several different data sets. We demonstrate the potential value of our method of handling specular highlights by building a system that can locate shiny, transparent objects, such as glassware, on table tops. We demonstrate our hybrid methods on pottery, and our use of reference images with face recognition experiments.

Whitening for Photometric Comparison of Smooth Surfaces under Varying Illumination

We consider the problem of image comparison in order to match smooth surfaces under varying illumination. In a smooth surface nearby surface normals are highly correlated. We model such surfaces as Gaussian processes and derive the resulting statistical characterization of the corresponding images. Supported by this model, we treat the difference between two images, associated with the same surface and different lighting, as colored Gaussian noise, and use the whitening tool from signal detection theory to construct a measure of difference between such images. This also improves comparisons by accentuating the differences between images of different surfaces. At the same time, we prove that no linear filter, including ours, can produce lighting insensitive image comparisons. While our Gaussian assumption is a simplification, the resulting measure functions well for both synthetic and real smooth objects. Thus we improve upon methods for matching images of smooth objects, while providing insight into the performance of such methods. Much prior work has focused on image comparison methods appropriate for highly curved surfaces. We combine our method with one of these, and demonstrate high performance on rough and smooth objects.

No Bot Expects the DeepCAPTCHA! Introducing Immutable Adversarial Examples, With Applications to CAPTCHA Generation

Recent advances in deep learning (DL) allow for solving complex AI problems that used to be considered very hard. While this progress has advanced many fields, it is considered to be bad news for Completely Automated Public Turing tests to tell Computers and Humans Apart (CAPTCHAs), the security of which rests on the hardness of some learning problems. In this paper, we introduce DeepCAPTCHA, a new and secure CAPTCHA scheme based on adversarial examples, an inherit limitation of the current DL networks. These adversarial examples are constructed inputs, either synthesized from scratch or computed by adding a small and specific perturbation called adversarial noise to correctly classified items, causing the targeted DL network to misclassify them. We show that plain adversarial noise is insufficient to achieve secure CAPTCHA schemes, which leads us to introduce immutable adversarial noise—an adversarial noise that is resistant to removal attempts. In this paper, we implement a proof of concept system, and its analysis shows that the scheme offers high security and good usability compared with the best previously existing CAPTCHAs.

Recognition Using Specular Highlights

We present a novel approach to pose estimation and model-based recognition of specular objects in difficult viewing conditions, such as low illumination, cluttered background, large highlights, and shadows that appear on the object of interest. In such challenging conditions, conventional features are unreliable. We show that under the assumption of a dominant light source, specular highlights produced by a known object can be used to establish correspondence between its image and the 3D model, and to verify the hypothesized pose and the identity of the object. Previous methods that use highlights for recognition make limiting assumptions such as known pose, scene-dependent calibration, simple shape, etc. The proposed method can efficiently recognize free-form specular objects in arbitrary pose and under unknown lighting direction. It uses only a single image of the object as its input and outputs object identity and the full pose. We have performed extensive experiments for both recognition and pose estimation accuracy on synthetic images and on real indoor and outdoor images.

Using specular highlights as pose invariant features for 2D-3D pose estimation

We address the problem of 2D-3D pose estimation in difficult viewing conditions, such as low illumination, cluttered background, and large highlights and shadows that appear on the object of interest. In such challenging conditions conventional features used for establishing correspondence are unreliable. We show that under the assumption of a dominant light source, specular highlights produced by a known object can be used to establish correspondence between its image and the 3D model, and to verify the hypothesized pose. These ideas are incorporated in an efficient method for pose estimation from a monocular image of an object using only highlights produced by the object as its input. The proposed method uses no knowledge of lighting direction and no calibration object for estimating the lighting in the scene. The evaluation of the method shows good accuracy on numerous synthetic images and good robustness on real images of complex, shiny objects, with shadows and difficult backgrounds.