Charles E. Jacobs

Image analogies

This paper describes a new framework for processing images by example, called “image analogies.” The framework involves two stages: a design phase, in which a pair of images, with one image purported to be a “filtered” version of the other, is presented as “training data”; and an application phase, in which the learned filter is applied to some new target image in order to create an “analogous” filtered result. Image analogies are based on a simple multi-scale autoregression, inspired primarily by recent results in texture synthesis. By choosing different types of source image pairs as input, the framework supports a wide variety of “image filter” effects, including traditional image filters, such as blurring or embossing; improved texture synthesis, in which some textures are synthesized with higher quality than by previous approaches; super-resolution, in which a higher-resolution image is inferred from a low-resolution source; texture transfer, in which images are “texturized” with some arbitrary source texture; artistic filters, in which various drawing and painting styles are synthesized based on scanned real-world examples; and texture-by-numbers, in which realistic scenes, composed of a variety of textures, are created using a simple painting interface.

Fast multiresolution image querying

We present a method for searching in an image database using a query image that is similar to the intended target. The query image may be a hand-drawn sketch or a (potentially low-quality) scan of the image to be retrieved. Our searching algorithm makes use of multiresolution wavelet decompositions of the query and database images. The coefficients of these decompositions are distilled into small “signatures” for each image. We introduce an “image querying metric” that operates on these signatures. This metric essentially compares how many significant wavelet coefficients the query has in common with potential targets. The metric includes parameters that can be tuned, using a statistical analysis, to accommodate the kinds of image distortions found in different types of image queries. The resulting algorithm is simple, requires very little storage overhead for the database of signatures, and is fast enough to be performed on a database of 20,000 images at interactive rates (on standard desktop machines) as a query is sketched. Our experiments with hundreds of queries in databases of 1000 and 20,000 images show dramatic improvement, in both speed and success rate, over using a conventional L1, L2, or color histogram norm. CR

Multiresolution video

We present a new representation for time-varying image data that allows for varying—and arbitrarily high—spatial and temporal resolutions in different parts of a video sequence. The representation, called multiresolution video, is based on a sparse, hierarchical encoding of the video data. We describe a number of operations for creating, viewing, and editing multiresolution sequences. These operations support a variety of applications: multiresolution playback, including motion-blurred “fast-forward” and “reverse”; constantspeed display; enhanced video scrubbing; and “video clip-art” editing and compositing. The multiresolution representation requires little storage overhead, and the algorithms using the representation are both simple and efficient. CR

Rapid modeling of animated faces from video

AbstractGenerating realistic 3D human face models and facial animations has been a persistent chal-lenge in computer graphics. We have developed a system that constructs textured 3D face modelsfrom videos with minimal user interaction. Our system takes images and video sequences of a facewith an ordinary video camera. After five manual clicks on two images to tell the system where theeye corners, nose top and mouth corners are, the system automatically generates a realistic looking3D human head model and the constructed model can be animated immediately. A user, with aPC and an ordinary camera, can use our system to generate his/her face model in a few minutes.Keywords: Face modeling, facial animation, geometric modeling, computer vision 1 Introduction One of the most interesting and difficult problems in computer graphics is the effortless generationof realistic looking, animated human face models. Animated face models are essential to computergames, film making, online chat, virtual presence, video conferencing, etc. So far, the most popularcommercially available tools have utilized laser scanners. Not only are these scanners expensive,the data are usually quite noisy, requiring hand touchup and manual registration prior to animatingthe model. Because inexpensive computers and cameras are widely available, there is great interestin producing face models directly from images. In spite of progress toward this goal, the availabletechniques are either manually intensive or computationally expensive.The goal of our system is to allow an untrained user with a PC and an ordinary camera to createand instantly animate his/her face model in no more than a few minutes. The user interface for theprocess comprises three simple steps. First, the user is instructed to pose for two still images. The useris then instructed to turn his/her head horizontally, first in one direction and then in the other. Third,the user is instructed to identify a few key points in the images. Then the system computes the 3Dface geometry from the two images, and tracks the video sequences to create a complete facial texturemap by blending frames of the sequence. The key observation is that even though it is difficult toextract dense 3D facial geometry from two images, it is possible to match a sparse set of corners anduse them to compute head motion and the 3D locations of these corner points. We can then fit a linearclass of human face geometries to this sparse set of reconstructed corners to generate the completeface geometry. Linear classes of face geometry and image prototypes have been demonstrated forconstructing 3D face models from images in a morphable model framework [3]. In this paper, weshow that linear classes of face geometries can be used to effectively fit/interpolate a sparse set of3D reconstructed points. This novel technique is the key to quickly generating photorealistic 3D facemodels with minimal user intervention.1

Adaptive document layout

How to automatically reformat, resize, and paginate electronic text and graphics so documents look as good on displays of any size as they do on paper.

Text recognition of low-resolution document images

Cheap and versatile cameras make it possible to easily and quickly capture a wide variety of documents. However, low resolution cameras present a challenge to OCR because it is virtually impossible to do character segmentation independently from recognition. In this paper we solve these problems simultaneously by applying methods borrowed from cursive handwriting recognition. To achieve maximum robustness, we use a machine learning approach based on a convolutional neural network. When our system is combined with a language model using dynamic programming, the overall performance is in the vicinity of 80-95% word accuracy on pages captured with a 1024/spl times/768 webcam and 10-point text.

Rapid modeling of animated faces from video images

Generating realistic 3D human face models and facial animations has been a persistent challenge in computer graphics. We have developed a system that constructs textured 3D face models from videos with minimal user interaction. Our system takes images and video sequences of a face with an ordinary video camera. After five manual clicks on two images to tell the system where the eye corners, nose top and mouth corners are, the system automatically generates a realistic looking 3D human head model and the constructed model can be animated immediately. A user, with a PC and an ordinary camera, can use our system to generate his/her face model in a few minutes. We will demonstrate the system at the conference.