Ryan Farrell

Birdlets: Subordinate categorization using volumetric primitives and pose-normalized appearance

Subordinate-level categorization typically rests on establishing salient distinctions between part-level characteristics of objects, in contrast to basic-level categorization, where the presence or absence of parts is determinative. We develop an approach for subordinate categorization in vision, focusing on an avian domain due to the fine-grained structure of the category taxonomy for this domain. We explore a pose-normalized appearance model based on a volumetric poselet scheme. The variation in shape and appearance properties of these parts across a taxonomy provides the cues needed for subordinate categorization. Training pose detectors requires a relatively large amount of training data per category when done from scratch; using a subordinate-level approach, we exploit a pose classifier trained at the basic-level, and extract part appearance and shape information to build subordinate-level models. Our model associates the underlying image pattern parameters used for detection with corresponding volumetric part location, scale and orientation parameters. These parameters implicitly define a mapping from the image pixels into a pose-normalized appearance space, removing view and pose dependencies, facilitating fine-grained categorization from relatively few training examples.

Deformable Part Descriptors for Fine-Grained Recognition and Attribute Prediction

Recognizing objects in fine-grained domains can be extremely challenging due to the subtle differences between subcategories. Discriminative markings are often highly localized, leading traditional object recognition approaches to struggle with the large pose variation often present in these domains. Pose-normalization seeks to align training exemplars, either piecewise by part or globally for the whole object, effectively factoring out differences in pose and in viewing angle. Prior approaches relied on computationally-expensive filter ensembles for part localization and required extensive supervision. This paper proposes two pose-normalized descriptors based on computationally-efficient deformable part models. The first leverages the semantics inherent in strongly-supervised DPM parts. The second exploits weak semantic annotations to learn cross-component correspondences, computing pose-normalized descriptors from the latent parts of a weakly-supervised DPM. These representations enable pooling across pose and viewpoint, in turn facilitating tasks such as fine-grained recognition and attribute prediction. Experiments conducted on the Caltech-UCSD Birds 200 dataset and Berkeley Human Attribute dataset demonstrate significant improvements of our approach over state-of-art algorithms.

Pose pooling kernels for sub-category recognition

The ability to normalize pose based on super-category landmarks can significantly improve models of individual categories when training data are limited. Previous methods have considered the use of volumetric or morphable models for faces and for certain classes of articulated objects. We consider methods which impose fewer representational assumptions on categories of interest, and exploit contemporary detection schemes which consider the ensemble of responses of detectors trained for specific pose-keypoint configurations. We develop representations for poselet-based pose normalization using both explicit warping and implicit pooling as mechanisms. Our method defines a pose normalized similarity or kernel function that is suitable for nearest-neighbor or kernel-based learning methods.

Open-vocabulary Object Retrieval

In this paper, we address the problem of retrieving objects based on open-vocabulary natural language queries: Given a phrase describing a specific object, e.g., “the corn flakes box”, the task is to find the best match in a set of images containing candidate objects. When naming objects, humans tend to use natural language with rich semantics, including basic-level categories, fine-grained categories, and instance-level concepts such as brand names. Existing approaches to large-scale object recognition fail in this scenario, as they expect queries that map directly to a fixed set of pre-trained visual categories, e.g. ImageNet synset tags. We address this limitation by introducing a novel object retrieval method. Given a candidate object image, we first map it to a set of words that are likely to describe it, using several learned image-to-text projections. We also propose a method for handling open-vocabularies, i.e., words not contained in the training data. We then compare the natural language query to the sets of words predicted for each candidate and select the best match. Our method can combine categoryand instance-level semantics in a common representation. We present extensive experimental results on several datasets using both instance-level and category-level matching and show that our approach can accurately retrieve objects based on extremely varied open-vocabulary queries. The source code of our approach will be publicly available together with pre-trained models at http://openvoc.berkeleyvision.org and could be directly used for robotics applications.

Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection

We introduce tools and methodologies to collect high quality, large scale fine-grained computer vision datasets using citizen scientists - crowd annotators who are passionate and knowledgeable about specific domains such as birds or airplanes. We worked with citizen scientists and domain experts to collect NABirds, a new high quality dataset containing 48,562 images of North American birds with 555 categories, part annotations and bounding boxes. We find that citizen scientists are significantly more accurate than Mechanical Turkers at zero cost. We worked with bird experts to measure the quality of popular datasets like CUB-200-2011 and ImageNet and found class label error rates of at least 4%. Nevertheless, we found that learning algorithms are surprisingly robust to annotation errors and this level of training data corruption can lead to an acceptably small increase in test error if the training set has sufficient size. At the same time, we found that an expert-curated high quality test set like NABirds is necessary to accurately measure the performance of fine-grained computer vision systems. We used NABirds to train a publicly available bird recognition service deployed on the web site of the Cornell Lab of Ornithology.

Decentralized discovery of camera network topology

One of the primary uses of camera networks is the observation and tracking of objects within some domain. Substantial research has gone into tracking objects within single and multiple views. However, few such approaches scale to large numbers of sensors, and those that do require an understanding of the network topology. Camera network topology models camera adjacency in the context of tracking: when an object/entity leaves one camera, which cameras could it appear at next? This paper presents a decentralized approach for estimating a camera networkpsilas topology based on sequential Bayesian estimation using a modified multinomial distribution. Central to this method is an information-theoretic appearance model for observation weighting. The distributed nature of the approach utilizes all of the sensors as processing agents in collectively recovering the network topology. Experimental results are presented using camera networks varying in size from 10-100 nodes.

Learning Higher-order Transition Models in Medium-scale Camera Networks

We present a Bayesian framework for learning higher- order transition models in video surveillance networks. Such higher-order models describe object movement between cameras in the network and have a greater predictive power for multi-camera tracking than camera adjacency alone. These models also provide inherent resilience to camera failure, filling in gaps left by single or even multiple non-adjacent camera failures. Our approach to estimating higher-order transition models relies on the accurate assignment of camera observations to the underlying trajectories of objects moving through the network. We addresses this data association problem by gathering the observations and evaluating alternative partitions of the observation set into individual object trajectories. Searching the complete partition space is intractable, so an incremental approach is taken, iteratively adding observations and pruning unlikely partitions. Partition likelihood is determined by the evaluation of a probabilistic graphical model. When the algorithm has considered all observations, the most likely (MAP) partition is taken as the true object trajectories. From these recovered trajectories, the higher-order statistics we seek can be derived and employed for tracking. The partitioning algorithm we present is parallel in nature and can be readily extended to distributed computation in medium-scale smart camera networks.

Localization in Multi-Modal Sensor Networks

We describe the design and implementation of solutions for localization problems in multi-modal wireless sensor networks. The problem of network self-localization, namely determining the positions of the nodes that comprise the network, is addressed optically using a set of pan-tilt-zoom (PTZ) cameras to search for a small light-source attached to each of the sensor nodes. Once the locations and headings of the networks nodes are estimated by the cameras, the network can be used to detect and estimate the location of objects traveling through it. Target localization is performed within the network, using information from magnetometers connected to the sensor nodes. We evaluate the performance of the proposed target localization algorithms through simulations and an implementation running on MicaZ motes. Simulation results show that the localization error for a 100-node network whose nodes are randomly deployed over an area of 100 x 100 m, can be less than 10 cm. Moreover, our initial implementation results show that the median of the localization error for magnetic targets in a 1m x 1m field is 7.1 cm.

Target localization in camera wireless networks

Target localization is an application of wireless sensor networks with wide applicability in homeland security scenarios including surveillance and asset protection. In this paper we present a novel sensor network that localizes with the help of two modalities: cameras and non-imaging sensors. A set of two cameras is initially used to localize the motes of a wireless sensor network. Motes subsequently collaborate to estimate the location of a target using their non-imaging sensors. Our results show that the combination of imaging and non-imaging modalities successfully achieves the dual goal of self- and target-localization. Specifically, we found through simulation and experimental validation that cameras can localize motes deployed in a 100 mx100 m area with a few cm error. Moreover, a network of motes equipped with magnetometers can, once localized, estimate the location of magnetic targets within a few cm.

Distributed inference for network localization using radio interferometric ranging

A localization algorithm using radio interferometric measurements is presented. A probabilistic model is constructed that accounts for general noise models and lends itself to distributed computation. A message passing algorithm is derived that exploits the geometry of radio interferometric measurements and can support sparse network topologies and noisy measurements. Simulations on real and simulated data show promising performance for 2D and 3D deployments.