Matthew Johnson

Semantic texton forests for image categorization and segmentation

We propose semantic texton forests, efficient and powerful new low-level features. These are ensembles of decision trees that act directly on image pixels, and therefore do not need the expensive computation of filter-bank responses or local descriptors. They are extremely fast to both train and test, especially compared with k-means clustering and nearest-neighbor assignment of feature descriptors. The nodes in the trees provide (i) an implicit hierarchical clustering into semantic textons, and (ii) an explicit local classification estimate. Our second contribution, the bag of semantic textons, combines a histogram of semantic textons over an image region with a region prior category distribution. The bag of semantic textons is computed over the whole image for categorization, and over local rectangular regions for segmentation. Including both histogram and region prior allows our segmentation algorithm to exploit both textural and semantic context. Our third contribution is an image-level prior for segmentation that emphasizes those categories that the automatic categorization believes to be present. We evaluate on two datasets including the very challenging VOC 2007 segmentation dataset. Our results significantly advance the state-of-the-art in segmentation accuracy, and furthermore, our use of efficient decision forests gives at least a five-fold increase in execution speed.

Semantic Photo Synthesis

Composite images are synthesized from existing photographs by artists who make concept art, e.g., storyboards for movies or architectural planning. Current techniques allow an artist to fabricate such an image by digitally splicing parts of stock photographs. While these images serve mainly to “quickly”convey how a scene should look, their production is laborious. We propose a technique that allows a person to design a new photograph with substantially less effort. This paper presents a method that generates a composite image when a user types in nouns, such as “boat”and “sand.”The artist can optionally design an intended image by specifying other constraints. Our algorithm formulates the constraints as queries to search an automatically annotated image database. The desired photograph, not a collage, is then synthesized using graph‐cut optimization, optionally allowing for further user interaction to edit or choose among alternative generated photos. An implementation of our approach, shown in the associated video, demonstrates our contributions of (1) a method for creating specific images with minimal human effort, and (2) a combined algorithm for automatically building an image library with semantic annotations from any photo collection.

Generalized Descriptor Compression for Storage and Matching

Smarter phones have made handheld computer vision a reality, but limited bandwidth, storage space and processing power prevent mobile phones from leveraging the full body of existing research. In particular, common techniques which use feature detectors and descriptors to solve problems in image matching and augmented reality cannot be used due to their space and processing requirements. We propose a general descriptor compression method which reduces descriptor size and provides fast descriptor matching without requiring decompression. By demonstrating how to apply our method to the commonly used SIFT, SURF and GLOH descriptors, we show its effectiveness in reducing size and increasing accuracy. In all cases, we reduce the size of the descriptor by an order of magnitude and achieve higher accuracy at a detection rate of 95%.

Improved Image Annotation and Labelling through Multi-Label Boosting.

The majority of machine learning systems for object recognition is limited by their requirement of single labelled images for training, which are difficult to create or obtain in quantity. It is therefore impractical to use methods or techniques which require such data to build object recognizers for more than a relatively small subset of object classes. Instead, far more abundant multilabel data provides a ready means to create object recognition systems which are able to deal with large numbers of classes. In this paper we present a new object recognition system named MLBoost which learns from multi-label data through boosting and improves on state-of-the-art multi-label annotation and labelling systems. The system is trained on images with accompanying text and at no time is told which parts of each image correspond to which words, and as such the process is unsupervised. Having once been trained it is able to give segment labels and a list of descriptive words (an annotation) for any novel image.

Semantic Texton Forests

The semantic texton forest is an efficient and powerful low-level feature which can be effectively employed in the semantic segmentation of images. As ensembles of decision trees that act directly on image pixels, semantic texton forests do not need the expensive computation of filter-bank responses or local descriptors. They are extremely fast to both train and test, especially compared with k-means clustering and nearest-neighbor assignment of feature descriptors. The nodes in the trees provide (i) an implicit hierarchical clustering into semantic textons, and (ii) an explicit local classification estimate. The bag of semantic textons combines a histogram of semantic textons over an image region with a region prior category distribution. The bag of semantic textons can be used by an SVM classifier to infer an image-level prior over categories, allowing the segmentation to emphasize those categories that the SVM believes to be present. We will examine the segmentation performance of semantic texton forests on two datasets including the VOC 2007 segmentation challenge.

Cross Modal Disambiguation

We consider strategies for reducing ambiguity in multi-modal data, particularly in the domain of images and text. Large data sets containing images with associated text (and vice versa) are readily available, and recent work has exploited such data to learn models for linking visual elements to semantics. This requires addressing a correspondence ambiguity because it is generally not known which parts of the images connect with which language elements. In this paper we first discuss using language processing to reduce correspondence ambiguity in loosely labeled image data. We then consider a similar problem of using visual correlates to reduce ambiguity in text with associated images. Only rudimentary image understanding is needed for this task because the image only needs to help differentiate between a limited set of choices, namely the senses of a particular word.