Krishnan Ramnath

Edge foci interest points

In this paper, we describe an interest point detector using edge foci. Unlike traditional detectors that compute interest points directly from image intensities, we use normalized intensity edges and their orientations. We hypothesize that detectors based on the presence of oriented edges are more robust to non-linear lighting variations and background clutter than intensity based techniques. Specifically, we detect edge foci, which are points in the image that are roughly equidistant from edges with orientations perpendicular to the point. The scale of the interest point is defined by the distance between the edge foci and the edges. We quantify the performance of our detector using the interest points repeatability, uniformity of spatial distribution, and the uniqueness of the resulting descriptors. Results are found using traditional datasets and new datasets with challenging non-linear lighting variations and occlusions.

Car make and model recognition using 3D curve alignment

We present a new approach for recognizing the make and model of a car from a single image. While most previous methods are restricted to fixed or limited viewpoints, our system is able to verify a cars make and model from an arbitrary view. Our model consists of 3D space curves obtained by backprojecting image curves onto silhouette-based visual hulls and then refining them using three-view curve matching. These 3D curves are then matched to 2D image curves using a 3D view-based alignment technique. We present two different methods for estimating the pose of a car, which we then use to initialize the 3D curve matching. Our approach is able to verify the exact make and model of a car over a wide range of viewpoints in cluttered scenes.

Increasing the density of Active Appearance Models

Active appearance models (AAMs) typically only use 50-100 mesh vertices because they are usually constructed from a set of training images with the vertices hand-labeled on them. In this paper, we propose an algorithm to increase the density of an AAM. Our algorithm operates by iteratively building the AAM, refitting the AAM to the training data, and refining the AAM.We compare our algorithm with the state of the art in optical flow algorithms and find it to be significantly more accurate. We also show that dense AAMs can be fit more robustly than sparse ones. Finally, we show how our algorithm can be used to construct AAMs automatically, starting with a single affine model that is subsequently refined to model non-planarity and non-rigidity.

Detecting and Reconstructing 3D Mirror Symmetric Objects

We present a system that detects 3D mirror-symmetric objects in images and then reconstructs their visible symmetric parts. Our detection stage is based on matching mirror symmetric feature points and descriptors and then estimating the symmetry direction using RANSAC. We enhance this step by augmenting feature descriptors with their affine-deformed versions and matching these extended sets of descriptors. The reconstruction stage uses a novel edge matching algorithm that matches symmetric pairs of curves that are likely to be counterparts. This allows the algorithm to reconstruct lightly textured objects, which are problematic for traditional feature-based and intensity-based stereo matchers.

AutoCaption: Automatic caption generation for personal photos

AutoCaption is a system that helps a smartphone user generate a caption for their photos. It operates by uploading the photo to a cloud service where a number of parallel modules are applied to recognize a variety of entities and relations. The outputs of the modules are combined to generate a large set of candidate captions, which are returned to the phone. The phone client includes a convenient user interface that allows users to select their favorite caption, reorder, add, or delete words to obtain the grammatical style they prefer. The user can also select from multiple candidates returned by the recognition modules.

Rapidly Deployable Video Analysis Sensor units for wide area surveillance

This paper presents an overview of self-contained automated video analytics units that are man-portable and constitute nodes of a large-scale distributed sensor network. The paper highlights issues with traditional video surveillance systems in volatile environments such as a battle field and provides solutions to them in the form of Rapidly Deployable Video Analysis sensors. We discuss scientific and engineering aspects of the system and present the outcome of a field deployment in an exercise conducted by the Office of Naval Research.

Mining text snippets for images on the web

Images are often used to convey many different concepts or illustrate many different stories. We propose an algorithm to mine multiple diverse, relevant, and interesting text snippets for images on the web. Our algorithm scales to all images on the web. For each image, all webpages that contain it are considered. The top-K text snippet selection problem is posed as combinatorial subset selection with the goal of choosing an optimal set of snippets that maximizes a combination of relevancy, interestingness, and diversity. The relevancy and interestingness are scored by machine learned models. Our algorithm is run at scale on the entire image index of a major search engine resulting in the construction of a database of images with their corresponding text snippets. We validate the quality of the database through a large-scale comparative study. We showcase the utility of the database through two web-scale applications: (a) augmentation of images on the web as webpages are browsed and (b)~an image browsing experience (similar in spirit to web browsing) that is enabled by interconnecting semantically related images (which may not be visually related) through shared concepts in their corresponding text snippets.

Distributed Sensor Networks for Visual Surveillance

Automated video analysis systems consist of large networks of distributed heterogeneous sensors. Such systems require extraction, integration, and representation of relevant data from sensors in real time. This book chapter identifies some of those major challenges and proposes solutions to them. In particular, efficient video processing for high-resolution sensors, data fusion across multiple modalities, robustness to changing environmental conditions and video processing errors, and intuitive user interfaces for visualization and analysis are discussed. Enabling technologies to overcome these challenges are also discussed. The case study of a wide area video analysis system deployed at ports in the states of Florida and California, USA is also presented. The components of the system are also detailed and justified using quantitative and qualitative results.

A portable geo-aware visual surveillance system for vehicles

This paper presents the development of a portable surveillance system based on a network of six cameras, attitude sensors and distributed processors mounted on a HMMWV. The system extends the functionality of traditional fixed-installation intelligent visual surveillance (IVS) systems by tracking targets and detecting events in a geographic context and allowing the entire platform to be redeployed to different locations. This paper focuses on our solutions to the following core problems of 360° geo-aware, portable IVS: target geo-localization and tracking, track stitching within and across the six cameras, rule-based event detection and real-time visualization on a map-based interface. An important contribution is our method for automatically updating the geo-localization calibration based on attitude measurements when the vehicle is redeployed. Experimental results validate the performance of the system.