Sharathchandra U. Pankanti

Smart video surveillance: exploring the concept of multiscale spatiotemporal tracking

Situation awareness is the key to security. Awareness requires information that spans multiple scales of space and time. Smart video surveillance systems are capable of enhancing situational awareness across multiple scales of space and time. However, at the present time, the component technologies are evolving in isolation. To provide comprehensive, nonintrusive situation awareness, it is imperative to address the challenge of multiscale, spatiotemporal tracking. This article explores the concepts of multiscale spatiotemporal tracking through the use of real-time video analysis, active cameras, multiple object models, and long-term pattern analysis to provide comprehensive situation awareness.

A layered approach to robust lane detection at night

A layered approach is designed to address many of the real-world problems that an inexpensive lane detection system would encounter. A region of interest is first extracted from the image followed by an enhancement procedure to manipulate the shape of the lane markers. The extracted region is then converted to binary using an adaptive threshold. A model based line detection system hypothesizes lane position. Finally, an iterated matched filtering scheme estimates the final lane position. The developed system shows good performance when tested on real-world data that contains fluctuating illumination and a variety of traffic conditions.

A touch of money [biometric authentication systems]

This paper suggests the use of a new authentication system for credit cards based on biometric sensors that could dramatically curtail identity theft. The proposed system uses fingerprint sensors, though other biometric technologies, either alone or in combination, could be incorporated. It could be economical, protect privacy, and guarantee the validity of all kinds of credit card transactions, including ones that take place at a store, over the telephone, or with an Internet-based retailer. By preventing identity thieves from entering the transaction loop, credit card companies could quickly recoup their infrastructure investments and save businesses, consumers, and themselves billions of dollars every year.

Adaptive as-natural-as-possible image stitching

The goal of image stitching is to create natural-looking mosaics free of artifacts that may occur due to relative camera motion, illumination changes, and optical aberrations. In this paper, we propose a novel stitching method, that uses a smooth stitching field over the entire target image, while accounting for all the local transformation variations. Computing the warp is fully automated and uses a combination of local homography and global similarity transformations, both of which are estimated with respect to the target. We mitigate the perspective distortion in the non-overlapping regions by linearizing the homography and slowly changing it to the global similarity. The proposed method is easily generalized to multiple images, and allows one to automatically obtain the best perspective in the panorama. It is also more robust to parameter selection, and hence more automated compared with state-of-the-art methods. The benefits of the proposed approach are demonstrated using a variety of challenging cases.

Intelligent headlight control using camera sensors

This paper describes our recent work on intelligently controlling a vehicles headlights using a forward-facing camera sensor. Specifically, we aim to automatically control its beam state (high beam or low beam) during a night-time drive based on the detection of oncoming/overtaking/leading traffics as well as urban areas from the videos captured by the camera. A three-level decision framework is proposed which includes various types of image and video content analysis, an SVM-based learning mechanism and a frame-level decision making mechanism. Both video and context information have been exploited to accomplish the task. Online test drives as well as offline evaluations on tens of videos have validated the robustness and effectiveness of the proposed system.

Privacy Protection and Face Recognition

In this chapter, we describe the privacy issues surrounding the proliferation of digital imagery, particularly of faces, in surveillance video, online photo-sharing, medical records and online navigable street imagery. We highlight the growing capacity for computer systems to process, recognize, and index face images and outline some of the techniques that have been used to protect privacy while supporting ongoing innovation and growth in the applications of digital imagery.

Resilient mobile cognition: Algorithms, innovations, and architectures

The importance of the internet-of-things (IOT) is now an established reality. With that backdrop, the phenomenal emergence of cameras/sensors mounted on unmanned aerial, ground and marine vehicles (UAVs, UGVs, UMVs) and body worn cameras is a notable new development. The swarms of cameras and real-time computing thereof are at the heart of new technologies like connected cars, drone-based city-wide surveillance and precision agriculture, etc. Smart computer vision algorithms (with or without dynamic learning) that enable object recognition and tracking, supported by baseline video content summarization or 2D/3D image reconstruction of the scanned environment are at the heart of such new applications. In this article, we summarize our recent innovations in this space. We focus primarily on algorithms and architectural design considerations for video summarization systems.

Exploiting Color Strength to Improve Color Correction

Color information is an important feature for many vision algorithms including color correction, image retrieval and tracking. In this paper, we study the limitations of color measurement accuracy and explore how this information can be used to improve the performance of color correction. In particular, we show that a strong correlation exists between the error in hue measurements on one hand and saturation and intensity on the other hand. We introduce the notion of color strength, which is a combination of saturation and intensity information to determine when hue information in a scene is reliable. We verify the predictive capability of this model on two different datasets with ground truth color information. Further, we show how color strength information can be used to significantly improve color correction accuracy for the 11K real-world SFU gray ball dataset.

Multi-media compliance: A practical paradigm for managing business integrity

In virtually every business context there is a need to establish some form of monitoring system to ensure that employees comply with business processes and policies. Compliance failures range from organized theft to gaps in procedure that can be easily remedied through retraining. It is clearly important for businesses to capture and record these deviations to minimize loss prevention and maximize workplace safety and efficiency. In this workshop, we discuss the growing problem of compliance failure and how our system addresses this problem in a retail context to detect checkout-related fraud through the integration of visual and non-visual data.

Security, privacy, and health

Much of the current pervasive computing research concentrates on devices and the communication between them. However, an important aspect of pervasive devices is their interface with the physical world—particularly, how they acquire information about their users. One rich medium (and the dominant one through which people receive information) is vision. As such, we expect future pervasive computing environments to depend on vision for passive perception of people. The PeopleVision project at the IBM T.J.Watson Research Center is tackling this problem, focusing on the privacy issues involved in such visual information gathering, both in pervasive computing environments and in video surveillance systems. The PeopleVision system takes an object-oriented approach to video. It understands the video stream, decomposing it into people, objects, and areas of interest. It then abstracts or selectively re-renders this information based on the intended user. Client processes receive abstract information about people in the environment according to issued requests. Access control lists that can grant privileged processes access to richer (and more intrusive) information verify these requests. For example, the list might grant a face recognition security system access to facial images but tell the air conditioning process only how many people are in each room. Access control lists also govern the information delivered to security guards, supervisors, and ordinary users. Re-rendering delivers reconstructed video, which preserves some objects unchanged and blanks out other areas of the image or replaces the area with a computergraphics rendering that preserves relevant information. However, it does not convey more privacy-sensitive details. During ordinary use, a guard may only view silhouettes of people in the surveillance area, hiding irrelevant but privacy-sensitive information such as race, gender, and appearance. We have developed a privacy camera—a single device combining a camera and processor that implements video-understanding algorithms. With this device, we can ensure that the privacyintruding video is never available or only leaves the device in an encrypted form. All of the processed data leaving the device can also be encrypted, ensuring maximum privacy protection for the people in the pervasive computing environment. For more information, contact Andrew Senior at IBM T.J. Watson Research Center, PO Box 704 Yorktown Heights, NY 10598-0218; aws@watson.ibm.com.