Pinaki Sinha

Content without context is meaningless

We revisit one of the most fundamental problems in multimedia that is receiving enormous attention from researchers without making much progress in solving it: the problem of bridging the semantic gap. Research in this area has focused on developing increasingly rigorous techniques using the content. Researchers consider that Content is King and ignore everything else. In this paper, first we will discuss how this infatuation with content continues to be the biggest hurdle in the success of, ironically, content based approaches for multimedia search. Lately, many commercial systems have ignored content in favor of context and demonstrated better success. Given that the mobile phones are the major platform for the next generation of computing, context becomes easily available and more relevant. We show that it is not Content Versus Context; rather it is Content and Context that is required to bridge the semantic gap. In this paper, first we will discuss reasons for our approach and then present approaches that appropriately combine context with content to help bridge the semantic gap and solve important problems in multimedia computing.

Asynchronous Distributed Calibration for Scalable and Reconfigurable Multi-Projector Displays

Centralized techniques have been used until now when automatically calibrating (both geometrically and photometrically) large high-resolution displays created by tiling multiple projectors in a 2D array. A centralized server managed all the projectors and also the camera(s) used to calibrate the display. In this paper, we propose an asynchronous distributed calibration methodology via a display unit called the plug-and-play projector (PPP). The PPP consists of a projector, camera, computation and communication unit, thus creating a self-sufficient module that enables an asynchronous distributed architecture for multi-projector displays. We present a single-program-multiple-data (SPMD) calibration algorithm that runs on each PPP and achieves a truly scalable and reconfigurable display without any input from the user. It instruments novel capabilities like adding/removing PPPs from the display dynamically, detecting faults, and reshaping the display to a reasonable rectangular shape to react to the addition/removal/faults. To the best of our knowledge, this is the first attempt to realize a completely asynchronous and distributed calibration architecture and methodology for multi-projector displays

Summarization of personal photologs using multidimensional content and context

In this paper, we propose a framework for generation of representative subset summaries from large personal photo collections. These summaries will help in effective sharing and browsing of the personal photos. We define three salient properties: quality, diversity and coverage that an informative summary should satisfy. We propose methods to compute these properties using multidimensional content and context data. The objective of summarization is modeled as an optimization of these properties, given the size constraints. We also propose metrics which will evaluate the photo summaries based on their representation of the larger corpus and the ability to satisfy users information needs. We use a dataset of 40K personal photos collected by crawling photo sharing and storage sites of sixteen users. Our experiments show that the summarization algorithm works better than the baseline algorithms.

Classification and annotation of digital photos using optical context data

Other than the pixel information, a digital photo of today has a host of other information regarding the photo shooting event. These information are captured by different sensors present on the camera and are stored as metadata. In this paper we exploit this meta information and derive useful semantics about the digital photo. We also compare our results with classical relevance models used for automatic photo annotation. We create a dataset of digital photos containing all information and report results on it. We also make the dataset available to the community for further experiments.

Personal photo album summarization

Photo album summarization is the process of selecting a subset of photos from a larger collection which best preserves the information in the entire set and is semantically coherent. In this paper we propose a system which uses heterogeneous information sources associated with digital photos and generates a summary. Our algorithm adapts itself based on the type of event it is summarizing (Yearbook, Week or Single Day Event) We model the summarization problem as a retrieval problem based on different types of queries. We propose some evaluation metrics for the summary. We use an intuitive web based interface to present the results so that users can further explore the summary in an interactive way. This system is our submission to the CeWe Challenge for the Next Generation of Tangible Multimedia Products.

Summarization of archived and shared personal photo collections

The volume of personal photos hosted on photo archives and social sharing platforms has been increasing exponentially. It is difficult to get an overview of a large collection of personal photos without browsing though the entire database manually. In this research, we propose a framework to generate representative subset summaries from photo collections hosted on web archives or social networks. We define salient properties of an effective photo summary and model summarization as an optimization of these properties, given the size constraints. We also introduce metrics for evaluating photo summaries based on their information content and the ability to satisfy users information needs. Our experiments show that our summarization framework performs better than baseline algorithms.

Semantics In Digital Photos: A Contenxtual Analysis

Interpreting the semantics of an image is a hard problem. However, for storing and indexing large multimedia collections, it is essential to build systems that can automatically extract semantics from images. In this research we show how we can fuse content and context to extract semantics from digital photographs. Our experiments show that if we can properly model context associated with media, we can interpret semantics using only a part of high dimensional content data.

Extractive summarization of personal photos from life events

Manually sifting through large collections of personal photos shot at various life events is both tedious and inefficient. In this paper, we propose a photo summarization system which creates a representative subset summary by extracting photos from a larger set shot at an event (e.g., in a trip, birthday, etc). We define three properties that are necessary to generate an effective summary: relevance, diversity and coverage. We propose methods to compute them using multimodal content and context data. The objective for automatic photo summarization is formulated as an optimization of these properties. We discuss algorithms that solve the problem efficiently. A dataset of 7,700 photos from personal life events is created with user-generated ground truth summary. We also propose objective metrics to evaluate summaries automatically. Evaluations using both objective metrics and user feedback show our models can generate summaries which are much better than baselines.

Concept annotation and search space decrement of digital photos using optical context information

A modern digital camera is not just a single sensor capturing light. It is an ensemble of different sensors which capture independent contextual information about the photo shooting event. This is stored as metadata in the image. In this paper, we demonstrate how the optical metadata (data related to the optics of the camera) can be retrieved, interpreted and used along with content information for organizing and indexing digital photos. Our model is based on the physics of vision and operation of a camera. We use our algorithm on images from personal photo albums. Our results show that the optical metadata improves annotation performance and decreases the search space for retrieval.

Automatic Summarization of Personal Photo Collections

Photo taking and sharing devices (e.g., smart phones, digital cameras, etc) have become extremely popular in recent times. Photo enthusiasts today capture moments of their personal lives using these devices. This has resulted in huge collections of photos stored in various personal archives. The exponential growth of online social networks and web based photo sharing platforms have added fuel to this fire. According to recent estimates [46], three billion photos are uploaded on the social network Facebook per month. This photo data overload has created some major challenges. One of the them is automatic generation of representative overviews from large photo collections. Manual browsing of photo corpora is not only tedious, but also time inefficient. Hence, development of an automatic photo summarization system is not only a research but also a practical challenge. In this dissertation, we present a principled approach for generation of size constrained overview summaries from large personal photo collections.