Aditya Sankar

Digital heritage

The India Digital Heritage Project is a collaborative initiative between the industry and academia, with the aim of using novel techniques to efficiently capture and present various aspects of Indias diverse heritage, while at the same time advancing the state-of-the art in related research areas. As part of the Digital Heritage Project, we have built a prototype virtual tour of a South Indian temple that, for the first time, integrates technologies such as Photosynth and HDView, opening up new ways to interactively explore visually complex sites. These technologies are combined with audio, video and guided walkthroughs, to provide a compelling end user experience. The accompanying video highlights the key scenarios of our prototype.

Capturing indoor scenes with smartphones

In this paper, we present a novel smartphone application designed to easily capture, visualize and reconstruct homes, offices and other indoor scenes. Our application leverages data from smartphone sensors such as the camera, accelerometer, gyroscope and magnetometer to help model the indoor scene. The output of the system is two-fold; first, an interactive visual tour of the scene is generated in real time that allows the user to explore each room and transition between connected rooms. Second, with some basic interactive photogrammetric modeling the system generates a 2D floor plan and accompanying 3D model of the scene, under a Manhattan-world assumption. The approach does not require any specialized equipment or training and is able to produce accurate floor plans.

An interactive multimedia framework for digital heritage narratives

The cultural heritage of a region is conveyed by both tangible physical artifacts and intangible aspects in the form of stories, dance styles, rituals, etc. Hitherto, the task of creating digital representations for each of these aspects has been addressed in isolation, i.e. using specific media most suited to the artifact such as video, audio, three-dimensional (3D) models, scanning, etc. The challenge of bringing together these separate elements to create a coherent story, however, has remained unaddressed until recently. In this paper we present a unified digital framework that enables the integration of disparate representations of heritage elements into a holistic entity. Our approach results in a compelling and engaging narration that affords a unified user experience. Our solution supports both active (user-controlled explorations) and passive (watching pre-orchestrated narrations) user interactions. We demonstrate the capabilities of our framework through a qualitative user study based on two rich interactive narratives built using our framework: (1) history and folklore surrounding a temple in South India, and (2) a historical account of an educational institution also in South India.

In situ CAD capture

We present an interactive system to capture CAD-like 3D models of indoor scenes, on a mobile device. To overcome sensory and computational limitations of the mobile platform, we employ an in situ, semi-automated approach and harness the users high-level knowledge of the scene to assist the reconstruction and modeling algorithms. The modeling proceeds in two stages: (1) The user captures the 3D shape and dimensions of the room. (2) The user then uses voice commands and an augmented reality sketching interface to insert objects of interest, such as furniture, artwork, doors and windows. Our system recognizes the sketches and add a corresponding 3D model into the scene at the appropriate location. The key contributions of this work are the design of a multi-modal user interface to effectively capture the users semantic understanding of the scene and the underlying algorithms that process the input to produce useful reconstructions.

Interactive Room Capture on 3D-Aware Mobile Devices

We propose a novel interactive system to simplify the process of indoor 3D CAD room modeling. Traditional room modeling methods require users to measure room and furniture dimensions, and manually select models that match the scene from large catalogs. Users then employ a mouse and keyboard interface to construct walls and place the objects in their appropriate locations. In contrast, our system leverages the sensing capabilities of a 3D aware mobile device, recent advances in object recognition, and a novel augmented reality user interface, to capture indoor 3D room models in-situ. With a few taps, a user can mark the surface of an object, take a photo, and the system retrieves and places a matching 3D model into the scene, from a large online database. User studies indicate that this modality is significantly quicker, more accurate, and requires less effort than traditional desktop tools.

In-situ semantic 3D modeling

Semantic 3D models of indoor scenes enable compelling interior design applications such as remodeling, refurnishing and rearrangement of furniture. However, creating these models is still a challenging task. Most existing approaches are designed to work ex-situ or out of context, and rely on the modelers memory, photographs or measurements from the scene. We propose a novel in-situ, mobile capture system that leverages quick and easy semantic input from the user and offloads tedious reconstruction and modeling tasks to the computer. In this way, our system combines the advantages of automatic and manual CAD based methods to significantly reduce modeling time and effort. Our approach runs on commodity mobile devices and can potentially scale to a much larger audience of casual mobile phone users.