Anupama Mallik

Nrityakosha: Preserving the intangible heritage of Indian classical dance

Preservation of intangible cultural heritage, such as music and dance, requires encoding of background knowledge together with digitized records of the performances. We present an ontology-based approach for designing a cultural heritage repository for that purpose. Since dance and music are recorded in multimedia format, we use Multimedia Web Ontology Language (MOWL) to encode the domain knowledge. We propose an architectural framework that includes a method to construct the ontology with a labeled set of training data and use of the ontology to automatically annotate new instances of digital heritage artifacts. The annotations enable creation of a semantic navigation environment in a cultural heritage repository. We have demonstrated the efficacy of our approach by constructing an ontology for the cultural heritage domain of Indian classical dance, and have developed a browsing application for semantic access to the heritage collection of Indian dance videos.

Multimedia ontology learning for automatic annotation and video browsing

In this work, we offer an approach to combine standard multimedia analysis techniques with knowledge drawn from conceptual metadata provided by domain experts of a specialized scholarly domain, to learn a domain-specific multimedia ontology from a set of annotated examples. A standard Bayesian network learning algorithm that learns structure and parameters of a Bayesian network is extended to include media observables in the learning. An expert group provides domain knowledge to construct a basic ontology of the domain as well as to annotate a set of training videos. These annotations help derive the associations between high-level semantic concepts of the domain and low-level MPEG-7 based features representing audio-visual content of the videos. We construct a more robust and refined version of this ontology by learning from this set of conceptually annotated videos. To encode this knowledge, we use MOWL, a multimedia extension of Web Ontology Language (OWL) which is capable of describing domain concepts in terms of their media properties and of capturing the inherent uncertainties involved. We use the ontology specified knowledge for recognizing concepts relevant to a video to annotate fresh addition to the video database with relevant concepts in the ontology. These conceptual annotations are used to create hyperlinks in the video collection, to provide an effective video browsing interface to the user.

MOWL: An ontology representation language for web-based multimedia applications

Several multimedia applications need to reason with concepts and their media properties in specific domain contexts. Media properties of concepts exhibit some unique characteristics that cannot be dealt with conceptual modeling schemes followed in the existing ontology representation and reasoning schemes. We have proposed a new perceptual modeling technique for reasoning with media properties observed in multimedia instances and the latent concepts. Our knowledge representation scheme uses a causal model of the world where concepts manifest in media properties with uncertainties. We introduce a probabilistic reasoning scheme for belief propagation across domain concepts through observation of media properties. In order to support the perceptual modeling and reasoning paradigm, we propose a new ontology language, Multimedia Web Ontology Language (MOWL). Our primary contribution in this article is to establish the need for the new ontology language and to introduce the semantics of its novel language constructs. We establish the generality of our approach with two disperate knowledge-intensive applications involving reasoning with media properties of concepts.

Acquisition of multimedia ontology: an application in preservation of cultural heritage

A domain-specific ontology models a specific domain or part of the world. In fact, ontologies have proven to be an excellent medium for capturingpagebreak the knowledge of a domain. We propose an ontology learning scheme in this paper which combines standard multimedia analysis techniques with knowledge drawn from conceptual meta-data to learn a domain-specific multimedia ontology from a set of annotated examples. A standard machine-learning algorithm that learns structure and parameters of a Bayesian network is extended to include media observables in the learning. An expert group provides domain knowledge to construct a basic ontology of the domain as well as to annotate a set of training videos. These annotations help derive the associations between high-level semantic concepts of the domain and low-level media features. We construct a more robust and refined version of the basic ontology by learning from this set of conceptually annotated data. We show an application of our ontology-based framework for exploration of multimedia content, in the field of cultural heritage preservation. By constructing an ontology for the cultural heritage domain of Indian classical dance, and by offering an application for semantic annotation of the heritage collection of Indian dance videos, we demonstrate the efficacy of ou approach.

Learning ontology for personalized video retrieval

This paper proposes a new method for using implicit user feedback from clickthrough data to provide personalized ranking of results in a video retrieval system. The annotation based search is complemented with a feature based ranking in our approach. The ranking algorithm uses belief revision in a Bayesian Network, which is derived from a multimedia ontology that captures the probabilistic association of a concept with expected video features. We have developed a content model for videos using discrete feature states to enable Bayesian reasoning and to alleviate on-line feature processing overheads. We propose a reinforcement learning algorithm for the parameters of the Bayesian Network with the implicit feedback obtained from the clickthrough data.

An Ontology Based Personalized Garment Recommendation System

We present a novel method for content-based recommendation of media-rich commodities using probabilistic multimedia ontology. The ontology encodes subjective knowledge of experts that enables interpretation of media based and semantic product features in context of domain concepts. Our recommendation is based on semantic compatibility between the products and user profile in context of use. We use probabilistic knowledge representation and reasoning framework to achieve robust and flexible results. The approach has been validated with fashion preferences of several individuals with a large collection of Sarees, an ethnic dress for women in Indian subcontinent.

Ontology based context aware situation tracking

Ubiquitous intelligent devices have enabled provision of smart services to people in seamless way. Context-awareness helps understand current state-of-affairs or the situation in which presently the system is. This understanding helps the IoT application provide more relevant and smarter services based on situations that change over a period of time. In this paper, we propose a novel context-aware situation-tracking framework that makes use of an ontology. The ontology represents the conceptual model of a dynamic world, where situations evolve over time in changing contexts. The ontology provides the reasoning framework to infer about a situation based on the input context data as well as the past information of earlier situations. Future situations can be predicted with some belief based on current situation and incoming context data. The context data is acquired from sensor devices and external inputs. For every recognized situation, system recommends some actions to provide context-aware service. We use Multimedia Web Ontology Language (MOWL) to represents the ontology. MOWL proposes a probabilistic framework for reasoning with uncertainties linked with observation of context. It makes use of Dynamic Bayesian networks to predict and track the dynamically changing situations. We illustrate use of this framework for Smart Mirror use case.

Using Concept Recognition to Annotate a Video Collection

In this paper, we propose a scheme based on an ontological framework, to recognize concepts in multimedia data, in order to provide effective content-based access to a closed, domain-specific multimedia collection. The ontology for the domain is constructed from high-level knowledge of the domain lying with the domain experts, and further fine-tuned and refined by learning from multimedia data annotated by them. MOWL, a multimedia extension to OWL, is used to encode the concept to media-feature associations in the ontology as well as the uncertainties linked with observation of the perceptual multimedia data. Media feature classifiers help recognize low-level concepts in the videos, but the novelty of our work lies in discovery of high-level concepts in video content using the power of ontological relations between the concepts. This framework is used to provide rich, conceptual annotations to the video database, which can further be used to create hyperlinks in the video collection, to provide an effective video browsing interface to the user.

Archiving mural paintings using an ontology based approach

In this paper, we propose an archiving scheme for heritage mural paintings. The mural paintings typically depict stories from folk-lore, mythology and history. These narratives provide content-based correlations between different pieces of art. Our e-heritage scheme for archiving the mural paintings is based on an ontology which captures the background knowledge of these narratives. Media features and patterns derived from the mural content are used to enrich the ontology with multimedia data. We have used the multimedia web ontology language as our ontology representation scheme, as it allows perceptual modelling of domain concepts in terms of their media properties, as well as reasoning with uncertainties. Besides the mural content and its knowledge, the ontology also helps encode other aspects of the mural paintings like their painting style, color, physical location, time-period, etc., which are important parameters of their preservation. We propose a framework to provide cross-modal semantic linkage between semantically annotated content of a repository of Indian mural paintings, and a collection of labelled text documents of their narratives. This framework, based on a multimedia ontology of the domain, helps preserve the cultural heritage encoded in these artefacts.

Preservation of intangible heritage: a case-study of indian classical dance

Cultural heritage is encoded in a variety of forms. The task of preserving heritage involves preserving the tangible and intangible resources that broadly define that heritage. A significant aspect of intangible heritage resources are performing arts which include classical dance and music. Digital heritage resources include heritage artefacts in digitized form as well as the background knowledge that puts them in perspective. We present an ontology based approach to capture and preserve the knowledge with digital heritage artefacts. Since the artefacts are generally preserved in multimedia format, we propose the use of Multimedia Web Ontology (MOWL) that supports probabilistic reasoning with media properties of domain concepts, to encode the domain knowledge. We propose an architectural framework that includes a method to construct the ontology with a labelled set of training data and use of the ontology to automatically annotate new instances of digital heritage artefacts. The annotations enable creation of a semantic navigation environment in a cultural heritage repository. We have realized a proof of concept in the domain of Indian Classical Dance and present some results.