Sotiris Manitsaris

A Multimodal Approach for the Safeguarding and Transmission of Intangible Cultural Heritage: The Case of i-Treasures

Intangible Cultural Heritage (ICH) creations include, amongst other, music, dance, singing, theatre, human skills, and craftsmanship. These cultural expressions are usually transmitted orally and/or using gestures and are modified over a period of time, through a process of collective recreation. As the world becomes more interconnected and many different cultures come into contact, local communities run the risk of losing important elements of their ICH, while young people find it difficult to maintain the connection with the cultural heritage treasured by their elders. In this paper, we present a novel holistic approach for the safeguarding and transmission of ICH that goes beyond the mere digitization of ICH content. Based on multisensory technology for the capturing of ICH, the proposed approach enables the generation of completely novel cultural content. High-level semantics are extracted from the acquired data, enabling researchers to identify possible implicit or hidden correlations between different ICH expressions or interpretation styles and study the evolution of a specific ICH. These data, coupled with other cultural resources, are accessible through the i-Treasures Web-platform, which provides the means for supporting knowledge exchange between researchers as well as know-how transmission from ICH bearers to apprentices.

Vocal tract imaging system for post-laryngectomy voice replacement

The article describes a system that uses real time measurements of the vocal tract to drive a voice-replacement system for post-laryngectomy patients. Based on a thermoformed acquisition helmet, miniature ultrasound machine, and video camera, and incorporating Hidden Markov Model speech recognition, the device has been tested on three speakers, one of whom has undergone a total laryngectomy. Results show that the device obtains exploitable recognition rates, and that performances on normal and post-laryngectomy speakers are nearly identical. The technique can also enable voice communication for normal speakers in situations where silence must be maintained.

The i-Treasures Intangible Cultural Heritage dataset

In this paper, we introduce the i-Treasures Intangible Cultural Heritage (ICH) dataset, a freely available collection of multimodal data captured from different forms of rare ICH. More specifically, the dataset contains video, audio, depth, motion capture data and other modalities, such as EEG or ultrasound data. It also includes (manual) annotations of data, while in some cases additional features and metadata are provided, extracted using algorithms and modules developed within the i-Treasures project. We describe the creation process (sensors, capture setups and modules used), the dataset content and the associated annotations. An attractive feature of this ICH Database is that its the first of its kind, providing annotated multimodal data for a wide range of rare ICH types. Finally, some conclusions are drawn and the future development of the dataset is discussed.

A User-Adaptive Gesture Recognition System Applied to Human-Robot Collaboration in Factories

Enabling Human-Robot collaboration (HRC) requires robot with the capacity to understand its environment and actions performed by persons interacting with it. In this paper we are dealing with industrial collaborative robots on assembly line in automotive factories. These robots have to work with operators on common tasks. We are working on technical gestures recognition to allow robot to understand which task is being executed by the operator, in order to synchronize its actions. We are using a depth-camera with a top view and we track hands positions of the worker. We use discrete HMMs to learn and recognize technical gestures. We are also interested in a system of gestures recognition which can adapt itself to the operator. Indeed, a same technical gesture seems very similar from an operator to another, but each operator has his/her own way to perform it. In this paper, we study an adaptation of the recognition system by modifying the learning database with a addition very small amount of gestures. Our research shows that by adding 2 sets of gestures to be recognized from the operator who is working with the robot, which represents less than 1% of the database, we can improve correct recognitions rate by ~3.5%. When we add 10 sets of gestures, 2.6% of the database, the improvement reaches 5.7%.

Intelligent invariance techniques for music gesture recognition based on skin modelling

A computer vision methodology for the recognition of finger gestures performed on a music instrument has been recently developed and implemented in the PianOrasis system. PianOrasis recognises the gestures of all the five fingers simultaneously, but not in real-time. In this paper, an optimisation of the above methodology is presented, implying the recognition of finger musical gestures performed in space. Scale and rotation invariance techniques are integrated into the system increasing the recognition quality and reducing the processing time. Scale invariance rests on the deterministic modelling of the number of skin pixels in the image. The proposed modelling enable three different sets of filtering parameters for the hand segmentation process, overcoming a long and manual preliminary analysis. More flexible finger gestures, performed in space without music instrument, can be recognised because of the integration of the rotation invariance.

A Consensual and Non-ambiguous Set of Gestures to Interact with UAV in Infantrymen

In the context of using an Unmanned Aerial Vehicle (UAV) in hostile environments, gestures allow to free the operator of bulky control interfaces. Since a navigation plan is defined before the mission, only a few commands have to be activated during the mission. This allows a gestural symbolic interaction that maps commands to a set of gestures. Nevertheless, as gestures are not universal, this asks the question of choosing the proper gestures that are easy to learn memorize and perform. We propose a four step methodology for eliciting a gestural vocabulary, and apply it to this use case. The methodology consists of 4 steps: (1) collecting gestures through user creativity sessions, (2) extracting candidate gestures to build a catalogue, (3) electing the gesture vocabulary and (3) evaluating the non-ambiguity of it. We then discuss the relevance of the GV.

Offline statistical analysis of gestural skills in pottery interaction

The paper presents a methodology for offline statistical analysis of expert technical gestures applied in pottery interaction. The technical gestures are described using rotations of each segment of the upper-part of the body, including hands and head. The motion capture is based on a suit with inertial sensors. The results confirm the initial hypothesis that some of the body parts are highly involved in the interaction with the material (effective gestures) while others are used for obtaining specific body postures that facilitate the interaction (accompanying gestures). Principal Component Analysis has been used to confirm the hypothesis and Jackknife test to evaluate the recognition accuracy of the system using only the subset of the effective gestures. Separate databases for machine learning and testing are used to confirm that the machine is able to discriminate the technical gestures of the potter by using only information from body segments that are involved in effective gestures.

A Natural User Interface for Gestural Expression and Emotional Elicitation to Access the Musical Intangible Cultural Heritage

This article describes a prototype natural user interface, named the Intangible Musical Instrument, which aims to facilitate access to knowledge of performers that constitutes musical Intangible Cultural Heritage using off-the-shelf motion capturing that is easily accessed by the public at large. This prototype is able to capture, model, and recognize musical gestures (upper body including fingers) as well as to sonify them. The emotional status of the performer affects the sound parameters at the synthesis level. Intangible Musical Instrument is able to support both learning and performing/composing by providing to the user not only intuitive gesture control but also a unique user experience. In addition, the first evaluation of the Intangible Musical Instrument is presented, in which all the functionalities of the system are assessed. Overall, the results with respect to this evaluation were very promising.

Intangible Cultural Heritage and New Technologies: Challenges and Opportunities for Cultural Preservation and Development

Intangible cultural heritage (ICH) is a relatively recent term coined to represent living cultural expressions and practices, which are recognised by communities as distinct aspects of identity. The safeguarding of ICH has become a topic of international concern primarily through the work of United Nations Educational, Scientific and Cultural Organization (UNESCO). However, little research has been done on the role of new technologies in the preservation and transmission of intangible heritage. This chapter examines resources, projects and technologies providing access to ICH and identifies gaps and constraints. It draws on research conducted within the scope of the collaborative research project, i-Treasures. In doing so, it covers the state of the art in technologies that could be employed for access, capture and analysis of ICH in order to highlight how specific new technologies can contribute to the transmission and safeguarding of ICH.

A tabletop instrument for manipulation of sound morphologies with hands, fingertips and upper-body

We present a musical instrument, named the Embodied Musical Instrument (EMI) which allows musicians to perform free gestures with the upper--body including hands and fingers thanks to 3D vision sensors, arranged around the tabletop. 3D interactive spaces delimit the boundaries in which the player performs metaphorical gestures in order to play with sound synthesis engines. A physical-based sound synthesis engine and a sampler have been integrated in the system in order to manipulate sound morphologies in the context of electro-acoustic and electronic composition.