Nicolas Tsapatsoulis

Emotion recognition in human-computer interaction

Two channels have been distinguished in human interaction: one transmits explicit messages, which may be about anything or nothing; the other transmits implicit messages about the speakers themselves. Both linguistics and technology have invested enormous efforts in understanding the first, explicit channel, but the second is not as well understood. Understanding the other partys emotions is one of the key tasks associated with the second, implicit channel. To tackle that task, signal processing and analysis techniques have to be developed, while, at the same time, consolidating psychological and linguistic analyses of emotion. This article examines basic issues in those areas. It is motivated by the PKYSTA project, in which we aim to develop a hybrid system capable of using information from faces and voices to recognize peoples emotions.

Parameterized facial expression synthesis based on MPEG-4

In the framework of MPEG-4, one can include applications where virtual agents, utilizing both textual and multisensory data, including facial expressions and nonverbal speech help systems become accustomed to the actual feelings of the user. Applications of this technology are expected in educational environments, virtual collaborative workplaces, communities, and interactive entertainment. Facial animation has gained much interest within the MPEG-4 framework; with implementation details being an open research area (Tekalp, 1999). In this paper, we describe a method for enriching human computer interaction, focusing on analysis and synthesis of primary and intermediate facial expressions (Ekman and Friesen (1978)). To achieve this goal, we utilize facial animation parameters (FAPs) to model primary expressions and describe a rule-based technique for handling intermediate ones. A relation between FAPs and the activation parameter proposed in classical psychological studies is established, leading to parameterized facial expression analysis and synthesis notions, compatible with the MPEG-4 standard.

Face detection in color images and video sequences

Face detection is no longer necessarily correlated with face recognition. Instead it has been established as an important tool in the framework of many multimedia applications like indexing, scene classification and news summarization. Many face detection algorithms based on skin color characteristics have appeared in the literature. Most of them face generalization problems due to the skin color model they use. Moreover, their verification stage exclusively depends on simple shape features limiting the reliability of detection. In this work we present a Gaussian model of the skin color distribution whose parameters are re-estimated based on the current image I frame. In this way the generalization problem is limited. Furthermore the verification stage, applied in the detected skin segments, is based on a template matching variation providing a robust detection.

A snake model for object tracking in natural sequences

Abstract Tracking moving objects in video sequences is a task that emerges in various fields of study: video analysis, computer vision, biomedical systems, etc. In the last decade, special attention has been drawn to problems concerning tracking in real-world environments, where moving objects do not obey any afore-known constraints about their nature and motion or the scenes they are moving in. Apart from the existence of noise and environmental changes, many problems are also concerned, due to background texture, complicated object motion, and deformable and/or articulated objects, changing their shape while moving along time. Another phenomenon in natural sequences is the appearance of occlusions between different objects, whose handling requires motion information and, in some cases, additional constraints. In this work, we revisit one of the most known active contours, the Snakes, and we propose a motion-based utilization of it, aiming at successful handling of the previously mentioned problems. The use of the object motion history and first order statistical measurements of it, provide us with information for the extraction of uncertainty regions, a kind of shape prior knowledge w.r.t. the allowed object deformations. This constraining also makes the proposed method efficient, handling the trade-off between accuracy and computation complexity. The energy minimization is approximated by a force-based approach inside the extracted uncertainty regions, and the weights of the total snake energy function are automatically estimated as respective weights in the resulting evolution force. Finally, in order to handle background complexity and partial occlusion cases, we introduce two rules, according to which the moving object region is correctly separated from the background, whereas the occluded boundaries are estimated according to the objects expected shape. To verify the performance of the proposed method, some experimental results are included, concerning different cases of object tracking, indoors and outdoors, with rigid and deformable objects, noisy and textured backgrounds, as well as appearance of occlusions.

Facial Image Indexing in Multimedia Databases

Abstract:Pictures and video sequences showing human faces are of high importance in content-based retrieval systems, and consequently face detection has been established as an important tool in the framework of many multimedia applications like indexing, scene classification and news summarisation. In this work, we combine skin colour and shape features with template matching in an efficient way for the purpose of facial image indexing. We propose an adaptive two-dimensional Gaussian model of the skin colour distribution whose parameters are re-estimated based on the current image or frame, reducing generalisation problems. Masked areas obtained from skin colour detection are processed using morphological tools and assessed using global shape features. The verification stage is based on a template matching variation providing robust detection. Facial images and video sequences are indexed according to the number of included faces, their average colour components and their scale, leading to new types of content-based retrieval criteria in query-by-example frameworks. Experimental results have shown that the proposed implementation combines efficiency, robustness and speed, and could be easily embedded in generic visual information retrieval systems or video databases.

MuLVAT: A Video Annotation Tool Based on XML-Dictionaries and Shot Clustering

Recent advances in digital video technology have resulted in an explosion of digital video data which are available through the Web or in private repositories. Efficient searching in these repositories created the need of semantic labeling of video data at various levels of granularity, i.e., movie, scene, shot, keyframe, video object, etc. Through multilevel labeling video content is appropriately indexed, allowing access from various modalities and for a variety of applications. However, despite the huge efforts for automatic video annotation human intervention is the only way for reliable semantic video annotation. Manual video annotation is an extremely laborious process and efficient tools developed for this purpose can make, in many cases, the true difference. In this paper we present a video annotation tool, which uses structured knowledge, in the form of XML dictionaries, combined with a hierarchical classification scheme to attach semantic labels to video segments at various level of granularity. Video segmentation is supported through the use of an efficient shot detection algorithm; while shots are combined into scenes through clustering with the aid of a Genetic Algorithm scheme. Finally, XML dictionary creation and editing tools are available during annotation allowing the user to always use the semantic label she/he wishes instead of the automatically created ones.

Improved detection of breast cancer nuclei using modular neural networks

Discusses the analysis of nuclei in histopathological sections with a system that closely simulates human experts. The evaluation of immunocytochemically stained histopathological sections presents a complex problem due to many variations that are inherent in the methodology. In this respect, many aspects of immunocytochemistry remain unresolved, despite the fact that results may carry important diagnostic, prognostic, and therapeutic information. In this article, a modular neural network-based approach to the detection and classification of breast cancer nuclei stained for steroid receptors in histopathological sections is described and evaluated.

AN EMBEDDED SALIENCY MAP ESTIMATOR SCHEME: APPLICATION TO VIDEO ENCODING

In this paper we propose a novel saliency-based computational model for visual attention. This model processes both top-down (goal directed) and bottom-up information. Processing in the top-down channel creates the so called skin conspicuity map and emulates the visual search for human faces performed by humans. This is clearly a goal directed task but is generic enough to be context independent. Processing in the bottom-up information channel follows the principles set by Itti et al. but it deviates from them by computing the orientation, intensity and color conspicuity maps within a unified multi-resolution framework based on wavelet subband analysis. In particular, we apply a wavelet based approach for efficient computation of the topographic feature maps. Given that wavelets and multiresolution theory are naturally connected the usage of wavelet decomposition for mimicking the center surround process in humans is an obvious choice. However, our implementation goes further. We utilize the wavelet decomposition for inline computation of the features (such as orientation angles) that are used to create the topographic feature maps. The bottom-up topographic feature maps and the top-down skin conspicuity map are then combined through a sigmoid function to produce the final saliency map. A prototype of the proposed model was realized through the TMDSDMK642-0E DSP platform as an embedded system allowing real-time operation. For evaluation purposes, in terms of perceived visual quality and video compression improvement, a ROI-based video compression setup was followed. Extended experiments concerning both MPEG-1 as well as low bit-rate MPEG-4 video encoding were conducted showing significant improvement in video compression efficiency without perceived deterioration in visual quality.

Broadcast news parsing using visual cues: a robust face detection approach

Automatic content-based analysis and indexing of broadcast news recordings or digitized news archives is becoming an important tool in the framework of many multimedia interactive services such as news summarization, browsing, retrieval and news-on-demand (NoD) applications. Existing approaches have achieved high performance in such applications but heavily rely on textual cues such as closed caption tokens and teletext transcripts. We present an efficient technique for temporal segmentation and parsing of news recordings based on visual cues that can either be employed as a stand-alone application for non-closed captioned broadcasts or integrated with audio and textual cues of existing systems. The technique involves robust face detection by means of color segmentation, skin color matching and shape processing, and is able to identify typical news instances like anchor persons, reports and outdoor shots.

Efficient face detection for multimedia applications

Face detection is becoming an important tool in the framework of many multimedia applications. Several face detection algorithms based on skin color characteristics have appeared in the literature. Most of them have generalization problems due to the skin color model they use. We present a study which attempts to minimize the generalization problem by combining the M-RSST color segmentation algorithm with a Gaussian model of the skin color distribution and global shape features. Moreover by associating the resultant segments with a face probability we can index and retrieve facial images from multimedia databases.