Matthias Wimmer

Learning Local Objective Functions for Robust Face Model Fitting

Model-based techniques have proven to be successful in interpreting the large amount of information contained in images. Associated fitting algorithms search for the global optimum of an objective function, which should correspond to the best model fit in a given image. Although fitting algorithms have been the subject of intensive research and evaluation, the objective function is usually designed ad hoc, based on implicit and domain-dependent knowledge. In this article, we address the root of the problem by learning more robust objective functions. First, we formulate a set of desirable properties for objective functions and give a concrete example function that has these properties. Then, we propose a novel approach that learns an objective function from training data generated by manual image annotations and this ideal objective function. In this approach, critical decisions such as feature selection are automated, and the remaining manual steps hardly require domain-dependent knowledge. Furthermore, an extensive empirical evaluation demonstrates that the obtained objective functions yield more robustness. Learned objective functions enable fitting algorithms to determine the best model fit more accurately than with designed objective functions.

Audiovisual Behavior Modeling by Combined Feature Spaces

Great interest is recently shown in behavior modeling, especially in public surveillance tasks. In general it is agreed upon the benefits of use of several input cues as audio and video. Yet, synchronization and fusion of these information sources remains the main challenge. We therefore show results for a feature space combination, which allows for overall feature space optimization. Audio and video features are thereby firstly derived as low-level-descriptors. Synchronization and feature combination is achieved by multivariate time-series analysis. Test-runs on a database of aggressive, cheerful, intoxicated, nervous, neutral, and tired behavior in an airplane situation show a significant improvement over each single modality.

Learning Robust Objective Functions for Model Fitting in Image Understanding Applications

Model-based methods in computer vision have proven to be a good approach for compressing the large amount of information in images. Fitting algorithms search for those parameters of the model that optimise the objective function given a certain image. Although fitting algorithms have been the subject of intensive research and evaluation, the objective function is usually designed ad hoc and heuristically with much implicit domain-dependent knowledge. This paper formulates a set of requirements that robust objective functions should satisfy. Furthermore, we propose a novel approach that learns the objective function from training images that have been annotated with the preferred model parameters. The requirements are automatically enforced during the learning phase, which yields generally applicable objective functions. We compare the performance of our approach to other approaches. For this purpose, we propose a set of indicators that evaluate how well an objective function meets the stated requirements.

A Person and Context Specific Approach for Skin Color Classification

Skin color is an important feature of faces. Various applications benefit from robust skin color detection. Depending on camera settings, illumination, shadows, peoples tans, and ethnic groups skin color looks differently, which is a challenging aspect for detecting it automatically. In this paper, we present an approach that uses a high level vision module to detect an image specific skin color model. This model is then used to adapt parametric skin color classifiers to the processed image. This approach is capable to distinguish skin color from extremely similar colors, such as lip color or eyebrow color. Its high speed and high accuracy make it appropriate for real time applications such as face tracking and recognition of facial expressions

A Model Based Approach for Expressions Invariant Face Recognition

This paper describes an idea of recognizing the human face in the presence of strong facial expressions using model based approach. The features extracted for the face image sequences can be efficiently used for face recognition. The approach follows in 1) modeling an active appearance model (AAM) parameters for the face image, 2) using optical flow based temporal features for facial expression variations estimation, 3) and finally applying classifier for face recognition. The novelty lies not only in generation of appearance models which is obtained by fitting active shape model (ASM) to the face image using objective functions but also using a feature vector which is the combination of shape, texture and temporal parameters that is robust against facial expression variations. Experiments have been performed on Cohn-Kanade facial expression database using 62 subjects of the database with image sequences consisting of more than 4000 images. This achieved successful face recognition rate up to 91.17% using binary decision tree (BDT), 98.6% using Bayesian Networks (BN) with 10-fold cross validation in the presence of six different facial expressions.

Facial expression recognition for human-robot interaction: a prototype

To be effective in the human world robots must respond to human emotional states. This paper focuses on the recognition of the six universal human facial expressions. In the last decade there has been successful research on facial expression recognition (FER) in controlled conditions suitable for human-computer interaction [1,2,3,4,5,6,7,8]. However the human-robot scenario presents additional challenges including a lack of control over lighting conditions and over the relative poses and separation of the robot and human, the inherent mobility of robots, and stricter real time computational requirements dictated by the need for robots to respond in a timely fashion. Our approach imposes lower computational requirements by specifically adapting model-based techniques to the FER scenario. It contains adaptive skin color extraction, localization of the entire face and facial components, and specifically learned objective functions for fitting a deformable face model. Experimental evaluation reports a recognition rate of 70% on the Cohn-Kanade facial expression database, and 67% in a robot scenario, which compare well to other FER systems.

Experiences with an Emotional Sales Agent

With COSIMAB2B we demonstrate a prototype of a complex and visionary e-procurement application. The embodied character agent named COSIMA is able to respect a customer’s preferences and deals with him or her via natural speech. She expresses various emotions via mimic, gesture, combined with speech synthesis, and COSIMA is even able to consider the customer’s emotions via mimic recognition. As first observations show, this is a very promising approach to improve the bargaining with the customer or the recommendation of products.

Adjusted pixel features for robust facial component classification

Within the last decade increasing computing power and the scientific advancement of algorithms allowed the analysis of various aspects of human faces such as facial expression estimation [20], head pose estimation [17], person identification [2] or face model fitting [31]. Today, computer scientists can use a bunch of different techniques to approach this challenge [4,29,3,17,9,21]. However, each of them still has to deal with non-perfect accuracy or high execution times. This is mainly because the extraction of descriptive features is challenging in real-world scenarios to any image understanding application. In this paper, we consider the extraction of more descriptive information from the image for face analysis tasks. Our approach automatically determines a set of characteristics that describe the conditions of the entire image. They are based on semantic information that describes the location of facial components, such as skin, lips, eyes, and brows. From these image characteristics, pixel features are determined that are highly tuned to the task of interpreting images of human faces. The extracted features are applied to train pixel-based classifiers, which is the straight-forward approach because this task suffers from high intra-class and small inter-class color variations due to changing context conditions such as the persons ethnic group or lighting condition. In contrast, more elaborate classifiers that additionally consider shape or region features are not real-time capable. The success of this approach relies on the fact that we do not manually select the calculation rules but we provide a multitude of features of various kinds, both color-related and space-related. A Machine Learning algorithm then decides which of them are important and which are not rendering the approach fast due to its pixel-based nature and accurate due to the highly descriptive features the same time.

A real time system for model-based interpretation of the dynamics of facial expressions

Our system runs at 10 fps on a 2.0 GHz processor and an image resolution of 640times480 pixels. High quality objective functions that are learned from annotated example images ensure both an accurate and fast computation of the model parameters. Our demonstrator for facial expression estimation has been presented at several events with political audience and on TV. However, the approach of robust face models fitting, forms the basis of various more applications such as gaze detection or gender estimation. The drawback of our approach is that the data base from which the objective function is learned needs to cover all aspects of face properties. If, for instance, the database did not contain images of bearded men the objective function will fail when confronted with such an image. Furthermore, the data base has to be manually annotated. Although no expert knowledge is required, this task requires a considerable amount of time. An online fitting demonstration is available.

Tailoring Model-Based Techniques to Facial Expression Interpretation

Computers have been widely deployed to our daily lives, but human-computer interaction still lacks intuition. Researchers intend to resolve these shortcomings by augmenting traditional systems with human-like interaction capabilities. Knowledge about human emotion, behavior, and intention is necessary to construct convenient interaction mechanisms. Today, dedicated hardware often infers the emotional state from human body measures. Similar to humans interpreting facial expressions, our approach acquires video information using standard hardware that does not interfere with people to accomplish this task. It exploits model-based techniques that accurately localize facial features, seamlessly track them through image sequences, and finally interpret the visible information. We make use of state-of-the-art techniques and specifically adapt most of the components involved to this scenario, which provides high accuracy and real-time capability. We base our experimental evaluation on publicly available databases and compare its results to related approaches. Our proof-of-concept demonstrates the feasibility of our approach and shows promising for integration into various applications.