André Mourão

Multimodal medical information retrieval with unsupervised rank fusion

Modern medical information retrieval systems are paramount to manage the insurmountable quantities of clinical data. These systems empower health care experts in the diagnosis of patients and play an important role in the clinical decision process. However, the ever-growing heterogeneous information generated in medical environments poses several challenges for retrieval systems. We propose a medical information retrieval system with support for multimodal medical case-based retrieval. The system supports medical information discovery by providing multimodal search, through a novel data fusion algorithm, and term suggestions from a medical thesaurus. Our search system compared favorably to other systems in 2013 ImageCLEFMedical.

Competitive affective gaming: winning with a smile

Human-computer interaction (HCI) is expanding towards natural modalities of human expression. Gestures, body movements and other affective interaction techniques can change the way computers interact with humans. In this paper, we propose to extend existing interaction paradigms by including facial expression as a controller in videogames. NovaEmötions is a multiplayer game where players score by acting an emotion through a facial expression. We designed an algorithm to offer an engaging interaction experience using the facial expression. Despite the novelty of the interaction method, our game scoring algorithm kept players engaged and competitive. A user study done with 46 users showed the success and potential for the usage of affective-based interaction in videogames, i.e., the facial expression as the sole controller in videogames. Moreover, we released a novel facial expression dataset with over 41,000 images. These face images were captured in a novel and realistic setting: users playing games where a players facial expression has an impact on the game score.

High-Dimensional Indexing by Sparse Approximation

In this paper we propose a high-dimensional indexing technique, based on sparse approximation techniques to speed up the search and retrieval of similar images given a query image feature vector. Feature vectors are stored on an inverted indexed based on a sparsifying dictionary for l0 regression, optimized to reduce the data dimensionality. It concentrates the energy of the original vector on a few coefficients of a higher dimensional representation. The index explores the coefficient locality of the sparse representations, to guide the search through the inverted index. Evaluation on three large-scale datasets showed that our method compares favorably to the state-of-the-art. On a 1 million dataset of SIFT vectors, our method achieved 60.8% precision at 50 by inspecting only 5% of the full dataset, and by using only 1/4 of the time a linear search takes.

Facial Expression Recognition by Sparse Reconstruction with Robust Features

Facial expression analysis relies on the accurate detection of a few subtle face traces. According to specialists [3], facial expressions can be decomposed into a set of small Action Units (AU) corresponding to different face regions. In this paper, we propose to detect facial expressions with sparse reconstruction methods. Inspired by sparse regularization and sparse over-complete dictionaries, we aim at finding the minimal set of face atoms that can represent a given expression. l 1 based reconstruction computes the deviation from the average face as an additive model of facial expression atoms and classify unknown expressions accordingly. We compared the proposed approach to existing methods on the well-known Cohn-Kanade (CK+) dataset [6]. Results indicate that sparse reconstruction with l 1 penalty outperforms SVM and k-NN baselines with the tested features. The best accuracy (97%) was obtained using sparse reconstruction in an unsupervised setting.

Inverse square rank fusion for multimodal search

Rank fusion is the task of combining multiple ranked document lists (ranks) into a single ranked list. It is a late fusion approach designed to improve the rankings produced by individual systems. Rank fusion techniques have been applied throughout multiple domains: e.g. combining results from multiple retrieval functions, or multimodal search where several feature spaces are common. In this paper, we present the Inverse Square Rank fusion method family, a set of novel fully unsupervised rank fusion methods based on quadratic decay and on logarithmic document frequency normalization. Our experiments created with standard Information Retrieval datasets (image and text fusion) and image datasets (image features fusion), show that ISR outperforms existing rank fusion algorithms. Thus, the proposed technique has comparable or better performance than existing state-of-the-art approaches, while maintaining a low computational complexity and avoiding the need for document scores or training data.

Crowdsourcing facial expressions for affective-interaction

The contribution of this paper is a dataset to foster an affective-interaction research and applications.Facial-expression images were captured while users played a game that responded to facial expressions.Statistical consensus techniques were used to merge 229,584 judgments obtained by crowdsourcing to produce high-quality labels for the 42,911 images. Affective-interaction in computer games is a novel area with several new challenges, such as detecting players facial expressions robustly. Many of the existing facial expression datasets are composed of a set of posed face images not captured in a realistic affective-interaction setting. The contribution of this paper is an affective-interaction dataset captured while users were playing a game that reacted to their facial-expressions. This dataset was the result of a framework designed for gathering affective-interaction data and annotating this data with high-quality labels. The first part of the framework is a computer game 15 planned to elicit a particular facial expressions that directly control the game outcome. Thus, the game creates a true and engaging affective-interaction scenario where facial-expressions data were captured. The proposed dataset is composed of a series of sequential video frames where faces were detected while users interacted with a game with their facial expressions. The second part of the framework is a crowdsourcing process designed to ask annotators to identify the facial-expression present in a given face image. Each face image was annotated with a facial-expression: happy, anger, disgust, contempt, sad, fear, surprise, and neutral. We examined how the annotators performance was affected by multiple variables, e.g., reward, judgment limits, golden questions. Once these parameters were tuned, we gathered 229,584 annotations for the whole 42,911 images. Statistical consensus techniques were then used to merge the annotators judgments and produce high-quality image-labels. Finally, we compared different classifiers trained on both ground-truth (expert) labels and crowdsourcing labels: we observed no differences in classification accuracy, which confirms the quality of the produced labels. Thus, we conclude that the proposed affective-interaction dataset provides a unique set of images of people playing games with their facial expressions and labels with a quality similar to that of expert labels (differences are less than 9%).

Scalable Multimodal Search with Distributed Indexing by Sparse Hashing

Multimedia search systems must deal with an increasingly large and heterogeneous amount of data. Several challenges exist when deploying real-world search engines for such data. Existing literature does not properly tackle the many efficiency issues that such task requires. In this paper, we address several of the key efficiency aspects required to deploy a distributed search engine, capable of handling several millions of multimedia documents. The search engine builds on a framework designed to: first, ease the distribution of documents and queries across cluster-nodes, second, index media efficiently for fast similarity search and third aggregate ranked results from several heterogeneous sources. Moreover, the proposed framework is flexible enough to support several state-of-the-art indexing and aggregation techniques. At the heart of the indexing architecture lies an inverse index structure optimized for sparse hashes, that speeds up the retrieval of similar descriptors. To leverage the distributed nature of the search framework, the proposed aggregation technique offers a low temporal complexity overhead and it is agnostic to the index type (a key aspect to support simultaneous modalities). A comprehensive evaluation with both general IR metrics and efficiency metrics, provides a unique assessment of the several efficiency bottlenecks faced by a search engine. In addition, we test the scalability of the search framework to multiple index sizes, i.e., up to 5 million documents per cluster-node.

Large-scale high-dimensional indexing by sparse hashing with l 0 approximation

In this paper we propose a large-scale high-dimensional indexing algorithm based on sparse approximation and inverted indexing. Our goal was to devise a method that smoothly scales to handle databases with over 100 million descriptors on a single machine. To meet this goal, we implemented an inverted indexed based on a sparsifying dictionary with l0 regression to assign documents to buckets. The sparsifying dictionary is optimized to reduce the data dimensionality, by concentrating the energy of the original vector on a few coefficients of a higher dimensional representation. These descriptors are added to an inverted index explores the locality of the coefficients of sparse representations to enable efficient pruned search. Evaluation on four large-scale datasets with multiple types of features showed that our method compares favorably to state-of-the-art techniques. On a 100 million dataset of SIFT descriptors, our method achieved 47.6 % precision at 50, by inspecting only 1 % of the full dataset, and by using only 1/20 of the time of a linear search.

Patient-Age Extraction for Clinical Reports Retrieval

Patient demographics are of great importance in clinical decision processes for both diagnosis, tests and treatments. Natural language is the standard in clinical case reports, however, numerical concepts, such as age, do not show their full potential when treated as text tokens. In this paper, we consider the patient age as a numerical dimension and investigate several Kernel methods to smooth a temporal retrieval model. We extract patient age from the clinical case narrative and extend a Dirichlet language to include the temporal dimension. Experimental results on a clinical decision support task, showed that our proposal achieves a relative improvement of 5.7% at the top 10 retrieved documents over a time agnostic baseline.

Low-Complexity Supervised Rank Fusion Models

Combining multiple retrieval functions can lead to notable gains in retrieval performance. Learning to Rank (LETOR) techniques achieve outstanding retrieval results, by learning models with no bounds on model complexity. Often, minor retrieval gains are attained at a significant cost in model complexity. This paper focuses on the research question:can less complex models achieve results comparable to LETOR models? In this paper, we investigate an approach for the selection and fusion of rank lists with low-complexity models. The described Learning to Fuse (L2F) algorithm, is a supervised rank fusion procedure that controls the model complexity by discarding rank lists that bring minor improvements to final rank. Evaluation results, on two different datasets, show that it is indeed possible to achieve a retrieval performance comparable to LETOR methods, using only 3-5% of the rank lists of the number of rank lists used by LETOR methods.