Fausto C. Fleites

Hierarchical disaster image classification for situation report enhancement

In this paper, a hierarchical disaster image classification (HDIC) framework based on multi-source data fusion (MSDF) and multiple correspondence analysis (MCA) is proposed to aid emergency managers in disaster response situations. The HDIC framework classifies images into different disaster categories and sub-categories using a pre-defined semantic hierarchy. In order to effectively fuse different sources (visual and text) of information, a weighting scheme is presented to assign different weights to each data resource depending on the hierarchical structure. The experimental analysis demonstrates that the proposed approach can effectively classify disaster images at each logical layer. In addition, the paper also presents an iPad application developed for situation report management using the proposed HDIC framework.

A Multimedia Semantic Retrieval Mobile System Based on HCFGs

A multimedia semantic retrieval system based on hidden coherent feature groups (HCFGs) can support multimedia semantic retrieval on mobile applications. The system can capture the correlation between features and partition the original feature set into HCFGs, which have strong intragroup correlation while maintaining low intercorrelation. The authors present a novel, multimodel fusion scheme to effectively fuse the multimodel results and generate the final ranked retrieval results. In addition, to incorporate user interaction for effective retrieval, the proposed system also features a user feedback mechanism that helps refine the retrieval results.

Correlation-based Feature Analysis and Multi-Modality Fusion framework for multimedia semantic retrieval

In this paper, we propose a Correlation based Feature Analysis (CFA) and Multi-Modality Fusion (CFA-MMF) framework for multimedia semantic concept retrieval. The CFA method is able to reduce the feature space and capture the correlation between features, separating the feature set into different feature groups, called Hidden Coherent Feature Groups (HCFGs), based on Maximum Spanning Tree (MaxST) algorithm. A correlation matrix is built upon feature pair correlations, and then a MaxST is constructed based on the correlation matrix. By performing a graph cut procedure on the MaxST, a set of feature groups are obtained, where the intra-group correlation is maximized and the inter-group correlation is minimized. Finally, one classifier is trained for each of the feature groups, and the generated scores from different classifiers are fused for the final retrieval. The proposed framework is effective because it reduces the dimensionality of the feature space. The experimental results on the NUSWIDE-Lite data set demonstrate the effectiveness of the proposed CFA-MMF framework.

Content-Based Multimedia Retrieval Using Feature Correlation Clustering and Fusion

Nowadays, only processing visual features is not enough for multimedia semantic retrieval due to the complexity of multimedia data, which usually involve a variety of modalities, e.g. graphics, text, speech, video, etc. It becomes crucial to fully utilize the correlation between each feature and the target concept, the feature correlation within modalities, and the feature correlation across modalities. In this paper, the authors propose a Feature Correlation Clustering-based Multi-Modality Fusion Framework FCC-MMF for multimedia semantic retrieval. Features from different modalities are combined into one feature set with the same representation via a normalization and discretization process. Within and across modalities, multiple correspondence analysis is utilized to obtain the correlation between feature-value pairs, which are then projected onto the two principal components. K-medoids algorithm, which is a widely used partitioned clustering algorithm, is selected to minimize the Euclidean distance within the resulted clusters and produce high intra-correlated feature-value pair clusters. Majority vote is applied to subsequently decide which cluster each feature belongs to. Once the feature clusters are formed, one classifier is built and trained for each cluster. The correlation and confidence of each classifier are considered while fusing the classification scores, and mean average precision is used to evaluate the final ranked classification scores. Finally, the proposed framework is applied on NUS-wide Lite data set to demonstrate the effectiveness in multimedia semantic retrieval.

A Visual Analytics Multimedia Mobile System for Emergency Response

We present a novel visual analytics system and multimedia enabled mobile application that allows emergency management (EM) personnel access to timely and relevant disaster situation information. The system is able to semantically integrate text-based emergency management disaster situation reports with related disaster imagery taken in the field by EM responders and community residents. In addition, through an intuitive and seamless Apple iPad application, users are able to interact with the system in diverse places and conditions and thus provide a more effective response. The system is demonstrated via its iPad application which aims at providing relevant and actionable information.

Efficiently integrating MapReduce-based computing into a Hurricane Loss Projection model

Homeowner insurance is a critical issue for Floridians because of the periodic threat hurricanes pose to Florida. Providing fairness into the rate-making policy process, the state of Florida has developed the Florida Public Hurricane Loss Model (FPHLM), an open, public hurricane risk model to assess the risk of wind damage to insured residential properties. For each input property portfolio, the FPHLM processes a large amount of data to provide expected losses over tens of thousand of years of simulation, for which computational efficiency is of paramount importance. This paper presents our work in integrating the atmospheric component into the FPHLM using MapReduce, which resulted in a highly efficient computing platform for generating stochastic hurricane events on a cluster of computers. The experimental results demonstrate the feasibility of utilizing MapReduce for risk modeling components.

Enhancing Product Detection With Multicue Optimization for TV Shopping Applications

Smart TVs allow consumers to watch TV, interact with applications, and access the Internet, thus enhancing the consumer experience. However, the consumers are still unable to seamlessly interact with the contents being streamed, as it is highlighted by TV-enabled shopping. For example, if a consumer is watching a TV show and is interested in purchasing a product being displayed, the consumer can only go to a store or access the Web to make the purchase. It would be more convenient if the consumer could interact with the TV to purchase interesting items. To realize this use case, products in the content stream must be detected so that the TV system notifies consumers of possibly interesting ones. A practical solution must address the detection of complex products, i.e, those that do not have a rigid form and can appear in various poses, which poses a significant challenge. To this end, a multi cue product detection framework is proposed for TV shopping. The framework is generic as it is not tied to specific object detection approaches. Instead, it utilizes appearance, topological, and spatio-temporal cues that make use of a related, easier to detect object class to improve the detection results of the target, more difficult product class. The three cues are jointly considered to select the best path that occurrences of the target product class can follow in the video and thus eliminate false positive occurrences. The empirical results demonstrate the advantages of the proposed approach in improving the precision of the results.

Correlation-based re-ranking for semantic concept detection

Semantic concept detection is among the most important and challenging topics in multimedia research. Its objective is to effectively identify high-level semantic concepts from low-level features for multimedia data analysis and management. In this paper, a novel re-ranking method is proposed based on correlation among concepts to automatically refine detection results and improve detection accuracy. Specifically, multiple correspondence analysis (MCA) is utilized to capture the relationship between a targeted concept and all other semantic concepts. Such relationship is then used as a transaction weight to refine detection ranking scores. To demonstrate its effectiveness in refining semantic concept detection, the proposed re-ranking method is applied to the detection scores of TRECVID 2011 benchmark data set, and its performance is compared with other state-of-the-art re-ranking approaches.

Enabling Enriched TV Shopping Experience via Computational and Temporal Aware View-Centric Multimedia Abstraction

Smart TVs have realized the convergence of TV, Internet , and PC technologies, but still do not provide a seamless content interaction for TV-enabled shopping. To purchase interesting items displayed in a TV show, consumers must resort to a store or the Web, which is an inconvenient way of purchasing products. The fundamental challenge in realizing such a use case consists of understanding the multimedia content being streamed. Such a challenge can be realized by utilizing object detection to facilitate content understanding though it has to be executed as a computationally bound process so that consumers are provided with a responsive and exciting user interface. To this end, we propose a computational- and temporal-aware multimedia abstraction framework that facilitates the efficient execution of object detection tasks. Given computational and temporal rate constraints, the proposed framework selects the optimal video frames that best represent the video content and allows the execution of the object detection task as a computationally bound process. In this sense, the framework is computationally scalable as it can adapt to the given constraints and generate optimal abstraction results accordingly. Additionally, the framework utilizes “object views” as the basis for the frame selection process, which depict salient information and are represented as regions of interest (ROI). In general, an ROI can be a whole frame or a region that discards background information. Experimental results demonstrate the computational scalability of the proposed framework and the benefits of using the regions of interest as the basis of the abstraction process.

A SEMANTIC INDEX STRUCTURE FOR MULTIMEDIA RETRIEVAL

To be effective multimedia retrieval mechanisms, index methods must provide not only efficient access but also meaningful retrieval by addressing challenges in multimedia retrieval. This article presents the AH+-tree, a height-balanced, tree-based index structure that efficiently incorporates high-level affinity information to support Content-Based Image Retrieval (CBIR) through similarity queries. The incorporation of affinity information allows the AH+-tree to address the problems of semantic gap and user perception subjectivity inherent to multimedia retrieval. Based on the Affinity-Hybrid Tree (AH-Tree), the AH+-tree utilizes affinity information in a novel way to eliminate the I/O overhead of the AH-Tree while maintaining the same functionality and quality of results. We explain the structure of the AH+-tree and implement and analyze algorithms for tree construction and similarity queries (range and nearest neighbor). Experimental results demonstrate the superior I/O efficiency of the AH+-tree over that of the AH-Tree and the M-tree without a detrimental impact on real-time costs of the retrieval process.