Weiming Lu

Cross-Modal Learning to Rank via Latent Joint Representation

Cross-modal ranking is a research topic that is imperative to many applications involving multimodal data. Discovering a joint representation for multimodal data and learning a ranking function are essential in order to boost the cross-media retrieval (i.e., image-query-text or text-query-image). In this paper, we propose an approach to discover the latent joint representation of pairs of multimodal data (e.g., pairs of an image query and a text document) via a conditional random field and structural learning in a listwise ranking manner. We call this approach cross-modal learning to rank via latent joint representation (CML2R). In CML2R, the correlations between multimodal data are captured in terms of their sharing hidden variables (e.g., topics), and a hidden-topic-driven discriminative ranking function is learned in a listwise ranking manner. The experiments show that the proposed approach achieves a good performance in cross-media retrieval and meanwhile has the capability to learn the discriminative representation of multimodal data.

Topic aspect-oriented summarization via group selection

Abstract The summarization is desirable to efficiently apprehend the gist of the huge amount of data and becomes a significant challenge in many applications such as news article summarization and social media mining. Considering the summaries from multi-documents of one topic can describe various aspects of one given topic, this paper attempts to exploit appropriate priors to generate topic aspect-oriented summarization (abbreviated as TAOS). The underlying intuition of the proposed TAOS is that different topics can prefer different aspects and the different aspects can be represented by different preference of features(e.g., technical topic may prefer proper noun than sports topic). In order to materialize the intuition of TAOS, we first extract several groups of features according to topic factors, and then a group norm penalty (i.e., l 1 / l 2 norm) and latent variables are utilized to select overlapping groups of features. We compare our proposed approach with some state-of-the-art methods on DUC2003, DUC2004 datasets for text summarization and NUS-Wide dataset for image summarization. The results show our method can generate meaningful summarization in terms of ROUGE and Jensen–Shannon Divergence metrics.

Deep Compositional Cross-modal Learning to Rank via Local-Global Alignment

Cross-modal retrieval is a very hot research topic that is imperative to many applications involving multi-modal data. Discovering an appropriate representation for multi-modal data and learning a ranking function are essential to boost the cross-media retrieval. Motivated by the assumption that a compositional cross-modal semantic representation (pairs of images and text) is more attractive for cross-modal ranking, this paper exploits the existing image-text databases to optimize a ranking function for cross-modal retrieval, called deep compositional cross-modal learning to rank (C2MLR). In this paper, C2MLR considers learning a multi-modal embedding from the perspective of optimizing a pairwise ranking problem while enhancing both local alignment and global alignment. In particular, the local alignment (i.e., the alignment of visual objects and textual words) and the global alignment (i.e., the image-level and sentence-level alignment) are collaboratively utilized to learn the multi-modal embedding common space in a max-margin learning to rank manner. The experiments demonstrate the superiority of our proposed C2MLR due to its nature of multi-modal compositional embedding.

Kernelized sparse hashing for scalable image retrieval

Recently, hashing has been widely applied to large scale image retrieval applications due to its appealing query speed and low storage cost. The key idea of hashing is to learn a hash function that maps high dimensional data into compact binary codes while preserving the similarity structure in the original feature space. In this paper, we propose a new method called the Kernelized Sparse Hashing, which generates sparse hash codes with ?1 and non-negative regularizations. Compared to traditional hashing methods, our method only activates a small number of relevant bits on the hash code and hence provides a more compact and interpretable representation of data. Moreover, the kernel trick is introduced to capture the nonlinear similarity of features, and the local geometrical structure of data is explicitly considered in our method to improve the retrieval accuracy. Extensive experiments on three large-scale image datasets demonstrate the superior performance of our proposed method over the examined state-of-the-art techniques.

Web-based chinese calligraphy retrieval and learning system

Chinese calligraphy is a valuable civilization legacy and there are some web sites trying to help people enjoy and learn calligraphy. However, besides metadata-base searching, it is very difficult to find advanced services such as content-based retrieval or vivid writing process simulating for Chinese calligraphy. In this paper, a novel Chinese calligraphy retrieval and learning system is proposed: First, the scanned calligraphy pages were segmented into individual calligraphy characters using minimum-bounding box. Second, individual characters feature information was extracted and kept. Then, corresponding database was built to serve as a map between the feature data and the original data of individual character image. Finally, a retrieval engine was constructed and dynamic writing process was simulated to help learners get the calligraphy character they are interested in and watch how it was written.

D-Ocean: an unstructured data management system for data ocean environment

Together with the big datamovement,many organizations collect their own big data and build distinctive applications. In order to provide smart services upon big data, massive variable data should be well linked and organized to form Data Ocean, which specially emphasizes the deep exploration of the relationships among unstructured data to support smart services. Currently, almost all of these applications have to deal with unstructured data by integrating various analysis and search techniques upon massive storage and processing infrastructure at the application level, which greatly increase the difficulty and cost of application development.This paper presents D-Ocean, an unstructured data management system for data ocean environment. D-Ocean has an open and scalable architecture, which consists of a core platform, pluggable components and auxiliary tools. It exploits a unified storage framework to store data in different kinds of data stores, integrates batch and incremental processing mechanisms to process unstructured data, and provides a combined search engine to conduct compound queries. Furthermore, a so-called RAISE process modeling is proposed to support the whole process of Repository, Analysis, Index, Search and Environment modeling, which can greatly simplify application development. The experiments and use cases in production demonstrate the efficiency and usability of D-Ocean.

Structured Visual Feature Learning for Classification via Supervised Probabilistic Tensor Factorization

In this paper, structured visual feature learning aims at exploiting the intrinsic structural properties of mutually correlated multimedia collections (e.g., video frames or facial images) to learn a more effective feature representation for multimedia data classification. We pose structured visual feature learning as a problem of supervised tensor factorization (STF), which is capable of effectively learning multi-view visual features from structural tensorial multimedia data. In mathematics, STF is formulated as a joint optimization framework of probabilistic inference and ε-insensitive support vector regression. As a result, the feature representation obtained by STF not only preserves the intrinsic multi-view structural information on tensorial multimedia data, but also includes the discriminative information derived from the max-margin learning process. Using the learned discriminative visual features, we conduct a set of multimedia classification experiments on several challenging datasets, including images and videos, which demonstrate the effectiveness of our method.

Digital Library Engine: Adapting Digital Library for Cloud Computing

With the rapid growth of digital libraries, more data and smart services are involved. People come to recognize the importance of digital libraries and the convenience they might bring to the society. However, the cost of owning a digital library is quite high, and many institutions do not have the ability to run and maintain a digital library by themselves, especially for massive data and complex services which require lots of storage and computing resources. In this paper, we proposed the Digital Library Engine, which aims to provide a new Platform as a Service for fast developing and deploying digital libraries in cloud. To the best of our knowledge, it is the first work to create a PaaS system for digital libraries. With the help of Digital Library Engine, institutions only need to develop some service bundles, which can be deployed in the engine, and then their own digital libraries could be running well with features of scalability, reliability, security, extensibility, availability and manageability. The practice in CADAL and the experiments demonstrate the feasibility and efficiency of our engine.

KeyphraseDS: Automatic generation of survey by exploiting keyphrase information

Abstract In this paper, we present a novel document summarization mechanism called KeyphraseDS that can organize the scientific articles into multi-aspect and informative scientific survey by exploiting keyphrases. Keyphrases describe texts salience and central focus, which can serve as the component of aspects under specific topic. KeyphraseDS consists of three steps: keyphrase graph construction, semantic aspect generation and content selection. Keyprhases are firstly extracted through CRF-based model exploiting various features, such as syntactic features, correlation features, etc. Spectral clustering is then performed on keyphrase graph to generate different aspects, where the semantic relatedness between keyphrases is computed through knowledge-based similarity and topic-based similarity. The proposed semantic relatedness can not only utilize the statistical text signals efficiently but also overcome the data sparsity problem. Significant sentences are then selected with respect to the generated aspects through integer linear programming (ILP), which takes semantic relevance, semantic diversity, and keyphrase salience into consideration. Extensive experiments, measured by automatic evaluation and human evaluation, demonstrate the effectiveness of our mechanism for generating scientific survey.

Hybrid storage architecture and efficient MapReduce processing for unstructured data

Abstract As we are now entering the era of data deluge, how to efficiently manage these massive data is becoming a great challenge, especially for the exponentially growing unstructured data, which is far more than structured and semi-structured data. However, unstructured data is more complex for its variety. That is to say, different types of unstructured data have different file size, type and usage, which need different storage and processing for high efficiency. In this paper, we propose a hybrid storage architecture to store the pervasive unstructured data. This hybrid architecture integrates various kinds of data stores within a unified framework, where each type of unstructured data can find its suitable placement policy and it is transparent to users. In addition, we present several partitioning strategies based on the unified framework, which are beneficial to the MapReduce-based batch processing for these unstructured data. The experiments demonstrate that it is possible to build an efficient and smart system through the hybrid architecture and the partitioning strategies.