Xiaozhao Zhao

Mining pure high-order word associations via information geometry for information retrieval

The classical bag-of-word models for information retrieval (IR) fail to capture contextual associations between words. In this article, we propose to investigate pure high-order dependence among a number of words forming an unseparable semantic entity, that is, the high-order dependence that cannot be reduced to the random coincidence of lower-order dependencies. We believe that identifying these pure high-order dependence patterns would lead to a better representation of documents and novel retrieval models. Specifically, two formal definitions of pure dependence—unconditional pure dependence (UPD) and conditional pure dependence (CPD)—are defined. The exact decision on UPD and CPD, however, is NP-hard in general. We hence derive and prove the sufficient criteria that entail UPD and CPD, within the well-principled information geometry (IG) framework, leading to a more feasible UPD/CPD identification procedure. We further develop novel methods for extracting word patterns with pure high-order dependence. Our methods are applied to and extensively evaluated on three typical IR tasks: text classification and text retrieval without and with query expansion.

A Quantum Query Expansion Approach for Session Search

Recently, Quantum Theory (QT) has been employed to advance the theory of Information Retrieval (IR). Various analogies between QT and IR have been established. Among them, a typical one is applying the idea of photon polarization in IR tasks, e.g., for document ranking and query expansion. In this paper, we aim to further extend this work by constructing a new superposed state of each document in the information need space, based on which we can incorporate the quantum interference idea in query expansion. We then apply the new quantum query expansion model to session search, which is a typical Web search task. Empirical evaluation on the large-scale Clueweb12 dataset has shown that the proposed model is effective in the session search tasks, demonstrating the potential of developing novel and effective IR models based on intuitions and formalisms of QT.

Investigating query-drift problem from a novel perspective of photon polarization

Query expansion, while generally effective in improving retrieval performance, may lead to the query-drift problem. Following the recent development of applying Quantum Mechanics (QM) to IR, we investigate the problem from a novel theoretical perspective inspired by photon polarization (a key QM experiment).

A Query Expansion Approach Using Entity Distribution Based on Markov Random Fields

The development of knowledge graph construction has prompted more and more commercial engines to improve the retrieval performance by using knowledge graphs as the basic semantic web. Knowledge graph is often used for knowledge inference and entity search, however, the potential ability of its entities and properties for better improving search performance in query expansion remains to be further excavated. In this paper, we propose a novel query expansion technique with knowledge graph (KG) based on the Markov random fields (MRF) model to enhance retrieval performance. This technique, called MRF-KG, models the joint distribution of original query terms, documents and two expanded variants, i.e. entities and properties. We conduct experiments on two TREC collections, WT10G and ClueWeb12B, annotated with Freebase entities. Experiment results demonstrate that MRF-KG outperforms traditional graph-based models.

A Sequential Latent Topic-based Readability Model for Domain-Specific Information Retrieval.

In domain-specific information retrieval (IR), an emerging problem is how to provide different users with documents that are both relevant and readable, especially for the lay users. In this paper, we propose a novel document readability model to enhance the domain-specific IR. Our model incorporates the coverage and sequential dependency of latent topics in a document. Accordingly, two topical readability indicators, namely Topic Scope and Topic Trace are developed. These indicators, combined with the classical Surface-level indicator, can be used to rerank the initial list of documents returned by a conventional search engine. In order to extract the structured latent topics without supervision, the hierarchical Latent Dirichlet Allocation (hLDA) is used. We have evaluated our model from the user-oriented and system-oriented perspectives, in the medical domain. The user-oriented evaluation shows a good correlation between the readability scores given by our model and human judgments. Furthermore, our model also gains significant improvement in the system-oriented evaluation in comparison with one of the state-of-the-art readability methods.

Extending the Extreme Physical Information to Universal Cognitive Models via a Confident Information First Principle

The principle of extreme physical information (EPI) can be used to derive many known laws and distributions in theoretical physics by extremizing the physical information loss K, i.e., the difference between the observed Fisher information I and the intrinsic information bound J of the physical phenomenon being measured. However, for complex cognitive systems of high dimensionality (e.g., human language processing and image recognition), the information bound J could be excessively larger than I (J ≫ I), due to insufficient observation, which would lead to serious over-fitting problems in the derivation of cognitive models. Moreover, there is a lack of an established exact invariance principle that gives rise to the bound information in universal cognitive systems. This limits the direct application of EPI. To narrow down the gap between I and J, in this paper, we propose a confident-information-first (CIF) principle to lower the information bound J by preserving confident parameters and ruling out unreliable or noisy parameters in the probability density function being measured. The confidence of each parameter can be assessed by its contribution to the expected Fisher information distance between the physical phenomenon and its observations. In addition, given a specific parametric representation, this contribution can often be directly assessed by the Fisher information, which establishes a connection with the inverse variance of any unbiased estimate for the parameter via the Cramer–Rao bound. We then consider the dimensionality reduction in the parameter spaces of binary multivariate distributions. We show that the single-layer Boltzmann machine without hidden units (SBM) can be derived using the CIF principle. An illustrative experiment is conducted to show how the CIF principle improves the density estimation performance.

On modeling rank-independent risk in estimating probability of relevance

Estimating the probability of relevance for a document is fundamental in information retrieval. From a theoretical point of view, risk exists in the estimation process, in the sense that the estimated probabilities may not be the actual ones precisely. The estimation risk is often considered to be dependent on the rank. For example, the probability ranking principle assumes that ranking documents in the order of decreasing probability of relevance can optimize the rank effectiveness. This implies that a precise estimation can yield an optimal rank. However, an optimal (or even ideal) rank does not always guarantee that the estimated probabilities are precise. This means that part of the estimation risk is rank-independent. It imposes practical risks in the applications, such as pseudo relevance feedback, where different estimated probabilities of relevance in the first-round retrieval will make a difference even when two ranks are identical. In this paper, we will explore the effect and the modeling of such rank-independent risk. A risk management method is proposed to adaptively adjust the rank-independent risk. Experimental results on several TREC collections demonstrate the effectiveness of the proposed models for both pseudo-relevance feedback and relevance feedback.

A Confident Information First Principle for Parameter Reduction and Model Selection of Boltzmann Machines

Typical dimensionality reduction (DR) methods are data-oriented, focusing on directly reducing the number of random variables (or features) while retaining the maximal variations in the high-dimensional data. Targeting unsupervised situations, this paper aims to address the problem from a novel perspective and considers model-oriented DR in parameter spaces of binary multivariate distributions. Specifically, we propose a general parameter reduction criterion, called confident-information-first (CIF) principle, to maximally preserve confident parameters and rule out less confident ones. Formally, the confidence of each parameter can be assessed by its contribution to the expected Fisher information distance within a geometric manifold over the neighborhood of the underlying real distribution. Then, we demonstrate two implementations of CIF in different scenarios. First, when there are no observed samples, we revisit the Boltzmann machines (BMs) from a model selection perspective and theoretically show that both the fully visible BM and the BM with hidden units can be derived from the general binary multivariate distribution using the CIF principle. This finding would help us uncover and formalize the essential parts of the target density that BM aims to capture and the nonessential parts that BM should discard. Second, when there exist observed samples, we apply CIF to the model selection for BM, which is in turn made adaptive to the observed samples. The sample-specific CIF is a heuristic method to decide the priority order of parameters, which can improve the search efficiency without degrading the quality of model selection results as shown in a series of density estimation experiments.

Encoding physiological signals as images for affective state recognition using convolutional neural networks

Affective state recognition based on multiple modalities of physiological signals has been a hot research topic. Traditional methods require designing hand-crafted features based on domain knowledge, which is time-consuming and has not achieved a satisfactory performance. On the other hand, conducting classification on raw signals directly can also cause some problems, such as the interference of noise and the curse of dimensionality. To address these problems, we propose a novel approach that encodes different modalities of data as images and use convolutional neural networks (CNN) to perform the affective state recognition task. We validate our aproach on the DECAF dataset in comparison with two state-of-the-art methods, i.e., the Support Vector Machines (SVM) and Random Forest (RF). Experimental results show that our aproach outperforms the baselines by 5% to 9%.Affective state recognition based on multiple modalities of physiological signals has been a hot research topic. Traditional methods require designing hand-crafted features based on domain knowledge, which is time-consuming and has not achieved a satisfactory performance. On the other hand, conducting classification on raw signals directly can also cause some problems, such as the interference of noise and the curse of dimensionality. To address these problems, we propose a novel approach that encodes different modalities of data as images and use convolutional neural networks (CNN) to perform the affective state recognition task. We validate our aproach on the DECAF dataset in comparison with two state-of-the-art methods, i.e., the Support Vector Machines (SVM) and Random Forest (RF). Experimental results show that our aproach outperforms the baselines by 5% to 9%.

Distributed Computation of Pure High-Order Word Dependence

Many data mining methods have been proposed to obtain useful word associations in text documents. However, it is of great challenge to efficiently discover “pure” high-order (n > 3) word association patterns, especially in the rapidly expanding data collection. Here, by “pure,” we mean that those words form an inseparable semantic entity, i.e., the high-order dependence that cannot be reduced to the random coincidence of low-order dependence. Our aim is to find pure high-order word associations in large data sets. This paper proposes a distributed pure dependence mining (DPDM) algorithm based on information geometry, which can efficiently mine the pure dependence between words. We also construct a distributed pure dependence mining framework (DPDMF) to mine pure high-order word associations from data sets. Extensive experiments show that DPDM algorithm can significantly improve the efficiency when mining the pure high-order patterns from large data sets. Finally, we apply the extracted high-order word association patterns in the text classification tasks, which will also achieve significant improvement.