Hosein Azarbonyad

Luhn Revisited: Significant Words Language Models

Users tend to articulate their complex information needs in only a few keywords, making underspecified statements of request the main bottleneck for retrieval effectiveness. Taking advantage of feedback information is one of the best ways to enrich the query representation, but can also lead to loss of query focus and harm performance in particular when the initial query retrieves only little relevant information when overfitting to accidental features of the particular observed feedback documents. Inspired by the early work of Luhn [23], we propose significant words language models of feedback documents that capture all, and only, the significant shared terms from feedback documents. We adjust the weights of common terms that are already well explained by the document collection as well as the weight of rare terms that are only explained by specific feedback documents, which eventually results in having only the significant terms left in the feedback model. Our main contributions are the following. First, we present significant words language models as the effective models capturing the essential terms and their probabilities. Second, we apply the resulting models to the relevance feedback task, and see a better performance over the state-of-the-art methods. Third, we see that the estimation method is remarkably robust making the models in- sensitive to noisy non-relevant terms in feedback documents. Our general observation is that the significant words language models more accurately capture relevance by excluding general terms and feedback document specific terms.

Exploiting Multiple Translation Resources for English-Persian Cross Language Information Retrieval

One of the most important issues in Cross Language Information Retrieval CLIR which affects the performance of CLIR systems is how to exploit available translation resources. This issue can be more challenging when dealing with a language that lacks appropriate translation resources. Another factor that affects the performance of a CLIR system is the degree of ambiguity of query words. In this paper, we propose to combine different translation resources for CLIR. We also propose two different methods that exploit phrases in the query translation process to solve the problem of ambiguousness of query words. Our evaluation results on English-Persian CLIR show the superiority of phrase based and combinational translation CLIR methods over other CLIR methods.

Using learning to rank approach for parallel corpora based cross language information retrieval

Learning to Rank (LTR) refers to machine learning techniques for training a model in a ranking task. LTR has been shown to be useful in many applications in information retrieval (IR). Cross language information retrieval (CLIR) is one of the major IR tasks that can potentially benefit from LTR to improve the ranking accuracy. CLIR deals with the problem of expressing query in one language and retrieving the related documents in another language. One of the most important issues in CLIR is how to apply monolingual IR methods in cross lingual environments. In this paper, we propose a new method to exploit LTR for CLIR in which documents are represented as feature vectors. This method provides a mapping based on IR heuristics to employ monolingual IR features in parallel corpus based CLIR. These mapped features are considered as training data for LTR. We show that using LTR trained on mapped features can improve CLIR performance. A comprehensive evaluation on the English-Persian CLIR suggests that our method has significant improvements over parallel corpora based methods and dictionary based methods.

Two-Way Parsimonious Classification Models for Evolving Hierarchies

There is an increasing volume of semantically annotated data available, in particular due to the emerging use of knowledge bases to annotate or classify dynamic data on the web. This is challenging as these knowledge bases have a dynamic hierarchical or graph structure demanding robustness against changes in the data structure over time. In general, this requires us to develop appropriate models for the hierarchical classes that capture all, and only, the essential solid features of the classes which remain valid even as the structure changes. We propose hierarchical significant words language models of textual objects in the intermediate levels of hierarchies as robust models for hierarchical classification by taking the hierarchical relations into consideration. We conduct extensive experiments on richly annotated parliamentary proceedings linking every speech to the respective speaker, their political party, and their role in the parliament. Our main findings are the following. First, we define hierarchical significant words language models as an iterative estimation process across the hierarchy, resulting in tiny models capturing only well grounded text features at each level. Second, we apply the resulting models to party membership and party position classification across time periods, where the structure of the parliament changes, and see the models dramatically better transfer across time periods, relative to the baselines.

Hierarchical Re-estimation of Topic Models for Measuring Topical Diversity

A high degree of topical diversity is often considered to be an important characteristic of interesting text documents. A recent proposal for measuring topical diversity identifies three elements for assessing diversity: words, topics, and documents as collections of words. Topic models play a central role in this approach. Using standard topic models for measuring diversity of documents is suboptimal due to generality and impurity. General topics only include common information from a background corpus and are assigned to most of the documents in the collection. Impure topics contain words that are not related to the topic; impurity lowers the interpretability of topic models and impure topics are likely to get assigned to documents erroneously. We propose a hierarchical re-estimation approach for topic models to combat generality and impurity; the proposed approach operates at three levels: words, topics, and documents. Our re-estimation approach for measuring documents’ topical diversity outperforms the state of the art on PubMed dataset which is commonly used for diversity experiments.

Words are Malleable: Computing Semantic Shifts in Political and Media Discourse

Recently, researchers started to pay attention to the detection of temporal shifts in the meaning of words. However, most (if not all) of these approaches restricted their efforts to uncovering change over time, thus neglecting other valuable dimensions such as social or political variability. We propose an approach for detecting semantic shifts between different viewpoints---broadly defined as a set of texts that share a specific metadata feature, which can be a time-period, but also a social entity such as a political party. For each viewpoint, we learn a semantic space in which each word is represented as a low dimensional neural embedded vector. The challenge is to compare the meaning of a word in one space to its meaning in another space and measure the size of the semantic shifts. We compare the effectiveness of a measure based on optimal transformations between the two spaces with a measure based on the similarity of the neighbors of the word in the respective spaces. Our experiments demonstrate that the combination of these two performs best. We show that the semantic shifts not only occur over time but also along different viewpoints in a short period of time. For evaluation, we demonstrate how this approach captures meaningful semantic shifts and can help improve other tasks such as the contrastive viewpoint summarization and ideology detection (measured as classification accuracy) in political texts. We also show that the two laws of semantic change which were empirically shown to hold for temporal shifts also hold for shifts across viewpoints. These laws state that frequent words are less likely to shift meaning while words with many senses are more likely to do so.

Telling How to Narrow it Down: Browsing Path Recommendation for Exploratory Search

Supporting exploratory search tasks with the help of structured data is an effective way to go beyond keyword search, as it provides an overview of the data, enables users to zoom in on their intent, and provides assistance during their navigation trails. However, finding a good starting point for a search episode in the given structure can still pose a considerable challenge, as users tend to be unfamiliar with exact, complex hierarchical structure. Thus, providing lookahead clues can be of great help and allow users to make better decisions on their search trajectory. In this paper, we investigate the behaviour of users when a recommendation engine is employed along with the browsing tool in an exploratory search system. We make use of an exploratory search system that facilitates browsing by mapping the data on a hierarchical structure. We designed and developed a path recommendation engine as a feature for this system, which given a text query, ranks different browsing paths in the hierarchy based on their likelihood of covering relevant documents. We conduct a user study comparing the baseline system with the featured system. Our main findings are as follows: We observe that, using the baseline system the users tend to explore the data in a breadth-first-like approach by visiting different data points at the same level of abstraction to choose one of them to expand and go deeper. Conversely, with browsing path recommendation (BPR) as a feature, the users tend to drive their search in a more depth-first-like approach by quickly going deep into the data hierarchy. While the users still incline to explore different parts of the search space by using BPR, they are able to restrain or augment their search focus more quickly and access smaller but more promising regions of the data. Therefore, they can complete their tasks with less time and effort

Sources of evidence for automatic indexing of political texts

Political texts on the Web, documenting laws and policies and the process leading to them, are of key importance to government, industry, and every individual citizen. Yet access to such texts is difficult due to the ever increasing volume and complexity of the content, prompting the need for indexing or annotating them with a common controlled vocabulary or ontology. In this paper, we investigate the effectiveness of different sources of evidence—such as the labeled training data, textual glosses of descriptor terms, and the thesaurus structure—for automatically indexing political texts. Our main findings are the following. First, using a learning to rank (LTR) approach integrating all features, we observe significantly better performance than previous systems. Second, the analysis of feature weights reveals the relative importance of various sources of evidence, also giving insight in the underlying classification problem. Third, a lean-and-mean system using only four features (text, title, descriptor glosses, descriptor term popularity) is able to perform at 97% of the large LTR model.

On Search Powered Navigation

main components of exploratory search. Search lets users dig in deep by controlling their actions to focus on and find just the information they need, whereas navigation helps them to get an overview to decide which content is most important. In this paper, we introduce the concept of search powered navigation and investigate the effect of empowering navigation with search functionality on information seeking behavior of users and their experience by conducting a user study on exploratory search tasks, differentiated by different types of information needs. Our main findings are as follows: First, we observe radically different search tactics. Using search, users are able to control and augment their search focus, hence they explore the data in a depth-first, bottom-up manner. Conversely, using pure navigation they tend to check different options to be able to decide on their path into the data, which corresponds to a breadth-first, top-down exploration. Second, we observe a general natural tendency to combine aspects of search and navigation, however, our experiments show that the search functionality is essential to solve exploratory search tasks that require finding documents related to a narrowdomain. Third, we observe a natural need for search powered navigation: users using a system without search functionality find creative ways to mimic searching using navigation.

Power Analysis for Interleaving Experiments by Means of Offline Evaluation

Evaluation in information retrieval takes one of two forms: collection-based offline evaluation, and in-situ online evaluation. Collections constructed by the former methodology are reusable, and hence able to test the effectiveness of any experimental algorithm, while the latter requires a different experiment for every new algorithm. Due to this a funnel approach is often being used, with experimental algorithms being compared to the baseline in an online experiment only if they outperform the baseline in an offline experiment. One of the key questions in the design of online and offline experiments concerns the number of measurements required to detect a statistically significant difference between two algorithms. Power analysis can provide an answer to this question, however, it requires an a-priori knowledge of the difference in effectiveness to be detected, and the variance in the measurements. The variance is typically estimated using historical data, but setting a detectable difference prior to the experiment can lead to suboptimal, upper-bound results. In this work we make use of the funnel approach in evaluation and test whether the difference in the effectiveness of two algorithms measured by the offline experiment can inform the required number of impression of an online interleaving experiment. Our analysis on simulated data shows that the number of impressions required are correlated with the difference in the offline experiment, but at the same time widely vary for any given difference.