Azin Ashkan

Novelty and diversity in information retrieval evaluation

Evaluation measures act as objective functions to be optimized by information retrieval systems. Such objective functions must accurately reflect user requirements, particularly when tuning IR systems and learning ranking functions. Ambiguity in queries and redundancy in retrieved documents are poorly reflected by current evaluation measures. In this paper, we present a framework for evaluation that systematically rewards novelty and diversity. We develop this framework into a specific evaluation measure, based on cumulative gain. We demonstrate the feasibility of our approach using a test collection based on the TREC question answering track.

A comparative analysis of cascade measures for novelty and diversity

Traditional editorial effectiveness measures, such as nDCG, remain standard for Web search evaluation. Unfortunately, these traditional measures can inappropriately reward redundant information and can fail to reflect the broad range of user needs that can underlie a Web query. To address these deficiencies, several researchers have recently proposed effectiveness measures for novelty and diversity. Many of these measures are based on simple cascade models of user behavior, which operate by considering the relationship between successive elements of a result list. The properties of these measures are still poorly understood, and it is not clear from prior research that they work as intended. In this paper we examine the properties and performance of cascade measures with the goal of validating them as tools for measuring effectiveness. We explore their commonalities and differences, placing them in a unified framework; we discuss their theoretical difficulties and limitations, and compare the measures experimentally, contrasting them against traditional measures and against other approaches to measuring novelty. Data collected by the TREC 2009 Web Track is used as the basis for our experimental comparison. Our results indicate that these measures reward systems that achieve an balance between novelty and overall precision in their result lists, as intended. Nonetheless, other measures provide insights not captured by the cascade measures, and we suggest that future evaluation efforts continue to report a variety of measures.

Evaluating architectural stability using a metric-based approach

Architectural stability refers to the extent software architecture is flexible to endure evolutionary changes while leaving the architecture intact. Approaches to evaluate software architectures for stability can be retrospective or predictive. Retrospective evaluation looks at successive releases of a software system to analyze how smoothly the evolution has taken place. Predictive evaluation examines a set of likely changes and shows the architecture can endure these changes. This paper proposes a metric-based approach to evaluate architectural stability of a software system by combining these two traditional analysis techniques. Such an approach performs on the fact bases extracted from the source code by reverse engineering techniques. We also present experimental results by applying the proposed approach to analyze the architectural stability across different versions of two spreadsheet systems

Estimating Ad Clickthrough Rate through Query Intent Analysis

Clickthrough rate, bid, and cost-per-click are known to be among the factors that impact the rank of an ad shown on a search result page. Search engines can benefit from estimating ad clickthrough in order to determine the quality of ads and maximize their revenue. In this paper, a methodology is developed to estimate ad clickthrough rate by exploring user queries and clickthrough logs. As we demonstrate, the average ad clickthrough rate depends to a substantial extent on the rank position of ads and on the total number of ads displayed on the page. This observation is utilized by a baseline model to calculate the expected clickthrough rate for various ads. We further study the impact of query intent on the clickthrough rate, where query intent is predicted using a combination of query features and the content of search engine result pages. The baseline model and the query intent model are compared for the purpose of calculating the expected ad clickthrough rate. Our findings suggest that such factors as the rank of an ad, the number of ads displayed on the result page, and query intent are effective in estimating ad clickthrough rate.

Characterizing commercial intent

Understanding the intent underlying users queries may help personalize search results and therefore improve user satisfaction. We develop a methodology for using the content of search engine result pages (SERPs) along with the information obtained from query strings to study characteristics of query intent, with a particular focus on sponsored search. This work represents an initial step towards the development and evaluation of an ontology for commercial search, considering queries that reference specific products, brands and retailers. The characteristics of query categories are studied with respect to aggregated users clickthrough behavior on advertising links. We present a model for clickthrough behavior that considers the influence of such factors as the location of ads and the rank of ads, along with query category. We evaluate our work using a large corpus of clickthrough data obtained from a major commercial search engine. Our findings suggest that query based features, along with the content of SERPs, are effective in detecting query intent. The clickthrough behavior is found to be consistent with the classification for the general categories of query intent, while for product, brand and retailer categories, all is true to a lesser extent.

Term-based commercial intent analysis

In this work, we investigate the contribution of query terms and their corresponding ad click rates on commercial intent of queries. A probabilistic model is proposed following the hypothesis that a query is likely to receive ad clicks based on contributions from its individual terms.

Cache content-selection policies for streaming video services

The majority of Internet traffic is now dominated by streamed video content. As video quality continues to increase, the strain that streaming traffic places on the network infrastructure also increases. Caching content closer to users, e.g., using Content Distribution Networks, is a common solution to reduce the load on the network. A simple approach to selecting what to put in regional caches is to put the videos that are most popular globally across the entire customer base. However, this approach ignores distinct regional taste. In this paper we explore the question of how a video content provider could go about determining whether or not they should use a cache filling policy based solely upon global popularity or take into account regional tastes as well. We propose a model that captures the overlap between inter-regional and intra-regional preferences. We focus on movie content and derive a synthetic model that captures “taste” using matrix factorization, similarly to the method used in recommender systems. Our model enables us to widely explore the parameter space, and derive a set of metrics providers can use to determine whether populating caches according to regional of global tastes provides better cache performance.

Impact of query intent and search context on clickthrough behavior in sponsored search

Implicit feedback techniques may be used for query intent detection, taking advantage of user behavior to understand their interests and preferences. In sponsored search, a primary concern is the user’s interest in purchasing or utilizing a commercial service, or what is called online commercial intent. In this paper, we develop a methodology for employing the content of search engine result pages (SERPs), along with the information obtained from query strings, to study characteristics of query intent, with a particular focus on sponsored search. Our work represents a step toward the development and evaluation of an ontology for commercial search, considering queries that reference specific products, brands, and retailers. Characteristics of query categories are studied with respect to aggregated user clickthrough behavior on advertising links. We present a model for clickthrough behavior that considers the influence of such factors as the location of ads and the rank of ads, along with query category. We evaluate our work using a large corpus of clickthrough data obtained from a major commercial search engine. In addition, the impact of query intent is studied on clickthrough rate, where a baseline model and the query intent model are compared for the purpose of calculating an expected ad clickthrough rate. Our findings suggest that query-based features, along with the content of SERPs, are effective in detecting query intent. Factors such as query category, the rank of an ad, and the total number of ads displayed on a result page relate to the context of the ad, rather than its content. We demonstrate that these context-related factors can have a major influence on expected clickthrough rate, suggesting that these factors should be taken into consideration when the performance of an ad is evaluated.

In the Mood to Click? Towards Inferring Receptiveness to Search Advertising

We present a method for modeling, and automaticallyinferring, the current interest of a user in searchadvertising. Our task is complementary to that of predictingad relevance or commercial intent of a query in the aggregate, since the user intent may vary significantly for the same query. To achieve this goal, we develop a fine-grained user interaction model for inferring searcher receptiveness to advertising. We show that modeling the search context and behavior can significantly improve the accuracy of ad clickthrough prediction for the current user, compared to the existing state-of-the-artclassification methods that do not model this additional session level contextual and interaction information. In particular, our experiments over thousands of search sessions from hundreds of real users demonstrate that our model is more effective at predicting ad clickthrough within the same search session. Our work has other potential applications, such as improving searchinterface design (e.g., varying the number or type of ads) based on user interest, and behavioral targeting (e.g., identifying users interested in immediate purchase).

Just One More: Modeling Binge Watching Behavior

Easy accessibility can often lead to over-consumption, as seen in food and alcohol habits. On video on-demand (VOD) services, this has recently been referred to as binge watching, where potentially entire seasons of TV shows are consumed in a single viewing session. While a user viewership model may reveal this binging behavior, creating an accurate model has several challenges, including censored data, deviations in the population, and the need to consider external influences on consumption habits. In this paper, we introduce a novel statistical mixture model that incorporates these factors and presents a first of its kind characterization of viewer consumption behavior using a real-world dataset that includes playback data from a VOD service. From our modeling, we tackle various predictive tasks to infer the consumption decisions of a user in a viewing session, including estimating the number of episodes they watch and classifying if they continue watching another episode. Using these insights, we then identify binge watching sessions based on deviation from normal viewing behavior. We observe different types of binging behavior, that binge watchers often view certain content out-of-order, and that binge watching is not a consistent behavior among our users. These insights and our findings have application in VOD revenue generation, consumer health applications, and customer retention analysis.