Diane Kelly

Understanding Ephemeral State of Relevance

Despite its dynamic nature, relevance is often measured in a context-independent manner in information retrieval practice. We look into this discrepancy. We propose a contextual relevance/usefulness measurement called ephemeral state of relevance (ESR), which is defined as the amount of useful information a user acquired from a clicked result as assessed just after examining the result during an interactive search session. We collect ESR and context-independent usefulness judgments through a laboratory user study and compare the two. We examine factors related to both judgments and examine their differences. Our study demonstrates a few advantages of ESR: it captures users real-time state of mind and perceptions; it measures how much useful information the user is able to acquire from a result rather than how much there is in the result; it better reflects users needs and criteria of useful results during a session, highlighting novelty as a salient factor. However, we also find that users may not be able to correctly assess the credibility of information during a session, which may reduce the reliability of the collected ESR judgments. We evaluate ESR, context-independent usefulness judgments, and TREC-style topical relevance judgments by correlating with user experience in a session. The results demonstrate that switching the judgment criterion from topical relevance to usefulness is fruitful, but moving from context-independent judgments to contextual ones has only limited advantages with respect to its cost and complexity. Our study enriches current understanding on the dynamics of relevance in a search session and identifies both opportunities and challenges for collecting contextual relevance judgments.

Engaged or Frustrated?: Disambiguating Emotional State in Search

One of the primary ways researchers have characterized engagement during information search is by increases in search behaviors, such as queries and clicks. However, studies have shown that frustration is also characterized by increases in these same behaviors. This research examines the differences in the search behaviors and physiologies of people who are engaged or frustrated during search. A 2x2 within-subject laboratory experiment was conducted with 40 participants. Engagement was induced by manipulating task interest and frustration was induced by manipulating the quality of the search results. Participants interactions and physiological responses were recorded, and after they searched, they evaluated their levels of engagement, frustration and stress. Participants reported significantly greater levels of engagement when completing tasks that interested them and significantly less engagement during searches with poor results quality. For all search behaviors measured, only two significant differences were found according to task interest: participants had more scrolls and longer query intervals when searching for interesting tasks, suggesting greater interaction with content. Significant differences were found for nine behaviors according to results quality, including queries issued, number of SERPs displayed and number of SERP clicks, suggesting these are potentially better indicators of frustration rather than engagement. When presented with poor quality results, participants had significantly higher heart rates than when presented with normal quality results. Finally, participants had lower heart rates and greater skin conductance responses when conducting interesting tasks than when conducting uninteresting tasks. This research provides insight into the differences in search behaviors and physiologies of participants when they are engaged versus frustrated and presents techniques that can be used by those wishing to induce engagement and frustration during laboratory IIR studies.

Second Workshop on Supporting Complex Search Tasks

There is broad consensus in the field of IR that search is complex in many use cases and applications, both on the Web and in domain specific collections, and both professionally and in our daily life. Yet our understanding of complex search tasks, in comparison to simple look up tasks, is fragmented at best. The workshop addresses many open research questions: What are the obvious use cases and applications of complex search? What are essential features of work tasks and search tasks to take into account? And how do these evolve over time--With a multitude of information, varying from introductory to specialized, and from authoritative to speculative or opinionated, when to show what sources of information? How does the information seeking process evolve and what are relevant differences between different stages? With complex task and search process management, blending searching, browsing, and recommendations, and supporting exploratory search to sensemaking and analytics, UI and UX design pose an overconstrained challenge. How do we evaluate and compare approaches? Which measures should be taken into account? Supporting complex search tasks requires new collaborations across the fields of CHI and IR, and the proposed workshop will bring together a diverse group of researchers to work together on one of the greatest challenges of our field.

SIGIR Initiative to Implement ACM Artifact Review and Badging

Recently, the ACM created a policy on Artifact Review and Badging, which presents a framework to help SIGs recognize repeatability, replicability and reproducibility in published research. While the ACM policy established a vocabulary and definitions, it did not prescribe procedures for implementation. Rather, the ACM has left this to each SIG to define given the variety of research traditions and approaches that exist with the ACM community. SIGs are not required to implement badging, but given the growing interest in this topic in the SIGIR community, a task force has been assembled to determine how badging might be implemented. This report describes the ACM policy on Artifact Review and Badging, introduces the task force, and presents survey data describing the SIGIR communitys opinions about this initiative.

Examining the impact of domain and cognitive complexity on query formulation and reformulation

The purpose of this analysis was to evaluate an existing set of search tasks in terms of their effectiveness as part of a shared infrastructure for conducting interactive IR research. Twenty search tasks that varied in their cognitive complexity and domain were assigned to 47 study participants; the 3,101 moves used to complete those tasks were then analyzed in terms of frequency of each type of move and the sequential patterns they formed. The cognitive complexity of the tasks influenced the number of moves used to complete the tasks, with the most complex (i.e., Create) tasks requiring more moves than tasks at other levels of complexity. Across the four domains, the Commerce tasks elicited more search moves per search. When sequences of moves were analyzed, seven patterns were identified; some of these patterns were associated with particular task characteristics. The findings suggest that search tasks can be designed to elicit particular types of search behaviors and, thus, allow researchers to focus attention on particular aspects of IR interactions.

Strategies for Finding and Evaluating Information about Personal Finance Topics: The Role of Government Information

In this paper, we present work-in-progress results from the stimulated recall portion of a U.S.-based lab study that investigated the influence of financial knowledge and cognitive abilities on the search performance, relevance assessments, and mental workload of adults searching the Internet for personal finance topics. Participants were asked to retrospectively think aloud while viewing screen recordings of one of their search tasks. Qualitative, inductive coding was applied to transcribed interviews. An early theme about government websites and information emerged in the data analysis and that is the topic of this paper. For all three tasks, participants prioritized and valued information from U.S. government websites over that of commercial websites, which seems to contradict recent national surveys indicating low levels of trust in government information sources and the government in general. Our findings suggest that for certain topics, especially those associated with high levels of uncertainty, people might resort to more basic search and evaluation behaviors.

SIGIR Community Survey on Preprint Services

Recently, there has been a growing number of questions about the relationship between preprints and double-blind peer-review at the SIGIR conference. These questions come from authors who wish to post papers which are under review to preprint servers such as arXiv.org, as well as reviewers who become aware of author identities through such postings, and are subsequently, unsure of how to proceed in a double-blind review process. A review of current conference publication guidelines was conducted, along with a survey of SIGIR community members to gain insight about their behaviors, practices and opinions. The paper presents results of this survey, as well as recommendations about submission and review policies for the SIGIR conference.