Michael J. Cole

Search behaviors in different task types

Personalization of information retrieval tailors search towards individual users to meet their particular information needs by taking into account information about users and their contexts, often through implicit sources of evidence such as user behaviors. Task types have been shown to influence search behaviors including usefulness judgments. This paper reports on an investigation of user behaviors associated with different task types. Twenty-two undergraduate journalism students participated in a controlled lab experiment, each searching on four tasks which varied on four dimensions: complexity, task product, task goal and task level. Results indicate regular differences associated with different task characteristics in several search behaviors, including task completion time, decision time (the time taken to decide whether a document is useful or not), and eye fixations, etc. We suggest these behaviors can be used as implicit indicators of the users task type.

Task and user effects on reading patterns in information search

We report on an investigation into people’s behaviors on information search tasks, specifically the relation between eye movement patterns and task characteristics. We conducted two independent user studies (n = 32 and n = 40), one with journalism tasks and the other with genomics tasks. The tasks were constructed to represent information needs of these two different users groups and to vary in several dimensions according to a task classification scheme. For each participant we classified eye gaze data to construct models of their reading patterns. The reading models were analyzed with respect to the effect of task types and Web page types on reading eye movement patterns. We report on relationships between tasks and individual reading behaviors at the task and page level. Specifically we show that transitions between scanning and reading behavior in eye movement patterns and the amount of text processed may be an implicit indicator of the current task type facets. This may be useful in building user and task models that can be useful in personalization of information systems and so address design demands driven by increasingly complex user actions with information systems. One of the contributions of this research is a new methodology to model information search behavior and investigate information acquisition and cognitive processing in interactive information tasks.

Inferring user knowledge level from eye movement patterns

The acquisition of information and the search interaction process is influenced strongly by a persons use of their knowledge of the domain and the task. In this paper we show that a users level of domain knowledge can be inferred from their interactive search behaviors without considering the content of queries or documents. A technique is presented to model a users information acquisition process during search using only measurements of eye movement patterns. In a user study (n=40) of search in the domain of genomics, a representation of the participants domain knowledge was constructed using self-ratings of knowledge of genomics-related terms (n=409). Cognitive effort features associated with reading eye movement patterns were calculated for each reading instance during the search tasks. The results show correlations between the cognitive effort due to reading and an individuals level of domain knowledge. We construct exploratory regression models that suggest it is possible to build models that can make predictions of the users level of knowledge based on real-time measurements of eye movement patterns during a task session.

Personalization of search results using interaction behaviors in search sessions

Personalization of search results offers the potential for significant improvement in information retrieval performance. User interactions with the system and documents during information-seeking sessions provide a wealth of information about user preferences and their task goals. In this paper, we propose methods for analyzing and modeling user search behavior in search sessions to predict document usefulness and then using information to personalize search results. We generate prediction models of document usefulness from behavior data collected in a controlled lab experiment with 32 participants, each completing uncontrolled searching for 4 tasks in the Web. The generated models are then tested with another data set of user search sessions in radically different search tasks and constrains. The documents predicted useful and not useful by the models are used to modify the queries in each search session using a standard relevance feedback technique. The results show that application of the models led to consistently improved performance over a baseline that did not take account of user interaction information. These findings have implications for designing systems for personalized search and improving user search experience.

Exploring and predicting search task difficulty

We report on an investigation of behavioral differences between users in difficult and easy search tasks. Behavioral factors that can be used in real-time to predict task difficulty are identified. User data was collected in a controlled lab experiment (n=38) where each participant completed four search tasks in the genomics domain. We looked at user behaviors that can be obtained by systems at three levels, distinguished by the time point when the measurements can be done. They are: 1) first-round level at the beginning of the search, 2) accumulated level during the search, and 3) whole-session level by the end of the search. Results show that a number of user behaviors at all three levels differed between easy and difficult tasks. Models predicting task difficulty at all three levels were developed and evaluated. A real-time model incorporating first-round and accumulated levels of behaviors (FA) had fairly good prediction performance (accuracy 83%; precision 88%), which is comparable with the model using the whole-session level behaviors which are not real-time (accuracy 75%; precision 92%). We also found that for efficiency purpose, using only a limited number of significant variables (FC_FA) can obtain a prediction accuracy of 75%, with a precision of 88%. Our findings can help search systems predict task difficulty and adapt search results to users.

User Activity Patterns During Information Search

Personalization of support for information seeking depends crucially on the information retrieval systems knowledge of the task that led the person to engage in information seeking. Users work during information search sessions to satisfy their task goals, and their activity is not random. To what degree are there patterns in the user activity during information search sessions? Do activity patterns reflect the users situation as the user moves through the search task under the influence of his or her task goal? Do these patterns reflect aspects of different types of information-seeking tasks? Could such activity patterns identify contexts within which information seeking takes place? To investigate these questions, we model sequences of user behaviors in two independent user studies of information search sessions (N = 32 users, 128 sessions, and N = 40 users, 160 sessions). Two representations of user activity patterns are used. One is based on the sequences of page use; the other is based on a cognitive representation of information acquisition derived from eye movement patterns in service of the reading process. One of the user studies considered journalism work tasks; the other concerned background research in genomics using search tasks taken from the TREC Genomics Track. The search tasks differed in basic dimensions of complexity, specificity, and the type of information product (intellectual or factual) needed to achieve the overall task goal. The results show that similar patterns of user activity are observed at both the cognitive and page use levels. The activity patterns at both representation layers are able to distinguish between task types in similar ways and, to some degree, between tasks of different levels of difficulty. We explore relationships between the results and task difficulty and discuss the use of activity patterns to explore events within a search session. User activity patterns can be at least partially observed in server-side search logs. A focus on patterns of user activity sequences may contribute to the development of information systems that better personalize the users search experience.

Predicting users' domain knowledge from search behaviors

This study uses regression modeling to predict a users domain knowledge level (DK) from implicit evidence provided by certain search behaviors. A user study (n=35) with recall-oriented search tasks in the genomic domain was conducted. A number of regression models of a persons DK, were generated using different behavior variable selection methods. The best model highlights three behavior variables as DK predictors: the number of documents saved, the average query length, and the average ranking position of documents opened. The model is validated using the split sampling method. Limitations and future research directions are discussed.

Dynamic assessment of information acquisition effort during interactive search

We present a method to measure some aspects of cognitive effort by a user while reading during a search session. We measured reading eye movement properties and patterns of eye movement in a user study (n=32) of participants carrying out realistic journalism IR work tasks. The results show the cognitive effort measures correlate positively with the information task characteristics that, by hypothesis, contribute to task difficulty. They also correlate well with participant self-assessed task difficulty. Our methodology can be applied to eye tracking data in any (textual) information setting and used to give dynamic estimates of these aspects of cognitive processing during search.

Linking search tasks with low-level eye movement patterns

Motivation -- On-the-task detection of the task type and task attributes can benefit personalization and adaptation of information systems. Research approach -- A web-based information search experiment was conducted with 32 participants using a multi-stream logging system. The realistic tasks were related directly to the backgrounds of the participants and were of distinct task types. Findings/Design -- We report on a relationship between task and individual reading behaviour. Specifically we show that transitions between scanning and reading behaviour in eye movement patterns are an implicit indicator of the current task. Research limitations/Implications -- This work suggests it is plausible to infer the type of information task from eye movement patterns. One limitation is a lack of knowledge about the general reading model differences across different types of tasks in the population. Although this is an experimental study we argue it can be generalized to real world text-oriented information search tasks. Originality/Value -- This research presents a new methodology to model user information search task behaviour. It suggests promise for detection of information task type based on patterns of eye movements. Take away message -- With increasingly complex computer interaction, knowledge about the type of information task can be valuable for system personalization. Modelling the reading/scanning patterns of eye movements can allow inference about the task type and task attributes.

Task Topic Knowledge vs. Background Domain Knowledge: Impact of Two Types of Knowledge on User Search Performance

Domain knowledge affects search success and behaviors but different aspects of domain knowledge, i.e., topic knowledge and background knowledge, has not normally been distinguished. We were interested in finding out if the two would have different effects on search performance for typical research search tasks, and we report empirical evidence from a user study. Participants’ topic knowledge and background knowledge were measured separately as different aspects of domain knowledge. Our results indicate that topic knowledge and background knowledge impact search performance at different levels. While topic knowledge was significantly correlated with performance measures for individual tasks, background knowledge was significantly correlated with performance at more general levels, over multiple task sessions. This finding furthers our understanding of the role of domain knowledge in search activities.