Martijn C. Willemsen

Explaining the user experience of recommender systems

Research on recommender systems typically focuses on the accuracy of prediction algorithms. Because accuracy only partially constitutes the user experience of a recommender system, this paper proposes a framework that takes a user-centric approach to recommender system evaluation. The framework links objective system aspects to objective user behavior through a series of perceptual and evaluative constructs (called subjective system aspects and experience, respectively). Furthermore, it incorporates the influence of personal and situational characteristics on the user experience. This paper reviews how current literature maps to the framework and identifies several gaps in existing work. Consequently, the framework is validated with four field trials and two controlled experiments and analyzed using Structural Equation Modeling. The results of these studies show that subjective system aspects and experience variables are invaluable in explaining why and how the user experience of recommender systems comes about. In all studies we observe that perceptions of recommendation quality and/or variety are important mediators in predicting the effects of objective system aspects on the three components of user experience: process (e.g. perceived effort, difficulty), system (e.g. perceived system effectiveness) and outcome (e.g. choice satisfaction). Furthermore, we find that these subjective aspects have strong and sometimes interesting behavioral correlates (e.g. reduced browsing indicates higher system effectiveness). They also show several tradeoffs between system aspects and personal and situational characteristics (e.g. the amount of preference feedback users provide is a tradeoff between perceived system usefulness and privacy concerns). These results, as well as the validated framework itself, provide a platform for future research on the user-centric evaluation of recommender systems.

Understanding choice overload in recommender systems

Even though people are attracted by large, high quality recommendation sets, psychological research on choice overload shows that choosing an item from recommendation sets containing many attractive items can be a very difficult task. A web-based user experiment using a matrix factorization algorithm applied to the MovieLens dataset was used to investigate the effect of recommendation set size (5 or 20 items) and set quality (low or high) on perceived variety, recommendation set attractiveness, choice difficulty and satisfaction with the chosen item. The results show that larger sets containing only good items do not necessarily result in higher choice satisfaction compared to smaller sets, as the increased recommendation set attractiveness is counteracted by the increased difficulty of choosing from these sets. These findings were supported by behavioral measurements revealing intensified information search and increased acquisition times for these large attractive sets. Important implications of these findings for the design of recommender system user interfaces will be discussed.

Process models deserve process data : comment on Brandstätter, Gigerenzer, and Hertwig (2006)

Comments on the article by E. Brandstätter, G. Gigerenzer, and R. Hertwig. Resolution of debates in cognition usually comes from the introduction of constraints in the form of new data about either the process or representation. Decision research, in contrast, has relied predominantly on testing models by examining their fit to choices. The authors examine a recently proposed choice strategy, the priority heuristic, which provides a novel account of how people make risky choices. The authors identify a number of properties that the priority heuristic should have as a process model and illustrate how they may be tested. The results, along with prior research, suggest that although the priority heuristic captures some variability in the attention paid to outcomes, it fails to account for major characteristics of the data, particularly the frequent transitions between outcomes and their probabilities. The article concludes with a discussion of the properties that should be captured by process models of risky choice and the role of process data in theory development.

Each to his own: how different users call for different interaction methods in recommender systems

This paper compares five different ways of interacting with an attribute-based recommender system and shows that different types of users prefer different interaction methods. In an online experiment with an energy-saving recommender system the interaction methods are compared in terms of perceived control, understandability, trust in the system, user interface satisfaction, system effectiveness and choice satisfaction. The comparison takes into account several user characteristics, namely domain knowledge, trusting propensity and persistence. The results show that most users (and particularly domain experts) are most satisfied with a hybrid recommender that combines implicit and explicit preference elicitation, but that novices and maximizers seem to benefit more from a non-personalized recommender that just displays the most popular items.

Choice by Value Encoding and Value Construction: Processes of Loss Aversion.

Loss aversion and reference dependence are 2 keystones of behavioral theories of choice, but little is known about their underlying cognitive processes. We suggest an additional account for loss aversion that supplements the current account of the value encoding of attributes as gains or losses relative to a reference point, introducing a value construction account. Value construction suggests that loss aversion results from biased evaluations during information search and comparison processes. We develop hypotheses that identify the influence of both accounts and examine process-tracing data for evidence. Our data suggest that loss aversion is the result of the initial direct encoding of losses that leads to the subsequent process of directional comparisons distorting attribute valuations and the final choice.

Rating support interfaces to improve user experience and recommender accuracy

One of the challenges for recommender systems is that users struggle to accurately map their internal preferences to external measures of quality such as ratings. We study two methods for supporting the mapping process: (i) reminding the user of characteristics of items by providing personalized tags and (ii) relating rating decisions to prior rating decisions using exemplars. In our study, we introduce interfaces that provide these methods of support. We also present a set of methodologies to evaluate the efficacy of the new interfaces via a user experiment. Our results suggest that presenting exemplars during the rating process helps users rate more consistently, and increases the quality of the data.

Evaluating recommender systems with user experiments

Proper evaluation of the user experience of recommender systems requires conducting user experiments. This chapter is a guideline for students and researchers aspiring to conduct user experiments with their recommender systems. It first covers the theory of user-centric evaluation of recommender systems, and gives an overview of recommender system aspects to evaluate. It then provides a detailed practical description of how to conduct user experiments, covering the following topics: formulating hypotheses, sampling participants, creating experimental manipulations, measuring subjective constructs with questionnaires, and statistically evaluating the results.

The meaning of indifference in choice behavior: Asymmetries in adjustments embodied in matching

Abstract Measuring and assessing people’s preferences is an intricate task that can be realized by different methods. Evidently, the different methods do not necessarily yield consistent results. In the present paper we examine the matching procedure and demonstrate inconsistencies within the method itself. It is shown that the inconsistencies stem from: (1) asymmetry in upward and downward matching (i.e., matching options by increasing or lowering values do not yield the same tradeoffs) and (2) participants’ tendency to match (adjust) the options such that they become more similar, eventually facilitating the choice process. Matching is the most straightforward method of eliciting indifference. Hence, the observed inconsistencies, encapsulated in the matching procedure, have direct implications for the construction of indifference curves (equal-utility contours) that are examined in the final section.

Negative-based prominence: the role of negative features in matching and choice

Previous research related to the prominence effect (e.g., Tversky, Sattath, & Slovic, 1988) has mainly focused on the causes and boundary conditions of the effect. This article investigates the determinants of prominence and explores a negative-based prominence effect in which the negative attribute becomes the prominent one. Using a matching-choice procedure, Experiments 1 and 2 show that the negative dimension became the prominent one under two different cover stories, suggesting that the negative feature looms larger in choice than in matching. The robustness of negative-based prominence was further demonstrated in a direct choice task without matching (Experiment 3) and was shown to be resistant to the impact of added positive features (Experiment 4). Finally, Experiments 5 and 6 distinguish between two determinants of prominence: intrinsic prominence caused by attribute importance and negative-based prominence caused by negative attribute values. By lowering the values on the positive dimension and enhancing the values on the negative dimension, one can reverse the prominence effect. The results suggest that, compared to a matching task, choice leads to enhanced sensitivity to negative features. The relations between negative characteristics of an attribute and other factors that determine prominence are discussed in the final section.

Percentage scaling: a new method for evaluating multiply impaired images

Data compression of digitized images usually leads to pictures with two or more perceptually distinguishable coding artifacts. In this paper, percentage scaling is presented as a new method for analyzing the perceived quality of multiply impaired images. By this method, subjects assess, per coding level, the proportion each artifact contributes to the overall impairment of the perceived quality of the coded image. These proportions are expressed in percentages such that the total sum is always 100%. As an illustration, percentage scaling has been employed to evaluate the perceived quality of JPEG-coded images comprising three types of coding artifacts: blockiness, ringing and blur. In one experiment, eight subjects expressed in percentages how much they believed these artifacts contributed to the overall impairment. In addition, overall impairment as well as the perceived strengths of blockiness, ringing and blur was assessed on 11-point numerical category scales. The percentages were found to vary with the coding level. Blockiness appeared to be the most dominant artifact at high compression levels, whereas ringing turned out to be the most annoying artifact at low compression levels. The results support the claim that perceptually distinguishable coding artifacts can be represented by a set of orthogonal vectors in a multidimensional Euclidean space.