Bart P. Knijnenburg

Explaining the user experience of recommender systems

Research on recommender systems typically focuses on the accuracy of prediction algorithms. Because accuracy only partially constitutes the user experience of a recommender system, this paper proposes a framework that takes a user-centric approach to recommender system evaluation. The framework links objective system aspects to objective user behavior through a series of perceptual and evaluative constructs (called subjective system aspects and experience, respectively). Furthermore, it incorporates the influence of personal and situational characteristics on the user experience. This paper reviews how current literature maps to the framework and identifies several gaps in existing work. Consequently, the framework is validated with four field trials and two controlled experiments and analyzed using Structural Equation Modeling. The results of these studies show that subjective system aspects and experience variables are invaluable in explaining why and how the user experience of recommender systems comes about. In all studies we observe that perceptions of recommendation quality and/or variety are important mediators in predicting the effects of objective system aspects on the three components of user experience: process (e.g. perceived effort, difficulty), system (e.g. perceived system effectiveness) and outcome (e.g. choice satisfaction). Furthermore, we find that these subjective aspects have strong and sometimes interesting behavioral correlates (e.g. reduced browsing indicates higher system effectiveness). They also show several tradeoffs between system aspects and personal and situational characteristics (e.g. the amount of preference feedback users provide is a tradeoff between perceived system usefulness and privacy concerns). These results, as well as the validated framework itself, provide a platform for future research on the user-centric evaluation of recommender systems.

Understanding choice overload in recommender systems

Even though people are attracted by large, high quality recommendation sets, psychological research on choice overload shows that choosing an item from recommendation sets containing many attractive items can be a very difficult task. A web-based user experiment using a matrix factorization algorithm applied to the MovieLens dataset was used to investigate the effect of recommendation set size (5 or 20 items) and set quality (low or high) on perceived variety, recommendation set attractiveness, choice difficulty and satisfaction with the chosen item. The results show that larger sets containing only good items do not necessarily result in higher choice satisfaction compared to smaller sets, as the increased recommendation set attractiveness is counteracted by the increased difficulty of choosing from these sets. These findings were supported by behavioral measurements revealing intensified information search and increased acquisition times for these large attractive sets. Important implications of these findings for the design of recommender system user interfaces will be discussed.

Making Decisions about Privacy: Information Disclosure in Context-Aware Recommender Systems

Recommender systems increasingly use contextual and demographical data as a basis for recommendations. Users, however, often feel uncomfortable providing such information. In a privacy-minded design of recommenders, users are free to decide for themselves what data they want to disclose about themselves. But this decision is often complex and burdensome, because the consequences of disclosing personal information are uncertain or even unknown. Although a number of researchers have tried to analyze and facilitate such information disclosure decisions, their research results are fragmented, and they often do not hold up well across studies. This article describes a unified approach to privacy decision research that describes the cognitive processes involved in users’ “privacy calculus” in terms of system-related perceptions and experiences that act as mediating factors to information disclosure. The approach is applied in an online experiment with 493 participants using a mock-up of a context-aware recommender system. Analyzing the results with a structural linear model, we demonstrate that personal privacy concerns and disclosure justification messages affect the perception of and experience with a system, which in turn drive information disclosure decisions. Overall, disclosure justification messages do not increase disclosure. Although they are perceived to be valuable, they decrease users’ trust and satisfaction. Another result is that manipulating the order of the requests increases the disclosure of items requested early but decreases the disclosure of items requested later.

Each to his own: how different users call for different interaction methods in recommender systems

This paper compares five different ways of interacting with an attribute-based recommender system and shows that different types of users prefer different interaction methods. In an online experiment with an energy-saving recommender system the interaction methods are compared in terms of perceived control, understandability, trust in the system, user interface satisfaction, system effectiveness and choice satisfaction. The comparison takes into account several user characteristics, namely domain knowledge, trusting propensity and persistence. The results show that most users (and particularly domain experts) are most satisfied with a hybrid recommender that combines implicit and explicit preference elicitation, but that novices and maximizers seem to benefit more from a non-personalized recommender that just displays the most popular items.

Evaluating recommender systems with user experiments

Proper evaluation of the user experience of recommender systems requires conducting user experiments. This chapter is a guideline for students and researchers aspiring to conduct user experiments with their recommender systems. It first covers the theory of user-centric evaluation of recommender systems, and gives an overview of recommender system aspects to evaluate. It then provides a detailed practical description of how to conduct user experiments, covering the following topics: formulating hypotheses, sampling participants, creating experimental manipulations, measuring subjective constructs with questionnaires, and statistically evaluating the results.

Privacy Aspects of Recommender Systems

The popularity of online recommender systems has soared; they are deployed in numerous websites and gather tremendous amounts of user data that are necessary for recommendation purposes. This data, however, may pose a severe threat to user privacy, if accessed by untrusted parties or used inappropriately. Hence, it is of paramount importance for recommender system designers and service providers to find a sweet spot, which allows them to generate accurate recommendations and guarantee the privacy of their users. In this chapter we overview the state of the art in privacy enhanced recommendations. We analyze the risks to user privacy imposed by recommender systems, survey the existing solutions, and discuss the privacy implications for the users of recommenders. We conclude that a considerable effort is still required to develop practical recommendation solutions that provide adequate privacy guarantees, while at the same time facilitating the delivery of high-quality recommendations to their users.

Receiving Recommendations and Providing Feedback: The User-Experience of a Recommender System

This paper systematically evaluates the user experience of a recommender system. Using both behavioral data and subjective measures of user experience, we demonstrate that choice satisfaction and system effectiveness increase when a system provides personalized recommendations (compared to the same system that provides random recommendations). We furthermore demonstrate that despite privacy issues, this higher choice satisfaction and system effectiveness increases users’ intention to provide feedback about their preference. Due to an intention-behavior gap, this may however not necessarily influence the users’ actual feedback behavior.

Workshop on user-centric evaluation of recommender systems and their interfaces

1. WORKSHOP GOALS In his keynote speech at the 2009 Recommender Systems conference, Francisco Martin indicated that the main challenge in recommender system industry is not to discover algorithms that provide good recommendations, but to provide users with a usable and intuitive interface for presenting these recommendations and eliciting feedback [1]. Several researchers have argued that other factors that may influence the user experience have not received the amount of attention they deserve [2][3][4]. Unfortunately, the research on ‘Human-Recommender Interaction’ is scarce. While algorithm optimization and off-line testing using measures like RMSE are standard procedure in the recommender systems community, theorizing about consumer decision processes and measuring user experience in online tests is much less common.

What a tangled web we weave: lying backfires in location-sharing social media

Prior research shows that a root cause of many privacy concerns in location-sharing social media is peoples desire to preserve offline relationship boundaries. Other literature recognizes lying as an everyday phenomenon that preserves such relationship boundaries by facilitating smooth social interactions. Combining these strands of research, one might hypothesize that people with a predisposition to lie would generally have lower privacy concerns since lying is a means to preserve relationship boundaries. We tested this hypothesis using structural equation modeling on data from a survey administered nationwide (N=1532), and found that for location-sharing, people with a high propensity to lie actually have increased boundary preservation concerns as well as increased privacy concerns. We explain these findings using results from semi-structured interviews.

Conducting user experiments in recommender systems

There is an increasing consensus in the field of recommender systems that we should move beyond the offline evaluation of algorithms towards a more user-centric approach. This tutorial teaches the essential skills involved in conducting user experiments, the scientific approach to user-centric evaluation. Such experiments are essential in uncovering how and why the user experience of recommender systems comes about.