Pearl Pu

Evaluating recommender systems from the user's perspective: survey of the state of the art

A recommender system is a Web technology that proactively suggests items of interest to users based on their objective behavior or explicitly stated preferences. Evaluations of recommender systems (RS) have traditionally focused on the performance of algorithms. However, many researchers have recently started investigating system effectiveness and evaluation criteria from users’ perspectives. In this paper, we survey the state of the art of user experience research in RS by examining how researchers have evaluated design methods that augment RS’s ability to help users find the information or product that they truly prefer, interact with ease with the system, and form trust with RS through system transparency, control and privacy preserving mechanisms finally, we examine how these system design features influence users’ adoption of the technology. We summarize existing work concerning three crucial interaction activities between the user and the system: the initial preference elicitation process, the preference refinement process, and the presentation of the system’s recommendation results. Additionally, we will also cover recent evaluation frameworks that measure a recommender system’s overall perceptive qualities and how these qualities influence users’ behavioral intentions. The key results are summarized in a set of design guidelines that can provide useful suggestions to scholars and practitioners concerning the design and development of effective recommender systems. The survey also lays groundwork for researchers to pursue future topics that have not been covered by existing methods.

Critiquing-based recommenders: survey and emerging trends

Critiquing-based recommender systems elicit users’ feedback, called critiques, which they made on the recommended items. This conversational style of interaction is in contract to the standard model where users receive recommendations in a single interaction. Through the use of the critiquing feedback, the recommender systems are able to more accurately learn the users’ profiles, and therefore suggest better recommendations in the subsequent rounds. Critiquing-based recommenders have been widely studied in knowledge-, content-, and preference-based recommenders and are beginning to be tried in several online websites, such as MovieLens. This article examines the motivation and development of the subject area, and offers a detailed survey of the state of the art concerning the design of critiquing interfaces and development of algorithms for critiquing generation. With the help of categorization analysis, the survey reveals three principal branches of critiquing based recommender systems, using respectively natural language based, system-suggested, and user-initiated critiques. Representative example systems will be presented and analyzed for each branch, and their respective pros and cons will be discussed. Subsequently, a hybrid framework is developed to unify the advantages of different methods and overcome their respective limitations. Empirical findings from user studies are further presented, indicating how hybrid critiquing supports could effectively enable end-users to achieve more confident decisions. Finally, the article will point out several future trends to boost the advance of critiquing-based recommenders.

Enriching buyers' experiences: the SmartClient approach

In electronic commerce, a satisfying buyer experience is a key competitive element. We show new techniques for better adapting interaction with an electronic catalog system to actual buying behavior. Our model replaces the sequential separation of needs identification and product brokering with a conversation in which both processes occur simultaneously. This conversation supports the buyer in formulating his or her needs, and in deciding which criteria to apply in selecting a product to buy. We have experimented with this approach in the area of travel planning and developed a system called SmartClient Travel which supports this process. It includes tools for need identification, visualization of alternatives, and choosing the most suitable one. We describe the system and its implementation, and report on user studies showing its advantages for electronic catalogs.

SmartClients : Constraint Satisfaction as a Paradigm for Scaleable Intelligent Information Systems

Many information systems are used in a problem solving context. Examples are travel planning systems, catalogs in electronic commerce, or agenda planning systems. They can be made more useful by integrating problem-solving capabilities into the information systems. This poses the challenge of scaleability: when hundreds of users access a server at the same time, it is important to avoid excessive computational load.We present the concept of SmartClients: lightweight problem-solving agents based on constraint satisfaction which can carry out the computation- and communication-intensive tasks on the users computer. We present an example of an air travel planning system based on this technology.

Evaluating example-based search tools

A crucial element in consumer electronic commerce is a catalog tool that not only finds the product for the user, but also convinces him that he has made the best choice. To do that, it is important to show him ample choices while keeping his interaction effort below an acceptable limit. Among the various interaction models used in operational e-commerce sites, ranked lists are by far the most popular tool for product navigation and selection. However, as the number of product features and the complexity of users criteria increase, a ranked lists efficiency becomes less satisfactory. As an alternative, research groups from the intelligent user interface community have developed various example-based search tools, including SmartClient from our laboratory. These tools not only perform personalized search, but also support tradeoff analysis. However, despite the academic interest, example-based search paradigms have not been widely adopted in practice. We have examined the usability of such tools on a variety of tasks involving selection and tradeoff. The studies clearly show that example-based search is comparable to ranked lists on simple tasks, but significantly reduces the error rate and search time when complex tradeoffs are involved. This shows that such tools are likely to be useful particularly for extending the scope of consumer e-commerce to more complex products.

A new development in camera calibration calibrating a pair of mobile cameras

A new method of calibration is proposed for active visual sensing for use in 3D analysis. The method, based on an adaptation of the two camera plane m model, permits the calibration of a mobile camera as a function of the position and orientation of the camera.

Designing example-critiquing interaction

In many practical scenarios, users are faced with the problem of choosing the most preferred outcome from a large set of possibilities. As people are unable to sift through them manually, decisions support systems are often used to automatically find the optimal solution. A crucial requirement for such a system is to have an accurate model of the users preferences.Studies have shown that people are usually unable to accurately state their preferences up front, but are greatly helped by seeing examples of actual solutions. Thus, several researchers have proposed preference elicitation strategies based on example critiquing. The essential design question in example critiquing is what examples to show users in order to best help them locate their most preferred solution.In this paper, we analyze this question based on two requirements. The first is that it must stimulate the user to express further preferences by showing the range of alternatives available. The second is that the examples that are shown must contain the solution that the user would consider optimal if the currently expressed preference model was complete so that he select it as a final solution.

User-Involved Preference Elicitation for Product Search and Recommender Systems

We address user system interaction issues in product search and recommender systems: how to help users select the most preferential item from a large collection of alternatives. As such systems must crucially rely on an accurate and complete model of user preferences, the acquisition of this model becomes the central subject of our paper. Many tools used today do not satisfactorily assist users to establish this model because they do not adequately focus on fundamental decision objectives, help them reveal hidden preferences, revise conflicting preferences, or explicitly reason about tradeoffs. As a result, users fail to find the outcomes that best satisfy their needs and preferences. In this article, we provide some analyses of common areas of design pitfalls and derive a set of design guidelines that assist the user in avoiding these problems in three important areas: user preference elicitation, preference revision, and explanation interfaces. For each area, we describe the state-of-the-art of the developed techniques and discuss concrete scenarios where they have been applied and tested.

Adaptation Using Constraint Satisfaction Techniques

Case adaptation, a central component of case-based reasoning, is often considered to be the most difficult part of a case-based reasoning system. The difficulties arise from the fact that adaptation often does not converge, especially if it is not done in a systematic way. This problem, sometimes termed the assimilation problem, is especially pronounced in the case-based design problem solving domain where a large set of constraints and features are processed. Furthermore, in the design domain, multiple cases must be considered in conjunction in order to solve the new problem, resulting in the difficulty of how to efficiently combine the cases into a global solution for the new problem.

Decision Tradeoff Using Example-Critiquing and Constraint Programming

We consider constructive preference elicitation for decision aid systems in applications such as configuration or electronic catalogs. We are particularly interested in supporting decision tradeoff, where preferences are revised in response to the available outcomes. In several user-involved decision aid systems we designed in the past, we were able to observe three generic tradeoff strategies that people like to use. We show how a preference model based on soft constraints is well-suited for supporting these strategies. Such a framework provides an agile preference model particularly powerful for preference revision during tradeoff analysis. We further show how to integrate the constraint-based preference model with an interaction model called example-critiquing. We report on user studies which show that this model offers significant advantages over the commonly used ranked-list model, especially when the decision problem becomes complex.