Rong Hu

A user-centric evaluation framework for recommender systems

This research was motivated by our interest in understanding the criteria for measuring the success of a recommender system from users point view. Even though existing work has suggested a wide range of criteria, the consistency and validity of the combined criteria have not been tested. In this paper, we describe a unifying evaluation framework, called ResQue (Recommender systems Quality of user experience), which aimed at measuring the qualities of the recommended items, the systems usability, usefulness, interface and interaction qualities, users satisfaction with the systems, and the influence of these qualities on users behavioral intentions, including their intention to purchase the products recommended to them and return to the system. We also show the results of applying psychometric methods to validate the combined criteria using data collected from a large user survey. The outcomes of the validation are able to 1) support the consistency, validity and reliability of the selected criteria; and 2) explain the quality of user experience and the key determinants motivating users to adopt the recommender technology. The final model consists of thirty two questions and fifteen constructs, defining the essential qualities of an effective and satisfying recommender system, as well as providing practitioners and scholars with a cost-effective way to evaluate the success of a recommender system and identify important areas in which to invest development resources.

Evaluating recommender systems from the user's perspective: survey of the state of the art

A recommender system is a Web technology that proactively suggests items of interest to users based on their objective behavior or explicitly stated preferences. Evaluations of recommender systems (RS) have traditionally focused on the performance of algorithms. However, many researchers have recently started investigating system effectiveness and evaluation criteria from users’ perspectives. In this paper, we survey the state of the art of user experience research in RS by examining how researchers have evaluated design methods that augment RS’s ability to help users find the information or product that they truly prefer, interact with ease with the system, and form trust with RS through system transparency, control and privacy preserving mechanisms finally, we examine how these system design features influence users’ adoption of the technology. We summarize existing work concerning three crucial interaction activities between the user and the system: the initial preference elicitation process, the preference refinement process, and the presentation of the system’s recommendation results. Additionally, we will also cover recent evaluation frameworks that measure a recommender system’s overall perceptive qualities and how these qualities influence users’ behavioral intentions. The key results are summarized in a set of design guidelines that can provide useful suggestions to scholars and practitioners concerning the design and development of effective recommender systems. The survey also lays groundwork for researchers to pursue future topics that have not been covered by existing methods.

Enhancing collaborative filtering systems with personality information

Collaborative filtering (CF), one of the most successful recommendation approaches, continues to attract interest in both academia and industry. However, one key issue limiting the success of collaborative filtering in certain application domains is the cold-start problem, a situation where historical data is too sparse (known as the sparsity problem), new users have not rated enough items (known as the new user problem), or both. In this paper, we aim at addressing the cold-start problem by incorporating human personality into the collaborative filtering framework. We propose three approaches: the first is a recommendation method based on users personality information alone; the second is based on a linear combination of both personality and rating information; and the third uses a cascade mechanism to leverage both resources. To evaluate their effectiveness, we have conducted an experimental study comparing the proposed approaches with the traditional rating-based CF in two cold-start scenarios: sparse data sets and new users. Our results show that the proposed CF variations, which consider personality characteristics, can significantly improve the performance of the traditional rating-based CF in terms of the evaluation metrics MAE and ROC sensitivity.

Video Stabilization Using Scale-Invariant Features

Video Stabilization is one of those important video processing techniques to remove the unwanted camera vibration in a video sequence. In this paper, we present a practical method to remove the annoying shaky motion and reconstruct a stabilized video sequence with good visual quality. Here, the scale invariant (SIFT) features, proved to be invariant to image scale and rotation, is applied to estimate the camera motion. The unwanted vibrations are separated from the intentional camera motion with the combination of Gaussian kernel filtering and parabolic fitting. It is demonstrated that our method effectively removes the high frequency noise motion, but also minimize the missing area as much as possible. To reconstruct the undefined areas, resulting from motion compensation, we adopt the mosaicing method with Dynamic Programming. The proposed method has been confirmed to be effective over a widely variety of videos.

Acceptance issues of personality-based recommender systems

To understand users acceptance of the emerging trend of personality-based recommenders (PBR), we evaluated an existing PBR using the technology acceptance model (TAM). We also compare it with a baseline rating-based recommender in a within-subject user study. Our results show that while the personality-based recommender is perceived to be only slightly more accurate than the rating-based one, it is much easier to use. The side-by-side comparison also reveals that users significantly favor the personality-based recommender and have a significantly higher intention to use such a system again. Therefore, we believe that if users accepted rating-based recommenders, they are most likely to accept personality-based recommenders and personality-based recommenders have a high likelihood to be widely adopted despite the fact that rating-based recommenders are now the industry norm. We further point out some preliminary guidelines on how to design personality-based recommender systems.

A comparative user study on rating vs. personality quiz based preference elicitation methods

We conducted a user study evaluating two preference elicitation approaches based on ratings and personality quizzes respectively. Three criteria were used in this comparative study: perceived accuracy, user effort and user loyalty. Results from our study show that the perceived accuracy in two systems is not significantly different. However, users expended significantly less effort, both perceived cognitive effort and actual task time, to complete the preference profile establishing process in the personality quiz-based system than in the rating-based system. Additionally, users expressed stronger intention to reuse the personality quiz-based system and introduce it to their friends. After using these two systems, 53% of users preferred the personality quiz-based system vs. 13% of users preferred the rating-based system, since most users thought the former is easier to use.

Personality-based recommender systems: an overview

Personality is a critical factor which influences peoples behavior and interests. There is a high potential that incorporating users characteristics into recommender systems could enhance recommendation quality and user experience. The goal of this tutorial is to give an overview of personality-based recommender systems and discuss challenges and possible research directions in this topic.

A study on user perception of personality-based recommender systems

Our previous research indicates that using personality quizzes is a viable and promising way to build user profiles to recommend entertainment products Based on these findings, our current research further investigates the feasibility of using personality quizzes to build user profiles not only for an active user but also his or her friends We first propose a general method that infers users music preferences in terms of their personalities Our in-depth user studies show that while active users perceive the recommended items to be more accurate for their friends, they enjoy more using personality quiz based recommenders for finding items for themselves Additionally, we explore if domain knowledge has an influence on users perception of the system We found that novice users, who are less knowledgeable about music, generally appreciated more personality-based recommenders Finally, we propose some design issues for recommender systems using personality quizzes.

Enhancing recommendation diversity with organization interfaces

Research increasingly indicates that accuracy cannot be the sole criteria in creating a satisfying recommender from the users point of view. Other criteria, such as diversity, are emerging as important characteristics for consideration as well. In this paper, we try to address the problem of augmenting users perception of recommendation diversity by applying an organization interface design method to the commonly used list interface. An in-depth user study was conducted to compare an organization interface with a standard list interface. Our results show that the organization interface indeed effectively increased users perceived diversity of recommendations, especially perceived categorical diversity. Furthermore, 65% of users preferred the organization interface, versus 20% for the list interface. 70% of users thought the organization interface is better at helping them perceive recommendation diversity versus only 15% for the list interface.

Design and user issues in personality-based recommender systems

Recommender systems have emerged as an intelligent information filtering tool to help users effectively identify information items of interest from an overwhelming set of choices and provide personalized services. Studies show that personality influences human decision making process and interests. However, little research has ventured in incorporating it into recommender systems. The utilization of personality characteristics into recommender systems and the exploration of user perception to such systems are the focuses of my thesis. The overall goal is to develop an efficient personality-based recommender system and to arrive at a series of design guidelines from the perspective of human computer interaction. In this paper, I present my up-to-date results on a proposed personality-based music recommender prototype, user perception investigations, and my ongoing research about addressing new user problem by utilizing personality characteristics. Finally, I shall present future works.