F. Maxwell Harper

The MovieLens Datasets: History and Context

The MovieLens datasets are widely used in education, research, and industry. They are downloaded hundreds of thousands of times each year, reflecting their use in popular press programming books, traditional and online courses, and software. These datasets are a product of member activity in the MovieLens movie recommendation system, an active research platform that has hosted many experiments since its launch in 1997. This article documents the history of MovieLens and the MovieLens datasets. We include a discussion of lessons learned from running a long-standing, live research platform from the perspective of a research organization. We document best practices and limitations of using the MovieLens datasets in new research.

Predictors of answer quality in online Q&A sites

Question and answer (Q&A) sites such as Yahoo! Answers are places where users ask questions and others answer them. In this paper, we investigate predictors of answer quality through a comparative, controlled field study of responses provided across several online Q&A sites. Along with several quantitative results concerning the effects of factors such as question topic and rhetorical strategy, we present two high-level messages. First, you get what you pay for in Q&A sites. Answer quality was typically higher in Google Answers (a fee-based site) than in the free sites we studied, and paying more money for an answer led to better outcomes. Second, we find that a Q&A sites community of users contributes to its success. Yahoo! Answers, a Q&A site where anybody can answer questions, outperformed sites that depend on specific individuals to answer questions, such as library reference services.

The quest for quality tags

Many online communities use tags - community selected words or phrases - to help people find what they desire. The quality of tags varies widely, from tags that capture akey dimension of an entity to those that are profane, useless, or unintelligible. Tagging systems must often select a subset of available tags to display to users due to limited screen space. Because users often spread tags they have seen, selecting good tags not only improves an individuals view of tags, it also encourages them to create better tags in the future. We explore implicit (behavioral) and explicit (rating) mechanisms for determining tag quality. Based on 102,056 tag ratings and survey responses collected from 1,039 users over 100 days, we offer simple suggestions to designers of online communities to improve the quality of tags seen by their users.

Exploring Question Selection Bias to Identify Experts and Potential Experts in Community Question Answering

Community Question Answering (CQA) services enable their users to exchange knowledge in the form of questions and answers. These communities thrive as a result of a small number of highly active users, typically called experts, who provide a large number of high-quality useful answers. Expert identification techniques enable community managers to take measures to retain the experts in the community. There is further value in identifying the experts during the first few weeks of their participation as it would allow measures to nurture and retain them. In this article we address two problems: (a) How to identify current experts in CQA? and (b) How to identify users who have potential of becoming experts in future (potential experts)? In particular, we propose a probabilistic model that captures the selection preferences of users based on the questions they choose for answering. The probabilistic model allows us to run machine learning methods for identifying experts and potential experts. Our results over several popular CQA datasets indicate that experts differ considerably from ordinary users in their selection preferences; enabling us to predict experts with higher accuracy over several baseline models. We show that selection preferences can be combined with baseline measures to improve the predictive performance even further.

An economic model of user rating in an online recommender system

Economic modeling provides a formal mechanism to understand user incentives and behavior in online systems. In this paper we describe the process of building a parameterized economic model of user-contributed ratings in an online movie recommender system. We constructed a theoretical model to formalize our initial understanding of the system, and collected survey and behavioral data to calibrate an empirical model. This model explains 34% of the variation in user rating behavior. We found that while economic modeling in this domain requires an initial understanding of user behavior and access to an uncommonly broad set of user survey and behavioral data, it returns significant formal understanding of the activity being modeled.

Talk amongst yourselves: inviting users to participate in online conversations

Many small online communities would benefit from increased diversity or activity in their membership. Some communities run the risk of dying out due to lack of participation. Others struggle to achieve the critical mass necessary for diverse and engaging conversation. But what tools are available to these communities to increase participation? Our goal in this research was to spark contributions to the movielens.org discussion forum, where only 2% of the members write posts. We developed personalized invitations, messages designed to entice users to visit or contribute to the forum. In two field experiments, we ask (1) if personalized invitations increase activity in a discussion forum, (2) how the choice of algorithm for intelligently choosing content to emphasize in the invitation affects participation, and (3) how the suggestion made to the user affects their willingness to act. We find that invitations lead to increased participation, as measured by levels of reading and posting. More surprisingly, we find that invitations emphasizing the social nature of the discussion forum increase user activity, while invitations emphasizing other details of the discussion are less successful.

The context, content & community collage: sharing personal digital media in the physical workplace

Online social media services enable people to share many aspects of their personal interests and passions with friends, acquaintances and strangers. We are investigating how the display of social media in a workplace context can improve relationships among collocated colleagues. We have designed, developed and deployed the Context, Content and Community Collage, which runs on large LCD touchscreen computers installed in eight locations throughout a research laboratory. This proactive display application senses nearby people via Bluetooth phones, and responds by incrementally adding photos associated with those people to an ambient collage shown on the screen. This paper describes the motivations, goals, design and impact of the system, highlighting the ways the system has increased interactions and improved personal relationships among coworkers at the deployment site. We also look at how the creation of a shared physical window into online media has affected the use of that media.

Letting Users Choose Recommender Algorithms: An Experimental Study

Recommender systems are not one-size-fits-all; different algorithms and data sources have different strengths, making them a better or worse fit for different users and use cases. As one way of taking advantage of the relative merits of different algorithms, we gave users the ability to change the algorithm providing their movie recommendations and studied how they make use of this power. We conducted our study with the launch of a new version of the MovieLens movie recommender that supports multiple recommender algorithms and allows users to choose the algorithm they want to provide their recommendations. We examine log data from user interactions with this new feature to under-stand whether and how users switch among recommender algorithms, and select a final algorithm to use. We also look at the properties of the algorithms as they were experienced by users and examine their relationships to user behavior. We found that a substantial portion of our user base (25%) used the recommender-switching feature. The majority of users who used the control only switched algorithms a few times, trying a few out and settling down on an algorithm that they would leave alone. The largest number of users prefer a matrix factorization algorithm, followed closely by item-item collaborative filtering; users selected both of these algorithms much more often than they chose a non-personalized mean recommender. The algorithms did produce measurably different recommender lists for the users in the study, but these differences were not directly predictive of user choice.

Insert movie reference here: a system to bridge conversation and item-oriented web sites

Item-oriented Web sites maintain repositories of information about things such as books, games, or products. Many of these Web sites offer discussion forums. However, these forums are often disconnected from the rich data available in the item repositories. We describe a system, movie linking, that bridges a movie recommendation Web site and a movie-oriented discussion forum. Through automatic detection and an interactive component, the system recognizes references to movies in the forum and adds recommendation data to the forums and conversation threads to movie pages. An eight week observational study shows that the system was able to identify movie references with precision of .93 and recall of .78. Though users reported that the feature was useful, their behavior indicates that the feature was more successful at enriching the interface than at integrating the system.

Using Groups of Items for Preference Elicitation in Recommender Systems

To achieve high quality initial personalization, recommender systems must provide an efficient and effective process for new users to express their preferences. We propose that this goal is best served not by the classical method where users begin by expressing preferences for individual items this process is an inefficient way to convert a user’s effort into improved personalization. Rather, we propose that new users can begin by expressing their preferences for groups of items. We test this idea by designing and evaluating an interactive process where users express preferences across groups of items that are automatically generated by clustering algorithms. We contribute a strategy for recommending items based on these preferences that is generalizable to any collaborative filtering-based system. We evaluate our process with both offline simulation methods and an online user experiment. We find that, as compared with a baseline rate-15-items interface, (a) users are able to complete the preference elicitation process in less than half the time, and (b) users are more satisfied with the resulting recommended items. Our evaluation reveals several advantages and other trade-offs involved in moving from item-based preference elicitation to group-based preference elicitation.