Loren G. Terveen

Evaluating collaborative filtering recommender systems

Recommender systems have been evaluated in many, often incomparable, ways. In this article, we review the key decisions in evaluating collaborative filtering recommender systems: the user tasks being evaluated, the types of analysis and datasets being used, the ways in which prediction quality is measured, the evaluation of prediction attributes other than quality, and the user-based evaluation of the system as a whole. In addition to reviewing the evaluation strategies used by prior researchers, we present empirical results from the analysis of various accuracy metrics on one content domain where all the tested metrics collapsed roughly into three equivalence classes. Metrics within each equivalency class were strongly correlated, while metrics from different equivalency classes were uncorrelated.

Using Social Psychology to Motivate Contributions to Online Communities

Under-contribution is a problem for many online communities. Social psychology theories of social loafing and goal-setting can lead to mid-level design goals to address this problem. We tested design principles derived from these theories in four field experiments involving members of an online movie recommender community. In each of the experiments participated were given different explanations for the value of their contributions. As predicted by theory, individuals contributed when they were reminded of their uniqueness and when they were given specific and challenging goals. However, other predictions were disconfirmed. For example, in one experiment, participants given group goals contributed more than those given individual goals. The article ends with suggestions and challenges for mining design implications from social science theories.

PHOAKS: a system for sharing recommendations

The feasibility of automatic recognition of recommendations is supported by empirical results. First, Usenet messages are a significant source of recommendations of Web resources: 23% of Usenet messages mention Web resources, and ?>0% of these mentions are recommendations. Second, recommendation instances can be machine-recognized with nearly 90% accuracy. Third, some resources are recommended by more than one person. These multiconfirmed recommendations appear to be significant resources for the relevant community. Finally, the number of distinct recommenders of a resource is a tallying, and redistributing recom-

The dynamics of mass interaction

Usenet may be regarded as the world’s largest and fastest growing conversational application. In 1988 there were fewer than 500 newsgroups. Current estimates vary, but at the time of our data collection in December 1996, there were over 17,000 newsgroups, with approximately 3 million users worldwide (Harrison, 1994). This growth has been achieved without any centralized organization or governing body (King, 1997). The ubiquity of Usenet, and the fact that it supports conversations between hundreds or even thousands of participants, provides the opportunity to study what we term mass interaction. However, we currently lack basic data about Usenet interactions. The current paper analyses over 2.15 million messages produced by 659,450 people in 500 representative newsgroups collected over 6 months. We provide descriptive data about newsgroup demographics, communication strategies, and interactivity. We then derive predictions from the common ground model of communication to test predictions about how these parameters interact.

Creating, destroying, and restoring value in wikipedia

Wikipedias brilliance and curse is that any user can edit any of the encyclopedia entries. We introduce the notion of the impact of an edit, measured by the number of times the edited version is viewed. Using several datasets, including recent logs of all article views, we show that an overwhelming majority of the viewed words were written by frequent editors and that this majority is increasing. Similarly, using the same impact measure, we show that the probability of a typical article view being damaged is small but increasing, and we present empirically grounded classes of damage. Finally, we make policy recommendations for Wikipedia and other wikis in light of these findings.

Social matching: A framework and research agenda

Social matching systems bring people together in both physical and online spaces. They have the potential to increase social interaction and foster collaboration. However, social matching systems lack a clear intellectual foundation: the nature of the design space, the key research challenges, and the roster of appropriate methods are all ill-defined. This article begins to remedy the situation. It clarifies the scope of social matching systems by distinguishing them from other recommender systems and related systems and techniques. It identifies a set of issues that characterize the design space of social matching systems and shows how existing systems explore different points within the design space. It also reviews selected social science results that can provide input into system design. Most important, the article presents a research agenda organized around a set of claims. The claims embody our understanding of what issues are most important to investigate, our beliefs about what is most likely to be true, and our suggestions of specific research directions to pursue.

Does “authority” mean quality? predicting expert quality ratings of Web documents

For many topics, the World Wide Web contains hundreds or thousands of relevant documents of widely varying quality. Users face a daunting challenge in identifying a small subset of documents worthy of their attention. Link analysis algorithms have received much interest recently, in large part for their potential to identify high quality items. We report here on an experimental evaluation of this potential. We evaluated a number of link and content-based algorithms using a dataset of web documents rated for quality by human topic experts. Link-based metrics did a good job of picking out high-quality items. Precision at 5 is about 0.75, and precision at 10 is about 0.55; this is in a dataset where 0.32 of all documents were of high quality. Surprisingly, a simple content-based metric performed nearly as well; ranking documents by the total number of pages on their containing site.

Overview of human-computer collaboration

Abstract The paper derives a set of fundamental issues from a definition of collaboration, introduces two major approaches to human-computer collaboration, and surveys each approach, showing how it formulates and addresses the issues. It concludes by proposing some themes that should characterize a unified approach to human-computer collaboration.

Constructing, organizing, and visualizing collections of topically related Web resources

For many purposes, the Web page is too small a unit of interaction and analysis. Web sites are structured multimedia documents consisting of many pages, and users often are interested in obtaining and evaluating entire collections of topically related sites. Once such a collection is obtained, users face the challenge of exploring, comprehending and organizing the items. We report four innovations that address these user needs: (1) we replaced the Web page with the Web site as the basic unit of interaction and analysis;(2) we defined a new informationstructure, the clan graph, that groups together sets of related sites; (3) we augment the representation of a site with a site profile, information about site structure and content that helps inform user evaluation of a site; and (4) we invented a new graph visualization, the auditorium visualization, that reveals important structural and content properties of sites within a clan graph. Detailed analysis and user studies document the utility of this approach. The clan graph construction algorithm tends to filter out irrelevant sites and discover additional relevant items. The auditorium visualization, augmented with drill-down capabilities to explore site profile data, helps users to find high-quality sites as well as sites that serve a particular function.

WP:clubhouse?: an exploration of Wikipedia's gender imbalance

Wikipedia has rapidly become an invaluable destination for millions of information-seeking users. However, media reports suggest an important challenge: only a small fraction of Wikipedias legion of volunteer editors are female. In the current work, we present a scientific exploration of the gender imbalance in the English Wikipedias population of editors. We look at the nature of the imbalance itself, its effects on the quality of the encyclopedia, and several conflict-related factors that may be contributing to the gender gap. Our findings confirm the presence of a large gender gap among editors and a corresponding gender-oriented disparity in the content of Wikipedias articles. Further, we find evidence hinting at a culture that may be resistant to female participation.