Benjamin Kille

The plista dataset

Releasing datasets has fostered research in fields such as information retrieval and recommender systems. Datasets are typically tailored for specific scenarios. In this work, we present the plista dataset. The dataset contains a collection of news articles published on 13 news portals. Additionally, the dataset comprises user interactions with those articles. We inctroduce the datasets main characteristics. Further, we illustrate possible applications of the dataset.

Benchmarking News Recommendations in a Living Lab

Most user-centric studies of information access systems in literature suffer from unrealistic settings or limited numbers of users who participate in the study. In order to address this issue, the idea of a living lab has been promoted. Living labs allow us to evaluate research hypotheses using a large number of users who satisfy their information need in a real context. In this paper, we introduce a living lab on news recommendation in real time. The living lab has first been organized as News Recommendation Challenge at ACM RecSys’13 and then as campaign-style evaluation lab NEWSREEL at CLEF’14. Within this lab, researchers were asked to provide news article recommendations to millions of users in real time. Different from user studies which have been performed in a laboratory, these users are following their own agenda. Consequently, laboratory bias on their behavior can be neglected. We outline the living lab scenario and the experimental setup of the two benchmarking events. We argue that the living lab can serve as reference point for the implementation of living labs for the evaluation of information access systems.

Recommender systems challenge 2012

The Recommender System Challenge 2012 invited participants to work on two tracks with real-world datasets and to submit their contributions that would be related to specific problem contexts. First of all, it asked participants to develop new algorithms and to compare them to other algorithms in given settings; in addition, it asked participants to explore with new recommendation methods, services, as well as added-value services related to recommendation.

Stream-Based Recommendations: Online and Offline Evaluation as a Service

Providing high-quality news recommendations is a challenging task because the set of potentially relevant news items changes continuously, the relevance of news highly depends on the context, and there are tight time constraints for computing recommendations. The CLEF NewsREEL challenge is a campaign-style evaluation lab allowing participants to evaluate and optimize news recommender algorithms online and offline. In this paper, we discuss the objectives and challenges of the NewsREEL lab. We motivate the metrics used for benchmarking the recommender algorithms and explain the challenge dataset. In addition, we introduce the evaluation framework that we have developed. The framework makes possible the reproducible evaluation of recommender algorithms for stream data, taking into account recommender precision as well as the technical complexity of the recommender algorithms.

Overview of NewsREEL’16: Multi-dimensional Evaluation of Real-Time Stream-Recommendation Algorithms

Successful news recommendation requires facing the challenges of dynamic item sets, contextual item relevance, and of fulfilling non-functional requirements, such as response time. The CLEF NewsREEL challenge is a campaign-style evaluation lab allowing participants to tackle news recommendation and to optimize and evaluate their recommender algorithms both online and offline. In this paper, we summarize the objectives and challenges of NewsREEL 2016. We cover two contrasting perspectives on the challenge: that of the operator (the business providing recommendations) and that of the challenge participant (the researchers developing recommender algorithms). In the intersection of these perspectives, new insights can be gained on how to effectively evaluate real-time stream recommendation algorithms.

Personalizing tags: a folksonomy-like approach for recommending movies

Movie recommender systems attempt to find movies which are of interest for their users. However, as new movies are added, and new users join movie recommendation services, the problem of recommending suitable items becomes increasingly harder. In this paper, we present a simple way of using a priori movie data in order to improve the accuracy of collaborative filtering recommender systems. The approach decreases the sparsity of the rating matrix by inferring personal ratings on tags assigned to movies. The new tag ratings are used to find which movies to recommend. Experiments performed on data from the movie recommendation community Moviepilot show a positive effect on the quality of recommended items.

Real-time Recommendation of Streamed Data

This tutorial addressed two trending topics in the field of recommender systems research, namely A/B testing and real-time recommendations of streamed data. Focusing on the news domain, participants learned how to benchmark the performance of stream-based recommendation algorithms in a live recommender system and in a simulated environment.

KMulE: a framework for user-based comparison of recommender algorithms

Collaborative Filtering Recommender Systems come in a wide variety of variants. In this paper we present a system for visualizing and comparing recommendations provided by different collaborative recommendation algorithms. The system utilizes a set of context-aware, hybrid, and other collaborative filtering solutions in order to generate various recommendations from which its users can pick those corresponding best to their current situation (i.e. context). All user interaction is fed back to the system in order to additionally improve the quality of the recommendations. Additionally, users can explicitly ask the system to treat certain recommenders as more important than others, or disregard them completely if the list of recommended movies is not to their liking.

News Recommendation in Real-Time

Recommender systems support users facing information overload situations. Typically, such situations arise as users have to choose between an immense number of alternatives. Examples include deciding what songs to listen to, what movies to watch, and what news article to read. In this chapter, we outline the case of suggesting news articles. This task entails a number of challenges. First, news collections do not remain relevant unlike movies or songs. Users continue to request novel contents. Second, users avoid creating consistent profiles thus reject login procedures. Third, requests arrive in enormous streams. Having short consumption times, users quickly request the next article to read. Handling these challenges requires adaptations to existing recommendation strategies as well as developing novel ones.

Evaluation of Cross-Domain News Article Recommendations

This thesis will investigate methods to increase the utility of news article recommendation services. Access to different news providers allows us to consider cross-domain user preferences. We deal with recommender systems with continuously changing item collections. We will be able to observe user feedback from a real-world recommendation system operating on different domains. We will evaluate how results from existing data sets correspond to actual user reactions.