Lukas Lerche

What Recommenders Recommend – An Analysis of Accuracy, Popularity, and Sales Diversity Effects

In academic studies, the evaluation of recommender system (RS) algorithms is often limited to offline experimental designs based on historical data sets and metrics from the fields of Machine Learning or Information Retrieval. In real-world settings, however, other business-oriented metrics such as click-through-rates, customer retention or effects on the sales spectrum might be the true evaluation criteria for RS effectiveness. In this paper, we compare different RS algorithms with respect to their tendency of focusing on certain parts of the product spectrum. Our first analysis on different data sets shows that some algorithms – while able to generate highly accurate predictions – concentrate their top 10 recommendations on a very small fraction of the product catalog or have a strong bias to recommending only relatively popular items than others. We see our work as a further step toward multiple-metric offline evaluation and to help service providers make better-informed decisions when looking for a recommendation strategy that is in line with the overall goals of the recommendation service.

What recommenders recommend: an analysis of recommendation biases and possible countermeasures

Most real-world recommender systems are deployed in a commercial context or designed to represent a value-adding service, e.g., on shopping or Social Web platforms, and typical success indicators for such systems include conversion rates, customer loyalty or sales numbers. In academic research, in contrast, the evaluation and comparison of different recommendation algorithms is mostly based on offline experimental designs and accuracy or rank measures which are used as proxies to assess an algorithm’s recommendation quality. In this paper, we show that popular recommendation techniques—despite often being similar when compared with the help of accuracy measures—can be quite different with respect to which items they recommend. We report the results of an in-depth analysis in which we compare several recommendations strategies from different perspectives, including accuracy, catalog coverage and their bias to recommend popular items. Our analyses reveal that some recent techniques that perform well with respect to accuracy measures focus their recommendations on a tiny fraction of the item spectrum or recommend mostly top sellers. We analyze the reasons for some of these biases in terms of algorithmic design and parameterization and show how the characteristics of the recommendations can be altered by hyperparameter tuning. Finally, we propose two novel algorithmic schemes to counter these popularity biases.

Using graded implicit feedback for bayesian personalized ranking

In many application domains of recommender systems, explicit rating information is sparse or non-existent. The preferences of the current user have therefore to be approximated by interpreting his or her behavior, i.e., the implicit user feedback. In the literature, a number of algorithm proposals have been made that rely solely on such implicit feedback, among them Bayesian Personalized Ranking (BPR). In the BPR approach, pairwise comparisons between the items are made in the training phase and an item i is considered to be preferred over item j if the user interacted in some form with i but not with j. In real-world applications, however, implicit feedback is not necessarily limited to such binary decisions as there are, e.g., different types of user actions like item views, cart or purchase actions and there can exist several actions for an item over time. In this paper we show how BPR can be extended to deal with such more fine-granular, graded preference relations. An empirical analysis shows that this extension can help to measurably increase the predictive accuracy of BPR on realistic e-commerce datasets.

Beyond "Hitting the Hits": Generating Coherent Music Playlist Continuations with the Right Tracks

Automated playlist generation is a special form of music recommendation and a common feature of digital music playing applications. A particular challenge of the task is that the recommended items should not only match the general listeners preference but should also be coherent with the most recently played tracks. In this work, we propose a novel algorithmic approach and optimization scheme to generate playlist continuations that address these requirements. In our approach, we first use collections of shared music playlists, music metadata, and user preferences to select suitable tracks with high accuracy. Next, we apply a generic re-ranking optimization scheme to generate playlist continuations that match the characteristics of the last played tracks. An empirical evaluation on three collections of shared playlists shows that the combination of different input signals helps to achieve high accuracy during track selection and that the re-ranking technique can both help to balance different quality optimization goals and to further increase accuracy.

Session-based item recommendation in e-commerce: on short-term intents, reminders, trends and discounts

Many e-commerce sites present additional item recommendations to their visitors while they navigate the site, and ample evidence exists that such recommendations are valuable for both customers and providers. Academic research often focuses on the capability of recommender systems to help users discover items they presumably do not know yet and which match their long-term preference profiles. In reality, however, recommendations can be helpful for customers also for other reasons, for example, when they remind them of items they were recently interested in or when they point site visitors to items that are currently discounted. In this work, we first adopt a systematic statistical approach to analyze what makes recommendations effective in practice and then propose ways of operationalizing these insights into novel recommendation algorithms. Our data analysis is based on log data of a large e-commerce site. It shows that various factors should be considered in parallel when selecting items for recommendation, including their match with the customer’s shopping interests in the previous sessions, the general popularity of the items in the last few days, as well as information about discounts. Based on these analyses we propose a novel algorithm that combines a neighborhood-based scheme with a deep neural network to predict the relevance of items for a given shopping session.

Recommending Based on Implicit Feedback

Recommender systems have shown to be valuable tools for filtering, ranking, and discovery in a variety of application domains such as e-commerce, media repositories or document-based information in general that includes the various scenarios of Social Information Access discussed in this book. One key to the success of such systems lies in the precise acquisition or estimation of the user’s preferences. While general recommender systems research often relies on the existence of explicit preference statements for personalization, such information is often very sparse or unavailable in real-world applications. Information that allows us to assess the relevance of certain items indirectly through a user’s actions and behavior (implicit feedback) is in contrast often available in abundance. In this chapter we categorize different types of implicit feedback and review their use in the context of recommender systems and Social Information Access applications. We then extend the categorization scheme to be suitable to recent application domains. Finally, we present state-of-the-art algorithmic approaches, discuss challenges when using implicit feedback signals in particular with respect to popularity biases, and discuss selected recent works from the literature.

Efficient optimization of multiple recommendation quality factors according to individual user tendencies

An efficient post-processing scheme for recommendation lists is proposed.It can adjust quality factors, like diversity, of a list to match user tendencies.Compromises on accuracy are kept low.The method is compared with other post-processing algorithms from the literature.It can be used to build novel fine-grained personalization approaches. Recommender systems are among the most visible applications of intelligent systems technology in practice and are used to help users find items of interest, for example on e-commerce sites, in a personalized way. While past research has focused mainly on accurately predicting the relevance of items that are unknown to the user, other quality criteria for recommendations have been investigated in recent years, including diversity, novelty, or serendipity. Considering these additional factors, however, often leads to the following two challenges. First, in many application domains, trade-offs like diversity vs.accuracy have to be balanced. Second, it is not always clear how much diversity or novelty is desirable in practice.In this work, we propose a novel parameterizable optimization scheme that re-ranks accuracy-optimized recommendation lists in order to cope with these challenges. Our method is both capable of considering multiple optimization goals at the same time and designed to consider individual user tendencies regarding the different quality factors, like diversity. In contrast to previous work, the method is not restricted to a specific underlying item ranking algorithm and its generic design allows the algorithm to be parameterized according to the requirements of the application domain. Experimental evaluations with different datasets show that balancing the quality factors with our method can be done with a marginal or no loss in ranking accuracy. Given that our method can be applied in various domains and within the narrow time constraints of online recommendation, our work opens new opportunities to design novel finer-grained personalization approaches in practical applications.

Item Familiarity as a Possible Confounding Factor in User-Centric Recommender Systems Evaluation

Abstract User studies play an important role in academic research in the field of recommender systems as they allow us to assess quality factors other than the predictive accuracy of the underlying algorithms. User satisfaction is one such factor that is often evaluated in laboratory settings and in many experimental designs one task of the participants is to assess the suitability of the system-generated recommendations. The effort required by the user to make such an assessment can, however, depend on the user’s familiarity with the presented items and directly impact on the reported user satisfaction. In this paper, we report the results of a preliminary recommender systems user study using Mechanical Turk, which indicates that item familiarity is strongly correlated with overall satisfaction.

Adaptive Recommendation-based Modeling Support for Data Analysis Workflows

RapidMiner is a software framework for the development and execution of data analysis workflows. Like many modern software development environments, the tool comprises a visual editor which allows the user to design processes on a conceptual level, thereby abstracts technical details, and thus helps the user focus on the core modeling task. The large set of pre-implemented data analysis operations available in the framework, as well as their logical dependencies, can, however, be overwhelming in particular for novice users. In this work we present an intelligent add-on to the RapidMiner framework that supports the user during the modeling phase by recommending additional operations to insert into the currently developed data analysis workflow. In the paper, we first propose different recommendation techniques and evaluate them in an offline setting using a pool of several thousand existing workflows. Second, we present the results of a laboratory study, which show that our tool helps users to significantly increase the efficiency of the modeling process.

Supporting the Design of Machine Learning Workflows with a Recommendation System

Machine learning and data analytics tasks in practice require several consecutive processing steps. RapidMiner is a widely used software tool for the development and execution of such analytics workflows. Unlike many other algorithm toolkits, it comprises a visual editor that allows the user to design processes on a conceptual level. This conceptual and visual approach helps the user to abstract from the technical details during the development phase and to retain a focus on the core modeling task. The large set of preimplemented data analysis and machine learning operations available in the tool, as well as their logical dependencies, can, however, be overwhelming in particular for novice users. In this work, we present an add-on to the RapidMiner framework that supports the user during the modeling phase by recommending additional operations to insert into the currently developed machine learning workflow. First, we propose different recommendation techniques and evaluate them in an offline setting using a pool of several thousand existing workflows. Second, we present the results of a laboratory study, which show that our tool helps users to significantly increase the efficiency of the modeling process. Finally, we report on analyses using data that were collected during the real-world deployment of the plug-in component and compare the results of the live deployment of the tool with the results obtained through an offline analysis and a replay simulation.