Maarten van Someren

A Bias-Variance Analysis of a Real World Learning Problem: The CoIL Challenge 2000

The CoIL Challenge 2000 data mining competition attracted a wide variety of solutions, both in terms of approaches and performance. The goal of the competition was to predict who would be interested in buying a specific insurance product and to explain why people would buy. Unlike in most other competitions, the majority of participants provided a report describing the path to their solution. In this article we use the framework of bias-variance decomposition of error to analyze what caused the wide range of prediction performance. We characterize the challenge problem to make it comparable to other problems and evaluate why certain methods work or not. We also include an evaluation of the submitted explanations by a marketing expert. We find that variance is the key component of error for this problem. Participants use various strategies in data preparation and model development that reduce variance error, such as feature selection and the use of simple, robust and low variance learners like Naive Bayes. Adding constructed features, modeling with complex, weak bias learners and extensive fine tuning by the participants often increase the variance error.The CoIL Challenge 2000 data mining competition attracted a wide variety of solutions, both in terms of approaches and performance. The goal of the competition was to predict who would be interested in buying a specific insurance product and to explain why people would buy. Unlike in most other competitions, the majority of participants provided a report describing the path to their solution. In this article we use the framework of bias-variance decomposition of error to analyze what caused the wide range of prediction performance. We characterize the challenge problem to make it comparable to other problems and evaluate why certain methods work or not. We also include an evaluation of the submitted explanations by a marketing expert. We find that variance is the key component of error for this problem. Participants use various strategies in data preparation and model development that reduce variance error, such as feature selection and the use of simple, robust and low variance learners like Naive Bayes. Adding constructed features, modeling with complex, weak bias learners and extensive fine tuning by the participants often increase the variance error.

Modeling developmental transitions on the balance scale task

Periods of relatively stable, rule-like behavior alternated with short transition periods characterize cognitive development on reasoning tasks like the balance scale task. Each transition gives rise to an improvement in behavior, until a phase is reached in which performance is flawless or improvement is not worthwhile given the necessary extra effort. Several computational models have been developed to capture the developmental phenomena associated with the balance scale task. These models, which originate from different computational traditions, explain the main phenomena of development. Recently, empirical phenomena have been reported that these models cannot easily accommodate. We propose a computational model that is implemented in ACT-R and that is based on the evaluation of success of applied knowledge, combined with a mechanism to construct new knowledge by searching for differences between the left- and right-hand sides of presented balance scale problems. This model accounts for the main empirical phenomena as well as for the recently reported empirical phenomena such as learning without feedback.

A Roadmap for Web Mining: From Web to Semantic Web

The purpose of Web mining is to develop methods and systems for discovering models of objects and processes on the World Wide Web and for web-based systems that show adaptive performance. Web Mining integrates three parent areas: Data Mining (we use this term here also for the closely related areas of Machine Learning and Knowledge Discovery), Internet technology and World Wide Web, and for the more recent Semantic Web. The World Wide Web has made an enormous amount of information electronically accessible. The use of email, news and markup languages like HTML allow users to publish and read documents at a world-wide scale and to communicate via chat connections, including information in the form of images and voice records. The HTTP protocol that enables access to documents over the network via Web browsers created an immense improvement in communication and access to information. For some years these possibilities were used mostly in the scientific world but recent years have seen an immense growth in popularity, supported by the wide availability of computers and broadband communication. The use of the internet for other tasks than finding information and direct communication is increasing, as can be seen from the interest in “e-activities” such as e-commerce, e-learning, e-government, e-science.

Machine learning for vessel trajectories using compression, alignments and domain knowledge

In this paper we present a machine learning framework to analyze moving object trajectories from maritime vessels. Within this framework we perform the tasks of clustering, classification and outlier detection with vessel trajectory data. First, we apply a piecewise linear segmentation method to the trajectories to compress them. We adapt an existing technique to better retain stop and move information and show the better performance of our method with experimental results. Second, we use a similarity based approach to perform the clustering, classification and outlier detection tasks using kernel methods. We present experiments that investigate different alignment kernels and the effect of piecewise linear segmentation in the three different tasks. The experimental results show that compression does not negatively impact task performance and greatly reduces computation time for the alignment kernels. Finally, the alignment kernels allow for easy integration of geographical domain knowledge. In experiments we show that this added domain knowledge enhances performance in the clustering and classification tasks.

Combining ship trajectories and semantics with the simple event model (SEM)

Bridging the gap between low-level features and semantics is a problem commonly acknowledged in the Multimedia community. Event modeling can fill the gap. In this paper we present the Simple Event Model (SEM) and its application in a Maritime Safety and Security use case about Situational Awareness. We show how we abstract over low-level features, recognize simple behavior events using a Piecewise Linear Segmentation algorithm, and model the events as instances of SEM. We apply deduction rules, spatial proximity reasoning, and semantic web reasoning in SWI-Prolog to derive abstract events from the recognized simple events. The use case described in this paper come from the Dutch Poseidon project.

Prolog programming techniques

In this paper we introduce the concept of a Prolog programming technique. This concept is then distinguished both from that of an algorithm and that of a programming cliché. We give examples and show how a knowledge of them can be useful in both programming environments and in teaching programming skills. The extraction of the various techniques is outlined. Finally, we discuss the problem of representing techniques where we conclude that the most promising approach is the development of a suitable meta-language.

GAMYGDALA: An Emotion Engine for Games

In this paper we present GAMYGDALA, an emotional appraisal engine that enables game developers to easily add emotions to their Non-Player Characters (NPC). Our approach proposes a solution that is positioned between event coding of affect, where individual events have predetermined annotated emotional consequences for NPCs, and a full blown cognitive appraisal model. Instead, for an NPC that needs emotions the game developer defines goals and annotates game events with a relation to these goals. Based on this input, GAMYGDALA produces an emotion for that NPC according to the well-known OCC model. In this paper we provide evidence for the following: GAMYGDALA provides black-box Game-AI independent emotion support, is efficient for large numbers of NPCs, and is psychologically grounded.

Abstracting and reasoning over ship trajectories and web data with the Simple Event Model (SEM)

Bridging the gap between low-level features and semantics is a problem commonly acknowledged in the Multimedia community. Event modeling can fill this gap by representing knowledge about the data at different level of abstraction. In this paper we present the Simple Event Model (SEM) and its application in a Maritime Safety and Security use case about Situational Awareness, where the data also come as low-level features (of ship trajectories). We show how we abstract over these low-level features, recognize simple behavior events using a Piecewise Linear Segmentation algorithm, and model the resulting events as instances of SEM. We aggregate web data from different sources, apply deduction rules, spatial proximity reasoning, and semantic web reasoning in SWI-Prolog to derive abstract events from the recognized simple events. The use case described in this paper comes from the Dutch Poseidon project.

Model class selection and construction: beyond the Procrustean approach to machine learning applications

Machine Learning was primarily inspired by human learning. In a branch of Artificial Intelligence scientists tried to build systems that reproduce forms of human learning. Currently the methods that were discovered in this way have been elaborated and are applied to tasks that are not performed by humans at all. For example, one of the most popular applications is the analysis of consumer data to predict buying behaviour. This has not traditionally been viewed as an interesting form of human intelligence.

Navigation behavior models for link structure optimization

Analysis of existing methods for automatic optimization of link structures shows that these methods rely heavily on assumptions about the preferences and navigation behavior of users. Authors often do not state these assumptions explicitly and do not evaluate whether the assumptions are consistent with the actual behavior of the users of the site. This is a serious deficiency as experiments with simulated users show that incorrect assumptions can easily lead to inefficient link structures. In this work we present a framework that gives a systematic overview of alternative assumptions. On the basis of the framework we can select a set of assumptions that best matches the navigation behavior of the users in the site’s log files. We also present a method for optimizing hierarchical navigation menus on the basis of the selected assumptions. This method can be used interactively under full control of a web master. The system proposes modifications of the structure and explains why these modifications lead to more efficient menus. Evaluation by means of a case study shows that the modifications that are proposed effectively reduce the expected navigation time while preserving the coherence of the menu structure.