Tom Croonenborghs

Transfer Learning in Reinforcement Learning Problems Through Partial Policy Recycling

We investigate the relation between transfer learning in reinforcement learning with function approximation and supervised learning with concept drift. We present a new incremental relational regression tree algorithm that is capable of dealing with concept drift through tree restructuring and show that it enables a Q-learner to transfer knowledge from one task to another by recycling those parts of the generalized Q-function that still hold interesting information for the new task. We illustrate the performance of the algorithm in several experiments.

Accelerometry-Based Home Monitoring for Detection of Nocturnal Hypermotor Seizures Based on Novelty Detection

Nocturnal home monitoring of epileptic children is often not feasible due to the cumbersome manner of seizure monitoring with the standard method of video/EEG-monitoring. We propose a method for hypermotor seizure detection based on accelerometers attached to the extremities. From the acceleration signals, multiple temporal, frequency, and wavelet-based features are extracted. After determining the features with the highest discriminative power, we classify movement events in epileptic and nonepileptic movements. This classification is only based on a nonparametric estimate of the probability density function of normal movements. Such approach allows us to build patient-specific models to classify movement data without the need for seizure data that are rarely available. If, in the test phase, the probability of a data point (event) is lower than a threshold, this event is considered to be an epileptic seizure; otherwise, it is considered as a normal nocturnal movement event. The mean performance over seven patients gives a sensitivity of 95.24% and a positive predictive value of 60.04%. However, there is a noticeable interpatient difference.

Generalized ordering-search for learning directed probabilistic logical models

Recently, there has been an increasing interest in directed probabilistic logical models and a variety of languages for describing such models has been proposed. Although many authors provide high-level arguments to show that in principle models in their language can be learned from data, most of the proposed learning algorithms have not yet been studied in detail. We introduce an algorithm, generalized ordering-search, to learn both structure and conditional probability distributions (CPDs) of directed probabilistic logical models. The algorithm upgrades the ordering-search algorithm for Bayesian networks. We use relational probability trees as a representation for the CPDs. We present experiments on blocks world domains, a gene domain and the Cora dataset.

Detecting rare events using extreme value statistics applied to epileptic convulsions in children

OBJECTIVE Nocturnal home monitoring of epileptic children is often not feasible due to the cumbersome manner of seizure detection with the standard method of video electroencephalography monitoring. The goal of this paper is to propose a method for hypermotor seizure detection based on accelerometers that are attached to the extremities. METHODS Supervised methods that are commonly used in literature need annotation of data and hence require expert (neurologist) interaction resulting in a substantial cost. In this paper an unsupervised method is proposed that uses extreme value statistics and seizure detection based on a model of normal behavior that is estimated using all recorded and unlabeled data. In this way the expensive interaction can be avoided. RESULTS When applying this method to a labeled dataset, acquired from 7 patients, all hypermotor seizures are detected in 5 of the 7 patients with an average positive predictive value (PPV) of 53%. For evaluating the performance on an unlabeled dataset, seizure events are presented to the system as normal movement events. Since hypermotor seizures are rare compared to normal movements, the very few abnormal events have a negligible effect on the quality of the model. In this way, it was possible to evaluate the system for 3 of the 7 patients when 3% of the training set was composed of seizure events. This resulted in sensitivity scores of 80%, 22% and 90% and a PPV of 89%, 21% and 44% respectively. These scores are comparable with a state-of-the-art supervised machine learning based approach which requires a labeled dataset. CONCLUSIONS A person-dependent epileptic seizure detection method has been designed that requires little human interaction. In contrast to traditional machine learning approaches, the imbalance of the dataset does not cause substantial difficulties.

Multi-agent relational reinforcement learning

In this paper we report on using a relational state space in multi-agent reinforcement learning. There is growing evidence in the Reinforcement Learning research community that a relational representation of the state space has many benefits over a propositional one. Complex tasks as planning or information retrieval on the web can be represented more naturally in relational form. Yet, this relational structure has not been exploited for multi-agent reinforcement learning tasks and has only been studied in a single agent context so far. In this paper we explore the powerful possibilities of using Relational Reinforcement Learning (RRL) in complex multi-agent coordination tasks. More precisely, we consider an abstract multi-state coordination problem, which can be considered as a variation and extension of repeated stateless Dispersion Games. Our approach shows that RRL allows to represent a complex state space in a multi-agent environment more compactly and allows for fast convergence of learning agents. Moreover, with this technique, agents are able to make complex interactive models (in the sense of learning from an expert), to predict what other agents will do and generalize over this model. This enables to solve complex multi-agent planning tasks, in which agents need to be adaptive and learn, with more powerful tools.

Semi-automated Video-based In-home Fall Risk Assessment

The development of an in-home fall risk assessment tool is under in- vestigation. Several fall risk screening tests such as the Timed-Get-Up-and-Go-test (TGUG) only provide a snapshot taken at a given time and place, where automated in-home fall risk assessment tools can assess the fall risk of a person on a contin- uous basis. During this study we monitored four older people in their own home for a period of three months and automatically assessed fall risk parameters. We selected a subset of fixed walking sequences from the resulting real-life video for analysis of the time needed to perform these sequences. The results show a sig- nificant diurnal and health-related variance in the time needed to cross the same distance. These results also suggest that trends in the transfer time can be detected with the presented system.

Bridging the gap between real-life data and simulated data by providing a highly realistic fall dataset for evaluating camera-based fall detection algorithms

Fall incidents are an important health hazard for older adults. Automatic fall detection systems can reduce the consequences of a fall incident by assuring that timely aid is given. The development of these systems is therefore getting a lot of research attention. Real-life data which can help evaluate the results of this research is however sparse. Moreover, research groups that have this type of data are not at liberty to share it. Most research groups thus use simulated datasets. These simulation datasets, however, often do not incorporate the challenges the fall detection system will face when implemented in real-life. In this Letter, a more realistic simulation dataset is presented to fill this gap between real-life data and currently available datasets. It was recorded while re-enacting real-life falls recorded during previous studies. It incorporates the challenges faced by fall detection algorithms in real life. A fall detection algorithm from Debard et al. was evaluated on this dataset. This evaluation showed that the dataset possesses extra challenges compared with other publicly available datasets. In this Letter, the dataset is discussed as well as the results of this preliminary evaluation of the fall detection algorithm. The dataset can be downloaded from www.kuleuven.be/advise/datasets.

Validation of the kinect for gait analysis using the GAITRite walkway

Accurate, non-intrusive and straightforward techniques for gait quality analysis can provide important information concerning the fall risk of a person. For this purpose an algorithm was developed which can measure step length and step time using the Kinect depth image. The validity of the measured step length and time is determined using the GAITRite walkway as a ground truth. The results of this validation confirm that the Kinect is well-suited for determining general parameters of a walking sequence (a Spearmans Correlation Coefficient (SCC) of 0.94 for average step length and 0.75 for average step time per walk), but we furthermore show that determining accurate results for single steps is more difficult (SCC of 0.74 for step length and 0.43 for step time for each step), making it harder to measure more complex gait parameters such as e.g. gait symmetry.

Automatic Fall Risk Estimation using the Nintendo Wii Balance Board

In this paper, a tool to assess a person´s fall risk with the Nintendo Wii Balance Board based on Center of Pressure (CoP) recordings is presented. Support Vector Machine and K-Nearest Neighbours classifiers are used to distinguish between people who experienced a fall in the past twelve months and those who have not. The classifiers are trained using data recorded from 39 people containing a mix of students and elderly. Validation is done using 10-fold cross-validation and the classifiers are also validated against additional data recorded from 12 elderly. A cross-validated average accuracy of 96.49% +/- 4.02 is achieved with the SVM classifier with radial basis function kernel and 95.72% +/- 1.48 is achieved with the KNN classifier with k=4. Validation against the additional dataset of 12 elderly results in a maximum accuracy of 76.6% with the linear SVM.

Automated in-home gait transfer time analysis using video cameras

Previous studies have shown that gait speed is an important measure of functional ability in the elderly. Continuous monitoring of the gait speed of older adults in their home environment may therefore allow the detection of changes in gait speed which could be predictive of health changes of the monitored person. In this study, a system consisting of multiple wall-mounted cameras that can automatically measure the time an older adult needs to cross a predefined transfer zone in the home environment is presented. The purpose of this study is the preliminary validation of the algorithm of the camera system which consists of several preprocessing steps and the automatic measurement of the transfer times. This validation is done through data collection in the homes of four older adults for periods varying from eight to twelve weeks. Trends in the measured transfer times are visualised and subsequently compared with the results of clinical assessments obtained during the acquisition period such as Timed-Get-Up-and-Go tests. The results indicate that it is possible to identify long-term trends in transfer times which can be indicative of adverse health-related events.