Philippe Preux

Sparse Temporal Difference Learning Using LASSO

We consider the problem of on-line value function estimation in reinforcement learning. We concentrate on the function approximator to use. To try to break the curse of dimensionality, we focus on non parametric function approximators. We propose to fit the use of kernels into the temporal difference algorithms by using regression via the LASSO. We introduce the equi-gradient descent algorithm (EGD) which is a direct adaptation of the one recently introduced in the LARS algorithm family for solving the LASSO. We advocate our choice of the EGD as a judicious algorithm for these tasks. We present the EGD algorithm in details as well as some experimental results. We insist on the qualities of the EGD for reinforcement learning.

Fitness Landscapes and Performance of Meta-Heuristics

We perform a statistical analysis of the structure of the search space of some planar, euclidian instances of the traveling salesman problem. We want to depict this structure from the point of view of iterated local search algorithms. The objective is two-fold: understanding the experimentally known good performance of metaheuristics on the TSP and other combinatorial optimization problems; designing new techniques to search the space more efficiently. This work actually led us to design a hybrid genetic algorithm that competes rather well with other local search heuristics for the TSP, notably Junger et al.’s version of ILK. This work also opens promising horizons to the study of other combinatorial optimization problems such as the quadratic assignment problem.

A generic architecture for adaptive agents based on reinforcement learning

In this paper, we present MAABAC, a generic model for building adaptive agents: they learn new behaviors by interacting with their environment. These agents adapt their behavior by way of reinforcement learning, namely temporal difference methods. MAABAC is presented in its generality and then, different instantiations of the generic model are presented and experiments are reported. These experiments show the strength of this way of learning.

“Virtual laboratory environment” (VLE): a software environment oriented agent and object for modeling and simulation of complex systems

A method of monitoring the status of a current ovulation cycle of an individual human female subject, involving testing of the body fluid concentration of an analyte of significance in relation to the status of the ovulation cycle, such as urinary E3G, during at least part of the pre-ovulation phase of the current ovulation cycle of the individual subject, and identification from the results of such testing an analyte concentration change indicative of imminent ovulation, relative to an analyte concentration reference value that has been adapted to the individual human subject on the basis of analyte concentration test data obtained from the individual human subject during one or more previous ovulation cycles.

Scale transfer modeling: using emergent computation for coupling an ordinary differential equation system with a reactive agent model

This article deals with the coupling of analytical models with individual based models design with the reactive agents paradigm. Such a coupling of models of different natures is motivated by the need to find a way to model scale transfer in large complex systems, i.e. to model how low level of organization can be made to influence upper level and vice versa. This is a fundamental issue, and more particularly in ecological modeling where models are a real scientific tool of investigation. Individuals and populations are not described at the same scale of time and space but it is known that they act on each others. Based on this example, we model individuals in their environment and the population dynamics. While behavior is best modeled using an algorithmic framework (the reactive agent paradigm), population dynamics (because of the number of interacting entities) is best modeled using numerical models. We propose the use of the concept of emergent computation as a framework for coupling heterogeneous formalisms. In the same time, it is crucial to be aware of the consequences of the simplifications and of the choices that are made in the reactive agent model, such as the topology of space and various parameters. In this article, we discuss these issues and our approach on a case study drawn from marine ecology and we show that it is possible to find classical mathematical functional responses with a reactive agent system. Then, we propose a methodology to deal with the coupling of heterogeneous formalism useful in any kind of system modeling.

A Bit-Wise Epistasis Measure for Binary Search Spaces

The epistatic variance has been introduced by Davidor as a tool for the evaluation of interdependences between genes, thus possibly giving clues about the difficulty of optimizing functions with genetic algorithms (GAs). Despite its theoretical grounding in Walsh function analysis, several studies have shown its weakness as a predictor of GAs results. In this paper, we focus on binary search spaces and propose to measure epistatic effect on the level of individual genes, an approach that we call bit-wise epistasis. We give examples of this measure on several well-known test problems, then we take into account this supplementary information to improve the performances of evolutionary algorithms. We conclude by pointing towards possible extensions of this concept to real size problems.

Sequential approaches for learning datum-wise sparse representations

In supervised classification, data representation is usually considered at the dataset level: one looks for the “best” representation of data assuming it to be the same for all the data in the data space. We propose a different approach where the representations used for classification are tailored to each datum in the data space. One immediate goal is to obtain sparse datum-wise representations: our approach learns to build a representation specific to each datum that contains only a small subset of the features, thus allowing classification to be fast and efficient. This representation is obtained by way of a sequential decision process that sequentially chooses which features to acquire before classifying a particular point; this process is learned through algorithms based on Reinforcement Learning.The proposed method performs well on an ensemble of medium-sized sparse classification problems. It offers an alternative to global sparsity approaches, and is a natural framework for sequential classification problems. The method extends easily to a whole family of sparsity-related problem which would otherwise require developing specific solutions. This is the case in particular for cost-sensitive and limited-budget classification, where feature acquisition is costly and is often performed sequentially. Finally, our approach can handle non-differentiable loss functions or combinatorial optimization encountered in more complex feature selection problems.

Climbing Up NP-Hard Hills

Evolutionary algorithms are sophisticated hill-climbers. In this paper, we discuss the ability of this class of local search algorithms to provide useful and efficient heuristics to solve NP-hard problems. Our discussion is illustrated on experiments aiming at solving the job-shop-scheduling problem. We focus on the components of the EA, pointing out the importance of the objective function as well as the manner the operators are applied. Experiments clearly show the efficiency of local search methods in this context, the trade-off between “pure” and hybrid algorithms, as well as the very good performance obtained by simple hill-climbing algorithms. This work has to be regarded as a step towards a better understanding of the way search algorithms wander in a fitness landscape.

Datum-wise classification: a sequential approach to sparsity

We propose a novel classification technique whose aim is to select an appropriate representation for each datapoint, in contrast to the usual approach of selecting a representation encompassing the whole dataset. This datum-wise representation is found by using a sparsity inducing empirical risk, which is a relaxation of the standard L0 regularized risk. The classification problem is modeled as a sequential decision process that sequentially chooses, for each datapoint, which features to use before classifying. Datum-Wise Classification extends naturally to multi-class tasks, and we describe a specific case where our inference has equivalent complexity to a traditional linear classifier, while still using a variable number of features. We compare our classifier to classical L1 regularized linear models (L1-SVM and LARS) on a set of common binary and multi-class datasets and show that for an equal average number of features used we can get improved performance using our method.

Basis Expansion in Natural Actor Critic Methods

In reinforcement learning, the aim of the agent is to find a policy that maximizes its expected return. Policy gradient methods try to accomplish this goal by directly approximating the policy using a parametric function approximator; the expected return of the current policy is estimated and its parameters are updated by steepest ascent in the direction of the gradient of the expected return with respect to the policy parameters. In general, the policy is defined in terms of a set of basis functions that capture important features of the problem. Since the quality of the resulting policies directly depend on the set of basis functions, and defining them gets harder as the complexity of the problem increases, it is important to be able to find them automatically. In this paper, we propose a new approach which uses cascade-correlation learning architecture for automatically constructing a set of basis functions within the context of Natural Actor-Critic (NAC) algorithms. Such basis functions allow more complex policies be represented, and consequently improve the performance of the resulting policies. We also present the effectiveness of the method empirically.