Katja Verbeeck

An intelligent hyper-heuristic framework for CHeSC 2011

The present study proposes a new selection hyper-heuristic providing several adaptive features to cope with the requirements of managing different heuristic sets. The approach suggested provides an intelligent way of selecting heuristics, determines effective heuristic pairs and adapts the parameters of certain heuristics online. In addition, an adaptive list-based threshold accepting mechanism has been developed. It enables deciding whether to accept or not the solutions generated by the selected heuristics. The resulting approach won the first Cross Domain Heuristic Search Challenge against 19 high-level algorithms.

Learning agents for the multi-mode project scheduling problem

Intelligent optimization refers to the promising technique of integrating learning mechanisms into (meta-)heuristic search. In this paper, we use multi-agent reinforcement learning for building high-quality solutions for the multi-mode resource-constrained project scheduling problem (MRCPSP). We use a network of distributed reinforcement learning agents that cooperate to jointly learn a well-performing constructive heuristic. Each agent, being responsible for one activity, uses two simple learning devices, called learning automata, that learn to select a successor activity order and a mode, respectively. By coupling the reward signals for both learning tasks, we can clearly show the advantage of using reinforcement learning in search. We present some comparative results, to show that our method can compete with the best performing algorithms for the MRCPSP, yet using only simple learning schemes without the burden of complex fine-tuning.

A new hyper-heuristic as a general problem solver: an implementation in HyFlex

This study provides a new hyper-heuristic design using a learning-based heuristic selection mechanism together with an adaptive move acceptance criterion. The selection process was supported by an online heuristic subset selection strategy. In addition, a pairwise heuristic hybridization method was designed. The motivation behind building an intelligent selection hyper-heuristic using these adaptive hyper-heuristic sub-mechanisms is to facilitate generality. Therefore, the designed hyper-heuristic was tested on a number of problem domains defined in a high-level framework, i.e., HyFlex. The framework provides a set of problems with a number of instances as well as a group of low-level heuristics. Thus, it can be considered a good environment to measure the generality level of selection hyper-heuristics. The computational results demonstrated the generic performance of the proposed strategy in comparison with other tested hyper-heuristics composed of the sub-mechanisms from the literature. Moreover, the performance and behavior analysis conducted for the hyper-heuristic clearly showed its adaptive characteristics under different search conditions. The principles comprising the here presented algorithm were at the heart of the algorithm that won the first international cross-domain heuristic search competition.

An investigation on the generality level of selection hyper-heuristics under different empirical conditions

The present study concentrates on the generality of selection hyper-heuristics across various problem domains with a focus on different heuristic sets in addition to distinct experimental limits. While most hyper-heuristic research employs the term generality in describing the potential for solving various problems, the performance changes across different domains are rarely reported. Furthermore, a hyper-heuristics performance study purely on the topic of heuristic sets is uncommon. Similarly, experimental limits are generally ignored when comparing hyper-heuristics. In order to demonstrate the effect of these generality related elements, nine heuristic sets with different improvement capabilities and sizes were generated for each of three target problem domains. These three problem domains are home care scheduling, nurse rostering and patient admission scheduling. Fourteen hyper-heuristics with varying intensification/diversification characteristics were analysed under various settings. Empirical results indicate that the performance of selection hyper-heuristics changes significantly under different experimental conditions.

Multi-agent Learning Dynamics: A Survey

In this paper we compare state-of-the-art multi-agent reinforcement learning algorithms in a wide variety of games. We consider two types of algorithms: value iteration and policy iteration. Four characteristics are studied: initial conditions, parameter settings, convergence speed, and local versus global convergence. Global convergence is still difficult to achieve in practice, despite existing theoretical guarantees. Multiple visualizations are included to provide a comprehensive insight into the learning dynamics.

The effect of the set of low-level heuristics on the performance of selection hyper-heuristics

The present study investigates the effect of heuristic sets on the performance of several selection hyper-heuristics. The performance of selection hyper-heuristics is strongly dependant on low-level heuristic sets employed for solving target problems. Therefore, the generality of hyper-heuristics should be examined across various heuristic sets. Unlike the majority of hyper-heuristics research, where the low-level heuristic set is considered given, the present study investigates the influence of the low-level heuristics on the hyper-heuristics performance. To achieve this, a number of heuristic sets was generated for the patient admission scheduling problem by setting the parameters of a set of parametric heuristics with specific values. These values were set such that nine heuristic sets with different improvement capabilities, speed characteristics and size were generated. A group of hyper-heuristics with certain selection mechanisms and acceptance criteria having dissimilar intensification/diversification abilities were taken from the literature enabling a comprehensive analysis. The experimental results indicated that different hyper-heuristics perform superiorly on distinct heuristic sets. The results can be explained and hence result in hyper-heuristic design recommendations.

A learning-based optimization approach to multi-project scheduling

The present paper introduces a learning-based optimization approach to the resource-constrained multi-project scheduling problem. Multiple projects, each with their own set of activities, need to be scheduled. The objectives dealt with here include minimization of the average project delay and total makespan. The availability of local and global resources, precedence relations between activities, and non-equal project start times have to be considered. The proposed approach relies on a simple sequence learning game played by a group of project managers. The project managers each learn their activity list locally using reinforcement learning, more specifically learning automata. Meanwhile, they learn how to choose a suitable place in the overall sequence of all activity lists. All the projects need to arrive at a unique place in this sequence. A mediator, who usually has to solve a complex optimization problem, now manages a simple dispersion game. Through the mediator, a sequence of feasible activity lists is eventually scheduled by using a serial schedule generation scheme, which is adopted from single project scheduling. It is shown that the sequence learning approach has a large positive effect on minimizing the average project delay. In fact, the combination of local reinforcement learning, the sequence learning game and a forward-backward implementation of the serial scheduler significantly improves the best known results for all the MPSPLIB datasets. In addition, several new best results were obtained for both considered objectives: minimizing the average project delay and minimizing the total makespan.

Priority awareness: towards a computational model of human fairness for multi-agent systems

Many multi-agent systems are intended to operate together with or as a service to humans. Typically, multi-agent systems are designed assuming perfectly rational, self-interested agents, according to the principles of classical game theory. However, research in the field of behavioral economics shows that humans are not purely self-interested; they strongly care about whether their rewards are fair. Therefore, multi-agent systems that fail to take fairness into account, may not be sufficiently aligned with human expectations and may not reach intended goals. Two important motivations for fairness have already been identified and modelled, being (i) inequity aversion and (ii) reciprocity. We identify a third motivation that has not yet been captured: priority awareness.We show how priorities may be modelled and discuss their relevance for multi-agent research.

Optimal convergence in multi-agent MDPs

Learning Automata (LA) were recently shown to be valuable tools for designing Multi-Agent Reinforcement Learning algorithms. One of the principal contributions of LA theory is that a set of decentralized, independent learning automata is able to control a finite Markov Chain with unknown transition probabilities and rewards. We extend this result to the framework of Multi-Agent MDPs, a straightforward extension of single-agent MDPs to distributed cooperative multi-agent decision problems. Furthermore, we combine this result with the application of parametrized learning automata yielding global optimal convergence results.

Solving multi-stage games with hierarchical learning automata that bootstrap

A solderless electrical connector comprising a printed circuit board and electrical contact terminals mounted therein. A printed circuit board is fabricated to the construction stage wherein conductive circuitry interconnects plated through holes formed therethrough. The exposed circuitry and plated through holes are covered with a conformal coating of insulative material which seals the outer surfaces of the conductive materials from the environment and its effects. The insulative coating obviates the need for solder to protect the conductive materials as well as precludes the use of solder for electrical interconnection therewith. The plated through holes are constructed for press fit insertion of contact terminals in tight frictional engagement therein. The press fit insertion of a contact having an angular edge portion effectively penetrates the insulative coating by deforming it away from that portion of the plated through hole brought to bear against the contact. A mechanical and electrical interconnection is thereby provided between the plated through hole and contact terminal of the press fit assembly while the utilization of solder and the requisite heat thereof is effectively eliminated.