Gustav Šourek

Learning to predict soccer results from relational data with gradient boosted trees

We describe our winning solution to the 2017’s Soccer Prediction Challenge organized in conjunction with the MLJ’s special issue on Machine Learning for Soccer. The goal of the challenge was to predict outcomes of future matches within a selected time-frame from different leagues over the world. A dataset of over 200,000 past match outcomes was provided to the contestants. We experimented with both relational and feature-based methods to learn predictive models from the provided data. We employed relevant latent variables computable from the data, namely so called pi-ratings and also a rating based on the PageRank method. A method based on manually constructed features and the gradient boosted tree algorithm performed best on both the validation set and the challenge test set. We also discuss the validity of the assumption that probability predictions on the three ordinal match outcomes should be monotone, underlying the RPS measure of prediction quality.

Learning Predictive Categories Using Lifted Relational Neural Networks

Lifted relational neural networks (LRNNs) are a flexible neural-symbolic framework based on the idea of lifted modelling. In this paper we show how LRNNs can be easily used to specify declaratively and solve learning problems in which latent categories of entities, properties and relations need to be jointly induced.

Learning to Detect Network Intrusion from a Few Labeled Events and Background Traffic

Intrusion detection systems (IDS) analyse network traffic data with the goal to reveal malicious activities and incidents. A general problem with learning within this domain is a lack of relevant ground truth data, i.e. real attacks, capturing malicious behaviors in their full variety. Most of existing solutions thus, up to a certain level, rely on rules designed by network domain experts. Although there are advantages to the use of rules, they lack the basic ability of adapting to traffic data. As a result, we propose an ensemble tree bagging classifier, capable of learning from an extremely small number of true attack representatives, and demonstrate that, incorporating a general background traffic, we are able to generalize from those few representatives to achieve competitive results to the expert designed rules used in existing IDS Camnep.

Lifted relational neural networks: Efficient learning of latent relational structures

We propose a method to combine the interpretability and expressive power of firstorder logic with the effectiveness of neural network learning. In particular, we introduce a lifted framework in which first-order rules are used to describe the structure of a given problem setting. These rules are then used as a template for constructing a number of neural networks, one for each training and testing example. As the different networks corresponding to different examples share their weights, these weights can be efficiently learned using stochastic gradient descent. Our framework provides a flexible way for implementing and combining a wide variety of modelling constructs. In particular, the use of first-order logic allows for a declarative specification of latent relational structures, which can then be efficiently discovered in a given data set using neural network learning. Experiments on 78 relational learning benchmarks clearly demonstrate the effectiveness of the framework.

Pruning Hypothesis Spaces Using Learned Domain Theories

We present a method to prune hypothesis spaces in the context of inductive logic programming. The main strategy of our method consists in removing hypotheses that are equivalent to already considered hypotheses. The distinguishing feature of our method is that we use learned domain theories to check for equivalence, in contrast to existing approaches which only prune isomorphic hypotheses. Specifically, we use such learned domain theories to saturate hypotheses and then check if these saturations are isomorphic. While conceptually simple, we experimentally show that the resulting pruning strategy can be surprisingly effective in reducing both computation time and memory consumption when searching for long clauses, compared to approaches that only consider isomorphism.

Stacked structure learning for lifted relational neural networks

Lifted Relational Neural Networks (LRNNs) describe relational domains using weighted first-order rules which act as templates for constructing feed-forward neural networks. While previous work has shown that using LRNNs can lead to state-of-the-art results in various ILP tasks, these results depended on hand-crafted rules. In this paper, we extend the framework of LRNNs with structure learning, thus enabling a fully automated learning process. Similarly to many ILP methods, our structure learning algorithm proceeds in an iterative fashion by top-down searching through the hypothesis space of all possible Horn clauses, considering the predicates that occur in the training examples as well as invented soft concepts entailed by the best weighted rules found so far. In the experiments, we demonstrate the ability to automatically induce useful hierarchical soft concepts leading to deep LRNNs with a competitive predictive power.

Dynamic system modeling of evolutionary algorithms

Evolutionary algorithms are population-based, metaheuristic, black-box optimization techniques from the wider family of evolutionary computation. Optimization algorithms within this family are often based on similar principles and routines inspired by biological evolution. Due to their robustness, the scope of their application is broad and varies from physical engineering to software design problems. Despite sharing similar principles based in common biological inspiration, these algorithms themselves are typically viewed as black-box program routines by the end user, without a deeper insight into the underlying optimization process. We believe that shedding some light into the underlying routines of evolutionary computation algorithms can make them more accessible to wider engineering public. In this paper, we formulate the evolutionary optimization process as a dynamic system simulation, and provide means to prototype evolutionary optimization routines in a visually comprehensible framework. The framework enables engineers to follow the same dynamic system modeling paradigm, they typically use for representation of their optimization problems, to also create the desired evolutionary optimizers themselves. Instantiation of the framework in a Matlab-Simulink library practically results in graphical programming of evolutionary optimizers based on data-flow principles used for dynamic system modeling within the Simulink environment. We illustrate the efficiency of visual representation in clarifying the underlying concepts on executable flow-charts of respective evolutionary optimizers and demonstrate features and potential of the framework on selected engineering benchmark applications.

Visual data-flow framework of evolutionary computation

Visual representation of information, allowing to quickly communicate and share ideas, forms an important part of scientific and engineering progress, with applications varying from physics to software design. Engineers naturally utilize graphs and flowcharts to clarify concepts and prototype their applications. Traditionally, wide variety of engineering applications from civil to control engineering can be formulated in the form of an optimization problem. For some of the most challenging optimization problems, population-based optimizers from the evolutionary computation family were proven useful in finding high quality solutions. In this paper we present a new data-flow framework to integrate these two worlds of visual representation and engineering optimization through VisualEA - a new Matlab-Simulink library for visual programming of evolutionary computation algorithms under the paradigm of dynamic systems, and demonstrate its features and potential.