Elia Liitiäinen

Non-parametric residual variance estimation in supervised learning

The residual variance estimation problem is well-known in statistics and machine learning with many applications for example in the field of nonlinear modelling. In this paper, we show that the problem can be formulated in a general supervised learning context. Emphasis is on two widely used non-parametric techniques known as the Delta test and the Gamma test. Under some regularity assumptions, a novel proof of convergence of the two estimators is formulated and subsequently verified and compared on two meaningful study cases.

On Nonparametric Residual Variance Estimation

In this paper, the problem of residual variance estimation is examined. The problem is analyzed in a general setting which covers non-additive heteroscedastic noise under non-iid sampling. To address the estimation problem, we suggest a method based on nearest neighbor graphs and we discuss its convergence properties under the assumption of a Hölder continuous regression function. The universality of the estimator makes it an ideal tool in problems with only little prior knowledge available.

Residual variance estimation in machine learning

The problem of residual variance estimation consists of estimating the best possible generalization error obtainable by any model based on a finite sample of data. Even though it is a natural generalization of linear correlation, residual variance estimation in its general form has attracted relatively little attention in machine learning. In this paper, we examine four different residual variance estimators and analyze their properties both theoretically and experimentally to understand better their applicability in machine learning problems. The theoretical treatment differs from previous work by being based on a general formulation of the problem covering also heteroscedastic noise in contrary to previous work, which concentrates on homoscedastic and additive noise. In the second part of the paper, we demonstrate practical applications in input and model structure selection. The experimental results show that using residual variance estimators in these tasks gives good results often with a reduced computational complexity, while the nearest neighbor estimators are simple and easy to implement.

Bounds on the mean power-weighted nearest neighbour distance

In this paper, bounds on the mean power-weighted nearest neighbour distance are derived. Previous work concentrates mainly on the infinite sample limit, whereas our bounds hold for any sample size. The results are expected to be of importance, for example in statistical physics, non-parametric statistics and computational geometry, where they are related to the structure of matter as well as properties of statistical estimators and random graphs.

On the statistical estimation of Rényi entropies

Estimating entropies is important in many fields including statistical physics, machine learning and statistics. While the Shannon logarithmic entropy is the most fundamental, other Rényi entropies are also of importance. In this paper, we derive a bias corrected estimator for a subset of Rényi entropies. The advantage of the estimator is demonstrated via theoretical and experimental considerations.

On the Curse of Dimensionality in Supervised Learning of Smooth Regression Functions

In this paper, the effect of dimensionality on the supervised learning of infinitely differentiable regression functions is analyzed. By invoking the Van Trees lower bound, we prove lower bounds on the generalization error with respect to the number of samples and the dimensionality of the input space both in a linear and non-linear context. It is shown that in non-linear problems without prior knowledge, the curse of dimensionality is a serious problem. At the same time, we speculate counter-intuitively that sometimes supervised learning becomes plausible in the asymptotic limit of infinite dimensionality.

A continuous regression function for the Delaunay calibration method

Abstract The Delaunay tessellation and topological regression is a local simplex method for multivariate calibration. The method, developed within computational geometry, has potential for applications in online analytical chemistry and process monitoring. This study proposes a novel approach to perform prediction and extrapolation using Delaunay calibration method. The main property of the proposed extension is the continuity of the estimated regression function also outside the calibration domain. To support the presentation, an application in estimating the aromatic composition in Light Cycle Oil by Near Infrared spectroscopy is discussed.

Delaunay Tessellation and Topological Regression: An Application to Estimating Product Properties from Spectroscopic Measurements

Abstract The Delaunay tessellation and topological regression is a local simplex method for multivariate calibration. The method, developed within computational geometry has potential for applications in analytical chemistry and process monitoring. This study investigates the applicability of the method for estimating the aromatic composition in Light Cycle Oil (LCO) by Near Infrared (NIR) spectroscopy.

Long-Term prediction of time series using state-space models

State-space models offer a powerful modelling tool for time series prediction. However, as most algorithms are not optimized for long-term prediction, it may be hard to achieve good prediction results. In this paper, we investigate Gaussian linear regression filters for parameter estimation in state-space models and we propose new long-term prediction strategies. Experiments using the EM-algorithm for training of nonlinear state-space models show that significant improvements are possible with no additional computational cost.

Student performance in computing education: an empirical analysis of online learning in programming education environments

ABSTRACT The benefits of using online exercises have been analysed in terms of distance learning, automatic assessment and self-regulated learning. In this study, we have not found a direct proportional relationship between student performance in the course exercises that use online technologies and the exam grades. We see that the average submission rate to these online exercises is not positively correlated with the exercise points. Yet, our results confirm that doing exercises along supports student learning and skill accumulation equipping them with the knowledge of programming. While the student performance in programming courses is affected by factors such as prior background in programming, cognitive skills and the quality of teaching, completing the course exercises via learning-by-doing is an indispensable part of teaching. Based on the student feedback from the course survey, the students are highly satisfied with using online technologies as part of learning.