Wim De Mulder

A survey on the application of recurrent neural networks to statistical language modeling

HighlightsWe explain in detail the different steps in computing a language model based on a recurrent neural network.We survey the applications and findings based on the current literature.We survey the methods for reducing computational complexity. In this paper, we present a survey on the application of recurrent neural networks to the task of statistical language modeling. Although it has been shown that these models obtain good performance on this task, often superior to other state-of-the-art techniques, they suffer from some important drawbacks, including a very long training time and limitations on the number of context words that can be taken into account in practice. Recent extensions to recurrent neural network models have been developed in an attempt to address these drawbacks. This paper gives an overview of the most important extensions. Each technique is described and its performance on statistical language modeling, as described in the existing literature, is discussed. Our structured overview makes it possible to detect the most promising techniques in the field of recurrent neural networks, applied to language modeling, but it also highlights the techniques for which further research is required.

A generalization of inverse distance weighting and an equivalence relationship to noise-free Gaussian process interpolation via Riesz representation theorem

Abstract In this paper, we show the relationship between two seemingly unrelated approximation techniques. On the one hand, a certain class of Gaussian process-based interpolation methods, and on the other hand inverse distance weighting, which has been developed in the context of spatial analysis where there is often a need for interpolating from irregularly spaced data to produce a continuous surface. We develop a generalization of inverse distance weighting and show that it is equivalent to the approximation provided by the class of Gaussian process-based interpolation methods. The equivalence is established via an elegant application of Riesz representation theorem concerning the dual of a Hilbert space. It is thus demonstrated how a classical theorem in linear algebra connects two disparate domains.

Extending Gaussian process emulation using cluster analysis and artificial neural networks to fit big training sets

ABSTRACT Gaussian process (GP) emulation is a relatively recent statistical technique that provides a fast-running approximation to a complex computer model, given training data generated by the considered model. Despite its sound theoretical foundation, GP emulation falls short in practical applications where the training dataset is very large, due to numerical instabilities in inverting the correlation matrix. We show how GP emulation can be extended to handle large training sets by first dividing the training set into smaller subsets using cluster analysis, then training an emulator for each subset, and finally combining the emulators using an artificial neural network (ANN). Our work has also conceptual relevance, as it shows how to solve a big data problem by introducing a local level in input space, where each emulator specialises in a certain subregion, and a global level, where the identified local features of the computer model are combined into a global view.

A sensitivity analysis of probabilistic sensitivity analysis in terms of the density function for the input variables

ABSTRACT Probabilistic sensitivity analysis (SA) allows to incorporate background knowledge on the considered input variables more easily than many other existing SA techniques. Incorporation of such knowledge is performed by constructing a joint density function over the input domain. However, it rarely happens that available knowledge directly and uniquely translates into such a density function. A naturally arising question is then to what extent the choice of density function determines the values of the considered sensitivity measures. In this paper we perform simulation studies to address this question. Our empirical analysis suggests some guidelines, but also cautions to practitioners in the field of probabilistic SA.