Jonathan J. Oliver

Minimum Message Length Segmentation

The segmentation problem arises in many applications in data mining, A.I. and statistics, including segmenting time series, decision tree algorithms and image processing. In this paper, we consider a range of criteria which may be applied to determine if some data should be segmented into two or regions. We develop a information theoretic criterion (MML) for the segmentation of univariate data with Gaussian errors. We perform simulations comparing segmentation methods (MML, AIC, MDL and BIC) and conclude that the MML criterion is the preferred criterion. We then apply the segmentation method to financial time series data.

MML Estimation of the Parameters of the Sherical Fisher Distribution

The information-theoretic Minimum Message Length (MML) principle leads to a general invariant Bayesian technique for point estimation. We apply MML to the problem of estimating the concentration parameter, κ, of spherical Fisher distributions. (Assuming a uniform prior on the field direction, μ, MML simply returns the Maximum Likelihood estimate for μ.) In earlier work, we dealt with the von Mises circular case, d=2. We say something about the general case for arbitrary d ≥ 2 and how to derive the MML estimator, but here we only carry out a complete calculation for the spherical distribution, with d=3. Our simulation results show that the MML estimator compares very favourably against the classical methods of Maximum Likelihood and marginal Maximum Likelihood (R.A. Fisher (1953), Schou (1978)). Our simulation results also show that the MML estimator compares quite favourably against alternative Bayesian methods.

Discriminant analysis when the classes arise from a continuum

Abstract Sometimes two class linear discriminant analysis is applied to situations in which the classes are formed by partitioning an underlying continuum. In such cases, a reasonable assumption is that the underlying continuous “response” variable forms a joint multivariate normal distribution with the predictors. We compare the error rate of linear discriminant analysis with that of the optimal classification rule under these conditions, showing that linear discriminant analysis leads to a decision surface parallel to, but shifted from, the decision surface of the optimal rule and that the two rules can lead to very different error rates.

The Kindest Cut: Minimum Message Length Segmentation

We consider some particular instances of the segmentation problem. We derive minimum message length (MML) expressions for stating the region boundaries for some one and two dimensional examples. It is found the message length cost of stating region boundaries is dependent on the noise of the data in the separated regions and also the ‘degree of separation’ of the two regions.

Averaging over decision stumps

In this paper, we examine a minimum encoding approach to the inference of decision stumps. We then examine averaging over decision stumps as a method of generating probability estimates at the leaves of decision trees.

Bayesian Estimation of the Von Mises Concentration Parameter

The von Mises distribution is a maximum entropy distribution. It corresponds to the distribution of an angle of a compass needle in a uniform magnetic field of direction, μ, with concentration parameter, κ. The concentration parameter, κ, is the ratio of the field strength to the temperature of thermal fluctuations. Previously, we obtained a Bayesian estimator for the von Mises distribution parameters using the information-theoretic Minimum Message Length (MML) principle. Here, we examine a variety of Bayesian estimation techniques by examining the posterior distribution in both polar and Cartesian co-ordinates. We compare the MML estimator with these fellow Bayesian techniques, and a range of Classical estimators. We find that the Bayesian estimators outperform the Classical estimators.

Averaging over decision trees

Pruning a decision tree is considered by some researchers to be the most important part of tree building in noisy domains. While there are many approaches to pruning, the alternative of averaging over decision trees has not received as much attention. The basic idea of tree averaging is to produce a weighted sum of decisions. We consider the set of trees used for the averaging process, and how weights should be assigned to each tree in this set. We define the concept of afanned set for a tree, and examine how the Minimum Message Length paradigm of learning may be used to average over decision trees. We perform an empirical evaluation of two averaging approaches, and a Minimum Message Length approach.

A decision graph explanation of protein secondary structure prediction

The machine-learning technique of decision graphs, a generalization of decision trees, is applied to the prediction of protein secondary structure to infer a theory for this problem. The resulting decision graph provides both a prediction method and an explanation for the problem. Many decision graphs are possible for the problem. A particular graph is just one theory or hypothesis of secondary structure formation. Minimum message length encoding is used to judge the quality of different theories. It is a general technique of inductive inference and is resistant to learning the noise in the training data. The method was applied to 75 sequences from nonhomologous proteins comprising 13 K amino acids. The predictive accuracy for three states (extended, helix, other) was in the range achieved by current methods.<<ETX>>

DISSOLVE: A System for the Generation of Human Oriented Solutions to Algebraic Equations

In general, competent algebraists can easily recognize expressions which hold high potential for the successful application of commonly applied algebraic transformations, such as transferring terms to the other side of an equation and factoring out common factors. Novice students, however, have difficulty identifying these expressions and assessing the promise of the application of particular transformations. In order to teach these skills, an Intelligent Tutoring System must be able to reason about algebraic expressions in a way which is accessible to a student. In this paper, we describe a mechanism for the representation and manipulation of algebraic expressions which is able to characterize algebraic transformations commonly performed by algebraists. This mechanism has been implemented in a system called DISSOLVE which generates explainable and intuitively appealing solutions to algebraic equations at the high-school level.

Lexical Access using Minimum Message Length Encoding

A method for deriving equivalence classes for lexical access in speech recognition is considered, which automatically derives equivalence classes from training data using unsupervised learning and the Minimum Message Length Criterion. These classes model insertions, deletions and substitutions in an input phoneme string due to mis-recognition and mis-pronunciation, and allow unlikely word candidates to be eliminated quickly. This in turn allows a more detailed examination of the remaining candidates to be carried out efficiently.