Sean B. Holden

Drug design by machine learning: support vector machines for pharmaceutical data analysis

We show that the support vector machine (SVM) classification algorithm, a recent development from the machine learning community, proves its potential for structure-activity relationship analysis. In a benchmark test, the SVM is compared to several machine learning techniques currently used in the field. The classification task involves predicting the inhibition of dihydrofolate reductase by pyrimidines, using data obtained from the UCI machine learning repository. Three artificial neural networks, a radial basis function network, and a C5.0 decision tree are all outperformed by the SVM. The SVM is significantly better than all of these, bar a manually capacity-controlled neural network, which takes considerably longer to train.

3D analysis of facial morphology

Dense surface models can be used to analyze 3D facial morphology by establishing a correspondence of thousands of points across each 3D face image. The models provide dramatic visualizations of 3D face‐shape variation with potential for training physicians to recognize the key components of particular syndromes. We demonstrate their use to visualize and recognize shape differences in a collection of 3D face images that includes 280 controls (2 weeks to 56 years of age), 90 individuals with Noonan syndrome (NS) (7 months to 56 years), and 60 individuals with velo‐cardio‐facial syndrome (VCFS; 3 to 17 years of age). Ten‐fold cross‐validation testing of discrimination between the three groups was carried out on unseen test examples using five pattern recognition algorithms (nearest mean, C5.0 decision trees, neural networks, logistic regression, and support vector machines). For discriminating between individuals with NS and controls, the best average sensitivity and specificity levels were 92 and 93% for children, 83 and 94% for adults, and 88 and 94% for the children and adults combined. For individuals with VCFS and controls, the best results were 83 and 92%. In a comparison of individuals with NS and individuals with VCFS, a correct identification rate of 95% was achieved for both syndromes. This article contains supplementary material, which may be viewed at the American Journal of Medical Genetics website at http://www.interscience.wiley.com/jpages/0148‐7299/suppmat/index.html.

Performance Degradation in Boosting

AdaBoost boosts the performance of a weak learner by training a committee of weak learners which learn different features of the training sample space with different emphasis and jointly perform classification or regression of each new data sample by a weighted cumulative vote.We use RBF kernel classifiers to demonstrate that boosting a Strong Learner generally contributes to performance degradation, and identify three patterns of performance degradation due to three different strength levels of the underlying learner. We demonstrate that boosting productivity increases, peaks and then falls as the strength of the underlying learner increases. We highlight patterns of behaviour in the distribution and argue that AdaBoosts characteristic of forcing the strong learner to concentrate on the very hard samples or outliers with too much emphasis is the cause of performance degradation in Strong Learner boosting. However, by boosting an underlying classifier of appropriately low strength, we are able to boost the performance of the committee to achieve or surpass the performance levels achievable by strengthening the individual classifier with parameter or model selection in many instances. We conclude that, if the strength of the underlying learner approaches the identified strength levels, it is possible to avoid performance degradation and achieve high productivity in boosting by weakening the learner prior to boosting...

Generalization and PAC learning: some new results for the class of generalized single-layer networks

The ability of connectionist networks to generalize is often cited as one of their most important properties. We analyze the generalization ability of the class of generalized single-layer networks (GSLNs), which includes Volterra networks, radial basis function networks, regularization networks, and the modified Kanerva model, using techniques based on the theory of probably approximately correct (PAC) learning which have previously been used to analyze the generalization ability of feedforward networks of linear threshold elements (LTEs). An introduction to the relevant computational learning theory is included. We derive necessary and sufficient conditions on the number of training examples required by a GSLN to guarantee a particular generalization performance. We compare our results to those given previously for feedforward networks of LTEs and show that, on the basis of the currently available bounds, the sufficient number of training examples for GSLNs is typically considerably less than for feedforward networks of LTEs with the same number of weights. We show that the use of self-structuring techniques for GSLNs may reduce the number of training examples sufficient to guarantee good generalization performance, and we provide an explanation for the fact that GSLNs can require a relatively large number of weights.

On the practical applicability of VC dimension bounds

This article addresses the question of whether some recent Vapnik-Chervonenkis (VC) dimension-based bounds on sample complexity can be regarded as a practical design tool. Specifically, we are interested in bounds on the sample complexity for the problem of training a pattern classifier such that we can expect it to perform valid generalization. Early results using the VC dimension, while being extremely powerful, suffered from the fact that their sample complexity predictions were rather impractical. More recent results have begun to improve the situation by attempting to take specific account of the precise algorithm used to train the classifier. We perform a series of experiments based on a task involving the classification of sets of vowel formant frequencies. The results of these experiments indicate that the more recent theories provide sample complexity predictions that are significantly more applicable in practice than those provided by earlier theories; however, we also find that the recent theories still have significant shortcomings.

Cross-validation for binary classification by real-valued functions: theoretical analysis

This paper concerns the use of real-valued functions for binary classification problems. Previous work in this area has concentrated on using as an error estimate the ‘resubstitution’ error (that is, the empirical error of a classifier on the training sample) or its derivatives. However, in practice, cross-validation and related techniques are more popular. Here, we devise new holdout and cross-validation estimators for the case where real-valued functions are used as classifiers, and we analyse theoretically the accuracy of these.

PAC-like upper bounds for the sample complexity of leave-one-out cross-validation

When designing a pattern classifier it is often the case that we have available a supervised learning technique and a collection of training data, and we would like to gain some idea of what the error probability of our classifier will be after training. A popular way of approaching this problem in practice is to use some form of error estimate, one of the most popular estimates being the cross-validation estimate. In this paper we address the following question: if we have n training examples what is the probability that the leave-one-out cross-validation estimate differs from the actual error probability by more than a constant c? We derive upper bounds on this probability for the closure algorithm and the deterministic l-inclusion graph prediction strategy.

Machine Learning for First-Order Theorem Proving

We applied two state-of-the-art machine learning techniques to the problem of selecting a good heuristic in a first-order theorem prover. Our aim was to demonstrate that sufficient information is available from simple feature measurements of a conjecture and axioms to determine a good choice of heuristic, and that the choice process can be automatically learned. Selecting from a set of 5 heuristics, the learned results are better than any single heuristic. The same results are also comparable to the prover’s own heuristic selection method, which has access to 82 heuristics including the 5 used by our method, and which required additional human expertise to guide its design. One version of our system is able to decline proof attempts. This achieves a significant reduction in total time required, while at the same time causing only a moderate reduction in the number of theorems proved. To our knowledge no earlier system has had this capability.

Average-case learning curves for radial basis function networks

The application of statistical physics to the study of the learning curves of feedforward connectionist networks has to date been concerned mostly with perceptron-like networks. Recent work has extended the theory to networks such as committee machines and parity machines, and an important direction for current and future research is the extension of this body of theory to further connectionist networks. In this article, we use this formalism to investigate the learning curves of gaussian radial basis function networks (RBFNs) having fixed basis functions. (These networks have also been called generalized linear regression models.) We address the problem of learning linear and nonlinear, realizable and unrealizable, target rules from noise-free training examples using a stochastic training algorithm. Expressions for the generalization error, defined as the expected error for a network with a given set of parameters, are derived for general gaussian RBFNs, for which all parameters, including centers and spread parameters, are adaptable. Specializing to the case of RBFNs with fixed basis functions (basis functions having parameters chosen without reference to the training examples), we then study the learning curves for these networks in the limit of high temperature.

Modeling the Model Athlete: Automatic Coaching of Rowing Technique

Watching athletes allows coaches to provide both vital feedback on how well they are performing and on ways to improve their technique without causing or aggravating injuries. The thoroughness and accuracy of this traditional observation method are limited by human ability and availability. Supplementing coaches with sensor systems that generate accurate feedback on any technical aspect of the performance gives athletes a fall back if they do not have enough confidence in their coachs assessment. A system is presented to model the quality of arbitrary aspects of rowing technique found to be inconsistently well performed by a set of novice rowers when using an ergometer. Using only the motion of the handle, tracked using a high-fidelity motion capture system, a coach trains the system with their idea of the skill-level exhibited during each performance, by labeling example trajectories. Misclassification of unseen performances is encouragingly low, even for unknown performers.