Lucas R. Hope

A bayesian metric for evaluating machine learning algorithms

How to assess the performance of machine learning algorithms is a problem of increasing interest and urgency as the data mining application of myriad algorithms grows Rather than predictive accuracy, we propose the use of information-theoretic reward functions The first such proposal was made by Kononenko and Bratko Here we improve upon our alternative Bayesian metric, which provides a fair betting assessment of any machine learner We include an empirical analysis of various Bayesian classification learners.

Causal discovery with prior information

Bayesian networks (BNs) are rapidly becoming a leading tool in applied Artificial Intelligence (AI). BNs may be built by eliciting expert knowledge or learned via causal discovery programs. A hybrid approach is to incorporate prior information elicited from experts into the causal discovery process. We present several ways of using expert information as prior probabilities in the CaMML causal discovery program.

The Evaluation of Predictive Learners: Some Theoretical and Empirical Results

With the growth of interest in data mining, there has been increasing interest in applying machine learning algorithms to real-world problems. This raises the question of how to evaluate the performance of machine learning algorithms. The standard procedure performs random sampling of predictive accuracy until a statistically significant difference arises between competing algorithms. That procedure fails to take into account the calibration of predictions. An alternative procedure uses an information reward measure (from I.J. Good) which is sensitive both to domain knowledge (predictive accuracy) and calibration. We analyze this measure, relating it to Kullback-Leibler distance. We also apply it to five well-known machine learning algorithms across a variety of problems, demonstrating some variations in their assessments using accuracy vs. information reward. We also look experimentally at information reward as a function of calibration and accuracy.

Bayesian Information Reward

We generalize an information-based reward function, introduced by Good (1952), for use with machine learners of classification functions. We discuss the advantages of our function over predictive accuracy and the metric of Kononenko and Bratko (1991). We examine the use of information reward to evaluate popular machine learning algorithms (e.g., C5.0, Naive Bayes, CaMML) using UCI archive datasets, finding that the assessment implied by predictive accuracy is often reversed when using information reward.

An information-theoretic causal power theory

A metric of causal power can assist in developing and using causal Bayesian networks. We introduce a metric based upon information theory. We show that it generalizes prior metrics restricted to linear and noisy-or models, while providing a metric appropriate to the full representational power of Bayesian nets.