John F. Elder

Induction and polynomial networks

Induction plays a major role in a wide variety of application domains. Because of this broad range of applicability a variety of approaches have been suggested and employed to discover general models from data. A key goal in these approaches is to perform well on data not seen during the model construction process. This paper surveys the variety of techniques available for induction and categorizes them by their degree of automation. The authors then examine in more detail polynomial networks which are induction methods that grew out of cybernetics and early neural network research. The authors conclude the paper with suggested directions for continued work in polynomial networks.

Heuristic Search for Model Structure: the Benefits of Restraining Greed

Inductive modeling or “machine learning” algorithms are able to discover structure in high-dimensional data in a nearly automated fashion. These adaptive statistical methods — including decision trees, polynomial networks, projection pursuit models, and additive networks — repeatedly search for, and add on, the model component judged best at that state. Because of the huge model space of possible components, the choice is typically greedy; that is, optimal only in the very short term. In fact, it is usual for the analyst and algorithm to be greedy at three levels: when choosing a 1) term within a model, 2) model within a family, and 3) family within a wide collection of methods. It is better, we argue, to “take a longer view” in each stage. For the first stage (term selection) examples are presented for classification using decision trees and estimation using regression. To improve the third stage (method selection) we propose fusing information from disparate models to make a combined model more robust. (Fused models merge their output estimates but also share information on, for example, variables to employ and cases to ignore.) Benefits of fusing are demonstrated on a challenging classification dataset, where the task is to infer the species of a bat from its chirps.

Combining estimators to improve performance

Despite the diverse pedigrees of Data Mining methods, the underlying algorithms fall into a handful of families, whose properties suggest their likely performance on a given dataset. One typically selects an algorithm by matching its strengths to the properties of one’s data. Yet, performance surprises, where competing models rank differently than expected, are common; model inference, even when semi-automated, seems to yet be as much art as science. Recently however, researchers in several fields have discovered that a simple technique combining competing models almost always improves classification accuracy. (Such “bundling” is a natural outgrowth of Data Mining, since much of the model search process is automated, and candidate models abound.) This tutorial will describe an interdisciplinary collection of powerful model combination methods including bundling, bagging, boosting, and Bayesian model averaging and briefly demonstrate their positive effects on scientific, medical, and marketing case studies. The instructors will show why this simple, new idea will often improve a model’s accuracy and stability (robustness).

Thriving as a data miner in the real world

Meaningful work is a deep human need. We all yearn to contribute to something greater than ourselves, be listened to, and work alongside friendly peers. Data mining consulting is a powerful way to use technical skills and gain these great side benefits. The power of analytics and its high return on investment makes ones expertise welcome virtually everywhere. And the variety of projects and domains encountered leads to continual learning as new problems are met and solved. Teaching and writing are possible, and there is great satisfaction in seeing ones work actually implemented and used, potentially touching millions. Still, in industry, one has the joy and hazards of working closely with other humans, where final success can depend as much on others as oneself, and on social as well as technical issues. In my experience, business risk strongly outweighs technical risk in whether a solution is used. I will share some hard-won lessons learned on how to best succeed, both technically and socially, in the results-oriented world of industry.Meaningful work is a deep human need. We all yearn to contribute to something greater than ourselves, be listened to, and work alongside friendly peers. Data mining consulting is a powerful way to use technical skills and gain these great side benefits. The power of analytics and its high return on investment makes ones expertise welcome virtually everywhere. And the variety of projects and domains encountered leads to continual learning as new problems are met and solved. Teaching and writing are possible, and there is great satisfaction in seeing ones work actually implemented and used, potentially touching millions. Still, in industry, one has the joy and hazards of working closely with other humans, where final success can depend as much on others as oneself, and on social as well as technical issues. In my experience, business risk strongly outweighs technical risk in whether a solution is used. I will share some hard-won lessons learned on how to best succeed, both technically and socially, in the results-oriented world of industry.