Timothy Masters

Restricted Boltzmann Machines

When it comes to practical aspects of training RBMs, “A Practical Guide to Training Restricted Boltzmann Machines” by Geoffrey Hinton (2010) can’t be beat. Also, the numerous technical papers by Geoffrey Hinton cover specific aspects of RMBs in glorious detail. Finally, deeplearning.net is an incredible resource. Because of this wealth of material, I will avoid unnecessary duplication. This chapter will be limited to an outline of the essentials of RBMs, in other words, the information necessary to understand and use the programs presented here.

Resampling for Assessing Parameter Estimates

We often collect a random sample of cases from a population, let this sample interact with a model in some way, and then examine with interest a number (or several numbers) that result from the interaction. For example, we may use the random sample to train a model and then examine one or more of the model’s learned parameters. More often, we apply a previously trained model to the cases and compute a measure of the model’s performance so that we may judge the model’s worth and perhaps even extrapolate its future performance. On page 53 we saw an excellent method for computing confidence bounds for individual prediction errors. On page 121 we saw that this same method could be used to compute confidence bounds for clusters of future gains or costs obtained from a classification scheme. The subject of this chapter is somewhat different, though nevertheless related. Here, we are not concerned with performance on individual or small groups of future cases. Rather, we compute a single measure that describes some aspect of the model, and then we judge the quality of this measurement.

Using the DATAMINE Program

This chapter serves as a user’s manual for the DATAMINE program, which demonstrates the algorithms presented in this book. Each menu selection is discussed in its own section.

DEEP Operating Manual

This chapter presents a concise operating manual for DEEP 2.0. The first section lists every menu option along with a short description of its purpose and the page number on which more details can be found if the short description is not sufficient.

Assessment of Class Predictions

The previous chapter focused on models that make numeric predictions. This chapter deals with models whose goal is classification. It must be understood that the distinction is not always clear. In particular, almost no models can be considered to be pure classifiers. Most classification models make a numeric prediction (of a scalar or a vector) and then use this numeric prediction to define a classification decision. Thus, the real distinction is not in the nature of the model but in the nature of the ultimate goal. The implications of this fact will resound throughout the chapter.

Resampling for Assessing Prediction and Classification

The most common procedure for assessing the performance of a prediction or classification model is to split the data collection into two subsets. The model is trained with one dataset, the training set, and then tested with a completely independent dataset, the test set or validation set or out-of-sample set. (The choice of term is usually personal preference.) A performance measure, such as mean squared error, median absolute error, profit factor, cost, or any other custom figure, is computed for the independent set. This performance measure provides a quality judgment for the model. Naive researchers stop here. But enlightened researchers go one step further. They use the methods described in Chapter 3 to evaluate the reliability of the performance measure.

Fun with Eigenvectors

Suppose we measure the height and weight of a collection of people. We could make a plot of the results, using an asterisk for each person. The horizontal position is determined by the person’s height, and the vertical position is determined by the person’s weight. The resulting plot might look something like that shown in Figure 11-1.

Miscellaneous Resampling Techniques

Previous chapters explored some uses for resampling techniques. In Chapter 3 we saw that the bias and variance of parameter estimators could themselves be estimated. This included model parameters as well as performance measures based on independent test data. Then in Chapter 4 we saw that performance measures for a model could be safely obtained from the very same data that was used to train the model. In this chapter we will explore assorted methods for using resampling to improve the performance of models. In particular, we will witness a marvelous phenomenon: it will be shown how a model whose performance is only slightly better than random guessing can be used to create a super-model whose performance is markedly better than the original. These techniques are extremely expensive in terms of computational requirements. However, in situations in which performance is more important than cost, resampling methods for model building are priceless.

Displaying Relationship Anomalies

Naive measures of association between variables, such as linear correlation, are primarily sensitive to gross relationships, those patterns that are easy to detect, see, and describe. In prior chapters we examined measures that go beyond such naivete and are able to detect more subtle dependencies between variables, in other words, anomalies in otherwise uncomplicated relationships. But what if we want a visual representation of the pattern that connects them? In this chapter we present several ways of doing this.

Assessment of Numeric Predictions

Most people divide prediction into two families: classification and numeric prediction. In classification, the goal is to assign an unknown case into one of several competing categories (benign versus malignant, tank versus truck versus rock, and so forth). In numeric prediction, the goal is to assign a specific numeric value to a case (expected profit of a trade, expected yield in a chemical batch, and so forth). Actually, such a clear distinction between classification and numeric prediction can be misleading because they blend into each other in many ways. We may use a numeric prediction to perform classification based on a threshold. For example, we may decree that a tissue sample should be called malignant if and only if a numeric prediction model’s output exceeds, say, 0.76. Conversely, we may sometimes use a classification model to predict values of an ordinal variable, although this can be dangerous if not done carefully. Ultimately, the choice of a numeric or classification model depends on the nature of the data and on a sometimes arbitrary decision by the experimenter. This chapter discusses methods for assessing models that make numeric predictions.