Mahesh V. Joshi

Evaluating boosting algorithms to classify rare classes: comparison and improvements

Classification of rare events has many important data mining applications. Boosting is a promising meta-technique that improves the classification performance of any weak classifier. So far, no systematic study has been conducted to evaluate how boosting performs for the task of mining rare classes. The authors evaluate three existing categories of boosting algorithms from the single viewpoint of how they update the example weights in each iteration, and discuss their possible effect on recall and precision of the rare class. We propose enhanced algorithms in two of the categories, and justify their choice of weight updating parameters theoretically. Using some specially designed synthetic datasets, we compare the capability of all the algorithms from the rare class perspective. The results support our qualitative analysis, and also indicate that our enhancements bring an extra capability for achieving better balance between recall and precision in mining rare classes.

A three-dimensional approach to parallel matrix multiplication

A three-dimensional (3D) matrix multiplication algorithm for massively parallel processing systems is presented. The P processors are configured as a virtual processing cube with dimensions pl, p2, and p3 proportional to the matrices dimensions-M, N, and K. Each processor performs a single local matrix multiplication of size Mlp, x Nlp, x Wp,. Before the local computation can be carried out, each subcube must receive a single submatrix of A and B. After the single matrix multiplication has completed, U/p3 submatrices of this product must be sent to their respective destination processors and then summed together with the resulting matrix C. The 3D parallel matrix multiplication approach has a factor of P1 less communication than the 20 parallel algorithms. This algorithm has been implemented on IBM POWERparallelTM SP2 systems (up to 216 nodes) and has yielded close to the peak performance of the machine. The algorithm has been combined with Winograds variant of Strassens algorithm to achieve performance which exceeds the theoretical peak of the system. (we assume the MFLOPS rate of matrix multiplication to be 2 MNK.)

On evaluating performance of classifiers for rare classes

Predicting rare classes effectively is an important problem. The definition of effective classifier, embodied in the classifier evaluation metric, is however very subjective, dependent on the application domain. In this paper a wide variety of point-metrics are put into a common analytical context defined by the recall and precision of the target rare class. This enables us to compare various metrics in an objective, domain-independent manner. We judge their suitability for the rare class problems along the dimensions of learning difficulty and levels of rarity. This yields many valuable insights. In order to address the goal of achieving better recall and precision, we also propose a way of comparing classifiers directly based on the relationships between recall and precision values. It resorts to a composite point-metric only when recall-precision based comparisons yield conflicting results.

Predicting Rare Classes: Comparing Two-Phase Rule Induction to Cost-Sensitive Boosting

Learning good classifier models of rare events is a challenging task. On such problems, the recently proposed two-phase rule induction algorithm, PNrule, outperforms other non-meta methods of rule induction. Boosting is a strong meta-classifier approach, and has been shown to be adaptable to skewed class distributions. PNrules key feature is to identify the relevant false positives and to collectively remove them. In this paper, we qualitatively argue that this ability is not guaranteed by the boosting methodology. We simulate learning scenarios of varying difficulty to demonstrate that this fundamental qualitative difference in the two mechanisms results in existence of many scenarios in which PNrule achieves comparable or significantly better performance than AdaCost, a strong cost-sensitive boosting algorithm. Even a comparable performance by PNrule is desirable because it yields a more easily interpretable model over an ensemble of models generated by boosting. We also show similar supporting results on real-world and benchmark datasets.

Topic learning from few examples

This paper describes a semi-supervised algorithm for single class learning with very few examples. The problem is formulated as a hierarchical latent variable model which is clipped to ignore classes not of interest. The model is trained using a multistage EM (msEM) algorithm. The msEM algorithm maximizes the likelihood of the joint distribution of the data and latent variables, under the constraint that the distribution of each layer is fixed in successive stages. We demonstrate that with very few positive examples, the algorithm performs better than training all layers in a single stage. We also show that the latter is equivalent to training a single layer model with corresponding parameters. The performance of the algorithm was verified on several real-world information extraction tasks.

A high performance matrix multiplication algorithm for MPPs

A 3-dimensional (3-D) matrix multiplication algorithm for massively parallel processing systems is presented. Performing the product of two matrices C=β C+α A B is viewed as solving a 2-dimensional problem in the 3-dimensional computational space. The three dimensions correspond to the matrices dimensions m, k, and n: A ∈ Rm×k, B ∈Rk×n, and C ∈Rm×n. The p processors are configured as a “virtual” processing cube with dimensions p1, P2, and p3. The cubes dimensions are proportional to the matrices dimensions-m, n, and k. Each processor performs a local matrix multiplication of size m/p1 × n/p2 × k/p3, on one of the sub-cubes in the computational space. Before the local computation can be carried out, each sub-cube needs to receive sub-matrices corresponding to the planes where A and B reside. After the single matrix multiplication has completed, the sub-matrices of C have to be reassigned to their respective processors. The 3-D parallel matrix multiplication approach has, to the best of our knowledge, the least amount of communication among all known parallel algorithms for matrix multiplication. Furthermore, the single resulting sub-matrix computation gives the best possible performance from the uni-processor matrix multiply routine. The 3-D approach achieves high performance for even relatively small matrices and/or a large number of processors (massively parallel). This algorithm has been implemented on IBM Power-parallel SP-2 systems (up to 216 nodes) and have yielded close to the peak performance of the machine. For large matrices, the algorithm can be combined with Winograds variant of Strassens algorithm to achieve “super-linear” speed-up. When the Winograd approach is used, the performance achieved per processor exceeds the theoretical peak of the system.