Shu Kay Angus Ng

Using the EM algorithm to train neural networks: misconceptions and a new algorithm for multiclass classification

The expectation-maximization (EM) algorithm has been of considerable interest in recent years as the basis for various algorithms in application areas of neural networks such as pattern recognition. However, there exists some misconceptions concerning its application to neural networks. In this paper, we clarify these misconceptions and consider how the EM algorithm can be adopted to train multilayer perceptron (MLP) and mixture of experts (ME) networks in applications to multiclass classification. We identify some situations where the application of the EM algorithm to train MLP networks may be of limited value and discuss some ways of handling the difficulties. For ME networks, it is reported in the literature that networks trained by the EM algorithm using iteratively reweighted least squares (IRLS) algorithm in the inner loop of the M-step, often performed poorly in multiclass classification. However, we found that the convergence of the IRLS algorithm is stable and that the log likelihood is monotonic increasing when a learning rate smaller than one is adopted. Also, we propose the use of an expectation-conditional maximization (ECM) algorithm to train ME networks. Its performance is demonstrated to be superior to the IRLS algorithm on some simulated and real data sets.

Speeding up the EM algorithm for mixture model-based segmentation of magnetic resonance images

Mixture models implemented via the expectation-maximization (EM) algorithm are being increasingly used in a wide range of problems in pattern recognition such as image segmentation. However, the EM algorithm requires considerable computational time in its application to huge data sets such as a three-dimensional magnetic resonance (MR) image of over 10 million voxels. Recently, it was shown that a sparse, incremental version of the EM algorithm could improve its rate of convergence. In this paper, we show how this modified EM algorithm can be speeded up further by adopting a multiresolution kd-tree structure in performing the E-step. The proposed algorithm outperforms some other variants of the EM algorithm for segmenting MR images of the human brain.

Multivariate Skew t Mixture Models: Applications to Fluorescence-Activated Cell Sorting Data

In many applied problems in the context of pattern recognition, the data often involve highly asymmetric observations. Normal mixture models tend to overfit when additional components are included to capture the skewness of the data. Increased number of pseudo-components could lead to difficulties and inefficiencies in computations. Also, the contours of the fitted mixture components may be distorted. In this paper, we propose to adopt mixtures of multivariate skew t distributions to handle highly asymmetric data. The EM algorithm is used to compute the maximum likelihood estimates of model parameters. The method is illustrated using a flurorescence-activated cell sorting data.

Ensemble Approach for the Classification of Imbalanced Data

Ensembles are often capable of greater prediction accuracy than any of their individual members. As a consequence of the diversity between individual base-learners, an ensemble will not suffer from overfitting. On the other hand, in many cases we are dealing with imbalanced data and a classifier which was built using all data has tendency to ignore minority class. As a solution to the problem, we propose to consider a large number of relatively small and balanced subsets where representatives from the larger pattern are to be selected randomly. As an outcome, the system produces the matrix of linear regression coefficients whose rows represent random subsets and columns represent features. Based on the above matrix we make an assessment of how stable the influence of the particular features is. It is proposed to keep in the model only features with stable influence. The final model represents an average of the base-learners, which are not necessarily a linear regression. Test results against datasets of the PAKDD-2007 data-mining competition are presented.

A two‐way clustering framework to identify disparities in multimorbidity patterns of mental and physical health conditions among Australians

Multimorbidity is present in more than one quarter of the population in Australia, and its prevalence increases with age. Greater multimorbidity burden among individuals is always associated with poor health-related outcomes, including quality of life, health service utilization and mortality, among others. It is thus significant to identify the heterogeneity in multimorbidity patterns in the community and determine the impact of multimorbidity on individual health outcomes. In this paper, I propose a two-way clustering framework to identify clusters of most significant non-random comorbid health conditions and disparities in multimorbidity patterns among individuals. This framework can establish a clustering-based approach to determine the association between multimorbidity patterns and health-related outcomes and to calculate a multimorbidity score for each individual. The proposed method is illustrated using simulated data and a national survey data set of mental health and wellbeing in Australia.

On modifications to the long-term survival mixture model in the presence of competing risks

A mixture model for long-term survivors has been adopted in various fields such as biostatistics and criminology where some individuals may never experience the type of failure under study. It is directly applicable in situations where the only information available from follow-up on individuals who will never experience this type of failure is in the form of censored observations. In this paper, we consider a modification to the model so that it still applies in the case where during the follow-up period it becomes known that an individual will never experience failure from the cause of interest. Unless a model allows for this additional information, a consistent survival analysis will not be obtained. A partial maximum likelihood (ML) approach is proposed that preserves the simplicity of the long-term survival mixture model and provides consistent estimators of the quantities of interest. Some simulation experiments are performed to assess the efficiency of the partial ML approach relative to the full M...

Modelling inpatient length of stay by a hierarchical mixture regression via the EM algorithm

The modelling of inpatient length of stay (LOS) has important implications in health care studies. Finite mixture distributions are usually used to model the heterogeneous LOS distribution, due to a certain proportion of patients sustaining a longer stay. However, the morbidity data are collected from hospitals, observations clustered within the same hospital are often correlated. The generalized linear mixed model approach is adopted to accomodate the inherent correlation via unobservable random effects. An EM algorithm is developed to obtain residual maximum quasi-likelihood estimation. The proposed hierarchical mixture regression approach enables the identification and assessment of factors influencing the long-stay proportion and the LOS for the long-stay patient subgroup. A neonatal LOS data set is used for illustration.

On clustering by mixture models

Finite mixture models are being increasingly used to model the distributions of a wide variety of random phenomena and to cluster data sets; see, for example, (2000a). We consider the use of normal mixture models to cluster data sets of continuous multivariate data, concentrating on some of the associated computational issues. A robust version of this approach to clustering is obtained by modelling the data by a mixture of t distributions (Peel and McLachlan, 2000). The normal and t mixture models can be fitted by maximum likelihood via the EM algorithm, as implemented in the EMMIX software of the authors. We report some recent results of (2000) on speeding up the fitting process by an an incremental version of the EM algorithm. The problem of clustering high-dimensional data by use of the mixture of factor analyzers model (McLachlan and Peel, 2000b) is also considered. This approach enables a normal mixture model to be fitted to data which have high dimension relative to the number of data points to be clustered.

Constrained mixture models in competing risks problems

We consider the problem of modelling the failure-time distribution, where failure is due to two distinct causes. One approach is to adopt a two-component mixture model where the components correspond to the two different causes of failure. However, routine application of this approach with typical parametric forms for the component densities proves to be inadequate in modelling the time to a re-replacement operation or death after the initial replacement of the aortic valve in the heart by a prosthesis, such as a xenograft valve. Hence we consider modifications to the usual mixture model approach to handle situations where there exists a strong dependency between the failure times of the distinct causes. With these modifications, a suitable model is able to be provided for the distribution of the time to a re-replacement operation conditional on the age of the patient at the time of the initial replacement operation. The estimate so obtained by the probability that a patient of a given age will undergo a re-replacement operation provides a useful guide to heart surgeons on the type of valve to be used in view of the patients age. Copyright (C) 1999 John Wiley & Sons, Ltd.

Clustering via Mixture Regression Models with Random Effects

In this paper, we consider the use of mixtures of linear mixed models to cluster data which may be correlated and replicated and which may have covariates. For each cluster, a regression model is adopted to incorporate the covariates, and the correlation and replication structure in the data are specified by the inclusion of random effects terms. The procedure is illustrated in its application to the clustering of gene-expression profiles.