Hamparsum Bozdogan

Model Selection and Akaike's Information Criterion (AIC): The General Theory and Its Analytical Extensions.

During the last fifteen years, Akaikes entropy-based Information Criterion (AIC) has had a fundamental impact in statistical model evaluation problems. This paper studies the general theory of the AIC procedure and provides its analytical extensions in two ways without violating Akaikes main principles. These extensions make AIC asymptotically consistent and penalize overparameterization more stringently to pick only the simplest of the “true” models. These selection criteria are called CAIC and CAICF. Asymptotic properties of AIC and its extensions are investigated, and empirical performances of these criteria are studied in choosing the correct degree of a polynomial model in two different Monte Carlo experiments under different conditions.

Choosing the Number of Component Clusters in the Mixture-Model Using a New Informational Complexity Criterion of the Inverse-Fisher Information Matrix

This paper considers the problem of choosing the number of component clusters of individuals within the context of the standard mixture of multivariate normal distributions. Often the number of mixture clusters K is unknown, but varying and needs to be estimated. A two-stage iterative maximum-likelihood procedure is used as a clustering criterion to estimate the parameters of the mixture-model under several different covariance structures. An approximate component-wise inverse-Fisher information (IFIM) for the mixture-model is obtained. Then the informational complexity (ICOMP) criterion of IFIM of this author (Bozdogan 1988, 1990a, 1990b) is derived and proposed as a new criterion for choosing the number of clusters in the mixture-model. For comparative purposes, Akaike’s (1973) information criterion (AIC), and Rissanen’s (1978) minimum description length (MDL) criterion are also introduced and derived for the mixture-model. Numerical examples are shown on simulated multivariate normal data sets with a known number of mixture clusters to illustrate the significance of ICOMP in choosing the number of clusters and the best fitting model.

Mixture-Model Cluster Analysis Using Model Selection Criteria and a New Informational Measure of Complexity

Analysis of clusters by means of mixture distribution, called mixture-model cluster analysis, has been one of the most difficult problems in statistics. But theoretical work, coupled with the development of new computational tools in the past ten years, has been made it possible to overcome some of the intractable technical and numerical issues that have limited the widespread applicability of mixture-model cluster analysis to complex real-word problems. The development of new objective analysis techniques had to wait the emergence of information-based model selection procedure to overcome difficulties with cinventional techniques within the context of the mixture-model cluster analysis. See, e.g., Bozdogan (1992), Windham and Cutler (1993) (in this volume)

On the information-based measure of covariance complexity and its application to the evaluation of multivariate linear models

This paper introduces a new information-theoretic measure of complexity called ICOMP as a decision rule for model selection and evaluation for multivariate linear models. The development of ICOMP is based on the generalization and utilization of the covariance complexity index of van Emden (1971) in estimation of the multivariate linear model. ICOMP is motivated by Akaikes (1973) Information Criterion (AIC), but it is a different procedure than AIC. In linear or nonlinear statistical models ICOMP uses an information-based characterization of: (i) the covariance matrix properties of the parameter estimates of a model starting from their finite sampling distributions, and (ii) the complexity of the inverse-Fisher information matrix (i-FIM) as a new criterion of achievable accuracy of the model As a result, it provides a trade-off between the accuracy of the parameter estimates and the interaction of the residuals of a model via the measure of complexity of their respective covariances. It controls the risk...

Multi-Sample Cluster Analysis Using Akaike's Information Criterion.

SummaryMulti-sample cluster analysis, the problem of grouping samples, is studied from an information-theoretic viewpoint via Akaikes Information Criterion (AIC). This criterion combines the maximum value of the likelihood with the number of parameters used in achieving that value. The multi-sample cluster problem is defined, and AIC is developed for this problem. The form of AIC is derived in both the multivariate analysis of variance (MANOVA) model and in the multivariate model with varying mean vectors and variance-covariance matrices. Numerical examples are presented for AIC and another criterion calledw-square. The results demonstrate the utility of AIC in identifying the best clustering alternatives.

Informational complexity criteria for regression models

This paper pursues three objectives in the context of multiple regression models: 1) To give a rationale for model selection criteria which combine a badness of fit term (such as minus twice the log likelihood) with a measure of complexity of a model. 2) To investigate the asymptotic consistency properties of the class of ICOMP criteria first in the case when one of the models considered is the true model and to introduce and establish a consistency property for the case when none of the models is the true model. 3) To investigate the finite sample behavior of ICOMP criteria by means of a simulation study where none of the models considered is the true model.

Ideal Point Discriminant Analysis.

A new method of multiple discriminant analysis was developed that allows a mixture of continuous and discrete predictors. The method can be justified under a wide class of distributional assumptions on the predictor variables. The method can also handle three different sampling situations, conditional, joint and separate. In this method both subjects (cases or any other sampling units) and criterion groups are represented as points in a multidimensional euclidean space. The probability of a particular subject belonging to a particular criterion group is stated as a decreasing function of the distance between the corresponding points. A maximum likelihood estimation procedure was developed and implemented in the form of a FORTRAN program. Detailed analyses of two real data sets were reported to demonstrate various advantages of the proposed method. These advantages mostly derive from model evaluation capabilities based on the Akaike Information Criterion (AIC).

A novel Hybrid RBF Neural Networks model as a forecaster

We introduce a novel predictive statistical modeling technique called Hybrid Radial Basis Function Neural Networks (HRBF-NN) as a forecaster. HRBF-NN is a flexible forecasting technique that integrates regression trees, ridge regression, with radial basis function (RBF) neural networks (NN). We develop a new computational procedure using model selection based on information-theoretic principles as the fitness function using the genetic algorithm (GA) to carry out subset selection of best predictors. Due to the dynamic and chaotic nature of the underlying stock market process, as is well known, the task of generating economically useful stock market forecasts is difficult, if not impossible. HRBF-NN is well suited for modeling complex non-linear relationships and dependencies between the stock indices. We propose HRBF-NN as our forecaster and a predictive modeling tool to study the daily movements of stock indices. We show numerical examples to determine a predictive relationship between the Istanbul Stock Exchange National 100 Index (ISE100) and seven other international stock market indices. We select the best subset of predictors by minimizing the information complexity (ICOMP) criterion as the fitness function within the GA. Using the best subset of variables we construct out-of-sample forecasts for the ISE100 index to determine the daily directional movements. Our results obtained demonstrate the utility and the flexibility of HRBF-NN as a clever predictive modeling tool for highly dependent and nonlinear data.

Measurement and Multivariate Analysis

Diversity is characteristic of the information age and also of statistics. To date, the social sciences have contributed greatly to the development of handling data under the rubric of measurement, while the statistical sciences have made phenomenal advances in theory and algorithms. Measurement and Multivariate Analysis promotes an effective interplay between those two realms of research-diversity with unity. The union and the intersection of those two areas of interest are reflected in the papers in this book, drawn from an international conference in Banff, Canada, with participants from 15 countries. In five major categories - scaling, structural analysis, statistical inference, algorithms, and data analysis - readers will find a rich variety of topics of current interest in the extended statistical community.

Information complexity criteria for detecting influential observations in dynamic multivariate linear models using the genetic algorithm

We develop a new information theoretic approach for detecting influential observations in dynamic linear models of multivariate time series known as vector autoregressions (VARs). Our approach consists of two stages. In the first, we use a Genetic Algorithm (GA) with Bozdogans informational complexity (ICOMP) criterion as the fitness function to select a near optimal subset VAR model. In the second stage, we use ICOMP with case-deletion on the subset VAR chosen by the GA to detect influential observations. Our approach yields an intuitive, practical and rigorous two-dimensional graphical representation of influential observations in multivariate time series data that accounts for both lack-of-fit and model complexity in one criterion function. We demonstrate our approach on multivariate macroeconomic time series data.