Geng-Shen Fu

Diversity in Independent Component and Vector Analyses: Identifiability, algorithms, and applications in medical imaging

Starting with a simple generative model and the assumption of statistical independence of the underlying components, independent component analysis (ICA) decomposes a given set of observations by making use of the diversity in the data, typically in terms of statistical properties of the signal. Most of the ICA algorithms introduced to date have considered one of the two types of diversity: non-Gaussianity?i.e., higher-order statistics (HOS)?or, sample dependence. A recent generalization of ICA, independent vector analysis (IVA), generalizes ICA to multiple data sets and adds the use of one more diversity, dependence across multiple data sets for achieving an independent decomposition, jointly across multiple data sets. Finally, both ICA and IVA, when implemented in the complex domain, enjoy the addition of yet another type of diversity, noncircularity of the sources?underlying components. Mutual information rate provides a unifying framework such that all these statistical properties?types of diversity?can be jointly taken into account for achieving the independent decomposition. Most of the ICA methods developed to date can be cast as special cases under this umbrella, as well as the more recently developed IVA methods. In addition, this formulation allows us to make use of maximum likelihood theory to study large sample properties of the estimator, derive the Cramer-Rao lower bound(CRLB) and determine the conditions for the identifiability of the ICA and IVA models. In this overview article, we first present ICA, and then its generalization to multiple data sets, IVA, both using mutual information rate, present conditions for the identifiability of the given linear mixing model and derive the performance bounds. We address how various methods fall under this umbrella and give examples of performance for a few sample algorithms compared with the performance bound. We then discuss the importance of approaching the performance bound depending on the goal, and use medical image analysis as the motivating example.

Independent Vector Analysis: Identification Conditions and Performance Bounds

Recently, an extension of independent component analysis (ICA) from one to multiple datasets, termed independent vector analysis (IVA), has been a subject of significant research interest. IVA has also been shown to be a generalization of Hotellings canonical correlation analysis. In this paper, we provide the identification conditions for a general IVA formulation, which accounts for linear, nonlinear, and sample-to-sample dependencies. The identification conditions are a generalization of previous results for ICA and for IVA when samples are independently and identically distributed. Furthermore, a principal aim of IVA is identification of dependent sources between datasets. Thus, we provide additional conditions for when the arbitrary ordering of the estimated sources can be common across datasets. Performance bounds in terms of the Cramér-Rao lower bound are also provided for demixing matrices and interference to source ratio. The performance of two IVA algorithms are compared to the theoretical bounds.

Independent vector analysis, the Kotz distribution, and performance bounds

The recent extensions of independent component analysis (ICA) to exploit source dependence across multiple datasets, termed independent vector analysis (IVA), have thus far only considered two multivariate source distribution models: the Gaussian and a second-order uncorrelated Laplacian distribution. In this paper, we introduce the use of the Kotz distribution family as a more flexible source distribution model which exploits both second and higher-order statistics. The Cramér-Rao lower bound (CRLB) for IVA performance prediction is shown to be analogous to the bound for blind source separation (BSS). Lastly, we provide an analytic expression for the CRLB when the sources follow the multivariate power exponential (MPE) subclass of distributions within the Kotz family.

Blind Source Separation by Entropy Rate Minimization

By assuming latent sources are statistically independent, independent component analysis separates underlying sources from a given linear mixture. Since in many applications, latent sources are both non-Gaussian and have sample dependence, it is desirable to exploit both properties jointly. In this paper, we use mutual information rate to construct a general framework for analysis and derivation of algorithms that take both properties into account. We discuss two types of source models for entropy rate estimation-a Markovian and an invertible filter model-and give the general independent component analysis cost function, update rule, and performance analysis based on these. We also introduce four algorithms based on these two models, and show that their performance can approach the Cramér-Rao lower bound. In addition, we demonstrate that the algorithms with flexible models exhibit very desirable performance for “natural” data.

An efficient multivariate generalized Gaussian distribution estimator: Application to IVA

Due to its simple parametric form, multivariate generalized Gaussian distribution (MGGD) has been widely used for modeling vector-valued signals. Therefore, efficient estimation of its parameters is of significant interest for a number of applications. Independent vector analysis (IVA) is a generalization of independent component analysis (ICA) that makes full use of the statistical dependence across multiple datasets to achieve source separation, and can take both second and higher-order statistics into account. MGGD provides an effective model for IVA as well as for modeling the latent multivariate variables-sources-and the performance of the IVA algorithm highly depends on the estimation of the source parameters. In this paper, we propose an efficient estimation technique based on the Fisher scoring (FS) and demonstrate its successful application to IVA. We quantify the performance of MGGD parameter estimation using FS and further verify the effectiveness of the new IVA algorithm using simulations.

Algorithms for Markovian source separation by entropy rate minimization

Since in many blind source separation applications, latent sources are both non-Gaussian and have sample dependence, it is desirable to exploit both non-Gaussianity and sample dependency. In this paper, we use the Markov model to construct a general framework for the analysis and derivation of algorithms that take both properties into account. We also present two algorithms using two effective source priors. The first one is a multivariate generalized Gaussian distribution and the second is an autoregressive model driven by a generalized Gaussian distributed process. We derive the Cramér-Rao lower bound and demonstrate that the performance of the algorithms approach the lower bound especially when the underlying model matches the parametric model. We also demonstrate that a flexible semi-parametric approach exhibits very desirable performance.

Complex Independent Component Analysis Using Three Types of Diversity: Non-Gaussianity, Nonwhiteness, and Noncircularity

By assuming latent sources are statistically independent, independent component analysis (ICA) separates underlying sources from a given linear mixture. Since in many applications, latent sources are non-Gaussian, noncircular, and have sample dependence, it is desirable to exploit all these properties jointly. Mutual information rate, which leads to the minimization of entropy rate, provides a natural cost for the task. In this paper, we establish the theory for complex-valued ICA giving Cramér-Rao lower bound and identification conditions, and present a new algorithm that takes all these properties into account. We propose an effective estimator of entropy rate and a complex-valued entropy rate bound minimization algorithm based on it. We show that the new method exploits all these properties effectively by comparing the estimation performance with the Cramér-Rao lower bound and by a number of examples.

An efficient entropy rate estimator for complex-valued signal processing: Application to ICA

Estimating likelihood or entropy rate is one of the key issues in many signal processing problems. Mutual information rate, which leads to the minimization of entropy rate, provides a natural cost for achieving blind source separation (BSS). In many complex-valued BSS applications, the latent sources are non-Gaussian, noncircular, and possess sample dependence. Consequently, an effective estimator of entropy rate that jointly considers all three properities of the sources is required. In this paper, we propose such an entropy rate estimator that assumes the sources are generated by invertible filters. With this new entropy rate estimator, we propose a complex entropy rate bound minimization algorithm. Simulation results show that the new method exploits all three properties effectively.

A data-driven solution for abandoned object detection: Advantages of multiple types of diversity

The automated detection of abandoned objects is a quickly developing and widely researched field in video processing with specific application to automated surveillance. In the recent years, a number of approaches have been proposed to automatically detect abandoned objects. However, these techniques require prior knowledge of certain properties of the object such as its shape and color, to classify the foreground objects as abandoned object. The performance of tracking-based approaches degrades in complex scenes, i.e., when the abandoned object is occluded or in the case of crowding. In this paper, we propose a data-driven approach based on independent component analysis (ICA) for object detection. We demonstrate the success of the proposed ICA-based methodology with examples of videos with complex scenarios. We also show that algorithm choice plays an important role in performance, in particular when multiple types of diversities are taken into account and demonstrate the importance of order selection.

Likelihood Estimators for Dependent Samples and Their Application to Order Detection

Estimation of the dimension of the signal subspace, or order detection, is one of the key issues in many signal processing problems. Information theoretic criteria are widely used to estimate the order under the independently and identically distributed (i.i.d.) sampling assumption. However, in many applications, the i.i.d. sampling assumption does not hold. Previous approaches address the dependent sample issue by downsampling the data set so that existing order detection methods can be used. By discarding data, the sample size is decreased causing degradation in the accuracy of the order estimation. In this paper, we introduce two likelihood estimators for dependent samples based on two signal models. The likelihood for each signal model is developed based on the entire data set and used in an information theoretic framework to achieve reliable order estimation performance for dependent samples. Experimental results show the desirable performance of the new method.