Milad Kharratzadeh

Bayesian topic model approaches to online and time-dependent clustering

Clustering algorithms strive to organize data into meaningful groups in an unsupervised fashion. For some datasets, these algorithms can provide important insights into the structure of the data and the relationships between the constituent items. Clustering analysis is applied in numerous fields, e.g., biology, economics, and computer vision. If the structure of the data changes over time, we need models and algorithms that can capture the time-varying characteristics and permit evolution of the clustering. Additional complications arise when we do not have the entire dataset but instead receive elements one-by-one. In the case of data streams, we would like to process the data online, sequentially maintaining an up-to-date clustering. In this paper, we focus on Bayesian topic models; although these were originally derived for processing collections of documents, they can be adapted to many kinds of data. The main purpose of the paper is to provide a tutorial description and survey of dynamic topic models that are suitable for online clustering algorithms, but we illustrate the modeling approach by introducing a novel algorithm that addresses the challenges of time-dependent clustering of streaming data.

Specialized hybrid learners resolve Rogers' paradox about the adaptive value of social learning

Culture is considered an evolutionary adaptation that enhances reproductive fitness. A common explanation is that social learning, the learning mechanism underlying cultural transmission, enhances mean fitness by avoiding the costs of individual learning. This explanation was famously contradicted by Rogers (1988), who used a simple mathematical model to show that cheap social learning can invade a population without raising its mean fitness. He concluded that some crucial factor remained unaccounted for, which would reverse this surprising result. Here we extend this model to include a more complex environment and limited resources, where individuals cannot reliably learn everything about the environment on their own. Under such conditions, cheap social learning evolves and enhances mean fitness, via hybrid learners capable of specializing their individual learning. We then show that while spatial or social constraints hinder the evolution of hybrid learners, a novel social learning strategy, complementary copying, can mitigate these effects.

Semi-parametric Order-based Generalized Multivariate Regression

In this paper, we consider a generalized multivariate regression problem where the responses are monotonic functions of linear transformations of predictors. We propose a semi-parametric algorithm based on the ordering of the responses which is invariant to the functional form of the transformation function. We prove that our algorithm, which maximizes the rank correlation of responses and linear transformations of predictors, is a consistent estimator of the true coefficient matrix. We also identify the rate of convergence and show that the squared estimation error decays with a rate of

Sparse multivariate factor regression

o(1/\sqrt{n})

Invariancy of Sparse Recovery Algorithms

. We then propose a greedy algorithm to maximize the highly non-smooth objective function of our model and examine its performance through extensive simulations. Finally, we compare our algorithm with traditional multivariate regression algorithms over synthetic and real data.

Order-based generalized multivariate regression

We introduce a sparse multivariate regression algorithm which simultaneously performs dimensionality reduction and parameter estimation. We decompose the coefficient matrix into two sparse matrices: a long matrix mapping the predictors to a set of factors and a wide matrix estimating the responses from the factors. We impose an elastic net penalty on the former and an ℓ1 penalty on the latter. Our algorithm simultaneously performs dimension reduction and coefficient estimation and automatically estimates the number of latent factors from the data. Our formulation results in a non-convex optimization problem, which despite its flexibility to impose effective low-dimensional structure, is difficult, or even impossible, to solve exactly in a reasonable time. We specify a greedy optimization algorithm based on alternating minimization to solve this non-convex problem and provide theoretical results on its convergence and optimality. Finally, we demonstrate the effectiveness of our algorithm via experiments on simulated and real data.

Online time-dependent clustering using probabilistic topic models

In this paper, a property for sparse recovery algorithms, called invariancy, is introduced. The significance of invariancy is that the performance of the algorithms with this property is less affected when the sensing (i.e., the dictionary) is ill-conditioned. This is because for this kind of algorithms, there exists implicitly an equivalent well-conditioned problem, which is being solved. Some examples of sparse recovery algorithms will also be considered and it will be shown that some of them, such as SL0, Basis Pursuit (using interior point LP solver), FOCUSS, and hard thresholding algorithms, are invariant, and some others, like Matching Pursuit and SPGL1, are not. Then, as an application example of the invariancy property, a sparse-decomposition-based method for direction of arrival estimation is reviewed, and it is shown that if an invariant algorithm is utilized for solving the corresponding sparse recovery problem, the spatial characteristics of the sensors will have essentially no effect on the final estimation, provided that the number of sensors is large enough.

Channel aware decentralized detection in wireless sensor networks

In this paper, we consider a generalized multivariate regression problem where the responses are monotonic functions of linear transformations of predictors. We propose a semi-parametric algorithm based on the ordering of the responses which is invariant to the functional form of the transformation function. We prove that our algorithm, which maximizes the rank correlation of responses and linear transformations of predictors, is a consistent estimator of the true coefficient matrix. We also identify the rate of convergence and show that the squared estimation error decays with a rate of o(1/√n). We then propose a greedy algorithm to maximize the highly non-smooth objective function of our model and examine its performance through simulations. Finally, we compare our algorithm with traditional multivariate regression algorithms over synthetic and real data.

How to use real-valued sparse recovery algorithms for complex-valued sparse recovery?

We introduce an online, time-dependent clustering algorithm that employs a dynamic probabilistic topic model. The proposed algorithm can handle data that evolves over time and strives to capture the evolution of clusters in the dataset. It addresses the case where the entire dataset is not available at once (e.g., the case of data streams) but an up-to-date clustering of the data at any given time is required. One of the main challenges of the data stream setting is that the computational cost and memory overhead must stay bounded as the number of data points increases. Our proposed algorithm has a Dirichlet process-based generative component combined with a sequential Monte Carlo sampler for posterior inference. We also introduce a novel modification to the sampling process, called targeted sampling, which enhances the performance of the SMC sampler. We test the performance of our algorithm with both synthetic and real datasets.

Weblog Analysis for Predicting Correlations in Stock Price Evolutions

Most of the channel aware decentralized detection methods are actually semi-decentralized in the sense that all the peripheral sensors transmit a quantized function of their observations to a central node (fusion center) and the final decision is made there. In this paper, we propose a fully decentralized channel aware algorithm for decentralized detection which is based on gossip. In our method nodes try to reach consensus by communicating only with neighbouring nodes. First, we propose our method where the channels between the nodes are ideal (i.e., no fading or noise) and show that it converges to the same solution as the centralized scheme, and hence it is globally optimum. Then, we modify it for the case of noisy channels with flat fading and show by simulation that our algorithm still converges to the optimum solution.