Dimitris Berberidis

Online Censoring for Large-Scale Regressions with Application to Streaming Big Data

On par with data-intensive applications, the sheer size of modern linear regression problems creates an ever-growing demand for efficient solvers. Fortunately, a significant percentage of the data accrued can be omitted while maintaining a certain quality of statistical inference with an affordable computational budget. This work introduces means of identifying and omitting less informative observations in an online and data-adaptive fashion. Given streaming data, the related maximum-likelihood estimator is sequentially found using first- and second-order stochastic approximation algorithms. These schemes are well suited when data are inherently censored or when the aim is to save communication overhead in decentralized learning setups. In a different operational scenario, the task of joint censoring and estimation is put forth to solve large-scale linear regressions in a centralized setup. Novel online algorithms are developed enjoying simple closed-form updates and provable (non)asymptotic convergence guarantees. To attain desired censoring patterns and levels of dimensionality reduction, thresholding rules are investigated too. Numerical tests on real and synthetic datasets corroborate the efficacy of the proposed data-adaptive methods compared to data-agnostic random projection-based alternatives.

Online reconstruction from big data via compressive censoring

This is an era of data deluge with individuals and pervasive sensors acquiring large and ever-increasing amounts of data. Nevertheless, given the inherent redundancy, the costs related to data acquisition, transmission, and storage can be reduced if the per-datum importance is properly exploited. In this context, the present paper investigates sparse linear regression with censored data that appears naturally under diverse data collection setups. A practical censoring rule is proposed here for data reduction purposes. A sparsity-aware censored maximum-likelihood estimator is also developed, which fits well to big data applications. Building on recent advances in online convex optimization, a novel algorithm is finally proposed to enable real-time processing. The online algorithm applies even to the general censoring setup, while its simple closed-form updates enjoy provable convergence. Numerical simulations corroborate its effectiveness in estimating sparse signals from only a subset of exact observations, thus reducing the processing cost in big data applications.

Adaptive censoring for large-scale regressions

Albeit being in the big data era, a significant percentage of data accrued can be overlooked while maintaining reasonable quality of statistical inference at affordable complexity. By capitalizing on data redundancy, interval censoring is leveraged here to cope with the scarcity of resources needed for data exchanging, storing, and processing. By appropriately modifying least-squares regression, first- and second-order algorithms with complementary strengths that operate on censored data are developed for large-scale regressions. Theoretical analysis and simulated tests corroborate their efficacy relative to contemporary competing alternatives.

Data sketching for large-scale Kalman filtering

In an age of exponentially increasing data generation, performing inference tasks by utilizing the available information in its entirety is not always an affordable option. This paper puts forth approaches to render tracking of large-scale dynamic processes via a Kalman filter affordable, by processing a reduced number of data. Three distinct methods are introduced for reducing the number of data involved in the correction step of the filter. Toward this goal, the first two methods employ random projections and innovation-based censoring to effect dimensionality reduction and measurement selection, respectively. The third method achieves reduced complexity by leveraging sequential processing of observations and selecting a few informative updates based on an information-theoretic metric. Simulations on synthetic data compare the proposed methods with competing alternatives, and corroborate their efficacy in terms of estimation accuracy over complexity reduction. Finally, monitoring large networks is considered as an application domain, with the proposed methods tested on Kronecker graphs to evaluate their efficiency in tracking traffic matrices and time-varying link costs.

Online censoring for large-scale regressions

As every day 2.5 quintillion bytes of data are generated, the era of Big Data is undoubtedly upon us. Nonetheless, a significant percentage of the data accrued can be omitted while maintaining a certain quality of statistical inference with a limited computational budget. In this context, estimating adaptively high-dimensional signals from massive data observed sequentially is challenging but equally important in practice. The present paper deals with this challenge based on a novel approach that leverages interval censoring for data reduction. An online maximum likelihood, least mean-square (LMS)-type algorithm, and an online support vector regression algorithm are developed for censored data. The proposed algorithms entail simple, low-complexity, closed-form updates, and have provably bounded regret. Simulated tests corroborate their efficacy.

Distributed recursive least-squares with data-adaptive censoring

The deluge of networked big data motivates the development of computation- and communication-efficient network information processing algorithms. In this paper, we propose two data-adaptive censoring strategies that significantly reduce the computation and communication costs of the distributed recursive least-squares (DRLS) algorithm. Through introducing a cost function that underrates the importance of those observations with small innovations, we develop the first censoring strategy based on the alternating minimization algorithm and the stochastic Newton method. It saves computation when a datum is censored. The computation and communication costs are further reduced by the second censoring strategy, which prohibits a node updating and transmitting its local estimate to neighbors when its current innovation is less than a threshold. For both strategies, a simple criterion for selecting the threshold of innovation is given so as to reach a target ratio of data reduction. The proposed censored D-RLS algorithms guarantee convergence to the optimal argument in the mean-square deviation sense. Numerical experiments validate the effectiveness of the proposed algorithms.

Data Sketching for Large-Scale Kalman Filtering

In an age of exponentially increasing data generation, performing inference tasks by utilizing the available information in its entirety is not always an affordable option. The present paper puts forth approaches to render tracking of large-scale dynamic processes affordable, by processing a reduced number of data. Two distinct methods are introduced for reducing the number of data involved per time step. The first method builds on reduction using low-complexity random projections, while the second performs censoring for data-adaptive measurement selection. Simulations on synthetic data, compare the proposed methods with competing alternatives, and corroborate their efficacy in terms of estimation accuracy over complexity reduction.

Quickest convergence of online algorithms via data selection

Big data applications demand efficient solvers capable of providing accurate solutions to large-scale problems at affordable computational costs. Processing data sequentially, online algorithms offer attractive means to deal with massive data sets. However, they may incur prohibitive complexity in high-dimensional scenarios if the entire data set is processed. It is therefore necessary to confine computations to an informative subset. While existing approaches have focused on selecting a prescribed fraction of the available data vectors, the present paper capitalizes on this degree of freedom to accelerate the convergence of a generic class of online algorithms in terms of processing time/computational resources by balancing the required burden with a metric of how informative each datum is. The proposed method is illustrated in a linear regression setting, and simulations corroborate the superior convergence rate of the recursive least-squares algorithm when the novel data selection is effected.