Sergio Valcarcel Macua

Consensus-based distributed principal component analysis in wireless sensor networks

Principal component analysis is a powerful technique for data analysis and compression, with a wide range of potential applications in wireless sensor networks. However, its centralized implementation, with a fusion center collecting all the samples, is inefficient in terms of energy consumption, scalability, and fault tolerance. Previous distributed approaches reduce the communication cost, but not the lack of flexibility, as they require multi-hop communications if the network is not fully connected. We present two fully distributed consensus-based algorithms that are guaranteed to converge to the global results, using only local communications among neighbors, regardless of the data distribution or the sparsity of the network: CB-DPCA is based on finding the eigenvectors of local covariance matrices, while CB-EM-DPCA is a distributed version of the expectation maximization algorithm. Both offer a flexible trade-off between the tightness of the achieved approximation and the associated communication cost.

Distributed Policy Evaluation Under Multiple Behavior Strategies

We apply diffusion strategies to develop a fully-distributed cooperative reinforcement learning algorithm in which agents in a network communicate only with their immediate neighbors to improve predictions about their environment. The algorithm can also be applied to off-policy learning, meaning that the agents can predict the response to a behavior different from the actual policies they are following. The proposed distributed strategy is efficient, with linear complexity in both computation time and memory footprint. We provide a mean-square-error performance analysis and establish convergence under constant step-size updates, which endow the network with continuous learning capabilities. The results show a clear gain from cooperation: when the individual agents can estimate the solution, cooperation increases stability and reduces bias and variance of the prediction error; but, more importantly, the network is able to approach the optimal solution even when none of the individual agents can (e.g., when the individual behavior policies restrict each agent to sample a small portion of the state space).

Robust Worst-Case Analysis of Demand-Side Management in Smart Grids

Demand-side management presents significant benefits in reducing the energy load in smart grids by balancing consumption demands or including energy generation and/or storage devices in the user’s side. These techniques coordinate the energy load so that users minimize their monetary expenditure. However, these methods require accurate predictions in the energy consumption profiles, which make them inflexible to real demand variations. In this paper, we propose a realistic model that accounts for uncertainty in these variations and calculates a robust price for all users in the smart grid. We analyze the existence of solutions for this novel scenario, propose convergent distributed algorithms to find them, and perform simulations considering energy expenditure. We show that this model can effectively reduce the monetary expenses for all users in a real-time market, while at the same time it provides a reliable production cost estimate to the energy supplier.

Distributed static linear Gaussian models using consensus

Algorithms for distributed agreement are a powerful means for formulating distributed versions of existing centralized algorithms. We present a toolkit for this task and show how it can be used systematically to design fully distributed algorithms for static linear Gaussian models, including principal component analysis, factor analysis, and probabilistic principal component analysis. These algorithms do not rely on a fusion center, require only low-volume local (1-hop neighborhood) communications, and are thus efficient, scalable, and robust. We show how they are also guaranteed to asymptotically converge to the same solution as the corresponding existing centralized algorithms. Finally, we illustrate the functioning of our algorithms on two examples, and examine the inherent cost-performance trade-off.

Distributed linear discriminant analysis

Linear discriminant analysis (LDA) is a widely used feature extraction method for classification. We introduce distributed implementations of different versions of LDA, suitable for many real applications. Classical eigen-formulation, iterative optimization of the subspace, and regularized LDA can be asymptotically approximated by all the nodes through local computations and single-hop communications among neighbors. These methods are based on the computation of the scatter matrices, so we introduce how to estimate them in a distributed fashion. We test the algorithms in a realistic distributed classification problem, achieving a performance near to the centralized solution and a significant improvement of 35% over the non-cooperative case.

Diffusion gradient temporal difference for cooperative reinforcement learning with linear function approximation

We introduce a diffusion-based algorithm in which multiple agents cooperate to predict a common and global state-value function by sharing local estimates and local gradient information among neighbors. Our algorithm is a fully distributed implementation of the gradient temporal difference with linear function approximation, to make it applicable to multiagent settings. Simulations illustrate the benefit of cooperation in learning, as made possible by the proposed algorithm.

Cooperative off-policy prediction of Markov decision processes in adaptive networks

We apply diffusion strategies to propose a cooperative reinforcement learning algorithm, in which agents in a network communicate with their neighbors to improve predictions about their environment. The algorithm is suitable to learn off-policy even in large state spaces. We provide a mean-square-error performance analysis under constant step-sizes. The gain of cooperation in the form of more stability and less bias and variance in the prediction error, is illustrated in the context of a classical model. We show that the improvement in performance is especially significant when the behavior policy of the agents is different from the target policy under evaluation.

Underwater Electromagnetic Sensor Networks, Part II: Localization and Network Simulations

In the first part of the paper, we modeled and characterized the underwater radio channel in shallow waters. In the second part, we analyze the application requirements for an underwater wireless sensor network (U-WSN) operating in the same environment and perform detailed simulations. We consider two localization applications, namely self-localization and navigation aid, and propose algorithms that work well under the specific constraints associated with U-WSN, namely low connectivity, low data rates and high packet loss probability. We propose an algorithm where the sensor nodes collaboratively estimate their unknown positions in the network using a low number of anchor nodes and distance measurements from the underwater channel. Once the network has been self-located, we consider a node estimating its position for underwater navigation communicating with neighboring nodes. We also propose a communication system and simulate the whole electromagnetic U-WSN in the Castalia simulator to evaluate the network performance, including propagation impairments (e.g., noise, interference), radio parameters (e.g., modulation scheme, bandwidth, transmit power), hardware limitations (e.g., clock drift, transmission buffer) and complete MAC and routing protocols. We also explain the changes that have to be done to Castalia in order to perform the simulations. In addition, we propose a parametric model of the communication channel that matches well with the results from the first part of this paper. Finally, we provide simulation results for some illustrative scenarios.

Location-aided distributed primary user identification in a cognitive radio scenario

We address a cognitive radio scenario, where a number of secondary users performs identification of which primary user, if any, is transmitting, in a distributed way and using limited location information. We propose two fully distributed algorithms: the first is a direct identification scheme, and in the other a distributed sub-optimal detection based on a simplified Neyman-Pearson energy detector precedes the identification scheme. Both algorithms are studied analytically in a realistic transmission scenario, and the advantage obtained by detection pre-processing is also verified via simulation. Finally, we give details of their fully distributed implementation via consensus averaging algorithms.

Learning in constrained stochastic dynamic potential games

We extend earlier works on continuous potential games to the most general case: stochastic time varying environment, stochastic rewards, non-reduced form and constrained state-action sets. We provide conditions for a Markov Nash equilibrium (MNE) of the game to be equivalent to the solution of a single control problem. Then, we address the problem of learning this MNE when the reward and state transition models are unknown. We follow a reinforcement learning approach and extend previous algorithms for working with constrained state-action subsets of real vector spaces. As an application example, we simulate a network flow optimization model, in which the relays have batteries that deplete with a random factor. The results obtained with the proposed framework are close to optimal.