Hana Straková

Distributed QR factorization based on randomized algorithms

Most parallel algorithms for matrix computations assume a static network with reliable communication and thus use fixed communication schedules. However, in situations where computer systems may change dynamically, in particular, when they have unreliable components, algorithms with randomized communication schedule may be an interesting alternative. We investigate randomized algorithms based on gossiping for the distributed computation of the QR factorization. The analyses of numerical accuracy showed that known numerical properties of classical sequential and parallel QR decomposition algorithms are preserved. Moreover, we illustrate that the randomized approaches are well suited for distributed systems with arbitrary topology and potentially unreliable communication, where approaches with fixed communication schedules have major drawbacks. The communication overhead compared to the optimal parallel QR decomposition algorithm (CAQR) is analyzed. The randomized algorithms have a much higher potential for trading off numerical accuracy against performance because their accuracy is proportional to the amount of communication invested.

Scalable and Fault Tolerant Orthogonalization Based on Randomized Distributed Data Aggregation

Highlights ► We present the push-flow algorithm (PF), a new distributed data aggregation algorithm (DDAA). ► PF has better resilience properties than previously existing DDAAs. ► PF has very good asymptotic scaling behavior on hypercube topologies. ► Based on DDAAs, we design the new rdmGS algorithm for orthogonalizing a set of vectors in a decentralized distributed fashion. ► rdmGS is capable of producing fully accurate results even if several nodes fail permanently.

A Distributed Eigensolver for Loosely Coupled Networks

We introduce a new distributed eigensolver (dOI) for square matrices based on orthogonal iteration. In contrast to standard parallel eigensolvers, our approach performs only nearest neighbor communication and provides much more flexibility with respect to the properties of the hardware infrastructure on which the computation is performed. This is achieved by utilizing distributed summation methods with randomized communication schedules which do not require global synchronization across the nodes. Our algorithm is particularly attractive for loosely coupled distributed networks with arbitrary network topologies and potentially unreliable components. Our distributed eigensolver dOI is based on a novel distributed matrix-matrix multiplication algorithm and on an extension of a distributed QR factorization algorithm proposed earlier. We illustrate the advantages of dOI in terms of higher flexibility with respect to the underlying network and lower communication cost compared to a related distributed eigensolver by Kempe and McSherry. Moreover, we experimentally illustrate how the overall communication cost of dOI is further reduced by adapting the accuracy of each distributed summation during the orthogonal iteration process.

Fault Tolerance Properties of Gossip-Based Distributed Orthogonal Iteration Methods☆

Abstract In this paper, we investigate and compare the fault tolerance properties and resilience of gossip-based distributed orthog- onal iteration algorithms for the in-network computation of the extreme eigenpairs of matrix. Gossip-based algorithms have many attractive properties, especially for loosely coupled distributed and decentralized systems, like P2P networks or sensor networks. Due to their randomized communication schedule and the fact that communication happens only between nearest neighbors, they are highly flexible with respect to the topology of the underlying system. Moreover, such algorithms have a big potential for high resilience against various types of failures. Lately, several gossip-based distributed eigensolvers based on orthogonal iteration method have been introduced. However, the performance of these algorithms in the presence of failures has not been analyzed yet. We illustrate that convergence properties, the numerical accuracy achieved, as well as resilience properties of gossip-based distributed orthogonal iteration are basically determined by the choice of the distributed data aggregation algorithm (DDAA) which is required within the algorithm for performing distributed reduction operations (such as summation or averaging) across the system. In particular, we illustrate that when using the proper combination of DDAA and distributed orthogonal iteration method, high accuracy can be achieved and even silent message loss can be tolerated without any loss in numerical accuracy.

Robust distributed orthogonalization based on randomized aggregation

The construction of distributed algorithms for matrix computations built on top of distributed data aggregation algorithms with randomized communication schedules is investigated. For this purpose, a new aggregation algorithm for summing or averaging distributed values, the push-flow algorithm, is developed, which achieves superior resilience properties with respect to node failures compared to existing aggregation methods. On a hypercube topology it asymptotically requires the same number of iterations as the optimal all-to-all reduction operation and it scales well with the number of nodes. Orthogonalization is studied as a prototypical matrix computation task. A new fault tolerant distributed orthogonalization method (rdmGS), which can produce accurate results even in the presence of node failures, is built on top of distributed data aggregation algorithms.

Distributed Gram-Schmidt orthogonalization based on dynamic consensus

We propose a novel distributed QR factorization algorithm for orthogonalizing a set of vectors in a wireless sensor network. The algorithm originates from the classical Gram-Schmidt orthogonalization which we formulate in a distributed way using the dynamic consensus algorithm. In contrast to existing distributed QR factorization algorithms, all elements of matrices Q and R are computed simultaneously and updated iteratively after each transmission. Assuming synchronous message broadcasting and communication only with neighboring nodes without any central computing unit (fusion center), we prove convergence of the algorithm. We investigate the algorithm in terms of numerical accuracy and we discuss the influence of the initial data distribution on the algorithm performance. Moreover, we provide a comparison with existing distributed QR algorithms in terms of communication cost and memory requirements, and we illustrate the comparison by simulations.

Improving Fault Tolerance and Accuracy of a Distributed Reduction Algorithm

Most existing algorithms for parallel or distributed reduction operations are not able to handle temporary or permanent link and node failures. Only recently, methods were proposed which are in principal capable of tolerating link and node failures as well as soft errors like bit flips or message loss. A particularly interesting example is the pushflow algorithm. However, on closer inspection, it turns out that in this method the failure recovery often implies severe performance drawbacks. Existing mechanisms for failure handling may basically lead to a fall-back to an early stage of the computation and consequently slow down convergence or even prevent convergence if failures occur too frequently. Moreover, state-of-the-art fault tolerant distributed reduction algorithms may experience accuracy problems even in failure free systems. We present the push-cancel-flow (PCF) algorithm, a novel algorithmic enhancement of the push-flow algorithm. We show that the new push-cancel-flow algorithm exhibits superior accuracy, performance and fault tolerance over all other existing distributed reduction methods. Moreover, we employ the novel PCF algorithm in the context of a fully distributed QR factorization process and illustrate that the improvements achieved at the reduction level directly translate to higher level matrix operations, such as the considered QR factorization.

Analysis and Comparison of Truly Distributed Solvers for Linear Least Squares Problems on Wireless Sensor Networks

The solution of linear least squares problems across large loosely connected distributed networks (such as wireless sensor networks) requires distributed algorithms which ideally need very little or no coordination between the nodes. We first provide an extensive overview of distributed least squares solvers appearing in the literature and classify them according to their communication patterns. We are particularly interested in truly distributed algorithms which do not require a fusion centre, cluster heads or any multi-hop communication. Beyond existing methods, we propose the novel least squares solver PSDLS, which utilises a recently developed distributed QR factorisation algorithm. All communication between nodes is exclusively performed within the push-sum algorithm for distributed aggregation.

Abstract: Gossip-Based Distributed Matrix Computations

We investigate randomized distributed algorithms for matrix computations over loosely coupled distributed systems, such as P2P networks or sensor networks. In this poster, we discuss orthogonalization methods and orthogonal iteration. These algorithms are very well understood in the sequential or in the classical parallel context, and they are important building blocks for many algorithms in numerical linear algebra.

Poster: Gossip-Based Distributed Matrix Computations

EXTENDED ABSTRACT SUBMITTED SEPARATELY