Pavel Tvrdík

Towards auction algorithms for large dense assignment problems

AbstractnIn this paper, we focus on the problem of solving large-scale instances of the linear sum assignment problem by auction algorithms. We introduce a modified auction algorithm, called look-back auction algorithm, which extends the forward auction algorithm by the ability of reusing information from previous bids. We show that it is able to reuse information from the previous bids with high efficiency for all tested types of input instances. We discuss then the design and implementation of a suite of sequential and distributed memory auction algorithms on a Linux cluster with the evaluation on several types of input instances of the linear sum assignment problem. Our results show that the look-back auction algorithm solves sequentially nearly all types of dense instances faster than other evaluated algorithms and it is more stable than the forward-reverse auction algorithm for sparse instances. Our distributed memory auction algorithms are fully memory scalable.n

Minimal Quadtree Format for Compression of Sparse Matrices Storage

Computations with sparse matrices are widespread in scientific projects. Commonly used storage formats (such as COO or CSR) are not suitable for I/O file operations with sparse matrices due to their high space complexities. Memory-efficient formats are still under development. In this paper, we present a new storage format called the Minimal quadtree (MQ) as well as algorithms for converting matrices from common storage formats to the MQ format. We compare the space complexity of common storage formats and of the MQ format and prove that the idea of using the quadtree as the data structure for sparse matrices is viable.

File System Security in the Environment of Non-dedicated Computer Clusters

Clusters built from commodity computers are popular because of their cost efficiency. There are two basic types of such clusters. A prevalent type is a dedicated cluster where all machines participating in the cluster belong to a single administrative domain. Then there are non-dedicated clusters, where the cluster itself forms its own administrative domain, but the participating computers retain their own administrative domains. Non-dedicated clusters have a better potential for attracting computer owners to participate in a cluster, as they do not have to give up control over their computers. On the other hand, these clusters have much higher security demands than the dedicated ones. In this paper, we focus on the security issues of distributed file systems in the environment of the non-dedicated clusters. We will identify the issues implied by the environment, propose our solution, and demonstrate the feasibility by the experimental implementation.

Necklaces and scalability of Kautz digraphs

In this paper, the following results are reported. The notion of Kautz necklaces, similar to de Bruijn ones, is introduced. A linear-time algorithm for generating Kautz necklaces in lexicographic ordering is described and a formula for enumerating the Kautz necklaces is given. A one-to-one mapping between de Bruijn and Kautz vertices preserving the necklaces is presented. Then the notion of necklaces is generalized and a simple necklace-based algorithm for factorization Kautz digraphs is described. Using this factorization, a construction of d-regular partial line digraphs of d-regular Kautz digraphs is described. This result implies that evenly distributed in the space between d-regular Kautz digraphs of diameter D and D+1 there are d-2 digraphs with the same routing and connectivity properties. Finally, a generalization of the method enables to construct partial line digraphs of any order with the connectivity close to d. These results make the Kautz digraphs unique in that sense that they are not only the very dense digraphs, but also very well incrementally scalable.<<ETX>>

Different Approaches to Distributed Compilation

Source code compiling is a non-trivial task that requires many computing resources. As a software project grows, its build time increases and debugging on a single computer becomes more and more time consuming task. An obvious solution would be a dedicated cluster acting as a build farm, where developers can send their requests. But in most cases, this solution has a very low utilization of available computing resources which makes it very ineffective. Therefore, we have focused on non-dedicated clusters to perform distributed compilation, where we could use users computers as nodes of a build farm. We compare two different approaches: distcc, which is an open-source program to distribute compilation of C/C++ code between several computers on a network and Clondike, which is a universal peer-to-peer cluster that is being developed at the Czech Technical University in Prague. A very complex task able to test deeply both systems is a compilation of a Linux Kernel with many config options. We have run this task on a cluster with up to 20 computers and have measured computing times and CPU loads. In this paper, we will present the results of this experiment that indicate the scalability and utilization of given resources in both systems. We also discuss the penalty of a generic solution over a task-specific one.

Scalable Parallel Generation of Very Large Sparse Benchmark Matrices

We present a method and an accompanying algorithm for scalable parallel generation of sparse matrices intended primarily for benchmarking purposes, namely for evaluation of performance and scalability of generic massively parallel algorithms that involve sparse matrices. The proposed method is based on enlargement of small input matrices, which are supposed to be obtained from public sparse matrix collections containing numerous matrices arising in different application domains and thus having different structural and numerical properties. The resulting matrices are distributed among processors of a parallel computer system. The enlargement process is designed so its users may easily control structural and numerical properties of resulting matrices as well as the distribution of their nonzero elements to particular processors.

Porting clondike to heterogeneous platforms

Clustering plays an important role in todays computer networks. We can achieve a higher efficiency of the whole network infrastructure by simply using idle computing power of ordinary workstations. The Clondike project aims to create a universal non-dedicated peer-to-peer cluster where every participating node can benefit from its membership in the cluster. The peer-to-peer approach does not contain any single point of failure, so a high availability is guaranteed by design. So far, we have tested Clondike only in our laboratory with homogeneous computer network architecture, all the computer nodes were the same. In this paper, we report on experiments with moving the Clondike cluster closer to a real environment: with porting Clondike to a real office network with heterogeneous computers. We have run a distributed compilation of the Linux Kernel on both platforms to verify our results and to assess the weaknesses of the current solution and to identify further development needs.

Downsampling Algorithms for Large Sparse Matrices

Mapping of sparse matrices to processors of a parallel system may have a significant impact on the development of sparse-matrix algorithms and, in effect, to their efficiency. We present and empirically compare two downsampling algorithms for sparse matrices. The first algorithm is independent of particular matrix-processors mapping, while the second one is adapted for cases where matrices are partitioned among processors according to contiguous chunks of rows/columns. We show that the price for the versatility of the first algorithm is the collective communication performed by all processors. The second algorithm uses more efficient communication strategy, which stems from the knowledge of mapping of matrices to processors, and effectively outperforms the first algorithm in terms of running time.

Security System for Overlapping Non-dedicated Clusters

Techniques for building local area network clusters differ from those used for large scale harvesting of idle computing power. The overlapping non dedicated clusters (ONDC) architecture is trying to mix advantages of both techniques. Clusters build in ONDC style can be deployed both on small scale local networks, but as well in large scale over the Internet deployments. In this paper we analyze the security requirements of ONDC and describe our solution. The solution was implemented for the Clondike clustering system, but the same approach can be used for any other ONDC system. The proposed system is inspired by the security mechanisms of existing P2P grid systems and various trust management systems, but it is tailored to exactly match the ONDC requirements. An important aspect of the proposed solution is a combination of locally issued identity based access control and certificates based delegated authorizations. While the identity based access control is the most intuitive way how to express trust for the users of the system, delegated authorizations are an answer for more complex cases. The unique feature of the proposed system is configurable mechanism for authorization distributions and storage that enables users to trade-off a local information availability with local storage requirements.

Using Bootstraping Principles of Contemporary P2P File-Sharing Protocols in Large-Scale Grid Computing Systems

As contemporary distributed systems reach their scalability limits, their architects search ways to push their boundaries further. One of the approaches is to convert a given distributed system into a peer-to-peer (P2P) structure. This approach causes more overhead compared to a single-master or multi-master architecture, but on larger scales (if properly designed) does not reach any boundaries. The Clondike project started as a grid computing system, but now it aims to create a universal non-dedicated P2P cluster where every participating node can benefit from its membership in the cluster. The P2P approach eliminates the single-point-of-failure problems and high availability is guaranteed by design. So far in the Clondike architecture, a bootstrap protocol was based on broadcasts, which became a limiting factor for the scalability of the whole cluster. We have compared existing P2P communication protocols used in a variety of P2P systems and contemporary P2P file-sharing protocols in order to find a suitable solution for Clondike. We have chosen Kademlia protocol and proven its logaritmic scalability in the Clondike environment. This paper presents results of our analysis and implementation and a series of measurement results comparing the old and new communication protocols.