Yu Jiadi

Distributed high-performance web crawler based on peer-to-peer network

Distributing the crawling activity among multiple machines can distribute processing to reduce the analysis of web page. This paper presents the design of a distributed web crawler based on Peer-to-Peer network. The distributed crawler harnesses the excess bandwidth and computing resources of nodes in system to crawl the web. Each crawler is deployed in a computing node of P2P to analyze web page and generate indices. Control node is another node to being in charge of distributing URLs to balance the load of the crawler. Control nodes are organized as P2P network. The crawler nodes managed by the same control node is a group. According to the ID of crawler and average load of the group, crawler can decide whether transmits the URL to control node or hold itself. We present an implementation of the distributed crawler based on Igloo and simulate the environment to evaluate the balancing load on the crawlers and crawl speed.Distributing the crawling activity among multiple machines can distribute processing to reduce the analysis of web page. This paper presents the design of a distributed web crawler based on Peer-to-Peer network. The distributed crawler harnesses the excess bandwidth and computing resources of nodes in system to crawl the web. Each crawler is deployed in a computing node of P2P to analyze web page and generate indices. Control node is another node to being in charge of distributing URLs to balance the load of the crawler. Control nodes are organized as P2P network. The crawler nodes managed by the same control node is a group. According to the ID of crawler and average load of the group, crawler can decide whether transmits the URL to control node or hold itself. We present an implementation of the distributed crawler based on Igloo and simulate the environment to evaluate the balancing load on the crawlers and crawl speed.

Differentially private multidimensional data publication

Multidimensional data provides enormous opportunities in a variety of applications. Recent research has indicated the failure of existing sanitization techniques (e.g., k-anonymity) to provide rigorous privacy guarantees. Privacy-preserving multidimensional data publishing currently lacks a solid theoretical foundation. It is urgent to develop new techniques with provable privacy guarantees. ∈-Differential privacy is the only method that can provide such guarantees. In this paper, we propose a multidimensional data publishing scheme that ensures ∈-differential privacy while providing accurate results for query processing. The proposed solution applies nonstandard wavelet transforms on the raw multidimensional data and adds noise to guarantee ∈-differential privacy. Then, the scheme processes arbitrarily queries directly in the noisy wavelet-coefficient synopses of relational tables and expands the noisy wavelet coefficients back into noisy relational tuples until the end result of the query. Moreover, experimental results demonstrate the high accuracy and effectiveness of our approach.