Efficient Distributed k-clique Mining for Large Networks Using MapReduce

Abstract

Mining cliques of a network is an important problem that has many applications in different fields like social networks, bioinformatics, and web analysis. In most applications, mining fixed sized cliques, known as k-cliques, is enough. However, mining cliques of a large network is very challenging using current solutions, and it takes a considerable time using a commodity machine. Also, very large networks cannot be efficiently loaded into memory of a single machine. To overcome these limitations, we have proposed a solution named KCminer, which is based on state space search and can be totally fitted into the MapReduce framework. Using the MapReduce framework, it is possible to run KCminer on cloud computing platforms and hence, process very large networks in feasible time. Our experiments which were performed on a cloud computing platform with 100 machines show that KCminer is both fast and scalable. Besides the MapReduce framework, KCminer executes efficiently on parallel shared memory systems. We performed some experiments on a commodity multicore desktop and showed that KCminer can effectively use the power of all cores. The experimental results show that even using a single thread, KCminer is much faster than available serial tools like MACE.