Yang Shuqiang

A MapReduce and Information Compression Based Social Community Structure Mining Method

As the rapid development of social media, social community structure mining has become a popular research field in recent years. But traditional social community mining methods are not able to effectively deal with the data of large scale networks. We firstly introduce an information compression based community mining model in this paper, and with the help of the model, we transform the community mining problem into optimal information coding problem. And then propose a parallel computing method CInfoMR based on the MapReduce parallel framework to mine the social community structure. In the InfoMR, map tasks are responsible for splitting network data into a plenty of subsets, each reduce task is responsible for accomplishing community clustering by means of loop iteration on its subset, and finally all the results from the reduce phase are merged together to output. Theoretical analysis and related experiments verify the validity of the work in this paper. The results of the accuracy experiments show that, the accuracy of the InfoMR is much higher than that of Fast GN and PDST algorithm. The performance experiments on 2 real dataset and 2 simulative dataset show that InfoMR is able to accomplish the task of mining social community in a relatively short period of time on big data social networks.

Research and implement of real-time data loading system IMIL

With rapid development of the Internet and communication technology, massive data has been accumulated in many web-based applications such as deep web applications and web search engines. Increasing data volumes pose enormous challenges to data-loading techniques. This paper presents a data loading system in real time, the IMIL (Internet Monitoring Information Loader) that is used in RT-IMIS (Real-time Internet Monitoring Information System), which monitors real-time internet flux, manages network security, and collects a mass of Internet real-time information. IMIL consists of an extensible fault-tolerant hardware architecture, an efficient algorithm for bulk data loading using SQL*Loader and exchange partition mechanism, optimized parallelism, and guidelines for system tuning. Performance studies show the positive effects of these techniques with loading speed of every Cluster, increasing from 220 million records per day to 1.2 billion per day, and achieving the top loading speed of 6TB data when 10 Clusters are in parallel. This framework offers a promising approach for loading other large and complex databases.