Lijun Cai

Big data visualization collaborative filtering algorithm based on RHadoop

With the rapid growth of various data, it is becoming increasingly important to extract useful information from big data. While the analysis tools of big data visualization is very rare, in this paper, we propose a new big data visualization algorithm analysis integrated model. The model integrates the processing of big data and the visualization of data as a whole. It is a good analysis tool of timely big data visualization. We use hadoop_1.X as the data storage and use R as the compiler environment in the model. If you are skilled in R, it is easy to design kinds of paralleling algorithms, and analyze and process the kinds of big data. Secondly we design and implement a paralleled collaborative filtering algorithm with the model. Finally we analyze the various performance indicators with kinds of experiments. The indicators show that the model has good scalability and easy operability, and contains all the advantages of Map Reduce. In conclusion, the big data visualization algorithm analysis integrated model has high performance to process and visualize the big data.

MTAD: a multitarget heuristic algorithm for virtual machine placement

Cloud data centers are facing increasingly virtual machine (VM) placement problems, such as high energy consumption, imbalanced utilization of multidimension resource, and high resource wastage rate. In order to solve the virtual machine placement problems in large scale, three algorithms are proposed. Firstly, we propose a physical machine (PM) classification algorithm by analyzing pseudotime complexity and find out an important factor (the number of physical hosts) that affects the efficiency, which improves running efficiency through reduction number of physical hosts; secondly, we present a VM placement optimization model using multitarget heuristic algorithm and figure out the positive and negative vectors of three goals using matrix transformation so as to provide the mapping of VMs to hosts by comparing distance with positive and negative vectors such that the energy consumption is saved, resources wastage of occupied PM is lowered, multidimension resource utilization is optimized, and the running time is shortened. Finally, we consider the poor placement efficiency problem of large-scale virtual serial requests and design a concurrent VM classification algorithm using the K-means method. Simulation experiments validate the performance of the algorithm in four aspects, including placement efficiency, resources utilization balance rate, wastage rate, and energy consumption.

Fast Community Detection Based on Distance Dynamics

The distance dynamics model is excellent tool for uncovering the community structure of a complex network. However, one issue that must be addressed by this model is its very long computation time in large-scale networks. To identify the community structure of a large-scale network with high speed and high quality, in this paper, we propose a fast community detection algorithm, the F-Attractor, which is based on the distance dynamics model. The main contributions of the F-Attractor are as follows. First, we propose the use of two prejudgment rules from two different perspectives: node and edge. Based on these two rules, we develop a strategy of internal edge prejudgment for predicting the internal edges of the network. Internal edge prejudgment can reduce the number of edges and their neighbors that participate in the distance dynamics model. Second, we introduce a triangle distance to further enhance the speed of the interaction process in the distance dynamics model. This triangle distance uses two known distances to measure a third distance without any extra computation. We combine the above techniques to improve the distance dynamics model and then describe the community detection process of the F-Attractor. The results of an extensive series of experiments demonstrate that the F-Attractor offers high-speed community detection and high partition quality.

An Improved Community Detection Algorithm Based on the Distance Dynamics

To solve the slow convergence or non-convergence problem of traditional distance dynamics method, an improved community detection algorithm based on distance dynamics is proposed in this paper for speeding up the time efficient and accuracy of algorithm. Our improved algorithm firstly analyzes the node cohesiveness and slow convergence problems. Secondly, the neighbor cohesion is defined and two new interaction patterns are designed to enhance the accuracy of community. Finally, the convergence coefficient is defined to judge whether the dynamic interaction process is in the slow convergence or non-convergence status. In the dynamic interaction process, when the percentage of non-converged distance is less than convergence coefficient, the final value of the distances are pre-judged and the whole algorithm is over. The experimental results show our improved algorithm has a fast convergent speed and accurate detection results.

Locality-Aware and Energy-Aware Job Pre-Assignment for Mapreduce

Cloud Map-Reduce (CMR) is an advantage Map-Reduce platform and has been aroused more and more attention. To further balance the performance of job secluding among job cost, execution time and energy consumption, a locality-aware and energy-aware job pre-assignment algorithm is proposed for Map-Reduce of CMR in this paper. Firstly, the importance of rack in data locality and energy saving is analyzed. Secondly, a capacity pre-judged method is developed to measure the idea capacity of one rack for different jobs where the energy-efficient is defined to measure the balance statues of rack usage among job cost, execution time and energy consumption in job scheduling. Finally, based on the pre-judged idea capacity of racks, job pre-assignment method is proposed to centrally assign one job to virtual machines of several booked racks for saving energy and reducing communication. By comparing with other three algorithms, the extensive experimental results show our algorithm has good performance on job execution time, cross rack traffic, and energy consumption.

Queuing-Oriented Job Optimizing Scheduling In Cloud Mapreduce

Cloud MapReduce, as an implementation of MapReduce framework on Cloud for big data analysis, is facing the unknown job makespan and long wait time problem, which have seriously affected the service quality. The Inefficient virtual machine allocation is one critical causing factor. Based on the M/M/1 model, a new queuing equation is built to ensure the virtual machine with the high efficiency. By jointing queuing equation and objectives function, a two variables equation group is designed to compute the desired virtual machine number for different jobs. According to the desired virtual machine number of each job, we developed a queuing-oriented job optimizing scheduling algorithm, called QTJS, to optimal job scheduling and enhance the resource utilization in Cloud MapReduce. Extensive experiments show that our QTJS algorithm consumes less job execution time and performs better efficiency than other three algorithms.

K-Hop Community Search Based on Local Distance Dynamics

Community search aims at finding a meaningful community that contains the query node and also maximizes (minimizes) a goodness metric, which has attracted a lot of attention in recent years. However, most of existing metric-based algorithms either tend to include the irrelevant subgraphs in the identified community or have computational bottleneck. Contrary to the user-defined metric algorithm, how can we search the natural community that the query node belongs to? In this paper, we propose a novel community search algorithm based on the concept of k-hop and local distance dynamics model, which can natural capture a community that contains the query node. Extensive experiments on large real-world networks with ground-truth demonstrate the effectiveness and efficiency of our community search algorithm and has good performance compared to state-of-the-art algorithm.

Overlapping community detection based on link graph using distance dynamics

The distance dynamics model was recently proposed to detect the disjoint community of a complex network. To identify the overlapping structure of a network using the distance dynamics model, an overlapping community detection algorithm, called L-Attractor, is proposed in this paper. The process of L-Attractor mainly consists of three phases. In the first phase, L-Attractor transforms the original graph to a link graph (a new edge graph) to assure that one node has multiple distances. In the second phase, using the improved distance dynamics model, a dynamic interaction process is introduced to simulate the distance dynamics (shrink or stretch). Through the dynamic interaction process, all distances converge, and the disjoint community structure of the link graph naturally manifests itself. In the third phase, a recovery method is designed to convert the disjoint community structure of the link graph to the overlapping community structure of the original graph. Extensive experiments are conducted on the LF...

Active Learning Method for Constraint-Based Clustering Algorithms

Semi-supervision clustering aims to improve clustering performance with the help of user-provided side information. The pairwise constraints have become one of the most studied types of side information. According to the previous studies, such constraints increase clustering performance, but the choice of constraints is critical. If the constraints are selected improperly, they may even degrade the clustering performance. In order to solve this problem, researchers proposed some learning methods to actively select most informative pairwise constraints. In this paper, we presents a new active learning method for selecting informative data set, which significantly improves both the Explore phase and the Consolidate phase of the Min-Max algorithm. Experimental results on the data set of UCI Machine Learning Repository, using MPCK-means as the underlying constraint-based semi-supervised clustering algorithm, show that the proposed algorithm has better performance.