Lin Shang

Minimum cost attribute reduction in decision-theoretic rough set models

In classical rough set models, attribute reduction generally keeps the positive or non-negative regions unchanged, as these regions do not decrease with the addition of attributes. However, the monotonicity property in decision-theoretic rough set models does not hold. This is partly due to the fact that all regions are determined according to the Bayesian decision procedure. Consequently, it is difficult to evaluate and interpret region-preservation attribute reduction in decision-theoretic rough set models. This paper provides a new definition of attribute reduct for decision-theoretic rough set models. The new attribute reduction is formulated as an optimization problem. The objective is to minimize the cost of decisions. Theoretical analysis shows the meaning of the optimization problem. Both the problem definition and the objective function have good interpretation. A heuristic approach, a genetic approach and a simulated annealing approach to the new problem are proposed. Experimental results on several data sets indicate the efficiency of these approaches.

On an optimization representation of decision-theoretic rough set model

Decision-theoretic rough set model can derive several probabilistic rough set models by providing proper cost functions. Learning cost functions from data automatically is the key to improving the applicability of decision-theoretic rough set model. Many region-related attribute reductions are not appropriate for probabilistic rough set models as the monotonic property of regions does not always hold. In this paper, we propose an optimization representation of decision-theoretic rough set model. An optimization problem is proposed by considering the minimization of the decision cost. Two significant inferences can be drawn from the solution of the optimization problem. Firstly, cost functions and thresholds used in decision-theoretic rough set model can be learned from the given data automatically. An adaptive learning algorithm and a genetic algorithm are designed. Secondly, a minimum cost attribute reduction can be defined. The attribute reduction is interpreted as finding the minimal attribute set to make the decision cost minimum. A heuristic approach and a particle swarm optimization approach are also proposed. The optimization representation can bring some new insights into the research on decision-theoretic rough set model.

Generalized attribute reduct in rough set theory

Attribute reduction plays an important role in the areas of rough sets and granular computing. Many kinds of attribute reducts have been defined in previous studies. However, most of them concentrate on data only, which result in the difficulties of choosing appropriate attribute reducts for specific applications. It would be ideal if we could combine properties of data and user preference in the definition of attribute reduct. In this paper, based on reviewing existing definitions of attribute reducts, we propose a generalized attribute reduct which not only considers the data but also user preference. The generalized attribute reduct is the minimal subset which satisfies a specific condition defined by users. The condition is represented by a group of measures and a group of thresholds, which are relevant to user requirements or real applications. For the same data, different users can define different reducts and obtain their interested results according to their applications. Most current attribute reducts can be derived from the generalized reduct. Several reduction approaches are also summarized to help users to design their appropriate reducts.

Binary particle swarm optimisation for feature selection: A filter based approach

Based on binary particle swarm optimisation (BPSO) and information theory, this paper proposes two new filter feature selection methods for classification problems. The first algorithm is based on BPSO and the mutual information of each pair of features, which determines the relevance and redundancy of the selected feature subset. The second algorithm is based on BPSO and the entropy of each group of features, which evaluates the relevance and redundancy of the selected feature subset. Different weights for the relevance and redundancy in the fitness functions of the two proposed algorithms are used to further improve their performance in terms of the number of features and the classification accuracy. In the experiments, a decision tree (DT) is employed to evaluate the classification accuracy of the selected feature subset on the test sets of four datasets. The results show that with proper weights, two proposed algorithms can significantly reduce the number of features and achieve similar or even higher classification accuracy in almost all cases. The first algorithm usually selects a smaller feature subset while the second algorithm can achieve higher classification accuracy.

Three-Way Decisions Solution to Filter Spam Email: An Empirical Study

A three-way decisions solution based on Bayesian decision theory for filtering spam emails is examined in this paper. Compared to existed filtering systems, the spam filtering is no longer viewed as a binary classification problem. Each incoming email is accepted as a legitimate or rejected as a spam or undecided as a further-exam email by considering the misclassification cost. The three-way decisions solution for spam filtering can reduce the error rate of classifying a legitimate email to spam, and provide a more meaningful decision procedure for users. The solution is not restricted to a specific classifier. Experimental results on several corpus show that the three-way decisions solution can get a better total cost ratio value and a lower weighted error.

A multi-objective particle swarm optimisation for filter-based feature selection in classification problems

Feature selection has the two main objectives of minimising the classification error rate and the number of features. Based on binary particle swarm optimisation (BPSO), we develop two novel multi-objective feature selection frameworks for classification, which are multi-objective binary PSO using the idea of non-dominated sorting (NSBPSO) and multi-objective binary PSO using the ideas of crowding, mutation and dominance (CMDBPSO). Four multi-objective feature selection methods are then developed by applying mutual information and entropy as two different filter evaluation criteria in each of the proposed frameworks. The proposed algorithms are examined and compared with a single objective method on eight benchmark data sets. Experimental results show that the proposed multi-objective algorithms can evolve a set of solutions that use a smaller number of features and achieve better classification performance than using all features. In most cases, NSBPSO achieves better results than the single objective algorithm and CMDBPSO outperforms all other methods mentioned above. This work represents the first study on multi-objective BPSO for filter-based feature selection.

An optimization viewpoint of decision-theoretic rough set model

This paper considers an optimization viewpoint of decisiontheoretic rough set model. An optimization problem is proposed by considering the minimization of the decision cost. Based on the optimization problem, cost functions and thresholds used in decision-theoretic rough set model can be learned from the given data automatically. An adaptive learning algorithm Alcofa is proposed. Another significant inference drawn from the solution of the optimization problem is a minimum cost based attribute reduction. The attribute reduction can be interpreted as finding the minimal attribute set to make the decision cost minimum. The optimization viewpoint can bring some new insights into the research on decision-theoretic rough set model.

BINARY PSO AND ROUGH SET THEORY FOR FEATURE SELECTION: A MULTI-OBJECTIVE FILTER BASED APPROACH

Feature selection is a multi-objective problem, where the two main objectives are to maximize the classification accuracy and minimize the number of features. However, most of the existing algorithms belong to single objective, wrapper approaches. In this work, we investigate the use of binary particle swarm optimization (BPSO) and probabilistic rough set (PRS) for multi-objective feature selection. We use PRS to propose a new measure for the number of features based on which a new filter based single objective algorithm (PSOPRSE) is developed. Then a new filter-based multi-objective algorithm (MORSE) is proposed, which aims to maximize a measure for the classification performance and minimize the new measure for the number of features. MORSE is examined and compared with PSOPRSE, two existing PSO-based single objective algorithms, two traditional methods, and the only existing BPSO and PRS-based multi-objective algorithm (MORSN). Experiments have been conducted on six commonly used discrete datasets with a relative small number of features and six continuous datasets with a large number of features. The classification performance of the selected feature subsets are evaluated by three classification algorithms (decision trees, Naive Bayes, and k-nearest neighbors). The results show that the proposed algorithms can automatically select a smaller number of features and achieve similar or better classification performance than using all features. PSOPRSE achieves better performance than the other two PSO-based single objective algorithms and the two traditional methods. MORSN and MORSE outperform all these five single objective algorithms in terms of both the classification performance and the number of features. MORSE achieves better classification performance than MORSN. These filter algorithms are general to the three different classification algorithms.

MULTI-OBJECTIVE EVOLUTIONARY ALGORITHMS FOR FILTER BASED FEATURE SELECTION IN CLASSIFICATION

Feature selection is a multi-objective problem with the two main conflicting objectives of minimising the number of features and maximising the classification performance. However, most existing feature selection algorithms are single objective and do not appropriately reflect the actual need. There are a small number of multi-objective feature selection algorithms, which are wrapper based and accordingly are computationally expensive and less general than filter algorithms. Evolutionary computation techniques are particularly suitable for multi-objective optimisation because they use a population of candidate solutions and are able to find multiple non-dominated solutions in a single run. However, the two well-known evolutionary multi-objective algorithms, nondominated sorting based multi-objective genetic algorithm II (NSGAII) and strength Pareto evolutionary algorithm 2 (SPEA2) have not been applied to filter based feature selection. In this work, based on NSGAII and SPEA2, we develop two multi-objective, filter based feature selection frameworks. Four multi-objective feature selection methods are then developed by applying mutual information and entropy as two different filter evaluation criteria in each of the two proposed frameworks. The proposed multi-objective algorithms are examined and compared with a single objective method and three traditional methods (two filters and one wrapper) on eight benchmark datasets. A decision tree is employed to test the classification performance. Experimental results show that the proposed multi-objective algorithms can automatically evolve a set of non-dominated solutions that include a smaller number of features and achieve better classification performance than using all features. NSGAII and SPEA2 outperform the single objective algorithm, the two traditional filter algorithms and even the traditional wrapper algorithm in terms of both the number of features and the classification performance in most cases. NSGAII achieves similar performance to SPEA2 for the datasets that consist of a small number of features and slightly better results when the number of features is large. This work represents the first study on NSGAII and SPEA2 for filter feature selection in classification problems with both providing field leading classification performance.

XCSc: A NOVEL APPROACH TO CLUSTERING WITH EXTENDED CLASSIFIER SYSTEM

In this paper, we propose a novel approach to clustering noisy and complex data sets based on the eXtend Classifier Systems (XCS). The proposed approach, termed XCSc, has three main processes: (a) a learning process to evolve the rule population, (b) a rule compacting process to remove redundant rules after the learning process, and (c) a rule merging process to deal with the overlapping rules that commonly occur between the clusters. In the first process, we have modified the clustering mechanisms of the current available XCS and developed a new accelerate learning method to improve the quality of the evolved rule population. In the second process, an effective rule compacting algorithm is utilized. The rule merging process is based on our newly proposed agglomerative hierarchical rule merging algorithm, which comprises the following steps: (i) all the generated rules are modeled by a graph, with each rule representing a node; (ii) the vertices in the graph are merged to form a number of sub-graphs (i.e. rule clusters) under some pre-defined criteria, which generates the final rule set to represent the clusters; (iii) each data is re-checked and assigned to a cluster that it belongs to, guided by the final rule set. In our experiments, we compared the proposed XCSc with CHAMELEON, a benchmark algorithm well known for its excellent performance, on a number of challenging data sets. The results show that the proposed approach outperforms CHAMELEON in the successful rate, and also demonstrates good stability.