Hussein Almuallim

Learning Boolean concepts in the presence of many irrelevant features

Abstract In many domains, an appropriate inductive bias is the MIN-FEATURES bias, which prefers consistent hypotheses definable over as few features as possible. This paper defines and studies this bias in Boolean domains. First, it is shown that any learning algorithm implementing the MIN-FEATURES bias requires ⊖(( ln ( l δ ) + [2 p + p ln n])/e) training examples to guarantee PAC-learning a concept having p relevant features out of n available features. This bound is only logarithmic in the number of irrelevant features. For implementing the MIN-FEATURES bias, the paper presents five algorithms that identify a subset of features sufficient to construct a hypothesis consistent with the training examples. FOCUS-1 is a straightforward algorithm that returns a minimal and sufficient subset of features in quasi-polynomial time. FOCUS-2 does the same task as FOCUS-1 but is empirically shown to be substantially faster than FOCUS-1. Finally, the Simple-Greedy, Mutual-Information-Greedy and Weighted-Greedy algorithms are three greedy heuristics that trade optimality for computational efficiency. Experimental studies are presented that compare these exact and approximate algorithms to two well-known algorithms, ID3 and FRINGE, in learning situations where many irrelevant features are present. These experiments show that—contrary to expectations—the ID3 and FRINGE algorithms do not implement good approximations of MIN-FEATURES. The sample complexity and generalization performance of the FOCUS algorithms is substantially better than either ID3 or FRINGE on learning problems where the MIN-FEATURES bias is appropriate. These experiments also show that, among our three heuristics, the Weighted-Greedy algorithm provides an excellent approximation to the FOCUS algorithms.

An efficient algorithm for optimal pruning of decision trees

Abstract Pruning decision trees is a useful technique for improving the generalization performance in decision tree induction, and for trading accuracy for simplicity in other applications. In this paper, a new algorithm called OPT-2 for optimal pruning of decision trees is introduced. The algorithm is based on dynamic programming. In its most basic form, the time and space complexities of OPT-2 are both Θ (nC), where n is the number of test nodes in the initial decision tree, and C is the number of leaves in the target (pruned) decision tree. This is an improvement over the recently published OPT algorithm of Bohanec and Bratko (which is the only known algorithm for optimal decision tree pruning) especially in the case of heavy pruning and when the tests of the given decision tree have many outcomes. If so desired, the space required by OPT-2 can further be reduced by a factor of r at the cost of increasing the execution time by a factor that is bounded above by (r + 1) 2 (this is a considerable overestimate, however). From a practical point of view, OPT-2 enjoys considerable flexibility in various aspects, and is easy to implement.

Two methods for learning ALT-J/E translation rules from examples and a semantic hierarchy

This paper presents our work towards the automatic acquisition of translation rules from Japanese-English translation examples for NTTs ALT-J/E machine translation system. We apply two machine learning algorithms: Hausslers algorithm for learning internal disjunctive concept and Quinlans ID3 algorithm. Experimental results show that our approach yields rules that are highly accurate compared to the manually created rules.

3 DEVELOPMENT AND APPLICATIONS OF DECISION TREES

Publisher Summary This chapter presents a basic method for automatically constructing decision trees from examples. It reviews various extensions of this basic procedure. The chapter provides a sample of real-world applications for which the decision tree learning approach has been shown to be successful. Considerable effort has been put to develop methods that induce the desired classification knowledge from a given set of pre-classified examples. Constructing classifiers in the form of decision trees has obtained much popularity. Decision trees have the advantage of being comprehensible by human experts and of being directly convertible into production rules. When used to handle a given case, a decision tree not only gives the solution for that case, but also mentions the reasons behind its choice. These features are very important in typical application domains in which human experts seek tools to help them in performing their job. Another advantage of using decision trees is the ease and efficiency of their construction compared to that of other classifiers such as neural networks.

Turning majority voting classifiers into a single decision tree

This paper addresses the issues of intelligibility, classification speed, and required space in majority voting classifiers. Methods that classify unknown cases using multiple classifiers (e.g. bagging, boosting) have been actively studied in recent years. Since these methods classify a case by taking majority voting over the classifiers, the reasons behind the decision cannot be described in a logical form. Moreover, a large number of classifiers is needed to significantly improve the accuracy. This greatly increases the amount of time and space needed in classification. To solve these problems, a method for learning a single decision tree that approximates the majority voting classifiers is proposed in this paper. The proposed method generates if-then rules from each classifier, and then learns a single decision tree from these rules. Experimental results show that the decision trees by our method are considerably compact and have similar accuracy compared to bagging. Moreover, the proposed method is 8 to 24 times faster than bagging in classification.

Simplification of majority‐voting classifiers using binary decision diagrams

Various versions of the majority-voting classification method have been proposed in recent years as a strategy for improving classification performance. This method generates multiple decision trees from training examples and performs majority voting of classification results from these decision trees in order to classify test examples. In this method, however, since the target concept is represented in multiple decision trees, its readability is poor. This property makes it ineffective in knowledge-base construction. To enable the majority-voting classification method to be applied to knowledge-base construction, this paper proposes a simplification method that converts the entire majority-voting classifier into compact disjunctive normal form (DNF) formulas. A significant feature of this method is the use of binary decision diagrams (BDDs) as internal expressions in the conversion process to achieve high-speed simplification. A problem that must be addressed here is the BDD input variable ordering scheme. This paper proposes an ordering scheme based on the order of variables in the decision trees. The simplification method has been applied to several real-world data sets of the Irvine Database and to data from medical diagnosis domain. It was found that the description size of the majority-voting classifier after simplification was on the average from 1.2 to 2.7 times that of a single decision tree and was less than one-third the size of a majority-voting classifier before simplification. Therefore, the method is effective in reducing the description size and should be applicable to the knowledge acquisition process. Using the input variable ordering scheme proposed here, high-speed simplification of several seconds to several tens of seconds is achieved on a Sun SPARC-server 10 workstation.

A revision learner to acquire verb selection rules from human-made rules and examples

This paper proposes a learning method that automatically acquires English verb selection rules for machine translation using a machine learning technique. When learning from real translation examples alone, many examples are needed to achieve good translation quality. It is, however, difficult to gather a sufficiently large number of real translation examples. The main causes are verbs of low frequency and the frequent usage of the same sentences. To resolve this problem, the proposed method learns English verb selection rules from hand-made translation rules and a small number of real translation examples. The proposed method has two steps: generating artificial examples from the hand-made rules, and then putting those artificial examples and real examples into an internal learner as the training set. The internal learner outputs the final rules with improved verb selection accuracy. The most notable feature of the proposed learner is that any attribute-type learning algorithm can be adopted as the internal learner. To evaluate the validity of the proposed learner, English verb selection rules of NTTs Japanese-English Machine Translation System ALT-J/E are experimentally learned from hand-made rules and real examples. The resultant rules have better accuracy than either those constructed from the real examples or those that are hand-made.