Izabela Szczęch

Multicriteria Attractiveness Evaluation of Decision and Association Rules

The work is devoted to multicriteria approaches to rule evaluation. It analyses desirable properties (in particular the property M, property of confirmation and hypothesis symmetry) of popular interestingness measures of decision and association rules. Moreover, it analyses relationships between the considered interestingness measures and enclosure relationships between the sets of non-dominated rules in different evaluation spaces. Its main result is a proposition of a multicriteria evaluation space in which the set of non-dominated rules will contain all optimal rules with respect to any attractiveness measure with the property M. By determining the area of rules with desirable value of a confirmation measure in the proposed multicriteria evaluation space one can narrow down the set of induced rules only to the valuable ones. Furthermore, the work presents an extension of an apriori-like algorithm for generation of rules with respect to attractiveness measures possessing valuable properties and shows some applications of the results to analysis of rules induced from exemplary datasets.

Analysis of symmetry properties for bayesian confirmation measures

The paper considers symmetry properties of Bayesian confirmation measures, which constitute an important group of interestingness measures for evaluation of rules induced from data. We demonstrate that the symmetry properties proposed in the literature do not fully reflect the concept of confirmation. We conduct a thorough analysis of the symmetries regarding that the confirmation should express how much more probable the rules hypothesis is when the premise is present rather than when the premise is absent. As a result we point out which symmetries are desired for Bayesian confirmation measures and which are truly unattractive. Such knowledge is a valuable tools for assessing the quality and usefulness of measures.

Measures of rule interestingness in various perspectives of confirmation

Confirmation is a useful concept for assessing the impact of the premise on the conclusion of a rule induced from data. Interpretation of probabilistic relationships between premise and conclusion of a rule led to four mathematical formulations of confirmation, called perspectives. The logical equivalence of these perspectives and the resulting general definition of confirmation underline the known qualitative aspect of the concept of confirmation. The quantitative aspect of confirmation is handled by definitions of particular confirmation measures. In this paper, we relate the qualitative and quantitative aspects by introducing a property of monotonicity of measures with respect to left- and right-hand side probabilities defining the perspectives. This new property permits consideration of confirmation measures in association with particular perspectives. We also identify several other properties that valuable confirmation measures should possess. Particular care is devoted to discussion of behavior of confirmation measures monotonic in different perspectives with respect to symmetry properties, taking also into account two new perspectives of Bayesian confirmation. We also prove that confirmation measures monotonic in the six perspectives are exhaustive in the sense that their set is closed under transformations related to symmetry properties. Finally, we verify which confirmation measures enjoy these properties.

Finding Meaningful Bayesian Confirmation Measures

The paper focuses on Bayesian confirmation measures used for evaluation of rules induced from data. To distinguish between many confirmation measures, their properties are analyzed. The article considers a group of symmetry properties. We demonstrate that the symmetry properties proposed in the literature focus on extreme cases corresponding to entailment or refutation of the rules conclusion by its premise, forgetting intermediate cases. We conduct a thorough analysis of the symmetries regarding that the confirmation should express how much more probable the rules hypothesis is when the premise is present rather than when the negation of the premise is present. As a result we point out which symmetries are desired for Bayesian confirmation measures. Next, we analyze a set of popular confirmation measures with respect to the symmetry properties and other valuable properties, being monotonicity M, Ex1 and weak Ex1, logicality L and weak L. Our work points out two measures to be the most meaningful ones regarding the considered properties.

Visual-Based Detection of Properties of Confirmation Measures

The paper presents a visualization technique that facilitates and eases analyses of interestingness measures with respect to their properties. Detection of properties possessed by these measures is especially important when choosing a measure for KDD tasks. Our visual-based approach is a useful alternative to often laborious and time consuming theoretical studies, as it allows to promptly perceive properties of the visualized measures. Assuming a common, four-dimensional domain of the measures, a synthetic dataset consisting of all possible contingency tables with the same number of observations is generated. It is then visualized in 3D using a tetrahedron-based barycentric coordinate system. Additional scalar function - an interestingness measure - is rendered using colour. To demonstrate the capabilities of the proposed technique, we detect properties of a particular group of measures, known as confirmation measures.

Alternative normalization schemas for Bayesian confirmation measures

Analysis of rule interestingness measures with respect to their properties is an important research area helping to identify groups of measures that are truly meaningful. In this article, we analyze property Ex1, of preservation of extremes, in a group of confirmation measures. We consider normalization as a mean to transform them so that they would obtain property Ex1 and we introduce three alternative approaches to the problem: an approach inspired by Nicod, Bayesian, and likelihoodist approach. We analyze the results of the normalizations of seven measures with respect to property Ex1 and show which approaches lead to the desirable results. Moreover, we extend the group of ordinally non-equivalent measures possessing valuable property Ex1.

Selected group-theoretic aspects of confirmation measure symmetries

The paper considers symmetry properties of rule interestingness measures (in particular: measures of confirmation). Many authors have studied various symmetries, however the discussion on which sets of symmetries should be taken into account, let alone why the particular symmetries are desirable or not, has still not produced a generally recognized consensus. Furthermore, the results published so far neglect the fact that symmetries can be the subject of group theory-based considerations. This paper aims at solving those problems by introducing group-theoretic interpretations of symmetries, indicating that all symmetries can be treated as permutations, and compositions of symmetries as compositions of permutations. Such an interpretation allows us to apply the well-known group-theoretic results to symmetries. In particular, using this approach, we reveal the phenomenon of incompleteness occurring in sets of symmetries considered by different authors, and propose an effective way of controlling it. Moreover, we demonstrate that assessing the symmetries as either desirable or undesirable, as introduced by these authors, brings in inconsistencies, which become evident under the permutation-based interpretation. Finally, we present group-theoretic guidelines to the design of such symmetry sets that are free of the incompleteness and inconsistency phenomena but remain meaningful in the context of rule evaluation.

Visual-based analysis of classification measures and their properties for class imbalanced problems

Abstract With a plethora of available classification performance measures, choosing the right metric for the right task requires careful thought. To make this decision in an informed manner, one should study and compare general properties of candidate measures. However, analysing measures with respect to complete ranges of their domain values is a difficult and challenging task. In this study, we attempt to support such analyses with a specialized visualisation technique, which operates in a barycentric coordinate system using a 3D tetrahedron. Additionally, we adapt this technique to the context of imbalanced data and put forward a set of measure properties, which should be taken into account when examining a classification performance measure. As a result, we compare 22 popular measures and show important differences in their behaviour. Moreover, for parametric measures such as the Fβ and IBAα(G-mean), we analytically derive parameter thresholds that pinpoint the changes in measure properties. Finally, we provide an online visualisation tool that can aid the analysis of measure variability throughout their entire domains.With a plethora of available classification performance measures, choosing the right metric for the right task requires careful thought. However, analyzing measures with respect to complete ranges of their values is a difficult and challenging task. In this study, we attempt to support such analyses with a specialized visualization technique, which operates in a barycentric coordinate system using a 3D tetrahedron. Additionally, we adapt this technique to the context of imbalanced data and put forward a set of properties which should be taken into account when selecting a classification performance measure. As a result, we compare 21 popular measures and show important differences in their behavior. Finally, we provide an online visualization tool that can aid the analysis of complete ranges of performance measures.

Can Confirmation Measures Reflect Statistically Sound Dependencies in Data? The Concordance-based Assessment

Abstract The paper considers particular interestingness measures, called confirmation measures (also known as Bayesian confirmation measures), used for the evaluation of “if evidence, then hypothesis” rules. The agreement of such measures with a statistically sound (significant) dependency between the evidence and the hypothesis in data is thoroughly investigated. The popular confirmation measures were not defined to possess such form of agreement. However, in error-prone environments, potential lack of agreement may lead to undesired effects, e.g. when a measure indicates either strong confirmation or strong disconfirmation, while in fact there is only weak dependency between the evidence and the hypothesis. In order to detect and prevent such situations, the paper employs a coefficient allowing to assess the level of dependency between the evidence and the hypothesis in data, and introduces a method of quantifying the level of agreement (referred to as a concordance) between this coefficient and the measure being analysed. The concordance is characterized and visualised using specialized histograms, scatter-plots, etc. Moreover, risk-related interpretations of the concordance are introduced. Using a set of 12 confirmation measures, the paper presents experiments designed to establish the actual concordance as well as other useful characteristics of the measures.

Tetrahedron: Barycentric Measure Visualizer

Each machine learning task comes equipped with its own set of performance measures. For example, there is a plethora of classification measures that assess predictive performance, a myriad of clustering indices, and equally many rule interestingness measures. Choosing the right measure requires careful thought, as it can influence model selection and thus the performance of the final machine learning system. However, analyzing and understanding measure properties is a difficult task. Here, we present Tetrahedron, a web-based visualization tool that aids the analysis of complete ranges of performance measures based on a two-by-two contingency matrix. The tool operates in a barycentric coordinate system using a 3D tetrahedron, which can be rotated, zoomed, cut, parameterized, and animated. The application is capable of visualizing predefined measures (86 currently), as well as helping prototype new measures by visualizing user-defined formulas.