Steven N. Thorsen

The ROC manifold for classification systems

We define the ROC manifold and CC manifold as duals in a given sense. Their analysis is required to describe the classification system. We propose a mathematical definition based on vector space methods to describe both. The ROC manifolds for n-class classification systems fully describe each system in terms of its misclassifications and, by conjunction, its correct classifications. Optimal points which minimize misclassifications can be identified even when costs and prior probabilities differ. These manifolds can be used to determine the usefulness of a classification system based on a given performance criterion. Many performance functionals (such as summary statistics) preferred for CC manifolds can also be evaluated using the ROC manifold (under certain constraints). Examples using the ROC manifold and performance functionals to compete classification systems are demonstrated with simulated and applied disease detection data.

A description of competing fusion systems

A mathematical description of fusion is presented using category theory. A category of fusion rules is developed. The category definition is derived for a model of a classification system beginning with an event set and leading to the final labeling of the event. Functionals on receiver operating characteristic (ROC) curves are developed to form a partial ordering of families of classification systems. The arguments of these functionals point to specific ROCs and, under various choices of input data, correspond to the Bayes optimal threshold (BOT) and the Neyman-Pearson threshold of the families of classification systems. The functionals are extended for use over ROC curves and ROC manifolds where the number of classes of interest in the fusion system exceeds two and the parameters used are multi-dimensional. Choosing a particular functional, therefore, provides the qualitative requirements to define a fusor and choose the best competing classification system.

A Boolean Algebra of receiver operating characteristic curves

A reasonable starting place for developing decision fusion rules of families of classification systems is using the logical AND and OR rules. These two rules, along with the unary rule NOT, can lead to a Boolean algebra when a number of properties are shown to exist. This paper examines how these rules for classification system families comprise a Boolean algebra of systems. This Boolean algebra of families is then shown under assumptions of independence to be isomorphic to a Boolean algebra of receiver operating characteristic (ROC) curves. These decision fusion rules produce ROC curves which become the bounds by which to test non-Boolean, possibly non-decision fusion rules for performance increases. We give an example to demonstrate the usefulness of this Boolean algebra of ROC curves.

ROC Manifolds of Multiple Fused Independent ATR Systems

n Automatic Target Recognition (ATR) system with N possible output labels (or decisions) will have N(N − 1) possible errors. The Receiver Operating Characteristic (ROC) manifold was created to quantify all of these errors. When multiple ATR systems are fused, the assumption of independence is usually made in order to mathematically combine the individual ROC manifolds for each system into one ROC manifold. This paper will investigate the label fusion (also called decision fusion) of multiple classification systems that have the same number of output labels. Boolean rules do not exist for multiple symbols, thus, we will derive possible Boolean-like rules as well as other rules that will yield label fusion rules. The formula for the resultant ROC manifold of the fused classification systems which incorporates the individual classification systems will be derived. Specifically, given a fusion rule and two classification systems, the ROC manifold for the fused system is produced. We generate formulas for the Boolean-like OR rule, Boolean-like AND rule, and other rules and give the resultant ROC manifold for the fused system. Examples will be given that demonstrate how each formula is used.

A category theory description of multisensor fusion

Data fusion as a science has been described in the literature in great detail by many authors, particularly over the last two decades. These descriptions are, for the vast majority, non-mathematical in nature and have lacked the symbolism and clarity of mathematical precision. This paper demonstrates a way of describing the science of data fusion using diagrams and category theory. The description begins using category theory to develop a clear definition of what fusion is in a mathematical sense. The definitions of fusion rules and fusors show how a notion of ”betterness” can be defined by developing appropriate functionals. Using a simple diagram of a multisensor process, an explanation of how receiver operating characteristic (ROC) curves can provide an appropriate functional to compare fusion rules, fusors, and even classifiers. A partial ordering of a finite number of fusors can then be created.

Quantifying the robustness of classification systems

Automatic Target Recognition (ATR) systems performance is quantified using Receiver Operating Characteristic (ROC) curves (or ROC manifolds for more than two labels) and typically the prior probabilities of each labeled-event occurring. In real-world problems, one does not know the prior probabilities and they have to be approximated or guessed, but usually one knows their range or distribution. We derive an objective functional that quantifies the robustness of an ATR system given: (1) a set of prior probabilities, and (2) a distribution of a set of prior probabilities. The ATR system may have two labels or more. We demonstrate the utility of this objective functional with examples, and show how it can be used to determine the optimal ATR system from a family of systems.

Describing data fusion using category theory

The process of multi-sensor data fusion and its taronomy is abstracted and described in the language of category theory. Categories are developed for sensors, data sets, processors, feature sets, classifiers, and label sets. Fusion rules are defined and shown to hold a unique role within the various categories. Fusion processes can then be described as the optimization affusion des within rhe appropriate categorj

Multi-performance fusion of classification systems

Given two legacy exploitation systems, whose performances are known, one might wish to determine if combining these together using some rule would yield a new exploitation system with improved performance. This is the fusion process. Often there are several performance objectives one would consider in this process. We investigate the fusion process based upon multiple performances. This is related to multi-objective optimization, but is different in some aspects. In this paper we pose a multi-performance problem for combining two classifications systems and derive the multi-performance fusion theory. A classification system with M possible output labels will have M(M-1) possible errors. The Receiver Operating Characteristic (ROC) manifold was created to quantify all of these errors. The assumption of independence is usually made to simply the mathematics of combining the individual systems into one system. Boolean rules do not exist for multiple symbols, thus, Boolean-like rules were created that would yield label fusion rules. An M-label system will have M! consistent rules. The formula for the resultant ROC manifold of the fused classification systems which incorporates the individual classification systems previously was derived. For the multi-performance problem we show how the set of permutations of the label set is used to generate all of the consistent rules and how the permutation matrix is incorporated into a single formula for the ROC manifold. Examples will be given that demonstrate how the solution to the multi-performance fusion problem relates to the solution of the single performance fusion problem.

Confidence of a ROC Manifold

A Classification system such as an Automatic Target Recognition (ATR) system with N possible output labels (or decisions) will have N(N-1) possible errors. The Receiver Operating Characteristic (ROC) manifold was created to quantify all of these errors. Finite truth data will produce an approximation to a ROC manifold. How well does the approximate ROC manifold approximate the TRUE ROC manifold? Several metrics exist that quantify the approximation ability, but researchers really wish to quantify the confidence in the approximate ROC manifold. This paper will review different confidence definitions for ROC curves and will derive an expression for confidence of a ROC manifold. The foundation of the confidence expression is based upon the Chebychev inequality..

The effects of correlation on the performance of ATR systems

The Receiver Operating Characteristic (ROC) curve is typically used to quantify the performance of Automatic Target Recognition (ATR) systems. When multiple systems are to be fused, assumptions are made in order to mathematically combine the individual ROC curves for each of these ATR systems in order to form the ROC curve of the fused system. Often, one of these assumptions is independence between the systems. However, correlation may exist between the classifiers, processors, sensors and the outcomes used to generate each ROC curve. This paper will demonstrate a method for creating a ROC curve of the fused systems which incorporates the correlation that exists between the individual systems. Specifically, we will use the derived covariance matrix between multiple systems to compute the existing correlation and level of dependence between pairs of systems. The ROC curve for the fused system is then produced, adjusting for this level of dependency, using a given fusion rule. We generate the formula for the Boolean OR and AND rules, giving the exact ROC curve for the fused system, that is, not a bound.