Giles M. Foody

Status of land cover classification accuracy assessment

The production of thematic maps, such as those depicting land cover, using an image classification is one of the most common applications of remote sensing. Considerable research has been directed at the various components of the mapping process, including the assessment of accuracy. This paper briefly reviews the background and methods of classification accuracy assessment that are commonly used and recommended in the research literature. It is, however, evident that the research community does not universally adopt the approaches that are often recommended to it, perhaps a reflection of the problems associated with accuracy assessment, and typically fails to achieve the accuracy targets commonly specified. The community often tends to use, unquestioningly, techniques based on the confusion matrix for which the correct application and interpretation requires the satisfaction of often untenable assumptions (e.g., perfect coregistration of data sets) and the provision of rarely conveyed information (e.g., sampling design for ground data acquisition). Eight broad problem areas that currently limit the ability to appropriately assess, document, and use the accuracy of thematic maps derived from remote sensing are explored. The implications of these problems are that it is unlikely that a single standardized method of accuracy assessment and reporting can be identified, but some possible directions for future research that may facilitate accuracy assessment are highlighted.

Thematic Map Comparison: Evaluating the Statistical Significance of Differences in Classification Accuracy

The accuracy of thematic maps derived by image classification analyses is often compared in remote sensing studies. This comparison is typically achieved by a basic subjective assessment of the observed difference in accuracy but should be undertaken in a statistically rigorous fashion. One approach for the evaluation of the statistical significance of a difference in map accuracy that has been widely used in remote sensing research is based on the comparison of the kappa coefficient of agreement derived for each map. The conventional approach to the comparison of kappa coefficients assumes that the samples used in their calculation are independent, an assumption that is commonly unsatisfied because the same sample of ground data sites is often used for each map. Alternative methods to evaluate the statistical significance of differences in accuracy are available for both related and independent samples. Approaches for map comparison based on the kappa coefficient and proportion of correctly allocated cases, the two most widely used metrics of thematic map accuracy in remote sensing, are discussed. An example illustrates how classifications based on the same sample of ground data sites may be compared rigorously and highlights the importance of distinguishing between one- and two-sided statistical tests in the comparison of classification accuracy statements.

A relative evaluation of multiclass image classification by support vector machines

Support vector machines (SVMs) have considerable potential as classifiers of remotely sensed data. A constraint on their application in remote sensing has been their binary nature, requiring multiclass classifications to be based upon a large number of binary analyses. Here, an approach for multiclass classification of airborne sensor data by a single SVM analysis is evaluated against a series of classifiers that are widely used in remote sensing, with particular regard to the effect of training set size on classification accuracy. In addition to the SVM, the same datasets were classified using a discriminant analysis, decision tree, and multilayer perceptron neural network. The accuracy statements of the classifications derived from the different classifiers were compared in a statistically rigorous fashion that accommodated for the related nature of the samples used in the analyses. For each classification technique, accuracy was positively related with the size of the training set. In general, the most accurate classifications were derived from the SVM approach, and with the largest training set the SVM classification was significantly (p < 0.05)more accurate (93.75%) than that derived from the discriminant analysis (90.00%) and decision tree algorithms (90.31%). Although each classifier could yield a very accurate classification, > 90% correct, the classifiers differed in the ability to correctly label individual cases and so may be suitable candidates for an ensemble-based approach to classification.

Approaches for the production and evaluation of fuzzy land cover classifications from remotely-sensed data

Abstract Remote sensing is an attractive source of data for land cover mapping applications. Mapping is generally achieved through the application of a conventional statistical classification, which allocates each image pixel to a land cover class. Such approaches are inappropriate for mixed pixels, which contain two or more land cover classes, and a fuzzy classification approach is required. When pixels may have multiple and partial class membership measures of the strength of class membership may be output and, if strongly related to the land cover composition, mapped to represent such fuzzy land cover. This type of representation can be derived by softening the output of a conventional ‘hard’ classification or using a fuzzy classification. The accuracy of the representation provided by a fuzzy classification is, however, difficult to evaluate. Conventional measures of classification accuracy cannot be used as they are appropriate only for ‘hard’ classifications. The accuracy of a classification may, ho...

Predictive relations of tropical forest biomass from Landsat TM data and their transferability between regions

The full realization of the potential of remote sensing as a source of environmental information requires an ability to generalize in space and time. Here, the ability to generalize in space was investigated through an analysis of the transferability of predictive relations for the estimation of tropical forest biomass from Landsat TM data between sites in Brazil, Malaysia and Thailand. The data sets for each test site were acquired and processed in a similar fashion to facilitate the analyses. Three types of predictive relation, based on vegetation indices, multiple regression and feedforward neural networks, were developed for biomass estimation at each site. For each site, the strongest relationships between the biomass predicted and that measured from field survey was obtained with a neural network developed specifically for the site (r>0.71, significant at the 99% level of confidence). However, with each type of approach problems in transferring a relation to another site were observed. In particular, it was apparent that the accuracy of prediction, as indicated by the correlation coefficient between predicted and measured biomass, declined when a relation was transferred to a site other than that upon which it was developed. Part of this problem lies with the observed variation in the relative contribution of the different spectral wavebands to predictive relations for biomass estimation between sites. It was, for example, apparent that the spectral composition of the vegetation indices most strongly related to biomass differed greatly between the sites. Consequently, the relationship between predicted and measured biomass derived from vegetation indices differed markedly in both strength and direction between sites. Although the incorporation of test site location information into an analysis resulted in an increase in the strength of the relationship between predicted and actual biomass, considerable further research is required on the problems associated with transferring predictive relations.

Feature Selection for Classification of Hyperspectral Data by SVM

Support vector machines (SVM) are attractive for the classification of remotely sensed data with some claims that the method is insensitive to the dimensionality of the data and, therefore, does not require a dimensionality-reduction analysis in preprocessing. Here, a series of classification analyses with two hyperspectral sensor data sets reveals that the accuracy of a classification by an SVM does vary as a function of the number of features used. Critically, it is shown that the accuracy of a classification may decline significantly (at 0.05 level of statistical significance) with the addition of features, particularly if a small training sample is used. This highlights a dependence of the accuracy of classification by an SVM on the dimensionality of the data and, therefore, the potential value of undertaking a feature-selection analysis prior to classification. Additionally, it is demonstrated that, even when a large training sample is available, feature selection may still be useful. For example, the accuracy derived from the use of a small number of features may be noninferior (at 0.05 level of significance) to that derived from the use of a larger feature set providing potential advantages in relation to issues such as data storage and computational processing costs. Feature selection may, therefore, be a valuable analysis to include in preprocessing operations for classification by an SVM.

Sub-pixel land cover composition estimation using a linear mixture model and fuzzy membership functions

Abstract Mixed pixels occur commonly in remotely-sensed imagery, especially those with a coarse spatial resolution. They are a problem in land-cover mapping applications since image classification routines assume ‘pure’ or homogeneous pixels. By unmixing a pixel into its component parts it is possible to enableinter alia more accurate estimation of the areal extent of different land cover classes. In this paper two approaches to estimating sub-pixel land cover composition are investigated. One is a linear mixture model the other is a regression model based on fuzzy membership functions. For both approaches significant correlation coefficients, all >0·7, between the actual and predicted proportion of a land cover type within a pixel were obtained. Additionally a case study is presented in which the accuracy of the estimation of tropical forest extent is increased significantly through the use of sub-pixel estimates of land-cover composition rather than a conventional image classification.

Measuring and modelling biodiversity from space

The Earth is undergoing an accelerated rate of native ecosystem conversion and degradation and there is increased interest in measuring and modelling biodiversity from space. Biogeographers have a long-standing interest in measuring patterns of species occurrence and distributional movements and an interest in modelling species distributions and patterns of diversity. Much progress has been made in identifying plant species from space using high-resolution satellites (QuickBird, IKONOS), while the measurement of species movements has become commonplace with the ARGOS satellite tracking system which has been used to track the movements of thousands of individual animals. There have been significant advances in land-cover classifications by combining data from multi-passive and active sensors, and new classification techniques. Species distribution modelling has been growing at a striking rate and the incorporation of spaceborne data on climate, topography, land cover, and vegetation structure has great potential to improve models. There have been significant advances in modelling species richness, alpha diversity, and beta diversity using multisensors to quantify land-cover classifications and landscape metrics, measures of productivity, and measures of heterogeneity. Remote sensing of nature reserves can provide natural resources managers with near real-time data within and around reserves that can be used to support conservation efforts anywhere in the world. Future research should focus on incorporating recent spaceborne sensors, more extensive integration of available spaceborne imagery, and the collection and dissemination of high-quality field data. This will improve our understanding of the distribution of life on earth.

An evaluation of some factors affecting the accuracy of classification by an artificial neural network

Abstract Artificial neural networks are attractive for the classification of remotely sensed data. However, a wide range of factors influence the accuracy with which a data set may be classified. In this paper, the effect of four factors on the accuracy with which agricultural crops may be classified from airborne thematic mapper (ATM) data was investigated. These factors related to the dimensionality of the remotely sensed data, the neural network architecture, and the characteristics of the training and testing sets. A total of 288 classifications were performed and their accuracies evaluated. The artificial neural networks were able to classify the data to high accuracies, with kappa coefficients of up to 0.97 obtained, but the accuracy derived was highly dependent on the factors investigated. A log-linear modelling approach was used to evaluate the simultaneous effect of the factors on classification accuracy. Variations in the dimensionality of the data set, as well as the training and testing set ch...

Identifying terrestrial carbon sinks: Classification of successional stages in regenerating tropical forest from Landsat TM data

Abstract Remote sensing has generally been used to study the role of tropical forests as a source of atmospheric carbon, primarily through land-use change, such as deforestation, and biomass burning. Regeneration of forest on previously cleared areas, however, is a significant carbon sink . The strength of this carbon sink is dependent on the age and composition of the regenerating forest. The ability to identify regenerating forest classes that may differ in terms of carbon sink strength was investigated with Landsat TM data of a test site near Manaus, Brazil. A number of forest age classes were defined from a time series of Landsat sensor data, and their separability in Landsat TM data was assessed by maximum likelihood classifications. A high level of class separability was observed with a weighted kappa coefficient of 0.8569 obtained for a classification of six forest regeneration classes. Of the classification errors observed most were found to be associated with the youngest forest age class. At the test site, however, two main successional pathways were followed and the differences between areas of forest of the same age but on different pathways was most apparent with the youngest forests. Splitting the regenerating forests by the successional pathway was found to increase classification accuracy, with a weighted kappa coefficient of 0.9315 observed for an 11 class classification. A range of tropical forest classes that vary in strength as a carbon sink could therefore be identified accurately from Landsat TM data. Although the broader generality of the results requires further investigation, this indicates the potential to use image classifications to scale-up point measurements of the carbon flux between regenerating forest classes and the atmosphere over large areas.