Mahesh Pal

Feature Selection for Classification of Hyperspectral Data by SVM

Support vector machines (SVM) are attractive for the classification of remotely sensed data with some claims that the method is insensitive to the dimensionality of the data and, therefore, does not require a dimensionality-reduction analysis in preprocessing. Here, a series of classification analyses with two hyperspectral sensor data sets reveals that the accuracy of a classification by an SVM does vary as a function of the number of features used. Critically, it is shown that the accuracy of a classification may decline significantly (at 0.05 level of statistical significance) with the addition of features, particularly if a small training sample is used. This highlights a dependence of the accuracy of classification by an SVM on the dimensionality of the data and, therefore, the potential value of undertaking a feature-selection analysis prior to classification. Additionally, it is demonstrated that, even when a large training sample is available, feature selection may still be useful. For example, the accuracy derived from the use of a small number of features may be noninferior (at 0.05 level of significance) to that derived from the use of a larger feature set providing potential advantages in relation to issues such as data storage and computational processing costs. Feature selection may, therefore, be a valuable analysis to include in preprocessing operations for classification by an SVM.

Some issues in the classification of DAIS hyperspectral data

Classification accuracy depends on a number of factors, of which the nature of the training samples, the number of bands used, the number of classes to be identified relative to the spatial resolution of the image and the properties of the classifier are the most important. This paper evaluates the effects of these factors on classification accuracy using a test area in La Mancha, Spain. High spectral and spatial resolution DAIS data were used to compare the performance of four classification procedures (maximum likelihood, neural network, support vector machines and decision tree). There was no evidence to support the view that classification accuracy inevitably declines as the data dimensionality increases. The support vector machine classifier performed well with all test data sets. The use of the orthogonal MNF transform resulted in a decline in classification accuracy. However, the decision‐tree approach to feature selection worked well. Small increases in classifier accuracy may be obtained using more sophisticated techniques, but it is suggested here that greater attention should be given to the collection of training and test data that represent the range of land surface variability at the spatial scale of the image.

Support vector machine‐based feature selection for land cover classification: a case study with DAIS hyperspectral data

This paper present the results of a support vector machine (SVM) technique and a genetic algorithm (GA) technique using generalization error bounds derived for SVMs as fitness functions (SVM/GA) for feature selection using hyperspectral data. Results obtained with the SVM/GA‐based technique were compared with those produced by random forest‐ and SVM‐based feature selection techniques in terms of classification accuracy and computational cost. The classification accuracy using SVM‐based feature selection was 91.89%. The number of features selected was 15. For comparison, the accuracy produced by the use of the full set of 65 features was 91.76%. The level of classification accuracy achieved by the SVM/GA approach using 15 features varied from 91.87% to 92.44% with different fitness functions but required a large training time. The performance of the random forest‐based feature selection approach gave a classification accuracy of 91.89%, which is comparable to the accuracy achieved by using the SVM and SVM/GA approaches using 15 features. A smaller computational cost is a major advantage associated with the random forest‐based feature selection approach. The training time for the SVM‐based approach is also very small in comparison to the SVM/GA approach, thus suggesting the usefulness of random forest‐ and SVM‐based feature selection approaches in comparison to the SVM/GA approach for land cover classification problems with hyperspectral data. Further, a higher classification accuracy was achieved with a combination of 20 selected features in comparison to the level of accuracy obtained using 15 features, but the difference in accuracy was not significant. To validate the results, SVM‐, SVM/GA‐ and random forest‐based feature selection approaches were compared with a maximum noise transformation based feature extraction technique. Results show an improved performance using these techniques in comparison to the maximum noise transformation‐based feature extraction technique.

Support vector regression based modeling of pier scour using field data

This paper investigates the potential of support vector machines based regression approach to model the local scour around bridge piers using field data. A dataset of consisting of 232 pier scour measurements taken from BSDMS were used for this analysis. Results obtained by using radial basis function and polynomial kernel based Support vector regression were compared with four empirical relation as well as with a backpropagation neural network and generalized regression neural network. A total of 154 data were used for training different algorithms whereas remaining 78 data were used to test the created model. A coefficient of determination value of 0.897 (root mean square error=0.356) was achieved by radial basis kernel based support vector regression in comparison to 0.880 and 0.835 (root mean square error=0.388 and 0.438) by backpropagation neural and generalized regression neural network. Comparisons of results with four predictive equations suggest an improved performance by support vector regression. Results with dimensionless data using all three algorithms suggest a better performance by dimensional data with this dataset. Sensitivity analysis suggests the importance of depth of flow and pier width in predicting the scour depth when using support vector regression based modeling approach.

Evaluation of SVM, RVM and SMLR for Accurate Image Classification With Limited Ground Data

The accuracy of a conventional supervised classification is in part a function of the training set used, notably impacted by the quantity and quality of the training cases. Since it can be costly to acquire a large number of high quality training cases, recent research has focused on methods that allow accurate classification from small training sets. Previous work has shown the potential of support vector machine (SVM) based classifiers. Here, the potential of the relevance vector machine (RVM) and sparse multinominal logistic regression (SMLR) approaches is evaluated relative to SVM classification. With both airborne and spaceborne multispectral data sets, the RVM and SMLR were able to derive classifications of similar accuracy to the SVM but required considerably fewer training cases. For example, from a training set comprising 600 cases acquired with a conventional stratified random sampling design from an airborne thematic mapper (ATM) data set, the RVM produced the most accurate classification, 93.75%, and needed only 7.33% of the available training cases. In comparison, the SVM yielded a classification that had an accuracy of 92.50% and needed 4.5 times more useful training cases. Similarly, with a Landsat ETM+ (Littleport, Cambridgeshire, UK) data set, the SVM required 4.0 times more useful training cases than the RVM. For each data set, however, the classifications derived by each classifier were of similar magnitude, differing by no more than 1.25%. Finally, for both the ATM and ETM+ (Littleport) data sets, the useful training cases by SVM and RVM had distinct and potentially predictable characteristics. Support vectors were generally atypical but lay in the boundary region between classes in feature space while the relevance vectors were atypical but anti-boundary in nature. The SMLR also tended to mostly, but not always, use extreme cases that lay away from class boundary. The results, therefore, suggest a potential to design classifier-specific intelligent training data acquisition activities for accurate classification from small training sets, especially with the SVM and RVM.

Application of support vector machines in scour prediction on grade-control structures

Research into the problem of predicting the maximum depth of scour on grade-control structures like sluice gates, weirs and check dams, etc., has been mainly of an experimental nature and several investigators have proposed a number of empirical relations for a particular situation. These traditional scour prediction equations, although offer some guidance on the likely magnitude of maximum scour depth, yet applicable to a limited range of the situations. It appears from the literature review that a regression mathematical model for predicting maximum depth of scour under all circumstances is not currently available. This paper explores the potential of support vector machines in modeling the scour from the available laboratory and field data obtained form the earlier published studies. To compare the results, a recently proposed empirical relation and a feed forward back propagation neural network model are also used in the present study. The outcome from the support vector machines-based modeling approach suggests a better performance in comparison to both the empirical relation and back propagation neural network approach with the laboratory data. The results also suggest an encouraging performance by the support vector machines learning technique in comparison to both empirical relation as well as neural network approach in scaling up the results from laboratory to field conditions for the purpose of scour prediction.

M5 model tree application in daily river flow forecasting in Sohu Stream, Turkey

This study investigate the potential of M5 model tree in predicting daily stream flows in Sohu river located within the municipal borders of Ankara, Turkey. The results of the M5 model tree was compared with support vector machines. Both modelling approaches were used to forecast up to 7-day ahead stream flow. A comparison of correlation coefficient and root mean square value indicates that M5 model tree approach works equally well to the SVM for same day discharge prediction. The M5 model tree also works well up to 7-day ahead discharge forecasting in comparison of SVM with this data set. An advantage of using M5 model tree approach is the availability of simple linear models to predict the discharge as well use of less computational time.

Margin-based feature selection for hyperspectral data

Abstract A margin-based feature selection approach is explored for hyperspectral data. This approach is based on measuring the confidence of a classifier when making predictions on a test data. Greedy feature flip and iterative search algorithms, which attempts to maximise the margin-based evaluation functions, were used in the present study. Evaluation functions use linear, zero–one and sigmoid utility functions where a utility function controls the contribution of each margin term to the overall score. The results obtained by margin-based feature selection technique were compared to a support vector machine-based recurring feature elimination approach. Two different hyperspectral data sets, one consisting of 65 bands (DAIS data) and other with 185 bands (AVIRIS data) were used. With digital airborne imaging spectrometer (DAIS) data, the classification accuracy by greedy feature flip algorithm and sigmoid utility function was 93.02% using a total of 24 selected features in comparison to an accuracy of 91.76% with full set of 65 features. The results suggest a significant increase in classification accuracy with 24 selected features. The classification accuracy (93.4%) achieved by the iterative search margin-based algorithm with 20 selected features using sigmoid utility function is also significantly more accurate than that achieved with 65 features. To judge the usefulness of margin-based feature selection approaches, another hyperspectral data set consisting of 185 features was used. A total of 65 selected features were used to evaluate the performance of margin-based feature selection approach. The results suggest a significantly improved performance by greedy feature flip-based feature selection technique with this data set also. This study also suggest that margin-based feature selection algorithms provide a comparable performance to support vector machine-based recurring feature elimination approach.

Multinomial logistic regression-based feature selection for hyperspectral data

Abstract This paper evaluates the performance of three feature selection methods based on multinomial logistic regression, and compares the performance of the best multinomial logistic regression-based feature selection approach with the support vector machine based recurring feature elimination approach. Two hyperspectral datasets, one consisting of 65 features (DAIS data) and other with 185 features (AVIRIS data) were used. Result suggests that a total of between 15 and 10 features selected by using the multinomial logistic regression-based feature selection approach as proposed by Cawley and Talbot achieve a significant improvement in classification accuracy in comparison to the use of all the features of the DAIS and AVIRIS datasets. In addition to the improved performance, the Cawley and Talbot approach does not require any user-defined parameter, thus avoiding the requirement of a model selection stage. In comparison, the other two multinomial logistic regression-based feature selection approaches require one user-defined parameter and do not perform as well as the Cawley and Talbot approach in terms of (i) the number of features required to achieve classification accuracy comparable to that achieved using the full dataset, and (ii) the classification accuracy achieved by the selected features. The Cawley and Talbot approach was also found to be computationally more efficient than the SVM-RFE technique, though both use the same number of selected features to achieve an equal or even higher level of accuracy than that achieved with full hyperspectral datasets.

M5 model tree for land cover classification

Tree based regression models like a M5 algorithm represent a promising development in machine learning research. A recent study suggests that a M5 model tree algorithm can be used for classification problems after some modification. This letter explores the usefulness of a M5 model tree for classification problems using multispectral (Landsat‐7 Enhanced Thematic Mapper Plus (ETM+)) for a test area in eastern England. Classification accuracy achieved by using a M5 model tree is compared with a univariate decision tree with and without using boosting. Results show that the M5 model tree achieves a significantly higher level of classification accuracy than a decision tree and works equally well to a boosted decision tree. Further, a model tree based classification algorithm works well with small as well as noisy datasets.