Critical Analysis of Cross-Validation Methods and Their Impact on Neural Networks Performance Inflation in Electroencephalography Analysis

Abstract

The performance of a brain–computer interface (BCI) system is usually measured by its classification accuracy. This creates motivation to increase system accuracies. This article investigates the variation of accuracy values in emotion recognition studies and their relation to cross-validation methods. The literature shows values of accuracies ranging from 60% to 99% while using similar classifiers when tested on the same data set. The study included a literature review of 65 articles testing their algorithms on the DEAP data set up until 2019. Moreover, the study involved the reimplementation of a neural network classifier in both nested and nonnested cross-validation methods. The results show accuracies up to 90% when using nonnested cross validation, while nested cross-validation achieves accuracies around 60%. This article aims to motivate researchers to clearly describe their cross-validation method to avoid confusing other researchers when benchmarking their algorithms.