EEG-based Texture Roughness Classification in Active Tactile Exploration with Invariant Representation Learning Networks
Ozan Ozdenizci, Safaa Eldeeb, Andac Demir, Deniz Erdogmus, Murat Akcakaya
EEEG-based Texture Roughness Classification in Active Tactile Explorationwith Invariant Representation Learning Networks
Ozan Özdenizci a,b,1, ∗ , Safaa Eldeeb c,1 , Andaç Demir a , Deniz Erdoğmuş a , Murat Akçakaya c a Department of Electrical and Computer Engineering, Northeastern University, Boston, MA, USA b Institute of Theoretical Computer Science, Graz University of Technology, Graz, Austria c Department of Electrical and Computer Engineering, University of Pittsburgh, Pittsburgh, PA, USA
Abstract
During daily activities, humans use their hands to grasp surrounding objects and perceive sensory infor-mation which are also employed for perceptual and motor goals. Multiple cortical brain regions are knownto be responsible for sensory recognition, perception and motor execution during sensorimotor processing.While various research studies particularly focus on the domain of human sensorimotor control, the relationand processing between motor execution and sensory processing is not yet fully understood. Main goalof our work is to discriminate textured surfaces varying in their roughness levels during active tactile ex-ploration using simultaneously recorded electroencephalogram (EEG) data, while minimizing the varianceof distinct motor exploration movement patterns. We perform an experimental study with eight healthyparticipants who were instructed to use the tip of their dominant hand index finger while rubbing or tappingthree different textured surfaces with varying levels of roughness. We use an adversarial invariant repre-sentation learning neural network architecture that performs EEG-based classification of different texturedsurfaces, while simultaneously minimizing the discriminability of motor movement conditions (i.e., rub ortap). Results show that the proposed approach can discriminate between three different textured surfaceswith accuracies up to 70%, while suppressing movement related variability from learned representations.
Keywords: haptics, texture roughness, active tactile exploration, EEG, invariant representations, neuralnetworks, deep learning, adversarial learning
1. Introduction
Active tactile exploration refers to the process of exploring surrounding objects to retrieve sensory infor-mation [1]. Dynamic active movement between human skin and an object’s surface may occur during thisprocess. Active tactile exploration involves several parts of the human body which are characterized by highdensity sensory receptors, high sensory acuity and large sensory and motor cortical representation [2]. Thehigh density mechanoreceptors on the human skin contribute significantly to the discrimination betweendifferent objects varying in pattern, texture and shape [3, 4]. Mechanoreceptors in the epidermis and dermislayers of the human fingertips include distinct small, large, slow and rapidly adapting receptive fields whichenable high spatial resolution [3, 4]. Active tactile object exploration generates simultaneous cutaneous andproprioceptive feedbacks, whose combination is referred as haptic feedback. Proprioceptive signals originatefrom joint, muscle and skin mechanoreceptors, and are related to joint movement and position [2].The primary somatosensory cortex (S1) is known to be the main brain region responsible for processingsensory information related to active exploration of objects [3]. This region is also known to have a directassociation with the primary motor cortex, which is also involved in processing of active exploratory mo-tor movements. During active tactile exploration, S1 cortex demonstrates neural activity that reflects the ∗ Corresponding author: [email protected] Equal contribution
Accepted for publication at Biomedical Signal Processing and Control February 19, 2021 a r X i v : . [ ee ss . SP ] F e b ctivation of several peripheral mechanoreceptors, as well as joint movements [5]. Moreover, movementsassociated with tactile exploration modulates the transmission of the tactile input to S1 cortex, addingfurther complications to the processing of somatic sensory signals [5]. It has also been shown that a vastmajority of neural activity in S1 cortex significantly contributes to the processing of tactile input duringactive tactile exploration compared to passive exploration [6, 7]. Accordingly, active tactile exploration pro-vides substantial information about the surface properties and demonstrate better discrimination thresholdscompared to passive exploration [1, 2, 8].Sensorimotor processing is generally visualized as a sequence of function units that are sequentiallyactivated that initiates from sensory input and proceeds until motor execution [9–11]. Several other researchstudies on human brain function suggest that motor and sensory processing are coupled, where motor actionsand perception are in affect with each other [12–14]. A study by Melink et. al. [15], investigated these twoviews of sensorimotor processing (i.e., the classical view and the alternative coupling perspective). Theirelectroencephalography (EEG) based exploratory study supports the alternative sensorimotor processinghypothesis that sensory perception and motor responses are coupled.Research studies investigating EEG-based cortical activity in response to active tactile exploration aregenerally explored within passive touch experimental designs [16–18]. To the best of our knowledge, thereexists only a few studies which explores the relation between recorded EEG and the roughness of theexplored surface in active touch [19, 20]. Yet, EEG-based classification of textured surfaces that vary intheir roughness levels, while minimizing the potential influence of motor movement type during active tactileexploration remains to be explored.In this study, we aim to classify different textured surfaces during active tactile exploration that vary inthe roughness level on a single trial basis using simultaneously recorded EEG data. Our experimental studydesign includes two motor movement conditions, rubbing or tapping the textured surface, and three differentlevels of surface roughness (i.e., smooth, medium rough and rough). We propose an invariant representationlearning neural network that allows classification of textured surface roughness levels, while minimizing thediscriminability of the motor movement type in a systematic way using an adversarial training approach [21].The overarching goal of this study is to use such a framework to develop a system that mimics the sensationof surfaces with varying levels of roughness during active exploration. These systems would ideally be usedin a wide range of applications such as teleoperation, surgical training of physicians in virtual environments,or remote control of robotic prostheses [22–26].
2. Experimental Study Design
A total of eight right-handed healthy participants (age: 26 ± During the experiments the participants sat in front of a computer screen and were instructed to usethe tip of their dominant hand index finger while rubbing or tapping three different textured surfaces withvarying levels of roughness. An illustration of the experimental setup is provided in Figure 1(a) (cf. [20]for further information regarding the experimental setup design). The computer screen provided visualcues to the participants via a graphical user interface we developed using the Psychtoolbox software [27].These cues marked the beginning of each condition. The experiment consisted of a total of six conditions,two movement type (i.e., rub and tap); and three textures surfaces (i.e., flat smooth, medium rough andrough). For each condition we instructed each participant to rub or tap the selected surface. Each completemovement across the textured surface was considered a single complete trial.Three synthesized textured surfaces with varying levels of roughness (i.e., smooth flat, medium rough, andrough) were generated using MATLAB (MathWorks, USA) and fabricated with Stereolithography (ViperSLA System, 3Dsystems, USA). The smooth flat surface is an even regular surface without any form of2 a) (b)Figure 1: (a) Illustration of the experimental setup during a trial where the participant is instructed to tap on the texturemounted on the force transducer. (b) Printed medium rough 3D texture mounted on the force transducer. roughness. The roughness of the medium rough and rough surfaces represented by the power spectraldensity was controlled by the following expression: φ ( | k | ) = C, if k l < = | k | < = k r .C ( | k | k r ) − H ) , if k r < = | k | < = k s . , otherwise . (1) where C is the roughness amplitude, k l , k r , k s are the lower roll-off and upper cutoff wave numbers and H is the Hurst roughness exponent. The values for each surface in this study are chosen as follows: themedium rough surface (H = 0.5, C = ∗ , k l = k r = 16, k s = 64) and rough surface (H = 0.5, C = ∗ , k l = k r = 32, k s = 256). For the third texture, simply a smooth flat surface was used. Eachsynthetic textured surface was mounted on a force transducer (as illustrated in Figure 1(b) for the mediumrough texture), and adjusted on a table in reach of the participants. Throughout the experiments, EEG data were collected from 14 EEG channels placed over the frontaland somatosensory cortices. These channels were placed according to the international 10-20 system [28] aschannel locations: F3, F4, FC3, FC4, C1, C3, C5, CZ, C2, C4, C6, CP1, CPZ and CP2. We used the leftmastoid as a reference and FPz as the ground electrode. Force data generated from touching the texturedsurfaces were also recorded using a force and torque transducer (NANO17 F/T transducer, ATI IndustrialAutomation, USA). Two g.USBamp amplifiers (g.tec medical engineering GmbH, Graz, Austria) were usedto record and synchronize both EEG and force data. The first amplifier, which is used for EEG dataacquisition, has a sampling frequency of 1200 Hz and uses a 4th order notch filter with cutoff frequencies 58and 62 Hz, and an 8th order bandpass filter with cutoff frequencies of 2 and 62 Hz. The second amplifierwas connected to the force transducer and used to synchronize the collected force data and digitize it withsampling rate of 1200 Hz. The force data was used to mark the beginning of each trial for the continuouslyrecorded EEG data. We downsampled EEG data to 300 Hz sampling rate offline, and segmented into trialswith duration of one third of a second following the touch (i.e., ∼
330 ms).
3. Methods
Deep neural networks have been widely explored as generic feature extractors for EEG in various clas-sification tasks [29–33]. We recently explored such discriminative models in the context of deep invariant3 nput EEG: X Feature Encoder: z = f ( X ; γ e )TemporalConvolutions SpatialConvolutions Spatio-temporalConvolutions Classifier q γ c ( l s | z )Adversary q γ a ( l m | z ) Figure 2: Overall adversarial invariant representation learning network flow, illustrating the feature encoder, classifier andadversary networks. Network layer specifications are further provided in Table 1. representation learning [34–36], to illustrate potential capabilities of EEG-based neural network models tocensor specific nuisance information inherent within the learned feature extractors [21, 37, 38]. Our pro-posed neural network aims to learn texture roughness discriminative features while minimizing the influenceof movement conditions (rub or tap) during active tactile exploration. We realize this using an adversarialtraining approach which infers a nuisance (i.e., motor movement activity) invariant latent space within adiscriminative setting, hence suppressing motor movement related cortical activity from representations thatare capable of classifying texture roughness levels.
We consider a subject-specific EEG data set { ( X i , l si , l mi ) } ni =1 with n number of trials, where X i ∈ R CxT denotes the multichannel EEG data collected from C channels and T number of discretized time samples, l si denotes the label for the roughness of a texture surface, and l mi denotes the binary nuisance variable(i.e., rub or tap motor movement). Since no ordinal relationship exists in the categorical variables l s and l m , we represent them as one-hot encoded label vectors. Nuisance variations denoted by l m are assumed tobe involved in the data generation process, and therefore we employ an adversarial training approach thatcan discriminate l s invariant of l m . We premise the underlying data generation process on our assumptionthat l s and l m are disentangled such that l s ∼ p ( l s ) and l m ∼ p ( l m ) with X ∼ p ( X | l s , l m ) .The network flow is outlined in Figure 2. A deterministic feature encoder network with parameters γ e projects the time-series EEG data onto a lower dimensional latent feature space, z = f ( X ; γ e ) . This latentvector is passed to a classifier network with parameters γ c , and an adversary network with parameters γ a separately. The goal of classifier network is to infer surface texture label l s , whereas that of adversary is toinfer the nuisance variations l m evoked by motor activity. In other words, the classifier network is trained tomaximize the likelihood q γ c ( l s | z ) , and the adversary network maximizes its likelihood q γ a ( l m | z ) . However,the joint feature encoder network parameters are trained using a loss function that combines these twoobjectives within a conflicting relationship, by maximization of the likelihood q γ c ( l s | z ) and minimization ofthe likelihood q γ a ( l m | z ) . This adversarial training behavior of the adversary network enables learning of l m -invariant feature representations. Overall, the joint feature encoder aims to learn l s -discriminative features,while the antagonistic adversarial loss enables concealing of l m -relevant information from the feature encoder.This corresponds to the following min-max objective for adversarial model training: min γ e ,γ c max γ a E [ − log q γ c ( l s | f ( X ; γ e )) + λ log q γ a ( l m | f ( X ; γ e ))] , (1)to obtain the optimal network parameters ˆ γ e , ˆ γ c , ˆ γ a , where λ > is the adversarial regularization weightfor nuisance-invariant decoding. λ = 0 would correspond to a traditional neural network training proce-dure without any manipulation. In case λ < , the encoder would be trained to learn particularly motormovement dependent features for texture roughness classification, which is not the intention of this study.Parameter updates are performed alternatingly. At each iteration, firstly the loss of the adversary is back-propagated to update the parameters of the adversary network γ a towards maximizing its log-likelihood(i.e., max objective). Afterwards, the overall loss is back-propagated to update the encoder and classifier4 lgorithm 1 Adversarial invariant representation learning network training [21]
Input: { ( X i , l si , l mi ) } ni =1 , λ > Output: ˆ γ e , ˆ γ c , ˆ γ a Randomly select initial parameters: γ e , γ c , γ a for epochs do for minibatches do Sample a minibatch: { ( X b , l sb , l mb ) } Bb =1 Update γ a with stochastic gradient ascent by: ∇ γ a (cid:80) Bb =1 λ log q γ a ( l mb | z b = f ( X b ; γ e )) Update γ e , γ c with stochastic gradient descent by: ∇ γ e ,γ c (cid:80) Bb =1 [ − log q γ c ( l sb | z b ) + λ log q γ a ( l mb | z b )] end for end for parameters γ e and γ c (i.e., min objective). Algorithm 1 outlines the stochastic model training process[21]. Implementations for the invariant EEG representation learning networks are publicly available at:https://github.com/oozdenizci/AdversarialEEGDecoding. Table 1 outlines the neural network architecture specifications that we used for EEG-based texture rough-ness classification. We determined the feature encoder layers in accordance with the EEGNet convolutionalneural network (CNN) model [31]. Network input EEG data has dimensionality of 14 channels by 100 dis-cretized time samples (i.e., 1/3 second segments of signals with 300 Hz sampling rate). The feature encodernetwork sequentially performs: (1) temporal convolutions that resemble to frequency bandpass filtering oper-ations, (2) depthwise spatial convolutions [39] to extract bandpass specific information from distinct corticalsources over all EEG sensors, and (3) spatio-temporal separable convolutions [39] for summarization of allextracted information. Differently from the original EEGNet model [31], we re-selected the temporal andspatial convolution kernel lengths in consistency with our data set structure. Initial temporal convolutionkernel size is chosen as 75, indicating that as low as 4 Hz of periodic signal activity will be captured fromthe input signals sampled at 300 Hz. Spatial convolution kernel length was determined as 14 by the numberof EEG channels. All temporal convolution operations involved zero padding to keep the temporal lengthsame. None of the convolution operations had a bias term. We used batch normalization after convolutionoperations [40], and dropout layers [41] with p = 0 . after average pooling operations.Representations z = f ( X ; γ e ) that are learned at the output of the feature encoder are provided as inputto two separate single dense layer networks (i.e., classifier and adversary). Both of these decision makinglayers perform linear classification. Classifier performs 3-class decoding of surface texture roughness, andadversary performs binary classification of motor movement type (rub or tap).Evaluations were performed on a within-subject cross-validation basis by learning eight user-specificinvariant representation learning neural networks. For each subject, we considered equal number of trialsstratified by labels across six class conditions (i.e., two movement types and three textures), with total dataset sample sizes of 2190, 426, 768, 438, 570, 762, 678, 708 for subjects 1 to 8 respectively. We generated theuser-specific model training, validation, and test sets with randomly selected 70%, 10% and 20% portions ofthe trials available from each subject. The 20% test set split is specifically generated in 5-folds and repeated10 times. Hence, all subject-specific classification analyses are performed by 10x5-fold cross-validation.Networks were trained with a minibatch size of 40 training trials for at most 500 epochs. Early trainingstopping was performed if the classifier loss of the validation set did not reach a new lowest value for 10consecutive epochs. We considered the model parameters which resulted in the lowest validation loss aftercompletion of model training. Adam optimizer [42] was used for parameter updates once per batch. Theoverall network consisted of a total number of 1,661 trainable parameters. We implemented the adversarialneural network training protocol using the Tensorflow deep learning library [43] with the Keras API [44].5 able 1: Invariant representation learning neural network architecture specifications. Network Layer Filters × (Kernel Size) Output Dim. F e a t u r e E n c o d e r Input EEG: X (1 , , Conv2D × (1 ,
75) (8 , , (8 , , × (14 ,
1) (16 , , (16 , , (1 ,
4) (16 , , Dropout ( p = 0 . ) (16 , , Separable Conv2D × (1 ,
16) (16 , , (16 , , (1 ,
8) (16 , , Dropout ( p = 0 . ) + Flatten (1 , Dense + Softmax (48 ,
3) (1 , ˆ l s (1 , Dense + Softmax (48 ,
2) (1 , ˆ l m (1 ,
4. Results
Our main goal is to classify different textured surfaces that vary in their roughness levels based on EEGrecordings during active tactile exploration. While we choose to exploit a convolutional neural network modelfor feature learning and classification, we further benefit from the promise of such models in learning nuisance-invariant discriminative representations. Figure 3 presents an overview of subject-specific classificationaccuracies of the classifier and adversary networks on the test sets after model learning. Horizontal axes inFigure 3 denote the binary adversary network classification accuracies for rub versus tap motor movementtype classification, whereas the vertical axes denote the 3-class decoding accuracies for surface textureroughness. Center marks of each colored box denote the averages across 5 training folds and 10 repetitions.The width of the boxes in both dimensions denote ± standard deviation intervals of the accuracies.In cases where λ = 0 , the classifier is trained by the traditional categorical cross-entropy minimization.However we also learn an adversary network solely in parallel, which simply monitors the motor movementvariant information leakage on the encoded latent representations z . The non-adversarial model ( λ = 0 )3-class classification results lead to an average accuracy across all subjects (maximum is . forsubject 7) as observed by the y-axis values in Figure 3 plots. Simultaneously, we observed that indeedthere is a significant amount of unwanted motor movement related information exploited in these textureroughness classification models. This is reflected by an average adversary network accuracy of . (ashigh as . for subject 2, x-axes in the Figure 3 plots) when λ = 0 for binary decoding of rub versustap movement using simple linear classification with latent z . This supports our hypothesis on the necessityto censor motor movement relevant nuisance features from the latent representations during discriminativemodel learning for texture roughness classification.Our adversarial regularization approach as λ > successfully manipulates latent representations by re-ducing the adversary accuracies (i.e., leakage) towards minimizing the variance of distinct motor explorationmovement patterns, while keeping the classifier performance stable for texture roughness decoding. Onaverage across eight subjects, repetitions and folds, the . adversary network leakage of regular modellearning ( λ = 0 ) can be gradually reduced to . , . , . , . , . and . with adversarialregularization λ values of . , . , . , . , . , . respectively. Such controllable invariance enables sup-pression of nuisance-related information down to near chance-level ( ) decoding, while keeping classifieraccuracies as stable as possible.Figure 4 presents a summary of the differences between accuracies obtained with λ > models and the λ = 0 condition. Horizontal axis represents the models for different λ values, while the vertical axis represents6 C l a ss i f i e r A cc u r ac y ( % ) =0=0.01=0.02=0.05=0.1=0.2=0.5 (a) Subject 1
50 60 70 80Adversary Accuracy (%)3040506070 C l a ss i f i e r A cc u r ac y ( % ) (b) Subject 2
45 50 55 60 65Adversary Accuracy (%)304050607080 C l a ss i f i e r A cc u r ac y ( % ) (c) Subject 3
40 50 60 70Adversary Accuracy (%)3035404550556065 C l a ss i f i e r A cc u r ac y ( % ) (d) Subject 4
45 50 55 60 65Adversary Accuracy (%)303540455055 C l a ss i f i e r A cc u r ac y ( % ) (e) Subject 5
50 60 70Adversary Accuracy (%)3040506070 C l a ss i f i e r A cc u r ac y ( % ) (f) Subject 6
50 60 70 80Adversary Accuracy (%)304050607080 C l a ss i f i e r A cc u r ac y ( % ) (g) Subject 7
50 60 70 80Adversary Accuracy (%)3035404550556065 C l a ss i f i e r A cc u r ac y ( % ) (h) Subject 8Figure 3: Classifier versus adversary accuracies on the test sets of each subject after model learning. For each colored box,center marks denote the means across 5 training folds and 10 repetitions, and box widths denote ± the differences of accuracies averaged across subjects. Note that a very strong adversarial regularization canlead to losing class-discriminative information for the main classification task, hence hindering the textureroughness decoding accuracies (e.g., with λ = 0 . ). Here we present our results in an exploratory fashion,by demonstrating performances with all λ values. For further model selection and online use, an optimal λ value based on the validation set classifier and adversary accuracy can be selected [21].
5. Discussion
The ultimate goal of this work is to build an EEG-guided system that can provide the sensation ofobjects with different surface roughness levels. During our daily activities we engage in active explorationof our surroundings, and neural responses to this type of exploration reflect both motor movement andsensory perception related information. The human brain processes motor and sensory related informationin a coupled way, where both information are in affect with each other [12–15]. Moreover, this activetactile exploration provides substantial information about the surface properties and demonstrate betterdiscrimination thresholds compared to passive exploration [1, 2, 8]. To that extent, we highlight the need toinvestigate discriminative models that are sensitive to sensory information while at the same time invariantto varying motor movement related cortical activity patterns.In order to empirically assess the feasibility of this problem, we present the current experimental activetactile exploration study with healthy subjects while the participants use their index finger to either rubor tap surfaces with three different texture roughness levels. We propose to use an adversarial invariantrepresentation learning neural network model to classify the three different texture roughness levels using ∼
330 ms EEG data segments recorded in response to these movements, while minimizing the discriminability7 A cc . D i ff e r e n ce w . r . t . = AdversaryClassifier
Figure 4: Difference of accuracies with respect to λ = 0 (i.e., non-adversarial, regular CNN) for each adversarial regularizationweight λ , averaged across subjects. Red denotes the differences in binary adversary network accuracies, and blue denotes thedifferences in 3-class classifier network accuracies. of the motor movement type. Our results show that the proposed approach can discriminate between threedifferent textured surfaces with accuracies up to 70%, hypothetically by exploiting event-related potentialsgenerated in response to sensory information [45]. Furthermore we also demonstrate the inherent existenceof undesired motor movement related activity embedded in the predictive EEG features (as presented inFigure 3), pointing to the necessity of our approach in censoring motor movement relevant variabilityfrom the texture roughness discriminative representations during model learning. Overall, our proposedadversarial inference approach successfully demonstrates its capability to manipulate latent representationstowards minimizing the discriminability of distinct motor exploration movement patterns, while keeping theclassifier performance stable for texture roughness decoding.Natural characteristics of EEG signals are particularly an important factor that motivates: (1) our ad-versarially learned neural network invariance approach to this complex neural decoding problem, and (2)our specific spatio-temporal feature learning network architecture choices. Our basis argument regarding (1)arises from the fact that EEG signals, by nature, are collected and analyzed as a superposition of multipletask-specific and task-irrelevant neural oscillations [46]. Since multi-channel time-series EEG signals do nottrivially dissociate multiple cortical processes occurring at the same time, this characteristic feature took asignificant role in motivating the invariant representation learning concept of our model. Our adversarialinvariance learning setting is one approach against this phenomena towards disentangling specifically definednuisance variability, which in this case is simultaneously present motor cortical de-/modulations. In differentsettings this invariance could address another a priori defined nuisance variability (e.g., rhythmic eye-blinks).Our choices regarding (2) simply indicate that the specific recording and pre-processing pipeline of EEGsignals constitute deterministic choices for the network architecture. Our decision to use a generic convolu-tional network architecture based on EEGNet [31] is motivated by an approach to initially perform temporalconvolutions analogous to frequency filtering activities for multi-channel EEG signals, spatial convolutionsfor localization of task-discriminative cortical sources/regions for each subject, followed by spatio-temporalconvolution layers for summarization of extracted temporal and spatial features. Furthermore, as outlinedin Section 3.2, our temporal and spatial convolution kernel sizes were chosen in consistency with the utilizedEEG signal’s recording and pre-processing pipeline (e.g., temporal convolution kernels were determined asone fourth of the EEG sampling rate to sufficiently capture periodic signal activity above 4 Hz). Hence theEEG signal characteristics implicitly determines the proposed model specifications in our study.Our use of an invariant representation learning network offers the flexibility to manipulate the learnedrepresentations during training to be invariant to particular pre-defined nuisance variability (i.e., activetouch movement condition), leading to significantly reduced motor movement related information leakagefrom the learned texture-discriminative representations. However one important limitation of our approach isthat a very strong adversarial regularization can lead to loss of class-discriminative information for the mainclassification task, hence hindering the texture roughness decoding accuracies (as illustrated in Figure 4 with λ = 0 . ). An immediate potential improvement to our current approach will be to address the systematic8ubject-specific model selection issue for the optimal λ choice, which yields non-confounding results in termsof task discriminative information loss. Another important limitation of our study is the lack of subject-specific multi-session decoding stability analyses, due to our single-session experimental study design. Toproceed along this line, a methodological improvement towards learning both motor movement as well assession-invariant representations can be potentially studied.
6. Conclusion
The overarching goal of this work is to build an EEG guided system that could provide and mimic thesensation of objects with varying levels of roughness. Applications of this system can include teleoperation,training of physicians training in virtual environments or remote robotic prosthesis control. Our daily lifeactivities engage active exploration of our surroundings, and EEG responses to this type of explorationreflects both motor movement related and sensory perception information. Hence, we shed light onto theneed to investigate discriminative models that are sensitive to sensory information while at the same timeinvariant to varying motor movement related cortical activity patterns.In this paper, we show that it is possible to discriminate surfaces with varying levels of roughness duringactive tactile exploration using EEG as an input to the proposed neural network architecture. Results showthat the proposed approach can discriminate between three different textured surfaces with accuracies up to70%, while suppressing movement related variability from learned representations. In summary, our aim ofthis study was to develop a methodology to extract and select EEG features that can classify various textureswith different levels of roughness using EEG that are invariant to motor activity. Such EEG features will beused to design principles for model-based optimal EEG-guided haptic feedback system in our future work.
Acknowledgments
Our work was supported by NSF grants IIS-1717654, IIS-1715858, IIS-1915083, IIS-1844885, CPS-1544895, CBET-1804550, M3X-20040457, and NIH grant 2R01DC009834.
References [1] S. J. Lederman, R. L. Klatzky, Haptic perception: A tutorial, Attention, Perception, & Psychophysics 71 (7) (2009)1439–1459.[2] J. J. Gibson, Observations on active touch, Psychological Review 69 (6) (1962) 477.[3] M. Borich, S. Brodie, W. Gray, S. Ionta, L. Boyd, Understanding the role of the primary somatosensory cortex: opportu-nities for rehabilitation, Neuropsychologia 79 (2015) 246–255.[4] V. E. Abraira, D. D. Ginty, The sensory neurons of touch, Neuron 79 (4) (2013) 618–639.[5] C. E. Chapman, Active versus passive touch: factors influencing the transmission of somatosensory signals to primarysomatosensory cortex, Canadian Journal of Physiology and Pharmacology 72 (5) (1994) 558–570.[6] M. Blatow, E. Nennig, A. Durst, K. Sartor, C. Stippich, fMRI reflects functional connectivity of human somatosensorycortex, Neuroimage 37 (3) (2007) 927–936.[7] H. Singh, M. Bauer, W. Chowanski, Y. Sui, D. Atkinson, S. Baurley, M. Fry, J. Evans, N. Bianchi-Berthouze, The brain’sresponse to pleasant touch: an EEG investigation of tactile caressing, Frontiers in Human Neuroscience 8 (2014) 893.[8] M. Hollins, S. R. Risner, Evidence for the duplex theory of tactile texture perception, Perception & Psychophysics 62 (4)(2000) 695–705.[9] M. N. Shadlen, W. T. Newsome, Neural basis of a perceptual decision in the parietal cortex (area LIP) of the rhesusmonkey, Journal of Neurophysiology 86 (4) (2001) 1916–1936.[10] L. P. Sugrue, G. S. Corrado, W. T. Newsome, Choosing the greater of two goods: neural currencies for valuation anddecision making, Nature Reviews Neuroscience 6 (5) (2005) 363–375.[11] J. I. Gold, M. N. Shadlen, The neural basis of decision making, Annual Review of Neuroscience 30.[12] A. K. Engel, A. Maye, M. Kurthen, P. König, Where’s the action? the pragmatic turn in cognitive science, Trends inCognitive Sciences 17 (5) (2013) 202–209.[13] P. König, N. Wilming, K. Kaspar, S. K. Nagel, S. Onat, Predictions in the light of your own action repertoire as a generalcomputational principle, Behavioral and Brain Sciences 36 (3) (2013) 219.[14] V. Gallese, G. Lakoff, The brain’s concepts: The role of the sensory-motor system in conceptual structure, CognitiveNeuropsychology 22 (3-4) (2005) 455–479.[15] A. Melnik, W. D. Hairston, D. P. Ferris, P. König, EEG correlates of sensorimotor processing: independent componentsinvolved in sensory and motor processing, Scientific Reports 7 (1) (2017) 1–15.
16] C. Genna, F. Artoni, C. Fanciullacci, C. Chisari, C. M. Oddo, S. Micera, Long-latency components of somatosensoryevoked potentials during passive tactile perception of gratings, in: 38th Annual International Conference of the IEEEEngineering in Medicine and Biology Society (EMBC), IEEE, 2016, pp. 1648–1651.[17] A. Moungou, J.-L. Thonnard, A. Mouraux, EEG frequency tagging to explore the cortical activity related to the tactileexploration of natural textures, Scientific Reports 6 (2016) 20738.[18] C. Genna, C. Oddo, C. Fanciullacci, C. Chisari, S. Micera, F. Artoni, Bilateral cortical representation of tactile roughness,Brain Research 1699 (2018) 79–88.[19] A. Moungou, E. Vezzoli, C. Lombart, B. Lemaire-Semail, J.-L. Thonnard, A. Mouraux, A novel method using EEG tocharacterize the cortical processes involved in active and passive touch, in: IEEE Haptic Symposium 2016, 2016.[20] S. Eldeeb, J. Ting, D. Erdogmus, D. Weber, M. Akcakaya, EEG-based texture classification during active touch, in: IEEE29th International Workshop on Machine Learning for Signal Processing (MLSP), 2019, pp. 1–6.[21] O. Özdenizci, Y. Wang, T. Koike-Akino, D. Erdoğmuş, Learning invariant representations from EEG via adversarialinference, IEEE Access 8 (2020) 27074–27085.[22] C. Melchiorri, Robot Teleoperation, Springer London, 2013, pp. 1–14.[23] P. Kremer, T. Wimbock, J. Artigas, S. Schatzle, K. Johl, F. Schmidt, C. Preusche, G. Hirzinger, Multimodal telepresentcontrol of DLR’s rollin’ JUSTIN, in: IEEE International Conference on Robotics and Automation, 2009, pp. 1601–1602.[24] K. J. Kuchenbecker, D. Ferguson, M. Kutzer, M. Moses, A. M. Okamura, The touch thimble: Providing fingertip con-tact feedback during point-force haptic interaction, in: Symposium on Haptic Interfaces for Virtual Environment andTeleoperator Systems, 2008, pp. 239–246.[25] C. Pacchierotti, A. Tirmizi, G. Bianchini, D. Prattichizzo, Enhancing the performance of passive teleoperation systemsvia cutaneous feedback, IEEE Transactions on Haptics 8 (4) (2015) 397–409.[26] C. Pacchierotti, L. Meli, F. Chinello, M. Malvezzi, D. Prattichizzo, Cutaneous haptic feedback to ensure the stability ofrobotic teleoperation systems, The International Journal of Robotics Research 34 (14) (2015) 1773–1787.[27] D. H. Brainard, S. Vision, The psychophysics toolbox, Spatial Vision 10 (1997) 433–436.[28] G. H. Klem, H. O. Lüders, H. Jasper, C. Elger, et al., The ten-twenty electrode system of the international federation,Electroencephalography and Clinical Neurophysiology 52 (3) (1999) 3–6.[29] P. Bashivan, I. Rish, M. Yeasin, N. Codella, Learning representations from EEG with deep recurrent-convolutional neuralnetworks, in: International Conference on Learning Representations, 2016.[30] R. T. Schirrmeister, J. T. Springenberg, L. D. J. Fiederer, M. Glasstetter, K. Eggensperger, M. Tangermann, F. Hutter,W. Burgard, T. Ball, Deep learning with convolutional neural networks for EEG decoding and visualization, Human BrainMapping 38 (11) (2017) 5391–5420.[31] V. J. Lawhern, A. J. Solon, N. R. Waytowich, S. M. Gordon, C. P. Hung, B. J. Lance, EEGNet: a compact convolutionalneural network for EEG-based brain–computer interfaces, Journal of Neural Engineering 15 (5) (2018) 056013.[32] F. Fahimi, Z. Zhang, W. B. Goh, T.-S. Lee, K. K. Ang, C. Guan, Inter-subject transfer learning with an end-to-end deepconvolutional neural network for EEG-based BCI, Journal of Neural Engineering 16 (2) (2019) 026007.[33] A. Craik, Y. He, J. L. Contreras-Vidal, Deep learning for electroencephalogram (EEG) classification tasks: a review,Journal of Neural Engineering 16 (3) (2019) 031001.[34] Q. Xie, Z. Dai, Y. Du, E. Hovy, G. Neubig, Controllable invariance through adversarial feature learning, in: Advances inNeural Information Processing Systems, 2017, pp. 585–596.[35] G. Lample, N. Zeghidour, N. Usunier, A. Bordes, L. Denoyer, M. Ranzato, Fader networks: Manipulating images bysliding attributes, in: Advances in Neural Information Processing Systems, 2017, pp. 5967–5976.[36] D. Moyer, S. Gao, R. Brekelmans, A. Galstyan, G. Ver Steeg, Invariant representations without adversarial training, in:Advances in Neural Information Processing Systems, 2018, pp. 9084–9093.[37] O. Özdenizci, Y. Wang, T. Koike-Akino, D. Erdoğmuş, Transfer learning in brain-computer interfaces with adversarialvariational autoencoders, in: 9th International IEEE/EMBS Conference on Neural Engineering (NER), IEEE, 2019, pp.207–210.[38] O. Özdenizci, Y. Wang, T. Koike-Akino, D. Erdoğmuş, Adversarial deep learning in EEG biometrics, IEEE Signal Pro-cessing Letters 26 (5) (2019) 710–714.[39] F. Chollet, Xception: Deep learning with depthwise separable convolutions, in: Computer Vision and Pattern Recognition,2017, pp. 1251–1258.[40] S. Ioffe, C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, arXivpreprint arXiv:1502.03167.[41] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: a simple way to prevent neural networksfrom overfitting, The Journal of Machine Learning Research 15 (1) (2014) 1929–1958.[42] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, in: International Conference on Learning Represen-tations, 2015.[43] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al.,Tensorflow: A system for large-scale machine learning, in: 12th { USENIX } Symposium on Operating Systems Design andImplementation ( { OSDI }16), 2016, pp. 265–283.[44] F. Chollet, Keras (2015).[45] S. Ballesteros, F. Munoz, M. Sebastian, B. Garcia, J. M. Reales, ERP evidence of tactile texture processing: Effectsof roughness and movement, in: World Haptics 2009-Third Joint EuroHaptics conference and Symposium on HapticInterfaces for Virtual Environment and Teleoperator Systems, IEEE, 2009, pp. 166–171.[46] E. Niedermeyer, F. L. da Silva, Electroencephalography: basic principles, clinical applications, and related fields, LippincottWilliams & Wilkins, 2005.