Robustness of classification ability of spiking neural networks
aa r X i v : . [ s t a t . M L ] J a n Nonlinear Dynamics manuscript No. (will be inserted by the editor)
Robustness of classification ability of spiking neural networks
Jie Yang · Pingping Zhang · Yan Liu
Received: 15 Jul 2014 / Accepted: 29 May 2015
Abstract
It is well-known that the robustness of artificialneural networks (ANNs) is important for their wide rangesof applications. In this paper, we focus on the robustness ofthe classification ability of a spiking neural network whichreceives perturbed inputs. Actually, the perturbation is al-lowed to be arbitrary styles. However, Gaussian perturba-tion and other regular ones have been rarely investigated.For classification problems, the closer to the desired point,the more perturbed points there are in the input space. Inaddition, the perturbation may be periodic. Based on thesefacts, we only consider sinusoidal and Gaussian perturba-tions in this paper. With the SpikeProp algorithm, we per-form extensive experiments on the classical XOR problemand other three benchmark datasets. The numerical resultsshow that there is not significant reduction in the classifica-tion ability of the network if the input signals are subject tosinusoidal and Gaussian perturbations.
Keywords
Robustness · Spiking neural networks · Gaussian perturbation · Classification
Lots of biological experiments and theoretical analysis havedemonstrated that the speed and scale of processing infor-mation by biological neural networks are much faster andlarger than by manual methods [1] [2]. Inspired by animals’central nervous systems in particular the brain, many kinds
Jie Yang · Pingping ZhangSchool of Mathematical Sciences, Dalian University of Technology,Dalian 116024, China.E-mail: [email protected]; [email protected] LiuSchool of information Science and Engineering, Dalian PolytechnicUniversity, Dalian 116034, China.E-mail: [email protected] of artificial neural networks (ANNs) and training methodshave presented to mimic animals’ behavior characteristics.ANNs are distributed mathematical models that process in-formation parallelly [3] [4]. They have been used to solvea wide variety of tasks that are hard to solve by using or-dinary rule-based programming, including computer visionand speech recognition [5] [6] [7] [8]. These networks baseon the characteristics and scales of data and the complex-ity of systems. By adaptively adjusting the weights whichare connected between different nodes in adjacent layers,ANNs can achieve the purpose of processing information.As a special class of ANNs, spiking neural networks (SNNs)can simulate spikes generated between animal dendrites andaxons of neurons [4] [9]. With temporal information codingin single spikes to process information, they were proved tobe a type of strong anthropomorphic networks.However, due to many uncontrollable factors, such asnoising inputs, individual spike decay times, thresholdingweights, the abilities of processing information by spikingneural networks may be affected [10] [11]. In order to getan available network architecture, it is important to do someresearch on its robustness. Because of encoding input vari-ables by time differences between pulses, spiking neuronsare particularly sensitive to the input signals. Recently, thishas led to several explorations on the computational abili-ties and learning performance of neuromorphic networks ofspiking neurons with noise [12] [13]. However, these workshave not considered the type of perturbations and the robust-ness of classification ability of SNNs.To analyze the robustness of SNNs’ classification abil-ities, in this paper we performed a series of numerical ex-periments on the classical XOR problem and other threebenchmark datasets ( i.e. , Iris dataset, Wisconsin breast can-cer dataset and StatLog landsat dataset) with the SpikePropalgorithm [14]. Notably, the closer the perturbed inputs tothe desired points, the more perturbed points there are in the
Jie Yang et al.
Fig. 1
The structure of a simple spiking neural network. input space. What’s more, the perturbation may be periodicin practice. These facts led us to consider the sinusoidal andGaussian perturbations in this paper.To summarize, our main contributions include: – As far as we know, this is the first work to validate the ro-bustness of classification ability of SNNs which receiveperturbed inputs. – Two kinds of perturbations were considered for robust-ness of classification ability of a SNN. In fact, perturba-tions could be arbitrary styles performing on the inputs.We only focus on sinusoidal and Gaussian perturbations. – On the classical XOR problem and other three bench-mark datasets, we evaluate the classification ability ofSNNs and show its robustness experimentally.
Compared with traditional neural networks (such as BP),SNNs have several differences in network architectures. Themost important one is that there are multiple synaptic termi-nals and the specific synaptic delay between spiking neuronsin adjacent layers. In addition, due to the fast temporal en-coding which is very different from traditional rate-codednetworks, spiking neurons can significantly improve com-plex non-linear classification performances [4] [14].2.1 The architectures of SNNsA simple feed-forward SNN with multiple input spiking neu-rons and one output spiking neuron is shown in Fig 1. Thenetwork architecture consists of one input layer, one hiddenlayer and one output layer, denoted by I, H and O respec-tively. Each connection between different layers comprisesseveral synapses and each neuron receives a set of spikesfrom all its previous neurons. Formally, assuming that dur-ing the simulation interval each neuron generates at mostone spike and fires when the internal state variable reachesa threshold, the state variable x j of neuron j receives out-puts from all its previous neurons as a weighted sum of thepre-synaptic contributions: x j ( t ) = Σ i ∈ D j Σ mk = w ki j y ki ( t ) (1) where D j denotes the set of pre-synaptic neurons associatedwith neuron j , w i j is the weight associated with synapticterminal k , and y ki ( t ) represents a delayed pre-synaptic po-tential (PSP) for each terminal, y ki ( t ) = ε ( t − t i − d k ) (2)with ε ( t ) a spike-response function. The time t i is the firingtime of pre-synaptic neuron i , and d k is the delay associatedwith the synaptic terminal k . The firing time is determined asthe first time when the state variable reaches the threshold.The spike-response function is always described as the form ε ( t ) = (cid:26) t τ exp ( − t τ ) , t > , t ≤ τ models the membrane potential decay time constantthat determines the rise and decay time of the PSP.2.2 Learning algorithmThe basic SpikeProp algorithm [14] is performed and wechoose the least mean square as the error-function. Givendesired spike times { t dj } and actual firing times { t j } , we canderive the form of the error-function E = Σ j ( t j − t dj ) . (4)For error back-propagation, the weights update rule is fol-lowed: w k + i j ( t j ) = w ki j ( t j ) + ∆ w ki j ( t j ) (5)Define δ j = t dj − t j Σ il w li j ∂ y li ( t j ) ∂ t j (6)In the output layer, the basic weight adaptation function forneurons is derived as ∆ w ki j ( t j ) = − η y ki ( t j )( t dj − t j ) Σ il w li j ∂ y li ( t j ) ∂ t j = − η y ki ( t j ) δ j . (7)For the hidden layers, the weight adaptation function forneurons is given by ∆ w khi ( t j ) = − η y kh ( t j ) Σ j { δ j Σ k w ki j ∂ y ki ( t j ) ∂ t j } Σ nl w lni ∂ y ln ( t j ) ∂ t j (8)where η is the learning rate of the network. For more details,see [14] [15] [16]. obustness of classification ability of spiking neural networks 3 x = ˜ x + σ (9)with ˜ x being the original input, and σ the noise term. Theformula means if ˜ x = ( a , b ) , the perturbed input ˜ x will be ( a + σ , b + σ ) . Table 1
Encoded inputs and outputs for XOR problemInput patterns Output patterns0 0 160 6 106 0 106 6 16 (1)Sinusoidal perturbationsThe original input ˜ x is perturbed by a sinusoidal perturba-tion term, which can be expressed as˜ x = ˜ x + A sin ( π y ) (10)where A is a constant between (0,1] to control the perturba-tion amplitude, and y is a random vector and its componentvalues belong to [0,1]. The numbers of components associ-ated with ˜ x and y are the same.(2)Gaussian perturbationsThe original input ˜ x is perturbed by a Gaussian perturba-tion. This is different from the sinusoidal perturbation, andcan be expressed as˜ x = ˜ x + ˜ x ( I − exp ( − r / ) · sgn ( l )) (11)where I is an identity matrix, r is a random vector whosecomponent values belong to [0,1] and sgn ( l ) is the signalfunction associated with l . Here the numbers of componentsassociated with ˜ x , r and l are also the same. Since our aim is to assess the robustness of classificationability of SNNs, we don’t need to design complex SNNs.We train a simple spiking neural network on XOR problemand other three benchmark datasets.4.1 XOR problemFirstly, the input and output signals of spiking neural net-works are coded as in [14]. If
Max and
Min are extremalvalues of a variable x (e.g.an input signal), we can encode itas a spike fired in the time f ( x ) = x − MinMax − Min · L (12)with the length of coding interval L .For a better representation of spike-time patterns, we canassociate a 0 with a “late” firing time and a 1 with an “early”firing time. With specific values 0 and 6 for the respective in-put times, we lead to the temporally encoded XOR [14] [20]showing in Table 1. In the table, the input numbers repre-sent spike times (i.e. late firing times and early firing times)in milliseconds. The actual input patterns contain a setting Jie Yang et al.
Table 2
The result of the correct classification with sinusoidal per-turbations. ROS: Rates of the correct classification without sinusoidalperturbations, RWS: Rates of the correct classification with sinusoidalperturbations. Epoches A ROS RWS50 0.001 90.50 91.0050 0.01 89.50 87.8050 0.1 91.00 88.9050 0.2 88.50 87.2050 0.5 87.50 82.2450 0.8 87.50 85.58 threshold by adding a third input neuron. Here we define thedifference between the times equivalent with “0” and “1” asthe coding interval L = τ = m =
16) were configurated between each pair ofsynaptic neurons in adjacent layers.Using computer simulations, we randomly generated 400perturbed samples for different r to see the distribution ofperturbed samples as shown in Fig 2. Considering the com-putational complexity and time, we only used 160 perturbedsamples for testing. We varied A and r ∗ (the least upperbound of all components to the random vector r )to controlperturbation amplitudes. The network reliably learned theXOR patterns with η = .
01. With different perturbations,we got the correct classification rate of the spiking neuralnetwork. The average of correct classification rates with andwithout perturbed data are compared in Table 2 and Table 3.As results are reported in Table 2 and Table 3, the clas-sification accuracies associated with original data are differ-ent but almost equal (about 90%).This is mainly because theinput data were reordered after each epoch. The larger per-turbation was performed to the input data, the more greatlyrates of correct classification with sinusoidal disturbancesdecreased (see Table 2). When we disturbed the input datawith Gaussian perturbations, the network did not got simi-lar results (see Table 3). The correct classification rates ofthe network did not fall too much. The reason may be thatmost of the perturbed data by Gaussian perturbations wereclustered around the desired value. Therefore, it indicates
Table 3
The result of the correct classification with Gaussian pertur-bations. ROG: Rates of the correct classification without Gaussian per-turbations, RWG: Rates of the correct classification with Gaussian per-turbations. Epoches r ∗ σ ROG RWG50 0.1 92.00 88.6250 0.2 92.00 88.4550 0.3 89.50 87.8050 0.4 89.50 88.1550 0.5 88.50 85.00100 0.1 88.25 88.66100 0.2 89.75 89.60100 0.3 91.75 90.86100 0.4 91.00 89.76100 0.5 88.25 88.81 spiking neural networks have strong anti-interference abili-ties.4.2 Other three benchmarksTo further validate robustness of classification ability of SNNsin practice, we consider the following three benchmarks withrealistic significance. As XOR problem does, we first adoptthe method in [14] to encode continuous input variables inspike times.
Specially, for a variable n with a range[ I nmin ,..., I nmax ],we use N neurons with Gaussian receptive fields to encodethe input variable. For a neuron i , its center was set to I nmin +( i − ) / ·{ I nmax − I nmin } / ( N − ) and width σ = / β ·{ I nmax − I nmin } / ( N − ) . We set β = . m = obustness of classification ability of spiking neural networks 5 y r*=3.0,sigma=0.5 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 200.20.40.60.811.21.41.61.82 x y r*=3.0,sigma=1.0 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 200.20.40.60.811.21.41.61.82 x y r*=3.0,sigma=1.50 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 200.20.40.60.811.21.41.61.82 x y r*=2.0,sigma=1.0 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 200.20.40.60.811.21.41.61.82 x y r*=4.0,sigma=1.0 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 200.20.40.60.811.21.41.61.82 x y r*=8.0,sigma=1.0 Fig. 2
The perturbed samples for different r ∗ . The desired point is (1,1) and the closer to the desired point, the more perturbed points there are.Besides, the perturbed points become more scattered when r ∗ increases. Table 4
The result of the correct classification with sinusoidal pertur-bations on Iris dataset. ROS: Rates of the correct classification withoutsinusoidal perturbations, RWS: Rates of the correct classification withsinusoidal perturbations.Epoches A ROS RWS500 0.001 96.50 95.71500 0.01 94.60 93.84500 0.1 91.73 88.90500 0.2 91.50 88.40500 0.5 89.35 87.62500 0.8 88.50 87.58 (2) Wisconsin breast cancer (Original) datasetThe breast cancer (Original) dataset [22] is from the Uni-versity of Wisconsin Hospitals and contains 699 case en-tries, divided into benign and malignant cases. Each casehas 9 measurements and each measurement is assigned aninteger between 1 and 10, with larger numbers indicatinga greater likelihood of malignancy. In our experiments, weencoded each measurement with 7 equally spaced neuronscovering the input range. We set 15 hidden layer neurons
Table 5
The result of the correct classification with Gaussian pertur-bations on Iris dataset. ROG: Rates of the correct classification withoutGaussian perturbations, RWG: Rates of the correct classification withGaussian perturbations.Epoches r ∗ ROG RWG750 0.1 96.10 96.02750 0.2 94.80 94.43750 0.3 94.50 94.05750 0.4 91.50 90.13750 0.5 89.56 88.001000 0.1 96.21 96.661000 0.2 95.75 94.901000 0.3 94.25 93.761000 0.4 91.00 89.741000 0.5 89.25 88.851500 0.1 96.25 96.011500 0.2 95.35 94.601500 0.3 91.27 90.861500 0.4 91.08 90.671500 0.5 89.21 89.01 and 2 output layer neurons. The results are presented in Ta-ble 6 and Table 7.
Jie Yang et al.
Table 6
The result of the correct classification with sinusoidal pertur-bations on Wisconsin breast cancer (Original) dataset. ROS: Rates ofthe correct classification without sinusoidal perturbations, RWS: Ratesof the correct classification with sinusoidal perturbations.Epoches A ROS RWS1500 0.001 97.50 97.601500 0.01 97.34 97.201500 0.1 95.60 95.531500 0.2 95.50 93.801500 0.5 96.02 94.841500 0.8 93.56 91.68
Table 7
The result of the correct classification with Gaussian pertur-bations on Wisconsin breast cancer (Original) dataset. ROG: Rates ofthe correct classification without Gaussian perturbations, RWG: Ratesof the correct classification with Gaussian perturbations.Epoches r ∗ ROG RWG1000 0.1 95.75 96.061000 0.2 95.85 94.601000 0.3 94.75 94.861000 0.4 92.00 91.961000 0.5 91.57 91.171500 0.1 97.40 97.521500 0.2 97.13 96.591500 0.3 95.57 93.861500 0.4 96.03 95.451500 0.5 93.54 91.60 (3) StatLog landsat datasetTo test the robustness of SNNs on a larger dataset, weinvestigated the Landsat dataset as described in the StatLogsurvey of machine learning algorithms [23]. This datasetconsists of a training set of 4435 cases and a test set of 2000cases and contains 6 ground cover types (classes). Each sam-ple contains the values of a 33 pixel patch and each pixel isdescribed by 4 spectral bands. For classification of a singlepixel, each case contains the values of a 33 pixel patch, witheach pixel described by 4 spectral bands, totaling 36 inputsper case. For each band, we used the average value of cor-responding bands of 9 pixels as a new band of a pixel. Thenthe case was represented with one average pixel and eachseparate band of it was encoded with 25 neurons. In this setof experments The results obtained by the Statlog survey aresummarized in Table 8 and Table 9.From the above results, we could conclude that the ap-plication of SNNs with SpikeProp algorithm on temporallyencoded versions of benchmark problems yields a satisfac-tory robustness for sinusoidal and Gaussian perturbations.
In this work, the robustness of the classification ability ofSNNs has been investigated by disturbing input signals with
Table 8
The result of the correct classification with sinusoidal pertur-bations on StatLog landsat dataset. ROS: Rates of the correct classi-fication without sinusoidal perturbations, RWS: Rates of the correctclassification with sinusoidal perturbations.Epoches A ROS RWS6000 0.001 85.50 85.616000 0.01 85.17 84.806000 0.1 85.00 84.906000 0.2 85.21 85.206000 0.5 84.50 82.326000 0.8 83.10 82.04
Table 9
The result of the correct classification with Gaussian pertur-bations on StatLog landsat dataset. ROG: Rates of the correct clas-sification without Gaussian perturbations, RWG: Rates of the correctclassification with Gaussian perturbations.Epoches r ∗ ROG RWG6000 0.1 85.30 85.026000 0.2 85.07 84.456000 0.3 83.50 82.836000 0.4 83.46 83.156000 0.5 81.56 81.007500 0.1 85.60 85.627500 0.2 85.00 84.757500 0.3 84.50 83.807500 0.4 82.58 82.157500 0.5 81.80 80.97 different perturbation methods not only for the classical XORproblem but also for some other complicated realistic prob-lems. From the experiments results, it can be concluded thatnevertheless perturbations will affect the classification abil-ity of SNNs, the classification ability does not decrease dra-matically and the networks have a certain anti-interferencecapability.