Evolving Spiking Neural Networks for Nonlinear Control Problems
Huanneng Qiu, Matthew Garratt, David Howard, Sreenatha Anavatti
EEvolving Spiking Neural Networks for NonlinearControl Problems
Huanneng Qiu
School of Engineering and Information TechnologyThe University of New South Wales atthe Australian Defence Force Academy
Canberra, [email protected]
Matthew Garratt
School of Engineering and Information TechnologyThe University of New South Wales atthe Australian Defence Force Academy
Canberra, [email protected]
David Howard
Robotics and Autonomous Systems GroupDATA61 at CSIRO
Brisbane, [email protected]
Sreenatha Anavatti
School of Engineering and Information TechnologyThe University of New South Wales atthe Australian Defence Force Academy
Canberra, [email protected]
Abstract —Spiking Neural Networks are powerful computa-tional modelling tools that have attracted much interest becauseof the bioinspired modelling of synaptic interactions betweenneurons. Most of the research employing spiking neurons hasbeen non-behavioural and discontinuous. Comparatively, thispaper presents a recurrent spiking controller that is capableof solving nonlinear control problems in continuous domainsusing a popular topology evolution algorithm as the learningmechanism. We propose two mechanisms necessary to the decod-ing of continuous signals from discrete spike transmission: (i) abackground current component to maintain frequency sufficiencyfor spike rate decoding, and (ii) a general network structure thatderives strength from topology evolution. We demonstrate thatthe proposed spiking controller can learn significantly faster todiscover functional solutions than sigmoidal neural networks insolving a classic nonlinear control problem.
Index Terms —Spiking Neural Networks, neuroevolution, re-current networks
I. I
NTRODUCTION
Artificial Neural Networks (ANNs) are learning based com-putational systems inspired by the structure of animal brains.The current generation of neural networks which are dominantin the computational intelligence community are sigmoidalneural networks, whose neurons consist of two components:a weighted sum of inputs and a sigmoidal activation functiongenerating the output accordingly. Despite their exceptionalperformance in a variety of tasks, they are vaguely related totheir biological counterparts and do not approximate the signaltransmission in biological neural systems.The idea of Spiking Neural Networks (SNNs) [1], therefore,is to bridge the gap between neuroscience and computationalintelligence, by employing biologically realistic neuron modelsto carry out computation. However, over the past decade mostof the work analysing information processing in SNNs has been focused on non-behavioural functionality, e.g., characterrecognition [2] and approximation [3]. Meanwhile, there isalso a rising interest in behaviourally functional SNNs, whichaddresses neural activities in closed-loop interaction with theenvironment [4]–[6].The most apparent advantage of neurocontrollers is thatneural networks can learn to perform satisfactory tasks withoutexplicit models of the plants. This is highly preferred in situa-tions where accurate models are difficult to obtain. SNNs canbe more suitable controllers than non-spiking neural systemsbecause: (i) they are bioinspired plastic learning architecturesand can provide faster information processing as observed inbiological neural systems [7]; (ii) they are fundamentally com-putationally more powerful than non-spiking neural systems[1]; and (iii) massive network implementation on hardwarehas already been realised [8], [9], as spiking neuron modelscan be represented using simple electric circuits.One issue of SNN implementation is that the learning pro-cess is challenging using gradient based methods, partly due tothe difficulty of extracting gradient information from discreteevents, but also because of the recurrent architecture that isessential to preserve memory through internal connections. Inthis paper, we use the popular NEAT algorithm [10] as thelearning mechanism to generate action selection policies andconsequently to seek functional network compositions. NEATis an ideal neuroevolution strategy due to (i) the efficacy ofautomatic topology altering along with connection weight; and(ii) compatibility with the discrete nature of SNNs.The main purpose of this work is to derive continuousmotion actions from discrete spike trains in a reinforcementlearning task. We claim this has not yet been fully resolvedbecause of two problems: decrease of spike activities throughsynapses and low resolution of conventional decoding meth- a r X i v : . [ c s . N E ] M a r ds. Therefore, we construct a recurrent spiking network withtwo essential settings: a background current to deal withfrequency loss, and a decoding method based on a weightedfiring rate. We show that the proposed spiking controller hasbetter performance than its sigmoidal counterpart in solvingthe classic pole balancing problem.Organization of the rest of this paper is as follows. SectionII will briefly describe the spiking neuron model and themechanism to preserve spike activities, followed by a reviewof the NEAT algorithm in Section III. Section IV presents theexperimental setup to test the controller learning algorithm andresults of both spiking and non-spiking approaches. Finally,Discussion and Conclusion will be covered in Section V andSection VI. II. C OMPUTATION WITH S PIKES
Unlike traditional neural networks, the information trans-mission in SNNs relies on electrical pulses. These so-called spikes are discrete events that occur at certain points in time.The information carried by spikes is not by the amplitude orform, but rather by the number and the timing of spikes.A spike is fired when the membrane potential of the presy-naptic neuron exceeds its threshold; then it will travel throughthe synapses and arrive at all forward-connected postsynaptic neurons. One incoming spike would increase the membranepotential of the neuron, which will decrease gradually untilit reaches the resting potential if no other arriving spikeis observed within a certain period. Therefore, a cluster ofincoming spikes would be necessary for a neuron to generatea spike.
A. Neural Model
Several spiking neuron models have been proposed over pastdecades [11]. The two-dimensional Izhikevich model [12] isused in this paper, because of its simplicity whilst being ableto produce rich firing patterns to simulate biological neuronsby adjusting only a few parameters. This model is formulatedby two ordinary differential equations: ˙ v = 0 . v + 5 v + 140 − u + I (1) ˙ u = a ( bv − u ) (2)with after-spike resetting following:if v ≥ v t , then (cid:26) v = cu = u + d (3)where v represents the membrane potential of the neuron; u represents a recovery variable; I represents the synapticcurrent injected into the neuron; v t is a threshold value; and a, b, c and d are dimensionless parameters to form differentspike patterns [12]. Fig. 1 shows the voltage response of anIzhikevich neuron when injected with a square wave currentsignal.Firing time t ( f ) is defined as the moment of v crossingthreshold v t from below: t ( f ) : v ( t ( f ) ) = v t and ˙ v ( t ( f ) ) > (4) Fig. 1. Voltage response of an Izhikevich neuron to a square wave currentsignal I corresponds to a = 0 . b = 0 . c = − d = 2 and v t = 30 mV. A spike train is then denoted as the sequence of spike times: s ( t ) = (cid:88) f δ ( t − t ( f ) ) (5)where δ ( t ) is the Dirac δ function . B. Spike Transmission
When interfacing the spiking controller with its plant, onecritical problem we should consider is how to decode spikesinto output commands and encode sensing data into spikes.The information representation of spiking neural systemscan be distinguished between rate coding and temporal coding schemes [13]. In a rate coding scheme, neural information isencoded into the number of spikes occurring during a giventime window, aka. firing rate of spikes. In a temporal codingscheme, the context is encoded into the exact timing betweenpresynaptic and postsynaptic spikes.In this paper the rate coding scheme is adopted, as it iseasier to implement. Mean firing rate is averaged during agiven time window, which will be further decoded to generatethe output signal.Encoding of sensing data will go through two stages. Inputvariables are first normalized within the range of [0,1], afterwhich, the rescaled signal will be linearly converted into acurrent value that will be injected into forward-connectedneurons through synapses. This current encoding method iscommonly used in literature, and is compatible with differentcoding schemes, as the neuron receiving larger input currentswill not only fire at higher rate, but also be earlier to spike[14].In a rate coding framework, the neuron will fire at a steady-state firing rate f , given the injected current I is fixed intime. Therefore, the spike train frequency can be defined as afunction of the magnitude of the input stimulus. Fig. 2 showsthe curve of mean firing rate of an Izhikevich neuron whenthe input current varies from zero to 200 nA. https://en.wikipedia.org/wiki/Dirac delta functionig. 2. f - I curve of an Izhikevich neuron with parameters as: a = 0 . b =0 . c = − d = 2 and v t = 30 mV. It is critical to maintain sufficient spike transmission in orderto calculate a smooth firing rate. However, as mentioned, acluster of incoming spikes are generally required to force aneuron to spike. Therefore, there tends to be a transmissionloss as spikes travel through the synapses. In this paper, wetry to resolve this problem by simply adding a backgroundcurrent, with which neurons will fire at a certain frequencyeven if there is no other input stimulus. Such an effect canalso be found in biology, where synaptic background noisecan significantly affect the neuron characteristics [15].III. N
EUROEVOLUTION
Despite the success of gradient methods in training tra-ditional multilayer neural networks, their implementation onSNNs is still problematic when extracting gradient informationfrom output spike times. Instead, population-based neuroevo-lution(NE) is an ideal learning architecture for our settings –recurrent network learning and evolution of network topology.
A. Evolutionary Algorithms (EAs)
EAs [16] are an abstraction of the natural evolution process.A general scheme of EAs is given in Algorithm 1. One ofthe cornerstones of EAs is competition based selection, whichis widely known as survival of the fittest – individuals thatare more adapted to the environment have a higher chance tocreate offspring or survive. Another basis lies in phenotypevariation. To generate different individuals, variation opera-tions (i.e. recombination (also termed crossover ) and mutation )are applied, producing new individuals that loosely resembletheir parents. The combination of variation and selectionthereby leads to a population that is better adapted to theenvironment or better able to complete a given task.However, EAs are likely to converge around one singlesolution as evolution goes on – a phenomenon known as genetic drift [16], which can lead to premature convergenceand consequently getting stuck at local optimum, because thepopulation may quickly converge on whatever network thathappens to perform best in the initial population. Therefore,
Algorithm 1
General Scheme of Evolutionary Algorithms INITIALISE a population of candidate solutions EVALUATE each individual & ASSIGN fitness values while (terminate condition not met) do SELECT parents RECOMBINATION parents MUTATE offspring EVALUATE new candidates SELECT individuals for the next generation end while the NEAT algorithm [10] is introduced in this paper toovercome this problem. B. NeuroEvolution of Augmenting Topology (NEAT)
NEAT is a powerful evolutionary approach for neural net-work learning which evolves network topologies along withconnection weights. The efficacy of NEAT is guaranteed by: (i)historical markings to solve the variable-length genome prob-lem; (ii) speciation to protect innovation and preserve networkdiversity, to avoid premature convergence; and (iii) incrementalstructural growth to avoid troublesome hand design of networktopology.Typical EAs are difficult to genetically crossover neural net-works with variant topologies, because they can only operatewithin fixed-sized genome space. Recombination of divergentgenomes tends to produce damaged offspring. However, sim-ilar network solutions sharing similar functionalities can beencoded using completely different genomes – a phenomenonknown as the
Competing Convention Problem [10]. To addressthis problem, NEAT uses historical markings which act asartificial evidence to track the origin of genes. When two genesshare the same historical marking, they are categorised asalleles. Therefore, NEAT can match up genomes representingsimilar network structures and allow mating in a rationalemanner.NEAT also uses an explicit fitness sharing scheme [16] as apopulation management approach to preserve network multi-modality. Historical markings are used as a measurement ofthe genetic similarity of network topologies, based on which,genomes are speciated into different niches (also termed species ). Individuals clustered into the same species will sharetheir fitness score together [16]. The fitness of each individualis scaled according to the number of individuals falling inthe niche. As competition within the species becomes moreintense, solutions will have a lower fitness. Thus, the sparselypopulated species will become more attractive, avoiding anyone taking over. Therefore, innovations will be protectedwithin niches to have time to optimize.Finally, an incremental growth mechanism is used in NEATto discover the least complex effective neural topology, bybeginning searching minimal network structure and graduallyexpanding to more complex networks during evolution.The NEAT algorithm has been successfully applied todifferent control problems [10], [17], [18]. However, imple-entation of NEAT has been restricted to traditional neuralsystems, with one exception [19] targeting spiking networks.
C. Speciation Measurement
NEAT use a compatibility distance function δ to determinethe similarity of network solutions. When the distance betweenany two individuals is smaller than a threshold δ t , they arecategorized into the same species. The compatibility distance δ is defined as: δ = c EN + c DN + c W (6)where E and D denote the number of excess and disjoint genes; W denotes the average connection weight difference; N is the total number of genes; c , c and c are user-definedcoefficients for altering the significance of these factors.In our second task illustrated later, we will apply a slightmodification to this function by taking account of the influenceof the decoding method. There will be a more detaileddescription in Section IV-D.IV. E XPERIMENTS
We evaluate the system’s performance using a classic non-linear control benchmark – the pole balancing problem (alsoknown as the inverted pendulum problem). This problem is notonly inherently unstable, but also capable of varying degreesof complexity by limiting the state variables provided to thecontroller, which makes it ideal for designing and testingnonlinear control methods.Previous attempts to solve this problem using SNNs withfixed-topology can be found in [20], [21]. In this paper, wetake a different approach using NEAT. Results of the pro-posed spiking controller are benchmarked against the originalsigmoidal counterpart. The original NEAT C++ source codeis available publicly , which is tailored to be amenable to ournetwork model. All the experiments are programmed in C++and performance analysis is carried out using MATLAB. A. Benchmark Problem
Fig. 3 shows the cart-pole system to be controlled, whichconsists of a cart that can move left or right within a boundedone-dimensional track, and a pole that is hinged to the cart.The problem is to balance the pole upright for as long aspossible by applying a force F t to the cart parallel to thetrack. The system has four state variables: x – cart position θ – pole angle ˙ x – cart velocity ˙ θ – pole angular velocityFor simplicity, we neglect the friction of cart on track andthat of pole on cart. The system is then formulated by twononlinear differential equations [22]: http://nn.cs.utexas.edu/?neat-c Fig. 3. Cart-pole system, taken from [22]. ¨ θ t = g sin θ + cos θ ( − F t − m p l ˙ θ sin θm c + m p ) l ( 43 − m p cos θm c + m p ) (7) ¨ x t = F t + m p l ( ˙ θ sin θ − ¨ θ cos θ ) m c + m p (8)where g = 9 . m/s denotes the gravitational acceleration; m c = 1 . kg denotes the mass of cart and m p = 0 . kgdenotes the mass of pole; l = 0 . m is half length of the pole.The discrete form of state variables are updated following: x t +1 = x t + τ ˙ x t ˙ x t +1 = ˙ x t + τ ¨ x t θ t +1 = θ t + τ ˙ θ t ˙ θ t +1 = ˙ θ t + τ ¨ θ t (9)with τ representing the time step.A force generated by the spiking controller at each timestep will be used to update the state variables following (7), (8)and (9). A failure signal is generated when the cart reaches thetrack boundary, which is ± . meters from the track centre, orif the pole tilts beyond the failure angle, which is ± degrees(or about 0.21 radian) from the vertical. B. Experimental Setup
We first start with the basic balancing task with completestate variables. This Markovian problem can act as a base per-formance measurement before we go to the more challengingnon-Markovian version without velocity information. In bothtasks the cart-pole system model is unknown to the spikingcontroller.The controller contains a population of 150 networks, whichwill be evolved using a combined EA, including a (150, 150)Evolution Strategy [16] plus species-based Elitism. Per epoch,the champion of each species is duplicated when the numberof networks in that species is larger than 5. The best ofnetworks in each species are allowed to reproduce, after which,ll parents are discarded and the remaining 150 offspring willform the next generation.Each network will be evaluated and assigned a fitness value.Fitness is defined as the number of time steps that the balancedcriteria is not violated. Otherwise a failure signal is generatedand evaluation is moved on to the next network.
C. Pole Balancing with Velocity
The first task is to balance the pole with velocity infor-mation. At initialization, each network consists of 5 spikingneurons – 4 input nodes each receiving one state variableand one output node generating a force applied to the cart;and 4 connections each connecting one input node to theoutput node, respectively. We use a probabilistic rate codingmethod here. The normalized input variables are encoded asa probability to generate a spike. At each time step, thisprobability will be used to determine the firing status of theinput neuron. Firing rate is then calculated based on the outputnode. A binary force ( F t = ± Newtons) is generated eachtime to be applied to the cart.During evolution, hidden nodes are allowed to be addedwith a probability of 0.03. Connections are added with aprobability of 0.1. The time step τ in (9) is set to 0.02 seconds.A successful solution is identified when it is able to balancethe pole for 100,000 time steps, which is equivalent to around30 minutes of simulated time.We apply NEAT to evolve both SNNs and sigmoidal net-works. Each test is run for 60 episodes. Table I shows asummary of fewest generations needed to complete the task. Afailure run means the controller fails to find a solution within100 generations. Both approaches failed once in 60 runs. Forthose successful runs, it is interesting to notice that the spikingcontroller takes fewer generations to solve the task.Example runs of both tests are shown in Fig. 4 and Fig. 5.The spiking approach has more jerky outputs due to theprobability coding method we use to encode sensing data.Smaller output errors can be observed when the magnitudeof the force F t is set smaller. D. Pole Balancing without Velocity
Our main task is to balance the pole without velocityinformation. Instead of using bang-bang control, we use acontinuous output force within [-10, 10] Newtons for this morechallenging problem. A normalized force F n is first calculatedbased on a weighted sum of firing rates from connectedneurons following a modified sigmoid function, which is thenscaled and shifted to generate the force F t : F n = 11 + exp( − σ (cid:80) w i r i ) (10) F t = 10(2 F n − (11)where σ is a positive decay variable, which will be automat-ically tuned during evolution; r i denotes the firing rate ofthe i th connected neuron and w i denotes the correspondingconnection weight. TABLE IF
EWEST G ENERATIONS R EQUIRED TO F IND
A S
UCCESSFUL S OLUTIONFOR M ARKOVIAN P OLE B ALANCING P ROBLEM
Best Worst Median a Mean a Failure RateNEAT original 3 55 17.5 21.68 1/60NEAT-SNN 1 92 9.5 16.55 1/60 a Values are calculated assuming failure runs take 101 generations.Fig. 4. NEAT original: cart position x and pole angle θ of the pole balancingtask with velocity information.Fig. 5. NEAT-SNN: cart position x and pole angle θ of the pole balancingtask with velocity information. The aforementioned compatibility distance function (6) ismodified by considering σ in (10). A component of σ differ-ence (denoted as D σ ) is added to the original compatibilitydistance: δ = c EN + c DN + c W + c D σ (12)Fig. 6 shows a possible network topology during evolution.Inputs are cart position and pole angle. Connections andnodes will be added based on a probability of 0.1 and 0.03.Recurrence is allowed within spiking neurons, facilitatinginternal calculation of derivatives. t x t F t w r w r Fig. 6. Spiking network topology that allows internal recurrence. Networkinputs consist of cart position x t and pole angle θ t . Output force F t iscalculated based on a weighted sum of firing rates (cid:80) w i r i , in which r i denotes the firing rate of the i th connected neuron and w i denotes thecorresponding connection weight.TABLE IIF EWEST G ENERATIONS R EQUIRED TO F IND
A S
UCCESSFUL S OLUTIONFOR N ON -M ARKOVIAN P OLE B ALANCING P ROBLEM
Best Worst Median a Mean a Failure RateNEAT original 8 96 44 50.35 7/60NEAT-SNN 5 65 21.5 24.70 0/60 a Values are calculated assuming failure runs take 101 generations.
The initial state variables are set to 0, except the poleangle is set to 3 degrees. Time step τ is set to 0.01 seconds.Successful solutions are dictated if the pole is balanced for5000 time steps.Similarly, we apply NEAT to the spiking controller andcompare the results against the original NEAT. A summaryis shown in Table II. The proposed spiking controller isessentially better than sigmoidal networks in solving thisproblem. It requires fewer generations to find a functionalsolution. It also has a lower failure rate over 60 runs, showingthe potential to be more adaptive. Further, the Mann-Whitney U -test is used to assess the statistic difference between the twosets of samples. The p value is smaller than 0.01, showing thatthe spiking controller has significantly better performance.To visualize the evolution progress, we average the bestnetworks’ fitness values over 60 runs. Fig. 7 shows themean and standard deviation of fitness values of both testsat successive generations. In the beginning, only some of thenetworks can find a path to optimization, thus introducinga large fitness deviation. As evolution goes on, individualswith higher fitness values will gradually take over the entirepopulation.Fig. 8 and Fig. 9 show the status of cart position, pole angleand the output force applied to the cart during a successful run.In both approaches, all these three variables are oscillatingaround a certain point. Fig. 7. Best networks’ mean fitness values in progress over 60 runs, witherror bars denoting the standard deviation.Fig. 8. NEAT original: cart position x , pole angle θ and force F t applied to thecart of the non-Markovian pole balancing task (without velocity information).Fig. 9. NEAT-SNN: cart position x , pole angle θ and force F t applied to thecart of the non-Markovian pole balancing task (without velocity information). . D ISCUSSION
The design of functional SNNs is considered to be difficult,because SNNs behave as complex systems with transientdynamics [7]. Therefore, parameter setting of spiking networkmodels to solve a given task is non-trivial and still not entirelyresolved. Apart from the neuron model, synaptic dynamicswith transmission delay also acts as a significant componentto the computation power of SNNs. We argue evolution can bebeneficial to the automatic tuning of these network parameters.Another active implementation problem is to develop ameaningful approach to transform continuous variables intoa spike representation, and vice versa. Rate coding meth-ods either require a relatively long time or large group ofhomogeneous neurons to calculate a smooth firing rate. Onthe other hand, temporal coding is considered to be morebiologically plausible. Although not yet fully understood, thiscoding scheme is evidentially able to provide faster signalprocessing and hence less reaction time.VI. C
ONCLUSION
NEAT is a performance-guaranteed neuroevolution method.This algorithm is tailored in this work to be amenable toSNNs. Through the experiments, we have demonstrated thatSNNs can solve continuous control problems by maintainingsufficient spike activities and decoding from weighted spikefrequencies. The proposed spiking controller has shown tobe more rapid than sigmoidal networks in finding functionalsolutions. Our work is a first step toward robot controland the results have encouraged further experimentation onmore challenging dynamic systems. We expect ensemble-based implementations [23] would provide redundancy toneuroevolution and hence achieve robust performance. We arealso looking at integration of synaptic plasticity as a meansto online-offline hybrid learning [24]. Our end target is toimplement the spiking controller on flight systems such asflapping wing Micro Air Vehicles.R
EFERENCES[1] W. Maass, “Networks of spiking neurons: The third generation ofneural network models,”
Neural Networks , vol. 10, no. 9, pp. 1659 –1671, 1997.[2] K. L. Rice, M. A. Bhuiyan, T. M. Taha, C. N. Vutsinas, and M. C. Smith,“FPGA implementation of Izhikevich spiking neural networks for char-acter recognition,” in , Dec 2009, pp. 451–456.[3] L. F. Abbott, B. DePasquale, and R.-M. Memmesheimer, “Buildingfunctional networks of spiking model neurons,”
Nature Neuroscience ,vol. 19, pp. 350 EP –, Feb 2016, perspective.[4] D. Floreano, N. Schoeni, G. Caprari, and J. Blynel, “Evolutionarybits’n’spikes,” in
Proceedings of the eighth international conference onArtificial life , 2003, pp. 335–344.[5] R. Batllori, C. Laramee, W. Land, and J. Schaffer, “Evolving spikingneural networks for robot control,”
Procedia Computer Science , vol. 6,pp. 329 – 334, 2011, complex adaptive sysytems.[6] D. Howard and A. Elfes, “Evolving spiking networks for turbulence-tolerant quadrotor control,”
Artificial Life 14 , 2014.[7] H. Paugam-Moisy and S. Bohte,
Computing with Spiking NeuronNetworks . Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, pp.335–376. [8] P. A. Merolla, J. V. Arthur, R. Alvarez-Icaza, A. S. Cassidy, J. Sawada,F. Akopyan, B. L. Jackson, N. Imam, C. Guo, Y. Nakamura, B. Brezzo,I. Vo, S. K. Esser, R. Appuswamy, B. Taba, A. Amir, M. D. Flickner,W. P. Risk, R. Manohar, and D. S. Modha, “A million spiking-neuronintegrated circuit with a scalable communication network and interface,”
Science , vol. 345, no. 6197, pp. 668–673, 2014.[9] J. S. Seo, B. Brezzo, Y. Liu, B. D. Parker, S. K. Esser, R. K.Montoye, B. Rajendran, J. A. Tierno, L. Chang, D. S. Modha, andD. J. Friedman, “A 45nm CMOS neuromorphic chip with a scalablearchitecture for learning in networks of spiking neurons,” in , Sept 2011, pp. 1–4.[10] K. O. Stanley and R. Miikkulainen, “Evolving neural networks throughaugmenting topologies,”
Evolutionary Computation , vol. 10, no. 2, pp.99–127, 2002.[11] W. Gerstner and W. Kistler,
Spiking Neuron Models: An Introduction .New York, NY, USA: Cambridge University Press, 2002.[12] E. M. Izhikevich, “Simple model of spiking neurons,”
IEEE Transactionson Neural Networks , vol. 14, no. 6, pp. 1569–1572, Nov 2003.[13] F. Theunissen and J. P. Miller, “Temporal encoding in nervous systems:A rigorous definition,”
Journal of Computational Neuroscience , vol. 2,no. 2, pp. 149–162, Jun 1995.[14] D. Gamez, A. K. Fidjeland, and E. Lazdins, “iSpike: a spiking neuralinterface for the iCub robot,”
Bioinspiration & Biomimetics , vol. 7,no. 2, p. 025008, 2012.[15] J.-M. Fellous, M. Rudolph, A. Destexhe, and T. Sejnowski, “Synapticbackground noise controls the input/output characteristics of singlecells in an in vitro model of in vivo activity,”
Neuroscience , vol. 122,no. 3, pp. 811 – 829, 2003.[16] A. E. Eiben, J. E. Smith et al. , Introduction to Evolutionary Computing ,2nd ed., ser. Natural Computing Series. Springer, Berlin, Heidelberg,2015.[17] D. Pardoe, M. Ryoo, and R. Miikkulainen, “Evolving neural networkensembles for control problems,” in
Proceedings of the 7th AnnualConference on Genetic and Evolutionary Computation , ser. GECCO’05. New York, NY, USA: ACM, 2005, pp. 1379–1384.[18] J. F. Shepherd III and K. Tumer, “Robust neuro-control for a microquadrotor,” in
Proceedings of the Genetic and Evolutionary ComputationConference , Portland, OR, July 2010, pp. 1131–1138.[19] A. Vandesompele, F. Walter, and F. Rhrbein, “Neuro-evolution of spikingneural networks on SpiNNaker neuromorphic hardware,” in , Dec 2016, pp.1–6.[20] M. C. Vasu and E. J. Izquierdo, “Information bottleneck in controltasks with recurrent spiking neural networks,” in
Artificial NeuralNetworks and Machine Learning – ICANN 2017 , A. Lintas, S. Rovetta,P. F. Verschure, and A. E. Villa, Eds. Cham: Springer InternationalPublishing, 2017, pp. 236–244.[21] T. S. Kang and A. Banerjee, “Learning deterministic spiking neuronfeedback controllers,” in , May 2017, pp. 2443–2450.[22] A. G. Barto, R. S. Sutton, and C. W. Anderson, “Neuronlike adaptiveelements that can solve difficult learning control problems,”
IEEETransactions on Systems, Man, and Cybernetics , vol. SMC-13, no. 5,pp. 834–846, Sept 1983.[23] D. Howard, L. Bull, and P.-L. Lanzi, “A cognitive architecture based ona learning classifier system with spiking classifiers,”
Neural ProcessingLetters , vol. 44, no. 1, pp. 125–147, Aug 2016.[24] G. Howard, E. Gale, L. Bull, B. de Lacy Costello, and A. Adamatzky,“Evolution of plastic learning in spiking networks via memristiveconnections,”