Benchmarking Deep Spiking Neural Networks on Neuromorphic Hardware
Christoph Ostrau, Jonas Homburg, Christian Klarhorst, Michael Thies, Ulrich Rückert
BBenchmarking Deep Spiking Neural Networkson Neuromorphic Hardware
Christoph Ostrau [0000 − − − , Jonas Homburg [0000 − − − ,Christian Klarhorst , Michael Thies , and Ulrich Rckert Technical Faculty, Bielefeld University, Bielefeld, Germany [email protected]
Abstract.
With more and more event-based neuromorphic hardwaresystems being developed at universities and in industry, there is a grow-ing need for assessing their performance with domain specific measures.In this work, we use the methodology of converting pre-trained non-spiking to spiking neural networks to evaluate the performance loss andmeasure the energy-per-inference for three neuromorphic hardware sys-tems (BrainScaleS, Spikey, SpiNNaker) and common simulation frame-works for CPU (NEST) and CPU/GPU (GeNN). For analog hardwarewe further apply a re-training technique known as hardware-in-the-looptraining to cope with device mismatch. This analysis is performed for fivedifferent networks, including three networks that have been found by anautomated optimization with a neural architecture search framework.We demonstrate that the conversion loss is usually below one percent fordigital implementations, and moderately higher for analog systems withthe benefit of much lower energy-per-inference costs.
Keywords:
Spiking Neural Networks · Neural Architecture Search · Benchmark.
Diverse event-based neuromorphic hardware systems promise the accelerated ex-ecution of so called spiking neural networks (SNN), also referred to as the thirdgeneration of neural networks [14]. The most prominent representatives of thisclass of hardware accelerators include the platforms Braindrop [16], BrainScaleS[22], DYNAPs [15], Loihi [5], SpiNNaker [8] and Truenorth [1]. With the di-versity of hardware accelerators comes a problem for potential end-users: whichplatform is suited best for a given spiking neural network algorithm, possibly re-specting inherent resource requirements for embedding in mobile robots or smartdevices. Usually, this question is answered by evaluating a set of benchmarks onall qualified systems, which measure the state-of-the-art and quantify progressin future hardware generations (see e.g. [4])). Here, we face two major challengeswith neuromorphic hardware. First, there is no universal interface to all hard-ware/software simulators despite some projects like PyNN [6]. Second, there arequite a few promising network models and learning strategies, but still “the” a r X i v : . [ c s . N E ] A p r C. Ostrau et al. algorithm for spiking neural networks is missing. One recent system overarchingnetwork is the cortical microcircuit model [2,13]. A follow-up publication [21]shows, how this benchmark has driven platform specific optimization that, inthe end, improves the execution of various networks on the SpiNNaker platformconfirming the value of benchmarks. However, it is also an example of a platformspecific implementation to reach maximal performance on a given system.One commonly agreed application for spiking neural networks is the con-version of conventionally trained artificial neural networks (ANN) to rate-basedSNNs [7]. Although this is not using SNNs in their most efficient way, it is apragmatic approach that is suitable to be ported to different accelerators, inde-pendent of their nature. In this work, we use this approach for evaluating fivedistinct networks, either defined by hardware restrictions, by already publishedwork, or by employing neural architecture search (NAS) with Lamarck ML [11]to optimize the network topology. We evaluate these networks on BrainScaleS,Spikey [20], and SpiNNaker as well as the CPU simulator NEST [9] and theCPU/GPU code-generation framework GeNN [25]. Furthermore, we use a re-training approach with neuromorphic hardware-in-the-loop (HIL) proposed in[23] to unlock the full potential of the analog neuromorphic hardware systems.Section 2 outlines the target systems, the software environment, and the usedmethods. Section 3 presents the results, including neuron parameter optimiza-tion, and accuracy along with energy measurements for all target platforms.
In the following we introduce all target systems and the software environmentas well as the methodology followed.
All target systems in this work support the simulation or emulation of leakyintegrate-and-fire neurons with conductance-based synapses, although especiallyanalog systems are limited to specific neuron models.
NEST is a scaleable soft-ware simulator suited to simulate small as well as extensive networks on computeclusters. It is used in version 2.18 [12] executed with four threads on an IntelCore i7-4710MQ mobile processor.
GeNN [25] is a code generation frameworkfor the simulation of SNNs. In its current release version (4.2.1) , it supportsgenerating code for a single-threaded CPU simulation or for graphics processingunits (GPU) supporting NVIDIA CUDA. Networks are evaluated on a NVIDIAGeForce 1080 TI GPU; runtimes are measured for networks without recordingany spikes due to the overhead of getting spikes back from GPU, which effec-tively stops the simulation at every time step and copies the data between GPUand CPU. For this publication we make use of single precision accuracy and allsimulators use a time step of 1 ms. However, NEST is using an adaptive time-step to integrate the neuron model. The fully digital many-core architecture Here, we use the most recent GeNN from github (end of April 2020)enchmarking Deep Spiking Neural Networks on Neuromorphic Hardware 3
SpiNNaker [8] comes in two different sizes, which are both used in this work.The smaller SpiNN3 system is composed of four chips; the larger SpiNN5 boardconsists of 48 chips. A single chip comprises 18 ARM968 general purpose CPUcores, with each simulating up to 255
IF_cond_exp neurons. The system runs inreal-time, simulating 1 ms of model time in 1 ms wall clock time. SpiNNaker isused with the latest released software version 5.1.0 using PyNN 0.9.4. Finally, wemake use of two mixed-signal (analog neural circuits, digital interconnect) sys-tems: First, the
Spikey system [20] supports the emulation of 384 neurons with256 synapses each. The emulated neuron model is subject to restricted parame-ter ranges (e.g. four bit weights, limited time constants) with some parametersprescribed by the hardware (e.g. the membrane capacitance). The system runs ata speedup of 10 , . µ s to emulate 1 ms of model time.Second, Spikey’s successor BrainScaleS [22] shares many of Spikey’s proper-ties. Most notably is the now fully parameterizable neuron model, as well as theusage of wafer-scale integration, combining 384 accessible HICANN chips on asingle wafer for a full system. Each chip implements 512 neuron circuits with 220synapses each, where up to 64 circuits can be combined to form a single virtualneuron, allowing more robust emulations and a higher synapse fan-in.While all of these platforms formally support the
PyNN
API [6], the sup-ported API versions differ between simulators impeding the portability of code.
Cypress [24] is a C++ framework abstracting away these differences. ForNEST, Spikey and SpiNNaker the framework makes use of their PyNN in-terfaces, however, for BrainScaleS and GeNN a lower-level C++ interface isused. Furthermore, the proposed networks studied below are part of the S piking N eural A rchitecture B enchmark Suite (SNABSuite)[17,18], which also coversbenchmarks like low-level synthetic characterizations and application-inspired(sub-)tasks with an associated framework for automated evaluation.Energy measurements have been taken with a Ruideng UM25C power meter(SpiNNaker, Spikey), with a PeakTech 9035 for CPU simulations, or with theNVIDIA smi tool. There is no possibility for remote energy measurements onthe BrainScaleS system. Thus, the values have been estimated from the numberof pre-synaptic events using published data in [23]. This work is based on the idea of [3,7], where a pre-trained artificial neuralnetwork is converted into a SNN. In this case, we train several multi-layer per-ceptrons that differ in size to classify MNIST handwritten digits. The train-ing uses standard batch-wise gradient-descent in combination with error back-propagation. Conversion exploits that the activation curve of a LIF neuron re-sembles the ReLU activation curve, such that float (analog) values of the ANNbecome spike rates in the SNN. All weights of the ANN are normalized to https://github.com/hbp-unibi/cypress The code for this and other work can be found at https://github.com/hbp-unibi/SNABSuite
C. Ostrau et al. N e u r on I D Fig. 1.
Output spikes for converted networks. Left: Output spikes of a network thathas been trained using a softmax layer as the last layer. Right: The same networktrained with only ReLU activation functions. the maximal weight of the full network, and then scaled to a maximal valueeither given by restrictions of the hardware platform (e.g. 4 bit weights onSpikey/BrainScaleS) or determined by parameter optimization (see below fordetails). Similarly, other parameters of the SNN are found by extensive parame-ter tuning or are fixed due to hardware constraints. Neuron biases are not easilyand efficiently mapped to SNNs, which is why we set all bias terms to zero inthe training process of the ANN. In contrast to [7], we found that using a soft-max layer as the last layer in the ANN for training does not necessarily decreasethe performance of the SNN. However, using soft-max will lead to an increasednumber of spikes for all rejected classes (cf. Figure 1).As the Spikey platform is very limited in size and connectivity, the smallestand simplest network (referred to as
Spikey network ) consists of a single hiddenlayer with 100 neurons and no inhibitory connections. Spikey requires separationof excitation and inhibition at the neuron level and consists of two separate chipswith limited connectivity between them. Thus, we only used positive weights andachieved the best performance using a hinge loss, which increases the weightsfor the winner neurons and decreases weights for the second place neuron only.Due to the acceleration factor of Spikey and BrainScaleS, communication band-width limits the usable spike rates. Too high rates (input as well as inter-neuronrates) will inevitably lead to spike loss that would reduce the performance ofthe network. This naturally restricts the parameter space to be evaluated. Still,there is a significant performance loss when applying the conversion process foranalog systems. Perfect conversion requires that every synapse with the sameweight and every neuron behaves in the same way, referring to identical activa-tion curves. On analog systems, however, we have to deal with temporal noiseperturbing the membrane voltage, trial-to-trial variation and analog mismatchbetween circuits [19]. As shown in [24], such a hardware network will perform atroughly 60-70% accuracy compared to a simulator, even after platform specificparameter tuning. [23] proposed to train the pre-trained neural network againwhile replacing the outputs of the ANN with spike rates recorded from hardwareemploying back-propagation to train a device specific network. All details canbe found in [23] (Figure 7). enchmarking Deep Spiking Neural Networks on Neuromorphic Hardware 5
Lamarck ML [11] is a modular and extensible Python library for applicationdriven exploration of network architectures. This library allows to define a classof network architectures to be examined and operations to modify and combinethose architectures. These definitions are then used by a search algorithm toexplore and evaluate network architectures in order to maximize an objectivefunction. For this work, the limitations of the neuromorphic hardware systemscompared to state-of-the-art processing units are the leading motivation for theapplied restrictions. The applied layer types are limited to fully connected lay-ers which may be arranged in a nonsequential manner resulting in an acyclicdirected graph structure. To preserve the structural information of a single neu-ral network in the exploration process, a meta graph is created to contain thecurrent network and the meta graph of the networks which were involved in cre-ating it. This process is unbounded and accumulates structural information overseveral generations in the meta graph. To forget unprofitable information, themeta graph is designed to dismiss structural information that has not been usedin the last five exploration steps. One exploration step consists of combining themeta graph of two network architectures and sampling a new path in this metagraph in order to create an improved architecture. A new architecture is createdby sampling a path based on the quality of its best architecture and amendingit with elements that have not been examined before.The exploration procedure is performed by a genetic algorithm configuredwith a generation size of 36 network architectures of which 20 are selected basedon an exponential ranking to create new architectures for the next generation.This next generation is created with an elitism replacement scheme that pre-serves the best two network architectures of the previous generation. In total 75generations have been created in the NAS to find an architecture that achievesat least 97% evaluation accuracy. Above this threshold, an architecture is definedto be better if it requires less than 100 neurons for increasing the accuracy by1%. The first two parts of this section present the parameter tuning process used forthe converted SNNs. Details of four different networks are shown, the smallestone was defined by the restrictions of the Spikey platform, while the remainingnetworks were picked from the neural architecture search. The final part gathersthe results for all networks including one model taken from literature.
This is the simplest network used in this work. As described above, it is motivatedby the hardware restriction of the Spikey neuromorphic hardware system and https://github.com/JonasDHomburg/LAMARCK_ML C. Ostrau et al.
Fig. 2.
Visualization of the down-scaled and converted images. The top left row showsthe first five images of the MNIST training data set. The bottom left row shows down-scaled images using 3 × uses a 89 × ×
10 layout which requires images to be scaled down using 3 × . – The maximal weight determines the incoming activity per neuron. If cho-sen too high, the neuron operates in its non-linear and saturating range nearthe maximum output frequency. – The leakage/membrane time constant describes the time window inwhich the neuron integrates incoming input. Too small values would requirehigh frequencies for encoding analog values while higher numbers lead tosaturation effects. – The sample presentation time increases accuracy with higher values,which in turn require more energy and time. – A higher frequency range of input pixels improves the pixel approxima-tion accuracy, but is subject to saturation of neurons.Figure 3 shows parameter sweeps over the two most essential neuron parametersfor the training set. The images show large areas of high relative accuracy forthe analog platforms. On the simulated platforms, one can see the discussedeffects of saturating neurons at high weights/time constants. Here, the area of
10 20 300 . . NEST
10 20 300 . . SpiNNaker . . . Spikey M a x . b it - w e i gh t BrainScaleS . . . A cc u r ac y τ m in ms0 . . . . .
000 10 . . . . . M a x . S yn a p ti c w e i gh ti n µ S Fig. 3.
Sweep over the maximal input frequency. Weights for BrainScaleS are set vialow level digital weights (0 to 15).enchmarking Deep Spiking Neural Networks on Neuromorphic Hardware 7
250 500 750 1000Presentation time in s0 . . . . A cc u r ac y
50 100 150Max. Input Spike Frequency in 1 / s SpiNNakerGeNNNESTBrainScaleSSpikey Fig. 4.
Sweep over the sample presentation time (left) and the maximal input frequency(right) high relative accuracy is rather narrow. Therefore, careful parameter tuning hasto be done.Taking a look at the most relevant conversion parameters, Figure 4 showsthe accuracy in relation to the sample presentation time and the maximal spikeinput frequency. First, simulating more than 200 ms will result in minor improve-ments only. Analog platforms converge a bit slower (which is partially causedby different neuron parameters used in the simulation), and the benefits of us-ing presentation times larger than 200 ms are minor again. However, prolongedpresentation times can cancel out some of the temporal noise on membranevoltages and synapses. Second, all platforms gain significantly from frequencieslarger than 40 Hz. However, due to communication constraints in the acceler-ated analog platforms, the accuracy decreases for values above 60 Hz. Here, twobandwidth restrictions may play a major role: input spikes are inserted into thedigital network using FPGAs. Any spike loss is usually reported by the respec-tive software layer. However, on the wafer, there might be additional loss in theinternal network, which is not reported. Output rates of hidden and ouput layersare a second source of potential spike loss which is only partially reported forthe Spikey system (by monitoring spike buffers), but happens silently on theBrainScaleS system. The Spikey system reports full buffers for larger frequen-cies, which is why we assume that this is the major cause for spike loss on bothsystems.To reach a high efficiency on larger systems, like SpiNN5 or GPUs, it is crucialto fully utilize them. Therefore, we used several parallel instances of the samenetwork each classifying a separate portion of the data. In our setup this is con-trolled by choosing the batch size: a smaller batch size leads to more independentbatches processed in parallel. On SpiNNaker the hardware size and the requirednumber of processor cores per network instance determine the parallelism. OnGeNN the working memory required to compile the GPU code is the determin-ing factor. The latter is a limitation caused by using separate populations perlayer, which could be merged to possibly lead to an increased parallelism of thenetworks, but not necessarily to increased efficiency.
C. Ostrau et al.
Neurons A cc u r a c y Input: 78486652353210SoftMax
Neurons-Accuracy Pareto
Input: 784431010SoftMax Input: 78452353210SoftMax
Test Acc as seen by GATest Acc as seen by GATest Acc
Fig. 5.
Results of the optimization process. Highlighted are three candidates networksat the pareto front with their respective network layout.
The optimization process was driven by two major goals: to reach an accuracylarger than 97% and at the same time to reduce the network size in terms of thenumber of neurons. Results in Figure 5 reveal, that this not necessarily leads tonetworks with a single hidden layer. Furthermore, the sequential neural networksoutperformed all evaluated non-sequential architectures. We have chosen threecandidates on the pareto-front for evaluation on neuromorphic hardware: – the network with the highest evaluation accuracy ( NAStop , 97.71%) – the optimal network with the best trade-off ( NAS129 , 97.53%) – a small network with still sufficient accuracy ( NAS63 , 96,76%)
Table 3.3 collects the results for all target platforms. Most striking is the en-ergy efficiency of the analog platforms, which is two orders of magnitude highercompared to other simulators. Furthermore, HIL training recovers most of theconversion losses found for these platforms (despite the four bit weight accu-racy). Larger networks have not been evaluated either due to size restrictions,or because combined spike rates of input pixels are too high to get any rea-sonable results. The SpiNNaker system, in both variants, performs on the sameefficiency level as a CPU/GPU implementations although its technology is mucholder (130 nm vs. 22 nm CPU vs. 16 nm GPU). Furthermore, there is less than enchmarking Deep Spiking Neural Networks on Neuromorphic Hardware 9
Table 1.
Results from all converted networks. † Reduced number of neurons per corefrom its default 255 to 200 and × further reduced to 180 together with a slowed-downsimulation (factor 2).Platform Accuracy Wall clock time Energy per Inference Batchsizein % in ms in mJ Spikey Network (ANN accuracy: 90.13%)Spikey 0.6533 350 0.21 2500Spikey HIL 0.8499 350 0.21 100BrainScaleS 0.6165 900 0.33 10000BrainScaleS HIL 0.8387 900 0.36 10000SpiNN3 0.8841 264000 79 480SpiNN5 0.8840 23100 61 42NEST 0.8898 70542 316 2500GeNN CPU 0.8911 5070 10 10000GeNN GPU 0.8887 2623 21 100
NAS63 (ANN accuracy: 96,76%)SpiNN3 0.9604 368500 109 670SpiNN5 0.9604 30800 80 56NEST 0.9637 217252 952 10000GeNN CPU 0.9629 16659 31 10000GeNN GPU 0.9632 17881 145 160
NAS129 (ANN accuracy: 97,53%)SpiNN3 0.9686 458700 138 834SpiNN5 0.9725 38500 105 70NEST 0.9710 263134 1247 10000GeNN CPU 0.9742 20436 38 10000GeNN GPU 0.9734 18495 153 200
NAStop (ANN accuracy: 97,71%)SpiNN3 † † × † one percent loss in accuracy due to the conversion in almost all cases. However,for the large networks the system was performing at its limits, and we had toreduce the maximal number of neurons per core. Of course, this can be miti-gated by further reducing the number of neurons per core or slowing down thesystem with respective negative impacts on the energy per inference. Interestingdifferences have been found for NEST: in some cases the accuracy is a bit lower,but the energy per inference is one order higher than for the GeNN CPU simu-lation. The latter is mainly due to the more accurate integrator employed by theNEST simulator (especially the adaptive time step in the integrator), which isalso responsible for the significant energy gap between the two CPU simulatorsNEST and GeNN. Furthermore, the multi-threaded execution of NEST does notreduce the computation time compared to GeNN. With the increase of networkcomplexity there is next to no increase in GPU execution time, indicating thatdespite parallelization of the networks, the GPU is still not utilized fully for thesmaller networks (there are 3969-86,760 simultaneously simulated neurons forthe GPU depending on the network). Still, for the larger networks, the GPUimplementation is the fastest simulation available.The last network in Table 3.3 is taken from [7], as the network weights arepublished within the respective repository. The layout is 784 × × × We have demonstrated the capability of all target platforms to simulate con-verted deep neural networks. The loss in the conversion process is negligiblein many cases, and for analog platforms we successfully employed retraining toreach high accuracy. Furthermore, we calculated the used energy-per-inferencefor all networks and platforms, quantifying the efficiency vs. accuracy trade-offof analog platforms. The digital SpiNNaker platform is highly efficient if fullyutilized. If primarily simulation time needs to be optimized, GeNN and its GPUbackend allow fast and efficient simulation of SNNs. The approach used in thiswork is not the most efficient way of using spiking neural networks. However, therate-coding applied here can be replaced with a more efficient time-to-first-spikeencoding, using only a few spikes with much faster response times, which hasrecently been demonstrated on analog hardware [10]. Therefore, the results fromthis work must be seen as a conservative measure for the relative efficiency ofSNNs on neuromorphic hardware. Furthermore, we did not make use of convolu-tional networks, because these currently cannot be mapped well to neuromorphichardware. enchmarking Deep Spiking Neural Networks on Neuromorphic Hardware 11
Funding/Acknowledgment
The research leading to these results has received funding from the EuropeanUnion Seventh Framework Programme (FP7) under grant agreement no 604102and the EU’s Horizon 2020 research and innovation programme under grantagreements No 720270 and 785907 (Human Brian Project, HBP). It has beenfurther supported by the Cluster of Excellence Cognitive Interaction Technology“CITEC” (EXC 277) at Bielefeld University, which is funded by the GermanResearch Foundation (DFG). Furthermore, we thank the Electronic Vision(s)group from Heidelberg University and Advanced Processor Technologies Re-search Group from Manchester University for access to their hardware systemsand continuous support and James Knight from the University of Sussex forsupport regarding our GeNN implementation.
References
1. Akopyan, F., et al.: TrueNorth: Design and Tool Flow of a 65 mW 1 MillionNeuron Programmable Neurosynaptic Chip. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (10), 1537–1557 (10 2015).https://doi.org/10.1109/TCAD.2015.24743962. van Albada, et al.: Performance Comparison of the Digital Neuromorphic Hard-ware SpiNNaker and the Neural Network Simulation Software NEST for aFull-Scale Cortical Microcircuit Model. Frontiers in Neuroscience (5 2018).https://doi.org/10.3389/fnins.2018.002913. Cao, Y., et al.: Spiking Deep Convolutional Neural Networks for Energy-EfficientObject Recognition. International Journal of Computer Vision (1), 54–66 (52015). https://doi.org/10.1007/s11263-014-0788-34. Davies, M.: Benchmarks for progress in neuromorphic computing. Nature MachineIntelligence (9), 386–388 (2019)5. Davies, M., et al.: Loihi: A Neuromorphic Manycore Proces-sor with On-Chip Learning. IEEE Micro (1), 82–99 (2018).https://doi.org/10.1109/MM.2018.1121303596. Davison, A.P.: PyNN: a common interface for neuronal networksimulators. Frontiers in Neuroinformatics (January), 11 (2008).https://doi.org/10.3389/neuro.11.011.20087. Diehll, P.U., et al.: Fast-Classifying, High-Accuracy Spiking Deep Net-works Through Weight and Threshold Balancing. Proceedings of the In-ternational Joint Conference on Neural Networks (2015).https://doi.org/10.1109/IJCNN.2015.72806968. Furber, S.B., et al.: Overview of the SpiNNaker system architec-ture. IEEE Transactions on Computers (12), 2454–2467 (12 2013).https://doi.org/10.1109/TC.2012.1429. Gewaltig, M.O., Diesmann, M.: { NEST } ( { NEural } Simulation Tool). Scholarpedia (4), 1430 (2007)10. G¨oltz, J., et al.: Fast and deep neuromorphic learning with time-to-first-spike cod-ing (12 2019)11. Homburg, J.D., et al.: Constraint Exploration of Convolutional Network Archi-tectures with Neuroevolution. In: Advances in Computational Intelligence. pp.2 C. Ostrau et al.735–746. Springer International Publishing (2019). https://doi.org/10.1007/978-3-030-20518-8 6112. Jordan, J., et al.: NEST 2.18.0 (6 2019).https://doi.org/10.5281/ZENODO.260542213. Knight, J.C., Nowotny, T.: GPUs Outperform Current HPC and Neuromor-phic Solutions in Terms of Speed and Energy When Simulating a Highly-Connected Cortical Model. Frontiers in Neuroscience (December), 1–19 (2018).https://doi.org/10.3389/fnins.2018.0094114. Maass, W.: Networks of spiking neurons: The third generation of neu-ral network models. Neural Networks (9), 1659–1671 (12 1997).https://doi.org/10.1016/S0893-6080(97)00011-715. Moradi, S., et al.: A scalable multicore architecture with heterogeneous mem-ory structures for Dynamic Neuromorphic Asynchronous Processors (DYNAPs).IEEE transactions on biomedical circuits and systems (1), 106–122 (2018).https://doi.org/10.1109/TBCAS.2017.275970016. Neckar, A., et al.: Braindrop: A Mixed-Signal Neuromorphic Architecture With aDynamical Systems-Based Programming Model. Proceedings of the IEEE (1),144–164 (1 2019). https://doi.org/10.1109/JPROC.2018.288143217. Ostrau, C., et al.: Comparing Neuromorphic Systems by Solving Sudoku Problems.In: Conference Proceedings: 2019 International Conference on High PerformanceComputing & Simulation (HPCS). IEEE (2019)18. Ostrau, C., et al.: Benchmarking of Neuromorphic Hardware Systems. In: Neuro-inspired Computational Elements Workshop (NICE 20), March 1720, 2020, Hei-delberg, Germany. Association for Computing Machinery (ACM) (2020)19. Petrovici, M.A., et al.: Characterization and compensation of network-level anoma-lies in mixed-signal neuromorphic modeling platforms. PLoS ONE (10) (2014).https://doi.org/10.1371/journal.pone.010859020. Pfeil, T., et al.: Six networks on a universal neuromorphic com-puting substrate. Frontiers in Neuroscience (7 FEB), 11 (2013).https://doi.org/10.3389/fnins.2013.0001121. Rhodes, O., et al.: Real-time cortical simulation on neuromorphic hard-ware. Philosophical Transactions of the Royal Society A: Mathemati-cal, Physical and Engineering Sciences (2164), 20190160 (2 2020).https://doi.org/10.1098/rsta.2019.016022. Schemmel, J., et al.: A wafer-scale neuromorphic hardware systemfor large-scale neural modeling. In: Proceedings of 2010 IEEE Inter-national Symposium on Circuits and Systems. pp. 1947–1950 (2010).https://doi.org/10.1109/ISCAS.2010.553697023. Schmitt, S., et al.: Neuromorphic hardware in the loop: Training a deep spik-ing network on the BrainScaleS wafer-scale system. In: 2017 InternationalJoint Conference on Neural Networks (IJCNN). pp. 2227–2234. IEEE (5 2017).https://doi.org/10.1109/IJCNN.2017.796612524. St¨ockel, A., et al.: Binary associative memories as a benchmark for spiking neu-romorphic hardware. Frontiers in computational neuroscience (August), 71(2017). https://doi.org/10.3389/fncom.2017.0007125. Yavuz, E., et al.: GeNN: a code generation framework for acceler-ated brain simulations. Scientific reports6