Ultra-Low-Power FDSOI Neural Circuits for Extreme-Edge Neuromorphic Intelligence
Arianna Rubino, Can Livanelioglu, Ning Qiao, Melika Payvand, Giacomo Indiveri
11 Ultra-Low-Power FDSOI Neural Circuits forExtreme-Edge Neuromorphic Intelligence
Arianna Rubino, Can Livanelioglu, Ning Qiao, Melika Payvand
Member, IEEE , and GiacomoIndiveri
Senior Member, IEEE
Abstract —Recent years have seen an increasing interest inthe development of artificial intelligence circuits and systems foredge computing applications. In-memory computing mixed-signalneuromorphic architectures provide promising ultra-low-powersolutions for edge-computing sensory-processing applications,thanks to their ability to emulate spiking neural networks in real-time. The fine-grain parallelism offered by this approach allowssuch neural circuits to process the sensory data efficiently byadapting their dynamics to the ones of the sensed signals, withouthaving to resort to the time-multiplexed computing paradigm ofvon Neumann architectures. To reduce power consumption evenfurther, we present a set of mixed-signal analog/digital circuitsthat exploit the features of advanced Fully-Depleted Silicon onInsulator (FDSOI) integration processes. Specifically, we explorethe options of advanced FDSOI technologies to address analogdesign issues and optimize the design of the synapse integratorand of the adaptive neuron circuits accordingly. We presentcircuit simulation results and demonstrate the circuit’s abilityto produce biologically plausible neural dynamics with compactdesigns, optimized for the realization of large-scale spiking neuralnetworks in neuromorphic processors.
Index Terms —Edge computing, silicon neurons, FDSOI, ultra-low-power, slow synaptic dynamics, IoT, real-time, analog circuit
I. I
NTRODUCTION
A technological revolution is in the making where more andmore Internet of Things (IoT) and edge-computing devices arebeing produced to sense and process signals, for example inenvironmental or health monitoring applications, and extractrelevant information locally, without resorting to cloud com-puting or transferring large amounts of data to remote datacenters. This poses a serious challenge in terms of memory andpower consumption requirements for IoT systems, especiallywhen they are expected to operate autonomously in a compactpackage directly on the sensed signals (i.e., in “extreme-edge” computing application scenarios). Due to the limitationsof Dennard scaling law [1] and the von Neumann memory
This paper is supported in part by the European Union’s Horizon 2020ERC project NeuroAgents (Grant No. 724295), by European Union’s Horizon2020 research and innovation programme under grant agreement No 871737(project BeFerroSynaptic), and by Toshiba Corporation. This research workwas also partially supported by H2020 MeM-Scales project (871371) andby the ECSEL Joint Undertaking (JU) under grant agreement No 826655.The JU receives support from the European Union’s Horizon 2020 researchand innovation programme and Belgium, France, Germany, Netherlands,Switzerland.A. Rubino, N. Qiao, M. Payvand, and G. Indiveri are with the University ofZurich and ETH Zurich, Institute of Neuroinformatics, Zurich, Switzerland.(e-mail: [email protected], [email protected], [email protected], [email protected]). C. Livanelioglu is with ETH Zurich, Zurich, Switzerland.(e-mail: [email protected]). bottleneck problems [2], [3], a disruptive change is needed inthe development of new memory and computing technologiesto be able to sustain these processing requirements under tightpower and size constraints.A promising computational paradigm that can support theultra-low-power implementation of “extreme-edge” computingprocessing tasks is that of Spiking Neural Networks (SNNs)and attractor dynamics [4]–[6]. In particular, it has been shownthat recurrent SNNs provide a valuable algorithmic basis forefficient processing of temporal signals: the rich dynamics ofthese networks are instrumental in minimizing the amount ofmemory resources required to process, recognize, and classifylong temporal sequences of data [7]. In addition, recent studiessuggest that longer time constants in the SNN synapse andneuron models are very beneficial in lengthening the so-called “fading memory” of the recurrent network [8]. Thebest-suited approach to implement such networks in hardwarewhich minimizes power consumption and area is that of usingmixed-signal neuromorphic circuits [9], [10]. By exploiting thetemporal properties of such circuits to adapt them to the tem-poral dynamics of the signals being processed, it is possibleto implement an optimal “matched filter” approach that mini-mizes power consumption and maximizes the Signal to NoiseRatio (SNR) [11]. This approach forgoes the need for storingthe data and the state of the processing elements, since theyoperate in real-time directly on the signals being acquired bythe sensor. By combining the adaptive analog signal processingstrategies of these neuromorphic circuits with digital event-based asynchronous communication schemes, it is possibleto build large-scale multi-core neuromorphic processors thatcombine the best of both (analog and digital) world for low-power signal processing, computation, and communication.These processors typically operate with sub-mWatt power-consumption figures and support the emulation of a wide rangeof SNN models, for solving artificial intelligence tasks directlyon the sensory signals, as they are acquired (e.g., see [12],[13]). We refer to the combination of this technology withthis approach as “ extreme-edge neuromorphic intelligence ”.The key enabling features of neuromorphic intelligencecircuits are twofold: (i) They operate in continuous real-timeon sensory signals, dissipating power only when the databecomes available and (ii) their operation speed is adapted(typically slowed down) to the time-scale of the signals beingprocessed. For natural signals such as speech, human gestures,bio-signals, and a wide range of environmental signals, thesetime constants are the biologically plausible ones that rangefrom fractions of milli-seconds to seconds. To optimize edge- a r X i v : . [ c s . ET ] J u l computing applications that operate on these types of inputsignals, a crucial precondition is to be able to support theprocessing of data on these different biological time scales.It is challenging to realize a similar range of timescalesusing conventional technologies, as they require very largecapacitors and/or big digital memory storage blocks whichlimits the scalability of these circuits. As the technology nodesmove toward deep sub-micron processes, the increased leakagecurrent limits the time constants and also poses a challengein terms of the static power consumption. Moreover, as thetechnology node scales down and the transistor’s channellength decreases, its parameter variations (e.g. the thresholdvoltage) increase, and device mismatch increases even further.In this paper, we present sub-threshold neuron and synapsecircuits that have been designed to implement large-scalemulti-neuron multi-core neuromorphic computing architec-tures using a 22 nm Fully-Depleted Silicon on Insulator(FDSOI) process. In Section II we show how it is possibleto implement bio-physically complex neural and synapticdynamics using ultra-low-power analog circuits in advancedscaled processes, by analyzing the features of the 22 nmFDSOI technology and addressing the analog design issuesthat arise from the advanced scaling. In particular, we exploitthe properties of the FDSOI technology to design a synapsecircuit that can reach 6 sec-long time constant, operatingreliably with femto-Ampere currents. We also optimize thedesign of the FDSOI silicon neuron circuit recently proposedin [14] to further reduce the device mismatch effects.In Section III we present circuit simulation results high-lighting how the synapse and neuron dynamics change as afunction of their bias parameters. We also characterize thepower consumption figures as a function of the neuron averagefiring rate, and quantify the effect of device mismatch withMonte Carlo analysis simulations.II. M ETHODS
A. FDSOI process advantages
Neuromorphic analog circuits typically use transistors op-erated in the sub-threshold or weak-inversion regime [15]–[17], using currents that range from fractions of pico-Amperesto hundreds of nano-Amperes. Indeed, to emulate biologi-cally plausible dynamics, with time constants of the orderof tens of milli-seconds, while using small capacitors (e.g.,of the order of pico-Farads), it is necessary to limit thecurrents to pico-Ampere amplitudes. Furthermore, to im-plement circuits with even longer time-scales, for exampleto emulate homeostatic plasticity phenomena that last mul-tiple seconds or tens of seconds, it would be necessaryto reduce the currents even below femto-Ampere ampli-tudes. However, the non-ideal transistor effects of advancedComplementary Metal-Oxide-Semiconductor (CMOS) tech-nology nodes (e.g., drain-induced-barrier-lowering (DIBL),band-to-band tunnelling (BTB), gate-induced-drain-leakage(GIDL), or random dopant fluctuation (RDF)) produce leakagecurrents that are well above the single pico-Ampere digits andseverely limit the functionality of subthreshold neuromorphiccircuits. To design neuromorphic circuits in advanced technologynodes, one option is to resort to above-threshold circuits, eitheraccelerating time constants by a factor of × V th can be further reduced with forward back-gate biasing. LVT are conventional-well devices which haveopposite doping types between channel and well and the V th can be further increased with reverse back-gate biasing. B. Key sub-circuits in 22 nm FD-SOI processes
In the neuron and synapse designs presented in this section,we use the 1.2 V I/O Low Threshold Transistors (LVT) at a V dd of 0.8 V. These transistors are thick gate oxide, conventionalwell devices with high enough V th which offer ultra-lowleakage baseline, essential for achieving ultra-low leakagecurrent levels. Moreover, due to their large utilizable range ofchannel lengths and minimal channel doping, these transistorsoffer less device mismatch.In both designs the capacitances are implemented usingAlternate Polarity Metal On Metal (APMOM) structures. Thedensity of these devices depends on the value and on thenumber of used metal layers: larger capacitances can be imple-mented with more metal layers to attain a higher capacitancedensity. However, as these devices exhibit parasitic resistanceeffects that scale with their area, there are severe limitationsto the maximum capacitance that can be attained.
1) The neuromorphic synapse circuit:
Here we present anFDSOI synapse circuit based on the DPI [22], [23] which
Fig. 1: Schematic of the DPI synapse circuit. Input spikes are applied to the V pre node. The output EPSC I syn has an amplitudeproportional to V w and decays exponentially with a time constant τ syn directly proportional to C syn and inversely proportionalto I τ .models the synaptic response behavior as a first order linearsystem. The schematic diagram of the circuit is shown inFig. 1. Input spikes applied to the V pre node get integratedinto an EPSC which obeys the following dynamics: τ syn dI syn dt + I syn = I gain I τ I w (1)where I syn is the synapse output current, I gain is a referencecurrent, I τ is a current equivalent to I gain / , and I w is asubthreshold current set by V w . The synapse time constant τ syn is defined as: τ syn := C syn U T κ I τ where C syn is the DPI capacitor, U T is the thermal voltage KT /q , and κ is the subthreshold slope factor [17]. To increasethe circuit time constant it is therefore necessary to eitherincrease capacitor sizes or to reduce I τ currents. Increasingcapacitor sizes however is problematic, because of large arearequirements and area-dependent leakage drawbacks (e.g.,with APMOM structures). So the most viable solution is tominimize the I τ current. By exploiting the features of FDSOItechnology and analog circuit design techniques, the synapsecircuit presented on Fig. 1 can reliably produce I τ currents ofthe order of femto-Amperes. This allows the circuit to reach τ syn values of up to 6 s with compact synaptic C syn capacitorsthat have capacitance values below 1 pF (specifically, 821 fFwith an APMOM structure of 12 × µ m in our design).Furthermore, by setting the DPI gain term to a constantratio ( I gain /I τ = 4 ) we ensure that changes in the synapsetime constant do not affect the maximum synapse currentamplitude.The transistors in the synapse circuit are operated with apower-supply voltage V dd = ± M S ) whosebody contact was set to gnd for achieving higher synapticefficacy. To improve the synaptic efficacy further, the NFETsof the differential pair ( M S and M S ) are designed withhigh WL ratio so that they accommodate a lower V gs drop andprovide enough V ds headroom for M S and M S to remainin saturation while V syn discharges to lower voltages.To achieve even lower leakage currents and higher outputimpedance for the I τ and I gain current mirrors, and allowthem to operate correctly with sub-pico-Ampere currents, weadopted the self-cascoding technique proposed in [24]. Thetransistor self-cascoded configuration is denoted in the figureby “x2” symbol.The operation of the DPI synapse circuit is controlled by theinput V pre , which is a pulse signal representing the incomingspikes from the previous synaptic neurons. When V pre is at gnd there is no current flowing in the bottom branch ( M S , M S and M S ) and I τ keeps V syn charged at V dd , switchingOFF M S . When V pre is at V dd , V syn discharges with a speedset by I w - I τ and as M S starts switching ON, the DPIsynapse circuit injects I syn . When V pre returns to gnd , V syn charges back to V dd with a rate set by I τ , switching M S backto the OFF state.
2) The silicon neuron circuit:
It has been recently arguedthat SNNs can accomplish remarkable learning and inferenceperformance figures, if they are endowed with complex dy-namics which comprise multiple and diverse time-scales [8].To support such networks, we propose the use of the Adaptive-Exponential Integrate and Fire (AdExp-I&F) silicon neuroncircuit [10], [14], [25]–[27]. The AdExp-I&F neuron modelhas been shown to be able to reproduce a wide range of spiking behaviors and explain a wide set of experimentalmeasurements from pyramidal neurons [28], [29]. Similar tothe Izhikevich neuron model [30], it is a two-variable modelwith a “fast” variable that describes the dynamics of themembrane potential and includes an activation term with anexponential voltage dependence, and a “slow” variable thatdescribes the spike-frequency adaptation mechanism. This isa negative-feedback mechanism which decreases the effectof the input current to the neuron with every output spike,therefore acting as a high-pass filter which reduces the neuronfiring rate in response to instantaneous increases in the input.The equations that describe the original computationalmodel [28] are the following:
C dVdt = − g L ( V − E L ) + g L ∆ T · e V − VT ∆ T − w + I (2) τ w dwdt = a ( V − E L ) − w (3)where V represents the neuron membrane potential, C itsmembrane capacitance, g L the leak conductance, E L the rest-ing potential, I the neuron’s input current, and w is the slowvariable that represents the after-hyperpolarizing current ofbiological neurons responsible for their spike-frequency adap-tation behavior [31]. The term ∆ T represents the exponentialslope factor, V T the neuron’s spiking threshold potential, a the adaptation weight, and τ w the adaptation time constant.At every spike, the neuron is reset to the resting potential andthe adaptation variable is increased by a .The AdExp-I&F FDSOI neuron we propose is depicted inFig. 2. It is a current-mode circuit, in which the currentsrepresent the state variables. Therefore the V and w variablesof the computational model described by Eqs.(2) and (3)are represented in the circuit by the currents I mem and I ahp respectively. By adopting the same translinear-circuitanalysis techniques used for the DPI synapse circuit, anddescribed in [10], and using very low leakage currents suchthat I leak (cid:28) I in , it is possible to express the circuit dynamicsas: C mem ddt I mem = − g L I mem + g L f ( I mem ) − g L I ahp + g L I in (4) τ ahp ddt I ahp = I a − I ahp (5)where g L and τ ahp are defined as: g L := κ I leak U T τ ahp := C ahp U T κ I τ ahp , and where I a represents the adaptation weight for theslow variable, and f ( I mem ) is a current produced by thepositive-feedback block transistors ( M N – M N ) which hasbeen shown to well fit a positive-exponent exponential func-tion [32].The AdExp-I&F FDSOI circuit schematic can be subdividedinto different functional blocks: An input DPI ( M L – M L )models the neuron’s leak conductance (LEAK). A current-based positive feedback module ( M N – M N ) models the TABLE I: Capacitance values and sizes used in the neurondesign C mem C ahp C ref C pex C cc Value 821 fF 1 pF 102 fF 136 fF 116 fFWidth 12 µ m 14 µ m 5.8 µ m 6 µ m 6 µ mLength 12 µ m 14 µ m 5.5 µ m 7 µ m 6 µ m neuron’s Sodium (Na+) activation and inactivation channels,and is coupled to a low-power current comparator (CC) block( M C – M C ) which triggers a spike as soon as the membranecurrent I mem exceeds the spiking threshold parameter I thr . Aspike reset circuit with refractory period functionality ( M K – M K ) models the neuron’s Potassium (K+) channels. Anegative feedback LPF circuit implemented with an additionalinstance of a DPI ( M A – M A ) (AHP) emulates Calcium-dependent after-hyperpolarization Potassium currents observedin real neurons to produce the spike-frequency adaptationmechanism. This circuit is driven each time the neuron pro-duces an output spike event, which is conveyed to a pulseextender circuit (Fig. 3) to lengthen the duration of thespike-event and ensure proper sub-threshold operation of theDPI. Finally, an asynchronous digital handshaking (HS) block( M HS – M HS ) implements the interface to Address-EventRepresentation (AER) circuits for transmitting the spikes asaddress-events to AER routers and destinations. This blockgenerates the Req and
Ack signals used to implement a four-phase handshaking cycle with the destination AER circuits: atrest, when the neuron current is below the spiking threshold,
Req and
Ack are both set to gnd . As the I mem crosses thespiking threshold, provided Ack is still at gnd , Req is set to V dd . Once the AER receiver consumes the event request andsets Ack to V dd the neuron resets and Req is pulled back to gnd . As the AER receiver senses this change, it should lowerthe
Ack signal, and the cycle can repeat.To minimize leakage currents we reduced the Early effectof critical transistors by using a pseudo-cascode split-transistorsub-threshold technique [33] (see transistors M L and M A ofFig. 2). As for the DPI synapse schematic of Fig. 1, the “ × V leak ), its spike-frequency adaptation properties ( V a and V τ _ ahp ), its refractory period ( V ref ), and its spiking threshold( V thr ). As the DPI circuits of the LEAK and AHP blocks havebeen configured to have a gain term I gain /I τ = 1 (similar tohow the gain term was set to in the synapse circuit), the V leak and V τ _ ahp signals can be tuned by modifying the I gain and I gain _ ahp currents respectively.The spiking threshold parameter V thr controls the cur-rent comparator block ( M C – M C ). This is a novel circuit,modified from the one originally proposed in [34] to reduce Fig. 2: AdExp-I&F neuron circuit schematic: In grey the input DPI LPF, in pink the positive-feedback and the currentcomparator, in yellow the reset block, in light blue the handshake block and in green the spike-frequency adaptation block.Adapted from [14].the neuron’s static and dynamic power consumption. It hasbeen introduced to decouple the slow and gradual changesof the neuron’s membrane potential dynamics (representedby the I mem current) from the digital switching mechanismrequired to generate a spike. This is necessary to minimizethe switching time of the digital circuits, during which theycan dissipate large amounts of power. The V inCC voltage ofthe comparator circuit is set by the competition between thecurrents representing the spiking threshold, I thr , and I mem .If I mem is smaller than I thr , this node is actively clampedto V dd . However as I mem approaches I thr , V inCC dropssharply to produce a spike event with very low dynamic powerconsumption figures. The transistor M C was introduced inthe CC block to reduce the neuron’s power consumptionin its resting (OFF) state: when I in =0, the node V inCC isdriven to V dd by M C which turns on a discharging path to gnd via M C . As both M C and M C are conducting, thereexists a direct path between V dd and gnd which undesirablyadds to the static power consumption figure. The additionof M C to this circuit breaks the discharging path to gnd ,further reducing the neuron static power consumption. The C CC capacitance ensures that at each input current thresholdcrossing corresponds only one spike. Fig. 3: Pulse extender circuit schematic. Adapted from [14].The pulse extender circuit depicted in Fig. 3 is used toextend the spike-event pulse created by the neuron and to drivethe spike-frequency adaptation circuit ( M A – M A ). When aspike is produced and Req goes to V dd , the transistor M E discharges the node V pex to gnd . As a consequence, the node spike _ extB is discharged to gnd switching ON M A of Fig. 2.The length of the spike _ extB pulse is set by the V B biasvoltage and the capacitance C pex . The higher the value of V B , the slower V pex is charged back to V dd and hence thelarger the extension. III. R
ESULTS
A. Synaptic circuit simulation
The simulations presented in this section demonstrate howthe DPI synapse presented in Section II-B1 can achieve verylarge time constants and high synaptic efficacy. The resultsdemonstrating the integration and steady-state profiles of thesynapse EPSC ( I syn ) are shown in Fig. 4a and 4c respectively.To assess the corresponding synaptic time constants ( τ syn ), weplot the linear fits to the natural logarithm of the normalizedEPSC in Fig. 4b and 4d.To measure the circuit’s dynamic range we stimulate itwith a pulse train of 50 Hz rate, with each pulse lasting100 nsec; and we sweep I τ to cover a wide spectrum of timeconstants extending from 50 msec to 6 sec. To demonstrate theintegration property of the synapse at lower I τ values (i.e.,larger time constants), we apply the pulse train for 1 sec, adjustthe weight I w accordingly to obtain a peak integrated responseof 1 nA, and measure the decay time of I syn . Moreover, toobserve the different steady-state response behavior, we extendthe pulse train stimulation duration to 5 sec, fix I w to 500 nAand use smaller time constant values which are comparablewith the input pulse train inter-spike interval.Figure 4a shows the synapse response in integration modewith the time constants in the range 250 msec–6 sec. Here,the inter-spike interval of 20 msec is much smaller than thesynaptic time constants, and thus the EPSC charges up to1 nA for all I τ values. However, to obtain equal peak EPSCmagnitudes we had to make small adjustments of I w for thevalues of I τ between 5 and 50 fA, and larger adjustmentsfor I τ =1 fA and I τ =100 fA. As the value of I τ decreases tofA values, the APMOM capacitor parasitic effects becomenon-negligible: specifically, the capacitor’s leak increases theeffective value of I τ . For ideal values of I τ =1 fA, the effectivesynaptic efficacy term I gain I τ is in practice much less than thenominal value of 4. To compensate for this effect it is thereforenecessary to increase the value of I w (e.g., see solid line inFig. 4a).Figure 4c shows how the synapse reaches a steady-statewith shorter time constants and a longer stimulus duration. Inthis mode of operation, the discharge of the synapse duringthe inter-spike interval balances out with the charge inducedby the spikes. Hence, the steady-state value scales with thetime constant. In addition, the EPSC at steady-state featuresfluctuations around its peak value due to the ongoing rapidsynaptic charging-discharging process.The fits to the synaptic current data illustrated in Fig. 4band 4d verify that the synapse behaves as a first order systemas Eq. (1) suggests. Table II compares the theoretical time con-stants to the values obtained by linear fitting. The table showsthat the fitting results are in very good agreement with the the-ory for fast synaptic dynamics while the difference increasesas the synapse becomes slower. This is due to the APMOMcapacitor leakage building on I τ which limits the maximumtime constant of the circuit. This limitation is particularlysignificant for I τ =1 fA, where the capacitor leakage dominatesthe synaptic discharging process. As a consequence, the timeconstant saturates at τ syn =5.81 sec above which the circuit TABLE II: Comparison between the measured and theoreticaltime constants τ syn of the DPI synapse for C syn =821 fF, κ =0.75, and U T =25 mV I τ Theoretical Linear Fit I τ Theoretical Linear Fit1 fA 27.37 sec 5.81 sec 100 fA 274 msec 270 msec5 fA 5.47 sec 3.42 sec 200 fA 137 msec 138 msec10 fA 2.74 sec 2.21 sec 300 fA 91 msec 93 msec20 fA 1.37 sec 1.26 sec 400 fA 68 msec 70 msec50 fA 547 msec 535 msec 500 fA 55 msec 56 msec cannot extend. Based on the results, the capacitor leakage isfound to constitute a 3–4 fA of peak baseline current whichis much higher than the cumulative leakage of all transistors.Although the reduction in the transistor leakage has increasedthe available time constant range, the capacitive leakage wouldneed to be reduced to sub-femto-Ampere regime as well, inorder to increase the time constants further up to 30 sec asthe theory suggests. Increasing the capacitance size is nota viable option, since the capacitor leakage scales up withits area as well. Moreover, this capacitive leakage is voltagedependent hence utilizing different synaptic weights or pulseconfigurations can result in slight deviations in the availabletime constant range.In general, the designed DPI synapse circuit with ultra-lowleakage capability can offer a wide dynamic range of timeconstants with sub-pico-Ampere I τ values and can generateEPSC on the order of nano-Amperes for 100 nsec of pulseduration. The ultra-low current operation of the circuit makestime constants up to several seconds achievable with capacitorsbelow 1 pF which reduces the layout area and enables denserintegration in addition to the benefit of reducing the powerconsumption of the overall circuit. B. Neuron circuit simulation
In this section we present the FDSOI neuron simulationresults, demonstrating examples of biologically plausible be-haviors, characterizing its power consumption properties, andquantifying the effects of device mismatch on its responseproperties. Figure 5 shows the response of the neuron to astep input current for different parameter settings: Fig. 5ashows the membrane current I mem for two different valuesof I ref . As the I ref increases, the refractory period is shorterand hence the neuron’s maximum spiking frequency increases.Figure 5b and 5c show the I mem for different values of I thr when keeping the same I in (Fig. 5b) and when changing I in (Fig. 5c) to obtain the same spiking frequency as I thr changes. As I thr increases the neuron is less facilitated tospike as it has to integrate more current to reach the threshold.Higher I thr leads to more time needed in the integrating phaseand therefore less frequent spikes, which gives lower spikingfrequency.Figure 6a and 6b show the neurons F-I curve for different I ref bias settings. The neurons average firing rate increaseslinearly with the input current, until it reaches a saturation levelthat depends on the refractory period setting (see Fig. 6a). Thesaturation frequency depends on the duration of the refractoryperiod, as the I ref increases the refractory period is shorter (a) (b)(c) (d) Fig. 4: Synapse I syn EPSC measurements and fits with the natural logarithm of the normalized EPSC for estimating its timeconstants τ syn : (a) & (b) for a 50 Hz stimulus applied for 1 sec with synaptic time constants in the range 250 msec–6 sec. (c)& (d) for a 50 Hz stimulus applied for 5 sec, with synaptic time constants in the range 50 msec–150 msec, causing the synapseto reach different steady state levels. (a) (b) (c) Fig. 5: FDSOI neuron biologically plausible behaviour: (a) Membrane current I mem for two different values of I ref , (b) I mem for two different values of I thr keeping the same I in , (c) I mem for three different values of I thr changing I in to obtain thesame spiking frequency.and hence the neuron’s maximum spiking frequency increases.For shorter refractory periods (Fig. 6b) the maximum spikingfrequency is linear with the input current for a larger range of I in and it reaches saturation only for very high values of I in .When I ref is 1 µ A, hence when the refractory period is veryshort (few hundreds of ns), the maximum spiking frequencydoes not saturate for the chosen I in range.Figure 6c shows the neuron spiking frequency versus inputcurrent (F-I curve), for different settings of the I gain / I leak biasratio. We modify the default ratio ( I gain / I leak =1) to highervalues. As expected, increase in the ratio results in the increaseof the neuron’s firing rate. Figure 7 demonstrates the spike-frequency adaptation be-havior, obtained by appropriately tuning the relevant parame-ters in the AHP block of Fig. 2 and measuring the neuron’sstep response to a constant injection current. At each spike the I ahp current increases, decreasing the total current charging C mem . When I ahp reaches the steady state, the neuron spikingfrequency remains constant to a lower value compared to theinitial one. C. Energy per spike
Once proven that the design is able to reproduce a biolog-ically plausible behavior, we evaluated whether it can imple- (a) (b) (c)
Fig. 6: Firing rate vs Input current (F-I) curve: (a) for lower values of I ref hence longer refractory period, (b) for higher valueof I ref hence shorter refractory period and (c) for different values of I gain / I leak ratio. Adapted from [14].Fig. 7: Spike-frequency adaptation: Membrane current andafter-hyperpolarization current trace over time. Adapted from[14].ment massively parallel large-scale neuromorphic processors.The energy per spike is equal to: EnergyF req · T ime = P ower · T imeF req · T ime = P owerF req (6)where
Energy , the total energy consumed, is the product oftotal power consumed (
P ower ) and simulation time (
T ime )and
F req is the maximum spiking frequency. As is shownin Fig. 8, the energy per spike for lower frequencies is inthe order of tens of pJ, while it decreases to 1 pJ for higherfrequencies. This is due to the fact that in lower frequenciesthe inverters spend more time in their high gain region withboth transistors conducting, since the V mem at their inputis charging much slower compared to the case with higherfrequencies.We compare the energy per spike of our proposed neuronwith previously published state-of-the-art neuromorphic pro-cessors in Table III.The neuron designed in this work consumes less energy perspike compared to a similar circuit [34] in a similar technology(28 nm FDSOI) at a biological plausible spiking frequency(30 Hz). Moreover, the circuits used in [34] and our workhave similar V dd and C mem , which we can consider as the Fig. 8: Energy per spike estimation: Energy per spike vs Firingrate.TABLE III: Energy per spike comparison with previous works Work [12] [19] [34]
This work
Techn. 180 nm 28 nm 28 nm 22 nmCMOS CMOS FDSOI FDSOIType Mixed Mixed Mixed Mixed V dd predominant capacitance for power consumption. Therefore,according to the scaling factor, the energy per spike of thecircuit proposed in [34] will be similar when scaled to the22 nm process. Hence the differences reported here can beexplained by the optimizations made at the circuit design level.The neuron circuit energy consumption at higher frequen-cies is compared with the Sigma-Delta neuron proposedin [35], which is one of the most recent mixed-signal siliconneuron circuit designs presented in the literature. Since theSigma-Delta neuron presented in [35] was optimized foroperation at higher frequencies in a range of 1 kHz to 10 MHz,we compare the energy per spike between these circuits inthese ranges: the neuron proposed in this work consumes1 [email protected] kHz, approximately one order of magnitude less thanthe Sigma-Delta neuron (10 pJ).
40 55 65 75 9585 100
Firing rate [Hz]
Number = 500Mean = 70.4618Std Dev = 9.26272N° of samples
35 45 50 60 70 9080 105 110405080110120
Fig. 9: Monte Carlo analysis result distribution of the neuron circuit. Adapted from [14].
D. Monte Carlo Analysis
To evaluate the sensitivity of the circuit to device mismatchwe ran a series of Monte Carlo simulations. We performed thisanalysis with 500 runs for this neuron circuit, with DC currentinjected in the LEAK block in Fig. 2, and with bias currents setto obtain a firing rate of approximately 70 Hz while switchingoff the spike-frequency adaptation circuit.The mean of the distribution obtained is centered around theexpected value ( ≈
70 Hz) and the standard deviation is equalto 9.26 (see Fig. 9). The variability of the neuron circuit isthus 13 %. Our analysis found that this variability is governedby the LEAK block and the first part of the CC, where thecomparison between I mem and I thr is made ( M C and M C ).In particular, in the LEAK block the transistors more affectedby process variation are M L and M L . The K+ block, inparticular M K , also shows sensitivity to device mismatch,but it is negligible compared to the other two blocks.IV. C ONCLUSION
We determined process and circuit parameters in order toimplement efficient (low power and slow dynamics) analogneuron circuits using an advanced scaled 22 nm FDSOI pro-cess. We optimized the design of the synapse and neuroncircuits for producing biologically plausible neural dynamics,with time constants matched to those of natural signals, suchas speech or bio-signals.The presented silicon synapse circuit can achieve timeconstants of up to 6 sec without having to increase the synapticcapacitance C syn over 1 pF. The neuron circuit presentedhas an energy per spike of tens of pJ for lower frequenciesand pJ for higher frequencies, which is considerably lowercompared to an analogous neuron design implemented in a180 nm CMOS process [12]. Furthermore, it consumes lesscompared to a more recent design [34] at biologically plausiblefrequencies and it consumes one order of magnitude less compared to the state-of-the-art neuron circuit [35] at higherfrequencies. We studied the mismatch sensitivity of the neuroncircuit by performing Monte Carlo simulations and identifiedthe parts of the circuit that are most critical to be optimizedfor variations, showing how the more sensitive sub-parts of thesilicon neuron circuit are the LEAK block and the first partof the CC block. In summary in this paper we demonstratehow it is possible to exploit the features of advanced 22 nmFDSOI processes to design complex analog circuits that canbe used to implement low-power neuromorphic processors foredge computing sensory-processing tasks and, more generally,“neuromorphic intelligence” applications.A CKNOWLEDGMENT
We are grateful to Mohammad Ali Sharif Shazileh, ManuNair, and Elisa Donati for fruitful discussions on themanuscript and circuit design.R
EFERENCES[1] B. Davari, R. Dennard, and G. Shahidi, “CMOS scaling for highperformance and low power – the next ten years,”
Proceedings of theIEEE , vol. 83, no. 4, pp. 595–606, April 1995.[2] J. Backus, “Can programming be liberated from the von Neumannstyle?: a functional style and its algebra of programs,”
Communicationsof the ACM , vol. 21, no. 8, pp. 613–641, 1978. [Online]. Available:http://doi.acm.org/10.1145/359576.359579[3] G. Indiveri and S.-C. Liu, “Memory and information processing inneuromorphic systems,”
Proceedings of the IEEE , vol. 103, no. 8, pp.1379–1397, 2015.[4] W. Maass and H. Markram, “On the computational power of circuitsof spiking neurons,”
Journal of computer and system sciences , vol. 69,no. 4, pp. 593–616, 2004.[5] D. Zambrano and S. M. Bohte, “Fast and efficient asynchronous neuralcomputation with adapting spiking neural networks,” arXiv e-prints ,2016.[6] E. O. Neftci, “Data and power efficient intelligence withneuromorphic learning machines,” iScience [7] M. Lukoševiˇcius and H. Jaeger, “Reservoir computing approaches torecurrent neural network training,” Computer Science Review , vol. 3,no. 3, pp. 127–149, 2009.[8] G. Bellec, D. Salaj, A. Subramoney, R. Legenstein, and W. Maass,“Long short-term memory and learning-to-learn in networks of spikingneurons,” in
Advances in Neural Information Processing Systems , 2018,pp. 787–797.[9] C. Mead, “Neuromorphic electronic systems,”
Proceedings of the IEEE ,vol. 78, no. 10, pp. 1629–36, 1990.[10] E. Chicca, F. Stefanini, C. Bartolozzi, and G. Indiveri, “Neuromorphicelectronic circuits for building autonomous cognitive systems,”
Proceed-ings of the IEEE , vol. 102, no. 9, pp. 1367–1388, 9 2014.[11] G. Indiveri and Y. Sandamirskaya, “The importance of space and timefor signal processing in neuromorphic agents,”
IEEE Signal ProcessingMagazine , vol. 36, no. 6, pp. 16–28, 2019.[12] S. Moradi, N. Qiao, F. Stefanini, and G. Indiveri, “A scalable multicorearchitecture with heterogeneous memory structures for dynamic neuro-morphic asynchronous processors (DYNAPs),”
Biomedical Circuits andSystems, IEEE Transactions on , vol. 12, no. 1, pp. 106–122, Feb. 2018.[13] F. Bauer, D. Muir, and G. Indiveri, “Real-time ultra-low power ECGanomaly detection using an event-driven neuromorphic processor,”
Biomedical Circuits and Systems, IEEE Transactions on , 2019, (inpress).[14] A. Rubino, M. Payvand, and G. Indiveri, “Ultra-low power silicon neu-ron circuit for extreme-edge neuromorphic intelligence,” in
InternationalConference on Electronics, Circuits, and Systems, (ICECS), 2019 , 112019, pp. 458–461.[15] C. Mead,
Analog VLSI and Neural Systems . Reading, MA: Addison-Wesley, 1989.[16] E. Vittoz, “Analog VLSI implementation of neural networks,” in
Hand-book of Neural Computation , E. Fiesler and R. Beale, Eds. Oxford andBristol: Oxford University Press and Institute of Physics Publishing,1996, ch. E1.3.[17] S.-C. Liu, J. Kramer, G. Indiveri, T. Delbruck, and R. Douglas,
AnalogVLSI:Circuits and Principles . MIT Press, 2002.[18] J. Schemmel, L. Kriener, P. Müller, and K. Meier, “An accelerated analogneuromorphic hardware system emulating NMDA- and calcium-basednon-linear dendrites,” in
Neural Networks (IJCNN), 2017 InternationalJoint Conference on . IEEE, 2017, pp. 2217–2226.[19] C. Mayr, J. Partzsch, M. Noack, S. Hänzsche, S. Scholze, S. Höppner,G. Ellguth, and R. S. 2, “A biological-realtime neuromorphic systemin 28 nm CMOS using low-leakage switched capacitor circuits,”
IEEETransactions on Biomedical circuits and systems , vol. 10, no. 1, pp. 243– 254, 2016.[20] F. Folowosele, R. Etienne-Cummings, and T. Hamilton, “A CMOSswitched capacitor implementation of the Mihalas-Niebur neuron,” in
Biomedical Circuits and Systems Conference, (BioCAS), 2009 . IEEE,Nov. 2009, pp. 105–108.[21] F. Folowosele, A. Harrison, A. Cassidy, A. Andreou, R. Etienne-Cummings, S. Mihalas, Niebur, and T. Hamilton, “A switched capacitorimplementation of the generalized linear integrate-and-fire neuron,”in
International Symposium on Circuits and Systems, (ISCAS), 2009 .IEEE, May 2009, pp. 2149–2152.[22] C. Bartolozzi, S. Mitra, and G. Indiveri, “An ultra low power current–mode filter for neuromorphic systems and biomedical signal processing,”in
Biomedical Circuits and Systems Conference, (BioCAS), 2006 . IEEE,2006, pp. 130–133.[23] C. Bartolozzi and G. Indiveri, “Synaptic dynamics in analog VLSI,”
Neural Computation , vol. 19, no. 10, pp. 2581–2603, Oct 2007.[24] V. Saxena and R. J. Baker, “Compensation of CMOS op-amps usingsplit-length transistors,” in
Circuits and Systems (MWSCAS), 2008 IEEE51st International Midwest Symposium on . IEEE, 2008, pp. 109–112.[25] G. Indiveri, “A low-power adaptive integrate-and-fire neuron circuit,”in
International Symposium on Circuits and Systems, (ISCAS), 2003 ,vol. IV. IEEE, May 2003, pp. 820–823.[26] P. Livi and G. Indiveri, “A current-mode conductance-based siliconneuron for address-event neuromorphic systems,” in
International Sym-posium on Circuits and Systems, (ISCAS), 2009 . IEEE, May 2009, pp.2898–2901.[27] G. Indiveri, B. Linares-Barranco, T. Hamilton, A. van Schaik,R. Etienne-Cummings, T. Delbruck, S.-C. Liu, P. Dudek,P. Häfliger, S. Renaud, J. Schemmel, G. Cauwenberghs, J. Arthur,K. Hynna, F. Folowosele, S. Saighi, T. Serrano-Gotarredona,J. Wijekoon, Y. Wang, and K. Boahen, “Neuromorphic siliconneuron circuits,”
Frontiers in Neuroscience
Journal ofneurophysiology , vol. 94, no. 5, pp. 3637–3642, 2005.[29] R. Naud, N. Marcille, C. Clopath, and W. Gerstner, “Firing patterns inthe adaptive exponential integrate-and-fire model,”
Biological Cybernet-ics , vol. 99, no. 4–5, pp. 335–347, November 2008.[30] E. Izhikevich, “Which model to use for cortical spiking neurons?”
IEEE Transactions on Neural Networks , vol. 15, no. 5, pp. 1063–1070,September 2004.[31] G. E. Ha and E. Cheong, “Spike frequency adaptation in neurons of thecentral nervous system,”
Experimental neurobiology , vol. 26, no. 4, pp.179–185, 2017.[32] G. Indiveri, F. Stefanini, and E. Chicca, “Spike-based learning with ageneralized integrate and fire silicon neuron,” in
International Sympo-sium on Circuits and Systems, (ISCAS), 2010 . Paris, France: IEEE,2010, pp. 1951–1954.[33] N. Qiao, C. Bartolozzi, and G. Indiveri, “An ultralow leakage synapticscaling homeostatic plasticity circuit with configurable time scales up to100 ks,”
IEEE Transactions on Biomedical Circuits and Systems , 2017.[34] N. Qiao and G. Indiveri, “Scaling mixed-signal neuromorphic processorsto 28nm FD-SOI technologies,” in
Biomedical Circuits and SystemsConference, (BioCAS), 2016 . IEEE, 2016, pp. 552–555.[35] M. V. Nair and G. Indiveri, “An ultra-low power sigma-delta neuroncircuit,” in , May 2019, pp. 1–5.
Arianna Rubino received her B.Sc degree in 2017from Politecnico di Milano, Italy in biomedicalengineering and her M.S degree in 2019 in biomedi-cal engineering with specialization in bioelectronicsfrom the Swiss Federal Institute of Technology inZurich, Switzerland. Since September 2019 she isworking toward the Ph.D. degree at the Instituteof Neuroinformatics, University of Zurich and ETHZurich, Zurich, Switzerland. Her research inter-ests include the design of ultra-low power mixed-signal circuits for neuromorphic edge computingand biomedical applications using advanced transistor processes and theimplementation of biologically plausible learning algorithms on-chip.
Can Livanelioglu received his B.Sc degree in Elec-trical and Electronics Engineering from Middle EastTechnical University, Ankara, Turkey in 2019, withspecialization in Electronics and Biomedical. He isnow pursuing his M.Sc. degree in Biomedical Engi-neering with specialization in bioelectronics at ETHZurich, Switzerland and working at the Institute ofNeuroinformatics, ETH Zurich and University ofZurich, Switzerland. His research interests includethe design of analog/digital integrated circuits, solidstate electronics, novel semiconductor device archi-tectures and neuromorphic structures. Ning Qiao received the bachelor’s degree in mi-croelectronics and solid-state electronics from Xi’anJiaotong University, Xi’an, China, in 2006, and thePh.D. degree in microelectronics from the Instituteof Semiconductors, Chinese Academy of Sciences,China, in 2012, with a focus on ultra-low-powerlow-noise mixed-signal circuits in SOI process. Hejoined the Institute of Neuroinformatics, Universityof Zurich and ETH Zürich, Switzerland, as a Post-Doctoral Researcher, in 2012, where he is involvedin developing mixed-signal multicore neuromorphicVLSI circuits and systems. His current research interests include ultra-low-power subthreshold mixed-signal neuromorphic VLSI circuits and systems,parallel neuromorphic computing architectures, and fully asynchronous event-driven computing and communication circuits and systems.
Melika Payvand received her B.Sc degree in 2010from University of Tehran, Iran in electrical engi-neering and her M.S. and Ph.D. degree in electricaland computer engineering from University of Cali-fornia Santa Barbara in 2012 and 2016 respectively.Currently, she is a post-doctorate researcher at theInstitute of Neuroinformatics, University of Zurichand ETH Zurich. Her research activities and interestis in exploiting the physics of the computationalsubstrate for real-time sensory processing. Specifi-cally, she is interested in exploiting the physics ofthe computational substrate in event-based neuromorphic chips to enable lowpower and highly dense solutions for wearable and IoT applications.