CalciumGAN: A Generative Adversarial Network Model for Synthesising Realistic Calcium Imaging Data of Neuronal Populations
Bryan M. Li, Theoklitos Amvrosiadis, Nathalie Rochefort, Arno Onken
CCalciumGAN: A Generative Adversarial NetworkModel for Synthesising Realistic Calcium ImagingData of Neuronal Populations
Bryan M. Li [email protected] Theoklitos Amvrosiadis [email protected] Nathalie Rochefort , [email protected] Arno Onken [email protected] School of Informatics, University of Edinburgh Centre for Discovery Brain Sciences, University of Edinburgh Simons Initiative for the Developing Brain, University of Edinburgh
Abstract
Calcium imaging has become a powerful and popular technique to monitor theactivity of large populations of neurons in vivo . However, for ethical considerationsand despite recent technical developments, recordings are still constrained to alimited number of trials and animals. This limits the amount of data available fromindividual experiments and hinders the development of analysis techniques andmodels for more realistic size of neuronal populations. The ability to artificiallysynthesize realistic neuronal calcium signals could greatly alleviate this problemby scaling up the number of trials. Here we propose a Generative AdversarialNetwork (GAN) model to generate realistic calcium signals as seen in neuronalsomata with calcium imaging. To this end, we adapt the WaveGAN architectureand train it with the Wasserstein distance. We test the model on artificial datawith known ground-truth and show that the distribution of the generated signalsclosely resembles the underlying data distribution. Then, we train the model onreal calcium signals recorded from the primary visual cortex of behaving miceand confirm that the deconvolved spike trains match the statistics of the recordeddata. Together, these results demonstrate that our model can successfully generaterealistic calcium imaging data, thereby providing the means to augment existingdatasets of neuronal activity for enhanced data exploration and modeling. Recordings of neuronal activities from behaving animals are essential for the study of informationprocessing in the brain. With the advancement of neural recording techniques, such as electrophysi-ological recordings and calcium imaging, it has become increasingly easier to obtain high qualityneuronal activity data in vivo . However, due to ethical considerations, the acquired datasets are oftenlimited by the number of trials or the duration of each trial on a live animal. This poses a problem forassessing analysis techniques that take into account higher-order correlations [4, 24–26]. Even forlinear decoders, the number of trials can be more important for determining coding accuracy than thenumber of neurons [27]. Code available at github.com/bryanlimy/calciumganPreprint. Under review. a r X i v : . [ q - b i o . N C ] S e p enerative models of neuronal activity hold the promise of alleviating the above problem by enablingthe synthesis of an unlimited number of realistic samples for assessing advanced analysis methods.Recently, the use of deep generative models on neuronal population spike train data has becomeincreasingly popular. Latent Factor Analysis via Dynamical Systems (LFADS, Pandarinath et al. [20])uses the Variational Autoencoders framework to learn the population dynamics in latent representation(using recurrent neural networks) and extract ’denoised’ single-trial firing rates from neural spikingdata. Spike-GAN [16] demonstrated that GAN can model neural spikes that accurately matchthe statistics of real recorded spiking behaviour from a small number of neurons. Moreover, thediscriminator in Spike-GAN is able to learn to detect which population activity pattern is the relevantfeature, and this can provide insights into how a population of neurons encodes information. Rameshet al. [21] trained a conditional GAN [15], conditioned on the stimulus, to generate multivariate binaryspike trains. They fitted the generative model with recorded data in the V1 area of macaque visualcortex, and GAN generated spike trains were able capture the firing rate and pairwise correlationstatistics better than the dichotomized Gaussian model and a deep supervised model.All of the aforementioned generative models operate on population spike trains. Spike trains arediscrete in nature meaning that they cannot be subject to any continuous increment or decrement.Hence it remains a difficult task to optimize deep generative models for discrete data with back-propagation, which is key for training deep neural networks.Population spike trains can be obtained from different recording techniques, each having advantagesand weaknesses. Electrophysiological recordings have high temporal resolution. However, thismethod is not without flaws [9]. For instance, a single microelectrode can only detect activity fromfew neurons in close proximity, and extensive pre-processing is required to infer single-unit activityfrom a multi-unit signal. Disentangling circuit computations in neuronal populations of a largescale remains a difficult task, hence resulting in recordings with low spatial resolution but hightemporal resolution [22]. On the other hand, calcium imaging recordings have high spatial resolutionand low temporal resolution [28]. This technique, which assesses changes in intracellular calciumconcentration as a proxy for neuronal spiking activity, has become a powerful imaging technique tomonitor large neuronal populations activity in vivo . The continuous nature of calcium fluorescencesignals makes optimization via back-propagation a much more straightforward task as compared tospike-train data. Thus, calcium imaging datasets are more attractive candidates for training generativemodels.In this work, we explore the feasibility of using the Generative Adversarial Network (GAN) frameworkto synthesize calcium imaging data, as a method to scale-up or augment the amount of neuronalpopulation activity data. We validate the method on artificial data with known ground-truth andwe synthesize data mimicking real two-photon calcium (Ca ) imaging data as recorded from theprimary visual cortex of a behaving mouse [10, 19]. The original generative adversarial network framework, introduced by Goodfellow et al. [7], plays amin-max game where the generator G attempts to generate convincing samples from the latent space Z , and the discriminator D learns to distinguish between generated samples and real samples X .The GAN framework has shown promising results across various domains, such as image and audiogeneration, unsupervised translation and many more [12, 5, 29]. However, the originally proposedoptimization objective was difficult to train, and was prone to mode collapse. Instead of the originalobjective of minimizing the Jensen-Shannon divergence between the original data distribution andgenerated data distribution, Arjovsky et al. [1] proposed Wasserstein GAN (WGAN) which minimizesthe smoother Earth-Mover’s (1st Wasserstein) distance of the two data distributions. In WGAN,the weights of the discriminator (critic) were restricted within a predefined range (weight clipping)in order to enforce the 1-Lipschitz condition and to compute the Wasserstein distance. Gulrajaniet al. [8] further improved the objective function with gradient penalty (WGAN-GP), instead ofgradient clipping, in order to enforce the Lipschitz condition more effectively. In this work, we usethe WGAN-GP formulation of the loss function without the need of incorporating any information of2he neural activities into the objective function: L D = E z ∼ P Z [ D ( G ( z ))] − E x ∼ P X [ D ( x )] + λ E ˆ x ∼ P ˆ X [( (cid:107) ∇ ˆ x D (ˆ x ) (cid:107) − ] where λ denotes the gradient penalty coefficient and ˆ X are samples taken between the true data andgenerator’s data distribution.For the network architecture, we adapted the WaveGAN architecture by Donahue et al. [5], whichhas shown promising results in audio signal generation. The generator uses 1-dimensional transposeconvolution layers to up-sample the spatial dimension. Each transpose convolution layer wasfollowed by batch normalization and Leaky ReLU activation. We added a dense layer with sigmoidactivation as output layer in the generator. We also replaced Batch Normalization [11] with LayerNormalization [2] in order to make the operation compatible with the WGAN framework. Samplesgenerated using transpose convolution often exhibit the "checkerboard" artifacts described by Odenaet al. [18]. In the context of signal generation, the discrimination could exploit the periodic artifactspattern and learn a naive policy to reject generated samples. Donahue et al. [5] proposed the PhaseShuffle mechanism in the discriminator to address the aforementioned issue. The Phase Shift layerrandomly shifts the activated units after each convolution layer by − n to n in the time domain, inorder to distort the periodic pattern. Hence, the resulting samples constitute a more challengingtask for the discriminator. In our network, we incorporated the Phase Shift operation, as well asusing a kernel size that is divisible by the stride size, as suggested in Odena et al. [18]. This ledto a noticeable improvement in the generated samples. The discriminator is largely a mirror of thegenerator’s architecture, except for layer normalization which we do not use in the discriminator.Instead, we apply the Phase Shuffle operation after each convolution layer.To improve the model learning performance and stability, the calcium signals were scaled to the rangebetween 0 and 1 by normalizing with the maximum training set calcium signal. Correspondingly,we chose sigmoid activation in the output layer of the generator and then re-scale the signals to itsoriginal range before inferring their spike trains. The model architecture can be found in Table 1. We trained both the generator and discriminator with the WGAN framework, with 5 discriminatorupdate steps for each generator update step. We then used Adam optimizer [13] to optimize bothnetworks, with a learning rate of λ = 10 − , β = 0 . and β = 0 . . To speed up the trainingprocess, we implemented mixed precision training in our codebase. As a result, we are able to train ourmodel with a batch size of 128 on a single NVIDIA RTX 2080 TI GPU. The exact hyper-parametersbeing used in this work can be found in Table 2. Spike analysis with ElephantNormalizationSegmentationOASIS CalciumGAN OASISrecorded signal segmented signals inferred spikes synthetic signalsrandom noise inferred synthetic spikes
Figure 1: Pipeline diagram of a CalciumGAN analysis. White boxes illustrate data in differentprocessing stages. Blue boxes illustrate analysis steps and techniques.We devised a consistent model analysis pipeline to evaluate the quality of samples generated by themodel, as well as its ability to generalize, in the context of neuronal population spiking activities.The complete model analysis pipeline is shown in Figure 1.Since we evaluate our model performance in terms of spike activities, we needed a deconvolutionalgorithm to infer the spike trains from calcium signals. In this work, we used the Online Active3et method to Infer Spikes (OASIS) deconvolution algorithm [6] for its fast online deconvolutionperformance.We apply the Electrophysiology Analysis Toolkit (Elephant, NeuralEnsemble [17]) to measure spiketrain similarities and statistics. We evaluate the performance of our model with the following spiketrain statistics and similarities: a) mean firing rate for evaluating single neuron statistics; b) pairwisePearson correlation coefficient for evaluating pairwise statistics; c) pairwise van-Rossum [23] distancefor evaluating general spike train similarity. Importantly, we evaluate these quantities across thewhole population for each neuron or neuron pair and each short time interval (100 ms) and comparethe resulting distributions over these quantities obtained from training data as well as generated data.We therefore validate the whole spatiotemporal first- and second-order statistics as well as generalspike train similarities.
We propose CalciumGAN as a generative model to synthesize realistic calcium traces as imaged fromneuronal populations. To validate our model, we used artificial data with known ground-truth as wellas real data recorded from the primary visual cortex of behaving mice.
In order to verify that CalciumGAN is able to learn the underlying distribution and statistics of thetraining data, we generated our own ground-truth dataset with pre-defined mean and covariance usingthe dichotomized Gaussian (DG) model [14]. The model uses a multivariate normal distribution togenerate latent continuous random variables which are then thresholded to generate binary variablesrepresenting spike trains. The DG model has mean vector and covariance matrix as free parameters.To generate data from this model, we used the sample means and sample covariances obtained fromreal recorded data (see Section 3.3). In alignment with the recorded data, we generated correlatedspike trains for N = 102 neurons with a duration of 899 seconds and at 24Hz, hence a matrix withshape (21576 , . In order to obtain calcium-like signals c from spike trains s with length T , weconvolved the generated spike trains with a calcium response kernel and added noise, as described inFriedrich et al. [6]: s t = g × s t − + s t ≤ t ≤ Tc = b + s + σu u ∼ N (0 , where g denotes a finite impulse response filter, b is the baseline value of the signal and σ is thenoise standard deviation. In our work, we set g = 0 . , σ = 0 . and b = 0 . We scale the signalsrange to 0 to 1. The data is then segmented using a sliding window along the time dimension with astride of 2 and a window size of T = 2048 (around 85 seconds in experiment time). We apply thesegmentation procedure to both the signal and spike data, hence resulting in two matrices with shape (9754 , , . Examples of signals and spikes generated from the DG model can be found inFigure 7a. We first fit CalciumGAN to the artificial dataset sampled from the dichotomized Gaussian distribution.We trained CalciumGAN for 400 epochs with 8,754 samples and held out 1,000 samples for evaluation.Since we defined the model from which we generated the training dataset, we can validate the statisticsof the dataset generated by CalciumGAN on the known ground-truth directly. Examples of generatedsignals and spikes can be found in Figure 7b.We estimated the mean firing rates and the covariances of data generated by CalciumGAN andcompared it to the DG ones (Figure 2). We plotted the values of 5 samples for each neuron andneuron-pair, and sorted them by their mean in ascending order. Our model is able to reliably capturethe firing rate very well, with root mean square error of 0.0997Hz. The variation of the firing rateacross samples matched with those of the ground-truth data. The majority of the neuron pairs havelow correlation which was also found in the generated data. The neuron pairs that have highly positiveand highly negative covariance also have a greater variation across samples.4
Neuron F i r i n g r a t e ( H z ) DGCalciumGAN (a)
Neuron Pair C o v a r i a n c e DGCalciumGAN (b)
Figure 2: CalciumGAN trained on the dichotomized Gaussian dataset with known ground-truth.(a) Mean firing rate of each neuron. (b) Neuron pairwise covariance. Blue dots represent DG data andorange crosses present generated data. 5 randomly selected samples for each neuron and neuron-pairwere displayed in both graphs, where the order on the x-axis was sorted by the mean of the firing rateand covariance respectively. In (b), only every th pair is displayed, for clarity. Here, we compareboth the trend and variation of the generated data statistics with the dichotomized Gaussian data. After validating our model on data with known ground-truth, we applied CalciumGAN on real two-photon calcium imaging data recorded in the primary visual cortex of mice performing a virtual realitytask. The data were collected with the same setup as specified in Pakan et al. [19] and Henschkeet al. [10]. Head-fixed mice were placed on a cylindrical treadmill, and navigated a virtual corridorrendered on two monitors that covered the majority of their visual field. A lick spout was placed infront of the mice, where a water drop would be made available to the mice as a reward if it lickedat the correct location within the virtual environment. Hence, the mice would learn to utilize boththe visual information and the self-motion feedback in order to maximize the rewards. Neuronalactivity was monitored from the same primary visual cortex populations over multiple consecutivebehavioural sessions. In this work, we are using neuron population data recorded on the th dayof the experiment, where the mice were quite familiar with the virtual environment and the giventask. In this particular recording, neurons were labelled with GCamP6f, and N = 102 neurons wererecorded at a sampling rate of 24Hz, the mouse performed 204 trials in 898.2 seconds (raw datashape (21556 , ). Due to the fact that GAN models require a significant amount of training data,information about the trial and position of the mice in the virtual environment were not used in thiswork.We applied OASIS AR1 deconvolution algorithm to infer the spike activities from the recordedcalcium signals, and performed the same normalization and segmentation steps as mentioned inSection 3.1. Both calcium signals and inferred spike trains have shape (9754 , , . Figure 3ashows examples of the recorded calcium signals and inferred spike trains. There are multiplechallenges for both the generator and discriminator to learn from the calcium imaging signals. Sincedata were segmented with a sliding window and the information of the trial was not used, somesamples might consist of abnormal signal activity, such as a peak being cropped off. Generatedsignals could have the same number of peaks or ranges, though might not preserve the peak anddecay characteristics of calcium imaging data. Real and synthetic activity from less active neuronsmight be more difficult for the discriminator to distinguish due to the absence of prominent spikingcharacteristics. We tested CalciumGAN with the data recorded from the primary visual cortex of a trained mouse (seeSection 3.3). Similar to the DG analysis, we trained the model for 400 epochs, with 8,754 trainingsamples, and 1,000 samples were held out for evaluation. Note that since we are not taking the trial5
Time (s) Neuron recorded signalinferred spike
Time (s) Neuron (a)
Time (s) Neuron synthetic signalinferred spike
Time (s) Neuron (b)
Figure 3: Calcium signals and inferred spike trains (in gray) of randomly selected neurons. (a) showsthe recorded data (in blue) and (b) shows synthetic data (in orange) generated by CalciumGANtrained on recorded data. Note that the generated data should not be identical with the recorded data,because CalciumGAN should not replicate the signals and it could generate a sample correspondingto a different trial.and position of the mice in the virtual environment into consideration when training the model, thegenerated data and the evaluation data do not have a one-to-one mapping.We first inspect the generated data and the deconvolved spike trains visually. The calcium signals andinferred spike trains of 6 randomly selected neurons from a randomly selected sample are shown inFigure 3. Both the synthetic raw traces as well as the inferred spikes visually match the characteristicsof the recorded ones.We then compared the spiking characteristics across the whole population. Figure 4 shows theinferred spike trains of the complete 102 neurons population from a randomly selected sample of thereal and the synthetic data, with the distribution histogram plotted on the x and y axis. The syntheticdata mimicks the firing patterns across neurons and across time remarkably well with occasionalsmall deviations in the rates at particular temporal intervals. Notably, the samples are clearly notidentical meaning that the network did not just replicate the training set data.In order to examine if CalciumGAN is able to capture the first and second order statistics of therecorded data, we measured the mean firing rate, pairwise correlation, and van-Rossum distance (seeFigure 5). The 3 randomly selected neurons shown in Figure 5a all have very distinct firing ratedistributions, and CalciumGAN is able to model all of them relatively well, with KL divergence of0.42, 0.11 and 0.09 with respect to the recorded firing rate over 1000 samples. We show the pairwisevan-Rossum distance of the same neuron between recorded and generated data across 50 samplesin Figure 5c as sorted heatmaps. Less active neurons, such as neuron 75, have a low distance valueacross samples, mainly due to the scarcity of firing events. Conversely, a high frequency neuron, suchas neuron 6, exhibits a clear trend of lower distance values in the diagonal of the heatmap, implyingthe existence of a pair of recorded and generated sample that are similar. In order to ensure thatthe data generated by our model capture the underlying distribution of the training data, we alsocompute the KL divergence between the distributions of the above-mentioned metrics (see Figure 6).CalciumGAN was able to model all 3 of the statistics of the recorded data, with most samples havingKL divergence values of less than 1.5. Note that we measure the pairwise distance of the same neuronacross 50 samples in Figure 5c, whereas in Figure 6c, we measure pairwise van-Rossum distance ofeach neuron with respect to other neurons within the same sample.Next, we examined whether CalciumGAN is able to learn from neural activities that are more stochas-tic and potentially less correlated. To this end, we trained the model on the neuronal populationsdata recorded on the first day of the mice experiment. Figure 9 shows the raster plot of a randomlyselected sample and Figure 11 shows the first and second order statistics of the generated samples,similar to the plots we presented above. Overall, CalciumGAN was able to capture the statistics and6
10 20 31 41 52 62 72 83Time (s)020406080100 N e u r o n recorded synthetic Figure 4: Raster plot of inferred real and synthetic spike trains of a randomly selected samplegenerated by CalciumGAN trained on recorded data. Blue markers indicate recorded data and orangemarkers indicate generated data. The histograms on the x and y axis indicate number of spikes overthe temporal dimension and neuron population respectively.underlying distribution of the real calcium imaging data acquired in the primary visual cortex ofawake, behaving mice. Despite the recent advancement and popularity of calcium imaging of neuronal activity in vivo ,the number of trials and the duration of imaging sessions in animal experiments is limited due toethical and practical considerations. This work provides a readily applicable tool to fit a GenerativeAdversarial Network model on calcium signals, enabling the generation of more data that matchesthe statistics of the provided data.We demonstrated that the GAN framework is capable of synthesizing realistic fluorescent calciumindicator signals similar to those imaged in the somata of neuronal populations of behaving animals.To achieve this, we adapted the WaveGAN [5] architecture with the Wasserstein distance trainingobjective. We generated artificial neuronal activities using a dichotomized Gaussian model, showingthat CalciumGAN is able to learn the underlying distribution of the data. We then fitted our model toimaging data from the primary visual cortex of a behaving mice. Importantly, we could show that thestatistics of the synthetic spike trains match the statistics of the recorded data.To infer spike trains from the real and synthetic calcium traces, we used the OASIS deconvolutionalgorithm, a method which is particularly fast. Speed was a crucial characteristic for evaluating a largenumber of trials. Nevertheless, we found that this advantage often came at the cost of performance inthe form of clearly missed spikes (c.f. Figure 3). However, we stress that these shortcomings applyto both the real data and the synthetic data in exactly the same way. In the end, we use the inferredspikes as a way to validate the plausibility of the synthesized traces. The comparison is fair as long asreal and synthetic deconvolutions are subject to the same biases.7 .0 0.1 0.2 0.3Hz0100200300400500 C o un t Neuron C o un t Neuron C o un t Neuron recordedsynthetic (a) C o un t Sample C o un t Sample C o un t Sample recordedsynthetic (b) synthetic trial r e c o r d e d t r i a l Neuron synthetic trial r e c o r d e d t r i a l Neuron synthetic trial r e c o r d e d t r i a l Neuron (c)
Figure 5: First and second order statistics of data generated from CalciumGAN trained on therecorded data. Shown neurons and samples were randomly selected. (a) Mean firing rate distributionover 1000 samples per neuron. (b) Pearson correlation coefficient distribution. (c) van-Rossumdistance between recorded and generated spike trains over 50 samples. Heatmaps were sorted wherethe pair with the smallest distance value was placed at the top left corner, followed by the pair withthe second smallest distance at the second row second column, and so on.One potential future direction for this work is to provide a meaningful interpretation for the latentgenerator representation z . In many image generation tasks with GANs [3, 12] it has been shownthat the output image can be modified or targeted by interpolating the latent variable that is fed tothe generator. Similarly, one could potentially have final control of the generated calcium signalsby exploring the synthetic calcium signals generated after interpolating samples in the latent space.Thereby, one could generate calcium imaging data that resemble the neuronal activities of an animalperforming a particular novel task. Another interesting research direction would be using a GAN tolearn the relationship between different neuronal populations, or to reveal changes in activity of thesame neuronal population in different training phases of an animal learning a behavioral task. Thiscould be achieved by using, for instance, CycleGAN [29], an unsupervised learning model that canlearn the mapping between two distributions without paired data, as a potential model architecture.8 .0 0.5 1.0 1.5 2.0 2.5 3.0KL divergence0510152025 C o un t Mean firing rate (a) C o un t Correlation (b) C o un t van-Rossum distance (c) Figure 6: KL divergence of recorded data and generated data distributions. (a) Mean firing rateof each neuron over 1000 trials. (b) Pairwise Pearson correlation coefficient over 1000 trials. (c)Pairwise spike train van-Rossum distance over 1000 trials. The mean KL divergence of each statisticsare 0.45, 0.08 and 0.58 respectively.
Acknowledgments and Disclosure of Funding
We thank the GENIE Program and the Janelia Research Campus, specifically V. Jayaraman, R. Kerr,D. Kim, L. Looger, and K. Svoboda, for making GCaMP6 This work was funded by the Engineeringand Physical Sciences Research Council (grant EP/S005692/1 to A.O.), the Wellcome Trust and theRoyal Society (Sir Henry Dale fellowship to N.R.), the Marie Curie Actions of the European Union’sFP7 program (MC-CIG 631770 to N.R.), the Simons Initiative for the Developing Brain (to N.R.),the Precision Medicine Doctoral Training Programme (MRC, the University of Edinburgh (to T.A.)).
References [1] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. arXiv preprintarXiv:1701.07875 , 2017.[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprintarXiv:1607.06450 , 2016.[3] Piotr Bojanowski, Armand Joulin, David Lopez-Paz, and Arthur Szlam. Optimizing the latentspace of generative networks. arXiv preprint arXiv:1707.05776 , 2017.[4] Emery N Brown, Robert E Kass, and Partha P Mitra. Multiple neural spike train data analysis:state-of-the-art and future challenges.
Nature neuroscience , 7(5):456–461, 2004.[5] Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. In
Inter-national Conference on Learning Representations , 2019. URL https://openreview.net/forum?id=ByMVTsR5KQ .[6] Johannes Friedrich, Pengcheng Zhou, and Liam Paninski. Fast online deconvolution of calciumimaging data.
PLoS computational biology , 13(3):e1005423, 2017.[7] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, SherjilOzair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani,M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors,
Advances in NeuralInformation Processing Systems 27 , pages 2672–2680. Curran Associates, Inc., 2014. URL http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf .[8] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville.Improved training of wasserstein gans. In
Advances in neural information processing systems ,pages 5767–5777, 2017.[9] Kenneth D Harris, Rodrigo Quian Quiroga, Jeremy Freeman, and Spencer L Smith. Improvingdata quality in neuronal population recordings.
Nature neuroscience , 19(9):1165, 2016.[10] Julia U Henschke, Evelyn Dylda, Danai Katsanevaki, Nathalie Dupuy, Stephen P Currie,Theoklitos Amvrosiadis, Janelle MP Pakan, and Nathalie L Rochefort. Reward associationenhances stimulus-specific representations in primary visual cortex.
Current Biology , 2020.911] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network trainingby reducing internal covariate shift. arXiv preprint arXiv:1502.03167 , 2015.[12] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans forimproved quality, stability, and variation. arXiv preprint arXiv:1710.10196 , 2017.[13] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprintarXiv:1412.6980 , 2014.[14] Jakob H Macke, Philipp Berens, Alexander S Ecker, Andreas S Tolias, and Matthias Bethge.Generating spike trains with specified correlation coefficients.
Neural computation , 21(2):397–423, 2009.[15] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprintarXiv:1411.1784 , 2014.[16] Manuel Molano-Mazon, Arno Onken, Eugenio Piasini*, and Stefano Panzeri*. Synthesizingrealistic neural population activity patterns using generative adversarial networks. In
Inter-national Conference on Learning Representations , 2018. URL https://openreview.net/forum?id=r1VVsebAZ .[17] NeuralEnsemble. Elephant. URL https://github.com/NeuralEnsemble/elephant .[18] Augustus Odena, Vincent Dumoulin, and Chris Olah. Deconvolution and checkerboardartifacts.
Distill , 2016. doi: 10.23915/distill.00003. URL http://distill.pub/2016/deconv-checkerboard .[19] Janelle MP Pakan, Stephen P Currie, Lukas Fischer, and Nathalie L Rochefort. The impact ofvisual cues, reward, and motor feedback on the representation of behaviorally relevant spatiallocations in primary visual cortex.
Cell reports , 24(10):2521–2528, 2018.[20] Chethan Pandarinath, Daniel J O’Shea, Jasmine Collins, Rafal Jozefowicz, Sergey D Stavisky,Jonathan C Kao, Eric M Trautmann, Matthew T Kaufman, Stephen I Ryu, Leigh R Hochberg,et al. Inferring single-trial neural population dynamics using sequential auto-encoders.
Naturemethods , page 1, 2018.[21] Poornima Ramesh, Mohamad Atayi, and Jakob H Macke. Adversarial training of neuralencoding models on population spike trains. 2019.[22] Hernan Gonzalo Rey, Carlos Pedreira, and Rodrigo Quian Quiroga. Past, present and future ofspike sorting techniques.
Brain research bulletin , 119:106–117, 2015.[23] MCW van Rossum. A novel spike distance.
Neural computation , 13(4):751–763, 2001.[24] Shreya Saxena and John P. Cunningham. Towards the neural population doctrine, 2019. ISSN18736882.[25] Benjamin Staude, Stefan Rotter, and Sonja Grün. Cubic: cumulant based inference of higher-order correlations in massively parallel spike trains.
Journal of computational neuroscience , 29(1-2):327–350, 2010.[26] Ian H. Stevenson and Konrad P. Kording. How advances in neural recording affect data analysis.
Nature Neuroscience , 14(2):139–142, feb 2011. ISSN 1097-6256. doi: 10.1038/nn.2731. URL .[27] Carsen Stringer, Michalis Michaelos, and Marius Pachitariu. High precision coding in visualcortex. bioRxiv , 2019. doi: 10.1101/679324. URL .[28] Ziqiang Wei, Bei-Jung Lin, Tsai-Wen Chen, Kayvon Daie, Karel Svoboda, and Shaul Druck-mann. A comparison of neuronal population dynamics measured with calcium imaging andelectrophysiology. bioRxiv , page 840686, 2019.[29] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-imagetranslation using cycle-consistent adversarial networks. In
Proceedings of the IEEE internationalconference on computer vision , pages 2223–2232, 2017.10 upplementary material
Table 1: The generator (a) and discriminator (b) architecture of CalciumGAN. The generator consistsof 4,375,740 parameters, and the discriminator consists of 4,110,273 parameters.Layer Output shapeInput (n, 32)Dense (n, 2048)LeakyRelu (n, 2048)Reshape (n, 64, 32)Conv1DTranspose (n, 128, 320)LayerNorm (n, 128, 320)LeakyRelu (n, 128, 320)Conv1DTranspose (n, 256, 256)LayerNorm (n, 256, 256)LeakyRelu (n, 256, 256)Conv1DTranspose (n, 512, 192)LayerNorm (n, 512, 192)LeakyRelu (n, 512, 192)Conv1DTranspose (n, 1024, 128)LayerNorm (n, 1024, 128)LeakyRelu (n, 1024, 128)Conv1DTranspose (n, 2048, 102)LayerNorm (n, 2048, 102)LeakyRelu (n, 2048, 102)Dense (n, 2048, 102)Sigmoid (n, 2048, 102) (a) Generator architecture
Layer Output shapeInput (n, 2048, 102)Conv1D (n, 1024, 64)LeakyRelu (n, 1024, 64)PhaseShift (n, 1024, 64)Conv1D (n, 512, 128)LeakyRelu (n, 512, 128)PhaseShift (n, 512, 128)Conv1D (n, 256, 192)LeakyRelu (n, 256, 192)PhaseShift (n, 256, 192)Conv1D (n, 128, 256)LeakyRelu (n, 128, 256)PhaseShift (n, 128, 256)Conv1D (n, 64, 320)LeakyRelu (n, 64, 320)Flatten (n, 20480)Dense (n, 1) (b) Discriminator architecture
Table 2: CalciumGAN hyperparamtersHyper-parameters ValueFilters 64Kernel size 24Stride 2Noise dimension 32Critic update 5 λ gradient penalty 10Batch size 128Epochs 400Learning rate 0.0001Phase shift 1011 Time (s)
Neuron
DG signalDG inferred spike
Time (s)
Neuron (a)
Time (s)
Neuron synthetic signalinferred spike
Time (s)
Neuron (b)
Figure 7: Calcium signals and inferred spike trains (in gray) of randomly selected neurons. (a) showsthe dichotomized Gaussian data (in blue) and (b) shows synthetic data (in orange) generated byCalciumGAN trained on the DG data. Notice that the artificial signal data transformed from DGspike data (see Section 3.1) do not have the peak and decay characteristics of typical calcium imagingdata.
Time (s) Neuron recorded signalinferred spike
Time (s) Neuron (a)
Time (s) Neuron synthetic signalinferred spike
Time (s) Neuron (b)
Figure 8: Calcium signals and inferred spike trains (in gray) of randomly selected neurons. (a) showsthe recorded data (in blue) and (b) shows synthetic data (in orange) generated by CalciumGANtrained on the recorded data with no Phase Shift (see Section 2.1). Notice the sharp rise to peakfollowed by a tail of decaying signal which is outside of the window shown for CalciumGAN.12
20 41 62 83Time (s)020406080100 N e u r o n recorded synthetic Figure 9: Raster plot of inferred real and synthetic spike trains of a randomly selected samplegenerated by CalciumGAN trained on calcium imaging data recorded on day one of the animalexperiment. Blue markers indicate recorded data and orange markers indicate generated data. Thehistograms on the x and y axis indicate number of spikes over the temporal dimension and neuronpopulation respectively. Compared to recordings acquired on the 4 th day of the experiment, mostneurons recorded in the untrained animal are more active. C o un t Mean firing rate (a) C o un t Correlation (b) C o un t van-Rossum distance (c) Figure 10: KL divergence of calcium data recorded on day 1 of the animal experiment and generateddata distributions. (a) Mean firing rate of each neuron over 1000 trials. (b) Pairwise Pearsoncorrelation coefficient over 1000 trials. (c) Pairwise spike train van-Rossum distance over 1000 trials.The mean KL divergence of each statistics are 0.38, 0.06 and 0.60 respectively.13 .0 0.2 0.4 0.6Hz020406080 C o un t Neuron C o un t Neuron C o un t Neuron recordedsynthetic (a) C o un t Sample C o un t Sample C o un t Sample recordedsynthetic (b) synthetic trial r e c o r d e d t r i a l Neuron synthetic trial r e c o r d e d t r i a l Neuron synthetic trial r e c o r d e d t r i a l Neuron (c)(c)