Fast deep learning correspondence for neuron tracking and identification in C.elegans using synthetic training
Xinwei Yu, Matthew S. Creamer, Francesco Randi, Anuj K. Sharma, Scott W. Linderman, Andrew M. Leifer
FFast deep learning correspondence for neuron tracking andidentification in
C. elegans using synthetic training
Xinwei Yu , Matthew S. Creamer , Francesco Randi , Anuj K. Sharma , Scott W.Linderman , Andrew M. Leifer Department of Physics, Princeton University, Princeton, NJ, United States of America Princeton Neuroscience Institute, Princeton University, Princeton, NJ, United Statesof America Department of Statistics, Stanford University, Stanford, CA, United States of America Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, United States ofAmerica* [email protected]
Abstract
We present an automated method to track and identify neurons in
C. elegans , called“fast Deep Learning Correspondence” or fDLC, based on the transformer networkarchitecture. The model is trained once on empirically derived synthetic data and thenpredicts neural correspondence across held-out real animals via transfer learning. Thesame pre-trained model both tracks neurons across time and identifies correspondingneurons across individuals. Performance is evaluated against hand-annotated datasets,including NeuroPAL [1]. Using only position information, the method achieves 80.0%accuracy at tracking neurons within an individual and 65.8% accuracy at identifyingneurons across individuals. Accuracy is even higher on a published dataset [2].Accuracy reaches 76.5% when using color information from NeuroPAL. Unlike previousmethods, fDLC does not require straightening or transforming the animal into acanonical coordinate system. The method is fast and predicts correspondence in 10 msmaking it suitable for future real-time applications.
Introduction
The nervous system of the nematode
C.elegans is well characterized, such that each ofthe 302 neurons are named and have stereotyped locations across animals [3–5]. Thecapability to find corresponding neurons across animals is essential to investigate neuralcoding and neural dynamics across animals. Despite the worm’s overall stereotypy, thevariability in neurons’ spatial arrangement is sufficient to make predicting neuralcorrespondence a challenge. For whole-brain calcium imaging [6, 7], identifying neuronsacross animals is additionally challenging because the nuclear localized markers that areused tend to obscure morphological features that would otherwise assist in neuralidentification.An ideal method for finding neural correspondence in
C. elegans shouldaccommodate two major use cases. The first is tracking neurons within an individualacross time as the animal’s head moves and deforms. Here the goal is to be able to saywith confidence that a neuron imaged in a volume taken at time t is the same asanother neuron taken from a volume imaged at time t . Tracking across time is neededJanuary 21, 2021 1/22 a r X i v : . [ q - b i o . Q M ] J a n o extract calcium dynamics from neurons during freely moving population calciumimaging [6–8]. Additionally, very fast real-time tracking will be needed to guideclosed-loop optogenetic stimulation of neurons in moving animals as those methodsmove to larger neural populations [9–12].The second and more general use case is finding neural correspondence acrossindividuals. Often this is to identify the name of a neuron with respect to theconnectome [3] or a gene expression atlas [13]. Even when a neuron’s name cannot beascertained, being able to identify which neurons are the same across recordings allowsresearchers to study neural population codes common across individuals.For both use cases, a method to find neural correspondence is desired that isaccurate, fast, requires minimal experimental training data and that generalizes acrossanimal pose, orientation, imaging hardware, and conditions. Furthermore, an idealmethod should not only perform well when restricted to neural positioning informationbut, should also be flexible enough to leverage genetically encoded color labelinginformation or other features for improved accuracy when available. Multicolor strainsare powerful new tools that use multiple genetically encoded fluorescent labels to aidneural identification [1, 14] (we use one of those strains, NeuroPAL [1], for validating ourmodel). However, some applications, like whole-brain imaging in moving worms, are notyet easily compatible with the multicolor imaging required by these new strains, sothere remains a need for improved methods that use position information alone.A variety of automated methods for C. elegans have been developed that addresssome, but not all of these needs. Most methods developed so far focus on finding theextrinsic similarity [15] between one neuron configuration, called a test, and anotherneuron configuration called a template. Methods like these deform space to minimizedistances between neurons in the template and neurons in the test and then attempt tosolve an assignment problem [16]. For example, a simple implementation would be touse a non-rigid registration model, like Coherent Point Drift (CPD) [17] to optimize awarping function between neuron positions in the test and template. More recentnon-rigid registration algorithms like PR-GLS [18] also incorporate relative spatialarrangement of the neurons [19].Models can also do better by incorporating the statistics of neural variability.NeRVE registration and clustering [20], for example, also uses a non-rigid point setregistration algorithm [21] to find a warping function that minimizes the differencebetween a configuration of neurons at one time point and another. But NeRVE furtherregisters the test neurons onto multiple templates to define a feature vector and thenfinds neural correspondence by clustering those feature vectors. By using multipletemplates, the method implicitly incorporates more information about the range andstatistics of that individual animal’s poses to improve accuracy.A related line of work uses generative models to capture the statistics of variabilityacross many individual worms. These generative models specify a joint probabilitydistribution over neural labels and the locations, shapes, sizes, or appearance of neuronsidentified in the imaging data of multiple individuals [22–25]. These approaches arebased on assumptions about the likelihood of observing a test neural configuration, givenan underlying configuration of labeled neurons. For example, these generative modelsoften begin with a Gaussian distribution over neuron positions in a canonical coordinatesystem and then assume a distribution over potentially non-rigid transformations of theworm’s pose for each test configuration. Then, under these assumptions, the most likelyneural correspondence is estimated via approximate Bayesian inference.The success of generative modeling hinges upon the accuracy of its underlyingassumptions, and these are challenging to make for high-dimensional data. Analternative is to take a discriminative modeling approach [26]. For example, recentwork [2] has used conditional random fields (CRF) to directly parameterize aJanuary 21, 2021 2/22onditional distribution over neuron labels, rather than assuming a model for thehigh-dimensional and complex image data. CRF allows for a wide range of informativefeatures to be incorporated in the model, such as the angles between neurons, or theirrelative anterior-posterior positions, which are known to be useful for identifyingneurons [27]. Ultimately, however, it is up to the modeler to select and hand curate aset of features to input into the CRF.The next logical step is to allow for much richer features to be learned from the data.Artificial neural networks are ideal for tackling this problem, but they requireimmensely large training sets. Until now, their use for neuron identification has beenlimited. For example, in one tracking algorithm, artificial neural networks provide onlythe initialization, or first guess, for non-rigid registration [19].Our approach is based on a simple insight: it is straightforward to generate verylarge synthetic datasets of test and template worms that nonetheless are derived frommeasurements. We use neural positions extracted from existing imaging datasets, andthen apply known, nonlinear transformations to warp those positions into new shapesfor other body postures. Furthermore, we simulate the types of noise that appear in realdatasets, such as missing or spurious neurons. Using these large-scale synthetic datasets,we can train an artificial neural network to map the simulated neural positions back tothe ground truth. Given sufficient training data (which we can generate at will), thenetwork learns the most informative features of the neural configurations, rather thanrequiring the user to specify them by hand. Realistic synthetic or augmented datasetslike these have been key to cracking other challenging problems in neural and behavioraldata analysis [28, 29], and have already shown promising potential for trackingneurons [19].In this work, we use synthetic data to train a Transformer network, an artificialneural network architecture that has shown great success in natural language processingtasks [30]. Transformers incorporate an attention mechanism that can leveragesimilarities between pairs of inputs to build a rich representation of the input sequencefor downstream tasks like machine translation and sentiment prediction. We reasonedthis same architecture would be well-suited to extract spatial relationships betweenneurons in order to build a representation that facilitates finding correspondence toneurons in a template worm.Not only is the Transformer well-suited to learning features for the neuralcorrespondence problem, it also obviates the need to straighten [31] the worm inadvance. Until now, existing methods have either required the worm to be straightenedin preprocessing [2, 22] or explicitly transformed them during inference [23, 24].Straightening the worm is a non-trivial task, and it is especially error-prone forcomplicated poses such as when the worm rolls along its centerline.Finally, one of the main advantages of the Transformer architecture is that itpermits parallel processing of the neural positions using modern GPU hardware. Incontrast to existing methods, which have not been optimized for speed, the Transformercan make real-time predictions once it has been trained. This speed is a necessary steptoward future experiments with real-time, closed-loop, targeted delivery of optogeneticstimulation of large populations of neurons in freely moving animals.
Materials and methods
Datasets
Our model was trained on a synthetic dataset derived from recordings of 12 freelymoving animals. The model’s performance was evaluated on a different set of 12held-out recordings: one moving recording and 11 immobile recordings. The recordingJanuary 21, 2021 3/22 able 1. List of emission filters for multicolor imaging.
Filter label Filters (Semrock part n.)F1 FF01-440/40F2 FF01-607/36F3 FF02-675/67 + FF01-692/LP
Table 2. Imaging channels used.
Channel Excitation λ (nm) Emission window (nm) [filter] Primary Fluorophorech0 405 420-460 [F1] mtagBFPch1 488 589-625 [F2] CyOFPch2 561 589-625 [F2] tagRFP-tch3 561 692-708 [F3] mNeptuneof moving animals used strain AML32 wtfIs5[P rab-3 ::NLS::GCaMP6s;P rab-3 ::NLS::tagRFP] and was first reported in [20]. Immobile recordings to evaluateperformance used strain AML320 (otIs669[NeuroPAL] V 14x; wtfIs145 [pBX +rab-3::his-24::GCaMP6::unc-54] ) derived from NeuroPAL strain OH15262 [1]. Allstrains and datasets in this study used nuclear localized fluorescent reporters.Neural configurations acquired as part of this study have been posted in an OpenScience Foundation repository with DOI:10.17605/OSF.IO/T7DZU available at https://dx.doi.org/10.17605/OSF.IO/T7DZU .Model performance was also evaluated on a publicly accessible dataset from [2]available at https://github.com/shiveshc/CRF_Cell_ID , commit . Imaging
To image neurons in the head of freely moving worms, we used a dual-objectivespinning-disk based tracking system [7] (Yokogawa CSU-X1 mounted on a NikonEclipse TE2000-S). Fluorescent images of the head of a worm were recorded through a40x objective with both 488- and 561-nm excitation laser light as the animal crawled.The 40x objective translated up and down along the imaging axis to acquire 3D imagestacks at a rate of 6 head volumes/s.To image neurons in the immobile multi-color NeuroPAL worms [1] we modified oursetup by adding emission filters in a motorized filter wheel (Prior ProScan-II), andadding a Stanford Research Systems SR474 shutter controller (with SR475 shutters) toprogrammatically illuminate the worm with different wavelength laser light. We usethree lasers with light of different wavelengths: 405 nm (Coherent OBIS-LX 405 nm 100mW), 488 nm (Coherent SAPPHIRE 488 nm 200 mW), and 561 nm (CoherentSAPPHIRE 561 nm 200 mW). Only one laser at a time reached the sample, through a40x oil-immersion objective (1.3 NA, Nikon S Fluor). The powers measured at thesample, after spinning disk and objective, were 0.14 mW (405 nm), 0.35 mW (488 nm),and 0.36 mW (561 nm). In the spinning disk unit, a dichroic mirror (ChromaZT405/488/561tpc) separated the excitation from the emission light. The latter wasrelayed to a cooled sCMOS camera (Hamamatsu ORCA-Flash 4.0 C11440-22CU),passing through the filters mounted on the filter wheel (Table 1). Fluorescent imageswere acquired in different “channels”, i.e. different combinations of excitationwavelength, emission filter, and camera exposure time (Table 2). The acquisition wasperformed using a custom software written in LabVIEW that specifies the sequence ofchannels to be imaged, and controls shutters, filter wheel, piezo translator, and camera.After setting the z position, the software acquires a sequence of images in the specifiedchannels.January 21, 2021 4/22 reprocessing and Segmentation
We extracted the position of individual neurons from 3D fluorescent images to generatea 3D point cloud, (Fig. 1A). This process is called segmentation and the fDLC model isagnostic to the specific choice of the segmentation algorithm.For recordings of strains AML32, we used a segmentation algorithm adopted from[20]. We first applied a threshold to find pixels where the intensities are significantlylarger than the background. Then, we computed the 3D Hessian matrix and itseigenvalues of the intensity image. Candidate neurons were regions where the maximaleigenvalue was negative. Next, we searched for the local intensity peaks in the regionand spatially disambiguated peaks in the same region with a watershed separationbased on pixel intensity.For recordings of NueroPAL strains, we used the same segmentation algorithm asin [1]. The publicly accessible dataset from [2] used in Fig. 4 had already beensegmented prior to our use.
Generating synthetic point clouds for training
We developed a simulator to generate a large training set of synthetic animals withknown neural correspondence. The simulator takes as its input the point cloudscollected from approximately 4,000 volumes spread across recordings of 12 freely movinganimals. For each volume, the simulator performs a series of stochastic deformationsand transformations to generate 64 new synthetic individuals where the ground truthcorrespondence between neurons in the individuals and the original point cloud isknown. A total of 2 . × synthetic point clouds were used for training.The simulator introduces a variety of different sources of variability and real-worlddeformations to create each synthetic point cloud (Fig. 1B,E). The simulator starts bystraightening the worm in the XY plane using its centerline so that it now lies in acanonical worm coordinate system. Before straightening, Z is along the optical axis andXY are defined to be perpendicular to the optical axis and are arbitrarily set by theorientation of the camera. After straightening, the animal’s posterior-anterior axis liesalong the X axis. To introduce animal-to-animal variability in relative neural position, anon-rigid transformation is applied to the neuron point cloud against a templaterandomly selected from recordings of the real observed worms using coherent point drift(CPD) [17]. To add variability associated with rotation and distortion of the worm’shead in the transverse plane, we apply a random affine transformation to the transverseplane. To simulate missing neurons and segmentation errors, spurious neurons arerandomly added, and some true neurons are randomly removed, for up to 20% of theobserved neurons. To introduce variability associated with animal pose, we randomlydeform the centerline of the head. Lastly, to account for variability in animals’ size andorientation, a random affine transformation in XY plane is applied that rescaled theanimal’s size by up to 5%. With those steps, the simulator deforms a sampled wormand generates a new synthetic worm with different orientation and posture whilemaintaining known correspondence.Whenever practical, the magnitude of the simulator’s perturbations was informed byobservation of real worms. For example the centerlines generated by the simulator weredirectly sampled from recordings of real individuals. The magnitude of added Gaussiannoise was set to have a standard deviation of 0 . µm , as this roughly matched ourestimate of variability observed by eye.January 21, 2021 5/22 eep Learning Correspondence Model Overview and input
The deep learning correspondence model (fDLC) is an artificial neural network based onthe Transformer [30] architecture (Fig. 1C) and is implemented in the automaticdifferentiation framework PyTorch [32]. The fDLC model takes as input the positionalcoordinates of a pair of worms, a template worm a , and test worm, b (Fig. 1F). For eachworm, approximately 120 neurons are segmented and passed to the fDLC model. Architecture
The model works as an encoder, which maps the input neuron coordinates( a , a , ..., a n , b , b , ..., b m ) to continuous embeddings ( u , u , ...u n , v , v , ..., v m ). Themodel is composed of a stack of N = 6 identical layers. Each layer consists of twosub-layers: a multi-head self-attention mechanism [30], and a fully connectedfeed-forward network. The multi-head attention mechanism is the defining feature ofthe transformer architecture and makes the architecture well-suited for finding relationsin sequences of data, such as words in a sentence or, in our case, spatial locations ofneurons in a worm. Each head contains a one-to-one mapping between the nodes in theartificial network and the C. elegans neurons. In the transformer architecture, featuresof a previous layer are mapped via a linear layer into three attributes of each node,called the query, the key and the value pairs. These attributes of each node contain highdimensional feature vectors which, in our context, represent information about theneuron’s relative position. The multi-head attention mechanism computes a weight foreach pair of nodes (corresponding to each pair of
C. elegans neurons). The weights arecalculated by performing a set computation on the query and key. The output iscalculated by multiplying this resultant weight by the value. In our implementation, weset the number of heads in the multi-head attention module to be 8 and we set thedimension of our feature vectors to be 128. We choose the best set of thehyperparameters by evaluating on a validation set, which is distinct from the trainingset and also from any data used for evaluation. A residual connection [33] and layernormalization [34] are employed for each sub-layer, as is widely used in artificial neuralnetworks.
Calculating probabilities for potential matches
The fDLC model generates a high dimensional ( d = 128) embedding u i for neuron i from the template worm and v j for the neuron j from the test worm. The similarity ofa pair of embeddings, as measured by the inner product (cid:104) u i , v j (cid:105) , determines theprobability that the pair is a match. Specifically, we define the probability that neuron i in the template worm matches neuron j in the test worm as p ij , where p ij = e (cid:104) ui,vj (cid:105) (cid:80) mk =1 e (cid:104) ui,vk (cid:105) . (1)Equivalently, the vector p i = ( p i , . . . , p im ) is modeled as the “softmax” function of theinner products between the embedding of neuron i and the embeddings of all candidateneurons 1 , . . . , m . The softmax output is non-negative and sums to one so that p i canbe interpreted as a discrete probability distribution over assignments of neuron i .We also find the most probable correspondence between the two sets of neurons bysolving a maximum weight bipartite matching problem where the weights are given bythe inner products between test and template worm embeddings. This is a classiccombinatorial optimization problem, and it can be solved in polynomial time using theHungarian algorithm [35].January 21, 2021 6/22 nd-user output The fDLC model returns two sets of outputs to the end user. One is the algorithm’sestimate of the most probable labels for each neuron in the test worm; i.e. the solutionto the maximum weight bipartite matching problem described above. The other is anordered list of alternative candidate labels for each individual neuron in the test wormand their probabilities ranked from most to least probable.
Training
The model was trained on 2 . × synthetic animals derived from recordings of 12individuals. The model was trained only once and the same trained model was usedthroughout this work.Training is as follows. We performed supervised learning with ground truth labels ofneuron names provided by the synthetically generated data. A cross-entropy lossfunction was used. If neuron i and neuron j has the same name, the cross-entropy lossfunction favors the model to output p ij = 1. If neuron i and neuron j have differentnames, the loss function favors the model to output p ij = 0. The model was trained for12 hours on a 2.40 GHz Intel machine with NVIDIA Tesla P100 GPU. Evaluating model performance and comparing against othermodels
Minimum confidence threshold
Accuracy of the fDLC model is calculated by including all neural matches between thetest and template regardless of confidence, except for in the case of Fig. 3. In thatanalysis only, we sought to reduce the number of matches to better compare to theNeRVE method which only matches 80% of the neurons. We therefore excludedmatches below a minimum confidence threshold of 0.05 from our calculation of accuracy.The tradeoff between accuracy and coverage is explored further in Fig. 4G.
Coherent Point Drift
We use Coherent Point Drift (CPD) Registration [17] as a baseline with which tocompare our model’s performance. In our implementation, CPD is used to find theoptimal non-rigid transformation to align the test worm with respect to the templateworm. We then calculated the distance for each pair of the neurons from thetransformed test worm and the template worm. We used the Hungarian algorithm [35]to find the optimal correspondence that minimizes the total squared distance for allmatches.
Color Model
The recently developed NeuroPAL strain [1] expresses four different genetically encodedfluorescent proteins in specific expression patterns so that a human can better identifyneurons across animals. Manual human annotation based on these expression patternsserves as ground truth in evaluating our model’s performance throughout this work. InFig. 5B we also explored combining color information with our fDLC model. To do sowe developed a simple color matching model that operated in parallel to our positionbased fDLC model. Outputs of both models were then combined to predict the finalcorrespondence between neurons.Our color matching model consists of two steps: First, the intensity of each of thecolor channels is normalized by the total intensity. Then the similarity of color for eachJanuary 21, 2021 7/22 ig 1. Fast Deep Learning Correspondence Model. (Caption continued on thenext page.)January 21, 2021 8/22 ig 1. Fast Deep Learning Correspondence Model (A-D) schematic of trainingand analysis pipeline for using the fast Deep Learning Correspondence (fDLC) model topredict correspondence between neurons across individuals. (A) Volumetric images offluorescent labeled neuronal nuclei are segmented to extract neuron positions. (Scalebar, 10 µm ). (B) Synthetic training data is generated with a simulator. The simulatortransforms the neural positions of a real worm and introduces noise to generate newsynthetic individuals. Approximately N = 10 neuron configurations without labelsfrom 12 moving worms were used to generate 2 . × labeled synthetic worms fortraining. (C) During training, the fDLC model finds optimal internal parameters tominimize the difference between predicted neural labels and true labels in pairs ofsynthetic worms. (D) Given position and labels for neurons in real worm A and positionfor neurons in real worm B, the trained model predicts corresponding labels for worm B.(E) Detailed schematic of the simulator from panel B. (F) Transformer architecture ofthe fDLC model. The position features of a template worm with n neurons and a testworm with m neurons are taken as input. The features are computed via a multi-headattention mechanism. ‘Add & Norm’ refers to an addition and layer normalization step. a and b are neuron positions and u and v are embeddings for the template and test,respectively. We choose the number of layers N = 6 and the embedding dimension d emb = 128 by evaluating the performance on a held-out validation set.pair of neurons is measured as the inverse of the Kullback–Leibler divergence betweentheir normalized color features.To calculate the final combined matching matrix, we add the color similarity matrixto the position matching log probability matrix from our fDLC model. The similaritymatrix of color is multiplied by a factor λ . We chose λ = 60 so that the amplitude ofvalues in the similarity matrix of color is comparable to our fDLC output. We note thematching results are not particularly sensitive to the choice of λ . The most probablematches are obtained by applying Hungarian algorithm on the combined matchingmatrix. Code
Source code in Python is provided for the model, for the simulator, and for training andevaluation. A jupyter notebook with a simple example is also provided. Code isavailable at https://github.com/XinweiYu/fDLC_Neuron_ID
Results
Fast deep learning correspondence accurately labels neuronsacross synthetic individuals
We developed a fast deep learning correspondence (fDLC) model that seeks to find thecorrespondence between configurations of
C. elegans neurons in different individuals orin the same individual across time (Fig. 1). We used a deep learning artificial neuralnetwork architecture, called the transformer architecture [30], that specializes at findingpairs of relations in datasets. The transformer architecture identified similarities acrossspatial relations of neurons in a test and a template to identify correspondences betweenthe neurons.Within a single individual, neural positions vary as the worm moves, deforms, andchanges its orientation and pose. Across isogenic individuals, there is an additionalsource of variability that arises from the animal’s development. In practice, furtherJanuary 21, 2021 9/22ariability also arises from experimental measurements: neuron positions must first beextracted from fluorescent images, and slight differences in label expression, imagingartifacts, and optical scattering all contribute to errors in segmenting individual neurons.We created a simulator to model these different sources of variability and used it togenerate realistic pairs of empirically derived synthetic animals with knowncorrespondence between their neurons for training our model (Fig. 1B, E). Thesimulator took configurations of neuron positions from real worms as inputs and thenscaled and deformed them, forced them to adopt different poses sampled from realworms, and then introduced additional sources of noise to generate many new syntheticindividuals. We then trained our fDLC model on these experimentally derived syntheticindividuals of different sizes and poses.Training our model on the empirically derived synthetic data offered advantagescompared to experimentally acquired data. First, it allowed us to train on largerdatasets than would otherwise be practical. We trained on 2 . × syntheticindividuals, but only seeded our simulator with unlabeled neural configurations fromexperimentally acquired recordings of 12 individuals. Second, we did not need toprovide ground truth correspondence because the simulator instead generates its ownground truth correspondence between synthetic individuals, thereby avoiding a tediousand error prone manual step. We note we still use human annotation for validation, butthis requires much smaller data sets. Third, by using large and varied synthetic data,we force the model to generalize its learning to a wide range of variabilities in neuralpositions and we avoid the risks of overtraining on idiosyncrasies specific to our imagingconditions or segmentation. Overall, we reasoned that training with synthetic datashould make the model more robust and more accurate across a wider range ofconditions, orientations and animal poses than would be practical with experimentallyacquired datasets.We trained our fDLC model on 2 . × synthetic individuals (Fig. 1C andmethods) and then evaluated its performance on 200 additional held-out synthetic pairsof individuals which had not been accessible to the model during training (Fig. 2).Model performance was evaluated by calculating the accuracy of the models’ predictedcorrespondence with respect to the ground truth in pairs of synthetic individuals. Oneindividual is called the “test” and the other is the “template”. Accuracy is reported asthe fraction of neurons, present in both the test and the template, that were correctlymatched. Our fDLC model achieved 96.6% average accuracy on the 200 pairs ofheld-out synthetic individuals. We compared this performance to that of Coherent PointDrift (CPD) [17], a classic registration method used for automatic cell annotation. CPDachieved 30.8% mean accuracy on the same held-out synthetic individuals. Ourmeasurements show that the fDLC model significantly outperforms CPD at findingcorrespondence in synthetic data. For the rest of the work, we use experimentallyacquired human annotated data to evaluate performance. fDLC accurately tracks neurons within an individual across time We next evaluated the fDLC model’s performance at tracking neurons within anindividual over time, as is needed, for example, to measure calcium activity in movinganimals [6, 7]. We evaluated model performance on an experimentally acquired calciumimaging recording of a freely moving
C. elegans (strain AML32) from [20] in which ateam of human experts had manually tracked and annotated neuron positions over time.This recording had been excluded from the set of recordings used by the simulator. Wesampled neuron configurations from 201 different time points during this recording toform 200 pairs of configurations upon which to evaluate the fDLC model. Each pairconsisted of a test and template. The template was always from the same time point t ,while the test was taken to be any of the other 200 time points. We applied theJanuary 21, 2021 10/22 ig 2. fDLC accurately predicts neuron labels of synthetic worms (A)Schematic of evaluation pipeline. fDLC model performance is evaluated on pairs ofsynthetic worms with known labels that had been held out from training. Given neuralpositions in worms A and B, and neuron labels from A, the model predicts neuronlabels for B. Accuracy is the percent of labeled neurons, present in both A and B, thatthe model correctly predicts. (B) Model performance of a Coherent Point DriftRegistration (CPD) is compared to the fDLC model on 200 randomly selected pairs ofheld-out synthetic individuals, without replacement. ( p = 2 . × − , Wilcoxonsigned rank test).January 21, 2021 11/22 ig 3. Tracking neurons within an individual across time. (A) Schematicshows how the pose and orientation of a freely moving animal change with time. Blackdot indicates head. (B) Pipeline to evaluate the fDLC model at tracking neurons withinan individual across time. The fDLC model takes in positional features of a templateneuron configuration from one time t of a freely moving worm, and predicts thecorrespondence at another time t , called the test. Recording is from [20]. Ground truthneuron labels are provided by manual human annotation. The same time point is usedas the template for all 200 template-test pairs. (C) Performance of fDLC andalternative models at tracking neurons within an individual are displayed in order ofmean performance. CPD refers to Coherent Point Drift. NeRVE(1) refers to therestricted NeRVE model that has access to only the same template as CPD and fDLC.NeRVE(100) refers to the full NeRVE model which uses 100 templates from the sameindividual to make a single prediction. A Wilcoxon signed rank significance test offDLC’s performance compared to CPD, NeRVE(1) and NeRVE(100) yields p = 4 . × − , .
006 and 3 . × − respectively. Boxplots show median andinterquartile range.January 21, 2021 12/22re-trained fDLC model to the 200 pairs of neuron configurations and compared themodel’s predicted correspondence to the ground truth from manual human tracking(Fig. 3). Across the 200 pairs, the fDLC model showed an average accuracy of 80.0%.We emphasize that the fDLC model achieved this high accuracy on tracking a real wormusing only neuron position information even though it is trained exclusively on syntheticdata.We compared the performance of our fDLC model to that of CPD Registration, andto Neuron Registration Vector Encoding and clustering (NeRVE), a classical computervision model that we had previously developed specifically for tracking neurons within amoving animal over time [20] (Fig. 3B). NeRVE only matches approximately 80% ofneurons, so for the sake of comparison, in this analysis (but not others), we onlyconsider comparable number of matches from the fDLC, see “minimum confidencethreshold” in “Materials and Methods.”) fDLC clearly outperformed CPD achieving80.0% accuracy compared to CPD’s 58.5%.Both CPD and fDLC predict neural correspondence of a test configuration bycomparing only to a single template. In contrast, the NeRVE method takes 100templates, where each one is a different neuron configuration from the same individual,and uses them all to inform its prediction. The additional templates give the NeRVEmethod extra information about the range of possible neural configurations made by thespecific individual whose neurons are being tracked. We therefore compared the fDLCmodel both to the full NeRVE method and also to a restricted version of the NeRVEmethod in which NeRVE had access only to the same single template as the fDLCmodel. (Under this restriction the NeRVE method no longer clusters and the methodcollapses to a series of gaussian mixture model registrations [21].) In this way, we couldcompare the two methods when given the same information. fDLC’s mean performanceof 80.0% was very similar but statistically significantly more accurate than therestricted NeRVE model (mean 79.1%, p = 6 . × − , Wilcoxon signed rank test). Thefull NeRVE model that had access to additional templates outperformed the fDLCmodel slightly (82.0% p = 3 . × − , Wilcoxon signed rank test). We conclude thatthe fDLC model is suitable for tracking individual neurons across time and performssimilarly to the NeRVE method.In the following sections, we further show that the fDLC method is orders ofmagnitude faster than NeRVE. Moreover, unlike NeRVE which can only be used withinan individual, fDLC is also able to predict the much more challenging neuralcorrespondence across individuals. fDLC is fast enough for future real-time tracking Because it relies on an artificial neural network, the fDLC model finds correspondencefor a set of neurons faster than traditional methods (Table 3). From the time that aconfiguration of segmented neurons is loaded onto a GPU, it takes only an average of 10ms for the fDLC model to label all neurons on a 2.4 GhZ Intel machine with anNVIDIA Tesla P100 GPU. If not using a GPU, the model labels neurons in 50 ms. Incontrast, on the same hardware it takes CPD 930 ms and it takes NeRVE on averageover 10 seconds. The fDLC model may be a good candidate for potential closed-looptracking applications because its speed of 100 volumes per second is an order ofmagnitude faster than the 6 to 10 volumes per second recording rate typically used inwhole-brain imaging of freely moving
C. elegans [6, 7]. We note that for a completeclosed-loop tracking system, fast segmentation algorithms will also be needed inaddition to the fast registration and labeling algorithms presented here. The fDLCmodel is agnostic to the details of the segmentation algorithm so it is well suited to takeadvantage of fast segmentation algorithms when they are developed.The fDLC model uses built-in libraries to parallelize the computations for labeling aJanuary 21, 2021 13/22 able 3. Time required to find neural correspondenceMethod Time(s/Volume)
CPD [17] 0.93NeRVE(1) [20] 10NeRVE(100) [20] > Table shows the measured time per volume required for different models to predictneural correspondence of a single volume. Time required is measured after neuronsegmentation is complete and a configuration of neural positions has been loaded intomemory. The same hardware is used for all models.single volume, and this contributes to its speed. In particular, each layer of the neuralnetwork contains thousands of artificial neurons performing the same computation.Computations for each neuron in a layer can all be performed in parallel and modernGPUs have as many as 3,500 CUDA cores.In practice, the method is even faster for post-processing applications (not-realtime)because it is also parallelizable at the level of each volume. Labeling one volume has nodependencies on any previous volumes and therefore each volume can be processedsimultaneously. The number of volumes to be processed in parallel is limited only bynumber the of volumes that can be loaded onto the memory of a GPU. When trackingduring post-processing in this work, we used 32 volumes simultaneously. fDLC accurately finds neural correspondence across individuals
Having shown that fDLC performs well at identifying neurons within the sameindividual, we wanted to address its capability to identify neurons across differentanimals. Identifying corresponding neurons across individuals is crucial for studying thenervous system. However, finding neural correspondence across individuals is morechallenging than within an individual because there is variability in neuronal positionfrom both the animal’s movement as well as from development. To evaluate the fDLCmodel’s performance at finding neural correspondence across individuals, we applied thesame synthetically-trained fDLC model to a set of 11 manually annotated NeuroPALworms.NeuroPAL worms contain genetically encoded multicolor fluorescently labeled neurallandmarks to aid neural identification [1]. For each of the 11 Neuropal recordings,neurons were automatically segmented and manually annotated based on the neuron’sposition and color features as described in [1] (see Fig. 4A,B). Across the 11 animals, ahuman assigned a ground-truth label to a mean of 43% of segmented head neurons,providing approximately 58 labeled neurons per animal (Fig. 4C). The remainingneurons were not confidently labeled and thus not included in this analysis. We selectedas template the recording that contained the largest of confidently labeled humanannotated neurons. We evaluated our model by comparing its predicted correspondencebetween neurons in the other 10 test datasets and this template, using only positioninformation. All 11 ground-truth recordings were held-out in that they were notinvolved in the generation of the synthetic data that had been used to train the model.We applied the synthetically-trained fDLC model to each pair of held-out NeuroPALtest and template recordings and calculated the accuracy as the fraction of labeledneurons present in both the test and the template that was correctly matched. Acrossthe 10 pairs of NeuroPAL recordings using position information alone, the fDLC modelhad an accuracy of 65.8%, significantly higher than the CPD method’s accuracy of54.7% ( p = 0 . ig 4. fDLC model finds neural correspondence across individuals. (A)Fluorescence image shows neuronal nuclei of a NeuroPAL worm. A single optical slice isshown from an optical stack. (Scale bar, 10 µm ). Genetically encoded color labels inNeuropal animals aid ground truth manual neural identification [1] and are used here toevaluate performance. Black dots indicate neurons found via automatic segmentation.(B) Locations of all segmented neurons from A. Neurons that additionally have a humanannotated label are shown in green. Those that a human was unable to label are red.(C) Number of segmented neurons (mean 133.6) and subset of those that were givenhuman annotations (mean 57.5) is shown for 11 NeuroPAL individuals. Box plot showsmedian and interquartile range. (D) Pipeline to evaluate fDLC model performanceacross NeuroPAL individual is shown. Predicted labels are compared with humanannotated labels to compute accuracy. (E) Performance of the fDLC model and CPD isshown evaluated on NeuroPAL recordings using position information alone. Accuracy isthe fraction of labeled neurons present in both test and template that are correctlymatched. Performance is evaluated on 10 pairs of 11 recordings, where the template isalways the same (Worm A). ( p = 0 . p = 0 . i in the test recording, the fDLC model returns the probability withwhich that neuron corresponds to each possible neuron j in the template, p ij . Wewondered whether we could better use the likelihood information about potentialmatches generated by the algorithm. A Hungarian algorithm finds the most probablematch by considering all p ij s for all neurons in the test. By default we use this bestmatch in evaluating performance. But the p ij s also provide the user with a list ofalternative matches ranked by their likelihood. We therefore also assessed the accuracyfor the top 3 most likely candidate neurons. In this context, we define accuracy as thefraction of neurons for which the correct match is included in the list of the top 3candidates. As before, the denominator is taken to be the number of ground-truthlabeled neurons that appear in both test and template. When considering the top 3neurons, the fDLC model achieves an accuracy of 84.4% using only position information. Validating on an alternative dataset
Data quality, selection criteria, human annotation, hardware and preprocessing can allvary from lab-to-lab making it challenging to directly compare methods. To validate ourmodel against different measurement conditions and to allow for a direct comparisonwith another recent method, we applied our fDLC model to a previously publisheddataset of 9 NeuroPAL individuals [2] (Fig. 4F). This public dataset used differentimaging hardware and conditions and was annotated by human experts from a differentgroup. On this public dataset, our method achieved 78.9% accuracy while CPD achieved59.4%. When assessing the top 3 candidate accuracy, the fDLC model performance was91.3%. The fDLC model performance was overall higher on the published datasetcompared to our newly collected dataset presented here. This suggests that our methodperforms well when applied to real-world datasets in the literature.
Table 4. Comparison of model performance on additional datasetMethod Accuracy N Reported inCPD 59% 8 This workCRF (open atlas) ≈
40% 9 [2]CRF (data driven atlas) 74% 9 [2]fDLC N indicates the number of template-test pairs used tocalculate accuracy. (CRF method uses an atlas as the template, whereas we randomlytake one of the 9 individuals and designate that as the template). CPD and fDLCperformance on this dataset are also shown in Fig. 4F.We further compared the fDLC model to a recent model called Conditional RandomFields (CRF) from [2]. We compared the reported performance of the CRF model on thepublished dataset to the performance of the fDLC model evaluated on the same dataset(Table 4). The CRF model has two variants an “open atlas” and a “data-driven atlas.”fDLC accuracy is notably higher than the reported CRF performance for either variant,although we are unable to test for statistical significance. We note the fDLC methodalso offers other advantages compared to the CRF approach in that the fDLC method isoptimized for speed and avoids the need to transform the worm into a canonicalcoordinate system. Taken together, we conclude that the fDLC model’s accuracycompares favorably to that of the CRF model while also providing other advantages.January 21, 2021 16/22 radeoff between performance and coverage Our model predicts correspondence for all pairs of neurons, but the confidence of thepredicted label varies by pair. Therefore, there is an inherent tradeoff between theaccuracy of matched neurons (performance) and the fraction of neurons matched(coverage).The probability p ij in Equation 1 (in Materials and methods) serves as anestimate of the confidence level. By default the fDLC model matches all neuronsregardless of confidence and, with the exception of Fig. 3, that is how performance isevaluated in this work. (In Fig. 3 a minimum confidence was imposed to better compareto the NeRVE approach, which also only matches a subset of neurons. Details aredescribed in methods section “Minimum confidence threshold.”)In practice, it is often desirable to focus on only those neurons that are matchedwith very high confidence. We therefore tried restricting the model to only includematches above certain confidence thresholds. As the confidence threshold increased,model accuracy increased at the cost of the number of labeled neurons (Fig. 4G). For athreshold of p = 0 .
05 we observed an increase in the accuracy on our Neuropal datasetfrom 65.8% to 73.0% while only reducing coverage from 100% to 78.0% of the totalneuron matches (intersection of test and template). By increasing the threshold to 0.99,the accuracy increases to 79.2% but now only 38.1% of the neurons are matched. Thisfeature of the model allows the experimenter to transparently tune the trade-offbetween prediction accuracy and coverage.
Incorporating color information
Our method only takes positional information as input to predict neural correspondence.However, when additional features are available, the position-based predictions from thefDLC model can be combined with predictions based on other features to improveoverall performance. As demonstrated in [1], adding color features from a NeuroPALstrain can reduce the ambiguity of predicting neural correspondence. We applied a verysimple color model to calculate the similarity of color features between neuron i in thetest recording to every possible neuron j in the template. The color model returnsmatching probabilities, p c ij based on the Kullback-Liebler divergence of the normalizedcolor spectra in a pair of candidate neurons (details described in Materials andmethods). The color model is run in parallel to the fDLC model (Fig. 5A). Overallmatching probabilities p all ij that incorporate both color and position information arecalculated by combining the color matching probabilities p c ij with the positionprobabilities p ij . The Hungarian algorithm is run on the combined matching algorithmto predict the best matches.Adding color information increased the fDLC model’s accuracy from 65.8% to 76.5%(Fig. 5B) when evaluated on our dataset, and improved the accuracy in every recordingpair. The top 3 candidate labels attained 92.4% accuracy. Accuracy was calculatedfrom a comparison to human ground truth labeling, as before.We chose a trivially simple color model in part to demonstrate the flexibility withwhich the fDLC model framework can integrate information about other features. Sinceour simple color model utilized no prior knowledge about the distributions of colors inthe worm, we would expect a more sophisticated color model, for example, the statisticalmodel used in [1], to do better. And indeed that model evaluated on a different datasetis reported to have a higher performance with color than our model on our dataset (86%reported accuracy in [1] compared to 77% for the fDLC evaluated here). But thatmodel also performs much worse than fDLC when both are restricted to use only neuralposition information (50% reported accuracy for [1] compared to 66% for the fDLC).Together, this suggests the fDLC model framework can take advantage of additionalfeature information like color and still perform well when such information is missing.January 21, 2021 17/22 ig 5. fDLC performance when incorporating color features (A) Pipeline toevaluate fDLC performance across animals with additional color features. A simple colormodel is added in parallel to the fDLC model to use both color and position informationfrom 11 NeuroPAL recordings. Accuracy is calculated from ground truth humanannotation and is the fraction of labeled neurons present in both test and template thatare correctly matched . Matching probabilities from the color and fDLC models arecombined to form the final matching probabilities. (B) Accuracy of the position-onlyfDLC model and the combined fDLC and color model are evaluated on 11 NeuroPALrecordings (same recordings as in Fig. 4). p = 5 . × − , Wilcoxon signed rank test.January 21, 2021 18/22 iscussion The fDLC model finds neural correspondence within and across individuals with anaccuracy that compares favorably to other methods. The model focuses primarily onidentifying neural correspondence using position information alone. The fDLC modelframework also makes it easy to integrate other features. We demonstrated that colorinformation could be added by integrating the fDLC model with a simple color model toincrease overall accuracy. We expect that performance would improve further with amore sophisticated color model that takes into account the statistics of the colors in aNeuroPAL worm [1].The fDLC model framework offers a number of additional advantages beyondaccuracy. First, it is versatile and general. The same pre-trained model performed wellat both tracking neurons within a freely moving individual across time and at findingneural correspondence across different individuals. It achieved even higher accuracy ona publicly accessible dataset acquired on different hardware with different imagingconditions from a different group. This suggests that the framework should beapplicable to many real-world datasets.Second, the model reports the relative confidence in its estimate of each neuron’scorrespondence. An experimenter can therefore tune the overall accuracy of the model’spredictions, at the cost of leaving some neurons unmatched, by simply setting a desiredminimum confidence. Similarly, the model also provides probability estimates of allpossible matches for each neuron, not just the most likely. This allows an experimenterto consider a collection of possible matches such as the top 3.In contrast to previous methods, an advantage of the fDLC method is that it doesnot require the worm to be straightened, axis aligned, or otherwise transformed into acanonical coordinate system. This eliminates an error-prone and often manual step.Instead, the fDLC model finds neural correspondence directly from neural positioninformation even in worms that are in different poses or orientations.Importantly, the model is trained entirely on synthetic data, which avoids the needfor large experimentally acquired ground truth datasets to train the artificial neuralnetwork. Acquiring ground truth neural correspondence in
C. elegans is time consuming,error prone, and often requires manual hand annotation. The ability to train the fDLCmodel with synthetic data derived from measurements alleviates this bottleneck andmakes the model attractive for use with other organisms with stereotyped nervoussystems where ground truth datasets are similarly challenging to acquire.The model is also fast and finds neural correspondence of a new neural configurationin 10 ms. This speed is sufficient to keep up with real-time calcium imaging applicationsand will be valuable for future efforts combining large scale population calcium imagingwith closed loop targeted optogenetic stimulation in freely moving animals [9–12]. Wenote that to be used in a real-time closed loop application, our fDLC model would needto be combined with faster segmentation algorithms because current segmentationalgorithms are too slow for real-time use. Because segmentation can be easilyparalellized, we expect that faster segmentation algorithms will be developed soon.Many of the advantages listed here stem from the fDLC model’s use of thetransformer architecture [30] in combination with supervised learning. The transformerarchitecture, with its origins in natural language processing, is well suited to find spatialrelationships within a configuration of neurons. By using supervised learning onempirically-derived synthetic training data of animals in a variety of different poses andorientations, the model is forced to learn relative spatial features within the neuronsthat are informative for finding neural correspondence across many postures andconditions. Finally, the transformer architecture leverages recent advances in GPUparallel processing for speed and efficiency, which will help pave the way for real-time,closed-loop optogenetic experiments in freely moving worms.January 21, 2021 19/22 cknowledgments
We thank Ev Yemini and Oliver Hobert of Columbia University for strain OH15262. Weacknowledge productive discussions with John Murray of University of Pennsylvania.This work used computing resources from the Princeton Institute for ComputationalScience and Engineering. Research reported in this work was supported by the SimonsFoundation under awards SCGB
Additional information
Author contributions
Xinwei Yu, Conceptualization, Software, Formal analysis, Investigation, Methodology,Visualization, Validation, Writing-original draft, Writing—review and editing; MatthewS. Creamer, Investigation, Collected Data, Writing—review and editing; FrancescoRandi, Resources, Designed optics and related software libraries, Writing—review andediting; Anuj Sharma, Resources, Writing—review and editing, Performed alltransgenics; Scott W. Linderman, Conceptualization, Funding acquisition,Writing—review and editing; Andrew M Leifer, Conceptualization, Supervision,Funding acquisition, Writing—original draft, Project administration, Writing—reviewand editing
References
1. Yemini E, Lin A, Nejatbakhsh A, Varol E, Sun R, Mena GE, et al. NeuroPAL: Amulticolor atlas for whole-brain neuronal identification in C. Elegans. Cell.2020;doi:10.1016/j.cell.2020.12.012.2. Chaudhary S, Lee SA, Li Y, Patel DS, Lu H. Automated Annotation of cellidentities in dense cellular images. bioRxiv. 2020; p. 2020.03.10.986356.doi:10.1101/2020.03.10.986356.3. The structure of the nervous system of the nematode
Caenorhabditis elegans .Philosophical Transactions of the Royal Society of London B, Biological Sciences.1986;314(1165):1–340. doi:10.1098/rstb.1986.0056.4. Sulston JE. Post-embryonic development in the ventral cord of Caenorhabditiselegans. Philosophical Transactions of the Royal Society of London Series B,Biological Sciences. 1976;275(938):287–297. doi:10.1098/rstb.1976.0084.5. Witvliet D, Mulcahy B, Mitchell JK, Meirovitch Y, Berger DR, Wu Y, et al.Connectomes across development reveal principles of brain maturation in C.elegans. bioRxiv. 2020;doi:10.1101/2020.04.30.066209.6. Venkatachalam V, Ji N, Wang X, Clark C, Mitchell JK, Klein M, et al.Pan-neuronal imaging in roaming
Caenorhabditis elegans . Proceedings of theNational Academy of Sciences. 2016;113(8):E1082–E1088.doi:10.1073/pnas.1507109113.January 21, 2021 20/22. Nguyen JP, Shipley FB, Linder AN, Plummer GS, Liu M, Setru SU, et al.Whole-brain calcium imaging with cellular resolution in freely behaving
Caenorhabditis elegans . Proceedings of the National Academy of Sciences.2016;113(8):E1074–E1081. doi:10.1073/pnas.1507110112.8. Lagache T, Hanson A, Fairhall A, Yuste R. Robust single neuron tracking ofcalcium imaging in behaving Hydra. bioRxiv. 2020; p. 2020.06.22.165696.doi:10.1101/2020.06.22.165696.9. Leifer AM, Fang-Yen C, Gershow M, Alkema MJ, Samuel ADT. OptogeneticManipulation of Neural Activity in Freely Moving Caenorhabditis Elegans.Nature Methods. 2011;8(2):147–152. doi:10.1038/nmeth.1554.10. Stirman JN, Crane MM, Husson SJ, Wabnig S, Schultheis C, Gottschalk A, et al.Real-Time Multimodal Optical Control of Neurons and Muscles in FreelyBehaving Caenorhabditis Elegans. Nature methods. 2011;8(2):153–158.doi:10.1038/nmeth.1555.11. Kocabas A, Shen CH, Guo ZV, Ramanathan S. Controlling Interneuron Activityin Caenorhabditis Elegans to Evoke Chemotactic Behaviour. Nature.2012;490(7419):273–277. doi:10.1038/nature11431.12. Shipley FB, Clark CM, Alkema MJ, Leifer AM. Simultaneous OptogeneticManipulation and Calcium Imaging in Freely Moving C. Elegans. Frontiers inNeural Circuits. 2014;8:28. doi:10.3389/fncir.2014.00028.13. Hammarlund M, Hobert O, Miller DM, Sestan N. The CeNGEN Project: TheComplete Gene Expression Map of an Entire Nervous System. Neuron.2018;99(3):430–433. doi:10.1016/j.neuron.2018.07.042.14. Toyoshima Y, Wu S, Kanamori M, Sato H, Jang MS, Oe S, et al. An AnnotationDataset Facilitates Automatic Annotation of Whole-Brain Activity Imaging of C.Elegans. bioRxiv. 2019; p. 698241. doi:10.1101/698241.15. Bronstein AM, al e. Rock, Paper, and Scissors: Extrinsic vs. Intrinsic Similarityof Non-Rigid Shapes; 2007.16. Lagache T, Lansdell B, Tang J, Yuste R, Fairhall A. Tracking Activity In aDeformable Nervous System With Motion Correction and Point-Set Registration.bioRxiv. 2018; p. 373035. doi:10.1101/373035.17. Myronenko A, Xubo Song. Point Set Registration: Coherent Point Drift. IEEETransactions on Pattern Analysis and Machine Intelligence.2010;32(12):2262–2275. doi:10.1109/TPAMI.2010.46.18. Ma J, Zhao J, Yuille AL. Non-Rigid Point Set Registration by Preserving Globaland Local Structures. IEEE Transactions on Image Processing. 2016;25(1):53–64.doi:10.1109/TIP.2015.2467217.19. Wen C, Miura T, Fujie Y, Teramoto T, Ishihara T, Kimura KD.Deep-learning-based flexible pipeline for segmenting and tracking cells in 3Dimage time series for whole brain imaging. bioRxiv. 2018;doi:10.1101/385567.20. Nguyen JP, Linder AN, Plummer GS, Shaevitz JW, Leifer AM. AutomaticallyTracking Neurons in a Moving and Deforming Brain. PLOS ComputationalBiology. 2017;13(5):e1005517. doi:10.1371/journal.pcbi.1005517.January 21, 2021 21/221. Jian B, Vemuri BC. Robust Point Set Registration Using Gaussian MixtureModels. IEEE Transactions on Pattern Analysis and Machine Intelligence.2011;33(8):1633–1645. doi:10.1109/TPAMI.2010.223.22. Bubnis G, Ban S, DiFranco MD, Kato S. A Probabilistic Atlas for CellIdentification. arXiv:190309227 [q-bio]. 2019;.23. Varol E, Nejatbakhsh A, Sun R, Mena G, Yemini E, Hobert O, et al. StatisticalAtlas of C. elegans Neurons. In: Medical Image Computing and ComputerAssisted Intervention – MICCAI 2020. Springer International Publishing; 2020. p.119–129.24. Nejatbakhsh A, Varol E, Yemini E, Venkatachalam V, Lin A, Samuel ADT, et al.Extracting neural signals from semi-immobilized animals with deformablenon-negative matrix factorization; 2020.25. Nejatbakhsh A, Varol E. Neuron Matching in C. Elegans With RobustApproximate Linear Regression Without Correspondence. In: Proceedings of theIEEE/CVF Winter Conference on Applications of Computer Vision; 2021. p.2837–2846.26. Bishop CM. Pattern recognition and machine learning. 2006;.27. Long F, Peng H, Liu X, Kim SK, Myers E. A 3D Digital Atlas of C. Elegans andIts Application to Single-Cell Analyses. Nature methods. 2009;6(9):667–672.doi:10.1038/nmeth.1366.28. Lee J, Mitelut C, Shokri H, Kinsella I, Dethe N, Wu S, et al. YASS: Yet AnotherSpike Sorter applied to large-scale multi-electrode array recordings in primateretina. bioRxiv. 2020;.29. Mathis MW, Mathis A. Deep learning tools for the measurement of animalbehavior in neuroscience. Current opinion in neurobiology. 2020;60:1–11.30. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al.Attention is All You Need; 2017.Available from: https://arxiv.org/pdf/1706.03762.pdf .31. Peng H, Long F, Liu X, Kim SK, Myers EW. Straightening CaenorhabditisElegans Images. Bioinformatics. 2008;24(2):234.32. Paszke A, Gross S, Chintala S, Chanan G, Yang E, Devito Z, et al. Automaticdifferentiation in PyTorch; 2017.33. He K, Zhang X, Ren S, Sun J. Deep Residual Learning for Image Recognition. In:2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Las Vegas, NV, USA: IEEE; 2016. p. 770–778. Available from: http://ieeexplore.ieee.org/document/7780459/http://ieeexplore.ieee.org/document/7780459/