Fast reconstruction of single-shot wide-angle diffraction images through deep learning
Thomas Stielow, Robin Schmidt, Christian Peltz, Thomas Fennel, Stefan Scheel
FFast reconstruction of single-shot wide-anglediffraction images through deep learning
T Stielow, R Schmidt, C Peltz, T Fennel and S Scheel
Institut f¨ur Physik, Universit¨at Rostock, Albert-Einstein-Straße 23–24, D-18059Rostock, GermanyE-mail: [email protected]
E-mail: [email protected]
E-mail: [email protected]
Abstract.
Single-shot X-ray imaging of short-lived nanostructures such as clustersand nanoparticles near a phase transition or non-crystalizing objects such as largeproteins and viruses is currently the most elegant method for characterizing theirstructure. Using hard X-ray radiation provides scattering images that encode two-dimensional projections, which can be combined to identify the full three-dimensionalobject structure from multiple identical samples. Wide-angle scattering using XUVor soft X-rays, despite yielding lower resolution, provides three-dimensional structuralinformation in a single shot and has opened routes towards the characterization ofnon-reproducible objects in the gas phase. The retrieval of the structural informationcontained in wide-angle scattering images is highly non-trivial, and currently noefficient rigorous algorithm is known. Here we show that deep learning networks,trained with simulated scattering data, allow for fast and accurate reconstruction ofshape and orientation of nanoparticles from experimental images. The gain in speedcompared to conventional retrieval techniques opens the route for automated structurereconstruction algorithms capable of real-time discrimination and pre-identification ofnanostructures in scattering experiments with high repetition rate – thus representingthe enabling technology for fast femtosecond nanocrystallography.
Submitted to:
Machine Learning: Science and Technology a r X i v : . [ phy s i c s . a t m - c l u s ] A p r ast reconstruction of single-shot wide-angle diffraction images through deep learning
1. Introduction
Sources of soft and hard X-rays with large photon flux such as free electron lasers[1, 2] have enabled the high-resolution imaging of unsupported nanosystems such asviruses [3], helium droplets [4, 5, 6], rare-gas clusters [7], or metallic nanoparticles [8].For reproducible samples, a set of scattering images taken for different orientationsin the small-angle scattering limit, each delivering a two-dimensional projection ofthe object’s density, can be used to retrieve its three-dimensional structure usingconventional reconstruction algorithms [9, 10]. Short-lived and non-reproducible objects,however, elude the repeated acquisition of several images required for the tomographicreconstruction from small-angle scattering. The partial three-dimensional informationcontained in wide-angle scattering enables to overcome this main deficiency, for the prizeof an even more complicated inversion problem [5, 8, 11]. Finding a fast reconstructionmethod thus remains the major obstacle for exploiting the potential of wide-anglescattering for genuine single-shot structure characterization.Two aspects distinguish wide-angle from small-angle scattering. First, theprojection approximation is no longer valid due to substantial contributions of thelongitudinal component of the wavevector, such that the curvature of the Ewald sphereplays an important role. Second, for the wavelength range for which wide-anglescattering is realized, the refractive index of most materials deviates substantially fromunity, and hence multiple scattering, absorption, backpropagating waves, and refractionall have to be accounted for. Currently, all these constraints can only be met bysolving the full three-dimensional scattering problem by, e.g., finite-difference time-domain (FDTD) methods, gridless discrete-dipole approximation (DDA) techniques,or appropriate approximate solutions based on multislice Fourier transform (MSFT)techniques [6, 12].These methods allow, for an assumed geometry model of the nanoparticle, todescribe their wide-angle scattering patterns. However, the determination of thegeometry from those patterns is highly nontrivial, as there exists no rigorous inversionmethod. Consequently, the existing applications of wide-angle scattering had to bebased on a parametrized geometry model whose parameters can be determined by aniterative forward fit, e.g. by an ensemble of optimization trajectories in phase space asemployed in the simplex Monte Carlo approach in [6]. Because for every iteration step,at least one forward simulation has to be performed, this method is only applicableto a small data set and for a sufficiently simple geometry model [6]. Hence, there isan urgent need for efficient reconstruction methods that can be used in real time for alarge data set. Here we present a proof-of-principle study that shows, by consideringicosahedra, that a neural network, trained with simulated scattering images, establishesa high-quality reconstruction method of particle size and orientation with unprecedentedspeed.Machine learning using neural networks, and deep learning in particular, are ideallysuited for the extraction of structural parameters from scattering images, as this is ast reconstruction of single-shot wide-angle diffraction images through deep learning
2. Experimental and theoretical framework
The choice of icosahedra as test objects was motivated by their ubiquity in nature,ranging from viruses [3, 9, 27, 10] to rare-gas [28] and metal clusters [8]. Focussingon the last example, which already constitutes a wide-angle scenario (see figure 1a),we compute scattering images of icosahedral silver clusters with a range of sizes andspatial orientations using an MSFT algorithm [6], representing the training data. Theemployed generalized multi-sliced Fourier transform (MSFT) algorithm includes aneffective treatment of absorption [8].We numerically generated ∼ ,
000 individual scattering images for clusters witha uniform size distribution (30 nm ≤ R ≤
160 nm) and random orientations in thefundamental domain of the icosahedron, which represent perfect theoretical data. Whenrepresenting spatial rotations by unit quaternions (see Appendix A for details), thefundamental domain of the icosahedron has the shape of a dodecahedron in imaginaryspace [29], which is simply connected and possesses a natural metric, unlike other lowerdimensional representations such as Euler angles. Furthermore, any arbitrary rotationin the axis-angle representation may be projected into this domain by determining thedistance to the closest quaternion associated to one of the symmetry rotations.The ultimate goal is to analyze realistic scattering data that are obtained fromexperiments with various imperfections. Therefore, the neural network should not betrained solely using the ideal theoretical data, but also with appropriately augmenteddata [30, 31, 32]. In that way, the network will be trained to focus on physically ast reconstruction of single-shot wide-angle diffraction images through deep learning Figure 1. Wide-angle scattering setup.
Nanoparticles of icosahedral shape andvarying size and orientation are interrogated by soft X-ray radiation from a free electronlaser (FEL) [8]. The resulting wide-angle scattering pattern is simulated by employingan MSFT algorithm. The Convolutional Neural Network (CNN) computes a parameterrepresentation of size and spatial orientation of the nanoparticle from the scatteringimage. experiment simulationnoise hole crop sim. exp.
Figure 2. Image Augmentations.
The ideal theoretical scattering images areaugmented by image defects that account for experimental imperfections. They canbe divided into quality defects such as noise or blur, and experiment-specific featuressuch as the central hole and the limited size of the detector. We randomly combine allimage effects, and in addition apply them in a well-defined order to generate imagesthat closely resemble experimental data. relevant features. Here, we augment our data by adding noise, blur, spatial offsets, acentral hole, as well as blind spots and cropping of the images. These augmentationfeatures address common experimental imperfections associated with photon noise,limited detector resolution, source-point and beam-pointing jitter, transmission of thehigh-intensity primary beam, and detector segmentation and finite size (see figure 2).These augmentations (see Appendix D for details) increase the training set 11-fold. ast reconstruction of single-shot wide-angle diffraction images through deep learning
3. Network Design and Training
Based upon the quaternion representation of rotations, we can find a unique, bijectiveparametrization of arbitrary icosahedra of uniform density by a vector with five scalarentries: the four components of the quaternions and the radius of the circumsphere. Onthe other hand, the associated scattering patterns can be understood as two-dimensionalsingle-channel (or grayscale) images with a size of 128x128 pixels. For the regressiontask of assigning a parameter vector to an image, we utilize the ResNet architectureof a convolutional neural network with 34 layers. This architecture was found to bothoffer the complexity needed for learning the specified task while also keeping the numberof free parameters as low as possible to counteract overfitting and minimizing trainingtimes. + + + + + + + + + + + + + + + + I npu tI m age 7 x C on v o l u t i on , S t r i de22 x M a x P oo li ng G l oba l A v e r age P oo li ng F u ll y C onne c t ed O u t pu t V e c t o r x x x x
64 32 x x
64 16 x x
128 8 x x
256 4 x x
512 512 5 + Plus OperationIdentity Shortcut1x1 Convolution
Figure 3. Network Design
The network architecture is based upon the ResNetscheme described in [33]. The first two layers are used for an initial lateral dimensionreduction, while all other convolution filters are encapsuled in residual blocks. Eachblock consists of two consecutive convolutional layers whose output is added to theinitial input and, by this, implement a residual calculation. The last residual block isfed into an average pooling operation compressing the tensor shape into a vector with512 entries from which the output five-vector is calculated by a fully connected layer.
The exact network structure is visualized in Figure 3. It is composed of an initial7x7 convolution layer with stride 2 and 64 filters, followed by a 2x2 max pooling forlateral dimension reductions, feeding into four stacks of 3, 4, 6 and 4 consecutive residualblocks as defined in [33] with 64, 128, 256 and 512 filters each, respectively. The firstconvolution of each stack has stride 2 and consequently the corresponding identityshortcut is implemented by 1x1 convolutions. Behind the final residual layer followsa global average pooling operation, reducing the tensor size to a 512-vector from whichthe terminal 5-vector is computed by a fully connected layer. All activation functionsare set to tanh.Upon training, the network parameters were optimized to minimize the mean- ast reconstruction of single-shot wide-angle diffraction images through deep learning a MaxMean % prediction error f r a c t i on b scatter simulations ( ) p r ed i c t i one rr o r ( % )
5k 10k 20k25k
MCS set size NN Figure 4. Performance Validation. a
The performance of the network is validatedby computing the relative mean prediction error (blue bins). The majority of the meanprediction errors is below 1%, with a minor quantile observing large errors that aremainly attributed to unphysical predictions. The maximal errors in each parameter(red bins) also remain mostly below 1%. b Evaluated on 30 random samples of thetraining set, the Monte-Carlo-Simplex algorithm reached a median accuracy (blue line)of 0 .
37% within 50 iterations. Each iteration step is estimated to require on average fourscattering simulations, the horizontal axis denotes the number of scattering simulationsduring a single MCS run. The blue shaded area covers the 90% quantile of the best-fitruns for each image, visualizing the error margin of the MCS method. The performanceof neural networks trained with subsets of the training set of different sizes are markedby red dots with shaded areas for the respective 90% quantiles. The correspondingtraining times are also expressed in units of scatter simulations. In order to achievecomparable median accuracy and error margin, the number of required scatteringsimulations for the training of a neural network corresponds to only a few MCSreconstructions.
For comparison with established forward fitting methods, we also reconstructed30 images of the test set with a state-of-the-art Monte Carlo simplex procedure, asused in [6]. For each image, the reconstruction started from 50 random initial pointsin parameter space, and 50 simplex iteration steps have been taken. The convergence ast reconstruction of single-shot wide-angle diffraction images through deep learning ∼ . ∼
17 h for the completetraining set. The subsequent training of the network on the complete data set takesadditional ∼ ∼
21 h or the equivalent of 31k scatteringimage calculations to yield the ready-to-use neural network. After successful training,the evaluation of a single image takes only 5 ms, which is a negligible numerical effortcompared to that required during forward fitting. Hence, already for a small number ofto be reconstructed images, the computational overhead for training data set generationand actual network training can be compensated by the exceptional reconstruction speedwhilst still providing reconstruction results of comparable accuracy (see figure 4b).We demonstrate the network’s ability in recognizing structures in imperfectexperimental images by applying it to data taken from [8], where two icosahedral clustershave been identified among the images (left column in figure 5). The reconstructedsize and spatial orientation (central column in figure 5) are validated to reproducethe experimental scattering images (right column in figure 5) with very high accuracy.Our results match the reconstructed data published in [8], with the remaining smalldeviations having to be attributed to the approximations used in the forward scatteringapproach rather than the neural network.
4. Summary
We have shown that, using a deep-learning technique based on augmented theoreticalscattering data, neural networks enable the accurate and fast reconstruction ofwide-angle scattering images of individual icosahedral nanostructures. Our resultsdemonstrate that a network, which has only been trained on theoretical data, can beemployed for the analysis of experimental scattering data, with image processing timeson the millisecond time scale. The neural network reaches the same level of accuracyas established forward fitting methods based on Monte Carlo Simplex algorithms.Although the reconstruction of a single image using the neural network is orders ofmagnitude faster than the direct optimization, the generation of the training data andsubsequent training of the network requires a substantial constant overhead. However,the reconstruction speed of the network compensates the extra effort after only a fewscattering images.Motivated by the performance of this method, we anticipate that a generalizationto a wide range of particle morphologies will be feasible. Combined with pre-selectionalgorithms as utilized in [6], this may evolve into a classification tool for archimedean ast reconstruction of single-shot wide-angle diffraction images through deep learning Experiment Reconstruction Validation a r =
151 nm b r =
144 nm
Figure 5. Reconstruction from experimental data.
Experimental datafrom [8] (left column, permitted by Creative Commons CC-BY 4.0 license ( http://creativecommons.org/licenses/by/4.0/ )), are evaluated by the neural network.The reconstructed spatial orientation in the laser propagation direction is shown inthe middle column. The reconstructed radii are very close to those given in [8]. Thetheoretical scattering patterns associated with these reconstructions reproduce theexperimental images very well, including low-intensity features (right column). Theintensity of the theoretical patterns is clipped at a maximum intensity in order tomimic the nonlinear response of the detector. bodies. The envisaged combination of identification of arbitrary three-dimensionalshapes with short processing times is anticipated to represent the enabling technology fora fully automated analysis of scattering data and real-time reconstruction of ultrafastnanoscale dynamics probed at the next generation of X-ray light sources with highrepetition rate — with major implications for a broad range of physical, chemical andbiological applications.
Acknowledgments
T.S. gratefully acknowledges a scholarship from “Evangelisches Studienwerk Villigst”.S.S. acknowledges financial support from Deutsche Forschungsgemeinschaft (DFG) viathe SPP 1929 “Giant interactions in Rydberg systems”. T.F. acknowledges financialsupport from the DFG via the Heisenberg program (No. 398382624) and the BMBF(project 05K16HRB). This work was partially supported by the NEISS project of theEuropean Social Fund (ESF) (reference ESF/14-BM-A55-0007/19) and the Ministry ofEducation, Science and Culture of Mecklenburg-Vorpommern, Germany. The authorsthank Dr. Katharina Sander for sharing her insights into the MSFT technique and Dr.Ingo Barke for providing information on the experimental data aquisition. ast reconstruction of single-shot wide-angle diffraction images through deep learning Data Availability
The data that support the findings of this study are available from the correspondingauthor upon reasonable request.
Appendix A. Icosahedral Symmetry
The icosahedron is one of the five platonic solids and is spanned by 20 equilateraltriangle faces, intersecting with 30 edges and twelve corners. It possesses three-foldrotation symmetry axes C about the center-of-mass of each triangle, two-fold axes C about the center of each edge and five-fold axes about each corner, which together formthe icosahedral rotation group I . The 60 symmetry rotations imply that any rotation ofa body with icosahedral symmetry is 60-fold degenerate. Hence, the mapping of three-dimensional rotation representations, such as Euler-angle or axis-angle representations,to icosahedral orientations are not unique, but have to be constrained in their parameterrange. The fundamental domain of rotations has an exceptionally simple form inquaternion representation of rotations, where it forms a dodecahedron in imaginaryspace [29].Quaternions Q are the four-dimensional extension of the complex numbers withthree imaginary units i , j and k fulfilling the relations i = j = k = ijk = − − ji. With real coefficients q i , any quaternion may be written as q = q +i q +j q +k q .Imaginary quaternions ( q ≡
0) are isomorphic to the space R , implying that all vectors a = ( a , a , a ) can be represented by quaternions as q a = i a + j a + k a . The sumof two vectors then translates into the sum of two quaternions, whereas the quaternionproduct contains both the scalar product of two cartesian vectors (in its real part) andtheir cross product (in the imaginary part). The rotation by an angle α of any vector a about a unit vector n can thus be expressed by the product of the quaternion q a withthe unit quaternion q rot = cos( α/
2) + sin( α/
2) ( n x i + n y j + n z k). Hence, any rotation canbe projected into the fundamental domain by applying all inverse symmetry rotationsand selecting the one yielding the smallest rotation angle. For the training of a neuralnetwork, the quaternion representation has the additional advantage of providing auseful metric for the distance between rotations. Appendix B. Dataset Generation
The scattering patterns used for training are created by using the MSFT algorithmdescribed in [8]. In accordance with the experiment described therein, we simulate thescattering of ultra-short XUV pulses with wavelength λ = 13 . a abs = 12 . ast reconstruction of single-shot wide-angle diffraction images through deep learning ×
128 pixelswith random background noise, which is stored as a grayscale image. The rotationquaternions are sampled uniformly from the fundamental domain, while the size of theclusters range from 30 to 160 nm. With this procedure, a dataset of 25 058 imageshas been generated, one fifth of which has been reserved for validation during training.Another set of 5000 is generated for final testing indepentently from the training process.
Appendix C. Error Measures
The five parameters of our icosahedron representation reconstructed by the neuralnetwork cover very different parameter ranges. For training purposes, both theparameters of the radius and the real part of the rotation quaternion are linearly scaled tothe interval [0 , Appendix D. Image Augmentation
Prior to training the neural network, image augmentation is applied to the dataset. Theaugmentation is performed by applying eleven different filters to each ideal scatteringpattern, and randomly adding the newly generated images to the training set. Thesefilters can be divided into five groups: trivial, noise, blur, cropping and successiveapplication. The trivial filter is the identity mapping, leaving the image unchanged.Noise is applied both with uniform distribution with a randomly chosen intensity uptohalf the maximum signal, changing every pixel by a random margin as well as salt-and-pepper statistics, where random pixels are set to either minimal or maximal signal.Blurring is performed by convoluting with a Gaussian kernel with randomly chosenradius of upto five pixels, and by jitter distortion. Cropping filters delete differentparts of the image, mainly to account for the characteristics of real detectors. Imagesare either center-cropped for limited detector size, a central hole of random radius isdeleted to simulate the shadow of a beam dump, images are shifted or uneven detectorsensitivity is simulated by attenuating parts of the image. Finally, we both randomly ast reconstruction of single-shot wide-angle diffraction images through deep learning
References [1] Chapman H N, Barty A, Bogan M J, Boutet S, Frank M, Hau-Riege S P, Marchesini S, WoodsB W, Bajt S, Benner W H et al.
Nature Physics Science et al.
Nature et al.
Science et al.
Nature communications et al. Physical review letters et al.
New Journal of Physics Nature communications et al. Physical review letters
Optics Express Nature
Annual Review of Physical Chemistry [13] Hinton G E and Salakhutdinov R R 2006 Science
Nature arXiv preprint arXiv:1903.10563 [16] Wang L 2016
Physical Review B Nature Physics Nature Physics Physical Review Letters
Science
Nature Physics npj Quantum Information Scientific reports arXiv preprint arXiv:1902.06876 [25] Laanait N, Romero J, Yin J, Young M T, Treichler S, Starchenko V, Borisevich A, Sergeev A andMatheson M 2019 arXiv preprint arXiv:1909.11150 [26] Zimmermann J, Langbehn B, Cucini R, Di Fraia M, Finetti P, LaForge A C, Nishiyama T,Ovcharenko Y, Piseri P, Plekan O et al. Physical Review E et al. Nature Photonics The Journal of chemical physics Le Journal de Physique Colloques C3–165 ast reconstruction of single-shot wide-angle diffraction images through deep learning [30] Krizhevsky A, Sutskever I and Hinton G E 2012 Imagenet classification with deep convolutionalneural networks Advances in neural information processing systems pp 1097–1105[31] Dosovitskiy A, Springenberg J T, Riedmiller M and Brox T 2014 Discriminative unsupervisedfeature learning with convolutional neural networks
Advances in Neural Information ProcessingSystems pp 766–774[32] Perez L and Wang J 2017 arXiv preprint arXiv:1712.04621 [33] He K, Zhang X, Ren S and Sun J 2016 Deep residual learning for image recognition