Reconstruction of nanoscale particles from single-shot wide-angle FEL diffractions patterns with physics-informed neural networks
MModel-independent reconstruction of nanoscale particles from single-shotwide-angle FEL diffractions patterns with physics-informed neural networks
Thomas Stielow and Stefan Scheel
Institut für Physik, Universität Rostock, Albert-Einstein-Straße 23–24, D-18059 Rostock, Germany (Dated: January 25, 2021)Single-shot wide-angle diffraction imaging is a widely used method to investigate the structureof non-crystallizing objects such as nanoclusters, large proteins or even viruses. Its main advan-tage is that information about the three-dimensional structure of the object is already containedin a single image. This makes it useful for the reconstruction of fragile and non-reproducible par-ticles without the need for tomographic measurements. However, currently there is no efficientnumerical inversion algorithm available that is capable of determining the object’s structure in realtime. Neural networks, on the other hand, excel in image processing tasks suited for such purpose.Here we show how a physics-informed deep neural network can be used to reconstruct completethree-dimensional object models on a voxel grid from single two-dimensional wide-angle scatteringpatterns. We demonstrate its universal reconstruction capabilities for silver nanoclusters, where thenetwork uncovers novel geometric structures that reproduce the experimental scattering data withvery high precision.
I. INTRODUCTION
The imaging of systems of nanometer size is ofgreat importance for many branches in biological,chemical and physical sciences. The laws of waveoptics demand the usage of wavelengths in the x-ray regime. However, the large energy carried byeach photon rapidly damages such delicate sam-ples [1]. The deterioration of the sample duringthe imaging process can be avoided if the sampleimage is generated on a much shorter timescalethan that on which the destruction process, e.g.Coulomb explosion [2], occurs. This requirement isfulfilled by imaging using high-intensity ultra-shortfemtosecond pulses, as produced by free electronlasers [3, 4]. Since the object’s features and thewavelength are comparable, the resulting imageis dominated by scattering features and, in orderto reveal the underlying real-space image, furtherprocessing is necessary [3]. To date, improvementsin object reconstruction allowed the investigationof ever smaller unsupported nanosystems such asviruses [5–7], helium droplets [8–10], rare-gas clus-ters [11], or metallic nanoparticles [12].For very short wavelengths, i.e. hard x-rays, thescattering occurs predominantly at small angles.In this case, the scattering process can be under-stood in the Fraunhofer limit, and the scatteringfield is the two-dimensional Fourier transform ofthe projected electron density. A subsequent iter-ative phase retrieval then allows to reconstruct thistwo-dimensional density projection with high fi-delity from a single scattering pattern [5, 13]. Fur-ther, individual scattering images of an ensembleof identical objects can be merged to obtain thethree-dimensional object density [6, 7, 14]. Fornon-reproducible targets, such tomographic tech-niques cannot be employed as only a single scat-tering image is available. In this situation, three-dimensional information can be extracted from wide-angle reflexes of the scattering pattern [15],which require longer wavelengths. Recent theo-retical works indicate the completeness of suchthree-dimensional information encoded in wide-angle scattering signals [16, 17]. Yet, they posea significantly more complicated inversion prob-lem compared to the small-angle reconstructionmethod [9, 12, 15]. Thus far, these reconstruc-tions mostly rely on iterative forward fitting meth-ods that are based on simulations of the scat-tering process of a suitably parametrized objectmodel [9, 10, 12]. While highly successful, the re-peated scattering simulations are computationallyexpensive and are restricted to the assumed objectmodel.Recent years have seen rapid development inimage processing and reconstruction techniquesbased on deep learning methods [18–20]. Theseconcepts have already found broad applications instatistical physics, particle and accelerator physics[21–25], material sciences [21, 26–28], as well as forapproximating solutions to differential equations[29, 30]. In diffractive imaging, deep learning tech-niques have been explored for the efficient recon-struction of both small-angle and wide-angle im-ages. Phase retrieval and subsequent Fourier inver-sion with convolutional neural networks has beendemonstrated for simulated small-angle scatteringpatterns [31], and have been expanded to three di-mensions for the reconstruction of object densitiesfrom complete Fourier volumes [32]. On the ex-perimental side, the pre-selection of automaticallyrecorded scattering patterns into various categorieshas been implemented as a classification task [10],and generative learning helped to reveal commonfeatures in patterns connected to object classes andimaging artifacts [33]. Recently, shape and orienta-tion of icosahedral silver nanoclusters were recon-structed from experimental wide-angle scatteringpatterns using a neural network trained solely on a r X i v : . [ phy s i c s . d a t a - a n ] J a n simulated training data [34]. This was achieved byutilizing a convolutional neural network that, com-bined with data augmentation techniques, is capa-ble of processing experimental images that sufferfrom a variety of physically relevant artifacts anddefects.In this article, we present a neural network ap-proach for reconstructing shape and orientation ofarbitrary nanoclusters from single-shot wide-anglescattering images that does not depend on theparametrization of the object model. Instead, weuse a voxel model of the object density similar tothat used in small-angle scattering [32]. For that,an encoder-decoder architecture is employed thatrealizes the transition from the two-dimensionalimage to the three-dimensional object space. Theinterpolation beyond the underlying training dataset is improved by implementing physics-informedlearning, in which the theoretical scattering modelitself is included in the loss function.The article is organized as follows. In Sec. II,we briefly review the scattering simulation methodthat is based on the multi-slice Fourier transform(MSFT) algorithm, and we introduce the construc-tion of the basis set and its augmentations. Thedesign of the neural network including the physics-informed training scheme is presented in Sec. III.Its capabilities and limits are discussed in Sec. IV,followed by the evaluation of experimental data inSec. V and some concluding remarks in Sec. VI. II. MODELLING AND SIMULATINGSCATTERING OF SILVERNANOCLUSTERS
Scattering experiments with light in the x-rayregime are known to reveal structure informationsuch as geometric shapes, spatial orientation andsize of nanoparticles, in some cases also their in-ternal structure [6, 35]. Here, we focus on the re-construction of silver nanoparticles that had beenilluminated with soft x-rays from a free electronlaser with wavelength λ = 13 . nm. At this wave-length, scattering off these clusters with sizes be-tween ... nm can then be regarded as in thewide-angle limit. The nanoparticles are producedby a magnetron sputtering source in a cluster beammachine. The generated stream of nanoclustersshows a wide range of shapes and sizes, meaningthat the particle shapes occur to a certain extentrandomly. Moreover, each individual experimentis non-reproducible as the Coulomb explosion pre-vents multiple illumination. It is also known thatthe particles emerging from the source have notyet relaxed to an equilibrium state at the time ofillumination, hence geometric structures such asicosahedra have been found [12, 34] that are notexpected to be stable for large particle sizes.Due to the lack of a direct inversion algorithm for the reconstruction of geometric information froma single-shot wide-angle scattering image, compar-ative methods such as forward fitting have beenemployed [10, 12, 36]. The theoretical scatteringpatterns are generated using a multi-slice Fouriertransform (MSFT) algorithm that takes absorp-tion into account but neglects multiple scatter-ing events as well as momentum transfer to thenanoparticle. Because of the short absorptionlength of . nm in silver, this algorithm gives veryaccurate results. Most importantly, it can be rep-resented as a linear tensor operation which makesit suitable for efficient parallel computation.For an efficient implementation of a reconstruc-tion algorithm, a suitable parametrization of theobject is needed. Typically, this means a restric-tion of the class of object shapes to a finite setof highly symmetric base solids with relatively fewdegrees of freedom. For nanoparticles out of equi-librium, however, transient shapes need not neces-sarily be highly symmetric. This in turn implies atrade-off between reconstruction accuracy and nu-merical efficiency. Already in the case of only fewparameters, neural networks outperform conven-tional forward fitting based on Monte Carlo sim-plex methods [34], which is expected to becomeeven more prominent with increasing number ofdegrees of freedom. The limiting case is to rep-resent the object on a discrete three-dimensionalgrid; such representations are commonly used forthe reconstruction of real-space objects from a se-ries of images using deep neural networks [37]. Inthe realm of scattering physics, this representationhas been employed for the reconstruction of a re-producible nanoparticle from a three-dimensionalscattering pattern that has been compiled from aseries of small-angle scattering images [32]. Weshow here that the discretized three-dimensionalobject can be reconstructed from a single wide-angle scattering pattern using deep neural net-works. A. Object classes for training the neuralnetwork
The training of a neural network requires asuitably chosen set of training data. In orderto account for a large variety of (convex) ob-ject shapes that still contain some symmetry, wechoose a basis set that contains all Platonic solids,all Archimedean solids (except the snub dodeca-hedron), the decahedron and truncated twinnedtetrahedron, as well as spheres and convex polyhe-dra with fully random vertices. This set is depictedin Fig. 1. Further, these base solids have beenstretched and squashed along one of their sym-metry axes, and have been randomly scaled androtated for maximum flexibility. Despite the stillfinite number of objects, it is expected that a large
FIG. 1. The basis set of 21 shapes contains all Platonic and Archimedean solids (except for the snub dodeca-hedron) and, additionally, the decahedron, the truncated twinned tetrahedron, spheres and polyhedra with fullyrandomized vertices, defined by enclosing 50 random space points. enough portion of object space is covered, and thatthe neural network is capable of interpolating ef-ficiently between them. Note, however, that someof the included objects (such as the tetrahedron)are highly unlikely to ever occur in an experimentbut are included nonetheless.
B. Scattering simulation
The training data are obtained numericallyby employing the MSFT scattering framework.All objects have been rasterized on a three-dimensional grid of × × points andare stored as flattened png images. For each ob-ject, the corresponding scattering intensity pat- tern is calculated using the MSFT algorithm. Thelateral dimensions of the object are padded to × pixels upon simulation, and the result-ing real transfer momentum space covers × pixels. As the transverse intensity decreases ex-ponentially away from the image center, the in-tensity values are scaled logarithmically in orderto preserve important scattering features at largetransfer momenta. In addition, in order to simu-late detector dark noise, a random constant offsetis being applied before scaling. Each image is thennormalized and stored as a png image. As theobject rasterization as well as the MSFT scatter-ing calculations require considerable computationtimes, a data set of 140 000 objects has been pre-generated and stored. C. Simulating experimental artifacts byimage augmentation
The theoretical training data contains the max-imal amount of information regarding the lightscattering off a nanoparticle allowed by scatteringand detection physics. However, in experimentalsituations, technical limitations often obscure someof the information necessary to, e.g. identify theshape of a particle. For example, all images containa central hole that protects the detector from thecentral unscattered beam. This is such a promi-nent artifact that a neural network is very likely toregard this as the most important feature, whereasthe information about the shape of the particle re-sides in the outer fringes of the scattering pattern.Therefore, such defects have to be included in thetraining of the network from the outset.In Ref. [34] it was demonstrated that data aug-mentation techniques can be used to simulate thesemeasurement artifacts and to train a neural net-work that is robust against such effects. Weextend this augmentation approach by introduc-ing additional filters and on-the-fly augmentation.Rather than pre-generating a set of augmented im-ages, here we apply random augmentations at eachtraining step. Hence, every time the network ispresented with the same data point, a random aug-mentation filter is being selected, which helps toprevent overfitting.Examples of all used augmentation filters areshown in Fig. 2. The augmentation functions uni-form noise , salt & pepper noise , shift , central hole and blind spot have been implemented as describedin Ref. [34]. The cropping filter has been modifiedto simultaneously apply rectangular and circularcropping masks with random sizes. The Poisso-nian noise filter has been implemented by addinga random matrix sampled from a Poissonian dis-tribution with variance λ = 1 . to the normal-ized scattering pattern, while the shot noise filtermultiplies the scattering pattern with a randomPoissonian matrix with variance λ = 10 r +1 where r is an uniform random number from the inter-val [0 , . These filters account for the Poissonianbackground counts as well as the discrete nature ofphotons in the low-intensity limit. The simulatedexperiment filter is implemented by a consecutiveapplication of the shot noise , shift , blind spot , de-tector saturation , central hole , cropping , and shift filters. III. DESIGN AND TRAINING OF THESCATTERING RECONSTRUCTIONNETWORK
In classical image processing, the task of creatinga three-dimensional model from one or more two- simulation simulatedexperimentuniform noise Poissonian noise salt &pepper noiseshot noise shift central holecropping detectorsaturation blind spot
FIG. 2. Image augmentation is used to feed the neuralnetwork scattering patterns with various defects to in-crease its prediction robustness. Each simulated scat-tering pattern (top left) is modified with one of thenine fundamental filters (bottom × square) or acombination of them (top right) to mimic experimen-tally obtained scattering patterns. dimensional images is a well-known problem thatcan be efficiently tackled using neural networks[37, 38]. The reconstruction of a discretized three-dimensional object from a two-dimensional single-channel image requires a dimension conversion,which is commonly solved with encoder-decoderarchitectures. In this case, the input image is pro-jected into a latent space from which the conver-sion into the output space is performed. Whenimplementing multi-view reconstructions of macro-scopic objects from photographic images, addi-tional recurrent elements within the latent spaceare required [37].The architecture we developed for single-shotscattering reconstructions is depicted in Fig. 3. InputConv Layer 2D, 64, 7x7, Stride 2Max Pool, 2x2Conv Layer 2D, 128Conv Layer 2D, 128Max Pool, 2x2Conv Layer 2D, 256Conv Layer 2D, 256Max Pool, 2x2Conv Layer 2D, 512Conv Layer 2D, 512Max Pool, 2x2Conv Layer 2D, 1024Conv Layer 2D, 1024Max Pool, 2x2Conv Layer 2D, 2048Conv Layer 2D, 2048Max Pool, 2x2Flatten Dense, 2048 ReshapeUpsampling, 2x2x2Conv Layer 3D, 1024Conv Layer 3D, 1024Upsampling, 2x2x2Conv Layer 3D, 512Conv Layer 3D, 512Upsampling, 2x2x2Conv Layer 3D, 256Conv Layer 3D, 256Upsampling, 2x2x2Conv Layer 3D, 128Conv Layer 3D, 128Upsampling, 2x2x2Conv Layer 3D, 64Conv Layer 3D, 64Upsampling, 2x2x2Conv Layer 3D, 64Conv Layer 3D, 64Conv 3D, 1, 1x1x1, SigmoidOutput
FIG. 3. Neural network with encoder-decoder struc-ture. The encoder (left column) consists of five residualblocks each containing two consecutive 2D convolutionlayers with × kernels. The filter size is doubledwith each residual block, while the lateral dimensionsare reduced by pooling layers. The latent space (bot-tom) is one-dimensional and is further connected by adense layer. After reshaping, the decoder (right col-umn) applies × × upsampling operations followedby two 3D convolution layers each. All convolutionlayers are regularized with a dropout ratio of 0.2 andbatch normalization is applied before the leaky ReLUactivation. The encoder section of the network in the left col-umn is constructed as a residual convolutional lat-eral compressor. An initial pickup layer with 7 × × × to × × elements. Following that is a sequenceof five residual blocks, each halving the lateral sizefurther while doubling the number of filters. Everyresidual block consists of two consecutive convolu-tion layers as well as an identity shortcut which are combined by a summation layer [39]. Eachconvolution layer has a kernel size of × and isactivated by the leaky ReLU function lReLU( x ) = (cid:26) x if x > , . x otherwise. (1)after regularization by batch normalization anddropout. Within the latent space, an additionalfully connected layer with 2048 neurons is em-ployed. The decoder (right column of Fig. 3) is de-signed in reverse with upsampling layers instead ofpooling and three-dimensional convolution layers.Unlike the encoder, the decoder does not employresidual operations and is instead of linear struc-ture, as residual connections were found to offerno improvement in the prediction quality while in-creasing the training time significantly. The finalcompression of the filter dimension into the out-put tensor of size × × × is performed bya three-dimensional convolution operation with a × × kernel and sigmoid activation, as the out-put tensor is of binary character. The full networkhas now approximately 200 million free parame-ters. A. Physics-Informed Learning
Classical supervised learning consists of com-paring the predictions p made by the neural net-work on the training inputs x to the correspondingground truth targets y , and calculating a loss scoreas illustrated in Fig. 4(a). However, a straightfor-ward implementation of this idea is unfeasible inour situation. Silver has a rather short absorptionlength of . nm at the relevant photon energies,which is much shorter than the cluster diametersthat range from 63 to 320 nm. As a result, theincoming radiation does not penetrate the entirenanoparticle and, in particular, has no access tothose parts of the scattering object that are fur-thest away from the radiation source. This is turnmeans that a significant part of the object doesnot contribute to the scattering image. However,the penalizing loss function forces the neural net-work to attempt to reconstruct those regions forwhich very little information is contained in theinput image. Hence, the neural network is eitherforced to complete the object from symmetric pro-jections (which is indeed observed to some degree),or is driven into significant overfitting.In order to ensure that the neural network learnsonly from physically relevant information, we pro-pose the calculation of a loss score in scatteredspace, which is shown in Fig. 4(b). Instead of com-paring the prediction p with the target y directlyby the mean binary crossentropy H ( y , p ) = 1 N N (cid:88) i,j,k =1 (cid:104) y i,j,k log( p i,j,k )+ (1 − y i,j,k ) log(1 − p i,j,k ) (cid:105) , (2)both p and y are used as inputs for the MSFTalgorithm, and the loss is calculated by the meansquared distance of the resulting scattering pat-terns, scaled logarithmically. This so called scatterloss can be expressed as L s ( y , p ) = 1 M M (cid:88) i,j =1 (cid:20) log (cid:18)(cid:12)(cid:12)(cid:12) E MSFT ( y ) i,j (cid:12)(cid:12)(cid:12) + (cid:15) (cid:19) − log (cid:18)(cid:12)(cid:12)(cid:12) E MSFT ( p ) i,j (cid:12)(cid:12)(cid:12) + (cid:15) (cid:19)(cid:21) , (3)with some chosen noise level (cid:15) , and where E MSFT is the normalized electric-field distribution obtainedby the MSFT algorithm. In this way, the traininggoal of the neural network is moved from predictingthe real-space shape of an object to generating anobject volume that reproduces the input scatteringpattern.Although the terminal layer of the neural net-work is sigmoid activated, this activation does notenforce the binary nature of our particle model.Therefore, we introduce an additional regulariza-tion term to the loss function (3) by penalizingnon-binary object voxels with the binary loss func-tion L b ( y , p ) = 1 N N (cid:88) i,j,k =1 ( p i,j,k ) (1 − p i,j,k ) . (4)The binary loss function (4) is weighted by a factor . compared to the scatter loss (3) to ensure op-timal convergence. This is an instance of physics-informed learning [29, 30] where physical laws areincorporated in the training function. ( a ) Classical Supervised Learning ( b ) Physical Loss Learning
FIG. 4. In classical supervised learning (a), the loss score is determined by the binary crossentropy between thenetwork prediction and the target entry of each data pair. In the physical learning scheme (b), the loss scoreis calculated within the scatter space rather than the object space. This is done by simulating the scatteringpattern of both the network prediction as well as the target object, and calculating their mean squared difference(scatter loss). To enforce the binary nature of the object model, an additional regularization function (binaryloss) is applied to the prediction.
B. Network Training
The neural network was implemented andtrained within the TensorFlow 2.3.1 Keras frame-work and Python 3.6.6. The binary loss regular-ization and scatter loss were both implemented asTensorFlow functions, thereby enabling backprop-agation on GPU devices during training. We havechosen the adaptive moments (ADAM) gradientdescent optimizer for optimal convergence. Thetraining dataset was pre-generated, and scatteringpatterns were stored as png images, while objectdensities were rescaled and saved as × × numpy arrays to minimize hardware access andprocessing times. The data set contains 140 000samples in total and has been split into a trainingand a validation set with a ratio . The trainingset was re-shuffled before each epoch, and data wasread from the hard drive and randomly augmentedon-the-fly. The validation data was not augmentedin order to monitor the peak reconstruction ca-pability. Training was performed on a dedicatedGPU server with two Intel Xeon Silver 4126 CPUsand four Nvidia RTX2080ti GPUs. Distribution ofeach training batch over all four GPUs allowed amaximum batch size of 32. We found the optimaltraining duration to be 50 epochs for sufficient con-vergence. The corresponding learning curve of thenetwork used throughout this manuscript is shownin Fig. 5. The total training time accumulated to63h. trainingvalidation0 20 40 60 80 1000.0000.0010.0020.0030.0040.0050.006 epoch l o ss FIG. 5. The training loss of the neural network con-verges within 50 full cycles of the training set to a nearhalt. The loss on the validation set follows a similartrajectory, but is consistently smaller than the trainingloss, due to the absence of augmentations and regular-ization.
A consistent result over different training runsfrom independent random initializations couldonly be achieved by applying regularization in ev-ery layer. Batch normalization counteracts thetendency to no-object predictions. Simultaneously,dropout regularization prevents the neural networkfrom converging to non-physical predictions, whichmay produce similar scattering patterns but arenon-binary point clouds in object space that do notcorrespond to solid convex (or at least star-shaped) bodies. The combined effect of these regularizationis that the training loss in Fig. 5 shows no over-fitting compared to the validation loss. However,this cannot rule out the possibility of overfitting toeither the underlying set of solids or the augmen-tations used.
IV. PREDICTION CAPABILITY OF THENEURAL NETWORK ( a ) input pattern simulationground truth prediction ( b ) input pattern simulationground truth prediction FIG. 6. Scattering patterns and real-space objectshapes are reproduced by the neural network for mostobjects of the test set, such as the rhombicosidodeca-hedron (a). For some examples, the predicted object isreconstructed without the far side or sports a shallowdome in the beam direction (b), both of which have nosignificant impact on the scattering pattern.
During training of the neural network, we bench-marked its prediction capabilities on the valida-tion set which was generated from the same objectspace as the training set. In order to test its in-terpolating power, we created an additional testset of object data unknown to the network. Thesebodies were created by truncating the previouslyscaled and stretched object classes along randomsymmetry axes, thus breaking some of the symme-tries and creating new shapes. In this way, a totalof 1000 new objects were created.In the majority of cases, the neural network iscapable of detecting the new deformations. An ex-ample is shown in Fig. 6(a), corresponding to aheavily truncated rhombicosidodecahedron. Theobject prediction of the neural network (bottomright) closely resembles the ground truth of the ob-ject (bottom left), while their scattering patternsare nearly indistinguishable (top row in Fig. 6(a)).This implies that, due to its physics-informedtraining, the neural network does not merely inter-polate between known shapes, but rather composesan hitherto unknown object from facets associatedwith distinct reflexes in the scattering pattern.Conversely, this also implies that objects areonly constructed from real-space features that im-pact the scattering pattern. An example is shownin Fig. 6(b), where two significant effects can beobserved. First, the far side of the predicted ob-ject (bottom right) is featureless. This was ex-pected because of the strong absorption of the in-coming radiation which prevents a significant con-tribution from the scattering off these regions. Thesame effect was also observed on the validationset and even the training set. The neural net-work then either cuts off the far side completely,or replaces it with a smooth droplet shape. Sec-ond, the flat front facet of the input object (bottomleft) is being converted into a shallow dome. Sur-faces oriented close to perpendicular with respectto the incoming beam are particularly difficult toreconstruct, as the strongest associated reflexes ap-pear in the backscattering direction. These reflexeswould only be observable in a π detector config-uration, for which the MSFT algorithm does notgive reliable results. A simplified two-dimensionalmodel of this effect is shown in Fig. 7, where atriangular shaped dome (orange object) is beingadded to a flat facet of a trapezoidal base (blackobject). The corresponding one-dimensional scat-tering intensity profiles are almost indistinguish-able, in particular given a finite detector resolu-tion.Delicate features of the real-space object appearat large transverse transfer momentum, that is,at large detection angles. During augmentation,this region is quite often cropped, giving the neu-ral network the incentive to gather its informationfrom the inner regions of small transfer momen-tum. This restriction is motivated by the limiteddetection angle of typical experiments. In order - - q ( nm - ) s c a tt e r e d i n t e n s i t y FIG. 7. The scattered intensity signals of a truncatedtriangle with a footprint of . nm and of the sameobject equipped with a shallow tip of of its heightare almost identical. to understand the effect of cropping, we show inFig. 8 the reconstructed images from the same in-put data pair for a series of ever smaller detectionangles. As expected, with smaller available trans-fer momenta, the reconstruction quality decreasesbecause information on sharp features is lost. As aconsequence, edges and corners appear smoothed,while the facets are still recognizable. crop. input prediction pred. scatter FIG. 8. Shrinking the angular span of the detectionrange (left column) leads to the loss of high-frequencyinformation in the scattering pattern. Thus, the neu-ral network predictions (central column) appear lesscrisp, and corners and edges are rounded, while thecorresponding scatter simulation (right column) stillmatches the input pattern within the input region(framed by gray mask).
V. NEURAL NETWORKRECONSTRUCTION OF EXPERIMENTALDATA
So far, the neural network has been tested onsynthetic data that capture the relevant scatter-ing physics, and that have been augmented inorder to mimic expected experimental artifacts.The trained network is now being used to recon-struct experimental single-shot wide-angle scatter-ing data of silver nanoclusters [12]. Our choicehas been informed by the existence of classical re-constructions using forward fitting methods withparametrized polyhedra, which provides the op-portunity for direct comparison between the meth-ods.In Fig. 9, we compare the reconstructed nan-oclusters from both the forward fitting (green ob-jects in central column) and the neural network(grey objects in central column). The left col-umn contains the experimental data from Ref. [12],whereas the right column depicts the simulatedscattering profiles of the neural network predic-tions. We have explicitly shown the detection areato indicate the region which the neural networkaims to reproduce. As discussed above, due tothe lack of available large transfer momenta, thereconstructed objects by the neural network havesmoother edges and corners. In comparison, theforward fit assumes the existence of sharp featureswhich is unsupported given only the available in-formation. Also, as expected from the above dis-cussion, the far sides of the reconstructed objectsare either missing or being replaced by a smoothdroplet, and shallow domes appear on their fronts.Notwithstanding, the main facets are being re-constructed reliably, resulting in structures withglobally similar features. However, the neural net-work predicts more elongated bodies which repro-duce the softer interference patterns in the scat-tering reflexes. Moreover, the reconstructed bod-ies are no longer perfectly symmetric as assumedin the parametrized model, but show local defectsthat break certain symmetries. Note that the ex-perimental scattering patterns show distinct asym-metries which can only be explained be relaxingthe requirement of symmetric bodies. As a result,the scattering patterns simulated from the neuralnetwork predictions match the experimentally ob-tained patterns almost perfectly.A particularly striking result is the star-shapedpattern with five-fold symmetry (5th row inFig. 9). Previously, this has been attributed toan icosahedron, as this was the only shape in theparametrized model with the correct symmetry.Instead, the neural network predicts an elongateddecahedron of similar size. A regular decahedronwould produce a scattering pattern with ten-foldsymmetry. However, the elongation of a decahe-
FIG. 9. The neural network is tested with the ex-perimental scattering patterns from Ref. [12] (left col-umn, permitted by Creative Commons CC-BY 4.0license ( http://creativecommons.org/licenses/by/4.0/) ) and the corresponding shape candidates ob-tained by forward fitting (green solids). The neuralnetwork predictions are shown in gray. The simu-lated scattering patterns (right column) show excellentagreement with the input pattern inside the availableregion (confined by the gray masks). dron breaks that symmetry in the scattering pat-tern, resulting in two distinct sets of five reflexes0each with different intensities. The extracted elon-gation factor along the symmetry axis is approxi-mately . . This result shows that the neural net-work reconstruction can help in detecting shapes ofnanoparticles that would not have been expectedfrom equilibrium cluster physics. VI. SUMMARY
We have developed a neural network that is ca-pable of reconstructing three-dimensional objectdensities of silver nanoclusters from single-shotwide-angle scattering patterns. By including thescattering physics into the penalty function usedfor training of the neural network, the networklearned to construct an object that produces theexact same scattering pattern rather than to rec-ognize previously seen object classes. This impliesthat the neural network is able to reliably recon-struct object shapes outside its training set. It isthus able to predict transient nanocluster struc-tures that would not be expected from equilibrium cluster formation theory. Our method is not re-stricted to the example of silver nanoclusters dis-cussed here. The same network structure can beused for any system for which the scattering prop-erties (such as absorption lengths) are known, anda numerical algorithm to generate training dataexists. Combined with the fast evaluation times inthe µ s range, this paves the way to a fully auto-mated reconstruction of the complete structure ofnanoparticles from single-shot wide-angle scatter-ing images in real time. ACKNOWLEDGMENTS
T. S. acknowledges financial support from“Evangelisches Studienwerk Villigst”. This workwas partially funded by the European Social Fund(ESF) and the Ministry of Education, Science andCulture of Mecklenburg-Western Pomerania (Ger-many) within the project
NEISS – Neural Extrac-tion of Information, Structure and Symmetry inImages under grant no ESF/14-BM-A55-0007/19. [1] R. Neutze, R. Wouts, D. Van der Spoel, E. Weck-ert, and J. Hajdu, Nature , 752 (2000).[2] Z. Jurek, G. Faigel, and M. Tegze, The Euro-pean Physical Journal D-Atomic, Molecular, Op-tical and Plasma Physics , 217 (2004).[3] H. N. Chapman, A. Barty, M. J. Bogan, S. Boutet,M. Frank, S. P. Hau-Riege, S. Marchesini, B. W.Woods, S. Bajt, W. H. Benner, et al. , NaturePhysics , 839 (2006).[4] K. Gaffney and H. Chapman, Science , 1444(2007).[5] M. M. Seibert, T. Ekeberg, F. R. Maia, M. Svenda,J. Andreasson, O. Jönsson, D. Odić, B. Iwan,A. Rocker, D. Westphal, et al. , Nature , 78(2011).[6] T. Ekeberg, M. Svenda, C. Abergel, F. R. Maia,V. Seltzer, J.-M. Claverie, M. Hantke, O. Jönsson,C. Nettelblad, G. Van Der Schot, et al. , Physicalreview letters , 098102 (2015).[7] K. Ayyer, A. J. Morgan, A. Aquila, H. DeMirci,B. G. Hogue, R. A. Kirian, P. L. Xavier, C. H.Yoon, H. N. Chapman, and A. Barty, Optics Ex-press , 37816 (2019).[8] L. F. Gomez, K. R. Ferguson, J. P. Cryan, C. Ba-cellar, R. M. P. Tanyag, C. Jones, S. Schorb,D. Anielski, A. Belkacem, C. Bernando, et al. , Sci-ence , 906 (2014).[9] D. Rupp, N. Monserud, B. Langbehn, M. Sauppe,J. Zimmermann, Y. Ovcharenko, T. Möller,F. Frassetto, L. Poletto, A. Trabattoni, et al. , Na-ture communications , 493 (2017).[10] B. Langbehn, K. Sander, Y. Ovcharenko, C. Peltz,A. Clark, M. Coreno, R. Cucini, M. Drabbels,P. Finetti, M. Di Fraia, et al. , Physical review let-ters , 255301 (2018). [11] D. Rupp, M. Adolph, T. Gorkhover, S. Schorb,D. Wolter, R. Hartmann, N. Kimmel, C. Reich,T. Feigl, A. De Castro, et al. , New Journal ofPhysics , 055016 (2012).[12] I. Barke, H. Hartmut, D. Rupp, L. Flückiger,M. Sauppe, M. Adolph, S. Schorb, C. Bostedt,R. Treusch, C. Peltz, S. Bartling, T. Fennel, K.-H.Meiwes-Broes, and T. Möller, Nature communi-cations , 6187 (2015).[13] S. Marchesini, H. He, H. N. Chapman, S. P.Hau-Riege, A. Noy, M. R. Howells, U. Weierstall,and J. C. Spence, Physical Review B , 140101(2003).[14] K. Ayyer, P. L. Xavier, J. Bielecki, Z. Shen,B. J. Daurer, A. K. Samanta, S. Awel, R. Bean,A. Barty, M. Bergemann, et al. , Optica , 15(2021).[15] K. S. Raines, S. Salha, R. L. Sandberg, H. Jiang,J. A. Rodríguez, B. P. Fahimian, H. C. Kapteyn,J. Du, and J. Miao, Nature , 214 (2010).[16] K. Engel, arXiv preprint arXiv:2008.00935(2020).[17] K. Engel and B. Laasch, arXiv preprintarXiv:2009.10414 (2020).[18] G. E. Hinton and R. R. Salakhutdinov, Science , 504 (2006).[19] Y. LeCun, Y. Bengio, and G. Hinton, Nature ,436 (2015).[20] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning (MIT Press, 2016) .[21] G. Carleo, I. Cirac, K. Cranmer, L. Daudet,M. Schuld, N. Tishby, L. Vogt-Maranto, andL. Zdeborová, arXiv preprint arXiv:1903.10563(2019). [22] P. Baldi, P. Sadowski, and D. Whiteson, Naturecommunications , 1 (2014).[23] G. Kasieczka, T. Plehn, M. Russell, and T. Schell,Journal of High Energy Physics , 6 (2017).[24] G. Kasieczka, T. Plehn, A. Butter, K. Cran-mer, D. Debnath, B. M. Dillon, M. Fairbairn,D. A. Faroughy, W. Fedorko, C. Gay, L. Gouskos,J. F. Kamenik, P. T. Komiske, S. Leiss, A. Lis-ter, S. Macaluso, E. M. Metodiev, L. Moore,B. Nachman, K. Nordstrom, J. Pearkes, H. Qu,Y. Rath, M. Rieger, D. Shih, J. M. Thompson,and S. Varma, SciPost Phys. , 14 (2019).[25] N. Meinert, Search for Rare b to Open-CharmTwo-Body Decays of Baryons at LHCb , Ph.D. the-sis, Rostock University (DE) (2020).[26] N. Laanait, Q. He, and A. Y. Borisevich, arXivpreprint arXiv:1902.06876 (2019).[27] N. Laanait, J. Romero, J. Yin, M. T. Young,S. Treichler, V. Starchenko, A. Borisevich,A. Sergeev, and M. Matheson, arXiv preprintarXiv:1909.11150 (2019).[28] C. Chen, Y. Zuo, and W. Ye, Nat Comput Sci ,46 (2021).[29] M. Raissi, P. Perdikaris, and G. E. Karni-adakis, Journal of Computational Physics ,686 (2019).[30] M. Raissi, A. Yazdani, and G. E. Karniadakis,Science , 1026 (2020).[31] M. J. Cherukara, Y. S. Nashed, and R. J. Harder,Scientific reports , 1 (2018). [32] H. Chan, Y. S. Nashed, S. Kandel,S. Hruszkewycz, S. Sankaranarayanan, R. J.Harder, and M. J. Cherukara, arXiv preprintarXiv:2006.09441 (2020).[33] J. Zimmermann, B. Langbehn, R. Cucini,M. Di Fraia, P. Finetti, A. C. LaForge,T. Nishiyama, Y. Ovcharenko, P. Piseri,O. Plekan, et al. , Physical Review E ,063309 (2019).[34] T. Stielow, R. Schmidt, C. Peltz, T. Fennel, andS. Scheel, Machine Learning: Science and Tech-nology , 045007 (2020).[35] K. Sander, Reconstruction Methods for Single-shotDiffractive Imaging of Free Nanostructures withUltrashort X-ray and XUV Laser Pulses , Ph.D.thesis, Rostock University (DE) (2018).[36] K. Sander, C. Peltz, C. Varin, S. Scheel,T. Brabec, and T. Fennel, Journal of Physics B:Atomic, Molecular and Optical Physics , 204004(2015).[37] C. B. Choy, D. Xu, J. Gwak, K. Chen, andS. Savarese, in European conference on computervision (Springer, 2016) pp. 628–644.[38] C. Niu, J. Li, and K. Xu, in
Proceedings of theIEEE conference on computer vision and patternrecognition (2018) pp. 4521–4529.[39] K. He, X. Zhang, S. Ren, and J. Sun, in