Adaptive 3D convolutional neural network-based reconstruction method for 3D coherent diffraction imaging
AAdaptive ML for CDI
Adaptive 3D convolutional neural network-based reconstruction methodfor 3D coherent diffraction imaging
Alexander Scheinker a) and Reeju Pokharel b) Los Alamos National Laboratory, Los Alamos, NM 87545, USA (Dated: 25 August 2020)
We present a novel adaptive machine-learning based approach for reconstructing three-dimensional (3D) crystals fromcoherent diffraction imaging (CDI). We represent the crystals using spherical harmonics (SH) and generate correspond-ing synthetic diffraction patterns. We utilize 3D convolutional neural networks (CNN) to learn a mapping between 3Ddiffraction volumes and the SH which describe the boundary of the physical volumes from which they were generated.We use the 3D CNN-predicted SH coefficients as the initial guesses which are then fine tuned using adaptive modelindependent feedback for improved accuracy.
I. INTRODUCTION
In-situ characterization of detailed 3D views of defects andinterfaces and their evolution at the mesoscale (few nm - hun-dreds of µ m) are required to develop microstructure-awarephysics-based models and to design advanced materials withtailored properties . Coherent Diffraction Imaging (CDI) isa non-destructive X-ray imaging technique providing 3D mea-surements of sample electron density at nm resolution fromwhich sub nm atomic displacement estimates can be calcu-lated to understand deformation for µ m sized non-crystallinespecimens .CDI has now been applied for a wide range of scientificstudies including biology, physics and engineering . CDI hasbeen used to measure the 3D structures of individual viruses and bacteria , for imaging quantum dots , to image redblood cells infected with malaria , for 3D imaging of humanchromosomes , for imaging the 3D electron denisty of largeZnO crystals , using only partially coherent light , for mea-suring 3D lattice distortions due to defect structures in ion-implanted nano-crystals , and for measuring dislocations inpolycrystalline samples .The CDI technique records only the intensity of the com-plex diffraction pattern originating from the illuminated sam-ple volume, in which all phase information is lost. If the phaseinformation in the diffraction signal could be measured, thena simple inverse Fourier transform would provide the 3D elec-tron density which generated the diffraction pattern. Many it-erative numerical methods for achieving phase retrieval havebeen developed which map measured diffraction patterns toelectron density . The existing CDI phase reconstructionmethods are sometimes very lengthy processes requiring ex-tensive fine tuning by expert users. Existing algorithms forinverting diffraction signals to produce a real-space image aresometimes brute-force and usually very computationally ex-pensive. Additionally, iterative phase retrieval algorithms re-quire expert knowledge, relying on a wide range of experi-ence and expertise in various fields. Standard methods aresensitive to small variation in diffraction signals and different a) Electronic mail: [email protected] b) Electronic mail: [email protected] users may produce inconsistent reconstructions depending onthe experience of the user and the choice of initial guessesfor the parameters while expert users are able to combine var-ious conventional algorithms such as error reduction, differ-ence map, shrinkwrap and hybrid input-output to be capableof exploiting the available frequency information in a CDImeasurement, utilizing robust treatments of measurement sig-nal noise . The challenges of iterative phase retrieval makeit a good candidate for utilizing machine learning methods.Although ML methods cannot substitute for traditional algo-rithms, they do have the potential to help with the speed ofobtaining reconstructions by providing an initial guess whichis then fine tuned by traditional methods to achieve accurateresults.Machine learning (ML) tools, such as deep neural net-works, have recently grown in popularity due to their abilityto learn input-output relationships of large complex systems.Neural networks have been used to speed up lattice quantumMonte Carlo simulations , for studying complicated manybody systems , for depth prediction in digital holography ,and combined with model-independent feedback for adap-tively controlling particle accelerator beams .Recently 2D convolutional neural networks (CNN) havebeen utilized for speeding up diffraction-based reconstruc-tions. CNNs have been developed to directly map 2D diffrac-tion amplitude measurements to the amplitudes and phasesof the 2D objects from which they originated, presenting anapproach for orders of magnitude faster 2D amplitude andphase reconstructions for CDI . CNNs have also been re-cently developed for orders of magnitude faster mapping ofelectron backscatter diffraction (EBSD) patterns to crystalorientations . In this work we utilize Tensorflow and the au-tomatic differentiation capabilities of the software package, arecent application of automatic differentiation to problem ofphase retrieval is given in . II. SUMMARY OF MAIN RESULTS
A graphical summary of the method proposed in this workis shown in Figure 1. We present an adaptive ML approachto the reconstruction of 3D object with uniform electron den-sities from synthetic diffraction patterns. Our adaptive MLframework utilizes a combination of a 3D convolutional neu- a r X i v : . [ phy s i c s . c o m p - ph ] A ug daptive ML for CDI 2ral network together with an ensemble of model-independentadaptive feedback agents to reconstruct 3D volumes basedonly on CDI diffraction measurements. The algorithm uses3D diffracted intensities as inputs and provides outputs in theform of spherical harmonics which describe the surfaces of the3D objects with uniform densities that generated the diffractedintensities. III. MATHEMATICAL BACKGROUND
Ideally, the goal of the the CDI measurements would beto record the complex diffracted scalar wavefield ψ ( w ) = | ψ ( w ) | exp [ i φ ( w )] , which is related to the Fourier transformof the electron density, ρ ( r ) of the sample, where r = ( x , y , z ) is the sample space and w = ( w x , w y , w z ) is reciprocal spacecoordinates, respectively. If such a measurement could bemade, the 3D electron density could be reconstructed by sim-ply performing an inverse Fourier transform. Unfortunately,when a coherent X-ray passes through a material with electrondensity ρ ( r ) , what is recorded on a detector is the intensity ofthe diffracted light, given by I ( w ) = ¨ ρ ( r ) ρ (cid:63) ( r ) exp [ i q ( r − r )] d r d r = ψ ( w ) ψ (cid:63) ( w )= | ψ ( w ) | exp [ i φ ( w )] exp [ − i φ ( w )]= | ψ ( w ) | , (1)with all of the phase information lost . Reconstructing ˆ ψ ( w ) requires lengthy phase retrieval algorithms which are typi-cally carried out after the experiments and performed by ex-pert users. A. Spherical harmonics shape descriptors
Our approach is to represent the unknown electron densityinside a 3D object by a collection of basis vectors in the formof spherical harmonics, which describe the surface that en-closes the volume of material of interest. Spherical harmon-ics are a generalization of the 1D Fourier series for represent-ing functions defined on the unit sphere. For any D >
0, theHilbert space of real square-integrable functions defined overthe interval x ∈ [ , D ] is defined as L [ , D ] = (cid:26) f ( x ) : ˆ D | f ( x ) | dx < ∞ (cid:27) (2)with the inner product of any f , g ∈ L [ , D ] defined as (cid:104) f ( x ) , g ( x ) (cid:105) = ˆ D f ( x ) g ( x ) dx . (3)Distance between functions in L [ , D ] is defined by the metric (cid:107) f − g (cid:107) = (cid:104) f − g , f − g (cid:105) = ˆ D | f ( x ) − g ( x ) | dx . (4) It is well known from Fourier analysis that any function f ( x ) ∈ L [ , D ] can be approximated arbitrarily closely by a linearcombination of the basis functions ϕ c , n ( x ) = cos (cid:18) π nxD (cid:19) , ϕ s , n ( x ) = sin (cid:18) π nxD (cid:19) , n ∈ N . (5)If a sequence of functions f N are defined as f N ( x ) = c + N ∑ n = [ c n ϕ c , n ( x ) + s n ϕ s , n ( x )] , (6) c = D (cid:104) f ( x ) , (cid:105) = D ˆ D f ( x ) dx , (7) c n > = D (cid:104) f ( x ) , ϕ c , n ( x ) (cid:105) = D ˆ D f ( x ) ϕ c , n ( x ) dx , (8) s n = D (cid:104) f ( x ) , ϕ s , n ( x ) (cid:105) = D ˆ D f ( x ) ϕ s , n ( x ) dx , (9)then lim N → ∞ (cid:107) f − f N (cid:107) = . (10)For any function s ( θ , φ ) defined on the surface of thesphere, where θ ∈ [ , π ] and φ ∈ [ , π ] are the spherical co-ordinates, the function can be approximated arbitrarily accu-rately with a representation of the form s N ( θ , φ ) = N ∑ l = l ∑ m = − l a ml Y ml ( θ , φ ) , (11)where the coefficients are found by the inner product a ml = (cid:104) s ( θ , φ ) , Y ml ( θ , φ ) (cid:105) = ˆ π ˆ π r ( θ , φ ) Y ml ∗ ( θ , φ ) sin ( θ ) d θ d φ . (12)By approximating in terms of the basis of spherical harmon-ics, we assume that we can find a star-convex approximationof a surface s ( θ , φ ) . IV. ADAPTIVE MACHINE LEARNING FOR PHASERETRIEVAL
To determine the unknown electron density ρ ( r , θ , φ ) , wemake the assumption that the electron density is non-zero onlywithin some compact set and that the density is uniform withinsome bounding surface ∂ ρ ( r , θ , φ ) = s ( θ , φ ) of the form ρ ( r , θ , φ ) = (cid:40) d , | r | ≤ s ( θ , φ ) , | r | ≥ s ( θ , φ ) . (13)Note that in this proof-of-concept work, we are consideringsolid objects of uniform density d without internal structures.Our 3D reconstruction approach is to find a set of coefficientsˆ a ml up to order l = N , y = ( ˆ a , . . . , ˆ a ml , . . . , ˆ a NN ) , which de-fine a surface that approximates s ( θ , φ ) by constructingˆ s ( θ , φ ) = N ∑ l = l ∑ m = − l ˆ a ml Y ml ( θ , φ ) , (14)daptive ML for CDI 3 | Ψ ( w )|
3D CNN y (0) ES initial shape r(θ,φ)
3D Fourier TransformAmplitude
Iterative ES dV w C(a ml ) = y (n+1) Iterativeloopshapeupdateprediction cost function FIG. 1. The 3D CNN’s output is used as the initial condition for ES tuning.
50 50 50 252525 5553×3×3 1×1×1normalizationlayer3D convolutionlayer| Ψ ( w )| y FIG. 2. Overview of the 3D CNN directly using the intensity of the Fourier transform as input with a final output of dimension 28 of thecoefficient of the even spherical harmonics Y ml for l ≤ y = ( y ,..., y ) = ( a , a − ,..., a ,..., a ) . which in turn defines an electron densityˆ ρ ( r , θ , φ ) = (cid:40) d , | r | ≤ ˆ s ( θ , φ ) , | r | ≥ ˆ s ( θ , φ ) , (15)that approximates ρ ( r , θ , φ ) . In order to find the appropri-ate spherical harmonics, we calculate the amplitude of the 3DFourier transform | F ( ˆ ρ ( r , θ , φ )) | which represents the am-plitude of a complex scalar diffracted wavefield | ˆ ψ ( w ) | andthen compare it to the ground-truth synthetic 3D diffractionpattern. A. 3D Convolutional Neural Network
Our approach uses a combination of a 3D convolutionalneural network together with model-independent adaptivefeedback. Convolutional neural networks are very powerfultools that can learn relationships between parameters in com-plex systems and in this case can directly utilize spatial in-formation to learn 3D features from the 3D amplitude of theFourier transform. The architecture of the 3D CNN networkdeveloped for our problem is shown in Figure (2).We point out that in mapping 3D Fourier transform inten-sities to spherical harmonic coefficients, a CNN is only ableto predict the even l -valued harmonics Y m , Y m , Y m , . . . for thefollowing reason.Consider two volumes described by surfaces which are per- turbations of a sphere, of the form s ± ( θ , φ ) = Y ( θ , φ ) ± ε Y ml ( θ , φ ) , (16)where l is odd. The odd l -valued harmonics are themselvesodd functions because all real spherical harmonics satisfy: ( − ) l Y ml ( θ , φ ) = Y ml ( π − θ , π + φ )= ⇒ − Y ml odd ( θ , φ ) = Y ml odd ( π − θ , π + φ ) . Therefore the two surfaces in (16) can be rewritten as s + ( θ , φ ) = Y ( θ , φ ) + ε Y ml ( θ , φ ) , s − ( θ , φ ) = Y ( θ , φ ) + ε Y ml ( π − θ , π + φ ) , (17)which are simply reflections and so the intensities of theFourier transforms of their volumes ρ ± are indistinguishablebecause of the lost phase information. When teaching a neu-ral network to map diffraction patterns to coefficients, we endup giving it inputs generated from two different surfaces andvolumes, with their corresponding spherical harmonic coeffi-cients as the correct outputs: s + = ⇒ ρ + = ⇒ | F ( ρ + ) | = | F ( ρ ± ) | = ⇒ CNN = ⇒ { , ε } , s − = ⇒ ρ − = ⇒ | F ( ρ − ) | = | F ( ρ ± ) | = ⇒ CNN = ⇒ { , − ε } . Because in this case the Fourier intensities are exactly thesame after learning over thousands of random data sets, theneural network is confused and at best can predict only theaverage value of 0 for all of the odd spherical harmonic coef-ficients.daptive ML for CDI 4
FIG. 3. Test vs prediction values shown for the even-valued a lm coefficients for 200 test structures along with the standard deviation of theerror for each coefficient. This problem can be confirmed numerically in three ways:1). If only positive values of odd harmonics are used to gen-erate volumes then the CNN learns how to map diffractionpatters to both even and odd harmonics, but this limits its ap-plicability because realistic objects have shapes that are com-posed of both odd and even harmonic components. 2). If thenetwork is tasked with only identifying the magnitude of theodd harmonics it is able to learn the relationship, but result-ing structure predictions must then iterate through all of thepossible ± combinations to find those which best match thegiven Fourier transform intensity to calculate the correct ob-ject shape, but the created object’s orientation will not neces-sarily match that of the target. 3). Finally, if the CNN is giventhe actual 3D electron densities or the 3D Fourier transforms(not just intensities) which contain the phase information asinputs, then it learns to map all spherical harmonics correctly,but this is not helpful for our problem, where the goal is tomake predictions solely based on diffracted intensities.This limitation of a neural network approach was also doc-umented in where they found that the neural network wouldsometimes predict objects that "are twin images of each other,and that they can be obtained from each other through acentrosymmetric inversion and complex conjugate operation.Both images are equivalent solutions to the input diffrac-tion pattern." Because of the limitations described above, theCNN was trained to map 3D diffracted intensities to only theeven valued spherical harmonic coefficients that describe theboundaries of the volumes whose Fourier transforms gener-ated those intensities.Data for training the 3D CNN was generated by samplingcoefficients from uniform distributions with ranges:ˆ a ml ∈ [ − c ml , c ml ] , c ml = . + l + | m | , and generating training volumes based on surfaces of the formˆ s ( θ , φ ) = Y ( θ , φ ) + N = ∑ l = l ∑ m = − l ˆ a ml Y ml ( θ , φ ) , (18)where the large Y ( θ , φ ) value ensured that we are workingwith a well defined surface perturbed by higher order compo-nents similar to naturally occurring complex grain shapes.We generated 500,000 training sets of 49 coefficients, for l = , . . . , N =
6, each of which was used to generate a surfaceand a volume bound by that surface to perform a 3D Fouriertransform. The input to the CNN was the intensity of the 3DFourier transform and the output of the CNN was a 28 di-mensional vector which was compared to the 28 even spher-ical harmonic coefficients Y ml for l ≤ y = ( y , . . . , y ) =( ˆ a , ˆ a − , . . . , ˆ a , . . . , ˆ a ) via the cost function: C CNN ( y ) = ∑ j = (cid:12)(cid:12) y j − a ml (cid:12)(cid:12) = ∑ l odd = l ∑ m = − l | ˆ a ml − a ml | . (19)CNN performance is illustrated in Figures 3 and 4 wherethe predictive accuracy of 200 unseen test sets is shown alongwith the lowest and highest prediction accuracy shapes.In this setup CNN performance is accurate, but can onlymake predictions for the even spherical harmonic coefficients.Furthermore, for use in experiments, the accuracy of a learnedmodel-based approach such as a CNN may suffer dependingon experimental setup changes and may require very lengthyexperiment-specific retraining.In order to predict the even spherical harmonics and also tomake these results more robust to a wide range of experimen-tal conditions, the next step of this approach is to use a model-independent algorithm that adjusts all spherical harmonic co-efficients directly based on matching individual 3D Fourierdaptive ML for CDI 5 FIG. 4. Detailed view of the best and worst performers out of 200 test structures that had not been seen during training of the CNN. transforms. The model independent approach is also capableof handling very large numbers of coefficients as shown belowwhen tuning up to 225 spherical harmonics simultaneously.
B. Model-independent tuning
For the adaptive part of this work, we utilize a model-independent extremum seeking (ES) algorithm which haswas originally developed for control and optimization of un-certain and time-varying systems by simultaneously tuninglarge numbers of coupled parameters based only on noisymeasurements . This bounded form of ES has been ana-lytically studied with convergence proofs for general non-differentiable dithers , has been proven to converge to op-timal controllers for unknown systems , and has been ap-plied to automatically control charged particle beams in parti-cle accelerators .The ES method is applicable to n-dimensional dynamic sys-tem of the form d y dt = f ( y , p , t ) , (20)ˆ C = C ( y , t ) + n ( t ) , (21)where y = ( y , . . . , y n ) are physical quantities of interest,such as diffraction patterns of electron densities. The p =( p , . . . , p m ) are controlled parameters, such as the sphericalharmonics that define the surface of a volume and t is time.The function f may be an unknown function governing thesystem’s dynamics. ˆ C is a measurement of an analytically un-known function C ( y , t ) that is noise-corrupted by an unknownfunction of time, n ( t ) , and depends on both the parameter val-ues y and on time due to a time-varying system environment.In our approach, we compare the intensities of the measureddiffraction and generated diffraction wavefields and quantifythe difference with the numerical cost function whose mini- mization is our goal: C ( y ) = µ ( V w ) ˚ V w (cid:12)(cid:12) | ψ ( w ) | − | ˆ ψ ( w ) | (cid:12)(cid:12) dV w , (22)where integration is performed over a volume in reciprocalspace V w of measure µ ( V w ) = ( w max − w min ) .In experimental applications of such a method, uncertaintywould come from the unknown electron densities that we aretrying to find and from uncertainties (such as misalignmentof components and drifts in X-ray coherence volume, wave-length, and flux) in the experimental setup.The parameters that we tuned were the Y lm coefficients y = (cid:16) y , . . . , y j , . . . , y ( + N ) (cid:17) = ( ˆ a , . . . , ˆ a ml , . . . , ˆ a NN ) , (23)which define the boundary surface of an unknown volume asin Equation (11) and the function that we are minimizing is C ( y ) as defined in (28). The ES algorithm perturbs parametersaccording to the dynamics dy j dt = (cid:112) αω j cos (cid:0) ω j t + k ˆ C ( y , t ) (cid:1) , (24)where ω j = ω r j and r i (cid:54) = r j for i (cid:54) = j . In (24) α is a dither-ing amplitude which can be increased to escape local minima.Once the dynamics have settled near an equilibrium point of(24), which may be a local minimum of C , each parameterwill continue to oscillate about its local optimal value with amagnitude of (cid:112) α / ω j . The term k > ω (cid:29) d y dt = − k α ∇ y C ( y , t ) , (25)which tracks the time-varying minimum of the unknown func-tion C ( y , t ) with respect to y ( t ) although using only its noise-corrupted measurement ˆ C as input. The reason behind con-vergence is that the evolution of the coupled parameters y j isdaptive ML for CDI 6decoupled and made orthogonal relative to the inner productin the L [ , t ] Hilbert space as defined in (3)lim ω i , ω j → ∞ (cid:10) cos ( ω i t ) , cos ( ω j t ) (cid:11) = . (26)Details and analytical proofs are available in .For iterative optimization as done in this work, we replacethe continuous time dynamics (24) with their discrete-time ap-proximation and make iterative updates according to y j ( n + ) = y j ( n ) + ∆ t (cid:112) αω j cos (cid:0) ω j ∆ t n + k ˆ C ( n ) (cid:1) , (27)which is a finite difference approximation of (24) for ∆ t (cid:28) C F ( ρ ) = µ ( V w ) × ˚ V w (cid:12)(cid:12) | ψ ( w ) | − | ˆ ψ ( w ) | (cid:12)(cid:12) dV w , (28) C ρ = µ ( ρ ) × ˚ V | ˆ ρ ( r , θ , φ ) − ρ ( r , θ , φ ) | dV , (29) C ∆ ρ = × | µ ( ˆ ρ ) − µ ( ρ ) | , (30) µ ( V w ) = ˚ V w | ψ ( w ) | dV w , V w = [ w min , w max ] , (31) µ ( ρ ) = ˚ V ρ ( r , θ , φ ) dV . (32)The quantity C F ( ρ ) is a measure of the percent differencebetween the intensity of the target and reconstructed Fouriertransforms. Convergence would mean we have matched theintensity of the Fourier transforms. However, it does not guar-antee the correct shape due to missing phase information, andthe same 3D object could be rotated or reflected. The quan-tity C ρ is a measure of the mismatch between volumes whichis non-zero when the objects have the same shape, but are ofdifferent orientations. Finally, the quantity C ∆ ρ is a measureof shape convergence which subtracts the total volumes oc-cupied by the two shapes and therefore will converge to zerowhen the two shapes are the same even if they have differentorientations.We created 100 random 3D shapes, generated their 3DFourier transforms, and fed the intensities of those transformsinto the 3D CNN. The predictions of the CNN were then usedas the starting point for the ES algorithm. Results of ES con-vergence for 100 random 3D shapes are shown in Figures 5and 6. Looking at the top images in the second and thirdcolumns of Figure 5 it is clear that the CNN-based objectshad Fourier transform intensity errors, C F ( ρ ) of 40% relativeto their full spectrum measures as defined in (31) and volumeerrors, C ∆ ρ of approximately 3%. The bottom images of thesecond and third columns show that by the end of convergencethe average intensity error was 8.4% and volumetric error was0.19%.In Figure 5, it is evident that on average all of the quantities C , C F ( ρ ) , C ∆ ρ , and C ρ converge towards zero; however C ρ hasseveral large outliers that never converge which implies thatthe densities being created are of the correct shape, but wrong orientation. The last column of Figure 5 is showing the er-rors between predicted ˆ a ml and correct a ml values. The greenbackground in Figure 5 highlights the even valued coefficientswhich we expect to match exactly while the odd componentsare expected to sometimes not converge due to the ambigu-ity introduced by the lack of phase information in the Fouriertransform’s intensity, as discussed above. Overall, the resultsof Figure 5 confirm that the ES approach is very robust andis able to find the correct object shape with the possibility ofan incorrect orientation in space, as expected. Figure 6 showsthree examples of exact agreement between 3D test objectsand their ES-based reconstructions. C. Adaptive ML for experimental data
In order to further demonstrate the robustness of the adap-tive ML approach, we applied it to an experimentally mea-sured 3D crystal volume that was obtained using high energydiffraction microscopy (HEDM) . HEDM is used for non-destructive measurements of spatially resolved orientation ( ∼ µ m and 0.01 ◦ ), grain resolved orientation, and elasticstrain tensor ( 10 − ) from representative volume elementswith hundreds of bulk grains in the measured microstructure(mm ) . HEDM measurements at multiple states of a sam-ple’s evolution can be used as inputs to inform and validatecrystal plasticity models . For a broad overview of HEDMand its many applications the reader is refereed to and themultiple references within.To test the robustness of our adaptive ML approach to astructure that had never been seen by neither the CNN nor theES algorithm during its tuning and design, we picked out asingle 3D grain from a polycrystalline copper sample whichwas measured with the HEDM technique at the AdvancedPhoton Source (APS) . The intensity of the 3D Fourier trans-form of this volume was fed into the CNN which providedan estimate of the first 28 even a ml coefficients. These werethen fed as initial guesses into the ES adaptive feedback al-gorithm which had the freedom to tune all 225 coefficients ofthe l ∈ { , . . . , } Y ml ( θ , φ ) spherical harmonics in order tomatch the generated and measured diffraction patterns. The3D shape and 2D slices of the amplitude and phase of the re-constructed particle results of convergence are shown in Fig-ures 7 and 8.The HEDM grain is relatively large with ∼ µ m diame-ter and is therefore too large to be imaged with existing CDItechniques due to light energy and coherence length limita-tions of existing light sources. Nevertheless, for testing theproposed method, the morphology of the HEDM crystal wasinteresting in its complexity and was similar to what has beenmeasured by Bragg CDI techniques as applied to quantum dotnanoparticles . Furthermore, advanced light sources such asthe planned Linac Coherent Light Source II (LCLS-II) freeelectron laser (FEL) and the APS Upgrade (APSU) are ex-pected to have increased transverse and longitudinal coher-ence lengths with techniques such as self seeding , to im-age larger than 1 µ m diameter crystals using high-energy CDIcombined with HEDM .daptive ML for CDI 7 FIG. 5. The top images of the first four columns show the convergence of the cost function C as defined in (28) together with convergence ofthe quantities defined in (28)-(30). The bottom images of the first four columns show histograms of converged values after the final iteration.The top image of the last column shows the errors of the converged ˆ a ml coefficients with the green band highlighting the even coefficientsrelative to which there is no ambiguity in the intensity of the Fourier transform. The bottom image of the last column shows the a ml errorstogether with mean and standard deviation as well as the bounds of the random distributions from which they were generated (blue/dashed).FIG. 6. Convergence of the ES algorithm for 3 different structures A, B, and C. In this demonstration the algorithm always started withonly Y (cid:54) = × ×
50 volume representations as seen by the error r ( θ , φ ) showing the initial and final differences between the volume bounding surfacesmapped onto the sphere. The last column shows the convergence of all parameter errors as the algorithm is able to find the correct values forall 36 Y lm settings in each case. V. CONCLUSIONS
In this proof-of-concept work, we demonstrate reconstruc-tions of arbitrary 3D shapes, while assuming no contribution of internal lattice distortions to the diffraction signal. Oneimmediate limitation of this method is that by parameteriz-ing surfaces as single valued functions over the 2D domain ( θ , φ ) ∈ [ , π ] × [ , π ] , we are limiting ourselves to producingonly star convex shapes. Star convex shapes are ones in whichdaptive ML for CDI 8 ABC
FIG. 7. Result of using the 3D CNN output as the initial guess for first the 49 a lm coefficients ( a , a − , a , a ,..., a ) , followed by ES finetuning of all 225 coefficients ( a ,..., a ) . The top row (A) shows the first measured state of the HEDM structure from various views. Thesecond row (B) shows the CNN-ES convergence results. The third row (C) is showing the same as (B) with shading for easier 3D visualization. -20 0 20 w x -20-1001020 w y -20 0 20 w x -20-1001020 w z -20 0 20 w y -20-1001020 w z -20 0 20 w x -20-1001020 w y -20 0 20 w x -20-1001020 w z -20 0 20 w y -20-1001020 w z -20 0 20 w x -20-1001020 w y -20 0 20 w x -20-1001020 w z -20 0 20 w y -20-1001020 w z -0.2-0.100.10.2 -20 0 20 w x -20-1001020 w y -20 0 20 w x -20-1001020 w z -20 0 20 w y -20-1001020 w z -202 -20 0 20 w x -20-1001020 w y -20 0 20 w x -20-1001020 w z -20 0 20 w y -20-1001020 w z -202 -20 0 20 w x -20-1001020 w y -20 0 20 w x -20-1001020 w z -20 0 20 w y -20-1001020 w z -202 ABC DEF w z = 50 w y = 50 w x = 50w z = 50 w y = 50 w x = 50w z = 50 w y = 50 w x = 50measuredCNN-ESdifference FFT amplitude FFT phase FIG. 8. Orthogonal slices through the 3D amplitude and 3D phase of the FT are shown. The top left row (A) shows the amplitudes of theHEDM measurement. The middle left row (B) shows the amplitudes of the CNN-ES reconstruction. The bottom left row (C) shows thedifference between (A) and (B), note the reduced color scale range. The rows (D), (E), and (F) show the same for the FT phase. daptive ML for CDI 9a line can be drawn from the center point to the outer edgewithout intercepting any other edges and therefore do not in-clude more complex surfaces which are not simply connected,such as a donut-like shape with holes. Generalization of thisapproach to a larger class of shapes will be the study of futurework by utilizing surface parameterization which decomposesa 3D particle surface onto three orthogonal directions .Furthermore, the method presented here can be readilyextended to reconstruct additional phases for crystals withinternal structures due to inherent defects and dislocationsby several methods including the use of generative convolu-tional neural networks or by extending the adaptive model-independent process to include more degrees of freedom. Al-though the CNN model is trained only on synthetic diffractiondata, the adaptive framework will readily account for noise inthe experimental data for robust reconstruction of 3D crystalswith internal structures, which is a topic of future work.
ACKNOWLEDGMENTS
This work was supported by the US Department of Energythrough the Los Alamos National Laboratory. Los AlamosNational Laboratory is operated by Triad National Security,LLC, for the National Nuclear Security Administration of U.SDepartment of Energy (Contract No. 89233218CNA000001).We acknowledge the support provided by the Institute of Ma-terials Science Rapid Response project RR2020-R&D-1.
VI. DATA AVAILABILITY
The data that support the findings of this study are availablefrom the corresponding authors upon reasonable request. D. L. McDowell, “A perspective on trends in multiscale plasticity,” Inter-national Journal of Plasticity , 1280–1309 (2010). G. Crabtree and J. Sarrao, “Opportunities for mesoscale science,” MRS bul-letin , 1079–1088 (2012). J. Miao, P. Charalambous, J. Kirz, and D. Sayre, “Extending the method-ology of x-ray crystallography to allow imaging of micrometre-sized non-crystalline specimens,” Nature , 342–344 (1999). I. K. Robinson, I. A. Vartanyants, G. Williams, M. Pfeifer, and J. Pitney,“Reconstruction of the shapes of gold nanocrystals using coherent x-raydiffraction,” Physical review letters , 195505 (2001). M. A. Pfeifer, G. J. Williams, I. A. Vartanyants, R. Harder, and I. K. Robin-son, “Three-dimensional mapping of a deformation field inside a nanocrys-tal,” Nature , 63–66 (2006). J. Clark, X. Huang, R. Harder, and I. Robinson, “High-resolution three-dimensional partially coherent diffraction imaging,” Nature communica-tions , 1–6 (2012). J. Clark, X. Huang, R. Harder, and I. Robinson, “High-resolution three-dimensional partially coherent diffraction imaging,” Nature communica-tions , 1–6 (2012). I. Robinson and R. Harder, “Coherent x-ray diffraction imaging of strain atthe nanoscale,” Nature materials , 291–298 (2009). J. Miao, R. L. Sandberg, and C. Song, “Coherent x-ray diffraction imag-ing,” IEEE Journal of selected topics in quantum electronics , 399–410(2011). C. Song, H. Jiang, A. Mancuso, B. Amirbekian, L. Peng, R. Sun, S. S. Shah,Z. H. Zhou, T. Ishikawa, and J. Miao, “Quantitative imaging of single,unstained viruses with coherent x rays,” Physical review letters , 158101(2008). J. Miao, K. O. Hodgson, T. Ishikawa, C. A. Larabell, M. A. LeGros,and Y. Nishino, “Imaging whole escherichia coli bacteria by using single-particle x-ray diffraction,” Proceedings of the National Academy of Sci-ences , 110–112 (2003). J. Miao, Y. Nishino, Y. Kohmura, B. Johnson, C. Song, S. H. Risbud,and T. Ishikawa, “Quantitative image reconstruction of gan quantum dotsfrom oversampled diffraction intensities alone,” Physical review letters ,085503 (2005). G. J. Williams, E. Hanssen, A. G. Peele, M. A. Pfeifer, J. Clark, B. Abbey,G. Cadenazzi, M. D. de Jonge, S. Vogt, L. Tilley, et al. , “High-resolution x-ray imaging of plasmodium falciparum-infected red blood cells,” Cytome-try Part A: The Journal of the International Society for Analytical Cytology , 949–957 (2008). Y. Nishino, Y. Takahashi, N. Imamoto, T. Ishikawa, and K. Maeshima,“Three-dimensional visualization of a human chromosome using coherentx-ray diffraction,” Physical review letters , 018101 (2009). X. Huang, R. Harder, S. Leake, J. Clark, and I. Robinson, “Three-dimensional bragg coherent diffraction imaging of an extended zno crystal,”Journal of applied crystallography , 778–784 (2012). J. Clark, X. Huang, R. Harder, and I. Robinson, “High-resolution three-dimensional partially coherent diffraction imaging,” Nature communica-tions , 1–6 (2012). F. Hofmann, E. Tarleton, R. J. Harder, N. W. Phillips, P.-W. Ma, J. N. Clark,I. K. Robinson, B. Abbey, W. Liu, and C. E. Beck, “3d lattice distortionsand defect structures in ion-implanted nano-crystals,” Scientific reports ,45993 (2017). M. J. Cherukara, R. Pokharel, T. S. O’Leary, J. K. Baldwin, E. Maxey,W. Cha, J. Maser, R. J. Harder, S. J. Fensin, and R. L. Sandberg, “Three-dimensional x-ray diffraction imaging of dislocations in polycrystallinemetals under tensile loading,” Nature communications , 1–6 (2018). J. Miao, D. Sayre, and H. Chapman, “Phase retrieval from the magnitudeof the fourier transforms of nonperiodic objects,” JOSA A , 1662–1669(1998). J. R. Fienup, “Reconstruction of an object from the modulus of its fouriertransform,” Optics letters , 27–29 (1978). J. R. Fienup, “Phase retrieval algorithms: a comparison,” Applied optics , 2758–2769 (1982). V. Elser, “Solution of the crystallographic phase problem by iterated projec-tions,” Acta Crystallographica Section A: Foundations of Crystallography , 201–209 (2003). C.-C. Chen, J. Miao, C. Wang, and T. Lee, “Application of optimizationtechnique to noncrystalline x-ray diffraction microscopy: Guided hybridinput-output method,” Physical Review B , 064113 (2007). S. Marchesini, “Invited article: A unified evaluation of iterative projectionalgorithms for phase retrieval,” Review of scientific instruments , 011301(2007). H. H. Bauschke, P. L. Combettes, and D. R. Luke, “Hybrid projection–reflection method for phase retrieval,” JOSA A , 1025–1034 (2003). D. R. Luke, “Relaxed averaged alternating reflections for diffraction imag-ing,” Inverse problems , 37 (2004). D. Sayre, “Imaging processes and coherence in physics,” Springer LectureNotes in Physics , 229–235 (1980). A. Colombo, D. E. Galli, L. De Caro, F. Scattarella, and E. Carlino, “Facingthe phase problem in coherent diffractive imaging via memetic algorithms,”Scientific reports , 1–12 (2017). G. Fevola, E. Bergbäck Knudsen, T. Ramos, D. Carbone, and J. Wenzel An-dreasen, “A monte carlo ray-tracing simulation of coherent x-ray diffractiveimaging,” Journal of Synchrotron Radiation (2020). S. Maddali, P. Li, A. Pateras, D. Timbie, N. Delegan, A. Crook, H. Lee,I. Calvo-Almazan, D. Sheyfer, W. Cha, et al. , “General approaches forshear-correcting coordinate transformations in bragg coherent diffractionimaging. part i,” Journal of Applied Crystallography (2020). P. Li, S. Maddali, A. Pateras, I. Calvo-Almazan, S. O. Hruszkewycz,W. Cha, V. Chamard, and M. Allain, “General approaches for shear-correcting coordinate transformations in bragg coherent diffraction imag-ing. part ii,” Journal of Applied Crystallography (2020). P. Godard, M. Allain, V. Chamard, and J. Rodenburg, “Noise models forlow counting rate coherent diffraction imaging,” Optics express , 25914–25934 (2012). daptive ML for CDI 10 S. Li, P. M. Dee, E. Khatami, and S. Johnston, “Accelerating lattice quan-tum monte carlo simulations using artificial neural networks: Applicationto the holstein model,” Physical Review B , 020302 (2019). E. Rrapaj and A. Roggero, “Exact representations of many body interac-tions with rbm neural networks,” arXiv preprint arXiv:2005.03568 (2020). T. Shimobaba, T. Kakue, and T. Ito, “Convolutional neural network-basedregression for depth prediction in digital holography,” in (IEEE, 2018) pp.1323–1326. A. Scheinker, A. Edelen, D. Bohler, C. Emma, and A. Lutman, “Demon-stration of model-independent control of the longitudinal phase space ofelectron beams in the linac-coherent light source with femtosecond resolu-tion,” Physical review letters , 044801 (2018). M. J. Cherukara, Y. S. Nashed, and R. J. Harder, “Real-time coherentdiffraction inversion using deep generative networks,” Scientific reports ,1–8 (2018). Y.-F. Shen, R. Pokharel, T. J. Nizolek, A. Kumar, and T. Lookman, “Con-volutional neural network-based method for real-time orientation indexingof measured electron backscatter diffraction patterns,” Acta Materialia ,118–131 (2019). S. Kandel, S. Maddali, M. Allain, S. O. Hruszkewycz, C. Jacobsen, andY. S. Nashed, “Using automatic differentiation as a general framework forptychographic reconstruction,” Optics express , 18653–18672 (2019). A. Scheinker, “Model independent beam tuning,” in
Proceedings of the2013 International Particle Accelerator Conference, Shanghai, China (2013) p. TUPWA068. A. Scheinker and D. Scheinker, “Bounded extremum seeking with discon-tinuous dithers,” Automatica , 250–257 (2016). A. Scheinker and D. Scheinker, “Extremum seeking for optimal controlproblems with unknown time-varying systems and unknown objective func-tions,” International Journal of Adaptive Control and Signal Processing(2020), https://doi.org/10.1002/acs.3097. R. Pokharel, J. Lind, S. F. Li, P. Kenesei, R. A. Lebensohn, R. M. Suter, andA. D. Rollett, “In-situ observation of bulk 3d grain evolution during plastic deformation in polycrystalline cu,” International Journal of Plasticity ,217–234 (2015). R. Pokharel, “Overview of high-energy x-ray diffraction microscopy(hedm) for mesoscale material characterization in three-dimensions,” in
Materials Discovery and Design (Springer, 2018) pp. 167–201. R. Pokharel, J. Lind, A. K. Kanjarla, R. A. Lebensohn, S. F. Li, P. Kene-sei, R. M. Suter, and A. D. Rollett, “Polycrystal plasticity: comparisonbetween grain-scale observations of deformation and simulations,” Annu.Rev. Condens. Matter Phys. , 317–346 (2014). A. Pandey and R. Pokharel, “Machine learning enabled surrogate crystalplasticity model for spatially resolved 3d orientation evolution under uni-axial tension,” arXiv preprint arXiv:2005.00951 (2020). P. Emma, R. Akre, J. Arthur, R. Bionta, C. Bostedt, J. Bozek, A. Brach-mann, P. Bucksbaum, R. Coffee, F.-J. Decker, et al. , “First lasing and oper-ation of an r a ngstrom-wavelength free-electron laser,” nature photonics ,641 (2010). G. Ju, M. J. Highland, C. Thompson, J. A. Eastman, P. H. Fuoss, H. Zhou,R. Dejus, and G. B. Stephenson, “Characterization of the x-ray coherenceproperties of an undulator beamline at the advanced photon source,” Journalof synchrotron radiation , 1036–1047 (2018). S. Maddali, M. Allain, W. Cha, R. Harder, J.-S. Park, P. Kenesei, J. Almer,Y. Nashed, and S. O. Hruszkewycz, “Phase retrieval for bragg coherentdiffraction imaging at high x-ray energies,” Physical Review A , 053838(2019). C. Brechbühler, G. Gerig, and O. Kübler, “Parametrization of closed sur-faces for 3-d shape description,” Computer vision and image understanding , 154–170 (1995). B. Zhao, D. Wei, and J. Wang, “Particle shape quantification using rotation-invariant spherical harmonic analysis,” Géotechnique Letters , 190–196(2017). D. Wei, J. Wang, and B. Zhao, “A simple method for particle shape gener-ation with spherical harmonics,” Powder technology330