Generative SToRM: A novel approach for joint alignment and recovery of multi-slice dynamic MRI
Qing Zou, Abdul Haseeb Ahmed, Prashant Nagpal, Sarv Priya, Rolf Schulte, Mathews Jacob
AALIGNMENT & JOINT RECOVERY OF MULTI-SLICE DYNAMIC MRI USING DEEPGENERATIVE MANIFOLD MODEL
Qing Zou, Abdul Haseeb Ahmed, Prashant Nagpal, Stanley Kruger, Mathews Jacob.
University of Iowa
ABSTRACT
We introduce a novel unsupervised deep generative man-ifold model for the recovery of multi-slice free-breathingand ungated cardiac MRI from highly undersampled mea-surements. The proposed scheme represents the multi-slicevolume at each time point as the output of a deep convolu-tional neural network (CNN) generator, which is driven bylatent vectors that capture the cardiac and respiratory phaseat the specific time point. The main difference between theproposed method and the traditional CNN approaches is thatthe proposed scheme learns the network parameters fromonly the highly undersampled data rather than the extensivefully-sampled training data. We also learn the latent codesfrom the undersampled data using the stochastic gradient de-scent. Regularizations on the network and the latent codesare introduced to encourage the learning of smooth imagemanifold and the latent codes for each slice have the samedistribution. The main benefits of the proposed scheme are(a) the ability to align multi-slice data and capitalize on theredundancy between the slices; (b) the ability to estimate thegating information directly from the k-t space data; and (c)the unsupervised learning strategy that eliminates the needfor extensive training data.
1. INTRODUCTION
Breath-held CINE imaging, which provides valuable indica-tors of abnormal structure and function, is an integral part ofcardiac MRI exams. The data is often acquired in the multi-slice fashion to preserve good contrast between myo-cardiumand blood. A challenge with the multi-slice scheme is thepotential for mis-matches between slices resulting from in-consistent breath-holds. Another challenge is the difficultyin acquiring data from subjects who cannot hold their breath(e.g. pediatric subjects, COPD subjects) for long durations.While self-gating and manifold methods were introduced forfree-breathing imaging. Self-gating methods [1, 2, 3, 4, 5] usek-space navigators to estimate the cardiac/respiratory phase,followed by the binning and recovery of measured data. Man-ifold approaches including our smoothness regularization on
This work is supported by grants NIH 1R01EB019961-01A1, 1R01AG067078-01A1 and R01EB019961-02S. manifolds (SToRM) approach [6, 7, 8], which perform soft-gating based on k-space navigators are emerging as power-ful alternatives to self-gating. However, all of these schemesperform the independent recovery of multi-slice data and failto capitalize on the interslice redundancies. Moreover, themanifold approaches usually need the storage of all the im-age frames in the time series, which requires high memorydemand.In this work, we propose a deep generative model for thereconstruction of multi-slice cine MRI. We model the multi-slice volume at each time point as a smooth non-linear func-tion of low-dimensional latent vectors. The highly non-linearfunction is represented using the deep convolutional neuralnetwork as we can use the implicit regularization brought inby the CNN. It is worth mentioning that the proposed doesnot require the extensive fully sampled training data, whichis not available in our setting. The parameters in the pro-posed scheme are learnt from only the highly undersampledmeasurements. Furthermore, the proposed method does notrequire k-space navigators to estimate the motion patterns.We note that for the multi-slice data, the cardiac and res-piratory motion during the acquisition of the different slicesare different. So we will use different latent vectors for eachslice, while the generator will be the same for all volumes.The parameters of the generator and the latent time-series foreach slice are jointly learned from the measured data of allthe slices. As mentioned above, the manifold approaches suf-fer from the high memory demand. While for the proposedscheme, the memory footprint of the algorithm is determinedby the network parameters θ and the latent vectors z , andhence is orders of magnitude smaller than that of manifoldapproaches.We propose to jointly learn the network parameters andthe latent vectors from the undersampled measurements. Wetry to minimize the cost (cid:80) Mi =1 (cid:80) Nt =1 (cid:107)A it ( G θ [ z it ]) − b it (cid:107) for the image recovery. The gradient of the non-linear func-tion ||∇ z G θ || is regularized to have a smooth image mani-fold. Besides, as we expect the image frames in the time se-ries vary smoothly in time, we penalize the smoothness of thelatent vectors in the optimization. More importantly, in or-der to make sure that all the slices are aligned properly, wealso add Kullback-Leibler (K-L) divergence [9] penalty onthe latent vectors for each slices to enforce that the latent vec- a r X i v : . [ ee ss . I V ] J a n ors of each slice will have the same distribution. Once thenetwork parameters and the latent vectors are optimized, thegenerator can be excited using the latent variables of any slice,when it generates aligned multi-slice data with matching car-diac/respiratory phases. The proposed scheme is illustrated inFig. 1. Fig. 1 . Illustration of the proposed scheme with 3 slices dataset. Thelatent vectors are fed into the deep generative model G θ , which thengenerates the multi-slice image volume in the time series. The latentvectors and the parameters θ of the generative model are learnedfrom the measured k-t space data.
2. METHODS
In this work, we model the image volume at the time point t during the acquisition of the i th slice, denoted by ρ ( i, t ) , asthe non-linear mapping ρ ( i, t ) = G θ [ z it ] . Here, z it are the low (2-4) dimensional latent vectors corre-sponding to slice i at a specific time point t , while G θ is a deepCNN generator, whose weights are denoted by θ .Note that we use the same network for all the slices, whichfacilitates the exploitation of the spatial redundancies betweenthe slices and is also memory efficient. We propose to jointlyestimate the network parameters θ and the latent variables ofthe different slices from the measured multi-slice data C ( z , θ ) = M (cid:88) i =1 N (cid:88) t =1 (cid:107)A it ( G θ [ z it ]) − b it (cid:107) + λ (cid:107)∇ z G θ (cid:107) (cid:124) (cid:123)(cid:122) (cid:125) network regularization + λ R ( z ) (cid:124) (cid:123)(cid:122) (cid:125) latent regularization (1)Here, A i,t corresponds to the measurement operator, whichextracts the i th slice and evaluates its multichannel Fouriertransform. Since the smoothness of the image manifold inthis setting is dependent upon the gradient of the non-linear function G θ , we regularize the weights of the generator tominimize the risk of high gradient values which results in thesituation that similar latent vectors are mapped to very dif-ferent images. We also add penalties on the latent vectors tofurther constrain the solution. The first penalty on the latentvectors is the smoothness penalty as we want to have that theimage frames vary smoothly in the time series. The secondpenalty for the latent vectors is the K-L divergence penalty onthe latent vectors for each slice. This penalty ensures that thelatent vectors for each slice will have the same distribution.This penalty is important because we finally generate alignedmulti-slice data with matching cardiac/respiratory phases us-ing the latent variables of any slice. If the latent vectors ofdifferent slices have different distribution, the generator maygenerate image frames without any meaning for the slices thatare not corresponding to latent vectors that is chosen.The parameters of the network and the latent vectors arejointly learned in an unsupervised fashion from the measuredk-t space data. We use ADAM optimization to determine theoptimal parameters. One can see that the proposed approachonly relies on the undersampled k-t space data. It does notrequire to have a large amount of fully-sampled training datato learn the parameters.
3. EXPERIMENTS3.1. Dataset and imaging experiments
This research study was conducted using human subject data.The institutional review board at the local institution approvedthe acquisition of the data, and written consent was obtainedfrom the subject. The experiments in this paper are based on awhole-heart multi-slice dataset collected in the free-breathingmode using a golden angle spiral trajectory. The acquisitionof the data was performed on a GE 3T scanner. The sequenceparameters were: TR= 8.4 ms, FOV= 320 mm x 320 mm, flipangle= 18 degrees, slice thickness= 8 mm.Results were generated using an Intel Xeon CPU at 2.40GHz and a Tesla V100-PCIE 32GB GPU. We binned the datafrom six spiral interleaves corresponding to 50 ms temporalresolution. The entire dataset corresponds to 522 frames. Weomit the first 22 frames and used the remaining 500 framesfor STORM reconstructions, which is used as ground truthfor comparisons. In all the studies, we assumed the latentvariables to be two dimensional since the main source of vari-ability in the data correspond to cardiac and respiratory mo-tion.
We study the impact of K-L divergence penalty in Fig. 2. InFig. 2 (a), we showed the multi-slice reconstructions withoutusing the K-L divergence penalty. The diastole and systolephases for each slice are exhibited. At the bottom of the fig-ure, we showed the plots of the latent vectors for each slice.rom the plots of the latent vectors in Fig. 2 (a), one cansee that the distribution of the latent vectors for each sliceare very different, which results in that when we feed on ofthe latent vectos into the generator, the reconstructions for theslices that are not corresponding to the chosen latent vectorsare bad. This is further confirmed by the images shown inFig. 2 (a). In Fig. 2 (b), we showed the multi-slice recon-structions by adding the K-L divergence penalty. From theplots of the latent vectors, we can see that the latent vectorsfor each slice will have the same distribution when the K-Ldivergence penalty is added. This results in the good recon-structions for each slice.
We compare the proposed multi-slice generative manifoldapproach with analysis SToRM [7]. Note that the analysisSToRM approach perform the independent recovery of multi-slice data. The reconstruction results are shown in Fig. 3. Thequantitative comparisons are made using the Signal-to-ErrorRation (SER) defined as
SER = 20 · log || x orig |||| x orig − x recon || . Here x orig and x recon represent the ground truth and the re-constructed image. The unit for SER is decibel (dB).We use the k-space data of 150 frames for the reconstruc-tions for comparison. The SToRM reconstructions using thedata of 500 frames are used as the ground truth. The resultsshow that the generative manifold approach is able to reducenoise and alias artifacts compared to analysis SToRM. We at-tribute the improved performance to spatial regularization of-fered by the CNN generator, which is absent in the analysisSTORM formulation. Furthermore, we note that unlike theanalysis schemes, the proposed scheme does not use k-spacenavigators to estimate the motion states; the latent variablesare estimated from the measured k-space data itself.
4. CONCLUSION
We introduce a generative manifold representation for thealignment and joint recovery of multi-slice dynamic MRIfrom highly undersampled measurements. The deep CNNgenerator is used to lift low-dimensional latent vectors tothe smooth image manifold and this proposed scheme doesnot require fully-sampled training data. We jointly optimizethe CNN generator parameters and the latent vectors basedon the undersampled data. During the training, the norm ofthe gradients of the generator is penalized to the learning ofa smooth surface/manifold, while temporal gradients of thelatent vectors are penalized to encourage the time series tobe smooth. Comparisons with existing methods suggest theutility of the proposed scheme in dynamic images. (a) Reconstructions without K-L divergence penalty(b) Reconstructions with K-L divergence penalty
Fig. 2 . Illustration of the impact of the K-L divergence penalty. (a)shows the multi-slice reconstructions without using the K-L diver-gence penalty. The latent vectors corresponding to slice
5. REFERENCES [1] Jeffrey Tsao, Peter Boesiger, and Klaas P Pruessmann,“k-t blast and k-t sense: dynamic mri with high frame rateexploiting spatiotemporal correlations,”
Magnetic Reso-nance in Medicine: An Official Journal of the Interna-tional Society for Magnetic Resonance in Medicine , vol.50, no. 5, pp. 1031–1042, 2003.[2] Sajan Goud Lingala, Yue Hu, Edward DiBella, and Math-ews Jacob, “Accelerated dynamic mri exploiting spar- ig. 3 . Illustration of the framework of the proposed scheme and comparison with existing methods. We compared the proposed multi-slicegenerative manifold approach with SToRM approach. We used the SToRM reconstructions from the data of 500 frames (SToRM500) as theground truth for quantitative comparison. For the comparisons, we used the data of 150 frames for the reconstruction. From the reportedSER, one can see that the proposed multi-slice generative manifold approach offers better reconstructions than the SToRM methods. We alsoplot the latent variables of 150 frames in time series for the proposed method. We showed four different phases for 2 different slices thatare reconstructed in the time series: systole in End-Expiration (E-E), systole in End-Inspiration (E-I), diastole in End-Expiration (E-E) anddiastole in End-Inspiration (E-I). The latent vectors corresponding to the four different phases are indicated in the plot of the latent vectors. sity and low-rank structure: kt slr,”
IEEE transactions onmedical imaging , vol. 30, no. 5, pp. 1042–1054, 2011.[3] Li Feng, Robert Grimm, Kai Tobias Block, Hersh Chan-darana, Sungheon Kim, Jian Xu, Leon Axel, Daniel KSodickson, and Ricardo Otazo, “Golden-angle radialsparse parallel mri: combination of compressed sensing,parallel imaging, and golden-angle radial sampling forfast and flexible dynamic volumetric mri,”
Magnetic res-onance in medicine , vol. 72, no. 3, pp. 707–717, 2014.[4] Li Feng, Leon Axel, Hersh Chandarana, Kai TobiasBlock, Daniel K Sodickson, and Ricardo Otazo, “Xd-grasp: golden-angle radial mri with reconstruction of ex-tra motion-state dimensions using compressed sensing,”
Magnetic resonance in medicine , vol. 75, no. 2, pp. 775–788, 2016.[5] Anthony G Christodoulou, Jaime L Shaw, ChristopherNguyen, Qi Yang, Yibin Xie, Nan Wang, and Debiao Li, “Magnetic resonance multitasking for motion-resolvedquantitative cardiovascular imaging,”
Nature biomedicalengineering , vol. 2, no. 4, pp. 215–226, 2018.[6] Sunrita Poddar and Mathews Jacob, “Dynamic mri usingsmoothness regularization on manifolds (storm),”
IEEEtransactions on medical imaging , vol. 35, no. 4, pp.1106–1115, 2015.[7] Abdul Haseeb Ahmed, Ruixi Zhou, Yang Yang, PrashantNagpal, Michael Salerno, and Mathews Jacob, “Free-breathing and ungated dynamic mri using navigator-lessspiral storm,”
IEEE Transactions on Medical Imaging ,2020.[8] Sunrita Poddar, Yasir Q Mohsin, Deidra Ansah, BijoyThattaliyath, Ravi Ashwath, and Mathews Jacob, “Man-ifold recovery using kernel low-rank regularization: Ap-plication to dynamic imaging,”
IEEE Transactions onComputational Imaging , vol. 5, no. 3, pp. 478–491, 2019.9] Dong Yu, Kaisheng Yao, Hang Su, Gang Li, and FrankSeide, “Kl-divergence regularized deep neural networkadaptation for improved large vocabulary speech recogni-tion,” in2013 IEEE International Conference on Acous-tics, Speech and Signal Processing