Noise Entangled GAN For Low-Dose CT Simulation
Chuang Niu, Ge Wang, Pingkun Yan, Juergen Hahn, Youfang Lai, Xun Jia, Arjun Krishna, Klaus Mueller, Andreu Badal, KyleJ. Myers, Rongping Zeng
116th International Meeting on Fully 3D Image Reconstruction in Radiology and Nuclear Medicine 19 - 23 July 2021, Leuven, Belgium
Noise Entangled GAN For Low-Dose CT Simulation
Chuang Niu , Ge Wang , Pingkun Yan , Juergen Hahn , Youfang Lai , Xun Jia , Arjun Krishna , Klaus Mueller , Andreu Badal , KyleJ. Myers , and Rongping Zeng Department of Biomedical Engineering, Center for Biotechnology & Interdisciplinary Studies, Rensselaer Polytechnic Institute, Troy, NY USA Department of Radiation Oncology, UT Southwestern Medical Center, Dallas, TX USA Computer Science Department, Stony Brook University, Stony Brook, NY USA Division of Imaging, Diagnostics and Software Reliability, OSEL, CDRH, U.S. Food and Drug Administration, Silver Spring, MD USA
Abstract
We propose a Noise Entangled GAN (NE-GAN) for sim-ulating low-dose computed tomography (CT) images from a higherdose CT image. First, we present two schemes to generate a clean CTimage and a noise image from the high-dose CT image. Then, giventhese generated images, an NE-GAN is proposed to simulate differentlevels of low-dose CT images, where the level of generated noise canbe continuously controlled by a noise factor. NE-GAN consists of agenerator and a set of discriminators, and the number of discriminatorsis determined by the number of noise levels during training. Com-pared with the traditional methods based on the projection data that areusually unavailable in real applications, NE-GAN can directly learnfrom the real and/or simulated CT images and may create low-dose CTimages quickly without the need of raw data or other proprietary CTscanner information. The experimental results show that the proposedmethod has the potential to simulate the realistic low-dose CT images.
An excess of x-ray exposure from computed tomography(CT) examinations could lead to the development of cancer,and thus optimizing CT protocols according to the as low asreasonably achievable (ALARA) principle has become im-portant. Low-dose CT (LDCT) simulation techniques havedeveloped as an effective tool to help determine the lowestdose in accordance with the ALARA principle, thereby cir-cumventing the repetition of CT examinations with differentexposure conditions for the same patients. However, reduc-ing the radiation dose will inevitably increase the noise levelin the reconstructed CT images and may compromise theaccuracy of a radiologist’s diagnostic decision. To this end,a lot of LDCT denoising methods have been proposed toimprove the image quality. Recently, deep-learning-baseddenoising methods have been shown a potential to achievethe superior denoising performance, if properly trained witha large number of CT images. In this context, the results withLDCT simulation methods can help train and test the robust-ness of denoising methods or other image analysis modelsapplied to the LDCT images.Traditionally, LDCT simulation tools insert random noiseto the raw sinogram data and reconstruct the noisy data tosimulate LDCT images [1]. However, neither raw data northe precise parameters of a CT imaging system are generallyaccessible without an established collaboration with the CTvendor. To circumvent the use of raw data, projection datacan be approximated by forward projecting from the CTimage, which are then added with noise and reconstructed using CT simulation software [2]. However, these sinogram-based methods are usually time-consuming and the simulatedprojection data may not truly reflect the real conditions sothe simulated LDCT noise is likely still not perfect. Recently,Shan et al. designed a specific GAN with a conditional batchnormalization layer to simulate LDCT noise from a random2-dimensional Gaussinan noise vector in the latent space [3].However, it is difficult for this method to generate realisticLDCT images from the Gaussian noise without explicit priorinformation of the LDCT noise.In this work, we treat the LDCT simulation as a transfor-mation from a higher dose CT (HDCT) image to the LDCTimages. Specifically, we first generate a clean CT image anda high-dose noise image from the HDCT image, and thentrain a noise entangled GAN (NE-GAN) to generate differentlevels of LDCT images via entangling the high-dose noiseimage scaled by different noise factors into the clean CTimage. The advantages of the proposed framework for LDCTsimulation are: 1) The generated high-dose noise image ex-plicitly contains the prior of noise and imaging system. 2)The NE-GAN can learn from both the simulated and real CTimages, so that it has the potential to generate realistic LDCTimages. 3) Once the model has been trained, the simulationspeed for LDCT images is very fast.
Ideally, before simulating the LDCT image from the HDCTimage, the noise component should be removed from theHDCT image and then low-dose noises are simulated andadded to the denoised image. In practice, the HDCT imagesare usually regarded as the clean image and the high-dosenoises are ignored. Although the magnitudes of high-dosenoises are low, they do contain the prior information of CTnoises and the imaging system to some extent. Based onabove observations, we propose to simulate an LDCT imagethrough two steps: the fist step is to generate a clean imageand a high-dose noise image from the HDCT image, and thesecond step is to generate different levels of LDCT imagesby entangling the high-dose noise component scaled with aspecific noise factor into the clean CT image. a r X i v : . [ ee ss . I V ] F e b Figure 1:
Generation of noise image.
Figure 2:
A denoising network trained using HDCT as input andLDCT as target. The predicted "LDCT" is a cleaner CT imagerather than a noisier.
For generating the high-dose noise image that preserves theprior information of the CT noise and imaging system, wepresent two schemes, i.e., through a denoising network orthrough CT simulation, as illustrated in Fig. 1. For thedenoising scheme, the HDCT image is first forwarded into adenoising model to obtain a clean CT image, which is thensubtracted from the input HDCT image to generate the high-dose noise image. The denoising model is trained by directlymapping high-dose CT image to the low-dose CT image,as shown in Fig. 2 . By doing this, the trained model cangenerate the denoised image instead of the images with morenoises, which is consistent to the findings of Noise2Noise [4].The denoising scheme can extract the real prior informationfrom the real HDCT images, which are then transformed toLDCT images with specific noise level by NE-GAN. TheCT simulation scheme is to use traditional sinogram-basedmethods to simulate a set of higher-dose noise images byvirtually scanning the real HDCT image, and the real HDCTimage is regarded as the clean CT image, as shown in Fig.1. Then, NE-GAN takes the simulated noise image and theHDCT image as inputs to generate a set of LDCT images withdifferent levels of noise. In this scheme, the sinogram-basedmethod is only used to simulate a single dose of images, andother lower dose of images can be generated by NE-GAN tosave computation time.
In this Subsection, we describe the details of the proposednoise entangled GAN (NE-GAN). As shown in Fig. 3, NE-
Figure 3:
Framework of NE-GAN.
GAN consists of a generator G and a set of discriminators D = { D j } , j = , · · · , S , where the number of discriminators S is equivalent to the number of lower-dose levels in the train-ing set. Specifically, the generator G is a encoder-decodernetwork that takes a clean CT image and a noise image scaledwith a noise factor as inputs, and outputs a LDCT image cor-responding to the input noise factor. All discriminators sharethe same network architecture. Each discriminator is to de-termine whether the generated image of a predefined level isreal.To train NE-GAN, we need a set of training samples { x i , n i , k j , x ji } , i = , · · · , N , j = , · · · , S , where x i and n i denote the clean CT image and high-dose image respectively, N is the total number of HDCT images, x ji denotes the cor-responding LDCT image, and S is number of noise levels ordiscriminators, k j denotes the noise factor that is a positivereal number and a larger value corresponds to a higher noiselevel or lower image quality. The loss function is: L = S ∑ j = E X j [ log D j ( x j )] + E X (cid:2) log ( − D j ( G ( x , n · k j )))+ | x − G ( x , n · k j ) | + | x − G ( x , ) | (cid:3) . The first two items in the loss function are the adversariallosses that train D to maximize the probability of assigningthe correct label to both real LDCT images and the generatedones from G and train G to minimize probability of assigningthe correct label for D , the third item is a data fidelity lossto constrain the generated LDCT images to keep the samecontents as those in the input images, and the fourth item is areconstruction loss to ensure that the generated CT imagesare exactly the clean images when the noise factor is zero.After training, only the generator G is retained to simulatedifferent levels of LDCT images given the clean CT image,the high-dose noise image, and the specific noise factor, i.e.,ˆ x j = G ( x , n · k j ) . It is noted that although the noise factorin the training stage is predefined as a limited number offixed values according to the training dataset, it could be anyvalue in the testing stage beyond the predefined values in thetraining stage. With increasing the value of noise factor, thenoise level of the simulated LDCT image will increase. We adopted the same generator and discriminator networksas those in CycleGAN [5]. The architecture of the denoisingnetwork was the same as the generator network. Duringtraining, we used the Adam method to optimize the NE-GANmodel with a batch of 8 128 ×
128 randomly cropped imagepatches. The initial learning rate was set to 0.0002 during thefirst 200 epochs and the learning rate was linearly decay tozero in the following 200 epochs. The momentum terms ofAdam were set to 0.5 and 0.999. The noise factor k j is set tothe ratio of the input dose level to the target dose level, seeSubsections 3.2 and 3.3 for details. In this study, we used a multi-dose of real CT image datasetfrom [6], in which the CT images were collected from anony-mous cadavers and each of them was repeatedly scanned fourtimes using four different radiation doses. In our experiments,we selected a sub-dataset that contains 261 groups of CT im-ages for training and 251 groups of CT images for testing,each group includes four 512 ×
512 FBP reconstructed im-ages that have the same contents but different noise indicesof 10, 20, 30, and 40. Here the noise index is approximatelyequal to standard deviation of CT number in the central re-gion of the image of a uniform phantom, and used to definethe image quality.
Figure 4:
Results of NE-GAN on simulated dataset.
Figure 5:
Results of NE-GAN on simulated dataset with addi-tional noise factors beyond training.
In this subsection, we used the simulation scheme as de-scribed in Subsection 2.1 to generate high-dose noise images.Specifically, we used the CatSim [7] simulator to simulateLDCT images of four different dose levels correspondingto X-ray tube currents of 90 mA, 70 mA, 50 mA, and 30mA, as shown in Fig. 4. The simulated CT image of 90mA is used as the HDCT image and the real CT image withnoise index of 10 is regarded as the clean CT image, thusthe high-dose noise image is the difference between them.With these images, NE-GAN was trained and noise factorscorresponding to 70 mA, 50 mA, and 30 mA were set to 1.3,1.8, and 3.0 respectively. The results in this setting are shownin Fig. 4, we can see that the proposed method can simulatethe different levels of LDCT images and the learned noiselevels are similar to those simulated with CatSim. The NE-GAN simulated results with different noise factors that werenot used in the training stage are shown in Fig. 5, where thenumber indicates the noise factor. Particularly, NE-GAN-0means that the scale factor is zero and in this case no noisesare added, consistent with the constraint in the loss functionas described in Subsection 2.2. Also, when increasing thenoise factor, the noise magnitude of simulated LDCT imageincreases and the image looks more noisier.
Figure 6:
Results of noise power spectrum.
In addition, we evaluated the statistical property of noisepower spectra (NPS) of the NE-GAN generated LDCT im-ages [8]. Specifically, we repeatly generated HDCT noiseimage with CatSim and simulated the LDCT images withNE-GAN by 50 times. Then, the 64 ×
64 image patches(green box) were cropped to calcuate the NPS, as shown inFig. 6. The NPS of the NE-GAN generated LDCT imagesis similar to that of the targets, which indicates the proposeddeep-learning-based method has the ability to preserve thestatistical properties of noise.
In this subsection, the proposed NE-GAN model was directlytrained on the real dataset. Here the denoising scheme wasfirstly used to decompose the high-dose CT image with noiseindex of 10 to a clean CT image and a high-dose noise image.Then the NE-GAN was trained to map these decomposedimages to the LDCT images with specific noise indices. Thenoise factors corresponding to noise indices of 20, 30, and40 were set to 2.0, 3.0, and 4.0 respectively. The results on
Figure 7:
Results of NE-GAN on real dataset.
Figure 8:
Results of NE-GAN on real dataset with additionalnoise factors beyond training. this real dataset set are shown in Fig. 7, where the first rowshows the denoised clean image and the high-dose noise im-age decomposed from the real HDCT image with noise indexof 10, the second row presents the real LDCT images withdifferent noise indices, the third row gives the correspondingNE-GAN generated LDCT images with the same noise in-dices, and the last row shows the reference results by directlyadding the scaled noise image with the same noise factorsinto the clean image. By comparing the second and the thirdrow, we can see that the simulated images of different noiseindices are similar to the corresponding real LDCT images.The results in the last row demonstrates that simply scalingthe extracted noise image and adding it back to the cleanimage cannot generate images matching with the real LDCTimages, while the proposed NE-GAN has the ability to si-multaneously transfer and merge the high-dose noise imageinto the clean image to simulate more realistic LDCT images.More simulation results with NE-GAN with different noisefactors beyond training are also shown in Fig. 7. Similarly,noise level increased continuously with the noise factors.
We presented a low-dose CT simulation method based ondeep learning. Visual comparison and NPS-based noise prop-erty evaluation have demonstrated the effectiveness of theproposed method. One main advantage of the proposed NE-GAN is the high speed for simulation, which is extremelyimportant when simulating a large number of LDCT images.
Figure 9:
Virtual CT workflow for robustness evaluation of LDCTdenoising algorithms.
For example, NE-GAN could be applied into a virtual CTworkflow for robustness evaluation of LDCT denoising algo-rithms by generating a large number of LDCT images withthe ground-truth, as shown in Fig. 9. Specifically, a CTgeneration method [9] is first used to generate many phan-toms, which are forwarded to Monte Carlo (MC) simulationtools [10] to simulate the HDCT images. Then, NE-GANcan simulate a large number of LDCT images with a fastspeed. Finally, the large number of LDCT images with theground-truth can be used to test the robustness of the LDCTdenoising algorithms. In the future, we will further improvethe simulation quality by adding some statistical constrainsin the NE-GAN loss function, such as those based on thenoise variance map and the noise power spectra.
References [1] P. Massoumzadeh, C. F. H. S. Don, K. T. Bae, et al. “Validation ofCT dose-reduction simulation”.
Med. Phys.
36 (2009), pp. 174–189.[2] S. E. Divel and N. J. Pelc. “Accurate Image Domain Noise Inser-tion in CT Images”.
IEEE Transactions on Medical Imaging
Developments in X-RayTomography XII . Vol. 11113. 2019, pp. 287 –296.[4] J. Lehtinen, J. Munkberg, J. Hasselgren, et al. “Noise2Noise:Learning Image Restoration without Clean Data”.
ICML . Vol. 80.2018, pp. 2965–2974.[5] J.-Y. Zhu, T. Park, P. Isola, et al. “Unpaired Image-to-ImageTranslation using Cycle-Consistent Adversarial Networks”.
ICCV .2017.[6] Q. Yang, M. Kalra, A. Padole, et al. “Big Data from CT Scanning”.2015.[7] B. D. M. et al. “CatSim: a new computer assisted tomography sim-ulation environment”.
Medical Imaging 2007: Physics of MedicalImaging . Vol. 6510. 2007, pp. 856 –863.[8] R Zeng, N Petrick, M. Gavrielides, et al. “Approximations ofnoise covariance in multi-slice helical CT scans: impact on lungnodule size estimation”.
Phys. Med. Biol.
56 (2011), pp. 6223–42.[9] A. Krishna and K. Mueller. “Medical (CT) image generation withstyle”.
Fully Three-Dimensional . Vol. 11072. 2019, pp. 542 –545.[10] X. Jia, H. Yan, X. Gu, et al. “Fast Monte Carlo simulation forpatient-specific CT/CBCT imaging dose calculation”.