Blind stain separation using model-aware generative learning and its applications on fluorescence microscopy images
BBlind stain separation using model-awaregenerative learning and its applications onfluorescence microscopy images
Xingyu Li
University of Alberta, Edmonton, Alberta, Canada [email protected]
Abstract.
Multiple stains are usually used to highlight biological sub-stances in biomedical image analysis. To decompose multiple stains forco-localization quantification, blind source separation is usually performed.Prior model-based stain separation methods usually rely on stains’ spa-tial distributions over an image and may fail to solve the co-localizationproblem. With the advantage of machine learning, deep generative mod-els are used for this purpose. Since prior knowledge of imaging modelsis ignored in purely data-driven solutions, these methods may be sub-optimal. In this study, a novel learning-based blind source separationframework is proposed, where the physical model of biomedical imagingis incorporated to regularize the learning process. The introduced model-relevant adversarial loss couples all generators in the framework and lim-its the capacities of the generative models. Further more, a training algo-rithm is innovated for the proposed framework to avoid inter-generatorconfusion during learning. This paper particularly takes fluorescence un-mixing in fluorescence microscopy images as an application example ofthe proposed framework. Qualitative and quantitative experimentationon a public fluorescence microscopy image set demonstrates the superi-ority of the proposed method over both prior model-based approachesand learning-based methods.
Keywords:
Blind source separation · model-aware generative learning · Fluorescence microscopy image.
In biomedical image analysis, multiple stains are usually used to highlight biolog-ical substances of interest and their interactions in tissue samples for quantita-tive and qualitative analysis. Fluorescence staining and histopathology stainingare the representative staining protocols widely adopted in tissue sample analy-sis. One issue in stained images is mixing/blurred colors due to co-localization ofstained biological substances. To regain the information provided by the contrastof individual stains, stain separation that facilitates the quantitative evaluationof degree of co-localization is highly desired in biological research. a r X i v : . [ ee ss . I V ] F e b Xingyu Li
Stain unmixing, or stain separation, is a specific source separation processthat separates the mixing stains in a biomedical image into a collection of single-stained planes/images. Model-based unmixing algorithms rely on specific mixingmodels, which can be formulated as either linear or nonlinear. For instance, fluo-rescence imaging follows a linear mixing model [1] and H&E stained histopathol-ogy imaging is characterized by the non-linear Beer–Lambert law [2]. In earlyliterature, stain unmixing is usually formulated as a model-based inverse prob-lem and many classic matrix decomposition techniques, for instance, ICA [3] andSVD [4], were deployed. Later, to address such a underdetermined source separa-tion problem, various regularization terms such as sparsity [7] and non-negativeconstraint [5][6] are introduced in model-based methods. However, these statis-tics approaches heavily rely on stains’ distributions in an image and usually failfor weak stained instances.With the advance of deep learning, blind source separation (BSS) problemsare now tackled by generative models [8][9]. Particularly for stain separation,the special case of BSS, deep models (such as U-Net [10] and generative adver-sary networks (GAN) [11][12]) can be built based on a set of training samplesthat consists of pairs of multi-color biomedical images and their decompositions.Different from the model-based methods relying on the statistics of stains inan image, deep generative models learn the decomposition function from thetraining set and thus are less sensitive to stains’ distribution in a specific image.In this work, a novel end-to-end model-aware GAN-based BSS frameworkis proposed for stain separation in biomedical images. Unlike the previous deeplearning BSS works which are only data-driven, the specific imaging model isincorporated in the proposed scheme. Briefly, in the proposed framework, mul-tiple deep nets are trained to generate single-stained images. By exploiting theimaging model, a model-relevant adversarial loss is used to couple individualgenerator nets in the learning process. Additionally, a new training algorithmis innovated, where the model-relevant adversarial loss is involved in trainingand a learning strategy is investigate to avoid inter-generator error propagation.Experimentation on fluorescence separation in biomedical images demonstratesthat prior knowledge of imaging model improves the learning performance andoutperforms prior arts. In summary, the contribution of the paper are as follows: – To our best of knowledge, it is the first time in literature to innovate a model-aware deep learning framework for biomedical image stain separation. – We incorporate generative learning and physical imaging models within theproposed framework. The solution is robust to variations in stains’ statisticsin an image. Compared to prior deep models, the proposed method generatesdecomposition results in higher quality. – A novel end-to-end training algorithm is proposed to optimize the learningefficiency. The model-relevant loss is incorporated in training to couple in-dividual generators toward the optimal solution and accelerate the learningprocess in the late training phase. itle Suppressed Due to Excessive Length 3
Framework Overview:
Stain separation decomposes a multi-stained image I into multiple images I i , each containing biological substances of interest stainedby one stain. Fig. 1 depicts the overview system architecture. In the framework, N generator nets are used for single-stained image generation. After the imagesyntheses module which takes the corresponding physical imaging model as thebasis, a discriminator net is incorporated to yield a model-relevant adversarialloss. Note that the generator G i : I → I i and discriminator D are trainable andthat the synthesis module S : { I , ..., I N } → I adopts a specific imaging modelof the on-hand problem to couple the N generators. Fig. 1.
Overview of the model-aware stain separation learning architecture, where theorange modules represent learning models with trainable parameters and the blue mod-ule is defined by the deterministic imaging model. During training, each G i first targetsto minimize the L loss between the ground-truth and its generated single-stained im-age only; when L losses are small, the model-relevant adversarial loss is incorporatedto accelerate generators’ optimization. During test, G i are used for stain separation. It is noteworthy that though the proposed framework adopts the adversar-ial loss, it is distinct from the classical GAN scheme. Instead of using multipleGAN nets to generate individual single-stained images, only one discriminator D is used to couple all generators in the learning processing. The model-awareadversarial loss drives all generators to work hard together to compete againstthe discriminator. Consider the subsystem between the input I and its synthesisˆ I , let’s denote it as F ◦ S where F = { G , ..., G N } : I → { I , ..., I N } is the de-composition function contributed by all generator nets and ◦ represents moduleconcatenation. Then the proposed model-aware scheme can be rephrased as tofind the decomposition function F such that F ◦ S is an identity function, i.e. F ◦ S : I → I . To this end, the proposed model-aware adversarial loss is designedas a function of F ◦ S , uniting all generators in training.One may notice that error associated with G i may propagate to other genera-tors G j for j (cid:54) = i through the model-aware adversarial loss, consequently causing Xingyu Li failure. To avoid such inter-generator confusion in learning, a novel end-to-endtraining algorithm is proposed to decouple generators at the early training phase.
Loss Functions:
As illustrated in Fig. 1, there are two categories of loss func-tions in the proposed scheme. For each generator G i , an L loss is used toquantify the difference between the generated single-stained image ˆ I i and itsground truth I i , i.e. L iL ( G i ) = E [ | I i − ˆ( I i ) | ] = E [ | I i − G i ( I ) | ] . (1)The reason that L loss is preferred over the common L loss is that L losstends to generate smooth/blue images.For the proposed architecture, we innovate a model-aware adversarial lossthat couples all G i s. Based on the design, F ◦ S : I → I is equivalent to anautoencoder with specific intermediate representations defined by the decom-position ground truth. Then F ◦ S and D compose a conditional GAN, where F ◦ S targets to fool the discriminator D and D tries to improve its classificationperformance. Hence, the model-aware adversarial loss is L cGAN ( D, F ◦ S ) = E [log D ( I, I )] + E [log(1 − D ( I, F ◦ S ( I )))] . (2)Note that since the synthesis module S is deterministic and well-defined by aspecific imaging model, all trainable parameters in the subsystem F ◦ S originatefrom F = { G , ..., G N } . This suggests that each generator G i ties to minimizethe adversarial loss in Eqn. (3) while the discriminator D tries to maximize it.The advantage of the model-aware adversarial loss is to couple all generatorsfor learning augmentation. But it also brings in a risk that the inter-generatorconfusion collapses the whole BSS system. Specifically, stain decomposition is anunder-determined problem as there may be numerous sets of { ˆ I i , ..., ˆ I N } satisfy-ing the imaging model S . Hence, if the single-stained image ˆ I i generated by G i greatly deviates from its true value I i , the corresponding error may propagate toother generators through the adversarial loss. To address this issue, rather thandirectly training F ◦ S towards the identity function via the adversarial loss, weenforce the generators G i focusing on estimation of decomposition ground-truthby L loss in Eqn. (2); G i s are coupled through the model-aware adversarial lossonly after generators have converged to the region that is close to the groundtruth I i . In this way, the learning process will be accelerated even at the latephase of training. In sum, the overall loss function is L ( D, F ◦ S ) = N (cid:88) i =1 L iL + λ L cGAN ( D, F ◦ S ) , (3)where λ is a hyper-parameter to weight the adversarial loss in the overall targetfunction. In the early training phase, λ = 0 to enable the generators to learn fromthe decomposition ground truth I i . In the late phase of training when L loss issmall, λ > itle Suppressed Due to Excessive Length 5 optimal solution to the problem is F ∗ = arg min F max D L ( D, F ◦ S ) = arg min F max D (cid:34) N (cid:88) i =1 L iL + λ L cGAN ( D, F ◦ S ) (cid:35) . Training Algorithm:
In this work, the standard procedures in GAN update[11] is followed - we alternate between one gradient descent step on discriminator D and then one step on generators G i . The specific end-to-end training procedureis described in Algorithm 1. Algorithm 1:
Minibatch stochastic gradient descent training of theproposed model-aware BBS learning
Input:
Training data set: multi-color images I s and its BBS groudtruth I i , i = 1 , ..., N , the number of sourse N , adversarial lossweight λ , and adversarial loss involvement parameter α Output:
Parameters of generators θ G i and deccriminator θ D for number of training iterations do Sample minibatch of m training data { I , ..., I m } ;Update the discriminator D by ascending the stochastic gradient: ∇ θ D m (cid:80) mj =1 (cid:2) log D ( I j , I j ) + log(1 − D ( I j , F ◦ S ( I j ))) (cid:3) ;Sample minibatch of m training data { I , ..., I m } and { I i , ..., I mi } ; if first − α percent iterations then Update the generators G i by ascending the stochastic gradients: ∇ θ Gi m (cid:80) mj =1 | I ji − G i ( I ji ) | ; else Update the generators G i by ascending the stochastic gradients: ∇ θ Gi m (cid:80) mj =1 (cid:104) | I ji − G i ( I ji ) | + λ log(1 − D ( I j , F ◦ S ( I j ))) (cid:105) ; endend Fluorescence microscopy is a technique used to visualize and identify the distri-bution of proteins or other molecules of interest stained with fluorescent stains.Because fluorescence microscopy is able to precisely distinguish individual anti-gens, it is the most popular cell and molecular biological research tool to studydynamical changes in the cell. To highlight multiple molecules of interest withina sample and analyze their interactions, multi-color fluorescence microscopy isused.Fluorescence unmixing aims to decompose a multi-color fluorescence mi-croscopy image into multiple images, each containing proteins or other moleculesof interest stained with one fluorescent stain only. Mathematically, given thatthere are N types of fluorescent stains in an image and let V i and D i representthe i th fluorescent stain’s spectrum vector and staining density map that con-sists of stain proportions over all pixels, a fluorescence microscopy image can be Xingyu Li formulated as a linear combination of the N staining density maps [1]: I = V × D = [ V , ..., V N ] × [ D , ..., D N ] T = N (cid:88) i =1 V i D i = N (cid:88) i =1 I i , (4)where I and I i are the observed multi-color fluorescence microscopy image andits i th decomposed instance, i.e. the single-stained image associated with the i th fluorescent stain, respectively. [ . ] T represents matrix transpose.To achieve fluorescent stain separation, model-based approaches in literatureexploit the imaging model in Eqn. (4) and estimate stains’ spectrum matrix V from a query image. Then staining density maps D is obtained by matrix inverseoperation, and single-stained image I i is derived by I i = V i D i . Among matrixinverse methods, NMF based approaches achieves top performance [5], wherenon-negative properties of D and V physically are considered during matrixfactorization. However, NMF-based solutions usually have weak ability to handlemolecules’ co-localization.In this paper, we use fluorescence separation as the application of the pro-posed framework. Different from model-based approaches that searches optimal V and D first, the proposed method takes Eqn. (4) as the image synthesis modeland directly generates a set of seperation results { I i } . In the next section, weevaluate the proposed stain separation framework on fluorescence images andcompare its performance with prior arts. We use image set BBBC020 from the Broad Bioimage BenchmarkCollection [15] in this study. In the high-resolution RGB-format fluorescence mi-croscopy images of murine bone-marrow derived macrophages from C57BL/6mice, nuclei were labeled by DAPI in blue and cell surface was stained byCD11b/APC in yellow [16]. In addition to multicolor images containing twofluorescent stains, single-stained images (i.e. DAPI images and CD22b/APC im-ages) are provided in the image set. Since the cells possess an irregular shape andsome of the macrophages touch or overlap each other, this dataset can be usedto assess an algorithm’s ability to deal with co-localization cell data [15]. In thisstudy, each high-resolution image is divided into 30 256 ×
256 patches. In thisway, we collect 2250 image patches coming from 750 decomposition patch pairs,each pair containing one multi-color image and its source separation groundtruth (i.e. one DAPI image and one CD11b/APC image). Then we randomlypicked 10% image pairs as test cases and the rest are used to train the proposedBBS model.
Network Architectures & Hyperparameter Setting:
In this study, weadopt the generators and discriminator from those in [12]. Specifically, U-Net256[14] is used to realize the generators, where the input tensor has a size of 256 × itle Suppressed Due to Excessive Length 7 Compared to the encoder-decoder structure, the skip path in U-Net helps tomaintain cell/molecule structures in images. Regarding to the discriminator, a70 ×
70 PatchGAN which aim to classify whether 70 ×
70 overlapping imagepatches are real or fake is adopted. It is noteworthy that the U-Net and thePatchGAN are applicable on any color images whose resolution is larger than256 × β = 0 . , β = 0 . α = 75 , λ = 0 . Comparison with State-of-the-Art Methods:
NMF based methods achievedtop performance in model-based fluorescence microscopy image separation. Hence,we include the NMF based method [5] in this comparison evaluation and use itas the baseline. Since there is no existing learning based approaches proposedfor biomedical image stain separation, two common generative learning modelsare evaluated. In specific, since we have two fluorescence stains in the image set,either 2 U-Nets [14] with the L loss or 2 pix2pix GAN [12] are used. Evaluation Metrics:
Both qualitative and quantitative evaluations are con-ducted. In qualitative comparison, spectral separation results generated by dif-ferent methods are visually examined. We also compute three major image qual-ity metrics (MSE, PSNR, and SSIM [18]) between the decomposition results andthe ground truth for quantitative comparison.
Fig. 2 presents examples of stain separation obtained by different methods. Asillustrated in the figure, the NMF-based method [5] and U-nets [14] fail to sepa-rate co-localized fluorescent stains in the images. Though the pix2pix GAN [12]obtains better results, block-check artifacts which usually occur in GAN modelsare observed, especially along image boundaries. Compared to prior arts, theproposed method yields the top source separation results.Table 1 records the quantitative evaluation of the examined methods. Forthe first column which corresponds to MSE, smaller values suggest better de-composition results. For PSNR and SSIM, the large the values are, the betterthe performance is. From the table, all three metrics advocates to the superi-ority of the proposed method, and the pix2pix GAN ranks the second despiteof its check-block artifact. This quantitative comparison is consistent with ourqualitative visual examination.
Table 1.
Quantitative evaluation of BBS via major image quality metrics, where thebest values are marked black.NMF method [5] U-nets [14] pix2pix GAN [12] proposed methodMSE 73.65 88.13 8.86
PSNR 30.67 32.97 39.13
SSIM 0.92 0.89 0.95
Xingyu Li
Fig. 2.
Samples of fluorescence separation using different methods. Images in the sec-ond row correspond to BSS ground truth. The NMF method [5] and U-Net [14] mayfail to handle co-localization; Images generated by the pix2pix GAN [12] appears withnoticeable check-block effects, while the proposed method generates smooth results asshown in the figure (It is recommended to compare the results between pix2pix GANand the proposed method by enlarging the figure in PDF).itle Suppressed Due to Excessive Length 9
The good performance of the proposed method is due to two reasons. First,let’s compare the U-Net based method and the proposed method. It should benoted that both approaches adopt the U-Net as the generators. The major dif-ference is that the proposed method introduces a model-aware adversarial lossin training. Inherent from GAN, the discriminator always tries to find a betterobjective function that distinguishes the reconstruction and the original input.As a result, instead of passively approaching the decomposition ground truthby L loss, the generator works hard to compete against the discriminator. Sec-ond, let’s focus on the pix2pix GAN [12] and the proposed method. Since bothapproaches adopt the adversarial game in training, they outperform the U-Netbased model. But different from the model that needs 2 independent pix2pixGAN to generate the spectral decomposition results, the proposed method in-novates the use of one discriminator to couple all generators via the fluorescencemicroscopy images. Because of the prior knowledge uniting all generators in thescheme, the proposed method has less freedom in image generation compared tothe pix2pix GAN method. This study proposed a novel deep learning based framework for BSS in biomed-ical images. In particular, it utilized the physical imaging model to augmentthe generative learning. In addition, the learning scheme was further empoweredand regularized by a model-aware loss. Both qualitative and quantitative experi-ments had validated the efficiency of the proposed method by public fluorescencemicroscopy image set. Future work would aim at evaluating the proposed stainunmixing scheme on other BSS scenarios such as multi-stained histo-pathologyimages.
References
1. Yi, HJ., et al.: Multilevel, hybrid regularization method for reconstruction of fluo-rescent molecular tomography. Applied Optics, vol. 51, no. 7, pp. 975–986, 2012.2. Ruifrok, A., and Johnston, D., : Quantification of histochemical staining by colordeconcolution. Anal. Quant. Cytol. Histol. Int. Acad. Cytol. Amer. Soc. Cytol., vol.23, no. 4, pp. 291–299, 2001.3. Cardoso, JF.: Blind signal separation: statistical principles. Processing of the IEEE, (10), 2009–2025 (1998).4. Macenko, M., et al.: A method for normalizing histology slides for quantitativeanalysis. In: IEEE International Symposium on Biomedical Imaging: From Nano toMacro (ISBI). pp. 1107–1110 (2009).5. Montcuquet, AS. et al.: Nonnegative matrix factorization: a blind spectra separationmethod for in vivo fluorescent optical imaging. Journal of Biomedical Optics, (5),056009 (2010).6. Li, X., et al.: Circular mixture modeling of color distribution for blind stain separa-tion in pathology images. IEEE journal of biomedical and health informatics, (1),150–161 (2017).0 Xingyu Li7. Kopriva, I. et al.: Blind decomposition of low-dimensional multi-spectral image bysparse component analysis. Journal of Chemometrics, (11), 590–597 (2009).8. Subakan., C, et al., Generative adversarial source separation. arVix (2017).9. Kong, QK., et al.: Single-channel signal separation and deconvolution with gen-erative adversarial networks. In: 28th International Joint Conference on ArtificialIntelligence, pp. 2747–2753 (2019).10. Ronneberger, O., et al.: U-Net: Convolutional networks for biomedical imagesegmentation. In: International Conference on Medical Image Computing andComputer-Assisted Intervention (MICCAI), pp. 234–241. (2015).11. Goodfellow IJ., et al.: Generative adversarial Nets. In: 28th Conference on NeuralInformation Processing Systems (NIPS), (2014).12. Isola, F., et al.: Image-to-image translation with generative adversarial networks.In: Conference on Computer Vision and Pattern Recognition (CVPR), (2017).13. Mao, X., et al.: Least squares generative adversarial networks. In: Conference onComputer Vision and Pattern Recognition (CVPR), (2017).14. Johnson, J.,et al.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision (ECCV), 2016.15. Lyosa, V., et al.: Annotated high-throughput microscopy image sets for validation.Nature Methods, , 637, 2012.16. Wenzel, J. et al.: Measurement of TLR-induced macrophage spreading by auto-mated image analysis: differential role of Myd88 and MAPK in early and late re-sponses. Frontiers in Physiology, (2011).17. Kingma, D., et al.: Adam: A method for stochastic optimization. In: InternationalConference on Learning Representations(ICLR), 2015.18. Wang, Z., et al.: Image quality assessment: From error visibility to structural sim-ilarity. IEEE Transactions on Image Processing,13