Multi-class probabilistic atlas-based whole heart segmentation method in cardiac CT and MRI
Tarun Kanti Ghosh, Md. Kamrul Hasan, Shidhartho Roy, Md. Ashraful Alam, Eklas Hossain, Mohiuddin Ahmad
DDate of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
Digital Object Identifier 10.1109/ACCESS.2017.DOI
Multi-class probabilistic atlas-basedwhole heart segmentation method incardiac CT and MRI
TARUN KANTI GHOSH , MD. KAMRUL HASAN , SHIDHARTHO ROY , MD. ASHRAFULALAM , EKLAS HOSSAIN (Senior Member, IEEE), MOHIUDDIN AHMAD (Member, IEEE) Department of Biomedical Engineering, Khulna University of Engineering & Technology, Khulna-9203, Bangladesh Department of Electrical and Electronic Engineering, Khulna University of Engineering & Technology, Khulna-9203, Bangladesh Department of Electrical Engineering & Renewable Energy, Oregon Renewable Energy Center (OREC), Oregon Institute of Technology, OR 97601, USA
Corresponding author: Md. Kamrul Hasan (e-mail: [email protected]).
ABSTRACT
Accurate and robust whole heart substructure segmentation is crucial in developing clinicalapplications, such as computer-aided diagnosis and computer-aided surgery. However, segmentation ofdifferent heart substructures is challenging because of inadequate edge or boundary information, thecomplexity of the background and texture, and the diversity in different substructures’ sizes and shapes. Thisarticle proposes a framework for multi-class whole heart segmentation employing non-rigid registration-based probabilistic atlas incorporating the Bayesian framework. We also propose a non-rigid registrationpipeline utilizing a multi-resolution strategy for obtaining the highest attainable mutual informationbetween the moving and fixed images. We further incorporate non-rigid registration into the expectation-maximization algorithm and implement different deep convolutional neural network-based encoder-decodernetworks for ablation studies. All the extensive experiments are conducted utilizing the publicly availabledataset for the whole heart segmentation containing MRI and CT cardiac images. The proposedapproach exhibits an encouraging achievement, yielding a mean volume overlapping error of . forCT scans exceeding the state-of-the-art results by a margin of . in terms of the same metric. As theproposed approach provides better-results to delineate the different substructures of the heart, it can be amedical diagnostic aiding tool for helping experts with quicker and more accurate results. INDEX TERMS
Bayesian framework, Deep convolutional neural network, Non-rigid registration, Proba-bilistic atlas, Whole heart segmentation.
I. INTRODUCTION
In this section, we first specify the challenges and motivationsof this article in subsection I-A. Secondly, we review sev-eral recent literature in subsection I-B for the particularizeddifficulties in subsection I-A. Finally, in subsection I-C, wesummarize our contributions in this article.
A. PROBLEM PRESENTATION
Medical imaging and computing technologies have revolu-tionized modern medicine and healthcare, which are increas-ingly crucial for the treatments and diagnosis of differentCardiovascular Diseases (CVDs) [1], [2]. Currently, differentnon-invasive cardiac imaging assessments, such as MagneticResonance Imaging (MRI), Computed Tomography (CT),Ultrasound (US), Positron Emission Tomography (PET), andSingle Photon Emission Computed Tomography (SPECT), are being used widely for clinical and diagnostic applica-tions in cardiology [3], [4]. Among all of those imagingmodalities, the MRI and CT scans have essential functionsin the non-invasive evaluation of CVDs through extensiveexperimentation, and clinical applications [1], [5]. The CTimages are more commonly employed than MRI images dueto their high-speed retrieval and more affordable expense,whereas the MRI images have no ionizing radioactivity andoutstanding soft-tissue contrast. However, most of the currentclinically convenient image examination approaches eitherattunes for the MRI or CT images alone [1], [5].Delineating different vital volumetric substructures fromthe whole volumetric medical images are of great signifi-cance for clinical practice for quantifying the morphologi-cal and pathological changes [1]. Precise segmentation aidsthe subsequent quantitative evaluation of the Volume of
VOLUME 4, 2016 a r X i v : . [ ee ss . I V ] F e b hosh et al. : Multi-class probabilistic atlas-based whole heart segmentation method in cardiac CT and MRI Interests (VOIs). It also avails specific diagnosis, forecastof prognosis, computer-aided diagnosis, radiation therapy,computer-aided surgery, intra-operative guidance, and surgi-cal planning [1], [6]. For instance, liver segmentation from D abdominal CT scans is a crucial prerequisite for tumorresection, computer-assisted live donor transplantation, andminimally invasive surgery interventions [7]–[9]. Besides,for the treatment of CVDs, including radio-frequency ab-lation and surgical preparation of crucial congenital heartdiseases, volumetric cardiac MR image segmentation is nec-essary [10]–[12].Currently, the Whole Heart Segmentation (WHS) is an im-perative preliminary action for a wide range of clinical treat-ments. For example, the pathology localization and accurateventricular dimensions [13], [14], which aims to delineateseven different heart substructures, as outlined in Table 1,from the whole cardiac images (see in Fig. 1). The preciseheart quantification requires the subtle segregation of dif-ferent heart substructures. For instance, the ejection portionand the myocardial mass are estimated from the segmentedventricular and the myocardial results, respectively, whichare critical pointers of cardiac disorder detection. The wholeheart’s manual delineation is labor-intensive and tedious,necessitating almost eight hours for an individual subject[15]. Consequently, the designing of computer-aided meth-ods to investigate medical images automatically are highlydemanding. However, an automated segmenting of a wholeheart is also challenging due to the cardiac anatomical shapevariations and the indefinite borders among different heartsubstructures [1], as depicted in Fig. 1. Achieving entirelycomputerized WHS is arduous due to the following hurdles: • Wide shape, structure, and boundary variations of thecardiac anatomy and their subsequent cardiac imaging • Indefinite boundaries, with inadequate edge informa-tion, between different heart substructures in cardiacimages • Low cardiac image quality and seldom they blend withother visually alike substructures (see in Fig. 1)
B. RECENT METHODS
Convolution Neural Network (CNN)-based Deep Learning(DL) strategies are the most commonly employed medicalimage segmentation method in different application fieldsand modalities [16]–[21]. Table 2 bestows various DL-basedand other techniques for the WHS of different substructures,including their corresponding utilized datasets and identicaloutcomes. UNet, initially proposed in [31], is mostly appliedCNN structure for the WHS [4], [18]–[20], [27], [29], [32]–[34], which can be summarized as follows:The authors in [27] suggested a framework consistingof two 3D-UNets, where the first network was employedto localize the bounding box encompassing the heart, andthe second network was used for the fine segmentation ofdifferent substructures. They also employed principal com-ponent analysis-based image augmentation for enhancing theWHS results. An UNet-based Omega-Net was introduced in [18] consisting of a set of UNet for fine-grained WHS.By explicitly finding the VOIs and turning the input imageto a standard orientation, this method adequately accom-modated the loss of segmentation precision produced bydistinguishing between training and testing images. Faster R-CNN (FRCNN) and UNet were combined in [19] namingas a CFUN. With the FRCNN’s precise localization abilityand UNet’s strong segmentation capability, CFUN neededsolely one-step detection and segmentation deduction to re-ceive the VOIs. The authors also used a novel loss functionbased on border information to accelerate the training andenhance the WHS results. The author in [32] applied anapproach combining with shape context knowledge encodedin volumetric shape models consisting of three primary steps:explorer segmentation with orthogonal 2D-UNet, shape con-text estimation, and refining segmentation with UNet andshapes context. The additional shape information, also calledshape context, was applied for implementing explicit 3Dshape knowledge to CNN. A multi-depth fused 3D-UNetwas applied in [34] to the initial network for more reliableextracting context information. The authors also introduceda hybrid loss incorporating focal loss into the dice functionto mark the volume size asymmetry among various ventric-ular substructures. The authors in [33] developed a pipelinewith three quintessential steps: employment of 3D-UNet forcoarse detection of VOIs to alleviate surrounding tissues’impact, artificially augmentation of the training dataset byextracting different VOIs, and a refined 3D-UNet for seg-mentation refinement using the augmented training dataset.The authors in [20] joined the attention tool into the gradientexpanding process for enhancing the coarse segmentationinformation with less computation expense. Moreover, theyintroduced the Category Attention Boosting (CAB) moduleinto the 3D-UNet network and constructed a new multi-scale boosting model, CAB-UNet, extending the network’sgradient flow and making sole usage of the low-resolutionfeature information.However, the segmentation strategies relying on the UNet-based structure suffer from redundant fusion tactics, impos-ing concatenation only at the identical scale feature mapsof the encoder and decoder [16], [17], [35]. There are someother DL- and Atlas-based techniques for the WHS [5], [15],[21], [25], [26], [28], [30], [36], which are also reviewed andsummarized as follows:A deep poincaré map encapsulating prior knowledge witha dynamical system was applied in [36]. A CNN-basedapproach had then navigated an ambassador over the cardiacMRI image, moving to approach a route that outlined the tar-get substructures. Different CT or MRI plans, such as coro-nal, sagittal, and axial, are explored for an adaptive fusion in[5] with a multi-planar deep CNN-based method. The authorsindependently built and trained three CNNs, with the sameconstructive arrangement, for those three planes and finally,combined them to obtain the segmented VOIs. A pipelinehaving two Fully Convolutional Networks (FCNs) was im-plemented in [30], where the first CNN localized the bound- VOLUME 4, 2016 hosh et al. : Multi-class probabilistic atlas-based whole heart segmentation method in cardiac CT and MRI
TABLE 1: Details of different heart substructures to be segmented from the whole chest images and their anatomical locations.
Substructures Acronym Anatomical position and functionsLeft Ventricular cavity LV The bottom left portion of the heart below the left atrium for pumping oxygenated blood to all tissuesRight Ventricular cavity RV The bottom right portion of the heart below the right atrium for pumping oxygen-depleted blood to the lungsLeft Atrial cavity LA The upper left portion of the heart to receive oxygenated blood from the four pulmonary veinsRight Atrial cavity RA The upper right portion of the heart for returning deoxygenated blood from the body to the RVMyocardium of LV Myo The myocardium is the muscular middle layer of the wall of the heart for pumping blood around the bodyAscending Aorta AO The ascending aorta is connected to the heart’s LV to allow the flow of blood from the heart into the aortaPulmonary Artery PA The pulmonary artery begins at the base of the heart’s RV to deliver oxygen-depleted blood to each similar lung
FIGURE 1: Illustration of complexities for achieving robust WHS in the employed cardiac MM-WHS-2017 dataset withcorresponding ground-truth, borrowed from [1], where the left three columns display the three orthogonal views of a cardiac CTimage and its corresponding WHS result, and the right three columns exhibit example cardiac MRI data and the WHS result.The LV, RV, LA, RA, Myo, AO, PA, and background in the first columns have a different shape, texture, and spatial locationthan the other two CT columns (second and third). Similar complexities are notified for the MRI scans (last three columns).TABLE 2: Several published literature for WHS with their employed datasets and achievements conferring varying metrics,such as mDSC, mVOE, and mSn, respectively, for mean dice similarity coefficient, mean volume overlapping error, and meansensitivity.
MetricsDifferent methods Year Datasets mDSC mVOE mSnA multi-modality atlas with a distinct label merging method based on a multi-scale patchapproach and a unique global atlas ranking system [15]
MM-WHS-2017 .
899 0 . − An automated approach using dilated convolutional neural networks for aggregatingfeatures at various scales through convolutional layers with too fewer parameters [22]
HVSMR-2016 .
865 0 . − A multi-planar deep CNNs with an adaptive merging procedure utilizing correspondinginformation from the separate planes of the 3D scans for enhanced delineations [5]
MM-WHS-2017 .
851 0 .
259 0 . An atlas-based segmentation approach with a two-stage registration pipeline, where themajority voting and STEPS algorithms were used for the merging of the labels [23]
HVSMR-2016 .
900 0 . − A method having three crucial actions: heart identification from landmark detection,heart isolation using mathematical shape model, and segmentation using learning-basedvoxel classification and local phase analysis [24]
HVSMR-2016 .
760 0 . − A deeply-supervised fractal network, where the multi-paths with different receptive fieldswere established in a self-similar fractal system to obtain the hierarchical features [25]
HVSMR-2016 .
760 0 . − Two-stage UNet framework, where the first stage detected the VOIs and the second stageaccurately segmented the heart substructures [4]
MM-WHS-2017 .
793 0 . − A semi-supervised approach, where the student model acquires from labeled target dataand also searches unlabeled target data and labeled data by two teacher models [26]
MM-WHS-2017 .
860 0 . − Category attention boosting module connecting the deep network estimation graph withthe boosting approach [20]
HVSMR-2016 .
894 0 . − A CNN-based architecture, which includes principal component analysis as asupplementary data enlargement routine [27]
MM-WHS-2017 .
890 0 . − A deep heterogeneous feature gathering network (HFANet) to wholly employcorresponding knowledge from various views of 3D cardiac data [28]
HVSMR-2016 .
942 0 . − MM-WHS-2017 .
909 0 . − AAPM-2017 .
883 0 . − An adversarial training strategy for training the networks using the UNet as a generatorin adversarial network [29]
HVMSR-2016 .
870 0 .
230 0 . Cascading of two volumetric FCNs, where the first network aimed at locating the cardiacarea, during the second segmented different cardiac and great vessel substructures [21]
HVSMR-2016 .
942 0 .
109 0 . A pipeline of two FCNs, where the first network localizes the bounding box’s centerencompassing all heart substructures, while the second network segments them [30]
MM-WHS-2017 .
908 0 . − VOLUME 4, 2016 et al. : Multi-class probabilistic atlas-based whole heart segmentation method in cardiac CT and MRI ing box’s center around the different substructures and thesucceeding second CNN concentrated on these regions forsegmenting the organs. The authors in [28] developed a deepHeterogeneous Feature Aggregation Network (HFANet) forentirely exploiting corresponding information from 3D car-diac data. They utilized asymmetrical 3D kernels and poolingfor obtaining heterogeneous features in identical encodingroutes. Therefore, distinguishable features were extractedfrom a specific view, and necessary contextual informationwas kept. Then, they employed a content-aware multi-planarconcatenation for aggregating meaningful features to boostthe WHS performance. Further, to overcome the model size,they also devised a new DenseVoxNet model by sparsifyingskip connections trained in an end-to-end manner. A Dual-Teacher strategy was suggested in [26], where the studentmodel acquired precisely the knowledge of unlabeled targetdata from intra-domain teachers by fostering prediction tex-ture and the shape priors embedded in labeled source datafrom inter-domain teachers via information distillation. Theauthors also examined the utility of concurrently leverag-ing unlabeled data and well-known cross-modality data forthe segmentation. A 3D fractal network for effective com-puterized segmentation technique was introduced in [25].The designed network took full convolutional constructionto implement effective, well-defined, and volume-to-volumeprognostication. Prominently, by recursively employing asingle augmentation rule, the authors assembled the networkin a novel self-similar fractal system and consequently pro-moted it in consolidating hierarchical evidence for precisesegmentation. They also employed a deep supervision toolto mitigate the vanishing gradients obstacle and increaseour network’s training effectiveness on inadequate medicalimage datasets. The authors in [15] offered a multi-scalepatch for hierarchical local atlas ranking. Their segmentationapproach exercised multi-modality atlases from MRI andCT and embraced a new label merging method based onthe recommended multi-scale patch policy and a new globalatlas ranking scheme. Both the local and global atlas rankingactions used the information-theoretic criteria to estimate theassociation between the target image and the atlases fromversatile modalities.
C. OUR CONTRIBUTION
While many approaches have already been developed and im-plemented for the WHS, there is still room for performanceimprovement. This article proposes a statistical WHS methodthat joins the prior anatomical information described byprobabilistic atlas into the Bayesian inference for delineatingseven different heart substructures (see in subsection I-A) incardiac CT and MRI images. Our multi-class WHS frame-work is based on the proposed non-rigid registration pipelinefor atlas construction (see in subsection II-B1), utilizing amulti-resolution strategy to obtain the highest possible mu-tual information between moving and fixed CT or MRI im-ages. Different parameters of the registration algorithm in ourpipeline, such as optimizer, interpolator, metric, resampling technique, and transformation, are tuned for achieving betterspatial alignment between moving and fixed CT or MRIimages. We also develop various fusion strategies of multipleatlases to delineate the different anatomical structures forablation studies. We further incorporate non-rigid registrationinto expectation-maximization for the WHS to compare withthe proposed statistical segmentation algorithm. Besides, wehave implemented other CNN-based supervised methods,where we employ three variants of encoder-decoder net-works. The best performing CNN network is compared withthe proposed statistical pipeline. We validate all the extensiveexperiments utilizing the publicly available dataset namedMM-WHS-2017 (see details in subsection II-A). The submit-ted pipeline exceeds state-of-the-art results for the WHS onthe used dataset to our most trustworthy knowledge.The rest of the sections are manifested as follows: sectionII illustrates the used datasets to bear extensive experimentsand the different methodologies. Section III describes theobtained results accompanying with a precise analysis andstate-of-the-art connections. Finally, section IV terminatesthe paper with prospective future acts.
II. MATERIALS AND METHODS
This section elaborates on the materials and methodologies inthe article. We present the utilized dataset and our proposedWHS schemes in subsections II-A and II-B, respectively.Subsections II-C and II-D respectively explain other imple-mented CNN-based methods for WHS and the hardware &metrics used to evaluate the experimentation.
A. DATASET
All the comprehensive experiments were conducted utiliz-ing publicly available MM-WHS- dataset [1], as it iscommonly used in recent articles (see in Table 2), whichcontains cardiac whole heart volumetric MRI and CT data.The VOIs include the seven different substructures in theutilized WHS dataset, as described earlier in subsection I-A(see in Table 1). The different substructures, such as LV, RV,LA, RA, Myo, AO, and PA (see in Fig. 1) of both the CTand MRI are labeled as , , , , , , and , respectively. We aim to segment those seven organsfrom both the cardiac CT and MRI scans. Hence, we havetermed it a multi-class (8-classes) segmentation task, includ-ing the background and seven different heart substructures.All the experimental WHS approaches are assessed followinga leave-one-out evaluation strategy [37].The CT and MRI sequences were collected from a -slice CT scanner (Philips Medical Systems, The Nether-lands) and a . T clinical scanner (Philips Healthcare, TheNetherlands), respectively. The volumes from different scan-ners were stored as NIfTI file format in differing imageproperties and resolutions to provide imperfect training datato promote more robust algorithms’ construction [38]. Theformer volumetric CT images were obtained in axial view,incorporating the entire heart from the topmost abdominalto the aortic arch with an in-plane resolution of . × . VOLUME 4, 2016 hosh et al. : Multi-class probabilistic atlas-based whole heart segmentation method in cardiac CT and MRI mm, and the standard slice thickness of . mm. The lattervolumetric MRI images were accumulated with a resolutionof × × mm and reconstructed to nearby × × mm. B. PROPOSED METHOD
The WHS’s proposed method essentially consists of twointegral parts, such as
Atlas construction and segmentationstrategy , where we integrate different strategies to select thebest performer for WHS. The elaborate discussion of thesetwo crucial parts of our proposed framework is manifested inthe following two subsections, II-B1 and II-B2, respectively.
1) Registration and Atlas Construction
Fig. 2 demonstrates the proposed pipeline for atlas-basedWHS, where the registration is the crucial integral step todeform a moving CT or MRI image to align with a fixedCT or MRI image spatially. Let us consider that I T is anunseen fixed cardiac image (CT or MRI) to be segmented( L T ). { ( I i , L i ) | i = 1 , ..., N } is a set of moving imagesto build atlases, where I i , L i , and N are the i th intensityimage, its corresponding i th label image, and the numberof member images in the reference volumes, respectively.We perform moving-to-fixed registration (non-rigid) to con-struct a deformed set, { ( I iD , L iD ) | i = 1 , ..., N } , where I iD and L iD are the resultant deformed intensity imageand its corresponding deformed label image. The deformedintensity image ( I iD ) and its corresponding deformed labelimage ( L iD ) are averaged to construct probabilistic atlas ( I A )and probabilistic label ( L A ), respectively, using (1) and (2),respectively. I A ( x, y, z ) = 1 N N (cid:88) i =1 I iD ( x, y, z )= 1 N N (cid:88) i =1 I i ( T NR i ( x, y, z )) , (1) L A ( x, y, z ) = 1 N N (cid:88) i =1 L iD ( x, y, z )= 1 N N (cid:88) i =1 L i ( T NR i ( x, y, z )) , (2)where T NR i is a transformation between i th pair of I i and I T .For M voxels and k ∈ { , , ..., K = 7 } target substructureregions, both the I A ( x, y, z ) and L A ( x, y, z ) probabilisticatlas is a matrix with N × K elements, and each element( P nk ∈ ( I A or L A ) ) represents the anatomical knowledgeabout the heart provided by training samples on the priorprobability of n th voxel belonging to a particular tissue class k . The intensity image ( I i ) is appointed as the moving imagefor deforming to the unseen fixed images ( I T ) using a non-rigid transformation ( T NR ) [39]–[41]. The algorithm of T NR is a combination of a global and local transformations as T NR ( x, y, z ) = T global ( x, y, z ) + T local ( x, y, z ) . The global ( T global ) is an affine transformation allowing scaling, trans-lation, rotation, and shearing of I i , whereas the local ( T local )is a free-form deformation model based on B-splines [42].However, in our pipeline, we formulate the registration asan optimization problem in (3) for maximizing the similaritybetween I i and I T employing the T NR . T NR = argmin T NR Υ[ T NR ; I T ( x, y, z ) , I i ( x, y, z )] , (3)where T NR is an optimal transformation to spatially align I i ( T NR ( x, y, z )) to I T . Υ is a cost function, which we mini-mize by designing a pipeline combining different algorithms.The cost function ( Υ ) is optimized using adaptive stochasticgradient descent optimizer [43]. Other crucial algorithmsin our proposed pipeline, such as interpolator, metric, andresample-interpolator, are the B-Spline algorithm with anorder of , mutual information [44] with histogram bins of , and B-Spline algorithm with an order of , respectively.A multi-resolution strategy, with resolutions of , is usedto bypass local minima [45]. We use Elastix- . [46] toimplement our proposed registration pipeline. In this work,a full probabilistic atlas is built and evaluated following aleave-one-out evaluation strategy [37].
2) Segmentation Strategies
This subsection presents and designs several integral strate-gies in our proposed pipeline (see in Fig. 2) to delineatedifferent anatomical substructures (see in Table 1); once themoving images are deformed to a fixed image employing anon-rigid registration. a: Multi-atlas Label Propagation
The easiest and quickest technique to assign a label to eachvoxel of an input test image is the label propagation ofdeformed labels ( L iD ) to the unseen test image space [47]–[49]. Multi-atlas Label Propagation (MALP) trades betterwith the registration errors comparing a single atlas andbetter accounts for anatomical variability [50]. The MALPalso strengthens over single label propagation, as it canreject outliers (minority) of the deformed labels ( L iD ). TheMajority Voting Fusion (MVF) [15] is a MALP approach,which counts the number of atlases provided the same labelfor a test voxel, n , in the same spatial location. The resultantsegmented VOI ( L T ) can be estimated from L iD , as in (4): L T ( n ) = argmax k ∈{ ,...,K } N (cid:88) i =1 Γ( L iD ( n ) , k ) , (4)where { , , ..., K } is a set of K (= ) labels of the heartanatomy (see in subsection II-A), Γ( L iD ( n ) , k ) is a countingfunction, as defined in (5). Γ( L iD ( n ) , k ) = (cid:40) if L iD ( n ) = k, if L iD ( n ) (cid:54) = k. (5) https://elastix.lumc.nl/doxygen/index.html VOLUME 4, 2016 et al. : Multi-class probabilistic atlas-based whole heart segmentation method in cardiac CT and MRI
Averaging
Registration
Atlas intensity images
N1Atlas label images
Probabilistic atlasTest imageDeformed atlas images and propagated labelsN1
FIGURE 2: Illustration of probabilistic atlas-based segmentation for the whole heart segmentation employing different fusionstrategies, where we use non-rigid registration for atlas construction.The other method of MALP is the estimation median valueof all the candidate labels in a particular voxel location.The median value of each test voxel ( n ) from the deformedlabels ( L iD ) also provides a robust estimation of resultantlabel ( L T ) [51], which can be formulated as L T ( n ) = M edian ( L kS ( n )) , ∀ k ∈ K , where L kS ( n ) , ∀ k ∈ K , is astacked (in th dimension) deformed label of L iD , ∀ i ∈ N ,which has N -values for all voxels, n . b: Probabilistic Atlas-based Segmentation (PAS) In our propsoed PAS framework, we define the actual mov-ing labels as X and the target fixed image as Y , wherethe components of X and Y are prepared by a spatiallocation expressed by n ∈ J , wherever J is the simplis-tic D rectangular grid index ( x, y, z ) . Let us consideringthat X = ( x , x , ..., x M ) , Y = ( y , y , ..., y M ) , and A = ( a , a , ..., a M ) are the sample realizations of labelimage, intensity image, and probabilistic label, respectively,where M is the total voxel number. Example space of X is indicated by Ω x , where Ω x = { x : x n ∈ { k =1 , , ..., K } , ∀ n ∈ J } . The probability atlas is K -vector a j = ( a j , a j , ..., a jK ) , ∀ j ∈ M , where all the ingredientcorresponds to expectation of K -different heart substruc-tures. The prior probability for all voxels can be expressedas P ( x n = k ) = a nk , ∀ n ∈ J ; k ∈ K .The hindrance comprises determining the label X thatadequately illustrates the provided observation Y accordingto any loss function. As a decision rule, we chose MAP(maximum a posteriori) and the formula for the realizationof estimating of X as ˆ x = argmax x P ( X = x | Y = y ) .The posterior probability ( P ( X | Y ) ) can be written as themultiplication of probability distribution ( P ( Y | X ) ), alsonamed tissue model [37], [52], and prior probability ( P ( X ) ), according to the Bayes theorem. P ( Y | X ) is defined by signalintensity tissue models immediately formed from the scansand hand-operated segmentation of the data set. An intensityvalue’s histogram is constructed for each heart substructureconsidering the given volumes’ voxels, which belong to it,using hand-operated segmentation. In this work, we estimatethe probability distribution P ( Y | X ) of the given image Y forthe provided appropriate segmentation X from training data.Algorithm 1 shows detail process of estimating P ( Y | X ) us-ing the number of bins ( N b ) for the histograms as . Fig. 3 Algorithm 1:
Estimation of P ( Y | X ) of the image Y for given segmentation X . Initialize the counters as C kb = 0 , ∀ b ∈ N b ; k ∈ K for all training images do Rescale the image ([ ∼ N b − ]) and extract VOI Accumulate N k , ∀ k ∈ K with corresponding pixelnumbers. Compute histograms H k , ∀ k ∈ K , and accumulatein counters C k + = H k , ∀ k ∈ K end for Normalize histograms C k /N k , ∀ k ∈ K Scale the histograms as C kb = C kb (cid:80) Kk =1 C kb , ∀ b ∈ N b ; k ∈ K , so that (cid:80) Kk =1 C kb = 1 , ∀ b ∈ N b exhibits an illustration of the Bayesian voxel classificationalgorithm consolidating the application of the probabilisticatlas of the WHS. Besides, the probabilistic atlas gives theprobability distribution P ( X ) once it has been mapped ontothe target space employing the corresponding registration VOLUME 4, 2016 hosh et al. : Multi-class probabilistic atlas-based whole heart segmentation method in cardiac CT and MRI
Probabilistic atlas label {T}
Bayesian frameworkP(X|Y) = argmax P (Y|X) P (X) P (Y|X) P (X) Y Probabilistic intensity image {A} {T} T {A} Target {T}
Segmented result {T}
LV RV AO PARA LA Myo
Tissue model (
P(Y|X) ) Non-rigid registration X FIGURE 3: Voxel classification algorithm overview for the WHS, where the probabilistic atlas labels are transformed to theunseen test image space { T } utilizing the atlas’ anatomical image. The probabilistic atlas, also known as the tissue models, andthe unseen objective image are given to the Bayesian interface as a prior probability P ( X ) , conditional probability P ( Y | X ) and set of intensity values Y , respectively. The Bayesian interface estimates the posterior probability for the segmentation ofgiven X maximizing the P ( X ) · P ( Y | X ) .method utilized in its creation. c: Expectation Maximization (EM) The EM algorithm is an iterative approach that classifies datapoints with the Gaussian Mixture Model (GMM), updatesmodel parameters with the newly classified data, and clas-sifies data points with the new parameters. In our implemen-tation, we use EM to learn the parameters of the GMM in anunsupervised fashion for the segmentation of different heartsubstructures from the whole cardiac CT or MRI images. Letus assume that X = ( x , x , ..., x M ) is an unseen targetCT or MRI image, where x i , ∀ i ∈ M , is d -dimensionalvoxel intensities and M is total voxel numbers. Suppose that x i , ∀ i ∈ M , is formed in an IID order from an underlyingdensity of GMM model ( P ( X ) ) with K ingredients, as in (6). P ( X | Θ) = K (cid:88) k =1 α k P k ( X | Z k , θ k ) , (6)where P k ( X | Z k , θ k ) , α k , and Z k are the k th ingredient ofGMM with parameters θ k , mix-up weights describing theprobability that a randomly chosen X is formed by k th element, and a vector of K pointer variables that are jointlyindependent, respectively, where (cid:80) Kk =1 α k = 1 . Therefore,the entire set of parameters for a GMM with K elements is Θ = { α , ..., α K , θ , ..., θ K } . We can model P k ( X | Z k , θ k ) as Gaussian density function as in (7) with the parameters of θ k = { µ k , Σ k } . P k ( X | θ k ) = 1(2 π ) d/ | Σ k | / · e ( − ( X − µ k ) T Σ − k ( X − µ k )) , (7)where ∀ k ∈ K . The parameters in the above equation, suchas µ k and Σ k , respectively denote the mean and co-varianceor standard deviation of k th Gaussian density function, P k ( X | θ k ) . From the known GMM parameters and using theBayes rule, the membership probabilities of a given observedvector ( x i ∈ X ) can be written, as in (8): W ik = α k · P k ( x i | θ k ) (cid:80) Kj =1 α j · P k ( x i | θ j ) , ∀ k ∈ K, (8)where (cid:80) Kk =1 W ik = 1 . In our pipeline, we propose to initialparameters of the EM algorithm using the probabilistic atlas.The convergence of the EM algorithm is recognized by esti-mating the log-likelihood value, as in (9), after each iterationand stopping when it seems not to increase significantly fromone iteration to the succeeding. log ( l (Θ)) = M (cid:88) i =1 log ( P ( X | Θ))= M (cid:88) i =1 (cid:18) log (cid:16) K (cid:88) k =1 α k · P k ( X | Z k , θ k ) (cid:17)(cid:19) . (9)The optimization of the EM algorithm in our pipeline es-sentially includes the following two steps to obtain optimalparameters of the GMM. VOLUME 4, 2016 et al. : Multi-class probabilistic atlas-based whole heart segmentation method in cardiac CT and MRI
Expectation-step (E-step):
It includes the following threesteps.Step 1: Denote the current parameter values (such as α k , µ k ,and Σ k , ∀ k ∈ K ) as Θ , where (cid:80) Kk =1 α k = 1 .Step 2: Compute membership probabilities ( W ik ) having thesize of M × K using the equation mentioned earlierin (8), ∀ x i ∈ X , ≤ i ≤ M , ∀ k ∈ K , where (cid:80) Kk =1 W ik = 1 .Step 3: Compute the log-likelihood ( L ) using (9) for thecurrent parameters ( Θ ), which is termed as L old . Maximization-step (M-step):
The matrix of membershipweights ( W ik ) is used to update the parameters Θ using thefollowing three steps.Step 1: Update the mixture weights ( α k , ∀ k ∈ K ) by using α newk = M (cid:80) Mi =1 W ik , where (cid:80) Kk =1 α k = 1 .Step 2: Update parameters, θ k = µ k , Σ k , ∀ k ∈ K , by usingthe following equations, µ newk = 1´ M k M (cid:88) i =1 W ik · x i , ∀ k ∈ K, Σ newk = 1´ M k M (cid:88) i =1 W ik · (cid:0) x i − µ newk (cid:1)(cid:0) x i − µ newk (cid:1) T , where ´ M k = M (cid:88) i =1 W ik , ∀ k ∈ K. Step 3: Compute the log-likelihood ( L ) using (9) for the up-dated parameters ( Θ new ), which is termed as L new .We run the EM algorithm until the convergence, which meanswe repeat until there is no significant change between L old and L new . Finally, the membership probabilities ( W ik ) areused to delineate WHS’s different substructures. d: PAS+EM In our proposed PAS+EM algorithm, we use posterior proba-bilities ( P ( X | Y ) ) and membership probabilities ( W ik ) fromPAS and EM algorithms, respectively, to delineate the dif-ferent structures for WHS. For doing so, we use the follow-ing equation, as in (10), to obtain the probabilities for ourPAS+EM algorithm. P P ASEM = W ik · P ( X | Y ) , ∀ k ∈ K, (10)where all P P ASEM , W ik , and P ( X | Y ) are M × K matrix.Finaly, in the end, we use P P ASEM to delineate the differentstructures for WHS.
C. CNN-BASED METHODS
We have implemented a CNN-based semantic segmenta-tion method besides our proposed atlas-based segmentationmethod to perform comprehensive ablation studies.Segmentation of medical image has attained enormousprogress notably since after the proposing of UNet[31]. Currently, CNN-based networks have been extensively practiced for the medical imaging domain, exceeding con-ventional image analysis techniques relying on hand-craftedfeatures [53]. However, the CNN-based network for segmen-tation incorporates two fundamental elements: the encoderand the decoder [31]. An encoder consists of convolutionaland pooling layers. The convolutional layers produce fea-ture maps, whereas the pooling layers continuously decreasethese feature maps’ dimension to gain more critical featureswith more eminent spatial invariance [17]. The decreasedresolution feature maps also enlarge the maps’ field-of-viewand diminish the computational expense [54]. The decoderprojects the distinctive lower resolution features discoveredby the encoder onto the higher resolution pixel space toachieve a compact pixel-wise labeling [55]. However, thesimple encoder-decoder network, named EDNet, in our im-plementation is depicted in Fig 4. The encoder in EDNet i npu t CT / M R I S e g m e n t e d V O I Encoder Decoder
Conv2D Pool2D Up2D Conv2D
FIGURE 4: A pictorial presentation of EDNet applying a pre-trained VGG-16 network as an encoder for transferring theprevious ImageNet knowledge to our WHS task. The samenumber of pooling and upsampling layers are used in theencoder and decoder to regain the output’s input resolutions.is a VGG-16 [56] with the pre-trained weight on ImageNet[57], which has five Convolutional Blocks (CB) and thirteenconvolutional layers. Each CB’s output is an input to the nextCB through a pooling layer with a stride of × . Hence, theencoder’s output feature map has a resolution of m/ × n/ for an input resolution of m × n . However, the decoder hasfive blocks to obtain the input resolutions of the output WHSmasks ( m × n ), where we apply 2D upsampling, with a strideof × , convolution with a kernel of × , and a batchnormalization [58] in each decoder block.However, the decreased feature maps due to pooling un-dergo spatial knowledge elimination injecting roughness,poor border knowledge, checkerboard artifacts, over-, andunder-segmentation in the segmented substructures [17],[31], [54], [59]. To overcome these problems, the authors in[31] introduced skip connections in a UNet, permitting thedecoder to retrieve the associated features discovered at allencoder steps that were missed due to subsampling in theencoder. The feature maps from the encoder’s antecedentlayers are concatenated with the decoder’s identical scalethrough the appliance of skip connections. Applying the skip VOLUME 4, 2016 hosh et al. : Multi-class probabilistic atlas-based whole heart segmentation method in cardiac CT and MRI connection of the popular UNet, we propose a VGG-UNet,where we have also employed the VGG-16 network as anencoder, as shown in Fig. 5. In our VGG-UNet, we apply i npu t CT / M R I S e g m e n t e d V O I Encoder Decoder
Conv2D Pool2D Up2D Conv2D
FIGURE 5: The architecture of our VGG-UNet, where weconcatenate the encoder’s features with the same scale fea-tures of the decoder to compensate for the lost spatial in-formation in the encoder. The same number of pooling andupsampling layers are used in the encoder and decoder toregain the output’s input resolutions.the skip connections holding ladder-like compositions [60]motivated by UNet to succeed in the pooling weaknesses. Allpooled layers of our network are concatenated channel-wiseto a deconvoluted feature map with identical dimensions,where it acts as an offsetting link for the spatial knowledgedropped due to subsampling in the encoder.Again, the authors in [54] combined features at variedcoarseness levels of the encoder in their popular FCN topolish the output segmented VOIs. In this article, we proposea VGG-FCN, as shown in Fig. 6, with the pre-trained VGG-16 network in the encoder. The output feature map of suchan encoder has -times fewer resolutions as VGG-16 hasfive pooling layers. The employment of × upsamplingcan output the segmented mask with the same size as theinput image, called FCN32s. Such an upsampling producesa coarse or rough mask as it utilizes only global informationfrom the more in-depth high-level features. The in-depth fea-tures are achieved while operating deeper, which causes thespatial location information lost. That indicates that outputfrom shallower layers has more location knowledge. If wecombine both local and global information, it can enhancethe segmentation result. The output from the fifth pooling ofVGG-16 is × upsampled and fused with the fourth poolingin our network. Then, the combined map is × upsampledand again linked with third pooling. Finally, we perform × upsampling, which provides a segmented WHS mask. Hence,the proposed VGG-FCN is also named VGG-FCN8s, wherewe use both the local and global information to obtain thefinal mask.We implemented three semantic segmentation networkvariants to compare them with the proposed Atlas-basedsegmentation method of WHS. We apply IoU as an objective i npu t CT / M R I S e g m e n t e d V O I Encoder Decoder c c
Conv2D Pool2D Up2D Conv2D
FIGURE 6: The structure of our VGG-FCN8s, where weapply a pre-trained VGG-16 as a feature learner. The outputWHS masks are obtained from the fused feature maps fromthe fifth, fourth, and third pooling layers of VGG-16. Thesame number of pooling and upsampling layers are usedin the encoder and decoder to regain the output’s inputresolutions.metric ( M seg ) in (11) throughout the WHS training for allnetworks, as mentioned above. M seg ( y, ˆ y ) = N (cid:88) i =1 y i × ˆ y iN (cid:88) i =1 y i + N (cid:88) i =1 ˆ y i − N (cid:88) i =1 y i × ˆ y i , (11)where the parameters, such as y , ˆ y , and N , respectivelyindicate the actual label, prognosticated label, and the totalvoxel numbers. The multiplicative term of y and ˆ y in theabove equation is the estimation of identity (intersection)connecting the actual and predicted VOIs. The binary cross-entropy is utilized as the WHS’s cost functions by thenetworks, such as EDNet, VGG-UNet, and VGG-FCN8s,optimized employing the Adam optimizer [61] with initialLearning Rate ( LR ) and exponential decay rates ( β , β )as LR = 0 . , β = 0 . , and β = 0 . , respectively,without AMSGrad variant. The LR is decreased subsequent epochs by . if validation loss obstructs progressing.The initial epochs are set as , which is suspend the net-work training utilizing a callback function after the validationloss becomes stagnated. D. EVALUATION
The extensive experimentations are conducted on a com-puter operating on
Windows-10 system with the followinghardware configuration: Intel ® Core TM i - HQ CPU @ . GHz processor with Install memory (RAM): . GB and GeForce GTX GPU with GB GDDR memory.The proposed pipeline and CNN-based models are designedutilizing the MATLAB R2020a and Python programminglanguage with various Python and Keras APIs. VOLUME 4, 2016 et al. : Multi-class probabilistic atlas-based whole heart segmentation method in cardiac CT and MRI
We use mean Volume Overlapping Error (mVOE), meanSensitivity (mSn), and mean Dice Similarity Coefficient(mDSC) to quantify the segmentation accuracy, which isdefined as follows: mV OE = 1 S S (cid:88) j =1 (cid:18) − T P j T P j + F N j + F P j (cid:19) ,mSn = 1 S S (cid:88) j =1 T P j T P j + F N j ,mDSC = 1 S S (cid:88) j =1 · T P j · T P j + F N j + F P j , where S , T P , F N , and
F P respectively denote the numberof slices in an unseen test image, true-positive region (VOIas a VOI), false-negative region (VOI as a background), andfalse-positive region (background as a VOI). mSn determinesthe percentage of correctly segmented VOI of the true VOI,whereas mVOE and mDSC estimate the difference and simi-larity between true and segmented VOIs.
III. RESULTS AND DISCUSSION
This section is dedicated to the presentation of extensiveexperimental results. In subsection III-A, we present andinterpret the results of eight different approaches (describedin subsections II-B & II-C) utilizing the both CT and MRIscans. We have grouped those eight distinct methods intothree categories: CNN-, MALP-, and probabilistic atlas-basemethods, where each group has several methods. Firstly, weperform ablation studies among different methods in eachgroup on the same dataset in subsections III-A1, III-A2, andIII-A3, respectively. Finally, in subsection III-B, we analyzethe best methods from all the groups for both the MRI andCT scans, as well as we compare other seventeen differentpublished methods with our best performing method on thesame dataset.
A. EXPERIMENTAL RESULTS
Table 3 demonstrates the obtained results from differentmethods for the WHS on both the CT and MRI scans expli-cating different quantitative metrics (see in subsection II-D).Fig. 7 exhibits the Box and Whisker visualization of allthe methods showing the spreads and centers of the DSC,VOE, and Sn metrics for all the 20 test images, either CTor MRI scans. Fig. 8 (a) and Fig. 8 (b) display the qualitativesegmentation results for both the CT (top) and MRI (bottom),respectively, from all the methods.
1) Results for CNN-based methods
The experimental results in Table 3 quantitatively demon-strate that the EDNet is the worst-performing method thanthe other two CNN-based methods, such as VGG-UNet andVGG-FCN8s, by the margins of . and . for CTscans and . and . for MRI scans, respectively, concerning the mDSC. Those two networks also defeat theEDNet for the other two metrics (mVOE and mSn) withsignificant margins (see in Table 3). Although those threenetworks have the same number of convolutional and poolinglayers in the encoder and decoder, they constructionallyvary in skip connection employment (see details in sub-section II-C). The results on both the imaging modalitiesreveal that the appliance of skip connection has outputtedbetter-segmented substructures, as the local information fromthe shallower layers is utilized to reconstruct output masksthrough the skip connection (see the results in Table 3).Again, it is seen from Fig. 7 that the DSC, VOE, and Snfrom the EDNet are sparse for both the CT and MRI scans,with significantly fewer median metrics, which demonstratesthat EDNet produces scattered results for each of the testingcases. On the other hand, the upper- and lower-whisker forall three metrics are closer for VGG-FCN8s and VGG-UNetthan the EDNet (see in Fig. 7), which shows better-robustnessof them comparing the EDNet.Furthermore, the qualitative results in Fig. 8 depictthat both the VGG-UNet and VGG-FCN8s provide better-segmented substructures of both the modalities (CT andMRI) than the EDNet. In some examples of the EDNet, noneof the target substructures are segmented, which provideVOE as . (see upper-whisker in Fig. 7 (b)). Theencoders in all three networks have 13-convolutional and 5-pooling layers, where the in-depth features obtained fromthem have lost spatial location information due to poolingin the encoders. Hence, the output masks from EDNet haveless local information, which is solved in the VGG-UNet andVGG-FCN8s due to the concatenation of local informationthrough the skip connection, providing an alternative pathfor the gradient with backpropagation. It is experimentallyvalidated in our investigations that the additional skippingpaths are beneficial for improving the segmentation resultsof different heart substructures.Again, it is also remarkable that VGG-FCN8s even ex-ceeds the VGG-UNet (see the corresponding results in Ta-ble 3, Fig. 7, and Fig. 8) by the margins of . and . respectively for CT and MRI scans in terms of mDSC.The former network further outperforms the latter networkconcerning the other two metrics, such as mVOE and mSn,with significant margins for the CT scans. The concate-nation of low-level features from the encoder’s antecedentlayers with the equivalent decoder scale in VGG-UNet isthe possible reason for failing VGG-UNet than VGG-FCN8s[16], [35]. Unessentially, an aggregation of the correspondingscale feature maps from the beginning layer of the encoder,is observed as a weakness of the UNet, as it imposes anundesirable merging procedure, forcing aggregation barely atthe corresponding scale feature maps of the encoder and de-coder, which was similarly experimentally validated in [35]and our previous article for other medical imaging modality[16]. On the other hand, in VGG-FCN8s, we fuse the featuremaps from the encoder’s different coarseness starting fromthe third block of the encoder, making it a winner of three VOLUME 4, 2016 hosh et al. : Multi-class probabilistic atlas-based whole heart segmentation method in cardiac CT and MRI
TABLE 3: The quantitative WHS results from our different methods utilizing both the CT and MRI scans. Best metrics areunderlined.
CT MRIDifferent methods mVOE mSn mDSC mVOE mSn mDSCEDNet . ± .
080 0 . ± .
135 0 . ± .
126 0 . ± .
070 0 . ± .
154 0 . ± . VGG-UNet . ± .
054 0 . ± .
086 0 . ± .
048 0 . ± .
049 0 . ± .
120 0 . ± . VGG-FCN8s . ± .
053 0 . ± .
062 0 . ± .
052 0 . ± .
051 0 . ± .
128 0 . ± . Median . ± .
063 0 . ± .
051 0 . ± .
044 0 . ± .
169 0 . ± .
163 0 . ± . MALP MVF . ± .
051 0 . ± .
046 0 . ± .
033 0 . ± .
180 0 . ± .
169 0 . ± . PAS . ± .
046 0 . ± .
026 0 . ± .
027 0 . ± .
162 0 . ± .
126 0 . ± . EM . ± .
040 0 . ± .
063 0 . ± .
063 0 . ± .
038 0 . ± .
100 0 . ± . PAS+EM . ± .
144 0 . ± .
097 0 . ± .
107 0 . ± .
197 0 . ± .
187 0 . ± . E D N e t V G G - U N e t V G G - F C N s M A L P M A L P P A S E M P A S E M M e t r i c s ( D S C , V O E , & S n ) DSCVOESn (a) WHS results utilizing CT scans E D N e t V G G - U N e t V G G - F C N s M A L P M A L P P A S E M P A S E M M e t r i c s ( D S C , V O E , & S n ) DSCVOESn (b) WHS results utilizing MRI scans
FIGURE 7: The Box and Whisker visualization of DSC (blue), VOE (black), and Sn (red) for the WHS utilizing (a) CT and(b) MRI scans employing eight different methods, as described in subsections II-B & II-C. The MALP and MALP indicatea MALP method with the median and MVF schemes, respectively.implemented CNN-based WHS approaches.
2) Results for MALP-based methods
The label propagated segmentation results utilizing differentdeformed atlas images are quantitatively and qualitativelymanifested in Table 3, Fig. 7, and Fig. 8. The median andMVF schemes of level propagation, as described in subsec-tion II-B2, generate the heart segmentation results with themVOEs of . and . , respectively, for CT scansand . and . , respectively, for MRI scans. Table 3demonstrates that the MVF scheme outperforms the medianmethod of MALP by the margins of . and . re-spectively for CT and MRI modalities for mDSC. The formerMALP method also outperforms the latter MALP methodwith significant margins concerning the other two metrics(mVOE and mSn). The Box and Whisker visualization ofall three metrics in Fig. 7 for both the methods demonstratethe superiority of the MVF scheme than the median strategyfor both the imaging modalities. Fig. 8 (top) and Fig. 8(bottom) exhibit the qualitative results respectively for CTand MRI scans for both the median (MALP ) and MVF (MALP ) schemes. Those results qualitatively confirm thatthe segmented substructures from MALP suffer from theoutliers (see in fifth and sixth columns of Fig. 8), wheremost of the organs are erroneously labeled with other organs.All the experimental results reveal that the MVF scheme hasbetter dealt with the outliers as it counts the majority ofthe labels from the voting candidates, whereas the medianmethod estimates the median values of the counter, whichmay not be matched by the majority voters.
3) Results for probabilistic atlas-based methods
The WHS results of the probabilistic atlas are exhibited inTable 3, Fig. 7, and Fig. 8, where we employ our threemethods, such as PAS, EM, and PAS+EM (see details insubsection II-B2). The PAS, EM, and PAS+EM schemes ofprobabilistic atlas provide the heart segmentation results withthe mVOEs of . , . , and . , respectively, forCT scans and . , . , and . , respectively, forMRI scans. Table 3 exhibit that the PAS scheme exceeds theother two methods, such as EM and PAS+EM, by the marginsof . and . for CT scans concerning the mDSC, VOLUME 4, 2016 et al. : Multi-class probabilistic atlas-based whole heart segmentation method in cardiac CT and MRI
Ground-truth
EDNet VGG-FCN8s VGG-UNet
MALP MALP PAS EM
PASEM
VOE=0.157 VOE=0.908
VOE=0.065
VOE=0.733 VOE=0.132VOE=0.187VOE=0.561VOE=0.704 VOE=0.654VOE=0.633VOE=0.853 VOE=0.491VOE=0.893VOE=0.200VOE=0.314VOE=0.355VOE=0.684VOE=0.661VOE=0.821 VOE=0.418
VOE=0.588VOE=0.919
VOE=0.199VOE=0.319VOE=0.655VOE=0.475
VOE=0.292
VOE=0.911VOE=0.074VOE=0.225 VOE=0.185VOE=0.505
VOE=0.259
VOE=0.551VOE=0.564
VOE=0.087VOE=0.822
VOE=0.945VOE=0.225 VOE=0.449 (a) Qualitative WHS results in CT scans applying different methods
Ground-truth
EDNet VGG-FCN8s VGG-UNet
MALP MALP PAS EM
PASEM
VOE=0.582VOE=0.118 VOE=0.895VOE=0.697
VOE=0.241
VOE=1.00 VOE=0.700 VOE=0.366 VOE=0.210 VOE=0.103 VOE=0.849 VOE=0.249VOE=0.900VOE=0.099VOE=0.214
VOE=0.335
VOE=0.688VOE=0.738VOE=1.00VOE=0.953 VOE=0.645 VOE=0.667 VOE=0.361 VOE=0.231VOE=0.776VOE=0.733VOE=0.725VOE=0.982 VOE=0.893VOE=0.975VOE=0.521VOE=0.637
VOE=0.788VOE=0.980
VOE=0.887VOE=0.766 VOE=0.655VOE=0.810
VOE=0.766
VOE=0.952 (b) Qualitative WHS results in MRI scans applying different methods
FIGURE 8: The qualitative WHS results from our different methods in CT (top) and MRI (bottom) scans, where the first threeand last two rows of both the figures are for the best- and worst-performing examples for the best-performing method (PAS),respectively. The MALP and MALP indicate a MALP method with the median and MVF schemes, respectively. VOLUME 4, 2016 hosh et al. : Multi-class probabilistic atlas-based whole heart segmentation method in cardiac CT and MRI respectively, whereas it also beats them by the margins of . and . for MRI scans, respectively, in terms ofthe mDSC. Similarly, the EM and EM+PAS are also defeatedby the proposed PAS method for mVOE and mSn for both thechest imaging modalities with considerable margins (see inTable 3). The spreads and centers of the DSC, VOE, and Snmetrics for all the 20 test images (either CT or MRI scans), asexhibited in Fig. 7, also reveal the supremacy of the proposedPAS method over the other two methods (EM and EM+PAS).The proposed PAS technique estimates the voxel class andassigns a target voxel label from the posterior distributionof the Bayesian framework, where the posterior distributionis determined from the likelihood tissue model and prioranatomical knowledge by a MAP rule. Such an estimation ofposterior distribution quantifies its expected probability valueand the uncertainty associated with it, which probably makesthe PAS algorithm a winner than the EM algorithm. As ourEM algorithm is a single variate, it has less representation ofthe heart substructure features. The EM algorithm is proneto converging to local minima when dealing with a singlevariate dataset, which provides an overfitted model for theheart segmentation.However, incorporating the PAS algorithm with an EMalgorithm extends the WHS outcomes by the margins of . and . in terms of mDSC respectively for CTand MRI modalities. Still, the PAS technique is a defeatingmethod. Again, Fig. 8 (top) and Fig. 8 (bottom) show thequalitative WHS outcomes for the CT and MRI scans forthe PAS, EM, and PAS+EM designs, respectively. Thoseresults show that the segmented substructures from EM andPAS+EM yield the outliers substructures (see in eight andninth columns of Fig. 8), where the EM results suffer severelyby the outliers. The embodiment of PAS with the EM hassignificantly reduced the outlier effect from the WHS resultsof the EM technique, as qualitatively depicted in Fig. 8. B. RESULTS COMPARISON
Comparing all the results as mentioned earlier, the proposedPAS is the best performing method for the WHS for boththe utilized dataset modalities, such as CT and MRI scans,which are manifested and visualized in Table 3, Fig. 7 andFig. 8. The proposed and designed methods are classifiedinto three categories: CNN-based, MALP-based, and prob-abilistic atlas-based, where the former approach works onthe 2D slice of the 3D CT and MRI modalities, while theformer two methods work on the whole 3D CT and MRIvolumes. The experimentation reveals that the 3D image-based approach conquers the 2D image-based techniques.Our experimental WHS results also demonstrate that the 3Datlas-based approaches beat CNN-based 2D methods. Thisresult’s probable reasons are that we take only axial slicesfrom the 3D CT and MRI scans to train and evaluate thenetwork. Although the 2D CNNs have much lighter compu-tation and higher inference speed, it neglects the informationbetween adjacent slices, which hinders the improvementof segmentation accuracy. Hence, the 3D CNNs can be a powerful model for learning representations for volumetricdata and perceiving the volumetric spatial information, whichwill be analyzed for the same WHS task in the future.Fig. 9 reflects the Box and Whisker presentation of themDSC of our proposed PAS algorithm on both the CT andMRI modalities for seven different substructures of the heart(see subsection I-A). The CT scans produce the best WHSresults than the MRI scans for all the substructures in termsof mean and median values of the mDSC. The substructure’ssegmentation results utilizing the MRI images also have ahigher interquartile range than the CT images (see in Fig. 9),which indicates less robustness of the MRI images for theWHS. WHS’s poor results utilizing the MRI images overCT images are also reported in previously published articles[1], [5], [30], [38]. The segmentation results, especially forLA, RA, Myo, AO, and PA, utilizing the CT images, arepraiseworthy as they have a high median value and signif-icantly less interquartile range than MRI images. Although
LV RV LA RA Myo AO PA0.30.40.50.60.70.80.91.0 D S C Utilizing CT scansUtilizing MRI scans
FIGURE 9: The Box and Whisker visualization of DSC eachsubstructure segmentation of WHS utilizing CT (black) andMRI (blue) scans. Our best performing segmentation method(PAS) for CT and MRI scans is employed to obtain thisvisualization.MRI images’ performance is less than CT, merging themcan further improve the WHS results with the multi-modalinput data, as it will have better feature presentations of dif-ferent heart substructures [38]. Therefore, the future workingdirection can concentrate on the multi-modal utilization ofthe MM-WHS-2017 dataset applying this article’s proposedtechniques.Table 4 shows the WHS results from our best perform-ing method and other state-of-the-art methods on the samedataset utilizing both the CT and MRI scans. The compara-tive results, as in Table 4, demonstrate that the proposed PASmethod with the CT scans outperforms the recent state-of-the-art approach in [38] by the margins of . and . concerning the mVOE and mDSC, respectively. The authorsin the second-best method (MMTLNet) [38] transferred theMRI image information from the source domain to the targetCT domain through the adversarial training without con-sidering their spatial alignment, which could be the viablereasons for realizing the WHS multi-modality approach. The VOLUME 4, 2016 et al. : Multi-class probabilistic atlas-based whole heart segmentation method in cardiac CT and MRI
TABLE 4: Comparative results of our best performing method for WHS and state-of-the-art methods for the same task on boththe MRI and CT scans and same dataset. Best and second-best metrics are underlined and double-underlined, respectively.
MetricsDifferent methods Year Modalities mVOE mSn mDSCMulti-scale patch and multi-modality atlas-based WHS [15]
CT+MRI . − . Multi-atlas registration-based approach [62]
MRI . − . Multi-planar CNN [5]
MRI .
259 0 .
831 0 . Multi-planar CNN [5] CT .
187 0 .
866 0 . Multi-label FCN [30]
MRI . − . Multi-label FCN [30] CT . − . CFUN: faster R-CNN + 3D UNet [19] CT . − . HFANet: deep heterogeneous feature aggregation network [28] CT . − . Two-stage UNet with adaptive threshold window [4]
CT+MRI . − . Multi-depth fusion of 3D UNet combining local and global features [34] CT . − . Averaging of 10 different algorithm [1]
MRI . − . Averaging of 10 different algorithm [1] CT . − . Dual-Teacher: integrating intra-domain and inter-domain teachers [26]
CT+MRI . − . MMTLNet: multi-modality transfer learning with adversarial training [38]
MRI . − . MMTLNet: multi-modality transfer learning with adversarial training [38] CT . − . MvMM-RegNet: multivariate mixture model with registration framework [63]
MRI . − .
3D UNet incorporating of the principal component analysis for augmentation [64] CT . − . Our proposed probabilistic atlas-based WHS .
291 0 .
825 0 . Our proposed probabilistic atlas-based WHS .
145 0 .
919 0 . other two well-performing methods (see in Table 4), such asMulti-label FCN [30] and Multi-depth fusion of 3D UNetcombining local and global features [34], also carry possibledrawbacks for being defeated by our proposed approach. Theformer strategy applied two stages, wherein the first stage, theVOIs are selected for the essential second stage. The errorsin VOI selection could lead to erroneous heart segmentationresults. The latter approach employed the fusion from thedifferent depths of the CNN network, where they fused theinput block to the along with other depths to generate thefinal WHS VOIs. The concatenation of the input block withthe same scale outermost decoder’s block probably hamperthe precise output, as it is also proven in our VGG-UNet andVGG-FCN8s experiments (see in Table 3), and the articles in[16], [35]. IV. CONCLUSIONS AND FUTURE WORK
Accurate in the whole heart’s segmentation is crucial indeveloping clinical applications, although it is very challeng-ing due to diverse artifacts, image variability, and patient-specific properties. This article has introduced and exploreda robust and accurate pipeline for the WHS utilizing twodifferent heart imaging modalities, such as CT and MRIscans. It is experimentally validated that incorporating theprior anatomical knowledge represented by probabilistic at-las into the Bayes inference to delineate seven different heartsubstructures has better-segmented results while utilizing theCT scans and employing our multi-resolution non-rigid regis-tration pipeline. As the 2D CNNs fail to provide satisfactoryWHS results, future research will focus our research directionon training and evaluating networks in multiple directions(coronal, sagittal, and axial) to combine all the directionalWHS results. We will also design an end-to-end 3D seg-mentation network for comprehensive ablation studies. Therecommended pipeline will be applied to other domains forvolumetric medical image segmentation to verify its versatil- ity and generability.
CONFLICT OF INTEREST
The authors have not any conflicts to disclose this research.
REFERENCES [1] X. Zhuang, L. Li, C. Payer, D. Štern, M. Urschler, M. P. Heinrich, J. Oster,C. Wang, Ö. Smedby, C. Bian, et al., “Evaluation of algorithms for multi-modality whole heart segmentation: an open-access grand challenge,”Medical image analysis, vol. 58, p. 101537, 2019.[2] S. Mendis, P. Puska, B. Norrving, W. H. Organization, et al., Globalatlas on cardiovascular disease prevention and control. World HealthOrganization, 2011.[3] D. Kang, J. Woo, C. J. Kuo, P. J. Slomka, D. Dey, and G. Germano, “Heartchambers and whole heart segmentation techniques,” Journal of ElectronicImaging, vol. 21, no. 1, p. 010901, 2012.[4] T. Liu, Y. Tian, S. Zhao, X. Huang, and Q. Wang, “Automatic whole heartsegmentation using a two-stage u-net framework and an adaptive thresholdwindow,” IEEE Access, vol. 7, pp. 83 628–83 636, 2019.[5] A. Mortazi, J. Burt, and U. Bagci, “Multi-planar deep segmentationnetworks for cardiac substructures from mri and ct,” in InternationalWorkshop on Statistical Atlases and Computational Models of the Heart.Springer, 2017, pp. 199–206.[6] Q. Dou, L. Yu, H. Chen, Y. Jin, X. Yang, J. Qin, and P.-A. Heng, “3d deeplysupervised network for automated segmentation of volumetric medicalimages,” Medical image analysis, vol. 41, pp. 40–54, 2017.[7] T. Heimann, B. Van Ginneken, M. A. Styner, Y. Arzhaeva, V. Aurich,C. Bauer, A. Beck, C. Becker, R. Beichel, G. Bekes, et al., “Comparisonand evaluation of methods for liver segmentation from ct datasets,” IEEEtransactions on medical imaging, vol. 28, no. 8, pp. 1251–1265, 2009.[8] A. Radtke, S. Nadalin, G. Sotiropoulos, E. Molmenti, T. Schroeder,C. Valentin-Gamazo, H. Lang, M. Bockhorn, H. Peitgen, C. Broelsch,et al., “Computer-assisted operative planning in adult living donor livertransplantation: a new way to resolve the dilemma of the middle hepaticvein,” World journal of surgery, vol. 31, no. 1, p. 175, 2007.[9] H.-P. Meinzer, M. Thorn, and C. E. Cárdenas, “Computerized planning ofliver surgery—an overview,” Computers & Graphics, vol. 26, no. 4, pp.569–576, 2002.[10] J. Peters, O. Ecabert, C. Meyer, H. Schramm, R. Kneser, A. Groth,and J. Weese, “Automatic whole heart segmentation in static magneticresonance image volumes,” in International Conference on Medical ImageComputing and Computer-Assisted Intervention. Springer, 2007, pp.402–410.[11] D. F. Pace, A. V. Dalca, T. Geva, A. J. Powell, M. H. Moghari, andP. Golland, “Interactive whole-heart segmentation in congenital heart VOLUME 4, 2016 hosh et al. : Multi-class probabilistic atlas-based whole heart segmentation method in cardiac CT and MRI disease,” in International Conference on Medical Image Computing andComputer-Assisted Intervention. Springer, 2015, pp. 80–88.[12] A. Atehortúa, M. A. Zuluaga, S. Ourselin, D. Giraldo, and E. Romero,“Automatic segmentation of 4d cardiac mr images for extraction of ven-tricular chambers using a spatio-temporal approach,” in Medical Imaging2016: Image Processing, vol. 9784. International Society for Optics andPhotonics, 2016, p. 978435.[13] F. Zhao, H. Hu, Y. Chen, J. Liang, X. He, and Y. Hou, “Accurate segmen-tation of heart volume in cta with landmark-based registration and fullyconvolutional network,” IEEE Access, vol. 7, pp. 57 881–57 893, 2019.[14] V. H. C. de Albuquerque, D. d. A. Rodrigues, R. F. Ivo, S. A. Peixoto,T. Han, W. Wu, and P. P. Rebouças Filho, “Fast fully automatic heart fatsegmentation in computed tomography datasets,” Computerized MedicalImaging and Graphics, vol. 80, p. 101674, 2020.[15] X. Zhuang and J. Shen, “Multi-scale patch and multi-modality atlases forwhole heart segmentation of mri,” Medical image analysis, vol. 31, pp.77–87, 2016.[16] M. K. Hasan, M. A. Alam, M. T. E. Elahi, S. Roy, and R. Martí, “Drnet:Segmentation and localization of optic disc and fovea from diabeticretinopathy image,” Artificial Intelligence in Medicine, p. 102001, 2020.[17] M. K. Hasan, L. Dahal, P. N. Samarakoon, F. I. Tushar, and R. Martí,“Dsnet: Automatic dermoscopic skin lesion segmentation,” Computers inBiology and Medicine, p. 103738, 2020.[18] D. M. Vigneault, W. Xie, C. Y. Ho, D. A. Bluemke, and J. A. Noble, “ ω -net(omega-net): fully automatic, multi-view cardiac mr detection, orientation,and segmentation with deep neural networks,” Medical image analysis,vol. 48, pp. 95–106, 2018.[19] Z. Xu, Z. Wu, and J. Feng, “Cfun: Combining faster r-cnn and u-net network for efficient whole heart segmentation,” arXiv preprintarXiv:1812.04914, 2018.[20] X. Ding, Y. Peng, C. Shen, and T. Zeng, “Cab u-net: An end-to-endcategory attention boosting algorithm for segmentation,” ComputerizedMedical Imaging and Graphics, vol. 84, p. 101764, 2020.[21] T. Han, R. F. Ivo, D. d. A. Rodrigues, S. A. Peixoto, V. H. C. de Albu-querque, and P. P. Rebouças Filho, “Cascaded volumetric fully convolu-tional networks for whole-heart and great vessel 3d segmentation,” FutureGeneration Computer Systems, 2020.[22] J. M. Wolterink, T. Leiner, M. A. Viergever, and I. Išgum, “Dilated convo-lutional neural networks for cardiovascular mr segmentation in congenitalheart disease,” in Reconstruction, segmentation, and analysis of medicalimages. Springer, 2016, pp. 95–102.[23] M. A. Zuluaga, B. Biffi, A. M. Taylor, S. Schievano, T. Vercauteren, andS. Ourselin, “Strengths and pitfalls of whole-heart atlas-based segmenta-tion in congenital heart disease patients,” in Reconstruction, Segmentation,and Analysis of Medical Images. Springer, 2016, pp. 139–146.[24] C. Wang, Q. Wang, and Ö. Smedby, “Automatic heart and vessel seg-mentation using random forests and a local phase guided level setmethod,” in Reconstruction, Segmentation, and Analysis of Medical Im-ages. Springer, 2016, pp. 159–164.[25] L. Yu, X. Yang, J. Qin, and P.-A. Heng, “3d fractalnet: dense volumetricsegmentation for cardiovascular mri volumes,” in Reconstruction, segmen-tation, and analysis of medical images. Springer, 2016, pp. 103–110.[26] K. Li, S. Wang, L. Yu, and P.-A. Heng, “Dual-teacher: Integrating intra-domain and inter-domain teachers for annotation-efficient cardiac segmen-tation,” in International Conference on Medical Image Computing andComputer-Assisted Intervention. Springer, 2020, pp. 418–427.[27] M. Habijan, H. Leventi´c, I. Gali´c, and D. Babin, “Whole heart segmen-tation from ct images using 3d u-net architecture,” in 2019 InternationalConference on Systems, Signals and Image Processing (IWSSIP). IEEE,2019, pp. 121–126.[28] H. Zheng, L. Yang, J. Han, Y. Zhang, P. Liang, Z. Zhao, C. Wang, and D. Z.Chen, “Hfa-net: 3d cardiovascular image segmentation with asymmetricalpooling and content-aware fusion,” in International Conference on MedicalImage Computing and Computer-Assisted Intervention. Springer, 2019,pp. 759–767.[29] X. Dong, Y. Lei, T. Wang, M. Thomas, L. Tang, W. J. Curran, T. Liu, andX. Yang, “Automatic multiorgan segmentation in thorax ct images usingu-net-gan,” Medical physics, vol. 46, no. 5, pp. 2157–2168, 2019.[30] C. Payer, D. Štern, H. Bischof, and M. Urschler, “Multi-label wholeheart segmentation using cnns and anatomical label configurations,” inInternational Workshop on Statistical Atlases and Computational Modelsof the Heart. Springer, 2017, pp. 190–198.[31] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networksfor biomedical image segmentation,” in International Conference on Medi- cal image computing and computer-assisted intervention. Springer, 2015,pp. 234–241.[32] C. Wang and Ö. Smedby, “Automatic whole heart segmentation usingdeep learning and shape context,” in International Workshop on StatisticalAtlases and Computational Models of the Heart. Springer, 2017, pp.242–249.[33] Q. Tong, M. Ning, W. Si, X. Liao, and J. Qin, “3d deeply-supervised u-netbased whole heart segmentation,” in International Workshop on StatisticalAtlases and Computational Models of the Heart. Springer, 2017, pp.224–232.[34] C. Ye, W. Wang, S. Zhang, and K. Wang, “Multi-depth fusion networkfor whole-heart ct image segmentation,” IEEE Access, vol. 7, pp. 23 421–23 429, 2019.[35] Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang, “Unet++:Redesigning skip connections to exploit multiscale features in imagesegmentation,” IEEE transactions on medical imaging, vol. 39, no. 6, pp.1856–1867, 2019.[36] Y. Mo, F. Liu, D. McIlwraith, G. Yang, J. Zhang, T. He, and Y. Guo, “Thedeep poincaré map: A novel approach for left ventricle segmentation,” inInternational Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2018, pp. 561–568.[37] A. Gubern-Mérida, M. Kallenberg, R. Martí, and N. Karssemeijer, “Multi-class probabilistic atlas-based segmentation method in breast mri,” inIberian Conference on Pattern Recognition and Image Analysis. Springer,2011, pp. 660–667.[38] X. Liao, Y. Qian, Y. Chen, X. Xiong, Q. Wang, and P.-A. Heng, “Mmtlnet:Multi-modality transfer learning network with adversarial training for 3dwhole heart segmentation,” Computerized Medical Imaging and Graphics,vol. 85, p. 101785, 2020.[39] D. Rueckert, L. I. Sonoda, C. Hayes, D. L. Hill, M. O. Leach, and D. J.Hawkes, “Nonrigid registration using free-form deformations: applicationto breast mr images,” IEEE transactions on medical imaging, vol. 18, no. 8,pp. 712–721, 1999.[40] C. R. Maurer, D. L. Hill, A. J. Martin, H. Liu, M. McCue, D. Rueckert,D. Lloret, W. A. Hall, R. E. Maxwell, D. J. Hawkes, et al., “Investigationof intraoperative brain deformation using a 1.5-t interventional mr system:preliminary results,” IEEE transactions on medical imaging, vol. 17, no. 5,pp. 817–825, 1998.[41] M. Lorenzo-Valdés, G. I. Sanchez-Ortiz, R. Mohiaddin, and D. Rueckert,“Atlas-based segmentation and tracking of 3d cardiac mr images usingnon-rigid registration,” in International conference on medical imagecomputing and computer-assisted intervention. Springer, 2002, pp. 642–650.[42] P. H. Eilers and B. D. Marx, “Flexible smoothing with b-splines andpenalties,” Statistical science, pp. 89–102, 1996.[43] S. Klein, J. P. Pluim, M. Staring, and M. A. Viergever, “Adaptive stochasticgradient descent optimisation for image registration,” International journalof computer vision, vol. 81, no. 3, p. 227, 2009.[44] D. Mattes, D. R. Haynor, H. Vesselle, T. K. Lewellen, and W. Eubank,“Pet-ct image registration in the chest using free-form deformations,”IEEE transactions on medical imaging, vol. 22, no. 1, pp. 120–128, 2003.[45] I. Isgum, M. Staring, A. Rutten, M. Prokop, M. A. Viergever, andB. Van Ginneken, “Multi-atlas-based segmentation with local decisionfusion—application to cardiac and aortic segmentation in ct scans,” IEEEtransactions on medical imaging, vol. 28, no. 7, pp. 1000–1010, 2009.[46] S. Klein, M. Staring, K. Murphy, M. A. Viergever, and J. P. Pluim,“Elastix: a toolbox for intensity-based medical image registration,” IEEEtransactions on medical imaging, vol. 29, no. 1, pp. 196–205, 2009.[47] A. Klein and J. Hirsch, “Mindboggle: a scatterbrained approach to auto-mate brain labeling,” NeuroImage, vol. 24, no. 2, pp. 261–280, 2005.[48] C. Ciofolo and C. Barillot, “Atlas-based segmentation of 3d cerebralstructures with competitive level sets and fuzzy control,” Medical imageanalysis, vol. 13, no. 3, pp. 456–470, 2009.[49] Y. Wu, S. K. Warfield, I. L. Tan, W. M. Wells III, D. S. Meier, R. A. vanSchijndel, F. Barkhof, and C. R. Guttmann, “Automated segmentation ofmultiple sclerosis lesion subtypes with multichannel mri,” NeuroImage,vol. 32, no. 3, pp. 1205–1215, 2006.[50] M. Cabezas, A. Oliver, X. Lladó, J. Freixenet, and M. B. Cuadra, “Areview of atlas-based segmentation for magnetic resonance brain images,”Computer methods and programs in biomedicine, vol. 104, no. 3, pp.e158–e177, 2011.[51] I. S. Amiri, O. A. Akanbi, and E. Fazeldehkordi, A machine-Learningapproach to phishing detection and defense. Syngress, 2014. VOLUME 4, 2016 et al. : Multi-class probabilistic atlas-based whole heart segmentation method in cardiac CT and MRI [52] H. Park, P. H. Bland, and C. R. Meyer, “Construction of an abdominalprobabilistic atlas and its application in segmentation,” IEEE Transactionson medical imaging, vol. 22, no. 4, pp. 483–492, 2003.[53] N. Tajbakhsh, L. Jeyaseelan, Q. Li, J. N. Chiang, Z. Wu, and X. Ding,“Embracing imperfect datasets: A review of deep learning solutions formedical image segmentation,” Medical Image Analysis, p. 101693, 2020.[54] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networksfor semantic segmentation,” in Proceedings of the IEEE conference oncomputer vision and pattern recognition, 2015, pp. 3431–3440.[55] A. Garcia-Garcia, S. Orts-Escolano, S. Oprea, V. Villena-Martinez,P. Martinez-Gonzalez, and J. Garcia-Rodriguez, “A survey on deep learn-ing techniques for image and video semantic segmentation,” Applied SoftComputing, vol. 70, pp. 41–65, 2018.[56] K. Simonyan and A. Zisserman, “Very deep convolutional networks forlarge-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.[57] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet:A large-scale hierarchical image database,” in 2009 IEEE conference oncomputer vision and pattern recognition. Ieee, 2009, pp. 248–255.[58] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deepnetwork training by reducing internal covariate shift,” arXiv preprintarXiv:1502.03167, 2015.[59] A. Odena, V. Dumoulin, and C. Olah, “Deconvolution and checkerboardartifacts,” Distill, vol. 1, no. 10, p. e3, 2016.[60] A. Rasmus, M. Berglund, M. Honkala, H. Valpola, and T. Raiko, “Semi-supervised learning with ladder networks,” in Advances in neural informa-tion processing systems, 2015, pp. 3546–3554.[61] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”arXiv preprint arXiv:1412.6980, 2014.[62] M. P. Heinrich and J. Oster, “Mri whole heart segmentation using dis-crete nonlinear registration and fast non-local fusion,” in InternationalWorkshop on Statistical Atlases and Computational Models of the Heart.Springer, 2017, pp. 233–241.[63] X. Luo and X. Zhuang, “Mvmm-regnet: A new image registration frame-work based on multivariate mixture model and neural network estimation,”in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2020, pp. 149–159.[64] M. Habijan, H. Leventi´c, I. Gali´c, and D. Babin, “Neural network basedwhole heart segmentation from 3d ct images,” International journal ofelectrical and computer engineering systems, vol. 11, no. 1, pp. 25–31,2020.
TARUN KANTI GHOSH was born in Jashore,Khulna, Bangladesh, in 1983 and earned the B.S.and M.S. degrees in biotechnology and genetic en-gineering from the Khulna University of Khulna,Bangladesh, in 2011 and another M.S. degree inbiomedical engineering from Khulna Universityof Engineering & Technology (KUET), Khulna,Bangladesh, in 2016. From 2006 to 2008, he wasa Research Fellow with the Animal Cell Cultureand Molecular Biology Laboratory. He is currentlya Ph.D. student in biomedical engineering at KUET. He is the author ofthree international journal and conference articles. His research interests in-clude biomedical signal and image processing, bioelectricity, bioinformatics,biomedical systems modeling and simulation, computational intelligence inbiomedical engineering.
MD. KAMRUL HASAN was born in Tangail,Bangladesh, in 1992 and received B. Sc. and M.Sc. engineering degrees in Electrical and Elec-tronic Engineering (EEE) from Khulna Universityof Engineering & Technology (KUET) in 2014and 2017, respectively. He received another M.Sc. in Medical Imaging and Application (MAIA)from France (University of Burgundy), Italy (Uni-versity of Cassino and Southern Lazio), and Spain(University of Girona) as an Erasmus scholar in2019. Currently, Mr. Hasan is serving as an Assistant Professor at KUETin the EEE department. He analyzed different medical image modalitiesand machine learning during the MAIA study to build a generic computer-aided diagnosis system. His research interest includes medical image anddata analysis, machine learning, deep convolutional neural network, medicalimage reconstruction, and surgical robotics in minimally invasive surgery.Mr. Hasan is currently a supervisor of several undergraduate students onthe classification, segmentation, and registration of medical images withdifferent modalities. He has already published many research articles onmedical image and signal processing in different international journals andconferences.
SHIDHARTHO ROY received the B.Sc. degreein electrical and electronic engineering from theKhulna University of Engineering & Technology(KUET), in 2020. His undergraduate thesis wason “EEG based Brain-Computer Interaction withdifferent stimuli using machine learning.” He iscurrently working on Bio-signal processing, Med-ical Imaging, 3D Reconstruction of Medical Im-ages, Renewables, and Computer-Assisted Inter-ventions. His research interests include artificialintelligence, brain-computer interface, and deep learning. Mr. Roy awardedseven national awards so far for his work. His previous works were presentedat TENSYMP 2020, ICAEE 2019, ICIECE 2019, EICT 2019, and ArtificialIntelligence in Medicine (AIIM, Elsevier).
MD. ASHRAFUL ALAM is studying Electricaland Electronic Engineering (EEE) at Khulna Uni-versity of Engineering & Technology (KUET).Currently, he is working on Medical Imaging. Hisinterests include medical image and data process-ing, computer vision, and deep learning. Mr. Alamawarded two national awards so far for his work inthe idea development project. His previous workswere presented at IEEE Access and Artificial In-telligence in Medicine (AIIM, Elsevier). VOLUME 4, 2016 hosh et al. : Multi-class probabilistic atlas-based whole heart segmentation method in cardiac CT and MRI
EKLAS HOSSAIN (M’09, SM’17) received hisPh. D. from the College of Engineering and Ap-plied Science at University of Wisconsin Milwau-kee (UWM). He received his MS in Mechatron-ics and Robotics Engineering from InternationalIslamic University of Malaysia, Malaysia, in 2010and BS in Electrical and Electronic Engineeringfrom Khulna University of Engineering & Tech-nology, Bangladesh, in 2006. Dr. Hossain has beenworking in distributed power systems and renew-able energy integration for the last ten years, and he has published severalresearch papers and posters in this field. He is now involved with severalresearch projects on renewable energy and grid-tied microgrid system atOregon Tech, as an Assistant Professor in the Department of ElectricalEngineering and Renewable Energy since 2015. He is a senior memberof the Association of Energy Engineers (AEE). He is currently servingas an Associate Editor of IEEE Access. He is working as an AssociateResearcher at the Oregon Renewable Energy Center (OREC). He is aregistered Professional Engineer (PE) in the state of Oregon, USA. He isalso a Certified Energy Manager (CEM) and Renewable Energy Professional(REP). His research interests include modeling, analysis, design, and controlof power electronic devices; energy storage systems; renewable energysources; integration of distributed generation systems; microgrid and smartgrid applications; robotics, and advanced control system. He is the winnerof the Rising Faculty Scholar Award in 2019 from the Oregon Institute ofTechnology for his outstanding contribution to teaching. With his dedicatedresearch team, Dr. Hossain is looking forward to exploring methods to makeelectric power systems more sustainable, cost-effective, and secure throughextensive research and analysis on energy storage, microgrid system, andrenewable energy sources.
MOHIUDDIN AHMAD received his BS de-gree with Honors Grade in Electrical and Elec-tronic Engineering (EEE) from Chittagong Uni-versity of Engineering and Technology (CUET),Bangladesh, and his MS degree in Electronics andInformation Science (major – Biomedical Engi-neering) from Kyoto Institute of Technology ofJapan in 1994 and 2001, respectively. He receivedhis Ph.D. degree in Computer Science and Engi-neering (CSE) from Korea University, Republic ofKorea, in 2008. From November 1994 to August 1995, he served as a part-time Lecturer in the Department of EEE at CUET, Bangladesh. From August1995 to October 1998, he served as a Lecturer in the Department of EEEat Khulna University of Engineering & Technology (KUET), Bangladesh.In June 2001, he joined the same Department as an Assistant Professor. InMay 2009, he joined the same Department as an Associate Professor, andnow he is a full professor. Moreover, Dr. Ahmad served as the Head ofthe Department of Biomedical Engineering (BME) from October 2009 toSeptember 2012. Prof. Ahmad served as the Head of the Department of EEEfrom September 2012 to August 2014. From July 2014, Prof. Ahmad hasbeen serving as the sub-project manager of the UGC, HEQEP, Sub-Project,CP