A survey on shape-constraint deep learning for medical image segmentation
AA SURVEY ON SHAPE - CONSTRAINT DEEP LEARNING FORMEDICAL IMAGE SEGMENTATION
A P
REPRINT
Simon Bohlender
Department of Computer ScienceTU DarmstadtDarmstadt, Germany
Ilkay Oksuz
Computer Engineering DepartmentIstanbul Technical UniversityIstanbul, TurkeySchool of Biomedical Engineering Imaging SciencesKing's CollegeLondon, U.K.
Anirban Mukhopadhyay
Department of Computer ScienceTU DarmstadtDarmstadt, GermanyJanuary 20, 2021 A BSTRACT
Since the advent of U-Net, fully convolutional deep neural networks and its many variants havecompletely changed the modern landscape of deep learning based medical image segmentation.However, the over dependence of these methods on pixel level classification and regression hasbeen identified early on as a problem. Especially when trained on medical databases with sparseavailable annotation, these methods are prone to generate segmentation artifacts such as fragmentedstructures, topological inconsistencies and islands of pixel. These artefacts are especially problematicin medical imaging since segmentation is almost always a pre-processing step for some downstreamevaluation. The range of possible downstream evaluations is rather big, for example surgical planning,visualization, shape analysis, prognosis, treatment planning etc. However, one common thread acrossall these downstream tasks is the demand of anatomical consistency. To ensure the segmentation resultis anatomically consistent, approaches based on Markov/ Conditional Random Fields, StatisticalShape Models are becoming increasingly popular over the past 5 years. In this review paper, a broadoverview of recent literature on bringing anatomical constraints for medical image segmentation isgiven, the shortcomings and opportunities of the proposed methods are thoroughly discussed andpotential future work is elaborated. We review the most relevant papers published until the submissiondate. For quick access, important details such as the underlying method, datasets and performanceare tabulated. K eywords Medical Image Segmentation · Shape Priors · Shape Models · CRF · MRF · Active Contours
Semantic segmentation is the task of predicting the cate-gory of individual pixels in the image which has been oneof the key problems in the field of image understandingand computer vision for a long time. It has a vast range ofapplications such as autonomous driving (detecting road signs, pedestrians and other road users), land use and landcover classification, image search engines, medical field(detecting and localizing the surgical instruments, describ-ing the brain tumors, identifying organs in different imagemodalities). This problem has been tackled by a combina-tion of machine learning and computer vision, approachesin the past. Despite their popularity and success, deep a r X i v : . [ ee ss . I V ] J a n PREPRINT - J
ANUARY
20, 2021learning era changed main trends. Many of the problemsin computer vision - semantic segmentation among them- have been solved with convolutional neural networks(CNNs) .Incorporating prior knowledge into traditional image seg-mentation algorithms has proven useful for obtaining moreaccurate and plausible results. The highly constrainednature of anatomical objects can be well captured withlearning based techniques. However, in most recent andpromising techniques such as CNN based segmentation itis not obvious how to incorporate such prior knowledge.Segmenting images that suffer from low-quality and lowsignal-to-noise ratio without any shape constraint remainsproblematic even for CNNs. Though it has been shownthat incorporation of shape prior information significantlyimproves the performance of the segmentation algorithms,incorporation of such prior knowledge is a tricky practicalproblem. In this work, we provide an overview of effortsof shape prior usage in deep learning frameworks.
There already appeared a variety of review papers aboutshape modelling and deep learning for medical image seg-mentation in the recent past. McInerney and Terzopoulos(1996) presents various approaches that apply deformablemodels. Peng et al. (2013) deals with different categoriesof graph-based models where meaningful objects are rep-resented by sub-graphs. The review by Heimann andMeinzer (2009) is about statistical shape models and con-centrates especially on landmark-based shape representa-tions. Elnakib et al. (2011) also reviews different shapefeature based models, that include statistical shape mod-els, as well as deformable models. A more recent reviewby Nosrati and Hamarneh (2016) provides insights intosegmentation models that incorporate shape informationas prior knowledge. Later surveys of Litjens et al. (2017),Razzak et al. (2017), Rizwan I Haque and Neubert (2020)and Lei et al. (2020) shift their focus to deep learningapproaches. Hesamian et al. (2019) and Taghanaki et al.(2019) present different network architectures and trainingtechniques, whereas Jurdi et al. (2020) take it a step furtherand reviews prior-based loss functions in neural networks.Since deep learning became the method of choice for manycomputer vision tasks, including medical image segmenta-tion, we focus our review on models that combine neural networks with explicit shape models in order to incorporateshape knowledge into the segmentation process. Segmen-tation models solely based on neural networks usually donot incorporate any form of shape knowledge. They arebased on traditional loss functions that only regard objectsat pixel level and do not evaluate global structures. Thepapers we present in this review improve these networks bycombining them with additional models that are especiallybuilt with shape in mind. This is also the point that delimitsthis review from existing surveys which either focus mostlydeep learning approaches or on traditional shape and de-formable model methods, but not on the combination ofboth.The explicit models applied in this review can be dividedinto three main categories as shown in Figure 1: 1) Condi-tional or Markov Field models that establish connectionsbetween different pixel regions 2) Active/Statistical ShapeModels that learn a special representation for valid shapes3) Active Contour Models or snakes that use deformablesplines for shape detection. These models are either ap-plied as pre-processing steps to create initial segmentations,post-processing steps to refine the neural network segmen-tations, or used in multi-step models consisting of variousmodels along a specific pipeline.We are aware that the field is heavily shifting from explicitways of modeling shape to more implicit approaches wherenetworks are trained in an end-to-end way.Up and com-ing Works propose more intelligent loss functions that nolonger require additional explicit shape modelling, but onlyconsist of a single neural network. Zhang et al. (2020a)proposed a new geometric loss for lesion segmentation.Other examples are Mohagheghi and Foruzan (2020) andHan et al. (2020) where the loss contains shape priors. Liet al. (2020) introduces a spatially encoded loss with aspecial shape attention mechanism. Clough et al. (2019b)uses a topology based loss function.However the overwhelming majority of articles combineneural networks and explicit models to introduce shapeknowledge. This combination often stems from a ratherprincipled engineering design choice (as shown in Fig-ure 1) which is not detailed in any of the previous reviewarticles. This review focuses on this overarching designprinciple of shape constraint which, along with being aquick access guide to explicit approaches, will work as aresearch catalyzer of implicit constraints.
Neural NetworksActive / StatisticalShape ModelsConditional / MarkovRandom Field Active Contour Models /Leve-Set Methods forPost-Processing forEnd-to-EndTraining forPost-Processing forPrior-Knowledge MultistepApproach forPost-Processing forPre-Processingcombined with
Figure 1: Overview of related work approaches
Markov Random Fields (MRF) Li (1994) belong to thedomain of graphical models and model relationships be-tween pixels or high-level features with a neighborhoodsystem. The label probability of a single pixel is therebyconditioned on all neighboring pixels which allow to modelcontextual constraints. The maximum a posteriori proba-bility (MAP) can then be calculated by applying the Bayesrule. Conditional Random Fields (CRF) Lafferty et al.2
PREPRINT - J
ANUARY
20, 2021(2001) are an extension of MRFs and allow to incorpo-rate arbitrary global features over regions of pixels. Formedical image segmentation this means that they gener-ate smooth edges by using this global knowledge aboutsurrounding regions which is a reason why the are often ap-plied alongside neural networks to perform medical imagesegmentation.
CRFs used for postprocessing
The largest categoryof methods that utilize CRFs or MRFs apply them as apos-tprocessing step. A large portion of papers focus onthe straight-forward approach where the CNN generatesinitial segmentations maps which are directly passed toa CRF or MRF model as inputs for further refinements.These approaches are evaluated on a variety of anatomiesand mostly differ in the utilized network architectures butfollow the same idea. They are applied to lung nodules(Yaguchi et al. (2019), Gao et al. (2016)), retinal vessel(Fu et al. (2016b)), brain tumor (Zhao et al. (2016), Liet al. (2017a)), cervical nuclei (Liu et al. (2018)), eyesclera (Mesbah et al. (2017)), melanoma (Luo and Yang(2018)), ocular structure (Nguyen et al. (2018)), left atrialappendage (Jin et al. (2018)), lymph node (Nogues et al.(2016)), liver (Dou et al. (2016)) and prostate cancer le-sion (Cao et al. (2019)) segmentation tasks. A slightlydifferent approach for skin lesion detection by Qiu et al.(2020) is based on the same idea, but uses not just a singleCNN network, but an ensemble of seven or fifteen whichare combined inside the CRF. Two other approaches tohighlight here for brain region (Zhai and Li (2019)) andoptical discs in fundus image (Bhatkalkar et al. (2020))segmentation integrate a special attention mechanism intotheir networks with the motivation to improve the segmen-tations by detecting and exploiting salient deep features.Another special version that operates on weakly segmentedbounding box images for fetal brain & lung segmentationis introduced by Rajchl et al. (2017). Given the initialweak segmentations, the model iteratively optimizes thepixel predictions with a CNN followed by a CRF to obtainthe final segmentation maps.Instead of CRFs, Shakeri et al. (2016) use a MRF to im-pose volumetric homogenity on the outputs of a CNN forsubcortical region segmentation. MRFs are also utilizedin the approach shown by Xia et al. (2019) for kidney seg-mentation where the MRF is integrated into a SIFT-Flowmodel.Besides these classical approaches, another method thatcame up focused on cascading CNN networks that gener-ate segmentations in a coarse-to-fine fashion. Wachingeret al. (2018) use this strategy with a first network thatsegments fore- from background pixels in brain MRIs anda second one that classifies the actual brain regions. Thesame method is also used by Shen and Zhang (2017) forbrain tumor segmentation, by Dou et al. (2017) for liverand whole heart segmentation, and by Christ et al. (2016)for liver-based lesion segmentation.A somewhat different cascading structure, for brain tu-mor segmentation, is introduced by Hu et al. (2019)where multiple subsequent CNNs are used to extract more discriminative multi-scale features and to capture depen-dencies. Feng et al. (2020) extend this version on thetask of brain tumor segmentation with the introduction ofresidual connections that improve the overall performance.Similar to the cascading methods, there are CNNs with twopathways that combine two parallel networks on differentresolution levels that aim for capturing larger 3D contexts.The approach was originally introduced by Alansary et al.(2016) for placenta segmentation, but was also appliedin Cai et al. (2017) to the task of pancreas segmentation.Kamnitsas et al. (2017) proposes another related approachwhere two parallel networks, a FCN that extracts a roughmask and a HED that outputs a contour, are fused inside aCRF. In the approach by Shen et al. (2018) that deals withbrain tumor segmentation, a third path is added where intotal three concurrent FCNs are trained based on differentfiltered (gaussian, mean, median) input images. After eachnetwork an individual CRF is applied and their results arefused in a linear regression model. f P ape r s Method
CRFACMASM
Figure 2: Overview of relevant papers per year for eachcategory
Training CNN and CRF models end-to-end
The ideaof integrating CRF models directly into neural networksorigins from the task of semantic image segmentation andwas introduced by Zheng et al. (2015). They combine thestrengths of both models into a unified framework thatallows end-to-end training. Broken down, the basic taskof CRFs is to minimize an energy term with an iterativemean field approximation. Since CRFs are graphical mod-els, each iteration step can be formulated as a stack ofCNN layers. Multiple iterations can then be implementedby repeatedly executing this stack or alternatively as anequivalent Recurrent Neural Network (RNN). The result-ing network is denoted as a CRF-RNN and can be appliedon top of any CNN architecture. Fu et al. (2016a) are thefirst to transfer this method to medical image segmenta-tions with a model called
DeepVessel for the task of retinalvessel segmentation. For the same task Luo et al. (2017)achieve similar results by using a slightly deeper base CNNnetwork with more convolution layers. Besides retinal ves-sel, CRF-RNN approaches are applied to a variety of other3
PREPRINT - J
ANUARY
20, 2021anatomical structures. Zhao et al. (2016) applies them tobrain tumor segmentation and extend it with some addi-tional pre- and post-processing steps later on Zhao et al.(2018b). Xu et al. (2018) uses a V-Net combined withCRF-RNN for bladder segmentation and in Monteiro et al.(2018) they are also applied on brain tumor as well asprostate segmentation with 3D multi-modal images. Ana-logues Chen and de Bruijne (2018) utilizes a U-Net as theirbase-network to deal with white matter lesion segmenta-tion. On the same idea as CRF-RNN Deng et al. (2020) uses a CRF-Recurrent Regression based Neural Network(CRF-RRNN) integrated with a heterogeneous CNN forbrain tumor segmentation where the combined network canalso be trained end-to-end. Instead of using a full RNN,Zhang et al. (2020d) propose a method where MRF is inte-grated into the segmentation network as a block of localand global convolution layers that take the CNN output asunary potentials to calculate the corresponding pairwisepotentials.Table 1: CNNs combined with CRF / MRF models
Authors Anatomy Title Method
CRF / MRF used for post-processingLi et al.(2017a) Brain Tumor Low-Grade Glioma Segmentation Based onCNN with Fully Connected CRF CRF refines CNN segmentationWachingeret al. (2018) Brain Re-gion DeepNAT: Deep convolutional neuralnetwork for segmenting neuroanatomy CRF refines hierarchical CNNsegmentationsHu et al.(2019) Brain Tumor Brain Tumor Segmentation UsingMulti-Cascaded Convolutional NeuralNetworks and Conditional Random Field FC-CRF refines segmentations of threeCNNsShen andZhang (2017) Brain Tumor Fully connected CRF with data-driven priorfor multi-class brain tumor segmentation Multiple FC-CRFsKamnitsaset al. (2017) Brain Lesion Efficient Multi-Scale 3D CNN with FullyConnected CRF for Accurate Brain LesionSegmentation FC-CRF refines two-pathway CNNAlansary et al.(2016) Placenta Fast Fully Automatic Segmentation of theHuman Placenta from Motion CorruptedMRI FC-CRF refines two-pathway CNNShakeri et al.(2016) Sub-corticalregions Sub-cortical brain structure segmentationusing F-CNN’s MRF refines FCNN segmentationZhai and Li(2019) Brain region An Improved Full Convolutional NetworkCombined with Conditional Random Fieldsfor Brain MR Image SegmentationAlgorithm and its 3D VisualizationAnalysis FC-CRF refines CNN with attentionDou et al.(2016) Liver 3D Deeply Supervised Network forAutomatic Liver Segmentation from CTVolumes FC-CRF refines 3D FCNN with 3Dsupervision mechanismDou et al.(2017) Heart 3D Deeply Supervised Network forAutomated Segmentation of VolumetricMedical Images FC-CRF refines cascading U-NetsChrist et al.(2016) Liver Automatic Liver and Lesion Segmentationin CT Using Cascaded Fully ConvolutionalNeural Networks and 3D ConditionalRandom Fields FC-CRF refines cascaded FCNsFu et al.(2016b) Retinal Ves-sel Retinal vessel segmentation via deeplearning network and fully-connectedconditional random fields FC-CRF refines CNN with side-outputsJin et al.(2018) Left atrial ap-pendage Left Atrial Appendage Segmentation UsingFully Convolutional Neural Networks andModified Three-Dimensional ConditionalRandom Fields FC-CRF combines slices of FCNCai et al.(2017) Pancreas Pancreas Segmentation in MRI usingGraph-Based Decision Fusion onConvolutional Neural Networks CRF refines results from FCN and HEDnetworkXia et al.(2019) Kidney Deep Semantic Segmentation of Kidneyand Space-Occupying Lesion Area Basedon SCNN and ResNet Models Combinedwith SIFT-Flow Algorithm MRF refines combined ResNet and SCNN PREPRINT - J
ANUARY
20, 2021Table 1: CNNs combined with CRF / MRF models
Authors Anatomy Title Method
Rajchl et al.(2017) Fetal Brain /Lung DeepCut: Object Segmentation fromBounding Box Annotations usingConvolutional Neural Networks Iterative CRF and CNNNogues et al.(2016) Lymph Node Automatic Lymph Node ClusterSegmentation Using Holistically-NestedNeural Networks and StructuredOptimization in CT Images CRF refines HNN (FCN + DSN)segmentationsYaguchi et al.(2019) Lung Nod-ules 3D fully convolutional network-basedsegmentation of lung nodules in CT imageswith a clinically inspired data synthesismethod CRF refines 3D FCN segmentationsGao et al.(2016) Lung Segmentation label propagation using deepconvolutional neural networks and denseconditional random field CRF refines CNN segmentationsFeng et al.(2020) Brain Tumor Study on MRI Medical ImageSegmentation Technology Based onCNN-CRF Model CRF refines DCNN segmentationsLiu et al.(2018) Cervical Nu-clei Automatic segmentation of cervical nucleibased on deep learning and a conditionalrandom field Locally FC-CRF refines Mask-RCNNsegmentationShen et al.(2018) Brain Tumor Brain tumor segmentation using concurrentfully convolutional networks andconditional random fields Concurrent FCN refined by FC-CRFMesbah et al.(2017) Eye Sclera Conditional random fields incorporateconvolutional neural networks for humaneye sclera semantic segmentation Initial CNN boundaries refined by CRFLuo and Yang(2018) Melanoma Fast skin lesion segmentation via fullyconvolutional network with residualarchitecture and CRF CRF refines FCN segmentationsBhatkalkaret al. (2020) Fundus Op-tic Disk Improving the Performance ofConvolutional Neural Network for theSegmentation of Optic Disc in FundusImages Using Attention Gates andConditional Random Fields FC-CRF refines CNN segmentationsQiu et al.(2020) Skin Lesion Inferring Skin Lesion Segmentation WithFully Connected CRFs Based on MultipleDeep Convolutional Neural Networks CRF refines segmentations of DCNNensembleNguyen et al.(2018) Ocular struc-tures Ocular structures segmentation frommulti-sequences mri using 3d unet withfully connected crfs FC-CRF refines CNN segmentationsCao et al.(2019) Prostate can-cer lesions Prostate Cancer Detection andSegmentation in Multi-parametric MRI viaCNN and Conditional Random Field Selective Dense CRF refines CNNsegmentationsCNN and CRF trained end-to-endZhao et al.(2018b) Brain Tumor A deep learning model integrating FCNNsand CRFs for brain tumor segmentation. Combination of FCNN and CRF-RNNMonteiro et al.(2018) Prostate /Brain Tumor Conditional Random Fields as RecurrentNeural Networks for 3D Medical ImagingSegmentation Combination of FCNN and CRF-RNNFu et al.(2016a) Retinal Ves-sel DeepVessel: Retinal Vessel Segmentationvia Deep Learning and ConditionalRandom Field Combination of CNN and CRF-RNN layersChen andde Bruijne(2018) White matterhyperintensi-ties An End-to-end Approach to SemanticSegmentation with 3D CNN andPosterior-CRF in Medical Images Combination of U-Net and FC-CRFXu et al.(2018) Bladder Automatic bladder segmentation from CTimages using deep CNN and 3D fullyconnected CRF-RNN Combination of CNN and CRF-RNN PREPRINT - J
ANUARY
20, 2021Table 1: CNNs combined with CRF / MRF models
Authors Anatomy Title Method
Deng et al.(2020) Brain Tumor Deep Learning-Based HCNN andCRF-RRNN Model for Brain TumorSegmentation Combination of HCNN and CRF-RRNNZhang et al.(2020d) Prostate ARPM-net: A novel CNN-basedadversarial method with Markov RandomField enhancement for prostate and organsat risk segmentation in pelvic CT images CNN combined with MRF blockZhao et al.(2016) Brain tumor Brain tumor segmentation using a fullyconvolutional neural network withconditional random fields CRF integrated into FCNNLuo et al.(2017) Retinal Ves-sel Efficient CNN-CRF network for retinalimage segmentation Combination of CNN and CRF
The second category of model assumptions often com-bined with CNNs are active shape models (ASM) Cooteset al. (1995) or probabilistic active shape models (PASM).ASMs require a training set with a fixed number of man-ually annotated landmark points of the segmented object.Each point represents a particular part of the object andhas to be in the same position over all images. These an-notated shapes are then iteratively matched and a meanshape is derived. The landmark points show different vari-abilities that are modeled by a Point Distribution Model(PDM). Performing a principal component analysis (PCA)and weighting the eigenvectors allows creating new shapesin the allowed variability range. For detecting an object inan unknown image an algorithm is used that updates poseand shape parameters iteratively to improve the match untilconvergence. An extension to this approach are probabilis-tic ASMs (PASM) Wimmer et al. (2009). They impose aweaker constraint on shapes which allows more flexiblecontours with more variations from the mean shape. Thisis achieved by introducing a probabilistic energy functionwhich is minimized in order to fit a shape to a given image.The model’s ability to generalize is thereby improved andthe segmentation results outperform standard ASMs.
Shape Models for post-processing
Though CNN basedsegmentation models yield good segmentation results, theytend to produce anatomically implausible segmentationmaps that can contain detached islands or holes at partswhere they do not occur in reality. Since shape modelsrepresent valid and anatomically plausible shapes, it makessense to apply them in post-processing steps to regular-ize initial CNN segmentations and transform them into avalid shape domain. Xing et al. (2016) take up this ideaand apply it to nucleus segmentation. The initial segmen-tations are generated by a CNN and the post-processingstep includes a sparse selection-based shape model fortop-down shape inference, which is more insensitive toobject occlusions compared to PCA-based shape models,and an additional deformable model for bottom-up shapedeformation. Also Hsu (2019) follows this strategy for seg-mentation and tracking of the left ventricle. They swap out the CNN for a Faster-RCNN and use an improved ASMthat allows to obtain matching points in greater ranges.Fauser et al. (2019) continue on improving the ASM byusing a probabilistic ASM that is more flexible and al-lows leaving the shape space. The segmentation of the leftventricle is performed by combining the results of threeCNN-PASM models for each dimension. Another modi-fied ASM is proposed by Medley et al. (2020). The authorsuse Expectation-Maximization to deal with outliers duringoptimizing the ASM. They also evaluate different ASMfeatures and conclude that a CNN that learns the inputfeature maps for the EM-ASM performs best. Besidesimproving on the ASM a different approach by Karimiet al. (2019) aims for generating better predictions with anensemble of U-Net like CNN models with different filtersand parameters. In their approach a SSM model, based onthe thresholded segmentations from all individual models,is only applied if the disagreement between the ensemblemodels becomes to high. Instead of using the CNN forgenerating segmentation maps, it is also sufficient to onlypredict bounding boxes as initializations for ASMs. Suchan approach is applied by Tabrizi et al. (2018) on kidneysegmentation where a fuzzy-ASM produces the final seg-mentations. Li et al. (2018) also uses a CNN for boundingbox prediction, but adds an intermediate step before utiliz-ing a statistical shape model for myocardial segmentation,in which a random forest classifier builds probability mapsfrom the given bounding boxes. Another tree model, morespecific an adaptive feature learning probability boostingtree (AFL-PBT) is also utilized by He et al. (2018) as aninitial step to classify voxels for prostate segmentation. Asubsequent CNN then extracts boundary probability mapsand a three-level ASM is employed to generate final seg-mentations.
Shape Models for prior knowledge
In this second para-graph we present some papers where the shape models areapplied pre-hoc before any deep learning network. Twostraight forward models for this category are proposedby Cheng et al. (2016) and Fan et al. (2020). In Fanet al. (2020) a 3D U-Net-like CNN segments Itra-Cholearanatomy based on initial segmentations from an ASMand the original CT images. Cheng et al. (2016) on the6
PREPRINT - J
ANUARY
20, 2021other hand use a CNN for refining initial segmentationsfrom an Active Appearance Model (AAM) that producesonly coarse prostate segmentations. The AAM is basicallyan extended shape model that adds an additional texturemodel for better fitting capabilities. The other two modelsalready introduce some pipeline-like approaches, but useboth a shape model as prior knowledge. The pipeline forsubcortical region segmentation in Duy et al. (2018) startswith a pre-processing SVM that classifies sagittal slicesinto groups of similar shape. The prior ASM then createsrough segmentations for each group which are finalizedby a CNN. Further the authors propose an optional CRFmodel for post-processing. Nguyen et al. (2019) introducethe ASM as a more traditional prior for uveal melanomasegmentation where it is used as a constraining term for aCRF model that is based on Grad-CAM (class activationmaps) heatmaps. The final segmentations are again gen-erated with a U-Net that combines the CRF with originalinput CTs. B l adde r B r a i n B r ea s t C a r d i a c D en t a l E a r E y e F e t u s K i dne y K nee L i v e r Lung L y m ph N ode N u c l e i P an c r ea s P l a c en t a P r o s t a t e S k i n f P ape r s Method
CRFACMASM
Figure 3: Overview of anatomical structures examined inthe relevant papers
Pipeline approaches with multiple CNN and ASMmodels
The last category for combining shape modelsand neural networks contains all approaches that consist ofdifferent models arranged along pipelines. The motivationis to process input images stage-wise or in a coarse-to-fineway that allows to capture more information and hence re-sult in more accurate segmentation maps. In the models byTack et al. (2018) for knee menisci, Ambellan et al. (2019)for knee bone & cartilage, and Brusini et al. (2020) for hip-pocampus segmentation, the pipelines combine multipleCNNs and SSMs. All three start with initial 2D U-Netsregularized by SSMs which are used to extract smaller 3D subvolumes. Tack et al. (2018) and Ambellan et al. (2019)apply an additional 3D U-Net afterwards, whereas Brusiniet al. (2020) uses three U-Nets and averages their predic-tions to obtain final segmentations. Ambellan et al. (2019)further continues after this step and utilizes a second 3DSSM model to obtain the knee bone segmentations andeven applies a third U-Net to segment the cartilage after-wards. Besides these typical pipelines, there are also somehybrid approaches we count to this category that integrateshape models and neural networks. They use special CNNsthat directly predicts the parameters of an SSM, which arethe shape coefficients (weights for the modes of variations),the pose parameters. Qin et al. (2020) use such a SSM-Netinside a small pipeline for prostate segmentation. Theypropose an inception-based network that directly predictsparameters of the SSM which can be back-translated intoa prostate contour prediction. Parallel to this, a residualU-Net generates probability maps from the inputs. Thefinal segmentations are generated by averaging the outputsof both models. The method of Tilborghs et al. (2020)for left ventricle segmentation is based on the same idea,but removes the small pipeline. Instead they modify theCNN and add a third output which is an actual distancemap. A special loss function is used to train the network to-ward optimizing the segmentation map alongside the SSMparameters. A nearly identical approach by Karimi et al.(2018) is applied to prostate segmentation. Their CNNpredicts center position of the prostate, the shape model pa-rameters, and a rotation vector which are passed to a finallayer that outputs the coordinates of the landmark pointswhich resemble the a final segmentation map. Schocket al. (2020) relies on the same method for knee bone &cartilage segmentation, but extend it with additional pre-and post-processing steps. They add a preprocessing 2DU-Net that detects initial bone positions and crop the vol-ume into subvolumes which only contain the femur or tibiabone. Afterwards their SSM-Net comes into place thatpredicts the SSM parameters and the actual landmarks ina subsequent PCA layer. An additional fine-tuning stepthen generates the cartilage segmentations with a 3D U-Net based on subvolumes centered at the bones’ landmarkpoints. Rather than integrating the SSM and CNN, Ma et al.(2018) introduces a Bayesian model that integrates both,the CNN and a robust kernel SSM (RKSSM) for the taskof pancreas segmentation. At first the RKSSM is initial-ized to fit the detected ROI of a Dense U-Net. A GaussianMixture Model afterwards guides the shape adaption anditeratively projects the adapted shape onto the RKSSMuntil convergence which results in the final segmentationmap.Table 2: CNNs combined with Active Shape Models
Authors Anatomy Title Method
ASM for post-processingXing et al.(2016) Nucleus An Automatic Learning-Based Frameworkfor Robust Nucleus Segmentation Shape Model refines CNN segmentation PREPRINT - J
ANUARY
20, 2021Table 2: CNNs combined with Active Shape Models
Authors Anatomy Title Method
He et al.(2018) Prostate Automatic Magnetic Resonance ImageProstate Segmentation Based on AdaptiveFeature Learning Probability Boosting TreeInitialization and CNN-ASM Refinement Three-level-ASM refines segmentations ofCNNFauser et al.(2019) TemporalBone Toward an automatic preoperative pipelinefor image-guided temporal bone surgery Probabilistic ASM refines 2D U-NetsegmentationLi et al. (2018) Myocardial Fully Automatic Myocardial Segmentationof Contrast Echocardiography SequenceUsing Random Forests Guided by ShapeModel ASM refines random-forest segmentationsinitialized by a CNNMedley et al.(2020) Left Ventri-cle Deep Active Shape Model for RobustObject Fitting ASM initialized with CNN generatedfeatures mapsKarimi et al.(2019) Prostate Accurate and robust deep learning-basedsegmentation of the prostate clinical targetvolume in ultrasound images SSM refines segmentations from ensembleof CNNsTabrizi et al.(2018) Kidney Automatic kidney segmentation in 3Dpediatric ultrasound images using deepneural networks and weighted fuzzy activeshape model Fuzzy ASM segmentations based on DNNgenerated bounding boxesHsu (2019) Left Ventri-cle Automatic Left Ventricle Recognition,Segmentation and Tracking in CardiacUltrasound Image Sequences ASM improves R-CNN segmentations fordetection and trackingASM as prior-knowledgeDuy et al.(2018) Brain Re-gion Accurate brain extraction using ActiveShape Model and Convolutional NeuralNetworks CNN refines ASM segmentationsCheng et al.(2016) Prostate Active appearance model and deep learningfor more accurate prostate segmentation onMRI 2D-CNN refines segmentations from anActive Appearance ModelFan et al.(2020) Intra-CholearAnatomy Combining model- and deep-learning-basedmethods for the accurate and robustsegmentation of the intra-cochlear anatomyin clinical head CT images U-Net refines ASM segmentationsNguyen et al.(2019) UvealMelanoma A novel segmentation framework for uvealmelanoma based on magnetic resonanceimaging and class activation maps U-Net segmentations based on a CRF thatuses ASM as prior knowledgePipelines with multiple ASM and CNN models & Hybrid approachesAmbellan et al.(2019) Knee Bone /Cartilage Automated Segmentation of Knee Boneand Cartilage combining Statistical ShapeKnowledge and Convolutional NeuralNetworks: Data from the OsteoarthritisInitiative Three CNN and two SSM modelsTack et al.(2018) KneeMenisci Knee Menisci Segmentation usingConvolutional Neural Networks: Data fromthe Osteoarthritis Initiative 3D CNN and SSM initialized by 2D modelsBrusini et al.(2020) Hippocampus Shape Information Improves theCross-Cohort Performance of DeepLearning-Based Segmentation of theHippocampus ASM as input for CNNMa et al.(2018) Pancreas A novel bayesian model incorporating deepneural network and statistical shape modelfor pancreas segmentation U-Net and SSM segmentations combinedwithin Bayesian modelQin et al.(2020) Prostate A weakly supervised registration-basedframework for prostate segmentation viathe combination of statistical shape modeland CNN Segmentations combined of U-Net andSSM-Net predictions PREPRINT - J
ANUARY
20, 2021Table 2: CNNs combined with Active Shape Models
Authors Anatomy Title Method
Tilborghs et al.(2020) Left Ventri-cle Shape Constrained CNN for Cardiac MRSegmentation with Simultaneous Predictionof Shape and Pose Parameters Hybrid approach where CNN generatessegmentations and ASM parametersKarimi et al.(2018) Prostate Prostate segmentation in MRI using aconvolutional neural network architectureand training strategy based on statisticalshape models CNN predicts segmentations and 3D-ASMparametersSchock et al.(2020) Knee Bone& Cartilage A Method for Semantic Knee Bone andCartilage Segmentation with Deep 3DShape Fitting Using Data from theOsteoarthritis Initiative CNN that predicts segmentations and3D-ASM parameters is refined by U-Net
A last type of models that often combined with deep learn-ing models to incorporate shape knowledge are
ActiveContour Models (ACM) Kass et al. (1988) , also knownas snakes . A snake is a deformable controlled continuityspline that is pushed towards edges or contours by mini-mizing an energy function under the influence of differentforces and constraints. It consists of an internal energy thatkeeps the contour continuous and smooth, an image energythat attracts it to contours, and an external constraint forcethat adds user-imposed guidance. A similar approach are level set functions (LSF) introduced by Andrew (2000)and firstly applied to image segmentation by Malladi et al.(1995). An LSF is a higher dimensional function where acontour is defined as its zero level set. With a speed func-tion , derived from the image, that controls the evolution ofthe surface over time, a Hamilton-Jacobi partial differentialequation can be obtained.
ACM models for post-processing
Since ACM modelsare based on the idea of evolving a contour, it makes senseto apply them as a post-processing step to improve an ini-tial segmentation map. An early model by Middleton andDamper (2004) uses only a simple multilayer perceptron(MLP) that creates binary pixel-wise boundary predictionsfor lung segmentation. Since these are very rough andcontain misclassifications the ASM is used to improve andclose the contour. Salimi et al. (2018) is also based on anMLP, but adds an vector field convolution to the ACM tomake it more robust for prostate segmentation.However, the more recent ACM post-processing modelsare exclusively based on different CNN architectures andare applied to a variety of anatomies. Li et al. (2017b) usea FCN that is refined by a classic ACM for left ventriclesegmentation. The same approach is taken by Guo et al.(2019) for liver segmentation and Zhao et al. (2018a) uti-lize it for nucleus segmentation. In the approaches by Xuet al. (2019) the ACM refinements are not yet the final stepsand additional adaptive ellipse fitting is used to segmentbreast nuclei. Hu et al. (2018) and Fang et al. (2019) trans-fer the basic refinement method to breast tumor detectionwith a phase-based ACM that improves over multiple iter-ations. A different slightly modified ACM post-processing method is based on geodesic computations and is furtherused by Ma and Yang (2019) for dental root segmenta-tion and Nunes et al. (2020) for lung segmentation. Zhanget al. (2020b) also introduces a special ACM that integratesa fourth-order partial differential equation and segmentsplaque based on an initial R-CNN segmentation. Insteadof just refining an initial CNN predicted per-pixel segmen-tation map, da Silva et al. (2020) use a Chan-Vese ACM togenerate prostate segmentation on DCNN coarsely classi-fied superpixels which only represent rough initializationfor the contour model. The authors of Kot et al. (2020) fur-ther separate the two models where the CNN masks bonetissue which is removed for the ACM to segment braintumors. The last special approach in the ACM category byZhang et al. (2020c) is a hybrid model that integrates anACM into a U-Net. The resulting deep active contour net-work (DACN) is end-to-end trainable with a special ACMbased loss function and automatically segments cervicalcells and skin lesions. Besides ACM, another large numberof approaches rely on level set functions (LSF). Same asbefore a CNN is used for generating initial segmentationmaps which are then refined by the LSF. Hatamizadeh et al.(2019) uses this for brain, liver, lung segmentation, Gonget al. (2019) for pancreas segmentation, Carbajal-Deganteet al. (2020) for ventricle and liver segmentation, and Xieet al. (2020) for left ventricle segmentation. Some extraprocessing is made in Yang et al. (2021) for dental pulp seg-mentation where the initial CNN segmentations are usedto calculate elliptic curves which are used to guide the LSF.In general, for the LSF it is often sufficient to initializethem only with a rough bounding boxes or region of inter-est annotations. So, Liu et al. (2019) use a Faster RCNNto generate location boxes of left atriums which serve asinput for the LSF after Otsu thresholding. Avendi et al.(2016) inserts an additional step between CNN ROI detec-tion and LSF segmentation where the initial left-ventricleshape is inferred with an stacked auto-encoder. In com-parison to these two approaches, in Cha et al. (2016) theCNN is not used to predict ROI, but to classify if an ROIis part of the bladder. The outputs are then refined by threedifferent 3D LSF and a final 2D LSF afterwards. Anotheridea is to use recurrent pipelines where the segmentationsare refined iteratively. Such an approach is introduced byTang et al. (2017) where both models are integrated into a9
PREPRINT - J
ANUARY
20, 2021FCN-LSF. The method is used for left ventricle and liversegmentation with semi-supervised training where the LSFgradually refines the segmentation and backpropagates aloss to improve the FCN. Hoogi et al. (2017) proposeda different iterative process. Hereby the CNN estimatesif the zero level set is inside, outside or near the lesionboundary. Based on these the LSF parameters are calcu-lated and the contour is evolved. The process then repeatsuntil convergence.
Using a CNN to refine ACM segmentations
Besidesthe majority of approaches that use ACMs for post-processing, there are also methods where ACMs are usedto obtain the initial segmentations or are guided by CNNs. The earliest of these approaches by Ahmed et al. (2009)uses an ACM to remove skull tissue from images andapplies a simple artificial neural network to classify theremaining brain regions. Rupprecht et al. (2016) intro-duce an approach where the ACM is guided by the CNN.The ACM generated rough segmentations of the left ven-tricle. A CNN then predicts vectors on patches aroundeach pixel of this initial contour that point towards closesobject boundary points and are used to further evolve thecontour. The latest method for this category by Kasinathanet al. (2019) also uses the ACM to generate initial seg-mentations, more specific it segments all lung nodules.A post-processing CNN afterwards classifies them or re-moves false positives.Table 3: CNNs combined with Active Contour Models
Authors Anatomy Title Method
ACM for post-processingMiddleton andDamper (2004) Lung Segmentation of magnetic resonanceimages using a combination of neuralnetworks and active contour models ACM refines MLP segmentationSalimi et al.(2018) Prostate Fully automatic prostate segmentation inMR images using a new hybrid activecontour-based approach ACM refines MLP segmentationLi et al.(2017b) Left Ventri-cle Left ventricle segmentation by combiningconvolution neural network with activecontour model and tensor voting inshort-axis MRI ACM refines FCN segmentationHu et al.(2018) BreastTumor Automatic tumor segmentation in breastultrasound images using a dilated fullyconvolutional network combined with anactive contour model Phase-based ACM refines dilated FCNsegmentationGuo et al.(2019) Liver Automatic liver segmentation byintegrating fully convolutional networksinto active contour models ACM refines multi-branch FCNsegmentationZhao et al.(2018a) Nucleus Improved Nuclear Segmentation onHistopathology Images Using aCombination of Deep Learning and ActiveContour Model Hybrid ACM refines multi-branch FCNsegmentationHatamizadehet al. (2019) Liver / BrainLesion /Lung Deep Active Lesion Segmentation ACM refines signed distance maps fromFC-CNNTang et al.(2017) Liver / LeftVentricle A Deep Level Set Method for ImageSegmentation Level-set ACM refines FCN segmentationsiterativelyCha et al.(2016) Bladder Urinary bladder segmentation in CTurography using deep-learningconvolutional neural network and level sets Multiple level-set functions segment CNNoutput ROIsHoogi et al.(2017) Liver Lesion Adaptive Estimation of Active ContourParameters Using Convolutional NeuralNetworks and Texture Analysis Level-set function iteratively improvesCNN segmentationFang et al.(2019) BreastTumor Combining a Fully Convolutional Networkand an Active Contour Model forAutomatic 2D Breast Tumor Segmentationfrom Ultrasound Images Phase-based ACM refines initial contoursfrom dilated FCNNXu et al.(2019) Breast Can-cer Nuclei Convolutional neural network initializedactive contour model with adaptive ellipsefitting for nuclear segmentation on breasthistopathological images ACM refines CNN segmentations PREPRINT - J
ANUARY
20, 2021Table 3: CNNs combined with Active Contour Models
Authors Anatomy Title Method
Ma and Yang(2019) Teeth Automatic dental root CBCT imagesegmentation based on CNN and level setmethod ACM refines CNN segmentationsCarbajal-Degante et al.(2020) Ventricles Active contours for multi-regionsegmentation with a convolutional neuralnetwork initialization Phase level-set function refines CNNsegmentationsLiu et al.(2019) Left Atrium A Framework for Left AtriumSegmentation on CT Images withCombined Detection Network and LevelSet Model 3D level-set model initialized by FasterRCNNYang et al.(2021) Teeth Accurate and automatic tooth imagesegmentation model with deepconvolutional neural networks and level setmethod Level-set based on contours derived fromU-Net predictionsNunes et al.(2020) Lung Adaptive Level Set with region analysis viaMask R-CNN: A comparison againstclassical methods ACM improves Mask R-CNNsegmentationsXie et al.(2020) Left Ventri-cle Automatic left ventricle segmentation inshort-axis MRI using deep convolutionalneural networks and central-line guidedlevel set approach Level-set model improves CNNinitializationGong et al.(2019) Pancreas Convolutional Neural Networks BasedLevel Set Framework for PancreasSegmentation from CT Images Level-set model based on initial contourfrom CNNZhang et al.(2020c) Cervical Cell/ Skin Lesion Deep Active Contour Network for MedicalImage Segmentation ACM integrated into CNN that learns initialparameters (end-to-end)Zhang et al.(2020b) Plaque Faster R-CNN, fourth-order partialdifferential equation and global-local activecontour model (FPDE-GLACM) for plaquesegmentation in IV-OCT image ACM initialized with bounding box fromR-CNNda Silva et al.(2020) Prostate Superpixel-based deep convolutional neuralnetworks and active contour model forautomatic prostate segmentation on 3DMRI scans ACM refines DCNN segmentationsKot et al.(2020) Brain Tumor U-Net and Active Contour Methods forBrain Tumour Segmentation andVisualization ACM refines U-Net segmentationsAvendi et al.(2016) Left Ventri-cle A combined deep-learning anddeformable-model approach to fullyautomatic segmentation of the left ventriclein cardiac MRI CNN and AE initialize level set functionCNN refines ACMKasinathanet al. (2019) Lung Tumor/ Nodule Automated 3-D Lung Tumor Detection andClassification by an Active Contour Modeland CNN Classifier CNN refines multiple ACM segmentationsRupprechtet al. (2016) Left ventric-ular cavity Deep Active Contour CNN refines ACMAhmed et al.(2009) Brain A Hybrid Approach for Segmenting andValidating T1-Weighted Normal Brain MRImages by Employing ACM and ANN ANN based on ACM preprocessed images
An alternative approach to integrating shape priors intonetwork-based segmentation was presented in Lee et al.(2019). Here, the segmentation started with a candidateshape which was topologically correct (and approximatelycorrect in terms of its shape), and the network was trained to provide the appropriate deformation to this shape suchthat it maximally overlapped with the ground truth segmen-tation.Such methods can be considered to have a ‘hard prior’rather than the ‘soft-prior’ of the methods presented abovein the sense that the end result can be guaranteed to have11
PREPRINT - J
ANUARY
20, 2021the correct shape. However, this approach may be limitedby a requirement that the initial candidate shape be veryclose to an acceptable answer such that only small shapedeformations are needed. A further potential issue is thatthe deformation field provided by the network may need tobe restricted to prevent the shape from overlapping itselfand consequently changing its topology.The differentiable properties of persistent homology Edels-brunner et al. (2000) make it a promising candidate for theintegration of topological information into the training ofneural networks. The key idea is that it measures the pres-ence of topological features as some threshold or lengthscale changes. Persistent features are those which existfor a wide range of filtration values, and this persistenceis differentiable with respect to the original data. Therehave recently been a number of approaches suggested forthe integration of PH and deep learning, which we brieflyreview here.In Chen et al. (2018) a classification task was considered,and PH was used to regularise the decision boundary. Typ-ical regularisation of a decision boundary might encour-age it to be smooth or to be far from the data. Here, theboundary was encouraged to be simple from a topologi- cal point of view, meaning that topological complexitiessuch as loops and handles in the decision boundary werediscouraged. Rieck et al. (2018) proposed a measure ofthe complexity of a neural network using PH. This mea-sure of ‘neural persistence’ was evaluated as a measure ofstructural complexity at each layer of the network, and wasshown to increase during network training as well as beinguseful as a stopping criterion.PH is applied to image segmentation, but the PH calcu-lation has typically been applied to the input image andused as a way to generate features which can then be usedby another algorithm. Applications have included tumoursegmentation Qaiser et al. (2016), cell segmentation As-saf et al. (2017) and cardiac segmentation from computedtomography (CT) imaging Gao et al. (2013). RecentlyClough et al. (2019a) proposed to use PH not to the inputimage being segmented, but rather to the candidate seg-mentation provided by the network. In an extended workClough et al. Clough et al. (2020) the topological informa-tion found by the PH calculation can be used to provide atraining signal to the network, allowing an differentiableloss function to compare the topological features presentin a proposed segmentation, with those specified to existby some prior knowledge.
As the deep learning research effort for medical imagesegmentation is consolidating towards incorporating shapeconstraints to ensure downstream analysis, certain patternsare emerging as well. In the next few subsections, we dis-cuss such clear patterns and emerging questions relevantfor the progress of research in this direction.
With the maturity of research, this field is clearly mov-ing beyond post-/pre-hoc setting towards more systematicend-to-end training approaches. This effect is depicted inFigure 4 where the paper counts are aggregated from thiswork and Jurdi et al. (2020). The maturity of deep learn-ing frameworks (especially PyTorch), novel architectures(especially generative modeling) and automatic differentia-tion make it possible to incorporate complex shape-basedloss functions during training. With the availability ofthese tools, large models can be trained with tailored shapestreams in the model architecture to incorporate shapeinformation. f P ape r s Post-/pre-hocEnd-to-end
Figure 4: Temporal trend towards end-to-end approaches
The ability to incorporate additional information usingshape as a prior can aid in reducing the total number ofnecessary annotations in achieving a good segmentation.The shape priors can useful in generating controlled dataaugmentations for the medical image analysis task in handand reduce the number of unrealistic augmentations. Thiswould be instrumental in particular in the case of rarediseases, where there is not enough of data and manualannotations to train a neural network. The shape priorsthat are giving clues about the expected pathology in suchcases can lead to better segmentation accuracy in the finaloutput.12
PREPRINT - J
ANUARY
20, 2021
One common theme identified by last few decades worthresearch on shape modeling is the difficulty in represent-ing the pathological shapes. While the "typical shapes"i.e. normal shapes lie in a low-dimensional sub-manifold,the pathological cases have a long tail in the distribution(e.g. congenital heart diseases). That is normal shapesare self-similar but pathological cases contain atypicalshapes along with typical pathologies. Traditional lin-earized shape modeling had trouble addressing this issuewhereas the non-linear modeling of shape statistics had itsissue in terms of intractable numerics. Whether a neuralapproach can address this overarching problem of encod-ing pathological shapes is an open problem. Unfortunately,from our literature search, we have not found any cleardirection to address this perennial issue of shape modeling.
While the shape constraints are becoming increasinglycommonplace for medical image segmentation, we believethe visual perception and human comprehension plays asignificant role behind the interest of the community. Themore general question of real world effectiveness of thesemethods are not often studied. For example, how effectivethese shape constraints are under noisy annotation is anopen question? While the segmentation quality is most of-ten measured by the Dice metric, Maier-Hein et al. (2018)has already prescribed to move beyond Dice to evaluate thesegmentation quality. Topological accuracy of anatomicalstructures is increasingly used as an evaluation metric toaddress the shortcomings of classical image segmentationevaluation metric in medical image analysis Byrne et al.(2020). Finally, segmentation is typically a mean to anend. As such, the effectiveness of these segmentation tech-niques should be measured quantitatively for downstreamevaluation tasks such as visualization, planning Fauser et al.(2019) etc.
Bringing prior knowledge about the shape of the anatomyfor semantic segmentation is a rather well-trodden idea.The community is devising new ways to incorporate suchprior knowledge in deep learning models trained with fre-quentist approach. While the Bayesian interpretation ofdeep learning segmentation networks is an upcoming trend,it is already shown that under careful considerations, priorknowledge about the shape can be incorporated even infrequentist approaches with significant success.We see the future research concentrating more on end-to-end networks with the overarching theme of learning usingAnalysis-by-synthesis. Early work has demonstrated theeffectiveness of shape constraints in federated learning andthis will be a major direction in the coming years.We believe the community needs to address the issues dis-cussed in Section 6 before shape constrained segmentation can be considered as a trustworthy technology in practi-cal medical image analysis. To this end, we can think ofshape constrained segmentation as a technical buildingblock within a bigger image analysis pipeline rather than astand-alone piece of technology. For example, in the caseof surgical planning and navigation pipeline, such shapeconstraints can be meaningful provided the performanceis thoroughly validated under pathological cases with mul-tiple quality metrics. Important steps have already beentaken in this direction. In short, along with exciting results,shape constrained deep learning for segmentation opensup many possible research questions for the next few years.Proper understanding and answering those hold the key totheir successful deployment in the real clinical scenario.
References
M. Masroor Ahmed, Dzulkifli Bin Mohamad, and Mo-hammed S. Khalil. A hybrid approach for segment-ing and validating t1-weighted normal brain MR im-ages by employing ACM and ANN. In Ajith Abra-ham, Azah Kamilah Muda, Nanna Suryana Herman,Siti Mariyam Shamsuddin, and Yun-Huoy Choo, editors,
First International Conference of Soft Computing andPattern Recognition, SoCPaR 2009, Malacca, Malaysia,December 4-7, 2009 , pages 239–244. IEEE ComputerSociety, 2009. doi: 10.1109/SoCPaR.2009.56. URL https://doi.org/10.1109/SoCPaR.2009.56 .Amir Alansary, Konstantinos Kamnitsas, Alice David-son, Rostislav Khlebnikov, Martin Rajchl, ChristinaMalamateniou, Mary A. Rutherford, Joseph V. Ha-jnal, Ben Glocker, Daniel Rueckert, and BernhardKainz. Fast fully automatic segmentation of the hu-man placenta from motion corrupted MRI. In
Med-ical Image Computing and Computer-Assisted Inter-vention - MICCAI 2016 - 19th International Confer-ence, Athens, Greece, October 17-21, 2016, Proceed-ings, Part II , pages 589–597, 2016. doi: 10.1007/978-3-319-46723-8\_68. URL https://doi.org/10.1007/978-3-319-46723-8_68 .Felix Ambellan, Alexander Tack, Moritz Ehlke, and Ste-fan Zachow. Automated segmentation of knee bone andcartilage combining statistical shape knowledge and con-volutional neural networks: Data from the osteoarthri-tis initiative.
Medical Image Analysis , 52:109–118,2019. doi: 10.1016/j.media.2018.11.009. URL https://doi.org/10.1016/j.media.2018.11.009 .Alex M. Andrew.
Level Set Methods and Fast March-ing Methods: Evolving Interfaces in ComputationalGeometry, Fluid Mechanics, Computer Vision, andMaterials Science , by J.A. sethian, cambridge uni-versity press, cambridge, uk, 2nd edn 1999 (firstpublished 1996 as
Level Set Methods ) xviii + 420pp., ISBN (paperback) 0-521-64557-3, (hardback) 0-521-64204-3 (pbk, £18.95).
Robotica , 18(1):89–92,2000. URL http://journals.cambridge.org/action/displayAbstract?aid=34609 .13
PREPRINT - J
ANUARY
20, 2021Rabih Assaf, Alban Goupil, Mohammad Kacim, and Va-leriu Vrabie. Topological persistence based on pix-els for object segmentation in biomedical images. In , pages 1–4. IEEE,2017.M. R. Avendi, Arash Kheradvar, and Hamid Jafarkhani.A combined deep-learning and deformable-model ap-proach to fully automatic segmentation of the left ventri-cle in cardiac MRI.
Medical Image Anal. , 30:108–119,2016. doi: 10.1016/j.media.2016.01.005. URL https://doi.org/10.1016/j.media.2016.01.005 .B. J. Bhatkalkar, D. R. Reddy, S. Prabhu, and S. V.Bhandary. Improving the performance of convolutionalneural network for the segmentation of optic disc infundus images using attention gates and conditional ran-dom fields.
IEEE Access , 8:29299–29310, 2020. doi:10.1109/ACCESS.2020.2972318.Irene Brusini, Olof Lindberg, J-Sebastian Muehlboeck,Örjan Smedby, Eric Westman, and Chunliang Wang.Shape information improves the cross-cohort per-formance of deep learning-based segmentation ofthe hippocampus.
Frontiers in Neuroscience , 14:15, 2020. ISSN 1662-453X. doi: 10.3389/fnins.2020.00015. URL .Nick Byrne, James R Clough, Giovanni Montana, and An-drew P King. A persistent homology-based topologicalloss function for multi-class cnn segmentation of cardiacmri. arXiv preprint arXiv:2008.09585 , 2020.Jinzheng Cai, Le Lu, Yuanpu Xie, Fuyong Xing, and LinYang. Pancreas segmentation in MRI using graph-baseddecision fusion on convolutional neural networks. In
Medical Image Computing and Computer Assisted In-tervention - MICCAI 2017 - 20th International Con-ference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part III , pages 674–682, 2017.doi: 10.1007/978-3-319-66179-7\_77. URL https://doi.org/10.1007/978-3-319-66179-7_77 .Ruiming Cao, Xinran Zhong, Sepideh Shakeri, Amirhos-sein Mohammadian Bajgiran, Sohrab Afshari Mirak,Dieter Enzmann, Steven S. Raman, and Kyung HyunSung. Prostate cancer detection and segmentation inmulti-parametric MRI via CNN and conditional ran-dom field. In , pages 1900–1904. IEEE, 2019. doi: 10.1109/ISBI.2019.8759584. URL https://doi.org/10.1109/ISBI.2019.8759584 .Erik Carbajal-Degante, Steve Avendaño, LeonardoLedesma, Jimena Olveres, and Boris Escalante-Ramírez.Active contours for multi-region segmentation witha convolutional neural network initialization. In Pe-ter Schelkens and Tomasz Kozacki, editors,
Optics,Photonics and Digital Technologies for Imaging Ap-plications VI , volume 11353, pages 36 – 44. Interna-tional Society for Optics and Photonics, SPIE, 2020. doi: 10.1117/12.2556928. URL https://doi.org/10.1117/12.2556928 .Kenny Cha, Lubomir Hadjiiski, Ravi Samala, Heang-Ping Chan, Elaine M. Caoili, and Richard H. Co-han. Urinary bladder segmentation in ct urographyusing deep-learning convolutional neural network andlevel sets.
Medical Physics , 43:1882–1896, 04 2016.doi: 10.1118/1.4944498. URL https://doi.org/10.1118/1.4944498 .Chao Chen, Xiuyan Ni, Qinxun Bai, and Yusu Wang.TopoReg: A Topological Regularizer for Classifiers. arXiv 1806.10714 , 2018.Shuai Chen and Marleen de Bruijne. An end-to-end ap-proach to semantic segmentation with 3d CNN andposterior-crf in medical images.
CoRR , abs/1811.03549,2018. URL http://arxiv.org/abs/1811.03549 .Ruida Cheng, Holger R. Roth, Le Lu, Shijun Wang, BarisTurkbey, William Gandler, Evan S. McCreedy, Harsh K.Agarwal, Peter L. Choyke, Ronald M. Summers, andMatthew J. McAuliffe. Active appearance model anddeep learning for more accurate prostate segmentationon MRI. In
Medical Imaging 2016: Image Process-ing, San Diego, California, USA, February 27, 2016 ,page 97842I, 2016. doi: 10.1117/12.2216286. URL https://doi.org/10.1117/12.2216286 .Patrick Ferdinand Christ, Mohamed Ezzeldin A. Elshaer,Florian Ettlinger, Sunil Tatavarty, Marc Bickel, PatrickBilic, Markus Rempfler, Marco Armbruster, Felix Hof-mann, Melvin D’Anastasi, Wieland H. Sommer, Seyed-Ahmad Ahmadi, and Bjoern H. Menze. Automaticliver and lesion segmentation in CT using cascadedfully convolutional neural networks and 3d conditionalrandom fields. In
Medical Image Computing andComputer-Assisted Intervention - MICCAI 2016 - 19thInternational Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II , pages 415–423, 2016.doi: 10.1007/978-3-319-46723-8\_48. URL https://doi.org/10.1007/978-3-319-46723-8_48 .J. Clough, N. Byrne, I. Oksuz, V. A. Zimmer, J. A. Schn-abel, and A. King. A topological loss function for deep-learning based image segmentation using persistent ho-mology.
IEEE Transactions on Pattern Analysis andMachine Intelligence , pages 1–1, 2020.James R Clough, Ilkay Oksuz, Nicholas Byrne, Julia ASchnabel, and Andrew P King. Explicit topological pri-ors for deep-learning based image segmentation usingpersistent homology. In
International Conference on In-formation Processing in Medical Imaging , pages 16–28.Springer, 2019a.James R. Clough, Ilkay Öksüz, Nicholas Byrne,Veronika A. Zimmer, Julia A. Schnabel, and Andrew P.King. A topological loss function for deep-learningbased image segmentation using persistent homology.
CoRR , abs/1910.01877, 2019b. URL http://arxiv.org/abs/1910.01877 .14
PREPRINT - J
ANUARY
20, 2021Timothy F. Cootes, Christopher J. Taylor, David H. Cooper,and Jim Graham. Active shape models-their training andapplication.
Computer Vision and Image Understanding ,61(1):38–59, 1995. doi: 10.1006/cviu.1995.1004. URL https://doi.org/10.1006/cviu.1995.1004 .Giovanni Lucca França da Silva, Petterson Sousa Diniz,Jonnison Lima Ferreira, João Vitor Ferreira França,Aristófanes C. Silva, Anselmo Cardoso de Paiva, andElton Anderson Araújo de Cavalcanti. Superpixel-baseddeep convolutional neural networks and active contourmodel for automatic prostate segmentation on 3d MRIscans.
Medical Biol. Eng. Comput. , 58(9):1947–1964,2020. doi: 10.1007/s11517-020-02199-5. URL https://doi.org/10.1007/s11517-020-02199-5 .Wu Deng, Qinke Shi, Miye Wang, Bing Zheng, and NingNing. Deep learning-based HCNN and CRF-RRNNmodel for brain tumor segmentation.
IEEE Access ,8:26665–26675, 2020. doi: 10.1109/ACCESS.2020.2966879. URL https://doi.org/10.1109/ACCESS.2020.2966879 .Qi Dou, Hao Chen, Yueming Jin, Lequan Yu, Jing Qin,and Pheng-Ann Heng. 3d deeply supervised networkfor automatic liver segmentation from CT volumes. In
Medical Image Computing and Computer-Assisted In-tervention - MICCAI 2016 - 19th International Con-ference, Athens, Greece, October 17-21, 2016, Pro-ceedings, Part II , pages 149–157, 2016. doi: 10.1007/978-3-319-46723-8\_18. URL https://doi.org/10.1007/978-3-319-46723-8_18 .Qi Dou, Lequan Yu, Hao Chen, Yueming Jin, Xin Yang,Jing Qin, and Pheng-Ann Heng. 3d deeply super-vised network for automated segmentation of volumetricmedical images.
Medical Image Analysis , 41:40–54,2017. doi: 10.1016/j.media.2017.05.001. URL https://doi.org/10.1016/j.media.2017.05.001 .Nguyen Ho Minh Duy, Nguyen Manh Duy, MaiThanh Nhat Truong, Pham The Bao, and Thanh BinhNguyen. Accurate brain extraction using active shapemodel and convolutional neural networks.
CoRR ,abs/1802.01268, 2018. URL http://arxiv.org/abs/1802.01268 .Herbert Edelsbrunner, David Letscher, and Afra Zomoro-dian. Topological persistence and simplification. In
Foundations of Computer Science , pages 454–463.IEEE, 2000.Ahmed Elnakib, Georgy Gimel’farb, Jasjit S. Suri, andAyman El-Baz.
Medical Image Segmentation: A BriefSurvey , pages 1–39. Springer New York, New York,NY, 2011. ISBN 978-1-4419-8204-9. doi: 10.1007/978-1-4419-8204-9_1. URL https://doi.org/10.1007/978-1-4419-8204-9_1 .Yubo Fan, Dongqing Zhang, Jianing Wang, Jack H. No-ble, and Benoit M. Dawant. Combining model- anddeep-learning-based methods for the accurate and ro-bust segmentation of the intra-cochlear anatomy in clin-ical head CT images. In Ivana Isgum and Bennett A. Landman, editors,
Medical Imaging 2020: Image Pro-cessing, Houston, TX, USA, February 15-20, 2020 ,volume 11313 of
SPIE Proceedings , page 113131D.SPIE, 2020. doi: 10.1117/12.2549390. URL https://doi.org/10.1117/12.2549390 .Zhou Fang, Mengyun Qiao, Yi Guo, Yuanyuan Wang,Jiawei Li, Shichong Zhou, and Cai Chang. Combin-ing a fully convolutional network and an active con-tour model for automatic 2d breast tumor segmentationfrom ultrasound images.
Journal of Medical Imagingand Health Informatics , 9:1510–1515, 09 2019. doi:10.1166/jmihi.2019.2752. URL https://doi.org/10.1166/jmihi.2019.2752 .Johannes Fauser, Igor Stenin, Markus Bauer, Wei-HungHsu, Julia Kristin, Thomas Klenzner, Jörg Schipper, andAnirban Mukhopadhyay. Toward an automatic preoper-ative pipeline for image-guided temporal bone surgery.
Int. J. Comput. Assist. Radiol. Surg. , 14(6):967–976,2019. doi: 10.1007/s11548-019-01937-x. URL https://doi.org/10.1007/s11548-019-01937-x .Naiqin Feng, Xiuqin Geng, and Lijuan Qin. Study onMRI medical image segmentation technology based onCNN-CRF model.
IEEE Access , 8:60505–60514, 2020.doi: 10.1109/ACCESS.2020.2982197. URL https://doi.org/10.1109/ACCESS.2020.2982197 .Huazhu Fu, Yanwu Xu, Stephen Lin, Damon Wing KeeWong, and Jiang Liu. Deepvessel: Retinal vessel seg-mentation via deep learning and conditional randomfield. In
Medical Image Computing and Computer-Assisted Intervention - MICCAI 2016 - 19th Interna-tional Conference, Athens, Greece, October 17-21,2016, Proceedings, Part II , pages 132–139, 2016a.doi: 10.1007/978-3-319-46723-8\_16. URL https://doi.org/10.1007/978-3-319-46723-8_16 .Huazhu Fu, Yanwu Xu, Damon Wing Kee Wong, andJiang Liu. Retinal vessel segmentation via deep learningnetwork and fully-connected conditional random fields.In , pages 698–701, 2016b. doi: 10.1109/ISBI.2016.7493362. URL https://doi.org/10.1109/ISBI.2016.7493362 .Mingchen Gao, Chao Chen, Shaoting Zhang, Zhen Qian,Dimitris Metaxas, and Leon Axel. Segmenting the pap-illary muscles and the trabeculae from high resolutioncardiac CT through restoration of topological handles.In
IPMI , pages 184–195. Springer, 2013.Mingchen Gao, Ziyue Xu, Le Lu, Aaron Wu, IsabellaNogues, Ronald M. Summers, and Daniel J. Mol-lura. Segmentation label propagation using deep con-volutional neural networks and dense conditional ran-dom field. In , pages 1265–1268, 2016.doi: 10.1109/ISBI.2016.7493497. URL https://doi.org/10.1109/ISBI.2016.7493497 .15
PREPRINT - J
ANUARY
20, 2021Zhaoxuan Gong, Zhenyu Zhu, Guodong Zhang, DazheZhao, and Wei Guo. Convolutional neural networksbased level set framework for pancreas segmentationfrom CT images. In
Proceedings of the Third Inter-national Symposium on Image Computing and Dig-ital Medicine, ISICDM 2019, Xi’an, China, August24-26, 2019 , pages 27–30. ACM, 2019. doi: 10.1145/3364836.3364842. URL https://doi.org/10.1145/3364836.3364842 .Xiaotao Guo, Lawrence H. Schwartz, and Binsheng Zhao.Automatic liver segmentation by integrating fully con-volutional networks into active contour models.
Med-ical Physics , 07 2019. doi: 10.1002/mp.13735. URL https://doi.org/10.1002/mp.13735 .Sang Yoon Han, Hyuk Jin Kwon, Yoonsik Kim, andNam Ik Cho. Noise-robust pupil center detectionthrough cnn-based segmentation with shape-prior loss.
IEEE Access , 8:64739–64749, 2020. doi: 10.1109/ACCESS.2020.2985095. URL https://doi.org/10.1109/ACCESS.2020.2985095 .Ali Hatamizadeh, Assaf Hoogi, Debleena Sengupta,Wuyue Lu, Brian Wilcox, Daniel L. Rubin, and DemetriTerzopoulos. Deep active lesion segmentation.
CoRR ,abs/1908.06933, 2019. URL http://arxiv.org/abs/1908.06933 .Baochun He, Deqiang Xiao, Qingmao Hu, and FucangJia. Automatic magnetic resonance image prostate seg-mentation based on adaptive feature learning proba-bility boosting tree initialization and CNN-ASM re-finement.
IEEE Access , 6:2005–2015, 2018. doi:10.1109/ACCESS.2017.2781278. URL https://doi.org/10.1109/ACCESS.2017.2781278 .Tobias Heimann and Hans-Peter Meinzer. Statisticalshape models for 3d medical image segmentation: Areview.
Medical Image Analysis , 13(4):543–563, 2009.doi: 10.1016/j.media.2009.05.004. URL https://doi.org/10.1016/j.media.2009.05.004 .Mohammad Hesam Hesamian, Wenjing Jia, XiangjianHe, and Paul J. Kennedy. Deep learning techniquesfor medical image segmentation: Achievements andchallenges.
J. Digit. Imaging , 32(4):582–596, 2019.doi: 10.1007/s10278-019-00227-x. URL https://doi.org/10.1007/s10278-019-00227-x .Assaf Hoogi, Arjun Subramaniam, Rishi Veerapaneni, andDaniel L. Rubin. Adaptive estimation of active con-tour parameters using convolutional neural networksand texture analysis.
IEEE Trans. Med. Imaging , 36(3):781–791, 2017. doi: 10.1109/TMI.2016.2628084. URL https://doi.org/10.1109/TMI.2016.2628084 .Wei-Yen Hsu. Automatic left ventricle recognition, seg-mentation and tracking in cardiac ultrasound imagesequences.
IEEE Access , 7:140524–140533, 2019.doi: 10.1109/ACCESS.2019.2920957. URL https://doi.org/10.1109/ACCESS.2019.2920957 .Kai Hu, Qinghai Gan, Yuan Zhang, Shuhua Deng, FenXiao, Wei Huang, Chunhong Cao, and Xieping Gao. Brain tumor segmentation using multi-cascaded convo-lutional neural networks and conditional random field.
IEEE Access , 7:92615–92629, 2019. doi: 10.1109/ACCESS.2019.2927433. URL https://doi.org/10.1109/ACCESS.2019.2927433 .Yuzhou Hu, Yi Guo, Yuanyuan Wang, Jinhua Yu, JiaweiLi, Shichong Zhou, and Cai Chang. Automatic tumorsegmentation in breast ultrasound images using a di-lated fully convolutional network combined with anactive contour model.
Medical Physics , 46, 10 2018.doi: 10.1002/mp.13268. URL https://doi.org/10.1002/mp.13268 .Cheng Jin, Jianjiang Feng, Lei Wang, Heng Yu, Jiang Liu,Jiwen Lu, and Jie Zhou. Left atrial appendage segmen-tation using fully convolutional neural networks andmodified three-dimensional conditional random fields.
IEEE J. Biomedical and Health Informatics , 22(6):1906–1916, 2018. doi: 10.1109/JBHI.2018.2794552. URL https://doi.org/10.1109/JBHI.2018.2794552 .Rosana El Jurdi, Caroline Petitjean, Paul Honeine,Veronika Cheplygina, and Fahed Abdallah. High-levelprior-based loss functions for medical image segmen-tation: A survey.
CoRR , abs/2011.08018, 2020. URL https://arxiv.org/abs/2011.08018 .Konstantinos Kamnitsas, Christian Ledig, Virginia F. J.Newcombe, Joanna P. Simpson, Andrew D. Kane,David K. Menon, Daniel Rueckert, and Ben Glocker.Efficient multi-scale 3d CNN with fully connected CRFfor accurate brain lesion segmentation.
Medical ImageAnalysis , 36:61–78, 2017. doi: 10.1016/j.media.2016.10.004. URL https://doi.org/10.1016/j.media.2016.10.004 .Davood Karimi, Golnoosh Samei, Claudia Kesch, Guy Nir,and Septimiu E. Salcudean. Prostate segmentation inMRI using a convolutional neural network architectureand training strategy based on statistical shape mod-els.
Int. J. Comput. Assist. Radiol. Surg. , 13(8):1211–1219, 2018. doi: 10.1007/s11548-018-1785-8. URL https://doi.org/10.1007/s11548-018-1785-8 .Davood Karimi, Qi Zeng, Prateek Mathur, ApekshaAvinash, Sara Mahdavi, Ingrid Spadinger, Purang Abol-maesumi, and Septimiu E. Salcudean. Accurate androbust deep learning-based segmentation of the prostateclinical target volume in ultrasound images.
Medical Im-age Anal. , 57:186–196, 2019. doi: 10.1016/j.media.2019.07.005. URL https://doi.org/10.1016/j.media.2019.07.005 .Gopi Kasinathan, Selvakumar Jayakumar, Amir H. Gan-domi, Manikandan Ramachandran, Simon James Fong,and Rizwan Patan. Automated 3-d lung tumor de-tection and classification by an active contour modeland CNN classifier.
Expert Syst. Appl. , 134:112–119,2019. doi: 10.1016/j.eswa.2019.05.041. URL https://doi.org/10.1016/j.eswa.2019.05.041 .Michael Kass, Andrew P. Witkin, and Demetri Terzopou-los. Snakes: Active contour models.
International PREPRINT - J
ANUARY
20, 2021
Journal of Computer Vision , 1(4):321–331, 1988. doi:10.1007/BF00133570. URL https://doi.org/10.1007/BF00133570 .Estera Kot, Zuzanna Krawczyk, Krzysztof Siwek, andPiotr S. Czwarnowski. U-net and active contourmethods for brain tumour segmentation and visualiza-tion. In , pages 1–7. IEEE, 2020. doi:10.1109/IJCNN48605.2020.9207572. URL https://doi.org/10.1109/IJCNN48605.2020.9207572 .John D. Lafferty, Andrew McCallum, and Fernando C. N.Pereira. Conditional random fields: Probabilistic mod-els for segmenting and labeling sequence data. In
Pro-ceedings of the Eighteenth International Conferenceon Machine Learning (ICML 2001), Williams College,Williamstown, MA, USA, June 28 - July 1, 2001 , pages282–289, 2001.Matthew Chung Hai Lee, Kersten Petersen, NickPawlowski, Ben Glocker, and Michiel Schaap. TETRIS:Template transformer networks for image segmentationwith shape priors.
IEEE transactions on medical imag-ing , 2019.Tao Lei, Risheng Wang, Yong Wan, Xiaogang Du, Hongy-ing Meng, and Asoke K. Nandi. Medical image seg-mentation using deep learning: A survey.
CoRR ,abs/2009.13120, 2020. URL https://arxiv.org/abs/2009.13120 .Lei Li, Xin Weng, Julia A. Schnabel, and Xiahai Zhuang.Joint left atrial segmentation and scar quantificationbased on a DNN with spatial encoding and shape at-tention. In Anne L. Martel, Purang Abolmaesumi,Danail Stoyanov, Diana Mateus, Maria A. Zuluaga,S. Kevin Zhou, Daniel Racoceanu, and Leo Joskow-icz, editors,
Medical Image Computing and ComputerAssisted Intervention - MICCAI 2020 - 23rd Interna-tional Conference, Lima, Peru, October 4-8, 2020,Proceedings, Part IV , volume 12264 of
Lecture Notesin Computer Science , pages 118–127. Springer, 2020.doi: 10.1007/978-3-030-59719-1\_12. URL https://doi.org/10.1007/978-3-030-59719-1_12 .Stan Z. Li. Markov random field models in computervision. In
Computer Vision - ECCV’94, Third Euro-pean Conference on Computer Vision, Stockholm, Swe-den, May 2-6, 1994, Proceedings, Volume II , pages361–370, 1994. doi: 10.1007/BFb0028368. URL https://doi.org/10.1007/BFb0028368 .Yuanwei Li, Chin Pang Ho, Matthieu Toulemonde, NavtejChahal, Roxy Senior, and Meng-Xing Tang. Fully au-tomatic myocardial segmentation of contrast echocar-diography sequence using random forests guided byshape model.
IEEE Trans. Med. Imaging , 37(5):1081–1091, 2018. doi: 10.1109/TMI.2017.2747081. URL https://doi.org/10.1109/TMI.2017.2747081 .Zeju Li, Yuanyuan Wang, Jinhua Yu, Zhifeng Shi, Yi Guo,Liang Chen, and Ying Mao. Low-grade glioma segmen-tation based on cnn with fully connected crf.
Journal of Healthcare Engineering , 2017:1–12, 06 2017a. doi:10.1155/2017/9283480. URL https://doi.org/10.1155/2017/9283480 .Zewen Li, Adan Lin, Xuan Yang, and Junhao Wu. Leftventricle segmentation by combining convolution neu-ral network with active contour model and tensor vot-ing in short-axis MRI. In ,pages 736–739, 2017b. doi: 10.1109/BIBM.2017.8217746. URL https://doi.org/10.1109/BIBM.2017.8217746 .Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi,Arnaud Arindra Adiyoso Setio, Francesco Ciompi,Mohsen Ghafoorian, Jeroen A. W. M. van der Laak,Bram van Ginneken, and Clara I. Sánchez. A survey ondeep learning in medical image analysis.
Medical Im-age Anal. , 42:60–88, 2017. doi: 10.1016/j.media.2017.07.005. URL https://doi.org/10.1016/j.media.2017.07.005 .Yashu Liu, Kuanquan Wang, Gongning Luo, and Heng-gui Zhang. A framework for left atrium segmenta-tion on CT images with combined detection networkand level set model. In ,pages 1–4. IEEE, 2019. doi: 10.23919/CinC49843.2019.9005853. URL https://doi.org/10.23919/CinC49843.2019.9005853 .Yiming Liu, Pengcheng Zhang, Qingche Song, Andi Li,Peng Zhang, and Zhiguo Gui. Automatic segmentationof cervical nuclei based on deep learning and a condi-tional random field.
IEEE Access , 6:53709–53721, 2018.doi: 10.1109/ACCESS.2018.2871153. URL https://doi.org/10.1109/ACCESS.2018.2871153 .Wenfeng Luo and Meng Yang. Fast skin lesion segmen-tation via fully convolutional network with residual ar-chitecture and CRF. In , pages 1438–1443. IEEE ComputerSociety, 2018. doi: 10.1109/ICPR.2018.8545571. URL https://doi.org/10.1109/ICPR.2018.8545571 .Yuansheng Luo, Lu Yang, Ling Wang, and Hong Cheng.Efficient cnn-crf network for retinal image segmenta-tion. In Fuchun Sun, Huaping Liu, and Dewen Hu,editors,
Cognitive Systems and Signal Processing , pages157–165, Singapore, 2017. Springer Singapore. ISBN978-981-10-5230-9.Jingting Ma, Feng Lin, Stefan Wesarg, and Marius Erdt. Anovel bayesian model incorporating deep neural networkand statistical shape model for pancreas segmentation.In Alejandro F. Frangi, Julia A. Schnabel, Christos Da-vatzikos, Carlos Alberola-López, and Gabor Fichtinger,editors,
Medical Image Computing and Computer As-sisted Intervention - MICCAI 2018 - 21st InternationalConference, Granada, Spain, September 16-20, 2018,Proceedings, Part IV , volume 11073 of
Lecture Notesin Computer Science , pages 480–487. Springer, 2018.17
PREPRINT - J
ANUARY
20, 2021doi: 10.1007/978-3-030-00937-3\_55. URL https://doi.org/10.1007/978-3-030-00937-3_55 .Jun Ma and Xiaoping Yang. Automatic dental root CBCTimage segmentation based on CNN and level set method.In
Medical Imaging 2019: Image Processing, San Diego,California, United States, 16-21 February 2019 , page109492N, 2019. doi: 10.1117/12.2512359. URL https://doi.org/10.1117/12.2512359 .Lena Maier-Hein, Matthias Eisenmann, Annika Reinke,Sinan Onogur, Marko Stankovic, Patrick Scholz, TalArbel, Hrvoje Bogunovic, Andrew P. Bradley, AaronCarass, Carolin Feldmann, Alejandro F. Frangi, Pe-ter M. Full, Bram van Ginneken, Allan Hanbury, Ka-trin Honauer, Michal Kozubek, Bennett A. Landman,Keno März, Oskar Maier, Klaus H. Maier-Hein, Bjo-ern H. Menze, Henning Müller, Peter F. Neher, Wiro J.Niessen, Nasir M. Rajpoot, Gregory C. Sharp, KorsukSirinukunwattana, Stefanie Speidel, Christian Stock,Danail Stoyanov, Abdel Aziz Taha, Fons van der Som-men, Ching-Wei Wang, Marc-André Weber, GuoyanZheng, Pierre Jannin, and Annette Kopp-Schneider. Isthe winner really the best? A critical analysis of com-mon research practice in biomedical image analysiscompetitions.
CoRR , abs/1806.02051, 2018. URL http://arxiv.org/abs/1806.02051 .Ravi Malladi, James A. Sethian, and Baba C. Vemuri.Shape modeling with front propagation: A level set ap-proach.
IEEE Trans. Pattern Anal. Mach. Intell. , 17(2):158–175, 1995. doi: 10.1109/34.368173. URL https://doi.org/10.1109/34.368173 .Tim McInerney and Demetri Terzopoulos. Deformablemodels in medical image analysis: a survey.
Medi-cal Image Anal. , 1(2):91–108, 1996. doi: 10.1016/S1361-8415(96)80007-7. URL https://doi.org/10.1016/S1361-8415(96)80007-7 .Daniela O. Medley, Carlos Santiago, and Jacinto C. Nasci-mento. Deep active shape model for robust objectfitting.
IEEE Trans. Image Process. , 29:2380–2394,2020. doi: 10.1109/TIP.2019.2948728. URL https://doi.org/10.1109/TIP.2019.2948728 .Russel Mesbah, Brendan McCane, and Steven Mills. Con-ditional random fields incorporate convolutional neu-ral networks for human eye sclera semantic segmen-tation. In , pages 768–773. IEEE, 2017. doi:10.1109/BTAS.2017.8272768. URL https://doi.org/10.1109/BTAS.2017.8272768 .Ian Middleton and Robert Damper. Segmentation ofmagnetic resonance images using a combination ofneural networks and active contour models.
Medi-cal engineering & physics , 26:71–86, 02 2004. doi:10.1016/S1350-4533(03)00137-1. URL https://doi.org/10.1016/S1350-4533(03)00137-1 .Saeed Mohagheghi and Amir Hossein Foruzan. Incor-porating prior shape knowledge via data-driven loss model to improve 3d liver segmentation in deep cnns.
Int. J. Comput. Assist. Radiol. Surg. , 15(2):249–257,2020. doi: 10.1007/s11548-019-02085-y. URL https://doi.org/10.1007/s11548-019-02085-y .Miguel Monteiro, Mário A. T. Figueiredo, and Arlindo L.Oliveira. Conditional random fields as recurrent neuralnetworks for 3d medical imaging segmentation.
CoRR ,abs/1807.07464, 2018. URL http://arxiv.org/abs/1807.07464 .Huu-Giao Nguyen, Alessia Pica, Philippe Maeder, AnnSchalenbourg, Marta Peroni, Jan Hrbacek, Damien C.Weber, Meritxell Bach Cuadra, and Raphael Sznit-man. Ocular structures segmentation from multi-sequences MRI using 3d unet with fully connectedcrfs. In Danail Stoyanov, Zeike Taylor, FrancescoCiompi, Yanwu Xu, Anne L. Martel, Lena Maier-Hein,Nasir M. Rajpoot, Jeroen van der Laak, Mitko Veta,Stephen J. McKenna, David R. J. Snead, EmanueleTrucco, Mona Kathryn Garvin, Xin Jan Chen, andHrvoje Bogunovic, editors,
Computational Pathologyand Ophthalmic Medical Image Analysis - First Inter-national Workshop, COMPAY 2018, and 5th Interna-tional Workshop, OMIA 2018, Held in Conjunctionwith MICCAI 2018, Granada, Spain, September 16-20, 2018, Proceedings , volume 11039 of
Lecture Notesin Computer Science , pages 167–175. Springer, 2018.doi: 10.1007/978-3-030-00949-6\_20. URL https://doi.org/10.1007/978-3-030-00949-6_20 .Huu-Giao Nguyen, Alessia Pica, Jan Hrbacek, Damien C.Weber, Francesco La Rosa, Ann Schalenbourg, RaphaelSznitman, and Meritxell Bach Cuadra. A novel seg-mentation framework for uveal melanoma in magneticresonance imaging based on class activation maps. InM. Jorge Cardoso, Aasa Feragen, Ben Glocker, EnderKonukoglu, Ipek Oguz, Gozde B. Unal, and Tom Ver-cauteren, editors,
International Conference on MedicalImaging with Deep Learning, MIDL 2019, 8-10 July2019, London, United Kingdom , volume 102 of
Pro-ceedings of Machine Learning Research , pages 370–379. PMLR, 2019. URL http://proceedings.mlr.press/v102/nguyen19a.html .Isabella Nogues, Le Lu, Xiaosong Wang, Holger Roth,Gedas Bertasius, Nathan Lay, Jianbo Shi, YohannesTsehay, and Ronald M. Summers. Automatic lymphnode cluster segmentation using holistically-nested neu-ral networks and structured optimization in CT im-ages. In
Medical Image Computing and Computer-Assisted Intervention - MICCAI 2016 - 19th Inter-national Conference, Athens, Greece, October 17-21,2016, Proceedings, Part II , pages 388–397, 2016. doi:10.1007/978-3-319-46723-8\_45. URL https://doi.org/10.1007/978-3-319-46723-8_45 .Masoud S. Nosrati and Ghassan Hamarneh. Incorpo-rating prior knowledge in medical image segmenta-tion: a survey.
CoRR , abs/1607.01092, 2016. URL http://arxiv.org/abs/1607.01092 .18
PREPRINT - J
ANUARY
20, 2021Virgínia Xavier Nunes, Aldísio Gonçalves Medeiros, Fran-cisco H. S. Silva, Gabriel M. Bezerra, and Pedro P. R.Filho. Adaptive level set with region analysis viamask R-CNN: A comparison against classical meth-ods. In , pages 1–8. IEEE, 2020. doi:10.1109/IJCNN48605.2020.9206664. URL https://doi.org/10.1109/IJCNN48605.2020.9206664 .Bo Peng, Lei Zhang, and David Zhang. A survey ofgraph theoretical approaches to image segmentation.
Pattern Recognit. , 46(3):1020–1038, 2013. doi: 10.1016/j.patcog.2012.09.015. URL https://doi.org/10.1016/j.patcog.2012.09.015 .Talha Qaiser, Korsuk Sirinukunwattana, Kazuaki Nakane,Yee-Wah Tsang, David Epstein, and Nasir Rajpoot. Per-sistent homology for fast tumor segmentation in wholeslide histology images.
Procedia Computer Science , 90:119–124, 2016.Chunxia Qin, Xiaojun Chen, and Jocelyne Troccaz. Aweakly supervised registration-based framework forprostate segmentation via the combination of statisticalshape model and CNN.
CoRR , abs/2007.11726, 2020.URL https://arxiv.org/abs/2007.11726 .Yuming Qiu, Jingyong Cai, Xiaolin Qin, and Ju Zhang.Inferring skin lesion segmentation with fully connectedcrfs based on multiple deep convolutional neural net-works.
IEEE Access , 8:144246–144258, 2020. doi:10.1109/ACCESS.2020.3014787. URL https://doi.org/10.1109/ACCESS.2020.3014787 .Martin Rajchl, Matthew C. H. Lee, Ozan Oktay, Kon-stantinos Kamnitsas, Jonathan Passerat-Palmbach, Wen-jia Bai, Mellisa Damodaram, Mary A. Rutherford,Joseph V. Hajnal, Bernhard Kainz, and Daniel Rueck-ert. Deepcut: Object segmentation from boundingbox annotations using convolutional neural networks.
IEEE Trans. Med. Imaging , 36(2):674–683, 2017. doi:10.1109/TMI.2016.2621185. URL https://doi.org/10.1109/TMI.2016.2621185 .Muhammad Imran Razzak, Saeeda Naz, and Ahmad Zaib.Deep learning for medical image processing: Overview,challenges and future.
CoRR , abs/1704.06825, 2017.URL http://arxiv.org/abs/1704.06825 .Bastian Rieck, Matteo Togninalli, Christian Bock, MichaelMoor, Max Horn, Thomas Gumbsch, and Karsten Borg-wardt. Neural persistence: A complexity measure fordeep neural networks using algebraic topology. arXivpreprint arXiv:1812.09764 , 2018.Intisar Rizwan I Haque and Jeremiah Neubert. Deeplearning approaches to biomedical image segmentation.
Informatics in Medicine Unlocked , 18:100297, 2020.ISSN 2352-9148. doi: https://doi.org/10.1016/j.imu.2020.100297. URL .Christian Rupprecht, Elizabeth Huaroc, Maximilian Baust,and Nassir Navab. Deep active contours.
CoRR , abs/1607.05074, 2016. URL http://arxiv.org/abs/1607.05074 .Ahad Salimi, Mohammad Ali Pourmina, and Moham-mad Shahram Moin. Fully automatic prostate seg-mentation in MR images using a new hybrid activecontour-based approach.
Signal, Image and VideoProcessing , 12(8):1629–1637, 2018. doi: 10.1007/s11760-018-1320-y. URL https://doi.org/10.1007/s11760-018-1320-y .Justus Schock, Marcin Kopaczka, Benjamin Agthe, JieHuang, Paul Kruse, Daniel Truhn, Stefan Conrad, Ger-ald Antoch, Christiane Kuhl, Sven Nebelung, and DoritMerhof. A method for semantic knee bone and cartilagesegmentation with deep 3d shape fitting using data fromthe osteoarthritis initiative. In Martin Reuter, ChristianWachinger, Hervé Lombaert, Beatriz Paniagua, OrcunGoksel, and Islem Rekik, editors,
Shape in MedicalImaging - International Workshop, ShapeMI 2020, Heldin Conjunction with MICCAI 2020, Lima, Peru, October4, 2020, Proceedings , volume 12474 of
Lecture Notesin Computer Science , pages 85–94. Springer, 2020.doi: 10.1007/978-3-030-61056-2\_7. URL https://doi.org/10.1007/978-3-030-61056-2_7 .Mahsa Shakeri, Stavros Tsogkas, Enzo Ferrante, SarahLippé, Samuel Kadoury, Nikos Paragios, and IasonasKokkinos. Sub-cortical brain structure segmentation us-ing f-cnn’s. In , pages 269–272, 2016.doi: 10.1109/ISBI.2016.7493261. URL https://doi.org/10.1109/ISBI.2016.7493261 .Guangyu Shen, Yi Ding, Tian Lan, Hao Chen, andZhiguang Qin. Brain tumor segmentation using concur-rent fully convolutional networks and conditional ran-dom fields. In
Proceedings of the 3rd International Con-ference on Multimedia and Image Processing, ICMIP2018, Guiyang, China, March 16-18, 2018 , pages 24–30. ACM, 2018. doi: 10.1145/3195588.3195590. URL https://doi.org/10.1145/3195588.3195590 .Haocheng Shen and Jianguo Zhang. Fully connectedcrf with data-driven prior for multi-class brain tu-mor segmentation. pages 1727–1731, 09 2017. doi:10.1109/ICIP.2017.8296577. URL https://doi.org/10.1109/ICIP.2017.8296577 .Pooneh R. Tabrizi, Awais Mansoor, Juan J. Cerrolaza,James Jago, and Marius George Linguraru. Automatickidney segmentation in 3d pediatric ultrasound imagesusing deep neural networks and weighted fuzzy ac-tive shape model. In , pages 1170–1173.IEEE, 2018. doi: 10.1109/ISBI.2018.8363779. URL https://doi.org/10.1109/ISBI.2018.8363779 .Alexander Tack, Anirban Mukhopadhyay, and Stefan Za-chow. Knee menisci segmentation using convolutionalneural networks: Data from the osteoarthritis initia-tive.
Osteoarthritis and Cartilage , 26, 03 2018. doi:19
PREPRINT - J
ANUARY
20, 202110.1016/j.joca.2018.02.907. URL https://doi.org/10.1016/j.joca.2018.02.907 .Saeid Asgari Taghanaki, Kumar Abhishek, Joseph PaulCohen, Julien Cohen-Adad, and Ghassan Hamarneh.Deep semantic segmentation of natural and medical im-ages: A review.
CoRR , abs/1910.07655, 2019. URL http://arxiv.org/abs/1910.07655 .Min Tang, Sepehr Valipour, Zichen Vincent Zhang, DanaCobzas, and Martin Jägersand. A deep level setmethod for image segmentation. In
Deep Learningin Medical Image Analysis and Multimodal Learningfor Clinical Decision Support - Third InternationalWorkshop, DLMIA 2017, and 7th International Work-shop, ML-CDS 2017, Held in Conjunction with MIC-CAI 2017, Québec City, QC, Canada, September 14,2017, Proceedings , pages 126–134, 2017. doi: 10.1007/978-3-319-67558-9\_15. URL https://doi.org/10.1007/978-3-319-67558-9_15 .Sofie Tilborghs, Tom Dresselaers, Piet Claus, Jan Bogaert,and Frederik Maes. Shape constrained CNN for car-diac MR segmentation with simultaneous prediction ofshape and pose parameters.
CoRR , abs/2010.08952,2020. URL https://arxiv.org/abs/2010.08952 .Christian Wachinger, Martin Reuter, and TassiloKlein. Deepnat: Deep convolutional neural net-work for segmenting neuroanatomy.
NeuroImage ,170:434–445, 2018. doi: 10.1016/j.neuroimage.2017.02.035. URL https://doi.org/10.1016/j.neuroimage.2017.02.035 .Andreas Wimmer, Grzegorz Soza, and Joachim Horneg-ger. A generic probabilistic active shape model fororgan segmentation. In
Medical Image Computing andComputer-Assisted Intervention - MICCAI 2009, 12thInternational Conference, London, UK, September 20-24, 2009, Proceedings, Part II , pages 26–33, 2009.doi: 10.1007/978-3-642-04271-3\_4. URL https://doi.org/10.1007/978-3-642-04271-3_4 .Kaijian Xia, Hongsheng Yin, and Yu-Dong Zhang. Deepsemantic segmentation of kidney and space-occupyinglesion area based on SCNN and resnet models combinedwith sift-flow algorithm.
J. Medical Systems , 43(1):2:1–2:12, 2019. doi: 10.1007/s10916-018-1116-1. URL https://doi.org/10.1007/s10916-018-1116-1 .Lipeng Xie, Yi Song, and Qiang Chen. Automaticleft ventricle segmentation in short-axis MRI usingdeep convolutional neural networks and central-lineguided level set approach.
Comput. Biol. Medicine ,122:103877, 2020. doi: 10.1016/j.compbiomed.2020.103877. URL https://doi.org/10.1016/j.compbiomed.2020.103877 .Fuyong Xing, Yuanpu Xie, and Lin Yang. An auto-matic learning-based framework for robust nucleus seg-mentation.
IEEE Trans. Med. Imaging , 35(2):550–566, 2016. doi: 10.1109/TMI.2015.2481436. URL https://doi.org/10.1109/TMI.2015.2481436 . Jun Xu, Lei Gong, Guanhao Wang, Cheng Lu, Han-nah Gilmore, Shaoting Zhang, and Anant Madabhushi.Convolutional neural network initialized active contourmodel with adaptive ellipse fitting for nuclear segmen-tation on breast histopathological images.
Journal ofMedical Imaging , 6:1, 02 2019. doi: 10.1117/1.JMI.6.1.017501. URL https://doi.org/10.1117/1.JMI.6.1.017501 .Xuanang Xu, Fugen Zhou, and Bo Liu. Automatic blad-der segmentation from CT images using deep CNNand 3d fully connected CRF-RNN.
Int. J. Comput.Assist. Radiol. Surg. , 13(7):967–975, 2018. doi: 10.1007/s11548-018-1733-7. URL https://doi.org/10.1007/s11548-018-1733-7 .Atsushi Yaguchi, Kota Aoyagi, Akiyuki Tanizawa, andYoshiharu Ohno. 3d fully convolutional network-basedsegmentation of lung nodules in CT images with aclinically inspired data synthesis method. In
Medi-cal Imaging 2019: Computer-Aided Diagnosis, SanDiego, California, United States, 16-21 February 2019 ,page 109503G, 2019. doi: 10.1117/12.2511438. URL https://doi.org/10.1117/12.2511438 .Yunyun Yang, Ruicheng Xie, Wenjing Jia, Zhaoyang Chen,Yunna Yang, Lipeng Xie, and BenXiang Jiang. Accu-rate and automatic tooth image segmentation modelwith deep convolutional neural networks and level setmethod.
Neurocomputing , 419:108 – 125, 2021. ISSN0925-2312. doi: https://doi.org/10.1016/j.neucom.2020.07.110. URL .Jiemin Zhai and Huiqi Li. An improved full convolu-tional network combined with conditional random fieldsfor brain MR image segmentation algorithm and its 3dvisualization analysis.
J. Medical Systems , 43(9):292:1–292:10, 2019. doi: 10.1007/s10916-019-1424-0. URL https://doi.org/10.1007/s10916-019-1424-0 .Hang Zhang, Jinwei Zhang, Rongguang Wang, Qi-hao Zhang, Susan A. Gauthier, Pascal Spincemaille,Thanh D. Nguyen, and Yi Wang. Geometric loss fordeep multiple sclerosis lesion segmentation.
CoRR ,abs/2009.13755, 2020a. URL https://arxiv.org/abs/2009.13755 .Huaqi Zhang, Guanglei Wang, Yan Li, and Hongrui Wang.Faster r-cnn, fourth-order partial differential equationand global-local active contour model (FPDE-GLACM)for plaque segmentation in IV-OCT image.
SignalImage Video Process. , 14(3):509–517, 2020b. doi:10.1007/s11760-019-01578-2. URL https://doi.org/10.1007/s11760-019-01578-2 .Mo Zhang, Bin Dong, and Quanzheng Li. Deep activecontour network for medical image segmentation. InAnne L. Martel, Purang Abolmaesumi, Danail Stoy-anov, Diana Mateus, Maria A. Zuluaga, S. Kevin Zhou,Daniel Racoceanu, and Leo Joskowicz, editors,
Med-ical Image Computing and Computer Assisted Inter-vention - MICCAI 2020 - 23rd International Confer-ence, Lima, Peru, October 4-8, 2020, Proceedings, Part PREPRINT - J
ANUARY
20, 2021 IV , volume 12264 of Lecture Notes in Computer Sci-ence , pages 321–331. Springer, 2020c. doi: 10.1007/978-3-030-59719-1\_32. URL https://doi.org/10.1007/978-3-030-59719-1_32 .Zhuangzhuang Zhang, Tianyu Zhao, Hiram Gay, WeixiongZhang, and Baozhou Sun. Arpm-net: A novel cnn-basedadversarial method with markov random field enhance-ment for prostate and organs at risk segmentation inpelvic CT images.
CoRR , abs/2008.04488, 2020d. URL https://arxiv.org/abs/2008.04488 .Lei Zhao, Tao Wan, Hongxiang Feng, and ZengchangQin. Improved nuclear segmentation on histopathol-ogy images using a combination of deep learning andactive contour model. In
Neural Information Process-ing - 25th International Conference, ICONIP 2018,Siem Reap, Cambodia, December 13-16, 2018, Pro-ceedings, Part VI , pages 307–317, 2018a. doi: 10.1007/978-3-030-04224-0\_26. URL https://doi.org/10.1007/978-3-030-04224-0_26 .Xiaomei Zhao, Yihong Wu, Guidong Song, Zhenye Li,Yong Fan, and Yazhuo Zhang. Brain tumor segmenta-tion using a fully convolutional neural network with con- ditional random fields. In
Brainlesion: Glioma, MultipleSclerosis, Stroke and Traumatic Brain Injuries - SecondInternational Workshop, BrainLes 2016, with the Chal-lenges on BRATS, ISLES and mTOP 2016, Held in Con-junction with MICCAI 2016, Athens, Greece, October17, 2016, Revised Selected Papers , pages 75–87, 2016.doi: 10.1007/978-3-319-55524-9\_8. URL https://doi.org/10.1007/978-3-319-55524-9_8 .Xiaomei Zhao, Yihong Wu, Guidong Song, Zhenye Li,Yazhuo Zhang, and Yong Fan. A deep learning modelintegrating fcnns and crfs for brain tumor segmen-tation.
Medical Image Analysis , 43:98–111, 2018b.doi: 10.1016/j.media.2017.10.002. URL https://doi.org/10.1016/j.media.2017.10.002 .Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet, Zhizhong Su, Dalong Du,Chang Huang, and Philip H. S. Torr. Conditional ran-dom fields as recurrent neural networks. In , pages1529–1537, 2015. doi: 10.1109/ICCV.2015.179. URL https://doi.org/10.1109/ICCV.2015.179https://doi.org/10.1109/ICCV.2015.179