A Survey and Analysis on Automated Glioma Brain Tumor Segmentation and Overall Patient Survival Prediction
AArchives of Computational Methods in Engineering manuscript No. (will be inserted by the editor)
A Survey and Analysis on Automated Glioma Brain TumorSegmentation and Overall Patient Survival Prediction
Rupal R. Agravat · Mehul S. Raval
Received: date / Accepted: date
Abstract
Glioma is the most deadly brain tumor withhigh mortality. Treatment planning by human expertsdepends on the proper diagnosis of physical symptomsalong with Magnetic Resonance(MR) image analysis.Highly variability of a brain tumor in terms of size,shape, location, and a high volume of MR images makesthe analysis time-consuming. Automatic segmentationmethods achieve a reduction in time with excellent re-producible results. The article aims to survey the ad-vancement of automated methods for Glioma brain tu-mor segmentation. It is also essential to make an objec-tive evaluation of various models based on the bench-mark. Therefore, the 2012 - 2019 BraTS challenges databaseevaluates state-of-the-art methods. The complexity oftasks under the challenge has grown from segmenta-tion (Task1) to overall survival prediction (Task 2) touncertainty prediction for classification (Task 3). Thepaper covers the complete gamut of brain tumor seg-mentation using handcrafted features to deep neuralnetwork models for Task 1. The aim is to showcase acomplete change of trends in automated brain tumormodels. The paper also covers end to end joint modelsinvolving brain tumor segmentation and overall survivalprediction. All the methods are probed, and parametersthat affect performance are tabulated and analyzed.
Rupal R. AgravatSchool of Engineering and Applied Science,Ahmedabad University,Ahmedabad, India.E-mail: [email protected] S. RavalSchool of Engineering and Applied Science,Ahmedabad University,Ahmedabad, India.E-mail: [email protected]
Keywords
Deep Learning · Medical Image Analysis · Brain Tumor Segmentation · Magnetic ResonanceImaging · Overall Survival Prediction
From the days the medical images are captured throughthe imaging devices and digitally preserved, researchershave started to build computerized automated and semi-automated analysis techniques to address a variety ofproblems like detection, segmentation, classification, andregistration. Till 1990s, medical image analysis was donewith pixel processing techniques to detect edge/line withfilters, regions based on the similarity of pixels or fitmathematical models to detect lines/elliptical shapes.Afterwards, shape models, atlas models and probabilis-tic models had become successful for medical imageanalysis, where from the observations(training data)the model learns to predict the unseen data. This trendhad moved towards the machine learning models wherefeatures are extracted from the data and fed into thecomputer to make it learn the underlying class/patternas per the input features and prediction of unseen datais made in the future. This approach has changed thetrend of human-dependent system to machine depen-dent system where the machine learns from the ex-ample data. Such algorithms works very well with thehigh dimensional feature space to find the optimal de-cision boundary. Here the only thing which is not doneby computer is the feature extraction, which leads tothe era of deep learning where the computer learn theoptimal set of features for the problem at hand. Thedeep learning models transform the input data fromimage/audio/video/text to output data which is loca-tion/presence/spread and incrementally learns high di- a r X i v : . [ ee ss . I V ] J a n Rupal R. Agravat, Mehul S. Raval mensional features with the help of set of intermediatelayers between input and output layers. Medical imageanalysis has reached the extent where such algorithmsplay a significant role in the early detection of the dis-ease based on the initial symptoms leading to bettertreatments. The deadly disease like cancer(Glioma-thecancerous brain tumor), if detected in the early stage,can increase the life expectancy.1.1 Brain Tumors and Their TypesWhen the natural cycle of the tissues in the brain breaksand growth becomes uncontrollable, it results in a braintumor[56]. The brain tumor is of two types, (1) primaryand (2) secondary. The primary tumor starts with thebrain tissues and grows within the brain, whereas sec-ondary tumor spreads in the brain and from the othercancerous organs[24]. More than 100 types of brain tu-mors are named based on the tissue and the brain partwhere it starts to grow. Out of all these tumors, Gliomais the most life-threatening brain tumor. It occurs inthe glial cells of the brain. The severity grades of theGlioma tumors depends on[115]. – the tumorous cell growth rate. – blood supply to the tumorous cells. – presence of necrosis (dead cells at the center of thetumor). – location of the tumor within the brain. – confined area of the tumor. – its structural similarity with the healthy cells.Grade I and II, are known as Low-Grade Glioma(LGG), which are benign tumors. Grade III and IV areknown as High-Grade Glioma (HGG), which are ma-lignant tumors. When symptoms like nausea, vomiting,fatigue, loss of sensation or movement, difficulty in bal-ancing, persist for the longer duration, it is advisable togo through the imaging screening to know the internalstructural changes in the brain due to tumor.1.2 Brain Imaging ModalitiesVarious imaging techniques are used for brain tumorscreening which includes positron emission tomogra-phy(PET), Computed Tomography(CT) and MagneticResonance Imaging(MRI). In PET, the radioactive traceris injected in the body, which captures the high level ofchemical activities of the disease infected body part.In CT, the X-ray tube rotates around the patient’sbody and emits a narrow X-ray beam, which is passedthrough the patient body to generate cross-sectionalimages of the brain. MRI uses a strong magnetic field around the patient’s body, which aligns the protons inthe body to the generated magnetic field, which is fol-lowed by the passing of radiofrequency signals to thebody. When the current is off, the protons emit theenergy and tries to align with the magnetic field. Theemitted energy form an image which records the re-sponse of various tissues of the brain. There are twotypes of MRI: 1)fMRI(Functional MRI): It measuresthe brain activities from the change in the blood flow,2) sMRI(Structural MRI): It captures the anatomy andpathology of the brain. The proposed article uses thesMRI as the focus to deal with the pathology in thebrain. Various modalities of sMRI captures responsesof the tissues which leads to distinct biological infor-mation in the images. Various modalities of sMRI are: – Diffusion Weighted Image(DWI) MRI: MRimaging technique measuring the diffusion of watermolecules within tissue voxels. DWI is often used tovisualize hyperintensities. – FLAIR MRI: an MRI pulse sequence which sup-presses the fluid (mainly cerebrospinal fluid (CSF))and enhances edema. – T1w MRI: basic MRI pulse sequence that captureslongitudinal relaxation time (time constant requiredfor excited protons to return to equilibrium) differ-ences of tissues. – T1Gd MRI: a contrast enhancing agent, Gadolin-ium is injected into the body and after that T1 se-quence is acquired. This contrast enhancing agentshortens the T1 time which results in bright appear-ance of blood vessels and pathologies like tumor. – T2w MRI: basic MRI pulse sequence that capturestransverse relation time(T2) differences of tissues.In general for brain tumors, CT and MRI are thecommonly used techniques. Both the imaging techniquesare essential, and the differences between CT and MRIare listed in Table 1. Fig. 1 shows the imaging dif-ference between CT and MRI, along with its variousmodalities. Healthy brain tissues are of three classes:1) Gray Matter(GM), 2) White Matter (WM), and 3)Cerebrospinal Fluid(CSF). Such soft tissue detailing iscaptured clearly in MRI compared to CT.Further different MRI modalities capture responsesof various brain tissues differently. The tumor adds onemore tissue class in the brain. Fig. 2 shows the differ-ence in the appearance of tumor in CT and MRI.Table 2 shows the intensity variation of the tumor indifferent MRI modalities. The tumor class overlaps thenormal tissue intensities, e.g., in T1 MR images GM,CSF, and tumor appear to be dark, whereas, in T2 GM,CSF and tumor appear to be bright. It is desirable touse the combination of various modality MRI images for
Survey and Analysis on Automated Glioma Brain Tumor Segmentation and Overall Patient Survival Prediction 3
Table 1: Difference between CT and MRI[93].
Advantages and DisadvantagesCT MRI
Non-invasive. Non-invasive.Fast. Accurate.Cheap. Expensive.Less accurate. More accurate.Coarse tissue details. Fine tissue details.Single modality Various modalities to recordthe reaction of different tis-sues differently (T1, T2, T1c,T2c, FLAIR, DWI).Generates 3D images. Generates 3D images.Images can be in axial, coro-nal and sagittal views. Images can be in axial, coro-nal and sagittal views.
RisksCT MRI
Harms unborn babies. Reacts with the metals in thebody due to magnetic field.A small dosage of radiation. The loud noise of the ma-chine causes hearing issues.Reacts to the use of dyes. Increases body temperaturewhen exposed for a longerduration.Imaging difficulties in case ofclaustrophobia.(a) (b)(c) (d)
Fig. 1: Imaging modalities[55] a) CT b) T1 MRI c) T2MRI and d) DWI MRI. (a) (b)
Fig. 2: Appearance of Tumor in a) CT and b) MRI[66].the analysis purpose[4]. The rest of the article focuseson the methods based on MR images.Table 2: The appearance of Normal Brain Tissues andTumor in various MRI modalities.
GM WM CSF TumorT1
Dark White Dark Dark T2 Light Dark White Bright
T1c
Dark White Dark Bright
FLAIR
Light Dark Dark Bright
The availability of the benchmark dataset has boostedthe research in the area of computer-assisted analysisfor brain tumor segmentation. Various types of meth-ods for the segmentation task includes semi-automatedand automated methods. The semi-automated methodsrequire user input to initiate the process. Circular au-tomata and other random field methods require seedpoint, diameter, or rough boundary selection for fur-ther computation. Atlas-based methods try to fit thepathological image with a healthy image to locate theabnormal brain area. Pathological atlas creation is an-other approach to determine the abnormality in thebrain. Expectation maximization methods iterativelyrefine the categories of the brain voxels from the inputof Gaussian Mixture Models or atlas. The automatedmethods include machine learning methods, which usethe features of the image for voxel classification. Lateron, the deep learning methods, specifically ConvolutionNeural Network(CNN) based methods, have shown suc-cess in the field of semantic segmentation, and suchmethods are adopted widely for brain tumor segmenta-tion. The CNN methods with various architectures aswell as ensemble approaches have proven to be the bestmethods for the segmentation task. In addition to thesegmentation task, the survival prediction task predictsthe survival days of patients. The contribution of thepaper is as follows:
Rupal R. Agravat, Mehul S. Raval
1. It is the most exhaustive review covering brain tu-mor imaging modalities, challenges in medical im-age analysis, evaluation metrics, BraTS dataset evo-lution, Pre-processing and post processing methods,segmentation methods, proposed architecture, hard-ware and software for implementation, overall sur-vival predictions, and limitations.2. It exhaustively traces the development in brain tu-mor segmentation by covering the models based onhandcrafted features to deep neural networks. It helpsto understand state-of-the-art development more com-prehensively.3. A fair comparison among models is made by cov-ering BraTS benchmark dataset. The methods areclassified, their parameters tabulated and analyzedfor performance.4. The paper also covers a survey on end to end meth-ods for brain tumor segmentation and overall sur-vival prediction. It helps to understand the impactof segmentation on overall survival prediction.The flow of the article is as follows: section 2 coversthe challenges for computer-aided medical image anal-ysis. Section 3 covers the problem statement, dataset,and evaluation framework. In contrast, section 4 in-cludes segmentation methods using hand-crafted fea-tures with limitations, section 5 covers segmentationand OS prediction using CNN methods, section 6 coversthe limitations of tumor segmentation and OS predic-tion methods followed by a conclusion and discussionin section 7.
Volumetric brain MRI images are analyzed and inter-preted by human experts (neurologists, radiologists) tosegment various brain tissues as well as to locate thetumor. This analysis is time-consuming. Besides, thistype of segmentation is non-reproducible. Accuracy ofbrain tumor segmentation, which is desirable to planproper treatment like medication or surgery, highly de-pends on the human expert with utmost precision. Thecomputer-aided analysis helps a human expert to lo-cate the tumor in less time as well as it regeneratesthe analysis results. The intended analysis by comput-erized methods requires appropriate input with correctworking methods. Input to the method may face thefollowing challenges:1. Low signal to noise ratio (SNR) and artifacts inraw MRI data are mainly due to electronic interfer-ences in the receiver circuits, radiofrequency emis-sions due to thermal motion of the ions in the pa-tient body, coils, and electronic circuits in MRI scan- ners. This random fluctuation reduces the imagecontrasts due to signal-dependent data bias[49].2. Non-uniformity is an irrelevant additional intensityvariation throughout the MRI signal. Possible causesof non-uniformity are radio-frequency coils, acquisi-tion pulse sequence, the geometry and nature of thesample.3. The unwanted information acquired by the MR ma-chines along with the brain images like skull, fat,and skin.4. The intensity profile of MR images may vary due tothe variety of MRI machine configuration.5. Publicly available brain tumor images for computer-aided analysis are very less. The collection of MRimages from various hospitals has privacy or confi-dentiality related issues.6. The class imbalance problem is another major issuein medical image analysis. The images for the abnor-mal class might be challenging to find because theabnormal classes are rare compared to the normalclasses.
The focus on the methods to solve medical related is-sues has increased since the late 1990s, which is appar-ent by looking at the gradual increase in semi-automatedor automated methods for tumor segmentation, as shownin Fig. 3. By considering the same, the main focus ofthe article is on the Glioma brain tumor segmentation.An additional task is about the survival prediction tech-niques of the patients suffering from Glioma.Fig. 3: Papers on PubMed with keywords ‘Brain TumorSegmentation’.
Survey and Analysis on Automated Glioma Brain Tumor Segmentation and Overall Patient Survival Prediction 5 compares and evaluates the methods on the samemeasure. The BraTS challenge dataset[13],[11],[12] hasthe following characteristics: – It contains multi-parametric MRI pre and post -operative scans in T1, T1Gd, T2, and T2-FLAIRvolumes(The post operative scans were omitted since2014); – The dataset contains images with different clinicalprotocols(2D or 3D) and various scanners from mul-tiple institutions(1.5T or 3T); – The dataset set includes the images with prepro-cessing for their harmonization and standardizationwithout affecting the apparent image information; – It has co-registration to the same anatomical tem-plate, interpolation to a uniform isotropic resolution(1 mm ), and skull-stripping.Initially, the clinical images in the dataset were very lessand it was challenging to compare methods based on theresults on such a small number of images. The compari-son is possible with increase in number of sample imagesand accurate generation of the ground truth images.The ground truth is generated based on the evaluationby more than one expert to avoid inter-observer vari-ability. The growth of the dataset from its inception isas shown in Table 3.Four different types of intra-tumoral structures areuseful for ground truth generation: edema, enhancingcore, non-enhancing(solid) core, and necrotic(or fluid-filled) core as shown in Fig 4. An annotation protocolwas used by expert raters to annotate each case manu-ally. Then the segmentation results from all raters werefused to obtain a single unanimity segmentation foreach subject as the ground truth. The validation of thesegmentation methods is based: 1) Whole Tumor (WT):all intra-tumoral substructures, 2) Tumor Core(TC):enhancing tumor, necrosis, and non-enhancing tumorsubstructures and 3) Enhancing Tumor(ET): includesonly enhancing substructure. IPP - ipp.cbica.upenn.edu
Fig. 4: Intra-tumoral structures appearance on threeimaging modalities with manual annotations. (a) Top:whole tumor (yellow), Bottom:FLAIR, (b) Top: tumorcore (red), Bottom:T2, (c) Top: enhancing tumor struc-tures (light blue), surrounding the cystic/necrotic com-ponents of the core (green), bottom:T1c, (d) Fusion ofthe three labels [100].
On account of sufficient dataset availability, the addi-tional task of overall survival(OS) prediction is intro-duced in BraTS Challenge since 2017. This task focuseson the OS prediction of HGG patients. The dataset in-cludes age, survival days, and resection status: GrossTotal Resection (GTR) or Sub-Total Resection (STR)information for HGG patients in addition to the images.The task is to classify the patients in: long-survivors( > <
10 months). The detailed descriptionof OS prediction task is given in Table 4. In 2019,the additional task included the quantification of un-certainty prediction in segmentation. This task focuseson the uncertainty prediction in the context of gliomatumor segmentation.3.2 Evaluation Metrics for Brain TumourSegmentation and OS predictionThe standard evaluation framework for tumor segmen-tation and OS prediction includes the following metrics.1. Dice Similarity Coefficient (DSC) (or F1 measure):It is the overlap of two objects divided by the to-tal size of both the objects. True Positive(TP) isthe outcome where the model correctly predicts thepositive class. In contrast, False Positive(FP) is theoutcome where the model incorrectly predicts thenegative class to be positive. False Negative(FN) isthe outcome where the model incorrectly predictsthe positive class to be negative.
DSC = 2
T P T P + F P + F N (1)
Rupal R. Agravat, Mehul S. Raval
Table 3: Growth of the BraTS dataset[13],[11],[12].
Year Total Im-ages Training Im-ages ValidationImages Test Images Tasks Type of Data
Table 4: The distribution of BraTS dataset[13],[11],[12] features in survival classes.
Year <
10 months) Long survivors(between 10 and 15months) Long Survivors( >
15 months) count Age ( µ ± σ ) OS days( µ ± σ ) count Age ( µ ± σ ) OS days( µ ± σ ) count Age ( µ ± σ ) OS days( µ ± σ )2017 163 Age 65 65.44 ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ±
2. Jaccard Similarity Coefficient: It is known as theintersection over the union of two different sets.
Jaccard = T PT P + F P + F N (2)3. Sensitivity: It is a measure that correctly identifiestumorous voxels.
Sensitivity = T PT P + F N (3)4. Hausdorff distance: It measures how far two subsetsof a metric space are from each other. If x and y betwo non-empty subsets of a metric space (M,d), thentheir Hausdorff distance d H ( x, y ) can be defined by d H ( x, y ) = max { sup x ∈ X inf y ∈ Y d ( x, y ) , sup y ∈ Y inf x ∈ X d ( x, y ) } (4)Where sup represents the supremum and inf theinfimum.5. The OS prediction is measurable with accuracy, whichis defined to be the quality of being precise.As DSC is the most commonly used evaluation metric,this article compares all the methods based using DSCunless specified explicitly.3.3 Image Pre-processingMedical image preprocessing plays a significant role inthe appropriate input to the computer-assisted analy-sis techniques. The preprocessed images help to get theaccurate outputs as such images show proper voxel rela-tionships. As mentioned in section 3.1, the dataset hasimages with different clinical protocols and scanners;the variability is to standardize, and it is necessary toput all the scans on a single scale. In addition to the im-age registration, uniform isotropic resolution, and skull-stripping, following preprocessing further improve theimage input: – Bias Field Correction(BFC): Bias field is a multi-plicative field added in the image due to the mag-netic field and radio signal in the MR machine. Au-thors in [139] have suggested a bias field correctiontechnique. – Intensity Normalization(IN): Different modality im-ages have a separate intensity scale. They must mapto the same range. Standardization of all the scansconsiders zero mean and unit variance. – Histogram Matching(HM): Due to the different con-figuration of MR machines, the intensity profile ofthe acquired images may vary. Intensity profiles areto be brought to the same scale using the histogrammatching process. – Noise Removal(NR): Noise in MR image at the timeof acquisition is due to the radio signal, or the mag-netic field. Various noise filtering techniques are use-ful for the removal of noise.3.4 Image Post-processingThe segmentation output generated by computer-assistedmethods may contain false segmentation in the imagedue to improper or incorrect feature selection. The seg-mentation improves by applying post-processing tech-niques like:1. Connected Component Analysis (CCA): CCA groupsthe voxels based on the connectivity depending onthe similar voxel intensities values. The connectedcomponents which are very small are excluded fromthe result as such components are considered to befalse positives due to spurious segmentation results.2. Conditional Random Field(CRF): The classifier pre-dicts the voxel class based on the features relatedto that voxel, which does not depend on the neigh-bouring relationship of that voxel with other nearbyvoxels. CRF takes this relationship into considera-tion and builds a graphical model to implement thedependencies between the predictions.3. Morphological Operations: Such operations are ap-plied to adjust the voxel value based on the valueof the other voxels in its neighborhood according tothe size and shape.
The methods in this section are divided into two cate-gories: interactive and non-interactive. The interactivemethods require user input in form of tumor diam-eter, boundary or seed point selection. Random fieldbased methods belong to this category. Non-interactivemethods do not require user input. First group of thiscategory detects abnormality. In atlas based methods,the abnormal image is non-linearly mapped to the nor-mal/input specific atlas to identify the abnormality whichin general is the area of the image where the atlas map-ping fails. The other approach is Expectation Max-imisation(EM), where the normal and abnormal vox-els intensity distribution is learnt with Gaussian Mix-ture Models(GMM) or probabilistic atlases. The second
Rupal R. Agravat, Mehul S. Raval group of this category is machine learning approaches.The clustering approach groups the voxels in numbersof clusters such that one of these clusters will result intumorous voxels group. In random forest(RF) and neu-ral network(NN) approaches, high dimensional featuresof the images are given for training. The trained modellater classifies the unseen voxels. The detailed descrip-tion of all these approaches is covered in the followingsub-sections.4.1 Random Field Based MethodsAuthors in [52] took user input for the largest possibletumor diameter from HGG images to find Volume of In-terest(VoI) for tumor and background from T1C MRIimages, followed by Cellular Automata(CA) to obtainthe probability maps for both the regions. The level setsurface was further applied to those probability mapsto get the final probability maps. They had further ex-tended their approach in [51], which considered mul-timodal images (T1C and FLAIR images) to segmenttumor, edema as well as LGG images as well.The semi-automatic method in [50] took the roughtumor region boundary as user input and fine-tuned itwith the global and local active contour-based model.The tumor region breaks into sub-regions with adap-tive thresholding based on a statistical analysis of theintensities of various tumor regions to separate edemafrom active tumor core. The process repeats for all theslices of a patient. In [35], the main contribution wasto incorporate soft model assignments into the calcula-tions of model-free affinities, which were then integratedwith model-aware affinities for multilevel segmentationby weighted aggregation algorithm.In [145], Random Walk(RW) based interactive aswell as an iterative method was applied to fine-tune thetumor boundary. RW was applied as an edge-weightedgraph in the discrete feature space based on the vari-ation of the distribution density of the voxels in thefeature space. The user made an initial tumor seed se-lection for tumors as well as edema. Afterward, RW wasapplied to feature space as well as on the image. If theuser did not approve the results, then the segmentationprocess is reinitiated.In [42], Hidden Markov Random Field (HMRF) basedmodel was used with modified Pott’s Model to panel-ized the neighboring pixels belonging to different classes.In [137], various modality intensity images along withits neighborhood voxel intensities fed into the map-reduce Hidden Markov Model (HMM), and the modelwas corrected iteratively based on the class labels.In [131], Gabor filter bank based Bayesian classifica-tion follows by MRF classification. Initially, each voxel divides into its constituent class by applying the Gaborfilter bank to the input vector (made up of intensitiesof four modalities at a voxel) to classify the voxel in fivedifferent classes (GM, WM, CSF, tumor, and edema).Next, MRF based classifier applies to the tumor as wellas edema classes. It uses voxel intensity and spatial in-tensity differences over the neighbor voxels. The [38],uses the Non-Negative Matrix to find voxel clusters,which shows the tumor and level set methods to fine-tune the region boundary.4.2 Atlas Based MethodsAuthors in [112], initially identified the abnormal braintissues by registering the tumorous brain image withhealthy tissues brain atlas. This step was followed byidentifying the presence of edema using T2 images, andfinally, geometric and spatially constraints were appliedto detect the tumor and edema regions. Authors in[36] applied an affine transformation to the atlas im-age to globally match the patient. The lesion was seg-mented using the Adaptive Template Moderated Spa-tially Varying statistical Classification (ATM SVC) al-gorithm. This atlas was then manually seeded by an ex-pert with a single voxel placed on the estimated originof the patient’s lesion, which was followed by the non-linear demons registration algorithm with the model oflesion growth to deform the seeded atlas to match thepatient. Four volumes of the contrast-enhanced agentwith meningioma implement the model.The paper [78] applied the semi-automatic method,which required user input to give the seed point fortumor, radius for each tumor, and seed point for eachregular tissue class. The random walk generates tumorpriors using initial tumor seeds. The patient-specific at-las was modified to accommodate tumor classes, usinga tumor growth model. Empirical Bayes model usedthe EM framework to update the posterior of tumor,growth model parameters as well as a patient-specificatlas. This work was then extended in [10], where in-stead of single seed point for various labels, multipleseed points were taken into consideration to find theintensity mean and variance of a specific label. Thiswork focused on preoperative MRI scans. Whereas, itwas further extended in [148] to include post-operativescans along with additional features to the GMM. Herethe need for manual selection of seed point was alsoomitted.
Survey and Analysis on Automated Glioma Brain Tumor Segmentation and Overall Patient Survival Prediction 9 k = 5. These five nearestneighbors generate the test patch label.The [32], uses the knowledge-based multispectralanalysis on the top of the unsupervised clustering tech-nique. The rule-based expert system then extracted theintracranial region. The training was performed on threeT1, Proton Density(PD), and T2 images, whereas thetest on thirteen such volumes. The [122], uses the rough-set approximation to fine-tune the prediction done byk-means clustering. In [116], initially, the enhanced probabilistic fuzzyC-means clustering was applied to get the rough esti-mation of the tumor region. This estimation and clus-ter centroid is given to the gradient-vector-flow snakemodel to refine the tumor boundary. The [126], learnsa Sparse dictionary of size 4x4 for different tissue typesusing four image modalities followed by logistic regres-sion for the tissue classification. This initial stage clas-sifies the image voxels in various classes. This step wasfollowed by k-means clustering, which uses a very highdimensional feature vector as input to find the classifi-cation of the tumor as well as the edema region as anoverlap of the output of the previous step.4.5 Random Forest Based methodsThe [95] implements a generative- discriminative model.The generative model estimates tissue probability us-ing density forest (similar to GMM). The discriminativemodel implements classification forest, which took as aninput 51 features (gradient information first-order tex-ture features and symmetry-based features, prior tissueprobabilities based on the density forest) to generatethe probability of the tissue. This probability was thensupplied to the CRF to fine-tune the result.Authors in [155] use context-aware spatial features,along with the tissue appearance probability generatedby Gaussian Mixture Model, to train decision forest.The [17], works on Random forest classification withCRF regularization to predict the probability of tis-sue in the multiple class, i.e., GM, WM, CSF, Edema,Necrotic core and enhancing tumor. 28-dimensional fea-ture vector (includes the intensity of each modality alongwith first-order statistics like mean, variance, skewness,kurtosis, energy, and entropy computed from local patchesaround each voxel in each modality.In [47], each voxel was characterized by signal modal-ity as well as spatial prior. The averaging across a 3x3x3cube removes the noise. The random forest trains usinglocal features (intensity or priors), as well as context-specific features (region-based features or symmetry-based features). The forest was made up of 30 treeswith a depth of 20 for each tree and trained on thesynthetic data.The [156], uses a discriminative multiclass classifica-tion forest. It uses spatially non-local context-sensitivehigh dimensional features along with the prior proba-bility for the tissue to be classified. Prior probabilityis available to the GMM. It classifies in three classes,i.e., background, edema, and tumor. Three types of fea-tures train 40 tree with a depth of 20 each, and suchfeatures were intensity difference, intensity mean dif-ference, and intensity range on a 3D line to check for structural changes. 2000 combination was selected todesign the decision trees. The [46], trains the randomforest of 50 trees each of depth 25 on 120000 sampleswith 324 features comprises of intensity, neighborhoodinformation, context information, and texture informa-tion.In [119], Random Forest classifier used intensities,the difference of intensities (T1 - T2, T1- Flair, T1– T1c), and the texture-based features such as fractaland texton to train RF using three-fold cross-validation.They further extended their work in [120] for the post-processing, where the connected component analysis re-moves tiny regions in the 3D volume, and holes in be-tween the according to the neighboring region.In [48], Extremely Randomized Trees(ExtraTrees)were used, which introduces more randomness at thetime of training. The classifier was trained on 208 fea-tures extracted from all the four modalities, which in-clude intensity values, local histograms, first-order statis-tics, second-order statistics, and basic histogram-basedsegmentation. In the paper, the ExtraTrees trains withthe best threshold rather than the threshold derivedfrom the individual features. In [75], pixel classificationwas done with ten random forests with ten trees each,which are trained in parallel to reduce the training timeand finally merged into a single forest with Gini im-purity. One thousand samples for tumorous class and1000 samples for the non-tumorous class trains the RF.Classification forest in [96] used 237 features which in-cluded appearance specific features (image intensities,first-order texture features, and gradient features), con-text sensitive features (ray feature, symmetry intensitydifference features). Authors in [88], [97], [82], [98], [43],[129], [80], [109] had used random forest classifier withcombination of various intensity based features, gradi-ent based features, texture based features, rotation in-variant features.In [19], tensor features were extracted along withmean, entropy and standard deviation features. Au-thors in [123] extracted features for supervoxels frommulti scale images and created sparse feature vectors tosegment whole tumor. Sub-regions of tumor were thenseparated using CRF.4.6 Neural Network Based methodsIn [20], Grouping Artificial Immune Network(GAIN)takes as an input voxel intensity from 2D as well as3D slices, statistical features and texture features fortraining as well as segmentation of brain MRI images.Information is in the form of bits from various imagemodalities. In [2], Convolutional Restricted BoltzmannMachine trains GMM and spatial tissue priors. Table 5 summarizes the methods for the type ofpre-preprocessing, dataset, the number of images usedas well as the DSC achieved. The DSC is shown forthe various tumor sub-regions which includes WT, ET,TC and Edema(ED) for training, validation and testdatasets.The DSC comparison of different methods for tumorsegmentation is shown in Fig. 5. The atlas and RF-based methods performed well compared to all otherapproaches.Random field methods, atlas-based methods, expec-tation maximization methods, and clustering methodsdo not use any post-processing techniques. Random for-est based methods, authors in [46], [120] and [75] usedconnected component analysis, [96], [97], [129] appliedspatial regularization and authors in [88] and [43] ap-plied morphological operations to refine the segmenta-tion output.Table 5: Summarization of segmentation methods usinghandcrafted features. Survey and Analysis on Automated Glioma Brain Tumor Segmentation and Overall Patient Survival Prediction 11
Ref. Pre-processing Dataset
Random Field methods[52] - Custom 29 TrainingTC:0.89TestTC:0.80[51] - BraTS2012 30 TC:0.69,ED:0.37[50] - BraTS2013 30 TC:0.82[35] IN Custom 20 TrainingTC:0.70,ED:0.66TestTC:0.66,ED:0.61[145] NR BraTS2012 30 TC:0.53,ED:0.25[42] - BraTS2013 30 HGGWT:0.84,TC:0.54,ET:0.67LGGWT:0.81,TC:0.54,ET:0.11[131] - BraTS2012 28 TC:0.66,ED:0.56[137] BFC BraTS2013 30 HGTC:0.62,ED:0.59Atlas Based Methods[78] BFC, IN BraTS2014 200 ValidationWT:0.86,TC:0.79,ET:0.59TestWT:0.88,TC:0.83,ET:0.72[10] BFC, IN BraTS2015 186 TrainingWT:0.88,TC:0.77,ET:0.68[148] NR, HM BraTS2016 200 WT:0.89,TC:0.77,ET:0.67Expectation Maximisation Based Methods[99] - BraTS2012 30 HGGTC:0.55,ED:0.57LGGTC:0.24,ED:0.42[118] - BraTS2012 30 HGGTC:0.58,ED:0.60LGGTC:0.32,ED:0.36[149] - BraTS2012 30 TC:0.31,ED:0.35[138] - BraTS2012 30 TC:0.43,ED:0.55
Table5 – continued . . .Ref. Pre-processing Dataset [150] IN BraTS2013 30 HGGWT:0.83,TC:0.74,ET:0.68LGGWT:0.83,TC:0.58,ET:0.51[110] - BraTS2015 200 TrainingWT:0.74,TC:0.55,ET:0.54Clustering Based Methods[34] - BraTS2013 30 HGGWT:0.79,TC:0.60,ET:0.59LGGWT:0.76,TC:0.64,ET:0.44[122] - BraTS2013BraTS2015 200 WT:0.82,TC:0.71,ET:0.72[116] - Custom 15 TC:0.82[126] - BraTS2012 30 TC:0.30,ED:0.39Random Forest Based Methods[95] - BraTS2013 30 HGGWT:0.80,TC:0.69,ET:0.69LGGWT:0.76,TC:0.58,ET:0.20[155] - Custom 40 TC:0.85,NC:0.75,ET:0.80[17] BFC, IN,HM BraTS2012 30 HGG-realTC:0.62,ED:0.61LGG-realTC:0.49,ED:0.35[47] HM BraTS2012 30 HGG-realTC:0.68,ED:0.56LGG-realTC:0.52,ED:0.29[156] BFC BraTS2012 30 HGG-realTC:0.71,ED:0.70LGG-realTC:0.62,ED:0.44[46] BFC, HM BraTS2013 30 HGGWT:0.83,TC:0.70,ET:0.75LGGWT:0.72,TC:0.47,ET:0.212 Rupal R. Agravat, Mehul S. Raval
Table5 – continued . . .Ref. Pre-processing Dataset [119] BFC, HM BraTS2013 30 HGGWT:0.92,TC:0.91,ET:0.88LGGWT:0.92,TC:0.91,ET:0.88[120] - BraTS2014 200 TrainingWT:0.81,TC:0.66,ET:0.71[48] BFC, HM BraTS2013 208 WT:0.83,TC:0.71,ET:0.68[75] HM, IN BraTS2014 200 TrainingWT:0.84,TC:0.68,ET:0.72Valid/TestWT:0.87,TC:0.76,ET:0.64[96] NR, IN,BFC BraTS2013 30 TrainingWT:0.83,TC:0.66,ET:0.58ChallengeWT:0.84,TC:0.73,ET:0.68[88] BFC, IN BraTS2015 252 WT:0.75,TC:0.60,ET:0.56[97] - BraTS2015 65 WT:0.83,TC:0.69,ET:0.63[82] BFC, NR,IN BraTS2013 30 WT:0.87,TC:0.88[98] - BraTS2013 30 ValidWT:0.79,TC:0.75,ET:0.66ChallengeWT:0.83,TC:0.76,ET:0.71[43] BFC, HM BraTS2016 200 WT:0.80,TC:0.72,ET:0.73[129] IN BraTS2015 274 TrainingWT:0.87,TC:0.72,ET:0.75[80] IN BraTS2015 274 TrainingWT:0.84,TC:0.72,ET:0.71[109] BFC BraTS2017 285 TrainingWT:0.64,TC:0.49,ET:0.47TestWT:0.63,TC:0.41,ET:0.42[19] IN, HM BraTS2017 285 ValidationWT:0.79,TC:0.67,ET:0.61TestWT:0.77,TC:0.61,ET:0.50
Table5 – continued . . .Ref. Pre-processing Dataset [123] - BraTS2018 285 ValidationWT:0.80,TC:0.63,ET:0.57TestWT:0.73,TC:0.58,ET:0.50Neural Network Based Methods[20] - BraTS2013 30 WT:0.73,TC:0.59,ET:0.63[2] - BraTS2015 200 TestWT:0.81,TC:0.68,ET:0.65
The limitations associated with the methods work-ing on handcrafted features are as follows:1. Identifying tissue probability classes: tumor tissueintensities overlap with that of the healthy tissues,as mentioned in Table 1; in such a case identify-ing the probable class for tumorous tissue is quitechallenging.2. Atlas matching (healthy or tumorous atlas): Usu-ally, the brain atlas contains the normal brain tis-sue distribution map. Due to the deformation of thehealthy tissue by the tumor, the atlas matching ofa tumorous brain may result in the wrong map3. Manual seed point identification for the tumor or itssubparts: Almost all semi-automated methods re-quire some initial selection for the tumorous voxel,its diameter, or its rough outer boundary. The se-lection depends on the expert. Its repetition over allthe slices of the brain is a time-consuming task.4. Feature extraction from the images: RF training de-pends on the features extracted from the brain im-ages. All the MRI modalities contain different bio-logical information. This variation in the informa-tion complicates the task of feature extraction aswell as selection to training RF.5. Discontinuity: The results generated by such meth-ods are spurious, which increases the chances of falsepositives. The proper post-processing techniques arerequired to fine-tune the generated results.
Deep Neural Network(DNN) is an artificial intelligencefunction which mimics the human brain working fordata processing and pattern creation in decision mak-ing. There are mainly four reasons contributing to theirsuccess:
Survey and Analysis on Automated Glioma Brain Tumor Segmentation and Overall Patient Survival Prediction 13
Fig. 5: Comparison of segmentation methods using hand-crafted features (DSC for validation/Test for Tumor/WTof all images/HGG).1. The DNN models solved the problem in an end-to-end manner. The models learn the features from thedata automatically with help of various functions.The feature learning improves from simple featuresat initial layers to complex features at deeper layersof the model. The automatic feature learning haseliminated the need of domain expertise.2. The computational capabilities of the hardware interms of GPU and efficient implementation of themodel on GPU with various open source librarieshave made the training of the DNN 10 to 50 timesfaster than CPU.3. Efficient optimization techniques for robust learn-ing contribute to the success of DNN for optimalnetwork performance.4. The availability of benchmark datasets allow train-ing and testing of various deep learning models tobe implemented successfully.The exponential growth of the usage of DNN tech-niques to solve the variety of problems is shown in Fig.6. A similar growth pattern is identified for solving thebrain tumor segmentation problem, which is as shownin Fig. 7.The general block diagram of any deep learning tech-nique is as shown in Fig 8. The crucial task is to get thelabeled data set. After the availability of the dataset, itis divided into training and validation sets, followed byappropriate pre-processing techniques as per the taskon hand. Actual DNN applies to the training data,which makes the network learn the network parame-ters. The output of DNN is spurious at some brainarea, and post-processing fine-tunes the segmentationresult. And at last, the evaluation framework measuresthe performance of the network. Fig. 6: The growth rate of research papers with thekeyword ‘deep learning’ on PubMed.Fig. 7: The growth rate research papers with the key-words ‘deep learning’ and ‘brain tumor segmentation’on PubMed.5.1 Evolution of DNNIn medical image analysis, the semantic segmentationtask is common, e.g., segmentation of organ, lesion.The Convolutional Neural Network(CNN), specific type
Fig. 8: Generalize DNN network.of DNN architecture gained its popularity from 1990with the architecture of LeNet[81], which was two lay-ers of architecture. After the availability of fast GPUsand other computing facilities, over fifteen years later,AlexNet was proposed by authors in [77] with five con-volutional layers. The CNNs are designed with vari-ety of layers (like convolution layers, non-linearity lay-ers, pooling layers, fully-connected layers), regulariza-tion, optimization and normalization, loss functions, aswell as network parameters initializations. Authors in[18] have nicely explained the architectural elements ofCNN which are as follows:
Convolution layer: extracts representative features fromthe input. It achieves: 1)weight-sharing mechanism,2)exploits local connectivity of input, and 3) pro-vides shift invariance at some extent.
Non-linearity layer: provides sparse representation ofinput space which achieves data variability invari-ance and computationally efficient representation.Types of non-linearity layers are Rectified LinearUnit(ReLU), Leaky ReLU(LReLU), Parametric ReLU(PReLU), S-shaped ReLU(SReLU), Maxout and itsvariants, Exponential Linear Unit(ELU) and its vari-ants.
Pooling/subsampling layer: extracts prominent fea-tures from non-overlapping neighborhood. It is usedto 1) reduce no. of parameter, 2) reduce over-fitting,and 3) achieve translation invariance. Commonlyused pooling techniques are max pooling and av-erage pooling.
Fully connected layer: converts 2D features to 1Dfeature vector. It helps to predict input image classlabel.
Loss functions: improves learning process by improv-ing within class similarity and between class sepa-rability.
Regularizations: deals with over-fitting issues. Com-monly used regularization techniques are L1 and L2regularizations, dropout, early stopping, batch nor-malization.
Optimization: used for proper updates of network pa-rameters during backpropogation. Various techniquesof optimization includes Nesterov accelerated gradi-ent descent, adaptive gradient algorithm (Adagrad),Root Means Square Propogation (RMSProp)
Weight initialization and normalization: boosts thelearning process by helping the weight update withproper initial values.The convolution layers extract features from the in-put by applying kernels to it. The output feature mapdepends on the type of kernel and its size. At the initiallayers simple features are extracted from the input likeedge or lines. The gradual increase of the network depthrequires higher number of feature maps to extract com-plex shapes[5]. The activation function is applied to thefeature maps to learn the non-linear relationship withinthe data and allows the errors to backpropagate to theinitial layers for accurate parameter updates. Increasein the network depth exponentially increases the net-work parameters, which is highly computationally ex-pensive. To balance the growth of the network parame-ters, pooling layers are introduced to down sample theinput feature maps and reduce its spatial size by consid-ering only the prominent features. The fully connectedlayers at the end of the network flatten the result ofthe input layers before actual classification. The lossfunction at the classification layer calculates the errorin the prediction. Based on this error, the network pa-
Survey and Analysis on Automated Glioma Brain Tumor Segmentation and Overall Patient Survival Prediction 15 rameters are updated using the gradient descent meth-ods by backpropagation. Commonly used loss functionsare: – Cross Entropy loss function: J = − N (cid:32) (cid:88) voxels y true · log ˆ y pred (cid:33) (5) – Dice Loss function: J = 1 − (cid:80) voxels y true y pred + (cid:15) (cid:80) voxels y true + (cid:80) voxels y pred + (cid:15) (6)Here, N = N umber of voxels, y true = ground truth label, and y pred = network predicted label .The CNNs, with convolution layers followed by fully-connected layers, classifies entire image in a single cat-egory. GoogleNet(Inception)[134] and InceptionV3[135]networks have introduced the inception module whichimplements the kernels of different sizes to reduce net-work parameters. ResNet[136] has introduced the resid-ual connection between the convolution layers such thatit learns the identity function which allows effectivetraining of deeper networks. In DenseNet[61] layers arevery narrow and adds very less number of feature mapsto the network which again allows to design deeper ar-chitectures and training is efficient as each layer hasdirect access to the gradient of the loss function.For semantic segmentation, the CNN can simply beused to classify each voxel of the image individually bypresenting it with several patches extracted around theparticular voxel. Each voxel of the image is classifiedwith the same process, resulting in the segmentationof the entire image. This ‘sliding-window’ approach re-peats the convolution operations for adjacent patches ofneighboring voxels. The improvement to this approachis the replacement of fully-connected layers with con-volution layers, which generates the probability map ofthe entire input image rather than generating outputfor a single voxel. Such networks are known to be FullyConvolutional Neural Networks(FCNN). FCN[85] is atype of FCNN where skip connections are introducedto reconstruct the high resolution image. U-Net[121],a very known highly adapted network architecture fortumor segmentation has taken the encoder - decoderapproach where every encoding layer is connected withits peer decoding layer with skip connection to recon-struct the dimension as well to get the particular spa-tial information from the encoding layer. SegNet[8] andDeepLab[28] are the other type of FCNN architectureadopted to solve the problem of brain tumor segmenta-tion. 5.2 Handling Class Imbalance ProblemIn medical image analysis, finding the number of abnor-mal images as compared to the normal images is diffi-cult as abnormality like the tumor is rare. This prob-lem is called ‘class imbalance’. All images in this articleare of the tumor; the class imbalance issue persists be-cause, in the single brain volume, the proportion of thetumor is less compared to average brain volume. Evenon a single brain slice, the area of the tumor is smallcompared to the brain part, which leads to a class im-balance problem. The proportion of the brain volume(BV) and back ground volume (BGV) with respect totumor volume for BraTS 2019 dataset is as shown inTable 6.Table 6: The proportion of non tumor brain volume(NTBV) and non tumor background volume (NTBGV)vs. tumorous volume(TV). %NTBV % Necrosis % Edema % ETBV 75.75 0.7 17.78 5.77BGV 99.11 0.03 0.65 0.21 The following approaches address data imbalanceproblem. – Patch sampling: The patch sampling-based methodcan mitigate the imbalanced data problem. The sam-pling process includes the equiprobable patches fromall the tumorous regions as well as the non-tumorousregion. – Improvement in loss functions: Some of the loss func-tions, when used in its raw form, may not suit thetumor segmentation task, as they consider balanceddatasets. These functions adopt imbalanced datasetwith modifications as:1. The weighted cross-entropy loss function: Thevoxel-wise class prediction averaged for all vox-els may lead to error if the class is imbalancedin the image. The issue is in a weighted cross-entropy function where the weight is associatedwith all the voxels prediction. Since the back-ground regions dominate the training set, it isreasonable to incorporate the weights of multi-ple classes into the cross-entropy such that moreweight given to the voxels of the positive class.
Loss
W CE = (cid:88) voxels (cid:88) classes y true · log ˆ y pred (7)2. Generalized Dice Loss Function: Authors in [132]proposed to use the class rebalancing properties of the generalized dice. It provides a robust andaccurate deep-learning loss function for unbal-anced tasks. Loss
GDL = 1 − (cid:80) classes w (cid:80) voxels y true y pred (cid:80) classes w (cid:80) voxels y true + y pred (8)3. Focal Loss Function: The detection task uses fo-cal loss. It encourages the model to down-weighteasy examples and focuses training on hard neg-atives. Formally, the focal loss defines a modu-lating factor to the cross-entropy loss and a pa-rameter for class balancing [84]. Loss
F L ( p t ) = − α t (1 − p t ) γ log ( p t ) p t = (cid:40) p t if y i = 11 − p t otherwise (9)where y(cid:15) { , − } is the ground-truth class, and p t (cid:15) { , − } is the estimated probability for theclass with label y = 1. The weighting parameter α deals with imbalanced dataset. The focusingparameter γ smoothly adjusts the rate at whicheasy examples are down-weighted. Setting γ > γ = 0. – Augmentation techniques: Most of the time, a largenumber of labels for training are not available forseveral reasons. Labeling the dataset requires anexpert in this field, which is expensive and time-consuming. Training large neural networks from lim-ited training data causes the over-fitting problem.Data augmentation is a way to reduce over-fittingand increase the amount of training data. It cre-ates new images by transforming (rotated, trans-lated, scaled, flipped, distorted, and adding somenoise such as Gaussian noise) the ones in the train-ing dataset. Both the original image and createdimages are input to the neural network. For exam-ple, a large variety of data augmentation techniquesinclude random rotations, random scaling, randomelastic deformations, gamma correction augmenta-tion, and mirroring on the fly during training.5.3 CNN Methods Classification for TumorSegmentationThe classification of CNNs for tumor segmentation usesthe combination of the design aspects as shown in theFig. 9.
Input type:
The network may take 2D/3D input inform of patches or images. The CNN with fully-connected layers classify the centre voxel of the patchwhereas FCNN predicts multiple or all voxels ofthe patch/image. The network may take multi-scalepatches to extract coarse and fine details of the in-put.
Output Type:
The output of the network depends onthe problem to solve. It predicts single output forthe classification problem and multiple voxel outputfor the semantic segmentation problem.
Type of network:
The CNN approach indicates theconvolution network with fully-connected layers atthe end whereas FCN indicates the network with allconvolution layers.
Ensemble Approach:
Ensemble approach can be clas-sified into serial and parallel approaches. In the se-rial approach multiple networks combine in series tofine tune the end output. The input of one networkdepends on the output of the other. In the paral-lel approach multiple networks work in parallel andtake the same/different input to gather the compre-hensive details from the input. The final output ofthe network is decided based on the majority votingor averaging of all the network output.The evolution of CNN based methods for the tumorsegmentation is as shown in Fig. 10. Some of well-knownCNN architectures for brain tumor segmentation arerepresented in Fig. 11, Fig. 12, Fig. 13 and Fig. 14.The architecture of [54] is a two pathway CNN whichtakes 2D multi-resolution input patches, applies theconvolution operations and concatenate the output ofboth the pathways. The deepmedic[70] also follows twopathways with 3D multi-resolution input patches andincorporates the residual connections and predicts theoutput for multiple voxels at a time.U-net[121] is an encoder-decoder architecture withskip-connections between the peer layers of analysis andsynthesis path. This architecture has gained most pop-ularity. Anisotropic[142] architecture followed the serialensemble approach. The first network segments wholetumor, second segments tumor core and considers theoutput of first network and finally third network seg-ments enhancing tumor with help of second networkoutput.Initially, shallow CNN performs the voxel-based im-age segmentation. The authors in [54] proposed voxel-wise classification using CNN multipath way architec-ture. One pathway uses 2D patches of size 32x32, andthe other uses fully-connected input of 5x5 patch sizehaving center pixel same as 32x32 patch. Patch selec-tion was made such that the labels are equiprobable.L1 and L2 regularizations overcome overfitting. In [140],
Survey and Analysis on Automated Glioma Brain Tumor Segmentation and Overall Patient Survival Prediction 17
Fig. 9: Design aspects of CNN architectures.Fig. 10: Variational growth of CNN for tumor segmentation.Fig. 11: TwopathwayCascadedCNN Architecture[54].Fig. 12: Deep Medic Architecture [70].
Fig. 13: U-net architecture[121].Fig. 14: Anisotropic Cascaded Architecture[142].voxel-wise class probability prediction uses separate 3DCNNs for HGG and LGG images. The final probabil-ity classified the voxel into the six classes. In [157], fivelayers deep 2D CNN architecture performs voxel-wiseclassification.Gradually the depth of CNN had increased to ac-commodate more layers in the network. In [108], 2Ddeep CNN with fully connected output layers sepa-rates HGG and LGG. This approach was further ex-tended by [117] with the two-phase process along with aweighted loss function-initially, the network trains usingequiprobable patches that follow actual patch trainingwithout the class imbalance problem. In [53], authorshad designed 2D InputCascadedCNN, which took theoutput of TwoPathCNN to train other 2DCNN withthe input images. After the successful implementationof FCN by the authors in [85], authors in [70] pro-posed two pathway architecture, where both pathwaysincluded residual connections and trained on the differ-ent input patch sizes. As the network was fully convo-lutional, multiple voxels of the input patch label at atime.The [151], uses the 2D FCNN approach along withCRF. The FCNN trains on patches and CRF on slices. In [26], cascaded encoder-decoder like FCNNs alongwith residual connections. The first FCNN segmentsthe whole tumor, followed by the internal tumor regionsby the second FCNN. Authors in [90] had proposedencoder-decoder FCNN based architecture to segmentvarious tumor sub-regions. In [21], authors had pro-posed three different FCNN architecture and showedthat the architectures with multi-resolution features per-formed better compared to single-resolution architec-ture. Authors in [86] implemented Dilated Residual Net-work for patch based training where equiprobable patcheswere supplied to the network for training.Authors in [41] adopted U-net architecture for braintumor segmentation. Authors in [44] modified the U-net, which took 3D input, and the depth of the net-work reduces to three. Authors in [63], had optimizedthe training of the network proposed in their previouswork [62]. In [27], authors had proposed novel encoder-decoder architecture that had worked well on multiplebiomedical image segmentation problems. Various otherCNN based approaches where single CNN was used forsegmentation task were [107], [64], [124], [111], [23], [74],[22], [67], [58], [89], [102], [94].
Survey and Analysis on Automated Glioma Brain Tumor Segmentation and Overall Patient Survival Prediction 19
The ensemble of CNNs performs better compared tosingle CNN, as in [71], where authors had implementedan ensemble of seven networks using DeepMedic, FCN,and U-net along with variations of those three net-works. They also tried different three approaches forpre-processing on all these networks. The output of in-dividual networks with all the pre-processing generatesthe final label. Authors in [91] had extended their workproposed in [90] where the dense module and dilatedmodule were introduced in the encoder decoder cas-caded architecture of two networks. The pooling layerswere replaced with dilated convolution layers.The authors in [101] had implemented the ensembleof ten encoder-decoder based architectures, which in-cluded the auto-encoder stream to reconstruct the orig-inal image for additional information and regularizationpurpose. Authors in [92] extended their approach pro-posed in [69]. They used a combination of U-net andDensenet with U-net like architecture containing denseblocks of dilated convolution. The network of [151] wasextended in [152] to create an ensemble of three net-works which were trained on three image views. Twonetwork cascaded path was used in [33], where CoarseSegmentation Network segmented WT and Fine Seg-mentation Network segmented the sub-regions of thenetwork. Both the networks used 3D U-net with fourlevel deep architecture.Authors in [142] used three networks (WNet, TNet,and ENet) to prepare the cascaded path. The multi-scale prediction averages for the private network. Thesenetworks train on three-views of images (axial, coro-nal, sagittal), and results from averages to generate thefinal segmentation output. In [153], the cascaded net-work was proposed, which initially segmented the wholetumor followed by tumor core, and enhancing tumorsegmentation refinement. In [79] authors had proposedcascaded UNet with three networks. The network pro-cessed downsampled input and generated output waspassed to the next network in the cascaded sequence.Authors in [146] proposed multi-scale mask 3D U-netswith atrous spatial pyramid pooling layers, where WTsegmentation generated by first networks was passed tothe second for TC generation, which in sequence passedto the final network to generate ET output. Other en-semble based CNN approaches were explained in [87],[73], [60], [143], [7], [31], [25], [147].The comparison of the methods summarized in Ta-ble 8 is based on the pre-processing techniques, DNNarchitecture, activation function, loss function, post-processing, and DSC achieved. The pre-precessing tech-niques considered for the comparison are:1. Intensity Clipping: 1 % of highest and lowest fre-quencies are clipped. 2. Bias Field Correction.3. Z-score normalization: Z = ( x − µ ) /σ .4. Histogram matching: Histogram of all the images ismatch with the reference histogram.5. Image normalization: Min-max normalization.6. Intensity standardization with Nyul approach [104].7. Image denoising: applies noise filtering for e.g. Gaus-sian noise filtering.8. Intensity rescaling : rescaling the intensity range be-tween some specific limits.The post-processing techniques used for segmentationresult improvement are:1. Connected Component Analysis: Analyse the con-nected components and removes the component withthe volume below some threshold.2. Conditional Random Field.3. Morphological Operators to remove false positivesand fill the holes4. relabelling the output label: Enhancing tumor labelsbelow some threshold are relabelled as necrosis.Fig. 15 shows the pictorial representation of theDSC of various CNN methods for whole tumor segmen-tation. The average DSC of CNN is 0.86, deep CNN andFCN is 0.87, and for ensemble approach is 0.89. The en-semble of the CNNs/FCNNs learns the robust featuresfrom the input.5.4 Hardware and Software for DNN Hardware : DNN methods have gained its popular-ity after availability of the Graphical Processing Units(GPU). Nowadays efficient parallel processing for ma-nipulation of large amount of data is applicable withhelp of General Purpose GPU(GPGPU). The comput-ing libraries like CUDA and OpenGL allows the efficientimplementation of the processing code on GPU. Theperformance of the GPU highly depends on GPU com-puting cores (CUDA cores), Tensor processing cores,Thermal Design Power (TDP), and on board GPU mem-ory. Various types of GPUs used for the implementationof the CNN methods for segmentation task are as shownin Table 7. As the computing capacity of the GPU in-creases, it allows more complex networks with highernumber of network parameters to be trained with lesstime.
Software : The open software library packages pro-vide implementation of various neural network oper-ations like convolution. Most popular python librarypackages used for DNN method implementations areCaffe[68], Tensorflow[1], Theano[16], and PyTorch[106].Some of the third-party packages which work on the
Table 7: GPU specifications[103].
Year GPUType CUDACores TensorCores TDP(Watts) RAM(GB) top of these networks are Keras[30], Lasagne[39], andTensorLayer[40].
Survey and Analysis on Automated Glioma Brain Tumor Segmentation and Overall Patient Survival Prediction T a b l e : Su mm a r i z a t i o n o f S e g m e n t a t i o n m e t h o d s u s i n g C NN a r c h i t e c t u r e s . R e f . P r e - p r o c e ss i n g I npu t M o d a l - i t y + A u g - m e n t a t i o n P a t c h / I m a g e I npu t v i e w N e t w o r k A r c h i t e c - t u r e N e t - w o r k s E n s e m b l e T y p e L o ss F un c - t i o n P o s t - p r o c e ss i n g D a t a s e t D S C M e a n C NN [ ] T , T , T c , F L A I R D p a t c h e s a x i a l D C NN - S o f t m a x B r a T S C h a ll e n g e W T : . , TC : . , E T : . [ ] , d o w n s a m p li n g T , T , T c , F L A I R D p a t c h e s a x i a l D C NN - S o f t m a x - B r a T S T r a i n i n g W T : . , TC : . , E T : . D ee p C NN [ ] , T , T , T c , F L A I R , R o t a - t i o n a t ce r t a i n a n g l e s D p a t c h e s a x i a l D C NN - C a t e go r i c a l C r o ss - E n t r o p y B r a T S W T : . , TC : . , E T : . [ ] , , T , T , T c , F L A I R D m u l - t i s c a l e p a t c h e s a x i a l D C NN c a s c a d e d , p a r a ll e l S o f t m a x B r a T S T e s t S e t W T : . , TC : . , E T : . [ ] , , T , T , T c , F L A I R , h o r - i z o n t a l a nd v e r t i c a l fl i p - p i n g2 D p a t c h e s a x i a l D C NN - W e i g h t e d c r o ss - e n t r o p y w i t h L nd L r e g - u l a r i z a - t i o n B r a T S T e s t S e t W T : . , TC : . , E T : . F C NN [ ] T , T , T c , F L A I R + w i t h fl i pp i n ga r o und a m i d s ag i tt a l p l a n e D m u l - t i s c a l e p a t c h e s a x i a l D F C NN p a r a ll e l - B r a T S W i t h o u t r e s i d - u a l c o nn ec t i o n s W T : . , TC : . , E T : . W i t h r e s i d - u a l c o nn ec t i o n W T : . , TC : . , E T : . [ ] , T c , T , F L A I R D m u l - t i s c a l e p a t c h e s a x i a l D F C NN c a s c a d e d - B r a T S C h a ll e n g e : W T : . , TC : . , E T : . L e a d e r b oa r d : W T : . , TC : . , E T : . Rupal R. Agravat, Mehul S. Raval T a b l e c o n t i nu e d ... R e f . P r e - p r o c e ss i n g I npu t M o d a l - i t y + A u g - m e n t a t i o n P a t c h / I m a g e I npu t v i e w N e t w o r k A r c h i t e c - t u r e N e t - w o r k s E n s e m b l e T y p e L o ss F un c - t i o n P o s t - p r o c e ss i n g D a t a s e t D S C M e a n [ ] T , T , T c , F L A I R , R o - t a t i o n a t d e g r ee , b i a s fi e l d c o rr ec t e d i m ag e s , A dd i - t i o n a l I m ag e s D p a t c h e s a x i a l D F C NN c a s c a d e dS o f t m a x w i t h L r e g - u l a r i z a - t i o n - B r a T S C h a ll e n g e : W T : . , TC : . , E T : . [ ] - T , T , T c , F L A I R D p a t c h e s a x i a l D F C NN - s i g m o i d - B r a T S T r a i n i n g W T : . , TC : . , E T : . [ ] - T , T , T c , F L A I R D m u l - t i s c a l e p a t c h e s a x i a l D F C NN p a r a ll e l S o f t m a x - B r a T S T r a i n i n g 3 D n e t W T : . , TC : . , E T : .
63 3 D N e t W T : . , TC : . , E T : .
74 3 D N e t W T : . , TC : . , E T : . [ ] T , T , T c , F L A I R D p a t c h e s a x i a l D F C NN - D i ce L o ss - B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . T e s t s e t : W T : . , TC : . , E T : . [ ] T , T , T c , F L A I R D I m ag e s a x i a l D F C NN - S o f t m a x - B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . T e s t : W T : . , TC : . , E T : . [ ] - T , T c , F L A I R D I m ag e s a x i a l D F C NN --- B r a T S V a li d a t i o n : W T : . , TC : . , E T : . T e s t : W T : . , TC : . , E T : . [ ] T , T , T c , F L A I R D I m ag e s a x i a l D F C NN - D i ce l o ss + c r o ss e n t r o p y , B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . T e s t s e t : W T : . , TC : . , E T : . [ ] , T , T , T c , F L A I R D I m ag e s a x i a l D F C NN -- , B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . T e s t s e t : W T : . , TC : . , E T : . [ ] T , T , T c , F L A I R D I m ag e s a x i a l D U - n e t - D i ce l o ss - B r a T S T r a i n i n g : W T : . , TC : . , E T : . [ ] , , T , T , T c , F L A I R D p a t c h e s a x i a l D U - n e t --- B r a T S W T : . , TC : . , E T : . [ ] T , T , T c , F L A I R D I m ag e s a x i a l D F C NN - M u l t i s c a l e w e i g h t e d l o ss - B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . T e s t s e t : W T : . , TC : . , E T : . Survey and Analysis on Automated Glioma Brain Tumor Segmentation and Overall Patient Survival Prediction T a b l e c o n t i nu e d ... R e f . P r e - p r o c e ss i n g I npu t M o d a l - i t y + A u g - m e n t a t i o n P a t c h / I m a g e I npu t v i e w N e t w o r k A r c h i t e c - t u r e N e t - w o r k s E n s e m b l e T y p e L o ss F un c - t i o n P o s t - p r o c e ss i n g D a t a s e t D S C M e a n [ ] T , T , T c , F L A I R D p a t c h e s a x i a l D F C NN - W e i g h t e d D i ce l o ss + C r o ss E n - t r o p y L o ss B r a T S V a li d a t i o n s e t W T : . , TC : . , E T : . T e s t s e t W T : . , TC : . , E T : . [ ] - T , T , T c , F L A I R D I m ag e s a x i a l D F C NN - C r o ss - e n t r o p y l o ss - B r a T S T r a i n i n g s e t W T : . , TC : . , E T : . [ ] , T , T , T c , F L A I R D I m ag e s a x i a l D F C NN - W e i g h t e d c r o ss e n t r o p y l o ss + g e n e r - a li ze d d i ce l o ss - B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . T e s t s e t : W T : . , TC : . , E T : . [ ] - T , T , T c , F L A I R D I m ag e s a x i a l D F C NN - D i ce l o ss - B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . T e s t s e t : W T : . , TC : . , E T : . [ ] - T , T , T c , F L A I R D I m ag e s a x i a l D U - n e t - C r o ss E n - t r o p y - B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . [ ] T , T , T c , F L A I R D I m ag e s a x i a l D F C NN - S o f t m a x - B r a T S T e s t s e t : W T : . , TC : . , E T : . E n s e m b l e o f C NN s [ ] , n ( T , T c ) , T , T , T c , F L A I R D m u l - t i s c a l e p a t c h e s a x i a l D C NN c a s c a d e d , p a r a ll e l s o f t m a x w i t h L nd L r e g - u l a r i z a - t i o n - B r a T S T r a i n i n g H GG / L GG W T : . , TC : . , E T : . W T : . , TC : . , E T : . [ ] , , nd i t s d i ff e r e n t c o m - b i n a t i o n T , T , T c , F L A I R D m u l - t i s c a l e p a t c h e s a x i a l D F C NN P a r a ll e l - B r a T S V a li d a t i o n W T : . , TC : . , E T : . T e s t s e t W T : . , TC : . , E T : . Rupal R. Agravat, Mehul S. Raval T a b l e c o n t i nu e d ... R e f . P r e - p r o c e ss i n g I npu t M o d a l - i t y + A u g - m e n t a t i o n P a t c h / I m a g e I npu t v i e w N e t w o r k A r c h i t e c - t u r e N e t - w o r k s E n s e m b l e T y p e L o ss F un c - t i o n P o s t - p r o c e ss i n g D a t a s e t D S C M e a n [ ] , T c , T , F L A I R D I m ag e s a x i a l, c o r o - n a l, s ag i t - t a l D F C NN P a r a ll e l - B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . T e s t s e t : W T : . , TC : . , E T : . [ ] ,i n t e n s i t y s c a l e ,i n t e n s i t y s h i f t , fl i p T , T , T c , F L A I R D I m ag e s a x i a l D F C NN - A w e i g h t e d a v e r ag e o f L , D i ce , K L - B r a T S V a li d a t i o nS e t W T : . , TC : . , E T : . T e s t s e t W T : . , TC : . , E T : . [ ] n t h e i nd i - v i du a l v o l u m e T , T , T c , F L A I R D I m ag e s a x i a l, c o r o - n a l, s ag i t - t a l D F C NN P a r a ll e l -- B r a T S V a li d a t i o n s e t W T : . , TC : . , E T : . T e s t s e t W T : . , TC : . , E T : . [ ] T , T , T c , F L A I R D p a t c h e s a x i a l, c o r o - n a l, s ag i t - t a l D F C NN P a r a ll e l o f c a s - c a d e s D i ce L o ss - B r a T S V a li d a t i o n s e t W T : . , TC : . , E T : . T e s t s e t W T : . , TC : . , E T : . [ ] -- D m u l t i s c a l e p a t c h e s a x i a l - p a r a ll e l - , B r a T S V a li d a t i o n s e t E T : . , W T : . , TC : . T e s t s e t E T : . , W T : . , TC : . [ ] , T , T , T c , F L A I R D i m ag e s a x i a l D F C NN c a s c a d e d M e a n D i ce L o ss - B r a T S V a li d a t i o n s e t W T : . , TC : . , E T : . [ ] , T , T , T c , F L A I R D p a t c h e s a x i a l D F C NN c a s c a d e d C r o ss e n t r o p y - B r a T S V a li d a t i o n s e t W T : . , TC : . , E T : . [ ] T , T , T c , F L A I R D i m ag e s a x i a l D F C NN c a s c a d e dS i g m o i d B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . T e s t s e t : W T : . , TC : . , E T : . [ ] , T , T , T c , F L A I R D p a t c h e s a x i a l D F C NN c a s c a d e d D i ce l o ss - B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . T e s t s e t : W T : . , TC : . , E T : . [ ] T , T , T c , F L A I R D m u l - t i s c a l e p a t c h e s a x i a l D C NN p a r a ll e l S o f t m a x - B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . T e s t s e t : W T : . , TC : . , E T : . [ ] - T , T , T c , F L A I R D I m ag e s a x i a l, c o r o - n a l, s ag i t - t a l D U - N e t p a r a ll e l D i ce L o ss - B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . T e s t : W T : . , TC : . , E T : . Survey and Analysis on Automated Glioma Brain Tumor Segmentation and Overall Patient Survival Prediction T a b l e c o n t i nu e d ... R e f . P r e - p r o c e ss i n g I npu t M o d a l - i t y + A u g - m e n t a t i o n P a t c h / I m a g e I npu t v i e w N e t w o r k A r c h i t e c - t u r e N e t - w o r k s E n s e m b l e T y p e L o ss F un c - t i o n P o s t - p r o c e ss i n g D a t a s e t D S C M e a n [ ] - T , T , T c , F L A I R D I m ag e s a x i a l D V - n e t c a s c a d e d w e i g h t e d D i ce + c r o ss e n t r o p y - B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . [ ] T , T , T c , F L A I R D P a t c h e s a x i a l D U - N e t p a r a ll e l t o c a s - c a d e d D i ce l o ss - B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . T e s t s e t : W T : . , TC : . , E T : . [ ] - T , T c , F L A I R D I m ag e s a x i a l D U - N e t c a s c a d e d D i ce l o ss - B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . T e s t s e t : W T : . , TC : . , E T : . [ ] , , T , T c , F L A I R D I m ag e s a x i a l D F C NN p a r a ll e l t o c a s - c a d e d D i ce l o ss - B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . T e s t s e t : W T : . , TC : . , E T : . [ ] T , T , T c , F L A I R + t r a n s f o r m a - t i o n a ndn o i s e b l u rr i n g3 D I m ag e s a x i a l D F C NN ( T e s t t i m e a u g - m e n t a t i o n ) c a s c a d e d D i ce l o ss - B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . T e s t s e t : W T : . , TC : . , E T : . [ ] T , T , T c , F L A I R D P a t c h e s a x i a l D F C NN p a r a ll e l D i ce l o ss - B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . T e s t s e t : W T : . , TC : . , E T : . [ ] T , T , T c , F L A I R D I m ag e s a x i a l, c o r o - n a l, s ag i t - t a l D F C NN p a r a ll e l -- B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . [ ] T , T , T c , F L A I R D I m ag e s a x i a l, c o r o - n a l, s ag i t - t a l D F C NN p a r a ll e l -- B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . T e s t s e t : W T : . , TC : . , E T : . [ ] , T , T , T c , F L A I R D I m ag e s a x i a l D U - n e t p a r a ll e l w e i g h t e d c a t e - go r i c a l c r o ss e n t r o p y l o ss - B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . T e s t s e t : W T : . , TC : . , E T : . Rupal R. Agravat, Mehul S. Raval T a b l e c o n t i nu e d ... R e f . P r e - p r o c e ss i n g I npu t M o d a l - i t y + A u g - m e n t a t i o n P a t c h / I m a g e I npu t v i e w N e t w o r k A r c h i t e c - t u r e N e t - w o r k s E n s e m b l e T y p e L o ss F un c - t i o n P o s t - p r o c e ss i n g D a t a s e t D S C M e a n [ ] T , T , T c , F L A I R D I m ag e s a x i a l D V - n e t p a r a ll e l g e n e r a li ze d d i ce l o ss B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . T e s t s e t : W T : . , TC : . , E T : . [ ] T , T , T c , F L A I R D I m ag e s a x i a l, c o r o n a l, s ag i tt a l D F C NN p a r a ll e l d i ce l o ss - B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . T e s t s e t : W T : . , TC : . , E T : . [ ] , , T , T , T c , F L A I R D I m ag e s a x i a l, c o r o - n a l, s ag i t - t a l D F C NN p a r a ll e l m u l t i c l a ss d i ce l o ss - B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . T e s t s e t : W T : . , TC : . , E T : . [ ] T , T , T c , F L A I R D I m ag e s a x i a l D F C NN p a r a ll e l c a t e go r i c a l c r o ss e n t r o p y - B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . T e s t s e t : W T : . , TC : . , E T : . [ ] T , T , T c , F L A I R D I m ag e s + D m u l t i r e s o l u t i o n p a t c h e s a x i a l D F C NN p a r a ll e l - , B r a T S V a li d a t i o n s e t : W T : . , TC : . , E T : . T e s t s e t : W T : . , TC : . , E T : . Survey and Analysis on Automated Glioma Brain Tumor Segmentation and Overall Patient Survival Prediction 27 – The bounding box is applied around the brain to re-duce the input which contains zero intensity (blank)voxels. – The convolution layer is added in the dense module.The results for the network are in Table 9, which in-cludes minor variations in the network, different inputimage sizes, different loss functions, and the number ofimages in training. The network is trained on 85% ofthe total image and validated on the remaining 15%images of the BraTS 2019 dataset. The online tool pro-vided by the organizer generates the evaluation metricfor the training set as well as a separate validation setin addition to the training set.The DSC comparison of training and validation setsfor these variations are as shown in Fig. 17. The seg-mentation results show that the method overfits thetraining data and does not generate good results forthe unknown validation set. The segmentation resultsof the sample images from the training set for correctas well as incorrect segmentation are shown in Fig. 18and Fig. 19. The network does not distinguish betweenthe subregions where the tumor appearance is homoge-neous for all its subregions and intensity is same as thenormal brain tissues. 5.6 End to End methods for tumor segmentation andOS predictionSince 2017, the second task of survival prediction wasintroduced in the challenge. Some of the methods hadparticipated in end-to-end solutions in the challenge,i.e., segmentation followed by survival prediction.In [125], an ensemble of RF and CNN segments thetumor and random forest regressor(RFR) was used topredict the overall survival days using 240 features outof 1366 different features (Kaplan-Meier was used tofind the relevant and useful features). Authors in [69]had modified U-net with Full Resolution Residual Net-work(FRRN) and Residual Unit(RU) units along withweight scaling dropout. The survival prediction ANNworked with linear activation function on four selectedfeatures.The variant of U-net was used in [62], which took3D input and included context module and localiza-tion module in each level of the architecture. The seg-mentation result was generated based on the element-wise summation of the output from the decoder lay-ers. Survival prediction was the average of RFR andmultilayer perceptron(MLP). RFR trains on 517 fea-tures extracted from three tumor sub-regions using theradiomics package[141]. The output of RFR and MLPaverages 15 MLPs design with three hidden layers, eachwith 64 neurons.Authors in [9] implemented 3D U-net with threestages of encoder-decoder architecture. Regression modelbased radiomics features selection trains MLP for OSprediction. Whereas in [144], two 3D U-nets uses four-stage encoder-decoder architecture, the first networksegmented the whole tumor, and the second one seg-mented the tumor sub-region. In addition to four con-ventional modalities, they used the additional image asan input, which is the T1c-T1 subtracted image. Thisimage provides additional information for the tumorcore region. They used only the age feature was used topredict OS using linear regressor. The approach pre-sented in [83] used FCNN named FCRN-101, whichderives from pre-trained SegNet and U-net architec-ture. Three path network combines the result of threeviews, i.e., axial, coronal, and sagittal. The OS predic-tion used SPNet, the fully-connected CNN, which tookfour modalities and the network segmentation result asinput to predict the probability of OS prediction.The [45], uses an ensemble of six 3D U-net type net-works with variation in the input size, number of encod-ing/decoding blocks, and feature maps at every layer.The OS prediction uses linear regression with groundtruth image volume, surface area, age, and resectionstatus. Features were input to the network after z-score
Fig. 15: DSC comparison for whole tumor segmentation of CNN methods.Fig. 16: Three stage U-net architecture [3].normalization. Authors in [114] had implemented FCNand generated results for three axes and used majorityvoting to generate the final segmentation results. ForOS prediction, ten features (focusing on necrosis andactive tumor) from the segmentation results were gen-erated, and mean PCA and standard deviation PCA totrain RF on the GTR images.In [133] an ensemble of three networks (U-net[121],DFKZNet[62], and CA-CNN[142]) was used, and ma-jority voting applied for final segmentation. The OSprediction used RF with 14 radiomics features selectedfrom various modality images, Laplacian of GaussianImages and wavelet decomposed images. Authors in[6] had implemented 2D U-net architecture with threestages for tumor segmentation and age, volumetric, andshape features of the whole tumor were used to predictOS. All the approaches did not use the location-based in-formation of the tumor and its sub-regions. In contrast,[72], uses twenty-one brain parcellation regions as an in-put along with the four MR modalities. It emphasizesthe number of tumor regions in those specific parcella-tion areas. Those twenty-five input channels were givenas input the ensemble of 3D U-net as well as the ensem-ble of DeepMedic architectures with different kernel andinput patch sizes. Tractrographic features from networksegmented regions trains SVM classifiers with the lin-ear kernel to predict OS. Authors in [3] implements 2DU-net of three stages with dense blocks at every en-coder level, and the feature set of [6] of necrosis tumorsub-region for OS prediction.The authors in [128] combines the features fromVGG16 based FCN and texton maps to generate thefeatures and supply them to RF classifier to generatethe segmentation result. RF is also used for OS pre-
Survey and Analysis on Automated Glioma Brain Tumor Segmentation and Overall Patient Survival Prediction 29
Table 9: Comparison of model variations of [3].
Model Architecturalchange InputModalities
Fig. 17: DSC comparison for training and validation seta) whole tumor b)tumor core c) ehhancing tumor. diction using volumetric as well as age feature of thepatient.Authors in [154], [60], [65], [37] attempts the end-to-end segmentation approach.Table 10 compares the segmentation results of end-to-end methods. The approaches for pre-processing andpost-processing techniques are as specified in section 5.3and Table 11 provides details related to the survivalprediction.
Fig. 18: Training set image, correct segmentation a)Original FLAIR image b) ground truth segmentationc) model 1 d) model 2 e) model 3 f) model 4 g) model5 h) model 6. (a) (b)(c) (d)(e) (f)(g) (h)
Fig. 19: Training set image, incorrect segmentation a)Original FLAIR image b) ground truth segmentationc) model 1 d) model 2 e) model 3 f) model g) model 5h) model 6.
Survey and Analysis on Automated Glioma Brain Tumor Segmentation and Overall Patient Survival Prediction T a b l e : E nd - t o - e nd M e t h o d s , T a s k : B r a i n T u m o r S e g m e n t a t i o n . R e f . P r e - p r o c e ss i n g I npu t M o d a l - i t y + A u g - m e n t a t i o n P a t c h / I m a g e I npu t v i e w N e t w o r k A r c h i t e c - t u r e N e t - w o r k s E n s e m b l e T y p e L o ss F un c - t i o n P o s t - p r o c e ss i n g D a t a s e t D S C M e a n [ ] T , T , T c , F L A I R D I m ag e s a x i a l D F C NN P a r a ll e l W e i g h t e d F o c a l L o ss - B r a T S V a li d a t i o n E T : . , W T : . , CT : . T e s t s e t E T : . , W T : . , TC : . [ ] f o ll o w e db y c li pp i n ga nd s c a li n g T , T , T c , F L A I R D p a t c h e s a x i a l D F C NN - W e i g h t e d D i ce l o ss - B r a t s V a li d a t i o n s e t W T : . , CT : . , E T : . T e s t s e t W T : . , TC : . , E T : . [ ] , T , T , T c , F L A I R D P a t c h e s a x i a l D C NN + R a nd o m F o r e s t - S o f t m a x - B r a T S H G T e s t s e t E T : . , W T : . , TC : . [ ] - T , T , T c , F L A I R D I m ag e s a x i a l D F C NN --- B r a T S V a li d a t i o n s e t W T : . , TC : . , E T : . T e s t s e t W T : . , TC : . , E T : . [ ] , , T , T , T c , F L A I R D I m ag e s a x i a l D F C NN --- B r a T S V a li d a t i o n : E T : . , W T : . , TC : . T e s t E T : . , W T : . , TC : . [ ] , T , T , T c , F L A I R D + D P a t c h e s a x i a l D F C NN c a s c a d e d t o p a r a l - l e l - , B r a T S T r a i n i n g : E T : . , W T : . , TC : . [ ] , T , T , T c , F L A I R , R a n - d o m F li p D P a t c h e s a x i a l D F C NN p a r a ll e l W e i g h t e d U n i - f o r m L o ss - B r a T S V a li d a t i o n s e t W T : . , TC : . , E T : . [ ] T , T , T c D I m ag e s a x i a l F C NN - S o f t m a x B r a T S T r a i n i n g s e t W T : . [ ] , r a nd o m fl i p - p i n g , ga u ss i a n n o i s e T , T , T c , F L A I R D I m ag e s a x i a l F C NN p a r a ll e l -- B r a T S V a li d a t i o n s e t E T : . , W T : . , TC : . T e s t s e t E T : . , W T : . , TC : . Rupal R. Agravat, Mehul S. Raval T a b l e c o n t i nu e d ... R e f . P r e - p r o c e ss i n g I npu t M o d a l - i t y + A u g - m e n t a t i o n P a t c h / I m a g e I npu t v i e w N e t w o r k A r c h i t e c - t u r e N e t - w o r k s E n s e m b l e T y p e L o ss F un c - t i o n P o s t - p r o c e ss i n g D a t a s e t D S C M e a n [ ] , T , T , T c , F L A I R D P a t c h e s a x i a l D F C NN -- , , f a l s e p o s i - t i v e s r e - m o v a l B r a T S V a li d a t i o n s e t E T : . , W T : . , TC : . T e s t s e t E T : . , W T : . , TC : . [ ] T , T , T c , F L A I R , ( T c - T ) D p a t c h e s a x i a l D F C NN c a s c a d e d D i ce l o ss - B r a T S V a li d a t i o n s e t E T : . , W T : . , TC : . T e s t s e t E T : . , W T : . , TC : . [ ] - T , T , T c , F L A I R , b i n a r y b r a i n p a r ce ll a t i o n i m ag e s D p a t c h e s a x i a l D F C NN p a r a ll e l -- B r a T S V a li d a t i o n s e t E T : . , W T : . , TC : . T e s t s e t E T : . , W T : . , TC : . [ ] , , T , T , T c , F L A I R d I m ag e s a x i a l D F C NN p a r a ll e l t o c a s - c a d e d F o c a l l o ss B r a T S V a li d a t i o n : E T : . , W T : . , TC : . T e s t : E T : . , W T : . , TC : . [ ] - T , T , T c , F L A I R D I m ag e s a x i a l, c o r o - n a l, s ag i t - t a l D F C NN p a r a ll e l W e i g h t e d c r o ss e n t r o p y + g e n - e r a li ze d d i ce l o ss - B r a T S V a li d a t i o n : E T : . , W T : . , TC : . T e s t : E T : . , W T : . , TC : . [ ] T , T c , F L A I R D I m ag e s a x i a l D F C NN --- B r a T S V a li d a t i o n : E T : . , W T : . , TC : . T e s t : E T : . , W T : . , TC : . [ ] , T , T , T c , F L A I R D I m ag e s a x i a l D F C NN p a r a ll e l c o n f u s i o n l o ss + m u l t i c l a ss d i ce l o ss , B r a T S V a li d a t i o n : E T : . , W T : . , TC : . T e s t : E T : . , W T : . , TC : . [ ] T , T , T c , F L A I R D I m ag e s a x i a l D F C NN - D i ce - B r a T S - [ ] T , T , T c , F L A I R D I m ag e s a x i a l D F C NN - F o c a l l o ss - B r a T S V a li d a t i o n s e t E T : . W T : . TC : . Survey and Analysis on Automated Glioma Brain Tumor Segmentation and Overall Patient Survival Prediction 33
Table 11: End-to-end Methods, Task 2: OS Prediction.
Ref. Method [83] 2D CNN - BraTS2017 Valid:55Test:45[62] RFR andMLP 66 BraTS2017 52.6[125] RFR 240 BraTS2017 Valid:66.7Test:57.9[69] ANN 4 BraTS2017 Valid:42.4Test:56.8[45] Linear Regres-sion Model 9 BraTS2018 Valid:32.1[114] PCA + RF 10 BraTS2018 Test:61[133] RF 14 BraTS2018 Valid:46.4Test:61SecondRank[9] MLP 468 BraTS2018 Valid:57.1Test:55.8[144] Linear Regres-sor 1 BraTS2018 Valid:50Test:55.8[72] SVM with lin-ear kernel - BraTS2018 Valid:35.7Test:41.6[6] RF 13 BraTS2018 59[3] RF 13 BraTS2019 Valid:58.6Test:57.9[128] RF 4 BraTS2017 Valid:48.5Test:41.1[154] CNN+XBoost 183 +CNNfeatures BraTS2017 Training:63.3[60] ensemble ofXboost, SVM,MLP, DT,RF, LDA 900 BraTS2018 Test:51.9[14] MLP 83 BraTS2018 Valid:54[65] ANN 50 BraTS2018 Valid:67.9Test:46.8[37] Xboost 195 BraTS2017 Valid:50
The tumor segmentation result depends on the architec-tural design (from shallow CNN to the ensemble/cascaded CNN), amount of training data, input pre-processing,type of input(2D/3D), network optimization as well aspost-processing of the generated output. Still, the DLmethods have certain limitations, which include: – Over-fitting: The common problem of the DNN basedapproach is over-fitting. It may occur due to the un-availability of the ample amount of labeled trainingdata for brain tumor segmentation, which refers toa model that has an excellent performance on thetraining dataset but does not perform well on newdata. The over-fitting problem can be handled ei-ther by reducing the network complexity (in termsof network layers and parameters) or by generat-ing an ample amount of training data using imageaugmentation techniques. The augmentation tech-niques produce new images by performing data trans-formations and the corresponding ground truth thatincludes operations of scaling, rotation, translation,brightness variation, elastic deformations, horizon-tal flipping, and mirroring. – Class imbalance: Class imbalance is another issuein tumor segmentation where the background classdominates the foreground class(tumor). Class im-balance can be handle by proper training data sam-pling, improved loss functions, and augmentationtechniques.The highest reported accuracy for the survival pre-diction task does not exceed 63% [45]. It is due to thedependency on the extraction of the features on the seg-mentation results. The incorrect segmentation results inwrong feature extraction for survival prediction. The bi-ological importance of the extracted features also playsan important role. If the relevance of the features isnot known correctly, the survival prediction cannot beaccurate. Besides, the importance of tumor sub-regionsplays a role in feature extraction. The dataset includesall the pre-operative scans and does not give any otherinformation like the success of tumor removal, post-operative treatments to the patients, and the responseof the patients to such treatments. Chances of tumorreoccurrence are high if the patient is exposed to theradiation environment. Features related to it may fur-ther improve the OS prediction.
The availability of benchmark dataset(BraTS) has grownthe field of computer-assisted medical image analysis forbrain tumor segmentation. The article covers detailedliterature survey for brain tumor segmentation tech-niques. The tumor segmentation is approached withvarious techniques like semi-automated and automated methods. The semi-automated methods works on theinput provided by user. Such methods suffer from limi-tations like manual seed point selection, atlas creation.Methods were gradually improved to include machinelearning techniques like clustering, RF, ANN. The lim-itation of such methods is the selection of the featuresfor training, which requires the knowledge of biologicalinformation of the image. The need of domain knowl-edge is removed by CNNs - deep neural network. CNNextracts high level features at deeper layers successivelyfrom the low level features from the preceding layers.This feature learning has improved the performance ofCNN for tumor segmentation. The improvements toCNN are its variants, FCNN and ensemble of CNN/FCNN.The detailing of all the methods along with pre-processing,post-processing, prominent highlight of the methodsand evaluation measure is given in Table 5, Table 8,Table 10 and Table 11.The ensemble generates robust segmentation resultsas well as improvements in the accuracy of the network.All these methods generate spurious segmentation re-sults, which improves with the help of post-processingtechniques like connected component analysis, spatialregularization and morphological operations to fine-tunethe output. The class imbalance is the primary concernin the training of CNN for medical image analysis. Thebalanced input selection and loss function for a positiveclass will resolve the issue. Due to the unavailability ofless amount of data for training, it may be possiblethat the network overfits the training data. The reg-ularization and dropout resolve such issues. Moreover,as Bayesian CNN[127] handles epistemic and aleatoricuncertainties in the presence of limited data and knowl-edge, such type of network is useful for semantic seg-mentation. Adaptive loss[15], approximates varieties ofloss functions with single latent variable. This functioncan further be thought of to solve the problem of seman-tic segmentation. Despite the popularity such methodshave following limitations: – computational efficiency – memory requirement: As the depth of the networkincreases, the number of network parameters increases.This increase in the parameter requires additionalmemory as well as time to tune those on each epoch. – Ample amount of annotated training data require-ment: The annotated data generation is itself a chal-lenge as this process is very time consuming and theannotation results may change depending on vari-ability of the expert. In addition specific annotationtools are required by the expert for proper delin-eation and annotation purpose. In place of voxelannotation, image labeling for presence/absence oftumor is less time consuming, does not require much expertise and specialized annotation tool. Use ofthese image labeling in addition to the voxel label-ing may help the network to learn relevant featuresfor segmentation with less annotated data.
Acknowledgements
The authors would like to thank NVIDIACorporation for donating the Quadro K5200 and QuadroP5000 GPU used for this research, Dr. Krutarth Agravat(Medical Officer, Essar Ltd) for clearing our doubts related tomedical concepts, Po-yu Kao, and Ujjawal Baid for their con-tinuous support and help, Dr. Spyros and his entire team forBraTS dataset. The authors acknowledge continuous supportfrom Professor Sanjay Chaudhary, Professor N. Padmanab-han, and Professor Manjunath Joshi for this work.
References
1. Abadi M, Barham P, Chen J, Chen Z, Davis A,Dean J, Devin M, Ghemawat S, Irving G, Isard M,et al. (2016) Tensorflow: A system for large-scalemachine learning. In: 12th { USENIX } Symposiumon Operating Systems Design and Implementation( { OSDI } Survey and Analysis on Automated Glioma Brain Tumor Segmentation and Overall Patient Survival Prediction 35 convolutional neural network for predicting sur-vival in gliomas. In: International MICCAI Brain-lesion Workshop, Springer, pp 369–37910. Bakas S, Zeng K, Sotiras A, Rathore S, AkbariH, Gaonkar B, Rozycki M, Pati S, DavatzikosC (2015) Glistrboost: combining multimodal mrisegmentation, registration, and biophysical tu-mor growth modeling with gradient boosting ma-chines for glioma segmentation. In: BrainLes 2015,Springer, pp 144–15511. Bakas S, Akbari H, Sotiras A, Bilello M, RozyckiM, Kirby J, Freymann J, Farahani K, DavatzikosC (2017) Segmentation labels and radiomic fea-tures for the pre-operative scans of the tcga-gbmcollection. the cancer imaging archive (2017)12. Bakas S, Akbari H, Sotiras A, Bilello M, RozyckiM, Kirby J, Freymann J, Farahani K, DavatzikosC (2017) Segmentation labels and radiomic fea-tures for the pre-operative scans of the tcga-lggcollection. The Cancer Imaging Archive 28613. Bakas S, Akbari H, Sotiras A, Bilello M, Rozy-cki M, Kirby JS, Freymann JB, Farahani K, Da-vatzikos C (2017) Advancing the cancer genomeatlas glioma mri collections with expert segmen-tation labels and radiomic features. Scientific data4:17011714. Banerjee S, Mitra S, Shankar BU (2018) Multi-planar spatial-convnet for segmentation and sur-vival prediction in brain cancer. In: InternationalMICCAI Brainlesion Workshop, Springer, pp 94–10415. Barron JT (2019) A general and adaptive robustloss function. In: Proceedings of the IEEE Con-ference on Computer Vision and Pattern Recog-nition, pp 4331–433916. Bastien F, Lamblin P, Pascanu R, Bergstra J,Goodfellow I, Bergeron A, Bouchard N, Warde-Farley D, Bengio Y (2012) Theano: new fea-tures and speed improvements. arXiv preprintarXiv:1211559017. Bauer S, Fejes T, Slotboom J, Wiest R, Nolte LP,Reyes M (2012) Segmentation of brain tumor im-ages based on integrated hierarchical classificationand regularization. In: MICCAI BraTS Workshop.Nice: Miccai Society, p 1118. Bernal J, Kushibar K, Asfaw DS, Valverde S,Oliver A, Mart´ı R, Llad´o X (2019) Deep convo-lutional neural networks for brain image analysison magnetic resonance imaging: a review. Artifi-cial intelligence in medicine 95:64–8119. Bharath HN, Colleman S, Sima DM, Van Huffel S(2017) Tumor segmentation from multimodal mriusing random forest with superpixel and tensor based feature extraction. In: International MIC-CAI Brainlesion Workshop, Springer, pp 463–47320. Buendia P, Taylor T, Ryan M, John N (2013) Agrouping artificial immune network for segmenta-tion of tumor images. Multimodal Brain TumorSegmentation 121. Casamitjana A, Puch S, Aduriz A, VilaplanaV (2016) 3d convolutional neural networks forbrain tumor segmentation: a comparison of multi-resolution architectures. In: International Work-shop on Brainlesion: Glioma, Multiple Sclerosis,Stroke and Traumatic Brain Injuries, Springer, pp150–16122. Casamitjana A, Cat`a M, S´anchez I, CombaliaM, Vilaplana V (2017) Cascaded v-net using roimasks for brain tumor segmentation. In: Interna-tional MICCAI Brainlesion Workshop, Springer,pp 381–39123. Castillo LS, Daza LA, Rivera LC, Arbel´aez P(2017) Brain tumor segmentation and parsingon mris using multiresolution neural networks.In: International MICCAI Brainlesion Workshop,Springer, pp 332–34324. Center RM (2019 (accessed April 6, 2020))Health Encyclopedia. URL
25. Chandra S, Vakalopoulou M, Fidon L, BattistellaE, Estienne T, Sun R, Robert C, Deutsch E, Para-gios N (2019) Context aware 3d cnns for brain tu-mor segmentation. brainles 2018. Springer LNCS11384:299–31026. Chang PD (2016) Fully convolutional deep resid-ual neural networks for brain tumor segmenta-tion. In: International Workshop on Brainlesion:Glioma, Multiple Sclerosis, Stroke and TraumaticBrain Injuries, Springer, pp 108–11827. Chen L, Bentley P, Mori K, Misawa K, Fuji-wara M, Rueckert D (2018) Drinet for medical im-age segmentation. IEEE transactions on medicalimaging 37(11):2453–246228. Chen LC, Papandreou G, Kokkinos I, Murphy K,Yuille AL (2017) Deeplab: Semantic image seg-mentation with deep convolutional nets, atrousconvolution, and fully connected crfs. IEEE trans-actions on pattern analysis and machine intelli-gence 40(4):834–84829. Chen W, Liu B, Peng S, Sun J, Qiao X (2018)S3d-unet: separable 3d u-net for brain tumor seg-mentation. In: International MICCAI BrainlesionWorkshop, Springer, pp 358–36830. Chollet F, et al. (2018) Keras: The python deeplearning library. Astrophysics Source Code Li-brary
31. Choudhury AR, Vanguri R, Jambawalikar SR,Kumar P (2018) Segmentation of brain tumors us-ing deeplabv3+. In: International MICCAI Brain-lesion Workshop, Springer, pp 154–16732. Clark MC, Hall LO, Goldgof DB, Velthuizen R,Murtagh FR, Silbiger MS (1998) Automatic tu-mor segmentation using knowledge-based tech-niques. IEEE transactions on medical imaging17(2):187–20133. Colmeiro RR, Verrastro C, Grosges T (2017) Mul-timodal brain tumor segmentation using 3d con-volutional networks. In: International MICCAIBrainlesion Workshop, Springer, pp 226–24034. Cordier N, Menze B, Delingette H, Ayache N(2013) Patch-based segmentation of brain tissues35. Corso JJ, Sharon E, Dube S, El-Saden S, SinhaU, Yuille A (2008) Efficient multilevel brain tu-mor segmentation with integrated bayesian modelclassification. IEEE transactions on medical imag-ing 27(5):629–64036. Cuadra MB, Pollo C, Bardera A, Cuisenaire O,Villemure JG, Thiran JP (2004) Atlas-based seg-mentation of pathological mr brain images usinga model of lesion growth. IEEE transactions onmedical imaging 23(10):1301–131437. Dai L, Li T, Shu H, Zhong L, Shen H, ZhuH (2018) Automatic brain tumor segmentationwith domain adaptation. In: International MIC-CAI Brainlesion Workshop, Springer, pp 380–39238. Dera D, Raman F, Bouaynaya N, Fathallah-Shaykh HM (2016) Interactive semi-automatedmethod using non-negative matrix factorizationand level set segmentation for the brats chal-lenge. In: International Workshop on Brainlesion:Glioma, Multiple Sclerosis, Stroke and TraumaticBrain Injuries, Springer, pp 195–20539. Dieleman S, Schl¨uter J, Raffel C, Olson E,Sønderby SK, Nouri D, et al. (2015) Lasagne: Firstrelease. DOI 10.5281/zenodo.27878, URL http://dx.doi.org/10.5281/zenodo.27878
40. Dong H, Supratak A, Mai L, Liu F, OehmichenA, Yu S, Guo Y (2017) Tensorlayer: a versatile li-brary for efficient deep learning development. In:Proceedings of the 25th ACM international con-ference on Multimedia, pp 1201–120441. Dong H, Yang G, Liu F, Mo Y, Guo Y (2017) Au-tomatic brain tumor detection and segmentationusing u-net based fully convolutional networks. In:annual conference on medical image understand-ing and analysis, Springer, pp 506–51742. Doyle S, Vasseur F, Dojat M, Forbes F (2013)Fully automatic brain tumor segmentation frommultiple mr sequences using hidden markov fields and variational em. Procs NCI-MICCAI BraTS pp18–2243. Ellwaa A, Hussein A, AlNaggar E, Zidan M, ZakiM, Ismail MA, Ghanem NM (2016) Brain tumorsegmantation using random forest trained on iter-atively selected patients. In: International Work-shop on Brainlesion: Glioma, Multiple Sclerosis,Stroke and Traumatic Brain Injuries, Springer, pp129–13744. Feng X, Meyer C (2017) Patch-based 3d u-netfor brain tumor segmentation. In: InternationalConference on Medical Image Computing andComputer-Assisted Intervention (MICCAI)45. Feng X, Tustison N, Meyer C (2018) Brain tumorsegmentation using an ensemble of 3d u-nets andoverall survival prediction using radiomic features.In: International MICCAI Brainlesion Workshop,Springer, pp 279–28846. Festa J, Pereira S, Mariz JA, Sousa N, SilvaCA (2013) Automatic brain tumor segmentationof multi-sequence mr images using random deci-sion forests. Proceedings of NCI-MICCAI BRATS1:23–2647. Geremia E, Menze BH, Ayache N, et al. (2012)Spatial decision forests for glioma segmentationin multi-channel mr images. MICCAI Challengeon Multimodal Brain Tumor Segmentation 3448. Goetz M, Weber C, Bloecher J, Stieltjes B,Meinzer HP, Maier-Hein K (2014) Extremely ran-domized trees based brain tumor segmentation.Proceeding of BRATS challenge-MICCAI pp 006–01149. Goyal B, Agrawal S, Sohi B (2018) Noise is-sues prevailing in various types of medical images.Biomedical & Pharmacology Journal 11(3):122750. Guo X, Schwartz L, Zhao B (2013) Semi-automatic segmentation of multimodal brain tu-mor using active contours. Multimodal Brain Tu-mor Segmentation 2751. Hamamci A, Unal G (2012) Multimodal brain tu-mor segmentation using the tumor-cut method onthe brats dataset. Proc MICCAI-BRATS pp 19–2352. Hamamci A, Kucuk N, Karaman K, Engin K,Unal G (2011) Tumor-cut: segmentation of braintumors on contrast enhanced mr images for radio-surgery applications. IEEE transactions on medi-cal imaging 31(3):790–80453. Havaei M, Dutil F, Pal C, Larochelle H, JodoinPM (2015) A convolutional neural network ap-proach to brain tumor segmentation. In: BrainLes2015, Springer, pp 195–208
Survey and Analysis on Automated Glioma Brain Tumor Segmentation and Overall Patient Survival Prediction 37
54. Havaei M, Davy A, Warde-Farley D, Biard A,Courville A, Bengio Y, Pal C, Jodoin PM,Larochelle H (2017) Brain tumor segmentationwith deep neural networks. Medical image anal-ysis 35:18–3155. Healthcareplex (2016 (accessed April 10,2020)) CT Scan vs. MRI. URL https://healthcareplex.com/mri-vs-ct-scan/
56. Hopkins J (2019 (accessed April 6, 2020))Health. URL
57. Hu X, Li H, Zhao Y, Dong C, Menze BH, Pi-raud M (2018) Hierarchical multi-class segmenta-tion of glioma images using networks with multi-level activation function. In: International MIC-CAI Brainlesion Workshop, Springer, pp 116–12758. Hu Y, Xia Y (2017) 3d deep neural network-basedbrain tumor segmentation using multimodalitymagnetic resonance sequences. In: InternationalMICCAI Brainlesion Workshop, Springer, pp 423–43459. Hu Y, Liu X, Wen X, Niu C, Xia Y (2018)Brain tumor segmentation on multimodal mrimaging using multi-level upsampling in decoder.In: International MICCAI Brainlesion Workshop,Springer, pp 168–17760. Hua R, Huo Q, Gao Y, Sun Y, Shi F (2018)Multimodal brain tumor segmentation using cas-caded v-nets. In: International MICCAI Brainle-sion Workshop, Springer, pp 49–6061. Huang G, Liu Z, Van Der Maaten L, WeinbergerKQ (2017) Densely connected convolutional net-works. In: Proceedings of the IEEE conference oncomputer vision and pattern recognition, pp 4700–470862. Isensee F, Kickingereder P, Wick W, Bendszus M,Maier-Hein KH (2017) Brain tumor segmentationand radiomics survival prediction: Contribution tothe brats 2017 challenge. In: International MIC-CAI Brainlesion Workshop, Springer, pp 287–29763. Isensee F, Kickingereder P, Wick W, Bendszus M,Maier-Hein KH (2018) No new-net. In: Interna-tional MICCAI Brainlesion Workshop, Springer,pp 234–24464. Islam M, Ren H (2017) Multi-modal pixelnet forbrain tumor segmentation. In: International MIC-CAI Brainlesion Workshop, Springer, pp 298–30865. Islam M, Jose VJM, Ren H (2018) Glioma prog-nosis: Segmentation of the tumor and survivalprediction using shape, geometric and clinical in-formation. In: International MICCAI BrainlesionWorkshop, Springer, pp 142–153 66. Janssen PM, Hoff EI (2012) Teaching neuroim-ages: Subacute intracerebral hemorrhage mimick-ing brain tumor. Neurology 79(21):e183–e18367. Jesson A, Arbel T (2017) Brain tumor seg-mentation using a 3d fcn with multi-scale loss.In: International MICCAI Brainlesion Workshop,Springer, pp 392–40268. Jia Y, Shelhamer E, Donahue J, Karayev S, LongJ, Girshick R, Guadarrama S, Darrell T (2014)Caffe: Convolutional architecture for fast featureembedding. In: Proceedings of the 22nd ACM in-ternational conference on Multimedia, pp 675–67869. Jungo A, McKinley R, Meier R, Knecht U,Vera L, P´erez-Beteta J, Molina-Garc´ıa D, P´erez-Garc´ıa VM, Wiest R, Reyes M (2017) Towardsuncertainty-assisted brain tumor segmentationand survival prediction. In: International MICCAIBrainlesion Workshop, Springer, pp 474–48570. Kamnitsas K, Ferrante E, Parisot S, Ledig C, NoriAV, Criminisi A, Rueckert D, Glocker B (2016)Deepmedic for brain tumor segmentation. In:International workshop on Brainlesion: Glioma,multiple sclerosis, stroke and traumatic brain in-juries, Springer, pp 138–14971. Kamnitsas K, Bai W, Ferrante E, McDonagh S,Sinclair M, Pawlowski N, Rajchl M, Lee M, KainzB, Rueckert D, et al. (2017) Ensembles of multiplemodels and architectures for robust brain tumoursegmentation. In: International MICCAI Brainle-sion Workshop, Springer, pp 450–46272. Kao PY, Ngo T, Zhang A, Chen JW, ManjunathB (2018) Brain tumor segmentation and tracto-graphic feature extraction from structural mr im-ages for overall survival prediction. In: Interna-tional MICCAI Brainlesion Workshop, Springer,pp 128–14173. Kermi A, Mahmoudi I, Khadir MT (2018) Deepconvolutional neural networks using u-net for au-tomatic brain tumor segmentation in multimodalmri volumes. In: International MICCAI Brainle-sion Workshop, Springer, pp 37–4874. Kim G (2017) Brain tumor segmentation us-ing deep fully convolutional neural networks.In: International MICCAI Brainlesion Workshop,Springer, pp 344–35775. Kleesiek J, Biller A, Urban G, Kothe U, BendszusM, Hamprecht F (2014) Ilastik for multi-modalbrain tumor segmentation. Proceedings MICCAIBraTS (brain tumor segmentation challenge) pp12–1776. Kori A, Soni M, Pranjal B, Khened M, Alex V,Krishnamurthi G (2018) Ensemble of fully convo-lutional neural network for brain tumor segmenta- tion from magnetic resonance images. In: Interna-tional MICCAI Brainlesion Workshop, Springer,pp 485–49677. Krizhevsky A, Sutskever I, Hinton GE (2012) Im-agenet classification with deep convolutional neu-ral networks. In: Advances in neural informationprocessing systems, pp 1097–110578. Kwon D, Akbari H, Da X, Gaonkar B, DavatzikosC (2014) Multimodal brain tumor image segmen-tation using glistr. MICCAI brain tumor segmen-tation (BraTS) challenge manuscripts pp 18–1979. Lachinov D, Vasiliev E, Turlapov V (2018) Gliomasegmentation with cascaded unet. In: Interna-tional MICCAI Brainlesion Workshop, Springer,pp 189–19880. Le Folgoc L, Nori AV, Ancha S, Criminisi A (2016)Lifted auto-context forests for brain tumour seg-mentation. In: International Workshop on Brainle-sion: Glioma, Multiple Sclerosis, Stroke and Trau-matic Brain Injuries, Springer, pp 171–18381. LeCun Y, Bottou L, Bengio Y, Haffner P (1998)Gradient-based learning applied to documentrecognition. Proceedings of the IEEE 86(11):2278–232482. Lefkovits L, Lefkovits S, Szil´agyi L (2016) Braintumor segmentation with optimized random for-est. In: International Workshop on Brainlesion:Glioma, Multiple Sclerosis, Stroke and TraumaticBrain Injuries, Springer, pp 88–9983. Li Y, Shen L (2017) Deep learning based mul-timodal brain tumor diagnosis. In: InternationalMICCAI Brainlesion Workshop, Springer, pp 149–15884. Lin TY, Goyal P, Girshick R, He K, Doll´ar P(2017) Focal loss for dense object detection. In:Proceedings of the IEEE international conferenceon computer vision, pp 2980–298885. Long J, Shelhamer E, Darrell T (2015) Fully con-volutional networks for semantic segmentation. In:Proceedings of the IEEE conference on computervision and pattern recognition, pp 3431–344086. Lopez MM, Ventura J (2017) Dilated convolu-tions for brain tumor segmentation in mri scans.In: International MICCAI Brainlesion Workshop,Springer, pp 253–26287. Ma J, Yang X (2018) Automatic brain tumorsegmentation by exploring the multi-modalitycomplementary information and cascaded 3dlightweight cnns. In: International MICCAI Brain-lesion Workshop, Springer, pp 25–3688. Maier O, Wilms M, Handels H (2015) Image fea-tures for brain lesion segmentation using randomforests. In: BrainLes 2015, Springer, pp 119–130 89. Marcinkiewicz M, Nalepa J, Lorenzo PR, DudzikW, Mrukwa G (2018) Segmenting brain tumorsfrom mri using cascaded multi-modal u-nets.In: International MICCAI Brainlesion Workshop,Springer, pp 13–2490. McKinley R, Wepfer R, Gundersen T, WagnerF, Chan A, Wiest R, Reyes M (2016) Nabla-net: A deep dag-like convolutional architecturefor biomedical image segmentation. In: Interna-tional Workshop on Brainlesion: Glioma, Multi-ple Sclerosis, Stroke and Traumatic Brain Injuries,Springer, pp 119–12891. McKinley R, Jungo A, Wiest R, Reyes M (2017)Pooling-free fully convolutional networks withdense skip connections for semantic segmentation,with application to brain tumor segmentation.In: International MICCAI Brainlesion Workshop,Springer, pp 169–17792. McKinley R, Meier R, Wiest R (2018) Ensemblesof densely-connected cnns with label-uncertaintyfor brain tumor segmentation. In: InternationalMICCAI Brainlesion Workshop, Springer, pp 456–46593. Media H (2004 (accessed April 10, 2020)) CTScan vs. MRI. URL
94. Mehta R, Arbel T (2018) 3d u-net for braintumour segmentation. In: International MICCAIBrainlesion Workshop, Springer, pp 254–26695. Meier R, Bauer S, Slotboom J, Wiest R, Reyes M(2013) A hybrid model for multimodal brain tu-mor segmentation. Multimodal Brain Tumor Seg-mentation 31:31–3796. Meier R, Bauer S, Slotboom J, Wiest R, ReyesM (2014) Appearance-and context-sensitive fea-tures for brain tumor segmentation. Proceedingsof MICCAI BRATS Challenge pp 020–02697. Meier R, Karamitsou V, Habegger S, Wiest R,Reyes M (2015) Parameter learning for crf-basedtissue segmentation of brain tumors. In: BrainLes2015, Springer, pp 156–16798. Meier R, Knecht U, Wiest R, Reyes M (2016)Crf-based brain tumor segmentation: alleviatingthe shrinking bias. In: International workshop onbrainlesion: glioma, multiple sclerosis, stroke andtraumatic brain injuries, Springer, pp 100–10799. Menze BH, Geremia E, Ayache N, Szekely G(2012) Segmenting glioma in multi-modal imagesusing a generative-discriminative model for brainlesion segmentation. Proc MICCAI-BRATS (Mul-timodal Brain Tumor Segmentation Challenge) 8100. Menze BH, Jakab A, Bauer S, Kalpathy-CramerJ, Farahani K, Kirby J, Burren Y, Porz N,
Survey and Analysis on Automated Glioma Brain Tumor Segmentation and Overall Patient Survival Prediction 39
Slotboom J, Wiest R, et al. (2014) The multi-modal brain tumor image segmentation bench-mark (brats). IEEE transactions on medical imag-ing 34(10):1993–2024101. Myronenko A (2018) 3d mri brain tumor seg-mentation using autoencoder regularization. In:International MICCAI Brainlesion Workshop,Springer, pp 311–320102. Nuechterlein N, Mehta S (2019) 3d-espnet withpyramidal refinement for volumetric brain tu-mor image segmentation. brainles 2018. SpringerLNCS 11384:245–253103. Nvidia (2020 (accessed April 28, 2020)) Nvidia.URL