A Comprehensive Review of Computer-aided Whole-slide Image Analysis: from Datasets to Feature Extraction, Segmentation, Classification, and Detection Approaches
Chen Li, Xintong Li, Md Rahaman, Xiaoyan Li, Hongzan Sun, Hong Zhang, Yong Zhang, Xiaoqi Li, Jian Wu, Yudong Yao, Marcin Grzegorzek
NNoname manuscript No. (will be inserted by the editor)
A Comprehensive Review of Computer-aidedWhole-slide Image Analysis: from Datasets toFeature Extraction, Segmentation, Classification andDetection Approaches
Chen Li · Xintong Li · Md Rahaman · Xiaoyan Li · Hongzan Sun · Hong Zhang · Yong Zhang · Xiaoqi Li · Jian Wu · Yudong Yao · Marcin Grzegorzek
Received: date / Accepted: date
Abstract
With the development of computer-aided diagnosis (CAD) and imagescanning technology,
Whole-slide Image (WSI) scanners are widely used in thefield of pathological diagnosis. Therefore, WSI analysis has become the key tomodern digital pathology. Since 2004, WSI has been used more and more in CAD.Since machine vision methods are usually based on semi-automatic or fully auto-matic computers, they are highly efficient and labor-saving. The combination ofWSI and CAD technologies for segmentation, classification, and detection helpshistopathologists obtain more stable and quantitative analysis results, save laborcosts and improve diagnosis objectivity. This paper reviews the methods of WSIanalysis based on machine learning. Firstly, the development status of WSI andCAD methods are introduced. Secondly, we discuss publicly available WSI datasetsand evaluation metrics for segmentation, classification, and detection tasks. Then,the latest development of machine learning in WSI segmentation, classification,and detection are reviewed continuously. Finally, the existing methods are stud-ied, the applicability of the analysis methods are analyzed, and the applicationprospects of the analysis methods in this field are forecasted.
Keywords
Whole-slide image analysis · computer-aided diagnosis · featureextraction · image segmentation · image classification · object detection Chen Li, Xintong Li, Md Rahaman, Xiaoqi Li and Jian WuMicroscopic Image and Medical Image Analysis Group, College of Medicine and BiologicalInformation Engineering, Northeastern University, 110169, Shenyang, PR ChinaChen Li E-mail: [email protected] Li, Hongzan Sun, Hong Zhang and Yong ZhangChina Medical University, 110122, Shenyang, ChinaYudong YaoDepartment of Electrical and Computer Engineering, Stevens Institute of Technology, Hobo-ken, NJ 07030, USAMarcin GrzegorzekInstitute of Medical Informatics, University of Luebeck, Luebeck, Germany a r X i v : . [ c s . C V ] F e b Chen Li et al.
Whole-slide Image (WSI), also generally mention as “virtual microscopy”, pur-poses to imitate typical light microscopy in a computer-generated model [1]. Peopleusually think of whole-slide imaging as an image acquisition method. It is possibleto transform the whole glass slide into a digital form [2]. Furthermore, the “digitalslides” are used for humans observation or performing them to automated imageanalysis [3].The processing of whole-slide imaging is performed by the WSI system. AWSI system has a scanner, networked computer(s), and possibly a server or cloudsolution for storage, display (e.g. tablet, etc.), and compatible software for imagecreation, management, and analysis [4] [5] [6]. The first part applies technical hard-ware (scanner) to digitize glass slides, generates a sizable classical digital image(so-called “digital slide”) accordingly. The second part exploits technical software(ie, virtual slide viewer) to view and/or analyze these huge digital images [7].WSI devices have different looks and performance, but overall, the WSI scan-ner includes the following parts: an optical microscope system with a camera, anacquisition system, computer hardwares/softwares, scanning softwares, and a dig-ital slide viewer. Supplemental components include the feeder or image processingsystems [1]. (1) (2) (3) (4)
Fig. 1: The workflow of the proposed whole-slide imaging. (1) is the glass slide [8].(2) whole mount glass slides. (3) whole slide imaging scanner. (4) digital wholeimage [1].As shown in Fig. 1, (1) is the pathologic biopsy, (2) is the whole mount glassslides, (3) is whole slide imaging scanner, (4) is the obtained WSIs. The optical microscope system is the essential part of the WSI scanner, especially the lensoptics and the camera because it can determine the quality of the images. Chargedcoupled device (CCD) sensors on cameras that can convert analog signals intodigital signals. There are two major methods of slide acquisition. One is areascanning, the other is line scanning. The area scanner moves on the sample block itle Suppressed Due to Excessive Length 3 by block and section by section, that is, after stopping at each position to capturean image, it is repositioned to the next position. The line scanner is smooth andcontinuous movement and fast scanning [9]. After choosing a range of interest on aslide, adjust focus, and scan the slide [8]. If WSI scanners have a Z-stacking facility(scan slides at different focal planes along the vertical Z-axis and stack images ontop of each other to produce composite multi-planar images [1]), they can bettercenter on particular areas of interest [4]. Owing to the images generated by theWSI systems are large, the visual field of a computer should be bigger than thevisual field of a traditional microscope over four times [10].With the accelerating development of science and technology, the WSI systemhas progressed rapidly. WSI offers higher quality and resolution images with anno-tation [3]. The scanner with fast scanning speed has improved image quality andreduced storage costs [2]. The digital approach also can reduce the time of trans-porting glass slides and the risk of breakage and fading [11] [12] [13]. Moreover,the digital slides do not deteriorate over time [5].WSI infuses into many fields such as E-education, virtual workshops, andpathology aspect. Now, there is a growing need for pathology to improve qual-ity, patient safety, and diagnostic accuracy. These causes and economic pressuresto consolidate and centralize diagnostic services [11]. Moreover, WSI can boost dis-tinct pathology practices, so it is generally used in pathology [14]. Digital pathologynetworks based on WSI systems can solve some difficult problems with pathology.For example, WSI can be explored by several observers from different areas at thesame time. Discussions using WSI can save the time needed for transferring glassslides to distant places for attaining second minds and teleconsultation [2] [13] [15].WSI equivalently broadens the scope of cytopathology where virtual slides are usedfor numerous intents like telecytology, quality activities (e.g. archiving and profi-ciency testing), and education (e.g. virtual atlases) [4]. It will also let pathologistsbecome more efficient, precise, and creative at quantifying prognostic biomarkerslike HER2/neu (c-erbB-2). But also, crucially, WSI develops CAD in combinationwith the continually developing computer artificial intelligence, big data, and cloudtechnology. Nowadays, WSI technology is very advance and offers the pathologycommunity novel clinical, nonclinical, and research image-related applications [1].1.2 The Development of WSI AnalysisThe traditional pathological section analysis method requires specially trainedpathologists to look for areas of interest under the microscope one by one, and thenanalyze and diagnose based on professional knowledge. Traditional manual analysisof pathological images has many drawbacks and problems. There are no quanti-tative indicators, so the qualitative analysis results cannot be reproduced [16].Moreover, most doctors have tight working conditions, heavy workload, and timepressure. In this case, the human cognitive process is easily disturbed, leading to incomplete diagnosis and misdiagnosis [17]. Although traditional slide analysis isaccurate, it can be deeply personal. It is available for the same person to evaluatea slide one day and to get different conclusions the following week. Besides, theprocedure is a challenging and time-consuming task [9]. Therefore, CAD is a moreefficient, accurate, and intuitive method.
Chen Li et al.
The computer-aided reading slide can help pathologists improve diagnosis ac-curacy and detection rate and reduce the overall misdiagnosis rate. Moreover, thecomputer is not affected by fatigue and human error and provides better assis-tance to doctors [17] [18]. It is also a valuable tool to reduce the workload ofclinicians [19]. While reducing pathologists’ workload and improving efficiency, itcan also perform intuitive quantitative analysis of pathological conditions. Theseare better than manual reading slides. Computer-aided viewing of WSI is nowrapidly developing. WSI provides the pathology field unique clinical, nonclinical,and analysis of image-related applications [1].In recent years, the pathological WSI analysis performed by CAD doctors, ithas been widely used in different cancer fields (ie, breast cancer, prostate cancer,gastric cancer, neuroblastoma). The scope of applications focuses on disease clas-sification, early screening, tissue localization, and benign and malignant diagnosis.Common tasks with CAD include classification, segmentation, and detection.For example, in the work of [20], automatic detection and sequencing systembased on Gleason pattern recognition is proposed for the automatic detection ofhigh-grade prostate cancer. In the field of breast cancer, the work of [21] makesthe segmentation of WSI images of breast biopsy with biologically significant tis-sue markers. The study of [22] trains a modified version of the residual network(ResNet) to classify different types of colorectal polyps on WSIs. At present, thedevelopment trend of computer-aided viewing of WSIs is shown in Fig. 2. T o t a l N u m b e r o f R e l a t e d W o r k s Year
All Classification Segmentation Detection Other
Fig. 2: The development trend of computer-aided viewing of WSIs. The horizontaldirection shows the time. The vertical direction shows the cumulative number ofrelated works in each year. itle Suppressed Due to Excessive Length 5
As shown in Fig. 2, as the years are getting closer and closer, technologycontinues to advance. There are more and more cases of using computers to assistdiagnosis. The number of cases in the three main applications of classification,segmentation, and detection has increased year by year. The number of cases inother applications are growing, such as retrieval [23], localization [24]. Beginningin 2008, CAD viewing WSI has helped pathologists begin to realize it significantly.Since 2014, there has been an increasing trend in the number of computer-aidedpathologists diagnosed with WSI. Gradually by 2020, the growth rate of CAD hasincreased, reflecting the vigorous development of this technology.Besides, to explain and clarify computer-aided pathologists’ work context inviewing WSI, an organization chart is shown in Fig. 3. The figure shows the gen-eral process of CAD and processing WSI. It shows seven important steps in thehistopathology image analysis system, including data acquisition, image presenta-tion, image preprocessing, feature extraction, data post-processing, classifier de-sign, and system evaluation.
Medical Demands Computer-aided ApproachesHistopathological Data Acquisition Histopathological image Image Pre-processing Feature Extraction Post-processing Classifier Design System Evaluation texture featurecolor featureshape featurelocal featuresspatial featuresenergy featureGlobal featuresManual craft featuresMachine learnt featuresbackground region detectionPCAColor Normalizationdataset augmentationdeconvolution ROI selection down-sampleColor space conversion segmentation down-samplePCARemove regionsfilterMorphological processing breast WSIProstate WSIGlioblastoma WSINeuroblastoma WSI KNNSVMK-meansNaive Bayes classifierRandom Forest classifierlogistic regressionShallow learning based classifier Classification accuracyClassification varianceClassification error ratePrecision Recall DiceROC and AUCSensitivity and specificityF1 score and 95% confidence intervalsPhysical examinationMicroscope selectionPreparation optionsSection preparationSpecimen receptionfixationembeddingsectioningstaining Colon Cancer WSIlung cancer WSIGastric WSIliver cancer WSI Deep learning based classifier … … … … … … …
Fig. 3: The organization chart of histopathology WSI analysis using computer-aided analysis approaches in this paper.In Fig. 3, histopathological data initially obtained from the medical field, 2-Dor 3-D digital microscopic images are first captured by various imaging equipment(e.g., optical microscopy), and then saved in a specific color space (e.g., Red, Greenand Blue (RGB) color space). The 3rd step is the image pre-processing step, theproperties of images are improved by dataset augmentation, segmentation, and so on, which is an importation preparation for the feature extraction step. In thenext step, feature extraction is implemented, where the image can be representedby its attributes (shape, texture, and color features), or layout features (globaland local features), or extraction style (manual or automatic). These feature ex-traction categories are not separate, but can be converted into other categories
Chen Li et al. using appropriate methods. After that, the post-processing step takes the respon-sibility to enhance the extracted features, where filter, morphological processing,normalization are always used. Also, the classifier can be classified as shallow ordeep according to its learning structure. Finally, various numerical and intuitivemethods are used to evaluate the classification system, such as classification ac-curacy, classification error rate, sensitivity, and specificity. Besides, each step isnot independent, but is closely connected with other steps through informationfeedback. Therefore, the entire CAD viewing WSI system is an organic whole [25].1.3 Motivation of This ReviewNow WSI technology has applications in many fields. For example, to perform pre-liminary diagnosis of surgical pathology, and perform intraoperative frozen sectiondiagnosis through remote consultation [26] [27], and seek expert advice withoutincurring international transportation costs or delays [28]. WSI also provides ad-vantages in tumor diagnosis, prognosis, and targeted therapy. It can also facilitateteachers and students in teaching [11]. Therefore, the research field of WSI analysisthrough CAD systems is significant. To the best of our knowledge, there exist somesurvey papers that summarize WSI analysis (e.g., the reviews in [3], [29–39]). Inthe following part, the summary of survey papers related to the WSI analysis ispresented.The survey of [3] reviews the current status of WSI pathology, including su-pervision and verification, remote and routine pathological diagnosis, educationaluse, implementation issues, and cost-benefit analysis of WSI in routine clinicalpractice. However, this article only focuses on the application of CAD systems inWSI analysis. This review rarely mentions this, and only 12 references are aboutWSI.The survey of [29] reviews the latest CAD techniques for digital histopathology.This article also briefly introduces new image analysis technologies developed andapplied in the United States and Europe for some specific histopathology-relatedproblems. More than 130 papers on CAD have been summarized and only threearticles are about WSI.The survey of [30] reviews the WSI informatics method of histopathology, re-lated challenges and future research opportunities. However, this article reviewsimage quality control, feature extraction of image attributes captured at pixels,object, and semantic levels, image features for predictive modeling, and data andinformation visualization for diagnostic or predictive applications. It does not dis-cuss the entire process of CAD and the viewing of WSI. More than 130 papershave been summarized. However, only three articles are about WSI.The survey of [31] reviews the analysis methods of histopathological imagesof breast cancer, introduces the process of tissue preparation, staining, and slide digitization, and then discusses different image processing techniques and appli-cations, from tissue staining analysis to CAD, and the prognosis of breast cancerpatients. Although the histopathological images discussed in the article are WSIs,they are only about breast cancer and not comprehensive. More than 110 papershave been summarized. However, only four articles are about WSI. itle Suppressed Due to Excessive Length 7
The survey of [32] provides a comprehensive overview of the graph-based meth-ods explored so far in digital histopathology. More than 170 papers have beensummarized. However, only four articles are about WSI.The survey of [33] reviews the latest methods of large-scale medical image anal-ysis, which are mainly based on computer vision, machine learning, and informa-tion retrieval. Then, they comprehensively reviewed the algorithms and technolo-gies related to the main processes in the pipeline, including feature representation,feature indexing, and search. However, WSI appears only in the sample dataset,and no actual analysis is performed. Of the more than 250 papers summarized inthis paper, only three mention WSI.The survey of [34] introduces the application of digital pathological imageanalysis using machine learning algorithms, solve some specific analysis problems,and propose possible solutions. However, there are only 11 articles related to WSIon the topics we are interested in. More than 120 papers have been summarized.But only 11 articles are about WSI.The survey of [35] introduces the general situation of artificial intelligence, abrief history in the medical field and the latest developments in pathology, andthe future prospects of pathology driven by it. This review only briefly mentionsWSI in the part of the pathology application imaging and example datasets. Ofthe more than 70 papers summarized in this paper, only four mention WSI.The survey of [39] introduces the technical aspects of WSI, its application indiagnostic pathology, training and research, and its prospects. It highlights thebenefits, limitations, and challenges of delaying the use of this technology in dailypractice. But this article only focuses on computer-aided pathologists to view WSIand its application in diagnosis, which are not discussed in this review. Of the 50references, 20 are about WSI.From the existing review papers mentioned above, we can find that many re-searchers are concerned about the current status and development trend of WSItechnology itself, and hundreds of related works have been systematically summa-rized and discussed in those review papers. However, all these survey papers useWSI format datasets as examples only, and do not aim to introduce the detailedintroduction of computer-aided pathologists to review WSI technology. Therefore,we present this review paper to analyze all related works using CAD combinedwith WSI in the past few decades. This survey summarizes more than 210 relatedworks from 2004 to 2020. The audience for this review is related researchers in thefield of medical imaging and medical professionals.1.4 Structure of This Review
This structure of this paper is as follows: Sec. 2 summarizes the related datasetsand commonly used evaluation methods. Sec. 3 illustrates frequently used featureextraction methods. Sec. 4, 5, and 6 present the related work of segmentation,classification, and detection using WSI and CAD technology. After the overviewof different works, the most frequently used approaches are analyzed in Sec. 7.Finally, Sec. 8 concludes this review with prospective future direction.
Chen Li et al.
In this section, we have discussed some commonly used datasets and evaluationmetrics for the classification, segmentation, and detection tasks.2.1 Publicly Available Datasets about WSITo better analyze the CAD using WSI technology, we have summarized somefrequently used publicly available datasets in our study. Tab. 1 shows the necessaryinformation of these datasets. Two of the most commonly used datasets are TheCancer Genome Atlas (TCGA) [40] and the Camelyon datasets [41]. Both datasetsare often used for classification and detection. At the same time, we find two WSIdatasets named TUPAC16 [42] and Kimia Path24 [43]. TUPAC dataset is widelyused, such as mitosis detection, prediction of breast tumor proliferation, automaticscoring (classification), and so on. Kimia Path24 is often used for classificationand retrieval [43] [44]. The basic information of the common datasets are shownin Table. 1.Table 1: The basic information of the publicly available used datasets.
Databases Year Field Number of images or or size
TCGA 2006 Cancer related Over 470 TBNLST Pathology Images 2009 Lung Around 1250 H&E slidesBreakHis 2015 Breast cancer 9,109 microscopic imagesTUPAC16 2016 Tumor mitosis Around 821 H&E slidesCamelyon 2017 Breast cancer Around 3TBKimia Path24 2017 Pathology Images 24 WSIs
TCGA is a project jointly launched by The National Cancer Institute (NCI) andthe National Human Genome Research Institute (NHGRI) in 2006 [40]. It containsclinical data, genome variation, mRNA expression, miRNA expression, methyla-tion and other data of various human cancers (including subtypes of tumors). Thedatabase is designed to use high-throughput genomic analysis techniques to helppeople developing a better understanding of cancer and improve the ability to pre-vent, diagnose, and treatment [45]. While TCGA main work focuses on genomicsand clinical data, it also accumulates a large number of WSIs in patient’s tissue.
Since WSI datasets are much larger than other datasets, to facilitate viewing,David et al. [46] proposes an integrated network platform named Cancer DigitalSlide Archive (CDSA) to accommodate all WSI in TCGA. Since the dataset con-tains many types of cancers, it has a wide range of uses. Fig. 4 below is an exampleof WSIs in an adrenal cortical carcinoma in the TCGA database. itle Suppressed Due to Excessive Length 9
Fig. 4: An example of WSI in an adrenal cortical carcinoma in the TCGAdatabase [40].
The Camelyon Challenge is hosted by International Symposium on BiomedicalImaging (ISBI) [41]. The whole competition dataset (Camelyon16, Camelyon17 )are derived from sentinel lymph nodes of breast cancer patients contains WSIsof Hematoxylin and Eosin (H&E) stained node sections [47] [48]. Therefore, theCamelyon dataset is suitable for the automatic detection and classification ofbreast cancer in WSI. The data of Camelyon16 are from the Radboud Univer-sity Medical Centre and the University of Utrecht Medical Centre. The Came-lyon16 dataset is composed of 170 phase I lymph node WSIs (100 normals and 70metastatics) and 100 Phase II WSIs (60 normals and 40 metastatics), and the testdataset consisted of 130 WSIs from two universities. The Camelyon16 dataset isused as training values for the evaluation of Camelyon17. Fig. 5 is a pathologicalpicture of a lymph node in Camelyon. The left side belongs to normal cell tissue,and the right cell has been swallowed and occupied by cancer cells.
The TUPAC16 challenge is held in the context of the MICCAI [42]. TUPAC16main challenge dataset consists of 821 TCGA WSIs with two types of tumor pro-liferation data. 500 for training, 321 for testing. In addition to the main challengedataset, there are two secondary datasets (area of interest and mitotic detection).
The area of interest auxiliary dataset contains 148 cases that are randomly se-lected from the training dataset. The mitotic test dataset consisted of WSIs of73 breast cancer cases from three pathological centers. Of the 73 cases, 23 areAMIDA13 challenge [49]. The remaining 50 cases previously used to assess theinterobserver agreement for mitosis counting are from two other pathology centers
Fig. 5: A pathological image of a lymph node in Camelyon [41]. The left sidebelongs to normal cell tissue, and the right cell has been swallowed and occupiedby cancer cells.in the Netherlands. So the dataset is mainly used for automatic detection of tumormitosis or other regions of interest(ROI). Fig. 6 shows some examples of mitosismaps in H&E breast cancer slices, with green arrows marking mitosis.
Fig. 6: Some examples of mitosis diagrams in H&E breast cancer slices in theTUPAC16 [42], with green arrows marking mitosis. itle Suppressed Due to Excessive Length 11
This dataset is consciously and manually selected from 350 WSIs from differentbody parts so that the 24 WSIs clearly represented different texture patterns. Sothis dataset is more like a computer vision dataset (as opposed to a pathologydataset) because visual attention is spent on the diversity of patterns rather thanon anatomy and malignancy [43]. Therefore, this dataset is mainly used for classi-fication and retrieval of histopathological images. The 24 WSIs thumbnails in thisdataset are shown in Fig. 7.Fig. 7: 24 WSIs thumbnails in Kimia Path24 database [43].
The confusion matrix is used to observe the performance of the model in eachcategory, and the probability of each category can be calculated. The specific styleof the confusion matrix is shown in Tab. 2.Table 2: Confusion matrix of basic evaluation indexs.
Data Class Classified as Pos Classified as Neg
Pos True Positive(TP) False Negative(FN)Neg False positive(FP) True Negative(TN)
According to the confusion matrix, the True Positive Rate (TPR) can be de-fined as TP/(TP + FN), which represents the proportion of the actual positiveinstances in all positive instances of the positive class predicted by the classifier,the False Postive Rate (FPR) can be defined as FP/(FP+TN), which representsthe proportion of the actual negative instances in all negative instances of thepositive class predicted by the classifier. It can be seen that the mathematicalexpressions of the following evaluation metrics are shown in Table. 3 [50].Table 3: Evaluation metrics.
Acc , P , R , Se , Sp and F1 denote accuracy, precision,recall, sensitivity, specificity and F1 score, respectively. Assessments Formula Assessments Formula
Acc TP + TNTP + TN + FP + FN Se TPTP + FN P TPTP + FP Sp TNTN + FP R TPTP + FN F1 PRP + R Image segmentation [51] is the segmentation of images with existing targets andprecise boundaries. The commonly used indicators are accuracy, precision, recall,F-measure, sensitivity, and specificity. These metrics we have discussed in Sec. 2.2.1and their mathematical expressions are given Tab. 2. Dice co-efficient (D) andJaccard index (J) are popular segmentation evaluation indexes in recent years.Dice co-efficient (D) represents the ratio of the area intersected by two individualsto the total area, that is, the similarity between ground truth and the segmentation result graph. If the segmentation is perfect, the value is 1. Then, if S stands forthe segmentation result graph and G stands for ground truth, the expression ofDice co-efficient (D) is given in Eq. (1). D ( S, G ) = 2 | A ∩ G || A | + | G | (1) itle Suppressed Due to Excessive Length 13 Jaccard Index (J) represents the intersection ratio of two individuals, which issimilar to Dice co-efficient. The formula is given in Eq. (2). J ( S, G ) = | A ∩ G || A ∪ G | (2) Classification [52] is the operation of determining the properties of objects in theimage. In the field of digital histopathology we studied, some are the classificationof cancer [53], some are the operation of selecting ROI [54], and some are theidentification of cancer regions [55]. The purpose of classification is achieved by theconstructed classifier. The performance indicators used to evaluate these classifiersare critical to the final results. Accuracy is the most commonly used indicators toevaluate classifiers. Precision, recall, sensitivity, specificity, and F1 score are widelyused to evaluate classifiers. Accuracy,precision, recall, F-measure, sensitivity, andspecificity we have discussed in Sec. 2.2.1 and their mathematical expressions aregiven in Tab. 2. With the continuous improvement of classification requirementsin practical applications, ROC (Receiver Operating Characteristic), AUC (AreaUnder ROC Curve), a non-traditional measurement standard, have emerged. ROCis a curve drawn on a two-dimensional plane with FPR as the abscissa and TPR asthe ordinate. It can reflect the sensitivity and specificity of the continuous variablesas a comprehensive indicator. It can also solve the problem of class imbalance inthe actual dataset. AUC quantifies the area under the ROC curve into a numericalvalue to make the results more intuitive.
Detection [56] is another common task in analyzing histopathological WSIs. De-tection is not only to determine the attributes of the region identified in WSI,but also to identify and obtain more detailed results. Because of the similaritybetween testing and classification, most of the evaluation indexes are the sameas the classification, including accuracy, precision, recall, F-measure, sensitivity,and specificity that we have discussed in Sec. 2.2.1. However, in WSI detection,it is difficult to locate, determine and quantify multiple lesions. Therefore, FROC(Free Receiver Operating Characteristic Curve) [57] is proposed to evaluate thedetection results. FROC curve is a small variation of the ROC curve. It is a curvedrawn on a two-dimensional plane with FP as the horizontal coordinate and TPRas the vertical coordinate. This allows the detection of multiple lesion areas on asingle WSI. tasks. The basic commonly used evaluation indicators are accuracy, precision, re-call, sensitivity, and specificity. In terms of classification, there are comprehensiveindicators such as AUC. Dice co-efficient and Jaccard index in segmentation indi-cators have become popular in recent years, and FROC in detection indicators canbe used for positioning, qualitative and quantitative analysis of multiple lesions.
Traditional image feature extraction is generally divided into three steps: prepro-cessing, feature extraction, and feature processing. Then using machine learningmethods to segment and classify the features. The purpose of preprocessing is toeliminate interference factors and highlight characteristic information. The mainmethods are: image standardization [58] (adjust the image size); image normaliza-tion [59] (adjust the image center of gravity to 0). The main purpose of featureprocessing is to eliminate features with a small amount of information and reducethe amount of calculation. The common feature processing method is principalcomponents analysis [60].Among them, feature extraction is a crucial step. Converting input data into aset of features is called feature extraction [61]. The main goal of feature extractionis to obtain the most relevant information from the original data and representthe information in a lower-dimensional space [62]. Therefore, in this section, wemainly summarize the features extracted in WSI for CAD. The types of extractedfeatures are shown in Fig. 8.
Traditional Deep LearningFeature
Color Feature Texture Feature Shape Feature
Fig. 8: The types of feature extraction methods. itle Suppressed Due to Excessive Length 15
Color is an important feature, which is widely used for image representation [63].The color of an image is invariant to rotation, translation, or scaling. Color char-acteristics are defined according to a specific color space or model [61]. Manycolor spaces are used in the literature such as RGB [64], HSV ((Hue SaturationValue)) [65], and LAB [66]. Common color features include color histograms, colormoments, and color coherence vector(CCV) [67]. Among the papers we summa-rized, 24 papers used color features [64–66, 68–88].
RGB-based Color Features
We have found 12 studies that utilized RGB feature extraction technique [64,68, 71, 72, 74, 76–78, 80, 81, 87, 88].The color features extracted by [68] are combined with the color and entropyinformation extracted from the RGB image channel. In [64], the vector extractedwith the RGB feature indicates that the feature vector of each pixel is the localentropy of the red-green difference calculated in the square neighborhood aroundthe pixel. In [71], the mean and standard deviations are calculated as first-orderand second-order statistical features from the three RGB channels, and there aresix features. The author in [72] extracts core RGB features. The color featuresof [74] are described by the average, standard deviation, minimum and maximumvalues of the three color channels in the RGB color space in the candidate area andtwo other areas. In Fig. 10 an example of features distribution image showing thespatial distribution of the cell nuclear diameter in [80]. The work of [76] extractsthe PVS function of each R, G, and B channel (8 pixel value statistics). PVS iscomposed of the minimum, maximum, sum, average, and standard deviation of theconstituent pixels, and the lower quartile, median and upper quartile are composedof values in a specific color channel. Fig. 9 shows the RGB feature extraction andclassification in [76].Fig. 9: An image after RGB feature extraction and classification in [76]. This figure corresponds to Fig.2 (2) in the original paper.The author in [77] extracts the variance in each color channel of RGB: s2R,s2G, s2B, the variance (maximum value) between the peaks of each color channels2. The work of [78] uses color saturation and RGB color. [80] extracts color features from the RGB channels. Fig. 10 shows an example of feature distributionof the cell nuclear in diameter [80].Fig. 10: An example of feature distribution image in [80]. This figure correspondsto Fig.10 in the original paper.The author in [81] extracts RGB histogram, and overlay features. [87] extractsthe first-order statistics of 14 color channels, and the color histogram of eachRGB channel is 8 ×
38 bin histogram. In [88], for each 2D superpixel (for example,grayscale superpixel), two statistics are calculated of mean and standard deviationof the pixel value. For 3D superpixels (such as RGB superpixels), eight statisticsare calculated the mean and standard deviation of the pixel value for each colorchannel and each RGB superpixel, then this as a color function.
HSV-based Color Features
There are four papers on color feature extraction based on HSV [69,71, 76,78].In [69], the extracted color feature is the hue channel converted from the HSVcolor space of the original image. In [71], from the three HSV channels, the averageand standard deviation are calculated as first-order and second-order statisticalfeatures, and a total of six features are extracted. [76] extracts the PVS functionof each H, S and V channel. [78] uses color saturation and value and RGB coloras a function in HSV color space.
LAB-based Color Features
There are five papers on color feature extraction based on LAB [65,68,71,79,86].In [68], the extracted color features are composed of color and entropy in-formation extracted from LAB image channels. In [71], from the three channelsof CIELAB, the average and standard deviation are calculated as first-order and itle Suppressed Due to Excessive Length 17 second-order statistical features, and a total of six features are extracted. [79]extracted the color histogram calculated in LAB space. Fig. 11 shows the colorhistogram mentioned in the paper. Cutting the WSI into a patch is a normal oper-ation in the image processing process. (a,b,c,f,g,h) in Fig. 11 are the image blocksafter the WSI slice.Fig. 11: (a) Original 120 ×
120 pixel patch, (b) deconvolved color channel thatshows good contrast for nuclei dyed with haematoxylin, (c) deconvolved colorchannel that shows good contrast for eosin dye, (d) LBP histogram calculated onhaematoxylin channel, (e) LBP histogram calculated on eosin channel,(f, g, h) L,a, and b channels of the image in LAB color space,(i, j, k) color histograms of L,A and B channels. At the end, LBP histograms on two channels are concatenatedto produce the first set of features, color histograms on LAB channels are concate-nated to produce the second set of features. These figure are from [79]. This figurecorresponds to Fig.3 in the original paper.The color feature of [65] is the LAB histogram of the color. [86] uses the colorhistogram calculated for each channel in the CIE-Lab space as the color feature.
Others Color Features
Other papers related to color feature extraction total of six [73, 75, 77, 82–85].[73] and [75] extract the color information in WSIs as features. [77] extracts theaverage value of the variance, saturation, brightness of HSI and the hue value of thecolor model µ . [82] extracts the color channel histogram as the color feature. [83] extracts the histogram of the three-channel HSD color model as color features. [85]extracts the color information in WSI as features. [84] extracts rough color features.The rough feature is the use of feature analysis of the diversity of rough areasin WSI to roughly characterize their shape, color, and texture. The fine featurerefers to the more comprehensive image features extracted from the slice to expressdeeper features. The feature histogram extracts in [84] is shown in Fig. 12.Fig. 12: Histogram distribution of features used in Elastic Net models, showingthe number of models in which features of a given class appear. The number offeatures is normalized based on the total number of features represented for eachclass. Bar patterns represent the feature class. Most features for each class appearin all, or nearly all models, as expected if they have diagnostic value. This figureis from [84]. This figure corresponds to Fig.7 in the original paper.From the above review, we can see that in terms of color feature extraction,the RGB color features are used most frequently, focusing on the period from 2009to 2018. Then HSV color feature and LAB (CIE-LAB) color feature. The datesare 2012 to 2014 and 2009 to 2017. In other papers, HSI and HSD color modelsare used as color characteristics. The texture feature describes the surface properties of the object corresponding to the image or image area. Unlike the color feature, the texture feature is notbased on the feature of pixels. It needs to be calculated in the area containingmultiple pixels. Texture feature is an effective method when judging images withlarge differences in thickness and density. However, when the thickness, density,feature are easy to distinguish between the information. However, difficult for the itle Suppressed Due to Excessive Length 19 usual texture features to accurately reflect the differences between the textureswith different human visual perception.Commonly used texture information description methods are: statistical meth-ods ( such as gray-level co-occurrence matrix (GLCM) [89]), geometric methods(such as voronoi checkerboard feature method [90]), model methods (such as ran-dom fields [91]), and signal processing methods (such as wavelet transform [92]).Among the papers we summarized, 57 papers used texture features [47, 53, 55, 65,66, 68, 71, 73–77, 79, 82–86, 88, 93–128].
Local Binary Pattern-based Texture Features
There are 14 papers on texture feature extraction based on local binary pattern(LBP) [43,47,65,66,76,79,86,94,96,97,100,107,118,127]. In [43,65,66,76,86,94,96,97,100,107], texture histograms of LBP features are extracted as texture features.Fig. 13 shows the calculation of the LBP feature of a given pixel in [96].Fig. 13: (a) The conventional LBP operator; (b) Circular pattern used to computerotation invariant uniform patterns. This figure corresponds to Fig.5 in [96].Texture features compute using LBP for small image patches are extractedfrom [79]. In [118], multi-resolution LBP is extracted as the texture feature. In [47],different structural texture features are extracted, including LBP features. In[127], LBP, MRC LBP feature are extracted as texture features.
Haralick-based Texture Features
Five papers involve texture feature extraction based on Haralick [68,93,94,119,121]. In [93], 4 Haralick features are the most suitable for discriminating stromafrom PCA. Haralick features are extracted as texture features in [68, 94, 119].In [121], extracts 57 subcellular location features, including Haralick texture fea-tures and DNA overlapping the features (experiments). The experimental imagesin [119] and the extracted haralick features are shown in Fig. 14.
GLCM-based Texture Features
Eight papers involve texture feature extraction based on GLCM [55, 77, 82, 88, 95,98, 104, 114].In [95], the extraction of the mean and variance of the range of values within thelocal neighborhoods and entropy and homogeneity of co-occurrence histograms astexture features. Co-occurrence features are extracted as texture features from [55] ( a ) ( b )( c ) ( d ) Fig. 14: (a, b) is the experimental image, (c, d) is the Haralick intensity texture.This figure corresponds to Fig.2 in [119].and [98]. 16 features such as mean and variance are combined to form a groupof 18 features [104]. In [77], co-occurrence matrix, correlation, and energy areextracted as texture features. In [82], GLCM features are extracted. In [114], co-occurrence matrix statistics are extracted for each hyperpixel. In [88], 2D hyper-pixel texture is obtained by using the spatial grayscale symbiosis matrix and 1pxdisplacement vector of 3D hyper-pixel. From the co-occurrence matrix, the secondmoment of angle, contrast, correlation, sum of squares, deficit moment, average,sum variance, sum entropy, entropy, difference variance, difference entropy, andcorrelation information measures 1 and 2 are calculated. The average value of the13 parameters obtained is the characteristic descriptor.
Filter and Scale-invariant Feature Transform(SIFT)-based Texture Features
There are five related papers on filter and SIFT based texture feature extrac-tion [47, 102, 113, 114, 122].In [102], first-order statistics, second-order statistics, and gabor filter featuresare used as texture features. In [113], a Gauss-like texture filter is applied toextract texture features. Fig. 15 shows the uniform distribution of histogram filterresponse in [113].In [114], the gray histogram statistics extracted from the filter bank responsefor each hyperpixel. Different structural texture features, such as SIFT features, are extracted from [47]. The vlfeat implementation of MSER (Maximally StableExtremal Regions) and SIFT is used by extracting from [122].
Others Texture Features itle Suppressed Due to Excessive Length 21
Fig. 15: Patches of three different tissue types (in RGB) corresponding to (a) fat,(b) stroma, and (c) epithelial cells -top- along with their normalized luminancechannel images (after converting them from RGB to LAB) -middle-, and maximalfilter responses after convolving Gaussian-like filters at all directions of onescale(( σ x, σ y) = (1,3)) -bottom-, and plot showing their histogram of filter responsemagnitudes. This figure corresponds to Fig.3 in [113].There are a total of 24 related papers based on the extraction of other texturefeatures [53, 71, 73, 75, 83–85, 99, 101, 103, 105, 106, 109–112, 117, 118, 123–126, 128].Texture Parameters: DNM1, DNM2,DNM3, DNM1-2, DNM1-3, DNM2-3, DNM2-2-3, DT,DN are extracted from [53]. The feature vector extracted in [53] is shownin Fig. 16.Fig. 16: Tissue microtextures identified using image processing. The first threecolumns show examples of cell nuclei belonging to nucleus morphology categoriesNM1, NM2, and NM3, respectively. The textures identified as ECM and AT rep-resent, respectively the collagen-rich stroma, and fat and tissue-devoid regions of the slide. This figure corresponds to Fig.2 in [53].The texture is applied to the cytoplasmic region around the nucleus in [99].In [101], texture features are used to identify areas with high or low intensityvariability in the image. Average, standard deviation, contrast, correlation, energy, entropy, and uniformity are extracted from [71] as texture features. Texton-basedtexture is extracted from [103]. In [73], texture information is used as feature.In [75], quantitative image features are extracted to capture its texture. Nucleartexture features are extracted from the chromatin content and distribution in [105].Each area is tagged according to its texture description in [106]. Intensity on thebasis of histograms of the sum and difference images are extracted as texturefeatures in [109]. In [83], texture features are extracted for classification. In [110],the texture feature is extracted from the cell image and compressed into a binarycode. These compressed features are stored in a hash table that allows constanttime access across many images. In [111], the texture features of each nucleus areextracted. In [112], the area is rendered using a manually positioned texture unit.The Fig. 17 shows the procedural structure and texture rendered in [112].Fig. 17: Procedural structure and texture: (a) Procedural 3D model of a lobu-lar epithelial layer. (b) Slice of (a) showing 2D lobular patterns. (c) Proceduralcell rendering (random layout, arbitrary colors). This figure corresponds to Fig.5in [112].[84], use the riesz texture features. The texture features of slides stained withKi67 are extracted from [117]. The author of [118] extracts Histograms of OrientedGradients (HOG), and Fisher Vectors (FV) as texture features. The kernel texturefeatures are extracted in [85] and [120]. In [123], texture features are extractedfrom the ROI. In [124], the texture of the subregion is extracted for statisticalanalysis and classification. In [125], the standard deviation (variance) within thearea defined by the contour is used as the texture feature. [126], extract the first-and second-order texture features. A total of 166 texture features are extractedfrom the convolved hematoxylin (nuclear staining) channel in [128]. The shape feature is just what the name suggests. Under normal circumstances,there are two ways to represent shape features. One is contour features, and the other is regional features. The contour feature of the image is mainly for the outerboundary of the image, and the regional feature of the image is related to the entireshape area [129]. Commonly used shape feature extraction methods include bound-ary feature method (such as hough transform method [130]), geometric parametermethod (such as moment, area, circumference [131]), fourier descriptors [132], and itle Suppressed Due to Excessive Length 23 other methods. There are 15 papers that use shape features among the papers wehave summarized [70, 72, 74, 75, 84, 93, 99, 101, 105, 111, 120, 124, 133–135].
Basic Geometric Parameter-based Shape Feature
Among the papers that used shape feature extraction, seven papers extractedbasic geometric shape features [70,72,74,101,105,133,134]. The feature extractionsteps in six of the papers are all used to classify, segment, or detect the task beforeit is used to better represent the image. In [133], the major axis length to minoraxis length ratio of a best-fit ellipse is extracted as the shape feature to eliminatefalse regions. In [70], eosinophilic-object shape features (pixel area, elliptical area,major-minor axes lengths, eccentricity, boundary fractal, bending energy, convexhull area, solidity, perimeter, and count) are extracted. The author in [134] ex-tracts two morphometric features, the mean nuclear area and standard deviationof the nuclear area, using a fully automatic segmentation method on WSIs. Theauthor in [72] extracts basic morphologic features and calculates its odds ratio formalignant tumors. The author in [74] extracts compactness, eccentricity, firmness,and sphericity as shape features. The author in [105] extractes perimeter, eccen-tricity, circularity, major axis length, minor axis length as geometric shape feature.Fig. 18 shows the morphological characteristic spectrum of the image in [105]. Theseventh paper [101] is the morphological feature extraction for post-processing.Fig. 18: Image analysis enables the reliable and objective characterization of tis-sues through a process of image normalization, image segmentation, and featureextraction.
Other Shape Features
In the early days, some existing library functions and third-party existing func- tions were usually used to directly extract features. In [93,99,120,124], the nuclearmorphological features of the nuclei in WSI are extracted as the shape features ofthe images.Over time, many other advanced extraction methods have emerged. There arealso four papers on other shape feature extraction [75, 84,111,135]. [135] extractes multiple sharpness features. In [75], the author extractes 461 quantitative im-age features capturing the texture, color, shape, and topological properties of ahistopathological image. [111] extracts precise quantitative morphometric features.There are shape features in the core feature group extracted by [84].3.2 Deep Learning Feature ExtractionConvolutional Neural Network (CNN) is widely used to extract the deep learningfeatures in various WSI analysis tasks. In the papers, a total of 53 papers usedCNN for deep learning feature extraction [22, 44, 136–184].The basic configuration of CNN is the convolutional layer, pooling layer andfully connected layer [185]. These three layers can be stacked. Take the input ofthe previous layer as the output of the next layer, and finally get N feature maps[186] with very low dimensions. Because it is an end-to-end learning model, it canlearn more fully and extract features better [187]. The convolutional layer acts asa feature extractor, and the neurons in the convolutional layer are arranged intofeature maps. Since different feature maps in the same convolution have differentweights, N features can be extracted at each position [188] [189]. Deep Learning Features of the VGG Series
In CNN, several classical improved network structures are often applied toextract deep features on WSI. VGGNet is an improvement based on the originalframework of [190]. The full name of VGG is Visual Geometry Group, whichbelongs to the Department of Science and Engineering of Oxford University. Itcan be applied to face recognition, image classification, etc. VGGNet increases thenetwork depth by adding more convolutional layers and fixing other parametersof the network framework [191]. All layers use 3 × Fig. 19: The VGG-16 network produces a high-dimensional feature vector for eachindividual tile from an input image. This figure corresponds to Fig.1 in [156]. itle Suppressed Due to Excessive Length 25
Deep Learning Features of the ResNet Series
ResNet is another widely used CNN structure. The full name of ResNet isResidual Network and the proponent is Balduzzi D. ResNet has pushed deep learn-ing to a new level, reducing the error rate to a level lower than that of humansfor the first time. The residual module in ResNet makes the network deeper, butwith lower complexity. It also makes the network easier to optimize and solvesthe problem of vanishing gradient [192]. The bottlen neck layer in ResNet uses 11networks, which expands the dimension of the featuremap and greatly reduces theamount of calculation [193]. Among the papers we reviewed, the papers that useResNet to extract deep learning features include [22,150,161,164,165,167,169,181].
Deep Learning Features of the U-net Series
The full name of U-net is Unity Networking. it is a network architecture estab-lished to solve the problem of medical image segmentation. This structure is basedon FCN (Fully Convolutional Neural Network). It adds an upsampling stage andadds many feature channels, allowing more original image texture information inhigh-resolution layers, using valid for convolution throughout, ensuring that theresults obtained are based on no missing context features [194]. In the paper wesummarized, the use of U-net for deep learning feature extraction have [195–198].
Other Deep Learning Feature
There are other improved structures based on CNN, such as GoogLeNet [138].The Google Academic team carefully prepares GoogLeNet to participate in theILSVRC 2014 competition. The main idea is to approximate the optimal sparsestructure by building a dense block structure to improve performance withoutincreasing the amount of calculation. The initial version of GoogLeNet appearedin [199].There are also other improved structures based on Recurrent Neural Networks(RNN) [174] and LSTM [158], which are used to extract deep learning features.RNN appeared in the 1980s, and its prototype has seen in the Hopfield neuralnetwork model proposed by American physicists in 1982 [200]. RNN has a strongprocessing ability for variable length sequence data. Therefore, it is very effectivefor data with time-series characteristics and can mine time series information andlanguage information in the data. The Long Short-Term Memory (LSTM) modelappeared because of the drawbacks of RNN, and LSTM can solve the problem ofgradient disappearance in RNN [201].3.3 Summary
It can be seen from the content we reviewed above, in the traditional featureextraction, color feature, texture feature, and shape feature are the three mostcommonly used features. The texture feature is the most used. In the papers wesummarized, from 2004 to 2019, a total of 51 papers used texture features. Thesecond is color features, which are generally based on the three color spaces ofRGB, HSV, and LAB. Among them, the RGB color space is the most commonly used. The least applied is the shape feature. For more details and analysis in thisregard, see the detailed introduction in the following chapters.Over times, the level of science and technology has also continuously improved.As can be seen from the papers we summarized, since 2016, deep learning featureshave been gradually applied to this day. The specific deep learning network ar-chitecture will be introduced in a separate method analysis later. Table. 4 is asummary of the CAD methods used for feature extraction in WSI. itle Suppressed Due to Excessive Length 27
Table 4: Summary of the CAD methods used for feature extraction in WSI (Tra-ditional (T)).
Feature TypeMethodReferenceYear Team DetailsT Color [68] 2009 J Kong, O Sertel Combined Color and entropyinformation extracted from RGBand LAB image channelsT Color [64] 2010 V. Roullier RGB feature vectorT Color [69] 2012 Siddharth Samsi Hue channel for HSV color space conversionT Color [70] 2012 Sonal Kothari RGB, CIELAB, HSV, mean and standarddeviation in each channel in three spacesT Color [71] 2012 Hatice Cinar Akakin Nuclear RGB featuresT Color [72] 2013 Brian T. Collins color informationT Color [73] 2013 Nandita Nayak The average, standard deviation,minimum and maximum values ofeach color channelin the RGB spaceT Color [74] 2013 M. Veta \ T Color [75] 2013 Kothari S PVS for each R, G, B and H, S, V channelT Color [76] 2013 Andr´e Homeyer Variance of RGB, /mu hue of HIST Color [77] 2013 Hazem Hiary RGB color and color saturationand value in HSV spaceT Color [78] 2014 Pinky A. Bautista RGB color and color saturationand value in HSV space RGBT Color [79] 2014 Ezgi Mercan Lab space color histogramT Color [80] 2014 Fang-Cheng Yeh RGB color channelsT Color [81] 2015 Litjens, G RGB histogram featuresT Color [82] 2015 Michaela Weingant color channel histogramT Color [83] 2015 Ruoyu Li Histogram of the three-channelHSD color modelT Color [65] 2016 Mercan E LAB histograms for colorT Color [84] 2016 Barker J \ T Color [66] 2016 Mercan, C LAB histograms for ColorT Color [85] 2016 Brieu N, Pauly O \ T Color [86] 2017 Caner Mercan color histogram of each channel in CIE-LAB spaceT Color [87] 2018 Angel Cruz-Roa color Histograms 8 × \ T Texture [53] 2006 Sokol Petushi 9 texture parameters: DNM1, DNM2, DNM3,DNM1-2, DNM1-3, DNM2-3, DNM1-2-3, DT, DNT Texture [94] 2008 Olcay Sertel LBP features,Haralick featuresT Texture [95] 2009 O Sertel \ T Texture [68] 2009 J Kong, O Sertel Four textural Haralick featuresT Texture [96] 2009 O Sertel, J Kong LBP featuresT Texture [97] 2010 Vincent Roullier LBP histogramT Texture [98] 2011 Matthew D. DiFranco Feature maps(the mean and standard from HS),Co-occurrence texture features
Table 4: Continue: Summary of the CAD methods used for feature extraction inWSI.
T Texture [99] 2011 Jun Kong Texture and gradient featuresT Texture [100] 2011 Vincent Roullier LBPT Texture [101] 2011 Michael Grunkin \ T Texture [102] 2011 Kien Nguyen Gabor filter featuresT Texture [55] 2012 Scott Doyle Co-occurrence FeaturesT Texture [71] 2012 Hatice Cinar Akakin Mean, standard deviation, contrast,correlation, energy,entropy and uniformityT Texture [103] 2012 Harshita Sharma Texton-based textureT Texture [73] 2013 Nandita Nayak \ T Texture [104] 2013 Liping Jiao Combine mean, varianceand other 17 features whichare extracted by GLCM methodT Texture [74] 2013 M. Veta \ T Texture [75] 2013 Kothari S \ T Texture [105] 2013 Jun Kong \ T Texture [76] 2013 Andr´e Homeyer \ T Texture [77] 2013 Hiary H, Alomari R S \ T Texture [106] 2014 Apou G, Naegel B \ T Texture [65] 2014 Ezgi Mercan LBPT Texture [107] 2015 Bejnordi, B. E LBPT Texture [108] 2015 Harshita Sharma GLCM FeaturesT Texture [109] 2015 Zaneta Swiderska \ T Texture [82] 2015 Michaela Weingant GLCM featuresT Texture [83] 2015 Ruoyu Li \ T Texture [110] 2015 Xiaofan Zhang \ T Texture [111] 2015 Lee AD Cooper \ T Texture [112] 2015 Gregory Apou \ T Texture [113] 2015 Peikari M \ T Texture [65] 2016 Mercan E LBPT Texture [84] 2016 Barker J \ T Texture [114] 2016 Bejnordi B E \ T Texture [115] 2016 Zhao Y GLCMT Texture [116] 2016 Harder N Co-occurrence FeatureT Texture [117] 2016 Shirinifard A \ T Texture [118] 2016 Gadermayr M HOG,LBP,FVT Texture [119] 2016 Leo P, Lee G Haralick featuresT Texture [66] 2016 Mercan, C LBPT Texture [85] 2016 Brieu N, Pauly O \ T Texture [120] 2017 Saltz, J \ T Texture [43] 2017 Babaie M LBPT Texture [121] 2017 Hu J X Haralick textureT Texture [47] 2017 Bejnordi B E SIFT,LBP,GLCMT Texture [122] 2017 Valkonen M VLFeat implementation of MSER and SIFTT Texture [123] 2018 Jeffrey J. Nirschl \ T Texture [124] 2018 Hongming Xu \ T Texture [125] 2018 Hiroshi Yoshida \ T Texture [126] 2018 W. Han \ itle Suppressed Due to Excessive Length 29 Table 4: Continue: Summary of the CAD methods used for feature extraction inWSI.
T Texture [88] 2018 Mork¯unas M \ T Texture [86] 2018 Caner Mercan \ T Texture [127] 2018 Olivier Simon LBP,mrcLBP featureT Texture [128] 2019 S Klimov \ T Shape [93] 2004 JamesDiamondPhDa \ T Shape [99] 2011 Jun Kong \ T Shape [101] 2011 Michael Grunkin \ T Shape [133] 2012 Cheng Lu \ T Shape [70] 2012 Sonal Kothari \ T Shape [134] 2012 Mitko Veta \ T Shape [135] 2013 Lopez X M Multiple sharpness featuresT Shape [72] 2013 Brian T. Collins \ T Shape [74] 2013 M. Veta \ T Shape [75] 2013 Kothari S \ T Shape [105] 2013 Jun Kong \ T Shape [111] 2015 Lee AD Cooper \ T Shape [84] 2016 Barker J \ T Shape [138] 2016Dayong Wang Aditya Khosla \ T Shape [120] 2017 Saltz, J \ T Shape [124] 2018 Hongming Xu \ DL CNN [136] 2016 Puerto M \ DL CNN [137] 2016 Sharma H \ DL CNN [138] 2016Dayong Wang Aditya Khosla GoogLeNetDL CNN [139] 2016 Ge¸cer B \ DL CNN [140] 2016 Sirinukunwattana K NEP coupled with CNNDL CNN [141] 2016 Hou L \ DL CNN [142] 2016 Sheikhzadeh F CNN,FCNDL CNN [143] 2017 Cruz-Roa A \ DL CNN [144] 2017 Wollmann T DNNDL CNN [145] 2017 Ara´ujo T Patch-wise trained CNNDL CNN [195] 2017 B´andi P FCN,U-netDL CNN [146] 2017 Bejnordi B E Context-aware stacked CNNDL CNN [147] 2017 Sharma H Selected self-designed CNN architectureDL CNN [22] 2017 Korbar B A modified version of a ResNet architectureDL CNN [148] 2017 Jimenez-del-Toro O \ DL CNN [149] 2017 Xu Y \ DL CNN [150] 2017 Korbar B ResNetDL CNN [151] 2017 Ghosh A Deep convolutional networkDL CNN [152] 2017 Das K, Karri S P K self-designed CNNDL CNN [153] 2018 Cui Y FCNDL CNN [154] 2018 Farhad Ghazvinian Zanjani Apply CRFs over latent spacesof a trained deep CNNDL CNN [155] 2018 Courtiol P ResNetDL CNN [44] 2018 Meghana Dinesh Kumar VGG,AlexNetDL CNN [156] 2018 Dmitrii Bychkov VGGDL CNN [157] 2018 Baris Gecer FCNDL RNN [158] 2018 Jian Ren LSTM
Table 4: Continue: Summary of the CAD methods used for feature extraction inWSI.
DL CNN [159] 2018 David Tellez Self-designed CNNDL CNN [160] 2018 K Sirinukunwattana Self-designed CNNDL CNN [161] 2018 Scotty Kwok Inception-Resnet-v2DL CNN [162] 2018 Kausik Das A MIL framework for CNNDL CNN [163] 2018 Huangjing Lin FCNDL CNN [164] 2018 Campanella G VGG and ResNetDL CNN [165] 2018 Jiang S ResNetDL CNN [166] 2018 X Wang VGGDL CNN [167] 2018 Junni Shou DenseNetDL CNN [168] 2018 David Tellez \ DL CNN [196] 2019 Nikhil Seth U-NetDL CNN [169] 2019 Christof A. Bertram ResNetDL CNN [170] 2019 Jiayun Li VGGDL CNN [171] 2019 Liu Y InceptionDL CNN [172] 2019 Gabriele Campanella A MIL framework for CNNDL CNN [173] 2019 Xingzhi Yue VGGDL RNN [174] 2019 Sam Maksoud LSTMDL CNN [175] 2019 S Bilaloglu PathCNN(Self-designed)DL CNN [176] 2019 Huangjing Lin VGGDL CNN [177] 2019 H Yu, X Zhang DNNDL CNN [178] 2019 Adit B. Sanghvi, MISM \ DL \ [179] 2019 Shujun Wang RMDLDL CNN [180] 2019 Kohlberger T ConvFocus(based CNN)DL CNN [197] 2019 Seth N, Akbar S U-netDL CNN [181] 2019 Xu J ResNet and DenseNetDL CNN [198] 2020 Feng Y, Hafiane A U-netDL CNN [182] 2020 Chen P, Shi X DeepCIN(CNN,BLSTM)DL CNN [183] 2020 Sornapudi S, Addanki R \ DL CNN [184] 2020 Pantanowitz L Self-designed CNN
In recent years, with the increasing size and quantity of medical images, com-puters must facilitate processing and analysis. In particular, computer algorithmsfor delineating anatomical structures and other areas of interest are becoming in-creasingly important in assisting and automating specific histopathological tasks.These algorithms are called image segmentation algorithms [202].Image segmentation refers to the process of dividing a digital image into multi-ple segments, namely a set of pixels. The pixels in a region are similar according to some homogeneity criteria (such as color, intensity, or texture), to locate and iden-tify objects and boundaries in the image [203]. The practical applications of imagesegmentation include: filtering noise images, medical applications (locating tumorsand other pathologies, measuring tissue volume, computer-guided surgery, diag-nosis, treatment planning, anatomical structure research) [204], locating objects itle Suppressed Due to Excessive Length 31 in satellite images (roads, forests, etc.), facial recognition, fingerprint recognition,etc. The selection of segmentation techniques and the level of segmentation de-pends on the specific type of image and the characteristics of the problem beingconsidered [205].In the process of medical image segmentation, the details required in the seg-mentation process largely depend on the clinical application of the problems [206] [207].The purpose of segmentation is to improve the visualization process to deal withthe detection process more effectively. Medical image segmentation is faced withmany problems because the quality of the segmentation process is affected [208].When there is noise in the image, there will be uncertainty, which makes it difficultto classify the image [209]. The reason is that the intensity value of the pixel hasbeen modified due to noise in the image. Such a change in pixel intensity valuewill disturb the uniformity of the image intensity range [210]. Therefore, to dealwith this uncertainty, image segmentation plays a crucial role in medical diagnosticsystems [211].As a crucial step in CAD pathologists, segmentation techniques have flourishedin recent years. As shown in Fig. 2, from 2010 to 2020, the number of papersusing segmentation WSI technology to assist doctors in diagnosis has increasedfrom 2 to 28. According to the papers we have reviewed, segmentation is dividedinto five different techniques including thresholding-based, region-based, graph-based, clustering-based, deep learning, and other image segmentation methods.Its composition and structure diagram are shown in Figure. 20.
Thresholding-based
Methodologies of Image Segmentation
Region-based Graph-based Clustering-based
Deep Learning Others
Fig. 20: The composition of the segmentation method in WSI.4.1 Thresholding based Segmentation MethodThreshold segmentation is a classic method in image segmentation. It uses thedifference in grayscale between the target and the background to be extracted inthe image, and divides the pixel level into several categories by setting a thresh-old to achieve the separation of the target and the background [51, 212]. Thethreshold segmentation method is simple to calculate, and can always use closed and connected boundaries to define non-overlapping regions. Images with a strongcontrast between the target and the background can provide better segmentationeffect [213].Among them, the selection of the optimal threshold is a significant issue. Com-monly used threshold selection methods are: manual experience selection method, histogram method [214], maximum between-class variance method (OTSU) [215],adaptive threshold method. Because the threshold segmentation method is simpleto implement, the amount of calculation is small, and the performance is relativelystable, it has become the most in image segmentation. As the basic and mostwidely used segmentation technology, it has been used in many fields. Among thereviewed papers, five are based on threshold based segmentation [99,133,216–218].In [99], an image analysis tool for segmentation and characterization of cell nu-clei is developed. The microscopic images of glioblastoma from the TCGA projectare used. To reliably identify cell nuclei, a fast hybrid gray-scale reconstructionalgorithm is applied to the image to normalize the background area degraded byartifacts produced by tissue preparation and scanning [219]. This operation sepa-rates the foreground from the normalized background and allows simple thresholdprocessing to identify the nucleus.In [133], a computer-aided technique is proposed for segmentation and analy-sis of the whole slide skin histopathological images. Before using the segmentationtechnique, determine the single-color channel that provides good discrimination in-formation between the epidermis and dermis regions. Then multi-resolution imageanalysis is used in the proposed segmentation technique. First, a low-resolutionimage of the WSI is generated. Then, the global threshold method and shapeanalysis to segment low-resolution images are used. Based on the segmented skinarea, the layout of the skin is determined, and a high-resolution image block ofthe skin is generated for further manual or automatic analysis. Experiments on 16different whole slide skin images show that the technology has high performance,92% sensitivity, 93% accuracy, and 97% specificity are achieved.In [216], a new method is proposed to segment severely aggregated overlappingcores. The proposed method first involves applying a combination of global andlocal thresholds to extract foreground regions.In [217] and [220], a highly scalable and cost-effective image analysis frameworkbased on MapReduce is proposed, and a cloud-based implementation is provided.The framework adopts a grid-based overlap segmentation scheme and providesparallelization of image segmentation based on MapReduce. In the segmentationstep, a threshold method is applied to segment the nucleus.In [218], the segmentation of tumor and non-tumor areas on the WSIs datasetsof osteosarcoma histopathology. The method in this article combines pixel-basedand object-based methods, using tumor attributes, such as nucleus clusters, den-sity, and circularity, and using multi-threshold Otsu segmentation technology tofurther classify tumor regions as live and inactive. The pan-fill algorithm clus-ters similar pixels into cell objects and calculates the cluster data to analyze thestudied area further. The final experimental results show that for all the sampleddatasets used, the accuracy of the method in question in identifying live tumorsand coagulative necrosis is 100%, while the accuracy of fibrosis and acellular/lowcell tumors is about 90%. The WSI effect after multi-threshold Otsu segmentationis shown in Figure. 21. itle Suppressed Due to Excessive Length 33
Fig. 21: Otsu output showing more blue color. This figure corresponds to Fig.3in [218].technology, it uses the similarity of the object’s gray distribution and background.Generally, region-based image segmentation methods contain two categories: wa-tershed segmentation and region growing.
Watershed Segmentation
The watershed algorithm draws on the theory of morphology and is a region-based image segmentation algorithm. In this method, an image is regarded as atopographic map, and the gray value corresponds to the height of the terrain.High gray values correspond to mountains, and low gray values correspond tovalleys. If rain falls on the surface, the low-lying area is a basin, and the ridgebetween the basins is called a watershed. Watershed is equivalent to an adaptivemulti-threshold segmentation algorithm [221].In [99], overlapping nuclei are separated using the watershed method. In [134],an automatic cell nucleus segmentation algorithm is used to extract size-relatedmorphometric features of cell nuclei and analyze their prognostic value in malebreast cancer. The segmentation process consists of four main steps: preprocessing,watershed segmentation controlled by multi-scale markers, post-processing, andmerging of multi-scale results. The overall process of this automatic segmentationmethod is shown in the Figure. 22. In [222], the same automatic segmentationmethod as in [134] is used in H&E stained breast cancer histopathology images.
In [216], to segment the overlapping nuclei gathered in the foreground region,seed markers are obtained using morphological filtering and intensity-based regiongrowth. Then the seed watershed and separate the aggregated nuclei are applied.Finally, a post-processing step of identifying positive nuclear pixels is added toeliminate false pixels. Some segmentation results are shown in Fig. 23. In [217]
Fig. 22: Overview of the automatic nuclei segmentation method. This figure cor-responds to Fig.1 in [134].and [220], watershed technology is used to separate overlapping nuclei in objectsis used.Fig. 23: Some segmentation results. This figure corresponds to Fig.7 in [216].
Region Growing
Region growing is an image segmentation method of serial region segmenta-tion. Region growth refers to starting from a certain pixel and gradually addingneighboring pixels according to specific criteria. When certain conditions are met, itle Suppressed Due to Excessive Length 35 the regional growth is terminated, that is, The region’s growth depends on theselection of the initial point (seed point), growth criteria, and termination condi-tions [223].The region growing is relatively a common method. It can achieve the bestperformance when there is no prior knowledge available, and it can be used to seg-ment more complex images. However, the regional growth method is iterative, andspace and time costs are relatively high [224]. Among the WSI-based segmentationtasks we have summarized, there is one paper related to region growth [216].4.3 Graph-based Segmentation MethodGraph-based segmentation is a classic image segmentation algorithm. The algo-rithm is a greedy clustering algorithm based on the graph. Its advantages includesimple implementation and faster speed [225]. Many popular algorithms are basedon this method [226].Graph-based segmentation first expresses image as a graph in graph theory,so that, each point in a pixel is regarded as a vertex v i ∈ V , and each pixel and8 adjacent pixels (eight neighborhoods) form a graph edge e i ∈ E , so a graph G = ( V, E ) is constructed. The weight of each side of the graph is the relation-ship between the pixel and the neighboring pixels, which expresses the similaritybetween the neighboring pixels. Treat each node (pixel) as a single area, and thenmerge it according to the parameters of the area and the internal difference to getthe final segmentation [227].Because WSIs are usually very large, they are stored as pyramids of tiledimages, so that they can be processed in a hierarchical manner, that is, low-resolution to determine the area of interest, high-resolution image classification,top-down segmentation. Among the papers we reviewed, there are three papersthat combine graph-based segmentation methods with multi-resolution [64,97,100].In [64], [97] and [100], the mitosis in WSI of breast tissue is extracted. Theimage is simplified by discrete regularization, and clustering is performed by un-supervised 2-mean clustering. Clustering is performed in a specific area divided atthe previous resolution. The obtained clusters are expanded with finer resolutionlevels through pixel duplication and refined in specific areas. At the last resolutionlevel, the mitotic figure is extracted. The segmentation result of mitosis in [64] isshown in the Figure. 24.
Fig. 24: The composition of the segmentation method in WSI. This figure corre-sponds to Fig.3 in [64] k -means algorithm is more commonly used in clustering algorithms. The clus-ter center point is obtained by calculating the average of all pixels in each clusterin the sample. The basic working principle of the k -means algorithm is to receivethe parameter k input by the user, divide the given n data sample points into k groups on average, take the input k points as the cluster centers to be converged,and calculate the other clusters. The Euclidean distance from the sampling pointto the k convergence centers. And compare the distance between all samplingpoints and the convergence center point. The classification is made by comparingthe minimum form of Euclidean distance. Then after repeated iterations, the meanvalue of k clusters is successively obtained. Until the performance criterion func-tion, clustering is the best, the overall error is the smallest, and the best clusteringeffect is obtained [228].Among the papers we reviewed, the papers on k -means clustering and segmen-tation are [77] and [218]. In [77], the author uses k -means unsupervised learning forWSI segmentation, which produces a highly robust correctness result equivalentto supervised learning, which is 95 .
5% accuracy. In [218], the k -means clusteringtechnique with color normalization is used for tumor separation.4.5 Deep Learning based SegmentationAmong the papers we reviewed, 10 papers used deep learning methods for WSIsegmentation [21, 149, 153, 160, 195–198, 220, 229].In [195], two different CNN structures, FCN and U-net, are used to segmentand accurately identify tissue slices. Here, the two methods are compared with thetraditional foreground extraction (FESI) algorithm based on structural informa-tion. These three methods are applied to 54 WSIs, and the average value of theYakoka index and the standard deviation of the Yakoka index are used for evalu-ation. The final U-net result is the best (Jaccard index is 0.937). The qualitativeeffects of different algorithms are shown in Figure. 25.In [149], CNN-based ImageNet is used to extract features and convert them intohistopathological images. And Support Vector Machine (SVM) is used to definesegmentation as a classification problem. This method is applied to the digitalpathology and colon cancer dataset of the MICCAI 2014 Challenge, and finallywon the first place with 84% test data accuracy.In [229], a simple and effective framework called Reinforced Auto-Zoom Net(RAZN) is proposed, which considers the accurate and fast prediction of breastcancer segmentation. RAZN learns a strategy network to decide whether to zoomon a given area of interest. Because the amplification action is selective, RAZN is robust to unbalanced and noisy ground truth labels and effectively reducesoverfitting. Finally, the method is evaluated on the public breast cancer dataset.It can be seen from the experimental results that RAZN is superior to single-scaleand multi-scale baseline methods, and obtains better accuracy with lower inferencecost. itle Suppressed Due to Excessive Length 37 Fig. 25: Qualitative results for the different algorithms. This figure corresponds toFig.2 in [195]In [153], an automatic end-to-end deep neural network algorithm is proposedto segment the single core. The kernel-boundary model is introduced to predictthe kernel and its boundary simultaneously using FCN. Given the color normal-ized image, the model directly outputs the estimated kernel map and boundarymap. After post-processing, the final segmented core is produced. A method forextracting and assembling overlapping blocks is designed to seamlessly predict thecores in a large WSI. The final result proves the effectiveness of the data expan-sion method for cell nucleus segmentation tasks. The experiment shows that thismethod is superior to the prior art method and it is possible to accurately segmentWSI within an acceptable time.In [160], different architectures are systematically compared to evaluate howthe inclusion of multi-scale information affects segmentation performance. The ar-chitectures are shown in Figure. 26. It uses a public breast cancer dataset anda locally collected prostate cancer dataset. The result shows that the visual en-vironment and scale play a vital role in the classification of histological images.In [21], a new encoder-decoder architecture is proposed to solve the seman-tic segmentation problem of breast biopsy WSI. The designed new architecturecontains four new functions: (1) Input Perceptual Coding Block (IA-RCU), whichenhances the input inside the encoder to compensate for the loss of informationdue to down-sampling operations, (2) densely connected decoding network and (3)additional sparsely connected decoding network to effectively combine the multi-level features aggregated by the encoder, and (4) a multi-resolution network for context-aware learning, which uses densely connected fusion blocks to combine dif-ferent resolutions rate output. The architecture is shown in Figure. 27. The resultafter segmentation is shown in Figure. 28.In [196] and [197], several U-net architectures-DCNN designed to output proba-bility maps are trained to segment ductal carcinoma in-situ (DCIS) WSI in wireless
Fig. 26: Used architectures. This figure corresponds to Fig.2 in [160]sensor networks and verified the minimum required to achieve excellent accuracyat the slide level good patch vision. U-net is trained five times to achieve the besttest results (DSC = 0.771, F1 = 0.601), which means that U-net benefits fromseeing wider contextual information.In [198], a multi-scale image processing method is proposed to solve the segmen-tation problem of liver cancer in histopathological images. These eight networksare compared, and then the network most suitable for liver cancer segmentation isselected. Through a comprehensive comparison of performance, U-net is selected.The local color normalization method of pathological images is used to solve the in-fluence of the background, and then a seven-layer Gaussian pyramid representationis established for each WSI to obtain a multi-scale image set. The trained U-netis fine-tuned at each level to obtain an independent model. Then, shift croppingand weighted overlap are used in the prediction process to solve block continuity.Finally, the predicted image is mapped back to the original size, and a votingmechanism is proposed to combine the multi-scale predicted images. The experi-mental data are the verification images of the 2019 MICCAI PAIP Challenge. Theevaluation results show that this algorithm is better than other algorithms.4.6 Other Segmentation Methods
In [230], a new algorithm for segmenting cell nuclei is used. Before determiningthe precise shape of the cell nucleus by the elastic segmentation algorithm, theproposed algorithm uses a voting scheme and prior knowledge to locate the cellnucleus. After removing noise through the mean shift and median filtering, theCanny edge detection algorithm is used to extract edges. Since the nucleus is itle Suppressed Due to Excessive Length 39
Fig. 27: Multi-resolution encoder-decoder network structure diagram. This figurecorresponds to Fig.4 in [21].observed to be surrounded by cytoplasm, its shape is roughly elliptical, and theedges adjacent to the background are removed. The random hough transform ofthe ellipse finds the candidate kernel, and then processes it through the levelset algorithm. The algorithm is tested and compared with other algorithms in adatabase containing 207 images obtained from two different microscope slides, andthe results passed the positive predictive value (PPV) and TPR. The high valueof, is displayed, resulting in a high measurement value of 96 . segmentation of the image is provided in an effective manner. Then other subse-quent steps are executed. The schematic diagram of the segmentation process isshown in Figure. 29.In [110], a robust segmentation method is developed to accurately delineateROI (eg, cells) using hierarchical voting and repelling active contours. Its segmen- Fig. 28: Segmentation result. The first line describes aggressive cases, while thesecond line describes benign cases. This figure corresponds to Fig.9 in [21].Fig. 29: Path computation in horizontal and vertical strips, leading to an imagepartition. This figure corresponds to Fig.4 in [106] tation is based on active contours with repelling terms [231]. The exclusion term isused to prevent the contour lines from intersecting and merging. Based on the de-tection result, the circle is associated with each detected cell as the initial contour.The final segmentation result is shown in Figure. 30.In [83], a new ROI search and segmentation algorithm based on superpixelsis proposed. First, the initial recognition of the ROI is obtained by gathering su- itle Suppressed Due to Excessive Length 41
Fig. 30: Segmentation results of different methods on a randomly picked patch.From left to right: original image and ours. This figure corresponds to Fig.3 in [110] . perpixels at low magnification. Then, by marking the corresponding pixels, the superpixels are mapped to the higher magnification image. This process is re-peated several times until the segmentation is stable. This method is differentfrom the previous classic segmentation methods based on superpixels [232, 233].This algorithm provides image segmentation with topology preservation.In [85], threshold processing is performed on the foreground posterior imageto detect the foreground area, and spot detection algorithms such as MSER andGaussian difference are used further to identify the brightness of the image. Ac-cording to the classification function combining suitable candidate parameters areselected. The remaining area is further divided by calculating the minimum pathof the posterior mapping between the concave points and checking the goodnessof fit of the candidate area in turn. The schematic diagram after segmentation isshown in Figure. 31.4.7 SummaryAs can be seen from the content we reviewed above, machine learning is used incombination with WSI technology to assist diagnosis. In the field of image segmen-tation, used methods include thresholding-based segmentation, region-based seg-mentation, graph-based segmentation, clustering-based segmentation, deep learning-based segmentation, and other segmentation methods. We can know that the graph-based segmentation method is a more classic im-age algorithm, so it became popular earlier, and many algorithms are based on thegraph-based segmentation method. Thresholding-based segmentation methods areoften used in WSI in combination with the watershed algorithm in region-basedsegmentation methods. In the papers we reviewed, three of them used both meth-ods at the same time. Then clustering-based segmentation methods and some
Fig. 31: The schematic diagram after segmentation is shown in the figure. Left:H&E (a)-(b), IHC-nuclei (c), and CD8 (d) regions; Center: posterior maps formarked nuclei and cells (blue channel if applicable), for homogeneous nuclei (redchannel) and for textured nuclei (green channel); Right: outline of the segmentationresult. This figure corresponds to Fig.3 in [85].other segmentation methods appeared to be used. Until 2017, deep learning meth-ods are widely used, and the application of deep learning to WSI segmentationbegan to get good results. Among them, the U-net architecture based on multi- resolution has been used many times. Table. 5 is a summary of the CAD methodfor segmentation technology in WSI. itle Suppressed Due to Excessive Length 43
Table 5: Summary of the CAD methods used for segmentation in WSI.
Method ReferenceYear Team DetailsThresholding [99] 2011 Jun Kong Simple threshold watershed methodThresholding [133] 2012 Cheng Lu Otsu thresholdThresholding [216] 2013 Jie Shu Automatic threshold method(foreground and backgroundclassification), regiongrowth, watershedsegmentation (seriouslyclustered overlappingcores for segmentation)Thresholding [217] 2016 Vo H Parallelization ofimage segmentation basedon MapReduceThresholding [218] 2017 Arunachalam H B The combination of pixel-based andobject-based methods (k-means andnon-tumor images, and multi-levelOtsu threshold segmentation)Region-based(Watered) [99] 2011 Jun Kong Simple thresholdwatershed methodRegion-based(Watered) [134] 2012 Mitko Veta Watershed segmentationcontrolled bymulti-scale markersRegion-based(Watered) [222] 2013 Veta M Marker-controlledwatershed segmentation,with multiple scales anddifferent markers(automatic nuclear segmentation)Region-based(Watered) [216] 2013 Jie Shu Automatic threshold method(foreground and backgroundclassification), region growth,watershed segmentation(seriously clustered overlappingcores for segmentation)Region-based(Watered) [217] 2016 Vo H Parallelization of ImageSegmentation Basedon MapReduceRegion-based(Watered) [220] 2019 Hoang Vo Core segmentation(MapReduce architecture)Region-based(Region growing) [216] 2013 Jie Shu Automatic threshold method(foreground and backgroundclassification), region growth,watershed segmentation (seriouslyclustered overlapping coresfor segmentation)
Table 5: Continue: Summary of the CAD methods used for segmentation in WSI.
Graph-based [64] 2010 V. Roullier Integrates graph-basedsegmentation (discrete semi-supervised clustering)and multi-resolutionsegmentation (clusterspace refinement)Graph-based [97] 2010 Vincent Roullier Integrates graph-basedsegmentation (discrete semi-supervised clustering)and multi-resolutionsegmentation (clusterspace refinement)Graph-based [100] 2011 Vincent Roullier Multi-resolution segmentationmethod (regularization framework,histogram construction,histogram 3 mean clustering,partition and spatial refinement)Clustering [77] 2013 Hazem Hiary k -means clusteringClustering [218] 2017 Arunachalam H B The combination of pixel-based and object-based methods(k-means and multi-levelOtsu threshold)Deep Learning [195] 2017 B´andi P Using FCN and U-netto organize backgroundsegmentationDeep Learning [149] 2017 Xu Y SVM-CNNDeep Learning [229] 2018 Nanqing Dong High-resolution imagesemantic segmentationframework-ReinforcedAuto-Zoom Net (RAZN) (FCN)Deep Learning [153] 2018 Cui Y Supervised FCN methodfor nuclear segmentationin histopathological imagesDeep Learning [160] 2018 K Sirinukunwattana Combining multipleCNNs of differentscales with LSTMDeep Learning [21] 2018 Sachin Mehta Multi-resolutionencoder-decoder networksemantic segmentationDeep Learning [220] 2019 Hoang Vo Core segmentation(MapReduce architecture)Deep Learning [196] 2019 Nikhil Seth Multi-resolutionU-net architectureDeep Learning [197] 2019 Seth N, Akbar S Multi-resolutionU-net architectureDeep Learning [198] 2020 Feng Y, Hafiane A Multi-resolutionseven-layer pyramid U-net itle Suppressed Due to Excessive Length 45 Table 5: Continue: Summary of the CAD methods used for segmentation in WSI.
Other [230] 2012 Christoph Bergmeir Random Hough transformellipse fitting cellnucleus to segment the nucleus(level set algorithm)Other [106] 2014 Apou G, Naegel B The path calculation inthe horizontal andvertical stripsOther [110] 2015 Xiaofan Zhang (Robust segmentation method)Use Euclidean distance.Based on active contourswith repelling termsOther [83] 2015Ruoyu Li and Junzhou Huang The superpixels are clusteredat low magnification toobtain ROI. Then, thesuperpixels are mappedto the higher magnificationimage. This process isrepeated several times.Other [85] 2016 Brieu N, Pauly O It is further dividedby calculating theminimum path of theposterior mapping betweenthe concave points andchecking the goodness offit of the candidatearea in turn
Image classification, as the name suggests, is to have a fixed set of classificationlabels, and then for the input image, find a classification label from the classifica-tion label set, and finally assign the classification label to the input image. It isat the heart of computer vision and is the most fundamental issue that forms thebasis for other computer vision tasks such as positioning, detection, and segmen-tation [186] [234]. It is widely used in practice. While a simple task for humans,it can be challenging for computer systems. Many seemingly different problems incomputer vision (such as object detection and segmentation) can be reduced toimage classification.In the analysis of pathological images, the most studied task is CAD. It alsohelps the pathologist make a diagnosis. The diagnostic process is the task of map- ping one or more WSIs to a disease category. Since errors produced by machinelearning systems are different from those produced by human pathologists [138],the use of computer-aided design systems can improve classification accuracy [34].In recent years, due to the progress of computer technology, histopathologicalimage classification has gradually become a research hotspot in the field of medical image processing. To the human anatomy area and the pathological changes areato carry on the accurate classification, may the maximum degree doctor accurate,the rapid diagnosis condition. This is of great significance to the further diagnosisof doctors and the further treatment of patients.In the papers we summarized, there are around 54 articles from 2004 to 2020 onimage classification using WSI techniques to assist pathologists in diagnosis. Wecan see from the development trend in Fig. 2 that the application of classificationhas increased. This reflects the wide application of classification technology. Fromthese papers, we can briefly summarize the classification methods they appliedincluding traditional machine learning algorithms, deep learning algorithm, andother methods.5.1 Traditional Machine Learning based Classification MethodAmong the papers we reviewed, there are 18 papers involve the classification ofWSI by using traditional machine learning algorithms for CAD.
SVM-based Classification Method
SVM is a type of supervised machine learning technique. It was first publishedin 1963 by Vladimir N. Vapnik and Alexander Y. Lerner [235]. It uses the hy-pothesis space of linear functions in hyperspace [236] and trains with the learningalgorithm of optimization theory, which realizes the learning bias derived from sta-tistical learning theory. The purpose of classification by SVM is to find an effectivecomputational method to learn good separation hyperplanes in hyperspace [237].SVM is designed for binary classification. when an SVM is applied to a multi-classclassification problem, it internally splits the task into multiple binary classifica-tion problems and uses several SVMs to solve them [238] [239]. A total of tenpapers in our review involve the use of SVMs for WSI classification.In [98], a tile-based approach is proposed to generate clinically relevant prob-ability maps of prostate cancer in prostate WSIs. The probability of cancer ex-istence is calculated from the response of each classifier in the ensemble. Beforeclassification, texture feature extraction and spatial filtering are performed. Theclassification is then performed using either an RF or an SVM (linear and radialkernels). Different feature subsets and different subsampled training data strate-gies are used for performance comparison. The final best classification result isobtained by Radial Basis Function (RBF) kernel SVM, which reports an AUCvalue of 95 . codes. The encoder is used to train the classifier on a small amount of label data.Multi-class regularization support vector classification is used, the regularizationparameter is 1, and the polynomial kernel is 3. In terms of data, two datasetsfrom (i) glioblastoma multiforme (GBM) and (ii) clear cell renal cell carcinoma(KIRC) of TCGA are used. The classification accuracy rates of 84 .
3% and 80 . itle Suppressed Due to Excessive Length 47 Fig. 32: This figure shows the utility of using classification probabilities for heatmap visualization at the tile level. There is a good difference between strong posi-tive and negative results (red versus blue). This figure corresponds to Fig.16 in [98].are obtained respectively. The classification results of heterogeneous GBM tissuesections are shown in Figure. 33.In [104], an SVM based classifier is used to classify colon cancer WSI andnon-colon cancer WSI. Then, 18 simple features (such as gray-level mean andgray-level variance) and 16 texture features extracted by the GLCM method areselected as the feature set. The final result shows that when all features are used,the mean values of accuracy, recall and F-measure are 96 . . . is to randomly select image blocks from the entire tissue area, divide them intosmall blocks, and perform Gaussian texture filtering on them. The texture filterresponses for each texture are combined and statistical indicators are obtainedfrom the histogram of the response. Then, the visual word bag pipeline is usedto combine extracted features to form a word histogram for each image block. Fig. 33: Two examples of classification results of a heterogeneous GBM tissuesections. The left and right images correspond to the original and classificationresults, respectively. Color coding is black (tumor), pink (necrosis), and green(transition to necrosis). This figure corresponds to Fig.4 in [73].The SVM classifier is trained using the computed lexical histograms to distinguishclinically relevant and unrelated plaques. Finally, the ROC is 0.87. It can be provedthat texture features can be used to classify important areas in WSI.In [118], to improve the generalization ability of the classification model forkidney WSI, a domain adaptive method is proposed, in which the classifier istrained on the data from the source domain to present a small number of user-labeled samples from the target domain. Efficient linear SVM is used to avoidwaiting time during interaction. In a comparison between interactive and non-interactive domain adaptation, it is observed that interactive domain adaptationhas a positive effect on the classification performance.In [241], SVM is used to predict the presence of cancer by WSI of sentinellymph nodes, and an equivalent fuzzy model is developed to improve the inter-pretability. The SVM model consists of 50 support vectors, and the accuracy rate is . . itle Suppressed Due to Excessive Length 49 through a multi-resolution frame. Next, an epidermal analysis is performed, inwhich a set of epidermal features reflecting the nucleus morphology and spatialdistribution are calculated. While performing epidermal analysis, dermal analysisis also performed, in which the dermal cell nucleus is segmented, and a set ofstructural and cytological features are calculated. Finally, by using a multi-classSVM with extracted epidermal and dermal features, the skin melanocyte imagesare classified into different categories. It is known from the experimental resultsthat the classification accuracy of this technology is over 95%. Random Forest-based Classification Method
RF is a popular machine learning algorithm, often used in classification tasksin various fields [242–249]. RF is a collection of tree structure classifiers [250].Each tree relies on the value of a randomly selected vector distributed in the sameway among all trees in the forest [251]. Each tree in the forest will vote once,assigning each input to the most likely category label. This method is fast androbust to noise, and is a successful ensemble that can identify nonlinear patternsin data. It can easily handle numerical and categorical data [249]. One of the mainadvantages of RF is that even if more trees are added to the forest, it causesoverfitting [252]. In the papers we have summarized, there are five papers mainlyusing RF classifiers for related WSI classification.In [76], it is mainly to quantify the necrotic part in histological WSIs. First, thethreshold principle is used to eliminate the background. Color and texture featuresare then extracted. After the feature classification step is performed, another post-processing step is performed to process the misclassified isolated tiles, that is, usingthe principle of context classification and spatial context information to re-evaluateall uncertainly classified slices. The sections are then merged, and each section isidentified by merging all the non-background slices joined by the edges. Finally,the results are evaluated. Naive Bayes classifier, k -nearest neighbor ( k NN), and RFclassifier are used in the post-processing classifier. From the experimental results,it can be known that the RF classifier has the best results, and the HSV-basedfeatures are better than RGB.In [137], the main task is to automatically classify cancer from histopatholog-ical stomach images. The classification performance of traditional image analysismethods and deep learning methods are quantitatively compared. First, the datais augmented. In traditional image analysis methods, the classifier used is a RFclassifier.Metastatic breast cancer is identified in [138]. First, the background is automat-ically detected based on the threshold. And by comparing GoogLeNet, AlexNet,VGG16 and FaceNet. GoogLeNet is selected to generate tumor probability heatmaps,and a RF classifier is used to classify metastatic WSIs and negative WSIs, andfinally the AUC of 0.925 is obtained.
In [253], normal sections and tumor sections from histological images of lymphnode tissue are classified. The first step is to remove unnecessary information.The CNN model is then reconstructed or trained to segment the tumor area. Thespecific details are in Sec. 6. After the tumor area is segmented, the features areextracted, and the RF classifier is used for classification. Finally the result withan AUC score of 0.94 is obtained.
In [128], the risk of recurrence of DCIS is classified. First of all, the color nor-malization and down sampling tasks is performed. Secondly, the texture featuresare extracted. Then, input these features into the RF classifier to predict the highand low risk of recurrence. It is convenient for the doctor to give the correspondingdiagnosis and treatment plan. The final result shows that the classifier significantlypredicts the 10-year risk of recurrence during training (accuracy = 0.87, sensitivity= 0.71, and specificity = 0.91).
Others Traditional Machine Learning Classification Method
In addition to the two commonly used classification methods of SVM andRF, some other methods are also used to classify histopathological WSI, such asBayesian classifier [254] and k NN [255] classifier.In [94, 96], a system has been developed for quantitative analysis of neuroblas-toma WSIs, which included stromal rich and stromal poor. The developed methodis based on the Gaussian pyramid method with multi-resolution. WSIs includenon-overlapping image blocks and parallel processing of image blocks, which iscarried out by the parallel computing module developed previously. Then, thetexture features are extracted and the optimal subset is executed. Next, the fea-ture selection of the sequential float forward selection (SFFS) method is adopted,and the confidence degree is calculated by k NN classification. If the confidencelevel falls below the set threshold, switch to a higher resolution. The experimentalresults show that the overall classification accuracy is 95%, and the calculationamount is reduced by 60%.In [68], WSI is graded for neuroblastoma biopsy. The texture features obtainedfrom the segmentation components of the tissue are extracted and processed byan automatic classifier group. This automatic classifier group Multiple Classifiers: k NN, linear discriminant analysis (LDA) & k NN, LDA & nearest mean (NM),correlation LDA (CORRLDA) & k NN, CORRLDA & NM, LDA & Bayesian andSVM with a linear kernel. The output of multiple classifiers is then selected us-ing a simple two-step classifier combination mechanism consisting of voting andweighting processes. The automatic classifier group is trained in multi-resolutionframe with different levels of differentiation. The trained classification system istested on 33 WSIs. Finally, the classification accuracy is 87 . itle Suppressed Due to Excessive Length 51 In [256], basal cell carcinoma WSIs are studied as an integrated unsupervisedcharacteristic. First, a set of feature detectors are learned from a set of patchesrandomly sampled in the image set. The detectors will capture the most commonpatterns by simulating the automatic encoder neural network. The image is thenrepresented using the convolution or BOF method. This representation is achievedusing the feature detector learned in the previous step. Next, the representationobtained from the convolution or BOF method to train the binary classificationmodel, the softmax regression classifier. Basal cell carcinoma includes differentcategories of cancer and non-cancer carcinoma. The final result of the system isshown in Figure. 34. The best results in AUC are obtained, which are superior tothe most advanced 7% and 98 . .
1% ( P (cid:28) . In this part, the classification is mainly introduced, that is, the ROI include fivediagnostic categories. The classifier is designed based on CNN, which uses thefeatures learned by CNN to classify the detected ROI. The CNN structure diagramdesigned is shown in Figure. 35. Then the post-processing of WSI classification iscarried out. WSIs are classified according to the prediction of most categories in the remaining cancer areas. The results show that the efficiency is improved byabout 6.6 times with sufficient accuracy.Fig. 35: Designs of the CNN. This figure corresponds to Figure.4.6 in [139].In [140], it mainly includes two parts, namely, the detection and classificationof cell nuclei. The detection part proposes a spatially constrained CNN for nucleardetection. The details are in Sec. 6. Classification uses a new NEP combined withCNN. None of the proposed detection and classification methods require nuclearsegmentation. And the proposed method is applied to colorectal adenocarcinomaWSIs, and the final classification result obtains a higher F1 score. The final detec-tion and classification results are shown in the Figure. 36.
Fig. 36: The detection and classification are the result graphs under 20 timesmagnification. This figure corresponds to Figure.9.(c) in [140]. itle Suppressed Due to Excessive Length 53
In [141], an expectation maximization method is proposed. The spatial re-lationship of patches is used to locate the distinguished patches robustly. Thismethod is applied to the subtype classification of glioma and non-small cell lungcancer. The classification module uses patch-level CNN and trains a decision fu-sion model as a two-level model. The first level (patch-level) model is based onexpectation maximization, combined with CNN output patch-level prediction. Inthe second level (image-level), the patch-level predicted histogram is input to theimage-level multiple logistic regression or SVM.In [43], a new dataset, Kimia Path24, is introduced for image classificationand retrieval in digital pathology. The WSIs of 24 different textures are generatedto test patches. Especially, the patch classification is based on LBP histograms,bag-of-words approach, and CNN.In [145], a method to classify WSI for breast biopsies using CNN is proposed.The proposed network architecture can retrieve information on different scales.First, the image includes 12 non-overlapping patches, and then the piece-by-piecetraining CNN and CNN+SVM classifier are used to calculate the patch-level prob-ability. Finally, one of three different patch probability fusion methods is usedto obtain image classification results. These three methods are majority voting(choosing the most common patch as the image tag), maximum probability (choos-ing the patch category with high probability as the image tag), and probability(the category with the largest sum of patch-level probability). The final resultsclassify the images into four categories: invasive, in situ, benign and normal. Theproposed system achieves the overall sensitivity of about 81% of cancer patchclassification.In [146], a context-aware stacked CNN is proposed to classify WSIs as nor-mal/benign, DCIS, and invasive ductal carcinoma (IDC). The first is to train aCNN and use high pixel resolution to capture cell level information. The charac-teristic responses generated by the model are then inputted to the second CNNand superimposed on the first CNN. The system had an AUC of 0.962 for binaryclassification of non-malignant and malignant slides, and a three-level accuracyrate of 81 .
3% for normal/benign classification of WSIs, DCIS, and IDC.In [147], gastric cancer WSI is automatically classified. The traditional imageanalysis method and deep learning method are proposed and compared quantita-tively. In the traditional analysis method, GLCM, Gabor filter bank response, LBPhistogram, gray level histogram, HSV histogram, and RGB histogram are used forclassification and RF. In terms of the deep learning method, AlexNet is proposedas a deep convolution framework. The structure of the network is shown in Fig-ure. 37. According to the experiment, the overall classification accuracy of thecancer classification proposed by AlexNet is 0.6990, and the overall classificationaccuracy of necrosis detection is 0.8144.In [22], different types of colorectal polyps on WSIs are classified to help pathol-ogists diagnose them. Here is a modified version of the ResNet structure. The wholeWSI is divided into patches and then applied to the ResNet. If at least five patcheson a WSI are recognized as this class, the average confidence level is 70%. If there is no cancer type in the patch, the WSI is considered normal. Finally, the accuracyis 93 . . . Fig. 37: Proposed AlexNet architecture. This figure corresponds to Figure.4in [147].data is passed through the 11-layers of CNN model. Details of the structure areshown in Figure. 38. Finally, the classification accuracy rate reached 47 .
31% in thetest data and 100% in the training data.Fig. 38: The deep neural network architecture and visualization of intermediatelayers features. This figure corresponds to Figure.4 in [121].In [149], a deep CNN is proposed to conduct large-scale histopathological im-age classification, segmentation, and visualization. In the part of the classification,WSI is first divided into patches, and background is discarded, and then the se-lected patches are input into the network to obtain 4096-dimensional CNN featurevectors. The final feature vectors of the image are assembled through softmax.Then feature selection is carried out to remove redundant and irrelevant features.Finally, SVM is used for classification. When using the MICCAI challenge dataset,the classification accuracy is 97 . last fully connected layer with a convolutional layer. Finally, the average accuracyrate is 91 . itle Suppressed Due to Excessive Length 55 on the dataset. Then the network is trained on the patch dataset, and the trainednetwork structure is shown in Figure. 39Fig. 39: Architecture of the DCN used in approach. This figure corresponds toFigure.1 in [151].In [152], a network structure based on CNN is proposed. The tissue section areaof WSI is analyzed with multiple resolution methods. That is, the class posteriorestimate of each view at a specific magnification is obtained from the CNN at aspecific magnification, and then the posterior-estimate of random multiple viewsat a multiple magnification is voted to filter to provide a slide-level diagnosis.According to the experimental results, the final classification accuracy is 94 . ± . . ± . . ± . . ± . tion output diagram is generated as shown in Figure. 41.In [159], a two-part approach is proposed to classify WSIs. Firstly, the encoderis trained in an unsupervised way, and the tissue blocks on WSI are mapped to theembedded vector, and the sliding window is used to form the stack of the featuremap of WSI. Then, the CNN classifier is trained based on the compact repre- Fig. 40: The architecture of the networks for the unsupervised domain adaptation.This figure corresponds to Figure.1 in [257].Fig. 41: The classification result of the proposed method in Camelyon16 dataset.This figure corresponds to Figure.4 in [155].sentation of WSIs. There are three types of encoders trained here: convolutionalautoencoder (CAE), variational autoencoder (VAE) and a new method based oncontrast training. Experiments show that the new contrast encoder is superior toCAE and VAE.In [88], WSI colorectal cancer tumor tissue is classified. The simple linear it-erative clustering (SLIC) algorithm is proposed by Achanta et al. [232] is mainly applied to generate superpixels, and superpixels are used to annotate the image.Therefore, the selected area is divided into superpixels. Then, color texture fea-tures are extracted and dimensionality reduction is used to regenerate compositefeatures. Experiment shows that the superpixel method is suitable for using dif-ferent classification algorithms based on machine learning. itle Suppressed Due to Excessive Length 57
In [161], a multi classification of breast cancer in WSI is presented. They pro-pose a deep learning framework, which is mainly divided into two stages, usingmicroscopic images and WSI to achieve the purpose of classification. After thetwo types of images are patched, the microscopic images are used for Inception-ResNet-v2 to train the classifier. The WSIs are then subsampled and convertedfrom RGB to CIE-LAB color space and segments the foreground from and back-ground. Then the extraction of hard examples and patch classifier is retrained. Theprediction results are aggregated from the block by block prediction back to im-age prediction and WSI annotation. This method is applied to ICIAR 2018 GrandChallenge on breast cancer histology images, with an accuracy rate of 87%, farexceeding the second place. The specific working process is shown in Figure. 42.Fig. 42: Overview of the framework. This figure corresponds to Figure.1 in [161].
In [86], a weakly supervised method for multi-classification of breast histopathol-ogy WSIs is proposed. Firstly, ROI is extracted by zooming, translation, and fixa-tion. Then, the color and texture information and structural features are extractedin CIE-LAB space. Next, four different multi-instance multi-label (MIML) learn- ing algorithms are used to predict the slide level and ROI level in the image. Theresult is an average classification accuracy of 78% at slide level 5-Class.The author in [162] also proposes a multiple instance learning (MIL) frame-work based on breast cancer WSI. This framework is based on CNN and intro-duces a new pooling layer that enables the patch in WSI to aggregate the mostinformative features. This pooling layer is a new layer of the multi-instance pool(MIP), which introduces MIL as an end-to-end learning process into DNN torealize WSIs classification. At the end, high classification sensitivity of 93 . . . .
95% are achieved with four different magnifications usingthe public dataset.The study in [164] is also based on the MIL classification for prostate cancer.Slide tiling is the first run at different magnification and its bags are generated.Then, model training is carried out to find the tiling with the highest positiveand negative probability in the slide, which is used to give more importance tothe under-representative examples. Then, based on AlexNet, ResNet, VGG clas-sification, given a threshold, if at least one instance is positive, then the WSI iscalled positive and the slide is negative if all instances are negative. The optimalmodels are Resnet34 and VGG11-BN, with AUC of 0.976 and 0.977, respectively.The method of [172] is roughly the same as that of [164].In [166], a weakly supervised learning approach is used to classify WSI lungcancer. Firstly, the improved FCN based on patch-level from WSI is used for cancerprediction to find the different regions as the prediction model of patch-level. Whenthe probability of a block exceeds the threshold, it is retrieved. Then, context-based feature selection and aggregation from the retrieved parts are performedto construct the global feature descriptor. Finally, the global feature descriptoris input into the standard RF classifier. Finally, a high classification accuracy of97 .
1% is obtained.In [167], CNN is mainly used to classify gastric cancer in WSI. Firstly, thedifference of threshold value and color feature is used to extract the tissue andconduct morphological processing. Then, it is separated into patches, and the datais expanded by flipping. Then, the existing CNN architecture is utilized to conductexperiments on patch-level and slide-level. Finally, good results are obtained onDenseNet-201.In [170], a multi-resolution classification of prostate WSIs based on attentionmechanism is performed. The MIL model based on attention is used to extracttransient features. The overall process is separated into two parts. The first partis to classify cancer and non-cancer, using attention-based clustering tile selection.The second part is to conduct cancer classification research with higher resolution.Finally, an average classification accuracy rate of 85 .
11% is obtained, and the newperformance of prostate cancer classification is realized.In [179], multi-instance deep learning is also used to classify WSI gastric cancerimages. A method named RMDL method is proposed. Similar to the above MIL method, it separated into two stages. The first phase also trains a localizationnetwork to select the discriminative instances. In the second stage, the RMDLnetwork is used for image-level label prediction. The network is composed of local-global feature fusion, instance recalibration, and multi-instance pooling modules.The specific working process is shown in Figure. 43. itle Suppressed Due to Excessive Length 59
Fig. 43: Overview of the framework. This figure corresponds to Figure.2 in [179].5.3 Others Classification MethodAmong the papers we have reviewed, seven papers are based on classification [53,80, 93, 95, 116, 125, 259], and they involve techniques related to machine learning.In [93], a machine vision system is developed, which uses its morphological fea-tures and texture analysis to classify the sub-regions in WSIs. However, it focuseson the application of features and does not introduce the classification part of themachine vision system in detail. The study in [116] also do not make a specialdescription of the classifier, but only focused on the features.In [53], the LNKNET software package is used to classify breast cancer WSIs.LNKNet integrates neural networks, statistical and machine learning classification,clustering, and feature selection algorithms into a modular package.In [95], a classification framework is proposed to classify the degree of neurob-lastoma differentiation. The main point here is that a new method of structuringstructural features are introduced, and its classification method is also based onprobability, namely mapping decision rules.In [80], the pixel-based stain classification is carried out to classify the IHCstain and the nearest neighbor classification, mainly using the nearest neighbor andmorphological methods. Then, the density distribution of the identified IHC stains are calculated by the kernel density estimator. After that the staining distributionon WSI is obtained.Both [259] and [125] are used to evaluate the classification accuracy of an au-tomated image analysis system, E-pathologist. Among them, [259] is for colorectalbiopsy, and [125] is for gastric biopsy. k NN.Since 2008, traditional machine learning algorithms have taken the mainstream po-sition in WSI classification. In 2015, deep learning algorithms began to be widelyused, obtaining classification results with higher accuracy than traditional ma-chine learning algorithms. Among them, MIL has many applications. Table. 6 is asummary of the CAD method of classification technology in WSI. itle Suppressed Due to Excessive Length 61
Table 6: Summary of the CAD methods used for classification in WSI.
Method ReferenceYear Team DetailsSVM [98] 2011Matthew D. DiFranco RF, linear SVMs, RBF SVMs are used to classifya collection of 19 B * channel featuresSVM [73] 2013 Nandita Nayak Multi-class regularized SVM classificationwith regularized parameters 1 and 3 polynomial kernelsSVM [104] 2013 Liping Jiao SVMSVM [240] 2015 Cheng Lu SVMSVM [82] 2015 Michaela Weingant RF feature selection, class balancetraining sample subsampling andSVM classificationSVM [113] 2015 Peikari M An SVM with a RBF kernelSVM [118] 2016 Gadermayr M The efficient linear C-SVMSVM [241] 2017 Shukla P SVMSVM [124] 2018 Hongming Xu MSVMRF [76] 2013 Andr´e Homeyer Naive Bayes classifier, k NNclassifier, RF classifierRF [137] 2016 Sharma H \ RF [138] 2016 Dayong Wang The post processing uses theRF classifier for classificationRF [253] 2017 Jamaluddin M F \ RF [128] 2019 S Klimov \ k NN [94] 2008 Olcay Sertel A multi-resolution classification system isdeveloped based on the improved k NN classifier k NN [96] 2009 O Sertel, J Kong Multi-resolution decomposition of trainingimages is performed by Gaussianpyramid method, and improved k NNBayesian [55] 2012 Scott Doyle Strong Bayesian multi-resolution classifierMixed classifiers [68] 2009 J Kong, O Sertel A simple two-step classifier combinationmechanism consisting of voting and weightingprocesses is chosen to aggregatethe output of multiple classifiers.DL [256] 2015 John Arevalo Softmax,Linear-SVMDL [84] 2016 Barker J Elastic Net classifierDL [139] 2016 Ge¸cer B CNNDL [140] 2016 Sirinukunwattana K NEP is combined with CNNDL [141] 2016 Hou L CNN,EM algorithm, SVMDL [43] 2017 Babaie M Applied LBP, the dictionary approachand CNNs to classify patchesDL [145] 2017 Ara´ujo T CNNDL [146] 2017 Bejnordi B E Contextual Awareness Stacked CNNDL [147] 2017 Sharma H The AlexNet deep convolutional comparisonwith RF is made, and a CNN architecture is proposedDL [22] 2017 Korbar B ResnetDL [121] 2017 Hu J X CNNDL [149] 2017 Xu Y CNNDL [150] 2017 Korbar B ResNetDL [151] 2017 Ghosh A AlexNetDL [152] 2017 Das K, Karri S P K CNN
Table 6: Continue: Summary of the CAD methods used for classification inWSI(Deep learning (DL)).
DL [257] 2018 Jian Ren Siamese neural network andGAN networkDL [155] 2018 Courtiol P Modify the pre-trained DCNN modelto introduce an additional setof full connection layers for contextreclassification from instancesDL [157] 2018 Baris Gecer CNNDL [159] 2018 David Tellez CNNDL [88] 2018 Mork¯unas M CNNDL [161] 2018 Scotty Kwok Inception-Resnet-v2DL [86] 2018 Caner Mercan MIML frameworkDL [162] 2018 Kausik Das Based on VGGDL [164] 2018 Campanella G AlexNet, VGG11-BN, ResNet18,ResNet34 (MIL)DL [166] 2018 X Wang CNN (find discriminative region)RF (classification)DL [167] 2018 Junni Shou CNNDL [170] 2019 Jiayun Li Attention-based clustering and CNNDL [172] 2019 Gabriele Campanella CNN RNNDL [179] 2019 Shujun Wang New RMDL networkOthers [93] 2004 JamesDiamondPhDa A machine vision system is developed, butthe emphasis is on the application of featuresOthers [53] 2006 Sokol Petushi LNKNET packageOthers [95] 2009 O Sertel Classified according to the distance betweenclasses and the distance between classesOthers [80] 2014 Fang-Cheng Yeh Classification to calculate IHCstain density distributionOthers [116] 2016 Harder N Emphasis is on the applicationof featuresOthers [259] 2017 Yoshida H Compare human pathologist and E-PathologistOthers [125] 2018 Hiroshi Yoshida E-Pathologist
To completely understand an image, one should accurately estimate the objects’concept and location in each image [260]. Detection determines whether one ormore specific category instances exist in an image or not [261]. Detection canprovide valuable information for various fields. For example, remote sensing imagedetection [262] can provide useful information related to geology, meteorology,water conservancy, and face recognition [263]. It is widely used in military and public security criminal investigations, and biomedical image detection [264] makeit convenient for doctors in clinical diagnosis and pathological research.As one of the most common tasks for CAD pathologists to view WSI, thedetection method has developed rapidly in recent years. Figure. 2, the number ofdetection cases is increasing from 2009 to 2019, which reflects the development of itle Suppressed Due to Excessive Length 63 detection technology. Besides, the basic content of the CAD view WSI detectionmethod is shown in Figure. 44. As shown in Figure. 44, all CAD WSI methods fordisease detection can be roughly divided into three categories. The first category istraditional detection methods (such as SVM, image enhancement, etc.), the secondcategory is an ensemble learning method, and the last category is deep learningmethod.
Traditional Deep LearningDetection Method
Support Vector Machine (SVM)Random ForestDecision TreeImage Enhancement Convolutional Neural Networks(CNN)Fully Convolutional Neural Network(FCN) ………
Ensemble Learning
RankboostAdaBoost
Fig. 44: The structure of diseases detection methods in CAD.6.1 Traditional Detection MethodIn this part, we select the traditional detection methods that appear in the papersand summarize these methods.
In [104], colon cancer is detected by WSI, use SVM to develop a classifier, andselect 18 simple features (such as gray-level variance and gray-level mean) and 16texture features (such as GLCM) to form features. Finally, the experiment uses 3-fold cross-validation and achieves an average precision of 96 . . . . In [127], detect glomeruli on WSI of thin kidney tissue biopsy, LBP featurevector adaptation is used to train the SVM model, the experimental results haveobtained high precision ( > > divide the pixels into normal areas and bleeding areas, and then detect in WSIby applying threshold method and extended area minimum method in the areaof tumor proliferation, the experimental result shows that automatic evaluation isbetter than manual evaluation.In [102], since there is no previous information about cancer, a fusion of cell andstructural features are used, and a patch-based method is used to detect prostatecancer using an SVM classifier with a RBF and a parameter c of 1. When thethreshold is 90 pixels, it provides the most satisfactory detection results (TPR =78%, FPR = 6%). In [122], the WSI rough segmentation simplifies the background, and the coloris normalized, the adaptive threshold method is used to obtain the binary image.Moreover, the watershed method is used to divide the nucleus. Then, LBP featuresare extracted. After, different models are used for metastasis detection. Accordingto the experimental results, the RF achieves an AUC value of 0.97.In [81], the SLIC algorithm is used to calculate superpixels, and after removingthe pixel background, the color and LBP features of the remaining tissue pixels arecalculated. Then, the RF classifier is used to detect the superpixels that are mostlikely to be cancer, and use the RF to input the highest resolution graphics andgland features to detect the area that may have cancer. The experimental resultThe experimental result obtaines an AUC of 0.96, and reaches 0.4 specificity at1.0 sensitivity.In [107], Similar to [81], both use the multi-scale superpixel classificationmethod. The experimental result of detecting breast cancer is that the ROC is0.958, and the AUC for the tile analysis in comparison is 0.932. In [114], similarto [81] and [107], it detects DCIS in the WSIs of breast histopathology.In [83], the superpixel classification method is also used to obtain the initialrecognition of the ROI by gathering superpixels at low magnification. Then, bymarking the corresponding pixels, the superpixels are mapped to the higher mag-nification image. This process is repeated several times until the segmentationis stable. Finally, the RF classifier and SVM respectively mark the superpixelsrepresented by the selected features and quickly detect the ROI in the WSI. Ex-periments prove that the superpixel results suggested in the article are better thanSLIC and non-superpixel.
There is an article each using DT classifier, clustering algorithm, linear discrimi-nator and machine learning organization classifier [74, 117, 135, 265].The author in [135] uses the multiple sharpness feature method to identifythe blur area on the WSI. Using different blur features (such as features related to the co-occurrence matrix and image gradient), the best blur detection results areobtained by using a DT classifier (i.e., 98 .
56% and 96 .
63% classification accuracyand small hardware investment).In [265], Ki67 immunohistochemistry-based WSI technology is used to detecttumor areas with high proliferation activity, a hybrid clustering method, referred to itle Suppressed Due to Excessive Length 65 as Seedlink, is developed in the paper. This tool greatly improves the pathologist’sidentification of hot-spots consistency.In [74], the author uses the level set method to extract candidate objects, anduse two sets of features (the baseline set describing the size, shape, and color of theextracted candidate objects and the extension of texture information in additionto the baseline set) input into the linear discriminator, divide candidate objectsinto mitotic figures or fake objects to detect mitosis.In [117], the staining and texture characteristics of Ki67 stained glass slides isused to divide the tissues in WSI into living tumor tissue, necrotic tissue, and back-ground. And use a tissue classifier based on machine learning, which is trained fivetimes magnified images to be applied to living tumor tissue to detect phenotypicchanges. In the end, an accuracy of 95% in each area is achieved.We found three papers about non-machine learning. In [266], by introducing amovement operator, it can effectively amplify the chromaticity difference betweentissue folds and other tissue components to detect tissue folds in WSI. In [267],similar to [266], the weighted difference between the color saturation and brightnessof the image pixels is used as the offset factor of the original RGB color of theimage to enlarge tissue folds.In [268], the author compares the histopathological characteristics of dermatitiscases detected by WSI and traditional microscopy. Although WSI is not as effectiveas traditional microscopes, it is sufficient to examine the pathological features oftenencounter in dermatitis cases and get the effect that can be used.6.2 Deep Learning based Detection MethodsIn this section, the relevant content of using deep learning algorithms to detecthistopathological images with WSI is briefly summarized.
The following [47, 143, 154, 168, 180, 253, 269] are all detection classifiers based onCNN. In [154], CRF is applied to the latent space of the trained deep CNN, andthe compact features extracted from the middle layer of the CNN are regardedas observations in the fully connected CRF model to detect invasive breast can-cer. Experiments show that the average FROC score of tumor area detection inhistopathology WSIs increased by about 3 . . . .
77% in terms of pixel-by-pixel evaluation compared to manually annotated regions of IDC.In [47], the author evaluates the performance of automated deep learning algo-rithms in detecting metastasis in lymph node H&E tissue sections of women withbreast cancer and compares it with a pathologist’s diagnosis in a diagnostic envi-ronment. The dataset is collected from 399 patients who underwent breast cancer surgery at the Radboud University Medical Center (RUMC) and Utrecht Univer-sity Medical Center (UMCU) in the Netherlands. The algorithm in the paper issignificantly better than the pathologist’s artificial algorithm.In [168], a method for training and evaluating CNNs in breast cancer WSIsmitosis detection is proposed. In the three tasks challenged by Tupac, the perfor-mance of the proposed method is independently evaluated.In [180], an autofocus quality detector called ConFocus is developed to de-tect and quantify the out-of-focus area and severity on WSIs. When compared topathologist-graded focus quality, ConvFocus achieved Spearman rank coefficientsof 0.81 and 0.94 on two scanners and reproduced the expected OOF patterns fromZ-stack scanning. The article also evaluates the impact of OOF on the accuracyof the most advanced metastatic breast cancer detector, and finds that as OOFincreases, the performance continues to decline.[253] is divided into two parts. The first part is to use CNN to detect possibletumor locations in WSI, and the second part is to use the detected results toextract features to classify normal or tumors. Using Camelyon16 dataset, whichconsists of 160 negatives and 110 positives WSIs for training, and 50 positivesand 80 negatives for testing. The method has a better AUC result at 0.94 thanthe winner of Camelyon16 Challenge with an AUC of 0.925. The CNN structuredesigned in [253] is shown in Figure. 45.Fig. 45: 12 convolutional layer including the fully connected layer. This model wasalso inspired from VGG model.In [269], CNN is trained to detect IDC in WSI. At the end, 71 .
80% of F-measure (F1) and 84 .
23% of balanced accuracy are obtained. The overall detectionframework of [269] is shown in Figure. 46.
Fig. 46: Overall detection framework of [269]. itle Suppressed Due to Excessive Length 67
The following [139,157,163,176] are processed with 4-layers of FCN to achieve thepurpose of detection.The study in [139] is composed of two tasks, retrieval and classification. The re-trieval part is composed of four layers of FCN. Through the feedforward processingof FCN-1, the salient area is detected from WSI, and each connected componentabove the threshold is enlarged on the input image and processed by FCN-2. Thisprocess lasts four times, and a significant area of WSI can be detected. Then CNNclassifies and finally determines the breast cancer diagnosis results.In [157], a system is proposed for diagnosing WSI of breast biopsy. Firstly, fourFCNs are used for saliency detection and multi-scale localization of ROI. Then,convolutional network is used for cancer classification. Finally, the saliency mapand the classification map are combined. The test result shows that its accuracyrate is roughly indistinguishable from the pathologist’s prediction.In [163], an improved FCN layer is used to input the WSI of any size, and thestandard FCN layers are converted into the anchor layer. Fast and dense ScanNetinference with the anchor layer makes the network faster. The result of its detectionof cancer metastasis shows excellent performance on the Camelyon16 Challengedataset.Similar method is applied in [163] and [176] .
The following references [87, 140, 175] are used for detection after combining orsimplifying the CNN structure with other methods.In [87], a new type of efficient adaptive sampling based on probability gradientand quasi-Monte Carlo sampling is used, combined with a CNN-based classifier.Applied to the detection of invasive breast cancer on WSI. The experimental resultshows that the Dice-coefficient is 76%, which is an efficient strategy.In [140], in conventional colon cancer WSI proposes a space-constrained CNNfor nuclear detection. The new NEP is used in conjunction with CNN to moreaccurately predict the type of cell detection. The test result shows that this articleproduces an higher average F1 score of detection.In [175], a simplified CNN, named PathCNN, is used to detect outliers of WSIin whole cancer. The WSI used in the experiment is downloaded from the genomedata sharing database of the TCGA.In [171], Lymph Node Assistant (LYNA), a tool for breast cancer lymph nodemetastasis detection based on Inception-v3, is evaluated for its application andclinical practice. The data is obtained from two sources: the Camelyon16 challengecontaining 399 slides, and a private dataset containing 108 slides from 20 patients(86 tissue blocks). When applying the second dataset, LYNA achieved an AUC of99 . In [20], the image is sampled based on the kernel density, and the taint de-convolution and feature description are used to extract image features. Then, anenhanced version of the rank boost integrated method (using multiple weak classi-fiers to obtain better performance of the final rank) is utilized to rank and detecthigh-level prostate cancer. The experiment shows the mean AUC is 0 . ± . . ± . itle Suppressed Due to Excessive Length 69 Table 7: Summary of the CAD methods used for detection in WSI (Traditional(T), Deep learning (DL), Ensemble Learning (EL), Breast Cancer SurveillanceConsortium (BCSC)).
TypeReferenceYear Team Data DetailsT [102] 2011 Kien Nguyen \ SVM with RBF kernel and c = 1T [104] 2013 Liping Jiao \ SVMT [108] 2015 Harshita Sharma \ SVMT [54] 2015 Zaneta Swiderska \ SVM with Gaussian kernel functionT [126] 2018 W. Han \ SVMT [127] 2018 Olivier Simon \ SVMT [81] 2015 Litjens, G \ RFT [83] 2015 Ruoyu Li NLST RFT [107] 2015 Bejnordi, B. E \ RFT [114] 2016 Bejnordi, B. E \ RFT [122] 2017 Valkonen M \ RFT [266] 2009 PA Bautista Massachusetts General Hospital Color shift factorT [267] 2010 PA Bautista Massachusetts General Hospital Color shift factorT [74] 2012 M. Veta UMCU Linear discriminatorT [265] 2012 Lopez X M \ A hybrid clustering methodT [135] 2013 Lopez X M \ DTT [117] 2016 Shirinifard A \ A tissue classifier based on machine learningT [268] 2016 Vyas N S \ Compare WSI and traditional microscopyDL [269] 2014 Angel Cruz-Roa \ CNNDL [253] 2017 Jamaluddin M F Camelyon16 CNNDL [47] 2017 Bejnordi, B. E RUMC and UMCU CNNDL [143] 2017 Angel Cruz-Roa TCGA CNNDL [154] 2018Farhad Ghazvinian Zanjani Camelyon17 CNNDL [168] 2018 David Tellez TNBC,TUPAC CNNDL [180] 2019 T. Kohlberger \ CNNDL [139] 2016 Ge¸cer B \ FCNDL [157] 2018 Ge¸cer B registries associated withthe BCSC Consortium FCNDL [163] 2018 Huangjing Lin Camelyon16 FCNDL [176] 2019 Huangjing Lin Camelyon16 FCNDL [140] 2016 Sirinukunwattana K \ New NEP is used inconjunction with CNNDL [87] 2018 Angel Cruz-Roa HUP,CWRU,CINJ,TCGA CNN and adaptive samplingDL [175] 2019 S Bilaloglu TCGA PathCNNDL [171] 2019 Liu Y Camelyon16,a separate dataset LYNA (Based on Inception-v3)EL [55] 2010 S. Doyle \ AdaBoostEL [20] 2017 Huang C H TCGA Rankboost
This section analyzes prominent methods in different tasks.
Method typeDeep learning Popular method
Multi- resolution U-net
The structural characteristics of the method
U-shaped structure and skip- connection
Common tasksSegmentation
Fig. 47: The popular methods for segmentation in WSIs.Its U-shaped structure and its skip-connection are its structural characteristics.The U-shaped structure can be used to extract deep features by down-samplingand down-dimension-reduction, and then up-sampling to obtain more accurateoutput images. This kind of end-to-end network can achieve good results in medicalimage segmentation. There is another multi-resolution encoder-decoder networkfor breast cancer segmentation [21], which is similar to the U-net method. However,U-net structure can only be predicted on a single scale, so it cannot cope with scalechanges well. Moreover, training does not have good generalization ability when theconvolutional layer is increased [270]. Breast cancer is the most commonly used [21,64, 97, 106, 134, 196, 197, 222, 229] tasks in segmenting WSI to assist pathologists indiagnosis.7.2 Analysis of Classification and Detection Applications in WSIAccording to the review of classification applications of WSI in CAD, it can be seenthat SVM is the most frequently used technique in traditional machine learning.However, basic SVM cannot achieve excellent classification results. For example,in [73], multi-class regularized SVM is used to classify tumors, and the accuracyrate is only 84%. In [113], the RBF SVM is used for classification, and only 0.87AUC is obtained. However, if SVM is combined with SVM of other kernel functionsor other kinds of classifiers to train ensemble learning classifiers, the results will behighly improved. For example, in [98], the AUC of prostate cancer is 0.95 when it is classified. However, this ensemble learning method has low efficiency, slow runningspeed, and requires a large number of parameters. For some more complex tasks,a large number of calculations are needed [271].In the traditional machine learning algorithm, the Gaussian pyramid methodis also commonly combined with k NN to improve the classification accuracy. The itle Suppressed Due to Excessive Length 71
Gaussian pyramid is based on simple down-sampling plus Gaussian filtering. Theoriginal image is continuously sampled by decreasing order, and a series of imagesof different sizes are obtained, from large to small, from bottom to top, to forma tower model. In this way, images of different resolutions can be obtained, thusimproving the accuracy of classification. For example, in [94], the improved k NNcombined with the Gaussian pyramid is used to classify neuroblastoma, and theclassification accuracy of 95% is obtained. However, in the classification problemof unbalanced datasets, the defects of k NN are obvious. Due to the influence ofsample distribution, the minority class will be more biased towards the majorityclass discrimination [272].In the deep learning classification algorithm, most of them are based on CNN.CNN is better than traditional machine learning methods in processing high-dimensional data, and because of the convolutional layer, it can automaticallyextract features for learning. In recent years, MIL and neural networks are oftenused to carry out classification tasks for medical WSIs. For example [86, 162, 164,172, 179]. MIL is a learning problem with Multiple example packages as trainingunits. A bag is marked as a positive-class multi-example package if it containsat least one positive instance. Conversely, the negative example is also true. Themulti-example learning method can effectively reduce the noise and improve theclassification accuracy of the prediction. In [164,172], the AUC of its classificationis 0.98, a good result. However, the CNN-based neural network is an end-to-endarchitecture, similar to a black box, which is weak in interpretation [273]. Thepopular methods in classification for WSIs are as shown in Figure. 48.
Method type
Deep learning Popular methodSVM Combined approachOther classifiersTraditional machine learning
KNNBased on CNN Gaussian pyramidMultiple instance learning Characteristics of method
Classifier complementation
Multiscale samplingReduce non-target interference
Practical task
Classification
Fig. 48: The popular methods in classification for WSIs.In the task of WSIs classification, the classification of breast cancer [113, 138,139,145,146,157,161,162], prostate cancer [55,82,98,170,257], and colon cancer [22,88,104,140] is commonly used. Gastric cancer [137,147,167], neuroblastoma [68,94], and melanocytic tumor on skin [124, 240] have also been studied.There are many researches on medical WSIs detection that are related to clas-sification. For example, [139, 140, 157, 253], they are all detected first and thenclassified. Most of the tasks that use WSIs alone as an adjunct to treatment arefor breast cancer detection [74, 87, 122, 143, 154, 163, 168, 171].
In addition to the feature extraction methods we reviewed, there are some otherfeature extraction methods that grab our attention and can be used in WSI tech-nology.In [274], a method is proposed to extract the structural features of buildingfacades through texture fusion. After texture fusion, the gradient amplitude ofelements reduced, and the gradient amplitude of structural features can be keptconstant. The interference of texture to structural feature extraction can be elim-inated by this method of texture fusion. If we apply this method to the extractionof traditional features, it may improve the availability of features and get betterresults.In [275], LeNet, AlexNet, and VGG-16 based deep learning models are used torealize the detection of lung cancer. This experiment is applied to the ComputedTomography (CT) image dataset. The combination of the AlexNet model and k NN classifier is used to obtain the best accuracy of 98 . . There are some segmentation methods that are worth noting in other fields andcan be tried with WSI for CAD.In [278], a method of gastric histopathological image segmentation based onthe layered CRF is introduced. This method can automatically locate the cancernest information in the stomach image, and because the CRF can represent the spatial relationship, a higher order term can be established on this basis and ap-plied to the image based post-processing, which further improves the segmentationperformance. The model shows high subdivision performance and effectiveness. In[279], CRF is also applied to the segmentation of environmental microorganismImages. So we can also apply it to WSI technology. itle Suppressed Due to Excessive Length 73
In [280], an image segmentation method is presented that is faster than su-perpixel is proposed. It separates into dense and sparse methods. Then, a newintensive method can achieve superior boundary adherence by exploring alterna-tive mid-level segmentation strategies are proposed. This method is a very effectivehierarchical segmentation method. But in this case, it applied to natural images,and we can also try to apply it to WSI.In [281], a multi-channel weighted region scalable fitting (M-WRSF) segmenta-tion model for medical image segmentation is proposed. In this M-WRSF model,a new penalty term is introduced to improve the numerical stability and the timeinterval is increased to improve the iteration efficiency. The new edge detectionfunction is used to improve the segmentation performance. Based on the originalmodel, the Gaussian kernel function is added to enhance the robustness.In [282], a method of automatic segmentation of coronary artery based ongrowing algorithm is proposed. Firstly, 2D U-net is used to automatically locate theinitial seed points and the growth strategy, and then a growth algorithm combinedwith the 3D triangulation network is proposed. The improved 3D U-net is used forcoronary artery segmentation. This method adopts residual block and two-phasetraining. The input data of the network is set as the neighborhood block of theseed point. And according to the Iterative termination condition, it determineswhether the segmentation is stopped.
The following is an introduction to some classification methods used in other fields,which can be used in WSI for CAD.In [283], a spectral-spatial classification algorithm based on spectral-spatialfeature fusion of spatial coordinates is proposed to classify hyperspectral images.Active learning is introduced to improve performance. The method of combiningspectral information with spatial information can solve the noise interference.In [284], two new Privacy Supporting Binary Classifier Systems are proposedto classify the Magnetic Resonance Imaging (MRI) images of the brain. LSB Sub-stitution Steganographic method is used to protect the privacy of the patient. Wecan apply this approach to histopathological WSI. [285] also classifies hyperspec-tral images. Deep SVM is used, and the results obtained by this method are betterthan those obtained by other classifiers. Hopefully, this method can be extendedto WSI datasets.In [286], a Content-Based Microscopic Image Analysis (CBMIA) approachesare proposed to classify microscopic images of microorganisms. This method isbased on the computer semi-automatic or automatic method, so it is very effectiveand saves manpower and material resources. We can also apply this approach to medicine.In [287], a CNN-based gender classification method for near-infrared periocularimages is proposed. In other words, the neural network is used to extract featuresand SVM is used for classification. In other words, directly using neural networkclassification to achieve advanced performance.
There are a number of tests that are used in other areas that can be attemptedfor WSI for CAD.In [288], the target in the video is detected. The method used is based ontraditional background subtraction and artificial intelligence detection. Mask R-CNN is used to judge whether there is a segmentation object in the candidateregion and to segment it.In [289], a soft-computing based approach for automatic detection of pul-monary nodules is proposed. This method is applied to the CT images. Firstly,threshold processing, gray-scale morphology, and other preprocessing are used,and then random undersampling is used to deal with unbalanced problems. Then,a combination of particle swarm optimization (PSO) and stacking integration isused to detect.In [290], to deal with the complex scene of the target, a Feature Guide Networkis proposed. The multi-scale feature extraction module (MFEM) is used to obtainmulti-scale context information for each level of abstraction. Finally, a loss functionthat outperforms the widely used cross-entropy loss is designed. This methoddoes not require pretreatment and is efficient. [291] is also used for significancedetection, an artificial neural network regressor is trained to refine the significancemap. If applied to the histopathological WSI for significance detection is verypromising.In [292], semantic context is used to carry out multiple concept detection ofstill images. The first is to generate semantic descriptors using a set of test scoresfor a single concept. This advanced feature is pushed as input to the target multi-concept detector. The second method detects the target multiple concepts andtheir categories, and then aggregates the results of the two treatments. Combiningsemantic context with the use of features based on deep learning yields goodresults.
In this paper, image analysis methods based on machine learning using WSI tech-nology for CAD are summarized. The applied datasets, evaluation methods, fea-ture extraction, segmentation, classification, and detection in the task are ana-lyzed and summarized. By reviewing all the related works, we can find that themost frequently-used datasets in Sect. 2,feature extraction in Sect. 3, segmenta-tion methods in Sect. 4, classification methods in Sect. 5, and detection methodsin Sect. 6, respectively.Through the review of relevant work, we can find the commonly used meth-ods of these three tasks. With time and the progress of science and technology,deep learning algorithm has gradually replaced the traditional machine learning algorithm.TCGA [40] and Camelyon [41] are the two commonly used datasets in thecommon datasets summarized by us. In terms of feature extraction, color fea-tures, texture features, shape features, and deep learning features are the mostcommonly used. In the segmentation work, it separated into thresholding-based itle Suppressed Due to Excessive Length 75 segmentation, region-based segmentation, graph-based segmentation, clustering-based segmentation, deep learning related segmentation and other methods. Thesetraditional methods are simple to calculate, but sensitive to noise, so they are notrobust. And the segmentation method based on U-net has become the mainstreamin recent years. Classification work is the most studied. In the classification work,the combination of ensemble learning for the traditional classifier, MIL, and neu-ral network has better recognition ability. Most of the testing work is carried outtogether with the classification work. In addition, the deep learning method basedon CNN has achieved excellent performance in segmentation, classification, anddetection tasks, which will contribute to the early detection, diagnosis, and treat-ment of patients.In the future, the combination of WSI technology and machine learning to helppathologists assist in diagnosis is promising. In recent years, CAD research hasmainly focused on the breast, stomach, colon, and nervous systems, etc., and theresearch field can be expanded to a wider extent in the future. Second, there is stilla lack of large-scale, comprehensive, and fully annotated WSI datasets. Finally, itwould be very useful to develop a network that requires less computation, requiresless hardware and can be interpreted.
Acknowledgements
This work is supported by National Natural Science Foundation ofChina (No. 61806047). We thank Miss Zixian Li and Mr. Guoxian Li for their importantdiscussion. We also thank B.E. Xiaoming Zhou, B.E. Jinghua Zhang and B.E. Jining Li, fortheir Important technical supports.
Conflict of Interest
The authors declare that they have no conflict of interest.
References
1. N. Farahani, A.V. Parwani, L. Pantanowitz, Pathol Lab Med Int (23-33), 4321 (2015)1.1, 1, 1.1, 1.22. S. Al-Janabi, A. Huisman, M. Nap, R. Clarijs, P.J. Van Diest, Journal of clinical pathol-ogy (12), 1107 (2012) 1.1, 1.13. L. Pantanowitz, P.N. Valenstein, A.J. Evans, K.J. Kaplan, J.D. Pfeifer, D.C. Wilbur,L.C. Collins, T.J. Colgan, Journal of pathology informatics (2011) 1.1, 1.1, 1.34. E.A. El-Gabry, A.V. Parwani, L. Pantanowitz, Diagnostic Histopathology (12), 456(2014) 1.1, 1.15. A. Saco, J.A. Bombi, A. Garcia, J. Ram´ırez, J. Ordi, Pathobiology (2-3), 79 (2016)1.1, 1.16. L. Pantanowitz, J.H. Sinard, W.H. Henricks, L.A. Fatheree, A.B. Carter, L. Contis, B.A.Beckwith, A.J. Evans, A. Lal, A.V. Parwani, Archives of Pathology and LaboratoryMedicine (12), 1710 (2013) 1.17. R.S. Weinstein, A.R. Graham, L.C. Richter, G.P. Barker, E.A. Krupinski, A.M. Lopez,K.A. Erps, A.K. Bhattacharyya, Y. Yagi, J.R. Gilbertson, Human pathology (8), 1057(2009) 1.18. W. Amin, S.J. Srintrapun, A.V. Parwani, Expert Opinion on Medical Diagnostics (10),1173 (2008) 1, 1.19. C. Higgins, Biotechnic & Histochemistry (5), 341 (2015) 1.1, 1.210. M.G. Rojo, G.B. Garc´ıa, C.P. Mateos, J.G. Garc´ıa, M.C. Vicente, International journalof surgical pathology (4), 285 (2006) 1.16 Chen Li et al.11. F. Ghaznavi, A. Evans, A. Madabhushi, M. Feldman, Annual Review of Pathology: Mech-anisms of Disease , 331 (2013) 1.1, 1.312. P. Camparo, L. Egevad, F. Algaba, D.M. Berney, L. Boccon-Gibod, E. Comp´erat, A.J.Evans, R. Grobholz, G. Kristiansen, C. Langner, et al., Apmis (4), 298 (2012) 1.113. J. Webster, R. Dunstan, Veterinary pathology (1), 211 (2014) 1.114. T.C. Cornish, R.E. Swapp, K.J. Kaplan, Advances in anatomic pathology (3), 152(2012) 1.115. S. Al-Janabi, A. Huisman, A. Vink, R.J. Leguit, G.J.A. Offerhaus, F.J. Ten Kate, M.R.Van Dijk, P.J. Van Diest, Journal of clinical pathology (2), 152 (2012) 1.116. S. Singh, P. Gupta, M.K. Sharma, International Journal of Engineering Science andTechnology (5), 4228 (2010) 1.217. L.S. Goggin, R.H. Eikelboom, M.D. Atlas, Otolaryngology–Head and Neck Surgery (4 suppl), s21 (2007) 1.218. M.I. Fazal, M.E. Patel, J. Tye, Y. Gupta, European journal of radiology , 246 (2018)1.219. P.J. Lisboa, A.F. Taktak, Neural networks (4), 408 (2006) 1.220. C.H. Huang, D. Racoceanu, in Medical Imaging 2017: Digital Pathology , vol. 10140 (In-ternational Society for Optics and Photonics, 2017), vol. 10140, p. 101400A 1.2, 6.3, 6.4,721. S. Mehta, E. Mercan, J. Bartlett, D. Weaver, J. Elmore, L. Shapiro, in (IEEE, 2018), pp. 663–672 1.2,4.5, 4.5, 27, 28, 5, 7.122. B. Korbar, A.M. Olofson, A.P. Miraflor, C.M. Nicka, M.A. Suriawinata, L. Torresani,A.A. Suriawinata, S. Hassanpour, Journal of pathology informatics (2017) 1.2, 3.2, 3.2,4, 5.2, 6, 7.223. Y. Ma, Z. Jiang, H. Zhang, F. Xie, Y. Zheng, H. Shi, Y. Zhao, IEEE journal of biomedicaland health informatics (4), 1114 (2016) 1.224. R.S. Alomari, R. Allen, B. Sabata, V. Chaudhary, in Medical Imaging 2009: Computer-Aided Diagnosis , vol. 7260 (International Society for Optics and Photonics, 2009), vol.7260, p. 726016 1.225. C. Li,
Content-based microscopic image analysis , vol. 39 (Logos Verlag Berlin GmbH,2016) 1.226. Y. Huang, Y. Lei, Q. Wang, D. Li, L. Ma, L. Guo, M. Tang, G. Liu, Q. Yan, L. Shen,et al., Diagnostic pathology (1), 1 (2018) 1.327. B. Boyce, Biotechnic & Histochemistry (6), 381 (2017) 1.328. A.M. Boroujeni, A. Dehghani, E. Yousefi, R. Gupta, American Journal of Clinical Pathol-ogy (suppl 1) (2016) 1.329. M.N. Gurcan, L.E. Boucheron, A. Can, A. Madabhushi, N.M. Rajpoot, B. Yener, IEEEreviews in biomedical engineering , 147 (2009) 1.330. S. Kothari, J.H. Phan, T.H. Stokes, M.D. Wang, Journal of the American Medical Infor-matics Association (6), 1099 (2013) 1.331. M. Veta, J.P. Pluim, P.J. Van Diest, M.A. Viergever, IEEE Transactions on BiomedicalEngineering (5), 1400 (2014) 1.332. H. Sharma, N. Zerbe, S. Lohmann, K. Kayser, O. Hellwich, P. Hufnagl, Diagnostic pathol-ogy (1) (2015) 1.333. Z. Li, X. Zhang, H. M¨uller, S. Zhang, Medical image analysis , 66 (2018) 1.334. D. Komura, S. Ishikawa, Computational and structural biotechnology journal , 34(2018) 1.3, 535. H.Y. Chang, C.K. Jung, J.I. Woo, S. Lee, J. Cho, S.W. Kim, T.Y. Kwak, Journal ofpathology and translational medicine (1), 1 (2019) 1.336. J.A. Nichols, H.W.H. Chan, M.A. Baker, Biophysical reviews (1), 111 (2019) 1.337. S. Wang, D.M. Yang, R. Rong, X. Zhan, G. Xiao, The American journal of pathology (9), 1686 (2019) 1.338. N. Dimitriou, O. Arandjelovi´c, P.D. Caie, Frontiers in Medicine (2019) 1.339. N. Kumar, R. Gupta, S. Gupta, Journal of Digital Imaging (2020) 1.340. T.C.G. Atlas. Website (2006). https://portal.gdc.cancer.gov/ (6), giy065(2018) 2.1, 2.1.2, 5, 842. M. Veta, Y.J. Heng, N. Stathonikos, B.E. Bejnordi, F. Beca, T. Wollmann, K. Rohr,M.A. Shah, D. Wang, M. Rousson, et al., Medical image analysis , 111 (2019) 2.1,2.1.3, 6itle Suppressed Due to Excessive Length 7743. M. Babaie, S. Kalra, A. Sriram, C. Mitcheltree, S. Zhu, A. Khatami, S. Rahnamayan,H.R. Tizhoosh, in Proceedings of the IEEE Conference on Computer Vision and PatternRecognition Workshops (2017), pp. 8–16 2.1, 2.1.4, 7, 3.1.2, 4, 5.2, 644. M.D. Kumar, M. Babaie, H.R. Tizhoosh, in (IEEE, 2018), pp. 1–8 2.1, 3.2, 3.2, 445. K. Tomczak, P. Czerwi´nska, M. Wiznerowicz, Contemporary oncology (1A), A68(2015) 2.1.146. D.A. Gutman, J. Cobb, D. Somanna, Y. Park, F. Wang, T. Kurc, J.H. Saltz, D.J. Brat,L.A. Cooper, J. Kong, Journal of the American Medical Informatics Association (6),1091 (2013) 2.1.147. B.E. Bejnordi, M. Veta, P.J. Van Diest, B. Van Ginneken, N. Karssemeijer, G. Litjens,J.A. Van Der Laak, M. Hermsen, Q.F. Manson, M. Balkenhol, et al., Jama (22), 2199(2017) 2.1.2, 3.1.2, 3.1.2, 3.1.2, 3.1.2, 3.1.2, 4, 6.2.1, 6.4, 748. P. Bandi, O. Geessink, Q. Manson, M. Van Dijk, M. Balkenhol, M. Hermsen, B.E. Be-jnordi, B. Lee, K. Paeng, A. Zhong, et al., IEEE transactions on medical imaging (2),550 (2018) 2.1.249. M. Veta, P.J. Van Diest, S.M. Willems, H. Wang, A. Madabhushi, A. Cruz-Roa, F. Gon-zalez, A.B. Larsen, J.S. Vestergaard, A.B. Dahl, et al., Medical image analysis (1), 237(2015) 2.1.350. M. Sokolova, G. Lapalme, Information processing & management (4), 427 (2009) 2.2.151. R.M. Haralick, L.G. Shapiro, Computer vision, graphics, and image processing (1),100 (1985) 2.2.2, 4.152. P. Kamavisdar, S. Saluja, S. Agrawal, International Journal of Advanced Research inComputer and Communication Engineering (1), 1005 (2013) 2.2.353. S. Petushi, F.U. Garcia, M.M. Haber, C. Katsinis, A. Tozeren, BMC medical imaging (1), 14 (2006) 2.2.3, 3.1.2, 3.1.2, 16, 4, 5.3, 654. Z. Swiderska, T. Markiewicz, B. Grala, W. Kozlowski, in International Conference onComputer Analysis of Images and Patterns (Springer, 2015), pp. 1–12 2.2.3, 6.1.1, 6.4,755. S. Doyle, M. Feldman, J. Tomaszewski, A. Madabhushi, IEEE transactions on biomedicalengineering (5), 1205 (2010) 2.2.3, 3.1.2, 3.1.2, 4, 5.1, 6, 6.3, 6.4, 7, 7.256. N.R. Pal, S.K. Pal, Pattern recognition (9), 1277 (1993) 2.2.457. J.P. Egan, G.Z. Greenberg, A.I. Schulman, The Journal of the Acoustical Society ofAmerica (8), 993 (1961) 2.2.458. S. Shen, W. Sandham, M. Granat, in (IEEE, 2003),pp. 149–152 359. A.M. Khan, N. Rajpoot, D. Treanor, D. Magee, IEEE Transactions on Biomedical En-gineering (6), 1729 (2014) 360. S. Jhajharia, H.K. Varshney, S. Verma, R. Kumar, in (IEEE, 2016), pp.1896–1901 361. D. ping Tian, et al., International Journal of Multimedia and Ubiquitous Engineering (4), 385 (2013) 3, 3.1.162. G. Kumar, P.K. Bhatia, in (IEEE, 2014), pp. 5–12 363. S.R. Kodituwakku, S. Selvarajah, Indian Journal of Computer Science and Engineering (3), 207 (2004) 3.1.164. V. Roullier, V.T. Ta, O. Lezoray, A. Elmoataz, in (IEEE, 2010), pp. 153–156 3.1.1, 3.1.1, 4,4.3, 24, 5, 7.165. E. Mercan, S. Aksoy, L.G. Shapiro, D.L. Weaver, T.T. Bruny´e, J.G. Elmore, Journal ofdigital imaging (4), 496 (2016) 3.1.1, 3.1.1, 3.1.1, 3.1.2, 3.1.2, 466. C. Mercan, E. Mercan, S. Aksoy, L.G. Shapiro, D.L. Weaver, J.G. Elmore, in MedicalImaging 2016: Digital Pathology , vol. 9791 (International Society for Optics and Photon-ics, 2016), vol. 9791, p. 979108 3.1.1, 3.1.2, 3.1.2, 467. G. Pass, R. Zabih, J. Miller, in
Proceedings of the fourth ACM international conferenceon Multimedia (1997), pp. 65–73 3.1.168. J. Kong, O. Sertel, H. Shimada, K.L. Boyer, J.H. Saltz, M.N. Gurcan, Pattern Recognition (6), 1080 (2009) 3.1.1, 3.1.1, 3.1.1, 3.1.2, 3.1.2, 4, 5.1, 6, 7.28 Chen Li et al.69. S. Samsi, A.K. Krishnamurthy, M.N. Gurcan, Journal of computational science (5), 269(2012) 3.1.1, 3.1.1, 470. S. Kothari, J.H. Phan, A.O. Osunkoya, M.D. Wang, in Proceedings of the ACM Confer-ence on Bioinformatics, Computational Biology and Biomedicine (2012), pp. 218–2253.1.1, 3.1.3, 3.1.3, 471. H.C. Akakin, M.N. Gurcan, IEEE transactions on information technology in biomedicine (4), 758 (2012) 3.1.1, 3.1.1, 3.1.1, 3.1.1, 3.1.2, 3.1.2, 3.1.2, 472. B.T. Collins, L.E. Collins, American journal of clinical pathology (6), 736 (2013)3.1.1, 3.1.1, 3.1.3, 3.1.3, 473. N. Nayak, H. Chang, A. Borowsky, P. Spellman, B. Parvin, in (IEEE, 2013), pp. 410–413 3.1.1, 3.1.1, 3.1.2,3.1.2, 3.1.2, 4, 5.1, 33, 6, 7.274. M. Veta, P.J. van Diest, J.P. Pluim, in Medical Imaging 2013: Digital Pathology , vol.8676 (International Society for Optics and Photonics, 2013), vol. 8676, p. 867607 3.1.1,3.1.1, 3.1.2, 3.1.3, 3.1.3, 4, 6.1.3, 7, 7.275. S. Kothari, J.H. Phan, M.D. Wang, Journal of pathology informatics (2013) 3.1.1,3.1.1, 3.1.2, 3.1.2, 3.1.2, 3.1.3, 3.1.3, 476. A. Homeyer, A. Schenk, J. Arlt, U. Dahmen, O. Dirsch, H.K. Hahn, Computerized Med-ical Imaging and Graphics (4), 313 (2013) 3.1.1, 3.1.1, 9, 3.1.1, 3.1.2, 3.1.2, 4, 5.1,677. H. Hiary, R.S. Alomari, V. Chaudhary, IET image processing (5), 464 (2013) 3.1.1,3.1.1, 3.1.1, 3.1.1, 3.1.2, 3.1.2, 4, 4.4, 578. P.A. Bautista, N. Hashimoto, Y. Yagi, Journal of pathology informatics (2014) 3.1.1,3.1.1, 3.1.1, 3.1.1, 479. E. Mercan, S. Aksoy, L.G. Shapiro, D.L. Weaver, T. Brunye, J.G. Elmore, in (IEEE, 2014), pp. 1179–1184 3.1.1,3.1.1, 11, 3.1.2, 3.1.2, 3.1.2, 480. F.C. Yeh, Q. Ye, T.K. Hitchens, Y.L. Wu, A.V. Parwani, C. Ho, Journal of PathologyInformatics (2014) 3.1.1, 3.1.1, 3.1.1, 10, 4, 5.3, 681. G. Litjens, B.E. Bejnordi, N. Timofeeva, G. Swadi, I. Kovacs, C. Hulsbergen-van de Kaa,J. van der Laak, in Medical Imaging 2015: Digital Pathology , vol. 9420 (InternationalSociety for Optics and Photonics, 2015), vol. 9420, p. 94200B 3.1.1, 3.1.1, 3.1.1, 4, 6.1.2,6.4, 782. M. Weingant, H.M. Reynolds, A. Haworth, C. Mitchell, S. Williams, M.D. DiFranco, in
International Workshop on Machine Learning in Medical Imaging (Springer, 2015), pp.280–287 3.1.1, 3.1.1, 3.1.2, 3.1.2, 4, 5.1, 6, 7.283. R. Li, J. Huang, in
International Workshop on Patch-based Techniques in Medical Imag-ing (Springer, 2015), pp. 120–127 3.1.1, 3.1.1, 3.1.2, 3.1.2, 3.1.2, 4, 4.6, 5, 6.1.2, 6.4,784. J. Barker, A. Hoogi, A. Depeursinge, D.L. Rubin, Medical image analysis , 60 (2016)3.1.1, 3.1.1, 12, 3.1.2, 3.1.2, 3.1.2, 3.1.3, 3.1.3, 4, 5.2, 685. N. Brieu, O. Pauly, J. Zimmermann, G. Binnig, G. Schmidt, in Medical Imaging 2016:Image Processing , vol. 9784 (International Society for Optics and Photonics, 2016), vol.9784, p. 978410 3.1.1, 3.1.1, 3.1.2, 3.1.2, 3.1.2, 4, 4.6, 31, 586. C. Mercan, S. Aksoy, E. Mercan, L.G. Shapiro, D.L. Weaver, J.G. Elmore, IEEE trans-actions on medical imaging (1), 316 (2017) 3.1.1, 3.1.1, 3.1.1, 3.1.2, 3.1.2, 4, 5.2, 6,7.287. A. Cruz-Roa, H. Gilmore, A. Basavanhally, M. Feldman, S. Ganesan, N. Shih,J. Tomaszewski, A. Madabhushi, F. Gonz´alez, PloS one (5), e0196828 (2018) 3.1.1,3.1.1, 3.1.1, 4, 6.2.3, 6.4, 7, 7.288. M. Mork¯unas, P. Treigys, J. Bernataviˇcien˙e, A. Laurinaviˇcius, G. Korvel, Informatica (1), 75 (2018) 3.1.1, 3.1.1, 3.1.1, 3.1.2, 3.1.2, 4, 5.2, 6, 7.289. P. Mohanaiah, P. Sathyanarayana, L. GuruKumar, International journal of scientific andresearch publications (5), 1 (2013) 3.1.290. M. Tuceryan, A.K. Jain, IEEE transactions on pattern analysis and machine intelligence (2), 211 (1990) 3.1.291. R. Chellappa, S. Chatterjee, IEEE Transactions on Acoustics, Speech, and Signal Pro-cessing (4), 959 (1985) 3.1.292. O. Pichler, A. Teuner, B.J. Hosticka, Pattern Recognition (5), 733 (1996) 3.1.293. J. Diamond, N.H. Anderson, P.H. Bartels, R. Montironi, P.W. Hamilton, Human pathol-ogy (9), 1121 (2004) 3.1.2, 3.1.2, 3.1.3, 3.1.3, 4, 5.3, 6itle Suppressed Due to Excessive Length 7994. O. Sertel, J. Kong, H. Shimada, U. Catalyurek, J.H. Saltz, M. Gurcan, in Medical Imag-ing 2008: Computer-Aided Diagnosis , vol. 6915 (International Society for Optics andPhotonics, 2008), vol. 6915, p. 69150P 3.1.2, 3.1.2, 3.1.2, 4, 5.1, 6, 7.2, 7.295. O. Sertel, U.V. Catalyurek, H. Shimada, M.N. Gurcan, in
International Conference onMedical Image Computing and Computer Assisted Intervention (MICCAI 2009) (2009),pp. 7–18 3.1.2, 3.1.2, 4, 5.3, 696. O. Sertel, J. Kong, H. Shimada, U.V. Catalyurek, J.H. Saltz, M.N. Gurcan, Patternrecognition (6), 1093 (2009) 3.1.2, 3.1.2, 13, 4, 5.1, 697. V. Roullier, O. L´ezoray, V.T. Ta, A. Elmoataz, in International Symposium on VisualComputing (Springer, 2010), pp. 539–548 3.1.2, 3.1.2, 4, 4.3, 5, 7.198. M.D. DiFranco, G. O’Hurley, E.W. Kay, R.W.G. Watson, P. Cunningham, Computerizedmedical imaging and graphics (7-8), 629 (2011) 3.1.2, 3.1.2, 4, 5.1, 32, 6, 7.2, 7.299. J. Kong, L.A. Cooper, F. Wang, D.A. Gutman, J. Gao, C. Chisolm, A. Sharma, T. Pan,E.G. Van Meir, T.M. Kurc, et al., IEEE Transactions on Biomedical Engineering (12),3469 (2011) 3.1.2, 3.1.2, 3.1.2, 3.1.3, 3.1.3, 4, 4.1, 4.2, 5100. V. Roullier, O. L´ezoray, V.T. Ta, A. Elmoataz, Computerized Medical Imaging andGraphics (7-8), 603 (2011) 3.1.2, 3.1.2, 4, 4.3, 5101. M. Grunkin, J. Raundahl, N.T. Foged, in Signal Transduction Immunohistochemistry (Springer, 2011), pp. 143–154 3.1.2, 3.1.2, 3.1.2, 3.1.3, 3.1.3, 4102. K. Nguyen, A.K. Jain, B. Sabata, Journal of pathology informatics (2011) 3.1.2, 3.1.2,4, 6.1.1, 6.4, 7103. H. Sharma, A. Alekseychuk, P. Leskovsky, O. Hellwich, R.S. Anand, N. Zerbe, P. Hufnagl,Diagnostic pathology (1), 134 (2012) 3.1.2, 3.1.2, 3.1.2, 4104. L. Jiao, Q. Chen, S. Li, Y. Xu, in World Congress on Medical Physics and BiomedicalEngineering May 26-31, 2012, Beijing, China (Springer, 2013), pp. 1283–1286 3.1.2,3.1.2, 4, 5.1, 6, 6.1.1, 6.4, 7, 7.2105. J. Kong, L.A. Cooper, F. Wang, J. Gao, G. Teodoro, L. Scarpace, T. Mikkelsen, M.J.Schniederjan, C.S. Moreno, J.H. Saltz, et al., PloS one (11), e81049 (2013) 3.1.2, 3.1.2,3.1.2, 3.1.3, 3.1.3, 4106. G. Apou, B. Naegel, G. Forestier, F. Feuerhake, C. Wemmert, in , vol. 1 (IEEE, 2014),vol. 1, pp. 309–319 3.1.2, 3.1.2, 3.1.2, 4, 4.6, 29, 5, 7.1107. B.E. Bejnordi, G. Litjens, M. Hermsen, N. Karssemeijer, J.A. van der Laak, in Med-ical Imaging 2015: Digital Pathology , vol. 9420 (International Society for Optics andPhotonics, 2015), vol. 9420, p. 94200H 3.1.2, 3.1.2, 4, 6.1.2, 6.4, 7108. H. Sharma, N. Zerbe, I. Klempert, S. Lohmann, B. Lindequist, O. Hellwich, P. Hufnagl, in (IEEE, 2015), pp. 1–6 3.1.2, 4, 6.1.1, 6.4, 7109. Z. Swiderska, A. Korzynska, T. Markiewicz, M. Lorent, J. Zak, A. Wesolowska,L. Roszkowiak, J. Slodkowska, B. Grala, Analytical cellular pathology (2015) 3.1.2,3.1.2, 3.1.2, 4110. X. Zhang, H. Su, L. Yang, S. Zhang, in
Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (2015), pp. 5361–5368 3.1.2, 3.1.2, 3.1.2, 4, 4.6, 30, 5111. L.A. Cooper, J. Kong, D.A. Gutman, W.D. Dunn, M. Nalisnik, D.J. Brat, Laboratoryinvestigation (4), 366 (2015) 3.1.2, 3.1.2, 3.1.2, 3.1.3, 3.1.3, 4112. G. Apou, F. Feuerhake, G. Forestier, B. Naegel, C. Wemmert, in (IEEE, 2015), pp.154–159 3.1.2, 3.1.2, 3.1.2, 17, 4113. M. Peikari, M.J. Gangeh, J. Zubovits, G. Clarke, A.L. Martel, IEEE transactions onmedical imaging (1), 307 (2015) 3.1.2, 3.1.2, 15, 4, 5.1, 6, 7.2, 7.2114. B.E. Bejnordi, M. Balkenhol, G. Litjens, R. Holland, P. Bult, N. Karssemeijer, J.A. VanDer Laak, IEEE transactions on medical imaging (9), 2141 (2016) 3.1.2, 3.1.2, 3.1.2,3.1.2, 4, 6.1.2, 6.4, 7115. Y. Zhao, E.F. Black, L. Marini, K. McHenry, N. Kenyon, R. Patil, A. Balla,A. Bartholomew, in (IEEE, 2016), pp. 165–174 3.1.2, 4116. N. Harder, M. Athelogou, H. Hessel, A. Buchner, R. Sch¨onmeyer, G. Schmidt, C. Stief,T. Kirchner, G. Binnig, in (IEEE, 2016), pp. 807–810 3.1.2, 4, 5.3, 6117. A. Shirinifard, S. Thiagarajan, P. Vogel, A. Sablauer, Journal of Histochemistry & Cy-tochemistry (5), 301 (2016) 3.1.2, 3.1.2, 3.1.2, 4, 6.1.3, 70 Chen Li et al.118. M. Gadermayr, M. Strauch, B.M. Klinkhammer, S. Djudjaj, P. Boor, D. Merhof, in International Conference on Image Analysis and Recognition (Springer, 2016), pp. 616–622 3.1.2, 3.1.2, 3.1.2, 3.1.2, 3.1.2, 4, 5.1, 6119. P. Leo, G. Lee, N.N. Shih, R. Elliott, M.D. Feldman, A. Madabhushi, Journal of MedicalImaging (4), 047502 (2016) 3.1.2, 3.1.2, 14, 4120. J. Saltz, A. Sharma, G. Iyer, E. Bremer, F. Wang, A. Jasniewski, T. DiPrima, J.S.Almeida, Y. Gao, T. Zhao, et al., Cancer research (21), e79 (2017) 3.1.2, 3.1.2, 3.1.3,3.1.3, 4121. J.X. Hu, Y.Y. Xu, H.B. Shen, et al., in (IEEE, 2017), pp. 599–604 3.1.2, 3.1.2, 4, 5.2, 38, 6122. M. Valkonen, K. Kartasalo, K. Liimatainen, M. Nykter, L. Latonen, P. Ruusuvuori,Cytometry Part A (6), 555 (2017) 3.1.2, 3.1.2, 3.1.2, 4, 6.1.2, 6.4, 7, 7.2123. J.J. Nirschl, A. Janowczyk, E.G. Peyster, R. Frank, K.B. Margulies, M.D. Feldman,A. Madabhushi, PloS one (4), e0192726 (2018) 3.1.2, 3.1.2, 3.1.2, 4124. H. Xu, C. Lu, R. Berendt, N. Jha, M. Mandal, Computerized medical imaging andgraphics , 124 (2018) 3.1.2, 3.1.2, 3.1.2, 3.1.3, 3.1.3, 4, 5.1, 6, 7.2125. H. Yoshida, T. Shimazu, T. Kiyuna, A. Marugame, Y. Yamashita, E. Cosatto,H. Taniguchi, S. Sekine, A. Ochiai, Gastric Cancer (2), 249 (2018) 3.1.2, 3.1.2, 3.1.2,4, 5.3, 6126. W. Han, C. Johnson, M. Gaed, J. Gomez, M. Moussa, J.L. Chin, S.E. Pautler, G. Bau-man, A.D. Ward, in Medical Imaging 2018: Digital Pathology , vol. 10581 (InternationalSociety for Optics and Photonics, 2018), vol. 10581, p. 105810Q 3.1.2, 3.1.2, 3.1.2, 4,6.1.1, 6.4, 7127. O. Simon, R. Yacoub, S. Jain, J.E. Tomaszewski, P. Sarder, Scientific reports (1), 1(2018) 3.1.2, 3.1.2, 3.1.2, 4, 6.1.1, 6.4, 7128. S. Klimov, I.M. Miligy, A. Gertych, Y. Jiang, M.S. Toss, P. Rida, I.O. Ellis, A. Green,U. Krishnamurti, E.A. Rakha, et al., Breast Cancer Research (1), 83 (2019) 3.1.2,3.1.2, 3.1.2, 4, 5.1, 6129. D. Zhang, G. Lu, Pattern recognition (1), 1 (2004) 3.1.3130. K.P. Philip, E.L. Dove, D.D. McPherson, N.L. Gotteiner, W. Stanford, K.B. Chandran,IEEE Transactions on Medical Imaging (2), 235 (1994) 3.1.3131. K. Huang, H. Yan, Pattern Recognition (1), 9 (1997) 3.1.3132. G. Zhang, Z.m. Ma, L.q. Niu, C.m. Zhang, Journal of Central South University (2),488 (2012) 3.1.3133. C. Lu, M. Mandal, in (IEEE, 2012), pp. 5355–5359 3.1.3, 3.1.3, 4, 4.1, 5134. M. Veta, R. Kornegoor, A. Huisman, A.H. Verschuur-Maes, M.A. Viergever, J.P. Pluim,P.J. Van Diest, Modern pathology (12), 1559 (2012) 3.1.3, 3.1.3, 4, 4.2, 22, 5, 7.1135. X.M. Lopez, E. D’Andrea, P. Barbot, A.S. Bridoux, S. Rorive, I. Salmon, O. Debeir,C. Decaestecker, PloS one (12), e82710 (2013) 3.1.3, 3.1.3, 4, 6.1.3, 7136. M. Puerto, T. Vargas, A. Cruz-Roa, in (IEEE, 2016), pp. 1–7 3.2, 4137. H. Sharma, N. Zerbe, I. Klempert, O. Hellwich, P. Hufnagl, Diagnostic Pathology (8)(2016) 3.2, 4, 5.1, 6, 7.2138. D. Wang, A. Khosla, R. Gargeya, H. Irshad, A.H. Beck, arXiv preprint arXiv:1606.05718(2016) 3.2, 3.2, 4, 5, 5.1, 6, 7.2139. B. Ge¸cer, Diss. Bilkent University (2016) 3.2, 4, 5.2, 35, 6, 6.2.2, 6.4, 7, 7.2140. K. Sirinukunwattana, S.E.A. Raza, Y.W. Tsang, D.R. Snead, I.A. Cree, N.M. Rajpoot,IEEE transactions on medical imaging (5), 1196 (2016) 3.2, 4, 5.2, 36, 6, 6.2.3, 6.4, 7,7.2141. L. Hou, D. Samaras, T.M. Kurc, Y. Gao, J.E. Davis, J.H. Saltz, in Proceedings of theieee conference on computer vision and pattern recognition (2016), pp. 2424–2433 3.2,4, 5.2, 6142. F. Sheikhzadeh, M. Guillaud, R.K. Ward, arXiv preprint arXiv:1612.09420 (2016) 3.2, 4143. A. Cruz-Roa, H. Gilmore, A. Basavanhally, M. Feldman, S. Ganesan, N.N. Shih,J. Tomaszewski, F.A. Gonz´alez, A. Madabhushi, Scientific reports , 46450 (2017) 3.2,4, 6.2.1, 6.4, 7, 7.2144. T. Wollmann, K. Rohr, in Bildverarbeitung f¨ur die Medizin 2017 (Springer, 2017), pp.249–253 3.2, 4145. T. Ara´ujo, G. Aresta, E. Castro, J. Rouco, P. Aguiar, C. Eloy, A. Pol´onia, A. Campilho,PloS one (6), e0177544 (2017) 3.2, 4, 5.2, 6, 7.2itle Suppressed Due to Excessive Length 81146. B.E. Bejnordi, G. Zuidhof, M. Balkenhol, M. Hermsen, P. Bult, B. van Ginneken,N. Karssemeijer, G. Litjens, J. van der Laak, Journal of Medical Imaging (4), 044504(2017) 3.2, 4, 5.2, 6, 7.2147. H. Sharma, N. Zerbe, I. Klempert, O. Hellwich, P. Hufnagl, Computerized Medical Imag-ing and Graphics , 2 (2017) 3.2, 4, 5.2, 37, 6, 7.2148. O. Jimenez-del Toro, S. Ot´alora, M. Atzori, H. M¨uller, in International Workshop onPatch-based Techniques in Medical Imaging (Springer, 2017), pp. 149–157 3.2, 4149. Y. Xu, Z. Jia, L.B. Wang, Y. Ai, F. Zhang, M. Lai, I. Eric, C. Chang, BMC bioinformatics (1), 1 (2017) 3.2, 4, 4.5, 4.5, 5, 5.2, 6150. B. Korbar, A.M. Olofson, A.P. Miraflor, C.M. Nicka, M.A. Suriawinata, L. Torresani,A.A. Suriawinata, S. Hassanpour, in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (2017), pp. 69–75 3.2, 3.2, 4, 5.2, 6151. A. Ghosh, S. Singh, D. Sheet, in (IEEE, 2017), pp. 1–6 3.2, 4, 5.2, 39, 6152. K. Das, S.P.K. Karri, A.G. Roy, J. Chatterjee, D. Sheet, in (IEEE, 2017), pp. 1024–1027 3.2, 4, 5.2,6153. Y. Cui, G. Zhang, Z. Liu, Z. Xiong, J. Hu, Medical & biological engineering & computing (9), 2027 (2019) 3.2, 4, 4.5, 4.5, 5154. F.G. Zanjani, S. Zinger, et al., in Medical imaging 2018: Digital pathology , vol. 10581(International Society for Optics and Photonics, 2018), vol. 10581, p. 105810I 3.2, 4,6.2.1, 6.4, 7, 7.2155. P. Courtiol, E.W. Tramel, M. Sanselme, G. Wainrib, arXiv preprint arXiv:1802.02212(2018) 3.2, 4, 5.2, 41, 6156. D. Bychkov, N. Linder, R. Turkki, S. Nordling, P.E. Kovanen, C. Verrill, M. Walliander,M. Lundin, C. Haglund, J. Lundin, Scientific reports (1), 1 (2018) 3.2, 3.2, 19, 4157. B. Gecer, S. Aksoy, E. Mercan, L.G. Shapiro, D.L. Weaver, J.G. Elmore, Pattern recog-nition , 345 (2018) 3.2, 4, 5.2, 6, 6.2.2, 6.4, 7, 7.2158. J. Ren, K. Karagoz, M. Gatza, D.J. Foran, X. Qi, in Medical Imaging 2018: ImagingInformatics for Healthcare, Research, and Applications , vol. 10579 (International Societyfor Optics and Photonics, 2018), vol. 10579, p. 1057904 3.2, 3.2, 4159. D. Tellez, J. van der Laak, F. Ciompi, Med. Imag. Deep Learn (2018) 3.2, 4, 5.2, 6160. K. Sirinukunwattana, N.K. Alham, C. Verrill, J. Rittscher, in
International Conferenceon Medical Image Computing and Computer-Assisted Intervention (Springer, 2018), pp.192–200 3.2, 4, 4.5, 4.5, 26, 5161. S. Kwok, in
International conference image analysis and recognition (Springer, 2018),pp. 931–940 3.2, 3.2, 4, 5.2, 42, 6, 7.2162. K. Das, S. Conjeti, A.G. Roy, J. Chatterjee, D. Sheet, in (IEEE, 2018), pp. 578–581 3.2, 4, 5.2,6, 7.2, 7.2163. H. Lin, H. Chen, Q. Dou, L. Wang, J. Qin, P.A. Heng, in (IEEE, 2018), pp. 539–546 3.2, 4, 6.2.2,6.4, 7, 7.2164. G. Campanella, V.W.K. Silva, T.J. Fuchs, arXiv preprint arXiv:1805.06983 (2018) 3.2,3.2, 3.2, 4, 5.2, 6, 7.2165. S. Jiang, J. Liao, Z. Bian, K. Guo, Y. Zhang, G. Zheng, Biomedical optics express (4),1601 (2018) 3.2, 3.2, 4166. X. Wang, H. Chen, C. Gan, H. Lin, Q. Dou, Q. Huang, M. Cai, P.A. Heng, Med. Imag.Deep Learn (2018) 3.2, 3.2, 4, 5.2, 6167. J. Shou, Y. Li, G. Yu, G. Li, OpenReview : MIDL 2018 (2018) 3.2, 3.2, 4, 5.2, 6, 7.2168. D. Tellez, M. Balkenhol, I. Otte-H¨oller, R. van de Loo, R. Vogels, P. Bult, C. Wauters,W. Vreuls, S. Mol, N. Karssemeijer, et al., IEEE transactions on medical imaging (9),2126 (2018) 3.2, 4, 6.2.1, 6.4, 7, 7.2169. C.A. Bertram, M. Aubreville, C. Marzahl, A. Maier, R. Klopfleisch, Scientific Data (1),1 (2019) 3.2, 3.2, 4170. J. Li, W. Li, A. Gertych, B.S. Knudsen, W. Speier, C.W. Arnold, arXiv preprintarXiv:1905.13208 (2019) 3.2, 3.2, 4, 5.2, 6, 7.2171. Y. Liu, T. Kohlberger, M. Norouzi, G.E. Dahl, J.L. Smith, A. Mohtashamian, N. Olson,L.H. Peng, J.D. Hipp, M.C. Stumpe, Archives of pathology & laboratory medicine (7),859 (2019) 3.2, 4, 6.2.3, 6.4, 7, 7.22 Chen Li et al.172. G. Campanella, M.G. Hanna, L. Geneslaw, A. Miraflor, V.W.K. Silva, K.J. Busam,E. Brogi, V.E. Reuter, D.S. Klimstra, T.J. Fuchs, Nature medicine (8), 1301 (2019)3.2, 4, 5.2, 6, 7.2173. X. Yue, N. Dimitriou, O. Arandjelovic, arXiv preprint arXiv:1902.03582 (2019) 3.2, 3.2,4174. S. Maksoud, A. Wiliem, K. Zhao, T. Zhang, L. Wu, B. Lovell, in International Conferenceon Medical Image Computing and Computer-Assisted Intervention (Springer, 2019), pp.432–441 3.2, 3.2, 4175. S. Bilaloglu, J. Wu, E. Fierro, R.D. Sanchez, P.S. Ocampo, N. Razavian, N. Coudray,A. Tsirigos, bioRxiv p. 633123 (2019) 3.2, 4, 6.2.3, 7176. H. Lin, H. Chen, S. Graham, Q. Dou, N. Rajpoot, P.A. Heng, IEEE transactions onmedical imaging (8), 1948 (2019) 3.2, 3.2, 4, 6.2.2, 6.4, 7177. H. Yu, X. Zhang, L. Song, L. Jiang, X. Huang, W. Chen, C. Zhang, J. Li, J. Yang, Z. Hu,et al., arXiv preprint arXiv:1910.03729 (2019) 3.2, 4178. A.B. Sanghvi, E.Z. Allen, K.M. Callenberg, L. Pantanowitz, Cancer cytopathology (10), 658 (2019) 3.2, 4179. S. Wang, Y. Zhu, L. Yu, H. Chen, H. Lin, X. Wan, X. Fan, P.A. Heng, Medical imageanalysis , 101549 (2019) 3.2, 4, 5.2, 43, 6, 7.2180. T. Kohlberger, Y. Liu, M. Moran, P.H.C. Chen, T. Brown, J.D. Hipp, C.H. Mermel,M.C. Stumpe, Journal of Pathology Informatics (2019) 3.2, 4, 6.2.1, 6.4, 7181. J. Xu, C. Cai, Y. Zhou, B. Yao, G. Xu, X. Wang, K. Zhao, A. Madabhushi, Z. Liu,L. Liang, in European Congress on Digital Pathology (Springer, 2019), pp. 100–108 3.2,3.2, 4182. P. Chen, X. Shi, Y. Liang, Y. Li, L. Yang, P.D. Gader, Computer Methods and Programsin Biomedicine , 105630 (2020) 3.2, 4183. S. Sornapudi, R. Addanki, J. Stanley, W.V. Stoecker, R. Long, R. Zuna, S.R. Frazier,S. Antani, medRxiv (2020) 3.2, 4184. L. Pantanowitz, G.M. Quiroga-Garza, L. Bien, R. Heled, D. Laifenfeld, C. Linhart,J. Sandbank, A.A. Shach, V. Shalev, M. Vecsler, et al., The Lancet Digital Health (8),e407 (2020) 3.2, 4185. M.M. Rahaman, C. Li, Y. Yao, F. Kulwa, M.A. Rahman, Q. Wang, S. Qi, F. Kong,X. Zhu, X. Zhao, Journal of X-ray Science and Technology (5), 1 (2020) 3.2186. W. Rawat, Z. Wang, Neural computation (9), 2352 (2017) 3.2, 5187. W. Zhiqiang, L. Jun, in (IEEE, 2017),pp. 11,104–11,109 3.2188. Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, Proceedings of the IEEE (11), 2278 (1998)3.2189. Y. LeCun, Y. Bengio, G. Hinton, nature (7553), 436 (2015) 3.2190. A. Krizhevsky, I. Sutskever, G.E. Hinton, Communications of the ACM (6), 84 (2017)3.2191. K. Simonyan, A. Zisserman, arXiv preprint arXiv:1409.1556 (2014) 3.2192. K. He, X. Zhang, S. Ren, J. Sun, in Proceedings of the IEEE conference on computervision and pattern recognition (2016), pp. 770–778 3.2193. K. He, X. Zhang, S. Ren, J. Sun, in
European conference on computer vision (Springer,2016), pp. 630–645 3.2194. O. Ronneberger, P. Fischer, T. Brox, in
International Conference on Medical imagecomputing and computer-assisted intervention (Springer, 2015), pp. 234–241 3.2195. P. B´andi, R. van de Loo, M. Intezar, D. Geijs, F. Ciompi, B. van Ginneken, J. van derLaak, G. Litjens, in (IEEE, 2017), pp. 591–595 3.2, 4, 4.5, 25, 5196. N. Seth, Automated localization of breast ductal carcinoma in situ in whole slide images.Ph.D. thesis, University of Toronto (Canada) (2019) 3.2, 4, 4.5, 4.5, 5, 7.1197. N. Seth, S. Akbar, S. Nofech-Mozes, S. Salama, A.L. Martel, in
European Congress onDigital Pathology (Springer, 2019), pp. 67–74 3.2, 4, 4.5, 4.5, 5, 7.1198. Y. Feng, A. Hafiane, H. Laurent, arXiv preprint arXiv:2007.12935 (2020) 3.2, 4, 4.5, 4.5,5199. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke,A. Rabinovich, in
Proceedings of the IEEE conference on computer vision and patternrecognition (2015), pp. 1–9 3.2200. J.J. Hopfield, Proceedings of the national academy of sciences (8), 2554 (1982) 3.2201. S. Hochreiter, J. Schmidhuber, Neural computation (8), 1735 (1997) 3.2itle Suppressed Due to Excessive Length 83202. D.L. Pham, C. Xu, J.L. Prince, Annual review of biomedical engineering (1), 315 (2000)4203. R.C. Gonzalez, R.E. Woods, Digital image processing , 1 (2007) 4204. D.D. Patil, S.G. Deore, International Journal of Computer Science and Mobile Computing (1), 22 (2013) 4205. R. Dass, S. Devi, International Journal of Computer Science and Mobile Computing(2012) 4206. S. Masood, M. Sharif, A. Masood, M. Yasmin, M. Raza, Current Medical Imaging (1),3 (2015) 4207. T. Zuva, O.O. Olugbara, S.O. Ojo, S.M. Ngwira, Canadian Journal on Image Processingand Computer Vision (3), 20 (2011) 4208. V. Shrimali, R. Anand, V. Kumar, IETE technical review (1), 8 (2009) 4209. W. Birkfellner, Applied medical image processing: a basic course (CRC Press, 2016) 4210. S.S. Al-Amri, N.V. Kalyankar, S.D. Khamitkar, arXiv preprint arXiv:1002.1148 (2010) 4211. B.S. He, F. Zhu, Y.G. Shi, in
Advanced Materials Research , vol. 760 (Trans Tech Publ,2013), vol. 760, pp. 1590–1593 4212. K.S. Fu, J. Mui, Pattern recognition (1), 3 (1981) 4.1213. P.K. Sahoo, S. Soltani, A.K. Wong, Computer vision, graphics, and image processing (2), 233 (1988) 4.1214. A. Rosenfeld, P. De La Torre, IEEE Transactions on Systems, Man, and Cybernetics (2), 231 (1983) 4.1215. N. Ohtsu, Proc. 4th IJCPR, Tokyo, 1978 (1978) 4.1216. J. Shu, H. Fu, G. Qiu, P. Kaye, M. Ilyas, in (IEEE, 2013), pp.5445–5448 4.1, 4.2, 23, 4.2, 5217. H. Vo, J. Kong, D. Teng, Y. Liang, A. Aji, G. Teodoro, F. Wang, in VLDB Workshopon Data Management and Analytics for Medicine and Healthcare (Springer, 2016), pp.62–77 4.1, 4.2, 5218. H.B. Arunachalam, R. Mishra, B. Armaselu, O. Daescu, M. Martinez, P. Leavey,D. Rakheja, K. Cederberg, A. Sengupta, M. NI’SUILLEABHAIN, in
PACIFIC SYM-POSIUM ON BIOCOMPUTING 2017 (World Scientific, 2017), pp. 195–206 4.1, 21,4.4, 5219. L. Vincent, IEEE transactions on image processing (2), 176 (1993) 4.1220. H. Vo, J. Kong, D. Teng, Y. Liang, A. Aji, G. Teodoro, F. Wang, Distributed and ParallelDatabases (2), 251 (2019) 4.1, 4.2, 4.5, 5221. L. Vincent, P. Soille, IEEE Transactions on Pattern Analysis & Machine Intelligence (6), 583 (1991) 4.2222. M. Veta, P.J. Van Diest, R. Kornegoor, A. Huisman, M.A. Viergever, J.P. Pluim, PloSone (7), e70221 (2013) 4.2, 5, 7.1223. S. Hojjatoleslami, J. Kittler, IEEE Transactions on Image processing (7), 1079 (1998)4.2224. R. Adams, L. Bischof, IEEE Transactions on pattern analysis and machine intelligence (6), 641 (1994) 4.2225. P.F. Felzenszwalb, D.P. Huttenlocher, International journal of computer vision (2),167 (2004) 4.3226. K.E. Van de Sande, J.R. Uijlings, T. Gevers, A.W. Smeulders, in (IEEE, 2011), pp. 1879–1886 4.3227. D.B. West, et al., Introduction to graph theory , vol. 2 (Prentice hall Upper Saddle River,NJ, 1996) 4.3228. J. MacQueen, et al., in
Proceedings of the fifth Berkeley symposium on mathematicalstatistics and probability , vol. 1 (Oakland, CA, USA, 1967), vol. 1, pp. 281–297 4.4229. N. Dong, M. Kampffmeyer, X. Liang, Z. Wang, W. Dai, E. Xing, in
Deep Learning in Med-ical Image Analysis and Multimodal Learning for Clinical Decision Support (Springer,2018), pp. 317–325 4.5, 4.5, 5, 7.1230. C. Bergmeir, M.G. Silvente, J.M. Ben´ıtez, Computer methods and programs inbiomedicine (3), 497 (2012) 4.6, 5231. L.D. Cohen, CVGIP: Image understanding (2), 211 (1991) 4.6232. R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, S. S¨usstrunk, IEEE transactions onpattern analysis and machine intelligence (11), 2274 (2012) 4.6, 5.2233. K. Yamaguchi, D. McAllester, R. Urtasun, in European Conference on Computer Vision (Springer, 2014), pp. 756–771 4.64 Chen Li et al.234. A. Karpathy, L. Fei-Fei, in
Proceedings of the IEEE conference on computer vision andpattern recognition (2015), pp. 3128–3137 5235. V. Vapnik, A.Y. Lerner, Avtomat. i Telemekh (6), 774 (1963) 5.1236. S.T. Roweis, L.K. Saul, science (5500), 2323 (2000) 5.1237. S. Kumar, Z. Khan, A. Jain, International Journal of Advanced Computer Research (3),55 (2012) 5.1238. C.N. Kamath, S.S. Bukhari, A. Dengel, in Proceedings of the ACM Symposium on Doc-ument Engineering 2018 (2018), pp. 1–11 5.1239. G. Madzarov, D. Gjorgjevikj, I. Chorbev, Informatica (2) (2009) 5.1240. C. Lu, M. Mandal, Pattern Recognition (8), 2738 (2015) 5.1, 6, 7.2241. P. Shukla, S. Verma, et al., in TENCON 2017-2017 IEEE Region 10 Conference (IEEE,2017), pp. 1588–1592 5.1, 6242. D.R. Cutler, T.C. Edwards Jr, K.H. Beard, A. Cutler, K.T. Hess, J. Gibson, J.J. Lawler,Ecology (11), 2783 (2007) 5.1243. B. Ghimire, J. Rogan, J. Miller, Remote Sensing Letters (1), 45 (2010) 5.1244. P.O. Gislason, J.A. Benediktsson, J.R. Sveinsson, Pattern Recognition Letters (4), 294(2006) 5.1245. L. Guo, N. Chehata, C. Mallet, S. Boukir, ISPRS Journal of Photogrammetry and RemoteSensing (1), 56 (2011) 5.1246. X.W. Chen, M. Liu, Bioinformatics (24), 4394 (2005) 5.1247. A. ¨Oz¸cift, Computers in biology and medicine (5), 265 (2011) 5.1248. M. Seera, C.P. Lim, Expert Systems with Applications (5), 2239 (2014) 5.1249. J.I. Titapiccolo, M. Ferrario, S. Cerutti, C. Barbieri, F. Mari, E. Gatti, M.G. Signorini,Expert systems with applications (11), 4679 (2013) 5.1250. L. Breiman, Machine learning (1), 5 (2001) 5.1251. Z. Masetic, A. Subasi, Computer methods and programs in biomedicine , 54 (2016)5.1252. A. Chaudhary, S. Kolhe, R. Kamal, Information Processing in Agriculture (4), 215(2016) 5.1253. M.F. Jamaluddin, M.F.A. Fauzi, F.S. Abas, in (IEEE, 2017), pp. 90–95 5.1, 6,6.2.1, 6.4, 7, 7.2254. P. Domingos, M. Pazzani, Machine learning (2-3), 103 (1997) 5.1255. T. Hastie, R. Tibshirani, IEEE transactions on pattern analysis and machine intelligence (6), 607 (1996) 5.1256. J. Arevalo, A. Cruz-Roa, V. Arias, E. Romero, F.A. Gonz´alez, Artificial intelligence inmedicine (2), 131 (2015) 5.2, 34, 6257. J. Ren, I. Hacihaliloglu, E.A. Singer, D.J. Foran, X. Qi, in International Conferenceon Medical Image Computing and Computer-Assisted Intervention (Springer, 2018), pp.201–209 5.2, 40, 6, 7.2258. T. Durand, N. Thome, M. Cord, in
Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (2016), pp. 4743–4752 5.2259. H. Yoshida, Y. Yamashita, T. Shimazu, E. Cosatto, T. Kiyuna, H. Taniguchi, S. Sekine,A. Ochiai, Oncotarget (53), 90719 (2017) 5.3, 6260. Z.Q. Zhao, P. Zheng, S.t. Xu, X. Wu, IEEE transactions on neural networks and learningsystems (11), 3212 (2019) 6261. A. Vedaldi, V. Gulshan, M. Varma, A. Zisserman, in (IEEE, 2009), pp. 606–613 6262. L. Zhang, L. Zhang, D. Tao, X. Huang, B. Du, IEEE transactions on geoscience andremote sensing (8), 4955 (2013) 6263. R.L. Hsu, M. Abdel-Mottaleb, A.K. Jain, IEEE transactions on pattern analysis andmachine intelligence (5), 696 (2002) 6264. P.M. Szczypi´nski, A. Klepaczko, in Biomedical texture analysis (Elsevier, 2017), pp. 315–347 6265. X.M. Lopez, O. Debeir, C. Maris, S. Rorive, I. Roland, M. Saerens, I. Salmon, C. De-caestecker, Cytometry Part A (9), 765 (2012) 6.1.3, 7266. P.A. Bautista, Y. Yagi, in (IEEE, 2009), pp. 3669–3672 6.1.3, 6.4, 7267. P.A. Bautista, Y. Yagi, Journal of pathology informatics (2010) 6.1.3, 6.4, 7268. N.S. Vyas, M. Markow, C. Prieto-Granada, S. Gaudi, L. Turner, P. Rodriguez-Waitkus,J.L. Messina, D.M. Jukic, Journal of Pathology Informatics (2016) 6.1.3, 7itle Suppressed Due to Excessive Length 85269. A. Cruz-Roa, A. Basavanhally, F. Gonz´alez, H. Gilmore, M. Feldman, S. Ganesan,N. Shih, J. Tomaszewski, A. Madabhushi, in Medical Imaging 2014: Digital Pathology ,vol. 9041 (International Society for Optics and Photonics, 2014), vol. 9041, p. 9041036.2.1, 6.2.1, 46, 6.4, 7270. G. Du, X. Cao, J. Liang, X. Chen, Y. Zhan, Journal of Imaging Science and Technology (2), 20508 (2020) 7.1271. L. Weihua, Process Automation Instrumentation (2013) 7.2272. Y. Li, X. Zhang, in Pacific-Asia Conference on Knowledge Discovery and Data Mining (Springer, 2011), pp. 321–332 7.2273. C. ZHANG, SCIENTIA SINICA Informationis (12), 1612 (2013) 7.2274. Y. Wang, J. Xi, Y. Ma, IEEE Access , 195799 (2020) 7.3.1275. M. To˘ga¸car, B. Ergen, Z. C¨omert, Biocybernetics and Biomedical Engineering (1), 23(2020) 7.3.1276. Y. Zheng, Z. Jiang, F. Xie, H. Zhang, Y. Ma, H. Shi, Y. Zhao, Pattern Recognition ,14 (2017) 7.3.1277. Y. Zhao, Y. Ding, X. Zhao, Electronics Letters (22), 1849 (2016) 7.3.1278. C. Sun, C. Li, J. Zhang, M.M. Rahaman, S. Ai, H. Chen, F. Kulwa, Y. Li, X. Li, T. Jiang,Biocybernetics and Biomedical Engineering (4), 1535 (2020) 7.3.2279. J. Zhang, C. Li, F. Kulwa, X. Zhao, C. Sun, Z. Li, T. Jiang, H. Li, S. Qi, BioMed ResearchInternational (2020) 7.3.2280. F.L. Galv˜ao, S.J.F. Guimar˜aes, A.X. Falc˜ao, Pattern Recognition , 107532 (2020)7.3.2281. H. Jiang, S. Piao, M.Z. Qadir, Q. Guo, Electronics Letters (7), 386 (2019) 7.3.2282. J. Cui, H. Guo, H. Wang, F. Chen, L. Shu, L.C. Li, Journal of X-Ray Science andTechnology (Preprint), 1 (2020) 7.3.2283. C. Mu, J. Liu, Y. Liu, Y. Liu, IEEE Access , 6768 (2020) 7.3.3284. S. Devi, M.N. Sahoo, S. Bakshi, Biocybernetics and Biomedical Engineering (2020) 7.3.3285. O. Okwuashi, C.E. Ndehedehe, Pattern Recognition p. 107298 (2020) 7.3.3286. C. Li, K. Wang, N. Xu, Artificial Intelligence Review (4), 577 (2019) 7.3.3287. A. Manyala, H. Cholakkal, V. Anand, V. Kanhangad, D. Rajan, Pattern Analysis andApplications (4), 1493 (2019) 7.3.3288. H. Park, S. Park, Y. Joo, IEEE Access , 80010 (2020) 7.3.4289. J. Mukherjee, M. Kar, A. Chakrabarti, S. Das, Biocybernetics and Biomedical Engineer-ing (2020) 7.3.4290. S. Mohammadi, M. Noori, A. Bahri, S.G. Majelan, M. Havaei, Pattern Recognition p.107303 (2020) 7.3.4291. V.K. Singh, N. Kumar, S. Madhavan, Artificial Intelligence Review pp. 1–42 (2019) 7.3.4292. A. Hamadi, H. Lattar, M.E.B. Khoussa, B. Safadi, Pattern Analysis and Applications23