Machine Learning Methods for Histopathological Image Analysis: A Review
Jonathan de Matos, Steve Tsham Mpinda Ataky, Alceu de Souza Britto Jr., Luiz Eduardo Soares de Oliveira, Alessandro Lameiras Koerich
RReview
Machine Learning Methods for HistopathologicalImage Analysis: A Review
Jonathan de Matos * , Steve Tsham Mpinda Ataky , Alceu de Souza Britto Jr. , LuizEduardo Soares de Oliveira and Alessandro Lameiras Koerich École de Technologie Superiéure, Université du Québec, Montréal, QC, Canada;[email protected], [email protected], [email protected] Universidade Estadual de Ponta Grossa, Ponta Grossa, PR, Brazil Pontifícia Universidade Católica do Paraná, Curitiba, PR, Brazil; [email protected] Universidade Federal do Paraná, Curitiba, PR, Brazil; [email protected] * Correspondence: [email protected]: date; Accepted: date; Published: date
Abstract:
Histopathological images (HIs) are the gold standard for evaluating some types of tumorsfor cancer diagnosis. The analysis of such images is not only time and resource consuming, but alsovery challenging even for experienced pathologists, resulting in inter- and intra-observer disagreements.One of the ways of accelerating such an analysis is to use computer-aided diagnosis (CAD) systems.In this paper, we present a review on machine learning methods for histopathological image analysis,including shallow and deep learning methods. We also cover the most common tasks in HI analysis, suchas segmentation and feature extraction. In addition, we present a list of publicly available and privatedatasets that have been used in HI research.
Keywords:
Histopathological Images; Machine Learning; Review.
1. Introduction
Current hardware capabilities and computing technologies provide the ability of computers tosolve problems in many fields. The medical field nobly employs technologies as a means of improvingpopulations’ health and life quality. Medical computer-aided diagnosis is one of the suitable examplesthereof. Amongst the aforementioned diagnosis, image-based diagnosis such as magnetic resonanceimaging (MRI), X-rays, computed tomography (CT), and ultrasound have been attracting growing interestof scientists and academics. Likewise, histopathological images (HIs) are another kind of medical imagingobtained by means of microscopy of tissues from biopsies, which brings to the specialists their ability toobserve tissues characteristics in a cell basis (Figure 1).Cancer is a disease with high mortality rates in developed and in developing countries. In additionto causing death, the costs for related treatment are high and have an impact on the public and on theprivate healthcare system, penalizing, therefore, the government and the population. According as itis mentioned by Torre et al. [1], the mortality rate among high-income countries is stabilizing or evendecreasing due to programs regarding the risk factors reduction (e.g. smoking, over-weighting, physicalinactivity) and due to treatment improvements. In low and middle-income countries mortality ratesare rising due to the increase in risk factors. One of the key points of improvements in treatment is theearly detection of tumors. In fact, in 140 out of 184 countries, breast cancer is the most prevalent typeof cancer among women [2]. Imaging exams like mammography, ultrasound or CT can diagnose the
Journal Not Specified , xx a r X i v : . [ c s . C V ] F e b ournal Not Specified , xx , 5 2 of 45 presence of masses growing in breast tissue, notwithstanding the confirmation of which type of tumorcan only be accomplished by means of a biopsy. Biopsies, in turn, take more time to provide a result dueto the acquisition procedure (e.g. fine-needle aspiration or open surgical biopsy), the tissue processing(creation of slide with the staining process) and finally pathologist visual analysis. Naturally, pathologistanalysis is a highly specialized and time-consuming task prone to inter and intra-observer discordance[3]. Furthermore, the staining process can cause the variance in the process of analysis. Hematoxylinand eosin (H&E), although both are the most common and accessible type of stain, they can neverthelessproduce different color intensities depending on the brand, storage time, and temperature. Therefore,computer-aided diagnosis (CAD) can increase pathologists’ throughput and improve the confidence ofresults by not only adding reproducibility to the diagnosis process but also reducing observer subjectivity. (a) (b)Figure 1. Example of ( a ) benign and ( a ) malignant HIs [4] One important feature in cancer diagnosis is the observation of nuclei. Tumors like ductal carcinomaand lobular carcinoma present an irregular growing on epithelial cells at these structures. A high numberof nuclei or a high number of mitotic cells in a small region can indicate the presence of an irregular growthof tissue, representing a tumor. An HI can capture this feature, but besides the nuclei, it will capture otherhealthy tissues that can be seen in images of benign tumors. Stroma is a type of tissue that shows the samecharacteristics in parts of malignant and benign images. Selecting more relevant patches could improvethe classification processes.In the last years, we have experienced an increasing use of machine learning (ML) methods in CADand HI analysis. ML methods have been used in the pathological diagnosis of cancer in different tissuesor organs such as breast, prostate, skin, brain, bones liver, and others. ML methods have also potentialadvantages in HI analysis. ML methods have been widely used in segmentation, feature extraction andclassification of HIs. HIs have rich geometric structures and complex textures, which are different fromthe visual characteristics of macro vision images used in other machine learning tasks such as objectrecognition, face recognition, scene reconstruction or event detection.In this review we attempt to capture the most relevant works of the last decade that employ MLmethods for HI analysis. We present a comprehensive overview of ML methods for HI analysis includingsegmentation, feature extraction and classification. The motivation is to understand the developmentand use of ML methods in HI analysis and discover the future potential of ML methods in HI analysis.Furthermore, this review aims to address the following three research questions:1. Which ML methods have been used for HI classification and how HIs are provided to the ML methods(raw images or pre-processed images or extracted features)? This question aims at identifying whichmonolithic classifiers, ensembles of classifiers or DL methods have been frequently used to classifyHIs. ournal Not Specified , xx , 5 3 of 45
2. Which elements of HIs are considered the most important ones and how they are obtained? Thisquestion aims at identifying which types of tissues or structures can be identified using ML methods.3. What are the trends that have been dominating HI analysis? This question aims at identifying whatare the most promising ML methods for HI analysis for for the near future.The main contributions of this paper are: (i) it covers a period of exponential change in the computervision, from the handcrafted features to representation learning methods; (ii) it is a comprehensive review,which does not focus on HIs of specific tissues or organs; (iii) it categorizes the works according to thetask: segmentation, feature extraction, classification, and representation learning and classification. Thisallows researchers to compare their works in the same context. There are several survey and review papersrelated to HI analysis, which are presented in Section 7. Different from such previous reviews or surveyson HIs that only focus on HIs of specific tissues or organs, or on a single learning modality (supervised,unsupervised or DL techniques), in this review we cover different approaches, methodologies, datasets,and experimental results, so that readers can identify possible opportunities for future research in HIanalysis.This paper is organized as follows. Section 2 proposes a taxonomy to categorize the ML methodsused in HIs as well as an overview of the process of selecting journals and proceedings. Section 3presents the segmentation methods that attempt to identify important structures in HIs, which may helpto diagnosis. Section 4 presents the feature extraction methods that have been used to represent HIs forfurther classification. Section 5 presents the shallow methods that have been used for classifying the maintypes of tissues and tumors in HIs. Given the importance and the growing interest in DL methods, Section 6is devoted to present the recent approaches for HI analysis that employ such methods. Section 7 bringstogether other reviews and surveys papers that have been published recently, as well as a compilationof several HI datasets that have been used in the last decade. Finally, in the last section we present theconclusions and perspectives of future works.
2. Taxomony and Overview
Based on the three research questions presented previously, we have created a search query whichwas slightly adapted to each search engine. We have searched for references comprising the period ofbetween 2008 and 2020 into five research portals (engines): IEEE Xplore, ACM Digital Library, ScienceDirect, Web of Science and Scopus. Table 1 presents the number of results obtained with the search query.We have searched based on the title, abstract and keywords for all search engines, except for Science Direct.In this case, we added the full-text search also, because the number of relevant works was very low. ((histology AND image) or (histopathology AND image) or (eosin AND hematoxylin)) and (("machine learning") or ("artificialintelligence") or ("image processing")) ournal Not Specified , xx , 5 4 of 45 Table 1.
Number of results without exclusion criteria, and after the application of the first and secondexclusion criteria.
Search Number of PapersEngine Search Query After 1 st Filter After 2 nd Filter
IEEE Xplore 96 68 -ACM Digital Library 5 3 -Science Direct 1752 161 -Web of Science 409 54 -Scopus 252 67 -
Total
Based on these results, the first exclusion criterion was based on the title and abstract. Most of theexclusions in this step were due to papers that mentioned "image processing" in the text, but the sense ofthe term was linked to the process of digitizing HIs for visual analysis by pathologists. Another exclusioncriterion was the presence of the terms eosin and hematoxylin or histopathology to exclude medical imagesthat were not the focus of this review, such as CT, MRI or radiology images. Finally, we have eliminatedthe duplicated articles resulting and we ended up with 353 articles. The second exclusion criterion wasbased on the full-text reading to evaluate the adherence of the paper’s contents to the goal of this review,which has excluded almost 50% of the papers retained by the first filtering. Therefore, we have ended upwith 178 articles. Besides the papers selected using the search query, we have also included several papersrelated to the HI datasets used in many references cited in this review, as well as, some other referencesthat discuss about specific ML methods and techniques that are also referred in many of the selectedreferences.
Figure 2.
Taxonomy used to classify HI works in this review. ournal Not Specified , xx , 5 5 of 45 This review focuses on ML methods for HI analysis. Therefore, we have categorized the ML methodsaccording to the most common ML tasks as shown in Figure 2. The top-level categories are: segmentation,feature extraction, shallow methods and deep methods. Notwithstanding DL approaches can be employedfor both segmentation and classification, we proposed this division to highlight how the recent advancesin DL have impacted the research on HI analysis, causing a paradigm shift to DL methods over traditionalML methods.Segmentation of HIs was a very popular category during the first years covered by this review.Most of the works were based on image processing techniques, such as filtering, thresholding, contourdetection techniques, while others rely on ML methods, such as classification and unsupervised learningat pixel level. Besides, inasmuch as the annotation for segmentation is a very time-consuming task,it is also common to find unsupervised methods along with the supervised ones. Most of the earlyworks used segmentation to highlighting information in HIs to specialists. Feature extraction aims atfinding discriminative characteristics in HIs and at aggregating them into a feature vector to train MLalgorithms. Most shallow classifiers and ensemble methods use such a vector representation to learn linearor non-linear decision boundaries. We divided the category of shallow methods into two subcategories:monolithic classifiers and ensemble methods. Ensemble methods combine several diverse base models toreduce bias and/or variance in predictions as well as to improve the accuracy of predictions. The worksthat fall within both subcategories require a previous step of feature extraction.Finally, the category of deep methods contains works focused on supervised and unsupervisedlearning of different architectures of deep neural networks. Most of the works within this category areend-to-end learning approaches, which integrate representation learning and decision-making.The number of publications related to the field of this research is presented in Figure 3. Based onFigure 3 it is possible to note that the research on the topic has been increasing in last years. The searchwas accomplished in January 2020, and regardless of the latter having been performed at the beginning ofthe year, some publications of the same year were found. It is also possible to note a great increase in theuse of DL methods, while ensembles and feature extraction kept their rates. Table 2 shows the number ofpublications per journal between 2008 and 2020 apropos of the subject of this review and Table 3 showsthe top 15 journals in number of publications.
Articles published by year
UnsupervisedSegmentationFeature ExtractionClassificationReviewsDeep LearningEnsemble
Figure 3.
Number of articles per year after filtering and organizing according to the main subjects. ournal Not Specified , xx , 5 6 of 45 Table 2.
Top 20 journals per publication between 2008 and 2020.
Journal Title Area
Computerized Medical Imaging and Graphics CHM 15Medical Image Analysis CHM 13Pattern Recognition C 6Computers in Biology and Medicine CM 5IEEE Transactions on Medical Imaging CEH 5Expert Systems with Applications CE 4IEEE Transactions on Biomedical Engineering E 4Information Sciences C 4Applied Soft Computing C 3Computer Methods and Programs in Biomedicine CM 3Cytometry Part A M 3Procedia Computer Science C 3Artificial Intelligence in Medicine CM 2Computational and Structural Biotechnology Journal BC 2IEEE Access CE 2IEEE Journal of Biomedical and Health Informatics BCE 2Informatics in Medicine Unlocked M 2Journal of Medical Imaging M 2Methods MB 2Micron B 2B: Biochemistry, C: Computing, E: Engineering, H: Health Sciences, M: Medicine.
Table 3.
Top 15 conferences by number of publications between 2008 and 2020
Conference
IEEE Intl Symp on Biomedical Imaging (ISBI) 9IEEE Intl Conf on Healthcare Informatics, Imaging and Systems Biology 2Intl Conf of the IEEE Engineering in Medicine and Biology Society 2Intl Conf on Bioinformatics, Computational Biology and Health Informatics 2Intl Conf on Information Technology in Medicine and Education (ITME) 2Intl Conf on Pattern Recognition (ICPR) 2Intl Symp on Medical Information Processing and Analysis 2Medical Image Computing and Computer-Assisted Intervention 2Medical Imaging: Digital Pathology 2ACM Symp on Applied Computing 1IEEE International Conference on Systems, Man, and Cybernetics (SMC) 1IEEE Intl Conf on Bioinformatics and Biomedicine (BIBM) 1IEEE Intl Conf on Image Processing (ICIP) 1IEEE Intl Conf on Systems, Man, and Cybernetics (SMC) 1IEEE Intl Symp on Multimedia 1
3. Segmentation Methods for HIs
Typically, pathologists look for tissue regions relevant to the disease being diagnosed. HI segmentationusually aims to label regions of pixels according to the structure that they may represent. For instance, theidentification of nuclei structures can be used to extract morphological features, such as the number ofnuclei per region, their size, and format, which may be very helpful to diagnose a tumor. In this sectionwe present several approaches for segmenting HIs where most of them are either based on supervised orunsupervised ML methods. The former requires HI datasets with region annotation of while the latterdoes not require any type of annotation. ournal Not Specified , xx , 5 7 of 45 The k -means algorithm is an unsupervised ML method for clustering that has been used forsegmentation of pixel regions, and in the context of this review, it represents the core of fourteensegmentation methods, as shown in Table 4. Fatakdawala et al. [5] proposed a methodology basedon the expectation-maximization of the geodesic active contour for detecting lymphocyte nuclei, which isable to identify four structures: lymphocyte nuclei, stroma, cancer nuclei, and background. The processinitiates with segmentation by a k -means algorithm, which clusters pixels of similar intensities andafterwards such clusters are improved with an expectation-maximization algorithm. The contours areidentified based on the magnetic interaction theory. After contours have been defined, an algorithmsearches for the concavity of contours, meaning that there is nuclei overlapping. The experiments wereconducted using a breast cancer dataset. A multi-scale segmentation with k -means is the subject of studyof Roullier et al. [6]. This work uses the same idea of the pathologist to analyze a whole slide image(WSI). The segmentation starts at a lower magnification factor and finishes at a higher magnificationwhere it is easier to identify mitotic cells. The result of the clustering algorithm aims to identify regions ofinterest in each magnification. Rahmadwati et al. [7] employed the k -means algorithm to help classify HIs.Although the focus is not on the k -means per se, but on Gabor filters, this clustering method is essentialin the segmentation process. Peng et al. [8] used k -means and principal component analysis (PCA) tosplit HIs into four types of structures: glandular lumen, stroma, epithelial-cell cytoplasm, and cell nuclei.Subsequently, morphological operations of closing and filling are performed. He et al. [9] used a mixtureof local region-scalable fitting and k -means to segment cervix HIs. Fatima et al. [10] used k -means forsegmentation followed by skeletonization and shock graphs to identify nuclei in the previously segmentedimage. If the shock graph provides a confidence value smaller than 0.5 for nucleus identification, a secondattempt of identification is made using a multilayer percepton (MLP). This hybrid approach achieves92.5% of accuracy in nucleus identification.Mazo et al. [11] also used k -means to segment cardiac images in three classes: connective tissues, lightareas, and epithelial tissue. A flooding algorithm processes light areas in order to merge its result withepithelial regions and improve the final result. Finally, plurality rule was used to assign cells into flat,cubic, and cylindrical. This method achieved a sensitivity of 85%. This work was extended in Mazo et al. [12]. Tosun et al. [13] proposed segmentation based on k -means that clusters all pixels into three categories(purple, pink, white), which are further divided into three subcategories. The object-level segmentationbased on clustering achieved 94.89% of accuracy against 86.78% for pixel-level segmentation. Nativ et al. [14] presented a k -means clustering based on morphological features of lipid droplets previouslysegmented using active contours models. A decision tree (DT) was used to verify the rules that lead to theclasses obtained by the clustering. The correlation with pathologist evaluations reached 97%. A two-step k -means is used by Shi et al. [15] in order to segment follicular lymphoma HI. The first step segmentsnuclei and another type of tissues into two clusters. The next step segments "another type tissue" areafrom the previous step into three classes (nuclei, cytoplasm and extracellular spaces). The final step is awatershed algorithm to extract better contours of nuclei. The difference between the manual segmentationand automated was of around 1%. Brieu et al. [16] presented a segmentation approach based on k -means.The result of k -means segmentation is improved and simplified using a sequence of thresholds that attemptto preserve the form of objects. The key point of such a method is not the segmentation, but nucleusdetection. Shi et al. [17] used k -means to cluster pixels represented in the L*a*b color space using pixelneighborhood statistics. A thresholding step improves contours detection of fat droplets, and humanspecialists analyze morphological information related to the droplets to come up with a diagnosis. Shi et al. [18] proposed a segmentation method that considers the local correlation of each pixel. A first clusteringperformed by a k -means algorithm generates a poorly segmented cytoplasm, and a second clustering that ournal Not Specified , xx , 5 8 of 45 does not consider the nuclei identified by the first clustering is performed. Finally, a watershed transformis applied to complete the segmentation.Other clustering algorithms have also been used to segment HIs. The work proposed by Liu et al. [19] used the iterative self-organizing data analysis technique (ISODATA) to cluster cell images and createprototypes. Hafiane et al. [20] studied two strategies for initialization of clustering methods: geodesicactive contours and multi-phase vector level sets. The last one proved to be more efficient when usingspatial constraint fuzzy c-means, with accuracy values of 68.1% and 67.9% respectively, and k -meansachieved 60.6% in this case. He et al. [21] presented segmentation based on Gaussian mixture models. Theirmethodology uses the stain color features (hematoxylin with blue color and eosin in pink and red) to applytwo segmentation steps in the red channel and other channels subsequently. It does not present groundtruth comparison, only visual results compared to k -means. [22] presented a quasi-supervised approachbased on nearest neighbors to cluster an unlabeled dataset based on itself and another labeled dataset. Acomparison between quasi-supervised approach and support vector machine (SVM) has shown that SVMpresents a better performance but it requires labeled data. Yang et al. [23] proposed a system for contentrecovery based on a three-step method that uses histogram features. The first two steps use dissimilaritymeasures of histograms to find candidate images. The last step uses mean shift clustering. The area underthe curve (AUC) of the proposed method is 0.87, which is better than 0.84 achieved by the method based onlocal binary patterns (LBP) features. A mitotic cell detection system using a dictionary of cells is presentedby Sirinukunwattana et al. [24]. A shrinkage/thresholding method groups intensity features representedby a sparse coding to create a dictionary. This method achieved 80.5% and 77.9% of F-score on Asperio andHamatsu subsets of MITOS dataset, respectively. Huang [25] proposed a semi-supervised method basedon exclusive component analysis (XCA) that uses the separation of stains to improve the performance.This method needs a small interaction of the user, who must provide a set of references from nuclei andfrom the cytoplasm. Finally, it is worth mentioning that unsupervised methods based on DL approacheshave also been proposed for segmenting HIs. We will present some recent works in Section 6. Table 4.
Summary of publications on unsupervised ML methods for HI segmentation.
Reference Year Tissue / Organ Method
Liu et al. [19] 2008 Lymph nodes ISODATATosun et al. [13] 2009 Colorectal k -meansHafiane et al. [20] 2009 Prostate Spatial constraint fuzzy c -meansHe et al. [21] 2010 Cervix Gaussian mixture modelsFatakdawala et al. [5] 2010 Breast k -meansRoullier et al. [6] 2011 Breast k -meansRahmadwati et al. [7] 2011 Uterus k -meansPeng et al. [8] 2011 Prostate k -meansHe et al. [9] 2011 Uterus k -meansOnder et al. [22] 2013 Colorectal Quasi-supervised nearest neighborsFatima et al. [10] 2014 Brain k -meansNativ et al. [14] 2014 Liver k -meansYang et al. [23] 2014 Prostate Mean shift, SimilaritySirinukunwattana et al. [24] 2015 Breast Dictionary, ThresholdingHuang [25] 2015 Breast XCAMazo et al. [11] 2016 Cardiac k -meansShi et al. [15] 2016 Lymph nodes k -meansBrieu et al. [16] 2016 Lung k -meansShi et al. [17] 2017 Liver k -meansShi et al. [18] 2017 Lymph nodes k -means ournal Not Specified , xx , 5 9 of 45 In this section, we present the works related to HI segmentation which are based on supervisedML approaches. Most of the works presented in this section are based on classification algorithms andtherefore, they require labeled datasets in which pixels or pixel regions are annotated. Table 5 summarizesthe recent publications on supervised ML methods used for the segmentation purpose, where eight out offourteen works are based on SVM classifiers.Yu and Ip [26] presented an approach to encode HIs using a patching procedure and a method calledspatial hidden Markov model (SHMM). Each patch is represented by a feature vector that uses a mixtureof Gabor energy and gray-level features. The SHMM showed improvements from 4% to 17% in multipletissues in comparison to a hidden Markov model. The work of Arteta et al. [27] uses the concept of extremalregions on gray-scale images to identify nuclei on HIs. In order to identify the threshold of extremalregions, which are organized in overlap tree, they used an SVM classifier. This approach achieved 88.5%of F1-score against 69.8% achieved by the state-of-the-art, considering the number of cells found aftersegmentation. Janssens et al. [28] presented a segmentation procedure to identify muscular cells. First asegmentation based on thresholding identifies connective tissues and cells. Then, an SVM receives thesegmented regions and classify them recursively into three classes (connective tissue, clump of cells andcells) until only connective and cell tissues appear. This approach achieved an F-score of 62%, whichwas the state-of-the-art at that time. Saraswat and Arya [29] proposed a segmentation procedure with anon-dominated sorted genetic algorithm (NSGA-II) and a threshold classifier. The NSGA-II generates thethreshold for feature values from ground-truth images. The comparison between learned thresholds andfeature values generates the segmentation. Breast cancer prognosis is the subject of the study of Qu et al. [30]. They used an SVM to perform pixel-wise classification to separate nuclei from the stroma. A secondstep based on a watershed algorithm identifies nuclei. The approach achieved 72% of accuracy usingpixel-level, object-level, and semantic-level features. Salman et al. [31] proposed a segmentation methodbased on k -NN to analyze WSIs. The method computes histograms from patches of 64 ×
64 pixels extractedfrom the H&E channels obtained by color deconvolution. The best accuracy was 73.2% using histogramsof both H&E channels. Chen et al. [32] proposed a method based on pixel-wise SVM to identify stroma andtumor nests. Nuclei segmentation is carried out by a watershed algorithm, which results in 314 object-levelfeatures and 16 semantic-level features. The feature dimensionality was reduced using the analysis offeature importance. Geessink et al. [33] used a normal density-based quadratic discriminant classifier(QDA) to segment colorectal images. The segmentation uses the L*a*b color space with a threshold toeliminate background pixels and HSV color space to classify the remaining pixels. After classification,errors are corrected based on histological constraints. The algorithm produced an error rate of 0.6% fortumor quantification which, according to the authors, is lower than the error of pathologists (4.4%). Zarella et al. [34] trained an SVM to distinguish stained pixels from unstained pixels. For such an aim, they selectedmanually positively stained pixels and negatively stained pixels from a set of representative images inHSV color space. The SVM identifies regions of interest for further analyses. Santamaria-Pang et al. [35]proposed an algorithm to enhance and improve general segmentation methods by utilizing a cell shaperanking function. The shape of the cells detected by the watershed transform is used to train an SVM,which discriminates real cells from false positives. Wang et al. [36] proposed the use wavelet decomposition,region growing, double strategy splitting model and curvature scale space to highlight nucleus regionsfor further classification. Textural and shape features are extracted from nuclei and feature selection iscarried out based on genetic algorithms and SVM. The best results were 91.5% and 91.6% for sensitivityand specificity, respectively. Arteta et al. [37] improved the post-processing step of the method proposed byArteta et al. [27]. Nucleus regions are refined using a surface. Two nucleus regions have their optimal areadefined by a smoothness factor. The improvement provided 91% of F1-score in the same dataset. A nuclei ournal Not Specified , xx , 5 10 of 45 segmentation was proposed by Brieu and Schmidt [38] based on an adaptive neighborhood provided by aregression tree. A comparison showed an improvement of 9% relative to a nuclei segmentation withoutadaptive thresholding. Finally, Song et al. [39] presented a nuclear segmentation as a cascade of two-classclassification problem. An effective learning formulation was proposed by adapting sparse convolutionalmodels across the different layers in order to estimate the latent morphology information. For improvingthe region probabilities, low-level appearance and high-level contextual features from original imagesand probability maps estimated, respectively, are integrated into a new sequence of probabilistic binaryDTs. The outcome led to a reliable contour set for each nucleus and final complete contour inferences. Theexperimental results over 26,500 nuclei from the Farsight, KIRC, and Kumar datasets showed that theproposed method achieved better performance than other automated segmentation approaches. Again,it is worth mentioning that supervised methods based on DL approaches have also been proposed forsegmenting HIs. We will present such recent works in Section 6. Table 5.
Summary of publications on supervised ML methods for HI segmentation.
Reference Year Tissue / Organ Classifier
Yu and Ip [26] 2008 Gastric SHMMArteta et al. [27] 2012 Breast Structured SVMJanssens et al. [28] 2013 Muscle SVMSaraswat and Arya [29] 2014 Skin NSGA-II, ThresholdQu et al. [30] 2014 Breast SVMSalman et al. [31] 2014 Prostate k -NNChen et al. [32] 2015 Breast SVMGeessink et al. [33] 2015 Colorectal QDAZarella et al. [34] 2015 Breast SVMSantamaria-Pang et al. [35] 2015 Epithelium SVMWang et al. [36] 2016 Breast GA + SVMArteta et al. [37] 2016 Breast Structured SVMBrieu and Schmidt [38] 2017 NA Regression treeSong et al. [39] 2019 Breast, prostate, kidney, liver, stomach, bladder DTNA: Not available.
4. Feature Extraction for HIs
Supervised shallow methods depend on the feature extraction from raw data before performingclassification. HI problems require a transformation of the image pixels into meaningful features prior toclassification. Feature extraction methods process images and provide a reasonable number of featuressummarizing the information contained in the image. In fact, feature extraction methods aim not onlyto reduce the dimensionality of the input, but also to highlight relevant information related to theproblem (presence/absence or amount of a certain element, texture, shape, histogram, etc.) providing arepresentation independent on translation, scale, and rotation. Several different types of features havebeen used with HIs, such as shape, size, texture, fractal, or even combination of these features. Table 6summarizes the articles related to feature extraction.Object-level and morphometric features like shape and size are particularly important for diseasegrading and diagnosis. Ballarò et al. [40] proposed the segmentation of HIs to identify unhealthy orhealthy megakaryocytes, structures from which morphometric features are extracted. Petushi et al. [41]employed the Otsu algorithm to highlight nuclei and then extracted different features such as inside radialcontact, inside line contact, area, perimeter, area-perimeter ratio, curvature, aspect ratio, and major axisalignment. Feature vectors are built by the concatenation of histograms of all these features. Madabhushi et al. [42] presented an approach for predicting disease outcome from multiple modalities including MRI, ournal Not Specified , xx , 5 11 of 45 digital pathology, and protein expression. For histopathology images, they used graph-based featuressuch as Voronoi diagram (total area of all polygons, polygon area, polygon perimeter, polygon chordlength), Delaunay triangulation (triangle side length, triangle area), minimum spanning tree (edge length),and nuclear statistics (density of nuclei, distance to nearest nuclei in different pixel radius) to representthe spatial arrangement of nuclei. Song et al. [43] applied thresholding and watershed transform toextract features like cystic cytoplasm length, cystic mucin production, and cystic cell density. These threefeatures are used to train different classifiers. The experimental results showed that these three featuresoutperformed morphological features (shape and size) achieving 90% of accuracy against 64%. Besidesthat, the combination of these features with morphological features achieved only 85% of accuracy. Thesystem described by Gorelick et al. [44] for prostate cancer detection and classification uses a segmentationstep to identify super pixels, where the segmented images are represented by morphometric and geometricfeatures. The framework for cytological analysis and breast cancer diagnosis presented by Filipczuk et al. [45] employed morphometric features. After isolation of nuclei from the images, for each nucleus theycalculated area, perimeter, eccentricity, major and minor axis length, luminance mean and variance, anddistance to the centroid of all nuclei. Ozolek et al. [46] performed the classification of follicular lesionson thyroid tissue. After a preprocessing step for nucleus segmentation, the chromatin texture of nucleiwith linear optimal transport provides features for the final classification. Fukuma et al. [47] comparedspatial-level and object-level descriptors like Voronoi tessellation, Delaunay triangulation, minimumspanning tree, elliptical, convex hull, bounding box and boundaries. Object-level features reached 99.07%of accuracy at best case against 82.88% achieved by the spatial ones. Morphometric features can also beobtained from other structures like glands, which are easier to identify due to the difference of the lumenand other cellular structures. This is the subject in the work presented by Loeffler et al. [48] which usesinverse compactness and inverse solidness as measures for gland alteration on prostate cancer. The featureswere obtained based on the area (object and convex hull area) and perimeter of threshold highlightedobjects. Marugame et al. [49] used morphometric features extracted from image objects indicating nuclearaggregations to represent three categories of ductal carcinomas in breast HIs. The number of pixels, length,and thickness of the objects reflect their size and shape. Osborne et al. [50] employed four geometricalfeatures extracted from nuclei after segmentation to melanoma diagnoses in skin HIs. The four featuresare the ratio of the area of nuclei to the area of cytoplasm, the ratio of the perimeter of a nucleus to itsarea, the ratio of the major axis length of a nucleus to its minor axis length, and the ratio of the number ofnuclei to the area of cytoplasm. The multi-view approach to detect prostate cancer presented by Kwakand Hewitt [51] extracted morphological and intensity features from multiple resolutions. Features likearea, compactness, smoothness, roundness, convex hull ratio, major-minor axis ratio, extent, boundingcircle ratio, distortion, and shape context are extracted from lumens and epithelial nuclei, as well asother relational features between them. Olgun et al. [52] introduced a feature extractor for HIs, which isbased on the local distributions of objects, which are segmented by color intensity. The feature extractormeasures the distance between an object and its neighborhood. The proposed method outperformed other13 methods that use textural and structural features.Texture descriptors have become quite popular in HI analysis due to the different types of texturesfound in HIs. For instance, high/low concentration of nuclei and stroma present quite different patternsof textures. For this reason, several researchers have been investigating a large spectrum of texturaldescriptors for HI classification. Descriptors based on GLCM had been used by several authors torepresent textures in HI. Kuse et al. [53] used GLCM as features with a pre-segmentation process basedon unsupervised mean-shift clustering. Such a method reduces color variety to segment the image usingthresholds. After this process nuclei are identified and have the overlapping removed by a contour andarea restrictions. Finally, GLCM features are extracted from the segmented image used for classification.Caicedo et al. [54] combined seven feature extraction methods, including GLCM, and create a kernel-based ournal Not Specified , xx , 5 12 of 45 representation of the data on each feature type. Kernels are used inside an SVM to find similarity betweendata and to implement a content retrieval mechanism. Fernández-Carrobles et al. [55] presented a featureextraction method based on frequency and spacial textons. The use of textons implies that images arerepresented by a reduced vocabulary of textures. Features used for the classification are histograms oftextons and GLCM features extracted from texton maps. They also evaluated the impact of differentcolormaps on these features. Best classification results (98.1%) were achieved by combining six colormodels and GLCM for textons. Despite the fact that GLCM requires a gray-level image, the conversion ofthe H&E color image to gray-level is affected by the variability of the staining color, so in the end GLCM isalso affected.Another descriptor that is very often used to represent texture is the Local Binary Pattern (LBP). Mazo et al. [12] proposed the classification of cardiac tissues into five categories using a patching approach thataims to optimize the patch size to improve the representation. The texture of HIs was described usingLocal Binary Patterns (LBP), LBP Rotation Invariant (LBPri), Haralick features and different concatenationsbetween them. Haralick features include contrast, angular second moment, homogeneity, correlation,entropy, and first and second correlation measures. Peyret et al. [56] applied LBP in the context ofmultispectral HIs. They used an SVM to evaluate the proposed LBP, which aligns all spectra and usespixels from all other bands. It also uses a multi-scale kernel size. This feature extractor reached 99% ofaccuracy compared to 88.3% achieved by the standard LBP and 95.8% reached by the concatenated spectraLBP. Bruno et al. [57] used a curvelet transform to handle multiscale HIs. The LBP algorithm extractsfeatures from curvelets coefficients which are reduced by an ANOVA analysis. The algorithm proposedby Phoulady et al. [58] uses adaptive and iterative thresholding to find nuclei area and extracts textureinformation using LBP and histograms of oriented gradients (HOG). The proposed method achieved 93.3%of accuracy against 92.3% of the second-best method. The work presented by Reis et al. [59] focused onthe stroma maturity to evaluate breast cancer. The features for the stroma are Basic Image Features (BIF),obtained by convolving images with a bank of derivatives-of-Gaussian filters, and LBP with multiplescales for the neighborhood. Gertych et al. [60] presented a system for prostate cancer classification, whichalso uses LBP as feature. The best accuracy was 68.4% for cancer detection. Balazsi et al. [61] presented aninvasive ductal breast carcinoma detector that extracts patches by tesselation without the square shapeconstraint. A set of 16,128 features derived from multiple histograms and LBP (multiple radii) using L*a*b,gray-scale and RGB color spaces are used to represent each patch. Atupelage et al. [62] extracted featuresusing fractal geometry analysis, and compare them with Gabor filter bank, Leung-Malik filter bank, LBPand GLCM features. The proposed approach outperformed the other methods achieving 95% of accuracy.Huang et al. [63] proposed a two-step feature extraction approach composed of a receptive fieldfor detecting regions of interest and a sparse coding. The sparse coding groups features from patchesof the same region. The mean and covariance matrix of receptive fields and sparse coding are the finalfilters. Noroozi and Zakerolhosseini [64] proposed an automated method for discriminating basal cellcarcinoma tumor from squamous cell carcinoma tumor in skin HIs using Z-transform features, which areobtained from the combination of Fourier transform features. Wan et al. [65] used a dual-tree complexwavelet transform (DT-CWT) to represent the images in the context of mitosis detection in breast cancerdetection. Generalized Gaussian distribution and symmetric alpha-stable distribution parameters wereused as features. Chan and Tuszynski [66] also used fractal dimension features for breast cancer detection.These features perform well for an HI magnification of 40 × to distinguish between malignant and benigntumors.Recently, deep features have become very popular in several image classification tasks, including HIs.Niazi et al. [67] presented a CAD system for bladder cancer that focuses on the extraction of epitheliumfeatures with segmentation using an automatic color deconvolution matrix construction. Spanhol et al. [68]used deep features from a pre-trained AlexNet to classify breast benign and malignant tumors. Vo et al. [69] ournal Not Specified , xx , 5 13 of 45 presented a method for feature extraction based on the combination of CNNs and boosting tree classifiers(BTC). This method utilizes an ensemble of inception CNNs to extract visual features from multi-scaleimages. In the first stage, data augmentation methods were employed. Afterwards, ensembles of CNNswere trained to extract multi-context information from multi-scale images. The latter stage extracted bothglobal and local features of breast cancer tumors. George et al. [70] proposed an approach for breast cancerdiagnosis, which extracts features from nuclei based on CNNs. The methodology consists of differentapproaches for extracting nucleus features from HIs and select the most discriminative spatially sparsenucleus patches. A pre-trained set of CNNs was used to extract features from such patches. Subsequently,features belonging to individual images are fused using 3-norm pooling to obtain image level features.Finally, several works use or combine different feature categories in an attempt to capture informationfrom both textures and geometrical structures found in HIs. Leo et al. [71] presented a method forquantifying instability of features across four prostate cancer datasets with known variations due tostaining, preparation, and scanning platforms. They evaluated five families of features: graph-basedfeatures, which include first- and second-order descriptors of Voronoi diagrams, Delaunay triangulations,minimum spanning trees, and gland density; gland shape features, which measure the average shape ofall the glands in an image, and include the lumen boundaries and the resulting area, perimeter, distance,smoothness, and Fourier descriptors; co-occurring gland tensor features, which capture the disorder ofneighborhoods of glands as measured by the entropy of orientation of the major axes of glands withina local neighborhood; subgraph features, which describe the connectivity and clustering of small glandneighborhoods using gland centroids; Haralick texture features. Yu et al. [72] investigated the best featuresfor characterizing lung cancer. The authors extracted objective quantitative image features such as Haralicktexture features of the nuclei (sum entropy, InfoMeas, difference variance, angular second moment), edgeintensity of the nuclei, texture features of the cytoplasm and intensity distribution of the cytoplasm,Zernike shape, texture and radial distribution of intensity. Caicedo et al. [73] proposed a low-level tohigh-level mapping to facilitate imaging retrieval. This mapping process consists of gray and colorhistograms, LBP, Tamura texture histogram, Sobel histogram, and invariant feature histograms. Pang et al. [74] proposed a CAD system for lung cancer detection, which uses textural features such as LBP,GLCM, and Tamura, shape features such as SIFT, global features and morphological features. Kruk et al. [75] used morphometric, textural, and statistical (histogram) features to describe nuclei for clear-cell renalcarcinoma grading. Genetic algorithm and Fisher discriminant were used to select the most importantfeatures. Basavanhally et al. [76] proposed a multi field-of-view (FOV) classification scheme to recognizelow versus high-grade ductal carcinoma from breast HIs. It uses a multiple patch size procedure for WSIto analyze whether morphological or textural or graph-based features is the most relevant to each patchsize. Tashk et al. [77] presented a complete framework for breast HI classification that estimates mitoticpixels in L*a*b color space. A combination of LBP, morphometric, and statistical features are extractedfrom mitotic candidates. Cruz-Roa et al. [78] proposed a patching method on HI slides to create smallregions and extract scale-invariant feature transform (SIFT), luminance level, and discrete cosine transformfeatures to create a bag-of-words. Semantic features are high-level information that can be associated withHIs to aid their classification.Orlov et al. [79] compared four color spaces (RGB, L*a*b, gray-scale and RGB) with H&E representationand eleven features such as Zernike, Chebychev, Chebyshev-Fourier, color histograms, GLCM, Tamura,Gabor, Haralick, edge statistics and others to represent lymph node HIs. De et al. [80] propose a fusionof several feature types for uterine cervical cancer HI classification. They used a 62-dimensional featurevector based on GLCM, Delaunay triangulation and weighted density distribution. Vanderbeck et al. [81]used morphological, textural and pixel neighboring statistics features to represent seven categories ofwhite regions of liver HIs. Kandemir et al. [82] proposed a MIL approach to detect Barrett’s cancer inHIs. They used cell-level morphometric features such as central power sums, area, radius, perimeter, and ournal Not Specified , xx , 5 14 of 45 roundness of segments, maximum, mean, and minimum intensity, and intensity covariance, variance,skewness, and kurtosis within regions and patch-level features such as LBP, SIFT and color histogramsfrom segmented images using the watershed algorithm. Coatelen et al. [83][84] proposed a feature selectionmethod of liver HI classification based on morphometric features such as area, compactness, perimeter,aspect ratio, Zernick moment, etc., textural features such as GLCM, LBP, fractal dimension, Fourierdistance, etc., and structural or graph-based features such as number of nodes/edges, modularity, pi,eta, theta, beta, alpha, gamma and Shimbel indexes, etc. Two greedy algorithms (fselector and in-houserecursive) selected features in a pool of 200 features where the fitness function was implemented byan SVM classifier. Michail et al. [85] highlighted nuclei using connected-component labeling to classifycentroblast and non-centroblasts cells. Morphometric, textural and color features are used as features. Das et al. [86] proposed the so-called geometric- and texture-aware features, which are based on Hu momentsand fractal dimensional, respectively. Such a set of features was applied to detect geometrical and texturalchanges in nuclei to discriminate mitotic and non-mitotic cells. The method proposed by Kong et al. [87]classifies neuroblastomas using textural and morphological features. It considers that pathologists usemorphological features for their analysis and textural features can be easily extracted. They also use GLCMfeatures and sequential floating forward selection to select features.In summary, given the rich geometric structures and complex textures that may be found in HIs, mostof the works combine different types of features. Morphometric features are important to characterizegeometric structures, but they are more complex to obtain since they require complex pre-processing,e.g. find the contour of nuclei to count them. On the other hand, textural features such as LBP andGLCM usually do not require a previous segmentation of HIs. Finally, the most recent methods of featureextraction are focused on deep features. They can be interpreted as a sequence of filters that can detect bothgeometric structures and textures. Therefore, deep features and deep methods seem to be very promisingmethods for HI analysis.
5. Shallow Methods for HI Classification
ML algorithms trained in a supervised fashion can accomplish different HI analyses such asidentifying types of tumors and tissues, nucleus features (e.g. mitosis phases) or specific characteristics insome organs (e.g. fat inside the liver or the size of epithelial tissue on the cervix). We present in this sectionthe ML methods based on shallow classifiers. We start by presenting some works that employ single(monolithic) classifiers followed by classification methods based on ensemble (multiple) of classifiers.Both shallow and ensemble methods depend on a previous stage of feature extraction because they relyon handcrafted feature vectors to learn discriminant functions. Therefore, most of the feature extractionmethods presented in Section 4 can be used together with the methods presented in this section.
Different ML methods for supervised learning have been employed in HI analysis, such as supportvector machines (SVM), decision trees (DT), naïve Bayes (NB), k -nearest neighbors ( k -NN), multilayerperceptron (MLP), among others. Table 7 summarizes the works reviewed in this section in terms ofclassification algorithm, tissue or organ from where the HI was obtained and the publication year.SVMs are the most used classification algorithm for HIs. Several works have employed SVM withdifferent feature categories. Mazo et al. [12] proposed the classification of cardiac tissues into five categoriesusing a patching approach that aims to optimize the patch size to improve the representation. A cascadeof linear SVMs separate tissues in four classes, followed by a polynomial SVM, which classifies one ofthese four classes in two sub-classes. Osborne et al. [50] employed segmentation and morphologicalfeatures with an SVM classifier to melanoma diagnoses in skin HIs. The propose approach achieved ournal Not Specified , xx , 5 15 of 45 Table 6.
Summary of publications devoted to feature extraction from HIs.
Reference Year Tissue / FeatureOrgan
Caicedo et al. [73] 2008 Skin Color and gray histograms, LBP, TamuraBallarò et al. [40] 2008 Bone MorphometricMarugame et al. [49] 2009 Breast MorphometricKong et al. [87] 2009 Brain Textural, morphologicalKuse et al. [53] 2010 Lymph nodes GLCMOrlov et al. [79] 2010 Lymph nodes Zernike, Chebychev, Chebyshev-Fourier, colorhistograms, GLCM, Tamura, Gabor, Haralick,edge statisticsPetushi et al. [41] 2011 Breast MorphometricMadabhushi et al. [42] 2011 Prostate Voronoi diagram, Delaunay triangulation,minimum spanning tree, nuclear statisticsOsborne et al. [50] 2011 Skin MorphometricCaicedo et al. [54] 2011 Skin Gray, color, invariant feature, Sobel, TamuraLBP, SIFTHuang et al. [63] 2011 Breast Receptive field, sparse codingCruz-Roa et al. [78] 2011 Skin SIFT, luminance, DCTLoeffler et al. [48] 2012 Prostate MorphometricSong et al. [43] 2013 Pancreas MorphometricGorelick et al. [44] 2013 Prostate Morphometric, geometricFilipczuk et al. [45] 2013 Breast MorphometricAtupelage et al. [62] 2013 Blood Fractal dimensionBasavanhally et al. [76] 2013 Breast Morphological, textural, graph-basedDe et al. [80] 2013 Uterus GLCM, Delaunay triangulation, weighteddensity distributionOzolek et al. [46] 2014 Thyroid Linear optimal transportOlgun et al. [52] 2014 Colorectal Local object patternMichail et al. [85] 2014 Lymph nodes Morphometric, textureVanderbeck et al. [81] 2014 Liver Morphological, textural, pixel neighboringstatisticsKandemir et al. [82] 2014 Esophagus Morphometric, LBP, SIFT, color histogramsFernández-Carrobles et al. [55] 2015 Breast TextonsGertych et al. [60] 2015 Prostate LBPTashk et al. [77] 2015 Breast LBP, morphometric, statisticalCoatelen et al. [83][84] 2015 Liver Morphometric, GLCM, LBP, fractal dimension,Graph-basedBalazsi et al. [61] 2016 Breast LBPFukuma et al. [47] 2016 Brain Object, spatialLeo et al. [71] 2016 Prostate Graph-based, shape, entropy, subgraphconnectivity, texturePhoulady et al. [58] 2016 Uterus HOG, LBPBruno et al. [57] 2016 Breast Curvelet transform, LBPNoroozi and Zakerolhosseini [64] 2016 Skin Z-transform coefficientsNiazi et al. [67] 2016 Bladder MorphometricYu et al. [72] 2016 Lung Quantitative, textureChan and Tuszynski [66] 2016 Breast Fractal dimensionKwak and Hewitt [51] 2017 Prostate MorphometricReis et al. [59] 2017 Breast BIF, LBPMazo et al. [12] 2017 Cardiac LBP, HaralickWan et al. [65] 2017 Breast Wavelet transform, Gaussian distribution,Symmetric alpha-stableSpanhol et al. [68] 2017 Breast DeepDas et al. [86] 2017 Oral Hu’s moment, fractal dimension, entropyPang et al. [74] 2017 Lung LBP, GLCM, Tamura, SIFT, global, morphometricKruk et al. [75] 2017 Kidney Morphometric, textural, and statisticalPeyret et al. [56] 2018 Prostate LBPVo et al. [69] 2019 Breast DeepGeorge et al. [70] 2019 Breast Deep ournal Not Specified , xx , 5 16 of 45
90% of accuracy. Malon et al. [88] compared the agreement of three pathologists and a ML method thatuses deep features to train an SVM classifier to locate mitotic nuclei in HIs. The accuracy achievedby the SVM was 63.6% and 98.6% for positive and negative cases respectively, which was close to twopathologists’ performance. Only one pathologist performed 99.2% and 94.5% on positive and negativesamples, respectively. Atupelage et al. [62] used fractal features and an SVM to classify non-neoplastictissues and tumors and to grade hepatocellular carcinoma HIs into five classes. The proposed approachachieved 95% of correct classification rate for five classes and outperformed other methods that usestexture features. Olgun et al. [52] introduced and approach for representation and classification of colontissue HIs, which is based on the local distributions of objects. This approach was evaluated using anSVM and compared with other 13 methods that use textural and structural features. It outperformedall methods achieving an accuracy of 93%. Wan et al. [65] used a dual-tree complex wavelet transform(DT-CWT) to represent breast HIs for mitosis detection. Generalized Gaussian distribution and symmetricalpha-stable distribution parameters were the features used for classification with an SVM. The proposedmethod achieved 73% of F-score outperforming most of other methods compared in their study. Chan andTuszynski [66] used fractal features and an SVM classifier for breast cancer detection. They achieved 97.9%of F-score for an HI magnification of 40 × to distinguish between malignant and benign tumors. On theother hand, on multiclass problem, they reached a F-score of only 55.6%. Caicedo et al. [73] proposed alow-level to high-level mapping to facilitate imaging retrieval. This mapping uses color, texture and shapefeatures to train 18 SVMs. The experimental results compared the low-level and high-level (semantic)features, which obtained 67% against 80% of precision respectively, showing that the mapping from lowto semantic-level features contributes favorably to the classification process. Vanderbeck et al. [81] usedSVM to classify white regions of liver HIs among seven classes. The best accuracy of 93.5% was achievedfor the combination of all features into a 413-dimensional feature vector. They also compared the resultsbased on images labeled by different pathologists. In an extension of the work of De et al. [80], Guo et al. [89] presented an automatic orientation detection for the epithelium with more features and usedan SVM classifier. Harai and Tanaka [90] presented a colorectal CAD system, which starts with an Otsuthresholding of red channel to separate nuclei, background, and stroma. An SVM classifier achieves 78.3%of accuracy against 67% of a method based on texture features. Peikari et al. [91] proposed a nucleussegmentation pipeline based on multi-level thresholding and watershed algorithm on the L*a*b color space.The nucleus classification uses a cascade of SVMs. The cascade phase initially separates lymphocytesfrom epithelial tissue and then classify epithelial in benign and malignant. An interesting comparisonwith two pathologists’ evaluation shows that the agreement between pathologists was 89% and betweenthe automated system was 74% and 75%. The classification of ovarian cancer is the subject of study ofBenTaieb et al. [92]. The proposed method localizes regions of cancer in WSI using a multi-scale mechanismconsidering that each tumor type has specific characteristics which are better detected at different scales.The method automatically selects a ROI based on multiple scales. The latent variable of the latent SVM(LSVM) used for classification is the presence of a patch at a particular scale on the classification of thatregion. The LSVM approach achieved the accuracy of 76.2%, which outperforms CNNs by 26%. Zhang et al. [93] proposed a multi-scale classification that uses sparse coding implemented by means of Fisherdiscriminant analysis to construct a visual dictionary with SIFT features. The multi-scale approach usingSVMs achieved the accuracy of 81.6%, which outperformed the state-of-the-art (79.5%). Korkmaz andPoyraz [94] proposed a classification framework based on minimum redundancy maximum relevancefeature selection and least square SVM (LSSVM). They claimed the accuracy of approximately 100%with only four false negatives for benign tumors in a three-class problem. No further comparisons wereperformed.Bayesian, DT, NN, k -NN and other supervised ML algorithms have also been used for classificationof HIs. Several works have employed such classifiers with different feature categories. Marugame ournal Not Specified , xx , 5 17 of 45 et al. [49] proposed a simple classifier based on Bayes theory to classify ductal carcinomas into threecategories. Specialists consulted by authors claimed that the simple classifier provides, together with themorphological features, a better way to understand the results. Spanhol et al. [68] used deep features froman AlexNet to classify breast benign and malignant tumors. The deep features were used with a logisticregression classifier. This approach achieved 90.3% of correct classification rate for 200 × magnificationfactor and outperformed a baseline (87.8%) that used texture features. De et al. [80] proposed grading ofuterine cervical cancer using an LDA classifier. A specialist manually segmented the images to identifytumors and split them into ten segments for feature extraction. A voting strategy combined resultsfrom the segments. The best grading result was 70.5% for the whole epithelium against 62.3% for thevertically segmented epithelium. Mete and Topaloglu [95] evaluated eleven different color spaces forrepresenting HIs. The combination of a spherical coordinate transform and DT achieved the best accuracy,outperforming SVM and NB classifiers. Sidiropoulos et al. [96] proposed a classification algorithm basedon a probabilistic neural network (PNN) implemented on GPUs to grade cases of rare brain tumors. Theadvantage is the reduced processing time that allows an exhaustive feature combination search. Fordemonstration purposes, a comparison of CPU- and GPU-based algorithms showed that the GPU versiontakes 278 times less computation time than the CPU version for a feature vector with 20 attributes. Thework presented by Michail et al. [97] classifies follicular lymphomas using a preprocessing step to segmentimages based on intensity thresholds and an expectation maximization algorithm. The segmented cells areclassified by LDA, achieving a detection rate of 82.6%. A random kitchen sink (RKS) classifier is used byBeevi et al. [98] to identify mitotic nuclei on breast cancer HIs. Nuclei are identified using thresholding inthe red channel. Local active contour model selects and models nuclei. The approach achieved F-score of96.0% for RKS and 83.4% for RFs on MITOS 2014 dataset. A CAD system proposed by Jothi and Rajam [99]used HI converted to gray-scale giving priority to the red channel. Otsu thresholding guided by particleswarm optimization segments the gray-scale images and noise segments are reduced using area constraintsbased on the nuclei size. The closest matching rule (CMR) classifier achieved the accuracy of 100% against99.5% for NB. Awan et al. [100] studied the classification of mitosis on breast cancer using a dataset labeledwith the four major phases of mitosis. Classes are imbalanced, posing a challenge for the classification.They proposed a data augmentation method based on PCA and its eigenvectors and compared it to thesynthetic minority over-sampling technique. Barker et al. [101] used a patching procedure based on a gridover the WSI to grade brain tumor type. Each patch has general features clustered using k -means. The finalclassification is performed over the features of the nuclei identified in the clustering step using elastic netmodel. The proposed model outperformed the methods from the 2014 MICCAI Pathology Classificationchallenge.Multiple instance learning (MIL) has also been used for classification of HIs. Several works haveemployed such MIL methods with different classifiers and feature categories. MIL is a weakly supervisedlearning paradigm that considers that instances are naturally grouped in labeled bags, without the needthat all the instances of each bag have individual labels. The MIL method proposed in [82] to comparethree MIL SVMs: SIL-SVM, MI-SVM and mi-SVM with mi-Graph. mi-Graph achieved accuracy of 87%against 69% of mi-SVM. The proposed methodology is based on patching. All images are previouslysegmented using the watershed algorithm. Another work from the same research group carried out abenchmark of MIL SVM methods, finding out that MILBoost gives better accuracy for instance-levelapproach (66.7%) and mi-Graph performs better in bag-level prediction (72.5%) [102]. A stain separation isperformed in Cosatto et al. [103] using a support vector regressor (SVR) to identify regions of interest (ROI),which is a high occurrence of hematoxylin in low-level magnification. This work uses an MIL approachbecause ROIs are not labeled but the WSIs, so all ROIs from a positive slide to receive positive labels. MILuses MLP for classification, but it requires a modified loss function to represent the one-positive rule for aslide, which means that if in the prediction only one ROI appears as positive, the entire slide is positive. In ournal Not Specified , xx , 5 18 of 45 the comparison between the MIL approach and SVM classification, the SVM required ROI labeling. TheAUC of MIL was 0.95 against 0.94 of SVM with the advantage of reducing labeling efforts. Xu et al. [104]introduced MCIL, an MIL-based method that uses patching procedure to create instance-level Gaussianclassifiers which are clustered using the k -means algorithm. The work performs comparisons with regularimage-level classification methods and MIL methods. The fully supervised method presented F-score of76.6% (using patch labeling) and the proposed method achieved 69.9%. MCIL achieved 71.7% and 60.1%in another dataset (not patch labeled) with constrained and unconstrained MCIL respectively against25.3% of MIL-Boosting. Sudharshan et al. [105] compared different MIL approaches to the diagnostic ofbreast cancer patients. In this approach, every patient is seen as a bag, which is labeled with her diagnosis.Therefore, HIs do not need to be individually labeled, as they can share the bag label. Instances arepatches extracted from the corresponding HIs, considering different magnification factors (40 × , 100 × ,200 × and 400 × ). The hypothesis is that a bag-based (patches) analysis is valuable for the analysis of HIsin comparison with single instance (entire image). The experiments were carried out on the BreakHisdatabase using CNN, 1-NN, QDA, RF and SVM classifiers and the best accuracy of 92.1% was achievedfor 40 × magnification by non-parametric MIL.Finally, several works compare the performance of different monolithic classifiers on HIs, but withoutcombining their predictions. Ballarò et al. [40] proposed the segmentation of HIs to identify unhealthyor healthy megakaryocytes. The classification is carried out by a k -NN and DTs. Song et al. [43] traineddifferent classifiers such as SVM, k -NN, neural networks, and Naïve Bayes (NB) on morphometric features.The experimental results showed that these three features outperform morphological features achieving90% of accuracy against 64%. Besides that, the combination of these features with morphological featuresachieves only 85% of accuracy. Bruno et al. [57] used a curvelet transform to handle multi-scale HIs. Texturefeatures extracted from curvelets coefficients which are reduced by an ANOVA analysis and evaluatedusing DTs, SVM, RF and polynomial classifiers. They achieved an AUC of 1.00 which is higher than thebest previous method (0.986). Pang et al. [74] proposed a CAD system for lung cancer detection. Sparsecontribution analysis selects non-redundant features, which are used to train SVM, RF and extreme learningmachine (ELM). Another contribution is the concave-convex variation, which consists of measuring theconcavity of all nuclei in an image and use such a measurement to weight the probabilities of the classifiers.This method achieved the accuracy of 98.74%, which is slightly better than RFs (97.68%). Orlov et al. [79]compared four color spaces (RGB, L*a*b, gray-scale and RGB) with H&E representation. A weighted k -NN achieved the best results (99%) followed by an RBF network and NB with 99% and 90% of accuracy,respectively. The best results were achieved for a color space called eosin representation. Irshad et al. [106]presented a multimodal approach with multispectral images focusing on selecting the best spectral bandsfor classification of mitotic cells on MITOS 2012 dataset. SVM, DT, and MLP are used for classificationpurpose. SVM achieved the best F-score (63.7%) using only eight best bands, which is higher than thestate-of-the-art (59%). WSI is the core of the work proposed by Homeyer et al. [107], which compares k -NN, NB and RFs for classification of slides based on a patching procedure and textural and intensityfeatures. RFs with a group of all features achieved the best result (94.7%). Khan et al. [108] proposed aframework for malignant cell classification in breast cytology images. Selected features train SVM, NBand RF classifiers. At the end, an ensemble method is employed to combine the classifiers based on themajority voting. The experiments have shown the accuracy of 98.0% in the detection and classificationof malignant cells. Kurmi et al. [109] presented an approach consisting of nuclei localization in HIs andfurther classification as benign or malignant using MLP and SVM models. MLP achieved the best averageaccuracy of 95.03%. ournal Not Specified , xx , 5 19 of 45 Table 7.
Summary of publications focusing on HI classification based on monolithic classifiers.
Reference Year Tissue / Organ Classifier
Caicedo et al. [73] 2008 Skin SVMBallarò et al. [40] 2008 Bone DT, k -NNMete and Topaloglu [95] 2009 Skin DT, NB, SVMMarugame et al. [49] 2009 Breast BayesOrlov et al. [79] 2010 Lymph nodes k -NN, NB, RBFOsborne et al. [50] 2011 Skin SVMMalon et al. [88] 2012 Breast SVMSidiropoulos et al. [96] 2012 Brain PNNDe et al. [80] 2013 Uterus LDAAtupelage et al. [62] 2013 Liver SVMCosatto et al. [103] 2013 Gastric MLP (MIL)Homeyer et al. [107] 2013 Liver k -NN, NB, RFSong et al. [43] 2013 Pancreas k -NN, NB, NN, SVMIrshad et al. [106] 2014 Breast DT, MLP, SVMXu et al. [104] 2014 Colorectal Gaussian (MIL)Kandemir et al. [82] 2014 Gastric SVM (MIL)Olgun et al. [52] 2014 Colon SVMVanderbeck et al. [81] 2014 Liver SVMCoatelen et al. [83] [84] 2014 Liver SVMMichail et al. [97] 2014 Lymph nodes LDAHarai and Tanaka [90] 2015 Colorectal k -NNKorkmaz and Poyraz [94] 2015 Breast SVMKandemir and Hamprecht [102] 2015 Gastric SVM (MIL)Chan and Tuszynski [66] 2016 Breast SVMGuo et al. [89] 2016 Uterus SVMBeevi et al. [98] 2016 Breast RKSJothi and Rajam [99] 2016 Thyroid VPRS + CMRBarker et al. [101] 2016 Brain Elastic netBruno et al. [57] 2016 Breast DT, Polynomial, RF, SVMPang et al. [74] 2017 Liver ELM, RF, SVMWan et al. [65] 2017 Breast SVMMazo et al. [12] 2017 Cardiac SVMPeikari et al. [91] 2017 Breast SVMBenTaieb et al. [92] 2017 Ovary SVMZhang et al. [93] 2017 Lung SVMSpanhol et al. [68] 2017 Breast Logistic regressionSudharshan et al. [105] 2019 Breast SVM, k -NN, QDA, RF, CNN (MIL)Khan et al. [108] 2019 Breast NB, RF, SVMKurmi et al. [109] 2019 Breast SVM, MLP ournal Not Specified , xx , 5 20 of 45 Ensembles approaches combine the predictions of multiple base classifiers in an attempt to improvegeneralization and/or robustness over a single classifier. Several researchers have proposed combiningclassifiers for improving the performance of HI approaches. Table 8 summarizes the works reviewed inthis section in terms of type of base classifier and combination strategy, tissue or organ from where the HIwas obtained and the publication year.Zarella et al. [34] employed classification using an ensemble of SVMs on ROIs segmented from WSI.Multiple "weak" classifiers trained with subsets of features and different parameters combined with aweighted sum (WS) function achieved the accuracy of 88.6%. Daskalakis et al. [110] used a preprocessingstep of segmentation to enhance nuclei and extract morphological and textural features. A multi-classifierapproach combines k -NN, linear least squares minimum distance (LLSMD), statistical quadratic Bayesian,SVM, and PNN using majority vote, minimum, maximum, average, and product rules. PNN achievedthe best accuracy of 89.6% for a base classifier while the ensemble method achieved 95.7% with themajority vote rule. The method proposed by Kong et al. [87] classifies neuroblastomas using textural andmorphological features. An ensemble approach combining k -NN, LDA, NB and SVM classifiers usingthe weighted voting (WV) rule achieved the accuracy of 87.8%. Meng et al. [111] proposed an ensembleof principal component classifiers (PCC). This ensemble classified 25 patches of each image, which arerepresented by 50 features. The accuracy achieved on a liver dataset was 96.41% using the majority vote(MV) rule compared to 95.09% achieved by a 3-NN. The same approach achieved 99.4% of accuracy onlymphoma classification against 92.08% achieved by the Adaboost approach. A CAD system composed ofa staining separation module, densitometric and texture feature extraction and an AdaBoost algorithmwas proposed by Wang and Yu [112]. The proposed system achieved accuracy of 94.37% against 86.44%of the best base classifier ( k -NN) trained on raw H&E images. The system described by Gorelick et al. [44] uses a segmentation step to identify super pixels. An Adaboost algorithm classifies the segmentedimages represented by morphometric and geometric features. The system achieved the accuracy of 85%. Aframework for cytological analysis is presented by Filipczuk et al. [45]. Morphometric features representnuclei obtained after segmentation. The proposed method uses a combination of random subspaces withperceptrons to create an ensemble. The comparison showed that the ensemble approach achieved accuracyof 96.0% compared to 90% achieved by a boosting algorithm. Vink et al. [113] proposed a nucleus detectionmethod based on two Adaboost stages. The first step is based on features extracted from stain separatedimages. The second Adaboost refines the result of the first with line-based features. An optimal activecontour refines the results from the second ensemble achieving the accuracy of 95.02%. Phoulady et al. [114] proposed an ensemble of Otsu thresholding algorithms with certain constraints and morphologicaloperations. Four segmentation algorithms are responsible for the segmentation, but each image canhave characteristics that would require different parameters for the segmentor set. The final result ofsegmentation is one among 18 segmentor sets that have most parameters shared with the set of segmentorsthat presented less difference in the segmentation. This approach achieved accuracy of 84.3% compared to77.4% achieved by other methods.Di Franco et al. [115] used an ensemble of SVM classifiers, where each model is trained with avariation of images pre-processed by Gaussian filters and color spaces. The classifiers are combined usingthe average rule and the best AUC value achieved was 0.978. Albashish et al. [116] proposed a featureselection method that uses the entropy of a feature in relation to a class as a redundancy criterion andconstraints in this value and the inter-feature entropy. SVM classifiers specialized in one subtype of tissuederived from prior segmentation are combined using the sum rule. The performance of ensemble approachusing 37 features (94.08%) is only 0.2% better than the best SVM with recursive feature learning (RFE)method using 46 features. A comparison of multiple classifiers and features is presented by Huang and ournal Not Specified , xx , 5 21 of 45 Kalaw [117]. A set of monolithic classifiers is compared with Adaboost implemented with SVM, DT,and RF. Adaboost achieved 97.8% of accuracy. Fernández-Carrobles et al. [118] presented a classificationframework for WSI with a bagging of DTs and GLCM features which achieved 0.995 for AUC and 98.13%for true positives. The multi-view approach presented by Kwak and Hewitt [51] extracts features frommultiple resolutions. A boosting algorithm combining linear SVMs and the features from multiple viewsachieved 0.98 of AUC compared to 0.96 of the concatenation of views. Kruk et al. [75] used morphometric,textural, and statistical features to describe nuclei for classification. An ensemble made up of SVM and RFclassifiers and trained with a subset of features resulting from the feature selection achieved the accuracyof 96.7%, which was higher than the state-of-the-art (93.1%) and the best single SVM classifier (91.1%).An Adaboost ensemble is used by Romo-Bucheli et al. [119] to grade skin cancer. The ensemble classifiesimages described by features created with graph theory to represent the nuclei distribution. The ensembleachieved 72% of accuracy. A multi field-of-view (FOV) classification scheme is proposed by Basavanhally et al. [76]. It uses a multiple patch size procedure for WSI that firstly analyzes which features are the mostrelevant to each patch size. After that, it uses a RF to aggregate multiple FOV patches. They do not presenta baseline for accuracy comparison, only the AUC result, showing better values for nucleus architecturefeatures to recognize low versus high-grade ductal carcinoma.
Table 8.
Summary of recent publication on ensemble approaches for HI analysis.
Reference Year Tissue / Base Classifier CombinationOrgan Rule/Function
Daskalakis et al. [110] 2008 Thyroid k -NN, LLSMD, SQ-Bayes, Vot, Min, Max,SVM, PNN Sum, ProdKong et al. [87] 2009 Neuroblastoma k -NN, LDA, Bayesian, SVM WVMeng et al. [111] 2010 Liver, PCC WVLymphocytesDiFranco et al. [120] 2011 Prostate SVM and RF MVWang and Yu [112] 2013 Lung DT AdaboostGorelick et al. [44] 2013 Prostate DT AdaboostFilipczuk et al. [45] 2013 Breast SVM, Perceptron PerceptronVink et al. [113] 2013 Breast DT, Stumps AdaboostBasavanhally et al. [76] 2013 Breast RF MVPhoulady et al. [114] 2014 Uterus Otsu segmentors SimilarityZarella et al. [34] 2015 Lymphoma SVM WSDi Franco et al. [115] 2015 Prostate SVM AvgGertych et al. [60] 2015 Prostate SVM, RF MVTashk et al. [77] 2015 Breast SVM, RF MVAlbashish et al. [116] 2015 Prostate SVM SumHuang and Kalaw [117] 2016 Prostate k -NN, SVM, DT, RF, LDA, AdaboostQDA, NBBalazsi et al. [61] 2016 Breast RF MVWright et al. [121] 2016 Colorectal RF MVFernández-Carrobles et al. [118] 2016 Breast DT Sum, VarianceKwak and Hewitt [51] 2017 Prostate SVM BoostingKruk et al. [75] 2017 Kidney SVM + RF MVValkonen et al. [122] 2017 Breast RF MVRomo-Bucheli et al. [119] 2017 Skin NA AdaboostNA: Not available, WV: Weighted vote, MV: Majority vote, Avg: Average, Min: Minimum, Max: Maximum,Sum: Summation, Prod: Product. Tashk et al. [77] presented a complete framework for HI classification. They employ maximumlikelihood estimation to obtain the mitotic pixels in L*a*b color space. A cascading classification is ournal Not Specified , xx , 5 22 of 45 performed firstly with SVM and next with RFs. A comparison shows that this method achieves theaccuracy of 96.5% against 82.4% of the best previous result in MITOS 2012 dataset. Gertych et al. [60]presented a system for prostate cancer classification, which consists of SVM and RF classifiers. SVMseparates the stroma and epithelium and the RF identifies benign/normal and carcinogenic tissue. Thebest accuracy was 68.4% for cancer detection. Balazsi et al. [61] extended the work described in [123]. Theauthors used simple linear iterative clustering (SLIC) to extract patches by tesselation. A set of multiplehistogram and texture features are extracted from the L*a*b, gray-scale and RGB color spaces of each patch.This number of features is suitable for a RF classifier, which achieved 79.51% of F-score for tessellatedpatches, compared to 77.57% of squared patches and 71.80% of the baseline. SLIC is also applied by Wright et al. [121] in a pipeline for colorectal cancer to initially segment images. Histogram and texture featuresextracted from the HSV color space; likewise, statistics from H&E channels were extracted, in addition toGLCM as features. A comparison showed that the proposed work achieved the accuracy of 79% against75% from their previous work for RF. Valkonen et al. [122] presented a system for the classification of WSI.The segmentation step uses Otsu, morphological operations and histological constraints. The classificationalgorithms, including RF, SVM, k -NN and logistic regression were trained with textural, morphometric,and statistical features extracted from random patches of segmented images. RF achieved the best accuracy(93%). A comparison between different ensemble approaches to classify patches of WSI is presented in[120]. A set of 114 features were selected and ranked using RFs. Based on the selected and ranked features,multiple linear and RBF SVMs and RF classifiers were built. The aggregation function was the majorityvote. The AUCs were 0.955, 0.951 and 0.948 for RBF SVM, RF and linear SVM respectively. The bestprevious result was 0.935.
6. Methods Based on Deep Learning (DL)
DL methods are gaining the attention of the scientific community due to recent achievements to solvecomplex machine learning problems on large datasets. A convolutional neural network (CNN) is able tolearn in a single optimization process, both a representation and a decision boundary. However, CNNsusually require a huge amount of data for adequate training in order to avoid overfitting problems, butmost of the HI datasets have only a few patients and hundreds of images. Data augmentation [124] [125]and transfer learning [126] are two possible approaches to circumvent the lack of data in HI datasets. Forinstance, ImageNet, which has more than 14 million images, is one of the most common datasets usedfor training CNNs for object recognition. Data augmentation generates new HIs from existing ones byusing affine transformations or morphological operations. Another common way of data augmentation ispatching HIs, which consists in producing the effect of selecting pieces of a HI with the same structure butthat belong to different classes. On the other hand, the transfer learning method reuses CNNs previouslytrained in large datasets, which usually belongs to a different domain from the target problem. Thepre-trained CNNs can be used in two ways: to extract features from HIs and use these features withshallow classifiers, as already described in Sections 4 and 5; to fine-tune such CNNs on an HI dataset,which means that filters learned on a large dataset will be adapted to the HI dataset. Recently, DL methodshave been employed in HI analysis. Table 9 summarizes the works reviewed in this section in terms ofnetwork architecture, tissue or organ from where the HI was obtained and the publication year.Malon et al. [88] were one the first authors to employ DL methods in HI analysis. They used aclassical LeNet-5, a 7-layer CNN architecture proposed by Lecun et al. [127], which in 1998 to learn arepresentation from HIs previously segmented with an SVR. The features extracted by the CNN wereclassified by an SVM. The purpose of the classification was to find mitotic nuclei. The remarkable aspectof this work is the comparison between machines and three pathologists. The pathologists showed aCohen Kappa factor of 0.13 and 0.44 in the best case, emphasizing the inter-observer problem. Kainz et al. ournal Not Specified , xx , 5 23 of 45 [128] presented two CNNs based on the LeNet-5 architecture for segmentation and classification of glandsin tissue of benign and malignant colorectal cancer. The first CNN separates glands from background,while the second CNN identifies gland-separating structures. Experimental results on Warwick-QU colonadenocarcinoma and GlaS@MICCAI2015 challenge datasets showed a tissue classification accuracy of 98%and 95%, respectively.Some works used CNNs based on the AlexNet architecture proposed by Krizhevsky et al. [129] in2012. AlexNet is similar to LeNet-5 but it has 12 layers, with more filters per layer, and with stackedconvolutional layers. Stanitsas et al. [130] employed the AlexNet CNN to classify breast cancer HIs. Theycompared the CNN results with some handcrafted feature extractors and shallow classifiers and theyconcluded that the CNN was not able to outperform the shallow methods. Spanhol et al. [131] evaluatedarchitectures based on AlexNet CNN for the problem of breast cancer HI classification. The experimentalresults on the BreaKHis dataset showed that the CNN achieved mean accuracy rates between 81.7% and88.6%, depending on the magnification, at patient level, which is better than other shallow ML approachesbased on textural features. Sharma et al. [132] also used an AlexNet CNN as well as other custom CNNarchitectures to classify benign and malignant tumors. Due to the small sample size, authors had to carryout data augmentation by patching and affine transforms. For cancer classification, 11 WSIs produced231,000 images. For necrosis detection, four WSIs produced a total of 47,130 images for training. BothAlexNet and the custom CNN architectures compare favorably to most handcrafted features and a RFclassifier. Budak et al. [133] proposed an end-to-end model based on a pre-trained AlexNet CNN anda bidirectional LSTM (BLSTM) for detecting breast cancer in HIs. The convolutional layers are used toencode HIs into a high-level representation, which is flattened and fed into the BLSTM. Experimentalresults on the BreaKHis dataset showed that the proposed model achieved the best average accuracy of96.32% for the magnification factor of 200 × . Moreover, for the magnification factor of 40 × , 100 × , and400 × , the average accuracy was 95.69%, 93.61%, and 94.29%, respectively.Some works use CNNs based on the inception architecture proposed by Szegedy et al. [134]. Theinception modules have parallel paths where the image is passed through filters of different dimensions(1 ×
1, 3 ×
3, 5 × et al. [137]compared AlexNet and Inception-V1, handcrafted features and SVM, and features extracted by CNNs toclassify regions of colon histology images as either gland or non-gland. The combination of handcraftedfeatures with an SVM and the prediction of a CNN showed the best results. They used data augmentationwith rotations and mirroring for handcrafted features and CNNs. Yan et al. [138] integrated a pre-trainedInception-V3 with a BLSTM for classifying breast cancer HIs into normal, benign, in situ carcinoma, orinvasive carcinoma. The method consists of dividing HIs into 12 small patches on average. Afterwards, afine-tuned Inception-V3 CNN extracts features from the patches, where a 5,376-dimensional feature vectoris made up of the concatenation of the weights of the last three layers of the CNN. Such feature vectors arethe input of a BLSTM compounded of four layers to fuse the features of the 12 small patches and comeup to an image-wise classification. The experiments show that such an approach achieved the averageaccuracy of 91.3%. de Matos et al. [126] proposed a classification approach for breast cancer HIs that usestransfer learning to extract features from HIs using an Inception-V3 CNN pre-trained with the ImageNetdataset. The proposed approach improved the classification accuracy in 3.7% using the feature extractiontransfer learning and an additional 0.7% using the irrelevant patch elimination. ournal Not Specified , xx , 5 24 of 45 Deep residual neural network (ResNet) [139] is another architecture that has been used in theclassification of HIs. The residual block alleviates the problem of training very deep networks. Khosravi et al. [140] evaluated the versatility of CNNs on eight different datasets of breast, lung, and bladder tissueswith H&E and immunohistochemistry images (IHC). Such an evaluation included Inception and ResNetCNNs, the combination of both CNNs, as well as the concept of transfer learning. Results showed agood performance in spite of using the raw images without any pre-processing. Vizcarra et al. [141] fusedCNN and SVM outputs for HI classification. The pipeline consists of extracting SURF features for theshallow learner (SVM) and color normalization (Reinhard method) and image resizing (downsampling)for fine-tuning Inception-V3 and Inception-ResNet-V2 CNNs, pre-trained on the ImageNet dataset. CNN.The output from both shallow and deep learners are fused for final prediction. Experimental results onthe BACH dataset showed a moderate accuracy of 79% and 81% achieved by the SVM and the CNN,respectively. On the other hand, the fusion of SVM and CNN outputs outperformed the individuallearners, achieving the accuracy of 92%. Zerhouni et al. [142] proposed the use of a wide residual CNNto classify mitotic and non-mitotic pixels in breast HIs. The CNN is trained on mitotic and non-mitoticpatches extracted from the ground truth images. Experimental results on the MICCAI TUPAC Challengedataset showed that the wide residual CNN outperformed most of other approaches. Gandomkar et al. [143] proposed the MuDeRN framework aiming at classifying HIs into benign or malignant, and nextinto four subtypes. In the first stage, a ResNet with 152 layers has been trained to classify HI patchesof different magnification factors as whether benign or malignant. Afterwards, the results thereof weresubdivided into four subcategories of malignant and benign likewise. Lastly, for each patient, the diagnosiswas conducted by combining the output of the ResNet using a meta-DT. MuDeRN achieved at the firststage the accuracy of 98.52%, 97.90%, 98.33%, and 97.66% for 40 × , 100 × , 200 × , and 400 × magnificationfactors, respectively. In the second stage, MuDeRN achieved the accuracy of 95.40%, 94.90%, 95.70%, and94.60% for 40 × , 100 × , 200 × , and 400 × magnification factors, respectively. For patient-level diagnosis, inturn, MuDeRN achieved the accuracy of 96.25%, considering the eight classes. Brancati et al. [144] alsoused a ResNet to detect invasive ductal carcinoma as well as to classify lymphoma subtypes. First, theconvolutional layers are trained in an unsupervised way for extracting useful features to reconstruct theinput image. On the other hand, the fully connect layers are trained in a supervised way. In both cases, thesoftmax classifier produces a probability of the input image belonging to a given class. Talo [145] presentedan approach based on pre-trained ResNet-50 and DenseNet-161 CNN models for automatic classificationof gray-scale and color HIs. The results achieved by both CNNs outperformed the existing studies in theliterature, with 95.79% of total accuracy for the gray-scale images. ResNet-50 achieved 97.77% of totalaccuracy to classify color HIs.Another CNN architecture that has been used in HI classification is the VGG-net, which is a veryuniform architecture that has 16 convolutional layers with a large number of 3 × et al. [146] used a VGG-net CNN [147] for classification of tissue into epithelium, stroma, and fat followed bya VGG16 CNN for classifying stroma into normal stroma or tumor-associated stroma. The first CNNachieved a pixel-level accuracy of 95.5%, while the second CNN achieved a pixel-level binary accuracy of92.0%. The authors employed data augmentation by randomly rotating and flipping patches, as well as byrandomly jittering the hue and saturation of pixels in the HSV color space. The work presented by Xu et al. [148] segments and distinguishes glands. They proposed an approach combining a fully convolutionalnetwork (FCN) for the foreground segmentation channel, a faster region-based CNN (R-CNN) for theobject detection channel, and a holistically-nested edge detector CNN for the edge detection channel. Allthree CNNs are based on the VGG16 CNN. The results of these three CNNs feed another CNN that outputsa segmented image. Data augmentation by affine and elastic transformation is carried out to enhanceperformance and avoid overfitting. The proposed approach achieved state-of-the-art results on the datasetfrom the MICCAI 2015 Gland Segmentation Challenge. Kumar et al. [149] developed a variant of VGG16 ournal Not Specified , xx , 5 25 of 45 CNN architecture, which replaces the fully connected layers by different classifiers. The approach consistsof stain normalization and data augmentation, which uses images with and without normalized stain. Theaugmented dataset is applied to the fused VGG16, where features are taken at the global average poolinglayer. Finally, the binary classification is carried out by SVM and RF classifiers. Experiments conducted oncanine mammary tumor (CMTHis) and breast cancer HIs (BreakHis), which are both randomly split intotraining (70%) and test (30%) sets. The approach achieved the accuracy of 97%, and 93% on BreakHis andCMTHis datasets, respectively.Other CNN architectures have also been used in HI classification, such as DenseNet [150] andMobileNet [151]. Kassani et al. [152] proposed an approach for classification of breast cancer HIs based onan ensemble of three pre-trained CNNs, namely VGG19, MobileNet, and DenseNet. Stain normalization,data augmentation, fine-tuning and hyper-parameter tuning were used to improve the performance of theCNNs. The multi-model ensemble method achieved better performance than single classifiers with theaccuracy of 98.13%, 95.00%, 94.64%, and 83.10% for BreakHis, ICIAR, PatchCamelyon, and Bioimagingdatasets, respectively. Yang et al. [153] introduced the use of additional region-level supervision forclassifying breast cancer HIs with a DenseNet-169 CNN pre-trained on ImageNet. For this purpose, ROIsare localized and used to guide the attention of the classification network simultaneously. This processactivates neurons in regions relevant to diagnose while suppressing activation in irrelevant and noisy areas.Hence, the prediction of the network is based on the regions which a pathologist expects the network tofocus on. Such an approach achieved the accuracy of 93% on the BACH dataset.Finally, several works proposed custom CNN architectures for HI classification, which are usuallybased on some well-known architectures. The authors attempt to optimize mainly the number andthe dimension of kernels and the number of layers. Bayramoglu et al. [154] proposed two differentCNN architectures, both with 10 layers, for breast cancer HI classification. The first CNN predicts onlymalignancy, while the second one predicts both malignancy and image magnification level simultaneously.Experimental results on the BreaKHis dataset showed that the magnification independent CNN approachimproved the performance of magnification specific model, and that the results are comparable withprevious state-of-the-art results obtained by handcrafted features. They also used data augmentationbased on affine transformations.Albarqouni et al. [155] introduced a CNN for aggregating annotations from crowds in conjunctionwith learning a model for a challenging classification task. During learning from crowd annotationsphase, the CNN architecture is augmented with an aggregation layer to aggregate the ground-truth fromthe crowdvote matrix. Experimental results on the AMIDA13 dataset showed that the proposed CNNarchitecture was robust to noisy labels and positively influences the performance. Cruz-Roa et al. [123]proposed a custom 3-layer CNN to classify patches of WSI as invasive ductal carcinoma (breast cancer) ornot. Patches ended up labeled due to the region labeling. Some regions of the WSI such as backgroundand adipose cells were excluded manually and were not patched. Patches were pre-processed using colornormalization and the YUV color space. CNN outperformed an RF trained on the best handcraft featureextractor by 4%. Compared to other works, this one has a simple protocol and uses a small network,but it was one of the precursors of CNNs to analyze HIs. Ciompi et al. [156] proposed an 11-layer CNNto analyze the impact of stain normalization in the training and evaluation pipeline of an automaticsystem for CRC tissue classification. Experimental results on the CRC dataset validated the performanceof the proposed CNN as well as the role of stain normalization in CRC tissue classification. Kwak andHewitt [157] proposed a 6-layer CNN to identify prostate cancer and compared it with other CNNs(AlexNet, VGG, GoogLeNet, ResNet) as well as with shallow classifiers such as SVM, RF, k -NN and NB.Experimental results on four tissue microarrays showed that the 6-layer CNN achieved an AUC of 0.974and it outperformed all other approaches either based on handcrafted features with shallow classifiers orother CNN architectures. ournal Not Specified , xx , 5 26 of 45 Roy et al. [158] proposed five custom CNN architectures for classification of patches of breast cancerHIs. The approach consists of extracting patches, classify them and compare the result of individual patcheswith the one of the whole image. The output is considered correct if there is an agreement between the classlabels of all extracted patches and the class label of the HI. They have also boosted the number of trainingsamples per class using affine transformation for data augmentation. Experimental results on the ICIAR2018 challenge dataset showed that a 14-layer CNN achieved the best results: patch-wise classificationaccuracy of 77.4% and 84.7% for four and two classes respectively; image-wise classification accuracyof 90.0% and 92.5% for four and two classes, respectively. de Matos et al. [124] proposed a 7-layer CNNarchitecture based on texture filters that has fewer parameters than traditional CNNs but is able to capturethe difference between malignant and benign tissues with relative accuracy. The experimental results onthe BreakHis dataset showed that the proposed texture CNN achieves 85% of accuracy for classifyingbenign and malignant tissues. The authors also employed data augmentation based on composed randomaffine transforms including flipping, rotation, and translation. Ataky et al. [125] proposed a novel approachfor augmenting HI dataset considering the inter-patient variability by means of image blending usingthe Gaussian-Laplacian pyramid. Experimental results on the BreakHis dataset with a texture CNN [124]have shown promising gains vis-à-vis the majority of DA techniques presented in the literature. Theresearch carried out by Gecer et al. [159] presented a method for breast diagnosis based on WSIs. Thismethod aims at classifying images into five categories. At first, a salience detection was performed by apipeline consisting of four sequential 9-layer CNNs based on the VGG-net [147] architecture for multi-scaleprocessing of the WSIs, considering different magnifications for localization of diagnostically pertinentROIs. Afterwards, a patch-based multi-class CNN is trained on representative ROIs resulting from theconsensus of three experienced pathologists. Finally, the final slide-level diagnosis is obtained by fusingthe salience detector and the CNN for pixel-wise labeling of the WSIs by a majority vote rule. Theyclaimed that the CNNs used for both detection and classification outperformed competing methods thatused handcrafted features and statistical classifiers. Moreover, the proposed method achieved resultscomparable to the diagnoses provided by 45 pathologists on the same dataset. Experiments using 240WSIs showed the five-class slide-level accuracy of 55%.Wang et al. [160] employed a bilinear CNN (BCNN), which consists of two individual CNNs, whoseoutputs of the convolutional layers are multiplied with outer product at each corresponding spatiallocation, resulting in the quadratic number of feature maps. The input of both CNNs is H&E imageswith the H and E channels separated in a pre-processing stage by a color decomposition algorithm. Theproposed BCNN-based algorithm achieves the best performance with a mean classification accuracy of92.6%. Compared to other CNN-based algorithms, BCNN improves at least 2.4% on classification accuracyon the CRC dataset. Li et al. [162] presented an automatic method for mitosis detection based on semanticsegmentation. Such a method used a CNN in which a novel label with concentric circles was addedinstead of a single-pixel representation of mitosis. The inner circle represents a mitotic region whereasthe ring around the inner circle is a "middle ground". This concentric loss allows training the semanticsegmentation CNN with weakly annotated mitosis data. The semantic segmentation employed on breastcancer HIs to seek out mitotic cells achieved the F-score of 0.562, 0.673, 0.669, on ICPR2014, MITOSIS,AMIDA13, and TUPAC16 datasets, respectively. Hou et al. [161] proposed a semi-supervised approach thatuses a sparse convolutional autoencoder (CAE) with a crosswise constraint that decomposes patches fromHIs into foreground (e.g. nuclei) and background (e.g. cytoplasm). Such a CAE initializes a supervisedCNN, which carries out nucleus detection, feature extraction, and classification/segmentation in anend-to-end fashion. The experimental results not only showed that the proposed approach outperformedother approaches, but also the noteworthiness of the crosswise constraint in boosting performance. Theproposed CAE-CNN achieved results comparable to the state-of-the-art using only 5% of training dataneeded by other methods. Sheikh et al. [163] proposed a four-input 24-layer custom CNN for classification ournal Not Specified , xx , 5 27 of 45 Table 9.
Summary of publications using DL methods in HI analysis.
Reference Year Tissue / Organ Network Architecture
Malon et al. [88] 2012 Breast LeNet-5Cruz-Roa et al. [123] 2014 Breast 3-layer CustomStanitsas et al. [130] 2016 Breast AlexNetSpanhol et al. [131] 2016 Breast AlexNetBayramoglu et al. [154] 2016 Breast 10-layer CustomAlbarqouni et al. [155] 2016 Breast AggNet CustomLi et al. [137] 2016 Gland AlexNet, Inception-V1Zerhouni et al. [142] 2017 Breast Wide ResNetBejnordi et al. [146] 2017 Breast VGG-net, VGG16Wang et al. [160] 2017 Colorectal Bilinear CustomCiompi et al. [156] 2017 Colorectal 11-layer CustomKainz et al. [128] 2017 Colorectal LeNetSharma et al. [132] 2017 Gastric AlexNet, CustomXu et al. [148] 2017 Gland VGG16Kwak and Hewitt [157] 2017 Prostate 6-layer CustomKhosravi et al. [140] 2018 Breast, Lung, Bladder Inception-V1, ResNetGandomkar et al. [143] 2018 Breast ResNetHou et al. [161] 2019 Gland, Breast CAE+CNN CustomLi et al. [162] 2019 Breast FCN CustomVizcarra et al. [141] 2019 Breast Inception-V3, Inception-ResNet-V2Brancati et al. [144] 2019 Breast ResNetBudak et al. [133] 2019 Breast AlexNet, BLSTMKassani et al. [152] 2019 Breast VGG19, MobileNet, DenseNetYang et al. [153] 2019 Breast DenseNet-169Roy et al. [158] 2019 Breast 11-layer to 14-layer CustomGecer et al. [159] 2019 Breast 9-layer CustomYan et al. [138] 2019 Breast Inception-V3, BLSTMTalo [145] 2019 Breast ResNet-50, DenseNet-161Kassani et al. [152] 2019 Breast VGG19, MobileNet, DenseNetYang et al. [153] 2019 Breast DenseNet-169de Matos et al. [124] 2019 Breast 7-layer Texture Customde Matos et al. [126] 2019 Breast Inception-V3Kumar et al. [149] 2020 Breast VGG16Ataky et al. [125] 2020 Breast 7-layer Texture CustomSheikh et al. [163] 2020 Breast 24-layer Custom ournal Not Specified , xx , 5 28 of 45 of HIs that fuses multi-resolution hierarchical feature maps at different layers. The proposed modellearns different scale image patches to account for the overall structures and texture features of cells.Experimental results on ICIAR2018 and BreaKHis datasets showed that the proposed model outperformedexisting state-of-the-art models. Table 10.
Summary of the reviews and surveys on HIs and ML approaches.
Reference Year Image Subject Journal or ConferenceType He et al. [164] 2012 HI Segmentation, feature Comp Methods Progrextraction, classification BiomedIrshad et al. [165] 2014 HI, Nuclei extraction, segmentation, IEEE Reviews BiomedIHC Feature extraction, classification EngDeshmukh and Mankar [166] 2014 HI, IHC Segmentation Intl Conf Electr Syst SigOther Proc Comp TechnAkhila and Preethymol [167] 2015 HI Nuclei segmentation, Intl Conf Innov Informclassification Emb Comm SysVeta et al. [168] 2015 HI Results of MITOS2013 Challenge Medical Image AnalysisNawaz and Yuan [169] 2016 Various Tumor ecology Cancer LettersMadabhushi and Lee [170] 2016 HI Detection, segmentation, feature Medical Image Analysisextraction, classificationSaha et al. [171] 2016 HI Slide preparation, staining, Tissue and Cellmicroscopic, imaging,preprocessing, segmentation,feature extraction, classificationChen et al. [172] 2017 HI Image analysis of H&E slides Tumor BiologyRobertson et al. [173] 2017 Various DL Translat ResearchCosma et al. [174] 2017 HI, Deep and shallow methods Expert Sys AppOtherTosta et al. [175] 2017 HI Segmentation for lymphocytes Inform Medicine UnlockedLitjens et al. [176] 2017 MI DL for medical images Medical Image AnalysisCataldo and Ficarra [177] 2017 HI Feature extraction Comput Struct Biotechn JAswathy and Jagannath [178] 2017 HI Image processing, classification Inform Medicine UnlockedLi et al. [179] 2018 MI Content retrieval Medical Image AnalysisKomura and Ishikawa [180] 2018 HI Datasets and ML methods Comput Struct Biotechn JZhou et al. [181] 2020 HI Classical and deep neural IEEE Accessnetworks, classificationKrithiga and Geetha [182] 2020 HI Image enhancement, segmentation, Archives Computfeature extraction, classification Methods EngMI: Medical images; IHC: Immunohistochemistry images.
7. Reviews, Surveys and Datasets
This section brings a summary of the reviews and surveys related to HIs and ML methods. Asshown in Table 10, we have found nineteen works in this category. Reviews and surveys publishedbetween 2012 and 2015 highlight mainly approaches for nucleus segmentation and classification. Onthe other hand, recent publications are focused on classification of whole medical images. The reviewspresented by Saha et al. [171], Nawaz and Yuan [169], Chen et al. [172] and Robertson et al. [173] werepublished in medical journals and provided a deeper view of the histology information. However, suchpublications overlooked aspects related to ML methods. For instance, Nawaz and Yuan [169] analyzedthe characteristics of tumors and presented a brief study on how computational methods can deal withHIs. Komura and Ishikawa [180] presented the use of ML methods in HI as well as several HI datasets.Litjens et al. [176] reviewed DL methods applied to a variety of medical images, including HIs. Zhou et al. [181] presented a comprehensive overview of breast HI analysis techniques based on both classical andDL methods and publicly HI datasets. Finally, Krithiga and Geetha [182] presented a systematic review of ournal Not Specified , xx , 5 29 of 45 breast cancer detection, segmentation and classification on HIs focused on the performance evaluation ofML and DL techniques to predict breast cancer recurrence rates.Given the importance of datasets for the research on HI, we have also compiled in Tables 11 and 12, alist of the datasets that have been used in experiments of several works we covered in this review. Weincluded the dataset reference, year of creation, their contents in terms of the number of images andpatients, and references to the papers that have used them. Table 11.
Summary of publicly available HI datasets.
Year Reference Dataset Reference Dataset Size et al. [53] [183] 20 Img2010 Meng et al. [111] [184] 528 Img, 265 Img, 376 Img2012 Arteta et al. [27] [183] 20 Img2014 Irshad et al. [106] [185] 200 Img2015 Sirinukunwattana et al. [24] [185] 50 WSI2015 Huang [25] [186] NA2016 Beevi et al. [98] [187] 96 Img2016 Arteta et al. [37] [183] 20 Img2016 Yu et al. [72] [188,189] 2,186 WSI, 294 Img2016 Chan and Tuszynski [66] [190] 82 Pat, 7,909 Img2016 Barker et al. [101] [186,191] 45 Img, 604 Img2016 Huang and Kalaw [117] [186] 682 Img2017 Das et al. [86] [185] 15 Img2017 Reis et al. [59] [192] 55 WSI2017 Mazo et al. [12] [193] 3,000 Img2017 Valkonen et al. [122] [194] 170 WSI, 100 WSI2017 Wan et al. [65] [185] 50 Img2017 Kruk et al. [75] [195,196] 70 Pat, 62 Pat, 94 Img2018 Sudharshan et al. [105] [190] 82 Pat, 7,909 Img2018 Hou et al. [161] [186], [191] NA2019 Gandomkar et al. [143] [190] 82 Pat, 7,909 Img2019 Kumar et al. [149] [190], CMTHis 82 Pat, 7,909 Img2019 Vo et al. [69] [190], [197] 82 Pat, 7,909 Img, 269 Img2019 Vizcarra et al. [141] [4] 400 Img, 30 WSI2019 Yan et al. [138] NA 249 Img2019 George et al. [70] [190],[197] 82 Pat, 7,909 Img, 269 Img2019 Budak et al. [133] [190] 82 Pat, 7,909 Img2019 Kurmi et al. [109] [190] 82 Pat, 7,909 Img2019 Kassani et al. [152] [190], [197], [194] 82 Pat, 7,909 Img, 269 Img2019 Yang et al. [153] [4] 400 Img, 30 WSI2019 Roy et al. [158] [4] 400 Img, 30 WSINA: Not available, Img: Images, Pat: Patients, WSI: Whole Slide Image. ournal Not Specified , xx , 5 30 of 45 Table 12.
Summary of datasets that are not publicly available.
Year Reference Dataset Reference Dataset Size et al. [40] NA 297 Img2008 Liu et al. [19] NA 480 Img2008 Caicedo et al. [73] NA 1,502 Img2008 Daskalakis et al. [110] NA 115 Img2009 Marugame et al. [49] NA 217 WSI2009 Mete and Topaloglu [95] NA 2 WSI2009 Kong et al. [87] NA 389 Img2009 Tosun et al. [13] NA 16 pat2009 Hafiane et al. [20] [199] 8 Img2010 Orlov et al. [79] NA 30 WSI2010 He et al. [21] NA NA2010 Fatakdawala et al. [5] NA 100 Img, 9 Pat2011 Huang et al. [63] NA 9 Slides, 36,000 Img, 40 × et al. [42] [200], [201], [202] 58 Pat, 100 Img, 20 Pat, 40 Img, 6 Pat2011 Cruz-Roa et al. [78] NA 1,502 Img basal, 2,828 Img tissues2011 Caicedo et al. [54] [203] 6,0002011 Petushi et al. [41] NA 30 WSI2011 Osborne et al. [50] NA 34 cases, 126 Img2011 Roullier et al. [6] NA NA2011 He et al. [9] NA NA2011 Peng et al. [8] NA 8 Pat, 62 Img2011 Rahmadwati et al. [7] NA 475 Img2011 DiFranco et al. [120] NA 14 Pat, 15 Img2012 Loeffler et al. [48] NA 125 Pat2012 Sidiropoulos et al. [96] NA 140 cases2013 Atupelage et al. [62] NA 109 Pat, WSI 369 Img2013 Song et al. [43] NA 11 slides, 7 Pat2013 Basavanhally et al. [76] [204], [205] 126 Pat, 29 Pat2013 De et al. [80] NA 62 Img2013 Homeyer et al. [107] NA 71 Img2013 Cosatto et al. [103] NA 12,726 Pat, 12,745 WSI, 26,879 Img2013 Janssens et al. [28] NA 111 Img2013 Onder et al. [22] NA 230 Img2013 Wang and Yu [112] NA 369 Img2013 Gorelick et al. [44] NA 50 WSI2013 Filipczuk et al. [45] NA 675 Img, 75 Pat2013 Vink et al. [113] NA 51 Img2014 Vanderbeck et al. [81] NA 59 Pat2014 Kandemir et al. [82] NA 97 Pat, 214 Tissue2014 Saraswat and Arya [29] [206] 30 Img2014 Olgun et al. [52] NA 3,236 Img, 258 Pat2014 Qu et al. [30] [207] 125 Pat, 1,180 Img2014 Fatima et al. [10] NA 5 Pat, 80 Img2014 Xu et al. [104] [208] 10 Img, 103 Img2014 Salman et al. [31] NA 20 Pat, 200 Img2014 Michail et al. [85] NA 300 Img2014 Ozolek et al. [46] NA 94 Pat2014 Nativ et al. [14] NA 54 Img, 9 Pat2014 Yang et al. [23] [209] 96 WSI2014 Phoulady et al. [114] NA 28,698 ImgNA: Not available, Img: Images, Pat: Patients, WSI: Whole Slide Image. ournal Not Specified , xx , 5 31 of 45 Year Reference Dataset Reference Dataset Size et al. [55] NA 40 WSI2015 Harai and Tanaka [90] NA 123 Pat, 400 Img2015 Chen et al. [32] NA 230 Pat, 1,150 Img2015 Korkmaz and Poyraz [94] NA 160 Img2015 Santamaria-Pang et al. [35] NA 350 Img2015 Tashk et al. [77] [185] 50 Img2015 Kandemir and Hamprecht [102] [210] 110 Img2015 Zarella et al. [34] NA 101 Pat2015 Gertych et al. [60] NA 210 Img2015 Di Franco et al. [115] [120], [211] 15 Img, 14 Pat, 9 Pat2015 Albashish et al. [116] NA 40 Pat, 149 Img2016 Leo et al. [71] NA 146 WSI2016 Noroozi and Zakerolhosseini [64] NA 33 Img2016 Fukuma et al. [47] NA 20 WSI2016 Bruno et al. [57] [212] 58 Img2016 Wang et al. [36] NA 68 Img2016 Niazi et al. [67] NA 15 WSI, 34 Img2016 Balazsi et al. [61] NA 66 Img2016 Wright et al. [121] Group [213] 157 Pat2016 Jothi and Rajam [99] NA 12 Pat, 219 Img,155 Img, 64 Img2016 Phoulady et al. [58] NA 39 Pat, 390 Img2016 Shi et al. [15] NA 47 WSI, 423 Img2016 Mazo et al. [11] [193] 200 Img2016 Brieu et al. [16] NA 90 Img2016 Fernández-Carrobles et al. [118] NA 170 WSI2017 Pang et al. [74] NA 96 WSI2017 Peikari et al. [91] NA 121 WSI, 64 Pat2017 BenTaieb et al. [92] NA 133 WSI2017 Zhang et al. [93] [214] 285 Img, 917 Img2017 Shi et al. [17] NA 200 Img2017 Shi et al. [18] [15] 47 WSI, 423 Img2017 Kwak and Hewitt [51] NA 771 Img2017 Romo-Bucheli et al. [119] NA 907 Img, 9 WSI2017 Brieu and Schmidt [38] NA 30 WSI2018 Peyret et al. [56] [215], [216], [217] 10 Img2019 Li et al. [162] ICPR2014 MITOSIS, NAAMIDA13, TUPAC162019 Gecer et al. [159] NIH-sponsored projects NA2019 Khan et al. [108] NA NA2019 Brancati et al. [144] D-IDC NA2019 Talo [145] Kimia Path24 NANA: Not available, Img: Images, Pat: Patients, WSI: Whole Slide Image.
8. Conclusion
In this paper, we have presented a review of the ML methods usually employed in analysis of HIs.This review revealed an increasing interest in the classification task, while the interest in other tasks suchas segmentation and feature extraction are in a clear declining in the last years, as shown in Tables 4 to 8,where the related works are arranged in ascending chronological order. We point out that the main reasonfor such a change is due to the introduction of DL methods, which are able to deal with raw HIs with alittle or even without any pre-processing step. Normalization is one of the most used preprocessing, but inthe early years other preprocessing methods such as thresholding, filtering, color models, had also been ournal Not Specified , xx , 5 32 of 45 used to improve the quality of HIs for subsequent tasks such as segmentation and feature extraction, oreven classification.In the years preceding the wide adoption of DL methods, several works had focused on identifyingnuclei in HIs, which are important structures to cancer diagnosis. Therefore, that lead to the exploitationof different segmentation approaches as reviewed in Section 3. Some works used the concept of semanticfeatures, based on the e.g. counting of nuclei, its relation to the stroma, the distance between nuclei. Stainnormalization is also a recurrent topic that has appeared in several works across the years covered by thisreview. Such an image processing method, which reduces the color and intensity variations present instained images, has been widely used even in conjunction with DL methods. Feature extraction methodswere the focus of interest of researchers between 2008 and 2016. Morphometric feature and textural featuressuch as GLCM, LBP and their variants have been the most frequent features used in HI analysis, eitheralone or in combination with other feature types. It is important to note that the shallow classifiers requirea feature extraction method. Again, the adoption of DL methods, which are able to learn representationand decision boundaries in a single optimization process, is probably the main cause of declining interestin feature extraction methods from 2016. Furthermore, pre-trained CNNs can also be used as featureextractors for HIs. Several works removed the fully connected layers of pre-trained CNNs and used theoutput of the last convolutional layer as feature vectors to feed shallow classifiers. Comparing Tables 7, 8and 9 we can say that DL approaches are becoming prevalent over shallow approaches in the last five years.Although studies are still necessary for understanding how these networks learn data representation,especially with respect to HIs.Finally, Tables 11 and 12 also help us to understand the increasing interest in HI analysis in the lastyears. We have found that most of the early works are based on small private datasets, what makesdifficult for other researchers that do not have access to such HI datasets to carry out research in this areaas well as to reproduce the scientific results. On the other hand, most of the recent works are based onpublic HI datasets, which are a great contribution to the science as they provide a way to researchers todevelop new methods and compare their performance with the existing ones. However, there is still a lackof large scale supervised WSI datasets.In conclusion, this review shows the evolution of HI analysis and the recent shift over DL methods.This review also provides valuable information to researchers in the field about datasets and other reviewsand surveys. Author Contributions:
Conceptualization, J.d.M., A.S.B.Jr., L.E.S.O. and A.L.K.; Methodology, J.d.M. and S.T.M.A.;Writing–original draft preparation, J.d.M. and S.T.M.A.; Writing–review and editing, A.L.K.; Supervision, A.S.B.Jr.,L.E.S.O. and A.L.K.; Funding acquisition, A.S.B.Jr. and A.L.K.; All authors have read and agreed to the publishedversion of the manuscript.
Funding:
This research was partially funded by Natural Sciences and Engineering Research Council of Canada(NSERC) Discovery grant number RGPIN-2016-04855 and by École de Technologie Supérieure, grant Développementde Collaborations Internationales de Recherche.
Conflicts of Interest:
The authors declare no conflict of interest.
Abbreviations
The following abbreviations are used in this manuscript: ournal Not Specified , xx , 5 33 of 45 AUC Area under the curveCAD Computer-aided diagnosisCNN Convolutional neural networkCT Computed tomographyDL Deep learningDNN Deep neural networkDT Decision treeELM Extreme learning machineGLCM Gray-level co-occurrence matrixHI Histophatologic imageH&E Hematoxylin and eosinHOG Histogram of oriented gradientsIHC Immunohistochemistry imagesImg ImagesLBP Local binary patternsML Machine learningMIL Multiple instance learningMLP Multilayer perceptronMRI Magnetic resonance imagingNSGA Non-dominated sorted genetic algorithmPat PatientsPCA Principal component analysisRCNN Recurrent convolutional neural networkRF Random forestROI Region of interestSHMM Spatial hidden Markov modelSIFT Scale-invariant feature transformSNN Synergistic neural networkSVM Support vector machineWSI Whole slide imageXCA Exclusive component analysis
References
1. Torre, L.A.; Bray, F.; Siegel, R.L.; Ferlay, J.; Lortet-Tieulent, J.; Jemal, A. Global cancer statistics.
CA CancerJournal for Clinicians , , 87–108. doi:10.3322/caac.21262.2. Torre, L.A.; Islami, F.; Siegel, R.L.; Ward, E.M.; Jemal, A. Global Cancer in Women: Burden and Trends. CEBPFocus: Global Cancer in Women , , 444–457. doi:10.1158/1055-9965.EPI-16-0858.3. Bellocq, J.P.; Anger, E.; Camparo, P.; Capron, F.; Chenard, M.P.; Chetritt, J.; Chigot, J.P.; Cochand-Priollet,B.; Coindre, J.M.; Copin, M.C.; Fléjou, J.F.; Galateau, F.; Gaulard, P.; Guiu, M.; Michiels, J.F.; Saint-André,J.P.; Scoazec, J.Y.; Vacher-Lavenu, M.C. Sécuriser le diagnostic en anatomie et cytologie pathologiquesen 2011. L’erreur diagnostique: entre discours et réalité. Annales de Pathologie , , S92–S94.doi:10.1016/j.annpat.2011.08.006.4. Aresta, G.; Araújo, T.; Kwok, S.; Chennamsetty, S.S.; Safwan, M.; Alex, V.; Marami, B.; Prastawa, M.; Chan, M.;Donovan, M.; Fernandez, G.; Zeineh, J.; Kohl, M.; Walz, C.; Ludwig, F.; Braunewell, S.; Baust, M.; Vu, Q.D.; To,M.N.N.; Kim, E.; Kwak, J.T.; Galal, S.; Sanchez-Freire, V.; Brancati, N.; Frucci, M.; Riccio, D.; Wang, Y.; Sun, L.;Ma, K.; Fang, J.; Kone, I.; Boulmane, L.; Campilho, A.; Eloy, C.; Polónia, A.; Aguiar, P. BACH: Grand challengeon breast cancer histology images. Medical Image Analysis , , 122 – 139. doi:10.1016/j.media.2019.05.010.5. Fatakdawala, H.; Xu, J.; Basavanhally, A.; Bhanot, G.; Ganesan, S.; Feldman, M.; Tomaszewski, J.E.; Madabhushi,A. Expectation-Maximization-Driven Geodesic Active Contour With Overlap Resolution (EMaGACOR): ournal Not Specified , xx , 5 34 of 45 Application to Lymphocyte Segmentation on Breast Cancer Histopathology.
IEEE Transactions on BiomedicalEngineering , , 1676–1689. doi:10.1109/TBME.2010.2041232.6. Roullier, V.; Lézoray, O.; Ta, V.T.; Elmoataz, A. Multi-resolution graph-based analysis of histopathologicalwhole slide images: Application to mitotic cell extraction and visualization. Computerized Medical Imaging andGraphics , , 603–615. doi:10.1016/j.compmedimag.2011.02.005.7. Rahmadwati.; Naghdy, G.; Ros, M.; Todd, C.; Norahmawati, E. Cervical Cancer Classification Using GaborFilters. IEEE 1st Intl Conf on Healthcare Informatics, Imaging and Systems Biology, 2011, pp. 48–52.doi:10.1109/HISB.2011.15.8. Peng, Y.; Jiang, Y.; Eisengart, L.; Healy, M.A.; Straus, F.H.; Yang, X.J. Segmentation of prostatic glands inhistology images. IEEE Intl Symp on Biomedical Imaging: From Nano to Macro, 2011, pp. 2091–2094.doi:10.1109/ISBI.2011.5872824.9. He, L.; Long, L.R.; Antani, S.; Thoma, G.R. Multiphase Level Set Model with Local K-means Energy forHistology Image Segmentation. IEEE 1st Intl Conf on Healthcare Informatics, Imaging and Systems Biology,2011, pp. 32–39. doi:10.1109/HISB.2011.35.10. Fatima, K.; Arooj, A.; Majeed, H. A New Texture and Shape Based Technique for Improving MeningiomaClassification. Microscopy Research and Technique , , 862–873. doi:10.1002/jemt.22409.11. Mazo, C.; Trujillo, M.; Alegre, E.; Salazar, L. Automatic recognition of fundamental tissues on histology imagesof the human cardiovascular system. Micron , , 1–8. doi:10.1016/j.micron.2016.07.001.12. Mazo, C.; Alegre, E.; Trujillo, M. Classification of cardiovascular tissues using LBP based descriptors and acascade SVM. Computer Methods and Programs in Biomedicine , , 1–10. doi:10.1016/j.cmpb.2017.06.003.13. Tosun, A.B.; Kandemir, M.; Sokmensuer, C.; Gunduz-Demir, C. Object-oriented texture analysis for theunsupervised segmentation of biopsy images for cancer detection. Pattern Recognition , , 1104–1112.doi:10.1016/j.patcog.2008.07.007.14. Nativ, N.I.; Chen, A.I.; Yarmush, G.; Henry, S.D.; Lefkowitch, J.H.; Klein, K.M.; Maguire, T.J.; Schloss,R.; Guarrera, J.V.; Berthiaume, F.; Yarmush, M.L. Automated image analysis method for detecting andquantifying macrovesicular steatosis in hematoxylin and eosin-stained histology images of human livers. LiverTransplantation , , 228–236. doi:10.1002/lt.23782.15. Shi, P.; Zhong, J.; Huang, R.; Lin, J. Automated Quantitative Image Analysis of Hematoxylin-Eosin StainingSlides in Lymphoma Based on Hierarchical Kmeans Clustering. 8th Intl Conf on Information Technology inMedicine and Education, 2016, pp. 99–104. doi:10.1109/ITME.2016.0031.16. Brieu, N.; Pauly, O.; Zimmermann, J.; Binnig, G.; Schmidt, G. Slide-Specific Models for Segmentation ofDifferently Stained Digital Histopathology Whole Slide Images. Medical Imaging 2016: Image Processing,2016, Vol. 9784, Proceedings of SPIE . doi:10.1117/12.2208620.17. Shi, P.; Chen, J.; Lin, J.; Zhang, L. High-throughput fat quantifications of hematoxylin-eosin stainedliver histopathological images based on pixel-wise clustering.
Science China Information Sciences , .doi:10.1007/s11432-016-9018-7.18. Shi, P.; Zhong, J.; Huang, R.; Lin, J. Automated quantitative image analysis of hematoxylin-eosin staining slidesin lymphoma based on hierarchical Kmeans clustering. 8th Intl Conf on Information Technology in Medicineand Education, 2017, pp. 99–104. doi:10.1109/ITME.2016.0031.19. Liu, B.; Liu, Y.; Zhang, J.; Zeng, Y.; Wang, W. Application of the synergetic algorithm on the classification oflymph tissue cells. Computers in Biology and Medicine , , 650–658. doi:10.1016/j.compbiomed.2008.03.003.20. Hafiane, A.; Bunyak, F.; Palaniappan, K. Evaluation of level set-based histology image segmentation usinggeometric region criteria. IEEE Intl Symp on Biomedical Imaging: From Nano to Macro, 2009, pp. 1–4.doi:10.1109/ISBI.2009.5192968.21. He, L.; Long, L.R.; Antani, S.; Thoma, G.R. Local and global Gaussian mixture models for hematoxylin andeosin stained histology image segmentation. 10th Intl Conf on Hybrid Intelligent Systems, 2010, pp. 223–228.doi:10.1109/HIS.2010.5600019.22. Onder, D.; Sarioglu, S.; Karacali, B. Automated labelling of cancer textures in colorectal histopathology slidesusing quasi-supervised learning. Micron , , 33–42. doi:10.1016/j.micron.2013.01.003. ournal Not Specified , xx , 5 35 of 45
23. Yang, L.; Qi, X.; Xing, F.; Kurc, T.; Saltz, J.; Foran, D.J. Parallel content-based sub-image retrieval usinghierarchical searching.
Bioinformatics , , 996–1002. doi:10.1093/bioinformatics/btt623.24. Sirinukunwattana, K.; Khan, A.M.; Rajpoot, N.M. Cell words: Modelling the visual appearanceof cells in histopathology images. Computerized Medical Imaging and Graphics , , 16–24.doi:10.1016/j.compmedimag.2014.11.008.25. Huang, C.H. Semi-supervised color decomposition for histopathological images using exclusivecomponent analysis. IEEE Intl Workshop on Machine Learning for Signal Processing, 2015.doi:10.1109/MLSP.2015.7324359.26. Yu, F.; Ip, H.H.S. Semantic content analysis and annotation of histological images. Computers in Biology andMedicine , , 635–649. doi:10.1016/j.compbiomed.2008.02.004.27. Arteta, C.; Lempitsky, V.; Noble, J.A.; Zisserman, A. Learning to Detect Cells Using Non-overlapping ExtremalRegions. Medical Image Computing and Computer-Assisted Intervention, 2012, Vol. 7510, Lecture Notes inComputer Science , pp. 348–356.28. Janssens, T.; Antanas, L.; Derde, S.; Vanhorebeek, I.; den Berghe, G.V.; Grandas, F.G. Charisma: An integratedapproach to automatic H&E-stained skeletal muscle cell segmentation using supervised learning and novelrobust clump splitting.
Medical Image Analysis , , 1206–1219. doi:10.1016/j.media.2013.07.007.29. Saraswat, M.; Arya, K.V. Supervised leukocyte segmentation in tissue images using multi-objective optimizationtechnique. Engineering Applications of Artificial Intelligence , , 44–52. doi:10.1016/j.engappai.2013.09.010.30. Qu, A.; Chen, J.; Wang, L.; Yuan, J.; Yang, F.; Xiang, Q.; Maskey, N.; Yang, G.; Liu, J.; Li, Y. Two-step segmentationof Hematoxylin-Eosin stained histopathological images for prognosis of breast cancer. IEEE Intl Conf onBioinformatics and Biomedicine, 2014, pp. 218–223. doi:10.1109/BIBM.2014.6999158.31. Salman, S.; Ma, Z.; Mohanty, S.; Bhele, S.; Chu, Y.T.; Knudsen, B.; Gertych, A. A machine learning approach toidentify prostate cancer areas in complex histological images. Advances in Intelligent Systems and Computing , , 295–306. doi:10.1007/978-3-319-06593-9_26.32. Chen, J.M.; Qu, A.P.; Wang, L.W.; Yuan, J.P.; Yang, F.; Xiang, Q.M.; Maskey, N.; Yang, G.F.; Liu, J.; Li, Y. Newbreast cancer prognostic factors identified by computer-aided image analysis of HE stained histopathologyimages. Scientific Reports , . doi:10.1038/srep10690.33. Geessink, O.G.F.; Baidoshvili, A.; Freling, G.; Klaase, J.M.; Slump, C.H.; Van Der Heijden, F. Toward automaticsegmentation and quantification of tumor and stroma in whole-slide images of H&E stained rectal carcinomas.Progress in Biomedical Optics and Imaging - Proceedings of SPIE, 2015, Vol. 9420. doi:10.1117/12.2081665.34. Zarella, M.D.; Breen, D.E.; Reza, M.A.; Milutinovic, A.; Garcia, F.U. Lymph Node Metastasis Status in BreastCarcinoma Can Be Predicted via image Analysis of Tumor Histology. Analytical and Quantitative Cytopathologyand Histopathology , , 273–285.35. Santamaria-Pang, A.; Rittscher, J.; Gerdes, M.; Padfield, D. Cell Segmentation and Classification by HierarchicalSupervised Shape Ranking. IEEE 12th Intl Symp on Biomedical Imaging, 2015, pp. 1296–1299.36. Wang, P.; Hu, X.; Li, Y.; Liu, Q.; Zhu, X. Automatic cell nuclei segmentation and classification of breast cancerhistopathology images. Signal Processing , , 1–13. doi:10.1016/j.sigpro.2015.11.011.37. Arteta, C.; Lempitsky, V.; Noble, J.A.; Zisserman, A. Detecting overlapping instances in microscopy imagesusing extremal region trees. Medical Image Analysis , , 3–16. doi:10.1016/j.media.2015.03.002.38. Brieu, N.; Schmidt, G. Learning Size Adaptive Local Maxima Selection for Robust Nuclei Detection inHistopathology Images. IEEE 14th Intl Symp on Biomedical Imaging, 2017, pp. 937–941.39. Song, J.; Xiao, L.; Molaei, M.; Lian, Z. Multi-layer boosting sparse convolutional model for generalized nuclearsegmentation from histopathology images. Knowledge-Based Systems , , 40–53.40. Ballarò, B.; Florena, A.M.; Franco, V.; Tegolo, D.; Tripodo, C.; Valenti, C. An automated image analysismethodology for classifying megakaryocytes in chronic myeloproliferative disorders. Medical Image Analysis , , 703–712. doi:10.1016/j.media.2008.04.001.41. Petushi, S.; Zhang, J.; Milutinovic, A.; Breen, D.E.; Garcia, F.U. Image-based histologic grade estimation usingstochastic geometry analysis. Progress in Biomedical Optics and Imaging - Proceedings of SPIE, 2011, Vol. 7963.doi:10.1117/12.876346. ournal Not Specified , xx , 5 36 of 45
42. Madabhushi, A.; Agner, S.; Basavanhally, A.; Doyle, S.; Lee, G. Computer-aided prognosis: Predicting patientand disease outcome via quantitative fusion of multi-scale, multi-modal data.
Computerized Medical Imagingand Graphics , , 506–514. doi:10.1016/j.compmedimag.2011.01.008.43. Song, J.W.; Lee, J.H.; Choi, J.H.; Chun, S.J. Automatic differential diagnosis of pancreatic serous andmucinous cystadenomas based on morphological features. Computers in Biology and Medicine , , 1–15.doi:10.1016/j.compbiomed.2012.10.009.44. Gorelick, L.; Veksler, O.; Gaed, M.; Gomez, J.A.; Moussa, M.; Bauman, G.; Fenster, A.; Ward, A.D. Prostatehistopathology: Learning tissue component histograms for cancer detection and classification. IEEE Transactionson Medical Imaging , , 1804–1818. doi:10.1109/TMI.2013.2265334.45. Filipczuk, P.; Krawczyk, B.; Wo´zniak, M. Classifier ensemble for an effective cytological image analysis. PatternRecognition Letters , , 1748–1757. doi:10.1016/j.patrec.2013.04.030.46. Ozolek, J.A.; Tosun, A.B.; Wang, W.; Chen, C.; Kolouri, S.; Basu, S.; Huang, H.; Rohde, G.K. Accurate diagnosisof thyroid follicular lesions from nuclear morphology using supervised learning. Medical Image Analysis , , 772–780. doi:10.1016/j.media.2014.04.004.47. Fukuma, K.; Prasath, V.B.S.; Kawanaka, H.; Aronow, B.J.; Takase, H. A Study on Nuclei Segmentation, FeatureExtraction and Disease Stage Classification for Human Brain Histopathological Images. Procedia ComputerScience , , 1202–1210. doi:10.1016/j.procs.2016.08.164.48. Loeffler, M.; Greulich, L.; Scheibe, P.; Kahl, P.; Shaikhibrahim, Z.; Braumann, U.D.; Kuska, J.P.; Wernert, N.Classifying Prostate Cancer Malignancy by Quantitative Histomorphometry. The Journal of Urology , , 1867–1875. doi:10.1016/j.juro.2011.12.054.49. Marugame, A.; Kiyuna, T.; Ogura, M.; Saito, A. Categorization of HE stained breast tissue samplesat low magnification by nuclear aggregations. IFMBE Proceedings, 2009, Vol. 25, pp. 173–176.doi:10.1007/978-3-642-03882-2-45.50. Osborne, J.D.; Gao, S.; Chen, W.b.; Andea, A.; Zhang, C. Machine Classification of Melanoma and Nevi fromSkin Lesions. ACM Symp on Applied Computing; , 2011; pp. 100–105. doi:10.1145/1982185.1982210.51. Kwak, J.T.; Hewitt, S.M. Multiview boosting digital pathology analysis of prostate cancer. Computer Methodsand Programs in Biomedicine , , 91–99. doi:10.1016/j.cmpb.2017.02.023.52. Olgun, G.; Sokmensuer, C.; Gunduz-Demir, C. Local object patterns for the representation andclassification of colon tissue images. IEEE Journal of Biomedical and Health Informatics , , 1390–1396.doi:10.1109/JBHI.2013.2281335.53. Kuse, M.; Sharma, T.; Gupta, S. A classification scheme for lymphocyte segmentation in H&E stained histologyimages. Lecture Notes in Computer Science , , 235–243. doi:10.1007/978-3-642-17711-8_24.54. Caicedo, J.C.; González, F.A.; Romero, E. Content-based histopathology image retrieval using a kernel-basedsemantic annotation framework. Journal of Biomedical Informatics , , 519–528. doi:10.1016/j.jbi.2011.01.011.55. Fernández-Carrobles, M.M.; Bueno, G.; Déniz, O.; Salido, J.; García-Rojo, M.; Gonzández-López, L. Frequentialversus spatial colour textons for breast TMA classification. Computerized Medical Imaging and Graphics , , 25–37. doi:10.1016/j.compmedimag.2014.11.009.56. Peyret, R.; Bouridane, A.; Khelifi, F.; Tahir, M.A.; Al-Maadeed, S. Automatic classification of colorectal andprostatic histologic tumor images using multiscale multispectral local binary pattern texture features andstacked generalization. Neurocomputing , , 83–93. doi:10.1016/j.neucom.2017.05.010.57. Bruno, D.O.T.; do Nascimento, M.Z.; Ramos, R.P.; Batista, V.R.; Neves, L.A.; Martins, A.S. LBP operators oncurvelet coefficients as an algorithm to describe texture in breast cancer tissues. Expert Systems with Applications , , 329–340. doi:10.1016/j.eswa.2016.02.019.58. Phoulady, H.A.; Zhou, M.; Goldgof, D.B.; Hall, L.O.; Mouton, P.R. Automatic quantification and classificationof cervical cancer via Adaptive Nucleus Shape Modeling. IEEE Intl Conf on Image Processing, 2016, pp.2658–2662. doi:10.1109/ICIP.2016.7532841.59. Reis, S.; Gazinska, P.; Hipwell, J.H.; Mertzanidou, T.; Naidoo, K.; Williams, N.; Pinder, S.; Hawkes, D.J.Automated Classification of Breast Cancer Stroma Maturity from Histological Images. IEEE Transactions onBiomedical Engineering , , 2344–2352. doi:10.1109/TBME.2017.2665602. ournal Not Specified , xx , 5 37 of 45
60. Gertych, A.; Ing, N.; Ma, Z.; Fuchs, T.J.; Salman, S.; Mohanty, S.; Bhele, S.; Velásquez-Vacca, A.;Amin, M.B.; Knudsen, B.S. Machine learning approaches to analyze histological images of tissuesfrom radical prostatectomies.
Computerized Medical Imaging and Graphics ,
46, Part 2 , 197–208.doi:10.1016/j.compmedimag.2015.08.002.61. Balazsi, M.; Blanco, P.; Zoroquiain, P.; Levine, M.D.; Burnier Jr., M.N. Invasive ductal breast carcinomadetector that is robust to image magnification in whole digital slides.
Journal of Medical Imaging , .doi:10.1117/1.JMI.3.2.027501.62. Atupelage, C.; Nagahashi, H.; Yamaguchi, M.; Abe, T.; Hashiguchi, A.; Sakamoto, M. Computational gradingof hepatocellular carcinoma using multifractal feature description. Computerized Medical Imaging and Graphics , , 61–71. doi:10.1016/j.compmedimag.2012.10.001.63. Huang, C.H.; Veillard, A.; Roux, L.; Loménie, N.; Racoceanu, D. Time-efficient sparse analysis ofhistopathological whole slide images. Computerized Medical Imaging and Graphics , , 579–591.doi:10.1016/j.compmedimag.2010.11.009.64. Noroozi, N.; Zakerolhosseini, A. Computer assisted diagnosis of basal cell carcinoma using Z-transformfeatures. Journal of Visual Communication and Image Representation ,
40, Part A , 128–148.doi:10.1016/j.jvcir.2016.06.014.65. Wan, T.; Zhang, W.; Zhu, M.; Chen, J.; Achim, A.; Qin, Z. Automated mitosis detection in histopathologybased on non-gaussian modeling of complex wavelet coefficients.
Neurocomputing , , 291–303.doi:10.1016/j.neucom.2017.01.008.66. Chan, A.; Tuszynski, J.A. Automatic prediction of tumour malignancy in breast cancer with fractal dimension. Royal Society Open Science , . doi:10.1098/rsos.160558.67. Niazi, M.K.K.; Parwani, A.V.; Gurcan, M.N. Computer-Assisted bladder cancer grading: α -shapes for colorspace decomposition. Progress in Biomedical Optics and Imaging - Proceedings of SPIE, 2016, Vol. 9791.doi:10.1117/12.2216967.68. Spanhol, F.A.; Oliveira, L.S.; Cavalin, P.R.; Petitjean, C.; Heutte, L. Deep features for breast cancerhistopathological image classification. 2017 IEEE International Conference on Systems, Man, and Cybernetics,2017, pp. 1868–1873.69. Vo, D.M.; Nguyen, N.Q.; Lee, S.W. Classification of breast cancer histology images using incremental boostingconvolution networks. Information Sciences , , 123–138.70. George, K.; Faziludeen, S.; Sankaran, P.; Paul, J.K. Deep Learned Nucleus Features for Breast CancerHistopathological Image Analysis based on Belief Theoretical Classifier Fusion. IEEE Region 10 Conf(TENCON), 2019, pp. 344–349.71. Leo, P.; Lee, G.; Shih, N.N.C.; Elliott, R.; Feldman, M.D.; Madabhushi, A. Evaluating stability ofhistomorphometric features across scanner and staining variations: prostate cancer diagnosis from whole slideimages. Journal of Medical Imaging , . doi:10.1117/1.JMI.3.4.047502.72. Yu, K.H.; Zhang, C.; Berry, G.J.; Altman, R.B.; Re, C.; Rubin, D.L.; Snyder, M. Predicting non-small cell lungcancer prognosis by fully automated microscopic pathology image features. Nature Communications , .doi:10.1038/ncomms12474.73. Caicedo, J.C.; Gonzalez, F.A.; Romero, E. A semantic content-based retrieval method for histopathology images.Information Retrieval Technology, 2008, Vol. 4993, Lecture Notes in Computer Science , pp. 51–60.74. Pang, W.; Jiang, H.; Li, S. Sparse Contribution Feature Selection and Classifiers Optimized by Concave-ConvexVariation for HCC Image Recognition.
BioMed Research Intl , . doi:10.1155/2017/9718386.75. Kruk, M.; Kurek, J.; Osowski, S.; Koktysz, R.; Swiderski, B.; Markiewicz, T. Ensemble of classifiers and wavelettransformation for improved recognition of Fuhrman grading in clear-cell renal carcinoma. Biocybernetics andBiomedical Engineering , , 357–364. doi:10.1016/j.bbe.2017.04.005.76. Basavanhally, A.; Ganesan, S.; Feldman, M.; Shih, N.; Mies, C.; Tomaszewski, J.; Madabhushi,A. Multi-Field-of-View Framework for Distinguishing Tumor Grade in ER+ Breast Cancer FromEntire Histopathology Slides. IEEE Transactions on Biomedical Engineering , , 2089–2099.doi:10.1109/TBME.2013.2245129. ournal Not Specified , xx , 5 38 of 45
77. Tashk, A.; Helfroush, M.S.; Danyali, H.; Akbarzadeh-jahromi, M. Automatic detection of breast cancer mitoticcells based on the combination of textural, statistical and innovative mathematical features.
Applied MathematicalModelling , , 6165–6182. doi:10.1016/j.apm.2015.01.051.78. Cruz-Roa, A.; Caicedo, J.C.; González, F.A. Visual pattern mining in histology image collections using bag offeatures. Artificial Intelligence in Medicine , , 91–106. doi:10.1016/j.artmed.2011.04.010.79. Orlov, N.V.; Chen, W.W.; Eckley, D.M.; Macura, T.J.; Shamir, L.; Jaffe, E.S.; Goldberg, I.G. AutomaticClassification of Lymphoma Images With Transform-Based Global Features. IEEE Transactions on InformationTechnology in Biomedicine , , 1003–1013. doi:10.1109/TITB.2010.2050695.80. De, S.; Stanley, R.J.; Lu, C.; Long, R.; Antani, S.; Thoma, G.; Zuna, R. A fusion-based approach for uterinecervical cancer histology image classification. Computerized Medical Imaging and Graphics , , 475–487.doi:10.1016/j.compmedimag.2013.08.001.81. Vanderbeck, S.; Bockhorst, J.; Komorowski, R.; Kleiner, D.E.; Gawrieh, S. Automatic classification ofwhite regions in liver biopsies by supervised machine learning. Human Pathology , , 785–792.doi:10.1016/j.humpath.2013.11.011.82. Kandemir, M.; Feuchtinger, A.; Walch, A.; Hamprecht, F.A. Digital pathology: Multiple instancelearning can detect Barrett’s cancer. IEEE 11th Intl Symp on Biomedical Imaging, 2014, pp. 1348–1351.doi:10.1109/ISBI.2014.6868127.83. Coatelen, J.; Albouy-Kissi, A.; Albouy-Kissi, B.; Coton, J.P.; Sifre, L.; Joubert-Zakeyh, J.; Dechelotte, P.; Abergel,A. A feature selection based framework for histology image classification using global and local heterogeneityquantification. 36th Annual Intl Conf of the IEEE Engineering in Medicine and Biology Society, 2014, pp.1937–1940. doi:10.1109/EMBC.2014.6943991.84. Coatelen, J.; Albouy-Kissi, A.; Albouy-Kissi, B.; Coton, J.P.; Maunier-Sifre, L.; Joubert-Zakeyh, J.; Dechelotte,P.; Abergel, A. A subset-search and ranking based feature-selection for histology image classification usingglobal and local quantification. Intl Conf on Image Processing Theory, Tools and Applications (IPTA), 2015, pp.313–318. doi:10.1109/IPTA.2015.7367154.85. Michail, E.; Dimitropoulos, K.; Koletsa, T.; Kostopoulos, I.; Grammalidis, N. Morphological and texturalanalysis of centroblasts in low-thickness sliced tissue biopsies of follicular lymphoma. Annual Intl Conf of theIEEE Engineering in Medicine and Biology Society, 2014, pp. 3374–3377.86. Das, D.K.; Mitra, P.; Chakraborty, C.; Chatterjee, S.; Maiti, A.K.; Bose, S. Computational approach for mitotic celldetection and its application in oral squamous cell carcinoma. Multidimensional Systems and Signal Processing , , 1031–1050. doi:10.1007/s11045-017-0488-6.87. Kong, J.; Sertel, O.; Shimada, H.; Boyer, K.L.; Saltz, J.H.; Gurcan, M.N. Computer-aided evaluation ofneuroblastoma on whole-slide histology images: Classifying grade of neuroblastic differentiation. PatternRecognition , , 1080–1092. doi:10.1016/j.patcog.2008.10.035.88. Malon, C.; Brachtel, E.; Cosatto, E.; Graf, H.P.; Kurata, A.; Kuroda, M.; Meyer, J.S.; Saito, A.; Wu, S.; Yagi, Y.Mitotic figure recognition: Agreement among pathologists and computerized detector. Analytical CellularPathology , , 97–100. doi:10.3233/ACP-2011-0029.89. Guo, P.; Banerjee, K.; Stanley, R.J.; Long, R.; Antani, S.; Thoma, G.; Zuna, R.; Frazier, S.R.; Moss,R.H.; Stoecker, W.V. Nuclei-Based Features for Uterine Cervical Cancer Histology Image AnalysisWith Fusion-Based Classification. IEEE Journal of Biomedical and Health Informatics , , 1595–1607.doi:10.1109/JBHI.2015.2483318.90. Harai, Y.; Tanaka, T. Automatic Diagnosis Support System Using Nuclear and Luminal Features. Intl Conf onDigital Image Computing: Techniques and Applications, 2015, pp. 1–8. doi:10.1109/DICTA.2015.7371235.91. Peikari, M.; Salama, S.; Nofech-Mozes, S.; Martel, A.L. Automatic cellularity assessment from post-treatedbreast surgical specimens. Cytometry Part A , , 1078–1087. doi:10.1002/cyto.a.23244.92. BenTaieb, A.; Li-Chang, H.; Huntsman, D.; Hamarneh, G. A structured latent model forovarian carcinoma subtyping from histopathology slides. Medical Image Analysis , , 194–205.doi:10.1016/j.media.2017.04.008. ournal Not Specified , xx , 5 39 of 45
93. Zhang, R.; Shen, J.; Wei, F.; Li, X.; Sangaiah, A.K. Medical image classification based on multi-scale non-negativesparse coding.
Artificial Intelligence in Medicine , , 44–51. doi:10.1016/j.artmed.2017.05.006.94. Korkmaz, S.A.; Poyraz, M. Least Square Support Vector Machine and Minumum Redundacy MaximumRelavance for Diagnosis of Breast Cancer from Breast Microscopic Images. Procedia - Social and BehavioralSciences , , 4026–4031. doi:10.1016/j.sbspro.2015.01.1150.95. Mete, M.; Topaloglu, U. Statistical comparison of color model-classifier pairs in hematoxylin and eosin stainedhistological images. IEEE Symp on Computational Intelligence in Bioinformatics and Computational Biology,2009, pp. 284–291. doi:10.1109/CIBCB.2009.4925740.96. Sidiropoulos, K.; Glotsos, D.; Kostopoulos, S.; Ravazoula, P.; Kalatzis, I.; Cavouras, D.; Stonham, J. Realtime decision support system for diagnosis of rare cancers, trained in parallel, on a graphics processing unit. Computers in Biology and Medicine , , 376–386. doi:10.1016/j.compbiomed.2011.12.004.97. Michail, E.; Kornaropoulos, E.N.; Dimitropoulos, K.; Grammalidis, N.; Koletsa, T.; Kostopoulos, I. Detection ofcentroblasts in H&E stained images of follicular lymphoma. 22nd Signal Processing and CommunicationsApplications Conf, 2014, pp. 2319–2322. doi:10.1109/SIU.2014.6830728.98. Beevi, S.K.; Nair, M.S.; Bindu, G.R. Detection of Mitotic Nuclei in Breast Histopathology Images using LocalizedACM and Random Kitchen Sink based Classifier. 38th Annual Intl Conf of the IEEE Engineering in Medicineand Biology Society, 2016, pp. 2435–2439.99. Jothi, J.A.A.; Rajam, V.M.A. Effective segmentation and classification of thyroid histopathology images. AppliedSoft Computing , , 652–664. doi:10.1016/j.asoc.2016.02.030.100. Awan, R.; Aloraidi, N.; Qidwai, U.; Rajpoot, N. How divided is a cell? Eigenphase nuclei for classification ofmitotic phase in cancer histology images. IEEE-EMBS Intl Conf on Biomedical and Health Informatics, 2016,pp. 70–73. doi:10.1109/BHI.2016.7455837.101. Barker, J.; Hoogi, A.; Depeursinge, A.; Rubin, D.L. Automated classification of brain tumor type inwhole-slide digital pathology images using local representative tiles. Medical Image Analysis , , 60–71.doi:10.1016/j.media.2015.12.002.102. Kandemir, M.; Hamprecht, F.A. Computer-aided diagnosis from weak supervision: A benchmarking study. Computerized Medical Imaging and Graphics , , 44–50. doi:10.1016/j.compmedimag.2014.11.010.103. Cosatto, E.; Laquerre, P.F.; Malon, C.; Graf, H.P.; Saito, A.; Kiyuna, T.; Marugame, A.; Kamijo, K. Automatedgastric cancer diagnosis on H&E-stained sections; ltraining a classifier on a large scale with multiple instancemachine learning. Progress in Biomedical Optics and Imaging - Proceedings of SPIE, 2013, Vol. 8676.doi:10.1117/12.2007047.104. Xu, Y.; Zhu, J.Y.; Chang, E.I.C.; Lai, M.; Tu, Z. Weakly supervised histopathology cancer image segmentationand classification. Medical Image Analysis , , 591–604. doi:10.1016/j.media.2014.01.010.105. Sudharshan, P.; Petitjean, C.; Spanhol, F.; Oliveira, L.E.; Heutte, L.; Honeine, P. Multiple instance learning forhistopathological breast cancer image classification. Expert Systems with Applications , , 103–111.106. Irshad, H.; Gouaillard, A.; Roux, L.; Racoceanu, D. Multispectral band selection and spatial characterization:Application to mitosis detection in breast cancer histopathology. Computerized Medical Imaging and Graphics , , 390–402. doi:10.1016/j.compmedimag.2014.04.003.107. Homeyer, A.; Schenk, A.; Arlt, J.; Dahmen, U.; Dirsch, O.; Hahn, H.K. Practical quantification ofnecrosis in histological whole-slide images. Computerized Medical Imaging and Graphics , , 313–322.doi:10.1016/j.compmedimag.2013.05.002.108. Khan, S.U.; Islam, N.; Jan, Z.; Din, I.U.; Khan, A.; Faheem, Y. An e-health care services framework for thedetection and classification of breast cancer in breast cytology images as an IoMT application. Future GenerationComputer Systems , , 286–296.109. Kurmi, Y.; Chaurasia, V.; Ganesh, N.; Kesharwani, A. Microscopic images classification for cancer diagnosis. Signal, Image and Video Processing , pp. 1–9.110. Daskalakis, A.; Kostopoulos, S.; Spyridonos, P.; Glotsos, D.; Ravazoula, P.; Kardari, M.; Kalatzis, I.; Cavouras,D.; Nikiforidis, G. Design of a multi-classifier system for discriminating benign from malignant thyroid ournal Not Specified , xx , 5 40 of 45 nodules using routinely H&E-stained cytological images. Computers in Biology and Medicine , , 196–203.doi:10.1016/j.compbiomed.2007.09.005.111. Meng, T.; Lin, L.; Shyu, M.L.; Chen, S.C. Histology Image Classification Using Supervised Classification andMultimodal Fusion. IEEE Intl Symp on Multimedia, 2010, pp. 145–152. doi:10.1109/ISM.2010.29.112. Wang, C.W.; Yu, C.P. Automated morphological classification of lung cancer subtypes using H&E tissue images. Machine Vision and Applications , , 1383–1391. doi:10.1007/s00138-012-0457-x.113. Vink, J.P.; Van Leeuwen, M.B.; Van Deurzen, C.H.M.; De Haan, G. Efficient nucleus detector in histopathologyimages. Journal of Microscopy , , 124–135. doi:10.1111/jmi.12001.114. Phoulady, H.A.; Chaudhury, B.; Goldgof, D.; Hall, L.O.; Mouton, P.R.; Hakam, A.; Siegel, E.M. Experimentswith large ensembles for segmentation and classification of cervical cancer biopsy images. IEEE Intl Conf onSystems, Man, and Cybernetics, 2014, pp. 870–875. doi:10.1109/SMC.2014.6974021.115. Di Franco, M.D.; Reynolds, H.L.M.; Mitchell, C.; Williams, S.; Allan, P.; Haworth, A. Performance assessmentof automated tissue characterization for prostate H&E stained histopathology. Medical Imaging 2015: DigitalPathology, 2015, Vol. 9420, Proceedings of SPIE . doi:10.1117/12.2081787.116. Albashish, D.; Sahran, S.; Abdullah, A.; Adam, A.; Abd Shukor, N.; Pauzi, S.H.M. Multi-scoring Featureselection method based on SVM-RFE for prostate cancer diagnosis. 5th Intl Conf on Electrical Engineering andInformatics, 2015, pp. 682–686.117. Huang, C.H.; Kalaw, E.M. Automated classification for pathological prostate images usingAdaBoost-based Ensemble Learning. IEEE Symp Series on Computational Intelligence, 2016, pp. 1–4.doi:10.1109/SSCI.2016.7849887.118. Fernández-Carrobles, M.M.; Serrano, I.; Bueno, G.; Déniz, O. Bagging Tree Classifier and TextureFeatures for Tumor Identification in Histological Images.
Procedia Computer Science , , 99–106.doi:10.1016/j.procs.2016.07.030.119. Romo-Bucheli, D.; Corredor, G.; Garcia-Arteaga, J.D.; Arias, V.; Romero, E. Nuclei Graph Local Features forBasal Cell Carcinoma Classification in Whole Slide Images. 12th Intl Symp on Medical Information Processingand Analysis, 2017, Vol. 10160, Proceedings of SPIE . doi:10.1117/12.2257386.120. DiFranco, M.D.; O’Hurley, G.; Kay, E.W.; Watson, R.W.G.; Cunningham, P. Ensemble based system forwhole-slide prostate cancer probability mapping using color texture features.
Computerized Medical Imaging andGraphics , , 629–645. doi:10.1016/j.compmedimag.2010.12.005.121. Wright, A.I.; Magee, D.; Quirke, P.; Treanor, D. Incorporating Local and Global Context for Better AutomatedAnalysis of Colorectal Cancer on Digital Pathology Slides. Procedia Computer Science , , 125–131.doi:10.1016/j.procs.2016.07.034.122. Valkonen, M.; Kartasalo, K.; Liimatainen, K.; Nykter, M.; Latonen, L.; Ruusuvuori, P. Metastasis detectionfrom whole slide images using local features and random forests. Cytometry Part A , , 555–565.doi:10.1002/cyto.a.23089.123. Cruz-Roa, A.; Basavanhally, A.; Gonzalez, F.; Gilmore, H.; Feldman, M.; Ganesan, S.; Shih, N.; Tomaszewski, J.;Madabhushi, A. Automatic detection of invasive ductal carcinoma in whole slide images with ConvolutionalNeural Networks. Medical Imaging 2014: Digital Pathology, 2014, Vol. 9041, Proceedings of SPIE .doi:10.1117/12.2043872.124. de Matos, J.; de Souza Britto Jr., A.; de Oliveira, L.E.S.; Koerich, A.L. Texture CNN for Histopathological ImageClassification. 32nd IEEE Intl Symp on Computer-Based Medical Systems (CBMS), Cordoba, Spain, 2019, pp.580–583. doi:10.1109/CBMS.2019.00120.125. Ataky, S.T.M.; de Matos, J.; de Souza Britto Jr., A.; de Oliveira, L.E.S.; Koerich, A.L. Data Augmentation forHistopathological Images Based on Gaussian-Laplacian Pyramid Blending. Intl Joint Conf on Neural Networks,Glasgow, UK, 2020, pp. 1–8.126. de Matos, J.; de Souza Britto Jr., A.; Oliveira, L.E.S.; Koerich, A.L. Double Transfer Learning for Breast CancerHistopathologic Image Classification. Intl Joint Conf on Neural Networks, Budapest, Hungary, 2019, pp. 1–8.doi:10.1109/IJCNN.2019.8852092. ournal Not Specified , xx , 5 41 of 45 Proceedings of the IEEE , , 2278–2324. doi:10.1109/5.726791.128. Kainz, P.; Pfeiffer, M.; Urschler, M. Segmentation and classification of colon glands with deep convolutionalneural networks and total variation regularization. PeerJ , . doi:10.7717/peerj.3874.129. Krizhevsky, A.; Sutskever, I.; Hinton, G.E. ImageNet Classification with Deep Convolutional Neural Networks.In Advances in Neural Information Processing Systems 25 ; Curran Associates, Inc., 2012; pp. 1097–1105.130. Stanitsas, P.; Cherian, A.; Li, X.; Truskinovsky, A.; Morellas, V.; Papanikolopoulos, N. Evaluation of featuredescriptors for cancerous tissue recognition. 23rd Intl Conf on Pattern Recognition, 2016, pp. 1490–1495.doi:10.1109/ICPR.2016.7899848.131. Spanhol, F.A.; Oliveira, L.S.; Petitjean, C.; Heutte, L. Breast cancer histopathological image classificationusing Convolutional Neural Networks. Intl Joint Conf on Neural Networks, 2016, pp. 2560–2567.doi:10.1109/IJCNN.2016.7727519.132. Sharma, H.; Zerbe, N.; Klempert, I.; Hellwich, O.; Hufnagl, P. Deep convolutional neural networks forautomatic classification of gastric carcinoma using whole slide images in digital histopathology.
ComputerizedMedical Imaging and Graphics , , 2–13. doi:10.1016/j.compmedimag.2017.06.001.133. Budak, Ü.; Cömert, Z.; Rashid, Z.N.; ¸Sengür, A.; Çıbuk, M. Computer-aided diagnosis system combining FCNand Bi-LSTM model for efficient breast cancer detection from histopathological images. Applied Soft Computing , , 105765.134. Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.E.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; Rabinovich, A.Going deeper with convolutions. IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2015,pp. 1–9. doi:10.1109/CVPR.2015.7298594.135. Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; Wojna, Z. Rethinking the Inception Architecture for ComputerVision. CoRR , abs/1512.00567 , [1512.00567].136. Szegedy, C.; Ioffe, S.; Vanhoucke, V.; Alemi, A.A. Inception-v4, Inception-ResNet and the Impact of ResidualConnections on Learning. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence. AAAIPress, 2017, pp. 4278–4284.137. Li, W.; Manivannan, S.; Akbar, S.; Zhang, J.; Trucco, E.; McKenna, S.J. Gland segmentation in colon histologyimages using hand-crafted features and convolutional neural networks. IEEE 13th Intl Symp on BiomedicalImaging, 2016, pp. 1405–1408. doi:10.1109/ISBI.2016.7493530.138. Yan, R.; Ren, F.; Wang, Z.; Wang, L.; Zhang, T.; Liu, Y.; Rao, X.; Zheng, C.; Zhang, F. Breast cancerhistopathological image classification using a hybrid deep neural network. Methods .139. He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. 2016 IEEE Conference onComputer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. IEEE ComputerSociety, 2016, pp. 770–778. doi:10.1109/CVPR.2016.90.140. Khosravi, P.; Kazemi, E.; Imielinski, M.; Elemento, O.; Hajirasouliha, I. Deep Convolutional Neural NetworksEnable Discrimination of Heterogeneous Digital Pathology Images. {EBioMedicine} , , 317–328.doi:10.1016/j.ebiom.2017.12.026.141. Vizcarra, J.; Place, R.; Tong, L.; Gutman, D.; Wang, M.D. Fusion In Breast Cancer Histology Classification. 10thACM Intl Conf on Bioinformatics, Computational Biology and Health Informatics, 2019, pp. 485–493.142. Zerhouni, E.; Lányi, D.; Viana, M.; Gabrani, M. Wide residual networks for mitosis detection. IEEE 14th IntlSymp on Biomedical Imaging, 2017, pp. 924–928. doi:10.1109/ISBI.2017.7950667.143. Gandomkar, Z.; Brennan, P.C.; Mello-Thoms, C. MuDeRN: Multi-category classification of breasthistopathological image using deep residual networks. Artificial intelligence in medicine , , 14–24.144. Brancati, N.; De Pietro, G.; Frucci, M.; Riccio, D. A Deep Learning Approach for Breast Invasive DuctalCarcinoma Detection and Lymphoma Multi-Classification in Histological Images. IEEE Access , , 44709–44720.145. Talo, M. Automated classification of histopathology images using transfer learning. Artificial Intelligence inMedicine , , 101743. ournal Not Specified , xx , 5 42 of 45 IEEE Transactions on Biomedical Engineering , , 2901–2912.doi:10.1109/TBME.2017.2686418.149. Kumar, A.; Singh, S.K.; Saxena, S.; Lakshmanan, K.; Sangaiah, A.K.; Chauhan, H.; Shrivastava, S.; Singh, R.K.Deep feature learning for histopathological image classification of canine mammary tumors and human breastcancer. Information Sciences , , 405–421.150. Huang, G.; Liu, Z.; van der Maaten, L.; Weinberger, K.Q. Densely Connected ConvolutionalNetworks. 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 2261–2269.doi:10.1109/CVPR.2017.243.151. Howard, A.G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; Weyand, T.; Andreetto, M.; Adam,H. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. CoRR , abs/1704.04861 .152. Kassani, S.H.; Kassani, P.H.; Wesolowski, M.J.; Schneider, K.A.; Deters, R. Classification of HistopathologicalBiopsy Images Using Ensemble of Deep Learning Networks. Proceedings of the 29th Annual InternationalConference on Computer Science and Software Engineering; , 2019; CASCON ’19, p. 92–99.153. Yang, H.; Kim, J.Y.; Kim, H.; Adhikari, S.P. Guided Soft Attention Network for Classification of Breast CancerHistopathology Images. IEEE transactions on medical imaging .154. Bayramoglu, N.; Kannala, J.; Heikkila, J. Deep Learning for Magnification Independent Breast CancerHistopathology Image Classification. 23rd Intl Conf on Pattern Recognition, 2016, Intl Conf on PatternRecognition, pp. 2440–2445.155. Albarqouni, S.; Baur, C.; Achilles, F.; Belagiannis, V.; Demirci, S.; Navab, N. AggNet: Deep Learning FromCrowds for Mitosis Detection in Breast Cancer Histology Images.
IEEE Transactions on Medical Imaging , , 1313–1321. doi:10.1109/TMI.2016.2528120.156. Ciompi, F.; Geessink, O.; Bejnordi, B.E.; de Souza, G.S.; Baidoshvili, A.; Litjens, G.; van Ginneken, B.; Nagtegaal,I.; van der Laak, J. The importance of stain normalization in colorectal tissue classification with convolutionalnetworks. IEEE 14th Intl Symp on Biomedical Imaging, 2017, pp. 160–163. doi:10.1109/ISBI.2017.7950492.157. Kwak, J.T.; Hewitt, S.M. Nuclear Architecture Analysis of Prostate Cancer via Convolutional Neural Networks. IEEE ACCESS , , 18526–18533. doi:10.1109/ACCESS.2017.2747838.158. Roy, K.; Banik, D.; Bhattacharjee, D.; Nasipuri, M. Patch-based system for Classification of Breast Histologyimages using deep learning. Computerized Medical Imaging and Graphics , , 90–103.159. Gecer, B.; Aksoy, S.; Mercan, E.; Shapiro, L.G.; Weaver, D.L.; Elmore, J.G. Detection and classification of cancerin whole slide breast histopathology images using deep convolutional networks. Pattern recognition , , 345–356.160. Wang, C.; Shi, J.; Zhang, Q.; Ying, S. Histopathological image classification with bilinear convolutional neuralnetworks. 39th Annual Intl Conf of the IEEE Engineering in Medicine and Biology Society, 2017, pp. 4050–4053.doi:10.1109/EMBC.2017.8037745.161. Hou, L.; Nguyen, V.; Kanevsky, A.B.; Samaras, D.; Kurc, T.M.; Zhao, T.; Gupta, R.R.; Gao, Y.; Chen, W.; Foran,D.; others. Sparse autoencoder for unsupervised nucleus detection and representation in histopathologyimages. Pattern recognition , , 188–200.162. Li, C.; Wang, X.; Liu, W.; Latecki, L.J.; Wang, B.; Huang, J. Weakly supervised mitosis detection in breasthistopathology images using concentric loss. Medical image analysis , , 165–178.163. Sheikh, T.S.; Lee, Y.; Cho, M. Histopathological Classification of Breast Cancer Images Using a Multi-ScaleInput and Multi-Feature Network. Cancers , , 2031. ournal Not Specified , xx , 5 43 of 45 Computer Methods and Programs in Biomedicine , , 538–556. doi:10.1016/j.cmpb.2011.12.007.165. Irshad, H.; Veillard, A.; Roux, L.; Racoceanu, D. Methods for Nuclei Detection, Segmentation, and Classificationin Digital Histopathology: A Review - Current Status and Future Potential. IEEE Reviews in BiomedicalEngineering , , 97–114. doi:10.1109/RBME.2013.2295804.166. Deshmukh, B.S.; Mankar, V.H. Segmentation of Microscopic Images: A Survey. Intl Conf on Electronic Systems,Signal Processing and Computing Technologies, 2014, pp. 362–364. doi:10.1109/ICESC.2014.68.167. Akhila, E.; Preethymol, B. Detection of malignant tissues: Analysis on segmentation of histologyimages. Intl Conf on Innovations in Information, Embedded and Communication Systems, 2015, pp. 1–4.doi:10.1109/ICIIECS.2015.7192913.168. Veta, M.; van Diest, P.J.; Willems, S.M.; Wang, H.; Madabhushi, A.; Cruz-Roa, A.; Gonzalez, F.; Larsen, A.B.L.;Vestergaard, J.S.; Dahl, A.B.; Cire¸san, D.C.; Schmidhuber, J.; Giusti, A.; Gambardella, L.M.; Tek, F.B.; Walter,T.; Wang, C.W.; Kondo, S.; Matuszewski, B.J.; Precioso, F.; Snell, V.; Kittler, J.; de Campos, T.E.; Khan, A.M.;Rajpoot, N.M.; Arkoumani, E.; Lacle, M.M.; Viergever, M.A.; Pluim, J.P.W. Assessment of algorithms for mitosisdetection in breast cancer histopathology images. Medical Image Analysis , , 237–248.169. Nawaz, S.; Yuan, Y. Computational pathology: Exploring the spatial dimension of tumor ecology. CancerLetters , , 296–303. doi:10.1016/j.canlet.2015.11.018.170. Madabhushi, A.; Lee, G. Image analysis and machine learning in digital pathology: Challenges andopportunities. Medical Image Analysis , , 170–175. doi:10.1016/j.media.2016.06.037.171. Saha, M.; Mukherjee, R.; Chakraborty, C. Computer-aided diagnosis of breast cancer using cytological images:A systematic review. Tissue and Cell , , 461–474. doi:10.1016/j.tice.2016.07.006.172. Chen, J.M.; Li, Y.; Xu, J.; Gong, L.; Wang, L.W.; Liu, W.L.; Liu, J. Computer-aided prognosis onbreast cancer with hematoxylin and eosin histopathology images: A review. Tumor Biology , .doi:10.1177/1010428317694550.173. Robertson, S.; Azizpour, H.; Smith, K.; Hartman, J. Digital image analysis in breast pathology—from imageprocessing techniques to artificial intelligence. Translational Research , pp. –. doi:10.1016/j.trsl.2017.10.010.174. Cosma, G.; Brown, D.; Archer, M.; Khan, M.; Pockley, A.G. A survey on computational intelligenceapproaches for predictive modeling in prostate cancer.
Expert Systems with Applications , , 1–19.doi:10.1016/j.eswa.2016.11.006.175. Tosta, T.A.A.; Neves, L.A.; do Nascimento, M.Z. Segmentation methods of H&E-stained histological images oflymphoma: A review. Informatics in Medicine Unlocked , , 35–43.176. Litjens, G.; Kooi, T.; Bejnordi, B.E.; Setio, A.A.A.; Ciompi, F.; Ghafoorian, M.; van der Laak, J.A.W.M.; vanGinneken, B.; Sánchez, C.I. A survey on deep learning in medical image analysis. Medical Image Analysis , , 60–88.177. Cataldo, S.D.; Ficarra, E. Mining textural knowledge in biological images: Applications, methods and trends. Computational and Structural Biotechnology Journal , , 56–67. doi:10.1016/j.csbj.2016.11.002.178. Aswathy, M.A.; Jagannath, M. Detection of breast cancer on digital histopathology images: Present status andfuture possibilities. Informatics in Medicine Unlocked , , 74–79. doi:10.1016/j.imu.2016.11.001.179. Li, Z.; Zhang, X.; Müller, H.; Zhang, S. Large-scale retrieval for medical image analytics: A comprehensivereview. Medical Image Analysis , , 66–84. doi:10.1016/j.media.2017.09.007.180. Komura, D.; Ishikawa, S. Machine Learning Methods for Histopathological Image Analysis. Computational andStructural Biotechnology Journal , pp. –. doi:10.1016/j.csbj.2018.01.001.181. Zhou, X.; Li, C.; Rahaman, M.M.; Yao, Y.; Ai, S.; Sun, C.; Wang, Q.; Zhang, Y.; Li, M.; Li, X.; Jiang, T.; Xue, D.; Qi,S.; Teng, Y. A Comprehensive Review for Breast Histopathology Image Analysis Using Classical and DeepNeural Networks.
IEEE Access , , 90931–90956. doi:10.1109/ACCESS.2020.2993788.182. Krithiga, R.; Geetha, P. Breast Cancer Detection, Segmentation and Classification on Histopathology ImagesAnalysis: A Systematic Review. Arch Computat Methods Eng . doi:10.1007/s11831-020-09470-w. ournal Not Specified , xx , 5 44 of 45 Journal of pathologyinformatics , , 8–8. doi:10.4103/2153-3539.112693.186. TCGA. The Cancer Genome Atlas Program. http://cancergenome.nih.gov/. Accessed: 2019-08-16.187. Roux, L. MITOS-ATYPIA-14 - MITOS & ATYPIA 14 Contest Home Page. https://mitos-atypia-14.grand-challenge.org/, 2014. Accessed: 2019-08-16.188. Network, C.G.A.R. Comprehensive molecular profiling of lung adenocarcinoma. Nature , , 543–550.doi:10.1038/nature13385.189. Marinelli, R.J.; Montgomery, K.; Liu, C.L.; Shah, N.H.; Prapong, W.; Nitzberg, M.; Zachariah, Z.K.; Sherlock,G.J.; Natkunam, Y.; West, R.B.; van de Rijn, M.; Brown, P.O.; Ball, C.A. The Stanford Tissue Microarray Database. Nucleic acids research , , D871–D877. doi:10.1093/nar/gkm861.190. Spanhol, F.A.; Oliveira, L.S.; Petitjean, C.; Heutte, L. A Dataset for Breast Cancer Histopathological ImageClassification. IEEE Transactions on Biomedical Engineering , , 1455–1462. doi:10.1109/TBME.2015.2496264.191. Medical Image Computing and Computer-Assisted Intervention (MICCAI) , Vol. 8673,
Lecture Notes in ComputerScience . Springer, 2014. doi:10.1007/978-3-319-10404-1.192. Saifuddin, S.R.; Devlies, W.; Santaolalla, A.; Cahill, F.; George, G.; Enting, D.; Rudman, S.; Cathcart, P.;Challacombe, B.; Dasgupta, P.; Galustian, C.; Chandra, A.; Chowdhury, S.; Gillett, C.; Van Hemelrijck,M. King’s Health Partners’ Prostate Cancer Biobank (KHP PCaBB).
BMC cancer , , 784–784.doi:10.1186/s12885-017-3773-8.193. Mazoa, C.; Trujilloa, M.; Alegreb, E.; Salazar, L. Banco de Imagenes Histologicas sobre el Sistema CardiovascularHumano. http://biscar.univalle.edu.co/, 2016. Accessed: 2019-08-16.194. Bejnordi, B.E.; Veta, M.; van Diest, P.J.; van Ginneken, B.; Karssemeijer, N.; Litjens, G.; van der Laak, J.A.W.M.;the CAMELYON16 Consortium. Diagnostic Assessment of Deep Learning Algorithms for Detection of LymphNode Metastases in Women With Breast Cancer. JAMA , , 2199–2210. doi:10.1001/jama.2017.14585.195. Kruka, S.M. Fuhrman grades nuclei. http://michalkruk.pl/FDataset.zip, 2016. Accessed: 2019-08-16.196. Kruka, S.M. Fuhrman grades images. http://michalkruk.pl/Images.zip, 2016. Accessed: 2019-08-16.197. Araujo, T.; Aresta, G.; Castro, E.; Rouco, J.; Aguiar, P.; Eloy, C.; Polonia, A.; Campilho, A.Classification of breast cancer histology images using convolutional neural networks. PLoS ONE , . doi:10.1371/journal.pone.0177544.198. Yu, F.; Ip, H.H. Semantic content analysis and annotation of histological images. Computers in Biology andMedicine , , 635 – 649. doi:10.1016/j.compbiomed.2008.02.004.199. Hafiane, A.; Bunyak, F.; Palaniappan, K. Clustering initiated multiphase active contours and robustseparation of nuclei groups for tissue segmentation. 19th Intl Conf on Pattern Recognition, 2008, pp. 1–4.doi:10.1109/ICPR.2008.4761744.200. Doyle, S.; Feldman, M.; Tomaszewski, J.; Madabhushi, A. A Boosted Bayesian Multiresolution Classifier forProstate Cancer Detection From Digitized Needle Biopsies. IEEE Transactions on Biomedical Engineering , , 1205–1218. doi:10.1109/TBME.2010.2053540.201. Monaco, J.P.; Tomaszewski, J.E.; Feldman, M.D.; Hagemann, I.; Moradi, M.; Mousavi, P.; Boag, A.;Davidson, C.; Abolmaesumi, P.; Madabhushi, A. High-throughput detection of prostate cancer inhistological sections using probabilistic pairwise Markov models. Medical Image Analysis , , 617 –629. doi:10.1016/j.media.2010.04.007.202. Lee, G.; Doyle, S.; Monaco, J.; Madabhushi, A.; Feldman, M.D.; Master, S.R.; Tomaszewski, J.E. A knowledgerepresentation framework for integration, classification of multi-scale imaging and non-imaging data: ournal Not Specified , xx , 5 45 of 45 Preliminary results in predicting prostate cancer recurrence by fusing mass spectrometry and histology. 2009IEEE Intl Symp on Biomedical Imaging: From Nano to Macro, 2009, pp. 77–80. doi:10.1109/ISBI.2009.5192987.203. Caicedo, J.C.; González, F.A.; Triana, E.; Romero, E. Design of a Medical Image Database with Content-BasedRetrieval Capabilities. Advances in Image and Video Technology. Springer Berlin Heidelberg, 2007, pp.919–931.204. Basavanhally, A.; Ganesan, S.; Shih, N.; Mies, C.; Feldman, M.; Tomaszewski, J.; Madabhushi, A. A boostedclassifier for integrating multiple fields of view: Breast cancer grading in histopathology. 2011 IEEE Intl Sympon Biomedical Imaging: From Nano to Macro, 2011, pp. 125–128. doi:10.1109/ISBI.2011.5872370.205. Basavanhally, A.; Feldman, M.; Shih, N.; Mies, C.; Tomaszewski, J.; Ganesan, S.; Madabhushi, A.Multi-field-of-view strategy for image-based outcome prediction of multi-parametric estrogen receptor-positivebreast cancer histopathology: Comparison to Oncotype DX.
Journal of Pathology Informatics , , 1.doi:10.4103/2153-3539.92027.206. Saraswat, M.; Arya, K.; Sharma, H. Leukocyte segmentation in tissue images using differential evolutionalgorithm. Swarm and Evolutionary Computation , , 46 – 54. doi:10.1016/j.swevo.2013.02.003.207. Wang, L.W.; fang Yang, G.; Chen, J.W.; Yang, F.; ping Yuan, J.; Sun, S.; Chen, C.Q.; bai Hu, M.; Li, Y. A clinicaldatabase of breast cancer patients reveals distinctive clinico-pathological characteristics: a study from centralChina. Asian Pacific journal of cancer prevention ,
15 4 , 1621–6.208. Lezoray, O.; Cardot, H. Cooperation of color pixel classification schemes and color watershed: a study formicroscopic images.
IEEE Transactions on Image Processing , , 783–789. doi:10.1109/TIP.2002.800889.209. Yang, L.; Qi, X.; Xing, F.; Kurc, T.; Saltz, J.; Foran, D.J. Center for Biomedical Imaging & Informatics. http://pleiad.umdnj.edu/CBII/Bioinformatics/, 2013. Accessed: 2019-08-16.210. Langer, R.; Rauser, S.; Feith, M.; Nährig, J.M.; Feuchtinger, A.M.E.; Frieß, H.; Hoefler, H.; Walch, A.K.Assessment of ErbB2 (Her2) in oesophageal adenocarcinomas: summary of a revised immunohistochemicalevaluation system, bright field double in situ hybridisation and fluorescence in situ hybridisation. ModernPathology , , 908–916.211. Cheng, L.; Jones, T.D.; Pan, C.X.; Barbarin, A.; Eble, J.N.; Koch, M.O. Anatomic distribution and pathologiccharacterization of small-volume prostate cancer ( < ModernPathology , , 1022–1026. doi:10.1038/modpathol.3800431.212. Drelie Gelasca, E.; Obara, B.; Fedorov, D.; Kvilekval, K.; Manjunath, B.S. A biosegmentation benchmark forevaluation of bioimage analysis methods. BMC Bioinformatics , , 368. doi:10.1186/1471-2105-10-368.213. Group, Q.C. Adjuvant chemotherapy versus observation in patients with colorectal cancer: a randomisedstudy. The Lancet ,370