Ink removal from histopathology whole slide images by combining classification, detection and image generation models
Sharib Ali, Nasullah Khalid Alham, Clare Verrill, Jens Rittscher
IINK REMOVAL FROM HISTOPATHOLOGY WHOLE SLIDE IMAGES BY COMBININGCLASSIFICATION, DETECTION AND IMAGE GENERATION MODELS
Sharib Ali (cid:63)
Nasullah Khalid Alham (cid:63), † Clare Verrill † Jens Rittscher (cid:63) (cid:63)
Institute of Biomedical Engineering, Department of Engineering Science,University of Oxford, Oxford † Nuffield Department of Surgical Sciences and Oxford NIHR Biomedical Research Centre (BRC),University of Oxford, John Radcliffe Hospital, Oxford
ABSTRACT
Histopathology slides are routinely marked by pathologistsusing permanent ink markers that should not be removed asthey form part of the medical record. Often tumour regionsare marked up for the purpose of highlighting features or otherdownstream processing such an gene sequencing. Once digi-tised there is no established method for removing this in-formation from the whole slide images limiting its usabil-ity in research and study. Removal of marker ink from thesehigh-resolution whole slide images is non-trivial and com-plex problem as they contaminate different regions and inan inconsistent manner. We propose an efficient pipeline us-ing convolution neural networks that results in ink-free im-ages without compromising information and image resolu-tion. Our pipeline includes a sequential classical convolutionneural network for accurate classification of contaminated im-age tiles, a fast region detector and a domain adaptive cy-cle consistent adversarial generative model for restoration offoreground pixels. Both quantitative and qualitative resultson four different whole slide images show that our approachyields visually coherent ink-free whole slide images.
Index Terms — Digital histopathology, Marker ink re-moval, Deep learning, CNN, GANs
1. INTRODUCTION
Histopathology slides are routinely marked by pathologistsusing permanent ink makers before imaging that should notusually be removed as these markings form part of the medi-cal record. Once the slides are digitised, these markings con-taminate the images. Such images are not advisable to be usedfor research and educational purposes as these markings of-ten contain information which could bias results of a study.Slides are for example marked with “EPE” referring to extra-prostatic extension could be a visual prompt for a pathologist.In addition, Digital images contaminated with ink mark-ers also obstruct further analysis as many regions could besubmerged into a dark ink blots appearing in digital images
Fig. 1 : Marked histology slides.
Shown are two whole slideimages which have been marked by pathologists. For exam-ple, the circle on the left indicates an area of tumour thatshould be used for subsequent sequencing. Our goal is to re-move such ink marks from digitised images so that these slidescan be used for teaching and computational analysis. and absence of clear structure of the underlying cells. To en-able the use of these slides we propose a method for removingthese ink marks and use domain adaptive cycle consistent ad-versarial generative model for restoration of foreground pix-els. This work is based on the assumption that the ink marksdo not obscure any diagnostically relevant information.Ink (referred to “marker ink” in this paper) markingscan of course be made by pens of different colours. Thesemarkings can be on the tissue or outside around the tissue orboth. Also, they are often composed of inconsistent patternscomprising of various elements like letters, circles, arrows,numbers or dots or a combination of them (see Fig. 1). Inkremoval from corrupted histopathogy images is very chal-lenging as these tissue samples also possess inter- and intra-texture variabilities (e.g. due to inconsistency in staining).Established methods that are capable of compensating forstaining variations cannot be used to remove ink marks. Ma-cenko et al. [1] convert RGB colours into corresponding opti-cal density values to compensate for variations in hematoxylinand eosin staining. Niethammer and colleagues [2] proposea colour correction model based on a similar approach butan improved estimation technique. A look-up table (LUT)based on the dye concentration absorbed by the sample was a r X i v : . [ c s . C V ] M a y ig. 2 : Unrealistic hallucinations.
The naive application ofgenerative models using CycleGAN [5] can lead to undesiredresults. built in [3] to correct for staining inconsistencies. Such colourtransformations will not be sufficient to remove high opacitysignals such as permanent ink markings. Given the texturevariation of the original tissue image static transformationslike those stored in look up tables cannot be used to restorethe original image.Recent developments in deep learning based adversarialnetworks [4] allows the modelling of complex data distri-butions that might not be captured by classical approaches.When compared to classical stain normalisation approachesCycle consistent Generative Adversarial Networks (Cycle-GANs [5]) can lead to improved results [6]. While thesemodels are very powerful, the example shown in Figure 2illustrates that any naive application of such models can leadto implausible results. In order to minimize such unrealistichallucinations, we propose a fully automatic CNN-based ap-proach that includes: 1) separating background ink corruptionfrom foreground ink corruption, 2) identifying regions in im-age tiles with ink, and sparse or compactly arranged cells orcytoplasms and 3) domain learning, separately for sparse anddensely arranged cell or cytoplasmic distributions. Finally,we have evaluated our approach on four different whole slideimages contaminated with marker inks.
2. METHOD
We propose a fully automatic convolution neural networkbased approach (see Fig. 3) for the classification and restora-tion of whole slide images containing ink marks in differentcolours. Our model includes: 1) Sequential CNN based archi-tecture for binary classification, 2) Yolov3 [7] for detection ofbounding boxes for ink corrupted and cell cluster regions, and3) CycleGAN [5] for domain learning. Each of these processare described below. The code for the presented pipeline withtrained weights and sample images used in this paper hasbeen made publicly available . This tool is only for researchuse and is not validated for use in a diagnostic setting. We propose to use a sequential convolution neural networkarchitecture shown in Figure 4 for our two step binary clas- https://github.com/sharibox/histopathology-inkRemoval Fig. 3 : Proposed appraoch
Rather than applying a globaltransformation model, CNN based binary detectors are usedto delineate the ink markings. Subsequently, generative mod-els are used to replace the missing information. First bi-nary classification block outputs ink contaminated image tileswhich is then fed to another sequential CNN classifier thatseparates image tiles with only background ink from fore-ground ink. Identified image tiles having only background inkare replaced with clean image tile (white background in ourcase) while the foreground ink contaminated tile goes to theforeground ink and cluster localization block.
Fig. 4 : Binary classifier.
The identical CNN architecture isused to classify if an entire tile or a pixel location contains inkmarks (see Figure 3). The model has 433K trainable parame-ters and permits rapid training and near real-time execution. sification: 1) classification of image tiles contaminated withink from non-ink ones, and 2) classification of backgroundcontamination from the foreground. We have used cross en-tropy as loss function and an Adam optimizer with learningrate of 0.00001. The used model has only 433K trainable pa-rameters. A total of samples were utilized to train thetile classification model and samples were used for pixellevel background/foreground classification. In each case, wehave used 40% samples for validation and the tiles incorpo-rate from 4 whole slide images used for training only.
Precise localization of the ink contaminated areas within eachof image tiles can substantially help in preserving underly-ing tissue information. That is, only the areas that are con-taminated with ink are restored while uncontaminated regionis kept intact (see Fig. 5 (a), top, pink box). Secondly, wea) (b) (c)
Fig. 5 : Ink removal examples
Top: a) Identified contamina-tion of a portion of the image, (b) and (c) identified contam-ination of the entire tile. Bottom: restoration results usingcycleGAN with (a-b) dense and (c) sparse domain weights. Itis to be noted that only the idenified corrupted part was re-stored in (a). Pink and green bounding-boxes on the top rowrepresent ink and cell cluster areas, respectively. identify image tiles with clustered cells or cytoplasm (seeFig. 5 (a-b), top, green box). We have used fast and accu-rate Yolov3 single-shot detector [7]. It predicts simultane-ously class and bounding box coordinates using convolutionfilters and skip connections in multi-scale approach. We havetrained darknet-53 model of Yolov3 for 2 class (ink and clus-ter) using bounding box annotations on 550 image tiles.
Generative adversarial networks [4] have become a power-ful tool for image generation. As the original uncorruptedimage is not available to us, we use CycleGAN which al-lows to map source domain to target domain without needfor a paired image-to-image mapping. The model consists oftwo generator-discriminator pairs that operates cyclically formapping image in one domain ( A ) to another domain ( B ).While, generator G B generates image similar to domain B ,discriminator D B evaluates its truthfulness. The generatedimages are then mapped to domain A back utilizing anothergenerator-discriminator pair of domain A . In order to achievecycle-consistent mapping function an l regularization wasintroduced in [5]. Similar cyclic process is repeated in reversedirection for mapping an image in domain B → A . In gist,the generator and discriminator plays a game until Nash equi-librium is achieved, i.e., the generator’s distribution becomessame as the desired distribution.Because of the variation of the underlying data distribu-tion domain learning in histopathology data needs to be care-fully defined. In order to prevent from over stretching of two distributions, we propose to utilize two different data distribu-tion pairs based on cell or cytoplasmic mass clusters, namely:a) dense and b) sparse. We utilized ≈ samples eachfor training upto 400 epochs with 100 iterations. During therestoration process, the distribution is identified by our objectdetector (see Section 2.2) for which corresponding domainmapping is applied. In Fig. 5 (a-b) domain learning fromdense distribution is applied while sparse in case of Fig. 5(c) yielding a very high-quality restorations.
3. EXPERIMENTS
Eight Formalin-Fixed Paraffin-Embedded (FFPE) H&E (Hema-toxylin and Eosin) stained prostate × mm slides scannedusing the Objective Imaging desktop scanner at a resolution40X were used for this study. These slides included inkmarks in four different colours (green, blue, black and red)that appeared separately or in combination. High-resolutionimage tiles ( × pixels) generated by the scannerwere used for our ink removal experiments. For training,these image tiles were scaled to appropriate resolutions (seeTable 1). However, the trained weights for both localizationand image generation were applied to original image tileresolution. Restored image tiles were converted to BigTiffwhole slide image format using OIWorkspaceConverter for inspection by pathologists. Whole slide images wereviewed in full detail using an standalone workspace viewer(OISwsViewer ). The average size of each whole slide imagewas around 4.5 GB.We evaluated our sequential CNN model using an aver-age of 549 image tiles and Yolov3 model using 100 imagetiles with ground truth annotations. In order to obtain quan-titative results for cycleGAN, we used CMYK color spacetransformation to add ink color markings (from only back-ground inked tiles in our dataset) on 20 non-corrupted imagetiles in different proportions. Table 1 shows average accuracy ( ¯ Acc.) of 95% for our se-quential CNN model and a mean average precision (mAP) of75% for Yolov3. Our binary classifier has light weight (5.2MB) and is computationally very fast both for training (10mins) and at test time (0.08 s). Experimental results on cy-cleGAN using distributed weights (sparse-dense) showed animprovement in all the image quality metrics, notably 28.38dB to 28.73 dB, 0.69 to 0.71, and 0.75 to 0.78, respectivelyfor mean values of PSNR (Peak Signal to Noise Ratio), SSIM(Structure Similarity Measure) and VIF (Visual InformationFidelity). Some notable improvements for 5 simulated inkedimage tiles are provided in Tab. 2. It is worth noting that thetile ethod time/epochs ¯ Acc./mAP ∗ Size Test load time Train samples Checkpoint sizemins./ × ∗ × × Table 1 : Quantitative information for different networks used in our proposed ink removal pipeline. All timings are providedfor NVIDIA Tesla P100 GPU.
Tile PSNR SSIM VIFInked Restored Inked Restored Inked
Restored
Table 2 : Quantitative results.
Image quality improvementfor restoration of 5 simulated inked tile images using pro-posed sparse-dense CycleGAN .
Fig. 6 : Background ink removal.
Ink removal from back-ground of histopathology whole slide images using only CNN-based binary classifier. Left: original image corrupted withink, right: restored image. At most times removing only back-ground markings is sufficient.
Visual analysis for fast and accurate ink removal from back-ground only utilizing solely our binary sequential CNN isshown in Fig. 6. Other markers like dots and circles asin Fig. 6 alone do not contain any evidential informationonce the surrounding texts are removed. 5 out of 8 of ourwhole slide images had marker inks mostly on the back-ground. Fig. 7 (left) shows marker ink mostly on foregroundpixels that also include parts of tissues. We utilize our com-
Fig. 7 : Foreground and background ink removal.
Inkremoval from foreground and background of histopathologywhole slide images using our pipeline. Left: original imagecorrupted with ink, right: restored image. Images have beenscaled to 10% of original image size.
Fig. 8 : Failed cases.
Added textures in restored image tiles. plete pipeline process to realistically remove and restore inkcontaminated regions. Fig. 7 (right) shows restored wholeslide image which is visually coherent and marker ink hasbeen substantially removed from both background and fore-ground. Fig. 8 presents some of failure results where therestoration is not perfect and addition of some texture can beclearly observed.
4. CONCLUSION
We have proposed a content-aware and fully automatic deeplearning based pipeline for efficient removal of marker inksfrom whole slide images without sacrificing information. Weperformed both quantitative and qualitative evaluations ofeach of the method separately which showed the efficacyof the methods. This work addresses a key issue faced bypathologists regarding usability of whole-slide images forstudy and research.
Acknowledgement
SA, NK and CV are supported by the NIHR Oxford BRC. JRis funded by EPSRC EP/M013774/1 Seebibyte. . REFERENCES [1] M. Macenko, M. Niethammer, J. S. Marron, D. Bor-land, J. T. Woosley, Xiaojun Guan, C. Schmitt, and N. E.Thomas, “A method for normalizing histology slidesfor quantitative analysis,” in
IEEE International Sym-posium on Biomedical Imaging (ISBI) , June 2009, pp.1107–1110.[2] Marc Niethammer, David Borland, J. S. Marron, JohnWoosley, and Nancy E. Thomas, “Appearance normaliza-tion of histology slides,” in
Machine Learning in MedicalImaging . 2010, pp. 58–66, Springer Berlin Heidelberg.[3] Pinky A. Bautista and Yukako Yagi, “Staining correc-tion in digital pathology by utilizing a dye amount table,”
Journal of Digital Imaging , vol. 28, no. 3, pp. 283–294,2015.[4] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza,Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C.Courville, and Yoshua Bengio, “Generative adversarialnets,” in
Annual Conference on Neural Information Pro-cessing Systems (NIPS) , Dec 2014, pp. 2672–2680.[5] J. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpairedimage-to-image translation using cycle-consistent adver-sarial networks,” in
IEEE International Conference onComputer Vision (ICCV) , Oct 2017, pp. 2242–2251.[6] M. Tarek Shaban, Christoph Baur, Nassir Navab, andShadi Albarqouni, “Staingan: Stain style transfer fordigital histological images,”
CoRR , vol. abs/1804.01601,2018.[7] Joseph Redmon and Ali Farhadi, “Yolov3: An incre-mental improvement,” arXiv preprint arXiv:1804.02767arXiv preprint arXiv:1804.02767