Improve Global Glomerulosclerosis Classification with Imbalanced Data using CircleMix Augmentation
Yuzhe Lu, Haichun Yang, Zheyu Zhu, Ruining Deng, Agnes B. Fogo, Yuankai Huo
IImprove Global Glomerulosclerosis Classification withImbalanced Data using CircleMix Augmentation
Yuzhe Lu a , Haichun Yang b , Zheyu Zhu a , Ruining Deng a , Agnes B. Fogo b , and Yuankai Huo aa Department of Electrical Engineering & Computer Science, Vanderbilt University, Nashville,TN, USA 37235 b Department of Pathology, Microbiology & Immunology, Vanderbilt University MedicalCenter, Nashville, TN, USA 37232
ABSTRACT
The classification of glomerular lesions is a routine and essential task in renal pathology. Recently, machinelearning approaches, especially deep learning algorithms, have been used to perform computer-aided lesion char-acterization of glomeruli. However, one major challenge of developing such methods is the naturally imbalanceddistribution of different lesions. In this paper, we propose CircleMix, a novel data augmentation technique,to improve the accuracy of classifying globally sclerotic glomeruli with a hierarchical learning strategy. Differ-ent from the recently proposed CutMix method, the CircleMix augmentation is optimized for the ball-shapedbiomedical objects, such as glomeruli. 6,861 glomeruli with five classes (normal, periglomerular fibrosis, obso-lescent glomerulosclerosis, solidified glomerulosclerosis, and disappearing glomerulosclerosis) were employed todevelop and evaluate the proposed methods. From five-fold cross-validation, the proposed CircleMix augmen-tation achieved superior performance (Balanced Accuracy= 73 . . Keywords: fine-grained image classification, imbalanced data, CircleMix, global glomerulosclerosis
1. INTRODUCTION
The identification of non-sclerotic and sclerotic glomeruli is essential to compute percentage of global glomeru-losclerosis, a quantitative measurement corresponding to several critical clinical outcomes. With fine-graineddefinition, globally sclerotic glomeruli, also called global glomerulosclerosis, can be characterized into three cat-egories: obsolescent glomerulosclerosis, solidified glomerulosclerosis, or disappearing glomerulosclerosis. Asglobally sclerotic glomeruli occur with both aging and kidney diseases, the fine-grained phenotype would providemore precise evidence to support both scientific research and clinical decision making. However, differentiatingthese patterns typically requires heavy manual efforts by trained clinical experts, which is not only tedious, butalso labor-intensive. Therefore, there is a strong need to develop automatic classification algorithms to performfine-grained glomerulosclerosis classification, especially with an increasingly large amount of digitized data fromwhole slide imaging (WSI).In the past few years, many studies have been conducted to classify different glomerular lesions usingcomputer-aided approaches.
However, there are few, if any, studies that have developed deep learning ap-proaches for fine-grained classification of glomerular lesions to characterize the globally sclerotic glomeruli intothree categories: obsolescent glomerulosclerosis, solidified glomerulosclerosis, and disappearing glomerulosclero-sis. Such fine-grained characterization is challenging, as the available data are typically highly imbalanced. Forinstance, the prevalence of obsolescent glomerulosclerosis is naturally much higher than solidified or disappearingglomerulosclerosis, leading to the technical difficulty which is well known as the “imbalanced classes problem”.In this paper, we propose CircleMix, a novel data augmentation technique, to improve the accuracy forclassifying non-sclerotic and sclerotic glomeruli, as well as fine-grained sub-types of globally sclerotic glomeruli.Our CircleMix algorithm is inspired by the prevalent CutMix augmentation, yet is optimized for ball-shaped Further author information: (Send correspondence to Yuankai Huo)Yuankai Huo: E-mail: [email protected] a r X i v : . [ q - b i o . Q M ] J a n
0% Mix 20% Mix20% Mix50% Mix50% Mix 20% Mix20% Mix50% Mix CutMixCircleMixNormalGlobal SclerosisInputs
Figure 1. This figure shows the examples of performing different data augmentation strategies. The left panel shows theexamples of glomerular image patches, which can be achieved from either object detection or manual annotation. Theglomeruli are typically located in the central location within the image patches. In the upper right panel, the morphologicalfeatures from one glomerulus can be largely lost when performing CutMix. By contrast, the CircleMix maintains themorphological features from both glomeruli with different percentages of mixture. biomedical objects, such as glomeruli in this study (Figure 1). To further enhance the performance of imbalancedclasses, the training is modeled as a hierarchical training procedure. To train and evaluate the deep learningalgorithms, we collected images from 6,861 glomeruli with five classes (normal, periglomerular fibrosis, obsolescentglomerulosclerosis, solidified glomerulosclerosis, and disappearing glomerulosclerosis)To summarize, the contribution of this work is three-fold: • We proposed the CircleMix, a novel data augmentation algorithm that is optimized for ball-shaped biomed-ical objects. • We evaluated the performance of hierarchical learning strategy on the fine-grained classification of glomeruliwith imbalanced data distribution. • To the best of our knowledge, this is the biggest study so far (6,861 glomeruli) to investigate deep learningbased image classification on both (1) non-sclerotic vs. sclerotic glomeruli, and (2) fine-grained classificationof obsolescent, solidified, and disappearing glomerulosclerosis.
2. RELATED WORK
The groundbreaking learning capability provided by deep learning algorithms comes largely from the unprece-dented large number of parameters in neural networks. To improve the generalizability of the deep neuralnetworks, applying data augmentations is typically an inevitable step to introduce extra randomness to thetraining data. Beyond the standard single image based augmentation strategies, Yun et al. proposed a noveldata augmentation strategy, which is called CutMix, by mixing different images as new training data. By cuttingand pasting patches among training images, CutMix forced the deep networks to provide partial decisions on amixed image, which achieved the superior performance compared with benchmarks (e.g., Mixup ). However,one major problem of CutMix is that the random patch-based image fusion might lose discriminative featuresrom the source images. To optimize the mixing procedure for ball-shaped biomedical objects, the novel CircleMixalgorithm is proposed in this study.In image classification, there is a long-lasting issue called imbalanced classes problem. The problem occurswhen the numbers of samples are considerably imbalanced (e.g., one class can have ten times or more samplesthan another), where the predictions from the trained neural networks are typically biased to the majority class.Many previous efforts have been made to improve the performance on imbalanced data, such as data sampling, cost-sensitive learning, and their combination.
14, 15
In this study, we explored the effect of hierarchical learningstrategy to perform fine-grained classification on an imbalanced glomerular cohort.
3. METHODS3.1 CircleMix
In this paper, we propose CircleMix, a novel data augmentation technique optimized for ball-shaped glomeruli(Figure 1). Firstly, the start and end points on the sides of the image are randomly generated. Then, togetherwith the image center and corners between the start and end points, a polygon mask is produced, which is thenfilled with the corresponding pixels from the other training image. We define I ∈ R H × W × C as an input imagewith H × W resolution and C channels (e.g., three channels for RGB). Y is the one-hot-vector label of class forimage I . By performing CircleMix, a new training sample ( ˜ I, ˜ Y ) is formed by combining two training images( I A , Y A ) and ( I B , Y B ). The procedure is presented as the following equations˜ I = M (cid:12) I A + ( − M ) (cid:12) I B ˜ Y = λY A + (1 − λ ) Y B , (1)where M ∈ { , } H × W is a polygon mask for filling image A , while ( − M ) is the remaining polygon region forfilling image B . “ (cid:12) ” is element-wise multiplication. λ is calculated by ( r end − r start ) / r start , r end ∼ Uniform (0 , , r start , r end = min ( r start , r end ) , max ( r start , r end ) (2)In implementation, the CircleMix is performed by randomly combining two training samples within the samemini batch, according to Eq.1. Our proposed training framework is defined as a hierarchical architecture, as shown in Figure 2.Concretely, we trained a five-class classifier first. Then, we used the best five-class model from validation tofine-tune three children classifiers, with one to re-verify the classification of normal and periglomercular fibrosis,one to re-verify the classification of three global glomerulosclerosis types, and one to re-verify global solidifiedand global disappearing types. With each of these children classifier, we combined its predictions with that ofthe five-class classifier to produce the final results. Specifically, if prediction from the parent classifier falls intothe set of classes the child classifier is re-verifying, the final prediction will be decided by the child classifier.
4. EXPERIMENTS AND RESULTS4.1 Data
The human nephrectomy tissues were acquired from 23 patients, whose tissues were routinely processed andparaffin-embedded, with 3 µ m thickness sections cut and stained with PAS. 6,861 glomeruli were extractedfrom WSI using the EasierPath semi-manual annotation software. Then, all glomeruli were manually labeled,including 2,757 normal glomeruli, 2,206 periglomerular fibrosis glomeruli, 1,525 global obsolescent glomeruli, 135global solidified glomeruli, and 238 global disappearing glomeruli. The images were resized to 256 ×
12 3 42 3 43 4 EfficientNet-B0-C5EfficientNet-B0-C3EfficientNet-B0-C2 0 12 3 4 ✓ ✓✓ ✓ ✓ ✓ ✓ ⟳ ⟳ ⟳ ✓ ✓✓ ⟳ ⟳ ⟳ ⟳ ✓ ✓ ✓ . N o r m a l . P e r i g l o m e r u l a r F i b r o s i s . G l ob a l O b s o l e s ce n t . G l ob a l S o li d i f i e d4 . G l ob a l D i s a pp ea r i ng Figure 2. This figure shows the hierarchical learning framework. The left panel shows the imbalanced data distributionof our data. The right panel shows the hierarchical design. EfficientNet-B0-C5 is used to classify all five classes, and thenused to fine-tune children classifiers. Specifically, EfficientNet-B0-C3 is fine-tuned to perform classification on classes “2”,“3”, and “4”, EfficientNet-B0-C2 is fine-tuned to perform classification on classes “3” and“4”, and EfficientNet-B0-NC2is fine-tuned to perform classification on classes “0” and “1”.
In the experiments, EfficientNet-B0 is employed as the backbone model of classification due to its high efficiencyof learning large-scale images. We adapted the EfficientNet-B0 model pretrained on ImageNet by customizingthe fully-connected layers based on our tasks. The model was trained and tested with standard five-fold cross-validation. Briefly, the data was split into five folds at subject level, where each fold was withheld as testingdata once. The remaining data for each fold was split as 75% training data and 25% validation data. Therefore,for each fold, the final split was 60% training, 20% validation, and 20% test. To avoid data contamination, allglomeruli from a patient were used either for training, validation or testing.The model was trained using cross entropy loss with stochastic gradient descent optimizer and a batch size of16. We started with a learning rate of 0.01 and decayed it by 10 half way through the total number of trainingepochs. We used both balanced accuracy and balanced F F F The basic data augmentations we used include horizontally and vertically flipping 50% of all training images andrandomly cropping 0 −
16 pixels. They are applied to all the experiments in this study.
We evaluated the performance of CircleMix by training EfficientNet-B0 as (1) a standard binary classifier, and(2) a five-class classifier, without performing the hierarchical training. The binary classifier (“Binary” in Table1) classified all images as two classes: global glomerulosclerosis or others. The five-class classifier (“Five-class”in Table 1) performed the fine-grained five class classification. able 1. Non-hierarchical Training. Binary Five-classACC F1 ACC F1EfficientNet-B0 % % % %* “Binary” is the binary classification results of global glomerulosclerosis vs. others, while “Five-class” is thefine-grained five class classification. “ACC” is the balanced accuracy score. “F1” is the balanced F Table 2. Hierarchical Training.
C5 C5+NC2 C5+C3 C5+C2ACC F1 ACC F1 ACC F1 ACC F1EfficientNet-B0 % 67.8% 68.6% 67.0%EfficientNet-B0+CutMix % % % 66.7% % * “NC2”, “C2”, “C3”, and “C5” represent EfficientNet-B0-NC2, EfficientNet-B0-C2, EfficientNet-B0-C3,and EfficientNet-B0-C5, respectively. “C5+X” indicates the merged results using EfficientNet-B0-C5 andEfficientNet-B0-X.From the results, when applied the proposed CircleMix augmentation, the model achieved superior perfor-mance on both binary classification and five-class classification tasks in terms of balanced accuracy (ACC) andbalanced F F thanks to a much larger dataset. For thefive-class classification task, CircleMix helps to improve the balanced accuracy by over 3% and balanced F Non-hierarchicalLearning (C5)HierarchicalLearning(C5+NC2) Baseline CutMix CircleMix
Figure 3. This figure shows the detailed confusion matrix of different data augmentation and learning strategies. .4.2 Hierarchical Training
Next, we evaluated the performance of hierarchical training with different hierarchical combinations (Table 2).The “C5”, “C3”, “C2”, and “NC2” represented the four deep networks in Figure 2.Based on the experimental results, while EfficientNet-B0-C5 and EfficientNet-B0-NC2 together producesslightly better results, other combinations generally give inferior performance. We observed a performancedegredation of the “C3”and “C2” classifiers compared to the “C5” classifier. This might be because EfficientNet-B0-C3 and EfficientNet-B0-C2 are trained with too few data points due to the imbalanced nature of the dataset.The confusion matrices from the combination of EfficientNet-B0-C5 and EfficientNet-B0-NC2 are presentedtogether with those from the non-hierarchical experiments in Figure 3.
5. CONCLUSIONS
In this paper, we proposed CircleMix, a novel data augmentation algorithm optimized for ball-shaped biomedicalimage classification, which is able to outperform the baseline and the state-of-the-art CutMix augmentation inglomerular classification task. To address the imbalanced classes problem, we evaluated the performance ofthe hierarchical training strategy on the fine-grained glomerular classification task. Though this strategy showsmixed results, the best overall performance was nonetheless achieved by combining the CircleMix augmentationwith hierarchical training, compared with other experiments.
6. ACKNOWLEDGMENTS
This work was supported by NIH NIDDK DK56942(ABF). This work has not been submitted for publication orpresentation elsewhere.
REFERENCES [1] Marcantoni, C., Ma, L.-J., Federspiel, C., and Fogo, A. B., “Hypertensive nephrosclerosis in african ameri-cans versus caucasians,”
Kidney international (1), 172–180 (2002).[2] Marsh, J. N., Matlock, M. K., Kudose, S., Liu, T.-C., Stappenbeck, T. S., Gaut, J. P., and Swamidass,S. J., “Deep learning global glomerulosclerosis in transplant kidney frozen sections,” IEEE transactions onmedical imaging (12), 2718–2728 (2018).[3] Zeng, C., Nan, Y., Xu, F., Lei, Q., Li, F., Chen, T., Liang, S., Hou, X., Lv, B., Liang, D., et al., “Identifi-cation of glomerular lesions and intrinsic glomerular cells types in kidney diseases via deep learning,” TheJournal of Pathology .[4] Uchino, E., Suzuki, K., Sato, N., Kojima, R., Tamada, Y., Hiragi, S., Yokoi, H., Yugami, N., Minamiguchi,S., Haga, H., et al., “Classification of glomerular pathological findings using deep learning and nephrologist-aicollective intelligence approach,” medRxiv , 2019–12 (2020).[5] Ginley, B., Lutnick, B., Jen, K.-Y., Fogo, A. B., Jain, S., Rosenberg, A., Walavalkar, V., Wilding, G.,Tomaszewski, J. E., Yacoub, R., et al., “Computational segmentation and classification of diabetic glomeru-losclerosis,”
Journal of the American Society of Nephrology (10), 1953–1967 (2019).[6] Ginley, B., Jen, K.-Y., Rosenberg, A., Rossi, G. M., Jain, S., and Sarder, P., “Fully automated classificationof glomerular lesions in lupus nephritis,” in [ Medical Imaging 2020: Digital Pathology ], , 113200Y,International Society for Optics and Photonics (2020).[7] Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y., “Cutmix: Regularization strategy to train strongclassifiers with localizable features,” in [ Proceedings of the IEEE International Conference on ComputerVision ], 6023–6032 (2019).[8] Yang, H., Deng, R., Lu, Y., Zhu, Z., Chen, Y., Roland, J. T., Lu, L., Landman, B. A., Fogo, A. B., and Huo,Y., “Circlenet: Anchor-free detection with circle representation,” arXiv preprint arXiv:2006.02474 (2020).[9] Zhu, Z., Lu, Y., Deng, R., Yang, H., Fogo, A. B., and Huo, Y., “EasierPath: An Open-source Tool forHuman-in-the-loop Deep Learning of Renal Pathology,” arXiv e-prints , arXiv:2007.13952 (July 2020).[10] Shorten, C. and Khoshgoftaar, T. M., “A survey on image data augmentation for deep learning,”
Journalof Big Data (1), 60 (2019).11] Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D., “mixup: Beyond empirical risk minimization,” arXiv preprint arXiv:1710.09412 (2017).[12] Chawla, N. V., Bowyer, K. W., Hall, L. O., and Kegelmeyer, W. P., “Smote: synthetic minority over-sampling technique,” Journal of artificial intelligence research , 321–357 (2002).[13] Ling, C. X. and Sheng, V. S., “Cost-sensitive learning and the class imbalance problem,” (2008).[14] Tang, Y., Zhang, Y.-Q., Chawla, N. V., and Krasser, S., “Svms modeling for highly imbalanced classifica-tion,” IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) (1), 281–288 (2008).[15] Huang, C., Li, Y., Loy, C. C., and Tang, X., “Learning deep representation for imbalanced classification,”in [ Proceedings of the IEEE conference on computer vision and pattern recognition ], 5375–5384 (2016).[16] Tan, M. and Le, Q. V., “Efficientnet: Rethinking model scaling for convolutional neural networks,” arXivpreprint arXiv:1905.11946 (2019).[17] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L., “Imagenet: A large-scale hierarchical imagedatabase,” in [2009 IEEE conference on computer vision and pattern recognition