Computer Aided Detection of Anemia-like Pallor
Sohini Roychowdhury, Donny Sun, Matthew Bihis, Johnny Ren, Paul Hage, Humairat H. Rahman
CComputer Aided Detection of Anemia-like Pallor
Sohini Roychowdhury ∗ , Donny Sun , Matthew Bihis , Johnny Ren , Paul Hage , Humairat H. Rahman Abstract — Paleness or pallor is a manifestation of bloodloss or low hemoglobin concentrations in the human bloodthat can be caused by pathologies such as anemia. This workpresents the first automated screening system that utilizes pallorsite images, segments, and extracts color and intensity-basedfeatures for multi-class classification of patients with high pallordue to anemia-like pathologies, normal patients and patientswith other abnormalities. This work analyzes the pallor sites ofconjunctiva and tongue for anemia screening purposes. First,for the eye pallor site images, the sclera and conjunctivaregions are automatically segmented for regions of interest.Similarly, for the tongue pallor site images, the inner andouter tongue regions are segmented. Then, color-plane basedfeature extraction is performed followed by machine learningalgorithms for feature reduction and image level classificationfor anemia. In this work, a suite of classification algorithmsimage-level classifications for normal (class 0), pallor (class1) and other abnormalities (class 2). The proposed methodachieves 86% accuracy, 85% precision and 67% recall in eyepallor site images and 98.2% accuracy and precision with100% recall in tongue pallor site images for classificationof images with pallor. The proposed pallor screening systemcan be further fine-tuned to detect the severity of anemia-likepathologies using controlled set of local images that can thenbe used for future benchmarking purposes.
Index Terms:
Anemia, pallor, classification, feature ex-traction, gradient filter, Frangi-filterI. I
NTRODUCTION
Paleness or pallor is a manifestation of anemia, definedas abnormally low hemoglobin concentrations in the blood,which can be caused by blood loss, malnutrition, or otherpathologies. Hemoglobin serves a vital role in the blood,carrying oxygen to the tissues. While anemia affects around1.6 billion people worldwide, it is known to affect womenand preschool children 5-8 times more than men [1]. TheCenters for Disease Control and Prevention estimate that thenumber of visits to emergency departments in North Americawith anemia as the primary hospital discharge diagnosishave been steadily increasing from 1990-2011 [2]. Chronicanemia can contribute to problems such as chronic fatigue, ormore severe problems such as heart failure, limb ischemia,and pregnancy complications. With such large percentagesof the present-day population at risk of the detrimentalimpacts of anemia, the design of computer aided diagnostic(CAD) systems that can screen patients with anemia from thenormal patients by detecting pallor non-invasively becomesnecessary. CAD and point of care (POC) applications areaimed at providing quick “expert” diagnostics for screening Department of Electrical and Computer Engineering, University ofWashington, Bothell, WA-98011. ∗ Email: [email protected]. Department of Environmental and Occupational Health, University ofSouth Florida, Tampa, FL-33612 and resourcefulness of treatment and care-delivery protocols.Also, such systems are useful especially in a telemedicineparadigm, where, the patient and the care-provider maynot be geographically co-located. Facial images have beenextensively useful for security, authentication, identificationand expression detection purposes [3]. This work is aimedat utilizing facial pallor site images for anemia-like medicalscreening applications.This paper makes three key contributions. First, spatial,color-based and gradient based features are analyzed todetect the optimal combination for prediction of patientpallor. We observe that Frangi-filtering and gradient filteringenhance the image separability for pallor severity in eye andtongue pallor site images, respectively. Second, a hierarchicalclassification strategy is proposed using an optimal set ofcolor-based and intensity-based features for screening nor-mal, anemic and abnormal pallor site images. We observe72-86% separability of normal from abnormal images foreye and tongue pallor site images, respectively. Third, thediscriminating contribution of each pallor site image foranemia-like pallor is analyzed. Here, we observe that theeye pallor site has higher area under Receiver OperatingCharacteristic curves (AUC) when compared to that of thetongue pallor site images.II. M
ATERIALS AND M ETHODS
Based on prior works that analyze pallor site images foranemia-like diagnosis [4], this work focusses on the eye im-ages with visible conjunctiva and tongue images for anemia-like pallor detection. There are two primary objectives ofthe analysis presented in this work. First, the color andregion-based features in each pallor site image are analyzedto determine the most discriminating features for pallorclassification. Second, the importance of the eye and tonguepallor sites are assessed to identify the most significant pallorsite for anemia-like pathology classification. A description ofthe image data sets under analysis and the proposed methodsare given below.
A. Data
A set of 27 eye images and 56 tongue images are collectedand manually annotated for subjective pallor indices. Theseimages represent uncontrolled imaging conditions and a widevariety in patient demographics. Each pallor site image hasdimensions ranging from [155x240] to [960x1280] pixelswith storage size of 8kB to 252kB per image. Also, everypallor site image that is gathered from public domain sources,is manually annotated for pallor severity grade. While grade0 refers to normal patients, grade 1 refers to patients a r X i v : . [ c s . C V ] M a r ith anemia-like pathologies and grade 2 is indicative ofpathologies/abnormalities associated with the specific pallorsite that is not anemia-like. Examples of sample imagescorresponding to each severity grade from the eye and tonguepallor site images are shown in Fig. 1. The goal of the overallautomated system is to classify each pallor site image withoutput class label [0, 1, 2], representative of patient’s anemia-like pallor.For the eye and tongue pallor site images, the numbers ofimages belonging to the sample class labels representative ofseverity [0,1,2] are [6, 7, 14], and [18, 3, 35] respectively. Forhomogeneous processing purposes, each image ( I ) is resizedto [125x125] pixels each. For automated pattern recognitionfrom the pallor site images, the eye and tongue data sets arepartitioned into training and test data sets, respectively. Dueto the limited data sizes, feature learning and data modelingis performed using 5-fold cross validation for the eye imagesand 3-fold cross validation for tongue images [5]. This is toensure similar proportions of class sample data in each ofthe folded data sets. Fig. 1. Example images of eye and tongue pallor sites with varying pallorseverity grades.
B. Proposed Methods
Since analysis of pallor site images for anemia-likepathologies is a novel idea, there are no existing methodsin literature. However, based on the variabilities introducedby the data sets, two data models are analyzed for pallorseverity grade classification tasks. The first model ( M ) isdesigned to detect specific spatial regions of interest (ROIs)that are indicative of patient pallor. In this model, the pallorsite images are segmented to extract several ROIs, followedby extraction of pixel intensity features corresponding tocolor plane and gradient images within the ROIs. Next,the intensity-based features are ranked to detect the mostdiscriminating set of features from the training data set thatensure maximum accuracy in the validation data set [6][5].Finally, the most optimal feature set is utilized for pallorseverity grade classification.The second model ( M ) is designed to detect the mostsignificant color planes and gradient images for pallor clas-sification. For each pre-processed color plane image, a mask‘ g ’ of the eye or tongue region is detected. Color planetransformations are then applied to each pallor-site image,thus resulting in the following 12 image planes: red, green, blue, hue, saturation, intensity (from RGB to HSV transfor-mation), lightness, a-color plane, b-color plane (from RGBto Lab transformation), luminance, 2 chrominance planes(from RGB to Ycbcr transformation). Next the first ordergradient filtered image in horizontal and vertical directionsis extracted from each color image plane ( I G ) and super-imposed on the color image plane itself, thereby resultingin 12 additional images. Finally, each color image plane isFrangi-filtered [7] to extract the second order edges ( I F ) andsuperimposed on the image itself, generating 12 additionalimages per pallor site image. Using this process, 36 color andedge enhanced images are extrapolated per pallor site image.Using cross-validation method, the 36 extrapolated imagesare ranked to identify the most discriminating image colorand edge enhancement procedure for pallor classification.
1) Image Segmentation:
The first step for model M involves spatial segmentation of the pallor site image intoseveral ROIs. For the eye pallor site images, the sclera andconjunctiva regions while for the tongue pallor site imagesthe inner and outer tongue regions would constitute thedifferent ROIs. For segmentation of eye images, the firststep is detection of the iris, followed by detection of thesurrounding sclera, followed by conjunctiva detection. Thesteps for detecting R iris , R sclera , R conj as the iris, sclera andconjunctiva regions, respectively are shown in (1)-(5). First,the scaled red plane image in [0,1] pixel range is thresholdedto detect dark regions with area greater than 100 pixels onlyin (1). These regions are represented by R . The R iris isthe dark region with the most elliptical shape (i.e., highestratio of major axis length ( φ ) and minor axis lengths ( ψ )in (2). Next, the green plane image within the masked g region is subjected to watershed transformation using circularstructuring element ( se ) of radius 5 in (3). This results inseveral sub-region segmented image W . The iris region isremoved from the image W followed by detection of theremaining sub-regions in W that intersect with the edge of R iris in (4). Since the sclera region lies right outside the iris,the edges of the sclera region and the iris region intersect.Finally, the conjunctiva region is detected as the remainingregions in mask g after removing the iris and sclera regions in(5). For the tongue pallor site images, the masked green planeimage within masked region g is subjected to watershedtransformation, thereby resulting in image W with severalsub-regions R . Next, the outer edge of the tongue is detectedin image E using the ‘Sobel’ filter. The sub-regions in R thatintersect with the outer tongue edge regions represent theouter regions in the tongue ( R outer ). The remaining regionsin the R after removing the R outer regions represent theinner tongue regions ( R inner ). R ← ( I red < . , Area ( R ) > . (1) R iris ← arg R max φ R ψ R . (2) W ← W atershed [ I green ◦ g, se ] . (3) R sclera ← ( W − R iris ) ∩ edge ( R iris ) . (4) R conj = g − [ R iris + R sclera ] . (5) ) Color Planes and Gradient Feature Extraction: Formodel M , 54 features are extracted per image using pixelintensity-features from color and gradient transformed im-ages from various segmented sub-regions in each image asshown in Fig. 2. For the eye images, 27 features are extractedfor the sclera and conjunctiva region each, corresponding tothe max, mean, variance of pixels in the following imageplanes: red, blue, green, hue, saturation, intensity, I Ggreen , I Fgreen . For the tongue images, 27 similar features are ex-tracted for the inner and outer tongue regions, each. (a)(b)Fig. 2. Examples of color and gradient plane images. Top row: I Ggreen magnitude, direction, I Fgreen . Middle row: red, green, blue color planes.Bottom row: hue, saturation, intensity color planes for (a) Eye. (b) Tongueimages, respectively.
3) Classification:
The final step in data models M and M involve classification using a family of data modelsimplemented on the Microsoft Azure Machine LearningStudio (MAMLS) platform for scalability. These classifiers,called Azure-based Generalized flow for Medical ImageClassification (AGMIC) [6], involve grid-search based hyper-parameterization of several data models and selection of theoptimal data model with highest classification accuracy onthe validation data set. Here, automated parametrization of 8sets of data models is performed [6] including support vectormachines, logistic regression, boosted decision tree, decisionforest, decision jungle, neural networks, Poisson regressionand k-nearest neighbors. At the the end of the trainingprocess, one optimal data model with lowest classification error is selected and used for classification of the test datasamples. III. E XPERIMENTS AND R ESULTS
The images from tongue and eye data sets are analyzedseparately. Since the data sets contain 3 classes of datasamples, 2-step hierarchical classification is performed [8]for data separability analysis. In the first hierarchical step, thenormal images are separated form the abnormal ones (class0 vs. class 1, 2) or images with anemia are separated fromthe non-anemic ones (class 1 vs. class 0, 2). In the secondhierarchical step, the remaining class samples are separated,i.e., (class 1 vs. class 2) or (class 0 vs. class 2), respectively.Three categories of experiments are performed to identifythe most discriminating set of intensity-based, spatial, andcolor-based features useful for classification of pallor severitygrade. First, the intensity-based features extracted per imageusing model M are subjected to feature ranking followedby double cross-validation [6] to identify the optimal setof intensity-based features. Second, the color-plane transfor-mations applied in model M are analyzed to identify themost significant spatial and color-planes. Third, the optimalintensity-based features are used for classification in model M and the optimal color planes are used to classify theimages using model M . A. Intensity-based Feature Learning
The 54 intensity-based features extracted per pallor siteimage in model M are ranked using the F-score, MutualInformation and Chi-squared scoring methods [6] and multi-class classification. We observe that for both the eye andtongue data sets, the 27 intensity-based features correspond-ing to the color planes, gradient and Frangi-filtered imagesfrom the conjunctiva region and the inner tongue regions,respectively, are optimal for classification of normal patientsfrom abnormal ones. However, all the 54 intensity-basedfeatures corresponding to the conjunctiva and sclera regionsin the eye and the inner and outer regions in the tongue aresignificant for classification of anemic images from abnormalones. This observation is inline with the domain knowledgeregarding the appearance of the conjunctiva in eye and innertongue regions for identifying normal patients and analysisof all regions in the eye and tongue for further detection ofabnormalities. B. Color-plane based Feature Learning
The 36 color and gradient planes extrapolated per pallorsite image using model M are analyzed for multi-classclassification performances. For the eye data set with 27images, [36x27=972 images] and for the tongue data setwith 56 images, [36x56=2016 images] are subjected toclassification. The rate of correct classification for each colorand gradient plane image is analyzed to identify the mostdiscriminating planes. We observe that for the eye data set,images ( I Fhue ) and ( I Fsat ) result in the maximum classificationaccuracy of 56%. For tongue images, lightness and a-channelplanes ( I GL , I Ga ) achieve maximum classification accuracy of65%. . Classification Performance Analysis For the eye data set, model M with AGMIC flow isimplemented with 27 intensity-based features from conjunc-tiva region for step-1 of hierarchical classification followedby 54 intensity-based features from sclera and conjunctivaregions for step-2 of hierarchical classification, respectively.The classification performance of models M and M onthe eye images are shown in Table I. Here, we observe thatthe M model implemented with decision forest data modelhas the best image classification performance. TABLE IP
ERFORMANCE A NALYSIS OF P ALLOR CLASSIFICATION M ODELS ON E YE I MAGES .Model M1,AGMIC M2,AGMICTask 0/1,2 1/2 0/1,2 1/2PR 0.85 0.57 0.45 0.42RE 0.67 0.8 0.57 0.42Acc 0.86 0.67 0.53 0.74AUC 0.75 0.675 0.41 0.49
For the tongue data set, model M with AGMIC flowis implemented with 27 intensity-based features from in-ner tongue region for step-1 of hierarchical classificationfollowed by 54 intensity-based features from inner andouter tongue regions for step-2 of hierarchical classification,respectively. The classification performance of models M and M on the tongue images are shown in Table II.Here, we observe that the M model implemented withboosted decision trees data model has the best screeningperformances. TABLE IIP
ERFORMANCE A NALYSIS OF P ALLOR CLASSIFICATION M ODELS ON T ONGUE I MAGES .Model M1,AGMIC M2,AGMICTask 1/0,2 0/2 0/1,2 1/2PR 0.982 0.51 0.8 0.77RE 1 0.53 0.81 0.87Acc 0.982 0.61 0.72 0.73AUC 0.83 0.574 0.78 0.67
IV. C
ONCLUSIONS AND D ISCUSSION
In this work, we present a variety of image-based featureextraction, segmentation and data modeling approaches forclassification and screening of anemia-like pallor using fo-cused facial pallor site images. We perform three categoriesof experiments on eye and tongue pallor site images thatare acquired from the public domain. The first category ofexperiments demonstrates that image intensity-based featurescorresponding to some specific spatial ROIs are significantfor separating normal images from abnormal ones that mustbe further analyzed by specialists. Here, we find that theconjunctiva region in the eye and the inner tongue regionsare significant for identification of normal images and ab-normal images from eye and tongue pallor site images, respectively. The second category of experiments detects themost discriminating color and gradient plane-transformedimages that are significant for classification of image-basedpallor. This experiment demonstrates that Frangi-filtered hueand saturation color planes and first-order gradient filteredluminance channel planes are most significant for pallorclassification using eye and tongue images, respectively. Ouranalysis leads to detection of intensity-based features fromconjunctiva region in the hue and saturation color planes su-perimposed with Frangi-filtered edges for optimal separationof normal images from anemic or abnormal images usingthe eye pallor site images. Also, intensity-based featuresfrom the inner tongue regions in the luminance color planessuperimposed with gradient filtered edges are significant forclassification of abnormal images from normal and anemicones using the tongue pallor site images. In the third categoryof experiments, we observe that the image segmentationand classification results in 86% screening accuracy foreye images while color-transformations and gradient filteringleads to 98% screening accuracy for tongue images. Thus,the proposed system is capable of severity screening foranemia using facial pallor site images in under 20 secondsof computation time per image.Future works will be directed towards analysis of ad-ditional data sets acquired under controlled imaging con-ditions. Since the data sets under analysis in this workrepresent a huge variety of imaging condition variabilities,the observations from the experimental analysis are moregeneralizable yet limited in classification capabilities. Futureefforts will be directed towards correlation of the automatedpallor severity grade obtained from the facial pallor siteimages with respect to the actual patient hemoglobin countfor pre-clinical evaluations.R
EFERENCES[1] G. A. Stevens, M. M. Finucane, L. M. De-Regil, C. J. Paciorek,S. R. Flaxman, F. Branca, J. P. Pe˜na-Rosas, Z. A. Bhutta, M. Ezzati,N. I. M. S. Group et al. , “Global, regional, and national trends inhaemoglobin concentration and prevalence of total and severe anaemiain children and pregnant and non-pregnant women for 1995-2011:a systematic analysis of population-representative data,”
The LancetGlobal Health arXiv preprintarXiv:1511.02407 , 2015.[4] S. S. Yalc¸in, S. ¨Unal, F. G¨umr¨uk, and K. Yurdak¨ok, “The validity ofpallor as a clinical sign of anemia in cases with beta-thalassemia,”
TheTurkish journal of pediatrics , vol. 49, no. 4, p. 408, 2007.[5] V. Cherkassky and F. Mullier, “Learning from data,”
John Wiley andsons, New York , 1998.[6] S. Roychowdhury and M. Bihis, “Ag-mic: Azure-based generalized flowfor medical image classification,”
IEEE Access , vol. PP, no. 99, pp. 1–14, 2016.[7] S. Roychowdhury, D. D. Koozekanani, and K. K. Parhi, “Automateddetection of neovascularization for proliferative diabetic retinopathyscreening,” in , Aug 2016, pp.1300–1303.[8] S. Roychowdhury, D. Koozekanani, and K. Parhi, “Dream: Diabeticretinopathy analysis using machine learning,”