High Definition image classification in Geoscience using Machine Learning
HHigh Definition image classification inGeoscience using Machine Learning
Yajun An − − − , Zachary Golden , Tarka Wilcox , and RenzhiCao − − − X ] University of Washington-Tacoma, Tacoma, WA, 98402 Pacific Lutheran University, Tacoma, WA, 98447 Wiss, Janney, Elstner Associates, Inc. Denver, CO, 80235 [email protected]; [email protected]; [email protected]
Abstract.
High Definition (HD) digital photos taken with drones arewidely used in the study of Geoscience. However, blurry images are oftentaken in collected data, and it takes a lot of time and effort to distin-guish clear images from blurry ones. In this work, we apply Machinelearning techniques, such as Support Vector Machine (SVM) and NeuralNetwork (NN) to classify HD images in Geoscience as clear and blurry,and therefore automate data cleaning in Geoscience. We compare theresults of classification based on features abstracted from several math-ematical models. Some of the implementation of our machine learningtool is freely available at: https://github.com/zachgolden/geoai.
Keywords: blur detection · data cleaning · machine learning · computervision Introduction
Blur in Geoscience photos is a very common phenomenon, especially when takenby drones (Blur could come from defocus, camera shake, motion, etc.). Manyphotos are not useful due to blur even for HD photo. It is particularly challengingfor data cleaning in Geoscience, even for a clear image. Other than conventionalblur detection, we also need to be careful with false positive for blurs. Currently,it takes days at a time to clean data by picking out blurry images manually. Inthis work, we develop tools using machine learning techniques for data cleaningin Geoscience.Methods have been proposed for blur detection based on the three strate-gies: Full-reference image quality assessment (FR-IQA) techniques that comparea reference and a test image and predict the perceptual quality of the test imagein terms of a scalar value representing an objective score; No-reference imagequality assessment (NR-IQA) techniques that measure the perceptual qualityof an image without access to the reference image; and Reduced-reference im-age quality assessment (RR-IQA) techniques that provide a solution that liesbetween FR and NR models [3]. a r X i v : . [ ee ss . I V ] S e p Y. An et al.
Mathematical models have been used for those blur detection methods. Hsuand Chen proposed a blur detector using support vector machine with imagegradient model [2]; De and Masilamani proposed a image sharpness measure inthe frequency domain using the Fast Fourier Transform [3]; Tong et al proposedwavelet transforms in blur detection [5]. In this work, we present a survey ofseveral mathematical models for feature analysis and construct several metricsfor measuring blurriness of a given clear image to itself with added noise. Wealso explore machine learning technique for this problem since it has been suc-cessfully and widely applied to solve problems in different fields, including imagerecognition, bioinformatics, voice recognition, drug discovery, etc. [6,7,8,9,10]
The input data in this work is a grayscale image of size M × N , which is rep-resented as M × N pixels while each pixel contains an integer between 0 to255. We follow Hsu and Chen’s work [2] and compute the gradient of an image at eachpixel: let F ( i, j ) denote the pixel of image F at pixel ( i, j ). Then the gradient at( i, j ) is the vector (cid:68) ∂F∂x , ∂F∂y (cid:69) (cid:12)(cid:12)(cid:12)(cid:12) ( i,j ) . We will use the gradient magnitude and the gradient direction as features of image F : G = (cid:115)(cid:18) ∂F∂x (cid:19) + (cid:18) ∂F∂y (cid:19) , θ = arctan (cid:18) ∂F / ∂x∂F / ∂y (cid:19) . (1)In a clear image, shapes should have sharp edges and angles, therefore thedifference between pixels would be “big”, producing a large gradient magnitudeand small angle. Whereas blurry images will give blurry edges and roundedangles. We base our first feature on pixel count and feed the histogram of gradientinformation to the machine: Input data : Image F of size M × N Output feature : clear coefficient α .1. Compute the gradient of image and store as a M × N array.2. Compute the gradient magnitude and gradient direction, then convertthe M × N arrays to two vector of length M N .3. Compute the percentage of pixels with high gradient magnitude (withpreset threshold) over all pixels, we call this the clear coefficient α . igh Definition image classification in Geoscience using Machine Learning 3 α for clear image is magnitudes higher than those for blurry images,and we can always adjust a preset threshold to normalize our output feature. (a) Clear (b) Spin 1 Fig. 1: Clear image, and generated blurry image with radial spin blur extent 1.Radial spin blur comes from the instability of drone imaging.
The image gradient model is based on edge sharpness information. Inspired by[4], we decompose an image array into multiple rank 1 matrices, call eigen-images , via the SVD decomposition. Each eigen-image correspond to a singularvalue: blurred images to larger singular values, and sharp images to smallersingular values. This is because blurry images suppress lose high-frequency in-formation, which is reflected in less significant eigen-images.Consider a grayscale image F of size M × N , performing SVD on F to get F = U ΣV T . (2) Y. An et al. −500 0 500 100015002000250030000.00000.00050.00100.00150.00200.00250.00300.0035
Gradient magnitude of clear −200−150−100−50 0 50 100 150 2000.0000.0010.0020.0030.0040.0050.006
Gradient direction of clear
Fig. 2: Histogram of gradient magnitude and gradient direction for clear image.Clear coefficient α is 0.0111 −500 0 500 100015002000250030000.0000.0020.0040.0060.0080.0100.0120.014 Gradient magnitude of spin1 −200−150−100−50 0 50 100 150 2000.000.010.020.030.040.050.060.07
Gradient direction of spin1
Fig. 3: Histogram of gradient magnitude and gradient direction for spin 1 image.Clear coefficient α is 0.0001. (a) Vertical 10 (b) Spin 10 Fig. 4: Generated vertical shift and radial spin blur images with shift extent 10and spin extent 10. Vertical or horizontal shift blur comes from the constantmovement of drones in geo-survey, and severe radial spin can be from strongwind, for example. igh Definition image classification in Geoscience using Machine Learning 5 −500 0 500 100015002000250030000.0000.0020.0040.0060.0080.0100.0120.0140.0160.018
Gradient magnitude of vertical10 −200−150−100−50 0 50 100 150 2000.0000.0050.0100.0150.0200.0250.0300.0350.0400.045
Gradient direction of vertical10
Fig. 5: Gradient histograms for vertical blur 10. Clear coefficient α is 5e-07 −500 0 500 100015002000250030000.0000.0020.0040.0060.0080.0100.0120.014 Gradient magnitude of spin10 −200−150−100−50 0 50 100 150 2000.000.010.020.030.040.050.060.070.08
Gradient direction of spin10
Fig. 6: Gradient histogram for spin 10. Clear coefficient α is 0The column-row multiplication of U Σ and V T allows writing F as the sum of r (the rank of F ) rank 1 matrices: F = U ΣV T = σ u v T + ... + σ r u r v Tr , (3)here u i , v i denote columns of U and V , σ i the singular values, arranged indescending order. Then image F is written as a weighted sum of eigen-images u i v Ti , with weights σ i .In image compression with SVD, we use low rank approximation with eigen-images: with a preset k , we obtain compressed image F k = σ u v T + ... + σ k u k v Tk of F . The compression omits details of the image, which correspond to omittingsmaller singular values and their corresponding eigen-images. Thus, the fewersmall singular values we have, the less detail an image contains. See Figure 7.We then define a SVD blur degree β , and propose the following algorithm forblur detection: Input : Image F of size M × N Output feature : SVD blur degree β .1. Compute SVD decomposition of the M × N image. Denote the singularvalues by { σ i } ni =1 Y. An et al.(a) Reconstructed with all 3000 singularvalues (b) Reconstructed with the 50 biggest sin-gular values
Fig. 7: Clear image reconstructed with all singular values versus the 50 biggestsingular values. Details were lost with fewer singular values.2. Compute blur degree: β = (cid:80) ki =1 σ i (cid:80) ni =1 σ i , where k is preset by user.3. The blur of image is measured by the blur degree. The higher the degreeis, the more blurry the image is.We performed experiments with images from our geo-survey (sizes around3000 by 4000), and have found when taking k = 300, β for blurry images werehigher than 0.63, whereas for clear we get below 0.51; for k = 50, β for blurryimages were all higher than 0.3, and lower than 0.2 for clear images. Independentof k value, blurry and clear images have a distinct linear range for blur degree. An image is an array of data containing information in the spatial domain.Fourier transform of an image contains information in the frequency domain,which means the rate of change of intensity per pixel. In a region of a grayscalepicture, if the change from white to black (or vice verse) takes many pixels,then the change is slow and corresponds to low frequency; if the process onlytakes a few pixels, then the intensity varies fast, corresponding to high frequency.Sharper image has more high frequency components, but blurry image has lesshigh frequency components. We use this characteristic of images to train themachine to distinguish clear and blurry images.Fourier transform (FT) is an efficient computational method for image pro-cessing, and we use FT histogram vector as training feature [3]:
Input : Image F of size M × N Output feature : FT histogram vector for image and clear estimate γ . igh Definition image classification in Geoscience using Machine Learning 7 Fourier transform of clear (a) FFT of Clear image
Fourier transform of vertical10 (b) FFT for vertical 10 image
Histogram of FFT, for clear image (c) Histogram of FFT of clear image
Histogram of FFT, for vertical10 image (d) Histogram of FFT for vertical 10
Fig. 8: 2D FFT of clear image and vertical 10 image. More higher frequencycomponents in the center in clear image. In the histogram, we see more higherfrequencies in clear image.1. Compute the shifted FT of image as a M × N array, then convert the M × N array to a vector of length M N . Denote by s .2. find the maximum of the centered FT component: a = max ( abs ( s ))3. Count the total number T H of pixels in image whose pixel value >threshold (specified by user). [3] takes threshold = a/ γ .Figure 8 shows the 2D FFT and corresponding histograms for clear andvertical 10 images. The 2D FFT figures show that clear images have more bigfrequencies in the center, and blurry images have fewer. Moreover, the FFT ofvertical shift blur images shows the shift and direction of the shift. As expected,normalized clear estimate γ is the biggest for clear image (0.0427), and as spinextent gets bigger, the clear estimate gets smaller: spin 1 is 0.0075, and spin 10is 0.0017. Although FFT histogram “thinks” vertical blur is clearer than spinblur with a clear coefficient of 0.0032, clear estimates of all blur images are onemagnitude smaller than that of the clear image. We thus conclude this is a validfeature for blur detection in our data. Y. An et al.
With all three feature formulations, our experiments demonstrate that they areeffective in distinguishing blurry images from clear ones. Image gradient modelis image independent, although it depends on the geometric information of theimage. This could be especially useful when studying image from Geoscience.SVD blur detection decomposes images into different levels of details, and mea-sures blur by the percentage of singular values corresponding to non-detailedinformation. FFT studies the frequency of pixels in an image: sharper imagescorresponds to larger frequencies, and more larger frequencies implies clearerimages.In the future, we will incorporate more machine learning techniques to detectblurry images in a large library of HD geo-survey image data with the abovefeature models. We have already done experiments with support vector machine(SVM), and are working on implementation with neural network (NN) and deeplearning technoques.
References
1. Ali, U. & Mahmood, M. Analysis of blur measure operators for single image blursegmentation. Applied Sciences. , 807 (2018)2. Hsu, P. & Chen, B. Blurred image detection and classification.3. De, K. & Masilamani, V. Image sharpness measure for blurred images in frequencydomain. Procedia Engineering. pp. 149–158 (2013)4. Su, Bolan, Shijian Lu, & Chew Lim Tan. Blurred image region detection and classi-fication. In Proceedings of the 19th ACM international conference on Multimedia,1397-1400 (2011)5. Tong, Hanghang, Mingjing Li, Hongjiang Zhang, & Changshui Zhang. Blur de-tection for digital images using wavelet transform. In 2004 IEEE internationalconference on multimedia and expo (ICME)(IEEE Cat. No. 04TH8763), , 17–20.(2004)6. Lecun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature. , 436–444 (2015)7. Lee, J., Jun, S., Cho, Y., Lee, H., Kim, G., Seo, J. & Kim, N. Deep learning in med-ical imaging: general overview. Korean Journal Of Radiology. , 570–584 (2017)8. Hou, J., Wu, T., Cao, R. & Cheng, J. Protein tertiary structure modeling drivenby deep learning and contact distance prediction in CASP13. Proteins: Structure,Function, And Bioinformatics. , 1165–1178 (2019)9. Staples, M., Chan, L., Si, D., Johnson, K., Whyte, C. & Cao, R. Artificial Intelligencefor Bioinformatics: Applications in Protein Folding Prediction.10. Stephenson, N., Shane, E., Chase, J., , J., Ries, D., Justice, N., Zhang, J., Chan, L.& Cao, R. Survey of machine learning techniques in drug discovery. Current DrugMetabolism. , 185–193 (2019)11. Conover, M., Staples, M., Si, D., Sun, M. & Cao, R. AngularQA: proteinmodel quality assessment with LSTM networks. Computational And MathematicalBiophysics.7