Hyperspectral Imaging Technology and Transfer Learning Utilized in Identification Haploid Maize Seeds
HHyperspectral Imaging Technology and Transfer LearningUtilized in Identification Haploid Maize Seeds
Wen-Xuan Liao ∗ , Xuan-Yu Wang ∗ , Dong An ∗ , Yao-Guang Wei ∗∗ College of Information and Electriacl Engineering China Agricultural University,Beijing 100083,China { vane,wangxuanyu,andong,weiyaoguang } @cau.edu.cn Abstract —It is extremely important to correctly identify thecultivars of maize seeds in the breeding process of maize.In this paper, the transfer learning as a method of deeplearning is adopted to establish a model by combining withthe hyperspectral imaging technology. The haploid seeds can berecognized from large amount of diploid maize ones with greataccuracy through the model. First, the information of maizeseeds on each wave band is collected using the hyperspectralimaging technology, and then the recognition model is builton VGG-19 network, which is pre-trained by large-scale com-puter vision database (Image-Net). The correct identificationrate of model utilizing seed spectral images containing 256wave bands (862.5-1704.2nm) reaches 96.32%, and the correctidentification rate of the model utilizing the seed spectralimages containing single-band reaches 95.75%. The experi-mental results show that, CNN model which is pre-trained byvisible light image database can be applied to the near-infraredhyperspectral imaging-based identification of maize seeds, andhigh accurate identification rate can be achieved. Meanwhile,when there is small amount of data samples, it can still realizehigh recognition by using transfer learning. The model notonly meets the requirements of breeding recognition, but alsogreatly reduce the cost occurred in sample collection.
1. Introduction
Seed is the foundation of agricultural engineering.Continuously growing population and constantly changingglobal climate determine that there is a need for moreeffective breeding strategies for new species, especially tomaximumly improve speed and accuracy of the identifica-tion and sorting of target seeds in the breeding process.Mixture of different types of seeds will directly result in thereduction of purity in breeding experiment, which furtherleads to a reduction in the yield of crops. Maize is oneof the most widely planted commercial crops in China,the maize haploid breeding technique based on biologicalinduction has become the key to breed new species of maize.Relevant haploid breeding technique helps effectively reducethe breeding time [1], save costs of labor and materialsand it is of great importance to germplasm improvement.However, the occurrence rate of maize haploid under naturalconditions is about 1%, while the occurrence rate rises to8%-15% after artificial induction([2],[3],[4]), that is, under the breeding environments, there are just 8%-15% of seedsbelonging to haploid, while the rest of them are of diploid.Therefore, it has always been the key point to realize rapid,accurate and non-destructive identification and sorting hap-loid maize seeds.The widely adopted seed identification methods atpresent include: morphological method [5], protein elec-trophoresis [11], DNA molecular marker technology tests[12], genetic marker method [13]. However the first threemethods are expensive and time-consuming and requirespracticed operators [14]. The genetic marker method isnot a good choice for automated machine vision sorting[15]. To collect information without damaging the seeds,the possible technological means is mainly lossless optics,such as machine vision technology [16]. These technologiesdisplay limited ability to collect information. Because theinformation that the machine vision system mainly col-lects mainly includes the information of external featuresof seeds, such as color, texture, contour, etc., it seemsless effective compared with the near infrared spectroscopytechnology with which the spectral features relevant to thechemical contents that the seeds contain are collected [17].Hyperspectral imaging technology refers that the nearinfrared imaging is applied at every wavelength point bycombining the information of near infrared spectroscopywith the information of near-infrared images, that is, it canbe used to generalize the space information and the nearinfrared spectroscopy information of collected samples. Thenear infrared spectroscopy is very sensitive to hydrocarbons,hydroxyls, and amines among the organic matters, and cor-respondingly, information such as proteins, starch, moisture,and fat etc. that the sample contains can be reflected [24] [6].In recent years, high-spectrum technology makes successin the field of food quality inspection [7], and it is alsoadopted in the task of seed detection and sorting. [8][9].Based on above, the hyperspectral imaging technology isfeasible to collect the infoemation on the diversity of haploidand diploid maize seeds organics or spatial shapes.CNN model displays excellent performance in computervision, and it has been successfully applied in the field ofimage recognition such as face recognition etc. However, thelarge-capacity model whose training starts from randomlyinitialized weights requires a large number of data sets,while it is impossible to provide such a large number of datasets in the maize seeds identification study. These problems a r X i v : . [ c s . C V ] M a y an be solved by transfer learning in the field of computervision and machine learning. Compared with the modelsthat use nothing but small data set with randomly initial-ized weights, it is proved that this new adjustment processcan improve the generalization of the model [10], and theconvergence starts when there is just a few of iterationtimes. Currently, a lot of researchers transfer CNN pre-trained in Image-Net data set to other vision identificationtasks. For example, Matthew et al. [21] applied it to Caltech-101 and Caltech-256 image classifications, and Maxime etal.[22] applied it to Pascal VOC 2007 and 2012 data setimage classifications, and Gustavo et al. [23] applied it tomedical X-ray image classification. Therefore, CNN modelpre-trained through Image-Net not only can be transferredto other visible light data set, but also to the data setof images of other wave bands. At present, there is noresearch touching upon the possibility of transferring CNNmodel pre-trained through Image-Net to the data set ofhyperspectral images. However, based on investigation, webelieve it is feasible. In this paper, the network is initializedusing the pre-trained deep learning model available frompublic sources, and re-training is conducted using relativelysmaller training set so that the classification can be realizedbased on the hyperspectral image of maize seeds.In this article, the main contributions are as follows:1.In our study, we design the haploid maize seeds iden-tification model consisting transfer learning technology andhyperspectral imaging technology that samples maize seedsembryo surface as images data.2.The correct recognition rate of maize haploid seedsreaches 96.32% if the model built on the high-spectrum im-ages of maize seeds containing 256 wave bands is adopted.The average correct recognition rate of maize seeds is94.08% and its highest rate reaches 95.85% if the modelbuilt on single-band and high-spectrum images of maizeseeds is adopted. Among them, the optimal wave bandis identical with the correct recognition rate of full-waveband hyperspectral image. Sampling and modeling withsingle-band samples data can effectively reduce the costof breeding sorting, despite expensive full-band samples.Hence, the acquisition cost can be reduced and the speedwould be improved through utilizing optical filter coveringthe full-wave band near-infrared camera.3.Our study can expend not only to identification ofother agricultural products seeds, but also to other fields.
2. Materials and Method
Samples adopted in the experiment are species calledZhengdan 958 Maize bred through induction of high-oilcrossbreed carrying R1-nj genetic marker, which is devel-oped by Maize Improvement Center of China AgriculturalUniversity. Among which, there are 100 haploid seeds and100 diploid seeds. After each maize seed is dried, dehy-drated and numbered, the samples are stored at 5 ◦ C, and
Figure 1. Samples of haploid and diploid maize seeds (left) and theirhyperspectral images taken at 962.7, 1132.3, 1364.4nm wavebands (right). TABLE 1. THE NUMBER, VARIETTY, AND MEASURMENT OFSAMPLE.
Acquisition mode Variety Haploid Diploid
HIS Zhengdan-958 100 100 their images are obtained using near-infrared hyperspectralimging technology. Considering great difference betweenembryo surface and non-embryo surface, the image of em-bryo surface of each seed is adopted in the experiment, thatis, the embryos surface of each seed is positioned towardsthe light during acquisition. To maximally reduce the influ-ence of instrumental parameter drifts on the measurementresults, the alternate sampling is performed between haploidseed and decreases during the actual sampling process.In theexperiment, we will process the collected data as follows: (a)different column matrix is set for labeling the hyperspectralimage of 100 haploid seed of Zhengdan 958 and the hyper-spectral image of 100 Diploid seed; (b) seeds are randomlyselected from haploid and diploid, and the images of these80 seeds are taken as the test set, while the remaining imagesare put into the network training as a training set.
The hyperspectral images were collected by the push-broom GaiaSorter hyperspectral system. The hyperspectralimaging system is mainly comprised of four components,that is, uniform light source, spectral camera, mobilecontrol platform and computer. The uniform light sourceis comprised of two sets of bromine tungsten lamp,while the light source gives out uniform light throughthermal radiation. As for the spectral camera, Image--N17Espectrum near-infrared-enhanced hyperspectral camera(Zolix Instruments Co., Ltd.) is adopted, which integratesthe imaging spectrometer of Imspector series and theInGaAs CCD camera, and cameras spectral region rangesfrom 862.9 to 1704.2nm containing 256 wave bands, whichcovers near-infrared wave band; its spectral resolution is5nm, and pixel 320x256, and slit width 30m. The systemmobile control platform is controlled by stepping motor,and the image acquisition is performed using Spectra View,an image acquisition software. If it is ensured that theimage is not distorted, the platform movement speed isset to 0.27 cm/s, and the exposure time is 35 ms. Eachimage acquired is a three-dimensional image (x, y, λ ),and the collected image was a (320x2000x256) imageube. In order to reduce interference from the externalenvironment, the images are captured inside the camerabellows. As for the measurement errors of hyperspectralimage caused by light source fluctuation and dark currentis corrected by formula1based on black and white reference. R cur = R sam − R dar R whi − R dar (1) R cur Denotes calibrated sample images, and R sam de-notes original sample images, and R dar denotes dark ref-erence image, and R whi denotes white reference image.The dark reference image can be captured by covering thecamera view with lens cap; the white reference image canbe captured by replacing measurement object with the purewhite board, and completely covering one frame of cameraafter being lighted. All the calibrated sample images will beapplied to the experimental analysis thereafter. Calibrated sample images still contain the backgroundinformation during acquisition. To separate the true infor-mation of the seed from the background, we have adoptedself-adaptive threshold segmentation and masking (Huanget al. 2016) to extract the region of interests (ROIs), andthe procedures are follows: (a) within the region of interests(ROIs) extracted on 60 (1064.8nm), the contrast ratio ofseeds to the background is the highest. The maximum valueof background is selected as the threshold value for imagebinarization; (b) the peripheric coordinate of each sample isobtained to form a binary image, and the rectangular areaof each sample is determined. Binary masks are generatedthrough rectangular regions to obtain ROIs of 256 bands; (c)the true information of the seed is obtained by multiplyingeach seeds ROI with its corresponding binary image, so thatthe interference of background information can be removed.The input layer of VGG-19 network requires a resolutionratio of the image of 224x224, while the resolution ratio ofthe pre-treated image of seed is 141x111, which does notmeet the network requirements. We adopted the method ofbackground filling to modify the seed image into 224x224(Figure 3), so it satisfies the requirements of the input layerof the network, while it does not change the informationcontained in the image.The software used to process the seed sample images isMATLAB 2016a (USA, MathWorks Company).
In the experiment, convolutional neural networks VGG-19[18] is utilized to extract the features of the seed imageand identify haploid. The VGG network was proposed bythe Oxford Vision Group and it won the championship inthe 2014 ILSVR competition.VGG-19 is comprised of 19 layers of network, contain-ing 16 convolutional layer and 3 full connection layers.
TABLE 2. MODELS FOR IMAGES OF DIFFERENT RANGE OFWAVE LENGTH
Range of wave length(nm) Ave acc(%) Ave consumed time(s)
In each convolutional layer, ReLU is taken as activationfunction. In full connection layer 1 and full connection layer2, dropout is adopted to prevent over-fitting; finally, softmaxis adopted as loss function. The first 18 layers are used forfeature extraction and the last layer is used for classification.The network uses nothing but small convolution kernel at3x3, which is the smallest size capable of capturing alldirections and central concept. Multiple convolution layersat 3x3 display greater nonlinearity than one convolutionallayer of large size, making the function more deterministicand greatly reducing the number of parameters. Due tonetwork depth, and because small size of convolution kernelhelps realize implicit regularization, VGG network starts toconverge when there is just a few of iteration times. There-fore, we have selected VGG-19 network in accomplishingthe identification task of maize seeds haploid.
First, the network goes through sufficient pre-training onImage-Net data set, and the last full connection layer of net-work is 1000-Dimensional tensor. Although the pre-trainednetwork isn’t capable of identifying the maize haploid,it provides excellent initial value to haploid identificationnetwork. A good initial value is quite critical to the networktraining. We have modified the last full connection layer ofnetwork (Figure 2), and the output is set as 2, correspondingto haploid and diploid. The network is re-trained usingthe maize images, and the weight of pre-trained networkis fine-tuned, after which the network can be applied tothe identification of maize haploid. In the training process,stochastic gradient descent (SGD) parameter optimization isadopted, and the global learning rate is 0.0001. The learningrate of last full connection layer is 0.002, and MiniBatchsizeis 90, and MaxEpochs is 500. If the average accuracy ofthe first 50 iterations reaches 98%, then the training isterminated early. Cross-validation is performed to work outmean, maximum value and standard deviation of correctrecognition rates for five times. The training process of themodel contains random process, and the highest value refersto the higher results that the model can achieve in multipletraining processes. The experimental process is shown inFigure 3.
3. Results and Discussion
The experimental result of modeling of maize hyper-spectral images taken at 256bands (862.9-1704.2nm) showsthat average accurate identification reaches 96.32%. The igure 2. The structure of the modified transfer-learning neutral network improved from Image-Net to classifying maize seeds.Figure 3. Main steps of identification of haploid maize seeds based improved pre-trained neutral network.Figure 4. The training accuracy convergence of network on train set, whichstarted converging at 60 times iteration and finally stop at 113 times.Figure 5. The change of average and maximum accuracies of 25 groups,each of which includes 10 wavebands, during five times cross-validations. network convergence speed is shown in Figure 4. Thenetwork converges after about 60 times of iteration andthe training is stopped after 113 times of iteration is fin-ished. The maize haploid can be identified at high accu-
Figure 6. The change of average and maximum accuracies of each singlewavebands during five times cross-validations. racy using the model, which meets the actual applicationrequirements. The model training features fast convergencespeed, short training time and reduced time cost. Figure5 shows the experimental result of modeling for 10-waveband images. In the experiment, the hyperspectral imagesof maize seeds are divided into 25 groups, in which 10wave bands are classified into one group: 1-10,11-20,21-30, . . . , 241-250. As seen from the Figure 5, the av-erage accurate identification rate reaches 90% and above;the highest correct identification rate reaches 93.75% andabove. The average accurate identification rate within theinterval of 221-230 (1593.5-1622.1nm) reaches 95.85%, andthe highest correct identification rate within the intervalof 81-90 (1135.6-1165.8nm) reaches 99.5%. The standarddeviation of average accuracy among different groups is0.009, and the standard deviation of maximum value be-tween different groups is 0.015, with relatively small inter-group difference. Figure 6 shows the experimental result ofmodeling of single-band hyperspectral image of maize seedsn the experiment, one CNN model is built using single-bandimages every five wave bands, and a total of 51 modelsare built. The average accurate recognition rate of modelsreaches 90% and above, and the highest correct recognitionrates exceed 93.75%, and the average accurate recognitionrate of the 115th wave band (1249.1nm) reaches 95.75%,and the highest correct recognition rate of the 220th waveband (1590.4nm) reaches 100%. The standard deviation ofaverage accuracy among different groups is 0.012, and thestandard deviation of maximum value between differentgroups is 0.015, with relatively small difference betweendifferent wave bands. In three groups of experiments, theaverage standard deviation shown by cross-validation is0.018, indicating that the model is stable. Table 2 is theresults comparison of above three modeling experiments.As can be seen from the table, the time consumed bysingle-band image modeling is the shortest. when the dataused in modeling is reduced, the correct recognition rate ofmodel doesn’t decrease significantly. Our research resultsshowed that, after pre-trained VGG-19 network is fine-tuned, the constructed new CNN model can be applied to theidentification of hyperspectral image of maize seeds. Amongexisting researches, through generalization, we can find that,most of models used to identify and sort out seeds arecomplicated system combination formed by nesting multiplemodels. For example, Zhang et al.[19] has adopted a kind ofPCA-GLCM-LS-SVM iteration combination method whensorting out the seeds. Compared with seeds soring modelsin the past, the model adopted in the paper doesnt requireother complex algorithms in performing special feature ex-traction works, as CNN can learn relevant features fromthe data and classify them. Through the transfer learning,the purpose of pre-trained networks can be expanded, andthe problems of sparse sample data can be solved. Thegeneralization performance can be significantly improvedwith a small number of marked samples. This feature isreflected in the single-band experiment. From the experi-ment, we can find that, the difference between the highestaverage recognition rate in the single-band experiment andthe average recognition rate in full-wave band experimentis less than 1%, so the reduction of the training data doesnot reduce the sorting accuracy of the model. Comparedwith full-wave band model, the single-band model has toface an additional problem, wave band selection. In previousexperiments, the important bands are selected according towave band or Characteristics to improve the accuracy andspeed of identification and sorting [20]. The single-bandexperimental results show that, the standard deviation ofaverage accurate recognition rates among different wavebands is 0.012, and the standard deviation of maximumrecognition rates is 0.015. The wave band doesnt have mucheffect on the model. Most of the models constructed by thewave band can satisfy the requirement of sorting tasks. Noother algorithm is needed in selecting the featured waveband.
4. Conclusions and Future Work
In the paper, we have built the soring model for identifi-cation haploid maize seeds adopting technology of transferlearning and pre-trained VGG-19 network fine-tuned bymaize hyperspectral images. The haploid maize seeds can beidentified from the maize samples mixed with large amountof diploid seeds with high identification rate. Within therange of 862.9-1704.2nm including 256 bands, hyperspectralimaging system collects feature information of haploid anddiploid maize seeds. The model is applicable to both imagestaken at full-wave bands and images taken at single-bandimages, the single-band model is featured by low acquisitioncost and fast speed. From the perspective of application,it can expand to both accurate maize breeding works andlarge-scale maize seeds sorting in the seeds breeding. Ourfuture works include the following (a) Extend the modelsapplication to other application areas. (b) Test the robustnessof the model by applying it in different complicated envi-ronment (such as complicated background and illuminationcondition) (c) Introduce the unsupervised method into themodel frame, and improve the autonomous learning abilityof features of different categories. Finally, our goal is tointegrate our method into the work system that can be usedin the site, and to realize automatized operation.
References [1] A. Murigneux, D. Barloy, P. Leroy, and M. Beckert,“Molecular andmorphological evaluation of doubled haploid lines in maize. 1. Homo-geneity within DH lines,”
Theoretical and applied genetics , vol. 86, no.7, pp. 837-842, 1993.[2] S. T. Chalyk, and V. A. Rotarenco, “The use of matroclinous maizehaploids for recurrent selection,”
Russian Journal of Genetics , vol. 37,no. 12, pp.1382-1387, 2001.[3] S. J. Chen, and T. M. Song, “Identification haploid with high oil xeniaeffect in maize,”
Acta Agronomica Sinica , vol. 29, no. 4, pp. 587-590,2003.[4] V. Prigge, C. Snchez, B. S. Dhillon, W. Schipprack, J. L. Araus, M.Bnziger, and A. E. Melchinger, “Doubled haploids in tropical maize: I.Effects of inducers and source germplasm on in vivo haploid inductionrates,”
Crop science , vol. 51, no. 4, pp. 1498-1506, 2011.[5] J. J. G Snchez, M. M. Goodman, and J. O. Rawlings, “Appropriatecharacters for racial classification in maize,”
Economic Botany , vol.47, no. 1, pp. 44-59.1993.[6] L. Pan, R. Lu, Q. Zhu, J. M. McGrath, and K. Tu, “Measurementof moisture, soluble solids, sucrose content and mechanical propertiesin sugar beet using portable visible and near-infrared spectroscopy,”
Postharvest Biology and Technology , vol. 102, pp. 42-50, 2015.[7] A. A. Gowen, C. O’Donnell, P. J. Cullen, G. Downey, and J. M. Frias,“Hyperspectral imaging-an emerging process analytical tool for foodquality and safety control,”
Trends in Food Science & Technology ,vol.18,no. 12, pp. 590-598, 2007.[8] W. Kong, C. Zhang, F. Liu, P. Nie, and Y. He, “Rice seed cultivaridentification using near-infrared hyperspectral imaging and multivari-ate data analysis,” sensors , vol.13, no.7, pp.8916-8927, 2013.[9] M. Huang, J. Tang, B. Yang, and Q. Zhu, “Classification of maize seedsof different years based on hyperspectral imaging and model updating,”
Computers and Electronics in Agriculture , vol. 122, pp.139-145,2016.[10] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable arefeatures in deep neural networks?”,
In Advances in neural informationprocessing systems , pp. 3320-3328, 2014.11] A. JUGRAN, I. D. BHATT, and R. S. RAWAL, “Characterization ofagro-diversity by seed storage protein electrophoresis: Focus on ricegermplasm from Uttarakhand Himalaya,”
India. Rice Science , vol. 17,no. 2, pp. 122-128, 2010.[12] S. Ye, Y. Wang, D. Huang, J. Li, Y. Gong, L. Xu, and L. Liu, “Geneticpurity testing of F1 hybrid seed with molecular markers in cabbage(Brassica oleracea var. capitata),”
Scientia Horticulturae , vol. 155, pp.92-96, 2013.[13] H. Wang, J. Liu, X. Xu, Q. Huang, S. Chen, P. Yang, et al, “Fully-automated high-throughput NMR system for screening of haploidkernels of maize (Corn) by measurement of oil content,”
PloS one ,vol. 11, no. 7, e0159444, 2016.[14] S. Mahesh, D. S. Jayas, J. Paliwal, and N. D. G. White, “Hyperspec-tral imaging to classify and monitor quality of agricultural materials,”
Journal of Stored Products Research , vol.61, pp. 17-26, 2015.[15] A. E. Melchinger, W.Schipparck, T. Wrschum, S. Chen, and F.Technow, “Rapid and accurate identification of in vivo-induced haploidseeds based on oil content in maize,”
Scientific reports , vol. 3, pp. 2129,2013.[16] M. H. Olesen, N. Shetty, R. Gislum, and B. Boelt, “Classificationof viable and non-viable spinach (Spinacia oleracea L.) seeds bysingle seed near infrared spectroscopy and extended canonical variatesanalysis,”
Journal of Near Infrared Spectroscopy , vol. 19, no. 3, pp.171-180, 2011.[17] D. Wu, P. Nie, Y. He, Z. Wang, and H. Wu, “Spectral multivariable se-lection and calibration in visible-shortwave near-infrared spectroscopyfor non-destructive protein assessment of spirulina microalga powder,”
International journal of food properties , vol. 16, no. 5, pp.1002-1015,2013.[18] S. Zhong, K. Li, and R. Feng, “Deep convolutional hamming rankingnetwork for large scale image retrieval,”
Proceedings of the 11th WorldCongress on Intelligent Control and Automation , pp. 1018-1023, 2014.[19] X. Zhang, F. Liu, Y. He, and X. Li, “Application of hyperspectralimaging and chemometric calibrations for variety discrimination ofmaize seeds,”
Sensors , vol.12, no. 12, pp. 17234-17246, 2012.[20] H. Cen, R. Lu, D. P. Ariana, and F. Mendoza, “Hyperspectral imaging-based classification and wavebands selection for internal defect detec-tion of pickling cucumbers,”
Food and bioprocess technology , vol. 7,no. 6, pp. 1689-1700,2014.[21] M. D. Zeiler, R. Fergus, “Visualizing and understanding convolutionalnetworks,”
Proceedings of European conference on computer vision ,2014.[22] M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Learning and trans-ferring mid-level image representations using convolutional neuralnetworks,” Proceedings of Computer Vision and Pattern Recognition,pp. 1717-1724, 2014.[23] G. Carneiro, J. Nascimento, and A. P. Bradley, “Unregistered mul-tiview mammogram analysis with pre-trained deep learning models,”