PIRM2018 Challenge on Spectral Image Super-Resolution: Dataset and Study
Mehrdad Shoeiby, Antonio Robles-Kelly, Ran Wei, Radu Timofte
PPIRM2018 Challenge on Spectral ImageSuper-Resolution: Dataset and Study
Mehrdad Shoeiby , Antonio Robles-Kelly , Ran Wei , Radu Timofte DATA61 - CSIRO, Black Mountain Laboratories, ACT 2601, Australia Faculty of Sci., Eng. and Built Env., Deakin University, VIC 3216, Australia Computer Vision Laboratory, D-ITET, ETH Zurich, Switzerland [email protected]@[email protected]
Abstract.
This paper introduces a newly collected and novel dataset(StereoMSI) for example-based single and colour-guided spectral imagesuper-resolution. The dataset was first released and promoted duringthe PIRM2018 spectral image super-resolution challenge. To the best ofour knowledge, the dataset is the first of its kind, comprising 350 regis-tered colour-spectral image pairs. The dataset has been used for the twotracks of the challenge and, for each of these, we have provided a splitinto training, validation and testing. This arrangement is a result of thechallenge structure and phases, with the first track focusing on example-based spectral image super-resolution and the second one aiming at ex-ploiting the registered stereo colour imagery to improve the resolution ofthe spectral images. Each of the tracks and splits has been selected to beconsistent across a number of image quality metrics. The dataset is quitegeneral in nature and can be used for a wide variety of applications inaddition to the development of spectral image super-resolution methods.
Keywords:
Super-resolution, Hyperspectral, Multispectral, RGB, Stereo
Imaging spectroscopy devices can capture an information-rich representation ofthe scene comprised by tens or hundreds of wavelength-indexed bands. In con-trast with their trichromatic (colour) counterparts, these images are composedof as many channels, each of these corresponding to a particular narrow-bandsegment of the electromagnetic spectrum [1]. Thus, imaging spectroscopy hasnumerous applications in areas such as remote sensing [2,3], disease diagnosisand image-guided surgery [4], food monitoring and safety [5], agriculture [6],archaeological conservation [7], astronomy [8] and face recognition [9].Recent advances in imaging spectroscopy have seen the development of sen-sors where the spectral filters are fully integrated into the complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) detectors. Theseare multi-spectral imaging devices which are single-shot and offer numerous ad-vantages in terms of speed of acquisition and form-factor [10,11]. However, one a r X i v : . [ c s . C V ] M a y Mehrdad Shoeiby et al. of the main drawbacks of these multispectral systems is the low raw spatialresolution per wavelength-indexed band in the image. Hence, super-resolvingspectral images is crucial to achieving a much improved spatial resolution inthese devices.Note that, during recent years, there has been a steady improvement in theperformance of example-based single image SR methods [12,13,14,15]. This ispartly due to the wide availability of various benchmark datasets for developmentand comparison. For example, the dataset introduced by Timofte et al. [16,17],[18,19,20,21], Urban100 [22], and DIV2K [23] are all widely available.Similar to RGB or grey-scale super-resolution, recently example-based tech-niques for spectral image super-resolution have started to appear in the lit-erature [24]. However, in contrast to their RGB and grey-scale counterparts,multispectral/hyperspectral datasets suitable for the development of single im-age super-resolution are not as abundant or easily accessible. For example, theCNN-based method in [24] was developed by putting together three differenthyperspectral datasets. The first of these, the CAVE [25] consists of only 35hyperspectral and RGB pairs gathered in a laboratory setting and controlledlighting using a camera with tunable liquid crystal filters. Similarly, the seconddataset from Harvard [26] contains fifty hyperspectral images captured with atime-multiplexed 31-channel camera with an integrated liquid crystal tunablefilter. The third dataset is that in [27], which includes 25 hyperspectral imagesof outdoor urban and rural scenes also captured using a tunable liquid-crystal fil-ter. Probably the largest spectral dataset to date with more than 250 31-channelspectral images is the one introduced with the NTIRE 2018 challenge on spectralreconstruction from RGB images [28].Moreover, while the topic of spectral image super-resolution utilizing colourimages, i.e. , pan-sharpening, has been extensively studied [29,30,31] so as to de-velop efficient example-based super-resolution methods, stereo registered colour-spectral datasets are limited to small number of hyperspectral images. One ofthe very few examples is that of the datasets in [32], where the authors intro-duced a stereo RGB and near infrared (NIR) dataset of 477 images and proposea multispectral SIFT (MSIFT) method to register the images. However, thedataset is promoted in the context of scene recognition. In addition, the NIRimages are comprised of only one wavelength-indexed band. Similarly, in [33],the authors introduce an RGB-NIR image dataset of approximately 13 hoursvideo with only one band dedicated to NIR images. The dataset was gatheredin an urban setting by mounting the cameras on a vehicle.In this paper we introduce a novel dataset of colour-multispectral imageswhich we name StereoMSI. Unlike the above two RGB-NIR datasets, the datasetwas primarily developed for the PIRM2018 spectral SR challenge [34] and com-prised 350 registered stereo RGB-spectral image pairs. The StereoMSI datasetis hence large enough to help develop deep learning spectral super-resolutionmethods. Moreover, it is, to the best of our knowledge, the first of its kind. As Refer to https://pirm2018.org/ for the spectral SR challenge and the datasetdownload links.IRM2018 Challenge on Spectral Image Super-Resolution: Dataset 3 a result, the paper is organised as follows. We commence by introducing thedataset. We then present a number of image quality metrics over the datasetand the proposed splits for training, validation and testing. Then we presenta brief review of the challenge and elaborate upon the results obtained by itsparticipants. Finally, we discuss other potential applications of the dataset andconclude on the developments presented here.
Fig. 1.
A sample image from the StereoMSI dataset. Here we show the RGB imageand the 14 wavelengths channels of the multispectral camera indicated by λ i , i = { , , . . . , } . All wavelengths are in nm and, for the sake of better visualisation, wehave gamma-corrected the 14 channels by setting γ = 0 . As mentioned above, here we propose the StereoMSI dataset. The dataset is anovel RGB-spectral stereo image dataset for benchmarking example-based singlespectral image and example-based RGB-guided spectral image super-resolutionmethods. The dataset is developed for research purposes only.
The 350 stereo pair images were collected from a diverse range of scenery in thecity of Canberra, the capital of Australia. The nature of the images ranges fromopen industrial to office environments and from deserts to rainforests. In Figures2 and 3 we display validation images for the former and latter, respectively.It is worth noting that, during acquisition time, we paid particular attentionto the exposure time and image quality as the stereo pairs were captured usingdifferent cameras. One is an RGB XiQ camera model MQ022CG-CM and theother is a XiQ multispectral camera model MQ022HG-IM-SM4x4 covering theinterval [470 − nm ] in the visible spectral range. Mehrdad Shoeiby et al.
Fig. 2.
Validation images for the Track 1 of the PIRM2018 challenge. Each of thepanels corresponds to the normalised spectral power of one of the validation images, i.e. the norm of the spectra per-pixel normalised to unit maximum over the image.
Fig. 3.
Validation images of Track 2 of the PIRM2018 challenge. In the left-hand andthird columns we show the normalised spectral power of the spectral imagery, whereasthe second and third columns show their registered RGB image pairs.IRM2018 Challenge on Spectral Image Super-Resolution: Dataset 5
Raw RGB pixel Raw spectral pixel
Fig. 4.
Illustration of raw pixels for RGB and spectral cameras. The RGB camera usedto acquire the images of our dataset has × × . nm , 477 . nm , 500 . nm , 489 . nm , 553 . nm , 562 . nm , 590 . nm , 577 . nm ,599 . nm , 612 . nm , 615 .
9, 617 . nm , 510 . nm , 523 . nm , 548 . nm , 537 . nm . The original spectral images were processed and cropped to the resolution480 ×
240 so as to allow the stereo RGB images to be resized to a resolution2 times larger in each axis, that is 960 × × × Table 1.
Summarised dataset and camera properties
Image properties StereoMSI Track1 Track2 TestingWhole Training Validation Whole Training Validation
Number of images 350 240 200 20 130 100 10 20
Spectral RGBLR × × × × Image resolution (80 × × × × × × Mehrdad Shoeiby et al.
After collecting the StereoMSI 350 images, the two invalid wavelength-indexedbands on the IMEC sensor were removed. We then registered the images usingFlownet2.0 [35] and used MATLAB’s imresize function to obtain lower resolu-tion versions of each image by downscaling them by factors of × × To quantitatively assess our StereoMSI dataset, and to provide a baseline for fu-ture benchmarking, we have performed image upsampling by applying a bicubickernel. Python’s imresize function from the scikit-image toolbox was used toperform bicubic upsampling. With the upsampled images in hand, we have thencomputed a number of image quality metrics so as to compare the performanceof current and future example-based spectral super-resolution algorithms. Tothis end, we up-sampled the lower-resolution images in the dataset by × × For more information on the imresize function, go to https://scikit-image.org/ IRM2018 Challenge on Spectral Image Super-Resolution: Dataset 7
Table 2.
Mean and standard deviation (in parenthesis) for the evaluation metrics underconsideration for each of the two down sampling factors, i.e. , × ×
3, for the wholedataset and the testing split used for both tracks of the PIRM2018 Example-basedSpectral Image Super-resolution challenge.Dataset Downsampling MRAE SID APPSA MSE PSNR SSIMsplit FactorStereoMSI × × × × Table 3.
Mean and standard deviation (in parenthesis) for the evaluation metricsunder consideration for each of the two down sampling factors, i.e. , × ×
3, forthe training and validation splits used in the Track 1 of the PIRM2018 Example-based Spectral Image Super-resolution challenge and the full set of images (the testing,training and validation splits combined).Dataset Downsampling MRAE SID APPSA MSE PSNR SSIMsplit FactorFull set × × × × × × Table 4.
Mean and standard deviation (in parenthesis) for the evaluation metricsunder consideration for each of the two down sampling factors, i.e. , × ×
3, forthe training and validation splits used in the Track 2 of the PIRM2018 Example-based Spectral Image Super-resolution challenge and the full set of images (the testing,training and validation splits combined).Dataset Downsampling MRAE SID APPSA MSE PSNR SSIMsplit FactorFull dataset × × × × × × The PIRM2018 challenge has a twofold motivation. Firstly, the notion that, byusing machine learning techniques, single image SR systems can be trained toobtain reliable multispectral super-resolved images at testing. Secondly, that byexploiting the higher resolution of the RGB images registered onto the spectralimages, the performance of the algorithms can be further improved.Track 1 focuses on to the problem of super-resolving the spatial resolutionof spectral images given training image pairs, whereby one of these is an LRand the other one is an HR image, i.e. the ground truth reference image. Theaim is hence to obtain × × IRM2018 Challenge on Spectral Image Super-Resolution: Dataset 9
Table 5.
Mean and standard deviation (in parenthesis) for the evaluation metrics underconsideration for the winners (IVRL Prime [36], and VIDAR [37]) of both tracks of thePIRM2018 Example-based Spectral Image Super-resolution challenge. For the sake ofreference, we also show the results yielded by up-sampling the LR ( ×
3) testing imagesusing a bicubic kernel.Team Track MRAE SID APPSA MSE PSNR SSIMIVRL Prime 1 0.07 0.00006 0.06 1246673 36.7 0.82VIDAR 1 0.11 0.00018 0.08 3414849 32.2 0.62IVRL Prime 2 0.07 0.00005 0.05 852268 38.2 0.86VIDAR 2 0.09 0.00011 0.08 1940939 34.5 0.75Bicubic upsampling ( ×
3) 0.21 0.00035 0.11 6353052 29.3 0.47 x2 downsampled x2 upsampled using bicubic interpolation Output of the network of IVRL_Primex3 downsampled x3 upsampled using bicubic interpolation Output of the network of VIDAR
Fig. 5.
Performance of IVRL Prime, and VIDAR teams on image 124 from the Track2 testing split, compared to bicubic upsampled LR × × × × × ×
3) obtained using a bicubic kernel. All the imagery in the panels correspondsto the normalized spectral power image, and for the sake of better visualization, wehave gamma-corrected the 14 channels by setting γ = 0 . the networks and algorithms used at the challenge, we would like to refer theinterested reader to [34]. In this paper, we have introduced the StereoMSI dataset comprising of 350 stereospectral-colour image pairs. The dataset is a novel one which is specifically struc-tured for multispectral super-resolution benchmarking. Although it was acquiredwith spectral image super-resolution in mind, it is quite general in nature. Hav-ing a ColorChecker present in every image, it can also be used for a numberof other learning-based applications. Moreover, it also provides lower-resolutionimagery and training, validation, and testing splits for both colour-guided andexample-based learning applications. We have also presented a set of quality im-age metrics applied to the images when up-sampled using a bicubic kernel and,in doing so, provided a baseline based upon an image resizing approach widelyused in the community. We have also provided a summary of both tracks inthe PIRM2018 spectral image super-resolution challenge and shown the resultsobtained by the respective winners.
Acknowledgments
The PIRM2018 challenge was sponsored by CSIRO’s DATA61, Deakin Univer-sity, ETH Zurich, HUAWEI, and MediaTek.
References
1. Robles-Kelly, A., Huynh, C.P.: Imaging spectroscopy for scene analysis. SpringerScience & Business Media (2012)2. Goetz, A.F.: Three decades of hyperspectral remote sensing of the earth: A personalview. Remote Sensing of Environment (2009) S5–S163. Hasan, M., Jia, X., Robles-Kelly, A., Zhou, J., Pickering, M.R.: Multi-spectral re-mote sensing image registration via spatial relationship analysis on sift keypoints.In: Geoscience and Remote Sensing Symposium (IGARSS), 2010 IEEE Interna-tional, IEEE (2010) 1011–10144. Lu, G., Fei, B.: Medical hyperspectral imaging: a review. Journal of biomedicaloptics (1) (2014) 0109015. Feng, Y.Z., Sun, D.W.: Application of hyperspectral imaging in food safety inspec-tion and control: a review. Critical reviews in food science and nutrition (11)(2012) 1039–10586. Elarab, M., Ticlavilca, A.M., Torres-Rua, A.F., Maslova, I., McKee, M.: Estimat-ing chlorophyll with thermal and broadband multispectral high resolution imageryfrom an unmanned aerial system using relevance vector machines for precision agri-culture. International Journal of Applied Earth Observation and Geoinformation (2015) 32–427. Liang, H.: Advances in multispectral and hyperspectral imaging for archaeologyand art conservation. Applied Physics A (2) (2012) 309–323IRM2018 Challenge on Spectral Image Super-Resolution: Dataset 118. Bell, J.F., Wellington, D., Hardgrove, C., Godber, A., Rice, M.S., Johnson, J.R.,Fraeman, A.: Multispectral imaging of mars from the mars science laboratorymastcam instruments: Spectral properties and mineralogic implications along thegale crater traverse. In: AAS/Division for Planetary Sciences Meeting Abstracts.Volume 48. (2016)9. Xie, Z., Jiang, P., Zhang, S., Xiong, J.: Hyperspectral face recognition based onspatio-spectral fusion and local binary pattern. In: AOPC 2017: Optical Sensingand Imaging Technology and Applications. Volume 10462., International Societyfor Optics and Photonics (2017) 104620C10. Wu, D., Sun, D.W.: Advanced applications of hyperspectral imaging technologyfor food quality and safety analysis and assessment: A reviewpart i: Fundamentals.Innovative Food Science & Emerging Technologies (2013) 1–1411. Bigas, M., Cabruja, E., Forest, J., Salvi, J.: Review of cmos image sensors. Micro-electronics journal (5) (2006) 433–45112. Timofte, R., Agustsson, E., Van Gool, L., Yang, M.H., Zhang, L., Lim, B., Son,S., Kim, H., Nah, S., Lee, K.M., et al.: Ntire 2017 challenge on single image super-resolution: Methods and results. In: Computer Vision and Pattern RecognitionWorkshops (CVPRW), 2017 IEEE Conference on, IEEE (2017) 1110–112113. Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using verydeep convolutional networks. In: Proceedings of the IEEE conference on computervision and pattern recognition. (2016) 1646–165414. Lai, W.S., Huang, J.B., Ahuja, N., Yang, M.H.: Deep laplacian pyramid networksfor fast and accurate superresolution. In: IEEE Conference on Computer Visionand Pattern Recognition. Volume 2. (2017) 515. Timofte, R., Gu, S., Wu, J., Van Gool, L., et al.: Ntire 2018 challenge on single im-age super-resolution: Methods and results. In: The IEEE Conference on ComputerVision and Pattern Recognition (CVPR) Workshops. (June 2018)16. Timofte, R., De Smet, V., Van Gool, L.: A+: Adjusted anchored neighborhoodregression for fast super-resolution. In: Asian Conference on Computer Vision,Springer (2014) 111–12617. Timofte, R., De Smet, V., Van Gool, L.: Anchored neighborhood regression forfast example-based super-resolution. In: Proceedings of the IEEE InternationalConference on Computer Vision. (2013) 1920–192718. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented naturalimages and its application to evaluating segmentation algorithms and measuringecological statistics. In: Computer Vision, 2001. ICCV 2001. Proceedings. EighthIEEE International Conference on. Volume 2., IEEE (2001) 416–42319. Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparserepresentation. IEEE transactions on image processing (11) (2010) 2861–287320. Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: International conference on curves and surfaces, Springer(2010) 711–73021. Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M.L.: Low-complexitysingle-image super-resolution based on nonnegative neighbor embedding. (2012)22. Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformedself-exemplars. In: Proceedings of the IEEE Conference on Computer Vision andPattern Recognition. (2015) 5197–520623. Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution:Dataset and study. In: The IEEE Conference on Computer Vision and PatternRecognition (CVPR) Workshops. Volume 3. (2017) 22 Mehrdad Shoeiby et al.24. Li, Y., Hu, J., Zhao, X., Xie, W., Li, J.: Hyperspectral image super-resolutionusing deep convolutional neural network. Neurocomputing (2017) 29–4125. Yasuma, F., Mitsunaga, T., Iso, D., Nayar, S.K.: Generalized assorted pixel camera:postcapture control of resolution, dynamic range, and spectrum. IEEE transactionson image processing (9) (2010) 2241–225326. Chakrabarti, A., Zickler, T.: Statistics of real-world hyperspectral images. In:Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on,IEEE (2011) 193–20027. Foster, D.H., Nascimento, S.M., Amano, K.: Information limits on neural iden-tification of colored surfaces in natural scenes. Visual neuroscience21