A Noise-aware Enhancement Method for Underexposed Images
AA Noise-aware Enhancement Method forUnderexposed Images
Chien-Cheng Chien
Tokyo Metropolitan University
Tokyo, Japan
Yuma Kinoshita
Tokyo Metropolitan University
Tokyo, Japan
Hitoshi Kiya
Tokyo Metropolitan University
Tokyo, Japan
Abstract —A novel method of contrast enhancement is proposedfor underexposed images, in which heavy noise is hidden. Underlow light conditions, images taken by digital cameras have lowcontrast in dark or bright regions. This is due to a limiteddynamic range that imaging sensors have. For these reasons,various contrast enhancement methods have been proposed sofar. These methods, however, have two problems: (1) The lossof details in bright regions due to over-enhancement of contrast.(2) The noise is amplified in dark regions because conventionalenhancement methods do not consider noise included in images.The proposed method aims to overcome these problems. In theproposed method, a shadow-up function is applied to adaptivegamma correction with weighting distribution, and a denoisingfilter is also used to avoid noise being amplified in dark regions.As a result, the proposed method allows us not only to enhancecontrast of dark regions, but also to avoid amplifying noise, evenunder strong noise environments.
Index Terms —Contrast enhancement, Image enhancement,Noise aware, Shadow-up function, Retinex, Denoising filter
I. I
NTRODUCTION
To overcome a limited dynamic range that imaging sensorshave, various contrast enhancement methods have so far beenproposed. The histogram equalization (HE) is one of themost popular algorithms for contrast enhancement, and thereare various extended versions of the HE. However, thesehistogram-based methods cause the loss of details in brightregions due to the over-enhancement. Contrast enhancementmethods based on the Retinex theory have also been studied[1]–[3]. Although these methods can enhance the contrastwhile preserving details in bright areas, they also have a noiseamplification problem as with histogram-based methods.To avoid the noise amplification problem, some contrast en-hancement methods have been proposed [1], [2], [4]. However,they do not preserve details in bright areas, although, they canreduce some noise.Because of such a situation, we proposes a novel imagecontrast enhancement method based on both the Retinextheory and a noise aware shadow-up function. The proposedmethod can enhance image contrast without over-enhancementand noise amplification. A shadow-up function is used forpreventing over-enhancement and the loss of details in brightregions. In addition, the use of a mapping function designed byusing adaptive gamma correction with weighting distribution(AGCWD) [5] allows not only to enhance contrast in darkregions, but also to avoid amplifying noise. In an experiment, the proposed method is compared withconventional contrast enhancement methods, including state-of-the-art ones. Experimental results show that the pro-posed method can produce high quality images without over-enhancement and noise amplification.II. RELATED WORKSRelated works are summarized here.
A. Retinex theory
Retinex theory is based on the relation, S = R · L ,where original image S is the product of illumination L andreflectance R. When the information of only one surround isused for the conversion of each pixel, its approach is calledSingle-Scale Retinex (SSR) [6]. In SSR, halo artifacts occurunnaturally in the boundary of regions with large gradientvalues. To solve this problem, Multi-Scale Retinex (MSR) [7]was proposed. However, since a logarithmic transformationis used, MSR still causes a problem that the results donot stabilize due to the influence of noise in dark areas.Simultaneous reflection & illumination estimation (SRIE) [8]and weighted variation model (WVM) [1] are also Retinex-based methods. These methods have a good performance forimages without noise, but some strange areas are generatedin strong noise environments. Therefore, many outstandingmethods [2], [9], [10] have been proposed to improve thequality of images, and preserve more details. B. Image enhancement
The histogram equalization (HE) [11] is one of the mostpopular algorithms for contrast enhancement [12] and variousextended versions of HE have been proposed [5], [13]–[17].Contrast enhancement using adaptive gamma correction withweighting distribution (AGCWD) [5] aims to prevent over-enhancement and under-enhancement caused by using adap-tive gamma correction and a modified probability distribution.However, the over-enhancement and the loss of contrast inbright areas are still caused under the use of these histogram-based methods. Some noise hidden in the darkness is alsoamplified. Because of such a situation, a number of histogram-based contrast enhancement methods have been proposed toprevent the noise amplification. In the methods, a shrinkagefunction is used for preventing the noise amplification. Lowlight image enhancement based on two-step noise suppression a r X i v : . [ c s . MM ] A p r ig. 1. Flowchart of the proposed method. Fig. 2. Example of mapping curves. (LLIE) [4] uses both noise level function (NLF) and justnoticeable difference (JND) for contrast enhancement withnoise suppression. Although this method can reduce somenoise, it does not preserve details in bright areas as withhistogram-based methods. Another way for enhancing imagesis to use a multi-exposure image fusion method by usingphotos with different exposures [18]–[21]. C. Deoising filter
Image denoising has a great tradition in the research fieldof signal processing because of its fundamental role in manyapplications. In particular, block-matching and 3D filtering(BM3D) [22] is one of the most successful advances. In thispaper, BM3D is used as one of noise suppression techniques.Our purpose is not only to enhance contrast with noisesuppression, but also to preserve details in bright regions basedon Retinex theory.III. P
ROPOSED METHOD
The novelty and the detail of the proposed method areexplained here, and the outline of the proposed method isshown in Fig 1.
A. Decomposition based on Retinex
As shown in Fig. 1, an input RGB image X = { X R , X G , X B } is transformed to an HSV image X HSV = { H , S , V } , where H , S and V are hue, saturation and bright-ness images, respectively. An excellent weighted variationalmodel (WVM) was proposed for simultaneous reflectance andillumination estimation [1]. We use this model for decompos-ing V into illumination layer I and reflectance layer R , where R has almost no noise, but I includes, due to the work of themodel. B. Contrast Enhancement I is enhanced by using two key technologies: Shadow-up function and AGCWD. The use of a shadow-up functionaims to avoid the loss of details in bright areas due to overenhancement, and example is shown in Fig. 2. A shadow-up function, which consists of a nonlinear part and a linear part,is given by I (cid:48) ( x, y ) = (cid:40) T ( I ( x, y )) , if I ( x, y ) < I th I ( x, y ) , otherwise , (1)where I ( x, y ) ∈ [0 , is the intensity of illumination layer ata coordinate ( x, y ) , T ( I ( x, y )) is a monotonically increasingfunction, and I th is an upper limit of the nonlinear partfor avoiding over enhancement in bright areas. Contrast isenhanced only when I ( x, y ) is less than the threshold value I th , according to (1).To determine a proper threshold value I th for illuminationlayer, we take into account the luminance distribution of theillumination layer. Let it be H = { ( x, y ) : I th < I ( x, y )
By enhancing the illumination layer I , an adjusted illu-mination I (cid:48) is obtained. Then an enhanced V (cid:48) is computedby V (cid:48) ( x, y ) = I (cid:48) ( x, y ) · R ( x, y ) . Finally, an YUV image Y Y UV = { Y , U , V (cid:48) } is obtained by using V (cid:48) , H , and S ,according to the model [1]. D. Denoising technique Y Y UV still has some noise in dark areas, since R includesnoise, though the Retinex theory allows us to avoid enhancing a) Original image (b) Reflectance layer (c) Illumination layer (d) Enhanced illumination layerFig. 3. Example of weighted variation model (WVM), and example of illumination layer is enhanced by AGCWD with shadow-up function.(a) Original image (b) AGCWD [5] (c) WVM [1] (d) Proposed methodFig. 4. Experimental Results. the noise. Therefore, a denoising technique is required tofurther improve the visual quality. Block-matching and 3Dfiltering (BM3D) [22] is chosen as a denoising method inthis paper. In our implementation, for further cutting thecomputational load, BM3D is applied to only Y channel. Afterthe denoising, an enhanced RGB image X (cid:48) is computed byusing Y (cid:48) , U and V . IV. S IMULATION
A. Simulation condition
We used six images for our simulation, where four imageswere from LIME [2], and two other images were taken by a digital camera Canon 5D mark IV under the conditions:ISO 12800 and safe shutter speed. Because the ISO valuewas very high, the two images contained a lot of noise asshown in Fig. 4(a). We carried out a simulation to comparethe proposed method with conventional contrast enhancementmethods, AGCWD [5] and WVM [1] and LIME [2]. Weadopted the th percentile as I th . B. Simulation results1) Visual comparison:
We picked up two images taken bythe camera resulting images, subjectively in Fig. 4. The secondcolumns in Fig. 4 are the enlarged view of red boxes in the first
ABLE IEXPERIMENTAL RESULTS (NIQE)Method Original AGCWD [5] WVM [1] LIME [2] ProposedImage 1
Image 3 4.366 4.169
Image 5 6.497 6.807 6.718 7.487
Image 6 3.226 column, so that we clearly see the difference in dark and brightareas, and noise. From Fig. 4(b), it is confirmed that AGCWDover-enhanced bright areas, but clearly enhanced dark areas.Further, we easily see a lot of noise in the image. WVM notonly enhanced noise but also changed the white balance indark areas. Also, we easily observe unusual purple areas inFig. 4(c). In contrast, the proposed method provided almostsame quality as that of the original image in bright areas.Moreover, the image had less noise than AGCWD and WVMin dark areas.
2) Objective evaluation:
A blind image quality assess-ments, called natural image quality evaluator (NIQE) [23] wasused to objectively evaluate the quality of enhanced images.Here, a matlab function niqe() and its default model were usedfor the evaluation. Since the default model used in niqe() weretrained with noisy images, a lower NIQE score represents thatthe evaluated image has less noise.Table I shows NIQE scores for six images. From the table,we can confirm that the proposed method averagely had lowerscores than the other methods including the state-of-the-artones. Hence, the proposed method was demonstrated to beeffective to avoid noise amplification while enhancing imagecontrast. V. C
ONCLUSION
We proposed a novel image contrast enhancement methodbased on both the Retinex theory and a noise aware shadow-up function. The proposed method can enhance the contrastof images without over-enhancement and noise amplification.Experimental results showed that the proposed method suc-cessfully enhances contrast, while preserving details in brightregions and suppressing some noise in dark regions.R
EFERENCES[1] X. Fu, D. Zeng, Y. Huang, X. P. Zhang, and X. Ding, “A weighted vari-ational model for simultaneous reflectance and illumination estimation,”in 2016 IEEE Conference on Computer Vision and Pattern Recognition(CVPR), June 2016, pp. 2782-2790.[2] X. Guo, Y. Li, and H. Ling, “ LIME: Low-light image enhancement viaillumination map estimation,” IEEE Trans. Image Process., vol. 26, no.2, pp. 982-993, Feb. 2017.[3] Edwin H Land, “The retinex theory of color vision,” Scientific American,237(6):108-129, 1977.[4] Su. H, Jung. C, “Low Light Image Enhancement Based on Two-Step Noise Suppression,” IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP), pp.1977-1981., ICASSP, 2017.[5] S.-C. Huang, F.-C. Cheng, and Y.-S. Chiu, “ Efficient contrast enhance-ment using adaptive gamma correction with weighting distribution,”IEEE Transactions on Image Processing,pp. 1032-1041, 2013. [6] G. Hines, Z. Rahman, D. Jobson and G. Woodell,
Single-scale retinexusing digital signal processors , Global Signal Processing Conference,pp.1-6, 2005.[7] Z. Rahman, D. Jobson, and G. A. Woodell,
Multiscale retinex for colorimage enhancement , Proc. IEEE Intl. Conf. Image Process, 1996.[8] X. Fu, Y. Liao, D. Zeng, Y. Huang, X.-P. Zhang, and X. Ding,
A prob-abilistic method for image enhancement with simultaneous illuminationand reflectance estimation , IEEE Transactions on Image Processing,24(12):4965-4977, 2015.[9] D. J. Jobson, Z.-U. Rahman, and G. A. Woodell,
A multiscale retinexfor bridging the gap between color images and the human observationof scenes ,Image Processing, IEEE Transactions on, 6(7):965-976, 1997.[10] X. Ren, M. Li, W.-H. Cheng, and J. Liu,
Joint enhancement anddenoising method via sequential decomposition , IEEE InternationalSymposium on Circults and Systems (ISCAS), 2018[11] S. Pizer, R. Johnston, J. Ericksen, B. Yankaskas, and K. Muller,
Contrast-limited adaptive histogram equalization: Speed and effective-ness , in Proc. 1st Conf. Visual. Biomed. Comput, pp. 337-345, 1990.[12] H. K. Sawant and M. Deore,
A comprehensive review of image en-hancement techniques , International Journal of Computer Technologyand Electronics Engineering (IJCTEE),1(2),39-44,2010.[13] K. Zuiderveld,
Contrast Limited Adaptive Histogram Equalization ,Elservier, pp. 474-485, 1994.[14] Y. Kim,
Contrast enhancement using brightness preserving bi-histogramequalization , IEEE Trans. Consum. Electron, vol. 43, no. 1, pp. 1-8, Feb,1997.[15] Y. Wan, Q. Chen, and B. Zhang,
Image enhancement based on equalarea dualistic sub-image histogram equal- ization method , IEEE Trans.Consum. Electron, vol. 45, no. 1, pp. 68-75, Feb, 1999.[16] X. Wu, X. Liu, K. Hiramatsu, K. Kashino,
Contrast-accumulated his-togram eqalization for image enhancemen , The International Conferenceon Image Processing (ICIP),pp. 3190-3194, 2017.[17] C. C. Chien, Y. Kinoshita, S. Shiota, H. Kiya,
A retinex-based imageenhancement scheme with noise aware shadow-up function , in Proc.SPIE 11049, International Workshop on Advanced Image Technology(IWAIT), 110492K, 2019.[18] Y. Kinoshita, T. Yoshida, S. Shiota, and H. Kiya,
Multi-exposure imagefusion based on exposure compensation , IEEE, International Conferenceon Acoustics, Speech, and Signal Processing (ICASSP), pp. 1388-1392,2018.[19] Y. Kinoshita, S. Shiota, and H. Kiya,
Automatic exposure compensationfor multi-exposure image fusion , IEEE, International Conference onImage Processing (ICIP), pp. 1-5, 2018.[20] Y. Kinoshita and H. Kiya, “Automatic exposure compensation usingan image segmentation method for single-image-based multi-exposurefusion,”
APSIPA Trans. Signal Inf. Process. , vol. 7, e22, pp. 1–10, Dec.2018.[21] Y. Kinoshita and H. Kiya, “Scene Segmentation-Based LuminanceAdjustment for Multi-Exposure Image Fusion,”
IEEE Trans. ImageProcess. , to be published, doi: 10.1109/TIP.2019.2906501.[22] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising bysparse 3d transform-domain collaborative filtering,”