Multi-focus Image Fusion Based on Similarity Characteristics
11 Abstract
A novel multi-focus image fusion algorithm performed in spatial domain based on similarity characteristics is proposed incorporating with region segmentation. In this paper, a new similarity measure is developed based on the structural similarity (SSIM) index, which is more suitable for multi-focus image segmentation. Firstly, the SSNSIM map is calculated between two input images. Then we segment the SSNSIM map using watershed method, and merge the small homogeneous regions with fuzzy c-means clustering algorithm (FCM). For three source images, a joint region segmentation method based on segmentation of two images is used to obtain the final segmentation result. Finally, the corresponding segmented regions of the source images are fused according to their average gradient. The performance of the image fusion method is evaluated by several criteria including spatial frequency, average gradient, entropy, edge retention etc. The evaluation results indicate that the proposed method is effective and has good visual perception. Introduction
Image fusion provides a mathematical model to integrate complementary and redundant information from multiple images of the same scene into a composite image, which is more suitable for the purpose of human visual perception and computer-processing tasks, such as segmentation, feature extraction and target recognition. Important applications of image fusion include medical science [1], remote sensing [2,17,18], face recognition [3] and object tracking [19,20]. In this paper we concentrate on multi-focus image fusion [4,5,16]. Due to the limited depth-of-focus of optical lenses, it is impossible to acquire an image that contains all relevant objects in focus through a shooting. To achieve an image where all objects are in focus, a multi-focus image fusion technique is desirable. Image fusion is generally performed at three different levels of information representation including pixel level, feature level and decision level respectively. According to recent research [6-8], feature level image fusion based on regions has become an important method. Several researchers incorporated the regional feature information in multi-resolution decomposition [9,10], while the inverse transform process may lose some information [11]. A recent trend for region based image fusion method is to implement the segmentation and fusion in spatial domain [11,12]. Li and Yang [11] segment the averaged image via traditional image segmentation without considering the complementary and redundant information, which is likely to produce segmentation mistakes. A partition strategy according to the
SSIM -map of the input images has been proposed in [12], however, the segmentation result of multi-focus images is less reliable since different focus areas can’t be clearly distinguished by SSIM. Fig.1 depicts our proposed image fusion process based on similarity characteristics and region segmentation for two source images. It gives a description of our segmentation process: non-similarity based segmentation incorporating watershed and fuzzy c-means clustering algorithm FCM(NSIWF). Firstly, the SSIM map and the sign for different regions are calculated between two input images, and our signed structural non-similarity (SSNSIM) map is generated by combining SSIM and the sign. To obtain an accurate and reliable region segmentation result, we merge the small regions which are generated by watershed using FCM. After achieving the final segmentation result, the corresponding segmented regions of the source images are fused according to the fusion rule. Fig.2 shows the fusion process of three source images based on joint region segmentation. The joint region segmentation is based on segmentation of two source images. Three different segmentation results are obtained by using NSIWF. To achieve a more reasonable final segmentation result that considering the information of all the source images, the joint region segmentation result is generated by integrating all of the three segmentation results.
Multi-focus Image Fusion Based on Similarity Characteristics
Ya-Qiong Zhang, Xiao-Jun Wu * , Hui Li (School of IoT Engineering, Jiangnan University, Wuxi, 214122, China) [email protected], [email protected], [email protected] The main contributions of this paper include: (1)
A new similarity measure is developed: signed structural non-similarity (SSNSIM) index, which leads to a more reliable region segmentation process for multi-focus image fusion. (2)
A joint region segmentation method for three source images based on segmentation of two images is used to obtain a more accurate segmentation result. The remainder of the paper is organized as follows. In Section 2, the proposed similarity measure SSNSIM is described. Section 3 provides a brief introduction to the segmentation method and joint region method. Our fusion rule is presented in Section 4. The experimental results are presented in Section 5 and conclusions are drawn in Section 6. Similarity measures
Structural similarity index(SSIM)
It is necessary to analyze the similarity of source images, because the purpose of image fusion is to create a composite image, which preserves the complementary characteristics and removes the redundant information.
SSIM has been used as similarity index in [12]. Given two images X and Y , the SSIM value for each pixel is evaluated as [13]: ( , ) [ ( , )] [ ( , )] [ ( , )]
SSIM i j l i j c i j s i j = (1) where , and are parameters used to adjust the relative importance of the three components. The luminance ( , ) l i j , contrast ( , ) c i j , and structure ( , ) s i j components for each pixel are defined as follows:
12 2 1 i j i j CX Yl i j i j i j CX Y += + + (2)
22 2 2
X YX Y i j i j Cc i j i j i j C += + + (3) ( , )( , ) ( , ) ( , )
XYX Y i j Cs i j i j i j C += + (4) where ( , ) X i j and ( , ) Y i j are the means of central pixel and neighborhood pixels in the local window for X and Y respectively. ( , ) X i j and ( , ) Y i j are the standard deviation between central pixel and neighborhood pixels in the local window for X and Y respectively. ( , ) XY i j is the cross-correlation of ( , ) X X i j − and ( , ) y Y i j − in the local window centered on pixel ( , ) i j for X and Y . Signed structural non-similarity(SSNSIM)
The
SSIM has already been applied to region segmentation [12]. As shown in Fig.3(c), different focus areas are not be clearly distinguished by
SSIM -map, which would reduce the accuracy of segmentation and fusion performance. Generally, the similarity between two source images in different focus areas is small, while the non-similarity is large. To make the feature difference between the two kinds of regions more apparent, a new similarity measure is developed based on
SSIM by appending sign for different focus areas: signed structural non-similarity (
SSNSIM ) index.
Figure 1: Fusion process for two source images Figure 2: Fusion process for three source images
Let
SNSIM be the structural non-similarity, the
SNSIM value for each pixel is:
𝑆𝑁𝑆𝐼𝑀(𝑖, 𝑗) = 1 − 𝑆𝑆𝐼𝑀(𝑖, 𝑗) (6)
Let
SSNSIM be the signed structural non-similarity, the
SSNSIM value for each pixel is defined as: ( , ) ( , ) ( , )
SSNSIM i j S i j SNSIM i j = (7) where S is the sign for different focus areas, defined by ( , ) X i j and ( , ) Y i j in (3):
1, ( , ) ( , )( , ) 1, ( , ) ( , )
X YX Y i j i jS i j i j i j + = − (8) where + and − denote the pixel belonging to the focus areas in image X and image Y , respectively. In apparently different focus areas, the SNSIM is large and the sign S is opposite, thus SSNSIM are very different. Fig.3(d) shows the
SSNSIM -map for two source images, higher gray value indicates larger
SSNSIM value. As it can be seen, different focus areas are clearly distinguished. Thus, the use of SSNSIM can lead to a more reliable segmentation process. (a) (b) (c) (d) Image segmentation
Segmentation based on watershed and FCM
Different source image regions have different similarity so that the similarity correlation is regional. Therefore, the similarity map should be partitioned into regions. The watershed algorithm is a common and powerful technique for image segmentation, and has recently received increased attention for its high segmentation accuracy and low computational load. However, this transformation is sensitive to noise, and usually results in over segmentation. The over segmentation makes it hard to identify meaningful regions. To reduce the over segmentation, firstly, the morphological H-minima transform [14] is performed on the gradient map of
SSNSIM -map as a modification, then the watershed algorithm is performed on the modified gradient map. Two sub-images at the top-left 50
50 pixels of Fig.3 are exemplified in Fig.4 to illustrate the process of forming regions better. The segmentation output of the watershed algorithm is shown in Fig.4(a). To deal with the over segmented output of the watershed algorithm, we need to merge the over segmented regions. Fuzzy c-means clustering algorithm (FCM) is an effective solution to this problem. Let num be the number of regions segmented by watershed algorithm and the SSNSIM values for the n -th region is (1 ) n SSNSIM n num , we use the mean SSNSIM (MNS) as the data for clustering: n Mn nkn
MNS SSNSIM kM = = (9) where n M is the number of pixels in the n -th region. Let 𝑁 𝑐𝑙𝑢𝑠𝑡𝑒𝑟 be the number of clusters. The FCM algorithm is used to cluster the feature (1 ) n MNS n num , we obtain 𝜇 𝑚𝑛 (1 ≤ 𝑚 ≤ 𝑁 𝑐𝑙𝑢𝑠𝑡𝑒𝑟 , 1 ≤ 𝑛 ≤ 𝑛𝑢𝑚) , [0,1] mn denotes the membership of the feature n MNS in the m -th cluster. The classification strategy is given as: n MNS belongs to the m -th cluster when 𝜇 𝑚𝑛 =𝑚𝑎𝑥 𝑙 {𝜇 𝑙𝑛 , 𝑙 = 1,2, . . . , 𝑁 𝑐𝑙𝑢𝑠𝑡𝑒𝑟 } . The feature n MNS is partitioned into 𝑁 𝑐𝑙𝑢𝑠𝑡𝑒𝑟 groups, and consequently, regions segmented by watershed algorithm are partitioned into 𝑁 𝑐𝑙𝑢𝑠𝑡𝑒𝑟 classes. Fig.4(b) shows the clustering result of FCM when 𝑁 𝑐𝑙𝑢𝑠𝑡𝑒𝑟 is 5, where the regions of the same class are labeled by the same grey value. Then, we merge the regions by dealing with the regional boundary as follows: the boundaries of the regions belong to the same class can be directly included in this class, and the boundaries of the different classes will be included in the class to which its feature is most close. Fig.4(c) shows the final segmentation result. (a) (b) (c) Joint region segmentation
For the fusion of three source images, we proposed a joint region segmentation method based on the similarity between all the source images. Given three images X , Y and Z , XY R is the segmentation result obtained with X and Y , XZ R is the Figure 3: Similarity maps of multi-focus images (a) and (b). (c)
SSIM -map. (d)
SSNSIM -map. Figure 4: An example of region segmentation. (a) Segmentation output of the watershed algorithm. (b) Clustering result of FCM. (c) Final segmentation result. segmentation result obtained with X and Z , YZ R is the segmentation result obtained with Y and Z . Joint region segmentation is a process to incorporate XY R , XZ R and YZ R , and generates a more accurate segmentation result. Fig.5 shows an example of generating a joint region map of XY R and XZ R . Let XY r denotes a region from XY R , and XZ r denotes a region from XZ R , we set the following incorporation rules: (1) If XY r is the same as XZ r , one region is generated in the joint region map: J XY XZ r r r = = . (2) If XY r is disjoint with XZ r , two regions are generated in the joint region map: J XY r r = , J XZ r r = . (3) If one region is entirely included in the other one, e.g.
XY XZ r r , two regions are generated in the joint region map: J XY r r = , J XZ XY r r r = − . (4) If XY r and XZ r are partially overlapped, three regions are produced in the joint region map: J XY XZ r r r = , J XY J r r r = − , J XZ J r r r = − . (a) (b) (c) The final segmentation result
XYZ R is obtained by generating a joint region map of J R and XZ R . In this paper, we use a simple way to implement the joint region segmentation according to the above rules. Assuming XY R , XZ R and YZ R are obtained by the FCM with a clustering number N , the joint region segmentation procedure is realized as follows: (1) The regions of XY R , XZ R and YZ R are labeled by the following three arithmetic progression respectively:
1, 2,..., N ,
1, 2( 1),..., ( 1)
N N N N + + + , ( 1) , 2( 1) ,..., ( 1) N N N N + + + . Assuming A , B and C are the label matrixes of XY R , XZ R and YZ R , which is shown in Fig.6. (2) The label matrix of joint regions is:
ABC
M A B C = + + (10)
Fig.6(d) is the label-map of joint regions, in which every number labels a region. The label matrix
ABC M indicates the final segmentation result XYZ R . (a) (b) (c) (d) Fusion rule
Here we adopt an average gradient sensitivity to evaluate the clarity of the image, it can reflect the features of image contrast in minute details and texture. The average gradient is defined as follows:
M Ni j
F i j F i j F i j F i jg M N − −= = − + + − += − − (11) where F is the fused image, M N is the size of the fused image. We compare the average gradient of the corresponding regions of the source images to decide which should be used to construct the fused image. Let’s take the fusion of three source images as an example: ,,, X Y X zi i i i iY X Y zi i i i i iZ X Z Yi i i i i
RoX g g andg gRoF RoY g g andg gRoZ g g andg g = (12) where i RoF is the i -th region of the fused image, Xi g , Yi g and Zi g are the average gradients of the i -th region of image X , Y and Z , respectively. Experimental results and discussions
In this section, subjective tests and objective measurements are used for qualitatively and quantitatively assessing the performance of the proposed methodology in two source images fusion and three source images fusion respectively. The evaluation criteria include variance (V), spatial frequency (SF), average gradient (AG), entropy (H), mutual information (MI) and edge retention ( / AB F Q ) [15]. These metrics can be effectively used to measure how well the source images are integrated into the fused image via the fusion algorithm. For all these measurements, the larger values indicate the better fusion performance. The fused images of different fusion schemes on two source images are shown in Fig.7. Fig.7(c, d) show the segmentation results in different schemes, in which the same kind of regions are labeled by the same grey value. Fig.7(c) shows the segmentation result with RSIM [12], from which we can clearly see the defect that regions in different focus areas are under the same class. While our
Figure 5: An example of generating a joint region map of XY R and XZ R . (a) XY R . (b) XZ R . (c) Joint region map J R . Figure 6: An example of joint region segmentation when the clustering number is 3. (a) Label matrix A of XY R . (b) Label matrix B of XZ R . (c) Label matrix C of YZ R . (d) Label matrix ABC M for XYZ R . segmentation result in Fig.7(d) has better segmentation performance. Table 1 shows the corresponding objective evaluation, as it can be seen, our proposed method is inferior to RSIM only in MI compared with RSIM [12] and has better visual perception. Scheme Evaluation V SF AG H MI RSIM [12] 51.3127 23.5837 12.0317 7.5888
Proposed (a) (b) (c) (d) (e) (f)
In order to test the robustness of our proposed algorithm, we simulate on other four pairs of two-focus images as shown in the columns of Fig.8(a) and (b). The fusion results of
RSIM [12] and our proposed method are presented in the columns of Fig.8(c) and Fig.8(d), respectively. Table 2 shows the objective comparison of different fusion schemes for the four pairs of original images in Fig.8. As we can see from Table 2, our method is inferior to
RSIM [12] only in H for the first row of source images, which indicates that our method out performs
RSIM [12] in two-focus image fusion.
Source images (row number in Fig.8)
Scheme Evaluation V SF AG H MI 1 RSIM [12] 51.0066 16.0091 6.6301
2 RSIM [12] 47.8008 15.7682 8.6477 7.4323 1.3983 Proposed
3 RSIM [12] 53.1642 23.7357 11.0372 7.5090 1.5211 Proposed
4 RSIM [12] 46.5970 17.4105 6.7841 7.0250 1.6404 Proposed (a) (b) (c) (d)
Table 3 shows the effect of different fusion schemes on three source images in Fig.9 and Fig.10. The rectangular area in Fig.9 (e) indicates selecting fuzzy area of the source images. Fig.10 (j) shows the fused image with RSSF [11], in which the fuzzy part in the white circle indicates the segmentation error in the black circle of Fig.10(i). As it can be seen, our proposed method is more effective compared with RSSF [11] in both visual presentation and objective evaluation.
Image Scheme Evaluation SF AG H / AB F Q Fig.9 RSSF [11] 18.8296 7.7765 6.8012 0.6805 Proposed
Fig.10 RSSF [11] 24.4513 11.7633 7.3854 0.4929 Proposed
Figure 7: (a) Left focus image. (b) Right focus image. (c) Segmentation result with RSIM [12]. (d) Segmentation result with our proposed method. (e) Fused image with RSIM [12]. (f) Fused image with proposed method.
Table 2: Performance for Different Fusion Schemes with Fig.8
Table 1: Performance for Different Fusion Schemes with Fig.7
Table 3: Performance for different fusion schemes
Figure 8: Fusion results of other images. (a) and (b) Source images. (c) Fused image by RSIM [12]. (d) Fused image by our proposed method. (a) (b) (c) (d) (e)
Fig.10 (d-f) shows the fused image with segmentation result obtained by only two of the three source images, the fused image with joint segmentation result is shown in Fig.10 (h) and has better visual perception. Table 4 shows the effect of different segmentation results on three source images. As it can be seen, our joint region segmentation method is more reasonable and accurate and lead to a better fusion performance.
Image Segmentation result Evaluation V SF AG / AB F Q Fig.10 A 23.9290 11.5754 7.3949 0.5071 B 23.0867 11.4341
ABC M (a) (b) (c) (d) (e) (f) (g) (h) (i) (j) Conclusions
In this paper, we proposed a new multi-focus image fusion method performed in spatial domain based on similarity characteristics incorporating with region segmentation. Our proposed SSNSIM leads to a more reliable region segmentation process for multi-focus image fusion. Considering the complementary and redundant information between all the source images, our joint region segmentation method for three source images based on segmentation of two images generates a more accurate segmentation result. Experimental results conducted several groups of images show the proposed method is effective and the perceptive quality of the fused image is enhanced.
Figure 10: (a)-(c) Three source images. (d) Fused image with segmentation result obtained by (a) and (b). (e) Fused image with segmentation result obtained by (a) and (c). (f) Fused image with segmentation result obtained by (b) and (c). (g) Joint segmentation result. (h) Fused image with joint segmentation result. (i) Segmentation result with RSSF [11]. (j) Fused image with RSSF [11].
Table 4: Performance for different segmentation results
Figure 9: (a)-(c) Three source images. (d) Fused image with proposed method. (e) Fused image with RSSF [11].
References [1]
Garg,s., Ushah Kiran, K.Mohan, R.Tiwary. Multilevel Medical Image Fusion using Segmented Image by Level Set Evolution with Region Competition, 27 th Annual International Conferrence of the Engineering in Medicine and Biology Society, 17-18 Jan, pp.7680-7683, 2006. [2]
Xiao qing Luo. Study on the Theory of Multi-Sensor Image Fusion and Its Applications. [PhD thesis]: Jiangnan University, 2010. [3]
Tao Wu, Xiao-Jun Wu, Xing Liu, Xiao-Qing Luo. New Method using Feature Level Image Fusion and Entropy Component Analysis for Multimodal Human Face Recognition [J]. Procedia Engineering. Volume 29, Pages 3991-3995, 2012. [4]
Helmy A. Eltoukhy, Sam Kavusi. A computationally efficient algorithm for multi-focus image reconstruction, Proc.SPIE 5017, Sensors and Camera Systems for Scientific, Industrial, and Digital Photography Applications IV, 332 (May 14, 2003); doi:10.1117/12.476754 [5]
Xuan Yang, Wanhai Yang, Jihong Pei. Different focus points images fusion based on wavelet decomposition[C], Information Fusion, vol.1, Page(s): MOD3/3 - MOD3/8. 10-13, July 2000. [6]
Piella G. A region-based multiresolution image fusion algorithm [J]. IEEE, 2:1557-1564, 2002. [7]
SU Dongxue, WU Xiaojun. Image Fusion Based on Multi-feature Fuzzy Clustering[J]. Journal of Computer-Aided Desing & Computer Graphics, 18(6):838-843, 2006. [8]
Lewis JJ, OCallaghan R J. Nikolov S G. et al. Region-based image fusion using complex wavelets [C]//Proceedings of the 7th International Conference on Information Fusion.Stockholm, Sweden:International Journal on Information Fusion, pp.555-562, 2004. [9]
G. Piella, A general framework for multiresolution image fusion: from pixels to regions, Information Fusion 4(4): 259–280, 2003. [10]
Xiao-Jun Wu, Dong-Xue Su, Xiao-Qing Luo, A new similarity function for region based image fusion incorporating Gabor filters and fuzzy c-means clustering, Proc.SPIE 6625, 66250Z, 2007. [11]
S. Li, B. Yang, Multifocus image fusion using region segmentation and spatial frequency, Image and Vision Computing 26 (7): 971–979, 2008. [12]
Xiao-yan Luo, Jun Zhang, Qiong-hai Dai, A regional image fusion based on similarity characteristics, Signal Processing 92(5):1268-1280, 2012. [13]
Z. Wang., A.C. Bovik, H.R. Sheikh, E.P. Simoncelli, Image quality assessment: from error visibility to structural similarity, IEEE Transactions on Image Processing 13(4): 600–612, Apr. 2004. [14]
R.J. O’Callaghan, D.R. Bull, Combined morphological spectral unsupervised image segmentation, IEEE Transactions on Image Processing 14(1): 49–62, January .2005. [15]
Xiao-Qing Luo, Xiao-Jun WU. New metric of image fusion based on region similarity. Optical Engineering, 49(4):047006-1-047006-13, 2010. [16]
Hui Li, Xiao-Jun Wu. Multi-focus image fusion using dictionary learning and low-rank representation [C]//International Conference on Image and Graphics. Springer, Cham, 2017: 675-686. [17]
Hui Li, Xiao-Jun Wu. Infrared and visible image fusion using Latent Low-Rank Representation[J]. arXiv preprint arXiv:1804.08992, 2018. [18]
Xiaoqing Luo, Zhancheng Zhang, Xiao-Jun Wu. A novel algorithm of remote sensing image fusion based on shift-invariant Shearlet transform and regional selection[J]. AEU-International Journal of Electronics and Communications, 2016, 70(2): 186-197. [19]