Automatic Spatially-Adaptive Balancing of Energy Terms for Image Segmentation
AAutomatic Spatially-Adaptive Balancing ofEnergy Terms for Image Segmentation
Josna Rao , Ghassan Hamarneh , and Rafeef Abugharbieh Biomedical Signal and Image Computing Lab,University of British Columbia, Canada Medical Image Analysis Lab, Simon Fraser University, Canada [email protected],[email protected],[email protected]
Abstract.
Image segmentation techniques are predominately based onparameter-laden optimization. The objective function typically involvesweights for balancing competing image fidelity and segmentation regular-ization cost terms. Setting these weights suitably has been a painstaking,empirical process. Even if such ideal weights are found for a novel image,most current approaches fix the weight across the whole image domain,ignoring the spatially-varying properties of object shape and image ap-pearance. We propose a novel technique that autonomously balancesthese terms in a spatially-adaptive manner through the incorporation ofimage reliability in a graph-based segmentation framework. We validateon synthetic data achieving a reduction in mean error of 47% (p-value << Key words:
Adaptive regularization, adaptive weights, image segmen-tation, energy minimization, energy functional, optimization, spectralflatness, noise detection
Robust automated image segmentation is a highly desirable goal that contin-ues to defy solution. In medical images for example, natural and pathologicalvariability may result in complicated and unpredictable image and shape fea-tures. Current segmentation methods are predominantly based on optimizationprocedures that produce so called ‘optimal’ segmentations at their minimum.Optimization methods typically incorporate a tradeoff between two classes ofcost terms: data fidelity and regularization. This actually is the case not only insegmentation, but also in image registration, shape matching, and other com-puter vision tasks. This basic tradeoff scheme is ubiquitous, relating to Occam’srazor and Akaike/Bayesian information criteria [1], and is seen in many forms,such as likelihood versus prior in Bayesian methods [2] and loss versus penaltyin machine learning [3]. Therefore, any advancement in controlling the balance a r X i v : . [ c s . C V ] J un J. Rao, G. Hamarneh, and R. Abugharbieh between competing cost terms will benefit many related applications and algo-rithmic formulations in medical image analysis. Optimization-based segmenta-tion methods that are fragile and highly sensitive to this tradeoff are plentiful,including active contours techniques [4][5][6][7], graph cut methods [8], and opti-mal path approaches [9]. For simplifying the exposition of the ideas in this paper,we will adopt the simplified but general form of the cost or energy function: E ( S | I, α, β ) = αE int ( S ) + βE ext ( S | I ) (1)where S is the segmentation and I is the image. E int is the internal cost termcontributing to the regularization of the segmentation, most often by enforcingsome smoothness constraints, in order to counteract the effects of imaging ar-tifacts. E ext is the external cost term contributing to the contour’s conformityto desired image features, e.g., edges. The weights α and β are typically setempirically by the users based on their judgment of how to best balance therequirements for regularization and adherence to image content. In most cases,this is a very difficult task and the parameters may be unintuitive for a typicalnon-technical end user, e.g. a clinician, who lacks knowledge of the underlyingalgorithm’s inner working. Also the resultant segmentations can vary drasticallybased on how this balance is set. Avoiding the practice of ad-hoc setting of suchweights is the driving motivation for our work here.To the best of our knowledge, regularization weights have traditionally beendetermined empirically and are fixed across the image domain (i.e. do not varyspatially). In Pluempitiwiriyawej et al [7], the weights are changed as the opti-mization progresses , albeit in an ad-hoc predetermined manner. McIntosh andHamarneh [10] demonstrated that adapting the regularization weights acrossa set of images is necessary in addressing the variability in real clinical imagedata. However, neither approach varies the weights spatially across the imageand hence are not responsive or adaptive to local features within a single image.Image regions with noise, weak or missing boundaries, and/or occlusions arecommonly encountered in real image data. For example, degradation in medi-cal images can occur due to tissue heterogeneity (“graded decomposition” [11]),patient motion, or imaging artifacts, e.g. echo dropouts in ultrasound or non-uniformity in magnetic resonance. In such cases and in order to increase seg-mentation robustness and accuracy, more regularization is needed in less reliableimage regions which suffer from greater deterioration. Although an optimal reg-ularization weight can be found for a single image in a set [10], the same weightmay not be optimal for all regions of that image. Spatially adapting the regular-ization weights provide greater control over the segmentation result, allowing itto adapt not only to images with spatially varying noise levels and edge strength,but also to objects with spatially-varying shape characteristics, e.g. smooth insome parts and jagged in others.Some form of spatially adaptive regularization over a single image appearedin a recent work by Dong et al [12]. For segmenting an aneurysm, they variedthe amount of regularization based on the surface curvature of a pre-segmented vessel. The results demonstrated improvements due to adaptive regularization. utomatic Spatially-Adaptive Balancing of Energy Terms 3 However, the regularization weights did not rely on the properties of the imageitself, which limited the generality of the method. Kokkinos et al [13] investigatedthe use of adaptive weights for the task of separating edge areas from texturedregions using a probabilistic framework, where the posterior probabilities of edge,texture, and smoothness cues were used as weights for curve evolution. Similarily,Malik et al [14] and, very recently, Erdem and Tari [15] tackled the problem oftexture separation and selected weights based on data cues. However, while thesemethods focused on curve evolution frameworks, our current work focuses ongraph-based segmentation. Additionally, we emphasize balancing the cost termsby adapting regularization for images plagued by noisy and weak or diffusededge problems rather than textured patterns in natural images, which we leaveas future work.In this paper, we advocate the strong need for spatially-adaptive balancing ofcost terms in an automated, robust, data-driven manner to relax the requirementon the user to painstakingly tweak these parameters. We also demonstrate howexisting fixed-weight approaches (even if globally optimized) are often inadequatefor achieving accurate segmentation. To address the problem, we propose a noveldata-driven method for spatial adaptation of optimization weights. We developa new spectral flatness measure of local image noise to balance the energy costterms at every pixel, without any prior knowledge or fine-tuning.We validate our method on synthetic, medical, and natural images and com-pare its performance against two alternative approaches for regularization: us-ing the best possible spatially-fixed weight, and using the globally optimal set ofspatially-varying weights as found automatically through dynamic programming.The rest of this paper is organized as follows; Section 2 presents a briefoverview of our segmentation process, the formulation for our proposed relia-bility method along with a formulation of the globally optimum graph searchapproach. Section 3 presents qualitative comparisons of our method to bothglobally-optimal and fixed parameter-based methods and reports quantitativeanalysis of the resulting error. Section 4 presents our conclusions and an overviewof future work planned.
Our formulation employs energy-minimizing boundary-based segmentation, wherethe objective is to find a contour that correctly separates an object from back-ground. We begin by formulating the energy of a contour and specifying how theregularization term is weighted in our definition. We then present our approachfor a data-driven spatially adaptive regularization method, and end the sectionwith a brief discussion of the globally-optimum parameter method.
We embed a parametric contour C ( q ) = C ( x ( q ) , y ( q )) : [0 , → Ω ⊂ R inimage I : Ω → R . We use a single adaptive weight w ( q ) ∈ [0 ,
1] that varies over
J. Rao, G. Hamarneh, and R. Abugharbieh the length of the contour and re-write (1) as: E ( C ( q ) , w ( q )) = (cid:90) ( w ( q ) E int ( C ( q )) + (1 − w ( q )) E ext ( C ( q ))) dq (2)where E ext ( C ( q )) = 1 − |∇ I ( C ( q )) | / max Ω |∇ I ( C ( q )) | (3)penalizes weak boundaries and E int ( C ( q )) = | dC ( q ) /dq | (4)is the length of the contour. To minimize E with respect to C ( q ) in (2), we adopta discrete formulation of the optimization problem. We model the image as agraph where each pixel is represented by a vertex v i and edges e ij = (cid:104) v i , v j (cid:105) thatcapture the pixel’s connectedness (e.g. 8-connectedness in 2D images). A localcost c ij = c ( e ij ) = wE int ( v i , v j ) + (1 − w ) E ext ( v i ) (5)is assigned to each edge e ij , where E int ( v i , v j ) is the Euclidean distances between v i and v j ( e.g. √ E = (cid:80) e ij ∈ C c ij represents the optimalsolution for the segmentation and is found by solving a minimal path problem,similar to [9], using Dijkstra’s algorithm [16]. Our approach for balancing the cost terms is to gauge the levels of signal vs.noise in local image regions. We estimate the edge evidence G ( x, y ) and noiselevel N ( x, y ) in each region of the image and set w ( x, y ) in (2) such that regionswith high noise and low boundary evidence (i.e. low reliability) have greaterregularization, and vice versa. Hence, w ( x, y ) is mapped to image reliability R ( x, y ) as w ( x, y ) = 1 − R ( x, y ) (6)where R ( x, y ) = (1 − N ( x, y )) G ( x, y ) . (7)Assuming additive white noise, uncorrelated between pixels, we estimate spatially-varying noise levels N ( x, y ) using local image spectral flatness (SF). SF is awell-known Fourier-domain measure that has been employed in audio signalprocessing and compression applications [17][18]. SF exploits the property thatwhite noise exhibits similar power levels in all spectral bands and results in aflat power spectrum, whereas uncorrupted signals have power concentrated incertain spectral bands and result in a more impulse-like power spectrum. Weextend the SF measure to 2D and measure N ( x, y ) as N ( x, y ) = exp (cid:16) π (cid:82) π − π (cid:82) π − π ln S ( ω x , ω y ) dω x dω y (cid:17) π (cid:82) π − π (cid:82) π − π S ( ω x , ω y ) dω x dω y (8) utomatic Spatially-Adaptive Balancing of Energy Terms 5 where S ( ω x , ω y ) is the 2D power spectrum of the image and ( ω x , ω y ) are spatialfrequencies. We use G ( x, y ) = max ( |∇ I x ( x, y ) | , |∇ I y ( x, y ) | ), where ∇ I x ( x, y )and ∇ I y ( x, y ) represent the x and y components of the image gradient. Wechose this measure rather than the standard gradient magnitude for its rotationalinvariance in the discrete domain. A theoretically appealing and intuitive approach for setting the regularizationweight is to optimize E in (2) for the weight w ( q ) itself in addition to opti-mizing the contour. In our discrete setting, this involves a ‘three dimensional’graph search that computes the globally optimal, spatially-adaptive regulariza-tion weight w ( q ), in conjunction with the contour’s spatial coordinates, i.e. weoptimize ˜ C ( q ) = ( x ( q ) , y ( q ) , w ( q )).In this formulation, each vertex in the original graph is now replaced by K vertices representing the different choices of the weight value at each pixel. Inaddition, graph edges now connect vertices corresponding to neighboring imagepixels for all possible weights. Note that the optimal path ˜ C ( q ) cannot passthrough the same ( x ( q ) , y ( q )) for different w , i.e. only a single weight can beassigned per pixel. Our graph search abides by this simple and logical constraint.The optimal C ( q ) and w ( q ) that globally minimize (2) are again calculated usingdynamic programming but now on this new ( x, y, w ) graph.There are, however, three main drawbacks to this globally optimum (in( x, y, w )) method: (i) it does not explicitly encode image reliability, even thoughregularization is essential in regions with low reliability; (ii) this approach willencourage a bimodal behavior of the regularization weight: w ( q ) = (cid:26) E int ( q ) > E ext ( q )1 otherwise (cid:27) , (9)and (iii) combining the weight and segmentation optimization into one processreduces the generality of the method. In short, even though optimal with re-spect to E in (2), the solution is incorrect and, as we later demonstrate, inferiorto the spatially adaptive balancing of energy cost terms proposed in Sec. 2.1.Furthermore, by combining the weight optimization and contour optimizationprocesses into one, we reduce the generality of the method. Finding globally-optimal weights for other segmentation frameworks would require significantchanges to the energy minimization process. We first performed quantitative tests on 16 synthetic images carefully designedto cover extreme shape and appearance variations (one example is shown in Fig. This is similar in spirit to [19] and [20] where they also optimize for a non-spatialvariable: vessel radius or scale, in addition to the spatial coordinates of the segmen-tation contour. J. Rao, G. Hamarneh, and R. Abugharbieh spatially-fixed regularization weight, set to the value producing the smallest (via brute force search) segmentation error. (a) Synthetic sinusoidal image I ( x, y ) (b) Synthetic sinusoidal image I ( x, y )(c) Edge evidence measure G ( x, y ) (d) Edge evidence measure G ( x, y )(e) Noise level estimate N ( x, y ) (f) Noise level estimate N ( x, y )(g) Total reliability measure R ( x, y ) (h) Total reliability measure R ( x, y ) Fig. 1.
Synthetic image with spatially varying noise and blurring (both increas-ing from right to left) and with changing boundary smoothness (smooth on theleft and jagged on the right). Black intensities corresponds to 0 and white to 1. utomatic Spatially-Adaptive Balancing of Energy Terms 7
We quantitatively examined our method’s performance using ANOVA test-ing on 25 noise realizations of each image in the dataset, where the error wasdetermined by the Hausdorff distance to the ground truth contour. Our methodresulted in a mean error (in pixels) of 6.33 (std. dev. 1.36), whereas the bestfixed-weight method had a mean error of 12.05 (std. dev. 1.61), and the globally-optimum weight method had a mean error of 33.06 (std. dev. 3.66). Furthermore,for each image, we found our method to be significantly more accurate with allp values << (a) Segmentation of image in Fig. 1(a)(b) Segmentation of image in Fig. 1(b) Fig. 2.
Color is essential for proper viewing, please refer to the e-copy . Con-tours obtained from: ( red ) proposed adaptive weights, ( green ) lowest-error fixedweight, and ( cyan ) globally optimum weight.We also tested our method on clinical MR images of the corpus callosum(CC), which exhibits the known problem of a weak boundary where the CC meetsthe fornix (Fig. 3(a)). Note how the contour obtained using globally optimalweights exhibits an optimal, yet undesirable, bimodal behavior (either blue or redin Fig. 3(a)) completely favoring only one of the terms at a time. In comparison,our method automatically boosts up the regularization (stronger red in Fig. 3(b))
J. Rao, G. Hamarneh, and R. Abugharbieh at the CC-fornix boundary producing a better delineation. The segmentationresults of all three methods for the same image are shown in Fig. 4. (a) (b)
Fig. 3. ( Color figure, refer to e-copy ). Results of (a) globally-optimum weightmethod and (b) proposed adaptive-weight method for a corpus callosum MRimage. The coloring of the contours reflects the value of the spatially-adaptiveweight. The same color map is used for both figures, with pure blue correspondingto w = 0 for pure red for and w = 1.We also tested our method on MR data from BrainWeb [21]. Fig. 5 shows thesegmentation of the cortical surface in a proton density (PD) image with a noiselevel of 5%. This example is a difficult scenario due to the high level of noise andlow resolution of the image. Our proposed method provided a smoother contourwhile conforming to the cortical boundary when compared to the other methods(although the difference was not too large).We also tested our method on natural images, such as the leaf shown in Fig.6(a). The resulting reliability measure (Fig. 6(b)) has lower image reliability and,hence, higher regularization at the regions of the leaf obscured by snow, whereasreliable boundaries light up (bright white boundary segments). The resultingsegmentations are shown in Fig. 7. We proposed a novel approach for addressing a ubiquitous problem that plaguesmany energy minimization segmentation techniques; how to balance the weightsof competing energy terms. Our technique spatially adapts the regularizationweight based on local spectral-flatness and data-driven image reliability mea-sures. We emphasize two contributions of our work: (i) regularization must varyspatially and must increase where image evidence is less reliable, and (ii) wediscriminate between signal edges (object boundaries) versus noise edges byextending a spectral flatness measure that is well established in audio signal utomatic Spatially-Adaptive Balancing of Energy Terms 9
Fig. 4. ( Color figure, refer to e-copy ). Segmentation results of the CC in the MRimage of Fig. 3.
Green contour is result of the proposed spatially adaptive weightmethod, red contour is result of best fixed-weight method, and cyan contour isresult of the globally-optimum weight method.
Fig. 5. ( Color figure, refer to e-copy ). Contours produced by using proposedadaptive weight ( red ), globally-optimum weight ( cyan ), best fixed-weight ( blue ).Ground truth contour is shown in green . Note the improved regularization usingour method.
Fig. 6.
Segmenting a natural image. (a) Original leaf image. (b) Reliability cal-culated by our proposed method.
Fig. 7. ( Color figure, refer to e-copy ). Segmentation results of the leaf in Fig.6(a) from our method ( green ), fixed-weight of 1 ( black ), 0.5 ( blue ), and 0 ( red ). utomatic Spatially-Adaptive Balancing of Energy Terms 11 processing to 2D. By analyzing the spectral flatness of the observed signal, aspatially-varying evidence of signal versus noise is derived fully automatically(without any tuning), which we use to spatially adapt the regularization using aclear reliability metric. We note that there is no restriction against using othernoise estimation methods in our proposed reliability-modulated regularization,making our method of broader interest. Using a large synthetic dataset exhibitingextreme variations of image deterioration and boundary shape, we demonstratedstatistically significant reduction in segmentation error compared to using thebest fixed weight, and to a globally-optimal, spatially-varying approach that usesdynamic programming to optimize for the regularization weight in conjunctionwith contour.Our current work focused on minimal-path approaches for segmentation.However, our approach of reliability-based regularization is applicable to a widerange of energy-minimizing segmentation techniques. We are currently extend-ing our approach to other variational and graph-based segmentation approachessuch as [4][8][22]. Additionally, we intend to expand our technique to handle en-ergy functionals where multiple weights balance the different energy terms andto explore alternative reliability measures. Another important conclusion fromour work is that globally optimal weights do not appear to be desirable. Weintend to further explore this issue in more detail in the future. References
1. Burnham, K.P., Anderson, D.R.: Multimodel inference: Understanding AIC andBIC in model selection. Sociological Methods & Research (2) (2004) 291–3042. Akselrod-Ballin, A., Galun, M., Gomori, M.J., Brandt, A., Basri, R.: Prior knowl-edge driven multiscale segmentation of brain MRI. In Ayache, N., Ourselin, S.,Maeder, A.J., eds.: MICCAI (2). Volume 4792 of Lecture Notes in Computer Sci-ence., Springer (2007) 118–1263. Zhao, P., Yu, B.: Stagewise lasso. Journal of Machine Learning Research (2007)2701–27264. Kass, M., Witkin, A., Terzopoulos, D.: Snakes: Active contour models. Interna-tional Journal of Computer Vision (4) (1988) 321–3315. Caselles, V., Kimmel, R., Sapiro, G.: Geodesic active contours. InternationalJournal of Computer Vision (1) (1997) 61–796. Osher, S.J., Paragios, N.: Geometric Level Set Methods in Imaging, Vision, andGraphics. Springer-Verlag (2003)7. Pluempitiwiriyawej, C., Moura, J.M.F., Wu, Y.J.L., Ho, C.: STACS: New activecontour scheme for cardiac MR image segmentation. IEEE Transactions on MedicalImaging (5) (2005) 593–6038. Boykov, Y., Funka-Lea, G.: Graph cuts and efficient N-D image segmentation. Int.J. Comput. Vision (2) (2006) 109–1319. Barrett, W.A., Mortensen, E.N.: Interactive live-wire boundary extraction. Medi-cal Image Analysis (1997) 331–34110. McIntosh, C., Hamarneh, G.: Is a single energy functional sufficient? Adaptive en-ergy functionals and automatic initialization. In Ayache, N., Ourselin, S., Maeder,A.J., eds.: MICCAI (2). Volume 4792 of Lecture Notes in Computer Science.,Springer (2007) 503–5102 J. Rao, G. Hamarneh, and R. Abugharbieh11. Udupa, J.K., Grevera, G.J.: Go digital, go fuzzy. Pattern Recognition Letters (6) (2002) 743–75412. Dong, B., Chien, A., Mao, Y., Ye, J., Osher, S.: Level set based surface capturingin 3D medical images. In Metaxas, D.N., Axel, L., Fichtinger, G., Sz´ekely, G., eds.:MICCAI (1). Volume 5241 of Lecture Notes in Computer Science., Springer (2008)162–16913. Kokkinos, I., Evangelopoulos, G., Maragos, P.: Texture analysis and segmentationusing modulation features, generative models, and weighted curve evolution. IEEETrans. Pattern Analysis and Machine Intelligence (1) (2009) 142–15714. Malik, J., Belongie, S., Leung, T.K., Shi, J.: Contour and texture analysis forimage segmentation. International Journal of Computer Vision (1) (2001) 7–2715. Erdem, E., Tari, S.: Mumford-shah regularizer with contextual feedback. Journalof Mathematical Imaging and Vision (1) (2009) 67–8416. Dijkstra, E. : A note on two problems in connection with graphs. NumericalMathematics (5) (1959) 269–27117. Jayant, N.S., Noll, P.: Digital Coding of Waveforms. Prentice-Hall, EnglewoodCliffs (1984)18. Taubman, D.S., Marcellin, M.W.: JPEG 2000: Image Compression Fundamentals,Standards and Practice. Kluwer Academic Publishers, Norwell, MA, USA (2001)19. Poon, K., Hamarneh, G., Abugharbieh, R.: Live-vessel: Extending livewire forsimultaneous extraction of optimal medial and boundary paths in vascular images.In Ayache, N., Ourselin, S., Maeder, A.J., eds.: MICCAI (2). Volume 4792 ofLecture Notes in Computer Science., Springer (2007) 444–45120. Li, H., Yezzi, A.: Vessels as 4-D curves: Global minimal 4-D paths to extract 3-Dtubular surfaces and centerlines. IEEE Transactions on Medical Imaging (9)(2007) 1213–122321. Cocosco, C.A., Kollokian, V., Kwan, R.K.S., Evans, A.C.: BrainWeb: Online in-terface to a 3D MRI simulated brain database. In Friberg, L., Gjedde, A., Holm,S., Lassen, N.A., Nowak, M., eds.: Third International Conference on FunctionalMapping of the Human Brain, NeuroImage. Volume 5., Academic Press (1997)22. Chan, T.F., Vese, L.A.: Active contours without edges. IEEE Transactions onImage Processing10