Misdirected Registration Uncertainty
Jie Luo, Karteek Popuri, Dana Cobzas, Hongyi Ding, William M. Wells III, Masashi Sugiyama
MMisdirected Registration Uncertainty
Jie Luo , , Karteek Popuri , Dana Cobzas , Hongyi Ding William M. Wells III , and Masashi Sugiyama , Graduate School of Frontier Sciences, The University of Tokyo, Japan Radiology Department, Brigham and Women’s Hospital, Harvard Medical School,USA School of Engineering Science, Simon Fraser University, Canada Computing Science Department, University of Alberta, Canada Department of Computer Science, The University of Tokyo, Japan Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute ofTechnology, USA RIKEN Center for Advanced Intelligence Project, Japan [email protected]
Abstract.
Being a task of establishing spatial correspondences, medicalimage registration is often formalized as finding the optimal transforma-tion that best aligns two images. Since the transformation is such an es-sential component of registration, most existing researches conventionallyquantify the registration uncertainty, which is the confidence in the esti-mated spatial correspondences, by the transformation uncertainty. In thispaper, we give concrete examples and reveal that using the transformationuncertainty to quantify the registration uncertainty is inappropriate andsometimes misleading. Based on this finding, we also raise attention toan important yet subtle aspect of probabilistic image registration, that iswhether it is reasonable to determine the correspondence of a registeredvoxel solely by the mode of its transformation distribution.
Keywords:
Image registration, Uncertainty
Medical image registration is a process of establishing anatomical or functionalcorrespondences between images. It is often formalized as finding the optimaltransformation that best aligns two images [1]. Since many important clinicaldecisions or analysis are based on registered images, it would be useful to quantifythe intrinsic uncertainty, which is a measure of confidence in solutions, wheninterpreting the image registration results.Among all methods that characterize the uncertainty of non-rigid imageregistration, the most mainstream, or perhaps the most successful frameworkis probabilistic image registration (PIR) [2,3,4,5,6,7,8,9]. Unlike point-estimateregistration methods that report a unique set of transformation parameters,PIR models the transformation parameters as a random variable and estimatesa distribution over them. PIR methods can be broadly categorized into dis-crete probabilistic registration (DPR) and continuous probabilistic registration a r X i v : . [ c s . C V ] M a y Authors Suppressed Due to Excessive Length (CPR). The transformation distribution estimated by DPR and CPR have dif-ferent forms. DPR discretizes the transformation space into a set of displacementvectors. Then it uses discrete optimization techniques to compute a categoricaldistribution as the transformation distribution [2,5,6,9]. CPR is essentially aBayesian registration framework, with the estimated transformation given by amultivariate continuous posterior distribution [3,4,7,8]. A remarkable advantageof PIR is that its registration uncertainty can be naturally obtained from thedistribution of transformation parameters, and further utilized to benefit thesubsequent clinical tasks[4,10,11].
Related Work
Image registration refers to the process of finding spatial cor-respondences, hence the uncertainty of registration should be a measure of theconfidence in spatial correspondences. However, since the transformation is suchan essential component of registration, in the PIR literature, most existing worksdo not differentiate the transformation uncertainty from the registration uncer-tainty. Indeed, the conventional way to quantify the registration uncertainty isto employ summary statistics of the transformation distribution. Applications ofvarious summary statistics have been found in previous researches: the Shannonentropy and its variants of the categorical transformation distribution were usedto measure the registration uncertainty of DPR [5]. Meanwhile, the variance [3],standard deviation [8], inter-quartile range [4] and covariance Frobenius norm[7] of the transformation distribution were used to quantify the registration un-certainty of CPR. In order to visually assess the registration uncertainty, eachof these summary statistics was either mapped to a color scheme, or an objectoverlaid on the registered image. By inspecting the color of voxels or the ob-ject’s geometry, clinicians can infer the registration uncertainty, which suggeststhe confidence they can place in the registered image.It is acknowledged that registration uncertainty should be factored into clin-ical decision making. This work mainly investigates whether those summarystatistics of the transformation distribution truly give insight into the registra-tion uncertainty. If clinicians are misdirected from the registration uncertaintyto the transformation uncertainty, and hence be conveyed by the false amount ofuncertainty with respect to the established correspondence, it can cause detri-mental effects on their performance.In the following sections, we use concrete examples and reveal that using thetransformation uncertainty to quantify the registration uncertainty is inappro-priate and sometimes misleading. Based on this finding, we also raise attentionto an important yet subtle aspect of PIR, that is whether it is reasonable todetermine the correspondence of a registered voxel solely by the mode of itstransformation distribution.
Most existing works do not differentiate the transformation uncertainty from theregistration uncertainty. In this section, we give concrete examples and further isdirected Registration Uncertaintyisdirected Registration Uncertainty
Most existing works do not differentiate the transformation uncertainty from theregistration uncertainty. In this section, we give concrete examples and further isdirected Registration Uncertaintyisdirected Registration Uncertainty point out that it is inappropriate to quantify the registration uncertainty by thetransformation uncertainty. For the convenience of illustration, we use RandomWalker Image Registration (RWIR) method as the PIR scheme in all examples[2,5,6]. In the RWIR setting, let I f and I m respectively be the fixed and moving image I f , I m : Ω I → R , Ω I ⊂ R d , d = 2 or
3. RWIR discretizes the transformation spaceinto a set of K displacement vectors D = { d k } Kk =1 , d k ∈ R d . These displacementvectors radiate from voxels on I f and point to their candidate transformationlocations on I m . The corresponding label for d k , which can be intensity values ortissue classes at those locations, are stored in I = { I ( d k ) } Kk =1 . For every voxel v i , the algorithm computes a unity-sum probabilistic vector P ( v i ) = { P k ( v i ) } Kk =1 as the transformation distribution. P k ( v i ) is the probability of displacement vec-tor d k . In a standard RWIR, the algorithm takes a displacement vector thathas the highest probability in P ( v i ) as the most likely transformation d m . Thecorresponding label of d m in I is assigned to voxel v i as its established corre-spondence.Conventionally, the uncertainty of registered v i is quantified by the Shan-non entropy of the transformation distribution P ( v i ). Since RWIR takes d m asits“point-estimate”, the entropy provides a measure of how disperse the rest ofdisplacement vectors in D are from d m . If other displacement vectors are allequally likely to occur as d m , then the entropy is maximal, because it is com-pletely uncertain which displacement vector should be chosen as the most likelytransformation. When the probability of d m is much higher than the other dis-placement vectors, the entropy decreases, and it is more certain that d m is theright choice. For example, assuming P ( v l ) and P ( v r ) are two discrete transfor-mation distribution for voxels v l and v r respectively. As shown in Fig.1, P ( v l )is uniformly distributed, and its entropy is E ( P ( v l )) = 2. P ( v r ) has an obviouspeak, hence its entropy is E ( P ( v r )) ≈ .
36, which is lower than E ( P ( v l ). Fig. 1.
Discrete distribution P ( v l ) and P ( v r ). For a registered voxel, the entropy of its transformation distribution is usuallymapped to a color scheme. Clinicians can infer how uncertain the registration isby the color of that voxel. However, does the conventional uncertainty measure,
Authors Suppressed Due to Excessive Length
Fig. 2. (a)The RWIR setting of a hypothetical example; (b)Bar chart of the transfor-mation distribution P ( v ). which is the entropy of transformation distribution, truly reflect the uncertaintyof registration?In a hypothetical RWIR example, assuming v on I f is the voxel we want toregister. As shown in Fig.2(a), v ’s transformation space D = { d k } k =1 is a setof 6 displacement vectors. P ( v ) = { P k ( v ) } k =1 is the computed distribution of D . The corresponding labels for displacement vectors in D are image intensitiesstored in I = { I ( d k ) } k =1 . For clarity, suppose that there are only two differentintensity values in I , one is 50 and the other is 200. The color of squares inFig.2(a) indicates the appearance of that intensity value. We can observe that d has the highest probability in D , hence its corresponding intensity I ( d ) = 50will be assign to the registered v .Fig.2(b) is a bar chart illustrating the transformation distribution P ( v ).Although P ( v ) has its mode at P ( v ) , the whole distribution is more or lessuniformly distributed. The transformation distribution’s entropy E ( P ( v )) ≈ .
58 is close to the maximal. Therefore, the conventional uncertainty measurewill suggest that the registration uncertainty of v is high. Once clinicians knewits high amount of registration uncertainty, they would place less confidence in v ’s current appearance.The conventional way to quantify the registration uncertainty seems useful.However, its correctness is questionable. In the same v RWIR example, let’stake into account the intensity value I ( d k ) associated with each d k and forman intensity distribution. As shown in Fig.3(a), even if d , d , d and d aredifferent displacement vectors, they correspond to the same intensity value asthe most likely displacement vector d . As we accumulate the probability for allintensity values in I , it is clear that 50 is the dominate intensity. Interestingly,despite being suggested of having high registration uncertainty by the conven-tional uncertainty measure, the intensity distribution in Fig.3(b) indicates thatthe appearance of registered v is quite trustworthy. In addition, the entropyof the intensity distribution is as low as 0.63, which also differs from the highentropy value computed from the transformation distribution.This counter-intuitive example implies that high transformation uncertaintydoes not guarantee high registration uncertainty. In fact, the amount of trans-formation uncertainty can hardly guarantee any useful information about theregistration uncertainty at all. More precisely, in the PIR setting, the trans-formation R T is modeled as a random variable. The corresponding label R L , isdirected Registration Uncertaintyisdirected Registration Uncertainty
58 is close to the maximal. Therefore, the conventional uncertainty measurewill suggest that the registration uncertainty of v is high. Once clinicians knewits high amount of registration uncertainty, they would place less confidence in v ’s current appearance.The conventional way to quantify the registration uncertainty seems useful.However, its correctness is questionable. In the same v RWIR example, let’stake into account the intensity value I ( d k ) associated with each d k and forman intensity distribution. As shown in Fig.3(a), even if d , d , d and d aredifferent displacement vectors, they correspond to the same intensity value asthe most likely displacement vector d . As we accumulate the probability for allintensity values in I , it is clear that 50 is the dominate intensity. Interestingly,despite being suggested of having high registration uncertainty by the conven-tional uncertainty measure, the intensity distribution in Fig.3(b) indicates thatthe appearance of registered v is quite trustworthy. In addition, the entropyof the intensity distribution is as low as 0.63, which also differs from the highentropy value computed from the transformation distribution.This counter-intuitive example implies that high transformation uncertaintydoes not guarantee high registration uncertainty. In fact, the amount of trans-formation uncertainty can hardly guarantee any useful information about theregistration uncertainty at all. More precisely, in the PIR setting, the trans-formation R T is modeled as a random variable. The corresponding label R L , isdirected Registration Uncertaintyisdirected Registration Uncertainty Fig. 3. (a)Bar chart of the transformation distribution P ( v ) taking into account I ( d k ).The color of each bar indicates the appearance of I ( d k ); (b)Intensity distribution ofthe registered v . consisting of intensity values or tissue classes, is a function of R T , so it is alsoa random variable. Even if R T and R L are intuitively correlated, given differenthyper parameters and priors, there is no guaranteed statistical correlation be-tween these two random variables. Therefore, it’s inappropriate to measure thestatistics of R L by the summary statistics of R T .In practice, for many PIR approaches, the likelihood term is often based onvoxel intensity differences. In case there is no strong informative prior, theseapproaches tend to estimate “flat” transformation distribution for voxels in ho-mogeneous intensity regions. Transformation distributions of these voxels areusually more diverse than their intensity distributions, and therefore they aretypical examples of how the conventional uncertainty measure, that is usingthe transformation uncertianty to quantify the registration uncertainty, tends toreport false results [3,5].In the following real data example, as shown in Fig.4(a), I f and I m are twobrain MRI images arbitrarily chosen from the CUMC12 dataset. After perform-ing RWIR, we obtain the registered moving image I rm . To give more insight intothe misleading defect of conventional uncertainty measures, we take a closer lookat two voxels, v c at the center of a white matter area on the zoomed I rm , and v e near the boundary of a ventricle. As can be seen from Fig.4(b), the transforma-tion distribution of v c is more uniformly distributed than that of v e . Therefore,conventional entropy-based methods will report v c having higher registrationuncertainty than v e . However, like the hypothetical example in Fig.3, we takeinto account the corresponding intensities and form a new intensity distribution.Since the intensity distribution is no longer categorical, we can employ othersummary statistics, such as the variance, to measure the uncertainty. It turnsout that the registered v e has larger intensity variance than v c , which againreveals that the conventional uncertainty measure is misleading. Point-estimate registration methods output a unique transformation, and estab-lish the correspondence I rm by assigning the corresponding label of its transfor-mation to each voxel on I f . PIR methods output a transformation distribution,yet they still seek to establish a “point-estimate” correspondence. Since the Authors Suppressed Due to Excessive Length
Fig. 4. (a)Input and result of the CUMC12 data example; (b)The transformation dis-tribution of v c and v e in the RWIR; (c)Intensity distributions of registered v c and v e . Fig. 5. (a)The RWIR setting of the second hypothetical example; (b)Bar chart of thetransformation distribution P ( v ) taking into account I ( d k ); (c)Intensity distributionof the registered v . transformation mode is the most likely transformation, the common standardfor PIR to establish the correspondence I rm is assigning the corresponding la-bel of its transformation mode to each voxel on I f . However, is it reasonable todetermine the correspondence solely by the transformation mode?In another hypothetical example, assuming v on I f is the voxel we want toregister. As shown in Fig.5(a), the transformation D = { d k } k =1 is a set of 4displacement vectors. P ( v ) = { P k ( v ) } k =1 is the estimated distribution of D .The corresponding intensity labels of all displacement vectors in D are storedin I = { I ( d k ) } k =1 . In RWIR, the transformation mode d m is the displacementvector with the highest probability. Therefore, d is the transformation mode,and I ( d m ) = I ( d ) will be assigned to the registered v . The probability of d is considerably higher than that of other displacement vectors. Based on therelatively low entropy of the transformation distribution P ( v ), the intensity ofregistered v should be trustworthy. However, once again we take into accountthe intensity value I ( d k ) associated with each d k , and form an intensity dis-tribution. Surprisingly enough, Fig.5(c) shows that the corresponding intensityof the transformation mode I ( d m ) = 50 is no longer the most likely intensity.Displacement vectors d , d and d are all less likely transformations, yet theircombined corresponding intensities outweigh I ( d ). isdirected Registration Uncertaintyisdirected Registration Uncertainty
Fig. 4. (a)Input and result of the CUMC12 data example; (b)The transformation dis-tribution of v c and v e in the RWIR; (c)Intensity distributions of registered v c and v e . Fig. 5. (a)The RWIR setting of the second hypothetical example; (b)Bar chart of thetransformation distribution P ( v ) taking into account I ( d k ); (c)Intensity distributionof the registered v . transformation mode is the most likely transformation, the common standardfor PIR to establish the correspondence I rm is assigning the corresponding la-bel of its transformation mode to each voxel on I f . However, is it reasonable todetermine the correspondence solely by the transformation mode?In another hypothetical example, assuming v on I f is the voxel we want toregister. As shown in Fig.5(a), the transformation D = { d k } k =1 is a set of 4displacement vectors. P ( v ) = { P k ( v ) } k =1 is the estimated distribution of D .The corresponding intensity labels of all displacement vectors in D are storedin I = { I ( d k ) } k =1 . In RWIR, the transformation mode d m is the displacementvector with the highest probability. Therefore, d is the transformation mode,and I ( d m ) = I ( d ) will be assigned to the registered v . The probability of d is considerably higher than that of other displacement vectors. Based on therelatively low entropy of the transformation distribution P ( v ), the intensity ofregistered v should be trustworthy. However, once again we take into accountthe intensity value I ( d k ) associated with each d k , and form an intensity dis-tribution. Surprisingly enough, Fig.5(c) shows that the corresponding intensityof the transformation mode I ( d m ) = 50 is no longer the most likely intensity.Displacement vectors d , d and d are all less likely transformations, yet theircombined corresponding intensities outweigh I ( d ). isdirected Registration Uncertaintyisdirected Registration Uncertainty Fig. 6. (a)Input and result of the BRATS data example; (b,c,d,e)Intensity distributionsof v b , v c , v d and v e ; (f)Approximate locations of v b , v c , v d and v e . The above example implies that the corresponding label of the transforma-tion mode can differ from the most likely correspondence that is given by the fulltransformation distribution. This example makes sense because in the previoussection we have pointed out that, in PIR, the transformation R T and correspon-dence R L are both regarded as random variables. Since there is no guaranteedstatistical correlation between R T and R L , the mode of R T ’s distribution is notguaranteed to be the mode of R L ’s distribution.As illustrated in Fig.6(a), we generate another example that register a MRIimage I f , which is arbitrarily chosen from the BRATS dataset, with synthet-ically distorted itself using RWIR. In this example, we investigate intensitydistributions of four registered voxels v b , v c , v d and v e , which are shown inFig.6(b),(c),(d),(e) respectively. In Fig.6, the red circle indicates the Most LikelyIntensity (MLI) given by the full transformation distribution, the orange circleindicates the corresponding intensity of the transformation mode I ( d m ), and thegreen circle is the Ground Truth (GT) intensity. We can observe that for v b , theMLI and I ( d m ) are both equal to the GT. On the other hand, for v c , v d and v e ,their MLIs are indeed not equal to their I ( d m ). This experiment does supportour point of view that the corresponding label of the transformation mode I ( d m )is not guaranteed to be the most likely label given by the full transformationdistribution. However, at this stage, we can not conclude which one is betterwith respect to the registration accuracy for PIR.As we conduct more experiments, we come across another interesting finding.As can be seen in Fig.6(c), the MLI of registered v c is equal to the GT intensityand more accurate than I ( d m ). Yet for v d and v e , unexpectedly, it is their I ( d m ) more closer to the GT than their MLI. Voxels like v d and v e can befound very frequently in our experiments using other real data. This surprisingresult indicates that utilizing the full transformation distribution can actuallygive worse estimation than using the transformation mode alone.Some existing researches have reported that it was beneficial to utilize theregistration uncertainty, which is information obtained from the full transforma- Authors Suppressed Due to Excessive Length tion distribution, in some PIR-based tasks [9,10,11]. However, the above findingmake us wonder whether utilizing the full transformation distribution could al-ways improve the performance.It is noteworthy that the above finding is based on RWIR. In PIR, the corre-lation between the transformation R T and the correspondence R L is influencedby the choice of hyper parameters and priors. Other PIR approaches that usedifferent transformation, regularization and optimization models, hence havingdifferent hyper parameters and priors, can certainly yield different findings thanRWIR. However, we still suggest that researchers should analyze and investigatethe credibility of the full transformation distribution before using it. Previous studies don’t differentiate the transformation uncertainty from the reg-istration uncertainty. In this paper, we point out that, in PIR the transformation R T and the correspondence R L are both random variables, so it is inappropriateto quantify the uncertainty of R L by the summary statistics of R T . We havealso raised attention to an important yet subtle aspect of PIR, that is whetherit is reasonable to determine the correspondence of a registered voxel solely bythe mode of its transformation distribution. We reveal that the correspondinglabel of the transformation mode is not guaranteed to be the most likely cor-respondence given by the full transformation distribution. Finally, we share ourconcerns with respect to another intriguing finding, that is utilizing the fulltransformation distribution can actually give worse estimation.Findings presented in this paper are significant for the development of PIR.We feel it is necessary to share our findings to the registration community. References
1. Sotiras, A., Davatzikos, C., Paragios, N.: Deformable Medical Image Registration:A Survey. IEEE. TMI. 32(7), 1153–1190 (2013)2. Cobzas, D., Sen, A.: Random Walks for Deformable Registration. In: MICCAI.LNCS, vol. 6892, pp. 557–565. Springer, Toronto (2011)3. Simpson, I.J.A., Schnabel, J.A., Norton, I., Groves, A.R., Andersson, J.L.R., Wool-rich, M.W. : Probabilistic Inference of Regularisation in Non-rigid Registration.NeuroImage. 59, 2438-2451 (2012)4. Risholm, P., Janoos, F., Norton, I., Golby, A.J., Wells III, W.M.: Bayesian Charac-terization of Uncertainty in Intra-subject Non-rigid Registration. Med. Image Anal.17(5), 538-555 (2013)5. Lotfi, P., Tang, L., Andrews, S., Hamarneh, G.: Improving Probabilistic Image Reg-istration via Reinforcement Learning and Uncertainty Evaluation. In: MLMI. LNCS,vol. 8184, pp. 187–194. Springer, Nagoya (2013)6. Popuri, K., Cobzas, D., Jagersand, M.: A Variational Formulation for Discrete Reg-istration. In: MICCAI. LNCS, vol. 8151, pp. 187–194. Springer, Nagoya (2013)7. Wasserman, D., Toews, M., Niethammer, M, Wells III, W.M.: Probabilistic Diffeo-morphic Registration: Representing Uncertainty. In: WBIR. LNCS, vol. 8545, pp.72–82. Springer, London (2014) isdirected Registration Uncertaintyisdirected Registration Uncertainty