Correction of Chromatic Aberration from a Single Image Using Keypoints
CCorrection of Chromatic Aberration from a Single Image UsingKeypoints
BENJAMIN T. CECCHETTO,
Department of Computer Science, The University of British Columbia
In this paper, we propose a method to correct for chromatic aberration in asingle photograph. Our method replicates what a user would do in a photoediting program to account for this defect. We find matching keypoints ineach colour channel then align them as a user would.CCS Concepts: • Computing methodologies → Computationalphotography ; Image processing . Additional Key Words and Phrases: chromatic aberration, image pro-cessing, color,
Chromatic aberration (also known as colour fringing) is a phenom-ena where different wavelengths of light refract through differentparts of a lens system. Thus, the colour channels may not align asthey reach the sensor/film. This is most notable in cheaper lenses, itis also noticeable at higher resolutions. We desire an image free ofchromatic aberrations is simple, so all the planes are in focus. For asimple lens system this amounts to misaligned colour channels (red,green and blue). The misalignment is due to a uniform scaling andtranslation. There are many types of other chromatic aberrations.Complex lens assemblies introduce new distortions. We propose amethod to deal with the simple, more common scenario. We laterdiscuss possible ways to deal with more complex ones.
Many domains have sought to correct chromatic aberration. A majorgoal is to improve accuracy of measurement. This could be eitherin a lab or more constrained capture scenarios such as of space.One such method is in active optics [Willson and Shafer 1991]. Foractive optics, instead of taking one exposure for the colour channelssimultaneously, the user takes one for each exposure at differenttimes. Each time, the focus changes to match the channel’s optimalplane. This accounts for the aberration in the lens so the threechannels will align in the final result. This requires a change fromtraditional hardware as well as three exposures at three points intime. This limits capture for dynamic scenes, as the object may bemoving in that time span.Another proposed method is to model the edge displacementswith cubic splines to compute a nonlinear warp of the image [Boultand Wolberg 1992]. This method cannot handle blurring/defocus inthe image plane along with saturation effects. It breaks down in re-gions that are either underexposed or overexposed. A different tech-nique exists for correcting lateral chromatic aberration where onecalculates the aberrations and uses warping to compensate [Mallonand Whelan 2007]. This only works for calibrated cameras/imageswhich may not be workable for a more simple user.Chromatic aberrations can also determine depth informationwithout prior calibration [Garcia et al. 2000]. Recently, an analysisof aberrations show they are not a simple shift of colour planes. Wecan get a corrected result by computing a non-linear warp [Kang 2007]. This algorithm handles many different categorized cases ofaberrations. Its only limitations are artifacts caused by saturation,and slow, non-linear, performance.Since we want to deal with the simple case, we may want toremove perspective distortions. A few algorithms have solved thisproblem in the past [Swaninathan et al. 2003]. If we undistort a lensdistortion, we should end up with only needing to solve for thetranslation/scaling of the colour channels to align them.A recently proposed metric, L , can determine a depth matte froma colour-filtered aperture [Bando et al. 2008]. This metric measureshow collinear and correlated points are in 3D (colour space). Itis then used to find a disparity from their custom colour-filteredaperture. Instead of concentrating on the optical derivation, we instead con-sider an image-based one. An artist can align colour channels of animage to correct the aberration. They align the channels by movingthe corners of the images to transform each channel’s image. Weattempt to automate this artist driven result.The main idea is to align two of the colour channels to the thirdone. The green channel has the least amount of aberration as it is inthe middle of the colour spectrum. In some of the results, we matchto the red channel for simplicity. Choosing a different channel tomatch to results in a scaled version of the corrected image.Our solution is to find keypoints in all channels, and find wherethe keypoints match to the fixed channel. At the same time, weminimize the aberration metric L . This is like finding disparitiesin stereo, but in 3 channels instead of two images. Once we havethese matchings, we can prune the results based on how well thematchings are. Then we need to find a transformation from thenon-fixed colour planes to the fixed one. We restrict the transfor-mation to a scale and translation. We solve the issues that arisefrom saturation by ignoring saturated regions and their neighbours.Since the algorithm is linear, it is quite efficient. Before we start describing how to find the keypoints, and disparities,it is best to define how aligned the colour channels are. It is notsuitable to do cross-correlation, as that only tells us how alignedtwo colour channels are. We want to see how aligned three are. Asdescribed above, there exists such a metric L [Bando et al. 2008]. Fora given neighbourhood around a pixel ( x , y ) , with eigenvalues of thecovariance matrix trix λ i and covariance matrix diagonal elements σ r , σ д , σ b , L ( x , y ) = λ λ λ σ r σ д σ b a r X i v : . [ c s . G R ] F e b enjamin T. Cecchetto (a) (b) Fig. 1. (a) A photo with chromatic aberration. (b) A correction for it using our algorithm. [Foundation 2009] (a) (b)
Fig. 2. L values over the images in Figure 1. This value is essentially how collinear the colour points in thisneighbourhood are in RGB-space. These are visualized in Figure 3.The lower this value is, the more collinear the points are and thehigher the less collinear. As mentioned in the appendix of the paper,this can be considered to be related to cross-correlation, and is thusexactly what we want to use. How to choose this neighbourhood sizeis a different story. The smaller it is, the less statistics we have aboutthat neighbourhood and thus may have a worse matching. The largerit is, the better chance we have of a matching, however the longer ittakes. This value is bounded between 0 and 1, as mentioned in thepaper. If we show this for every pixel as in Figure 2, we can see thatimages without chromatic aberration show very little misalignmentover the whole image. Another justification of using this metricis that according to the colour lines model [Omer and Werman2004], colour points in RGB-space of the whole image will lie alongdifferent colour lines. If we look at smaller neighbourhoods, then wecan assume the points will also lie along either a line, or intersectionof lines. This measures the collinearity of those neighbourhoods. Ifwe search for an ideal alignment, then we want to maximize thecollinearity, thus minimizing L . The first task is to find regions where the keypoints would be useful.We want to find regions where the alignment measure L is very high,but at the same time we want to be certain there is a good possibilityfor a good alignment. Choosing regions in the image with high L values is costly, as we have to compute L for every pixel and examinethat image. Also, it is not guaranteed to give us a good pixel, such assmooth regions which may have ambiguous results. A good choicewould be to find high gradient regions and use points from those.Specifically where we know there is an edge nearby. This gives usa better match since we know the other two channels should havea high gradient in that region too. We randomly sample from thenorm of the gradient image with gradient sufficiently high within athreshold. In addition, if we want to pay the cost of the L image, wecan compute it only in regions where the norm of the gradient issufficiently high. If we multiply L by the norm of the gradient image,and threshold it, we can sample points that are highly unalignedwith enough information around them to be aligned. If we want to align two channels to the remaining one (say aligngreen and blue to the red channel), we need to shift over all combi-nations of possible windows, with different scales. The idea is wewant the neighbourhood of the green and blue channels to be corre-lated with the red, and those channels we know may be elsewherewith a different scale. Thus, we want to minimize the misalignment L ( x , y ) subject to a shifted and scaled window in both green andblue channels so we can write L ( x , y ; d Gx , d Gy , σ G , d Bx , d By , σ B ) orrection of Chromatic Aberration from a Single Image Using Keypoints Fig. 3. Colour points ( r , д , b ) in neighbourhood clusters around a pixel. Left:From a control image with no aberrations chosen randomly. Right: From aphoto with aberrations, chosen at regions of obvious colour fringing. (a) (b) Fig. 4. Zoomed in area from Figure 1. (a) Original photo. (b) Corrected.
With disparities d x , d y and relative scales σ . Where we iterateover all acceptable disparities (as in stereo) and all acceptable scales.We know however, that the disparities and cale difference should-nâĂŹt be too large (unless one has a truely horrible lens) so we canlimit the search to local neighbourhoods in that sixtuple. In fact, ifthe scale difference is decently small (which it usually is in the caseof aberrations) we can simply look for disparities to find the scaleaspect of the transform. Thus we can write L as L ( x , y ; d Gx , d Gy , d Bx , d By ) To be able to handle different scales, one could simply do an imagepyramid based approach as in other computer vision papers. This isunnecessary, as most aberrations are not that distant in the scaledomain. The reason why this works is if we have a perfect edge(all one colour on one side, then all the other colour on the otherside of the edge) in an image then the colours in the neighbourhoodwill cluster into two distinct regions. Since we are dealing with anatural image and edges are not perfect, these clusters will connectin a line as there will be a gradient from one colour to the other. Ifthe image is misaligned, then this region will become more spreadout. Now if we consider a multicoloured region, we get a morecomplicated shape. However, if we find disparities that minimizesthis clusterâĂŹs collinearity, we should get a better aligned imagesince weâĂŹre minimizing the spread of the whole shape.
Although weâĂŹve found keypoints and disparities for those key-points in the other two channels, they might not give us goodinformation. For example, if the best matching L value for the pointneighbourhood was high, we shouldnâĂŹt want to use that point asit is not a very well aligned point. We only want to use points thathave gone from high L value (unaligned) to low L value (aligned).Since we only chose points that are unaligned, we just need to pruneaway the points that remain unaligned. Thus we only choose pointswith a low enough new L value. Alternatively, since we know wewant to do a scale and translation for each channel (3 degrees offreedom for each) we only need 2 keypoints (each being 2 dimen-sional). Thus we could choose the 2 keypoints with the lowest L value. Other methods could include weighting the keypoints basedon the L parameter and havenâĂŹt been fully explored. In practice,thresholding by the right amount is sufficient for good results. Now that we have a set of good points, we can solve for the transfor-mation pretty easily. Let us consider the red and green channels fornow. We have ( p Rx , p Ry ) chosen in the fixed red channel and a pointtranslated by disparity in the green channel ( p Gx , p Gy ) . We have theequation: (cid:169)(cid:173)(cid:171) σ G t Gx σ G t Gy (cid:170)(cid:174)(cid:172) (cid:169)(cid:173)(cid:171) p Rx p Ry (cid:170)(cid:174)(cid:172) = (cid:169)(cid:173)(cid:171) p Gx p Gy (cid:170)(cid:174)(cid:172) Thus we can rearrange it for this point pair as: (cid:18) p Rx p Ry (cid:19) (cid:169)(cid:173)(cid:171) σ G t Gx t Gy (cid:170)(cid:174)(cid:172) = (cid:18) p Gx p Gy (cid:19) If we have a second point we can solve this system for σ G , t Gx , t Gy ,the scale, and translation respectively. In practice, weâĂŹd wantmore than just two points because the points might have only agood local solution and not global. Thus weâĂŹd want points fromdifferent regions of the image. enjamin T. Cecchetto (a) (b) (c) Fig. 5. (a) A synthetically aberrated mountain image via translation in different colour channels. (b) A corrected version using our algorithm. (c) The groundtruth image. (a) (b) (c)
Fig. 6. (a) An image with exaggerated translation and scale shift in two colour channels. (b) A corrected version using our algorithm. (c) The same correctedversion showing disparities and keypoints. (a) (b) (c)(d) (e) (f)
Fig. 7. A set of extreme results. (a) An image taken with a lens with large chromatic aberrations. (d) A corrected version after using our algorithm. Similarly for(b) and (e). (c) An image with distortion [Art and the Zen of Design 2009] and our algorithm result in (f). Note the properly aligned stand but misaligned horsehead. orrection of Chromatic Aberration from a Single Image Using Keypoints So far this seems to work quite well as seen in Figure 1 and moreclosely in Figure 4. In Figure 5 we have a synthetic image beingcorrected. As we can see the correction is quite similar to the originalimage. A lot of the high frequency details are a little blurred becausethe misaligned image has a lower resolution than the original imageand thus we cannot get those details back. In Figure 6 we have alarger scale change between the different channels. Our algorithmsolves this pretty well too. We can see the keypoints in the thirdimage of this figure and some matchings arenâĂŹt always correctsuch as the one in the bottom left pointing in the wrong direction.In Figure 7, there are images with extreme lens distortion where ouralgorithm is expected to fail. Some of these include photos takenwith a lens assembly that exaggerates chromatic aberrations. Thesephotos are blurry because it is very hard to focus with this assembly.On the top we have the original photos and the bottom we have thecorrected versions. Notice that the box is actually worse than theoriginal whereas the tripod has a slight improvement. These photosare hard to deal with because the aberration is not as apparent sinceall of the colour channels are blurry.
If we reduce the neighbourhood size to compute L , we get more false-positives in the correlations. This is true because more disparitiesgive a low number. The colours will cluster into a spherical region ina smooth neighbourhood, whereas we want lines. It will just choosea random disparity in this case. For the blurry images in Figure 7, thisis a similar phenomena and there isnâĂŹt enough information in theimage for this method to work well and reliably. Many disparities inthis image regardless of direction gave L a value below 0.01 whereless than that is considered a âĂŹgoodâĂŹ alignment in regionswith more detail.It was mentioned earlier that we may weight the keypoints andtheir appropriate disparities based on how much they reduced thealignment measure. Each row of equation 2 would be multipliedby a function of its associated L . Different linear and squared errorweighting based on L have been attempted with little change inresults. One could try to normalize the weights somehow instead ofjust using L directly.Also, there might be other statistical methods to explore to prunethe keypoints such as computing the translation/scaling and gettingrid of the outliers using RANSAC. The computationally expensivepart of this algorithm is determining the disparities where we have a4D loop (without handling extremely large scale changes). After wehave the disparities itâĂŹs much quicker to deal with the keypointdata, especially since we need so few keypoints.Another option to consider speedups is perhaps a hierarchicalapproach. One could try to solve the problem with a reduced resolu-tion image and gradually work our way to the full resolution imageusing the lower resolution information.What isnâĂŹt clear but is worth exploring is a statement made ear-lier saying that unwarping a distorted image will yield an image witha chromatic aberration that can be corrected with a scale/translation.This is worth exploring, as the undistortions relatively fast as wellas this algorithm. Another approach to find keypoints would be to segment theimage into many cells and pick an appropriate keypoint from eachas in section 3.3. This will allow enough information from differentparts of the image to make a global warp more accurate than justchoosing random points in the acceptable regions.One other thing that has been explored is doing the same thing ingradient domain. Some initial results have shown that it didnâĂŹtwork as well as the natural image formations as in our figures. Itis worth exploring trying to align the gradients in a different way,perhaps using chamfer alignment on the edges of the channels. We have presented an method that corrects chromatic aberrationsin a single image without any use of calibration. Also, since thismethod is keypoint based and linear, the method is efficient. Ourmethod also handles the case of saturation, since it can ignore thoseregions by not choosing keypoints near them.
ACKNOWLEDGMENTS
Wolfgang Heidrich for the idea for the project. Gordon Wetzsteinfor his lens.
REFERENCES
Art and the Zen of Design. 2009.
Chromatic aberration and the DMC-FZ18
ACM SIGGRAPH Asia 2008 papers . 1–9.Terrance E Boult and George Wolberg. 1992. Correcting chromatic aberrations usingimage warping.. In
CVPR . 684–687.Wikimedia Foundation. 2009.
Chromatic Aberration (comparison) . https://en.wikipedia.org/wiki/Chromatic_aberration
Proceedings 15th International Conference onPattern Recognition. ICPR-2000 , Vol. 1. IEEE, 762–765.Sing Bing Kang. 2007. Automatic removal of chromatic aberration from a single image.In . IEEE, 1–8.John Mallon and Paul F Whelan. 2007. Calibration and removal of lateral chromaticaberration in images.
Pattern recognition letters
28, 1 (2007), 125–135.Ido Omer and Michael Werman. 2004. Color lines: Image specific color representation.In
Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision andPattern Recognition, 2004. CVPR 2004. , Vol. 2. IEEE, II–II.R Swaninathan, Michael D Grossberg, and Shree K Nayar. 2003. A perspective ondistortions. In2003 IEEE Computer Society Conference on Computer Vision and PatternRecognition, 2003. Proceedings.