LLocal Fourier Slice Photography
CHRISTIAN LESSIG ∗ , Institute for Simulation and Graphics, Otto-von-Guericke-Universität Magdeburg, Germany (cid:1)(cid:1)(cid:1)(cid:1)(cid:1)(cid:1)(cid:1)(cid:1)(cid:1)(cid:1)(cid:1)(cid:1)(cid:1)(cid:1)(cid:1)(cid:1) (cid:2)(cid:2)(cid:2)(cid:2)(cid:2)(cid:2)(cid:2)(cid:2)(cid:2)(cid:2)(cid:2)(cid:2)(cid:2)(cid:2)(cid:2)(cid:2) (cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3) (cid:4)(cid:4)(cid:4)(cid:4)(cid:4)(cid:4)(cid:4)(cid:4)(cid:4)(cid:4)(cid:4)(cid:4)(cid:4)(cid:4)(cid:4)(cid:4) (cid:1) (cid:5) = (cid:2) (cid:5) = (cid:3) (cid:5) = (cid:4) (cid:5) = L err x10 Fig. 1.
Right:
Refocused images obtained with local Fourier slice photography directly from a sparse wavelet representation of the light field.
Left:
Reconstructionerror as a function of the nonzero coefficients in the spare representation, demonstrating that the image fidelity degrades gracefully as storage requirementsare reduced. The plot also shows the output sensitivity of our technique, with the largest error obtained when the in-focus region in the image is largest.
Light field cameras provide intriguing possibilities, such as post-capture re-focus or the ability to synthesize images from novel viewpoints. This comes,however, at the price of significant storage requirements. Compression tech-niques can be used to reduce these but refocusing and reconstruction requireso far again a dense pixel representation. To avoid this, we introduce localFourier slice photography that allows for refocused image reconstructiondirectly from a sparse wavelet representation of a light field, either to obtainan image or a compressed representation of it. The result is made possibleby wavelets that respect the “slicing’s” intrinsic structure and enable us toderive exact reconstruction filters for the refocused image in closed form.Image reconstruction then amounts to applying these filters to the lightfield’s wavelet coefficients, and hence no reconstruction of a dense pixelrepresentation is required. We demonstrate that this substantially reducesstorage requirements and also computation times. We furthermore ana-lyze the computational complexity of our algorithm and show that it scaleslinearly with the size of the reconstructed region and the non-negligiblewavelet coefficients, i.e. with the visual complexity.CCS Concepts: •
Computing methodologies → Computational pho-tography ; Image compression ; •
Mathematics of computing → Compu-tation of transforms .Additional Key Words and Phrases: light field camera, Fourier slice theorem,wavelets
ACM Reference Format:
Christian Lessig. 2019. Local Fourier Slice Photography. 1, 1 (October 2019),16 pages. https://doi.org/10.1145/1122445.1122456
Author’s address: Christian Lessig, [email protected], Institute for Simu-lation and Graphics, Otto-von-Guericke-Universität Magdeburg, Universitätsplatz 2,Magdeburg, 39106, Germany.Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from [email protected].© 2019 Copyright held by the owner/author(s). Publication rights licensed to ACM.XXXX-XXXX/2019/10-ART $15.00https://doi.org/10.1145/1122445.1122456
Light field cameras, which record the full four-dimensional plenopticfunction, open up many possibilities for both consumer, e.g. [Nget al. 2005], and professional applications, e.g. [Levoy et al. 2006].Prime examples are post-capture refocus and the ability to obtainimages where every depth is in focus. The possibilities, however,come at the price of considerable storage requirements for the lightfield data sets. Compression techniques can alleviate these but imagereconstruction and light field processing typically require again adense representation.To avoid this, we propose local Fourier slice photography , an al-gorithm to compute refocused images directly from a light field’ssparse wavelet representation. Our work draws inspiration fromNg’s seminal Fourier slice photography [2005] where image recon-struction is performed in the frequency domain using the projec-tion slice theorem. We combine this work with a recent advance-ment of the slice theorem [Lessig 2018a] that uses carefully chosenwavelets to allow for an efficient projection from a signal’s com-pressed wavelet representation. To apply this result to refocusedimage reconstruction, we extend it to a sheared, local projection sliceequation that establishes closed-form, shear-dependent reconstruc-tion kernels for the projected signal. With these, a refocused imagecan be obtained directly from a light field’s compressed waveletcoefficients using an inverse transform. We also derive an exten-sion that enables one to directly obtain sparse, refocused imagesfrom a sparse light field data set. Our experimental results confirmthat our approach yields high fidelity, refocused images directlyfrom compressed light fields without the need to obtain a densepixel representation. They also demonstrate that errors that ariseat high compression rates mainly manifest themselves through losthigh-frequency detail, i.e. without distracting artifacts.The sparsity that reduces storage requirements also reduces com-putational costs. We show this experimentally and verify it througha theoretical analysis that establishes a linear dependence on thenumber of nonzero wavelet coefficients. Because of the spatial lo-calization of the wavelets, the costs of our technique depend on thelight field’s angular resolution. This was not the case for Fourierslice photography [Ng 2005] although there one cannot easily take , Vol. 1, No. 1, Article . Publication date: October 2019. a r X i v : . [ c s . G R ] O c t • Lessig !!" ! frequency domain(1) (2) (3) (4) Fig. 2. Overview of our approach: (1) A light field ℓ ( x , y , u , v ) in the two plane parameterization is represented using polar wavelets ψ s ( x , u ) and ψ r ( y , v ) (defined in Eq. 1), exploiting the separability of the refocusing problem. (2) The wavelets are defined in polar coordinates in the Fourier domain. They arehence naturally compatible with the restriction to a line through the origin which implements refocused image reconstruction in the Fourier domain [Ng2005]. (3) The restriction of the sheared polar wavelet ˆ ψ s ( S − T ( ξ x , ξ u )) to the ξ x axis defines the one dimensional wavelet ˆ ζ αs ( ξ x ) = ˆ ψ s ( S − T ( ξ x , )) (andanalogous for ( y , v ) ). (4) The inverse Fourier transform of ˆ ζ αs ( ξ x ) , which can be computed in closed form, provides the exact, spatial reconstruction filters toobtain a refocused image from the wavelet coefficients ℓ sr of the light field. advantage of redundancy in the data. The localization also enablesus to obtain all-focus images, which is not possible using Fourierslice photography.A conceptual overview of our approach is provided in Fig. 2 andthe computations required in an implementation are summarizedin Algorithm 1. The remainder of the paper, which expounds on thedetails, is structured as follows. After reviewing related work in thenext section, we provide in Sec. 3 the necessary background on thepolar wavelets that are used in our work. In Sec. 4 we derive thesheared local Fourier slice equation and develop our technique toobtain a refocused image directly from a light field’s sparse waveletrepresentation. Experimental results on refocused image reconstruc-tion and all-focus images as well as details on our reference im-plementation are presented in Sec. 5. We conclude in Sec. 7 with adiscussion of possible directions for future work. In computer graphics, light fields were introduced by Levoy andHanrahan [1996] and Gortler et al. [Gortler et al. 1996]. Initially,these were mainly of academic interest, e.g. [Chai et al. 2000], but inthe 2000s practical means to capture real-world light field data setsbecame available [Ng et al. 2005; Venkataraman et al. 2013; Wilburnet al. 2005]. With these, the generation, processing, and display oflight fields has become an important research direction [Ihrke et al.2016; Wu et al. 2017]. In the following, we will therefore focus onrelated work most pertinent to our own.Ng [2005] showed that post-capture refocus can be formulated inthe frequency domain using the Fourier projection slice theoremand that this provides an asymptotic speedup compared to the pixeldomain. Our work is inspired by Ng’s and we combine it with arecent result in optics [Lessig 2018a] that extends the slice theoremto a spatially localized form using wavelets. The extension relieson the use of polar wavelets, which is a family of wavelets definedseparably in polar coordinates in the Fourier domain [Unser andChenouard 2013; Unser et al. 2011; Unser and Van De Ville 2010]. These include a wide range of steerable wavelets [Freeman andAdelson 1991; Perona 1991; Simoncelli and Freeman 1995] as well ascurvelets [Candès and Donoho 2005a,b] and ridgelets [Candès andDonoho 1999; Donoho 2000]. The present work also exploits theseparability of polar wavelets in polar frequency coordinates andwe extend [Lessig 2018a] to include the shearing that implementsrefocusing. We also benefit from the sparsity available with curvelet-like constructions [Candès and Donoho 2004; Donoho 2000], whichensures an efficient sparse representation of light field data sets.Light field imaging in the Fourier domain was also consideredby Shi et al. [2014]. Their objective was to circumvent the sparsitydegradation that results when the discrete instead of the continuousFourier transform is used in numerical calculations. This is no issuefor our technique since our reconstruction kernels are obtainedin the continuous Fourier domain. Furthermore, while Shi et al.require a nonlinear optimization to obtain sparsity we use simplethresholding and rely on the compatibility of polar wavelets with thestructure of natural images in frequency space [Candès and Donoho2005a; Mallat 2009, Ch. 9]. The design of of light field cameras andtheir sensors has been analyzed comprehensively by Liang andRamamoorthi [2015]. Although these authors also perform theiranalysis in the Fourier domain, this work is orthogonal to ours sincewe assume we have a preprocessed light field data set as input.For image synthesis, the frequency representation of the lightfield has been analyzed in a series of papers starting with the semi-nal work by Durand et al. [2005]. The work showed, for example,that the shearing that implements refocusing in the Fourier domainis the general expression for the transport of the light field in thetwo plane parametrization [Chai et al. 2000]. Closest to our work areFourier-based approaches for depth of field rendering [Belcour et al.2013; Lehtinen et al. 2011; Soler et al. 2009]. This work, however,aims at finding optimal sampling rates for the light field in MonteCarlo renderers while we assumes a (largely noise free) light fieldon the camera is available. Because of the curse of dimensionality,image synthesis applications also typically do not employ an explicit , Vol. 1, No. 1, Article . Publication date: October 2019. ocal Fourier Slice Photography • 3
Fig. 3.
Left:
Conceptual construction of polar wavelets using window func-tions separable in polar coordinates.
Right:
Directional, curvelet-like polarwavelets in the frequency domain (middle) and the spatial domain (right). basis representation of the light field, which is a key part of thepresent work. Sen, Darabi, and Xiao [Sen et al. 2011] used compres-sive sensing to reduce the sampling rate for depth of field rendering.The polar wavelets employed in our work provide a sparse repre-sentation for image and light field data that would be well suitedfor compressive sensing.Vagharshakyan, Bregovic, and Gotchev [2018] proposed the useof shearlets for light field reconstruction from a limited set of per-spective views. Shearlets can be seen as a stereographic projectionof polar wavelets with the directional localization controlled by theparabolic scaling also used for curvelets. The authors exploit thesparsity afforded by the shearlet transform to obtain an efficientalgorithm for the reconstruction. However, they do not exploit thatimage reconstruction is naturally formulated in polar coordinatesin the frequency domain, which at the heart of our work. In fact, tothe best of our knowledge, with shearlets no closed form solutionfor the reconstruction kernels would be available.Learning-based techniques for image reconstruction from lightfields have also received considerable attention in recent work,e.g. [Kalantari et al. 2016; Levin et al. 2008; Xu et al. 2018; Yoon et al.2015]. The objective there is typically to perform the reconstructionfrom a reduced set of measurements, and it is hence orthogonal toour work. In the spirit of [Vagharshakyan et al. 2018], we believethat our polar wavelet representation of a light field might providea useful pre-processing for learning-based reconstruction since itremoves redundancy while respecting the intrinsic structure.A variety of approaches for the compression of light field datasets have been proposed in the literature [Viola et al. 2017; Wu et al.2017], for example adapting techniques used for image compression,e.g. [Alves et al. 2018]; developing custom ones for slices or the full4D light field, e.g. [Aggoun 2006; Aggoun and Mazri 2008; Contiet al. 2014; Kundu 2012]; or extending video compression schemesby exploiting that a light field can be seen as a sequence of imagesrecorded from a set of nearby vantage points, e.g. [Dai et al. 2015;Vieira et al. 2015]. Our use of polar wavelet for transform coding isdictated by our objective to refocus from the compressed represen-tation. However, we do not provide a full compressions technique,e.g. we do not consider the choice of color space, gamma correction,and quantization.
Wavelets are functions that are well localized in both the spatialand frequency domain. Through this, they enable an efficient andsparse representation of signals and can reduce the computationalcosts of numerical computations, e.g. [DeVore 2006; Stevenson 2009]. α = α = α = α = α = - -
10 10 20 - - Fig. 4. Reconstruction kernel ζ α , s ( x ) in Eq. 12 for different values of α . In multiple dimensions, wavelets are typically constructed as ten-sor products of one dimensional ones. Polar wavelets [Unser andChenouard 2013], in contrast, are defined separably in polar coordi-nates in Fourier space, which leads to many desirable properties.Polar wavelets are constructed using a compactly supported radialwindow ˆ h (| ξ |) , which controls the overall frequency localization,and an angular one, ˆ γ ( ¯ ξ ) , which controls the directionality, with ξ being the frequency variable and ¯ ξ = ξ /| ξ | . The mother wavelet isthus given by ˆ ψ ( ξ ) = ˆ γ ( ¯ ξ ) ˆ h (| ξ |) , cf. Fig. 3, left. The whole family offunctions used to represent arbitrary signals is then generated bydilation by 2 j , j ≥
0, translation by k ∈ Z , and rotation by θ t with t ∈ { , · · · , T j } .The angular window ˆ γ ( ¯ ξ ) is conveniently described using a Fourierseries in the polar angle θ ξ , i.e ˆ γ ( ¯ ξ ) = (cid:205) n β tj , n e inθ ξ . In the fre-quency domain, a polar wavelet is thus given byˆ ψ s ( ξ ) ≡ ˆ ψ jkt ( ξ ) = j π (cid:16) (cid:213) n β tj , n e inθ ξ (cid:17) ˆ h ( − j | ξ |) e − i ⟨ ξ , j k ⟩ (1)with the Fourier series coefficients β tj , n controlling the angular local-ization. In the simplest case, β n is the Kronecker delta δ n and onehas isotropic, bump-like wavelet functions, cf. Fig. 2, left. Conversely,when the support of the β tj , n is over all integers Z then one candescribe angular windows that are compactly supported in the polarangle θ ξ . Eq. 1 then encompasses ridgelets [Candès and Donoho1999; Donoho 2000] and second generation curvelets [Candès andDonoho 2005b], cf. Fig. 3, right, which provide quasi optimally sparserepresentations for image-like signals.A beneficial property of polar wavelets is that their spatial rep-resentation, given by the inverse Fourier transform of Eq. 1, canbe computed in closed form. Using the Fourier transform in polarcoordinates one obtains [Lessig 2018b] ψ s ( x ) ≡ ψ jkt ( x ) = j π (cid:213) n i n β tj , n e inθ x h n (| j x − k |) (2)where h n (| x |) is the Hankel transform of ˆ h (| ξ |) of order n . For ˆ h (| ξ |) we employ the window proposed for the steerable pyramid [Freemanand Adelson 1991; Portilla and Simoncelli 2000] since h n (| x |) thenhas a closed form expression [Lessig 2018b]. Note that the angularlocalization of the wavelets, which is described by the β tj , n in Eq. 1and Eq. 2, is invariant under the Fourier transform and only modifiedby the factor of i n = e inπ / that implements a rotation by π / , Vol. 1, No. 1, Article . Publication date: October 2019. • Lessig As indicated in Fig. 3, left, the radial mother window ˆ h (| ξ |) isdefined away from the origin and dilation by 2 − j centers it at higherand higher frequencies as j ≥ д (| ξ |) is required that has support inthe disk-neighborhood around ξ = h (| ξ |) ). Thetranslates of the inverse Fourier transform of ˆ д (| ξ |) yield the socalled scaling functions ϕ k ( x ) and in our case these will always beisotropic. To simplify notation we will write ψ − , k ( x ) ≡ ϕ k ( x ) ; werefer to [Daubechies 1992, Ch. 5] for more details on the concept ofscaling functions.The polar wavelets in Eq. 2 together with the just defined scalingfunctions provide a Parseval tight frame for L ( R ) . Thus any func-tion f ( x ) ∈ L ( R ) can be represented as [Unser and Chenouard2013] f ( x ) = ∞ (cid:213) j = − (cid:213) k ∈ Z T j (cid:213) t = (cid:10) f ( y ) , ψ jkt ( y ) (cid:11)(cid:124) (cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32) (cid:123)(cid:122) (cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32) (cid:125) f jkt ψ jkt ( x ) . (3)where ⟨ , ⟩ is the L inner product. Although the above frame isredundant, since it is Parseval tight it still affords most of the con-veniences of an orthonormal basis, e.g. the primary and dual framefunctions coincide and the norm of the signal equals those of theexpansion coefficients. For an isotropic frame the redundancy isthereby 1 + / + / + · · · = / T j >
1. As for curvelets,where T j grows according to a parabolic scaling law as a functionof j , anisotropic representations are typically sparser, which cancompensate for the larger redundancy.Since the definition of polar wavelets in the frequency domainuses a compactly supported radial window ˆ h (| ξ |) , the wavelets havenon-compact support in space. Thus, a finite signal representationis not exact, since it requires a truncation of the basis functions. Fig. 5. Image formation model used in our work. Shown is a cross sectionof the optical system with the red line representing a typical ray passingthrough the lens. Fig. 6. Contour plots of curvelet-like, directional polar wavelets in the fre-quency domain at two different orientations (different rows) and with andwithout shear (different columns) for α = . . Shown are also the shearedreconstruction filters ˆ ζ αs . It is apparent that only those ˆ ψ s ( x ) with an ori-entation close to the ξ x -axis (dashed grey), i.e. to the slicing direction, yield ˆ ζ αs that contribute to the projected signal. Nonetheless, with a sufficiently large apron region around an im-age, an arbitrary accuracy can be attained; in our experiments anapron of 4 pixels sufficed to meet the requirements of photographicapplications. To obtain the wavelet representation of a signal, wecompute it using a coarse-to-fine, fast wavelet transform-like algo-rithm where on each level the computations can either be performedusing discrete convolutions with filter taps in the spatial domain orby multiplication with the windows in the frequency domain.To simplify notation, we will typically employ the multi-index s = ( j , k , t ) introduced in Eq. 1 and, when confusion might arise,write s = ( j s , k s , t s ) . The index s runs over the set S that a prioriincludes all scales, translations, and orientations. The cardinality ofa set will be denoted by | · | , e.g. |S| . In this section we derive local Fourier slice photography, our imagereconstruction technique from a sparse polar wavelet representa-tion of a light field. We begin by fixing notation and recalling theimage formation model. At the end we analyze the computationalcomplexity of the technique as well as sources of error.
We use the two-plane parametrization [Chai et al. 2000] for the lightfield ℓ ( x , y , u , v ) with ( x , y ) being the coordinates on the image plane , Vol. 1, No. 1, Article . Publication date: October 2019. ocal Fourier Slice Photography • 5 Fig. 7. Projection of sheared 2D Gaussian (left, bottom). The maximum errorwith the sheared local Fourier slice equation is . × − . and ( u , v ) those on the lens, see Fig. 5 for a schematic depiction. Wealso assume ℓ ( x , y , u , v ) is non-zero only over the camera sensorin ( x , y ) and over the lens in ( u , v ) and that it already contains thecos ( θ ) / F foreshortening factor, where θ is the angle between theray and the image plane normal.For image formation we use the same model as [Ng 2005]. Hence,the image I ( x , y ) is obtained from the light field as I ( x , y ) = α ∫ R u ∫ R v ℓ (cid:0) x / α + ( − / α ) u , y / α + ( − / α ) v , u , v (cid:1) du dv where α = F / F ′ is the refocusing parameter, with F and F ′ being theoriginal and new distance of the sensor to the lens plane, respectively,cf. Fig. 5. By changing the notation for the light field to ℓ ( x , u ; y , v ) ,the last equation can be written more compactly as I ( x , y ) = ∫ R u ∫ R v ℓ (cid:0) S α ( x , u ) T ; S α ( y , v ) T (cid:1) du dv (4)where the shear matrix S α is given by S α = (cid:18) / α − / α (cid:19) . (5)The shear S α amounts to the transport of the light field in the camerafrom the original sensor location to the refocused one [Chai et al.2000; Durand et al. 2005].To obtain a form of ℓ ( x , u ; y , v ) that is amenable to compression,e.g. by thresholding small coefficients, we represent it in the polarwavelets introduced in Sec. 3. Respecting the separable structure ofthe refocusing in Eq. 4, that is performing one transform over x - u and a second one over y - v , we obtain ℓ ( x , u ; y , v ) = (cid:213) ( s , r )∈L ℓ sr ψ s ( x , u ) ψ r ( y , v ) (6)where L is the index set for the representation that runs over allcoefficients s = ( j s , k s , t s ) and r = ( j r , k r , t r ) . Inserting this repre-sentation into Eq. 4 yields I ( x , y ) = ∫ R u ∫ R v (cid:213) ( s , r )∈L ℓ sr ψ s (cid:0) S α ( x , u ) T (cid:1) ψ r (cid:0) S α ( y , v ) T (cid:1) du dv (7) = (cid:213) ( s , r )∈L ℓ sr ∫ R u ψ s (cid:0) S α ( x , u ) T (cid:1) du ∫ R v ψ r (cid:0) S α ( y , v ) T (cid:1) dv . // Input: Sampled light field ℓ in ( x , y ) × ( u , v ) parameterization Precomputation: ( ℓ ) // 1. Wavelet projection of light field in ( x , u ) and ( y , v ) ℓ ψ = { ℓ sr } = FWT ( ℓ ) ∈ R | S |×| S | // 2. Computation of reconstruction filter, possibly sampling // it for interpolation ζ αs ( x ) = F − x (cid:16) ˆ ψ s (cid:0) S − Tα ( ξ x , ξ u ) T (cid:12)(cid:12) ξ u = (cid:1)(cid:17) end // Input: shear α , resolution N for reconstruction Reconstruction: ( α , N ) // 1. Determine locations for reconstruction x = (cid:8) − − j max − N / α , · · · , − j max − N / α (cid:9) // 2. Evaluate projection of sheared locations k αs = j s P x ( S − k s ) // 3. Evaluate ζ αs ( x − k αs ) for all x i and k αs Z = { ζ αs ( x i − k αs )} i , s ∈ R n ×|S | // 4. Reconstruction of n × n raw image I = Z ℓ ψ Z T endAlgorithm 1: Local Fourier slice photography algorithm for α -sheared image reconstruction from wavelet compressed represen-tation (for single color channel).The last equation shows that it suffices to determine the effect ofthe sheared projection for the basis functions ψ s and ψ r and thatthis can be done independently for each of them.Next, we will thus study general, two-dimensional sheared projec-tion using polar wavelets. This will yield our sheared local Fourierslice equation. We return to imaging in Sec. 4.3. Let f ( x , u ) be a two dimensional signal. We consider the shearedprojection д ( x ) = ∫ R u f (cid:0) S α ( x , u ) T (cid:1) du (8)where S α is given by Eq. 5. When f ( x , u ) is represented in polarwavelets the equation becomes д ( x ) = (cid:213) s ∈S f s ∫ R u ψ s (cid:0) S α ( x , u ) T (cid:1) du . (9)Inspired by Ng’s work [2005], we will seek a numerically practicalsolution to Eq. 9 in the Fourier domain; a depiction of our approachis shown in Fig. 2. By the Fourier slice theorem, the integral in thelast equation can be written as ∫ R u ψ s (cid:0) S α ( x , u ) T (cid:1) du = F − x (cid:16) α − ˆ ψ s (cid:0) S − Tα ( ξ x , ξ u ) T (cid:12)(cid:12) ξ u = (cid:1)(cid:17) (10)where S − Tα ( ξ x , ξ u ) T | ξ u = is a linear slice in ξ x - ξ u frequency space.By expanding ˆ ψ s using the definition in Eq. 1, defining ξ = ( ξ x , ) T , , Vol. 1, No. 1, Article . Publication date: October 2019. • Lessig α = α = α = α = α = - -
10 10 20 P x ( S - k s )- - - Fig. 8. Coupling coefficient γ sq in Eq. 17 for j s = j q and k q = as afunction of P x ( − j S − k s ) . and writing out the inverse Fourier transform we obtain ∫ R u ψ s (cid:0) S α ( x , u ) T (cid:1) du = j α − ( π ) / (cid:213) n β tj , n (11) × ∫ R ξx e inθ S − Tα ξ ˆ h (cid:0) − j | S − Tα ξ | (cid:1) e − i ⟨ ξ , j S − k ⟩ e iξ x x dξ x (cid:124) (cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32) (cid:123)(cid:122) (cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32) (cid:125) ζ α , ns ( x ) = ζ α , nj s , t s (cid:0) x − P x ( j S − k s ) (cid:1) , that is, ζ s , αn ( x ) is the inverse Fourier transform of the ξ x -dependentterm. Importantly, for our choice of the radial window the ζ s , αn ( x ) have a closed form expression, see Appendix A, and the above equa-tion can therefore easily be realized numerically. Furthermore, theoriginal shift k s of ψ s becomes after slicing P x ( − j s S − k s ) , where P x is the projection onto the x -axis. This means the shape of the ζ s , αn ( x ) remains independent of k s . Eq. 11 furthermore shows thatthe angular localization coefficients β tj , n are invariant under theinverse Fourier transform. The sheared projection of an arbitrarypolar wavelet ψ x ( x ) is thus ζ αs ( x ) ≡ j α − ( π ) / (cid:213) n β tj , n ζ α , ns ( x ) = ∫ R u ψ s (cid:0) S α ( x , u ) T (cid:1) du . (12)and inserting Eq. 12 into Eq. 9 we obtain д ( x ) = (cid:213) s ∈S f s ∫ R u ψ s (cid:0) S α ( x , u ) T (cid:1) du = (cid:213) s ∈S f s ζ αs ( x ) . (13)Eq. 13 is our sheared local Fourier slice equation with the ζ αs ( x ) being the reconstruction filters that implement projection directlyfrom the wavelet representation. A simple verification of Eq. 13 for atwo dimensional Gaussian, for which the ground truth has a closedform solution, is provided in Fig. 7.The reconstruction kernels ζ αs ( x ) are wavelet-like in that they arecompactly supported in the frequency domain around ξ x = j s andwell localized in the spatial domain. The former holds since ˆ ζ αs ( ξ ) isa slice of a compactly supported wavelet centered at this frequency,cf. Fig. 6, and the latter since the sliced window has the same decayas ˆ ψ s ( ξ ) , see Fig. 4. The wavelet-like properties enable a local, sparsereconstruction with a coefficient f s only having a non-negligibleeffect to д ( x ) in a small neighborhood around the projected loca-tions 2 − j s P x ( S − k s ) . Thus, only the wavelet coefficients defined atlocations in a sheared tube in the u -direction above a location x ′ contribute to д ( x ′ ) , see Fig. 9 left. We can therefore think of Eq. 13as a wavelet representation of the projected signal with the wavelets !! !" " ! (cid:1)!!" !! !" ! ! !!!"! " Fig. 9.
Left:
Original lattice of frame function locations (grey) and its shear(light red) for α = . . The projection (red circles) of the sheared latticeare on a denser grid, ensuring that the reconstruction filters ζ αs are on agrid that is sufficiently fine to meet the Nyquist criterion for the increasedbandlimit that arises through the shearing. The bluish region schematicallyindicates the set of coefficients that determines the projection at the bluishlocation on the x -axis, indicating the locality of our approach. Right:
Thediscrete Fourier transform in Fourier Slice Photography [Ng 2005] yields asignal on a discrete lattice of frequencies (grey). The evaluation directions S − T ξ does not lie on the lattice and the resampling introduces error. ζ αs ( x ) which are located at non-canonical locations P x ( − j s S − k s ) .In fact, with isotropic wavelets and α = д ( x ) = (cid:213) j , k x (cid:16) (cid:213) k u f j , k x , k u (cid:17)(cid:124) (cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32) (cid:123)(cid:122) (cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32) (cid:125) f j , k x ζ j , k x ( x ) , (14a)which is a standard, one dimensional wavelet representation of д ( x ) .Our result then also coincides with those of [Lessig 2018a].Next to the spatial position of ψ s ( x ) , the contribution of a coeffi-cient f s to the projected signal д ( x ) also depends on the orientationof ψ s ( x ) , or, equivalently, on the corresponding β tj , n , cf. Eq. 12. Asis apparent from Fig. 6, the magnitude of the reconstruction filters ζ αs ( x ) is non-negligible only when the direction S − Tα ξ overlaps theeffective support of the wavelets in the frequency domain. Whencurvelet- or ridgelet-like wavelets are used, i.e when ˆ ψ s ( ξ ) has strongdirectional localization, then only a sheared wedge or a small num-ber of wedges from the polar tiling of the frequency plane haveeffective support over the direction. Hence, only these directionsneed to be considered in the sum over s in the sheared Fourier sliceequation in Eq. 13.Our result relies on the use of polar wavelets that, through theirdefinition in polar coordinates in frequency space, are compatiblewith the intrinsic structure of the projection, i.e. with a restriction toa line through the origin in the Fourier domain. With tensor productwavelets, e.g. using Daubechies-type discrete wavelets, one wouldhave a different, skew slicing through the axis-aligned frequencywindow for every α . The reconstruction kernel would then nothave a closed form solution and, since the wavelets have no simpledescription, even determining them numerically would be difficult. , Vol. 1, No. 1, Article . Publication date: October 2019. ocal Fourier Slice Photography • 7 (cid:1) (cid:1) (cid:1) (cid:1) (cid:1)(cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1)(cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1)!!" (cid:2) (cid:1)!$"! (cid:3) (cid:1)!$"$ (cid:1)! (cid:4) "(cid:1)! !(cid:4) (cid:5)" ! ! (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1)(cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) ! ! (cid:2) ! " (cid:3) ! (cid:1) !" !! %+!!" !! '+!!" !! ,+!!" !! "+""""!""+""""! (cid:1)(cid:2)(cid:3)(cid:4)(cid:5)(cid:3)(cid:6)(cid:7)(cid:1)(cid:5)(cid:2)(cid:8)(cid:6)(cid:9)(cid:7)(cid:10)(cid:2)(cid:11)(cid:1)(cid:2)(cid:3)(cid:4)(cid:5)(cid:3)(cid:6)(cid:7)(cid:1)(cid:5)(cid:2)(cid:8)(cid:6)(cid:9)(cid:7)(cid:10)(cid:2)(cid:11) (cid:1)! (cid:12)(cid:1)(cid:4)(cid:5)(cid:12)(cid:6)(cid:5)(cid:6)(cid:13)(cid:6)(cid:5)(cid:6)(cid:11)(cid:9)(cid:6)(cid:12)(cid:9)(cid:4)(cid:3)(cid:6)(cid:14)(cid:6)(cid:5)(cid:5)(cid:2)(cid:5)(cid:1)(cid:2)(cid:3)(cid:4)(cid:5)(cid:3)(cid:6)(cid:7) " (cid:15)(cid:16)(cid:17) (cid:12)(cid:9)(cid:4)(cid:3)(cid:6)(cid:14)(cid:6)(cid:5)(cid:5)(cid:2)(cid:5)(cid:1)(cid:2)(cid:3)(cid:4)(cid:5)(cid:3)(cid:6)(cid:7) (cid:1)! (cid:12)(cid:1)(cid:4)(cid:5)(cid:12)(cid:6) " (cid:15)(cid:16)(cid:17) (cid:1)(cid:2)(cid:2) (cid:3)(cid:2)(cid:2) (cid:4)(cid:2)(cid:2) (cid:5)(cid:2)(cid:2) (cid:6)(cid:2)(cid:2)!(cid:2)(cid:7)(cid:3)(cid:2)(cid:7)(cid:3)(cid:2)(cid:7)(cid:5)(cid:2)(cid:7)(cid:8)(cid:2)(cid:7)(cid:9)(cid:1)(cid:7)(cid:2) (cid:1)!(cid:1)(cid:2)(cid:3) Fig. 10. Sheared projection with anisotropic, curvelet-like wavelets.
Left:
Reconstruction error as a function of the compression rate.
Middle: ∥ ζ jkt ( x )∥ as afunction of the orientation t ; as expected from Fig. 6 the norm is non-negligible only when t is approximately aligned with the projection direction. Right:
Reconstruction with all directions and only those where ∥ ζ jkt ( x )∥ is non-negligible. We now return to refocused image reconstruction from the polarwavelet representation of a light field. Using the local Fourier sliceequation and inserting Eq. 7 into Eq. 12 we obtain I ( x , y ) = (cid:213) ( s , r )∈L ℓ sr ζ αs ( x ) ζ αr ( y ) . (15)The ζ αs ( x ) thus provide the reconstruction filters that implement α -refocused image reconstruction directly from the wavelet coef-ficients ℓ sr of the light field, that is without the need to obtain adense pixel representation. In practice, a displayable representationof an image is obtained by evaluating Eq. 15 for every pixel (possi-bly with multiple samples to increase the quality). We summarizethe computations that are required for image reconstruction usingEq. 15 in Algorithm 1.Eq. 15 can also be thought of as a separable wavelet representationof the image with the wavelets ζ αs ( x ) and ζ αr ( y ) located at the non-dyadic locations P x ( − j S − k ) . Note that the spatial and directionallocality ζ αs ( x ) discussed in Sec. 4.2 immediately carries over to Eq. 15and, for example, a coefficient ℓ sr contributes to the image I ( x , y ) only when ˆ ψ s ( ξ x , ξ u ) and ˆ ψ r ( ξ y , ξ v ) are oriented along the slicingdirection, cf. again Fig. 6. It is often useful to directly determine the compressed representationof an image from a compressed light field, i.e. without first havingto obtain a dense pixel representation as an intermediate step. Wewill show next how the result of Sec. 4.2 can be extended towardsthis end.We assume that the light field is again provided in the polarwavelet representation in Eq. 6 so that the projected light field isgiven by Eq. 15. Assuming a separable wavelet basis ψ q ( x ) ψ p ( y ) isused for the image, the corresponding expansion coefficients aregiven by ℓ qp = (cid:68) I ( x , y ) , ψ q ( x ) ψ p ( y ) (cid:69) (16) = (cid:213) ( s , r )∈L ℓ sr (cid:68) ζ αs ( x ) , ψ q ( x ) (cid:69) x (cid:68) ζ αr ( y ) , ψ p ( y ) (cid:69) y . By introducing γ sq = (cid:68) ζ αs ( x ) , ψ q ( x ) (cid:69) (17)we can write this more compactly as ℓ qp = (cid:213) ( s , r )∈L ℓ sr γ sq γ rp . (18)When ψ q ( x ) and ψ p ( y ) are one dimensional, bandlimited “polar”wavelets that use the same radial window ˆ h ( ξ x ) as in Sec. 3, thenthe γ -coefficients in Eq. 17 can be computed in closed form; theexpressions are provided in the accompanying Mathematica code.By the compact support of the wavelets in the frequency domain, the γ sq are then, for moderate α , non-negligible only when max ( , | j q − |) ≤ j s ≤ | j q + | ; when | α − | is large then max ( , | j q − |) ≤ j s ≤| j q + | holds. A plot of the γ sq as a function of P x ( − j S − k s ) fordifferent values of α and j s = j q is provided in Fig. 8. As can be seenthere, the coefficients have a shape similar to ζ αs ( x ) , since ˆ ζ αs ( ξ ) isessentially a smoothed box function, and in particular they havethe same spatial decay.A consequence of Eq. 18 is that sparsity in the wavelet repre-sentation of a reconstructed image is induced by sparsity in thoseof the light field. In particular, for fixed ( q , p ) only the ℓ sr with | k q − P x ( S − Tα k s )| ≲ − j q and | k p − P x ( S − Tα k r )| ≲ − j p contributeto ℓ qp , since the wavelets decay in space with dyadic dilation fromlevel to level. Hence, when all ℓ sr in the sheared tube above k q and k p are negligible, cf. Fig. 9, then also ℓ qp is negligible. The coeffi-cient ℓ qp can also become small through cancellation, since boththe ℓ sr and γ sq are signed. Intuitively, this happens, for example,when a region is defocused in the sheared projection and hencethe wavelet coefficients on fine levels there have to be small. Suchdecay estimates are beyond the scope of the present paper and willbe investigated elsewhere; existing result in this direction can befound in [Quinto 1993, 2007]. In the following, we will analyze the computational complexity ofAlgorithm 1. We assume that the reconstruction is performed for aregion A ⊆ [ , ] , which is potentially a subset of the normalizedoriginal image plane [ , ] , with size | A | and that R is the samplingrate per pixel in the reconstructed image. The input light field is , Vol. 1, No. 1, Article . Publication date: October 2019. • Lessig !" !"" (cid:1)!!" (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1)(cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) (cid:1) ! ! (cid:2) ! " (cid:3) ! (cid:1) !" Fig. 11. Sheared projection for the two dimensional dragon image in Fig. 12using classical and local Fourier slice photography.
Right:
Projections anddifferences to pixel projection for a representative α . Bottom left: L , L , and L ∞ errors of the projection as a function of the shearing angle α . Top left:
Projection as a function of the interpolation order for classical Fourier slicephotography. assumed to have resolution N x × N x × N u × N u × ζ αs ( x ) ζ αr ( y ) has effective support 2 − j s W × − j r W , where W is awindow function-dependent constant (see Fig. 4). The computa-tional costs K sr that arise for each coefficient ℓ sr are thus K sr = c ζ − j s − j r W R (19)where c ζ is the cost for evaluating ζ αs ( x ) at one point. The numberof coefficients on level ( j s , j r ) in a dense wavelet representation is (cid:12)(cid:12) L j s , j r (cid:12)(cid:12) = · − ( j max − j s ) − ( j max − j r ) N x N u T j s T j r | A | (20)since at each level the resolution is reduced by a factor of 2 in eachdirection. As before, T j is the number of different orientations onlevel j and the factor of 4 = · / K j s , j r = K sr (cid:12)(cid:12) L j s , j r (cid:12)(cid:12) for image reconstruction on level ( j s , j r ) are for a dense polar wavelet representation thus K j s , j r = · − j max j r + j s N x N u T j s T j r | A | c ζ W R . (21)For the costs K j max across all levels we then have K j max = · P j max N x N u T j s T j r | A | c ζ W R . (22)where P j max = − j max ( j max + − ) .Two facts can be exploited to reduce the costs K j max . First, withanisotropic, curvelet-like frame functions only the α -sheared onesoverlapping the slicing direction are required, cf. Fig. 6. These areof order one so that the costs K αj max are K αj max = · c T P j max N x N u | A | c ζ W R . (23)where c T = O( ) . The increased redundancy of a directional repre-sentation system hence does not affect the costs since the number offrame functions overlapping the slicing direction remains constant.The second reduction of the costs K j max arises from the sparsityin a light field’s wavelet representation, i.e. that only a sparse set Fig. 12. Comparison between our approach and reconstruction in the pixeldomain for α = . . The left image is split in the middle between the pixelreconstruction (left) and the polar wavelet reconstruction (right). The imageon the right shows the difference image, magnified by a factor of 25. Theerror is on the level of those incurred by naïve slicing in the pixel domain. L α , ϵj s , j r , consisting of non-negligible ones with respect to a compres-sion parameter ϵ (in the simplest case of hard thresholding, ϵ givesthe threshold), suffices to represent the signal. The ratio betweenthe number of coefficients in the full and sparse representations isknown as compression ratecr ϵj s , j r = |L j s , j r ||L α , ϵj s , j r | . (24)Assuming it is independent of the level, i.e. cr ϵj s , j r = cr ϵ , whensparsity and directionality are exploited the costs K α , ϵj max then are K α , ϵj max = · c T P j max N x N u cr ϵ | A | c ζ W R . (25)With basis dependent constants being ignored, this becomes inbig-O notation K α , ϵj max = O (cid:16) P j max N x N u cr ϵ | A | R (cid:17) . (26)This shows that the costs scale linearly in the area that is to be re-constructed and the sampling rate and as 1 / cr ϵ in the compressionrate. Thus, as the number of coefficients in the sparse represen-tation L α , ϵj s , j r decreases and the compression rate grows also thecomputational costs decrease.The complexity of Ng’s Fourier slice photography [Ng 2005] is,in our notation, O (cid:0) | A | R N x (cid:1) . Although both works exploit that theprojection becomes trivial in the Fourier domain, the spatial localiza-tion of the wavelets in our approach introduces again the directionalresolution parameter N u in Eq. 26. However, with the wavelets wealso have a dependence on the compression rate cr ϵ . This can com-pensate for the N u factor, depending on the rate cr ϵ that can beattained for the light field. Although theoretical characterizations ofcr ϵ exist, see e.g. [Mallat 2009, Ch. 6, Ch. 9], these require technicalassumptions about the signal that are difficult to precisely meet inpractice. We will hence not pursue a further theoretical analysishere. Nonetheless, the practical utility of wavelets for the compres-sion of image like signals, and hence that significant compressionrates can be attained, is by now well established, as is evidenced bytheir use in the JPEG2000 standard. , Vol. 1, No. 1, Article . Publication date: October 2019. ocal Fourier Slice Photography • 9 Fig. 13. Reconstructions with subsets of all wavelet levels, demonstrating that not all are required to obtain acceptable reconstructions. Furthermore, the errorincreases gracefully as the number of levels decreases. The contribution by the individual levels is shown in Fig. 19.
Ng [2005] discusses two sources of error for image reconstruction,namely roll off error and aliasing. We avoid roll-off error sinceour reconstruction kernels ζ αs are the exact ones for our analysiswavelets, which in turn provide a Parseval tight frame; when thewavelets are used up to level j max then all signals with bandlimit2 j π can be represented exactly. Aliasing is of concern because theshearing can increase the bandlimit (the change in the bandlimitcan be seen, e.g. in bottom right plot in Fig. 6). In our approach,this implies that for α < ζ αs have a bandlimit beyond thoseof the original polar wavelets ψ s . Consequently, they also need tobe defined over a finer grid than the original wavelets to allow forperfect reconstruction. But, as shown in Fig. 9, left, the shearingalso affects the grid over which the basis functions are defined andthrough this the ζ αs are inherently defined on a lattice that has theappropriate density.Beyond roll off error and aliasing, a third source of error in factarises in Ng’s work [2005]. As depicted in Fig. 9, right, the dis-crete Fourier transform yields a lattice of discrete frequencies in theFourier domain. Except in the trivial case when α =
1, the slicingdirection S − T ξ does not lie on the lattice and the resampling ontoa regular grid along the slicing direction provides an additionalsource of error. This error does not occur in our approach since ourbasis functions are defined in the continuous Fourier domain andwe analytically compute the projection as a function of α ∈ R .The main source of inaccuracies in our approach are in practicethose introduced by a finite truncation of the basis functions or,equivalently, of the filter taps used in the fast transform. As discussedin Sec. 3, these can be ameliorated by using appropriate paddingand filter sizes. In this section we present experimental results that verify the prac-ticality of the image reconstruction technique developed in the lastsection. Additional results as well as the raw images for most of thepresented figures are provided in the supplementary material.
We implemented a custom light field film class for the pbrt ren-derer [Pharr and Humphreys 2010] so that we could easily vary thespatial and angular resolutions as well as the optical properties oflight field data sets. The film class directly records light fields inthe two plane parameterization, applies the foreshortening factorof cos θ / F and also computes a depth map for the scene. We usedpbrt to generated synthetic light fields for a dragon scene and a villainterior, both with a resolution of 1025 × × × ×
3. Wealso performed experiments with photographic light fields from theStanford Lytro light field archive. We selected 20 light fields fromdifferent categories, all with a resolution of 541 × × × × α values and tone mappingparameters) were applied in each case, which might be sub-optimalin individual instances. Brightness variations that can be seen insome of the results for varying α stem from (independent) tonemapping. We developed a Mathematica reference implementation of Algo-rithm 1 (available in the supplementary material) and a basic, multi-threaded C++ implementation. Since the reconstruction filters ζ αs ( x ) are relatively expensive to evaluate, cf. Appendix A, we sampledthem in a preprocessing step and interpolated at runtime (the er-ror introduced through the interpolation was below 10 − and thusnegligible for photographic applications). Post-processing bias wasavoided by using one sample per pixel for image reconstruction andno interpolation filter on the image plane. “Reference” solutionswere similarly computed using naïve projection in the pixel domainwithout filtering of the light field data sets or the pixel data.For the polar wavelets, we used filter taps of size 81 ×
81 and,as mentioned earlier, an apron of 4 pixels. Larger values did notimprove the reconstruction. Note that our algorithm for refocusedimage reconstruction is itself parameter free. http://lightfields.stanford.edu/LF2016.html, Vol. 1, No. 1, Article . Publication date: October 2019. ●●●●●●●●●●●●●●●● ■■■■■■■■■■■■■■■■ ◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆ ▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲ ▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼ ● flowers _ plants1 ■ flowers _ plants2 ◆ flowers _ plants3 ▲ flowers _ plants4 ▼ flowers _ plants5 L err x10 α = ●●●●●●●●●●●●●●●● ■■■■■■■■■■■■■■■■ ◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆ ▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲ ▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼ ● flowers _ plants1 ■ flowers _ plants2 ◆ flowers _ plants3 ▲ flowers _ plants4 ▼ flowers _ plants5 L err x10 α = ●●●●●●●●●●●●●●●● ■■■■■■■■■■■■■■■■ ◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆ ▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲ ▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼ ● flowers _ plants1 ■ flowers _ plants2 ◆ flowers _ plants3 ▲ flowers _ plants4 ▼ flowers _ plants5 L err x10 α = ●●●●●●●●●●●●●●●● ■■■■■■■■■■■■■■■■ ◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆ ▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲ ▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼ ● flowers _ plants1 ■ flowers _ plants2 ◆ flowers _ plants3 ▲ flowers _ plants4 ▼ flowers _ plants5 L err x10 α = Fig. 14. Log-linear plots of L reconstruction error as a function of the nonzero coefficients for five different photographic light fields (flowers_plants_1to flowers_plants_5 from left to right) and α = . , . , . , . (from top to bottom). The vertical grey dashed line in the plots on the left indicates thecoefficients required for the uncompressed representation. Shown are reconstructed images obtained from the full wavelet representation to show the effect ofchanging α (see Fig. 18 for a depiction of compressed ones). To study the effect of sparsification on image reconstruction qual-ity we implemented a simple transform coding scheme with hardthresholding, i.e. we set to zero all coefficients whose magnitudeis below a threshold ϵ dependent on the light field’s L norm tocompensate for overall brightness differences between the data sets.Results will be reported using either the number of nonzero coeffi-cients in the light fields ϵ -sparse wavelet representation (denotedby nzs) or the compression rate (denoted by cr), i.e. the number ofnonzero coefficients in a sparse over the total number in a densewavelet representation. The number of nonzero coefficients pro-vides an indication of the storage requirements although it is anupper bound since our wavelet representation only provides thetransform coding step of a full compression scheme and we do notconsider blocking, quantization, entropy coding and other aspectsthat are critical in practical compression algorithms. To demonstrate the correctness of the sheared local Fourier sliceequation as well as to gain some understanding of various concep-tual aspects we performed experiments on two dimensional signalsyielding a one dimensional projection.
Basic verification.
We verified the correctness of the local Fourierslice equation using the sheared projection of a two dimensionalGaussian for which an analytic solution exists. As shown in Fig 7, ourreconstruction matches the analytic one very well with a maximumerror on 1 . × − . This is of the same order as the reconstruction error of the 2D input signal, and hence attributable to inaccuraciesin the transform yielding the wavelet coefficients. Non-smooth signals and effect of α . To obtain insights on the be-havior of our technique for “natural images” as well as to understandthe effect of α on the reconstruction quality we considered the pro-jection of a monochromatic dragons scene image (derived from theimage in Fig. 12). The results in Fig. 11, right, demonstrate that theerror is smaller than what can be perceived visually and on thesame order as differences resulting with different reconstructionkernels for the pixel domain projection (not shown). Shown in thefigure is also a quantitative analysis of the error as a function of α , demonstrating that only a mild dependence on the angle exists.The results, furthermore, reveal that our technique provides slightlylower error rates than classical Fourier slice photography [Ng 2005].The plot on the top left of Fig. 11 indicates that the higher errors forthe technique result from the interpolation from the axis-alignedDFT grid that is required for it, see Fig. 9. Since our wavelets aredefined in the continuous Fourier domain and the restriction to aslice is performed there, no such interpolation is required. Sparsity and Directionality.
For the dragon scene image we alsostudied the effect of sparsity as well as the gains that are possibleusing oriented, curvelet-like frame functions. The left plot in Fig. 10shows the error as a function of the compression rate with hardthresholding. As one would expect for a wavelet representation,very accurate reconstructions are possible with a small fraction ofthe full coefficient set. Furthermore, the error increases smoothlywith the compression rate. The middle plot in Fig. 10 depicts the , Vol. 1, No. 1, Article . Publication date: October 2019. ocal Fourier Slice Photography • 11 ●●●●●●●●●●●●●●●● ■■■■■■■■■■■■■■■■ ◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆ ▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲ ▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼ ● general1 ■ general2 ◆ general3 ▲ general4 ▼ general5 L err x10 α = ●●●●●●●●●●●●●●●● ■■■■■■■■■■■■■■■■ ◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆ ▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲ ▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼ ● general1 ■ general2 ◆ general3 ▲ general4 ▼ general5 L err x10 α = ●●●●●●●●●●●●●●●● ■■■■■■■■■■■■■■■■ ◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆ ▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲ ▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼ ● general1 ■ general2 ◆ general3 ▲ general4 ▼ general5 L err x10 α = ●●●●●●●●●●●●●●●● ■■■■■■■■■■■■■■■■ ◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆ ▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲ ▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼ ● general1 ■ general2 ◆ general3 ▲ general4 ▼ general5 L err x10 α = Fig. 15. Log-linear plots of L reconstruction error as a function of the nonzero coefficients for five different photographic light fields (general_1 to general_5from left to right) and α = . , . , . , . (from top to bottom). The vertical grey dashed line in the plots on the left indicates the coefficients required forthe uncompressed representation. Shown are reconstructed images obtained from the full wavelet representation to show the effect of changing α (see Fig. 18for a depiction of compressed ones). norm ∥ ζ jkt ( x )∥ of the reconstruction kernel ζ jkt ( x ) as a function ofthe orientation t of the wavelets. Because of the localized supportof the directional wavelets in the frequency domain, cf. Fig. 6, only ζ jkt ( x ) whose orientation t matches the slicing direction S − Tα ξ arenon-negligible. The right plot in Fig. 10 verifies that orientations farfrom the projection direction do not have to be taken into accountfor reconstruction, i.e. the reconstruction error is sufficiently smallwhen these are ignored. Furthermore, since for typical values α does not fundamentally change the direction, some orientationsare irrelevant, e.g. t ≤ π / t ≥ π / ϵ -threshold. ∥ ζ jkt ( x )∥ can hencebe understood as a signal independent form of sparsification. Refocusing.
The reconstruction of a refocused image is providedin Fig. 12. The comparison to the projection in the pixel domain,also shown in the image, verifies that our result is visually indistin-guishable and that the differences between both are on the orderof what one would obtain with different reconstruction kernels forprojection in the pixel domain.Fig. 13 shows reconstructions with only a subset of levels andFig. 19 the contribution made by individual levels. It can be seenthat the error increases gracefully as one disregards finer levels withthe reconstructed images becoming smoother but largely withoutobjectionable artifacts, though some ringing can be observed for j ≤ − j ≤ . . .
43% of the computation time for all levels for j ≤ − j ≤ j ≤
1, respectively, substantial savings for out-of-focus regionsare possible when a depth map is known.
Sparsity.
Fig. 14 and Fig. 15 show the relationship between thenumber of nonzero coefficients and L reconstruction errors for 10different photographic light fields for α = . , . , . , .
35 (top tobottom in each figure). For α = .
6, in both figures one data set (thefifth in Fig. 14 and the third in Fig. 15) yields considerably largererrors than the other ones. For the two data sets large areas withhigh frequency foliage, requiring many small wavelet coefficientsfor an accurate representation, are in focus for α = . α . Correspondingly, one has a consistentlylow reconstruction error. For the remaining data sets there is foreach α one region with high visual complexity in focus and thereconstruction errors remain thus relatively constant. Qualitativelyequivalent results hold for the L and L ∞ norms. Results for 10 dif-ferent photographic light fields can be found in the supplementarymaterial.Fig. 18 provides a visual comparison of reconstructed images asthe sparsity in the light field’s wavelet representation increases. Thefigure demonstrates that the image fidelity degrades gracefully asthe compression rate increases. Furthermore, when the error be-comes visible then it amounts to a lack of high frequency features , Vol. 1, No. 1, Article . Publication date: October 2019. ����� ������� ��������� ���������� ����� ����� ��� × �� � ��� × �� � ��� × �� � ��� × �� � ��� × �� � ��� × �� � ������������������� � � ��� ������� ����� ������� ��������� ���������� ����� ����� � × �� � � × �� � � × �� � � × �� � � × �� � �������������� � � ��� ������ _ ������ _ � ( ���� ) � ������� _ � ( ������ ) Fig. 16. Comparison of the reconstruction error as a function of the nonzero coefficients in the sparse representation for our technique and the transformcoding step of JPEG. Note that for the JPEG-compressed light fields a dense pixel representation is required to reconstruct refocused images.
Left:
Resultsfor synthetic dragon light field with resolution × × × × (see Fig. 12). Right:
Log-linear plot for two representative photographic light fields(flowers_plants_4 (fourth column) in Fig. 14 (blue) and general_5 (fifth column) in Fig. 15 (yellow)). but largely without objectionable artifacts (image sequences as afunction of the compression rate are provided in the supplemen-tary material). The storage requirements for the coefficients of thecompressed light fields in Fig. 18 (with cr >
1) are 2 .
23 GB, 1 .
27 GB,660 MB, 122 .
39 MB, and 40 .
23 MB, respectively. As a comparison,the dense light field, required for projection in the pixel domainor Fourier slice photography, requires 13 GB of storage and theuncompressed wavelet representation 48 GB.In Fig. 16 we report the L error in the reconstructed image as afunction of the number of nonzero coefficients for our local Fourierslice photography and a JPEG-compressed representation of thelight field (similar to [Alves et al. 2018]). For the latter, the sparserepresentation was obtained by considering each slice in the lightfield (for fixed u-v index) as an image and applying the transformcoding step of JPEG, consisting of mask-weighted quantization inthe discrete cosine transform (DCT) domain over 8 × All-in-focus reconstruction.
Fig. 20, right, shows a reconstructionof the dragons scene as well as of the checkerboard ground planewith a depth dependent α value so that the entire scene is in focus.Slight artifacts are visible around the dragon silhouettes, since wedo not take the varying support of the ζ αs ( x ) into account and onlysample the depth map at the location of the reconstruction kernels. Performance.
Fig. 17 shows the relative execution time as a func-tion of the compression rate. Although our implementation is notparticularly optimized and we only use the Eigen library for the sparse wavelet representation, the results demonstrate that sparsitycan lead to a substantial reduction in computation time. The plotsalso show the execution time decreases approximately as 1 / cr, asone would expect from our analysis of the computational complexityin Sec. 4.5, see in particular Eq. 26.The absolute computation time of image reconstruction is cur-rently approximately two minute for a 1025 × × × × Our experimental results demonstrate the practical viability of Algo-rithm 1 for the reconstruction of refocused images from the sparsewavelet representation of a light field. We verified that high fidelityimages can be obtained from a highly sparse representation and thatthe error increases gracefully with the compression rate. Further-more, our experiments show that the error depends on the visualcomplexity of the in-focus region, which can be exploited when adepth map is available. Additionally, we demonstrated that simple (cid:31)(cid:31)(cid:31)(cid:31)(cid:31)(cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31)(cid:31)(cid:31)(cid:31)(cid:31)(cid:31)(cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31)(cid:31)(cid:31)(cid:31)(cid:31)(cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31)(cid:31)(cid:31)(cid:31)(cid:31)(cid:31)(cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31)(cid:31)(cid:31)(cid:31)(cid:31)(cid:31)(cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) (cid:31) flowers _ plants 1 (cid:31) flowers _ plants 2 (cid:31) flowers _ plants 3 (cid:31) flowers _ plants 4 (cid:31) flowers _ plants 5
200 400 600 800 1000 cr0.20.40.60.81.0rel. t
Fig. 17. Relative execution time for the light fields in Fig. 14. The resultsdemonstrate that the use of sparsity can lead to substantially lower com-putation times. See Fig. 18 for reconstructed images with the respectivecompression rates. , Vol. 1, No. 1, Article . Publication date: October 2019. ocal Fourier Slice Photography • 13 ●●●●●●●●●●●●●●●● ■■■■■■■■■■■■■■■■ ◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆ ● L ■ L ◆ L ∞ ( x0.05 ) x10 ●●●●●●●●●●●●●●●● ■■■■■■■■■■■■■■■■ ◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆ ● L ■ L ◆ L ∞ x10 Fig. 18. Reconstruction errors as a function of the number of nonzero coefficients in the sparse wavelet representation of the data sets and reconstructedimages for α = . for the dragon scene and α = . for the tree blossoms. Shown are also difference images compared to the uncompressed reference. Theresults demonstrate that a reconstruction from a sparse wavelet representation of the light fields yields high fidelity images and that even with a very smallnumber of nonzero coefficients, and a correspondingly high compression rates, the errors manifest themselves mainly as missing high frequency details, e.g.on the back of the red dragon, but there are largely no disruptive visual artifacts. Animations for the reconstructed image are provided in the supplementarymaterial. hard thresholding of polar wavelet coefficients is, at least for mod-erate to high compression rates, competitive with the transformcoding step of JPEG, which uses highly optimized masks.Our results also show that sparsity in the wavelet representationcan lead to a reduction in the computation time through the smallernumber of coefficients that has to be processed, although our imple-mentation is currently slower than projection in the pixel domainwhen a dense representation of a light field is directly available. Al-gorithm 1 is easily parallelized by exploiting that the reconstructionfor each pixel is independent, i.e. there is N x × N y data parallel work.This makes it directly amenable to a GPU implementation whereone could also take advantage of half-precision, which is sufficientto obtain artifact free images. This would also provide benefits onthe embedded processors typically available in cameras.Fig. 20 showed first results on the reconstruction of all-focus im-ages from the compressed wavelet representation. While currentlynot artifact free when the depth map contains discontinuities, the re-sults verify the potential of our approach to obtain all-focus images,which is not possible using Fourier slice photography where a fixed α has to be used. To remove the current artifacts, the depth mapneeds to be preprocessed in a mip-map-like manner so that an aver-age depth can be sampled at each wavelet level and it might also benecessary to restrict the support of wavelets at depth discontinuities. The local Fourier slice photography algorithm developed in theforegoing suggests many directions for future work. In our work, we consider the sparse wavelet representation oflight fields, which corresponds to the transform coding step of acompression technique. To make local Fourier slice photographypractical for applications, this has to be extended to a full compres-sion scheme by also considering, for example, blocking, quantization,and color coding. Our comparison to the transform coding step ofJPEG indicates already that the sparse representation will translateto significantly reduced storage requirements also in practice.Our technique would also benefit from additional work on thewavelet representation. For example, the radial window ˆ h (| ξ |) wecurrently employ does not provide very good decay in the spatialdomain and, similar to [Ward et al. 2015], one could investigate howbetter radial windows can be constructed. One should also furtherinvestigate the trade off between increased sparsity for directional,curvelet-like representations and the higher computational costsfor evaluating these. Our results in Sec. 4.4 showed that sparsityin the light field induces sparsity in the reconstructed image. Aquantitative description of this could potentially lead to a furtherreduction of the storage requirements as well as computation times.By representing the light field in polar wavelets, many existingtechniques for editing and processing are no longer directly avail-able. However, we believe that many of them can be translated tothe polar wavelet domain and it might, in fact, provides advantagesto conventional approaches. For instance, the feature-aware resiz-ing of Gastal and Oliveira [2017] is naturally formulated in polarcoordinates and could hence be performed directly in a sparse rep-resentation. We believe that other tasks, such as the shearlet-based , Vol. 1, No. 1, Article . Publication date: October 2019. Fig. 19. For the dragon scene in Fig. 13, contributions by different levelsfor j s = − , · · · , (columns) and j r = − , · · · , (rows) magnified by thefactor shown in the inset. The results verify that very fine levels can beomitted when performance is of importance. light field reconstruction from a sparse set of views in [Vaghar-shakyan et al. 2018] or learning based techniques such as [Kalantariet al. 2016] could also benefit from the efficacy of a polar waveletrepresentation.In the present work we considered refocused image reconstruc-tion. Another important application of light fields is novel viewsynthesis. It would be interesting to investigate if this can also beperformed directly from a sparse polar wavelet representation.Our current approach for image reconstruction exploits the sep-arability of the refocusing problem so that two dimensional polarwavelets are sufficient. This is sub-optimal concerning the compres-sion rates that can be attained for the light field and also since oneobtains a separable wavelet representation for the reconstructedimage. With 4D polar wavelets, which can be constructed as anextension of the polar wavelets used in the present work, cf. [Wardand Unser 2014], the entire light field could be represented in onewavelet basis and the projection of the data would then yield twodimensional, curvelet-like polar wavelets. In our opinion, this isboth theoretically and practically an interesting direction for futurework. In this paper, we presented local Fourier slice photography, an al-gorithm to reconstruct refocused images from a sparse waveletrepresentation of a light field. For this, we derived a sheared localFourier slice equation, which extends the local Fourier slice equationof [Lessig 2018a] to the case of sheared projection. The equation
Fig. 20. Reconstructions for α = . (left) and with α determined based ona depth map (right) to obtain all-focus reconstructions ( α is restricted to [ . , . ] which leaves the region nearest to the camera still out of focus).Because we use only a single high resolution depth map and do not considerthe variable support of the reconstruction kernels ζ αs as a function of level,artifacts are visible around the silhouettes of the dragons. provides analytic, wavelet-like reconstruction kernels for obtaininga refocused image directly from a light field’s wavelet coefficients.The direct reconstruction from a sparse representation is at the heartof our work and it avoids the need for a dense light field representa-tion that exists for techniques in the literature. We experimentallyverified that high fidelity images can be reconstructed from a highlysparse representations of a light field, providing the potential forsignificant storage requirements, and our results demonstrate thatimage quality degrades gracefully as the compression rate increases.Furthermore, also the computational costs can be reduced by directlyreconstructing images from the sparse wavelet representation. Weanalyzed this theoretically and demonstrated efficiency gains as afunction of compression rate experimentally. Because of the spatiallocalization of the wavelets, the costs of our technique depend onthe light field’s angular resolution. This was not the case for FourierSlice Photography [Ng 2005] but the localization enables us, forexample, to obtain all-focus images, which is not possible using thistechnique. ACKNOWLEDGMENTS
The comments by the anonymous reviewers helped to considerablyimprove the manuscript, in particular their insistence to try thealgorithm on photographic light fields. The Stanford light field groupis acknowledged for making the Lytro data sets available to thecommunity. First ideas for the project were developed while theauthor was a post-doc in Marc Alexa’s computer graphics group , Vol. 1, No. 1, Article . Publication date: October 2019. ocal Fourier Slice Photography • 15 at TU Berlin. Many thanks also to Eugene Fiume for continuingsupport.
REFERENCES
A. Aggoun. 2006. A 3D Dct Compression Algorithm For Omnidirectional IntegralImages. In , Vol. 2. IEEE, II–517–II–520. https://doi.org/10.1109/ICASSP.2006.1660393A. Aggoun and M. Mazri. 2008. Wavelet-based compression algorithm for still omni-directional 3d integral images.
Signal, Image and Video Processing
2, 2 (jun 2008),141–153. https://doi.org/10.1007/s11760-007-0044-1G. Alves, M. P. Pereira, M. B. de Carvalho, F. Pereira, C. L. Pagliari, V. Testoni, andE. A.B. da Silva. 2018. A Study on the 4D Sparsity of JPEG Pleno Light Fields Usingthe Discrete Cosine Transform. In . IEEE, 1148–1152. https://doi.org/10.1109/ICIP.2018.8451583L. Belcour, C. Soler, K. Subr, N. Holzschuch, and F. Durand. 2013. 5D Covariance Tracingfor Efficient Defocus and Motion Blur.
ACM Transactions on Graphics
32, 3 (jun2013), 1–18. https://doi.org/10.1145/2487228.2487239E. Candès and D. L. Donoho. 2004. New Tight Frames of Curvelets and OptimalRepresentations of Objects with Piecewise $Cˆ2$ Singularities.
Communications onPure and Applied Mathematics
57, 2 (feb 2004), 219–266. https://doi.org/10.1002/cpa.10116E. J. Candès and D. L. Donoho. 1999. Ridgelets: A Key to Higher-Dimensional Inter-mittency?
Royal Society of London Philosophical Transactions Series A
357 (1999),2495–+.E. J. Candès and D. L. Donoho. 2005a. Continuous curvelet transform: I. Resolution ofthe Wavefront Set.
Applied and Computational Harmonic Analysis
19, 2 (sep 2005),162–197. https://doi.org/10.1016/j.acha.2005.02.003E. J. Candès and D. L. Donoho. 2005b. Continuous curvelet transform: II. Discretizationand Frames.
Applied and Computational Harmonic Analysis
19, 2 (sep 2005), 198–222.https://doi.org/10.1016/j.acha.2005.02.004J.-X. Chai, S.-C. Chan, H.-Y. Shum, and X. Tong. 2000. Plenoptic Sampling. In
Proceedingsof the 27th annual conference on Computer graphics and interactive techniques -SIGGRAPH ’00 . ACM Press, New York, New York, USA, 307–318. https://doi.org/10.1145/344779.344932C. Conti, P. T. Kovacs, T. Balogh, P. Nunes, and L. D. Soares. 2014. Light-field videocoding using geometry-based disparity compensation. In . IEEE,1–4. https://doi.org/10.1109/3DTV.2014.6874724F. Dai, J. Zhang, Y. Ma, and Y. Zhang. 2015. Lenselet image compression scheme basedon subaperture images streaming. In . IEEE, 4733–4737. https://doi.org/10.1109/ICIP.2015.7351705I. Daubechies. 1992.
Ten Lectures on Wavelets . Society for Industrial and AppliedMathematics, Philadelphia, PA, USA.R. A. DeVore. 2006. Optimal Computation. In
Proceedings of the International Congressof Mathematicians . Madrid, Spain.D. L. Donoho. 2000. Orthonormal Ridgelets and Linear Singularities.
SIAM Jour-nal on Mathematical Analysis
31, 5 (aug 2000), 1062. https://doi.org/10.1137/S0036141098344403F. Durand, N. Holzschuch, C. Soler, E. Chan, and F. Sillion. 2005. A Frequency Analysisof Light Transport.
ACM Transactions on Graphics (Proceedings of SIGGRAPH 2005)
24 (2005).W. T. Freeman and E. H. Adelson. 1991. The design and use of steerable filters.
IEEETransactions on Pattern Analysis and Machine Intelligence
13, 9 (1991), 891–906.https://doi.org/10.1109/34.93808E. S. L. Gastal and M. M. Oliveira. 2017. Spectral remapping for image downscaling.
ACM Transactions on Graphics
36, 4 (jul 2017), 1–16. https://doi.org/10.1145/3072959.3073670S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen. 1996. The Lumigraph. In
SIGGRAPH ’96: Proceedings of the 23rd annual conference on Computer graphics andinteractive techniques . ACM, New York, NY, USA, 43–54. https://doi.org/10.1145/237170.237200I. Ihrke, J. Restrepo, and L. Mignard-Debise. 2016. Principles of Light Field Imaging:Briefly revisiting 25 years of research.
IEEE Signal Processing Magazine
33, 5 (sep2016), 59–69. https://doi.org/10.1109/MSP.2016.2582220N. K. Kalantari, T.-C. Wang, and R. Ramamoorthi. 2016. Learning-based view synthesisfor light field cameras.
ACM Transactions on Graphics
35, 6 (nov 2016), 1–10. https://doi.org/10.1145/2980179.2980251S. Kundu. 2012. Light field compression using homography and 2D warping. In .IEEE, 1349–1352. https://doi.org/10.1109/ICASSP.2012.6288140J. Lehtinen, T. Aila, J. Chen, S. Laine, and F. Durand. 2011. Temporal light field re-construction for rendering distribution effects. In
ACM SIGGRAPH 2011 paperson - SIGGRAPH ’11 , Vol. 30. ACM Press, New York, New York, USA, 1. https://doi.org/10.1145/1964921.1964950 C. Lessig. 2018a. A local Fourier slice equation.
Optics Express
26, 23 (nov 2018), 29769.https://doi.org/10.1364/OE.26.029769C. Lessig. 2018b.
Polar Wavelets in Space . Technical Report. arXiv:1805.02061 https://arxiv.org/abs/1805.02061A. Levin, W. T. Freeman, and F. Durand. 2008. Understanding Camera Trade-Offsthrough a Bayesian Analysis of Light Field Projections. In
Computer Vision âĂŞECCV 2008 . Springer Berlin Heidelberg, Berlin, Heidelberg, 88–101. https://doi.org/10.1007/978-3-540-88693-8_7M. Levoy and P. Hanrahan. 1996. Light Field Rendering. In
SIGGRAPH ’96: Proceedingsof the 23rd annual conference on Computer graphics and interactive techniques . ACM,New York, NY, USA, 31–42. https://doi.org/10.1145/237170.237199M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz. 2006. Light Field Microscopy.In
ACM Trans. Graph. (Proceedings of ACM SIGGRAPH 2006) , Vol. 25. ACM Press,New York, New York, USA, 924. https://doi.org/10.1145/1179352.1141976C.-K. Liang and R. Ramamoorthi. 2015. A Light Transport Framework for LensletLight Field Cameras.
ACM Transactions on Graphics
34, 2 (mar 2015), 1–19. https://doi.org/10.1145/2665075S. G. Mallat. 2009.
A Wavelet Tour of Signal Processing: The Sparse Way (third ed. ed.).Academic Press.R. Ng. 2005. Fourier Slice Photography.
ACM Transactions on Graphics
24, 3 (jul 2005),735. https://doi.org/10.1145/1073204.1073256R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan. 2005. Light fieldphotography with a hand-held plenoptic camera. (2005).P. Perona. 1991. Deformable kernels for early vision. In
Proceedings. 1991 IEEE ComputerSociety Conference on Computer Vision and Pattern Recognition . IEEE Comput. Sco.Press, 222–227. https://doi.org/10.1109/CVPR.1991.139691M. Pharr and G. Humphreys. 2010.
Physically Based Rendering: From Theory to Im-plementation (second ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA,USA.J. Portilla and E. P. Simoncelli. 2000. A Parametric Texture Model Based on JointStatistics of Complex Wavelet Coefficients.
International Journal of Computer Vision
40, 1 (2000), 49–70. https://doi.org/10.1023/A:1026553619983E. T. Quinto. 1993. Singularities of the X-Ray Transform and Limited Data Tomographyin Rˆ2 and Rˆ3.
SIAM Journal on Mathematical Analysis
24, 5 (sep 1993), 1215–1225.https://doi.org/10.1137/0524069E. T. Quinto. 2007. Local algorithms in exterior tomography.
J. Comput. Appl. Math.
ACM Transactions on Graphics
34, 1 (dec 2014), 1–13. https://doi.org/10.1145/2682631E. P. Simoncelli and W. T. Freeman. 1995. The steerable pyramid: a flexible architecturefor multi-scale derivative computation. In
Proceedings., International Conference onImage Processing , Vol. 3. IEEE Comput. Soc. Press, 444–447. https://doi.org/10.1109/ICIP.1995.537667C. Soler, K. Subr, F. Durand, N. Holzschuch, and F. Sillion. 2009. Fourier depth offield.
ACM Transactions on Graphics
28, 2 (apr 2009), 1–12. https://doi.org/10.1145/1516522.1516529R. Stevenson. 2009. Adaptive wavelet methods for solving operator equations: Anoverview. In
Multiscale, Nonlinear and Adaptive Approximation . Springer Berlin Hei-delberg, Berlin, Heidelberg, 543–597. https://doi.org/10.1007/978-3-642-03413-8_13M. Unser and N. Chenouard. 2013. A Unifying Parametric Framework for 2D SteerableWavelet Transforms.
SIAM Journal on Imaging Sciences
6, 1 (jan 2013), 102–135.https://doi.org/10.1137/120866014M. Unser, N. Chenouard, and D. Van De Ville. 2011. Steerable Pyramids and TightWavelet Frames in $L_{2}({R}ˆ{d})$.
IEEE Transactions on Image Processing
20, 10 (oct2011), 2705–2721. https://doi.org/10.1109/TIP.2011.2138147M. Unser and D. Van De Ville. 2010. Wavelet Steerability and the Higher-Order RieszTransform.
IEEE Transactions on Image Processing
19, 3 (mar 2010), 636–652. https://doi.org/10.1109/TIP.2009.2038832S. Vagharshakyan, R. Bregovic, and A. Gotchev. 2018. Light Field Reconstruction UsingShearlet Transform.
IEEE Transactions on Pattern Analysis and Machine Intelligence
40, 1 (jan 2018), 133–147. https://doi.org/10.1109/TPAMI.2017.2653101K. Venkataraman, D. Lelescu, J. Duparré, A. McMahon, G. Molina, P. Chatterjee, R.Mullis, and S. Nayar. 2013. PiCam: an ultra-thin high performance monolithiccamera array.
ACM Transactions on Graphics
32, 6 (nov 2013), 1–13. https://doi.org/10.1145/2508363.2508390A. Vieira, H. Duarte, C. Perra, L. Tavora, and P. Assuncao. 2015. Data formats forhigh efficiency coding of Lytro-Illum light fields. In . IEEE, 494–497. https://doi.org/10.1109/IPTA.2015.7367195I. Viola, M. Rerabek, and T. Ebrahimi. 2017. Comparison and Evaluation of Light FieldImage Coding Approaches.
IEEE Journal of Selected Topics in Signal Processing
11, 7(oct 2017), 1092–1106. https://doi.org/10.1109/JSTSP.2017.2740167, Vol. 1, No. 1, Article . Publication date: October 2019.
J. P. Ward, P. Pad, and M. Unser. 2015. Optimal Isotropic Wavelets for Localized TightFrame Representations.
IEEE Signal Processing Letters
22, 11 (nov 2015), 1918–1921.https://doi.org/10.1109/LSP.2015.2448233J. P. Ward and M. Unser. 2014. Harmonic singular integrals and steerable wavelets inL2(Rd).
Applied and Computational Harmonic Analysis
36, 2 (mar 2014), 183–197.https://doi.org/10.1016/J.ACHA.2013.03.006B. Wilburn, N. Joshi, V. Vaish, E.-V. Talvala, E. Antunez, A. Barth, A. Adams, M. Horowitz,and M. Levoy. 2005. High Performance Imaging Using Large Camera Arrays.
ACMTrans. Graph. (Proceedings of ACM SIGGRAPH 2005) (2005). https://graphics.stanford.edu/papers/CameraArray/G. Wu, B. Masia, A. Jarabo, Y. Zhang, L. Wang, Q. Dai, T. Chai, and Y. Liu. 2017. LightField Image Processing: An Overview.
IEEE Journal of Selected Topics in SignalProcessing
11, 7 (oct 2017), 926–954. https://doi.org/10.1109/JSTSP.2017.2747126Z. Xu, K. Sunkavalli, S. Hadap, and R. Ramamoorthi. 2018. Deep image-based relightingfrom optimal sparse samples.
ACM Transactions on Graphics
37, 4 (jul 2018), 1–13.https://doi.org/10.1145/3197517.3201313Y. Yoon, H.-G. Jeon, D. Yoo, J.-Y. Lee, and I. S. Kweon. 2015. Learning a Deep Con-volutional Network for Light-Field Image Super-Resolution. In . IEEE, 57–65. https://doi.org/10.1109/ICCVW.2015.17
A SPATIAL REPRESENTATION OF RECONSTRUCTIONFILTERS ζ αs When ˆ h (| ξ |) is the wavelet window from the steerable pyramid, the ζ α , ms ( x ) in Eq. 12 are given by ζ α , ns ( x ) = √ x π − log ( ) + iπ log ( ) (cid:32)(cid:18) − α (cid:19) α + (cid:33) − iπ log ( ) × e − in ( π − tan − (( − α ) α )) (cid:16) α x (cid:17) − iπ log ( ) × π iπ log ( ) A + (cid:32)(cid:18) − α (cid:19) α + (cid:33) iπ log ( ) B where A = (− iαx ) iπ log ( ) ( iαx ) iπ log ( ) (cid:16) e iπn (cid:0) D − − D − (cid:1) − (cid:0) D + − D + (cid:1)(cid:17) B = e iπn ( iαx ) iπ log ( ) (cid:0) D − − D − (cid:1) − (− iαx ) iπ log ( ) (cid:0) D + − D + (cid:1) and D ± d = Γ (cid:169)(cid:173)(cid:173)(cid:173)(cid:173)(cid:171) ± iπ log ( ) , ± iπxαd (cid:114)(cid:16) − α (cid:17) α + (cid:170)(cid:174)(cid:174)(cid:174)(cid:174)(cid:172) ..