Estimating Homogeneous Data-driven BRDF Parameters from a Reflectance Map under Known Natural Lighting
EEstimating Homogeneous Data-driven BRDF Parameters from a ReflectanceMap under Known Natural Lighting
Victoria L. Cooper James C. Bieron Pieter PeersCollege of William & Mary
Abstract
In this paper we demonstrate robust estimation of themodel parameters of a fully-linear data-driven BRDFmodel from a reflectance map under known natural lighting.To regularize the estimation of the model parameters, weleverage the reflectance similarities within a material class.We approximate the space of homogeneous BRDFs usinga Gaussian mixture model, and assign a material class toeach Gaussian in the mixture model. We formulate the es-timation of the model parameters as a non-linear maximuma-posteriori optimization, and introduce a linear approx-imation that estimates a solution per material class fromwhich the best solution is selected. We demonstrate the effi-cacy and robustness of our method using the MERL BRDFdatabase under a variety of natural lighting conditions, andwe provide a proof-of-concept real-world experiment.
1. Introduction
Data-driven appearance models [11, 12] express theBidirectional Reflectance Distribution Function (BRDF) ofa homogeneous material as a linear combination of a largeset of measured “basis”
BRDFs. The key assumption isthat this large set of basis BRDFs covers the full space ofBRDFs, and any BRDF in this space can be representedas convex combination of these basis BRDFs, thereby in-heriting all the intricate reflectance details present in themeasured basis BRDFs that can be difficult to model withanalytical BRDF models. Compared to analytical BRDFmodels that require an expensive and fragile non-linear opti-mization to estimate the model parameters from reflectancemeasurements, data-driven BRDF models, by virtue of itslinear nature, only require a linear least squares to estimatethe model parameters. Recent advances have shown greatpromise in reconstructing a data-driven BRDF from veryfew measurements [13, 25]. However, these methods relyon controlled directional or point lighting. A key problem in generalizing prior methods to natural lighting is that theseprior methods require a non-linear encoding (e.g., logarith-mic) to compress the dynamic range of the the basis BRDFsin order to regularize the estimation of the model parame-ters. Such non-linear encoding can only be undone after lin-ear parameter estimation if the observations consist of directBRDF observations (i.e., a single view and a single lightdirection per observation). In contrast, observations undernatural lighting are the result of an integration of the BRDFtimes lighting over all directions, and only linear transfor-mations of the BRDF are transparent to this integration.In this paper we aim to narrow the gap between inverserendering with data-driven BRDF models and analyticalBRDF models under natural lighting while retaining the ro-bustness and simplicity of linear parameter estimation fordata-driven models. We consider our work a first explo-ration in this direction that demonstrates that robust lineardata-driven BRDF model parameter estimation under nat-ural lighting is feasible, rather than introducing a practicaland/or competitive method to current advanced inverse ren-dering methods that use an analytical BRDF models as abasis. To focus our exploration, we will a-priori assume thatthe natural lighting is known and that we have a full char-acterization of the material reflectance under this lightingcondition in the form of a reflectance map [18].We desire to retain the advantages of a linear parameterestimation process, and therefore avoid non-linear encodedbasis BRDFs, and directly estimate the data-driven BRDFmodel parameters from unmodified basis BRDFs. To reg-ularize the estimation of the model parameters from a re-flectance map under natural lighting, we leverage the re-flectance similarities between BRDFs in a material class.Intuitively, we expect that it is easier to express the BRDF asa combination of a small set of similar materials than froma large set of BRDFs that span a larger spectrum of morevaried materials. We therefore, first approximate the spaceof homogeneous BRDFs with a Gaussian mixture model.Each normal distribution in the Gaussian mixture modelrepresents a material class, and we assign each basis ma-1 a r X i v : . [ c s . G R ] J un erial to the class with the highest likelihood. We formulatethe estimation of the model parameters as a maximum a-posteriori optimization that maximizes the likelihood thatthe model parameters explain the observations, as well asthe likelihood that the model belongs to the material class.However, this formulation is highly non-linear and difficultto minimize. We therefore exploit the additional observa-tion that in high dimensional spaces everything is distant,and approximate the maximum a-posteriori optimization byan efficient linear least squares approximation per materialclass. Finally, we select the most likely provisional leastsquares solution based on the maximum a-posteriori error.We demonstrate the efficacy of our solution using theMERL BRDF database under a variety of natural lightingconditions. Furthermore, we provide a proof-of-conceptreal-world experiment to demonstrate that our results gener-alize beyond the ideal simulated experiments on the MERLBRDF database.
2. Related Work
We focus this discussion of prior work on the two keyproperties of our method: reflectance modeling under nat-ural lighting, and appearance modeling with a data-drivenreflectance model. We refer to the surveys of Dorsey etal . [5], and Weinmann and Klein [23] for an in-depth gen-eral overview of appearance modeling.
Reflectance Modeling under Natural Lighting
A firstsubset of methods models surface reflectance from mul-tiple photographs under natural lighting. Oxholm andNishino [15] model shape and homogeneous reflectancefrom multiple photographs under known natural lighting.Palma et al . [16], Dong et al . [4], and Zhou et al . [27] re-cover spatially-varying surface reflectance under unknown natural lighting from a dense sampling of multiple views ormultiple rotations of a subject with known shape. Xia etal . [24] extended the method of Dong et al . [4] to modelspatially-varying reflectance under unknown natural light-ing and unknown shape. These model all rely on non-linearreflectance models and estimation processes. In contrast,we employ a linear data-driven BRDF model and rely on alinear estimation process.A second subset of methods models surface reflectancefrom just a single photograph of an object under naturallighting. In seminal work, Ramamoorthi and Hanrahan [17]lay out a spherical harmonics framework for estimatinggeneral homogeneous reflectance functions modeled by aspherical harmonics expansion. Romeiro et al . [19, 20]model the homogeneous surface reflectance using a bivari-ate data-driven model from an object with known shape un-der known and unknown natural lighting respectively. Simi-larly, Lombardi et al . [10] also estimate natural lighting andhomogeneous surface reflectance modeled by the DSBRDFreflectance model [14]. Finally, Barron and Malik [2] re- cover shape, lighting and spatially-varying albedo from asingle photograph under unknown natural lighting. How-ever, Barron and Malik only consider diffuse reflectance.Our method espouses the same overall goal as this sec-ond subset of methods. A reflectance map can potentiallybe obtained from a single observations of a convex objectof known shape (e.g., sphere) or using the deep learningmethod of Rematas et al . [18]. However, we explicitely de-sire to recover a data-driven model [11] based on real-worldmeasured reflectance.A third subclass of methods relies on deep learning toinfer reflectance properties under unknown natural lightingfrom a single image. Li et al . [7] and Ye et al . [26] es-timate the parameters of an analytical BRDF model [22]for a spatially-varying material. Both Li et al . and Ye etal . focus on augmenting the training data with unlabeledphotographs in order to reduce the number of required la-beled training data (i.e., measured SVBRDFs). Li et al . [8]present a network structure and a novel post-processing stepbased on conditional random fields to estimate spatially-varying reflectance parameters for an analytical micro-facetBRDF model [21]. Finally, Li et al . [9] propose a cascad-ing network structure to iteratively estimate and refine theshape and spatially-varying surface reflectance. All of theabove methods express the surface reflectance using an an-alytical BRDF model. In contrast, we express the surfacereflectance using a more expressive data-driven model, al-beit limited to a homogeneous material and under known natural lighting.
Data-driven Reflectance Model
In seminal work, Ma-tusik et al . [11] presented a data-driven BRDF model thatexpresses the surface reflectance as a weighted combina-tion of a large set of measured BRDFs. To handle the largedynamic range between the specular peaks and the diffusereflectance, a log-encoding is first applied to the measuredbasis BRDFs. Matusik et al . propose two models: a PCAbased D linear model, and non-linear, charting based, D model. In follow up work, Matusik et al . [12] usethe linear PCA model and show that well selected andcontrolled view-light direction pairs are sufficient for esti-mating the BRDF. Nielsen et al . [13] show that by addinga Tikhonov regularization to the estimation of a log-relativeencoded linear data-driven model, a good BRDF estimatecan be obtained from less than optimized and controlledview-light direction pairs, and for photographs of a spherelit by optimized directional light sources. Xu et al . [25]build on the method of Nielsen et al ., and show that with animproved error metric, a log-relative encoded linear data-driven model can be recovered from just near-field ob-servations (photographs) under controlled directional light-ing. All of the above methods estimate a data-driven BRDFfrom observations under directional lighting, and regularizethe estimation using a non-linear encoding of the measured2RDFs. In contrast, our method uses a fully linear modeland reconstructs the data-driven BRDF model from a re-flectance map under uncontrolled known natural lighting.
3. Overview
Data-driven BRDF
The reflectance behavior of a homoge-neous material is described by the bidirectional reflectancedistribution function (BRDF) ρ ( ω i , ω o ) : a D function de-fined as the ratio of incident irradiance for an incident direc-tion ω i over the outgoing radiance for an outgoing direction ω o .In this paper, we follow the data-driven BRDF modelof Matusik et al . [12] that characterizes the BRDF ρ as alinear combination of a large set of n measured materials b i , i ∈ [1 , n ] . The underlying idea is that the set of mea-sured BRDFs spans the space of BRDFs, and any material’sBRDF should lie in this space: ρ = Bw, (1)where we stack the BRDF ρ and basis BRDFs b i in a vectorof length p , and form the matrix B by stacking each basisvector in a column: B = [ b , ..., b n ] . The model parame-ters are stacked in a vector w of n scalar weights. We di-rectly use the BRDF parameterization of the MERL BRDFdatabase [11], and p = 90 × × . Furthermore, similaras in Nielsen et al . [13], we consider each color channel ofthe MERL BRDFs as a basis BRDF, and thus n = 300 .Due to the large dynamic range between specular peaksversus diffuse reflectance, prior work [11, 13, 25] has ap-plied a non-linear compression function ζ to make the esti-mation of w less sensitive to errors on the (large) specularpeaks: ρ (cid:48) = B (cid:48) w (cid:48) , (2)where B (cid:48) = [ ζ ( b ) , ..., ζ ( b n )] . An expansion ζ − is ap-plied to the compressed BRDF ρ (cid:48) after computation of theweights. A common compression function is the logarith-mic function, in which case Equation 2 becomes a homo-morphic factorization. Natural Lighting
Prior work relied on point sample mea-surements of ρ for a set of incoming-outgoing directionpairs to estimate the weights w . In contrast, in this paperwe aim to estimate the weights w from an observation un-der natural lighting. Assuming the lighting L is distant (i.e.,it only depends on the incident direction ω i = ( φ i , θ i ) ), andignoring interreflections, we can formulate the observed ra-diance y as: y ( ω o ) = (cid:90) Ω ρ ( ω i , ω o ) cos( θ i ) L ( ω i ) dω i , (3)where cos( θ i ) is the foreshortening, and Ω is the upperhemisphere of incident directions. Due to linearity of light transport, we can express Equation 3 in terms of corre-sponding basis observations y : y = Y w, (4)where the weights w are the same as in Equation 1, andthus can be used to reconstruct ρ . The basis images Y = [ y , ..., y n ] are the observations of the measured ba-sis BRDFs b i under the same conditions: y i = (cid:90) Ω b i ( ω i , ω o ) cos( θ i ) L ( ω i ) dω i . (5) Problem Statement
As noted before, the dynamic rangecompression function ζ is essential in obtaining good data-driven BRDF reconstructions, even in the case of a verydense point sampling of light and view directions [1]. How-ever, this compression function cannot be used when lin-early estimating the weights w from observations under nat-ural lighting. This can be seen by inserting Equation 2in Equation 5: ζ ( y i ) (cid:54) = y ζi = (cid:90) Ω ζ ( b i ( ω i , ω o )) cos( θ i ) L ( ω i ) dω i . (6)In other words, the non-linear compression of the obser-vation is not equivalent to the observation under naturallighting of the non-linearly compressed BRDFs. While nota problem for the basis BRDFs b i , since we can generatethe corresponding images y ζi with any rendering system di-rectly from the non-linear encoded basis BRDFs ζ ( b i ) , it isa problem for ρ , because we can only observe y the result-ing radiance of ρ under natural lighting, not the reflectedradiance of its non-linear compressed form ζ ( ρ ) , and hencewe do not have access to y ζ . Consequently, the key prob-lem we aim to address in this paper is to find the data-drivenweights w from the observation y without relying on a non-linear compression function ζ and/or a non-linear optimiza-tion procedure for estimating the weights w . Maximum a-posteriori Optimization
Formally, our goalis to find the most likely weights w , relying on a linear esti-mation process, such that the conditional probability of thereconstructed data-driven homogeneous BRDF ρ is maxi-mized given a reflectance map y under known natural light-ing L : argmax w P ( ρ | y ) . (7)We will assume that the observations are in the form of ahigh dynamic range reflectance map (i.e., a full characteri-zation of the reflectance radiance of a homogeneous BRDFfor a fixed lighting condition). In the remainder of this pa-per, we will assume that the reflectance map is provided inthe form of a visualization of a sphere under the target illu-mination.3sing Bayes’ theorem, we can formulate the maximuma-posteriori (MAP) estimation of w as: argmax w P ( y | ρ ) P ( ρ ) P ( y ) . (8)Rewriting in terms of the log-likelihood, and noting that P ( y ) is constant (i.e., the observation is given), we obtain: argmin w (log P ( y | ρ ) + log P ( ρ )) . (9)In order to solve this minimization problem, we needa model of the likelihood of the BRDF estimation ρ (sec-tion 4), and a model for the conditional probability of theobservation y given the estimated BRDF ρ , and an efficientlinear strategy for solving this minimization (section 5).
4. BRDF Likelihood Modeling
Gaussian Mixture Model
We propose to model the likeli-hood of BRDFs by a Gaussian mixture model (GMM): P ( ρ ) = k (cid:88) j =1 π j N ( ρ | µ j , Σ j ) , (10)where π j are the mixing coefficients of the j -th normal dis-tribution N with mean µ j and covariance matrix Σ j . Expectation-Maximization
An effective method for com-puting the parameters
Θ = ( π, µ, Σ) is the ExpectationMaximization algorithm using the MERL BRDFs b i as ob-servations. For this we define a latent variable γ j ( b i ) thatindicates the likelihood of the j -th Gaussian given a MERLBRDF b i : γ j ( b i ) = P ( j | b i ) , (11) = P ( j ) P ( b i | j ) P ( b i ) , (12) = π j N ( b i | µ j , Σ j ) (cid:80) kj =1 π j N ( b i | µ j , Σ j ) . (13)Expectation minimization iterates between estimating thelatent variable γ j ( b i ) (E-step, Equation 13), and the modelparameters (M-step): π j = 1 n n (cid:88) i γ j ( b i ) , (14) µ j = (cid:80) ni γ j ( b i ) b i π j , (15) Σ j = (cid:80) ni γ j ( b i )( b i − µ j )( b i − µ j ) T π j . (16)We iterate until the log-likelihood over the MERL BRDFsconverges: log P ( B | Θ) = n (cid:88) i log k (cid:88) j π j N ( ρ | µ j , Σ j ) . (17) Specular Plastics/PaintsMetalsPlastics/PhenolicsDiffuse/Glossy
Figure 1. D multi-dimensional scaling of the projected MERLBRDFs ˆ U T B and a color-coding of the respective material classesderived from the D approximation of the BRDF likelihood mod-eled by a Gaussian mixture model.
To bootstrap the EM algorithm, we perform a standardk-mean clustering, and initialize π j as the ratio of assignedBRDFs to the j -th cluster over the total number of MERLBRDFs (i.e., n ). Curse of Dimensionality
A practical problem is that thenumber of observations n is significantly lower than the di-mensionality of the space (i.e., p ).We therefore apply a sin-gular value decomposition (SVD) to express the observa-tions in a n dimensional space U : B = U SV T . (18)However, this is still a dimensional space. A key issueis that even for a moderate number of dimensions any dis-tance is very large, and thus the distance to the means µ j are large too. Consequently, the likelihood of each Gaus-sian mixture (Equation 13) will always be very low and itcan potentially cause numerical instabilities. To resolve thisissue, we perform expectation maximization in a reducedspace, and only keep the coefficients belonging to the N largest singular values. In other words, we perform expecta-tion maximization (i.e., soft clustering) on a projection to an N dimensional subspace, and approximate the likelihood: P ( ρ ) ≈ P ( ˆ U T ρ ) , where ˆ U is the N dimensional basis (i.e.,the first N vectors in U ). Discussion
We found that N = 4 offers a good balance be-tween accuracy and numerical stability. A second parameterthat needs to be set is the number of Gaussian mixtures K .If the number of Gaussians is too low, then P ( ˆ U T ρ ) onlyoffers a coarse approximation. However, we also found thatfor increasing number of K , the algorithm tends to subdi-vide the same Gaussian distribution, essentially overfittingto ’special case’ BRDFs (such as Steel which exhibits ac-quisition artifacts). In practice we found that K = 4 offersa good approximation that nicely categorizes the materialsin four recognizable distinct material classes: “diffuse andglossy” materials ( materials), “plastics/phenolics” ( “metals” ( ), and “specular plastics/paints” ( materials); we determine membership to a material classby assigning the material to the material class with the max-imum γ j ( b i ) likelihood. Figure 1 shows a plot of a Dmulti-dimensional scaling of the D projected coordinatesof the MERL BRDFs, as well as a color-coding to indicatefor which material class the material has the highest affinity.Note that even though the diffuse-like material class con-tains materials, the multi-dimensional scaling placesthem all close together. Please refer to the supplementalmaterial for an exhaustive list of which material belongs towhich material class.
5. Data-driven Model Estimation
MAP Estimation
We express the likelihood of the obser-vation given an estimate of the BRDF as: P ( y | ρ ) = N ( Y w − y | µ, σ ) , (19)where µ and Σ is the expected mean error and standard de-viation on the reconstructions, and Y w is the rendering ofthe estimated BRDF under the target natural lighting. Weassume that the mean error is close to zero ( µ = 0 ), and σ is proportional to the expected measurement error (e.g.,camera noise).Given the likelihood P ( ˆ U T ρ ) expressed by the Gaussianmixture model (Equation 10), we can then formulate theMAP estimation (Equation 9) as: argmin w || Y w − y || σ + log (cid:88) j π j N ( ˆ U T Bw | µ j , Σ j ) . (20)The first term is the data term that indicates how well (avisualization of) the BRDF ρ = Bw can explain the ob-servation y , and the second term indicates how plausiblethe reconstructed BRDF ρ (projected in the dimensionalspace ˆ U ) is.However, directly solving for the BRDF weights w usingEquation 20 is not practical because of two key issues:1. Non-linear:
Equation 20 is highly non-linear and dif-ficult to optimize due to the sum of the log-likelihoodsin the second term.2.
Gaussian Mixture Model Accuracy for P ( ρ ) ≈ P ( ˆ U T ρ ) : We approximated the likelihood of theBRDF by a dimensional Gaussian mixture model.This reduction in dimensionality was necessary due tothe curse of dimensionality. However, it also implic-itly assumes that the BRDF lies not too far from thespace of plausible BRDFs. Since the likelihood is onlydetermined based on dimensions (and thus only regu-larizes these four), the other dimensions can be set to any value (including unreasonable values that resultin an implausible BRDF). Linear MAP Approximation
To alleviate the above twopractical issues, we exploit the observation that the likeli-hood of a basis BRDF b i belonging to a material class m isfor most basis BRDFs equivalent to an indicator function: γ j ( b i ) ≈ δ i,m . (21)This implies that the overlap between the Gaussians in theGaussian mixture model is limited. Armed with this obser-vation, we therefore propose to compute a candidate BRDFfor each material class j ∈ [1 , k ] : argmin w ( j ) (log P ( y | ρ, j ) + log P ( ρ | j )) . (22)Given the set of candidate solutions w (cid:48) = { w (1) , .., w ( k ) } ,we then pick the best candidate that best reconstructs theBRDF. Per-Material Class Linear Data Term
We define the data-term similarly as in the general non-linear case, except thatwe only use the basis BRDFs that belong to the same mate-rial class: log P ( y | ρ, j ) = || Y ( j ) w ( j ) − y || , (23)where Y ( j ) is the set of observations that correspond to thebasis BRDFs assigned to the j -th material class (i.e., thematerials b i for which γ j ( b i ) is maximal). Per-Material Class Linear Likelihood Term
We ex-press the per-material class likelihood by a single Gaussianmodel. We directly compute this probability on the BRDFweights w ( j ) : P ( ρ | j ) = N ( w ( j ) , µ (cid:48) j , Σ (cid:48) j ) , (24)where: µ (cid:48) j = ( c j , ..., c j ) T , and c j is the number of ba-sis BRDFs in the j -th material class. Note that Y ( j ) µ (cid:48) j isequivalent to the mean BRDF of the material class, and µ (cid:48) j the corresponding coordinate in the j-th BRDF subspace. Linear Least Squares Estimation
Both Equation 23 and(the log likelihood of) Equation 24 are quadratic terms thatdefine a linear system in terms of w that can be solved us-ing a regular linear least squares. However, both terms canhave a vastly different magnitude. The magnitude of thedata-term depends on the error on the rendered image of theestimated BRDF. This image error depends on the resolu-tion, the overall intensity of the lighting, and the reflectivityof the material. Similarly, the magnitude of the likelihoodterm depends on the number of basis BRDFs per materialclass. We therefore add a balancing term: λ j = λ || y || c j , (25)5here || y || is the total squared pixel intensities in the ob-servation. We expect that the overall intensity of the obser-vation is directly proportional to the lighting intensity andreflectivity of the BRDF, and hence the overall scale of theimage error. λ is a user set constant that depends on thequalities of the lighting. An ill-conditioned lighting condi-tion requires a larger λ value (e.g., a low frequency lightingenvironment is ill-conditioned for estimating specular prop-erties [17]). In practice we found that λ = 0 . works wellfor many lighting environments, and forms a good startingpoint for fine-tuning λ .The final linear least squares is: argmin w ( j ) (cid:32) || Y ( j ) w ( j ) − y || + λ j || w ( j ) − µ (cid:48) j || Σ j (cid:33) . (26) Selection
Ideally, we would like to select the best candidatesolution from w (cid:48) by evaluating Equation 20. However, bya-priori assuming that a BRDF belongs to a material class j , it is possible that there is a significant mismatch betweenthe target material and the material class. For example, at-tempting to model a mirror-like specular material using thediffuse material class is unlikely to produce a satisfactoryresult. Consequently, we cannot simply rely on the likeli-hood P ( ˆ U T ρ ) based on the 4 dimensional Gaussian mix-ture model to select the best solution from w (cid:48) (i.e., the other dimensions can be arbitrarily wrong). We will thereforefurther exploit the observation of the limited overlap of theGaussians in the mixture model, and approximate the solu-tion per material class by enforcing that it lies in the convexhull of the subspace spanned by the BRDFs assigned to thematerial class, and only rely on Equation 19 to pick the bestcandidate from w (cid:48) . We ignore the standard deviation (i.e., σ = 1 ) in Equation 19 as it only acts as a scale (in thelog-likelihood) that does not affect the selection of the bestreconstruction (i.e., minimum log-likelihood). Color
Our discussion until now only considered mono-chrome BRDFs; we used all color channels from the MERLBRDFs as separate basis BRDFs. A straightforward strat-egy for estimating a non-monochrome BRDF with threecolor channels, would be to execute the estimation sepa-rately for each color channel, and combine the three re-constructed monochrome BRDF into a single RGB BRDF.However, it is possible that a solution from a different mate-rial classes j is selected for each of the three color channels.Because the set of basis BRDFs for each material class aredisjunct, there can be slight differences in the constructedBRDF shape for each color channel, which in turn can resultin color artifacts in the combined BRDF. We circumventthis potential problem by combining the three color chan-nels after obtaining the candidate BRDFs, and performingthe selection on the RGB BRDF instead of each color chan-nel separately. Hence, each color channel will be recon-structed with the same set of basis BRDFs. Algorithm Summary
In summary, given a reflectance map y under known natural lighting L , and given a user providedbalance parameter λ , we compute the data-driven BRDF ρ = Bw as:1. We precompute the Gaussian mixture model using theEM algorithm detailed in section 4. Note, this pre-computation only needs to happen once for the MERLBRDF database, and is independent of the lighting.2. We precompute Y by rendering a sphere with each ba-sis BRDF b i under the natural lighting (Equation 5).This precomputation needs to happen for every light-ing condition.3. We compute the candidate solutions w (cid:48){ r,g,b } for eachmaterial class by solving the linear least squaresin Equation 26 per color channel.4. We combine the monochrome BRDFsto a 3-channel BRDF: w (cid:48) = { ( w (cid:48) r, , w (cid:48) g, , w (cid:48) b, ) , ..., ( w (cid:48) r,k , w (cid:48) g,k , w (cid:48) b,k ) } .5. Finally, we select the candidate solution from w (cid:48) thatminimizes Equation 19.
6. Results
Experiment Setup
We demonstrate our method on simu-lated reflectance maps in order to fully control all parame-ters. We generate the reflectance maps under natural light-ing, by rendering a sphere lit by a light probe [3] usingMitsuba [6]; as noted in section 3, we will directly usethis rendered image as a representation of the reflectancemap. All generated images are radiometrically linear, andwe only tone map them for display. All results shown inthis paper were tone mapped by a simple gamma . cor-rection and a virtual exposure (i.e., scale factor) of . ; allpixel values above 1.0 and below 0.0 are clipped to the re-spective clipping values. We use the BRDFs in the MERLdatabase [11] for generating reflectance maps. For eachMERL BRDF, we compute a novel Gaussian mixture modelon the remaining MERL BRDFs (i.e., we exclude thebasis BRDF corresponding to any of the three color chan-nels of the BRDF), and only use these MERL BRDFsfor reconstruction. Consequently, any reconstruction of aBRDF from the MERL BRDF database is computed usinga different set of basis BRDFs. As noted in the prior sec-tions, we compute the Gaussian mixture model on a N = 4 dimensional reduced space, and use K = 4 Gaussians inthe mixture model. All reconstructions are generated with afixed balancing factor λ = 0 . . Reconstruction Results
Figure 2 shows reconstructions of selected materials under two different light probes (i.e., Eucalyptus Grove and
Galileo’s Tomb ). For each recon-struction (and the reference), we show a visualization of the6 econstructed under Reconstructed underReference Eucalyptus Grove Galileo S p ec . O r a ng e P h e no li c C o l on i a l M a p l e G r ee n L a t e x S t ee l G r ee n M e t a l . P a i n t G r ee n M e t a l . P a i n t Y e ll o w M a tt e P l a s ti c Figure 2. Data-driven BRDF reconstructions from a reflectance map under the
Eucalyptus Grove and the
Galileo’s Tomb light probe. Wevisualize the reference and reconstructed BRDFs under the
Uffizi Gallery light probe and a directional light. inear Least Material ClassReference Squares Diffuse Plastics/Phenolics Metals Spec. Plastics/Paints D a r k B l u e P a i n t Observation Log-likelihood: 0.001 0.012 0.570 0.071 V i o l e t A c r y li c Observation Log-likelihood: 0.666 0.141 2.952 0.487 C h r o m e S t ee l Observation Log-likelihood: 304.217 27.631 19.580 35.744 R e d M e t a lli c P a i n t Observation Log-likelihood: 53.195 1.484 5.391 1.205Figure 3. Reconstructions for each material class for selected materials observed under the Uffizi Gallery light probe, and revisualizedunder the
Eucalyptus Grove light probe and directional lighting. We list the log-likelihood error on the observations, and mark the bestsolution. In addition we provide a comparison against a naive linear least squares reconstruction with the full MERL BRDFs.
Uffizi Gallery ; different than the lighting condi-tion under which the BRDF was reconstructed) and a direc-tional light (i.e., a slice of the BRDF for a single incidentdirection for all outgoing directions). These results showthat our method is able to reconstruct plausible BRDFs fora wide range of materials from a reflectance map under nat-ural lighting. We refer to the supplemental material for thereconstructions under different natural lighting conditionsfor all MERL BRDFs.
Per-Material Class Reconstruction
Figure 3 illustrates,for a selection of materials, reconstructed under the Uf-fizi Gallery light probe, that the reconstructions per mate-rial class are different, and that depending on the materiala different class’ reconstruction is selected. We show a vi-sualization of the reference BRDF and the reconstructionsper cluster under a natural lighting condition (i.e.,
Euca-lyptus Grove ) and a directional light. We also list the log-likelihood of the observation given the BRDF (Equation 19)below each cluster, and mark the final selected solution (i.e.,minimum). For reference, we also show the linear leastsquares solution: argmin w || Y w − y || . As expected thisyields the lowest reconstruction error (since it explicitelyoptimizes for this). However, the linear least squares solu-tion does not always yield a plausible result when visual-ized under a different lighting condition. This is not onlyclearly visible under the directional light source, but alsounder other natural lighting conditions other than the orig-inal observed lighting (e.g., the black spot in the center ofthe visualizations under the Eucalyptus Grove light probefor
Steel and
Red Metallic Paint ). Furthermore, we observethat not all clusters’ reconstructions appear to be plausible.However, the selection process tends to pick the most plau-sible reconstruction.
Comparison: Single Material Class Reconstruction
Togain insight in the importance of reconstructing the BRDFper material class, we compare the reconstruction qualityof the BRDF from a single material class to our multi-material class solution (Figure 4). Our results demonstratethat using a single material class improves on a naive linearleast squares. However, our solution with multiple mate-rial classes outperforms the single material class case. Notethat we optimized λ for the single cluster case to produce anas optimal result given the lighting conditions. In this casewe reconstructed the BRDF under Grace Cathedral lightingusing a λ = 0 . for the single cluster case. Note, that thesingle material class reconstruction (Equation 26) is simi-lar to Nielsen et al .’s [13] method, without applying a non-linear encoding of the BRDF. Additional minor differencesare that Nielsen et al . subtract the median instead of themean before computing the linear least squares and assumea unit standard deviation. Furthermore, the single class re-construction is also similar to Romeiro et al .’s [19] method, using a linear data-driven BRDF model instead of the bi-variate model. Since we a-priori assume a linear BRDFmodel, we want to explore the differences between the re-construction methods, not the BRDF model representations.In general, we found that overall our method outper-forms a single material class reconstruction. The singlematerial reconstruction tends to work equally well on phe-nolic and plastic materials as these are similar in BRDFshape to the mean material. However, the single materialclass reconstruction fails for diffuse and metal-like materi-als. While less strong than for the naive least squares, fordiffuse materials we can observe a central “spike” visibleunder the directional lighting. For the metal-like materials,we typically observe strong ringing artifacts. Captured Reflectance Map Validation
To validate ourmethod on other materials than the MERL BRDF database,we performed the following proof-of-concept experiment.We acquired three spheres with different materials (i.e.,
Dense Orange Foam , Blue Plastic , and
Dark Bronze ) un-der two different natural lighting environments shown inthe insets. Next, we estimate data-driven BRDF parametersunder the indoor
Chapel lighting, and rerender the sphereunder the outdoor lighting. We mask out any measuredreflectance values that deviate from the expected measure-ment conditions (e.g., the dimple on the
Blue Plastic , andthe near field reflection from the stand). As can be seenin Figure 5, the rerendered reflectance maps closely resem-ble the acquired reference maps. Note, the
Dark Bronze ma-terial exhibits anisotropic reflectance which adversely im-pacts the reconstruction. Nevertheless, the reconstructedBRDF remains plausible. For reference, we also include aleast squares data-driven BRDF reconstruction. In addition,we also show visualizations of the reconstructed BRDFs litby a directional light source to better demonstrate the plau-sibility of the reconstructions.
Discussion
While our selection criterion does in the ma-jority of cases select the best reconstruction from the dif-ferent material classes, we found that in a few cases casesit does not select the best reconstruction, and a better re-construction can be observed in another material class. Ide-ally, the selection criterion should not only include the dataterm Equation 19, but evaluate the full non-linear MAPestimation loss (Equation 20). We observe that for caseswhere our current selection criterion prefers a suboptimalsolution, that the accompanying likelihood term P ( ˆ U T ρ ) isrelatively large. However, a challenge is that the range ofthe data-term and the likelihood cover a different range dueto: (a) the ommission of a standard deviation scale in thedata-term, and (b) the dimension reduction in the likelihoodterm. Finding a good balancing term is non-trivial and aninteresting avenue for future research.We currently used a λ balancing factor of . for all ourreconstructions. This λ is a compromise to produce the9 inear Least Single Material Multiple MaterialReference Squares Class Reconstruction Class Reconstruction C h r o m e S t ee l B r o w n F a b r i c S p ec i a l W a l nu t Y e ll o w P a i n t Figure 4. Comparison between naive linear least squares data-driven BRDF reconstruction, single material class reconstruction, and ourmultiple material class reconstruction. For diffuse-like materials, both the linear least squares and single material class reconstructionsexhibit a central “spike” visible under the directional lighting. For metals, strong ringing artifacts can be observed for both the linear leastsquares and single material class solutions. best result over all materials. Despite the material class andscene dependent scale factor (Equation 25), we observe thatthis lambda terms tends to affect the “diffuse” and “plas-tics / phenolics” stronger, and the ”metals” and “specularplastics / paints” less. These latter two material classes ex-hibit not only a lower number of materials (for which wecompensate), but we can also observe in Figure 1 that theyare also spread out further. Consequently, the density ofthese material classes is significantly lower. This lower den-sity implies that the material class is very diverse in BRDFtypes and that the MERL BRDF database does not denselysample these material types. Taking in account this den-sity difference is another interesting avenue for future work.In general, we find that reconstructions from these materialclasses are less often selected.
Relation to Prior Work
Matusik et al . [11] showed thatthe log-encoded BRDF space can be accurately modeledby a D linear subspace and a D non-linear manifold.While a linear model is computationally more convenient, anon-linear model offers a tighter fit to the space of BRDFs,and consequently, it contains less implausible BRDFs. OurGaussian mixture based model can be seen as a piecewiselinear approximation of the non-linear manifold of BRDFs.In contrast to Matusik et al ., we work directly on the spacespanned by the basis BRDF (i.e., without log-encoding). However, as shown in Figure 1 this manifold is highlynon-linear too. While less tight than a full D non-linearmodel, our Gaussian mixture models strikes a balance be-tween tightness and the ability to robustly identify the piece-wise linear subspace to which the observations under natu-ral lighting belong.An implausible BRDF lies inside the linear subspacespanned by the linear model, but outside the non-linearBRDF manifold. Ideally, we would like to bias these im-plausible solution towards the non-linear manifold to ob-tain a more plausible solution. Tikhonov regularizationbiases the reconstruction towards a mean BRDF, assum-ing that the solution is more plausible when closer to themean. However, such a regularization is only efficient ifthe modeled space resembles a hypersphere. Nielsen etal . [13] model the BRDF space as a hypersphere by scal-ing the PCA basis BRDFs by the singular values. However,as noted before, the BRDF space is highly non-linear andsuch a hypersphere is not a tight model. Intuitively, biasingthe reconstruction of a diffuse material towards the mean ormedian BRDF is suboptimal; the mean or median BRDFcontains a rough specular lobe. Consequently, it is possi-ble that biasing pushes the solution towards a point awayfrom the non-linear manifold. Our solution represents thespace of BRDFs as a sum of (rescaled) hyperspheres: the10 inear Least Material ClassReference Squares Diffuse Plastics/Phenolics Metals Spec. Plastic/Paints O r a ng e F o a m B a ll Observation Log-likelihood: 0.5963 0.6614 7.7250 1.5751 B l u e P l a s ti c Observation Log-likelihood: 24.8899 11.8852 12.8056 12.8498 D a r k B r on ze Observation Log-likelihood: 0.7201 0.4698 1.1173 0.6966Figure 5. Reconstructions for each material class for captured materials observed under indoor natural lighting, and revisualized underoutdoor natural lighting and directional lighting. For each natural lighting condition we also provide a reference photograph. We list thelog-likelihood error, and mark the best solution. In addition we provide a comparison against a naive linear least squares reconstruction. µ (cid:48) j ) of the local hypersphere(rescaled by Σ (cid:48) j ). Since each Gaussian subspace is moretight, biasing towards the mean has a lower likelihood ofending away from the non-linear manifold.
7. Conclusion
In this paper we presented a novel method for estimatingthe parameters of a fully linear data-driven BRDF modelfrom a reflectance map under uncontrolled, but known, nat-ural lighting. Our estimation method does not require anynon-linear optimization, and only requires solving linearleast squares problems. Our method requires modest pre-computations: a Gaussian mixture model clustering for thebasis BRDFs, and for each natural lighting conditions, ren-derings of each basis material. We demonstrated the ac-curacy and robustness of our method on the MERL BRDFdatabase, and validated our method on real-world measure-ments.For future work we would like to explore better selectioncriteria and a per-material class λ j density correction factor. Acknowledgments
This work was supported in part byNSF grant IIS-1350323 and gifts from Google, Activision,and Nvidia.
References [1] M. M. Bagher, J. Snyder, and D. Nowrouzezahrai. A non-parametric factor microfacet model for isotropic brdfs.
ACMTrans. Graph. , 35(5):159:1–159:16, 2016.[2] J. T. Barron and J. Malik. Shape, illumination, and re-flectance from shading.
IEEE PAMI , 2015.[3] P. Debevec. Light probe gallery. , 1998.[4] Y. Dong, G. Chen, P. Peers, J. Zhang, and X. Tong.Appearance-from-motion: Recovering spatially varying sur-face reflectance under unknown lighting.
ACM Trans.Graph. , 33(6):193:1–193:12, 2014.[5] J. Dorsey, H. Rushmeier, and F. Sillion.
Digital Modelingof Material Appearance . Morgan Kaufmann Publishers Inc.,2008.[6] W. Jakob. Mitsuba: Physically based renderer. , 2010.[7] X. Li, Y. Dong, P. Peers, and X. Tong. Modelingsurface appearance from a single photograph using self-augmented convolutional neural networks.
ACM Trans.Graph. , 36(4):45:1–45:11, July 2017.[8] Z. Li, K. Sunkavalli, and M. K. Chandraker. Materials formasses: Svbrdf acquisition with a single mobile phone im-age. In
ECCV , 2018.[9] Z. Li, Z. Xu, R. Ramamoorthi, K. Sunkavalli, and M. Chan-draker. Learning to reconstruct shape and spatially-varyingreflectance from a single image.
ACM Trans. Graph. , 37(6),Dec. 2018. [10] S. Lombardi and K. Nishino. Reflectance and illuminationrecovery in the wild.
IEEE PAMI , 38(1):129–141, 2016.[11] W. Matusik, H. Pfister, M. Brand, and L. McMillan. A data-driven reflectance model.
ACM Trans. Graph. , 22(3):759–769, July 2003.[12] W. Matusik, H. Pfister, M. Brand, and L. McMillan. Ef-ficient isotropic BRDF measurement. In
Rendering Tech-niques , pages 241–248, 2003.[13] J. B. Nielsen, H. W. Jensen, and R. Ramamoorthi. On opti-mal, minimal brdf sampling for reflectance acquisition.
ACMTrans. Graph. , 34(6), Oct. 2015.[14] K. Nishino and S. Lombardi. Directional statistics-based re-flectance model for isotropic bidirectional reflectance distri-bution functions.
J. Opt. Soc. Am. A , 28(1):8–18, Jan 2011.[15] G. Oxholm and K. Nishino. Shape and reflectance estimationin the wild.
IEEE PAMI , 38(2):376–389, Feb. 2016.[16] G. Palma, M. Callieri, M. Dellepiane, and R. Scopigno. Astatistical method for svbrdf approximation from video se-quences in general lighting conditions.
Comput. Graph. Fo-rum , 31(4):1491–1500, 2012.[17] R. Ramamoorthi and P. Hanrahan. A signal-processingframework for inverse rendering. In
Proceedings of the 28thAnnual Conference on Computer Graphics and InteractiveTechniques , SIGGRAPH ’01, pages 117–128, 2001.[18] K. Rematas, T. Ritschel, M. Fritz, E. Gavves, and T. Tuyte-laars. Deep reflectance maps. In
CVPR , 2016.[19] F. Romeiro, Y. Vasilyev, and T. Zickler. Passive reflectome-try. In
ECCV , pages 859–872, 2008.[20] F. Romeiro and T. Zickler. Blind reflectometry. In
ECCV ,pages 45–58, 2010.[21] B. Walter, S. R. Marschner, H. Li, and K. E. Torrance. Mi-crofacet models for refraction through rough surfaces. In
Rendering Techniques , pages 195–206, 2007.[22] G. J. Ward. Measuring and modeling anisotropic reflection.
SIGGRAPH Comput. Graph. , 26(2):265–272, 1992.[23] M. Weinmann and R. Klein. Advances in geometry andreflectance acquisition. In
ACM SIGGRAPH Asia, CourseNotes , 2015.[24] R. Xia, Y. Dong, P. Peers, and X. Tong. Recovering shapeand spatially-varying surface reflectance under unknown il-lumination.
ACM Trans. Graph. , 35(6), December 2016.[25] Z. Xu, J. B. Nielsen, J. Yu, H. W. Jensen, and R. Ra-mamoorthi. Minimal brdf sampling for two-shot near-fieldreflectance acquisition.
ACM Trans. Graph. , 35(6):188:1–188:12, Nov. 2016.[26] W. Ye, X. Li, Y. Dong, P. Peers, and X. Tong. Single pho-tograph surface appearance modeling with self-augmentedCNNs and inexact supervision.
Computer Graphics Forum ,37(7), Oct 2018.[27] Z. Zhou, G. Chen, Y. Dong, D. Wipf, Y. Yu, J. Snyder, andX. Tong. Sparse-as-possible svbrdf acquisition.
ACM Trans.Graph. , 35, November 2016., 35, November 2016.