Spectral DiffuserCam: lensless snapshot hyperspectral imaging with a spectral filter array
Kristina Monakhova, Kyrollos Yanny, Neerja Aggarwal, Laura Waller
RResearch Article Vol. X, No. X / April 2016 / Optica 1
Spectral DiffuserCam: lensless snapshothyperspectral imaging with a spectral filter array K RISTINA M ONAKHOVA , K
YROLLOS Y ANNY , N
EERJA A GGARWAL , AND L AURA W ALLER Department of Electrical Engineering & Computer Sciences, University of California, Berkeley, CA, 94720, USA UCB/UCSF Joint Graduate Program in Bioengineering, University of California, Berkeley, CA, 94720, USA * Corresponding authors: [email protected], [email protected] † These authors contributed equally to this workCompiled June 16, 2020
Hyperspectral imaging is useful for applications ranging from medical diagnostics to crop mon-itoring; however, traditional scanning hyperspectral imagers are prohibitively slow and expen-sive for widespread adoption. Snapshot techniques exist but are often confined to bulky bench-top setups or have low spatio-spectral resolution. In this paper, we propose a novel, compact,and inexpensive computational camera for snapshot hyperspectral imaging. Our system con-sists of a repeated spectral filter array placed directly on the image sensor and a diffuser placedclose to the sensor. Each point in the world maps to a unique pseudorandom pattern on the spec-tral filter array, which encodes multiplexed spatio-spectral information. A sparsity-constrainedinverse problem solver then recovers the hyperspectral volume with good spatio-spectral reso-lution. By using a spectral filter array, our hyperspectral imaging framework is flexible and canbe designed with contiguous or non-contiguous spectral filters that can be chosen for a givenapplication. We provide theory for system design, demonstrate a prototype device, and presentexperimental results with high spatio-spectral resolution. © 2020 Optical Society of Americaunder the terms of the OSA Open Access Publishing Agreement http://dx.doi.org/10.1364/optica.XX.XXXXXX
1. INTRODUCTION
Hyperspectral imaging systems aim to capture a 3D spatio-spectral cube containing spectral information for each spatiallocation. This enables the detection and classification of differentmaterial properties through spectral fingerprints, which cannotbe seen with an RGB camera alone. Hyperspectral imaging hasbeen shown to be useful for a variety of applications, from cropmonitoring to medical diagnostics, multispectral microscopy,and food quality analysis [1–9]. Despite the potential utilityof hyperspectral imaging, commercial hyperspectral camerasrange from $25,000 - $100,000. This high price point and thelarge size are prohibitive for most consumer technology andhave limited the widespread use of hyperspectral imagers.Traditional hyperspectral imagers rely on scanning either thespectral or spatial dimension of the hyperspectral cube withspectral filters or line-scanning [10–12]. These methods can be slow and generally require precise moving parts, increasing thecamera complexity. More recently, snapshot techniques haveemerged, enabling capture of the full hyperspectral data cubein a single shot. Some snapshot methods trade-off spatial res-olution for spectral resolution by using a color filter array orsplitting up the camera’s field-of-view (FOV). Computationalimaging approaches can circumvent this trade-off by spatio-spectrally encoding the incoming light, then solving a compres-sive sensing inverse problem to recover the spectral cube [13].These systems are typically table-top instruments with bulky re-lay lenses, prisms, or diffractive elements, suitable for laboratoryexperiments, but not the real world. Recently, several compactsnapshot hyperspectral imagers have been demonstrated byencoding spatio-spectral information within a single optic, en-abling a practical form factor [14–16]. Using a single optic tocontrol both the spectral and spatial resolution, they measurecontiguous spectral bins within a given spectral band. a r X i v : . [ ee ss . I V ] J un esearch Article Vol. X, No. X / April 2016 / Optica 2 Fig. 1.
Overview of the Spectral DiffuserCam imaging pipeline, which reconstructs a hyperspectral datacube from a single-shot 2Dmeasurement. The system consists of a diffuser and spectral filter array bonded to an image sensor. A one time calibration proce-dure is used to measure the point spread function (PSF) and filter function. Images are reconstructed using a non-linear inverseproblem solver with a sparsity prior. The result is a 3D hyperspectral cube with 64 channels of spectral information for each of448 ×
320 spatial points, generated from a 2D sensor measurement that is 448 ×
320 pixels.Here, we propose a new encoding scheme that takes advan-tage of recent advances in patterned thin film spectral filters [17],as well as lensless imaging, to achieve high-resolution snapshothyperspectral imaging in a small form factor. Our system con-sists of a repeating spectral filter array placed directly onto theimaging sensor and a randomizing phase mask (i.e. diffuser)placed a small distance away from the sensor, as in the Diffuser-Cam architecture [18]. The diffuser spatially multiplexes theincoming light, such that each spatial point in the world mapsto many pixels on the camera. The spectral filter array thenspectrally encodes the incoming light via a structured erasurefunction. The multiplexing effect of the diffuser allows recov-ery of scene information from a subset of sensor pixels, so weare able to recover the full spatio-spectral cube without the lossin resolution that would result from using a non-multiplexingoptic, such as a lens.Our encoding scheme enables hyperspectral recovery in acompact and inexpensive form factor. The spectral filter ar-ray can be manufactured directly on the sensor, costing under$5 for both the diffuser and the spectral filter array at scale.A key advantage of our system over previous work in com-pact snapshot hyperspectral imagers is that it decouples thespectral and spatial responses, enabling a flexible design inwhich contiguous or non-contiguous spectral filters with user-selected bandwidths can be chosen. This should find use in task-specific/classification applications [19–21], where one may wishto measure multiple non-contiguous spectral bands, or havehigher-resolution spectral sampling for certain bands. Givensome conditions and scene sparsity, the spectral sampling isdetermined by the spectral filters and the spatial resolution isdetermined by the autocorrelation of the diffuser response.We present theory for our system, simulations to motivate theneed for a diffuser, and experimental results from a prototypesystem. The main contributions of our paper are:1. A novel framework for snapshot hyperspectral imaging thatcombines compressive sensing with spectral filter arrays,enabling compact and inexpensive hyperspectral imaging.2. Theory and simulations analyzing the system’s spatio-spectral resolution for objects with varying complexity.3. A prototype device demonstrating snapshot hyperspectralrecovery on real data from natural scenes.
2. RELATED WORK
A. Snapshot Hyperspectral Imaging
There have been a variety of snapshot hyperspectral imagingtechniques proposed and evaluated over the past decades. Mostapproaches can be categorized into the following groups: spec-tral filter-based methods, coded aperture methods, speckle-based methods, and dispersion-based methods.
Spectral filter array methods use tiled spectral filter arrayson the sensor to recover the spectral channels of interest [22].These methods can be viewed as an extension of Bayer filtersfor RGB imaging. As the number of filters increases (increasingthe spectral resolution), the spatial resolution decreases. Forinstance, with an 8 × × worse in each direction than that of thecamera sensor. Demosaicing methods have been proposed toimprove upon this, however they rely on intelligently guessinginformation that is not recorded by the sensor [23]. Our systemuses a spectral filter array, but combines it with a randomizingdiffuser in a lensless imaging architecture, allowing us to recoverclose to the full spatial resolution of the sensor, which is notpossible with traditional lens-based spectral filter array methods. Coded aperture methods use a coded aperture, in combina-tion with a dispersive optical element (e.g. a prism or diffrac-tive grating), in order to modulate the light and encode spatial-spectral information [13, 24–26]. These systems are able to cap-ture hyperspectral images and videos but tend to be large table-top systems consisting of multiple lenses and optical compo-nents. In contrast, our system has a much smaller form factor,requiring only a camera sensor with an attached spectral filterarray and a thin diffuser placed close to the sensor.
Speckle-based methods use the wavelength dependence ofspeckle from a random media to achieve hyperspectral imaging.This has been demonstrated for compact spectrometers [27, 28]and extended to hyperspectral imaging [14, 15]. These systemscan be compact, since they require only a sensor and scatteringmedia as their optic; however their spectral resolution is lim-ited by the speckle correlation through wavelengths. This ischallenging to design for a given application, since the spatialand spectral resolutions are highly coupled. In contrast, oursystem uses spectral filters that can easily be adjusted for a givenapplication and can be selected to have variable bandwidth ornon-uniform spectral sampling.
Dispersive methods utilize the dispersion from a prism or esearch Article Vol. X, No. X / April 2016 / Optica 3 high NA lens low NA lens Spectral DiffuserCam super-pixelfilter pixel
Fig. 2.
Motivation for Multiplexing. A high-NA lens captureshigh-resolution spatial information, but misses the yellowpoint source, since it comes into focus on a spectral filter pixeldesigned for blue light. A low-NA lens blurs the image ofeach point source to be the size of the spectral filter’s super-pixel, capturing accurate spectra at the cost of poor spatialresolution. Our DiffuserCam approach multiplexes the lightfrom each point source across many super-pixels, enabling thecomputational recovery of both point sources and their spectrawithout a loss in spatial resolution. Note that a simplified 3 × B. Lensless Imaging
Lensless, mask-based imaging systems do not have a main lens,but instead use an amplitude or phase mask in place of imagingoptics. These systems have been demonstrated for very com-pact, small form factor 2D imaging [32–34]. They are generallyamenable to compressive imaging, due to the multiplexing na-ture of lensless architectures; each point in the scene maps tomany pixels on the sensor, allowing a sparse scene to be com-pletely recovered from a subset of sensor pixels [35]. Or, onecan reconstruct higher-dimensional functions like 3D [18] orvideo [36] from a single 2D measurement. In this work, we usediffuser-based lensless imaging to spatially-multiplex light ontoa repeated spectral filter array, then reconstruct 3D hyperspectralinformation with spatial resolution better than the array super-pixel size, despite the missing information due to the array.
Forward Model filter for Contributions from each spectral band + measurement ... filter fordiffuser ...... Fig. 3.
Image formation model for a scene with two pointsources of different colors, each with narrow-band irradiancecentered at λ y (yellow) and λ r (red). The final measurementis the sum of the contributions from each individual spectralfilter band in the array. Due to the spatial multiplexing of thelensless architecture, all scene points v ( x , y , λ ) project informa-tion to multiple spectral filters, which is why we can recover ahigh-resolution hyperspectral cube from a single image, aftersolving an inverse problem.
3. SYSTEM DESIGN OVERVIEW
Our system leverages recent advances in both spectral filterarray technology and compressive lensless imaging to decouplethe spectral and spatial design in ways that are not possiblewhen using a single optic to encode spectral-spatial information.Furthermore, the spectral filter arrays can be deposited directlyon the camera sensor. With a diffuser as our multiplexing optic,the system is compact and inexpensive at scale.To motivate our need for a multiplexing optic instead of animaging lens, let us consider three candidate architectures: onewith a high numerical aperture (NA) lens whose diffraction-limited spot size is matched to the filter pixel size, one witha low-NA lens whose diffraction-limited spot size is matchedto the super-pixel size, and one with our diffuser as a multi-plexing optic. Figure 2 illustrates these three scenarios with asimplified spectral filter array consisting of 3 × N pixel , the spectral filter array has square filters of size N filter ,and each 3 × super-pixel of size N super-pixel , where N pixel < N filter < N super-pixel .In the high-NA lens case, a spectrally-narrow point sourcein the scene will be imaged onto a single pixel of the sensor,and thus will only be measured if it is within the passband ofthe filter it is imaged to; otherwise it will be filtered out andnot recorded (Fig. 2 (left)). In the low-NA lens case, each pointsource will be imaged to an area the size of the filter array super-pixel, and thus recorded by the sensor correctly, but at the priceof low spatial-resolution (matched to the the super-pixel size)Fig. 2 (middle). In contrast, a multiplexing optic can avoid thegaps in the measurement of the high-NA lens and achieve betterresolution than the low-NA case.A diffuser multiplexes the light from each point source suchthat it hits many filter pixels, covering all of the spectral bands.Given conditions on the scene sparsity and system incoherence,images could potentially be recovered with a spatial resolution esearch Article Vol. X, No. X / April 2016 / Optica 4 on the order of the camera pixel size, Fig. 2 (right). In practice,the spatial resolution of our system will be bounded by the au-tocorrelation of the point spread function (PSF), as detailed inSec. 7, and the diffuser PSF must span multiple super-pixelsto ensure that each point in the world is captured. Since com-pressive recovery is used to recover a 3D hyperspectral cubefrom a 2D measurement, the resolution is a function of the scenecomplexity, as described in Sec. 7.
4. IMAGING FORWARD MODEL
Given our chosen design with a diffuser placed in front of asensor that has a spectral filter array on top of it, in this sectionwe outline a forward model for the optical system, illustratedin Fig. 3. This model is a critical piece of our iterative inversealgorithm for hyperspectral reconstruction and will also be usedto analyze spatial and spectral resolution.
A. Spectral filter model
The spectral filter array is placed on top of an imaging sensor,such that the exposure on each pixel is the sum of point-wisemultiplications with the discrete filter function, L [ x , y ] = K − ∑ λ = F λ [ x , y ] · v [ x , y , λ ] , (1)where · denotes point-wise multiplication, v [ x , y , λ ] is the spec-tral irradiance incident on the filter array and F λ [ x , y ] is a 3Dfunction describing the transmittance of light through the spec-tral filter for K wavelength bands, which we call the filter function .In this model, we absorb the sensor’s spectral response into thedefinition of F λ [ x , y ] . Our device’s filter function is determinedexperimentally (see Sec 6.C) and shown in Fig. 4(b). This can begeneralized to any arbitrary spectral filter design and does notassume alignment between the filter pixels and the sensor pixels.Here, we focus on the case of a repeating grid of spectral filters,where each ’super-pixel’ consists of a set of narrow-band filters.Our device has a 8 × × B. Diffuser model
The diffuser (a smooth pseudorandom phase optic) in our sys-tem achieves spatial multiplexing; this results in a compact formfactor and enables reconstruction with spatial resolution betterthan the super-pixel size via compressed sensing. The diffuseris placed a small distance away from the sensor and an apertureis placed on the diffuser to limit higher angles. The sensor planeintensity resulting from the diffuser can be modeled as a convo-lution of the scene, v [ x , y , λ ] with the on-axis PSF, h [ x , y ] [33]: w [ x , y , λ ] = crop (cid:16) v [ x , y , λ ] [ x , y ] ∗ h [ x , y ]) (cid:17) (2)where [ x , y ] ∗ represents a discrete 2D linear convolution over spa-tial dimensions. The crop function accounts for the finite sensorsize. We assume that the PSF does not vary with wavelength andvalidate this experimentally in Sec. 6.B. However, this model canbe easily extended to include a spectrally-varying PSF, h [ x , y , λ ] if there is more dispersion across wavelengths. C. Combined model
Combining the spectral filter model with the diffuser model, wehave the following discrete forward model: b = K − ∑ λ = F λ [ x , y ] · crop (cid:16) h [ x , y ] [ x , y ] ∗ v [ x , y , λ ] (cid:17) (3) = K − ∑ λ = F λ [ x , y ] · w [ x , y , λ ] (4) = Av . (5)The linear forward model is represented by the combined op-erations in matrix A . Figure 3 illustrates the forward modelfor several point sources, showing the intermediate variable w [ x , y , λ ] , which is the scene convolved with the PSF, beforepoint-wise multiplication by the filter function. The final imageis the sum over all wavelengths.
5. SPECTRAL RECONSTRUCTION
To recover the hyperspectral datacube from the 2D measurement,we must solve an underdetermined inverse problem. Sinceour system falls within the framework of compressive sensingdue to our incoherent, multiplexed measurement, we can use l minimization to solve this problem. We use a weighted 3D totalvariation (3DTV) prior on the scene, as well as a non-negativityconstraint, and a low rank prior on the spectrum. This can bewritten as:ˆ v = arg min v ≥ (cid:107) b − Av (cid:107) + τ (cid:107)∇ xy λ v (cid:107) + τ (cid:107) v (cid:107) ∗ , (6)where ∇ xy λ = [ ∇ x ∇ y ∇ λ ] T is the matrix of forward finite dif-ferences in the x , y , and λ directions, (cid:107) · (cid:107) ∗ represents the nu-clear norm, which is the sum of singular values. τ and τ arethe tuning parameters for the 3DTV prior and low rank priors,respectively. We use the fast iterative shrinkage-thresholdingalgorithm (FISTA) [37] with weighted anisotropic 3DTV to solvethis problem according to [38].
6. IMPLEMENTATION DETAILS
We built a prototype system using a CMOS sensor, a hyperspec-tral filter array provided by Viavi Solutions (Santa Rosa, CA)[17]and an off-the-shelf diffuser (Luminit 0.5°) placed 1cm awayfrom the sensor. The sensor has a resolution of 659 ×
494 pixels(with a pixel pitch of 9.9 µ m ), which we crop down to 448 × ×
20 super-pixels, each with an 8 × µ m in size, covering a little over 4 sensor pixels, andhas a distinct narrow-band spectral response. The alignmentbetween the sensor pixels and the filter pixels is unknown, sorequires a calibration procedure, detailed in Sec. 6A. A. Filter Function Calibration
To calibrate the filter function, including the spectral sensitivityof both the sensor and the spectral filter array, we use a Corner-stone 130 1/3m motorized monochromator (Model 74004). Themonochromater creates a narrow-band source of 5nm full-widthhalf-maximum (FWHM) and we measure the filter response(without the diffuser) while sweeping the source by 8nm incre-ments from 378nm to 890nm. The result is shown in Fig. 4(b). esearch Article Vol. X, No. X / April 2016 / Optica 5
Fig. 4.
Experimental calibration of Spectral DiffuserCam. (a)The caustic PSF (contrast-stretched and cropped), before pass-ing through the spectral filter array, is similar at all wave-lengths. (b) The spectral response with the filter array only(no diffuser). (Top left) Full measurement with illuminationby a 458nm plane wave. The filter array consists of 8 × ×
20 super-pixels. (Top right)Spectral responses of each of the 64 color channels. (Bottom)Spectral response of a single super-pixel as illumination wave-length is varied with a monochromater.
B. PSF Calibration
We also need to calibrate the diffuser response by measuring thediffuser PSF pattern without the spectral filter array. Becausethe diffuser is relatively smooth with large features (relative tothe wavelength of light), the PSF remains relatively constantas a function of wavelength, as shown in Fig. 4(a). Hence, weonly need to calibrate for a single wavelength by capturing asingle point source calibration image [18]. However, this is nottrivial because the spectral filter array is bonded to the sensorand cannot be removed easily. In our setup, we instead take advantage of the fact that our filter array is smaller than oursensor size, so we can measure the PSF using the edges of the rawcamera sensor, by shifting the point source to scan the differentparts of the PSF over the raw sensor area. In a system where thefilter size is matched to the sensor, this trick will not be possible,but an optimization-based approach could be used to recoverthe PSF from measurements of randomly shifted point sources.
7. RESOLUTION ANALYSIS
Here, we derive our theoretical resolution and experimentallyvalidate it with our prototype system. First, we compute theexpected two-point spatial and spectral resolution, based on thePSF autocorrelation and spectral filter passband, respectively.Since our resolution is scene-dependent, we expect the resolutionto degrade with scene complexity; to attempt to characterizethis, we present theory for multi-point resolution based on thecondition number analysis introduced in [18]. We compare oursystem against those with high-NA and low-NA lens instead ofa diffuser. Our results indicate a spatial resolution of ∼ .25 super-pixels and a spectral resolution of ∼ b. Experimental Spatial Resolution λ =
626 nm λ =
466 nm λ =
522 nm a. Theoretical Spatial Resolution pixels −20 −10 0 10 20pixels0.40.60.81.0 vertical autocorrelation horizontal autocorrelation
Fig. 5.
Spatial Resolution analysis. (a) The theoretical resolu-tion of our system, defined as the half-width of the autocorre-lation peak at 70% its maximum value, is 0.15 super-pixels. (b)Experimental two-point reconstructions demonstrate betterthan 0.3 super-pixel resolution across all wavelengths.
A. Two-point Resolution
Spatial resolution of our system, in terms of the two-point res-olution, will be bounded by the resolution from the diffuserwithout the spectral filter array. The expected resolution of alensless imager can be defined as the autocorrelation peak half-width at 70% the maximum value [33], as shown in Fig. 5(a).For our system, this is ∼ esearch Article Vol. X, No. X / April 2016 / Optica 6 b. Spectral Resolutiona. Spectral Detection n o r m a l i z e d i n t e n s i t y profile at 681 nm010.5 profile at 553 nm010.5 i n t e n s i t y r e s o l u t i o n ( n m ) i n t e n s i t y
23 nm
Fig. 6.
Spectral resolution analysis. (a) Sample hyperspectralreconstructions of narrow-band point sources overlaid on topof each other, with shaded lines indicating the ground-truthspectra. For each case, the recovered spectral peak matchesthe true wavelength within 5nm. (b) Two-point spectral reso-lution varies from 23 nm to 46 nm, as determined by applyingRayleigh’s criterion to a reconstruction of synthetically addedpoint sources with different wavelengths.
Spectral resolution is determined by the spectral channelsof the filter array. In this case, we expect to be able to resolve64 spectral channels spaced 8nm apart across a range from 378-898nm. To validate, we scan a point source across those wave-lengths using a monochrometer. Figure 6(a) shows a sampleof the spectral reconstructions overlaid over each other, withthe shaded blocks indicating the ground-truth monochrometerspectra. Our reconstructions closely match the ground-truthpeaks. The small red peaks around 400nm are artifacts from themonochrometer used for calibration, which emitted a 2nd peakaround 400nm for the longer wavelengths. To determine the two-point spectral resolution of our system, we synthetically addthe raw data from point sources at two different wavelengthsand reconstruct a scene of two points at the same spatial po-sition with varying separations in wavelength. We determinethe spectral resolution by applying the Rayleigh criterion to thespectral dimension of the reconstruction. Figure 6(b) shows themeasured spectral resolution and how it varies with wavelength.Our system achieves 30nm two-point spectral resolution acrossmost of the 390-900nm spectral range.
B. Multi-point resolution
Because our image reconstruction algorithm contains nonlin-ear regularization terms, our reconstruction resolution will beobject dependent. Hence, two-point resolution measurementsare not sufficient for fully characterizing the system resolution,and should be considered a best case scenario. To better predictreal-world performance, we perform a local condition numberanalysis, as introduced in [18], that estimates resolution as afunction of object complexity. The local condition number is aproxy for how well the forward model can be inverted, givenknown support, and is useful for systems such as ours in whichthe full A matrix is never explicitly calculated [39].The local condition number theory states that given knowl-edge of the a priori support of the scene, v , we can form a b. 3D condition number (Spatio-Spectral) a. 2D condition number (Spatial) x y x y λ dd λ c o n d i t i o n n u m b e r separation distance (super-pixels)separation distance (super-pixels) S p e c t r a l D i ff u s e r C a m l o w N A l e n s h i g h N A l e n s number of objects number of objects Fig. 7.
Condition number analysis for Spectral DiffuserCam,as compared to a low-NA or high-NA lens. (a) Condition num-bers for the 2D spatial case (single spectral channel) are calcu-lated by generating different numbers of points on a 2D grid,each with separation distance d . (b) Condition numbers forthe full spatio-spectral case are calculated on a 3D grid. A con-dition number below 40 is considered to be good (shown ingreen). The diffuser has a consistently better performance forsmall separation distances than either the low-NA or the high-NA lens. The diffuser can resolve objects as low as 0.25 super-pixels apart for more complex scenes, whereas the low-NAlens requires larger separation distances closer to 1 super-pixelaway.sub-matrix consisting only of columns of A corresponding to thenon-zero voxels. The reconstruction problem will be ill-posedif any of the sub-matrices of A are ill-conditioned, which canbe quantified by the condition number of the sub-matrices. Theworst-case condition number will be when sources are near eachother, therefore we compute the condition number for a group ofpoint sources with a separation varying by an integer number ofvoxels and repeat this for increasing numbers of point sources.In Fig. 7, we calculate the local condition number for twocases: the 2D spatial reconstruction case, considering only asingle spectral channel, and the 3D case, considering pointswith varying spatial and spectral positions. For comparison, wealso simulate the condition number for a low-NA and high-NAlens, as introduced in Sec. 3. The results show that our diffuserdesign has a consistently lower condition number than eitherthe low- or high-NA lens, having a low condition number forseparation distances of greater than ∼ ∼ esearch Article Vol. X, No. X / April 2016 / Optica 7 Fig. 8.
Simulated hyperspectral reconstructions comparing ourSpectral DiffuserCam, a low-NA lens and a high-NA lens. (a)Resolution target consisting of pairs of three bars illuminatedby narrow-band 634nm (red), 570nm (green), 474nm (blue),and a broadband (white) sources. (b-d) Reconstructions ofthe resolution target by (b) our Spectral DiffuserCam, (c) low-NA lens, and (d) high-NA lens, showing the raw data, false-colored reconstruction (top) and λ y sum projection (bottom)for each. The diffuser achieves higher spatial resolution andbetter accuracy than the low-NA and the high-NA lens. C. Simulated Resolution Target Reconstruction
Next, we validate the results of our condition number analysisthrough simulated reconstructions of a resolution target withdifferent spatial locations illumination by different sources (red,green, blue and white light source) (Fig. 8). For each simulation,we add additive Gaussian noise with a variance of 1 × − and run the reconstruction for 2,000 iterations of FISTA with3DTV. Our system resolves features that are 0.3 super-pixelsapart, whereas the low-NA lens can only resolve features thatare roughly 1 super-pixel apart and the high-NA lens results ingaps, validating our predicted reconstruction resolution.
8. EXPERIMENTAL RESULTS
Experimental reconstructions of a broadband USAF resolutiontarget displayed on a computer monitor, and a grid of RGB LEDsare shown in Fig. 9. We resolve points that are .38 super-pixelsapart, which is slightly worse than the expected 0.25 super-pixelresolution. For the RGB LED scene, the ground truth spectral
Fig. 9. (a) Experimental reconstruction of a resolution targetshowing the xy sum projection (top) and λ y sum projection(bottom), demonstrating resolution of 0.4 super-pixels. (b)Experimental reconstruction of 10 multi-colored LEDs in agrid (four red LEDs on left, four green LEDs in middle andtwo blue at right). We show the xy sum projection (top) and λ y sum projection (bottom). The LEDs are clearly resolvedspatially and spectrally. Spectral line profiles from within eachcolor LED are compared with the ground truth spectra from aspectrometer, showing an accurate spectral reconstruction. Aline profile shows that the points are ∼
9. DISCUSSION
Our work presents a new hyperspectral imaging modality thatcombines a color filter array and lensless imaging techniques foran ultra-compact and inexpensive hyperspectral camera. Thisimaging modality is flexible and the spectral filters can easilybe swapped out for different wavelengths; one can non-linearlysample a wide range of wavelengths (which is difficult withmany previous snapshot hyperspectral imagers). This imagingmodality could be useful for numerous applications, especiallysince the price is several orders of magnitude lower than cur-rently available hyperspectral cameras.Currently, we experimentally achieve a spatial resolutionof 0.37 super-pixels, or 6 sensor pixels. In future designs, weshould be able to achieve the full sensor resolution by specificallydesigning the randomizing optic instead of using an off-the shelf esearch Article Vol. X, No. X / April 2016 / Optica 8
Fig. 10.
Experimental hyperspectral reconstructions. (a-c) Reconstructions of color images displayed on a computer monitor and (d)a cup and scissors placed in front of the imager. The false color images and x λ sum projections are shown, as well as spectral lineprofiles for four spatial points in each scene.diffuser. This could be achieved using methods such as end-to-end optical design [40, 41].This system has two main limitations: light-throughput andscene-dependence. Due to our use of narrow-band spectral filterarrays, much of the light is filtered out by the filters. This pro-vides good spectral accuracy and discrimination, but at the costof low light throughput. In addition, since the light is spread bythe diffuser over multiple pixels, the signal to noise ratio (SNR)is further decreased. Hence, our imager is not currently suitablefor low-light conditions. This light-throughput limitation canbe mitigated in the future using end-to-end design of both thespectral filters as well as the phase mask in order to increaselight-throughput while maintaining spatio-spectral resolutionand accuracy. Further, the flexibility of the spectral filter meansthat application-specific designs can use only the set of wave-lengths necessary for a particular task, without unnecessarilymeasuring unimportant wavelengths. This will improve bothlight throughput (because more sensor area will be dedicatedto each spectral band) and spatial resolution (because the super-pixels will be smaller) in cases where fewer wavelengths thanused here are needed.Our second limitation is scene-dependence, as our reconstruc-tion algorithm relies on object sparsity (e.g. sparse gradients).It is difficult to predict performance and one might suffer arti-facts if the scene is not sufficiently sparse. Advances in machinelearning and inverse problems seek to provide better signal rep- resentations, enabling the reconstruction of more complicated,denser scenes. In addition, machine learning could be useful inspeeding up the reconstruction algorithm as well as potentiallyutilizing the imager more directly for a higher-level task, suchas classification [42].
10. CONCLUSION
We demonstrated a compact inexpensive snapshot imaging sys-tem that encodes hyperspectral information using a diffuser andspectral filter array. The spectral filter array encodes spectralinformation onto the sensor and the diffuser multiplexes theincoming light such that each point in the world maps to manyspectral filters. The multiplexed nature of the measurementallows us to use compressive sensing to reconstruct high spatio-spectral resolution from a single 2D measurement. We providedan analysis for the expected resolution of our imager and experi-mentally characterized the two-point and multi-point resolutionof the system. Finally, we showed example reconstructions of acomplex spatio-spectral scene.
FUNDING INFORMATION
This work was supported by the Gordon and Betty Moore Foun-dation Data-Driven Discovery Initiative Grant GBMF4562, andSTROBE: A National Science Foundation Science & TechnologyCenter under Grant No. DMR 1548924. Kristina Monakhova esearch Article Vol. X, No. X / April 2016 / Optica 9 and Kyrollos Yanny acknowledge funding from the National Sci-ence Foundation Graduate Research Fellowship Program (NSFGRFP) (DGE 1752814). The camera and spectral filter array wereprovided by Viavi Solutions (Santa Rosa, CA).
ACKNOWLEDGMENTS
The authors would like to thank Viavi Solutions (Santa Rosa,CA), and particularly Bill Houck, for their technical help andsupport, as well as Nick Antipa and Grace Kuo for helpfuldiscussions.
DISCLOSURES
The authors declare no conflicts of interest.
REFERENCES
1. S. Delalieux, A. Auwerkerken, W. W. Verstraeten,B. Somers, R. Valcke, S. Lhermitte, J. Keulemans, andP. Coppin, “Hyperspectral reflectance and fluorescenceimaging to detect scab induced stress in apple leaves,” Re-mote. sensing , 858–874 (2009).2. R. T. Kester, N. Bedard, L. S. Gao, and T. S. Tkaczyk,“Real-time snapshot hyperspectral imaging endoscope,” J.biomedical optics , 056005 (2011).3. G. Lu and B. Fei, “Medical hyperspectral imaging: a re-view,” J. biomedical optics , 010901 (2014).4. D.-W. Sun, Hyperspectral imaging for food quality analysis andcontrol (Elsevier, 2010).5. A. Gowen, C. O’Donnell, P. Cullen, G. Downey, and J. Frias,“Hyperspectral imaging–an emerging process analyticaltool for food quality and safety control,” Trends food sci-ence & technology , 590–598 (2007).6. H. Akbari, L. Halig, D. M. Schuster, B. Fei, A. Osunkoya,V. Master, P. Nieh, and G. Chen, “Hyperspectral imagingand quantitative analysis for prostate cancer detection,” J.biomedical optics , 076005 (2012).7. G. Lu, L. V. Halig, D. Wang, X. Qin, Z. G. Chen, and B. Fei,“Spectral-spatial classification for noninvasive cancer detec-tion using hyperspectral imaging,” J. biomedical optics ,106004 (2014).8. A. Orth, M. J. Tomaszewski, R. N. Ghosh, and E. Schonbrun,“Gigapixel multispectral microscopy,” Optica , 654–662(2015).9. W. Huang, J. Li, Q. Wang, and L. Chen, “Developmentof a multispectral imaging system for online detection ofbruises on apples,” J. Food Eng. , 62–71 (2015).10. R. O. Green, M. L. Eastwood, C. M. Sarture, T. G. Chrien,M. Aronsson, B. J. Chippendale, J. A. Faust, B. E. Pavri,C. J. Chovit, M. Solis et al. , “Imaging spectroscopy and theairborne visible/infrared imaging spectrometer (aviris),”Remote. sensing environment , 227–248 (1998).11. N. Gat, “Imaging spectroscopy using tunable filters: a re-view,” in Wavelet Applications VII, vol. 4056 (InternationalSociety for Optics and Photonics, 2000), pp. 50–64.12. C. Zhang, M. Rosenberger, A. Breitbarth, and G. Notni, “Anovel 3d multispectral vision system based on filter wheelcameras,” in (IEEE, 2016), pp. 267–272.13. A. Wagadarikar, R. John, R. Willett, and D. Brady, “Sin-gle disperser design for coded aperture snapshot spectralimaging,” Appl. optics , B44–B51 (2008). 14. S. K. Sahoo, D. Tang, and C. Dang, “Single-shot multi-spectral imaging with a monochromatic camera,” Optica ,1209–1213 (2017).15. R. French, S. Gigan, and O. L. Muskens, “Speckle-basedhyperspectral imaging combining multiple scattering andcompressive sensing in nanowire mats,” Opt. letters ,1820–1823 (2017).16. D. S. Jeon, S.-H. Baek, S. Yi, Q. Fu, X. Dun, W. Heidrich,and M. H. Kim, “Compact snapshot hyperspectral imagingwith diffracted rotation,” ACM Trans. Graph. (2019).17. S. Saxe, L. Sun, V. Smith, D. Meysing, C. Hsiung, A. Houck,M. Von Gunten, C. Hruska, D. Martin, R. Bradley et al. , “Ad-vances in miniaturized spectral sensors,” in Next-GenerationSpectroscopic Technologies XI, vol. 10657 (International Soci-ety for Optics and Photonics, 2018), p. 106570B.18. N. Antipa, G. Kuo, R. Heckel, B. Mildenhall, E. Bostan,R. Ng, and L. Waller, “DiffuserCam: lensless single-exposure 3D imaging,” Optica. , 1–9 (2018).19. K. Chao, C. Yang, Y. Chen, M. Kim, and D. Chan,“Hyperspectral-multispectral line-scan imaging system forautomated poultry carcass inspection applications for foodsafety,” Poult. science , 2450–2460 (2007).20. R. M. Levenson, D. T. Lynch, H. Kobayashi, J. M. Backer,and M. V. Backer, “Multiplexing with multispectral imag-ing: from mice to microscopy,” ILAR journal , 78–88(2008).21. A. Hennessy, K. Clarke, and M. Lewis, “Hyperspectralclassification of plants: A review of waveband selectiongeneralisability,” Remote. Sens. , 113 (2020).22. P.-J. Lapray, X. Wang, J.-B. Thomas, and P. Gouton, “Multi-spectral filter arrays: Recent advances and practical imple-mentation,” Sensors. , 21626–21659 (2014).23. S. Mihoubi, O. Losson, B. Mathon, and L. Macaire, “Multi-spectral demosaicing using pseudo-panchromatic image,”IEEE Transactions on Comput. Imaging , 982–995 (2017).24. M. E. Gehm, R. John, D. J. Brady, R. M. Willett, and T. J.Schulz, “Single-shot compressive spectral imaging with adual-disperser architecture,” Opt. express , 14013–14027(2007).25. X. Lin, Y. Liu, J. Wu, and Q. Dai, “Spatial-spectral encodedcompressive hyperspectral imaging,” ACM Transactionson Graph. (TOG) , 1–11 (2014).26. X. Cao, T. Yue, X. Lin, S. Lin, X. Yuan, Q. Dai, L. Carin,and D. J. Brady, “Computational snapshot multispectralcameras: Toward dynamic capture of the spectral world,”IEEE Signal Process. Mag. , 95–108 (2016).27. B. Redding, S. F. Liew, R. Sarma, and H. Cao, “Compactspectrometer based on a disordered photonic chip,” Nat.Photonics , 746 (2013).28. M. Chakrabarti, M. L. Jakobsen, and S. G. Hanson,“Speckle-based spectrometer,” Opt. letters , 3264–3267(2015).29. S.-H. Baek, I. Kim, D. Gutierrez, and M. H. Kim, “Compactsingle-shot hyperspectral imaging using a prism,” ACMTransactions on Graph. (TOG) , 1–12 (2017).30. M. A. Golub, A. Averbuch, M. Nathan, V. A. Zheludev,J. Hauser, S. Gurevitch, R. Malinsky, and A. Kagan, “Com-pressed sensing snapshot spectral imaging by a regulardigital camera with an added optical diffuser,” Appl. op-tics , 432–443 (2016).31. J. Hauser, M. A. Golub, A. Averbuch, M. Nathan, V. A.Zheludev, and M. Kagan, “Dual-camera snapshot spectralimaging with a pupil-domain optical diffuser and com- esearch Article Vol. X, No. X / April 2016 / Optica 10 pressed sensing algorithms,” Appl. Opt. , 1058–1070(2020).32. M. S. Asif, A. Ayremlou, A. Sankaranarayanan, A. Veer-araghavan, and R. G. Baraniuk, “Flatcam: Thin, lenslesscameras using coded aperture and computation,” IEEETransactions on Comput. Imaging , 384–397 (2016).33. G. Kuo, N. Antipa, R. Ng, and L. Waller, “Diffusercam:diffuser-based lensless cameras,” in Computational OpticalSensing and Imaging, (Optical Society of America, 2017), pp.CTu3B–2.34. J. Tanida, T. Kumagai, K. Yamada, S. Miyatake, K. Ishida,T. Morimoto, N. Kondou, D. Miyazaki, and Y. Ichioka,“Thin observation module by bound optics (tombo): con-cept and experimental verification,” Appl. optics , 1806–1813 (2001).35. R. Fergus, A. Torralba, and W. T. Freeman, “Random lensimaging,” (2006).36. N. Antipa, P. Oare, E. Bostan, R. Ng, and L. Waller, “Videofrom stills: Lensless imaging with rolling shutter,” in (IEEE, 2019), pp. 1–8.37. A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding algorithm for linear inverse problems,” SIAMjournal on imaging sciences , 183–202 (2009).38. U. S. Kamilov, “A parallel proximal algorithm foranisotropic total variation minimization,” IEEE Transac-tions on Image Process. , 539–548 (2016).39. E. J. Candès and C. Fernandez-Granda, “Towards a math-ematical theory of super-resolution,” Commun. on pureapplied Math. , 906–956 (2014).40. V. Sitzmann, S. Diamond, Y. Peng, X. Dun, S. Boyd, W. Hei-drich, F. Heide, and G. Wetzstein, “End-to-end optimiza-tion of optics and image processing for achromatic ex-tended depth of field and super-resolution imaging,” ACMTransactions on Graph. (TOG) , 1–13 (2018).41. Y. Peng, Q. Sun, X. Dun, G. Wetzstein, W. Heidrich, andF. Heide, “Learned large field-of-view imaging with thin-plate optics,” ACM Transactions on Graph. (TOG) , 219(2019).42. K. Monakhova, J. Yurtsever, G. Kuo, N. Antipa, K. Yanny,and L. Waller, “Learned reconstructions for practical mask-based lensless imaging,” Opt. express27