Depth extraction from a single compressive hologram
11 Depth extraction from a single compressivehologram
Baturay Ozgurun and Mujdat Cetin,
Fellow, IEEE
Abstract —We propose a novel method that records a singlecompressive hologram in a short time and extracts the depth ofa scene from that hologram using a stereo disparity technique.The method is verified with numerical simulations, but thereis no restriction on adapting this into an optical experiment.In the simulations, a computer-generated hologram is firstsampled with random binary patterns, and measurements areutilized in a recovery algorithm to form a compressive hologram.The compressive hologram is then divided into two parts (twoapertures), and these parts are separately reconstructed to forma stereo image pair. The pair is eventually utilized in stereodisparity method for depth map extraction. The depth maps ofthe compressive holograms with the sampling rates of 2, 25, and50 percent are compared with the depth map extracted from theoriginal hologram, on which compressed sensing is not applied.It is demonstrated that the depth profiles obtained from thecompressive holograms are in very good agreement with thedepth profile obtained from the original hologram despite thedata reduction.
Index Terms —Stereo image processing, holography, com-pressed sensing.
I. I
NTRODUCTION
Holography is a technique to record and reconstruct a three-dimensional (3D) object. To record a hologram, a coherentlight source is usually divided into two arms: reference andobject beams. The object beam first illuminates the 3D object,and then it reflects toward a beam splitter. The beam carriesphase and amplitude information related to the object. Toextract the phase information, the reference beam is alsorequired. Therefore, the reference and object beams are mergedon the beam splitter. Interference of two beams, which is alsocalled a hologram, is recorded with a camera for a numericalreconstruction.To extract depth from the hologram, several methods havebeen developed. The Fresnel propagation method is one ofthe widely used techniques for hologram reconstruction. Thismethod enables us to calculate the depth of microscopic ob-jects; however, it requires numerical phase unwrapping whenthe phase is wrapped for distances longer than a wavelength[1]. Phase unwrapping is useful for microscopic objects, but
This work was not supported by any organization.B. Ozgurun is with the Department of Biomedical Engineering,University of Rochester, Rochester, NY 14627, USA (e-mail: [email protected]).M. Cetin is with the Department of Electrical and Computer Engineer-ing, University of Rochester, Rochester, NY 14627, USA (e-mail: [email protected]).B. Ozgurun is also with the Faculty of Engineering and Natural Sciences,Sabanci University, Istanbul 34956, TurkeyB. Ozgurun is also with the School of Engineering and Natural Sciences,Istanbul Medipol University, Istanbul 34810, Turkey this is not sufficient for macroscopic objects because of theirsize. Phase shifting is another method for depth extraction,but limited depth of field restricts the depth acquisition formacroscopic objects [2], [3]. Other researchers have shownthat a dual beam illumination can be utilized to obtain twophase-contrast images, and subtraction of these images canprovide depth of macroscopic objects. However, 2 π jumpsreduce the efficiency of this method [4], [5]. In addition, itwas demonstrated that the gray level variance method can beused to extract depth of macroscopic objects, but this techniqueworks mostly when a highly textured object is used [6], [7],[8].Pitk¨aaho and Naughton presented a study for the depthextraction from a single hologram [9]. They sharply dividedthe single hologram along the horizontal direction into twoseparated holograms. Each separated hologram is equally sizedwith the single hologram but contains half of the intensityvalues of it. After intensity division, the separated hologramswere independently reconstructed to form a stereo image pair.Eventually, the image pair was utilized in a stereo disparitymethod to get the depth information related to the object. Wewere inspired by the study of Pitk¨aaho and Naughton and wehave recently demonstrated depth extraction for macroscopicobjects from experimentally recorded holograms [10]. We alsohave shown that depth extraction is mostly independent fromthe division directions (horizontal, vertical, and diagonal) aswell as the division types (gradual and sharp). Although thestudy of Pitk¨aaho and Naughton as well as our previousstudy demonstrated that depth of small and macroscopicobjects could be extracted from a single hologram, high-speed recording and high-speed depth extraction are still chal-lenging problems because of huge data volumes [11]. High-speed depth extraction could be possible with our previouslyproposed approach in [10] because depth can be extractedfrom a single hologram alone, but there is also the desire torecord a hologram in a short time. In this Letter, we proposea method that records a hologram faster using the compressedsensing (CS) framework and extracts depth information from asingle compressive/estimated hologram using a stereo disparitymethod. II. I MAGING VIA C OMPRESSED S ENSING
In conventional imaging, a camera records an image bysampling a scene at the Nyquist rate, and it collects N measurements, where N is the total number of image pixels.Once the image is recorded, it is usually compressed toreduce its dimension. To perform compression, the image is a r X i v : . [ ee ss . I V ] F e b first represented as a sparse image by utilizing a sparsify-ing transform. Then, the most significant coefficients of thetransform domain representation of the image are kept, andthe rest of the coefficients are thrown away. Eventually, theapproximated transform coefficients is back transformed whilekeeping only the most significant coefficients [12]. Given thatmany coefficients are thrown away, one could argue samplingthe scene at the Nyquist rate wastes hardware. The compressedsensing (CS) framework is quite different from conventionalimaging. It does not have to meet the Nyquist sampling rateand then carry out compression operations, rather it needs only M measurements, where M (cid:28) N , to recover a scene. The CSframework has been utilized for a variety of applications suchas radar imaging [13] and magnetic resonance imaging [14],but here we focus on the application of CS in optics. Once ofthe first applications of CS in optics was the development ofa single-pixel camera [15]. In that application, a camera is notused for recording a scene. The scene is first sampled withpseudorandom patterns, which are generated by a digital mi-cromirror device (DMD). Generated random patterns constructa sampling matrix Φ , where Φ ∈ (cid:60) M × N . The inner productsbetween the random patterns and the scene are collected by aphotodiode, which generates measurements y , where y ∈ (cid:60) M .If the scene is sparse enough, a sensing matrix A , where A ∈ (cid:60) M × N , can be constructed only from the samplingmatrix Φ . However, if the scene is dense, the sensing matrix A must be formed with the product of the sampling matrix and asparsifying matrix Ψ , where Ψ ∈ (cid:60) N × N . This operation canbe described as A = ΦΨ . The sparsifying matrix Ψ representsa scene sparsely in an appropriate transform domain. Once themeasurements y and the sensing matrix A are formed, theyare utilized in a non-linear recovery algorithm to estimate thescene ˆ x , where ˆ x ∈ (cid:60) N . Most recovery algorithms are basedon an (cid:96) minimization problem. This can be mathematicallydescribed below with an assumption that measurements arecorrupted by a bounded noise n , i.e. y = Ax + n , where x ∈ (cid:60) N is the scene, and (cid:107) n (cid:107) ≤ ε . ˆ x = min x (cid:107) x (cid:107) s.t. (cid:107) y − Ax (cid:107) ≤ ε (1)There are some restrictions for adapting of CS into anoptical configuration. First, the scene image must be sparseor it must be represented as a sparse image. A dense imagecan be sparsified by utilizing sparsifying matrices. However,it should be considered that the level of sparsity of the imageaffects the performance of the recovery algorithm. Second, thesampling patterns must satisfy the restricted isometry property(RIP). Fortunately, the RIP constraint can be satisfied when thesampling patterns are made from the random binary patternsthat can be easily generated by the DMD [16].In the literature, CS is usually applied to holography forimage reconstruction and data security applications [17], [18],[19]. However, CS can also potentially enable one to recorda scene or a hologram in a short time since it samples thescene with a DMD instead of a camera, and it requires asmall number of measurements to recover the scene. The framerate of a typical DMD on the market is almost 330 timeshigher than that of a camera. In addition, it was demonstrated that it may be possible to recover a scene with a samplingrate of only 2 percent [15]. Considering of the frame rateof the DMD and the ability of CS for recovering the scenewith few measurements, an optical configuration based onCS can record a scene 2 or 3 times faster. Here, we claimthat data collection time of a single hologram can be reducedby utilizing CS. In addition to recording a single hologramfaster, depth extraction is performed from the recorded singlehologram using a stereo disparity method.III. M ETHOD
To demonstrate our claim, a computer-generated hologram(CGH) of the Venus statue, which is provided by DavidBlinder et al. as an open access file, is utilized [20]. TheCGH (1920 x 1080 pixels with a pixel pitch of 8 µm ) andits numerical reconstruction with the Fresnel approximationmethod are presented in Fig. 1. The data dimension of theCGH is high, and this increases the execution time. To reducethe computational cost, the CGH is first transformed intothe Fourier domain, and the low frequency region (one-tenthof the bandwidth) is extracted and then back transformed.This operation compresses the hologram size by 100 times.Although the sharp transitions of the original hologram (1920x 1080 pixels) disappear in the small hologram (192 x 108pixels), most of the information about the structure of theVenus statue is preserved. In this study, this compressedhologram is used for all numerical calculations instead ofthe original hologram, and the small hologram is hereinafterreferred to as the CGH. (a)(b) Fig. 1. The CGH (a) of the Venus statue, and its numerical reconstruction(b) with the Fresnel approximation method.
In the simulation-based experiments, the CGH is consid-ered as a holographic scene, and the DMD is considered to be placed in front of the CGH or a beam splitter thatcombines object and reference beams. The CGH is sampledwith random binary patterns since the DMD can produce thistype of patterns. Inner products between the random binarypatterns and the CGH present measurements. In a real opticalconfiguration, the measurements are usually collected by aphotodiode or photomultiplier tube (PMT). A sensing matrixis constructed from the product of a sampling matrix and asparsifying matrix. The sampling matrix is created from therandom binary patterns while the discrete cosine transform(DCT) is selected as the sparsifying matrix. The measurementsand the sensing matrix are utilized in the NESTA algorithm,which is one of the open source CS recovery algorithms [21].The NESTA algorithm produces an estimated CGH or a com-pressive hologram. The CGH reconstruction and the recon-structions of the compressive holograms with sampling ratesof 2, 25, and 50 percent are shown in Fig. 2. All numericalreconstructions are performed with the Fresnel approximationmethod. The reconstruction result of the CGH is slightly betterthan the reconstruction results of the compressive holograms.We demonstrated that it is possible to record a hologram about1.6 times faster. This corresponds to the 2 percent samplingrate case and assumes the frame rate of the DMD is 330 timeshigher than that of the camera.(a) (b)(c) (d)
Fig. 2. The numerical reconstructions of the CGH (a) and the compressiveholograms with the sampling rate of 50 percent (b), and 25 percent (c) and 2percent (d). The reconstructions are performed with the Fresnel approximationmethod.
Once the compressive holograms are acquired, depth pro-files are also obtained. We applied our previous study [10],which is based on the depth extraction from a single holo-gram, to the compressive holograms. To extract depth froma single compressive hologram, the compressive hologram isfirst divided gradually into two parts (two apertures) alongthe horizontal direction. Each of the separated holograms isequally sized with the single compressive hologram, but eachof them contains almost half of the intensity weights of thesingle hologram. Division direction does not influence the ac-curacy of the depth information significantly; however, gradualdivision provides uniform illumination on the reconstruction,which increases the accuracy of the depth [10]. After thehologram division is performed, two apertures are separately reconstructed with the Fresnel approximation method to forma stereo image pair. The stereo image pair and the separatedholograms are presented in Fig. 3.
Fig. 3. The gradual intensity divisions of the compressive hologram (the firstrow) with the sampling rate of 25 percent, and their numerical reconstructions(the stereo image pair) with the Fresnel approximation method (the secondrow).
To extract depth, stereo disparity estimation is performedon the stereo image pair. The disparity technique generatesdepth map values, which are associated with depth of scenepoints and are usually shown as a gray-scale image. A smalldepth map value represents as a dark pixel in the gray-scaleimage and corresponds to a distant scene point. Similarly, ahigh depth map value or a bright pixel corresponds to a closescene point. In the literature, there exist various stereo disparitytechniques. Here, we utilized the normalized cross-correlation(NCC) algorithm for the depth extraction, since this algorithmis robust to intensity offsets and contrast changes although itis computationally costly [22]. The NCC algorithm calculatesa correlation peak over two rectangular ( k × k ) blocks on thestereo image pair. These blocks are separately located on eachstereo image pair, and they are called reference R ( x, y ) andcandidate C ( x, y ) blocks. Calculation of the NCC is performedaccording to N CC = k (cid:80) x =1 k (cid:80) y =1 (cid:101) R ( x, y ) (cid:101) C ( x + ∆ , y ) (cid:115) k (cid:80) x =1 k (cid:80) y =1 (cid:101) R ( x, y ) k (cid:80) x =1 k (cid:80) y =1 (cid:101) C ( x + ∆ , y ) (2)where (cid:101) R ( x, y ) = R ( x, y ) − R ( x, y ) , (cid:101) C ( x + ∆ , y ) = C ( x +∆ , y ) − C ( x, y ) , and ∆ denotes any shifts applied. R ( x, y ) and C ( x, y ) are the mean pixel values over the referenceand candidate blocks, respectively. Once the first NCC valueis calculated (∆ = 0) , the candidate block is shifted onecolumn for the second NCC calculation (∆ = 1) . The shiftingoperation is usually finalized when the shifting amount reacheshalf of the image size. This process provides a number of NCCvalues. The maximum value is picked and registered for thecenter pixel of the reference block. The overall operation mustbe repeated for the other pixels of the stereo image pair. Thisprovides a depth map for the stereo image pair. IV. E
XPERIMENTAL R ESULTS
Selection of the block size in the NCC algorithm is animportant issue. The block size should be large enough foraccurate matching, and it should be small enough for the lessprojective distortion effects. We used an empirical method todefine the block size, and it was found that the best block sizefor our stereo image pairs was (23 × in terms of estimateddepth map accuracy. Once the depth maps of each hologram(compressive holograms and CGH) are acquired with themethod described above, each of them is separately mergedwith their numerical reconstructions. The reconstructed imagescombined with the depth maps are illustrated in Fig. 4. Thedepth profiles of the Venus statue, corresponding to the linesdisplayed on the depth maps in Fig. 4, along the frontal axisare presented in Fig. 5. The results show that the normalizeddepth profile of the compressive holograms with samplingrates of 2, 25, and 50 percent are very good agreementwith the normalized depth profile of the CGH. These resultsdemonstrates that it is possible to extract depth from a singlecompressive hologram, and that the depth extraction quality isrobust to reductions in sampling rate.(a) (b)(c) (d) Fig. 4. The merging of the hologram reconstructions with the normalizeddepth maps. The depth map of the CGH (a), and also the depth maps of thecompressive holograms with sampling rate of 50 percent (b), 25 percent (c),and 2 percent (d). The depth profile lines of the Venus statue along the frontalaxis are also illustrated on the depth maps.
V. C
ONCLUSION
We have presented a method that not only records a holo-gram faster using the compressed sensing (CS) framework butalso extracts a depth map from a recorded single compressivehologram. CS can be utilized for recording holograms in ashort time, since it requires a small number of measurements toacquire a scene and uses a high-speed sampling device (DMD).In addition, depth can be extracted from the compressivehologram accurately. The results demonstrate that the depth
Fig. 5. The normalized depth profiles for each depth map along the frontalaxis. The profile colors correspond to the colors presented on the depth mapsof the Venus statue. profiles of the compressive holograms are almost the samewith the depth profile of the computer-generated hologram(CGH) although the hologram reconstructions are not exactlysame. This shows that depth extraction does not depend on thehologram reconstruction results or sampling rates so much.R
EFERENCES[1] U. Schnars and W. Jueptner,
Digital Holography: Digital HologramRecording, Numerical Reconstruction, and Related Techniques . Berlin,Germany: Springer, 2005.[2] I. Yamaguchi, J. Kato, and H. Matsuzaki, “Measurement of surface shapeand deformation by phase-shifting image digital holography,”
Opt. Eng. ,vol. 42, no. 5, pp. 1267 – 1271, 2003.[3] I. Yamaguchi and T. Zhang, “Phase-shifting digital holography,”
Opt.Lett. , vol. 22, no. 16, pp. 1268–1270, 1997.[4] D. V. Prieto and J. Garcia-Sucerquia, “Three-dimensional surface con-touring of macroscopic objects by means of phase-difference images,”
Appl. Opt. , vol. 45, no. 25, pp. 6381–6387, 2006.[5] S. M. Sol´ıs, M. S. Hern´andez-Montes, and F. M. Santoyo, “Tympanicmembrane contour measurement with two source positions in digitalholographic interferometry,”
Biomed. Opt. Express , vol. 3, no. 12, pp.3203–3210, 2012.[6] S. Frey, A. Thelen, S. Hirsch, and P. Hering, “Generation of digitaltextured surface models from hologram recordings,”
Appl. Opt. , vol. 46,no. 11, pp. 1986–1993, 2007.[7] L. Ma, H. Wang, Y. Li, and H. Jin, “Numerical reconstruction of digitalholograms for three-dimensional shape measurement,”
J. Opt. A: PureAppl. Opt. , vol. 6, no. 4, pp. 396–400, 2004.[8] C. P. McElhinney et al. , “Depth-independent segmentation of macro-scopic three-dimensional objects encoded in single perspectives ofdigital holograms,”
Opt. Lett. , vol. 32, no. 10, pp. 1229–1231, 2007.[9] T. Pitk¨aaho and T. J. Naughton, “Calculating depth maps from digitalholograms using stereo disparity,”
Opt. Lett. , vol. 36, no. 11, pp. 2035–2037, 2011.[10] B. ¨Ozg¨ur¨un, D. ¨O. Tayyar, K. ¨O. Agis¸, and M. ¨Ozcan, “Three-dimensional image reconstruction of macroscopic objects from a singledigital hologram using stereo disparity,”
Appl. Opt. , vol. 56, no. 13, pp.F84–F90, 2017.[11] D. Blinder et al. , “Signal processing challenges for digital holographicvideo display systems,”
Signal Process. Image Commun. , vol. 70, pp.114–130, 2019.[12] S. Mallat,
A Wavelet Tour of Signal Processing . Elsevier, 1999.[13] L. C. Potter, E. Ertin, J. T. Parker, and M. Cetin, “Sparsity andcompressed sensing in radar imaging,”
Proceedings of the IEEE , vol.98, no. 6, pp. 1006-1020, 2010. [14] M. Lustig, D. L. Donoho, J. M. Santos, and J. M. Pauly, “Compressedsensing MRI,” in
IEEE Signal Processing Magazine , vol. 25, no. 2, pp.72-82, 2008.[15] M. F. Duarte et al. , “Single-pixel imaging via compressive sampling,”
IEEE Signal Processing Magazine , vol. 25, no. 2, pp. 83-91, 2008.[16] E. J. Cand`es and M. B. Wakin, “An introduction to compressivesampling,”
IEEE Signal Processing Magazine , vol. 25, no. 2, pp. 21-30, 2008.[17] T. Leportier and M-C. Park, “Holographic reconstruction by compressivesensing,”
J. Opt. , vol. 19, no. 6, 2017.[18] M. M. Marim, M. Atlan, E. Angelini, and J-C. Olivo-Marin, “Com-pressed sensing with off-axis frequency-shifting holography,”
Opt. Lett. ,vol. 35, no. 6, pp. 871–873, 2010.[19] H. Di et al. , “Multiple-image encryption by compressive holography,”
Appl. Opt. , vol. 51, no. 7, pp. 1000–1009, 2012.[20] D. Blinder et al. , ”Open access database for experimental validationsof holographic compression engines,” in , pp. 1–6,2015.[21] S. Becker, J. Bobin, and E. J. Cand`es, “NESTA: A fast and accuratefirst-order method for sparse recovery,”
SIAM J. Imaging Sci. , vol. 4,no. 1, pp. 1–39, 2011.[22] S. Satoh, “Simple low-dimensional features approximating NCC-basedimage matching,”