Hongda Wang

Deep learning microscopy

We demonstrate that a deep neural network can significantly improve optical microscopy, enhancing its spatial resolution over a large field-of-view and depth-of-field. After its training, the only input to this network is an image acquired using a regular optical microscope, without any changes to its design. We blindly tested this deep learning approach using various tissue samples that are imaged with low-resolution and wide-field systems, where the network rapidly outputs an image with remarkably better resolution, matching the performance of higher numerical aperture lenses, also significantly surpassing their limited field-of-view and depth-of-field. These results are transformative for various fields that use microscopy tools, including e.g., life sciences, where optical microscopy is considered as one of the most widely used and deployed techniques. Beyond such applications, our presented approach is broadly applicable to other imaging modalities, also spanning different parts of the electromagnetic spectrum, and can be used to design computational imagers that get better and better as they continue to image specimen and establish new transformations among different modes of imaging.

Sparsity-based multi-height phase recovery in holographic microscopy

High-resolution imaging of densely connected samples such as pathology slides using digital in-line holographic microscopy requires the acquisition of several holograms, e.g., at >6–8 different sample-to-sensor distances, to achieve robust phase recovery and coherent imaging of specimen. Reducing the number of these holographic measurements would normally result in reconstruction artifacts and loss of image quality, which would be detrimental especially for biomedical and diagnostics-related applications. Inspired by the fact that most natural images are sparse in some domain, here we introduce a sparsity-based phase reconstruction technique implemented in wavelet domain to achieve at least 2-fold reduction in the number of holographic measurements for coherent imaging of densely connected samples with minimal impact on the reconstructed image quality, quantified using a structural similarity index. We demonstrated the success of this approach by imaging Papanicolaou smears and breast cancer tissue slides over a large field-of-view of ~20 mm2 using 2 in-line holograms that are acquired at different sample-to-sensor distances and processed using sparsity-based multi-height phase recovery. This new phase recovery approach that makes use of sparsity can also be extended to other coherent imaging schemes, involving e.g., multiple illumination angles or wavelengths to increase the throughput and speed of coherent imaging.

Computational out-of-focus imaging increases the space–bandwidth product in lens-based coherent microscopy

The space–bandwidth product (SBP) of modern objective lenses is often significantly larger than the pixel count of opto-electronic image sensor chips, and, therefore, much of the information transmitted by the optical system cannot be adequately sampled or digitized. To resolve this mismatch, microscopes are in general designed to maintain the resolution of the optical system while significantly wasting the field of view (FOV) and SBP of the objective lens. We introduce a wide-field and high-resolution coherent imaging method that uses a stack of out-of-focus images to provide much better utilization of the SBP of an objective lens. We demonstrate our approach on a benchtop microscope by using a demagnification camera adapter to match the active area of the image sensor chip to the FOV of an objective lens. We show that the resulting spatial undersampling caused by capturing a large FOV can be mitigated through an iterative pixel super-resolution algorithm that uses e.g., ∼three to five slightly out-of-focus images, yielding an ∼8-fold increase in the SBP of the microscope. Furthermore, the same pixel super-resolution algorithm also achieves phase retrieval, revealing the optical phase information of the specimen. We compared our method against traditional off-axis and phase-shifting digital holographic microscopy modalities and demonstrated at least 3-fold reduction in the number of images required to achieve the same SBP. This technique could be used to maximize the throughput and SBP of lens-based coherent imaging and holography systems and inspire new microscopy designs that benefit from the inherent autofocusing steps of a scanning microscope to increase its SBP.

Deep Learning Enhanced Mobile-Phone Microscopy

Mobile-phones have facilitated the creation of field-portable, cost-effective imaging and sensing technologies that approach laboratory-grade instrument performance. However, the optical imaging interfaces of mobile-phones are not designed for microscopy and produce spatial and spectral distortions in imaging microscopic specimens. Here, we report on the use of deep learning to correct such distortions introduced by mobile-phone-based microscopes, facilitating the production of high-resolution, denoised and colour-corrected images, matching the performance of benchtop microscopes with high-end objective lenses, also extending their limited depth-of-field. After training a convolutional neural network, we successfully imaged various samples, including blood smears, histopathology tissue sections, and parasites, where the recorded images were highly compressed to ease storage and transmission for telemedicine applications. This method is applicable to other low-cost, aberrated imaging systems, and could offer alternatives for costly and bulky microscopes, while also providing a framework for standardization of optical images for clinical and biomedical applications.

3D imaging of optically cleared tissue using a simplified CLARITY method and on-chip microscopy

Using lens-free holographic microscopy, we demonstrated 3D imaging in optically cleared tissue over a thickness of 0.2 mm. High-throughput sectioning and optical imaging of tissue samples using traditional immunohistochemical techniques can be costly and inaccessible in resource-limited areas. We demonstrate three-dimensional (3D) imaging and phenotyping in optically transparent tissue using lens-free holographic on-chip microscopy as a low-cost, simple, and high-throughput alternative to conventional approaches. The tissue sample is passively cleared using a simplified CLARITY method and stained using 3,3′-diaminobenzidine to target cells of interest, enabling bright-field optical imaging and 3D sectioning of thick samples. The lens-free computational microscope uses pixel super-resolution and multi-height phase recovery algorithms to digitally refocus throughout the cleared tissue and obtain a 3D stack of complex-valued images of the sample, containing both phase and amplitude information. We optimized the tissue-clearing and imaging system by finding the optimal illumination wavelength, tissue thickness, sample preparation parameters, and the number of heights of the lens-free image acquisition and implemented a sparsity-based denoising algorithm to maximize the imaging volume and minimize the amount of the acquired data while also preserving the contrast-to-noise ratio of the reconstructed images. As a proof of concept, we achieved 3D imaging of neurons in a 200-μm-thick cleared mouse brain tissue over a wide field of view of 20.5 mm2. The lens-free microscope also achieved more than an order-of-magnitude reduction in raw data compared to a conventional scanning optical microscope imaging the same sample volume. Being low cost, simple, high-throughput, and data-efficient, we believe that this CLARITY-enabled computational tissue imaging technique could find numerous applications in biomedical diagnosis and research in low-resource settings.

Deep learning achieves super-resolution in fluorescence microscopy

We present a deep learning-based method for achieving super-resolution in fluorescence microscopy. This data-driven approach does not require any numerical models of the imaging process or the estimation of a point spread function, and is solely based on training a generative adversarial network, which statistically learns to transform low resolution input images into super-resolved ones. Using this method, we super-resolve wide-field images acquired with low numerical aperture objective lenses, matching the resolution that is acquired using high numerical aperture objectives. We also demonstrate that diffraction-limited confocal microscopy images can be transformed by the same framework into super-resolved fluorescence images, matching the image resolution acquired with a stimulated emission depletion (STED) microscope. The deep network rapidly outputs these super-resolution images, without any iterations or parameter search, and even works for types of samples that it was not trained for.

A robust holographic autofocusing criterion based on edge sparsity: Comparison of Gini index and Tamura coefficient for holographic autofocusing based on the edge sparsity of the complex optical wavefront

The Sparsity of the Gradient (SoG) is a robust autofocusing criterion for holography, where the gradient modulus of the complex refocused hologram is calculated, on which a sparsity metric is applied. Here, we compare two different choices of sparsity metrics used in SoG, specifically, the Gini index (GI) and the Tamura coefficient (TC), for holographic autofocusing on dense/connected or sparse samples. We provide a theoretical analysis predicting that for uniformly distributed image data, TC and GI exhibit similar behavior, while for naturally sparse images containing few high-valued signal entries and many low-valued noisy background pixels, TC is more sensitive to distribution changes in the signal and more resistive to background noise. These predictions are also confirmed by experimental results using SoG-based holographic autofocusing on dense and connected samples (such as stained breast tissue sections) as well as highly sparse samples (such as isolated Giardia lamblia cysts). Through these experiments, we found that ToG and GoG offer almost identical autofocusing performance on dense and connected samples, whereas for naturally sparse samples, GoG should be calculated on a relatively small region of interest (ROI) closely surrounding the object, while ToG offers more flexibility in choosing a larger ROI containing more background pixels.

3D on-chip microscopy of optically cleared tissue

Traditional pathology relies on tissue biopsy, micro-sectioning, immunohistochemistry and microscopic imaging, which are relatively expensive and labor-intensive, and therefore are less accessible in resource-limited areas. Low-cost tissue clearing techniques, such as the simplified CLARITY method (SCM), are promising to potentially reduce the cost of disease diagnosis by providing 3D imaging and phenotyping of thicker tissue samples with simpler preparation steps. However, the mainstream imaging approach for cleared tissue, fluorescence microscopy, suffers from high-cost, photobleaching and signal fading. As an alternative approach to fluorescence, here we demonstrate 3D imaging of SCMcleared tissue using on-chip holography, which is based on pixel-super-resolution and multi-height phase recovery algorithms to digitally compute the sample’s amplitude and phase images at various z-slices/depths through the sample. The tissue clearing procedures and the lens-free imaging system were jointly optimized to find the best illumination wavelength, tissue thickness, staining solution pH, and the number of hologram heights to maximize the imaged tissue volume, minimize the amount of acquired data, while maintaining a high contrast-to-noise ratio for the imaged cells. After this optimization, we achieved 3D imaging of a 200-μm thick cleared mouse brain tissue over a field-of-view of <20mm2 , and the resulting 3D z-stack agrees well with the images acquired with a scanning lens-based microscope (20× 0.75NA). Moreover, the lens-free microscope achieves an order-of-magnitude better data efficiency compared to its lens-based counterparts for volumetric imaging of samples. The presented low-cost and high-throughput lens-free tissue imaging technique enabled by CLARITY can be used in various biomedical applications in low-resource-settings.

Super-resolution through out-of-focus imaging in lens-based microscopy (Conference Presentation)

The limited space-bandwidth-product of microscopy systems in general forces users to sacrifice either resolution or field-of-view (FOV). Here we introduce a wide-field and high-resolution imaging method that uses a stack of out-of-focus images of the specimen to increase the space-bandwidth-product of lens-based microscopes. Although modern microscope objective-lenses are designed for high-resolution imaging and can achieve a relatively large FOV, often the active area of the imager chip sets a limitation. To best utilize the full field-of-view of an objective-lens in our microscope, we first added a demagnification camera adaptor (e.g., 0.35×) to match the CCD sensor chip active area to the FOV of a 10X objective-lens (~5 mm2) that has an NA of 0.3. This demagnification, while increasing the FOV, downgrades the image resolution and results in pixelation. We illustrate that this spatial undersampling can be overcome through an iterative pixel super-resolution algorithm that uses a stack of out-of-focus images of the sample to restore a high-resolution image across a large FOV. We demonstrated the success of this approach using a resolution test-target and showed that our technique reduces the number of measurements required to achieve the same effective space-bandwidth-product using e.g., lateral scanning and digital stitching of different FOVs. Phase retrieval capability of this approach is also demonstrated by reconstructing unstained Papanicolaou (Pap) smear samples without the need for phase-contrast objective-lenses. This technique might be useful to maximize the throughput of lens-based optical imaging systems and inspire new microscopy designs that utilize auto-focusing steps to increase resolution.

High resolution computational on-chip imaging of biological samples using sparsity constraint (Conference Presentation)

Microscopic imaging of biological samples such as pathology slides is one of the standard diagnostic methods for screening various diseases, including cancer. These biological samples are usually imaged using traditional optical microscopy tools; however, the high cost, bulkiness and limited imaging throughput of traditional microscopes partially restrict their deployment in resource-limited settings. In order to mitigate this, we previously demonstrated a cost-effective and compact lens-less on-chip microscopy platform with a wide field-of-view of >20-30 mm^2. The lens-less microscopy platform has shown its effectiveness for imaging of highly connected biological samples, such as pathology slides of various tissue samples and smears, among others. This computational holographic microscope requires a set of super-resolved holograms acquired at multiple sample-to-sensor distances, which are used as input to an iterative phase recovery algorithm and holographic reconstruction process, yielding high-resolution images of the samples in phase and amplitude channels. Here we demonstrate that in order to reconstruct clinically relevant images with high resolution and image contrast, we require less than 50% of the previously reported nominal number of holograms acquired at different sample-to-sensor distances. This is achieved by incorporating a loose sparsity constraint as part of the iterative holographic object reconstruction. We demonstrate the success of this sparsity-based computational lens-less microscopy platform by imaging pathology slides of breast cancer tissue and Papanicolaou (Pap) smears.