Autofocusing technologies for whole slide imaging and automated microscopy
Zichao Bian, Chengfei Guo, Shaowei Jiang, Jiakai Zhu, Ruihai Wang, Pengming Song, Zibang Zhang, Kazunori Hoshino, Guoan Zheng
11 Autofocusing technologies for whole slide imaging and automated microscopy
Zichao Bian , Chengfei Guo , Shaowei Jiang ,1,* , Jiakai Zhu , Ruihai Wang , Pengming Song , Zibang Zhang , Kazunori Hoshino , and Guoan Zheng University of Connecticut, Department of Biomedical Engineering, Storrs, CT, 06269, USA University of Connecticut, Department of Electrical and Computer Engineering, Storrs, CT, 06269, USA These authors contributed equally to this work * E-mail: [email protected] (S. J.) or [email protected] (G. Z.)
Abstract:
Whole slide imaging (WSI) has moved digital pathology closer to diagnostic practice in recent years. Due to the inherent tissue topography variability, accurate autofocusing remains a critical challenge for WSI and automated microscopy systems. The traditional focus map surveying method is limited in its ability to acquire a high degree of focus points while still maintaining high throughput. Real-time approaches decouple image acquisition from focusing, thus allowing for rapid scanning while maintaining continuous accurate focus. This work reviews the traditional focus map approach and discusses the choice of focus measure for focal plane determination. It also discusses various real-time autofocusing approaches including reflective-based triangulation, confocal pinhole detection, low-coherence interferometry, tilted sensor approach, independent dual sensor scanning, beam splitter array, phase detection, dual-LED illumination, and deep-learning approaches. The technical concepts, merits, and limitations of these methods are explained and compared to those of a traditional WSI system. This review may provide new insights for the development of high-throughput automated microscopy imaging systems that can be made broadly available and utilizable without loss of capacity.
Keywords : whole slide imaging, digital pathology, focus quality, focus map, deep learning.
1. Introduction
The process of analyzing pathology slides using an optical microscope has remained relatively unchanged until recently. In a regular process, pathologists move the microscope stage to different positions to identify areas of interest, which can be further analyzed by switching to a higher magnification objective lens. The focusing of the slide is manually performed using the focus knob of the microscope platform. Although this traditional slide reviewing process remains the gold standard in diagnosing a large number of diseases including almost all types of cancers, it is highly subjective on the other hand: different pathologists may arrive at different conclusions and the same person may also give different conclusions at different time points. In terms of workflow efficiency, this process is labor-intensive and can be easily disrupted when a pathologist bumps a slide to a high magnification objective lens . Similarly, it can be disrupted when the pathologist switches to a different objective lens and performs manual focusing of the slide. After the reviewing process, the slides must be kept accessible, clean and protected, creating additional storage and labor demands . Since the current slide reviewing process is based on subjective opinions of pathologists, there is a need for quantitative and streamlined assessment of histology slides. Quantitative characterization of pathology imagery is not only important for reducing inter- and intra-observer variations in diagnosis but also to better understand the biological mechanisms of the disease process . Recent clinical guidelines have begun to require quantitative evaluations as part of the effort towards better patient risk stratification . For example, breast cancer staging requires the counting of mitotic cells. A whole slide imaging (WSI) system is designed to replace the traditional microscope for quantitative and streamlined slide reviewing. It was first developed based on a robotic microscope platform in the 1990s . The essential components of a WSI system include the following: 1) a microscope with objective lenses, 2) robotics to move slides, 3) one or more image sensors for image acquisition and autofocusing, and 4) software for management. In the acquisition process, a typical WSI system captures hundreds of high-resolution images that are subsequently aligned and stitched together to create a complete and seamless representation of the original whole tissue section . The stitched whole slide image can provide a digital equivalent of the original glass slide on the microscope. The pathologists can then view, navigate, change magnification, and annotate the virtual slide with speed and ease. Digital pathology using WSI is now advancing into clinical workflow for better and faster predication, diagnosis, and prognosis of cancers and other diseases . A major milestone was accomplished in 2017 when the U.S. Food and Drug Administration approved the first WSI scanner for primary diagnostic use in the U.S. . The new generation of pathologists trained on digital pathology promises further growth of the field in the coming decades. Another driving force for the development of digital pathology is the recent advancement of artificial intelligence (AI) in medical diagnosis . In particular, deep-learning approaches have been demonstrated for automated analysis of microscopic pathology images with performance comparable to that by human experts . An augmented reality microscope has also recently been developed to provide real-time integration of AI in the slide inspection process . In this augmented reality microscope platform, two modules are attached to a regular upright microscope. The first module is a digital camera that captures high-resolution images of the same field of view as one observes through the eyepiece ports. The second module is a microdisplay that projects digital information into the eyepiece ports. In a typical implementation, the captured image from the camera will be processed by a deep learning algorithm to produce a heatmap that predicts tumor probability. The outline of the predicted tumor regions will then be projected to the eyepiece ports via the microdisplay. As such, the pathologists can observe the original specimen overlaid with the AI-assisted information through the eyepiece ports. A fundamental challenge with WSI, automated microscopy, and augmented reality microscopy has been the ability to acquire high-quality, in-focus images at high speed. For a high numerical aperture (NA) objective lens, the depth of field is on the orders of 1 µm. The small depth of field poses a difficulty to track the axial topography variations that inherently exist in solid tissue samples . If the specimen is not placed within the depth of field of the objective lens, the image quality of the acquisition will be degraded, leading to rescanning and workflow delays. Several studies have implicated poor focus as the main culprit for poor image quality in WSI . For augmented reality microscopy, defocus blur can occur to the captured images due to the optical path length difference between the eyepiece ports and the camera port. This optical path length difference varies for different objective lenses. As a result, it is challenging to maintain the in-focus position for the camera when the pathologist keeps switching to different objective lenses in the slide reviewing process. Furthermore, some pathologists may have certain vision conditions such as myopia. Instead of adjusting the diopter on the eyepieces, they may prefer to adjust the focus knob to bring the sample into focus for their eye observation. The captured image through the camera port, on the other hand, will be out-of-focus due to the introduced optical path length difference. To address these challenges in augmented reality microscopy, a real-time autofocusing module is needed to acquire high-quality, in-focus images at high speed. Here we review and discuss different autofocusing techniques for WSI and automated microscopy in general. A list of commercially-available WSI scanners and automated microscopy systems are provided in Table 1. The employed autofocusing techniques are listed in the last column and they can be categorized into three groups: 1) pre-scan focus map approach, 2) real-time reflective autofocusing, and 3) real-time image-based autofocusing. In the following, we will first review the traditional pre-scan focus map approach in Section 2. We will discuss the choice of different focus measures for determining the best focal position. In Section 3, we will review the reflective autofocusing approaches, including intensity detection via confocal pinhole, triangulation with oblique illumination, and low-coherence interferometry. In Section 4, we will review and discuss various real-time image-based autofocusing approaches, including tilted sensor, independent dual sensor scanning, beam splitter array, phase detection, dual-LED illumination, and deep-learning approaches. The technical concepts, merits, and limitations of these methods are explained and compared to those of a traditional focus map approach. In Section 5, we will summarize our discussion and provide perspectives for future development. This review may provide new insights for the development of high-throughput automated microscopy systems that can be made broadly available and utilizable without loss of capacity. Vendor Model Imaging mode Slide capacity Scanning speed (15 mm x 15 mm region) Sensor type Autofocusing method
Zeiss Axio Scan.Z1 Brightfield, Fluorescence 12 or 100 slides 20× 240 sec/slide 3 CCD sensor, sCMOS sensor Focus map Olympus VS200 Brightfield, Darkfield, Phase contrast, Polarization, Fluorescence 210 slides 20×: 80 sec/slide Area sensor Focus map Hamamatsu NanoZoomer S360 Brightfield 360 slides 20×: ~30 sec/slide 40×: ~30 sec/slide TDI sensor Focus map Huron TissueScope LE120 Brightfield 120 slides 20×: <60 sec/slide Area sensor Focus map Ventana iScan HT Brightfield 360 slides 20×: <45 sec/slide 40×: <72 sec/slide Information not available
Focus map
Leica Aperio AT2 DX Brightfield 6 or 400 slides 20×: <72 sec/slide TDI sensor Focus map Aperio GT 450 Brightfield 450 slides 40×: 32 sec/slide TDI sensor Tilted sensor 3DHistech Pannoramic 1000 Brightfield 1000 slides 20×:<60 sec/slide 40×:<60 sec/slide Area sensor Focus map Pannoramic 250 Flash III Brightfield, Fluorescence 250 slides 20×: 35 sec/slide 40×: 95 sec/slide 3 CCD sensor, sCMOS sensor Focus map Philips Ultra fast scanner Brightfield 300 slides 40×: 60 sec/slide TDI sensor Tilted sensor Nikon Eclipse Ti2 (Perfect Focus) Brightfield, Phase contrast, Fluorescence 1 slide Not available Area sensor Triangulation with oblique illumination Olympus IXplore (TruFocus) Brightfield, Phase contrast, Fluorescence 1 slide Not available Area sensor Triangulation with oblique illumination Thorlabs EV103 Brightfield, Fluorescence 4 slides 20×: <70 sec/slide 40×:<200sec/slide TDI sensor Low-coherence interferometry Omnyx (now Inspirata) VL120 Brightfield 120 slides 40×: 80 sec/slide 60×: 200 sec/slide Area sensor Independent dual sensor scanning
Table 1.
A list of commercially-available WSI scanners and automated microscopy systems. Note: every attempt was made to include accurate data in this table at the time of writing this article. The autofocusing methods were best estimated based on the product instruction manuals and related patents.
2. Focus map surveying
Focus map surveying is the most adopted autofocusing method in commercially available WSI systems. Manufactories are in favor of using this approach because of two main reasons: 1) it requires no additional optical hardware and is robust for different types of samples, and 2) no or less intellectual property issue. Here we will first discuss the choice of focus measure in Section 2.1. We will then discuss focus map generation and focus quality control in Section 2.2.
The principle of this method is shown in Figure 1, where the camera is used to acquire z-stack images of the specimen when the sample or the objective lens is axially scanned to different positions. From the resulting z-stack, a certain figure of merit of each image, such as image contrast, entropy, spatial frequency content, is extracted for measuring the quality of focus. It is also common to acquiring images while calculating the figure of merit, and choosing the image corresponding to the peak (or valley) of the figure of merit, or by performing a search to optimize the figure of merit. By repeating this searching process for different tiles of the microscope slide, the well-focused digital whole slide image can be obtained.
Figure 1.
The traditional axial scanning procedure for autofocusing. For a selected region of interest, a z-stack is acquired and used to determine the focus position using a certain figure of merit.
An important aspect of this approach is to choose a proper figure of merit to measure the quality of focus. When the specimen is in focus, the captured image should demonstrate large image contrast, a large range of intensity values, and sharp edges. Quantitatively, a good figure of merit should be acutely sensitive to focus, monotonically decreasing and symmetric about the peak, and contains no prominent local maxima outside of the peak, as shown in Figure 1. Accuracy is clearly of utmost importance. In the case of WSI and automated microscopy, minimizing the computation time is also critical. Several previous studies have evaluated and compared a list of common focus measures . Table 2 lists a dozen common focus measures that are intuitive and computationally simple. In general, they can be categorized into 4 groups : (1) derivative-based measures such as Brenner gradient, Tenenbaum gradient, energy Laplace, Gaussian derivative, sum of wavelet coefficients, ratio of wavelet coefficients, power-weighted average, and power log-log slope, (2) statistical-based measures such as image contrast, normalized variance, auto-correlation, and standard deviation-based correlation, (3) histogram-based measures such as histogram range, histogram entropy, and weight histogram sum, and (4) intuitive-based measures such as thresholded content. Focus measure
Equation
Comments
Brenner gradient = ∑ ∑ ( + 2, ) − ( , ) , where ( , ) is the captured 2D intensity image. High autofocusing accuracy for different samples . Tenenbaum gradient = ∑ ∑ ( ( , ) + ( , ) ) , where ( , ) and ( , ) are the resultant images by convoluting ( , ) with the kernels [-1 0 1; -2 0 2; -1 0 1] and [1 2 1;0 0 0;-1 2 -1], respectively. Well performed for subsampled images and robust to random noises . Energy Laplace _ = ∑ ∑ [ ( − 1, ) + ( + 1, ) + ( , − 1) + ( , + 1) + 4 ( − 1, )] Well performed for tuberculosis detection . Gaussian derivative = ∙ ∑ ∑ [ ( , ) ∗ ( , , )] + ( , ) ∗ ( , , ) , where and are the first-order Gaussian derivatives in x- and y-direction at scale . Robust against noises with a proper selection of parameter 33 . Sum of wavelet coefficients _ = ∑ | ( , )| + | ( , )| + | ( , )| , where is the corresponding window in the DWT sub-regions. , and are the level-1 two-dimension DWT sub-regions. A common derivative-based focus measure . Ratio of wavelet coefficients _ = ⁄ , = ∑ ∑ ( , ) + ( , ) + ( , ) , = ∑ ( , ) , where is the Kth level DWT low-frequency sub-region. , and are the level-n two-dimension DWT sub-regions. Well performed for common microscopic images . Power-weighted average ( ) = ∑ ∑ [ ( , ) ∗ ( , )] ∑ ∑ ( , ) and _ = ∑ ( ) ∑ ( ) ⁄ , where ( , ) is high-pass or band-pass filter, ∗ stands for the convolution operator, ( , ) is the grey level intensity of pixel (x, y) at z position. m is an integer chosen by the user for different applications. Well performed for phase-contrast autofocusing . Power log-log slope is the log-log slope of the one-dimensional power spectral density of image I , where = log( ( ( )) ) and FT denotes as Fourier transform. Well performed for focus quality control in high-content screening . Image contrast = ( − ) ( + )⁄ , where and are the maximum and minimum grey level intensity, respectively. A common statistical-based focus measure . Normalized variance _ = 1 ( ∙ ∙ )⁄ ∑ ∑ ( ( , ) − ) , where is the mean gray level of the image. Well performed for blood smear and pap smear autofocusing . Auto-correlation = ∑ ∑ ( , ) ∙ ( + 1, ) − ∑ ∑ ( , ) ∙ ( + 2, ) Well performed for fluorescence microscopy . Standard deviation-based correlation _ = ∑ ∑ ( , ) ∙ ( + 1, ) − · ∙ Robust to noises . Histogram range = max (ℎ( ) > 0) − m (ℎ( ) > 0) , where ℎ( ) is image histograms (i.e., the number of pixels with intensity in an image). Performance depends on samples and imaging methods . Histogram entropy = − ∑ · log ( ) , where = ℎ( ) ( ∙ )⁄ is the probability of a pixel with intensity . Well performed for sinusoidal and binary images . Weight histogram sum = ∑ ℎ( ) ∙ ( , ) ∙ 10 , where the fifth root and fifth potency are empirical results. Well performed for fluorescence bacterial samples . Thresholded content _ = ∑ ∑ ( , ) , where ( , ) ≥ . is the threshold Fast computation; a good choice for the coarse searching . Table 2.
Common figure of merits for measuring the focus quality.
With a chosen focus measure for certain applications, the next step is to estimate the focus position using the calculated focus measure from the acquired images. A fitted function can be used to find the peak (or valley) from the figure of merit data points, obviating the need to acquire images near the focus. The choice of curve fitting model directly affects the number of images needed. Typical fitting models include polynomial , Lorentzian , and Gaussian models . A polynomial fit may closely approximate the figure of merit data points that are close to the focal plane. An n th -order function, however, requires a minimum of n +1 images to be acquired, thus drastically increasing image acquisition time when a higher-order fitting curve is employed. It may also fail if the focus plane is substantially outside of the depth of field. Yazdanfar et al. have demonstrated a Lorentzian function for fitting the Brenner gradient focus measure . Using this empirical model, only 3 images are needed to determine the focal plane. Similarly, Gaussian fitting model with 3 unknown parameters has been demonstrated for fluorescence microscopy with an electrically tunable lens . The choice of fitting model is an important topic for each of the chosen focus measure and the related microscopy applications. Further research in this direction is highly desired. The focus measures listed in Table 2 are mainly designed for incoherent microscopy with intensity-only measurements. Another important property of light wave is phase, which characterizes the optical delay accrued during propagation. Light detectors such as image sensors and photographic plates can only measure intensity variation of the light waves. Phase information is lost during the recording process. Consequently, phase measurement often involves additional experimental complexity, typically by requiring light interference with a known field , or via a phase retrieval process where the complex amplitude is recovered from intensity measurements . Coherent microscopy uses both intensity and phase as the focus measure. The autofocusing process can be performed after the data has been acquired. As one example, Fourier ptychography is a coherent microscopy technique that has been demonstrated for WSI . Unlike in conventional microscopy where resolution and imaging field of view need to be traded off against each other, Fourier ptychography can achieve both high resolution and wide field of view via a low-NA objective lens and angle-varied illumination. Regular WSI platform stitches the captured intensity images in the spatial domain to expand the field of view. Fourier ptychography, on the other hand, stitches the information in the Fourier domain to expand the spatial frequency bandwidth. Autofocusing of Fourier ptychography is performed in the ptychographic phase retrieval process , where the defocus pupil aberration can be jointly recovered with the complex object . At the end of the reconstruction, the synthesized information in the Fourier domain generates a high-resolution, complex-valued object image that retains the original large field of view set by the low-NA objective lens. Similar coherent imaging procedures can also be performed at the detection path via aperture or diffuser modulation . In this case, the recovered complex wavefront can be digitally propagated to any plane along the optical axis after reconstruction. A focus measure with both intensity and phase can be used to determine the best focal plane of the object . A detailed discussion on coherent microscopy and the related focus measures are beyond the scope of this review article. In the following, we focus our discussions on regular incoherent microscopy. By repeating the z-stack autofocusing process for every tile, it is straightforward to generate a high-resolution, well-focused whole slide image of the specimen. However, as indicated above, the autofocusing process can take a significant amount of time to acquire z-stacks at multiple positions. Assuming a rate of 20 frames per second to acquire images, surveying focus at 5 different focal positions would take 0.25 seconds per tile. As a result, an image with 500 tiles can take as much as 150 seconds to acquire, not including the deceleration, acceleration, settling time for moving the slide to different lateral and axial positions. Therefore, it is not a feasible solution to perform autofocusing on every tile using the traditional image-based focus measure approach. To address the time burden, many WSI systems create a focus map prior to scanning, or survey focus points every n tiles or lines, in effect skipping areas to save time . The number and the locations of the focus points are often made user selectable. Figure 2(a) shows the procedures of the focus map surveying approach. The system will first select focus points based on the sample’s feature and distribute them evenly over the entire slide. Each focus point is triangulated to create a focus map of the tissue surface, in effect filling in the blanks. Delaunay triangulation is a typical method for generating the focus map . As shown in Figure 2(b), line scanners typically achieve better autofocusing performance than traditional 2D area sensors because linear sensors can change focus at a shorter interval. Regular 1D and 2D image sensors need to have high illumination intensity to quickly register light levels before the sample motion causes smearing of the image. Time delay integration (TDI) sensor overcomes this illumination limitation by having multiple rows of elements that each shift their partial measurements to the adjacent row synchronously with the motion of the image across the array of elements . TDI sensors are often the choice of low-light applications such as fluorescence microscopy with low photon budgets. The disadvantage of TDI sensor is the requirement of precisely synchronized sample scanning for generating an image. Rescan of the sample is needed for imaging multiple depths or fluorescence channels. Precise co-localization of different depths or different fluorescence colors can be a challenge for post-acquisition processing. The use of TDI sensors also lacks the imaging flexibility for research microscopy in general. An alternative approach to generating the focus map is to perform autofocusing in every n tiles, termed ‘skipping tiles’ in Figure 2(b). In this case, it assumes the focused tile shares the same focus position with its adjacent tiles. The focusing performance is, however, worse than the focus map approach as it may contain more out-of-focus regions as shown in Figure 2(b). The skipping tiles approach, on the other hand, does not need to travel back to a certain axial position with sub-micron accuracy. The requirement of motion repeatability is not as stringent as that in the focus map approach. Nevertheless, more focus points can increase the accuracy of the overall focusing performance for both approaches, at the expense of additional time for autofocusing. Figure 2. (a) Focus map generation procedures. The green bars represent the calculated figure of merits at different focus points. The red bars represent the interpolated focus points. (b) Comparison between the focus map approach and the skipping tiles approach. The green crosshairs represent the focus points used to calculate the focus map. The blue dashed lines represent the focus positions interpolated between the selected focus points. Red boxes represent the focal plane for each field of view using a 2D image sensor or a 1D linear sensor. Each red box can be adjusted in the z-axis during a scan. Modified from Ref. . After the high-resolution specimen images are acquired, it is often necessary to review the images for focus quality control and determine whether certain regions need to be re-scanned. Similarly, in high-content screening for drug discovery and genome analysis, it is important to identify out-of-focus images for obtaining a clean, unbiased image dataset. Complicating this task is the fact that one only has a single-z-depth image instead of a z-stack for analysis. An absolute measure of image focus on a single image in isolation, without other user-specified parameters, is needed in this case. In the past years, various approaches have been demonstrated for no-reference focus quality assessment, including gradient map , contrast map , phase coherency , cumulative probability of blur detection , visual system’s equalization of spatial frequency , among others. Jimenez et al. have tested several quality assessment metrics on a database of pathology slides and reported that the cumulative probability of blur detection is most effective among the 6 tested metrics . Another emerging direction for focus quality control is to convert the image assessment process into a classification task using a neural network . For example, Senaras et al. reported a ‘DeepFocus’ network to identify out-of-focus regions in histopathological images . Discussions of deep-learning approaches will be given in Section 4.6.
3. Reflective-based autofocusing
Reflective-based autofocusing aims to detect the axial location of a reference plane, which is usually the interface between glass and liquid where the cells residue or the air-glass interface at the bottom of the cell culture vessels. During experiments, the focus drift correction system will repetitively find the axial location of the reference plane and maintain a constant distance between the objective lens and the reference plane through a motorized axial driver. In Section 3.1, we will discuss a confocal pinhole approach to locate the interfaces. In Section 3.2, we will discuss how to use the reflective light displacement to locate the reference plane in real-time. In Section 3.3, we will discuss a low-coherence interferometry approach to locate the sample switched by two interfaces in real-time. Liron et al. reported a laser reflective autofocusing approach using confocal pinhole detection in 2006 . The optical setup is shown in Figure 3, where a laser beam is expanded and focused onto the substrate of the sample (highlighted in red). The reflective light from the substrate passes through a confocal pinhole and reaches the photodetector (highlighted in yellow). The fraction of laser intensity reflected at an interface is roughly proportional to the square of the refractive index difference. For biological specimens located in water (or aqueous buffers) above a glass / plastic plate, the reflection from the glass-air interface is about 4% of the incident beam and the reflection from the glass-water interface is only 0.4%. The inset of Figure 3 shows a measured intensity curve by axially scanning the objective lens to different positions. The first strong peak corresponds to the air-glass interface and the second weaker peak corresponds to the sample-glass interface. Solid and dashed lines are results for 100-µm and 200-µm pinhole. Increasing the confocal pinhole size can broaden the width of the peaks as indicated by the dashed line in Figure 3. This adjustment can reduce some unwanted interference speckles and facilitate the data analysis process. A two-stage operation was employed to perform the autofocusing process. The first stage, termed ‘long peak detection search’, is to locate the strong peak via high-speed axial scanning of the objective lens. With the location of the first strong peak, the position of the second peak can be estimated by adding the thickness of the glass substrate. The second stage, termed ‘local peak search’, performs precise peak search over a relatively short range. While this confocal detection approach can perform precise autofocusing, its main drawback is the requirement of axial scanning to get the trace curve shown in Figure 3. Another drawback is the orders of magnitude difference in strength for the two peaks. The weaker peak can easily be overwhelmed by the first strong peak, especially for lower magnification objective lenses. In Section 3.2, we will discuss a strategy to address the first drawback, i.e., to locate the first peak position without performing axial scanning. In Section 3.3, we will discuss another strategy to address both drawbacks, i.e., to reduce the signal strength from the first peak and to locate both peaks without axial scanning. Figure 3.
An autofocusing system using confocal pinhole detection. Laser light is expanded and focused on the substrate of the sample. The reflective light, highlighted in yellow, is passed through a confocal pinhole and detected by the photodetector. Inset in the top right shows the measured intensity signals by axially scanning the objective lens to different positions. The first strong peak corresponds to the air-glass interface and the second weaker peak corresponds to the sample-glass interface. Solid and dashed lines are results for 100 µm and 200 µm pinhole. Modified from Ref. . To locate the axial position of an interface without axial scanning, one can illuminate the sample with a tilted incident angle and measure the lateral displacement of the reflected beam. The triangulation concept for microscopy autofocusing can be dated back to the patent by Reinheimer in 1973 . In this patent, Reinheimer proposed to restrict a shaped illumination beam to occupy only half of the pupil aperture cross-section. As such, the beam reflected from a surface will have different lateral displacements when the sample surface is placed at different axial positions. The reflected light from the sample surface is detected by two photoelectric transducers for differential measurement. The differential signal detected by these two transducers is used to drive the focus knob. For example, if the sample surface is placed at the in-focus position, the reflected light will be directed to the boundary of the two transducers. The resulting differential signal is 0 and no adjustment is needed. If the sample surface is positioned above the in-focus plane, the reflected light will shift to one of the transducers. The differential signal is then used to drive down the sample stage. Similarly, if the sample surface is positioned below the in-focus plane, the differential signal from the two transducers drives up the sample stage. There are some further refinements and developments of this original patent in the 1980s and 1990s . These developments are, in general, about how to better detect the beam size and the positional shift to infer the defocus distance. Similar schemes have also been reported in more recent literatures . Figure 4.
The Nikon Perfect Focus System. Light from an infrared LED is shaped by a line aperture and a half-moon mask for illuminating the sample substrate at a tilted angle. The reflected light is detected by a linear CCD. Inset shows the detected line traces when the sample substrate is scanned to different defocused positions. Two offset adjustment lenses are used to maintain the focus at the desired positional offset from the coverslip surface. Modified from Ref. . Figure 4 shows the adoption of the triangulation idea in a modern microscope system, marketed as Nikon Perfect Focus System (PFS) . This system maintains focus by detecting and tracking the position of the coverslip surface in real-time. It employs a near-infrared 870-nm LED as the light source and a linear CCD sensor as the detector (other detectors such as four-quant photodiode and area sensor can also be used here). Predefined by the user is an offset between the reference plane and the axial location of the desired focused image. Different from the original patent by Reinheimer, the PFS system introduces two offset adjustment lenses in Figure 4 to maintain the focus at the desired positional offset from the coverslip surface. When the user changes the offset distance, the distance of the two offset adjustment lenses changes, resulting in a shift of the line position detected by linear CCD (inset of Figure 4). The positional shift generates a signal to move the objective lens along the axial direction until the line position is centered at the linear CCD again. The PFS system is mainly designed to image living cells housed in imaging chambers equipped with a standard coverslip. For fixed specimens like tissue slides that are mounted in a high refractive index medium (which closely matches that of the coverslip), the refraction index difference may not generate sufficient signal to detect the interface surface. Likewise, tissue slides with strong absorption profiles often scatter a considerable amount of light, leading to excessive hunting or errors in the focus drift correction system. Plastic tissue culture dishes are also not recommended, as the boundary surface may not be detectable due to insufficient offset . The idea of using optical coherence tomography (OCT) for autofocusing was proposed in a patent by Wei and Hellmuth in 1996 . The general concept is to locate the sample position using the axial depth reflectivity profile called A-scan, which contains scattering information of sample structures along the axial direction. In the original patent, an on-axis configuration is used to perform autofocusing of an ophthalmologic surgical microscope. However, it is not suitable for high-resolution imaging of tissue slides covered by glass. The main difficulty is the overlap between the large signal reflected by glass surfaces and the weak signal reflected from the sample. Locating the sample position with submicron accuracy is challenging given the large signals reflected from the glass surfaces.
Figure 5.
Low-coherence interferometry for reflective real-time autofocusing. A superluminescent diode is used as a low-coherence light source. The light illuminates the sample from a tilted incident angle. As such, most reflected light from the glass surface will not be coupled back to the interferometry system. The axial depth reflectivity profile (i.e., A-scan) is measured using a spectrometer. The recovered sample position is used to move the z stage or the objective lens. Adapted from Ref. . One solution to this problem is to substantially reduce the light reflected from glass surfaces while keeping the sample scattering light relatively unchanged. Figure 5 demonstrates such a solution by using an off-axis configuration, where the light illuminates the sample at a tilted incident angle . As such, the light directly reflected from the glass surface will not be coupled back to the interferometry system. In Figure 5, a broadband superluminescent diode is used as the low-coherence light source. The axial depth reflectivity profile (i.e., A-scan) is measured using a spectrometer in a Fourier-domain OCT setup. The sample position can be calculated by performing a Fourier transform of the captured spectrum and used to move the objective lens to the in-focus position. Since OCT is sensitive to refraction index variations within the sample, this approach can handle transparent samples that may be challenging for the traditional focus map approach. The disadvantages, perhaps, are the complicated Fourier-domain OCT setup, the precise optical alignment, and the high maintenance of the system.
4. Real-time image-based autofocusing
The pre-scan focus map approach requires the acquisition of a z-stack for each focus point. The sample needs to be scanned to different x-y positions for acquiring multiple z-stacks to generate the focus map. In many WSI systems, the overhead time for generating the focus map is a substantial portion of the total scanning time. In this section, we will discuss several real-time image-based autofocusing approaches without the need for generating the focus map.
The traditional focus map approach uses the same image sensor to both survey the focus and acquire the image. In between two image acquisitions, there is a certain amount of ‘dead time’ to read out the data to the memory. As a result, the main camera cannot be used to survey the focus during this ‘dead time’. An independent secondary image sensor has been proposed to survey the focus in parallel . Figure 6 shows the principle and operation of this concept. In Figure 6(a), an independent camera, termed focusing sensor, is used to survey the focus while the main camera captures the high-resolution sample images. During the scanning process, the stage is in continuous motion and the motion blur is eliminated by using short pulses of light during imaging. As shown in Figure 6(b), the focusing sensor acquires three autofocus images, each at a slightly different focal plane. Based on these three images, the system calculates the optimal focus position and moves the sample to that focal plane , where the main camera takes a high-resolution image. When the main camera is reading out image data, the autofocusing is repeated for the next tile position to predict its optimal focal plane ahead. Since the stage is in continuous motion during this process, the captured three focus images only share a small region of overlap (Figure 6(c)). Only the overlapping region is used to calculate the correct focal position. The autofocusing performance of this system has been validated with various tissue sections . The average focusing error is ~0.30 µm for the continuous motion scheme. Around 95% of tiles fall within the system’s depth of field. Figure 6.
Independent dual sensor scanning for real-time image-based autofocusing. (a) The optical scheme, where a high-speed focusing camera is used to survey the focus in parallel with the main camera. (b) The focusing sensor acquires three autofocus images, each at a slightly different focal plane. The system calculates the optimal focus position and moves the sample to that focal plane, where the main camera takes a high-resolution image. (c) The stage is in continuous motion during this process and the captured three images only share a small region of overlap. Modified from Ref. . In the independent dual sensor scanning scheme discussed above, multiple images are acquired to calculate the focus position when the sample is moved to different focal planes. In a patent published in 2010, Virag et al. proposed to use a beam splitter array to allow capturing images at different focal planes on the same image sensor . Figure 7 shows the imaging setup, where the focusing optics comprises a main imaging camera and a secondary focusing camera. A beam splitter array is used to split and direct the light beam to different regions of the focusing sensor. As such, the system can capture images at multiple focal planes at the same time. The 45-degree semi-reflective surfaces in the beam splitter array are chosen to assure that all beams reflected by the surfaces have roughly the same intensities. With the image captured by the focusing sensor, a certain focus measure and fitting model can be used to infer the optimal focus position. In additional to autofocusing, this scheme can also be modified for real-time multiplane microscopy , which finds important applications in volumetric imaging of biological samples. Figure 7.
Beam splitter array for real-time image-based autofocusing. A beam splitter array is used to split and direct the light beam to different regions on the focusing sensor. As such, the system can capture images at multiple focal planes for determining the optimum focus position. Modified from Ref. . The tilted sensor approach uses a tilted focusing sensor to image an oblique cross-section of the sample. The optimum focus position can be inferred by locating the peak of the contrast curve in real time. The concept of this approach was originally proposed in a patent by Dong et al. in 2005 . There are some further refinements and developments of this original concept by Philips and Leica . Arguably, it is one of the most successful autofocusing technologies employed in existing commercially available WSI systems. Figure 8 shows the principle and operation of the tilted sensor concept. In Figure 8(a), the focusing sensor is tilted at θ angle with respect to the parfocal image plane. The imaging and focusing sensors can be either 2D area sensors or 1D linear sensors. The overlapping position between the focusing sensor and the parfocal imaging plane is termed ‘parfocal point’ in Figure 8(b). The focusing range is determined by Z range . With a larger tilted angle, a longer focusing range can be expected. During the scanning process, both sensors capture images of the sample. For each pixel of the captured data, a contrast value can be determined based on the surrounding pixel values. Consider a 1D image data ( ) as an example, the contrast value ( ) can be calculated via ( ) =∑ | ( ) − ( − )| , where m define the surrounding range for the calculation. A contrast curve can then be obtained by dividing the focusing sensor contrast value by the imaging sensor contrast value , as shown in Figure 8(c). The peak of the contrast curve determines the pixel having the highest contrast value, i.e., the best focal position. The parfocal point can also be plotted on the contrast curve. In Figure 8(c), the pixel distance Δ N between the parfocal point and the peak contrast point on the curve indicates a physical distance along the axial direction. This distance represents the distance between the current position of the objective lens and the optimal focus position of the objective lens, i.e., one needs to axially move the objective lens by this distance for best focusing. While the imaging sensor is centered at the field of view of the objective lens, the focusing sensor can be shifted away from the center of the field of view. As such, the focusing sensor ‘sees’ the image data before the imaging sensor ‘sees’ the same region. Similarly, a ‘volume camera’ consisted of multiple linear CCDs coupled with fibers can be arranged with a tilted angle for autofocusing . Bravo et al reported the use of 9 sensors coupled with fibers to acquire images at different focal planes for real-time image-based autofocusing . Figure 8.
Tilted sensor for real-time image-based autofocusing. (a) The optical scheme, where a tilted sensor is used to infer the optimum focus position during the scanning process. (b) The overlapping position between the focusing sensor and the parfocal imaging plane is termed ‘parfocal point’. (c) Contrast curve for determining the optimal focus position. The pixel distance (Δ N ) between the parfocal point and the peak contrast point indicates a physical distance by which one needs to adjust the objective lens for optimal focusing. Modified from Ref. . Phase detection autofocusing has been used in most digital single-lens reflex cameras (DLSR) . It is typically achieved by dividing the incoming light into pairs of images. It then measures the distance between the two images and infers the defocus amount. The ‘phase’ here is referred to the translational shift between the two images (or the phase shift in the Fourier domain). Inspired by the phase detection concept in photography, we have developed an autofocusing add-on kit to perform WSI using a regular microscope . As shown in Figure 9(a), two pinhole-modulated cameras are attached to the eyepiece for phase detection autofocusing. By adjusting the positions of the pinholes, one can effectively change the view angles through the two eyepiece ports. If the sample is placed at the in-focus position, the two captured images will be identical. If the sample is placed at an out-of-focus position, the sample will be projected at two different view angles, causing a translational shift in the two captured images. The translational shift is proportional to the defocus distance of the sample. Therefore, by identifying the translational shift of the two captured images via phase correlation, the optimal focal position of the sample can be recovered without a z-scan. Figure 9.
Phase detection for real-time image-based autofocusing. (a) Two pinhole-modulated cameras are attached to the eyepiece ports for phase detection autofocusing. If the sample is placed at an out-of-focus position, it will be imaged at two different view angles, causing a translational shift in the two captured images through the eyepiece ports. Modified from Ref. . (b) A dual-pinhole mask is placed at the pupil plane for light modulation. The captured image from the focusing sensor contains two copies of the object and the defocus distance can be recovered based on the translational shift between the two copies. Modified from Ref. . (c) A wedge plate is inserted into the pupil plane to direct half of the beam to a slightly tilted angle. As such, the captured image from the focusing sensor contains two copies of the sample separated by a certain distance. Similarly, the defocus distance can be recovered from the translational shift of the two copies. Modified from Ref. . Figure 9(b) shows another autofocusing configuration using the phase detection concept . A dual-pinhole mask is placed at the pupil plane to modulate the light from the sample. Instead of using two pinhole-modulated cameras, only one focusing sensor is used to capture the image modulated by the dual-pinhole mask. In this case, the captured image from the focusing sensor contains two copies of the sample and the translational shift of these two copies is proportional to the defocus distance. Inset of Figure 9(b) shows a raw image captured by the focusing sensor, where two copies of the sample can be seen from this image. The distance between the two copies can be recovered via autocorrelation analysis shown in Figure 9(b). Figure 9(c) shows a similar phase detection scheme by Silvestri et al. . Same as the dual-pinhole modulation approach, only one camera is used for the focusing purpose. A wedge plate is inserted into the pupil plane to direct half of the beam to a slightly tilted angle. As such, the captured image from the focusing sensor contains two copies of the sample separated by a certain distance. The defocus distance can be recovered from the translational shift of the two copies. For the configurations shown in Figure 9(a) and 9(b), pinhole masks are used to restrict the light at the pupil plane. Therefore, they have relatively long autofocusing ranges. The system in Figure 9(c), on the other hand, has a short autofocusing range. Using the dual-pinhole mask does not prevent its applications in fluorescence microscopy. One can choose a beam splitter cube to direct the strong excitation light through the dual-pinhole mask. Weak fluorescence emissions from the sample can be directed to the imaging camera. The configuration in Figure 9(b) has been demonstrated for fluorescence WSI . Dual-LED illumination has recently been demonstrated for single-frame autofocusing while the sample is in continuous motion . Figure 10(a) shows one of the reported configurations where two near-infrared LEDs are placed at the back focal plane of the condenser lens for sample illumination . These two LEDs illuminate the sample from two different incident angles and they can be treated as spatially coherent light sources. A hot mirror is used to direct the near-infrared light to the focusing sensor shown in Figure 10(a). As such, the captured image from the focusing sensor contains two copies of the sample separated by a certain distance. In particular, the focusing sensor is placed at a preset offset distance with respect to the imaging sensor. When the sample is placed at the in-focus position, the captured image from the focusing sensor contains two copies of the sample profile. Similar to the dual-pinhole mask approach, one can recover the defocus distance by identifying the separation of the two copies through autocorrelation analysis. The preset offset arrangement in Figure 10(a) is used to improve the accuracy of autocorrelation analysis when the defocus distance is small. It can also generate out-of-focus contrast for transparent specimens. If the sample motion direction is perpendicular to the direction of the translational shift, the autofocusing process can be implemented even with continuous sample motion. This dual-LED scheme has also been demonstrated for focus map surveying with only one main camera . Figure 10(b1) shows a further development of the dual-LED approach using color multiplexed illumination . In this scheme, a color LED array is used for sample illumination. For regular brightfield image acquisition, all LED elements are turned on as shown in the left part of Figure 10(b1). In between two brightfield acquisitions, a red and a green LED are turned on for color-multiplexed illumination. If the sample is placed at an out-of-focus position, the red and the green copy will be separated by a certain distance, as shown in the insets of Figure 10(b1). One can then identify the translational shift of the red- and green-channel images by maximizing the image mutual information or cross-correlation . The resulting translational shift is used for dynamic focus correction in the scanning process. Figure 10(b2) shows an open-source WSI platform, termed OpenWSI, based on the color-multiplexed dual-LED autofocusing scheme . This OpenWSI platform is built with low-cost, off-the-shelf components including a programmable LED array, a photographic lens, and a computer numerical control (CNC) router. Coarse axial adjustment is performed using the CNC router and the precise adjustment is performed using the ultrasonic motor ring within the photographic lens. The system has a resolution of ~0.7 µm using a 20X objective lens. It can acquire a whole slide image of 225 mm region in ~2 mins. Since a programable LED array is used for sample illumination in this system, it can also be used for quantitative phasing imaging via Fourier ptychography. Figure 10.
Dual-LED illumination for single-frame autofocusing. (a) Two near-infrared LEDs are placed at the back focal plane of the condenser lens for illuminating the sample from two different angles. A hot mirror is used to direct the near-infrared light to the focusing sensor with a preset offset. The defocus distance is related to the separation of the two-copy image captured by the focusing sensor. (b1) Color-multiplexed dual-LED illumination for single-frame autofocusing. A red and a green LED are turned on for generating a red and green copy on the color image sensor. (b2) OpenWSI system based on the color-multiplexed dual-LED autofocusing scheme. Modified from Ref. . Deep learning has been demonstrated as a powerful tool for solving inverse problems. With the advent of accelerated computing and deep learning frameworks such as TensorFlow and PyTorch, researchers have also explored various deep learning-based solutions for autofocusing . As shown in Figure 11, the reported deep-learning solutions can be, in general, categorized into two groups. Figure 11.
Deep learning approaches for autofocusing. (a) A neural network is trained to output the defocus distance from an input defocused image. (b) A neural network is trained to output an in-focus image based on the input defocused image.
The first group is to predict the defocus distance or to locate the out-of-focus regions based on one or more input defocused images . For example, Jiang et al . employed a convolutional neural network (CNN) to estimate the defocus distance based on the transform- and multi-domain inputs . By adding the Fourier spectrum and the autocorrelation of the spatial image as the input, the performance and the robustness can be improved compared to that only with the spatial image as the input. Dastidar et al . further improved the performance by using the difference of two defocused images as the input of the CNN . Shajkofci et al. reported the use of a CNN-based sharpness function as the focus measure for three-shot autofocusing . Pinkard et al . designed a fully connected Fourier neural network with the additional off-axis LEDs as the illumination source to predict the defocus distance . Yang et al . and Kohlberger et al . have developed networks to quantify and localize the out-of-focus regions in WSI. The severity of the out-of-focus regions is treated as a classification problem with 30 classes . The second group of developments is to output an in-focus image based on an input defocused image . The network is, essentially, to perform blind deconvolution. Typically network architectures include U-net and conditional generative adversarial network (cGAN) . For example, Wu et al . have employed a cGAN to virtually refocus a two-dimensional fluorescence image onto user-defined three-dimensional (3D) surfaces by appending a pre-defined digital propagation matrix . It has also been shown that a blurry microscopy image acquired at an arbitrary out-of-focus plane can be virtually refocused to the in-focus position .
5. Summary and discussion
High-content images are desired in many fields of biomedical research as well as in clinical applications. Accurate and high-speed autofocusing remains a challenge for WSI and automated microscopy. This work has reviewed and discussed various autofocusing techniques from existing patents and journal papers. The technical concepts, merits, and limitations of these methods are explained and discussed. Autofocusing approach
Advantages
Disadvantages
Focus map ▪ No or less intellectual property issue ▪ Require no additional optical hardware ▪ Can be used for different imaging modalities ▪ Robust and widely adopted for WSI ▪ Require a z-stack for each focus point ▪ Mechanical repeatability is critical for sample positioning ▪ Challenging to handle transparent specimens Confocal pinhole ▪ High accuracy for locating the air-glass interface ▪ Require additional confocal optics ▪ Time-consuming for z-scan ▪ Reflection from other interfaces can be overwhelmed by the strong signal from the air-glass interface Triangulation with oblique illumination ▪ High accuracy for locating the air-glass surface of a standard coverslip ▪ Real-time autofocusing ▪ Require additional illumination and detection optics ▪ Only work for living cells housed in imaging chambers with a standard coverslip. Cannot work for microscope slides or thick plastic dish. Low-coherence interferometry ▪ Can handle transparent specimens ▪ Real-time autofocusing ▪ Expensive and complicated Fourier-domain OCT setup ▪ Precise optical alignment needed Independent dual sensor scanning ▪ Real-time image-based autofocusing during continuous sample motion ▪ Effectively avoid the ‘dead time’ of camera readout ▪ Require a secondary area camera and pulsed illumination ▪ Require the acquisition of three images for autofocusing with a small overlapping portion ▪ Relatively short autofocusing range Beam splitter array ▪ Real-time image-based autofocusing ▪ Require a secondary area camera ▪ Relatively short autofocusing range Tilted sensor ▪ Real-time image-based autofocusing ▪ Fully compatible with linear and TDI image sensor ▪ Fast calculation via contrast curve ▪ One of the most successful techniques deployed in commercially available WSI systems ▪ Require a secondary focusing sensor ▪ A transparent sample may lead to a wrong autofocusing calculation since out-of-focus regions have a higher contrast Phase detection ▪ Real-time image-based autofocusing ▪ Can handle transparent specimens via a preset offset of the focusing sensor ▪ Require additional camera(s) and relay optics for the pinhole mask ▪ Precise alignment needed for the pinhole mask ▪ Low-pass filtering of the pinhole mask may affect the accuracy of the correlation analysis Dual-LED illumination ▪ Real-time image-based autofocusing ▪ Can be implemented with continuous sample motion ▪ Can handle transparent specimens ▪ Relatively long autofocusing range due to the use of partially coherent dual-LED illumination ▪ Cost effective and compatible with most automated microscope platforms ▪ Only work for regular 2D thin slides Deep learning ▪ Allow single-shot autofocusing ▪ Require no additional optical hardware ▪ Relatively short virtual refocusing range ▪ Change of optical hardware may affect the autofocusing performance ▪ The system may fail for new features or new types of specimens that have not been trained before
Table 3.
Summary and comparison of different autofocusing techniques. We summarize the advantages and disadvantages of these techniques in Table 3. Among these techniques, the focus map approach is the most adopted technique in existing WSI systems due to its simplicity and the absence of intellectual property issues. The tilted sensor approach is another very successful technique employed in current Leica and Philips WSI systems. The recent dual-LED approach provides a cost-effective solution to develop WSI systems that can be made broadly available and utilizable without loss of capacity. The deep learning approach, on the other hand, is an emerging direction for tackling autofocusing problems without hardware modification. Further work is desired for improving its robustness and the generalization capability of handling different types of specimens. Some of the autofocusing techniques discussed here can also be employed in the augmented reality microscope system. For example, a secondary tilted sensor can be used for locating the optimal focus position in real-time. A motorized stage can be used to drive the main camera for capturing the in-focus, high-resolution sample images. In the medical realm, one strategy taken by the National Cancer Moonshot initiative to fight cancer cooperatively is to create an image database for different cases and connect scientists and pathologists for online collaboration. Coupling an automated microscope system with a proper autofocusing technique has the potential to convert various biological specimens into high-content images and address the challenge of high-throughput microscopy.
Acknowledgments
Z. B. and C. G. contributed equally to this work. P. S. acknowledges the support of the Thermo Fisher Scientific Fellowship. K. H. acknowledges the support of NSF 1809047. G. Z. acknowledges the support of NSF 1700941 and NSF 2012140.
References
1 Ghaznavi, F., Evans, A., Madabhushi, A. & Feldman, M. Digital imaging in pathology: whole-slide imaging and beyond.
Annu. Rev. Pathol. , 331-359 (2013). 2 Higgins, C. Applications and challenges of digital pathology and whole slide imaging. Biotech. Histochem. , 341-347 (2015). 3 Gurcan, M. N. et al. Histopathological image analysis: A review.
IEEE Rev. Biomed. Eng. , 147-171 (2009). 4 Amin, M. B. et al. The eighth edition AJCC cancer staging manual: continuing to build a bridge from a population ‐ based to a more “personalized” approach to cancer staging. CA Cancer J. Clin. , 93-99 (2017). 5 Ferreira, R. et al. in Proc. AMIA Annu. Fall Symp.
449 (American Medical Informatics Association). 6 Montalto, M. C., McKay, R. R. & Filkins, R. J. Autofocus methods of whole slide imaging systems and the introduction of a second-generation independent dual sensor scanning method.
J. Pathol. Inform. (2011). 7 Evans, A. J. et al. US Food and Drug Administration approval of whole slide imaging for primary diagnosis: A key milestone is reached and new questions are raised.
Arch. Pathol. Lab. Med. , 1383-1387 (2018).
8 Abels, E. & Pantanowitz, L. Current state of the regulatory trajectory for whole slide imaging devices in the USA.
J. Pathol. Inform. (2017). 9 Niazi, M. K. K., Parwani, A. V. & Gurcan, M. N. Digital pathology and artificial intelligence. Lancet Oncol. , e253-e261 (2019). 10 Tizhoosh, H. R. & Pantanowitz, L. Artificial intelligence and digital pathology: challenges and opportunities. J. Pathol. Inform. (2018). 11 Radakovich, N., Nagy, M. & Nazha, A. Artificial Intelligence in Hematology: Current Challenges and Opportunities. Networks , 6 (2020). 12 Dimitriou, N., Arandjelović, O. & Caie, P. D. Deep Learning for Whole Slide Image Analysis: An Overview. Front. Med. (2019). 13 Janowczyk, A. & Madabhushi, A. Deep learning for digital pathology image analysis: A comprehensive tutorial with selected use cases. J. Pathol. Inform. (2016). 14 Liu, Y. et al. Detecting cancer metastases on gigapixel pathology images. arXiv preprint arXiv:1703.02442 (2017). 15 Chen, P.-H. C. et al.
An augmented reality microscope with real-time artificial intelligence integration for cancer diagnosis.
Nat. Med. , 1453-1457 (2019). 16 Ianni, J. D. et al. tailored for Real-World: A Whole Slide Image Classification System Validated on Uncurated Multi-Site Data Emulating the Prospective Pathology Workload. Sci. Rep. , 1-12 (2020). 17 Ianni, J. D. et al. Augmenting the Pathology Lab: An Intelligent Whole Slide Image Classification System for the Real World. arXiv preprint arXiv:1909.11212 (2019). 18 Bejnordi, B. E. et al.
Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer.
JAMA , 2199-2210 (2017). 19 Gilbertson, J. R. et al.
Primary histologic diagnosis using automated whole slide imaging: a validation study.
BMC Clin. Pathol. , 4 (2006). 20 Massone, C. et al. Feasibility and diagnostic agreement in teledermatopathology using a virtual slide system.
Hum. Pathol. , 546-554 (2007). 21 Kohlberger, T. et al. Whole-slide image focus quality: Automatic assessment and impact on AI cancer detection.
J. Pathol. Inform. , 39 (2019). 22 Groen, F. C., Young, I. T. & Ligthart, G. A comparison of different focus functions for use in autofocus algorithms. Cytometry A , 81-91 (1985). 23 Sun, Y., Duthaler, S. & Nelson, B. J. Autofocusing in computer microscopy: selecting the optimal focus algorithm. Microsc. Res. Tech. , 139-149 (2004). 24 Liu, X., Wang, W. & Sun, Y. Dynamic evaluation of autofocusing for automated microscopic analysis of blood smear and pap smear. J. Microsc. , 15-23 (2007). 25 Yazdanfar, S. et al.
Simple and robust image-based autofocusing for digital microscopy.
Opt. Express , 8670-8677 (2008). 26 Redondo, R. et al. Autofocus evaluation for brightfield microscopy pathology.
J. Biomed. Opt. , 036008 (2012). 27 Pertuz, S., Puig, D. & Garcia, M. A. Analysis of focus measure operators for shape-from-focus. Pattern Recognit , 1415-1432 (2013). 28 Brenner, J. F. et al. An automated microscope for cytologic research a preliminary evaluation.
J. Histochem. Cytochem. , 100-111 (1976). 29 Yeo, T., Ong, S. & Sinniah, R. Autofocusing for tissue microscopy. Image. Vis. Comput , 629-639 (1993).
30 Osibote, O., Dendere, R., Krishnan, S. & Douglas, T. Automated focusing in bright ‐ field microscopy for tuberculosis detection. J. Microsc. , 155-163 (2010). 31 Subbarao, M., Choi, T.-S. & Nikzad, A. Focusing techniques.
Opt. Eng. , 2824-2837 (1993). 32 Russell, M. J. & Douglas, T. S. in Cytometry A , 1-9 (2000). 34 Yang, G. & Nelson, B. J. in Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003)(Cat. No. 03CH37453). ‐ based focus measure for microscopy imaging. Microsc. Res. Tech. , 987-995 (2007). 37 Bravo-Zanoguera, M., v. Massenbach, B., Kellner, A. L. & Price, J. H. High-performance autofocus circuit for biological microscopy. Rev. Sci. Instrum. , 3966-3977 (1998). 38 Price, J. H. & Gough, D. A. Comparison of phase ‐ contrast and fluorescence digital autofocus for scanning microscopy. Cytometry A , 283-297 (1994). 39 Oliva, M. A., Bravo-Zanoguera, M. & Price, J. H. Filtering out contrast reversals for microscopy autofocus. Appl. Opt. , 638-646 (1999). 40 Shen, F., Hodgson, L. & Hahn, K. in Methods Enzymol.
Vol. 414 620-632 (Elsevier, 2006). 41 Bravo-Zanoguera, M. E., Laris, C. A., Nguyen, L. K., Oliva, M. & Price, J. H. Dynamic autofocus for continuous-scanning time-delay-and-integration image acquisition in automated microscopy.
J. Biomed. Opt. , 034011 (2007). 42 Field, D. J. & Brady, N. Visual sensitivity, blur and the sources of variability in the amplitude spectra of natural scenes. Vision Res. , 3367-3383 (1997). 43 Bray, M.-A., Fraser, A. N., Hasaka, T. P. & Carpenter, A. E. Workflow and metrics for image quality control in large-scale high-content screens. J. Biomol. Screen. , 266-274 (2012). 44 Vollath, D. Automatic focusing by correlative methods. J. Microsc. , 279-288 (1987). 45 Vollath, D. The influence of the scene parameters and of noise on the behaviour of automatic focusing algorithms.
J. Microsc. , 133-146 (1988). 46 Santos, A. et al.
Evaluation of autofocus functions in molecular cytogenetic analysis.
J. Microsc. , 264-272 (1997). 47 Firestone, L., Cook, K., Culp, K., Talsania, N. & Preston Jr, K. Comparison of autofocus methods for automated microscopy.
Cytometry A , 195-206 (1991). 48 Zeder, M. & Pernthaler, J. Multispot live ‐ image autofocusing for high ‐ throughput microscopy of fluorescently stained bacteria. Cytometry A , 781-788 (2009). 49 Mendelsohn, M. L. & Mayall, B. H. Computer-Oriented analysis of human chromosomes—III. Focus. Comput. Biol. Med. , 137-150 (1972). 50 Böcker, W., Rolf, W., Müller, W. & Streffer, C. A fast autofocus unit for fluorescence microscopy. Phys. Med. Biol. , 1981 (1997). 51 Nayar, S. K. & Nakagawa, Y. Shape from focus. IEEE Trans. Pattern Anal. Mach. Intell. , 824-831 (1994).
52 Wang, Z. et al.
Compact multi-band fluorescent microscope with an electrically tunable lens for autofocusing.
Biomed. Opt. Express , 4353-4364 (2015). 53 Schnars, U., Falldorf, C., Watson, J. & Jüptner, W. in Digital Holography and Wavefront Sensing
Appl. Opt. , A52-A61 (2008). 55 Fienup, J. R. Phase retrieval algorithms: a comparison. Appl. Opt. , 2758-2769 (1982). 56 Zheng, G., Horstmeyer, R. & Yang, C. Wide-field, high-resolution Fourier ptychographic microscopy. Nat. Photon. , 739 (2013). 57 Rodenburg, J. M. Ptychography and related diffractive imaging methods. Adv. Imaging Electron Phys. , 87-184 (2008). 58 Rodenburg, J. & Maiden, A. in
Springer Handbook of Microscopy
Opt. Express , 4960-4972 (2014). 60 Song, P. et al. Full-field Fourier ptychography (FFP): Spatially varying pupil modeling and its application for rapid field-dependent aberration metrology.
APL Photonics , 050802 (2019). 61 Williams, A. J. et al. Fourier ptychographic microscopy for filtration-based circulating tumor cell enumeration and analysis.
J. Biomed. Opt. , 066007 (2014). 62 Dong, S. et al. Aperture-scanning Fourier ptychography for 3D refocusing and super-resolution macroscopic imaging.
Opt. Express , 13586-13599 (2014). 63 Bian, Z. et al. Ptychographic modulation engine: a low-cost DIY microscope add-on for coherent super-resolution imaging.
J. Phys. D: Appl. Phys. , 014005 (2019). 64 Song, P. et al. Super-resolution microscopy via ptychographic structured modulation of a diffuser.
Opt. Lett. , 3645-3648 (2019). 65 Jiang, S. et al. Wide-field, high-resolution lensless on-chip microscopy via near-field blind ptychographic modulation.
Lab Chip , 1058-1065, doi:10.1039/C9LC01027K (2020). 66 He, X., Jiang, Z., Kong, Y., Wang, S. & Liu, C. Fourier ptychography via wavefront modulation with a diffuser. Opt. Commun. , 125057 (2020). 67 Shen, C. et al.
Computational aberration correction of VIS-NIR multispectral imaging microscopy based on Fourier ptychography.
Opt. Express , 24923-24937 (2019). 68 Xu, J. et al. Accelerating wavefront-sensing-based autofocusing using pixel reduction in spatial and frequency domains.
Appl. Opt. , 3003-3012 (2019). 69 Langehanenberg, P., von Bally, G. & Kemper, B. Autofocusing in digital holographic microscopy.
3D Research , 4 (2011). 70 Memmolo, P. et al. Automatic focusing in digital holography and its application to stretched holograms.
Opt. Lett. , 1945-1947 (2011). 71 Ferraro, P., Coppola, G., De Nicola, S., Finizio, A. & Pierattini, G. Digital holographic microscope with automatic focus tracking by detecting sample displacement in real time. Opt. Lett. , 1257-1259 (2003). 72 Zhang, Y., Wang, H., Wu, Y., Tamamitsu, M. & Ozcan, A. Edge sparsity criterion for robust holographic autofocusing. Opt. Lett. , 3824-3827 (2017). 73 Lyu, M., Yuan, C., Li, D. & Situ, G. Fast autofocusing in digital holography using the magnitude differential. Appl. Opt. , F152-F157 (2017). 74 Li, W., Loomis, N. C., Hu, Q. & Davis, C. S. Focus detection from digital in-line holograms based on spectral l 1 norms. JOSA A , 3054-3062 (2007).
75 Dubois, F., Schockaert, C., Callens, N. & Yourassowsky, C. Focus plane detection criteria in digital holography microscopy by amplitude analysis.
Opt. Express , 5895-5908 (2006). 76 Thelen, A., Bongartz, J., Giel, D., Frey, S. & Hering, P. Iterative focus detection in hologram tomography. JOSA A , 1176-1180 (2005). 77 Memmolo, P., Paturzo, M., Javidi, B., Netti, P. A. & Ferraro, P. Refocusing criterion via sparsity measurements in digital holography. Opt. Lett. , 4719-4722 (2014). 78 Ren, Z., Chen, N. & Lam, E. Y. Automatic focusing for multisectional objects in digital holography using the structure tensor. Opt. Lett. , 1720-1723 (2017). 79 Bahrami, K. & Kot, A. C. A fast approach for no-reference image sharpness assessment based on maximum local variation. IEEE Signal Process. Lett. , 751-755 (2014). 80 Li, L., Xia, W., Lin, W., Fang, Y. & Wang, S. No-reference and robust image sharpness evaluation based on multiscale spatial and spectral features. IEEE Trans. Multimed. , 1030-1040 (2016). 81 Liu, Y. et al. Quality assessment for real out-of-focus blurred images.
J. Vis. Commun. Image Represent. , 70-80 (2017). 82 Liu, A., Lin, W. & Narwaria, M. Image quality assessment based on gradient similarity. IEEE Trans. Image Process. , 1500-1512 (2011). 83 Guan, J., Zhang, W., Gu, J. & Ren, H. No-reference blur assessment based on edge modeling. J. Vis. Commun. Image Represent. , 1-7 (2015). 84 Gvozden, G., Grgic, S. & Grgic, M. Blind image sharpness assessment based on local contrast map statistics. J. Vis. Commun. Image Represent. , 145-158 (2018). 85 Hassen, R., Wang, Z. & Salama, M. M. Image sharpness assessment based on local phase coherence. IEEE Trans. Image Process. , 2798-2810 (2013). 86 Leclaire, A. & Moisan, L. No-reference image quality assessment and blind deblurring with sharpness metrics exploiting fourier phase information. Journal of Mathematical Imaging and Vision , 145-172 (2015). 87 Narvekar, N. D. & Karam, L. J. A no-reference image blur metric based on the cumulative probability of blur detection (CPBD). IEEE Trans. Image Process. , 2678-2683 (2011). 88 Jiménez, A. et al. in Optics, Photonics and Digital Technologies for Imaging Applications IV. et al.
Focus Quality Assessment of High-Throughput Whole Slide Imaging in Digital Pathology.
IEEE Trans. Med. Imaging , 62-74 (2019). 90 Kang, L., Ye, P., Li, Y. & Doermann, D. in Proceedings of the IEEE conference on computer vision and pattern recognition. et al.
A shallow convolutional neural network for blind image sharpness assessment.
PLoS One (2017). 92 Senaras, C., Niazi, M. K. K., Lozanski, G. & Gurcan, M. N. DeepFocus: detection of out-of-focus regions in whole slide digital images using deep learning. PLoS One (2018). 93 Campanella, G. et al. Towards machine learned quality control: A benchmark for sharpness quantification in digital pathology.
Comput. Med. Imaging Graph. , 142-151 (2018). 94 Yang, S. J. et al. Assessing microscope image focus quality with deep learning.
BMC Bioinformatics , 77 (2018). 95 Liron, Y., Paran, Y., Zatorsky, N., Geiger, B. & Kam, Z. Laser autofocusing system for high ‐ resolution cell biological imaging. J. Microsc. , 145-151 (2006).
96 Reinheimer, G. Arrangement for automatically focussing an optical instrument. US patent 3,721,827 (1973). 97 Sato, M. & Matsuno, J. Apparatus for focusing on transparent objects. US patent 5,530,237 (1996). 98 Yonezawa, Y. Apparatus for detecting an in-focus position of a substrate surface having a movable light intercepting member and a thickness detector. US patent 5,483,079 (1996). 99 Ito, K., Musha, T. & Kato, K. Method and apparatus for detecting a focussing condition of an objective lens. US patent 4,422,168 (1983). 100 Noda, H., Dosaka, S. & Kurosawa, H. Automatic focusing apparatus which removes light reflected by a lower surface of a sample. US patent 5,317,142 (1994). 101 Kramer, P., Bouwhuis, G. & Day, P. E. Apparatus for reading a flat reflecting record carrier with autofocusing means. US patent 3,876,841 (1975). 102 Velzel, C. H. & Greve, P. F. Apparatus for optically reading a record carrier and correcting focus error. US patent 4,074,314 (1978). 103 Jorgens, R. & Faltermeier, B. Process and apparatus for the automatic focusing of microscopes. US patent 4,958,920 (1990). 104 Mueller, O. Method of and apparatus for the automatic focusing of stereoscopic microscopes. US patent 4,025,785 (1977). 105 Li, Q., Bai, L., Xue, S. & Chen, L. Autofocus system for microscope.
Opt. Eng. , 1289-1294 (2002). 106 Liu, C.-S. & Jiang, S.-H. A novel laser displacement sensor with improved robustness toward geometrical fluctuations of the laser beam. Meas. Sci. Technol. , 105101 (2013). 107 Liu, C.-S., Lin, Y.-C. & Hu, P.-H. Design and characterization of precise laser-based autofocusing microscope with reduced geometrical fluctuations. Microsyst. Technol. , 1717-1724 (2013). 108 Liu, C.-S. & Jiang, S.-H. Design and experimental validation of novel enhanced-performance autofocusing microscope. Appl. Phys B , 1161-1171 (2014). 109 Liu, C.-S., Wang, Z.-Y. & Chang, Y.-C. Design and characterization of high-performance autofocusing microscope with zoom in/out functions.
Appl. Phys B , 69-80 (2015). 110 Liu, C.-S. & Jiang, S.-H. Precise autofocusing microscope with rapid response.
Opt. Lasers Eng. , 294-300 (2015). 111 Silfies, J. S., Lieser, E. G., Schwartz, S. A. & Davidson, M. W. Nikon Perfect Focus System (PFS) et al.
Microscopy system with auto-focus adjustment by low-coherence interferometry. US patent 9,869,852 (2018). 114 McKay, R. R., Baxi, V. A. & Montalto, M. C. The accuracy of dynamic predictive autofocusing for whole slide imaging.
J. Pathol. Inform. (2011). 115 Virág, T., László, A., Molnár, B., Tagscherer, A. & Varga, V. S. Focusing method for the high-speed digitalisation of microscope slides and slide displacing device, focusing optics, and optical rangefinder. US patent 7,663,078 (2010). 116 Prabhat, P., Ram, S., Ward, E. S. & Ober, R. J. Simultaneous imaging of different focal planes in fluorescence microscopy for the study of cellular dynamics in three dimensions. IEEE Transactions on Nanobioscience , 237-242 (2004).
117 Abrahamsson, S. et al.
Fast multicolor 3D imaging using aberration-corrected multifocus microscopy.
Nat. Methods. , 60-63 (2013). 118 Descloux, A. et al. Combined multi-plane phase retrieval and super-resolution optical fluctuation imaging for 4D cell microscopy.
Nat. Photon. , 165-172 (2018). 119 Xiao, S. et al. High-contrast multifocus microscopy with a single camera and z-splitter prism. bioRxiv (2020). 120 Dong, R.-T., Rashid, U. & Zeineh, J. System and method for generating digital images of a microscope slide. US patent 2005/0089208A1 (2005). 121 Hulsken, B. & Stallinga, S. Sensor for microscopy. US patent 10,353,190 (2019). 122 Hulsken, B. Scanning imaging system with a novel imaging sensor with gaps for electronic circuitry. US patent 10,091,445 (2018). 123 Hulsken, B. Determining a polar error signal of a focus position of an autofocus imaging system. US patent 9,578,227 (2017). 124 Hulsken, B. Method for simultaneous capture of image data at multiple depths of a sample. US patent 9,910,258 (2018). 125 Hulsken, B. & Stallinga, S. Autofocus imaging. US patent 10,365,468 (2019). 126 Vink, J. P., Hulsken, B., Wolters, M., Van Leeuwen, M. B. & Shand, S. H. System for generating a synthetic 2D image with an enhanced depth of field of a biological sample. US patent 10,623,627 (2020). 127 Zou, Y., Crandall, G. J. & Olson, A. Real-time focusing in line scan imaging. US patent 9,841,590 (2017). 128 Olson, A., Saligrama, K., Zou, Y. & Najmabadi, P. Real-time autofocus focusing algorithm. US patent 10,459,193 (2020). 129 Price, J. H. Autofocus system for scanning microscopy having a volume image formation. US patent 5,932,872 (1999). 130 Kinba, A., Hamada, M., Ueda, H., Sugitani, K. & Ootsuka, H. Auto focus detecting device comprising both phase-difference detecting and contrast detecting methods. US patent 5,597,999 (1997). 131 Guo, K., Liao, J., Bian, Z., Heng, X. & Zheng, G. InstantScope: a low-cost whole slide imaging system with instant focal plane detection.
Biomed. Opt. Express , 3210-3216 (2015). 132 Liao, J. et al. Single-frame rapid autofocusing for brightfield and fluorescence whole slide imaging.
Biomed. Opt. Express , 4763-4768, doi:10.1364/BOE.7.004763 (2016). 133 Silvestri, L. et al. RAPID: Real-time image-based autofocus for all wide-field optical microscopy systems. bioRxiv , 170555 (2017). 134 Liao, J. et al.
Dual light-emitting diode-based multichannel microscopy for whole-slide multiplane, multispectral and phase imaging.
J. Biophotonics , e201700075, doi:10.1002/jbio.201700075 (2018). 135 Liao, J. et al. Terapixel hyperspectral whole-slide imaging via slit-array detection and projection.
J. Biomed. Opt. , 066503 (2018). 136 Liao, J. et al. Rapid focus map surveying for whole slide imaging with continuous sample motion.
Opt. Lett. , 3379-3382 (2017). 137 Jiang, S. et al. Rapid and robust whole slide imaging based on LED-array illumination and color-multiplexed single-shot autofocusing.
Quant Imaging Med Surg , 823-831 (2019).
138 Guo, C. et al.
OpenWSI: a low-cost, high-throughput whole slide imaging system via single-frame autofocusing and open-source hardware.
Opt. Lett. , 260-263, doi:10.1364/OL.45.000260 (2020). 139 Belthangady, C. & Royer, L. A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat. Methods. , 1-11 (2019). 140 Dastidar, T. R. & Ethirajan, R. Whole slide imaging system using deep learning-based automated focusing.
Biomed. Opt. Express , 480-491 (2020). 141 Jiang, S. et al. Transform-and multi-domain deep learning for single-frame rapid autofocusing in whole slide imaging.
Biomed. Opt. Express , 1601-1612 (2018). 142 Li, Q. et al. Rapid Whole Slide Imaging via Learning-based Two-shot Virtual Autofocusing. arXiv preprint arXiv:2003.06630 (2020). 143 Luo, Y., Huang, L., Rivenson, Y. & Ozcan, A. Single-shot autofocusing of microscopy images using deep learning. arXiv preprint arXiv:2003.09585 (2020). 144 Pinkard, H., Phillips, Z., Babakhani, A., Fletcher, D. A. & Waller, L. Deep learning for single-shot autofocus microscopy.
Optica , 794-797 (2019). 145 Rivenson, Y. et al. Deep learning microscopy.
Optica , 1437-1443 (2017). 146 Shajkofci, A. & Liebling, M. in arXiv preprint arXiv:2001.00667 (2020). 148 Wu, Y. et al. Three-dimensional virtual refocusing of fluorescence microscopy images using deep learning.
Nat. Methods. , 1323-1331 (2019). 149 Ren, Z., Xu, Z. & Lam, E. Y. Learning-based nonparametric autofocusing for digital holography. Optica , 337-344 (2018). 150 Wei, L. & Roberts, E. Neural network control of focal position during time-lapse microscopy of cells. Sci. Rep. , 1-10 (2018). 151 Ronneberger, O., Fischer, P. & Brox, T. in International Conference on Medical image computing and computer-assisted intervention.
Proceedings of the IEEE conference on computer vision and pattern recognition.1125-1134.