Computational interference microscopy enabled by deep learning.

Quantitative phase imaging (QPI) has been widely applied in characterizing cells and tissues. Spatial light interference microscopy (SLIM) is a highly sensitive QPI method due to its partially coherent illumination and common path interferometry geometry. However, SLIM's acquisition rate is limited because of the four-frame phase-shifting scheme. On the other hand, off-axis methods such as diffraction phase microscopy (DPM) allow for single-shot QPI. However, the laser-based DPM system is plagued by spatial noise due to speckles and multiple reflections. In a parallel development, deep learning was proven valuable in the field of bioimaging, especially due to its ability to translate one form of contrast into another. Here, we propose using deep learning to produce synthetic, SLIM-quality, and high-sensitivity phase maps from DPM using single-shot images as the input. We used an inverted microscope with its two ports connected to the DPM and SLIM modules such that we have access to the two types of images on the same field of view. We constructed a deep learning model based on U-net and trained on over 1000 pairs of DPM and SLIM images. The model learned to remove the speckles in laser DPM and overcame the background phase noise in both the test set and new data. The average peak signal-to-noise ratio, Pearson correlation coefficient, and structural similarity index measure were 29.97, 0.79, and 0.82 for the test dataset. Furthermore, we implemented the neural network inference into the live acquisition software, which now allows a DPM user to observe in real-time an extremely low-noise phase image. We demonstrated this principle of computational interference microscopy imaging using blood smears, as they contain both erythrocytes and leukocytes, under static and dynamic conditions.

INTRODUCTION

Quantitative phase imaging (QPI) has developed into an active field with the goal of providing a label-free alternative to biomedical imaging, complementary to the standard techniques relying on stains and fluorescent tags.[1] QPI yields the optical path length map associated with the specimen and, thus, informs about both the thickness and the refractive index of the structure of interest. Due to its quantitative and nondestructive nature, QPI has found important biomedical applications ranging from basic science to clinical diagnosis.[2] As the specimen refractive index reports on the dry mass density, QPI has been employed to study cell growth.[3-7] Analyzing the spatio-temporal fluctuations of dry mass provided a new way of monitoring intracellular transport and differentiating between diffusive and active processes.[8-10] Due to its sensitivity to nanometer scale optical path length changes, QPI is capable of measuring cell membrane fluctuations[11-14] and imaging unlabeled single microtubules.[15] The full holographic information (phase and amplitude) associated with a field scattered by a transparent object allows for tomographic reconstructions without ambiguity, as demonstrated by Wolf in 1969.[16] Thus, QPI-based tomography has been demonstrated by acquiring phase imaging data as a function of illumination angles,[17-19] scanning the object through focus,[20,21] and performing spectroscopic measurements.[22] This approach has been recently extended to second harmonic fields.[23] QPI has led to the discovery of new intrinsic markers for cancer diagnosis and prognosis without the variability generally introduced by stains.[24-31] More recently, QPI has been extended to strongly scattering specimens, such as embryos, spheroids, and acute brain slices,[32,33] which significantly expanded QPI’s range of applications.

In terms of methodology, QPI instruments are divided into off-axis[34-38] and phase shifting[32,39,40] geometries. Phase shifting methods (see Chap. 10 of Ref. 1) use time domain modulation and, as a result, preserve the maximum space-bandwidth product for a given imaging instrument at the expense of frame rate. Conversely, off-axis interferometry (see Chap. 9 in Ref. 1) operates on spatial modulation, which provides single shot fast imaging at the expense of space-bandwidth product. In addition, due to the spatial domain image processing, off-axis methods also generally result in higher spatial phase noise. This spatial noise is further amplified by speckles whenever monochromatic rather than broadband light is used (see Ref. 39 for an assessment of the spatial noise in both geometries).

An ideal QPI method would provide the low noise, high resolution associated with phase shifting interferometry, and single-shot performance associated with off-axis geometries. In this paper, we demonstrate that such a performance can be achieved by using deep learning to produce an image-to-image translation from single shot noisy data to phase-shifting low-noise images, on which the network was a priori trained. For generating the input data, we used diffraction phase microscopy (DPM),[34] an off-axis common path QPI technique, while for ground truth, we used images of the same field of view obtained by spatial light interference microscopy (SLIM),[39] which is a broadband light, phase-shifting, common path method. A U-Net[41] convolutional neural network was trained to infer a SLIM image from a DPM image as the input. The performance of this image translation was significant, with a peak signal-to-noise ratio (PSNR) around 30 and a Pearson correlation coefficient of 0.79. Finally, we integrated the inference algorithm into the acquisition software, which allows for real-time operation.

RESULTS

DPM and SLIM data collection

In order to acquire training data necessary to produce SLIM-quality images in a single-shot, we developed a combined SLIM-DPM system, which generates both images from the same field of view (Fig. 1). The DPM and SLIM modules were placed at the two side ports of a commercial inverted microscope (Axio Observer Z1, Zeiss). A coupled fiber green laser (λ = 532 nm) was used as illumination for DPM, with the condenser aperture closed to minimum. A collimated LED source (λ = 623,20 nm bandwidth) was used as illumination for SLIM (CellVista SLIM Pro, Phi Optics, Inc.), with the conventional ring illumination associated with phase contrast microscopy. Using the 20×/0.4 NA objective, the magnified image is replicated to either port by using a switch.

FIG. 1.

Schematic of the imaging setup. The system is built around an inverted microscope. We are using a 20×/0.4 NA objective. The two side ports connect to the DPM (right) and SLIM (left) modules. Thus, we obtain SLIM and DPM images on the same field of view. The focal lengths of lenses 1 and 2 are 100 and 200 mm, respectively. Lenses 3 and 4 have the same focal length. When switching between DPM and SLIM, prisms 1 and 2 are switched to different positions and the condenser is set to the PH1 set. Due to the magnification of the 4f system in DPM, a registration is needed to match the DPM and SLIM images.

The DPM principle has been presented in detail previously.[42] Briefly, a phase diffraction grating is placed at the image plane to replicate the spatial frequency content at the Fourier plane of lens 1 along multiple diffraction orders. The zeroth order is spatially filtered to remove all higher frequencies, the first order beam is passed unaltered, and all the remaining orders are completely blocked. Thus, following lens 2, the camera detects the interference between the image field and the filtered zeroth order, which acts as a reference field. There is a further 2.5× magnification produced by the L1–L2 4-f system. The resulting interferogram is processed using the well-known Hilbert transform to retrieve the phase map from a single recording. The exposure time for each DPM image was 2 ms throughout all our measurements. As illustrated in Fig. 1, the resulting DPM image is affected by background non-uniformities, which in the past have been mitigated using post processing.[43]

The SLIM module (see Ref. 39 for details) contains a phase-only spatial light modulator (SLM) at the pupil plane created by lens 3. A binary mask is created on the SLM to precisely match the phase contrast ring placed in the condenser aperture. By shifting the phase of the mask in increments of π/2, we record four intensity images using camera 2. The four frames are combined to obtain the quantitative phase map, as described in Ref. 39.

To validate our phase measurements, we performed a side-by-side comparison of the two measurements on a 1 μm polystyrene bead, immersed in oil (Zeiss, Immersol 518F), as illustrated in Fig. 2. We diluted one drop of bead suspension into 10 ml of ethanol and used a vortex mixer to disperse it. The refractive index of the beads is 1.588 at 623 nm wavelength, while that of the oil is 1.518. As a result, the expected phase shift from the bead using red led is where d is the height of the bead, n and n0 are the refractive index of bead and oil, respectively, and λ is the wavelength of light.

FIG. 2.

Oil-immersed 1 μm diameter polystyrene beads imaged by DPM and SLIM, (a) The cropped DPM phase map of the bead and the cross section plot. (b) The cropped SLIM phase map of the bead and the cross section plot. The measured phase values are comparable to the expected number, as detailed in the text.

For comparison, the DPM phase map, which was imaged with a green laser, was normalized by the ratio of two wavelengths. The values obtained by DPM and SLIM, 0.71 rad and 0.71 rad, are compatible with the expected value of 0.71 rad.

Training procedure

In order to use deep learning to infer SLIM images from DPM data, we employed the U-Net described in Fig. 3. Our model was a modified version of the U-Net.[41] We added residual connection within each feature-extracting block in the encoder path and batch normalization layers[44] between convolution and activation layers. Three drop-out layers were added in the encoder path and the bottleneck to avoid overfitting. Our model had far fewer trainable parameters than the original U-Net, as we reduced the number of channels in each layer to a quarter of the original proposed value. The model was optimized using adaptive moment estimation (Adam)[45] against the mean-squared error. All the input values were scaled into the range of [0, 1].

FIG. 3.

The U-net structure. The model has a symmetric layout and consists of three major parts: the encoder path, the bottleneck, and the decoder path. We added batch normalization layers between convolution and activation layers to stabilize the learning process. After every two convolutional operations, a residual connection was added for faster convergence. Three drop-out layers were also used to avoid overfitting. The kernel size was set to 3 × 3 and the number of kernels was reduced to a quarter of the original proposed number. A 400 × 400 region was randomly cropped from each original 1536 × 1536 image during training.

To demonstrate the deep-learning-enabled computational interference microscopy (CIM) operation on live cells, we used blood cell smears, which contain red blood cells and several types of white blood cells. More than 1200 images were recorded by both SLIM and DPM with over 100 cells in each field of view. To boost variation within our dataset, the images were cropped into random 400 × 400 pixel2 regions and fed into the U-net for training. The network was trained for 21 h on a single GTX 1070 graphic processing unit (GPU) using 1000 epochs.

Figure 4 illustrates the DPM input data (left column), SLIM ground truth (middle), and the resulting CIM (right). Visually, the U-Net is able to reduce the overall noise of the DPM input and produce a remarkably similar image to the SLIM ground truth. In order to quantify the performance of the neural network, we computed the peak signal to noise ratio (PSNR), the Pearson correlation coefficient, and structural similarity index measure (SSIM)[46] between the ground truth and the prediction. The mean-squared loss value on the training dataset and the validation dataset after each epoch is plotted in Fig. 5. The model checkpoint with lowest loss value on the validation dataset was selected as our end model for evaluation and deployment. Figure 5 also shows the result of the average PSNR, Pearson correlation, and SSIM calculated on the training, validation, and test datasets. The SSIM is computed using the following equation: where μ and μ are the mean values of x and y, respectively, σ2 and σ2 are the variance of x and y, respectively, and σ is the covariance of x and y. C1 and C2 are small constants used for numerical stability. The results are consistent among all three datasets, indicating that the model generalized well on the unseen test dataset. Figure 6 illustrated the CIM images with lowest, average, and highest Pearson correlation coefficients. The profiles of a red blood cell at the exact same position in both SLIM and CIM images were plotted. The results indicated that even in the worst-performing test cases, CIM was able to capture the sample structures. In most test cases, the CIM image and the SLIM image had a closely matched profile.

FIG. 4.

DPM input data (left column), SLIM ground truth data (center), and the U-net inference (right). All images share the same calibration and scale bar. The neural network correctly infers SLIM images from the DPM input with drastically reduced noise levels.

FIG. 5.

Model convergence and performance. After each epoch, we plotted the mean-squared loss value on both the training dataset and validation dataset. The model checkpoint with the lowest error on the validation dataset was selected as our end model for evaluation and deployment. The average PSNR, Pearson correlation coefficient, and structural similarity index measure (SSIM) of our model on the training, validation, and test datasets were computed. Our models generalized well on the unseen test dataset.

FIG. 6.

Comparison of the CIM images with the lowest, average, and highest Pearson correlation coefficient. The images were chosen from the test dataset. We plotted the profile of a chosen red blood cell at the exact same pixel position. The Pearson correlation coefficient (denoted as ρ) and the structural similarity index measure (SSIM, denoted as s) were computed for each image pair. CIM was able to capture all the structures properly. In CIM images with higher Pearson correlation coefficient and SSIM, the cross section profile matched closely to that of the SLIM image.

Real-time inference

To boost the usability of CIM, we integrated the inference algorithm into the real-time acquisition software for DPM (wDPM CellVista Pro, Phi Optics, Inc.). Figure 7 shows the user interface, with the DPM image being reconstructed and used as the input for inference. Supplemental video visualization 1 illustrates the real-time operation. The conversion from the noisy DPM images to CIM takes place at a push of a button. Note that translating the stage does not affect the quality of the inference, which works very well for both red and white blood cells. Thus, we envision that CIM can be readily used for automating large field of view and multi-well plate scanning. To integrate the trained model into our acquisition software, we built the same network architecture from scratch using the Nvidia TensorRT library and loaded the transposed trained weights. The computer was equipped with the Nvidia RTX2070 GPU. The software then ran the model inference on a 1536 × 1536 image at a speed of 58 ms/image. Thus, the resulting CIM is computed and displayed in real time at a rate of up to 14 frames/s. This performance can be boosted further by either lowering the pixel size or upgrading the GPU.

FIG. 7.

Live imaging interface. We demonstrate our live imaging system through the three channels: DPM, SLIM, and CIM channel (see visualization 1). The image shows the graphic user interface and a snapshot of CIM operation in real-time. Multimedia view: https://doi.org/10.1063/5.0041901.1

Supplemental video visualization 2 shows a CIM time lapse of a fresh unlabeled blood smear. We collected blood from a healthy volunteer and diluted one drop of blood with 10 ml phosphate-buffered saline. No effect was devoted to stabilizing the smear such that we can test the ability of CIM to operate on highly dynamic samples. These specimens were never imaged by SLIM, and thus, the network was not trained on these samples. Clearly, CIM performs very well and even reveals minute membrane fluctuation in individual red blood cells. The area of cell overlapping sometimes displays lower phase values than expected, but this appears to be an optical artifact, rather than a computational artifact, as it is also present in Fig. 4 (SLIM column). These data highlight the capability of CIM to run under flow conditions for applications such as flow cytometry. Figure 8 shows four sequential frames from the time lapse in visualization 2, taken 100 ms apart, with 2 ms exposure each. The cells selected in the rectangular boxes moved very fast, as can be visualized in the movie and these snapshots. These results prove that the CIM system can provide single-shot high speed measurements, as allowed by DPM, while the output has the high quality of SLIM images.

FIG. 8.

Dynamic imaging of blood cells with snapshots at different moments in time, 100 ms apart, as indicated. The scale bar and color bar are the same for all the frames. The full video can be visualized as visualization 2. In these frames, one can see cells (e.g., those in the rectangular boxes) that flow very fast. However, the CIM inference is accurate and operates in real-time. Multimedia view: https://doi.org/10.1063/5.0041901.2

METHODS

Deep learning

We formulated the problem as an image-to-image regression, where the deep neural network takes in a DPM image as the input and predicts a new image that is close to a SLIM image of the same field of view. Our neural network was a variant of U-Net, which has shown great performance on similar tasks with quantitative phase imaging data.[47] Unlike the stock U-Net, we added batch normalization layers between convolution and activation layers to stabilize the learning process. We also added in a residual connection after every two convolution operations (on the same field of view) for faster convergence and better performance. The model has a symmetric layout and consists of three major parts: an encoder path, a bottleneck and a decoder path. The encoder path captures contextual information in the image. It consists of four stages of convolutional and non-linear activation operations with residual connection. Each stage was followed by a 2 × 2 downsampling operation. The decoder path is almost symmetric to the encoder path, except that it has upsampling operations to combine low-resolution and high-resolution information and enables localization. The convolution kernel size within the network was set to 3 × 3 except for those used in residual connection, which was set to 1 × 1. The number of kernels in each stage was set to 16, 32, 64, and 128, respectively, reduced to a quarter of the proposed value in the original U-Net paper.[41] Thus, our model had only 3.3 × 106 trainable parameters. Based on the training results, it was apparent that this model was already complex enough to approximate the transform from DPM images to SLIM images. We picked the mean-squared error as our loss function and used the Adam optimizer with the default exponential decay rate for both moment estimates (0.9 and 0.999, respectively). We used PSNR, Pearson correlation coefficient, and structural similarity index measure (SSIM) to measure the performance of our network All the input images (DPM and SLIM) were scaled from [−π, π] to [0, 1]. We trained the model from scratch with a learning rate of 6 × 10−5 for 1000 epochs. The batch size was set to 4 during training. The mean squared loss value on both the training and the validation dataset was plotted after each epoch. To add more variation during the training process, we applied random cropping to each training image. A 400 × 400 crop was selected randomly from each original 1536 × 1536 image and fed into the model during one epoch. Since U-Net is fully convolutional, it can pick up features on these smaller crops and apply them later onto the larger images. This random cropping has two main advantages. First, it served as a form of data augmentation, contributing to better generalizability. Second, it reduced the training time and GPU memory requirement. The model was implemented using Tensor-Flow and the training was performed on a GTX 1070 GPU with 8 GB memory. The training took ~21 h. This training cost can be justified as it is a one-pass operation. The trained model can be applied to a larger number of red blood cell measurements, and overall, the time CIM saved can outweigh the training time cost. In addition, the training time can be reduced by using more recent GPUs.

Real-time implementation

To incorporate the trained model into our acquisition software, we first saved all the trained weights (convolution kernels and batch normalization parameters) into a single HDF5 file. Then, we constructed the same network architecture within our acquisition software using the Nvidia TensorRT API in C++. Due to the mismatch between weight formats in TensorElow and TensorRT, the weights were first transposed and then loaded into the network architecture in C++. Once the network was built, TensorRT optimized the inference procedure by enumerating different configuration of kernels. This optimization was necessary because the optimal configuration for inference differs from hardware to hardware. Thus, we over-lapped the model optimization with the software initialization by utilizing multi-threading in C++.

DISCUSSION

QPI has developed into several types of methods, characterized by different advantages and drawbacks according to the specific geometry. The laser DPM enables a high acquisition rate but is plagued with speckles and background phase noise. White light DPM overcomes the noise with broadband light at the expense of intensity loss; thus, it has a lower acquisition rate. Because of illumination NA, SLIM has a higher transverse resolution than the DPM based methods. However, the acquisition speed of SLIM is further limited by the four phase shifts.

Artificial intelligence (AI) has already demonstrated its potential to transform biomedical imaging. In microscopy, image-to-image translation techniques proved to be valuable in generating synthetic fluorescence images for cell biology applications[48,49] and digital staining for pathology applications.[50] Phase imaging with computational specificity (PICS) has been introduced recently to retrieve molecular specificity via deep learning, thus eliminating the phototoxicity and photobleaching associated with fluorescence tags[51] and intrinsic fertility makers.[52]

Here, we presented a different application of AI, where the goal was to combine the benefits of different QPI instruments. The trained model performed generally well in producing high quality images given relatively noisy input DPM images, attaining, on average, a Pearson correlation coefficient and SSIM around 0.8 on the test dataset. Even in the worst-performing test cases, CIM was able to keep the structure of the samples. CIM is a new QPI method that combines a high acquisition rate from DPM and low-noise and high resolution from SLIM, fitting into the category of QPI methods that are relatively inexpensive and high-speed. This approach allows for applications that involve both highly dynamic and sensitive measurements. Since phase is the intrinsic marker of transparent samples, we anticipate that our current model can be extended to other specimens via transfer learning with a small amount of new data, making our method extendible to new biological samples at low costs.