Interferometer-based structured-illumination microscopy utilizing complementary phase relationship through constructive and destructive image detection by two cameras.

In an interferometer-based fluorescence microscope, a beam splitter is often used to combine two emission wavefronts interferometrically. There are two perpendicular paths along which the interference fringes can propagate and normally only one is used for imaging. However, the other path also contains useful information. Here we introduced a second camera to our interferometer-based three-dimensional structured-illumination microscope (I(5)S) to capture the fringes along the normally unused path, which are out of phase by π relative to the fringes along the other path. Based on this complementary phase relationship and the well-defined phase interrelationships among the I(5)S data components, we can deduce and then computationally eliminate the path length errors within the interferometer loop using the simultaneously recorded fringes along the two imaging paths. This self-correction capability can greatly relax the requirement for eliminating the path length differences before and maintaining that status during each imaging session, which are practically challenging tasks. Experimental data is shown to support the theory.

Introduction

We reported earlier a resolution-enhancing scheme for three-dimensional (3D) widefield fluorescence microscope, I5S (Shao ), that greatly improves the resolution of optical microscopy to nearly 100 nm in all three dimensions. It combines structured-illumination microscopy (SIM; Heintzmann & Cremer, 1999; Gustafsson ; Gustafsson, 2000; Gustafsson ) with an interferometer using two opposing objective lenses (Hell & Stelzer, 1992; Gustafsson ). In SIM, the sample is illuminated by a periodic pattern, which mixes with the otherwise undetectable high-resolution information and, in spatial frequency space, moves it into the passband of the microscope in the form of moiré fringes. The down translated high-resolution information is then computationally recovered to produce an image with extended resolution (Heintzmann & Cremer, 1999; Gustafsson ; Gustafsson, 2000; Gustafsson ). SIM can double the conventional resolution in all three dimensions, resulting in a resolution of about 100 nm laterally and 300 nm axially (Gustafsson ). I5S is an extension of SIM in which a second objective lens is used and placed on the opposite side of the sample, and the two apertures are combined interferometrically (Hell & Stelzer, 1992; Gustafsson ; Shao ). As a result, the axial resolution is greatly improved both because the set of light-gathering angles of the microscope is enlarged and because the highest axial spatial frequencies in the illumination pattern are increased due to the interference of the illumination beams coming from the two opposing objectives. An isotropic 100-nm-scale resolution is achieved by this technique in all dimensions (Shao ).

In I5S, a collimated laser illumination beam is diffracted by a transmission phase grating and the central three diffraction orders are split by a beam splitter, resulting in six mutually coherent illumination beams being incident on the sample, one triplet of beams passing through each objective lens (Fig. 1). The illumination pattern that results from the interference of these six beams consists of 19 frequency components (Fig. 2D) and can be expressed as (Shao )

Fig. 1

The schematic drawing of an I5S microscope. The illumination light passes first through a transmission grating, which diffracts it into three beams (green lines), and then through a beam splitter, which splits each beam and directs three beams to each of the two opposing objective lenses. The same beam splitter combines the two beams of emission light (red) from the sample onto the camera. Normally, only one camera (CCD 1) is used to record the emission light from one of the two beam splitter exit ports. Here we introduced a second imaging path to record on CCD 2 the emission light from the other exit port. The movable objective lens can be positioned in X, Y and Z with respect to the stationary objective lens. Mirrors M3 and M4 can be translated together to adjust the path length difference. The grating can be rotated and laterally translated to control the orientation and lateral phase of the illumination pattern. Mirrors M3 is made partially transmissive to let conventional illumination come in from there so that I5S detection OTF can be measured.

Fig. 2

Principles of I5S explained in frequency space. (A) The generalized pupil function (GPF) of an I5S/I5M microscope, i.e. the portions of the wavefront detectable through the two objective lenses. In a conventional widefield microscope, only one of the two shells is detectable. The support (i.e. region of nonzero value) of the intensity detection OTF, which is the autocorrelation of the GPF, is shown in (B). In comparison, the detection OTF of a conventional microscope is just the middle region. (C) The six dots represent the six collimated illumination beams being incident upon the sample. Their interference creates a 19-component illumination intensity pattern (D) organized into five vertical lines corresponding to m=−2 to 2. (E) The effective I5S OTF support is shown in ky–kz cross-section. As a reference, the conventional OTF support is drawn in dashed contour in (E).

where r represents the lateral coordinates (x, y), p is the lateral component of the wave vector of each beam not coincident with the optical axis k(or the side beams), I(z) is the mth-order axial illumination pattern in real space corresponding to the mth axial row of illumination frequencies shown as dots in Figure 2D and is a phase angle (Gustafsson ; Shao ). Given the form of the illumination pattern in Eq. (1) and the condition that the axial position of the illumination pattern is fixed in relation to the focal plane of the microscope as the sample is refocused during 3D acquisition, the Fourier transform of an observed 3D image D can be expressed as follows (Gustafsson ; Shao ):

where is the sample information, O (Figs 3A–C) is the mth-order effective optical transfer function (OTF) obtained by convolving the detection OTF with the axial Fourier transform of I. The axially extended support, or the nonzero valued region, of O in I5S is attributed to both the high axial illumination frequencies (Fig. 2D) and a detection OTF with two axially extended support regions (or the sidebands; Fig. 2B) in addition to the usual OTF support region (or the central band) of the single-objective microscopy (Gustafsson ). Equation (2) shows that I5S improves the resolution laterally via frequency shift of the sample information and axially via the extended support in the effective OTFs.

Fig. 3

Experimentally measured and rotationally averaged I5S OTFs O of Eq. (2) for m= 0, ±1 and ±2 (A—C). (D) Demonstrates that m= 0 and m=±1 components partly overlap when they are placed where they belong in frequency space.

Image reconstruction and path length difference

The raw data of SIM, including I5S, is the summation of five components (Eq. (2)). These components can be separated given a series of images taken with five different lateral pattern's phases ( in Eqs (1) and (2); Gustafsson ; Shao ). Because the lateral pattern in our implementation is one-dimensional (1D), the same pattern needs to be applied multiple times with different angles (i.e. p in Eqs (1) and (2) with the same magnitude but different orientations) so as to isotropically extend the lateral resolution. A final image with 100 nm resolution in all three dimensions (Fig. 2E) is reconstructed by positioning all the separated information components back to their original locations in frequency space and then correcting for amplitude differences via a generalized Wiener filter (Gustafsson ; Shao ).

For I5S reconstruction to work, it is required that the raw image is acquired under the same conditions as the measured effective OTFs. In I5S or any interferometric set-up, one condition prone to variation in different imaging sessions is the path length difference of the interferometer loop. Before every experiment, significant time must be spent on digitally adjusting this path length difference to reach zero, which gives the maximum axial fringe contrast. Even if this preimaging adjustment is performed very accurately, instrumental drift will have caused different path length differences, i.e. different imaging conditions, in the actual data set and in the data set taken for the OTF determination, resulting in axially ringing artefacts in the reconstruction because of the unmatched OTF used. To make I5S easier to use, we need a scheme that can relax this very stringent requirement on path length difference consistency without inducing artefacts.

Another disadvantage of the current I5S set-up is that only the imaging interference fringes at one of the beam splitter's two exit ports are recorded, meaning half of the emission light is discarded. Capturing both halves of the emission light simultaneously would significantly improve the signal-to-noise ratio. Furthermore, because of the complementary phase relationship between the fringes at the two beam splitter exits (to be discussed next), the actual path length difference within the interferometer loop can be estimated from a pair of 3D images simultaneously acquired through the two exits. With this quantity known, one can then either choose a matching set of OTFs from a library of OTFs or, in a more attractive solution, synthesize a set of OTFs that can account for the estimated path length difference, and use those to reconstruct the data set. As a result, the requirement for achieving a specific path length difference for each experiment can be relaxed.

Path length difference estimation

First, we discuss how the path length difference can be estimated from an I5S data set. Given a known path length difference δ, the detection OTF of I5S can be divided into three regions based on different phases: b, 0 and −b, where b= 2πδ/λem and λem is the emission wavelength (Fig. 4A). These values are because of the fact that the detection OTF is an autocorrelation of the generalized pupil function (GPF; Fig. 2A) and the relative phase difference between the two shells of GPF (i.e. the two wavefronts collected separately by the two objective lenses) is b. Likewise for the illumination components (Fig. 4B), three groups can be distinguished based on the components’ phases: a, 0 and −a, where a= 2πδ/λex and λex is the excitation wavelength. Because the effective I5S OTF O is the convolution of the detection OTF with the mth illumination order (Eq. (2) and Fig. 3) and phases of the operands add in convolution, the phases in different regions of I5S effective OTFs are various additive combinations of 0, ±a and ±b. The colour-coded maps of phase in these OTFs are shown in Figures 4C–E.

Fig. 4

The maps of phase values in I5S detection OTF (A), illumination structures (B) and the effective OTFs for m= 0 (C), ±1 (D) and ±2 (E). The reason why most illumination components are stretched axially compared to Figure 2D was explained in detail in earlier publications (Gustafsson ; Shao ). The regions coloured differently correspond to different phases equal to various linear combinations of a and b, where a and b are the phase differences caused by the path length difference in the interferometer loop for the excitation and emission wavelengths, respectively. In reality, the dashed lines in (C) and (D) coincide, i.e. they represent the same lateral frequency.

Because the mth-order component of the I5S raw data is a subregion of the sample's Fourier transform—corresponding to the support of O shifted by mp (Eq. (2))—multiplied by O, this component's phase map is the addition of the sample's and the OTF's phases. Furthermore, components of different orders partially overlap with each other in frequency space (Fig. 3D). For example, the central line of the m= 1 component (the dashed line in Fig. 4D) should be placed p distance (Eqs (1) and (2)) from the central line of the m= 0 component; i.e. to the dashed line in Figure 4C. If we ignore the portions of the effective OTFs derived from the sidebands of the detection OTF for the moment and move the m= 1 component to its original position in frequency space, as illustrated in Figure 5, the green coloured part of the m= 1 component partially overlaps with the red coloured part of the m= 0 component (shaded areas in Fig. 5). Within this overlap, the raw image's m= 0 and 1 components at a certain frequency k are related by

Fig. 5

When the m= 1 component (Fig. 4D) is placed where it belongs in frequency space, it partly overlaps with the m= 0 component as indicated by the shaded areas. Shown here are only the regions because of the convolution of central band of the detection OTF (Fig. 4A, green) and the central three and two frequencies of the m= 0 and 1 illumination structure, respectively (Figs 2D and 4B).

where and are OTFs that are acquired with zero path length difference and is the mth-order component separated from Eq. (2) (the zero path length difference requirement on can be satisfied even for OTFs acquired with nonzero path length difference, to be discussed in OTF separation and synthesis). Linear regression can then be applied to solve Eq. (3) for a using all the frequencies within the overlap. This seems to be a straightforward method to estimate the path length difference, except that it requires a clean overlap between only the two central band-derived regions of the m= 0 and 1 components pointed to by the arrows in Figure 5. Unfortunately, superimposed on those two central band-derived regions are other signals derived from the sidebands of the detection OTF (Figs 3 and 4C–E), which make Eq. (3) invalid. Hence, we first need a solution for obtaining the uncontaminated overlaps as depicted in Figure 5 before the path length difference can be estimated using Eq. (3).

Estimation based on complementary phase relation

It can be proved (see Appendix) that in I5S, when the two emission wavefronts collected from the two objective lenses are coherently combined at the beam splitter, the phase of the interference fringes at the primary exit port (i.e. the illumination entry port, Fig. 1) of the beam splitter is exactly π different than that at the other exit port (the secondary exit). This results in the following relation between the detection OTFs acquired at the primary and the secondary exits: when the phases are ±b in the sidebands of the former, the phases are ±(b+π) in the sidebands of the latter. Then if I5S effective OTFs are acquired through the two exits, the same π phase difference exists between the two sets of OTFs in those regions because of the convolution of the illumination components with the sidebands of the detection OTF. Furthermore, as an image in frequency space is a multiplication of the object's Fourier transform by the OTF, this π phase difference in those sideband-derived regions is also there between two I5S data sets acquired simultaneously through the two exits for any sample. Consequently, the addition (or subtraction) of these two data sets results in only the central (or side) band-derived regions being nonzero in frequency space, provided that correct intensity normalization between the two images has been applied. If the addition image is used for extracting the overlap shown in Figure 5, the overlap would not be contaminated by the superimposed sideband-derived signals and therefore, the path length difference can be cleanly estimated as described in Eq. (3).

To implement the above idea, a second imaging path was added to I5S to capture the interference fringes through the secondary exit of the beam splitter (CCD 2 in Fig. 1). A pair of I5S point spread functions (PSF) was acquired simultaneously by the two CCDs and the rotationally averaged effective OTFs were generated (Gustafsson ; Shao ). The summations and subtractions of these two sets of OTFs are shown in Figure 6, which, as expected, contain mostly the subregions derived only from the central band and sidebands of the detection OTF, respectively.

Fig. 6

The effective I5S OTFs were acquired through the primary and secondary beam splitter exit and rotationally averaged, and their additions and subtractions, for m= 0, ±1 and ±2, are shown in (A)–(C) and (D)–(E), respectively. The regions derived from the sidebands of the detection OTF are mostly missing in the additions, although those from the central band are missing in the subtractions, as theoretically predicted.

To estimate the path length difference in each data set, a pair of raw data sets was acquired simultaneously by the two cameras. The image pair was then computationally aligned with each other to compensate the residual misalignment between the imaging paths. The misalignment, mainly because of lateral shift, rotation and magnification, only needs to be calibrated once every few months using scattered beads samples. The result of the calibration, a two-dimensional affine transform matrix, was saved for aligning the image pairs in subsequent experiments. The intensity ratio between the image pair after background subtraction was also calibrated and saved for future use. After alignment and intensity correction, each raw image was separated into the five information components (Eq. (2)), each corresponding to a different m of Eq. (2) (Shao ), and the m= 0 and 1 components of the images acquired by one camera were added to their counterparts acquired by the other, resulting in the new m= 0 and 1 components with little sideband-derived contamination. The overlap between the red subregion of the m= 0 component and the green subregion of the m= 1 component (Fig. 5) was then extracted and used for the estimation of the path length difference based on Eq. (3).

OTF separation and synthesis

Given the estimated path length difference, we still need a scheme for image reconstruction to take advantage of it. Here we introduce such a scheme based on the fact that I5S effective OTFs are the convolution of the detection OTF O with the 1D axial illumination structures (Eq. (2)). Suppose a measured I5S OTF can be decomposed into O and . Once the path length difference is estimated from a raw image, then it is trivial to modify the phases of O and by multiplying to the components of nonzero phases (Fig. 4B) and to the sidebands of O(Fig. 4A). The phase-modulated O and can then be convolved together to synthesize a set of OTFs that can account for the path length difference specific to the raw image.

To separate I5S OTFs into two convolving parts, one of the parts has to be known. There is a way to measure the detection OTF and I5S effective OTFs almost simultaneously as long as the mirror M3 in Figure 1 also transmits partially (20% is what we used) so that a conventional illumination path can be introduced through that mirror. With this new set-up (Fig. 1), the 3D I5S effective PSF and detection PSF can be acquired in an interleaved manner by alternating between the conventional and the structured-illumination path at each defocus step. We can then obtain rotationally averaged I5S effective OTFs and detection OTF from these PSFs.

The OTF separation problem is formulated as solving for in given O(k) and O(k). Because is a 1D function, this problem is equivalent to solving it in a series of 1D deconvolution problems for each lateral frequency kxy. Linear convolution can be transformed into a circular convolution, which can then be formulated as a vector (in this case the unknown ) being multiplied by a cyclic matrix, if the vectors have a sufficiently short support or are padded with enough zeros (Bertero & Boccacci, 1998, Chapter 2.7). A stack of linear equations, one for each kxy, can thus be formulated into an overdetermined system for solving . The axial illumination structures thus solved and the measured detection OTF are still modulated by an unknown phase because of the unknown path length difference during data acquisition. To estimate this path length difference, the same summation–subtraction and overlap extraction steps described above can be applied to the I5S effective OTFs measured by the two cameras. The estimated path length difference can then be removed from O and and, therefore, we can synthesize the I5S effective OTFs with zero path length difference ( in Eq. (3)) by convolving the zero phase-modulated O and , which are needed to correctly estimate path length difference in each I5S data set (Eq. (3)). For each of the three lateral pattern orientations, data-specific I5S effective OTFs are synthesized based on the estimated path length difference as described above and then used in reconstructing the normal I5S data set acquired through the primary beam splitter exit.

Results

Figure 7 shows the results of separating I5S illumination structures from the measured I5S effective OTFs as discussed above. The three curves in Figure 7 are the amplitudes of I5S illumination structure versus k in frequency space for m = 0, ±1 and ±2 (Eqs (1) and (2)).

Fig. 7

I5S illumination structure axial profiles in frequency space, obtained by deconvolving measured I5S detection OTF from the I5S effective OTFs. Black, red and blue curves correspond to order m= 0, ±1 and ±2 components, respectively.

Two pairs of dual-camera I5S images of a single-bead sample were acquired. Nonzero path length differences were intentionally introduced when acquiring these data sets. The reconstructions using a standard set of effective OTFs taken with little path length difference present show the expected ringing artefacts in the axial intensity profile through the bead centre (Figs 8A and C). The different polarities of the asymmetry in those plots indicate either positive or negative path length difference in the loop. For each data set, we performed phase difference estimation (the results are 0.15 and –0.9 radians for data set shown in Figs 8A and C, respectively) and data set-specific OTF synthesis as described above, and then reconstructed the images acquired through the primary beam splitter exit using the synthesized OTFs. The results are also displayed as axial intensity profiles in Figures 8B and D, which exhibit much less ringing than Figures 8A and C.

Fig. 8

Results showing the effects of path length difference estimation and correction. (A) and (C) The axial profiles of I5S reconstruction of a single bead, imaged with intentional path length differences estimated to be 0.15 and −0.9 radians, respectively. (B) and (D) Results using synthesized OTFs based on the estimated path length differences corresponding to (A) and (C), respectively. There are significantly less ringing artefacts in (B) and (D) than in (A) and (C).

Discussion

We introduced a method to estimate the path length difference present in each I5S data set with the assistance of a second camera recording fluorescence emission's interference fringes through the secondary exit of the beam splitter. The estimation method exploits the π phase difference relationship between the imaging interference fringes at the primary and secondary beam splitter exits and the well-defined Fourier space phase maps for all the information components constituting an I5S data set. Another way to think about the π phase difference is to recall that when the primary beam splitter exit has constructive interferences information, the new secondary beam splitter exit will have destructive interferences information. With the estimated phase difference, a data set specific set of OTFs can then be synthesized, because I5S effective OTFs are separable into a detection OTF convolving with the axial illumination structures, and used for reconstructing the data set. We have used a single bead imaging experiment to prove the principle. Reconstruction using a synthesized, as opposed to a standard, set of OTFs successfully removes most of the axial ringing artefacts that are because of an unknown path length difference. One major difficulty of I5S or any other interferometric microscope is the strict requirement on path length difference adjustment. Relaxing this requirement will significantly enhance the real-world applicability of I5S to biology. The basic principle demonstrated in this paper lays the groundwork for a scheme that can achieve such a goal. It does not, however, affect other factors limiting the application of I5S to biology, including the requirement of refractive index matching among the mounting medium, the immersion medium and the sample itself, and the requirement that the two objective lenses are closely matched in their optical properties. In our demonstration, the fluorescent bead was mounted in the same medium as the high refractive index (1.51) immersion medium.

As mentioned earlier, the image pair recorded on two cameras can also be used simply for improving the signal to noise. This can be accomplished by including in the generalized Wiener filter (Gustafsson ) the additional terms corresponding to the images and OTFs acquired by the second camera once the image pair is aligned.