Creating Virtual Hematoxylin and Eosin Images using Samples Imaged on a Commercial CODEX Platform.

Multiparametric fluorescence imaging through CODEX allows the simultaneous imaging of many biomarkers in a single tissue section. While the digital fluorescence data thus obtained can provide highly specific characterizations of individual cells and microenvironments, the images obtained are different from those usually interpreted by pathologists (i.e., hematoxylin and eosin [H&E] slides and 3,3'-diaminobenzidine-stained immunohistochemistry slides). Having the fluorescence data plus coregistered H&E or similar data could facilitate the adoption of multiparametric imaging into regular workflows, as well as facilitate the transfer of algorithms and machine learning previously developed around H&E slides. Since commercial CODEX instruments do not produce H&E-like images by themselves, we developed a staining protocol and associated image processing to make "virtual H&E" images that can be incorporated into the CODEX workflow. While there are many ways to achieve virtual H&E images, including the use of a fluorescent nuclear stain and tissue autofluorescence to simulate eosin staining, we opted to combine fluorescent nuclear staining (through 4',6-diamidino-2-phenylindole) with actual eosin staining. We also output images derived from fluorescent nuclear staining and autofluorescence images for additional evaluation. Copyright:

INTRODUCTION

Highly multiparametric fluorescence imaging is gaining increased use in research laboratories, particularly those involved in tumor microenvironments and related immunological research, wherein the accurate colocalization of many markers within individual and adjacent cells is important.[123] CODEX imaging[34] is available as a commercial system from Akoya and is one such multiparametric imaging system. Some academic pathology labs, and in particular, hematopathology divisions, are also taking notice, with the possibility of translation to clinical use becoming more likely as the methods are further developed. Pathologists, however, are trained to interpret histology using primarily hematoxylin and eosin (H&E) stained slides and immunohistochemically stained slides using 3,3′-diaminobenzidine and hematoxylin counterstain. Given the critical importance of proper interpretation of histology for clinical care, pathologists are understandably hesitant when asked to interpret histology using unfamiliar staining and imaging approaches. As machine learning algorithms for automated image interpretation in pathology have thus far also focused primarily on H&E-stained slides, transfer of some of that learning is desirable and could be facilitated by coregistered H&E staining in addition to the multiparametric biomarker data.

To incorporate multiparametric fluorescence imaging in clinical use, it is therefore desirable to, in addition to the fluorescence images, have H&E -stained images to compare with fluorescence images.[5] As our work involves a commercial CODEX instrument, we had hoped to perform regular H&E staining on the same tissue used for fluorescence imaging. In the CODEX system, tissue is attached to a glass coverslip rather than a microscope slide. The coverslip is sandwiched between two gaskets (or between a gasket and microscope stage adapter in the updated version) and serves as the bottom of a well, through which fluids are passed during the cyclical staining and imaging process that makes highly multiparametric imaging possible. A coverslip is required since the tissue is imaged on an inverted microscope, and microscope slides would be too thick for the optical objectives used in the usual system. In our experiments, roughly half of the coverslips crack in the process of removing them from the gaskets. Other investigators have reported staining and imaging the coverslips with eosin at the end of the regular CODEX imaging process, while the coverslips are still in place (i.e., still between the gaskets).[3] However, their report lacked further details regarding their approach.

We, therefore, developed our own protocol for creating images using CODEX coverslips that closely resemble traditional H&E-stained tissue sections (“virtual H&E” images, see for examples).[567891011] Similar to another report,[3] we leave the coverslip in place on the instrument and use the fluid well created by the coverslip, gaskets, and coverslip holder to stain with eosin. Since eosin is fluorescent and compatible with our Cy3 filter set,[12] it is simple to image within the existing system. Unfortunately, hematoxylin does not have similar properties. However, since hematoxylin is primarily used to stain the nuclei, we felt that substituting 4′,6-diamidino-2-phenylindole (DAPI) staining for hematoxylin would be sufficient for our purposes (and compatible with our existing DAPI filter set). We then apply relatively straightforward mathematical transformations, similar to those reported elsewhere,[5813] to the fluorescence images of the eosin and DAPI staining to create our virtual H&E images.

METHODS

Eosin and 4′,6-diamidino-2-phenylindole staining protocol

An 8.2% eosin Y working solution was prepared using 390 ml 95% ethanol, 50 ml 1% eosin Y, 5 ml 1% phloxine, and 2 ml glacial acetic acid, from which 300 ml were then mixed with 100 ml water and 2 ml glacial acetic acid to arrive at the final eosin Y working solution. A 50% solution of ethanol diluted in purified water was also prepared before staining. After completion of a normal CODEX imaging run and with the coverslip still in place on the microscope stage, the solution within the coverslip well was removed and replaced three times with 1 ml of 190 proof ethanol, waiting 1 min per wash, and being careful to pipet gently so as to not cause tissue to separate from the glass coverslip. This was then replaced with 500 μl of 190 proof ethanol solution with 0.5 μl of DAPI solution (Akoya Nuclear Reagent; Akoya cat. no. 7000003) and 0.5 μl of eosin working solution added. After incubating for 3 minutes, the solution within the well was then replaced three times with 1 ml of 190 proof ethanol. The well solution was then replaced twice with 1 ml of CODEX buffer (Akoya cat no. 7000001, diluted 10x in water), then replaced with 700 μl of CODEX buffer. The sample was imaged soon thereafter on a Keyence BZ-X800 microscope using DAPI and Cy3 filter sets (Chroma cat. no. 49000 and Chroma cat. no. 49004, respectively) and 0.75 NA Nikon 20x and 0.95 NA Nikon 40x Plan Apo lambda objectives.

Imaging

Imaging was performed on a Keyence BZ-X800 microscope (with built-in camera) with a Nikon 20x Plan Apo lambda NA 0.75 objective using “high resolution” camera mode. Alternatively, a Nikon 40x Plan Apo lambda NA 0.95 objective using “high sensitivity” or “high resolution” camera mode could be used. Using “high sensitivity” camera mode has the effect of binning four pixels (2 × 2) into one, which has the effect of reducing image resolution due to pixelation. In a prior iteration of our method, we created virtual H&E images using 20x magnification and “high sensitivity” camera mode, but the results suffered due to (1) pixelation effects and (2) overly sensitive imaging (i.e., 4x more sensitive due to binning), which made it necessary to use very short camera acquisition times (due to the brightness of the dyes’ fluorescence) and unnecessarily increased sensitivity to possible variations in reagent dilutions.

Conversion of raw fluorescence images to virtual hematoxylin and eosin images

Fluorescence images demonstrate signal intensity that is generally linear with respect to the number of fluorescent molecules present, while normal brightfield imaging of true H&E-stained slides is essentially a measurement of light absorbance by the dye molecules and follows a logarithmic relationship with respect to concentration of dye molecules.[13] We, therefore, modeled the conversion of fluorescence images to absorbance images using the following equations:

Ri, j = cR + (1 − cR) exp (−kDaRDi, j – kEbREi, j)

Gi, j = cG + (1 – cG) exp (−kDaGDi, j – kEbGEi, j)

Bi, j = cB + (1 – cB) exp (−kDaDi, j – kEbBEi, j)

where R, G, and B correspond to the red, green, and blue channels of the resulting virtual H&E image; i and j are pixel indices; c values correspond to a postulated expected minimum amount of light (brightfield out of focus light, etc.) that is transmitted for a maximally stained specimen for the given channels; a and b vectors correspond to the expected decrease in brightfield light intensity for increasing brightness seen by fluorescence microscopy for DAPI and eosin staining, respectively; kD and kE are scalars that are added for convenience for adjusting impact on image output based on DAPI and eosin input images, respectively, and to avoid modifying component values of a and b; D is the DAPI monochromatic fluorescence image; and E is the monochromatic eosin fluorescence image. Constants were adjusted until there was satisfactory correspondence with corresponding H&E images, after which only kD and kE were modified from sample to sample. The resulting RGB channels form a final virtual H&E image. By omitting the eosin image (E), a virtual hematoxylin-only stained image is seen, similar to what is used for conventional immunohistochemistry. Software for the conversion was developed in Python using a Jupyter notebook and a minimal set of modules that included NumPy, pandas, tifffile, and matplotlib. Updated source code and associated example data are available for download at https://github.com/SimonsonLab/VirtualHE_examples.

In addition to the conversion of images obtained through eosin and DAPI combination staining, image conversion was also performed on DAPI and autofluorescence images. This was performed for comparison with the DAPI/eosin staining images. The DAPI/autofluorescence images were collected using an Atto 550 filter set (Cy3 filter set) on the first cycle of the regular CODEX imaging (when autofluorescence is expected to be highest) with equivalent mathematical transformations, albeit with different chosen constants kD and kE.

RESULTS

Virtual H&E images were considered qualitatively good surrogates for regular H&E-stained images, as determined by two board-certified pathologists (PDS and JRF), very similar to that expected by regular H&E staining. An example application is shown in Figure 1, which demonstrates the utility of the virtual H&E staining in helping to identify eosinophilic inclusions/nucleoli in Hodgkin and Reed-Sternberg cells in a case of classic Hodgkin lymphoma [see also Supplementry Figures 1-3]. The protocol was time-sensitive, with gradual loss of eosin staining over time as it dissociated from the tissue specimens and increased generalized background fluorescence from the imaging buffer. Hence, imaging was performed directly after completion of the staining protocol. After imaging with eosin and DAPI, we experimented with tissue clearing by the CODEX instrument, followed by reimaging. Reimaging demonstrated that virtually all of the eosin was removed by the CODEX tissue clearing cycle.

Figure 1

Virtual H&E staining in a case of classic Hodgkin lymphoma after CODEX imaging. Some examples of Hodgkin/Reed-Sternberg cells are circled. The virtual H&E image, created using the DAPI and eosin fluorescence images, helps demonstrate eosinophilic nuclear inclusions, which by CODEX imaging costain with CD20. CODEX image: blue = DAPI, magenta = CD30, yellow = MUM1, cyan = CD20, and white = CD68. Images were captured on a Keyence BZ-X800 microscope with 20x Nikon Plan Apo lambda 0.75 NA objective with “high resolution” camera setting

Virtual H&E images created using DAPI and autofluorescence (captured through an Atto550 filter set using the first CODEX imaging cycle data) were qualitatively very similar to those captured in our post imaging DAPI-eosin staining protocol, but the autofluorescence images showed significant raster scanning photobleaching artifacts that were evident in the final images, more than that seen for eosin staining [see Supplementry Figure 4].

DISCUSSION

As brightfield H&E staining is predicted to remain the gold standard by which pathologists are trained to interpret histology images, presenting imaging data acquired through other methods in a virtual H&E representation is desirable.[567891013] Here, we have presented our method for use with a commercial CODEX imaging system[34] that allows us to use the same tissue section to create a virtual H&E image for direct comparison with the fluorescence imaging of many tissue markers. A similar process could be (and is often) followed for other new imaging systems when the samples cannot be (or cannot conveniently be) stained using H&E and brightfield imaging.

There are multiple methods that could be applied to create virtual H&E images from tissue imaged on a CODEX system. In addition to the method we have presented, another option includes using images of the tissue autofluorescence (collected, for example, using a GFP or Atto 550 filter set) for virtual eosin staining and DAPI fluorescence for virtual hematoxylin staining.[5] This approach eliminates manual staining at the risk of a mismatch between the autofluorescence signal and what would be the true eosin staining pattern. Despite this risk, for many applications, the difference will be small and not affect tissue interpretation. Nevertheless, we determined that actual eosin staining would be preferable, which prompted the development of a staining procedure and associated software. When both imaging approaches were directly compared, we found the eosin-stained imaging to be of higher quality, primarily due to the strong effects of photobleaching seen in the autofluorescence images, which were less pronounced in the eosin images (which is to be expected given the generally short photobleaching lifetime for most autofluorescence).

Given that the transformation to virtual H&E images involves applying exponential-decay transforms, some image detail can actually appear to be lost after transformation, which was particularly noticed in comparing transformed and nontransformed images of cell nuclei. By making the option available to switch between exponential-transformed and linear-transformation images, it is likely that pathologists will quickly adapt to and possibly even prefer having the nontransformed DAPI images, despite being less similar to true H&E images.

An important limitation of this approach is the lack of visualization or altered visualization of pigments present in the specimen since these will be displayed as some mix of H&E coloring. Examples include hemoglobin (appearing as magenta color rather than red) and melanin. For evaluation of pigments intrinsic to the sample, additional tissue sections with traditional H&E staining will likely be required. Furthermore, as eosin might have different fluorescence properties in different molecular environments (e.g., shifted emission spectrum when bound to protein, self-quenching effects for high molecule densities, etc.), we also recognize that, even with using the same molecule, the correlation between brightfield and fluorescence imaging may be more complex than our simple transformation allows, and more sophisticated transformations (e.g., nonlinear or deep learning GAN models) might ultimately yield better virtual H&E images. Finally, the images produced through fluorescence are a combination of eosin fluorescence and autofluorescence, so care must be taken to ensure sufficiently strong eosin staining is present to ensure the autofluorescence contribution is negligible.

In summary, we have presented a straightforward staining protocol and analysis algorithm that should be of interest to investigators who are using CODEX systems and wish to create virtual H&E images of the same tissue section used for imaging biomarkers. The images will be helpful for trained pathologists who are asked to interpret the images, whether for research questions or eventual clinical use.

Financial support and sponsorship

Financial support was provided through the Department of Laboratory Medicine and Pathology at the University of Washington and the Department of Pathology and Laboratory Medicine at Weill Cornell Medicine.

Conflicts of interest

There are no conflicts of interest.

SUPPLEMENTARY MATERIAL

Updated source code and associated example data are available for download at https://github. com/Simonson Lab/.

Virtual H&E imaging of a section of tonsil imaged using ×40 and “high sensitivity” camera setting. (a) 4',6-diamidino-2-phenylindole fluorescence image. (b) Eosin fluorescence image. (c and d) Virtual H&E brightfield image

Second example virtual H&E imaging of a tonsil, ×40 and “high sensitivity” camera setting. (a) 4',6-diamidino-2-phenylindole fluorescence image. (b) Eosin fluorescence image. (c and d) Virtual H&E brightfield image

Virtual H&E imaging of a particle preparation of acute monocytic leukemia aspirate. (a) Virtual hematoxylin image (achieved simply by setting kE = 0), ×40. (b) Vir tual H&E image, &40 magnification. (c and d) CD4 and CD163 CODEX fluorescence images, respectively, obtained at ×20

Virtual H&E images of a tonsil created using autofluorescence image instead of eosin fluorescence image, ×20, “high sensitivity” camera mode. (a and c) 4',6-diamidino-2-phenylindole fluorescence, eosin fluorescence, and virtual H&E brightfield images, respectively. (d) Photobleaching of the autofluorescence signal contributes to artifacts due to raster scanning often being present in final H&E images