Computational Photon Counting Using Multithreshold Peak Detection for Fast Fluorescence Lifetime Imaging Microscopy.

Time-resolved photon counting methods have a finite bandwidth that restricts the acquisition speed of techniques like fluorescence lifetime imaging microscopy (FLIM). To enable faster imaging, computational methods can be employed to count photons when the output of a detector is directly digitized at a high sampling rate. Here, we present computational photon counting using a hybrid photodetector in conjunction with multithreshold peak detection to count instances where one or more photons arrive at the detector within the detector response time. This method can be used to distinguish up to five photon counts per digitized point, whereas previous demonstrations of computational photon counting on data acquired with photomultiplier tubes have only counted one photon at a time. We demonstrate in both freely moving C. elegans and a human breast cancer cell line undergoing apoptosis that this novel multithreshold peak detection method can accurately characterize the intensity and fluorescence lifetime of samples producing photon rates up to 223%, higher than previously demonstrated photon counting FLIM systems.

Many optical technologies rely on time-resolved photon counting such as time of flight (ToF) imaging,[1] light detection and ranging (LiDAR),[2] and optical microscopy. Specifically, multiphoton fluorescence lifetime imaging microscopy (FLIM) often uses time-resolved photon counting to compute and spatially map fluorescence lifetime. The time between excitation and emission in the fluorescence process follows an exponential decay probability density function, and the mean time between excitation and emission is the fluorescence lifetime.[3] Fluorescence lifetime can be used to investigate various properties within biological samples. Notably, FLIM can be used to examine metabolism in biological samples via reduced nicotinamide adenine dinucleotide and nicotinamide adenine dinucleotide phosphate (NAD(P)H)[4,5] because protein-bound NAD(P)H has a longer fluorescence lifetime than “free” NAD(P)H.[6]

Presently, most photon counting FLIM systems use time-correlated single photon counting (TCSPC), in which analog electronics are used to count and time-tag incident photons relative to the pulsed laser source used for excitation.[7,8] However, TCSPC systems experience a dead time: the amount of time after a photon is counted that a subsequent photon cannot be counted. This is due to both the detector response function and the limited bandwidth of the additional analog electronics. When many photons arrive close together in time, it becomes more likely that photons will arrive while the system is “dead” and these additional informative photons will not be counted, biasing the fluorescence intensity and lifetime measurements, termed the “pile-up effect”. Thus, the dead time of the system determines the maximum acceptable photon rate (photon rate is the percentage of laser excitation pulses resulting in a photon count), above which the distortion due to pile-up in fluorescence lifetime and intensity becomes significant. Many standard TCSPC systems have a maximum acceptable photon rate of around 5–10%, meaning that the fluorescence signal must be attenuated so that 90–95% of laser pulses on average do not result in a photon count.[7,9]

One way to combat dead time is to use array detectors, such as silicon photomultipliers (SiPMs)[10] and single-photon avalanche diode (SPAD) arrays,[11] which are made up of many small single-pixel detectors that can work independently, though each individual pixel detector has a dead time. These detectors can suffer from lower quantum efficiency due to the geometrical limitation on fill factor, which is one barrier to autofluorescence imaging, which generally has a much lower intensity than fluorescent dyes. Regardless, high photon-rate-capable SPAD array systems have been developed and will be important for the future of fast FLIM.[12] As an alternative to multi-pixel detectors, pulse pile-up inspectors can be used to reject data when multiple photons arrive at the detector within a short period, providing a way to combat pile-up, but are still limited by system dead time.[13,14] An additional demonstrated strategy matched the dead time of a SPAD array TCSPC FLIM system to the laser repetition period to improve performance.[15] While these improvements have all contributed to enabling faster imaging, they all still are inherently limited by the dead time of the photon counting hardware employed. Direct pulse sampling, where the output of the detector is directly digitized and used to approximate photon counts,[16] can be used for rapid acquisition of FLIM but has poorer accuracy than photon counting methods due to the lack of single-photon resolution.[17] Time-gated methods, where fluorescence intensity is collected in varying time windows with respect to the laser pulse, can provide fast fluorescence lifetime contrast but has poorer temporal resolution and must employ either multiple detectors or use multiple acquisitions to acquire multiple time-gates.[18−20]

Recent advances in time to digital conversion (TDC)- and field-programmable gate array (FPGA)-based technologies have achieved dead times as low as 0.650 ns, and enabled fast TCSPC[21,22] and digital frequency domain[23] FLIM. However, thus far, the fastest single-detector dead time in a photon counting FLIM system (0.625 ns) has been demonstrated by our research group using computational photon counting on directly digitized photomultiplier tube (PMT) output by implementing a simple thresholded peak detection algorithm called Single-photon PEak Event Detection (SPEED).[17]

This demonstration used a high-bandwidth PMT, transimpedance amplifier, and fast digitization at 3.2 GS/s to collect data, and classified any local maxima (a digitized data point with a voltage greater than the points immediately before and after it) that was above the empirically determined single-photon threshold as a photon count using GPU-accelerated real-time processing. Similar approaches using fast digitization have previously been demonstrated for FLIM[24−26] and secondary electron counting in scanning electron microscopy.[27]

Here, we expand upon previous methods with higher bandwidth detection electronics and explore the implications of using a hybrid photodetector (HPD) versus using a PMT. PMTs have high gain variability due to the multiple dynode amplification, in which the randomness associated with signal amplification at each dynode is compounded. HPDs have a two-step amplification process consisting of electron bombardment and then avalanche gain, which leads to low gain variability and a more linear dependence of signal on the number of incident photons.[28,29] The signal created by a PMT when a single photon arrives at the detector is highly variable and cannot be easily distinguished from the signal created when multiple photons arrive nearly simultaneously; however, when using an HPD, single- and multiphoton responses can be distinguished.[28−31] Although this principle is known, to our knowledge, it has not previously been utilized in computational photon counting methods to count multiple photons arriving at the HPD within the detector response time.

Results and Discussion

In response to the arrival of one or more photons, HPDs and PMTs produce waveforms that have peaks of various heights. To examine the differences between HPD (R10467U-40, Hamamatsu) and PMT (H10721-210, Hamamatsu) photon peak height distribution, we examined two-photon fluorescence of NADH in concentrations from 1 to 5 mM on our custom FLIM system (similar to our previously published system;[17,32] see Figure S1) using both detectors and digitizing the amplified output at 5 GS/s. The peak height distributions (Figure ) match well with previously experimental[28,31] and theoretical[29] comparisons of HPD and PMT performance. By examining the local minima of the HPD and PMT peak height distributions, thresholds for one to five photons were set for the HPD and a threshold for a single photon was set for the PMT (dashed vertical lines, Figure ). These lines delineate whether a peak will be counted as 0, 1, 2, etc. photons. Inevitably there is some overlap between expected peak heights for different numbers of incident photons on the detector, which is one source of error for this method that is not present in traditional single-photon counting systems when photon rates are kept very low.

Figure 1

Peak height distribution for hybrid photodetector (HPD, blue) and photomultiplier tube (PMT, green) for 1–5 mM NADH using 30 mW of power incident on the sample. Dashed lines delineate between peak heights that arise from different numbers of incident photons on the detectors. The negative PMT output values were inverted for easier comparison.

To acquire time-resolved photon counts using computational photon counting, a standard optical setup for a two-photon fluorescence microscope was used and the fluorescence was collected in the epi-direction by an HPD (Figure S1). The detector output is amplified and directly sampled at 5 GS/s (Figure a), computationally converted into photon counts using GPU-accelerated processing[17,32,33] (Figure b), compressed into one 25 ns time block for each pixel composing of the average photon counts over two laser periods (Figure c), and then interleaved and shifted based on the time bin within the line of data that has the maximum value (Figure d). Digitization at 5 GS/s leads to interleaved sampling since the laser period (12.5 ns) is not a multiple of the sampling period (0.2 ns), so data can be averaged into a time period of twice the laser period (25 ns) and then interleaved to have 0.1 ns time bins over a 12.5 ns period. Interleaved sampling has similarly been applied for direct pulse sampling FLIM.[34] For fast and convenient lifetime calculation, phasor analysis[35] is applied to photon counts to determine fluorescence lifetime and phasor components g and s (Figure e).

Figure 2

Data processing steps for a single pixel of data using Single- and multiphoton PEak Event Detection (SPEED). (a) Raw data: HPD output that was amplified and then directly digitized at 5 GS/s. (b) Photon counts based on the multiple threshold model for the HPD. (c) Photon counts averaged into twice the laser period, 25 ns. (d) Photon counts interleaved and circularly shifted to have the maximum value at t = 0 ns. (e) Photon counts (black), and sine (red) and cosine (blue) multiplication factors that are used to calculate g and s components for phasor analysis.

The intensity linearity and fluorescence lifetime accuracy at different concentrations of NADH were examined using one to five photon thresholds for the HPD (Figure ) to determine the optimal number of thresholds. Using five photon thresholds provided the best intensity linearity (Figure a) and consistent lifetime estimation from 0.005 to 50 mM NADH (Figure b). Using five photon thresholds also provided accurate lifetime estimation for fluorescent dyes such as Rhodamine B and Fluorescein (Figure S2) and provided a better fit to a Poisson distribution of photon arrivals than a single photon threshold (Figure S3).

Figure 3

Intensity linearity and fluorescence lifetime consistency acquired using HPD for different concentration of NADH with constant excitation power. (a) Intensity linearity given in calculated photon rate as a function of NADH concentration. The black line represents a linear fit for NADH concentrations 0–10 mM and associated photon rates. (b) Calculated fluorescence lifetime of NADH over the same range of concentrations.

Our previous characterization of the performance of the SPEED algorithm using a PMT calculated the maximum acceptable photon rate as the upper limit where less than 10% of photon counts appear to be missed, and fluorescence lifetime was consistently estimated within 10% error.[17] Using the same metric, accurate intensity results are acquired using the HPD at a maximum NADH concentration of 20 mM, which had a counted photon rate of 223% with an estimated 9.75% loss of photon counts from a linear fit (black solid line, Figure a) calculated based on photon rate from lower concentrations. This is higher than our previously published maximum photon rate calculated using a PMT, of around 206%.[17] Using the HPD, we can confidently classify high-amplitude peaks as multiple photon counts. However, there is still pile-up experienced due to the nature of peak detection, which requires unique peaks, meaning that if the directly acquired signal does not decrease in the digitized point immediately after a photon arrives at the detector due to a high photon rate, photons will still be missed. The dead time is approximately twice the sampling period: 0.4 ns (Figure S4), as opposed to the previously achieved dead time of 0.625 ns achieved using digitization at 3.2 GS/s.[17] The theoretical maximum possible measurable photon count rate is the inverse of the dead time, which is 2.5 giga counts per second, though of course this is not practical and the performance of the system to characterize fluorescence lifetime at a rate that high would be poor. Future work could incorporate dead time corrections and compensations[36−38] to further improve results and achieve better accuracy at high photon rates.

In contrast to the small improvement in maximum acceptable photon rate, the fluorescence lifetime consistency across a wide range of photon rates is substantially improved compared to previous results. Our previously published system accurately estimated NADH fluorescence lifetime for photon rates of approximately 6–206%.[17] Now, with improved bandwidth and the multithreshold model for HPD photon counts, NADH fluorescence lifetime is accurately estimated for photon rates from 0.03 to 419% (Figure b) using five photon thresholds. Interestingly, using three photon thresholds showed more consistent lifetime estimation at high photon rates, likely due to both underestimating the maximum number of photons arriving within a time bin and underestimating subsequent photons due to pile-up. Finer temporal resolution and slightly shorter dead time are highly impactful in fluorescence lifetime estimation since the fluorescence lifetime of NADH in solution is short. With a fluorescence lifetime of only ca. 0.3–0.4 ns,[6] the probability that multiple photons will arrive in consecutive time bins is larger for 0.3125 ns time bins than for 0.200 ns time bins, leading to the superior performance in lifetime estimation for our higher bandwidth system. Additionally, fluorescence lifetime estimation at low photon rates is drastically improved using all of the data acquired in each line of the image (instead of each pixel of the image, as in previous demonstration[17]) to determine the time bin that is synchronized with the excitation pulse (Supplementary Note 1, Figure S5).

To characterize detector performance, we compared the single-photon temporal response by averaging the directly digitized signal of thousands of single- and multiphoton waveforms (Figure a), and the impulse response function in photon counts (Figure b) as the resultant timecourse of second harmonic generation (SHG), which unlike fluorescence is an instantaneous process. The waveform of the HPD response to photons had a full width at half-maximum (FWHM) of 0.630 ns (Figure a), which corresponds well to the manufacturer-specified 0.6 ns pulse width. The waveform of the PMT response had an FWHM of 1.613 ns (Figure a), similar to our previous estimation of 1.617 ns, which was collected using slightly lower bandwidth electronics.[17] However, when comparing the impulse response function using counted photons (Figure b), computed as the incident photon counts obtained over time for SHG, the HPD and PMT performed similarly, with FWHM values of 0.226 and 0.230 ns, respectively. The PMT had a small number of counts after the main response, which could be due to some signal reflection between the PMT, amplifier, and digitizer, although it was not observed in the HPD data, or could be due to afterpulsing of the PMT.[39,40] These results were obtained at relatively low photon rates, with a mean pixel photon rate of 0.32% (max 13.5%) for the HPD and 0.33% (max 7.5%) for the PMT to eliminate the effect pile-up would have on the estimated response functions; similar values were obtained using both detectors with higher photon rate data as well. Both the HPD and PMT are thus good detector candidates for computational photon counting using peak detection. Many traditional photon counting methods rely on discriminators, which have reduced capability when the detector waveform is longer. However, when using peak detection, the temporal impulse response computed using photon counts is only 0.04 ns different for these two detectors since peaks can be discriminated even if two photons arrive within the FWHM of the detector waveform. Though as previously stated, the HPD has the benefit of better performance due to the ability to clearly characterize arrivals of multiple photons within one time bin.

Figure 4

Impulse response functions using both an HPD (blue) and a PMT (green). (a) Average single-photon waveform, calculated using direct pulse sampling methods. (b) Impulse response function using computational photon counting, determined as the resultant temporal spread of signal from SHG, which should produce an instantaneous response.

To demonstrate the capabilities of a fast FLIM system equipped with high maximum photon rate detection, we acquired label-free images of live, nonimmobilized C. elegans at room temperature using 2.5 μs pixel dwell times, with only 200 laser pulses per pixel per frame and no frame averaging (Figure ). With 10–15 mW of incident power on the sample and two-photon excitation at 750 nm, live C. elegans produced photon rates up to over 600% within a single pixel with approximately 3% of pixels within the C. elegans producing photon rates over 100% (Figure c). Fluorescence was acquired within a 75 nm spectral band centered at 450 nm to target NAD(P)H autofluorescence. Additional videos show the motion of a mature C. elegans egg (Figure a, Video 1) and two C. elegans moving around (Figure b, Video 2). The NAD(P)H two-photon autofluorescence lifetime in C. elegans has been previously imaged and characterized using FLIM systems with significantly slower acquisition, such as with 160 μs pixel dwell time and using the average of eight images where C. elegans were immobilized by the addition of polystyrene beads.[41] In addition, human breast cancer cells produced measured photon rates that increased almost 10-fold in seconds during apoptosis (Figure S6, Supplementary Note 2), which shows that high maximum photon rate imaging capabilities are useful for high-dynamic-range processes in living samples.

Figure 5

NAD(P)H autofluorescence in live C. elegans over 135.7 s. Single frames acquired with 2.5 μs pixel dwell time are shown with no frame averaging. (a) Mature egg in its final stages before hatching (256 × 256 pixels). (b) Two C. elegans in motion (512 × 512 pixels). Scale bar: 40 μm. (c) Histogram of nonzero photon rates within pixels of all frames of both the mature egg from (a) (teal) and the two hatched C. elegans from (b) (yellow).

Although useful frames can be acquired with just 2.5 μs pixel dwell time, the transferring, processing, and saving of multiple gigabytes of raw data acquired for each frame presently limit the imaging speed. GPU-accelerated processing enables saving processed data (∼0.15 Hz) 2.5 times faster than saving raw data (∼0.06 Hz). FPGA-based peak detection methods could be used in future work to reduce the volume of data and enable higher frame rates that can fully utilize the capability to collect many photon counts with only 2.5 μs pixel dwell time. Another consideration is the number of imaging channels. Due to the limitation on data transfer speed, we only performed computational photon counting on one channel from a single detector, although a few digitizers exist that can acquire multiple channels of data at sufficiently high rates for accurate computational photon counting that could enable the use of two channels simultaneously. The use of multiple detectors has been employed previously to improve the FLIM throughput of TCSPC systems.[22,26,42] Furthermore, there are many more detectors than just HPDs and PMTs that could potentially be used in future work with similar fast digitization and computational photon counting such as electron-bombarded CCDs,[43] SiPMs,[10] SPAD arrays,[11] and ultrafast CMOS cameras.[44]

Faster photon counting technologies enable the ability to study fast biological dynamics and ameliorate some of the challenges and limitations associated with the translation of multiphoton microscopy to clinical imaging, such as long image acquisition times. Presently, portable multiphoton systems have been used for clinical imaging of autofluorescence and multiharmonic imaging.[45,46] Notably, the JenLab MPTflex system has been used for two-photon FLIM of NAD(P)H for in vivo human studies,[45] yet the image acquisition time is significantly longer than what could be achieved using computational photon counting methods and multiple photon thresholds, depending on the emitted photon rate distribution. Samples that only produce low (<5–10%) photon rates would optimally be imaged using TCSPC systems with good temporal resolution, but when the maximum photon rate emitted by a sample is higher, the presented methods allow for faster acquisition while maintaining photon counting accuracy. Thus, enabling photon counting in FLIM for photon rates up to 223% using fast electronics, an HPD, and computational photon counting with a multithreshold model can open up new research, commercial, and clinical possibilities for FLIM.

Methods

FLIM System

The microscopy system used for imaging is described in Figure S1 and is similar to previous setups.[12,19] Briefly, the tunable output of an 80 MHz pulsed laser source (InSight X3+, Spectra-Physics) was used in a laser scanning setup with two galvanometer mirrors (6230H, Cambridge Technologies) and a water immersion, 1.05 NA multiphoton excitation imaging objective lens (XLPLN25XWMP2, Olympus) to excite two-photon fluorescence and SHG, which were collected in the epi-direction using a dichroic mirror (FF665-Di01, Semrock) to separate excitation and emission. Another removable dichroic mirror was used to direct the emitted fluorescence or SHG to either an HPD (R10467U-40, Hamamatsu) or an analog output PMT (H10721-210, Hamamatsu). The signal was amplified by a 20 dB, 2.5 GHz bandwidth amplifier (HSA-X-2-20, Femto) and digitized by a high-speed digitizer (ADQ7WB, Teledyne SP Devices). For the determination of photon thresholds, raw data was saved and later analyzed in MATLAB 2021a (MathWorks). Real-time GPU-accelerated processing (using a GeForce RTX 2080, NVIDIA) was used to save processed compressed data after threshold analysis was completed. A custom LabVIEW software and GUI were used to control acquisition. Unless otherwise specified, a laser pulse centered at 750 nm with a 240 fs pulse duration at the sample plane was used for excitation, the FOV was 512 × 512 pixels (180 × 180 μm), and pixel dwell time was 2.5 μs.

Fluorophore and Standard Preparations

NADH powder was dissolved in a 1 M HEPES buffer to maintain a stable pH. Rhodamine B was dissolved in sterile water, and Fluorescein was dissolved in 100% ethanol. SHG was imaged using urea crystals.

C. elegans Sample Preparation

C. elegans growing on agar plates seeded with E. coli were obtained (Carolina Biological Supply Company) and left to grow for 2–4 days. For imaging, a small (<1 cm diameter) portion was cut out of the agar plate and placed in an imaging dish (P35G-0-10-C, MatTek). An average power of 10–15 mW was incident on the sample for imaging.