Temporal pixel multiplexing for simultaneous high-speed, high-resolution imaging.

We introduce an imaging modality that, by offsetting pixel-exposure times during capture of a single image frame, embeds temporal information in each frame. This allows simultaneous acquisition of full-resolution images at native detector frame rates and high-speed image sequences at reduced resolution, without increasing bandwidth requirements. We demonstrate this method using macroscopic and microscopic examples, including imaging calcium transients in heart cells at 250 Hz using a 10-Hz megapixel camera.

Conventional cameras capture images at a fixed spatial and temporal resolution. Events that occur faster than the detector integration time (ti) appear blurred (if moving), as an average light intensity (if changing intensity while stationary), or a combination thereof. We propose a temporal pixel multiplexing (TPM) paradigm for imaging that allows simultaneous high-speed and high-resolution image capture on a single detector1.

The method is based on subdividing a charge coupled device (CCD) array into m non-overlapping groups of n pixels (Fig. 1, Supplementary Fig. 1). Individual pixels in each group capture their part of the imaged scene sequentially so that at any instant, m spatially dispersed pixels are subsampling the image with an exposure duration te = ti × n−1. Since all detector pixels are exposed at some point during ti, the resulting full frame image is qualitatively similar to that of a conventionally acquired high-resolution image (Supplementary Fig. 2). In addition, n subframes can be extracted from co-exposed pixels in all m groups. The spatial resolution of each subframe is n-times lower than that of the full frame, but the image sequence has an n-times higher temporal resolution than can be achieved using the conventional image capture mode of the same detector. Thus, TPM embeds high-speed temporal information in an image, while increasing neither memory requirements nor the intrinsic frame rate of the camera. Static regions of the high resolution image are unchanged, while high-speed information is multiplexed into image regions that would normally appear blurred.

Figure 1

Schematic of TPM principle. (a) A detector (6 × 6 pixels) with a frame rate of 1 fps is organized into m = 9 exposure groups, each consisting of n = 4 light-detecting elements (arranged in a 2 × 2 grid pattern; see solid outline). Here, each numbered element corresponds to one pixel (in general, exposure elements can consist of multiple neighboring pixels). (b) Each pixel with the same ID in all exposure groups integrates light at the same time, for a quarter of the detector’s total exposure time (i.e. for te × n−1, here 0.25 s). The full-resolution frame is read out and digitized after 1 s. (c) A subframe is extracted from all pixels exposed over the first 0.25s (i.e. consisting of all m pixels with ID #1), a second subframe exposed from 0.25 to 0.5s is extracted from all m pixels with ID #2, and so on, to give four sequential 9-pixel frames, collected at 4 fps (i.e. n-times the inherent full-frame rate). The full resolution frame is simultaneously obtained.

A principal design requirement is that cohorts of spatially distinct subregions of the imaging chip are sequentially exposed for periods te shorter than ti of the camera. Subregion exposure control can be implemented on the illumination side, using patterned light techniques such as applied to super-resolution2 and confocal3 imaging. Alternatively, light can be blocked on the detector side, as done in dynamic filtering4 and single pixel camera5 applications. Finally, TPM could be incorporated into new CMOS sensor designs by offsetting pixel sample-and-digitization times. Here we present a prototype that uses a digital micro-mirror device6 (DMD) to transiently expose subregions of a CCD (see Methods).

TPM functionality is illustrated in Fig. 2. A high-resolution image of a macroscopic scene was captured using a 25 fps megapixel camera, while the DMD cycled through a sequence of 16 mirror patterns during each full frame acquisition (ti = 40 ms; te = 2.5 ms), yielding a final frame rate of the embedded lower-resolution image sequence of 400 fps (Fig. 2b). Since all pixels are exposed during ti, the unprocessed frame captures high spatial resolution details of the imaged scene. After extracting and combining simultaneously exposed pixels, the blurred regions in the high-resolution low-speed image are resolved as an image sequence of a drop of opaque liquid (milk) falling into a beaker filled with water (Fig. 2b, Supplementary Fig. 3, and Supplementary Video 1).

Figure 2

Functional TPM prototype. (a) A diagram of the prototype, which can capture microscopic (black lines, via L1) or macroscopic images (gray lines, via L3). (b) High resolution frame (right panel, 1,000 × 1,000 pixels, 25 fps) and 16 embedded high-speed sub-frames (left panels, 250 × 250 pixels, 400 fps) showing a milk drop falling into water, captured via L3. (c) Cardiac cell loaded with a calcium sensitive dye, imaged with a megapixel camera at ti = 100 ms. The DMD is programmed to sequentially expose a 5 × 5 pattern of exposure elements (where each element consists of 25 neighboring pixels), for te = 4 ms. Bright points are mirrors toggled to the ‘always on’ position, for alignment purposes. (d) Close up of cell regions during an action potential, showing intensity changes mapped to detector location (after background frame subtraction). Every exposure group (such as the one outlined in red) measures fluorescence intensity from a 25 × 25 pixel area. (e) Intensity vs. time plot for one of the exposure groups, extracted by sequentially plotting the intensity recorded by each of the 25 elements. Dark points correspond to pixel values in (d), gray points are extracted from frames before and after. (f) Average intensity values for all pixels in (d) clearly resolve the whole-cell calcium transient shape with improved SNR. Analysis of the calcium dynamics show that a normal beat is followed by one with an early and delayed after-contraction. Scale bars 4 mm in 2b, 6 μm in 2c.

The method is also useful for measuring intensity-coded processes, such as encountered in functional imaging of cardiac7 and neuronal8 cell-networks. Here, rapid ion concentration and membrane voltage changes can be mapped to light intensity using fluorescent probes. Since high-resolution detectors suffer noise and bandwidth limitations at high frame rates, and as image intensification technology imposes dynamic range limitations9, researchers typically rely on high-speed, high dynamic range, low-resolution detectors, ranging from 16×16 photodiode arrays7 to specialized low resolution CCD8 and CMOS10 cameras. Additional high spatial resolution cameras are required to interrelate structure and function.

We conducted a proof-of-principle microscopic fluorescence mapping study in rat cardiac myocytes. Cells were loaded with the calcium-sensitive dye Rhod-2 and imaged at 40× using an inverted microscope. High-resolution cell structure images (Fig. 2c,d) were captured while simultaneously measuring functional whole cell calcium transients at high speed (Fig. 2e,f). The effective frame rate of calcium transient capture is more than an order of magnitude greater than that of the camera itself (250 fps vs. 10 fps), and significantly exceeds the camera’s maximum region-of-interest or on-chip binning frame rates. Furthermore, despite light losses caused by transient pixel exposure, the fluorescent signal-to-noise ratio (SNR) is high, and equivalent to that of specialized high-speed low-resolution devices, such as photodiode arrays, dedicated high-speed CCDs, or photomultiplier tubes.

TPM is a novel contribution to advanced imaging techniques that embed information in images by modulating light. Lens arrays11 and shaped apertures12 are used to partially encode light fields in images, allowing users to change viewpoints or focal planes after acquisition. Patterned light has been used to embed 3D information in images13 and perform optical sections3 in microscopy applications. High-speed motion has also been targeted: coded exposure photography14 embeds spatial frequency detail in images by globally modulating exposure during acquisition, allowing motion blur to be algorithmically corrected. However, coded exposure methods depend on a known and reproducible motion path, and since pixels are repeatedly exposed during each frame, intensity changes in stationary objects (i.e. cells) cannot be measured. In contrast, TPM exposes pixel subsets at discrete times using one detector, a strategy that has previously been implemented using multiple detectors in high-speed dense camera arrays15.

TPM offers several advantages over conventional high-speed imaging methods. Not only does it provide simultaneous high-resolution and high-speed data sets using just one detector, but data throughput requirements are the same as that of a single high-resolution image sequence. This allows slow-scan cameras with their known advantages in terms of cost, SNR, and dynamic range9,11 to be used for high-speed imaging tasks. The system is modular (it can be added-on to existing detectors) and highly flexible: our present implementation allows for non-rectangular ROI shapes (pixel groups in the shape of cells, for example) and supports different frame rates for different areas of the CCD, which is not possible using conventional detectors.

Methods

Cell isolation

Rat ventricular myocyte isolation has been described in detail elsewhere16. In short, hearts were initially Langendorff-perfused with a modified Tyrode solution containing in mM: NaCl 128.0, KCl 2.6, CaCl2 2.0, MgSO4 1.18, KH2PO4 1.18, HEPES 10, Taurine 20, Glucose 11, and heparin (5,000 units l−1), pH = 7.4; followed by cardioplegic solution (K+ 20 mM; no heparin or calcium) to induce cardiac arrest. Upon induction of arrest, the perfusate was changed for 12 min to enzyme-containing solution (0.24 mg ml−1 BlendZymes III; Roche). The tissue was then harvested, minced, and gently agitated in 10 ml enzymatic solution. This solution is collected every 5 min (5 repeats), filtered, and centrifuged for 1 min at 18 g (Precision Durafuge 100). The supernatant is discarded, and the cell pellet re-suspended in cell culture medium (DMEM, Sigma). Cells were kept at room temperature.

Calcium Fluorescence

Cells were loaded with the calcium sensitive dye Rhod-2 for 20 min (Rhod-2AM 5 mM, Invitrogen), washed in Tyrode, and imaged after 10 min to allow for esterase-mediated dye cleavage. Dye-loaded cells were transferred to a glass-bottomed dish and imaged using an inverted microscope (Nikon Diaphot, 1.3 NA oil immersion lens) with a Rhod-2 fluorescence cube (Chroma, excitation 540/25 nm band pass, dichroic 565 nm, emission 605/655 nm band pass, excited with an Xe-Hg light source, Optiquip).

Optical path

The optical path is aligned so that an imaged scene is first focused on the DMD17 array, which in turn is focused on the camera. The camera and mirror array are positioned to allow imaging from the side port of a microscope (Nikon Diaphot, 40× oil immersion lens), or for macroscopic imaging using a suitable projection lens (e.g. lens L3 in Fig.2a). In both cases, light is focused to form an image on the DMD array. Light from the microscope side port is directed to the DMD using a plane mirror to an achromatic lens pair (100 mm focal length, Thorlabs). The Texas Instruments XGA+ (1,400 × 1,050 mirror) DMD array (Vialux) is mounted in a custom housing which is attached on a 6 axis micro positioning stage (MAX312 and APY002 from Thorlabs, M-TGN120 from Newport) for alignment of mirrors with camera pixels. Light from the DMD is focused on a camera (GC1380H, Sony ICX285 sensor, 1,360 × 1,024 pixels, Prosilica) using a 100 mm SLR lens (f/2 Makro-Planar T* Zeiss) so that a single mirror is in register with each pixel in the CCD (excess DMD mirrors are not employed for projection). For the experiments shown in Fig. 2, the center 1,000 × 1,000 pixel/mirror pairs are used. The camera and lens are coupled using Thorlabs 30 and 60 mm cage components with a rotation mount for camera angle adjustment (Thorlabs CRM1P), and positioned using a linear stage (Newport M423).

Alignment and synchronization

The camera and mirror array are aligned visually using custom software that allows live zoom display of 50 equidistant 10×10 mirror areas, each set so a single mirror is in the ‘on’ position, dispersed over the mirror array surface. Once aligned, the mirror array is set to repeatedly cycle through frame series of n mirror patterns, where each pattern is active for te ms. The camera is set to trigger off every nth DMD pattern by setting the camera frame duration to te × n + tr, where tr is the frame readout time. tr is small (0.184 ms for the GC1380H) compared to camera integration time (ti = te × n) and has thus been ignored in the schematic description of TPM elsewhere. Frames are continually saved to hard disk for later analysis.

Exposure patterns

The mirror array can be programmed to cycle through a series of binary images, so that each mirror acts as a pixel level shutter which can open and close faster than the camera exposure time ti. Individual mirrors are classified for programming purposes as being part of exposure elements, exposure groups, and subframe elements (Supplementary Fig. 4). Since mirrors are pixel-matched to the detector, a corresponding pattern is created on the image, and pixels and mirrors are functionally equivalent. Exposure elements (outlined in red in Supplementary Fig. 4) refer to locally adjacent mirrors that are programmed to have the same on/off state. Here, each exposure element consists of 4 mirrors (in contrast, in Fig. 1 each element corresponds to one mirror / one pixel). An exposure group (outlined in yellow in Supplementary Fig. 4) consists of a set of locally adjacent exposure elements that cycle through a sequence of on/off states, so that CCD pixels in-register with mirrors in the exposure group collect light at different times during ti. Mirrors in different exposure groups with the same ID (shown by black numbers in Supplementary Fig. 4) behave identically. Simultaneously exposed pixels are termed subframe elements, and can be extracted, conserving their spatial interrelation, to form a subframe (e.g. pixels forming the seventh subframe are shown outlined in blue). In the diagram in Supplementary Fig. 4, a (hypothetical) 12 × 12 mirror array is organized into four exposure groups (m = 4), each containing nine 4-pixel exposure elements (n = 9). Each subframe consists of four spatially dispersed co-exposed four-pixel elements (with the same ID) providing a temporal resolution that is n = 9 times better that the detector’s intrinsic frame rate.

In addition to the sequential patterns (Fig. 1, Supplementary Fig. 4), exposure patterns can be randomized to reduce potential effects of regularly spaced exposure elements in the high resolution image. Here, each exposure group has the same dimension and number of exposure elements as before, but the exposure sequence of individual elements (with the same ID) is shuffled using a random seed. This way, co-exposed elements do not display a periodic spatial pattern, and the high resolution image appears qualitatively more similar to a conventional image. High resolution images captured using a conventional technique, periodic exposure patterns and randomized exposure patterns are compared in Supplementary Fig. 2a-c.

For macroscopic scene imaging (Fig. 2b), the mirror array is divided into a grid pattern of 62,500 (1,000,000 × 16−1) exposure groups, each comprising 4 × 4 adjacent mirrors (here, exposure element size=1 pixel). Mirrors within each exposure group are programmed to reflect light to in-register pixels on the camera so that each pixel is exposed for 2.5 ms in a sequential fashion, and each 1,000 × 1,000 pixel frame is read out and stored after 40 ms.

For fluorescence imaging (Figure 2c-f), the mirror array is divided into 1,600 exposure groups, each comprising 625 (25 × 25) adjacent mirrors. In order to improve contrast, mirrors are further grouped into exposure elements of 25 (5 × 5) adjacent mirrors (pixels on the detector). Each exposure group consists of a 5 × 5 pattern of these multi-pixel elements. Each exposure element sequentially reflects light to 5 × 5 groups of in-register pixels for 4 ms, and each 1,000 × 1,000 pixel frame is read out and stored after 100 ms. The corner mirror in each exposure group is fixed to the ‘on’ position for some runs to aid alignment (Figs 2c,d).

Post-processing

Pixels exposed over the same interval are grouped in software to generate a higher-speed lower-resolution image sequence. Since the mirror and pixel dimensions are small (13.68 μm2 and 6.45 μm2 respectively), and image focus can be suboptimal, pixels integrate stray light from adjacent regions. In practice, the maximum contrast ratio at the individual pixel level (ratio of pixel intensity to that of its nearest neighbor) is approximately 4. For macroscopic scene imaging, where each pixel cluster consists of only one pixel, images are optionally sharpened prior to decoding with 3 × 3 Laplacian kernel.

For the fluorescence data shown in Figs 2c-f, sharpening is not used. Instead, the pixels along the border of each 5 × 5 pixel exposure element are discarded, and the remaining pixel values with identical ID are averaged for each exposure element. Contrast ratio using this scheme is high (> 1,000 : 1). Average cluster values are plotted as a function of time (4 ms intervals). In order to remove the effects of high spatial frequency structures on the analysis, the first high resolution image is used as a background image, which is subtracted from each subsequent high-resolution image before extracting the high speed calcium fluorescence time series. The time series data in Fig. 2e and Fig. 2f are unfiltered.

Image processing

High resolution TMP images contain high spatial frequency components that can lead to visual artifacts if reproduced or viewed at less than full resolution. The full resolution image in Fig. 2b was processed using a three pixel radius Gaussian blur to avoid these potential effects. The unaltered full 1000 × 1000 pixel image is included as Supplementary Figure 3.

Limitations

The method embeds high-speed information into static frames by transiently exposing pixels to light in a controlled fashion. As a consequence of this process, the majority of the light is deflected away from the sensor by the DMD. However, since each pixel collects light with high efficiency over its exposure period, signal quality can be high. Nonetheless, as light destined for neighboring elements in each exposure group is discarded for a fraction equivalent to (n−1) × n−1 of detector integration time, the method will be of low utility in highly light-limited situations where spatial pixel averaging (e.g. adjacent pixel binning) may be required to obtain a high quality signal.

Supplementary Video 1 400 fps movie taken using a 20 fps camera.

Two high resolution images (1000 × 1000 pixels, acquired at 25 fps, see Fig 2b for the first image in the high resolution sequence) are decoded into 32 250 × 250 pixel subframes with an effective frame rate of 400 fps, showing a milk drop falling into a beaker of water.