Nanophotonic color splitters for high-efficiency imaging.

Standard color imaging utilizes absorptive filter arrays to achieve spectral sensitivity. However, this leads to ∼2/3 of incident light being lost to filter absorption. Instead, splitting and redirecting light into spatially separated pixels avoids these absorptive losses. Herein we investigate the inverse design and performance of a new type of splitter which can be printed from a single material directly on top of a sensor surface and are compatible with 800 nm sensor pixels, thereby providing drop-in replacements for color filters. Two-dimensional structures with as few as four layers significantly improve fully color-corrected imaging performance over standard filters, with lower complexity. Being fully dielectric, these splitters additionally allow color-correction to be foregone, increasing the photon transmission efficiency to over 80%, even for sensors with fill-factors of 0.5. Performance further increases with fully 3D structures, improving light sensitivity in color-corrected imaging by a factor of 4 when compared to filters alone.

Introduction

Color imaging is most commonly achieved through absorptive Bayer filter arrays (depicted in Figures 1A and 1C), with each of three color pixel filters allowing ∼1/3 of the incident light to transmit to the active material below (Bayer, 1976). While this allows high selectivity of individual colors and thus simple image processing, the significant losses are becoming an increasing issue, particularly as the trend toward smaller, higher resolution sensors further reduces the incoming average photon count per pixel.

Figure 1

Schematic of working principle

(A) Cross-sectional diagram showing color filters (blue, red regions), which lose 50%–75% of incident light to the filter absorption, and can suffer additional losses due to inactive regions (gray bar) on the sensor front surface.

(B) Color splitters investigated here separate and focus light in the appropriate color pixel, obviating absorption loss, as well as the need for micro-lens arrays.

(C) Top view of a Bayer array, with colors corresponding to red, green, and blue color pixels, and the investigated splitter unit cell and actively simulated (through symmetry) regions outlined (dashed gray and yellow lines, respectively). The shaded regions represent the inactive sensor regions, with a 50% fill-factor.

An alternative approach to filtering is splitting the incident light by color, redirecting the appropriate wavelength range to a corresponding pixel (Figure 1B). Theoretically, this can increase the photon collection on average by 3×, with additional improvements possible for front-illuminated CMOS configurations (where often only 50% of the illuminated surface is photoactive) (Zhang et al., 2010), due to built-in lensing/funneling (Sounas and Alu, 2016; Johlin et al., 2018). Color splitting for imaging applications has been investigated previously through fairly complex processes (Chen et al., 2016), often involving high-index material processing steps (Sounas and Alu, 2016; Miyata et al., 2019; Chen et al., 2017; Nishiwaki et al., 2013; Tamang et al., 2019; Zhao et al., 2020), new elements in the far-field of the sensor (Miyata et al., 2019; Chen et al., 2017; Nishiwaki et al., 2013; Xiao et al., 2016; Camayd-Muñoz et al., 2020), and computationally expensive image reconstructions (Wang and Menon, 2015; Sahoo et al., 2017). Many designs additionally only work with a specific polarization of light, limiting their efficiency for normal imaging applications (Xu et al., 2010; Nguyen-Huu et al., 2011; Kanamori et al., 2006). Furthermore, as sensors now regularly feature pixel sizes down to 800 nm in pitch (Sony Corporation, 2018), color splitters need to be capable of handling this high pixel density as well, which to our knowledge has not yet previously been achieved, with many existing designs being 10-100× larger (Davis et al., 2017; Wang and Menon, 2015; Chen et al., 2017).

In essence, there are three competing objectives for either color splitters or filters: selectivity, transmissivity, and simplicity. Filters excel at selectivity and simplicity, but with low transmissivity, while splitters usually sacrifice simplicity and selectivity for increased transmissivity. While selectivity can to some degree be compensated for via processing (Wang and Menon, 2015; Hauser et al., 2019), so far the need for multiple additional materials, as well as precisely aligned additional layers away from the sensor surface have precluded wide adoption of color splitters for compact sensing.

In this work, we computationally explore the design process and performance of fully dielectric, low-index nanophotonic color splitters for imaging applications, with the splitter element printed directly on the surface (and thus in the near-field) of the active material layer. This would thereby allow these novel splitters to serve as simple drop-in replacements for the current absorptive filters, while significantly increasing the sensor transmissivity. Specifically, we investigate components containing a range of geometric complexity, from single-layer 2D patterns (producible through simple single-step photolithography) to fully 3D structures (fabricable through multi-photon lithography (Deubel et al., 2004), or holography (Yuan and Herman, 2016)). In all cases, we consider only a single low index (n = 1.5) dielectric material, printed directly on top of a complementary metal–oxide–semiconductor (CMOS) image sensor, which to our knowledge has not been investigated previously. We observe up to 2.0× improvements in photon collection efficiency even after rigorous (but computationally trivial) color correction, accompanied by the option to increase collection by 2.4× in grayscale imaging (e.g. for low-light imaging, night vision) on the same sensor.

Results

Splitter design

The general philosophy of this work is to utilize adjoint-based inverse design to determine structures that succeed at splitting broadband, randomly polarized light into a pixel array, based on three distinct wavelength bands: 400-500 nm (blue), 500-600 nm (green), and 600-700 nm (red). We investigate structures of varying complexity, from single-layer (fully 2D; metasurface) coatings, to four-layer (4× 2D) coatings (Figures 2A and 2B), producible through one to four standard photolithography steps respectively, to fully 3D photonic-crystal-like structures (Figures 2C and 2D).

Figure 2

Color splitter designs

(A) Isometric-view rendering, and (B) top-view of color-coded layers for the four-layer topology-optimized splitter structure.

(C) Isometric view and (D) top-view average density projection of fully 3D splitter structure. Dashed yellow lines in A and C represent the actively simulated region.

We utilize a two-step process for the overall design of the splitter geometry: First, topology optimization using a simple figure of merit is run for a range of initial conditions, returning a locally optimal structure for each provided starting point. This is followed by a more accurate analysis of the final designs, using a color-corrected efficiency metric to determine the single best structure those generated by the various initial conditions. The process for each of the steps follows, with additional details in the transparent methods.

Topology optimization

Adjoint-based inverse design is an efficient method for nanophotonic topology optimization (Rodriguez et al., 2018). This works by leveraging properly configured forward and reverse (adjoint) simulations to compute gradients of a figure of merit with respect to discrete changes in structural permittivity. By iterating this process, one can converge on a locally optimal solution to a figure of merit, given the initial conditions. This approach has been used widely in nanoscale optics to design components ranging from optical resonators (Lu et al., 2011), to integrated photonic circuits (Lalau-Keraly et al., 2013; Elesin et al., 2014; Piggott et al., 2015), to metasurfaces able to perform computational tasks (Estakhri et al., 2019; Liu et al., 2018).

Here, we utilize inverse design to maximize the concentration of on-band (correctly colored) light in the center of the corresponding pixel. Pixel sizes are fixed at 800 nm to correspond to the highest resolution CMOS detectors commonly available today (Sony Corporation, 2018), with an assumed active area fill-factor of 0.5, corresponding to 565 nm wide square active regions (Zhang et al., 2010). The optimization region total thicknesses was limited to 1000 nm, similar to the thickness of color filters (FUJIFILM Holdings, 2020). The material being structured is taken to be purely dielectric with a refractive index of n = 1.5 comparable to that of transparent photoresist, on a glass substrate of equivalent index, and with a background of air (n = 1.0) on the front surface and any material-absent areas of the structured region.

Each optimization iteration step consists of four broadband finite-difference time-domain simulations, corresponding to the two orthogonal light polarizations, each with one forward and one adjoint simulation. By orienting the simulation coordinate system diagonally with respect to the sensor layout, we can exploit the system symmetry, requiring only 1/4 of the splitter region to be actively simulated (dashed outlines in Figure 1C). For the 800 nm pixel size, the active simulation area thus has dimensions of 565 × 1130 nm.

The optimization figure of merit (FoM) is the function that the optimization takes gradients of to determine the evolution of the structure permittivity, and thus geometry. While ideally the final color-corrected response (discussed below) would be used for the optimization, such an equation is not differentiable making the computation of gradients impossible. To address this, we use a simplified FoM for the optimization, corresponding to the intensity of the electric field of on-band light () in the center of each color pixel immediately below the splitter surface, from broadband planewave illumination in the forward simulation. This corresponds to a simple adjoint simulation as well, corresponding to a single dipole emitter of the proper emission frequency again at the center of each pixel, scaled by the electric field from the forward simulation. This follows the general method of field localization optimization described in (Miller, 2012), and is coincidentally similar to method used by the concurrent work in (Camayd-Muñoz et al., 2020).

As adjoint-based topology optimizations are local optimization techniques, the choice of initial conditions have a significant impact on the resulting design and performance. For each layer complexity, a series of optimizations was run, ranging from fully filled to fully empty along with 9 equally spaced uniform index levels between as the starting point for the optimization. This allows us to utilize a full color-corrected efficiency calculation to compare the 11 resulting structures for each configuration and select those with the best overall performance. See the transparent methods supplemental document for additional details and Figure S1 for a flowchart of the optimization and simulation process.

Color-corrected efficiency

At the most basic level, a color splitter could be operated similar to a color filter, attenuating the incoming light to the point where the off-band color response (e.g. red light into the blue pixel region) is sufficiently low, and then determining the transmission efficiency of the on-band color response. However, since the signal from four single-color pixels are combined to create one RGB image pixel, one can use the correlations between the pixels to determine more accurately the colors in the image. This allows color selectivity to be improved, although still at a cost of some signal intenstiy. We here describe a general method for evaluating the overall color imaging efficiency using a simple matrix inversion process, applicable to any color isolating component (splitter or filter).

We begin by creating a 3 × 3 matrix, R of the color response, with rows corresponding to the red, green, and blue pixels, and columns being the average red, green, and blue color transmission into the respective pixel. In a theoretically perfect splitter, R would be the identity matrix, but in any real design there will be some degree of off-color transmission represented by non-zero off-diagonal elements. The inverse of the response matrix represents the linear combination of the pixel responses that would recover the identity matrix (I) and thus the ideal response, by using combinations of the color pixels to effectively cancel out the off-band responses. However, in order to avoid amplification of noise, the value of any element in the inverse matrix must be limited to 1 (i.e. we allow signal attenuation but not gain). Furthermore, the total performance of the system will be ultimately limited by the response of the worst-performing color-corrected element, and so the color responses must be uniformly scaled. We thus normalize the inverse matrix by the largest element in said matrix, thereby providing the gain-free color-correction matrix, Q, as

The color-corrected response to a uniform input is thus RQ, and is equal to , where η is the overall color-corrected efficiency.

It should be noted that while matrix inversion is often a computationally expensive process, the small 3 × 3 size of these response matrices, and the need for this to be performed only once during the splitter design phase makes the color correction process computationally trivial – during actual imaging, the correction is simply a uniform linear combination of the sensor's red, green, and blue signals.

Performance of splitters

The highest splitter performance was achieved by the fully 3D structures. The structure is portrayed in Figures 2C and 2D, and the color response is shown in Figure 3, with the Poynting vector components normal to the substrate at surface (representing the flux of energy) depicted in Figure 3B. It can be seen that all three colors are not only spatially isolated, but well focused into the center of the active region of the corresponding pixel, even without micro-lens arrays. The four-layer structure (shown in Figures 2A and 2B) appears similar in spatial power transmission (Figure 3A), still showing strong spectrally selective focusing of light onto the sensor surface. The selectivity, however, is slightly lower, particularly for the green pixel, as seen in Figure 3C.

Figure 3

Splitter operation

(A) Poynting vector -component (toward the sensor) within the three color bands (red, green, blue), visualized at the sensor surface. Pixel active regions are shown as dashed lines.

(B) Transmission of light from the full 2 × 2 pixel region into the sensor active region for the three different pixel colors. The 3D splitter (solid lines) shows lower selectivity than the Bayer filter (dashed lines), but the much higher transmissivity still leads to twice the color-corrected efficiency (34% vs. 17%, respectively).

The simulated transmission spectrum is shown in Figure 3C and D for four-layer and 3D splitters, respectively, and is compared to that of a representative commercial Bayer filter absorptive layers (FUJIFILM Holdings, 2020). It is observed that each color component of collection efficiency is higher than that of absorptive filters, indicating that light is successfully rerouted into the correct filter region. This is likely occurring through a combination of refractive and interference effects, allowing efficient broadband responses to be created in the near-field of the sensor (Johlin et al., 2018). It also can be noted that while the collection efficiency is much higher, the selectivity is lower. However, as discussed above, this can be compensated for using linear combinations of the color pixel readings via Equation 1 to produce the final fully color-corrected image, at a cost of some sensitivity. Our estimated efficiency of the 3D color splitter is 34% for fully color-corrected imaging, and 81% for grayscale imaging. This is compared to the Bayer filter, with color-corrected estimated efficiency of 17%, and grayscale efficiency of 33%.

Interestingly, the color separation mechanism of the splitters investigated herein appears to be distinct from those largely studied previously; the proximity of the color splitting structure prevents the use of angular redistribution to spectrally separate the light (Nishiwaki et al., 2013), relying instead on full spatial separation at the base of the splitter, with little redistribution in angular-space of the transmitted light. This provides additional benefits as well, in that the splitters designed here appear to perform better at off-angle illumination than previous designs that rely more on scattering or diffraction to reconfigure the angular distribution from the splitter (Camayd-Muñoz et al., 2020). For full characterization of these effects, see Figure S4.

Simulated full-color imaging

In order to validate the accuracy of the color correction, as well as visualize the increased sensitivity, it is instructive to directly compare the performance of the two sensor designs (Bayer filter vs. splitter) on realistic images. To do this, we use freely available 2000 × 2000 pixel hyperspectral images (Brainard, 2004) to simulate the operation of both the 3D color splitter, as well as the Bayer array for comparison. Through convolution of the per-pixel spectrum, first with a light source spectrum (the AM 1.5 sunlight spectrum is used here), followed by our color responses, and finally applying the linear color correction operation from Equation 1, we obtain the expected full-color image response for each design. A reference image with 50% efficiency and perfect color separation is shown in Figure 4B, along with the responses of filters and splitters after color correction, as well as in grayscale imaging.

Figure 4

Simulated imaging response

(A) Pixel transmission density functions for color splitters (red) and Bayer filters (blue), in color-corrected (solid) and grayscale (dashed) imaging, for the given test image.

(B) Reference image assuming perfect splitting at 50% efficiency.

(C and D) Color-corrected simulated images for comparing filter and splitter performance, respectively.

(E and F) Grayscale simulated images for filter and splitter, respectively.

By comparing the color-corrected response of the filter (Figure 4C) and splitter (Figure 4D) to the reference image, the splitters accurately reproduce the true colors, and with higher brightness than the standard Bayer filter. This is increasingly visible in the grayscale images (Figures 4E and 4F), where the splitter even surpasses the brightness of the 50% efficiency reference image. From the imaging efficiency distributions in Figure 4A, we can see that indeed in color imaging the collection efficiency is increased by a factor of 2.0, from 15.9% for the Bayer filter to 31.3% for the color splitter, and with the grayscale efficiency increasing by a factor of 2.4, from 33.6% to 80.2%. The difference in performance from the estimated efficiency arises the non-uniform color distribution of the reference image, responsible for the extended distribution in the splitter transmission density function as well.

Furthermore, the above comparisons all conservatively assume perfect micro-lens structures above the Bayer filter array, being both lossless and focusing all transmitted light into the active region. Without micro-lens arrays, the Bayer filter efficiency would be reduced by an additional factor of 2, with imperfect lenses somewhere in between. The splitters require no such micro-lens arrays, as they inherently focus the light into the active regions already.

It should be noted that the color correction used here is strict, in that all cross-color response must be eliminated. This is why even the Bayer filters, which have some color contamination, particularly in the green pixel response, show reduced efficiency when corrected as well. By limiting the strictness of this correction, any position on the trade-off continuum from no color correction to full color correction can be selected, and so for lower color contrast, increasing efficiencies can be realized.

Complexity and robustness

Reducing the layer complexity by an order of magnitude to just four discrete layers (structure shown in Figures 2A and 2B), still maintains much of the performance of the color splitters. In fully color-corrected imaging, an efficiency of 21% is obtained, still notably higher than filters, especially when compared to filters without micro-lens arrays. In grayscale imaging, performance is even slightly higher than full 3D splitters, at 83%. With two layers (Figure 5C), or even just a single-layer (Figure 5B), splitters are still able to produce full-color images, although at a fairly low efficiency. As expected, they maintain high grayscale performance, still without the need for any micro-lens array.

Figure 5

Performance vs. complexity trade-off

(A) Splitter performance, in terms of relative color-corrected imaging efficiency, shown for both in color (blue) and grayscale (red) imaging. Performance is normalized to Bayer filters with assumed perfect micro-lens structures (black solid), with the performance without such lenses shown as well (black dashed).

(B and C) Single- and two-layer structure geometries are shown in B and C, respectively.

This trend in performance vs. complexity is further clarified in Figure 5A, showing the simulated color-corrected imaging efficiency with respect to the number of splitter layers. It should be noted that fully 3D structures here are equivalent to 40-layer structures due to the finite optimization voxel size. In particular the large gain in performance between two- and four-layer configurations is clearly visible, with the imaging efficiency moving from substantially below the filter performance, to substantially above. This leap in performance indicates that the distinct layers spectrally separate light as it moves through the splitter, and demonstrating that the changes in structure topology (as splitter thickness is constant) progressively separates and focuses the three color bands. Further characterization of one- and two-layer splitter structures is shown in Figure S2.

Finally, in order for such color splitters to be usable in real devices, robustness against both defects and non-ideal operating conditions should be ensured. Due to the broadband response of the splitters, performance should be generally invariant to minor (uniform) expansion or contraction of the structure. Furthermore, the strong focusing within the active regions of the pixels allows some built-in robustness to misalignment, as shown in Figures 3A and 3B, and quantified in Figure S3. While all analysis here assumes randomly polarized light, we additionally ensure that such splitters perform well in arbitrary linear polarizations, as well as for non-normal incident light, as is common in normal imaging systems (see Figures S4–S6).

Discussion

Herein we have shown that efficient color splitters can be designed using low-index dielectric layered 2D or fully 3D structures directly on the surface of CMOS sensors. Performance significantly surpasses that of traditional filter arrays for splitter designs even with as few as four layers, with efficiency enhancements as high as 4 times over CMOS sensors without micro-lens arrays. Splitter structures show added functionality in the ability to choose any point in the color accuracy vs. sensitivity trade-off, permitting accurate color imaging, nearly lossless grayscale imaging, and anything in between, all on the same sensor. This could offer particular advantages in color biological imaging, where high efficiency, small pixel sizes, and color sensitivity are all desired (Wu et al., 2008; Farsiu et al., 2004). Furthermore, these splitter structures do not require any focusing micro-lens arrays, even when used with 800 nm pixel CMOS sensor configurations with active area fractions of 50%, further simplifying the use of such components.

The placement of the splitter in the near-field of the sensor surface offers unique benefits over previous color splitter configurations as well, both in design and performance. Removing the need for precise yet thick separation layers can simplify fabrication, and operating without angular redistribution of light transmitted to the sensor surface appears to provide increased invariance to off-normal illumination conditions, and may also reduce losses and cross talk in sensor as well (Zhang et al., 2010).

Finally, the color-corrected efficiency metric developed for the splitter design process here is general, and should be useful in comparing the performance of new splitter or filter structures as well. A similar design and analysis approach could additionally be utilized for other applications, such as infrared imaging or multi-junction photovoltaics, especially where splitting and focusing are both advantageous (Xiao et al., 2016).

Limitations of the study

All work presented here is computational, and so limitations and adjustments to designs may be necessary when experimentally fabricating such splitters. The structures discussed here are intended to be fabricable through either sequential photolithography (for layered structures), or printed directly with multi-photon lithography (Deubel et al., 2004). While feature sizes were specifically limited to allow reasonable (with the smallest features having 90 nm widths), the same process with either more or less restrictive limitations on sizes, numbers of layers, or a different refractive index material could be implemented as needed. The built-in robustness against expansion/contraction of the structures, as well as slight misalignments should ease fabrication requirements; however misalignment between layers of 2D structures could also hinder performance. Concurrent work by others has demonstrated the operation of 3D dielectric splitters (although with larger overall sizes, and positioned far above a sensor surface) in the microwave regime (Camayd-Muñoz et al., 2020), providing evidence that structured splitter devices should be readily achievable in the near future.

Resource availability

Lead contact

Further information and requests should be directed to the Lead Contact, Dr. Eric Johlin (ejohlin@uwo.ca).

Materials availability

This study did not use or generate any physical materials.

Data and code availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Methods

All methods can be found in the accompanying transparent methods supplemental file.