NEUTRON: Neural particle swarm optimization for material-aware inverse design of structural color.

Designing optical structures for generating structural colors is challenging because of the complex relationship between the optical structures and the color perceived by human eyes. Machine learning-based approaches have been developed to expedite this design process. However, existing methods solely focus on structural parameters of the optical design, which could lead to suboptimal color generation because of the inability to optimize the selection of materials. To address this issue, an approach known as Neural Particle Swarm Optimization is proposed in this paper. The proposed method achieves high design accuracy and efficiency on two structural color design tasks; the first task is designing environment-friendly alternatives to chrome coatings, and the second task concerns reconstructing pictures with multilayer optical thin films. Several designs that could replace chrome coatings have been discovered; pictures with more than 200,000 pixels and thousands of unique colors can be accurately reconstructed in a few hours.

Introduction

Structural color refers to the color generated through the light interaction with patterned or layered optical structures. It is more stable than colors produced from chemical pigments and serves as an environment-friendly alternative. However, designing the structures for producing desired colors is challenging because of the complex relationship between the optical structures and their spectral properties. In addition, color metamerism, i.e., different spectra may correspond to the same color perceived by human eyes (Foster et al., 2006; Best, 2017), makes the relationship between the structures and the perceived color more complex because multiple different structures could have the same color appearance. Human experts often design optical structures based on the understanding of the physical properties of structures, including multilayer thin films (Wang et al., 2018; Yang et al., 2019), metasurfaces (Sun et al., 2017; Yang et al., 2020), and self-assembled colloidal particles (Kim et al., 2011; Liu et al., 2021), to name a few. Owing to the complex relationship between the structures and the generated color, the manual design process is often slow and could lead to suboptimal color production.

Recently, machine learning-based optical inverse design approaches have been developed to predict optical structures that can achieve user-specified color properties (Liu et al., 2018b; Ma et al., 2019; Unni et al., 2021; Yeung et al., 2021; Yao et al., 2019; Jiang et al., 2020). These inverse design methods often involve training a machine learning model such as deep neural networks (Gao et al., 2019) or support vector machines (Huang et al., 2019) on a curated dataset that contains a large number of data points mapping structural parameters to the corresponding color, e.g., represented by coordinates in CIE xyY (Gao et al., 2019) or LAB color space (Sajedian et al., 2019; Dai et al., 2021). Though previous methods have been demonstrated to be efficient in designing a wide range of colors, they often require materials constituting the optical structures to be fixed. Because the refractive index of materials affects their reflection and absorption properties, it could be challenging or even impossible to produce specific colors when the materials are not appropriately selected. Thus, the first step of screening appropriate materials for latter inverse design with machine learning models still requires extensive effort from human experts and is difficult for people without sufficient prior experience, which should be adequately addressed for the wide adoption of the developed machine learning models. For optical multilayer thin-film design, the recently reported reinforcement learning approach (Wang et al., 2021) addresses the material selection challenge by searching the material and thickness design space simultaneously. However, this method only designs a single color at a time because the reward function for training the reinforcement learning algorithm has to be defined for a specific color. It could be impractical when many colors need to be designed, e.g., reproducing all the colors found in a paint shop or designing a large array of reflective color pixels to reconstruct a colored picture, which would take an extremely long time using the reinforcement learning method reported in our previous work. To address these issues, we propose an inverse structural color design method that can efficiently predict both the materials and structural parameters in a synergistic manner.

The proposed method termed Neural ParTicle SwaRm OptimizatioN (NEUTRON) is a hybrid approach that combines a Material-aware Multitask Mixture Density Network (M3DN) (Caruana, 1997; Bishop, 1994) and Particle Swarm Optimization (PSO) (Kennedy and Eberhart, 1995). Instead of searching the material and thickness space simultaneously, NEUTRON first predicts the most suitable materials and provides a diverse set of initial guesses of the thicknesses in the form of probability distributions that could fulfill the target color and then applies particle swarm optimization to fine-tune the initial thickness designs. We demonstrate the effectiveness of the proposed approach on two optical multilayer thin film design tasks. The results show that our approach can lead to accurate color inverse designs efficiently. We believe that the proposed approach can be readily extended to many other optical design tasks where material selection and structural designs are both important.

Background

Both numerical optimization and machine learning have been applied to the task of structural color design. Compared to machine learning, optimization-based methods for structural color design are often more accurate but suffer from low computational efficiency. Here, we briefly review the related works on numerical optimization and machine learning approaches for structural color design. For more comprehensive discussions on both approaches for broader nanophotonic design tasks, we refer readers to review articles (Campbell et al., 2019; So et al., 2020; Wiecha et al., 2021; Ma et al., 2021; Jiang et al., 2021).

Various optimization methods including PSO (Kennedy and Eberhart, 1995) and genetic algorithms (GAs) (Whitley, 1994) have been applied to design optical structures for structural color applications. While applying optimization methods to design optical structures for structural color, the designs are iteratively updated through the feedback from the electromagnetic simulations. Owing to the active feedback loop, optimization methods can often lead to highly accurate results but may suffer from low computational efficiency because of the large number of EM simulations (Schneider et al., 2019). Shokooh-Saremi and Magnusson applied both PSO and GA to the design of grating filters with a sharp reflection peak around 550 nm (Shokooh-Saremi and Magnusson, 2007). Both optimization methods took a few hours to converge for a single design target. Yang et al. applied PSO to design 2D grating structures for reflective structural color (Yang et al., 2013). The reflective color maintains CIEDE2000 less than eight for light incidence angles up to 45°. Rabady and Ababneh optimized the layer thicknesses of optical multilayer thin films with PSO (Rabady and Ababneh, 2014), where the alternating materials with high and low refractive index are fixed. Recently, Liu et al. applied GA to optimize the geometry parameters of metasurfaces with multiple meta-atoms in a single cell to produce highly pure primary red, blue, and green colors. Compared to metasurfaces without multiple meta-atoms, the achievable area in the CIE diagram is expanded by 106% (Liu et al., 2020). All aforementioned works only apply optimization methods to a few design targets because of the low computational efficiency.

Machine learning methods, especially deep learning, have been recently applied to the task of structural color designs. Unlike numerical optimizations, machine learning methods learn models from the data points that can later be used to efficiently predict designs corresponding to user-specified color targets. However, machine learning could lead to lower design performance than optimizations because of the lack of active evaluations with EM simulations. Tandem networks combine a forward network and an inverse network to directly predict the designs corresponding to a color target (Liu et al., 2018a). The forward network predicted optical properties given the design, whereas the inverse network performs the opposite. Gao et al. optimized the reflective color of silicon metasurface (Gao et al., 2019) and Dai et al. (2021) designed three layer Fabry-Perot color filters with tandem networks. However, both works can only output a single design given a user-specified color input, whereas our proposed method can output a set of potential solutions. The capability of outputting a set of solutions allows researchers to select designs that are amenable to the fabrication process (Ma et al., 2022). Mixture density networks (MDNs) model the potential designs as a mixture of Gaussian distributions and are able to output a set of designs. Most closely related to our work, Unni et al. applied MDNs to design multilayer thin films with high reflection (Unni et al., 2021). Unlike our proposed approach, their method fixes the material selections for the alternating high refractive index and low refractive index layers, a well-established structure for optical filter design. Moreover, they select the best design from the set of MDN’s outputted designs by comparing the optical properties predicted by a forward network. Instead of relying on predicting the optical properties with the forward model, our method uses PSO to fine-tune the thickness predictions by the MDN through the TMM simulations’ feedback. Thus, our method does not suffer from the potential prediction errors from the forward model. Owing to the ability to search the optimal materials and accurately fine-tune the designed thicknesses with PSO, our proposed method could achieve a better design accuracy, with improved efficiency for a large number of color designs.

Results

Here, we first provide an overview of the proposed method. Then, details on how the training dataset is generated are discussed. We highlight the exceptional performance of the proposed approach with results on two design tasks, including 1) environment-friendly chrome coating replacement design and 2) picture reconstructions. Finally, we conduct analysis to understand the effects of major components of the proposed method.

Overview of the proposed method

The proposed method consists of four steps for structural color designs (Figure 1). In the first step, we collect training data with a wide range of colors. Details of the dataset collection are described in the next section. Then, a novel neural network model called material-aware multitask mixture density network (M3DN) is trained on the collected dataset. Next, for a given color design target, M3DN outputs a set of potential designs with different material combinations and layer thicknesses. Finally, the set of initial designs are fine-tuned with PSO to obtain the final designs with optimized material selections and layer thicknesses.

Figure 1

Pipeline of NEUTRON

We used a dataset generated through transfer-matrix method to train a M3DN. After the training, M3DN can output a set of designs based on different candidate materials. The initial designs’ thicknesses are further fine-tuned through particle swarm optimization.

Dataset generation

Previous research shows that five-layer optical thin films with two absorbing layers sandwiched by two dielectric layers and a bottom metal reflecting layer (Figure 2A) can achieve high color purity and brightness (Yang et al., 2019), where the layers can be easily deposited by physical vapor evaporation. Owing to the high performance and feasibility for large-scale fabrications of such structures, we synthesize a dataset with diverse designs based on the same five-layer structural template by varying both the material and thickness of each layer. All designs are based on randomly sampled materials from ten candidate metal materials Au, Ag, Al, Cu, Cr, Ge, Ni, Ti, W, Zn, and ten dielectric materials Al2O3, Fe2O3, HfO2, MgF2, SiO2, Ta2O5, TiO2, ZnO, ZnS, and ZnSe. Both absorber layers and the bottom reflective layers are composed of metals, whereas the other two layers are based on dielectric materials. Including a wide range of candidate materials with different refractive indices makes it possible to search for the most suitable materials combinations for specific color targets.

Figure 2

Five-layer optical structure and the data distribution

(A) A five-layer optical layer thin-film structure for generating a wide range of reflective colors covering the sRGB color gamut. Both the materials and their thicknesses are designed to obtain the target RGB color.

(B) The dataset generation process. 400,000 data points are obtained through PSO for randomly sampled RGB color target and material combinations.

(C) The color distribution of the validation set is visualized in the CIE 1931 xy space. The data points achieve good coverage of the entire sRGB color gamut spanned by the standard Red, Blue, and Green colors.

(D) Randomly sampled RGB color target and the obtained RGB color through PSO.

While sampling the materials for the two absorbing layers, we introduce a constraint that the two adjacent absorber layers must be composed of different materials; otherwise, it is simply a single absorber material with a larger thickness, which cannot satisfy the design requirement based on previous findings (Yang et al., 2019). Thus, the total number of unique material combinations of the five-layer stack is . Because sRGB is widely used in display industry design and production, we aim to provide the best coverage over the sRGB color gamut through our model. Most previous studies randomly sample the layer thicknesses from a prefixed range to generate data points with different designs and color properties. However, this approach often leads to nonuniform coverage in the color space, i.e., the density of data points in a certain color region is higher than other regions in the color space. Because nonuniform data coverage could lead to an undesirable skewed inverse design performance where the inverse design model is more accurate for the color region with more data points, we propose to directly sample from the color target space uniformly to avoid this problem. To this end, instead of randomly sampling the thicknesses of the layers as in most previous studies, we randomly sample the sRGB color target from the 3D sRGB space with a value range [0, 255] for each dimension. In Figures 2C and 2D, we show that this random color target sampling ensures uniform coverage of the color space.

Then, for each randomly sampled material combination and color target pair, we use PSO to optimize the thickness of the top four layers to minimize the color difference between the color of the designed structure and the color target while fixing the bottom reflective layer at 100 nm. PSO is a widely used global optimization algorithm for optical designs (Shokooh-Saremi and Magnusson, 2007; Yang et al., 2013; Rabady and Ababneh, 2014). When solving a minimization problem, PSO maintains a group of particles (i.e., solutions) that individually explore the solution space and communicate with each other to share information about the explored solution space. All particles’ positions are iteratively updated based on the best solution each particle has found and the best solution found by the entire group until a convergence criterion is met. In our five-layer optical thin film design task, each particle’s “position” is a 4D vector corresponding to the top four layers’ thickness. The final position all particles converge to is the final thickness design obtained through PSO. More details on PSO can be found in the STAR Methods section. Throughout the optimization process, the reflection spectrum of multilayer designs is computed with transfer-matrix method (Byrnes, 2016) for the wavelength range [400, 700] nm with a step size of 10 nm. Thus, each material’s complex refractive index is described by a 62-dimensional vector for the entire 31 wavelength points. We use the Python package Colour (Mansecal et al., 2021) for the conversion of reflection spectrum to sRGB and Lab color coordinates. In addition, we use the industry-standard CIEDE2000 metric based on Lab coordinates to measure the color difference (Sharma et al., 2005) between the target color and the color obtained through the designs. After particle swarm optimization converges, if CIEDE2000 for the given material combination and the color target is lower than or equal to 2, we consider that the specific material combination allows accurate generation of the target color because CIEDE2000 lower than or equal to two is almost imperceivable by untrained eyes (ViewSonic, 2021), and we assign a label ; otherwise the label is . Overall, we observe that the average success ratio of obtaining the accurate color following the random data generation process is about 25%. Both classes of data points ( and ) are included in the final dataset. The Lab color coordinates and the sRGB coordinates for the same design have a one-to-one correspondence, and we only use Lab color coordinates for computing CIEDE2000. In contrast, sRGB coordinates are used explicitly as the color target. If users need to design specifically for a given Lab or CIE xyY target, sRGB coordinates can be uniquely obtained from the provided color coordinates defined on other color spaces.

In terms of the thickness of each layer, we set the thickness range for both dielectric layers from the range [5, 250] nm and the thickness of the absorbing layers to be in the range [5, 15] nm during the particle swarm optimization process. Again, the thickness of the bottom metal layer is fixed to be 100 nm, because it is used as a reflector whose thickness has a negligible effect on the reflective color as long, as the layer is thick enough to reflect the light completely. In addition, we constrain the thicknesses of all layers to be integers to allow fabrications. Thus, the size of the design space after considering variations in materials combinations and thickness designs is .

Because the small number of primitive materials limits variations in the refractive index data, we augment the dataset by randomly mixing two dielectrics or two metals to form synthetic dielectric or metal composite materials to broaden the training dataset distribution through convex combinations of their complex refractive indices. Inspired by our previous finding that the mixture of two thin material layers can be considered as a linear combination of the two materials and helpful to improve color purity Yang et al. (2019), we obtain the complex refractive index of synthetic mixture materials through the following formula:where and are the two sampled materials for ith layer, is the mixing factor for the ith layer, , and are the complex refractive indices for randomly selected two primitive materials and the composite material synthesized from them for the ith layer. The composite material refractive index is synthetic and does not require to be experimentally achievable. It only serves as additional data to improve the data variation so that a machine learning model can learn more efficiently from the data. Similar data augmentation strategies have been widely adopted in other fields (Zhang et al., 2018). Following the described data generation procedure, we generated a total of 400,000 data points and about half of the samples are based on primitive (i.e., actual) materials, whereas the others are based on synthetic materials via linear superposition. The entire dataset is randomly split into 380,000/10,000/10,000 data points for training/validation/testing. Importantly, both validation and test sets are based on primitive materials only, because we are interested in the model’s design accuracy for designs based on attainable actual materials, and mixture materials are introduced only in the training set for the data augmentation purpose. Unless stated otherwise, all results reported in the paper are based on the test sets. Each data point is a tuple comprised of the refractive index data for all five layers concatenated as a single vector with 310 dimensions (each material’s complex refractive data is a 62-dimensional vector), the thickness of top four layers obtained with PSO, the target color sRGB color , the obtained color through PSO , and the binary label , indicating whether the CIEDE2000 between and is lower than or equal to 2. We transform continuous variables to the range [-1, 1]. A pictorial illustration of the full data generation pipeline is included in Figure 2B.

Chrome coating replacement design

In the first task, we aim to demonstrate NEUTRON’s performance for designing a single target color. Specifically, we choose a highly practical application where we design an environmental-friendly five-layer optical thin film stack with an appearance similar to chrome, which could potentially replace the traditional chrome coatings produced by a highly toxic electroplating process that poses great dangers to both workers and the environment. This is an important task because chrome coatings are widely used in the automotive industry for aesthetic purposes because of their pleasing appearance. Traditionally, the chrome layer is electroplated to the surface of the object by submerging it in a chemical electrolyte solution that contains hexavalent chrome (Cr (6)). However, Cr (VI) is a strong human carcinogen and has been found to greatly increase the risk of lung cancer, nasal cancer, and sinus cancer OSHA (2013). Cr (VI) can also cause severe nasal septum ulcerations and perforations, gastritis, and gastrointestinal ulcers Lieberman et al. (1941); Lindberg and Hedenstierna (1983). Workers can be exposed to Cr (VI) at the workplace from the mist generated during plating. The electroplating process can also lead to air pollution through the emissions of toxic air containing cadmium and cyanide, which could impact the nervous system, hearts, and lungs of millions of people EPA (2005). Thus, it is highly desirable to develop alternative solutions that can produce the chrome appearance but do not require the dangerous chrome plating process.

In the material screening phase, we exclude Cr from the available metals and predicted the likelihood for all possible material combinations. Then, we sample initial thickness designs from the M3DN and fine-tune the designs with PSO for the top 100 combinations with the largest likelihoods. The emulated colors of the five best designs are shown in Figure 3. We also provide the detailed designs and CIEDE2000 of all five designs in Table 1. PSO significantly improved the initial designs predicted by the M3DN in terms of the CIEDE2000. All five final designs obtained CIEDE2000 lower than two and are highly promising for replacing the traditional Cr plating process through the environmental-friendly thermal evaporation process.

Figure 3

Single color design pipeline and chrome color design results

(A) The top K material combinations obtaining the target color is obtained through ranking the predicted probability vector for all allowed material combinations. Then a set of initial designs are sampled from the M3DN as the initial solutions for particle swarm optimization to fine-tune the thickness of each layer.

(B) Top five designs based on the final CIEDE2000 values. The particle swarm optimization step leads to significant performance improvement compared to the designs sampled directly from the mixture density network.

(C) CIEDE2000 before and after the particle swarm optimization fine-tuning for the top 100 material combinations.

(D) The reflection spectrum of the best design and a 100 nm thick Cr film.

Table 1

Chrome color designs based on M3DN only and the fine-tuned design by PSO (bold)

Design	AntireflectiveLayer (nm)	AbsorberLayer I (nm)	AbsorberLayer II (nm)	DielectricLayer (nm)	ReflectiveLayer (nm)	ΔE_00
I	HfO2	W	Ge	SiO2	Zn
	120/105	13/15	15/10	122/83	100/100	7.6/0.8
II	HfO2	Zn	Ge	Al2O3	Ni
	12/7	14/14	8/13	160/98	100/100	15.8/1.2
III	SiO2	Ag	Ge	SiO2	Au
	273/250	14/14	15/15	147/130	100/100	9.8/1.4
IV	HfO2	Zn	Ge	Al2O3	Au
	61/5	14/14	15/15	202/207	100/100	25.0/1.6
V	HfO2	Zn	Ge	MgF2	Au
	97/93	15/15	12/11	249/218	100/100	5.4/1.6

Picture reconstruction

Next, we apply NEUTRON to design optical thin film structures for a large set of color targets. We choose the picture reconstruction task (Gao et al., 2019; Dai et al., 2021), because it has been previously studied in machine learning-based structural color inverse design works. Because most high-resolution pictures have a large number of pixels with unique sRGB values, which could take a long time to design when the PSO fine-tuning step is involved, we quantize the pixel values and find the quantization step size of 10 to reduce the number of unique pixels of a full-color picture by more than X while maintaining the quality of the original pictures with more than 200,000 pixels. Details of picture quantization can be found in the STAR Methods section. The detailed design pipeline is provided in Figure 4A, where the top material combination is selected based on the average computed over the entire picture to be reconstructed. Note that we select the same material combination of all pixels across the entire picture to make it possible for fabrication through grayscale lithography Wang et al. (2018). We plot average distributions for all allowed material combinations in Figure 4B, which show that only a few material combinations can lead to high average success design rates and also demonstrate the importance of the material classification network. Again, in Figure 5C, we show that PSO significantly improved the average design accuracy upon the initial designs based on the trained MDN.

Figure 4

Picture reconstruction design pipeline

(A) Similar to the design pipeline of a single color, the material screening is the first phase of the design process for identifying the best material for each layer to reconstruct the entire picture. Unlike the single-color case, the average probability is computed for all target colors through which the material combination that gives the largest probability is selected.

(B) Average distribution for all 100,000 allowed material combinations.

(C) Average CIEDE2000 computed over entire images before and after particle swarm optimization fine-tuning.

Figure 5

Picture reconstruction results

The first column is the target picture. The second column is the reconstruction based on NEUTRON. The last column is without the particle swarm fine-tuning.

(A) The White Orchard by Vincent van Gogh.

(B) The Tulip Field by Vincent van Gogh.

(C) A photo of The Great Wall of China taken by photographer Severin Stalder.

(A and B) are reproduced with the permission from the Van Gogh Museum, Amsterdam (Vincent van Gogh Foundation).

(C) is reproduced with the permission from Wikimedia Commons.

As another demonstration, we applied our method to pictures obtained through neural style transfer Gatys et al. (2016), which is a computer vision method that can transform images to possess an appearance similar to a style source image. As shown in Figure 6, we first apply the neural style transfer method to transfer real photos into stylized pictures, and then we apply NEUTRON to obtain designs for reconstructing the transferred pictures.

Figure 6

Picture reconstruction results based on neural style transferred photos

(A) Flower.

(B) Street. Both original photos in (A and B) were taken by the authors.

(C) The source style painting Vase with Gladioli and Chinese Asters for generating the style transferred photos is reproduced with the permission from the Van Gogh Museum, Amsterdam (Vincent van Gogh Foundation).

Results in Figures 4 and 5 demonstrate that NEUTRON allows accurate picture reconstructions with almost unnoticeable errors. In addition, the entire design process can be accomplished within a few hours for high-resolution pictures (Table 2), which makes NEUTRON a highly practical algorithm for real applications. Although PSO also achieves good performance in terms of , the PSO results reported in Table 2 are based on the material combination predicted by the classification model of the trained M3DN and does not take the amount of time required for screening the materials when such a model is not available. In addition, when initializing the PSO with the initial designs predicted by the M3DN, the design accuracy can be further improved upon using randomly-initialized PSO.

Table 2

The best material combinations and with M3DN only, PSO only, and NEUTRON for reconstructing pictures in Figures 4 and 5

Picture	Material Combination					MDN	PSO	NEUTRON	Time (s)
White Orchard	Al2O3	Ge	Al	Al2O3	Zn	18.55	5.16	3.76	9,917
Tulip Field	Ta2O5	Au	Ti	Al2O3	Au	19.45	4.34	3.48	9,246
Great Wall	HfO2	Au	Ti	SiO2	Au	21.46	6.45	5.69	6,835
Flower	Ta2O5	Au	Ti	Al2 O3	Au	19.37	5.00	3.93	11,249
Street	ZnO	Ti	Au	MgF2	Au	21.60	4.45	3.28	7,349

The results reported for the last two pictures in italic font are for the style transferred photos. Bold values indicate the lowest errors (i.e., best performance among MDN, PSO, and NEUTRON).

All results reported in this section are based on optimizing the top four layers’ thickness to obtain designs across the entire picture. Though the same material combination is selected for the entire picture with the feasibility of fabrication in mind, it would still be highly challenging in practice. Thus, we also explored the feasibility of picture reconstruction when only varying the bottom dielectric layer while keeping all other layers’ thickness fixed. Compared to adjusting all four layers’ thickness, adjusting a single layer’s thickness leads to a deteriorated average CIEDE2000 of 13.36 (see Figure S1) compared to 3.48 when tuning the top four layers for the Tulip Field painting. When fabrication-friendly designs are preferred over design accuracy, varying a single layer could be applied for the picture reconstruction application.

Understanding algorithmic design choices of NEUTRON

NEUTRON involves multiple hyperparameters that affect its performance (Table 3). We conducted a series of experiments to understand the effects of three important algorithmic design choices for the proposed algorithm NEUTRON, including the number of mixture components, multitask learning, and dataset augmentation with mixture materials.

Table 3

Hyperparemeter search values for training the M3DN model

Hyperparameters	Values
Learning rate	loguniform(10^−5,10^−3)
Hidden dimension of FC layers	[128, 256, 512, 1024]
Number of mixtures	[200, 400, 800, 1600, 3200, 6400, 12800, 25600, 51200]
Weight decay factor	loguniform(10^−6,10^−4)
α	[1, 2, 5, 10]

Effect of number of mixture components

The optimal number of mixture components m is dependent on the one-to-many mapping between the input and the potential designs . A large m is required when many designs could lead to the similar color target. Here, we vary the number of mixtures from 200 to 51,200. As shown in Figure 7, leads to the best performance, whereas reducing m causes the validation CIEDE2000 to increase significantly.

Figure 7

Effects of important algorithmic design choices

All results are based on three runs initialized with different random seeds. The shaded area corresponds to one SD.

(A and B) Varying the number of mixtures. The network with 51,200 mixture components achieves the best validation performance among the nine experiments.

(C and D) Effect of weighting parameter for the classification loss.

(E and F) Effect of dataset augmentation with mixed materials. By adding 50,000 data points from mixed materials (A 150K), the trained network achieves a more significant improvement of CIEDE2000 over the dataset with 100,000 data points simulated from primitive materials (P 100K), compared to adding 50,000 additional data points simulated from only primitive materials (P 150K). Further increasing the dataset size to 400,000 data points with 50% of the data simulated from the mixed materials leads to further performance improvement of both validation CIEDE2000 and the validation AUC.

Effect of multitask learning

We vary the weighting parameter α to investigate if positive transfer exists between the material classification task and the design prediction task. Results in Figures 7C and 7D show that leads to the best CIEDE2000 measured on the validation set while the classification AUC is insensitive to nonzero α values. This indicates that positive knowledge transfer exists between the material classification task and the thickness prediction task. Thus, incorporating the material classification task not only allows users to select the best material combinations for downstream tasks but also improves the thickness inverse design accuracy because of the benefit of multitask learning.

Effect of dataset augmentation

Introducing data points simulated from a mixture of materials increases the variation of the input refractive data , which could allow the trained M3DN to generalize better. To test this hypothesis, we first train M3DN on a dataset with 100,000 data points simulated entirely based on primitive materials. Then, we add additional 50,000 data points that were either simulated from primitive materials or mixture materials and train two networks separately. By comparing the validation , we observe that adding 50,000 additional data points from mixture materials leads to greater performance improvement than adding 50,000 data points simulated from primitive materials.

Discussion

In both the chrome color design and the picture reconstruction tasks, NEUTRON achieves exceptional design accuracy efficiently through combining a material classification network, a mixture density network for thickness predictions and the final PSO fine-tuning. Unlike previous methods that either assume fixed materials (Gao et al., 2019; Dai et al., 2021) or search the materials and thickness designs simultaneously Wang et al. (2021), our method split the materials and thickness design into two stages to enable an approach that can search for the best materials and design the thickness accordingly in an efficient manner. The reason why splitting the material and thickness design is helpful can be understood by the fact that the entire design space is much larger than the material design space or the thickness design space alone. In our five-layer optical thin film design task, the material design space size is , and the thickness design space size is , which lead to a huge full design space with a size of when considering the material design space and the thickness design space simultaneously. With the two-step process, we first narrow down the promising material combinations with the material classification model and only optimize the thickness designs for the selected small set of materials. Thus, we only need to consider a design space that is one or multiple times of the thickness design space, which is a X complexity reduction compared to searching both materials and thickness in one step.

We combine M3DN and PSO by initializing the particle positions with designs sampled from the M3DN. This process is easy to implement and highly effective in obtaining optimal designs because of the probabilistic nature of the mixture density network. On both design tasks, we show that PSO significantly improved the initial solutions by the M3DN. This result is not surprising, because PSO involves iterative updates of the design parameters based on the feedback from optical simulations. In addition, the exceptional design accuracy after fine-tuning the initial M3DN designs with PSO indicates that probabilistic machine learning models that can output diverse predictions are highly compatible with PSO, which requires an initial population of designs to begin with.

Our experiment on the number of mixtures suggests that a strong degree of one-to-many mapping exists (Figures 7A and 7B) for the problem of inverse designing the structural color with a five-layer optical stack, thus a large number of mixture components of 51,200 is required to obtain an accurate inverse prediction of the designs. Owing to the limit of GPU memory, we cannot further increase the number of mixture components, but we expect the design accuracy to improve as the number is increased further.

Another interesting finding on multitask learning suggests that future optical inverse design models should leverage the task relationships by learning the main and auxiliary tasks simultaneously. As shown in the experiment for varying the classification loss weight α, we observe optimal inverse design performance at . In optical design problems, many tasks are related, such as the electric field distribution prediction, far-field intensity prediction, spectrum prediction, and color prediction, etc. Learning multiple tasks simultaneously could allow researchers to obtain high inverse design accuracy with fewer data points.

Moreover, our results show that data augmentation that increases the refractive index data variation by mixing the primitive materials is an efficient strategy to improve the inverse design accuracy. We believe such data augmentation should be incorporated for training material-aware inverse design models when possible.

NEUTRON combines a classification model, a probabilistic regression model, and particle swarm optimization for fine-tuning the designs predicted by the M3DN. Compared to using ML or the optimization approach alone, NEUTRON brings the best of both worlds to allow efficient and accurate inverse designs for large-scale design tasks.

In sum, we developed a material-aware inverse design algorithm that combines a multitask mixture density network and particle swarm optimization to select the optimal material combination and optimize structural parameters. On two practical tasks for producing reflective structural color with a five-layer optical stack, NEUTRON demonstrates the proposed algorithm’s exceptional design accuracy and efficiency. Though we develop NEUTRON for structural color design based on multilayer optical thin films, it can be adopted for a wide range of other optical design tasks that require optimizing the material selections and structural parameters.

Limitations of the study

The training dataset for NEUTRON consists of PSO-optimized designs. To achieve accurate design predictions, we collected 400K designs as the entire dataset. However, when applying NEUTRON to optical structures that require a longer simulation time, collecting 400K PSO-optimized designs might be infeasible. Thus, future study on improving the sample efficiency of NEUTRON is required.

STAR★Methods

Key resources table

Resource availability

Lead contact

Further information and requests should be directed to either Haozhu Wang (hzwang@umich.edu) or L. Jay Guo (guo@umich.edu).

Materials availability

This study did not generate new unique reagents.

Method details

Material-aware multitask mixture density networks

Mixture density networks (MDNs) have been previously applied for inverse problems extensively (Bishop, 1994; Li and Lee, 2019; Unni et al., 2021). Instead of mapping the input to a single output, which is done by deterministic neural network models, an MDN maps the input to the probability density function of a multivariate Gaussian Mixture with m isotropic Gaussian mixture components. Because each Gaussian mixture component can learn a different mean and standard deviation value, MDNs are able to learn a one-to-many mapping, which is critical to solving the often ill-posed inverse design problems with non-unique solutions. The probability density function given by a Gaussian mixture for a pair of refractive data input, thickness design, and corresponding color can be written as :where are the mixing weights, mean, and standard deviation outputted by the MDN with the input refractive data and the color target . D is the dimension of the design parameter vector , which equals 4 in our design problem. Note that the mixing weights sum up to one (i.e., ) so that the mixed function is a proper probability density function.

In addition to predicting the thickness, we also predict whether a given refractive index data can lead to accurate designs for a target color with CIEDE2000 2 (i.e., ) after the thickness of each layer has been optimized. To this end, we include a sub-network to form a material-aware multitask mixture density network (M3DN) that is composed of a mixture density network and a classification network (see Figure S2A).

To train the M3DN, we minimize the multitask loss function:where the first term on the right hand side is the negative log likelihood for training the M3DN, and the second term is the binary cross entropy loss for training the classifier, i.e.,

The hyperparameter α in the M3DN loss controls the knowledge sharing among the material screening task and the thickness prediction task. Note that, is used as input for training the M3DN while is used as input for training the classifier because the thickness corresponds to , which could differ slightly from the in the generated dataset because PSO may only be able to find solutions with a color coordinate slightly different from the target color, i.e., .

Through an extensive hyperparameter search based on the CIEDE2000 measured on the validation set (details can be found in the STAR Methods section), we found a seven-layer neural network with four separate output heads to give the best inverse design accuracy (see Figure S2A). The number of of Gaussian mixture components is 51,200. We visualize the learned probability density function and the mixing weights (see Figure S3). Note that we use the softplus activation function for the output head to ensure the standard deviation prediction is always greater than 0, and softmax activation function is used for the mixing weight output so that all mixing weights sum up to 1. We use ELU activations for all other layers except the last layer in the base network and the output layer for μ, where tanh is used to ensure their outputs are in the range [-1, 1].

The advantage of training a single multitask network for both classification and thickness prediction tasks is two-folded: 1) the classification module allows users to screen among possible material combinations to select those that could lead to an accurate generation of the target color; 2) improved sample efficiency through sharing knowledge among the related classification and thickness prediction tasks.

The test results for the classification AUC (area under the receiver operating curve, or AUROC) and the inverse design accuracy in terms of CIEDE2000 for both the target color and the PSO obtained color are visualized (see Figures S2B and S2C). The material classification network achieves excellent performance with an AUC of 0.91. Both average test CIEDE2000 values are slightly above ten and can be further improved by finetuning with the PSO.

We train the M3DN with PyTorch (Paszke et al., 2017) on NVIDIA RTX 3090 GPUs with 24 GB of internal memory. Details on implementation and hyperparameter tuning can be found in the STAR Methods section. Both our code and data are publicly available (links included in the key resource table).

Based on the downstream task, the material screening module can be used in various ways with the predicted probabilities . On one hand, when the goal is to select the best material corresponding to a single color (e.g. in the first example below searching for chrome color replacement), the top K material combinations with largest among all possible combinations can be chosen for further examinations of their performance. On the other hand, when researchers have a set of target colors to produce (e.g. in the second example of reproducing a color picture), the average over the set of target colors can be computed for ranking and selecting the material combinations.

Given the selected materials, the thickness designs are predicted based on the color target and the material index data. However, since the output of the mixture density network is probabilistic, it is possible that the thickness output by the trained mixture density network does not correspond to the optimal value. To address this issue, we further fine-tune the designs with PSO. Combining the dataset generation process, M3DN, and the PSO finetuning process completes the proposed NEUTRON method for structural color design.

In all of our experiments, for each material combination, we randomly sampled 32 thickness designs from the MDN to be used as the initial positions of the particle swarm optimization. Then, we optimize until convergence with a tolerance level of . In each optimization iteration, the designs are evaluated through the transfer matrix method (Byrnes, 2016) and compared with the target spectrum. Compared to randomly initializing the positions of the particles, starting with designs sampled from the M3DN allows PSO to find better solutions. More importantly, conventional PSO requires the materials to be provided by the user while NEUTRON predicts the best materials to be used with the PSO. Since the manual material screening process could be time-consuming, using NEUTRON for structural color design can significantly speed up the entire design process by directly predicting the best materials.

Picture quantization details

High-resolution pictures generally have more than unique pixel RGB values, which makes it impossible to directly reconstruct when iterative optimizations are is involved. In our work, the particle swarm optimization process for finetuning a single pixel requires seconds, and reconstructing the entire picture takes seconds. To speed up the reconstruction, we apply a quantization approach to group similar pixels values to the nearest integer values that can be divided by a quantization step size s. We explored quantized step size and found the step size led to the best tradeoff between the compression ratio of unique pixels values and the quality of the quantized picture. We compare the pictures before and after quantization with a step size 10 (see Figure S4). With a negligible difference in picture appearance, we achieved more than 40X compression ratio of unique pixel values for all pictures.

Model implementation details

For training the M3DN, we randomly search hyperparameters from the range listed in Table 3. The model with the best CIEDE2000 on the validation set has a learning rate of 0.000176, hidden dimension 128, the number of mixtures 51,200, weight decay of 0.0000117, and . We train the model on the training set for 500 epochs and store the model with the model checkpoint with the best validation CIEDE2000. Our reported results in the main text are all based on the best model unless stated otherwise.

In particle swarm optimization, each particle’s position at time t is updated with a velocity that is determined by both the best solution found by the particle itself and the best solution found by the entire group. In the optical multilayer thin-film design task, the position vector is the thickness designs for all layers. The velocity is the update to each layer’s thickness between two optimization steps. The velocity is calculated as:where is the best solution found by the ith particle by time step t, while is the best solution found by the entire group of particles by time step t. are two random numbers in the range [0, 1]. w is the inertia coefficient that controls how much the previous velocity is maintained, is the cognitive coefficient that controls how much the best solution found by the particle itself contributes to the velocity, and is the social coefficient that determines how the best solution found by the entire group alters the velocity. In our implementation, we set the based on the results from a manual search. After the velocity is obtained at the th step, we update the particle’s position by:

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Deposited data
Color datasets	This paper	Mendely Data:https://data.mendeley.com/datasets/54d84d88p8/1
Software and algorithms
NEUTRON code base	This paper	https://github.com/hammer-wang/NEUTRON
TMM Python package	GitHub	https://github.com/sbyrnes321/tmm
Color Python package	GitHub	https://github.com/colour-science/colour