Learning to see colours: Biologically relevant virtual staining for adipocyte cell images.

Fluorescence microscopy, which visualizes cellular components with fluorescent stains, is an invaluable method in image cytometry. From these images various cellular features can be extracted. Together these features form phenotypes that can be used to determine effective drug therapies, such as those based on nanomedicines. Unfortunately, fluorescence microscopy is time-consuming, expensive, labour intensive, and toxic to the cells. Bright-field images lack these downsides but also lack the clear contrast of the cellular components and hence are difficult to use for downstream analysis. Generating the fluorescence images directly from bright-field images using virtual staining (also known as "label-free prediction" and "in-silico labeling") can get the best of both worlds, but can be very challenging to do for poorly visible cellular structures in the bright-field images. To tackle this problem deep learning models were explored to learn the mapping between bright-field and fluorescence images for adipocyte cell images. The models were tailored for each imaging channel, paying particular attention to the various challenges in each case, and those with the highest fidelity in extracted cell-level features were selected. The solutions included utilizing privileged information for the nuclear channel, and using image gradient information and adversarial training for the lipids channel. The former resulted in better morphological and count features and the latter resulted in more faithfully captured defects in the lipids, which are key features required for downstream analysis of these channels.

1 Introduction

Nanomedicine uptake and effect on fat cells (adipocytes) can be explored using microscopy imaging techniques applied to stem cell derived cell cultures. The lipid droplets within the adipocytes play a key role in metabolism and are implicated in several pathologies, including cancer, diabetes and obesity [1]. They provide fuel for the organism and supply a safeguard for energy fluctuations [2]. Membrane surfaces of lipid droplets can contain hundreds of different proteins (such as perilipins, enzymes and trafficking proteins) that allow them to function as energy repositories and interact with other cellular components [3]. The metabolic demands of nanomedicine transfection can therefore cause physical remodeling of the lipid droplets as the cells respond to the increased energy demands and the resulting phenotypic signatures can be used to evaluate nanomedicine efficacy. Therapies that modify adipocyte phenotypes can also be used to adjust metabolic profiles towards more healthy states by encouraging catabolic lipid processing [4]. Furthermore, using nanoparticle delivery of mRNA to adipocytes, via subcutaneous injection, is a very promising next-generation approach for protein replacement therapies and vaccination [5]. Such mRNA based vaccinations are also currently being developed for Covid-19 [6].

A typical workflow in image cytometry utilizes fluorescent staining and imaging to extract relevant features from cells (such as morphology, count and intensity) and subsequently builds classifiers for various purposes based on these features. Labelling with fluorescent dyes is a well-established and effective method for identifying cellular components. However, imaging fluorescence stains does not come without drawbacks. They require sample preparation, are time-consuming, expensive and labor intensive to develop, and are often toxic to the cells. For example, Hoechst staining is commonly used to visualize cell nuclei because this blue-fluorescent dye binds to DNA, this binding however tends to disrupt normal DNA processing. The fluorescent dyes must also be excited using relatively short high-energy wavelengths resulting in the production of toxic free-radicals. While these problems can be mitigated using experimental design, they are impossible to avoid completely, particularly when performing time-lapse imaging experiments with many exposures [7]. If possible it would be better to circumvent the need for these fluorescent stains and short wavelength light exposures. Additionally, there is a limit to the number of cellular components that can be imaged simultaneously for a given sample due to spectral overlap of the requisite fluorescence dyes.

Bright-field images, on the other hand, are significantly easier to acquire, require very little sample preparation, and are not toxic to the cells. Hence, a less invasive method (and the one proposed herein) would be to predict the fluorescence images directly from their corresponding bright-field images. This translation needs to be as accurate as possible to ensure that the downstream analysis from the generated images is very close to that obtained from the real images.

In recent years deep learning has shown impressive results for image reconstruction and enhancement tasks in image cytometry including deblurring, denoising and super-resolution, and has recently shown promise as a virtual staining method [8, 9]. The “in-silico labeling” approach used by [10] is a particularly noteworthy example of the latter. They proposed a U-Net deep learning architecture [11] and inception-inspired modules [12] to generate fluorescence images given transmitted-light images as input. Their model was trained with pixel-wise cross entropy loss. Using only pixel-wise loss, however, their generated images in some instances lacked global coherence [13]. Also noted, in their virtual staining method for histological images, that training using only pixel-wise loss led to good localization but to poor resolution of finer cellular structures. There are many loss functions and architectural choices that can be explored to combat these issues. For instance, for generating crisp, realistic looking images generative adversarial networks (GANs; [14]) have shown great promise. As opposed to computing the losses over pixels, GANs work with losses at the image-level (or across patches of an image) and push the generated images towards the natural image manifold [15]. In its basic form a GAN plays out a zero-sum game between two networks—a generator and a discriminator. The generator creates counterfeit images which it hopes to deceive the discriminator with, whilst the discriminator attempts to correctly classify the real and fake images. Convergence is reached when the discriminator can no longer tell the difference between the images. For image-to-image translation tasks conditional GANs are often applied. The conditional discriminator sees both the input to the generator and the fake or real outputs. This can help alleviate the problem of artifacts that GANs can produce and was used in, for instance, the Pix2Pix algorithm [16]. In the field of virtual staining, conditional GANs have been used by, for example, [17] for histological staining of prostate core RGB images, and by [18] for histological staining of skin, liver and kidney tissue sections. Additional virtual staining methods and applications of note include the 3D U-Net-based approach used in [19], for 3D fluorescence reconstruction, and the dual adversarially trained autoencoder model used in [20]. This latter method used nuclear and cell fluorescence stains to train a first autoencoder, learning a probabilistic encoding for cell and nuclear shape, upon which a second “wedded” autoencoder was used for generating additional subcellular structures, conditional on this encoding.

A relatively new paradigm within machine learning is Learning Using Privileged Information (LUPI; [21]). The key idea of the LUPI paradigm is that the machine learner, the “student”, has an “intelligent teacher” that provides additional (privileged) information during the training phase, to improve and accelerate the learning process. Through LUPI the learner gains a better concept of similarity between training objects and also receives hidden insights (explanations from the teacher) to guide decision rule formation. The privileged information can be anything that potentially aids the networks during the training process, but that is not required or available during inference. The privileged information can help to focus the network’s attention on locations in the data that are difficult to predict. The way in which this is done in deep learning is to develop separate network streams, one for the main task and one based on the privileged information, and then to incorporate some means of guiding the main model with insights from the privileged information stream [22-26].

In bright-field images the nuclei are often not visually apparent and hence generating their fluorescence counterpart can be challenging. If there are significant errors in these generated fluorescence images the downstream segmentation of the nuclei will fail, a step that is crucial for obtaining subsequent measures of nuclear features. Furthermore a common approach for downstream analysis of the images is to use a CellProfiler [27] pipeline in which the nuclei (from the nuclear stain) are used for providing seeds to the segmentation algorithms that delineate the cell boundaries (the cytoplasm). Hence, the nuclear channel is very important and needs careful consideration. As a potential aid to the nuclear model we therefore apply LUPI using segmentation masks as privileged information. To the best of our knowledge, the work in this paper presents the first application of LUPI in virtual staining for image cytometry.

A shortcoming of several virtual staining studies in cytometry is an overly strong comparison of the true and generated images at the image level. For biological relevance the generated images should be compared based on various features extracted from the different cellular components. Evaluating synthetic images based on derived features has been proposed previously in [28]. In this paper, we propose a method to generate fluorescent labels for adipocyte cell images directly from bright-field z-stacks. This is done by constructing three different models, one each for nuclei, cytoplasm, and lipid droplets. The faithfulness of this translation is compared at both the image and feature level, but with more emphasis on the features.

The modelling and results presented in this paper were based on our winning solution to the Adipocyte Cell Imaging challenge hosted by AstraZeneca AB (AZ) and AI Sweden during November 2020. The features evaluated in the CellProfiler pipelines were based on those used to judge the competition. An overview of our proposed method and workflow is shown in Fig 1. A GitHub repository providing the code base for our modelling solutions is available at https://github.com/aktgpt/brevis.

Fig 1

Proposed method and workflow.

Cell cultures are imaged by fluorescence and bright-field imaging. These images are used to train and compare specialized models for generating each stain, based on biologically relevant features. The selected models are then used to virtually stain bright-field images. A. Nuclear virtual staining model which simultaneously learns the nuclear segmentation (the privileged information model, the outer-most decoder path) and the generated image for the nuclear channel (the inner-most decoder path), B. Virtual staining model for the cytoplasm and C. Virtual staining model for lipid droplets, the model utilizes a patch-level discriminator, used for adversarial training.

2 Data and evaluation

The data under consideration consists of adiopocyte cells imaged at 60x magnification. The cells were stained with Hoechst 33342 (blue, for the nuclei), Bodipy (green, for the lipid droplets) and Cell Tracker Deep Red (red, for the cytoplasm) and imaged with a robotic confocal microscope (Yokogawa CV7000). The cells were imaged with a bright-field z-stack of 7 images with an exposure of 150 ms, the fluorescence with an exposure of 300 ms for blue, 300 ms for green, and 500 ms for red with a numerical aperture of the objective of 1.2. The cells were cultured in 8 wells and imaged at 12 different sites. Each image is approximately 2556 × 2156 pixels with a pixel scale of 0.1083 μm and used 16 bits to represent the pixel values. The final dataset (8 wells) consisted of 96 bright-field images, all with corresponding fluorescence images (nuclei, lipid droplets and cytoplasm). The stage of the microscope was kept fixed in the x-y direction during imaging to preserve registration between the bright-field and fluorescence images.

2.1 Data preprocessing

The dynamic range of microscopy images are rarely the entire bit-depth (16 bits). The images were therefore scaled to 0-1 using their dynamic range. Segmentation masks for the nuclei were produced with CellProfiler (see section 2.3). For each identified nucleus the center coordinate was extracted to make it possible to crop patches in the original images containing nuclei.

The data was divided into three different splits, each having 5 wells for training and 2 for validation. Excluded from these three splits was one well that was set aside for final evaluation.

2.2 Image based evaluation

The generated images were compared to the ground truth images based on the mean absolute error (MAE), normalized towards the median of the ground truth images, and the structural similarity index measure (SSIM) [29]. The MAE measures the absolute deviation in pixel intensities between the generated image and the ground truth, for which a lower value is better. SSIM measures the similarity between overlapping sub-windows in the images based on their means, variances and co-variances and returns a value between zero and one, whereby more similar images have a higher value.

2.3 CellProfiler evaluation

From the generated and ground truth images a variety of morphological, intensity and count features were extracted using a CellProfiler pipeline ([27] version 4.06). The different cellular structures were initially identified based on intensity, size and shape, prior to feature extraction. Potential nuclear regions were identified by an adaptive thresholding method. Since the lipids can occlude and alter the appearance of the nuclei a wide range of diameters and shapes were used to filter out irrelevant objects. The objects were then filtered based on shape features to remove incorrectly identified nuclear objects. For the lipid droplets, small and large droplets were identified separately. Since the droplets have similar intensities they are identified by a global threshold across the expected range of diameters for the two groups. Lipids that were clumped together were distinguished based on their intensities. The defects in the lipid droplets were found by morphological reconstruction (a rolling-ball algorithm) where the image is successively eroded and reconstructed to identify dark holes within bright regions. Cytoplasmic regions were identified starting with nuclear objects as seeds. Here the combination of the cytoplasmic images and the lipids (which are contained within the cytoplasm) were utilized by averaging them together. Cell bodies were then identified by an adaptive threshold with a window size larger than that of nuclei. For all cellular structures features were extracted separately and are shown in Table 1.

Table 1

Extracted features from CellProfiler.

	Nuclei	Cytoplasm	Lipids
Morphology	AreaCompactnessForm FactorMean RadiusPerimeterSolidity	AreaCompactnessForm FactorMean RadiusPerimeterSolidity	AreaCompactnessForm FactorMean RadiusPerimeterSolidityGranularity
Intensity	Integrated IntensityMean IntensityStd Intensity	Integrated IntensityMean IntensityStd Intensity	Integrated IntensityMean IntensityStd Intensity
Count	Count Nuclei	Count Cells	Count Defective LipidsCount Lipids

Compactness is a measure of irregularities and holes in the object, where a filled circle will have a score of 1 and more irregular objects will have a score grater than 1. Form Factor measures the circularity of objects. Mean Radius measures the mean distance of any pixel in the object to a pixel outside of the object. Solidity measures the proportion of the area of the object against the convex hull. Granularity is a measure of size distribution over a fixed set of scales and gives one value per scale. Integrated Intensity is the sum of the pixel intensities within an object. Mean Intensity is the average pixel intensity within an object. Std Intensity is the standard deviation of the pixel intensities within an object.

The value for each feature is computed as the average over the objects in the image. The features were then compared to the features extracted from the ground truth by the MAE and Spearman correlation (denoted by ρ). The absolute error in each case was calculated with respect to the median values of the ground truth, expressed as where is the median value of the target. The correlation is calculated as the the average value across the three splits, calculated by a Fisher transform. The score of each feature group (morphology, count and intensity) is then calculated as the mean of the MAE of the features in each group.

3 Model descriptions

3.1 Base model

The foundation of the proposed models is a dense U-Net [30]. Like the traditional U-Net [11] this model consist of a contracting/downsampling path (encoder) and an expanding/upsampling (decoder) path, with concatenating skip connections between the two. These skip connections aid the upsampling path in recovering finer grained spatial information from the downsampling path. The dense U-Net extends this basic U-Net architecture through the use of dense blocks consisting of batch-normalization—ReLU—convolution—dropout operations that are densely connected (within a block) to one another in a feed forward fashion (i.e. there is a skip connection with the output of each quartet of operations to every subsequent quartet). On the contracting path there are also skip connections that hop over each dense block. These dense blocks and additional skip connections result in a network that has multi-scale deep supervision with feature propagation and reuse, permitting the training of deeper networks but with fewer parameters than would otherwise be required (see Fig 2 which illustrates the dense U-Net model). The transition down layers consist of spatial max pooling and the transition up layers pixel-shuffle followed by blurring to reduce checkerboard artifacts in the network outputs, as proposed in [31]. The dense U-Net has shown promising results in, for instance, predicting cell traction forces [32] and image restoration [33].

Fig 2

Illustration of the base model (Dense UNet).

The dense blocks is a series of batch-normalization—ReLU—convolution—dropout that are densely connected. The transition down layer is a spatial max pooling layer and the transition up is a pixel-shuffle followed by a blurring.

3.2 Nuclei model

Since generation of nuclear fluorescence stains based on bright-field images is a difficult task a LUPI-based framework was utilized. This was achieved through adding a supplementary decoder to the base dense U-Net model (see Fig 3). The additional decoder focuses on predicting the segmentation of nuclei and this privileged information is propagated to the image decoder by pixel-wise summation of the intermediate blocks.

Fig 3

Proposed network for the nuclear virtual staining model which simultaneously learns the nuclear segmentation (the privileged information model, the outer-most decoder path) and the generated image for the nuclear channel (the inner-most decoder path).

See Fig 4 for more information on how the information from the segmentation decoder is propagated to the image generation decoder.

Fig 4

Illustration of proposed loss weight module (LWM) and segmentation weight mask (SWM).

Each output from intermediate blocks are passed through a 1×1 convolution and compared to the resized ground truth. The loss map is then fused with the SWM to get the final loss.

Furthermore, inspired by [22] the network employs a multi-scale loss weight module (LWM, see Fig 4) to assign larger weight to the hard pixels. Each decoder block is passed through a convolutional layer to output the nuclear image predictions at each scale. The losses for each intermediate block were then fused with the segmentation weight mask (SWM) to form a weighted loss for the nuclear image. The SWM is employed to assign more weight to the regions where the segmentation is incorrect.

The model maps the input bright-field image stack I to y and y, the regression (based on the nuclear image) and the segmentation respectively. For the segmentation decoder binary cross-entropy loss was used. The SWM Z is defined by Y (output of the segmentation decoder) as where η is the weight assigned to misclassified pixels. The weighted reconstruction/regression loss is thus defined as Similarly, the weighted loss for the outputs of intermediate blocks and resized ground truth is defined as where k = {1, …, N} are the intermediate blocks in the decoder. The final loss for the nuclei channel is then where α is the weight for each loss function.

3.3 Lipid model

Virtual staining for the lipids channel was based on the same model as described in section 3.1. To enhance the performance of this model it was trained with an adversarial training scheme, using a conditional GAN approach, where the input to the discriminator included both the bright-field z-stacks and the fake or real fluorescence images. Hence, the discriminator not only asks whether the generated images look real, but also whether or not they look real conditioned on the bright-field images. Our discriminator also worked at the patch level, rather than at the entire image level, achieved by limiting the receptive field. Working with such patches promotes sharper outputs, better capturing local style characteristics, and to some extent introduces a form of style/texture loss [16]. In combination with this, the models were trained with a Relativistic Average Least Squares GAN (raLSGAN) loss, which is a combination of the relativistic GAN [34] and the least square GAN [35]. In traditional GANs the discriminator estimates the probability that an image is real, whereas in relativistic GANs the discriminator estimates the probability that a real image is more realistic than fake images and conversely, that a fake image is less realistic than a real image. This relativism passes more information to the generator and can significantly improve both generated image quality and training stability. The least square GAN replaces the probability with a least squares loss, i.e. instead of estimating the probability of an image being real or fake it minimizes the least square distance between the two. The losses for the generator and discriminator can be expressed as, Where D is the discriminator and G is the generator.

To complement the adversarial training a mean absolute error loss, , was employed to minimize deviations from the ground truth. To better capture defects in the lipids, a loss function based on (unnormalized) horizontal and vertical image gradients was added. This loss puts extra emphasis into reconstructing edges in the image and is calculated as where x in the generated image and y is the ground truth image. The total loss for the lipid models is then where α is the weight for each respective loss function. See Fig 5 for an illustration of proposed model used for generating the lipid stain.

Fig 5

Proposed model for virtual staining of the lipid droplets.

The base model is extended with a patch-level discriminator, used for adversarial training, in combination with intensity and gradient-based loss functions.

3.4 Cytoplasm model

The cytoplasm model employed the base model described in section 3.1. The red cytoplasmic stain was quite weak and had high intensity variations. Due to these issues, applying adversarial and/or gradient loss did not show significant improvements over the base model and in fact led to artifacts (bright intensity spots for the adversarial training and checkerboard patterns in some locations for the gradient loss). Hence, this model was trained using only loss.

3.5 Pyramidal weighted inference

To handle inference on large images (in this case 2156×2556 pixels) tiling can be applied, whereby the tiles are inferred by the network and then stitched together to obtain the full image. We adapt the pyramidal weighted tiling scheme from [36] and create weight masks for each tile, giving a higher weight to central pixels than to pixels at the image boundaries. The weights are reduced in a pyramidal fashion from the central pixel towards the edges. The inference is then performed on overlapping tiles to reduce GPU memory consumption and to have similar receptive fields during training and inference. The outputs are multiplied by the weight mask prior to averaging to create the full image. The method was applied with a tile size of 512×512 pixels and a stride of 256×256 pixels.

4 Experiments and results

4.1 Model training

For the nuclei model training the loss weights were set to α = 0.1, α = 0.05 and η for the SWM was set to 2. The learning rate was set to 10−3 with batch size of 12. Images were cropped with a size of 512x512 pixels, with a probability of p = 0.8 to be cropped around the center of the nuclei (with random offset) and p = 0.2 to be cropped at a random location. The maximum offset was set to 256 pixels to ensure that at least one nucleus was present in the image. Augmentations were done with horizontal flipping, vertical flipping, and randomly rotating the images with maximum 180 degrees. The models were then trained for 51,400 iterations.

The weights for the losses for the lipid model were set to α = 5, α = 0.01, and . The learning rate for the generator was set to 10−4 and 10−5 for the discriminator. The models were trained with a batch size of 4 and augmentations were performed with random crops (of 512x512 pixels), random horizontal and vertical flipping and random 90 degrees rotations. The models were trained for 30,000 iterations. For the first 1500 iterations the generator was trained in isolation to put it in a stronger starting position before introducing the discriminator.

The model for cytoplasm virtual staining took longer to converge and was thus trained for 75,000 iterations. The model was trained with a learning rate of 10−4 and a batch size of 4, together with the same augmentations as applied for the lipid model.

All the models were trained with AdamW [37] as the optimizer with weight decay of 0.01. The models for the lipids and cytoplasm channels had four dense blocks each in both the encoder and decoder paths. The models for nuclear channel had five blocks in each path, as preliminary experiments showed a higher capacity was beneficial in this case. The dropout rate was set to 0.2 and the growth rate of the dense blocks was set to 12. The best models were saved on the basis of the loss evaluated on the validation set.

4.2 Ablation analysis

To explore the effect of each loss and model configuration on the generated images an ablation study was performed. These effects were evaluated based on the pixel-wise mean absolute error and on the three groups of CellProfiler features (morphology, intensity and count). The scores were calculated for the outputs of the models across the three training-validation splits of the data, and represents the mean absolute error between the generated and ground truth images and the mean absolute error for each group of features. For each loss/model configuration a one-sided Mann-Whitney rank test [38] was performed to evaluate the significance of the improvements.

For the nuclear channel, the effect of three model configurations were evaluated: which uses only the base model described in section 3.1 and MAE as loss function; , which extends the base model by adding another decoder as described in section 3.2 with only MAE and cross-entropy loss as loss functions, and lastly: , which also adds the multi-scale loss modules.

For the lipid droplets the addition of gradient loss and adversarial loss was evaluated against using only the L1 loss.

The effect of the addition of each model/loss configuration for both nuclei and lipids can be seen in Table 2.

Table 2

Effect of each loss/model configuration on the generated images evaluated on the three groups of CellProfiler features for the nuclear and lipids channels.

The scores are the combined mean and standard deviation over the generated images against the ground truth across the three splits.

Channel	Configuration	MAE	CP Morphology	CP Intensity	CP Count	CP Tot
Nuclei	L_1	0.51 ± 0.070	0.05 ± 0.023	0.10 ± 0.038	0.11 ± 0.072	0.07 ± 0.022
	L_1+L_BCE	0.46 ± 0.048	0.05 ± 0.018	0.11 ± 0.051	0.08 ± 0.058	0.07 ± 0.021
	L_1+L_BCE+L_LWM	0.49 ± 0.049	0.03 ± 0.017	0.14 ± 0.070	0.07 ± 0.057	0.07 ± 0.024
Lipid droplets	L_1	0.14 ± 0.030	0.19 ± 0.028	0.42 ± 0.207	0.49 ± 0.189	0.26 ± 0.057
	L_1+L_grad	0.14 ± 0.030	0.18 ± 0.038	0.38 ± 0.213	0.45 ± 0.184	0.24 ± 0.059
	L_1+L_grad+L_adv	0.14 ± 0.029	0.16 ± 0.038	0.27 ± 0.158	0.25 ± 0.162	0.19 ± 0.047

For the nuclear channel, a performance improvement was seen in count and morphology features with the LWM (see Table 2). These improvements were significant against the base model (p < 0.05 for count; p ≪ 0.001 for morphology). The LWM was also significantly better for morphology when compared against the LUPI-based model without the LWM (p ≪ 0.001). The LWM focuses the model towards better shape outlines and punishes it for mislocation of nuclei (see Fig 6). This could be due to the fact that the LWM punishes errors in segmentation by increasing the weights of the reconstruction loss in those regions, hence the loss in correctly classified regions gets less attention as the network becomes better at segmentation.

Fig 6

A comparison between the LWM based model (applying LUPI) and the base model for virtual staining of the nuclear channel.

(A) Ground truth fluorescence image, (B) Output from LWM model (C) Output from the base model. The arrows in (C) point out a missing nucleus and an incorrectly shaped nucleus for the base model’s prediction. Images are displayed with the same dynamic range for visualization purpose.

As can be seen in Table 2, for the lipid droplets, an overall performance improvement was gained at the feature level with the inclusion of each additional loss. The gradient loss gave a significant improvement in morphology (p < 0.05), whereas for intensity and count the improvements were not significant (p = 0.122 and p = 0.135 respectively). The largest overall improvement came with the combination of both gradient and adversarial loss (p ≪ 0.001 for all feature groups), where the largest change for the better was for the count features. Both the adversarial and the gradient loss assist in the delineation of the lipids in the reconstruction of the defects within some of the lipids, relative to the base model, as can be seen in Fig 7.

Fig 7

Comparison of lipid defect reconstructions between the models.

(A) Ground truth fluorescence image, (B) Output from the model trained with , (C) Output from the model trained with . In (B) there is a better delineation of the lipids and an improved reconstruction of the lipid defects/dents than in (C). Images are displayed with the same dynamic range for visualization purpose.

An observation from the ablation studies, for both the nuclear and lipids channels, was that the pixel-wise MAE did not capture the full picture of model performance. For the lipid droplets, training without the adversarial loss gave the best MAE score. As this metric only summarises the closeness between the generated and ground truth images, it cannot capture all the aspects of the image reconstruction fidelity that may be of interest. This metric is also based on both the foreground and background in the images. In many cases the background is of little or no interest. On the other hand, the feature based metrics we explored are based solely upon on the foreground/objects of interest. The feature set was based upon the cell-level features that biologists routinely extract from such data, and hence they provide much better statistics with which to root model comparison and selection. We based our model choices primarily upon the feature level comparisons.

4.3 Final evaluation

The best performing model configurations were evaluated on the held out test set. The averages and standard deviations for the scores across the three models (from each training/validation split) are presented in Table 3. For the feature measures the Spearman correlation is also given. Radar plots, showing the models’ performance for each features are displayed in Fig 8, where the mean values for the ground truth and the generated images are displayed along with their correlations. Note that the mean granularity for the lipid droplets is the mean of the granularity measures across the different scales.

Table 3

Evaluation of the models for the three channels on the held out test set, where ρ denotes the Spearman correlation averaged across the results of the three models (from each training/validation split).

Channel	MAE	SSIM	CP Morphology		CP Intensity		CP Count		CP Tot
	Score	Score	Score	ρ	Score	ρ	Score	ρ	Score	ρ
Nuclei	0.47 ± 0.016	0.98 ± 0.001	0.04 ± 0.007	0.91	0.16 ± 0.070	0.58	0.09 ± 0.012	0.82	0.10 ± 0.010	0.77
Lipid droplets	0.14 ± 0.006	0.99 ± 0.001	0.16 ± 0.005	0.65	0.29 ± 0.010	0.57	0.210 ± 0.02	0.47	0.19 ± 0.001	0.52
Cytoplasm	0.18 ± 0.002	0.99 ± 0.000	0.09 ± 0.009	0.78	0.13 ± 0.022	0.68	0.15 ± 0.013	0.83	0.11 ± 0.012	0.59

Fig 8

Individual feature scores for the models evaluated on the test set.

The plots shows the mean of each feature across the images and display the results for both the generated and ground truth fluorescence images. The plots also include the Spearman correlation for each feature. Top row represents morphology and count features and bottom row shows intensity features. For the lipids: (D) lipids with defects; and (L) all lipid droplets.

A qualitative comparison of the results on the test data can be seen in Fig 9 which shows that the proposed models generate the fluorescent stains accurately. Fig 9A and 9D shows the generated and ground truth for the nuclei. As can be seen in the maximum projection of the bright-field z-stack (Fig 9G) generating the nuclei from the bright-field images poses significant challenges—nuclei not surrounded by lipid droplets are barely visible. However, the proposed method manages to find these nuclei well (see for instance right most nuclei in Fig 9A.1). The model also faithfully captures the shape of the nuclei, which is supported by the results in Table 2 and Fig 6, where the LWM improves significantly over the baseline model on the shape features. However, difficulties in predicting the nuclei can be seen also in Fig 9A.1 where the model misses a nucleus in a location where one should be present. Hence, although this problem of missing nuclei is reduced when using the LWM, as opposed to the base model, it has not been entirely eliminated. Occasionally, also the opposite problem occurs, where the model predicted a nucleus in a location where no nucleus was present. These errors tended to occur in small gaps in densely clustered lipid regions where nuclei are often located.

Fig 9

Comparison of images generated from the bright-field z-stack and the ground truth fluorescence images.

A-C. Generated images for the nuclei, lipid droplets and cytoplasm, with zoomed in regions showing some well reconstructed areas alongside some more problematic locations. D-F Ground truth images with corresponding zoomed in regions. G Maximum projection of the bright-field z-stack with corresponding zoomed in regions. Images are displayed with the same dynamic range per channel for visualization purpose.

Fig 9B and 9E show the generated and ground truth images for the lipid droplets. Since the droplets are quite visible in the bright-field (see for instance Fig 9G.2) this reconstruction task is significantly simpler. The main difficulties lies in reconstructing the droplets internal structures, such as the small defects. From Fig 7 and Table 2 the reconstruction of these defects improved significantly with the gradient based loss and the adversarial training.

The results for the cytoplasmic stain can be seen in Fig 9C which show that the model manages to predict the cytoplasmic stain with high fidelity. Fig 9F.1 and 9F.2 contain some examples of intensity variations that the model does not manage to recreate. This is not surprising as there is very little information about these variations in the bright-field data. As such large intensity fluctuations are rather sparse in the cytoplasm images, modelling approaches that try to accommodate for them, such as those including gradient and adversarial loss, may lead to hiccups where the models try to recreate such variations where they should not be.

Quantitatively we see a good correlation between features extracted from the ground truth images and the reconstructed images (Table 3). Looking at the individual feature scores for the reconstructed nuclei (Fig 8) the morphological and count features line up well with those extracted from the ground truth. The main difference being in the intensity features where the intensity from the generated images is (on average) lower than that from the ground truth images. This relates back to the problem of recreating the internal structures of the nuclei, where the generated nuclear objects tended to have a more uniform intensity.

From Fig 8, for lipid droplets, the count features from the generated images were lower an average. Although adding gradient loss and adversarial training improved the model for this channel, it still missed some of the defective lipids, which consequently resulted in a lower integrated intensity. The area features based on the generated images were higher on average. This is probably due to the fact that smaller lipids were often missed in the generated images (see Fig 7). Apart from these examples the model performs almost perfectly in the remaining feature categories.

Since the cytoplasm segmentation in the CellProfiler pipeline is bound to both the nucleus and the lipid droplets the scores for this channel are effected by the performance on these other two channels. This effect can be seen for instance in Fig 8, where the compactness score for the generated images is lower than for the ground truth, i.e. it predicts more compact objects with less irregularities and holes.

5 Conclusion

In this work we carefully considered the challenges posed by each imaging channel to tailor our solutions, whilst also fine-tuning the model selection based on the ablation analysis. The most tailored solution was for the nuclear channel. In many cases the nuclei have only a ghost-like presence in the bright-field images (see Fig 9). We therefore chose to apply the LUPI paradigm to guide the model for this channel, using segmentation masks as privileged information to guide for the predictions. This guidance led to significantly improved results which is in line with related work using LUPI for image related tasks [22-26].

For the lipids channel we found that adding gradient and especially adversarial loss gave significant improvements in delineating the lipids and in reconstructing the defective parts within them (Figs 7 and 9). Defective lipids become caved in, with intensity depressions, and are often clustered together inside the same cell—an active cell. Locating and quantifying these defects is an important ingredient of the nanomedicine drug discovery process, hence faithfully reconstructing these intensity depressions with our models is crucial.

Although we gave attention to how well our models performed at the image-level, our main focus was on creating models that were most faithful at the extracted feature-level. Indeed, in several cases the models that gained the highest score at the image level, based on the MAE, were not those that performed best at the feature-level. As downstream analysis from fluorescence images is based on the derived features, including the design of effective drug therapies, we believe that this focus is of higher benefit when applying deep learning to the task of creating virtually stained images.

As presented in this paper, generating the nuclear, cytoplasmic and lipid droplet images directly from bright-field images liberates the fluorescence channels so that, if some perturbation of the cells can be tolerated, they can be used to visualize other aspects of cell physiology (proteins, lipid subsets, molecular reporters etc.). In the context of nanoparticles and vaccines, for example, this enables additional measurements of particle uptake, cargo delivery and functional measurements [39]. Virtual staining also delivers results at a fraction of the cost and time required for traditional fluorescence imaging [13]. As our trained models are lightweight, and generate the virtually stained images relatively quickly, they could potentially be directly integrated into the image acquisition software connected to the microscopes to essentially enable “augmented microscopy” [40].

Transfer Alert

This paper was transferred from another journal. As a result, its full editorial history (including decision letters, peer reviews and author responses) may not be present.

30 Jun 2021

PONE-D-21-15339

Learning to see colours: generating biologically relevant fluorescent labels from bright-field images

PLOS ONE

Dear Dr. Harrison,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Jul 26 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Chi-Hua Chen, Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We note that Figure(s) 6, 8, and 9 in your submission contain copyrighted images. All PLOS content is published under the Creative Commons Attribution License (CC BY 4.0), which means that the manuscript, images, and Supporting Information files will be freely available online, and any third party is permitted to access, download, copy, distribute, and use these materials in any way, even commercially, with proper attribution. For more information, see our copyright guidelines: http://journals.plos.org/plosone/s/licenses-and-copyright.

We require you to either (1) present written permission from the copyright holder to publish these figures specifically under the CC BY 4.0 license, or (2) remove the figures from your submission:

a. You may seek permission from the original copyright holder of Figure(s) 6, 8, and 9 to publish the content specifically under the CC BY 4.0 license.

We recommend that you contact the original copyright holder with the Content Permission Form (http://journals.plos.org/plosone/s/file?id=7c09/content-permission-form.pdf) and the following text:

“I request permission for the open-access journal PLOS ONE to publish XXX under the Creative Commons Attribution License (CCAL) CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). Please be aware that this license allows unrestricted use and distribution, even commercially, by third parties. Please reply and provide explicit written permission to publish XXX under a CC BY license and complete the attached form.”

Please upload the completed Content Permission Form or other proof of granted permissions as an "Other" file with your submission.

In the figure caption of the copyrighted figure, please include the following text: “Reprinted from [ref] under a CC BY license, with permission from [name of publisher], original copyright [original copyright year].”

b. If you are unable to obtain permission from the original copyright holder to publish these figures under the CC BY 4.0 license or if the copyright holder’s requirements are incompatible with the CC BY 4.0 license, please either i) remove the figure or ii) supply a replacement figure that complies with the CC BY 4.0 license. Please check copyright information on all replacement figures and update the figure caption with source information. If applicable, please specify in the figure caption text when a figure is similar but not identical to the original image and is therefore for illustrative purposes only.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: PONE-D-21-15339

In this manuscript, the authors apply the established approach of label-free microscopy to an adipocyte dataset with two . They modify previous deep-learning approaches using a variation on deep learning called Learning Using Privileged Information. They also use a feature-based approach to evaluate generated images, rather than the pixel-based approaches used in previous label-free work. There are a three significant issues with this manuscript that need to be addressed. First, relevant prior work is not cited (as itemized below). Second, the authors did not compare their approach to current state-of-the-art methods Without these comparisons it is unclear whether or not the manuscript contributes an advance to the field. Third, the images used in the study are not being made publicly available as required by journal policy.

Major issues:

1. The title, abstract and introduction are largely written as if the authors invented the approach of label-free microscopy, which is not correct. All should be modified to make clear the relationship of their work to prior work clear.

2. The authors briefly mention Christiansen et al 2018 but do not mention the even more relevant papers from the Allen Institute for Cell Science (Johnson et al (2017) [https://arxiv.org/abs/1705.00092] and Ounkomol et al (2018) [https://doi.org/10.1038/s41592-018-0111-2]. They do not compare their approach to these state-of-the-art methods, which could been done either by applying the open source Allen Institute for Cell Science software to their images, or by applying their method to the publicly available images used by Ounkomol et al.

3. The idea of using features to evaluate synthetic images has been described previously by Zhao & Murphy (2007) [https://doi.org/10.1002/cyto.a.20487].

4. While using features is appropriate, the authors should also calculate pixel-wise correlations in order to be able to compare their results to the previous work.

5. In order to enable others to reproduce and possibly improve upon the work and to comply with PLoS ONE policy, all images used in the study should be made publicly available. This reviewer strongly urges that the manuscript not be published without making the images freely available.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

24 Jul 2021

Dear reviewer,

Thank you for carefully going through our manuscript. Please find our responses to your comments below. In the revised manuscript we have made updates in line with your comments.

Best regards, Philip J Harrison

PONE-D-21-15339

Learning to see colours: generating biologically relevant fluorescent labels from bright-field images

Review Comments to the Author

Reviewer #1: PONE-D-21-15339

In this manuscript, the authors apply the established approach of label-free microscopy to an adipocyte dataset with two . They modify previous deep-learning approaches using a variation on deep learning called Learning Using Privileged Information. They also use a feature-based approach to evaluate generated images, rather than the pixel-based approaches used in previous label-free work. There are a three significant issues with this manuscript that need to be addressed. First, relevant prior work is not cited (as itemized below). Second, the authors did not compare their approach to current state-of-the-art methods Without these comparisons it is unclear whether or not the manuscript contributes an advance to the field. Third, the images used in the study are not being made publicly available as required by journal policy.

Major issues:

1. The title, abstract and introduction are largely written as if the authors invented the approach of label-free microscopy, which is not correct. All should be modified to make clear the relationship of their work to prior work clear.

We did not intent to make it sound like we had invented label-free microscopy. We have added text in the abstract and references to additional articles (see response to your next comment) to deal with this concern.

2. The authors briefly mention Christiansen et al 2018 but do not mention the even more relevant papers from the Allen Institute for Cell Science (Johnson et al (2017) [https://arxiv.org/abs/1705.00092] and Ounkomol et al (2018) [https://doi.org/10.1038/s41592-018-0111-2]. They do not compare their approach to these state-of-the-art methods, which could been done either by applying the open source Allen Institute for Cell Science software to their images, or by applying their method to the publicly available images used by Ounkomol et al.

We agree that these two papers should have been cited in our manuscript. The Johnson et al paper we had not previously come across. Thank you for pointing out this paper to us, their method was interesting. We have added references to these two paper (see lines 66-72).

Concerning comparing our method to other state-of-the-art methods: As we tailored our methods to each imaging channel and our specific adipocyte image data, focusing on the derived features, and were not trying to devise a general method that would work for all virtual staining, we believe it would be difficult to devise a fair comparison. We can not directly apply these alternative methods directly to our dataset without making modifications to them. For instance, the Allen Institute for Cell Science software was developed for 3D reconstructions, not 2D as was our case. We have updated the text on line 102 to emphasize that we have build this method specifically for our adipocyte cell image data.

It is also the case that our method was actually compared against 7 other state-of-the-art methods in the competition organised by AstraZeneca and AI Sweden that we won. The 7 other teams used a variety of different and current approaches for the virtual staining. AstraZeneca and AISweden are in the process of writing a paper about the competition and the modelling solutions used by the competing teams. Unfortunately, this paper has not yet been completed nor published.

3. The idea of using features to evaluate synthetic images has been described previously by Zhao & Murphy (2007) [https://doi.org/10.1002/cyto.a.20487].

Thank you for bringing this paper to our attention, we have included a reference to it on lines 100-101.

4. While using features is appropriate, the authors should also calculate pixel-wise correlations in order to be able to compare their results to the previous work.

See previous response concerning comparing to other methods. We do however still report our pixel-level performance (MAEs) in Tables 2 and 3.

5. In order to enable others to reproduce and possibly improve upon the work and to comply with PLoS ONE policy, all images used in the study should be made publicly available. This reviewer strongly urges that the manuscript not be published without making the images freely available.

We would really like to make this data available, but unfortunately there are legal restrictions that make this impossible at the current time. At present access to the data can be requested from AI Sweden, although a sample of the data is freely available at https://www.ai.se/en/node/81535/adipocyte-cell-imaging-challenge.

Submitted filename: Responses to Reviewer.pdf

Click here for additional data file.

13 Aug 2021

PONE-D-21-15339R1

Learning to see colours: generating biologically relevant fluorescent labels from bright-field images

PLOS ONE

Dear Dr. Harrison,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Sep 27 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Chi-Hua Chen, Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors have made a few welcome additions based upon my previous comments. They have decided not to compare their method to previous methods. While I understand their choice, in the absence of such a comparison the methodological significance of the work remains unknown. Therefore the work is best described as the application of an existing approach to a different cell type. I therefore strongly feel that the title is not appropriate and should be changed to something like “Development of label-free microscopy models for adipocytes”.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

23 Aug 2021

RESPONSE TO EDITOR

I am pleased to resubmit the original paper by Håkan Wieslander, Ankit Gupta, Ebba Bergman, Erik Hallström and Philip J Harrison for publication consideration in PLOS ONE. This paper was originally entitled “Learning to see colours: generating biologically relevant fluorescent labels from bright-field images”, but in response to the reviewer’s suggestion we have now changed the title to “Learning to see colours: biologically relevant virtual staining for adipocyte cell images”.

We have reviewed our reference list to ensure that it is complete and correct. None of the cited papers have been retracted.

We hope you find our revised manuscript suitable for publication in PLOS ONE.

RESPONSE TO REVIEWER

We agree, in light of your comments, and those you made on the first draft of our manuscript that a less general title would be better. We have therefore changed the title of our manuscript to “Learning to see colours: biologically relevant virtual staining for adipocyte cell images”.

Submitted filename: 2nd_responses_to_reviewer.pdf

Click here for additional data file.

30 Sep 2021

Learning to see colours: biologically relevant virtual staining for adipocyte cell images

PONE-D-21-15339R2

Dear Dr. Harrison,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Chi-Hua Chen, Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

7 Oct 2021

PONE-D-21-15339R2

Learning to see colours: biologically relevant virtual staining for adipocyte cell images

Dear Dr. Harrison:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Professor Chi-Hua Chen

Academic Editor

PLOS ONE