Single-Image Super-Resolution Improvement of X-ray Single-Particle Diffraction Images Using a Convolutional Neural Network.

Femtosecond X-ray pulse lasers are promising probes for the elucidation of the multiconformational states of biomolecules because they enable snapshots of single biomolecules to be observed as coherent diffraction images. Multi-image processing using an X-ray free-electron laser has proven to be a successful structural analysis method for viruses. However, the performance of single-particle analysis (SPA) for flexible biomolecules with sizes ≤100 nm remains difficult. Owing to the multiconformational states of biomolecules and noisy character of diffraction images, diffraction image improvement by multi-image processing is often ineffective for such molecules. Herein, a single-image super-resolution (SR) model was constructed using an SR convolutional neural network (SRCNN). Data preparation was performed in silico to consider the actual observation situation with unknown molecular orientations and the fluctuation of molecular structure and incident X-ray intensity. It was demonstrated that the trained SRCNN model improved the single-particle diffraction image quality, corresponding to an observed image with an incident X-ray intensity (approximately three to seven times higher than the original X-ray intensity), while retaining the individuality of the diffraction images. The feasibility of SPA for flexible biomolecules with sizes ≤100 nm was dramatically increased by introducing the SRCNN improvement at the beginning of the various structural analysis schemes.

Introduction

Femtosecond X-ray free-electron lasers (XFELs)[1,2] are attracting attention as new probes for the elucidation of the multiconformational states of biomolecules at room temperature[3,4] because of their ability to measure a variety of instantaneous molecular structures as a snapshot. Although serial femtosecond crystallography has proven to be an effective method with low radiation damage,[5−7] the removal of the limitation of sample crystallization in single-particle analysis (SPA) remains challenging. Currently, three-dimensional (3D) assembled structures are reconstructed by processing multiple images obtained from relatively large, complex, and highly symmetric viruses.[8−10] A key challenge in this area is realizing the SPA of nanoscale flexible biomolecules with sizes ≤100 nm, such as ribosomes, nucleosomes, and membrane proteins, which are particularly important for drug discovery. The main difficulty that makes SPA of 100 nm size biomolecules complicated for processing multiple images is the significantly low signal-to-noise (S/N) ratio of the observed diffraction images caused by the low scattering ability of the biomolecules. An additional difficulty in processing multiple images is that flexible biomolecules exhibit a variety of conformations. Various experimental and analytical methods have been proposed to obtain better diffracted images for biomolecular imaging applications,[11−20] such as the use of 100 nm focusing mirrors.[21]

In SPA, a single biomolecule is injected into a vacuum with an unknown molecular orientation and possibly different conformations for nanoscale flexible biomolecules. When the X-ray hits the sample well within the appropriate beam position, a snapshot of the instantaneous structure of a single biomolecule is acquired as a diffraction image without phase information. This image is observed by a two-dimensional (2D) charge-coupled device (CCD) detector as pixel-wise information on the photon counts of a solid angle. By discretizing in units smaller than a solid angle of the reciprocal of the molecular size in wavenumber space, which is called the oversampling ratio, the phase information is retrieved by an algorithm such as the hybrid-input–output (HIO) algorithm.[22] However, the curvature of the Ewald sphere and noise affect the convergence of the phase retrieval calculation. In practical experiments, the high oversampling ratios required for phase recovery have significantly reduced counting photons per pixel. If the requirement of high oversampling ratios is relaxed and observation is allowed at a larger solid angle, the number of achievable photons per pixel will increase significantly and noise will be reduced.

In image processing, in addition to general image complementation methods, super-resolution (SR) imaging, which estimates high resolution (HR) images from low resolution (LR) images, has been proposed. In recent years, research on single-image SR using convolutional neural networks (CNNs) has remarkably advanced. Dong et al.[23] developed a super-resolution convolutional neural network (SRCNN), which employed CNN layers and achieved SR with higher accuracy than previous methods. Since the development of the SRCNN, various methods have been proposed, such as deeper network architectures,[24,25] methods for speeding up,[26] improvements on loss functions such as perceptual loss and training strategies,[27] and generative adversarial network (GAN)-based methods.[28,29] Most of these methods are based on supervised learning, where an artificially reduced LR image is created from a given ground truth high resolution (GTHR) image, and the system is trained to recover the original GTHR image from the LR image. Although these proposed methods have improved the performance of SR, it has been pointed out that neural network-based methods, especially GANs, do not necessarily reproduce the original image and generate artifacts.[30,31] Recently, unsupervised or weakly unsupervised learning,[32,33] where SR models are trained without corresponding HR images or SR is on few images,[34] has also been proposed.

In the 3D assembled structure reconstruction of viruses,[8−10] improvement of the S/N ratio of the diffraction images and orientation recovery were realized simultaneously using the expand–maximize–compression algorithm,[35−37] a statistical algorithm for maximizing posterior probability as a multi-image processing technique. As conventional 3D reconstruction methods[38] use many images to estimate an assembled 3D structure with a resolution higher than that provided by a single 2D diffraction image, SRCNN has shown promising results in computer vision for a variety of single images. Single-image SRCNN may be effective in single-particle biomolecular analysis. It could improve the quality of single 2D diffraction images from LR to HR and simultaneously solve the problems of noisy observation data and multiconformational states of nanoscale flexible biomolecules. However, incomplete and chaotic experimental data with unknown molecular orientations persist, making CNN training difficult. The preparation of large-scale annotated data in real-world space is essential for supervised learning.

This study is the first investigation of the possibility of using single-image SR techniques for X-ray diffraction image improvement to overcome the difficulties associated with the SPA of flexible biomolecules with sizes ≤100 nm. To compensate for the incomplete and chaotic experimental data describing nanoscale flexible biomolecules, we propose applying synthetically generated labeled data in silico using molecular dynamics simulations and a virtual X-ray diffraction image simulator. These virtual diffraction image data help employ various SR methods on the basis of the supervised learning framework. To avoid excessive artifacts here,[30,31] we use the SRCNN model, which is a relatively simple network. The training of network models using labeled synthetic data has been adapted to a variety of methods and has been reported to improve performance, increase generalization performance, and reduce the amount of real-world data used in training.[39−41] If an improved diffraction image is obtained from an individual noisy diffraction image by the trained SRCNN model in silico, the applicability of SPA to nanoscale flexible biomolecules is dramatically increased. For example, the number of necessary diffraction images for the 3D assembled structure reconstruction is reduced. Moreover, the multiconformational states of nanoscale flexible biomolecules are evaluated by using improved individual 2D diffraction images, which reflect the instantaneous structures of biomolecules, as input data of a pipeline for flexible 100 nm biomolecule structural analysis schemes, such as the HIO method for 2D real image reconstruction and the diffraction template matching method for the estimation of plausible 3D structural models (see Figure ).[42,43]

Figure 1

Schematic of a pipeline for 100 nm biomolecule structural analysis with single-image SRCNN (see Section S1).

Section describes the methods for the construction of SRCNN models with data preparation in silico and for the quantitative evaluation of image improvement. Section presents the main results of the diffraction image improvement achieved by the constructed SRCNN model, effects of the structural and incident X-ray intensity fluctuations on diffraction image improvement, and real image analysis by a phase retrieval algorithm. Section discusses the main contributions and the limitations of this work and topics for future research.

Methods

To avoid a practical problem for supervised learning of data labeling between LR images and HR images, data preparation in silico was actively used here. A mixed data set considering realistic experimental conditions,[44,45] incident X-ray intensity fluctuations, structural fluctuations, and various molecular orientations during observation was created in silico. Through this, we constructed a trained SRCNN model that significantly improved the X-ray diffraction image quality for nanoscale flexible biomolecules

Data Preparation and Image Data Set

As a test case, we conducted studies on X-ray diffraction image improvement for a 70S-ribosome molecule. Here, all diffraction image data sets were prepared by simulation using the structural models generated by the molecular dynamics simulation as described in Section S2. We simulated two types of X-ray diffraction images: EMV images as GTHR images and virtual experimental diffraction images as LR images, in which the quantum noise effect was considered. The latter was termed the experimental images in this study. These diffraction images were prepared using the diffraction image simulator (see Section S3) and considering the curvature of the Ewald sphere.

Considering the wide dynamic range of the diffraction images and possible zero values at some pixels, we added one to each pixel value and took the logarithm as a preprocessing step to obtain tractable numbers. The SRCNN output data were postprocessed to get a normal diffraction image by inverse conversion. Since our experimental images are created on the basis of Poisson noise distribution, one would argue that further image quality improvement may be achieved by applying a variance-stabilization scheme such as the Anscombe transformation. We found that the application of the Anscombe transformation before the logarithmic transformation results in almost the same quality of SRCNN transformed images (see Figure S1).

We employed several subsets of the virtual images for systematic model training and evaluation. The training images were used to update the model parameters through backpropagation. The validation images were used to monitor the training process. The test images were used to evaluate a trained model. The details of the preparation of these images are described in Section S5. Herein, we only emphasized that these image sets are carefully prepared; there is no overlap between different image sets. Moreover, there is no accidental bias in the distribution of molecular structure, X-ray intensity, and noise pattern between different image sets. In this way, the objectivity of the evaluation of the trained model is ensured.

In training the SRCNN model, partial images with 44 × 44 pixels were cropped from the training and validation images. From the training images, K = 64 patches that partially overlap each other were taken periodically at regular intervals from each image. For the validation images, K = 30 partial images were randomly selected from each image.

We trained the SRCNN model (see Section S6) using various combinations of training and validation data sets, which were indexed by Data set ID. Table S1 lists the 18 data sets employed in this study. Data set 18 was used for hyperparameter optimization (see Table S2).

Adopted Network Model and SRCNN Training

The SRCNN technique achieved high accuracy by replacing dictionary-based SR with a CNN. Figure S2 shows a diagram of an SRCNN. The network was composed of three convolution layers. We examined the single-channel (1ch) and the two-channel (2ch) models. The 1ch and 2ch models took a gray scale diffraction image data and a two-channel image as input, respectively. The former output was another gray scale image. In the first channel in the input, a gray scale image data was stored. In the second channel, the incident X-ray intensity value was stored. While there were several ways to define the value of input channel 2 in the 2ch model, we used I in eq S1 here. Our preliminary calculations revealed that the 1ch model exhibited more significant losses than the 2ch model. The loss function of the validation data tended to remain high after the loss function of the training data became small. We speculated that this was because the patterns of diffraction images change significantly depending on the incident X-ray intensity. Therefore, we employed the 2ch model in this study because of the intensity dependence of the diffraction pattern. The network was implemented using Keras (ver2.2.4),[46] a widely used framework for constructing deep learning models.

The training/validation/test data were used to train and evaluate the SRCNN model using the hold-out method. The mean square error (MSE) of partial images with 44 × 44 pixels with respect to the EMV images was adopted for the loss function. For the convolution of the intermediate layer, zero-padding was adopted to ensure that the input and output image sizes were the same, and a rectified linear unit was used for the activation function in the convolution layers. Model parameters were randomly initialized with the Glorot uniform distribution. Mini-batch training with a batch size of 128 was performed for up to 100 epochs using the early stopping method. The Adam optimizer[47] with a learning rate of 0.0003 was used for the parameter update. The iteration was terminated when the loss function did not improve five times in a row. After termination, the parameter set with the lowest loss value of the validation data was adopted as the trained model. In the test phase, an experimental image was input to the trained network model, and the output was an SRCNN image as the predicted HR image. Then, the obtained SRCNN images were evaluated by the R with respect to the corresponding EMV images calculated using the similarity detection algorithm (see Section ).

Hyperparameter optimization was performed (see Figure S3) to improve the generalization performance using a data set with various incident X-ray densities. A small set of images was employed in hyperparameter optimization (data set 18 in Table S1) to save computational cost. There is no overlap between the training, validation, and test image sets. The maximum value of ΔR̅ was 0.1598 for f1 = 5, f2 = 1, f3 = 3, n1 = 256, and n2 = 64. The results represented in the following sections were obtained using these parameter values.

Similarity Detection Algorithm

The similarity of a pair of diffraction images was evaluated using the following similarity determination algorithm[48,49] on the basis of the correlation function.[38,50] We introduced a noise reduction mechanism to reduce the influence of quantum noise by the integral correlation patterns I(k, α) in and α. Here, s was the number of observed photons described in eq S1, s̅ was the expected photon number, Nξ was the discrete number of pixels on the concentric circles, ξ = 2θ represented the scattering angle, and λ was the incident X-ray wavelength. The rotation α of the molecule with respect to the incident X-ray axis appeared as a rotation around the center of the diffraction intensity image. By considering α, the correlation pattern was immediately obtained to acquire 360° rotating correlation coefficients to the 2D plane. Here, the correlation pattern c (ξ, α) of a pair of diffraction images I and j was defined by the following equations:

Furthermore, we defined an integral correlation pattern using the following equations:where α̂ is the direction of the highest correlation value in the integral correlation pattern. The score is defined as the maximum value of I at k̂ in the direction of α̂, i.e., I(k̂, α̂). When one performs the normalization with the autocorrelation term of each diffraction image, the similarity of the pair of diffraction images is quantified as the R(k̂, α̂) score.Here, R(k̂, α̂) was denoted as R for simplicity. In particular, when image j was the corresponding EMV image of image i, we termed R(k̂, α̂) as the R score of image i.

Results and Discussion

Diffraction Image Improvement by SRCNN

To evaluate the diffraction image improvement achievable by using an SRCNN, training was conducted using various training data sets with different molecular orientations and incident X-ray intensities (A: 5 × 1012; C: 1 × 1013; E: 5 × 1013; F: 1 × 1014 photons pulse–1 μm–2, corresponding to data sets 1, 3, 5, and 7, respectively, in Table S1). As an example, the learning curve for the intensity of 1 × 1013 photons pulse–1 is shown in Figure S4, which confirms that the loss function sufficiently converges during both validation and training. The validation loss reaches the value near the optimum one within a few epochs. To the best of our knowledge, sometimes it occurs in the case of insufficient training data. We confirmed that by saving the model parameters at each epoch separately and constructing SRCNN models at all epochs, R of test images also drop rapidly. R scores of SRCNN images obtained by the SRCNN model with randomly initialized parameters are extremely low (the average value of R 0.021 was compared to 0.063 for experimental images). Figure presents the results obtained in the test cases at four different incident X-ray intensities and the histogram of pixel intensity. The experiment, SRCNN, EMV, and Poisson-NLPCA[51,52] (nonlocal principal component analysis) images (Section S9), a baseline for a conventional denoising method, were shown at each incident X-ray intensity. Here, the images created in the simulation considering the quantum shot-noise effect were called the experimental images. Because the diffraction X-ray intensity was a function of the wavenumber with considerable attenuation (i.e., it took a very wide range of values with a long tail), all diffraction images were displayed on a log scale. When one focuses on the high-wavenumber region, the experimental diffraction images had zero or single-photon counts for most pixels because the diffraction intensity was insufficient. This trend can also be observed in the histogram. Moreover, in the experimental images, the continuous speckle pattern was only observed in the central regions of the images. However, in the SRCNN images, continuous speckle patterns could be seen from near the centers of the images, where the diffraction intensity was relatively high in the middle, depending on the intensity. In contrast, in the Poisson-NLPCA images, continuous speckle patterns were observed only near the images’ centers. However, the speckle patterns in the Poisson-NLPCA images appeared to be different from those in the EMV images, suggesting that the SRCNN images are more suitable for structural analysis than Poisson-NLPCA images.

Figure 2

Diffraction images at incident X-ray intensities of 5 × 1012, 1 × 1013, 5 × 1013, and 1 × 1014 photons pulse–1 μm–2 and in different molecular orientations. The experimental images, images improved by the SRCNN, EMV images, and Poisson-NLPCA images are shown from left to right. A histogram of the X-ray intensity for each image is also shown.

To evaluate the degree of diffraction image improvement quantitatively, the integral correlation pattern based on an EMV image was calculated using the similarity detection algorithm,[47,48] and Figure displays the results. A comparison of the integral correlation patterns of the experiment (i.e., between the experimental and EMV images) and SRCNN images (i.e., between the SRCNN and EMV images) clearly revealed a high-correlation line in the SRCNN results for all incident intensities. The R score, which corresponded to the similarity of a pair of diffraction images described in eq , was evaluated. The results indicated that the R score of SRCNN was higher than that of the experimental images at all intensities: 5 × 1012 (R of the experiment: 0.0387; R of SRCNN: 0.265; R of NLPCA: 0.081), 1 × 1013 (R of the experiment: 0.0830; R of SRCNN: 0.327; R of NLPCA: 0.147), 5 × 1013 (R of the experiment: 0.242; R of SRCNN: 0.491; R of NLPCA: 0.312), and 1 × 1014 (R of the experiment: 0.348; R of SRCNN: 0.588; R of NLPCA: 0.419). This finding indicated that the improved diffraction images obtained using the SRCNN had patterns more similar to the corresponding EMV images than the experimental images to the EMV images. When one focuses on the Poisson-NLPCA improvement, the R scores of the Poisson-NLPCA images were higher than those of the experimental images. However, they were lower than those of the SRCNN images, and the degree of improvement was not high compared to that achieved using the SRCNN.

Figure 3

Integral correlation patterns at various incident X-ray intensities from those above 5 × 1012, 1 × 1013, 5 × 1013, and 1 × 1014 photons pulse–1 μm–2 and in different molecular orientations between diffraction images. The experiment vs EMV, SRCNN vs EMV, and Poisson-NLPCA vs EMV images are shown from left to right.

Figure shows the impact of applying the SRCNN model in the structure analysis protocol. In most structure analysis methods, the similarity detection of the pair of images is critical. An integral correlation pattern measures how well the similarity detection works. In the case of the four experimental images shown in Figure , correlation lines clearly appear for X-ray intensities of 1 × 1013, 5 × 1013, and 1 × 1014photons pulse–1 μm–2. In other words, the similarity detection can be performed at these intensities. However, for the experimental image with and intensity of 5 × 1012 photons pulse–1 μm–2, no clear correlation lines can be observed. Then, the similarity detection would fail. In contrast, in the case of the SRCNN images, correlation lines are visible in the integral correlation patterns for the intensity of 5 × 1012 photons pulse–1 μm–2. This implies that the structure analysis would become possible for an X-ray intensity of 5 × 1012 photons pulse–1 μm–2 by introducing the SRCNN model.

Next, to investigate the molecular orientation dependency of the image improvement, a test was performed on diffraction images of 320 orientations for each intensity using the trained SRCNN model. Figure presents the results for four different incident X-ray intensities (A: 5 × 1012; C: 1 × 1013; E: 5 × 1013; and F: 1 × 1014 photons pulse–1 μm–2, corresponding to data sets 1, 3, 5, and 7, respectively, in Table S1). Figure a shows the R scores of the SRCNN images were all higher than those of the experimental images. In addition, the R scores of the SRCNN images were higher than those of the Poisson-NLPCA images (Figure b). These results confirmed that the present SRCNN model effectively improved diffraction images with various molecular orientations.

Figure 4

(a) Similarity improvement by the SRCNN model. X: R score of the experiment. Y: R score of SRCNN. (b) Similarity improvement by the Poisson-NLPCA model. X: R score of the experiment. Y: R score of the experiment.

Effects of Structural and Incident X-ray Intensity Fluctuations on Diffraction Image Improvement

We investigated the effects of the structural fluctuations of the sample biomolecules and incident X-ray intensity fluctuations on the diffraction image improvement achieved using the SRCNN.

First, we studied the effects of the structural fluctuations on the diffraction image improvement. Structural-mixed training was performed using the diffraction image data sets (data sets 2, 4, 6, and 8 in Table S1) that involved two types of structures that differed from the reference structure by approximately 5 and 10 Å in root-mean-square deviation (RMSD). Subsequently, a test was performed using diffraction images of the reference structure (RMSD = 0 Å) that were not used in training. Figure a,b shows the results of the structurally mixed training at incident X-ray intensities of 5 × 1012 and 1 × 1013 photons pulse–1 μm–2. Structurally mixed training was conducted at all intensities, and the R score of the SRCNN was higher than that of the experiment. Thus, it was concluded that the constructed SRCNN model could resist structural fluctuations to some extent at a resolution of 5 Å.

Figure 5

Results of learning considering structural and incident X-ray intensity fluctuations. X: R score of the experiment. Y: R score of the SRCNN. (a) Test of the structural fluctuation using data set 2, which contained mixed data of RMSD = 5 and 10 Å to improve the diffraction image of RMSD = 0 Å at 5 × 1012 photons pulse–1 μm–2. (b) Test of the structural fluctuation using data set 4, which contained mixed data having RMSD = 5 and 10 Å, to improve the diffraction image of RMSD = 0 Å at 5 × 1013 photons pulse–1 μm–2. (c) Test of incident X-ray intensity fluctuation using data set 13, which was a mixed data set of diffraction images at 5 × 1012 and 1 × 1013 photons pulse–1 μm–2. (d) Test of structural and intensity fluctuations using data set 14, which was a mixed data set of RMSD = 5 and 10 Å to improve the diffraction image of an RMSD to 0 Å at 5 × 1012 and 5 × 1013 photons pulse–1 μm–2.

Next, we investigated the effects of the incident X-ray intensity fluctuations on the diffraction image improvement. X-ray intensity-mixed training was performed using the diffraction image data sets (data sets 9, 11, 13, and 15 in Table S1) that differed in incident X-ray intensity by at most three times. Figure c presents the results of tests performed with two different intensities using a mixed-intensity SRCNN training model. At each intensity, the R scores with all 320 different orientations were improved. The same tendency was observed in the training of mixed intensity data at other intensities (data sets 9, 11, and 15 in Table S1). These results indicated that the SRCNN model constructed in the presence of various incident X-ray intensities was effective for improving images regardless of the molecular orientation.

Finally, we investigated the effects of a more realistic experimental situation wherein both structural and incident X-ray intensity fluctuations existed simultaneously. The structural/incident X-ray intensity fluctuation mixing diffraction image data sets (data sets 10, 12, 14, and 16 in Table S1) were used to perform the structural/incident X-ray intensity-mixing training. Figure d depicted the testing results for two different intensities using a mixed training model. At all intensities, the R scores were improved at all molecular orientations of the reference structure, which was not used in training.

In summary, an effective single-image SRCNN model was constructed in the presence of structural and incident X-ray intensity fluctuations by using simulation data that reflect the actual situation of single-particle experiments.

Improvement of Intermediate-Intensity Diffraction Images Not Included in Training

Because the incident X-ray intensity continuously changed by up to 3 orders of magnitude in an actual experiment, this section addresses whether the SRCNN model constructed for an intermediate-intensity diffraction image not included in the training data was effective. The intermediate-intensity diffraction image improvement was performed using the incident X-ray mixed-intensity data set that did not include some intensity data (data set 18 in Table S1), which mimicked the actual XFEL experimental situation. The training data included intensities (A) 5 × 1012, (C) 1 × 1013, and (E) 5 × 1013 photons pulse–1 μm–2 but did not include intensities (B) 8 × 1012, (D) 3 × 1013, and (F) 1 × 1014 photons pulse–1 μm–2. Figure shows the results of testing all incident X-ray intensities using a training model in which missing mixed-intensity data training was performed. The images with intensities B, D, and E, which were not involved in the training data, were improved by the trained SRCNN model to the same extent as those with intensities A, C, and E. Thus, the constructed SRCNN model was effective for improving intermediate-intensity diffraction images not included in the training. We demonstrated that the SRCNN model created was effective for X-ray diffraction image improvement under actual experimental conditions.

Figure 6

Similarity improvement of the intermediate X-ray intensity data set not included in the training. X: R score of the experiment. Y: R score of the SRCNN. The X-ray incident intensity contained in the training data set is 5 × 1012, 1 × 1013, and 5 × 1013 photons pulse–1 μm–2, whereas that not included in the training data set is 8 × 1012, 3 × 1013, and 1 × 1014 photons pulse–1 μm–2.

Conversion of Diffraction Image Improvement into Incident X-ray Intensity

The trained SRCNN model’s performance was evaluated by converting the diffraction image improvement into a comparable incident X-ray intensity. Figure shows the constructed SRCNN model increased R for various incident X-ray intensities. The increase in R indicated that the SRCNN reduced the noise in the observed diffraction images and that the improved images were closer to the corresponding EMV images. The degree of noise in the observed diffraction images depended on the diffraction image intensity, which depended on the molecular size and incident X-ray intensity.[53] The following analysis was conducted to convert the increase in R into an increase in incident X-ray intensity. The averages and variances of the R scores of the experimental images were calculated for 320 different molecular orientations at various incident X-ray intensities. Then, the R versus incident X-ray intensity conversion curve was constructed (see Figure a), which also included the standard deviation with error bars centered on the mean R score of the experiment. For the R versus incident X-ray intensity conversion curve, the regression curve was obtained by fourth-order polynomial fitting using the function f(x) = ax4 + bx3 + cx2 + dx + e. The fitting parameters were obtained as a = −0.011959, b = 0.64769, c = −13.042, d = 115.95, and e = −384.65.

Figure 7

(a) Conversion between R and incident X-ray intensity with I on a log scale. X: Log10(I). Y: R score of the experiment. (b) Improvement ratio of the incident X-ray intensity between the SRCNN and experimental results. X: Log10(I): Experiment; Y1: Log10(I): SRCNN; Y2: SR/EX ratio; I: SRCNN/I: Experiment.

Using the X-ray mixed-intensity training results (see Figure ), R̅, which was the average R obtained using the SRCNN, was calculated for each intensity. Subsequently, R̅ was converted into the incident X-ray intensity using the R versus incident X-ray intensity conversion curve. Figure b presents the resulting improvement ratios with black dots and I of SRCNN/I of the experiment = SR/EX, shown by the red bar as a function of the X-ray intensity of the experimental image. According to SR/EX, the lower the incident X-ray intensity, the higher is the improvement rate. When one focuses on the intensities of 5 × 1012 to 1 × 1013 photons pulse–1 μm–2 (three red columns from the left in Figure b), which could be obtained in the current XFEL facilities, the improvement rate was comparable to an incident X-ray intensity of five to seven times that observed in the context of experimental observations, which was approximately an order of magnitude higher. In Figure , the relationship is shown between the S/N ratio (SNR), a widely used evaluation standard of image quality, and the R of our experimental images with respect to the corresponding EMV images. SNR is a monotonically increasing function of R, indicating that both SNR and R can be used as an image evaluation standard. However, the SNR shows only small variations when the R of experimental images is small. The small R region is important because real experiments are typically carried out with low X-ray intensity, resulting in the low quality (that is, R is small) of diffraction images. R provides a more reasonable measure of the diffraction image quality in such a condition. The constructed SRCNN model successfully improved the image quality, and the images were of similar quality to that of the diffraction images. Thus, we verified the effectiveness of the proposed method on the similarity detection between diffraction images for nanoscale flexible biomolecules, whose analysis was currently at the limit of what was achievable through the XFEL experiments. The use of the predicted SRCNN image in the structural analysis was expected to estimate a more accurate real image.

Figure 8

Conversion between SNR and R. X: R score of the experiment; Y: S/N ratio.

Improvement of the Oversampling Ratio and Real-Space Image Analysis

In single-particle coherent diffraction imaging, a real-space image is obtained by conducting oversampling and phase-retrieval calculations. Oversampling is the process of sampling at a frequency higher than the Nyquist frequency (i.e., linear oversampling ratio σ = 1) in the diffraction image. It was necessary to measure the diffraction image with a high oversampling ratio (linear oversampling ratio σ > 2) to perform phase retrieval calculations for an experimental diffraction image. This rigorous experimental condition accelerated the lack of diffraction intensity per pixel solid angle. We investigated whether the SRCNN model could improve the oversampling ratios of diffraction images and whether the real images could be recovered using the HIO algorithm for phase retrieval (see Section S10). If a finer diffraction image that satisfied the oversampling condition was obtained by the SRCNN model from the Nyquist frequency observation, it would dramatically relax the strict experimental conditions.

Figure depicts the results of the phase retrieval calculations using four types of diffraction images: the EMV image, SRCNN image, experimental image, and projection image, which was the FT image of the 2D electron density map. The projection image in the k-space had perfect point symmetry due to the Ewald sphere’s lack of curvature (Figure d). Phase retrieval analysis was performed 100 times using the HIO method with different initial phases, and each analysis consisted of 10 000 iterations. The lower parts in Figure a–f present the best real-space images obtained by the phase-retrieval calculation for each case. In the case of the projection image (see Figure d), we obtained the reconstructed real-space image, which was almost equal to the electron density plots (Figure g). However, in the case of the EMV image, the reconstructed real-space image was blurred (see Figure a) compared to the electron density image if there was no noise. These results indicated that the effect of the curvature of the Ewald sphere affected the phase retrieval calculation in the EMV, SRCNN, and experiment in this condition. Moreover, this tendency was confirmed by the error values obtained by HIO calculations (see Figure h,i). We examine the results of the SRCNN images and the experimental images, considering that they included the effects of the curvature of the Ewald sphere on the HIO method. Figure b,e and c,f shows the results of the phase retrieval calculations in the cases of 1 × 1014 and 5 × 1012 photons pulse–1 μm–2, respectively. In Figure e,f, the upscaled experimental diffraction images with σ = 1 did not work well because they did not satisfy the oversampling conditions, and the molecular shape was not retrieved correctly. In contrast, the real-space images obtained from the SRCNN images (Figure b,c) were similar to the electron density projection image, indicating that the phase retrieval calculation worked well.

Figure 9

Upper row of (a–f): diffraction images used as inputs for the phase retrieval calculations in the EMV, SRCNN, projection image, and experimental approach from upper left to lower right. The lower row of (a–f): real-space images obtained by the phase retrieval calculation have the maximum real-space C values compared with the G of the electron density projection image, as shown in the electron density plots. The error values in k-space (Error-k) and real-space (Error-r) obtained by the HIO method are also shown for the cases of (h) 1 × 1014 photons pulse–1 μm–2 and (i) 5 × 1012 photons pulse–1 μm–2.

We successfully demonstrated that a plausible real image could be recovered by phase retrieval for an experimental diffraction image that did not satisfy the phase retrieval conditions. It was possible to obtain a real image using the HIO method from an observed diffraction image under the condition that σ = 1 was satisfied with an incident X-ray intensity of 5 × 1012 photons pulse–1 μm–2 through SRCNN analysis in the case of ribosomes. Our proposed X-ray diffraction image improvement scheme is advantageous; the oversampling ratio could be improved because GTHR images are used to train the SRCNN model in silico in various molecular orientations and conformations. Consequently, it is possible to conduct experimental measurements with smaller oversampling ratios, which could increase the number of measured photons per pixel.

Conclusion

Herein, we constructed a practical SRCNN model by data augmentation in a virtual space using simulation data assuming actual experimental conditions. We showed that the constructed SRCNN model could be applied to diffraction images of all molecular orientations with 3-fold X-ray intensity fluctuations and 10 Å structural fluctuations. The constructed SRCNN model improved the quality of a single noisy X-ray diffraction image under actual experimental conditions, i.e., if the sample molecule had various conformational states and the incident X-ray intensity changed with each measurement. The diffraction image improvement rate was quantitatively evaluated using the similarity with the EMV diffraction image as a standard. We showed that the diffraction image improvement by the present SRCNN model corresponded to an increase in the incident X-ray intensity by three to seven times in the context of experimental observations.

We also confirmed that the SRCNN model contributes to noise reduction and effectively improves the oversampling ratio. The real-space image analysis shows that the proposed SRCNN model not only amplifies the intensity but also effectively reproduces the fine pattern of the diffraction image. It is expected that the improvement of the oversampling ratio is a powerful approach because it enables the relaxation of the experimental conditions by the Nyquist frequency observation. This method will be useful for studying the structure and dynamics of flexible nanoscale biomolecules and for accelerating drug discovery because it relaxes the stringent experimental conditions required by phase-search algorithms.

We discuss the advantage of using SRCNN images as an initial model for the structural analysis pipeline (see Figure ). First, it is expected to significantly reduce the number of images used for 3D structural analysis because the S/N ratio can be improved for a single diffraction image. Second, the harsh experimental conditions required for real-space image analysis using a phase retrieval algorithm can be significantly relaxed because the improved oversampling ratio allowed observations at the Nyquist frequency. Finally, the potential for the multiconformational analysis of biomolecules will be considerably increased because the diffraction image quality can be improved while maintaining the individuality of the image. For example, the similarity value improved the estimation accuracy of the plausible 3D structure by the template matching method. In conclusion, the improvement of the diffraction image by SRCNN dramatically enhanced the effectiveness of one diffraction image in which the individuality of biomolecules appeared.

Regarding the limitations of this work, the constructed SRCNN model depended on the training data sets because of supervised learning. We believed that we needed to augment our training data further by considering various molecular species (see Figure S5; generalization achievable by the SRCNN model with respect to molecular species) and realistically simulated experimental data. Although the current SRCNN model was insufficient, we expected that the pretrained SRCNN model conducting supervised learning in silico could be applied to real experimental data by combining it with real-world data. For example, we believe it was necessary to incorporate the beam stop region of the detector, Gaussian-like electronics noise of the detector, and fluctuations in the geometry of the detector and the sample into the learning process. Additionally, it would be necessary to examine the difference in the performance of the methods other than the SRCNN, including unsupervised methods, to observe which methods were effective in improving the XFEL images. Additionally, it was necessary to develop a workflow wherein various structural analysis methods were organically combined to construct a structural analysis pipeline for flexible 100 nm biomolecules. Therefore, high-performance computing could be used with parallel processing for an artificial intelligence-driven big data analysis system linked with single-particle experimental measurements.

Data and Software Availability

The software GROMACS is publicly available under academic license for research (https://www.gromacs.org). Keras is publicly available under open source software (https://github.com/keras-team/keras/releases/tag/2.4.0). The molecular operating environment is available as paid software (https://www.chemcomp.com/Products.htm) All relevant data are shown in the figures, listed in the tables, or included in the Supporting Information. The data sets of 18 in Table S1 and the constructing model that is presented in Figure can also be accessed at https://github.com/TokuhisaAtsushi/Improvement-of-X-ray-diffraction-image-using-SRCNN.