Literature DB >> 35817617

Fast Prediction of Binding Affinities of SARS-CoV-2 Spike Protein and Its Mutants with Antibodies through Intermolecular Interaction Modeling-Based Machine Learning.

Alexander H Williams1,2, Chang-Guo Zhan1,2.   

Abstract

Since the introduction of the novel SARS-CoV-2 virus (COVID-19) in late 2019, various new variants have appeared with mutations that confer resistance to the vaccines and monoclonal antibodies that were developed in response to the wild-type virus. As we continue through the pandemic, an accurate and efficient methodology is needed to help predict the effects certain mutations will have on both our currently produced therapeutics and those that are in development. Using published cryo-electron microscopy and X-ray crystallography structures of the spike receptor binding domain region with currently known antibodies, in the present study, we created and cross-validated an intermolecular interaction modeling-based multi-layer perceptron machine learning approach that can accurately predict the mutation-caused shifts in the binding affinity between the spike protein (wild-type or mutant) and various antibodies. This validated artificial intelligence (AI) model was used to predict the binding affinity (Kd) of reported SARS-CoV-2 antibodies with various variants of concern, including the most recently identified "Deltamicron" (or "Deltacron") variant. This AI model may be employed in the future to predict the Kd of developed novel antibody therapeutics to overcome the challenging antibody resistance issue and develop structural bases for the effects of both current and new mutants of the spike protein. In addition, the similar AI strategy and approach based on modeling of the intermolecular interactions may be useful in development of machine learning models predicting binding affinities for other protein-protein binding systems, including other antibodies binding with their antigens.

Entities:  

Mesh:

Substances:

Year:  2022        PMID: 35817617      PMCID: PMC9301770          DOI: 10.1021/acs.jpcb.2c02123

Source DB:  PubMed          Journal:  J Phys Chem B        ISSN: 1520-5207            Impact factor:   3.466


Introduction

The SARS-CoV-2 virus (which causes the COVID-19 disease) continues to remain the world health community’s number one priority as the pandemic prepares to enter its third year.[1] Although potent vaccines have been developed for the virus, widespread inoculation has been achieved primarily in well-developed nations, leaving vast areas of lesser developed nations in peril against the virus and its many variants.[2,3] Vaccine hesitancy within the United States has led to multiple additional waves of COVID-19 infections, in which the B.1.617.2 (Delta) and B.1.1.529 (Omicron) variants have played significant roles, especially within the southern United States.[3−9] The Delta variant is noted for both its increased infectivity and the severity of symptoms it causes.[7] Although vaccines still remain largely effective against the Delta variant, breakthrough infections are still possible, and the viral load carried by vaccinated individuals is unaffected, a testament to the infectivity of the Delta variant.[10−12] However, the resistance against mutations seen within the fast-tracked vaccines from Pfizer, Moderna, and Janssen is not carried over to the monoclonal antibody therapeutics that have also been approved for COVID-19 treatment.[13−17] Of note is Ly-CoV555, an antibody that received FDA emergency use authorization, which has dramatic reduction in efficacy when used against the B.1.617.1 (Kappa), Delta, and most notably the Omicron variant.[13,15,18−20] Mutations seen within the receptor binding domain (RBD) region of the spike protein are of particular interest when developing COVID-19 variant-resistant antibodies. Due to the similar positioning of angiotensin-converting enzyme II (ACE2) and many antibodies in relation to the spike protein’s RBD region (see Supporting Information Figure S1), the mutations that confer greater biding affinity to ACE2 may be deleterious to the binding of the antibody.[21] Of note is the L452R mutation and T478K mutation, both of which are capable of greatly decreasing binding affinity with multiple known antibodies on their own, including Ly-CoV555.[13] As variants continue to develop, it is important to develop an accurate and efficient methodology that can predict the effects on binding affinity these mutations may have. During the early months of the pandemic in 2020, our laboratory group was attempting to adapt our in silico protein/ligand binding affinity methodology to the spike protein and several engineered ACE2 proteins and ACE2 mimics to optimize the binding interactions between them. This methodology is a very convenient, computationally inexpensive way of estimating the Gibbs binding free energy using only energy minimization and molecular mechanics Poisson–Boltzmann surface area calculations (MM-PBSA), a methodology that our laboratory had seen great success with in multiple protein/ligand systems.[22−27] Concurrently, the first variant of major public note, the N501Y (later named B.1.1.1 or Alpha), began sweeping across the United Kingdom. Our group then decided to include a prediction for the binding affinity of this Alpha variant using the linear regression model that had been established for the spike protein and the engineered ACE2 protein and its mimics. Our prediction of 0.44 nM Kd for the Alpha variant was validated during the review process of our first spike/ACE2 model as the Alpha variant’s Kd was determined in vitro at 0.8 nM, less than a 0.5 nM difference.[28] Throughout 2021, additional waves of the COVID-19 virus manifested, introducing new, even more heavily mutated variants (e.g., B.1.617.2 and B.1.1.529) to the public. Using the same computational methodology, our laboratory was able to further predict the binding affinity of the Omicron variant at 22.63 nM, a prediction that was later validated during review, as in vitro experiments determined the Kd at 20.69 nM.[29] Since the publication of our previous methodologies focusing on the interactions between the spike protein and ACE2, a slew of new antibody structures binding with the spike protein have been deposited and have subsequently been used in in silico methodologies. Both coarse-grained (CG) and all-atom molecular dynamics (MD) simulations have been performed with numerous antibodies to study the changes in the binding modes induced by the mutation of the spike protein across the many variants of SARS-CoV-2.[30−32] These models have also been used to produce reasonable estimates of the binding affinity; however, these studies are more focused on a particular subset of antibodies, rather than producing a more generalizable model.[32] With these new antibody structures available, along with the empirically determined Kd values, we can update our model to accurately predict the binding affinity of these antibodies with the different variants of the spike protein. However, building a model that can predict the affinity of various antibodies with the spike protein broadly is a much more challenging task than building a model to determine how certain mutations will affect the binding of a singular protein/protein pair. Generalized linear regression models for the prediction of protein/protein interactions[33] do suffer in terms of their accuracy when compared to models such as our previously published ACE2/spike mutation model.[28] This loss in accuracy is inherent to the methodology as while a change in a residue in an otherwise unchanged protein structure can be seen as a small perturbation of the overall system, the use of many antibodies, each with their own unique binding mode with the spike protein (Figure S1), cannot. This issue is further compounded when considering the multitude of different charge/charge combinations between the spike protein and the antibody, which is variable due to several common mutations seen within the SARS-CoV-2 spike protein (e.g., L452R, T478K, etc.). To circumvent this loss in accuracy, we have constructed a multi-layer perceptron (MLP) neural network, a machine learning method that is able to accurately predict the binding affinity of a wide variety of antibodies, including several monoclonal antibody therapeutics on offer from Regeneron, AstraZeneca, and Eli Lilly.[34] By including multiple predictors including the gas and solvation free energy predictions from the MM-PBSA methodology, solvent-accessible surface areas (SASA) of the complex, hydrogen bonds between the antibody and spike protein, and the charges of each, we have developed and cross-validated such an artificial intelligence model which can accurately predict the binding affinity of various SARS-CoV-2 variants’ spike protein with numerous antibodies. We have then proceeded to use the model to predict the binding affinity of these antibodies with numerous variants of concern (VOCs) including the B.1.1.529 and the newly published “Deltamicron” (or “Deltacron”) variant identified within southern France.[35]

Methods

The methodology for preparing our MLP neural network models consists of three major stages: (1) structure preparation of spike/antibody structures, (2) energy minimization utilizing the Sander module of the Amber MD package followed by an MM-PBSA calculation, and finally, (3) the construction, training, and cross-validation of the MLP models.[28,36] Once the MLP models are validated for their accuracy in both the training and validation data sets, the model predictions may be averaged to mitigate local inaccuracies located within one model.

Modeling of Spike Protein/Antibody Complexes

For the first stage, multiple spike/antibody crystal and cryo-electron microscopy (cryo-EM) structures were obtained from the RCSB database (PDB IDs are available in Supporting Information Table S1), and the structures which had only one type of antibody binding to the spike protein and had a known binding affinity Kd value were chosen for use in the MLP model. Additional data points from mutated versions of these antibodies were taken from Liu et al.,[37] who measured the Kd for numerous antibodies with spike proteins that mimicked the B.1.617.1 and B.1.617.2 strains of the RBD region of the spike protein. Each of these structures was prepared using the PDB4AMBER module of the AmberTools suite, which removed any non-protein atoms from the structure.[38] These prepared structures then had their initial coordinates and parameters created using the tLEaP module of the AmberTools package; this package also assigned the positions for the hydrogens for each structure. For the non-wild-type data points, the structures were perturbed using the PyMol mutation tool to replace each residue with its respective mutation in the RBD region of the spike protein, making sure that the lowest-energy rotamer available was chosen to avoid large steric clashes with nearby residues.[44] For the second stage, to prepare the complexes for their interaction energy calculation with the MM-PBSA method, we employed our previous methodology that employs computationally efficient energy minimization over a two-step process. This methodology has been shown to accurately estimate the binding affinity in both protein/protein and protein/ligand systems[22−27] and has previously been used to engineer an anti-cocaine therapeutics.[39,40] This methodology, although more simplistic than other methodologies that employ all-atom or CG MD simulations, has major advantages. Sufficiently long MD simulations require extensive computational resources and can require days of computing time before a system may enter equilibrium. Additionally, over these long simulations, the limitations placed upon these MD simulations (e.g., force field cutoffs, periodic conditions, etc.) can introduce artifacts into the system, impacting the actual accuracy of the simulation. Usage of energy minimization also limits the number of snapshots required to determine the ΔG of the system as only one snapshot is needed as opposed to MD simulations or with using free-energy perturbation methods.[41,42] However, this methodology does assume that the structure used within the energy minimization/MM-PBSA calculations is close to its equilibrium state (i.e., unperturbed), and the amino acid mutations inserted into each of these structures are a small perturbation away from the equilibrium state, which can be returned with the energy minimization steps. The wild-type protein/protein complex structures were energy-minimized over a two-step process using the CUDA-optimized version of PMEMD in the AMBER20 package.[38] The first round of energy minimization used 1000 steps of steepest descent minimization followed by 4000 steps of conjugate gradient energy minimization, using 10 kcal/Å restraints upon the backbone alpha carbons of the proteins. The second round of energy minimization reduced these restraints to 2 kcal/Å. Interaction (binding) energies of the spike/antibody complexes were then evaluated using the MMPBSA.py module of AMBER20, using the MM-GBSA methodology.[36,38,43] The ΔGGB values were deconstructed into their constituent electrostatic (EEL), van der Waals (vdW), and solvation (SOLV) energies for use in training/validation of our model. Unlike our other empirical models, the residues mutated within the spike protein variants induced a change in charge, which caused overestimations of the ΔΔGEEL between variants using the standard MM-PBSA method. To account for these overestimations, we implemented a distance-dependent dielectric coefficient that we previously employed to correctly predict the binding affinity of ligands with the nicotinic acetylcholine receptors.[44−46] The distance between the charge centers of the antibody and spike protein was calculated by exporting the minimized parameters and topology of each in the PQR format. Using the product of each atom’s charge and XYZ coordinates, an average location for the overall charge in each structure was calculated, followed by determining the distance between these two points using the Euclidian distance formula. Finally, the SASA of each complex, antibody, and spike protein was calculated using a 1.4 Å probe via the CPPTRAJ module of the AMBER package.[47]

Construction of the MLP Neural Network Model

Once the necessary in silico parameters for each complex were generated, the in vitro Kd values were converted to kcal/mol units to match those of the MM-GBSA calculations. These converted Kd values were set as the response variable in the JMP 16.0 software package, utilizing the neural predictive modeling package.[48] The neural network structure consists of initial 11 input nodes (i.e., electrostatic, vdW, and solvation energy decomposed from the MM-GBSA calculation, long-range electrostatic energy (LRE), antibody and spike overall charges, distance between antibody and spike protein charge centers, hydrogen bonds between the spike protein and antibody, and the surface areas of the overall complex, spike protein, and the antibody surface), forward feeding into two hidden layers of 11 nodes, which is forward feeding into a single-output node, providing the binding free energy (ΔG) prediction of the antibody/protein complex in question (Figure ).
Figure 1

Schematic of the MLP neural network used to predict spike/antibody binding affinities. The network is divided into four sections: an input layer, two fully connected hidden layers, and a final singular output layer.

Schematic of the MLP neural network used to predict spike/antibody binding affinities. The network is divided into four sections: an input layer, two fully connected hidden layers, and a final singular output layer. This setup (Figure ) allows for each of the input variables collected in the previous stages to possess their own singular input node for weight training while simultaneously avoiding overfitting the data with an overly complex model.[49,50] Each node within the hidden layer uses the TanH activation function, which has previously been shown to be superior in accuracy and the root mean square error (RMSE) in MLP performance meta-analyses.[51] To train and cross-validate the model, the 48 data points collected were randomly split into training and validation sets, using 3 K-fold cross-validation, allowing each of our models to receive 67% (32 points) of the available data as a training set, while the other 33% (16 points) is used as the validation set to train the weights for each fold.[52] This cross-validation method allows for efficient use of the relatively small set of available Kd values measured for antibody/protein complexes and allows us to determine the generalizability of the model over the entire data set. Each K-fold neural-network model was trained over 10,000 iterations, and loss was measured using the RMSE of the validation set’s prediction versus the empirically determined binding affinity values. Finally, each K-fold’s prediction was added and averaged for the final spike/antibody Kd prediction. Once the MLP model was established, the structures for multiple antibody/spike complexes were reconstructed for each variant’s set of mutations using the PyMol mutation tool. The variants chosen for analysis are the VOCs listed by the CDC in June 2021. Much like our previous work investigating the effects of spike protein mutations with ACE2,[28] the mutations investigated were treated as small structural perturbations of the overall spike protein structure, which is made possible due to the availability of reliable experimental crystal or cryo-EM structures of these antibody/spike complexes, whose energy-minimized structures produce results consistent with in vitro data.[22−27] This methodology allows us to avoid long-timescale MD simulations of the spike/antibody complex that can introduce artifacts into the simulated system from the imperfect force field and from the truncation of long-distance interactions.[53−57] Additionally, these long MD simulations allow for deviations away from the original crystal/cryo-EM structures, thus losing the overall equilibrium state of the complex captured by these structural methods (Supporting Information Figure S17). These complex structures were similarly energy-minimized, and their interaction energies were calculated. The values obtained from the MM-GBSA calculation, along with the charge center, SASA, and H-bond measures, were then used within our MLP network to obtain the antibody-predicted affinity for the mutated spike protein. To obtain the predicted Kd values of these antibodies against the SARS-CoV-2 variants, the difference between the ΔGGB value of the WT spike protein versus each variant was applied to the ΔGexp obtained from in vitro experiments.

Results and Discussion

Structural Insights into the Binding Modes of VOCs

Binding Interactions of B.1.617.2 (Delta) Variant’s Spike Protein with Human ACE2

We have previously utilized the available structures of ACE2 binding with the spike protein to investigate the effects the N501Y mutation had on the interactions within the ACE2/spike interface.[28] Although the mutations of the Delta variant are still located in the RBD region of the spike protein, the L452R and T478K mutations are not located within the interface between the spike protein and ACE2. These mutations are either closer to the core of the spike protein (e.g., L452R) (Figure B) or are located on the receptor binding motif (RBM) but too far away to closely interact with any ACE2 residues (e.g., T478K) (Figure B). A commonality between these two residues is their change from an uncharged residue to a positively charged residue; this change alters the electrostatic surface of the spike protein and thus its affinity with ACE2, which has an overall charge of −27. With this large number of negative charges within ACE2, it is not surprising that many of the VOCs utilize such mutations within the RBD region to gain affinity with ACE2 [e.g., B.1.1.7 (Alpha), B.1.351 (Beta), B.1.427 (Epsilon), B.1.429 (Epsilon), B.1.525 (Eta), B.1.526 (Iota), B.1.526.1, B.1.617.1-2, P.1 (Gamma), P.2 (Zeta), C.37 (Lambda), and B.1.1.529 (Omicron)].[58] However, the distance of these residues from the interface between spike and ACE2 limits their effects on modifying any existing interactions between the two. This can be seen in Figure C,D, revealing the low rmsd between the two energy-minimized protein structures (rmsd = 0.046) (Figure A,B).
Figure 2

(A) Overall binding structure of the WT spike protein with ACE2, with L452 and T478 indicated in stick representation. (B) Overall binding structure of the B.1.617.2 (Delta) spike protein and ACE2, with L452R and T478K mutations in ball and stick representation. (C) Detailed representation of the WT spike and ACE2 binding mode, showing several hydrogen bonds of interest at the interface between the two proteins. (D) Detailed binding mode of Delta spike and ACE2, revealing limited deviation in position for the mutated spike protein’s residues. All hydrogen bonds in (C) are retained.

(A) Overall binding structure of the WT spike protein with ACE2, with L452 and T478 indicated in stick representation. (B) Overall binding structure of the B.1.617.2 (Delta) spike protein and ACE2, with L452R and T478K mutations in ball and stick representation. (C) Detailed representation of the WT spike and ACE2 binding mode, showing several hydrogen bonds of interest at the interface between the two proteins. (D) Detailed binding mode of Delta spike and ACE2, revealing limited deviation in position for the mutated spike protein’s residues. All hydrogen bonds in (C) are retained.

Mutational Escape of Spike VOCs against Antibodies

Although the Delta variant is known for its increased infectivity, its mutational escape against several antibodies is also notable. Although there are few examples in the literature of empirical binding data of antibodies against the B.1.617.1 (Kappa) spike protein, Liu et al. released a study of the effects of the T478K and E484Q mutations in the spike RBD region against several of their identified COVOX-series antibodies.[37] Two antibodies, COVOX-316 and COVOX-384, were notable for their drastic increase in binding Kd versus the B.1.617.1 protein and the WT spike, 200 and >500-fold, respectively. Identifying these Kd values against the Delta spike’s RBD region while also elucidating the cryo-EM structure of these antibody complexes with the spike protein provides an opportunity to establish a predictive model like our previously published model with the spike/ACE2 complex and enables to predict Kd values across a representative sample of known antibodies.[37] COVOX-316 and COVOX-384 bind to the RBM of the spike protein much in the same way as ACE2 does (Figure A,D). However, both antibodies are in much closer proximity to the L452 and E484 mutational sites, making them much more susceptible to the effects of these changes. Although the L452R mutation’s primary effect is the addition of deleterious +/+ charge interactions with the positively charged antibody, the E484Q mutation has a much larger effect. The mutated glutamine residue is unable to fit within the same pocket as the glutamic acid and thus loses two strong hydrogen bonds with the COVOX-316 (Figure B,C). In Covox-384, the mutation of E484Q is more devastating, removing a critical +/– charge interaction between E484 and R52 of COVOX-384 (Figure E,F). Additionally, a +/+ charge repulsion between the mutated R452 and H56 further adds to the loss in binding affinity between these two proteins (Table ). This drastic loss in interactions within COVOX-384 explains the “knockout” reported by Liu et al. regarding the Kd value of the complex. Several antibodies share a similar positioning of an arginine residue in the range of the L452R mutation including COVA2-04, C1A-C2, C1A-F10, C1A-B3, and C1A-B12 (Figures S6 and S9–S11), a potential issue if these antibodies were to be trialed against either the Kappa or Delta variants.
Figure 3

(A) Overall binding mode of COVOX-316 and the B.1.617.1 variant spike protein, with the mutated residues R452 and Q484 in green ball and stick representation. (B) Detailed binding mode of COVOX-316 with WT spike protein, showing the hydrogen bonds within the complex. (C) Binding mode of COVOX-316 with Delta spike protein. The E484Q mutation causes a loss of two hydrogen bonds between E484/S494 and E484/N54 of the spike and antibody, respectively, leaving only the hydrogen bond with Y33 with the Q484 of the mutated spike protein. (D) Overall binding mode of COVOX-384 and the Delta variant spike protein, with mutated residues in green ball and stick representation. (E) Binding mode of COVOX-384 with WT spike protein; E484 forms a +/– charge interaction with R52 of the antibody, which is destroyed upon mutation to Q484, along with a hydrogen bond with the backbone amine of F490 (F).

Table 1

Training Data Used for the Establishment of the MLP Neural Network, along with the In Vitro Experimental Binding Energies and Predicted ΔG Values (kcal/mol)

strainantibodyEEL (kcal/mol)VDW (kcal/mol)GAS (kcal/mol)SOLV (kcal/mol)LREa (Hartrees)antibody chargespike chargedistanceb (Å)complex surface (Å2)spike surface (Å2)antibody surface (Å2)H-bondsKdc (nM)ΔGd (kcal/mol)predicted ΔGe (kcal/mol)
WTAZD1061[37]–269.46–77.95–347.41265.56–0.055245.7026086.9718300.659303.8373.90–11.54–12.04
WTAZD8895[37]–99.10–72.96–172.06115.85–0.045239.1126638.5818504.489467.6261.00–12.35–11.84
WTC1A-B3[61]–128.24–116.13–244.36150.00–0.076334.7723735.0517006.079145.51876.30–9.77–9.95
WTC1A-C2[61]–321.24–122.33–443.57345.33–0.083349.0025301.6418396.429537.211114.10–10.78–10.36
WTC1A-F10[61]–216.73–114.17–330.89233.77–0.085341.7224711.4217710.949458.29955.70–9.96–10.07
WTCC12.1[62]–121.01–93.60–214.61221.08–0.054341.5922147.2016163.988829.571017.00–10.66–10.19
WTCC12.3[62]–87.60–100.28–187.88118.23–0.045451.7425532.8417837.539615.36614.00–10.78–10.76
WTCOVA2-04[63]–223.60–130.87–354.47253.52–0.074234.0624389.4717863.509118.65740.00–10.15–10.04
WTCOVOX-150[37]–237.88–130.69–368.56256.08–0.074355.1024272.6817905.289064.8780.57–12.69–12.64
WTCOVOX-158[37]–172.95–112.40–285.35198.85–0.064339.9923722.7216853.489218.8081.40–12.15–11.81
WTCOVOX-222[37]–235.74–118.04–353.78255.04–0.062233.2823836.5316887.709274.2270.25–13.18–12.89
WTCOVOX-253[37]–138.11–69.95–208.06162.06–0.043148.6925509.2617822.879089.2670.51–12.76–12.68
WTCOVOX-269[37]–114.77–138.27–253.04143.41–0.077346.7924935.2018424.499276.91100.52–12.74–12.76
WTCOVOX-278[37]–206.63–90.13–296.76207.93–0.066337.7725211.4017444.569509.0681.60–12.07–11.49
WTCOVOX-316[37]41.74–75.93–34.20–16.57–0.038341.9524697.5416854.099315.4720.38–12.93–13.22
WTCOVOX-384[37]–1.01–74.98–76.0022.88–0.037452.8426393.4618008.869635.8110.65–12.61–12.55
WTCOVOX-40[37]–143.09–116.85–259.95158.64–0.075452.1525831.5818107.2510191.22130.34–13.00–13.50
WTCR3022[64]–316.93–106.82–423.75323.02–0.073436.2325756.9018227.559606.00868.00–9.84–9.92
WTLY-CoV488[37,61]–409.96–87.61–497.57410.27–0.08020.0025032.3717670.799295.96954.00–9.98–9.65
WTLY-CoV555[37,65]–90.46–93.19–183.6699.17–0.058221.3826215.8018375.759674.6931.45–12.13–12.53
WTREGN10933[37,66]–216.00–77.40–293.40238.72–0.051292.3828940.3319871.4010808.0760.38–12.93–12.90
WTREGN10987[37,66]–41.04–63.99–105.0369.98–0.025218.2627916.5618461.5310639.0640.81–12.48–12.62
B.1.617.1AZD1061–226.77–79.91–306.68221.52–0.055446.6126333.5918247.789651.9976.70–11.22–11.59
B.1.617.1AZD8895–13.93–71.38–85.3128.90–0.045439.7727406.4819052.169704.1173.90–11.54–11.24
B.1.617.1COVOX-150–212.20–130.34–342.55221.40–0.074551.2324740.3518286.389177.15100.70–12.57–13.83
B.1.617.1COVOX-158–151.24–111.05–262.29172.04–0.064537.4624345.5417207.799508.15101.40–12.15–11.92
B.1.617.1COVOX-222–224.82–115.86–340.69238.36–0.072430.5523984.2917061.759305.89101.40–12.15–12.40
B.1.617.1COVOX-253–105.82–66.88–172.70127.81–0.043338.1525688.3317846.939220.5561.90–11.97–11.90
B.1.617.1COVOX-269–33.03–140.32–173.3563.19–0.067542.4925273.6818619.639407.03111.50–12.11–12.22
B.1.617.1COVOX-278–100.56–95.83–196.39107.79–0.056535.2925587.3217700.779736.49922.00–10.51–10.67
B.1.617.1COVOX-316311.56–82.17229.39–264.31–0.018534.9724674.6916764.889337.2811623.00–7.95–7.67
B.1.617.1COVOX-384255.82–76.57179.25–227.98–0.017643.0726777.6818192.439828.1834000.00–7.41–7.54
B.1.617.1COVOX-40–92.14–122.14–214.28109.53–0.065646.9426394.4518546.2910359.61120.33–13.01–12.49
B.1.617.1REGN10933–175.61–77.75–253.36195.96–0.051491.4129099.7920076.9610706.7360.13–13.57–13.55
B.1.617.1REGN10987–4.14–68.69–72.8330.77–0.035425.9128161.9418723.9010712.4040.44–12.84–12.58
B.1.617.2AZD1061–241.80–78.22–320.02269.66–0.065448.1126428.7018364.639616.3476.90–11.20–10.74
B.1.617.2AZD8895–68.24–75.81–144.05144.32–0.035437.6227252.9619050.939660.9361.50–12.11–12.24
B.1.617.2COVOX-150–188.94–129.24–318.19274.25–0.074548.8424743.1118245.389231.71100.77–12.51–13.31
B.1.617.2COVOX-158–190.27–108.09–298.36266.29–0.074534.9624337.7717167.349600.56102.40–11.83–11.79
B.1.617.2COVOX-222–214.29–117.07–331.36301.95–0.062426.4124166.8917096.969447.41100.52–12.74–12.22
B.1.617.2COVOX-253–207.08–69.15–276.23260.99–0.053339.4825698.7117939.009218.3170.96–12.38–12.42
B.1.617.2COVOX-269–5.13–140.46–145.59113.60–0.067540.8625517.5818815.959461.17100.73–12.54–12.44
B.1.617.2COVOX-278–157.34–95.21–252.55201.98–0.066534.4625848.1617877.369857.2186.40–11.25–11.22
B.1.617.2COVOX-316203.42–75.78127.64–118.28–0.038535.3226552.1118185.919809.7520.78–12.50–12.47
B.1.617.2COVOX-384141.35–71.8469.51–87.59–0.037642.5028084.6218938.3210393.0935.60–11.33–11.43
B.1.617.2COVOX-40–64.52–118.51–183.03143.12–0.065643.5526454.0118592.4410344.35140.47–12.80–11.71
B.1.617.2REGN10933–234.27–75.22–309.48308.17–0.051492.6828988.0419942.7510747.2960.74–12.53–12.75
B.1.617.2REGN10987–18.88–69.00–87.8899.78–0.035429.9328046.7918694.7910638.5250.38–12.93–12.71
B.1.617.2Ly-Cov555252.20–74.30177.90–187.88–0.046629.4225682.4019568.1011423.69431.00–10.31–9.58
               RMSE0.38

LRE interaction as calculated using the distance-dependent dielectric function.

Distance between the overall charge center of the spike and antibody proteins.

Experimentally determined Kd.

Experimental binding affinity[6] converted to Gibbs binding free energy: ΔGexp = −RT ln(Kd).

Computationally predicted ΔG in kcal/mol using the MLP Neural Network.

(A) Overall binding mode of COVOX-316 and the B.1.617.1 variant spike protein, with the mutated residues R452 and Q484 in green ball and stick representation. (B) Detailed binding mode of COVOX-316 with WT spike protein, showing the hydrogen bonds within the complex. (C) Binding mode of COVOX-316 with Delta spike protein. The E484Q mutation causes a loss of two hydrogen bonds between E484/S494 and E484/N54 of the spike and antibody, respectively, leaving only the hydrogen bond with Y33 with the Q484 of the mutated spike protein. (D) Overall binding mode of COVOX-384 and the Delta variant spike protein, with mutated residues in green ball and stick representation. (E) Binding mode of COVOX-384 with WT spike protein; E484 forms a +/– charge interaction with R52 of the antibody, which is destroyed upon mutation to Q484, along with a hydrogen bond with the backbone amine of F490 (F). LRE interaction as calculated using the distance-dependent dielectric function. Distance between the overall charge center of the spike and antibody proteins. Experimentally determined Kd. Experimental binding affinity[6] converted to Gibbs binding free energy: ΔGexp = −RT ln(Kd). Computationally predicted ΔG in kcal/mol using the MLP Neural Network.

Delta Variant’s Escape of Ly-CoV555

Upon closer inspection, the cause behind the large loss in efficacy of Ly-CoV555 when used against the Delta variant is clear when one looks at the antibody/spike interface, particularly at the L452R mutation site (Figure A,B)[7] The mutation from leucine to arginine constitutes a shift in both residue size and charge, putting it in stark contrast to the hydrophobic Ly-CoV555 I54 and L55 residues it interfaces with. Additionally, both the L452R and T478K mutations within the Delta variant introduce two additional positive charges into the spike protein, shifting the total charge of the spike protein from +2 to +4 and thus inducing multiple sources of positive/positive charge repulsion, the strongest of which coming from nearby R24 on the Ly-CoV555 light chain. These mismatches in residue affinity lead to a 20% loss in LRE energy between Ly-CoV555 and the WT/Delta variant spike protein (−31.4 to −25.1 kcal/mol, respectively) (Table ).
Figure 4

(A) Overall structure of the spike protein and Ly-CoV555 complex. (B) Interface of the L452 to R452 (in yellow ball and stick representation) mutation with Ly-CoV555, with nearby residues I54 and L55 clashing with the polar and charged arginine residue.

(A) Overall structure of the spike protein and Ly-CoV555 complex. (B) Interface of the L452 to R452 (in yellow ball and stick representation) mutation with Ly-CoV555, with nearby residues I54 and L55 clashing with the polar and charged arginine residue.

Construction and Validation of the MLP Model of Spike/Antibody Interactions

Our previous models would primarily use either the ΔGPB or the deconstructed van der Waals, electrostatic, and solvation energies to create a linear regression to predict the binding affinity, creating a generalized model for numerous classes of antibodies presents an additional challenge. The differences in spike/antibody charges and binding modes cause each class of antibodies (i.e., COVOX and CC12.x, each of which contains three or more examples available for linear regression and both have high sequence similarity and have been published contemporaneously) to be locally comparable with one another via linear regression analysis (Figure S14 and Table S2). However, when combining these two classes of antibodies, the ability to predict the binding affinity worsens significantly, with an R2 value of only 0.001 and an RMSE of nearly 1 kcal/mol (Figure S13), far below that of our previous models, which can achieve up to R2 = 0.924.[28,29,36] With these differences between the antibodies, a more complex model that can take into account the differences in the structure is required to accurately predict the binding affinity of each to the spike protein and its variants. Using the obtained interaction energy values and the additional structural information variables (Figure ) from the spike/antibody structures and their modified, mutated versions, we proceeded to train a four-layer perceptron neural network (MLP) machine learning model, using the converted Kd in kcal/mol unit as the response variable. Due to the limited number of available experimental data points for spike/antibody binding affinities, we decided to implement a methodology which would allow for the greatest utilization of the data. By partitioning the whole data set of spike/antibody binding affinity data into 67% training and 33% validation partitions (each data point labeled as either 0 or 1, respectively, within JMP 16.0 software,[48]Table S1), we effectively created three data sets (i.e., K-folds 1, 2, and 3) to train three separate models, each having a different combination of data points that were included in the training set. Additionally, this ensures that each data point is within the validation set once over the course of training the three neural network models, which alleviates the overtraining commonly seen within supervised learning tasks for the neural network models.[59] For each of the training partitions of the data, the models have both high accuracy and precision when compared to the experimentally derived values, with an average RMSE of 0.16 kcal/mol for all three K-folds (Figure ). Additionally, the R2 values for each K-fold range from 0.962 to 1.0 for the training sets, which is representative when compared to our previous linear regression-based prediction models of spike/ACE2 and GPCR/ligand binding affinities.[28,44,46,60] Additional covariance analysis of the 11 features used in comparison with the ΔGbind values used to train the model shows that the electrostatic (EDW), LRE, and charges of the spike/antibody proteins show both high correlation and covariance with the ΔG value (Tables S3 and S4). This correlation is expected as each of these variables is either directly force-field-based calculation of the interactions between the spike protein or is used to calculate the said force-field-based features (i.e., the charges used to calculate Coulombic potential). Although each of these parameters on its own does not have a strong correlation with the ΔGbind, their use within the neural network creates a model that can accurately predict these values.
Figure 5

Predicted binding free energy (kcal/mol) vs experimental binding free energy (kcal/mol) of antibody proteins and SARS-CoV-2 spike proteins listed in Table utilizing the neural networks trained on fold 1, 2 and 3 training and validation partitions of the spike/antibody binding free energy data. RMSE and R2 values for each K-fold’s training (0) and validation (1) partitions are displayed.

Predicted binding free energy (kcal/mol) vs experimental binding free energy (kcal/mol) of antibody proteins and SARS-CoV-2 spike proteins listed in Table utilizing the neural networks trained on fold 1, 2 and 3 training and validation partitions of the spike/antibody binding free energy data. RMSE and R2 values for each K-fold’s training (0) and validation (1) partitions are displayed. Once the ability for the models to accurately predict against each K-fold’s training partition had been confirmed (correlation matrix of the ΔGprediction to the used descriptors available in Table S5), we than proceeded to test their generalizability by introducing the models to their respective validation partitions of the data. These predictions also had low average RMSE values (0.40 kcal/mol in average), and the averaging of the K-folds’ predictions further improved this metric, decreasing the RMSE of our final prediction to 0.38 kcal/mol (Table ). While fold 2 does appear to be over-trained on the training partition, with a 0.0 kcal/mol training RMSE, the model’s generalizability is not impacted when compared to the fold’s validation set, which has a similar RMSE (0.35) when compared to the average K-fold validation RMSE (0.4 kcal/mol). Importantly, the K-folds’ models displayed an ability to predict when the spike protein’s binding affinity with a certain antibody would be dramatically decreased in the micromolar range, which will serve as a warning when utilizing this model with three antibody/variant combinations.

Prediction of Binding Affinity of Concerned Spike Variants with Various Antibodies via the MLP Neural Network

With the establishment and successful validation of our three K-folds’ neural networks, attention was then turned to the spike VOCs and how their mutations would affect a wide array of antibodies, especially the antibodies that are currently being used to treat SARS-CoV-2 infections. By utilizing the same computational methodology to generate the parameters needed (e.g., MM-PBSA, LRE interactions, surface area, etc.), each variant spike/antibody complex’s binding free energy was predicted using each the K-fold’s neural network and subsequently averaged to reduce the influence of any one K-fold model’s biases and reduce the overall RMSE of the predictions. The predicted binding affinity data are summarized in Table . As seen in Table , experimental Kd values have been reported in the literature for multiple antibodies with spike variants B.1.617.1 (Kappa) and B.1.617.2 (Delta). The predicted Kd values are all close to the corresponding experimental data, suggesting that the predictions provided in Table are reasonable.
Table 2

Predicted Kd (nM) Values of Antibodies with Multiple SARS-CoV-2 Spike Protein VOCs Using Our Neural Network Modelsa

     B.1.351B.1.427/B.1.429/B.1.526.1dB.1.525/P.2eB.1.526B.1.617.1B.1.617.2C.37P.1B.1.1.529 
 antibodyKdb,cΔGNNAY.1Betaepsilonetaiotakappadeltalambdagammaomicrondelta/omicron hybridf
clinical antibodiesAZD10613.91.75555272.83.13.2 (6.7)15 (6.9)262.13.822
 AZD8895120.40.40.73206 (3.9)1 (1.5)0.515050
 LY-CoV48854931313723.7484.13043253743
 LY-CoV5551.450.741.381.381.2110744.646.2105 (31)1.022.18288368
 REGN109330.380.40.930.930.330.110.110.110.4111.90.310.83
 REGN109870.810.64774.5151513160.6162.15
non-clinical antibodiesB3870.181140140130210200220170241501515
 C1A-B124.221.3517.417.47.8817.814.315.28.56.3224608.1311.9
 C1A-B376.356.716166.9310.812.99.866.7514.16.33.264.55
 C1A-C214.128.30.3570.3574.541.241.462.142.2912.60.1941.211.23
 C1A-F1055.746.22.942.944.719.055.614.931.8647.139.52.792.37
 CC12.117380.440.440.471.10.710.690.820.230.34343
 CC12.3141521215.16.26.56.35.91.2991515
 COVA2-0440500.90.92010.622200.80.80.8
 COVA2-39211107.57.56.7372439341789530530
 COVOX-1500.570.620.0990.0990.120.390.3315 (0.77)0.20 (0.77)20.850.860.86
 COVOX-1581.42.55521.61.42.1 (1.4)2.4 (2.6)1.3130.780.97
 COVOX-2530.510.572.32.31.11.41.82.1 (1.9)0.90 (0.96)0.690.782.12.1
 COVOX-2690.520.514.34.30.61.91.51.3 (1.5)0.86 (0.73)0.243.43.52.6
 COVOX-2781.64.3333.72.31.817 (22)6.7 (6.4)1.83.244.3
 COVOX-3160.380.246506501200150021002600 (1623)0.82 (0.78)3915004701600
 COVOX-3840.650.71220220380320019003200 (4000)4.7 (5.6)72400420250
 COVOX-400.340.151.51.52.74.92.50.80 (0.33)2.9 (0.47)1.56.83.13.1
 COVOX-884.40.832302301020190.39 (0.13)0.79 (5.8)17661115
 CR30226859808068163153370302915

Known variant experimental Kd values are set within parentheses. Ly-CoV555 shows decreased affinity with B.1.617.1 and B.1.617.2 in confirmation with in vitro data.[37]

Experimental Kd obtained by Liu et al. (2021).[37]

Experimental Kd obtained by Liu et al. (2021).[68]

RBD region of these noted strains contain the same mutations within their respective columns.

Neural network prediction of the wild-type strain.

Delta/Omicron hybrid mutations obtained by Colson et al. (2022).[35]

Known variant experimental Kd values are set within parentheses. Ly-CoV555 shows decreased affinity with B.1.617.1 and B.1.617.2 in confirmation with in vitro data.[37] Experimental Kd obtained by Liu et al. (2021).[37] Experimental Kd obtained by Liu et al. (2021).[68] RBD region of these noted strains contain the same mutations within their respective columns. Neural network prediction of the wild-type strain. Delta/Omicron hybrid mutations obtained by Colson et al. (2022).[35] Upon analysis of the screened antibodies versus the numerous SARS-CoV-2 variants’ spike proteins, a pattern emerged that many antibodies are unable to retain their affinity across the entire spectrum of variants. This is especially true with Ly-CoV555, which loses significant affinity toward B.1.617.1, B.1.617.2, and B.1.1.529.[13,37,67] Additionally, due to the high amount of similarity between the Omicron and “Deltamicron” RBD regions, the same antibodies that showed susceptibility to the Omicron variant show similar binding affinities, including Ly-CoV555 (Table ). However, the positioning of the antibody in relation to the spike protein has significant effects on its susceptibility to these variants. The antibody CR3022 (RCSB: 6YLA) has a unique positioning against the spike protein, where it instead binds in proximity to the β-sheets of the RBD region of the spike protein, rather than the ACE2 binding interface, where most other antibodies bind (Figure ).[64] This confers significant resistance for CR-3022 to the mutations that are common in the VOCs. This is reflected in the data, where CR-3022’s binding affinity has little variability when compared to that of the RBD binding antibodies and has no predicted knockouts among the variants tested. This unique positioning and high affinity to the spike protein could potentially be used as a dual-antibody treatment, wherein CR3022 is used as a secondary binder to the spike protein in combination with an antibody tailored to the SARS-CoV-2 VOC (e.g., B.1.617.2).
Figure 6

(A) Overall binding mode of CR3022 with the WT spike protein; important to note is the unique positioning of the antibody in the spike RBD region. Unlike Ly-CoV555 (B) and many other antibodies (Figure S1), CR3022 binds to a separate motif of the spike protein, separate from the commonly used RBM region utilized by the ACE2. This difference in positioning (C) leads to additional resistance against the spike protein mutations (Table ), which are tailored to increase binding affinity with ACE2 at the RBM.

(A) Overall binding mode of CR3022 with the WT spike protein; important to note is the unique positioning of the antibody in the spike RBD region. Unlike Ly-CoV555 (B) and many other antibodies (Figure S1), CR3022 binds to a separate motif of the spike protein, separate from the commonly used RBM region utilized by the ACE2. This difference in positioning (C) leads to additional resistance against the spike protein mutations (Table ), which are tailored to increase binding affinity with ACE2 at the RBM. Conversely, when looking at the trends from a variant-centric perspective, the Kappa, Delta, and Gamma variants all have substantial changes in their average binding affinity from 17 nM for the WT Kd values to 560, 240, and 260 nM, respectively. These large increases in Kd over a large class of antibodies can be tied to these variant’s mutation upon the RBD; although these three variants do not contain the same mutation, they do each contain a mutation which alters the overall charge balance of the (i.e., E484 for B.1.617.1 & P.1 and T478K for B.1.617.2). This change in charge balance will affect the more positively charged antibodies (e.g., COVOX-316 and COVOX-384) and decrease the binding affinity between the spike protein and antibody.

Conclusions

We have developed and validated an intermolecular interaction modeling-based MLP neural network that is able to replicate the experimentally determined in vitro binding affinities of a wide set of known antibodies toward the wild-type SARS-CoV-2 spike protein and various measurements of these same antibodies with both the Kappa and Delta variants of the spike protein. This neural network utilizes non-proprietary in silico-derived parameters, which can be obtained using the publicly available structures of these spike/antibody complexes. Using this model, we have predicted the binding affinities across multiple VOCs for each of these antibodies, including those from Eli Lilly and Regeneron, which are currently being used to treat SARS-CoV-2. In the case of the Ly-CoV555 antibody, the L452R and T478K mutations appear to be deleterious to the antibody’s binding to the protein, through a combination of the disruption of the charge/charge interactions between the antibody and spike protein and steric hindrance induced by the L452R mutation. Our model was also able to predict the deleterious effects of the B.1.1.529 (Omicron) variant’s mutations on the Ly-CoV555’s efficacy, with a predicted 200-fold decrease in binding affinity; this prediction has been borne out in clinical studies where Ly-Cov555 has little to no effect on treating the Omicron variant.[18,67] These effects carry over to the newly discovered “Deltamicron” variant, which shares significant similarity with the Omicron variant RBD region. Additionally, this method has revealed a resistance to mutations in the RBD region with the antibody CR3022, which bind peripherally to the RBD region of the protein; this resistance could be employed within a multi-antibody therapeutic to circumvent the loss of affinity seen with many other antibodies. This neural network methodology provides a quick and computationally inexpensive way to not only test currently existing antibodies against new SARS-CoV-2 variants but also to allow for the quick in silico screening of newly designed antibodies before their costly production and in vitro testing. In addition, our machine learning approach for fast protein–protein binding affinity prediction is based on modeling of intermolecular interactions between proteins. Compared to usually used machine learning models that often use hundreds or even thousand(s) of molecular descriptors as input nodes, an intermolecular interaction modeling-based machine learning model will only need to use a few intermolecular interaction descriptors as input nodes, such as only 11 input nodes in this study. The similar computational strategy and approach may be useful in development of machine learning models predicting binding affinities for other protein–protein binding systems, including other antibodies binding with their antigens. Further iterations of this model may focus on including additional mutation Kd values as they are reported along with implementing other machine learning methods (e.g., AdaBoost, Boosting Gradient, etc.), which can also produce models that show strong correlation with the empirically obtained mutational data (Figure S16). Additionally, further feature optimization may be performed by paring unnecessary features (a prospective five-feature model’s performance can be seen in Supporting Information S18).
  56 in total

1.  MMPBSA.py: An Efficient Program for End-State Free Energy Calculations.

Authors:  Bill R Miller; T Dwight McGee; Jason M Swails; Nadine Homeyer; Holger Gohlke; Adrian E Roitberg
Journal:  J Chem Theory Comput       Date:  2012-08-16       Impact factor: 6.006

2.  Modeling subtype-selective agonists binding with alpha4beta2 and alpha7 nicotinic acetylcholine receptors: effects of local binding and long-range electrostatic interactions.

Authors:  Xiaoqin Huang; Fang Zheng; Xi Chen; Peter A Crooks; Linda P Dwoskin; Chang-Guo Zhan
Journal:  J Med Chem       Date:  2006-12-28       Impact factor: 7.446

3.  Assessing the performance of the MM/PBSA and MM/GBSA methods. 6. Capability to predict protein-protein binding free energies and re-rank binding poses generated by protein-protein docking.

Authors:  Fu Chen; Hui Liu; Huiyong Sun; Peichen Pan; Youyong Li; Dan Li; Tingjun Hou
Journal:  Phys Chem Chem Phys       Date:  2016-08-10       Impact factor: 3.676

4.  The neutralizing antibody, LY-CoV555, protects against SARS-CoV-2 infection in nonhuman primates.

Authors:  Bryan E Jones; Patricia L Brown-Augsburger; Kizzmekia S Corbett; Kathryn Westendorf; Julian Davies; Thomas P Cujec; Christopher M Wiethoff; Jamie L Blackbourne; Beverly A Heinz; Denisa Foster; Richard E Higgs; Deepa Balasubramaniam; Lingshu Wang; Yi Zhang; Eun Sung Yang; Roza Bidshahri; Lucas Kraft; Yuri Hwang; Stefanie Žentelis; Kevin R Jepson; Rodrigo Goya; Maia A Smith; David W Collins; Samuel J Hinshaw; Sean A Tycho; Davide Pellacani; Ping Xiang; Krithika Muthuraman; Solmaz Sobhanifar; Marissa H Piper; Franz J Triana; Jorg Hendle; Anna Pustilnik; Andrew C Adams; Shawn J Berens; Ralph S Baric; David R Martinez; Robert W Cross; Thomas W Geisbert; Viktoriya Borisevich; Olubukola Abiona; Hayley M Belli; Maren de Vries; Adil Mohamed; Meike Dittmann; Marie I Samanovic; Mark J Mulligan; Jory A Goldsmith; Ching-Lin Hsieh; Nicole V Johnson; Daniel Wrapp; Jason S McLellan; Bryan C Barnhart; Barney S Graham; John R Mascola; Carl L Hansen; Ester Falconer
Journal:  Sci Transl Med       Date:  2021-04-05       Impact factor: 19.319

5.  Antibody resistance of SARS-CoV-2 variants B.1.351 and B.1.1.7.

Authors:  Pengfei Wang; Manoj S Nair; Lihong Liu; Sho Iketani; Yang Luo; Yicheng Guo; Maple Wang; Jian Yu; Baoshan Zhang; Peter D Kwong; Barney S Graham; John R Mascola; Jennifer Y Chang; Michael T Yin; Magdalena Sobieszczyk; Christos A Kyratsous; Lawrence Shapiro; Zizhang Sheng; Yaoxing Huang; David D Ho
Journal:  Nature       Date:  2021-03-08       Impact factor: 69.504

6.  Exploring the immune evasion of SARS-CoV-2 variant harboring E484K by molecular dynamics simulations.

Authors:  Leyun Wu; Cheng Peng; Yanqing Yang; Yulong Shi; Liping Zhou; Zhijian Xu; Weiliang Zhu
Journal:  Brief Bioinform       Date:  2022-01-17       Impact factor: 11.622

7.  Electrostatic Interactions Explain the Higher Binding Affinity of the CR3022 Antibody for SARS-CoV-2 than the 4A8 Antibody.

Authors:  Hung Nguyen; Pham Dang Lan; Daniel A Nissley; Edward P O'Brien; Mai Suan Li
Journal:  J Phys Chem B       Date:  2021-07-06       Impact factor: 2.991

Review 8.  Monoclonal antibody as a potential anti-COVID-19.

Authors:  Leila Jahanshahlu; Nima Rezaei
Journal:  Biomed Pharmacother       Date:  2020-06-04       Impact factor: 6.529

9.  Limited neutralization of authentic SARS-CoV-2 variants carrying E484K in vitro.

Authors:  Marek Widera; Alexander Wilhelm; Sebastian Hoehl; Christiane Pallas; Niko Kohmer; Timo Wolf; Holger F Rabenau; Victor M Corman; Christian Drosten; Maria J G T Vehreschild; Udo Goetsch; Rene Gottschalk; Sandra Ciesek
Journal:  J Infect Dis       Date:  2021-07-05       Impact factor: 5.226

10.  Effectiveness of Covid-19 Vaccines against the B.1.617.2 (Delta) Variant.

Authors:  Jamie Lopez Bernal; Nick Andrews; Charlotte Gower; Eileen Gallagher; Ruth Simmons; Simon Thelwall; Julia Stowe; Elise Tessier; Natalie Groves; Gavin Dabrera; Richard Myers; Colin N J Campbell; Gayatri Amirthalingam; Matt Edmunds; Maria Zambon; Kevin E Brown; Susan Hopkins; Meera Chand; Mary Ramsay
Journal:  N Engl J Med       Date:  2021-07-21       Impact factor: 91.245

View more

北京卡尤迪生物科技股份有限公司 © 2022-2023.