Covid-19.bioreproducibility.org: A web resource for SARS-CoV-2-related structural models.

The COVID-19 pandemic has triggered numerous scientific activities aimed at understanding the SARS-CoV-2 virus and ultimately developing treatments. Structural biologists have already determined hundreds of experimental X-ray, cryo-EM, and NMR structures of proteins and nucleic acids related to this coronavirus, and this number is still growing. To help biomedical researchers, who may not necessarily be experts in structural biology, navigate through the flood of structural models, we have created an online resource, covid19.bioreproducibility.org, that aggregates expert-verified information about SARS-CoV-2-related macromolecular models. In this article, we describe this web resource along with the suite of tools and methodologies used for assessing the structures presented therein.

INTRODUCTION

The spread of the novel coronavirus around the world has triggered an unprecedented response from the scientific community. Six months into the pandemic, PubMed already listed over 23,000 scientific papers with the terms COVID‐19 or SARS‐CoV‐2 in the title, and tens of analyses are reported daily in mass media around the globe. Understandably, first‐line research findings, including molecular structure determinations, depositions in the Protein Data Bank (PDB), and related results, are often made public on BioRxiv or MedRxiv before formal peer review. This approach delivers the latest results to scientists that develop treatments and vaccines without any delay but at the cost of elevated risk of mistakes and errors, which can mislead scientists performing follow‐up research and misinform the general public. The World Health Organization has even coined the portmanteau “infodemic” to describe the phenomenon of potentially misleading information overload.

As of July 27, 2020, the PDB has amassed 298 structural models of SARS‐CoV‐2‐related macromolecules, including proteins and RNA fragments. Structure‐based drug design depends on such molecular models, especially of complexes with candidate drugs slated for further development. However, the rapidly growing number of structures without corresponding publications and the potential mistakes associated with pandemic‐driven research can create confusion among biomedical researchers and could impede, rather than accelerate, drug development. Indeed, an analysis of the “entry history” of structures deposited to the PDB between January 24 and July 27, 2020 showed that as many as 56 out of the 182 (30.8%) SARS‐CoV‐2 structures (excluding PanDDA fragment screening deposits) required a major revision of the initial model, whereas only 360 of the other 6,328 (5.9%) structures deposited during that time period had any major revisions. For 15 of the SARS‐CoV‐2 structures, the revisions were significant and involved replacement of the atomic coordinates. Some of these revisions were triggered by our resource. Several structures have already been obsoleted and replaced with new deposits. In comparison, there were only seven coordinate replacements for all other structures deposited during that time (and among all the structures deposited in 2019, on average only less than 1 in 300 structures had a new version with coordinate replacement). The higher fraction of revised COVID‐related structures may in part be due to the celerity of the research, and in part to this and similar projects that requested original diffraction data, which prompted the authors of these structures to revisit their models. An additional factor, which may impede the use of molecular structures in biomedical research, is that they are sometimes presented in a way geared toward modeling and theoretical chemistry, but not for biomedical scientists that are not necessarily experts in protein crystallography.

In this article, we present covid‐19.bioreproduciblity.org, a web resource that organizes SARS‐CoV‐2 related structural information in a way that should be understandable and useful for a wider scientific community, and not only for structural biologists. The website also serves as a repository for examined and, if found to be suboptimal, corrected versions of PDB structures of SARS‐CoV‐2 proteins and RNA fragments, with a focus on assessing the small‐molecule ligands modeled in those structures. Moreover, we strive to re‐deposit the optimized structure models in the PDB, always in collaboration with the original authors. The validation tools and re‐refinement protocols used in this project can serve as a template for future molecular structure assessment efforts.

THE RESOURCE

Due to the rapid response in time of the pandemic, the covid‐19.bioreproducibility.org web resource was created in an agile, fast‐prototyping manner, focusing on speedy delivery and flexibility to accommodate changes. As a result, several new features are still being implemented. New data are evaluated and published on the website on a weekly basis, synchronized with the weekly releases of the PDB.

The website's main section (Figure 1) presents a table of assessed structures, preceded by a set of filters and export options. The filters (Figure 1A) allow users to select the experimental method, virus type, protein type, and ligand category. Our resource includes proteins and RNA fragments not only from SARS‐CoV‐2, but also a selection of structures of proteins from other closely related coronaviruses: HCoV, MERS, and SARS. To allow users to find specific fragments of the virus easily, the structures are categorized according to the function of the protein and the location of the corresponding genomic fragment. Ligands are categorized according to their involvement in the function of the protein to which they are bound. Ligands that may affect the protein function are called “functional ligands” (as opposed, for example, to the ligands that are artifacts of protein purification or crystallization, and there is no indication that their binding may affect the protein function). Users can also quickly filter for structures with or without functional ligands, protein–protein complexes, pathogen‐host interactions, and fragment screening results. The website also allows text searches (Figure 1B), and the data can be exported (Figure 1C). The selected structures are presented in a responsive table (Figure 1D) that adjusts the visibility of columns to the user's browser window. To preserve screen space, several pieces of information, such as the protein cartoon representation, full title, or the ligand diagram (Figure 1E), can be dynamically previewed upon mouse hover. Detailed information that cannot be presented directly in the table (such as re‐refinement details, full list of quality metrics, or associated files) can be viewed by expanding table rows (Figure 1F). For completeness, the 115 PanDDA models are listed, but we did not attempt to assess the quality of these models. PanDDA structures contain weakly bound potential ligands and thus may be of limited usefulness for translational research. With a few exceptions, we did not attempt to assess the quality of the cryo‐EM structures.

FIGURE 1

Structure section of the covid‐19.bioreproducibility.org web resource. A, Filter area, where the user can choose the experimental method, virus type, protein type, and ligand category. Multiple filters can be used simultaneously, and multiple options within each filter can be selected. B, Data can also be filtered using text search. C, Selected data can be exported to xlsx, csv, and pdf file formats. D, List of structures fulfilling user‐specified criteria. The table is responsive and adjusts the columns shown depending on the user's browser window width. E, Hover preview of ligands. A similar preview mechanism was implemented for the PDB column. F, Details of structures can be displayed by expanding table rows

Figure 2 presents some of the current statistics of covid‐19.bioreproducibility.org, excluding PanDDA fragment screening deposits. The resource hosts 160 X‐ray, 1 NMR, and 48 cryo‐EM structural models, only 13 of which had raw diffraction data available. As expected, X‐ray structures generally have a better resolution than the cryo‐EM structures (Figure 2A). According to the PQ1 metric, the overall quality of SARS‐CoV‐2 structures is somewhat better than that of other, usually older, coronavirus structures in the resource (Figure 2B), but similar to other structures generated during the same time frame. Structures of SARS‐CoV‐2 macromolecules were determined at 13 different synchrotrons, and four were determined at a home source (Figure 2C). Most structures were determined at the beamlines 19‐ID, 21‐ID‐F (APS), BL17U1, BL19U1 (SSRF), and I03 (Diamond). The proteins with the highest numbers of structures are the main protease (also known as NSP5, 3CLPro, or Mpro) and the spike protein (S). The spike protein is the only protein whose structure was determined for a pathogen‐host interaction complex, and the main protease has most structures with functional ligands or inhibitors (Figure 2D). Exactly 58 of the SARS‐CoV‐2 structures in the resource have ligands that are recognized as pharmacological agents, representing 22 different approved drugs.

FIGURE 2

Structure statistics of the covid‐19.bioreproducibility.org web resource. A, Histogram of structure resolutions for X‐ray (blue) and cryo‐EM (orange) structures. B, The PQ1 quality percentile (expressed as fraction, the higher the better) of X‐ray structures, with the different types of coronaviruses in the resource depicted by different symbols. C, Number of SARS‐CoV‐2 structures determined at different synchrotrons; names of five top beamlines are shown. Full names of synchrotrons abbreviated on the plot are given in Table S1. D, Number of structures of different proteins of SARS‐CoV‐2, colored by ligand category

The resource is implemented in Python and uses the Flask web framework (https://flask.palletsprojects.com/) to serve the website. The frontend makes use of the Bootstrap (https://getbootstrap.com/) and DataTables (https://datatables.net/) libraries to provide searching, preview, sorting, and detail‐view capabilities. All the required data processing is done with the Pandas library on the backend. Currently, all the information about structures presented on the website is organized in spreadsheets and stored in‐memory on the server. However, ultimately the data will be ported to a PostgreSQL database (http://www.postgresql.org) for better self‐consistency enforcement.

The website is updated weekly in a semi‐automatic fashion. Every week, we consult the update of the PDB's COVID‐19/SARS‐CoV‐2 Resources to obtain a list of the most recently released COVID‐19 related structures. We extract information about the structures from a copy of the PDBj Mine database installed as a schema within an in‐house annotated database of structures. Moreover, we track changes to existing deposits, including primary citations related to structures released prior to publication as well as updated versions of atomic coordinates. For structures determined by X‐ray crystallography, we also search for raw diffraction images in the Integrated Resource for Reproducibility in Macromolecular Crystallography or, if significant improvement might be expected, contact the authors with the request to submit the diffraction images. Moreover, we extract quality metrics related to structures by querying a locally installed copy of the PDBj's VRPT database schema. After the data have been automatically gathered from the above‐mentioned sources, they are processed by geometry checking, statistical, and validation tools, most of which were developed in‐house by the laboratories collaborating on this project. Finally, the structures are evaluated by a team of expert structural biologists who use a combination of the mined data, validation reports, and manual inspection of the protein models and associated electron density to examine potential problems. Careful attention is paid to all functional ligands and inhibitors contained in the structures. If potential problems are spotted, the diffraction data are re‐processed (whenever the raw data are available) and the models are re‐refined. The corrected models are made publicly available on our webserver. In addition, we always attempt to contact the original authors and encourage them to jointly re‐deposit the optimized models to the PDB. The details of the application of our structure evaluation tools and our structure correction protocol are discussed in the following sections.

EVALUATION OF THE STRUCTURE QUALITY

To make an informed decision whether a structure should be re‐refined or not, we use several criteria and tools to assess its quality. We check the overall geometry (Ramachandran outliers and rotamer outliers), the correlation between model and electron density map (especially for ligands), the presence of large peaks in difference electron density map, the placement of the macromolecular model in the unit cell, and whether R and Rfree are reasonable for the reported resolution.

Additionally, to provide a simple quantitative overview of the quality of X‐ray structures, we calculate and show on our website the PQ1 metric. PQ1 is the structure's quality percentile (from 0 to 100, the higher the better) based on Rfree, RSRZ score, Clashscore, Ramachandran outliers, and Rotamer outliers. Being a hybrid reciprocal‐ and real‐space global metric, PQ1 can be easily used to sort structures and compare their overall quality. The PQ1 metric is recalculated weekly for each structure.

Equipped with the above‐mentioned validation tools, expert structural biologists may decide to manually inspect each structure in Coot. If diffraction data are available, the potential gains of their manual re‐processing are analyzed. Using the calculated electron density maps, the main chain and side chains can be easily reviewed with Coot or Molstack. Special emphasis is put on unmodeled electron density blobs. Based on such a review, a decision whether to re‐refine the structure is made. Full re‐refinement is a laborious process and sometimes requires contact with the deposition authors. In cases of deposits that do not have primary citations, the identification of the principal investigator (PI) is not always an easy task. For that reason, the Commission on Biological Macromolecules of the International Union of Crystallography (IUCr), together with the IUCr Committee on Data, has asked the PDB to publicly disclose the e‐mail address of the PI (or depositing author) of each deposit.

STRUCTURE CORRECTION PROTOCOL

Once the decision to re‐refine has been made, we use the following protocol to improve the model. Some aspects of this protocol are general in nature, and the exact values may be changed for a particular structure. The protocol and decisions made for each structure are based on our extensive experience in protein structure determination, , , crystallographic software development, , published guidelines on structure refinement and structure quality, , , and previous campaigns of PDB structure re‐refinement , :

Diffraction data

If raw diffraction data are available, the results of automatic processing of images by HKL‐3000auto are examined to verify that the structure was determined in the correct space group and at optimal resolution. In cases of inconsistent results, we use the HKL‐3000 program suite with the implementation of corrections for X‐ray absorption, radiation decay, and anisotropic diffraction. , , Resolution cut‐offs are chosen based on the values of CC1/2, average signal‐to‐noise ratio 〈I/σ(I)〉, and R in the outermost resolution shell, as well as on the quality of the resulting electron density maps. , ,

ACHESYM

Each structure under inspection is placed in a standardized way in the reference unit cell using ACHESYM. Even though crystal structures can be presented with the molecular models located in various crystallographically equivalent locations, we seek to facilitate the process of comparisons of analogous structures for non‐crystallographers by placing the models as close to the origin of the unit cell as possible. The ACHESYM server takes into account the equivalence of the space group symmetry positions and adjusts the location of the model in the unit cell. As a result, the atomic coordinates and electron density maps of the re‐refined versions of isomorphous structures, that is, structures in the same space group and with differences for cell parameters a/b/c within 1.5% and cell angles within 5%, are standardized to the same location. This means that the macromolecules occupy similar positions in their corresponding unit cells and both the coordinates and electron density maps of isomorphous structures can be easily viewed as already superposed using any current computer graphics program.

Refinement in reciprocal space

The models subjected to restrained maximum posterior refinement in REFMAC with hydrogen atoms added in riding positions. For all standard protein residues, the REFMAC dictionary is used as a source of ideal stereochemical targets, whereas for nucleic acids the conformationally dependent restraints generated by the RestraintLib server , , are used. Automatic local non‐crystallographic symmetry restraints are utilized if more than one protein molecule is present in the asymmetric unit. The TLS Motion Determination server is used to define TLS groups and their initial parameters. Solvent molecules are not included in the TLS groups. The TLS parameterization is kept if confirmed by a significantly improved Rfree. Full anisotropic parametrization is utilized if the ratio of unique reflections to non‐hydrogen atoms is higher than 18 (corresponding to data‐to‐parameters ratio higher than two) and if it resulted in significant (over 2%) drop of Rfree. The Hamilton R‐factor‐ratio test is used to decide whether the choice of anisotropic parametrization is justified. For most structures, we manually adjusted stereochemical weights aiming at bond length and bond angle rmsd values in the range of 0.010–0.015 Å and 1.6–2.0°, respectively. Multiple tools integrated into Coot, , a standalone version of MolProbity, as well as the PDB validation tools are used for structure quality assessment.

Manual model correction

After each round of REFMAC refinement, the atomic model is manually inspected and corrected according to the following checklist:

Review unmodeled electron density blobs, which might represent ligands or residues missing from the polymer model.

Inspect all difference electron density peaks above 4.0 rmsd (5.0 rmsd if there are too many peaks). Inspect the strongest negative density peaks.

Inspect rotamer outliers, which may indicate incorrect placement of side chains, as well as residues with missing atoms.

Review density fit graphs and inspect poorly fitting residues; verify terminal residue placement; and inspect any gaps in the sequence.

Inspect Ramachandran outliers.

Once large electron density blobs have been modeled and major issues with the protein backbone have been addressed, look for potential water molecules to add, with peaks above 1.1 rmsd in the 2mFo‐DFc map and distances to protein H‐bonded atoms ranging from 2.4 to 4.0 Å.

Ligand modeling and metal identification

If unmodeled electron density blobs are found during manual corrections, they are considered as potential ligands. In such cases, we try to identify the ligand with the help of CheckMyBlob, fit the ligand in the density, and run no fewer than 10 REFMAC cycles. If the ligand does not have a proper stereochemical description in the standard REFMAC dictionary, new geometrical restraints are generated using the Grade Web Server and carefully checked before use. Since ligands originally modeled in the deposition may be incorrect, they are inspected visually and, if questionable, validated using CheckMyBlob. Similarly, it has been shown that a significant fraction of metal‐containing structures in the PDB have incorrect metal assignment or modeling. Therefore, special attention is given to metal identification by the CheckMyMetal validation server and, when possible, using anomalous maps calculated with data collected above and below the X‐ray absorption edge.

Challenging cases are discussed by at least two team members. Many structures are inspected by at least one other expert after the refinement has been completed. The revised structures are stored in the web resource described here, along with a description of the identified issues and changes made. However, if the changes are significant, the goal is to re‐deposit the re‐refined structure in the PDB, preferably together with the original authors, using the mechanism of re‐versioning. An example report for a structure re‐processed from original data and re‐refined according to the above protocol is presented in Figure 3 and Figure S1. A report showing a case when the original diffraction data were not available is presented in Figures S2 and S3.

FIGURE 3

Report showing the re‐processing and re‐refinement of PDB structure 7BRR. This structure will be re‐deposited to the PDB under a new PDB ID, due to significant changes in comparison with the original deposit. The second part of the report is presented in Figure S1

DISCUSSION

The goal of the covid‐19.bioreproducibility.org web resource is to gather macromolecular structures related to the SARS‐CoV‐2 virus and assess them using state‐of‐the‐art tools. Additionally, we aim to provide information that can be easily used by non‐structural biologists. That is why the structures are categorized according to the experimental method, virus type, protein type, and ligand category. We also attempt to facilitate quick overall structure assessment for general users by calculating aggregated quality metrics, such as the PQ1 percentile. Finally, we make sure that isomorphous structures solved in the same space group can be easily compared, by moving them into standardized location of the reference unit cell. Although non‐uniform model placement in the unit cell may not seem to be a serious issue for trained crystallographers, for many biomedical researchers it makes comparison harder as isomorphousstructures may appear to be completely different, leading to confusion and misinterpretations.

During the work on the server, we made several disturbing observations. First, in several cases, the deposited images were clearly not compatible with the diffraction data used for structure refinement. Second, some of the contacted scientists claimed that the diffraction data were deleted immediately after processing in order to save disk space. Third, several scientists did not respond to our request to provide their data, despite the IUCr recommendation and an appeal from the community to make diffraction data related to SARS‐CoV‐2 public (http://phenix‐online.org/pipermail/phenixbb/2020‐March/024556). However, in one case our request resulted in the original authors re‐depositing an optimized structure instead of depositing the diffraction data. All of the above facts show that the struggle for reproducibility of scientific results is an uphill battle, and suggest that leading scientific journals should do more , than run editorials about the need to improve the reproducibility of scientific results.

It is worth noting that the described web resource is not the only project established with the aim of validating, correcting, or providing additional information on COVID‐19 related structures. The most notable other examples have been set by Gerard Bricogne's group at Global Phasing (https://www.globalphasing.com/buster/wiki/index.cgi?Covid19), Jane and David Richardson's group at Duke University (http://kinemage.biochem.duke.edu/), and Andrea Thorn's group at the University of Würzburg (https://github.com/thorn-lab/coronavirus_structural_task_force).

With vaccines in late‐stage development , and the first reports of drugs increasing survival chances, the COVID‐19 pandemic will hopefully end soon. However, this may not necessarily be the end of the SARS‐CoV‐2 coronavirus, as it may evolve in yet unforeseen ways to evade vaccines and treatments. Therefore, we will keep improving the web resource presented herein, with the hope that it will remain useful to biologists during the present pandemic and that it will set standards for any future health crises.

AUTHOR CONTRIBUTIONS

Dariusz Brzezinski: Data curation; methodology; software; validation; visualization; writing‐original draft; writing‐review and editing. Marcin Kowiel: Data curation; investigation; software; validation; writing‐original draft; writing‐review and editing. David Cooper: Data curation; investigation; validation; writing‐review and editing. Marcin Cymborowski: Data curation; software; validation. Marek Grabowski: Formal analysis; software; writing‐review and editing. Alexander Wlodawer: Conceptualization; data curation; formal analysis; investigation; supervision; writing‐review and editing. Zbigniew Dauter: Formal analysis; investigation; validation; writing‐review and editing. Ivan Shabalin: Data curation; formal analysis; investigation; methodology; validation; writing‐original draft; writing‐review and editing. Miroslaw Gilski: Formal analysis; validation; writing‐review and editing. Bernhard Rupp: Data curation; formal analysis; investigation; methodology; validation; writing‐review and editing. Mariusz Jaskolski: Conceptualization; formal analysis; funding acquisition; methodology; supervision; validation; writing‐original draft; writing‐review and editing. Wladek Minor: Conceptualization; data curation; formal analysis; funding acquisition; project administration; resources; supervision; validation; writing‐original draft; writing‐review and editing.

CONFLICT OF INTEREST

One of the authors (WM) notes that he has also been involved in the development of software and data management and mining tools; some of them were commercialized by HKL Research and are mentioned in the paper. WM is the co‐founder of HKL Research and a member of the board. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

Supplementary Table S1 Synchrotron names and abbreviations.

Supplementary Figure S1. Report of an optimization when diffraction data were available (deposit 7BRR); part II.

Supplementary Figure S2. Report of minor structure optimization when diffraction data were not available (deposit 6VW1); part I.

Supplementary Figure S3. Report of minor structure optimization when diffraction data were not available (deposit 6VW1); part II.

Click here for additional data file.