EVAcon: a protein contact prediction evaluation service.

Here we introduce EVAcon, an automated web service that evaluates the performance of contact prediction servers. Currently, EVAcon is monitoring nine servers, four of which are specialized in contact prediction and five are general structure prediction servers. Results are compared for all newly determined experimental structures deposited into PDB ( approximately 5-50 per week). EVAcon allows for a precise comparison of the results based on a system of common protein subsets and the commonly accepted evaluation criteria that are also used in the corresponding category of the CASP assessment. EVAcon is a new service added to the functionality of the EVA system for the continuous evaluation of protein structure prediction servers. The new service is accesible from any of the three EVA mirrors: PDG (CNB-CSIC, Madrid) (http://www.pdg.cnb.uam.es/eva/con/index.html); CUBIC (Columbia University, NYC) (http://cubic.bioc.columbia.edu/eva/con/index.html); and Sali Lab (UCSF, San Francisco) (http://eva.compbio.ucsf.edu/~eva/con/index.html).

INTRODUCTION

Contacts between amino acid side chains are fundamental during protein folding, especially those contacts between pairs of residues that preserve the native state of the protein. These contacts are often well separated in the sequence but close in the protein 3D structure.

New methods to predict contacts, based on different approaches, are continuously appearing (1–6). The importance of contact prediction has been pointed out in international experiments, such as the Critical Assessment of protein Structure Prediction [CASP, (7)], where contact prediction is evaluated within the ab initio category, and the Critical Assessment of Fully Automated protein Structure Prediction [CAFASP, (8)] where these kind of predictions are evaluated as well.

These experiments revealed that contact prediction techniques are still not of sufficient quality to allow them to be used as the only source of information in protein structure prediction, even though, as demonstrated in the comparison carried out in the context of the CAFASP3 challenge, they can predict contacts of a similar quality to other ab initio methods, i.e. fragments-based methods, such as Rosetta (9), threading methods, such as GenTHREADER (10) and FUGUE (11), and comparative modelling servers, such as Pcomb (12).

The need to monitor progress in this field motivated us to develop an automatic contact prediction evaluation service that will allow the scientific community to compare results of these kind of predictions in a systematic way. In particular we are interested in establishing the limits reachable by contact prediction methods in comparison with other prediction approaches.

EVAcon evaluates raw contact predictions produced by the contact specialists and also evaluates, in terms of implicitly predicted 3D contacts, methods for the full reconstruction of protein models (i.e. comparative modelling and threading/fold recognition servers). The results of EVAcon allow a direct comparison between contact predictions produced by the specialists and predictions extrapolated from the 3D models produced by protein structure prediction servers. Overall, the results are potentially informative about the ability of the contact specialists to produce 3D restraints, something that could be used by other ab initio methods to produce protein models, or that could be used as additional information to select models generated by other prediction methods (13–15). EVAcon realises the continuous evaluation of protein structure prediction servers, using the facilities of the EVA system for assessing the servers, retrieving the new structures from PDB and building homogeneous datasets for testing the methods (16,17).

METHOD OVERVIEW

Analysis of contacts

A set of estimations are applied for the evaluation of the contact predictions following the standards commonly accepted in the field and used for the evaluations of CASP and CAFASP (see for the description of the evaluation). Two residues are considered to be in contact when the separation between their C-beta atoms (C-alpha for Gly) is ≤8 Å. There is also the possibility of re-evaluating the predictions with a variable distance threshold (which can be chosen by the user) for the definition of contact. For 3D models without side chains, the positions of the potential side chains are generated with MaxSprout (18).

Predictions from contact specialists are usually received as a list of predicted contacts sorted by a reliability value. We evaluate different levels of prediction, by selecting a number of predicted contact pairs proportional to the protein length. For 3D prediction servers that do not have values of reliability associated to each pairwise distance, we sample an equivalent number of predicted pairs. Evaluations are carried out for a number of pairs corresponding to: L/10, L/5, L/2, L and 2 times the protein length (L).

The predictions are evaluated in terms of accuracy (Acc), improvement over random (Imp), coverage (Cov) and Xd. The accuracy of a random prediction (AccRand) is obtained assuming that all the possible pairs of the experimental structure in the corresponding ranges of sequence distance separation are potential contacts (C). Xd is the weighted harmonic average difference between the distance distribution of the predicted contacts and the all-pairs distance distribution.

There are 15 distance bins covering the range from 0 to 60 Å. The sum runs for all the distance bins. di is the distance representing each bin, the upper limit (normalized to 60). Pip is the percentage of predicted pairs whose distance is included in the i bin. Pia is the same for all the pairs. Defined in this way, Xd > 0 indicates the positive cases where the population of predicted contacts' distances is shifted to lower distances (19). For a random prediction, Xd will have a value around zero.

The predictions are also evaluated for several values of sequence deviation from the predicted residue (delta, 14), giving the percentage of correctly predicted contacts within delta residues of the experimental contact. We apply delta values from 0 to 5. This means that a predicted contact between two residues i and j is considered correct if there is a contact observed between any (i − delta, i + delta) and any (j − delta, j + delta).

All these measures are calculated for three different ranges of contacts: short range contacts (at least 6 residues of sequence separation between the residues of the pair), medium range contacts (at least 12 residues of sequence separation between the residues of the pair) and long range contacts (at least 24 residues of sequence separation between the residues of the pair).

Servers participating

A set of prediction servers have been evaluated in EVAcon during the last 30 weeks (Table 1). We would like to invite any other automated servers to join the EVA evaluation system.

Table 1

Servers under EVAcon evaluation

Name	Method	Server location and URL	Reference
Contact prediction servers
PROFcon	Back propagation neural network that combines information from alignments, from one-dimensional predictions, from the region between two contacting residues, and from the average properties of the entire protein chain	CUBIC, Columbia University ()	–
GPCpred	Evolutionary algorithm (genetic programming) that selects residues and residue pairs likely to make contacts based solely on local sequence patterns extracted with the help of self-organizing maps	Stockholm Bioinformatics Center ()	(5)
CMAPpro (5 different versions)	Recurrent neural network implementations of a class of Bayesian networks called generalized input-ouput Hidden Markov Models (GIOHMMs)	Institute for Genomics and Bioinformatics, University of California-Irvine ()	(3)
PDGcon	Contact predictions based on correlated mutations	Protein Design Group, CNB-CSIC ()	(1)
3D structure prediction servers
FUGUE	Program for recognizing distant homologues by sequence-structure comparison. It utilizes environment-specific substitution tables and structure-dependent gap penalties, where scores for amino acid matching and insertions/deletions are evaluated depending on the local environment of each amino acid residue in a known structure.	Department of Biochemistry, University of Cambridge ()	(11)
WURST	Threading server with a structural scoring function, sequence profiles and optimized substitution matrices	Center for Bioinformatics, University of Hamburg ()	(20)
SAM-T99 and SAM-T02	Iterative Hidden Markov Model-based method for finding proteins similar to a target sequence	Biomolecular Engineering, University of California-Santa Cruz () ()	(21,22)
LIBELLULA	Neural network approach that improves the selection of correct folds from fold recognition results given by SAM-T99 and 3D-PSSM (23)	Protein Design Group, CNB-CSIC ()	(24)

Viewing results

The current interface provides an easy navigation for target structures and prediction methods (Figure 1):

Figure 1

Interfaces of the main EVAcon modules. The system is composed of three different modules: One of them provides a fast view of the results through static pages that are regenerated every week. The second one is composed of a query-based system that allows a flexible display of the results. The third module is a service for the direct evaluation of contact predictions that can be submitted by users and method developers.

EVAcon provides a fast graphical view with directly accessible results that give an overview of the performance of the servers. This initial view is composed of static pages that are regenerated every week. These pages include results (with graphs, tables and raw files for Acc, Imp, Cov and Xd) for each individual server, a summary of the mean accuracy over the weeks of the current year and also a comparison of the accuracies of the servers for the set of all the proteins during the current week.

The dynamic section provides an in-depth view of the results, suitable for method developers. A query-based system allows a flexible display of results. For example, it makes it possible to define subsets of proteins of interest common to all methods, and to visualize the comparative performances of the servers on these proteins. The results are presented as graphs, tables and raw files.

EVAcon includes an additional facility for the direct evaluation of contact predictions that is intended for developers in order to evaluate the predictions of their methods according to the EVA criteria. These can be submitted in contact prediction format or as PDB files together with the corresponding experimental structure.

Another possibility of the system is to perform evaluations with a variable distance cutoff for the definition of contact which can be chosen by the user (apart from the default—8 Å between C-beta atoms). This option can be accessed also from the main page of EVAcon, through the link ‘Choose a different contact distance and see results for the evaluated targets’. This option is also available for the evaluation of predictions submited by the user [point (iii) above].

It is still too soon to establish a clear ranking of the performance of the servers, due to the small number of weeks that the evaluation has been active. Despite this, the assessment yields very similar results for the contact specialist methods and the 3D structure prediction servers. For some protein targets, contact specialists are performing much better than the 3D prediction methods evaluated by EVAcon. For instance, protein 1pxe chain A (zinc binding domain of the neural zinc finger transcription factor 1), where GPCpred (5) has an accuracy of 50% with a coverage of contacts predicted of 27.3%, twice the accuracy and coverage of FUGUE (11), the best performing threading server for this protein. Another interesting example is the case of the target 1xjh chain A (redox switch domain of the Escherichia coli Hsp33), where the best result is produced by CMAPpro_band (3), with a prediction accuracy of 66.7% and a coverage of 20%, and where the best threading prediction, in this case done by WURST (20), is only able to achieve an accuracy of 58.3% and a coverage of 17.5%. These results seem to confirm our initial observations (8), and if confirmed in the following weeks with additional proteins and methods, it will demonstrate the utility of adding contact prediction methods to the set of tools used in fold recognition approaches.

DISCUSSION

The prediction of 3D contacts between amino acids is an active field of research, in which new methods are continually appearing.

We introduce here EVAcon, an automatic system for the continuous evaluation of contact predictions. With the EVAcon evaluation, we want to help developers to test their new methods, and compare their results with others. The dynamic part of the system is particularly interesting for them, since it allows the production of common subsets of proteins adequate for the equilibrated evaluation of the methods. But above all, we think EVAcon can be very useful for biologists as a way to check which servers produce better predictions (or predictions with certain characteristics of coverage, accuracy, …), based on an independent statistically reliable comparison. One of the reasons for that is that the comparison between methods based on the original publications is difficult since they are based on different datasets.

EVAcon can also be useful to test the potential use of contact prediction as additional restraints incorporated in other methods, for example, as assistance in the selection of correctly folded models. We also think that the system can be of interest for protein modellers searching for the best method from which to obtain additional structural information.

It will allow a statistically valid comparison between contact prediction methods and general protein structure prediction methods.

EVAcon performs evaluations once a week, like the other EVA evaluation modules. This continuous assessment produces rankings that are statistically more significant than those at CASP, an experiment that takes place over just 2–3 months every 2 years, over a relatively small number of target sequences. Unlike CASP the predictions are assessed entirely by automatic methods.

We hope to be able to increase the number of servers evaluated by EVAcon and to produce an accurate evaluation of the current methods when the evaluation server has been running for a sufficient number of weeks. Furthermore, we foresee using the results of EVAcon to evaluate the relation between prediction methods and classes of proteins, under the assumption that some contact methods might lead to better predictions for some specific class of proteins or structural motifs.