The application of statistical methods to cognate docking: a path forward?

Cognate docking has been used as a test for pose prediction quality in docking engines for decades. In this paper, we report a statistically rigorous analysis of cognate docking performance using tools in the OpenEye docking suite. We address a number of critically important aspects of the cognate docking problem that are often handled poorly: data set quality, methods of comparison of the predicted pose to the experimental pose, and data analysis. The focus of the paper lies in the third problem, extracting maximally predictive knowledge from comparison data. To this end, we present a multistage protocol for data analysis that by combining classical null-hypothesis significance testing with effect size estimation provides crucial information about quantitative differences in performance between methods as well as the probability of finding such differences in future experiments. We suggest that developers of software and users of software have different levels of interest in different parts of this protocol, with users being primarily interested in effect size estimation while developers may be most interested in statistical significance. This protocol is completely general and therefore will provide the basis for method comparisons of many different kinds.