Open Source Bayesian Models. 1. Application to ADME/Tox and Drug Discovery Datasets.

On the order of hundreds of absorption, distribution, metabolism, excretion, and toxicity (ADME/Tox) models have been described in the literature in the past decade which are more often than not inaccessible to anyone but their authors. Public accessibility is also an issue with computational models for bioactivity, and the ability to share such models still remains a major challenge limiting drug discovery. We describe the creation of a reference implementation of a Bayesian model-building software module, which we have released as an open source component that is now included in the Chemistry Development Kit (CDK) project, as well as implemented in the CDD Vault and in several mobile apps. We use this implementation to build an array of Bayesian models for ADME/Tox, in vitro and in vivo bioactivity, and other physicochemical properties. We show that these models possess cross-validation receiver operator curve values comparable to those generated previously in prior publications using alternative tools. We have now described how the implementation of Bayesian models with FCFP6 descriptors generated in the CDD Vault enables the rapid production of robust machine learning models from public data or the user's own datasets. The current study sets the stage for generating models in proprietary software (such as CDD) and exporting these models in a format that could be run in open source software using CDK components. This work also demonstrates that we can enable biocomputation across distributed private or public datasets to enhance drug discovery.

Introduction

For well over a decade, the cost of in vitro and in vivo screening of absorption, distribution, metabolism, excretion, and toxicity (ADME/Tox) properties of molecules has motivated efforts to develop various in silico methods to efficiently pre-filter candidates for actual physical testing.[1−29] By relying on very large, internally consistent datasets, large pharmaceutical companies have succeeded in developing highly predictive but ultimately proprietary models.[29−33] At one pharmaceutical company, for example, many of these models (e.g., volume of distribution, aqueous kinetic solubility, acid dissociation constant, distribution coefficient, microsomal clearance, CYP3A4 time-dependent inhibition)[30−36] as well as other endpoints[15,22] have achieved such high accuracy that they have essentially put the experimental assays out of business. It is likely that most large pharmaceutical companies can now perform experimental assays for a small fraction of compounds pre-filtered through the proprietary ADME/Tox and physicochemical property computational models, thus improving cost efficiency while minimizing in vitro and animal experimentation. Extra-pharma computational efforts have not been so successful, largely because they have, by necessity, drawn upon considerably smaller datasets, in many cases trying to combine information from the literature.[37−43] This situation, however, has improved with larger datasets publicly available in PubChem,[44,45] ChEMBL,[46−48] CDD,[49] and others, and some drug companies depositing their data (e.g., the recently deposited AstraZeneca data in ChEMBL), which can be useful for model building.[50−53]

ADME/Tox properties have been modeled by us[1,54−81] and many other groups[29,82] using an array of machine learning algorithms such as support vector machines,[59] Bayesian modeling,[19] Gaussian processes,[83] and many others.[84] A more exhaustive review of the different machine learning approaches is outside the scope of this work. These combined efforts at ADME/Tox model building have likely resulted in hundreds of published models which are, unfortunately, inaccessible to anyone but their authors in most cases. This limited access problem for published models is also likely the case with computational models for bioactivity or other physicochemical properties of interest. The ability to share such models freely still remains a major challenge when dealing with issues of proprietary samples or data, as repercussions for such for-profit pharmaceutical companies could be severe. The current development of technologies for open models and descriptors builds on established methodologies.[85−88] Datasets for quantitative structure–activity relationships (QSAR) have previously been represented in a reproducible way via QSAR-ML.[85] These methods also come with a reference implementation for the Bioclipse workbench,[86,87] which provides a graphical interface. There have been several early efforts at cheminformatics Web services; e.g., Indiana University provides access to cheminformatics methods (fingerprints, 2D depiction, and various molecular descriptors) and statistical techniques. These have been used to develop models for the NCI60 cancer cell lines.[89,90] In addition, there are Web tools for the prediction of bioactivities and physicochemical properties, like the Chemistry Activity Predictor (GUSAR).[91] Also, the Open Notebook Science (ONS) project[92] has developed models for solubility and melting point using web services based on open descriptors and algorithms. These tools all enable parties to collaborate publicly but do not facilitate private or selective collaboration.

We have previously demonstrated a proof of concept using open descriptors and modeling tools to model very large ADME datasets at Pfizer.[22] Models were constructed with open descriptors and keys (CDK + SMARTS) using open software (C5.0) and performed essentially identically to expensive proprietary descriptors and models (MOE2D + SMARTS + Rulequest’s Cubist) across all metrics of performance when evaluated on human liver microsomal stability (HLM), RRCK passive permeability, P-gp efflux, and aqueous solubility datasets.[22] Pfizer’s HLM dataset, used in this study, contained more than 230,000 compounds and covered a diverse range of chemistry space as well as addressing many therapeutic areas. The HLM dataset was split into a training set (80%) and a test set (20%) using the venetian blind splitting method. In addition, a newly screened set of 2310 compounds was evaluated as a blind dataset. All the key metrics of model performance, e.g., R2, RMSE, kappa, sensitivity, specificity, and positive predictive value (PPV), were nearly identical for the open source approach vs proprietary software (e.g., PPV = 0.80 vs 0.82).

Our goal is to enable extra-pharma drug discovery projects to exploit in silico machine learning methods that have, until now, been confined in practice to pharma and to a few knowledgeable academics. These methods better exploit limited screening resources and will enable such projects to cover more unexplored chemical space and to address ADME/Tox earlier in the discovery process. Extra-pharma projects represent a growing trend for commercial drug discovery[93−95] to be the principal efforts to find cures for many neglected diseases (e.g., tuberculosis, malaria, Chagas disease, visceral leishmaniasis, etc.), and thousands of orphan indications[96,97] will require more collaborations and data, and therefore model sharing. This approach has the potential to accelerate the discovery of promising drug-like lead compounds with acceptable properties in vivo and ultimately yield a significant impact on global health.

We now describe the creation of a reference implementation of a Bayesian model-building software module, which we have released as an open source component that is now included in the Chemistry Development Kit (CDK) project[98,99] and incorporated using the FCFP6 descriptors in the CDD Vault, which was also recently made open source.[100] We make use of the CDD Public database,[49] which has over 100 public datasets that can be used to generate community-based models, including extensive neglected infectious disease SAR datasets (malaria, tuberculosis, Chagas disease, etc.), and ADMEdata.com datasets that are broadly applicable to many projects. An accompanying paper uses this software to develop models on a much larger scale.[101]

Experimental Section

Data and Materials Availability

All computational models are available from the authors upon request. All molecules for malaria, tuberculosis, and cholera datasets from Table 1 are available in CDD Public (https://app.collaborativedrug.com/register), and the models from Table 2 are available from http://molsync.com/bayesian1.

Table 1

Datasets Used for Bayesian Models Created with CDD Models Using FCFP6 Fingerprints

model	datasets used and refs	cutoff for active	no. of molecules	three-fold ROCa
malaria (Plasmodium falciparum)	CDD Public datasets (MMV, St. Jude, Novartis, and TCAMS)[127−129]	3D7 EC50 <10 nM	184 actives, 19,824 inactives	0.97
TB (Mycobacterium tuberculosis)	CDD Public datasets from NIAID/SRI (MLSMR, CB2, kinase)[138−140]	Mtb inhibition >90%	6891 actives, 210,190 inactives	0.88
TB (Mycobacterium tuberculosis)	CDD Public datasets from NIAID/SRI (MLSMR, CB2, kinase, and ARRA)[138−141]	Mtb IC50 or IC90 <10 μM	3712 actives, 1145 inactives	0.89
TB (Mycobacterium tuberculosis)	CDD Public MLSMR single-point data	Mtb inhibition >90%	3986 actives, 210,447 inactives	0.87
TB (Mycobacterium tuberculosis)	CDD Public MLSMR dose–response	Mtb IC50 <10 μM and classed as active	624 actives, 1649 inactives	0.75
TB (Mycobacterium tuberculosis) efficacy in vivo mouse	CDD Public[112]	described in ref (112)	371 actives, 407 inactives	0.73
cholera	CDD Public in the TB ARRA dataset[141]	IC50 <5 μM	50 actives, 1874 inactives	0.93
Ames mutagenicity	ref (142)	Ames positive, active = 1	3501 actives, 3007 actives	0.83
mouse intrinsic clearance	data from ChEMBL	<10 μL/(min·g)	52 actives, 312 inactives	0.82
human intrinsic clearance	data from ChEMBL	≤10 μL/(min·g)	105 actives, 638 inactives	0.92
human intrinsic clearance	AZ data from ChEMBL[143]	≤10 μL/(min·mg)	496 actives, 604 inactives	0.80
Caco-2	proprietary data from ADMEdata.com	pH 6.5, cutoff >1×10–5	181 actives, 325 inactives	0.79
Caco-2	data from ChEMBL	cutoff >1×10–5	60 actives, 399 inactives	0.89
5-HT2B	ref (144)	active = 1, described in ref (144)	146 actives, 607 inactives	0.89
solubility	ref (145)	Log solubility = −5	1136 actives, 154 inactives	0.87
PXR activation	ref (146)	described in ref (146)	174 actives, 143 inactives	0.80
maximum recommended therapeutic dose	ref (147)	>10 mg/(kg·day)	350 actives, 813 inactives	0.85
blood brain barrier permeability	ref (28)	BBB positive, described in ref (28)	1472 actives, 432 inactives	0.92

ROC = receiver operator characteristic integral.

Table 2

Datasets Used for Bayesian Models Created for Use by MMDS, with ECFP6 Fingerprintsa

model	datasets used and refs	cutoff for active	no. of molecules	five-fold ROC
solubility	ref (145)	Log solubility = −5	1144 actives, 155 inactives	0.86
probe-like	ref (148)	described in ref (148)	253 actives, 69 inactives	0.76
hERG	ref (149)	described in ref (149)	373 actives, 433 inactives	0.85
KCNQ1	PubChem BioAssay: AID 2642[150]	using actives assigned in PubChem	301,737 actives, 3878 inactives	0.84
Bubonic plague (Yersina pestis)	PubChem single-point screen BioAssay: AID 898	active when inhibition ≥50%	223 actives, 139,710 inactives	0.81
Chagas disease (Typanosoma cruzi)	Pubchem BioAssay: AID 2044	with EC50 <1 μM, >10-fold difference in cytotoxicity as active	1692 actives, 2363 inactives	0.8
TB (Mycobacterium tuberculosis)	in vitro bioactivity and cytotoxicity data from MLSMR, CB2, kinase, and ARRA datasets[110]	Mtb activity and acceptable Vero cell cytotoxicity selectivity index = (MIC or IC90)/CC50 ≥10	1434 actives, 5789 inactives	0.73
malaria (Plasmodium falciparum)	CDD Public datasets (MMV, St. Jude, Novartis, and TCAMS)[127−129]	3D7 EC50 <10 nM	175 actives, 19,604 inactives	0.98

All eight models are ECFP6, with folding into 32,768 slots.

Laplacian-Modified Naïve Bayesian Definition and Pseudocode

Bayesian models have been a useful part of computer-aided drug discovery for many years and were popularized in Pipeline Pilot.[19,102,103] The statistical method is particularly useful for correlated structure-derived fingerprint bit strings with an activity measurement that has been classified as active or inactive on the basis of a selected threshold. Variants on the original Bayes theorem can be used to produce an estimate of the likelihood of activity for proposed compounds. For reproducing binary classifications, Bayesian methods have several appealing features. The model creation process is typically very fast and can be implemented in O(N) time, which means that an ordinary desktop computer can build and evaluate models with hundreds of thousands of compounds with minimal delay. When general purpose structure-derived fingerprints are used, the methods tend to be quite robust, which is in contrast to methods such as QSAR, which require some expertise to select appropriate descriptors, avoid overtraining, and ensure domain applicability.

Unlike many other applications of Bayesian methods, the use of chemical structure fingerprints as inputs means priors often numbers in the thousands, which produces scale distortions. Even if the prior probabilities are approximately 0.5 in each case, multiplying thousands of such values together tends to warp the distribution of the posterior probabilities to being asymptotically close to 0 or 1, which introduces numerical precision issues. By summing the logarithms of the ratios rather than multiplying the fractions, and incrementing the numerator and denominator, the precision issues are eliminated, and the resulting predictions tend to follow a linear distribution.

The main drawback with this particular Bayesian variant is that the resulting prediction is not a probability, but rather an arbitrary number that has no particular upper or lower bound. The results can be converted into a two-state classification by selecting a threshold, or into a probability-like value by picking a linear transformation function, but these are post-Bayesian calibrations that must be made by applying judgment criteria that are not an intrinsic part of the method.

We have used our work[100] on the reference extended connectivity ECFP and FCFP fingerprints to create a software class that allows Bayesian models to be created from a collection of molecules and activity data, used to predict probabilities for new molecules, and to serialize/deserialize the model as a structure text string that can be saved to a file or shared with any other package that implements the same functionality.

The Laplacian-modified naïve Bayesian (which we call Bayesian models for simplicity) formula uses a simple definition, which pre-supposes that each molecule has been described by enumerating a list of fingerprints that applies to it, and has a determination of whether it is active or inactive. For each fingerprint code in the entire dataset:where C is the contribution associated with the presence of a fingerprint hash code i, which is in turn derived from A and T, which are respectively the number of active molecules with the fingerprint and the total number of molecules with the fingerprint, while R is the overall fraction of actives.

Building the Bayesian model is a simple matter of determining the total set of fingerprints in the dataset and, for each of them, calculating the value of C. Any fingerprint that is theoretically possible but not encountered in the training set has an implied value of 0. Any fingerprint hash code that is observed equally often in active and inactive molecules (or not at all in either) has a ratio of 1, for which the log value is zero.

When making a prediction for an incoming molecule, the value is determined by adding up the contributions for each fingerprint hash code for the molecule:The resulting prediction, P, is an uncalibrated value: unlike for the conventional Bayes theorem, the result is not a probability and is generally not directly interpretable, meaning that there is no significance to either the scale or offset. Methods for interpreting these values will be discussed subsequently.

Creating a Bayesian model using this method is very fast and has favorable scalability properties, because it requires just two passes through the input collection: the total number of actives and inactives needs to be summed, and after that, each compound needs to be considered only individually. The total memory required to build the Bayesian model is bounded by the theoretical number of fingerprints. For each possible unique fingerprint hash code, it is necessary to store two integers (A, T) and derive one floating point value per fingerprint (C). For small, relatively dense fingerprint schemes, these can be stored in a flat array (e.g., when folding fingerprints into 1024 possible values), but for larger schemes with sparse occupancy it is better to use a dictionary object (e.g., when the full 32-bit range of ECFP6 or FCFP6 fingerprints is allowed).

The pseudocode for the model building is as follows:For making a prediction for an incoming molecule:

let T = empty dictionary (key: hash code i, value: total t)

let A = empty dictionary (key: hash code i, value: actives a)

for m in all molecules in training set:

determine list of fingerprints F for molecule m

for fingerprint hash code i in F:

increment T

if molecule m is active: increment A

let R = total actives/total molecules

let C = empty dictionary (key: hash code i, value: contribution v)

let L = unique list of keys for T

for i in L:

put C = log([A + 1]/[T·R + 1])

let m = molecular structure

determine list of fingerprints F for molecule m

let P = 0

for i in L:

if i is one of the fingerprints in F:

let P = P + C

Implementing these algorithms using a flat array rather than a dictionary object is analogous and differs only in the way indices are looked up.

Chemistry Development Kit

The method described in this article is implemented in the Chemistry Development Kit (CDK) project and made available under the terms of the Lesser Gnu Public License (LGPL). The latest version of the project can be obtained from its SourceForge host and underlying Git repository (http://sourceforge.net/p/cdk/code/ci/master/tree). The Bayesian modeling capabilities are available within the tools section, the main class for which is org.openscience.cdk.fingerprint.model.Bayesian.

Using the CDK library to create a new Bayesian model from a collection of molecule objects and boolean activity values is straightforward. For example, given the filename for an MDL SDfile with a field called “pIC50”, for which any molecule with a value of 6 or greater is considered active, the following Java code snippet can be used to create a serialized model:

The resulting serialized form can be stored for future use. If it is stored in a file, it can be easily retrieved and used to apply to a different SDfile, e.g.:

The first step is to read the model from the pre-existing file (“activity.bayesian”). The second step iterates over the input SDfile, while writing to another SDfile with two extra fields appended: “RawPrediction”, which contains the uncalibrated outcome from the modified Bayesian method, and “ScaledPrediction”, which contains the prediction that has been scaled using metrics originally derived from the internal cross-validation.

These two examples demonstrate the ‘create and consume’ use cases and can be easily adapted to scenarios besides reading and writing from files. Serialized Bayesian models can be embedded in any kind of text-friendly data structure, e.g., XML documents, JSON messages, SQL tables, etc. Use of models to provide predictions can be applied to a variety of invocations, such as command line tools, incorporation into modeling packages with a graphical interface, Web services accessible via API, etc.

File Format

For saving models for subsequent reuse, the information necessary to apply the model to make predictions for new molecules can be stored in a text-based file format. The molecules that were used to build the model are not included in the serialized form, nor are the fingerprints that were generated from them. This means that sharing a serialized model allows the recipient to make inferences on the basis of the original data without explicitly having access to it. For confidentiality purposes, sharing models without the underlying data is useful in a number of situations, but it should be noted that this cannot be considered as entirely foolproof: a determined hacker with some context would likely be able to make a well educated guess as to the actives contained in the training set.

Figure 1 shows an example of a serialized file. The default file extension is .bayesian, and the MIME type is chemical/x-bayesian. The text should be encoded as UTF-8 unicode, for which all of the content is limited to the ASCII subset, except for the freeform text notes. End of line should be encoded Unix-style, and floating point numbers can be encoded with a decimal point (e.g., 1.23, with a period symbol for the separator, invariant of localization) or scientific notation (e.g., 1.23 × 10–9). The format is case- and whitespace-sensitive. The body of the format consists of individual lines, each of which encodes a discrete property, and is of arbitrary length.

Figure 1

Example of a serialized file containing a very small Bayesian model. The default file extension is .bayesian, and the MIME type is chemical/x-bayesian.

The first line contains the header, which consists of the recognition sequence and essential information about the model. The first nine characters are always set to the ASCII characters for the string “Bayesian!” (hex: 42 61 79 65 73 69 61 6E 21), which can be used as a recognition sequence. This is useful for situations such as embedding in streams, within whitespace-padded subfields such as XML elements, or when the file extension or MIME type is unavailable or unreliable.

The recognition sequence is followed by four comma-separated fields: fingerprint type, folding length, calibration minimum, and maximum. Only the first two fields are mandatory, and parsers should ignore additional fields, in case the format is subsequently extended.

The fingerprint type must be one of ECFPn or FCFPn, where n is 0, 2, 4, or 6. These correspond to the eight different permutations of circular fingerprints that are implemented in the CDK library. The most commonly used values are ECFP6 and FCFP6. The variety of fingerprints may be extended at a later date. When a parser encounters a fingerprint type that it does not recognize, it should invoke an error pathway if there is any intention of applying the model to new molecules, since the ability to produce the exact same fingerprints is a pre-requisite. The folding length should either be 0 (no folding, i.e., full range of 32-bit integers) or a power of 2 (e.g., 512, 1024, 2048, etc.). The parser should fail for invalid folding lengths. The calibration minimum and maximum values are used to transform raw predictions into a probability-like range. Since this is calculated by analyzing the cross-validation metrics, which is an optional step, the information may not be available. If not included, then the model can only be used to generate raw prediction values. Note that sometimes the minimum and maximum values are equal, which can occur for datasets that are small or trivial. In this case, the degenerate value should be treated like a simple threshold, giving results of 0 or 1, rather than a probability-like transform.

The model specification ends with a line beginning with the string “!End”. This should be considered as the terminator sequence regardless of trailing characters or whitespace. All lines in between the header and footer can be examined out of order.

Lines that match the template {, where bit index is an integer and contribution is a floating point number, make up the payload of the model. The contributions-per-bit are typically stored in a dictionary object, since the bit coverage is sparse, i.e. usually not all of the possible bits are represented. If the fingerprints are folded, then the bit indices range from 0 to folding-1. If not folded, the indices are represented as signed integers: approximately half of the values will be negative.

For generating raw Bayesian predictions, all that is needed is fingerprint type, folding, and contribution list. All remaining lines are optional and must follow the general format of {, whereby category and key must be plain ASCII without whitespace, category must begin with an alphanumeric character, and value may contain any unicode characters, except for end-of-line. Duplicates are allowed. When software needs to parse, modify then write a model file, any optional lines that are not understood should be preserved as-is.

The optional data that are currently used by the CDK implementation include the following:

training:size and training:actives: the total number of compounds in the training set, and the number of actives, respectively

roc:auc: integral of the receiver operator characteristic, which is a number between 0 and 1

roc:type: the method used to partition the data for internal cross-validation, which is one of leave-one-out, three-fold, or five-fold.

roc:x and roc:y: two comma-separated lists of numbers from 0 to 1 which can be used to recreate the ROC curve visually. Note that for large datasets, the total number of points may be reduced in order to limit the impact on file size. This means that while the resolution is indistinguishable for graph plotting purposes, recalculating the integral from these points is less precise than using the stored roc:auc value.

note:title: ideally a short free-text description of the model that communicates to a scientist what data were being modeled. It should be expected to be displayed in a single line.

note:origin: a short free-text description that provides information about the provider of the model, be it the software algorithm or the source of the data, or both.

note:comment: a completely freeform field, which may be of any length. Since newlines are disallowed, multiple paragraphs of comments should be encoded by having multiple comment lines.

The file size for a serialized model depends on the size and diversity of the molecules. One of the main reasons for opting to fold the fingerprints is that it places a reasonable maximum limit on the file size. For example, a collection of 7000 molecules with experimentally determined activity against Mycobacterium tuberculosis using ECFP6 with no folding produced a file size of 1.4 Mb. Folding the fingerprints into 32,768 bits reduced the file size to 646 kb, and into 2048 down to 67 kb.[101]

CDD Models

The CDD Vault[104] product makes use of the fingerprinting functionality in the CDK to provide Bayesian model-building capabilities, which we have termed CDD Models. While the Bayesian implementation is proprietary, the underlying algorithm for model generation is equivalent to the method described in this article. Models can be created and used within the CDD Vault environment, and at any time they can be exported using the format described above, which means that they can be utilized by any software that either implements the requisite algorithms described in this article or makes use of the CDK library.

The CDD Models extension is part of CDD Vision in CDD Vault. A model is created by separating a set of molecules into two collections: those that could be considered ‘actives’ and those that can be considered ‘inactives’. These classes are then used to train the model, after a series of steps are taken to ensure logical consistency. These include ensuring that duplicates do not appear in either collection, that there is no overlap between the collections, and that each collection contains at least two molecules. The Standard InChIKey[105] of each molecule is used as the criteria for detecting and removing duplicates. These precautions are addressed in a series of pre-processing steps in the CDD Ruby on Rails application, where the modeling process and molecule management system is hosted, wherein the training sets are algorithmically curated via optimized raw SQL code.

Once the training set has passed these checks and pre-processing, the model is generated. CDD Vault uses the FCFP6[100,102,106] structural fingerprints to build the Bayesian statistical model.[107] This machine learning model is stored as a special type of protocol (category = Machine-Learning model), which provides an ROC plot generated by stratified three-fold cross-validation. This ROC plot is interactive, allowing the user to explore the sensitivity, specificity, and corresponding score cutoff at each point along the curve (Figure 2A). After the model has been created, each molecule in the user’s selected ‘project’ receives a relative score, applicability number (fraction of structural features shared with the training set), and maximum similarity number (maximum Tanimoto/Jaccard similarity to any of the “good” molecules). The model can be subsequently shared with both other users and the user’s other ‘projects’ to score any molecule of interest.

Figure 2

Example of the model output in CDD Models. (A) Model derived from whole-cell datasets from antimalarial screening across four CDD Public datasets (MMV, St. Jude, Novartis, and TCAMS), ∼20,000 EC50 values, cutoff < 10 nM. (B) Options for exporting a model from CDD.

The model can also be exported from CDD Vault by making use of the aforementioned .bayesian file format (Figure 2B). To render a serialized version of a model, CDD Vault feeds the training set structures into the serialization implementation described in the previous section. The connection between the Ruby code in CDD Vault and the Java-based serialization code is accomplished using RJB (Ruby-Java Bridge). Further details on using CDD Models are described in the Supporting Information.

Mobile Apps

Once the Bayesian model building was formalized as part of the CDK project, with a rigorously defined file format, it became a straightforward matter to implement the algorithm on other platforms. We have previously described the implementation of ECFP and FCFP fingerprints in a way that is agnostic to the specifics of any particular cheminformatics toolkit and so can be easily ported to other platforms, such that it is literally compatible with the original reference. We have taken the same approach with the Bayesian modeling and ported enough of the functionality to the iOS mobile platform such that models created with the CDK or CDD Vault using ECFP6 fingerprints can be parsed from within mobile apps and used to make predictions. Currently Bayesian model prediction capabilities have been incorporated into the Mobile Molecular DataSheet (MMDS) app (Figure 3), Approved Drugs, and MolPrime+ (all apps produced by Molecular Materials Informatics). Several useful Bayesian models have been packaged with the apps as default functionality, and it is also possible to import user-created models in order to make structure-based predictions within the mobile app.

Figure 3

Example of the Bayesian model implemented in the MMDS mobile app. (a) hERG model, based on literature data. (b) A molecule from a hERG paper.[151] (c) Results scored with this model (hERG measured IC50 = 24 nM) showing a visually intuitive atom coloring for this and other Bayesian models. This compound would appear to be an inhibitor of hERG and possibly KCNQ1 potassium channels.

Application to Datasets

To illustrate the utility of CDD Models we evaluated several datasets available in CDD public as well as in our own CDD Vaults (Table 1). These include screening datasets for malaria, tuberculosis, and cholera from whole-cell screens, in vivo data from mice treated with potential antituberculars, as well as several ADME/Tox properties such as Ames mutagenicity, mouse and human intrinsic clearance, Caco-2, 5-HT2B, solubility, PXR activation, maximum recommended therapeutic dose, and blood brain barrier permeability data. In all cases, three-fold ROC data were collated.

Several datasets were also selected for integration into mobile apps (including MMDS and Approved Drugs). These datasets included solubility, probe-like,[108] hERG, KCNQ1, screening data for whole-cell phenotypic screens against Bubonic plague as well as Chagas disease. In all cases, five-fold ROC data were collated, summarized, and compared to published results.

Results

Chemistry Development Kit

As mentioned previously, the reference implementation for the Bayesian algorithm and the underlying ECFP/FCFP fingerprints is available in the CDK library, which can be freely downloaded from Github. The open source implementation is also accompanied by a testing library, which runs a battery of tests to ensure that the basic functionality is operating as-described.

CDD Bayesian Models

CDD Models using FCFP6 fingerprints have been demonstrated using diverse datasets, such as from public phenotypic screening, and published ADME/Tox datasets (Table 1). In most cases, the three-fold cross-validation ROC values are >0.75. We have also illustrated with the tuberculosis (M. tuberculosis) and malaria (P. falciparum) screening datasets that very large dose–response or single-point datasets can be constructed by combining datasets in CDD Public. Other datasets were collected for this study by manual mining of ChEMBL (mouse intrinsic clearance, human intrinsic clearance, Caco-2). Although these datasets are generated from many different published datasets, the ROC values are a good starting point (>0.80) and comparable to those obtained from proprietary datasets. Several of these datasets (blood brain barrier permeability and PXR) were used recently for a comparison across SVM and Bayesian methods,[109] and the three-fold cross-validation ROC values were similar to those obtained with five-fold cross-validation in this study. The use of the ROC value in this way is a reasonable method to evaluate the utility of the computational models. However, ideally the use of an additional external test set would provide further confidence. The ROC values for the M. tuberculosis models are comparable to those published recently using a commercial tool. For example, in this study MLSMR single-point model three-fold ROC = 0.87 (Figure S1, five-fold ROC 0.87,[110] leave out 50% × 100 cross-validation ROC = 0.86[111]) and MLSMR dose–response model three-fold cross-validation ROC = 0.75 (leave out 50% × 100 cross-validation ROC = 0.73[111]), M. tuberculosis efficacy in mouse three-fold ROC = 0.73 (five-fold ROC = 0.73[112]), and Ames mutagenicity three-fold ROC = 0.83 (five-fold ROC = 0.84[109]).

Mobile App Bayesian Models

The models developed using the same underlying code in mobile apps used the ECFP6 descriptors, and all eight models described had five-fold ROC values >0.75 (Table 2). Several of these datasets have previously been used to generate SVM and Bayesian methods with FCFP6 descriptors[109] using other software. For example, the three-fold ROC for the probe-like dataset in this study was 0.76 (five-fold ROC = 0.73[108]), the three-fold ROC for the hERG dataset was 0.85 (five-fold ROC = 0.84[109]), and the three-fold ROC for the KCNQ1 dataset was 0.84 (five-fold ROC = 0.86[109]). The models derived with FCFP6 (Table 1) and ECFP6 descriptors (Table 2) can be compared; e.g., the three-fold ROC for the malaria dataset using FCFP6 was 0.97 (Table 1) (using ECFP6 for the same dataset five-fold ROC = 0.98, Table 2).

These examples of models generated previously and now with open source descriptors and algorithms suggest they are likely comparable (based on ROC values) and will be evaluated prospectively in future studies. We have also made the models in the mobile app freely accessible via the link http://molsync.com/bayesian1, which is summarized in Figure 4.

Figure 4

Screenshots summarizing the ROC plots and active and inactive compounds for eight models implemented in MMDS.

Discussion

We have recently suggested how providing computational models tightly integrated in software used for storing and sharing chemistry and biology data will be useful for decision making.[113] Some resources exist such as qsardb.org and ochem.eu for public model sharing and development,[114,115] while another, Chembench, provides a resource for creating and using models and other cheminformatics tools privately.[116] Our work, proposing that open source descriptors and algorithms are comparable to commercial software in performance,[22] will ideally lead to more sharing of computational models. At approximately the same time, QSAR-ML[85] was developed to enable standards for interoperability of QSAR models.[85,117] We now build on this prior work as the current study sets the stage for being able to generate a model in proprietary software such as CDD Vault and export a model in a format that could be run in open source software using CDK components. This is a significant advance, because it means that a shared Bayesian model can in principle be used by anyone, regardless of which commercial software packages they have licenses to, since the model capabilities are implemented by an open source toolkit that runs on essentially every desktop platform (CDK is written in cross-platform Java). The creation of additional products that implement the same identical reference algorithm, e.g., mobile apps,[100,118−123] makes use of shared models increasingly convenient. None of the existing Web sites for creating or storing QSAR models appear to offer this capability.

In terms of willingness to share models, sharing with collaborators is one thing, while sharing models openly with the community at large is another, but we have at least removed the main technology hurdle for fingerprint-based Bayesian models in this study. As previously noted, the shared models do not contain chemical structures or the fingerprints corresponding directly to them. However, the direct correlation between structural features and fingerprint does provide clues as to what active molecules an organization may have been using to build their models, and so this caveat must be taken into account when trustworthiness cannot be assumed. While additional security measures are appropriate for the world of proprietary high-value disease targets, this is much less of an issue for rare or neglected diseases, which is where we believe that open model sharing will have the greatest impact.[113] There has been considerable research and discussion on efforts to securely share chemistry data,[124−126] and some of these approaches could be implemented to encrypt models in future.

We have now described how implementation of Bayesian models with FCFP6 descriptors generated in the CDD vault enables the rapid creation of machine learning models from public datasets or the user’s own proprietary data. We also enable the resultant models to be selectively shared (or not) within CDD without having to disclose the underlying data—this represents a practical middle ground, where a trusted broker (CDD) allows a research group to share some of the benefits of their results, but not necessarily full access to the raw data, nor sufficient detail to reverse engineer it. Since sharing is not mandatory, and the option exists to export a model in an open format that can be used by anyone, this means that the full spectrum of model sharing options is available. Providing researchers with greater flexibility to designate and share models with specified collaborators, over particular time intervals, and with clear rights encourages data exchange by allowing researchers to share on terms they control. More fine-grained access control will expand the boundary of what models can be shared to fit the comfort levels of scientists (and their management and lawyers). From having been involved in a number of collaborations large and small, we have observed considerable variation in the need for security and desired degree of openness.

The possibility of using such models to further drug discovery for neglected diseases is of considerable interest, since the available software has traditionally catered to the proprietary market that provides most of the funding. There is now a significant amount of SAR data for rare and neglected diseases that is publicly available, and following up on open data with open source modeling algorithms is an important step. For example, pharmaceutical companies and other research groups have performed high-throughput screens on likely millions of compounds in the search for antimalarials, but they have generally only offered up the active compounds,[127−130] some of which are available in CDD Public. By selecting a cut-off for activity for the antimalarial data that is very stringent (e.g., <10 nM) in CDD Models one can construct a Bayesian model with a three-fold ROC = 0.97 upon combining four public datasets (Table 1). This model may be useful for virtual screening of future compound libraries and complements our other efforts at machine learning models for antimalarial research.[131] Ideally having access to the millions of other inactive compounds would also be useful, although one could imagine a company could just make a model available by selecting a cut-off for inactives as we have demonstrated herein.

Any efficiencies that can be gained in drug discovery would be highly desirable as it is widely known it is both time-consuming and very costly.[93,94] Therefore, the use of tools like computational models that can point out drug candidate liabilities earlier will have considerable value.[1,132,133] With a considerable percentage of drug failures attributed to ADME/Tox issues,[1,43] it is still important to assess these qualities early in the drug development process. Running experimental ADME/Tox assays on each compound for initial screening of chemical libraries is cost- and labor-intensive,[1,43] while computational approaches that rapidly and reliably predict these qualities are gaining more acceptance in the drug discovery community. It is therefore possible to exclude compounds that are most likely to exhibit undesirable ADME or toxicity problems sooner. We present an approach to drug discovery using computational methods for predicting whole-cell activity as well as ADME/Tox and physicochemical properties that can be broadly applied and do not have to be restricted to large companies with sophisticated software and big budgets. For example, modeling of microsomal metabolism has been used with large datasets,[22,29,35] and such models are now more accessible through availability of public data. The results summarized in Tables 1 and 2 suggest that reliable Bayesian models for various bioactivity and ADME/Tox models can be generated with simple fingerprint descriptors (FCFP6 and ECFP6) and the same Bayesian algorithm. This is enabled in such a way that experience in building computational models, while valuable, may not be essential to facilitate model generation, compound scoring, and interpretation.

The models described in Table 2 which are available in MMDS are now also freely accessible (http://molsync.com/bayesian1). Our main motivation for creating and disseminating this work is to enable the sharing of Bayesian models between a diverse set of toolkits and computing platforms. We have previously described our open source implementation of ECFP6 and FCFP6 fingerprints,[100] inspired by the original commercial implementation that was partially reported in the literature without the disclosure of key details, which remain a trade secret.[102,106] While there are several other examples of the general approach, our intent was to create a reference implementation and document it so that identical results could be reproduced. The algorithm herein is explicitly documented in a stepwise fashion, and the reference method is available publicly in source code, and hence can be used to compare against when re-implementing in another environment. We believe that taking such care to ensure that the algorithms can be implemented in a way that is 100% compatible with the formal reference removes a major barrier to scientific progress, since building and using models is no longer an isolated activity. We have deliberately taken a two-prong approach: by releasing a fully functional implementation as part of a popular open source toolkit, and also taking the effort to document the algorithm in fine grained detail, to encourage creators of commercial software to consider the advantages of interoperability within their own proprietary products.

Because the source code is a part of the CDK, the modeling functionality that we describe can be used in a variety of scenarios as-is. Any software environment that is capable of linking to a Java Virtual Machine (either directly or through a pipe) can make use of this functionality. Since the CDK is made available under the LGPL license, it can be incorporated into proprietary products as long as it is linked as a separate library, but for internal projects, back-end services for which the software is not distributed, or open source projects with a compatible license, it can be used essentially without restrictions. For wholly closed-source products, and platforms that are not compatible with the Java Virtual Machine, the methodology can be re-implemented without difficulty. The exact implementation of Bayesian model building and subsequent calibration is straightforward, and we have represented it in pseudocode form (see the accompanying paper for details of algorithms for additional analysis[101]). The CDK version is readily available to verify literal compatibility and can be used as a limitless source of validation data for direct comparison. Thus far, the method has been ported to Objective-C, in order to enable the use of Bayesian models within several different mobile apps (Figure 3) and CDD Vault as CDD Models. The use of CDD Models online in the CDD Vault data sharing platform to create Bayesian models, the use of mobile apps to apply them to small collections of proposed compounds, and integration into other products and scripts via the CDK library present a number of opportunities for making computational modeling potentially more useful and widespread. Currently structure–activity models are generally only able to be created and used by one specific platform, or if they have some portability, they often suffer from serious compatibility issues due to differences in the underlying technology (e.g., aromaticity models, ylide representations, SMARTS implementations, partial charge models, etc.) By releasing a well-documented reference implementation as open source and building powerful and useful functionality on top of it, we hope to encourage computational chemists and software creators to make use of this increased inter-operability.

Future work related to this project will include the implementation of further measures to assess model quality and the applicability[115,134−137] of a model to a test compound. In the accompanying paper,[101] we describe several additional algorithms, including calibration of raw Bayesian results to a probability-like scale, the effects of folding fingerprints into a smaller range, methods for extracting suitable validation test sets from large public datasets, automated determination of thresholds for active/inactive, and the impact of training set selection on internal cross-validation metrics. As others begin to use the new CDK functionality, CDD Models, and Bayesian functionality implemented in various mobile apps, we will expect to see further prospective and retrospective testing of the underlying technology and descriptions of the utility and limitations.