Phantom PAINS: Problems with the Utility of Alerts for Pan-Assay INterference CompoundS.

The use of substructural alerts to identify Pan-Assay INterference compoundS (PAINS) has become a common component of the triage process in biological screening campaigns. These alerts, however, were originally derived from a proprietary library tested in just six assays measuring protein-protein interaction (PPI) inhibition using the AlphaScreen detection technology only; moreover, 68% (328 out of the 480 alerts) were derived from four or fewer compounds. In an effort to assess the reliability of these alerts as indicators of pan-assay interference, we performed a large-scale analysis of the impact of PAINS alerts on compound promiscuity in bioassays using publicly available data in PubChem. We found that the majority (97%) of all compounds containing PAINS alerts were actually infrequent hitters in AlphaScreen assays measuring PPI inhibition. We also found that the presence of PAINS alerts, contrary to expectations, did not reflect any heightened assay activity trends across all assays in PubChem including AlphaScreen, luciferase, beta-lactamase, or fluorescence-based assays. In addition, 109 PAINS alerts were present in 3570 extensively assayed, but consistently inactive compounds called Dark Chemical Matter. Finally, we observed that 87 small molecule FDA-approved drugs contained PAINS alerts and profiled their bioassay activity. Based on this detailed analysis of PAINS alerts in nonproprietary compound libraries, we caution against the blind use of PAINS filters to detect and triage compounds with possible PAINS liabilities and recommend that such conclusions should be drawn only by conducting orthogonal experiments.

Introduction

The scientific community is in the grips of the data reproducibility crisis, highlighted by Nature’s “Challenges in Irreproducible Research” initiative.[1,2] Oftentimes in drug discovery, compounds active in primary biological screens show no activity in follow-up studies.[3−5] The measured effect of false positives may be due to various mechanisms including those that interfere with the assay detection technology such as autofluorescence, hydrogen peroxide production, metal chelation, chemical aggregation, etc.[2−6]

In a highly cited study,[6] Baell and Holloway analyzed compounds that showed activity in multiple assays and suggested that these compounds may interfere with the bioactivity detection technology. Such compounds were cleverly dubbed PAINS, or Pan-Assay INterference compoundS. In an effort to provide a tool to enhance reproducibility and reliability of true hit identification in drug discovery, the authors then identified 480 “PAINS alerts”, i.e., substructural features frequently found in PAINS, and suggested that these alerts could be used to flag false screening hits and annotate suspect compounds in screening libraries.[6]

Following the original publication,[6] which has garnered more than 800 citations according to Google Scholar at the time of this study, the concept of PAINS alerts (filters) has gained much attention, many supporters, and prompted many follow-up publications.[7−9] Several web-based applications relying on the original work by Baell and Holloway[6] have been developed to flag and filter compounds with PAINS alerts;[10,11] chemical databases, such as ZINC (http://zinc15.docking.org/) and ChEMBL (https://www.ebi.ac.uk/chembl/), also flag compounds containing PAINS alerts. On the scientific blogosphere, publications reporting compounds flagged with PAINS alerts as viable hits have been publicly ridiculed in a practice known as “PAINS-Shaming.”[12]

The wide acceptance of the PAINS concept by the scientific community and the availability of PAINS filters have made it common for researchers to triage virtual screening hits flagged with these alerts prior to experimental validation.[13] Similarly, lead compounds resulting from experimental screening campaigns have typically been deprioritized for follow-up studies if they contained PAINS alerts.[14] Furthermore, scientific journals have begun to recommend that all hit compounds, virtual or otherwise, should be passed through one of the publicly available PAINS filters before the manuscript is considered for publication. For instance, the Journal of Medicinal Chemistry requires that “active compounds from any source must be examined for known classes of assay interference compounds”.[15] The authors are asked to “provide firm experimental evidence in at least two different assays that reported compounds with potential PAINS liability are specifically active and their apparent activity is not an artifact”.[15] Thus, compounds with potential PAINS liability are those flagged with PAINS alerts.

Amidst the generally wide acceptance of PAINS, there have been a few voices cautioning about the overarching utility of the alerts. Several authors have noted that the application of these alerts could discard viable drug candidates because such alerts have actually been found in approved drugs.[16,17] More substantial criticism of PAINS alerts has emerged as well on Internet forums (but not in peer-reviewed publications). Aware of these concerns, in the course of our own recent virtual screening investigations, we re-examined the original study[6] from which the 480 PAINS alerts were derived. We noticed that the study[6] employed a relatively small (93 000 compounds) and proprietary library (complete chemical structures were not released) tested for one type of activity (protein–protein interaction inhibition) in just six HTS campaigns (three out of six targets were kept confidential) using a single detection technology (AlphaScreen).

Though considerable effort was made to divulge as much information as possible, due to the proprietary nature of the original study and unavailability of the chemical library explored therein, the detection of PAINS and the derivation alerts could not be fully and independently reproduced. That being said, upon further inspection of the 92 pages of Supporting Information,[6] we observed that more than half of the PAINS alerts were derived from one or two compounds only (Figure ), with 68% (328 out of the 480 alerts) found in four or fewer compounds only, and more than 30% (190 PAINS alerts) found in one compound only showing “pan-assay” activity (Figure ; Table S1). This preliminary analysis lead us to hypothesize that the majority of these alerts may have limited extrapolative power.

Figure 1

Probing the extrapolative power of PAINS alerts. A histogram showing the distribution of the number of PAINS alerts (amounting to 480 total) as the function of the number of compounds used to derive each alert. Note that 190 PAINS alerts were derived from one representative compound only whereas only 18 PAINS alerts were derived from samples including more than 100 compounds per alert.

Given the aforementioned limited sources of PAINS alerts, we decided to probe into the pan-assay activity of PAINS and the reliability of PAINS alerts by analyzing publicly available data on extensively assayed compounds. To this end, we have (i) assessed the robustness of PAINS alerts at flagging frequent hitters among compounds assayed using the AlphaScreen technology as reported in PubChem; (ii) scanned PubChem to investigate the level of pan-assay activity of compounds with and without PAINS alerts; (iii) examined the frequency of PAINS alerts in extensively assayed, yet consistently inactive compounds known as “Dark Chemical Matter”;[18] and (iv) profiled the PubChem-wide activity of FDA-approved drugs with and without PAINS alerts. Overall, using publicly available data, this study sought to evaluate the PAINS concept in general, with an additional focus on specific PAINS alerts established in the original investigation,[6] in order to provide both researchers and journal editors with insight into the utility of PAINS alerts as they currently stand.

Results and Discussion

Detection of PAINS in Chemical Libraries Tested with the AlphaScreen Technology

We have identified six PubChem assays that measure protein–protein interaction (PPI) inhibition using AlphaScreen, i.e., the same type of activity and the same technology employed in the original study.[6] The study design is shown in Figure . The six originally studied assays were run at the relatively high compound concentration of 25–50 μM in primary screens, which may account for the high rate of interference, and two of the assays used hexa-his/Ni anchors.[6] We have chosen these six PubChem assays, similar to a study by Schorpp et al.,[19] in order to assess the robustness of PAINS alerts to flag frequent hitters across a similar, but not identical, series of assays. It should be noted that the anchorage and screening concentrations reported in PubChem were different from original study. However, current PAINS filters look solely for the presence of specific functional groups in assayed chemicals regardless of the assay conditions; thus, the difference in these conditions does not invalidate the use of PAINS alerts in this investigation.

Figure 2

Study design for examining compounds in PubChem tested with AlphaScreen assay technology. Six assays targeting PPIs and using AlphaScreen were identified in PubChem. Only those compounds that were tested in all six assays were considered. Compounds were binned into two categories according to the number of active calls. Compounds in each category were then queried for PAINS alerts. The PubChem-wide bioassay activity of all compounds was then investigated.

As many as 153 339 unique compounds were found in PubChem to have been tested across all six assays (Table S2). Activity calls for each compound (Active, Inactive, and Inconclusive) were recorded as defined by the assay depositor. Compounds were then binned into two categories: “Frequent Hitters” (active calls in at least two out of six assays) and “Infrequent Hitters” (active calls in one or zero assays), which is the same threshold as established in the original study.[6] Both categories were first queried for the presence of PAINS substructural alerts using the SMARTS implementation from PubChem Promiscuity,[20] then confirmed using SYBYL Line Notation (SLN) implementation from FAF-Drugs3.[10] Four categories arose: “Frequent Hitters-PAINS” (FH-PAINS), “Frequent Hitters-No PAINS” (FH-NoPAINS), “Infrequent Hitters-PAINS” (IH-PAINS), and “Infrequent Hitters-No PAINS” (IH-NoPAINS). There was a concordance of ∼99.9% between the SMARTS and SLN implementations for flagging compounds with PAINS alerts (Table S3). The enrichment value (EV), which was previously defined[6] as the percentage of compounds active in at least two of the six assays relative to the number of compounds that displayed no activity across all six assays, was calculated to compare FH-PAINS vs IH-PAINS (Table S4 and S5).

The results of our analysis are shown in Table . There were 902 compounds in the FH category, and only 208 (23%) of these contained PAINS substructural alerts (FH-PAINS). The remaining 694 FH lacked any PAINS alerts (FH-NoPAINS). For the IH, 146 224 (96%) compounds lacked PAINS alerts (IH-NoPAINS), but 6413 compounds (4%) still contained the alerts (IH-PAINS). Comparing the numbers of IH-PAINS and FH-PAINS leads to the apparent conclusion that, for this series of assays, the majority of compounds containing PAINS alerts (97%) were actually infrequent hitters.

Table 1

Lack of Pan-Assay Activity for Compounds with PAINS Alerts in PubChema

compound categories	Ncompounds	luciferase	β-lactamase	fluorescence	all assays
FH-PAINSb	208	12% (93)	4% (9)	7% (312)	10% (546)
FH-NoPAINSb	694	6% (95)	2% (9)	3% (320)	5% (550)
IH-PAINSb	6413	3% (93)	1% (10)	2% (323)	2% (550)
IH-NoPAINSb	21500	1.5% (95)	0.5% (9)	1% (326)	1% (555)
Random-PAINSc	14611	3% (95)	1% (12)	2% (329)	3% (562)
Random-NoPAINSc	58722	2% (93)	0.6% (13)	0.8% (321)	1% (550)
Drugs-PAINSc	87	9% (71)	7% (40)	6% (223)	24% (602)
Drug-NoPAINSc	1373	5% (59)	5% (33)	3% (183)	15% (458)

The average fraction of activity calls for PAINS and non-PAINS (defined as containing or lacking PAINS alerts, respectively) across both detection technology-specific assays and all assays in PubChem. The average number of assays in which the compounds were tested are shown in parentheses.

Defined by the compound profile in PPI assays utilizing AlphaScreen.

Defined by presence or absence of PAINS alerts.

The enrichment value calculated for PAINS-containing compounds (FH-PAINS and IH-PAINS) was only 3.5% (Table S4 and S5). Furthermore, if IH-PAINS that were active in one assay only were taken into consideration, the overall EV fell to 3.2% (Tables S4 and S5). The analysis of this series of assays indicates that PAINS alerts are found much more frequently in nonpromiscuous compounds.

To probe whether or not this observation is only limited to assays related to PPIs using AlphaScreen, we investigated the PubChem-wide bioassay activity of the same compounds. Given that IH-NoPAINS constituted the overwhelming majority of all compounds discussed above (146 224 compounds), the PubChem-wide activity of all IH-NoPAINS was not evaluated due to computational constraints. Instead, a random subset of 21 500 IH-NoPAINS (Table S7) were evaluated; this number was selected to preserve approximately the same ratio of frequent to infrequent NoPAINS (1:3.5) as was observed for the PAINS (cf. Table ). PubChem Promiscuity[20,21] was used to retrieve the activity calls for all four aforementioned categories of compounds (FH-PAINS, IH-PAINS, FH-NoPAINS, and IH-NoPAINS) tested in luciferase-, β-lactamase-, and fluorescence-based assays (see Tables S4–S7). Lastly, we assessed activity calls across all bioassays in PubChem irrespective of the detection technology (Table ).

We found that across all assays, including those that have been reported as particularly susceptible to interference,[5] FH-PAINS were active in more assays than FH-NoPAINS; however, IH-PAINS were active in fewer assays than FH-PAINS (Table ). The reduced activity of IH-PAINS in the AlphaScreen assays, therefore, is not limited to this detection technology, as it can be observed over all reported assays in PubChem. Also, both categories of frequent hitters in AlphaScreen (FH-PAINS and FH-NoPAINS) showed greater PubChem-wide activity than infrequent hitters containing PAINS alerts (IH-PAINS). Therefore, the broader activity spectrum of these frequent hitters is independent of the presence or absence of any PAINS alerts, highlighting the importance of considering molecular entities as a whole rather than chemical fragments when trying to derive any structural rules governing assay promiscuity.

Analysis of PAINS Alerts in Chemical Libraries Tested with the AlphaScreen Technology

The specific alerts found in the above FH-PAINS and IH-PAINS categories were then investigated on an individual basis. In total, 163 individual PAINS alert types were observed in compounds among the two categories (Table S8). It should be noted that multiple PAINS alerts could be present within a single compound (Figure ).

Figure 3

Compounds with multiple PAINS alerts. Two representative compounds from the IH-PAINS and FH-PAINS categories that contain multiple PAINS alerts.

For the 208 FH-PAINS compounds, 41 individual PAINS alerts were detected (Table S8). Of these 41 alerts, only 7 alerts, i.e., quinone_A(370), mannich_A(296), ene_six_het_A(483), anil_di_alk_B(251), anil_di_alk_A(478), ene_one_hal(17), and imine_one_A(321), were found in more than 10 FH-PAINS compounds. The remaining 34 PAINS alerts were present in 10 or less FH-PAINS compounds.

For the 6314 IH-PAINS compounds, 162 individual PAINS alerts were detected (Table S8). Of these 162 alerts, 57 alerts were found in more than 10 IH-PAINS compounds. Moreover, 15 of these alerts were found in more than 100 IH-PAINS compounds. The anil_di_alk_A(478) alert, for example, appeared in 1083 IH-PAINS compounds.

Next, the PAINS alerts that were present in both the FH-PAINS and IH-PAINS were analyzed. Within these two categories, 40 individual PAINS alerts were shared, roughly ∼25% of all observed alerts. Only one alert, i.e., anil_no_alk_A(1), was unique to FH-PAINS (Table S8); however, only 1 compound possessed this alert, which is consistent with the limited sample size (1 compound) used to derive this alert (cf. Figure ). Similarly, 122 alerts were unique to IH-PAINS (Table S8). For this series of assays, ∼75% PAINS alerts present in the PubChem library analyzed herein (122 out of 163) were found only in IH.

The enrichment value (EV), defined as percentage of compounds active in at least two of the six assays relative to the number of compounds active in 1 or 0 assays, was calculated for each of 40 shared PAINS alerts (Table ). Only 6 alerts showed EVs greater or equal to 25%. However, 4 of these 6 alerts had less than 10 representative compounds. Therefore, 2 alerts, i.e., quinone_A(370) and quinone_D(2), were found in 10 or more compounds and had an EV greater or equal to 25%. The remaining 34 shared alerts had EVs less than 25%, and 32 of these 34 alerts had more than 10 representative compounds. Indeed, 6 shared alerts had EVs less than 1.0% despite being present in more than 100 compounds. For this series of assays, the vast majority of PAINS alerts were found among the IH-PAINS at much higher frequencies. The full analysis of all 40 shared alerts, including representative compound sizes and EVs, can be found in Table .

Table 2

PAINS Enrichment in Six PubChem Assays Employing AlphaScreena

Forty alerts were present in both FH-PAINS and IH-PAINS. Two alerts showed EVs greater or equal to 25% and were found in 10 or more total compounds. Six alerts had EVs below 1.0%.

Random PAINS in PubChem

We also evaluated the PubChem-wide activity of compounds tested in at least 25 separate bioassays based only on the presence or absence of 480 originally established PAINS alerts, i.e., irrespective of any perceived promiscuity across a selected series of specific assays. Randomly selected compounds that were evaluated in the previous section were excluded. The resultant data set contained 73 333 individual compounds. The structures of these compounds were searched for PAINS alerts (described above) and binned into two categories: Random-PAINS (14 611 compounds) and Random-NoPAINS (58 722 compounds). We compared these two categories following the same protocol as described in the previous section (Table ). The average pan-assay activity of Random-PAINS was just 3%, compared to an average of 1% for Random-NoPAINS (Table ), i.e., Random-PAINS were marginally more active than Random-NoPAINS. Additionally, of the 14 611 Random-PAINS only 752 compounds (5% of the total) showed activity in at least 10% of all assays. Of the remaining 13 859 Random-PAINS (95%) that were active in less than 10% of all assays, 1146 had no activity at all, despite being tested in an average of 443 assays (Tables S9 and S10). These results indicate that the mere presence of a PAINS substructure does not give rise to any observed pan-assay activity, nor any marked interference trends in luciferase-, β-lactamase-, or fluorescence-based assays. In fact, only two PAINS alert containing compounds, tanespimycin and dihydrexidine, were active in more than 50% of the assays (Figure ). In total 202 PAINS alerts were found among the Random-PAINS category. A PubChem-wide analysis of alerts in random PAINS is described in the Global Analysis of PAINS Alerts section.

Figure 4

Random-PAINS displaying pan-assay activity. Tanespimycin and dihydrexidine are active in 85% and 50% of all assays in PubChem, respectively.

Analysis of PAINS Alerts in Dark Chemical Matter

Following the observation that Random-PAINS can be consistently inactive across a large number of assays, we probed the so-called Dark Chemical Matter (DCM)[18] for the presence of PAINS alerts. DCM was defined by Wassermann et al.[18] as compounds that have not yet shown any activity when tested in a minimum of 100 assays. The complete data set of 139 352 DCM compounds, i.e. 128 997 PubChem and 10 355 Novartis DCM compounds, was downloaded from the Supporting Information of the respective study.[18]

The data set was examined with FAF3-Drugs,[10] and 3570 DCM compounds containing PAINS substructures were found, encompassing 109 of the 480 original PAINS alerts[6] (Table S11). Of these 109 PAINS alerts, 30 alerts were found in more than 10 compounds and 10 alerts were present in 100 or more compounds (Table ). This analysis shows that even extensively assayed compounds containing PAINS alerts may be consistently inactive.

Table 3

PAINS alerts enriched in Dark Chemical Mattera

Ten alerts are present in 100 or more in Dark Chemical Matter compounds (NDCM).

Global Analysis of PAINS Alerts

In this study, a total of 24 802 PAINS compounds were analyzed (208 FH-PAINS, 6413 IH-PAINS, 14 611 Random-PAINS, and 3570 DCM-PAINS) covering 220 specific PAINS alert types, which is ∼46% of the original PAINS alerts.[6] In order to determine if specific PAINS alerts correspond to compounds with elevated assay promiscuity, we performed a global analysis of the PubChem-wide activity of all PAINS compounds investigated herein (Tables S12–S15). Since there is no agreed upon threshold of pan-assay activity, we selected assay activity of at least 10% as an arbitrary classifier. The global assay activity associated with all alerts can be found in the SI.

Of these 220 PAINS alert types, 32 alerts had greater than 10% assay activity in either all assays, luciferase-, β-lactamase-, or fluorescence-based assays (Table S12). However, only 12 of these alerts were present in more than 10 compounds (Table ). It should be noted, however, that six of these alerts can also be found in DCM.

Table 4

PAINS Alerts Displaying Elevated Assay Activity in PubChema

Twelve alerts have more than 10 compounds (NPubChem) and greater than 10% assay activity in either all assays, luciferase-, β-lactamase-, or fluorescence-based assays (bold). Six alerts were also present in DCM (NDCM). The average numbers of assays in which the compounds were tested are shown in parentheses.

On the other hand, 176 (∼80%) of the total PAINS alerts analyzed (220) were active in less than 10% of all investigated assays and technologies and 88 alerts were present in DCM (Tables S13 and S14). Eighty-four of 176 alerts were present in more than 10 compounds (Table S13). Interestingly, 6 of these alerts were found in more than 1000 compounds (Table ). Finally, 12 alerts were found exclusively in DCM-PAINS (Table S15). Eleven of these alerts were found in less than 10 compounds, while 1 alert, i.e., hzone_phenol_B(215), was present in exactly 10 compounds.

Table 5

PAINS Alerts Not Displaying Elevated Assay Activity in PubChema

These 6 alerts are present in more than 1000 compounds (NPubChem) and have less than 10% assay activity in either all assays, luciferase-, β-lactamase-, or fluorescence-based assays. All six alerts were also present in DCM (NDCM). The average numbers of assays in which the compounds were tested are shown in parentheses.

There are 16 PAINS alerts that were derived from more than 150 compounds (cf. Figure ). Of all 480 alerts, these 16 alerts were created from the most underlying data in the original study.[6] Given the prevalence and heightened promiscuity of compounds possessing these alerts in the original study, we specifically investigated whether any compounds in our collection flagged by these 16 alerts display suspect assay trends (Table ). Aside from hzone_phenol_A(479) and hzone_phenol_B(215), which were found exclusively in DCM, 14 of these 16 alerts were frequently assayed and abundantly present in the public collection. All 14 alerts displayed less than 10% activity in all PubChem assays despite being tested on average in more than 500 assays each. Among these specific alerts, the quinone_A(370) alert demonstrated the highest activity in all assays (8.4%).

Table 6

Global Assay Activity of Compounds in PubChem Possessing the Top 16 PAINS Alertsa

The bioassay activities of 14 out of 16 alerts have been profiled with more compounds than were used to derive the alerts originally.[6] Two alerts were found only in DCM (NDCM). The average numbers of assays in which the compounds were tested are shown in parentheses.

Our findings using data in the public domain can be corroborated in part by other inquiries into the nature of promiscuous compounds. For instance, while attempting to use PAINS alerts to fill gaps in Eli Lilly’s promiscuity filters, Bruns and Watson observed that “PAINS queries matched 286 promiscuous compounds that passed the Lilly rules, compared to 3986 in the non-promiscuous set, for an enrichment factor of 4.0”.[22] Furthermore, they noted that “although 67 PAINS queries matched at least one promiscuous compound, only nine queries matched at least five promiscuous compounds and had an enrichment of at least 5.”[22] These findings are consistent with our observations that PAINS alerts in public data frequently flag nonpromiscuous compounds or are manifested in only a small number of promiscuous compounds.

On the other hand, another study on frequent hitter behavior by researchers at AstraZeneca showed elevated “Frequent-hitter Incidence %” for 10 out of 15 PAINS alerts.[23] Although the authors state that their “corporate data largely confirm previous observations of the PAINS classes”, this study only investigated part of the first tier of the 480 PAINS alerts, i.e., the 15 out of 16 alerts derived from more than 150 compounds (cf. Figure ), or ∼3% of all alerts.[23] As can be seen, in that study, one-third of the profiled alerts did not show elevated frequent-hitter behavior, which is, in part, aligned with our general observations (Tables and 6).

PAINS Alerts in Drugs

Other groups have noted that many drugs contain PAINS alerts,[6,16,17] and several careful and keen analyses have centered around this phenomenon.[24] It has also been observed that many of these PAINS alerts in drugs (but not all) map to poor ADMETox properties, such as quinone-containing drugs.[6] While this is an interesting observation, we view interference propensity and poor ADMETox properties as separate phenomena. Our group[25] as well as others[19] have also shown that a great majority of toxicity structural alerts, much akin to PAINS alerts, are overly sensitive and not predictive of actual in vitro or in vivo toxicity. Given that drug repurposing is currently widely used as a boon to traditional drug discovery,[26,27] we profiled the PubChem-wide bioassay activity of drugs with and without PAINS alerts (Table ).

A list of 1460 approved small-molecule drugs was compiled from Drugs@FDA (https://www.accessdata.fda.gov/scripts/cder/drugsatfda/). Structures for these drugs were searched for PAINS alerts.[10,20,21] We identified 87 small-molecule approved drugs possessing 25 individual PAINS alerts (Table S16). As observed in the preceding sections, Drugs-PAINS are more active than Drugs-NoPAINS, having activity in 24% and 15% of all bioassays in PubChem, respectively (Table ). According to current filters,[10,20] 16 of these drugs possess quinone PAINS alerts. The promiscuity of quinone-containing drugs have been extensively discussed in the PAINS literature[6,24] and is supported by our analysis (cf. Tables and 4). For instance, the chemotherapeutic doxorubicin, which contains the quinone_A(370) alert, has been tested in more than 4000 assays with active calls ∼85% of the time.

At the same time, however, the relationship between polypharmacology and PAINS has not yet been adequately explored. Many drugs show polypharmacological behavior and possibly derive their efficacy from interacting with multiple targets.[28] Indeed, a similar study on promiscuity in extensively assayed compounds found that drugs are more promiscuous than bioactive compounds,[28] which is evidenced in our analysis as well (Table ). Polypharmacology may well account for the increased activity of both Drugs-PAINS and Drugs-NoPAINS relative to the other categories (cf. Table ). While the phenomena of assay interference and polypharmacology have rightfully been contrasted,[28] there is very real possibility that a compound may both possess PAINS alerts and display polypharmacological behavior. Given that PAINS-containing drugs are now frequently used in drug repurposing screens, a larger discussion about the utility of PAINS alerts and polypharmacology should take place.

Beyond PAINS Substructures

PAINS concept has been widely accepted by many experienced medicinal chemists both in academia and the pharmaceutical industry. Indeed, the original study from which the PAINS alerts were derived and the impetus behind it are an important step toward reproducibility and the appropriate use of resources in drug discovery. However, our findings based on the analysis of public data suggest that many compounds containing PAINS alerts do not actually show high assay promiscuity, leading to the conclusion that these alerts should not be blindly used, in the absence of orthogonal experimental assays, to deprioritize a compound.

At the same time, it is undeniable that pan-assay interference compounds exist and care must be taken to avoid these compounds. Moreover, we recognize that true “PAINS” may be present in the data analyzed herein but have not been classified as such because the current alerts do not cover these compounds.[29] The issue of what constitutes a pan-assay interference compound thus remains unclear. For example, in How to Triage PAINS-Full Research, Dahlin and Walters define PAINS as “compounds that are recognized by the substructure filters reported by the Baell and Holloway article.”[29] By this definition, all alerts (filters) are treated equally, regardless of the underlying data used to derive the alert or the actual promiscuity of flagged compounds. Yet our analysis indicates that the identification of such compounds should not be restricted to substructures alone. Substructural alerts, PAINS or otherwise, do not take into consideration the whole molecular environment,[25] as illustrated by PAINS alerts manifesting in both promiscuous and frequently inactive compounds (DCM). Attempts should then be made to move beyond substructural or fragment-based alerts. For instance, Yang and co-workers in their “BadApple” algorithm have extended the identification of promiscuous compound to larger scaffolds.[30]

In recent publications by Alves et al.,[31−33] quantitative structure–activity relationship (QSAR) models were used in conjunction with structural alerts for toxicity to dramatically improve the accuracy of prediction of multiple toxicity end points over alerts alone. The authors of the present study advocate the development of a similar approach for PAINS alerts. Such publicly accessible models, if successful, could be employed even for proprietary compounds insofar as chemical descriptors of PAINS alert-containing compounds could be shared without divulging actual molecular structures (given the proprietary nature of most compounds used to derive and evaluate PAINS so far[6,22,23]). The challenge is to build externally predictive QSAR models capable of classifying PAINS versus non-PAINS compounds. Using such models, predictions of suspect compounds could be made, giving higher confidence in the utility of the alert and the nefarious nature of the compound.

Meanwhile, the concept of PAINS alerts, at the very least, needs a redefined set of “best practices”[29] that covers the appropriate use of alerts, which may include cross-referencing the promiscuity profiles of structurally similar compounds or alert types in the public domain, annotation of particularly susceptible assays, targets, and conditions, pointers to the appropriate controls, and a generally agreed upon definition of pan-assay activity. It would be of great value if a community-wide effort to screen and analyze a large set of commercially available compounds representing all current PAINS alerts against multiple targets in various assays was performed by several independent groups.

Conclusions

It is imperative to establish target selectivity for any compound considered a viable chemical probe or drug candidate through rigorously acquired experimental data and meaningful SAR. Future studies may well establish some generalized approach for detecting frequent hitters engendered by assay interference. However, until such approaches are developed and rigorously validated across a large number of molecules, researchers should be cautioned about using the current PAINS alerts as reliable indicators of nonspecific pan assay interference. Though it has been stated elsewhere that compounds flagged with PAINS alerts are not active in all assays or against all targets,[6,29] our analysis provides systematic and data-driven support of this claim across a large series of compounds, alerts, and assays. Our findings do demonstrate, with publicly available data at hand, that majority of the original PAINS alerts are not indicative of pan-assay compound promiscuity, that many compounds without PAINS alerts are as, if not more, promiscuous as those with the alerts, and that many compounds flagged by PAINS alerts show no activity. It is of great importance that reviewers and journal editors request experimental proofs of selectivity, such as orthogonal experimental assays, for hit and lead compounds reported in scientific manuscripts. However, the results of this study strongly suggest that such requests should not be based solely on the results of PAINS filters.[29]