Stimulated reporting: the impact of US food and drug administration-issued alerts on the adverse event reporting system (FAERS).

BACKGROUND: The US Food and Drug Administration (FDA) uses the Adverse Event Reporting System (FAERS) to support post-marketing safety surveillance programs. Currently, almost one million case reports are submitted to FAERS each year, making it a vast repository of drug safety information. Sometimes cited as a limitation of FAERS, however, is the assumption that "stimulated reporting" of adverse events (AEs) occurs in response to warnings, alerts, and label changes that are issued by the FDA.
OBJECTIVE: To determine the extent of "stimulated reporting" in the modern-day FAERS database.
METHODS: One hundred drugs approved by the FDA between 2001 and 2010 were included in this analysis. FDA alerts were obtained by a comprehensive search of the FDA's MedWatch and main websites. Publicly available FAERS data were used to assess the "primary suspect" AE reporting pattern for up to four quarters before, and after, the issuance of an FDA alert.
RESULTS: A few drugs did demonstrate "stimulated reporting" trends. A majority of the drugs, however, showed little evidence for significant reporting changes associated with the issuance of alerts. When we compared the percentage changes in reporting after an FDA alert with those after a sham "control alert", the overall reporting trends appeared to be quite similar. Of 100 drugs analyzed for short-term reporting trends, 21 real alerts and 25 sham alerts demonstrated an increase (greater than or equal to 1 %) in reporting. The long-term analysis of 91 drugs showed that 24 real alerts and 28 sham alerts demonstrated a greater than or equal to 1 % increase.
CONCLUSIONS: Our results suggest that most of modern day FAERS reporting is not significantly affected by the issuance of FDA alerts.

Key Points

Introduction

Because of financial and logistical hurdles, pre-approval clinical trials can never be large enough, or long enough, to identify and properly characterize all adverse effects that may occur once a drug is introduced to consumer populations. The gradual evolution of adverse event (AE) profiles across numerous drugs (e.g., thalidomide, sibutramine, cerivastatin, rofecoxib) [1-4], after they were approved by the US Food and Drug Administration (FDA), serves to underscore the preceding points.

Unfortunately, the time lag associated with the dissemination of relevant post-marketing AE information is also a significant concern. In fact, within 7 years after FDA approval, only half of a drug’s serious post-marketing AEs was listed in the Physician’s Desk Reference, a main source of AE information for many prescribers [5]. Such delays, combined with the aforementioned limitations of the pre-approval clinical trial process, reinforce the need for diligent post-marketing vigilance [5-8].

The FDA’s Adverse Event Reporting System (FAERS) [9] is a centralized computerized information database that is broadly used by the agency, and other pharmacovigilance experts, for post-marketing drug safety surveillance [6, 8, 10]. International government and related organizations also use spontaneous AE databases to identify post-approval drug safety concerns.

Challenges to using FAERS data, however, have been reported to include under-reporting [11, 12], the “Weber Effect” [13, 14], and “stimulated reporting” [15-19]. With regard to under-reporting, while it remains a significant limitation, recent efforts by the FDA, global regulatory agencies, and the healthcare industry itself are helping to increase AE reporting rates. Indeed, almost one million AE reports will be added to both the EudraVigilance [20] and FAERS databases this year alone [21]. FAERS now has a total of over seven million reports. With regard to the “Weber effect”, a recent FAERS study [22] demonstrated that it may be of less concern than it was in the past, likely owing to an increasing focus on the importance and utility of post-approval AE reporting by both regulatory and key healthcare players [22, 23]. Unfortunately, organized FAERS data are not readily accessible to healthcare professionals who instead rely heavily on safety information from drug label “inserts” that are often based predominantly on pre-approval clinical trial results and are frequently inadequate with regard to AE details [24].

To help guide prescribing decisions and arm healthcare professionals with emerging post-marketing safety risks, the FDA issues warnings [7, 18, 25], which have often effectively modified prescribing behaviors [17, 19]. However, the alerts are sometimes challenged by the scientific community [17, 26–29], numerous alerts can be required to trigger an effect [30], and too often they have little to no impact on prescribing patterns [17, 28, 31–33]. Occasionally, regulatory action by the FDA can also have the unintended consequence of depriving patients of efficacious and safe (when actually used as directed) medications [34-39].

“Stimulated reporting” refers to the concept that public disclosure of a safety issue by the issuance of, for example, an FDA alert will result in substantially increased AE reporting rates regarding the drug and/or the specific AE mentioned in such an alert. The term can also refer to clustering of AE reports triggered by the activities of consumer-based “support groups” and/or reporting activity related to litigation. Stimulated reporting has important implications regarding the utility of post-marketing AE data, as alert-driven shifts in reporting could impact the accuracy of comparative research and related analytical methods such as disproportionality analysis.

Although a number of studies have evaluated the impact of FDA alerts on drug use [16–19, 31, 33, 40–57], few have comprehensively studied the impact of FDA alerts on AE reporting. Past studies that have found evidence for stimulated reporting with regard to post-marketing AE data have generally examined solitary, or a single group of, drugs.

A 1998–2004 study showed that the reporting frequency of key AEs associated with statin use increased as a result of FDA warnings, but a general increase in broad AE reporting for drugs in the class was not observed [58]. A study of Italian AE databases indicated that three out of four regulatory warnings triggered increased AE reporting [38]. FDA-issued boxed warnings were linked to increases in FAERS reporting for multiple myeloma drugs [59]. Two French studies found evidence for stimulated reporting via changes in disproportionality [60, 61]. A study from The Netherlands found a media influence regarding cardiac arrhythmias linked to antihistamines [62]. An Australian study found an increase in reporting odds ratios after media publicity regarding links between a sleep medication and parasomnias and amnesia [63]. An FDA alert influenced FAERS reporting of pancreatitis linked to two diabetic medications [64]. FDA researchers detailed a large increase in rhabdomyolysis reports linked to statin drugs after an alert was issued on cerivastatin [65].

While select FDA alerts appeared to have impacted AE reporting rates, we have not found a broad study regarding FAERS reporting trends after alert(s) are issued. Therefore, we used a big data analytic platform [66] to quantify FAERS reporting trends over a 10-year period regarding 100 drugs that had been issued an FDA alert.

Methods

FAERS Data Preparation

Case reports that were missing or contained malformed key identification fields [Individual Safety Report number (ISR), patient number, drug sequence identification, or Medical Dictionary for Regulatory Activities (MedDRA®) AE term] were discarded. As long as the aforementioned key identification fields were contained in a given case report, allowable missing fields included: age, gender, weight, outcome, and condition. Cases were discarded if the drug name was found to be indeterminate or if the name was determined to not represent an FDA-approved drug (e.g. dietary supplements, foods). In an effort to exclude pre-approval AE case reports mistakenly logged into the FAERS, the date of receipt for a given case report must have occurred after the drug’s FDA approval date.

In instances where there were multiple, individual case safety reports for the same identification number, we selected the earliest reported case. For example, if a patient reported adverse effects from the same drug in 2008 and 2010 and we ran a search against all primary suspect cases for that drug across all dates, only the 2008 report would be included. However, if we were to run a search against all primary suspect cases for 2010 only, then the 2010 case would be included. Because the case counts included in our analysis were restricted by quarter, drug, and, when applicable, AE, we are confident that we obtained a complete profile of the primary suspect cases of interest.

Drug name text-mapping was accomplished as previously described by Hoffman et al. [66]. Drug names were normalized to RxNorm reference codes [67] using string searching and manual curation. National Drug File Reference Terminology [68] was used to provide ancillary information on class and mechanism of action.

AE information was coded according to MedDRA® version 16.1 [69]. AEs mentioned in each safety alert (see Electronic Supplemental Material) were mapped to MedDRA terms according to the following hierarchy: (1) AEs linked to specific medical conditions were coded verbatim to MedDRA Preferred Terms (i.e., “retinal detachment” was mapped to the MedDRA Preferred Term AE “retinal detachment”); (2) variations of specific AEs were also coded to MedDRA Preferred Terms (i.e., “heart attack” was mapped to the Preferred Term “myocardial infarction”); (3) AEs highlighting disorders pertaining to specific organs or parts of the body were coded to MedDRA System Organ Classes (i.e., “Central Nervous System Disorders” was mapped to the System Organ Class “Nervous system disorders”); and (4) generalized alert terms that encompassed multiple specific AEs were coded to MedDRA Standardized MedDRA® Queries (SMQs) (i.e., “hypertension” was mapped to all Preferred Terms that fall under the SMQ “Hypertension”).

“Primary suspect” designations in FAERS case reports were quantified in an attempt to restrict the analysis to those drugs directly suspected of causing the AE. (“Primary suspect” is a description chosen by the person who submitted a given case report and is their estimate of which drug, if the subject was taking more than one, was likely responsible for the observed AE).

Finally, drugs were mapped to their corresponding Anatomical Therapeutic Chemical (ATC) codes [70] in an attempt to determine if AE reporting patterns differed across therapeutic class (see Electronic Supplemental Material).

FDA Alert Data Collection

We searched for FDA alerts, and their associated issuance dates, via the FDA’s MedWatch [71] and main [72] websites.

A data mining platform specifically built to optimize FAERS data [66] was used to quantify “primary suspect” case counts up to four calendar quarters before, and four after, publicized FDA regulatory actions (Dear Healthcare Professional Letter, Drug Safety Communication, Early Communication, FDA Information for Healthcare Professionals, FDA Recall, Public Health Advisory, REMS Issued, and Safety Labeling Changes).

Statistical Analysis

For our primary analysis, the number of quarterly AE cases was collected for each drug for 2000–2012. Percentage differences were obtained for quarter two and quarter four prior, vs. quarter two and quarter four after, an FDA or a sham alert and displayed as a histogram. Percentiles were binned into 25 or 50 U increments starting with a minimum bin of “−99 to −75” and ending at a maximum “1,200–1,249” bin.

Prescription drugs approved from 2000 to 2010 were eligible for analysis. We assessed changes in short-term reporting by comparing the amount of total reports in the two quarters prior to the quarter the FDA alert was issued in with the two quarters following such an alert. The same method was used to assess longer-term reporting trends by comparing four quarters before with four quarters after the same FDA alert. Our inclusion criteria were: (1) two or four full quarters between the FDA-approval date and alert date for all two quarter and four quarter calculations, respectively, (2) a minimum of four quarters was needed between the alert date and quarter four 2012, (3) the drug needed to have a minimum of 40 primary suspect cases in the both the alert quarter and the quarter directly prior to the alert for the “all primary suspect” case report analysis, and (4) the drug needed to have a minimum of five AE/drug-specific cases in the both the alert quarter and the quarter directly prior to the alert for the “AE-specific” case report analysis. Over-the-counter and “street” drugs, vaccines, and broad, undefined compounds listed in FAERS were not analyzed.

For internal control analysis, we used each drug (“all event analysis”) as well as the specific AE/drug pairs as their own controls by assigning a “sham alert” five quarters before the actual FDA alert. We then evaluated the percentage changes in reports (for both two quarter and four quarter differences) for the sham alert in the same manner that we analyzed drugs with actual FDA alerts. We used the Mann–Whitney test to determine if there were statistical differences between reporting rates for real compared with sham alerts.

Results

Figure 1 shows both the short- and long-term percent changes in primary case reporting for drugs that were subjected to an FDA alert during the time period studied. One hundred drugs were analyzed for short-term (two quarters before and after alerts) effects, while 91 of those 100 drugs were analyzed for long-term (four quarters before and after alerts) effects.

Fig. 1

Short- and long-term percent changes in primary case reporting for drugs that were subjected to a US Food and Drug Administration alert during the time period studied. 2Q two quarters, 4Q four quarters

Figure 2 shows a short-term (two quarters prior vs. two quarters after) comparison of reporting changes for 43 out of 100 drugs that could serve as their own internal controls (by having a sham alert set at five quarters before their actual FDA alert). 21 real alerts and 25 sham alerts demonstrated an increase (greater than or equal to 1 %) in short-term reporting. If one were to define “stimulated reporting” as a 50 % or higher increase in reporting, then 7 out of 43 actual FDA alerts and 7 out of 43 of sham alerts would qualify.

Fig. 2

Short-term percent changes in primary case reporting for real alerts compared with sham alerts. 2Q two quarters

For Fig. 2, we ran a rank sum (Mann–Whitney) test comparing the two quarters prior vs. two quarters post results for actual alerts compared with sham alerts. The median percent change was not significantly different for actual alerts (1 %) vs. sham alerts (9 %) with U = 957 (p = 0.78).

Figure 3 shows a long-term (four quarters prior vs. four quarters after) comparison of reporting changes for 43 out of 100 drugs that could serve as their own internal controls (by having a sham alert set at five quarters before their actual FDA alert). 24 real alerts and 28 sham alerts demonstrated an increase (greater than or equal to 1 %) in long-term reporting. If one were to define “stimulated reporting” as a 50 % or higher increase in reporting then 12 out of 43 actual FDA alerts and 12 out of 43 sham alerts would qualify.

Fig. 3

Long-term percent changes in primary case reporting for real alerts compared with sham alerts. 4Q four quarters

For Fig. 3, we ran a rank sum (Mann–Whitney) test comparing the four quarters prior vs. four quarters post results for actual alerts compared with sham alerts. The median percent change was not significantly different for actual alerts (5 %) vs. sham alerts (30 %) with U = 1,022 (p = 0.40).

Figure 4 shows both the short- and long-term percent changes in reporting of specific AE/drug pairs noted in FDA alerts. 134 AE/drug pairs were analyzed for short-term (two quarters before and after alerts) effects, while 124 of those 134 AE/drug pairs were analyzed for long-term (four quarters before and after alerts) effects.

Fig. 4

Short- and long-term percent changes in specific adverse event/drug pair case reporting for drugs that were subjected to an FDA alert during the time period studied. 2Q two quarters, 4Q four quarters

Figure 5 shows a short-term (two quarters prior vs. two quarters after) comparison of reporting changes for 61 out of 134 AE/drug pairs that could serve as their own internal controls (by having a sham alert set at five quarters before their actual FDA alert). 26 real alerts and 25 sham alerts demonstrated an increase (greater than or equal to 1 %) in AE-specific reporting. If one were to define “stimulated reporting” as a 50 % or higher increase in reporting, then 10 out 61 actual FDA alerts and 14 out of 61 sham alerts would qualify.

Fig. 5

Short-term percent changes in specific adverse event/drug pair case reporting for real alerts compared with sham alerts. 2Q two quarters

For Fig. 5, we ran a rank sum (Mann–Whitney) test comparing the two quarters prior vs. two quarters post results for actual alerts compared with sham alerts. The median percent change was not significantly different for actual alerts (−3 %) vs. sham alerts (0 %) with U = 1,968 (p = 0.47).

Figure 6 shows a long-term (four quarters prior vs. four quarters after) comparison of reporting changes for 61 out of 134 AE/drug pairs that could serve as their own internal controls (by having a sham alert set at five quarters before their actual FDA alert). 30 real alerts and 49 sham alerts demonstrated an increase (greater than or equal to 1%) in AE-specific reporting. If one were to define “stimulated reporting” as a 50% or higher increase in reporting, then 17 out of actual FDA alerts and 33 out of 61 sham alerts would qualify.

Fig. 6

Long-term percent changes in specific adverse event/drug pair case reporting for real alerts compared with sham alerts. 4Q four quarters

For Fig. 6, we ran a rank sum (Mann–Whitney) test comparing the four quarters prior vs. four quarters post results for actual alerts compared with sham alerts. The median percent change was significantly lower for actual alerts (−4.8 %) than for sham alerts (59.3 %) with U = 2,603 (p = 0.000145).

Discussion

With regard to AE reporting, while we did find evidence for clear cases of “stimulated reporting” in a subset of drugs examined, the majority of FDA alerts did not appear to undergo substantial changes in reporting rates because of such warnings. When we compared AE reporting trends before and after both FDA alerts and sham alerts, the overall pattern of AE reporting was very similar between the groups. This lack of an overall “stimulated” reporting pattern held for the short-term (two quarters before and after an alert was issued) and long-term (four quarters before and after) analyses, as well as the all primary suspect cases and AE-specific/drug pair analysis.

Numerous factors are likely to influence whether a given FDA alert results in “stimulated” changes in drug use and/or AE reporting, including: how serious the AE mentioned in the alert is, whether the guidance requires increased workload for prescribers, the extent of FDA and media coverage, whether the scientific basis of the warning is widely accepted, the specificity of the warning, and whether the alert includes guidance for the selection of safer alternatives.

Traditional media coverage also shapes the dissemination of emerging drug safety information and can significantly affect the impact of FDA warnings (for a review, see Yong et al. [73]). One study found that 100 % of newspaper articles included at least one mention of a benefit of a newly approved drug, while only 32 % mentioned at least one harmful effect [74]. The Internet can also be a significant factor in behavioral changes to drug safety information [75]. Unfortunately, sometimes the message delivered by regulatory authorities is not the same one that is relayed by the media [73, 76–78].

Willingness to follow the FDA’s guidance is yet another variable that can have a significant impact. For example, in one study of REMS guidance, 71 % of prescribing physicians stated that it would be “somewhat” to a “very” significant barrier to have to endure a locally available, one-time training session of less than 2 h to guide them on the proper use of emerging opioids [28]. Along the same lines, another study demonstrated that prescribers are less likely to comply with guidance that entails increased patient contact [49]. Additionally, a specific FDA guidance suggesting “close supervision” of patients to monitor something as serious as increased risk of suicide appeared to be entirely disregarded [48].

As in most areas of risk communication, FDA warnings and alerts appear to be most effective when they are specific, offer alternative options, and are repeated. Indeed, other studies [17, 79–82] have suggested that FDA alerts must be concise, actively publicized through scientific and traditional media, and offer immediate prescribing options (not added educational programs or increased patient time) for warnings to trigger meaningful prescribing changes.

We are encouraged that modern FAERS data do not appear to suffer from the biases that would be introduced by significant shifts in reporting owing to the issuance of FDA alerts.

Limitations

The limitations of our study include the fact that we did not have data to correlate how much media coverage accompanied each FDA warning, nor any way to gauge how well each warning was communicated to, or received by, prescribers. We were not able to determine if any of the FDA alerts we analyzed were pre-dated by alerts issued in non-US countries. We measured the “primary suspect” case reports in an attempt to restrict the analysis to those drugs directly suspected of causing the AE, but such a selection process could have unintended consequences with regard to the analysis of stimulated reporting as the impact on secondary suspects was not studied. Scrip (drug usage) numbers were not available to correlate with AE counts. The inclusion criteria of 40 cases were intended to focus the analysis on drugs that had sizeable use rates and was also needed to make the dataset a manageable size. Such steps, however, may have excluded certain drugs where AE reporting trends went up or down after an alert. Using each drug as its own internal control necessitated the use of drugs that had at least nine quarters of FAERS data before an actual FDA alert, potentially biasing the sham alert analysis towards slightly older drugs. Our results are based upon the FAERS database alone, so the relation between our findings and smaller databases is unknown. The strength of this research is that it comprehensively examined a large number of individual drugs, included all FDA warning types, and analyzed trends over an extended period of time.

Conclusion

While there is no doubt that occasionally an FDA alert can have a profound influence on AE reporting rates for a given drug, we were struck by how similar the overall reporting trend distributions were between drugs with real compared with sham alerts. We therefore suggest that modern day AE reporting trends do not appear to be substantially affected by FDA alerts.

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 119 kb)

Adverse event (AE) databases such as the US Food and Drug Administration (FDA) Adverse Event Reporting System are assumed to suffer from “stimulated reporting”.

“Stimulated reporting” is the concept that public disclosure of a safety issue by the issuance of, for example, an FDA alert will result in substantially increased AE rates for the drug and/or specific safety issue mentioned in such an alert.

We analyzed both overall and AE-specific reporting before and after 100 FDA alerts and found no discernable pattern of increased reporting.

While certain drugs appeared to have their AE reporting trends affected by an FDA warning, most did not.