Assessment of factors associated with completeness of spontaneous adverse event reporting in the United States: A comparison between consumer reports and healthcare professional reports.

WHAT IS KNOWN AND
OBJECTIVE: The objectives of this study were to explore completeness of direct adverse event (AE) reports from consumers and healthcare professionals (HCPs), and to discuss the reasons completeness varied among reporters with different occupations.
METHODS: We used a total of 5475 direct AE reports to the United States (US) Food and Drug Administration (FDA) from the first and second quarters of 2016 and assessed completeness of basic information (eg, patient sex, age, weight) and information relevant to AEs (eg, suspect and concomitant drugs). Logistic regression analysis was conducted to evaluate the associations between report completeness and reporting backgrounds. RESULTS AND DISCUSSION: The completeness of AE reports from consumers was generally greater than that of reports from HCPs. Completeness of specific items varied among different occupations, which may reflect accessibility to, and/or availability of, relevant information for each type of reporter. There was a clear association between the proportion of 'known' ADRs in a report and completeness, suggesting that consumers and HCPs are likely to consult labelling information when reporting AEs. WHAT IS NEW AND
CONCLUSION: The quality of AE reports seemed to depend on information costs accrued to potential reporters. Researchers should consider the impact of database heterogeneity and possible sample selection bias when using spontaneous AE reports as a sample of events in the United States.

WHAT IS KNOWN AND OBJECTIVE

Direct reporting of adverse drug reactions (ADRs) to authorities was made available to patients and consumers worldwide in the 2000s. Access has been facilitated through legislative changes in the European Union (EU) and release of a new consumer‐friendly reporting form in the United States (US).1, 2, 3 However, the quality of reports made by consumers is of concern. For example, when introducing consumer reporting programs in the EU, an argument was made that patient reports tended to be easily influenced by mass media, which might result in increased reports with trivial or well‐known ADRs.4, 5, 6, 7, 8 Recent studies showed that ADRs reported by patients overlap with ADRs reported by healthcare professionals (HCPs), but the ADRs are reported in different ways. For example, patients tend to report reactions that affect quality of life, whereas these reactions are unlikely to be reported by HCPs.1, 2, 3, 4 Although the quality of ADR reports from patients and HCPs is comparable in many ways, it is possible that reporting behaviours may be different between consumers and HCPs because of obvious differences in professional expertise, motivation and practical restrictions related to submission of ADR reports. Given the growing number of direct ADR reports from patients in the United States, the EU and other jurisdictions, increased heterogeneity caused by diverse reporters and clinical environments would be a critical issue in monitoring drug safety with pharmacovigilance databases.2, 4, 5

Spontaneous ADR reporting systems such as the US Food and Drug Administration (FDA) adverse event reporting system (FAERS) and the EudraVigilance database are cornerstones of post‐marketing pharmacovigilance. They provide solid scientific and regulatory foundations for drug companies and regulators. However, there are several barriers to successful detection of safety signals, including incompleteness of ADR reports and under‐reporting, which are inevitable in a system that relies on voluntary actions. Most studies aimed at developing methods to detect ‘true’ safety signals have tried to minimize the impact of missing information using disproportional analysis and quality indicators.9, 10, 11, 12, 13, 14 For example, vigiGrade, which was developed by the Uppsala Monitoring Centre as a unique completeness indicator to conduct safety signal detection, applies additional weight to reports in vigiBase, the pharmacovigilance database used by the World Health Organization, to identify well‐documented ADRs. It is recognized as a useful tool to measure the amount of clinically relevant information because obvious confounders such as suspected causality between the drug and adverse events (AEs) do not interfere with the indicator.15 However, there has been no other indicator developed to measure the heterogeneity in quality of AE reports caused by diversification of reporters, because behavioural patterns of reporters (ie, when/where/how reporters report AEs directly/indirectly to the FDA) have not been fully elucidated. Developing practical indicators would entail methodological difficulties. For instance, it is necessary to consider ‘known’ ADRs included in relevant drug labels when evaluating the reporting rates of secondary suspect drug or concomitant drugs. Availability of information varies by reporters' occupation and/or clinical environment. Due to these difficulties, there has been no detailed investigation of heterogeneities underlying in AE reporting. Better understanding of behavioural patterns across reporters' occupation would serve to provide clues of how to handle such heterogeneities in pharmacovigilance activities.

The US FDA receives numerous voluntary AE reports directly and indirectly (ie, via manufacturers) from consumers and HCPs, and provides safety information for appropriate use of drugs in the form of safety alerts and monthly summaries of drug products that have undergone safety labelling changes. Regarding the quality of AE reports, a previous study showed that completeness of AE reports via manufacturers was much lower than that of direct AE reports.16 However, no comprehensive investigation of FAERS has evaluated differences in report completeness according to type (occupation) of reporter and clinical conditions. This study aimed to show similarities and differences in completeness of AE reports from consumers and HCPs in the United States and to explore the underlying mechanisms of AE reporting, especially focusing on the roles of availability of information (eg, previously known ADRs) and other circumstances surrounding reporters.

METHODS

We examined reports submitted by consumers and HCPs to the FAERS database in the first and second quarter of 2016. We chose the observation period to avoid the impact of spikes in the number of voluntary reports triggered by the introduction of consumer‐friendly reporting form FDA3500B in 2013 and also to compare the current findings with the results of descriptive analysis in a previous study.3 To evaluate the impact of candidate factors such as known ADRs and duration on market of primary suspect drug on reporting rates of reporters, out of 25 814 direct reports in the first and second quarter of 2016, 5475 which had information required for the evaluation were selected as shown in Figure 1. Descriptive statistics are summarized in Table 1. Of the 5475 direct reports, 5369 reports containing patient sex and age were used for logistic regression analyses.

Figure 1

Flow chart of data selection. To evaluate the impact of candidate factors such as previous drug information (eg, known ADRs and duration on market) on reporting rates of reporters, 5475 direct reports that met 4 criteria were identified. Filters 1 and 2 were required to determine if AEs were known ADRs of the primary suspect drug. Filter 3 was used to calculate the duration after drug approval in the United States. Filter 4 was used to calculate time‐to‐report

Table 1

Completeness of direct AE reports, categorized by 1st reporter occupation

Variables	All reports n = 5475 (%)	CNs		HCPs
		CNs n = 212 (%)	UNs n = 2162 (%)	PHs n = 2116 (%)	MDs n = 381 (%)	OTs n = 604 (%)
Patient sex	5412 (99)	203 (96)	2141 (99)	2091 (99)*	377 (99)*	600 (99)*
Patient age	5411 (99)	205 (97)	2130 (99)*	2095 (99)*	380 (100)	601 (100)*
Indication(s)	3910 (71)	163 (77)	1624 (75)	1634 (77)	121 (32)*,**	368 (61)*,**
Onset of treatment with primary suspect drug	2840 (52)	127 (60)	1298 (60)	1092 (52)**	89 (23)*,**	234 (39)*,**
Concomitant drug(s)	2184 (40)	117 (55)	1403 (65)*	501 (24)*,**	81 (21)*,**	82 (14)*,**
Secondary suspect drug(s)	788 (14)	32 (15)	219 (10)	302 (14)**	139 (36)*,**	96 (16)**
Patient weight	3309 (60)	165 (78)	1861 (86)*	745 (35)*,**	350 (92)*	188 (31)*,**

Abbreviations: CNs, consumers; HCPs, healthcare professionals; MDs, physicians; OTs, other HCPs; PHs, pharmacists; UNs, ‘unknown occupation’ reporters (including consumers using FDA3500B).

P < .01 (vs CNs).

P < .01 (vs UNs).

Data sources

We used the JAPIC AERS database, which is comprised of the FAERS database cleaned by the Japan Pharmaceutical Information Center (JAPIC). We used the sider database version 4.1, which provides ADRs described in drug labels of marketed chemical medicines in the United States, to determine whether suspected ADRs in reports were known (ie, written in the labels at the time of AE reporting).3 The results of mapping to the MedDRA dictionary used in the sider4.1 database were available under a Creative Commons Attribution‐Noncommercial‐Share Alike 4.0 License.

We collected data on safety labelling changes from monthly safety labelling changes listed on the FDA MedWatch website.17 Using archival data, we counted the number of times the primary suspect drug underwent safety labelling changes in the black box warning, warning and contraindication sections.

Definition of reporters

We classified reporters into consumers (CNs), pharmacists (PHs), physicians (MDs) and other HCPs (OTs) according to the reporter occupation listed in the form. Although the name of form FDA3500B is ‘Consumer Voluntary Reporting’, we defined the occupations of reporters using FDA3500B as ‘unknown’ (UN) because the form lacks an occupation column and because reporters using FDA3500B were different from those using the traditional FDA3500.3

Statistical analysis

We compared completeness of items in direct AE reports by reporter occupation. Patient sex, age, indications and onset of treatment with primary suspect drug were examined because they are used in vigiBase. We also focused on patient weight, concomitant drugs and secondary suspect drugs. We used the completeness of these three items as the objective variables in regression analysis because their completeness varied significantly among reporters, which agreed with the results of a previous study,3 and because they differed in information costs and background. Patient weight was accessible from patients and in clinical records, but was difficult for pharmacists to obtain. Concomitant drugs and secondary suspect drugs were interesting items because completeness of information may have reflected decisions on causality from reporter knowledge and drug therapy experience.

We examined the associations between the completeness of each of the three main items and the characteristics of the reporters, and the level of uncertainty regarding the AE, adjusting for types of primary suspect drugs and classes of AEs at SOC levels of MedDRA. The explanatory variables for previous history and information regarding primary suspect drugs were the proportion of known ADRs (known ADR rate) in the report, number of previous safety labelling changes and days on market. Known ADR rate was the ratio of the number of known ADRs of primary suspect drug to the total number of AEs. As previously reported,3 known ADRs were determined with ADR information on drug labels available in the SIDER4.1 database which was updated and released at Nov 2015. A dummy variable showing the number of days for which the primary suspect drug was on the market (ie, <5, or 5 years or more) was used to adjust the common trend of ADR reporting.18 Time‐to‐report (ie, days between the onset of AEs and report submission) and the number of ADRs in a report were used to as background controls.

We used mixed‐effects logistic regression models to account for within‐report variance due to heterogeneity caused by therapeutic circumstance, most of which was explained by the type of primary suspect drug. We performed regression analyses using the lme4 package in r (ver. 3.4.4, https://www.r-project.org). Chi‐squared test was used for inter‐group comparisons between the following seven pairs: CNs‐UNs, CNs‐PHs, CNs‐MDs, CNs‐OTs, UNs‐PHs, UNs‐MDs and UNs‐OTs. The significance threshold for comparisons in Tables 1 and 2 was set to 0.01. The significance threshold for Table 3 was set to 0.1 as in previous studies.19, 20

Table 2

Descriptive analysis of primary suspect drug information

Variables	All reports n = 5475 (%)	CNs		HCPs
		CNs n = 212 (%)	UNs n = 2162 (%)	PHs n = 2116 (%)	MDs n = 381 (%)	OTs n = 604 (%)
Known ADR rate
0.5‐1	2634 (48)	86 (41)	973 (45)	1125 (53)*,**	134 (35)**	316 (52)*,**
Number of previous labelling changes in BW, C or W sections
1 or more	2638 (48)	106 (50)	1078 (50)	1006 (48)	103 (27)*,**	345 (57)**
Years after approval
Five or more years	4061 (74)	165 (78)	1768 (82)	1407 (66)*,**	336 (88)*,**	385 (64)*,**

Abbreviations: ADR, adverse drug reaction; AEs, adverse events; BW, boxed warnings; C, contraindications; CNs, consumers; HCPs, healthcare professionals; MDs, physicians; OTs, other HCPs; PHs, pharmacists; UNs, ‘unknown occupation’ reporters (including consumers using FDA3500B); W, warnings.

P < .01 (vs CNs, chi‐squared test).

P < .01 (vs UNs, chi‐squared test).

Table 3

Results of logistic regression analysis

	Patient weight			Concomitant drug			Secondary suspect drug
	Estimate	SE	P‐value	Estimate	SE	P‐value	Estimate	SE	P‐value
Reporters' occupation
CN	(Ref.)			(Ref.)			(Ref.)
UN	1.401	0.513	.006**	0.636	0.323	.049*	−0.294	0.52	.572
PH	−1.716	0.514	<.001***	−0.863	0.327	.008**	0.235	0.51	.645
MD	1.575	0.696	.024*	−1.027	0.419	.014*	0.205	0.587	.727
OT	−0.937	0.55	.089*	−1.154	0.37	.002**	0.596	0.531	.261
Factors related to primary suspect drugs
Known ADR rate	−0.158	0.126	.21	−0.242	0.084	.004**	−0.402	0.126	.001**
Number of previous labelling changes	−0.184	0.066	.005**	−0.011	0.035	.759	−0.09	0.052	.084*
Five or more years after 1st approval	0.734	0.293	.012*	−0.163	0.162	.313	0.33	0.234	.158
Other background factors
Age	0.01	0.003	.002**	−0.001	0.002	.752	−0.001	0.003	.77
Sex, Male (base = female)	0.394	0.117	<.001***	−0.336	0.076	<.001***	0.121	0.101	.231
log10 (Time‐to‐report + 1)	1.106	0.331	<.001***	0.315	0.191	.099*	0.415	0.285	.145
Number of AEs	0.207	0.056	<.001***	0.077	0.026	.004**	0.036	0.033	.279
Interaction
log10 (Time‐to‐report + 1): CNs	(Ref.)			(Ref.)			(Ref.)
log10 (Time‐to‐report + 1): UNs	−1.154	0.348	<.001***	−0.26	0.2	.194	−0.137	0.302	.651
log10 (Time‐to‐report + 1): PHs	−0.614	0.348	.078*	−0.27	0.206	.19	0.038	0.3	.899
log10 (Time‐to‐report + 1): MDs	−0.909	0.492	.065*	−0.313	0.263	.233	−0.079	0.348	.82
log10 (Time‐to‐report + 1): OTs	−0.633	0.381	.097*	−0.515	0.249	.038*	−0.407	0.332	.22
Constant	−0.006	0.596	.993	0.043	0.368	.906	−3.284	0.57	<.001***

Regression analyses were adjusted by kinds of primary suspect drugs and 27 System Organ Classes of MedDRA for AEs.

Abbreviations: ADR, adverse drug reaction; AEs, adverse events; CNs, consumers; HCPs, healthcare professionals; MDs, physicians; OTs, other HCPs; PHs, pharmacists; Ref., reference; SE, standard error; UNs, ‘unknown occupation’ reporters (including consumers using FDA3500B).

P < .1.

P < .01.

P < .001.

RESULTS

A total of 5475 direct AE reports were eligible for our analyses (Figure 1). The reports consisted of 212, 2162, 2116, 381 and 604 reports from consumers, reporters with unknown occupations (ie, mostly consumers using FDA3500B), pharmacists, physicians and other HCPs, respectively.

Report completeness

The completeness of each item is summarized in Table 1. Patient sex and age were completed nearly perfectly in AE reports generated by all types of reporters. The completeness of indication(s) and onset of treatment with primary suspect drug were low in reports from physicians and other HCPs. Consumers tended to report concomitant drugs more completely than pharmacists, physicians and other HCPs. The completeness of secondary suspect drugs was greater in reports by physicians than in those generated by other reporters. The reporting rate of patient weight was lower in reports from pharmacists and other HCPs than in those from consumers.

Characteristics of reports regarding information on primary suspect drugs

Reports from pharmacists and other HCPs tended to have ADRs that were included in labels (Table 2). Physicians tended to report AEs of primary suspect drugs that had never undergone safety labelling changes in the boxed warning, contraindication or warning sections. Pharmacists and other HCPs were most likely to report AEs of primary suspect drugs that had recently entered the market.

Factors associated with report completeness

Regression analysis results for reporting of concomitant drugs, secondary suspect drug(s) and patient weight are summarized in Table 3. Consumers tended to report concomitant drugs to a greater extent that did any HCPs (P < .1). Pharmacists (P < .001) and other HCPs (P < .1) were significantly less likely to report patient weight. There was a clear negative association between known ADR rate and completeness for concomitant drugs and secondary suspect drug(s) (P < .01), but not for patient weight. The number of labelling changes was negatively associated with completeness of secondary suspect drug (P < .1) and patient weight (P < .01), but not with completeness of concomitant drugs. Longer periods on the market (five or more years after approval) were positively associated with completeness of patient weight (P < .1). There was a positive association between time‐to‐report and completeness of concomitant drugs (P < .1) and patient weight (P < .001). Reports for male patients tended to have high completeness of patient weight (P < .001), but low completeness of concomitant drugs (P < .001) compared with those in reports for female patients.

Completeness of patient weight differed between consumers using different reporting forms. FDA3500B users tended to complete patient weight at a higher rate than those who used FDA3500, but this tendency disappeared as time‐to‐report increased.

DISCUSSION

We investigated completeness of direct AE reports in the United States, with a particular focus on reporter occupation and several other important items contained in the reports. The results showed that report completeness for basic patient attributes such as age and sex was similarly high among all types of reporters, but completeness for some items significantly varied among different types of reporters. Interestingly, report completeness for concomitant drugs was much higher in reports from consumers than in those from HCPs. Patient weight was more likely to be reported by consumers and MDs. Regression analysis of patient weight, concomitant drugs and secondary suspect drug suggested that availability of, and/or accessibility to, information on patients and drugs, which depended on the occupational environment of the reporters, seemed to significantly impact completeness for all types of reporters. Furthermore, our findings indicated that consumers as well as HCPs might refer to drug labels to check previously known ADRs when reporting AEs.

As shown in Table 1, the proportion completed differed among items. This may have been because different items required different types of information that may have been differentially available to different reporter types. For example, patients and MDs were able to obtain patient weight information at much lower information costs than pharmacists in a typical healthcare setting. The observed differences in report completeness for patient weight may reflect differences in data accessibility/availability for each reporter occupation.

To investigate possible reasons for differences in completeness, we analysed three items (patient weight, concomitant drugs and secondary suspect drug) using regression analysis, each of which has unique characteristics in terms of accessibility/availability. Patient weight is personal information for which clinical expertise is not required. Concomitant drugs are also personal, may only be known to patients, and reporting may reflect characteristics of the reporter, such as eagerness or thoroughness. It is possible that clinical experts may omit concomitant drugs for which causality of the reported AEs is considered unlikely based on professional opinion. Decisions on whether to report secondary suspect drugs may also be complex. Reporting of secondary suspect drugs may require pharmacological knowledge, such as information about drug‐drug interactions, to assign causality to secondary suspect drug. Access to public and private drug information websites on the Internet, which allows access to additional background information on drug safety, is likely to influence completeness for secondary suspect drugs.

Regression analysis of the three items above showed that completeness for each item varied among reporters with different occupations, and that information costs incurred in various forms seemed to play important roles in determining which item(s) specific types of reporters were likely to complete in voluntary reports.

The tendency of consumers and MDs to report patient weight more often than pharmacists was supported by the regression analysis (Table 3). Concomitant drugs were more likely to be reported by consumers than by HCPs. In contrast, completeness for secondary suspect drug was not associated with reporter occupation. These findings suggested that accessibility to private information directly impacts completeness of reports. However, these findings may be explained by other plausible hypotheses, as indicated in previous studies.21 Consumers may be more meticulous and eager to report all information compared with HCPs. Healthcare professionals are often pressed for time, which may also explain our findings. Although testing of these alternative hypotheses is beyond the scope of this study, these factors may contribute to the broader understanding of the role of data accessibility/availability in completeness of reporting.

The negative association between the proportion of known ADRs and completeness shows the critical role previously available information plays in reporting possible AEs when there is a significant level of uncertainty. This negative association indicated that the presence of more known ADRs correlated with reporting of only the primary suspect drug, and not any secondary suspect drugs, suggesting that AE reports tend to focus on specific drug‐ADR pairs included in existing labelling. This finding suggests that potential reporters, including consumers and HCPs, with knowledge of AEs would refer to drug databases (on the Internet) and check the validity of their decisions on whether to report them. If a drug‐ADR pair was detected in drug databases, it was likely to be submitted because reporters were more convinced of causality between the drug and the ADR. A similar negative association for concomitant drugs, but not for patient weight, further supported this conjecture.

Of note, the observed tendency to report drug‐ADR pairs listed in drug labels does not always represent the purpose of the voluntary reporting system, particularly when this occurs excessively. Epidemiological risks accompanying this tendency include the possibility that unknown ADRs will be overlooked and the possibility that the prevalence of known ADRs may be overestimated through biased sample selection. Providing detailed information on drug safety obtained in clinical trials and through pharmacovigilance would enhance ADR reporting from laypersons, but might prompt specific types of reports related to specific ADR‐drug pairs, which may distort the true distribution and nature of ADRs. Our results suggest that an appropriate balance should be reached regarding how drug information should be interpreted in public databases.

The positive associations between time‐to‐report and completeness of patient weight and concomitant drugs may reflect that it takes time for reporters to obtain some type of information, even though they know how to access to it. This is in‐line with the result that time‐to‐report was not associated with completeness of secondary suspect drug, which is less likely to depend on just accessibility of information. Interestingly, the associations were different even between consumers using different reporting forms as discussed below.

The clinical conditions and environments in which reporters are treated and work are closely related to accessibility and availability of information, which may impact ADR report quality. Our regression analysis suggested that reports from consumers using the classical reporting form (FDA3500) were different from reports from consumers using the new form (FDA3500B) in completeness of the three items upon which we focused, which agreed with a previous study.3 The form FDA3500B, which was released in 2013 to facilitate spontaneous consumer reports, contains the same items as FDA3500, except for occupation. The instructions in FDA3500B on the current FDA homepage are easier for consumers to understand than FDA3500, even for first‐time users. FDA3500 users may be closely connected to healthcare professionals and clinical institutions, whereas FDA3500B users may report ADRs independently. Further investigations should focus on whether reports were submitted by the patient or by another agent(s), as this could result in various differences in information costs and quality of reports. However, there is currently no ‘agent’ flag on either form. The choice of reporting form is an indication of the characteristics of reporters, and thus, can be used to discuss reporting quality, and for data mining purposes.

The present study raises the possibility that practical incentives and restrictions in real‐world settings seem to affect the reporting behaviours in various ways, which might make a significant difference in the contents and quality of ADR reports including the level of completeness. Our findings of this study clarified how voluntary ADR reports in the United States actually vary according to the type of reporters and which background factors might lead to such variation. The possible associations detected in our analysis provide insights on how we could improve the quality of voluntary reports and make the best use of them for the purpose of data mining and meta‐analysis. One of the possible solutions to reduce the impact of heterogeneities in ADR reports on data mining would be to categorize the reports by the type of reporters and reporting forms, and to conduct subgroup analysis in addition to pooled analysis. The fact that some of the heterogeneities observed in our results seem to be ascribed to the reporting forms and/or environments implies that the regulator would be able to further accommodate the preferences and needs of potential reporters to improve the current reporting forms, websites and instructions relevant to voluntary reporting, although paying sufficient attention to potential biases in sampling and reporting.

This study had several limitations. We investigated completeness of reports as a quality measure, but quality of reports likely consists of many other dimensions. In this study, we treated completeness of items as an outcome reflecting various conditions and interactions in ADR reporting. Our findings do not necessarily suggest a simple proposal or solution to improve the quality of ADR reports because there are many trade‐offs between various elements of quality, and also between reporters. Deficiencies in publicly available data, including lack of information regarding whether a consumer or some other agent (eg, family members, lawyers) reported the event, cannot be overcome using the current database.

WHAT IS NEW AND CONCLUSION

The findings of this study suggested that the quality of voluntary ADR reporting depended on information costs accrued to reporters. Reports generated by individuals with different occupations were incomplete in different ways, reflecting restrictions specific to the types of reporters. It is impossible to completely eliminate information costs, so care must be taken to pay appropriate attention to the quality and content of reports based on the type of reporter. Our results indicated that consumers and HCPs tended to focus on different aspects of events and information, and researchers who use data mining procedures should consider these heterogeneities. It is likely that individuals tend to report known ADRs with much greater frequency than unknown AEs, which could result in sample selection biases during safety evaluations. These issues may be further complicated by increased access to several drug databases on the Internet. Although reporting based on sufficient and sound knowledge is of critical importance for the national pharmacovigilance system, regulators may need to encourage consumers to report ‘unknown’ events for which causality with drugs appears unlikely.

CONFLICT OF INTEREST

Tadashi Toki was employed by Daiichi Sankyo Co. Ltd during the study period. Shunsuke Ono declares no conflict of interest.