Incomplete evidence: the inadequacy of databases in tracing published adverse drug reactions in clinical trials.

BACKGROUND: We would expect information on adverse drug reactions in randomised clinical trials to be easily retrievable from specific searches of electronic databases. However, complete retrieval of such information may not be straightforward, for two reasons. First, not all clinical drug trials provide data on the frequency of adverse effects. Secondly, not all electronic records of trials include terms in the abstract or indexing fields that enable us to select those with adverse effects data. We have determined how often automated search methods, using indexing terms and/or textwords in the title or abstract, would fail to retrieve trials with adverse effects data.
METHODS: We used a sample set of 107 trials known to report frequencies of adverse drug effects, and measured the proportion that (i) were not assigned the appropriate adverse effects indexing terms in the electronic databases, and (ii) did not contain identifiable adverse effects textwords in the title or abstract.
RESULTS: Of the 81 trials with records on both MEDLINE and EMBASE, 25 were not indexed for adverse effects in either database. Twenty-six trials were indexed in one database but not the other. Only 66 of the 107 trials reporting adverse effects data mentioned this in the abstract or title of the paper. Simultaneous use of textword and indexing terms retrieved only 82/107 (77%) papers.
CONCLUSIONS: Specific search strategies based on adverse effects textwords and indexing terms will fail to identify nearly a quarter of trials that report on the rate of drug adverse effects.

Background

Both physicians and patients need to know the likelihood of adverse effects of drugs in order to assess the efficacy:risk ratio of a particular therapy. This applies to serious clinical effects that may cause significant morbidity or mortality, and to more "trivial" symptoms that may affect quality of life and drug compliance. Lists of adverse effects can be obtained from reference texts or pharmaceutical companies, but details of frequencies are often not available [1]. Randomised controlled trials, in conjunction with case reports and observational studies, can potentially provide useful evidence on the frequencies of adverse effects. However, some trials are too small for reliable estimates, making a systematic review of the data – ideally a meta-analysis – necessary, and in order to carry out an unbiased review, we need to retrieve a complete set of data.

We became aware, from our own meta-analytical work [2], that apparently comprehensive search strategies combining textwords (e.g. "drug name" and "adverse or side effect" and "trial") with indexing terms (e.g. "drug-AE" and "clinical trial") may not always succeed in retrieving records that we know to exist from hand searches. We therefore set out to determine the number of instances in which automated searches of the EMBASE and MEDLINE databases would fail to retrieve trials that we already knew contained adverse effects data.

Methods

We used a sample of papers culled from review articles in which the adverse effects of drug therapy were part of the analysis [2-5]. We manually selected papers that were known to have reported data on the frequency of adverse effects and checked to see if:

1. they had been indexed with the relevant terms for adverse effects, using the Silver Platter Information Retrieval System to check the indexing of their records on the MEDLINE and EMBASE electronic databases;

2. the authors mentioned adverse effects (or any related terms) in the title or abstract (thus enabling the paper to be found in an electronic search).

The absence of both features would make the search procedure far more difficult and time-consuming for those interested in identifying and retrieving papers that reported adverse effects data.

Details of the papers included in the review

Our sample consisted of reports of randomised controlled trials that had been retrieved through a combination of electronic searching and manual checking of full-text articles. All the papers (for a full list see ) reported the presence or absence of adverse effects and provided figures for the rates of adverse effects in both the treated and control arms.

The papers we surveyed came from review articles assessing:

1. the risk of gastrointestinal haemorrhage with aspirin – 37 papers reporting on 24 trials [2];

2. the risk of adverse effects with antihypertensive agents – 13 papers reporting on 13 trials [3];

3. the adverse effects of analgesics in postoperative pain – 28 papers on paracetamol (acetaminophen) and 29 on ibuprofen [4,5].

The aspirin review specifically studied gastrointestinal bleeding, so the free text terms "h(a)emorrhage", "bleed(ing)", "mel(a)ena", "h(a)ematemesis", and "gastrotoxic" were accepted as relevant, in addition to general terms such as "side/adverse effect". The other two reviews studied all adverse effects associated with the drugs. Since a potential reviewer would not know in advance of a search all the observed/reported adverse effects, we looked in the title and abstract for general terms such as "side/adverse effect" (which might be used to qualify specific symptoms, such as nausea), and accepted specific symptoms in the indexing fields only when they were clearly categorized as adverse effects, e.g. "nausea-CI" or "nausea-SE".

All the papers were indexed on either MEDLINE (n = 100), or EMBASE (n = 88), or both (n = 81). However, in seven papers the abstract was missing from the electronic database, and in those cases our findings are based on the textwords present in the abstracts of the printed versions.

Results

Papers that reported adverse effects were not consistently indexed in the two electronic databases (Figure 1). Of the 81 records logged in both MEDLINE and EMBASE, only 30 (37%) were indexed for adverse effects in both databases. In most instances, papers were indexed in one but not the other (26/81), or were not indexed in either (25/81). When each database was assessed individually, there was little difference in the proportion of papers indexed for adverse effects – MEDLINE 53/100 records (53%) and EMBASE 43/88 records (49%) – although, as is clear from Figure 1, they each indexed different sets of papers.

Figure 1

Assignment of indexing terms for adverse effects in 81 papers with records available in both Medline and EMBASE. All the papers were known to contain information on adverse effects.

Similarly, a large proportion of papers did not mention adverse effects (or related terms) in the title or abstract, and therefore would not have been detected using a textword search (Table 1). A combined search covering the two databases and using both the index and textword terms would still have failed to pick up 25 of the 107 papers that contained adverse effects data.

Table 1

Number of papers that could be retrieved based on the reporting of adverse effects.

Drug (total number of papers)	Aspirin (37)	Antihypertensive drugs (13)	Ibuprofen (29)	Paracetamol (28)
Adverse effects textwords present	18 (49%)	8 (62%)	20 (69%)	16 (57%)
Indexed in either Medline or EMBASE	28 (76%)	8 (62%)	18 (62%)	15 (54%)
Retrievable by a combined search	29 (78%)	9 (69%)	24 (83%)	20 (71%)

*Abstract from seven of the 107 papers were missing from the electronic databases; in these cases, the printed abstracts were analysed.

Full details of the textwords and indexing terms for adverse effects used in the papers can be found in Table 2, Table 3 and Table 4.

Table 2

"Adverse Effects" Textwords in Titles or Abstracts*

Drug	AE term as	Terms used
(no. of papers)	textword*	[frequency]
Aspirin	18	bleed(s/ing) [11], side effects [5], haemorrhag(e/ic) [4], adverse effects [3], safe(ty) [2], adverse event [1], gastrotoxic [1]
(37)		haematemesis [1], melaena [1]
Antihypertensive drugs	8	adverse effects [3], side effects [3], adverse reactions [1], risks [1]
(13)
Ibuprofen	20	tolerability/tolerance/well tolerated [9], adverse effects [8], safe(ty) [6], side effect(s) [5], adverse events [2], unwanted
(29)		effects [1]
Paracetamol	16	side effects [7], adverse effects [5], tolerability/tolerance/well tolerated [4], safe(r/ty) [3], adverse reactions [2], adverse
(28)		events [1]

*Abstracts from seven of the 107 papers were missing from the electronic databases; in these cases, the printed abstracts were analysed.

Table 3

Indexing for Adverse Effects in MEDLINE

Drug	Records	Indexed	Terms used
	avaliable	for AE	[frequency]
Aspirin	37	26	Aspirin-AE [25]
			Hemorrhage-CI [5]
			Platelet-aggregating-inhibitor-AE [3]
			Safety [1]
Antihypertensives	13	6	Antihypertensive agents-AE [5]
			Named drug*-AE [2]
Ibuprofen	23	12	Ibuprofen-AE [12] Antiinflammatory-agent-non-steroidal-AE [2]
			Analgesics-AE [1]
Paracetamol	27	9	Acetaminophen-AE [9]

*name of any antihypertensive drug (e.g. atenolol, enalapril, nifedipine)

Table 4

Indexing for Adverse Effects in EMBASE

Drug	Records	Indexed	Terms used
(no. of reports)	available	for AE	[frequency]
Aspirin	30	16	Adverse-reactions-titles [13]
(37)			Bleeding-SE [8]
			GI-hemorrhage-SE [5]
			Adverse-drug-reaction [4]
			Antithrombotic-agent-ADR [2]
			ASA-ADR [10], ASA-SE [1]
			Hematemesis-SE [1], Melena-SE [1]
			Stomach-hemorrhage-SE [1]
Antihypertensive drugs	9	3	Adverse-drug-reaction [3]
(13)			Adverse-reactions-titles [1]
Ibuprofen	26	13	Adverse-reactions-titles [12]
(29)			Specific symptom*-SE [9]
			Ibuprofen-ADR [8]
			Adverse-drug-reaction [3]
			Analgesic-agent-ADR [1]
			Drug-safety [1]
			Drug-tolerability [1]
			Nonsteroidal-antiinflammatory-agent-AE [1]
			Side effect [1]
Paracetamol	23	11	Adverse-drug-reactions-titles [10]
(32)			Adverse-drug-reaction [8]
			Paracetamol-ADR [2]
			Specific symptom*-SE [2]
			Drug-safety [1]

*any named symptom (e.g. nausea, vomiting)

Discussion

Our results show that automated searches of MEDLINE and EMBASE will not succeed in retrieving all trials with adverse effects data because:

• indexing terms for adverse effects were not assigned to some of the papers, even though they contained data on adverse effects frequencies;

• the authors made no mention of adverse effects (or related terms) in the title or abstract, although the paper itself contained numerical information about adverse effects.

This latter point explains the failure of textword searches; more importantly, it shows that manual checking of trial abstracts will not help reviewers in determining whether adverse effects data are available from a trial report.

There are several reasons for this failure to mention adverse effects in abstracts of papers or indexes of databases. Guidelines on the structured reporting of clinical trials are directed towards the reporting of therapeutic efficacy [6]. Authors may not feel that adverse effects deserve a mention in the confined space of an abstract, especially as there are already so many other requirements about the information they need to provide.

There are no specific rules concerning what merits indexing as "drug toxicity" or "side effect" in the EMBASE database, but general guidelines are that indexing should take place if significant information relevant to clinical use is presented in the article (Jos Hageman, personal communication). Indexing terms for adverse effects in the MEDLINE database are generally assigned only to papers in which the author devotes substantive discussion to the adverse effects (Ione Auston, Beth Van Lenten, personal communication). This usually occurs only when the author considers the effects to be significant or serious. However, most trials are not powered to detect significant differences in rates of adverse effects, and serious ones tend to be rare. Furthermore, few authors actually devote substantial amounts of space to safety data [7]. This may explain why only about half of the trials we surveyed in which adverse effects data were reported had been assigned adverse effects indexing terms.

Data on adverse effects could be more easily retrieved if trials that reported adverse effects were distinguishable in electronic databases from those that did not. However, about 23% of papers containing adverse effects data do not have any adverse effects textwords or indexing terms attached to them, and would therefore be missed by electronic searches. These papers can be identified only by using a broad search strategy with no specific adverse effects terms, followed by manual (and potentially laborious) searching of the full-text versions. We found this to be true whether we were searching for a specific adverse effect associated with a specific drug, or for all adverse effects associated with a specific drug or type of therapy.

It is important that search strategies should reliably retrieve all trials in which drug adverse effects are reported, so that we can readily access and use all the existing evidence. However, index-based searches for adverse effects may omit papers that do not report significant or serious adverse effects. Unreliable estimates of the rate or severity of drug adverse effects may occur as a result of these limitations in the current indexing system.

Although there has been some recent interest in improving the adequacy of adverse effects reporting in clinical trials [6,7] the benefits of better reporting may be wasted if steps are not taken to make the data more easily available. A number of relatively simple changes to both the reporting and indexing of drug adverse effects could significantly increase the accessibility of such evidence from future publications, and allow more reliable estimates of their significance to be given to patients. As such, we make the following proposals:

1. Indexers should agree on a unified system for indexing drug adverse effects. Any trial that reports on the frequency of adverse effects needs to be indexed, even if the adverse effects were judged not significant or not severe. Information on the absence of serious adverse effects is of no less value to those who are involved in making therapeutic decisions.

2. Journals that publish structured abstracts should require authors to mention drug adverse effects in the abstract whenever they are reported in randomised controlled trials, even if no adverse drug reactions occurred in the trial.

Conclusions

Implications for clinical practice

Clinical staff using quick and specific search methods for trial data on adverse effects will fail to retrieve some potentially relevant papers, and may retrieve a biased selection.

Implications for research

Researchers carrying out systematic reviews of adverse drug effects in clinical trials will need to carry out manual checking of full-text articles of all trials of the relevant drug in order to ascertain whether adverse effects data are reported or not.

Competing interests

None declared

Pre-publication history

The pre-publication history for this paper can be accessed here: