Educational intervention to improve physician reporting of adverse drug reactions (ADRs) in a primary care setting in complementary and alternative medicine.

BACKGROUND: Recent studies have shown that adverse drug reactions (ADRs) are underreported. This may be particularly true of ADRs associated with complementary and alternative medicine (CAM). Data on CAM-related ADRs, however, are sparse.Objective was to evaluate the impact of an educational intervention and monitoring programme designed to improve physician reporting of ADRs in a primary care setting.
METHODS: A prospective multicentre study with 38 primary care practitioners specialized in CAM was conducted from January 2004 through June 2007. After 21 month all physicians received an educational intervention in terms of face-to-face training to assist them in classifying and reporting ADRs. The study centre monitored the quantity and quality of ADR reports and analysed the results.To measure changes in the ADR reporting rate, the median number of ADR reports and interquartile range (IQR) were calculated before and after the educational intervention. The pre-intervention and post-intervention quality of the reports was assessed in terms of changes in the completeness of data provided for obligatory items. Interrater reliability between the physicians and the study centre was calculated using Cohen's kappa with a 95% confidence interval (CI). We used Mann Whitney U-test for testing continuous data and chi-square test was used for categorical data. The level of statistical significance was set at P < 0.05.
RESULTS: A total of 404 ADRs were reported during the complete study period. An initial 148% increase (P = 0.001) in the number of ADR reports was observed after the educational intervention. Compared to baseline the postinterventional number of ADR reportings was statistically significant higher (P < 0.005) through the first 16 months after the intervention but not significant in the last 4-month period (median: 8.00 (IQR [2.75; 8.75]; P = 0.605). The completeness of the ADR reports increased from 80.3% before to 90.7% after the intervention. The completeness of the item for classifying ADRs as serious or non-serious increased significantly (P < 0.001) after the educational intervention. The quality of ADR reports increased from kappa 0.15 (95% CI: 0.08; 0.29) before to 0.43 (95% CI: 0.23; 0.63) after the intervention.
CONCLUSION: The results of the present study demonstrate that an educational intervention can increase physician awareness of ADRs. Participating physicians were able to incorporate the knowledge they had gained from face-to-face training into their daily clinical practice. However, the effects of the intervention were temporary.

Background

The World Health Organization defines adverse drug reactions (ADRs) as 'a reaction which is noxious and unintended and which occurs at doses normally used in humans for prevention, diagnosis or therapy of disease, or for the modification of physiological functions' [1]. ADRs are considered to be a leading cause of morbidity and mortality [2]. Indeed, it has been shown that approximately 5.3% of hospital admissions were associated with ADRs [3].

Although conventional clinical studies on efficacy and safety are suitable for recognizing frequent ADRs and are required for the approval of a new drug, they often fail to detect rare ADRs [4]. As a result, a number of spontaneous reporting systems have been developed in recent decades to help detect serious, rare, and unexpected ADRs. These systems are characterized, however, by a high rate of underreporting, which varies depending on the types of ADRs and drugs in question [5]. A systematic review about determinants of under-reporting found that a large proportion of physicians did not report ADRs because they felt that these were well known or too trivial [6]. However the knowledge and attitudes of health professionals appear to be strongly related with reporting and are potentially modifiable factors [7].

Educational interventions have been shown to influence reporting rates [8-10]. In a cluster-randomized controlled trial in Portugal, a 10-fold increase in the rate of ADR reports was observed following an educational intervention [8]. In contrast to the targeted outreach visits employed in the Portuguese study, however, a solely passive method of dissemination based on mailing educational materials to selected clinicians did not lead to any improvements in the reporting of ADRs in another study [11].

Over the past 20 years, the use of complementary and alternative medicine (CAM), including homeopathic, anthroposophic, and herbal treatment, has increased markedly in Western industrialized nations [12,13]. This has created a challenging situation when it comes to detecting, evaluating, and reporting ADRs, because many patients – and, remarkably, even CAM physicians – believe that these kinds of treatments are not associated with risk [5]. Patient surveys have suggested that this assumption may be traced to the belief that CAM products are 'natural' and therefore harmless [14]. Assumptions like these have led to a discrepancy between the growing interest in CAM remedies and the limited data on their potential to cause ADRs [13,15-17]. The seriousness of this situation is compounded by the poor quality of most individual ADR reports [18].

The aim of the present study was therefore to analyse the impact of a face-to-face educational intervention and on the quantity and quality of ADR reports filed by a network of 38 CAM physicians working in the primary care sector in Germany. To assess the sustainability of this intervention we also assessed the duration of the effect of this intervention and analysed the impact of the intervention with regards to CAM and non CAM drugs.

Methods

Study design

The present study was designed as a prospective multicentre observational study within the EvaMed Pharmacovigilance Network, which aims to evaluate complementary remedies in primary care with regard to prescribing patterns, efficacy, and safety [19]. As the study is based on anonymised data on adverse drug reactions which are obligatory to report in Germany and no experimental research or intervention on patients were applied, no ethical approval was needed. Physicians were recruited through the German National Association of Anthroposophic Physicians (Gesellschaft Anthroposophischer Ärzte in Deutschland; GAÄD). A total of 362 physicians were contacted and informed about the EvaMed Network by standard mail and, in case of non-response, four weeks later by telephone. For a physician to be eligible to participate in the study, his or her medical practice had to meet a number of technical requirements, including the presence of a special computerized patient documentation system (DocExpert, DocConcept, TurboMed, Duria, AdamedPlus, Medistar), a local area network (LAN) connection, and Microsoft Windows and Internet Explorer (i.e. as client software). A total of 38 physicians from 12 of 16 German states fulfilled the technical requirements, gave informed consent and agreed to participate. None of them was working in a health centre. For electronic documentation the QuaDoSta system already known by the physicians was used, which has been described in detail elsewhere [20,21]. The QuaDoSta system was free for all physicians.

Setting

Anthroposophic medicine is a whole medical system founded in the 1920s by Rudolf Steiner and Ita Wegman. As a form of complementary medicine, it is regarded as an extension to conventional treatment, requiring that a physician work together with his or her patients to understand the meaning of their illness by carefully exploring the physical, mental, and spiritual aspects of their biography. Thus, although anthroposophic physicians employ conventional medical treatments such as surgery and medicine as needed, they also seek to stimulate patients' salutogenetic capacities by using unique anthroposophic therapies and remedies. Anthroposophic remedies include preparations of mineral, botanical, or zoological origin, as well as chemical substances that are either undiluted or based on the homeopathic principle of high dilution. Data on patients who were treated with any kind of drug (i.e. conventional, anthroposophic, homeopathic, or herbal) during the study period were included in the current analysis.

Educational intervention

An interactive educational intervention was designed to increase physician awareness of ADRs and to facilitate the transfer of knowledge into physicians' everyday routines (Figure 1). Starting in January 2004 the participating physicinans documented all ADRs for 21 months in the EvaMed network. After this time period each of the 38 physicians took part in a one-on-one ADR training session, which lasted approximately 1.5 hours. The sessions were held at each physician's place of work and were led by one specially trained study researcher.

Figure 1

Training and monitoring concept.

The educational intervention was divided into a theoretical and a practical part. The theoretical part consisted of a presentation on the economic and epidemiological importance of ADRs, as well as on the definition and classification of ADRs (i.e. in terms of seriousness, severity, and causality). Physicians were instructed to report all ADRs, including those that were mild or anticipated. Every physician received a manual summarizing the main points of the training session. The practical part of the intervention included a problem-based learning course that provided practical examples of how to document ADRs using their QuaDoSta software package. Each year, all physicians received two newsletters with details about the study and were invited to a meeting to discuss their experiences.

Two weeks after the educational intervention, physicians were called and asked if they had encountered any problems; if so, they were provided with support. A telephone hotline was available on weekdays during normal business hours.

Data collection

The study period lasted from January 2004 through June 2007. During the study period, the participating physicians continued to follow their routine documentation procedures, recording diagnoses and prescriptions for each consecutive patient using their existing, computerized patient documentation system. These data were exported electronically to a QuaDoSta system hosted in each practice by an on-site server to avoid double documentation. All ADRs that occurred during the study period were recorded by the physicians using a separate input mask.

Because many anthroposophic physicians often prefer not to use computers to take notes or enter data during patient consultations, a short paper version of the computerized ADR report form was developed. This allowed physicians to collect data by hand and transfer these to the QuaDoSta system after the consultation. Every ADR reported to the QuaDoSta was compensated with € 15.

The ADR report form included obligatory and voluntary items (Table 1). ADRs were classified as serious or non-serious according to International Conference on Harmonization (ICH) guidelines [22], as well as in terms of severity according to World Health Organization Adverse Reaction Terminology (WHO-ART) [22].

Table 1

Obligatory and voluntary items on the adverse drug reaction (ADR) report

Obligatory	Voluntary
Patient details
Patient initials*	Week of pregnancy
Date of birth*	Breastfeeding? (yes/no)
Gender*	Profession
Height*
Weight*
Drug
Name of drug*	If used previously, was the drug tolerated at the time?
Prescribed for*	Was the drug continued or read ministered after onset of ADR?
Date drug started
Date drug stopped
Suspected of causing ADR (yes/no/unsure)
Dosage
Diagnosis
Name*	Type of diagnose (primary, concomitant, secondary)
ICD-10 code*	Diagnosis confirmed on date
ADR
Symptom	Initial worsening of symptom
Severity according to WHO-ART
Serious or non-serious according to ICH	If serious, why?
Date ADR started
Date ADR stopped
Treatment of ADR completed? (yes/no)	Reason for not completing treatment of ADR
Causality

* Imported electronically from the computerized patient documentation system into QuaDoSta

Causality was assessed according to Uppsala Monitoring Centre (UMC) criteria as certain, probable/likely, possible, unlikely, conditional/unclassified, or unassessable/unclassifiable [23]. Causality assessment was based on the nature of the ADR; the association in time between drug administration and the ADR; possible confounding factors; the clinical plausibility of the ADR; dechallenge or rechallenge effects; expectancy according to the information given in the package leaflet.

An ADR report was regarded as complete if information had been provided for all obligatory items. Physicians were required to send their data to the study centre every two months. In the event of a serious ADR, they had to inform the study centre within 24 hours.

Monitoring programme

All ADR reports were monitored by the study centre and checked for

• completeness (i.e. data provided for all obligatory items)

• plausibility (e.g. association in time between drug administration and ADR)

• classification of ADR seriousness and severity according to ICH and WHO-ART criteria

• assessment of causality according to UMC criteria

To assess the quality of reports, the study centre performed its own assessment of reported ADRs according to ICH, WHO-ART, and UMC criteria. These steps took place independently of the participating physicians. Subsequently, the study centre phoned the physicians to confirm that their reports had been received and asked them to supply any missing data. The study centre also provided the physicians with information about ADRs that had been reported in the past or described in the literature in association with the drug in question. The whole data collection period lasted for 42 months (i.e. 21 months before and 21 months after the educational intervention).

Statistical analysis

To measure changes in the ADR reporting rate, the median number of ADR reports and interquartile range (IQR) were calculated before and after the educational intervention. The pre-intervention and post-intervention quality of the reports was assessed in terms of changes in the completeness of data provided for obligatory items. Interrater reliability between the physicians and the study centre was calculated using Cohen's kappa with a 95% confidence interval (CI). For the analysis of the duration of the effect, we followed the line of Figueiras et al. [8] and aggregated five categories: baseline period of the first 21 month, first 4-month after intervention and respective groups for each ensuing 4-month period respectively. The intervention effect in each 4-month period after the educational intervention was then compared to the baseline value. Finally we evaluated the intervention effect with regards to the quantities of ADRs in CAM and non-CAM medication. Changes in the classification and assessment behaviour of physicians in terms of the severity and causality of ADRs completed our analysis.

We used Mann Whitney U-test for testing continuous data (i.e. Number of ADRs) and chi-square test was used for categorical data (i.e. data on ADR completeness). The level of statistical significance was set at P < 0.05. Due to the pre-post study design in a small group of physicians we abstained from using additional adjusting techniques.

All analyses were performed using SPSS 15 for Windows.

Results

Physicians

From January 2004 through June 2007, a total of 38 primary care practitioners participated in the study. The average age of the physicians was 48.0 (SD ± 6.1) years and 55% were male. Of the participating physicians, 55% were general practitioners (GPs) and 45% were specialized GPs (23% paediatrics, 11% internal medicine, 11% other).

Adverse drug reactions

A total of 404 ADRs were reported in 381 patients during the study period. Twenty-three patients had two ADRs. Altogether, 20 ADRs were classified as serious and 384 as non-serious according to ICH criteria. Of the reported ADRs, 157 were classified according to WHO-ART as grade 1; 203 as grade 2; 40 as grade 3; and four as grade 4. All 20 of the serious ADRs were associated with conventional drugs.

Number of ADR reports

The total number of ADR reports rose from 116 before to 288 after the educational intervention (P < 0.001), which corresponds to an increase of 148%. A detailed overview of reporting rates before and after the intervention is given in Figure 2. Seven physicians who reported ADRs before the intervention did not report any ADRs after it.

Figure 2

Number of adverse drug reaction (ADR) reports before and after educational intervention.

Before the educational intervention, the median monthly reporting rate was 4.00 (IQR [3.00; 7.50]). The median number of ADRs per physician was 3.00 (IQR [0.00; 10.00]). The highest pre-intervention reporting rate (14 ADRs per month) was observed in the 21st month – i.e. after physicians had received a newsletter reminding them of their upcoming training session.

After the educational intervention, the median ADR reporting rate rose to 9.00 ADRs per physician (IQR [1.50; 16.00]), and the median number of ADR reports per month significantly increased to 14.00 (IQR [9.50; 19.50], P < 0.001). The duration of the educational intervention in terms of a 4-monthly trend in total ADR reporting is shown in Figure 3. The quantity of the effect decreased from a median of 21.50 (IQR [16.25; 24.50]) in the first post-intervention period to medians of 14.00 (IQR [10.50; 16.00]), 14.50 (IQR [12.50; 20.25]) and 15.5 (IQR [11.25; 19.75]) in the next three periods. Compared to baseline the postinterventional number of ADR reportings was statistically significant higher (P < 0.005) through the first 16 months after the intervention but not significant in the last 4-month period (median: 8.00 (IQR [2.75; 8.75]; P = 0.605).

Figure 3

Duration of the educational intervention in terms of a 4-monthly trend in total ADR reporting.

A subgroup analysis of the number of ADR reports finally revealed that the increment in ADR-reporting was solely and highly significant (P < 0.001) related to ADRs of non CAM drugs. While the number of ADRs of CAM remained stable (45 pre- intervention versus 45 post- intervention), the number of non CAM ADRs increased from 71 before intervention to 243 after intervention.

Completeness and quality of ADR reports

Completeness

The completeness of the obligatory items on the ADR reports, including information on all current medication, current diagnoses with ICD 10 codes, and description of ADR increased from 80.3% before to 90.7% after the educational intervention. The largest improvement was seen for the item 'Drug': the completeness of the sub-item 'Dosage' increased from 66.3% before to 93.0% (P < 0.001) after the intervention. Similarly, the completeness of the sub-item 'Date drug stopped' increased from 62.0% to 85.1% (P < 0.001). Although no improvement was seen for the sub-item 'Diagnosis name', this item was already complete on 96.6% of the ADR reports filed before the intervention. Finally, a statistically significant increase was observed in the completeness of the sub-item 'Serious or non-serious according to ICH' after the intervention (P < 0.001). Table 2 provides a more detailed overview of the changes in the completeness of these important items.

Table 2

Completeness of obligatory items on adverse drug reaction (ADR) reports

ADR Report	Before educational intervention		After educational intervention		P value
	n	%	n	%
ADR	116	100	288	100
Drug
Name of drug	110	94.8	279	96.9	0.325
Prescribed for	95	81.9	262	91.0	0.010*
Date drug started	99	85.3	280	97.2	< 0.001*
Date drug stopped	72	62.0	245	85.1	< 0.001*
Suspected of causing ADR	97	83.6	274	95.1	< 0.001*
Dosage	77	66.3	268	93.0	< 0.001*
Diagnosis
Name	112	96.6	279	96.9	0.868
ICD-10 code	111	95.6	279	96.9	0.556
ADR
Symptom	106	91.4	271	94.1	0.322
Severity according to WHO-ART	109	94.0	273	94.8	0.741
Serious or non-serious according to ICH	40	34.5	170	59.0	< 0.001*
Date ADR started	100	86.2	268	93.1	0.029*
Date ADR stopped	81	69.8	233	80.9	0.016*
Treatment of ADR completed?	96	82.8	268	93.1	0.002*
Causality	94	81.0	271	94.1	< 0.001*

*Significant (chi-square-test)

Quality

To evaluate the quality of ADR reports, interrater reliability between the physicians' and the study centre's classification of ADRs according to ICH criteria, as well as between the physicians' and the study centre's assessment of causality according to UMC criteria was calculated. There were no considerable changes in the quality of classification according to ICH criteria (pre-intervention: kappa 0.84, 95% CI: 0.64, 1 vs. post-intervention: kappa 0.87, 95% CI: 0.73, 1). For the causality assessment according to UMC criteria, kappa was 0.15 (95% CI: 0.08; 0.29) before and 0.43 (95% CI: 0.23; 0.63) after the educational intervention. Table 3 provides a detailed overview of the changes in physicians' assessment of seriousness, severity, and causality according to ICH, WHO-ART, and UMC criteria, respectively. Physicians assessed causality as certain in 55.2% of the ADR reports before the intervention, but as probable in 34.0% and possible in 44.4% of the reports after the intervention. Only 17.7% of the reports filed after the intervention included a causality assessment of certain.

Table 3

Changes in quantity and quality of ADR reports

	Before educational intervention		After educational intervention
	n	%	n	%
ADR	116	100.0	288	100.0
ICH classification (P = 0.566)
Serious	8	6.9	14	4.9
Non-serious	108	93.1	274	95.1
WHO-ART classification (P = 0.015*)
Grade 1 = mild	50	43.1	107	37.2
Grade 2 = moderate	46	39.7	157	54.5
Grade 3 = severe	18	15.5	22	7.6
Grade 4 = life threatening	2	1.7	2	0.7
Causality (P < 0.001*)
Certain	64	55.2	51	17.7
Probable/likely	24	20.7	98	34.0
Possible	19	16.4	128	44.4
Unlikely	4	3.4	5	1.7
Conditional/unclassified	0	0.0	0	0.0
Unassessable/unclassifiable	5	4.3	6	2.1

*Significant (chi-square test)

In total, 57.8% of the reports before and 77.4% of the reports after the intervention involved expected ADRs.

Discussion

This prospective, multicentre study in primary care in Germany shows the effects of an educational intervention designed to improve ADR reporting in a primary care setting of CAM physicians. A significant increase both in the quantity and quality of ADR reports was observed after physicians took part in the intervention. This finding indicates that the intervention increased physicians' awareness of ADRs and that the physicians were able to transfer the knowledge they had gained from face-to-face training into their everyday clinical practice. The impact of the intervention, however, was temporary.

Physicians' enhanced awareness of ADRs was reflected by an increase in the number of ADR reports submitted immediately after they had received the first newsletter. During the 16 months following the educational intervention, the ADR reporting rate was higher than it had been before the intervention. Subsequently, however, the reporting rate began to decrease and continued to do so, on the average, throughout the remaining study period. Similar results were found by Figueiras et al, who examined the effectiveness of educational outreach visits for improving ADR reporting by physicians. The maximal effect in their study was observed during the first four months after the intervention, and the differences remained statistically significant for 12 months [8]. In a study of an educational initiative in drug safety, Bracchi et al. found that training improved the rate and quality of ADR reporting, but the effect was also of only short duration [9].

In the literature, a lack of knowledge about ADRs is often considered to be a cause of underreporting [6,24,25]. The results of the present study show that the degree to which physicians were able to put the knowledge they had gained from face-to-face training into practice was remarkably high. This is demonstrated by an increase in the completeness of ADR reports from 80.3% before to 90.7% after the intervention. Nevertheless, a need for improvement was seen for the item 'Severity according to WHO-ART', as this was complete in only 59.0% of ADR reports after the intervention.

One focus of the educational intervention was on causality assessment. Improvements in this area were observed following the intervention, as can be seen in the shift that occurred from a predominance of reports indicating certain causality to reports in which causality was judged to be probable or possible. The present study was also able to provide some insight into ADR reporting in CAM. Quite interestingly, the intervention did not result in an increment in the number of reported CAM associated ADRs but to a triplication of non CAM related ADRs. Among patients and even CAM practitioners, there appears to be a widespread belief that CAM is 'natural' and therefore safe [14], which may lead to underreporting of ADRs associated with herbal remedies [26]. The results of the present study show that none of the ADRs associated with CAM were serious. A very low rate of serious ADRs might be one explanation for the low reporting rate observed among CAM physicians.

Spontaneous reporting systems have been developed to detect serious, rare, and unexpected ADRs. The results of the present study show that the proportion of expected ADRs reported by physicians was larger after an educational intervention. Expected ADRs can be used to help assess the safety of a drug, because they are not likely to be reported to the spontaneous reporting systems.

Cosentino et al. recommend including pharmacovigilance as a topic in continuing education programmes [10]. Our data suggests that continuing education is an important tool for increasing physicians' awareness of ADRs. Based on our results and the findings of Figueras et al. [8], we would recommend a yearly repetition of such educational interventions.

To date, ADRs have been reported primarily by physicians, but nurses and pharmacists can also play an important role [27-29]. However, in a similar interventional program in pharmacists the study of Herdeiro et al. showed that educational outreach visits improved ADR reporting in terms of quantity and relevance [30].

This study has several important limitations. Firstly, although the reporting rate decreased towards the end of the study, the study period was too short to evaluate long-term effects. In future studies, it would be helpful to determine whether annual training sessions might lead to improved results. Secondly, the study did not evaluate the total number of prescriptions in relation to the total number of ADR reports. Thirdly, reporting even mild and expected ADRs requires a great deal of motivation on the part of physicians; however, factors related to motivation were not evaluated (e.g. by questionnaire). Finally our study design with a concurrent control group and a small sample size did not allow to control for external interventions on the individual level (i.e. individual courses in continuing medical education) which could possibly have influence our results.

Conclusion

In conclusion, the results of the present study demonstrate that an educational intervention can increase physician awareness of ADRs, and that physicians were able to incorporate the knowledge they gained from face-to-face training into their everyday clinical practice. The effects of the educational intervention, however, were temporary. Further research is needed to determine whether extended or continuous educational measures might lead to more durable improvements in ADR reporting rates in everyday practice. This may be particularly important considering the growing popularity of CAM.

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

MT has made substantial contributions to conception and design, acquisition of data and analysis and interpretation of data. EJ participated in the design of the study and interpretation of data. AB has given final approval of the version to be published.

CMW, SNW and HM have given final approval of the version to be published. TO performed the statistical analysis. All Authors read and approved the final manuscript.

Pre-publication history

The pre-publication history for this paper can be accessed here: