Application of pharmacovigilance methods in occupational health surveillance: comparison of seven disproportionality metrics.

OBJECTIVES: The French National Occupational Diseases Surveillance and Prevention Network (RNV3P) is a French network of occupational disease specialists, which collects, in standardised coded reports, all cases where a physician of any specialty, referred a patient to a university occupational disease centre, to establish the relation between the disease observed and occupational exposures, independently of statutory considerations related to compensation. The objective is to compare the relevance of disproportionality measures, widely used in pharmacovigilance, for the detection of potentially new disease × exposure associations in RNV3P database (by analogy with the detection of potentially new health event × drug associations in the spontaneous reporting databases from pharmacovigilance).
METHODS: 2001-2009 data from RNV3P are used (81,132 observations leading to 11,627 disease × exposure associations). The structure of RNV3P database is compared with the ones of pharmacovigilance databases. Seven disproportionality metrics are tested and their results, notably in terms of ranking the disease × exposure associations, are compared.
RESULTS: RNV3P and pharmacovigilance databases showed similar structure. Frequentist methods (proportional reporting ratio [PRR], reporting odds ratio [ROR]) and a Bayesian one (known as BCPNN for "Bayesian Confidence Propagation Neural Network") show a rather similar behaviour on our data, conversely to other methods (as Poisson). Finally the PRR method was chosen, because more complex methods did not show a greater value with the RNV3P data. Accordingly, a procedure for detecting signals with PRR method, automatic triage for exclusion of associations already known, and then investigating these signals is suggested.
CONCLUSION: This procedure may be seen as a first step of hypothesis generation before launching epidemiological and/or experimental studies.

Introduction

The European Union Scientific Committee on Emerging and Newly Identified Health Risks (SCENHIR) stated recently "There are a number of reasons why an emerging issue was not identified at an appropriate time or its potential effects were not properly considered." [1]. The two first - among the 8 "reasons of past failure" identified - are "Inadequate monitoring/surveillance resulting in a failure to detect the presence of a disease and/or agent at an early stage" and "Failure to make important relevant information available to the risk assessors/risk managers".

In the occupational health field, occupational registers from insurances systems, because of their primary vocation of compensation, are devoted to already known "disease × exposure (D × E)" associations and are not supposed to report and detect new D × E associations. That's the reason why occupational diseases (OD) surveillance schemes have been set out by university physicians and researchers specialists of OD, with the ambition to take part efficiently to the occupational health surveillance. Some have developed a systematic collection of all OD cases by the mean of large samples of physicians from different specialties reporting continuously or periodically (surveillance action), as the Health and Occupation Research Network system does in UK [2-5]. Some rely on a highly qualitative approach consisting of the analysis by OD specialists (expert function) of the work attributability (imputability) of disease cases whose work-relatedness has raised questions for physicians (vigilance action).

"The French National Occupational Diseases Surveillance and Prevention Network (RNV3P)" is a French network of academic specialists of ODs, which belongs to the last category. Indeed, RNV3P collects since 2001 in its database, in standardised coded reports, all cases where a physician of any specialty, and throughout metropolitan France, referred a patient to a university OD centre, to establish the relation between the disease observed and one or several occupational exposures, independently of statutory considerations related to compensation [6-8].

One of the research aims of RNV3P is to improve its ability for early detection of new ODs, especially find out information that could be useful to achieve this objective within its growing database. In order to achieve this aim, methods for detecting emerging diseases that are currently used in occupational health and by vigilance systems in other healthcare fields were first analysed [9]. Most of the data mining methods used in health surveillance (especially syndromic surveillance) aimed at identifying unusual increases in the number of cases within chronological data [10-12]. Beside these methods, we identified the "disproportionality metrics" used in pharmacovigilance [13,14] as most relevant to our aim of revealing currently unknown or poorly-documented D × E associations ("Disease" might stand for "health effect" or "health event", and "exposure" for "drug" in the pharmacovigilance context). Some disproportionality metrics are frequency-based methods (chi2, proportional reporting ratio [PRR] which is used in the British pharmacovigilance database, reporting odds ratio [ROR], Yules' Q), some are based on Poisson's law (Poisson and Sequential Probability Ratio Test [SPRT2]), and some rely on Bayesian methods (as Bayesian Confidence Propagation Neural Network [BCPNN], which is used in the international pharmacovigilance database).

In pharmacovigilance, these methods are dedicated to "hypothesis generation", also called "signal generation", where signal stand for "reported information on a possible relationship between an adverse event and a drug, of which the relationship is unknown or incompletely documented previously" (World Health Organization [WHO] definition). These "Safety Data Mining" methods have shown high potentialities in the analysis of the very large pharmacovigilance spontaneous reporting databases (e.g., more than 250,000 reports annually for the Uppsala Monitoring Centre [UMC] "WHO" database). Retrospective studies have shown that the utilisation of these methods 1) confirmed signals that had been clinically first identified, 2) might highlight these new associations sooner, 3) might distinguish a specific adverse drug effect of a molecule, not shared by its whole therapeutic family [15]. Conversely we may also remark that they are few examples of new drug × health effect that have been first highlighted by these methods prospectively, probably because, these methods have not been used for such a long time, but mainly because the analysis of numerous signals is also time consuming for experts.

We first tested the PRR method on a RNV3P sample few years ago [16], and the interest of PRR in the RNV3P was also highlighted in a second publication with twice more data and the example of systemic sclerosis [8].

The objectives of this paper are to present the comparison of the seven previously cited pharmacovigilance methods tested on RNV3P, in order to choose the most promising method(s), and then propose a procedure for an utilisation of these method(s) on a routine basis in the RNV3P network.

Materials and Methods

Data

RNV3P data from 2001-2009 were analysed, including a total of 81,132 observations. The main codes used are International Classification of Diseases 10th Revision (ICD-10), International Standard Classification of Occupations (ISCO 88), Nomenclature d'Activités Françaises for activity sector (NAF 93), and a French code for exposures from the French national insurance system of salaried workers (CNAM-TS), at the origin of the European Occupational Diseases Statistics Classification of the causal agents of the ODs (EODS causal agents code). This last code may be downloaded on the following EU website: http://circa.europa.eu/Public/irc/dsis/hasaw/library?l=/occupational_statisstics/working_paper_18/_EN_1.0_

One key point of RNV3P data is that 75% of the observations notify only one exposure (17% with 2 exposures, 5% with 3, 2% with 4, < 1% with 5 exposures notified).

Methods

Pharmacovigilance methods are detailed in several publications, where especially formula and signal generation criteria may be found [17,18]. At the first level, each report generates n "D × E associations", n being the number of exposures notified in this report notifying one disease. At the aggregated level (the whole database being taken into account), the observed number of cases of each "D × E association" is compared with the estimation of the expected number of cases. This expected number of cases is calculated using data from the database only, reason why these methods are also named "numerator dependent". Pharmacovigilance methods generate a statistical signal when there is a discrepancy between the observed number of a D × E association within the database and the expected number of cases. Several methods are proposed to estimate the magnitude of the disproportion measure and its confidence interval. All are derived from a basic 2 × 2 contingence table generated for each D × E pair [16]. In other words, these methods are using the "background noise" summarized in the marginal counts of these 2 × 2 contingency tables.

Seven disproportionality metrics were applied and their results compared: the usual, frequency-based methods (n°1: PRR which is used in the British pharmacovigilance database, n°2: ROR, n°3: Yules, n°4: chi2), methods based on Poisson's law (n°5: Poisson and n°6 : SPRT2), as well as one of the two Bayesian methods that have been described to date (n°7: BCPNN), which is used by the WHO collaborative centre on pharmacovigilance (UMC) on the international pharmacovigilance database. Only the Empirical Bayes Screening (EBS) method, also named "Dumouchel method", was not tested, as it was more complex to implement.

Another point that should be kept in mind is that the same method may be used with different signal generation criteria. For instance there are two common PRR signal generation criteria for PRR method. The first one is a composite criterion requiring that the number of observed cases (a) is at least equal to 3, and that PRR and chi2 measures for this association are at least equal to 2 and 4 respectively: a ≥ 3 and PRR ≥ 2 and chi2 ≥ 4 (named "PRR1" in Table 1). The second is that the lower bound of its 95% confidence interval has to exceeded one: LI95% (PRR) > 1 (named "PRR2" in Table 1).

Table 1

Total number of D × E associations generating signals according to the 7 disproportionality metrics applied to RNV3P database (2001-2009)

All: all disease × exposure (D × E) associations generating a signal with the disproportionality metrics tested (number or percentage), C: part of the D × E associations generating a signal that are eligible for compensation according to criteria for French salaried workers (testifying of already well known occupational diseases), NC: part of the D × E associations not eligible for compensation; it's within this group that we might find new occupational diseases, PRR1: proportinal reporting ratio (PRR) with the following signal generation criterion; a≥3 & PRR≥2 & χ2≥4, PRR2: PRR with the following signal generation criteria; LI95% (PRR)>1, RNV3P: French National Occupational Diseases Surveillance and Prevention Network.

These results are based on the definition interval of the methods, which might slightly differ. As percentages are rounded off at the unit level, the sum of the columns C and NC might sometimes differ from one unit of the percentage notified in the All column.

*46% (n=221) have been reported only twice. †77% (n=585) have been reported only twice.

The comparison of the methods can't be done with sensitivity and specificity assessment, due to the absence of a Gold Standard allowing differentiating true positives (health events with causality relation to the notified exposure) from false positives (see Discussion). For that reason, we propose a comparison of the behaviours of the methods, notably of the ranking of the disproportionality measure affected to the D × E associations.

Software

All analyses are conducted with SPlus 6.1 (Insightful Co., Seattle, WA, USA).

Results

Structure of the D × E database

The 81,132 observations lead to 11,627 "D × E" associations. The filling percentage of the theoretical "D × E" matrix is of 1.2%, which is not so different from pharmacovigilance databases (e.g., 2.1% for the US Medwatch database in 2002 [19]), or 2.2% to 3.6% for the WHO Vigibase in 2005, according to the level of information (code precision) taken into account (personal communication with UMC). Most of the cases are reported once (63% of the associations). Four thousands and two hundreds ninety one D × E associations were reported more than twice and are potentially candidate for generating a signal. The distribution of the D × E associations, according to their number of observations (Fig. 1), is similar to the one of the French pharmacovigilance database. This shows a good comparability of RNV3P D × E matrix and the health event × drug matrix of the pharmacovigilance databases, according to their sparsity ("empty matrixes") and to the distribution of the size of theses associations. The main difference is that RVN3P database is built on a smaller number of cases. The distribution is also similar for the RNV3P "disease × occupation" and "disease × activity sector" matrices, that were also studied (results not shown).

Fig. 1

"Disease × exposure (D × E)" associations according to the number of times they have been notified in the French National Occupational Diseases Surveillance and Prevention Network (i.e., number of similar observations or reports).

Comparison of the number of signals generated by the different methods

Table 1 shows the number of signals generated by the 7 methods with their usual signal generation criterion, and if these signals concern potentially compensable diseases or not (because the potentially new OD belongs to the second category, the signals belonging to the first category will not have to be investigated). In terms of number of signal generated, we can distinguish two groups of methods. PRR2, ROR, Yules, BCPNN and chi2 generate a signal for 21 to 23% of the D × E associations, and only the 12-14% not eligible for compensation would deserve investigation. This percentage rises up to 58% to 66%, if we consider only the associations reported twice or more. PRR1 and SPRT2 are more "restrictive", as respectively 11% and 8% of all associations generate a signal (6% and 3% of associations not eligible for compensation would deserve investigation). Poisson method with its common signal generation criteria generates a signal for up to 25% of all associations. Table 1 also presents a quantitative assessment of the overlap of the signals generated by each method with the signals generated by PRR1 and PRR2 taken as references.

Comparison of disproportionality metrics with regards to their respective ranking of all D × E associations

The differences between PRR2 and the other methods might be understood when comparing the respective ranking of the D × E association by the different methods. This analysis shows that the behaviour of the 7 metrics is different according to the number of cases reported for each D × E association. This is illustrated by Fig. 2, which displays the rank of each D × E association according to PRR2, as a reference (x axis), compared to the rank of this association with other methods successively (y axis), highlighting in the same time (symbols) the number of reports for each D × E association. This figure shows the slight similar behaviours of PRR2 with the other frequentist method ROR, and its derivative Yules (which rank similarly the associations). We also notice only a very slight difference between PRR2 and BCPNN: BCPNN may give a slightly superior rank to associations with a low number of associations, whereas the "frequentist" method PRR2 is more "sensitive" to the highest number of reports. To the contrary, there are big differences in terms of ranking when considering PRR in one hand, and chi2, SPRT2 or Poisson to the other. For example, chi2 ranking distribution is biphasic with regards to PRR2, as it is very sensitive to D × E associations reported a high number of times, and at the same time sensitive to associations reported a low number of times, when this number is equal or nearly equal to the marginal count of the exposure or of the disease.

Fig. 2

Comparison of the behaviour of the proportional reporting ratio (PRR) method (x axis), with the disproportionality metrics reporting odds ratio (ROR), Yules, chi2, Poisson, Sequential Probability Ratio Test (SPRT2), Bayesian Confidence Propagation Neural Network (BCPNN), according to the number of reports in each disease × exposure associations (symbols). The associations represented near the origin of the axes have the lowest disproportionality measures, whereas the ones to the opposite have the highest measures and present the strongest signals. When associations are plotted near the bisecting line, a similar rank has been affected by the 2 disproportionality metrics compared. Conversely, when some associations are plotted lower (respectively higher) than the bisecting line, it means that they have been affected lower (respectively higher) disproportionality measures by the method represented on the y axis, than by the PRR method.

Finally, the relatively similar behaviour of PRR2 and BCPNN, despite their different theoretical backgrounds, is illustrated with the example of the systemic sclerosis (ICD-10: M34). One hundred seventy eight observations reported M34 as a main Diagnosis on 2001-2009 data. These data lead to 70 associations "M34 × exposure", of which 27 associations are reported more than twice. The ranking of the BCPNN measures of these "M34 × E associations", and the overlap with the signals generated by PRR1 and PRR2 are presented (Fig. 3). Difference in signal generation between BCPNN and PRR2 only concerns one association (which is just exceeding the BCPNN signal threshold), and differences with PRR1 only concern the associations reported only twice. Furthermore, their ranking of the highest signals is similar (Fig. 4).

Fig. 3

"Systemic Scleroderma × Exposure" associations reported twice or more, their number of reports, their measures with BCPNN method, whether they generate a signal (solid triangles) or not (empty triangles), and overlap with proportional reporting ratio signals (PRR1 in blue circles and PRR2 in red squares). BCPNN: Bayesian Confidence Propagation Neural Network, LI95% IC BCPNN: lower bound of 95% confidence interval for each BCPNN measure.

Fig. 4

"Systemic Scleroderma × Exposure" associations and their proportional reporting ratio (PRR) measures (squares), whether they generate a signal with either PRR2 (over the horizontal line LI95% IC PRR>1) or PPR1 (blue circles), and overlap with BCPNN signals (solid triangles). LI95% IC PRR: lower bound of 95% confidence interval for each PRR measure.

Discussion

In this work, 7 disproportionality metrics used in pharmacovigilance to highlight potential new "drugs × health events" associations were tested on RNV3P database to assess their relevance for detecting new "exposure × health events". Their results, in terms of number of signals generated and signal ranking were compared, and an illustration was given with the disease Systemic Scleroderma. This was a first step before choosing the most promising method(s), and proposing a procedure for an utilisation of these method(s) on a routine basis in the RNV3P network.

Choice of a method in the light of the comparisons made between these disproportionality metrics in this work and in the literature

Considering the above mentioned comparisons, and many examples accurately studied (notably the temporal trend of the signal), the PRR method was finally chosen for integration in a wider procedure of screening of RNV3P database, because more complex methods did not show a greater value with the RNV3P data. As discussed in the literature, the price of sophistication is these methods is the "increased cost of lack of transparency" [15], which seems so far not justified in our context.

There are few data in the literature about comparison of these methods, because, in the absence of a Gold Standard allowing differentiating true positives (health events with causality relation) from false positives, it is difficult to calculate their respective sensitivity and specificity. That's why, the estimations of sensitivity and specificity often rely on simulated data [17]. When considering all the D × E association generating a signal with their simulated data, the authors did not show many differences in specificity according to the methods, and slight differences in terms of sensitivity. When taking into account the ranking of all signals, they showed a less rate of false positives of BCPNN and EBS, but this superiority to frequentist methods disappeared soon when increasing the number of the most highly ranked combination taken into account, and when the number of observed cases is increasing (no more differences for associations reported at least 5 times). Finally they showed that all methods had a better performance, when ranking was done on the lower bound of their confidence interval that takes into account the variance of the disproportionality measure. We should nevertheless remember that these estimations of sensitivity and specificity remain relatives to the structure of the dataset they were tested on. That's the reason why, on other studies, PRR may have shown superiority in some contexts [20]. As Hauben and Bate [15] said, "Judicious implementation of all the methods gives comparable results and far greater variation in performance is seen owing to heterogeneity in implementation choices, such as threshold election/titration and the triage logic and procedures for investigation of signals". In the light of this comment, we could analyse the added value, for our purpose of some sophistications as the weighting of some D × E associations to allow them generate a signal sooner if they appear (e.g., if there has been suspicion due to experimental toxicological data, Quantitative Structure Activity Relationships [QSARs], or even clinical sentinel approach). Conversely, stratification according to age, sex or other variables is not yet relevant, as the number of cases per associations is already low (it would prevent many D × E associations to reach statistical significance).

Proposed procedure for the routine use in RNV3P

Accordingly, a procedure for detecting signals with PRR method, and then investigating these signals has been suggested: 1) detection of signals using the PRR method; 2) automatic sorting and elimination of signals generated that can lead to recognition of an OD as they reflect known associations; 3) investigation of each as yet unknown D × E association generating a signal, by investigating data within the database (number and source of cases, changes over time of the signal, distribution of the attributabilities assessed by the OD specialists, etc); 4) analysis of data in the literature using an algorithm explaining the level of proof from, on one hand human data (epidemiology and case reports), and to the other hand experimental data (toxicology), and using Bradford-Hill causality criteria. This procedure should be used in the next future on a routine basis, as soon as the new RNV3P information system-currently tested-will allow welcoming a complementary analysis module.

Interest and limits of this procedure

We think this procedure including disproportionality metrics represent an interesting supplemental quantitative tool that can be helpful to highlight some potentially emerging D × E associations which should require attention and specific investigations (cf. examples listed in the previous publications [8,16]).

Yet, it is important to remember that these methods are only the first step in a more comprehensive process, which requires evaluation of the relevance of the signals generated, and monitoring of such signals ("signal strengthening", "signal follow-up"). These methods are not able to demonstrate causality. They may be seen as a first step of hypothesis generation before launching epidemiological and/or experimental studies. For that reason, if these methods free us from time to screen "manually" the RNV3P database in the search of new associations not visible from usual statistic analyses, in turn they require from us much time to analyse the signals generated, and exclude the obvious false positive. Another limit is that these methods do not take into account the effects of multi-exposure. That's a reason why, we also develop methods that will take into account this important point [21].

Finally, we should not forget that to discover potentially new D × E associations, the first and most important point is that our scheme continuously increases its ability to capture those cases (i.e., that our "vigie/vigilance" action is well-known from a majority of physicians who would refer to us these cases), and that our coding allow information to be of high quality and accurate (dynamic evolution of the codes, homogeneity of coding practices, etc).