Data management and data analysis techniques in pharmacoepidemiological studies using a pre-planned multi-database approach: a systematic literature review.

PURPOSE: To identify pharmacoepidemiological multi-database studies and to describe data management and data analysis techniques used for combining data.
METHODS: Systematic literature searches were conducted in PubMed and Embase complemented by a manual literature search. We included pharmacoepidemiological multi-database studies published from 2007 onwards that combined data for a pre-planned common analysis or quantitative synthesis. Information was retrieved about study characteristics, methods used for individual-level analyses and meta-analyses, data management and motivations for performing the study.
RESULTS: We found 3083 articles by the systematic searches and an additional 176 by the manual search. After full-text screening of 75 articles, 22 were selected for final inclusion. The number of databases used per study ranged from 2 to 17 (median = 4.0). Most studies used a cohort design (82%) instead of a case-control design (18%). Logistic regression was most often used for individual-level analyses (41%), followed by Cox regression (23%) and Poisson regression (14%). As meta-analysis method, a majority of the studies combined individual patient data (73%). Six studies performed an aggregate meta-analysis (27%), while a semi-aggregate approach was applied in three studies (14%). Information on central programming or heterogeneity assessment was missing in approximately half of the publications. Most studies were motivated by improving power (86%).
CONCLUSIONS: Pharmacoepidemiological multi-database studies are a well-powered strategy to address safety issues and have increased in popularity. To be able to correctly interpret the results of these studies, it is important to systematically report on database management and analysis techniques, including central programming and heterogeneity testing.

Introduction

The need for post‐approval surveillance of drug safety has been widely recognized for more than four decades. From the early days of pharmacoepidemiology, initiatives have been undertaken to study safety and, most recently, effectiveness of medications using routinely collected healthcare data. Over the past decade, an increasing number of studies have been performed using healthcare databases from multiple countries, regions or healthcare organizations. Using data from multiple databases offers a number of potential advantages such as increased sample size (datasets become large enough to give precise estimates of medication risks and benefits even for rare outcomes and exposures) and generalizability (when similar results are found in studies utilizing the same methodology in heterogeneous populations). Furthermore, it offers a potential for using a standardized methodological approach across data sources.

A number of large initiatives have been launched to develop methods for combining data from multiple databases and registers. In the United States (U.S.), the Observational Medical Outcomes Partnership (OMOP)1 and the Mini‐Sentinel program2 have been run since 2007 and 2008, respectively. Data are often combined using the health maintenance organization (HMO) Research Network, a consortium of 19 large healthcare delivery organizations in the U.S.3 The Canadian Network for Observational Drug Effect Studies (CNODES), a distributed network of Canadian researchers and data centers, was launched in 2011.4, 5 European initiatives that address logistical and methodological problems of conducting multi‐database studies include EU‐ADR6 and IMI‐PROTECT.7 Recently, the Asian Pharmacoepidemiology Network (AsPEN),8 a collaboration including Asian countries, was also started. Besides these large programs, a handful of smaller research projects have also combined data from several healthcare databases.9, 10, 11, 12, 13

There are various ways to combine data from several independent databases, which all have different advantages and disadvantages. A combination of aggregate results does not require sharing of individual patient data and makes optimal use of locally available data (e.g. information on confounders). However, correcting for heterogeneity between the databases may not be fully possible when combining summary estimates. A combination of individual patient data opens more space for exploring and correcting for heterogeneity; it offers an opportunity to use exactly the same definitions of exposures, outcomes, covariates, and time windows. This approach may, however, result in a compromise when relevant information is lost if not available in all databases (possibly leading to larger residual confounding).

To our knowledge, no systematic review has been performed yet to select multi‐database studies and to illuminate which methods have most frequently been used to combine data. Our aim was therefore to identify pharmacoepidemiological studies using a pre‐planned multi‐database approach and to describe data management and data analysis techniques used for combining data.

Methods

Literature search

We conducted systematic literature searches in PubMed and Embase, as well as a manual literature search, to identify relevant multi‐database observational studies. We followed the PRISMA guideline (www.prisma‐statement.org).

The following inclusion criteria were applied: (i) peer‐reviewed pharmacoepidemiological study (defined as an observational study about the safety or effectiveness of medication); (ii) published from 2007 onwards; (iii) study subjects were selected from two or more independent healthcare databases (i.e. databases covering a different study population); (iv) all databases within the study were analyzed to answer the same research question; and (v) data were combined for a pre‐planned common analysis or quantitative synthesis (i.e. data were combined either at individual patient‐level or the estimates obtained from individual databases were combined in analyses). We excluded drug utilization studies, articles that focused on the detection of adverse drug reactions (ADRs) and other pharmacovigilance studies, studies that only reported results from separate databases without providing any combined estimates, purely methodological papers (e.g. describing methods to combine data, but not actually doing so in the paper), as well as studies published in languages other than English (exclusion criteria were applied sequentially).

Systematic search strategies for PubMed and Embase were developed under the supervision of a research librarian and included text words and relevant indexing to capture pharmacoepidemiological studies satisfying the inclusion criteria. The strategies were adapted to match the structure of each database and were based on search terms that included ‘drugs’, ‘databases’, and ‘epidemiology’/‘observational studies’ (using MeSH, Emtree, and free text terms); both databases were searched from 2007 to October 2013 (see e‐tables 1 and 2 for the full search strategy in Pubmed and Embase, respectively). The date filter from 2007 onwards was applied as our pilot searches did not identify any relevant publications before 2007. We therefore believed that it would help improve the precision of the search without negative impact on its sensitivity. One author (IE) screened the titles and abstracts of the identified articles using our inclusion and exclusion criteria to select articles eligible for full‐text screening.

The electronic literature search was supplemented with a manual search (2007 to December 2013). We identified publications listed on the following research projects' websites: (i) OMOP (http://omop.fnih.org and www.omop.org); (ii) Mini‐Sentinel (www.mini‐sentinel.org); (iii) the HMO Research Network (www.hmoresearchnetwork.org); (iv) EU‐ADR (www.euadr‐project.org); (v) IMI‐PROTECT (www.imi‐protect.eu); (vi) AsPEN (www.aspennet.asia); and (vii) CNODES (www.cnodes.ca). In addition, researchers within our team were consulted to add multi‐database projects that they were aware of (and that were not identified in the automatic and website searches). The titles and abstracts were screened to determine which articles were eligible for full‐text screening.

Two authors (IE and MB) reviewed the full‐text of all potentially relevant studies to determine final inclusion.

Data extraction process

A data extraction form was designed to extract relevant data from the selected studies. Data extraction was first performed by one reviewer (MB) with subsequent quality assurance performed by the second reviewer (IE). Disagreements were resolved through discussion with a third reviewer (MA).

Analysis of study characteristics

We retrieved information about the objective of the studies, the exposure, the outcome, the study design, and the number of different databases and countries. Further, we classified studies according to the methods that were used for individual‐level analyses and meta‐analyses. Regarding the meta‐analyses, we defined three different levels of combining data:

An aggregate level approach, in which separate analyses are performed on datasets from each database and overall results (adjusted effect estimates with confidence intervals) are collected for meta‐analysis. The analyses are usually ‘database‐optimized’ in the sense that the best available data for each database are being used. This approach allows using the normal statistical techniques for meta‐analysis, including random‐effects models, to account for heterogeneity of study results. Further, meta‐regression can be used for assessing variation in effects related to covariates, which may explain some of the overall heterogeneity.

A semi‐aggregate level approach, in which stratified datasets with event counts (and for cohort studies person time) are collected from all databases for one common analysis (e.g. a distributed data network). Datasets can be stratified on outcome, exposure, and covariate patterns (age, sex, time since initiation, and selected confounders). For a cohort study, this approach employs a Poisson regression model on tables of event numbers and person time stratified by exposure and covariate patterns. For a case–control study, a logistic regression model can similarly be used to analyze frequency tables of cases and controls stratified by different covariates.

An individual level approach, in which individual patient data are collected from all databases for one common analysis. In this scenario, the information from different databases has to be made compatible with regard to definitions of exposures, outcomes, covariates, and time windows. Heterogeneity of study populations and variation in the effects both of exposure and covariates between databases may affect the results. This may be accounted for using statistical techniques correcting for overall variation within and between databases.

With regard to data management, we were interested in which data were collected centrally and whether central programming or the use of distributed common programs was mentioned in the article. Moreover, motivations for conducting the study (as stated in the article) were classified. For all questions there was a category ‘not specified’ that could be ticked. This category is only shown in the result tables if there was at least one study with missing data. Frequencies were calculated to describe the study characteristics, data analysis techniques, data management, and motivations.

Results

The PubMed and Embase searches identified 3083 publications (see Figure 1). After the screening of titles and abstracts, 44 articles were selected for a full‐text screening. The manual search identified an additional 176 publications; 22 from OMOP, 64 from Mini‐Sentinel, 34 from the HMO Research Network (based on a Pubmed search, because no articles were found on the project's website), 36 from EU‐ADR, 11 from IMI‐PROTECT, 2 from AsPEN, 2 from CNODES, and 5 from individual projects (data not shown in the flowchart). After review of titles and abstracts we selected 31 articles for a full‐text screening. Figure 1 shows that from the 75 articles selected for full‐text screening, 22 were finally selected for inclusion.

Figure 1

Flowchart of the selection of articles. * No papers were excluded because they were non‐English (there were 122 non‐English papers but they were excluded because of other exclusion criteria)

E‐table 3 gives an overview of the 22 studies included in our review.9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30 Six studies were published between 2007 and 2010, while a majority of 16 studies was published between 2011 and 2013. For seven studies, there was a method objective in addition to an overall (mostly clinical) objective.10, 13, 15, 17, 18, 23, 27 Figure 2 shows which countries contributed to the multi‐database projects. From the 22 studies, there were 14 (64%) that used at least one database from the U.S. or Canada. Half of the studies (50%) used at least one European database. Within Europe, data from Great Britain were most often used: there were nine studies (41%) that used a British database (with the Clinical Practice Research Datalink (CPRD) being the most frequently used one, i.e. five times). Further, there were five studies that used at least one Scandinavian database (23%), five studies that used a database from the Netherlands (23%), and five studies that used an Italian database (23%).

Figure 2

Map of countries involved in multi‐database studies. Legend: ‐ the numbers reflect how many times the country (at least one database) was involved in a multi‐database study (the darker the color, the higher the number of studies) ‐ stripes: country not involved in any multi‐database study

Table 1 shows that most studies addressed safety (82%) rather than effectiveness issues (23%). The design that was most often used was the cohort study design (82%) as opposed to the case–control design (18%). The number of databases used per study ranged from 2 to 17 (median = 4.0), while the number of countries involved ranged from 1 to 6 (median = 2.5). The type of exposure that was most frequently studied was medication related to the nervous system (41%), for example antidepressants and dopamine agonists. In terms of outcomes, cardiac disorders were most frequently represented (36%), followed by all‐cause mortality (23%) and nervous system disorders (18%). The code system used to identify exposure was often not specified (82%), while for the outcome this was less often the case (18%).

Table 1

Objective, design, exposure, and outcome

	Number of studies (total: n = 22)	%
Objective category
Evidence generation	20	91%
Method / feasibility study	1	5%
Dual purpose (combination of the above)	1	5%
Study type (a)
Safety	18	82%
Effectiveness	5	23%
Design category
Cohort study	18	82%
Case–control study	4	18%
Number of databases
Range	[2,17]
Mean	5.9
Median	4.0
Number of countries
Range	[1,6]
Mean	2.4
Median	2.5
Drug exposure: ATC category
N—Nervous system	9	41%
A—Alimentary tract and metabolism	4	18%
C—Cardiovascular system	3	14%
M—Musculo‐skeletal system	3	14%
B—Blood and blood forming organs	1	5%
R—Respiratory system	1	5%
S—Sensory organs	1	5%
Outcome: All‐cause mortality/MedDRA SOC (a)
Cardiac disorders	8	36%
All‐cause mortality	5	23%
Nervous system disorders	4	18%
Congenital, familial and genetic disorders	3	14%
Gastrointestinal disorders	2	9%
Musculoskeletal and connective tissue disorders	2	9%
Blood and lymphatic system disorders	1	5%
Eye disorders	1	5%
Infections and infestations	1	5%
Pregnancy, puerperium, and perinatal conditions	1	5%
Renal and urinary disorders	1	5%
Respiratory, thoracic and mediastinal disorders	1	5%
Skin and subcutaneous tissue disorders	1	5%
Drug code systems (a)
ATC	4	18%
BNF	2	9%
Not specified	18	82%
Outcome code systems (a)
ICD‐9	13	59%
ICD‐10	8	36%
ICPC	2	9%
RCD	2	9%
CCP	1	5%
CPT	1	5%
mRS	1	5%
Not specified	4	18%

(a) One study could contribute to more than one category.

Abbreviations: ATC, Anatomical Therapeutic Chemical; BNF, British National Formulary; CCP, Canadian Classification of Diagnostic, Therapeutic and Surgical Procedures; CPT, Current Procedural Terminology; ICD‐9, International Classification of Diseases—9th revision; ICD‐10, International Classification of Diseases—10th revision; ICPC, International Classification of Primary Care; mRS, modified Rankin Scale; RCD, READ CODE Classification; SOC, system organ class.

Table 2 shows that logistic regression was most frequently used for individual‐level analyses (41%), followed by Cox regression (23%) and Poisson regression (14%). As meta‐analysis method, a majority of the studies used an individual level approach (73%) (see e‐table 4 for the exact categorization). There were three studies with a semi‐aggregate approach (14%), and six studies performed an aggregate meta‐analysis (27%). For two studies, the aggregate meta‐analysis was the only meta‐analysis that was conducted (because data were collected on an aggregate level). The other four aggregate meta‐analyses were conducted in addition to an individual patient data meta‐analysis (three studies) or a semi‐aggregate meta‐analysis (one study). There were four studies that collected semi‐aggregate datasets (18%), but one of them did not use this information in a meta‐analysis (reflected in the category ‘meta‐analysis: none’). A quantitative heterogeneity assessment was conducted in almost half of the studies (45%) without one specific test being most popular.

Table 2

Data analysis techniques and data management

	Number of studies (total: n = 22)	%
Individual‐level analyses
Logistic regression	9	41%
Cox proportional hazards model	5	23%
Poisson regression	3	14%
Incidence rate / incidence rate ratio	2	9%
Prevalence / prevalence ratio	1	5%
Relative risk	1	5%
Generalized linear model regression	1	5%
Exposure–time relation
Time‐dependent exposure	14	64%
Intention to treat (ever / never)	7	32%
Cumulative exposure (dose or time)	1	5%
Confounder control
Conventional	11	50%
Propensity score	7	32%
Disease risk score	1	5%
None	3	14%
Meta‐analysis method (a)
Individual	16	73%
Semi‐aggregate	3	14%
Aggregate	6	27%
Fixed effect	4	18%
Fixed effect / random effect	2	9%
None	1	5%
Heterogeneity assessment
Quantitative test	10	45%
I‐squared	2	9%
Chi‐squared	1	5%
Cochran's Q statistic	1	5%
Interaction by data source	1	5%
Kaplan–Meier stratified by database	1	5%
Not specified	4	18%
Qualitative statements only	1	5%
Not specified	11	50%
Programming
Central (leading center)	12	55%
Decentral	0	0%
Not specified	10	45%
Data collected centrally
Individual‐based register data	16	73%
Semi‐aggregate datasets	4	18%
Aggregate results	2	9%
Distributed common programs
Yes	5	23%
Not specified	17	77%

(a) One study could contribute to more than one category.

For 12 studies, central programming was mentioned in the publication. The use of distributed common programs was mentioned for five studies; in three cases these common programs were used in a semi‐aggregate level approach, and the other two cases were related to an aggregate level approach.

A majority of the articles motivated their multi‐database study by power or mentioned power as a strength of their study (86%) (data not shown in a table). Only one article explicitly mentioned rare exposure as an argument, and three articles made mention of a rare outcome. External validity was stressed in six publications (27%), while only three articles mentioned comparing different populations or databases as an argument to perform a multi‐database project (14%).

Discussion

Our systematic literature search identified 22 pharmacoepidemiological multi‐database studies, in which data were combined for a pre‐planned common analysis or quantitative synthesis. For individual‐level analyses, logistic regression was most frequently used, followed by Cox and Poisson regression. For meta‐analyses, 16 studies combined individual patient data, while a semi‐aggregate level analysis was conducted in three studies and an aggregate level analysis in six.

It was a challenge to capture multi‐database observational studies in a systematic literature search. Usually, the clinical topic was well represented in search terms or keywords, but the use of more than one independent database was much harder to capture. There is a Mesh term called ‘Multicenter study’, but this was on the one hand too broad (e.g. also involving studies that selected their patients from two different hospitals, which was not the definition of multi‐database study we were looking for) and on the other hand missing out on relevant articles. Among studies that used more than one database, it was almost impossible to distinguish studies with pooled estimates from studies that only showed results from the separate databases, using search terms. Because of these issues, we were quite liberal in defining the search strategy and manually screened over 3000 articles, which only resulted in 44 articles that were selected of a full‐text screening. Still, we missed out on 31 articles that were found in the additional manual search.

Among the studies finally selected for inclusion, the range of different exposures and outcomes was quite broad. This indicates that the upcoming trend of performing multi‐database observational studies stretches out over the entire field of pharmacoepidemiology. Approximately half of our selected studies used at least one database from the U.S. or Canada, and this percentage was similar for European databases. This indicates that Northern America and Europe currently take an almost equal part in contributing to this relatively new field of multi‐database research.

Regarding the methods of performing such a study, we found that the combination of individual patient data was the most frequently used technique. It should however be noted that one of our inclusion criteria was that data were combined for a pre‐planned common analysis. Thereby we excluded all ‘standard’ meta‐analyses that pool estimates from different studies together on a post‐hoc basis, i.e. the approach of combining results from published literature. This latter type of meta‐analysis is frequently used in the field of clinical trials and is probably also quite common in the field of observational studies, as it can be done relatively quickly without the need of getting access to patient‐level healthcare data. If this type of meta‐analysis would have been included in our review, the proportion of aggregate meta‐analyses would therefore have been much higher.

There were three studies that compared an individual level meta‐analysis to an aggregate level (fixed or random effect) meta‐analysis, using the same data.10, 13, 18 In all three cases, the results were similar between the two approaches. Importantly, these aggregate level analyses were designed with a pre‐specified common analysis plan, i.e. the same plan as was used for the individual level analyses, but now without combining the individual patient data but pooling database‐specific estimates together. From a logistic point of view, this type of aggregate meta‐analysis could be an interesting alternative to an individual level approach, as no individual patient data have to be transferred. Other methods that do not require sharing of individual patient data involve a distributed network approach such as the EU‐ADR15 or a case‐centered logistic regression approach as described by Toh and coworkers.27, 31

It was not always clearly stated in the articles how the data were combined. Especially the technique of combining individual patient data (instead of combining semi‐aggregate or aggregate data) was not always explicitly mentioned. Most of the articles clearly had a clinical focus, and descriptions of data management were often very short or completely missing. Central programming was mentioned for 12 studies, but the actual number of studies that did so is probably higher, because this technique is very likely for the studies that combined individual patient data (16 in total). However, the use of distributed common programs was frequently mentioned for semi‐aggregate and aggregate level approaches. This is a good practice, because different ways of statistical programming may lead to heterogeneity between results from different databases. In the individual level approach, it was often not clearly described if and how definitions were kept similar between data from different sources.

Power was the most stated reason for performing a multi‐database study. Comparing different populations or databases was seldom mentioned. Many papers, especially the individual patient data studies, did not show any characteristics of patients from the separate databases. Overall, only half of the studies performed a quantitative heterogeneity assessment.

There are important strengths and limitations to this review. To our knowledge, we were the first to perform a systematic literature search about methods used in multi‐database studies. Full‐text screening was performed by two independent reviewers, and data extraction was quality checked by a second reviewer as well. Even though we performed a systematic literature search, we probably missed relevant articles, because the formulation of a suitable search strategy was not straightforward. Our searches were limited from 2007 onwards; however, in our pilot searches with no date filter applied we were not able to identify any studies published prior to 2007 that would potentially fit our inclusion and exclusion criteria. Classification into the different levels of data combining was done to the reviewers' best effort, but was sometimes based on very little information. Classification of motivations to perform the study may have been susceptible to subjective interpretation.

In conclusion, multi‐database studies are becoming more popular in observational research. We feel that there is room for improvement in making clear to the reader how data from different databases were combined; on an individual, semi‐aggregate, or aggregate level. For all scenarios, it is useful to know how definitions of exposures, outcomes, confounders, and time‐windows were kept consistent across the databases. Further, it should be explained how data management was organized, which data were collected centrally and whether central programming or distributed common programs were used. It is useful to show characteristics of patients from the separate databases to enable the reader to evaluate whether there were important differences. When combining data from different databases, the performance of heterogeneity assessments should become common practice. Even if the objective of a study is clinical rather than methodological, all this information enables a better interpretation of the results of a multi‐database study.

Conflict of interest

Marloes T. Bazelier's employment at Utrecht University is funded by the CARING project (European Community's Seventh Framework Program grant agreement number 282526).

Frank de Vries is employed by Utrecht University as a senior researcher, conducting research coordinated by The Centre for Research Methods. The Centre for Research Methods has received unrestricted funding from the Netherlands Organization for Health Research and Development (ZonMW), the Dutch Health Care Insurance Board (CVZ), the Royal Dutch Pharmacists Association (KNMP), the private–public funded Top Institute Pharma (www.tipharma.nl, includes co‐funding from universities, government, and industry), the EU Innovative Medicines Initiative (IMI), the EU 7th Framework Program (FP7), and the Dutch Ministry of Health and industry (including GlaxoSmithKline, Pfizer, and others).

Marie L. De Bruin is employed by Utrecht University as a senior researcher conducting research in collaboration with the WHO Collaborating Centre for pharmaceutical policy and regulation. This Centre receives no direct funding or donations from private parties, including pharma industry. Research funding from public–private partnerships, e.g. IMI, TI Pharma (www.tipharma.nl) is accepted under the condition that no company‐specific product or company related study is conducted. The Centre has received unrestricted research funding from public sources, e.g. Netherlands Organization for Health Research and Development (ZonMW), the Dutch Health Care Insurance Board (CVZ), EU 7th Framework Program (FP7), Dutch Medicines Evaluation Board (MEB), and Dutch Ministry of Health.

Morten Andersen participates/has participated in research projects funded by AstraZeneca, Lundbeck, Merck Sharp & Dohme, Novartis, Nycomed, and Pfizer with grants received by the institutions where he has been employed. He has personally received fees for leading and teaching pharmacoepidemiology courses arranged by Medicademy, the Danish Association for the Pharmaceutical Industry.

The upcoming trend of performing multi‐database observational studies stretches out over the entire field of pharmacoepidemiology.

Regarding the methods of performing such a study, we found that the combination of individual patient data was the most frequently used technique.

There is room for improvement in making clear to readers how data from different databases were combined (on an individual level or aggregate level), how data management was organized, and whether central programming was used.

It is useful to show characteristics of patients from the separate databases and the performance of heterogeneity assessments should become common practice.

Ethics Statement

The authors confirm to have adhered to Ethics principles during all phases of the study.

Acknowledgements

The research leading to the results of this study has received funding from the European Community's Seventh Framework Programme (FP‐7) under grant agreement number 282526, the CARING project. The funding source had no role in study design, data collection, data analysis, data interpretation, or writing of the report.

Supporting info item

Click here for additional data file.