Impact of time-varying exposure on estimated effects in observational studies using routinely collected data: protocol for a cross-sectional study.

INTRODUCTION: Time-varying exposure is an important issue that should be addressed in longitudinal observational studies using routinely collected data (RCD) for drug treatment effects. How well investigators designed, analysed and reported time-varying exposure, and to what extent the divergence that can be observed between different methods used for handling time-varying exposure in these studies remains uncertain. We will conduct a cross-sectional study to comprehensively address this question. METHODS AND ANALYSIS: We have developed a comprehensive search strategy to identify all studies exploring drug treatment effects including both effectiveness and safety that used RCD and were published in core journals between 2018 and 2020. We will collect information regarding general study characteristics, data source profile, methods for handling time-varying exposure, results and the interpretation of findings from each eligibility. Paired reviewers will screen and extract data, resolving disagreements through discussion. We will describe the characteristics of included studies, and summarise the method used for handling time-varying exposure in primary analysis and sensitivity analysis. We will also compare the divergence between different approaches for handling time-varying exposure using ratio of risk ratios. ETHICS AND DISSEMINATION: No ethical approval is required because the data we will use do not include individual patient data. Findings will be disseminated through peer-reviewed publications. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This is the first study to systematically investigate the methodological and reporting quality regarding time-varying exposure, and to assess the impact of different approach on estimated treatment effects in observational studies of drug treatment effects using routinely collected data.

Only including reports published in core clinical journals between 2018 and 2020 may yield less generalisable findings.

Introduction

Investigators increasingly use routinely collected data (RCD) to evaluate drug treatment effects and support drug regulatory decisions.1 2 Such data include a broad array of data sources including electronic medical records (EMR), healthcare claims, administrative registries, epidemiological surveillance and monitoring of smart devices.3–5 Over the past decades, reports using RCD have proliferated.4–7 However, growing concerns have arisen regarding studies using RCD for evaluating drug treatment effects, particularly their methodological rigour and credibility of the results.8 9

As an important issue in longitudinal observational studies, treatment effect estimates are often confounded by time-varying exposure in routine practice. Time-varying exposure, including treatment discontinuing, switching and adding-on, are common in routine care.10–12 For instance, a retrospective cohort study included 16 351 patients with atrial fibrillation showed that 38.6% patients involved treatment switching during 1-year follow-up.13 Another study evaluated treatment patterns in patients with plaque psoriasis, and showed that among patients treated with ixekizumab, 57.7% patients discontinued treatment and 30.2% switched to alternative biologic over the 2-year follow-up.14

In the sophisticated data environment, selecting an optimal method to handle time-varying exposure are important. Inappropriate design and analytical approaches for time-varying exposure are often prone to biases and result in biased estimates.10 Using time‐varying analytical strategy, such as marginal structural model (MSM) with inverse probability weight (IPW), G-computation and structural nested models may be available approaches to handle time-varying variables.10 15 However, many previously published studies either ignore time-varying exposure, or excluding or censoring patients with time-varying exposure. In most cases, time-varying exposure is not a random mechanism, the reasons for time-varying exposure are often related to health events, such as disease progression, failure of therapeutic effects, adverse events. Excluding patients based on information during follow-up is potential to introduce selection bias, such as immortal time bias.16 17 In addition, using statistical models without considering time-varying variables may also result in biased treatment effect estimates.18 Studies showed that 11% results from marginal structural models were different from conventional models.18

Up to now, no study has systematically examined the issues related to the time-varying exposure in RCD studies for exploring drug treatment effects. Therefore, we will undertake a systematic literature survey of recently published studies to investigate the following: (1) how well did the investigators report time-varying exposure; (2) what are the study designs and analytical methods that investigators used for handling time-varying exposure; (3) whether the treatment effect estimates were consistent when using alternative methods for handling time-varying exposure, and to what extent do time-varying exposure may impact the effect estimates.

Methods and analysis

Overview of study design

This paper describes the protocol of a cross-sectional survey to investigate the reporting, methods and inferences of time-varying exposure in published observational studies that used typical RCDs to evaluate drug treatment effects, including effectiveness, safety or both. In this paper, time-varying exposure refers to treatment discontinuing, switching and adding-on during follow-up. We defined treatment discontinuation as individuals stopping treatment of interest during follow-up; treatment switching as individuals changing from the exposure group to the control group, or vice versa; treatment add-on as individuals receiving treatments from both exposure and control group (figure 1).

Figure 1

Summarises the scenario regarding time-varying exposure. (The solid lines represent that patients were exposed to drugs, while the dotted line represent that patients discontinued exposure. In the scenario above, patient A and patient B experienced treatment adding-on, patient C and patient D experienced treatment switching, patient E and patient F experienced treatment discontinuation.)

Through several internal group discussions and iterations with external experts, we have defined the criteria for including a study report and the strategy for searching reports from PubMed. In order to ensure study quality, teams of paired investigators (QRL, YXH, XZ, YJJ) will perform title and abstract screening and undertake full-text screening. Whereafter, methods-trained investigators (WW, ML, QH, JYX, MQW, YQX) will perform data collection in duplicate and independently. For all the study process, discrepancies will be addressed through discussion, or adjudication by a third reviewer (XS). Before data screening begins, we will also randomly select 10% citations for calibration exercises to ensure consistency among reviewers. For challenging items, thorough instructions will be developed after discussion and expert consultation. The study is planned to start in April 2022 and is expected to complete in October 2022.

Eligibility criteria

We will include a study if it meets all of the following:

An original study that exclusively used RCD to evaluate drug treatment effects on humans, including effectiveness and/or safety.

The study used a design that allows estimation of the effect of at least one individual drug.

Published in English.

We will exclude a report if it meets any of the following:

Unable to confirm, from the full reports, if the data used were collected for routine practice or for research purposes.

Exclusively evaluated treatment effect of complementary medicines.

Study primarily addressed the following questions: the incidence or prevalence of diseases, disease burden or risk factors.

No restrictions are applied to characteristics of study participants. We define a drug as pharmaceutical agent and biological products including therapeutic proteins, and monoclonal antibodies.19 We define RCD as those data that are generated and collected in the course of healthcare delivery without a priori research purpose. The definitions of registry vary across regions, organisations and institutions, and the opinion as to whether registries fall into an RCD category is inconsistent. For this study, we consider registries that serve for administrative purposes (eg, death or birth registry) as part of RCDs. We do not consider registries which—at least partially—actively collected data based on a study protocol or a plan with research purpose.3

Literature search

We have developed search strategy with the assistance of an information expert (LH), combining both MeSH (Medical Subject Headings) terms and free words. The key terms included routinely collected data, administrative claims data and EMRs. We also integrated the search strategy for electronic health records into our search. The search strategy for electronic health record was developed by the National Library of Medicine,20 and has been peer-reviewed by information specialist.21 22 Online supplemental appendix 1 presents the details of the search strategy. We will search PubMed in core journals for studies published between 1 January 2018 and 31 December 2020. We will use Abridged Index Medicus list to search core clinical journals in PubMed. The list included 118 journals in 2020, and covered all specialties of clinical medicine and public health.

Data collection

From each eligible study, we will collect information regarding general study characteristics, database characteristics, methods used for handling time-varying exposure and the interpretation of findings.

General characteristics

We will extract information on whether there is a protocol as stated by the investigators (yes vs no), whether the protocol was published (yes vs no), whether the study was registered, endorsement of RECORD (yes vs no), total number of participants, medical specialty (ie, dermatology, endocrinology, haematology, neurology, cardiology, respiratory, gastro intestinal, renal, rheumatology, infectious disease, oncology, intensive care, mental health), funding sources (ie, governmental or research organisational not for profit, private for profit, not funded or not reported), involvement of a methodologist (eg, epidemiologist, statistician, information expert), type of primary study design, type of outcomes reported (exclusively effectiveness outcomes, exclusively safety outcomes or both), specification of a primary outcome (yes vs no), type of primary outcome (safety or effectiveness).

We will determine if the primary outcome is a measure for drug effectiveness or safety using the following strategies: (1) if the investigators clearly specified the type of outcome (ie, effectiveness vs safety) or clearly stated the hypothesis, we will use investigators’ statement; (2) if no clear statement was made, we will make a judgement based on the putative relationship described in the rationale or the discussion section.

For each article, we will only record the primary comparison and primary outcome. The primary comparison will be selected according to the strategy: (1) if the study specified a primary comparison, we will select it as the primary comparison; (2) if the study did not specify a primary comparison, we will select the first reported comparison in the results section. We will select primary outcome according to previously published rules:23–25 (1) if a study clearly specified a primary outcome, we will use that outcome as the primary outcome; (2) if no primary outcome was pre-specified or more than one primary outcome was pre-specified, we will choose the first reported outcome in the part of methods; otherwise, the first reported outcome in the results section.

Database characteristics

We will extract information regarding data source characteristics from full texts of included studies. We will document information regarding types of primary databases (ie, EMR, claims data or both), database linkage, data source coverage (ie, international, national, regional or single centre), population coverage (number of participants included in the database), geographical region and time span of data source. We will document information regarding database characteristics according to the description of data source of included studies.

Time-varying exposure

We will record information on whether the author apply new user design for both the exposure and comparator, whether the author handle time-varying exposure during follow-up, what type of time-varying exposure did the author handle, how did the author define time-varying exposure, whether any time-varying statistical model was used to handle time-varying exposure (ie, time-dependent Cox model, MSM with IPW, structural nested failure time model) for primary analysis. We will also document the numbers of patients with time-varying exposure, whether alternative methods were used for handling time-varying exposure. We will document the results of primary analysis and all sensitivity analyses using alternate methods for time-varying exposure.

If authors clearly specified a primary analysis, we will use the primary analysis as reported. If there was no prespecified primary analysis, we will select the first reported analysis.

Interpretation of findings

We will record whether the claim of effect on primary outcome was consistent with predefined hypothesis and other outcomes, and whether authors provided external evidence (eg, evidence from external randomised controlled trials (RCTs)) or supportive rationale. We will record whether the authors noted potential bias introduced by time-varying exposure and how these potential biases could affect acceptance or rejection of null hypothesis.

Data analyses

We will qualitatively describe general characteristics, database characteristics and methods for time-varying exposures of included studies. We will summarise type of study design, type of agents of interest, type of primary outcome, area of diseases involved in the study and total number of participants included in the analysis. For database characteristics, we will summarise type of data sources used for analysis; data linkage across databases; time span of databases used for analysis; proportion of studies that used data sources from single centre, multicentre, national centre or international centre; and population coverage of data sources.

We will also summarise the type of time-varying exposure, the proportions of studies applying a new user design, proportions of studies handling time-varying exposure, method used for handling time-varying exposure, the numbers of patients with time-varying exposure, sensitivity analysis using alternative methods for time-varying exposure and the percentage of studies reporting the inconsistency results between sensitivity analysis and primary analysis. We will use numbers (percentages) for categorical variables, and mean (SD) or median (IQR) for continuous variables.

For studies that the primary outcome was binary clinical outcome and used alternative methods for handling time-varying exposure, we will additionally conduct the following analyses:

Compare the differences between sensitivity analysis and primary analysis.

Compare the differences between different conventional statistical model among studies using alternative conventional statistical model for handling time-varying exposure.

Compare the differences between different time-varying statistical model among studies using alternative time-varying statistical model for handling time-varying exposure (ie, time-varying Cox model vs MSM with IPW).

Compare the differences between conventional statistical model and time-varying statistical model among studies using both time-varying statistical model and conventional statistical model for handling time-varying exposure.

We will use ratio of risk ratios to compare the differences. Ratio of risk ratios will be measured as risk ratios of the primary analysis divided by risk ratios of alternative analyses. We will also pool ratios of risk ratios for the above comparison using random effects meta-analyses, respectively. In addition, we will compare effect estimates of RCD for outcome ascertainment with RCT with same research question. For studies exclusively reported OR, we will use the following formula to transform an OR to a risk ratio:26

RR=(OR÷(1−ACR×(1−OR)).

where RR=risk ratio, OR=odds ratio, ACR=assumed control risk.

If the study performed more than one alternative methods for time-varying exposure, we will choose the first outcome reported in the part of methods; otherwise, the first outcome reported in the results section. If more than one RCT were involved, we will pool risk ratios using random effects meta-analyses. All statistical analyses will be conducted in Stata/MP (V.16.0).

Patient and public involvement

No patient involved.

Discussion

For many years, observational studies that used RCD for exploring drug treatment effects have received substantial attentions, especially for the challenges about the credibility of findings from such studies. As a common issue in longitudinal observational studies using RCD, inappropriate approach for handling time-varying exposure may result in biased treatment effect estimates. The extent to which time-varying exposure may impact the treatment effect estimates are largely depends on the proportion of patients with time-varying exposure, and how the investigators design and analysis. Our study is specifically designed to thoroughly examine the design, analysis and reporting of time-varying exposure among studies that used RCD for exploring drug treatment effects. The resulting findings would support the development of recommendations for better handling time-varying exposure in such studies.

Strengths and limitations

Our study has some strengths. First, we will use rigorous methods to systematically identify eligible studies. Second, we will use standardised, pilot-tested forms developed by experts in pharmacoepidemiology and routinely collected health data and statisticians. Third, teams of methods-trained reviewers will conduct calibration exercise, and thoroughly collect independently. Finally, our study will include a large number of studies, which may allow us address multiple methodological problems regarding time-varying exposure.

Our study, however, has some limitations. First, we will only include studies published in core clinical journals between 2018 and 2020. Although this restriction of the publication may yield less generalisable findings, one would not expect that the methods for handling time-varying exposure in RCD studies exploring drug treatment effects could have a significant change in a relatively short period. Second, our assessment of the methodological quality of studies may be limited due to the insufficient reporting details. However, this issue is common across all literature survey, and we will make efforts to document the methodological characteristics.

Implications

Time-varying exposure in a common issue that should be addressed in longitudinal observational studies using RCD for drug treatment effects. Most of these studies, however, either ignore or underestimated the impact of time-varying exposure on effect estimates.15 Previously studies were mainly focused on time-varying exposure in RCTs, studies evaluating time-varying exposure in longitudinal observational studies is limited. No study has systematically investigated the methodological and reporting quality regarding time-varying exposure, and assessed the impact of different approach on estimated treatment effects in observational studies of drug treatment effects using RCD.

In the effort to addressing this evidence gap, we will systematically investigate the design, analysis and reporting of the time-varying exposure in RCD studies exploring drug treatment effects. The results will provide timely and up-to-date evidence about how such studies handled and reported the time-varying exposure, and to what extent the divergence that can be observed between different methods used for time-varying exposure. The findings will facilitate recommendations on design, analysis, reporting and interpretation regarding time-varying exposure and may have potential to improve causal inference and reduce bias in such studies. These will be great interest of researchers, editors and reviewers.