Lihua Li1,2,3, Meaghan S Cuerden4, Bian Liu1,3,5, Salimah Shariff6, Arsh K Jain4,6, Madhu Mazumdar1,2,3. 1. Institute for Healthcare Delivery Science, Icahn School of Medicine at Mount Sinai, New York, NY, USA. 2. Department of Population Health Science and Policy, Icahn School of Medicine at Mount Sinai, New York, NY, USA. 3. Tisch Cancer Institute, Icahn School of Medicine at Mount Sinai, New York, NY, USA. 4. London Health Sciences Centre, London, Ontario, Canada. 5. Institute for Translational Epidemiology, Icahn School of Medicine at Mount Sinai, New York, NY, USA. 6. Institute for Clinical Evaluative Sciences, Toronto, Ontario, Canada.
Abstract
INTRODUCTION: Statistical methods to assess the impact of an intervention are increasingly used in clinical research settings. However, a comprehensive review of the methods geared toward practitioners is not yet available. METHODS AND MATERIALS: We provide a comprehensive review of three methods to assess the impact of an intervention: difference-in-differences (DID), segmented regression of interrupted time series (ITS), and interventional autoregressive integrated moving average (ARIMA). We also compare the methods, and provide illustration of their use through three important healthcare-related applications. RESULTS: In the first example, the DID estimate of the difference in health insurance coverage rates between expanded states and unexpanded states in the post-Medicaid expansion period compared to the pre-expansion period was 5.93 (95% CI, 3.99 to 7.89) percentage points. In the second example, a comparative segmented regression of ITS analysis showed that the mean imaging order appropriateness score in the emergency department at a tertiary care hospital exceeded that of the inpatient setting with a level change difference of 0.63 (95% CI, 0.53 to 0.73) and a trend change difference of 0.02 (95% CI, 0.01 to 0.03) after the introduction of a clinical decision support tool. In the third example, the results from an interventional ARIMA analysis show that numbers of creatinine clearance tests decreased significantly within months of the start of eGFR reporting, with a magnitude of drop equal to -0.93 (95% CI, -1.22 to -0.64) tests per 100,000 adults and a rate of drop equal to 0.97 (95% CI, 0.95 to 0.99) tests per 100,000 per adults per month. DISCUSSION: When choosing the appropriate method to model the intervention effect, it is necessary to consider the structure of the data, the study design, availability of an appropriate comparison group, sample size requirements, whether other interventions occur during the study window, and patterns in the data.
INTRODUCTION: Statistical methods to assess the impact of an intervention are increasingly used in clinical research settings. However, a comprehensive review of the methods geared toward practitioners is not yet available. METHODS AND MATERIALS: We provide a comprehensive review of three methods to assess the impact of an intervention: difference-in-differences (DID), segmented regression of interrupted time series (ITS), and interventional autoregressive integrated moving average (ARIMA). We also compare the methods, and provide illustration of their use through three important healthcare-related applications. RESULTS: In the first example, the DID estimate of the difference in health insurance coverage rates between expanded states and unexpanded states in the post-Medicaid expansion period compared to the pre-expansion period was 5.93 (95% CI, 3.99 to 7.89) percentage points. In the second example, a comparative segmented regression of ITS analysis showed that the mean imaging order appropriateness score in the emergency department at a tertiary care hospital exceeded that of the inpatient setting with a level change difference of 0.63 (95% CI, 0.53 to 0.73) and a trend change difference of 0.02 (95% CI, 0.01 to 0.03) after the introduction of a clinical decision support tool. In the third example, the results from an interventional ARIMA analysis show that numbers of creatinine clearance tests decreased significantly within months of the start of eGFR reporting, with a magnitude of drop equal to -0.93 (95% CI, -1.22 to -0.64) tests per 100,000 adults and a rate of drop equal to 0.97 (95% CI, 0.95 to 0.99) tests per 100,000 per adults per month. DISCUSSION: When choosing the appropriate method to model the intervention effect, it is necessary to consider the structure of the data, the study design, availability of an appropriate comparison group, sample size requirements, whether other interventions occur during the study window, and patterns in the data.
Authors: Arsh K Jain; Ian McLeod; Cindy Huo; Meaghan S Cuerden; Ayub Akbari; Marcello Tonelli; Carl van Walraven; Rob R Quinn; Brenda Hemmelgarn; Matt J Oliver; Ping Li; Amit X Garg Journal: Kidney Int Date: 2009-05-13 Impact factor: 10.612