Slawa Rokicki1,2, Jessica Cohen3, Günther Fink4, Joshua A Salomon3, Mary Beth Landrum5. 1. Interfaculty Initiative in Health Policy, Harvard University, Cambridge, MA. 2. Geary Institute for Public Policy, University College Dublin, Dublin, Ireland. 3. Department of Global Health and Population, Harvard T.H. Chan School of Public Health, Boston, MA. 4. Swiss Tropical and Public Health Institute and University of Basel, Basel, Switzerland. 5. Department of Health Care Policy, Harvard Medical School, Boston, MA.
Abstract
BACKGROUND: Difference-in-differences (DID) estimation has become increasingly popular as an approach to evaluate the effect of a group-level policy on individual-level outcomes. Several statistical methodologies have been proposed to correct for the within-group correlation of model errors resulting from the clustering of data. Little is known about how well these corrections perform with the often small number of groups observed in health research using longitudinal data. METHODS: First, we review the most commonly used modeling solutions in DID estimation for panel data, including generalized estimating equations (GEE), permutation tests, clustered standard errors (CSE), wild cluster bootstrapping, and aggregation. Second, we compare the empirical coverage rates and power of these methods using a Monte Carlo simulation study in scenarios in which we vary the degree of error correlation, the group size balance, and the proportion of treated groups. Third, we provide an empirical example using the Survey of Health, Ageing, and Retirement in Europe. RESULTS: When the number of groups is small, CSE are systematically biased downwards in scenarios when data are unbalanced or when there is a low proportion of treated groups. This can result in over-rejection of the null even when data are composed of up to 50 groups. Aggregation, permutation tests, bias-adjusted GEE, and wild cluster bootstrap produce coverage rates close to the nominal rate for almost all scenarios, though GEE may suffer from low power. CONCLUSIONS: In DID estimation with a small number of groups, analysis using aggregation, permutation tests, wild cluster bootstrap, or bias-adjusted GEE is recommended.
BACKGROUND: Difference-in-differences (DID) estimation has become increasingly popular as an approach to evaluate the effect of a group-level policy on individual-level outcomes. Several statistical methodologies have been proposed to correct for the within-group correlation of model errors resulting from the clustering of data. Little is known about how well these corrections perform with the often small number of groups observed in health research using longitudinal data. METHODS: First, we review the most commonly used modeling solutions in DID estimation for panel data, including generalized estimating equations (GEE), permutation tests, clustered standard errors (CSE), wild cluster bootstrapping, and aggregation. Second, we compare the empirical coverage rates and power of these methods using a Monte Carlo simulation study in scenarios in which we vary the degree of error correlation, the group size balance, and the proportion of treated groups. Third, we provide an empirical example using the Survey of Health, Ageing, and Retirement in Europe. RESULTS: When the number of groups is small, CSE are systematically biased downwards in scenarios when data are unbalanced or when there is a low proportion of treated groups. This can result in over-rejection of the null even when data are composed of up to 50 groups. Aggregation, permutation tests, bias-adjusted GEE, and wild cluster bootstrap produce coverage rates close to the nominal rate for almost all scenarios, though GEE may suffer from low power. CONCLUSIONS: In DID estimation with a small number of groups, analysis using aggregation, permutation tests, wild cluster bootstrap, or bias-adjusted GEE is recommended.
Authors: Julia Raifman; Ellen Moscoe; S Bryn Austin; Mark L Hatzenbuehler; Sandro Galea Journal: JAMA Psychiatry Date: 2018-07-01 Impact factor: 21.596
Authors: Andrew J Admon; Thomas S Valley; John Z Ayanian; Theodore J Iwashyna; Colin R Cooke; Renuka Tipirneni Journal: Med Care Date: 2019-04 Impact factor: 2.983
Authors: John A Graves; Laura A Hatfield; William Blot; Nancy L Keating; J Michael McWilliams Journal: Health Aff (Millwood) Date: 2020-01 Impact factor: 6.301
Authors: Deliana Kostova; Cynthia H Cassell; John T Redd; Desmond E Williams; Tushar Singh; Lise D Martel; Rebecca E Bunnell Journal: Health Econ Date: 2019-08-28 Impact factor: 3.046
Authors: Shehnaz Alidina; Gopal Menon; Steven J Staffa; Sakshie Alreja; David Barash; Erin Barringer; Monica Cainer; Isabelle Citron; Amanda DiMeo; Edwin Ernest; Laura Fitzgerald; Hiba Ghandour; Magdalena Gruendl; Audustino Hellar; Desmond T Jumbam; Adam Katoto; Lauren Kelly; Steve Kisakye; Salome Kuchukhidze; Tenzing Lama; William Lodge Ii; Erastus Maina; Fabian Massaga; Adelina Mazhiqi; John G Meara; Stella Mshana; Ian Nason; Chase Reynolds; Cheri Reynolds; Hannington Segirinya; Dorcas Simba; Victoria Smith; Christopher Strader; Meaghan Sydlowski; Leopold Tibyehabwa; Florian Tinuga; Alena Troxel; Mpoki Ulisubisya; John Varallo; Taylor Wurdeman; Noor Zanial; David Zurakowski; Ntuli Kapologwe; Sarah Maongezi Journal: Int J Qual Health Care Date: 2021-06-29 Impact factor: 2.038