Elizabeth A Stuart1,2,3, Anna Rhodes4. 1. 1 Department of Mental Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA. 2. 2 Department of Biostatistics, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA. 3. 3 Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA. 4. 4 Department of Sociology, Johns Hopkins University, Baltimore, MD, USA.
Abstract
BACKGROUND: Given increasing concerns about the relevance of research to policy and practice, there is growing interest in assessing and enhancing the external validity of randomized trials: determining how useful a given randomized trial is for informing a policy question for a specific target population. OBJECTIVES: This article highlights recent advances in assessing and enhancing external validity, with a focus on the data needed to make ex post statistical adjustments to enhance the applicability of experimental findings to populations potentially different from their study sample. RESEARCH DESIGN: We use a case study to illustrate how to generalize treatment effect estimates from a randomized trial sample to a target population, in particular comparing the sample of children in a randomized trial of a supplemental programfor Head Start centers (the Research-Based, Developmentally Informed study) to the national population of children eligible for Head Start, as represented in the Head Start Impact Study. RESULTS: For this case study, common data elements between the trial sample and population were limited, making reliable generalization from the trial sample to the population challenging. CONCLUSIONS: To answer important questions about external validity, more publicly available data are needed. In addition, future studies should make an effort to collect measures similar to those in other data sets. Measure comparability between population data sets and randomized trials that use samples of convenience will greatly enhance the range of research and policy relevant questions that can be answered.
RCT Entities:
BACKGROUND: Given increasing concerns about the relevance of research to policy and practice, there is growing interest in assessing and enhancing the external validity of randomized trials: determining how useful a given randomized trial is for informing a policy question for a specific target population. OBJECTIVES: This article highlights recent advances in assessing and enhancing external validity, with a focus on the data needed to make ex post statistical adjustments to enhance the applicability of experimental findings to populations potentially different from their study sample. RESEARCH DESIGN: We use a case study to illustrate how to generalize treatment effect estimates from a randomized trial sample to a target population, in particular comparing the sample of children in a randomized trial of a supplemental program for Head Start centers (the Research-Based, Developmentally Informed study) to the national population of children eligible for Head Start, as represented in the Head Start Impact Study. RESULTS: For this case study, common data elements between the trial sample and population were limited, making reliable generalization from the trial sample to the population challenging. CONCLUSIONS: To answer important questions about external validity, more publicly available data are needed. In addition, future studies should make an effort to collect measures similar to those in other data sets. Measure comparability between population data sets and randomized trials that use samples of convenience will greatly enhance the range of research and policy relevant questions that can be answered.
Authors: Carol M Hamilton; Lisa C Strader; Joseph G Pratt; Deborah Maiese; Tabitha Hendershot; Richard K Kwok; Jane A Hammond; Wayne Huggins; Dean Jackman; Huaqin Pan; Destiney S Nettles; Terri H Beaty; Lindsay A Farrer; Peter Kraft; Mary L Marazita; Jose M Ordovas; Carlos N Pato; Margaret R Spitz; Diane Wagener; Michelle Williams; Heather A Junkins; William R Harlan; Erin M Ramos; Jonathan Haines Journal: Am J Epidemiol Date: 2011-07-11 Impact factor: 4.897
Authors: Benjamin Ackerman; Ian Schmid; Kara E Rudolph; Marissa J Seamans; Ryoko Susukida; Ramin Mojtabai; Elizabeth A Stuart Journal: Addict Behav Date: 2018-10-25 Impact factor: 3.913
Authors: Benjamin Ackerman; Catherine R Lesko; Juned Siddique; Ryoko Susukida; Elizabeth A Stuart Journal: Stat Med Date: 2020-11-26 Impact factor: 2.373