Elizabeth Tipton1, Kelly Hallberg2, Larry V Hedges3, Wendy Chan4. 1. 1 Department of Human Development, Teachers College, Columbia University, New York, NY, USA. 2. 2 University of Chicago - Urban Labs, Chicago, IL, USA. 3. 3 Institute for Policy Research, Northwestern University, Evanston, IL, USA. 4. 4 Quantitative Methods Division, Graduate School of Education, University of Pennsylvania, Philadelphia, PA USA.
Abstract
BACKGROUND: Policy makers and researchers are frequently interested in understanding how effective a particular intervention may be for a specific population. One approach is to assess the degree of similarity between the sample in an experiment and the population. Another approach is to combine information from the experiment and the population to estimate the population average treatment effect (PATE). METHOD: Several methods for assessing the similarity between a sample and population currently exist as well as methods estimating the PATE. In this article, we investigate properties of six of these methods and statistics in the small sample sizes common in education research (i.e., 10-70 sites), evaluating the utility of rules of thumb developed from observational studies in the generalization case. RESULT: In small random samples, large differences between the sample and population can arise simply by chance and many of the statistics commonly used in generalization are a function of both sample size and the number of covariates being compared. The rules of thumb developed in observational studies (which are commonly applied in generalization) are much too conservative given the small sample sizes found in generalization. CONCLUSION: This article implies that sharp inferences to large populations from small experiments are difficult even with probability sampling. Features of random samples should be kept in mind when evaluating the extent to which results from experiments conducted on nonrandom samples might generalize.
BACKGROUND: Policy makers and researchers are frequently interested in understanding how effective a particular intervention may be for a specific population. One approach is to assess the degree of similarity between the sample in an experiment and the population. Another approach is to combine information from the experiment and the population to estimate the population average treatment effect (PATE). METHOD: Several methods for assessing the similarity between a sample and population currently exist as well as methods estimating the PATE. In this article, we investigate properties of six of these methods and statistics in the small sample sizes common in education research (i.e., 10-70 sites), evaluating the utility of rules of thumb developed from observational studies in the generalization case. RESULT: In small random samples, large differences between the sample and population can arise simply by chance and many of the statistics commonly used in generalization are a function of both sample size and the number of covariates being compared. The rules of thumb developed in observational studies (which are commonly applied in generalization) are much too conservative given the small sample sizes found in generalization. CONCLUSION: This article implies that sharp inferences to large populations from small experiments are difficult even with probability sampling. Features of random samples should be kept in mind when evaluating the extent to which results from experiments conducted on nonrandom samples might generalize.
Keywords:
content area; education; methodological development
Authors: Jeanette W Chung; Karl Y Bilimoria; Jonah J Stulberg; Christopher M Quinn; Larry V Hedges Journal: Health Serv Res Date: 2017-08-21 Impact factor: 3.402
Authors: Luis Fernando Garcia de Oliveira; Tácito Pessoa Souza-Junior; Juliane Jellmayer Fechio; José Alberto Fernandes Gomes-Santos; Ricardo Camões Sampaio; Cristina Vasconcelos Vardaris; Rafael Herling Lambertucci; Marcelo Paes de Barros Journal: Brain Sci Date: 2022-05-31
Authors: Rachael D Dombrowski; Alex B Hill; Bree Bode; Kathryn A G Knoff; Hadis Dastgerdizad; Noel Kulik; James Mallare; Kibibi Blount-Dorn; Winona Bynum Journal: Nutrients Date: 2022-05-20 Impact factor: 6.706
Authors: Robert Y Lee; Erin K Kross; Lois Downey; Sudiptho R Paul; Joanna Heywood; Elizabeth L Nielsen; Kelson Okimoto; Lyndia C Brumback; Susan E Merel; Ruth A Engelberg; J Randall Curtis Journal: JAMA Netw Open Date: 2022-04-01