Nicholas J Parr1, Maria L Schweer-Collins2, Todd M Darlington3, Emily E Tanner-Smith2. 1. Department of Counseling Psychology and Human Services, University of Oregon, 5251 University of Oregon, Eugene, OR, 97403, USA; Prevention Science Institute, University of Oregon. 6217 University of Oregon, Eugene, OR, 97403, USA. Electronic address: nparr2@uoregon.edu. 2. Department of Counseling Psychology and Human Services, University of Oregon, 5251 University of Oregon, Eugene, OR, 97403, USA; Prevention Science Institute, University of Oregon. 6217 University of Oregon, Eugene, OR, 97403, USA. 3. Prevention Science Institute, University of Oregon. 6217 University of Oregon, Eugene, OR, 97403, USA; Department of Psychology, University of Oregon. 1227 University of Oregon, Eugene, OR, 97403, USA.
Abstract
INTRODUCTION: In the field of adolescent development, meta-analysis offers valuable tools for synthesizing and assessing cumulative research evidence on the effectiveness of programs, practices, and policies intended to promote healthy adolescent development. When examining the impact of a program implemented across multiple primary studies, variation is often observed in the methodological attributes of those primary studies, such as their implementation methods, program components, participant characteristics, outcome measurement, and the systems in which programs are deployed. Differences in methodological attributes of primary studies represented in a meta-analysis, referred to as complexity, can yield variation in true effects across primary studies, which is described as heterogeneity. METHODS: We discuss heterogeneity as a parameter of interest in meta-analysis, introducing and demonstrating both graphical and statistical methods for evaluating the magnitude and impact of heterogeneity. We discuss approaches for presenting characteristics of heterogeneity in meta-analytic findings, and methods for identifying and statistically controlling for aspects of methodological complexity that may contribute to variation in effects across primary studies. RESULTS: Topics and methods related to assessing and explaining heterogeneity were contextualized in the field of adolescent development using a sample of primary studies from a large meta-analysis examining the effectiveness of brief alcohol interventions for youth. We highlighted approaches currently underutilized in the field and provided R code for key methods to broaden their use. CONCLUSIONS: By discussing various heterogeneity statistics, visualizations, and explanatory methods, this article provides the applied developmental researcher a foundational understanding of complexity and heterogeneity in meta-analysis.
INTRODUCTION: In the field of adolescent development, meta-analysis offers valuable tools for synthesizing and assessing cumulative research evidence on the effectiveness of programs, practices, and policies intended to promote healthy adolescent development. When examining the impact of a program implemented across multiple primary studies, variation is often observed in the methodological attributes of those primary studies, such as their implementation methods, program components, participant characteristics, outcome measurement, and the systems in which programs are deployed. Differences in methodological attributes of primary studies represented in a meta-analysis, referred to as complexity, can yield variation in true effects across primary studies, which is described as heterogeneity. METHODS: We discuss heterogeneity as a parameter of interest in meta-analysis, introducing and demonstrating both graphical and statistical methods for evaluating the magnitude and impact of heterogeneity. We discuss approaches for presenting characteristics of heterogeneity in meta-analytic findings, and methods for identifying and statistically controlling for aspects of methodological complexity that may contribute to variation in effects across primary studies. RESULTS: Topics and methods related to assessing and explaining heterogeneity were contextualized in the field of adolescent development using a sample of primary studies from a large meta-analysis examining the effectiveness of brief alcohol interventions for youth. We highlighted approaches currently underutilized in the field and provided R code for key methods to broaden their use. CONCLUSIONS: By discussing various heterogeneity statistics, visualizations, and explanatory methods, this article provides the applied developmental researcher a foundational understanding of complexity and heterogeneity in meta-analysis.