Physical activity interventions in Latin America: expanding and classifying the evidence.

CONTEXT: Systematic reviews of public health interventions are useful for identifying effective strategies for informing policy and practice. The goals of this review were to (1) update a previous systematic review of physical activity interventions in Latin America which found that only school-based physical education had sufficient evidence to recommend widespread adoption; (2) assess the reporting of external validity elements; and (3) develop and apply an evidence typology for classifying interventions. EVIDENCE ACQUISITION: In 2010-2011, community-level, physical activity intervention studies from Latin America were identified, categorized, and screened based on the peer-reviewed literature or Brazilian theses published between 2006 and 2010. Articles meeting inclusion criteria were evaluated using U.S. Community Guide methods. External validity reporting was assessed among a subset of articles reviewed to date. An evidence rating typology was developed and applied to classify interventions along a continuum based on evidence about their effectiveness in the U.S. context, reach, adoption, implementation, institutionalization, and benefits and costs. EVIDENCE SYNTHESIS: Thirteen articles published between 2006 and 2010 met inclusion criteria and were abstracted systematically, yet when combined with evidence from articles from the previous systematic review, no additional interventions could be recommended for practice. Moreover, the reporting of external validity elements was low among a subset of 19 studies published to date (median=21% of elements reported). By applying the expanded evidence rating typology, one intervention was classified as evidence-based, seven as promising, and one as emerging.
CONCLUSIONS: Several physical activity interventions have been identified as promising for future research and implementation in Latin America. Enhanced reporting of external validity elements will inform the translation of research into practice.

Context

The growing burden of chronic disease in low- and middle-income countries has stimulated public health efforts focused on increasing physical activity. To address this burden, systematic reviews, or research syntheses, can provide a useful tool for identifying effective interventions that may help decision makers and practitioners prioritize limited resources.2, 3 Latin America represents a specific region of the world faced with a growing burden of noncommunicable diseases but lacking evidence concerning the effectiveness of community-level physical activity interventions.4, 5, 6 For example, a systematic review of evaluated physical activity interventions in Latin America published between 1980 and 2006 found that only school-based physical education had sufficient evidence to recommend widespread adoption.

Although the U.S. Community Guide has recommended eight community-level physical activity interventions among adults and youth, other systematic reviews, primarily among adults, have found limited support for the effectiveness of interventions targeting communities6, 8 or multistrategic community-wide interventions. Only one systematic review included non-English articles. Given the recent rapid growth in physical activity programs and research in Latin America,10, 11 an update of the previous literature review could prove beneficial in guiding further decisions concerning physical activity promotion in this region.

Although systematic reviews are helpful for informing public health practice, they are focused primarily on intervention effectiveness, favoring controlled and randomized designs.12, 13 Issues of potential population impact (e.g., reach, uptake in practice, contextual factors); feasibility; and costs rarely are considered yet influence real-world generalizability (external validity) and represent critical information for public health planning.14, 15, 16 Improving the reporting of external validity information may aid in designing future research studies and inform decision making about public health interventions.

Identifying best practices to address complex public health problems is challenging. Yet, doing nothing while waiting for more evidence is unacceptable, and evidence-based action should be focused on using the “best evidence available” as opposed to the “best evidence possible.”17, 18, 19 The L.E.A.D. framework (Locate Evidence, Evaluate Evidence, Assemble Evidence, Inform Decisions) was developed in 2008 by an IOM committee to address the evidence gap for complex decision making about public health problems such as obesity prevention. The L.E.A.D. framework articulates the need for considering external validity and for an expanded evidence typology.

Several other frameworks have been developed to classify public health interventions in general2, 21, 22, 23 and specifically targeting obesity or physical activity, with the goal of fostering evidence-based decision making.18, 24, 25 These frameworks are helpful when evaluating single programs on a one-by-one basis, or when resources permit extensive evaluation including gray literature. However, these existing frameworks do not provide direction on how to classify intervention strategies following a systematic review of peer-reviewed articles that vary in study design and quality of execution.

For example, they do not provide direction on how to classify interventions that do not achieve the rigorous threshold required for public health recommendations (e.g., based on U.S. Community Guide). They also do not address transferability of evidence from systematic reviews of interventions in high-income countries (e.g., Western countries where most evidence exists) to low- to middle-income countries (with limited evidence and resources for evaluation).13, 27 Region- or country-specific reviews are important but not always feasible for countries with limited resources for evaluation; therefore, guidance on prioritizing interventions based on evidence from high-income countries may be helpful while considering important contextual factors.

The goal of the current study was to accelerate the transfer of research to practice and identify research opportunities by (1) updating the prior systematic review of physical activity interventions in Latin America using U.S. Community Guide methods by adding published articles and Brazilian theses from 2006 to 2010; (2) assessing the reporting of external validity elements among a subset of articles included in the review; and (3) developing an evidence rating typology for physical activity interventions in Latin America and applying it to the updated review (Figure 1).7, 26 This review is part of a larger cross-national initiative, called Project GUIA (Guide for Useful Interventions for Activity in Brazil and Latin America), intended to promote evidence-based strategies to increase physical activity in Latin America.

Figure 1

Goals and key methods of the review

Evidence Acquisition

Overview of the Literature Review Update

Although the review was expanded to include only 5 more years of publications, an update was important given the rapid growth in evaluation and funding of physical activity interventions in Latin America in recent years10, 11 and desire to include the most recent evidence in the external validity assessment and application of the evidence typology. Updating the prior systematic review involved similar searching, screening, categorization, and abstraction procedures as described previously and modeled after the U.S. Community Guide.4, 7, 26, 28 All reviews were carried out in 2010–2011.

In the first phase, peer-reviewed literature and Brazilian theses were searched for studies on physical activity interventions. The following databases were searched systematically for entries appearing from 2006 to 2010: Biblioteca Virtual de Saúde (which includes LILACS, MEDLINE, MedCarib, OPAS/OMS, PAHO, and WHOLIS), SciELO, PubMed, Scopus, Web of Science, Transportation Research Information Systems, and CAPES. Studies of physical activity interventions were identified using 22 search terms in Portuguese, Spanish, and English related to physical activity, health promotion, and urban environment. Additional English search terms included intervention, program, policy, Latin America, Brazil, Colombia, Argentina, Chile, Bolivia, Venezuela, Peru, and Mexico. Other peer-reviewed articles and theses were located by (1) contacting Latin American researchers and practitioners; (2) manually searching 113 issues of seven Brazilian public health or physical activity journals from 2006 to 2010 and articles in the 2010 special issue of the Journal of Physical Activity and Health focused on Latin America; and (3) manually searching reference lists of identified articles.

Screening and Categorizing Studies

All intervention studies that met search criteria were synthesized by trained researchers into one-page summary tables in English. Using the synthesis tables, three reviewers each independently categorized the intervention studies by the U.S. Community Guide categories and categories specific to the Latin American review (Appendix A, available online at www.ajpmonline.org) and screened the studies for inclusion in a full abstraction process to assess the strength of the evidence. Decisions about categorization and selection of studies were based on the opinions of a majority of the reviewers.

Articles were assigned to only one intervention category and screened according to these seven criteria described more fully elsewhere: (1) evaluated an intervention with a significant focus on physical activity; (2) original investigation of intervention; (3) evaluated physical activity behavior or aerobic capacity as outcomes; (4) compared outcomes between groups with different levels of exposure to the intervention; (5) were conducted in a community setting (e.g., school, worksite) and were not therapeutic or rehabilitation interventions or that addressed a population group because the members shared a clinical condition; (6) did not solely involve one-to-one advice or counseling in a healthcare setting; and (7) were published in a format in which details about the intervention and its execution were available.

Abstracting Studies to Evaluate the Evidence and Calculate Net Effects

The abstraction phase followed the same procedure as those in the U.S. Community Guide and had been applied previously.4, 29 Two trained bilingual researchers independently abstracted each study and came to consensus on the suitability of the design and quality of intervention execution—the two measures by which the studies were evaluated. Design suitability was coded as greatest, moderate, and least using established criteria. Based on the number of limitations, quality of execution was categorized as good (0 or 1 limitations); fair (2–4); or limited (≥5). Limitations were counted in the following nine categories, as described elsewhere: (1) description of the study population and intervention; (2) sampling; (3) measurement of exposure; (4) measurement of outcome and independent variables; (5) confounding bias; (6) data analysis; (7) participation; (8) comparability and bias; and (9) other biases.

If an intervention had the potential to achieve strong or sufficient evidence of effectiveness using the Community Guide criteria, net effects were calculated as the relative, net percent change from baseline for participants in both the intervention and control groups. The formula for calculating the net intervention effect varied depending on the study design (with or without a control group and/or pre-intervention measurement of outcome). Net intervention effects were calculated for all reported measurements of aerobic capacity and physical activity behavior, for all time periods relative to baseline, and for all subgroups (e.g., gender, country of study). Sufficient and large intervention effects are based on the quality of the outcomes, magnitude of effect sizes, and population reach of the intervention. Results from the assessments of design suitability, quality of execution, and net intervention effects were combined to determine whether an intervention had strong, sufficient, or insufficient evidence of effectiveness for recommendation.

Assessing External Validity

Along with evaluating intervention effectiveness (internal validity), the systematic review included an assessment of external validity elements among a subset of reviewed articles. Articles were selected if they evaluated interventions that (1) possessed sufficient evidence for recommendation (i.e., school-based physical education)30, 31, 32, 33, 34, 35, 36, 37; (2) were candidates for recommendation (i.e., multicomponent instructional programs, based on number and quality of studies included in the review)38, 39, 40, 41, 42; or (3) included specific programs in Latin America, namely Academia da Cidade,43, 44, 45 Ciclovia, Curitibativa, and Agita São Paulo, with extensive practice-based evidence (i.e., evidence of institutionalization or spread to other communities) although with limited research-based evidence (i.e., limited number or quality of peer-reviewed articles) based on knowledge of the research team. The assessment included all articles of these specific intervention strategies reviewed to date.

An external validity assessment tool (EVAT) was developed based on previous work by others.49, 50, 51, 52 The EVAT adapted the RE-AIM model and was organized into five sections: (1) reach; (2) adoption; (3) implementation; (4) outcomes for decision making; and (5) maintenance and institutionalization (www.projectguia.org/en/research_publications.html). Although the items were the same for programs and environmental and policy interventions, the protocol definitions of items were tailored.

Examples of differences in definitions between the external validity factors are presented in Appendix B (available online at www.ajpmonline.org). With the exception of one item, all asked whether a specific element of external validity was reported, with response choices yes, no, or not applicable. Not applicable was assigned to studies in which the intervention was either mandatory, and thus no selection was involved (e.g., students in school physical education program for target population participation rate); only targeted one setting (e.g., citywide program for setting participation rate); or was delivered as part of standard care or curriculum (e.g., existing physical education teachers as delivery agent items).

Five trained abstractors completed an electronic form, with two abstractors assigned to independently assess each study and with subsequent discussion about any discrepancies. Interrater agreement was 96%–100% among a sample of seven articles. Descriptive analyses included the frequency of studies reporting each element, as well as the average number of elements reported per study.

Developing a Modified Evidence Rating Typology

A modified evidence rating typology was adapted from existing typologies.2, 18, 21, 22, 24 The evidence typology places various forms of evidence about physical activity interventions into categories ranging from “evidence-based” (i.e., based on systematic reviews) to “insufficient.” The current typology includes only research-tested interventions, defined here as broad intervention strategies that have undergone empirical investigations and for which their evaluation has been published in peer-reviewed articles or systematic reviews.

This typology and methodology for rating population interventions combine elements of internal and external validity as well as future translation potential. The internal validity elements are those from the U.S. Community Guide with minor adaptation, and the external validity and translation elements are based on the EVAT. The five criteria by which interventions are rated include (1) effectiveness; (2) reach; (3) feasibility; (4) sustainability; and (5) benefits and costs.

These criteria are described in Table 1 and rated based on whether they have support documentation and their plausibility. “Documented” means that written documentation must be provided in peer-reviewed articles. To be “plausible,” an intervention must possess a high degree of face validity for each criterion, as determined by expert opinion that can be derived from researchers and/or practitioners, similar to informed judgments that are used in assessing applicability in the U.S. Community Guide or for evaluability assessments.

Table 1

Evidence rating typology for research-tested interventionsa

Criteria		Evidence-based	Promising	Emerging	Insufficient
EffectivenessBased on direction of effect on public health outcomes, evidence source, and context	Direction of effect	+b	+	+	Mixed
	Source	Systematic review	Three or more peer-reviewed studiescorsystematic review in different country	Less than three peer-reviewed studiesc	Peer-reviewed studies
	Country or global context where implemented	Same	Same (for three or more studies)orDifferent (if systematic review)	Same	Same
ReachApplied to entire communities or individuals within large organization(s) with representation by lower-income populations		Documentedd	Documented	Plausiblee	Plausible
FeasibilityFeasible for replication in other settings/populations given complexity, costs, and political considerations		Plausible	Plausible	Plausible	Plausible
SustainabilityMaintained with institutional/agency support or integrated into existing programs and/or operating procedures to achieve desired outcomes over time		Plausible	Plausible	Plausible	Plausible
Benefits and costsPossesses health, economic, environmental, and/or social benefits to society that outweigh costs		Plausible	Plausible	Plausible	Plausible

Note: Ineffective includes the following types of interventions: (1) interventions that consistently show null or adverse effects; (2) interventions that show evidence of effectiveness but lack plausibility across one of more of the other criteria (reach, feasibility, sustainability, benefits, and costs).

Research-tested interventions include broad intervention strategies that have undergone empirical investigations and for which their evaluation has been published in peer-reviewed articles or systematic reviews.

Positive effects outweigh adverse effects (based on significance and magnitude of effect).

With at least fair quality of execution

Documented=written documentation in majority of peer-reviewed articles or theses from Latin America

Plausible=no written documentation required; however intervention must possess high degree of face validity as determined by expert opinion derived from researchers and/or practitioners.

In this evidence rating typology, evidence-based refers to interventions identified as effective in a systematic review of evidence from the country to which the evidence ratings are being applied. Promising includes interventions that (1) have at least three studies of fair or better quality in which the positive effects outweigh adverse effects or (2) systematic reviews of interventions carried out in different countries (e.g., the U.S.). Interventions in this category may have an insufficient number of studies to be eligible for a systematic review or may be candidates for a systematic review. Evidence-based and promising interventions must all have documentation that they can reach large populations with representation by low-income or underserved groups, and feasibility, benefits, and costs must all be plausible.

Emerging include interventions evaluated by fewer than three studies with positive effects that outweigh adverse effects. Insufficient interventions are those evaluated in peer-reviewed articles but show either mixed effects or positive effects with limited quality of execution. Ineffective interventions include those that (1) consistently show null or adverse effects or (2) show evidence of effectiveness but lack plausibility across one or more of the other criteria (reach, feasibility, sustainability, benefits, and costs).

Evidence Synthesis

Literature Review Update

Of the 2465 peer-reviewed articles and theses that met the search criteria for the updated review, as well as 113 journal issues manually searched, 32 articles and two theses were selected as physical activity interventions in Latin America (Figure 2). Of these, two articles evaluated associations between built environment characteristics and nonmotorized transportation.54, 55 Given the differences in analytic approaches and sources of articles, these and other built environment studies were recommended to be included in a separate, future systematic review.

Figure 2

Results from the search of Latin American peer-reviewed literature and Brazilian thesesa

aFrom the original and updated review

The remaining 32 articles/theses were screened a second time by multiple reviewers using the list of selection criteria defined previously. Only 13 of the 32 (41%) articles/theses met all of the criteria for inclusion in the abstraction process.34, 35, 38, 40, 41, 43, 44, 45, 47, 56, 57, 58, 59 The distribution of the articles from the past and current reviews are shown by level of screening, study characteristics, and intervention categories in Table 2, Figure 2, and Appendix C (available online at www.ajpmonline.org). Most interventions were conducted in Brazil and in school or community settings.

Table 2

External validity reporting and evidence rating for physical activity interventions in Latin America

Intervention category	Number of studiesa			Median (rangeb) number of external validity elements reported	Evidence rating
	Original review (Years 1980–2006)	Updated review (Years 2006–2010)	Total
Informational approaches
Community-wide campaigns	0	158	1	14	Promising
“Point-of-decision” prompts	160	0	1	—c	Promising
Classroom-based health education focused on information provision	361, 62, 63	0	3	—	Insufficient
Mass media campaigns	0	0	0	—	Insufficient
Delivery of short physical activity–related messages	164	0	1	—	Insufficient
Behavioral and social approaches
School-based physical education	531, 32, 33, 36, 37	135	6	14 (9–26)	Evidence-based
Physical activity classes in community settings	165	343, 44, 45	4	11 (8–15)	Promising
Multicomponent instructional programs	239, 42	338, 40, 41	5	11 (8–20)	Promising
Health education with component for turning off TV/video games	0	0	0	—	Insufficient
College-age physical education/health education	0	0	0	—	Insufficient
Family-based social support	0	0	0	—	Insufficient
Environmental and policy approaches
Creation of or enhanced access to places for physical activity combined with activities in informational outreach	0	0	0	—	Promising
Community-scale urban design and land-use policies and practices	—	—	Pendingd	—	Promising
Street-scale urban design and land-use policies and practices	—	—	Pendingd	—	Promising
Transportation policy and infrastructure changes	—	—	Pendingd	—	Insufficient
Community-wide policies and planning	146	147	2	9	Emerging
Total	14	9	23

Qualifying studies included all eligible studies, except those with limited quality of execution.

Where applicable (i.e., n>1 study)

Not assessed for external validity

Given the differences in analytic approaches and sources of articles, it was determined that these and other built environment studies should be included in a separate, future systematic review.

Assessment of the quality of execution and design suitability of all of the studies indicated that only those studies classified as multicomponent instructional programs had the potential to achieve sufficient evidence for recommendation based on U.S. Community Guide criteria.26, 29 Although the net effects of the five studies in this category suggested a positive pattern in effects on physical activity outcomes (Figure 3), multicomponent instructional programs could not be recommended based on U.S. Community Guide criteria for the following reasons: (1) the group sizes of many interventions were small (i.e., n≤103); (2) none used randomized designs and three studies lacked an external control group; (3) many effects were not significant; and (4) the interventions mostly targeted volunteers with limited reach. School-based physical education remained the only intervention with sufficient evidence for recommendation, whereas the other six evaluated interventions had an insufficient number of studies to support a recommendation given their design suitability (54% of studies in moderate or least category) or quality of execution (85% of studies in fair to limited category).

Figure 3

Net effect size plot for multicomponent instructional programs to promote physical activity

IPAQ, International Physical Activity Questionnaire; NSDAW, non–Seventh-Day Adventist women; SDAW, Seventh-Day Adventist women

External Validity Assessment

Among the 19 studies assessed for external validity, interventions included school-based physical education (n=8)30, 31, 32, 34, 35, 36, 37, 57; multicomponent instructional programs (n=5)38, 39, 40, 41, 42; physical activity classes in community settings (n=4)43, 44, 45, 65; community-wide policies and planning (n=1); and community-wide campaigns (n=1). The median reporting of elements was 21% (range=0%–100%; Appendix D, available online at www.ajpmonline.org). The median number of reported elements per study varied by region where the study was conducted and by intervention type. The five studies conducted on the U.S.-Mexico border reported a median of 17 elements (range 10–26), as compared with 12 elements (range 8–20) from the 14 studies carried out in Latin America. Studies classified as school-based physical education had the highest number of elements reported (median=15; Table 2).

A summary of the reported external validity elements is presented in Appendix D (available online at www.ajpmonline.org). Elements describing reach were reported by most studies, with the exception of representativeness. Elements related to adoption, such as recruitment, participation, and representativeness, were lacking in the majority of studies. No study reported setting-level representativeness, and for three studies, this element was considered not applicable because the intervention was implemented in only one setting.

Regarding implementation, nearly all studies reported intervention characteristics and frequency of exposure to the intervention. Few studies reported information about selection process, participation rate, training, and payment of delivery agents. More than two thirds of the studies reported outcomes in a way that could be compared to clinical guidelines or public health goals, and nearly one third reported other benefits of the intervention beyond physical activity. However, few studies reported effects on quality of life, adverse consequences, and cost of intervention. Information on maintenance and institutionalization was reported by less than half of the studies, with 42% reporting on acceptability of the intervention and only 16% reporting long-term effects of intervention on health-related outcomes.

Evidence Rating Typology

The evidence rating typology was applied to the reviewed interventions to better differentiate their levels of supporting evidence, particularly among those that could not be recommended based on U.S. Community Guide criteria alone. Among the physical activity interventions reviewed, one was classified as evidence-based school-based physical education (Table 2). Based on studies from Latin America, physical activity classes in community settings and multicomponent instructional programs were classified as promising. Based on sufficient or strong evidence for recommendation by the U.S. Community Guide,7, 28 other promising interventions included community-wide campaigns, point-of-decision prompts, creation of or enhanced access to places for physical activity with activities involving informational outreach, community-scale urban design and land-use policies and practices, and street-scale urban design and land-use policies and practices. Community-wide policies and planning were classified as emerging based on Latin American research evidence.

Discussion

Although publications of physical activity intervention evaluations in Latin America have increased in recent years, this updated systematic review of physical activity interventions found that school-based physical education continues to represent the only evidence-based intervention for recommendation using the U.S. Community Guide criteria. In order to include more information on external validity and maximize the utility of this review for decision makers, the criteria for evaluating evidence were expanded to incorporate information from interventions with an insufficient number of supporting studies as well as external validity information. In doing so, seven interventions emerged as promising approaches for increasing physical activity levels at a population level, including community-wide campaigns, point-of-decision prompts, physical activity classes in community settings, multicomponent instructional programs, creation of or enhanced access to places for physical activity combined with activities in informational outreach, community-scale urban design and land-use policies and practices, and street-scale urban design and land-use policies and practices.

The majority of interventions rated as promising were recommended approaches by the U.S. Community Guide with limited evidence from Latin America; the transferability of these interventions will need to be tested to determine their appropriateness in this region.13, 27, 66 Therefore, these interventions should be considered as priorities for future evaluation rather than evidence-based recommendations to practitioners. Although imperfect, by using a tiered approach for evidence classification, more information was available to judge the many interventions that originally were classified as insufficient based on the U.S. Community Guide criteria. Certainly, continued work and cross-national dialogue are needed to build on this important and understudied concept of cross-national transferability of evidence about community health interventions. Application of systematic approaches may assist in assessing the adaptability of effective interventions from one country to another.

The expanded evidence typology is a starting point for providing decision makers with a tool for translating research evidence into practice and for identifying evaluation gaps. Yet, it requires factoring information about intervention processes, contextual factors, and population impact that are not always available or encouraged in peer-reviewed publications. Indeed, the current review found that reporting of external validity elements was low among the 19 studies examined, with variation by type of intervention and country where implemented. Most studies failed to report key elements that are important to ensure translation of findings into implementation of interventions in various settings and contexts.

Other external validity reviews of English-language studies—including of childhood obesity prevention research, behavioral childhood obesity treatments, and health behavior change interventions—found similar low reporting across many external validity elements. For example, participation rate and representativeness of the target population were reported, respectively, by 68% and 5% of studies from this review and 76% and 14% from controlled health behavior change interventions. In addition, the reporting of representativeness and participation of settings, as well as reporting of consistent implementation of the program, costs, differential attrition by condition, and drop-out representativeness, were reported less commonly.51, 68, 70

The findings from the current review differed significantly (by ±10 percentage points) and fell outside the range of at least one of the other three external validity reviews51, 68, 69 for reporting of staff expertise or training (current, 10%; others, 60%–89%); long-term effects (current, 15%; others, 29%–74%); program sustainability (current, 40%; others, 0%–2%); and time needed to deliver interventions (current, 85%; others, 0%–68%). The EVAT also expanded on existing external validity assessment tools by modifying/adding items and protocol language to make the tool relevant to environmental and policy interventions as well as individually based programs.

Limitations

This study included several limitations. First, this review may have missed some studies published in non-Brazilian theses that were not indexed and therefore excluded from the literature databases searched. Similar to the original systematic review, this review was limited to Brazilian theses that were readily available using databases on the Internet or through professional contacts. Second, similar to the U.S. Community Guide, the current review provides information on “what to do” but limited guidance on “how to” do it. Studies were classified into intervention categories defined by the U.S. Community Guide with only slight modifications. Use of taxonomies may improve replicability and implementation in practice.

Third, the EVAT was designed to assess reporting of external validity elements and did not define under which conditions the effective interventions would be optimal. Moreover, the only criterion for assessing contextual factors in the evidence classification system was country or region. In addition, the plausibility of the criterion is subjective and based on informed expert opinion, not different from other established evidence rating systems.26, 53 The transferability of interventions to multiple settings, and other countries, will depend on the unique circumstances of the setting and program intended for implementation. As Green and Glasgow suggest, interpretation of evidence for one's local purposes requires the strategic combination of experimental evidence with local surveillance evidence, theory, professional judgment, and participatory planning with those who have local experience. The next phase of this review includes a qualitative assessment of external validity, including contextual factors. Future work is needed to develop frameworks for rating external validity elements.16, 23

Fourth, because a systematic assessment of the gray literature was not conducted, this review did not incorporate formally how practice-based evidence contributes to the continuum. Promising and emerging practices would likely include programs, environmental approaches, and policies that have proven sustainable in practice but whose evidence of effectiveness is limited to the gray literature or considered plausible based only on theory or face validity. A better marriage between research and practice is needed to maximize public investment and relevance of public health research for practice (e.g., building so-called practice-based evidence).20, 73, 74

Conclusion

Accelerating the translation of research (i.e., knowledge on which interventions are effective) into practice (i.e., informing how to implement interventions) will depend in part on funders' willingness to provide resources for and require evaluation of external validity components. Researchers also should be encouraged, particularly by journal editors and peer reviewers, to improve reporting of these factors. In addition, capacity-building among practitioners can be addressed in several ways: (1) workforce development in evidence-based decision making; (2) emphasis on translation of evidence to practice among public health leaders; and (3) building partnerships to support research translation. This review and evidence rating typology represents an important early step in the integration of both internal and external validity evidence that will contribute to dissemination and implementation research.