A Review of Implementation Outcome Measures of School-Based Physical Activity Interventions.

BACKGROUND: Measuring the implementation of school-based physical activity (PA) interventions is an important prerequisite in assessing their impact. Prior to conducting a study to assess the implementation of the daily physical activity (DPA) policy in Ontario, Canada, a literature review was conducted to identify existing survey instruments to measure 5 implementation outcomes: adoption, fidelity, implementation cost, reach, and sustainability.
METHODS: A search for survey instruments to assess these implementation outcomes at the teacher and school administrator levels was conducted in 7 bibliographic databases, as well as the gray literature. Each survey instrument was coded as assessing 1 of the 5 implementation outcomes if it included at least 1 item measuring the construct.
RESULTS: Twenty-three survey instruments were identified. None of the instruments were specifically developed to measure the implementation outcomes. Fidelity was the most common implementation outcome measured, followed by adoption. The least common implementation outcome measured was sustainability. Thirty-five percent of survey instruments assessed were previously tested for validity and 26% were previously tested for reliability.
CONCLUSIONS: Based on this review, a gap in available instruments to measure implementation outcomes of school-based PA programs was identified. An adapted theoretical framework, presented here, has potential application in future implementation studies.

Physical inactivity is a leading cause of cardiovascular disease, type II diabetes, and other chronic diseases among adults.1 Although obesity rates are rising across all age groups, childhood obesity is increasingly associated with these negative health outcomes in later stages of life.2 Moreover, evidence consistently indicates a dose‐response relationship between physical activity (PA) and positive health outcomes.3 These health benefits include skeletal, cardiovascular and metabolic health outcomes, and mental health.3, 4 School‐based PA interventions and initiatives are becoming more common as evidence of their positive health benefits continues to increase worldwide. Physical activity in schools is associated with increased physical health, and also with increased academic achievement, better classroom behavior, and improved self‐esteem.5

In 2005, the Ontario (Canada) Ministry of Education established Policy/Program Memorandum No. 138—daily physical activity (DPA), a school‐based initiative. The DPA policy requires all publicly funded school boards in the province to “ensure that all elementary students (grades 1‐8), including students with special needs, have a minimum of 20 minutes of sustained moderate to vigorous physical activity (MVPA) each school day during instructional time.”6 Policy and program interventions, such as the DPA policy, provide important opportunities for improving the health and well‐being of children. However, to increase the likelihood of producing positive outcomes, these initiatives must be implemented effectively.

Durlak and DuPre explain that, rather than solely evaluating program outcomes, knowledge of which program components are delivered and how well they are delivered is necessary to accurately interpret program outcomes.7 Often, policies and programs are deemed ineffective based on their outcomes but, in some instances, they may not have been implemented as intended.7, 8 For example, the Lifestyle Education for Activity Program (LEAP) was evaluated to determine whether a link existed between program implementation and outcomes such as engagement in vigorous PA among girls. Schools identified as low‐ and high‐implementing were compared to control schools. It was found that better program outcomes were directly associated with level of implementation.8 If program outcomes alone were evaluated, there likely would be mixed findings about the success of LEAP. Therefore, assessing the effectiveness of a program without understanding its implementation process would be limiting. Nonetheless, measuring implementation is challenging in itself, as recent studies suggest that there is a lack of theoretical knowledge of implementation processes.8, 9, 10 In summary, it is important to conceptualize implementation outcomes as distinct from, but related to, policy and program outcomes.9

The DPA policy implementation has not been formally evaluated at the provincial level since its release in 2005. To address this gap, a research team at Public Health Ontario (PHO) initiated a series of studies to evaluate the development and implementation of DPA in Ontario schools.11 The most recent of these studies, related to this article, explored the extent to which DPA was being implemented across Ontario elementary schools and classrooms in the 2013‐2014 school year. The study consisted of administering 2 online survey instruments to school administrators and teachers. To do this, a set of implementation outcome measures was needed. In this article, we describe the identification and assessment of existing implementation outcome measures for evaluating the DPA policy and, potentially, the implementation of other school‐based PA initiatives.

CONCEPTUAL FRAMEWORK

To inform our series of DPA studies, we adapted a conceptual framework from Chaudoir et al.10 They developed it to guide implementation research, building on previous studies by Durlak and Dupre7 and Damschroder et al.12 The framework addresses significant challenges in measuring implementation due to complex multilevel variables related to the intervention/innovation and its context. The framework posits that structural‐, organizational‐, provider‐, innovation‐, and patient‐level factors predict 5 implementation outcomes: adoption, fidelity, implementation cost, reach, and sustainability. Structural‐level factors include those that represent the sociocultural context within which a program, innovation or organization is situated. Organizational‐level factors are seen as being specific to the context in which implementation is taking place. These can include organizational culture and climate. Provider‐level factors include those that are specific to the individual implementing the program or innovation. Innovation‐level factors are those representing the program or innovation being implemented. Last, patient‐level factors are those specific to the individual or population receiving the innovation or program. These include factors such as motivation and personality traits.

For our studies of DPA implementation, we adapted the Chaudoir et al10 framework to include 6 levels instead of 5. These include structural‐, organizational‐macro‐, organizational‐micro‐, teacher‐, student‐, and innovation‐level factors that influence the 5 implementation outcomes (Figure 1). Structural‐level factors include the sociocultural context that the school is operating within. Organizational‐macro‐level factors include the school boards, health units, and organizations supporting schools, while organizational‐micro‐level factors include the schools and school administrators. Teacher‐level factors are those that enable or inhibit the provision of DPA. Student‐level factors are those that enable or inhibit participation in DPA. Last, innovation‐level factors are those positively or negatively affecting the utility of DPA such as other similar strategies. For purposes of the evaluation of DPA implementation in Ontario we focus here on the organizational‐micro‐ and teacher‐level factors.

Figure 1

DPA Study Framework, Adapted from Chaudoir et al10

The 5 implementation outcomes identified by Chaudoir et al10 based on a study by Proctor et al9 defined implementation outcomes as “the effects of deliberate and purposive actions to implement new treatments, practices, and services.”10 (p65) Two of the main functions of implementation outcomes are to serve as indicators of implementation success and key intermediate outcomes.9 For illustration, we defined the 5 implementation outcomes as they relate to the DPA evaluation. Adoption is considered to be the number and proportion of schools that initially decided to employ the DPA policy. It is often referred to as uptake. Fidelity is the extent to which the DPA policy is implemented as originally intended by policymakers. Upon analyzing the policy, we further divided fidelity into 7 specific components: duration, frequency, scheduling, structure, intensity, sustained PA, and inclusivity. According to the provincial policy, DPA has to occur for a minimum of 20 minutes (duration), each school day (frequency), during instructional time (scheduling), include a warm‐up and cool‐down (structure), be MVPA (intensity), be sustained (continuous PA), and include children with special needs (inclusivity). Implementation cost is a quantified measure that assesses the financial impact of DPA. Reach refers to the number of students who are receiving DPA. Last, sustainability is the scope in which DPA is embedded within the school's daily system.

Studies have shown that there are multiple approaches to measuring implementation outcomes including self‐report surveys, observation, and use of archival records archival.7, 9, 10 Proctor et al9 found that developing implementation outcome measures that would be useful in real world settings are needed. As the final DPA evaluation study focused on administering self‐report surveys to school administrators and teachers, this review of PA survey instruments, specifically, represented an important preliminary step to conducting that study. Its purpose was to identify measures currently available to assess implementation outcomes of school‐based interventions related to PA at the school administrator and teacher levels. In that way, it informed the development of measures, particularly implementation fidelity, for the DPA evaluation. Finally, the paper outlines an adapted theoretical framework relevant to assessing the implementation of school‐based PA policies and program interventions.

METHODS

Literature Search

A literature search was conducted to locate articles focused on evaluations of school‐based PA programs using MEDLINE, Embase, SCOPUS, ERIC, PsychINFO, SPORTSDiscus, and Physical Education Index databases. In addition, reference lists and gray literature were examined for potential papers, and articles recommended by research team members were also examined. It is important to note that this review is focused on the survey instruments used to assess various implementation outcomes related to school‐based PA interventions, not the results of a particular study that used the tool in its methodology. Combinations of controlled vocabularies and keywords representing the categories were developed and included: program, implementation outcomes, population, setting, tools, and measures. A test strategy for search terms was conducted on MEDLINE. This strategy was reviewed by a PHO librarian (BP), and then translated to the remaining databases. When the search strategy was adapted to gray literature searching and applied to Google, a 10‐page limit for records retrieval was set. The search was conducted between May 2013 and July 2013 and no date limit was placed on this search.

Screening Articles and Identifying Survey Instruments

The titles and abstracts of articles were screened and retained if they met the following inclusion criteria: English language, used a survey to evaluate a program, targeted teacher/school administrators as respondents to the survey(s), and focused on a school‐based PA intervention targeting grades 1‐8. To narrow the search, studies were limited to interventions that occurred during the school day. Therefore studies focused on before‐ or after‐school activities were excluded. Full‐text articles were then screened for the above inclusion criteria. If survey instruments were not included in the article, corresponding authors were contacted to obtain the full survey instrument and for information regarding its psychometric properties. If sufficient information regarding a survey was not provided in the article, a separate search of the survey was conducted through MEDLINE, Google Scholar, or Google search engine. Survey instruments were included if they assessed 1 of the 5 implementation outcomes discussed above. It is important to note that, although we were assessing surveys in relation to our defined implementation outcomes, the purposes of the studies reviewed were broad such as to assess program outcomes, implementation, progress, and/or PA policy.

Data Extraction

Three data extraction tables were created to review the survey instruments. The first table (Table 1) focused on survey characteristics including purpose, target population, method of delivery, number of items, time to complete, target sample size, and response rate. The second table was used to gather information on the extent to which survey instruments were assessed for validity and reliability (Table 2). The third table was used to code items within the survey instrument based on implementation outcomes: adoption, fidelity, implementation cost, reach, and sustainability (Table 3). As the review was intended to provide information applicable to our series of DPA studies, fidelity was further coded to attributes specific to the DPA policy requirements, such as measures of duration, frequency, scheduling, structure, MVPA, sustained PA, or inclusion. When assessing these implementation outcomes general definitions for each outcome were used.

Table 1

Description of Surveys

Questionnaire	Study	Purpose	Target Population	Method of Delivery	# Of Items	Time to Complete	Target Sample Size	Response Rate
School Physical Education Environment and Policy Survey	Zhu et al13	To understand physical education program factors and policies affecting Texas students' physical fitness as a result of Senate Bill 530	Teachers	Online	39	n/a	5651 schools (3241 elementary schools, 1245 middle schools, 1059 high schools, and 106 "mixed" schools)	45.6% (58.1% = elementary, 21.2% = middle school, 19.4% = high school, 1.3% = "mixed" school)
School Physical Activity Policy Assessment	Lounsbery et al14	To assess school physical activity policy related to physical education, recess, and other physical activity opportunities within schools	School physical education teachers	Online or in‐person	82	23 minutes	n/a	n/a
Module 3: Physical Education and Other Physical Activity Programs Score Card	Sherwood‐Puzzello et al15	One of 8 modules that assesses school‐based program needs relative to physical activity at 3 midwestern middle schools	Administration, teachers, coaches, school health care providers, students, and parents	In‐person	21	n/a	n/a	n/a
Province‐wide evaluation of action schools! British Columbia (BC) Principal Survey	Masse et al16	To identify factors that influence how well Action Schools! BC is received and implemented	School principals	Online	44	n/a	133	92% (122/133)
Province‐wide evaluation of action schools! British Columbia (BC) Teacher Survey	Masse et al16	To identify factors that influence how well Action Schools! BC is received and implemented.	School teachers (grades 4‐7)	Online	83	n/a	850	71% (587/850)
Telephone interview for school principals, Physical Education (PE) teachers, or designee	Barroso et al17	To assess awareness of and adherence to Senate Bill 42 (SB42) in Texas middle schools, and its impact on the frequency and quality of structured physical activity	School principals, physical education instructors, nurses, or designated personnel	Telephone interview	39	n/a	131	85% (112/131)
School Health Policies and Programs School Physical Activity Questionnaire	Lee et al18	To describe school‐level policies and programs specific to physical activity with an emphasis on practice	School‐level person most knowledgeable about physical activity	Computer‐assisted personal interviews	14	0‐30 minutes	1394	72% (988/1394)
School Health Policies and Programs Classroom Physical Activity Questionnaire	Lee et al18, 19	To describe instruction/al content and teaching practices used in teaching health instruction and physical education20	Teacher who provides the required physical education20	Computer‐assisted personal interviews	45	0‐30 minutes	1260	95% (1194/1260)
Daily Physical Activity (DPA) Online Survey	Alberta Education20	To gather school level information regarding perceptions of DPA and to better understand DPA implementation in Alberta in grades 1 to 9	Teachers and principals	Online	∼30	5 minutes	1901	20.4% (387/1901)
Physical Activity Survey	Kennedy et al21	To gather information on if and how the daily physical activity initiative is being implemented in Calgary schools	Elementary school principals and vice‐principals	Interview or electronic surveys	18	n/a	62	88.7% (55/62)
Irish School Physical Education Survey	Halbert and MacPhail22	To provide detailed information regarding the PE infrastructure in Irish post‐primary schools	Physical education teacher	Mail‐Back paper survey	25	n/a	405	74.8% (303/405)
Teacher Awareness and Implementation Survey	Lanier et al23	To examine factors associated with elementary school teacher awareness and implementation of school food and physical activity policies	Teachers	Online	27	n/a	4380	28% (1243/4380)
Physical Education Program Study Questionnaire	Perry24	To determine the extent to which selected school districts in the Commonwealth of Virginia were following Virginia school mandates and fulfilling requirements identified by the National Association of Sport and Physical Education (NASPE) and the American Heart Association (AHA)	Physical education teachers	Online	21	10 minutes	135	85% (116/135)
School Health Policies and Practices Questionnaire (2007, 2008, 2009, 2010)	Slater et al25	To examine the impact of state‐ and school district‐level policies on the prevalence of physical education and recess in a nationally representative sample of US public elementary schools across 3 years (2006‐2007, 2007‐2008, 2008‐2009)	Administrators (school principals or vice‐principals)	n/a	n/a	n/a	47 states, 690 districts, 1761 schools	54.6% (578 school, year 1); 70.6% (748 school, year 2); 61.8% (641 schools, year 3)
School Health Policies and Practices Questionnaire (2007, 2008, 2009, 2010)	Turner et al26	To track progress since the federal wellness policy requirement went into effect (2006‐2007 school year); to track changes as district wellness policies are strengthened and additional state and federal legislation is developed; and to examine school‐level implementation of district policies	Administrators (school principals or assistant principals), food service staff, other relevant staff	Mail‐back paper survey	∼100	n/a	1450 schools year 1; 1457 schools year 2	57.5% (837 schools, year 1); 74.4% (1084 schools, year 2)
Eat well be active principal questionnaire	Wilson et al27	To measure school healthy eating and physical activity environments and the links with parents/other organizations regarding healthy eating and physical activity	Principals	n/a	27	n/a	40	90% (36/40)
Physical Education (PE) Teacher Questionnaire 2008	Harris28	To evaluate the implementation of the Healthy Lifestyles Act in West Virginia	Physical education teacher	Mailed surveys	65	n/a	696	57% (398/696)
Physical Education Teacher Survey 2009	Harris et al28	To evaluate the implementation of the Healthy Lifestyles Act in West Virginia	Physical education teacher	Mailed surveys	54	n/a	∼688	60% (413/∼688)
Questionnaire	Patton29	To address teachers' perspectives on DPA in the Thames Valley District School Board, Ontario	Teachers	Paper survey	35	n/a	37 schools and 624 teachers in the schools	23% (145/624)
Principal Survey 2013	Veugelers and Storey30	To assess the impact of the APPLE Schools program	Principals	Paper survey	33	5 minutes	n/a	100%
2006 Georgia School Health Profile School Principal Questionnaire—Part II	Falb et al31	To measure the current status of comprehensive health education and policies related to school health and safety, and to monitor characteristics of health education practices	Principals (grades 6‐12)	Paper survey	18	n/a	390	71% (275/390)
2006 Kansas School Physical Activity Survey	Kimminau et al32	To examine the key policies and practices that affects public school children across Kansas	PE teachers	Online	52	20 minutes	∼695	37% (257/∼695)
Appendix 3: Sample School Survey	Ministry of Education6	For principals to update their implementation status and identify areas of need for the upcoming school year	Principals	Paper survey	9	n/a	n/a	n/a
School Health Environment Survey 2007‐2008	Manske et al33	To assess factors in the school environment that contributes to healthy eating and physical activity among children and youth	School administrator	Paper survey or online	93	30‐45 minutes	500 Ontario Schools	n/a

Table 2

Validity and Reliability of Survey Instruments

Questionnaire	Study	Validity	Reliability
School Physical Education Environment and Policy Survey	Zhu et al13	n/a	n/a
School Physical Activity Policy Assessment	Lounsbery et al14	✓	✓
Module 3: Physical Education and Other Physical Activity Programs Score Card	Sherwood‐Puzzello et al15	n/a	n/a
Province‐Wide Evaluation of Action Schools! British Columbia Principal Survey	Masse et al16	✓	✓
Province‐Wide Evaluation of Action Schools! British Columbia Teacher Survey	Masse et al16	✓	✓
Telephone Interview for School Principals, Physical Education Teachers, or Designee	Barroso et al17	n/a	n/a
School Health Policies and Programs School Physical Activity Questionnaire	Lee et al18	n/a	n/a
School Health Policies and Programs Classroom Physical Activity Questionnaire	Lee et al18	n/a	n/a
DPA Online Survey	Alberta Education20	n/a	n/a
Physical Activity Survey	Kennedy et al21	✓	✓
Irish School Physical Education Survey	Halbert and MacPahail22	n/a	n/a
Teacher Awareness & Implementation Survey	Lanier et al23	n/a	n/a
Physical Education Program Study Questionnaire	Perry et al24	✓	✓
School Health Policies and Practices Questionnaire (2007, 2008, 2009, 2010)	Slater et al25	✓	n/a
School Health Policies and Practices Questionnaire (2007, 2008, 2009, 2010)	Turner et al26	✓	n/a
Eat well be active principal questionnaire	Wilson et al27	n/a	n/a
Physical Education Teacher Questionnaire 2008	Harris et al28	n/a	n/a
Physical Education Teacher Survey 2009	Harris et al28	n/a	n/a
Questionnaire	Patton29	n/a	n/a
Principal Survey 2013	Veugelers and Storey30	n/a	n/a
2006 Georgia School Health Profile School Principal Questionnaire—Part II	Falb et al31	n/a	n/a
2006 Kansas School Physical Activity Survey	Kimminau et al32	n/a	n/a
Appendix 3: Sample School Survey	Ministry of Education6	n/a	n/a
School Health Environment Survey 2007‐2008	Manske et al33	✓	✓

DPA, daily physical activity; n/a, this information was not available.

Table 3

Implementation Outcomes Included

			Fidelity
Questionnaire	Study	Reach	Duration	Frequency	Scheduling	Structure	MVPA	Sustained PA	Inclusion	Adoption	Implementation Cost	Sustainability
School Physical Education Environment and Policy Survey	Zhu et al13		√	✓						✓	✓
School Physical Activity Policy Assessment	Lounsbery et al14	✓	✓	✓	✓	✓	✓				✓
Module 3: Physical Education and Other Physical Activity Programs Score Card	Sherwood‐Puzzello et al15		✓		✓				✓
Province‐Wide Evaluation of Action Schools! British Columbia Principal Survey	Masse et al16				✓					✓		✓
Province‐Wide Evaluation of Action Schools! British Columbia Teacher Survey	Masse16		✓	✓	✓		✓			✓		✓
Telephone Interview for School Principals, Physical Education Teachers, or Designee	Barroso et al17		✓	✓	✓					✓
School Health Policies and Programs School Physical Activity Questionnaire	Lee18		✓	✓	✓
School Health Policies and Programs Classroom Physical Activity Questionnaire	Lee18	✓	✓		✓	✓			✓
DPA Online Survey	Alberta Education20	✓		✓	✓		✓			✓	✓
Physical Activity Survey	Kennedy et al21		✓	✓	✓
Irish School Physical Education Survey	Halbert and MacPhail22								✓
Teacher Awareness & Implementation Survey	Lanier et al23		✓		✓
Physical Education Program Study Questionnaire	Perry24		✓			✓
School Health Policies and Practices Questionnaire (2007, 2008, 2009, 2010)	Slater et al25; Turner et al26		✓	✓		✓	✓			✓
eat well be active principal questionnaire	Wilson et al27				✓					✓
Physical Education Teacher Survey 2009	Harris et al28		✓				✓				✓
Physical Education Teacher Questionnaire 2008	Harris et al28		✓				✓				✓
Questionnaire	Patton29		✓	✓		x
Principal Survey 2013	Veugelers and Storey30			✓	✓					✓
2006 Georgia School Health Profile School Principal Questionnaire—Part II	Falb et al31			✓						✓
2006 Kansas School Physical Activity Survey	Kimminau et al32	✓	✓	✓		✓
Appendix 3: Sample School Survey	Ministry of Education6		✓	✓	✓
School Health Environment Survey 2007‐2008	Manske et al33	✓	✓	✓			✓			✓		✓

DPA, daily physical activity; MVPA, moderate to vigorous physical activity; PA, physical activity.

Survey Coding

Each survey instrument item was individually evaluated and coded by 2 members of the research team (S.S. and N.S.M.), as representing none or 1 or more of the implementation outcomes. We coded measures based on the general definitions discussed above. A measure had to include a minimum of 1 item to be coded as representing that outcome.

RESULTS

Literature Search Results

The literature search retrieved 5737 journal articles. After the screening process, 23 survey instruments were retained for further analysis. Figure 2 shows the literature search results. Seven descriptive properties of the survey instruments were assessed (Table 1). Items were predominantly closed‐ended in nature. The number of items on survey instruments ranged from approximately 10 to 115, with the average number of items per survey instrument being 47. Completion time was only available for 8 surveys and among these 8, time ranged from 10 to 45 minutes, with the average completion time being 18 minutes. Overall, 39% of surveys were delivered online, 35% were self‐administered, 26% were administered in‐person or by telephone, and 17% had no information on type of survey delivery. Some surveys were administered by more than 1 method. Response rate information was provided for those surveys that were part of a research study. Response rates ranged from 23% to 100%. Consistent with the target participants of the surveys included in our review, we found that 52% of surveys included school administrators and 65% of surveys included teachers as participants. Some surveys targeted both school administrators and teachers.

Figure 2

Literature Search Flow Diagram

Validity and Reliability of Survey Instruments

We assessed survey instruments for the inclusion of validity, reliability, and other information relevant to their development (Table 2). Results indicated that 35% of survey instruments (N = 8) were reportedly assessed for an aspect of validity: face validity (N = 4), validity of outcome variable using Spearmen rho correlation (N = 2), content validity (N = 1), and criterion validity (N = 1). In addition, 26% of survey instruments (N = 6) were reportedly assessed for reliability. Four of those specified the type of reliability: test‐retest reliability (N = 2) and reliability of scales using Cronbach alpha tests (N = 2). Some studies indicated that a prior review of the literature was conducted before developing items for the surveys. However, information regarding survey development was scarce among many of the reviewed studies.

Implementation Outcomes

The 23 survey instruments were coded to identify items related to implementation outcomes identified in our adapted conceptual framework and our specific interest in implementation fidelity (Table 3). Of these survey instruments, 43% included adoption, 24% included reach, 100% included fidelity, 22% included implementation cost, and 13% included sustainability. An example of an item coded as adoption was: “Has the school adopted the district's wellness policy for physical activity and nutrition?”31 When fidelity was broken down into its 7 components, 74% of surveys included duration, 61% included frequency, 57% included scheduling, 26% included structure, 30% included intensity, 13% included inclusion and no surveys included sustained (continuous) PA. The following are examples of the 3 most common components of fidelity (duration, frequency, and scheduling). An example of an item coded for duration was: “How long are the typical physical education class periods (counting time for changing/showering)?”13 An example of an item coded for frequency was: “How many physical education classes per week do students receive? (Provide the average)”14 An example of an item coded for scheduling was: “What type of class schedule do your students follow?”17 For the most part, as mentioned earlier these measures were not conceptualized specifically as implementation outcome measures in the various survey instruments assessed.

DISCUSSION

This review was conducted to address the perceived lack of available measures of implementation outcomes related to school‐based PA policies and programs in the scientific literature. Measures of implementation outcomes that could potentially be used to assess and evaluate PA policies/programs and, specifically for our purpose, the DPA policy in Ontario, were examined. Recognizing the importance and complexity of measuring implementation, we adapted Chaudoir et al's10 multilevel framework as the conceptual framework for our study of DPA policy implementation. Twenty‐three survey instruments were identified that measured 1 or more of adoption, fidelity, reach, implementation cost, and sustainability at the organizational‐micro and teacher levels according to our definitions above. Descriptive attributes and whether or not the validity and reliability of survey instruments were assessed were also examined in our review.

Description of Surveys and Validity and Reliability of Survey Instruments

We found considerable variation between survey instruments in terms of the number of items, time to complete, response rate, and target sample size. It is difficult to make conclusions about these findings because of the incomplete reporting of survey and survey instrument descriptions and development.

Overall, most survey instruments were not reported to have been assessed for validity and reliability. This is a common finding within both PA and implementation literature in general. For example, in a review of literature on measures of PA more generally, it was found that none of the survey instruments demonstrated adequate reliability and validity.36 McGraw et al37 found that more techniques for increasing the validity and reliability of program implementation measures related to PA need to be developed. Similarly, measures of implementation outcomes are developed with minimal analysis of validity and reliability.10 For that reason, it is suggested that any measure, especially in the area of implementation science, should be assessed for validity and reliability.10 Most of the survey instruments may not lend themselves to full psychometric testing because they are not in the form of scales nor have an outcome variable. However, it is still possible to assess face validity.21, 24, 25, 26

Only one of the studies included in our review16 explicitly discussed the use of a theoretical framework or foundation for the development of their survey instruments. Of the studies that were measuring implementation, none discussed specific implementation outcomes, indicating that they did not directly identify indicators of implementation success.

After assessing the 5 implementation outcomes it was evident that items measuring our components of fidelity were most common. Each survey instrument included one of the components of fidelity except for sustained (continuous) PA. Because the DPA policy contains many components, our definition of fidelity was necessarily broad. This could explain why all of the articles assessed contained at least 1 item that measured one of the DPA policy components. Of the components, duration, frequency, and scheduling were the most common measures (74%, 65%, and 61% respectively). Chinapaw et al36 similarly found that frequency and duration were commonly measured dimensions in PA surveys. Only 30% of articles included items assessing MVPA. Chinapaw et al36 found that intensity is a common measure in physical education studies; however, it is usually measured at the youth/student level using accelerometry and heart rate monitors. There were no items that were coded to measure sustained PA. Again, this might have been because it is usually measured by actually monitoring youth activity levels. Surprisingly, only 13% of surveys had an item that measured inclusion to determine whether all students can participate in an activity and how accommodations or modifications are incorporated into PA. This component may be more suitable to measure at the student level using observational methods such as SOFIT (System for Observing Fitness Instruction Time) where the researcher directly observes student PA levels and the activity environment during instructional time.38 This was not feasible for the DPA evaluation study as we were focused on measuring implementation at the teacher and school administrator levels.

Based on our review, adoption was the second most common implementation outcome measure (43% of survey instruments). Adoption and fidelity tend to occur early in the implementation process, which could explain why most measures found in this review measured those 2 outcomes. Sustainability was the least common measure (13% of survey instruments). This, again, could be explained by the fact that sustainability occurs later in the implementation process, and most surveys measured implementation at early stages.10

McGraw et al37 identified the need for greater conceptual clarity in defining implementation constructs when trying to measure school‐based PA and nutrition policies and programs. The framework we adapted by Chaudoir et al10 was crucial in helping us define and identify relevant items, and could be utilized by researchers when developing a study to measure implementation of school‐based policies and programs. We found that, the breakdown of fidelity into 7 components was helpful in identifying key items related to fidelity of school‐based PA programs. This approach could be useful to other PA programs as they usually contain some of these components.

Limitations

Our review of existing implementation outcome measures, reported here, contains some limitations. First, we were unable to obtain all the survey instruments we were interested in and requested. These missing instruments may have contained additional relevant items. We contacted authors to obtain missing information; however, we were unable to reach all authors.

In addition, although we restricted the search to surveys measuring PA during instructional time, we believe that these studies would likely include some similar items to studies measuring PA outside of instructional time. Another limitation is that, although the importance of including “patient‐level factors,” such as motivation and personality traits, has been emphasized for understanding implementation outcomes,10 it was not feasible to examine student‐level factors in this DPA study. Furthermore, many of our identified fidelity outcomes, such as intensity, structure, and inclusivity, could be considered student/child level outcomes rather than teacher/school‐administrator outcomes. However, our primary focus both conceptually and empirically, was on measurement of the school‐based structured opportunities for PA identified by administrators and teachers. Understanding implementation at the school administrator and teacher levels provides important information regarding the DPA policy.

Conclusions

Measuring implementation outcomes is critical when evaluating PA policies and programs. Understanding implementation is necessary in interpreting policy and program outcomes and, therefore, adequate measures are required for this purpose. The multilevel framework developed by Chaudoir et al10 is an excellent tool for school‐based programs because it accounts for factors from multiple levels influence implementation outcomes. This review contributes to the implementation science literature by extending it to school‐based PA initiatives. While we found no existing (intact) survey instruments that could be adapted for our study, specific items from several surveys were useful for developing the measurement instruments for our evaluation of DPA implementation in Ontario elementary schools. The current review also contributes to the development of measures of implementation outcomes specific to other PA policies/programs.

IMPLICATIONS FOR SCHOOL HEALTH

The school setting has been identified as having an important influence on the lives of children and youth, as it is where they spend most of their day.18, 34 School policies and programs that provide structured opportunities for health and encourage and support healthy behaviors are being recommended and implemented globally.34, 35 Evaluations are needed to determine the effectiveness of these policy and program interventions and assess their potential scale‐up on a population level. However, to evaluate these interventions, policy and program implementation and its determinants must also be assessed. Proctor et al's9 acknowledgment that there appears to be a lack of theoretical understanding of the underlying processes involved in implementation can be expanded to the PA field. It is important that specific implementation outcome measures are identified or developed, to evaluate school‐based PA interventions. This review provides researchers developing and evaluating school‐based PA policies and programs with: an overview of the importance of measuring implementation outcomes; an adapted theoretical framework that can be used to guide subsequent evaluations; and an assessment of existing measurement instruments in relation to a number of key implementation outcomes.