Formative Process Evaluation of the "Connect" Physical Activity Feasibility Trial for Adolescents.

Most interventions do not reach full implementation in real-world settings. Due to this issue, formative process evaluation during pilot programs can be especially useful to understand implementation strengths and areas for improvement so that full implementation can be reached in future iterations. This study demonstrated how a formative process evaluation of the Connect through Positive Leisure Activities for Youth (Connect) pilot informed course corrections for year 2 implementation. Connect is an intervention to promote a positive social motivational climate for physical activity (PA) in pre-existing after school programs. Connect ran 3 days a week for 8 weeks and had 2 components: a 30-minute "Get-to-Know-You" (GTKY) session and a 60-minute PA session. Formative evaluation was assessed using an observational tool and staff surveys. Changes in youth PA during program hours was assessed as a process outcome using the System for Observing Children's Activity and Relationships during Play (SOCARP). All Connect essential elements were assessed with the observational tool including (a) social goal-oriented support; (b) collaborative, cooperative play; (c) equal treatment/access; and (d) an inclusive and engaging climate. Adequate dose was achieved on all items in all sessions. Although GTKY and PA sessions both reached high fidelity in promoting equal treatment and access, success in reaching fidelity varied for the 3 remaining essential elements. Post-intervention staff surveys indicated acceptability/adoptability of the Connect program and SOCARP observations indicated significant increases in PA from baseline to post-intervention. Changes for year 2 implementation based on the findings are discussed.

Physical inactivity has been identified as a primary contributor of childhood obesity and related diseases,[1,2] with underserved youth (minority and low-income) at greatest risk of inactivity and its health consequences.[1,3] Afterschool programs (ASPs) have been recognized as an important context to support youth daily physical activity (PA) accrual[4-8] and provide substantial reach to underserved youth.[9] However, similar to interventions implemented within other youth settings (eg, school), PA interventions within ASPs have been only minimally successful in improving youth PA motivations and behaviors,[10,11] and prevalence rates of adolescent obesity have remained alarmingly high.[12]

Given the inherent complexity of community-based intervention, process evaluation has been identified as a critical and necessary component of intervention design. Adjustments in program implementation resulting from systematic monitoring and correction of program delivery (eg, dose, fidelity, reach) can help to significantly improve upon the minimal outcomes found across previous community-based obesity prevention interventions,[13-15] including those in the after school setting.[16] In addition, improved monitoring of the internal operations of an intervention (ie, implementation strengths and challenges, theorized change mechanisms) enables researchers to draw conclusions about what differentiates successful interventions from those that have been less successful.[14,17]

The purpose of this article is to demonstrate the utility of formative process evaluation in a year 1 pilot for improving the larger year 2 implementation of the Connect Through Positive Leisure Activities for Youth after school PA intervention (Connect), a feasibility trial implemented within pre-existing ASPs. Many health promotion interventions that have used process evaluation have used it for summative purposes post-intervention to avoid making a Type III error (concluding that an intervention is ineffective when in actuality the ineffectiveness is due to inadequate implementation). However, process evaluation can also be used for formative purposes to test the feasibility of intervention elements in a pilot prior to full implementation and to make implementation adjustments to ensure high dose, fidelity, and acceptance.[15,18] Depending on the stage of intervention development and implementation, researchers will use different components of process evaluation (see Moore et al[15] for review). For this study, which aimed to test the feasibility of implementing a newly developed intervention within pre-existing ASPs, it was particularly critical to examine whether all intervention components were deliverable (ie, dose), implemented in the way intended (ie, fidelity), and viewed as acceptable/adoptable to our community partners within the ASP setting. In addition, observed changes in PA (ie, SOCARP systematic observations) within the ASP from pre- to post-intervention was measured as a key process measure of feasibility to determine whether the intervention was effective for changing targeted behaviors/processes within the setting that have been shown to promote youth motivation and long-term engagement in PA. That is, improvements in youth PA during program hours function as further indication that targeted social motivational processes within the setting are being established for increasing inclusion and PA engagement among enrolled youth.

The Connect Pilot Intervention

The goal of Connect is to increase youth PA within ASPs by improving the PA social motivational climate. The climate-based approach for the feasibility trial is based on a theoretical framework that expands on the social motivational constructs highlighted by self-determination theory (SDT[19]) and achievement goal theory (AGT[20]) and previous research that has demonstrated the need for positive social interactions and an emphasis on social benefits and social affiliation goals of PA for improving long-term life style changes.[21-23] These social affiliation goals/benefits include (a) developing and strengthening friendships through PA; (b) feeling a sense of group belonging with their peers; and (c) feeling connected to, supported, and encouraged by staff.

Development and operationalization of the study’s essential elements (see Tables 1 and 2) for facilitating improvements in the 3 targeted social components were derived from the integration and extension of the theoretical frameworks of SDT and AGT and preliminary qualitative and observational studies the first author conducted within underserved ASPs that assessed the infrastructure, staff behaviors, and social motivational context for supporting youth PA (Zarrett et al., 2012a; Lawman et al., 2012; Zarrett et al., 2015; Zarrett et al., 2018[24-27]). Through 5-day systematic observations of 7 ASPs, using an observation tool that the author developed and tested (Zarrett et al., 2012b; Zarrett et al., 2013; Zarrett et al., 2015[28,29,26]) the first phase of this iterative process identified several critical social processes (eg, positive peer connections, inclusive and cooperative activities that foster group belongingness) and staff behaviors (eg, encouraging and participating with youth in PA) that were primary correlates of students’ moderate-to-vigorous (MV) PA, and the only predictor of MVPA for less active youth, and girls. Findings from the observation study were used to develop an initial set of essential elements (eg, Staff PA involvement and encouragement; inclusive/cooperative activities; respect and positive peer interactions; guided autonomy; PA social support). In the second phase, the author conducted interviews and focus groups with 7 ASP sites (26 staff, 102 youth) to get staff and youth input on the design and feasibility of initially developed program elements and to assess key physical (eg, equipment allocation), structural (eg, PA preferences, activities offered), interpersonal (eg, peer behaviors), and intrapersonal supports (eg, social skills) needed to promote and support youth social affiliation orientations toward PA (friendship, group belonging, and staff connection goals) within ASPs (Zarrett et al., 2018[27]). Content analysis (NVivo) identified key themes related to supporting youth PA social affiliation goals that were used to develop the final set of essential elements and their application. This community-based participatory approach to the development of “Connect,” which informed content, structure, and flexibility, aimed to enhance the likelihood that Connect will be adopted as an ongoing component of the training and curriculum implemented within ASPs.

Table 1.

Essential elements and observed staff-, peer-, and activity-based fidelity constructs for physical activity sessions.

Essential element	Program application and evaluation of physical activity sessions
	Staff	Youth	Activity
Moral, emotional, social goal-oriented support	• Verbal praise and encouragement for participation and achieving social goals• Reinforce respect to ensure emotional safety (no teasing, cliques)• Promotes acceptance and tolerance of mistakes• Encourage youth to play with more than one youth• Staff actively develop supportive connections/relationships with youth	• Altruistic behaviors (building confidence in each other, emotional and moral support)• No boasting, teasing, negative reactions to “defeat,” rivalry, conflict, or competitive body language• Provide praise and positive feedback• Communicate effectively with peers and staff• Encourage effort and fun	• Small group activities focused on friendship and affiliation• Continuous opportunities to build connections with new peers (eg, rotations)• Primary goal of activity is to assist in building friendships• Success is defined as achieving social goals
Collaborative, cooperative play	• Collaborative leadership style• Facilitation of “garden play”—informal-fun involvement• Invested and actively participate with youth in the activity (Modeling, demonstrating)	• Collaborative play and learning—playing well, as a team• Defining competence as achieving social goals• Build trust, friendship, reliance, and comfort with peers	• Activities are cooperative and require teamwork to achieve• Centered on team-based goals/skills rather than individual goals/skills• Youth teach their peers new activities• Grouping of athletes: Heterogeneous ability groupings
Equal treatment/access	• Equal treatment and perceived fairness	• Equal treatment of all peers• Acceptance of peers	• Equal allocation of space and equipment across program youth• Staff accessibility/availability equal across program youth
Inclusive and engaging	• Actively encourage all youth to become involved• Adults actively keep youth from being excluded• Positive affect: Appear to be enjoying and value the activity• Demonstrates confidence and clarity in implementing activities• Value and praise social development/accomplishments	• Accepting of peers of all skill levels, and unconcerned/accepting of mistakes• Ensure peers feel part of the group and valuable contributors to program• Components• Positive affect: Appear to be enjoying and value the activity	• Foster group identity and perceived belonging• Involve all youth, all skill levels, and no eliminations• Developmentally appropriate, engaging, challenging, fun, varied• Clear Rules: Youth know what is expected of them• Youth feel capable and able to participate successfully• Opportunities for youth to have choice, provide meaningful input• De-emphasize normative ability

Table 2.

Essential elements and observed staff-, peer-, and activity-based fidelity constructs for Get-to-Know-You sessions.

Essential Element	Program application and evaluation of Get-To-Know-You sessions
	Staff	Youth	Activity
Moral, emotional, social goal-oriented support	• Adults facilitate positive student interactions• Adults do not allow verbal “put downs” from youth• Adult leaders emphasize the social benefits of PA• Adult leaders have meaningful verbal interaction with youth (eg, calling youth by name)• Adult leaders have meaningful nonverbal interaction with youth (eg, tone of voice, listening and responding, getting on student’s level, good eye contact).	• Student interactions are positive (encourage one another, do not tease, pressure or discourage participation in any way)	• Activities offered are socially focused (eg, icebreakers, team-building exercises, discussions/brainstorming)• Activities provide opportunities to build and/or exercise social skills (good listening, taking turns, being a good friend)• Youth rotate between small groups at least once during the session
Collaborative, cooperative play	• Adult leader asks if youth have questions	• Youth actively seek out ways to learn more about one another and new ways of thinking from each other• Youth don’t self-select into group	• Youth develop new socially based PA activities in small groups and teach/learn these activities to/from one another• Group-based social PA goals are developed and progress is monitored (eg, learn everyone’s name, walk/run with a peer)
Equal treatment/access	• Adult leader listens to what youth have to say (nonverbal attending)• Adults enforce ground rules/discipline policy consistently• Adults don’t punish the larger group for the behavior of a few individuals• Adults ensure youth follow ground rules	• Youth listen to what their peers have to say• Youth demonstrate respect and appreciation for diversity
Inclusive and engaging	• Adults actively encourage all youth to become involved• The tone/atmosphere set by adults is welcoming, upbeat and positive• Adults indicate that the program (club) is for everyone• Adult leader explains overview to all youth• Adult leader fully answers youth’s questions	• All youth introduce themselves, share their ideas, personal experiences/ thoughts with the group	• Activities facilitate youth agency and leadership (Youth take an active role in guiding session discussions, developing and bringing new activity ideas to the program, and teaching their peers)

Abbreviation: PA, physical activity.

The final set of essential elements of the Connect intervention, derived from central themes identified across this earlier work, were focused on establishing a positive PA social climate across the multiple levels of the ASP setting including staff behaviors, peer interactions, and program structure (eg, activities offered) and included (a) moral, emotional, and social goal-oriented support and skill development (eg, friendship-building skills); (b) collaborative, cooperative play centered on friendship and informal-fun; (c) equal treatment and access; and (d) an inclusive and engaging climate for both youth and staff. The Connect PA curriculum and staff training were centered on these essential elements. (See Tables 1 and 2 for detailed description of each essential element.)

There were 2 primary components of the Connect curriculum: (a) a weekly small group “get-to-know you” (GTKY) session that provided guided social opportunities and (b) novel, socially oriented PA programming integrated within ASPs daily indoor/outdoor recreational time (see Zarrett et al., 2015[26] for description of the nature of indoor/outdoor recreation time typically provided within ASPs’ daily schedule). The “Get to Know You” (GTKY) small group sessions were designed to foster the intra- and interpersonal assets needed to build and support friendships, group cohesion, and staff connections through PA. The PA component of “Connect” was designed to promote a positive PA social experience and affiliation orientation toward PA[22,30,31] among participating youth. Youth had choice in games that were novel (ie, newly designed, never played by youth), inclusive, emphasized teamwork, and centered on social goals linked to the GTKY sessions. (See Table 2 for detailed description of each essential element.)

Purpose

Using data from an objective process evaluation observation tool, a staff post-intervention survey, and systematic observations of youth PA within the ASP, the purpose of this study was to test the feasibility of implementing a newly developed intervention within pre-existing ASPs. Findings are used to understand intervention and implementation strengths (ie, what worked well within the ASP setting), areas in need of improvement, and the impact of the intervention on process outcomes (ie, youth PA within the ASP). Given the challenges of developing and implementing translational interventions within real-world settings (eg, schools, communities), this study demonstrates the usefulness of formative process evaluation for making important adjustments to intervention implementation that can improve adoption and effectiveness of the larger year 2 trial.

Method

Setting/participants

The Y1 pilot (2 programs; 1 intervention vs 1 control) was implemented within school-based aftercare programs for middle school youth (grades 6-8) within a southeastern city in the United States. School recruitment criteria was 3-fold: schools were required to have an ASP that included time allotted for PA/recreation in their curriculum, were considered “underserved” (ie, 50% or more of the school student body were of minority status and/or received free or reduced lunch), and had a “Positive Youth Development” framework (eg, program mission/curriculum fostered overall well-being rather than a specified set of skills; eg, 4H, Boys and Girls Club of America[32,33]). Similar to other developmentally based ASPs, the daily curriculum of the ASP in our pilot included a short 15-minute snack/social time, a 1-hour homework session, and a 1.5-hour recreational session. Incorporated into the regular weekly curriculum of the ASP, all youth enrolled in the intervention ASP were invited and encouraged to participate in the Connect programming. However, to be eligible for participation in the study’s data collection (eg, youth surveys, accelerometers), youth were required to (a) be currently enrolled in the ASP, (b) have parental consent and youth assent to participate, and (c) be available for baseline and post-intervention measurement. Adolescents were excluded from participation if they (a) had a medical condition that would interfere with participation in PA or (b) were developmentally delayed such that the intervention/measurement materials were not appropriate. Programs were randomized to either the intervention or a wait-list control post-baseline data collection (ie, parallel randomized controlled trial [RCT] design; intervention allocation dictated by coin toss). This study is based on data collected solely in the intervention ASP and includes a process evaluation tool, a staff acceptability survey, and youth PA during program hours as measured using a well-validated systematic momentary time sampling observational technique. Three youth dropped out of the intervention sample post-randomization due to sporadic attendance (ie, competing extracurricular activities) and 2 students dropped out due to disenrollment in the ASP. One student did not meet inclusion/exclusion criteria (ie, developmental disability) to participate in the study (ie, measurement) but was still included and encouraged to participate in all Connect programming. A total of 34 students (55% female; 41.2% minority status; Mean Age = 12.4) were enrolled in the intervention ASP and participated in the Connect curriculum and at least 3 of the 4 staff were present each day. All youth in the sample had received parent consent and provided their own assent to participate in the study. All procedures performed in this study were approved by the University’s institutional ethics committee (Pro00037559) and were performed in accordance with the ethical standards of the APA.

Prior to implementation, all Connect intervention staff and ASP staff attended a 3-day training with our Physical Education consultants to learn how to implement all program activities and to establish a positive PA social climate. Staff were provided a daily activity schedule that outlined which activities to offer, and a detailed game guide that described how to set up the activity area and how to succinctly introduce and teach the activity to participating youth.

The 8-week PA intervention took place 3 days a week during the 1-hour recreational component of the program. The PA schedule each day included 3 socially oriented PA stations in which youth would rotate every 20 minutes. Fridays were designated as “choice days.” On choice days youth activity preference ratings were used to determine which activities were offered. For any youth who opted out of the main activities, a small alternative activity was also offered each program day. One day each week, an additional 30 minutes of the recreational session was dedicated to “Get-to-Know You” activities. Get-to-Know-You sessions involved student-led discussions and activities (eg, team building) as well as a long-term project in which they designed their own PA-focused health campaigns (ie, designed, recorded, and presented a video) to promote PA among peers outside the program. These activities were designed to build further group cohesion through fostering youth value and adoption of common/shared group goals, while simultaneously helping to reinforce positive individual and group-level cognitions/goals around PA.

Measures

Process evaluation observation tool

To assess implementation dose and fidelity of all intervention components, process evaluation, using an objective observation tool, was conducted once a week by a trained independent evaluator across the 8-week intervention. A validated and reliable process evaluation tool from a previous PA trial with adolescents[34] was adapted for this study. The final tool for this study was developed through an iterative process based on a well-established framework[35] and employed in previous research,[34,36] where investigators developed a list of critical constructs to be evaluated based on theory and previous research, and distributed this initial process evaluation tool to outside experts for review, modification, and final approval (see Tables 1 and 2 for the Essential Elements, definitions, and corresponding process evaluation items for PA and GTKY sessions, respectively). The final process evaluation instrument assessed implementation dose (present/not present) and fidelity (a 3-point scale indicating quality of implementation as 1 = not present, 2 = sometimes present, 3 = present during most/all of the observed session”) at the staff, peer, and activity level, using multiple items to assess each of the 4 essential elements.

The process evaluator received a central training session where they were provided a detailed overview of the study’s essential elements and the purpose of the process evaluation, orientation to the process evaluation form, and instruction and practice in data gathering skills using video clips of the ASP setting. Central training on observations was then followed with 2 practice observation days at the intervention site, in which the evaluator was paired with a gold-standard observer (the project director), discrepancies in ratings were discussed, and inter-rater reliability was achieved.

Staff acceptability survey

Acceptability was measured as part of a larger readiness assessment tool administered to each ASP staff (n = 4) at post-intervention. The measure included 5 items that assessed staff perceptions of the degree of challenge/ease in delivering the intervention (eg, “Program staff felt that integrating Connect activities into the program schedule was”: 1 = very difficult; 5 = very easy).

Youth physical activity

Changes in youth PA within program hours from baseline to post-intervention were assessed using the System for Observing Children’s Activity and Relationships during Play (SOCARP[37]) SOCARP uses time sampling techniques during which the activity level of each enrolled student is individually observed across a 2-minute period (ie, each 10-second observation interval is followed by a 10-second recording interval and so on). For each observation interval, youth activity is recorded on a range of 1 through 5 (1 = lying down, 2 = sitting, 3 = standing, 4 = walking, 5 = vigorous). Previously established inter-observer reliability rates for the SOCARP instrument range from 88% to 90% agreement[37,38] and were in a similar range of 89% to 94% for this study. Teams of 2 coders made continuous observations of daily activities throughout 5 program days at baseline and post. On average, each student was observed 2 times per day across the 5 program days. This number of observations/days is consistent with other PA interventions targeting children[39-41] and represents an accepted number of observations on which to determine intervention effectiveness. Each participant’s total PA scores across each of the observed weeks at baseline and post-intervention were summed and averaged by their total number of observations. For this study, observed changes in youth PA during ASP hours from baseline to post is used to assess the feasibility of the intervention for changing the targeted processes within the setting.

Analytical procedures

To determine dose and fidelity of implementation, percentages across all observation days were calculated separately for each essential element at the staff, peer, and activity levels, as well as for each essential element overall (see Tables 3 and 4). For dose (present/not present response options), percentages were calculated as the number of “present” responses out of the total number of sessions observed. For fidelity, percentages were first calculated for each item to indicate the number of observations that scored a 2 (“sometimes present”) or a “3” (present most/all the time) out of the total number of sessions observed. Drawing from acceptability standards set by several previous studies, dose and fidelity were deemed acceptable if the observed element was present most/all of the time (“3”) in 75% of the observed sessions.[17,42] For acceptability, the mean and range of staff responses indicated the degree to which staff perceived intervention adoption and implementation as feasible. To measure observed changes in youth PA from baseline to post-intervention, a paired t test was used to determine whether there were significant changes in PA from baseline to post.

Table 3.

Percentage of observations in which dose was achieved.

Session elements	%
Across elements
Youth were greeted by name by staff	100
Youth were greeted by peers	100
Ground rules were developed/reviewed	83
Adult leader provided overview of session to all youth	100
Topic/skill explained by adult leader	80
Topics/skill demonstrated by adult leader	75
Get to Know You session
Time allotted for social session	100
Guided social activities provided	100
Youth brainstorm, role play, etc	100
Summary/closure	100
Physical activity session
Time is allotted for PA session	100
Adult leader provided overview of session	100
PA choices listed	100
Physical activity alternate offered (youth participate)	86

Abbreviation: PA, physical activity.

Table 4.

Percentage of observations in which fidelity was achieved “some” (2) or “most/all” of the time (3) for GTKY and PA sessions.

Essential elements	GTKY sessions(%)			Physical activity sessions(%)
	Most/all the time	Some of the time	GTKY total	Most/all the time	Some of the time	PA total
Moral, emotional, social goal-oriented support
Staff	87.5	4.17	91.67	72.22	6.94	79.17
Peer	83.33	16.67	100	45.95	48.65	94.59
Activities	91.67	8.33	100	71.05	26.32	97.37
Total	88.10	7.14	95.24	58.02	29.01	87.02
Collaborative/cooperative play
Staff	66.67	33.33	100	72.22	6.94	79.17
Peer	40	40	80	37.93	22.41	60.34
Activities	83.33	16.67	100	54.17	16.67	70.83
Total	61.9	33.33	95.24	56.18	14.61	70.79
Equal treatment/access
Staff	93.75	0	93.75	98.18	1.82	100
Peer	75	25	100	–	–	–
Activities	–	–	–	89.47	10.52	100
Total	85.71	10.71	96.42	95.95	4.05	100
Inclusive/engaging
Staff	79.17	4.17	83.33	84.48	6.03	90.52
Peer	33.33	66.67	100	92.68	0	92.68
Activities	60	40	100	81.43	11.43	92.86
Total	68.57	20	88.57	85.02	6.61	91.63

Abbreviations: GTKY, Get-to-Know-You; PA, physical activity.

Results

Dose and fidelity were evaluated via 28 observations. Physical activity observations were conducted across all program implementation days (Mondays, Wednesdays, and Fridays; 22 observations). Get-to-Know-You session observations were conducted on Wednesdays (6 observation days).

Dose delivered

Adequate dose (≥75%) was achieved on all items in all sessions. For example, across all observed days, adult leaders provided a list of PA choices, and for GTKY sessions, guided social activities were provided. See Table 3 for all dose items and ratings. Although still within adequate range, findings indicated that there could be additional improvements with ASP staff demonstrating/participating in both GTKY and PA program components (75%) and explaining (80%) the topic/skill to be covered.

Fidelity

Moral, emotional, social goal-oriented support

Evaluation of the GTKY sessions indicated that implementation of Moral, Emotional, Social Goal-Oriented Support achieved adequate fidelity overall (88%) and at each level of analysis (ie, staff, peer, and activity). In contrast, implementation of Support during the PA sessions did not meet adequate fidelity at any of the levels of analysis (see Table 4). Among Support goals for PA, findings indicated that the greatest challenge for staff was in regularly “ . . . emphasizing the social goals/benefits of PA,” as this was only implemented consistently in 5% of observations. All other staff support goals were met with fidelity (≥75%). Among peer-level goals for Support, primary challenges involved keeping “student interactions positive” (57%) and getting youth to consistently provide “encouragement to one another that emphasizes fun, effort, and social goals” (31%). At the level of the Activity, it was somewhat challenging to consistently provide an activity that was “social in nature” (71% most/all the time), however, a social activity was provided at least “some of the time” for almost all sessions observed (97%).

Collaborative/cooperative play

Adequate fidelity for implementing Collaborative/Cooperative Play was not reached during GTKY or PA sessions (see Table 4). For GTKY sessions, staff behaviors and the climate set by the activity both reached adequate fidelity, indicating that staff provided adequate encouragement of collaborative/cooperative play and program elements related to collaboration and cooperation were implemented mostly as planned. However, Peer level implementation of collaborative/cooperative play (eg, “youth actively seek out ways to learn more about one another and new ways of thinking from each other”) was identified as particularly weak.

Collaborative/Cooperative play was particularly challenging to implement in the PA sessions and a notable area in need of modification/improvements for the Y2 intervention. Staff struggled mostly with “getting students’ feedback, or lead short discussions during a break or at the end of activities” (implemented most/all of the time in only 2 out of 15 observed sessions), and there was substantial difficulty at the peer-level in preventing youth from “emphasizing winning or being the best” or “self-selecting into groups.” Relatedly, at the activity-level, fidelity was only reached a little over half of observed sessions, with particular challenges in providing activities that “required teamwork, group goals/problem solving, and de-emphasized winning/losing” (implemented most/all the time in only 19% of observed sessions).

Equal treatment/access

Adequate fidelity for Equal Treatment/Access was achieved for both GTKY and PA sessions (see Table 4). This essential element was adequately implemented across the duration of the pilot intervention at all levels of analysis assessed.

Inclusive/engaging

Implementation of a consistently inclusive and engaging climate was not achieved across GTKY sessions. Although adequate fidelity was reached at the staff-level, at both the peer- and activity-level, implementation of an Inclusive/Engaging climate was particularly weak with students failing to consistently “introduce themselves, share their personal ideas/thoughts with the group” or “demonstrate agency/leadership (take an active role in guiding session discussions, teach their peers, develop/bring new activity ideas to the program).”

In contrast, adequate fidelity for meeting the elements of an Inclusive/Engaging climate was achieved across PA sessions. Within the essential element, all levels also achieved adequate fidelity, indicating that, along with Equal treatment/access, an inclusive/engaging climate was successfully implemented during the PA sessions across the duration of the pilot intervention (see Table 4).

Acceptability

Post-intervention ASP staff surveys indicated acceptability/adoptability of the Connect program. All staff “strongly agreed” (M = 5.00) that “the goals of Connect were communicated to them clearly,” and overall staff reported implementation of Connect to be “easy” to “very easy.” Specifically, all staff reported that it was either “easy” (3 staff) or “very easy” (1 staff, the program director) for staff in their program to “learn new games,” and “to learn how to integrate Connect into the program schedule” (M = 4.25 and M = 4.25, respectively). Staff also reported that “getting students to participate in Connect activities was ‘easy’” (M = 4.25; 3 staff indicated it was “easy” and 1 staff reported it was “very easy”) and that “communicating the benefits of the Connect activities to the students” was, on average, “very easy” (M = 4.75; 3 staff reported it was “very easy” and 1 staff reported it was “easy”).

Youth physical activity

A paired t test indicated significant observed increases in youth PA from baseline (M = 3.3, SD = 0.68) to post-intervention (M = 4.0, SD = 0.54) within the after school program setting, t (22) = 3.22, P < .01. The difference in youth mean PA scores between baseline and post is equivalent to youth activity moving, on average, from more ASP time spent in sedentary/light activity (eg, standing) to more time spent in moderate activity (eg, walking, jogging).

Discussion

The purpose of this study was to systematically monitor implementation of the year 1 “Connect” pilot to determine strengths and areas in need of improvement to meet study objectives in the larger Y2 trial. Given this study aimed to test the feasibility of implementing a newly developed intervention within pre-existing ASPs, it was particularly critical to examine whether all intervention components were deliverable (ie, adequate dose), implemented in the way intended (ie, adequate fidelity of intervention essential elements), acceptable/adoptable to our community partners (ie, ASP frontline program staff), and efficacious in changing targeted processes within the ASP setting (ie, increased youth PA during program hours as measured by SOCARP). To identify and adjust areas of program implementation that are not achieving dose, fidelity, acceptability, or changing targeted processes, can help to significantly improve upon the minimal outcomes found across previous community-based obesity prevention interventions,[13-15] including those in the after school setting.[16]

Findings indicated several strengths in initial implementation. In particular, adequate dose was achieved for all GTKY and PA sessions and frontline ASP staff perceived the Connect program as valuable, easy to implement, and feasible to infuse into the daily program. These findings indicate that it is possible to successfully implement all components of the intervention and that the program is acceptable and adoptable in pre-existing after school programs. In addition, significant increases in youth PA from baseline to post-intervention indicate the Connect program is capable of changing key targeted behaviors/processes within the setting.

In terms of reaching fidelity, findings are mixed with some components implemented as intended, and others for which Y2 implementation modifications/improvements are needed. For example, “Equal Treatment/Access” was implemented effectively across GTKY and PA sessions, indicating that establishing this essential element is feasible in ASPs and that our implementation strategies were effective and should be maintained. In contrast, and in need of improvement, Collaborative/Cooperative play did not reach fidelity for either the GTKY or PA sessions. For both GTKY and PA sessions, youth had difficultly adopting a new paradigm in which they took greater initiative in communicating and working together, and approaching activities with a cooperative rather than a competitive orientation. Aligned with SDT and AGT and our previous systematic observations of the social climate of youth ASPs (Zarrett et al., 2015[26]) and summer camps (Zarrett et al., 2012; Zarrett et al., 2013[43,29]) activities that are more cooperative, emphasize teamwork, and provide opportunities for positive peer interactions and building friendship are highly effective for improving youth PA engagement in ASPs. However, there are minimally observed instances of these critical social features in the daily experiences youth have with PA in ASPs. These challenges may be attributed to the value of competitiveness embedded within the organizational structures and cultural norms that characterize community and school-based sports programs. Changing the adolescent PA landscape will involve a continued focus on changing these larger norms centered on the competitiveness of sports and PA that emphasize performance (eg, winning, being the best) to a more inclusive “optimal challenge” framework (eg, personal and team goals of improvement) at all levels of the system (eg, staff, youth, quality of the activities). Previous SDT-based research has shown that a more inclusive, “optimally challenging” mastery climate is more likely to be achieved when youth are allocated into mixed-ability groupings, pace of learning is accommodated, personal and team improvement is emphasized, and youth are given opportunities to exercise leadership.[44] Increased consistency in incorporating such strategies into the PA curriculum, such as including older youth in junior leadership roles, and having staff allocate youth into mixed-ability teams may help further improve these cooperative and collaborative goals.

For the remaining 2 essential elements, findings indicated unique implementation strengths and challenges. Establishment of a climate that was socially-emotionally supportive was particularly successful in the GTKY sessions. In particular, the “Connect” small-group activities adequately emphasized friendship and affiliation. Staff and youth successfully demonstrated acceptance and respect while consistently providing praise and encouragement. In contrast, establishment of a socially-emotionally supportive climate was somewhat challenging for PA sessions. Although the activities provided were consistently oriented toward social goals and the development of social skills, staff did not consistently emphasize the social experiences afforded by these activities or optimize on building staff-peer and peer-peer connections through engagement in these activities. Moreover, along with its impact on collaboration and cooperation, challenges with minimizing youth competitiveness resulted in difficulties meeting peer support goals (eg, frustration with peers, teasing). The goal of “Connect” is to develop and implement innovative ways to change youth and staff schemas of PA from this competitive orientation to one that is more socially and emotionally supportive in nature.[26,45] In Y2 this will require that we work with staff to improve their comfort and delivery of these games so that they relay messages of cooperation, teamwork, and social emotional support. It may also be effective to optimize on the intimacy we were able to establish in the small group platform and conduct some of the smaller PA games and challenges within this group setting.

For the essential element of inclusion and engagement, the 2 program modules demonstrated the opposite strengths. During PA sessions the staff, peers, and activities were particularly effective in meeting criteria to facilitate wide involvement and high engagement (eg, youth took initiative; the activities offered met all youth skill levels). In contrast, findings indicated that the development of more intentional ways of prompting youth to connect with and learn from each other were necessary in future iterations of the GTKY sessions. Sharing and taking other’s perspectives requires behavioral autonomy, which is still emerging developmentally during the middle school years.[46] Therefore, future implementation may need to employ a more deliberate scaffolding model whereby ASP staff demonstrate how to communicate in this new way.

Despite the various implementation challenges identified in the Y1 pilot, the adequate dose, high staff acceptability of Connect programming, and the significant increases in youth PA from baseline to post-intervention indicate the Connect program, even in its current form of implementation, is feasible to infuse into ASPs daily schedule and is efficacious in changing key targeted behaviors/processes within the setting.

Implications for Y2 intervention implementation

Overall, our findings indicate that the “Connect” program is able to establish an inclusive climate, where all youth have equal treatment and access. Along with high acceptability ratings from the staff survey, fidelity to this essential element demonstrated substantial program and staff “buy-in” to the intervention, and promise for sustainability. However, increased focus on shifting the normative paradigm of the ASP climate to facilitate youth cooperative interactions, improve staff understanding of, and efficacy in, implementing the social goals of “Connect,” and curbing youth competitiveness will be required to improve program effectiveness. For Y2, several modifications will be made to training and implementation to address challenges identified by our Y1 formative process evaluation.

Meeting youth SDT needs for autonomy, mastery, and belongingness through providing youth opportunities to work with a heterogeneous group of individuals toward a shared goal/cause, and allowing youth the freedom to make real decisions and take leadership roles within a supportive structure have been shown to facilitate youth engagement (infuse adolescent participation with meaning), improved connection (eg, trust, acceptance) and greater affective ties among peers, staff, and the overall ASP community.[47,48] In particular, employing youth participation approaches that empower youth through power-sharing work with adults and peers, and where they advocate for specific changes that meet their unique needs, is one method shown to provide these types of opportunities for youth, and have demonstrated potential to increase youth engagement and health outcomes.[49,50] Enhancing the youth participatory activities within the GTKY sessions, so that youth are prompted to work together with ASP staff and a peer group to develop and advocate for a PA-related change in their ASP or school (http://yparhub.berkeley.edu/), has promise for addressing some of the engagement, collaboration, and inclusion barriers identified in this study.

Along with changes to the GTKY component, improvements made to the curriculum manual and staff training for year 2 are needed to directly target improvements in social emotional support and collaboration during PA sessions. For example, revisions to the “Connect” manual so that it includes additional cooperative warm-ups (orienting youth toward a teamwork, socially-based activity model) and a section of “Suggestions for Staff” that can provide helpful guidelines on how to initiate and maintain a supportive, cooperative climate during each activity may assist staff in consistently reinforcing this type of supportive climate. Staff training will also need to place greater emphasis on the importance of the social experiences of PA and autonomy promotion within the ASP and can include a “tool box” of methods/strategies to encourage youth to work together and support one another, de-emphasize winning and minimize competitiveness, and help increase youth engagement in decision making.

Previous research has demonstrated that when staff have a comprehensive understanding of the philosophies, goals and expectations of a specific program, and are provided the tools to feel efficacious in delivering the program, they are much more likely to endorse the program and implement it effectively.[51,52] Moreover, experiential learning, where program staff learn “best practices” by actively engaging in them, has been shown to be a highly effective training approach.[53] Therefore, training in Y2 will be designed to include multiple opportunities for ASP staff to practice emphasizing the social goals of activities and to capitalize on opportunities for obtaining feedback from youth to increase their comfort with implementation of “Connect” elements that may contrast with the current ASP norms.

Finally, along with training, other participatory efforts to increase ASP staff engagement/participation in “Connect” activities in year 2 will also be critical for ensuring a socially and emotionally supportive PA climate and to assist with adoption and sustainability of “Connect.” For example, to foster further “buy-in” from staff, we need additional staff input (ie, qualitative data) that can help us to identify potential initial barriers to implementation at baseline and to best meet staff needs (eg, capacity, resources), values, and strengths. Similarly, developing a process to acquire staff feedback throughout the study can help address any additional unanticipated challenges that arise during implementation and will ensure that we integrate staff wisdom, insights, and activity preferences to increase adoption and effectiveness.

Limitations and future directions

Although a significant strength of this study is the use of direct observations,[15] it is important to note that awareness of being observed can potentially influence staff and students’ behaviors (eg, positive response biases) leading to higher dose and fidelity than typical. However, considerable variation observed in fidelity for the majority of essential elements and across GTKY and PA sessions suggests that this did not inhibit the ability of the tool/approach to identify strengths and areas in need of improvement. Moreover, despite gaining a strong understanding of what worked well and what needs improvement, pairing our observation and survey methods with qualitative staff interviews in the future will provide greater insight into the underlying reasons for why some components work or do not work so well in the ASP setting. Finally, there is likely considerable variation in the culture of each ASP, and so even though findings from this study lend important insights into Y2 implementation, it will be important to use process evaluation alongside the larger Y2 trial because new problems are likely to emerge when the intervention is tested in a larger more diverse sample.[15]

Implications for practice and future research

This study demonstrated the usefulness of formative process evaluation to assess the feasibility of intervention components in a pilot prior to full implementation. Formative work can provide critical insights about intervention processes in real-world practice. Findings related to implementation strengths and areas for improvement can be used to modify the content and processes of the intervention to better match the setting during full implementation. In turn, higher intervention-setting fit will increase the likelihood of adoption and maximize study objectives/outcomes, such as behavior change.