Implementing Evidence-Based Teen Pregnancy-Prevention Interventions in a Community-Wide Initiative: Building Capacity and Reaching Youth.

PURPOSE: To describe efforts to implement evidence-based interventions (EBIs) within multicomponent, community-wide initiatives to reduce teen pregnancy.
METHODS: During 2011-2014, we collected information about the capacity (i.e., knowledge, confidence, training, and experience) of state and community-based organizations to support implementation of the following: EBIs, number and characteristics of youth served by EBIs, type of EBIs implemented, EBI settings, hours of training, and technical assistance provided. State and community-based organizations reported these data annually; however, training and technical assistance was reported monthly. We used aggregated data from these annual and monthly reports to describe the implementation of EBIs in the community-wide initiative project.
RESULTS: From baseline in 2011-2014, state and community-based organizations increased their capacities to support program partners in delivering EBIs. They provided 5,015 hours of technical assistance and training on topics, including ensuring adequate capacity, process and outcome evaluation, program planning, and continuous quality improvement. Program partners increased the number of youth reached by an EBI in targeted communities by 349%, from 4,304 in the first year of implementation in 2012 to 19,344 in 2014. Most youth in 2014 received sexuality education programs (59%), whereas smaller percentages received abstinence-based, youth development, and clinic-based programs. Most youth were reached through schools (72%) and community-based organizations (16%), and smaller numbers were reached in other settings (e.g., faith-based organizations, health centers).
CONCLUSIONS: Building and monitoring the capacity of program partners to deliver EBIs through technical assistance and training is important. In addition, partnering with schools leads to reaching more youth.

Many adolescent and reproductive health interventions are effective at influencing factors associated with teenage pregnancy [1,2]. However, because evidence-based interventions (EBIs) are not always implemented with quality and fidelity, outcomes may vary [3,4], and results may not mirror outcomes observed in the original efficacy research [5]. For programs to achieve population-level effects, interventions should reach large numbers of youth and be implemented with attention to fidelity, careful adaption, and by using an evidence-based approach.

With the U.S. Department of Health and Human Services Office of Adolescent Health, the Centers for Disease Control and Prevention (CDC) led a 5-year cooperative agreement with eight state and community-based organizations to implement multicomponent, community-wide initiatives (CWIs) to prevent teenage pregnancy among nine communities with high rates of teen births [6]. The Office of Population Affairs and CDC supported a ninth organization working in one additional community [6]. A key component of the initiative involved implementing EBIs shown to be effective in addressing risk and protective factors associated with preventing teen pregnancy. The CWIs also incorporated three key components to facilitate the implementation of EBIs: (1) mobilizing the community; (2) educating stakeholders; and (3) working with diverse communities. In addition, the CWIs incorporated a component to increase youth access to reproductive health care so that youth who are educated through EBIs have quality reproductive health services available.

The goals of the EBI component were as follows: increase the capacity of state and community-based organizations; support program partners in selecting, implementing, and evaluating EBIs; increase the number of youth within the target community exposed to EBIs; and prevent teen pregnancy. Program partners were local organizations that worked directly with youth and were willing to implement teen pregnancy-prevention EBIs.

The project required state and community-based organizations to partner with at least 10 program partners. A multilevel approach to supporting EBI delivery was developed for the initiatives and based on the Interactive Systems Framework for Dissemination and Implementation [4,7]. The state and community-based organizations provided program partners with formal training and technical assistance (T&TA) for selecting, implementing, and evaluating EBIs for the prevention of teen pregnancies.

Although experienced in teen pregnancy-prevention programming, the nine state and community-based organizations started the initiatives with varying levels of expertise for implementing EBIs. To improve state and community-based organizations’ capacities to support the work of their program partners, Healthy Teen Network, a nationally recognized expert in the implementation of teen pregnancy-prevention EBIs, provided T&TA. Healthy Teen Network was funded through a cooperative agreement with CDC and provided state and community-based organizations with train the trainer sessions about specific EBIs, which allowed the organizations to then train their program partners. Healthy Teen Network also provided state and community-based organizations with T&TA on a 10-step process for state and community-based organizations to use with their program partners to select, implement, and evaluate teen pregnancy-prevention EBIs. Referred to as Promoting Science-based Approaches using Getting to Outcomes (PSBA-GTO), this process was developed by adapting a general process for selecting, implementing, and evaluating EBIs [8] to specifically address teen pregnancy-prevention EBIs [9].

We describe monitoring and implementing EBI-related capacity-building efforts and the implementation of EBIs as part of the CWIs. In addition, we describe the methods used to assess the capacity of state and community-based organizations to support EBI delivery, as well as to measure the delivery of T&TA and the performance of program partners in implementing EBIs. We then explain the number of hours and the content of T&TA provided to state and community-based organizations, as well as the change in their capacity, followed by a description of the T&TA that state and community-based organizations provided to local program partners. Finally, we describe the number of youth served and the quality of the EBIs implemented by the local program partners. We conclude with a discussion of lessons learned through these efforts.

Methods

Assessing state and community-based organizations’ capacity to support EBI delivery

During 2011—2014, program staff from state and community- based organization completed an annual Web-based needs assessment to identify strengths and areas of potential growth in their ability to support local organizations in selecting, implementing, and evaluating EBIs to prevent teen pregnancies. Questions were asked about recent experiences and training in relevant topics, as well as their self-reports of knowledge and confidence for providing T&TA on PSBA-GTO to support EBI implementation.

Sixteen items addressed whether the respondent had any training (eight items) or experience (eight items) in PSBA-GTO-specific content areas (e.g., using logic models, assessing programs for fit) during the previous 2 years. All items were summed to create a total training or experience score that could range from 0 (no training) to 8 (trained in all areas) or 0 (no experience) to 8 (experience in all areas), respectively. Twenty-two items assessed PSBA-GTO-specific knowledge, and 22 items assessed confidence in using PSBA-GTO; these items included a 1—5 Likert-scale response option, with higher scores indicating the respondent had more knowledge or confidence in the question topic. For both knowledge and confidence, items were averaged to create a composite total knowledge and composite total confidence score. The knowledge and confidence scales demonstrated good reliability (alpha = .97 for each scale).

Training and technical assistance

During 2011—2014, the nine state and community-based organizations implementing the CWIs and Health Teen Network entered information regarding T&TA events into a Web-based system to track delivery and receipt of T&TA. Information about type of event (training or technical assistance), topic of T&TA (e.g., continuous quality improvement, program planning), and time spent providing T&TA was submitted to CDC in monthly reports beginning September 2011.

Program performance measures

Performance measures assessed the type of programs that were implemented in communities by program partners, the number and characteristics of youth participants, and the fidelity of implementation. State and community-based organizations also conducted participant surveys before and after interventions to assess outcomes, such as changes in adolescent knowledge and attitudes. Performance measure data from program partners were reported to CDC in an annual report. Surveys varied on the basis of the type of curriculum and the setting where programs were delivered; therefore, data could not be reported in aggregate across state and community-based organizations to understand the effect of EBIs on youth across sites.

To consider the level of risk for teen pregnancy among EBI participants, state and community-based organizations collected information about sexual behavior with pretest assessments. Some organizations did not collect data on sexual behavior because some program sites were not approved to ask questions regarding sexual behavior and some organizations were unable to survey all youth participants in all settings because of limited capacity (2012 = 38%, 2013 = 15%, and 2014 = 19%). Youth were asked if they had ever had sex or had sex during the past 3 months and if they had used a form of contraception at last sexual encounter.

The number of youth served was defined as the number who participated in at least one program session during the reporting period. The number of youth served was reported by gender, age, race/ethnicity, and special populations (i.e., youth who are in foster care, homeless, pregnant or parenting, or involved with the juvenile justice system).

Fidelity addresses how well the implementation adhered to the program’s model. State and community-based organizations reported on program activities that were delivered as planned by using self-report tools from program facilitators. They were encouraged to conduct observations of at least 10% of all sessions to assess fidelity; however, observations were limited because of the state and community-based organizations’ capacity to observe the many implementing partners. Fidelity was calculated as the percentage of all activities in an EBI that were delivered as planned.

Results

Addressing state and community-based organizations’ capacities to support program partners

During 2011—2014, Healthy Teen Network provided state and community-based organizations with 376 hours of training and 691 hours of technical assistance related to EBIs. The Healthy Teen Network provided 173 hours of training and 186 hours of technical assistance to directly support the implementation of EBIs, including curricula facilitator trainings, program implementer support, and master trainer skill building for state and community program technical assistance providers. One key direct support activity for EBI implementation was training in program delivery for specific curricula. During the first year of the project, Healthy Teen Network facilitated initial trainings about using PSBA-GTO for all state and community-based organizations and trainings on available EBI curricula. In 2011, Healthy Teen Network provided train the trainer sessions on Cuidate, Be Proud Be Responsible, Making A Difference, and Making Proud Choices. In 2012, Healthy Teen Network provided training of trainers and facilitators for Becoming a Responsible Teen, Be Proud Be Responsible Be Protected, and Reducing the Risk. In 2013, Healthy Teen Network provided training of trainers for Project AIM, Sexual Health and Adolescent Risk Prevention, and Sisters Informing Living Healing Empowerment.

The majority of T&TA provided was about the PSBA-GTO approach: 100 hours out of 376 of training and 380 hours out of 691 of technical assistance. State and community-based organizations’ capacities to use the PSBA-GTO approach were assessed. Thirty-two staff participated in the assessment as EBI program technical assistance providers with an average of 3.5 staff per organization and a range of 3—5 staff per organization. The organizations’ staff capacity for 2011—2014 are presented in Table 1. Technical assistance providers reported having received training on a mean of 5.2 topics related to PSBA-GTO at baseline in 2011, with the largest increase to 7.3 topics during 2011—2012. Mean scores in knowledge and confidence related to program delivery using PSBA-GTO increased each year, with the largest increase again during 2011—2012.

Table 1:

State and community-based organizational capacities to use the PSBA-GTO approach to provide training and technical assistance for teen pregnancy-prevention interventions were assessed across state and community-based organizations, 2011—2014

Capacity area	2011 M (SD)	2012 M (SD)	2013 M (SD)	2014 M (SD)
Number of training topics	5.2 (3.5)	7.3 (1.7)	7.6 (.7)	7.5 (1.2)
Knowledge ratings	3.2 (.9)	3.9 (.5)	4.0 (.5)	4.2 (.6)
Confidence ratings	3.0 (1.0)	3.7 (.8)	3.8(.7)	4.1 (.6)

M = mean; PSBA-GTO = Promoting Science-based Approaches by using Getting to Outcomes; SD = standard deviation.

Improving program partner capacity to implement EBIs

Program partners

During the first 3 years of implementation (2012—2014), state and community-based organizations partnered with 113 program partners. Data were reported for 79 program partners in 2012,112 program partners in 2013, and 107 program partners in 2014. Program partners included community-based organizations (46) (e.g., Boys and Girls Clubs, YMCA, and other nonprofit youth serving organizations), health care organizations (14), schools (12), faith-based organizations (11), school districts (5), and other organizations (25) (e.g., after school, military, foster care, juvenile justice, college). In 2014, the median number of program partners for each state and community-based organization was 8.5, with a range of 5—31.

T&TA provided by state and community-based organizations

State and community-based organizations provided direct T&TA to program partners from 2012 to 2014. A total of 2,171 hours of training was provided with a median of 251 hours, ranging from 21 hours to 445 hours for each program partner. A total of 2,844 hours of technical assistance was provided with a median of 228 hours and a range from 87 to 531 hours for each program partner. The majority of T&TA provided was related to ensuring adequate capacity (23%), process and outcome evaluation (17%), program planning (12%), and continuous quality improvement (12%). Other major topics were related to specific steps of PSBA- GTO (e.g., assessing needs and resources, program fit, goal setting).

Program delivery

Populations served

Across the 10 communities in the CWIs, the number of youth (aged 10—19 years or older) served per year by EBIs increased from 4,304 in 2012, to 12,085 in 2013 and 19,334 in 2014, for a total of 35,723 youth total (Table 2). Among youth served from 2012 to 2014 with data on these variables available, 57% were female and 49% were aged 14 years or younger; 38% were African-American, 34% Hispanic, and 28% identified as white.

Table 2:

Number of youth served across CDC/OAH community-wide initiative program partners, by age and race or ethnicity and percentage distribution by pregnancy risk factors, 2012—2014

Youth served by EBIs who reported demographic information[a]	2012	2013	2014	Project total	Percentage increase 2012—2014
Total numbers Age	4,304	12,085	19,344	35,733	349%
Age
<14	1,600	4,435	7,960	13,995	398%
15—16	1,163	3,330	4,235	8,728	264%
17—18	471	1,656	2,206	4,333	368%
19+	98	530	876	1,504	793%
Race or ethnicity
Hispanic	1,294	3,835	5,634	10,763	335%
African-American	1,853	3,819	6,306	11,978	240%
White	592	2,387	5,826	8,805	884%
% with pregnancy risk factor[b]
Ever had sex	33	46	36	39	—
Had sex during the past 3 months	24	28	25	26	—
Used a method of contraception at last sex[c]	53	53	59	56	—

CDC = Centers for Disease Control and Prevention; EBIs = evidence-based interventions; OAH = Office of Adolescent Health.

Youth served in this table includes only youth who completed a pre- or post- EBI assessment to report their age and race or ethnicity. Total numbers reflect total number of youth served, including youth who did not complete a pre- or post-EBI assessment.

Among youth asked about sexual behavior and use of contraception (N = 1,925 in 2012, N = 8,193 in 2013, and N = 11,719 in 2014). Change is not reported for these variables as they are not measured as outcomes of EBI participation.

Among those who had sex during the past 3 months.

As an indicator of baseline pregnancy risk among the populations served, some sites asked all participants about their sexual behavior and use of contraception at baseline. During 2012—2014, more than half of the youth participants (N = 21,837 of 35,723; 61%) were asked about ever having sex; of these, 40% reported having ever had sex, 27% reported having had sex during the past 3 months, and 56% of those who had sex during the past 3 months reported using some form of contraceptive at last sexual encounter.

Program type

Overall, the majority of implemented EBIs provided sexuality education (70%—74%) followed by abstinence-based curricula (13%—18%); a smaller proportion were youth development and clinic-based programs (Table 3). The EBIs implemented most often from 2012 to 2014 were Reducing the Risk (31%), Making Proud Choices (29%), Making A Difference (21%), and Be Proud Be Responsible (17%). Reducing the Risk, Making Proud Choices, and Be Proud Be Responsible are comprehensive sexuality education programs, and Making A Difference is an abstinence-based curriculum. These programs are curriculum based and delivered in 6—16 sessions. The percentage of youth who had ever had sex of those surveyed was higher for clinic-based programs than other settings. Sexuality education, as compared to abstinence-based or youth development EBIs, served more youth who reported having had sex. In many cases, abstinence-based and youth development programs were implemented among younger youth who are less likely to have had sex.

Table 3:

Number of youth served and number reporting ever having sex across CDC/OAH community-wide initiative program partners by type of program, 2012—2014

Type of program	Number of programs (%)			Youth served (%)			Total number who ever had sex (%)[a]
	2012	2013	2014	2012	2013	2014	2012—2014
Sexuality education	66 (74)	101 (70)	103 (72)	3,570 (72)	8,550 (70)	11,380 (59)	6,569 (42)
Abstinence based	13 (15)	26 (18)	19 (13)	916 (18)	2,776 (23)	4,019 (21)	1,131 (26)
Youth development	6 (7)	13 (9)	12 (8)	206 (4)	341 (3)	3,295 (17)	241 (34)
Clinic	4 (4)	5 (3)	9 (6)	262 (5)	505 (4)	738 (4)	722 (71)

CDC = Centers for Disease Control and Prevention; OAH = Office of Adolescent Health.

Total number of youth who ever had sex is only reported for youth who participated in surveys before interventions were implemented; therefore, it is not a percentage of total youth served.

Setting

The largest percentage of youth were reached through schools (ranging from 44.2% in 2012 to 71.6% in 2014) followed by community-based organizations (ranging from 14.5% to 15.8%) (Table 4). Youth were also reached by organizations serving special populations (i.e., foster care, pregnant and parenting teens, homeless shelters, and juvenile justice), faith-based organizations, health centers, the military (one organization partnered with a local military base), and colleges. Schools, community-based organizations, and faith-based organizations implemented greater numbers of EBI cycles than other settings. The proportion of youth who reported ever having sex at baseline was higher for youth served by organizations that worked with military settings (86%), health centers (74%), and special populations (56%) than organizations that worked with schools (30) or community-based organizations (45).

Table 4:

Number of youth served and number reporting ever having sex across CDC/OAH community-wide initiative program partners, by setting, 2012—2014

Setting	Number of EBIs (cycles)			Youth served (%)			Total number who ever had sex (%)[a]
	2012	2013	2014	2012	2013	2014	2012—2014
School	22 (88)	45 (287)	62 (530)	2,234 (44.2)	6,252 (51.4)	13,908 (71.6)	3,385 (30)
Community-based organization	27 (64)	38 (177)	45 (233)	733 (14.5)	2,629 (21.6)	3,075 (15.8)	2,266 (45)
Special populations	6 (6)	21 (62)	15 (71)	197 (3.9)	916 (7.5)	702 (3.6)	776 (56)
Military	0 (0)	2 (27)	4 (30)	0 (.0)	417 (3.4)	557 (2.9)	640 (86)
Faith-based organization	10 (34)	15 (48)	39 (9)	450 (9.0)	575 (4.7)	491 (2.5)	414 (36)
Health center	1 (−)	5 (−)	4 (−)	94 (1.9)	377 (3.1)	363 (1.9)	431 (74)
College	1 (2)	3 (6)	3 (12)	49 (1.0)	176 (1.5)	191 (1.0)	221 (63)
Other	8 (18)	16 (69)	3 (19)	323 (6.4)	830 (6.8)	145 (.8)	451 (46)

CDC = Centers for Disease Control and Prevention; OAH = Office of Adolescent Health.

Total number of youth who ever had sex is only reported for youth who participated in surveys before interventions were implemented; therefore, it is not a percentage of total youth served.

Fidelity

On average, fidelity scores were relatively high across program sites that reported this outcome (2012, N = 75; 2013, N = 120; and 2014, N = 130). From 2012 to 2014, mean selfreport fidelity scores ranged from 89% to 99% of activities being completed. Observations for fidelity of implementation were conducted for 36% of program partners in 2012, 53% in 2013, and 49% in 2014. On average, fidelity observation scores remained high across program sites; more than 82% of intervention activities were implemented as planned.

Discussion

Consistent with the goals of the EBI component of the CWI, state and community-based organizations increased their capacities to support EBI delivery among program partners in their communities from 2011 to 2014, and program partners served more youth with EBIs during 2012—2014. Our findings showed that technical assistance providers increased their knowledge and confidence in supporting PSBA-GTO use among program partners. During the planning year in 2011, state and community-based organizations received training that introduced them to PSBA-GTO, and technical assistance providers began receiving intense T&TA to support program delivery. Thus, increases in capacity were highest from the planning year (2011—2012) to the first year of implementation. It is not clear that these gains are associated with better program delivery, yet state and community-based organizations would not be able to support EBI delivery without adequate capacity and T&TA from national technical assistance providers.

T&TA was an important factor in supporting capacity-building efforts for program partners. State and community-based organizations provided T&TA on several topics directly related to program delivery, including curricula trainings, program planning, and continuous quality improvement, with most T&TA focused on ensuring adequate capacity. Specifically, state and community-based organizations worked to ensure that program partners were ready to deliver EBIs to meet project goals of reaching many youth. To contribute to program quality and to meet CDC evaluation requirements, state and community-based organizations also provided program partners with training on evaluation, data collection, data management, and data analysis.

EBIs offer the opportunity for population-level effects; with adequate support, communities can implement these programs. Many youth can be reached, but successful implementation requires time and technical assistance and training. Although fewer youth were reached during the first implementation year, their number more than doubled by the second year. Although data were incomplete on all participants, the project also reached sexually active youth. The proportion of sexually active youth was lower than the proportion of sexually active youth reported nationally (46.8%) by the Youth Risk Behavior Surveillance System in 2013 [7]. This discrepancy may be caused by the fact that some school partners were more likely to serve younger youth who may not yet be sexually active. Whereas schools reached the largest numbers of youth, community-based organizations, health centers, military, colleges, and special population settings reached a larger proportion of youth who were sexually experienced.

Lessons learned

There are several important lessons learned from the implementation of EBIs in CWIs. First, set clear expectations from the beginning. Initial CDC guidance required 10 program partners but did not specify how many youth should be reached; the focus was subsequently shifted from the number of partners to the number of youth served by EBIs. Second, some state and community-based organizations partnered with organizations that had limited reach and capacity. In 2012, CDC, with input from national technical assistance providers, implemented a continuous quality improvement process whereby performance data and recommendations based on the data were shared with state and community-based organizations in organization- specific reports. From this feedback process and new CDC guidance, state and community-based organizations were required to develop action plans to reach more youth. In some cases, better program planning with partners or creating new or different partnerships was required due to limited capacity to deliver programs and recruit and retain youth.

Strategic efforts to reach youth through EBIs to reduce teen pregnancy are critical—another important lesson learned. More youth can be reached by partnering with schools and delivering programs during the school day. Delaying sexual activity remains important, so having EBIs in middle and high schools to reach large numbers of sexually inexperienced students are critical. Given this project’s focus on increasing links to contraceptive and reproductive health services, vulnerable youth should also receive EBIs because many are sexually active and need clinical prevention awareness and services. The activities implemented in other components of the initiative supported implementation of EBIs. Community mobilization and stakeholder education activities assisted in building partnerships for EBI implementation, particularly in schools, while focusing on engaging diverse communities may have helped state and community-based organizations to identify hard-to-reach populations and those youth most at risk of pregnancy.

Some implementation barriers included the following: delays in training for state and community-based organizations and program partners; programs that were not packaged for training; an inability to fund program partners because of local agency policies; and a lack of support for implementation by schools in the community. The most commonly used programs at the beginning of the project were packaged with readily available training. As trainings became available for other curricula, state and community-based organizations communities began using more diverse programs, including youth development and special population-focused curricula (i.e., Project AIM and Sexual Health for Adolescent Risk Protection). Working with partners who (1) had experience delivering teen pregnancy-prevention programs, (2) had worked with state and community-based organizations before, and (3) had organizations with consistent access to youth all helped implementation efforts.

Limitations

Our findings are subject to several limitations. First, although we are beginning to understand how to assess and increase capacity, the relationship between capacity and quality programming to achieve desired youth outcomes is not clear. Further, the capacity measures were limited to topics specific to the program and did not include other indicators of capacity, such as organizational leadership, adaptability, partnerships, and financial and technological resources.

Second, the lack of consistent methods of collection for youth outcomes, limited the number of organizations collecting youth risk information and comparison groups for understanding the effects of the intervention. Communities implemented different programs to meet their specific needs, and outcomes should be measured based on the differing goals of each intervention. Given the focus on community- level change for the overall project, it was not possible to have comparison groups within the target community, and it was beyond the capacity of state and community-based organizations to track comparison youth outside of the community.

Other limitations include that although fidelity scores were generally high, there was considerable variability and most partners did not validate fidelity with observations. The number of youth served was determined based on participation in at least one session of an EBI, and information on attendance to capture the number of youth who participated in most program sessions was not consistently available and was therefore unable to be reported. Last, T&TA hours, which require consistent monitoring, may have been low because of a lack of recording by technical assistance providers.

In summary, EBIs can be implemented across settings in communities to reach many youth in a manner that ensures fidelity. To support implementation of EBIs, technical assistance and training from organizations with adequate capacity is required. Initiatives to expand EBIs can engage schools to increase the number of youth reached while also focusing on partnerships with organizations that reach higher risk youth. Further study is needed to understand factors that contribute to effective delivery and to understand community-level impacts.