Understanding quality improvement collaboratives through an implementation science lens.

Quality improvement collaboratives (QICs) have long been used to facilitate group learning and implementation of evidence-based interventions (EBIs) in healthcare. However, few studies systematically describe implementation strategies linked to QIC success. To address this gap, we evaluated a QIC on colorectal cancer (CRC) screening in Federally Qualified Health Centers (FQHCs) by aligning standardized implementation strategies with collaborative activities and measuring implementation and effectiveness outcomes. In 2018, the American Cancer Society and North Carolina Community Health Center Association provided funding, in-person/virtual training, facilitation, and audit and feedback with the goal of building FQHC capacity to enact selected implementation strategies. The QIC evaluation plan included a pre-test/post-test single group design and mixed methods data collection. We assessed: 1) adoption, 2) engagement, 3) implementation of QI tools and CRC screening EBIs, and 4) changes in CRC screening rates. A post-collaborative focus group captured participants' perceptions of implementation strategies. Twenty-three percent of North Carolina FQHCs (9/40) participated in the collaborative. Health Center engagement was high although individual participation decreased over time. Teams completed all four QIC tools: aim statements, process maps, gap and root cause analysis, and Plan-Do-Study-Act cycles. FQHCs increased their uptake of evidence-based CRC screening interventions and rates increased 8.0% between 2017 and 2018. Focus group findings provided insights into participants' opinions regarding the feasibility and appropriateness of the implementation strategies and how they influenced outcomes. Results support the collaborative's positive impact on FQHC capacity to implement QI tools and EBIs to improve CRC screening rates.

Background

Quality improvement collaboratives (QICs) emerged in the mid-1990s as a promising approach to facilitate group learning and systems change across a range of healthcare settings and topics. The Institute for Healthcare Improvement (IHI) “Breakthrough Series” model () is the most widely used collaborative approach, and has been replicated with thousands of healthcare teams and organizations (Institute for Healthcare Improvement, n.d.). Federally Qualified Health Centers (FQHCs) have a long history of participation in collaboratives, most notably the Health Resources and Services Administration (HRSA)-funded Health Disparities Collaboratives for improving the quality of chronic care and reducing disparities. As the popularity of QICs rapidly increased, a call for a stronger evidence-base was issued in 2004 given the lack of rigorous study designs demonstrating positive effects on patient outcomes (Mittman, 2004).

Fifteen years later, randomized controlled trials of QICs are still rare, but systematic reviews provide evidence that they achieve positive - although limited and variable - improvements in processes of care and clinical outcomes (Chin, 2010; Schouten et al., 2008; Wells et al., 2018). Schouten and colleagues reviewed nine controlled studies; two had little to no effect; five had mixed effects, and two had positive effects on outcomes (Schouten et al., 2008). Chin reviewed experimental, quasi-experimental, and descriptive studies that evaluated the HRSA Health Disparities Collaboratives, and found that some had a positive impact on processes of care and clinical outcomes whereas others only improved processes of care (Chin, 2010). Wells and colleagues conducted a systematic review of 64 studies of QICs that varied in settings, topics, and populations; 83% reported improvements in primary outcome measures (Wells et al., 2018). Although these findings are encouraging, authors noted that many studies provided insufficient description of the collaborative, its activities, and levels of engagement (Wells et al., 2018). The variability across QICs has been well-documented (Schouten et al., 2008; Wells et al., 2018; Strating et al., 2011), creating an imperative for detailed descriptions of collaborative components so that reasons for variation can be studied and effective models replicated. As Nadeem and colleagues note, collaboratives provide an opportunity to intervene on multi-level factors that are known to influence outcomes such as characteristics of the intervention, fit with providers and the organization, leadership, culture and climate, and alignment with the policy environment (Nadeem et al., 2016). Consequently, QICs in FQHCs can move beyond addressing diseases or conditions by increasing overall capacity to address population health (Chambers, 2018).

Implementation science, specifically the widely used compilation of implementation strategies published by Powell et al. (2015) and implementation outcomes published by Proctor et al. (2011), can support a more systematic exploration of how QICs lead to positive outcomes. The purpose of this paper is to explain how we applied an implementation science lens to our QIC evaluation by: 1) aligning collaborative and FQHC activities with implementation strategies from Powell’s taxonomy (Powell et al., 2015), and 2) measuring implementation and service outcomes as defined by Proctor (Proctor et al., 2011). Our goal is to describe the strategies used by the collaborative and explore the mechanisms through which the collaborative led to improvements in colorectal cancer (CRC) screening rates. The results will accelerate our understanding and application of effective QICs for not only cancer screening, but also broader prevention activities such as healthy eating and physical activity, thus improving quality of care, health outcomes, and population health.

Conceptual framework

In 2018, the American Cancer Society® (ACS) partnered with the North Carolina Community Health Center Association (NCCHCA) to sponsor a CRC screening QIC and invited UNC Chapel Hill, part of the Cancer Prevention and Research Control Network (CPCRN, www.cpcrn.org) to serve as the external evaluator. Fig. 1 provides an overview of the conceptual framework guiding the evaluation, with the implementation strategies informed by Powell (Powell et al., 2015) and the implementation and service outcomes informed by Proctor (Proctor et al., 2011).

Fig. 1.

Conceptual framework for the quality improvement collaborative.

The framework describes the implementation strategies employed at two levels: the level of the facilitation team (ACS and NCCHCA staff) that led the collaborative and the level of the implementation team (FQHC team members) that participated in the collaborative. The model also specifies the implementation, service, and population outcomes that the strategies were intended to achieve.

We applied terms from Powell’s taxonomy (Powell et al., 2015) to each of the implementation strategies listed in the conceptual framework and refer to them in the descriptions below (in italics).

Implementation strategies: collaborative facilitation team

The NCCHCA recruited 13 FQHCs to apply for the collaborative and selected nine to participate. They obtained formal commitments from the leadership of each FQHC in the form of a written agreement. ACS provided funding ($20,000 total) to cover transportation and lodging for FQHC staff to participate in the collaborative. Then, ACS staff conducted training for FQHC staff during two in-person collaborative meetings and monthly virtual meetings, with a focus on using quality improvement (QI) tools to implement CRC screening evidence-based interventions (EBIs). The QI tools were adapted from materials developed by the Institute for Healthcare Improvement, HRSA, and the Centers for Medicare and Medicaid Services (CMS) and included aim statements, current and future state process maps, root cause and gap analysis, and Plan-Do-Study-Act (PDSA) cycle templates (see Table 1). During the monthly virtual meetings, ACS shared graphs of each FQHC’s screening rates and facilitated discussions on barriers and facilitators (i.e., audit and provide feedback). ACS staff delivered phone and in-person coaching (i.e., provided facilitation) for each FQHC.

Table 1

Quality improvement tools and activities[a].

Tool	Activity
Aim statements	Teams used a standardized worksheet to lead them through the process of creating broad aims and SMART goals (specific, measurable, achievable, relevant and time-bound) for improving CRC screening rates.
Current and future state process maps	Teams were instructed in using process mapping as a tool to collaboratively document roles and responsibilities for each step in the current and proposed CRC screening process.
Gap and root cause analysis	Teams were introduced to the 5 Whys worksheet and Fishbone Diagram, both of which aid teams in identifying barriers and facilitators to CRC screening.
Plan-Do-Study-Act (PDSA) cycles	Teams used a standardized PDSA worksheet to guide them in planning, executing, and evaluating small improvement cycles for CRC screening processes.

Templates available at http://www.ihi.org/resources/Pages/Tools/Quality-Improvement-Essentials-Toolkit.aspx.

Implementation strategies: implementation team

As depicted in Fig. 1, the collaborative’s strategies were intended to build FQHC teams’ capacity to select and apply implementation strategies. Each FQHC convened an implementation team of three members to participate in the collaborative. ACS recommended QI staff, but centers ultimately determined the composition of their teams. Teams used QI tools to identify barriers and facilitators to CRC screening and to implement Plan-Do-Study-Act (PDSA) cycles (i.e., conduct cyclical small tests of change). The implementation teams also selected CRC screening EBIs on which to focus.

Implementation outcomes

To have an impact, the collaborative needed to achieve implementation outcomes including FQHC leadership adoption of the collaborative, FQHC team engagement in collaborative activities, and FQHC system implementation of CRC screening EBIs and implementation and spread of QI tools to other types of screening and health conditions.

System and population outcomes

The goal of the collaborative was to improve CRC screening rates and thereby reduce colorectal cancer morbidity and mortality, with a particular focus on reducing disparities.

Methods

Evaluation design

We used a pre-test/post-test single group design and mixed methods data collection. Sources of data included self-administered questionnaires and inventories using PDF forms and SurveyMonkey®, copies of completed QI tools, Poll Everywhere questions during monthly virtual meetings, monthly and annual screening rates from FQHCs’ electronic health records, and published data from HRSA’s Uniform Data System (UDS). All data collected by ACS and the NCCHCA were deidentified and sent to the UNC evaluation team for analysis. The UNC evaluators also conducted a post-collaborative focus group to capture contextual data. The evaluation plan was reviewed by the University of North Carolina Institutional Review Board and determined to be Not Human Subjects Research.

Implementation outcome measures

The evaluation assessed the collaborative’s impact on implementation outcomes. Table 2 provides an overview of the implementation outcomes, how they align with those described in Proctor’s taxonomy, and how they were defined and measured.

Table 2

Implementation outcomes.

Proctor et al.’s implementation outcomes (Proctor et al., 2011)	Collaborative implementation outcomes	Definition	Measurement
Adoption - setting	Adoption	FQHCs that signed the participant agreement (n = 9)	Center demographics from the Uniform Data System (Health Resources and Services Administration Health Center Program: 2018. Health Center Data, n.d.-b)
Adoption - provider	Engagement	FQHC and individual attendance at in-person collaborative meetings and monthly calls	Lobbyguard software and Poll Everywhere item
Implementation Fidelity	Implementation of QI tools	QI tools that were filled out and acted upon by FQHC staff during the collaborative	Log of completed QI tools
Penetration	Spread of QI tools	Tools that were applied to other screening/conditions or other populations	Baseline and follow-up inventory and post-collaborative focus group
Intervention fidelity	Implementation of CRC screening EBIs	Implementation of interventions recommended by the Guide to Community Preventive Services	Baseline and follow-up inventory
Effectiveness	CRC screening rates	Percentage of adults 50–75 years of age who had appropriate screening for colorectal cancer	The 2016, 2017, 2018 Uniform Data System (Health Resources and Services Administration Health Center Program: 2018. Health Center Data, n.d.-b)
Appropriateness	Influence of implementation strategies on CRC screening rates	Implementation staff perceptions of how collaborative strategies led to systems change and increase in screening rates	Post-collaborative focus group

Adoption

Data were collected on the nine FQHC systems that agreed to participate in the collaborative. Descriptors of the FQHCs (including all clinical sites) and their patient populations were reported by the participating centers or extracted from the UDS 2017 Health Center Program Awardee Data (Health Resources and Services Administration Health Center Program: 2017 Health Center Data, n.d.-a).

Engagement

Data were collected on FQHC staff participation in collaborative activities. LobbyGuard Visitor Management Software was used to track in-person meeting attendance. For the monthly virtual meetings, attendance was monitored through a Poll Everywhere question.

Implementation and spread of QI tools

ACS staff maintained a log of completed QI tools from each FQHC. Spread of the QI tools was assessed via the online inventory described below and through the post-collaborative focus group.

Implementation of CRC screening EBIs

The ACS administered baseline and follow-up inventories that assessed each FQHC’s Fecal Immunochemical Test (FIT) kit and colonoscopy capacity (i.e. “Are you currently experiencing any problems with colonoscopy capacity?”), use of CRC screening EBIs (e.g. reminder systems, patient education), and implementation of QI tools (i.e. “What QI Process tools did you use for colorectal cancer screening improvement after the collaborative?”). The CRC screening EBIs included in the inventory were those recommended by the Guide to Community Preventive Services (Community Preventive Services Task Force, n.d.). The electronic 39-item baseline inventory was completed between January and February 2018, and the follow-up 45-item inventory was completed between February and March 2019.

Influence of implementation strategies on outcomes

Thirteen participants representing seven FQHCs attended an in-person 90-minute focus group in February 2019, at the end of the collaborative. A structured focus group guide was used to inquire about participants’ perceptions of how implementation strategies led to improvements in CRC screening, and how the collaborative model could be improved.

System outcome measures

Changes in colorectal cancer screening rates

For the participating FQHCs, annual CRC screening rates for 2016, 2017, and 2018 are from the UDS (Health Resources and Services Administration Health Center Program: 2018. Health Center Data, n.d.-b). The SurveyMonkey® questionnaire captured monthly rates on a rolling year schedule.

Analysis plan

We used a mixed methods approach for data collection and analysis. Descriptive statistics were calculated for adoption, engagement, implementation of QI tools and CRC screening EBIs. In the baseline and follow-up inventories, qualitative data from open-ended responses were used to provide context for the quantitative findings. The focus group discussion was recorded, transcribed, and coded by two independent coders, using a content analysis approach (Hsieh and Shannon, 2005). The two reviewers met to reconcile any discrepancies in coding and to summarize findings into themes. Content analysis was guided by elements of the evaluation model with the goal of exploring participants’ perceptions of implementation strategies and the relationship between those strategies and implementation outcomes. The identified themes were reviewed with members of the facilitation team. To assess changes in screening rates before the collaborative (2016–2017) and during the collaborative (2017–2018), we calculated weighted averages and compared the percent change between the two time periods.

Results

Sample

Table 3 displays the characteristics of the nine participating FQHCs, which had between one and six clinical sites except for one large center with 23 clinical sites. In 2017, the number of patients per center between the ages of 50–74 ranged from 633 to 14,924. A large proportion of the FQHCs’ populations were racial minorities (mean = 65.6%); patients of Hispanic ethnicity (mean = 24.7%); and uninsured (mean = 36.7%). Each FQHC was asked to create a three-member implementation team; members included quality improvement staff, clinical administrators, and providers (physicians, mid-levels, nurses). Among those who completed the baseline inventory (n = 22), half had a low level of experience with quality improvement activities and a third had worked in an FQHC setting for less than one year.

Table 3

Organizational characteristics of participating FQHCs, 2017.

FQHC	Characteristics of FQHCs (N = 9)
	# of clinical sites	% of patients screened for CRC[a]	# of patients ages 50–74[a]	% racial minority[a,b]	% hispanic ethnicity[a,b]	% uninsured[a,c]
A	2	41.3	1280	46.9	3.2	9.3
B	23	35.6	14,924	51.9	20.5	33.9
C	1	19.8	2419	82.4	39.3	54.5
D	5	12.8	2921	86.2	5.6	25.1
E	2	25.9	854	29.0	15.6	30.3
F	5	45.0	5582	76.7	50.5	48.6
G	6	25.7	1539	92.5	26.1	9.1
H	3	17.1	1186	83.5	29.5	74.7
I	1	32.9	633	40.9	32.4	44.5

Data are from the 2017 UDS.

Race and ethnicity are for the whole patient population.

Insurance status is for patients 18 and over.

North Carolina had 40 FQHCs in 2018, 13 (32.5%) of which applied to participate in the QIC. Out of the 13, the ACS and the NCCHCA selected nine centers with the lowest screening rates and the capacity to fully engage in the QIC. Capacity was informally assessed based on the FQHCs’ successful participation in prior ACS and NCCHCA initiatives, verbal commitment to the QIC by center leadership, and priority the FQHC gave to CRC screening compared to other UDS clinical care measures.

Fig. 2 displays FQHC representation and individual attendance for the two in-person collaborative trainings and the monthly virtual meetings. All FQHCs sent team members to both in-person meetings, with the proportion of team members attending ranging from 100% (27/27) at the first meeting to 89% (24/27) at the second meeting. FQHC representation at the virtual meetings was 100% (9/9) for the first four months, then dropped to 67% (6/9) by the end of the collaborative. Individual team member attendance at the virtual meetings peaked at 74% (20/27) for the first four months, then decreased steadily over time with a low of 41% (11/27) in October and December.

Fig. 2.

Quality improvement collaborative attendance, 2018.

a No call was held in November.

Implementation of QI tools

Most FQHC teams completed all the QI tools. All nine teams completed and submitted aims statements, eight out of nine teams submitted their current state process map, and all nine submitted their future state process map. Eight out of nine teams completed their gap and root cause analysis, and seven out of nine submitted PDSA cycle worksheets. For the process maps, FQHC teams took different approaches. For example, some FQHC teams completed a map for each clinical site while others completed one for the entire FQHC system. Some sites completed a map for one process while others completed maps for multiple processes (see Appendices A and B for sample process maps).

As measured by the baseline and follow-up inventory, barriers to FIT-based CRC screening were reduced during the collaborative. Before the collaborative, two FQHC teams documented problems with limited testing supplies and patients not returning their stool cards. On the follow-up inventory, no sites reported problems with FIT kit capacity and all nine clinics were performing FIT kit tracking, compared to six at baseline. Problems with helping patients obtain colonoscopies, however, did not change over time. Three FQHC teams reported the exact same barriers on the inventory both pre- and post-collaborative, which included patients’ lack of insurance or inability to afford the procedure, too few providers who offer free or reduced-fee colonoscopies, and delays in processing paperwork for charity care. Table 4 displays the change in CRC screening EBIs that FQHCs were implementing before and after the collaborative. Standing orders for CRC screening were implemented by one additional FQHC. Patient reminders for screening increased for phone and mail, but not for text and patient portal. Provider prompts increased in the EHR, but not paper reminders. Reminders through huddles or nurse review decreased. Provider assessment and feedback increased, but not patient education.

Table 4

CRC screening EBIs implemented by FQHCs (N = 9).

		Yes (Pre)	Yes (Post)	Difference[a]
	CRC Screening Standing Orders	5	6	1
CRC Screening Reminders for Patients	Live Phone	3	5	2
	Automated Phone	1	2	1
	Mail	4	7	3
	Text Message	1	1	0
	Patient Portal	1	1	0
Provider Prompts and Type	Provider Prompt	5	6	1
	Prompts in EHR	1	4	3
	Paper reminders in chart	0	0	0
	Review scheduled patients in huddle	3	1	−2
	Nurse reviews screenings and alerts provider	1	0	−1
Provider Assessment and Feedback	EHR can provide reports of those due for screening, screening rates, and screening modality	4	7	3
Patient education on colorectal cancer screening options	One-on-One	9	9	0

Green = increase, yellow = no change, red = decrease.

Spread

In the follow-up inventory, participants reported applying the QI tools to diabetes, vaccinations for HPV and pneumonia, and workflows such as scheduling and no-shows.

Table 5 summarizes qualitative themes related to each of the implementation strategies, with illustrative quotations. Obtaining a formal commitment from FQHC directors and providing funding for travel motivated FQHC team members to participate in the collaborative activities. The in-person trainings were the most frequently mentioned and most highly valued strategy. Participants appreciated the comprehensive overview of the QI process and tools. Participants also valued the opportunity that in-person trainings provided for them to interact with peers and obtain hands-on experience with the QI tools. They even suggested that trainers spend less time on CRC screening EBIs so that more time could be devoted to QI processes and tools. Participants found it challenging to make time for the monthly virtual meetings but valued the facilitation provided by ACS staff. The audit and feedback on their CRC screening rates was helpful because it generated friendly competition and increased team accountability. Lastly, they felt the QI tools were highly valuable, with the exception of the Fishbone Diagram (a visualization tool for categorizing the potential causes of a problem in order to identify its root causes) and the Five Whys worksheet (an iterative interrogative technique used to explore the cause-and-effect relationships underlying a particular problem).

Table 5

Implementation strategies.

Implementation strategies	Themes	Quotations
Formal commitment	Motivated engagement	“I said we signed the contract to be on these calls. We have to do this, because we said we were going to. And if we’re not going to then we shouldn’t have done it.”
Funding	Motivated engagement	“I’m not going to lie, girls, we’re broke. That we were able to get it paid for, that sealed the deal.”
Training: in-person meetings	Highly valued for providing: An overview of the steps involved in the QI process An opportunity for peer networking Hands-on experience using QI processes and tools	They explained “why you have to go through all the steps, which in turn I could take to my team and say listen, guys, we have to do it this way because it makes more sense if we start it at the beginning instead of jumping to the end.”
		“I really like the interactive collaboration and bouncing ideas off- oh, this worked for them; let’s try it for us.” “Because people would think of things that I didn’t think of and it would make it so, okay, almost like an idea think box that I could pick from.”
		“By the time you actually got back to your health department, you had worked through this whole process and understood it, so that way you could start with your team and know what you were doing.”
Training: virtual meetings	Were difficult to prioritize over clinic needs	“When you have webinars in your health center, you get pulled away when there’s a crisis going on.”
Facilitation	Valued the personalized problem solving and support	“It was helpful to have an ACS member assigned to you because if I had questions, I could just send an email.”
Audit and feedback	Valued for: Generating friendly competition Holding them accountable	“Them knowing how they were performing and showing them how they were performing, and it was just a motivator to do it and to get everybody on board and everybody to work together.”
		“There are so many plates in the air all the time…so we couldn’t really lose momentum because we kept having to put in our data.”
Implementation team	Need to have the right people on the team	“It cannot just be QI people. There needs to be at least one clinical person on the team because those clinical people are the ones that have to document. They’re the ones that have to be in the face of the patient helping promote this.”
Using QI tools to identify barriers and facilitators	Valued the process maps as a way to identify barriers Did not value the Five Whys and Fishbone Diagrams	“We only had one referral specialist. We realized that she needed help, right there in that room during the boot camp…so we did bring on key staff throughout the year because of that current state and future state [process map].”
		“Maybe you don’t have to have the Five Whys and the Fishbone.”
		“The Five Whys was kind of an out there, like where does it go?”
Using QI tools to conduct cyclical, small tests of change	PDSA cycles allowed participants to see if a change worked and demonstrate progress	“If I implement a PDSA, I can see immediately if that worked.”

System outcomes

In 2017, eight of the participating FQHCs served a total of 31,338 patients ages 50–74 (the only year in which age-specific data were available from the ACS). The percentage of patients who received appropriate CRC screening increased for all but one of the FQHCs in 2017 (the year prior to the collaborative) and 2018 (the year in which the QI Collaborative took place). More importantly, the weighted average of the percentage of patients who had received appropriate screening increased by 8.0% from 2017 to 2018 as compared to 3.3% from 2016 to 2017 (see Table 6).

Table 6

Colorectal cancer screening rates for participating FQHCs: 2016, 2017, 2018.

FQHC “I” was not in operation until 2017 and was excluded from the analysis.

Discussion

Using an implementation science lens through which to evaluate QICs can promote standardized terminology and facilitate more detailed description of collaborative strategies, which allows for easier replication and synthesis of the model (Proctor et al., 2013). The goal of this paper was to use Powell’s (Powell et al., 2015) implementation strategies and Proctor’s (Proctor et al., 2011) implementation outcomes to describe the mechanisms through which the collaborative led to improvements in CRC screening rates. To further our understanding of these mechanisms, we used qualitative methods to capture participants’ perceptions of the collaborative activities and explain how they may have contributed to positive implementation outcomes.

The collaborative experience

In terms of engagement, we observed that individual participation in the virtual meetings declined over time, a pattern that has been observed in other QICs (Colon-Emeric et al., 2006). Requesting that all three implementation team members attend every meeting may have been too burdensome. However, nearly every FQHC sent at least one team member to all 11 collaborative gatherings. Focus group participants attributed this strong commitment to signing a contract and accepting additional funding. The composition of the participants on the implementation team varied between those who had substantial experience with quality improvement and those who had less experience and were new to the FQHC setting. While the study was not designed to analyze the influence of team member characteristics on implementation and outcomes, the consensus from the focus group findings was that the capacity-building was useful for all participants, regardless of their level of familiarity with quality improvement.

Almost all implementation teams provided documentation of how their clinics implemented QI tools, and they reported increased use of 7/13 CRC screening EBIs. A few of the obstacles to implementing EBIs that were mentioned in the focus group included lack of capacity in the EMR and slow uptake of patient portals. When asked about the content of the in-person trainings, focus group participants wanted the ACS to prioritize general QI capacity-building over specific information on CRC screening. One possible reason is that staff may have had a high level of knowledge about CRC screening EBIs prior to starting the QI Collaborative. A recent survey of FQHCs across eight states found that 77% were fully implementing at least one recommended CRC screening intervention from the Community Guide (Adams et al., 2018). Results from the follow-up inventory indicated that capacity for distributing FIT kits and following up with patients was improved as a result of the QIC. However, problems with colonoscopy capacity did not appear to be resolved at the end of twelve months. This is not surprising given that North Carolina does not have funds for diagnostic colonoscopies for the uninsured. FQHCs have had to develop relationships with local gastroenterology practices to donate endoscopy services or creatively manage revenue to supplement the cost of colonoscopies (Weiner et al., 2017). Barriers such as understaffing and lack of access to colonoscopy have broader organizational and policy-level implications which need to be addressed in future QIC research. Implementation science, with its emphasis on multi-level determinants and strategies, can inform how best to address these structural barriers.

Because the long-term goal for ACS is to replicate and disseminate the QIC model for all cancer screenings, there was an emphasis on collecting and systematically applying recommendations for improvement. In subsequent QICs, the ACS is incorporating the findings from this evaluation such as keeping the focus on quality improvement, increasing peer-to-peer best-practice sharing and troubleshooting, sharing identified screening rates every month, and having ACS staff conduct at least one site visit for every FQHC. Appendix C includes a table with recommendations from the post-collaborative focus group.

Impact on CRC screening rates

In the past several years, FQHCs have made significant strides in improving CRC screening among their patient population (National Colorectal Cancer Roundtable, n.d.; Riehman et al., 2018). In 2017, the FQHCs that participated in the QIC had an average weighted screening rate of 32.7%. Their rate was lower than the non-participating FQHCs in North Carolina (37.7%), and the national FQHC average (42.0%), which gave them room to improve. With one exception (FQHC B), the centers were relatively small with 6 or fewer clinical sites. Their patient populations, however, were diverse. As safety net providers, many served a patient population with a high proportion of racial/ethnic minorities and the uninsured, but three of the centers had less than 10% uninsured and/or less than 10% minorities. Despite this variability, all but one of the FQHCs increased their CRC screening rates over the one-year collaborative. There is evidence that the QIC had an impact as the weighted average of the percentage of patients who received appropriate screening was 4.7 points higher in 2018 than in 2017. Also, the percentage increase in rates for the cohort outpaced state-level and national trends. In North Carolina, overall CRC screening rate for the 31 centers that did not participate in the collaborative increased by 1.8% from 2017 to 2018. For all FQHCs in the U.S., the rate increased 2.1% from 2017 to 2018 (Health Resources and Services Administration Health Center Program: 2018. Health Center Data, n.d.-b). Interestingly, a randomized controlled trial of a QIC in 23 primary care clinics did not demonstrate statistically significant differences in CRC screening rates. The authors state “Advancing the knowledge base of QI interventions requires future reports to address how and why QI interventions work rather than simply measuring whether they work” (Shaw et al., 2013), which was the goal of this evaluation.

Strengths of the quality improvement collaborative model

One of the strengths of this QIC was that every center received facilitation (i.e., coaching) from an assigned ACS Primary Care Manager. All Care Managers had received extensive virtual training through the IHI Open School (http://www.ihi.org/education/IHIOpenSchool/) and in-person training through ACS and HealthTeamWorks (https://www.healthteamworks.org/). They provided core practice facilitator tasks noted by others (Bidassie et al., 2015) such as: context assessment; guiding PDSA cycles and use of process improvement tools; identifying resources and making referrals; holding teams accountable for plan implementation during site visits; and providing support and encouragement. Another strength was the quality and reputation of the facilitator team. In their taxonomy of attributes of successful collaboratives, AHRQ found that the degree of credibility of the host or convener and the leadership was an important component of the social system (Nix et al., 2018). The ACS is a well-respected organization with a high level of expertise in the topic area, and the NCCHCA is a trusted partner with the mission of serving its FQHC members. These two organizations were ideally suited to convene and support the QIC. Features of the QIC also mapped back to five essential features that determine short-term success, according to a study of 182 teams that participated in seven QICs (Strating et al., 2011) (see Table 7).

Table 7

Essential collaborative features that determine short-term outcomes.

Essential feature	CRC screening quality improvement collaborative
There is a specified topic	The topic was CRC screening
Clinical and quality improvement experts provide ideas and support for improvement	The ACS and the NCCHCA provided cancer screening and quality improvement expertise
Critical mass of teams from multiple sites willing to improve, share and participate	27 individuals from nine FQHCs participated; an average of eight FQHCs had representation at all 13 QI Collaborative meetings
Teams use a model for improvement	The curriculum was based on the IHI Model for Improvement (The Breakthrough Series: Mi’s Collaborative Model for Achieving Breakthrough Improvement Boston, MA, 2003)
Collaborative process involves a series of structured activities	Activities included creating a team, attending in-person and virtual meetings, completing QI tools, implementing screening interventions, and collecting and reporting data

The conceptual framework for the QIC could be generalized to preventive services beyond cancer screening such as nutrition and physical activity guidelines.

Strengths of the evaluation approach

This evaluation plan was guided by a conceptual model that specified implementation strategies and outcomes (e.g., adoption, engagement, and implementation of QI tools and CRC screening EBIs), which enhanced our ability to describe the collaborative activities in detail and link them to improvements in screening rates. Collecting completed QI tools such as aims statements, process maps, and gap and root cause analyses provided proof that the teams were executing EBIs and improvement strategies in a more systematic way, as recommended by Leeman et al. (2019). Because we documented both the collaborative-level implementation strategies as well as the FQHC-level strategies, the study findings contribute to a clearer pathway from the application of evidence to changes in the outcome. Wells et al. (2018) recommend that evaluations of QICs include site engagement measures, which we collected by measuring individual attendance and FQHC-representation at all meetings. Nadeem et al. (2013) advised researchers to provide detailed information on collaborative components, which we laid out in Table 1 and included in the Appendices. Our mixed methods approach and close collaboration with ACS and NCCHCA in interpreting the evaluation results speaks to Solberg’s (2005) observation that “Efforts to evaluate such QICs must therefore include those more subjective but important influences on care quality, not just particular changes in a few measures”. We specifically addressed implementation strategies and outcomes that are often missing from QIC clinical trials, thus promoting transparency and replicability.

Limitations

The results presented in this paper are based on an external evaluation of a real-world collaborative that was not part of a research study, and thus have some limitations. While selection bias may have played a role in the observed increases in CRC screening rates, randomly selected FQHCs with a high dropout rate would have defeated the funder’s purpose of creating a sustainable learning cohort. Given the lack of randomization and a control group, caution must be taken when attributing improvements in screening solely to the QIC. To strengthen our conclusions, we applied weighted averages to assess differences in CRC screening rates between 2016, 2017, and 2018. We also assessed rates among the non-participating FQHCs in North Carolina. Another limitation is that the practice inventory was not a standardized instrument and, in a few cases, the pre and post were completed by different team members. Future evaluations could benefit from standardized, validated measures completed by the same participants, when possible. Only seven of the nine FQHCs were able to travel to attend the in-person focus group, so two centers did not have their perspectives represented in the qualitative results. Sustainment of CRC screening EBIs is a critical implementation outcome which we have not addressed because post-collaborative data have not yet been collected. While nine organizations is a fairly small sample size, it is practical for the purposes of in-person meetings and on-site coaching. As this model is replicated across multiple cohorts of FQHCs, the ability to test for statistically significant changes in CRC screening rates will be increased.

Conclusion

Our evaluation approach is novel in that we incorporated measures and outcomes from implementation science, which enabled us to apply standardized terminology to describe what we observed while also exploring the mechanisms through which the QIC led to improvements in CRC screening rates. For cancer screening specifically, the model is ripe for dissemination to states that have partnerships with the ACS, primary care organizations, and other key stakeholders. The ACS is currently adapting the QIC approach for cancer prevention and control initiatives in North Carolina, South Carolina, Virginia, and Florida. They have engaged the University of South Carolina CPCRN as the external evaluator of a QIC on HPV vaccination with FQHCs in SC. Their evaluation plan builds on the activities described here and encompasses more frequent data collection to monitor and assess intervention outcomes (e.g., baseline, mid-program, and post-program). Additional qualitative data collection will further elucidate contextual factors influencing the QI interventions and CRC screening interventions in the FQHCs, such as how the QI intervention promotes expanded social networks for FQHCs for future collaboration. The ACS, primary care associations, and FQHCs are nationwide, providing an infrastructure that can rapidly and efficiently replicate the QI Collaborative model to deliver life-saving CRC screening to our country’s most vulnerable populations.

Appendix C.

Recommendations for improvements

	Recommendations
In-person and virtual meetings	Encourage FQHCs to create multidisciplinary quality improvement teams including clinical staff and leadership; offer CMEs to incentivize provider participation. Offer a pre-collaborative meeting or phone call to introduce participants to the project. Avoid scheduling collaborative activities at the time UDS data are due (January-February). Provide a platform for sharing resources, ideas and contact information, such as a website or shared folder. Facilitate peer networking by giving each FQHC an opportunity to speak during the virtual meetings Incorporate content on sustainability throughout the collaborative.
Data collection	Plan enough time between the start of the collaborative and the first data collection point for FQHCs to gather accurate data and show improvements. Consider a new data collection timeline, as the current rolling month approach (Feb to Jan, March to Feb) makes it hard to isolate recent improvements “because you really can’t see the changes until the end of the year when you’re doing a rolling month.” Make sure instructions regarding data collection are very clear.
Quality improvement tools	Present additional root cause analysis options other than the 5 Whys or the Fishbone Diagram. Emphasize that quality improvement tools can be used flexibly. Allow FQHCs ample time to practice using the tools during the in-person meetings.