Challenges in assessing population reach in a pragmatic trial.

Strategies and Opportunities to STOP Colon Cancer in Priority Populations (STOP CRC) was a pragmatic cluster-randomized trial conducted at federally qualified health centers and designed to "Reach" as many unscreened patients as possible by directly mailing them fecal screening tests. STOP CRC used an electronic health record registry to identify individuals' needing CRC screening and mail interventions to them. The registry was updated daily removing individuals completing CRC screening or those who no longer were clinic patients. Reach, a component RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance), is defined as the absolute number, percent, and representativeness of individuals "willing to participate in" or "exposed to" an initiative. We describe the complexities of measuring Reach in a pragmatic trial. Overall 21,134 patients were on the registry list for at least one day, with 18,226 remaining after removing patients completing screening before any mailings. Observed Reached (the percent of individuals exposed to the intervention) using each denominator was 30.7% and 35.6% respectively. Reach improved only modestly after accounting for factors that made it impossible for clinics to send mailings. Few differences were observed in demographic and health care utilization factors among individuals Reached versus not Reached, suggesting that health center Implementation was more influential than patients' willingness or ability to participate. A pragmatic definition of Reach that accounted for dynamic changes the absolute number eligible and the proportion exposed was more useful than traditional definitions of Reach. Actual Reach was dependent on Implementation and not patient level characteristics. Clinical Trials Registration Number: ClincalTrials.gov (NCT01742065).

RCT Entities:   Population Interventions Outcomes

Introduction

Research-tested programs to improve rates of CRC screening have shown that mailed fecal test (FIT) outreach leads to significant increases in CRC screening (Coronado et al., 2011; Green et al., 2013; Walsh et al., 2010; Sequist et al., 2009; Myers et al., 2007; Jean-Jacques et al., 2012; Dougherty et al., 2018). While these studies show promising results, none have resulted in widespread adoption of CRC screening practices, particularly in safety net clinics that serve population groups with the lowest rates of CRC screening. Strategies and Opportunities to STOP Colon Cancer in Priority Populations (STOP CRC) was a pragmatic study designed to increase CRC screening in safety net clinics. STOP CRC used a real-time electronic health record (EHR)-embedded registry to identify patients who need CRC screening and to mail FITs directly to them. The goal was to develop an efficient system for mailing FITs to as many age- and screening-eligible patients as possible (i.e. have maximum reach within the clinic population), increase CRC screening, and decrease screening disparities.

RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) is a framework that was developed to encourage consistent reporting of research results and to identify potential factors that might influence intervention impact (Glasgow et al., 1999; Glasgow et al., 2006). One component, Reach, has been defined as the percent and representativeness of individuals willing to participate to an initiative (RE-AIM, n.d.). Reach was designed to test the generalizability, the external validity, and the potential impact of randomized controlled trial (RCT) results. Excluding individuals from a trial who might face greater difficulty engaging in an intervention (e.g. unable to converse in the English language, chronic illnesses) or who may be less willing to provide informed consent might lead to differences in key characteristics between participants nonparticipants groups, and study findings might be less applicable to a broader population (Lee et al., 2017). Recently, Glasgow et al. expanded the definition of Reach beyond the level of “willingness to participate” to the proportion of individuals “exposed” to an initiative (Glasgow and Estabrooks, 2018). Other components of the RE-AIM model include Adoption, the numbers of settings invited and percent willing to host the initiative and reasons why or why not; Implementation, the degree to which the intervention was delivered as intended; Effectiveness, the outcomes achieved and consistency across groups; and Maintenance, of the initiative and long-term effectiveness.

STOP CRCs Reach was expected to be high, as there were few patient-level exclusions (end-stage renal failure and lack of a valid address) and patient consent was not required. However, less than half of the eligible patients were exposed to the intervention (i.e. mailed a FIT Kit). We hypothesized that factors beyond the control of the health centers (such as attrition of patients from the real-time registry, because they had not had a clinic visit in the past 12 months) impacted Reach. We describe here the Reach of the STOP CRC program and the complexities we encountered in attempting to apply standard definitions of Reach to our pragmatic trial.

Methods

STOP CRC (UH3 AT007782) was funded by the National Institutes of Health, Health Care Systems Research Collaboratory program, whose aim is to provide a framework of implementation methods and best practices that will enable pragmatic research within health-care systems (Riley et al., 2013; Collaboratory, 2017). Detailed reports of the trial design have been published (Coronado et al., 2014a; Coronado et al., 2018). We provide a brief overview of STOP CRC study components relevant to calculating Reach.

Study setting

STOP CRC was conducted in 26 individual clinics within 8 Federally Qualified Health Center organizations (FQHCs) in the states of California and Oregon. These clinics are members of OCHIN, a nonprofit computer solution company that provides EHRs and Health Information Technology support to health systems; all participating health centers used a single OCHIN-supported EHR system, Epic© (Verona, WI).

Patient eligibility and accrual

Patients were deemed eligible for CRC screening if they were aged 50–75, overdue for CRC screening based on EHR data, had an address, had a clinic visit in the prior 12 months, and had no ineligibility condition. Overdue for CRC screening was defined as absence of EHR evidence of a colonoscopy in the prior 9 years, a flexible sigmoidoscopy in the prior 4 years, or a FOBT in the prior 11 months. Ineligibility conditions included EHR evidence of prior CRC, colectomy, inflammatory bowed disease, end-stage renal failure, or having an “open” or un-resulted order for a fecal test in the prior 6 months or a referral to gastroenterology or for a colonoscopy or flex sigmoidoscopy in the prior 12 months. Patients meeting these criteria any time between clinic randomization and 12 months after clinic randomization were included in the study cohort.

Intervention

STOP CRC used real-time EHR data and an EHR-embedded registry to identify patients who needed CRC screening and to mail fecal immunochemical tests (FIT) directly to them. The interventions were implemented by regular clinic staff. The EHR registry (Reporting Workbench) was designed to generate materials for up to 3 mailings which included an introductory letter that explained the program, a mailed FIT kit (with wordless pictographic instructions and a postage paid envelope for returning the test), and a reminder letter (for those who had not returned the kit) (Coronado et al., 2014b). Some clinics sent an initial introductory letter with a phone number for patients to call if they had already completed CRC screening. Other clinics skipped this step and sent out the introductory letter and FIT kit as the first mailing. Written materials were in both English and Spanish (and Russian and Mandarin in one organization).

The EHR-embedded registry was updated daily, with patients dropped from the list of those needing mailings when they: had an order for, or completed, screening; reached age 75; had EHR evidence of a new ineligibility diagnosis (e.g. renal failure); were flagged for a bad address on record; or didn't have a clinic visit in the past 12 months. For purposes of defining Reach for this paper, we deemed individuals as “exposed to the intervention” if they were mailed an introductory letter.

Data collection and tracking

A novel aspect of the STOP CRC trial was the way in which data were captured for implementation as well as analysis. STOP CRC relied on a real-time EHR registry and dynamic eligibility criteria. The tools, updated nightly, produced lists of patients who were eligible for the intervention. A research tracking database captured patients in both intervention and usual care clinics who appeared on the study eligibility list, even if for a single day. Real-time tools were used to eliminate patients who had new evidence of completion of CRC screening (a completed FIT or other fecal screening test in laboratory data, or whose provider or clinic newly documented colonoscopy or other CRC screening completion). Additionally, patients who had not had a clinic visit in the prior 12-months were removed from the eligibility list, even though they might still need CRC screening. Patients removed from the registry list however remained in the STOP CRC denominator for the primary analysis.

Defining the denominator for reach

Usual care clinic patients were not invited to participate and received no interventions, thus STOP CRC usual care clinic patients were not included in the Reach denominator. The target population in the intervention clinics included all intervention clinic patients who were eligible for CRC screening for at least one day (N = 21,134, Fig. 1, Box A). This number served as the denominator for what we define as “Traditional Reach”.

Fig. 1

Strategies and opportunities to STOP colon cancer study flow for the reach analysis.

However, being overdue for CRC screening changed daily in our pragmatic trial. Many patients became current for CRC screening without receiving any study interventions. As shown in Fig. 1, this could happen from chart clean-up uncovering pre-randomization CRC tests that were not appropriately recorded (n = 174) or CRC screening occurring after an individual was determined eligible but before an introductory letter was sent (n = 2734). In both cases the patients would have been removed from the registry as they no longer needed the intervention. This effectively decreased to 18,226 the number of patients who were eligible for the program (Fig. 1, Box B), and we use this latter figure as the denominator for what we call “Pragmatic Reach”.

Defining the numerator for reach

The numerator for Reach (the number of people who were sent at least one mailing) was impacted by several factors as described in Fig. 1 and below.

No longer an active patient due to start-up delays: The research database captured patients who met study eligibility criteria (even for a single day) for both intervention and usual care clinics. Due to an upgrade to Epic, roll out of the intervention (the EHR-embedded registry) was delayed with no clinic being able to begin mailing letters until after the upgrade and testing was completed, approximately four months after clinic randomization. This delay led to 936 participants no longer satisfying the 12-month visit criterion used to define active clinic membership (and had no subsequent visits that would reclassify them as an active patient) and hence were no longer in the real-time registry of eligible patients. However, they remained in the denominator for both our measures of traditional and pragmatic Reach. Clinic-specific startup delays, EHR issues and staffing challenges, further compounded this problem, triggering an additional 343 individuals to drop off our lists due to the active membership criterion.

Bad addresses: Despite excluding individuals with addresses that didn't appear to be valid from our initial pool of participants, 103 individuals were newly identified as having an invalid address, with no available forwarding address. These patients still need to be reached but were unable to receive mailed interventions.

Provider opt outs: Providers could also use the EHR's health maintenance tool to postpone or discontinue screening for patients who declined or had a disqualifying illness or limited life-expectancy, which would remove the patients from the registry list. The study had no way to know whether this was because of patient self-report or provider judgement related to the patient's health or some other factor. This affected 758 individuals.

No mailings done: The most common reason for eligible patients not being reached was clinics not mailing the letters, with 9552 patients affected.

Analysis

We present Reach as simple proportions (percent) using several different denominators. The first includes individuals who were eligible for the intervention for at least one day (Traditional Reach). The second calculation removed individuals who later turned out to be ineligible (Pragmatic Reach) due to, for example, having been found to have completed CRC screening through chart review after the registry list was created. We also present other factors that decreased Reach (patient exposure to the intervention): including patients being dropped from the registry because of system and clinic delays before the clinics did their first mailings, or because the patients had an invalid address, or the provider had removed them from the registry that made it impossible for the participants to be reached by the clinics, even though these individuals may have needed CRC screening.

To assess the extent to which Reach varied by different patient characteristics, we first fit logistic regression models that included both the characteristic of interest as well as clinic level, thus providing a test of whether Reach varied across the levels of that characteristic that was adjusted for clinic level effects. We then used this model to estimate clinic specific probabilities of Reach for each level of the characteristic in question (i.e., gender specific Reach for each clinic), and finally combined these probabilities using the common, overall clinic distribution to arrive at adjusted subgroup-specific Reach estimates that are not influenced by varying clinic distributions across the subgroups. In classical epidemiologic parlance, for each level of the characteristic in question (e.g., for men and women) we used direct standardization to calculate clinic adjusted Reach estimates, which we then present in tabular form.

Results

Maximal obtainable reach

Table 1 shows the maximal obtainable percent Reach possible based on Traditional and Pragmatic Reach denominator definitions. In theory, maximal attainable reach was 100%, but as noted above, 2908 patients were determined to have completed CRC screening, and appropriately received no intervention. If we exclude these people from the denominator, the proportion of patients that should have been Reached was 18,826, or 86.2% of the Traditional Reach denominator. Maximal obtainable Reach, among individuals who continued to need the intervention (they were still overdue for CRC screening) was still potentially 100%.

Table 1

Maximum reach theoretically possible based on traditional and pragmatic definitions.

Definition of population	# Patients	Maximum attainable traditional reach	Maximum attainable pragmatic reach
Everyone that were included on the registry as not current for CRC screening (Box A)	21,134	100%	–
Excludes those who were current for CRC screening (Box B)	18,226	86.2%	100%
Excludes those who were no longer active clinic members following system-wide delay associated with EPIC upgrade (Box C)	17,251	81.6	94.7%
Excludes those who were no longer active clinic members following clinic-specific delays (Box D)	16,908	80.0%	92.8%
Reach possible if still on the list, not opting out, or postponed, and no bad address(Box E)	16,047	75.9%	88.0%

However, other factors out of the control of clinics made it impossible to Reach all patients who needed to be Reached. These reasons included patients being dropped from the registry because of system and clinic delays before the clinics did their first mailings, or because the patients had a missing or invalid address, or the provider had removed them from the registry. If we account for all these factors (Table 1), the best that clinic staff could have hoped to achieve was 75.9% for Traditional Reach and 88.0% for Pragmatic Reach (Box B, Fig. 1).

Percent reach observed

Table 2 shows the percent of STOP CRC patients Reached, with 30.7% and 35.6% of patients receiving at least one mailing using the Traditional and Pragmatic Reach denominators, respectively. Accounting for factors that were out of the clinics' control, system and clinic start-up delays, improved the percent of patients Reached only modestly to 38.4%. Accounting for patients who were removed from the list because of invalid addresses and provider's removing the patient from the registry list, only 40.5% of patients overall who still needed the intervention were Reached.

Table 2

Percent of Individuals Reached (Received at Least One Mailing) in the STOP CRC Study⁎.

	Percent reachedUsing the traditional reach denominatora(Box F/Box A)	Percent ReachedUsing the pragmatic reach denominatora(Box F/Box B)	Percent ReachedUsing the pragmatic reach denominator plus accounting for delays(Box F/Box D)	Percent ReachedUsing the pragmatic reach denominator plus accounting for delays, bad addresses, and provider opt-outs(Box F/Box E)
All	30.7%	35.6%	38.4%	40.5%
Health Center 1	71.8%	74.7%	80.9%	82.2%
Health Center 2	44.7%	53.7%	57.3%	60.6%
Health Center 3	42.3%	51.7%	55.8%	56.8%
Health Center 4	29.8%	35.3%	37.5%	38.3%
Health Center 5	25.1%	28.0%	29.4%	29.8%
Health Center 6	18.1%	20.8%	22.2%	24.3%
Health Center 7	27.1%	29.5%	32.2%	32.9%
Health Center 8	4.1%	4.8%	5.9%	6.8%

Based on the Pragmatic Reach denominator that removed individuals who completed CRC screening prior to a letter being mailed or within 12 months of clinic randomization if no letter was mailed.

Reach varied substantially between health centers, ranging from 74.7% to 4.8% among the health centers based on the Pragmatic Reach denominator (Table 2). Similarly, after accounting for all factors that might have removed patients from the list, making it impossible for clinics to implement mailings, percent Reached only increased modestly, with the highest performing health center completing 82.2% of mailings, and the lowest 6.8%.

Percent Reached varied little by patient characteristics (Table 3) and differences were mostly due to health center characteristics rather than individual patients' characteristics, resulting in only modest subgroup differences after clinic-level adjustment. However, a slightly higher percentage of younger patients (aged 50–64), Spanish speakers, and those with Medicaid insurance were Reached.

Table 3

Percent of patients reached by patient characteristics.

	Denominator	Percent reacheda,b
Age
50–64	14,592	36.5%
65–75	3634	32.3%
Gender
Female	9970	35.9%
Male	8256	35.3%
Hispanic
Non-Hispanic	15,738	35.1%
Hispanic	1859	39.5%
Hispanic+
Non-Hispanic	15,738	35.2%
Hispanic	1859	39.5%
Missing	629	34.4%
Race
Asian	1003	34.5%
Black	932	35.3%
Hawaiian/Pac Islander	63	34.3%
Native American	181	34.6%
Unknown	863	37.9%
White	15,184	35.4%
Language
English	14,823	35.0%
Other	1481	37.5%
Spanish	1355	41.6%
Unknown	567	26.9%
Insurance status index
Uninsured	3450	34.1%
Medicaid	6686	40.1%
Medicare	4933	31.7%
Commercial	2864	34.0%
Other	109	28.2%
Unknown	184	28.9%
Federal poverty level
<100%	7808	36.1%
100–150%	3123	37.7%
151–200%	1334	36.4%
200%+	1837	33.3%
Unknown	4124	34.0%
Urban rural
Rural	781	34.2%
Urban	17,445	35.7%
All	18,226	35.6%

Based on pragmatic reach.

Adjusted subgroup specific probabilities calculated using clinic specific probabilities calculated from a logistic regression model and then applied to the overall clinic distribution (see Methods).

Among those not Reached, Table 4 presents patient characteristics by the reasons why patients were not Reached. Uninsured individuals, and patients whose insurance status was unknown were somewhat more likely to have lost clinic eligibility because of not having had a clinic visit within 12 months, otherwise differences between subgroups were small. Lack of Implementation, i.e., no mailings, was the primary reason patients were not Reached.

Table 4

Patient characteristics of those not reached.

	Eligible	Fell off list	Bad address	Opted out	Postpone	No mailinga
Age
50–64	14,592	7.3%	0.7%	2.8%	1.4%	51.2%
65–75	3634	6.9%	0.2%	3.5%	0.8%	57.5%
Gender
Female	9970	6.9%	0.4%	2.8%	1.2%	53.0%
Male	8256	7.6%	0.7%	3.0%	1.3%	51.7%
Hispanic
Non-Hispanic	16,367	7.2%	0.5%	3.1%	1.3%	53.1%
Hispanic	1859	7.4%	0.8%	1.1%	1.0%	46.5%
Race
Asian	1003	5.9%	0.6%	1.7%	1.1%	38.7%
Black	932	6.3%	1.8%	2.6%	2.7%	42.4%
Hawaiian/Pac Islander	63	4.8%	0.0%	1.6%	3.2%	47.6%
Native American	181	8.8%	1.1%	2.8%	0.6%	53.0%
Unknown	863	9.5%	0.7%	1.2%	1.3%	42.2%
White	15,184	7.2%	0.5%	3.1%	1.2%	54.5%
Language
English	14,823	7.4%	0.5%	3.2%	1.3%	54.5%
Other	1481	5.1%	0.8%	2.4%	1.4%	35.7%
Spanish	1355	7.7%	0.9%	0.6%	0.9%	43.2%
Unknown	567	8.3%	0.4%	1.2%	0.2%	63.1%
Insurance status index
Uninsured	3450	10.5%	1.0%	1.8%	0.5%	45.6%
Medicaid	6686	6.0%	0.6%	3.4%	1.9%	46.4%
Medicare	4933	6.4%	0.4%	3.8%	1.1%	58.8%
Commercial	2864	6.7%	0.1%	1.9%	0.9%	64.1%
Other	109	11.9%	0.9%	0.9%	0.9%	47.7%
Unknown	184	19.6%	1.6%	1.6%	0.5%	46.7%
Federal poverty level
<100%	7808	7.1%	0.8%	3.1%	1.4%	49.4%
100–150%	3123	6.3%	0.4%	3.6%	1.3%	51.0%
151–200%	1334	7.1%	0.3%	3.5%	1.2%	54.3%
200%+	1837	7.9%	0.3%	2.6%	0.9%	60.2%
Unknown	4124	7.9%	0.5%	2.0%	1.0%	55.0%
Urban/rural
Rural	781	7.6%	0.4%	3.5%	1.4%	61.1%
Urban	17,445	7.2%	0.6%	2.9%	1.2%	52.0%
All	18,226	7.2%	0.6%	2.9%	1.3%	52.4%

Patient not current for CRC, remained on registry list, with no mailings completed.

Discussion

In the STOP CRC trial, we attempted to deliver a mailed CRC screening intervention to as many people as possible so as to increase CRC screening uptake and decrease CRC screening disparities. This is important because in US Federally Qualified Health Centers CRC screening is much lower than national averages (Hall et al., 2018; Sabatino et al., 2015). In STOP CRC we used the RE-AIM model definition of Reach to measure the absolute number eligible for and the proportion and representativeness or individuals exposed to the mailed program (Glasgow et al., 1999; Glasgow et al., 2006). We applied the traditional definition of Reach to our study and present a more flexible pragmatic definition (“Pragmatic Reach”) in a program that had real-time dynamically changing eligibility. After accounting for people who did not need the intervention (were found to be or became ineligible) maximally obtainable Pragmatic Reach was still 100%. However, actual Reach ranged from 74.7% to 4.8% across health centers and varied little across patient subgroups defined by age, gender, race/ethnicity, language, poverty status, and insurance status. In our study we found that Reach was dependent on Implementation and that patient level characteristics varied little between those Reached and not Reached.

We had originally hypothesized that factors beyond the health centers' control (“dropping from the registry list” due to system-wide and clinic delays) would be an important reason for lower than expected Reach. However, after accounting for this, percent Reached only increased modestly. Ultimately, lack of implementation was responsible for the low overall percent of patients Reached, with large variation between health centers. Low implementation also impacted Effectiveness. Overall adjusted FIT completion rates were 3.4% higher for interventions clinics (13.9%) compared to control clinics (10.4%) at 12 months (P = .05) (Coronado et al., 2018). In analyses that accounted for EHR implementation delays the adjusted FIT completion rate were 4.7% higher for intervention clinics. Across health centers, net differences in FIT completion rates were strongly correlated with Implementation or mailing rates (R2 = 0.87).

We also found that patient-level characteristics had little impact on Reach; instead differences in Reach were primarily driven by cluster- (health center) level characteristics. Differences between individuals Reached versus those not Reached, were not a function of patient “willingness to participate”, but instead their “exposure to” the intervention. In a typical patient-level randomized trial assent or “willingness to participate” is obtained prior to the intervention being offered. Other studies describing Reach generally describe patient willingness to participate in a study or initiative (Green et al., 2011; Yeary et al., 2018). Finlayson et al., described methods for increasing Reach – but their methods were directed at engaging patients' willingness to participate in programs (Finlayson et al., 2014). In contrast, in cluster trials such as STOP, individuals are included in the study cohort regardless of whether they are willing to participate. In STOP CRC, “willingness” could only be measured by completion of FIT, which defined study effectiveness.

Sweet et al., called for broadening of the definition of Reach and other RE-AIM components to evaluate the impact of multi-sector partnerships or public health initiatives, to determine whether the entire eligible population had the opportunity to participate in a program, rather than those specifically invited to participate in a study (Sweet et al., 2014). Reach was calculated on several different levels, including the population at large that was eligible (individuals living with spinal cord injuries in Canada), region (among 2 provinces), and among those contacted by the community programs, with the percent observed Reach of 3%, 37%, and 49% respectively. Reach in STOP CRC, was more dynamic, because patient status was subject to ongoing daily changes that typically occur as part of routine clinical care. More recently Glasgow and Estabrook, recommended measuring Reach in pragmatic studies or community initiatives as the percent of the target population either participating or exposed to the program, initiative, or intervention (Glasgow and Estabrooks, 2018). The RE-AIM web site continues to define Reach as the absolute number, percent, and representativeness of individuals who are willing to participate in a study and includes both intervention and usual care participants (RE-AIM, n.d.; RE-AIM Homepage, n.d.).

Glasgow et al. also recommended that Reach be used to calculate the potential Impact of an intervention or program, by multiplying the Reach (percent agreeing to participate) by Effectiveness of the intervention (Glasgow et al., 2006). However, the definition of impact as originally proposed is also changed when considering Pragmatic Reach. For example in our team's SOS study implemented within an integrated health care system and requiring verbal consent for participation, the net benefit of the mailed CRC screening program was 25% at 12 months (Green et al., 2013). If you multiplied this by the participation rate (signed consent was obtained from 39%), which assumes that those not willing to participate would also not complete the program, the net impact would be 10% (Green et al., 2012). In STOP CRC, where theoretically maximal obtainable Reach was closer to 100% the effect size would not need adjustment. However, if participants were not given the opportunity to “join” or be exposed to the program, the reverse might be true (Coronado et al., 2018). Had the intervention been delivered to all eligible patients, the potential impact would have been 15% or greater (effectiveness = (4.7% X 100%)/30%).

A limitation of our manuscript is that we do not describe factors related to implementation fidelity and variation that lead to low rates of percent reached (mailing of the initial letter). We have previously described some of the implementation factors that led to implementation failures including delays in activating the EHR tools, time burden of carrying out the mailings and lab test orders, difficulties with lab interfaces and having to wait for information technology support for registry-linkages, and changes in leadership and resulting reorganization of staff and processes (Coronado et al., 2017). We further describe implementation barriers in our primary outcomes paper, and the relationship between mailings and effectiveness within health centers (Coronado et al., 2018). Another limitation that we don't account for is that three health centers that were eligible to be included in the study declined participation (Re-AIM Adoption). We previously published factors related to Adoption in the STOP CRC trial (Coronado et al., 2016). Finally defining the targeted population (age-eligible individuals overdue for CRC screening) was dependent on the accuracy of the EHR registry, which we previously validated. Missing data might lead to misidentification of the target population, and the proportion needing to be Reached.

Conclusion

We found that, in a pragmatic cluster randomized trial that used a real-time, dynamic patient registry to identify and track eligible patients, the traditional definition of Reach “willingness to participate” was not meaningful. Emerging pragmatic definitions that measure dynamic changes in the absolute number eligible and the proportion exposed to the intervention were more useful. In our study we found that Reach was dependent on Implementation and that patient level characteristics varied little between those Reached and not Reached. More studies are needed to explore the use of Reach in health system-embedded pragmatic trials.