Randomized controlled trial of centralized vaccine reminder/recall to improve adult vaccination rates in an accountable care organization setting.

Our objectives were to assess 1) effectiveness of using Colorado's Immunization Information System (CIIS) to send out vaccine reminder/recalls (R/Rs) centrally vs. usual care for adult vaccine delivery within an accountable care organization (ACO) and 2) practice staff's perception of centralized R/R. From 9/2016 to 4/2017, we conducted a randomized controlled trial among adults enrolled in a Medicaid ACO at six healthcare entities. Adults were divided into two strata: 15,153 age 19-64 and 616 age 65+. Adults age 19-64 who needed influenza and/or Tdap vaccine, and adults age 65+ who needed influenza, and/or Tdap, and/or a pneumococcal vaccine were randomized to receive up to 3 R/Rs by autodialed telephone and mail or usual care. Documentation of receipt of any needed vaccines in CIIS within six months was the primary outcome. We assessed intervention effectiveness using mixed effect logistic regression. Thirteen semi-structured exit interviews were conducted with staff from each healthcare entity. The intervention was not associated with the primary outcome for the age 19-64 population [OR 1.06 (95% CI 0.98-1.15)] or age 65+ population [(OR 0.96 (0.69-1.32)]. Practice staff perceived the intervention to be beneficial and not burdensome. Perceived barriers included lack of availability of appointments and adults receiving only influenza vaccine when other vaccines were needed. In conclusion, centralized R/R was not effective at improving adult vaccination rates in a Medicaid ACO. Future studies should consider better harmonizing vaccine centralized R/Rs with vaccine delivery efforts within the practice setting. Clinical Trials Registration Number: NCT02133391.

RCT Entities:   Population Interventions Outcomes

Introduction

Because accountable care organizations (ACOs) are held responsible by payers for both the cost and quality of care for a defined population of patients, vaccination to prevent infectious diseases is a particularly compelling metric for ACOs (Kessell et al., 2015). By improving vaccination rates in their patient population, ACOs have the potential to incur cost savings through disease prevention (Orenstein and Ahmed, 2017; Ozawa et al., 2016; McLaughlin et al., 2015). Influenza epidemics in the U.S. have been estimated to result in 3.1 million hospitalized days and 31.4 million outpatient visits and cost an average of $10.4 billion annually (Molinari et al., 2007) and influenza vaccines have been shown to be cost-effective at preventing influenza disease (Ting et al., 2017). ACOs may also, by improving vaccination rates, be able to enhance payments by improving quality metrics (https://www.ncqa.org/hedis/measures/ (n.d.)).

The Community Guide for the Preventive Services Task Force recommends several evidence-based strategies to improve vaccinations including patient vaccine reminder and recall (R/R) (). Patient R/R interventions remind members of a target population that vaccinations are due (reminders) or are late (recall). Health care practices rarely implement R/R (Pereira et al., 2012) even though it is a recommended strategy to improve vaccination rates and has a body of evidence to support its use (Jacobson Vann and Szilagyi, 2005).

A way to reduce the burden on an individual practice in conducting a R/R and a way for an ACO to target members going to different practices is to conduct the R/R centrally using an Immunization Information System (IIS). IIS are confidential, population-based, computerized databases that record and consolidate all vaccination doses administered by participating providers to people residing in a given geopolitical area (). IIS exist in five cities, the District of Columbia, and in all U.S. states except New Hampshire (IISAR Data Participation Rates and Maps, n.d.). Pediatric trials have shown this approach to be effective and cost-effective (Kempe et al., 2013; Kempe et al., 2015; Kempe et al., 2017) and a previous trial in an adult safety-net population demonstrated this approach was effective at increasing influenza vaccination rates in adults age 65+ (Hurley et al., 2018a). Vaccination R/R using an IIS may be opportune among adults because adults often receive vaccines outside of the medical home (Lu et al., 2014), and depending on the local reporting of adult vaccination information to the IIS, IIS may contain a more complete record of vaccinations an adult has received. As of 2016, a total of 44% of adults aged ≥19 years in the U.S. had at least one vaccination administered during adulthood in their respective IIS ().

The objectives of this study were to assess 1) effectiveness of centralized vaccine R/R for adult seasonal influenza, pneumococcal, and Tdap vaccines using the Colorado Immunization Information System (CIIS) compared to usual care for adult Medicaid patients managed by an ACO, and 2) practice staff's perception of centralized R/R.

Methods

This study utilized an explanatory sequential mixed methods study design (Creswell and Plano Clark, 2018). The first phase, conducted from September 2016 to April 2017, was a stratified randomized controlled trial involving adult Medicaid patients enrolled in a regional accountable care organization comparing centralized IIS-based R/R for influenza, Tdap and PPSV23 or PCV13 to usual care. The second phase, conducted from July 2017 to February 2018, consisted of semi-structured exit interviews to enrich understanding and interpretation of the randomized controlled trial. The study was reviewed and approved by the Colorado Multiple Institutional Review Board (COMIRB #13-2395) and the Colorado Department of Public Health and Environment's Institutional Review Board.

Setting and participants

The study team collaborated with a regional ACO that manages Colorado Medicaid patients (RCCO) to recruit healthcare entities (single and multi-site practices) in the Denver metropolitan area and Northeastern Colorado. Practice eligibility criteria included: being an internal or family medicine practice, actively using the IIS for adult vaccination data at the onset of the study and having at least 50 adults enrolled in the RCCO. Practices were approached to participate by the RCCO and study staff over 10 months. All empaneled adults (had a visit within the previous year 8/2016–8/2017), who were age 19+ as of September 1, 2016, and who were deficient in at least one of the 3 vaccines being evaluated were included in the study. Individuals meeting these criteria were divided into two groups: 19–64 years and 65+ years.

After the randomized controlled trial and post preliminary analyses, semi-structured exit interviews were conducted with at least one clinical and one administrative staff from each healthcare entity. Staff eligibility criteria included experience in the larger trial and/or clinic vaccination processes and protocols. Clinical staff included nurses, providers, and medical assistants and administrative staff included directors and office managers. The study team worked closely with previously identified practice study champions to recruit staff to participate in interviews.

CIIS

CIIS receives client and vaccination data through live data entry into the web-enabled IIS application and through electronic transfers from data sources maintained by health care practitioners, state vital statistics, and insurers. CIIS also includes historical data about vaccinations given outside the state if entered by a participating Colorado practitioner, school or child care facility. Colorado is currently not a mandatory reporting state. However, as of 2017, 83% of adolescents aged 11 through 17 years in Colorado had 2 or more adolescent vaccinations recorded in CIIS and 63% of adults age 19+ in Colorado had 1 or more adult vaccinations recorded in CIIS (IISAR Data Participation Rates and Maps, n.d.). CIIS became a lifespan immunization registry in 2007. Vital records data for both births and deaths are uploaded to CIIS daily.

To ensure accurate data, eligible healthcare entities had to have been actively participating in CIIS as evidenced by at least one vaccination upload during the 6 months prior to the trial start. Individual demographic data obtained from the RCCO for all adults was uploaded to CIIS immediately before the trial, and, if any adult had no previous CIIS record, a new entry was made for that individual; 3103/15807 (19.6%) individuals across intervention and control required new entries.

Randomization

Randomization at the participant level into R/R or usual care groups was done using simple random sampling stratified by healthcare entity and study age group. Fig. 1 (CONSORT diagram) provides more detailed information. Participants and care providers were unblinded, but the source of the outcomes (CIIS) could not be biased by the allocation of the intervention.

Fig. 1

Consort diagram: adult vaccine reminder/recall trial, regional accountable care organization population, 2017.

Note: No participants were lost to follow-up. All patients were included in the analysis. ACO, accountable care organization. UTD, up to date.

Intervention

All participating sites were given Centers for Disease Control and Prevention (CDC) posters in English and Spanish promoting adult vaccinations to display in patient care areas. The control arm received usual care that did not include any reminders from the study team to receive vaccines.

Centralized R/R approach

Adults randomized to R/R were contacted up to three times over three to four months. Adults received up to two auto-dial phone calls followed by a postcard. Messages were personalized to include practice name and phone number, and were delivered in both English and Spanish. Participants were able to select via dial tone whether they received the message in English or Spanish. Postcards were printed in both English and Spanish. If a person's phone number was missing or deemed incorrect, they were sent a postcard only. Phone numbers were considered valid if they went to a live answer or voicemail and no one called to say they should not have been contacted; they were considered incorrect if they were a fax number, a discontinued phone number, or had no dial tone. Returned postcards were considered to have incorrect addresses. Auto-dialer and postcards indicated that the individual may need one or more of two or three vaccines (influenza, Tdap, or pneumococcal) and prompted recipients to call their clinic to schedule an appointment to discuss their vaccine needs. The recalls did not specify what vaccines were needed. During the study, adults had various options to opt-out of the study including pressing a number on a phone dial pad at the time of the phone recall or leaving a voicemail or email via contact information provided in the recall message. Adults who became up to date on the vaccines of interest or who had opted out between rounds were not contacted further.

Outcome measures

The primary study outcome was documentation of receipt of any of the two or three needed vaccines in CIIS within six months of the initiation of the R/R. This outcome could include receipt of a vaccine or documentation within CIIS of a previously received vaccine because both might be attributed to the R/R effort. The six-month time frame has been used in previous studies (Kempe et al., 2013; Kempe et al., 2015; Hurley et al., 2018a), is proximal enough to the R/R to be attributed to it, corresponded to the influenza season, and allowed time for the three reminders and for visits to be scheduled after a reminder. Secondary outcomes assessed location of receipt of vaccine and missed opportunities. For the purposes of this study, a missed opportunity meant an individual received one, but not all the vaccines of interest he/she was eligible for.

Staff interviews explored experiences with the R/R, current practices around vaccinations, perceived effectiveness of the collaborative centralized R/R, and potential barriers in vaccination delivery. Overall preliminary findings as well as individual practice level findings were presented, and participants were queried to help explain the findings. Interview guides were semi-structured and used a combination of open-ended questions and follow-up prompts. Interviews were conducted over the phone and were 30–45 min in-length. Interviews were digitally recorded and transcribed verbatim. Participants received $50 compensation.

Data analyses

The study was powered to detect an absolute difference of 2 percentage points in documentation of a new vaccination among adults age 19–64 whose vaccination status was not up-to-date at baseline between the two arms. Receipt of any of the three vaccines was modeled using generalized linear mixed-effects models with the logit link function. Primary clinic was included as a random effect to account for clustering within clinic. The independent variable included was whether they were randomized to intervention or control. The major analyses were intention to treat. All comparisons were set at a priori level of significance at p ≤ 0.05. All analyses were performed using SAS version 9.4 (SAS Institute, Cary, NC).

Qualitative data were analyzed using an iterative and team-based process using a constant comparative method and content analysis until saturation of themes was reached. Two qualitatively trained analysts inductively developed a code book and jointly reviewed and coded interview transcripts until no new codes were identified and there was strong code assignment agreement. Transcripts were first read and coded independently, double coded, and merged to achieve immersion. The analytic team debriefed until consensus was reached with code assignment, and met regularly with the study team to discuss emergent new codes and themes, and assess the results (Hsieh and Shannon, 2005; Elo and Kyngas, 2008; Ranney et al., 2015; Green et al., 2007). ATLAS.ti version 8.0 was used for data organization and management.

Results

Sample characteristics

Of the 15,807 patients randomized, 38 (0.2%) did not need any vaccines and were excluded from further analysis, leaving 15,769 participants. Since the trial commenced at the beginning of the influenza season, the expectation was that most participants would need an influenza vaccine at baseline. Fig. 1 shows a CONSORT diagram for the trial.

Table 1 compares characteristics of the populations randomized to the two intervention arms. There were no significant differences in baseline characteristics or up to date rates for the vaccines of interest between the arms in both age groups. Baseline Tdap vaccination rates were higher than national averages (53%-19–64 study population vs. 24.7% national average and 49% 65+ study population vs. 15.5% national average) for both populations; pneumococcal vaccination rates for the age 65+ population were significantly lower (28% study population vs. 64% national average) (Williams et al., 2017). Eighty-three percent (6577/7919) of the intervention population had a correct phone number and 79% (6218/7919) had a correct mailing address. Ninety-six percent (7572/7919) of the intervention population had either a correct address or phone number, and so was reached at least once by one of these means.

Table 1

Regional accountable care organization study population, 2017.

Variables	Level	19–64 groupa(n = 15,153)		≥65 groupa(n = 616)
		Intervention(n = 7612)	Control(n = 7541)	Intervention(n = 307)	Control(n = 309)
Gender	Female	62%	63%	64%	64%
Median age (IQR 25–75%)		36 (26–50)	36 (26–50)	71 (67–78)	70 (67–74)
Proportion of participants from healthcare entity	1-FQHC	35%	35%	32%	32%
	2-FQHC	46%	46%	51%	51%
	3-University Clinic	2%	2%	7%	7%
	4-Private Practice	2%	2%	1%	2%
	5-University Clinic	11%	11%	8%	8%
	6-Private Practice	4%	4%	1%	1%
Up to date on vaccine at baseline
	Influenza	0.1%	0.1%	0.7%	1.6%
	Tdap	53%	52%	48%	49%
	Pneumococcal	N/A	N/A	26%	28%

Tdap = Tetanus, diptheria, pertussis vaccine.

Pneumococcal = PPSV23 or PCV13.

FQHC = Federally Qualified Health Center.

No difference between intervention and control for baseline variables.

Exit interviews were conducted with 2–3 staff members from the six healthcare entities, including 6 administrators, 3 medical staff (nurses, MAs), and 4 providers (MDs, NPs).

Effectiveness

There were no significant differences in receipt of any individual vaccine between intervention and control arms for either population studied (Fig. 2). The intervention was not associated with receipt of any of the needed vaccines for the age 19–64 population (OR 1.06 (95% CI 0.98–1.15) or the age 65+ population (OR 0.96 (95% CI 0.69–1.32).

Fig. 2

Receipt of vaccines among those who needed a vaccine, adult vaccine reminder/recall trial, regional accountable care organization, 2017*.

Missed opportunities

In the age 19–64 group, of the 1275 participants who needed both an influenza vaccine and a Tdap vaccine and received any vaccine, only 5% (n = 62) received both vaccines at the first vaccine visit; 80% (n = 1024) received only an influenza vaccine, and 15% (n = 189) received only a Tdap vaccine. In the age 65+ group, of the 202 participants who needed more than one vaccine, only 6% (n = 13) received all needed vaccines at the first vaccine visit. There were no statistical differences between intervention and control in terms of missed opportunities (p ≥ 0.2 for all comparisons).

Location of receipt of vaccines

For the age 19–64 group, individuals who received vaccine during the trial received it at the following locations: 53% primary care clinic, 20% at another clinic outside of the primary care, 6% retail pharmacy, 6% hospital and 4% another location. Eleven percent of the age 19–64 group who appeared to have received any vaccine had done so by providers updating historical vaccine information into CIIS from personal vaccine records. For the age 65+ group, individuals who received a vaccine during the trial received it at the following locations: 61% primary care clinic, 15% at another clinic outside of primary care, 8% retail pharmacy, 4% hospital and 1% another location. Ten percent of the age 65+ group who appeared to have received any vaccine had done so by providers updating historical information from personal vaccine records into CIIS.

Standing orders

From key informant interviews, we discovered that four out of the six healthcare entities in the study reported using standing orders for adult vaccination; three reported using them for influenza vaccine and only one reported having them for all three vaccines.

Practice experience with and perceived impact of intervention

Table 2 shows results of key informant interviews reflecting attitudes regarding practice experience with and perceived impact of the intervention. Staff generally reported the experience had been beneficial, broadened the practice's impact on encouraging patients to receive vaccines and did not disrupt workflow.

Table 2

Practice experience with and perceived impact of R/R-qualitative themes and illustrative quotes, regional accountable care organization, 2017.

Theme	Respondent	Representative quotes
Positive experience participating in intervention	Provider	I would say, yes, especially if it was as low maintenance on our end as this was, right. Anything that's gonna be high yield in terms of getting more patients in to get their health maintenance done, including vaccines that doesn't take away our clinic staff time or money to do it, heck yeah.
	Administrator	I think that was why it [R/R] was so unnoticed. You know, it really worked really well with what we already do with our existing standing orders. It's not like these patients had to make appointments. They basically could just come in with their postcard and say that they were due for something and we could administer it. So I would say it really worked really well in tandem together.
Minimal burden on practice workflow	Provider	I mean, it's [answering calls] part of what they [call center] already do. They get, you know, tens of thousands of calls. It's just kind of in that. So it would be hard to say, well, you know, it increased by such and such an amount. I mean, each call center person can only take so many calls. So how many of those were because of the recall…
	Administrator	Yeah. It's [answering calls and scheduling appointments] part of their workflow already. We didn't have to make any changes at all to accommodate for these patients calling in or walking in.
Intervention perceived as beneficial to practice	Medical Support Staff	Just to get all our patients vaccinated, and then that would actually even help us out even more for when patients come in for not just an annual – we don't always catch it. So having someone else or something else helping us would make a huge difference.
	Administrator	I think that any way that we can provide reminders or education to patients is valuable. So I think it's learning different avenues where it's talking to them about it in clinic, if there's a mailing, if it's a call, exploring what those different methods are for encouraging patients to come in for those vaccines, is it something worth exploring. You know, just an annual reminder from us, I don't think it's enough. I think it's good to have other avenues for outreach.

Perceived barriers to the intervention

Table 3 shows results of key informant interviews reflecting perceived barriers to the intervention. Staff shared a variety of barriers that might have influenced the effect of the intervention including only focusing on delivering influenza vaccine and not checking for other vaccine needs, lack of access to appointments and prevalent reminders for influenza vaccine outside of the intervention that would have touched both intervention and control groups.

Table 3

Perceived barriers to R/R-qualitative themes and illustrative quotes, regional accountable care organization, 2017.

Theme	Respondent	Representative quotes
Missed opportunities to provide all needed vaccines	Medical Support Staff	…I feel like us as medical assistants, we forget to do it [check vaccines] because we're just so busy, and then the provider doesn't always catch it either. Then the patient ends up leaving. Sometimes we remember when the patient is already gone, so we try calling them but they don't answer, or they don't come back for an MA visit. So yeah, paying attention I think would help a lot.
	Administrator	I would like to think [staff checked EMR or CIIS for other vaccines during flu clinic], but honestly I'm gonna say probably not as regularly because we do run several flu clinics during flu season, and they are extremely busy. So it's probably not being done quite as well as it could be, and that is just based on volume
Missed opportunities to schedule vaccine appointments	Provider	… Because if you call, you know, we send you a reminder, and when you call to make the appointment, they're like, uh, well, it's great that you called, and call back because we can't make this appointment.
	Administrator	…it would be different if we had three people here in the clinic answering phones for patients, but there's probably 30 or 40 agents answering all these calls. So I don't know how we drill down to the level of questions that might be coming from somebody getting a reminder…
Advertisements for influenza vaccine reached all patients	Administrator	…We do one specific outreach to all of our patients during flu season just to let the know that they can come into [clinic] for their vaccine. We also do a mailing during flu season just to remind people to get their flu shots. And we have promotional materials in the clinics, so posters, and different things in the exam rooms to remind people to the importance of getting a flu shot.
Other barriers
• Inaccurate CIIS data for adults	Medical Support Staff	Interviewee: I've noticed that a lot of our adult population does not go into that [CIIS].Interviewer: So, you don't’ think that CIIS is really a great – a place for adults?Interviewee: Not for adults, and it could be because I'm so used to not seeing adults in there.
• Transient medicaid population	Provider	I mean, I can only guess from, I mean, the patient population that you were using in our [system]. I know that some of these patients are hard to get a hold of, and we spend a lot of time. Their numbers are not correct, their phone numbers are disconnected, they've moved.
• Lack of patient knowledge around Tdap and pneumococcal vaccines	Administrator	I think that some of the other vaccines, there is not always an understanding of what those are, and what they do, and the value in it. But when you talk about flu shots for the age range that you referenced, people understand what the flu is, that they've either experienced it, don't ever want to get it again…

Discussion

This is one of few studies evaluating the effect of a centralized vaccine R/R intervention in adults and the first study to assess this type of intervention in an ACO population. The intervention was ineffective at improving vaccination rates in adults age 19–64 with Medicaid managed by an ACO seeking care at a variety of practice types. Due to involving a Medicaid population, inherently a younger population, there were inadequate numbers of patients age 65+ in order to draw a conclusion about the effect of the intervention in this population. Practices generally reported the intervention was beneficial and not burdensome, but also identified a number of important barriers to the success of centralized R/R.

In an effort to explain these negative findings, it should be noted that this was a pragmatic trial focused on increasing patient demand for vaccine and did not address vaccine delivery at the practice level. A minority of participants who received the influenza vaccine and needed another vaccine targeted in this study received one. The frequency in which participants might have had a clinic visit and did not receive a vaccine is unknown, but the qualitative information confirms, at least in some instances, a focus on delivering influenza vaccine and neglecting the other vaccine needs a participant may have had. This could relate to influenza vaccine being delivered over a finite period of time and often in a setting dedicated to delivering influenza vaccine. The Standards for Adult Immunization call for assessing immunization needs at each visit and for strongly recommending needed vaccines and either administering them or referring for vaccinations (). A recent Internet Panel survey of a variety of providers including family physicians and general internists, as in this study, also demonstrated variable implementation of the Standards by vaccine type (Lutz et al., 2018). Behind this variable implementation of the Standards by vaccine type and a possible explanation for the low baseline pneumococcal vaccine rate in this study may be physician confusion regarding the pneumococcal vaccine recommendations that have been documented (Hurley et al., 2018b).

Standing orders, like R/R, are recommended by the Community Guide for Preventive Service Task Force to improve vaccination rates (Vaccination Programs, 2018). They authorize non-physician healthcare providers, where allowed by state law, to assess a client's vaccination status and administer vaccinations according to a protocol approved by an institution, physician, or other authorized provider (Vaccination Programs, 2018), and are a way for an organization to create a supportive environment for adult immunization and perhaps better adhere to the Standards for Adult Immunization. Based on the qualitative data, four out of the six healthcare entities in this study reported having standing orders for adult immunizations, but only one site had standing orders in place for all three vaccines in this study. Centralized R/R using an IIS at an institution using standing orders had some success improving influenza vaccine rates among adults age 65+ (Hurley et al., 2018a). Standing orders are not foolproof however, with one prior study showing only 23% of a study population consistently using standing orders for influenza and pneumococcal vaccine (Albert et al., 2012).

Importantly, despite perhaps a focus on influenza vaccine, we observed no difference between intervention and control groups in terms of receipt of this vaccine. The qualitative data indicate this might be a result of community and clinic specific advertising for influenza vaccine diluting the impact of the intervention and perhaps a lack of patient knowledge regarding Tdap and pneumococcal vaccines. Another explanation for the lack of effectiveness of the intervention that the qualitative data highlights is, at some clinics, participants not being able to make an appointment after being recalled. While it is encouraging that the intervention was viewed as not being burdensome, this may be reflective of a lack of urgency in scheduling on the part of the practice.

The impact of only targeting a Medicaid population is uncertain. Medicaid provides health benefits for the poor, disabled and elderly (Altman and Frist, 2015) who may be less likely to participate in their healthcare. A prior study demonstrated that Medicaid patients were more likely to not show up for scheduled appointments (Kaplan-Lewis and Percac-Lima, 2013). Perhaps it can be presumed they might be less diligent about scheduling an appointment to receive a vaccine, particularly when scheduling is difficult. One provider reported feeling that the transient nature of the Medicaid population may have been a barrier to the success of this trial.

In addition to what has already been discussed, the following limitations to this study should be noted. Some vaccinations received before or during the study at sites outside of the practice (at other clinics, worksites, community events, some pharmacies) may not have been captured in CIIS diminishing the apparent effectiveness of the intervention. The trial was conducted during a U.S. presidential election year which, given the amount of autodialer calling, may have lessened the effects of the autodial aspect of the study. The R/R messages were only in English and Spanish and some participants may have had other language preferences. The recalls did not specify which vaccines were needed. It would be difficult to replicate this study at a site that did not have an electronic interface with the IIS or at a site with a less mature IIS than Colorado's. The study required significant collaboration with CIIS to properly denominate the population, as not all adults were present in CIIS at the outset of the study. Qualitative data were gathered from six healthcare entities in a large urban area and are reflective of those sites' experiences with the intervention and their patient populations. Further studies would be necessary to understand the generalizability of our findings to different types of clinical settings and areas.

Despite this being a negative trial, some reservations about the accuracy of CIIS data noted by one study interviewee, and the fact that most IIS are currently preferentially populated with pediatric vaccination information, there are several factors indicating that centralized R/R from an IIS might be a stronger intervention for adult vaccination in the future. With time, IIS should become more populated with adult vaccine information as pediatric patients age into adulthood and the Standard for Adult Immunization to document adult vaccine administrations into an IIS gains traction (). IIS are particularly appealing for adult vaccine R/R because they consolidate vaccination information from a variety of vaccine providers and we have shown here, as has been shown elsewhere (Hurley et al., 2018a), adults receive vaccines at many locations. In order to be effective, centralized R/R for adult vaccines may require more integration of the intervention with the practice in terms of educating providers about vaccines, ensuring that appointments are scheduled after the recall and that practices follow the Standards for Adult Immunization. Future research should consider studying more generalizable populations and recalling for other non-influenza adult vaccinations.

Financial disclosure

No financial disclosures were reported by the authors of this paper.

Declaration of Competing Interest

The authors do not have any potential conflicts of interest to disclose.