Effect of Outreach Messages on Adolescent Well Child Visits and COVID-19 Vaccine Rates: An RCT.

OBJECTIVE: To determine effectiveness of text/telephone outreach messages, with and without COVID-19 vaccine information. STUDY
DESIGN: We conducted an intent-to-treat, multi-arm, randomized clinical trial with adolescents aged 12-17 years. Eligible patients did not have an AWC visit in the past year or scheduled in the next 45 days or an active electronic health record portal account. We randomized participants to the Standard Message, COVID-19 Vaccine Message, or No Message (control) group and delivered two text/telephone calls (per family preference) to the message groups. The primary outcome was AWC visit completion within 8 weeks, and secondary outcomes were AWC visit scheduled within 2 weeks and receiving COVID-19 vaccine within 8 weeks.
RESULTS: We randomized 1235 adolescents (mean [SD] age = 14 [1.5] years, 51.6% male, 76.7% Black, 4.1% Hispanic/Latinx, 88.3% publicly insured). The Standard Message group had higher odds of scheduling an AWC visit compared with the control group (OR, 2.07 [95% CI: 1.21 to 3.52]) and COVID-19 Vaccine Message group (OR, 1.66 [95% CI: 1.00 to 2.74]). The odds of completing an AWC visit did not differ significantly (Standard Message group vs. control group, OR = 1.35 [95% CI: 0.88 to 2.06]; COVID-19 Vaccine Message group vs. control group, OR = 1.33 [95% CI: 0.87 to 2.03]. In per-protocol analyses, adolescents in the Standard Message group were twice as likely as the control group to receive the COVID-19 vaccine (OR, 2.48 [95% CI: 1.05 to 5.86]).
CONCLUSION: Outreach messages were minimally effective. Efforts are needed to address widening disparities.

Numerous preventative services were delayed during the SARS-CoV-2 (COVID-19) pandemic, including adolescent well-care (AWC) visits.1, 2, 3 AWC visit adherence was challenging pre-pandemic, with only 30-50% of adolescents completing the recommended yearly evaluation.4, 5, 6 Following the pandemic shut-down, thousands of AWC appointments were cancelled or delayed and vaccination administration rates declined.7, 8, 9 Black and Latinx vaccination rates were found to be disproportionately worsening, , creating an imperative to address structural racism with intentional and deliberate intervention. , Additionally, increasing rates of obesity and depression have heightened the importance of AWC preventative services. ,

On May 10, 2021, the Pfizer-BioNTech COVID-19 Vaccine age eligibility expanded to adolescents 12-15 years of age, amending the EUA originally issued on December 11, 2020 for those >16 years old. With COVID-19 vaccine hesitancy more common in Black and Latinx and uninsured populations , possibly linked to pervasive mistreatment, medical mistrust, and poor access,

Systematic outreach approaches hold promise to encourage adolescents to return for care and complete vaccinations, including text and email messages, phone calls, mailings, school campaigns, community navigators, and home visitors.18, 19, 20 Outreach has proven benefit for completion of AWC visits21, 22, 23 and vaccinations,21, 22, 23, 24, 25, 26, 27 but it was uncertain whether this would be effective amid the pandemic and among Black and Latinx communities who experienced the collective suffering of the pandemic with additional layers of trauma.

We sought to assess the effect of automated text and telephone outreach messages to parents, with and without information about COVID-19 vaccine availability, on the scheduling and completion of AWC visits among adolescents due for preventative services. We hypothesized that outreach messages would be superior to no message, and messages with mention of COVID-19 vaccine availability would outperform messages without it given that a subset of our population might be early adopters and open to COVID-19 vaccination.

Methods

Study Design and Randomization

We conducted a multi-arm, randomized clinical trial (NCT04904744) from May 28-August 5, 2021 to remind patients due for an AWC visit. Study personnel generated a random allocation sequence stratified by clinic location using block randomization (with a block size of 3) and randomly assigned eligible patients (1:1:1) to 1 of 3 arms: Standard Message, Standard plus COVID-19 vaccine message (hereafter referred to as COVID-19 Vaccine Message), or control group (no message). Our Institutional Review Board approved the study, granting a waiver of informed consent.

Setting & Population

The study took place at three academic pediatric primary care practices. These practices serve a predominantly non-Hispanic Black, low-income population. Annually, these practices provide over 54,000 visits to 36,000 patients. Parents/legal guardians (hereafter referred to as parents) schedule appointments using a central scheduling center that is available Monday – Friday, 8:30 A.M.-5:00 P.M.

We included patients aged 12-17 years old that met the following eligibility criteria: (1) seen at one of our three primary care practices within two years, (2) no AWC visit in the last 365 days; (3) no AWC visit scheduled in the next 45 days; (4) preferred language of English and/or Spanish; (5) parent telephone number recorded in the hospital registration system (Figure 1). We also excluded patients with an active electronic health record (EHR) portal account (n=271, 17.8% of 1,522 potentially eligible patients) in order to conduct a separate trial focused on delivering portal messages to that population. At the time of this study, 32% of the entire patient population served by our practices had active portal accounts. We collected the following variables from our EHR to characterize our sample and for consideration as covariates in our analyses: parent communication preference, absence of past patient receipt of measles, mumps, rubella (MMR) vaccine and diphtheria, tetanus and acellular pertussis (DTaP) vaccine as a proxy for declining childhood immunizations, patient lifetime historical institutional no-show rate, and practice appointment availability. We also tracked community incidence of COVID-19.

Figure 1

CONSORT 2010 Flow Diagram

Interventions

We sent two automated outreach messages 24 hours apart (i.e., Thursday at 12:00 P.M. then Friday at 12:00 P.M.) because it was feasible to implement, and a recent adult intervention was successful with a short interval between two text messages. Messages were sent either by text or phone calls to parents using Televox, a HIPAA compliant platform, based on the parent’s preferred mode of reminder and preferred language. Parent preferences are elicited at first-time registration and recorded in the EHR. Interpreter Services at CCHMC translated automated messages into Spanish.

We crafted the messages delivered based on the extant literature and feedback from parents, as well as outreach medical assistants who call and text with our families daily. Please see the Supplement (available at www.jpeds.com) for examples of automated messages. The Standard Messages referenced the patient’s first name, reminding parents that their child was due for an AWC visit and asked them to call to schedule using the number provided. The COVID-19 Vaccine Messages included the information in the standard message plus mention that the COVID-19 vaccine was available for the child and parent during their future visit. Study personnel downloaded a report from the third-party vendor to ascertain receipt of text and phone call messages by parents.

Outcomes

Our primary outcome was AWC visit completion within 8 weeks of the date we sent the first message or the date of randomization for the control group. All practice locations maintained ample AWC appointment availability within two weeks throughout the study. Secondary outcomes included appointment scheduled within 2 weeks, and receipt of first COVID-19 vaccination within 8 weeks. Post hoc secondary outcomes included eligible patient’s receipt of tetanus-diphtheria-acellular pertussis (Tdap) vaccine, human papillomavirus (HPV) vaccine, and meningococcal conjugate vaccine (MCV4). We assessed all outcomes using data from our EHR. AWC completion was determined by a well-visit billing code. Vaccinations administered at any Cincinnati Children’s facility were recorded in our EHR. In addition, our team members can manually document vaccines received elsewhere. Our EHR imports vaccine administration data from external sources, including the Ohio immunization registry and Care Everywhere (Epic’s information exchange application). We were unable to accurately determine if a vaccine was offered but declined by the family.

Sample Size & Blinding

We calculated the sample size for the study based on the hypothesis that 20% of patients in the Standard Message group, 30% of patients in the COVID-19 Vaccine Message group, and 2% of those in the control group would complete an AWC visit within 8 weeks. We based our hypotheses on effect sizes documented for automated text reminders on immunization rates. To detect these differences in AWC visit completion with 80% power at 5% significance level, we required a sample size of 293 per group (879 subjects overall).

We blinded patients and their parents by concealing the clinical trial. We blinded clinical practice teams and our outcome assessor by concealing group allocation.

Statistical Analysis

We conducted descriptive analyses to characterize the participants in terms of demographics and other potential covariates. We used chi square tests or analysis of variance, as appropriate, to assess for differences across the three groups. We conducted intention-to-treat analysis to compare outcomes among the three randomized groups. We used logistic regression models to examine each of the following binary outcomes: scheduled appointment within 2 weeks, completed appointment within 8 weeks, receipt of vaccine within 8 weeks. We did not adjust for any covariates in these logistic regression models because the three arms were balanced on patient characteristics and potential covariates. Because our study design didn’t account for siblings or those living in the same household, we conducted a sensitivity analysis excluding patients who shared the same phone number. Because some patients did not receive messages as intended, we also conducted a per-protocol analysis excluding these patients.

Results

Baseline Characteristics

Of the 1,522 adolescents assessed for eligibility, 1,235 were eligible and randomized into either the Standard Message group (n = 412), COVID-19 Vaccine Message group (n = 411), or the control group (n = 412) (Figure 1 ). The three groups were very similar, with no statistically significant difference on any characteristics (Table 1 ). Most adolescents were Black (76.7%), Non-Hispanic (95.7%), and had public insurance (88.3%). Overall, 71.7% preferred to receive a text message and 28.3% preferred to receive a phone call. Few patients (0.73%) appeared to have declined childhood vaccine based on our proxy measure of not having received at least one past dose of MMR and DTaP. Community incidence rates of COVID-19 based on 7-day moving average remained low during the study, as increases in our local area due to the Delta variant of COVID-19 occurred in August 2021.

Table 1

Baseline Characteristics

Characteristic	Standard Message, No. (%)	COVID-19 Vaccine Message, No. (%)	Control Group, No. (%)	Total, No. (%)
Unique patients, No.	412	411	412	1235
Age
12	74 (18.0)	72 (17.5)	78 (18.9)	224 (18.1)
13	103 (25.0)	110 (26.8)	99 (24.0)	312 (25.3)
14	105 (25.5)	106 (25.8)	98 (23.8)	309 (25.0)
15	57 (13.8)	54 (13.1)	61 (14.8)	172 (13.9)
16	39 (9.5)	29 (7.1)	33 (8.0)	101 (8.2)
17	34 (8.3)	40 (9.7)	43 (10.4)	117 (9.5)
Sex
Female	192 (46.6)	210 (51.1)	196 (47.6)	598 (48.4)
Male	220 (53.4)	201 (48.9)	216 (52.4)	637 (51.6)
Race
Black or African American	313 (76.0)	319 (77.6)	315 (76.5)	947 (76.7)
White	69 (16.8)	60 (14.6)	65 (15.8)	194 (15.7)
Multiracial	10 (2.4)	17 (4.1)	13 (3.2)	40 (3.2)
Asian	3 (0.7)	2 (0.5)	6 (1.5)	11 (0.9)
American Indian & Alaska Native	1 (0.2)	0 (0)	0 (0)	1 (0.1)
Missing Data	16 (3.9)	13 (3.2)	13 (3.2)	42 (3.4)
Ethnicity
Non-Hispanic/Latinx	390 (94.7)	396 (96.4)	396 (96.1)	1182 (95.7)
Hispanic/Latinx	21 (5.1)	15 (3.6)	15 (3.6)	51 (4.1)
Not reported	1 (0.2)	0 (0)	1 (0.2)	2 (0.2)
Insurance
Public (i.e., Medicaid)	375 (91.0)	357 (86.9)	358 (86.9)	1090 (88.3)
Private	30 (7.3)	47 (11.4)	44 (10.7)	121 (9.8)
Self-pay	7 (1.7)	7 (1.7)	10 (2.4)	24 (1.9)
Communication preference
Text message	279 (67.7)	306 (74.5)	301 (73.1)	886 (71.7)
Phone call	133 (32.3)	105 (25.5)	111 (26.9)	349 (28.3)
Decline childhood vaccine
No past MMR and DTaP	3 (0.7)	4 (1.0)	2 (0.5)	9 (0.7)
Patient lifetime historical no-show rate, M (SD)	0.15 (0.3)	0.19 (0.4)	0.15 (0.3)	0.16 (0.3)

Intention-to-Treat Analyses

There were 43 (10.4%), 27 (6.6%), and 22 (5.3%) patients who scheduled an AWC visit within 2 weeks in the Standard Message, COVID-19 Vaccine Message, and control groups, respectively. The Standard Message group had significantly higher odds of scheduling an AWC visit compared with both the COVID-19 Vaccine Message group (OR, 1.66 [95% CI: 1.00-2.74]) and the control group (OR, 2.07 [95% CI: 1.21-3.52]) (Table 2 ; available at ). Across the Standard Message group, COVID-19 Vaccine Message group, and control group, there were relative differences in the rate of completed AWC visits within 8 weeks (13.6% vs. 13.4% vs. 10.4%), and adolescent receipt of COVID-19 vaccination within 8 weeks (3.9% vs. 2.4% vs. 1.9%), but there were no statistically significant differences in the odds of these outcomes. Among the 34 adolescents who received a COVID-19 vaccine within 8 weeks, 30 (88.2%) received the vaccine at our institution and 4 (11.8%) received the vaccine elsewhere in the community. Among those who were eligible to receive Tdap, HPV and/or MCV4 during the study period, the odds of receiving Tdap and MCV4 within 8 weeks differed significantly between groups, with the message groups outperforming the control group (Table 2). Rates of eligible patients receiving these vaccines within 8 weeks across the Standard Message group, COVID-19 Vaccine Message group, and control group were as follows: Tdap 13% vs. 17% vs. 2%; HPV 12% vs. 10% vs. 7%; MCV4 12% vs. 10% vs. 2%. By the end of the trial, the study population had high rates of vaccination with Tdap (89.5%), and first doses of MCV4 (89.5%) and HPV (82.8%), but a lower percentage of eligible participants had completed the HPV series (54.6%) or received the MCV4 booster (35.3%).

Table 2

Intention-to-treat analysis

Outcome	Comparison	OR (95% CI)	P Value
AWC visit scheduled within 2 weeks	Standard Message vs. Control Group	2.07 (1.21 - 3.52)	0.008
	COVID-19 Vaccine Message vs. Control Group	1.25 (0.70 - 2.23)	0.457
	Standard Message vs. COVID-19 Vaccine Message	1.66 (1.00 - 2.74)	0.049
AWC visit completed within 8 weeks	Standard Message vs. Control Group	1.35 (0.88 - 2.06)	0.165
	COVID-19 Vaccine Message vs. Control Group	1.33 (0.87 - 2.03)	0.193
	Standard Message vs. COVID-19 Vaccine Message	1.02 (0.68 - 1.52)	0.930
Receipt of COVID vaccination within 8 weeks	Standard Message vs. Control Group	2.04 (0.86 - 4.82)	0.104
	COVID-19 Vaccine Message vs. Control Group	1.26 (0.49 - 3.22)	0.631
	Standard Message vs. COVID-19 Vaccine Message	1.62 (0.73 - 3.61)	0.238
Receipt of Tdap within 8 weeks a	Standard Message vs. Control Group	6.50 (0.74 – 311.21)	0.117
	COVID-19 Vaccine Message vs. Control Group	8.66 (1.03 – 408.04)	0.045
	Standard Message vs. COVID-19 Vaccine Message	0.75 (0.19 – 2.89)	0.854
Receipt of HPV within 8 weeks b	Standard Message vs. Control Group	1.86 (0.94 – 3.70)	0.075
	COVID-19 Vaccine Message vs. Control Group	1.47 (0.72 – 3.03)	0.292
	Standard Message vs. COVID-19 Vaccine Message	1.27 (0.67 – 2.38)	0.465
Receipt of MCV4 within 8 weeks c	Standard Message vs. Control Group	5.44 (1.52 – 19.48)	0.009
	COVID-19 Vaccine Message vs. Control Group	4.59 (1.25 – 16.93)	0.022
	Standard Message vs. COVID-19 Vaccine Message	1.19 (0.51 – 2.74)	0.691

OR=Odds Ratio; 95% CI=95% Confidence Interval

Analysis includes all patients eligible to receive the Tdap during the 8-week study period (Total, n = 130; Standard Message group, n = 45; COVID-19 Vaccine Message group, n = 41; Control Group, n = 44). Odds ratios estimated from exact logistic regression analysis, and wide 95% CI due to small cell counts.

Analysis includes all patients eligible to receive the HPV series during the 8-week study period (Total, n = 608; Standard Message group, n = 208; COVID-19 Vaccine Message group, n = 195; Control Group, n = 205).

Analysis includes all patients eligible to receive the MCV4 vaccine or booster during the 8-week study period (Total, n = 344; Standard Message group, n = 116; COVID-19 Vaccine Message group, n = 106; Control Group, n = 122).

We identified 237 patients (19% of 1235 sample; 77 in Standard Message, 83 in COVID-19 Message, 77 in control groups) who shared the same parent phone number with one or more other patients included in the study. Excluding these patients did not change the overall pattern of results, but differences between Standard Message group and control group on AWC visit completion approached significance (p = 0.06). The proportion of those who received the Tdap reached statistical significance between the Standard Message group, COVID-19 Vaccine Message group, and the control group (13% vs. 15% vs. 0%; Fisher1s exact test p = 0.02).

Process Measures

Delivery of text messages had similar success rates across the Standard Message (84%) and COVID-19 Vaccine Message (83%) groups (Figure 1). Common reasons for text messages failing to be delivered included: 12% did not have SMS capabilities, 6% failed at carrier, and 1% previously blocked the number from which the messages were generated. Delivery of messages by phone was impacted by vendor technical difficulties on one day when no calls were sent, leading to the COVID-19 Vaccine Message group receiving only one rather than two phone calls. On the days phone calls were delivered, the success rates were similar to texting. Common reasons for phone call failures included: 1% had a busy tone or did not answer the phone and 0.4% did not have a working phone number. Overall, 342 (83%) of the Standard Message group and 242 (59%) of the COVID-19 Vaccine Message group received the intended intervention on both days. Therefore, we conducted a per-protocol analysis including only patients who received the intended intervention (a total of 996 patients), with the Standard Message group including recipients of text (n = 230) and phone (n = 112) messages, the COVID-19 Vaccine Message group only including recipients of text messages (n = 242), and the control group including those who received no messages (n = 412).

Per-Protocol Analyses

Per-protocol outcome analyses are presented in Table 3 (available at ). Overall, the pattern of results was similar to the intention-to-treat analyses. Across the Standard Message group, COVID-19 Vaccine Message group, and control group there were relative differences in the rate of scheduling an appointment within 2 weeks (11.4% vs. 6.6% vs. 5.3%). The Standard Message group had higher odds of scheduling an AWC visit within 2 weeks of receiving the first intervention message compared with the control group (OR, 2.28 [95% CI: 1.32-3.93]), but differences compared with the COVID-19 Vaccine Message group were no longer significant (OR, 1.82 [95% CI: 0.99-3.34]). Although there were relative differences in AWC completion within 8 weeks across the Standard Message, COVID-19 Vaccine Message and control groups (14.0% vs. 12.8% vs. 10.4%), there were no significant differences between the groups on odds of completing the AWC visit within 8 weeks. Completion of COVID-19 vaccine within 8 weeks differed across the Standard Message (4.7%), COVID-19 Vaccine Message (3.3%) and control (1.9%) groups, with significant differences in the odds between the Standard Message group and the control group (OR, 2.48 [95% CI: 1.05-5.86]). The Standard Message group still had higher odds of receiving an MCV4 compared with the control group, but differences between the COVID-19 Vaccine Message group and control group were no longer significant.

Table 3

Per-protocol analysis

Outcome	Comparison	OR (95% CI)	P Value
AWC visit scheduled within 2 weeks	Standard Message vs. Control Group	2.28 (1.32 - 3.93)	0.003
	COVID-19 Vaccine Message vs. Control Group	1.26 (0.65 - 2.44)	0.503
	Standard Message vs. COVID-19 Vaccine Message	1.82 (0.99 - 3.34)	0.054
AWC visit completed within 8 weeks	Standard Message vs. Control Group	1.4 (0.9 - 2.17)	0.132
	COVID-19 Vaccine Message vs. Control Group	1.26 (0.77 - 2.06)	0.356
	Standard Message vs. COVID-19 Vaccine Message	1.11 (0.68 - 1.8)	0.670
Receipt of COVID vaccination within 8 weeks	Standard Message vs. Control Group	2.48 (1.05 - 5.86)	0.039
	COVID-19 Vaccine Message vs. Control Group	1.73 (0.64 - 4.66)	0.281
	Standard Message vs. COVID-19 Vaccine Message	1.44 (0.6 - 3.41)	0.413
Receipt of Tdap within 8 weeks a	Standard Message vs. Control Group	7.76 (0.81 – 385.15)	0.089
	COVID-19 Vaccine Message vs. Control Group	6.58 (0.49 – 364.47)	0.208
	Standard Message vs. COVID-19 Vaccine Message	1.17 (0.20 – 8.45)	0.999
Receipt of HPV within 8 weeks b	Standard Message vs. Control Group	1.81 (0.88 – 3.70)	0.105
	COVID-19 Vaccine Message vs. Control Group	1.56 (0.68 – 3.57)	0.290
	Standard Message vs. COVID-19 Vaccine Message	1.16 (0.53 – 2.52)	0.715
Receipt of MCV4 within 8 weeks c	Standard Message vs. Control Group	5.39 (1.46 - 19.91)	0.012
	COVID-19 Vaccine Message vs. Control Group	3.31 (0.76 - 14.30)	0.110
	Standard Message vs. COVID-19 Vaccine Message	1.63 (0.54 - 4.94)	0.388

OR=Odds Ratio; 95% CI=95% Confidence Interval

Analysis includes all patients eligible to receive the Tdap during the 8-week study period (Total, n = 98; Standard Message group, n = 32; COVID-19 Vaccine Message group, n = 22; Control Group, n = 44). Odds ratios estimated from exact logistic regression analysis, and wide 95%CI due to data with small cell counts.

Analysis includes all patients eligible to receive the HPV series during the 8-week study period (Total, n = 483; Standard Message group, n = 171; COVID-19 Vaccine Message group, n = 107; Control Group, n = 205).

Analysis includes all patients eligible to receive the MCV4 vaccine or booster during the 8-week study period (Total, n = 279; Standard Message group, n = 92; COVID-19 Vaccine Message group, n = 65; Control Group, n = 122).

Discussion

In this clinical trial of automated text and telephone outreach messages among adolescents due for preventative services, the Standard Message positively impacted AWC visit scheduling but not completion. Both message groups outperformed the control group on MCV4 vaccine receipt within 8 weeks among those eligible suggesting messaging may ultimately impact routine adolescent vaccination. Per-protocol analyses suggest the Standard Message positively impacted adolescent receipt of the COVID-19 vaccine compared with the control group.

There was a significant increase in the scheduling of appointments within 2 weeks of receiving the Standard Message compared with the COVID-19 Vaccine Message and no message, indicating gentle outreach nudges may remind families of delayed care and prompt some action. The COVID-19 Vaccine Message group had rates of visit scheduling similar to the control group. Despite known vaccine hesitancy in many populations similar to ours,30, 31, 32, 33 we anticipated that notification of the COVID-19 vaccine availability directly in their primary care medical home, which is familiar and often trusted, would provide an incentive to families seeking access to the vaccine. It is possible that messaging about the COVID-19 vaccine deterred some vaccine-hesitant families concerned they would be pressured to accept the vaccine if they scheduled and completed an appointment. It is also possible that this difference was due to lower intervention fidelity in the COVID message group as this finding was no longer significant in per-protocol analyses.

The rate of AWC visit completion in our study was lower than previous studies,21, 22, 23 , , with estimates ranging from 14% to 72%. Our rate of 10.4% in the Standard Message group is similar to that achieved by O’Leary et al’s bi-directional text messaging intervention. Interventions by Szilagyi et al , , and Suh et al yielded much higher rates. There are plausible explanations for this. First, the duration of follow-up in these other studies was markedly longer, ranging from 6 to 18 months. We chose 8 weeks as a more proximal outcome measure given easy appointment access in our system and wanting to yoke the appointment to our specific intervention. Second, over half of the sample in previous studies , , had an AWC visit in the 12 months prior to intervention vs. zero in our study. Our families may have been under-utilizers of healthcare and less likely to return for AWC. Third, prior studies deployed higher intensity interventions. In contrast to other studies that provided many more messages over a longer time period, we provided two messages one day apart to examine the use of a simple, easy to implement strategy. Comparative studies included their whole patient population, whereas we excluded patients who had an active EHR portal account. This may have impacted results, as the included population who lacked portal access might be even harder to reach.. Simple outreach messages may not be sufficient to re-engage patients that have fallen away from the medical system, especially during the pandemic. Vaccine hesitancy, which appears to be increasing nationally, may also have deterred some families. It is unclear if the challenge to get families to return to well-care relates to worsening trust or skepticism of the medical system, or the profound impact of the pandemic on Black families including higher rates of unemployment and death, or other factors. , , Regardless, more is needed to help adolescents catch-up, specifically those most at-risk and traditionally underrepresented and not engaged by the medical community. Some have advocated for converting sick visits to AWC visits, when possible. Tiered approaches to outreach, including home visitation, may also be warranted.

The two-fold increase in odds of COVID-19 vaccine receipt observed in per-protocol analyses among adolescents in the Standard Message group compared with the control group was encouraging and consistent with strong evidence of effect for a variety of reminder systems increasing pediatric vaccine initiation. That said, the overall vaccination rates were extremely low: 3.9% vs. 1.8%. Over 80% of our sample had become eligible for COVID-19 vaccine when the EUA was expanded 3 weeks prior to receipt of intervention messages in this study. It is possible that many weren’t ready to accept the COVID-19 vaccine and that multiple messages and conversations will be needed to answer questions and address concerns.

Additional qualitative research is needed to develop effective strategies to promote uptake of the COVID-19 vaccine in culturally relevant ways. Framing of future outreach messages may be informed by recent intervention studies designed to increase influenza vaccination. Reminders were not effective at increasing initiation in a large pediatric study, regardless of whether the message was framed as gains or losses, but reminders did promote completion among those who needed a second shot. In contrast, reminder messages to adults who already had a visit scheduled were effective at increasing flu vaccine rates, with the most effective messages framed as “a vaccine has been reserved for you.” In addition to outreach, effective communication approaches are needed to engage vaccine hesitant adolescents and their parents during visits. It is unclear whether past approaches to address vaccine hesitancy when discussing routine childhood vaccines will remain effective in the context of COVID-19 vaccine.

Study strengths include the use of a clinically relevant intervention, a large, marginalized population, ascertainment of important outcomes (visit completion and vaccination rates), and an intention-to-treat design. Our study has several limitations. It was conducted in academic primary care practices serving low income, predominantly Black patients, which limits the generalizability of results. Due to challenges related to attribution, it is likely we included some adolescents who were not current patients of the practices. Our groups were contaminated with adolescents in the same household (i.e., the same parental phone number) being allocated to different groups. This biased our results towards the null hypothesis, though sensitivity analyses suggest the impact was minimal. There are currently barriers to identifying household members within EHRs, providing researchers and EHR vendors an opportunity to create innovative population management tools. Like other studies, delivery of automated messages was limited by incorrect phone numbers, lack of SMS capabilities, etc. In addition, we experienced technical difficulties for phone call delivery which further decreased the fidelity of our automated message intervention. Finally, the racial composition of our study team does not reflect that of our patient population. We engaged parents and practice outreach medical assistants to craft the messages but would have benefited from collaboration with family members throughout the research process.

This data reinforces the challenges recovering from the COVID-19 pandemic, as the delays in AWC threaten future health outcomes. However, we found messages had some beneficial spillover effects for vaccine rates. Additional study is needed to understand whether the timing of follow up messages could realize additional successes in the completion of adolescent preventative services and address widening disparities.