Veterans' response to an automated text messaging protocol during the COVID-19 pandemic.

The US Department of Veterans Affairs (VA) is using an automated short message service application named "Annie" as part of its coronavirus disease 2019 (COVID-19) response with a protocol for coronavirus precautions, which can help the veteran monitor symptoms and can advise the veteran when to contact his or her VA care team or a nurse triage line. We surveyed 1134 veterans on their use of the Annie application and coronavirus precautions protocol. Survey results support what is likely a substantial resource savings for the VA, as well as non-VA community healthcare. Moreover, the majority of veterans reported at least 1 positive sentiment (felt more connected to VA, confident, or educated and/or felt less anxious) by receiving the protocol messages. The findings from this study have implications for other healthcare systems to help manage a patient population during the coronavirus pandemic. Published by Oxford University Press on behalf of the American Medical Informatics Association 2020. This work is written by US Government employees and is in the public domain in the US.

INTRODUCTION

Robust evidence supports the value of integrating SMS/text-messaging interventions into healthcare delivery and population health practice. The US Department of Veterans Affairs (VA) Office of Connected Care developed and implemented an automated short-message service (SMS) application for guided patient self-management, named Annie. Developed by the VA, Annie is modeled after a system used by the National Health Service in England called Flo. The platform is hosted on Amazon Web Services Veterans Affairs Enterprise Cloud – Mobile Applications Platform. Annie makes use of the same shared services and databases VA health apps utilize, but it is not directly connected to the Computerized Patient Record System or My HealtheVet, VA’s patient health record and patient portal, respectively. Within Annie, clinicians can assign protocols to patients and view individual veteran responses. Veterans use SMS text to exchange messages with Annie or they can log in to exchange messages and see graphs of their readings.

Annie can deliver educational and motivational messages to veterans’ mobile phones, send reminders for health tasks, and query about their current health status. This can encourage veterans to adopt healthy habits and monitor progress toward wellness goals. The flexible design and branching logic allow clinical teams to develop protocols tailored to a variety of health conditions. Annie enables clinical teams to assign protocols to patients based on clinical and social conditions as well as user preferences; clinical teams may or may not review responses. In addition, for certain protocols, veterans can self-enroll. Currently, there are over 170 clinical protocols and around 19 000 Annie users. The VA used Annie as part of its coronavirus disease 2019 (COVID-19) response, adding several protocols, including a coronavirus precautions protocol (CPP).

After the first known cases of COVID-19 were diagnosed in the US in February 2020, many healthcare organizations, including VA facilities, were overwhelmed with calls seeking guidance. Within the VA, call center volume increased by 21% from February (3 488 478 total calls) to March (4 217 229 total calls). Similarly, the VA saw a sharp increase in secure messages. In February 2020, the number of secure messages sent and received by veterans ranged from a low of 354 832 (week of Feb 16) to 413 084 (week of March 1). In the following weeks, secure messaging continued to increase to a high of 622 858 the week of March 15.

On March 4, 2020 a request was made through the Veterans Health Administration Emergency Management Coordination Cell to develop an Annie protocol to assist veterans during COVID-19 response efforts. A team of clinical home telehealth (remote monitoring) and Annie experts convened on March 5 for a one-time, virtual advisory meeting to shape 2 protocols for COVID-19 support: CPP and Isolation/Quarantine. Protocol development and testing occurred March 6-9 and were presented to Emergency Management Coordination Cell clinical subject matter experts on March 10. Expert concurrence was obtained March 11 and Veterans Health Administration senior leadership approved the protocol for use nationally on March 13. It was made available the same day.

The CPP (Figure 1) is a 2-month protocol that delivers educational content on coronavirus precautions, guides veterans in monitoring their temperature and other clinical symptoms, and advises them to call their care team or a nurse triage line when they cross certain clinical thresholds. For the first 2 weeks the protocol was offered, check-ins were sent daily. Currently, messages include wellness questions sent Monday, Wednesday, Friday, Sunday (“Annie here. Are you feeling well today?”) and educational tips (eg, “Annie again. If you are ill, use a separate bedroom and bathroom if possible.”) sent Monday, Wednesday, and Friday. Annie always identifies herself, eg, “Annie here.”

Figure 1.

Flow map depicting the decision tree for the coronavirus precautions protocol wellness check questions. Not illustrated in the figure: messages always include some reference to “Annie,” either as the greeting (eg, “Annie here”) or at the end of the message (“-Annie”). For the wellness check questions, Annie asks for a response and provides predetermined response choices.

Veterans can self-enroll in this protocol or be added by a member of their care team. The Office of Connected Care reached out to veterans across the country encouraging them to enroll in Annie and subscribe to the CPP, and around 4200 self-enrolled. As the number grew, we decided to invite CPP subscribers to provide feedback on Annie’s utility and value to them during this period.

MATERIALS AND METHODS

A subset of the authors designed a set of 9 questions to assess the respondents’ geographic distribution, self-reported clinical status, reactions to the Annie CPP messages, and actions taken or not taken because of the messages they received. The questions included multiple formats in which the respondents could choose all responses that applied, as well as open-ended questions allowing free-text responses. The questions were loaded into SurveyMonkey and made available via internet link. On April 22, 2020, a broadcast message was sent to 11 630 users of the Annie application who had that feature enabled, which is 81% of all Annie users. At that time, there were approximately 4400 active users of Annie’s CPP (96% self-enrolled; the other 4% enrolled by a VA staff member). Data was exported from SurveyMonkey, April 23, 2020, 24 hours after the link was sent and transferred to an Excel spreadsheet for analysis. Data were summarized with descriptive statistics.

Questions that included open-ended responses were analyzed using an inductive coding approach, by a single team member (ie, allowing codes to emerge from the data rather than using a predetermined coding scheme) with an independent audit of all coding by a coauthor. This type of auditing procedure by a second person is considered an acceptable alternative to using independent coders for ensuring validity of the analysis. Questions raised by the audit were resolved by consensus calls as needed. For 2 questions, a secondary analysis of the content for each primary code was warranted based on the complexity of the responses to further reveal subpatterns. For example, for the primary code “Connected” for the question “What about Annie’s messages was helpful for you?” the first author then summarized the different types of subpatterns for “Connected” as subpatterns under this primary code.

RESULTS

Of the 11 630 veterans who received an invitation to participate, 1567 (13.5%) went to the SurveyMonkey™ website. Of those, 188 (12.0%) reported that they were not subscribed to the CPP Annie protocol, so were excluded from analysis. Of the remaining 1379 who were eligible, 245 (17.8%) did not complete the survey, but we retained partial responses; 1134 veterans responded to all questions. This response was from a total of 4400 veterans who were actually subscribed to the CPP protocol, so the calculated response rate is 25.8% (1134/4400).

Table 1 shows the questions that the respondents answered, along with a summary of the results for each question. Responses were received from veterans in all 50 states plus the District of Columbia and Puerto Rico. Five of the 6 most highly represented states in terms of ratio of respondents to veterans living in that state were within a single VA network (VA Midwest Health Care Network: Minnesota, South Dakota, North Dakota, Iowa, and Nebraska) that was an early adopter and promoter of Annie. Following Table 1 is further analysis of veterans’ free-text responses to open-ended questions and when they selected the “other” option for various questions.

Table 1.

Summary of survey responses

Questions & Responses	Number of respondents	Percentage of respondents (%)
In what state or US territory do you live?	1,291
California	102	7.9
Florida	107	8.3
Minnesota	120	9.3
Texas	103	8.0
Other (each <5%)	859	66.5
Why did you choose to subscribe to Annie's Coronavirus Precautions protocol messages? Please check all that apply to you.	1,305
My VA healthcare team recommended it	422	32.3
A VA publication recommended it	386	29.6
Another Veteran recommended it	22	1.7
I wanted tips to protect myself	592	45.4
I felt nervous or anxious	146	11.2
I felt sick	27	2.1
I wanted a resource in case I felt sick	458	35.1
I wanted to connect to VA without making a phone call	291	22.3
Other (please specify)	101	7.7
Have you ever replied “No” to Annie's question “Are you feeling well today?” ?	1305
Yes	249	19.1
No	1056	80.9
Have Annie's messages been helpful?	1301
Yes	977	75.1
No	66	5.1
Not Sure	258	19.8
Did you take any of these actions as a result of Annie's Coronavirus Precautions messages? Please check all that apply to you.	1134
Followed the stay well tips that Annie offered	803	70.8
Sent a secure message to my VA care team	92	8.1
Called my VA care team	58	5.1
Called my community (non-VA) care team or my local health department	17	1.5
Sought care from my VA care team or from a VA Emergency Department	19	1.7
Sought care in a community clinic, such as at an Emergency Department or Urgent Care	12	1.1
Recommended this Annie protocol to another Veteran	124	10.9
Stopped messages	4	0.4
None/no other actions	334	29.5
Other (please specify)	24	2.1
Would you have taken any of the following actions if you were not receiving Annie's Coronavirus Precautions messages? Please check all that apply to you.	1134
Sent a secure message to my VA care team	364	32.1
Called my VA care team	305	26.9
Called my community (non-VA) care team or my local health department	79	7.0
Sought care from my VA care team or from a VA Emergency Department	203	17.9
Sought care in a community clinic, such as at an Emergency Department or Urgent Care	119	10.5
Asked a friend or family member for advice	136	12.0
None/no other actions	481	42.4
Other (please specify)	46	4.1
What impact have Annie's Coronavirus Precautions messages had on you? Please check all that apply to you.	1134
Felt more connected to VA	669	59.0
Felt less nervous or anxious	323	28.5
Felt more confident about when to seek care	442	39.0
Felt more educated about coronavirus in general	380	33.5
Changed my behavior to align to Annie's tips	210	18.5
Opted not to call my VA care team because I had the information I needed from Annie's messages	118	10.4
Opted not to call my community (non-VA) care team or my local health department because I had the information I needed from Annie's messages	63	5.6
Opted not to seek face to face care because I knew of a better action from Annie's messages	77	6.8
Felt annoyed by the messages and/or stopped them	26	2.3
None/no impact	202	17.8
Other (please specify)	27	2.4

Why did you choose to subscribe to Annie’s coronavirus precautions protocol messages?

Frequent free-text responses in the “Other” field for the question “Why did you choose to subscribe to Annie’s Coronavirus Precautions protocol messages?” included reasons such as curiosity, health management, and simply to stay informed.

What about Annie’s messages was helpful for you?

This was an open-ended question. Table 2 shows a summary of the qualitative analysis conducted on the 808 free-text responses.

Table 2.

Summary of free-text responses toWhat about Annie’s messages was helpful for you?”, including coded categories, definitions for each category, frequency count, and example responses

Codes	Definitions	Count	Example free-text responses
Informative	Information provided is valued by veteran	195	I’m proud of the VA’s medical personnel for being so proactive in this pandemic and designing a tool that keeps me informed. The more I know, the more I can help myself and help others.
Remindful	Reminds me to do X or look out for Y	150	Reminds me of safety procedures I should be doing and anything new that I haven’t heard about.
Connected	Mentions contact with someone else, feeling like not being alone (not just feeling of security—actual connectedness with someone else); or just mentioning the daily check-in	124	She [Annie] lets me know that someone is there virtually, and if I need her, she will always be there 24/7/365.
Reassuring	Peace of mind, feeling safe, reassurance, comforting, less stress and/or anxiety	65	Just knowing there is a link and the brief msg asking ok or not helps being, even momentarily, in the moment and requires some action … a crutch for corona! Annie keeps reminding us on precautions, love how she keeps asking how we are feeling. She's there for us!! As is the VA!!
Caring	VA cares, is watching out for me, wants to make sure I am ok	50	It’s having someone checking up on you. It’s someone at the VA who cares, because most of the veterans I know feel as if the VA doesn’t. It made me feel better that someone cares about me and my health. They gave me a sense that, despite all the chaos, there was someone that cared just about me.
Advisory	More than just information—actual advice on what to do	47	Knowing that [if] I started feeling sick, suggestions re what to do would follow.
Encouraging	Mentioned encouragement, motivation, positive, uplifting, etc.	35	To know that Annie alerts keep me positive and happy to know that someone cares about my well being! Keeps me mindful of what I need to do if I have to go out.
Timely	Messages are timely; keeps veteran “up to date”	24	Made me feel up to date with information on COVID-19.
Focused	Helps veteran focus or keep on track	14	Helps to keep me on track protecting myself and my grandchildren, 2 of which I have see [sic] daily contact with while their mom works.
Clear	Messages are easy to follow	5	Simple, short, easy to understand and implement.
No code	Not enough info to assign a code or not relevant	99

A secondary analysis of the categories revealed subpatterns for the “Connected” category. These subpatterns included:

Veterans liked a feeling of contact with another person or entity (Annie). (49)

Many veterans specifically mentioned liking the daily contact. (35)

Veterans simply liked feeling connected in general. (15)

A dozen veterans specifically mentioned feeling connected with the VA. (12)

Seven veterans liked that their status was being checked on. (7)

Six veterans specifically mentioned that the messages made them feel “not alone.” (6)

Why were Annie’s messages not helpful to you?

This open-ended question received 61 responses. Most of those responses were coded as “Not enough information” (34); that is, they felt that the Annie messages did not provide enough information to be valuable to the user or that they shared repeated, common knowledge. Others reasons included “Too many messages” (3), the user did not receive a reply from Annie (3), the user did not need the help (3), and the user was limited in how they send messages to Annie (4). For example, user was unsure how to respond to the question “Are you feeling well today?” because “…you can have different ways of not feeling well.” Other responses were one-offs or had insufficient information to be coded.

Did you take any of these actions as a result of Annie’s coronavirus precautions messages?

Frequent free-text responses in the “Other” field mostly included users reporting that they were not receiving messages at all or messages that required action of them (13). Other responses had insufficient information in their response to be coded.

Would you have taken any of the following actions if you were not receiving Annie’s coronavirus precautions messages?

The most frequent type of free-text response in the “Other” field included veterans reporting that they would have sought information from alternative information sources such as the Centers for Disease Control (5), the World Health Organization (1), news outlets (4), a health department (2), medical professionals (2), or the internet in general (6). Other responses had insufficient information in their response to be coded.

What impact have Annie’s coronavirus precautions messages had on you?

Although most “Other” responses had insufficient information to be coded, a couple of users had a critique of the messages, the Annie application, or the process to sign up (2). Others simply indicated that they use Annie as an information source (3).

DISCUSSION

Survey findings support that the use of Annie’s CPP resulted in what is likely a substantial resource savings for the VA, as well as non-VA community healthcare. Of the 1134 veterans who answered the question “Would you have taken any of the following actions if you were not receiving Annie’s Coronavirus Precautions messages?”, 532 (47%) report they would have reached out to VA in at least 1 way (secure message, phone call, visit, or some combination of those). In addition, 159 (14%) veterans reported they would have interacted with community (non-VA) care (called or sought care). Of the 1134 veterans who answered the question “What impact have Annie’s coronavirus precautions messages had on you?”, 172 (15%) reported that they opted not to interact (message, call, or visit) with a healthcare provider (VA or community) because Annie provided the information they sought.

Moreover, 124 (11%) responses mentioned a sense of connection as something that was helpful in Annie’s messages. This is important when feelings of isolation due to lockdowns and closures begin to set in: the landing page for Mental Health Apps on mobile.va.gov saw an 8476% increase in pageviews from March to April.

Additionally, 879 (76%) of veterans reported at least 1 positive sentiment (felt more connected to VA, confident, or educated and/or felt less anxious) as 1 impact of CPP. This is in line with previous studies on Annie. The over 800 open-ended responses for the question “What about Annie’s messages was helpful for you?” also support the positive sentiment findings (Table 2).

CONCLUSION

The findings from this study have implications for other healthcare systems. The use of an automated SMS/text service to help manage a patient population during the COVID-19 pandemic is a cost-effective way to simultaneously encourage patient monitoring and provide updated information while demonstrating potential healthcare resource savings. In addition, because those who subscribed to the CPP messages were overwhelmingly positive about receiving them on a regular basis, we have shared the messages and flow map should any healthcare system want to use them as the country continues to navigate through the COVID-19 pandemic.

These findings should be interpreted within the context of certain limitations. First, as guidance is evolving, some changes should be expected to the protocol itself. Around 600 Veterans had opted out of receiving Annie CPP messages prior to our survey, and so those Veterans are not represented in our sample. However, that is still a small percentage of the active subscribers. Also, we did not collect data on how long each veteran has been subscribed to the CPP messages. The length of time a veteran has been receiving the messages may have influenced their responses. Lastly, a couple of free-text responses noted that responders were not receiving messages. Since completion was on a “self-selection” basis, it is possible that a few responses were not based on experience with CPP. However, that is true of many survey opportunities. Despite these limitations, the survey seems to suggest that the Annie CPP has had a substantial positive impact on the veteran population.

FUNDING

This work was supported by the Department of Veterans Affairs, Veterans Health Administration, Office of Health Informatics (10A7), Human Factors Engineering, Contract 36C776 19 D0017/36C776 19 N0142, awarded to Cognitive Medical Systems, Inc.

AUTHOR CONTRIBUTIONS

KLF, JJM, NRW, BV, JH, JJS conceived and designed the study. JH and BV were responsible for the survey data collection. BL, JH, JJM, JJS, and JMR contributed to the analysis and interpretation of the survey data. BV, BL, and NRW provided protocol information such as history and usage statistics. All authors revised the manuscript critically for important intellectual content. JJS had principal responsibility for drafting the manuscript. All authors approved the final version.