Rapid design and implementation of an integrated patient self-triage and self-scheduling tool for COVID-19.

OBJECTIVE: To rapidly deploy a digital patient-facing self-triage and self-scheduling tool in a large academic health system to address the COVID-19 pandemic.
MATERIALS AND METHODS: We created a patient portal-based COVID-19 self-triage and self-scheduling tool and made it available to all primary care patients at the University of California, San Francisco Health, a large academic health system. Asymptomatic patients were asked about exposure history and were then provided relevant information. Symptomatic patients were triaged into 1 of 4 categories-emergent, urgent, nonurgent, or self-care-and then connected with the appropriate level of care via direct scheduling or telephone hotline.
RESULTS: This self-triage and self-scheduling tool was designed and implemented in under 2 weeks. During the first 16 days of use, it was completed 1129 times by 950 unique patients. Of completed sessions, 315 (28%) were by asymptomatic patients, and 814 (72%) were by symptomatic patients. Symptomatic patient triage dispositions were as follows: 193 emergent (24%), 193 urgent (24%), 99 nonurgent (12%), 329 self-care (40%). Sensitivity for detecting emergency-level care was 87.5% (95% CI 61.7-98.5%). DISCUSSION: This self-triage and self-scheduling tool has been widely used by patients and is being rapidly expanded to other populations and health systems. The tool has recommended emergency-level care with high sensitivity, and decreased triage time for patients with less severe illness. The data suggests it also prevents unnecessary triage messages, phone calls, and in-person visits.
CONCLUSION: Patient self-triage tools integrated into electronic health record systems have the potential to greatly improve triage efficiency and prevent unnecessary visits during the COVID-19 pandemic.

INTRODUCTION

In December 2019, reports emerged from Wuhan, China of an outbreak of viral pneumonia caused by the novel Coronavirus SARS-CoV-2. The viral illness has since become a global pandemic, infecting over 2.3 million people and resulting in over 160 000 deaths as of April 18, 2020. On February 28, 2020, the first known case of community-spread 2019-novel coronavirus (COVID-19) in the United States was reported in Northern California. Health systems in California and across the country began to prepare for a surge in volume in both the ambulatory and inpatient settings.

One of the earliest effects on health systems was a sharp increase in the volume of phone calls, patient portal messages, and appointment requests from patients who had questions or concerns about COVID-19. The increased demand on ambulatory clinical capacity created several health system challenges. First, it was difficult to provide care to the patients who needed it most when front-line clinicians and staff were spending a large proportion of their time on triage. Second, the surge of patients walking into urgent care and primary care practices for advice created an infection control hazard. Third, in the setting of rapidly changing information and guidelines, it was difficult to maintain consistency in medical recommendations and advice. Lastly, patient experience suffered, with unusually long telephone hold times, delayed message responses, and limited appointment availability.

Electronic health record (EHR)-tethered patient portals enable patients to view test results, communicate with their care team, and schedule appointments, including telehealth visits. Portal use can improve satisfaction and engagement of both clinicians and patients. In most EHR-tethered patient portals, symptom triage and medical advice is only accessible through asynchronous secure messaging between patients and clinicians; inbound messages arrive unfiltered and without a triage mechanism to enhance clinical care efficiency, even in the case of repetitive and algorithmic tasks. More recently, some health systems have begun employing patient self-triage modules and symptom checkers as a first point of contact for patients with new symptoms. These tools have the potential to efficiently allocate resources by providing automated triage advice and by linking patients to the optimal level of care.

In response to this emergent demand upon our health system, we designed and rapidly implemented a patient portal-based self-triage and self-scheduling tool. The goal was to direct patients to targeted intake, advice, information and care for respiratory symptoms and COVID-19 concerns.

OBJECTIVE

To rapidly deploy a digital patient-facing self-triage and self-scheduling tool in a large academic health system to address the COVID-19 pandemic.

METHODS

Setting

University of California, San Francisco (UCSF) Health is a large academic health system consisting of 3 campuses, with nearly 1000 inpatient beds and 9 primary care practices serving approximately 90 000 patients. UCSF Health serves approximately 45 000 hospital admissions and 1.7 million outpatient visits annually. It is 1 of 5 academic medical centers within UC Health and the broader 10-campus University of California system. In February 2020, UCSF was one of the first health systems in the country to care for COVID-19-positive patients.

Tool

UCSF uses a commercially available EHR from Epic Systems (Verona, WI). We used an Epic toolkit to design and deploy our UCSF Coronavirus Symptom Checker. The toolkit is primarily intended to aid in patient appointment scheduling, allowing the use of complex, branching logic to determine the appropriate provider or visit type and can be accessed by patients through the EHR-tethered patient portal (MyChart).

Using this toolkit, we developed a module for symptom assessment, triage, and appointment scheduling for patients with concerning symptoms or questions about COVID-19. After answering a series of branched logic questions about exposures, symptoms, and comorbidities, patients are segmented into risk categories and directed to 1 of 4 endpoints. Basic demographic information, including age and gender, is automatically populated from the patient’s medical record. Patients can complete the self-assessment as frequently as they wish. All responses and interactions are stored as part of each patient’s medical record.

Population and outreach

The UCSF Coronavirus Symptom Checker was made accessible to all empaneled UCSF primary care patients with active patient portal accounts (approximately 61 000 patients). There was no widespread announcement or promotion of the tool. Patients were directed to the tool only when they sought care by phone or via the patient portal. In the patient portal, icons for the self-triage and self-scheduling tool were placed on the home page and adjacent to the option to “Send a Message” to the provider. Primary care clinics added instructions for accessing the tool to the phone messages that patients hear when calling about COVID-19. Messages were also sent daily to patients with upcoming appointments, instructing them to reschedule if they were experiencing respiratory symptoms. One week after launch, the link to the self-triage and self-scheduling tool was added to this upcoming appointment message so that symptomatic patients could be further triaged.

Clinical content

The clinical content of the self-triage and self-scheduling tool was designed to have high sensitivity for identifying severe disease, and high specificity when recommending self-care. It includes 3 categories of questions: exposures, symptoms, and comorbidities (Supplementary AppendixFigure 1). Exposure and travel questions were adapted from Centers for Disease Control and Prevention (CDC) guidelines and updated as the outbreak spread globally. Symptom and comorbidity questions were adapted from nursing telephone triage protocols, clinical practice guidelines,,, and early reports of clinical symptomatology of COVID-19.,, Over 15 national experts in infectious disease, acute respiratory tract infections, and ambulatory triage were consulted and provided input on this model. Question phrasing was continuously revised based on feedback from front-line triage and advice nurses and physicians to clarify any elements that caused patient confusion.

Figure 1.

Adult COVID-19 patient self-triage protocol.

The self-triage and self-scheduling tool protocol (Figure 1) includes 4 endpoints for symptomatic patients: self-care, nonurgent evaluation, urgent evaluation, and emergent evaluation. Patients with 1 of 4 life-threatening symptoms (chest pain, severe shortness of breath, facial cyanosis, or confusion) are instructed to call 911 or present to the emergency department (ED), and are also given the phone number to the UCSF COVID-19 Hotline if they prefer to be retriaged by a nurse. If patients are classified as “urgent” evaluation, they are directed to call the UCSF COVID-19 Hotline to make an appointment within 12 hours at the newly established UCSF Respiratory Screening Clinic. Patients classified as “nonurgent” are triaged to a visit in the UCSF Video Acute Care Clinic, ideally within 12 hours, but up to within 24 hours. One week after the tool’s launch, direct scheduling into the Video Clinic was activated for patients in the nonurgent endpoint. Patients determined to be at low risk of having severe illness are recommended to care for themselves at home and provided with return precautions and detailed instructions on symptom management.

Operational processes

After patients complete the self-triage and self-scheduling tool, their responses and the triage recommendation are automatically stored in the EHR as Patient Self-Triage encounters. Patients are instructed to initiate contact if further care is needed and they have not already self-scheduled. The Patient Self-Triage encounters are reviewed by clinicians only if patients call the triage hotline or clinic or make an appointment. There is no proactive outreach or review of all patient responses.

The UCSF COVID-19 Hotline, staffed by clinical navigators and registered nurses, uses the same UCSF Adult COVID-19 Triage Protocol (Figure 1 and Supplementary AppendixFigure 2) for patients who do not have access to, or are unable to use, the self-service portal tool.

Figure 2.

COVID-19 self-triage CONSORT diagram.

Triage outcomes

Care intensity was defined in descending order: emergency department, in-person visit, video visit, telephone call, patient portal message. We reviewed each patient’s highest level of care received within 48 hours of completing the self-triage tool. If the patient completed the self-triage multiple times, the result of the most recent completion was compared to the final disposition. For the purposes of determining test characteristics of the tool, telephone calls and patient messages were not considered visits. Daily reports of patient self-triage use and demographic information were extracted from the EHR database. Charts were manually reviewed for all patients who ultimately had an ED visit.

Differences in time to scheduling were compared using the Wilcoxon rank sum test. Statistical analysis was performed using R 3.6.4.

RESULTS

On February 29, 2020, UCSF Health assembled a multidisciplinary team to develop a patient portal-based self-triage tool. By March 6th, we completed the clinical design and workflow for the self-triage protocol. Building the protocol into the EHR required approximately 100 hours of in-house Epic developer time. The UCSF Coronavirus Symptom Checker was launched on March 12, 2020 and made available to approximately 60 865 UCSF primary care patients.

Through the first 16 days of use, the self-triage and self-scheduling tool was accessed 1327 times (defined as answering at least 1 question) and completed 1129 times by 950 unique patients (85% completion rate) (Figure 2). The median age of patients completing the symptom checker was 42 years (interquartile range [IQR] 34–56 years), 62% were female, and race/ethnicity was 49% Caucasian, 23% Asian, 12% Hispanic, 7% non-Hispanic black, and 10% other (Table 1). Seventy-three percent were commercially insured. By comparison, the overall primary care population at this health system has a median age between 40 and 50 years, is 67% female, 47% Caucasian, and 48% commercially insured.

Table 1.

Demographics of patients completing coronavirus symptom checker

Unique patientsa		948
Sex (% Male)		38.0
Patient age (years)	Median	41.6
	IQR	(33.6–55.8)
Race/ethnicity (%)	Caucasian	48.6
	Non-Hispanic black	7.5
	Hispanic or Latino	11.7
	Asian	22.4
	Other/Unknown	9.8
Marital status (%)	Married/Partnered	46.7
	Single/Separated	50.8
	Unknown/Declined	1.5
County type (%)	Urban	99.6
	Rural	0.4
Insurance type (%)	Commercial	72.5
	Medicare	13.0
	Medicaid	0.0
	Self-pay	0.9

Demographics unavailable for 2 patients.

Abbreviation: IQR, interquartile range.

Of completed sessions, 315 (28%) were asymptomatic, and 814 (72%) were symptomatic. Fifty-two asymptomatic patients (17%) had a positive travel or exposure history and received targeted information explaining how to self-quarantine and when to contact a physician. The remaining 263 asymptomatic patients (83%) did not report exposure or travel and were provided with general information and answers to frequently asked questions about COVID-19. Symptomatic patient triage dispositions were as follows: 193 emergent (24%), 193 urgent (24%), 99 nonurgent (12%), 329 self-care (40%). Use and disposition over time are shown in Figure 3.

Figure 3.

Coronavirus symptom checker use & disposition over time.

The highest level of care received within 48 hours of using the symptom checker is shown in Table 2. Of the 16 patients who completed the self-triage tool and went to the ED within 48 hours, 14 had been triaged to the emergent category for a sensitivity of detecting emergency-level illness of 87.5% (95% CI 61.7–98.5%) and specificity of 76.2% (95% CI 72.9–79.5%). Ninety-two patients (53%) triaged to emergent care did not complete a visit within 48 hours at UCSF Health. Of the 2 patients who went to the ED but were not triaged to emergency care, 1 was retriaged by the COVID-19 hotline and recommended for a video visit, but preferred to go to the ED. The other was unable to get an in person respiratory screening clinic appointment, so was advised to go to the ED. Of the 276 patients triaged to self-care, 169 (61%) had no appointments, calls or messages at UCSF Health within the next 48 hours, 84 (31%) sent a message or called the hotline, and 23 (8%) had an in-person visit. Sensitivity and specificity for recommending self-care were 53.7% (95% CI 49.1–58.3%) and 89.5% (95% CI 84.6–93.2%), respectively.

Table 2.

Highest level of care received by disposition category

Highest level of care received within 48 hours of using self-triage tool	Disposition category assigned by self-triage tool
	Emergent	Urgent	Nonurgent	Self-Care	Total
ED visit	14 (8%)	1 (1%)	1 (1%)		16
In-person visit	27 (16%)	43 (28%)	9 (11%)	6 (2%)	85
Video visit	41 (24%)	23 (15%)	36 (43%)	17 (6%)	61
Telephone call	19 (11%)	16 (10%)	2 (2%)	13 5%)	50
Patient portal message	41 (24%)	26 (17%)	12 (15%)	71 (26%)	150
No further action	32 (18%)	47 (30%)	23 (28%)	169 (61%)	271
Total	174	156	83	276	689

Abbreviation: ED, emergency department.

In the 10 days during which video visit direct scheduling was active, 29 patients (56% of patients in the nonurgent disposition category during that time) directly scheduled video visits. The patients who directly scheduled visits had a significantly shorter time from starting the self-triage and self-scheduling tool to scheduling a visit compared to those who called the hotline to schedule (median 2 minutes [IQR 1–6 minutes] vs. 2 hours 15 minutes [IQR 1 hour, 28 minutes–4 hours, 49 minutes], P = < .001).

DISCUSSION

We describe the rapid implementation of a patient portal self-triage and self-scheduling tool for COVID-19. The purpose of this tool is to provide patients with 24-hour access to personalized recommendations and information regarding COVID-19 and to improve ambulatory surge capacity through self-triage, self-scheduling, and avoidance of unnecessary in-person care. In the first 2 weeks of deployment, the tool was completed over 1000 times, and nearly 1 in 60 eligible patients completed it at least once. The tool was used primarily by symptomatic patients but also included more than 300 sessions by asymptomatic patients. The tool was designed to “do no harm”: to have high sensitivity to detect emergency-level illness and high specificity when recommending self-care, both of which were greater than 85%. Despite designing the tool with this conservative approach, the most frequent triage disposition was self-care. The majority of these patients did not make further contact with our health system during the subsequent 2 days. This tool may have therefore prevented hundreds of unnecessary encounters. Eliminating unnecessary in-person visits has the potential to prevent patient exposure to pathogens en route to clinic visits and in waiting rooms, reduce personal protective equipment use by clinic staff, and liberate front-line clinicians to focus on caring for sicker patients. One particularly important benefit of this tool is self-scheduling, which allows patients to be triaged and scheduled in a median of 2 minutes.

It is also worth noting that volume of use decreased in the second week of the studied period. “Shelter in place” went into effect on March 17 in San Francisco and 5 other Bay Area counties. Although these data cannot confirm a “flattening” of the curve, they do show a correlation: use of the UCSF Coronavirus Symptom Checker decreased beginning about 5 days after the order was implemented. It is also possible that initial pent up demand among patient portal users was extinguished after the first week of this tool’s availability.

Over a dozen COVID-19 symptom checkers are now available from private companies, academic medical centers, and government organizations like the CDC. However, this is the first known implementation of a COVID-19 symptom checker that is fully integrated with the patient’s medical record and enables direct appointment scheduling. Although public-facing self-triage tools may be more accessible to patients, their benefits are limited if those patients require further care. EHR integration allows for direct scheduling and recording of responses, eliminating the need for the patient to repeat the history. In a future state with an interoperable application programming interface-driven health IT ecosystem, these integrations will be possible with a wider array of third-party applications providing greater flexibility. However, in the current state, use of the EHR-tethered portal is critical to realizing the full operational benefit of self-triage tools. Additionally, there is an important distinction between many of the tools on the market that are meant to provide likelihood of having COVID-19 and those meant for triage. By designing this as a triage tool, we were able to base the clinical logic on well-established triage principles and clinical guidelines for upper respiratory infections, like influenza. We are adding predictors of severity for COVID-19 as more literature becomes available.

Although this is the first known analysis of a COVID-19 self-triage and self-scheduling tool, our results are consistent with the existing literature for self-triage tools and symptom checkers as a whole. Most studies report high sensitivity to detect severe illness and lower specificity.,, In an evaluation of 23 symptom checkers, Semigran et al reported appropriate triage advice in 80% of emergent cases, 55% of nonemergent cases, and 33% of self-care cases. Our data shows over half of patients triaged to the emergent category did not have a visit in our health system within 48 hours, which is consistent with Semigran’s finding, and may be explained by the limited ability of binary answers to elicit true emergency symptoms. However, the percentage of emergent cases decreased over time as we made changes to the wording of emergent symptom questions, suggesting the potential to improve performance with front-line clinician and patient feedback.

The self-triage and self-scheduling tool we implemented has several limitations. First, this intervention reached a limited patient demographic. It was only available to adult patients with portal access who have primary care physicians at UCSF and was only available in English. Privately insured patients were overrepresented among the population using the symptom checker compared to the primary care population as a whole, which is consistent with literature on the digital divide. Second, in our analysis of patient disposition, we were able to review only the data in our medical record system, so we may have missed events occurring at outside clinics or hospitals. Finally, we were limited in our ability to compare the effects of this intervention to a control group given the primary objective of quickly addressing a pressing health system concern and the unprecedented ambulatory surge precluding the use of an historic control.

This self-triage and self-scheduling tool was presented to our EHR vendor and is now part of their base content, meaning that it is available for use by their other customers. At UCSF, we have expanded access to this tool to pediatrics, affiliate locations, and UCSF specialty care patients who have primary care physicians outside the UCSF network. We are assisting other health systems in implementing similar self-triage tools. We are also working to deploy additional self-triage and assessment tools for use by patients after they are initially evaluated by a physician to help provide follow-up care for patients with respiratory illness.

Additional research is needed to improve symptom checkers for COVID-19. While other common respiratory diseases have robust machine learning and regression models for determining the likelihood of severe illness based on initial presenting factors, there is sparse literature of this type for COVID-19. A current limiting factor is a lack of data on confirmed cases. By compiling large datasets from standardized assessments of patients with respiratory illness during the COVID-19 pandemic, researchers will be able to create better predictive models. These can be used to improve specificity. Research is also needed to understand what percentage of symptom checker volume is redirected from other means of triage volume, and how much is newly generated (such as by interest in using the symptom checker or curiosity about a friend or family member’s symptoms). In this intervention, we avoided marketing the tool to asymptomatic patients to avoid artificially inducing demand, but it is still possible that patients who would not otherwise have presented for care found the symptom checker on the patient portal and completed it.

CONCLUSION

Patient self-triage tools have an enormous potential to improve triage efficiency and to quickly connect patients with the appropriate care venue, preventing unnecessary ED and urgent care visits and reducing potential infectious exposures and transmissions. During the COVID-19 pandemic, these functions are particularly critical. Although more work is needed to improve specificity of these tools, the operational benefits and 24-hour patient access alone warrant the consideration of use of similar tools in other health systems.

FUNDING

No external funding was used for this study.

AUTHOR CONTRIBUTIONS

All authors contributed to the design, data analysis, and writing of this manuscript.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

Click here for additional data file.