Implementation of Outpatient Infectious Diseases E-Consults at a Safety Net Healthcare System.

Background: Safety net healthcare systems have high patient volumes and significant demands for specialty care including infectious diseases (ID) consultations. Electronic ID consults (E-consults) can lessen this burden by providing an alternative to face-to-face ID referrals and decreasing financial, time, and travel constraints on patients. This system could increase access to ID care for patients in limited-resource settings.
Methods: We described characteristics of all outpatient ID E-consults at Parkland Health in Dallas, Texas, from March 2018 to February 2021. We used modeling to determine which characteristics influenced conversion of E-consults to clinic visits and integrated these data into a predictive model for face-to-face conversion.
Results: For 725 E-consults, common E-consult topics included 118 (16%) latent tuberculosis, 116 (16%) syphilis, and 76 (10%) gastrointestinal infections. Nearly two-thirds of E-consults (456 [63%]) were requested by primary care providers. The majority (78%) were resolved without a face-to-face ID visit. Osteomyelitis, nontuberculous mycobacterial, and gastrointestinal questions frequently required face-to-face visits at rates of 65%, 49%, and 32%, respectively. Our logistic regression model predicted the need for a face-to-face visit with 80% accuracy and an area under the receiver operating characteristic curve of 0.72. Conclusions: An outpatient ID E-consult program at a safety net healthcare system was an effective tool to provide timely input on common ID topics. E-consults were requested by a range of providers, and most were completed without a face-to-face visit. Predictive modeling identified important characteristics of E-consults and predicted conversion to face-to-face visits with reasonable accuracy.

Key Points: Infectious diseases E-consults are a useful tool for improving timely access to specialty care in safety net health systems, and predictive modeling can be used to improve efficiency of outpatient E-consult programs.

Electronic consultations (E-consults) provide access to specialty care through asynchronous telemedicine encounters without the need for an in-person patient visit. Utilizing a shared electronic platform, a specialist reviews patient data provided by the referring provider and makes recommendations. E-consults reduce time to recommendation by enabling referring providers to receive specialist consultation within hours or days instead of the weeks required for referring patients to an in-person appointment with the specialist [1]. E-consults are an appealing medium for the coronavirus disease 2019 (COVID-19) pandemic as they may reduce patient and provider exposure and face-to-face referrals, which in turn decreases wait times for appointments in specialty clinics [1-3].

Safety net healthcare systems provide care for patients regardless of their insurance status and often have substantial patient volumes and wait times. For safety net systems, the combination of scarcity of specialists, limited funding, and patient transportation difficulties provides a favorable environment for outpatient E-consult services [4-6]. An outpatient infectious diseases (ID) E-consult service allows more complex conditions to be managed by primary care providers (PCPs), thereby decreasing distance and transportation barriers and resource utilization [7]. Leveraging E-consults may improve outcomes by increasing the number of patients who receive timely ID specialty care [6].

While the popularity of E-consults increased during the COVID-19 pandemic, E-consults are still a novel tool and a clear understanding of which patient–condition(s)–provider combinations are appropriate for E-consult remains elusive. Referring a patient to an E-consult that proves inadequate could potentially delay treatment and increase effort and costs. We sought to describe the characteristics of outpatient ID E-consults at Parkland Health. Exploring patient and E-consult related factors (consultation topic, consultation text, and referring provider specialty), we labeled E-consult requests that were converted to a face-to-face visit as E-consults failures and attempted to identify conditions that would predict failed E-consult requests.

METHODS

Parkland Health

Parkland Health serves as the safety net healthcare provider in Dallas County, Texas, where according to United States Census data, 18.4% of residents had no health coverage in 2019 [8, 9]. Parkland Health is unique among safety net providers, as it offers emergency and hospital care in its main hospital, outpatient primary and specialty care, and medication dispensing. Parkland operates 30 outpatient clinics throughout the community that complete >1 million visits annually [8, 9]. In 2020, 48% of Parkland Health’s patients had Medicare/Medicaid, 40% were self-pay or charity, and 8% had commercial insurance [8].

E-Consult Program

The Parkland Health Outpatient ID E-consult program was implemented within the electronic health record system, Epic (Epic Systems Corporation, Verona, Wisconsin). Primary care and subspecialty providers could request E-consults using templates with multiple topics: (1) latent tuberculosis infection (LTBI), (2) nontuberculous mycobacterial (NTM) infections, (3) human immunodeficiency virus preexposure prophylaxis (PrEP), and (4) other. These topics were chosen based on extrapolation of previous literature demonstrating effectiveness in answering mycobacterial-related questions and a Parkland Health initiative to increase PrEP access for patients. The templates included topic-specific questions, symptom queries, and a free text option for providers to provide additional information. Within the “Other” template, providers were given choices for the reason for E-consult that included abnormal imaging, abnormal laboratory test, fevers, leukocytosis, abnormal culture results, medication question, vaccine question, and other. Referring providers completed the template and the ID specialists reviewed the information provided and the patient’s chart. Expected response time was <72 hours. Response templates included the specialist’s suggested diagnosis, recommendations, and rationale for the recommendations. The template also included any recommendations to convert the E-consult to a face-to-face visit and information on the approximate time that was required to complete the E-consult.

The ID E-consult service is structured with a single ID specialist or “superuser” answering most E-consults, though other users can answer E-consults. Since its inception, the E-consult program has been funded by Parkland Health, including reimbursement of specialists. Initially, the E-consult program was funded through full-time equivalent support to participating specialties. However, this reimbursement model lacked the flexibility necessary to allow for uninterrupted specialty provider coverage. Currently, specialists are reimbursed at a rate of 1.4 work relative value units per E-consult completed.

Data Collection

We reviewed all completed E-consults from program start (March 2018) through February 2021. We collected patient demographics such as age, sex, race, ethnicity, and payor status. From the referral, we collected the reason for E-consult, symptoms, free text comments, and the specialty of the referring provider. From the response, we collected the visit diagnoses, conversions to face-to-face visits, reported time effort by the ID specialist, recommended laboratory tests and imaging, outcome of the E-consult (eg, diagnostic or treatment recommendations), and free text responses by the ID expert.

E-consults were grouped into topics according to the diagnoses assigned to the encounter by the ID specialist and information provided in the free text of the referral (referral topic). Outcomes were categorized based on the recommendations of the specialist.

Analysis

We compared the number of E-consults converted to face-to-face visits for each referral topic category using χ2 testing for statistical significance. Alpha level was set a priori at .05 and all hypothesis testing was 2-sided. We used Python version 3.9.0 software to perform all statistical analysis.

Modeling

We employed 3 common machine learning algorithms: logistic regression, decision trees, and naive Bayes learners (Scikit-learn version 1.1.1) to determine which characteristics of E-consults were more likely to result in a conversion to a face-to-face visit. We determined useful characteristics through authors’ consensus of the collected data (eg, patient age, race, referring specialty), then converted the data into a machine-usable format. Model creation was performed iteratively. We employed a simple greedy approach to determine predictive attributes (variables that may be useful for a prediction task): creating the model with a single attribute and evaluating the receiver operating characteristic curve (ROC), a method to graphically display the ability of a model to distinguish a binary classifier at varying thresholds after the addition of another attribute. If the ROC did not decrease with the additional attribute, it was considered predictive.

We normalized attributes using one-hot encoding with the exception of patient age. The data were subsequently shifted by the mean to prevent the logistic regression model’s loss function from being overly sensitive to outliers, and divided by the standard deviation (SD) to ensure that data collected on different scales (eg, age vs patient gender) were comparably scaled. Missing data were accounted for by inserting a value of “0” in the data set; we chose this approach as certain datum, such as presence of fever, were always set to 1 or missing. Finally, the order of the data were randomized as the E-consults were initially ordered temporally.

We evaluated model performance using true-positive rate, false-positive rate, accuracy, and the ROC AUC (area under the curve). An ROC AUC of 1 indicates a perfect classifier while an ROC AUC of 0.5 indicates a classifier working no better than a random coin flip. We employed k-fold cross-validation, where k = 10, and computed the classifier’s performance when fit over all data. This 2-pronged approach ensured that the models were capable of appropriately fitting data while decreasing model overfitting. Confidence intervals were generated using 1000 bootstrapped samples. Since E-consults are intended to decrease workload and every E-consult converted to a face-to-face visit generates unnecessary work, we chose a cutoff for the model (70% sensitivity and 75% specificity) that balanced decreasing workload for the specialist conducting the E-consults with reducing unnecessary face-to-face referrals for the ID clinic.

RESULTS

E-Consult Characteristics

During the first 3 years of the outpatient ID E-consult program, 725 E-consults were requested and completed for 400 (55%) female and 325 (45%) male patients (Table 1). The mean patient age was 50 (standard deviation [SD], 15) years. The patient population included 444 (61%) White, 238 (33%) Black, 32 (4%) Asian, 3 (<1%) Native American, 2 (<1%) Pacific Islander, and 6 (1%) patients with unknown race. Nearly half of patients (353) identified as Hispanic/Latino. The majority of patients 392 (54%) were uninsured or funded through charity and 285 patients (39%) had Medicare/Medicaid. The remaining 48 patients (7%) had commercial insurance. The proportion of patients with an ordered ID E-consult who were uninsured or funded through charity was larger than the overall payor mix for the Parkland Health system (54% vs 40%, respectively).

Table 1.

Electronic Consultation Characteristics

Characteristic	No. (%)
Patient characteristics
Sex
Male	325 (45)
Female	400 (55)
Age, y, mean (SD)	50 (15)
Race
White	444 (61)
Black	238 (33)
Asian	32 (4)
Native American	3 (<1)
Pacific Islander	2 (<1)
Unknown	6 (1)
Ethnicity
Hispanic/Latino	353 (49)
Non-Hispanic/Latino	368 (51)
Payor status
Uninsured/charity	392 (54)
Government insurance	285 (39)
Private insurance	48 (7)
E-consult characteristics
Total No. of E-consults	725
Converted face-to-face	156 (22)
Time to completion
<10 min	194 (27)
10–15 min	340 (47)
15–20 min	164 (23)
>20 min	25 (3)
Referring specialty
Primary care	456 (63)
Gastroenterology	55 (8)
Hematology/oncology	36 (5)
Neurology	30 (4)
Plastics	29 (4)
Rheumatology	28 (4)
Dermatology	22 (3)
Otolaryngology	13 (2)
Podiatry	9 (1)
Ophthalmology	9 (1)
Pulmonology	10 (1)
Other	28 (4)
Topic
LTBI	118 (16)
Syphilis	116 (16)
SSTI	45 (6)
Osteomyelitis	34 (5)
NTM	51 (7)
Other respiratory[a]	53 (7)
GI infections[b]	76 (10)
Urinary[c]	41 (6)
Other[d]	191 (26)
Outcomes
Converted face-to-face	132 (18)
Treatment recommended	212 (29)
No further workup/treatment necessary	154 (21)
Additional workup advised	136 (19)
Other	91 (13)

Data are presented as No. (%) unless otherwise indicated.

Abbreviations: E-consult, electronic consultation; GI, gastrointestinal; LTBI, latent tuberculosis infection; NTM, nontuberculous mycobacteria; SD, standard deviation; SSTI, skin and soft tissue infection.

Includes topics labeled “pulmonary tuberculosis,” “positive sputum culture (non-NTM),” and “abnormal chest imaging.”

Includes the topics Helicobacter pylori, parasitic infections, cytomegalovirus, Clostridioides difficile, and other bacterial enteritis.

Includes both urinary tract infection and asymptomatic bacteriuria.

Includes e-consults on human immunodeficiency virus preexposure prophylaxis, nonsyphilis sexually transmitted infections, coronavirus disease 2019, viral hepatitis, vaccine counseling, and all other e-consults that did not fit into the previously defined categories.

PCPs requested most E-consults (n = 456 [63%]) (Table 1). The remaining requests were generated by (sub)specialty services including gastroenterology (55 [8%]), hematology/oncology (36 [5%]), rheumatology (28 [4%]), neurology (30 [4%]), and dermatology (22 [3%]). Twenty-five different specialties and subspecialties requested E-consults. Conversion to face-to-face visits occurred in 156 (22%) of E-consults. The remaining 569 (78%) were resolved without the patient being seen in the ID clinic.

The most common E-consult topics included LTBI treatment (16%), syphilis serology interpretation (16%), and gastrointestinal (GI) infections (10%). A detailed breakdown of all E-consult topics is shown in Supplementary Table 1. Most E-consults (74%) required <15 minutes’ effort. Only 25 (3%) required >20 minutes. E-consult topics were not evenly distributed among requesting specialties (Figure 1). For example, gastroenterology, dermatology, and rheumatology were likely to request E-consults for LTBI. Neurology and ophthalmology frequently asked for assistance interpreting syphilis serologies. All E-consults from podiatry focused on skin and soft tissue infections or osteomyelitis. E-consult topics remained consistent over time with the exception of an increase in E-consults related to COVID-19 throughout the course of the pandemic.

Figure 1.

E-consult topics, by referring specialty. Abbreviations: Hem/Onc, hematology/oncology; NTM, nontuberculous mycobacteria; TB, tuberculosis.

E-Consult Outcomes

E-consult requests generated a variety of outcomes. In 212 (29%) E-consults, the consultant recommended additional treatment. No further workup or treatment was recommended in 154 (21%), and in 136 (19%) additional workup was advised. In 132 (18%) E-consults, the consultant immediately recommended a face-to-face visit based on the initial E-consult request (Table 1), whereas in 24 cases, the conversion was only recommended after additional information had been obtained from the requester, for a total of 156 (22%) converted E-consults.

Some E-consult topics were significantly more likely to result in a conversion to a face-to-face clinic visit (Figure 2). E-consults for osteomyelitis, NTM, and GI infections were converted at the highest rates of 65%, 49%, and 32% respectively. LTBI, syphilis, and urinary questions were converted less frequently at rates of 12%, 8%, and 5% respectively (see also Supplementary Table 1). Using χ2 testing, the likelihood of an E-consult conversion was found to be dependent on the reason for the E-consult (P < .0001) when compared to the overall conversion rate of 22%. There were similar rates of face-to-face conversion between templated (24%) and nontemplated e-consults (21%).

Figure 2.

Number of E-consults converted face to face, by topic. Abbreviations: GI, gastrointestinal; LTBI, latent tuberculosis infection; NTM, nontuberculous mycobacteria; SSTI, skin and soft tissue infection.

Predictive Modeling

Figures 3 demonstrate the ROC AUC curves for all 3 models. Supplementary Table 2 summarizes the accuracy, ROC AUC, and SD of all 3 models when performed over cross-validation and all data, respectively. The logistic regression model demonstrated the best overall performance characteristics with respect to accuracy and average ROC AUC (0.8 and 0.72, respectively), and the confusion matrix for this final logistic regression model can be found in Supplementary Figure 1. The naive Bayes model performed similarly and the decision tree model performed the worst despite having the highest prediction accuracy over all data. This high accuracy is a result of model overfitting.

Figure 3.

Cross-validation receiver operating characteristic curves for the logistic regression model (A), naive Bayes model (B), and decision tree model (C). Abbreviations: AUC, area under the curve; ROC, receiver operating characteristic; SD, standard deviation.

Table 2 shows the top 10 and bottom 10 attributes associated with conversion to a face-to-face consult, including odds ratios and confidence intervals. E-consults unrelated to COVID-19, including lymphadenopathy, “cyst” in the free text, Asian race, and Hispanic/Latino ethnicity were associated with an E-consult being converted to a face-to-face consult. The attributes of urinary topic, vaccine questions, and E-consults related to COVID-19 were the top predictors for an E-consult not requiring conversion to a face-to-face consult.

Table 2.

Predictive Model Attributes: The Top 10 and Bottom 10 Attributes Associated With Conversion of an Electronic Consultation to a Face-to-Face Consultation

Attribute	Odds Ratio	(95% CI)
Top predictors
E-consult unrelated to COVID-19	1.49	(.06–.06)
Lymphadenopathy	1.42	(.14–.2)
“Cyst” included in free text	1.4	(.18–.27)
Race: Asian	1.4	(.17–.16)
Ethnicity: Hispanic/Latino	1.4	(.19–.17)
Topic: Osteomyelitis	1.4	(.17–.2)
Mycobacterium kansasii included in free text	1.39	(.14–.15)
Associated symptoms reported	1.38	(.18–.17)
Topic selected: Other	1.37	(.18–.19)
Reason for E-consult: Medication question	1.35	(.17–.19)
Bottom predictors
Topic: Urinary	0.63	(.18–.14)
Reason for E-consult: Vaccine question	0.66	(.07–.08)
E-consult related to COVID-19	0.67	(.06–.06)
Topic: Other respiratory	0.71	(.26–.23)
Not Hispanic/Latino	0.71	(.17–.19)
Referring specialty: Primary care	0.72	(.12–.1)
No associated symptoms reported	0.72	(.17–.18)
Race: White	0.74	(.19–.19)
Topic: LTBI	0.75	(.24–.23)
Reason for E-consult: Fevers	0.75	(.1–.1)

Abbreviations: CI, confidence interval; COVID-19, coronavirus disease 2019; E-consult, electronic consultation; LTBI, latent tuberculosis infection.

DISCUSSION

In this retrospective analysis of an outpatient ID E-consult program at a safety net health system, differences in the rates of conversion of E-consults to face-to-face visit were seen among E-consult referral diagnoses, indicating that certain conditions may be more appropriate for E-consults. We analyzed E-consult characteristics by creating a logistic regression model for a reasonably accurate prediction of conversion to face-to-face visit.

Characteristics of outpatient ID E-consult programs have been previously described at a few institutions in the United States [1, 10]. Wood et al described that fevers and musculoskeletal and skin infections were converted frequently to face-to-face evaluations [1]. Our study is the first to evaluate differences in conversion of E-consults to face-to-face visits by diagnosis and to predict conversion rates. Our experience suggests that osteomyelitis, NTM, and GI infections are less appropriate for E-consults as consultants frequently convert them to face-to-face visits. We hypothesize this may be a result of the need for in-person examination or the need for further imaging or laboratory investigations. We also demonstrate that a simple logistic regression model can accurately predict conversion rates based on the characteristics included in an E-consult referral.

Reasons for the E-consult factored significantly into conversions. Complex conditions like active TB or NTM infections predicted face-to-face conversion. Referrals for reasons where the ID specialists had less of an effect on morbidity and mortality (COVID-19, serology interpretation, LTBI) were less likely to be converted. The use of templates for E-consult topics had no appreciable effect on face-to-face conversion. The PrEP template was included as part of an initiative to increase PrEP access at Parkland Health, but was not widely utilized by referring providers throughout the study period.

Murthy et al described the use of an ID E-consult service in Ottawa, Canada. Despite differences in the healthcare systems, their E-consult program received similar proportions of consults for LTBI (14% vs 16%) and skin and soft tissue infections (7% vs 6%). They also had a similar rate of E-consult conversions to face-to-face referrals (25% vs 22%). The Murthy et al study also showed high rates of E-consults for Lyme disease and parasitic infections that were not present at our institution, indicating that geographic location and local infection patterns may influence a program’s E-consult usage [11].

Barnett et al described a large multispecialty E-consult program at a large safety net health system in Los Angeles County that improved access to specialist care for underserved populations [10]. Our study is the first to describe and analyze ID E-consult usage at a safety net institution in the United States. Unlike an ID E-consult program at an academic medical center, Parkland’s E-consult program had higher proportions of requests relating to LTBI, syphilis serologies, and musculoskeletal infections (9.4% vs 16%, 9.6% vs 16%, and 6.5% vs 11%, respectively), but had a similar proportion of E-consults converted to a face-to-face visit (22% vs 25%) [1]. This single comparison suggests that there may be variations in the usage of E-consult programs at different institutions, possibly due to differences in the patient populations.

Our analysis also highlights that the type of E-consult requests tend to be similar among referring providers from the same medical specialty. For instance, rheumatologists were more likely to ask about LTBI in anticipation of prescribing immunosuppression, and podiatrists are more likely to ask about skin and soft tissue infections and osteomyelitis. This finding highlights an opportunity to design continuing medical education to reinforce knowledge around common specialty-specific questions.

Our predictive model may be a useful tool to triage E-consults for appropriateness and automatically direct some to face-to-face encounters. Alerting the ID consultant that a specific request may be more likely to require a face-to-face visit may also reduce time spent on an E-consult. Alternatively, referring providers may be alerted by models that a face-to-face encounter could be an appropriate next action. Healthcare systems could use their own referral characteristics to create a model specific to their institution. However, further studies are needed to optimize these models for predictive accuracy and to explore their usefulness in clinical care.

Within our own institution, we plan to incorporate the predictive model to assist referring providers and ID specialists to efficiently triage E-consults. In the future, our hope is to also more fully leverage a natural language processing platform to analyze the text of a referral and provide clinical decision support to the end user to direct them to the most appropriate consult type.

While this manuscript on our infectious disease E-consult experience enhances understanding of the optimal use of E-consult programs, there is still a lack of large multicenter studies to explore their use and utility. Conducting such studies may be limited by institutional variations in E-consult systems and patient populations served. Further work is needed to specifically address whether ID E-consults save time and resources or decrease clinical burdens and obstacles for patients to specialty care.

There are several inherent limits to our study. First, we used a retrospective study design and collected data by chart review, which may have resulted in instances of human error. Second, our study was conducted at a single center and may not be applicable at other institutions. Third, our model was limited by our sample size. Furthermore, we did not evaluate patient outcomes and thus cannot comment on the appropriateness of the model to delineate the appropriate consult type based on this metric. Finally, we note that all models had decreased accuracy and ROC AUC during cross-validation. We hypothesize that this may be a result of the class imbalance within the data set (ie, the vast majority of the E-consults were not converted to a face-to-face meeting).

CONCLUSIONS

Our results provided a detailed look at how ID E-consults were used in a safety net healthcare system over a 3-year period including which specialties were using the program and the most common topics asked. We were also able to identify which topics were most appropriate for E-consult and use modeling to predict conversion to face-to face-visits based on referral characteristics.

Click here for additional data file.