Telehealth Made EASY: Understanding Provider Perceptions of Telehealth Appropriateness in Outpatient Rheumatology Encounters.

OBJECTIVE: The purpose of this study was to evaluate a novel scoring system, the Encounter Appropriateness Score for You (EASY), to assess provider perceptions of telehealth appropriateness in rheumatology encounters.
METHODS: The EASY scoring system prompts providers to rate their own encounters as follows: in-person or telehealth acceptable, EASY = 1; in-person preferred, EASY = 2; or telehealth preferred, EASY = 3. Assessment of the EASY scoring system occurred at a single academic institution from January 1, 2021, to August 31, 2021. Data were collected in three rounds: 1) initial survey (31 providers) assessing EASY responsiveness to five hypothetical scenarios, 2) follow-up survey (34 providers) exploring EASY responsiveness to 11 scenario modifications, and 3) assessment of EASYs documented in clinic care.
RESULTS: The initial and follow-up surveys demonstrated responsiveness of EASYs to different clinical and nonclinical factors. For instance, less than 20% of providers accepted telehealth when starting a biologic for active rheumatoid arthritis, although more than 35% accepted telehealth in the same scenario if the patient lived far away or was well known to the provider. Regarding EASY documentation, 27 providers provided EASYs for 12,381 encounters. According to these scores, telehealth was acceptable or preferred for 29.7% of all encounters, including 21.4% of in-person encounters. Conversely, 24.4% of telehealth encounters were scored as in-person preferred.
CONCLUSION: EASY is simple, understandable, and responsive to changes in the clinical scenario. We have successfully accumulated 12,381 EASYs that can be studied in future work to better understand telehealth utility and optimize telehealth triage.

INTRODUCTION

The COVID‐19 pandemic drastically altered the way rheumatologists provide clinical care, forcing the rapid adoption of telehealth for patients with rheumatoid arthritis (RA) and other rheumatologic disorders. More than 95% of health centers across the United States offered telehealth services during the COVID‐19 pandemic (1), and reports from rheumatology practices across the globe indicate that between 40% and 90% of rheumatology encounters occurred via telehealth during the initial COVID‐19 transition (2, 3, 4, 5, 6, 7, 8).

Telehealth is known to be safe and effective for patients with RA with low disease activity (9, 10, 11, 12). However, since the onset of COVID‐19, patients with the full range of rheumatologic diseases and disease activity levels have been seen by telehealth with unknown consequences. At present, there is no established method of identifying patients who are appropriate for telehealth care. Survey data from rheumatology providers and patients suggest that telehealth is an accessible and acceptable option for many patients, but it can be difficult to distinguish noninflammatory mimics from active inflammatory arthritis during a telehealth encounter because of the inability to perform a physical examination (13, 14, 15).

As concerns regarding the COVID‐19 omicron variant (B.1.1.529) and seasonal resurgence of the virus loom, there is an urgent need to develop actionable risk prediction tools that can distinguish patients who are appropriate for telehealth from those who need an in‐person encounter for adequate treatment. Therefore, we created the Encounter Appropriateness Score for You (EASY) to rate provider perceptions of telehealth appropriateness in individual rheumatology encounters, enabling us to identify factors that predict the utility of future telehealth encounters and to monitor our telehealth practice over time. Ultimately, our goal is to use EASY to optimize telehealth care for rheumatology patients and to develop a sustainable telehealth model of care for rheumatology providers and staff. In this study, we describe the development and initial findings of EASY.

MATERIALS AND METHODS

Development of EASY

We prioritized the following factors when creating the EASY rating system: 1) efficient documentation, 2) applicable to both in‐person and virtual encounters, 3) clear differentiation of patients who need in‐person care from those who could be seen by telehealth, and 4) ease of data extraction. These priorities were chosen to facilitate consistent collection of provider perceptions of telehealth appropriateness in all outpatient rheumatology encounters without overburdening providers. In addition, EASYs needed to be discrete variables that were easy to extract to enable inferential and predictive analyses of telehealth appropriateness. A literature review confirmed that no validated score was available that fit these priorities.

The EASY prompt and response options (Figure 1) were created as a smart phrase for the Epic electronic health record (EHR) that providers could enter into their clinic note templates, ensuring consistent documentation of EASY ratings for all in‐person and telehealth encounters (video or phone). In addition, the response options were converted into a smart data element, which linked the responses to other encounter data stored in our center's electronic data warehouse. This allowed our informatics team to retrieve and analyze EASYs without manual chart review.

Figure 1

Encounter Appropriateness Score for You (EASY) prompt and response options.

Three members of our research team (DLL, JD, and MEBC) piloted EASY in December 2020. EASY required less than 3 seconds to document in each encounter based on informal feedback. We then introduced EASY during a faculty meeting on January 5, 2021, during which we described its purpose, explained the response options, and gave instructions on how to incorporate it into clinical documentation. We then invited all MD and DO faculty, advanced practice providers (APPs), and fellows in our division to begin documenting the score in every clinical encounter.

Assessment of EASY

The EASY rating system is intended to reflect providers’ personal perceptions of telehealth appropriateness. There is no gold standard for this rating; thus, our assessment of EASY is focused on 1) variability in provider perception of telehealth appropriateness in similar clinical scenarios and 2) the reasons why providers may have different preferences for recommending an in‐person or telehealth encounter. During our assessment of EASY, we collected data in three rounds.

Initial survey

During a virtual divisional grand rounds on January 26, 2021, we asked rheumatology providers (n = 31) to assign EASY ratings to five hypothetical clinical scenarios using Poll Everywhere (Table 1). Each scenario included a patient with RA seen during a follow‐up encounter. Although the EASY rating system is intended to assess provider perceptions of telehealth appropriateness in all rheumatology encounters, the scenarios in the initial survey focused on a single rheumatic disease to understand responsiveness of EASY to clinical factors beyond the underlying rheumatic diagnosis. As such, the scenarios varied by degree of clinical complexity and included additional clinical information, such as comorbid osteoarthritis and fibromyalgia, RA disease activity, and treatment with conventional synthetic disease‐modifying antirheumatic drugs (csDMARDs) or biologic disease‐modifying antirheumatic drugs (DMARDs). The scenarios also included a patient‐reported Routine Assessment of Patient Index Data  3 (RAPID3) score, which is routinely collected during typical clinic visits at our institution. RAPID3 scores were categorized as follows: less than or equal to 1 = remission, 1.1 to 2 = low disease activity, 2.1 to 4 = moderate disease activity, and greater than 4 = high disease activity for patients with inflammatory arthritis. Meeting attendees also provided verbal feedback about the EASY rating system. The meeting was recorded, and investigators took notes on key feedback.

Table 1

Clinical scenarios for the initial survey (A‐E) and modifications to scenarios C and D for the follow‐up survey (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11)

	Description
Scenario for initial survey
A. Stable RA	50‐year‐old patient with RA on MTX 15 mg weekly, RAPID3 score <1, who presents for follow‐up with stable symptoms and needs medication toxicity laboratory monitoring
B. Severe RA flare, comorbid OA and FM	50‐year‐old patient with RA on MTX 15 mg weekly, also with FM and OA, who presents for follow‐up with worsening polyarticular joint pain, RAPID3 score increased from 2 to 6
C. Mild RA flare, csDMARD change	50‐year‐old patient with RA on MTX 15 mg weekly, no major comorbidities, who presents for follow‐up with mild increase in hand symptoms that feel similar to prior RA activity, RAPID3 score increased from 0.5 to 1.5; you are considering maximizing MTX or adding HCQ or SSZ
D. Severe RA flare, biologic initiation	50‐year‐old patient with RA on MTX 25 mg weekly, no major comorbidities, who presents for follow‐up with substantial increase in hand symptoms that feel similar to prior RA activity, RAPID3 score increased from 0.5 to 4; you are considering starting a biologic DMARD
E. Continued RA flare, biologic switch	50‐year‐old patient with RA on MTX 25 mg weekly and started a biologic DMARD 3 months ago at an in‐person visit for active RA presents for follow‐up with no improvement in symptoms, RAPID3 score still 5; you are considering switching biologics
Modifications to scenarios C and D for follow‐up survey
1. Distance <30 minutes	The patient lives within a 30‐minute drive from the clinic.
2. Distance >2 hours	The patient lives more than 2 hours away from the clinic.
3. Patient mobility issues	The patient has mobility issues that make transportation to the clinic difficult.
4. Patient well known to provider	You have been managing the patient for several years.
5. Patient not well known to provider	You have seen the patient once in the past.
6. Previous visit by telehealth	The patient's last appointment was a telehealth encounter.
7. Previous visit >6 months ago	The patient's last appointment was more than 6 months ago.
8. Previous visit with different provider	The patient's last appointment was with a different provider (such as an MD/DO or APP with whom you comanage patients).
9. Family support needed	The patient lives alone but has a family member who accompanies them to their clinic visits, provides supportive medical history, and assists the patient with decision‐making.
10. Interpreter needed	The patient requires an interpreter.
11. Medication nonadherence	The patient has a history of medication nonadherence.

Abbreviations: APP, advanced practice provider; csDMARD, conventional synthetic disease‐modifying antirheumatic drug; DMARD, disease‐modifying antirheumatic drug; FM, fibromyalgia; HCQ, hydroxychloroquine; MTX, methotrexate; OA, osteoarthritis; RA, rheumatoid arthritis; RAPID3, Routine Assessment of Patient Index Data  3; SSZ, sulfasalazine.

Follow‐up survey

On May 11, 2021, a Qualtrics electronic survey was sent to MD and DO faculty, APPs, and fellows (n = 34) in our practice to assess additional clinical factors that may influence EASY. This survey repeated scenarios C and D from the initial survey and then introduced 11 modifying factors to these scenarios (Table 1). Modifying factors included patient and provider factors, such as patients’ travel distance to the clinic, provider familiarity with the patient, and time since last appointment. Participants were asked to provide EASY ratings in response to each modifying factor. Participants were also asked to “provide any additional feedback about the EASY score and/or the ease of incorporating telehealth encounters into your practice” (free text).

EASY implementation

Data on the number and distribution of EASYs documented by each clinician over time from January 1, 2021, to August 31, 2021, were tabulated using EHR data. We excluded scores from providers with less than 20 EASY ratings to avoid including scores that were accidentally recorded by providers outside our division or by providers with minimal experience using EASY.

Statistical analysis

Descriptive statistics were used to summarize survey findings, EASYs for all collected data, the number of EASYs documented over time, and variability in EASY ratings between providers from January 1, 2021, to August 31, 2021.

Institutional review board statement

The study was designated as exempt human subjects research and was approved by the Institutional Review Board of Duke University Hospital (Pro00105997).

RESULTS

Initial survey results

The initial EASY survey was administered to 31 grand rounds attendees, including 12 rheumatology attending providers, 10 APPs, six rheumatology fellows, one internal medicine resident, and two medical students. Response rates to each survey question ranged from 90% to 100%. At the time of the survey, 17 of 31 (55%) reported already using EASY in their notes. Providers were asked to estimate the proportion of their own telehealth encounters over the last month that would have been more appropriate as in‐person encounters: 9 of 29 (31%) estimated 31% to 50% of encounters, 16 of 29 (55%) estimated 11% to 30% of encounters, and 4 of 29 (14%) estimated 0% to 10% of encounters.

The distribution of EASYs for the five clinical scenarios in the initial survey are shown in Figure 2. Overall, the majority of providers indicated that telehealth was acceptable or preferred for scenarios A and C. Scenario A (stable RA) was the only scenario in which any providers indicated that telehealth would be preferred over in‐person care (43% of responses), with the remainder indicating either in‐person or telehealth care was acceptable (57%); no providers preferred in‐person care for scenario A. In scenario C (mild RA flare, csDMARD change), 67% indicated either in‐person or telehealth care was acceptable and 33% preferred in‐person care. In contrast, more than 93% of providers preferred in‐person care for scenario B, in which a patient with RA with comorbid osteoarthritis and fibromyalgia developed a severe flare of joint pain. Similarly, the majority of providers preferred in‐person care for scenarios D (82%) and E (86%), in which a patient required either a new biologic DMARD or a switch in biologic DMARD for high disease activity.

Figure 2

Provider Encounter Appropriateness Score for You (EASY) rating responses to the five clinical scenarios in the initial survey. csDMARD, conventional synthetic disease‐modifying antirheumatic drug; FM, fibromyalgia; OA, osteoarthritis; RA, rheumatoid arthritis.

Follow‐up survey results

The follow‐up survey was sent to 34 providers, and 18 (53%) responded. Scenarios C and D were chosen for the follow‐up survey because they contained the most variability in provider responses on the initial survey, suggesting they were the most likely scenarios to change in response to different modifying factors. The proportion of follow‐up survey responses indicating telehealth was acceptable or preferred (EASY = 1 or 3) in scenarios C and D with and without modifying factors is shown in Table 2. Compared with the initial survey, fewer providers in the follow‐up survey indicated acceptance or preference for telehealth (EASY = 1 or 3) for scenario C (67% vs 44%). There was little difference between the initial survey and the follow‐up survey results for scenario D (18% vs 12%).

Table 2

Follow‐up survey results showing the impact of clinical context on the perception of telehealth utility

	Percentage of EASY ratings indicating either acceptance of telehealth (EASY = 1) or preference for telehealth (EASY = 3)
Modification	Scenario C: mild RA flare, csDMARD change	Scenario D: severe RA flare, biologic initiation
Original scenario	44%	12%
Patient mobility issues	89%	41%
Distance >2 hours	82%	36%
Patient well known to provider	71%	36%
Distance <30 minutes	44%	12%
Previous visit with different provider	41%	24%
Medication nonadherence	28%	18%
Family support needed	33%	6%
Previous visit >6 months ago	17%	0%
Previous visit by telehealth	12%	0%
Patient not well known to provider	11%	0%
Interpreter needed	6%	0%

Note: The results are presented as the percentage of providers (n = 18) who responded either telehealth acceptable (EASY = 1) or preferred (EASY = 3) after different modifications were applied to scenarios C and D from the initial survey.

Abbreviations: csDMARD, conventional synthetic disease‐modifying antirheumatic drug; EASY, Encounter Appropriateness Score for You; RA, rheumatoid arthritis.

The top three modifying factors that increased the proportion of respondents indicating acceptance or preference for telehealth for both scenarios C and D were mobility issues that make transportation to the clinic difficult, distance from the clinic greater than 2 hours, and patients well known to the provider. The four modifying factors that most decreased the proportion of respondents indicating acceptance or preference for telehealth (meaning providers favored in‐person encounters, EASY = 2) for both scenarios C and D were patients requiring an interpreter, patients who had only been seen once by the provider, patients seen by telehealth during their last appointment, and patients whose last appointment was more than 6 months ago. The modifying factors that minimally influenced provider EASYs were distance from the clinic less than 30 minutes, last appointment with another rheumatology provider from the same practice, patients with a history of medication nonadherence, and patients with a family member who provides supportive history and assists with patient decision‐making.

Provider feedback from the initial and follow‐up surveys

Verbal feedback from participants after the initial survey revealed that providers understood the meaning of the prompt and response options, and there were no requests to modify the EASY rating system. The only point of clarification was the portion of the EASY prompt stating “irrespective of the pandemic.” This indicates that EASY ratings should be based solely on the clinical scenario and should not be influenced by the current prevalence or severity of COVID‐19 or the influence of the pandemic on the patient's ability to attend a visit in‐person. The remainder of the verbal feedback from participants focused on additional factors that might influence EASY ratings, and this feedback was used to generate the scenario modifications for the follow‐up survey. Factors included patients’ travel time to the clinic, provider history with the patient, patients’ adherence to medication, length of time since last appointment, and the amount of support patients need during the visit (interpreters, family members in attendance, etc). In the free‐text feedback section of the follow‐up survey, no respondents commented on the EASY rating system or suggested any changes.

EASY implementation results

From January 1, 2021, to August 31, 2021, 12,394 outpatient encounters received an EASY rating from 30 providers. Three providers documented fewer than 20 EASY ratings (13 ratings in total); these ratings were excluded. After these exclusions, the final analysis (Table 3) included 12,381 outpatient encounter EASY ratings from 27 rheumatology providers (17 MD and DO attendings, 8 APPs, and 2 fellows). These 27 providers composed 77.1% of the 35 providers invited to document EASY.

Table 3

Provider EASY ratings for in‐person and telehealth visits (N = 12,381)

EASY rating	Total visits (N = 12,381), n (%)	In‐person visits (n = 10,460), n (%)	Telehealth visits
			Total telehealth (n = 1921), n (%)	Video (n = 1406), n (%)	Phone (n = 515), n (%)
In‐person preferred	8697 (70.2%)	8229 (78.7%)	468 (24.4%)	281 (20.0%)	187 (36.3%)
Either in‐person or telehealth acceptable	3286 (26.5%)	2152 (20.6%)	1134 (59.0%)	861 (61.2%)	273 (53.0%)
Telehealth preferred	398 (3.2%)	79 (0.8%)	319 (16.6%)	264 (18.8%)	55 (10.7%)

Abbreviation: EASY, Encounter Appropriateness Score for You.

During the study period, EASY ratings were documented for 10,460 in‐person and 1,921 telehealth encounters (1,406 video and 515 phone). Of all encounters (N = 12,381), providers rated 70.2% as in‐person preferred (EASY = 2), 26.5% as either in‐person or telehealth acceptable (EASY = 1), and 3.2% as telehealth preferred (EASY = 3). Providers scored 21.4% of in‐person encounters as telehealth acceptable or preferred. Conversely, almost a quarter (24.4%) of telehealth encounters were scored by the provider as in‐person preferred. This discordance between encounter type and encounter appropriateness was more pronounced for phone (36.3% in‐person preferred) than video (20.0% in‐person preferred) encounters. We also noted variability in EASY score distribution between different rheumatology providers (Figure 3). Provider preference for in‐person encounters (EASY = 2) ranged from 25.6% to 97.9% of encounters, with a median of 69.4% and interquartile range of 53.6% to 82.3%.

Figure 3

Encounter Appropriateness Score for You (EASY) score distribution by provider.

DISCUSSION

In this study, we present the development, implementation, and results of EASY to assess provider perceptions of telehealth appropriateness in individual rheumatology encounters. Our survey results demonstrate that rheumatology providers understand EASY and can apply the score to a variety of clinical scenarios. Furthermore, EASY is responsive to modifying factors that affect provider perceptions of telehealth care. The simplicity and efficiency of the EASY scoring system has enabled us to successfully recruit more than 75% of providers in our practice to document EASY in 12,381 encounters over a period of 8 months, thus providing insights into provider perceptions of telehealth appropriateness across the entire spectrum of rheumatologic conditions and disease severity in our practice. To our knowledge, EASY is the first scoring system designed to evaluate the appropriateness of telehealth care at the level of each individual patient encounter, and we plan to use these data to develop a sustainable telehealth care model for rheumatology patients and providers.

The initial and follow‐up provider surveys provide a preliminary guide to understanding the variability in provider acceptance of telehealth. In some scenarios, the clinical situation is the primary driver of telehealth acceptance by providers. For example, the majority of providers preferred in‐person care for scenario D (severe RA flare, biologic DMARD initiation) on the initial survey, and this scenario varied less than scenario C (mild RA flare, csDMARD change) in response to various modifiers in the follow‐up survey. In contrast, some clinical situations are more ambiguous, leaving room for provider preference and nonclinical factors to influence telehealth acceptance. For instance, there was substantial disagreement about telehealth appropriateness for scenario C (mild flare, csDMARD change), with 66% accepting telehealth on the initial survey and 44% on the follow‐up survey. Furthermore, this scenario was highly influenced by nonclinical scenario modifiers, with telehealth acceptance increasing to 89% if the patient had mobility issues and reducing to 6% if an interpreter were needed. We suspect that the scenarios and modifiers included in our surveys represent some but not all clinical and nonclinical factors that influence telehealth acceptance, and EASY will enable us to further explore these factors.

Previous survey data collected from rheumatology providers with telehealth experience from the Veterans Health Administration estimated that 20% to 29% of patients with certain rheumatologic diagnoses (gout, pseudogout, RA, fibromyalgia, osteoarthritis) could be appropriately managed via telehealth (8). In our study, we observed that telehealth was an acceptable or preferred option for almost one third (29.7%) of all outpatient rheumatology encounters. Furthermore, providers indicated that telehealth would have been an acceptable or preferred option for 21.4% of patients seen in‐person during the study period. These results suggest that telehealth has gained rapid acceptance among rheumatology providers during the COVID‐19 pandemic and challenge us to consider ways to optimize appropriate use of telehealth care through the next waves of the pandemic and beyond.

Although telehealth has been widely implemented by rheumatology practices during the COVID‐19 pandemic, there is still a concern that telehealth care may be inappropriate for some rheumatology patients. In a prepandemic telehealth study of 176 rheumatology patients in rural New Hampshire and Vermont, the three participating rheumatology providers noted that 19% of patients seen during telehealth encounters were inappropriate for telehealth care (16). We observed similar results in our study; providers indicated that 24.4% of telehealth encounters would have been more appropriate as in‐person encounters (EASY = 2). This discordance suggests that current triage processes are not adequate for differentiating patients appropriate for telehealth care from those who need an in‐person encounter for optimal care.

The present findings are subject to several limitations. EASY was developed within one health system; additional work is needed to test the score in other settings to increase generalizability. Furthermore, the scenarios presented in our surveys focused on patients with RA in a limited number of situations. Although additional surveys addressing more diseases and scenarios might be useful, the intention of our surveys was not to develop an exhaustive list of all factors that influence telehealth appropriateness but rather to demonstrate provider understanding of the scoring system and document responsiveness of the score to clinical and nonclinical factors. One set of factors that might be helpful to collect in future studies is provider characteristics because provider characteristics (eg, time in practice, familiarity with technology, clinician resilience, preference for face‐to‐face encounters, demographics) may influence EASY ratings (14). This study describes the initial development of EASY and, by design, does not include a full evaluation of reliability and validity of this measure. Consistent with standard measurement development methods shared among multiple disciplines, the methods used in this study are important foundational methods derived from validity theory (17, 18, 19). Key among these steps are clarifying the use case for EASY, describing initial evidence for content validity (agreement on the applicability of the measure within its context of use, common agreement on the usefulness of response choices, and common understanding of the question being asked and response choices), and identifying potential threats to validity, including variation among providers in how they choose a response to EASY. Nonetheless, additional studies should be conducted to evaluate intrarater and interrater reliability and validity in rheumatology applications. Finally, it is difficult to separate provider perceptions of telehealth appropriateness from the influence of the pandemic, such as vaccination status, mask wearing, and other factors related to COVID‐19. We attempted to account for this in our EASY prompt by stating “irrespective of the pandemic”; however, reassessing EASY ratings post pandemic would help further clarify the score's utility and meaning.

Our study opens the door for numerous additional lines of investigation. Now that we have EASY ratings for the entire spectrum of rheumatology patients from our practice, we are exploring numerous other clinical factors that influence telehealth appropriateness, such as diagnosis, disease activity, medications, etc. In addition, we are working to incorporate these clinical factors into a predictive model to improve our current telehealth triage practices, thus reducing the 20% to 25% discordance rate between the telehealth appropriateness score and the actual encounter modality observed in our initial data. The variability in EASY ratings between providers suggests that any use of these scores with a predictive model or other work must recognize and account for provider preference. Finally, future work is needed to understand how provider EASY ratings compare with patients’ perceptions of telehealth appropriateness.

In conclusion, we developed and evaluated EASY to investigate provider perceptions of telehealth appropriateness. EASY is simple, understandable, and responsive to changes in clinical and nonclinical factors. We have successfully implemented this score in our practice and accumulated 12,381 EASY ratings that can be studied in future work to better understand telehealth utility and optimize our telehealth triage practices.

AUTHOR CONTRIBUTIONS

All authors were involved in drafting the article or revising it critically for important intellectual content, and all authors approved the final version to be published. Dr. Smith had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. This research sponsored by Pfizer grant #2933673 was conducted by independent academic researchers at Duke University Hospital, and the sponsor had no influence on the project design, data analysis, manuscript creation, or resultant publication.

Study conception and design

Smith, Coles, Howe, Overton, Economou‐Zavlanos, Solomon, Zhao, Adagarla, Doss, Henao, Clowse, Leverenz.

Acquisition of data

Smith, Coles, Howe, Overton, Economou‐Zavlanos, Solomon, Zhao, Adagarla, Doss, Henao, Clowse, Leverenz.

Analysis and interpretation of data

Smith, Coles, Howe, Overton, Economou‐Zavlanos, Solomon, Zhao, Adagarla, Doss, Henao, Clowse, Leverenz.

Disclosure Form:

Click here for additional data file.