Transitioning to telehealth? A guide to evaluating outcomes.

Objectives: Telehealth use has surged since the onset of the coronavirus disease 2019 (COVID-19) pandemic, but the evaluation of telehealth outcomes and performance has not necessarily matched the pace of its uptake. In this article we aim to guide the design of a telehealth evaluation system encompassing all four domains of the outcome measurement framework developed by the National Quality Forum (NQF) - access to care, cost, experience, and effectiveness. We aim to achieve this through proposing survey items that can be distributed to patients or clinicians as a questionnaire and providing suggestions on areas of focus for evaluation studies.
Methods: Using PubMed and Google Scholar search engines, we performed a literature review of articles related to the evaluation of telehealth outcomes that were published in English since 2000.
Results: We found existing survey tools to assist the development of an evaluation questionnaire, and categorized items into the four NQF outcome domains. For each outcome domain, we also summarize existing work on evaluation and make recommendations on areas for future assessment. In particular, we found that telehealth accessibility and accommodations have been historically under-studied and provide tools to address this. Conclusions: Evaluating telehealth outcomes is critical to ensure efficient and high-quality care delivery, and we believe establishing an evaluation system will help practices assess and improve their telehealth systems as well as their ability to use telehealth to respond to the diverse needs of patients. Public Interest Summary: Since the start of the coronavirus disease 2019 (COVID-19) pandemic, telehealth use has been on the rise. Evaluating outcomes related to telehealth is critically important, but given the urgency of telehealth uptake, many health systems and practices may not yet have evaluation systems in place. This article guides the design of a telehealth evaluation system by proposing several validated and novel survey questions that can be used as part of a patient or clinician questionnaire and suggesting important measures of outcome for evaluation studies to assess across the four domains of telehealth quality as outlined by the National Quality Forum (NQF) - access to care, cost, experience, and effectiveness. We present tools to reach priority populations who often lack access to remote care, including older adults, underrepresented minorities, and people with disabilities.

Introduction

Telehealth uptake has been rapid since the start of the coronavirus disease 2019 (COVID-19) pandemic; an analysis of 31 billion private health care claim records showed that telehealth insurance claim lines increased from 0.24% to 13% of all claims lines between January and April 2020, representing more than a 54-fold increase [1]. Evaluating outcomes related to telehealth is critically important, but given the urgency of telehealth uptake, many health systems and practices may not yet have evaluation systems in place to measure quality outcomes and help guide future practice decision making.

In a 2016–2017 initiative, the national quality forum (NQF) created a framework to guide the measurement of telehealth quality which outlined four domains of outcomes for evaluation (Fig. 1 ): access to care, cost, experience, and effectiveness [2]. The effectiveness, experience, and cost of telehealth have been extensively assessed (see Supplementary Table 1). However, access to telehealth care is understudied. Patients who are Black, Hispanic, older, and less educated face barriers to accessing telehealth [3], 52 million Americans have limited digital literacy, and over a third of US households headed by a person over 65 years of age do not have a laptop [4]. This raises concerns that telehealth may only reach White, younger, and affluent Americans.

Fig. 1

The four NQF domains and corresponding subdomains.

In this article we demonstrate how to design a telehealth evaluation system, propose several validated and novel survey questions that can be used as part of a patient or clinician questionnaire, and discuss potential study designs that address the four NQF domains of telehealth quality. We include tools that help ascertain whether a medical practice is reaching priority populations that often lack access to remote care, including older adults, underrepresented minorities, and people with disabilities. According to telehealth consultants, doctors still lack “clear criteria about who should be seen, under what circumstances and for which conditions” [5]. A telehealth evaluation system can also help health systems make decisions about when to deploy telehealth or in-person care. By assessing telehealth outcomes, practices can expect to gain efficiencies, improve reach, patient and clinician experience, and be better equipped to respond to the health needs of patients during the pandemic and beyond.

Methods

To summarize survey questions and other studies performed on telehealth evaluation, we performed a literature review using the PubMed and Google Scholar search engines and key search terms “telehealth”, “evaluation”, “outcome”, and “measure”. We selected sources published in English within the last 20 years. Based on our findings, we list validated questions that practices can develop into an evaluation questionnaire to be distributed to patients and physicians within their telehealth program. Drawing on our experience as physician-scientists in digital health (EG, KR) and leading innovations in telehealth emergency care and research (KR), we also suggest novel questions and areas of focus for future studies, such as longitudinal studies, that can be conducted to further evaluate outcomes.

Results

We found that published studies frequently use surveys when assessing telehealth outcomes. Surveys can be quickly designed and deployed, making this evaluation method feasible for most practices. Questionnaires with established validity include the telehealth usability questionnaire (TUQ) [6] and the telemedicine satisfaction questionnaire (TSQ) [7]. Drawbacks of surveys include that they are often provided to patients who have successfully accessed telehealth, and therefore do not provide information about the barriers some patients may experience while attempting to access remote care.

In Table 1 , we summarize survey questions and group them into the four aforementioned NQF outcome domains – access to care, cost, experience, and effectiveness. Because it is important to assess the experience of telehealth for both patients and clinicians, we separately list items that can be adopted for patient and clinician questionnaires. Statements followed by a citation were adapted from published surveys with established validity and employ a Likert Scale (i.e. a 5-point scale ranging from “strongly agree” to “strongly disagree”). Items with an asterisk (*) are novel and are suggested by the authors.

Table 1

Sample Questionnaire Items for Patients and Clinicians.

Outcome	Description	Target	Questionnaire Items
Access to care	Affordability, availability, accessibility, accommodation, and acceptability5	Patients Clinicians	• Telehealth improves my access to healthcare services [6]
			• Telehealth provides for my healthcare needs [6]
			• Telehealth is an acceptable way to receive healthcare services [6]
			• I could easily talk to the clinician using the telehealth system [7]
			• I could hear the clinician clearly using the telehealth system [7]
			• Using the telehealth system, I could see the clinician as well as if we met in person [7]
			• I have reliable access to telehealth*
			• All my patients are able to access telehealth services equally*
			• I spend excessive time on my telehealth calls orienting patients to the telehealth technology*
			• My patients with poor technological literacy receive less optimal care with telehealth than others with higher digital literacy*
			• My older adult patients with disabilities (visual/hearing/mobility/cognitive impairments) have their needs met using telehealth*
			• Non-English speaking patients can communicate with me via telehealth*
Cost	Decreased travel expenditure, equipment and facilities, telecommunication fees, training, wages, potential decreased hospitalizations	Patients Clinicians	• Telehealth saves me time traveling to a hospital or specialist clinic [6]
			• The service fee for telemedicine is reasonable*
			• My internet access is affordable*
			• The additional equipment I purchased to use telehealth was affordable*
			• Telehealth visits take less time than in-person visits*
			• Telehealth led to reduced hospitalizations in my patients*
			• I was able to prevent hospital readmissions in my patients using telehealth*
			• I was able to prevent emergency department visits by using telehealth*
			• The cost of providing telehealth services is acceptable*
			• Equipment installation and maintenance
			• Workforce training hours
			• Communication fees (internet/phone/data transmission)
Experience	Ergonomics, technical quality, and patient-clinician interaction	Patients & clinicians	• It was easy to learn to use this system [6]
			• It was simple to use this system [6]
			• I like using the system [6]
			• I felt comfortable communicating with my patient/clinician [7]
			• The lack of direct physical contact with the patient/clinician was acceptable [8]
			• The technical quality (e.g. video/audio quality) was acceptable*
			• I felt that I was able to have sufficient emotional connection with my patient/clinician*
Effectiveness	System, clinical, operational and technical effectiveness	Patients Clinicians	• The system is able to do everything I would want it to be able to do [6]
			• I think the visits provided over the telehealth system are the same as in-person visits [6]
			• I got enough information via home telecare to understand my diagnosis [8]
			• I got enough information via home telecare to manage my treatment [8]
			• I have adequate diagnostic confidence with telehealth*
			• I have confidence that the patient understands the care plan*

The following paragraphs contain detailed descriptions of the four NQF outcome domains. Although all domains can be addressed in survey questions, we also make suggestions for how to address the evaluation of each domain using other study designs that perhaps provide a more rigorous means of evaluation, such as time series, cohort, or randomized trials. We also provide Supplementary Table 1 as an example collection of studies since 2015 that have assessed telehealth outcomes related to the NQF outcome domains, which can help inform future evaluation studies.

The four outcome domains

Access: Access is defined as the timely receipt of appropriate health care [9]. According to NQF guidelines, access includes five components – affordability, availability, accessibility, accommodation, and acceptability [2]. Affordability addresses the question of whether or not the costs associated with telehealth are acceptable to both patients and members of the care team. Availability assesses whether or not the telehealth system is able to provide the type of care required by the patient in a timely manner. Accessibility and accommodation are highly connected and jointly measure whether or not telehealth modalities can accommodate the diverse needs of patients and whether or not patients and clinicians are able to access telehealth resources when requested. And lastly, acceptability is a measure of utilization and addresses whether or not patients and clinicians accept the use of telehealth as a means of care delivery. Of these components, accessibility and accommodation are the least well-studied and arguably have the most impact on the reach of telehealth as access barriers can exclude vulnerable patient populations from receiving remote care. Populations that experience barriers to telehealth access include patients who are Black, Hispanic, older, less educated, and lack digital literacy [3]. Common barriers experienced by these groups are lack of access to technology and reliable internet access, poor technological literacy, and different abilities. For instance, some patients may have visual, hearing, mobility, or cognitive impairments. To ensure equitable access, we recommend assessing variability in telehealth use across social and demographic characteristics [10]. Additionally, language preference should be considered as non-English speaking patients may also lack access to telehealth services if interpretation services are not integrated into virtual care. To address these areas of accessibility and accommodation, we have compiled various tools in Table 2 that may help practices reach a wider patient population by increasing the capacity of telehealth systems to meet the many diverse needs of patients. These tools include tutorials and lessons designed to increase digital literacy in older adults and other populations, as well as a accessibility tools that help address audiovisual, mobility, and cognitive impairments while using technology. The tutorials and lessons can be integrated into a digital literacy training program to facilitate telehealth navigation for patients with limited digital literacy, and accessibility tools can be integrated into the telehealth platform based on patient need.

Table 2

Telehealth accessibility and accommodation resources.

Resource	Description
Skillful Senior web tutorial [11]	Offers interactive online tutorials on basic mouse, arrow, and typing skills
Generations on Line app/web tutorials [12]	Provides free downloadable app “Easy Tablet Help for Seniors” with basic interactive tutorials on how to search on the internet, FaceTime, Zoom, text, take photos, and email. Content also available as a website instead of an app.
Learn My Way online lessons [13]	Not specifically tailored to seniors, but offers a wide range of online lessons with video and audio on topics including device usage, online usage, online safety, video calling, email etc.
Goodwill Community Foundation online technology lessons [14]	Not specifically tailored to seniors, skill level ranging from basic computer skills, email, internet use and how to use different operating systems, to more advanced Office tips (Word, Excel, Powerpoint, etc.)
Microsoft Accessibility Tools [15]	Offers guides on how to use the accessibility features of Microsoft products, the six areas of accessibility include vision, hearing, neurodiversity, learning, mobility, mental health. Not specifically tailored to seniors, but vision, hearing and mobility accessibility might pertain to older adults as well as other populations.

Cost: A 2017 review of twenty-one telehealth cost studies found that for most major medical fields, telehealth is cost effective [16]. However, that does not mean that health systems and individual practices should neglect to consider cost evaluations. Costs can be viewed from both the patient and the system perspective. For the patient, telehealth-related costs include internet fees and the potential costs of telehealth devices. During the COVID-19 pandemic, some doctors have reported that patients stopped answering phone calls at the end of the month due to limits on minutes through their mobile carrier [5]. Understanding patient cost barriers can inform the timing and means of contacting patients. From the patient perspective, potential cost-saving areas when using telehealth include decreased travel expenditures (e.g. parking and gas costs), missed days at work, and shorter waiting and consultation times. For the system, telehealth-related costs include training time, wages, equipment and facilities, telecommunication costs, while savings include potential reductions in length of consultations, hospitalizations, readmissions, emergency department visits, and laboratory tests. For future telehealth cost evaluation studies, Dávalos et al. make several recommendations to address research gaps in telehealth cost analyses; they recommend randomized controlled trials whenever possible, including all stakeholder perspectives and using reliable context-specific monetary conversion factors to estimate economic benefits [17].

Experience: The NQF divides measures of experience into three subdomains: patient, family, and caregiver experience; care team member experience; and community experience. Of these three subdomains, patient, family and caregiver experience has been most assessed in the past, commonly in the form of satisfaction surveys. In comparison, there is a lack of existing literature on care team member experience and community experience. These two subdomains include acceptance of telehealth in the community at large, and the use of telehealth to “facilitate teamwork and the ongoing care of a patient, as well as the utility of the technology to provide necessary information to assist in the provision of care” [2]. These subdomains have great potential for future evaluation, and it would be beneficial to understand the capacity of telehealth to facilitate teamwork within a care team, as well as the level of telehealth acceptance across different communities. Furthermore, experience is often misguidedly gauged solely through satisfaction. In 2000, a report prepared for the Department of Health and Human Services found that patient satisfaction has been the most evaluated outcome of telehealth and that it was almost universally high in past studies [18]. This positively-skewed ceiling effect makes critical evaluation based on patient satisfaction challenging, thus it was concluded that other measures for experience should be targets for future evaluation. These alternative measures include the ease of use of technology, its technical quality, and the patient-clinician interaction.

Effectiveness: According to the NQF framework, effectiveness measures should include system, clinical, operational, and technical effectiveness. System effectiveness is the ability of the telehealth modality to coordinate care across various settings and share information between clinicians; clinical effectiveness refers to health outcomes, the compared effectiveness to in-person care and process measures of quality such as accuracy of the diagnoses; operational effectiveness measures the integration of telehealth within a care setting; technical effectiveness refers to the quality of data transmission to patients and data exchange between members of a care team [2]. While patient or clinician-perceived effectiveness can be quickly gauged through questionnaires, effectiveness is best measured through randomized controlled trials that take place over a longer span of time. We summarize randomized control trials published since 2015 that pertain to NQF telehealth outcome domains (Supplementary Table 1), and many can help inform future studies on effectiveness. Because telehealth uptake was limited prior to the COVID-19 pandemic, studies with high validity and reliability that are generalizable to large patient populations are still lacking [19]. With the large uptake of telehealth, more robust statistical analyses can be completed with greater sample sizes using quasi-experimental and other study designs, such as case-control or time series analysis [20]. For example, a 2016 case-control study examining the use of a telehealth program in emergency medical services found telehealth effective in reducing unnecessary emergency department ambulance transports [21], a 2018–2020 time series study investigating the effectiveness of a telehealth service for antenatal care found that telehealth integrated care enabled the reduction of in-person consultations without compromising pregnancy outcomes [22], and a study design has been published for a cluster-randomized trial to compare telehealth care with clinic-based care for uncontrolled hypertension [23].

Starting an evaluation system and using it to inform improvements

A site context-appropriate questionnaire can be designed using the suggested items in Table 1, and then deployed to patients and clinicians within the telehealth system. The questionnaire can be given to the patient or clinician after a session through email or as a screen pop-up if applicable. To increase response rates, we recommend notifying participants about the survey ahead of their appointment or during the call, and sending follow-up reminders, both of which have been found to significantly improve responsiveness [24]. We recommend oversampling of patients from priority populations - older, Black, Hispanic, or low-income groups - rather than sampling a consecutive or random group of patients, because otherwise responses will be too few to produce reliable estimates of access needs. Once responses are collected and analyzed, areas of improvement can be identified, and measures can be implemented to enhance performance. Simultaneously, areas of success can also be identified, and strategies can be developed for long term implementation after the immediate increase in telehealth use during the COVID-19 pandemic. We illustrate the process of initiating the evaluation and acting on the results in Fig. 2 . For example, if the questionnaire reveals patient challenges with telehealth usability (i.e. low score for “Experience” domain items in Table 1), pre-visit training programs (e.g. instructional videos, instructional pamphlets, or interactive lessons) can be developed by telehealth staff and provided to patients as a resource to help them navigate telehealth platforms. Based on context and available resources, in-depth semi-structured interviews with patients can also be conducted to better understand the specific areas of usability that need improvement before implementing changes.

Fig. 2

An example evaluation workflow.

To illustrate how an evaluation system may lead to tangible changes in how telehealth is implemented, we share a case study. In a study published in 2019 evaluating the Yukon Telehealth System in Canada, researchers were able to identify barriers and recommendations to improve the delivery of care to remote communities in Yukon using both quantitative and qualitative data collected from April to August in 2016 [25]. Quantitative data was obtained through usage logs and questionnaires that were administered to nurses and patients who had used telehealth, and qualitative data was obtained through focus groups and semi-structured interviews with telehealth stakeholders including physicians, nurses, managers and telehealth specialists. From the data researchers found that clinicians and patients were generally satisfied with the telehealth system and believed that telehealth had many benefits such as saving patients’ time and money by reducing the need to travel to hospitals, being a convenient method of accessing care, and preventing isolation in remote communities by providing a means to connect patients to clinicians through virtual visits. Researchers also identified limitations of the telehealth system, including complex workflows to use the technology which resulted in underutilization, understaffed telehealth teams and overreliance on a single telehealth coordinator, and the need for patients to travel to community health centers with telehealth units in order to participate in appointments. Based on these findings, the researchers provided recommendations for improvement such as reducing workflow complexity through adopting new telehealth platforms that enable clinicians to receive calls from their own office and initiate their own sessions, increasing telehealth training to more staff members beyond the telehealth coordinator, and providing patients with home-based service options. Informed by their evaluation study, the researchers were able to provide the Yukon government with targets for the expansion and improvement of the telehealth system.

Resources and future considerations

To complement evaluation questionnaires, other metrics can be assessed to understand telehealth performance, such as the usage logs used in the Yukon case study above. The American Medical Association Telehealth Implementation Playbook recommends that success metrics be defined before evaluation, such as telehealth visit volume, increased appointment availability due to telehealth, and reduced no-show rates for appointments [26]. These metrics can further guide the development of site-specific evaluation systems. Due to the highly context-dependent nature of telehealth performance, it is unfeasible to recommend a set frequency for evaluation across different settings. However, new telehealth systems can benefit from more frequent evaluations during early stages of operation and implementing changes accordingly until a steady rate for iterative assessment and improvement is achieved. With any significant change in workflow or care delivery model, performance should be re-assessed. Although to our knowledge there have not been any documented uses of the Continuous Quality Improvement (CQI) improvement management process in telehealth evaluation, there is potential for such a process to aid in systematic telehealth evaluation. In particular, the CQI strategy developed by the Institute for Healthcare Improvement (i.e. IHI Model of Improvement) could be useful in testing changes to telehealth workflow or evaluating a new telehealth system, as it is characterized by testing in short, rapid cycles and applying lessons learned to new cycles [27]. The use of CQI methods in telehealth could be a potential area of future research [28]. In addition, evaluation tools such as qualitative interviews and comparative studies are needed to better understand site-specific areas for improvement and the long-term implications of telehealth use. Table 3 summarizes telehealth resources and tools that can be used for designing evaluation studies. These resources include comprehensive reviews that summarize and discuss different evaluation frameworks, provide guidance on project design and data collection, and make recommendations on approaches to telehealth evaluation. In general, the resources recommend designing evaluations with multiple outcome domains similar to those recommended by the NQF, and agree that the inclusion of a variety of telehealth stakeholders is beneficial to a comprehensive assessment.

Table 3

Resources for telehealth evaluation design.

Resource	Description
National Quality Forum (NQF) telehealth measurement framework final report (2017) [2]	A report prepared by a multi-stakeholder committee that contains recommendations on various methods to measure the use of telehealth as a means of providing care. Includes a four-domain measurement framework that can serve to inform evaluation work on the impact of telehealth on the cost and quality of care and health outcomes.
2014 review article outlining practical approaches to telehealth evaluation [29]	Reviews evaluation frameworks and common challenges in telehealth evaluation, presents case study for evaluation. Recommends a pragmatic, multi-method, multi-phase evaluation approach that is flexible and can be adapted to the characteristics and challenges unique to each telehealth program.
Queensland Health Evaluation Resource Guide (2016) [30]	Comprehensive telehealth evaluation guide developed by the Department of Health in the State of Queensland, Australia. Provides guidance for project planning, design, data collection and analysis, and summarizes existing evaluation frameworks.
2003 synthesis of telehealth outcomes literature [31]	A University of Calgary description of the Canadian National Telehealth Outcomes Indicator Project. Provides a synthesis of telehealth outcomes literature and a Telehealth Outcomes Development (TOD) model to guide the process. Appendices provide additional information regarding international health outcomes projects, and potential tools.
The Model for ASsessment of Telemedicine applications (MAST) [32]	A 2012 paper presented MAST, a structure for telehealth assessment developed based on workshops and using existing models, consisting of seven domains: health problem and characteristics of the application, safety, clinical effectiveness, patient perspectives, economic aspects, organizational aspects, and socio-cultural, ethical and legal aspects.

Conclusion

Telehealth uptake has been rapid during the COVID-19 pandemic, increasing the need for telehealth evaluation to ensure standards of care delivery. This can be done by assessing the reach, cost, experience, and effectiveness of existing remote care practices. Special attention must be paid to ensuring priority populations - older adults, underrepresented minorities, the traditionally underserved, and those with disabilities - are included as telehealth is widely embraced. With measurement and iterative improvement of telehealth outcomes, health systems and patients can expect virtual visits to become a staple of medical care through the US.

Funding

None

Ethical approval

Not required

Patient consent

Not required

Declaration of Competing Interest

None declared