Rapid development of visualization dashboards to enhance situation awareness of COVID-19 telehealth initiatives at a multihospital healthcare system.

The COVID-19 pandemic has led to the rapid expansion of telehealth services as healthcare organizations aim to mitigate community transmission while providing safe patient care. As technology adoption rapidly increases, operational telehealth teams must maintain awareness of critical information, such as patient volumes and wait times, patient and provider experience, and telehealth platform performance. Using a model of situation awareness as a conceptual foundation and a user-centered design approach we describe our process for rapidly developing and disseminating dashboard visualizations to support telehealth operations. We used a 5-step process to gain domain knowledge, identify user needs, identify data sources, design and develop visualizations, and iteratively refine these visualizations. Through this process we identified 3 distinct stakeholder groups and designed and developed visualization dashboards to meet their needs. Feedback from users demonstrated the dashboard's support situation awareness and informed important operational decisions. Lessons learned are shared to provide other organizations with insights from our process.

INTRODUCTION

The COVID-19 pandemic has presented healthcare provider organizations with unique healthcare delivery challenges. Organizations have had to rapidly prepare for a surge of high acuity patients that could strain capacity while dramatically reducing routine nonessential care to minimize community spread of illness and protect healthcare workers. These changes pose many problems including managing the unique isolation requirements for infected patients, maintaining communication among distributed care teams, and meeting the needs of the broader patient population. Telehealth has emerged as a primary solution to these challenges that has allowed for clinicians to reach beyond the walls of the facilities within which they work.

MedStar Health, a 10-hospital system with over 280 outpatient clinics responded to the pandemic by accelerating its telehealth program in 3 ways. First, on-demand telehealth visits, referred to as “eVisits,” allowing patients to connect with a clinician through voice and/or video, were made available at no cost to any person living in Washington DC, Maryland, and Virginia. This was done to provide a low-barrier method for patients to engage with a clinician and reduce patient volumes at emergency departments and urgent care centers. Second, physicians from nearly all specialties, such as primary care and cardiology, were enabled with telehealth to conduct ambulatory patient appointments, referred to as “Video Visits,” with voice and/or video. This served to replace in-person appointments, limiting exposure between patients and healthcare workers. Finally, telehealth allowed clinicians to conduct voice and/or video meetings, referred to as “eConsults,” with each other so physical location did not limit the reach of specialized clinical expertise. These capabilities were developed over the course of a 3-week period and released to clinicians and affiliated professional across outpatient and inpatient settings in MedStar Health, during which the number of telehealth engagements increased from an average of 100 a day to over 5000 per day.

With this rapid expansion and dramatic increase in telehealth use, there was an immediate need for improved situation awareness of all telehealth operations to effectively monitor and proactively manage patient experience, healthcare provider experience, and platform performance. To achieve this, a 7-person multidisciplinary visualization project team was formed that included expertise in human factors, telehealth, data visualization, data science, and informatics. The team was formed from experts that had different research and operational roles across MedStar Health and had the capabilities required for this effort. This case report describes our methods for identifying operational end user needs and designing and developing visualizations to meet those needs. Finally, we provide recommendations for rapid visualization development.

MATERIALS AND METHODS

Foundations of telehealth visualization development

A human factors-based theoretical model of situation awareness (SA) guided conceptualization of visualization dashboards to meet the needs of telehealth stakeholders. The SA model postulates 3 different levels: perception, comprehension, and projection., Level 1, perception, is the most basic level and is the ability to perceive the current state and monitor specific data elements in the environment. Level 2, comprehension, is the ability to integrate and synthesize different data elements. Level 3, projection, is the ability to forecast or predict future states. During the initial phases of the organization’s adoption and scaling of telehealth technology, our team supported perception and comprehension needs through visualization of platform performance and utilization. More complex modeling approaches to achieve enhanced projection were delayed in order to fulfill these immediate needs.

Visualization development was driven by a user-centered design process that put the needs of end users at the forefront of design and development. This design process involved developing prototype visualizations, soliciting feedback from end users, and iteratively improving the visualizations. Because of the urgent need for visualizations, a rapid process of development and user feedback from domain experts and/or end users was developed.

Our multidisciplinary visualization team used a 5-step process to create and iteratively refine visualization dashboards to support the initial launch and ongoing development of the expansive telehealth program. This study was approved by the institutional review board.

Subject matter expert interviews to increase domain knowledge

With a visualization team new to telehealth there was an immediate need to increase domain knowledge. We conducted interviews with the 3 operational directors of the telehealth program and the 3 operational managers of the different areas of telehealth (on-demand visits, ambulatory visits, and provider consults). The interviews with the directors were focused on understanding the telehealth program, current and near-term operations, and identification of key stakeholders that required awareness of the telehealth program. The interviews with the operational managers were focused on understanding the technology, specific vendors supporting it, clinical use, and the data it produced. These interviews lasted for about 1 hour and were the beginning of an ongoing channel of communication about needs and visualization feedback. Notes were taken during the interviews and these notes were discussed by the visualization team.

User needs analysis and feature identification

Once stakeholder groups were defined from the subject matter expert interviews, we conducted a “quick and dirty” needs analysis with the operational directors and 1–2 members of each operational team to identify information needs for operational decision-making and executive awareness. User needs were elicited through 30-minute virtual meetings in which these stakeholders were asked about the information they needed. These needs were then synthesized through group discussion to create a short list of visualization requirements for each stakeholder group (Table 1).

Table 1.

Stakeholder needs and dashboard features

Stakeholder	Needs	Visualization Dashboard Features and Decisions Supported
Executives(SA level 1—perception)	Awareness of telehealth patient volume, provider usage, and patient experience across system. Simple representations to quickly highlight trends and deviations. Weekly reports to summarize current status in context of overall initiative.	Features:Summary metrics, weekly tables, bar graphs and line charts of volume, provider usage, and patient experience. Simple and straightforward terminology to enhance readability. Show current week’s data in context of all data. No interactivity or filtering. Enabled decisions about: Effectiveness of telehealth solutions in terms of patient reach Where to focus resources
Telehealth Operational Leaders(SA level 2—comprehension)	KPIs for each telehealth initiative, including overall volume, wait times, visit duration, provider utilization, technical issues, and patient experience. Daily refreshes to monitor prior-day telehealth operations and compare to previous days. Report aggregate summary metrics across different time periods.	Features:Dense presentation of specific summary metrics and bar charts representing overall volumes. Specific terminology describing complex KPIs. Incorporates end user date filtering allowing control over calculation of summary numbers. Enabled decisions about: Provider staffing levels Specific operational areas that requires further attention Areas for operational improvement
Telehealth Management Teams (SA levels 2 and 3—comprehension and projection)	Detailed understanding of patient volume, provider use, and platform performance for all telehealth initiatives. Ability to diagnose issues with near real-time data and high level of control over data filters. Specific representations to highlight patterns supporting projection and forecasting.	Features: Visualizes a large number of relevant data fields to uncover insights that answer operational questions Filters across large time windows and manipulate performance indicator parameters Drills down to individual provider or call-level information. Enabled decisions about: Technical enhancements to improve performance Where to focus clinician training efforts Need for additional equipment

Abbreviations: KPI, key performance indicator; SA, situation awareness.

Processing telehealth data sources

The visualization requirements for each stakeholder group were used to guide the acquisition and analysis of relevant data produced by the telehealth platforms. Our telehealth system relied on 2 independent platforms supported by different vendors, each with their own unique data architecture and structure. As neither vendor fully supported industry-standard application programming interfaces nor a common data format, data were retrieved from their proprietary data platforms using semi-automated scripts and ingested into structured relational database tables. Data were validated through a manual quality assurance process. Varying time constraints from each of our stakeholder groups, evolving data structures, and variable reliability of these data feeds required redundant and flexible extraction methods in order to maintain a resilient situation awareness infrastructure. Further, each platform had unique ways of recording constructs (visit time, visit duration, and dropped or disconnected calls), requiring data interrogation to determine appropriate metrics for operational needs. We were not able to apply any standard data format or coding system to these data.

Visualization design, development, and testing

Dashboards were developed using Tableau (Version 2020.1) was connected to the relational database. With the variety of stakeholder and end user needs, our visualization team divided into pairs consisting of a lead developer, driving dashboard development, and a support developer, handling data issues or providing additional context around each telehealth platform or program. These pairs would create prototype dashboards, starting with a basic layout of data feeds in a development environment then rapidly iterating with end users to ensure the dashboards met their needs. They would also share these iterations with the broader visualization team to solicit feedback on the design and highlight any data discrepancies between dashboards. After stakeholder and visualization team approval, dashboards would be checked by 1 member of the visualization team to ensure proper functionality and labeling before being uploaded to a production environment.

Dissemination and iterative refinement

Once in the production environment, these dashboards were disseminated to all relevant stakeholders as either a push (scheduled e-mail or message alert) or pull (on-demand availability on a network server). End users were engaged openly and continuously for feedback in order to ensure that dashboard design, elements, and functionality met their specific role-based needs. Each piece of feedback or feature request was then prioritized based on estimated level of effort and impact then fit into the visualization team’s development plan.

RESULTS

Through our 5-step visualization development process we identified distinct stakeholder groups based on their information needs and designed, developed, and implemented dashboards to meet those needs. The visualization team began work on March 16th with each of the 7 members dedicated full-time. The 5-step process was completed over 2 to 3 days for generation of each initial dashboard and then optimized on an ongoing basis.

Our interviews resulted in the identification of 3 different stakeholder groups and their respective SA awareness needs: healthcare system executives, telehealth leaders, and telehealth managers (Table 1). At the highest level of the organization, healthcare executives included the chief medical officer and others with responsibility for overall organizational operations. From our interviews with telehealth leaders, we identified that executive stakeholders needed weekly awareness of high-level metrics and trends to convey the overall activity of telehealth across the system (Table 1). Figure 1 shows an example of the eVisit executive summary dashboard.

Figure 1.

On-demand eVisit executive dashboard.

Telehealth leaders were identified as a second stakeholder group including 7 people charged with leading day-to-day telehealth operations for the healthcare system. These stakeholders requested daily awareness of key operational performance indicators with the ability to integrate information to determine if a specific operational area needed focused attention and resources (Table 1). These needs were met with an overview dashboard representing volume and platform performance across all 3 telehealth initiatives (Figure 2). These operational leaders also had access to all dashboards being developed and could investigate more detailed information through the operational manager dashboards as described below.

Figure 2.

Telehealth operational leaders’ dashboard.

Telehealth operational teams included managers and team members involved in active development and maintenance of each telehealth program. These stakeholders needed detailed information about their respective areas with the ability to diagnose where issues such as increased patient volumes, poor patient experiences, or dropped calls were occurring (Table 1). A dashboard was created for each program (see eVisit dashboard, Figure 3), and, wherever possible, features were developed to support projection needs for clinical staffing and platform technical issues. Finally, ad-hoc visualizations were produced on request to support specific issues or questions.

Figure 3.

On-demand eVisit operational team dashboard.

The primary users of these dashboards are healthcare system operational leaders and telehealth operational staff. These dashboards were not distributed to frontline clinical staff. Table 2 provides an approximate number of users and frequency of use and a summary of user feedback. Dashboard feedback was generally positive with several users expressing specific benefits, including efficiency of access to data and ease with which they could understand critical information compared to use of the raw data feeds or vendor-provided platforms with limited visualization capabilities. The primary limitations were a lack of connection to other data systems and lack of detail for diagnosing specific technical issues. Our visualization team also observed that developing these dashboards facilitated the centralization of data sources and standardization of metrics and terminology across the operation. Operational telehealth leader and management users also provided feedback on operational decisions and functions that were influenced by these dashboards. These included the following:

Table 2.

Dashboard users, frequency of use, and feedback

Stakeholder group	Approximate number of users	Approximate frequency of use	Feedback
Health System Executives	5	Weekly	Benefits: High level summary and gestalt of telehealth adoption and utilization across system during pandemic.Limitations: Telehealth data disconnected from other enterprise-level information systems related to patient care, administration, and billing.
Telehealth Operational Leaders	9	Daily	Benefits: Summary of telehealth platform KPIs for strategic planning and decision making; executive and cross-department reporting.Limitations: “Blind spots” of telehealth data (ie, lack of detailed utilization/diagnostic information, inability to connect to other information systems).
Telehealth Management Teams	30–40	Multiple times per day	Benefits: Ability to drill down into provider and call-level details; telehealth leadership, and cross-team reporting.Limitations: Lack of real-time information, lack of detailed diagnostic and utilization information, limitations in connecting to other enterprise-level information systems.

Abbreviations: KPI, key performance indicator.

Identifying when telehealth platform issues were occurring and deciding where to dedicate resources to troubleshoot technical/training issues.

Tracking Video Visit provider utilization across specialties to identify where additional workflow and/or training optimization is required.

Informing eVisit provider staffing decisions by knowing current and past patient volumes and wait times.

Identifying patient experience and patient follow-up needs by knowing which patients are not able to complete eVisits or Video Visits.

DISCUSSION

COVID-19 has required many healthcare organizations to rapidly expand telehealth services, requiring operational teams to have robust situation awareness of patient and provider experience as well as telehealth platform performance. To meet these needs, we formed a multidisciplinary visualization team with a situation awareness model as our conceptual framework and a rapid user-centered design approach. User feedback suggests the visualizations improved situational awareness and may have provided valuable information to better inform operational decisions. We also faced several challenges, including interoperability, poor vendor data standards, and challenges accessing real-time data. Limitations to our approach include: no formal interrater reliability when synthesizing interview content, no design for different devices (eg, mobile phones, tablets, etc), and no consideration for users with disabilities.

We have identified several lessons learned that may be valuable to other visualization development teams:

Use a lightweight user-centered approach: A rigorous user-centered design approach is not possible when the goal is to produce same day data visualizations. Use a lightweight approach by quickly identifying stakeholder groups and identifying at least 1 actual or representative user for feedback.

Basic (accurate) visualization is better than no visualization: User needs should be identified quickly and basic dashboards for immediate use should be developed as a starting point for rapid feedback cycles. Focus on meeting user needs as opposed to spending time on complex or comprehensive representations of data.

Clearly indicate development status and timeliness: Having users view dashboards that are still in development is necessary, given the pace of work; however, this information as well as the “freshness” of data being represented should be clearly marked.

Develop prioritization criteria for visualization and feature development: Work with a variety of stakeholders to develop clear criteria based on operational strategy on how to prioritize dashboard development effort.

Stabilize the data environment as much as possible: Reliably receiving data from telehealth vendors, particularly under rapid growth, can be challenging. Time should be invested to understand data source characteristics and ensure efficient methods for data ingestion and storage. Further, consider mapping telehealth data standard identifiers or existing datasets to enrich dashboards, as many stakeholder needs may require integrating telehealth data with data from other clinical information systems.

Metric design, user literacy, and user control should be based on audience: Be mindful of the audience and ensure that the metrics being presented meet audience needs. Terminology for executive-level audiences should be jargon-free. Certain dashboards should be designed as awareness for the occasional user and others as tools for superusers.

COVID-19 requires the rapid adoption and expansion of telehealth to protect both patients and providers. With this unprecedented shift comes the need for effective visualization of telehealth data to support operational needs. With a multidisciplinary visualization team, these needs can be met. Next steps include improvements to the data warehouse and gathering more feedback from end users for dashboard optimization.

FUNDING

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

AUTHOR CONTRIBUTIONS

All authors contributed to conceptualization of the case report and all authors reviewed and edited the case report. RAD and RMR led the writing.

CONFLICT OF INTEREST STATEMENT

None declared.