Lessons learned and keys to success: Provider experiences during the implementation of virtual oncology tumor boards in the era of COVID-19.

BACKGROUND AND OBJECTIVES: The COVID-19 pandemic required rapid adaptation of multidisciplinary tumor board conferences to a virtual setting; however, there are little data describing the benefits and challenges of using such a platform.
METHODS: An anonymous quality improvement survey was sent to participants of tumor board meetings at a large academic institution. Participants answered questions pertaining to the relative strengths and weaknesses of in-person and virtual settings.
RESULTS: A total of 335 responses (23.3% response rate) were recorded, and 253 met inclusion criteria. Respondents represented 25 different tumor board meetings, with colorectal, breast, and liver (18.6%, 17.0%, and 13.0%, respectively) being the most commonly attended. Virtual tumor boards were equivalent to in-person across 9 of 10 domains queried, while a virtual format was preferred for participation in off-site tumor boards. The lack of networking opportunities was ranked by physicians to be a significant challenge of the virtual format. Consistent leadership and organization, engaged participation of all attendees, and upgrading technology infrastructure were considered critical for success of virtual meetings.
CONCLUSIONS: The implementation of virtual tumor board meetings has been associated with numerous challenges. However, improving several key aspects can improve participant satisfaction and ensure excellent patient care.

INTRODUCTION

Providing integrated, multidisciplinary care to oncology patients has become a cornerstone for achieving improved outcomes. Multidisciplinary teams, also known as MDTs or Tumor Boards, are a valued setting for providers from numerous specialties to meet in person to collaborate and coordinate patient care. The concept of MDTs has existed in the medical community for some time, and numerous studies have shown improvements in patient management and outcomes across diverse specialties including gastrointestinal, breast, pediatrics, and rare tumors. However, as MDTs became more common at large academic centers, a notable shortcoming has been identified: providers practicing in rural or community settings do not always have a local MDT and they may be unable to travel to present rare cases at large specialized centers. As a result, several groups established virtual MDTs to address this critical need in rural community settings and showed virtual meetings to be feasible and effective in improving care. , , , ,

Despite studies showing that virtual MDTs were practical, in‐person meetings have remained the gold standard at large centers. However, unbeknownst to the providers who implemented virtual tumor boards in rural settings more than a decade ago, utilizing a virtual platform for these meetings would take on new importance during the COVID‐19 pandemic. As COVID‐19 cases increased dramatically in early 2020, providers had to balance partially competing interests: ensuring uninterrupted multidisciplinary care for patients while minimizing face‐to‐face meetings to protect their own safety and minimize infectious spread among colleagues. In response, the traditionally in‐person MDT meetings were rapidly adapted to a virtual platform.

At Cleveland Clinic, there are more than 20 different subspecialty oncology MDT meetings, all of which transitioned to virtual settings during COVID‐19. This rapid change was challenging, as providers who had little experience in how to organize these meetings were tasked with maximizing both patient care and participant satisfaction. Nearly a year after the COVID‐19 began, data regarding the results of implementing virtual MDT meetings remains sparse, with available reports limited to single specialties and small sample sizes. Therefore, the aim of this study was to conduct a quality improvement survey including all Cleveland Clinic departments to identify the keys to success and common pitfalls associated with virtual MDT meetings. We hypothesized that strong organizational leadership would be critical for success while technologic barriers would be an opportunity for improvement, regardless of subspecialty. We also anticipated that providers' preferences for virtual or in‐person MDT meetings would vary by role and training level.

METHODS

Survey design

A quality improvement survey was designed to solicit feedback regarding the benefits and challenges of transitioning from in‐person to virtual tumor board meetings in the midst of the COVID‐19 pandemic. The survey was approved by the Institutional Review Board of Cleveland Clinic and informed consent was waived. The survey was designed by the authors and was piloted before administration. The pilot consisted of a group of faculty from diverse specialties who reviewed the survey and provided feedback to the authors. The survey was then revised and reviewed again by the authors before final approval and distribution.

The final survey comprised 16 questions and was divided into three sections. The first section consisted of a single screening question which asked participants whether they had experience with both in‐person and virtual tumor boards. Respondents who answered “No” to this question were excluded from analysis. The second section collected basic demographic information regarding training status, years in practice, hospital affiliation, specialty of practice, role, and frequency of participation in tumor board meetings. More detailed personal demographic information was not collected, so as to maintain the anonymity of the participants. The third section asked responders to compare their experiences with in‐person and virtual tumor boards with regard to several meeting characteristics using a modified anchored Likert scale, where −10 = in‐person is better, 0 = no difference between them, and 10 = virtual is better. Participants' attitudes regarding several common hurdles encountered during virtual meetings were also queried using an anchored scale where 0 = no problem and 10 = very significant problem. Responders were also asked to rate the extent to which potential changes would improve virtual tumor boards using an anchored scale from 0 to 10 where 0 = would not improve it and 10 = would greatly improve it. The final Likert scale question asked individuals to indicate their attitudes regarding the following statement, “Despite modifications, virtual tumor boards just aren't as good as in‐person tumor boards” using a classic 5‐point Likert scale from “strongly agree” to “strongly disagree.”

Respondents

The survey was distributed to 1437 Cleveland Clinic caregivers via email distribution lists for each tumor board group. Responses were solicited from all tumor board specialties, including the following: adrenal, brain, breast, bone marrow transplant, colorectal, esophageal, general, genitourinary, upper gastrointestinal, gynecology oncology, head and neck, hematology, leukemia, liver, lung, lymphoma, melanoma, multiple myeloma, nonmelanoma skin cancer, pancreas, pediatric, pituitary, sarcoma, spine, and thyroid and parathyroid. Participants included both trainees (e.g., residents and fellows) as well as fully‐licensed providers practicing at Cleveland Clinic locations in Ohio and Florida. All responses were anonymous and were collected over a 30‐day period from September 10th, 2020 through October 10th, 2020.

Statistical analysis

Continuous variables are reported as means with SD or medians with interquartile ranges while categorical variables are reported with counts and percentages. Survey responses were analyzed using analysis of variance for numeric data and chi‐square for categorical data. Subgroup analyses were performed by training status (trainee vs. complete) and role (surgeon, nonsurgeon physician, and nurses/APPs/other). Between‐group comparisons were analyzed using Kruskal–Wallis and Mann–Whitney U tests as appropriate. All p values <0.05 were considered statistically significant. All analyses were performed using R (version 4.0.2, R Core Team 2020).

RESULTS

Survey participants

A total of 333 individuals responded to the survey, for a response rate of 23%. Of these, 253 completed the survey, 36 answered “No” to the first screening question and were excluded, and 44 began the survey but did not complete a sufficient number of questions to be included in the analysis. Twenty‐five (9.9%) respondents were trainees (resident or fellows) while the remaining 228 (90.1%) had completed their training. Forty‐eight surgeons, 123 nonsurgeon physicians, and 82 nurses, advanced practice providers, and other caregivers were included. Further details regarding respondents' roles and experience are provided in Table 1. Colorectal, breast, and liver (18.6%, 17.0%, and 13.0% of respondents, respectively) were the most commonly attended meetings.

Table 1

Survey respondents' characteristics

Variable	N (%)
Training status
Trainee	25 (9.9)
Completed training	228 (90.1)
Role
Nursing and APPs	62 (24.5)
Surgeon	48 (19.0)
Radiology and interventional radiology	42 (16.6)
Medical and radiation oncology	38 (15.0)
Pathologist	26 (10.3)
Other support staff (e.g., researcher, coordinator, patient liaison, others)	17 (6.7)
Medical specialists (e.g., endocrinology, pulmonology, hepatology, etc.)	15 (5.9)
Other specialists (e.g., psychologist, pharmacist, genetic counselor etc.)	5 (2.0)
Time in practice since completing training*
0–5 years	55 (24.1)
6–10 years	53 (23.2)
11–15 years	33 (14.5)
16+ years	86 (37.7)
Number of tumor boards attended per week
One	169 (66.8)
Two	66 (26.1)
Three	15 (5.9)
Four or more	3 (1.2)

Note: *1 response unknown.

Abbreviation: APP, advanced practice provider.

Ease of participation

The survey first asked respondents to evaluate ten different domains of tumor board meetings and whether they are more easily achieved in an in‐person format or virtual format. Eight of the ten domains had a median response score of 0, corresponding to the statement that there is no difference between in‐person and virtual formats. There was a small preference for an in‐person format with regards to the Quality of Case discussion (median score = −1, interquartile range [IQR] = −4–0). In contrast, Ease of Participating in Offsite Tumor Boards favored a virtual format (median score = 7, IQR = 3–10). When responses were grouped by Training Status (trainees vs. staff) a virtual format was favored for Ease of Reviewing Images by trainees (median score = 5, IQR = −0.5–8.5) but not by staff (median score = 0, IQR = −3–5) (p = 0.03, Figure 1). There was no significant difference in scores between trainees and staff across any other domain (all p > 0.05). When responses were divided by respondents' role (surgeons, non‐surgeon physicians, and other), there were no significant differences in scores across any domain (all p > 0.05).

Figure 1

Responses to the question “Compared to in‐person Tumor Board, please rate virtual Tumor Boards on a scale of −10 to +10 (−10 indicates that in‐person is better and+10 indicates than virtual Tumor Board is better).” Responses by Training Status (in training vs. completed training)

Challenges with virtual tumor boards

Respondents then evaluated six common challenges associated with a virtual tumor board format and rated the severity of each problem from zero to ten. Overall, the largest problem was the lack of opportunity to network with colleagues (median score = 5, IQR = 1–8). Three issues relating to the quality of the conversation (not being able to hear well, multiple people trying to talk at once, and the inability to see who is speaking) all had median scores of three. Not being able to see presentations and connectivity issues were the least problematic issues, with median scores of two. When analyzed by training status, the issue of multiple people trying to talk at once was more significant for those in training (median score = 4, IQR = 3–6) compared to staff (median score = 2, IQR = 1–5) (p = 0.03, Figure 2A). There were no significant differences between training status across the remaining domains (all p > 0.05). When analyzed by provider role, connectivity issues were considered to be more significant of a problem by surgeons and nonsurgeon physicians (median score = 3) compared to nurses/APPs/others (median score = 2) (p = 0.02). The lack of opportunity to network with colleagues was scored significantly differently between the groups, with surgeons scoring it as the most significant problem (median score = 6, IQR = 3–8.25) while nurses/APPs/others considered it less of an issue (median score = 3, IQR = 0–7) (p = 0.004) (Figure 2B).

Figure 2

Responses to the prompt, “On a scale of 0–10 where 0 represents no problem and 10 is a very significant problem, please score the biggest problems with virtual Tumor Board.”

(A) Responses by Training Status (in training vs. completed training).

(B) Responses by Role

Potential changes to improve virtual tumor boards

After evaluating the perceived problems with virtual tumor boards, respondents were then asked to rate how various changes could improve virtual tumor boards. The changes felt to have the most potential for improvement were incorporating video feeds in the meetings (median score = 3, IQR = 0–6) and combining virtual meetings with an in‐person component (median score = 3, IQR = 0–5). Of note, using a different interface was not felt to have potential to contribute significant improvement (median score = 1, IQR = 0–5). When analyzed by training status, there were no significant differences between groups across the four domains (all p > 0.05). The addition of video to virtual tumor boards was viewed differently across roles, with surgeons believing that it has the potential for significant improvement of virtual tumor boards (median score = 5, IQR = 2–7) compared to both nonsurgeon physicians and nurses/APPs/others (median score = 2, IQR = 0–5) (p = 0.03, Figure 3).

Figure 3

Responses to the question, “How much would the following changes improve virtual Tumor Board? 0 = would not improve it at all; 10 = would greatly improve it”. Responses by Role

Further perceptions of virtual tumor boards

When asked to respond “Yes” or “No” to the question, “If a pandemic was not occurring, would you still prefer virtual tumor board?” a total of 116 (46.0%) individuals answered “Yes.” When divided by training status, there was no significant difference in the percent of “Yes” responses (40.0% of trainees vs. 46.7% of staff, p = 0.7). Similarly, there was no significant difference in the “Yes” response rate when comparing nonsurgeon physicians, surgeons, and nurses/APPs/others (40.2%, 43.8%, and 56.1%, respectively, p = 0.08).

The survey additionally tried to elucidate whether inconsistent mask use and social distancing was affecting participants' format preference by asking, “If social distancing and mask use were mandatory and enforced, would you still prefer a virtual tumor board?” A total of 159 (63.1%) individuals answered “Yes.” There was no significant difference in responses when divided by training status, as 68.0% trainees and 62.6% staff answered “Yes” (p = 0.7). Similarly, there was no significant difference in the “Yes” response rate when comparing non‐surgeon physicians, surgeons, and nurses/APPs/others (62.3%, 58.3%, and 67.1%, respectively, p = 0.6).

Last, respondents were asked to rate their response to the statement, “Despite modifications, virtual tumor boards just aren't as good as in‐person tumor boards.” using a 5‐point Likert scale ranging from “Strongly disagree” to “Strongly agree.” A total of 50.6% of respondents answered either “Strongly agree” or “Somewhat agree” while 36.7% responded with “Strongly disagree” or “Somewhat disagree” (Figure 4). The distribution of responses was not significantly different when analyzed by training status (p = 0.9) or role (p = 0.2).

Figure 4

Responses to the statement, “Despite modifications, virtual tumor boards just aren't as good as in‐person tumor boards.”

DISCUSSION

This is the largest study to date that reports quantitative comparisons of in‐person and virtual tumor board meetings across multiple oncologic disciplines during the COVID‐19 pandemic. Our results show that care providers felt that in‐person and virtual formats were similarly effective across almost all domains, and there was little variation in responses based on training status or provider role. Furthermore, those aspects of virtual meetings identified as most problematic are well suited to actionable plans to improve caregivers' experience with a virtual format moving forward.

Despite the need to implement virtual tumor boards in early 2020 with relatively little time to plan, responses from our cohort 6 months after the transition showed that most aspects of these meetings were equally effective in both formats. This is consistent with findings from a smaller study by Sidpra and colleagues where 24 responders described high efficacy of virtual meetings compared to the traditional face‐to‐face format across 11 criteria 1 month after the implementation of virtual meetings. Although assessed using a different question format, another early study from Cathcart et al. with 18 participants in the United Kingdom reported high satisfaction ratings in regards to the quality of audio, visual, and group interactions during virtual tumor board meetings for breast disease in the first 4 months following the transition. These findings are consistent with our much larger cohort, as the respondents reported here considered audio and visual problems to be fairly minor (median scores of 3 or less on a scale of 0 to 10). The most notable benefit to a virtual format compared to in‐person meetings is the enhanced accessibility and ability to participate for providers with varied schedules and work locations. This particular benefit has been highlighted by several prior studies from the COVID‐19 era, , and when combined with the early literature describing the development of virtual tumor boards for exactly this purpose, , , we believe that this aspect of virtual meetings is likely one of the most durable benefits to both patients and providers if virtual meetings are maintained after COVID‐related precautions have receded.

While the results from our cohort suggest overall equivalence or superiority of a virtual platform over in‐person meetings, it is important to note that there are still several challenges that need to be addressed. The problem noted to be the most significant among the responders reported here was the decreased opportunity to network with colleagues when compared to the traditional face‐to‐face format. This is not unique to the current cohort, as a study by Rajasekaran et al. noted that of the seven different aspects of virtual meetings they queried, “Interaction with specialists” received the lowest score, and several respondents noted that in‐person communication was “grossly missed” when using a virtual format. When considering how to best improve virtual meetings moving forward, we believe that our subgroup analyses by respondent's role provides valuable insights. Particularly in regard to the issue of decreased opportunity for networking, this issue was most significant to surgeons and non‐surgeon physicians. Knowing this, we recommend piloting an alternative, but less frequent, in‐person meeting for these individuals in particular to provide the necessary forum for building inter‐personal relationships and camaraderie while being mindful not to overburden providers' schedules with too many additional meetings. Other potential venues including journal clubs or department or section meetings, can additionally supplement the virtual tumor board experience. It is interesting to note that in this cohort, just over 50% of respondents agreed (either somewhat or strongly) to the statement, “Despite modifications, virtual tumor boards just aren't as good as in‐person tumor boards.” This was surprising, as none of the ten domains of tumor board meetings specifically queried showed a strong preference for an in‐person format. Therefore, we postulate that this response may be the result of providers' initial frustrations with having to make a significant unexpected change to their workflow, the learning curve associated with rapidly implementing new imperfect technologies, or the emerging phenomenon known as “Zoom fatigue” which describes the mental exhaustion that accompanies extended periods of time spent in virtual meetings.

As one prior study reported, nearly 78% of respondents felt that virtual meetings are the future of cancer care and more than 90% felt that international virtual tumor boards are on the horizon. To maximize the benefits of continuing to use a virtual platform, we believe the feedback from the cohort described here highlights the importance of focusing on several key issues. When reviewing respondents' comments in the “free text” section of the survey, strong meeting leadership and organization were the most commonly mentioned keys to a successful virtual tumor board meeting. Institutions must work to ensure that all providers, regardless of their location, have the necessary technology infrastructure to be able to fully participate and minimize technology‐associated frustrations. As technological challenges are minimized, it is also critical that meeting leadership reinforce that engaged participation from all group members during the entirety of the session is critical to maintain meeting quality and patient care outcomes. Lastly, all providers must ensure that they are using virtual platforms that provide the encryption necessary to protect patient information and prevent security breaches (Table 2). Whenever implementing changes to established systems, it is of utmost importance to track outcomes and participants' feedback over time. Therefore, similar to the findings of the Head and Neck oncology group at University of Pittsburgh, we encourage the continued study of both patient outcomes and providers' experiences with virtual tumor board meetings over the coming months and years to determine if further changes are needed to maximize the benefits of this resource.

Table 2

Recommendations for a maximizing participant satisfaction and patient care utilizing virtual tumor boards

1. Ensure that the virtual platform software being used and all participants are using HIPPA‐compliant encryption on all devices

2. Prioritize consistent leadership and organization to create efficient meetings

3. Encourage active preparation and participation from all specialties and attendees

4. Remind participants to minimize “off‐screen” multi‐tasking and distractions

5. Maintain and upgrade technology infrastructure for easy participation across all locations

6. Offer participants training on the various features (e.g., screen sharing, mute/unmute, etc.) of the virtual platform software to minimize challenges during the meeting

7. Provide additional educational venues for physicians to create opportunities for networking and to foster inter‐departmental relationships

Because this project was initiated as a quality improvement study, it was not powered to detect any specific findings and is therefore limited in the extent to which strong conclusions can be drawn. Additionally, we were not able to achieve a high response rate; however, given that our final sample size is fivefold higher than all other available reports to date, we feel that this cohort provides valuable insights. Furthermore, this study is the first to incorporate feedback from numerous specialties and provider roles, allowing our findings to be more broadly applicable to other groups and institutions. Due to the need to protect respondents' anonymity, we limited the amount of biographical information that was collected, which may have prevented the reporting of other potentially insightful subgroup analyses.

CONCLUSION

Virtual platforms for tumor board meetings are likely here to stay, as they are equal to or preferred to in‐person meetings across almost all queried domains. This was largely independent of training status or role of the participants. Additions to the virtual tumor board that might further enhance acceptance include improved organization and meeting leadership, investment in technology infrastructure to ensure easy participation across all locations, and the use of other educational programming for physicians to maintain networking opportunities. Finally, institutions need to continue to monitor both patient outcomes and providers' experiences using this virtual format to ensure the highest quality of care and caregiver engagement.

FUNDING INFOMATION

FUNDING INFOMATION is not available.

CONFLICT OF INTERESTS

The authors declare that there are no conflict of interests.

SYNOPSIS

Virtual tumor boards were considered to be equivalent to in‐person meetings across 9 of 10 domains queried, while a virtual format was preferred for participation in off‐site tumor boards. Consistent organization and engaged participation of all attendees is critical for successful virtual meetings. Creating alternative opportunities for inter‐departmental networking would also be beneficial.