Implementing a telemedicine curriculum for internal medicine residents during a pandemic: the Cleveland Clinic experience.

Telemedicine training was not a substantial element of most residency programmes prior to the COVID-19 pandemic. Social distancing measures changed this. The Cleveland Clinic Internal Medicine Residency Programme (IMRP) is one of the largest programmes in the USA, which made the task of implementing a telemedicine curriculum more complex. Here we describe our experience implementing an effective, expedited telemedicine curriculum for our ambulatory resident clinics. This study was started in April 2020 when we implemented a resident-led curriculum and training programme for providing ambulatory telemedicine care. The curriculum was finalised in less than 5 weeks. It entailed introducing a formal training programme for residents, creating a resource guide for different video communication tools and training preceptors to safely supervise care in this new paradigm. Residents were surveyed before the curriculum to assess prior experience with telemedicine, and then afterward to assess the curriculum's effectiveness. We also created a mini-CEX assessment for residents to solicit feedback on their performance during virtual appointments. Over 2000 virtual visits were performed by residents in a span of 10 weeks. Of 148 residents, 38% responded to the pre-participation survey. A majority had no prior telemedicine experience and expressed only slight comfort with the modality. Through collaboration with experienced residents and faculty, we expeditiously deployed an enhancement to our ambulatory care curriculum to teach residents how to provide virtual care and help faculty with supervision. We share our insights on this experience for other residency programmes to use. © Author(s) (or their employer(s)) 2022. No commercial re-use. See rights and permissions. Published by BMJ.

Purpose of study

The first documented cases of COVID-19 in Ohio occurred on 9 March 2020.1 One day later, the Cleveland Clinic’s leadership announced that all in-person meetings were cancelled and ambulatory patient visits would be virtual. This was a major paradigm shift for our Internal Medicine Residency Programme (IMRP), where one-third of training time is spent in ambulatory clinics. Teaching residents at one of the largest programmes in the country to provide virtual visits on short notice was an enormous endeavour. In addition, we needed to make sure that all faculty had the tools they needed to precept residents in a completely virtual format.

A cohort of Cleveland Clinic faculty physicians have been performing virtual patient visits since 2014 using the American Well platform.2 They conducted 41 000 virtual visits in 2019 and over 100 000 such visits by February 2020.3 Residents, however, had not been trained in delivering virtual care prior to the COVID-19 pandemic. In any given week, our IMRP residents complete around 840 ambulatory patient encounters. The avoidance of in-person visits for routine care as a result of social distancing measures would have deprived our community of considerable primary care, and it would have limited learning opportunities for residents.

Our IMRP’s clinician-educator track (CET) residents began building a telemedicine curriculum before the start of the pandemic. The original plan was for a slow, stepwise rollout of the curriculum with the new interns in fall 2020. However, the urgent need created by the pandemic called for an accelerated implementation of this nascent telemedicine curriculum.

Study design

The Accreditation Council for Graduate Medical Education (ACGME) had planned to integrate telemedicine into the Common Program Requirements in July 2020. Considering the pandemic, precepted telemedicine encounters were permissible immediately.4 Our telemedicine curriculum allowed us to quickly train residents and preceptors to serve our community with virtual visits. We first assessed resident confidence in providing telemedicine care before the curriculum started in April 2020. In June and July 2020, we surveyed residents regarding the effectiveness of the curriculum. These surveys were determined to be exempt by the Cleveland Clinic Institutional Review Board (IRB).

Our ambulatory sites accommodate 148 residents and vary in size and patient populations. Residents function as primary care physicians during their clinic week (4+1 model) in nine different clinic sites. Implementing this programme involved preparing and standardising clinic workflow documents for residents and a precepting guide for faculty across the different ambulatory sites (online supplemental appendix A). We created an orientation presentation to standardise virtual visits inspired by external resources from the American College of Physicians (ACP), the Ohio State Medical Association (OSMA) and the ACGME. Telemedicine training for residents was conducted virtually by Zoom, a practice that has been widely used at other institutions to implement virtual curricula.5

Trainee pre-participation and post-participation surveys

Residents were surveyed before and after the telemedicine curriculum. The pre-participation survey was distributed by email starting in early April 2020 and asked about prior telemedicine experience and comfort level in providing telemedicine care (online supplemental appendix B). The post-participation survey was sent by email in June and July 2020 at the end of the academic year. It asked for feedback on resident satisfaction with and benefit from the telemedicine curriculum (online supplemental appendix C). Surveys were anonymous and not matched.

Communication competencies

We considered it paramount to teach our residents how to communicate and perform remote examinations with virtual tools, which require unique communication skills.6 Residents watched a recording of a simulated encounter prior to their first real encounter (online supplemental appendix D). This was done at the start of each ambulatory clinic week for all residents in a cohort (approximately 30 physicians) via Zoom and then was repeated for 5 weeks to reach all trainees. We demonstrated potential pitfalls of a virtual encounter (from technology-specific issues such as audio or connectivity problems, to challenges in virtual verbal communication) and paused periodically for discussion. This gave residents an opportunity to share their impressions with the group and brainstorm possible solutions. The Cleveland Clinic Communication Tips for Virtual Visits (online supplemental appendix E) was used to teach resident essential skills (eg, conveying respect, setting an agenda and providing reflective listening using non-verbal communication). This tip sheet was developed based on the Relationship: Establishment, Development, and Engagement (REDE) model, a validated model for teaching healthcare communication.7 The presentation also reviewed practical tips such as looking at the camera instead of the screen to provide the appearance of direct eye contact. Initially, virtual visits were conducted from dedicated patient rooms on campus, but eventually many visits were conducted remotely to provide appropriate social distancing for trainees and staff. Medical decisions were always discussed with and approved by attending physician preceptors.

Virtual physical examination competency

Physical examinations in telemedicine must be augmented to fit the virtual nature of the encounter. Two inherent challenges with virtual visits are the physician’s ability to describe the examination he/she needs, and the patient’s ability to perform the examination on video. In our programme, we taught the virtual physical examination with Zoom. We used problem-based questions and interactive cases, addressing symptoms ranging from ankle pain to vision changes. During these scenarios, we emphasised creativity. For example, we showed residents how to teach a patient the Ottawa ankle rules to address ankle pain (online supplemental appendix F), thus saving them a visit to the emergency department.

Telemedicine technology platform competency

Virtual encounters were initially offered using a variety of platforms including American Well, Doximity, Skype and Apple FaceTime. We created a resource slide deck with practical tips on how to install and use each of those platforms (online supplemental appendix G). One month later, our institution started using a secure version of Zoom integrated into our Epic electronic medical record (EMR) to streamline virtual visits. Our slide deck was updated to incorporate this new platform. Residents also viewed an institutional video that walked them through using Zoom within Epic for virtual visits.

Teaching documentation of virtual encounters

Residents and preceptors were taught how to appropriately document virtual encounters. New documentation elements to the standard ambulatory note included a statement that the encounter was conducted by phone or video, an estimate of the time it took to provide care and a description of the findings of the virtual physical examination, if performed. Preceptors also needed to attest that the visit was provided either by phone or video, and indicate whether they were present for a portion of the encounter. These documentation changes were facilitated with new templated phrases in Epic that all residents and staff physicians could use.

Virtual encounter formative trainee evaluation

We created a formal evaluation process to help residents improve their practice. Residents were still required to precept their patient care with a staff physician. The staff physician could join the encounter at any time, especially during the visit wrap-up where the resident covers the plan with the patient and answers questions. Residents were also able to get direct observation and feedback during collection of the patient’s history and their performance of the physical examination in the virtual setting. This mirrored the process for in-person visits.

We created a mini-clinical evaluation exercise (mini-CEX) for telemedicine centred around the ACGME core competencies8 (online supplemental appendix H). Each resident was encouraged to send an electronic mini-CEX survey to faculty to get actionable feedback on their virtual visit, preferably before the encounter. Mini-CEX evaluations were geared to evaluate for the nuances of the virtual encounter. Residents in our programme are required to complete 12 mini-CEXs per year, and a telemedicine mini-CEX may be included.

Precepting the virtual encounter

Dynamic changes in recommendations for trainee virtual visits by the ACGME and billing and supervision rules from the Centers for Medicare & Medicaid Services (CMS) made regular updates for virtual care preceptors paramount. Our precepting team clinic leaders collaborated to identify champions for maintaining compliance with ACGME and CMS rules and providing as-needed updates to all faculty, residents and support staff. This incorporated changes to workflow for encounter consent and the location of preceptors (direct or indirect supervision). It also entailed identification of the types of encounters where the CMS primary care exception,9 stipulating that the preceptor does not need to interact with the patient, could be used.

Results

Our team implemented the telemedicine curriculum within 4 weeks. All 148 ambulatory residents in our IMRP were trained in performing virtual visits and they completed more than 2000 such visits between early April and mid-June 2020. In total, 56 residents completed the orientation pre-participation survey with a total response rate of 38%. Forty-three residents completed the post-participation survey, with a total response rate of 29%. Of residents responding on the pre-participation survey, 76.8% had no prior telemedicine experience and indicated limited comfort with the modality (table 1). The post-participation survey assessed the residents’ perception of the quality of this curriculum in teaching telemedicine skills. Of these respondents, 46.5% were third-year residents (n=20) and 67.4% (n=29) reported learning new knowledge about telemedicine because of the curriculum that was offered (table 2). Residents selected strongly agree or agree to each of the eight statements on the post-participation by over 60% per question (table 2). Free response survey answers provided both positive and growth-oriented feedback. In regard to the content, many residents felt that it helped bridge the skill gap created by the pandemic. One respondent wrote “It was very helpful as I had no formal training in telemedicine before this time and I had to do virtual visits in my Resident PCP clinic during COVID-19.” Regarding the delivery of the content, some respondents wrote that the classroom course was ‘not interactive enough’ or that ‘too much of the presentation was obvious or common sense’. As of 1 December 2020, 62 of the 76 virtual visit/telemedicine mini-CEXs sent to preceptors have been completed.

Table 1

Resident telemedicine orientation pre-participation survey responses (n=56)

	PGY1n=26	PGY2n=19	PGY3n=11	All PGYn=56
Respondents with prior telehealth experience
None to date	73.1%	73.7%	90.9%	76.8%
Demonstration by attending	23.1%	26.3%	9.1%	21.4%
Medical School	3.8%	0%	0%	1.8%
Other	0%	0%	0%	0%
Respondents’ view of benefits of telemedicine*
Improved patient outcomes	23.1%	15.8%	27.3%	21.4%
Stronger physician–patient relationship	42.3%	42.1%	27.3%	39.3%
Access to data (eg, home measurements) or resources (eg, consultants)	46.2%	42.1%	36.4%	42.9%
Improved efficiency	61.5%	68.4%	54.5%	62.5%
Better access for patients	69.2%	89.5%	90.9%	80.4%
Other:	3.8%	10.5%	0%	5.4%
Better understanding of home environment and medications Access to specialists in remote areas
Respondents’ view of barriers to using telemedicine*
Providers lack of familiarity with the technology	34.6%	73.7%	72.7%	55.4%
Weaker physician–patient relationship	42.3%	47.4%	54.5%	46.4%
Too difficult for my patients to use	53.8%	63.2%	81.8%	62.5%
Lack of time to implement care	23.1%	0%	9.1%	12.5%
Privacy concerns	19.2%	10.5%	27.3%	17.9%
Reimbursement concerns	23.1%	52.6%	45.5%	37.5%
Documentation concerns	34.6%	26.3%	18.2%	28.6%
Lack of full physical examination	11.5%	10.5%	9.1%	10.7%
Other:	0%	0%	9.1%	1.8%
Challenging to connect with patients, especially for a first visit
Respondents’ comfort with conducting telehealth visits
Very comfortable	11.5%	5.3%	0%	7.1%
Comfortable	11.5%	36.8%	27.3%	23.2%
Neutral	42.3%	31.6%	45.5%	39.3%
Uncomfortable	34.6%	26.3%	18.2%	28.6%
Very uncomfortable	0.0%	0.0%	9.1%	1.8%
Respondents who expect to integrate telemedicine into future practice
Strongly agree	26.9%	47.4%	36.4%	35.7%
Agree	38.5%	31.6%	36.4%	35.7%
Neutral	23.1%	15.8%	9.1%	17.9%
Disagree	11.5%	5.3%	9.1%	8.9%
Strongly disagree	0%	0%	9.1%	1.8%

*Multiple responses permitted per respondent.

Table 2

Resident post-participation survey responses 12 weeks after starting the telemedicine curriculum (n=43)

	PGY1n=11	PGY2n=12	PGY3n=20	All PGYn=43
Overall, I was satisfied with this learning activity.
Strongly agree	9.1%	16.7%	40.0%	25.6%
Agree	18.2%	75.0%	40.0%	44.2%
Neutral	54.5%	8.3%	15.0%	23.3%
Disagree	9.1%	0%	5.0%	5.7%
Strongly disagree	9.1%	0%	0%	2.3%
I would recommend this learning activity to others.
Strongly agree	9.1%	16.7%	45.0%	27.9%
Agree	18.2%	75.0%	40.0%	44.2%
Neutral	54.5%	8.3%	10.0%	20.9%
Disagree	9.1%	0%	5.0%	4.7%
Strongly disagree	9.1%	0%	0%	2.3%
The learning activities and/or materials were effective in helping me learn the content.
Strongly agree	9.1%	16.7%	30.0%	20.9%
Agree	45.5%	75.0%	50.0%	55.8%
Neutral	27.3%	8.3%	20.0%	18.6%
Disagree	18.2%	0%	0%	4.7%
Strongly disagree	0%	0%	0%	0%
I learnt new knowledge and skills from the telemedicine curriculum.
Strongly agree	9.1%	8.3%	30.0%	18.6%
Agree	45.5%	75.0%	35.0%	48.8%
Neutral	27.3%	16.7%	35.0%	27.9%
Disagree	9.1%	0%	0%	2.3%
Strongly disagree	9.1%	0%	0%	2.3%
I will be able to apply the knowledge and skills learnt for future encounters.
Strongly agree	9.1%	16.7%	50.0%	30.2%
Agree	72.7%	75.0%	30.0%	53.5%
Neutral	0%	8.3%	20.0%	11.6%
Disagree	9.1%	0%	0%	2.3%
Strongly disagree	9.1%	0%	0%	2.3%
The scope of the material was appropriate for my needs.
Strongly agree	9.1%	16.7%	40.0%	25.6%
Agree	54.5%	75.0%	45.0%	55.8%
Neutral	18.2%	8.3%	10.0%	11.6%
Disagree	9.1%	0%	5.0%	4.7%
Strongly disagree	9.1%	0%	0%	2.3%
I found the material in this learning activity to be relevant and up-to-date.
Strongly agree	18.2%	25.0%	45.0%	32.6%
Agree	45.5%	75.0%	40.0%	51.2%
Neutral	27.3%	0%	15.0%	14.0%
Disagree	0%	0%	0%	0%
Strongly disagree	9.1%	0%	0%	2.3%
The content was relevant to my job-related needs.
Strongly agree	27.3%	25.0%	40.0%	32.6%
Agree	45.5%	75.0%	50.0%	55.8%
Neutral	9.1%	0%	10.0%	7.0%
Disagree	18.2%	0%	0%	4.7%
Strongly disagree	0%	0%	0%	0%

Discussion

Our experience demonstrates that teaching and precepting telemedicine in resident ambulatory clinics is feasible. Implementing any type of curriculum or training programme requires extensive preparation, involvement of multiple stakeholders and competing time constraints. The urgency of the COVID-19 pandemic and the resulting changes to healthcare delivery allowed us to move this process along quickly within a major tertiary medical centre, at one of the largest internal medicine residency programmes in the USA, and at an institution where faculty had significant prior experience providing telemedicine care before the pandemic.3 In addition, several authors (DJS, OG, BEM and ADS) were residents with a significant leadership role from the beginning. They were designing this curriculum before the pandemic and they collaborated with the IMRP ambulatory team to expedite its launch. These residents created the telemedicine training video (online supplemental appendix D), the telemedicine troubleshooting guide (online supplemental appendix G) and the first iteration of the PowerPoint slides for the telemedicine didactic course. They also helped design the pre-participation and post-participation surveys (online supplemental appendices B and C), and the mini-CEX form for residents to solicit faculty feedback on telemedicine encounters (online supplemental appendix H). They attended many of the virtual didactic courses to serve as facilitators and answer questions from their peers. Their leadership helped to accelerate the pace of implementation of the Cleveland Clinic IMRP telemedicine curriculum to meet an essential need precipitated by the pandemic.

Zoom was the modality for teaching the introductory telemedicine course due to social distancing needs, but other mainstream tools like Microsoft Teams, used by other programmes,5 would have also worked. We did not require advanced features such as topic-based threads or secure messaging, adding flexibility in the choice of a video platform. Trainings were led by a chief resident for groups of 30 learners on average, as this is the number of residents in each ambulatory week at our programme. In the future, this small size may permit the use of breakout rooms to simulate encounters and provide feedback. Recently, Sartori and colleagues demonstrated the use of an Objective Structured Clinical Examination (OSCE) to assess residents’ telemedicine-specific skills,10 and this would be a natural next step for our programme. We plan to build on this study to train more faculty to become facilitators for the simulation of the telemedicine ambulatory clinic encounters in the future.

In many regards, the process for performing, precepting and documenting a virtual visit parallels a traditional encounter. Our introductory course therefore focused on video communication skills, the virtual physical examination, documentation changes and technology troubleshooting to help build resident competency in this new framework. Many recent articles detail best practices for the virtual physical examination,11 and the neurological12 and musculoskeletal13 14 examinations in particular. One study showed non-inferiority of the virtual shoulder examination compared with the traditional ‘in person’ examination.15 In our experience, ailments that could be demonstrated by visual inspection were easier to assess in a virtual physical examination than problems of the heart, lung or gastrointestinal tract that may require a more detailed examination with auscultation, percussion and/or palpation. Another unexpected challenge was that it was cumbersome at first for residents to identify a video platform that would work for each patient. Early on, patients were more likely to adopt video calls when we offered a wide variety of choices. Our institution’s subsequent rollout of an Epic EMR update with embedded Zoom video made visual appointments easier and more preferred by patients.

To evaluate learner response to our curriculum, we used pre-participation and post-participation surveys modelled after the work of others in creating a longitudinal telemedicine curriculum.16 The pre-participation results indicated that a majority of residents had not had prior telemedicine experience, but that they felt it was an important skill to learn and it would be part of their future practice (table 1). Post-participation survey results indicated that this curriculum was perceived as effective at teaching telemedicine skills that could be applied immediately to serve patients (table 2). Free response answers indicated that while the information conveyed was beneficial, the presentation format should be reworked to make it more interactive. This feedback has already been used to restructure and improve the telemedicine curriculum for 2020–2021.

Now that our residents have had at least three ambulatory weeks of telemedicine encounters, our next step will be to assess their self-reported competency in providing telemedicine care by phone or video. Since most residents from the 2019–2020 academic year did not have prior telemedicine experience, it will be helpful to understand how residents’ skillsets have changed because of this curriculum and hands-on experience in clinic. We also plan to solicit preceptor feedback on the training programme. In addition, residents are now requesting formal objective feedback from their ambulatory preceptors on their performance with quarterly summative evaluations and on-demand mini-CEX evaluations specific to virtual visits. These feedback tools will help learners improve and provide a further assessment of this curriculum. The new interns in 2020–2021 will provide in-person visits exclusively for the first half of the year, and then will have at least one telemedicine slot per ambulatory week thereafter. We believe this will allow the newest trainees time to better understand the workflow at our institution and the tools available to serve their patients in a hands-on, supervised environment before those interns begin performing telemedicine encounters. They will also receive the Introduction to Telemedicine course that was offered to all residents at the start of the pandemic.

Limitations

Our limitations were a low survey response rate and variable faculty experience with precepting telemedicine. The response rate for the pre-participation survey was higher than the post-participation survey, but the response rate on both instruments was low. This is likely because of the accelerated pace that this curriculum was rolled out to meet the needs of our learners and patients. We had the most success with getting pre-participation surveys completed by providing time at the start of the virtual didactic course for residents to fill them out with their phones. We also kept the number of survey questions short to encourage participation. It was even more challenging to get learner post-participation feedback because of their busy work schedules and the ongoing pandemic. The post-participation survey response rate was also impacted by timing. It was distributed at the end of the academic year when many residents were preparing to graduate or advance to the next training year.

Lack of staff physician experience with precepting telemedicine encounters was also a limitation early on, but this improved with time, practice and the creation of a faculty preceptor guide (online supplemental appendix A). Since preceptors supervise resident care, the biggest change in workflow has been at the end of each encounter. If a resident wants the preceptor to examine the patient or be available on the line to listen to the wrap-up with the patient, the preceptor needs to either join the resident in person or connect to the encounter by three-way video or phone. This was something that our faculty mentors and learners adapted to quickly, and this would be a very manageable barrier within any other residency programme, especially those of smaller size. Despite challenges in assessment and ambulatory precepting, this curriculum has served the needs of our residents well, and it will be a sustainable element of our training programme.

Our experience with implementing a comprehensive telemedicine curriculum for internal medicine residents and their preceptors is notable for its quick turnaround time in response to the COVID-19 pandemic, its breadth in reaching 148 trainees and 62 preceptors at nine clinic sites, and for its involvement of resident leaders for implementation. Having resident champions significantly broadened the scope of this project and helped create a support network for their peers so that the first iteration of the telemedicine curriculum was a success. We hope our experience can guide other programmes in developing a lasting telemedicine curriculum that prepares physicians-in-training for a practice landscape that includes virtual visits.

Prior to March 2020, telemedicine training for residents and fellows was limited, even though telemedicine was quickly emerging as a widespread tool for providing care.

Telemedicine curricula for residents and fellows are quite new, but their development was accelerated by the COVID-19 pandemic.

A comprehensive curriculum for telemedicine can be expeditiously and effectively implemented at a large multisite residency programme.

Trainee leadership in building a curriculum can accelerate implementation and increase buy-in from both faculty and peers.