Telemedicine Training in the COVID Era: Revamping a Routine OSCE to Prepare Medicine Residents for Virtual Care.

BACKGROUND: During the rapid onset of the pandemic, clinicians transitioned from traditional outpatient practice to virtual modalities for providing routine care to patient panels. Like training programs nationwide, telemedicine training and assessment had not been systematically incorporated into our residency. In response, a scheduled Internal Medicine (IM) Objective Structured Clinical Examination (OSCE) was adapted to a remote modality to become virtual care-focused learning experience for trainees and to provide valuable feedback to educators.
METHODS: Standardized Patients (SPs) rated residents on their communication (including information gathering, relationship development and patient education), patient activation and satisfaction, and telemedicine skills. Analyses included a comparison of domain scores for residents who participated in both the 2020 remote and 2019 in-person OSCEs, and a review of written resident comments about the virtual OSCE.
RESULTS: During 2020's video visit OSCE (VOSCE), residents (n = 23) excelled at nonverbal communication but struggled with virtual physical exams and information gathering. In debrief, residents expressed substantial interest in more opportunity to practice virtual visit skills going forward. In comparing scores of the virtual care (2020) OSCE with the in-person (2019) version, the small subset of residents who participated in both assessments (n = 9) performed similarly on communication skills, patient satisfaction and activation. Patient education scores were significantly lower during the virtual care OSCE (P = .008).
CONCLUSION: Our reformulated OSCE accomplished 3 goals including; (1) physically distancing residents from SPs per COVID regulations, (2) providing residents with the opportunity to practice critical virtual visit skills, and (3) alerting our educators to curricular improvement areas. Our methods are useful for other institutions and have applications to the larger medical education community.

Introduction

The rapid onset of COVID-19 forced primary care clinicians to transition from traditional outpatient practice to telemedicine (ie, virtual video visits), both to triage potential COVID-positive patients and provide routine care. Like programs nationwide, telemedicine and virtual care provision training had not been systematically incorporated into our residency. In response to this gap and as mandated by social distancing recommendations during the emerging pandemic, our scheduled Internal Medicine (IM) objective structured clinical examination (OSCE) was adapted to take place over a remote, video modality in order to provide a virtual care-emphasized learning experience. The goal of this video visit OSCE (VOSCE) was to teach skills on best practices for conducting virtual visits and to provide educational leadership with valuable information on areas for curricular improvement.

The definition of telemedicine has evolved. Telemedicine is now defined as broadly incorporating the delivery of healthcare services and education from 1 geographical space to a separate one. Virtual clinical encounters, during which a clinical care provider and patient interact over a video tool, fall under the larger umbrella of telemedicine. Telemedicine communication curriculum and learner assessment has not been a big focus in medical education until recently. We developed a video telemedicine OSCE rating checklist using focus groups and iterative feedback with experienced telemedicine practitioners, along with live observation of virtual clinic encounters. Skills described in our assessment tool reflect practice guidelines established by the American Telemedicine Association and used in other specialties’ OSCEs.[4,5] Observation of video clinical encounters has shown that some communication behaviors, including expression of empathy and facilitation of medical history gathering, occur less often in televisits than during in-person encounters. Research is needed to fully understand factors contributing to satisfaction, patient-centered care, and practice adoption during virtual visits, not to mention the systemization and dissemination of successful physician training models.

For the purposes of 2020s VOSCE, we utilized a web-based conference platform to simulate a virtual visit in order to assess a specific set of telemedicine skills. The goal of this standardized assessment was to understand resident ability to engage in virtual visit-based care provision. Further, we sought to better identify which skills are necessary for virtual visit-specific curricula in an effort to maximize opportunity for successful adoption of this burgeoning modality by our residency and health system.

Methods

Our new VOSCE included 5 stations from our traditional in-person OSCE, 4 of which utilized video conference software to simulate the remote care environment and are reported in the current study (Table 1). One VOSCE case featured a standardized patient (SP) presenting with COVID-19 symptoms and was designed to assess our learner’s telemedicine-specific skills. Residents participated in 3 additional virtual care cases; (1) educating a patient on buprenorphine maintenance, (2) counselling an elderly patient on mammogram screening, and (3) addressing a patient’s frustration with documentation in their electronic health record. Cases offered virtual visit practice in common primary care scenarios and an opportunity to assess trainees’ remote physical examination and history-taking, prescribing, and communication and patient-centered care skills.

Table 1.

Virtual OSCE cases with standardized patient (SP) demographics.

Case information	Upper respiratory infection (URI)	Buprenorphine (bupe) maintenance	Mammography screening	Electronic medical record (EMR)
SP age (gender)	52 (male)	38 (male)	67 (female)	67 (female)
Chief complaint/SP concern	Nagging cough with temperature of 100.8 and COVID concern	Bi-weekly bupe and urine tox check-up	Check-up and referral for annual mammogram	Concern over problem list documentation error
Challenge to resident	Take focused history of URI symptoms	Interpret toxicology results	Assess risk and family history of patient	Identify and acknowledge the concern
	Perform virtual physical exam	Assess history of bupe use	Appropriately counsel against screening	Avoid blame and manage difficult situation
	Determine management plan for COVID possibility	Determine follow-up and management plan	Educate patient on alternatives to screening	Educate patient on benefits of technology
Technology interface	WebEx	WebEx	WebEx	WebEx

During the VOSCE, experts at our simulation center set up video conferencing materials in each room. Residents read the door note’s instructions, entered the exam room and signed on to the conferencing platform, signifying the start of the virtual clinical encounter. SPs, who signed on to the virtual encounter at the same time as residents, were stationed in different rooms in an effort to ensure social distancing per emerging regulation. Residents had 10 minutes to complete a virtual visit with the SP, which included taking a complete history, assessing the patient’s condition, and initiating next steps. Faculty members observed performance through the video conferencing system. Upon completion, the SP completed an extensive behaviorally-anchored checklist (3 point: not done, partly done, well done) on resident performance.

SPs rated residents on their core communication skills (including information gathering, relationship development, and patient education and counseling), patient activation and satisfaction, and our case-specific telemedicine items. Domain summary scores were calculated as mean percent rated “Well done.” Residents participated in a virtual group debrief at the end of the VOSCE and completed a post-visit questionnaire, consisting of Likert and open-ended questions on their experience and comfort in handling cases during the VOSCE. Analyses included a paired Wilcoxen signed-rank comparison of domain scores for a small subset of residents who participated in both the 2020 remote and 2019 in-person OSCEs, and a review of written resident comments about the VOSCE. Individual resident consent was not required as this project qualified as a curricular quality improvement project through the NYU Institutional Review Board (IRB).

Results

23/25 possible residents (92%) participated in 2020’s VOSCE, and 9 residents participated in both the 2019 and 2020 OSCE. Fewer than half (46%) of 2020’s residents (n = 23) performed well on the telemedicine skill domain (Table 2). Residents excelled in using nonverbal communication to enrich on-camera communication (100%), but struggled with virtual physical exams (13%), gathering information (4%), and optimizing technology (4%).

Table 2.

2020’s Telemedicine scores and comparison of virtual versus in-person domain scores.

Domain	Checklist item	Frequency of each item, % (n)			Behavioral descriptor of well done
		Not done	Partly done	Well done
Telemedicine items and scores for 2020’s virtual OSCE (N = 23)	Confirmed patient identifiers	61 (14)	39 (9)	0 (0)	Asked patient. to confirm name/DOB, call-back number & location
	Used non-verbal communication to enrich on-camera communication	0 (0)	0 (0)	100 (23)	Maintained eye contact with webcam throughout encounter, sat squarely in front of camera, at an appropriate distance
	Actively optimized technical aspects of the virtual encounter	91.3 (21)	4.3 (1)	4.3 (1)	Assessed sound quality, video quality, and backup plan if audio/video failed
	Exhibited comfort and confidence using video interface	0 (0)	0 (0)	100 (23)	Confident on camera, acknowledged and moved forward from technical glitches, and did not let video interface detract from natural conversation
	Utilized live video to augment information gathering	83 (19)	13 (3)	4 (1)	Did 2 or more of the following: Visually reconcile medications, witness reproducible symptoms, talk with onsite collateral
	Partnered with patient to perform physical exam	74 (17)	13 (3)	13 (3)	Asked patient to perform maneuvers or access peripheral monitoring device followed by verbal confirmation of findings with patient or collateral.
	Maintained appropriate computer etiquette during encounter	0 (0)	0 (0)	100 (23)	Paused video or provided clear explanation while documenting, searching another website, or having another screen open for the purpose of patient care
	Domain	2019 (in-person)			2020 (virtual)		P-value (Wilcoxon)
		Mean % well done		Range (SD)	Mean % well done	Range (SD)
Comparison of scores for learners participating in both OSCEs (N = 9)	Communication: Information gathering	83		0.70-0.93 (0.07)	84	0.70-0.90 (0.06)	.88
	Communication: Relationship development	94		0.84-1.00 (0.04)	93	0.86-1.00 (0.05)	.85
	Communication: Patient education	76		0.64-0.85 (0.07)	40	0.26-0.47 (0.07)	.008
	Patient satisfaction	80		0.64-0.92 (0.08)	72	0.63-0.88 (0.08)	.17
	Patient activation	66		0.36-0.86 (0.13)	65	0.57-0.79 (0.08)	.95
	Global Recommendation Scores (mean score of scaled 0-3 rating per case)
	SP willingness to recommend resident to others based on their communication	2.2		1.75-2.5 (0.23)	2	1.75-2.50 (0.24)	.3
	SP willingness to recommend resident to others based on their professionalism	2.5		2.33-2.78	2.5	2.25-2.78 (0.17)	.31

In review of results from the exit survey on attitudes and self-efficacy, few residents reported high confidence in their ability to conduct virtual visits before the VOSCE (25%), but a majority (80%) felt confident or very confident in their virtual visit abilities afterwards. Among VOSCE feedback, residents commented it was “so helpful to have just-in-time training for telemedicine,” and “[the] telehealth and COVID focus was necessary.”

In comparing data from the 2019 in-person to 2020’s virtual OSCEs, residents who participated in both assessments (n = 9) demonstrated consistent performance across most of the communication domains (Table 2). Satisfaction scores were marginally lower in the VOSCE when compared to the in-person. Performance in the patient education domain was significantly lower in the VOSCE (P = .008).

Discussion

We sought to better understand our how residents approach a virtual visit and the learning impact of a remote OSCE in order to identify areas for curricular improvement. In comparing mean scores between our in-person and socially distanced VOSCE, we noted that most resident physician communication skills transfer from physical clinic visits to virtual clinical encounters. However, patient education appears to be more difficult in remote visits, drawing attention to necessary curricular focus as telemedicine adoption in primary care increases. Historically physicians have used telephone calls as a remote panel management tool, and our results demonstrate that they have the skills to communicate clinical basics remotely, but until recently have not been expected, or trained, to deliver patient-centered care on video.

Patient education, activation, and satisfaction skills are essential for patient-centered care, with research in this area suggesting that relationships exist between these skills and longer-term care and health. While residents in our program excel at aspects of communication, patient education skills are often rated lower during routine OSCEs and was noted even lower in virtual visit simulation. We hypothesize that the video visit environment introduces additional barriers to incorporating patient education-specific skills such as diagram drawing/sharing, teach back with audio delay and time management. Our results leave room for future study. Further, trainees consistently struggled with core telemedicine skills like optimizing technology (sound, screen visualization) and virtual information gathering augmentation. Increasing exposure to best practices in patient-centered care while also providing opportunity to practice those skills in simulation is relevant not only to our training program, but to other institutions looking to introduce telemedicine curricula as well.

Post program survey results indicated a substantially higher degree of perceived self-efficacy in conducting virtual visits following participation in the VOSCE compared to prior to participation. Since the initial VOSCE, residents have been given additional materials on best practices for virtual visits that include training guides on video-based communication skills and YouTube tutorials covering virtual physical exams. Our VOSCE findings have helped us develop and implement a curricula on virtual care best practices which will be assessed in subsequent OSCEs. Further, resident feedback following 2020’s inaugural VOSCE indicated desire for a higher ratio of virtual clinical encounters in future simulation and assessment. We will continue to integrate telemedicine training and practice throughout the residency program in alignment with emerging standards, and prime residents for lifelong skill development as technology develops and virtual care becomes integrated in to all practices.

As the medical landscape shifted overnight, our VOSCE was implemented to empower trainees facing the pandemic. This performance-based competency assessment took on increased importance as policymakers and medical professionals accepted virtual clinical visits as a means of care provision. In addition to making residents aware of gaps in their own knowledge, results serve as a needs assessment, alerting our faculty and programmatic leadership to areas for telemedicine-specific curricular enhancement. Insight gained through the VOSCE will impact our residency training going forward. Our findings and lessons learned can provide valuable information for both clinical and education researchers leading training programs.