The Development of a Video Intervention to Improve Senior Medical Students' Performance on Outpatient Telephone Encounters: a Delphi Analysis and Randomized Controlled Trial.

Introduction: Postgraduate trainees address outpatient telephone calls (OTCs) with little prior training. This study determines the skills necessary for OTCs and examines whether a video intervention improves medical students' performance on simulated OTCs. Materials and
Methods: We utilized a Delphi technique to determine skills needed for OTCs and created a 9-min video teaching these skills. Senior medical students were randomized to Intervention (viewed video) and Control (did not view video) groups. Students were assessed pre-/post-intervention on simulated OTCs. The primary outcome was the between-group difference in improvement.
Results: The Delphi yielded 34 important skills with the highest focus on communication (n = 13) and triage (n = 6). Seventy-two students completed assessments (Control, n = 41; Intervention, n = 31). The score (mean ± SD) improved 4.3% in the Control group (62.3 ± 14.3% to 66.6 ± 25.0%) and 12.2% in the Intervention group (60.7 ± 15.2% to 72.9 ± 20.4%, p = 0.15). The effect size measured by Cohen's d was 0.55, considered effective (> 0.33) for an educational intervention. Conclusions: This project fills a gap in OTC training. The use of the Delphi technique, intervention development based on the results, and evaluation of efficacy is a process that could be reproduced for other educational gaps. Supplementary Information: The online version contains supplementary material available at 10.1007/s40670-021-01331-w. © International Association of Medical Science Educators 2021.

Introduction

New postgraduate trainees are often assigned telephone medicine responsibilities [1, 2] with little prior training [3, 4]. This may occur when covering an assigned panel of continuity clinic patients or when working after-hours (e.g., night float) [2]. Telephone encounters address a variety of patient issues such as answering questions, addressing concerns, communicating results, and discussing plans. Moreover, the importance of telemedicine (“the use of electronic information and communications technologies to provide and support health care when distance separates participants,” which includes synchronous telephone medicine [5], the focus of this study) is increasing dramatically due to the coronavirus disease 2019 (COVID-19) pandemic [6, 7]. Given the multiple challenges of remote communication, outpatient telephone calls (OTCs) have been shown to increase the potential for medical errors [1, 8–10]. As such, postgraduate trainees express discomfort with this task [11, 12].

There is a need for telephone medicine curricula to be incorporated in medical school [13]. The importance of telephone medicine has been recognized by the Accreditation Council for Graduate Medical Education Program Requirements for Family Medicine programs, which specifies that postgraduate trainees engage in telephone visits (IV.C.4.f.) [14] and incorporate telephone medicine as a Communication milestone (C-4) [15]. Additionally, other health professions such as nursing recognize the importance of telehealth and offer formal training and telehealth credentialing [16]. However, there is a general lack of guidelines or curricula available targeting learners at the medical student level [13]. The few existing guidelines pre-date the current ubiquitous use of Electronic Medical Records (EMR) [17, 18] or are time-intensive [19]. Therefore, there is a need for an updated OTC curriculum prior to the start of postgraduate training.

One well-described way to increase preparedness for postgraduate training is through Residency Preparation Courses (RPCs), which are courses at the end of medical school intended to better prepare senior students for postgraduate expectations [20, 21]. A strategy utilized in RPCs for both education and assessment is a mock paging curriculum in which students receive pages with simulated scenarios and must communicate accordingly with either a simulated nurse (inpatient scenarios) or a simulated patient (outpatient scenario). Mock paging curricula are generally well-received by students [22-26] and have been shown to improve confidence [27, 28], decrease anxiety [27], improve clinical decision making [25, 29, 30], improve communication skills [25, 27, 31], and improve attitudes toward interprofessional communication and collaboration with nurses [32].

We hypothesized that an educational intervention directed specifically at OTCs would improve student performance on mock outpatient pages (i.e., simulated OTCs). Our study addressed this question by first utilizing a Delphi technique to determine the OTC skills important to teach students. Second, we used this information to develop a brief video educational intervention. Finally, we tested the efficacy of the educational intervention with a randomized controlled trial involving senior medical students enrolled in RPCs at a single institution.

Materials and Methods

Delphi Technique

The Delphi technique is a method used to establish consensus among a group of experts for a given issue. It has been used in medical education extensively with good effect in multiple studies in both the medical and nursing education literature [33-35]. We performed a single-center, conventional, 2-round Delphi with electronic surveys created and distributed using Qualtrics survey software (Qualtrics International Inc., Provo, UT, USA).

Study Setting and Population for Delphi Method

We invited seven faculty members and eleven postgraduate trainees from six different fields (Family Medicine, Internal Medicine, Internal Medicine-Pediatrics, Obstetrics and Gynecology, Pediatrics, and Surgery). They were invited based on outpatient clinical experience, experience with trainees, and experience in transition-to-postgraduate education. Residents were invited because of their relatively recent experience transitioning from medical school to postgraduate training.

Delphi Survey Round 1

Respondents were asked to provide free-text responses to three questions regarding what an intern should know to address OTCs:

“What are the most important skills for interns to know when interacting with outpatients over the phone?”

“What are the most frequent challenges for which interns should be prepared when interacting with outpatients over the phone? What are the skills interns need to address these challenges?”

“What other things should be included in a video to teach interns about interacting with outpatients over the phone?”

The authors developed these questions based on their clinical experience. Similar responses were collated by one author and verified by another. Skills were categorized according to themes (e.g., triage, communication, medical knowledge, etc.). See Appendix 1 for the full survey.

Delphi Survey Round 2

The skills identified in the first round were sent via Qualtrics, and respondents were asked to rank the importance of the skills on a 4-point Likert scale (“Not at all important,” “Slightly important,” “Very important,” or “Extremely important”). A neutral middle point was excluded to compel participants to choose either importance or unimportance and to assist in calculations for consensus [33]. The survey in the second round also included questions regarding respondents’ demographics and clinical experience (Appendix 2).

Defining Consensus

Results were analyzed according to importance (important or not important) and degree of consensus (high consensus, approaching consensus, no consensus). High consensus for a skill being important was defined as ≥ 80% of respondents agreeing on a skill being “Extremely” or “Very important” [33]. Approaching consensus for a skill being important was defined as 50–79% agreeing on a skill being “Extremely” or “Very important.” High consensus for a skill not being important was defined as ≥ 80% of respondents agreeing on a skill being “Not at all” or “Slightly important.” Approaching consensus for a skill not being important was defined as 50–79% agreeing on a skill being “Not at all” or “Slightly important.”

Educational Intervention

The skills selected by the Delphi process were incorporated into a video intervention. This 8:38-min video presented a schematic and algorithmic framework by which to address OTCs. Select images from the video are available (see Fig. 3 and Supplementary Information). The video was a narrated Google Slides (Alphabet Inc., Mountain View, CA, USA) presentation recorded with Camtasia (TechSmith, Okemos, MI, USA).

Fig. 3

Select images from the educational video intervention which describe a the three main components for caring for patients remotely, i.e., b the setting in which the encounter takes place, c the interaction with the patient, and d foundational knowledge and skills

Randomized Controlled Trial

Study Setting and Population

Participants were senior medical students who were enrolled in specialty-specific RPCs (Family Medicine, Internal Medicine, Obstetrics and Gynecology, Pediatrics, or Surgery) in February and March 2019 based on their career interests. Participants were excluded from analysis if they dropped out of the RPCs at any time, or if they did not complete (e.g., due to other scheduled RPC requirements) or results were not available for both the pre- and post-intervention simulated OTCs. See Fig. 1 for further details of study population.

Fig. 1

Randomized controlled trial protocol. Students were first stratified within their Residency Preparation Courses (RPCs) then randomized. Then, both Intervention and Control groups were assessed with Mock Page 1. Subsequently, the Intervention group received the educational video and both groups were again assessed with Mock Page 2. Analysis was conducted according to the intention-to-treat principle. RPC residency preparation course, FM family medicine, IM internal medicine, OB/GYN obstetrics and gynecology, Peds pediatrics, Surg surgery

Assessment

Students were assessed on their ability to manage OTCs based on their performance responding to simulated OTCs within a larger mock paging curriculum. The assessment used trained registered nurses acting as standardized patients to page students with simulated outpatient scenarios; students would call the standardized patient to address their concern. These simulated OTCs have been used for several years as a part of our institution’s RPCs. The scenarios presented in these simulated OTCs are meant to represent outpatient scenarios common to the RPC specialty (e.g., the scenarios are different for Surgery compared to OB/GYN). Students were graded according to a previously established expert-validated rubric based on a weighted checklist for each case. They earned points for what they “Must do” and “Should do” and were penalized points for what they “Should not do” and “Must not do.” These points contributed to an Overall Score which was reported as a percentage. See Appendix 3 for an example case.

Randomization and Intervention

Students were stratified within their respective RPC and randomized into two groups, Intervention and Control. Students were randomized using a random number generator within Google Sheets (Alphabet Inc., Mountain View, CA, USA). Participants were blinded to randomization. The Control group received “usual care,” i.e., there was no placebo. The nurses (acting as standardized patients) were the outcome adjudicators and data collectors in that they evaluated the students according to the primary outcome and recorded their scores; the nurses were blinded to group assignments. The author who acted as data analyst was not blinded to group assignments; however, he did not participate in the data collection or any outcome adjudication.

Students received a total of two simulated OTCs. Both groups received Mock Page 1 over the course of 1 week and were scored by the adjudicator according to the rubric for the case. The Intervention group was then sent a link to the educational video via Qualtrics. The Qualtrics survey recorded whether a student opened the link to the video. The following week, both groups then received Mock Page 2 and were again scored. After the conclusion of the simulated OTCs, the Control group was sent the link to the educational video to view for edification (see Fig. 1).

Evaluation

After viewing the educational video, students were automatically redirected to two free-text survey questions:

What did you learn from this video that you did not learn previously during medical school?

What feedback do you have to improve this video for future students?

Outcomes

The primary outcome was the mean improvement in overall score pre- and post-intervention of Intervention compared to Control. The secondary outcome was the perceived value of the video intervention based on survey comments about what skills were learned.

Statistical Analysis

The primary outcome was examined using an intention-to-treat analysis. Differences between groups were determined using a multivariate repeated measures analyses for significance (p ≤ 0.05) and Cohen’s d for effect size. The threshold for an educational intervention to be considered effective is a Cohen’s d > 0.33 [36]. The data were analyzed using JMP Pro 14.2.0 (SAS Institute, Inc., Cary, NC, USA).

Results

Respondents included 4/7 faculty (57%) and 8/11 postgraduate trainees (73%) in four specialties (Internal Medicine, n = 5; Medicine-Pediatrics, n = 1; Obstetrics and Gynecology, n = 3; Surgery, n = 3) for a total response rate of 67%. See Table 1 for respondent characteristics. The first round of the Delphi process resulted in 144 ideas. These ideas were collated into 40 teachable skills within the categories of communication with the patient (n = 13), triage (n = 6), medical knowledge (n = 5), gathering information (n = 4), logistics of telephone encounters (n = 4), documentation/communication with other providers (n = 3), available resources (n = 3), prescribing medications (n = 1), and follow-up (n = 1). Eighteen skills were important, with high consensus. Sixteen skills were important, approaching consensus. Five skills were not important, approaching consensus. One skill was not important, with high consensus. All skills except the unimportant skill with high consensus were incorporated into the video intervention (see Table 2).

Table 1

Characteristics of respondents of Delphi process

Faculty/postgraduate trainee	Specialty	Gender	Years experiencea	Avg. clinic half-days per month	Avg. outpatient telephone calls per weekb	Avg. outpatient EMR messages per weekb
Faculty	IM	Female	15	20	0–5	20–50
Faculty	IM	Male	4	24	0–5	> 50
Faculty	OB/GYN	Female	13	8	0–5	10–20
Faculty	Surgery	Male	8	12	5–10	20–50
Trainee	IM	Female	3	4	0–5	5–10
Trainee	IM	Male	1	3	0–5	0–5
Trainee	IM	Male	3	4	5–10	20–50
Trainee	IM-Peds	Male	3	4	0–5	0–5
Trainee	OB/GYN	Female	2	12	5–10	20–50
Trainee	OB/GYN	Female	5	16	10–20	> 50
Trainee	Surgery	Female	2	8	0–5	0–5
Trainee	Surgery	Male	2	4	5–10	0–5

EMR electronic medical record, OB/GYN obstetrics and gynecology, IM internal medicine, IM-Peds internal medicine-pediatrics

aFor faculty, years of experience are since finishing postgraduate training (residency or fellowship). For postgraduate trainees, years of experience is postgraduate year

bOptions were 0–5, 5–10, 10–20, 20–50, > 50

Table 2

Results of modified Delphi process

Skill category	Skill
Important—high consensus
Available resources	Know how ancillary clinic staff (e.g., RNs, LPNs, MAs, SWs, PharmDs, secretaries) can help you
	Know how attendings and senior residents can help you
Communication with the patient	Be honest, even if you do not know the answer
	Communicate a clear assessment and plan
	Handle challenging situations (e.g., talkative patient, demanding patient, aggressive patient, “worried well,” when a patient does not agree with your assessment)
	Introduce self by name and role
	Know the limitations of remote conversations (e.g., diagnosis without thorough evaluation, absence of body language)
	Understand privacy (e.g., verify the patient's identity, know who can and cannot receive information about a patient, know what can and cannot be left over voicemail)
	Use effective interpersonal communication skills (e.g., listening skills, pace of speech, being open for questions, being polite, being patient)
	Use the teach back communication method (ask the patient to repeat the plan as s/he understands it)
Documentation/communication with other providers	Know how to document telephone encounters
	Know the importance of documentation for medico-legal purposes
Gathering information	Know how to gather essential information from the patient
	Know how to review the EMR for relevant data (e.g., recent operation, recent prescriptions)
Prescribing	Understand the appropriateness of prescribing certain medications over the phone (e.g., refills, antibiotics, controlled substances)
Triage	Communicate red flag symptoms for which the patient should go to the emergency department as anticipatory guidance when necessary
	Understand which situations can be handled on the phone, which require reassurance, what can wait until their next appointment, what should be seen in clinic urgently, and what should be handled in the emergency room
Important—approaching consensus
Available resources	Ensure patient is going through her/his primary physician for non-urgent primary care issues
Communication with the patient	Be empathetic
	Ensure it is a good time and place for the patient to receive information
	Know that one may have to repeat information for family members
Documentation/communication with other providers	Know how to add photos to a patient's chart
Follow-up	Know how to arrange follow-up, including when clinics are available
Gathering information	Know that practice is necessary in the setting of poor historians and not having EMR data available
Logistics of telephone encounters	Know the logistics of a remote encounter (i.e., via telephone or via the EMR “inbasket”)
	Know what the expectations are regarding in how much time one is expected to respond to the page
	Know whether one needs to be near a computer when on call
Medical knowledge	Handle questions about side effects
	Know the common reasons for outpatient telephone calls (e.g., common immediate and delayed post-operative issues, prescription refills, exacerbations of chronic conditions)
	Understand that medical decision making comes with experience and mentorship
Triage	Confirm what level of care is available in the patient's location (e.g., emergency room, urgent care)
	Make an appointment if the conversation is complex
	Understand that triaging outpatient scenarios comes with experience
Not important—approaching consensus
Communication with the patient	Send a letter after the telephone call if the conversation was complicated
	Set expectations in clinic prior to a telephone call with patients
Gathering information	Instruct the patient to send a photograph through the patient portal when clinically appropriate
Medical knowledge	Read to understand pathophysiology
Triage	Contact the patient's attending if considering emergency referral
Not important—high consensus
Medical knowledge	Handle questions about recalls

EMR electronic medical record, RN registered nurse, LPN licensed practical nurse, MA medical assistant, SW social worker, PharmD Doctor of Pharmacy

Participants

Eighty-nine students enrolled in the RPCs in February and March 2019. Eighty-five were stratified and randomized into Control (n = 46) and Intervention (n = 39) groups. Forty-one students in the Control group and 31 students in the Intervention group completed pre- and post-intervention assessments (Fig. 1). Of the 31 students that completed both pre- and post-intervention assessments in the Intervention group, 26 (84%) accessed the link to view the video.

Primary Outcome

The scores for both groups improved; Control group scores increased from 62.3 ± SD 14.3% to 66.6 ± 25.0% and Intervention group scores increased from 60.7 ± 15.2% to 72.9 ± 20.4% (p = 0.15). The intervention had a notable effect as indicated by the medium effect size (Cohen’s d = 0.55). Values are tabulated in Table 3 and scores are plotted in Fig. 2.

Table 3

Mock paging scores in Control and Intervention groups, pre- and postintervention

Group	n	Simulated OTC score mean (SD)			Cohen’s d effect sizea
		Simulated OTC 1	Simulated OTC 2	Improvement
Control	41	62.3% (14.3%)	66.6% (25.0%)	4.3% (25.2%)	0.55
Intervention	31	60.7% (15.2%)	72.9% (20.4%)	12.2% (18.9%)

OTC outpatient telephone call

aThe effect size measured by Cohen’s d between the Control and Intervention groups, using the pooled standard deviation of the first time point (Mock Page 1). The effect size may be classified as small (0.20 ≤ d < 0.50), medium (0.50 ≤ d < 0.80), and large (0.80 ≤ d) [44]. The threshold for an educational intervention to be considered effective is a Cohen’s d > 0.33[35]

Fig. 2

Plot of student overall scores. Each student’s scores are plotted for Mock Page 1 and Mock Page 2 with a line connecting the scores. OTC outpatient telephone call

Secondary Outcomes

Of the 77 students who accessed the video link, 60 students (78%) responded to the first free-text question and 47 (61%) responded to the second question. Students learned skills in categories that are reflective of those that emerged from the Delphi including: gathering information, communication with the patient, triage, available resources, and documentation / communication with other providers (Table 4). The most common suggestion to improve the video was to include more specific case examples during the video or in follow-up material that address common situations (n = 21) (Fig. 3).

Table 4

Skills that students reported learning from the video intervention that were otherwise not learned during medical school

Skill category	Skill
Available resources	To talk with nurses, residents, and attendings for advice about common calls and for help when needed (n = 5)
	To rely on ancillary staff and know how they can help (n = 2)
Communication with the patient	To/How to introduce themselves over the phone (n = 5)
	To ask the patient if s/he is able to talk freely and whether it is a good time to talk (n = 3)
	To use the teach back method (n = 2)
Documentation/communication with other providers	What information to include in a phone note (n = 4)
	To route the telephone note to the patient’s primary care physician or specialists (n = 3)
Gathering information	Ask the patient to provide vital signs to gather some objective date (n = 6)
	To do a full but brief History of Present Illness with patients (n = 5)
	To suggest that the patient send photos through the EMR (n = 3)
	That remote encounters are similar to in-person encounters (n = 3)
	To gather information from the EMR in addition to speaking with the patient (n = 2)
Triage	The importance of and how to triage outpatients over the phone (n = 9)
Follow-up	Know how to send a patient to the ED (i.e., the logistics for doing so)
Other	A general structured outline and approach for handling outpatient calls, even if all of the individual concepts had previously been taught (n = 22 students)

Students had differing levels of experience prior to the RPCs. Some students had little experience, as in “I’ve never had formal teaching in medical school on how to handle outpatient telephone encounters…” and, “I think the biggest thing is just having a framework for approach to the more remote management of outpatients because it’s not something we routinely get exposure to in medical school.” Another said, “It was a nice template for a type of patient encounter that we get little-to-no explicit training…during medical school.” Other students had more consequential prior experiences. For example, “…I feel like I learned this in school and was fortunate to have experience practicing this during my clerkships a couple of times.” Another student had experience based on their extracurricular activities: “I actually do feel like I’ve had a lot of practice with managing outpatients remotely in medical school as a result of my work with the Student-Run Free Clinic…I oversee all of the labs that come back for patients, create medical plans, and communicate those to patients. So luckily, I feel like I am already really familiar with the concepts in this video.”

Discussion

In this study, we demonstrated an evidence-informed process by which to develop an educational intervention related to managing OTCs and proved its efficacy based on a previously validated assessment. We demonstrated that the video was effective in improving student performance responding to simulated OTCs through the level of results according to the Drs. Kirkpatrick [37].

The Delphi technique showed utility in establishing consensus on a topic without available guidelines as an established way to “crowdsource” ideas from experts, as has been demonstrated previously in medical education literature [33, 34]. One particularly relevant example is a study by van Houwelingen et al. who used the Delphi technique to determine entrustable professional activities for nurses in the telehealth setting [35]. In our study, the skills deemed most important and with highest consensus were related to communication, followed by triage and logistics. This highlights the unique skillset involved in addressing OTCs beyond just the medical knowledge of outpatient medicine. Strengths included the variety of specialties represented, and the differing levels of experience that included early experienced clinicians as well as postgraduate trainees in order to create an educational video informed by different perspectives and that would be relevant regardless of students’ intended specialty. Our study also has relevance for a scoping review which is currently under protocol by Cooper and Alexander of the Joanna Briggs Institute seeking to examine the current evidence on the “components, skills and training required for conducting initial telephone consultations [38]. This scoping review will, reciprocally, put our results in the greater context of the current state of medical and allied health professionals training.

Our intervention to improve telephone medicine skills complements those designed by other researchers. Seal et al. designed a 1-h in-person training session and measured the frequency of gathering important information and self-reported student confidence during simulated telephone calls. They found that, compared to the control group, students who had completed the training gathered important information more frequently and had greater self-reported confidence in their performance [31]. Saba et al. designed a study in which students attended a 2-h in-person training session, followed by telephone calls with real patients, and finally a telephone call with a standardized patient. They found that, compared to the control group, students who had completed the program scored better on some measured communication skills [39]. Contrasting our intervention with the work of these researchers, the development of a video intervention rather than an in-person lecture was useful as it promoted standardization (i.e., can be used in the future regardless of lecturer availability or experience), flexibility of scheduling learning, and repeat viewing if desired. Further, while a considerable amount of time is required to initially develop the video, it requires no subsequent resources from educators. This also conforms with suggestions for providing self-directed learning materials [40] in response to the trial by Seale et al.[31].

While a rapid pivot to remote learning is important (especially in the context of the COVID-19 pandemic) [7, 41], it is essential to test the efficacy of learning interventions. There is certainly debate regarding the utility of randomized controlled trials (RCTs) in medical education [42-45], but it is generally accepted that trial design should reflect the nature of the question and should consider local context [44, 46]. We chose our trial design based on our ability to perform the study while all students at a single institution were completing their RPCs during the same two-month period, which lessens the problem of different training experiences and asynchrony often seen in graduate medical education [43]. Additionally, e-learning such as our video intervention is particularly suitable to RCTs [46] in that it standardizes the “teacher” [43] and therefore leads to a relatively simple intervention to improve a relatively narrow problem [47]. Furthermore, all students have access to the educational video in a timely manner and hence promotes a more equitable learning experience.

Student feedback uncovered a significant number of skills that had not been acquired elsewhere in medical school. Students recounted differing levels of exposure to OTCs, which illustrates the need for an educational intervention prior to the start of postgraduate training such as the one created in this study (or ones developed by other researchers [31, 39]).

There were a number of limitations to our single center trial. Limitations of the Delphi included the number of participants; the optimum number of participants is 15–30 [33], and the current study had 12 respondents. Further, none of the invited participants from Family Medicine or Pediatrics responded, which may limit generalizability. There were also limitations to the randomized controlled trial. First, there were a relatively small number of participants. Second, it is unknown how differing scores on mock paging cases correspond to clinical relevance or how student performance translates to performance during postgraduate training. Third, history bias was not controlled (i.e., students may have been exposed to other lectures regarding OTCs during the course of the trial); however, this was mitigated by randomizing students within each RPC group. The use of the intention-to-treat analysis led to including 5 students in the Intervention group who did not view the intervention, 4 of whom had an increase in their scores and one of whom had a decrease. This increases the probability of a Type I error (falsely rejecting the null hypothesis that our intervention has no effect on student performance). Finally, our video intervention addressed general telephone medicine skills, whereas our measurement (i.e., the grading rubric) addressed the specific skills relevant to each simulated OTC. That is, our intervention did not perfectly match our measurement. This was an intentional decision so as to make our intervention more broadly applicable to outpatient telephone calls, while still taking advantage of the expert-validated grading rubrics that our institution has used for many years. Further, the fact that our intervention was not designed to improve the specific skills graded on the rubric reduces the probability of a type II error.

Conclusions

In conclusion, this study illustrated a process by which to develop an educational intervention addressing OTCs and demonstrated its efficacy. In the future, we plan to re-record the video based on student feedback and expand the study to a larger group of students.

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 88 KB)

Supplementary file2 (DOCX 84 KB)

Supplementary file3 (DOCX 81 KB)

Supplementary file4 (MOV 17134 KB)