Effects of Recorded versus Live Teleconference Didactic Lectures on Medical Student Performance in the Surgery Clerkship.

OBJECTIVE: Due to the COVID-19 pandemic, numerous institutions converted medical education didactics to electronic formats including both live teleconference didactics and recorded faculty lectures. This study aims to compare the effect of recorded versus live teleconference didactic lectures on medical student examination scores during the surgery clerkship.
DESIGN: Medical students completing the Surgery Clerkship received a weekly series of didactic lectures taught by faculty via a teleconference (2020-2021 academic year) or recorded format (2021-2022 academic year). Performance outcomes included weekly quizzes, National Board of Medical Examiners (NBME) Surgery Shelf Exam, and clerkship Objective Structured Clinical Examination (OSCE) scores.
SETTING: University of Miami Miller School of Medicine. PARTICIPANTS: All second- (MS2) and third-year (MS3) medical students completing the Surgery Clerkship over two academic years (n = 312).
RESULTS: Students who received live teleconference lectures (n = 156) demonstrated higher average scores on weekly quizzes (89%) and the NBME shelf exam (76%) compared to those receiving recorded lectures (n = 156; 71% quiz, 70% shelf exam), both p < 0.001. There was a significant association with performance in the highest quartile (Q1) of weekly quiz scores and receiving live lectures (40% vs. recorded lectures 1%, p < 0.001). Comparing only MS3 students, mean weekly quiz scores and Q1 achievement were significantly higher (both p < 0.001) in the teleconference cohort with no significant difference in NBME shelf exam performance (p = 0.971). No difference in OSCE performance was observed between groups.
CONCLUSION: These results suggest that synchronous teleconferences may be more effective than recorded lectures for achieving institutional learning objectives on the surgery clerkship without any negative impact on NBME shelf exam or clinical evaluation parameters. This information should be used to inform future institutional clerkship design and educational initiatives.

INTRODUCTION

Safety precautions recommended by the Center for Disease Control (CDC) to limit the spread of the SARS-CoV-2 virus (COVID-19) included limited social gatherings and social distancing measures. , As a result, educational programs across the country were impacted and forced to develop new methods of instruction to limit in-person gatherings. All facets of medical education were affected with the Association of American Medical Colleges (AAMC) releasing guidelines suggesting that face-to-face educational formats be reduced and replaced by remote and virtual learning modalities.

In response to these guidelines, numerous institutions quickly adapted to asynchronous learning options with recorded lectures as well as synchronous virtual teleconferences for small-group learning activities. From the 2020-2021 to 2021-2022 academic years, our department of surgery converted our weekly didactic lecture series for medical students on the surgery clerkship from a synchronous electronic teleconference lecture to recorded video lectures. Despite the fact many medical schools across the country have converted to entirely online curriculums, , the optimal virtual format with which lecture material should be conveyed has yet to be fully investigated. Our study sought to compare the effect of synchronous teleconference lectures versus recorded video lectures on objective grading parameters including NBME examination and clinical performance scores during the surgery clerkship. We hypothesized that no significant difference in student performance measures would exist among students receiving either format of didactics.

METHODS

Data Source and Cohort Selection

Data from all medical students completing the surgery clerkship at a single institution from July, 2020 through January, 2022 were compiled and de-identified. This retrospective study was deemed exempt from review by the University of Miami Institutional Review Board.

Curriculum Design

During the surgery clerkship, a one-hour didactic lecture is given every week by an attending surgeon faculty member of the respective specialty corresponding to the lecture topic. The lectures involve a general overview of the most common disease epidemiology, pathophysiology, clinical presentation, diagnosis, and management pertaining to that subject. The didactic presentations were delivered as a one-hour lecture by a single attending faculty member with a power point presentation over the relative topic. Students were offered the ability to ask questions at any time during the live teleconference format and were given the ability to email lecturers at any time for the recorded lecturers. The didactic topics and quizzes included thyroid and parathyroid, inguinal hernia, esophagus and foregut, gallbladder and pancreas, appendix and small bowel, breast, trauma and vascular, critical care, and colon and anorectal disease. All medical students received either lectures delivered remotely via a live teleconference format (2020-2021 academic year; Zoom Video Communications, Inc., San Jose, California) or a recorded video format (2021-2022 academic year) that students could watch at any time. Across both study years, the didactic content presented in both recorded and live lectures was identical. Mandatory attendance and the completion of didactic materials by students was required for both study years, respectively. At the start of the 2021 to 2022 academic year, there was a curriculum shift in our institution such that MS2 students completed their surgery clerkship overlapping with MS3 students from the traditional curriculum. All MS2 and MS3 students during the 2021 to 2022 academic year received an identical surgery clerkship curriculum and experience in both their clinical rotation time, all didactic presentation content, and exam content. However, MS2 and MS3 students were assigned to separate rotation services independent of each other in order to prevent direct comparison between students by faculty members on student clinical evaluations. To prevent confounding due to differences in individual lecturer content, the same didactic presentation was given by the same faculty lecturers assigned to their topic across both study years. To account for this difference, a subgroup analysis was performed comparing only MS3 students across both academic years.

All medical students were administered a ten-question quiz about the lecture material as well as required readings from a general surgery text (Essentials of General Surgery, fifth edition, Peter F. Lawrence-Lippincott William and Wilkins, Philadelphia, Pennsylvania). The quiz questions administered were identical across all study years. In addition to the quizzes, students were also formally evaluated on the final day of the clerkship by the National Board of Medical Examiners Surgery (NBME) Shelf Exam similar to all other ACGME-accredited allopathic medical schools. Clinical evaluation parameters are utilized by 2 methods. First, students are evaluated by a standardized written evaluation by attending surgeons at the end of each surgery rotation. These evaluations are completed based on the collective input of all attending within the division that students rotate on and consist of a RIME (reporting, interpretation, management, and educator) model with a 5-point Likert scale to assess students, which are then converted into a percentage grade. All evaluations are reviewed by our institution's Surgery Clerkship Directors to ensure evaluations are standardized and prevent significant inter-evaluator variability. Second, each student completes an Objective Structured Clinical Exam (OSCE) in which they perform a history and physical exam on a standardized patient presenting with a common general surgical condition (acute appendicitis, acute cholecystitis, small bowel obstruction, etc) and complete a History and Physical write-up with an Assessment and Plan including at least 3 differential diagnoses. These exams are graded by both standardized patients (communication skills) and clinical surgeons (write-up). All grading of the OSCE by both standardized patient actors and attending physicians are standardized to a checklist of whether individual items were performed or recorded in the write-up in order to limit variation amongst evaluators. Of note, there was a change in the grading schema during the 2020 to 2021 academic year for the OSCE from a pass/fail system to a percentage score resulting in 41 (13%) live teleconference students having only a pass/fail score reported. A passing score on the OSCE was set as greater than or equal to 75%.

Statistical Analysis

Students were classified into quartiles from Q1 (top 25% of performers) to Q4 (bottom 25% of performers) based on each individual grading parameter. Continuous variables are reported as mean ± standard deviation (SD) for normally distributed variables and analyzed using Student's t-test. Categorical variables were compared among groups using chi-squared analysis or Fisher's exact test as appropriate. IBM SPSS Statistics version 28 (International Business Machines Corp, Armonk, New York) was utilized for statistical analysis. Statistical significance was defined as p < 0.05.

RESULTS

Analysis of All Medical Students Completing the Surgery Clerkship

There were 312 students identified during the study period, of which 50% (n = 156) received live teleconference lectures (2020-2021 academic year) and 50% (n = 156) received recorded lectures (2021-2022 academic year). All students were stratified into quartile performance ranging from the highest (Q1) to lowest (Q4) performance on each grading parameter of the surgical clerkship (Table 1 ). Mean performance scores on the overall clerkship, clinical evaluations, NBME shelf exam, and weekly quiz scores were significantly higher for the cohort receiving live teleconference didactics compared to the recorded cohort, all p < 0.001 (Table 2 ). Although percent scores were only available for 115 students from the live teleconference cohort due to a change in the grading schema during the 2020 to 2021 academic year, there was no difference in mean OSCE performance between both groups. All students had a pass/fail OSCE score available (n = 312), and there was no difference in pass rates (greater than or equal to 75% threshold) on the OSCE between either cohort (Table 3 ). Regarding performance in the top quartile of each individual grading parameter, students who received live teleconference didactics more frequently placed in the top quartile of the NBME shelf exam, quiz scores, clinical evaluations, and overall surgery clerkship grades (Table 4 ). There was no difference in placing in the top quartile of the OSCE in either group.

TABLE 1

Performance Quartiles on Grading Parameters of all Medical Students Completing the Surgical Clerkship

Surgery Clerkship Grading Parameters	Percent Score
Clinical evaluations
Q1	> 98.0
Q2	97.1-98.0
Q3	95.1-97.0
Q4	≤95.0
NBME shelf exam
Q1	>80.0
Q2	74.1-80.0
Q3	67.1-74.0
Q4	≤67.0
Weekly quiz score average
Q1	>90.0
Q2	83.1-90.0
Q3	73.3-83.0
Q4	≤73.3
OSCE
Q1	>96.0
Q2	92.1-96.0
Q3	88.1-92.0
Q4	≤88.0
Overall clerkship grade
Q1	>91.0
Q2	88.1-91.0
Q3	85.1-88.0
Q4	≤85.0

NBME, National Board of Medical Examiners; OSCE, Objective Structured Clinical Exam.

TABLE 2

Mean Performance Scores of All Medical Students Completing the Surgical Clerkship

Grading Parametera	Live Teleconference Cohort n=156	Recorded Cohort n=156	p value
Clinical evaluations	97.3 ± 1.7	94.7 ± 3.4	<0.001
NBME shelf exam	75.7 ± 8.3	69.5 ± 9.3	<0.001
Weekly quiz score average	88.8 ± 5.3	71.1 ± 11.6	<0.001
OSCE	91.0 ± 7.0b	90.9 ± 6.8	0.946
Overall clerkship grade	90.5 ± 3.5	84.2 ± 6.5	<0.001

Bolded values are statistically significant (p < 0.05).

NBME, National Board of Medical Examiners; OSCE, Objective Structured Clinical Exam.

Data presented as mean ± standard deviation.

Mean OSCE score for the live teleconference cohort representative of n = 115 students.

TABLE 3

Performance of all Medical Students on the Objective Structured Clinical Exam (OSCE) Completing the Surgical Clerkship Stratified by Pass/Fail Scores

Grading Parameter	Live Teleconference Cohort n = 156 (%)	Recorded Cohort n = 156 (%)	p Value
OSCE			0.361
Pass	151 (97)	153 (98)
Fail	5 (3)	3 (2)

TABLE 4

Performance of all Medical Students Completing the Surgical Clerkship on Objective Parameters Stratified By Top 25% Performance in the First Quartile (Q1)

Grading Parameter	Live Teleconference Cohort n = 156 (%)	Recorded Cohort n = 156 (%)	p Value
Q1 NBME shelf exam			<0.001
Yes	51 (33)	14 (9)
No	105 (67)	142 (91)
Q1 Weekly quiz score			<0.001
Yes	63 (40)	1 (1)
No	93 (60)	155 (99)
Q1 Clinical evaluation			<0.001
Yes	41 (26)	18 (12)
No	115 (74)	138 (89)
Q1 OSCE			0.106
Yes	20 (17)a	29 (19)
No	95 (83)a	127 (81)
Q1 Final grade			<0.001
Yes	68 (44)	5 (3)
No	88 (56)	151 (97)

Bolded values are statistically significant (p < 0.05).

NBME, National Board of Medical Examiners; OSCE, Objective Structured Clinical Exam.

Percentages given out of total number of students for which percent OSCE numeric scores were available (n=115).

Subgroup Analysis of only Third Year Medical Students Completing the Surgery Clerkship

During the 2021 to 2022 academic year there was an institutional curriculum change to introduce the surgery clerkship 1 year earlier resulting in both MS2 (n = 102) and traditional MS3 students (n = 54) completing the same curriculum and clerkship simultaneously. To account for this difference, a subgroup analysis was completed analyzing only MS3 students within the recorded lecture cohort (n = 54) compared to the MS3 teleconference cohort (n = 156). Regarding mean performance scores (Table 5 ), the synchronous teleconference cohort had higher clinical evaluations (p = 0.009), weekly quiz scores (p < 0.001), and overall clerkship grades (p < 0.001). There was no difference in mean NBME shelf or OSCE scores. The pass rates on the OSCE were similar between the MS3 live teleconference group (97%) and MS3 recorded lecture group (98%; p = 0.694). Overall, those in the live teleconference group comprised a greater proportion of the top quartile of performance in the weekly quiz scores and final surgery clerkship grades (both p < 0.001; Table 6 ). There was no difference in performance in the top quartile on the NBME shelf exam, clinical evaluations, or OSCE between both groups.

TABLE 5

Mean Performance Scores of Third-Year Medical Students (MS3) Completing the Surgical Clerkship from 2021 to 2022

Grading Parametera	MS3 Live Teleconference Cohort n = 156 (%)	MS3 Recorded Cohort n = 54 (%)	p Value
Clinical evaluations	97.3 ± 1.7	96.2 ± 2.5	0.009
NBME shelf exam	75.7 ± 8.3	75.8 ± 6.5	0.971
Weekly quiz score average	88.8 ± 5.3	74.7 ± 11.8	<0.001
OSCE	91.0 ± 7.0b	90.3 ± 7.2	0.548
Overall clerkship grade	90.5 ± 3.5	87.2 ± 3.5	<0.001

Bolded values are statistically significant (p < 0.05).

NBME, National Board of Medical Examiners; OSCE, Objective Structured Clinical Exam.

Data presented as mean ± standard deviation.

Mean OSCE score for the live teleconference cohort representative of n = 115 students.

TABLE 6

Performance of Third-Year Medical Students Completing the Surgical Clerkship on Objective Parameters Stratified by Performance in the Top 25% of Performance Scores Quartile (Q1)

Grading Parameter	MS3 Live Teleconference Cohort n = 156 (%)	MS3 Recorded Cohort n = 54 (%)	p Value
Q1 NBME shelf exam			0.056
Yes	51 (33)	10 (19)
No	105 (67)	44 (82)
Q1 Weekly quiz score			<0.001
Yes	63 (40)	1 (2)
No	93 (60)	53 (98)
Q1 Clinical evaluation			0.168
Yes	41 (26)	10 (19)
No	115 (74)	44 (81)
Q1 OSCE			0.131
Yes	20 (17)a	11 (20)
No	95 (83)a	43 (80)
Q1 Final grade			<0.001
Yes	68 (44)	5 (9)
No	88 (56)	49 (91)

Bolded values are statistically significant (p < 0.05).

NBME, National Board of Medical Examiners; OSCE, Objective Structured Clinical Exam.

Percentages given out of total number of students for which percent OSCE scores were available (n = 115).

DISCUSSION

The aim of this study was to elucidate the effect that live teleconference lectures compared to recorded video lectures have on examination and clinical performance scores during the surgery clerkship. Among all medical students, those who received live teleconference lectures had higher average overall scores measured by their overall clerkship grades, NBME shelf exam, clinical evaluation scores, and weekly quiz score averages. Furthermore, exposure to a synchronous teleconference format was associated with performance in the top quartile of students in similar grading parameters. When comparing only MS3 students, those who received live teleconference lectures had higher mean overall clerkship grades, clinical evaluations, and weekly quiz scores. Performance in the top quartile of weekly quizzes and the final clerkship grade was more frequent among MS3 students who received live teleconference lectures. There was no difference in mean or top quartile OSCE scores between either groups. Overall, these results suggest that synchronous teleconference lectures may be more optimal for institutional clinical didactic learning compared to recorded without any negative impact on NBME shelf exam or clinical evaluation performance.

Although the different effects on performance between recorded and in-person lectures have been well studied, , 7, 9 less is known about the impact of recorded versus virtual teleconference lectures. These two formats hold separate advantages and disadvantages for both the learner and instructor. Within our institution, one benefit of utilizing recorded lectures was the reduced administrative burden for course schedulers in coordinating lecture schedules with full-time attending surgeons’ availability. For medical students, an advantage of a recorded format during the surgical clerkship is the extra time it allots for students to participate in clinical activities as well as the freedom to utilize video-accelerating technologies to reduce their total amount of time in lecture. Previous studies have found that the majority of medical students prefer utilizing such video-accelerating features for recorded lectures as they feel such features can keep them more engaged and increase their speed of knowledge acquisition. Despite students’ broad acceptance of online education formats and their flexibility, others have found that medical trainees still do not favor complete replacement of face-to-face lectures with recorded lectures and report feeling less engaged with such sessions. - Virtual teleconference lectures provide another option for live learning, and students have reported no significant difference in preference between in-person and teleconference formats. , , However, teleconference formats require considerable electronic resources and investment by both trainees and lecturers, and these formats can suffer from audiovisual technical difficulties that can impair the lecture experience.

Among all medical students, mean scores and performance in the top quartile on objective metrics such as the overall clerkship grade and weekly quiz scores were higher for students receiving teleconference didactics. Given that the weekly quizzes directly correlated to the lecture material topics in both academic years, this grading parameter may indicate the effect of the two virtual options on medical student performance most directly. Furthermore, this relationship persisted in a head-to-head comparison of only third year medical students completing the surgical clerkship. Within our own institution, we previously demonstrated that examination scores on the NBME shelf exam and weekly quizzes did not decrease as a result of the conversion from in-person to virtual teleconference didactics. Taken together, these results may indicate that students perform more similarly when receiving in-person or live teleconference lectures compared to recorded lectures. When considering a more standardized national performance exam such as the NBME surgery shelf exam compared to our institutional quizzes, students who received live teleconference lectures achieved higher mean scores and made up a greater proportion of the top quartile of performers compared to those who received recorded lectures. Previous work has demonstrated institutional weekly quiz scores to be correlated with performance on the NBME surgery shelf exam. However, this relationship was not preserved when examining only third year students in both groups, and these results in NBME performance may also be reasonably explained by the differences in second and third year students completing the clerkship. Moreover, these results do not suggest inferiority of teleconference lectures compared to recorded lectures for medical student performance and learning on the medical student clerkship.

Although clinical evaluation scores were significantly higher for students who received synchronized teleconference didactics, there was no difference in performance in the top quartile of students who received either lecture format. Moreover, another clinical performance parameter, the OSCE, was not significantly associated with either lecture format. As both of these metrics are related more closely with clinical performance compared to the other parameters utilized in our study, these results may support previous studies that have found no significant correlation between objective grading parameters and clinical performance in the surgery clerkship. , Alternatively, it may be that the didactics in our institution do not significantly correlate with clinical performance and either format is comparable for this parameter.

Several significant limitations existed in the present study. Given the retrospective nature of this study, only individual clerkship performance scores were available for analysis without any accompanying demographic information. As a result, it was not possible to compare the effect of these demographic variables upon individual grade performance. Another factor that can affect student performance regardless of the lecture format is learner engagement. , Although attendance and completion of didactic materials was mandatory across study years, students’ level of interest and engagement was not prospectively surveyed in this study. As with any evaluative parameter, there is likely some degree of inter-evaluator variability with regards to clinical performance scores. Our institution provides standardized annual training to the faculty on how to assess the performance of medical students on the surgery clerkship to reduce the degree of inter-evaluator variability as much as possible. Furthermore, the lowest quartile of performance for the overall surgical clerkship was high (85% or lower), which suggests a level of grade inflation present in our institutional grading parameters. This continues to be a widespread phenomenon among medical schools with greater than 25% of institutions reported as assigning more than 40% of students to their highest grading category on the surgical clerkship. This is an issue that both our and other institutions should attempt to combat and continue to find alternative solutions for.

CONCLUSION

This is one of the first studies to directly compare the effect of synchronous teleconference versus recorded didactics on medical student examination performance scores on the surgery clerkship. Overall, medical students receiving live teleconference didactics had higher performance on weekly quizzes and overall clerkship grades. This relationship persisted when comparing only third year medical students. No significant difference was observed in shelf exam performance or placement in the top quartile of clinical evaluation and OSCE parameters among third year students. This suggests that synchronous teleconferences may be more effective than recorded lectures for achieving institutional learning objectives on the surgery clerkship without any negative impact on NBME shelf exam or clinical evaluation parameters. Future prospective studies should be undertaken to determine the most optimal incorporation of teleconference didactics to supplement in-person and recorded sessions to improve medical trainees’ education.

Authors' contribution statement

Study Conception and Design: CT Huerta and RA Saberi. Acquisition of Data: CT Huerta, RA Saberi, and LR Sands. Analysis/Interpretation of Data: CT Huerta. Drafting of Manuscript: CT Huerta. Critical Revision of Manuscript: All authors.

Statements and Declarations

The authors have no relevant disclosures.