Educational Approaches That Enhance Online Clinical Clerkship during the COVID-19 Pandemic.

Objective This cross-sectional national study determined which educational approaches are associated with the effectiveness of online clerkship for medical students. Method A survey was conducted for medical students at 78 medical schools in Japan from May 29 to June 14, 2020. It comprised the following aspects: (a) participants' profiles, (b) number of opportunities to learn from each educational approach (lecture, medical quiz, assignment, oral presentation, observation of a physician's practice, clinical skill practice, participation in interprofessional meetings, and interactive discussions with physicians) in online clerkship, (c) frequency of technical problems, and (d) educational outcome measurement (satisfaction, motivation, knowledge acquisition, skill acquisition, change in self-study time, and understanding of the importance of medical care team). Results Of the 2,640 respondents, 2,594 (98.3%) agreed to cooperate. Ultimately, 1,711 matched our inclusion criteria. All educational approaches but assignments were positively associated with satisfaction and motivation. All educational approaches excluding assignment submission and interprofessional meeting were positively associated with knowledge acquisition. Observation, practice, and interprofessional meeting were positively associated with skill acquisition. Only assignment submission was positively associated with the change in self-study time. Educational approaches excluding medical quizzes were positively associated with understanding the importance of the medical care team. Technical problems were negatively associated with motivation, knowledge acquisition, and skill acquisition. Conclusions Educators should implement various educational approaches, especially observation and practice, even in online clinical clerkship. They also need to minimize the technical problems associated with the Internet, as they reduce the effectiveness of online clerkship.

Introduction

Coronavirus disease 2019 (COVID-19) has significantly affected societies globally, particularly restricting face-to-face interactions (1). COVID-19 has also greatly impacted medical education (2-6), specifically face-to-face clinical clerkship. Some medical schools have been forced to suspend clinical clerkships following a city lockdown or a national emergency declaration (7). Globally, online clinical clerkship was proposed to replace face-to-face clinical clerkship during the COVID-19 pandemic (mid-COVID-19) (7). Various opinions and practical innovations have been presented to make online clinical clerkship a valuable alternative to face-to-face clerkship (8-10). However, which educational approaches can best enhance online clinical clerkship is unclear at present.

This study surveyed Japanese medical students to examine practical educational approaches to improving online clinical clerkship. We hypothesized that educational approaches would enhance the effectiveness of online clinical clerkship mid-COVID-19. Therefore, we investigated which educational approaches were associated with online clinical clerkship effectiveness for medical students as an alternative to face-to-face clerkship.

Materials and Methods

Survey design and data collection

This study adhered to the principles of the Declaration of Helsinki and was approved by the ethics committee of Kyushu University. We conducted a cross-sectional survey study to achieve our research objective.

Our participants were undergraduate medical school students in Japan. The study was conducted from May 29 to June 14, 2020. We investigated medical education for the period from April to May 2020, when a state of emergency was declared in Japan. The questionnaire included an informative letter summarizing the purpose of the research and an informed consent form, which was written briefly and clearly in Japanese to avoid misinterpretation. The questionnaire took 3-5 minutes to complete and was prepared using Google Forms and distributed via the LINE application.

This study's inclusion criteria were students 1) who had switched from face-to-face to online clinical clerkship during COVID-19, 2) who were in their fifth or sixth year (final two years) in a Japanese medical school, and 3) who had completed all the survey questions.

Questions included in the survey

The questionnaire consisted of two parts. In the first part, participants were asked about their profile. The second part of the survey was conducted for medical students who had switched from face-to-face to online clinical clerkship mid-COVID-19. It asked students how they felt about the effectiveness of online clinical clerkship mid-COVID-19 compared with the face-to-face clerkship they had been part of before the COVID-19 pandemic (pre-COVID-19). It also asked them what they experienced via online clinical clerkship and how frequently they experienced it.

Participant profile

The first part of this survey asked about participants' profile details, including their sex, year in medical school, university, and education style. To incorporate the variable of whether or not an area was infected with COVID-19 into the analysis, we categorized the location of universities into two groups according to the number of COVID-19-positive cases as of May 28, 2020 (Fig. 1). We also asked participants about their self-study time pre-COVID-19 (Supplementary material 1).

Figure 1.

Number of COVID-19-positive people in each prefecture of Japan on May 28, 2020. Universities were categorized into two groups: universities in prefectures with high numbers (≥1,000) of COVID-19 patients (red bar) and universities in prefectures that did not have high numbers (<1,000) of COVID-19 patients (blue bar; https://web.sapmed.ac.jp/canmol/coronavirus/japan.html). Specifically, universities located in Tokyo, Osaka, Kanagawa, Hokkaido, and Saitama prefectures (red on the map) were defined as COVID-19-infected areas. COVID-19: coronavirus disease 2019

Exposure and other factors

We examined the extent of students' exposure to factors associated with the effectiveness of online clinical clerkship. We decided which factors should be included in our survey based on the previous literature and recommendations for medical education. We asked about factors such as the department into which students rotated for their online clinical clerkship. Concerning the department, they were categorized into two groups: departments that were considered as difficult to conduct online clinical clerkship and those that did not (11). Specifically, general surgery (12), neurosurgery (13), orthopedics (14), anesthesiology (15), and otolaryngology (16) were categorized as departments for which online clinical clerkship was considered difficult. Factors related to educational approaches included lecture duration (17-19), lecture frequency per week (17,20,21), and opportunities to take quizzes (22,23), submit assignments (24,25), give oral presentations (26,27), observe physician practices (8,28), practice clinical skills (8,28), participate in interprofessional meetings (28-30), and discuss with physicians interactively (28,31). In addition, we incorporated the frequency of technical problems related to the Internet as a variable (32). The details of the choices for each answer are presented in Supplementary material 1.

Outcome measurement (Table 1)

Based on Kirkpatrick's model for assessing the usefulness of medical education and curriculum (30,33-35), we analyzed the following five outcome measurements of the effectiveness of online clinical clerkship: 1) degree of satisfaction, 2) degree of motivation, 3) amount of knowledge acquisition, 4) amount of skill acquisition, and 5) change in self-study time. We could not assess level 4a/4b (clinical processes/benefit to patients) in Kirkpatrick's assessment model, which is the highest level, because a long follow-up time is required to assess the clinical benefit for patients (30,35). In addition to these five outcomes, we incorporated the level of understanding the importance of the medical care team as the sixth outcome (36-38). Although this measurement was not incorporated into Kirkpatrick's evaluation model, an awareness of the importance of the medical care team is one of the important things medical students can learn via clinical clerkship.

We calculated the change in self-study time as the mid-COVID-19 time minus the pre-COVID-19 time. Answers to other questions were based on a 5-point Likert scale ranging from 1 (the level/amount during face-to-face clerkship was much higher than that during online clinical clerkship) to 5 (the level/amount during online clinical clerkship was much higher than that during face-to-face clerkship).

Total number of respondents and those included in this study (Fig. 2)

Of the 2,640 Japanese medical students who viewed the questionnaire, 2,594 (98.3%) consented to cooperate with the study. The sample size was calculated based on a 99% confidence interval and a 5% margin of error. A total of 18,195 students enrolled in the fifth and sixth years of Japanese medical universities in 2020. The 2,594 students who agreed to participate was four-fold more than the required sample size of 642. Respondents with missing answers were excluded. Of the remaining respondents, we included those who answered that they had experienced face-to-face clerkship in hospitals pre-COVID-19 and had switched to online clinical clerkship due to the COVID-19 pandemic. Ultimately, 1,671 responses were analyzed (Fig. 2).

Figure 2.

A flowchart showing the selection of suitable patients for inclusion in this study. We collected responses from 2,640 Japanese medical students. Of the 2,640 respondents, 2,594 (98.3%) consented to cooperate in this study. Respondents with missing answers for their profiles were excluded from the analysis (n=122). In Japan, fifth- and sixth-year students have completed the equivalent years of practice, so we also excluded responses from students in fourth year or below (n=19). The next step was to categorize the responses with respect to the type of education received mid-COVID-19. Sixteen respondents were excluded for inappropriate response content. Of the 2,432 remaining respondents, 1,758 reported that they had experienced face-to-face clerkship in hospitals pre-COVID-19 and had switched to online clinical clerkship due to the COVID-19 pandemic. Of the 1,758 respondents, those with missing responses regarding the content of the education exposed were excluded (n=87). Finally, 1,671 responses were used for the analysis of this study. COVID-19: coronavirus disease 2019, pre-COVID-19: before the COVID-19 pandemic, mid-COVID-19: during the COVID-19 pandemic

Statistical analyses

We used the SPSS ver. 27 software program (IBM, Armonk, USA) for the analysis. A two-sided p value of 0.05 was considered significant.

The linear mixed model analysis was used to determine which factors were associated with each outcome measurement of the effectiveness of online clinical clerkship mid-COVID-19. We assumed that the Likert scale provided linear ordinal data. Fixed effect predictors were (a) sex, (b) year in medical school, (c) location of university, (d) self-study time pre-COVID-19, (e) rotating departments with difficulty in online clinical clerkship, (f) lecture duration, (g) lecture frequency, (h) quiz, (i) assignment submission, (j) student oral presentation, (k) observation of practice, (l) clinical skill practice, (m) participation in interprofessional meeting, (n) interactive discussion with physicians, and (o) technical problems related to the Internet. Intercept and university were treated as random factors.

We conducted a sensitivity analysis to assess the robustness of our results (Supplementary material 2).

Patient and public involvement

Patients and/or the public were not involved in the design, conduct, reporting, or dissemination of this research.

Results

Summary of participants' profiles and their responses

Table 2 presents participants' profiles and the proportion of each answer to every question.

Table 2.

Summary of Participants’ Profiles and Answers.

Participants (n=1,671)
Profile of participants
Sex, No. (%)	Male: 979 (58.6), Female: 692 (41.4)
Year in medical school, No. (%)	5th: 707 (42.3), 6th: 964 (57.7)
Location of university, No. (%)	Infected area: 566 (33.9), Non-infected area: 1,105 (66.1)
Self-study time per day pre-COVID-19 (*1) No. (%)	1: 307 (18.4), 2: 480 (28.7), 3: 418 (25.0), 4: 208 (12.4), 5: 107 (6.4), 6: 60 (3.6), 7: 43 (2.6), 8: 20 (1.2), 9: 8 (0.5), 10: 5 (0.3), 11: 8 (0.5), 12: 2 (0.1), 13: 5 (0.3)
Factors in web-based education
Rotating departments with difficulty teaching online clinical clerkship or not, No. (%)	Yes: 1,104 (66.1), No: 567 (33.9)
Lecture duration (*2) No. (%)	1: 287 (17.2), 2: 54 (3.2), 3: 139 (8.3), 4: 250 (15.0), 5: 406 (24.3), 6: 329 (19.7), 7: 118(7.1), 8: 88 (5.3)
Lecture frequency per week (*3) No. (%)	1: 287 (17.2), 2: 141 (8.4), 3: 166 (9.9), 4: 138 (8.3), 5: 181 (10.8), 6: 139 (8.3), 7: 357 (21.4), 8: 61 (3.7), 9: 50 (3.0), 10: 28 (1.7), 11: 12 (0.7), 12: 63 (3.8), 13: 48 (2.9)
Opportunity to take quizzes (*4) No. (%)	1: 357 (21.4), 2: 240 (14.4), 3: 753 (45.1), 4: 206 (12.3), 5: 115 (6.9)
Opportunity to submit assignments (*4) No. (%)	1: 212 (12.7), 2: 171 (10.2), 3: 473 (28.3), 4: 382 (22.9), 5: 433 (25.9)
Opportunity to give oral presentations (*4) No. (%)	1: 700 (41.9), 2: 370 (22.1), 3: 400 (23.9), 4: 119 (7.1), 5: 82 (4.9)
Opportunity to observe physician practices (*4) No. (%)	1: 1,286 (77.0), 2: 238 (14.2), 3: 87 (5.2), 4: 25 (1.5), 5: 35 (2.1)
Opportunity to practice clinical skills (*4) No. (%)	1: 1,250 (74.8), 2: 263 (15.7), 3: 124 (7.4), 4: 14 (0.8), 5: 20 (1.2)
Opportunity to participate in interprofessional meetings (*4) No. (%)	1: 1,202 (71.9), 2: 244 (14.6), 3: 152 (9.1), 4: 37 (2.2), 5: 36 (2.2)
Opportunity to discuss with physicians interactively (*4) No. (%)	1: 798 (47.8), 2: 382 (22.9), 3: 303 (18.1), 4: 108 (6.5), 5: 80 (4.8)
Frequency of technical problems related to the Internet (*5) No. (%)	1: 575 (34.4), 2: 416 (24.9), 3: 296 (17.7), 4: 297 (17.8), 5: 87 (5.2)
Outcome measurement
Satisfaction level (*6) No. (%)	1: 517 (30.9), 2: 570 (34.1), 3: 348 (20.8), 4: 138 (8.3), 5: 98 (5.9)
Motivation level (*6) No. (%)	1: 543 (32.5), 2: 525 (31.4), 3: 381 (22.8), 4: 108 (6.5), 5: 114 (6.8)
Acquired knowledge (*6) No. (%)	1: 300 (18.0), 2: 503 (30.1), 3: 475 (28.4), 4: 260 (15.6), 5: 133 (8.0)
Acquired clinical skill (*6) No. (%)	1: 1,093 (65.4), 2: 363 (21.7), 3: 146 (8.7), 4: 35 (2.1), 5: 34 (2.0)
Change in self-study time (*7) No. (%):	-12: 1 (0.1), -11: 0 (0), -10: 1 (0.1), -9: 1 (0.1), -8: 1 (0.1), -7: 2 (0.1), -6: 4 (0.2), -5: 2 (0.1), -4: 10 (0.6), -3: 25 (1.5), -2: 45 (2.7), -1: 105 (6.3), 0: 302 (18.1), 1: 321 (19.2), 2: 342 (20.5), 3: 206 (12.3), 4: 129 (7.7), 5: 100 (6.0), 6: 31 (1.9), 7: 13 (0.8), 8: 12 (0.7), 9: 7 (0.4), 10: 6 (0.4), 11: 1 (0.1), 12: 4 (0.2)
Understanding level of the importance of the medical care team (*6) No. (%)	1: 901 (53.9), 2: 414 (24.8), 3: 277 (16.6), 4: 37 (2.2), 5: 42 (2.5)

*1: 13 choices in hourly increments: 1)

*2: 8 choices in 15-minute increments: 1) not at all, 2) ≤15 minutes per lecture, 3) ≥15 minutes and ≤30 minutes per lecture, 4) ≥30 minutes and ≤45 minutes per lecture, 5) ≥45 minutes and ≤60 minutes per lecture, 6) ≥60 minutes and ≤75 minutes per lecture, 7) ≥75 minutes and ≤90 minutes per lecture, and 8) ≥90 minutes per lecture.

*3: 13 choices: 1) not at all, 2) less than once a week, 3) about once a week, 4) about twice a week, 5) about three times a week, 6) about four times a week, 7) about five times a week, 8) about six times a week, 9) about seven times a week, 10) about eight times a week, 11) about nine times a week, 12) about 10 times a week, and 13) over 10 times a week.

*4: 5-point Likert scale: 1) the opportunities during face-to-face clerkship were much more than during online clinical clerkship to 5) the opportunities during online clinical clerkship were much more than during face-to-face clerkship.

*5: 5-point Likert scale: 1)=not at all, to 5)=very frequent.

*6: 5-point Likert scale: 1) the level/amount during face-to-face clerkship was much higher than that during online clinical clerkship to 5) the level/amount during online clinical clerkship was much higher than that during face-to-face clerkship.

*7: Subtraction of self-study time during face-to-face clerkship from that during online clinical clerkship

*pre-COVID-19: before the COVID-19 pandemic

Of the 1,671 participants included in the analysis, 692 (41.4%) were women, and 964 (57.7%) were in their final year of medical education. Overall, 1,105 (66.1%) students attended universities located in not COVID-19-infected areas. Concerning self-study time, 1,205 (72.1%) students studied fewer than 3 hours per day during the pre-COVID-19 clerkship period. Regarding department rotation, 1,104 (66.1%) students rotated within departments with difficulty in conducting online clinical clerkship. The most common lecture duration was ≥45 minutes and ＜60 minutes, with 406 (24.3%) students' responses falling within this time range. The most common lecture frequency was 5 times a week, with 357 (21.4%) students selecting this option. The number of students who believed that online clinical clerkship was more effective than face-to-face clerkship regarding the 5 outcome measures analyzed were as follows: satisfaction, 236 (14.2%); motivation, 208 (13.3%); acquired knowledge, 393 (23.6%); acquired skill, 69 (4.2%); and understanding the importance of the medical care team, 79 (4.9%).

Factors associated with the degree of satisfaction with online clinical clerkship

Supplementary material 3 shows the results of the mixed model analysis for degree of satisfaction with online clinical clerkship. The positive factors associated with satisfaction were gender (estimate=-0.12; p=0.012), year in medical school (estimate=0.21; p＜0.001), lecture frequency (estimate=0.04; p＜0.001), quizzes (estimate=0.10; p＜0.001), student oral presentation (estimate=0.08; p=0.003), observation of practice (estimate=0.17; p＜0.001), clinical skill practice (estimate=0.17; p＜0.001), participation in interprofessional meeting (estimate=0.15; p＜0.001), and interactive discussion with physicians (estimate=0.15; p＜0.001). The only associated negative factor was self-study time pre-COVID-19 (estimate=-0.03; p=0.028).

Factors associated with the degree of motivation for online clinical clerkship

Supplementary material 4 shows the results of the mixed model analysis for degree of motivation for online clinical clerkship. The positive factors associated with motivation were gender (estimate=-0.11; p=0.037), year in medical school (estimate=0.15; p=0.006), lecture frequency (estimate=0.02; p=0.033), quizzes (estimate=0.11; p＜0.001), student oral presentation (estimate=0.11; p＜0.001), observation of practice (estimate=0.19; p＜0.001), clinical skill practice (estimate=0.15; p＜0.001), participation in interprofessional meeting (estimate=0.13; p＜0.001), and interactive discussion with physicians (estimate=0.13; p＜0.001). The associated negative factors were self-study time pre-COVID-19 (estimate=-0.03; p=0.021) and technical problems related to the Internet (estimate=-0.07; p＜0.001).

Factors associated with the amount of acquired knowledge and clinical skill

Supplementary material 5 shows the results of the mixed model analysis for the amount of knowledge acquired in online clinical clerkship. The positive factors associated with acquired knowledge were year in medical school (estimate=0.12; p=0.028), lecture frequency (estimate=0.04; p＜0.001), quiz (estimate=0.17; p＜0.001), student oral presentation (estimate=0.09; p=0.001), observation of practice (estimate=0.22; p＜0.001), clinical skill practice (estimate=0.09; p=0.032), and interactive discussion with physicians (estimate=0.11; p＜0.001). The associated negative factors were self-study time pre-COVID-19 (estimate=-0.03; p=0.023) and technical problems related to the Internet (estimate=-0.05; p=0.011).

Supplementary material 6 shows the results of the mixed model analysis for the amount of clinical skill acquired in online clinical clerkship. The associated positive factors were rotating departments with difficulty in online clinical clerkship (estimate=0.08; p=0.024), observation of practice (estimate=0.26; p＜0.001), clinical skill practice (estimate=0.52; p＜0.001), and participation in interprofessional meeting (estimate=0.07; p=0.009).

Factors associated with the change in self-study time

Supplementary material 7 shows the results of the mixed model analysis for the change in self-study time. The associated positive factors were year in medical school (estimate=1.06; p＜0.001) and assignment submission (estimate=0.10; p=0.026). The only associated negative factor was self-study time pre-COVID-19 (estimate=-0.56; p＜0.001).

Factors associated with understanding the importance of the medical care team

Supplementary material 8 shows the results of the mixed model analysis for the understanding level of the importance of the medical care team. The associated positive factors were lecture frequency (estimate=0.02; p=0.030), assignment submission (estimate=0.04; p=0.031), student oral presentation (estimate=0.07; p＜0.001), observation of practice (estimate=0.25; p＜0.001), clinical skill practice (estimate=0.24; p＜0.001), participation in interprofessional meeting (estimate=0.16; p＜0.001), and interactive discussion with physicians (estimate=0.08; p＜0.001). The only associated negative factor was longer self-study time pre-COVID-19 (estimate=-0.02; p=0.034).

Fig. 3 summarizes the 95% confidence intervals of the mixed model estimate of each online clinical clerkship factor at each level based on Kirkpatrick's assessment model.

Figure 3.

Graphic summary of the 95% confidence intervals of the estimates of the independent variables in the linear mixed model analysis for each outcome. Variables with red plots and bars, whose 95% confidence intervals of the estimate calculated with the linear mixed model analysis were over 0, represent the positive factors associated with each outcome. Variables with blue plots and bars, whose 95% confidence intervals of the estimate were under 0, represent the negative factors associated with each outcome. (A) Degree of satisfaction with online clinical clerkship compared with previous face-to-face clerkship. (B) Degree of motivation for online clinical clerkship compared with previous face-to-face clerkship. (C) Amount of knowledge acquired in online clinical clerkship compared with previous face-to-face clerkship. (D) Amount of skill acquired in online clinical clerkship compared with previous face-to-face clerkship. (E) Change in self-study time between during online clinical clerkship and during face-to-face clerkship. (F) Degree of understanding of the importance of the medical care team in online clinical clerkship compared with previous face-to-face clerkship. COVID-19: coronavirus disease 2019

Results of a sensitivity analysis

Supplementary material 9 summarizes the 95% confidence intervals of the coefficients in the generalized linear mixed model analysis. The sensitivity analysis produced similar results to the original analysis of our study, thus confirming the robustness of our results.

Discussion

No previous studies have examined educational approaches that enhance the effectiveness of online practice. Our national-level cross-sectional study is the first to identify educational approaches in online clinical clerkship using Kirkpatrick's assessment model with 1,671 medical students in 87 medical schools across Japan.

Our study showed that lecture frequency contributes to enhancing online clinical clerkship more than lecture duration. When transitioning from face-to-face lectures to an online form, a 3-hour face-to-face lecture needs to be summarized in a 30-minute video, as research shows that students get bored with videos if they are too long (39). In our study mid-COVID-19, lecture frequency was found to be related to satisfaction, motivation, the acquisition of medical knowledge, and understanding the importance of the medical care team. In short, our results support the above-mentioned research finding. In addition, an increased lecture frequency may have helped students feel more connected to society mid-COVID-19 (40). Our results indicate that it is more effective to divide lectures into small sessions and increase their frequency than to deliver a singly long lecture all at once in online clinical clerkship mid-COVID-19.

Quizzes, assignment submissions, and student oral presentations contributed in different manners to online clinical clerkship's effectiveness, although they all reflected knowledge output. Quizzes contributed to satisfaction, motivation, and knowledge the most among the three output activities (estimates: 0.10, 0.11, 0.17). Student oral presentations also contributed to satisfaction, motivation, and knowledge. However, student oral presentations differed from quizzes, as they contributed to an understanding of the importance of the medical care team. Unlike the other two, assignment submission contributed to an increase in self-study time, indicating that self-study is encouraged by assignments. Conversely, it did not contribute to satisfaction, motivation, knowledge acquisition, or skill acquisition. Considering the contribution of each of the above factors to online clinical clerkship, educators and universities should incorporate all three factors, keeping in mind the features of each educational approach. Specifically, student oral presentations can only be undertaken by one person/team at a time and are time-limited, as they can only be commenced once everyone is online and present. Quizzes, however, have fewer time restrictions, as they can be administered to all medical students at once and can be completed by each student at their own pace. Assignment submissions are less time-sensitive in this respect, similar to quizzes, but the burden on physicians to provide appropriate feedback to medical students is greater for assignment submissions than with the other two educational approaches. Therefore, it is possible that adequate feedback was not able to be provided to medical students, as physicians were under immense strain during the early phase of the pandemic. This may also be why assignment submissions did not contribute to medical student satisfaction and motivation.

Observation of practice and clinical skill practice are important approaches in both online clinical and face-to-face clerkship. These approaches have traditionally only been feasible with on-site practice (11), but mid-COVID-19, educators have attempted to provide them online (8-10). Our results showed that observation and practice similarly contributed to satisfaction, motivation, and the understanding of the importance of the medical care team (estimate, observation vs. practice: 0.17 vs. 0.17, 0.19 vs. 0.15, 0.25 vs. 0.24). These factors also contributed to knowledge and skill. However, observation contributed more to the acquisition of knowledge than practice (estimate: 0.22 vs. 0.09), and practice contributed more to the acquisition of skill than observation (estimate: 0.26 vs. 0.52). Participation in interprofessional meetings is also an important activity in online clinical clerkship. Our results showed that it contributed to satisfaction, motivation, the acquisition of skill, and understanding of the medical care team's importance. All three approaches were found to be important educational content in online clinical clerkship. Observation and practice contributed to the greatest number of educational outcomes. Considering the features of each educational approach online, practice would have been restricted by technical issues. Specifically, practice often requires the presence of certain people and special equipment, such as physical examination practices that use models; it can thus be difficult to practice properly in an online setting. Furthermore, for observation, issues such as the protection of patient privacy and the reliability of online tools must be considered when observing online. Finally, regarding participation in interprofessional meetings, in addition to ensuring the protection of patient privacy, the implementation of interprofessional meetings itself may have been restricted at each facility during the early phase of the pandemic.

Interactive discussions with a physician play a great role in clinical clerkship and online clinical clerkship. They contributed to satisfaction, motivation, knowledge acquisition, and understanding of the medical care team's importance. Interactive discussions with physicians would be easier to perform on an online setting than observation, practice, and participation in interprofessional meetings.

Online technical problems are some of the most feared obstacles in web education. They negatively contributed to motivation and knowledge acquisition. Educators and universities must implement interactive discussions between physicians and medical students and prevent technical problems related to the Internet.

Regarding profile factors, women were less satisfied and motivated by online clinical clerkship than men. A previous study concluded that men were more likely to express higher confidence in self-assessments than women (34). Our study may support this conclusion, although there were no significant differences between men and women in other outcomes. The final year in medical school was also positively associated with satisfaction, motivation, acquired knowledge, and change in self-study time. An interpretation of this observation is that students in their final year of medical school must be fully prepared to become doctors. Therefore, they put in the most effort to learn mid-COVID-19 (41). Their longer self-study time pre-COVID-19 was negatively associated with satisfaction, motivation, acquired knowledge, change in self-study time, and understanding of the importance of medical care team. Students who spend too long self-studying may have higher medical education expectations than those who spend a shorter time.

Regarding changes in self-study time, students who had spent less time on self-study pre-COVID-19 increased their self-study time mid-COVID-19. This results may have been influenced by the restriction of extracurricular activities (e.g. sports, part-time job) due to the lockdown. Although departments such as surgery reportedly find it difficult to educate students on the web because of the focus on hospital visits (11), rotating such departments helped improve students' clinical skills in our study, possibly due to these departments implementing innovations in educating students online (e.g. video observations of surgery) (8-10).

Methodological considerations

Several limitations associated with the present study warrant mention. First, this cross-sectional questionnaire survey captures subjects' experiences and perceptions at only one point in time, retrospectively. Thus, recall bias is unavoidable. In particular, with respect to the level 2 Kirkpatrick's assessment model, objective outcome measurements should be made pre-COVID-19, and a quantitative comparison should be conducted using the same outcome measurement mid-COVID-19. However, because the COVID-19 pandemic was unexpected, quantitative measurements could not be made pre-COVID-19. For this reason, our study design had to be a cross-sectional survey. Even though this study is based on self-reported outcomes, our results could contribute to future online education because self-reported outcomes, especially satisfaction and motivation, are important factors affecting the effectiveness of education. In fact, one of the interests of medical educators is how to maintain or enhance students' satisfaction and motivation for online education (42-44). Second, we focused on medical students in Japan. Future research must consider the applicability of these results to other countries. Online clinical clerkship may be less useful in countries with limited healthcare resources or a lack of an Internet infrastructure. The impact of COVID-19 also varies across countries. Our study compared the utility of online clinical clerkship mid-COVID-19 and face-to-face clerkship pre-COVID-19. Therefore, we excluded students who had not experienced face-to-face clerkship pre-COVID-19. Further studies are needed to determine whether or not similar results are found for students without any experience of face-to-face clerkship pre-COVID-19. In addition, some communities continued face-to-face clerkship in hospitals mid-COVID-19, so studies evaluating the effectiveness of clinical clerkship mid-COVID-19 will be necessary. Finally, we did not account for all confounding factors. For example, the educational system and content differ among universities and departments. In addition, the departments in which students rotated differed between pre- and mid-COVID-19 and could not be compared directly.

Conclusion

The present web-based national-level survey for 1,671 medical students clarified that educational approaches, especially observation and practice, made online clinical clerkship efficient. Our results may help improve the online clinical clerkship program. Educators should implement various educational approaches, even in online clinical clerkship. Furthermore, they should aim to minimize technical problems, as these reduce the effectiveness of online clinical clerkship.

The authors state that they have no Conflict of Interest (COI).

Financial Support

This work was supported by JSPS KAKENHI (Grant Number 17H04097).

Detailed description of the choices for each answer.

Click here for additional data file.

Sensitivity analysis.

Click here for additional data file.

Linear mixed model analysis to characterize the satisfaction level of medical students with online clerkship (level 1 in Kirkpatrick’s assessment model).

Click here for additional data file.

Linear mixed model analysis to characterize the motivation level of medical students for online clerkship (level 2a in Kirkpatrick’s assessment model).

Click here for additional data file.

Linear mixed model analysis to characterize the amount of knowledge acquired in online clerkship (knowledge part of level 2b in Kirkpatrick’s assessment model).

Click here for additional data file.

Linear mixed model analysis to characterize the amount of clinical skill acquired in online clerkship (skill part of level 2b in Kirkpatrick’s assessment model).

Click here for additional data file.

Linear mixed model analysis to characterize the change in self-study time during online clerkship compared with during previous face-to-face clerkship pre-COVID-19 (level 3 in Kirkpatrick’s assessment model).

Click here for additional data file.

Linear mixed model analysis to characterize the understanding level of the importance of the medical care team in online clerkship.

Click here for additional data file.

Graphic summary of the 95% confidence interval of the coefficients of the independent variables in each outcome with the sensitivity analysis.

Click here for additional data file.

Table 1.

Educational Outcomes Based on Kirkpatrick’s Assessment Model.

Kirkpatrick’s assessment model (34,36)		Outcome measurement in our study
Level 1	Learners’ satisfaction	Satisfaction level with online clinical clerkship compared with previous face-to-face clerkship
Level 2a	Changes in learner attitudes	Motivation level for online clinical clerkship compared with previous face-to-face clerkship
Level 2b	Measures of learner knowledge and skill	1. Amount of knowledge acquired in online clinical clerkship compared with previous face-to-face clerkship
		2. Amount of skill acquired in online clinical clerkship compared with previous face-to-face clerkship
Level 3	Changes in learner behavior	Change in self-study time between during face-to-face clerkship and during online clinical clerkship
Level 4a	Changes to clinical processes	NA
Level 4b	Benefits to patients	NA
NA		Understanding level of the importance of the medical care team in online clinical clerkship compared with previous face-to-face clerkship

NA: not applicable