Evaluating knowledge retention and perceived benefits of medical webinar for professional development among Indonesian physicians.

PURPOSE: To assess knowledge retention of physicians after participating in a webinar series and its perceived benefits on daily practice and career development.
METHODS: The webinar series comprised six separate webinars about daily practice. Online questionnaires were sent to all participating physicians via email 3 months after the webinars. The questionnaire assessed knowledge retention through the difference between initial and follow-up post-test as well as the webinar series' benefits on daily practice and career development. Participants' demographic information, including their age, gender, education, year of graduation, and work details, were collected to compare outcomes between demographic groups.
RESULTS: A total of 689 responses were gathered, and 622 were analyzed. At follow-up, the median knowledge score was significantly lower than the initial median knowledge score (Z=-6.973, p<0.001). Participants' perception of the webinar series' benefits on daily practice and career development was very positive. A significant weak negative correlation was found between age and knowledge score at follow-up (rs=-0.157, p<0.001). Physicians who graduated more recently and worked for less than 3 years scored significantly higher on knowledge tests at follow-up. Meanwhile, perception score towards webinar series' benefits on daily practice was significantly higher among physicians with more extended work history. Male physicians scored significantly higher on perception score towards webinar series' benefits on career development.
CONCLUSION: Online continuing medical education programs like webinars can encourage physicians to maintain their competence, but further research on improving knowledge retention over time is necessary. Overall, physicians perceived webinars to be beneficial for their professional development.

Introduction

The profession of a physician is inherently demanding as it deals with the quality of life of another human being. Physicians everywhere are morally obliged to keep up with the latest information and knowledge on treatments and diseases. Indeed, the four principles of health care ethics, including beneficence and non-maleficence, have been widely adopted to guide physicians in their clinical decision-making. Various academic institutions and health experts agree that continuing medical education (CME) is one effective method that enables physicians to update their knowledge and skills throughout their careers continuously. This term describes educational activities aimed at physicians to maintain and improve their professional knowledge and skills, ultimately leading to better patient care. A review by Cervero and Gaines [1] proved that CME is strongly associated with better physician performance and patient health outcomes.

A study by Cullen et al. [2] strongly suggested that CME must evolve to suit the needs of modern physicians in today’s digital age. Nowadays, physicians are busier than ever as they are increasingly overwhelmed with long work hours, bureaucratic tasks, financial issues, stressful on-call duties, and other work-life demands [2,3]. The ongoing coronavirus disease 2019 (COVID-19) pandemic has further worsened the situation, as it has led to staff shortage in many hospitals and eventually higher workload and risk of burnout for the remaining health care providers [4]. The problems as mentioned earlier are particularly evident in low-to-middle-income countries, where a physician’s workload is more physically and mentally draining due to a shortage of physicians. In a 2014 report by World Health Organization, most provinces in Indonesia (87.8%) had a lower ratio of physicians per population than the recommended 1:1,000 ratio [5]. Therefore, as physicians struggle to meet patients’ rising needs, there is an urgent need for a more accessible CME delivery method that requires less effort, cost, and commute time. Modern CME must shift from implementing solely traditional, face-to-face courses to integrating online and interactive educational activities. An example of online CME activity is a webinar, an online training program that utilizes the videoconferencing platform to offer synchronous learning.

Online CME offers excellent flexibility for physicians to set their schedule and pace of learning. Moreover, as many online courses are often recorded, saved, and uploaded, physicians can conveniently access them anytime and review the materials. Data from Mayo Clinic Cardiovascular Review for Cardiology Boards and Recertification showed that while physicians spent up to 50% less time engaging with online courses, their knowledge significantly increased by three-fold from pre-test to post-test [2]. Previous studies also supported the effectiveness of online CME in terms of physicians’ satisfaction, knowledge gain, and behavioral changes [6,7]. Interestingly, a recent study concluded that physicians were open to the idea of online CME programs as a substitute for face-to-face courses as they encouraged more participation in CME, promoted more engagement, and reduced the need and time to travel [8].

Although the effectiveness of online CME can be assessed based on participants’ knowledge gain and satisfaction, what ultimately matters to patient care is how much new knowledge is retained for long-term memory and how this knowledge translates into changes in daily practice and personal development. Physicians’ newly acquired knowledge following online training can be significantly lost in as little as several days to 2 months [9]. In addition, teaching hospitals and medical educators should utilize CME to improve physicians’ knowledge and assist them in enhancing their clinical performance and career development [10]. This study aims to analyze knowledge retention of physicians 3 months after participating in a webinar training and how they perceive the webinar series’ benefits on their daily practice and career development.

Methods

This research is a follow-up study from our previous research that evaluated participants’ satisfaction and learning outcomes following the i-CORPS (Interactive Course for General Practitioners) webinar series held by Universitas Indonesia Hospital [11]. The hospital is a teaching hospital that is part of a prominent medical school in Indonesia, the Faculty of Medicine, Universitas Indonesia. Considering the problems that CME faces amid the pandemic, the purpose of the i-CORPS webinar series was to explore contact-free teaching through online webinars and evaluate its effectiveness. Its primary target audience was physicians but was also open to other health professionals around Indonesia. The webinar series consisted of six webinars with different themes conducted separately, live-streamed on the Zoom videoconferencing platform, and uploaded to YouTube afterward for future access. The webinar series was conducted between September and October 2020, with each webinar being held on 2 consecutive days for 1 hour each. Each webinar covered four or five subtopics, and the main themes included COVID-19 clinical approach, diseases affecting the special senses and nervous system, emergency cases management, nutrition, and metabolic disorders, medico-legal medicine, and other related fields. The Ethical Committee of Universitas Indonesia Hospital granted ethical clearance for this study (005/SKPE/KKO/2020/00). This study evaluates the knowledge retention of physicians who participated in the webinar series, how their daily practice has been impacted over a 3-month timeframe, and how their career development for the future has been affected.

1. Sample

The webinars were openly marketed to various health professions, including physicians, pharmacists, nurses, medical students, and other health providers residing in Indonesia. However, only those who were physicians and gave consent were included as participants for this study. Participation was voluntary. The scope of the sample recruitment was nationwide. Since each webinar was delivered and assessed independently, each subject could participate in as many webinars as they wanted to. Hence, if a physician registered as a participant in two webinars, he or she would be asked to complete two separate questionnaires.

2. Questionnaire and data collection

This follow-up study used online self-administered questionnaires that were sent to all eligible participants via email. A total sampling method was used. A questionnaire was created for each webinar and comprised three sections: section 1 (questions to assess knowledge retention), section 2 (webinar series’ benefits on daily practice), and section 3 (webinar series’ benefits on career development). Each questionnaire was administered 3 months after the date of each respective webinar. To evaluate the knowledge retention of participants, the same questions from each webinar’s post-test were included in the questionnaire. However, the list of knowledge questions for each webinar differed from one another as it was determined by the themes and subtopics covered. The knowledge test section was in the form of four-options multiple-choice questions with one correct answer. To obtain the final score for the knowledge test, each correct answer was given a score of 10 while the wrong answer was given a score of 0, and the points were then accumulated for each subject. This new score would then be compared with each subject’s post-test score 3 months ago to see any significant difference.

Meanwhile, in sections 2 and 3, the questionnaire included six questions about how the participants perceived the webinar series’ benefits on daily practice and six questions about how they perceived the webinar series’ benefits on career development. The questions were in the form of a 6-point Likert scale (1=very unsatisfied/unhelpful/unlikely; 6=very satisfied/helpful/likely) and adapted from “Handbook for graduate tracer studies” developed by Schomburg [12]. To obtain each subject’s perception score towards benefits of the webinar series on daily practice, the sum score from all six questions was divided by 36 and multiplied by 100. The same scoring formula was used to obtain each subject’s perception score towards the benefits of the webinar series on career development. Personal information about the participants’ age, gender, education, location of residence, graduation year as a general practitioner, work unit, and other work details was also asked. Overall, the questionnaire’s content for each webinar was identical to one another except for the knowledge test section. The questionnaire had been validated with an inter-item correlation score between 0.543–0.816 and a reliability score of 0.981.

3. Data analysis

Data were analyzed using IBM SPSS ver. 20.00 software (IBM Corp., Armonk, USA). Descriptive analysis was performed to describe the participants’ demographics and distribution of responses using count (N) and percentage (%). Testing for normality of the data was performed by looking at Kolmogorov-Smirnov test and skewness. If the Kolmogorov-Smirnov test was significant and skew values were not close to zero, the data would be considered not normally distributed. Hence, median values alongside their minimum–maximum values were presented, and non-parametric tests were chosen to analyze the data [13]. Wilcoxon test was used to assess the knowledge retention by evaluating the difference between initial post-test scores (shortly completed after the webinar) and follow-up post-test scores (completed 3 months after the webinar ended). Mann-Whitney U-test was used to compare knowledge test score at follow-up, perception score towards benefits of webinar series on daily practice, and perception score towards benefits of webinar series on career development between different demographic groups. The term “demographic group” refers to a subset of the general population categorized by a specific socio-demographic variable, such as gender. Spearman rank correlation test was used to assess the correlation between participant’s age and knowledge test score at follow-up. The cut-off level selected for determining statistically significant differences was set to p<0.05.

Results

1. Demographic information

Six hundred eighty-nine responses were gathered from all questionnaires, but 67 were excluded due to incomplete information. Hence, the total data analyzed was 622. Most of the study participants were female (66.6%), graduated in 2009 or after (70.3%), and had an undergraduate medical degree and/or general practitioner education level (90.2%). The distribution of participants was dominated by physicians who work in a health facility (94.2%) with the length of work for 3 years or less (51.4%). Complete demographic information is shown in Table 1.

Table 1.

Demographic Distribution of the Study Participants

Variable	No. (%)
Gender
Female	414 (66.6)
Male	208 (33.4)
Education[a)]
Undergraduate and/or general practitioner	561 (90.2)
Postgraduate	61 (9.8)
Year of graduation
Graduated in 2008 or prior	185 (29.7)
Graduated in 2009 or after	437 (70.3)
Type of work
Clinician	588 (94.5)
Non-clinician	34 (5.5)
Place of work
Health facility	586 (94.2)
Not health facility[b)]	36 (5.8)
Length of work
3 years or less	320 (51.4)
More than 3 years	302 (48.6)

Undergraduate and/or general practitioner education level refers to participants who had graduated with either bachelor of medicine or doctor of medicine (M.D.). In Indonesia, M.D. with a practicing license is equivalent to the general practitioner level. Postgraduate education level refers to participants who graduated with higher qualifications, including specialist physicians, consultants, physicians with a master’s degree, and physicians with Ph.D.

Examples of physicians who do not work in a health facility include those who work in governmental offices, universities, insurance companies, and telemedicine platforms.

2. Knowledge retention

The median knowledge test score at the 3-month follow-up survey was 10 points lower (60.00) than the initial median knowledge test score (70.00), and the difference was statistically significant (p<0.001) (Table 2).

Table 2.

Knowledge Retention Assessment

Tests	Median (min–max)	p-value
Knowledge test score–initial	70.00 (10.00–100.00)	<0.001[*]
Knowledge test score–follow-up	60.00 (10.00–100.00)

p<0.05; Wilcoxon test was used.

3. Perceived benefits of webinar series on daily practice and career development

Overall, the participants perceived that the webinar series positively impacted their daily practice and career development. The median score was 97.22 for perception on webinar series’ benefits on daily practice and 94.44 for perception on webinar series’ benefits on career development (Table 3). More details about the distribution of responses can be found in Supplements 1 and 2.

Table 3.

Participants’ Perception of the Webinar Series’ Benefits

Tests	Median (min–max)
Perception score towards webinar series’ benefits on daily practice	97.22 (61.11–100.00)
Perception score towards webinar series’ benefits on career development	94.44 (41.67–100.00)

4. The difference in scores between demographic groups

Spearman rank correlation test showed a statistically significant weak negative correlation between participants’ age and knowledge test score at follow-up (r s=-0.157, p<0.001). The older a physician was, the lower he or she scored on the knowledge test at follow-up. Comparison of scores between other demographic groups using bivariate analysis is summarized in Table 4. Statistically significant differences in knowledge test scores at 3-month follow-up were found between different groups based on graduation and length of work. Physicians who graduated more recently, in 2009 or after, had a higher median score (62.50) than those who graduated in 2008 or prior (60.00) (p=0.040).

Table 4.

Bivariate Analysis to Compare Scores against Demographic Groups

Variable		Knowledge test score at follow-up		Perception score towards webinar series' benefits on daily practice		Perception score towards webinar series' benefits on career development
		Median (min—max)	p-value	Median (min—max)	p-value	Median (min—max)	p-value
Gender			0.082		0.500		0.023[*]
	Female	60.00 (10.00-100.00)		97.22 (61.11-100.00)		91.67 (50.00-100.00)
	Male	62.50 (20.00-100.00)		97.22 (61.11-100.00)		97.22 (41.67-100.00)
Education			0.079		0.469		0.297
	Undergraduate and/or general practitioner	60.00 (10.00-100.00)		97.22 (61.11-100.00)		94.44 (41.67-100.00)
	Postgraduate	60.00 (10.00-90.00)		100.00 (61.11-100.00)		100.00 (52.78-100.00)
Year of graduation			0.040[*]		0.201		0.661
	Graduated in 2008 or prior	60.00 (10.00-100.00)		100.00 (66.67-100.00)		94.44 (41.67-100.00)
	Graduated in 2009 or after	62.50 (10.00-100.00)		94.44 (61.11-100.00)		94.44 (50.00-100.00)
Type of work			0.054		0.555		0.260
	Clinician	60.00 (10.00-100.00)		97.22 (61.11-100.00)		94.44 (41.67-100.00)
	Non-clinician	72.50 (10.00-100.00)		98.61 (75.00-100.00)		100.00 (75.00-100.00)
Place of work			0.724		0.323		0.386
	Health facility	60.00 (10.00-100.00)		97.22 (61.11-100.00)		94.44 (41.67-100.00)
	Not health facility	60.00 (10.00-100.00)		100.00 (75.00-100.00)		98.61 (75.00-100.00)
Length of work			<0.001[*]		0.047[*]		0.422
	3 years or less	70.00 (10.00-100.00)		94.44 (61.11-100.00)		91.67 (61.11-100.00)
	More than 3 years	60.00 (10.00-100.00)		100.00 (61.11-100.00)		94.44 (41.67-100.00)

p<0.05; Mann-Whitney U-test was used.

Similarly, physicians who had only worked for 3 years or less had a higher median score (70.00) than those who had worked longer than 3 years (60.00) (p<0.001). On the other hand, regarding perception score towards webinar series’ benefits on daily practice, physicians who had worked for longer than 3 years scored statistically significantly higher (100.00) than those who had not (94.44) (p=0.047). Regarding perception score towards webinar series’ benefits on career development, male physicians scored statistically significantly higher (97.22) than female physicians (91.67) (p=0.023). Other differences in scores between groups were not statistically significant.

Discussion

Despite concerns that any new knowledge can be quickly forgotten, especially after an extended period, there has been limited research into knowledge retention following online learning in the medical field. To our knowledge, this is the first study in Indonesia that investigated knowledge retention of physicians after online CME and its impact on their daily practice and career development. The study found a significant loss of knowledge among physicians 3 months after acquiring it from the webinar series. This result was consistent with Bell et al. [9], who observed significant retention loss after 3–8 days and no significant retention 55 days after the physicians completed an online tutorial on diabetes care guidelines. In one of their older studies, the physicians only retained 40% of their initial knowledge gains 4–6 months after completing an online tutorial on postmyocardial infarction care [14]. The retention loss observed in our study participants was not as extreme, but it remained an essential factor to address in order to improve online CME in the future. However, it is essential to note that this study’s findings might not be directly comparable to other studies’ findings on knowledge retention as the type of online CME might differ regarding the method of delivery and materials covered.

The reason for retention loss in our study participants might be the nature of the i-CORPS webinar series, which was delivered as a one-time event without a follow-up refresher course. A systematic review by Cook et al. [15] found four aspects associated with better learning outcomes in online learning for health professions education: interactivity, practice exercises, repetition, and feedback. The webinar series had incorporated three of these elements in its delivery, but the repetition of the materials was not well-addressed. All participants were given access to revisit the webinar materials online anytime on Youtube, but this was not reinforced and remained optional for them. Meanwhile, a randomized controlled trial involving 44 physicians assigned to an online CME course showed that the participants benefited from the repetition of the course, which was conducted 1 month after. The repetition was in the form of a 45-minute instructor-led web conference, and it was joined by 85% of the participants. A small but significant increase from participants’ scores at the post-test immediately after the course to participants’ scores at 12 weeks after the course [16]. Additionally, ensuring interactivity in learning is arguably one of the biggest challenges in conducting an online CME course. Although the i-CORPS webinar series has anticipated this problem by encouraging the instructors to be engaging, for example, using the “chat” and “polling” features in Zoom [11], this remains a limitation worth considering for future online CME courses. An interactive learning experience was associated with better knowledge retention, and failure to establish a two-way communication involving feedback may negatively affect participants’ attention span and retention level [17]. In a face-to-face educational course, the instructor can better engage with the participants since they are in the same room. Participants are also able to provide immediate feedback or ask questions directly to the instructor. However, in an online course, such two-way communication between instructor and participants can be greatly restricted due to multiple reasons such as lack of physical presence and poor Internet connectivity. A previous study found that students mainly struggled to pay attention and be motivated in an online course as they viewed the instructor on screen as “invisible”, “absent”, “passive”, and a “hybrid between a human and a computer”. This physical separation made it difficult for students to stay focused, get to know their instructor personally, and share the instructor’s enthusiasm for the subject [18]. Regarding connectivity, poor Internet connection can lead the videoconferencing platform to lag or freeze, therefore causing the audio or image to become out of sync. This eventually causes misunderstandings and communication delays between the instructor and participants.

Physicians’ age, education level, type of work, and place of work did not appear to influence their knowledge test scores at the follow-up survey. On the other hand, knowledge test scores at follow-up significantly differed by physicians’ age, year of graduation, and length of work. It was not surprising that physicians tended to absorb new knowledge much slower as they grew older. This phenomenon has significant implications on physicians’ retirement age, which has sparked debate over recent years about whether it must be regulated and its impact on the medical workforce. Some worry that aging physicians are more prone to making errors that disadvantage the patients, but others believe that allowing them to remain in practice can help mitigate the growing physician shortage in many countries [19-21].

Nevertheless, since age is just one of the many factors that affect older physicians’ performance, it must not be regarded as the sole predictor of competence [19-21]. One way to ensure aging physicians maintain their competence is by reinforcing CME and making it more accessible and convenient, which can be achieved by promoting online learning with the i-CORPS webinar series as an example. All webinars in the series were live-streamed on Zoom and recorded for future access, reducing the need for participants to spend excessive time, money, and travel on CME. This enabled not only younger but also older physicians to participate with great convenience. However, it must be noted that older physicians may feel uncomfortable or unfamiliar when using digital technologies, including video conferencing platforms. If this limitation is not correctly addressed, online CME can be counterproductive and hinder older physicians from participating.

Interestingly, this study also found that recently graduated physicians with less than 3 years working period scored significantly higher on the knowledge test than the more senior physicians. While it could be argued that fresh medical graduates might score higher because they could still remember a sheer amount of knowledge from their medical school years, we initially expected the senior physicians who had worked longer to score higher as they had more clinical exposure to patients in daily practice. The questions in the knowledge test covered clinical aspects of various diseases encountered in daily practice, which included their etiology, pathophysiology and pathogenesis, signs and symptoms, diagnosis, management, and prevention. This result further supported the findings in our previous study, in which we also discovered that physicians who graduated more recently (2015–2020) had higher average post-test scores than physicians who graduated in 2014 or prior in all six webinars [11]. We hypothesized that most recently graduated physicians could engage better with the webinar series and hence absorbed the material better than senior physicians. Hence, the importance of regularly updating clinical knowledge by participating in structured CME must be re-emphasized to all physicians regardless of their rank, seniority, or work history.

Although physicians with more extended work history scored lower on the knowledge test at follow-up, they perceived the webinar series’ benefits on daily practice more positively than those who had just begun medical practice. Those with more extended work history presumably had more experience in managing patients and could better apply the newly acquired knowledge from the webinar series to cases encountered in daily practice. Knowledge gain must not be used as the only parameter for determining the success of online CME since how relevant and applicable; it is to physicians’ work is as important, if not more so. Fortunately, there may be a mutualistic relationship between perception of online learning experience and knowledge gain. Cook and Thompson [22] implied that having a favorable perception of online learning was positively associated with improved physicians’ knowledge. Thus, future online CME must evaluate how well participants improve their knowledge, how satisfied they are with the course, and how they perceive its benefits in daily practice.

In this study, male physicians perceived the webinar series’ benefits on career development more positively than their female counterparts. In other words, male physicians could be putting more thought into how the webinar series might have influenced their careers more than female physicians. Although this trend could be multifactorial, it might reflect gender stereotypes among physicians to some extent. Previous studies already discussed this difference, in which they found that female physicians were less likely or slower to take on higher positions in medicine, pursue more advanced qualifications, and work in larger hospitals [23,24]. However, the difference in the rate of career development was more likely caused by female physicians’ tendency to bring up family and switch to part-time work rather than by gender discrimination [23-25]. Parenthood appeared to be negatively associated with career development, particularly for women [23]. This is especially true in Indonesia, where the traditional roles of men as breadwinners and women as family caregivers are still deeply rooted in its culture. Thus, utilizing the online form of CME may be a feasible strategy to give physicians from both genders equal opportunities as it offers a more flexible schedule that can be self-paced accordingly.

There are some limitations to this study. First, there was an unequal distribution of study participants based on specific demographic subgroups, resulting in a less accurate representation. In addition, since a physician could participate in more than one webinar and therefore would complete more than one questionnaire, such possibility of duplicate participation can lead to a less accurate representation. The cross-sectional nature of this study also limited its ability to investigate the long-term impact of the webinar series on physicians’ behavioral changes and quality of patient care. Since this study only evaluated one platform of online CME, which was webinar series, further investigation is needed to evaluate other platforms of online CME and eventually compare their efficacy. More research is also needed to determine the ideal time for conducting a follow-up refresher course after an online CME program to optimize knowledge retention. Refresher courses can be delivered in the form of case-based discussions to be more relevant to the physicians’ daily practice.

In conclusion, the decline of knowledge retention over time remains an important issue to address for future online CME programs. Properly structured refresher courses should be offered to physicians to help them retain the newly acquired knowledge, although the ideal interval from the initial course needs further study. The importance of participating in CME must be reemphasized to physicians of all ages, experiences, positions, and genders because CME is perceived to be beneficial for their clinical knowledge and competence in daily practice as well as career development. Utilizing the online platform of CME may allow for broader participation as it offers flexible learning and requires less time and cost.