Learning on the Go: Assessing Knowledge Gained From Medical Podcasts Created for Vulvovaginal Disease Education.

OBJECTIVE: The aim of the study was to evaluate the effectiveness of "The Vulva Diaries" podcast as a novel learning tool for vulvovaginal disease education.
MATERIALS AND METHODS: Medical students and residents were invited to participate in the study using social media advertisements. Online pretests and posttests, one based on a podcast episode regarding genital herpes and the other on lichen sclerosus, were used to assess changes in knowledge level pre- and post-podcast listening in medical students and residents respectively. A second posttest was sent out 2 weeks after the first to assess knowledge retention. Results were analyzed using paired t tests comparing mean scores before and after podcast.
RESULTS: In medical students, the average test score increased by 20% (n = 56, p < .001). Similarly, in residents the average test score increased by 23.1% (n = 22, p < .001). Medical students and residents rated their average preference for using podcasts as compared with other resources at 3.6 and 3.7/5, respectively. Furthermore, in both groups, there was no significant difference between average scores for posttest 1 versus posttest 2 written 2 weeks later suggested excellent knowledge retention.
CONCLUSIONS: "The Vulva Diaries" podcast increases knowledge on vulvovaginal disease and is an effective learning tool for health care trainees in women's health. This study emphasizes the role of podcasts as a valuable educational resource within gynecology. The success of such initiatives will hopefully bolster the effort to correct the lack of provider knowledge in treating vulvovaginal diseases.

The latest generation of medical learners has witnessed an explosive growth in podcast-based medical education resources.[1] The information age has led to the phasing out of traditional textbooks and the ushering in of #FOAMed (Free Open Access Medical Education as referred to on Twitter). Medical education podcasts are extensively used by residents and previous studies have shown that medical trainees may prefer podcasts over other learning resources.[2-7] This trend has been mirrored within obstetrics and gynecology; however, research in this area is extremely limited and reviews of podcasting in obstetrics and gynecology present a gap in the literature.[8,9] To our knowledge, there are currently no studies that objectively evaluate the educational efficacy of a podcast series in obstetrics and gynecology.

Vulvovaginal disease is extremely common and burdensome, and yet there is a lack of physician education and understanding in this area. National surveys have shown a lack of vulvovaginal education and training in gynecology residency programs in both Canada and the United States. Moreover, all these survey reports recommend evidence-based online resources to improve graduate education.[10-12] This knowledge gap in women’s health care prompted the creation of a medical podcast series for vulvovaginal disease education called “The Vulva Diaries.” As of August 2021, the podcast has more than 35,000 downloads with listeners in 92 countries. This study aims to evaluate the efficacy of “The Vulva Diaries” podcast as a new learning tool in medical trainees.

METHODS

This study involved a brief educational intervention to assess knowledge retention. Institutional research ethics board approval was obtained. “The Vulva Diaries” is a medical podcast series for vulvovaginal disease education created by the authors of this study that is accessible to the public (students, physicians, patients, etc.) through all major podcasting platforms (Apple Podcasts, Spotify, Google Play, etc.). Although the series is publicly accessible, the episodes are largely geared towards medical learners and health care professionals involved in women's health. Thirty-nine episodes were published on a biweekly basis from September 2019 to March 2021 featuring interviews with international experts on the prevention, diagnosis, treatment, and impact of many different lower genital tract human papillomavirus (HPV) and non-HPV–related conditions and dermatologic disorders. Two podcast episodes on common vulvovaginal diseases were selected to be the basis on which the podcast series was evaluated in this study. The study was carried out by using an online pretest and 2 posttests distributed before and after podcast listening, respectively.

Trainees were invited to participate in the study using social media advertisements on 13 medical student and resident Facebook pages, which were selected by connecting with student representatives at various medical schools (7 groups based at the University of Toronto, 4 groups based at other Ontario medical schools, and 2 groups based at US medical schools), and was posted to one of the author's Twitter account. The recruitment period was 6 months in length, from December 5, 2020, to June 5, 2021. The exact number of trainees that saw the advertisements is unknown but is expected to be in the 100 seconds. Advertisements contained a brief description of the study, including that participants were required to provide their e-mail address and how this information would be used, author contact information in the event that participants had any questions, and instructions to click on the below-listed links. The links directed participants to the pretest aimed toward medical students or toward residents. Learners enrolled by clicking the corresponding link and submitting the pretest. Medical students were assigned to listen and be tested on “The Vulva Diaries” episode on genital herpes and residents were assigned the episode on lichen sclerosus. Informed consent was obtained, participation was voluntary, and any medical students and residents in Canada and the United States were eligible to complete the knowledge tests.

The online tests were anonymous and were conducted through SurveyMonkey.com. Questions were designed by mapping out all content covered by the podcast episode, from which major topics were pulled and selected for assessment. Question design was guided by various standards for writing multiple-choice questions, published by US and Canadian university programs, and level of difficulty was catered toward the intended audience, being slightly greater for the questions aimed toward resident learners. All questions were then reviewed by obstetrics and gynecology medical education staff and piloted by a resident and medical student to ensure that the questions were comprehensive and appropriate. The pretest consisted of 20 questions including a consent form, 5 multiple-choice questions assessing demographics, 4 Likert scale questions evaluating self-rated confidence/knowledge and attitudes toward podcast use, and 10 knowledge-based multiple-choice questions. The Likert scale questions asked participants to rate their knowledge level and their confidence level in counseling a patient on the podcast topic on a scale from 1 to 5 (1 = none/very low; 2 = low; 3 = average; 4 = high; 5 = very high). They were then asked to rate their preference for using podcasts as a learning tool as compared with the other educational resources available to them (1 = least preferred; 2 = not preferred; 3 = neutral; 4 = preferred; 5 = most preferred) and to rate the general effectiveness of podcasts as a medical learning tool (1 = very ineffective; 2 = ineffective; 3 = neutral; 4 = effective; 5 = very effective). The knowledge-based questions constituted the participant’s score and covered disease prevalence, causes, diagnosis, common presentations, signs and symptoms, disease course, and treatments.

After submitting the pretest, links to the corresponding podcast episode and posttest were emailed to volunteer participants. The podcast episodes were both 15 minutes in length. The posttests contained the same 10 knowledge-based questions as the pretest in different orders. To minimize for a potential priming effect participants were not given their score upon completion of tests and were not informed of the correct answers. Participants were asked to rate the appropriateness of “The Vulva Diaries” podcast for their audience (1 = very inappropriate; 2 = inappropriate; 3 = neutral; 4 = appropriate; 5 = very appropriate) and to rate the effectiveness of specifically “The Vulva Diaries” podcast as a medical learning tool (1 = very ineffective; 2 = ineffective; 3 = neutral; 4 = effective; 5 = very effective). The first posttest was completed immediately after podcast listening and the second posttest was sent by e-mail 2 weeks after submission of the first posttest to assess knowledge retention. Upon literature review, only 2 articles evaluating podcasts in medical education assessed learning after 1 week, and none included a second posttest to assess knowledge retention.[13,14] To reduce the risk of small sample size and low study completion rate, which is a known limitation of voluntary questionnaire-based studies, 2 weeks was chosen as the metric for knowledge retention.[11,12,14] Furthermore, 2 weeks has been reported as an acceptable and commonly used interval for test-retest reliability, with longer intervals not showing significant differences in scores when compared with each other.[15,16] Pretest and posttest responses were linked using e-mail addresses provided by participants.

Knowledge scores were analyzed using paired t tests comparing an individual's prepodcast to their postpodcast scores. Self-rated knowledge level and confidence level in patient counseling were also compared before and after posting listening using paired t tests. Preference for using podcasts as compared with other educational resources and general effectiveness of podcasts as learning tools are reported as averages before and after podcast listening. Data from questions specifically assessing “The Vulva Diaries” podcast series (appropriateness for a medical student/resident audience and effectiveness as a medical learning tool) and the number of downloads are presented descriptively. Microsoft Excel was used to analyze data.

The sample size calculation was based on an effect size of 0.7, which is considered to be a medium anticipation of effect and appropriate for educational intervention. Using a p value of 0.05 or less to claim statistical significance and a power of 80%, and assuming an effect size of 0.7, a sample size of 19 learners per group was calculated.[17]

RESULTS

From December 5, 2020, to June 5, 2021, a total of 101 medical students and 46 residents opened the pretest. Eighty-three medical students and 37 residents went on to complete the pretest, with 27 medical students and 15 residents dropping out of the study after only completing the pretest. Therefore, a total of 56 medical students and 22 residents completed the pretest and posttest 1 (see Table 1). Forty of these 56 medical students (71%) completed posttest 2 two weeks later, assessing knowledge retention, and all 22 residents (100%) completed posttest 2. Most participants belonged to younger age groups, 57% of medical students selecting 18–24 years old as their age range and 63% of residents selecting 25–29 years old. Within the resident cohort, 68% were obstetrics and gynecology residents.

TABLE 1

Demographic Information of Participants

Demographic		Medical students (n = 56)	Residents (n = 22)
Year of study	1st year	10 (18%)	9 (41%)
	2nd year	16 (29%)	4 (18%)
	3rd year	27 (48%)	4 (18%)
	4th year	3 (5%)	4 (18%)
	5th year	0	1 (5%)
Age	18–24	32 (57%)	0
	25–29	22 (39%)	14 (63%)
	30–34	2 (4%)	5 (23%)
	35–39	0	3 (14%)
Medical specialty	Medical student/unspecialized	55 (98%)	1 (5%)
	Gynecology	0	15 (68%)
	Family medicine	1 (2%)	4 (18%)
	Neurology	0	2 (9%)
Province/state	Ontario	54 (96%)	14 (63%)
	Alberta	0	6 (27%)
	Manitoba	0	1 (5%)
	Quebec	0	1 (5%)
	Connecticut	2 (4%)	0

Data are presented as n (%).

Since its inception in September 2019, “The Vulva Diaries” has accrued 35,620 total downloads with listeners in 92 countries as of August 22, 2021. The total number of US downloads is 11,510 and “The Vulva Diaries” has a 5.0/5.0 rating on Apple Podcasts. Among medical students, the average pretest score before podcast listening was 69.6% and the average posttest score after podcast listening was significantly improved at 89.6% (n = 56, p < .001), which is 20% increase (see Table 2). The average pretest and posttest scores for residents before and after podcast listening, respectively, also significantly improved from 65.5% to 88.6% (n = 22, p < .001), which is a 23.1% increase. Self-rated knowledge level on the episode's topic and confidence level in counseling a patient about the episode's topic significantly improved after podcast listening in both medical students and residents (p < .001). Average preference for using podcasts as a learning tool as compared with other educational resources did not change before and after podcast listening in residents and slightly improved in medical students from 3.11 to 3.57/5. The average rating for general effectiveness of podcasts as medical learning tools increased after podcast listening in both medical students and residents. On average, both medical students and residents rated “The Vulva Diaries” as an appropriate resource for their audience and as an effective medical learning tool.

TABLE 2

Average Pretest Versus Posttest 1 Results Before and After Podcast Listening

	Medical students (n = 56)			Residents (n = 22)
Measure	Pretest	Posttest 1	Change in score, %	Pretest	Posttest 1	Change in score, %
Average score	69.6%	89.6%	+20a	65.5%	88.6%	+23.1a
Average self-rated knowledge level on podcast topic	2.66/5	3.45/5	+15.8a	2.59/5	3.05/5	+9.2a
Average confidence level in counseling a patient on podcast topic	2.09/5	3.43/5	+36.8a	2.32/5	3.32/5	+20a
Average preference for podcast use as compared with other educational resources	3.11/5	3.57/5		3.68/5	3.68/5
Average rating for general effectiveness of podcasts as medical learning tools	3.57/5	3.93/5		3.86/5	4.09/5
Average effectiveness of “The Vulva Diaries” podcast as a medical learning tool	N/A	4.32/5		N/A	4.23/5
Average appropriateness of “The Vulva Diaries” podcast for its audience	N/A	4.54		N/A	4.55

Data are outcome p value if statistically analyzed.

< .001.

N/A indicates not applicable.

In both medical students and residents, there was no significant difference observed between average scores for posttest 1 and posttest 2 written 2 weeks later (see Table 3). Similarly, there was no significant decrease in average self-rated knowledge level on the episode's topic 2 weeks later for residents and medical students. The average confidence level in counseling a patient about the episode’s topic did significantly decrease from 3.45 to 3.03/5 in medical students (n = 40, p < .001) and from 3.32 to 3.00/5 in residents (n = 22, p = .016).

TABLE 3

Average Posttest 1 Versus Posttest 2 Results Immediately and 2 Weeks After Podcast Listening

	Medical students (n = 40)		Residents (n = 22)
Measure	Posttest 1	Posttest 2	Posttest 1	Posttest 2
Average score	89.8%	88.5% (NS)	88.6%	85.9% (NS)
Average self-rated knowledge level on podcast topic	3.43/5	3.28/5 (NS)	3.05/5	3.00/5 (NS)
Average confidence level in counseling a patient on podcast topic	3.45/5	3.03/5a	3.32/5	3.00/5b

Data are outcome p value if statistically analyzed.

< .001.

< .05.

NS indicates not significant.

DISCUSSION

This study shows that “The Vulva Diaries” podcast is an effective learning tool for vulvovaginal disease education. Previous reviews on podcasting in medical education have cautioned against limiting evaluation to rater satisfaction and call for more rigorous examination of efficacy.[13,18,19] To our knowledge, this is the first report that has used objective measurements to evaluate an educational podcast series in obstetrics and gynecology. A study published in 2020 assessed the feasibility of an obstetrics and gynecology educational podcast that the group developed by measuring costs, downloads, and number of reviews and followers. Although they did not test pre– and post–podcast knowledge scores, their podcast also accrued a large number of downloads, highlighting that there is indeed a demand for such resources.[20]

Podcast-based learning has exhibited particular uptake in emergency medicine, with 89% and 91% of surveyed US and Canadian EM residents, respectively, reporting the use of educational podcasts.[3,17] Similar to the increased confidence level in patient counseling reported by residents and medical students after listening to “The Vulva Diaries,” US emergency medicine residents reported that podcasts changed their clinical practice.[2] A literature review on podcasting in medical education, spanning studies done in medical students and residents from several disciplines, summarizes that almost all articles reporting knowledge outcomes showed an improvement in post–podcast test scores. Of note, none of these studies assessed knowledge retention, whereas this study demonstrated the maintenance of increased knowledge scores 2 weeks later. Despite knowledge retention, participants' confidence in counseling patients about the episode’s topic 2 weeks later significantly decreased in residents and especially so in medical students, stressing the importance of integrating knowledge into real-world clinical counseling to maintain confidence. Apart from this, the previously referenced review concluded that most learners reported podcasts were useful for learning.[13] Usefulness and relevance for the intended audience have been recommended by international medical education experts as one of the top podcast quality indicators, and correspondingly, both residents and medical students rated “The Vulva Diaries” as an appropriate resource for their audience and as an effective learning tool.[19]

After listening to the podcast, medical students and residents rated their average preference for using podcasts as compared with other resources at 3.6 and 3.7/5, respectively, and this preference has been demonstrated by previous literature as well. Based on past studies, residents and medical students may prefer podcasts most over textbooks and other resources.[5,6] Some studies have even shown superior knowledge retention and learning satisfaction from podcasts as compared with textbooks.[7] With the current generation of medical learners becoming more and more technologically literate, it is important that medical education resources follow suit—which is supported by the fact that 87% of this study’s participants were below the age of 30 years. Podcasts allow for asynchronous on-demand learning, are convenient and time-efficient, cater to learners with different studying habits and learning styles, and improve engagement.[6,7,20] In addition, expanding podcast use for continuing medical education has been touted as a potential solution for providing continuing medical education in a safe and accessible manner during the COVID-19 global pandemic.[21]

Study limitations include a small sample size and a limited number of responses from residents. Interestingly, 100% of residents completed the follow-up knowledge retention test 2 weeks later; however, only 73% of medical students completed this test. A study evaluating podcast use in medical students reported much greater incompletion rates and listed lack of time and podcast length as the most cited reasons for incompletion of follow-up tests.[14] We attempted to optimize for this by selecting a podcast episode only 15 minutes in length and by measuring knowledge retention at 2 weeks. We suspect lack of time and residents’ busy schedules also account for the low number of resident participants overall.

The pretests and posttests were designed by the authors of “The Vulva Diaries” podcasts, and as such, another limitation is that the tests have not been previously validated. To address this weakness, questions were designed based on a thorough literature review of other studies assessing knowledge gain from podcasts. Level of confidence in counseling patients was included as a question based on a study that asked students to assess their comfort level in approaching patients before and after resource exposure.[6] This hinted toward a behavioral outcome, which unfortunately was not formally measured in this study. Any bias secondary to the authors of this article being the creators of “The Vulva Diaries” has been addressed by using objective measurements of knowledge gain. In addition, there is a possibility of priming effect, as the pretest and posttest contained the same questions in different orders. Therefore, participants could have used audio cues rather than improved knowledge to increase their scores. This was controlled for by not informing participants of their scores or of the correct answers so that they did not simply learn the right answers to the questions. That being said, this could have prompted participants to engage in other learning behaviors to confirm their answers, which is a potential confounder as learners were not asked if they engaged in any learning behaviors aside from podcast listening during the study period. Last, the sample pool may have been affected by voluntary response bias, in that people who are interested and learn best from podcasts may have volunteered. This has been listed as a limitation by most aforementioned studies evaluating podcasts in medical education[4,6,14,20] and may be largely unavoidable; however, studies that enrolled a general group of participants (e.g., emergency medicine residents at all levels of training across the United States), rather than those who volunteered to listen to podcasts, have also shown that podcasts are extensively used and preferred.[2,3,5] Randomized control studies have demonstrated that podcasts and live lectures provide equivalent knowledge retention and another study that randomly assigned students to learn from a book chapter or a podcast also found that podcast users scored significantly better on posttest s and showed significantly higher approval ratings.[7,22,23] This points toward the fact that the significance of this limitation may be minimal.

CONCLUSIONS

As new technologies are incorporated into medical education, their effectiveness needs to be evaluated. “The Vulva Diaries” has been shown to be an effective educational resource for vulvovaginal disease and can be used by any health care providers involved in women's health. This project emphasizes the role of podcasts as a valuable modern learning tool in the field of gynecology and in medicine in general. This will inform future podcast development, providing a new avenue for interprofessional learning and helping optimize medical education. With the success of “The Vulva Diaries” as a novel learning tool, we hope to see an increase in knowledge on managing vulvovaginal disease to ultimately improve patient outcomes and quality of life, correcting this large gap in women's health care.