Web-based teaching in point-of-care ultrasound: an alternative to the classroom?

OBJECTIVES: To evaluate two educational methods for point-of-care ultrasound (POC US) in order to: 1) determine participant test performance and attitudes in using POC US and 2) compare cost and preparation time to run the courses.
METHODS: This was a pilot study conducted at a county teaching hospital. Subjects were assigned to participate in either a large group course with live classroom lectures (Group A) or a group asked to watch 4.5 hours of online prerecorded lectures (Group B). Both groups participated in small-group hands-on training after watching the lectures. Both groups took a pre- and post-course exam, and completed course surveys. Cost and time spent running the courses were also compared.
RESULTS: Forty-seven physicians participated in the study. The pre-test and post-test scores between the two groups did not differ significantly. Of those with prior ultrasound experience, the majority of both groups preferred to continue classroom-based teaching for future courses. Interestingly, in the groups who had no ultrasound experience prior to their course participation, there was a higher percentage who preferred web-based teaching. Lastly, Group B was shown to have the potential to take less preparatory time when compared to Group A.
CONCLUSION: A web-based curriculum in POC US appears to be a promising and potentially time saving alternative to live classroom lectures and seems to offer similar educational benefits for the postgraduate learner.

Introduction

Point-of-care ultrasound (POC US) is a widely used clinical modality in the emergency department (ED), with over 90% of emergency medicine (EM) residency programs having incorporated it into their curriculum to bolster resident ultrasound (US) training.1–3 Due to the increasing demand for this critical skill, US training requirements for EM residents have become more well defined. However, there is less supportive evidence to guide POC US educators on how best to train physicians who are already in practice. Unlike residents, attendings are more likely to have established practice patterns that make POC US implementation more difficult, they may have less time to devote to voluntary didactic learning, and their role during clinical shifts differs such that learning a new clinical skill like POC US during a shift may be more challenging.4

The more recent introduction of technologically based educational tools in POC US education has mitigated these learning obstacles somewhat by allowing a migratory learning environment to exist that does not require the learner to actually be in a classroom. Integrating web-based material into a POC US curriculum with web-casting and web-streaming has garnered particular interest, as it may be more effective and efficient than the traditional classroom model.5–7 Thus, educational models designed to disseminate information using technology in the form of online lectures, instructional videos, blogs, or live feeds have grown exponentially.8,9

The purpose of this study was to evaluate two educational methods of teaching POC US to attendings: 1) a traditional method of teaching POC US with classroom instruction and small-group hands-on training with hired models and 2) pre-recorded web-based lectures followed by small-group hands-on training in the ED. We aimed to determine whether trainees who took the online lecture course with small-group hands-on training in the ED maintained similar learning outcomes in test performance and self-perceived attitudes regarding the course, at a similar or lower cost.

Materials and methods

This was a pilot study that was approved by our hospital institutional review board. This study took place over 6 weeks, from December to January 2014, with fellows and attendings from the Departments of Medicine and EM. All fellows and attendings were employees of an academic Level I trauma center with an annual ED census of 185,000. Inclusion criteria for this study were any participant who may be eligible to take a 1-day POC US course hosted by the Emergency Ultrasound Division (EUD) in the Department of EM or watch the web-based POC US instructional lectures and participate in small-group hands-on training. Participants were excluded from the study if they had completed a residency or practice-based pathway of emergency ultrasound (EUS) training.

The impetus behind the course was to provide those attendings and postgraduate learners with no or limited training in EUS with a course designed specifically for them, separate from the didactic training given to the residents throughout the year. None of the participants in this study had required or protected didactic time incorporated into their schedules for this particular course, so allocation of participants into groups was based on clinical schedules.

Twenty-five fellows and attendings were assigned to Group A. Members of Group A attended a 1-day, 10-hour POC US course, with classroom-based lectures and small-group hands-on training with hired models. All eight lectures, which covered basic and advanced applications of POC US, were given by instructors from the EUD. For the Group A course, ultrasound machines, meals, and models were provided. Audio-visual (AV) recording equipment was set up prior to the course, and all lectures were recorded and then uploaded by a department AV specialist. Group A had 4.5 hours of live classroom-style lectures given by members of the EUD and 4.5 hours of small-group hands-on training on models. Group A scanned in small groups, rotating every 45 minutes at five different stations. Small-group scanning occurred twice, in the morning and in the afternoon, between lectures. The instructor-to-student ratio was not more than 1:4.

Twenty-two fellows and attendings were assigned to Group B. Members of Group B were required to watch all the lectures that were recorded from Group A’s course online prior to meeting with members of the EUD for 4.5 hours to receive hands-on small-group instruction on patients in the ED. All videos could be viewed online on a secure server that was already being used for EM residency lectures. Group B members met and scanned in small groups in the ED, with each learner being given the opportunity to perform educational scans on live patients in the ED. Instructors were told to allow time for each learner to practice each application up to a maximum time of 4.5 hours in small groups. The instructor-to-student ratio was not more than 1:4.

Both groups completed a web-based examination before and after the course. Learners took exams (45 questions each for pre- and post-exam), which were created from a random pool of 150 questions (approximately five to seven questions were created in each pool for each POC US application). Exams were created on an online exam-building site that randomly created pre- and post-exams from pooled questions. Exam questions were written by the EUD, and the test material was based closely on the required core content recommended by the American College of Emergency Physicians’ “Emergency Ultrasound Guidelines”.2 Two voluntary post-course surveys were distributed; one in paper form that was handed out immediately after the conclusion of the course and a web-based survey 2 weeks later. Each survey contained different questions regarding the participants’ comfort using US and their thoughts about the method of teaching in each group. Costs for food, models, exam programs, instructors, and the time taken to prepare for both courses were recorded for each group.

Test scores were tallied before and after the courses using a data-collection tool within the web-based exam program ExamBuilder™. To determine the differences between Group A and B with regard to pre-test, post-test, and percentage improvement between these two tests, the Mann–Whitney U test was used for independent measures. For all statistical analyses, P-values <0.05 (two-tailed) were considered to be significant.

Results

Six participants from Group A and two participants from Group B were ultimately excluded from the statistical analysis because they either did not take the pre-exam prior to the course, the post-exam immediately after completing the course, or watch the web-based videos prior to the hands-on training. Over the course of the study, a total of 47 participants enrolled in the POC US course and participated in the study – 25 participants were assigned to Group A and 22 participants were assigned to Group B. Participants were fellows and attendings only. Both groups were comprised of a mixture of EM and subspecialty medicine physicians (Table 1). Group A and B mean pre-test scores were 67.5% (95% confidence interval [CI] =0.64–0.71) and 68.5% (95% CI =0.64–0.73), respectively (P=0.645). Group A and B mean post-test scores were 82.2% (95% CI =0.79–0.84) and 79.1% (95% CI =0.75–0.82), respectively (P=0.142). The mean percentage difference between Group A’s and Group B’s pre-test and post-test scores were 14.7% and 11.4%, respectively (P=0.254) (Table 2).

Table 1

Attending characteristics of group participants, n (%)

Years in specialty	Group A (N=25)			Group B (N=22)
	EM	IM	Pulm CC	EM	IM subspecialty	Pulm CC
0–5	0 (0.0)	0 (0.0)	11 (44.0)	3 (13.5)	4 (18.0)	0 (0.0)
6–10	3 (12.0)	0 (0.0)	0 (0.0)	1 (4.5)	5 (23.0)	0 (0.0)
11–15	5 (20.0)	0 (0.0)	1 (4.0)	5 (23.0)	0 (0.0)	0 (0.0)
16+	4 (16.0)	1 (4.0)	0 (0.0)	2 (9.0)	1 (4.5)	1 (4.5)

Note: IM includes the specific subspecialties it encompasses.

Abbreviations: EM, emergency medicine; IM, internal medicine; Pulm CC, pulmonary critical care.

Table 2

Participant pre- and post-course exam scores

Test score	Group A	Group B	P-value
Pre-test, mean	67.5%(95% CI =0.64–0.071)	68.5%(95% CI =0.79–0.84)	0.645
Post-test, mean	82.2%(95% CI =0.79–0.84)	79.9%(95% CI =0.75–0.82)	0.142
Mean difference in pre- and post-test scores in each group	14.7%	11.4%	0.254

Abbreviation: CI, confidence interval.

Twenty of the 47 participants who answered the paper-based survey had POC US training prior to this course (Table 3). Of those with prior US experience, 80% in Group A and 55% in Group B preferred to continue the classroom-based method of teaching for future courses, whereas 20% in Group A were interested in, and 45% in Group B preferred, web-based training in the future. Of the participants who answered the online survey who had no prior US experience, 67% in Group A and 30% in Group B preferred the classroom-based method of teaching for future courses, whereas 33% in Group A and 70% in Group B preferred web-based training in the future (Table 3). The Group A course cost $1400.00, and a total of 91 hours were spent organizing the course. The Group B course had no additional costs, and a total of 31 hours were spent organizing the course.

Table 3

Participant survey results

Survey question	Group A (N=25)		Group B (N=22)
EM/Other specialty, %	55/45		50/50
Had prior training in POC US before course*	Yes	No	Yes	No
	14/25 (56%)	11/25 (44%)	6/17 (35%)	11/17 (65%)
Prefer classroom-based teaching method for future courses**	Prior teaching in POC US	No prior teaching in POC US	Prior teaching in POC US	No prior teaching in POC US
	4/5 (80%)	6/9 (67%)	5/9 (55%)	3/10 (30%)
Prefer web-based teaching method for future courses***	Prior teaching in POC US	No prior teaching in POC US	Prior teaching in POC US	No prior teaching in POC US
	1/5 (20%)	3/9 (33%)	4/9 (45%)	7/10 (70%)

Notes:

From paper survey – n=25 for Group A; n=17 for Group B;

from online survey – n=14 for Group A and n=19 for Group B;

from online survey – n=14 for Group A and n=19 for Group B.

Abbreviations: EM, emergency medicine; POC US, point-of-care ultrasound.

Discussion

Our study has demonstrated that there were no significant differences between pre- and post-test scores between Group A and Group B. Additionally, there was no significant difference in the percentage improvement between the pre- and post-test exams between each group, suggesting that prerecorded video may impart rote knowledge as effectively as lectures that are given in a live setting.

The majority of both groups’ participants felt classroom lectures were still the superior method of teaching POC US. Interestingly, there was a higher percentage of those who preferred web-based teaching in the groups who had no experience with US prior to their course participation, which suggests a mere-exposure effect among those with prior more traditionally taught US experiences, and that with no prior training, novice learners may be more open to nontraditional ways of being taught a new skill.10

The concept of a “flipped classroom” model, where learners watch online modules and lectures outside and before class is transforming medical education. In this model, students are responsible for learning the educational content prior to class so that class time can be spent on interactive activities and discussion.11–13 Several studies have reported that students from undergraduate to postgraduate levels in multiple disciplines not only achieved examination scores similar to those obtained through the traditional classroom teaching model but also had equal or better self-perceived knowledge and general educational experiences.12,14,15 Issues related to cost and the difficulty of balancing clinical work with educational didactics due to duty hours and schedules are potentially mitigated by this model.16,17 From an instructor’s perspective, this model allows the face-to-face interaction to be dedicated to more in-depth and advanced discussion based on the basic concepts taught in the video lecture.17 From a learner’s perspective, the participant can control the pace at which information is presented. Instructors need only give the lecture once, but the information can be disseminated to a larger audience, repeatedly.

Limitations

The generalizability of these findings may be limited due to the fact this study was performed at a single center with a robust EUS division. As a result, the US instructors may be better equipped to offer high-fidelity modules and use innovative methods for learning, and have access to educational space and protected time to create a blended curriculum in POC US, than smaller programs or community sites. As a result, the cost of AV equipment, AV personnel, and servers to place online lectures in for this project were not included, and thus may not represent the true cost of running a Group B course outside of this institution. Based on the estimations of one local EM education group, the cost for cameras, laptops, mixing boards, microphones, and software can vary from a few hundred dollars upward to US$10,000. Additionally, only those not affiliated with the hospital and residency who taught in Group A were paid an honorarium. All other instructors were part of the EUD, where teaching is blended into an established salary, and no additional honoraria were given for participation in and preparation of the course. It should be noted that the financial benefit from running a Group B-type course would only materialize if the course were repeated, as no additional cost would ensue from the initial cost of setting up the course.

The fact that this was a pilot study made it more difficult to ascertain what the generalizability and magnitude of our findings were outside this institution. There were a set number of participants in the study based on the number of attendings and fellows who opted to take the course, so our conclusions are not powered to determine if the difference between the post-test scores were significant. Also, a multiple choice test had not been validated as an accurate assessment of competency prior to this course. For future endeavors, the study design will require a power calculation such that differences in outcomes of performance can be definitively determined.

Second, assignment to groups was not randomized but was based on the schedules of the participants, which presented some potential sources of bias. We attempted to mitigate this bias by choosing course dates where participants in both groups were not on the clinical schedule, and assigned participants to groups on those dates. Difficulties still arose due to other administrative conflicts that prevented them from being able to attend their initially assigned course dates, resulting in a small number of participants having to reschedule to a non-randomly assigned date. Additionally, the obvious differences in prior exposure to EUS, the role of US in the physician’s practice (EM versus pulmonary critical care, for instance), and the physician’s comfort with the concept of internet-based video learning may bias their preconceived beliefs about learning, none of which was explored in this study.

Third, the environment and conditions of small-group hands-on scanning were not the same for Group A and B. Though both groups had similar numbers of participants (no more than four to a group), Group A scanned healthy, young, hired models, while Group B scanned in the ED on real patients with potential pathologic findings. This was a potential source of bias, even though the premise of Group B was to decrease the overhead expenses incurred and time taken to obtain models for teaching. Participants had varying levels of US experience, which could pose as a potential inter-operator bias when calculating exam performance and making subsequent comparisons to the method of teaching.

In addition, there was no clinical component to the post-exam, which is arguably the most important aspect of learning a bedside skill such as POC US. To counter this, our next phase will include a 6-month follow-up with a proctored clinical examination in addition to an online exam to evaluate performance.

Conclusion

Learners who participated in a blended curriculum that incorporated web-based lectures followed by dynamic scanning in small groups had post-course exam scores that did not differ significantly from those achieved by their traditional classroom-learning colleagues. Additionally, a web-based teaching model in US may be less expensive as a teaching method when compared to traditional classroom teaching model if subsequent courses are taught from web-based lectures. More study is warranted to determine if this method results in equal or improved accuracy and quality of US scanning, as well as if material retention is maintained after the conclusion of the course.