Randomized study of effectiveness of computerized ultrasound simulators for an introductory course for residents in Brazil.

PURPOSE: This study aimed to assess the impact of ultrasound simulation (SonoSim) on educational outcomes of an introductory point-of-care ultrasound course compared to hands-on training with live models alone.
METHODS: Fifty-three internal medicine residents without ultrasound experience were randomly assigned to control or experimental groups. They participated in an introductory point-of-care ultrasound course covering eight topics in eight sessions from June 23, 2014 until July 18, 2014. Both participated in lecture and hands-on training, but experimental group received an hour of computerized simulator training instead of a second hour of hands-on training. We assessed clinical knowledge and image acquisition with written multiple-choice and practical exams, respectively. Of the 53 enrolled, 40 participants (75.5%) completed the course and all testing.
RESULTS: For the 30-item written exam, mean score of the experimental group was 23.1±3.4 (n=21) vs. 21.8±4.8 (n=19), (P>0 .05). For the practical exam, mean score for both groups was 8.7 out of 16 (P>0 .05).
CONCLUSION: The substitution of eight hours of ultrasound simulation training for live model scanning in a 24 hour training course did not enhance performance on written and image acquisition tests in an introductory ultrasound course for residents. This result suggests that ultrasound simulation technology used as a substitute for live model training on an hour-for-hour basis, did not improve learning outcomes. Further investigation into simulation as a total replacement for live model training will provide a clearer picture of the efficacy of ultrasound simulators in medical education.

RCT Entities:   Population Interventions Outcomes

Introduction

The benefits of point of care (POC) ultrasound are well-known; it is inexpensive, noninvasive, portable, and increasingly present in rural and less developed areas [1]. Early exposure to ultrasound in medical education increases users’ aptitude in diagnostic and procedural sonography [2,3]. The use of simulated clinical experiences (e.g., standardized patients and high-tech mannequins) to augment conventional teaching methods is well-established in medical schools, but the utility of simulation in ultrasound education has yet to be fully investigated [4]. This article discusses the impact of SonoSim ver. 2.8.1. (Ultrasound Training Solution, Santa Monica, CA, USA) ultrasound simulators on learning outcomes for medical residents taking an introductory ultrasound course in a hospital, Brazil.

Methods

Fifty-three internal medicine residents of the Universidade Federal de Ciências da Saûde de Porto Alegre at Santa Casa de Misericordia Hospital in Porto Alegre, Rio Grande do Sul, Brazil participated in an introductory POC ultrasound course of eight three-hour sessions over four weeks. Residents had no prior experience with ultrasound. Each session covered one topic: knobology, focused assessment with sonography for trauma (FAST) protocol, head and neck, abdominal, cardiac, pulmonary, musculoskeletal, and procedures. We provided an hour-long lecture and two hours of hands-on training for each session. For the hands-on training portion, we randomly assigned subjects to control ‘live model only’ (LM) group (two hours scanning live models) or the ‘live model plus simulator’ (LM+S) experimental group (one hour with live models and one hour of self-instruction using the SonoSim training modules).

Three well-trained ‘blinded for peer review’ medical students gave the lectures and hands-on instruction using one another as live models. The student-instructor ratio was less than or equal to 6 to 1. Four ultrasound devices (SonoSite S Series, Bothell, WA, USA) and a procedure training block (CAE Blue Phantom Regional Anesthesia Ultrasound Training Block, Sarasota, FL, USA) were used for training. The LM group was allowed two hours of practice with a live model, while the LM+S group spent one hour with live models and one hour with the simulator. During each live model session, subjects were divided into small groups in which they practiced insonating key structures highlighted in the prior lecture while learning the basic function of the ultrasound devices.

The simulator used in this study was a SonoSim. It provides integrated hands-on ultrasound training, didactic instruction, and assessment using real patient ultrasound cases. Through its replication of the tactile experience of probe manipulation, the simulator is designed to mimic the experience of scanning an actual patient. Users are allowed to practice image acquisition and interpretation in a risk-free environment and test their knowledge and skills in clinical scenarios and module knowledge assessment questions [5]. During the hour with the simulator, the LM group used the software training modules pertaining to the focus of that session. Their learning was largely independent, relying on the instruction and feedback provided by the software.

Fifty-three participants were enrolled and randomized before instruction began. Of the 53 participants enrolled, 40 (75.5%) met the necessary criteria (including attendance of all eight sessions and completion of both the written and practical exams), leaving the groups with n=21 in LM+S and n=19 in LM. This sample size was shown to be sufficient to reach a power, 0.8 with an expected standard deviation of written exam test scores, 5 (α=0.05). The outcome measures were written exam and practical exam scores. An unpaired t-test was performed on the written and practical exam scores of the LM and LM+S groups. Statistical significance was set at P<0.05. Data was analyzed with STATA ver. 9.0 (Stata Co., College Station, TX, USA).

Written exam

The written exam had 30 image or video-based multiple choice questions (MCQ) assessing the subjects’ clinical knowledge of ultrasound functionality, diagnostic methods, and image interpretation (Supplementary file 1). We did not administer a pre-test because enrollment requirements stipulated that participants have no prior ultrasound experience.

Practical exam

The 16-point practical exam was designed to assess the subjects’ ability to operate the ultrasound device, properly insonate key POC ultrasound structures, and utilize advanced imaging functions (Appendix 1). Subjects were allowed six minutes to scan a live model and save five images (Fig. 1): (1) right upper quadrant organs and landmarks; (2) parasternal long axis view of the heart; (3) parasternal short axis view of the heart; (4) visceral-parietal pleural interface (VPPI) on M-mode (for detection of pneumothorax); and (5) right carotid artery/internal jugular vein of the neck with color Doppler.

Fig. 1.

Ultrasound images used as an answer key for the practical exam. (A) Right upper quadrant view. (B) Parasternal long axis view. (C) Parasternal short axis view. (D) Visceral-parietal pleural interface with M-mode. (E) Carotid/internal jugular vein with color Doppler.

For full credit, each window needed to include landmarks or structures specified by the exam instructions, e.g., the left ventricle, left atrium, interventricular septum, mitral valve and aortic outflow tract for the parasternal long axis view. Some images required the use of special device functions, including M-mode for the insonation of the VPPI and color Doppler for the neck vasculature.

The subjects had linear, phased-array, and curvilinear probes during the exam. Proctors, present for timing purposes only, paused the time when the subjects verbally expressed a desire to change transducers. Subjects were not provided feedback about the quality of their images or any instruction for improvement. When they were satisfied with their images, the subjects saved them with a coded label to be used for identification during the grading process.

Images were graded back in the United States by faculty at the ‘blinded for peer review’ department of ultrasound education with student identity and timing of assessment blinded. Scores were calculated as the sum of points earned from each image (16 points total) through successful acquisition of items specified by the grading rubric. Points were granted to images with potential diagnostic utility in a clinical setting. The study was approved by the institutional review board (IRB) as exempt under IRB number HS# 2014-1170.

Results

Of the 53 students who participated in the course, 40 (75.5%) met the attendance requirements for all eight sessions and completed the 30-item MCQ written exam. The mean score of the LM+S group was 23.1±3.4 (n=21) vs. 21.8±4.8 (n=19) in the LM group (Fig. 2). There was no significant difference between the written exam scores of the two groups (P>0.05). Of 53 students who participated in the course, 41 (77.4%) met the attendance requirements for all eight sessions and completed the 16-item practical exam. The mean score for the LM+ S group was 8.7±4.8 (n=22) while the mean score for the LM group was also 8.7±3.9 (n=19) (P>0.05) (Fig. 2). Raw data of the results were available from Supplementary file 2.

Fig. 2.

Comparison of mean scores of the live model (LM) and live model plus simulator (LM+S) groups for the (A) written and (B) practical examinations. Vertical bars represent standard deviation.

Discussion

Technology is a vital component of effective simulation-based education in ultrasound [6]. Simulators such as the SonoSim Ultrasound Training Solution, Phantom Task Trainers, and SonoTrainer have already been adopted into medical education programs throughout the United States [7]. These technologies have demonstrated promise to teach specific clinical applications of ultrasound, e.g., the FAST protocol, the detection of fetal anomalies, and advanced procedures [8,9]. Yet the efficacy of ultrasound simulation, its educational outcomes, and effects on clinical competence remain unclear. A 2011 systematic review of ultrasound procedural simulation concluded that there was little evidence to suggest that the extensive use of simulation-based ultrasound education would yield improved competence [10].

This study investigated the effectiveness of ultrasound simulator technology in conjunction with traditional teaching methods, with the hope that the simulators could overcome the logistical barriers to providing ultrasound training to students. A similar educational study showed that an ultrasound curriculum based upon Podcast lectures, peer instruction, and a standardized formative evaluation for each learning session led to improved educational outcomes compared to a traditional lecture format with dedicated practice sessions [11]. The study in Brazil aimed to build on that success and investigate simulation as a possible alternative to live model training. By combining these two modalities, it could be possible to provide ultrasound education in institutions lacking local expertise. Podcast lectures are a solution to lack of ultrasound faculty, and simulator training could reduce the cost of ultrasound machines and live models.

The use of ultrasound simulators as a partial replacement for live model training did not significantly affect learning outcomes in this study. This result can be interpreted in two ways: the simulators were an unnecessary adjunct to traditional learning methods, or conversely, the simulators served as an adequate replacement for live model training when used for half of the dedicated learning time. Either hypothesis is consistent with showing no difference in post-simulation scores.

Proficiency of the English language was a requirement to participate in this study, but inevitably, students had variable fluency, which we did not assess. Individuals aware of their limited grasp of English signed up for the course because they did not want to miss the opportunity to learn ultrasound. This could have led to a discrepancy between an individual’s knowledge and their written exam score. Furthermore language barrier may have impaired learning itself. Our students had imperfect attendance; we excluded subjects who did not sign into every session but it is possible that some who did may have left early to handle medical emergencies. Nearly all of the internal medicine residents that were accessible for the study participated, but residents from other specialties that could have satisfied the requirements for participation were not available. The addition of a third group that received only simulation training would have provided a clearer picture of the effectiveness of the simulators.

In conclusion, the substitution of eight hours of ultrasound simulation training for live model scanning in a 24 hour training course did not enhance novice student performance on written and image acquisition tests in an international introductory ultrasound course for residents. This result suggests that ultrasound simulation technology, as used here to substitute for live model training on an hour-for-hour basis, did not improve learning outcomes. Further investigation into simulation as a total replacement for live model training will provide a clearer picture of the efficacy of ultrasound simulators in medical education.