The effect of dyad versus individual simulation-based ultrasound training on skills transfer.

CONTEXT: Dyad practice may be as effective as individual practice during clinical skills training, improve students' confidence, and reduce costs of training. However, there is little evidence that dyad training is non-inferior to single-student practice in terms of skills transfer.
OBJECTIVES: This study was conducted to compare the effectiveness of simulation-based ultrasound training in pairs (dyad practice) with that of training alone (single-student practice) on skills transfer.
METHODS: In a non-inferiority trial, 30 ultrasound novices were randomised to dyad (n = 16) or single-student (n = 14) practice. All participants completed a 2-hour training programme on a transvaginal ultrasound simulator. Participants in the dyad group practised together and took turns as the active practitioner, whereas participants in the single group practised alone. Performance improvements were evaluated through pre-, post- and transfer tests. The transfer test involved the assessment of a transvaginal ultrasound scan by one of two clinicians using the Objective Structured Assessment of Ultrasound Skills (OSAUS).
RESULTS: Thirty participants completed the simulation-based training and 24 of these completed the transfer test. Dyad training was found to be non-inferior to single-student training: transfer test OSAUS scores were significantly higher than the pre-specified non-inferiority margin (delta score 7.8%, 95% confidence interval -3.8-19.6%; p = 0.04). More dyad (71.4%) than single (30.0%) trainees achieved OSAUS scores above a pre-established pass/fail level in the transfer test (p = 0.05). There were significant differences in performance scores before and after training in both groups (pre- versus post-test, p < 0.01) with large effect sizes (Cohen's d = 3.85) and no significant interactions between training type and performance (p = 0.59). The dyad group demonstrated higher training efficiency in terms of simulator score per number of attempts compared with the single-student group (p = 0.03).
CONCLUSION: Dyad practice improves the efficiency of simulation-based training and is non-inferior to individual practice in terms of skills transfer.

RCT Entities:   Population Interventions Outcomes

Introduction

Simulation‐based training is an effective method of improving clinical skills, but is highly resource‐intensive.1, 2 The significant time and monetary costs associated with simulation‐based training have led medical educators to search for methods that improve the effectiveness and efficiency of training. Training in pairs – also known as dyad practice – may allow an increased learner‐to‐simulator ratio and thereby improve training efficiency. The effectiveness of dyad practice has been explored in the motor skills learning literature, which demonstrates that learners perform as well or better than those who practise alone despite decreased hands‐on experience.3, 4, 5, 6 Dyad practice may therefore be an attractive method of reducing the costs of simulation‐based medical education without compromising learning.

The conceptual theoretical framework underpinning the effectiveness of dyad practice involves peer‐assisted learning theory,7 neuroscience studies,8, 9 and cognitive load theory (CLT).10 Whereas peer learning theorists emphasise the importance of inter‐subjective cognitive co‐construction, shared knowledge and peer feedback,11 experimental motor learning studies involving dyad practice have attributed the benefits of dyad practice to the effects of peer observation.3, 4, 5, 6 The neurophysiological basis for the effects of observation has been provided by the identification of so‐called ‘mirror‐neurons’ located in the prefrontal motor cortex that are activated when an individual observes and performs the same action.8, 9 In other words, observation elicits an internal representation of the performed actions, which allows rehearsal without any actual hands‐on practice. Cognitive load theory provides yet another view on the potential mechanism of action behind dyad practice.12 From a CLT perspective, dyad practice may divide the cognitive load imposed on learners by shared information processing, thereby making learning more effective and efficient. However, dyad practice may also reduce the effectiveness of learning if the task is characterised by low cognitive load.13, 14

The effect of dyad practice within the simulated setting has been explored in several recent studies in medical education. In a study by Tolsgaard et al.,15 medical students were randomised to dyad or single practice during patient encounter management training in a skills laboratory. A retention test using simulated patients showed the dyad group performed better than the single‐practice group and demonstrated improved confidence in managing future patient encounters.15 In a study involving simulation‐based lumbar puncture training, Shanks et al.16 found no differences in pre‐, post‐ and retention test performances between students who practised alone and those who worked in dyads. However, the dyad group displayed greater pre‐ to post‐test gains than the single‐practice group without using additional practice time.16 Finally, a recent study involving virtual reality coronary angiography training did not demonstrate any significant differences in retention test performances between students who were allocated to individual or dyad practice.17 Consequently, there is increasing evidence that dyad practice improves efficiency for a wide range of clinical skills without compromising learning. However, it remains unknown whether dyad practice is as effective as individual practice in the context of skills transfer from simulated to clinical performances with patients.

The process of transfer describes the application of previously learned concepts to a novel problem.18 The transfer process is influenced by multiple factors19 and performance in a simulated setting may not necessarily reflect how learners perform in the clinical setting with patients.20, 21 Consequently, although dyad practice is as effective as individual practice during simulation‐based training, it does not guarantee equivalent skills transfer. The positive effects of dyad practice on confidence and metacognition reported in previous studies15, 17 indicate that dyad practice may actually improve the chances of successful skills transfer according to social cognitive theory.19, 22 However, the decreased amount of hands‐on experience may reduce dyad participants’ automaticity of skills during simulation‐based practice, which may impair subsequent skills transfer according to the motor skills learning literature.23, 24

The aim of this study was therefore to assess the effectiveness of dyad practice compared with individual practice in the context of the transfer of skills from simulation‐based training in complex skills to clinical performances with patients. We chose a non‐inferiority design to investigate whether dyad training was as effective as individual practice according to recent recommendations.25 In the present study, we focused on simulation‐based ultrasound training because ultrasonography is a complex skill involving technical and diagnostic aspects of performance.

Methods

This study was a randomised, parallel‐group, observer‐blinded, non‐inferiority trial using a 1 : 1 allocation ratio to compare dyad and single practice during ultrasound simulation between 1 January and 1 June 2013. We hypothesised non‐inferiority between dyad and single practice and used pre‐planned specified boundaries according to current recommendations.25 The study is reported according to the CONSORT statement, as well as the CONSORT extension on equivalence trials.26, 27 Sample size calculations for this trial were based on previously published studies (reported below). A flowchart outlining the study is shown in Fig. 1.

Figure 1

Flowchart of the study

Ethical approval was obtained from the Regional Ethical Committee of the Capital Region (protocol no. DRVK‐35596). The study protocol was registered at clinicaltrials.gov (protocol no. NCT01703975).

Participants and setting

Participants in this study were medical students in their final clinical rotations before medical licensing. The group was selected to ensure that all participants had an equal knowledge base and minimal practical ultrasound experience. The medical programme at the University of Copenhagen is a traditional 6 year curriculum which provides basic science teaching during the first half of the programme and clinical sciences teaching as well as clerkship training during the second half. During Year 4, all medical students are required to complete a 3‐hour course on basic abdominal ultrasonography.

Participants were contacted by e‐mail and again during the introductory lectures to their gynaecological rotations. The inclusion criterion required participants to be less than 4 months from medical graduation. Exclusion criteria denied the participation of students who had previously undertaken ultrasound courses or had any experience with any kind of virtual reality simulation or ultrasound in obstetrics and gynaecology. Ultrasound experience was defined as any hands‐on training and did not include the observation of examinations performed by other health care providers. Participants were enrolled on a consecutive basis until the required sample size was reached.

The study was conducted at the Department of Obstetrics and Gynaecology, Copenhagen University Hospital Rigshospitalet. All participants underwent simulation‐based ultrasound training. The gynaecological ambulatory unit was selected because it has a high flow of patients and thus the likelihood that participants would be able to perform transvaginal ultrasound scans was concomitantly high. The patient population included women scheduled for clinical control after medical abortion, as well as women with acute gynaecological conditions (e.g. missed abortion [missed miscarriage], lower abdominal pain). All patients provided written informed consent to their participation in this study. Eligible patients were non‐pregnant or in an early stage of pregnancy and were required to have a body mass index of < 25 kg m−2 to ensure some degree of test condition standardisation. All patients were required to be able to read and understand Danish in order to provide informed consent.

Interventions

Participants were randomised to dyad or single practice before beginning the training. Simulations were performed using a high‐fidelity transvaginal ultrasound simulator (ScanTrainer; MedaPhor Ltd, Cardiff, UK). The ultrasound simulator was equipped with a haptic device that provides realistic force‐feedback to the ultrasound probe. The B‐mode monitor output was obtained from real patients and presented according to the probe's positioning, while a three‐dimensional illustration simultaneously demonstrated probe orientation during the scans (Fig. S1). For this study, we selected the general gynaecological training modules, which included nine basic training exercises focused on probe orientation and systematic examination of a normal female pelvis.

Both groups were provided with a brief introduction (10 minutes) in order to familiarise group members with the simulated environment and the equipment. Participants were then instructed to complete a pre‐test to assess baseline performance before training, as well as a post‐test once they had finished training. They were instructed to perform a systematic ultrasound examination of the female pelvis which included the measuring of endometrial thickness within a time limit of 10 minutes. The 3‐D illustration was not enabled during the pre‐ and post‐tests and case difficulty was low (normal pelvic anatomy, module 1.1). All simulated and clinical ultrasound examinations were recorded to allow for subsequent performance assessment. All tests were completed individually; participants in the dyad group were not allowed to observe one another's performances.

Once the pre‐test was completed, participants were instructed to begin training using nine general gynaecological simulator modules. The training time was fixed for both groups at 2 hours, upon which the post‐test was completed. Participants in the dyad group were instructed to take turns as the active practitioner and to attempt each module once and then to move on to the next module. The participants in the single group, practising alone, were instructed to attempt each module twice. If any of the participants managed to complete all nine modules within 2 hours, they were instructed to repeat the modules from the beginning. Discussion was allowed between the participants working in pairs, but only one participant at a time was involved in hands‐on practice.

All participants were provided with two optional 10‐minute feedback sessions with one of two simulator instructors (MGT or MEM). The feedback was structured according to a framework for the assessment of ultrasound skills, the objective structured assessment of ultrasound skills (OSAUS) (Appendix S1).28 After 2 hours of training, the sessions were ended and participants were briefed on the transfer test scheduled for the following day.

Outcomes

The main outcome measure was clinical performance during ultrasound scans performed the day after the completion of simulation‐based training (transfer test). All participants were assessed on one transvaginal scan by one of two blinded raters (AO or BSO) using the OSAUS scale. Participants were instructed to perform a systematic examination of the uterus and lateral pelvic wall, as well as to assess free fluid in the pouch of Douglas (the recto‐uterine pouch), and to measure endometrial thickness. The rater provided the participant with a short oral introduction to the equipment and setting before the participant performed the scans. During the scans, the rater assisted with technical problems, but provided limited verbal instructions unless patient discomfort prompted intervention. Technical assistance included help with operating the buttons on the ultrasound dashboard (i.e. handling ‘knobology’). The participant was allowed to scan for a maximum of 10 minutes before the rater took over the examination. Participants were excluded from the analysis of transfer test performances if no eligible patients were booked or if no patients provided consent on the day following simulator training.

Secondary outcome measures included the simulator‐based pre‐ and post‐tests, which were assessed using the OSAUS by the same two raters who assessed the transfer tests. Whereas only one of the two raters assessed the transfer tests, both raters assessed all pre‐ and post‐tests. Finally, simulator metrics (i.e. the automated feedback provided by the simulator) were used to assess efficiency during training (Appendix S2).

Validity of assessment instruments and rater training

The OSAUS framework evaluates ultrasound competence and consists of five items pertaining to equipment knowledge, image optimisation, systematic examination, image interpretation, and documentation of findings (Appendix S1). Each item is rated on a 5‐point Likert scale (1 = poor performance, 5 = excellent performance). The OSAUS scale has demonstrated content validity,28 construct validity, and high inter‐rater reliability and internal consistency,29, 30 as well as evidence of structural validity.31 Finally, criterion‐based pass/fail standards have been established for the OSAUS scale in a previous study.30 The validity evidence of simulator metrics used in the assessment of simulated performances has been examined in a previous study.32 Only metrics that demonstrated construct validity were included in the analyses of the present study.

The two raters were specialty registrars with an interest in gynaecological ultrasound. The raters completed comprehensive training in assessing pre‐recorded ultrasound performances until rating consensus was reached, which occurred after three videos. The raters were instructed to assess performances according to the level that can be expected from a recently certified obstetrics and gynaecology consultant.

Randomisation and blinding

The participants were informed and enrolled by MEM and randomised to either dyad or single practice by an independent research fellow (TT) from another department within the same hospital (the Centre for Clinical Education). Randomisation was performed in blocks of two using a web‐based randomiser. The dyads were paired in the order in which they were randomised; hence they had no influence on partner choice. The two raters were blinded to the allocation of participants. Blinding was ensured by training either one dyad or two consecutive single‐practice participants at a time so that two participants were always available for the transfer test on the following day.

Sample size calculations

The primary outcome's non‐inferiority margin was based on the smallest difference considered to be clinically or educationally meaningful. We regarded a difference in OSAUS scores of < 4.6% to be within the non‐inferiority margin. This was based on previously published data, in which trainees demonstrated a mean ± standard deviation (SD) increase in OSAUS scores of 4.60 ± 3.74% for every 2 months of clinical training.30 Hence, dyad training was considered non‐inferior to single‐student training if differences in performance scores were above the non‐inferiority margin of −4.6%. Using a power of 90% and an α‐value of 0.05, 24 participants were needed to establish non‐inferiority between the two groups based on the procedure used to determine sample size in a non‐inferiority trial.33

Statistical analysis

All simulator and performance scores were calculated into percentages of maximum score. The mean OSAUS scores were calculated for the pre‐, post‐ and transfer test performances. Confidence intervals (CIs) for the differences in transfer test OSAUS scores between the two groups were calculated to assess the non‐inferiority hypothesis. The numbers of trainees who achieved transfer test OSAUS scores above a pre‐established criterion‐referenced pass/fail level of 50.0%30 were calculated and compared between the two groups using chi‐squared statistics.

A mixed‐design (2 × 2) repeated‐measures analysis of variance (anova) was performed using the pre‐ and post‐test scores to assess the main effects of training and to test for interactions with type of training (dyad or single). The simulator scores were calculated as the sum of metrics with established validity evidence. Training efficiency was assessed by dividing the simulator score on the final attempt on the nine modules by the total number of attempted modules. Simulator and efficiency scores were compared between the two groups using Mann–Whitney U‐tests.

Internal consistency for the OSAUS items was calculated using Cronbach's alpha; inter‐rater reliability for the pre‐ and post‐test assessments was calculated using intraclass correlation coefficients (ICCs).

Results

Thirty participants were randomised and completed the simulation‐based training; 24 of these completed the transfer test. Baseline characteristics of the participants and mean OSAUS scores on the pre‐, post‐ and transfer tests are shown in Table 1. There were no differences in the distribution of case pathology between groups  = 0.14, p = 0.93) (Table 1). The 95% CI for the differences in transfer test OSAUS scores between the two groups included zero but not the non‐inferiority margin (delta score 7.8%, 95% CI −3.8–19.6%). Hence, the clinical performances of the dyads were not inferior to those of the single‐practice students (p = 0.04), but neither were they superior (p = 0.19). However, a significantly higher number of dyads than single‐practice students achieved scores above the pre‐specified pass/fail level in the transfer test (dyads 71.4%, singles 30.0%;  = 4.03, p < 0.05). The distribution of scores is shown in Fig. 2.

Table 1

Baseline characteristics and performance scores of participants who trained alone (single group) or in pairs (dyad group)

Characteristics	Dyad group (n = 16)	Single group (n = 14)
Men, n	1	0
Women, n	15	14
Age, years, mean (range)	28.8 (27–34)	27.2 (23–31)
Feedback sessions during training, mean ± SD	0.9 ± 0.6	1.1 ± 0.7
Pre‐test OSAUS score, mean ± SD (%)	28.5 ± 4.2	24.9 ± 3.7
Post‐test OSAUS score, mean ± SD (%)	55.5 ± 6.3	49.3 ± 6.0
Transfer test OSAUS score, mean ± SD (%)	56.3 ± 14.8	48.4 ± 11.8
Distribution of cases, %
Incomplete abortion, missed abortion or complete abortion	50.0	50.0
Bleeding in relation to early pregnancy	33.3	25.0
Imminent abortion	16.7	25.0

OSAUS = objective structured assessment of ultrasound skills; SD = standard deviation.

Figure 2

Pre‐, post‐ and transfer test performances of participants trained in pairs (dyads) or alone (singles). OSAUS = objective structured assessment of ultrasound skills

There were no significant interactions between effects of training and type of training (dyad or single) (d.f.(1,28) = 0.72, p = 0.41). There was a significant main effect of simulation‐based training in terms of performance improvements from pre‐ to post‐test (d.f.(1,28) = 509.933, p < 0.01), with large effect sizes (Cohen's d = 3.85). There was no statistical difference between the two groups in final simulator scores (dyads: mean ± SD: 82.1 ± 9.6%; singles: mean ± SD 72.1 ± 21.3%; p = 0.22). However, the dyad group demonstrated significantly higher training efficiency with a mean ± SD of 5.88 ± 1.13 points/attempt compared with the single‐practice group, which achieved a mean ± SD of 2.79 ± 0.92 points/attempt (p < 0.01) (Fig. 3). The ICCs for inter‐rater reliability on the pre‐ and post‐test assessments were 0.66 and 0.40, respectively. Those for internal consistency on the pre‐ and post‐test assessments were 0.77 and 0.84, respectively.

Figure 3

Efficiency of simulation‐based training for participants who worked in pairs (dyads) and alone (singles), respectively. The number of attempts during the 2‐hour training session is plotted against the simulator scores obtained on the final attempt at the nine modules (i.e. the sum score of all passed simulator metrics)

Discussion

Participants in the dyad group had only half the amount of hands‐on time, but their performances with patients were non‐inferior to those of participants who practised alone. In fact, a significantly higher number of participants in the dyad group compared with those in the single‐practice group achieved performance scores above a pre‐established pass/fail level. Consequently, training efficiency was doubled without any sign of negative effects on transfer of learning. The fact that the dyad group needed significantly fewer attempts to achieve a certain simulator score supports the suggestion that dyad participants were able to learn from each other's errors and avoid them during subsequent performance. This confirms existing theories on the mechanisms underlying the efficacy of dyad practice from the motor skills learning and peer learning literature. However, in the present study we did not attempt to explore the relative contributions of observation, feedback, shared knowledge or cognitive co‐construction on the observed performances in the dyad group.

Of the 30 participants included, 24 completed the transfer test, thereby fulfilling the study's targeted sample size. Although this study was powered to demonstrate only non‐inferiority, the dyad group demonstrated superiority with regard to simulator training efficiency and transfer test pass rate. This study used assessment instruments for which validity evidence is documented, rater blinding, and thorough rater training prior to its commencement. Although simulator instructor blinding was not possible, we used standardised instruction and feedback protocols to limit any instructor bias. The fact that the effects of dyad training generalise across several practice domains supports the trustworthiness of the present results. However, the present study included only novice learners, who had limited clinical experience and no prior ultrasound training. By contrast with our findings, Shanks et al.16 hypothesised that novice learners may not benefit from dyad practice because they lack the knowledge and skills necessary for effective peer teaching. This was supported by an experimental study showing that peer‐assisted learning in pairs was inferior to computer‐assisted learning and teacher‐assisted learning.34 However, the training provided in our study included a combination of computer‐ and teacher‐assisted learning, and the effects of dyad practice for novice learners may therefore depend on some element of guidance during training.

In the present study, we examined the effect of dyad practice during complex skills training, but similar results have been demonstrated for simple skills training in a previous study involving lumbar puncture.16 Hence, the effectiveness of dyad practice may not exclusively rely on task complexity, as proposed by Kirschner et al.,13, 14 as the cognitive load imposed during initial simple skills training may be equivalent to the load after several hours of complex skills training. As cognitive load decreases with increasing levels of skills automaticity,10, 24 the benefits of dyad practice may be reduced and this reduction may occur independently of skill complexity. Interestingly, the time allocated to dyad practice during simulation‐based training in the study by Shanks et al.16 (24 minutes), the present study (2 hours), and the study by Räder et al.17 (3.5 hours) seems to be inversely related to the immediate gains in learning of the dyad participants. In the study by Shanks et al.,16 the dyad group demonstrated significantly greater pre‐ to post‐test gains than the single‐practice group, whereas there were no differences between groups in the present study and slightly worse performances in the dyad group in the study by Räder et al.17 The trade‐off between gains from dyad practice in terms of peer observation and shared information processing against the reduction in hands‐on experience may help to explain the reduced effects of dyad practice beyond initial training. Future qualitative studies may help to explore how these issues affect learning.

Limitations

The limitations of this study include its small sample size and the use of single‐case assessments for the transfer test. During the transfer tests, only one rater assessed performances, which may have impacted the reliability of ratings. Although the validity of the OSAUS scale has been explored in multiple previous studies involving different trainees and institutions,28, 29, 30, 31 the validity of a score does not necessarily transfer to a new setting with a different sample of participants. Furthermore, because of the large differences between pre‐ and post‐test performances, the raters may have been able to distinguish between the types of assessment in the simulated setting. Although this may have resulted in the overestimation of the immediate training effects for both groups, it does not affect the interpretation of the main outcome or affect the overall conclusions of this study as the raters were blind to the allocation of participants in the pre‐, post‐ and transfer tests. To limit the case‐mix effect, we aimed for strict selection criteria for the transfer test cases and found no significant differences between groups in the types of pathology participants encountered. To ensure the standardisation of the transfer test conditions between the two groups, we included only scans performed on the day after training. Hence, six of the 30 participants included were unable to complete the transfer test because we lacked eligible patients. We chose to limit the transfer test to include a single scan because aiming for a higher number of scans would have limited the completion rate further.

The use of different methods for assessing pre‐, post‐ and transfer test performances may have resulted in bias as there is some evidence that live performances are rated more highly than video‐recorded performances.35 Hence, the extent of skills transfer to the clinical setting may have been overestimated for both groups of participants as a result of rater leniency. Finally, we examined only the immediate effects of dyad practice on skills transfer and did not attempt to evaluate effects on skills retention. This has, however, been examined in several recent studies, in which no significant differences were found between dyad and single practice in the simulated setting after delays of 2–3 weeks.15, 16, 17, 36

Conclusions

The finding that dyad practice was non‐inferior to individual practice with regard to skills transfer is remarkable as dyad participants had only half the hands‐on training of single‐practice participants. Dyad practice may be used in multiple clinical disciplines to lower the time and monetary costs associated with early simulation‐based training. However, the type of guidance needed to facilitate the positive effects of dyad practice, as well as the effectiveness of dyad practice during different stages of learning are subjects for further studies.

Contributors

MGT contributed to the study design, provided instructions during simulator training, wrote the statistical analysis plans, monitored data collection throughout the trial, analysed the data and contributed to data interpretation. MEM contributed to the study design, provided instructions during simulator training and contributed to data interpretation. CR and AT contributed to the study design, the statistical analysis plans and data interpretation. BSO and AO contributed to the study design, were responsible for the assessment of ultrasound performances in both the clinical and simulated settings, and contributed to data interpretation. JLS and BO contributed to the conception and design of the study, to the funding application process and to data interpretation. All authors contributed to the critical revision of the paper and approved the final manuscript. All authors are responsible for the integrity of the data and are accountable for all aspects of the work.

Funding

this study presents independent research funded by a grant from the University of Copenhagen.

Conflicts of interest

none.

Ethical approval

this study complied with the Helsinki II Declaration on biomedical research. Ethical approval was obtained from the Regional Ethical Committee of the Capital Region (protocol no. DRVK‐35596).

Figure S1. Monitor output from the transvaginal ultrasound simulator.

Click here for additional data file.

Appendix S1. The objective structured assessment of ultrasound skills.

Click here for additional data file.

Appendix S2. Simulator metrics with validity evidence in terms of ability to discriminate between differences in performance in a group of competent performers (obstetrics and gynaecology consultants) and a group of non‐competent performers (medical students).

Click here for additional data file.