Skill Acquisition and Retention Following Simulation-Based Training in Pavlik Harness Application.

BACKGROUND: Simulation-based learning is increasingly prevalent in many surgical training programs, as medical education moves toward competency-based curricula. In orthopaedic surgery, developmental dysplasia of the hip is a commonly treated condition, where the standard of care for patients less than six months of age is an orthotic device such as the Pavlik harness. However, despite widespread use of the Pavlik harness and the potential complications that may arise from inappropriate application, we know of no previously described formal training curriculum for Pavlik harness application.
METHODS: We developed a video and model-based simulation learning module for Pavlik harness application. Two novice groups (residents and allied health professionals) were exposed to the module and, at pre-intervention, post-intervention, and retention testing, were evaluated on their ability to apply a Pavlik harness to the model. Evaluations were completed using a previously validated Objective Structured Assessment of Technical Skills (OSATS) and a global rating scale (GRS) specific to Pavlik harness application. A control group that did not undergo the module was also evaluated at two time points to determine if exposure to the Pavlik harness alone would affect skill acquisition. All groups were compared with a group of clinical experts, whose scores were used as a competency benchmark. Statistical analysis of skill acquisition and retention was conducted using t tests and analysis of variance (ANOVA).
RESULTS: Exposure to the learning module improved resident and allied health professionals' competency in applying a Pavlik harness (p < 0.05) to the level of the expert clinicians, and this level of competency was retained one month after exposure to the module. Control subjects who were not exposed to the module did not improve, nor did they achieve competency.
CONCLUSIONS: The simulation-based learning module was shown to be an effective tool for teaching the application of a Pavlik harness, and learners demonstrated retainable skills post-intervention. This learning module can form the cornerstone of formal teaching of Pavlik harness application for developmental dysplasia of the hip.

Simulation-based education has become increasingly prevalent in many surgical training programs as a result of restrictions in resident work hours, heightened concerns for patient safety, and a shift toward competency-based curricula[1-4]. Compared with traditional models of medical education, simulation offers a standardized approach to the acquisition and evaluation of technical skills, with limitless practice opportunities and no risk to patient safety[5]. Systematic reviews have shown that simulation-based education provides a learning environment in which the acquired technical skills are transferable to clinical practice[5-8] and are retained over time[9-11]. The features of effective simulation-based education include proficiency-based training, the provision of summative feedback, repetitive practice opportunities, and integration into the standard training curriculum[5,8].

Developmental dysplasia of the hip is a congenital hip joint disorder in newborns with a rate of one to thirty per 1000 live births[12]. The Pavlik harness is one of the most common orthotic devices used to treat developmental dysplasia of the hip in children who are less than six months of age. Used incorrectly, the harness can lead to serious complications, including osteonecrosis of the femoral head and femoral nerve palsy[13,14]. Despite this, we know of no previously described formal training curriculum for Pavlik harness application.

In this study, we developed a module for simulation learning that was based on a previously validated Objective Structured Assessment of Technical Skills (OSATS) for teaching Pavlik harness application, and we evaluated the educational outcomes of the simulation-based learning. We hypothesized that exposure to our learning module would improve novice learners’ skill to the level of that of experts and that these skills would be retained.

The primary objectives were to (1) develop a simulation-based learning module for Pavlik harness application, (2) assess the improvement in skill competence after exposure to the module, and (3) assess skill retention following exposure to the learning module.

Materials and Methods

This study was conducted under approval from our institution’s research ethics board.

Development of the Learning Module

The learning module incorporated the use of instructional media and an infant model. The instructional media was developed with one content expert (S.P.K.), who was recorded on video applying a Pavlik harness to the infant model. Video recording was completed using a GoPro video camera, which was fastened by the GoPro head strap to the expert’s forehead to enable an unobstructed, forward-facing demonstration from a first-person point of view, showing the application of the Pavlik harness. Verbal instructions were later incorporated as narration, as shown in Video 1 (online). The steps and instructions provided in the video followed those previously identified in a validated OSATS for Pavlik harness application[15]. Video editing was performed using Adobe Premiere Pro software (Adobe Systems) and was timed to allow the viewer the opportunity to practice on the simulator model as the steps were explained on the video. Two additional content experts (M.L.M. and C.S.B.) then evaluated the video using the OSATS to ensure that each step was accurately portrayed.

Video 1

Instructional video demonstrating the application of a Pavlik harness on the simulator model per the OSATS.

Procedure

The acquisition of skill competence in Pavlik harness application from exposure to the learning module was assessed using data from forty-six participants in four groups. The first group (experts) consisted of clinicians with expertise in Pavlik harness application and included three staff orthopaedic surgeons, five orthopaedic clinical fellows, and five orthotists. The second group (residents) consisted of ten junior orthopaedic residents. The third group (allied health professionals) consisted of eleven physiotherapists and one occupational therapist who work in pediatric subspecialty care. The fourth group (controls) consisted of eleven residents and allied health professionals who were randomly allocated via number draw. The sample size was derived using an alpha level of 0.05, power of 0.8, and an effect size of 0.6 based on the Cohen sample-size tables for analysis of variance (ANOVA)[16]. All subjects provided written consent to participate and be video recorded during this study.

Prior to any exposure to the learning module, participants in all groups were asked to apply a Pavlik harness (Wheaton Brace) to the simulator model while a GoPro camera was used to record their performance (testing time point 1). The camera was positioned on the participant’s forehead using the GoPro head strap, to provide a recording of the Pavlik harness application from a first-person point of view while ensuring that only the participant’s hands and the Pavlik harness infant model were captured in the video frame. We further de-identified participants by having them wear long-sleeved gowns and remove all jewelry. Video editing and sound removal were undertaken using Adobe Premiere Pro software.

The expert group was not exposed to the learning module and was only tested at time point 1. The expert group’s data were compared with those of the other three groups at each time point.

The resident and allied health groups were exposed to the learning module; participants were allowed as much time and as many repetition opportunities during the learning simulation as they deemed necessary to achieve competency. To assess skill acquisition, both of these groups were recorded and retested in their ability to apply a Pavlik harness following exposure to the learning module (testing time point 2). To assess skill retention, the two groups were also recorded and retested one month post-baseline (testing time point 3).

The control group was not exposed to the learning module but was retested thirty minutes after the baseline assessment to determine if exposure to the Pavlik harness without instruction had any effect on skill acquisition. An overview of the study protocol is presented in Figure 1.

Fig. 1

Study layout. The expert group included thirteen clinicians with expertise in Pavlik harness application, the resident group included ten junior residents, the allied health group included eleven physiotherapists and one occupational therapist, and the control group included eleven control subjects. Groups were tested at three time points: prior to the intervention, immediately after, and at one month after the intervention for retention testing.

Outcome Measures

One content expert (M.L.M.) evaluated all videos using the OSATS and a global rating scale (GRS) for the assessment of performance and final product, which are specific to Pavlik harness application. The OSATS, developed according to international expert consensus using Delphi methodology, is a twenty-five-item rating scale that evaluates the steps required for appropriate Pavlik harness application[15]. Inter-rater and test-retest reliability of all three measures are considered excellent at, respectively, 0.96 and 0.98 for the OSATS; 0.90 and 0.93 for the GRS performance; and 0.93 and 0.94 for the GRS product[15]. These previously established reliability measures obviated the need for multiple reviewers of the videos.

Statistical Analysis

Statistical analysis was undertaken using SPSS software, version 20 (IBM). Differences among the expert, resident, allied health, and control groups at each available time point were assessed using one-way ANOVA. The change in scores among baseline, post-intervention, and retention testing for the two test groups (residents and allied health professionals) were assessed using repeated-measures ANOVA. The change in scores between baseline and repeat testing for the control group were assessed using paired-sample t tests.

Results

None of the participants in either of the test groups (residents and allied health professionals) or the control group had previously applied a Pavlik harness. All self-reported their competence to be either “very weak” or “weak.” Participants in the expert group had prior clinical experience with Pavlik harness application, and all considered their competency in this task as either “good” or “very good.” Group mean scores and standard deviations at each time point are presented in Table I.

TABLE I

Mean Scores at Pre-Intervention (1), Post-Intervention (2), and One-Month Retention (3) Testing Time Points*

	OSATS			GRS Final Product			GRS Performance
Group	1	2	3	1	2	3	1	2	3
Residents	11.14 (3.61)	23.90 (1.26)	23.29 (2.22)	2.00 (0.84)	3.95 (0.50)	3.95 (0.81)	1.86 (0.79)	4.10 (0.54)	4.05 (1.02)
Allied health professionals	11.33 (2.90)	23.50 (1.45)	24.25 (1.22)	1.92 (0.79)	3.92 (0.52)	4.25 (0.62)	1.83 (0.72)	4.00 (0.43)	4.25 (0.97)
Controls	11.73 (2.83)	11.09 (2.84)	NA	1.91 (0.54)	1.91 (0.30)	NA	1.82 (0.60)	1.73 (0.47)	NA
Experts	21.1 (2.29)	NA	NA	4.38 (0.71)	NA	NA	4.28 (0.72)	NA	NA

The values are given as the mean, with the standard deviation in parentheses. NA = not applicable.

Pre-Intervention Testing

The overall ANOVA demonstrated that there were differences among the groups in the OSATS and GRS scores at time point 1 (p < 0.001). Post-hoc testing using the Tamhane T2 analysis demonstrated that the scores of the resident, allied health, and control groups at time point 1 did not differ significantly, but all of these groups had significantly lower scores than the expert group on each of the three measures (p < 0.001).

Post-Intervention Testing

The scores of the resident and allied health groups improved post-intervention, and there were no significant differences found at time point 2 among the expert, resident, and allied health groups for any of the three measures, with the exception of a slightly higher OSATS score in the resident group compared with the expert group (p = 0.006). Pre-intervention and post-intervention Pavlik harness application is shown in Figures 2 and 3, respectively.

Fig. 2

Improper application of a Pavlik harness prior to the educational intervention.

Fig. 3

Improved application of a Pavlik harness, with appropriate hip adduction and flexion, following the educational intervention.

Retention Testing

Overall ANOVA demonstrated no significant differences at time point 3 among the experts, residents, and allied health professionals for both GRS measures, but there was a significant difference in the OSATS scores at the one-month retention test. Post-hoc testing of the OSATS scores revealed that the significant difference was between the allied health group and the experts, with the allied health group scoring slightly higher on the OSATS (p = 0.005; 95% confidence interval [CI], 0.8 to 5.1).

Repeated-Measures Testing

Repeated-measures ANOVA of the scores of the resident and allied health groups demonstrated a significant difference between baseline and post-intervention testing and between baseline and retention testing for all three measures (p < 0.001). A comparison of post-intervention and retention testing demonstrated no significant differences in GRS scores for either the resident group (p = 0.31 for performance and p = 0.31 for product) or the allied health group (p = 0.12 and 0.99) and no significant difference in OSATS scores for the allied health group (p = 0.54). However, there was a small decrease in OSATS scores (p = 0.04) for the resident group at retention testing.

All individual participants in the resident and allied health groups were scored as competent on all three scales post-intervention and at retention testing.

Control Group Testing

While there were no significant differences among the resident, allied health, or control groups at pre-intervention testing, the controls were found to be significantly worse than all other groups at time point 2 for all three measures (p < 0.001). Paired-sample t tests revealed no difference for the control subjects between time point 1 and time point 2 testing for any of the measures. In addition, no participant in the control group was scored as competent on any of the measures at either time point.

Discussion

There has been a recent shift from traditional time-based residency training toward competency-based curricula. Evidence regarding improved resident satisfaction, efficiency in training, and proficiency using this model is emerging[17], and simulation-based education offers an effective educational basis for such training programs.

Developmental dysplasia of the hip is a very common pediatric orthopaedic condition, and the standard of care for patients who are less than six months of age is the use of an orthotic device such as the Pavlik harness. In our institution, the clinical management of these patients as well as the contraindications, complications, and use of orthotic devices are well taught. However, understanding the technical application of a Pavlik harness is an essential part of an orthopaedic residency curriculum for which, to our knowledge, no formal training exists. This study addresses the lack of previously published educational methods for teaching this skill and has demonstrated that the developed simulation module is effective among learners in achieving and retaining competency.

The importance of repetitive practice in simulation-based education is well known, but there is a lack of consensus in the literature regarding specific metrics[8]. The authors of previous studies have reported a range of repetitions deemed necessary for optimal acquisition of surgical skills, from nineteen practice sessions for medical knot-tying[18], to ten additional repetitions once competency had been achieved for a laparoscopic procedure[19], to enough repetitions until automaticity is reached[20]. However, for simpler tasks, it has been noted that overtraining may adversely affect performance[20]. In our study, learners were allowed as many practice opportunities with the instructional media and the simulator model as they thought necessary, and all learners were able to reach competency without requiring a set number of practice events. This suggests that Pavlik harness application can be learned relatively easily with our module and that it accommodates different learning styles.

Retention of skill is an important measure of any educational intervention. However, skill retention, skill decay, retraining, and interventions for skill maintenance are poorly understood[8]. There is some evidence to support retraining for skill maintenance, but there is little consensus regarding appropriate intervals for skill-retention testing. In one study of skill retention in training for robotic-assisted surgery, the authors reported no skill decay at one and three months of retention testing[21]. In another study of simulation-based training in laparoscopy, minor skill decay was noted in the first two weeks following training, but skills then stabilized in the seven-month follow-up, with no retraining in the interim[19]. These findings suggest that one-month retention testing is sufficient and that interval was, therefore, selected for this study. Among our test groups, we demonstrated the retention of skill at one month post-intervention with no evidence of decay in competency.

A group of novice learners who had not previously applied a Pavlik harness or been instructed on Pavlik harness application was included in this study. The groups that underwent exposure to the learning module improved in their Pavlik harness application skill to the level of the experts, while the control group showed no improvement. The residents scored higher than the experts in post-intervention OSATS scores. This difference reflects the nature of the OSATS checklist; the residents achieved more points on the task-specific checklist on which they were trained. The expert group may have missed a few points or specific steps, as they were not trained on the OSATS. However, the GRS final product and GRS overall performance scores were indistinguishable between the residents and the experts.

We also demonstrated that our module can be used by physical and occupational therapists to effectively learn Pavlik harness application. Allied health practitioners are increasingly delivering clinical care, and our study provides support for training these learners, who may serve as a useful adjunct to the surgeon’s clinical practice, in specialized procedures. A physiotherapist-directed model of care was previously shown to be effective in orthopaedics, where Ponseti treatment of idiopathic clubfoot by physiotherapists resulted in fewer interventions with similar clinical outcomes compared with surgeon-directed care[22].

While there is well-established evidence demonstrating that skills learned in a simulation environment are transferable to the clinical setting[5], the primary limitation of the present study was that the direct link to clinical outcome was not evaluated. However, this study establishes our module as an effective learning tool and is the first step toward incorporating simulation-based learning into studies of clinical impact and outcomes. Our instructional media tool and the affordability of our simulator model render study expansion both economic and feasible.

Another limitation of our study was that we did not provide our learners with summative feedback. It has previously been identified that skill retention may be improved on the basis of how and when feedback is provided during simulation training[18]. However, we chose not to provide summative feedback in this study because we feared that information might have been shared among participants, thus confounding our results. Despite this, all learners improved in their ability to apply a Pavlik harness to the level of an expert clinician and maintained this competency at retention testing.

In summary, the simulation-based learning module developed in this study has been shown to be an effective tool for teaching the application of a Pavlik harness, and learners demonstrated skill retention post-intervention. This learning module can form the cornerstone of formal teaching of Pavlik harness application for developmental dysplasia of the hip.