Practicing Procedural Skills Is More Effective Than Basic Psychomotor Training in Knee Arthroscopy: A Randomized Study.

BACKGROUND: Simulator-assisted arthroscopy education traditionally consists of initial training of basic psychomotor skills before advancing to more complex procedural tasks.
PURPOSE: To explore and compare the effects of basic psychomotor skills training versus procedural skills training on novice surgeons' subsequent simulated knee arthroscopy performance. STUDY
DESIGN: Controlled laboratory study.
METHODS: Overall, 22 novice orthopaedic surgeons and 11 experienced arthroscopic surgeons participated in this study, conducted from September 2015 to January 2017. Novices received a standardized introductory lesson on knee arthroscopy before being randomized into a basic skills training group or a procedural skills training group. Each group performed 2 sessions on a computer-assisted knee arthroscopy simulator: The basic skills training group completed 1 session consisting of basic psychomotor skills modules and 1 session of procedural modules (diagnostic knee arthroscopy and meniscal resection), whereas the procedural skills training group completed 2 sessions of procedural modules. Performance of the novices was compared with that of the experienced surgeons to explore evidence of validity for the basic psychomotor training skills modules and the procedural modules. The effect of prior basic psychomotor skills training and procedural skills training was explored by comparing pre- and posttraining performances of the randomized groups using a mixed-effects regression model.
RESULTS: Validity evidence was found for the procedural modules, as test results were reliable and experienced surgeons significantly outperformed novices. We found no evidence of validity for the basic psychomotor skills modules, as test scores were unreliable and there was no difference in performance between the experienced surgeons and novices. We found no statistical effect of basic psychomotor skills training as compared with no training (P = .49). We found a statistically significant effect of prior procedural skills training (P < .001) and a significantly larger effect of procedural skills training as compared with basic psychomotor skills training (P = .019).
CONCLUSION: Procedural skills training was significantly more effective than basic psychomotor skills training regarding improved performance in diagnostic knee arthroscopy and meniscal resection on a knee arthroscopy simulator. Furthermore, the basic psychomotor skills modules lacked validity evidence. CLINICAL RELEVANCE: On the basis of these results, we suggest that future competency-based curricula focus their training on full knee arthroscopy procedures. This could improve future education programs.

Knee arthroscopy is an essential procedure to master during residency training[18] and requires a specific psychomotor skill set.[13] Increased focus on patient safety has led to a shift toward simulation-based medical education of procedural skills,[17] and solid evidence shows its positive effects[9] with significant impact on patient-related outcomes.[35] Within knee arthroscopy, transfer of skills from computer-assisted simulators to live surgery on actual patients has been demonstrated.[8] However, considerable costs related to acquiring and maintaining surgical simulators as well as support staff and faculty[23] make it imperative to explore the effect of training curricula and find the most effective ways for resident surgeons to acquire the arthroscopic skills needed in the operation room.[2]

In 2011, a collaboration of the Arthroscopy Association of North America, the American Academy of Orthopaedic Surgeons, and the American Board of Orthopaedic Surgery initiated the Fundamentals of Arthroscopy Surgery Training (FAST) program, as a comprehensive training curriculum based on the assumption that “basic surgical skills are best developed sequentially. It assumes that advanced proficiency should be predicated upon successful completion of a basic skills curriculum.”[25] The program suggests approximately 20 hours for the completion of the Basic Triangulation Skills submodule.[26] However, evidence from laparoscopy questions whether focus on basic psychomotor skills is an effective way to learn more complex tasks and how good the transferability of skills between procedures really is.[19,20,28]

The aims of this study were to explore and compare the effects of basic psychomotor skills training and procedural skills training on novice surgeons’ subsequent procedural knee arthroscopy performance. We hypothesized that basic psychomotor skills training improves performance in subsequent simulated full procedures and that training basic psychomotor skills before advancing to procedural skills training would be a more effective strategy than training procedural skills alone.

Methods

Study Design

A single-center randomized educational and validation study was conducted from September 2015 to January 2017 in accordance with the Consolidated Standards of Reporting Trials statement.[21] For the randomized part of the study, a computerized 1:1 randomization with varying block size was done. Participation in the study was voluntary. Before enrollment, all participants were provided written information about the study and filled in a questionnaire regarding operative and simulator experience. All participants signed informed consent. This study was deemed exempt from ethical approval.

The ARTHRO Mentor (3D Systems)[1] computer-assisted simulator with the FAST and Advanced Knee software was used for standardized training and performance testing (Figure 1). The simulator consists of a computer and a stand with a fiberglass model of a right knee, with 2 premade portals, the anterolateral portal and the anteromedial portal. Two robot arms (PHANTOM Omni; Sensable) mimic the surgical tools (arthroscope, probe, shaver, punch, etc) and generate haptic feedback. The arthroscopic picture is generated on a screen on top of the stand and reflects the position of the surgical tools and external manipulation of the knee joint.

Figure 1.

Virtually generated arthroscopic pictures from the ARTHRO Mentor computer-assisted simulator. (A) Basic psychomotor skills module (FAST software). (B) Resection of a lateral meniscus flap tear (Advanced Knee software). FAST, Fundamentals of Arthroscopy Surgery Training.

Performances were measured using the automated simulator metrics, and the validity of this approach was established using the framework proposed by Messick.[30] Validity evidence was collected from the first 4 sources: content, response process, internal structure, and relations to other variables. No changes or updates were made to the simulator during the study.

Participants

Novice participants were recruited for the randomized part of the trial through invitations sent to all orthopaedic departments in eastern Denmark. The inclusion criterion for novices was being a medical doctor employed at an orthopaedic department as an intern or resident. The exclusion criterion was prior performance of unsupervised knee arthroscopy; this was chosen to make the novice group reflect the target group of the simulator training. We aimed at including approximately 10 participants in each group, as is often done in educational studies to ensure the clinical relevance and sufficient power to detect meaningful and statistically significant findings.

For the validation part of the study, we recruited 11 consultant orthopaedic surgeons with arthroscopic subspecialization and at least 100 independent knee arthroscopies. Exclusion criteria for both groups were (1) experience with the FAST and Advanced Knee modules on the ARTHRO Mentor simulator and (2) >2 hours of simulation-based arthroscopy training within the last year before enrollment.

Intervention

The novice participants had to prepare by reading a purpose-made booklet covering the fundamentals of knee arthroscopy, before receiving a standardized introductory 60-minute in-person lesson by an arthroscopy specialist (P.G.J.). The lesson included an introduction to the arthroscopy equipment, the use of angled optics, and triangulation, as well as the use of the most common instruments, including the probe, punches, and a shaver and their respective dangers. Subsequently, a standard diagnostic knee arthroscopy was demonstrated, and novices performed supervised training on a bench-top model (CLA) of a right knee. The novices were subsequently randomized to either the basic skills training (BST) group or the procedural training (PT) group and completed 2 sessions on the ARTHRO Mentor simulator over 2 days, aided by a medical student employed at the simulation center.

During the first training session, the BST group performed 3 repetitions of 2 basic psychomotor skills modules, Periscoping and Basic Probe Triangulation, before training 11 basic psychomotor skills modules included in the simulator’s FAST software. In the Periscoping module, a number of spheres inside boxes have to be visualized by use of the angled optics, whereas in the Basic Probe Triangulation module, a number of spheres have to be visualized and probed to train triangulation skills (Figure 1). The additional basic psychomotor skills modules range from image orientation and steadiness of the camera to tracking and probing of moving targets and measurement of dimensions with the tip of the probe, according to the FAST module 2 protocol.[26]

The PT group performed 3 repetitions of a diagnostic arthroscopy (Basic Probe Examination) and a resection of a lateral meniscus flap tear (Flap Tear). The last of the 3 repetitions yielded the pretraining performance of the PT group.

The Basic Probe Examination module is a diagnostic knee arthroscopy procedure where all compartments of the knee are systematically examined by visualizing and probing spheres that appear sequentially in a fixed order and that disappear after they have been probed. The Flap Tear module is a therapeutic task where a tear in the lateral meniscus must be visualized, resected with a punch, and shaved. A variety of straight and angled punches can be chosen.

In the second session, both groups performed 3 repetitions of the Basic Probe Examination and Flap Tear, with the last of these repetitions yielding the posttraining performance of both groups.

Three repetitions were chosen so that participants had enough time to familiarize themselves with the simulator and eliminate mistakes not related to their respective proficiency levels, while ensuring that the training sessions could be completed within a feasible time frame. Based on experience from a previous study exploring simulation-based knee arthroscopy[16] and similar studies in other subspecialties of orthopaedics[15] as well as other surgical specialties,[32] we have found that there is often an initial steep test performance curve when using a new simulator. This is unlikely to relate to learning but to familiarization of the simulator for experts and novices alike. Thus, using the first repetition could represent a source of bias.

The experienced surgeons had 1 session consisting of 3 repetitions of Periscoping and Basic Probe Triangulation, followed by 3 repetitions of Basic Probe Examination and Flap Tear. For all groups, standardized written help (eg, cues for image orientation, anatomic landmarks) was offered for the first 2 repetitions of a given module, whereas the participants received no help during the third repetition.

A time cap of 3 minutes was set on the shaving part of the Flap Tear module, as it was discovered during a pilot test that the simulator was not able to exactly reproduce how the shaver works in real life. For example, the software allowed the meniscus to be pushed below the articular surface of the tibia, leading some test takers to spend a long time where content would not reflect the underlying construct.

Outcomes

The primary outcome was a calculated composite z score corresponding to the combined overall performance in the last repetition of the 2 procedural modules: Basic Probe Examination and Flap Tear. The following simulator metrics provided in both modules were chosen by an orthopaedic consultant (A.G.): total movement of camera in centimeters, total movement of tools in centimeters, total time of blind movement of tools in seconds, total number of collisions (number of camera-tissue collisions, number of capsular collisions, and number of times that irrelevant structures were touched with the probe), and time to complete tasks in seconds. For all these metrics, a lower score was considered better.

The secondary outcomes were performance scores for the simulator metrics provided by the 2 psychomotor skills modules: Periscoping and Basic Probe Triangulation. Simulator metrics that were included in both modules were chosen and consisted of a precision score in percentages (where a higher score is better) and time to complete tasks in seconds (where a lower score is better).

Statistical Methods

Validity evidence for the basic psychomotor skills and the procedural modules was explored by calculating reliability indices (Cronbach alpha and Pearson r) for the relevant simulator metrics (evidence for internal structure). Furthermore, performances by the experienced surgeons and the 2 groups of novices were compared with independent-samples t tests and a mixed-effects regression model (evidence for relationship to other variables).

Using mixed-effects regression, the posttraining performance of the BST group was compared with the pretraining performance of the PT group to explore the effect of basic psychomotor skills training on the performance of procedural skills. The pretraining performance of the PT group was compared with its own posttraining performance to explore the effect of training procedural skills. Finally, the posttraining performances of the BST group and the PT group were compared to explore the potential difference in effect between basic psychomotor skills and procedural skills training.

In the mixed-effects regression, we specified groups as fixed effects; random effects were specified for the measures and the assessment to allow individual variation in the regression models for these facets. Maximum likelihood was used in the estimation. P values of mixed-effects regression coefficients were used to examine group differences. Differences in performance were considered significant at P < .05. Statistical analyses were made using Stata Version 16 (StataCorp LLC).

Results

A flowchart of participant enrollment and completion is shown in Figure 2. Overall, 33 participants were enrolled in the study. Of those, 22 were novice surgeons and were randomized. Eighteen randomized participants completed the study. Three novices in the PT group and 1 novice in the BST group did not complete the study: 2 were discontinued because of a simulator malfunction that ultimately led to the halt of data collection, and 2 were lost to follow-up and did not respond to contact. The BST group had performed significantly more supervised arthroscopies within the past year (P = .042); otherwise, the 2 groups had similar baseline characteristics (Table 1). Eleven experienced surgeons were included, and 10 completed the study. One withdrew because of a lack of time. The first training session of the BST group lasted 90 minutes, whereas all the other sessions were completed in <60 minutes by all participants.

Figure 2.

CONSORT flow diagram for the study participants. CONSORT, Consolidated Standards of Reporting Trials.

Table 1

Participant Characteristics

Characteristic	PT Group	BST Group	Experienced Surgeons
Age, y, median (range)	28 (26-55)	30 (27-31)	53 (41-61)
Sex, female:male, n	3:6	3:6	0:10
Dominant hand: right, n	9	8b	10
Experience, mo, mean ± SDc	15.44 ± 27	9.44 ± 5	233 ± 114
No. of supervised knee arthroscopies performed, mean (range)
Overall	0.22 (0-2)	2.89 (0-10)	1615 ± 1264d
Within past year	0.22 (0-2)e	2.33 (0-6)e	110 ± 46d
Days, mean ± SD
To complete course	37 ± 22	34 ± 19	NA
Between training sessions	11 ± 7	13 ± 8	NA

Blank cells indicates not available. BST, basic skills training; NA, not applicable; PT, procedural training.

Ambidextrous, n = 1.

Indicates length of time as an orthopaedic surgery intern/resident or an orthopaedic surgeon (including residency).

For the experienced group, number of unsupervised knee arthroscopies.

Significant difference between PT and BST groups (P = .042).

Validity Evidence

The internal structure of the basic psychomotor skills modules displayed a very low level of consistency, with a Cronbach alpha of 0.07 (P = .4). The test-retest reliability was also very weak, with a correlation of –0.03 (P = .9) for time to complete task and 0.09 (P = .7) for the precision score (Table 2). We found no statistically significant difference in performance scores between novices and experienced surgeons for the basic psychomotor skills modules (Table 3).

Table 2

Validity Evidence

		Validity Evidence
Source of Validity Evidence	Questions Related to Each Source of Evidence	Basic Psychomotor Skills Modules	Procedural Skills Modules
Content	Does the content reflect the underlying construct?	Tasks are aligned with construct	Tasks are aligned with construct
Response process	Are sources of bias reduced?	Standardized written instructions Only objective simulator metric scores are used	Standardized written instructions Only objective simulator metric scores are used
Internal structure	Is the test score reliable?	A very low level of reliability is seen: Cronbach alpha = 0.07 Very low level of correlation of scores between tasks: –0.03 for time to complete task (P = .9) and 0.09 for precision score (P = .7), both Pearson r	High level of reliability shown:   Cronbach alpha = 0.91 for each module   Cronbach alpha = 0.87 for the 2 modules combined
Relations to other variables	Does performance correlate with a known measure of competence?	No difference in scores between novices and experienced arthroscopic surgeons	Experienced surgeons performed 0.86 SD better than novices (P < .001)

Table 3

Performance Scores for Novices and Experienced Arthroscopic Surgeons in the Basic Psychomotor Modules

Module: Simulator Metric	Novices (n = 9)	Experienced Surgeons (n = 10)	P
Periscoping
Time score	122.7 ± 22.7	122.4 ± 39.4	.99
Precision score	42.7 ± 7.5	43.8 ± 13.7	.84
Basic Probe Triangulation
Time score	80.1 ± 31.3	63.9 ± 6.0	.13
Precision score	21.7 ± 7.5	23.3 ± 11.6	.71

Data are reported as mean ± SD.

The procedural modules displayed a high level of consistency between performance scores, with a Cronbach alpha of 0.91 for the scores in the Basic Probe Examination module, 0.91 for the scores in the Flap Tear module, and 0.87 for the 2 combined. Relationship to other variables showed that the performance of the experienced surgeons was 0.86 standard deviations better than the pretraining performance of the novices in the PT group (P < .001).

Effect of Basic Psychomotor Skills and Procedural Skills Training

We found no statistical effect of basic psychomotor skills training (P = .49). There was a statistically significant effect of procedural skills training, as the posttraining performance of the PT group was 0.54 standard deviations better than its pretraining performance (P < .001). There was a significantly larger effect of procedural skills training than basic psychomotor skills training, as the posttraining performance of the PT group was 0.40 standard deviations better than that of the BST group (P = .019) (Table 4).

Table 4

Effect of Basic Psychomotor Skills Training and Procedural Skills Training

Effect of basic psychomotor skills training

Comparison: BST group posttraining vs PT group pretraining

Result: No statistical difference between groups (P = .49)

Effect of procedural skills training

Comparison: PT group pretraining vs PT group posttraining

Result: Posttraining performance of the PT group 0.54 SD better than pretraining performance (P < .001)

Effect of basic psychomotor skills training vs procedural skills training

Comparison: BST group posttraining vs PT group posttraining

Result: PT group 0.40 SD better than BST group (P = .019)

BST, basic skills training; PT, procedural training.

Discussion

The most important finding of the present study is that procedural skills training was significantly more effective than basic psychomotor skills training with regard to performance in diagnostic knee arthroscopy and meniscal resection on a computer-assisted knee arthroscopy simulator, refuting our hypothesis of prior basic psychomotor skills training being a more effective strategy. Surprisingly, we found no significant effect of basic psychomotor skills training as compared with no training; hence, our results also did not support the hypothesis that basic psychomotor skills training improves performance in subsequent simulated full procedures. Although we cannot rule out that this is the result of a type 2 error attributed to a small sample size, there was a striking contrast to the significant effect of procedural skills training, even with the same sample size and with less total time spent training.

Our findings are somewhat contrary to those of Bouaicha et al,[5] who, in a group of arthroscopy-naïve medical students, found that three 1-hour training sessions on a low-fidelity arthroscopy simulator box resulted in improved performance on a high-fidelity computer-assisted knee arthroscopy simulator as compared with no training. It would be interesting to compare the low-fidelity simulator box–trained group with a group spending the same amount of time training on more complex procedure-specific tasks. Our results, however, were comparable with the results of Wang et al,[34] who found that 3 sessions of basic psychomotor skills training on a computer-assisted arthroscopy simulator resulted in improved performance in the same skills as compared with no training but not in better performance of diagnostic arthroscopies of knee and shoulder joints on a cadaver. This indicates that there might be limited transfer of skills from training fractionated tasks[29] to whole-procedure tasks and among different tasks. This is in line with the findings of Ferguson et al[13] that improved performance through practice with simulated diagnostic arthroscopy in either a knee or shoulder joint did not transfer to subsequent improved performance of simulated diagnostic arthroscopy in the other joint (shoulder or knee). Also, Ode et al[24] found that training to proficiency in simulated knee arthroscopy did not improve performance in wrist arthroscopy on cadaveric specimens. Likewise, in another area of orthopaedics, Dubrowski et al[12] suggested that practicing the entire task of bone plating yielded more learning than practicing isolated individual skills. In other surgical fields, Bjerrum et al[4] found little transfer of skills from one laparoscopic procedure to another, while Thomsen et al[31] found no evidence of transfer from one intraocular surgical procedure to another.

Thus, our findings imply that extensive training of basic psychomotor skills does not appear to be an effective training strategy. Commonly incorporating this type of skills training into surgical skills curricula may be an ineffective way to use our resources, given the number of hours required from supervisors and trainees and the wear and tear on the often costly simulators.[23,36] In our study, a short lecture directly followed by training the full procedure was sufficient to generate the wanted learning and improved procedural performance.

To our surprise, we found no evidence of validity for the 2 basic psychomotor skills modules (FAST), and our data did not give any indication that this could be a type 2 error. In other words, the FAST modules were so different from the daily clinical work of consultant arthroscopic surgeons that their extensive clinical experience did not result in superior performance on the basic simulator modules, at least for the available simulator metrics. This aligns with a study by Tofte et al[33] in which 3 metrics from another simulator (composite score, time to completion, and camera path length) did not significantly correlate with the number of arthroscopy cases performed by individual residents for the 3 FAST modules. Interestingly, Tofte et al found significant correlations between clinical experience and simulator performance for whole-procedure modules (diagnostic knee and shoulder arthroscopy). This aligns with our findings of evidence of validity in all 4 included domains for the 2 whole-procedure modules on the simulator.

Although not specifically mentioned, the proposed educational approach of the FAST program[26] is comparable to that of mastery learning,[10] where predefined proficiency is to be achieved before moving on to the next objective. Evidence suggests that this is indeed an effective strategy,[10] but validity evidence is imperative with regard to defining proficiency levels. In our setting, there was no evidence that the FAST modules measured any clinically relevant competencies, which makes it difficult to demand that trainees obtain certain predetermined (meaningless) scores. However, our findings add to the existing validity evidence for simulation-based procedural tasks, which makes it possible to implement the mastery learning approach in a training program with procedure-specific tasks. A recent systematic review and meta-analysis suggested that the impact of skills training on an arthroscopy simulator can vary widely depending on the simulator and the training methodology.[14] Logically this calls for more studies investigating the transferability of skills from simulators to actual operations, but to the same extent, it emphasizes the need for investigating the optimal content of training curricula, as addressed in the present study.

In light of our results and this discussion, we suggest that a training curriculum for basic knee arthroscopy focus on the practice of full procedures on a simulator—that is, reflecting the actual procedures that the trainee is going to perform—until a predefined proficiency is met. If available, moving on to practicing procedures on a cadaver might represent an effective next step before performing operations on live patients. However, there are significant costs and logistics related to cadaveric surgery. Likewise, the issue of standard setting and performance evaluation represents a great challenge with cadaveric surgery as compared with computer-assisted simulators, where performance metrics are objective and often readily available.

We acknowledge several limitations to our study. First is the small sample size, which introduces the possibility of type 2 error; however, this is comparable to similar studies in the field, with sample sizes ranging from 14 medical doctors and 18 medical students to 45 medical doctors.[3,7,11,13,16,24,27,33] We would have liked to continue inclusion of participants, but unfortunately data collection had to be stopped because of a simulator malfunction that led to 2 dropouts and study termination. The bench-top model used for the introductory lesson and the computer-assisted simulator included only right knee anatomy, and with regard to our results, training on left as well as right knee anatomy intuitively seems necessary to enhance dexterity. Moloney et al[22] found significantly more malpositioned sliding hip screws in the left hip versus the right hip, and Buyukdogan et al[6] suggested different setups for right versus left hip arthroscopies. This is an understudied area, but transfer of skills from one laterality to another is an obvious topic to explore for future educational studies. Finally, the effectiveness of prior basic psychomotor skills training on whole-procedure performance was examined only in a computer-assisted simulated environment—which in itself is a fractionated version of the surgical task of arthroscopy[29]—and should ideally be tested on performance in real surgery.

Intuitively, basic psychomotor training is effective and transfers to improved performance of arthroscopy procedures. However, this perceived effect was too small to be detected in our randomized study, which included 33 trainees and experienced consultants in orthopaedic surgery. In conclusion, our results indicate that training procedural skills on a virtual reality simulator is a more effective way to prepare trainees for performing arthroscopic procedures of the knee as compared with training basic psychomotor skills.