Predictors of Performance on the Arthrobox Arthroscopy Simulator for Medical Students.

PURPOSE: The purpose of this study was to analyze the effects of past participation in athletics, the playing of musical instruments and video games and other variables on medical students' performance on an arthroscopic simulator task as well as other assessments of visuospatial ability.
METHODS: We assessed 50 medical students by using previously validated tests of manual dexterity and spatial reasoning as well as performance on an arthroscopic surgical simulator. Inclusion criteria were to be 18 years of age or older and to be a student studying in the M.D. program at a single public state university. Exclusion criteria were previous use of an arthroscopic surgery training device or active participation in an actual arthroscopic surgery, defined as participating as a surgeon, resident trainee, physician's assistant, or other similarly credentialed professional. Students were also assessed by the use of a high-fidelity ultrasound simulator as a marker of visuospatial capacity. Students were then surveyed about lifestyle characteristics and personal attributes hypothesized to predict surgical skill, such as playing sports, instruments or video games.
RESULTS: A total of 49 participants were included in this study. High levels of athletic experience were significantly associated with improved performance on the arthroscopic surgical simulator (P = .008). Participants with higher levels of athletic experience were more likely to achieve competence on the arthroscopic surgical simulator (P = .006). Scores on the arthroscopic simulator task were significantly correlated with both ultrasound simulator shape-identification task scores and masked mirror-tracing task scores, as independent measures of visuospatial ability (P = .015 and P = .013, respectively).
CONCLUSIONS: This study provides evidence of a statistically significant correlation between increased experience in athletics and single-use test performance on an arthroscopic surgical simulator. Subjects who reported higher levels of experience in athletics were significantly more likely to achieve competence in the arthroscopic surgical simulator task. Finally, statistically significant correlations were found between subjects' performance scores on tasks assessed by the surgical simulator, masked mirror-trace assessment and ultrasound simulator. CLINICAL RELEVANCE: Simulator-based training and education allow for the development of arthroscopic skills prior to operating on a live patient in a clinical situation. This is an area of great interest in orthopaedic education. Our study evaluates parameters in a trainee that may relate to a higher performance level in technical skills on an arthroscopic surgical simulator.

A critical component of the comprehensive education of an orthopaedic surgical resident is acquiring proficiency in arthroscopic surgery. However, traditional paradigms of procedural training are often incompatible within this subset of surgical training because of potential complications of scope handoff from instructor to trainee.1, 2, 3, 4, 5, 6 Because of the Accreditation Council for Graduate Medical Education-mandated limits on residency training hours and the growing volume of arthroscopic surgery, residency programs have developed alternative and complementary training strategies to ensure the competency of graduating orthopaedic surgeons.7, 8, 9 The integration of arthroscopic surgical simulation into residency curricula allows trainees risk-free, standardized practice on devices that have been demonstrated to facilitate the development of dexterity, visuospatial awareness and triangulation abilities critical for arthroscopy.10, 11, 12, 13, 14, 15, 16, 17 Simulator-based training and education are highly regarded by trainees and attending surgeons alike, allowing the development of arthroscopy skills prior to stepping into the operating room.18, 19, 20, 21, 22, 23, 24, 25

Many residency programs have established proficiency-based progression curricula using simulator-based training. Used by the Arthroscopy Association of North America, proficiency-based progression models facilitate a stepwise, competency-based progression from low-fidelity to high-fidelity simulators., Standardization of simulation programs allows instructors to analyze trainees’ progress via learning curves, allowing identification of areas of strength and weakness as trainees progress through the curriculum. Prior studies have shown that most trainees demonstrate an initially rapid progression through the learning curve that gradually proceeds to plateau.,27, 28, 29, 30

Although the uses of arthroscopic simulation in postgraduate training have been well documented and continue to build in popularity, some have proposed the potential utility of implementing similar technologies for medical school training and assessment of senior medical students and junior residents. The Arthrobox (Arthrex, Naples, FL) is an experimentally validated, nonanatomic, low-fidelity arthroscopy simulator that has been demonstrated to significantly improve the performance of novice arthroscopists on high-fidelity simulator systems. The device consists of a small cube designed to represent the size of a larger joint space, such as a shoulder or knee, and features a press-fit assembly design. Interchangeable “skill plugs” can be inserted into the cube, and tasks are completed using a small probe and camera through a variety of ports in the device. The fixed, zero-degree camera projects via a USB port onto a laptop computer screen. Previous studies have demonstrated this simulator’s achievement of construct validity as well as its generalized acceptance as a useful tool for training in basic arthroscopy skills.

The present study aimed to assess the existence of nonsurgical experiential factors in early proficiency on a benchtop arthroscopic surgery simulator among first- and second-year medical students. Such “intrinsic” skill in surgery has been hypothesized to involve chiefly visuospatial abilities; thus, our current study employed 2 additional, experimentally validated assessments of visuospatial ability. The mirror-tracing task, which has been used extensively as a marker of visuospatial ability and operational dexterity, involves asking subjects to trace an object while visualizing only the reflection of their hand.33, 34, 35 Additionally, a shape-identification task using a high-fidelity ultrasound simulator was employed. This assessment is useful because ultrasound imaging requires substantial visuospatial skill to mentally map out a 3-dimensional field from 2-dimensional slices; it was, therefore, hypothesized that proficiency on this high-fidelity ultrasound simulator platform would correlate with proficiency on the arthroscopic simulator.36, 37, 38 The purpose of this study was to analyze the effects of past participation in the playing of athletics, musical instruments, video games, and other variables on medical students’ performance in an arthroscopic simulator task, as well as other assessments of visuospatial ability. We hypothesized that significant associations exist between these surveyed variables and performance on an arthroscopic simulator by medical students.

Methods

Fifty volunteers were recruited by e-mailed fliers sent to each member of the first- and second-year classes at a single United States allopathic medical school (240 total students). The flier included a link to an online registration portal, and the first 50 registrants meeting criteria were included in the study. Inclusion criteria were to be 18 years of age or older and to be a student studying in the M.D. program at a single public state university. Exclusion criteria were previous use of an arthroscopic surgery-training device or active participation in an actual arthroscopic surgery, defined as participating as a surgeon, resident trainee, physician’s assistant, or other similarly credentialed professional. Ultimately, no registered students met exclusion criteria. This study was conducted in a quiet room in the clinical skills center at the medical school. Participants received $20 cash for participation in this study.

Tasks

Ultrasound Object Recognition Task

Ultrasound object recognition tasks were completed using the SonoSim ultrasound simulator (SonoSim, Santa Monica, CA). Participants were instructed to use the device to identify the shape of a training block as quickly as possible. Subjects were allotted a 1-minute time limit and were given 2 attempts to identify each shape. Two different shapes were assessed: a cone and a cross. For the cone, the accepted answer was “cone.” For the cross, accepted answers were “cross,” “two crossed rods” and “crossed sticks.” Two incorrect guesses resulted in a failure for that trial. A score for each shape was calculated as the total time taken for a correct guess, with a 30-point penalty applied for an incorrect guess. In the event of 2 incorrect guesses, a 60-point penalty was applied to the final time at which the subject made the second guess.

Masked Mirror Tracing

A star shape was employed as the tracing shape. This shape was selected because the multiple component diagonal lines are considered more difficult to produce, increasing the technicality of this task., The task consisted of a wooden divider placed in front of a mirror such that participants were able to see only their hand and the shape in the mirror. The star shape consisted of 2 concentric stars separated by a 1 cm gap (Fig 1). With their hand underneath the apparatus, participants were instructed to trace between the 2 outlines as quickly as possible while visualizing their hand only via reflection. A score for each subject was calculated as the total time to complete the tracing, with a 5-second penalty applied for each error. An error was defined as each time the trace crossed outside of the boundaries of the 2 stars.

Fig 1

The star shape used in the masked mirror-trace assessment. Participants were instructed to trace between the outer and inner star as quickly as possible, while viewing their hand only through a mirror.

Arthroscopic Simulator Task

The task apparatus consisted of the simulator device placed on a desk at seated height, with the USB camera probe projecting onto a laptop screen directly adjacent to the simulator. Participants were given 1 minute to become familiar with the device without any task shapes inserted. The helix task was firmly fixed into place inside the box, and participants were given 10 minutes to move a small ring around the helix, using only the probe of the simulator (Fig 2). A score for each subject was calculated as the total time to complete the task, with a 5-second penalty applied for each port change.

Fig 2

An image of a medical student completing a task on the arthroscopic surgical simulator.

Survey

A 15-question survey was administered following task completion (Table 1). The survey was administered on a supplied tablet, handed to the participant by 1 of the members of the research team following task completion. Questions deemed “sensitive” or “personal” were given a “prefer not to respond” option; however, this option was not selected by any participants for any questions.

Table 1

A 15-Question Survey Was Administered Following Task Completion

1	Gender	Male; Female; Prefer not to Respond
2	Choose the option that most accurately describes your level of past experience playing sports:	(None); (Novice); (Recreational); (Intramural/Leisure]); (High School/Intercollegiate Club); (Professional/Varsity Collegiate Athletics)
3	Are you currently participating in sports?	Yes; No; Prefer not to respond
4	Choose the sports you have most often participated in, choosing up to 3.	(Categorical list of sports, with an “other” text-box option)
5	Choose the option that most accurately describes your level of past experience playing a musical instrument:	None; (0-2 hours per week); (3-5 hours per week); (6-10 hours per week); (10 or greater hours per week)
6	Do you currently practice playing a musical instrument?	Yes; No; Prefer not to respond
7	Select the instruments you play most commonly, choosing up to 3.	(Categorical list of instruments, with an "other" text-box option)
9	Do you play video games?	Yes; No; Prefer not to respond
10	Do you plan on pursuing a surgical specialty?	Yes; No; Prefer not to respond
11	Have you ever received formal ultrasound training?	Yes; No; Prefer not to respond
12	Select the option that most closely describes your handedness:	Right-handed; Left-handed; Ambidextrous
13)	Have you ever observed an arthroscopic surgical procedure?	Yes; No; Prefer not to respond
14)	Were 1 or more of your parents or legal guardians physicians?	Yes; No; Prefer not to respond
15)	Which year in your medical school training are you currently enrolled?	M1; M2; M3; M4

Data Analysis

Data were collected and entered into a secure Qualtrics standard data form immediately following each task’s completion. All data analyses were conducted with use of SPSS software (version 24.0; IBM, Armonk, NY). ANOVA analysis was conducted to identify significant predictors. Correlation values were assessed by Spearman rank correlation, and stepwise multiple linear regression analysis was employed to evaluate predictors. Significance was assessed at P ≤ 0.05 for all tests. Because of participant error in data-form completion for 1 subject, data from 49 subjects were included for final analysis. A post hoc power analysis was completed for 49 participants, demonstrating a high effect size of f = 0.56 and an α = 0.05 at a power of β = 0.8135859. This analysis assumed 6 groups of athletic experience (0-5) and a primary endpoint of score on the surgical simulator task.

Results

Data from 49 subjects were included for final analysis. An a priori 1-way fixed ANOVA power analysis was conducted for an effect size of f = 0.55 and an α = 0.05 at a power of β = 0.8, and it produced a predicted sample size of 54. Owing to feasibility constraints, 50 participants were recruited, and the sample included 28 males and 21 females. Of the subjects, 30 were first-year students, and 19 subjects were second-year students. After ANOVA analysis, increased level of sports experience was found to be significantly associated with higher proficiency scores on the arthroscopic surgical simulator (P = 0.008) (Table 2). The β coefficient was calculated as β =-56.926, indicating that for each increase in level of sports experience, a subject scored an average of 56.926 points better on the arthroscopic surgical simulator task. Level of sports experience included 6 categories (scored 0-5 on the data sheet) and ranged from “none” (0) to “professional/varsity collegiate” (Fig 3).

Table 2

ANOVA Analysis of the Effects of Hypothesized Predictors on Surgical Simulator Scores

Model	ANOVA Coefficients∗			t	Sig.
	Unstandardized Coefficients		Standardized Coefficients
	B	Std. Error	Beta
1
(Constant)	552.350	419.408		1.317	.196
Gender	29.163	79.992	.070	.365	.718
Athletics Experience	–60.991	21.852	–.483	–2.791	.008
Currently an Athlete	–128.785	80.855	–.282	–1.593	.120
Musical Experience	7.364	26.695	.048	.276	.784
Currently a Musician	–34.354	78.930	–.075	–.435	.666
Video Games	–23.732	79.623	–.055	–.298	.767
Intent to Become a Surgeon	81.482	65.820	.195	1.238	.224
Ultrasound Training	–37.081	67.926	–.087	–.546	.589
Handedness	208.593	230.214	.143	.906	.371
Have Observed Arthroscopy	–40.864	63.886	–.093	–.640	.527
Physicians as Parents	83.760	72.664	.169	1.153	.257
Year in Medical School	34.191	67.192	.080	.509	.614

ANOVA was used to identify surveyed variables with significant associations (Sig.) with the primary outcome, score on the surgical simulator task (dependent variable: arthroscopic surgical simulator score). Significance was assessed at P ≤ .05.

Fig 3

Athletic experience level is associated with improved performance on the surgical simulator task (P = 0.008, n = 49). 0 represents no sports experience, and 5 represents high-level sports experience as assessed by the questionnaire. Proficiency score was calculated as time-to-completion with a 5-second penalty applied for each port-change, such that a lower score represents improved performance.

Competence on the arthroscopic surgical simulator, defined as the ability to complete the task within the 10-minute time limit, was found to be significantly associated with increased experience in athletics, with P = 0.006 by independent samples t test (Tables 3 and 4). Subjects who achieved competence in the task reported an average experience level of 3.7 on a scale of 0-5 (n = 30), while those who failed to achieve competence reported an average experience level of 2.32 (n = 19), which corresponded to the categorical scale of 0 = no experience, 1 = novice, 2 = recreational, 3 = intramural/leisure, 4 = high school/intercollegiate club, and 5 = professional/varsity collegiate athletics.

Table 3

Average Levels of Athletics Experience Between Those Who Achieved Competence on the Surgical Simulator and Those Who Did Not

Group Statistics
Competence	N	Mean	Std. Deviation	Std. Error Mean
Athletics Experience (0-5)
Yes	30	3.70	1.393	.254
No	19	2.32	1.701	.390

Table 4

Independent Samples t Test Analysis of Average Levels of Athletics Experience Between Those Who Achieved Competence on the Surgical Simulator and Those Who Did Not

	t test for Equality of Means
	t	df	Sig. (2-Tailed)	Mean Difference	Std. Error Difference	95% Confidence Interval of the Difference
						Lower	Upper
Athletic experience∗	2.971	32.859	.006	1.384	.466	.436	2.332

Equal variances not assumed.

No significant association was seen between level of musical experience, video game experience, intent to become a surgeon, handedness, ultrasound training, or gender when assessing proficiency on the arthroscopic surgical simulator.

Spearman rank correlation analysis demonstrated a significant correlation between score on the arthroscopic surgical simulator task and score on the masked mirror tracing task (P = 0.013 and r = 0.351), representing a moderate positive correlation (Table 5). Additionally, arthroscopic surgical simulator scores were found to be significantly correlated with scores on the ultrasound simulator task (P = 0.015; r = 0.346).

Table 5

Spearman Rank Correlation for the Association Between the Surgical Simulator Task, the Mirror-Tracing Task and the Ultrasound Simulator Task Scores

	Mirror Tracing Task Score	Ultrasound Simulator Score	Surgical Simulator Score
Spearman rho
Mirror Tracing Task Score
Correlation coefficient	1.000	.212	.351∗
Sig. (2-tailed)		.144	.013
N	49	49	49
Ultrasound Simulator Score
Correlation coefficient	.212	1.000	.344∗
Sig. (2-tailed)	.144		.015
N	49	49	49
Surgical Simulator Score
Correlation coefficient	.351∗	.344∗	1.000
Sig. (2-tailed)	.013	.015
N	49	49	49

Correlation is significant at the 0.05 level (2-tailed).

Level of athletic experience was not found to be significantly associated with improved performance on the mirror-tracing task. Subjects with intent to pursue a surgical specialty and experience observing an arthroscopic procedure also showed a significant association with improved performance on the mirror-tracing task, with P = 0.008 and P = 0.024, respectively.

Among participants, the most common sports included track/cross country (n = 20), basketball (n = 14), football (n = 12) and tennis (n = 11) (Fig 4). Other sports included volleyball (n = 10), baseball (n = 7), soccer (n = 6), and hockey (n = 4). Participants self-reported a wide range of sports experience levels, with 5 responses of “none,” 7 responses of “novice,” 2 responses of “intramural/leisure,” 20 responses of “high school/intercollegiate club,” and 9 responses of “professional/varsity collegiate.”

Fig. 4

Sports participation among subjects: respondents played hockey (4), skiing (3), surfing (3), volleyball (10), track/cross country (20), soccer (6), baseball (7), tennis (11), lacrosse (2), basketball (14), football (12), golf (3), crew/rowing sports (2), snowboarding (1), softball (1), frisbee (1), table tennis (1), wrestling (1), martial arts (1), cheerleading (1), and swimming (1).

Discussion

This study found a statistically significant relationship between increased experience levels in athletics and early proficiency on an arthroscopic surgical simulator task (P = .008). The present study adds to the growing body of evidence that there may exist certain predictors of improved early performance with arthroscopic surgery training tools. By demonstrating a significant association between increased experience in athletics and scores on the arthroscopic surgical simulator task, this study helps to elucidate at least 1 potential source of variance in baseline arthroscopic skill between subsets of medical students. Additionally, the observation that greater experience in sports increased the likelihood of achieving competence on the arthroscopic surgical simulator task further supports this relationship. Interestingly, several prior reports disagreed about whether such an association between groups of trainees existed.,, This study offers 2 advantages over these reports; first, the sample size of n = 49 in this study increases the power to detect such a relationship. Second, stratifying participants across 6 levels of athletics involvement increased the objectivity of subjects’ self-reported experiences.

Most qualified candidates, with proper training, will ultimately achieve competence in the many disciplines of orthopaedic surgery. Research such as this aims only to provide a possible additional criterion for judicious use by selection committees and to characterize potential reasons for observed differences in early proficiency on such simulators. To this end, there has been much research into the differences in cognitive abilities between athletes and nonathletes. In their work, Jansen et al. demonstrated that soccer players showed significantly improved performance on a mental rotation test of visuospatial ability compared to nonathletes. In a meta-analysis by Voss et al., the authors found that male athletes showed increased mental-processing speed in laboratory-based assessments of cognitive ability. Moreover, other groups have found that visuospatial abilities developed through athletic experience are generalizable to nonsport-specific visuospatial tasks.45, 46, 47 Notably, in this study, the largest subgroup of athletes participated in track and cross country, sports not typically requiring substantial hand-eye coordination. This interesting association between non-hand/eye coordination athletic participation and improved initial performance on tasks requiring such coordination may warrant further investigation.

The observation of a significant correlation between scores on the surgical simulator task and both the mirror-tracing task and the ultrasound task provides validation that differences observed in performance on the surgical simulator may be related to differences in visuospatial ability. Interestingly, performance on the mirror-tracing task was not found to be significantly associated with increased levels of athletic experience. However, both intent to pursue a surgical specialty and previous observation of an arthroscopic procedure were both found to be significantly associated with improved performance on the mirror tracing task. The reasons for such associations are unclear.

A notable point of controversy exists between the results of this project and those of other groups. Congruent with the current study, the observation that musical experience and video game experience (VGE) do not predict ability on surgical simulators has been repeated by several different groups.,, However, several groups have found that VGE is, in fact, predictive of skill on similar surgical simulators.49, 50, 51 What, then, can account for this discrepancy? Further subanalysis of the methodology of these reports revealed that significant associations between VGE and skill on surgical simulators were found when VGE was assessed in a total-experience fashion, rather than the binary yes/no scale used in this study. For example, Jentzsch et al. reported VGE by assessing total hours over lifetime, similar to a pack-years approach. The authors of this study, then, posit that such a relationship may, in fact, exist and may not have been captured in this study because of the methodology employed. Therefore, future studies should use a standardized schema quantifying total hours of experience (i.e., hours per week) in assessing video game experience and related attributes. It is also possible that this study was underpowered to detect such a relationship with the methodology employed.

Given the relative paucity of rigorous protocols in this field, the present study intended to build on prior work by Bouaicha et al., who also tracked time to completion and portal changes on the same surgical simulator. To add additional objectivity by providing clear pass-or-fail criteria, the present study added a 10-minute time limit, which was determined by adding approximately 25% of the time to the 75th percentile finishing time of novices in the Bouaicha et al. trial. Additionally, because ultrasound was used in this study, an additional accepted measure of visual-spatial ability, the masked-mirror tracing was included.

Limitations

The authors of this study acknowledge several limitations in the present protocol. The simulator device used in this protocol was a low-fidelity simulator of arthroscopy, which implicitly cannot recapitulate the experience of surgery; thus, skill in this task should not be interpreted as equivalent to skill in actual surgery. Additionally, there was substantial variation in the specific sports played by each respondent, and these groups were not large enough for a meaningful comparison. Although participants were blinded to the purpose of the study during the protocol, the survey assessing participants’ characteristics was administered following completion of all tasks, so there is a possibility that perceived task performance could have biased participants’ responses. This study also failed to show relationships among many of the hypothesized predictor variables and surgical simulator scores; this may be because this study was underpowered to detect such relationships, which may, in fact, exist. This study was limited by the lack of external validation of the protocol currently existing in the literature. The construct validity of simulated training as a teaching method for surgery has also been challenged. Anastakis et al. found that when compared to historical controls, residents trained in a surgical skills center curriculum using bench-top simulators and animal simulated surgical models did not demonstrate significantly improved scores on the Objective Structured Assessment of Technical Skills examination. As such, the utility of improved and updated simulator devices in surgical training should be continually reassessed.

Conclusion

This study provides evidence of a statistically significant correlation between increased experience in athletics and single-use test performance on an arthroscopic surgical simulator. Subjects who reported higher levels of experience in athletics were significantly more likely to achieve competence on the arthroscopic surgical simulator task. Finally, statistically significant correlations were found between subjects' performance scores on tasks assessed on the surgical simulator, masked mirror-trace assessment and ultrasound simulator.