Validation of a Cognitive Task Simulation and Rehearsal Tool for Open Carpal Tunnel Release.

BACKGROUND: Carpal tunnel release is one of the most common surgical procedures performed by hand surgeons. The authors created a surgical simulation of open carpal tunnel release utilizing a mobile and rehearsal platform app. This study was performed in order to validate the simulator as an effective training platform for carpal tunnel release.
METHODS: The simulator was evaluated using a number of metrics: construct validity (the ability to identify variability in skill levels), face validity (the perceived ability of the simulator to teach the intended material), content validity (that the simulator was an accurate representation of the intended operation), and acceptability validity (willingness of the desired user group to adopt this method of training). Novices and experts were recruited. Each group was tested, and all participants were assigned an objective score, which served as construct validation. A Likert-scale questionnaire was administered to gauge face, content, and acceptability validity.
RESULTS: Twenty novices and 10 experts were recruited for this study. The objective performance scores from the expert group were significantly higher than those of the novice group, with surgeons scoring a median of 74% and medical students scoring a median of 45%. The questionnaire responses indicated face, content, and acceptability validation.
CONCLUSIONS: This mobile-based surgical simulation platform provides step-by-step instruction for a variety of surgical procedures. The findings of this study help to demonstrate its utility as a learning tool, as we confirmed construct, face, content, and acceptability validity for carpal tunnel release. This easy-to-use educational tool may help bring surgical education to a new-and highly mobile-level.

INTRODUCTION

The training of future surgeons is a lengthy endeavor, requiring substantial investment by medical schools and residency programs. It is further complicated by the ethical and safety considerations involved in novice surgeons operating on patients. This concern was cogently summarized in a surgical journal many decades ago-“a first experience has no place in the operating room” [1]-but we still find this to be the case for many residents. At the same time, residency programs are also now faced with mandated duty-hour limitations, restricting how much time a trainee is allowed to spend in the hospital. For surgical residents, this effectively means a prescribed and finite amount of time that can be spent in the operating room [2]. For both of these reasons, it is increasingly important to maximize preoperative learning and postoperative knowledge consolidation, as well as to provide educational tools that can be utilized outside of the hospital to allow surgical training to thrive.

Many institutions have begun to incorporate surgical simulations in an effort to expand their training curriculum beyond what is provided by random opportunity [3]. Early versions of these simulators were created to provide a user-guided 3-dimensional environment to explore the steps of an operation. They offer a type of rehearsal prior to encountering a real patient [4]. Some are quite elaborate and require a significant investment of time, space, and funding by the sponsoring department and are limited to one or few procedures. Many of these simulators are predominantly focused on teaching technical skills, with little focus on intraoperative decision-making.

In an effort to counteract some of the costs associated with simulations, many programs now use computer modules as a way to augment learning [5]. Our group, for example, created an online microsurgery curriculum that we have previously validated as a cheap and efficient way to provide complex surgical skill training to our plastic surgery residents [6]. As mobile technology becomes increasingly ubiquitous and powerful, it is now possible to create high-fidelity apps that provide true on-the-go learning for trainees of all stages. It is clear that current trainees are becoming increasingly comfortable with incorporating smartphones into their clinical practice, with as many as 55% of Accreditation Council for Graduate Medical Education trainees requesting app-based reference materials [7].

In this study, we assessed one such smartphone application, Touch Surgery app (Kinosis Ltd., London, UK). This platform may be described as a cognitive-task surgical simulation and rehearsal tool. In these simulations, operative decisions are mapped to a virtual 3-dimensional patient, allowing the users to review the steps of a surgical procedure. The user progresses through the surgical procedure by interacting with the virtual patient through every step of surgery (Fig. 1) and is required to answer multiple-choice questions. At the end of the simulation, an inbuilt assessment tool (Fig. 2) evaluates cognitive competence and maps an individual's learning curve. The aim of this study was to evaluate the construct, content, and face validity of the open carpal tunnel release surgical procedure on the Touch Surgery app.

Fig. 1

Screen shot from the simulation app

This screen shot shows a representative image of the smartphone-based surgical simulator. This phase of the simulation guides the user through the markings for an open carpal tunnel release. The trainee must utilize the touch screen to draw over the palm similarly to how they would carry out the procedure in the operating room.

Fig. 2

Built-in assessment tool

This is a screen shot from the built-in assessment engine within the simulation app. It is able to show users their most recent simulation score as well as to track their progress as they improve their surgical skills and technique.

METHODS

The platform

The Touch Surgery app offers a number of procedures that simulate common operations in a variety of surgical disciplines. This validation study utilizes the Open Carpal Tunnel Release module, which is broken up into two smaller procedures: patient preparation and operative open carpal tunnel release. Both modules have a learn function and a test function. In the learn mode, a user views an interactive step-by-step guide, while in test mode, a user completes questions as well as performs anatomically relevant swipes on the touch-screen platform to complete the procedure. In the test function, a built-in scoring algorithm evaluates the respondent's knowledge based on correct decision-making and accurate touch-screen interactions.

Participants

Participants were recruited through an open application process at a medical school in California. The novice cohort consisted medical students who had never performed or assisted in an open carpal tunnel release, while the expert cohort was defined as either attending-level surgeons or senior residents who had each completed more than 50 open carpal tunnel release procedures. Neither group had previous exposure to the Touch Surgery app.

Validation

Construct validity refers to the ability of a simulator to distinguish between experts and novices. In order to assess construct validation, participants were required to perform the test function of the open carpal tunnel release surgery module under observation. Participant scores were calculated and recorded by the app and sent to a secure server.

Content validity refers to the realism and surgical content of a simulation, and is assessed by experts. Face validation assesses the simulator as an educational tool. This requires the opinions of both senior and junior surgeons [89].

Content, face, and acceptability were assessed using a 5-point Likert scale in which participants rated the statements listed in Appendix 1 as follows.

(1) Very poor/very useless

(2) Poor/useless

(3) Acceptable

(4) Good/useful

(5) Very good/very useful

All questions and their respective domains are listed in Appendix 1.

Data analysis

Construct validation scores were assessed using the Mann-Whitney U test, as the data displayed a non-parametric distribution. P-values of <0.05 were considered to indicate statistical significance.

Questionnaire scores (content, face, and acceptability) were assessed using median scores, as these data also displayed a non-parametric distribution.

RESULTS

A total of 30 participants volunteered to participate in the study. Twenty of the participants were novices and 10 participants were experts.

For construct validation, the expert cohort scored significantly higher than the student cohort, with a median of 74% versus 45%, respectively (P<0.0001) (Fig. 3).

Fig. 3

Box plot demonstrating construct validity

Construct validity is the ability of a tool to demonstrate differences in skill levels. This box plot of the raw scores that participants received on the simulation shows a statistically significant difference between experts and novices, confirming construct validity.

On the feedback questionnaire, scores with a median of 4 or greater were reported for all statements regarding content (Table 1), face (Table 2), and acceptability (Table 3).

Table 1

Content validity

Content (answered by experts only)	Median agreement score
This simulation appears to be an accurate representation of the intended procedure	4
The anatomy in the simulation is accurate	4
The steps present are ordered appropriately	4
The surgical instruments are represented accurately	4
The graphics contained in this simulation are realistic	4

Table 2

Face validity

Face validation statement	Expert (median)	Novice (median)
The simulation could be used to teach the intended procedure	5	4
This simulation tool may improve patient safety	4	4
This simulation would be a useful for junior trainees	5	4
This simulation would be a useful for senior trainees	4	N/A

N/A, not available.

Table 3

Acceptability validation

Acceptability statements	Experts	Novices
Trainees should be encouraged to use this simulation and rehearsal tool during their free time	5	4
Trainees should be encouraged to use Touch Surgery for preoperative rehearsal	5	4
Touch Surgery should be incorporated into the surgical curriculum for simulation and operative rehearsal	4	4

DISCUSSION

The results of this study demonstrated that the Touch Surgery app has construct, face, and content validity for carpal tunnel release surgery. Construct validation corresponds to a statistically significant difference between novices and experts, which provides an evidence base for the scoring algorithm of the app. We believe that differences in techniques and/or the use of alternative instruments ultimately led to variability in the experts' score and the median expert score of 74%, as opposed to a gap in the knowledge of the experts.

Face and content validation jointly signify that a simulation teaches what it intends to teach and resembles the real world.

The plastic surgery milestone project assessment tool by the Accreditation Council for Graduate Medical Education and the American Board of Plastic Surgery lists surgical skills evaluation as one of the 6 components in accreditation. Items within this evaluation rubric pay particular attention to surgical marking, positioning, preparation, draping, surgical steps and flow of the operation [10], all of which are objectively assessed within the app. This is not to be confused with the clinical competence to perform the procedure, but could help trainees reach clinical milestones faster. Technical skills training still needs to be covered in the surgical curriculum, as well as observing a varied mix of cases and presentations.

Tremendous interest has emerged in simulations as a means to enhance medical education for a variety of reasons, including duty-hour restrictions and ethical concerns over the use of patients for training [11]. The Touch Surgery app provides residency programs with the ability to have residents simulate and safely rehearse surgical procedures outside of duty-hour restrictions through a simulation tool that is available on smartphones.

The use of cognitive task analysis in surgical training has been shown to accelerate skill acquisition and to improve performance in procedures such as central venous catheterization and laparoscopic cholecystectomy [1213]. It has also been demonstrated that in some cases, cognitive task analysis may be as effective or more effective than on-the-job training [14].

This app can also provide residency programs with objective data that could be used to measure a resident's cognitive knowledge of a procedure, meaning that institutions that adopt this app will be early movers into a competency-based curriculum. Moving towards curriculum integration is necessary to the ensure effective incorporation of simulations within a residency program, as the voluntary use of simulation tools has been shown to lead to minimal participation [1516]. A consensus also exists among the majority of junior residents that simulation should be made a mandatory component of training [1617].

We strongly believe that apps have the potential to be valuable additions to surgical learning for trainees of all levels as part of a training curriculum. The beauty of low-cost and easily delivered app-based content derives from the unfettered access to such content.

This study evaluated participants simulating a single procedure at one institution. Future work will expand to include multiple centers and multiple procedures. We aim to recruit an intermediate group and assess their scores. More importantly, as the validity of Touch Surgery app has been demonstrated, we can now begin to deploy the platform within the curriculum and observe the learning curves of residents and the transfer-ability of skills. At present, the platform only includes a single approach to carpal tunnel release, using our open technique. Further work needs to be done to demonstrate other open and minimally invasive techniques. It may also be feasible to use other instruments to perform the same objectives, and this should be incorporated into the simulations.

Touch Surgery app was demonstrated to have construct validity, with statistically significant differences in results observed between experts and novices. It has also demonstrated face and content validity based on ratings by our expert groups. This application has the potential to improve access to surgical education around the world.