Face, content, construct validity and training effect of touch surgery™ as a surgical decision-making trainer for novices in open appendicectomy.

BACKGROUND: Laparoscopic surgery has become the gold standard for many operations with significant benefits in morbidity and hospital recovery time. One such procedure is appendicectomy, which is overwhelmingly performed using the laparoscopic approach in the modern era. This has also meant that the number of cases involving traditional open appendicectomy has declined despite surgeons being expected to be able to convert to the open technique if required. One method to rehearse for theatre is the use of software applications. This paper investigates the validity of Touch Surgery™ as an education tool for surgical decision-making for novices, as well as its training effect in open appendicectomy.
METHOD: 70 participants will be recruited, consisting of 60 medical students (novices) and 10 surgical consultants (experts). For face, content, and construct validity, first attempt scores on the Touch Surgery™ Open Appendicectomy Test Module will be compared between novices and experts. For the training effect and knowledge decline elements of the study, novices will be further randomised into either the low intervention (control) group who will complete the simulation once, or to the high intervention group who will complete the simulation six times, with both novice groups asked to repeat the test one week later. All participants will also be requested to complete questionnaires regarding the stimulation.

Introduction

Background

With the advent of laparoscopic technology, the number of open operations such as open appendicectomy has gradually declined in recent decades [1], [2], [3], [4]. The inclusion of modern technology in surgical training has increased and represents an important component of modern surgical education [5], [6]. The use of software to assist in surgical teaching is widely documented in the literature. A large systematic review by Maertens et al. concluded that e-learning is as effective as other platforms for learning in surgery [7]. Some of these modern teaching methods include digital simulations, which have been shown to be effective at improving surgical performance [8].

Simulation training is one method of developing proficiency in a safe and risk-free environment, and is used in formal surgical training [9], [10], [11]. Touch Surgery™ is a free surgery simulator on iOS and Android developed primarily in the United Kingdom with contributors from surgeons around the world, and contains a simulation on open appendicectomy.

There is 1 study validating Touch Surgery™ as a surgical decision-making tool [12]. There is 1 study protocol on intercostal catheter insertion involving Touch Surgery™ [13], 1 study evaluating its use in training for hand surgical procedures [14], 2 studies on the application of Touch Surgery™ in training for laparoscopic cholecystectomy [15], [16], 1 study assessing its use in intramedullary femoral nailing [17], and 1 review article discussing the use of the software in breast surgery [18]. A relatively recent randomized trial also demonstrated the underlying potential of Touch Surgery™ to be incorporated into the surgical curriculum to improve technical skills among surgical trainees in low- and middle-income countries [19]. Touch Surgery™ has been proposed as an innovative simulation-based platform covering numerous simulations from many different specialties, which helps meet the demands of modern professional surgical training [20].

However, a thorough literature search in PubMed and EMBASE has not yielded any studies evaluating the efficacy of digital simulations in the cognitive performance of surgeons, surgical trainees or medical students in open appendicectomy. In the broader realm of open surgery, very few studies have focused on the use of simulation in the training of open surgical skills [21]. Within these few studies, Davies et al. highlighted the need for higher quality research [22]. This research project will investigate the use of Touch Surgery™ in less commonly performed procedures (i.e. open appendicectomy) and quantify knowledge decay, thereby assessing the need for repeated rehearsal. In addition, this project would be a key article that further validates or refutes against Touch Surgery™ as a valid educational tool.

Rationale

Simulation training has been shown to decrease the time to proficiency for novices and offer the chance to develop surgical skills in a risk-free environment [11], [23]. Virtual surgical application tools such as Touch Surgery™ create a platform that enables users to simulate and become exposed to different surgical procedures, with the potential to be used in the education of novices in these procedures. In addition, the evidence for the impact of simulation training on patient care is suggested to be positive, although unequivocal data is lacking and benefits may be confined to novices [8], [24], [25]. In fact, research from Belgium suggests that simulation training even during preclinical years has a significant positive effect on surgical performance during the first postgraduate surgical year [26].

Many simulators are costly and not readily accessible to novices. A cheap alternative is the plethora of medical apps to iOS, Android, and other platforms. Thus, if Touch Surgery™ can be used as a valid tool for educational purposes, the medical community would benefit from a free and widely accessible mobile device application to aid in surgical rehearsal.

Aim

The efficacy of digital simulations in improving the cognitive performance of surgeons, surgical trainees or medical students in open appendicectomy has not been previously studied. The primary aim is to determine if Touch Surgery™ is a valid educational training tool for open appendicectomy by assessing its face, content and construct validity. Furthermore, this project has an added component of knowledge decay, which has also not been previously studied. The secondary objectives include assessing the effectiveness of Touch Surgery™ as a training tool; whether repeated exposure to the simulation has an effect on knowledge decay after one week.

Method

Study type

This study is based on Sugand's studies on Intramedullary Femoral Nail [12]. Medical students, fellows, and consultant surgeons will be recruited to determine construct validity of Touch Surgery™. This determines whether the Touch Surgery™ Open Appendicectomy Test Module can differentiate between an inexperienced novice vs an experienced surgeon.

This study can be adapted for either multicentre or single institution recruitment.

Hypothesis

The hypothesis is that Touch Surgery™ would be a valid education tool as demonstrated on face, content, and construct validity, as well as effective through assessment of training effect for Open Appendicectomy. This is based on previous studies by Sugand et al. on the Touch Surgery™ Intramedullary Femoral Nail modules [12]. Based on recent systematic reviews on simulations and surgical outcomes for common procedures [27], [28], [29], it is reasonable to assume that if powered correctly, novices may demonstrate improved cognitive performance in open appendicectomy after brief interaction with the surgical simulation software.

It is also hypothesised that students will exhibit a knowledge decay effect after not using the app for one week, but those who underwent the high intensity intervention will have a higher baseline score than those who underwent the low intensity intervention.

Study population

Three different groups of participants are required, these are:

(1) Control group (Novices):

• Novices include medical students and non-board certified doctors.

• Will be administered test module once, and given the Likert 5-point scale questionnaire immediately after the test. The test will be repeated 1 week later.

• Participants will not have access to Touch Surgery™ in the interim.

(2) Intervention group (Novices):

• Novices including medical students and non-board certified doctors.

• Will be administered test module 6 times repeatedly, and given the Likert 5-point scale questionnaire immediately after the test. The test will be repeated 1 week later with administration of 6 repeated tests, to determine training effect and knowledge decay 1 week apart. 1 week was chosen for ease of follow-up and minimization of loss to follow-up. Participants are asked to repeat the test 6 times at 1 week to determine any differences in training effect (i.e. test score (%), time to complete module (seconds), attempts required to reach plateau effect (n)) compared to 1 week prior.

• Participants will not have access to Touch Surgery™ in the interim.

(3) Gold Standard group (Expert):

• Will comprise of consultant surgeons.

• Will be administered test module once only.

Inclusion criteria

Both cohorts will complete a pre-study questionnaire to ensure the inclusion criteria are met.

• Novice group: Have never observed or performed an open appendicectomy.

• Expert group: Are able to perform open appendicectomy independently.

Exclusion criteria

• Previous exposure to Touch Surgery™.

Participants recruitment

Sugand et al. 2015 showed statistical significance using 10 experts and 39 novices (p = 0.001) for the Touch Surgery™ Intramedullary Femoral Nail module. This study replicates Sugand et al.’s research to test construct validity of Touch Surgery™ in the Open Appendicectomy Test Module. A sample size calculation is not necessary as per consultation with Graduate Research School HDR StatsHelp Advisor at James Cook University. However, we plan to enrol 10 experts and 60 novices to better power our study to look for a difference. To achieve this, the following cohorts will be recruited.

COHORT 1: Novice group

Medical students naïve to open appendicectomy represent the novice group. This group will be recruited by word of mouth and consented to the study. The time involved for novices could range from 15 minutes to 1 hour, for 2 appointments one week apart. They will complete the following tasks:

Filling in screening questionnaire to ensure they meet the inclusion criteria for novices and whether any exclusion criteria apply. This standardises exposure to Touch Surgery™.

Randomisation to either control or intervention. This enables measurement of the training effect & knowledge decay element of the study. Participants in the low-intensity intervention will need to complete the test module once. Those in high-intensity will complete the test module 6 times (to assess training effect).

Brief hands-on introduction to Touch Surgery™ using the Mastectomy Part I module. This module is unrelated to open appendicectomy, and will be used to familiarise participants with selection of surgical decisions and swipe interactions.

Timed test module of open appendicectomy app, while being blinded to their final score (0–100%). This will serve as the novice score to determine construct validity.

Filling in post-module questionnaire for face validity only (based on a rating scale ranging from 1-5).

Participants randomised to control will return in one week, repeating the test module once to assess knowledge decay.

Participants randomised to intervention will return in one week to repeat the test module 6 times. This assesses knowledge decay. 1 week was chosen for ease of follow-up and minimization of loss to follow-up. Participants are asked to repeat the test 6 times at 1 week to determine any differences in training effect (i.e. test score (%), time to complete module (seconds), attempts required to reach plateau effect (n)) compared to 1 week prior.

COHORT 2: Expert group

The expert group will consist of consultant general surgeons or fellows who act as the gold standard as they have been certified to be able to independently perform open appendicectomy on patients. They would be recruited by word of mouth or email. Their time involved will be 15 minutes. They will complete the following tasks:

Filling in screening questionnaire to ensure they meet the inclusion criteria for experts, or whether the exclusion criteria apply. This standardises exposure to Touch Surgery™.

Brief hands-on introduction to Touch Surgery™ using the Mastectomy Part I module. This module is unrelated to open appendicectomy, and will be used to familiarise participants with selection of surgical decisions and swipe interactions.

Timed test module of open appendicectomy app, while being blinded to their final score (0–100%). This will serve as the expert score to determine construct validity.

Filling in post-module questionnaire for face & content validity (based on a rating scale ranging from 1-5).

Study design

Each participant will be blinded to the randomisation event and test score. They will be given 5 attempts to answer a question, before being guided to the next step by the exam administrator. Statistical analysis will be done using the student’s 2-tailed t-test for difference. Intervention includes:

All participants in all groups will perform the open appendicectomy test module.

Control group novices will perform the test once, then again 1 week later.

Intervention group novices will perform the test 6 times consecutively, then again 1 week later.

Expert group will perform the test once only as a gold standard.

Questionnaires

Pre-Study questionnaire

Demographic and inclusion/exclusion criteria questions on the questionnaire are as follows:

Name

Sex

Age

Contact number

Contact email

Level of medical training (medical student, fellow, consultant surgeon)

Number of years as doctor

Number of years as surgical consultant

Used Touch Surgery™ in the past (yes/no)

Observed/participated in an open appendicectomy before (yes/no)

Number of open appendicectomies assisted

Number of open appendicectomies performed

Post-Study questionnaire

A post-study questionnaire for face and content validity asks the participants to rate the app using a five-point Likert rating scale:

Is the Touch Surgery™ open appendicectomy module a useful training and assessment tool?

Is the Touch Surgery™ open appendicectomy module accurate, and does the flow of steps represent the open appendicectomy procedure in practice?

Would you use the app?

Is this app useful for surgical training?

Is this app useful for preoperative rehearsal?

Would you use this app to learn more procedures?

Would you want this app as part of the surgical training curriculum?

Do you feel this application is a valuable teaching tool for surgery?

Do you feel this application would allow novices to gain more confidence as the primary operator?

Do you feel this application would significantly decrease the learning curve for novices to perform operations?

Do you feel novices would understand the surgery better using Touch Surgery™ than traditional low-cost methods such as textbooks and videos?

Do you feel applications such as Touch Surgery™ should be formally incorporated into medical evaluation?

Additional comments:

Statistical analysis plan

A rigorous step-by-step process will be used to ensure a standardised experience for participants. After obtaining consent:

A paper screening questionnaire will be completed by participants to ensure they meet inclusion criteria and are not excluded from the study.

A paper post-module questionnaire will be completed by participants to assess face and content validity (if applicable).

Time taken to perform the task and achieve a test score will be recorded with a stopwatch to the nearest second.

All participants are assigned a unique alphanumeric identifier to maintain anonymity. De-identified data will be transferred to a secure electronic database, accessible only by researchers on google sheets.

All objective data will be recorded as median with Bonett-Price 95% confidence intervals. Data will be confirmed as nonparametric by reviewing distribution. Analysis will be performed using the Mann–Whitney U test for independent data, the Wilcoxon signed-rank test will be used for paired data. A result will be deemed significant when a two-tailed p-value is less than 0.05.

Risk and benefit to participants

This study will support or refute Touch Surgery™ as a valid cognitive assessment tool in surgical decision making. Future medical students would be able to determine whether they should use this app as a surgical training tool.

Ethics of study

Ethics Approval (HREC/17/QTHS/181) was given by the Townsville Hospital and Health Service Human Research Ethics Committee (HREC).

Information protection

Confidentiality

Identifiable data from the screening questionnaire will be de-identified. Identifiable data will only be used by the researchers to follow-up with participants (when assessing the knowledge decay element of the study) and will not be published in any way. It will be de-identified upon transfer to electronic format by assignment of a unique alphanumeric identifier. De-identified data will be used for statistical analysis to determine face, content, construct validity, and training effect. Confidentiality and anonymity are guaranteed for the participants.

Data storage and security

Data formats:

Paper copies of questionnaires will be locked in personal cabinets/storage containers

Electronic data will be locked by Google docs and only accessible by assigned collaborators: Dr. Casper Pretorius, Dr. Chi Lap Nicholas Tsang, Dr. Jerry Cao, Dr. Kapil Sugand, Ms. Jacqui Chiu. Electronic identifiers will be removed when data sets are complete and ready for formal statistical analysis.

Length of data storage

Paper copy data with identifiable information will be destroyed upon publication of the project in a peer-reviewed journal.

De-identified information will be stored for 5 years from the date of publication of results as per the Joint NHMRC/AVCC Statement on Guidelines on Research Practice (1997) Section 2.3.

Data disposal

All papers with identifiable information will be destroyed upon completion of the project, which is upon publication in a peer-reviewed journal. The same process used to handle patient sensitive information as per Queensland Health policies (likely 2-axis shredding or pulping) will be used. De-identified electronic information will be retained for 5 years after which it will be permanently deleted from the researcher's computer(s) via purging or overwriting.

Authors’ contributions

All authors have made substantial contributions to design of the study, drafting of the article, revisions, and approval of the final version for submission.

Informed consent

This research will involve informed consent from participants. Participants will be briefed and consented for participation. Participation is voluntary and will have no effect on the clinical teaching or grade medical students receive as a consequence of consenting/declining/withdrawing from this study.

Funding

This research was not funded by any specific grant.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.