Test-enhanced learning improves learner attendance during a laparoscopic box trainer simulation program.

BACKGROUND: Laparoscopy is the gold standard approach for many surgical procedures, but it is a complex skill to learn. Laparoscopic simulation training may help, but it is unclear how to best engage trainees in these programs. Test-enhanced learning (TEL) uses regular, well-defined assessments of performance throughout the training phase of learning. AIM: The aim of this study was to assess the effects of TEL on a laparoscopic simulation program involving a cohort of medical student volunteers.
MATERIALS AND METHODS: A prospective cohort study was performed with a convenience sample of 40 medical students. Students were recruited to participate in a ten-week laparoscopic simulation program. Twenty students participated in a laparoscopic surgical program with TEL ('TEL group'), and 20 students participated in a standard laparoscopic simulation program ('control group').
RESULTS: Attendance in the TEL group was significantly higher than in the standard group (71 vs 51.5%, P = 0.03). There was no difference between groups in mean time scores. Four themes were identified in qualitative data drawn from student surveys - personal traits and motivators, training context, clear goals and feedback enabling understanding of one's own performance.
CONCLUSION: Testing laparoscopic skills throughout a learning program, in conjunction with individualised feedback and tracking of learning trajectory, increases trainee attendance. Laparoscopic simulation training programs are encouraged to reflect on the pedagogic framework in which their procedural skills training operates.

INTRODUCTION

There is evidence that laparoscopic simulation training is helpful for surgical skill development. , , , , Trainees who utilise laparoscopic simulators have been reported to have a significant reduction in operative time compared to trainees without access to simulation programs. However, these outcomes are not uniform. Other trainee cohorts have failed to achieve either their training or practice goals through lack of engagement, and this risks poor return on investment from expensive training equipment. Simply having access to simulators is not sufficient to guarantee procedural performance improvement or motivation to complete simulation training. , Finding the optimal pedagogical framework in which to conduct training is important for laparoscopic simulation programs, but evidence‐based guidance is lacking.

BACKGROUND

Laparoscopy is the gold standard approach for many surgical procedures, but it is a complex skill to learn. Traditionally surgical trainees learned how to operate through apprenticeship with more senior surgeons, but changes in healthcare service delivery have made this approach challenging. High numbers of trainees in training programs, reduced working hours, increased number and complexity of operative procedures and reduced theatre exposure have resulted in trainees reporting decreased confidence in performing procedures at the end of speciality training.

Previous attempts to incentivise and increase surgical trainee utilisation of laparoscopic simulators have had mixed results. During the Incentivised Laparoscopy Practice Study (ILPS) by Nicol et al., surgical trainees were given a take‐home portable laparoscopic simulator, online modules, metrics with personalised feedback and eCertificates to help assist primary operating opportunities. Despite these incentivised efforts, the study found that there was generally poor engagement in the program. There are similar findings in other take‐home laparoscopic box trainer projects, with up to half of the recruited trainees not completing the laparoscopic curricula despite access to take‐home trainers. , Thinggaard et al. found trainees completing their take‐home laparoscopic program practiced heavily at the initiation of the program and at the end, before final assessments, with very limited practice logged by trainees in the intermediate period.

Following ILPS, Blackhall et al. performed a follow‐up qualitative study to further understand the barriers to engagement with laparoscopic simulation. They identified that competing commitments of surgical trainees was a major barrier to engagement with home‐based simulation. Trainees reported preferring to focus on ‘point scoring’ for career progression and struggled to see the value in some of the laparoscopic tasks. They were also unsatisfied with automated metric feedback and wanted individual personal feedback from surgical supervisors. The study concluded that scheduled simulation sessions which provide trainees with the opportunity for direct surgical specialist feedback may improve engagement.

With ‘best practice’ so far elusive, other pedagogic strategies need consideration to enhance trainee engagement. Test‐enhanced learning (TEL) has been demonstrated to be an effective strategy for knowledge development across a range of educational contexts. TEL uses regular, well‐defined assessments of performance throughout the training phase of learning. Testing is considered to support learning by encouraging retrieval of information or skills and foster effortful and deliberate practice. , Regular testing also provides the opportunity for direct and specific feedback from a supervisor with clearly benchmarked goals. , There is currently limited experience with the use of TEL in complex procedural work.

Our three study questions were the following:

Does the addition of TEL accelerate technical performance in a cohort of novice learners?

Does TEL increase learner engagement in a laparoscopic surgical program?

How do learners perceive the TEL experience in a laparoscopic simulation program?

MATERIALS AND METHODS

A prospective cohort study comparing medical student learner groups undertaking laparoscopic training with and without TEL was performed at a Queensland University, within the Faculty of Health Sciences and Medicine. This project was reviewed by the Ethics Committee at Bond University, and an ethical waiver was granted (project BL02596). Written consent from all participants was obtained.

Recruitment

All medical students from a single medical school were invited to participate in this study. Flyers and university social media were used to identify students interested in participating in the research project. The medical program currently has not included formal teaching of laparoscopic skills in the undergraduate degree.

Forty students were recruited and randomly allocated to two equal groups. Twenty students were allocated to a control group (‘control group’) performing a standard laparoscopic simulation program, and the remaining 20 students in the intervention group performed a laparoscopic simulation program with the addition of TEL (‘TEL group’).

Study procedure

All students completed laparoscopic exercises on standard box trainers over a ten‐week program. Students were asked to attend weekly face‐to‐face sessions with control and intervention groups. All students were provided a peer pair allocation for the duration of the program within their group. The sessions lasted 1 h, were performed on a weekend remote from student lectures and were facilitated by two to four surgical supervisors from general surgical and gynaecology clinical backgrounds. All students were provided with a laparoscopic program with ten exercises to complete; videos of the exercises to perform; and timed targets to achieve, with competent, advanced and elite levels identified (Appendix 1.3). Students were encouraged to obtain a competent time with mastery of tasks before moving onto the next exercise.

Instruction from surgical supervisors was provided to all students at all ten face‐to‐face weekly sessions. Feedback was verbal and tailored to each individual student. Supervisors were the same for both the TEL and control groups. The TEL group was formally tested and timed at the conclusion of each weekly session whether they felt they had achieved mastery of task, and the timed attempts were video recorded. Testing occurred during the allocated session time. Feedback to the TEL group students utilised the timed recorded videos which allowed specific feedback of technique performance. The control group was tested only at the beginning and end of the ten‐week program.

Data collection

All students were timed on the thread transfer exercise at the beginning of the program and on four exercises at the end: thread transfer, paperclip untangle, glove capping and dice stack (see Table 1). We initially aimed for baseline assessment of multiple skills; however, students did not have sufficient dexterity and skill to time record the more complex exercises at the beginning of the study. In addition, the students in the intervention TEL cohort were formally ‘tested’ every week on the exercise they were allocated to complete.

Table 1

Outline of the four exercises timed at the end of the laparoscopic simulation program

	Times to achieve (s)
	Competent	Advanced	Elite
Exercises to complete
Thread transfer	60	45	25
Dice stacking	90	60	32
Paper clip untangle	90	60	35
Glove tip capping	120	100	74

Data collection – other variables influencing trainee performance

Students completed a questionnaire at the beginning of the program, which included demographical details and a history of personal traits known to relate to hand–eye coordination, dexterity and improved baseline surgical ability identified in previous studies , , , (Appendix 1.1).

Data collection – learner experience

All students completing the final week timings were invited to complete an end‐of‐project questionnaire, addressing acceptability of the proposed curricula and feasibility of the program (Appendix 1.2). The survey was developed by the study authors as we found no suitable psychometrically tested instruments.

Data analysis

Linear regression analyses were performed to test for significant differences in task completion times and attendance between TEL and control groups. χ2 Tests were used to determine whether the TEL group was statistically different from the control group for characteristics or skills that could plausibly be associated with completion times. A P‐value of 0.05 was considered significant.

Survey data analysis

Three researchers (authors B.L., J.N. and B.V.) analysed the qualitative end‐of‐survey responses after completion of the study. We used an inductive approach, following the six‐step process outlined by Braun and Clarke. The material was independently reviewed by each researcher, who familiarised themselves with the data in Microsoft Word document form, identified key phrases and generated initial codes. The researchers then met and discussed candidate subthemes and themes identified in the data. Consensus was reached by iterative testing of the fit between codes and draft themes.

RESULTS

Descriptive statistics

Forty medical students participated in the program, with loss to follow‐up due to attendance in the last session resulting in 23 individuals with complete data. Baseline demographical data were collected for both TEL and control groups. There was no statistically significant difference in group characteristics and pre‐existing skills that could be plausibly associated with completion times (see Table 2). Small group numbers may have limited the interpretation of these variables between groups. Attendance in the TEL group was 143 out of 200 possible student attendances over the ten‐week program, equating to 71% overall attendance. The control group had 103 attendances, equating to 51.5% attendance.

Table 2

Baseline characteristics of TEL and control groups

	TEL group N (%)	Control group N (%)	P‐values
Sex
Female (%)	9 (60%)	5 (62%)	0.94
Male (%)	6 (40%)	3 (38%)
Age
<18 (%)	0 (0%)	1 (12.5)	0.17
18–24 (%)	14 (93%)	5 (62.5%)
25–34 (%)	1 (7%)	2 (25%)
Handedness
Right (%)	13 (86%)	8 (100%)	0.53
Left (%)	1 (7%)	0 (0%)
Ambidextrous (%)	1 (7%)	0 (0%)
Assisted in laparoscopic surgery
Yes (%)	5 (33%)	1 (13%)	0.39
No (%)	10 (67%)	7 (87%)
Interested in surgery
Yes (%)	13 (85%)	7 (88%)	0.91
No (%)	2 (13%)	1 (12%)
Woodwork
Yes (%)	3 (20%)	1 (13%)	0.60
No (%)	12 (80%)	7 (87%)
Plays musical instrument
Yes (%)	8 (53%)	3 (38%)	0.57
No (%)	7 (47%)	5 (62%)
Uses chopsticks
Yes (%)	7 (46%)	5 (63%)	0.38
No (%)	8 (53%)	3 (37%)
Plays competitive sport
Yes (%)	8 (53%)	4 (50%)	1.00
No (%)	7 (47%)	4 (50%)
Used surgical simulator
Yes (%)	3 (20%)	1 (13%)	0.60
No (%)	12 (80%)	7 (87%)
Plays video games
Yes (%)	9 (60%)	4 (50%)	0.75
No (%)	6 (40%)	4 (50%)

TEL, test‐enhanced learning.

Attendance

In a linear regression model attendance was significantly higher in the TEL group, P = 0.03. On average the TEL group attended two more sessions than the control group participants. Attendance in both groups declined with progression of the ten‐week program; however, this was most marked in the control group (see Fig. 1).

Figure 1

Attendance across the ten‐week program.

Task completion times

All students who attended in the final week were timed on exercises, including thread transfer, dice stack, paperclip untangle and glove capping. There was no difference between groups for mean time scores (see Table 3). Thread transfer was the only exercise tested at both the beginning and the end of the project. Both intervention and control groups had a significant improvement in mean times for thread transfer from the beginning to the end of the project, but there was no statistically significant difference between the groups’ mean time scores at the end of the project. Attendance was not a statistically significant predictor of task completion times.

Table 3

Performance of TEL and control groups

	TEL group	Control	P‐value
Participant number N	20	20
Attendance
Mean	8.5	7.8	0.03*
Standard deviation
Range	6–10	6–10
Participant number end of study N (%)	15 (75%)	8 (40%)
Thread transfer time (s)
Mean	42.8	47	0.44
Standard deviation	12.1	14
Range	29–74	25–67
Dice stacking time (s)
Mean	39.1	49.6	0.32
Standard deviation	19.9	28.3
Range	9–88	10–98
Paperclip untangling time (s)
Mean	45.1	73.8	0.08
Standard deviation	17.3	56.5
Range	17–90	29–198
Glove capping time (s)
Mean	116.4	160.8	0.17
Standard deviation	50.3	94.4
Range	38–221	37–354

TEL, test‐enhanced learning.

Qualitative survey results

Nineteen survey responses were received — 13 responses were from the TEL group (13 of 15 participants who completed the study completed the survey 87%), and six responses were from the control group (six of nine participants who completed the study completed the survey 67%). Four themes were identified: personal traits and motivation, training context, clear goals in the training process and feedback enabling understanding of one's own performance.

Personal traits and motivation

Students listed motivators which they felt were important for progress in this program. Personal traits such as self‐confidence, determination and competitiveness were frequently described.

‘Willingness to give a go, no matter how hard it was’(TEL group).

‘Previous experience with things that involve hand–eye coordination and fine movement dexterity, eg, sports (tennis, squash), gaming (PC, console), music (piano)’ (Control group).

‘Competitive traits both affected me positively and negatively. Being competitive made me try harder and attempt to be fast, however, I was less accurate when I was trying too hard’ (Control group).

Training context

Students listed factors relating to training context which they felt were associated with their performance including external pressures and stressors which tended to affect concentration and the ability to remain patient during tasks.

‘How tired I was feeling. How stressed I was. How easy the task is’ (TEL group).‘Outside stress made it difficult to concentrate at times’ (TEL group).

‘Exams and rotations so had to miss some weeks’ (TEL group).

Clear goals in the training process

The impact and importance of clear goals were highlighted by many students. Testing was considered to help with clear identification of goals as they progressed. Students also reflected on the impact of unclear goals impeding performance.

‘I found testing helpful because timing gave an end goal for each session and something to strive for’ (TEL group).

‘Having to be competent on one skill before progressing to the next allowed me to focus my energy and does gradually improve my overall skills’ (TEL group).

[with no clear goal] ‘I switch exercises regularly so never get good at any’ (Control group).

Feedback enabling understanding of one's own performance

Many students commented on different techniques and methods they used to better their own performance. They also found testing useful to help determine whether their performance was progressing. Students also commented on feelings of reward in seeing their own progress develop.

‘Through trying different methods to achieve the tasks I found the "ideal" method that worked for me’ (TEL group).

‘The testing was helpful – I performed better under pressure and gave a baseline of how I was progressing. It also gave something to strive towards and acted as a hurdle before moving to the next task’ (TEL group).

DISCUSSION

We found that the addition of TEL leads to a significant increase in learner attendance in a laparoscopic simulation program, but there was no change in the speed of performance of timed tasks. Students reported benefits of testing during procedural learning, including clear goal setting, increased motivation, self‐reflection and understanding of their own performance.

Testing is well known to have a positive effect on learning and skill development. TEL may compliment incentive strategies already trialled in laparoscopic simulation programs. Testing appears to encourage deliberate practice with clear objectives and goals. Feedback given during the testing process is often more specific, and quantitative measures or ‘scores’ of performance can be tracked for individual progress. Testing allows student performance to easily be benchmarked against ‘best’ or peer performance, which many go on to further motivate and increase engagement in a training program.

TEL is an important element of mastery learning, an approach that has also been well described in simulation‐based medical education. Originally proposed by Benjamin Bloom, the Healthcare Simulation Dictionary describes mastery learning as follows: ‘A student must first practice and study to meet the predetermine level criteria (>90%) through the formative assessment of a prerequisite domain prior to advancing in subject matter. If the learner does not achieve the level of mastery, information from the test is used to diagnose areas of deficiency necessary for additional prescriptive support and the student is later tested again. This cycle of feedback and corrective procedures is repeated until mastery is achieved at which point the student will move on to the next level’. A key feature of mastery learning is allowing the student to progress following individualised learning curves. Testing is the vehicle to confirm (or not) ‘mastery’. TEL does not require the predetermined criteria for a ‘pass’ but rather focuses on learning at every point along the learning curve.

Cook et al. performed a systematic review and meta‐analysis on mastery learning for health professionals using technology‐enhanced simulation and concluded that mastery learning was associated with higher‐learning outcomes. This effect was particularly large and statistically significant for procedural skills. However, mastery learning took longer than non‐mastery learning, and there were increased logistical difficulties on teachers and trainees during the mastery‐learning process. We suggest these disadvantages will also apply to our TEL approach. Further questions raised in the mastery‐learning systematic review included variations in how competency is defined, how it is assessed and how practice phase is implemented and the regulation of progression to more advanced skills.

There are unanswered questions in our TEL approach for laparoscopic surgery simulation. Does testing need to be frequent and low stakes or infrequent and high stakes? Should the tests occur at points identified by the learner or at the request of the supervisor? Testing often has negative associations for learners – as an evaluative ‘threat’. Testing to aid learning works to shift the paradigm from assessment of learning to assessment for learning (Martinex & Lipson 1989). , Frequent low‐stakes testing helps de‐emphasise the high‐risk apprehension some learners describe related to summative assessments. Several facilitation strategies may help, including allowing learners to test when they felt ready to do so, testing away from peers, asking learners to video record self‐times and allowing repeat testing to further lower the stakes of the test.

Surprisingly, TEL in our cohort appeared to motivate learners to strive for performance beyond proficiency of task. Students were given ‘competent, advanced and elite’ times to aim for as per Table 1. The end TEL group mean times for thread transfer, dice stacking and paperclip exercises were all within the ‘advanced’ times. TEL students were aware of whether their own times were competent, advanced or elite as the study progressed due to the regular testing and benchmarking of performance. Several TEL students achieved ‘elite’ times for exercises on week‐to‐week timings. Despite clearly defining competency, many TEL students continued to practice and test beyond the competency goal. It is difficult to discern whether this effect was due to the process of testing or the very specific feedback and self‐reflection involved in watching videos of their own tested performance. Is ‘mastery’ of a task our goal, or is it fostering a lifelong striving for improvement? Both may be attainable.

Interestingly in our study, the students in the control group who did not have testing reported negative features of unstructured practice, including ‘lack of concentration’ and ‘regularly switching of tasks so I never get good at any’. Observing this cohort of students, they appeared to ‘play’ with the equipment as opposed to deliberately practice tasks. It is possible that some of the problems encountered in engagement with other laparoscopic programs may be associated with trainees performing unstructured practice.

The themes we identified in our qualitative analysis included personal traits and motivation, training context, clear goals in the training process and feedback enabling understanding of one's own performance. It is not surprising that students reported the testing component to be particularly helpful in clear goal setting and assessment of whether their skills and performance were improving. The influence of personality traits on medical students’ learning behaviour has been well explored. A new question raised by our findings is whether participating in TEL approaches conversely shapes those traits throughout training.

There are several limitations to our study. Similar to other laparoscopic simulation programs, both the TEL and control groups had a reduction in student attendance in the mid portion of this project. The highest engagement levels were at the beginning and the end of the program, and this effect was most marked in the control group. This dropout rate is similar to other reported laparoscopic simulation studies. , , This highlights that there appear to be barriers to engagement in laparoscopic simulation programs requiring further research in future.

This project was conducted with medical students as opposed to trainees. Working with a medical student cohort allowed us to achieve a larger participant number who uniformly had no prior laparoscopic experience. This is more challenging in the trainee cohort where numbers are smaller, and there is variability in surgical skill set across the trainee cohort. The observed effect of testing in the medical student cohort may be different from that in the trainee cohort. Our numbers were small, affecting our ability to study the impact of other (non‐training‐related) predictor variables on performance. Many laparoscopic simulation research programs have had similar small cohort numbers due to the difficulty in managing and recruiting a larger trainee cohort. , , , , ,

Despite these limitations, our study has found that testing appeared to be a powerful tool to encourage deliberate practice and increase attendance and engagement in a laparoscopic simulation program. These findings may be helpful to other institutions considering curriculum design and training strategies for laparoscopic simulation programs. We raise new questions in defining mastery learning for laparoscopic skills, including the timing, nature and role of testing in mastery of laparoscopic procedures.

Testing of laparoscopic skills during the phase of learning in conjunction with individualised feedback and tracking of learning trajectory may increase trainee engagement with laparoscopic simulation programs. Laparoscopic simulation training programs are encouraged to reflect on the pedagogic framework in which their procedural skills training operates.

FUNDING INFORMATION

A Faculty of Health Sciences and Medicine Bond University grant of $3500 was used to purchase two EoSIM laparoscopic simulation trainees to assist in the completion of this project. There was no additional funding outside of usual employment arrangements for the authors.

AUTHOR CONTRIBUTION

B.L. designed the manuscript concept and contributed to data collection and writing and revising the manuscript. J.N. and S.J. contributed to data collection, initial manuscript drafts and revisions. D.W. provided statistical interpretation and support. D.A. contributed to revising the manuscript. V.B. provided overall project guidance and support and also contributed to writing and revising the manuscript.

Appendix S1. Supporting Information.

Click here for additional data file.