Distance learning improves attainment of professional milestones in the early years of surgical training.

OBJECTIVES: To assess the impact of a surgical sciences e-learning programme in supporting the academic development of surgical trainees during their preparation for professional examination.
BACKGROUND: In 2007, a 3-year online part-time Master of Surgical Sciences (MSc) degree programme was launched, utilizing an innovative platform with virtual case scenarios based on common surgical conditions addressed by the curriculum relating to the Membership Examination of the Royal Colleges of Surgeons (MRCS). Multiple-choice questions with feedback and discussion boards facilitated by expert clinical tutors provided formative assessment. Summative assessment comprised written examination at the end of each of the first 2 years (equivalent to MRCS level), culminating in submission of a research dissertation in year 3 toward an MSc.
METHODS: Students' age, gender, and level at entry to the programme were documented. Anonymized student feedback from 2008 to 2012 was examined using online questionnaires, and performance in the MSc programme was compared to MRCS examination outcomes for students who had consented to release of their results.
RESULTS: A total of 517 surgical trainees from 40 countries were recruited over the 6-year period, and 116 MSc students have graduated to date. Of 368 students, 279 (76%) were foundation doctors (interns) and had not commenced formal surgical training on enrolling in the MSc programme. However, level at entry did not influence performance (P > 0.05 across all 3 years). Average pass rates since the programme launched, for those students completing all of the required assessments, were 84% ± 11% in year 1, 85% ± 10% in year 2, and 88% ± 7% in year 3 of the MSc programme. MSc students had significantly higher MRCS pass rates than nonenrolled trainees (67% vs 51%, P < 0.01, n = 352). There was a significant correlation between MRCS examination performance and overall performance in the MSc (R = 58%; P < 0.01, n = 37). Of 248 respondents, 202 (81%) considered that the MSc would improve their chances of gaining a surgical training post, and 224 (90%) would recommend the programme to their peers.
CONCLUSIONS: The online MSc programme supports academic development of trainees in the early years of surgical training, is well received by students, and is associated with improved success in their professional examination.

Recent years have seen major changes in surgical training worldwide, with reduced working hours enforced under European law,1 and the introduction of run-through and core training programmes in the United Kingdom, which no longer expose the surgical trainee to all the common surgical conditions and operations that would have formed part of traditional surgical training. Trends to reduce working hours are also evident in North America2 and Australasia.3 In addition, there has been a lack of teaching provision of knowledge and consistent educational support as the trainee has approached the major compulsory examination of their early professional career, Membership examination of the Royal College of Surgeons (MRCS), normally taken during the first 2 years of basic surgical training. In response to these changes, a jointly awarded modular qualification, the Edinburgh Surgical Sciences Qualification (ESSQ—http://www.essq.rcsed.ac.uk), was launched in 2007 by the University of Edinburgh and the Royal College of Surgeons of Edinburgh (RCSEd). The programme was based on the MRCS curriculum but was tailored to be equally applicable to the early years of surgical training programmes worldwide.

The online ESSQ provides learning material tailored to the professional needs of surgical trainees, delivered in a flexible and readily accessible manner that enables the student to measure their progress through the individual components of the MRCS curriculum, while providing peer support and senior mentorship during this critical phase of their professional development. The online clinical case scenarios have specific objectives that underpin the curriculum in the applied surgical sciences of anatomy, radiology, physiology, pathology and bacteriology, with modules in the assessment and care of the surgical patient, research, and communication skills.

The aim of this study was to determine the impact of the ESSQ e-learning programme in supporting the early academic development of surgical trainees in their preparation for professional examination. The success of trainees in completing their MSc degree and in securing their professional examination (MRCS) was assessed and the quality of the programme determined by securing direct student feedback.

METHODS

The ESSQ programme is intended for doctors in the early stages of their careers (pre-MRCS), and therefore those who already possessed the MRCS or Fellowship of the Royal College of Surgeons (FRCS) were ineligible to enroll. Applications were accepted only from trainees in possession of a medical degree from their country of origin (MBChB or equivalent) and who were based within a training environment. The programme provides academic recognition of trainees' acquisition of knowledge and development; the ESSQ leads progressively to a Certificate (year 1—basic science relevant to surgery), Diploma (year 2—clinical surgery) and an MSc in Surgical Sciences (year 3—research component) over a 3-year period of part-time study that runs alongside and complements full-time clinical training. Students in the first 2 years of the programme have access to e-tutor–assessed collaborative discussion boards, where they are expected to lead debate and facilitate interaction with others and bring original material to support their arguments (http://demo.essq.ed.ac.uk/Surgical_Oncology_Discussion.asp), in addition to completion of multiple-choice questions (MCQs) and mini essays, with formative feedback provided. Summative assessment of knowledge is undertaken through an end-of-year examination comprising single best answer questions and extended matching item questions in a similar format to the MRCS examination. In addition, there is an in-course assessment score reflecting the student's performance in various aspects of the course throughout the year. During the second year, students are also introduced to the basic concepts relating to their academic development in preparation for the final year, in which students are afforded the opportunity to undertake a period of novel research leading to a dissertation and the award of MSc in surgical sciences. Although the primary objective of the third year is to consolidate academic principles and learning, many individuals go on to publish their work in peer-reviewed journals.

Academic data regarding performance and assessment (end-of-year scores) were sourced from student records relating to academic years 2007–2008 to 2011–2012 of the MSc in Surgical Sciences programme at the University of Edinburgh, in partnership with the Royal College of Surgeons of Edinburgh. Demographic data collected included age, gender, and the training post held at the time of entry to the programme. In years 1 and 2, students were assessed on the basis of completion of in-course MCQ, discussion board contributions, essays (year 2 only), and final examination marks. The relative weighting of in-course assessment increased over the 3 years: 25% in year 1, 30% in year 2, and 50% in year 3. In year 3, students completed a research project proposal and summary, an e-poster, and a final e-dissertation, all of which contributed to the end-of-year score. All examination questions were quality assured (including Angoff standard setting) and the examination and in-course assessment results were ratified by a programme Board of Examiners at the University of Edinburgh, which included 2 external examiners, both consultant surgeons with academic appointments at other UK institutions, which included responsibilities for postgraduate clinical teaching. The selection of external examiners and conduct of the board of examiners were done in accordance with the University's Taught Assessment Regulations and Guidelines.

Anonymized student feedback was examined over the same period using archived responses to online questionnaires. Such feedback was sought on agreed pro forma that were circulated to students at the end of each academic year. A search of PubMed and Web of Knowledge was undertaken using ESSQ student name, supervisor name, and research topic to identify published research articles and abstracts resulting from the dissertation topic.

The MRCS is designed especially for surgical trainees in Great Britian and Ireland and is normally taken during the first 24 months of surgical training. Part A of the MRCS is a written examination on generic surgical sciences and applied knowledge, whereas the Part B Objective Structured Clinical Examination integrates basic surgical scientific knowledge and its application to clinical surgery. An e-mail was sent to all students who had completed both years 1 and 2 of the ESSQ, requesting their consent for the programme team to obtain MRCS Part A examination scores. Where consent was given, MRCS results were obtained from the “Assessment and Quality Assurance Examinations” section of the RCSEd. In addition, pass rates (of 100%) were obtained from the RCSEd for all MRCS examinations held during 2008 and 2012. Data were separated into pass rates for ESSQ students and non-ESSQ candidates. Immediately after their correlation with performance in the MSc programme, all data relating to individuals' MRCS examination scores were anonymized.

Statistical analysis was performed using SPSS Statistics 19 and GraphPad Prism 3.0 software. Differences between group means were identified using Student t test for 2-group comparisons, and 1-way analysis of variance for multiple group comparisons. The influence of age, gender, and level at entry on end-of-year scores was assessed using the Cox proportional hazards regression model. Linear regression analysis was carried out to determine the relationship between ESSQ performance and MRCS examination scores. Data are expressed as mean ± SD unless otherwise stated. Statistical significance was accepted when P < 0.05.

RESULTS

A total of 517 surgical trainees from 40 countries (Fig. 1) were admitted to the programme over the 6-year (2007–2012) period observed. Of the 205 trainees who entered the 3-year programme during 2007–2009, 116 completed successfully and have graduated with an MSc degree (57%), 36 (18%) completed at Diploma level, 26 (13%) completed at Certificate level, 4 (2%) have returned from an interruption of study, and 23 (11%) withdrew early with no award. The recruitment and overall progression rates of the ESSQ programme over the 5 completed academic years are shown in Table 1.

FIGURE 1.

The online distance-learning ESSQ/MSc in Surgical Sciences attracts students from 40 different countries.

TABLE 1.

ESSQ Recruitment and Progression Data

The majority of students, 279 of the 368 (76%) whose level of work experience was known, entered as foundation doctors (interns) within 2 years of graduation from medical school and before commencing formal surgical training. Of the remaining 89 trainees, 68, 18, and 3 students were in the first, second, and third years, respectively, of a surgical training programme (Fig. 2). Level of work experience at entry to programme was not found to influence performance across each of the 3 years of the programme (Fig. 3). On multivariate analysis, age, gender, and level of work experience at entry were not found to be predictors of end-of-year scores (Table 2).

FIGURE 2.

Surgical training structure in the United Kingdom. The trainee has to demonstrate the acquisition of knowledge and technical skills in the workplace and negotiate examinations of the Royal College of Surgeons (Membership, MRCS, and Fellowship, FRCS), which are mandatory requirements in demonstrating competency in surgery.

FIGURE 3.

The influence of level at entry to the ESSQ programme and academic performance (2007–2012). Individual marks (with mean values) shown for students at the end of (A) year 1 (n = 279), (B) year 2 (n = 188), and (C) year 3 (n = 117) reveal that there is no significant effect of level of training upon entering the programme and end-of-year results (P = 0.26, 0.12, and 0.48 for years 1, 2, and 3, respectively; 1-way analysis of variance). FY1/2 = Foundation year doctors (year 1 or 2 after graduation); ST1/2 = first-/second-year surgical trainee, equivalent to first-/second-year core trainee (CT1/2); ST3 = third-year surgical trainee.

TABLE 2.

Multivariate Analysis of the Influence of Gender, Age, and Training Level at Entry, on Academic Performance on the ESSQ Programme

Average pass rates since the programme launched were 84% ± 11% in year 1, 85% ± 10% in year 2, and 88% ± 7% in year 3. MSc students had significantly higher MRCS pass rates than nonenrolled trainees (Fig. 4), for both parts A and B of the examination. Comparison of the groups sitting the MRCS examinations revealed that candidates were similar in terms of age and expected level of clinical experience: ESSQ and non-ESSQ candidates both had an average age of 26 years. Furthermore, there was a significant correlation between MRCS examination performance and overall performance in the first 2 years of the MSc programme (Fig. 5).

FIGURE 4.

Pass rate in 2008-2012 for students enrolled (ESSQ) or not enrolled (non-ESSQ) in the ESSQ. (A) Combined or overall examination pass rate (mean ± SD; *P < 0.01, n = 352; unpaired student t test) and (B) separate part A and B examination pass rate (mean ± SD; *P < 0.05, n = 223; †P < 0.01, n = 129; unpaired Student t test).

FIGURE 5.

Regression analysis showing the association between student performance in the first 2 years of the ESSQ and MRCS part A examination outcome (y = 0.6505x + 34.859; R2 = 58%; P < 0.01, n = 37) (95% confidence intervals).

Feedback from online, anonymous questionnaires revealed that 224 (90%) of 248 respondents would recommend the programme to their peers and the same percentage reported that they had a better understanding of the basic and applied surgical sciences than they would have otherwise had if they had not enrolled on the ESSQ programme. The majority of students (81%) considered that the ESSQ would improve their chances of gaining a surgical training post. In addition, 61% found the level of the MCQs within the ESSQ programme to be more difficult than those in the MRCS examination. On average, year 1 students felt that they needed to spend 10 ± 6 hours per week engaged in ESSQ activities to keep up-to-date with the timetable. Respondents consistently reported that they found the ESSQ an enjoyable experience. Search of PubMed and Web of Knowledge revealed that there have been 26 full research papers and 27 abstracts published to date relating to ESSQ research dissertations submitted May 2010 onwards.

DISCUSSION

The ESSQ/MSc in Surgical Sciences was established in 2007 as a response to changes in employment law in Europe, which reduced clinical exposure in the workplace.1 Contemporaneously, changes in surgical training were implemented with the aim of streamlining progression through training milestones and shortening the duration of training. The entire ESSQ programme, based on the MRCS curriculum, is taught through self-directed learning at a distance. It is delivered via a virtual learning environment which, allows students easy access to internal and external coursework, on an “anytime, anywhere” basis via the Internet on computer or mobile devices. The learning style provides students the options of a problem-oriented entry or systematic learning illustrated through subsequent problem solving, delivered in modules scheduled in blocks of time that accommodate and complement the student's clinical training demands.

Since its inception, the ESSQ has consistently exceeded its recruitment target with the highest student numbers of any postgraduate programme at the University of Edinburgh. Although targeted specifically at surgical trainees based in the United Kingdom, it has attracted more than 500 students from 40 countries in the last 6 years. The programme appears, therefore, to have a growing international reach with applicability to the early years of surgical training in many countries. This may be related to the current general reduction in working experience for doctors; the requirement to reduce working hours in Europe, North America, and Australasia1–3 has diminished considerably the trainee's exposure to both common and less frequently encountered surgical conditions.4,5 There is, therefore, a need to consolidate the applied knowledge that may be compromised by reduced exposure to the assessment of the patient in both the elective and emergency setting.6,7 We view distance-learning surgical educational programmes as an approach different from and complementary to traditional vocational methods, and which serve to enhance the learning experience of surgeons in training, not to replace but to complement experience in the workplace. The high recruitment to the programme and positive feedback from students would endorse this view.

The design of the ESSQ programme is underpinned by the pedagogical principles of e-learning, whereby students are supported as members of an online community with clearly defined learning objectives and use of materials that are active, engaging, collaborative, and relevant.8,9 Ease of navigation around the virtual learning environment and regular support from the programme team are critical in achieving successful online learning.10 The e-learning platform developed in-house (eeSURG, http://learning.essq.ed.ac.uk) allows students to navigate resources easily, follow the programme curriculum, access course material, and interact with course tutors and other students. Central to course design, Blanchard and Frasson11 emphasize the importance of roleplay to enhance constructivist learning. The activity model within the eeSURG platform is provided by Labyrinth (http://labyrinth.mvm.ed.ac.uk), a tool for authoring and delivering case narratives. Labyrinth was designed in-house with the goal of supporting development of richly engaging, narrative medical cases that invite users to take control of their own decisions—and, by extension, their own learning—and develop the critical analysis skills to face effectively the consequences of those decisions.12 It is noted that there was an appreciable “dropout” rate from the programme each year (10%–15%) but this was less than that observed in other distance learning masters programmes; online courses are associated with higher attrition rates than on-campus equivalents,13 with several studies reporting dropout rates for online masters programmes exceeding 40%.14

Although the online ESSQ/MSc in Surgical Sciences is designed to be studied part-time, alongside full-time clinical training in a hospital setting, the absence of a significant association between students' training level and academic performance on the ESSQ programme suggests that the content of the programme stands alone and is not necessarily dependent on previous surgical experience. Greater experience held by more advanced trainees may be offset by a longer period away from formal academic study, conferring an advantage to recent graduates who are more accustomed to intensive study. Furthermore, it is recognized that each student within a particular level of foundation (intern) or surgical training can have a very different profile of skills and experience compared with others at the same level, accounting for the lack of a relationship between performance and level at entry to the programme.

Given that one of the principal aims of the ESSQ programme was to prepare the surgical trainee better for the individual components of the MRCS examination that were covered in the first 2 years to Diploma level, monitoring pass rates for those students who have presented themselves for MRCS examination in the United Kingdom has provided a valuable assessment of outcome. The increased average pass rate for ESSQ students (∼15% higher) than non-ESSQ students add credence to the positive appraisal of the programme to date and will contribute to future student engagement and motivation. The correlation between performance in the ESSQ programme and MRCS examination scores and the positive feedback from students further demonstrates that the programme is fit-for-purpose. It is accepted that a potential limitation of the current analysis is that the ESSQ may not improve MRCS outcomes per se, but rather the ESSQ students are a self-selected group of highly motivated students who were more likely to be successful in the MRCS.

The programme, however, was also intended to support the academic development of the surgical trainee at an early stage, and it is noteworthy that 116 students who enrolled in the programme since its launch in 2007 continued to complete a postgraduate degree. This has provided them with a strong academic foundation irrespective of their final career intentions. It may encourage some to pursue an additional period of research training, whereas for others, the academic exposure may replace the traditional approach, which has required surgical trainees in the United Kingdom to pursue 2 or 3 years out of surgical programme training to secure a postgraduate qualification that may make them competitive to progress thought-advanced surgical training. Although, at the present time, there is insufficient follow-up of MSc postgraduates to determine whether their participation will impact on their future surgical training plans or academic development, it is encouraging to note that 26 full research articles have been published directly from content relating to the MSc dissertation project.

There is considerable potential for distance learning programmes to have a major impact on the future delivery of surgical training. The data from this study are relatively immature to determine whether it has contributed to a major change, but it has been demonstrated that the online ESSQ programme supports the academic development of trainees in the early years of surgical training, is well received by students, and is associated with improved success in their professional examination.