Alumni-based evaluation of a novel veterinary curriculum: are Nottingham graduates prepared for clinical practice?

INTRODUCTION: Outcomes-based education has been the core of the curriculum strategy of the Nottingham School of Veterinary Medicine and Science (SVMS) since its inception in 2006. As part of the ongoing curriculum evaluation, the first two graduating cohorts were invited to provide an appraisal of their preparation by the SVMS curriculum for their role in clinical practice. This paper provides brief accounts of the SVMS curriculum model, the development of the evaluation instrument and the findings of the alumni survey.
MATERIALS AND METHODS: The evaluation instrument contained 25 attributes expected of SVMS graduates. Alumni rated their preparation for practice in relation to each attribute.
RESULTS: The four highest rated characteristics were compassion for animals and the application of ethics to animal welfare; communication skills; recognising own limitations and seeking help and advice where needed and clinical examination skills. The four lowest rated were clinical case management and therapeutic strategies; dealing with veterinary public health and zoonotic issues; knowledge of current veterinary legislation and dealing with emergency and critical care cases. Free text responses were in line with these quantitative findings.
CONCLUSION: The results indicate that this sample of SVMS graduates were satisfied with their undergraduate education and felt well prepared for their role in clinical practice.

Introduction

Alumni-based evaluations are increasingly recognised as an important part of the process of ongoing curriculum evaluation. They provide insights into the undergraduate learning experience and students’ preparation for work as clinical practitioners (Bristol 2002, Hardin and Ainsworth 2007, Doucet and Vrins 2010). Together with the views of other stakeholders including staff, students, employers and clients, alumni evaluations can provide useful feedback to curriculum designers (Trent 2002, Edmondson 2004).

Within higher education, there has been increasing emphasis placed on the intended learning outcomes (ILOs) of a course and this has led to the development of outcomes-based curricula (Harden and others 1997, Davis 2003, Bok and others 2011). In essence, outcomes-based education (OBE) is based on the principle that the outcomes that students are expected to display at the end of the course are decided, and the curriculum is subsequently developed to enable it to achieve these outcomes (Spady 1988, Harden 1999a). The outcomes-based model has been adopted by professional healthcare courses (Harden and others 1997, Chambers 1998, Chapman 1999, Bok and others 2011), and following the introduction of the Royal College of Veterinary Surgeons (RCVS) day one competences in 2001 (RCVS 2014a), veterinary schools were encouraged to review their curricula with less emphasis on course content and more importance placed on the performance of graduates.

The implication of this approach for curricula in veterinary medicine is that the emphasis should primarily be on what graduating veterinary students can do, rather than on what they should know on ‘day one’. The approach has affinities with Miller's hierarchical model of assessment of knowledge, clinical skills, competence and performance (Miller 1990). Miller emphasised that although all levels need to be assessed the level of ‘does’ should be the priority in a medical curriculum (see Fig 1).

FIG 1:

Miller's pyramid of clinical competence (Miller 1990)

Critics of OBE have argued that, by virtue of defining specific outcomes, this approach limits potential achievement within education (McKernan 1993, Hussey and Smith 2002, Davis 2003); students are often disempowered (Rees 2004), and an emphasis on broad personal and social outcomes detracts from the academic rigour of a programme (O'Neil 1994). Whatever criticisms OBE has received in other fields, within veterinary education the performance of the new graduate veterinary surgeon cannot be ignored. Universities have a duty to the profession, the public, accrediting bodies and to their students to graduate vets who are ‘fit to practice’. Defining the components of veterinary clinical competence and ensuring all students achieve these standards at graduation is paramount for universities to fulfil this duty. As part of our ongoing evaluation, the opinions of the 2011 and 2012 cohorts of Nottingham graduates are reported here. Graduates were asked how prepared they considered they were for the challenges of clinical practice. The questions used in the survey were based on the ILOs of the curriculum.

Specifically the following research questions were posed:

What are the intended learning outcomes of the current veterinary medicine course, based on stakeholder opinion?

Does the course prepare students for the demands of clinical practice?

Context of the study: the SVMS curriculum

The School of Veterinary Medicine and Science (SVMS) at the University of Nottingham opened in 2006, and the first cohort of students graduated in 2011. Adopting a distributed curriculum model, SVMS has pioneered a novel and unique approach to veterinary education in the UK. Students at the SVMS follow a 5-year course; successful students are awarded the Bachelor of Veterinary Medical Sciences degree at the end of their third year and the Bachelor of Veterinary Medicine Bachelor of Veterinary Surgery at the end of their final year.

An outcomes-based approach informed the design of the SVMS curriculum, which is taught in systems-based modules following a spiral model (Harden 1999b). Each successive year of the curriculum builds new knowledge and expertise upon the knowledge and practice of the previous year (Fig 2). Students study each system with a focus on basic sciences in years 1 and 2 and then revisit each system in years 3 and 4 where the emphasis is on the clinical science. Alongside these systems-based modules, which are delivered in blocks of 2–11 weeks, run 3 long modules: personal and professional skills, animal health and welfare and veterinary public health. These modules run throughout the year and are integrated with the concurrent systems-based modules.

FIG 2:

The School of Veterinary Medicine and Science curriculum based on a spiral model

Clinical examples and case studies are provided to contextualise the basic science from the start of the course in year 1. There is an emphasis on practical work including laboratory techniques, clinical and professional skills. During third year, students undertake a 12-week individual research project followed by two modules comprising para-clinical sciences and practical techniques in preparation for the clinical modules. The fifth year of the course is a ‘lecture-free final year’ where students complete clinical placements at university clinical associate practices. This is consistent with a distributed curriculum model: there are no teaching hospitals for any of the veterinary species at the University of Nottingham and clinical experience is provided at local veterinary practices, referred to as ‘clinical associates’. Clinicians are employed by the SVMS to work collaboratively with each of the associate practices to oversee the teaching and assessment within each clinical rotation.

Students are assessed using a variety of formats including short answer questions and multiple-choice questions delivered online; practical objective structured clinical examinations assessments; coursework assignments; a reflective portfolio and during their clinical placements in final year directly observed procedural skills. All of these are in alignment with the intended learning outcomes of the curriculum.

Methods

Defining the ILOs and developing the graduate survey

Although ILOs had previously been written for the SVMS course prior to its inception in 2006, it was decided to review and redefine the ILOs to ensure they were representative of the current intentions of all teaching staff. The RCVS day one competences (RCVS 2014a) were considered as the learning outcomes for the study. However, these outcomes are not specific to the Nottingham graduate and were under review at the time the research was carried out. It was therefore decided to produce a set of outcomes that were a more appropriate reflection of the current SVMS curriculum.

The new learning outcomes were based on the results of a literature review, a subsequent focus group discussion and a survey of SVMS staff. These new learning outcomes provided the basis of the graduate survey of 2011 and 2012 that took place in 2012–2013. In this report, these are referred to as expected attributes of graduates rather than intended learning outcomes although this term is used in the undergraduate curriculum documents. Fig 3 summarises the development of the alumni survey.

FIG 3:

Development of the alumni survey used in the School of Veterinary Medicine and Science (SVMS) curriculum evaluation. ILOs, intended learning outcomes

Thirteen articles were identified in the literature search describing a total of 133 desired skills or attributes of veterinary graduates commencing a career in practice. These were reviewed for overlap or repetition and grouped according to the type of skill or attribute by the lead researcher.

At the same time, a focus group was formed of three clinicians, two non-clinical academic staff members and the school manager. The discussion was facilitated by the lead researcher and aimed to explore and define what they considered to be the key desirable attributes of a Nottingham veterinary graduate.

The findings of the focus group were then cross-referenced with the results of the literature search to produce a final list of 25 graduate attributes.

Validating the ILOs

In a wider survey, a questionnaire was sent to 33 academic members of SVMS staff and clinical associate employees to validate the final list of graduate attributes. This online questionnaire was delivered using a web-based survey tool: SurveyMonkey (www.surveymonkey.com). Participants were asked to rate each of the attributes for importance on a scale of 1–5 and also to provide any comments they had on the individual outcomes. Fig 4 provides an example section from the questionnaire.

FIG 4:

An example section from the online questionnaire asking staff to rate each of the attributes for importance to the School of Veterinary Medicine and Science graduate

Nineteen staff responded to the survey, a response rate of 57.6 per cent. All attributes were considered essential or very important, and therefore, none were removed from the list. The additional free text comments from respondents resulted in some minor changes to the wording of some of the attributes. The resulting modified list of intended graduate attributes was used in the graduate survey.

The graduate survey

In the survey, the graduates were asked to rate, on a five-point Likert-type scale (where 1=not at all prepared and 5=excellent preparation), how prepared they felt to perform each of the attributes of the SVMS courses in the context of their current employment. In addition, they were asked to rate how prepared they felt overall to do their current job and some general questions about their postgraduate experience. The descriptors and graduate attributes are shown in Box 1.

Introduction

Please try to consider the education you received while at Nottingham Vet School when answering the questions rather than any support or continuing professional development (CPD) you have received postgraduation.

Please use the free text boxes to enter any additional comments within each area.

Thinking about the demands of your current role, use the following options to tell us how prepared you feel for your job in each area:

Excellent preparation, I am able to carry out all requirements of my current job

Good preparation, I have needed minimal support or training since graduation

Adequate preparation, I felt somewhat prepared although have needed to use support and training since graduation

Poor preparation, insufficient coverage of this area in the course to be adequately prepared, significant support and training postgraduation have been necessary

Not at all prepared, the course did not prepare me at all to carry out the requirements of my current job in this area, further support and training have been essential

Areas of veterinary practice

Knowledge of underpinning basic science

(including normal structure and function of healthy animals; molecular, biochemical and cellular mechanisms; mechanisms of defence; animal management, nutrition and husbandry systems)

Veterinary clinical knowledge

(including disease aetiology and pathophysiology; mechanisms of disease transmission; knowledge of medical and surgical management)

Clinical and surgical skills

(including diagnostic imaging skills; basic surgical skills; anaesthesia skills and fluid therapy)

Clinical examination skills

(within all species and including animal handling and an understanding of animal behaviour)

Diagnostic reasoning abilities

(including compiling an appropriate differential diagnosis list; decision-making and identification of common and important conditions)

Clinical case management and therapeutic strategies

(including making therapeutic decisions; prescribing and dispensing appropriately and legally; performing drug dose calculations and reproductive management)

Dealing with emergency and critical care cases

(including initial assessment and triage of emergency cases)

Promoting preventative healthcare

Population health and epidemiology

Dealing with veterinary public health and zoonotic issues

(including notifiable diseases and an understanding of the importance of food safety)

Recognition for need and implementation of euthanasia

Veterinary practice and financial management

(including financial awareness in case management)

Recognising own limitations and seeking help and advice where needed

(including reflective practice and being open to constructive criticism)

Ability to seek, critically evaluate and use new information from a variety of sources

Knowledge of current veterinary legislation

Compassion for animals and the application of ethics to animal welfare

Awareness of professional responsibilities to patients, clients, colleagues, society

(including understanding the needs and maintaining respect for all clients; engagement in CPD and lifelong learning; putting evidence-based veterinary medicine into practice and respectfully challenging bad practice; maintaining a safe working environment; showing a caring attitude towards colleagues and shaping the future of the profession)

Communication skills

(including empathy and communication with clients; written communication skills, e.g., clinical records and referral letters; communication with veterinary colleagues and paraprofessionals)

Interpersonal and teamwork skills

(including leadership skills; showing initiative in the workplace; time management skills and confidence to take control of a situation)

‘Robustness’ —the ability to manage pressure and stress

Flexibility and ability to cope with change

Self-reflection and maintaining a work life balance

Systematic approach to problem solving and critical thinking

IT skills required for your current role

Research skills

The questionnaire was distributed to all 2011 (87) graduates in December 2012 and all 2012 (89) graduates in February 2013. All graduates were given the opportunity to complete the survey on paper and return in a prepaid envelope, or complete the survey online. Both cohorts were given four weeks to complete the survey. A reminder email was sent out two weeks before closure of the survey and a link posted on the alumni Facebook page.

Quantitative data were analysed in SPSS Statistics, V.17. Likert-type descriptors were converted to numerical responses. Changes had been made to the curriculum following the first cohort, which may have influenced the graduates’ perceived preparation for practice. Previous studies have identified a lack of confidence in female graduates facing the transition to practice compared with their male counterparts (Kogan and others 2004, Tomlin and others 2010). Therefore, Mann-Whitney U tests were used to compare the responses of 2011 and 2012 graduates and the responses of male and female graduates. A Kruskal-Wallis test was used to compare responses from graduates working in different practice types (small animal, equine, farm animal and mixed practice). For all statistical analyses, significance was measured as P≤0.05.

The free text responses from the survey were assigned an initial code and used to illuminate the quantitative data; the qualitative data were explored in greater detail using thematic analysis (Braun and Clarke 2006).

Results

The final version of the 25 expected attributes for a Nottingham veterinary graduate is shown in Box 1. Twenty-eight responses from the 2011 graduates (response rate 32.2 per cent) and 38 responses from the 2012 graduates (response rate of 42.7 per cent) were received, a total of 66 responses and an overall response rate of 37.5 per cent. The type of veterinary practice the graduates are working in is shown in Fig 5.

FIG 5:

Distribution of graduates working in different types of veterinary practices

On the five-point Likert scale, the combined mean of the 25 attributes was 4.19; the mean for overall competence to do the job for which the graduates were hired was 4.36. Table 1 summarises the responses for the four attributes for which they considered they were best prepared for and the four attributes for which they felt least prepared.

TABLE 1:

A summary of responses to the highest and lowest rated attributes

Attribute	Mean rating	sd	Median rating	Min rating	Max rating
Compassion for animals and the application of ethics to animal welfare	4.72	0.43	5.00	3.00	5.00
Communication skills	4.71	0.70	5.00	3.00	5.00
Recognising own limitations and seeking help and advice where needed	4.61	0.61	5.00	3.00	5.00
Clinical examination skills	4.58	0.59	5.00	3.00	5.00
Clinical case management and therapeutic strategies	3.86	0.77	4.00	3.00	5.00
Dealing with veterinary public health and zoonotic issues	3.70	0.83	4.00	2.00	5.00
Knowledge of current veterinary legislation	3.61	0.75	4.00	2.00	5.00
Dealing with emergency and critical care cases	3.59	0.77	4.00	2.00	5.00

There were no statistically significant differences in the responses from 2011 graduates compared with 2012 graduates or comparing responses from male and female graduates. But there were significant differences between graduates working in different practices for diagnostic reasoning abilities; dealing with veterinary public health and zoonotic issues; self-reflection and maintaining a work-life balance, illustrated in Table 2.

TABLE 2:

Differences in responses from graduates working in different types of practice

Attribute	Small animalMean	Equine practiceMean	Mixed practiceMean	Farm animalMean	χ2	P value
Diagnostic reasoning	4.07	4.00	4.56	4.00	8.74	0.033
Veterinary public health and zoonotic issues	3.93	3.17	3.72	3.13	14.57	0.002
Self-reflection and work-life balance	4.30	3.33	4.44	3.75	9.93	0.019

The free text comments generally supported the quantitative results for the highest rated attributes as demonstrated in the following quotes:

I never realised how much I would rely on our communication training from Day One after graduating - THANK YOU for preparing me!

Practice makes perfect! Glad that we had plenty of chances to examine ‘normal’ animals

Felt that Pebble pad helped with reflection and knowing personal limitations etc.

However, some graduates identified opportunities for development even in these highly valued aspects of the course. For example, communication skills and clinical examination:

We didn't cover referral letters - this would have been useful and paraprofessional communication

Excellent for dogs/cats/horses/cows etc but not good for rabbit, Guinea pigs, small furries, any exotics.

The free text responses also helped to explain why graduates felt less prepared in response to the lowest rated outcomes:

I have required some guidance since graduation. But emergency situations are difficult to prepare for at vet school as you can't experience them all before you leave

Although legislation has changed since these things were being taught e.g. pet passport regulations.

Having been given so little case responsibility during final year rotations, this was a big leap into day one in practice - some more responsibility would ease this transition I feel.

Following a thematic analysis of all comments, three themes were identified in the free response data: confidence, the course and managing the transition to practice. Only a summary of this analysis is provided here.

Confidence

Graduates described being confident and also lacking confidence regarding specific skills or topics. These comments reflected individual variation in how confident graduates are in their own abilities within specific subject areas and with more generic skills.

The course

Graduates referred to specific areas of the course in their comments, either that they felt were particularly well-covered or areas that they felt needed improvement.

Transition to practice

The final theme relates more generally to the transition to clinical practice. Many graduates describe this experience as stressful and challenging, for example:

Although I feel I have been well prepared for managing the pressures and stress of life in practice, this is still something I feel all new graduates will always struggle with early in their career, no matter how much preparation they are given.

Finally, case responsibility appears to be a significant change that graduates struggle to cope with:

I personally have struggled with the pressure - unsure how much more Vet School could do as ultimately you are never the responsible vet as a student

In the final section of the survey, the graduates were asked about their postgraduate experience. Most graduates (87.5 per cent) felt that their experience of work had matched up to their expectations. Also, 36.5 per cent of graduates stated that their eventual choice of work had been influenced by the Nottingham course and 95.2 per cent of graduates would still study veterinary science at Nottingham if they could repeat their time at university. These results are shown in Fig 6.

FIG 6:

Responses to three questions about the School of Veterinary Medicine and Science graduate experience

Discussion

The distribution of SVMS graduates working within different types of practice appears similar to that of veterinary graduates in the UK represented in a national survey (Robinson and Buzzeo 2013). Participants of this study felt very well prepared for their entry into the profession; this high level of satisfaction among Nottingham graduates is consistent with findings from other studies (AVS BVA 2012, Ipsos Mori 2013). Even the four lowest course attributes were rated by most as very good. Clinical integration and practical experience from year 1 of the course may have contributed to the graduates’ confidence in their professional role. Case availability and the limits of their responsibility as undergraduates may have resulted in minor criticisms. For example, despite the emphasis on a first opinion caseload, Nottingham students are still reliant on extra mural studies to maximise their surgical experience. This finding is common to the majority of veterinary graduates in the UK according to a recent survey (RCVS 2014b).

These results provide clear indications that the personal and professional skills module has been successful in providing communication skills training, encouraging reflective practice and providing students with an understanding of business and practice management. These skills had previously been identified as lacking in many veterinary curricula (Lloyd and Walsh 2002, Routly and others 2002, Gilling and Parkinson 2009).

The findings also raise the issue of case responsibility. Consistent with previous findings (Garrett 2009), this has been one of the most challenging aspects of working as a newly qualified veterinary surgeon for the participants in this study. However, this is problematic for all veterinary courses, as during undergraduate clinical placements responsibility for each case seen will ultimately lie with a qualified veterinary surgeon and not the student. There is therefore a need for novel methods of delivery and assessment that require the student to make decisions regarding diagnoses and clinical case management, where there are real consequences to their actions but no compromise to animal welfare. The use of high-fidelity simulation has been used to this effect within medical education (Gordon and others 2001, Barry Issenberg and others 2005) and should be considered as a potential strategy for enhancing clinical decision-making among veterinary students.

The transition to clinical practice has been described as a make or break period for veterinary graduates (Gilling and Parkinson 2009). Success is often dependent on the support and supervision provided by employers and this can be variable. Although initiatives exist that offer support to graduates (Boulton and McIntyre 2012, Hunting 2007), these rely on individuals actively seeking further guidance and support and are not mandatory. The introduction of the compulsory professional development phase by the RCVS (2007) is a step towards improved support for all graduates. However, this is still far removed from the continuum of medical education, in which postgraduate training continues to provide a focus for educational research and development (Calman and others 1999, Ovseiko and others 2014). Despite this mandatory postgraduate training and recent reforms to medical education, many junior doctors still find the transition from student to practitioner stressful (Brennan and others 2010). Medical graduates face similar challenges to those reported by the new graduate veterinary surgeons in this and previous studies. For example, case responsibility, clinical decision-making, managing critically ill patients and a lack of confidence in performing practical procedures (Prince and others 2004, Illing and others 2013). Consequently, there has been a call for an increase in workplace-based learning within a medical team prior to graduation (Illing and others 2013). The evidence from within veterinary and medical education therefore suggests a need for further research into the transition to independent practice and the interventions that will enhance curricula and help to prepare students.

There are of course limitations to the current study. This sample of SVMS alumni, other than a few minor reservations, considered they had been thoroughly prepared for their entry into the profession. However, the results of this survey cannot be regarded as a proof that outcome-based curricula are superior to other forms of curricula. It should also be acknowledged that the transition from university to clinical practice is a complex process with multiple influences, including personal, social and employment related factors in addition to the impact of previous veterinary education. Identifying direct relationships between the curriculum and preparation for practice is therefore challenging. Furthermore, graduate perception is not a direct measure of learning outcomes and self-evaluation has been shown to differ from employer evaluations (Woolliscroft and others 1993, Doucet and Vrins 2010). Finally, although response rates are consistent with similar studies performed elsewhere (Doucet and Vrins 2010, Danielson and others 2012) due to the sample size, the results may not represent the opinions of all alumni.

Despite these limitations, alumni surveys are increasingly used as part of the process known as outcomes assessment (Trent 2002, Kochevar 2004, Edmondson 2004) to provide valuable information in the evaluation of outcomes-based curricula. Arguably, alumni are the most important evaluators of an outcome-based curriculum because only they can match their experience of the curriculum against the demands of their current roles. Clients and employers can only evaluate the current work of the graduates. Quality assessors focus upon curriculum documents and interviews with academic staff and undergraduates. Academic staff can only offer their views on teaching and assessment, while students can comment on aspects of the curriculum but not on how it matches the demands of working life as a vet. Future research will combine these multifaceted evaluations of the curriculum and prioritise content and intended learning outcomes.

The purpose of outcomes assessment is to identify areas for potential improvement within the curriculum (Edmondson 2004, Kochevar 2004). It is essential that this information is fed back to curriculum designers and teaching providers so that the cycle is complete and the outcomes assessment exercise achieves its purpose. Potential areas for future improvement have been identified following the current study. One priority has been ensuring adequate preparation for work with all commonly encountered species, following the notable differences between the ratings for graduates working in different types of practice. While this is not surprising considering the varied nature of clinical practice, specific suggestions from alumni have been used to develop certain aspects of the curriculum.

The results of such outcomes assessments can only be interpreted in the context of the participating school. However, it is hoped that educators within other universities will identify with the areas for development within the SVMS curriculum, namely case responsibility and dealing with emergency patients in different types of veterinary practice. These areas may provide a focus for future research. Finally, a consistent approach to alumni evaluation as part of outcomes assessment would enable comparison of educational practice and new initiatives within different veterinary curricula.

Conclusion

We will continue to use alumni surveys as part of the SVMS course evaluation to minimise the effects of curriculum drift and maintain the original ethos of the course. Despite the limitations of the alumni survey, the evidence strongly indicates that the outcome-based curriculum at SVMS provides a firm foundation for entry into the veterinary profession. It is hoped that other UK veterinary schools engaged in developing their undergraduate curricula will find similar approaches and results.