A mixed-methods evaluation of the Educational Supervision Agreement for Wales.

OBJECTIVES: In a bid to promote high-quality postgraduate education and training and support the General Medical Council's (GMC) implementation plan for trainer recognition, the Wales Deanery developed the Educational Supervision Agreement (EdSA). This is a three-way agreement between Educational Supervisors, Local Education Providers and the Wales Deanery which clarifies roles, responsibilities and expectations for all. This paper reports on the formative evaluation of the EdSA after 1 year.
DESIGN: Evaluation of pan-Wales EdSA roll-out (2013-2015) employed a mixed-methods approach: questionnaires (n=191), interviews (n=11) with educational supervisors and discussion with key stakeholders (GMC, All-Wales Trainer Recognition Group, Clinical Directors). Numerical data were analysed in SPSS V.20; open comments underwent thematic content analysis. PARTICIPANTS: The study involved Educational Supervisors working in different specialties across Wales, UK.
RESULTS: At the point of data collection, survey respondents represented 14% of signed agreements. Respondents believed the Agreement professionalises the Educational Supervisor role (85%, n=159 agreed), increases the accountability of Educational Supervisors (87%; n=160) and health boards (72%, n=131), provides leverage to negotiate supporting professional activities' (SPA) time (76%, n=142) and continuing professional development (CPD) activities (71%, n=131). Factor analysis identified three principal factors: professionalisation of the educational supervisor role, supporting practice through training and feedback and implementation of the Agreement.
CONCLUSIONS: Our evidence suggests that respondents believed the Agreement would professionalise and support their Educational Supervisor role. Respondents showed enthusiasm for the Agreement and its role in maintaining high standards of training. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2017. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

The study contributes to the limited research on the impact of greater trainer recognition.

The sample includes Educational supervisors from all health boards in Wales working in many different specialties.

The study represents a snapshot in time at the introduction of the Educational Supervision Agreement; longitudinal follow-up is required.

Introduction

Securing and maintaining high-quality education for trainee doctors is key to ensuring excellent patient care and patient safety. In August 2012, the General Medical Council (GMC) published details of new arrangements for the formal recognition of medical trainers in secondary care.1 These arrangements apply to four categories of trainer: in postgraduate training, (1) educational supervisors and (2) named clinical supervisors; and in undergraduate education, (3) those responsible for overseeing students’ progress at medical school and (4) lead coordinators at each Local Education Provider (LEP). These arrangements use existing standards for postgraduate training as outlined in the Trainee Doctor 2 and the Tomorrow’s Doctors 3 and map to seven headings requiring evidence in appraisal.1 4 Providing a process for the formal recognition of medical trainers is designed to offer assurance to patients and the public that new doctors are developed with the appropriate knowledge, skills and behaviours. Regulation should support trainers as good professional role models for students and trainees to emulate.5

The Wales Deanery, in a bid to promote high standards of postgraduate education and training, developed and piloted a tripartite Agreement in early 2011. This pilot preceded the publication of the GMC’s Implementation Plan (2012) and formed the foundation of the ‘Educational Supervision Agreement’ (EdSA)6 which was designed to meet requirements for the recognition of one of the GMC categories of postgraduate trainer, namely the Educational Supervisor. Educational supervisors are clinicians who are responsible for the overall supervision and management of a trainee’s trajectory of learning and educational progress during a placement or series of placements.1

The Educational Supervision Agreement

The EdSA is a signed agreement between three parties: an individual Educational Supervisor, a LEP (the NHS Health Board or Trust, ie, the medical director) and the Wales Deanery (the Postgraduate Dean). Currently, it is an Agreement with educational and not named clinical supervisors who are trained individuals responsible for overseeing a specified trainee’s clinical work throughout a placement in a clinical or medical environment.1 We note that extension to this group is planned. At present, EdSA is for foundation, core and higher specialty training, not training in general practice. The Agreement is designed to be an explicit demonstration of the three parties’ responsibilities and is intended to enhance communication and accountability between those responsible for delivering and supporting educational supervision in secondary care. In signing the Agreement, the Educational Supervisor commits to undertaking a minimum of 8 hours of continuing professional development (CPD) per year that maps to at least two of the seven framework areas adopted by the GMC5 and to cover all areas within a 5-year cycle. In addition, educational supervisors must provide evidence of suitable attitudes and behaviours (for example, from multisource feedback, evaluations of teaching, placement feedback forms, GMC survey results, trainee audits and analysis of critical incidents) which is reviewed in an annual appraisal of the trainer role.

The LEP responsibilities under the Agreement include building into a trainer’s job plan a time tariff of 0.25 ‘supporting professional activities’ (SPA), equivalent to 1 hour per week, per trainee. This was established following the initial pilot and consideration of the workload involved in supervising trainees. Supervision requires time not only in direct educational supervision but also in mentoring, preparing reports, supporting trainees in difficulty, sitting on annual review of competence progression (ARCP) panels and undertaking CPD for the trainer role. The Agreement also requires the LEP to provide annual appraisal for the trainer role and participate in quality management processes which include feedback to the Deanery.

Wales Deanery responsibilities include monitoring the provision of time and implementing quality management processes relating to Educational Supervision. The Deanery also signs up to responsibilities related to selection of educational supervisors, supporting mechanisms for the delivery of induction and other essential training, promoting annual appraisal and rewarding excellence.

The EdSA sets out to achieve five main objectives:

To enhance communication and accountability between the Deanery, the LEP and the Educational Supervisor.

To recognise the role of the Educational Supervisor and raise the profile of their work.

To promote high standards of postgraduate medical education and training in Wales.

To contribute to safe, high-quality patient care.

To support the GMC’s implementation plan.

Roll-out of EdSA commenced across Wales on 13 November 2013. This paper reports on the formative evaluation of the EdSA after 1 year.

Methods

We adopted mixed methods, collecting data from telephone interviews and a questionnaire. In reporting these data we followed standard guidance (see supplementary material).

Telephone interviews

Eleven semi-structured telephone interviews were conducted by KW, with a purposive sample of Educational Supervisors across Wales (February–July 2014), from a range of specialties including medicine, obstetrics and gynaecology and public health and both men and women (n=8 male; n=3 female). Participants were sent an invitation email which included an information sheet and consent form. The telephone interview is widely used in research,7–9 and we judged it to be an appropriate means by which to gather opinion about the effectiveness and impact of EdSA from individuals across a wide geographical area.

The telephone interviews lasted between 20 and 40 min, were recorded and transcribed. We conducted ongoing thematic content analysis.10 An initial coding framework was developed informed by the evaluation protocol and the literature. Two members of the research team (KW and AB) who have substantial experience in qualitative research and analysis independently read a selection of early transcripts, discussed their coding and further revised the coding frame before reaching consensus. Data were organised and analysed using Nvivo V.10. At 11 interviews, we determined data coding saturation had occurred as no new themes were raised in the later interviews.11

The questionnaire

The questionnaire was informed by the aims of the EdSA and the interviews. Drafts were piloted and discussed with members of the Deanery responsible for the EdSA roll-out who were also Educational Supervisors. The questionnaire consisted of both closed and open free-text questions, thereby generating both quantitative and qualitative data.

All Educational and Named Clinical Supervisors across Wales were eligible to take part. Questionnaires were distributed in paper-based format at three Trainer and Educator Development Days (TEDD) hosted by the Wales Deanery across Wales (Cardiff, Swansea and Wrexham) between October and December 2014. These were free, CPD-accredited training days, for all practising Educational and Named Clinical Supervisors across Wales. Participants received questionnaires within their delegate packs and were asked to place completed questionnaires in collection boxes on their departure.

All questionnaire data were entered into IBM SPSS V.20. We reviewed variable frequencies and used exploratory factor analysis. Open comment data underwent thematic content analysis.10 A random selection of data entries (10%), including qualitative open comments, was checked for accuracy.

Ethical approval was granted by Cardiff University’s School of Postgraduate Medical and Dental Education Research Ethics Committee (03/01/2014).

Results

We present and discuss results from the questionnaires and interviews. As themes are replicated across interviews and questionnaires, we integrate results in our presentation of findings. Box 1 provides an overview of the themes raised in the telephone interviews. These shaped the survey questions.

General themes

Benefits

Formal recognition of role

Consolidation of tripartite agreement

Legitimation of time spent supervising

Potential benefits

Professionalising the role

Enhancing accountability

Ensuring quality

Points of concern

Recognising supporting professional activities’ time in practice

Monitoring the agreement

Assessing quality

Managing administrative workload

Barriers/enabler to implementation of the agreement

Size and staffing specialty

Predictability of work demands

Ability to attend training

Attitude of the Health Board and Deanery

Responses were received from individuals across all seven LEPs in Wales. The number of returns and the response rates for each of the regional events are presented in table 1. Overall, 191 surveys were returned, of these 62% of respondents were male.

Table 1

Response rate at each Trainer and Educator Development Days (TEDD) event

Location of TEDD event	Attendees (n)	% response rate by attendees (n)
Cardiff	218	44% (95)
Swansea	211	31% (66)
Wrexham	107	28% (30)
Totals	536	36% (191)

Eighty-seven per cent of respondents (n=167) indicated having already signed the EdSA. At the time of data collection, 1265 Educational Supervisors had signed the Agreement; our sample thus represents 14% of signed agreements across Wales. Respondents were from a range of specialties; well-represented specialties included anaesthetics (n=29), paediatrics and child health (n=26) and core medical training (n=25). The majority indicated being both an Educational and Named Clinical Supervisor (63%, n=121), while 26% (n=49) indicated solely being an Educational Supervisor and 9% (n=17) solely a Named Clinical Supervisor.

Survey respondents were asked to indicate for how many trainees they acted as Educational Supervisor. The majority reported having one trainee (39%, n=60; table 2).

Table 2

How many trainees do you currently have as an Educational Supervisor?

	No of trainees % (n)
	1	2	3	4	5
Educational Supervisor*	39% (60)	24% (37)	18% (28)	14% (22)	4% (6)

*Percentage is of 153 respondents.

Although 80% indicated having a job planning meeting within the last 12 months, 18% (n=34) had not yet had their job planning meeting and three individuals responded that their last job planning meeting was in excess of 12 months ago. Of those having a job planning meeting within the last 12 months, 76% (n=135) reported having their SPA time recognised within the job plan, while 24% (n=43) reported they had not. Most of those who did have their SPA time recognised reported getting time in work time (78%). Respondents were asked further about this; 66 respondents (35%) provided comment which we summarised (see table 3). Some respondents made more than one comment. Overall, more commented on difficulties.

Table 3

Open comments summary: are you getting supporting professional activities’ (SPA) time in work time?

	n
Generally negative responses (n=65)
Time not captured adequately/less SPA time in clinical time	17
Not supported/not getting the time in job plan/not recognised	14
Clinical commitments take precedence	11
Difficult to ensure when high workload/difficult to fit in	10
Not sure it has been included	5
Need new job plan	4
Where SPA time is scheduled	2
None for teaching	1
Make time up when workload low	1
Generally positive responses (n=11)
Enough time	6
Time allocated	5

Many comments were about the reality of getting the SPA time in practice. Some respondents reported not having their SPA time recognised and not feeling supported:

Trainer role not recognised/not discussed in job plan yet. (w.120.f)

I have discussed but do not feel supported. (w.13.f)

Some individuals indicated not having adequate time recognised within their job plan or being confused and unsure about it:

I’m not sure it has been taken into account. (c.194.f)

Not enough time in job plan given. Confusion about time allocated for Educational vs. Clinical Supervisors. (s.123.f)

A sizeable number of respondents reported clinical workload pressures which take precedence and impinge on allocated SPA time; these respondents found ‘it difficult to fit in’ and commented on completing work at home:

Clinical commitments can take precedence over SPA time. (c.14.f)

Yes in theory but in practice not possible due to current acuity of patients, gaps in junior doctor rotas, pressure of targets (eg, cancer), absent SpRs etc. (c.154.f)

My SPA time is in my job plan and allocated within work time, but in practice all my SPA is done ‘out of work’ time. (c.168.m)

I specifically have an SPA that is in the evenings and what I like about that is nobody can interrupt me and I can get those things done. (19: telephone interviewee)

Impact of the EdSA

To understand the perceived impact of the EdSA, we asked respondents to indicate along a 10-point scale (where one signified negative impact and 10 positive impact) the effect of the Agreement in relation to their roles as educational and named clinical supervisors, the quality of training, time with trainees and patient care. It is noteworthy from table 4 that all means are above 6, indicating a positive impact.

Table 4

Perceived impact of the Educational Supervision Agreement (EdSA) on the role of Educational Supervisors

In your unit or department, how do you think the EdSA will impact on your	Impact % (n) from negative to positive			Mean (range)
	1–4	5–7	8–10
Quality of training? (% of n=176)	1% (2)	66% (117)	32% (57)	6.57 (1–10)
Time with trainees? (% of n=175)	2% (3)	66% (116)	32% (56)	6.57 (3–10)
Work as a supervisor? (% of n=176)	4% (7)	70% (124)	26% (45)	6.38 (1–10)
Quality of patient care? (% of n=176)	6%(10)	76% (133)	19% (33)	6.11 (1–10)

We asked individuals to rate along a five-point scale, from ‘strongly disagree’ to ‘strongly agree’, whether they believed the Agreement enhances accountability for themselves as supervisors, for the LEP and the Deanery. Responses indicate that the great majority of participants believed the Agreement would enhance accountability for all parties (see table 5).

Table 5

Views on whether the Agreement enhances accountability

The Agreement enhances accountability for	Strongly agree	Agree	Neither agree nor disagree	Disagree	Strongly disagree
Me as a supervisor (% of n=183)	15% (28)	72% (132)	9% (16)	2% (3)	2% (4)
The Deanery (% of n=182)	11% (21)	70% (129)	14% (25)	3% (6)	1% (2)
The LEP (HB, Trust) (% of n=183)	12% (22)	60% (109)	20% (37)	6% (11)	2% (3)

LEP, Local Education Provider; HB, Health Board

In telephone interviews, nearly all spoke of the Agreement enhancing accountability (10/11) for all parties. Some interviewees hoped that the Agreement would improve Health Board compliance (7/11), “the important thing is that it forces our Health Board to make sure that we are looking after our junior doctors” (18: telephone interviewee).

When asked, the majority of survey respondents (85%) thought that the Agreement professionalises the role of the Educational Supervisor (table 6). The majority (76%) also thought the Agreement provides leverage to negotiate for the recognition of SPA time in job planning meetings. The comments reported earlier (table 3) provide some insight into what those disagreeing with this statement might be experiencing. From the telephone interviews, all agreed that the EdSA had the potential to professionalise their role as an Educational Supervisor. Some interviewees (4/11) mentioned that it would do so by screening out those who could not or did not want to commit to the supervisory role: “because now you have to train at it and you have to show that you’re continuing to train at it. I think that means that it should be seen in a different light, and it’s not now seen as automatic” (110: telephone interviewee).

Table 6

Responses to statements about potential benefits of the Agreement

	Strongly agree	Agree	Neither agree nor disagree	Disagree	Strongly disagree
The Agreement professionalises the role of the Educational Supervisor (n=188)	19% (35)	66% (124)	11% (21)	2% (4)	2% (4)
The Agreement gives me the leverage to negotiate for the recognition of SPA time in my job planning (n=187)	14% (26)	62% (116)	14% (26)	7% (14)	3% (5)
The Agreement helps to ensure the quality of trainers’ supervisory practice (n=186)	10% (18)	53% (99)	27% (50)	8% (15)	2% (4)
Having my supervisory role recognised formally will support me in the management of difficult trainees (n=184)	10% (18)	46% (84)	34% (62)	7% (13)	4% (7)

SPA, supporting professional activities.

The majority of respondents agreed (63% ‘agree’/‘strongly agree’) that the EdSA would ensure the quality of Educational Supervision, although it is worth noting a sizeable proportion (27%, n=50) ‘neither agreed nor disagreed’ (table 6). The results were similar for whether they thought that having their supervisory role recognised formally would help in the management of difficult trainees (table 6).

The responses to another set of statements are displayed in table 7. These report views on factors influencing the implementation of the Agreement. All telephone interviewees suggested that successful implementation of the Agreement would vary by specialty. This perspective was not fully supported by the wider questionnaire data. Fifty-six per cent (n=99) of respondents ‘agreed’/‘strongly agreed’ implementation would be easier in some specialties than others although 40% (n=72) ‘neither agreed nor disagreed’. However, when asked if implementation of the EdSA within their own specialty would be difficult, only 17% (n=32) of respondents ‘agreed’/‘strongly agreed’.

Table 7

Responses to statements related to the implementation of the Agreement

	Strongly agree	Agree	Neither agree nor disagree	Disagree	Strongly disagree
Implementation of the Agreement is easier in some specialties than others (n=178)	10% (17)	46% (82)	40% (72)	3% (6)	1% (1)
Extending the Agreement to Named Clinical supervisors will increase communication between educational and clinical supervisors (n=184)	5% (9)	42% (77)	41% (75)	11% (20)	2% (3)
I am confident that the Agreement has enough ‘teeth’ (n=186)	2% (3)	27% (50)	48% (90)	19% (36)	4% (7)
The Agreement is a ‘tick-box’ exercise (n=184)	7% (13)	19% (35)	34% (63)	33% (61)	7% (12)
It is difficult to implement the Agreement in my specialty (n=184)	2% (4)	15% (28)	30% (55)	44% (81)	9% (16)

All interviewees were positive about the inclusion of Named Clinical Supervisors in theory: “it would be brilliant because…we could actually get together and talk about it beforehand and organise [trainees’] activity better” (111: telephone interviewee). From the questionnaire data, 47% (n=86) either ‘agreed’ or ‘strongly agreed’ that extending the EdSA to Named Clinical Supervisors would increase communication between the Educational and Named clinical supervisor, although almost as many ‘neither agreed nor disagreed’ (41%, n=75).

Data from our telephone interviews suggested that the Agreement was a ‘tick-box’ exercise (8/11), “to me it has become a bit of a paper exercise, meaningless” (17: telephone interviewee). To explore whether this was a view shared more widely, we included a statement about this within the survey. Results from respondents suggested that this was not the broader view, as 40% (n=73) ‘disagreed’/‘disagreed strongly’ and a further 34% (n=63) ‘neither agreed nor disagreed’ (see table 7).

Respondents were also asked to indicate whether they were confident that the Agreement had enough ‘teeth’ to ensure a high standard of trainee supervisory practice. Many were undecided (48%; n=90) (table 7). Interestingly, among the interviewees the issue of monitoring and a lack of trust was raised (6/11); some interviewees felt that increasing the monitoring of Educational Supervisors indicated lack of trust. However, from the questionnaire data, the idea of monitoring (policing compliance) was welcomed by many. The notion of the Agreement having enough ‘teeth’ was also alluded to within the ‘free-text’ comments. This view was further supported by 62% (n=114) of respondents who indicated that they did not think or were not sure there was sufficient information available on how EdSA would be enforced. Respondents commented on a lack of information, concerns about the ability of the Agreement to sift out poor trainers and whether it would be properly enforced. Telephone interviews and open comments from questionnaires revealed robust and no nonsense approaches to upholding quality and accountability of Educational Supervisors and LEPs:

Know what penalties we must/will have if can’t fulfil responsibilities for training. (c.170.m)

Publish names of those trainers who have not signed Agreement, and remove SPA time from them. (w.122.f)

The accountability of programme directors and Educational Supervisors should be cross-checked regularly by talking to their colleagues. (c.122.m)

It will need policing and feedback and there need to be instances where people are being taken off their in Educational Supervision because they don’t fulfil their role or aren’t able to fulfil their role. Unless you do that it becomes a paper exercise and toothless. (14: telephone interviewee)

A further set of statements were presented to participants. The results are presented in table 8. The majority of respondents, 82% (n=153), ‘strongly agreed/agreed’ that they would like to have more feedback on their role as a supervisor; this is in line with suggestions by interviewees (5/11). Some interviewees said that the Agreement would enhance their access to training but raised concerns about their ability to attend sufficient training. To explore these concerns more widely, we asked respondents whether the Agreement would ensure they had better access to educational training for their role: of the sample, 71% (n=131) ‘strongly agreed’/‘agreed’. Furthermore, 48% (n=87) of respondents reported they would no longer have difficulty in negotiating time away from work to attend educational training sessions (as indicated by disagreement with the statement).

Table 8

Responses to statements about educational aspects of the Agreement and supervisory practice

	Strongly agree	Agree	Neither agree nor disagree	Disagree	Strongly disagree
I would like more feedback on my roles as a supervisor (n=186)	15% (28)	67% (125)	15% (27)	3% (5)	1% (1)
The Agreement will ensure I have better access to educational training for the role (n=185)	12% (22)	60% (109)	24% (44)	4% (7)	2% (3)
I am comfortable that I will be able to demonstrate I have undertaken the required CPD (n=186)	16% (30)	54% (101)	16% (30)	11% (21)	2% (4)
I will still have difficulty in being able to negotiate time away from work to attend educational training sessions (n=182)	7% (12)	26% (47)	19% (35)	41% (75)	7% (12)
I think the CPD requirements deter ‘good’ trainers from becoming supervisors (n=186)	4% (8)	19% (35)	25% (47)	44% (81)	8% (15)
I can complete all my administrative supervisory responsibilities in work time (n=183)	0% (0)	14% (26)	23% (43)	49% (90)	13% (24)
I feel that the Agreement undermines my current supervisory practice (n=184)	1% (2)	4% (8)	33% (60)	51% (93)	11% (21)

CPD, continuing professional development.

A concern about workload identified in the telephone interviews (6/11), was supported by questionnaire data. When respondents were asked to rate whether they completed all their administrative responsibilities within work time, no participants ticked ‘strongly agree’ and 62% (n=114) disagreed (table 8).

A key element of EdSA is CPD (a minimum of 8 hours of CPD per year). The Wales Deanery provides training opportunities, and in general, interviewees were positive about the role of the Deanery in helping them to fulfil the requirements of the Agreement:

I haven’t come across something that the Deanery wouldn’t do for us. (111: telephone interviewee)

At the moment updates have all been provided by the Deanery. We get frequent updates locally in the postgrad centre, and they have all the all-Wales one …which were great. (18: telephone interviewee)

Seventy per cent of questionnaire respondents indicated they felt comfortable in being able to demonstrate required levels of CPD.

One of the primary aims of the Agreement is to raise standards of postgraduate training and education and to help support good supervisory practice. Despite an interview, observation that the Agreement could undermine current Educational Supervision practice this view was not supported by the wider questionnaire data: 62% (n=114) indicated disagreed that their supervisory practice was undermined (see table 8).

With a view to advancing the Agreement, respondents were asked to comment on what they thought were barriers to implementation. We present a summary in table 9.

Table 9

What do you think are the barriers to the implementation of the Educational Supervision Agreement?

General themes	n
Time	35
Pressure on clinical work/service commitment	27
Inadequate job planning	19
Process of the Agreement/sign up	18
Buy-in (taking up of commitments)	14
Ability to enforce	9
Funding	6
Recognition of role	5
Lack of trainees	1

Respondents’ suggestions about what might hamper implementation were largely pragmatic. These include extant time and clinical workload pressures which resulted in concern that existing commitments would inevitably lead to a squeeze on individuals’ ability to be a supervisor:

Time, multiplicity of roles. The best doctors tend to deliver the most roles. It is actually likely that very busy clinicians/ managers/ academics will necessarily have to drop supervision. (s.129.f)

Priorities of service delivery and inadequate job planning were further perceived barriers:

Employers still consider service delivery more important than training educational CPD needs of doctors. (s.11.f)

I feel this a very significant first formal step in the right direction. Will take some time before this is universally reflected in people’s job plans as appraisal information. (s.124.f)

Furthermore, issues around upholding quality were noted, for instance with regard to CPD training and the need for enforcement of the Agreement:

Relevance of courses & CPD which are required annually & avoidance of repetition. (w.15.m)

No ‘teeth’ to make radical change needed within health boards. (c.119.f)

In an effort to make sense of the complexity of responses, we conducted a factor analysis. Following Field,12 we first confirmed that a high number of responses to statements correlated (13/18 correlated), checked that the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy was over the recommended value of 0.5 (KMO=0.87) and that all KMO values for individual items were greater than 0.5 (all were >0.87). Bartlett’s test of sphericity was significant (χ2(171)=1133.72, p<0.001) which indicated that correlations between items were sufficiently large for principal component analysis. Five components had eigenvalues over Kaiser’s criterion of 1 and in combination explained 56.3% of the variance. However, the scree plot showed a levelling off of eigenvalues after factor 3. Given this and the relatively small sample size, three factors were retained in the final analysis. Table 10 displays the factor loadings after varimax rotation. For our analysis, we have used a cut-off of 0.30.13 We described the factors as primarily being about: professionalising the role of Educational Supervisors (factor 1); supporting supervisory practice through training and feedback (factor 2) and implementation of the Agreement (factor 3).

Table 10

Summary of exploratory factor analysis results for the Educational Supervision Agreement questionnaire (n=191)

	Rotated factor loadings
	Factor 1	Factor 2	Factor 3
Item
The Agreement helps to ensure the quality of trainers' supervisory practice	0.773
The Agreement professionalises the role of the Educational Supervisor	0.770
Having my supervisory role recognised formally will support me in the management of difficult trainees	0.762
The Agreement enhances accountability for me as a supervisor	0.747
The Agreement enhances accountability for the Deanery	0.698
The Agreement will ensure I have better access to educational training for the role	0.682
The Agreement enhances accountability for: the LEP (LHB, Trust)	0.657	0.302
I am confident that the Agreement has enough 'teeth' to ensure a high standard of trainee supervisory practice	0.645		0.304
Extending the Agreement to Named Clinical Supervisors will increase communication between Educational and Named Clinical Supervisors	0.588
I am comfortable that I will be able to demonstrate I have undertaken the required CPD (8 hours minimum per year, of which at least 4 hours are accredited		0.702
I would like more feedback on my roles as a supervisor		0.641
I think the CPD requirements deter ‘good’ trainers from becoming supervisors		0.619
I can complete all my administrative supervisory responsibilities in work time			0.731
It is difficult to implement the Agreement in my specialty	0.334		0.665
I will still have difficulty in being able to negotiate time away from work to attend educational training sessions		0.365	0.620
I feel that the Agreement undermines my current supervisory practice		0.310	0.415
Eigenvalues	6.25	1.95	1.33
% of variance	32.90	10.25	6.99
Α	0.84	0.53	0.57

CPD, continuing professional development; LEP, Local Education Provider.

Figures in bold denote those items above 0.30.

Discussion

This paper presents findings from a formative evaluation of the first year of the roll-out the EdSA. The results can help to shape development of the Agreement to fit with trainers’ expectations and inform subsequent evaluation of EdSA. Given the recent GMC trainer standards, this work is timely and can inform others working to meet requirements. The Agreement can be understood for the most part, as being well received by the sample we surveyed. The Agreement successfully addresses the objective of supporting the GMC’s implementation plan and goes some way to ensuring commitment from each of the parties to the tenets of the Agreement which are commensurate with the GMC’s intentions.

A strength of the study was the way in which data from the interview was triangulated in the questionnaire. We have revealed a level of similarity in opinion across those in our sample as well as highlighting some difference of opinion (eg, adequacy of SPA provision, hours of CPD, extent to which such an Agreement should be enforced). Findings from the explorative factor analysis help us to identify and simplify the main purposes of the Agreement (professionalising the role and providing support) and the challenge (implementation). These key elements are commensurate with the aims and objectives of the GMC implementation plan.

Our results should be interpreted in context however, acknowledging that these data reflect a snapshot in time from a relatively small self-selected sample of Educational Supervisors attending a training event during the introductory period of the Agreement. Self-selection operated at two levels: first our sample included only those individuals who chose to attend the CPD event; second, we only have data from those who chose to complete and return the questionnaire. In the future, to enhance response rate, we will seek to build time into the programme for questionnaire completion. We also note that the group of signed-up Educational Supervisors was not static, increasing in number as the Agreement rolled out. This makes cohort comparison problematic. However, we can report that these findings are commensurate with those reported from the GMC Trainer Survey14 for educational and Named cClinical Supervisors in areas such as recognition for the role, SPA time and support for CPD. Other studies looking at the effectiveness of Educational Supervisors acknowledge practice variability15 and low educator appraisal16 provide argument for professionalising the role which is part of the purpose of the EdSA. We recognise that in respect of the EdSA, longer-term follow-up is needed.

The findings suggested that the EdSA will fulfil the majority of its aims or objectives: the Agreement is seen as enhancing communication and accountability between the Deanery, the LEP and the Educational Supervisor; and professionalising the role of Educational Supervisors through recognition of the role, SPA time tariffs in job plans and annual appraisal of the trainer role. Results show that there is an expectation that the Agreement will promote and enhance standards of postgraduate medical education and training in Wales. Qualitative descriptive data provided by respondents through open comments and telephone interviews shed light on the positive expectations of the Agreement to deliver on its principles, providing the authority to support and recognise the role of Educational Supervisors in practice. However, while respondents are proponents of safe, high quality patient care, the impact of the Agreement on patient care cannot yet be judged and needs further evaluation.

Conclusion

The evaluation shows the positive expectations of the Agreement to deliver on its principles, providing the quasi-regulatory authority to support, recognise and monitor the role of Educational Supervisors in practice. These findings provide positive reinforcement for the development and roll-out of EdSA and its potential extension to Named Clinical Supervisors.