Supervised learning events in the foundation programme: a UK-wide narrative interview study.

OBJECTIVES: To explore Foundation trainees' and trainers' understandings and experiences of supervised learning events (SLEs), compared with workplace-based assessments (WPBAs), and their suggestions for developing SLEs.
DESIGN: A narrative interview study based on 55 individual and 19 group interviews.
SETTING: UK-wide study across three sites in England, Scotland and Wales. PARTICIPANTS: Using maximum-variation sampling, 70 Foundation trainees and 40 trainers were recruited, shared their understandings and experiences of SLEs/WPBAs and made recommendations for future practice.
METHODS: Data were analysed using thematic and discourse analysis and narrative analysis of one exemplar personal incident narrative.
RESULTS: While participants volunteered understandings of SLEs as learning and assessment, they typically volunteered understandings of WPBAs as assessment. Trainers seemed more likely to describe SLEs as assessment and a 'safety net' to protect patients than trainees. We identified 333 personal incident narratives in our data (221 SLEs; 72 WPBAs). There was perceived variability in the conduct of SLEs/WPBAs in terms of their initiation, tools used, feedback and finalisation. Numerous factors at individual, interpersonal, cultural and technological levels were thought to facilitate/hinder learning. SLE narratives were more likely to be evaluated positively than WPBA narratives overall and by trainees specifically. Participants made sense of their experiences, emotions, identities and relationships through their narratives. They provided numerous suggestions for improving SLEs at individual, interpersonal, cultural and technological levels.
CONCLUSIONS: Our findings provide tentative support for the shift to formative learning with the introduction of SLEs, albeit raising concerns around trainees' and trainers' understandings about SLEs. We identify five key educational recommendations from our study. Additional research is now needed to explore further the complexities around SLEs within workplace learning. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

This is the first study to explore Foundation Programme trainee and trainers’ understandings and experiences of supervised learning events (SLEs) (compared with workplace-based assessments (WPBAs)).

The large number of narratives collected across England, Scotland and Wales enhances the transferability of our findings to other UK locations.

We had relatively low numbers of general practitioners (GP) and nurse trainers and trainees with GP and nurse trainer SLE/WPBA experiences so our findings are most relevant to SLEs conducted by hospital doctors.

Introduction

If you are a clinical educator or trainee doctor in today's National Health Service (NHS) in the UK, you will inevitably have participated in a ‘supervised learning event’ (SLE).1 SLEs review the personal development of trainee doctors, with an emphasis on patient safety.1 They were introduced into the UK Foundation Programme (UKFP) in 2012. SLEs specifically address concerns raised in the Collins report2 and previously published literature about assessment within the UKFP3; that trainees and trainers perceived workplace-based assessments (WPBAs) as excessive, onerous and therefore unvalued. Drawing on the same tools utilised within WPBAs (eg, Mini Clinical Evaluation Exercise: Mini-CEX, Direct Observation of Procedurals Skills: DOPS and Case-Based Discussion: CBD), SLEs are designed to: (1) highlight achievements and areas of excellence; (2) provide immediate feedback and suggest areas for further development; and (3) demonstrate engagement in the educational process (see ref. 1, pp. 57–59 for more details). Trainees are encouraged to complete a minimum number of SLEs spread evenly throughout their placements, with different trainers and covering diverse acute and long-term clinical problems.1 In this way, SLEs aim to facilitate a strong formative component of trainee doctors’ assessment.

Rather than indicating what a learner can/cannot do or knows (ie, summative assessment), formative assessments indicate the ‘gap’ between the learner's actual level of performance and the required standard, providing an indication of how performance could be improved to reach the required standard. Therefore, SLEs are designed to enable the provision of timely feedback about the effectiveness of care and the trainee's interactions with others, with a focus on the trainee's performance and development, which may identify areas of weakness requiring support and reflection. Thus, SLEs have the potential to be more meaningful for learning, motivating learners to ‘mastery goals’ such as understanding, rather than ‘performance goals’ like passing an examination.4 5

However, SLEs also have a summative role within the UKFP. Currently, evidence of SLEs must be included in every Foundation doctor's e-Portfolio, which in turn is a method of assessment of the Foundation doctor's success in achieving the outcomes described in the curriculum, and which educational supervisors use in the end of placement report. Thus, SLEs can be viewed broadly as information gathering activities that aim to benefit the quality of trainee learning, as well as monitoring their engagement with feedback for accountability purposes.

Effectiveness of the assessment tools

Previous research has examined the effectiveness of the assessment tools (eg, DOPS, Mini-CEX, CBD),6–8 drawing on van der Vleuten's utility equation9: educational impact × validity × reliability × cost-effectiveness × acceptability. Previous research has provided mixed results regarding their efficaciousness in terms of acceptability, reliability and validity: (1) the acceptability of WPBAs to trainees and trainers varies widely2 8 10–13; (2) reliability for the tools is frequently suboptimal14; and (3) the Mini-CEX and the ‘clinical encounter card’ appears to have high criterion validity in terms of strong and significant correlations with other assessment instruments.15 However, the cost-effectiveness and educational impact of the tools have been largely neglected. Indeed, few published articles have explored the educational impact of WPBA tools and there is therefore little evidence that they lead to improvements in performance.3 15

Effectiveness of WPBAs and SLEs

Research has also examined the effectiveness of WPBAs, albeit scant. What evidence there is suggests that WPBAs are reasonably ineffective, attributed to issues such as the suboptimal use of the tools for feedback.16 17 Some research suggests that the rating scales often utilised within the tools such as the Mini-CEX introduce artificiality into the assessment, concluding that open-ended comments may be more valuable as assessors are able to provide feedback in more ‘authentic’ terms.18 Additionally, there are issues with suboptimal learners being less likely to seek feedback.19 Outcomes such as learning, transfer of skills to new situations, or improved patient care are relatively unstudied, and when they are, conclusions drawn are limited due to weak study designs.

SLEs were introduced in 2012 to address these shortcomings but, so far, there has been no evidence to evaluate their success in doing so. Given that SLEs comprise similar tools to those used within the WPBAs but with fewer assessments and explicit formative goals, it is important that aspects such as acceptability and the educational utility of SLEs as a form of feedback are explored as a matter of priority. Given that acceptability and educational impact inter-relates with how trainees and trainers make sense of their experiences, emotions, identities and relationships, we felt it crucial to employ a narrative interview approach. We were therefore commissioned by the Academy of Medical Royal Colleges (AOMRC) to undertake an independent evaluation of the impact of the transition from WPBAs to SLEs.

Aims and research questions

This study is the first exploration of SLEs within the UKFP and aims to answer four research questions (RQ). (RQ1) What are participants’ understandings of SLEs and WPBAs and how do they differ between trainees and trainers? (RQ2) What are participants’ experiences of SLEs and WPBAs and how do they differ between trainees and trainers? (RQ3) How do participants make sense of their experiences through narrative? (RQ4) What are participants’ suggestions for how SLEs should be developed?

Methods

Study design

We conducted a qualitative study using group and individual interviews to elicit trainees’ and trainers’ understandings and personal incident narratives (PINs) of their experiences. We employed focus groups wherever possible because they can lead to richer data due to group dynamics (eg, synergism), but individual interviews were also utilised because of the difficulties in getting groups of clinicians together.20 Our study draws on social constructionist epistemology suggesting that there are multiple interpretations of reality and ways of knowing.21 We consider the individual and sociorelational aspects of stories of experience including how participants make sense of their SLE/WPBA experiences through narrative and how they share those stories and in doing so construct identities and trainee–trainer relationships.22

Sampling and recruitment

Following Deanery and Medical School authorisation, ethical approval was established at three sites in England, Scotland and Wales. Using maximum-variation sampling to attempt to obtain a greater range of understandings and experiences, we recruited Foundation doctors from year 1 to year 2 of the 2-year programme (F1s and F2s) with training experiences in hospital and general practice settings. We also recruited trainers across hospital and general practice settings, including clinical and educational supervisors and members of placement supervision groups such as specialist registrars, consultants and nurses. Using advice from our clinical reference group (see Acknowledgements section), we employed multiple recruitment strategies to maximise participation: (1) email; (2) physical notice-boards; (3) leaflets in strategic places (eg, medical libraries, common rooms); (4) snowballing through participant and trainee organisations (eg, BMA junior doctor committee); (5) social networking (eg, Facebook); and (6) face-to-face during formal curricula. We interviewed 110 participants (34 F1s, 36 F2s, and 40 trainers: see table 1 for participants’ characteristics). This overall sample and subsamples far exceeded the minimum sample size of 30 advocated by some qualitative scholars.23 Furthermore, we considered this to be the maximum number of participants we could feasibly interview given the time and financial constraints of our grant, another pragmatic consideration discussed by qualitative researchers.23

Table 1

Participant characteristics by group

Characteristic	Trainees (N=70)*	Trainers (N=40)*
Age
20–30	65 (93%)	2 (5%)
31–40	2 (3%)	13 (32%)
41+	3 (4%)	24 (61%)
Gender
Male	31 (44%)	24 (60%)
Female	39 (56%)	16 (40%)
Ethnicity
White	57 (81%)	37 (93%)
Non-white	13 (19%)	3 (8%)
Language
English	60 (86%)	36 (90%)
English as second language	10 (14%)	3 (8%)
Trainers’ years since graduation
0–10	–	8 (20%)
11–20	–	15 (38%)
21+	–	16 (41%)
Trainers’ years of PGME experience
0–10	–	26 (64%)
11–20	–	9 (23%)
21+	–	4 (11%)
Trainers’ specialties
Hospital (medical)†	–	16 (40%)
Hospital (surgical)	–	5 (13%)
Hospital (services)	–	8 (20%)
General practice	–	5 (13%)
Nurse	–	4 (10%)
Number of SLEs conducted
Median	8	6
Range	3–25	0–40
Had experience with tools as SLEs?‡
DOPS	42 (60%)	16 (40%)
Mini-CEX	46 (66%)	25 (63%)
CBD	45 (64%)	26 (65%)
DCT	10 (14%)	6 (15%)
Number of WPBA conducted
Median	19.5	30
Range	8–28	0–40
Had experience with tools as WPBAs?‡
DOPS	24 (34%)	20 (50%)
Mini-CEX	24 (34%)	30 (75%)
CBD	24 (34%)	30 (75%)

*These figures are rounded up to zero decimal places so may not always add up to 100%.

†Medical specialties included neurology, gastroenterology, rheumatology, anaesthesiology and psychiatry, surgical specialties included ophthalmology and orthopaedics, and services specialties included infectious diseases and dermatology.

‡These figures represent a free-text question asking participants to outline which tools they had used so numbers are likely to be under-estimates.

CBD, Case-based Discussion; DCT, Developing the Clinical Teacher; DOPS, Direct Observation of Procedural Skills; Mini-CEX, Mini Clinical Evaluation Exercise; SLEs, supervised learning events; WPBAs, workplace-based assessments.

Data collection

We conducted 55 individual and 19 group interviews (34 individual and 3 group interviews with trainers; 21 individual and 16 group interviews with trainees). All focus groups bar two were homogenous in terms of the type of study participant (ie, trainer or year-specific trainee groups). Interviews were recorded, transcribed and anonymised (mean length of focus groups 45:43 minutes:seconds (range 31:50–63:47) and individual interviews 36:38 minutes:seconds (range 17:37–69:50): total data around 46 h). Participants completed a personal details questionnaire, comprising demographic and education-related details enabling classification of sample characteristics by group, site and entire study. An interview schedule ensured consistency across multiple interviewers. Interviews began by exploring trainees’ and trainers’ understandings of SLEs and WPBAs. Using narrative interviewing, we encouraged participants to articulate their PINs of SLEs and WPBAs by asking a series of prompts around their narratives: Can you tell me about a memorable SLE/WPBA? What happened? Who was involved? Where did it happen? What did you do and why? How did you feel? What was the impact of that SLE/WPBA for trainee learning? We encouraged participants to narrate their SLE/WPBA experiences so that their views were grounded in actual lived experiences and we could understand how they made sense of those experiences, identities and relationships. Interviews continued until participants felt they had shared their experiences sufficiently. We then asked participants how they thought SLEs could be improved.

Data analysis

We employed multiple and complementary forms of analyses as per previously published research24: thematic and discourse analyses and in-depth narrative analysis of one exemplar PIN. We began with a primary level thematic analysis of the data called framework analysis (involving data familiarisation, thematic framework identification, indexing, charting, mapping and interpretation) to determine content-related and process-related themes (ie, what participants said and how they said it respectively).25 The identification and coding of process-related themes was akin to discourse analysis—that is, analysis of language-in-use in social interaction.26 We employed qualitative data analysis software (Atlas-Ti, V.7.0) to facilitate multianalyst coding of the data. This allowed us to explore patterns across our data qualitatively, such as possible differences in understandings between trainees and trainers, and sometimes quantitatively such as exploring differences in trainee and trainers’ SLE/WPBA experiences using descriptive (eg, frequencies and percentages) and univariate statistics (eg, χ2 tests). Finally, we present an in-depth narrative analysis of one exemplar PIN in this paper to illustrate how one trainee made sense of her workplace learning experiences, identities and relationships.27 We establish credibility in our study by describing our analytic methods, involving multiple data analysts and using illustrative quotes.28 Finally, we establish transferability through our inclusion of a large number of narratives from a diverse sample of trainees and trainers across three different UK countries.28

Results

Our thematic framework analysis identified seven themes in the data: one theme relating to our first research question (understandings of SLEs/WPBAs); four themes relating to our second research question (contextual codes for the PINs, processes of SLEs/WPBAs, factors facilitating learning in SLEs/WPBAs, and factors inhibiting learning); one theme relating to our third research question (how participants narrate their experiences); and one theme relating to our fourth research question (suggestions for improving SLEs).

RQ1: what are participants’ understandings of SLEs and WPBAs and how do they differ between trainees and trainers?

Many trainees and trainers admitted to not knowing what SLEs were, and this uncertainty was emphasised through hesitations (errs and ums), pauses, hedges (eg, ‘I guess’) and laughter. Some participants (eg, those new to training or new to the UK) were also unsure what WPBAs were but most seemed better able to explain WPBAs than SLEs.

Many trainers and F2s suggested that SLEs and WPBAs were conceptually and operationally the same. However, others did perceive them to be conceptually different, with SLEs having formative and WPBAs having summative aims. While participants volunteered a range of understandings for SLEs (eg, as learning, as assessment), they almost exclusively volunteered understandings of WPBAs as assessment (see table 2).

Table 2

Participants’ understandings of SLEs/WPBAs

Understandings	Description	Illustrative quote
SLE/WPBA as unknown	Understanding unclear	“I didn't really understand what they [SLEs] meant ((laughs)) to be honest erm” (Female F1, site 3)
SLE/WPBA as summative tool	SLEs/WPBAs’ purpose is to assess trainees’ abilities, and give ‘pass/fail’ results	“WPBA is more of a case of they've performed a task and have they understood what that task is or is it something you can sign off that they're competent to do” (Male Trainer, site 3)
SLE/WPBA as tick box exercise	SLEs/WPBAs demonstrate basic requirements are met with little educational value	“It's still tempting for an assessor to say “I'm really busy, we'll do a WPBA and we'll just tick whether it was excellent or not”” (Female F2, site 1)
SLE/WPBA as safety net	SLEs/WPBAs’ purpose is to ensure that trainees who struggle are identified	“I initiated a Mini-CEX [Mini Clinical Evaluation Exercise] in a clinic to try and get some ideas about why the registrar was getting these complaints…what it allowed me to do was to try and broach the subject of the complaints with the registrar but in a training environment” (Male Trainer, site 2)
SLE as formative tool	SLEs are a tool for developing, rather than assessing, trainees	“It is a learning event and you should be giving them feedback on the process there and then, and that should be used as a learning tool” (Female Trainer, site 2)
SLE as a formalisation process	SLEs open up a legitimate route for trainees to ask seniors to engage in their learning, ensuring that training processes occur within the workplace	“I think that's just formalising what we do normally, ward round teaching it's formalising that but also making it more time consuming because you have to write it all down” (Female Trainer, site 1)
SLE as individual assessments	An opportunity to assess competencies and knowledge at a single time-point	“Problem is it's just, the supervised learning events is just a one off thing, it's just like a little snapshot” (Female F1, site 2)
SLE as formal progression	SLEs demonstrate trainee progression, evidencing skill acquisition over time	“My understanding of the SLEs are they are opportunities to um, view and um, assess a trainee's um, progress, whether that's examination skills, whether that's clinical reasoning… ” (Male Trainer, site 3)
SLE as developmental process	SLEs provide trainees with an opportunity for holistic development. Unlike ‘formal progression’, the focus is on trainees’ personal perceptions of development	“she [consultant] was there all the time, she, when she wasn't there, you know, the first thing she said to me when she got back onto the ward on Monday morning, was “What does the latest gas show? What are you gonna do…? Are you gonna treat this…?”, so, so the whole thing was just this massive learning experience” (Female F2, site 3)
SLE as engagement opportunity	SLEs are an opportunity for trainers and trainees to have one-to-one time that may not otherwise happen	“the fact it's compulsory…that gives you something you can say to seniors “look, I need to do this, I'm sorry, but I have to do it”…it does mean you sit down and hopefully spend half an hour talking in a bit more detail… it does mean you've got an excuse to have that face-to-face…” (Male F2, site 2)
WPBA as a gut feeling	WPBAs are poorly defined and therefore assessing whether a trainee had passed is a ‘judgement call’	“because also like last year, somebody would give you all these meets or meets it more, but it's such a subjective thing” (Female F2, site 1)
Understandings linked with emotion	Understandings articulated with emotion talk	“I think it's six of one half-dozen of the other, I am not somebody who excels at that kind of assessment… errm and I get very anxious, I get very uptight and I don't shine… and it feeds into all my anxieties and insecurities about myself… and I think that probably skews my perception of them [SLE/WPBAs]…” (Female F2, site 3)

Mini-CEX, Mini Clinical Evaluation Exercise; SLEs, supervised learning events; WPBAs, workplace-based assessments.

Trainers seemed to volunteer understandings of SLEs as assessment and as a ‘safety net’ (ie, a diagnostic tool to help identify trainees who were ‘struggling’) more than trainees. However, only trainers defined WPBAs in this way. Another apparent difference we identified, was the extent of emotional talk (eg, negative emotion talk) employed by trainees when attempting to define SLEs and WPBAs. Trainees sometimes felt the formative focus relieved the pressure to perform and reduced anxieties.

RQ2: what are participants’ experiences of SLEs and WPBAs and how do they differ between trainees and trainers?

We outline key findings associated with four of our fragmentary themes (ie, themes that cross-cut all narratives) here: one contextual theme (covering the timing, location of SLEs/WPBAs, identity of trainer, type of tool and participant evaluation including the differences between trainees’ and trainers’ evaluations), and three conceptual themes all pertaining to participants’ lived experiences of SLEs/WPBAs (processes of SLEs and WPBAs; and factors facilitating and inhibiting learning within SLEs/WPBAs). It is important to indicate that narratives typically contain numerous elements including the narrator's commentary on their experience: also known as the ‘evaluation’.29 As per the interpretive approach, the analysts coded whole narratives to these codes depending on what participants said and how they said it. For example, narratives including mostly negative emotional talk (eg, ‘it was quite alarming’) would be coded to ‘negative evaluation’ and narratives including mostly positive emotional talk (eg, ‘it's nice to have nice things said about you’) would be coded to ‘positive evaluation’.

The context of SLE and WPBA narratives

We identified 333 narratives in the data (221 SLEs, 72 WPBAs; see table 3). Most SLEs and WPBAs narrated took place in hospital settings (n=253) and involved F1 doctors (n=185). Trainers within the incidents were usually hospital-based doctors (n=262), although some non-medical specialists (eg, nurses) also acted as trainers (n=15). CBD, DOPS and Mini-CEX were the most common tools narrated (totalling n=276). Finally, SLEs were overall more likely to be evaluated by the narrators positively (58%) than WPBA narratives (39%), and were less likely to be evaluated negatively by the narrators (13%) compared with WPBAs (22%: χ2=5.344, df=1, p=0.021). The descriptive statistics presented in table 3 illustrate more similarities than differences between trainees and trainers. Although trainees seemed to narrate more SLE experiences with positive evaluations (62%) compared with trainers (46%: χ2=0.000, df=1, p=1.000) and more WPBAs with negative evaluations (26%) compared with trainers (18%: χ2=0.237, df=1, p=0.627), these relationships were not statistically significant. However, trainees were more likely to narrate their SLE experiences positively (62%) compared with WPBAs (36%: χ2=5.148, df=1, p=0.023).

Table 3

Overview of personal incident narratives of supervised learning events and workplace-based assessments by participants: Frequencies (%)

	Overall*333	SLEs†						WPBAs†
		Total221		Trainee167		Trainer54		Total72		Trainee39		Trainer33
Where
Hospital	253	170	(76)	123	73)	47	(84)	58	(81)	31	(79)	27	(82)
GP Practice	20	17	(8)	12	(7)	5	(9)	2	(3)	0	(0)	2	(6)
Other	3	1	(0)	1	(1)	0	(0)	1	(1)	0	(0)	1	(3)
When
FY1	185	130	(59)	104	(62)	26	(48)	50	(69)	39	(100)	11	(33)
FY2	84	76	(34)	62	(37)	14	(26)	5	(7)	0	(0)	5	(15)
ST	10	4	(2)	0	(0)	4	(7)	2	(3)	0	(0)	2	(6)
Who (trainer)
Hospital Dr	262	176	(79)	139	(83)	37	(67)	57	(79)	29	(74)	28	(85)
Community Dr	26	21	(9)	12	(8)	9	(16)	3	(4)	0	(0)	3	(9)
Non-medic	15	11	(5)	4	(2)	7	(13)	3	(4)	2	(5)	1	(3)
No trainer	2	0	(0)	0	(0)	0	(0)	2	(3)	2	(5)	0	(0)
Which tool
CBD	106	78	(34)	59	(34)	19	(35)	16	(22)	5	(13)	11	(32)
Mini-CEX	85	61	(27)	47	(27)	14	(25)	17	(23)	9	(23)	8	(24)
DOPS	85	57	(25)	46	(27)	11	(20)	20	(27)	13	(33)	7	(21)
DCT	28	12	(5)	9	(5)	3	(5)	13	(18)	11	(28)	2	(6)
Other (eg, MSF)	6	2	(1)	1	(1)	1	(2)	2	(3)	0	(0)	2	(6)
Evaluation‡
Positive	173	128	(58)	103	(62)	25	(46)	28	(39)	14	(36)	14	(42)
Negative	56	29	(13)	23	(14)	6	(11)	16	(22)	10	(26)	6	(18)
Neutral	36	28	(13)	16	(10)	12	(22)	8	(11)	3	(8)	5	(15)
Contradictory	20	12	(5)	7	(4)	5	(9)	6	(8)	4	(10)	2	(6)

*Note that frequencies for SLEs and WPBAs across rows do not add up to the overall total because unclear narratives are excluded.

†Percentages are calculated within each group/column—that is, total, trainee, trainer. These also fall short of 100% because ‘unclear’ narratives are excluded.

‡As per the interpretive approach, analysts coded whole narratives to these codes depending on what participants said and how they said it (eg, narratives including mostly negative emotional talk eg, “it was quite alarming” would be coded to ‘negative evaluation’).

CBD, Case-based Discussion; DCT, Developing the Clinical Teacher; DOPS, Direct Observation of Procedural Skills; Mini-CEX, Mini Clinical Evaluation Exercise; MSF, Multi-source Feedback; SLEs, supervised learning events; WPBAs, workplace-based assessments.

Processes of SLEs and WPBA

SLEs and WPBAs were conducted in diverse ways, in terms of their initiation, tools employed, educational processes used and completion.

Initiating SLEs and WPBAs

WPBAs/SLEs were initiated by different parties, with different motivations and in different contexts. While SLEs should be trainee-initiated, trainers occasionally also initiated them, sometimes near the end of rotations (see table 4). Trainees and trainers described some trainees lacking proactivity to seek opportunities for SLEs/WPBAs. When trainees did initiate them, at times, they strategically chose a trainer they knew. This was sometimes done to enhance the learning experience, choosing someone they felt comfortable with, believed would engage in the process, and/or thought would support them in a positive way. At other times this was done with the intention of having a quick and easy experience where the trainer would just ‘tick the box’. Trainees often described feeling discomfort in asking for SLE/WPBA supervision and were often grateful when trainers initiated them. The initiation also varied in terms of the level of planning and organisation. Occasionally they were planned ahead of time, and this sometimes involved an element of rehearsal (particularly for the developing the clinical teacher tool: DCT). At other times, they were ad hoc, with opportunistic clinical encounters recognised as an opportunity for an SLE. Finally, they were sometimes initiated retrospectively, at times, long after the event, particularly when trainees had completed insufficient tools during their placements (see table 4).

Table 4

“I'll actively hunt”

Interviewer	…okay well can you think of any more stories with your SLEs [Supervised Learning Events]* because we've got different types I mean any DOPS [Direct Observation of Procedural Skills] maybe?
Helena	I don't find the DOPS very useful because one of the DOPS like taking blood or putting in a cannula we do that about a hundred times a day and obviously all our trainers know that we can do that and have seen that not sat and watched us put in a venflon but have seen all the venflons in the patients and they know that we put them in
Interviewer	right
Helena	so they don't really take the time to stand and assess and watch us put it in because they've seen people toing and froing with our venflons in their arms so they're like “yeah I'll sign that off no problem I know you can do a venflon”
Interviewer	okay so they're not really watching you they're just taking it on trust
Helena	yeah they can see the outcomes of the procedures that we've done rather than
Interviewer	have you had an SLE like that?
Helena	yeah um like I mean fairly straightforward procedures that we do every day there's not often enough time for trainers to actually stand and watch us do something as basic as taking blood they know we can take blood else we wouldn't be able to survive on the wards ((laughs)) so it's kind of taken for granted that we can do that
Interviewer	so when you got your SLE for that can you just tell me how that happened how did you go about getting the SLE for that?
Helena	um well just in the last job towards the end they always say “how are you doing with all the tick bo- have you got everything you need?” and I was a couple short on DOPS so my clinical fellow said “I obviously known you can do venflons I've sent you to go and [do] them and you've come back and said you've done them on numerous occasions I can easily sign that one off for you”
Interviewer	okay so again they initiated it rather than you yourself is that right in this particular case?
Helena	Yeah it can be both because I'll think “oh deadline coming up I'm a few short of this and this” and I'll actively hunt to- to go and find somebody that needs what I'm missing ((laughs))…

*Although the trainee is repeated asked about a Supervised Learning Event (SLE) experience, she provides a workplace-based assessment (WPBA) experience.

Tools used

Participants talked about the unique aspects of tools, their preferences and their ‘workability’. However, they were sometimes unsure or mistaken about what comprised an SLE/WPBA assessment tool, or conflated tools (eg, CBD with Mini-CEX). Participants discussed the practicalities of various tools, and suggested that some were less workable in certain specialties (eg, DOPS in psychiatry). Interestingly, many participants expressed clear preferences and dislike for certain tools. For example, some clinicians expressed a preference for Mini-CEX over CBD: Mini-CEX allowed them to observe ‘real’ performances of trainees and identify ‘struggling trainees’, whereas CBDs gave trainees opportunities to rehearse thereby masking potential difficulties. Other trainees expressed a preference for CBD over DOPS: CBDs led to ‘real learning’, whereas DOPS were ‘tick-box exercises’, simply signing off already-competent procedures.

Feedback

The educational activities highlighted included: (1) trainers’ observation of the trainee; (2) didactic teaching of knowledge/skills; (3) scaffolding trainees’ learning through strategic questioning; and (4) feedback (most commonly verbal feedback during the event and written feedback afterwards). Feedback quality was thought to vary. Positive experiences included personal, meaningful and constructive feedback for learning. Negative experiences included generalised (non-specific), inadequate, inconsistent (eg, contradictory verbal and written feedback from the same trainer), unconstructive/abusive or overly positive (and therefore educationally unhelpful) feedback. Trainees often wanted formative feedback to help improve their performance (ie, feedforward) rather than ticks (ie, feedback).

Finalising SLEs and WPBAs

Some participants described examples of trainers completing forms promptly, sometimes during the SLE/WPBA itself, with the feedback being a dialogue. However, finalising the SLE/WPBA process often involved chasing trainers to complete forms within e-Portfolios, which trainees perceived as frustrating and awkward. Trainers were also frustrated if they received the link to the form weeks after the SLE. Trainers and trainees described how written e-Portfolio feedback could be inadequate: while some trainees used trainer comments to promote reflection within their e-Portfolio, others seemed to lack motivation to read their e-Portfolio feedback. Occasionally trainers relied on hearsay or having a general overview of a trainee, rather than seeing events for themselves, signing trainees off without actually witnessing their performance, a subtheme we called ‘manipulating the system through short-cuts’ (see tables 4 and 5).

Table 5

Issues around supervised learning events/workplace-based assessments

Issue	Illustrative quote
Initiation	“I've done catheter insertion and I did that for the first time as a DOPS [Direct Observation of Procedural Skills] because while I was on call a lady needed to be catheterised and the SHO [Senior House Officer] said to me “have you done a catheter before? Do you want to do it as a DOPS for me?”” (Female F1, site 1)
Tools used	“… probably the Mini-CEX [Mini Clinical Evaluation Exercise] has been the most useful, I say that because we have a trainee who's currently in difficulty and we had an extra assessment for her a couple of months ago and it became clear that she could swat up for the CBD and was actually quite good at the CBD [Case-based Discussion] but in the Mini-CEX when you're in a clerk situation the patient is there you're seeing the whole package… it was the most valuable tool for us in this particular trainee because it seemed to pick out where the gaps were and it was quite alarming ((laughs)) where the gaps were ((said with laughter)) and that's the best tool we found for that particular trainee …” (Female Trainer, site 1)
Feedback	“there's no point somebody sitting down and filling in a form that takes you know a minute to complete and and all they say is “very good carry on”… because that fine it's nice to have nice things said about you but it doesn't really help in terms of training or feedback… give them something to reflect on” (Male Trainer, site 1)
Finalising	“I'm still waiting and that was about a month, maybe a month ago ((laughs))… I sent her [trainer] some erm reminder e-mails and I think probably… next week I'm gonna have to go up to her and say “Oh I sent you an e-mail, have I got your right e-mail address?” kind of thing but I don't really like chasing people… it's a bit uncomfortable kind of situation” (Female F2, site 3)

Factors facilitating and inhibiting learning in SLE/WPBAs

Participants described many factors that facilitated and inhibited learning throughout SLEs and WPBAs at four different levels: individual (eg, characteristics of individual trainees and trainers), interpersonal (eg, trainer–trainee relationships), cultural (eg, protected time) and technological (eg, e-forms; see table 6).

Table 6

Factors facilitating/inhibiting learning through supervised learning events/workplace-based assessments

Levels	Definition	Illustrative quotes
Individual	Trainee/trainer characteristics including the presence (facilitator) or absence (inhibitor) of: enthusiasm, motivation, and engagement; understanding of SLE/WPBAs; teaching/learning competence; self-reflection and self-awareness; organisational skills and confidence.	“but it seems to be sort of confusing the seniors as well because they're not too sure what's required of us… they're not too sure what the requirements are and to be honest when we first started it didn't seem like the academic office was too sure of the requirements either… so no one had a clue sort of how many we all needed…” (Female F2, site 1)
Interpersonal	Trainee-trainer relationship characterised by presence (facilitator) or absence (inhibitor) of: knowledge of the other person and continuity of relationship; mutual respect; like, warmth, and trust; an identification with the ‘other’ and a sense of connectedness; connection to the ‘team’ with shared goals.	“In a way it's needed really because of the way postgraduate medical training has been condensed and continuity of training has disappeared so you don't get the same mentorship and the same apprenticeship that you used to be because you're working with a number of different consultants depending on which day of the week it is and I think that's one of the things that is difficult actually for the trainers is that they may not see a lot of the trainees to get the background sense of how a trainee actually is so that they can then provide meaningful input related to a specific case…” (Male Trainer, site 1)
Cultural	Organisational characteristics including presence (facilitator) or absence (inhibitor) of: safe learning and assessment culture; protected time for supervised practice including observation and feedback; rotations with adequate durations; team-orientation with availability of registrar, consultant and non-medical trainers (eg, nurses); relevant tools for each specialty.	“I think the SLEs were a little bit easier [on my second rotation] because you got regs [registrars] to do it… the environment is very amenable to SLEs because you saw the same regs again and again and it's easy to follow-up versus another environment that's less so, let's say if you're working in orthopaedics not so much because their rotas don't exactly facilitate for seeing people on a regular basis and it's a different, separate teams and very much the FY1 more on the wards and that's why pretty much so it really is environment depended” (Male trainee, site 1)
Technological	Technological characteristics including presence (facilitator) or absence (inhibitor) of hardware (eg, computers, smartphones) and software (eg, online tools, Internet).	Int: How quickly do you complete their form, their e-Portfolio?MT: I tend to do them online at the time… primarily because I'm never more than two feet away from my iPad and so it's easy to um get them to log in either on a terminal and send me a link (Male Trainer, site 3)

RQ3: how do participants make sense of their experiences through narrative?

Participants narrated their SLEs/WPBAs with interesting discourse features (eg, pronominal, metaphoric and emotional talk and laughter), revealing how they constructed themselves, others and their relationships. In terms of pronouns, participants often referred to the ‘other’ as ‘them’, illustrating adversarial trainer–trainee relationships (eg, “they need a certain amount completed so particularly towards the end of placements you get a lot of reminders because you haven't done it ‘cause you haven't had time um and they're panicking ‘cause they need to get them” (Trainer, site 3). Participants’ metaphoric talk also illustrated how they understood the trainee-trainer relationship as adversarial, for example as war (eg, “we get at least one CBD… and questions get fired back and forward” (Trainee, site 2) and sport (eg, “I think it was… a win-win for both of us…. they realised where they were with it, they acknowledged that some of their deficiencies and I was able to form a game plan…” (Male Trainer, site 2). Participants employed positive and negative emotional talk throughout their narratives (eg, “the supervisors don't know their trainees because of the way the rotations work, and that must be very difficult I think… yes it is very difficult” (Female Trainer, site 2), and also laughter, in order to cope with the recounting of difficult stories (eg, “I'll talk about a good one I've had, because then we'll get on to the bad ones I've had ((laughs))” (Trainee, site 3).

To illustrate these themes in more depth, we next present one narrative exemplar from a trainee which demonstrates the complex interplay between what participants say and how they narrate their experiences in order to make sense of them, identities and relationships. We selected this narrative because it is fairly typical, illustrates a range of themes already discussed in this paper, and includes interesting discourse elements found across our data (see Rees et al30 for a further narrative analysis). Helena (a pseudonym) is a female F2. She narrates a WPBA experience from the end of her final F1 rotation. Her experience takes place in a medical setting within the hospital and involves her clinical fellow trainer. She recounts a fairly typical experience: “hunting” for outstanding WPBAs/SLEs near the end of rotations. In the following narrative, Helena explains how her trainer offers to sign off ‘inserting a venflon’ without observing her (see table 4), thus clearly indicating how trainees and trainers can manipulate the system through short cuts.

She constructs her own identity and that of her clinical teaching fellow through narrating her DOPS experience. Helena presents herself as a competent Foundation doctor by emphasising her day-to-day participation in the medical work of the hospital: taking blood and inserting venflons. She sees her competence in these procedures as without question, emphasised by her repeated comments about trainers “knowing” that she and her fellow Foundation doctors can insert venflons because they see evidence of them in patients’ arms. Helena suggests the obviousness of Foundation doctors’ competence, in that they would not be able to “survive on the wards” if they could not take blood. Helena positions her clinical fellow (and other trainers) as having insufficient time ‘to actually stand and watch’ trainees do basic procedures that they are competent in. Helena presents her trainer as knowledgeable and proactive because he checks she has completed her WPBAs for the end of her rotation. While he is partly constructed as helpful for offering to sign off a venflon insertion, he is simultaneously constructed as blasé in that her competence is ‘taken for granted’.

There are various discourse elements in Helena's narrative that are worthy of consideration, including her pronominal and metaphoric talk and laughter, all of which shed light on how she makes sense of this DOPS experience. In terms of her pronominal talk, she repeatedly positions herself as ‘we’ throughout her narrative (meaning me and the other Foundation doctors), and she repeatedly positions her clinical fellow as ‘they’ throughout the narrative (meaning him and other trainers). This use of ‘we’ and ‘they’, rather than ‘me’ and ‘him’, depersonalises and simultaneously generalises her experience, implying that all Foundation doctors commonly experience this event.31 Furthermore, this ‘them and us’ language within the narrative implies an oppositional relationship between trainees and trainers.31 In terms of metaphoric talk, Helena explains that she is “hunting” for patients in order to get DOPS signed off, and she is busy “surviving” on the wards by practising procedures competently. This latter metaphoric linguistic expression, for example, implies the common conceptual metaphor of medicine as war, and similar to the pronominal talk implies oppositional relationships between trainees and trainers.32 33 What is striking about these metaphoric linguistic expressions are that they are both accompanied by laughter, possibly for contextual coping (in the interactional moment of narrating the event) and non-contextual coping (due to uncomfortable feelings around the nature of what it is she's disclosing in her narrative).34 35 This laughter for coping suggests that experiences such as this (‘I don't find DOPS very useful’) can have a negative impact on trainees’ emotional learning experiences.

RQ4: What are participants’ suggestions for how SLEs should be developed?

In response to our final question (how do you think SLEs could be improved?), participants provided a range of suggestions at four different levels: individual (eg, improving trainees’ and trainers’ understanding and engagement), interpersonal (eg, improving trainer–trainee relationships), cultural (eg, shifting away from tick-box summative culture), and technological (eg, improving e-tools: see table 7).

Table 7

Suggested improvements to the supervised learning event process

Level	Definition	Illustrative quotes
Individual	Suggestions included improving trainee/trainers’ understandings of SLEs and their engagement with SLEs.	“I think that we would very much like to have a clearer idea about what it is we should be doing rather than having to make up what it is that we actually are doing” (Trainer, Site 3)
Interpersonal	Suggestions included increased opportunities for trainers to receive feedback from trainees, more regular trainee-trainer meetings, and a developmental approach to the trainee-trainer relationship.	Trainee 1: the same way we have to get evidence that we've done these things, I think that they [trainers] should also have evidence… they have to show examples that they have given feedback…so I think they should be required to do it as wellTrainee 2: I think that's a great idea that we give feedback on their feedback ((says laughingly))” (Trainees, Site 1)
Cultural	Suggestions included increased recognition for the roles of clinical/educational supervisors, increased diversity among trainers able to do SLEs, improved continuity in processes across the continuum of postgraduate medical education, increased clarity around the initiation of SLEs, shifting away from a tick-box culture and removing structures allowing for cheating.	“this is a tool…which is meant to be used in conjunction with the training that goes on and if the training that goes on isn't happening… if consultants aren't able to come and watch you in the clinic…for an hour an hour and a half to actually observe what you're doing if they're not in a position to be able to do that then it doesn't matter how good the tool is…I don't know how you make it better until you can actually release consultants and registrars and people to actually to give them time to say you know you're doing training” (Trainer, Site 1)
Technological	Suggestions included improving e-tools and platforms, and altering the system to reduce time spent chasing trainers to finalise the process.	“maybe if all the, all the feedback-ey things were right at the top of the form and the tickbox-ey things were further down… because the trouble with tick-boxes is, I've done it myself you know “yeah, yeah, yeah, yeah, yeah, fine, yeah, whatever”…you go into tick-box mode and and it's like “any further comment?” is “what, you want me to say MORE?!” ((laughs loudly))” (Trainee, Site 3)

Discussion

This independent evaluation, commissioned by the AOMRC, is the first of its kind to explore Foundation trainee and trainers’ understandings and experiences of SLEs compared with WPBAs since the introduction of SLEs in 2012.

Confusion reigned among participants about what SLEs were and how they differed from WPBAs. While participants ultimately volunteered diverse understandings of SLEs (eg, learning and assessment), they volunteered understandings of WPBAs that were almost exclusively assessment related. Trainers seemed more likely than trainees to volunteer understandings of SLEs as assessment and a ‘safety net’ to protect patients. That many trainers continue to understand SLEs as assessment means that they may continue to treat them as such, thereby jeopardising trainee learning.

The narratives illustrated that SLEs and WPBAs were conducted in diverse ways, with issues raised about their initiation, tools used, feedback and finalisation. Enthusiastic trainers and trainees and good relationships facilitated learning within SLEs/WPBAs, whereas time pressures and e-tools posed barriers to learning. SLE narratives were more likely to be evaluated positively than WPBA narratives. Trainees narrated more SLE experiences with positive evaluations and more narratives of WPBAs with negative evaluations. Some of these findings extend the already mixed evidence for WPBA in terms of its acceptability to trainees and trainers.2 10 36 Previous research, for example, indicates that feedback within the medical workplace can be suboptimal and numerous factors can hinder workplace learning, such as a lack of protected time for the trainee–trainer relationship.16 20 37 38

This study provides tentative support for the summative to formative shift in focus from WPBAs to SLEs initiated by the AOMRC.1 Furthermore, this study contributes to our understanding of the lived experiences of trainers and trainees, and provides quantitative data on differences in SLE/WPBA experiences between trainees and trainers. That trainees were more likely to report positive evaluations of their SLE experiences (and trainers not) suggests that trainees and trainers might want different things from SLEs/WPBAs (learning vs assessment respectively). Further, that participants constructed their own and others’ identities, and their relationships in numerous ways builds on other medical education research at the undergraduate level emphasising potential conflictual relationships between trainees and trainers31–33 39

Key suggestions to improve the SLEs included improving trainees’ and trainers’ understandings of SLEs, better trainee–trainer relationships through regular meetings and closing the ‘feedback loop’, improving the culture of workplace learning through formative learning rather than summative assessment, and improving the technology around SLEs, extending previous research within medical education.16 20 37–43

Methodological strengths and challenges of study

To the best of our knowledge, this is the first study to explore Foundation trainee and trainers’ understandings of SLEs and WPBAs, and their lived experiences. The large number of narratives collected, and our consistent findings across the three geographically dispersed sites, suggests that our results are transferable to other UK locations. Although our sample of trainees and trainers was intentionally diverse, we had relatively low numbers of GP and nurse trainers in our study, and relatively few trainees with GP and nurse trainer SLE/WPBA experiences. While this reflects the reality of training programme structures and processes, we must use caution when extrapolating our findings to GP settings and to GP and nurse trainers. Having employed qualitative methods, our sample is not necessarily representative, nor does it intend to be representative, of all UK trainers and trainees.

The geographical distance between sites and the need to collect large amounts of qualitative data in a relatively short time-frame (around 6 months) required multiple researchers across the three sites to undertake interviews and data analysis. Consistency was maintained across the researchers through training, the use of a discussion guide, regular meetings and use of a comprehensive coding framework. Finally, with around 46 h of qualitative data it was pragmatic for us to adopt different methods of data analysis to explore both the breadth and depth (and, therefore, the what's and how's) of participants’ experiences. Because of this voluminous data, we partly quantified it to identify patterns across our narratives that would otherwise be invisible.44 45 Some methodological purists would find this combination of quantitative and qualitative analyses problematic because of the different epistemologies underpinning these two approaches. However, we retained a process-orientated qualitative approach to our interpretation of numerical data.44 45

Implications for educational practice

Our recommendations are based on key findings from our research (what works and what does not work) and comments from our clinical reference group (see Acknowledgements section). First, we need to improve trainee and trainers’ understandings of SLEs. Both must understand the concepts of formative and summative assessment and be able to recognise good quality feedback; that feedback is a dialogic process; and how they can give, receive and seek feedback effectively within the workplace.46 Both need to appreciate the diversity of processes for conducting SLEs; know the tools and how they differ; and comprehend factors facilitating and hindering learning within SLEs.

Second, trainee–trainer relationships need to be improved. Good quality relationships, characterised by knowledge of the other person, mutual respect and trust, should be possible through prolonged engagement including multiple trainee–trainer meetings throughout rotations. We recognise that the pressures of service delivery make this recommendation challenging.

Third, the culture of workplace learning needs to be improved. The formative focus of SLEs could be emphasised further by rethinking the structures around SLEs, and particularly those structures that imply a summative focus. For example, SLEs should be undertaken at regular intervals with a cumulative formative impact over the course of a rotation, thereby allowing trainees to conduct SLEs in a meaningful way that is beneficial to their own personal and professional development, rather than encouraging a system of ‘hunting’ for SLEs at the end of a rotation to secure that ‘tick’.

Fourth, tools employed for SLEs need to be improved to emphasise their formative focus (eg, prioritising free-text comments) and making them easier to finalise (eg, applications for smartphones and tablets).5

Finally, we need to develop, assess and recognise trainers for the work they do including the provision of trainee feedback to trainers to close the ‘feedback loop’,46 and to be used as part of trainers’ annual appraisals. Furthermore, this process of feedback could form the basis of a trainer recognition programme, thus valuing the important role of the educator.

Implications for further research

The introduction of any new workplace-based initiative will benefit from investigation using a range of approaches. Further interview research is required using wider sampling (eg, capturing GP experiences) to more fully elucidate the themes identified in this paper. Also, additional qualitative (eg, longitudinal audio-diary, video-reflexive ethnography) and quantitative methodologies (eg, pragmatic cluster randomised trial) would be helpful to explore SLEs further. The latter could compare various outcomes (eg, trainee and trainer satisfaction, metrics around form completion) for an intervention group of trainers and/or trainees who have received theory-based training in giving, receiving and seeking formative feedback, compared with those not receiving the educational intervention. Ultimately, without such further research, it may be impossible to fully understand the complexities surrounding SLEs within workplace learning.