An exploratory study considering the potential impacts of high-fidelity simulation based education on self-evaluated confidence of non-respiratory physiotherapists providing an on-call respiratory physiotherapy service: a mixed methods study.

INTRODUCTION: Physiotherapists working on-call to provide emergency respiratory services report stress and lack of confidence in on-call scenarios. Simulation-based education (SBE) is a potential solution to improve confidence and reduce stress of on-call physiotherapists. In physiotherapy, use of SBE is sporadic. The aim of this study was to evaluate the addition of SBE to an on-call training programme on non-respiratory physiotherapists' self-evaluated confidence. Additionally, the study aimed to evaluate if SBE facilitates identification of learning needs.
METHODS: This cohort study took a mixed methods approach. Participants were recruited from staff providing on-call respiratory physiotherapy services at a UK hospital. Participants received traditional on-call training over 1 year, with SBE added the subsequent year, in a pre-post analysis design. Self-evaluated confidence was assessed with the Association of Chartered Physiotherapists in Respiratory Care Acute Respiratory/On-call Physiotherapy Self-evaluation of Competence (ACPRC) questionnaire. Two focus groups were conducted post-SBE.
RESULTS: There were 10 participants. Thematic analysis of focus groups indicated participants found SBE provided coping strategies for on-call working. Using coping strategies taught in SBE reduced stress levels and increased confidence of non-specialist on-call physiotherapists. ACPRC questionnaire scores significantly improved following the addition of SBE (median change 5.5%, p=0.034, r=0.57). SBE assisted in identification of learning needs through recognition of unconscious incompetence and reinforcement of conscious and unconscious competence.
CONCLUSIONS: SBE may improve self-evaluated confidence of non-specialist on-call physiotherapists. SBE assists in learning needs identification. SBE could enhance training of physiotherapists providing on-call respiratory services. Further larger trials investigating optimal methods of on-call physiotherapy postgraduate education are warranted. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Introduction

Physiotherapists in the UK work in a variety of specialties, and are permitted to do so, as long as they are working within their scope of practice.1 These specialties include respiratory care, and UK physiotherapists are often required to provide an on-call service and work outside of normal working hours. An on-call scenario is a situation where an acutely unwell patient will deteriorate without immediate respiratory physiotherapy assessment and treatment.2

Logistically, many hospital trusts employ non-respiratory physiotherapists to provide on-call services, thus these staff are expected to function out of hours as respiratory physiotherapists.3 There are concerns within the physiotherapy profession regarding the competence of these non-specialist staff, with risks of patients deteriorating when inexperienced staff are delivering complex treatments.4 There are considered to be four stages of competence: unconscious competence, unconscious incompetence, conscious competence and conscious incompetence.5 Where unconscious incompetence exists, physiotherapists will find it difficult to identify learning needs associated with that particular skill.5

Many physiotherapists find on-call situations stressful and report a lack of confidence in these scenarios, which is especially evident among newly qualified or non-respiratory staff.6 Novice physiotherapists report sources of stress when on-call services include: lone working, complexity of patients and working in critical care environments.7 A qualitative investigation into causes of stress in recently qualified physiotherapists revealed 58% reported being on-call as stressful, 51% perceived working with acutely unwell patients to be stressful and 49% stated working a weekend was stressful.6 According to the Yerkes-Dodson law (a term describing the inverted U relationship between stress and performance), there comes a point where high stress levels result in underperformance.8 Although performance has not been assessed in physiotherapists in an on-call situation, an on-call physiotherapist lacking confidence, or who feels highly stressed, is potentially more likely to make mistakes, which may impact on patient safety and effectiveness of treatment.

Simulation-based education (SBE) is a technique aimed at achieving set learning objectives through replication of, and immersion in, real-world experiences.9 This educational approach is grounded in adult learning theories, allowing learners to complete the stages of an experiential learning cycle: active experience, reflective observation, abstract conceptualisation and active experimentation.10 11 Experiential learning allows integration of theory and practice, facilitating changes in practice.12 SBE may be useful in educating on-call physiotherapists to address concerns around practitioner competence and confidence.13

SBE has been integrated into medical training, yet remains infrequent within physiotherapy training.14 Despite a UK Department of Health (2011) recommendation that SBE be integrated into health professionals’ training, a survey of UK physiotherapy departments found only 15% (n=24/155) of respondents used SBE.14 These low numbers were attributed to lack of access to equipment and inexperience in providing SBE.14 Evidence for SBE suggests it is limited to undergraduate physiotherapy training and focuses on student competency rather than changes in practice or patient safety.15–17 The evidence base in undergraduate education suggests SBE may also be a useful educational method for postgraduate physiotherapists.

The aim of this study was to evaluate, through a mixed methods approach, non-respiratory physiotherapists’ self-evaluated confidence following the addition of a novel educational modality (SBE) to an on-call training programme. Additionally, the study aimed to evaluate if SBE facilitated participants to identify their own learning needs.

Methods

This cohort study used a mixed methods approach. Participants were volunteers recruited from staff providing on-call respiratory physiotherapy services at an acute UK hospital trust. At the time of the study, there were 14 non-specialist physiotherapists working on-call eligible for the study. Participants were non-respiratory physiotherapists: defined as staff providing on-call services but not employed as a respiratory physiotherapist in their permanent job. In order to allow evaluation between the two education methods, participants had to have partaken in the on-call training programme, for a minimum of 1 year, prior to introduction of SBE. In accordance with local guidelines, participants were assessed as competent by a respiratory physiotherapist prior to working on-call. Respiratory physiotherapists, and staff who did not complete the entire year’s training programme, were excluded. On-call staff received standard on-call training in the first year (2012). In the second year (2013), SBE was provided in addition to the standard training (table 1).

Table 1

Summary of standard on-call training and on-call training with the addition of SBE

Standard on-call training (provided in 2012)	On-call training with the addition of SBE (provided in 2013)
Four training sessions provided quarterly. 90 min training sessions: Paper-based scenarios; Paediatric update; MDA competencies for equipment; Practical skills refresher. Mixed experience and specialty groups. Total of 6 hours of training.	Three training sessions provided four monthly interspersing SBE sessions. 1 hour long: o MDA competencies; o Paediatric update; o Practical skills refresher. In addition, two SBE training sessions lasting 3 ½ hours each spaced 6 months apart. Four to five candidates in a group for SBE. Mixed experience and specialty groups. Total 10 hours training.

MDA, Medical Devices Agency; SBE, simulation-based education.

SBE was provided at the research site’s simulation centre by simulation facilitators, including the simulation centre coordinator and two respiratory specialist physiotherapists. Laerdal SimMan essential (Laerdal Medical, Orpington, Kent, UK) and Laerdal SimBaby (Laerdal Medical) mannequins were used.

Eight scenarios were developed by experienced respiratory physiotherapists, and were reviewed by a clinical specialist physiotherapist and an academic respiratory physiotherapy researcher with SBE experience, who ensured clinical accuracy and appropriateness of learning aims. Scenarios were subsequently piloted on five respiratory physiotherapists, and modified, prior to inclusion in the SBE programme. The learning objectives for the scenarios are available as an online supplement (see online supplement 1), the main aims of the scenarios were focused around non-technical skills and human factors. At each half day session, four scenarios were undertaken: participants completed one scenario as the candidate and observed the other three scenarios. Each scenario lasted approximately 15 min. After completion of the scenario, a faculty-led debrief session followed lasting approximately 45 min following the diamond debrief model.18 Debrief was led by appropriately trained simulation facilitators and aimed to identify learning needs, and promote self and peer evaluation of performance. Staff were encouraged not to share content of the scenarios with their colleagues who were yet to attend SBE training sessions.

Data collection

Focus groups at the workplace were conducted within 3 months of completion of the training programme, lasted approximately 1 hour and were audio-recorded. The semi-structured questions for the focus group were piloted on two respiratory physiotherapists, who had experienced SBE during the development of the scenarios, but who were excluded from the main study. As a result of the pilot study, the semi-structured questions were refined (see online supplement 2). To reduce bias, the focus groups were facilitated by a medical respiratory physician with knowledge of respiratory physiotherapy, limited experience in SBE, experience conducting focus groups and who did not work with the focus group participants on a day-to-day basis.

The Association of Chartered Physiotherapists in Respiratory Care Acute Respiratory/On-call Physiotherapy Self-evaluation of Competence (ACPRC) questionnaire was developed to assist physiotherapists undertaking on-call duties to identify their self-perceived competence and confidence, thus facilitating identification of their learning needs.19 Each item of the ACPRC questionnaire was scored using a range of 0–4 (0 strongly disagree to 4 strongly agree). An ACPRC questionnaire score is calculated and reported as a percentage 0%–100%.19 Additionally, subsections of the ACPRC questionnaire can be calculated: assessment skills, treatment skills, treatment skills matrix, range, managing a call-out and confidence. A lower score indicates less and a higher score indicates more confidence in an on-call scenario.19 The ACPRC questionnaire is the only published tool allowing assessment of self-perceived competence and confidence of physiotherapists, and while further work is required to assess reliability and sensitivity of the tool, preliminary evidence suggests validity in this population.19 Participants completed the ACPRC questionnaire pre-SBE and post-SBE training. In addition, a demographics questionnaire was completed by participants.

All participants provided written informed consent.

Data analysis

Audio recordings were transcribed verbatim and checked for accuracy by the lead investigator via repeated listening of the recordings. QSR NVivo (V.10) was used for data management. Qualitative data were analysed by the lead investigator (SKM), who had prior experience of qualitative methods. A thematic analysis approach was undertaken, using an a priori framework, informed by the research questions.20 21 Thematic analysis is a qualitative analytical approach using codes to recognise, evaluate and describe patterns from a data set.20 22 Patterns were recognised through searching the transcripts for repetitions, metaphors and analogies, transitions, similarities and differences and linguistic connections.23 The thematic analysis took a deductive, theoretical approach, as the investigator used the research aims to inform and guide development of themes.20 However, there was an element of inductive strategy to the analysis, as themes were found that were not related to the research aims.22 Additional themes became apparent during the thematic analysis of written transcriptions, which had not been discovered during the pilot phase, further demonstrating an element of inductive approach. This thematic analysis identified themes at a semantic level using a realist approach. A semantic-level analysis means the investigator assumes participants are being explicit in their description of their experiences.20 A realist approach assumes the language participants used reflects their true meaning and experiences.20 The analytic method employed a process of familiarisation, generation of initial codes, searching for themes, reviewing themes and defining and naming themes.20–22 The initial themes were generated from the pilot data, which then provided the theory on which to conduct the thematic analysis. These themes were created as nodes in NVivo, which was then used to organise the written transcriptions, by highlighting and grouping codes under one node. Data for individual themes were reviewed in NVivo, allowing refinement and definition of the themes, using the theories described above.

Data from the ACPRC questionnaire were entered into a Microsoft Excel 2010 spread sheet. SPSS statistics (V.22) was used to analyse descriptive statistics; 95% CI and effect size were calculated for median differences in ACPRC score pre-SBE and post-SBE training using z-scores.24 25 The Wilcoxon signed rank test was employed to assess non-parametric, paired, ordinal data (ACPRC questionnaire results).

Results

A total of 10 participants were recruited to the study (figure 1). The demographics of participants in the focus groups are displayed in table 2.

Figure 1

Recruitment process. ACPRC; Association of Chartered Physiotherapists in Respiratory Care Acute Respiratory/On-call Physiotherapy Self-evaluation of Competence; SBE, simulation-based education.

Table 2

Characteristics of participants

Participant number	Gender	Time since qualification (years)	Job title	Previous SBE experience
1	Female	2.5	Senior physiotherapist: general (inpatients)	Nil
2	Female	3	Physiotherapist	Nil
3	Male	5	Senior physiotherapist: general (inpatients)	Nil
4	Female	4	Senior physiotherapist: general (inpatients)	Nil
5	Female	1.5	Physiotherapist	Nil
6	Male	2	Physiotherapist	Nil
7	Female	2.5	Senior physiotherapist: general (inpatients)	Nil
8	Female	7	Senior physiotherapist: neurology	Nil
9	Female	7	Team leader physiotherapist	Nil
10	Female	3	Senior physiotherapist: general (inpatients)	Nil

Self-evaluated confidence: ACPRC questionnaire

The median difference in overall ACPRC questionnaire scores pre-SBE and post-SBE was significant (median difference 5.5%, 95% CI 2% to 16%, p=0.034). The r score (0.57) suggests there was a large effect size.24 Figure 2 demonstrates the changes observed in the ACPRC questionnaire subsection scores. Except for the treatment skills section, all sections saw a statistically significant increase in scores following the introduction of SBE.

Figure 2

Changes in ACPRC questionnaire subsection scores pre-SBE and post-SBE. ACPRC; Association of Chartered Physiotherapists in Respiratory Care Acute Respiratory/On-call Physiotherapy Self-evaluation of Competence; SBE, simulation-based education.

Thematic analysis of the focus group data resulted in one theme related to the identification of learning needs; theme 4 ‘SBE facilitates the identification of learning needs’.

Thematic analysis of the focus groups resulted in three themes: ‘non-specialist on-call physiotherapists found on-call experiences stressful pre- SBE training’, ‘SBE training provides a structure in which to work when on-call’ and ‘SBE provides coping strategies for dealing with on-call-related stress’.

Theme 1: ‘non-specialist on-call physiotherapists found on-call experiences stressful pre-SBE training’

When asked to reflect on their experiences of on-call working prior to SBE, participants described on-call as causing feelings of stress and anxiety. Participants reported feeling ‘daunted’, ‘stressed’, ‘scared’, ‘terrified’ and ‘worried’ about being both on-call and in an on-call situation. These were perpetuated by fear of both the unknown and lone working without senior support. Participants also reported concern regarding their ability to function out of hours, especially if woken for a call in the ‘middle of the night’. The anticipation of receiving a phone call also heightened stress levels. Particular concerns were expressed over paediatrics and critical care. Furthermore, participants expressed anxiety levels increased when they were not working day to day in a cardiorespiratory clinical area. Participants reported their anxiety decreased when they were ‘prewarned’ about an at-risk patient, or where ‘preplanned call outs’ were arranged. Participants reported a reduction in anxiety as their clinical experience increased, this was particularly evident for participants who had worked in more than one hospital. Table 3 illustrates representative quotes for the theme ‘non-specialist on-call physiotherapists found on-call experiences stressful pre-SBE training’.

Table 3

Framework matrix for theme 1

Theme	Representative quotations
1. ‘Non-specialist on-call physiotherapists found on-call experiences stressful pre-SBE training’	“I was quite terrified of being on-call if I’m being honest”(Participant 5) ‘Sometimes that thought of not having someone immediately there’ (not having senior support while on-call) (Participant 4) ‘It’s just the unknown’ (in reference to being called out) (Participant 1)

Theme 2: ‘SBE training provides a structure in which to work when on-call’

Participants reported that crisis resource management (CRM) approaches used in SBE taught them to approach on-call situations systematically. Participants reported their assessment skills were improved through use of the ABCDE approach,26 which they felt reduced the chance of errors or omissions. Participants reported the Situation, Background, Assessment, Recommendation cogntive aid27 improved their non-technical skills, by enhancing communication with the multidisciplinary team and reducing anxiety surrounding receiving a phone call while on-call. Participants felt SBE improved both their clinical reasoning skills, and confidence in decision making. SBE exposed participants to rare clinical situations, which they reported helped them to feel more prepared for on-call working. Participants reported they felt SBE was particularly useful for newly qualified staff prior to on-call working. Participants did not feel SBE improved their technical skills. Table 4 illustrates representative quotes for the theme ‘SBE training provides a structure in which to work when on-call’.

Table 4

Framework matrix for theme 2

Theme

Representative quotations

2. ‘SBE training provides a structure in which to work when on-call’.

“having strategies to cope with, with things you gain from the sim centre then it makes your on call respiratory so much easier”(Participant 2) “It (SBE) taught me to just hang on a minute step back go through it systematically”(Participant 3) “I think I’m a lot more confident in decision making” (Participant 2) “It (SBE) gives you some exposure to that (rare clinical scenarios) and some idea of actually how you would manage it” (Participant 4) “I feel more confident” (Participant 2)

Theme 3: ‘SBE provides coping strategies for dealing with on-call-related stress’

Participants reported SBE duplicated the stress felt in an on-call situation. For example, participants were not given advice prior to the SBE on the content of their scenario; participants reflected this was useful as you cannot predict an on-call scenario. Participants reflected SBE aided them to identify how they respond, particularly emotionally, to stressful situations, specifically on-call. By teaching CRM, SBE gave non-specialist physiotherapists strategies to cope with the pressure of on-call working. Participants regarded themselves as being more confident in their clinical reasoning, resulting in reduced anxiety surrounding the lone working element of on-call. Participants reported the reflective nature and peer support gained from the debrief gave them skills to deal with on-call-related stress. Table 5 illustrates representative quotes for the theme ‘SBE provides coping strategies for dealing with on-call-related stress’.

Table 5

Framework matrix for theme 3

Theme

Representative quotations

3. ‘SBE provides coping strategies for dealing with on-call-related stress’.

“I think it’s quite a strong feeling as well it like it (HFS) brings out quite a few emotions, mainly anxiety but it helps (when you’re on-call) to think remember what you learnt in the sim centre”. (Participant 3) “I think you learn a lot about yourself and how you manage stressful environments”. (Participant 2) “My heart drops but I know how to cope with it rather than thinking oh my god” (in reference to receiving a phone call while on-call) (Participant 5) “And just being more structured you know like being taught ways in the reflection but about how to actually manage it” (on-call scenarios and stress) (Participant 1) “It’s those skills you can carry over perhaps not physio specific but dealing with stress” (Participant 3)

Theme 4: ‘SBE facilitates the identification of learning needs’

All participants reported SBE assisted them to identify learning needs by facilitating recognition of their weaknesses and areas for improvement. In addition, participants felt they learnt through their mistakes in the scenarios and were able to highlight areas of previously unconscious incompetence. All participants identified SBE facilitated reflective practice, hence improving the learning experience and their ability to translate learning into changes in their practice. In contrast, participants also reported SBE, particularly the debrief, identified their strengths and reinforced both their conscious and unconscious competence. Participants reported beneficial learning from all aspects of the experiential learning used within SBE. They found they could identify learning needs when they were the candidate carrying out the scenario, or observing the scenario and during the debrief. Table 6 illustrates representative quotes for the theme ‘SBE facilitates the identification of learning needs’.

Table 6

Framework matrix for theme 4

Theme	Representative quotations
4. ‘SBE facilitates the identification of learning needs’	“I think it was going to highlight weaknesses, which is actually a good thing” (Participant 2) “having people reflect back to you about how you did … was useful for me” (Participant 4) “Cause sometimes you not always sure that your positives are actual positives” (Participant 2)

Discussion

This investigation suggests SBE may improve self-evaluated confidence of non-respiratory physiotherapists providing an on-call service. From the thematic analysis, participants found SBE techniques supplied them with a framework for application during an on-call situation. As an example of using cognitive aids, participants found the ABCDE approach to assessment, in addition to exposure to rare clinical scenarios, provided coping strategies for managing on-call situations. Anecdotally, the ABCDE approach is taught throughout nursing and medicine26 28–30 but is not often taught, or applied, in physiotherapy practice.31 32 The ABCDE approach is now standard practice at the research centre, and benefits to staff are apparent from this investigation. The ACPRC questionnaire results demonstrated significant improvements in overall scores, but not all sectional scores improved. Given SBE in this course focused on non-technical skills, it is perhaps not surprising the ACPRC questionnaire sections relating to treatment (treatment skills and treatment matrix and range) did not reach significant changes in scores.

Of particular interest during thematic analysis was the theme ‘SBE provides coping strategies for dealing with on-call-related stress’. This theme suggests one benefit of SBE is its ability to reduce stress experienced by on-call physiotherapists which is a novel finding of this study. Using SBE techniques appears to have provided coping strategies, resulting in a calmer on-call experience, thus reducing stress. As previously discussed, high stress levels during on-call working are commonly reported6 and are associated with increased perceptions of support required, increased use of resources7 and reduced performance.8 While not directly assessed during this investigation, a reduction in stress is associated with reduced need for senior support and enhanced performance.7 8 It could therefore be suggested SBE may reduce the need for senior support.

Links have been demonstrated between competence and confidence.33 34 Incompetent individuals overestimate their performance and thus have overinflated confidence.33 In contrast, highly competent individuals underestimate their performance and thus lack confidence.33 For highly competent individuals, observation of peers enables accurate self-assessment and improves confidence.33 Given all participants in this investigation were deemed competent prior to the study, it is plausible the peer observation component of SBE was a contributing factor to improving participants’ self-assessed confidence, although the exposure to SBE was limited in this study and further SBE exposure may be required to enhance this effect.

The ability to self-assess, and thus identify learning needs, is essential in safeguarding patient safety by ensuring professionals perform tasks within their scope of practice.35 The task of self-assessment is a skill which is rarely formally taught.34 36 As was discovered from thematic analysis, SBE assisted participants to identify their learning needs. This finding is perhaps not surprising given the experiential learning methods used in SBE. Interestingly, participants found reinforcement of conscious and unconscious competence to be as useful as identification of unconscious incompetence.

There is an anecdotal growing trend towards integration of SBE into postgraduate physiotherapy on-call education.31 37–43 While there is an ever-growing evidence base for healthcare SBE, particularly for nurses and doctors,44 45 investigation into the impact of SBE on patient outcomes is lacking. Such investigation would require collaborative working across organisations, financial investment and active stakeholder engagement. They are difficult to conduct due to costliness, their time-consuming nature and the population sample size required.44 46 It could be suggested that the evidence base for SBE in other health professionals’ education could be extrapolated to physiotherapy. However, learning is contextual and it may not be appropriate to extrapolate evidence from other professionals to physiotherapists.

The use of a control group would have improved the methodology of this investigation. This study was conducted in an acute hospital trust with a high turnover of staff providing the on-call service, making a control group operationally challenging. As there was no control group in this study and the sample size was small, changes in ACPRC scores over time due to increase in experience cannot be excluded. The SBE was longer in duration than the previous method, creating a further confounding factor. SBE scenarios were limited to 15 min limiting the opportunity for physiotherapists to apply multiple treatment techniques. Furthermore, the direct impact of SBE on patient care was beyond the scope of this investigation, as was the clinical impact of an improvement in self-reported confidence. While the quantitative data demonstrated measurable improvements in the confidence of non-respiratory physiotherapists, the focus group data suggest current outcome measures, such as the ACPRC questionnaire, are not sensitive enough to detect all changes associated with SBE. Furthermore, the reliability and sensitivity of the ACPRC questionnaire has not been established, and it was chosen as an outcome measure in this investigation in lieu of a more appropriate outcome measure. The thematic analysis was undertaken by the lead researcher (SKM), who had knowledge of the participants’ clinical experience, experience and knowledge of SBE and delivered the SBE sessions, potentially creating a source of bias. Time constraints and cost limitations prevented a duplicate analysis of the focus group transcripts, which would have improved the dependability, and thus reliability, of the qualitative data.

Further investigations into the outcomes of SBE training in respiratory physiotherapy should aim to have larger sample sizes and larger cohorts to enable evaluation of the impact of SBE on changes in clinical practice and patient safety. While there is a growing evidence base for SBE, its impacts on patient care and outcomes have not been fully established.46 47 While researchers in healthcare education should strive to establish whether SBE impacts on patient care and outcomes, these trials will need to be large-scale investigations that are likely to be costly and time consuming.46 The development of an outcome measure measuring the impact of SBE on changes in clinical practice which can be used by multiprofessionals and is valid, reliable and sensitive to change would aid research in SBE. Increases in self-evaluated confidence (as measured by the ACPRC questionnaire) were observed in this investigation, but this study did not consider the duration these improvements were sustained for. Further investigation is warranted to establish the duration of improvements in confidence, in order to determine the required regularity of SBE. The design of this study involved SBE being delivered to a uniprofessional staff group. While some publications have considered outcomes of delivering SBE in an interprofessional model, these have been limited to the undergraduate population.48 49 Investigations into outcomes of interprofessional SBE in the postgraduate population are necessary.

Conclusion

In summary, SBE training may improve confidence of on-call physiotherapists. SBE training assists non-respiratory on-call physiotherapists to identify their learning needs, which has implications for self-regulation. Further large-scale trials with methodology, such as randomised controlled trials, accounting for confounding factors to investigate the optimal methods of on-call physiotherapy postgraduate education are warranted. Given the growing evidence base for SBE in the medical profession and other high-hazard industries, the physiotherapy profession should consider integration of SBE into postgraduate training.