Evaluation of resident satisfaction and change in knowledge following use of high-fidelity simulation teaching for anaesthesia residents.

BACKGROUND AND AIMS: Anaesthesia practice demands medical knowledge and skills as essential components for patient management in peri-operative emergencies. Since all residents are not exposed to such situations during their residency, training them using simulation technology could bridge this knowledge and skill gap. The aim of this study was to train and evaluate residents to manage anaesthesia emergencies on high fidelity simulators.
METHODS: Kirkpatrick model of program evaluation was carried out. Resident reaction was captured using a satisfaction questionnaire and the change in knowledge was assessed using pre-test and post-test Multiple Choice Questions (MCQs). Six scenarios were created and executed on a human patient simulator (HPS). All 22 residents participated in this teaching learning method. The steps of simulation teaching included pre-test, pre-briefing, orientation to manikins, performing/scribe, debriefing, feedback questionnaire, and post-test. The satisfaction questionnaire was administered following the second and fourth scenario.
RESULTS: 95% residents agreed on overall satisfaction, that it helps in building team dynamics and clinical reasoning. All students agreed that this teaching had positive professional impact. 14% residents felt they were anxious during the class. The items in the questionnaire had a Cronbach's α value of 0.9. The mean score for pre-test was 24.22 ± 7 (Mean ± SD) and the post-test was 47.18 ± 5.6, the difference between the scores were statistically significant (P = 0.007).
CONCLUSION: The use of high-fidelity simulation to train anaesthesia residents resulted in greater satisfaction scores and improved the residents' reasoning skills. Copyright:

INTRODUCTION

The use of high-fidelity simulation teaching learning tool is known to impart learning among students through experiential learning with reflection on action. Majority of the existing literature is from the western world who have been using simulation as a modality to teach, assess, for research or recertifying anaesthesiologists.[12345] Although simulation as a teaching tool is new, it has picked up great momentum and utility due to availability of skills and simulation centres in various parts of India. The Medical Council of India (MCI) has proposed the new Competency Based Medical Education (CBME) with attitude, ethics and communication (AETCOM) module has been implemented from August 2019 for the undergraduate and postgraduate curriculum.[67] For CBME to be implemented, skills and simulation centres will play a vital role as a training tool to achieve mastery level of learning. To incorporate AETCOM and team training programs into existing curriculum, skills and simulation centres are essential. Hence it is necessary for the teachers to be aware of the basics of simulation teaching.

Peri-operative emergencies related to anaesthesia requires prompt recognition and early intervention for successful positive outcome in patient care. All residents are not exposed to all these emergencies hence are not intellectually equipped to handle such situations later in their clinical practice.[8] Thus, to bridge this knowledge and practice gap, we have used simulation as a training modality. We hypothesise that simulation helps to integrate learning from books to learning in situ by improving their clinical reasoning and decision-making skills. In the present study, we evaluated the simulation training program delivered to the Anaesthesia residents.

The aim of this study was to incorporate simulation as a teaching learning tool for post graduates in Anaesthesia. Our primary objective was to train anaesthesia residents to recognise and manage perioperative emergencies on high fidelity simulators and secondary objective was to document residents' reaction to such training and record the change in knowledge using pre, post-test multiple choice questions.

METHODS

Institutional Ethics Committee permission was obtained (letter dated – 16-08-2017, number – FMMC/FMIEC/4338/2017). This study was conducted according to the principles of Helsinki. The study participants gave written informed consent to participate in the study.

After obtaining clearance from Institutional Ethics Committee, 'need assessment' was done by interviewing the faculty and conducting a focus group discussion for the residents to find out how many faculties were interested in teaching using these mannequins and to identify the topics of perioperative emergencies that had to be included in the timetable. Based on their inputs, six scenarios were created and implemented. The six perioperative emergencies chosen by consensus were high spinal anaesthesia, local anaesthetic systemic toxicity, intraoperative tachycardia, intraoperative cardiac arrest, amniotic fluid embolism and mitral stenosis.

This was a Prospective observational-Mixed method research study (Qualitative and quantitative) including all anaesthesia residents numbering to 22 from our institute. The pilot testing was done to calculate sample size to document resident's response and for change in knowledge using this innovative teaching methodology. The sample size of 22 anaesthesia residents met the requirement for our primary objective to train the residents using High fidelity simulators to document change in reaction and knowledge.

Steps taken to validate the scenarios created before implementing to the residents:

Select the case following a thorough literature search including standard text books

Write the specific learning objectives maximum of three in number

Prepare the pre-test MCQs and check for content validity with two other anaesthesia faculty

Create the scenario including the sequencing of events on the software

Prepare the room with the manikin to simulate the Operating Room setting with the necessary equipment and drugs

Pilot test the scenario for authenticity and reality with two other faculties and do any necessary changes

Implement the scenario to the residents. The 22 residents were divided into 3 batches comprising of 8, 8 and 6 students per group per day, respectively. Each case had to be therefore performed thrice.

The schedule of training and the broad topic to be implemented was notified to the residents in advance and were instructed to come prepared/read for the class. On the day of session, they were given a pre-test to assess their baseline understanding of the concepts related to the topic. This is followed by pre-briefing where the residents were given a brief history of the patient details with vitals, lab investigations and the time point when they are anaesthesia care providers. Later, each batch was divided into four residents per group, where two were the performers (actual care givers) and two were the observers/recorders whose job was to note down the sequence of events and interventions. This is followed by debriefing/reflection where all eight residents and two facilitators sat together to reflect on the actions taken by the residents in managing the case. The facilitators used the plus-delta method for debriefing. The performed actions/interventions that were beneficial for patient's outcome were applauded and reinforced while the gaps in performance or knowledge which need remediation were also highlighted for future corrective performance. The residents could discuss muddy concepts and clarify any other queries related to the event. The final step was the post-test and closure where residents verbalised their take home message and feedback about the facilitator and the case scenario.

To evaluate the entire training program, the Kirkpatrick four level training evaluation model was used.[9]

Level 1 (measures Reaction): the degree to which participants find the training favourable, engaging and relevant to their job (measured using post training questionnaires, interviews, printed or oral reports).

Level 2 (measures Learning): the degree to which participants acquire the intended knowledge, skills, attitude, confidence and commitment based on their participation in the training (measured using Pre-test post-test assessment, observation by peers/instructors, having a control group).

Level 3 (measures Behaviour): The degree to which participants apply what they learnt during the training when they are back on their job (measured using 360 degree feedback, workplace based assessment tools, direct observation of procedural skills, observable change in behaviour, interviews).

Level 4 (measures Results): the degree to which targeted outcomes occur as a result of the training and the support and accountability package (measured by calculating patient safety/reduction in litigations following this educational intervention).

After the second and fourth sessions, resident satisfaction regarding the teaching method was captured using a questionnaire (Annexure 1) designed by Calamassi D et al.[10] It included a total of 43 questions covering seven criteria namely-1. Overall satisfaction, 2. Facilitator and debriefing, 3. Clinical reasoning and self-effectiveness, 4. Team dynamics and team factors, 5. Professional impact, 6. Safeguard and materials, 7. Difficulty and distress.

To evaluate the second level of Kirkpatrick model for change in knowledge pre-test and post-test multiple choice questions (MCQs) were used. These questions were validated for content and relevance to the prepared scenario by two other anaesthesiologists not involved in simulation teaching.

Data was analysed using SPSS-IBM. Exploratory factor analysis was used to assess the underlying factor structure of the questionnaire. The reliability of questionnaire to record the resident satisfaction was analysed using the Cronbach's alpha. By definition, a Cronbach's α from 0.61 to 0.80 represents a substantial correlation and from 0.81 to 1.00 a good correlation. To test if intervention (Simulation training) improved critical thinking and decision making skills we used Extended matching questions in the form of Multiple choice questions (MCQs) as pre and post-tests. The average scores of pre-test and post-test scores were compared using the Student's T-test.

RESULTS

The items on the questionnaire had good correlation with Cronbach's alpha value of 0.90 and the correlations between items ranged from 0.6-0.84.

Table 1 shows the domains and items of the questionnaire and Table 2 shows the satisfaction score of participants for session two and session four for the various components used in the questionnaire. Following session four, 95% of the participants were satisfied with overall training programme. All the participants satisfied with the role of facilitator and debriefing session. About 95% of the participants agreed that this training session helps in developing clinical reasoning skills as well as improving team dynamics. All the participants felt that these simulation training will have professional impact. 95% students agreed that materials as well as safety measures are adequate to recreate the scenarios. Only 5% of the participants felt that these training sessions are difficult and can cause distress.

Table 1

Domains and items of resident satisfaction questionnaire

Domain 1: Overall satisfaction I was satisfied with the Fidelity/realism on simulation scenario Usefulness of simulation in work procedures The simulator allows learning through team work very effectively It was worth participating in the simulation The clinical case was realistic This scenario developed clinical reasoning skills through the simulation experience There was possibility of clinical learning through the simulationysDegree of effectiveness of the simulator in re-creating the scenario (effectiveness as far as the proposed difficulties were similar to those of real cases experienced)Possibility of learning by efficiently working in a team The simulator permits me to learn the necessary procedures for patient management The simulation was a valuable learning experience The simulation session has improved my level of professional training I was satisfied with usefulness of debriefing after the simulation

Domain 2: Facilitator and Debriefing I received feedback during the debriefing that helped me to learn The facilitator provided feedback during the debriefing that helped me to develop my clinical reasoning skills I had the opportunity to reflect on and discuss my performance during the debriefing The debriefing provided an opportunity to ask questions The facilitator provided constructive criticism during the debriefing The facilitator explained important things during the debriefing The facilitator made me feel comfortable and ease during the debriefing The facilitator’s questions helped me to learn The facilitator was an expert There was a degree of competence in the management of the debriefing by the facilitator There was a degree of competence in the management of the scenario by the facilitator Reflecting on and discussing the simulation enhanced my learning

Domain 3: Clinical reasoning and self-effectiveness The simulation caused me to reflect on my clinical ability The simulation developed my clinical reasoning skills The simulation tested my clinical ability The simulation helped me to recognise patient deterioration early The simulation helped me to recognise my clinical strengths and weaknesses The simulation helped me to apply what I learned from the case study

Domain 4: Team dynamics and team factors My team members were interested and paid attention during the debriefing I participated actively in the debriefing after the simulation My peers provided feedback on the performance of the team as well as performance of individuals

Domain 5: Professional Impact During simulation the fundamentals learnt, will be helpful for work procedures 80% of what you learned during the simulation can be applied to your work The simulation helped me understand what my role would be in a similar emergency situation

Domain 6: Safeguards and materials The safeguards/materials were adequate in order to recreate the scenario The Health-system technology were adequate in recreating the scenario The setting was suitable in that it created the scenario

Domain 7: Difficulty and distress During the simulation I felt uncomfortable The simulation was a stressful moment/a source of anxiety I found it difficult to face the clinical case during the simulation

Table 2

Satisfaction scores of the participants

Domain	Session 2			Session 4
	Agree	Neutral	Disagree	Agree	Neutral	Disagree
Overall satisfaction	18 (82%)	4 (18%)	0	21 (95%)	1 (5%)	0
Facilitator and Debriefing	21 (95%)	1 (5%)	0	22 (100%)	0	0
Clinical reasoning and self-effectiveness	20 (91%)	2 (9%)	0	21 (95%)	1 (5%)	0
Team dynamics and team factors	20 (91%)	2 (9%)	0	21 (95%)	1 (5%)	0
Professional Impact	21 (95%)	1 (5%)	0	22 (100%)	0	0
Safeguards and materials	11 (50%)	11 (50%)	0	21 (95%)	1 (5%)	0
Difficulty and distress	10 (14%)	8 (36%)	11 (50%)	1 (5%)	5 (23%)	16 (72%)

Kirkpatrick level 2 (change in knowledge) was tested using pre-test and post-test with 60 MCQs. The mean values of pre-test and post-test scores are 24.22 ± 7 and 47.18 ± 5.6 respectively. The difference between the pre-test and post-test scores is statistically significant (P = 0.007).

DISCUSSION

The Human Patient Simulator (HPS) which is a high-fidelity full body human simulator manufactured by CAE company is present in our institute. HPS has inbuilt technology to mimic physiological changes. It has reactive eyes, palpable pulses, heart sounds, bilateral breath sounds and bowel sounds. It can analyse expired CO2 and inhalational agents in real time, along with recognizing drugs delivered through barcoding technology. HPS has realistic upper airway which allows laryngoscopy and intubation, and also difficult airway module can be mimicked with tongue and posterior oropharyngeal swelling. We can perform needle cricothyrotomy, needle thoracocentesis and pericardiocentesis. It can withstand a defibrillator shock, and a nerve stimulator and be attached to elicit thumb twitch response to neuromuscular blockade. These mannequins are designed to automatically determine patient responses to use of interventions with appropriate physiological responses. The scenario performance station is wirelessly connected with instructor workstation for better communication. The instructor workstation provides the instructor with scenario development tools and programmable patient physiology to create an immersive, realistic training environment.

Simulation based medical education (SBME) has proven to be effective in improving the knowledge, skill and behaviour of health care professionals. Simulation has been effectively used to teach anaesthesia, procedural skills, surgical skills and crises resource management.[111213141516171819] There is evidence of simulation being used to train individuals, teams, environments, technical factors, system factors and patient factors.[1819202122] In this present study, we aimed to teach peri-operative emergencies targeting patient care as our ultimate goal. To document this process, we captured the resident's reaction and change in knowledge to this teaching modality.

This study showed that anaesthesia residents had greater satisfaction following use of high-fidelity simulation as a learning tool for peri-operative emergencies. We also observed that the ratings for all criteria in the satisfaction scores were better for the fourth scenario than the second, which implies that our role as facilitators also improved. The residents have agreed that their clinical reasoning and self-effectiveness have improved, they started recognizing their role as the team leader or team member. Following the session two, many residents were not satisfied with the materials/makeup of the mannequin hence we modified our preparation for the following sessions and tried to make it as real as possible, hence in fourth session the number of responses for this criterion have improved. Thus, this questionnaire has helped us to identify our lacunae and act on it. After session two, about 14% of the residents felt that this learning had induced anxiety. Similar observations were seen by James WP and Thomas G, where students experienced anxiety while using high-fidelity simulators.[2324] This amount of anxiety is expected from residents as they are subjected to pressure of performance in front of their peers and teachers. Also, this helped us to modify the method of pre-briefing and tried to make them at ease during performing in the following sessions, which reduced the anxiety among the participants. This study reinforced the findings of studies by Parck CS, McCrossin and Burlacu CL who stated that use of high-fidelity simulation training has increased the confidence and decision-making skills among residents.[42526]

In this study, we used the MCQs as a means of assessing the efficacy of simulation sessions. The post-test was given immediately after debriefing. There was a significant improvement in the post-test scores compared to the pre-test scores. Similar improved knowledge following simulation class were seen by Gaba DM and Solymos O.[2728] Also, Curran VR has shown that simulation training helps in retaining the skills and higher confidence level in performing these skills later.[29] However, in the present study, we did not follow up the residents after the sessions to check the retention level.

A systematic review on best evidence in medical education by Issenberg SB et al. have identified the features of high-fidelity simulation that facilitate learning such as curriculum integration, feedback/debriefing, deliberate practice, mastery learning, capturing clinical variation, individualised learning opportunities and team training.[30] In the present study, we have integrated simulation into the resident curriculum, structured feedback was given, students were allowed to practice repetitively and in teams. The scenarios used also captured the clinical variations. Therefore, we have used many of the principles of best practices in this teaching learning methodology.

We have used purposive sampling and included only residents from our medical college hence our sample size is small to generalise the findings. Hence a larger study with residents from different regions would be ideal to confirm our findings. In the present study, we recorded the participant's reaction and change in knowledge to this educational intervention. We have not tested the change in behaviour (Kirkpatrick level 3) and result on patient care (Kirkpatrick 4). A longer period of follow-up and recording of the observations is required to get the whole effectiveness of this program. The present study also did not incorporate interdisciplinary team work i.e., including surgery residents and nursing faculty etc. Hence the holistic team dynamics was lacking. Future studies incorporating these features would be ideal.

Simulation as an educational tool compliments clinical exposure. It is an effective teaching learning tool for our adult learners in implementing Competence Based Medical Education (CBME). Teachers need to be trained to use this tool for education. However, future research with a larger number of participants is required to document the positive participant's response and record the change in behaviour or improvement in patient's outcome.

CONCLUSION

Hence we conclude that the use of high-fidelity simulation to train residents for “Peri-operative Emergencies” has greater satisfaction scores, provides and improves resident's clinical reasoning knowledge and skills. Therefore simulation teaching needs to be incorporated in the regular curriculum.

Financial support and sponsorship

Logistic support-Father Muller Skills and Simulation Centre, Intellectual support-FAIMER regional centre PSG Coimbatore.

Conflicts of interest

There are no conflicts of interest.