A Mixed-Methods Realist Analysis of an Interdisciplinary Simulation Intervention for Psychiatry Residents.

OBJECTIVES: Simulation research in postgraduate psychiatry remains limited, with minimal studies on interdisciplinary involvement and mechanisms of change. To address these gaps, the authors implemented a mixed-methods realist analysis of an interdisciplinary simulation intervention administered to psychiatry residents.
METHODS: The University of Calgary implemented a simulation intervention for psychiatry residents. Eight junior residents participated in or observed 4 scenarios and eighteen senior residents participated in or observed 8 scenarios. Scenarios lasted 15 minutes with a pre-simulation orientation and post-scenario debrief. Most scenarios involved interdisciplinary staff. Scenarios included agitated and suicidal patients; treatment-related emergencies; and challenging conversations. All residents completed pre- and post-simulation surveys reporting confidence levels. Changes in confidence were analyzed using paired t tests and differences between junior and senior residents' confidence using ANOVA. Eleven residents participated in 2 focus groups. Transcripts were analyzed using a constant comparative model to identify contexts, mechanisms, outcomes, and the relationship between these realist categories. Key themes were extracted using generic theme analysis.
RESULTS: Aggregated survey data demonstrated statistically significant improvements in self-reported confidence for 7 of 8 proposed scenarios, with variations in confidence outcomes between junior and senior residents. Four themes emerged: (1) How Simulations Facilitate Learning, (2) The Role of Pre-simulation Instructions, (3) Factors Facilitating Confidence, (4) Positive Effects of Interdisciplinary Involvement.
CONCLUSIONS: This study identifies possible mechanisms for residents' self-reported improvements in learning and confidence, which may help programs tailor interventions. Furthermore, this study suggests there may be benefits to interdisciplinary simulations, with self-reported outcomes of improved collaboration and safety planning.

Simulations are an interactive learning experience meant to replicate real clinical scenarios. The process consists of a pre-simulation orientation, a simulated clinical scenario, and a post-scenario debrief. Simulations are used as an educational tool which allows learners to explore behaviors and solve problems in a relatively safe environment [1]. Literature suggests that simulations promote deeper learning, exposure to rare but serious events, and self-reflection, which are critical skills for trainees as residency programs transition to competency-based education models [1-8].

Although simulation interventions are well-studied within broader medical education literature, research on psychiatry-specific simulation interventions has been relatively limited [1-7]. Within psychiatry simulation literature, studies on interdisciplinary simulations remain limited [2, 3, 9]. Furthermore, there is a lack of assessment on which aspects of a simulation intervention may contribute to a desired outcome. A recent meta-analysis of psychiatry simulation literature has encouraged qualitative analyses to identify mechanisms of change [2, 3, 9].

The current study aims to address these gaps in the literature by using the realist lens to evaluate an interdisciplinary simulation intervention delivered to psychiatry residents. By studying an interdisciplinary simulation intervention, the authors hope to contribute to the minimal literature on the effects of interdisciplinary simulations within psychiatry [2, 3, 9]. In using a realist evaluation method, the authors’ goal was to develop an understanding of possible mechanisms of change within simulation interventions. Realist evaluations seek to answer, “what works for whom, in what circumstances, in what respects, and why?” [10]. In other words, a realist evaluation not only evaluates outcomes, but also emphasizes the importance of the hidden mechanisms that drive the process. It has been suggested that the realist evaluation is ideal when an intervention is broadly accepted as effective, but the goal is to optimize it or target a specific subgroup [10]. A mixed-methods approach is recommended for realist analyses and was employed in this study to allow for a richer exploration of the resident experience, including resident perspectives both before and after the intervention [11].

Methods

Study Design

In the 2018–2019 academic year, the University of Calgary Psychiatry Residency Program implemented an interdisciplinary simulation intervention for residents in their first, fourth, and fifth years of postgraduate training. Psychiatry residents in Calgary complete 5 years of postgraduate training, consisting of primarily in-person patient contact with both direct and indirect supervision. First year residents work in a mix of psychiatry and non-psychiatry settings and second to fifth year residents work exclusively in psychiatry. By fourth year, residents have had exposure to general adult, child and adolescent, and geriatric populations. Throughout all years of residency, residents routinely work on call in psychiatric emergency settings.

Eight first year psychiatry residents (referred to as “junior residents”) participated in 2 mandatory simulation sessions over 2 separate days, during which each resident participated directly in 1 scenario and observed 3 additional scenarios total. Eighteen fourth and fifth year psychiatry residents (referred to as “senior residents”) participated in 2 separate mandatory simulation sessions over 2 separate days, during which each resident participated directly in 2 scenarios and observed 6 additional scenarios total. Four scenarios were used with all residents (managing an agitated involuntary patient, managing a patient with akathisia, diagnosing and managing serotonin syndrome, and assessing and managing a suicidal patient). Senior residents engaged in four additional scenarios (sexual boundary violation, clozapine myocarditis, revoking a driver’s license from a patient with dementia, and an adverse outcome in electroconvulsive therapy).

All scenarios involved 1 standardized patient and most involved nursing, protective services, or psychiatrists as confederates. Confederates are members of the healthcare team acting in the simulation to enhance the fidelity, or believability, of the simulation [12]. A psychiatrist from the University of Calgary observed each scenario. A 5-min orientation was held before each simulation session, which involved multiple 15-min simulation scenarios (2 scenarios for junior residents’ sessions and 4 scenarios for senior residents’ sessions). A 20- to 30-min debrief was held after each scenario, which was led by the staff psychiatrist and included all residents and any confederates involved in the scenario. Simulations took place at the Medical Skills Centre in the University of Calgary.

All residents completed optional anonymous pre-simulation and post-simulation surveys. To preserve confidentiality, residents were assigned numbers to pair the pre- and post-surveys. Raw data was collated by team members not associated with the University of Calgary Psychiatry Residency Program. Six junior residents participated in an optional focus group held 8 weeks after their first simulation session. Five senior residents participated in a separate optional focus group held 4 weeks after their first simulation session. To preserve confidentiality, residents were assigned numbers in the focus group and did not mention their names or demographic data. Focus groups were transcribed by an independent online transcription service.

Ethics approval was received by the University of Calgary Conjoint Health Research Ethics Board (Ethics ID REB18-0810). Aggregate data from surveys and focus group transcripts will be stored for 5 years with no identifying information associated with the data. Informed consent was obtained from all participants prior to commencement of surveys and focus groups. Of note, this paper discusses a portion of the data produced from a larger study which involved both psychiatry residents and students enrolled in a University of Calgary Community Rehabilitation and Disability Studies program. The psychiatry resident data was extracted for this paper to focus solely on findings relevant to postgraduate psychiatry trainees.

Survey Analysis

Surveys collected demographic data and residents’ self-reported level of confidence in their ability to manage 8 different general scenarios both before and after the simulation sessions. Residents ranked their confidence using a 5-point Likert scale, where 1 represented “no confidence” and 5 represented “very confident.” The 8 scenarios were (1) managing a patient with suicidal ideation, (2) working with a nonverbal and frustrated patient, (3) managing an agitated patient, (4) working with an involuntary patient, (5) working with an intoxicated patient, (6) having a difficult conversation with a colleague where there is a difference of opinion, (7) working collaboratively with nursing to manage an adverse event, and (8) managing a psychiatric emergency. These 8 general scenarios were all directly encountered in at least one of the simulations and were chosen for their applicability to clinical practice. Confidence was chosen as a parameter as the authors hoped to further explore the nuances of changes in confidence and possible mechanisms contributing to this common finding [5, 6]. The limitations and possible utilities of measuring self-reported confidence are further explored in the “Discussion” section.

Self-reported confidence levels before and after the simulation were compared using a paired sample t test for the junior resident, senior resident, and aggregated data. The difference in confidence levels between junior residents and senior residents was determined by one-way analysis of variance (ANOVA). All statistical analyses were performed using SPSS version 26.0 software (SPSS INC, Chicago, IL, USA). Data were deemed statistically significant when p < 0.05. Of note, there was no correction for multiple comparisons given the exploratory nature of the analysis.

Focus Group Analysis

Focus groups were run by 2 trained members of the study team with semi-structured questions related to 3 topics: (1) how learners felt about their simulation experience; (2) what learners had learned from the simulation experience; and (3) how the experience has impacted their confidence and behaviors in day-to-day practice.

Transcripts were analyzed using a constant comparative model. In this model, responses are not grouped into predefined categories; rather, inductive reasoning is used to separate data into meaningful categories. Categories undergo definition changes as the data analysis proceeds. The number of incidents of each category and the relationships between categories are recorded. The results are then integrated into an explanatory model [13]. The realist framework dictated the type of categories and relationships derived from the data, with categories of interest being specific contexts, mechanisms, and outcomes identified by residents. The relationships of interest were connections between a given context, mechanism, and outcome (termed context-mechanism-outcome triads), a given context and mechanism (termed context-mechanism dyads), and a given mechanism and outcome (termed mechanism-outcome dyads). Three members of the 6-member study team coded each transcript to ensure inter-coder reliability. Of note, the study team included 2 psychiatry residents who were participants in the simulation intervention prior to joining the team; these residents were not involved in the coding of raw data from their cohort.

Using multi-coder interpretations, 69 contexts, 17 mechanisms, and 53 outcomes were initially identified in the data. As coding proceeded, similar codes were combined and codes that occurred fewer than 10 times total were removed to create a final 12 contexts, 16 mechanisms, and 14 outcomes. Throughout data collection, frequencies of context-mechanism-outcome triads, context-mechanism dyads, and mechanism-outcome dyads were tallied. The most frequent relationships were then analyzed using generic themes analysis and combined with key findings from the survey data to reveal pertinent themes.

Results

Participant Demographics

Five of 8 junior residents were female. Ages ranged from 24 to 28 with a mean of 26. Self-reported ethnicities were White for 4 residents; Chinese, Asian Canadian, and Arab for 1 resident each; and not reported for 1 resident.

Fourteen of 18 senior residents were female. Ages ranged from 26 to 39 with a mean of 32. Self-reported ethnicities were White for 12 residents; South Asian, Pakistani, African/Indian, and First Nations/White for 1 resident each; and not reported for 2 residents.

Survey Confidence Data

The key outcome of interest from the surveys was residents’ self-reported confidence ratings before and after the simulation sessions. For all scenarios, there was an improvement in confidence after the simulation sessions, the only exception being no change for senior residents in the suicidal patient scenario. Within the aggregated data set, the improvement in confidence was statistically significant for all the scenarios except the suicidal patient scenario. For senior residents, the improvement in confidence was statistically significant for 3 of the scenarios: having a difficult conversation with a colleague, working with a nurse to manage an adverse event, and managing a psychiatric emergency. For junior residents, the improvement in confidence was statistically significant for 4 of the scenarios: working with a nonverbal patient, an involuntary patient, an intoxicated patient, and having a difficult conversation with a colleague. This data is summarized in Table 1.

Table 1

Comparison of self-reported confidence ratings before and after the simulation intervention for aggregate, senior resident, and junior resident cohorts

Scenario	Aggregated dataa (n = 26)					Senior resident datab (n = 18)					Junior resident datac (n = 8)
	Mean pre-SIM rating	Mean post-SIM rating	Diff. post to pre	SD of diff. post to pre	p-value (sig < 0.05)	Mean pre-SIM rating	Mean post-SIM rating	Diff. post to pre	SD of diff. post to pre	p-value (sig < 0.05)	Mean pre-SIM rating	Mean post-SIM rating	Diff. post to pre	SD of diff. post to pre	p-value (sig < 0.05)
Managing a patient with suicidal ideation	4.00	4.19	0.19	0.90	0.28	4.28	4.28	0.00	0.84	1.00	3.38	4.00	0.63	0.92	0.10
Working with a nonverbal and frustrated patient	3.85	4.31	0.46	0.86	0.01*	4.44	4.61	0.17	0.71	0.33	2.50	3.63	1.13	0.84	< 0.01*
Managing an agitated patient	3.35	3.88	0.54	1.03	0.01*	3.67	4.11	0.44	0.98	0.07	2.63	3.38	0.75	1.17	0.11
Working with an involuntary patient	3.77	4.46	0.69	0.93	< 0.01*	4.11	4.44	0.33	0.69	0.06	3.00	4.50	1.50	0.93	< 0.01*
Working with an intoxicated patient	3.46	4.00	0.54	0.81	< 0.01*	3.94	4.28	0.33	0.77	0.08	2.38	3.38	1.00	0.76	< 0.01*
Having a difficult conversation with a colleague where there is a difference of opinion	3.08	3.96	0.89	1.11	< 0.01*	3.39	3.89	0.50	0.86	0.02*	2.38	4.13	1.75	1.17	< 0.01*
Working with nursing staff to collaboratively manage an adverse event	3.73	4.31	0.58	0.95	< 0.01*	3.89	4.33	0.44	0.78	0.03*	3.38	4.25	0.88	1.25	0.09
Managing a psychiatric emergency	3.04	3.85	0.81	1.02	< 0.01*	3.33	4.11	0.78	1.00	< 0.01*	2.38	3.25	0.88	1.13	0.06

For each of the above scenarios, residents rated their level of confidence in their ability to manage the scenario on a 5-point Likert scale, where 1 represented “no confidence” and 5 represented “very confident.” Confidence ratings were obtained immediately before the first simulation session (labelled “pre-SIM ratings”) and immediately after the second simulation session (labelled “post-SIM ratings”). The post-simulation ratings were compared to pre-simulation ratings using a paired sample t test for (a) the aggregate data, (b) the senior resident cohort, and (c) the junior resident cohort. Data were deemed statistically significant when p < 0.05, indicated by * in the above table

In comparing senior residents to junior residents, it is notable that senior residents had higher pre-simulation confidence ratings and that junior residents had a greater mean increase in confidence ratings in all scenarios. The difference between senior and junior residents’ confidence levels in the pre-simulation data was statistically significant for 7 of the 8 scenarios, the exception being working with nursing to manage an adverse event. After the simulation sessions, the difference between junior and senior residents’ confidence ratings was no longer statistically significant in 3 scenarios: working with an involuntary patient, a suicidal patient, and having a difficult conversation with a colleague. The comparison data is summarized in Table 2.

Table 2

The difference between senior residents’ and junior residents’ self-reported confidence ratings before and after the simulation intervention

Scenario	Difference in senior vs. junior data
	Difference in pre-SIMa	Pooled SD	p-value(sig < 0.05)	Difference in post-SIMb	Pooled SD	p-value(sig < 0.05)
Managing a patient with suicidal ideation	0.90	0.75	0.01*	0.28	0.39	0.11
Working with a nonverbal and frustrated patient	1.94	0.72	< 0.01*	0.98	0.65	< 0.01*
Managing an agitated patient	1.04	0.70	< 0.01*	0.73	0.81	0.04*
Working with an involuntary patient	1.11	0.64	< 0.01*	−0.06	0.59	0.83
Working with an intoxicated patient	1.56	0.93	< 0.01*	0.90	0.63	< 0.01*
Having a difficult conversation with a colleague where there is a difference of opinion	1.01	0.87	0.01*	−0.24	0.83	0.51
Working with nursing staff to collaboratively manage an adverse event	0.51	0.90	0.19	0.08	0.56	0.73
Managing a psychiatric emergency	0.95	0.86	0.02*	0.86	0.69	< 0.01*

For each scenario, the difference in confidence ratings between senior residents and junior residents was determined by one-way analysis of variance (ANOVA). The difference in ratings was analyzed for ratings made (a) before the simulation and (b) after the simulation. Data were deemed statistically significant when p < 0.05, indicated by * in the above table

Focus Group Data

The 10 most frequently identified context-mechanism-outcome triads derived from the focus group data are listed in Table 3 along with the frequencies of each relationship.

Table 3

Ten most frequent context-mechanism-outcome triads described in focus group data

Context	Mechanism	Outcome	Frequency
The active nature of the simulation intervention	The residents’ identification of their own learning gaps through confronting these gaps in the scenario	Better retention of learning compared to didactic interventions	24
The quality of pre-simulation instructions	The residents’ understanding of their role in the simulation setting	The feeling that abilities to perform in the scenario were limited	19
The quality of pre-simulation instructions	The fidelity of the simulation scenario allowing residents to immerse themselves in their role	The feeling of being unable to immerse oneself in the scenario	13
The quality of pre-simulation instructions	The residents’ understanding of their role in the simulation setting	The feeling of being unable to immerse oneself in the scenario	9
Residents’ pre-existing confidence in their skills	The opportunity to practically apply theoretical knowledge in the simulation setting	Recognizing the positive impact of pre-reading	8
Residents’ pre-existing confidence in their skills	The opportunity to practically apply theoretical knowledge in the simulation setting	Recognizing one’s ability to apply teaching to real-life practice	7
Opportunities to engage with the interdisciplinary team	The residents’ recognition of the value of interdisciplinary perspectives	Improved safety planning in real clinical settings	7
Opportunities to engage with the interdisciplinary team	The residents’ recognition of the value of interdisciplinary perspectives	Increased collaboration with the interdisciplinary team in real practice	6
The specific simulation scenario	The fidelity of the simulation scenario allowing residents to immerse themselves in their role	Better retention of learning compared to didactic interventions	6
The active nature of the simulation intervention	The residents’ identification of their own learning gaps through confronting these gaps in the scenario	Increased awareness of skills needing further development	6

The number of times a given context-mechanism-outcome triad was noted in the focus group data was recorded during data analysis. The above table displays the ten most frequently noted context-mechanism-outcome triads with the total frequency throughout all focus groups indicated in the final row

The 10 most frequently identified context-mechanism dyads were also of interest. The active nature of the simulation intervention was associated frequently with residents’ identification of their learning gaps through confronting these gaps in the scenario (frequency = 29); with residents’ perceptions of their ability to manage the scenario (frequency = 10); and with the opportunity for residents to practically apply their theoretical knowledge in the simulation (frequency = 9). Residents’ pre-existing confidence in their skills was also associated with the opportunity to practically apply their theoretical knowledge in the simulation (frequency = 25), and with residents’ feelings of confidence in the scenario (frequency = 11). The quality of pre-simulation instructions impacted residents’ understanding of their role in the simulation (frequency = 24) as well as the fidelity of the simulation and thus residents’ ability to immerse themselves in their role (frequency = 16). The specific simulation scenario also had an effect on the fidelity of the simulation and resident immersion in the scenario (frequency = 10). Opportunities to engage with the interdisciplinary team facilitated residents’ recognition of the value of interdisciplinary perspectives (frequency = 19). Receiving feedback from peers was associated with residents’ appreciation of their ability to change their practice based on the simulation learning (frequency = 9).

The 10 most frequently identified mechanism-outcome dyads were also of interest. Residents’ poor understanding of their role in the simulation was associated with feelings that their abilities to perform in the scenario were limited (frequency = 24) and that they were unable to immerse themselves in the scenario (frequency = 11). Residents’ identification of their learning gaps through confronting these gaps in the scenario (frequency = 24), the opportunity to discuss key learning in the debrief (frequency = 17), and residents’ appreciation of their ability to change their practice based on the simulation learning (frequency = 10) led to self-reported better retention of learning compared to didactic interventions. A lack of fidelity in the simulation led to residents being unable to immerse themselves (frequency = 21). Residents’ feelings of confidence in the scenario facilitated increased confidence in their ability to handle similar scenarios in real practice (frequency = 17). Residents’ sense of physical danger in the scenario also facilitated increased confidence in their ability to handle similar scenarios (frequency = 13). Residents’ feelings of fear in the simulation, however, were linked to the feeling that their abilities to perform in the scenario were limited (frequency = 13). Residents’ recognition of the value of interdisciplinary perspectives facilitated self-reported improved safety planning in real clinical settings (frequency = 11).

Themes Derived from the Data

Four clear themes in the survey and focus group data emerged, which highlight the most common outcomes and their contributing factors as described by resident participants. These are discussed in detail, each with an exemplifying quote from the transcripts.

Theme 1: How Simulations Facilitate Improved Learning

The broad outcome of improved learning was noted throughout the focus groups, which were held 1–2 months post-simulation. There were three nuanced learning outcomes self-reported by residents: (1) improved retention of learning points compared to didactic interventions; (2) increased awareness of their skills needing further development; and (3) a greater recognition of their ability to apply teaching to clinical practice. Each outcome was associated with unique contexts and mechanisms. Retention of learning was connected to the active nature of the simulation intervention, which allowed residents to identify their learning gaps through confronting these gaps in the scenario. Realistic scenarios allowing for high fidelity and discussions of key learning in the debrief were also connected to improved retention. The opportunity to receive feedback from peers in the debrief was uniquely connected to improved retention by encouraging residents to recognize their ability to change their practice based on the simulation learning. Increased awareness of skills needing development was fostered by the active nature of the simulation intervention and the opportunity this provided for residents to identify learning gaps. A greater recognition of the ability to apply teaching was associated with the opportunity for residents to practically apply their theoretical knowledge in the simulation setting. This opportunity was fostered by the active nature of the simulation as well as residents’ pre-existing confidence in their skills. These findings have been represented in their entirety as a logic diagram (Figure 1).

Fig. 1

Relationships between the contexts, mechanisms, and outcomes found to be associated with improved learning in focus group data

“…textbook reading is one thing, but when you see something, even if it’s artificial… it sticks in your mind better when you’ve actually seen a presentation versus just reading about it.”

Theme 2: The Important Role of Pre-simulation Instructions

A common theme throughout the focus groups was the effect of poor pre-simulation instructions on residents’ understanding of their role in the simulation and thus their ability to immerse themselves and perform well in the scenario. The pre-simulation instructions inadvertently provided limited detail around residents’ autonomy and capabilities in the simulation scenarios (that is, their ability to act independently and their access to various clinical tools). The negative impact on fidelity may have negatively impacted learning outcomes, given that fidelity was a key mechanism leading to better retention. Similarly, the negative effect on performance may have undermined residents’ opportunity to practically apply their theoretical knowledge and thus their recognition of their ability to apply teaching to practice. The negative effect on performance may also undermine improvements in confidence, given that feelings of confidence in the scenario were a key mechanism contributing to improved self-reported confidence.

“I think that I felt a little bit limited… I wasn’t too sure how far I should take it, because I wasn’t sure if I should be acting as the attending [psychiatrist].”

Theme 3: Factors Facilitating Improved Confidence

The survey data suggest that simulations improve residents’ self-reported confidence in their ability to manage real-life scenarios similar to those experienced in the simulation intervention. A key context that appears to play a role in this outcome is residents’ level of training. Junior residents had a greater mean increase in confidence ratings than senior residents for all scenarios. The increase in confidence for junior residents was significant enough in 3 scenarios (suicidal ideation, involuntary patient, and difficult conversation) that the difference between junior and senior confidence ratings shifted from being statistically significant pre-simulation to no longer statistically significant post-simulation.

The focus group data provided additional insights into other mechanisms that may facilitate the development of improved self-reported confidence. Confidence was described more specifically by residents as increased confidence in their ability to handle situations similar to the simulation scenarios in real life. The mechanisms which fostered this outcome were residents’ feelings of confidence in the scenario and a heightened sense of physical danger during the scenario. Conversely, feelings of fear in the scenario were associated with residents feeling that their abilities to perform were limited, which could undermine feelings of confidence in the scenario and thus undermine the improved confidence outcome.

“The [neuroleptic malignant syndrome] case that I encountered on call… I felt more confident going in and being confident about what my differential was and how I wanted to approach it.”

Theme 4: Positive Effects of Interdisciplinary Involvement

Throughout the focus groups, residents self-reported that the opportunity to engage with nursing and protective services in the simulation and debrief resulted in increased collaboration with the interdisciplinary team and improved safety planning in day-to-day practice. The key mechanism was a recognition of the value of interdisciplinary perspectives.

“I’ve started to communicate better with nursing staff as well as security about a patient, since they know [the patient] better. And kind of formulating a plan about safety, so I’ll talk with security about how the patient has been and what’s the best approach to approach the patient in terms of safety and what our plan would be if the patient gets agitated.”

Discussion

Improved learning was a broad outcome self-reported by residents throughout the focus groups, in keeping with current simulation literature [1, 2, 5–7]. The realist approach allowed for a more nuanced understanding of the type of learning that might take place during a simulation intervention and the aspects of the simulation that may contribute to learning outcomes. The most frequent specific learning outcomes self-reported by residents were improved retention compared to didactic interventions, increased awareness of skills needing further development, and increased recognition of one’s ability to apply teaching points to real-life practice.

While improved retention compared to didactic lectures is a documented outcome in existing simulation literature, increased awareness of skills needing development and increased recognition of one’s ability to apply teaching are more novel outcomes [14]. As postgraduate training programs adopt competency-based education models, such self-reflective skills are increasingly prioritized in training and practice [8]. The factors of the simulation intervention which were most often connected to these outcomes included the inherently active nature of participating in a simulation, which allowed residents to identify learning gaps and practically apply theoretical knowledge; the high fidelity of the simulation scenarios, which allowed for immersion into the role; and the debrief, which allowed for review of key learning points and discussions with peers that reinforced residents’ ability to change their practice. The roles of active participation, fidelity of a simulation, and the debrief/feedback in fostering learning outcomes are supported by existing literature [15, 16]. The current study has furthered this body of literature and provided additional understanding of which mechanisms may be associated with each specific learning outcome. This information may be helpful for programs hoping to maximize a particular learning outcome when developing a simulation intervention.

Notably, while active participation in the simulation was a key mechanism in learning outcomes, residents self-reported improved learning and confidence in all scenarios, not only those they directly participated in. While this may be accounted for by residents’ participation in all debriefs, it is feasible that the observer role may have also impacted learning outcomes. Existing literature suggests that observing a simulation may be as effective as participating for some learning outcomes [17-19]. Although the observer role was mentioned during focus groups, discussion of the observer role was minimal compared to the contexts and mechanisms noted above. The infrequent discussion of the observer role may be a reflection of residents’ perception that the observer role was less important, whether this was truly the case or not. There is evidence that learners may subjectively prefer direct participation over observation, even when the two roles have similar learning outcomes on objective measures [18, 19]. It is also possible that the benefits of the observer role were encompassed in discussions around the benefits of the debrief, which included observers. Existing literature supports the notion that including observers in the debrief enhances the utility of the observer role [17-19]. The limited discussion on the observer role may also be a result of the focus group structure, where residents were asked broad questions about learning without direction towards any specific simulation components. Ultimately, it is unclear exactly why the observer role was not extensively discussed by residents and the role it may have played in facilitating learning outcomes. Future qualitative analyses may consider looking directly at the interplay between participating, observing, and the debrief in learning outcomes.

An additional finding relevant to learning outcomes was the impact of minimal pre-simulation instructions on residents’ role. Residents reported that minimal instructions on their autonomy and capabilities in the simulation negatively impacted the simulation fidelity and their performance. Given the role of simulation fidelity and performance in the development of self-reported learning and confidence outcomes, minimal instructions on residents’ roles may undermine these outcomes. This hypothesized consequence of poor pre-simulation instructions is in keeping with an existing RCT which demonstrated that poor pre-simulation instructions reduced learners’ performance and subsequent development of confidence [20].

In addition to improved learning findings, improved perceived confidence was identified in the surveys and focus groups. This finding is consistent with existing literature demonstrating that improved confidence is a common outcome of postgraduate simulation interventions [5, 6]. The realist analysis allows for identification of possible mechanisms contributing to this outcome. Residents’ level of training appeared to play a role in development of confidence, with junior residents experiencing greater increases in confidence than senior residents. This was theorized to be due, at least in part, to residents’ level of experience. Existing literature demonstrates a connection between greater levels of experience and increased reports of confidence [21, 22]. Of note, junior residents had a broader range of pre-simulation confidence levels, which may be a confounding factor in their relatively greater change in confidence ratings due to regression to the mean. Similarly, a ceiling effect for senior residents might be a confounding factor, as senior residents had consistently higher baseline scores with less room to improve compared to junior residents. Additional factors contributing to improved self-reported confidence were participants’ feelings of confidence and sense of physical danger during the scenario. Interestingly, residents also identified that high levels of fear in the scenario contributed to decreased performance, which may undermine residents’ development of confidence. This balanced understanding of the role of arousal in performance and confidence during a simulation intervention is in keeping with the Yerkes-Dodson law as well as existing simulation literature, which suggests that negligible or excessive arousal could both result in reduced performance [23-25].

While improvements in self-reported confidence and the possible mechanisms contributing to this were clear outcomes of this study, the exact significance of residents’ improved confidence is unclear. There is a large body of literature focused on improved confidence as the primary outcome of simulation interventions [5, 6, 26]. However, there are also several studies suggesting that confidence does not necessarily correlate positively with competence and may not be a worthwhile outcome for study [22, 27, 28]. It is noted that these studies have assessed confidence at a given time in training rather than after a directed educational intervention. Conversely, there is literature demonstrating that improved confidence can positively correlate with competence in a given task after a directed educational intervention focused on that task [5, 29, 30]. There is also evidence that feedback and a formal debrief after a learning experience may facilitate a positive correlation between confidence and competence [31-33]. As such, it is possible that improved confidence in the current study may correlate with improved competence, though this cannot be confirmed without further objective analyses. Beyond a possible indication of improved competence, improved confidence may be a valuable outcome in its own right. There is literature suggesting that improved confidence may allow competent residents to appreciate their competency and further their efficacy; that it may reduce anxiety in independent practice; and that it may encourage perseverance in light of difficulties [26, 34, 35]. In future iterations of this research, the research team hopes to incorporate objective measures of competence and a greater focus on additional outcomes related to confidence in order to clarify the exact implications of this finding.

A final outcome of interest was the effect of including an interdisciplinary team in the simulation scenario and debrief. There is growing appreciation for the importance of interdisciplinary collaboration in delivering patient-centered care [36]. While there is literature demonstrating that interdisciplinary simulation interventions can improve interdisciplinary collaboration, there is minimal data on the impact of interdisciplinary simulations within psychiatry settings [2, 3, 9, 37–39]. Residents’ reports that the simulation intervention helped them appreciate the value of interdisciplinary perspectives, which resulted in self-reported improved collaboration and safety planning in real clinical settings, is a helpful addition to the current literature. The realist analysis has allowed for an understanding of a possible mechanism leading to these outcomes, namely inclusion of interdisciplinary team members in the debrief process, which provides helpful insight for programs hoping to maximize this outcome.

Limitations of this study include the relatively homogenous and small sample size, with all residents being part of the same residency program and only 11 residents participating in focus groups. Not all residents who participated in the simulation and surveys participated in the focus group and thus we cannot be sure whether some findings apply to all residents in the study or only to those who attended focus groups. Focus groups were voluntary and participants self-selected, which may have introduced a selection bias. Being in a focus group with well-known colleagues may have limited residents’ willingness to share their full or unique thoughts. It is possible that the survey on confidence may have biased residents into discussing this concept more than they otherwise would have during focus groups, though it is noted that residents discussed a wide variety of topics. Residents were not divided into junior and senior cohorts for focus group results, due to the low participation numbers and for simplicity. Based on analysis of the raw data sets, the two groups reported similar context-mechanism-outcome relationships with the same major themes emerging, however by combining data it is possible that minor differences between junior and senior residents were missed. Finally, the impact of gender, age, and ethnicity on self-reported confidence levels and on self-disclosure in the focus groups was not assessed and is an important consideration for future research.

In conclusion, while more literature on psychiatry simulation interventions is emerging, studies on the postgraduate psychiatry population remain relatively limited, with minimal studies on interdisciplinary involvement and on mechanisms of change [1–4, 9]. The current study adds to existing literature by demonstrating that self-reported improved learning outcomes and improved confidence appear to extend to the postgraduate psychiatry population; by revealing potential benefits of interdisciplinary involvement; and by providing insight into mechanisms that may foster the varied outcomes of simulation interventions. This more nuanced understanding of possible mechanisms of change may provide useful insights to programs developing simulation interventions by allowing them to tailor their interventions to maximize a desired outcome. While the purpose of this study was not to provide specific instructions for simulation delivery, the insights gained allow us to suggest the following broad considerations for programs developing simulation interventions for psychiatry residents:

Improved learning outcomes may be maximized by giving all residents the opportunity to participate directly; by incorporating a debrief which includes perspectives from both participants and observers; by maximizing the fidelity of scenarios; and by incorporating clear pre-simulation instructions on residents’ autonomy and capabilities in the simulation scenario.

If increasing perceived confidence levels is a high priority for training programs, there may be benefits to implementing simulations with more junior learners and ensuring inclusion of dangerous scenarios with a balanced approach to the level of fear generated.

If the goal is to promote improved collaboration and safety planning with interdisciplinary team members, the inclusion of interdisciplinary confederates in the simulation scenario and debrief should be considered.

Further research is required to confirm the self-reported outcomes identified in this study through objective measures. The impact of these outcomes on real-life practice would also be an important area for further research. Additional qualitative analyses of simulation interventions in the postgraduate psychiatry population will be valuable to add to the findings of the current study and provide support to the mechanisms of change identified or identify alternative mechanisms or understandings.