Simulation of Inpatient Medical Critical Events for Physicians at a Community Hospital.

Background: Critical events are common at community hospitals, yet physicians who lead them have had varying levels of training and involvement during their residency and professional development. Little is known about the impact of simulation to improve performance during inpatient critical events among community hospitalist physicians.
Objectives: To determine if hospitalist physicians reported sustained performance improvement regarding critical events as a result of simulation.
Methods: Physicians at a community hospital in Northern California participated in critical event simulation over one year. Self-assessment surveys (scale 1 through 5) were collected before, after, and at 1-month post-simulation. Differences in survey scores and post-simulation trends in total composite survey scores over a 1-month period were compared among participants.
Results: From February 2018 through February 2019, 25 of 32 eligible physicians (78%) participated in the simulations. Most were trained in internal medicine (76%), practiced primarily hospital medicine (72%), and had previous experience of at least 5 critical events per year (68%). Participants reported increases in mean survey scores (knowledge +0.8, familiarity +1.0, communication +1.2, technical skills +1.0) which were sustained at one month post-simulation (knowledge +0.8, familiarity +1.0, communication +1.3, technical skills +0.9) (all p < 0.0001). At one month post-simulation, participants who were clinic-based and had <5 years of post-residency experience had higher composite survey score differences compared to those who were hospital-based and had ≥5 years of experience, respectively (p < 0.05).
Conclusion: Simulation may lead to sustained performance improvement at critical events as reported by community hospitalist physicians. Further investigation is needed.

1. Introduction

Inpatient critical events are defined as severe, acute deteriorations in the clinical status of hospitalized patients. Examples of inpatient critical events include codes and rapid response team activations (RRTs) and are common at community hospitals.1–14 In one community-based study, there were 213 RRTs over a 14-month period and >10 code blue calls per 1000 discharges.15 Despite the frequency of critical events at community hospitals, hospitalist physicians who lead them have had varying levels of critical event training and involvement during their residency. A previous study found no standardized approach to critical care medicine education among internal medicine residents within a wide range of training environments. 16 In addition, early career physicians entering hospital medicine as a specialty after graduating from residency make up a growing proportion of hospitalists who lead codes and RRTs, and in a field traditionally comprised of internal medicine graduates, there is an increasing number of family medicine trained hospitalists in the United States. Family medicine residents spend less inpatient time during their residency because of the specific educational experiences in multiple areas required for graduation. Thus, simulation could help address gaps in physician knowledge and experience by providing a safe environment to improve performance in critical event situations without presenting any risk to patients.

While academic centers have reported data on simulation for inpatient critical events among healthcare providers,17–20 community-based studies on the use of critical event simulation specifically for non-academic hospitalist physicians are lacking. As of the time of this manuscript submission, we performed an Ovid MEDLINE® literature review combining medical subject headings and keyword search teams related to simulation, community hospitals, and critical events which yielded very few studies to date on this topic. Further, differences exist in organizational characteristics of codes and rapid response systems between academic centers and community hospitals,21 necessitating further investigation on the impact of simulation in community-based hospitals at which most patients receive inpatient care. One study found increased mortality at community hospitals compared to large teaching hospitals for common hospitalized conditions. 22 While the reasons for this association are not well understood, the finding may indicate a need for increased quality and safety initiatives at community hospitals for which critical event simulation may play a role. Another study surveyed physicians and nurses at a community hospital and found that familiarity with RRT criteria improved adherence to RRT activation, highlighting an opportunity for simulation to eliminate potential RRT barriers.23 As the Institute for Healthcare Improvement has indicated RRTs as a major initiative in its 100,000 Lives Campaign to decrease mortality,24 and medical simulation of inpatient critical events may increase competency and affect outcomes,25–27 this is an area that requires further investigation among physicians in the community hospital setting.

Thus, at our medical center, we conducted a pilot study of Simulation of Inpatient Medical critical events for Physicians At a Community hospiTal (the SIMPACT study), with the objective to determine if physicians reported improved and sustained knowledge, familiarity, communication and technical skills regarding inpatient critical events as a result of simulation participation.

2. Materials and methods

The SIMPACT study is a prospective cohort study conducted at Kaiser Permanente Santa Rosa Medical Center of physicians who practice hospital medicine at least part-time and who participated in simulated inpatient critical event scenarios. The primary objective was to determine if physicians reported sustained performance improvement at critical events as a result of simulation. The secondary objectives were to (1) compare mean differences of pre and post-simulation survey assessments, and to (2) compare trends in total composite scores over a 1-month time interval of post-simulation assessments between physician learners of different training backgrounds and levels of experience. The purpose of SIMPACT as a pilot study was to determine the feasibility and practicality of running critical event simulations among community hospital physicians at a single medical center, with the intent to conduct simulations on a larger scale across multiple medical centers within our healthcare system, Kaiser Permanente Northern California. The study was approved by the Kaiser Permanente Northern California Institutional Review Board.

Kaiser Permanente Santa Rosa Medical Center is a community-based medical center in Northern California that consists of a Joint Commission-accredited and licensed 173-bed acute care hospital and 5 medical office buildings. Dedicated hospitalist physicians provide continuous 24/7 inpatient care for patients admitted to the hospital. Hospitalized patients are also cared for by physicians who are primarily clinic-based and practice hospital medicine part-time. Additionally, the medical center has a healthcare simulation program which provides simulation experiences across multiple disciplines, coordinating with physician/nursing leadership and with departments such as quality and safety to address important and relevant issues within the medical center.

The Kaiser Permanente Santa Rosa Medical Center Simulation Program was created in 2012. Currently, the program is part of the medical center's Quality Care Management Department. The program is under the leadership of the Simulation Manager and the Simulation Medical Director. The program employs a full-time operations specialist, 12 standardized patients, and a family of high-fidelity simulators, as well as 2 low-fidelity simulators and several task trainers.

2.1. Recruitment

Simulation experience events were announced at Hospital Medicine Department meetings and by email. All physicians who actively practice hospital medicine at our medical center at least part-time were invited. Physicians who participated in previous code blue simulations mandated by medical center leadership in March 2016 were recruited to sign up on a voluntary basis; physicians hired after this time period were asked to sign up for 2 out of 4 sessions. Written consent was obtained from all participants, who were compensated for their participation with department administrative time and were blinded to the topics and content of each simulation session prior to participation.

2.2. Simulation design

Inpatient critical event simulations were held quarterly over the course of one year, with two 4-h sessions held each quarter for a total of 8 sessions. Scenarios were created by the Simulation Medical Director and developed with consideration for clinical importance and relevance in conjunction with and with input from multiple departments and specialties. Scenarios were reviewed by the Simulation Manager for feasibility and rehearsed prior to participation by study participants. The structure of each simulation session included a pre-scenario introduction, multiple scenarios, debriefings after each simulation, post-scenario didactics, and conclusion. Scenarios utilized various elements including a high-fidelity mannequin (SimMan®3G), standardized patients as patients and/or family members, and actors as nurses, respiratory therapists, nursing supervisors, specialists, and other clinical staff as appropriate. Participants took turns rotating as the Team Leader during the scenarios and also played various additional roles. Content chosen for the quarterly sessions were as follows: Quarter 1 – ventricular megacodes; Quarter 2 – thrombotic critical events (inpatient stroke alert, inpatient ST-elevation myocardial infarction [STEMI] alert, and massive pulmonary embolism); Quarter 3 – alcohol withdrawal and the combative patient; and Quarter 4 – RRTs (vital sign instability, altered mental status, and acute respiratory failure). Standardized patients who participated in simulations were trained a week prior to the simulations by the Simulation Medical Director and Simulation Manager on the case objectives, the simulation of symptoms, the expected actions of the study subjects, and the use of the standardized patient checklist. The standardized patient checklist for all simulations focused on the care experience aspect of the care provided during simulations. Standardized patients were included in the debriefings that followed each simulation.

2.3. Data collection

Surveys using Likert scales (1 through 5) eliciting responses in 4 domains (knowledge, familiarity, communication, and technical skill) were collected from all participants before, immediately after, and at 1-month post simulation session, from February 2018 through February 2019. Variables included gender, residency training (family medicine vs internal medicine), primary practice setting (clinic vs hospital), post-residency clinical experience (<5 years vs ≥ 5 years), previous critical event experience (<5 events per year encountered vs ≥ 5 events per year), and attendance at previously mandated code blue simulation sessions at our medical center, held in March 2016. The primary outcome was mean difference in survey scores, pre vs post-simulation and pre vs 1-month post-simulation. Using mean composite survey score (on a scale of 4 through 20) as a surrogate marker of overall self-reported performance rating, secondary outcomes included mean composite survey score differences immediately post-simulation and at 1-month post-simulation, and trends in mean composite score differences from post-simulation to 1-month post-simulation.

2.4. Statistical analysis

The nature of code blue, RRT, and other inpatient critical events in a community hospital setting allow for up to 2 physicians per scenario to optimize the educational experience of the simulation. As the hospital physician group at our medical center has approximately 30 physicians, we planned for 2 scenarios at each simulation session and 2 sessions per quarter, to allow for a total sample size of up to 32 participants to attend over the course of one year. We estimated that for physicians who participated in previous simulation training or had greater clinical experience, composite survey score differences would increase post-simulation by an average of 1 point, while for physicians who had not received previous simulation training or had <5 years of clinical experience, composite score differences would increase by a mean of 1.5 points. Using an alpha error of 0.05, we would need a sample size of 6 to achieve 80% power to detect a difference between the two groups. Thus, we were confident that with 32 participants in our study, we would detect a difference if present. Variables were reported as counts, percentages, standard deviations, and 95% confidence intervals. We used paired T tests to compare differences in pre vs post and pre vs 1-month post survey scores for the primary outcome. We also used paired T tests to compare mean composite survey score differences and trends in total composite score differences over a 1-month period between cohorts for the secondary outcomes. Statistical analyses were performed using SAS, version 9.3 (SAS Institute Inc., Cary, North Carolina).

3. Results

Between February 2018 and February 2019, 25 out of 32 eligible physicians (78%) participated in the inpatient critical event simulations held throughout the year. Fig. 1 details the demographics of study participants. Overall, most participants were trained in internal medicine (76%), practiced primarily hospital medicine (72%), and had previous experience of at least 5 critical events per year (68%).

Fig. 1

Characteristics of SIMPACT study participants, 2018–2019.

For the primary outcome, physicians who participated in critical event simulation reported significant increases in mean survey scores across all 4 domains elicited (knowledge +0.8, familiarity +1.0, communication +1.2, technical skills +1.0) which were sustained at one month post-simulation (knowledge +0.8, familiarity +1.0, communication +1.3, technical skills +0.9) (all p < 0.0001) (Table 1). Increases in mean survey scores were seen across the different inpatient scenarios by quarter (data not shown).

Table 1

Mean Survey scores of participants by learning Somains: Pre vs Post Simulation.

Survey Domains	Mean ScoreaPre Simulation	STD	Mean ScorePost*/1-MonthPost** Simulation	STD	P Value
Post Simulation
Knowledge	3.36	0.57	4.16	0.62	<0.0001
Familiarity	3.24	0.66	4.24	0.66	<0.0001
Communication	3.04	0.68	4.24	0.66	<0.0001
Technical Skills	3.08	0.49	4.08	0.70	<0.0001
1-Month Post Simulation
Knowledge	3.36	0.57	4.20	0.71	<0.0001
Familiarity	3.24	0.66	4.24	0.66	<0.0001
Communication	3.04	0.68	4.32	0.63	<0.0001
Technical Skills	3.08	0.49	3.96	0.79	<0.0001

Range of possible survey scores: 1 through 5.

Secondary outcomes included mean composite survey score differences of participants by variables, and trends in mean composite score differences from post-simulation to 1-month post-simulation. Female participants had higher mean composite score differences immediately after simulation and at 1-month post-simulation compared to males (p < 0.05). At 1-month post-simulation, participants who were primarily clinic-based and had <5 years of post-residency clinical experience also had significantly higher mean composite score differences compared to those who were primarily hospital-based and had 5 or more years of post-residency experience, respectively (p < 0.05) (Table 2). For trends in mean composite score differences over a 1-month interval post-simulation, participants who trained in family medicine demonstrated a significant rise compared to internal medicine-trained participants (p < 0.05) (Table 3), although there was no difference in overall composite score differences between the two groups at one month.

Table 2

Mean composite score differences by participant demographics: post simulation & 1-month post simulation.

Demographics	N (%)	Mean Composite Score Diff.a (Post - Pre Sim.)	STD	P Value	Mean Composite Score Diff. (1-Month Post - Pre Sim.)	STD	P Value
Sex
Male	15 (60%)	3.27	2.31	0.048	3.33	1.59	0.022
Female	10 (40%)	5.10	1.85		5.00	1.76
Residency Training
Family Medicine	6 (24%)	2.83	1.33	0.160	4.17	1.72	0.800
Internal Medicine	19 (76%)	4.37	2.43		3.95	1.90
Primary Practice Setting
Clinic	7 (28%)	4.71	2.50	0.340	5.14	2.34	0.049
Hospital	18 (72%)	3.72	2.22		3.56	1.42
Post-Residency Clinical Experience
<5 Years	10 (40%)	4.90	2.33	0.110	4.90	1.85	0.041
≥5 Years	15 (60%)	3.40	2.13		3.40	1.59
Previous Critical Event Experience
<5 Events Per Year	8 (32%)	4.38	2.00	0.590	4.50	1.85	0.360
≥5 Events Per Year	17 (68%)	3.82	2.46		3.76	1.82
Previous Simulation Experienceb
Yes	12 (48%)	4.42	2.02	0.393	4.42	2.07	0.282
No	13 (52%)	3.62	2.53		3.62	1.56

Maximum composite survey score is 20.

Defined by participation in 2016 medical center specific Code Blue simulation training (not inclusive of other simulation experiences).

Table 3

Post-simulation mean composite score change by participant demographics.

Demographic	Mean Composite Score Difference (1 Month Post Sim - Post Sim)	STD	P Value
Sex
Male	0.07	2.15	0.790
Female	−0.10	0.88
Residency Training
Family Medicine	1.33	1.37	0.030
Internal Medicine	−0.42	1.64
Primary Practice Setting
Clinic	0.43	0.53	0.260
Hospital	−0.17	2.01
Post-Residency Clinical Experience
<5 Years	0.00	1.49	1.000
≥5 Years	0.00	1.93
Previous Critical Event Experience
<5 Events Per Year	0.13	0.99	0.810
≥5 Events Per Year	−0.06	2.01
Previous Simulation Experiencea
Yes	0.00	1.13	1.000
No	0.00	2.20

Defined by participation in 2016 medical center specific Code Blue simulation training (not inclusive of other simulation experiences).

4. Discussion

To our knowledge, the SIMPACT study is the first of its kind to explore the impact of simulation on self-reported critical event performance among community-based hospitalist physicians and to describe the types of learners within this population who would derive the greatest benefit from simulation training.

While critical event simulation has been extensively studied in the academic environment among student learners and medical residents,28–32 little is known about the educational benefit of critical event simulation among hospitalist physicians in the community who have completed graduate medical education. Hospital medicine is the fastest growing medical specialty in the United States,33 and new hospitalist physicians begin their careers with varying levels of training in RRT and code blue response during their residency. Additionally, in a field traditionally dominated by internal medicine graduates who spend a large part of their residency on inpatient medicine, a growing proportion of hospitalists in the United States are now family medicine trained.34,35 Our study sheds new light on the positive impact of critical event simulation among early career hospitalists.

Limitations of our study include single center location, small sample size, absence of an observer assessment tool, and lack of direct correlation of critical event self-assessment ratings to clinical outcomes. Given that our population of interest was physicians at community-based hospitals (which are generally smaller in size than academic medical centers), SIMPACT was intended as a pilot study to determine if it was feasible and practical to run a community-based hospital study of critical event simulation. Our preliminary results revealed sustained critical event performance improvement as reported by hospital physicians at our medical center. Thus, a larger scale study is a potential next step through implementation of regional critical event simulation, as Kaiser Permanente Santa Rosa Medical Center is 1 of 21 medical centers that include hospital physicians who work for Kaiser Permanente, an integrated healthcare consortium, throughout Northern California. Future investigation to be conducted on a larger scale would include the use of validated observer assessment tools during simulations with interrater reliability (such as the Teamwork Assessment Scale36) and would follow clinical outcomes over time (such as hospital mortality, length of stay, and intensive care unit transfer).

Strengths of our study include longitudinal follow up, prospective design over the course of one year, and consistency of results across different types of simulation scenarios. Participants were surveyed one month after simulation and reported sustained improvements in all domains, generally about a 1-point post-simulation increase per domain. While it is difficult to interpret the clinical significance of this sustained increase at 30 days, the domains surveyed – knowledge, familiarity, communication, and technical skills – represent essential components of high-level performance at critical events. As an example, a code blue activation for a cardiac arrest requires knowledge of presenting cardiac rhythms, familiarity with the equipment utilized and the personnel comprising the code team, clear closed-loop communication among team members, and the technical skills of delivering effective shocks and administering quality chest compressions. All of these examples were covered in depth during our Quarter 1 ventricular megacode simulations, with feedback on the participants’ performance given during debriefings. A study of nearly 700 code blue calls revealed that factors such as presenting rhythm and duration of cardiopulmonary resuscitation were found to have a significant effect on survival, and problems encountered stemmed from issues with equipment and personnel.8 Thus, physicians who participated in our code blue simulations and reported sustained improvements in the surveyed domains may be better equipped than previously to identify factors for survival and mitigate problems that may arise. Additionally, our prospective cohort study design revealed differences in learner assessment by demographics and experience levels, with an overall trend that physicians who had less experience in hospitalized settings and with inpatient critical events gained greater knowledge, familiarity, and skills from participating in simulation. Positive results were consistent across different types of inpatient critical event scenarios, suggesting that simulation use to improve performance assessment as reported by participants can be used not only for codes and RRTs but various other inpatient critical events.

In conclusion, the findings of our pilot study support our hypothesis that physicians practicing hospital medicine reported increased and sustained confidence in managing critical events following simulation, with greater impact among early career physicians and those with less experience working in hospitalized settings. Further investigation on a larger scale with objective assessment tools is needed to see if results are reproducible and affect clinical outcomes.