Objective performance of emergency medical technicians in the use of mechanical cardiopulmonary resuscitation compared with subjective self-evaluation: a cross-sectional, simulation-based study.

OBJECTIVE: To evaluate the subjective and objective resuscitation performance of emergency medical technicians (EMTs) using mechanical cardiopulmonary resuscitation (MCPR) devices. DESIGN AND
SETTING: This was a cross-sectional simulation-based study where participants installed the MCPR device on a training manikin. PARTICIPANTS: We assessed EMT-Intermediates (EMT-Is) and EMT-Paramedics (EMT-Ps) of the Emergency Medical Services (Ambulance) Division of the Taipei City Fire Department. PRIMARY AND SECONDARY OUTCOME MEASURES: The primary outcome was the gap between self-perceived (subjective) and actual (objective) no-flow time during resuscitation, which we hypothesised as statistically insignificant. The secondary outcome was the association between resuscitation performance and personal attributes like knowledge, attitude and self-confidence.
RESULTS: Among 210 participants between 21 and 45 years old, only six were female. There were 144 EMT-Is and 66 EMT-Ps. During a simulated resuscitation lasting between four and a half and 5 min, EMTs had longer actual no-flow time compared with self-perceived no-flow time (subjective, 38 s; objective, 57.5 s; p value<0.001). This discrepancy could cause a 6.5% drop of the chest compression fraction in a resuscitation period of 5 min. Among the EMT personal factors, self-confidence was negatively associated with objective MCPR deployment performance (adjusted OR (aOR) 0.66, 95% CI 0.45 to 0.97, p=0.033) and objective teamwork performance (aOR 0.57, 95% CI 0.34 to 0.97, p=0.037) for EMT-Ps, whereas knowledge was positively associated with objective MCPR deployment performance (aOR 2.15, 95% CI 1.31 to 3.52, p=0.002) and objective teamwork performance (aOR 1.77, 95% CI 1.02 to 3.08, p=0.043) for EMT-Is. Moreover, regarding the self-evaluation of no-flow time, both self-satisfaction and self-abasement were associated with objectively poor teamwork performance.
CONCLUSIONS: EMTs' subjective and objective performance was inconsistent during the MCPR simulation. Self-confidence and knowledge were personal factors associated with MCPR deployment and teamwork performance. Both self-satisfaction and self-abasement were detrimental to teamwork during resuscitation. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This study compared the subjective and objective out-of-hospital cardiac arrest resuscitation performance of emergency medical technicians (EMTs) in a quantitative manner.

The simulation process involved a mechanical cardiopulmonary resuscitation that is used in the prehospital setting worldwide, indicating that the results might be applied to other regions.

The study had a large sample size of 210 EMTs to test the hypothesis with adequate statistical power.

Using quantitative methods, the presence or absence of the ‘Dunning–Kruger effect’ in prehospital care providers could also be investigated.

The study was limited by a non-randomised design and a fixed crew size of three EMTs in the simulation setting, which may inhere a risk of selection bias and limited generalisability to other crew sizes.

Introduction

Survival of patients following out-of-hospital cardiac arrest (OHCA) depends on many factors, including bystander cardiopulmonary resuscitation (CPR), public access defibrillation and well-organised emergency medical services (EMS).1–3 Among them, emergency medical technicians (EMTs) play a pivotal role because they are usually the first medical personnel resuscitating the victim. EMTs perform CPR under many challenging circumstances, for example, on a quickly moving ambulance, and high-quality manual CPR is difficult and may cause occupational injuries.4–7 Thus, mechanical CPR (MCPR) has been widely applied to improve resuscitation quality and EMT safety.8–10

Although studies have not shown that MCPR in the prehospital setting did improve patient outcomes compared with manual CPR,11–15 current guidelines do support the use of MCPR devices under some circumstances, such as ambulance transport.16 17 To achieve the effective application of MCPR, it is important to eliminate the interruptions of chest compression before the commencement of mechanical compression.8 18 19 As a result, the EMS authorities of Taipei City started to introduce MCPR before ambulance transport, without changing the initial manual CPR procedures, for every OHCA resuscitation since 2018. In the densely populated metropolitan area of Taipei City, MCPR benefits patients and EMTs during ambulance transport and in circumstances when patients need to be carried down staircases. Given that MCPR devices—LUCAS or AutoPulse—are now incorporated into the standard protocol for OHCA resuscitation in Taipei City, sufficient training for the rapid and correct deployment of MCPR devices is crucial.

For medical professionals, a strong belief in their capabilities is helpful for successful resuscitation techniques.20 21 However, studies have also revealed the detrimental effects of overconfidence in both medical diagnosis and management.22 23 Besides, overconfidence might be associated with less competence, first reported in the psychology study as the Dunning–Kruger effect.24 This phenomenon was observed in certain areas25–28 but has never been studied in healthcare providers like EMTs.

Because we had observed, during the quality control processes at the Taipei City Fire Department, that EMTs did not usually accurately perceive the duration of chest compression interruptions (no-flow time) during deployment of MCPR devices, we designed a study to test the gap between self-perceived (subjective) and actual (objective) no-flow time during resuscitation, and the association between resuscitation performance and personal attributes like knowledge, attitude and self-confidence. By evaluating and comparing EMTs’ subjective and objective MCPR performance, we investigated whether the Dunning–Kruger effect is identifiable in EMTs performing resuscitation.

Methods

Study setting and participants

This cross-sectional study investigated EMT-Intermediates (EMT-Is) and EMT-Paramedics (EMT-Ps) of the EMS (Ambulance) Division of the Taipei City Fire Department. Taipei City is the capital of Taiwan, with approximately 2.5 million registered residents. The city’s EMS has a two-tiered response system consisting of a basic life support plus defibrillator (BLS-D) team and an advanced life support (ALS) team. The BLS-D team comprises more than 600 active EMT-Is, who have each completed at least 264 hours of training. They perform defibrillation and insert the supraglottic airway (SGA). The ALS team comprises more than 100 active EMT-Ps, specially trained for at least 1280 hours. The ALS team providers are authorised to perform endotracheal intubation and intravenous injections of resuscitation medications, like epinephrine, atropine and amiodarone, etc.

Each of the 41 BLS-D stations in Taipei City has two BLS-D ambulances staffed by two EMT-Is, although sometimes there is a third member (usually a volunteer EMT). Each of the four ALS stations in Taipei City has three ALS ambulances staffed by three EMT-Ps.

The Taipei City Fire Department has a single dispatch centre to process all incoming EMS calls. All dispatchers are required to complete at least 40 hours of training on priority dispatch. BLS-D is the universal response for all dispatch calls. For OHCA cases, additional ALS teams would be dispatched to the scene with BLS-D teams.

Recruitment and simulation process

Study participants were selected from 734 EMTs in 45 Taipei City Fire Department stations between 19 January and 16 April 2019. To qualify for participation, EMTs must have completed: (1) the re-education training course in the previous year, (2) the 2-hour high-performance CPR training course and (3) the 4-hour MCPR training course.

We visited about two or three stations every few days for 4 months. There was no randomisation process, and the impromptu visits depended on our daily duties. Selected stations were not notified before visitation to prevent them from preparing beforehand. We explained the purpose and process of the study after arriving at each station and picked EMTs who were on BLS-D/ALS duties at the time. After written informed consent was acquired, the three EMTs were randomly assigned a role as either team leader, compressor or airway manager. EMTs who participated in the study completed a questionnaire before the test scenario, stating their personal data, resuscitation-related knowledge, attitude and self-confidence. We picked and tested a few more EMT teams at the station if time permitted.

The MCPR device in the test scenario was either LUCAS or AutoPulse, depending on which was available at each station. The CPR training manikin was a Resusci Anne quality CPR (QCPR) device produced by Laerdal Medical, which recorded chest compressions’ rate, depth and interruptions. The entire simulation process was recorded by two video cameras placed at the foot and head ends of the manikin, respectively.

The test scenario started with the compressor and airway manager performing CPR on the manikin with a 30:2 compression–ventilation ratio. Besides performing bag-valve-mask (BVM) ventilation, the airway manager also set up the automated external defibrillator (AED) and inserted the SGA device—i-gel. Ventilation was changed to one breath every five to six seconds after successfully placing the SGA device. During the second AED heart rhythm analysis, the compressor and airway manager switched roles, and the compressor and airway manager started MCPR deployment. The team leader observed the entire process and offered advice, encouragement or a helping hand to the team members. The simulation process ended once the MCPR device was deployed.

After the test scenario, each EMT self-evaluated their performance and no-flow time during the first heart rhythm analysis, BVM ventilation, i-gel insertion, second heart rhythm analysis and MCPR deployment, respectively. These analyses were documented as subjective outcomes. An expert committee consisting of two experienced paramedics gave objective scores by examining the videos and measured actual no-flow time using timers.

Questionnaire and score sheet

The questionnaire (online supplemental appendix I) covered personal data such as EMT certification levels, years of service and prior resuscitation experience. The first three sections contained questions about knowledge, attitude and self-confidence. They were designed according to a previous study by Hsieh et al.29 Knowledge was quantified by the number of questions answered correctly in section 2 of the questionnaire, with a score ranging from 0 to 10. Attitude was quantified by questions 1–6 in section 3, with a score ranging from 6 to 30. Self-confidence was quantified by questions 7–12 in section 3, with a score ranging from 6 to 30. The fourth section was the self-evaluation of performance and no-flow time.

The expert committee’s objective score sheet (online supplemental appendix II) contained sections on chest compression, airway management, MCPR deployment and overall performance—each included three to five scoring items. MCPR deployment performance was quantified by combining the scores ranging from 2 to 10 of items 1 and 2 in section 2. Teamwork performance was quantified by combining scores ranging from 4 to 20 of items 1–4 in section 3. The grading of airway management (BVM ventilation and i-gel operation), MCPR deployment and overall performance was done by two EMT-P instructors with at least 15 years of emergency management experience. The CPR quality gradings before MCPR deployment were automatically recorded by the QCPR device.

The questionnaire and the score sheet went through three Delphi rounds by five physicians, with all questions and scoring items achieving at least 80% of the content validity index.

Pilot test and sample size estimation

A pilot test with eight three-crew teams containing 12 EMT-Is and 12 EMT-Ps had been performed for questionnaire optimisation, scoring standard adjustment and sample size estimation. The study was estimated to require a sample size of 36 to achieve a power of 80% and a significance level of 5% (two-sided) to detect a mean difference of 4 s between subjective and objective no-flow time, assuming a SD of 8.3 s.

Statistical analysis

Continuous variables included no-flow time (in seconds) calculated by timers, questionnaire scores, age, years of service, number of OHCA dispatches, number of OHCA survivals and number of MCPR deployments. These variables were summarised in medians with IQR. Alternatively, categorical variables included the EMT (EMT-P or EMT-I) level, sex, appraisal of MCPR deployment and teamwork performance. Inferior performance was defined as scores lower than the 25th percentile, and superior performance was defined as scores higher than the 75th percentile. We chose to focus on the extremes of the upper and lower quartiles to evaluate factors contributing to the best and worse resuscitation performance. The categorical data were analysed with Pearson’s χ2 test. Inferential statistics were done using the Wilcoxon signed-rank (matched pairs) test, the two-sample Wilcoxon rank-sum (Mann–Whitney) test and simple and multivariable logistic regression analyses. Cohen’s Kappa statistics were used to measure inter-rater reliability. The linear regression model was applied to analyse the relationship between self-cognition and objective teamwork performance. Data were collected and analysed using R-3.5.3 and SPSS Statistics V.22.0 (IBM, USA).

Ethics approval

This study was approved by the Research Ethics Committee of the National Taiwan University (NTU-REC No. 201811HS013).

Patient and public involvement

Patients and the public were not involved in the study.

Results

Seventy groups (44 EMT-I groups, 20 EMT-P groups and 6 mixed groups of 2 EMT-Is and 1 EMT-P each) were included in this study, as shown in figure 1. Their baseline characteristics and self-confidence in MCPR deployment are shown in table 1. The subjective no-flow time was significantly longer than the objective no-flow time among all EMTs, as shown in table 2. This discrepancy also existed among EMT-P and EMT-I subgroups (table 2). EMT-Ps had better scores than EMT-Is in both subjective and objective MCPR deployment performance (table 2). Self-confidence was negatively associated with objective MCPR deployment performance among EMT-Ps, while knowledge was positively associated with objective MCPR deployment performance among EMT-Is (table 3). Self-confidence was negatively associated with objective teamwork performance among EMT-Ps, while knowledge was positively associated with objective teamwork performance among EMT-Is (table 4).

Figure 1

Overview of the recruitment process. CPR, cardiopulmonary resuscitation; EMT, emergency medical technician; EMT-I, EMT-Intermediate; EMT-P, EMT-Paramedic; MCPR, mechanical cardiopulmonary resuscitation.

Table 1

Baseline characteristics and self-confidence in MCPR deployment

	Participants (n=210)	EMT-I (n=144)	EMT-P (n=66)	P value	Confident† (n=72)	Unconfident (n=138)	P value
Age
21–28	58	49	9	0.006*	16	42	0.132
29–31	47	38	9		12	35
32–36	55	27	28		24	31
37–45	50	30	20		20	30
Sex
Male	204	140	64	0.919	72	132	0.174
Female	6	4	2		0	6
Service duration (years)
≤1	23	22	1	0.001*	3	20	0.02*
2–4	71	62	9		19	52
5–7	45	31	14		15	30
8–9	21	14	7		7	14
≥10	50	15	35		28	22
Number of OHCA dispatches
0–5	54	51	3	0.001*	18	36	0.152
6–14	51	44	7		14	37
15–30	54	41	13		16	38
31–500	51	8	43		24	27
ROSC number
0–2	116	104	12	0.001*	33	83	0.001*
3–5	41	29	12		12	29
6–60	53	11	42		27	26
Number of survivals to discharge
0–2	160	136	24	0.001*	44	116	0.001*
3–15	50	8	42		28	22
Number of CPC 1–2
0–2	165	138	27	0.001*	46	119	0.001*
3–15	45	6	39		26	19
Number of MCPR deployments
0–3	80	74	6	0.001*	17	63	0.019*
4–5	26	22	4		11	15
6–15	52	37	15		21	31
16–180	52	11	41		23	29

The data were analysed with Pearson’s χ2 test.

*P value less than 0.05.

†Self-confidence in MCPR deployment is measured as a score of 5 out of 5 in both questions 11 and 12 in section 3 of the questionnaire (online supplemental appendix I).

CPC, cerebral performance category; score of 1 or 2 indicates a good neurologic outcome for OHCA survivors; EMT-I, emergency medical technician-intermediate; EMT-P, emergency medical technician-paramedic; MCPR, mechanical cardiopulmonary resuscitation; OHCA, out-of-hospital cardiac arrest; ROSC, return of spontaneous circulation.

Table 2

Subjective versus objective no-flow time and MCPR deployment performance

		All EMTs (n=210)		P value
		Subjective no-flow time(in seconds)†	Objective no-flow time(in seconds)†
First AED analysis(and defibrillation, if needed)		10 (7–11)	16 (13–18)	<0.001*
BVM ventilation		3 (2–3)	3 (2–4)	0.008*
Insertion of i-gel		2 (1–3)	3 (2–4)	<0.001*
Second AED analysis(and defibrillation, if needed)		10 (7–15)	18 (14–21)	<0.001*
MCPR deployment		10 (8–15)	16 (13–23)	<0.001*
Overall		38 (27.75–47)	57.5 (49–68)	<0.001*
		EMT-P(n=66)	EMT-I(n=144)	p value
Subjective no-flow time during MCPR deployment (in seconds)†		8 (6–15)	10 (8–18.75)	0.002*
Objective no-flow time during MCPR deployment (in seconds)†		13.5 (10–19)	17 (14–24)	<0.001*
Subjective performance‡	Related operation	4.5 (4–5)	4 (3–5)	0.006*
	Correct deployment	5 (4–5)	4 (3–5)	0.001*
Objective performance‡	Related operation	5 (4–5)	4 (3–4.75)	<0.001*
	Correct deployment	5 (4–5)	4 (3–5)	0.018*

The data format is median with an IQR. The data were analysed with the two-sample Wilcoxon rank-sum (Mann–Whitney) test.

*P value less than 0.05.

†The subjective no-flow time comes from questions 12 and 13 in section 4 of the questionnaire (online supplemental appendix I); the objective no-flow time during MCPR deployment comes from item 3 in section 2 of the score sheet (online supplemental appendix II).

‡The subjective scores for related operation and correct deployment come from questions 7 and 8, respectively, in section 4 of the questionnaire (online supplemental appendix I); the objective scores of related operation and correct deployment come from items 1 and 2, respectively, in section 2 of the score sheet (online supplemental appendix II). Related operation refers to the initiation, pausing, and troubleshooting of the MCPR device, whereas correct deployment refers to the correct positioning of the MCPR device.

AED, automated external defibrillator; BVM, bag-valve-mask; EMT, emergency medical technician; EMT-I, emergency medical technician-intermediate; EMT-P, emergency medical technician-paramedic; MCPR, mechanical cardiopulmonary resuscitation.

Table 3

Factors† related to the MCPR deployment performance‡ among EMT-Ps and EMT-Is

	MCPR deployment performance among EMT-Ps						MCPR deployment performance among EMT-Is
	Inferior(n=24)	Superior(n=24)	OR(95% CI)	P value	aOR(95% CI)	P value	Inferior(n=41)	Superior(n=38)	OR(95% CI)	P value	aOR(95% CI)	P value
Age	35(32–40)	34(30.3–36)	0.92(0.82 to 1.03)	0.15	0.93(0.82 to 1.05)	0.214	30(27–34)	30(27–36)	1.01(0.95 to 1.07)	0.878
Service duration (years)	4(3–5)	5(3–5)	1.09(0.11 to 4.53)	0.986			2(2–3)	2(2–3)	1.71(0.49 to 6)	0.325
Number of OHCA dispatches	80(30–150)	50(30–100)	0.99(0.99 to 1.01)	0.657			6(5–15)	10(4.8–18)	1.02(0.98 to 1.05)	0.359
ROSC number	10(4–18)	10(5–15)	0.99(0.95 to 1.03)	0.632			1(0–2)	1(0–3)	1.02(0.89 to 1.16)	0.829
Number of survivals to discharge	5(1–9)	5(2.3–6.8)	0.96(0.82 to 1.17)	0.601			0(0–1)	0.5(0–1)	1.2(0.81 to 1.77)	0.362
Number of CPC 1–2	5(1–7)	4(2–6)	0.99(0.85 to 1.17)	0.958			0(0–1)	0(0–1)	1.1(0.71 to 1.72)	0.674
Number of MCPR deployments	32(10–50)	27.5(14–46)	0.99(0.98 to 1.01)	0.738			3(2–6)	4(2–10)	1.07(0.99 to 1.15)	0.101	1.08(0.99 to 1.18)	0.052
Knowledge	9(9–10)	9(9–10)	1.07(0.5 to 2.3)	0.855			8(8–9)	8(8–9)	2.02(1.25 to 3.27)	0.004*	2.15(1.31 to 3.52)	0.002*
Attitude	29(27–30)	28(26–30)	0.84(0.61 to 1.14)	0.263			27(25–29)	28(25.5–30)	1.03(0.92 to 1.15)	0.654
Self-confidence	29(28–30)	29(26–30)	0.66(0.46 to 0.96)	0.028*	0.66(0.45 to 0.97)	0.033*	26(22–29)	25(22–28)	0.98(0.89 to 1.08)	0.687

The data format is median with an IQR. The data were analysed with simple and multivariable logistic regressions.

*P value less than 0.05.

†The first seven factors are from section 1 of the questionnaire (online supplemental appendix I); the knowledge score is the number of questions answered correctly in section 2; attitude and self-confidence are quantified by questions 1–6 and questions 7–12, respectively, in section 3.

‡The MCPR deployment performance is quantified by the combined scores of items 1 and 2 in section 2 of the score sheet (online supplemental appendix II); inferior performance is defined as combined scores less than the 25th percentile, whereas superior performance is defined as scores higher than the 75th percentile.

aOR, adjusted OR; CPC, cerebral performance category; EMT-I, emergency medical technician-intermediate; EMT-P, emergency medical technician-paramedic; MCPR, mechanical cardiopulmonary resuscitation; OHCA, out-of-hospital cardiac arrest; ROSC, return of spontaneous circulation.

Table 4

Factors† related to the teamwork performance‡ among EMT-Ps and EMT-Is

	Teamwork performance among EMT-Ps						Teamwork performance among EMT-Is
	Inferior(n=24)	Superior(n=24)	OR(95% CI)	P value	aOR(95% CI)	P value	Inferior(n=41)	Superior(n=38)	OR(95% CI)	P value	aOR(95% CI)	P value
Age	35.5(34.8–38.5)	31(29–34.3)	0.84(0.74 to 0.96)	0.012*	0.87(0.74 to 1.03)	0.102	29(25–34)	30.5(27.3–36.8)	1.04(0.97 to 1.11)	0.301
Service duration (years)	5(3–5)	3.5(2–5)	0.6(0.37 to 0.98)	0.043*	0.88(0.45 to 1.72)	0.713	2(2–3)	2(2–3)	1(0.29 to 4.86)	0.932
Number of OHCA dispatches	89(30–150)	55(27.5–170)	0.99(0.99 to 1.01)	0.821			10(5–15)	7(5–15)	0.99(0.96 to 1.03)	0.681
ROSC number	10(4.8–15.8)	10(3.8–15)	0.98(0.94 to 1.03)	0.442			1(0–3)	1(0–3)	0.99(0.85 to 1.16)	0.886
Number of survivals to discharge	6(3.25–8.25)	5(1–7)	0.95(0.82 to 1.11)	0.542			1(0–1)	0(0–1)	0.79(0.49 to 1.28)	0.186	0.79(0.52 to 1.22)	0.286
Number of CPC 1–2	5(2.5–6.25)	3.5(1–6)	0.91(0.76 to 1.09)	0.322			0(0–1)	0(0–0.75)	0.79(0.49 to 1.28)	0.345
Number of MCPR deployments	30(14.3–48.5)	20(10–42.5)	0.99(0.98 to 1.01)	0.624			3(2–10)	4(2–9.8)	1(0.94 to 1.07)	0.958
Knowledge	9(8–9.3)	9(9–10)	1.61(0.72 to 3.59)	0.245			8(8–9)	9(8–9)	1.78(1.03 to 3.08)	0.038*	1.77(1.02 to 3.08)	0.043*
Attitude	29(28–30)	28(26–30)	0.71(0.5 to 1.01)	0.059	0.91(0.59 to 1.41)	0.667	27(25–28)	27(25–29.8)	1(0.88 to 1.15)	0.94
Self-confidence	30(29–30)	27(25.8–30)	0.46(0.27 to 0.78)	0.004*	0.57(0.34 to 0.97)	0.037*	26(22–28)	24(22–27.8)	0.92(0.81 to 1.04)	0.178	0.92(0.8 to 1.05)	0.204

The data format is median with an IQR. The data were analysed with simple and multivariable logistic regressions.

*P value less than 0.05.

†The first seven factors are from section 1 of the questionnaire (online supplemental appendix I); the knowledge score is the number of questions answered correctly in section 2; attitude and self-confidence are quantified by questions 1–6 and questions 7–12, respectively, in section 3.

‡The teamwork performance is quantified by the combined scores of items 1–4 in section 3 of the score sheet (online supplemental appendix II); inferior performance is defined as combined scores less than the 25th percentile, whereas superior performance is defined as scores higher than the 75th percentile.

aOR, adjusted OR; CPC, cerebral performance category; EMT-I, emergency medical technician-intermediate; EMT-P, emergency medical technician-paramedic; MCPR, mechanical cardiopulmonary resuscitation; OHCA, out-of-hospital cardiac arrest; ROSC, return of spontaneous circulation.

The no-flow time during MCPR deployment negatively correlated with objective MCPR deployment performance (figure 2A) and objective teamwork performance (figure 2B). Additionally, self-satisfaction and self-abasement in evaluating no-flow time were associated with poor objective teamwork performance, but only the former showed statistical significance (figure 3). Unlike self-confidence, self-satisfaction was assessed based on objective no-flow time greater than subjective no-flow time, whereas self-abasement was assessed based on subjective no-flow time greater than objective no-flow time.

Figure 2

Correlation between no-flow time during MCPR deployment (range, 7–53 s) and resuscitation performance among 210 EMTs (70 groups). (A) Correlation between no-flow time and MCPR deployment score (items 1 and 2 in section 2 of the score sheet; range, 2–10), R2=0.58. (B) Correlation between no-flow time and teamwork score (items 1–4 in section 3 of the score sheet; range, 5–20), R2=0.12. EMT, emergency medical technician; MCPR, mechanical cardiopulmonary resuscitation.

Figure 3

Association between self-cognition (objective no-flow time minus subjective no-flow time during MCPR deployment, in seconds; range, −14 to 42) and objective teamwork performance (items 1–4 in section 3 of the score sheet; range, 5 to 20). Closed dots represent EMT-Ps, whereas open dots represent EMT-Is. EMT-I, emergency medical technician-intermediate; EMT-P, emergency medical technician-paramedic; MCPR, mechanical cardiopulmonary resuscitation.

Cohen’s Kappa coefficients of the objective gradings of MCPR device operation (online supplemental appendix II, section 2, item 1), correct MCPR device deployment (online supplemental appendix II, section 2, item 2) and overall performance (online supplemental appendix II, section 3) were 0.803, 0.822 and 0.736, respectively. All the p values were less than 0.001.

Discussion

There were three major findings in this study. First, the subjective and objective MCPR performance of EMTs was inconsistent. The Dunning–Kruger effect was identifiable in EMTs performing resuscitation in our study. Second, many EMT personal factors such as self-confidence and knowledge contributed to resuscitation performance, ignoring EMT certification levels. Third, EMTs who were neither self-satisfied nor self-abased in evaluating no-flow time had better objective teamwork performance.

Both EMT-Ps and EMT-Is lacked sufficient self-awareness of their no-flow time during resuscitation. Among all EMTs, the median no-flow time was 19.5 s longer than the self-perceived length, which would cause a 6.5% drop in chest compression fraction in a 5-min resuscitation period. The discrepancy between subjective and objective no-flow time has already been reported, and objective feedback was recommended to decrease the duration.30 We posit that recognition of interruption time is not accurate while EMTs concentrate on specific emergency medical techniques, particularly during urgent OHCA resuscitations. In these situations, adequate leadership and team-focused resuscitation should play a beneficial role.31 For example, team leader verbalisation makes EMTs aware of errors or encourages them to keep up the good work, reinforcing positive performance outcomes.32 33

During MCPR deployment, EMT-Ps had subjective and objective median no-flow time of 8 and 13.5 s, respectively. These durations were both shorter than that of EMT-Is (10 and 17 s, respectively). The discrepancy between subjective and objective no-flow time was also smaller in the EMT-P group. This indicates that EMT-Ps are more capable of recognising and reducing unnecessary disruptions during resuscitation, which is expected because EMT-Ps are better trained. However, the five-and-a-half-second gap in the EMT-P group indicates that even well-trained EMTs have unrecognised shortcomings in their resuscitation process.

For EMTs, MCPR deployment confidence was positively associated with the number of completed deployments, as shown in table 1. This association has also been observed in other resuscitation techniques, such as endotracheal intubation.21 Operator confidence is likely to affect resuscitation performance. Davis et al.34 have revealed a positive association between self-confidence and ultrasound accuracy by novice emergency physicians. In contrast, we found a negative association between self-confidence and MCPR deployment performance among EMT-Ps. We speculate that operator overconfidence plays a detrimental role because excessive complacency leads to detail neglect during resuscitation.

For EMT-Is, knowledge was positively associated with MCPR deployment and teamwork performance. EMT-Is may be unfamiliar with OHCA resuscitation because OHCA cases comprise a small fraction of their usual emergency medical dispatches. We believe that their resuscitation performance and outcomes will improve by enhancing their knowledge base. As for EMT-Ps, the negative association between self-confidence and performance indicates that highly trained EMTs may be careless. We believe that these shortcomings can be improved with a debrief after resuscitation. The debriefing process can enhance objective perception about resuscitation performance, balancing self-satisfaction or self-abasement.

The no-flow time negatively correlated with objective MCPR deployment performance and objective teamwork performance. The latter played a minor role, with a more moderate slope of regression line. However, the negative correlation between no-flow time and objective teamwork performance indicates the merit of teamwork-focused resuscitation in favour of personal ability. Planning, leadership and communication are the key elements of successful coordination during resuscitation.35 36 Other studies also indicated that simulation-based training could improve quality of care and treatment outcomes of resuscitation.37 38 Therefore, EMS authorities should develop team-oriented and simulation-based training models for EMT training and re-education courses. Additionally, instructor-led, low-dose but high-frequency training may be beneficial by reinforcing resuscitation skills retention.39 Moreover, our study first observed the Dunning–Kruger effect in EMTs’ resuscitation performance. This finding is important for the training of EMTs because even just keeping in mind that the effect exists can help the trainers and trainees stay humble, accept criticism and keep learning new knowledge and skills.

This study has the following limitations. First, the participant recruitment process lasted nearly 4 months, and the impromptu visits and selections were not randomised, which may inhere a risk of selection bias. Although we visited stations without prior notification, the EMTs who were tested later in the recruitment period might still have been aware and able to prepare; they might have been informed of the details of our simulation from their colleagues. Regardless, the results of participants who tested earlier showed no significant difference to those who tested later. Second, we discovered that almost all stations used half-length manikins as their regular training materials. Due to the inaccurate anatomical structure around the axillary region, the MCPR devices (LUCAS or AutoPulse) could not be positioned correctly on most half-length manikins. This would affect the scores of MCPR deployment among many EMTs, yet those effects were minor enough to be ignored. Third, each BLS-D station was equipped with either LUCAS or AutoPulse, which were fundamentally different devices with different deployment techniques. However, we purposely did not control for the brand of MCPR device so that we could test EMTs with devices they were familiar with. Fourth, in our simulation study, we applied a fixed crew size of three EMTs, so the results might not be applied to EMS teams with a different size of resuscitation crew.40 Finally, despite applying the Delphi method to test the questionnaire, there were still limitations regarding the external validity of participant attributes such as attitude and self-confidence.

Conclusions

EMT’s subjective resuscitation performance and no-flow time did not match the objective results. Self-confidence was negatively associated with EMT-Ps’ MCPR deployment and teamwork performance. In contrast, knowledge was positively associated with EMT-Is’ performance, which indicated that the training courses for different EMT levels should focus on various aspects. The negative association between objective teamwork performance and self-satisfaction or abasement highlighted the importance of self-awareness in emergency resuscitation procedures. Finally, the Dunning–Kruger effect first observed in EMT’s resuscitation performance may provide insight into the adequate training strategy for EMTs.