Operating bedside cardiac ultrasound program in emergency medicine residency: A retrospective observation study from the perspective of performance improvement.

BACKGROUND: Point-of-care ultrasound is one of useful diagnostic tools in emergency medicine practice and considerably depends on physician's performance. This study was performed to evaluate performance improvements and favorable attitudes through structured cardiac ultrasound program for emergency medicine residents.
METHODS: Retrospective observational study using the point-of-care ultrasound (PoCUS) database in one tertiary academic-teaching hospital emergency department has been conducted. Cardiac ultrasound education and rotation program has been implemented in emergency medicine residency program. Structured evaluation sheet for cardiac ultrasound and questionnaire toward PoCUS have been developed. An early-phase and a late-phase case were selected randomly for each participant. Two emergency medicine specialists with expertise in PoCUS evaluated saved images independently. We used a paired t-test to compare the performance score of each phase and the results of the questionnaire. Multivariable linear regression analysis was conducted to evaluate the association between the characteristics of participants and performance improvements.
RESULTS: During the study period, a total of 1,652 bedside cardiac ultrasounds were administered. Forty-six examinations conducted by 23 emergency medicine residents were randomly selected for analysis. The performance score increased from 39.5 to 56.1 according to expert A and 45.3 to 62.9 according to expert B (p-value <0.01 for both). The average questionnaire score, which was analyzed for 17 participants, showed improvement from 18.9 to 20.7 (p-value <0.01). In multivariable linear regression analysis, younger age, higher early-phase score and higher confidence had a negative association with a greater improvement of performance, while the number of examinations had a positive association.
CONCLUSIONS: Bedside cardiac ultrasound performance and attitudes toward PoCUS have been improved through structured residency program.

Introduction

Point-of-care ultrasound (PoCUS) has been used frequently and variously in emergency clinical practice because it can reduce cost [1] and can be used as an additional diagnostic test [2] that provides important clinical information in a very short time. Procedures have also been proven to benefit from PoCUS, e.g., both guidance and confirmation [3, 4]. Because ultrasound performance depends on the physician’s ability, PoCUS training should be a core component in the education of emergency physicians, especially those who are in residency programs [5]. Adding PoCUS training to an education program is considered an important milestone, [6] and many programs have implemented such training, demonstrating feasibility and participant satisfaction [7, 8]. In the United States, PoCUS has been requirement of residency training in emergency medicine [9].

Cardiac ultrasound would be a good performance indicator of PoCUS due to its well established standards for application and evaluation [10]. Furthermore, cardiac ultrasound is known to be useful for diagnosis in emergency departments [11-14]. A recent study by Davood et al. showed that emergency medicine residents can perform bedside cardiac ultrasound in the emergency department after several workshops and that this procedure yields comparable quality to traditional cardiac ultrasound performed by cardiologists [15].

Improvements in practitioners’ skills, including in acquiring proper images and interpreting the results, have not been well investigated in previous studies. The evaluation tools are usually limited to objective structured clinical exams (OSCEs) or knowledge tests [16, 17]. In several studies, the diagnostic performance of clinical competency has usually been used to evaluate practitioner performance [7, 18]. To the best of our knowledge, methods to evaluate the performance level of bedside cardiac ultrasound in a real clinical emergency setting have not been developed well.

The purpose of this study is to evaluate the improvement of performance skill and interpretation level of bedside cardiac ultrasound among emergency medicine residents following a PoCUS education-rotation program, by reviewing acquired cardiac ultrasound images. The secondary outcome was a change in attitude and confidence toward PoCUS in emergency practice. We hypothesized that structured program for education and rotation would improves performance and results in favorable attitudes.

Methods

Study design and setting

A retrospective observational study was conducted based on the ultrasound database of the Seoul National University Hospital Department of Emergency Medicine. Seoul National University Hospital is a tertiary academic-teaching hospital in the metropolitan city of Seoul, and approximately 70,000 patients visit the emergency department annually. Emergency bedside ultrasound rotation schedule with education program has been in place since 2018, as part of a residency training program. The goal of this program is to improve the quality of clinical practice. Before implementation, ultrasound was an optional procedure in the emergency department that was conducted when physicians were willing to participate, which was not frequent enough to make ultrasound a standard part of clinical practice. The sample for this study comprised 2nd- to 4th-year residents, who were required to participate in the PoCUS program since it became a formal part of the curriculum and practice protocol in the department.

Study population

Emergency medicine residents who followed emergency bedside ultrasound rotation duties during the study period, were enrolled. Participants who did not save the whole basic views of ultrasound imaging in the picture archiving communication system or document their interpretations in medical records were excluded from the analysis.

PoCUS program

The PoCUS program, consisting of an education section and a practice section, was officially designed and implemented in April 2018. All residents had to take a comprehensive ultrasound workshop for basic echocardiography, including basic view, by a cardiologist early in their 2nd years. They then automatically participated in monthly PoCUS education programs if they had been assigned to adult or pediatric emergency departments in Seoul National University Hospital. Residents on other schedules, e.g., in other hospitals or departments, did not participate. The monthly education program included a 2-hour training session and a conference. In the training session, basic and advanced knowledge of PoCUS administration, including for the lung or abdomen, was reviewed, and the participants received hands-on practice administering bedside cardiac ultrasound using an ultrasound simulation machine (US Mentor, Simbionix). Interesting cases were selected by the residents as cases supporting the clinical usability of PoCUS and presented in a conference at the end of the month.

The PoCUS practice session was a requirement of the residency program. All residents had approximately 8–16 hours of ultrasound duties in the emergency department each month. They performed examinations whenever patients presented with chest pain, difficulty breathing, syncope or palpitation. Primary physicians could also request bedside cardiac ultrasounds to residents who are in ultrasound rotation. Residents do not care for patients as primary physicians while on an ultrasound rotation duty. A portable ultrasound machine was used for all procedures (M-turbo, SonoSite) (Vivid Q, GE). Video clips and images were saved and transferred to a picture archiving communication system. Official interpretations of the results were typed in a constructed format. PoCUS rounding was conducted every other day with PoCUS faculty and residents for quality improvement and feedback. All contents were reviewed and supported by the PoCUS faculty, which consisted of 4 emergency medicine specialists and an emergent medical technician. PoCUS faculty need to undergo comprehensive ultrasound education workshops regularly and perform bedside ultrasound in official emergency practice.

Data source and acquisition

The database system was operated and supervised by the PoCUS faculty using a structured registry that contains demographic and clinical information of patients. It includes date of examination, operator’s information and official interpretation.

We retrieved data from the PoCUS database to evaluate performance improvements. Two cases were selected for each resident, one in the first week of the whole period of PoCUS rotation program (early phase) and the other in the last (late phase). Independent research coordinators selected all cases randomly in each period.

Evaluation for performance

A structured evaluation sheet was developed based on the guidelines of emergency ultrasound standard reporting [19]. The evaluation sheet was designed to evaluate handling the machine, capturing feasible views for interpretation, adjusting depth and gain properly, achieving a complete set of views and interpreting clearly. A detailed introduction to the evaluation sheet is described in Table 1.

Table 1

Evaluation sheet for performance evaluation of cardiac ultrasound in PoCUS program.

View		Probe	Utility	Orientation	Anatomy	Interpretation	Image quality
Parasternal Long Axis View
Parasternal Short Axis View
	• AV level
	• MV level
	• Papillary muscle level
	• Apex level
Apical 4-Chamber View
Apical 5-Chamber View
Measure Index
	• IVC
	• EF

PoCUS, point-of-care ultrasound; AV, Aortic valve; MV, Mitral valve; IVC, Inferior vena cava; EF, Ejection fraction

Probe, Appropriate probe was used, score 1; Utility, Total gain adjustment, Time gain adjustment, Focus and depth are well controlled, score 2; Utility, 1 or 2 options have not been adjusted properly, score 1; Utility, 3 or 4 options have not been adjusted properly, score 0; Orientation, Orientation marker was positioned well, score 1; Anatomy, All structures were well identified, score 3; Anatomy, Any core structures was not identified, score 2; Anatomy, Only one structure was identified, score 1; Anatomy, Missed or could not determine specific view, score 0; Interpretation, All important finding has been documented, score 3; Interpretation, Several important findings have been documented, score 2; Interpretation, Some interpretation was inappropriate, score 1; Interpretation, No specific interpretation for view, score 0; Image quality, Seems no need for improvement, score 3; Image quality, Fine but need some improvement, score 2; Image quality, Can recognize specific view, but limited to interpret, score 1; Image quality, Cannot recognize specific view, score 0

Two independent emergency specialists were consulted to evaluate the cases. Two independent emergency specialists were consulted to evaluate the cases. They are PoCUS faculty members who have practical experience with more than 200 cardiac PoCUS and work as instructors for emergency physician and medical school student. Evaluation was based on video clips in a picture archiving communication system (PACS) and developed sheet. The specialists were blind to practitioner identity and phase. Specialists gave specific scores to 6 components of 7 basic views (parasternal long axis, parasternal short axis for 4 levels, 4-chamber and 5 chamber view) and 2 measure indexes (inferior vena cava and ejection fraction) based on the guidelines of the evaluation sheet, and the total score was calculated. One point was given for probe and orientation selection, 2 points were given for adjusting brightness properly, and 3 points were given for imaging well-identified structures of heart, providing a clinically sufficient interpretation and capturing high quality images. The maximum total score was 100 points in each case.

Survey of confidence and acceptance

A structured questionnaire that consisted of 7 questions regarding confidence in and attitudes about PoCUS use in emergency practice was developed. Each question is answered on a 5-point Likert scale of agreement. Detailed information on the questionnaire is provided in Table 2. All participants completed the questionnaire at the beginning and end of the PoCUS program. This questionnaire was used to judge quality improvements and applicability to residency programs.

Table 2

Questionnaire for survey of PoCUS program.

1. Can you use ultrasound in emergency department which you are working?
(including select probe, adjust gain and depth, saving image)
Response set	Strongly Disagree—Disagree—Neutral—Agree—Strongly Agree
2. How many patients are you applying ultrasound for one duty in daytime?
Response set	less than two—two—three—four—more than four
3. How many patients are you applying ultrasound for one duty in nighttime?
Response set	less than two—two—three—four—more than four
4. How much proportions of "Core Basic View" can you acquire properly in adult cardiac ultrasound?
(Parasternal long axis, Parasternal short axis, 4-chamber view, 5-chamber view, IVC)
Response set	0~20% - 20~40% - 40~60% - 60~80% - 80~100%
5. How much proportions of "Core Basic View" can you acquire properly in adult abdominal ultrasound?
(Liver, GB, Spleen, Kidney, Bladder, Abdominal Aorta)
Response set	0~20% - 20~40% - 40~60% - 60~80% - 80~100%
6. How much proportions of "Core Basic View" can you acquire properly in pediatric abdominal ultrasound?
(Liver, GB, Spleen, Kidney, Bladder, Abdominal Aorta)
Response set	0~20% - 20~40% - 40~60% - 60~80% - 80~100%
7. This question is about clinical utility of point-of-care ultrasound in emergency setting. How many ultrasounds will you perform in one duty after residency program?
Response set	less than two—two—three—four—more than four

* Response set, Likert 5-scale (1-2-3-4-5)

PoCUS, point-of-care ultrasound; IVC, inferior vena cava; GB, gallbladder

Statistical analyses

A descriptive analysis was conducted by calculating the mean and standard deviation for the demographics and characteristics of the participants. Total early- and late-phase scores were compared by paired t-test for each participant, and the intraclass correlation coefficient was calculated. Sensitivity analysis for each basic core view and questionnaire analysis were conducted using the same methods.

An additional multivariable linear regression analysis was conducted to evaluate the association between the demographics and characteristics of the participants and score improvements. A stepwise method was used to develop the optimized model. Score improvement was defined as the mean difference between each phase. We selected predictors such as year of graduation (YOG), age, time elapsed between assessments, early-phase score, number of PoCUS applications, duration of program participation and answers to the questionnaires about confidence in and acceptance of bedside cardiac ultrasound. All statistical analyses were conducted in R Studio 4.3.4.

Ethics approval

Ethics approval and consent to participate: The study complies with the Declaration of Helsinki, and its protocol was approved by the Seoul National University Hospital Institutional Review Board with a waiver of informed consent (IRB No. 1911-076-1078).

Results

Characteristics of study subjects

From April 2018 to February 2019, a total of 28 emergency medicine residents participated in the PoCUS program. 46 ultrasound data from 23 participants were analyzed in the performance evaluation, since 5 participants were excluded who did not save the whole basic ultrasound views or interpretations. In the survey analysis, 6 participants refused to fill out questionnaire. 22 residents conducted 35 pairs (35 early phase and 35 late phase) of questionnaires, since 13 residents participated POCUS each year during the study period. The demographics and characteristics of the study participants are described in Table 3. The mean period between the early and late phases, which is almost same as the gap between first and last rotation, was 20 weeks. The mean participation in the PoCUS program was 3 months, which was not the same for each individual due to rotation schedule including dispatching to other hospitals. They administered PoCUS nearly 40 times on average, and there was one exceptional practitioner who conducted 215 cardiac ultrasounds in 5 months.

Table 3

Characteristics of study population and bedside cardiac ultrasound.

		N	%
Total		23
YOG
	1	1	4.3
	2	9	39.1
	3	8	34.8
	4	5	21.7
		Mean	SD
Age		30.96	3.2
Period difference, week		20.87	10.83
Participated period, month		3.04	1.15
Score of Early phase		42.39	19.58
Score of Late Phase		59.52	19.54
Number of Cardiac Ultrasound Examinations		38.78	44.38
Prior Answers for Question about Confidence		2.76	0.97
Prior Answers for Question about Acceptance		2.88	1.32
Post Answers for Question about Confidence		3.88	0.7
Post Answers for Question about Acceptance		3.24	1.39

YOG, Year of graduate

Performance and survey evaluation

Table 4 and Fig 1 shows the results of the performance evaluations conducted by two emergency specialists using the structured sheet and questionnaire evaluations. A total of 46 cases from 23 participants were analyzed, and the mean score changed significantly from 39.5 to 56.1 according to expert A and 45.3 to 62.9 according to expert B. The interclass correlation coefficients were 0.85 and 0.84 for the early and late phases, respectively. The total average questionnaire score improved from 18.9 to 20.7. The most improved question was about confidence in acquiring the core basic view using adult cardiac ultrasound (Q4 in Table 2), which changed from 3.14 to 3.83.

Table 4

Performance evaluation of cardiac ultrasound and acceptance for PoCUS program.

	Expert A	Expert B	ICC, 95% CI	Questionnaire
Early phase, mean (SD)	39.5 (21.2)	45.3 (19.2)	0.85 (0.61–0.94)	18.9 (4.65)
Late phase, mean (SD)	56.1 (21.7)	62.9 (18.5)	0.84 (0.52–0.94)	20.7 (4.51)
p-value of Shapiro-Wilk normality test	0.56	0.84
p-value for paired t-test	<0.01	<0.01		<0.01

PoCUS, point-of-care ultrasound; ICC, Interclass correlation coefficiency; CI, Confidence Interval; SD, Standard deviation

Fig 1

Evaluation of performance improvement in bedside cardiac ultrasound and survey about confidence and acceptance toward PoCUS.

PoCUS, point-of-care ultrasound.

Sensitivity analysis for detailed components

We conducted additional sensitivity analysis for each component of the bedside cardiac ultrasound core basic view. There was significant improvement in performance based on each expert’s evaluation of the parasternal long axis; parasternal short axis, aortic valve; parasternal short axis, mitral valve; parasternal short axis, papillary muscle; and 4-chamber view. The interclass correlation coefficient was mostly between 0.65 and 0.8, except for the inferior vena cava view (0.43 and 0.23) (Table 5).

Table 5

Sensitivity analysis for cardiac ultrasound performance evaluation.

View	Phase	Expert A	Expert B	ICC, 95% CI
Parasternal Long Axis	Early phase, mean (SD)	7.96 (2.67)	8.26 (2.70)	0.84 (0.66–0.93)
	Late phase, mean (SD)	10.0 (2.06)	10.3 (1.75)	0.65 (0.33–0.83)
	p-value for paired t-test	<0.01	<0.01
Parasternal Short Axis, AV level	Early phase, mean (SD)	4.65 (4.48)	4.91 (3.69)	0.86 (0.69–0.94)
	Late phase, mean (SD)	7.30 (4.83)	7.39 (3.74)	0.65 (0.33–0.84)
	p-value for paired t-test	<0.01	<0.01
Parasternal Short Axis, MV level	Early phase, mean (SD)	6.09 (3.87)	7.78 (3.23)	0.61 (0.23–0.82)
	Late phase, mean (SD)	8.52 (4.18)	9.83 (3.52)	0.77 (0.48–0.90)
	p-value for paired t-test	0.03	0.04
Parasternal Short Axis, Papillary muscle level	Early phase, mean (SD)	5.70 (4.20)	6.87 (4.26)	0.89 (0.64–0.96)
	Late phase, mean (SD)	8.04 (4.81)	9.65 (3.65)	0.73 (0.41–0.88)
	p-value for paired t-test	0.04	<0.01
Parasternal Short Axis, Apex level	Early phase, mean (SD)	3.39 (4.19)	4.43 (4.59)	0.86 (0.67–0.94)
	Late phase, mean (SD)	5.17 (5.39)	6.09 (5.73)	0.77 (0.54–0.90)
	p-value for paired t-test	0.18	0.25
4-Chamber View	Early phase, mean (SD)	3.87 (4.30)	4.83 (3.89)	0.86 (0.67–0.94)
	Late phase, mean (SD)	7.26 (4.39)	8.13 (4.35)	0.77 (0.54–0.90)
	p-value for paired t-test	<0.01	<0.01
5-Chamber View	Early phase, mean (SD)	2.96 (3.67)	3.39 (4.35)	0.86 (0.67–0.94)
	Late phase, mean (SD)	3.74 (4.66)	4.96 (5.23)	0.77 (0.54–0.90)
	p-value for paired t-test	0.51	0.28
Inferior Vena Cava View	Early phase, mean (SD)	3.26 (2.56)	3.78 (2.32)	0.43 (0.04–0.71)
	Late phase, mean (SD)	3.91 (2.48)	4.96 (1.64)	0.23 (0–0.56)
	p-value for paired t-test	0.27	0.84

ICC, Interclass correlation coefficiency; CI, Confidence Interval; SD, Standard deviation; AV, Aortic valve; MV, Mitral valve

Regression model for improvement of performance

In multivariable linear regression analysis, we found that age, early-phase score and prior high confidence had a negative association with performance improvement; the beta coefficients were -2.6 (-4.8 to -0.4) for age, -0.4 (-0.7 to -0.1) for early-phase score, and -11.0 (-18.6 to -3.4) for prior answers to questions about confidence. Otherwise, the number of cardiac ultrasound examinations had a positive association with improved performance, with a beta coefficient of 0.4 (0.2 to 0.5) (Table 6).

Table 6

Multivariable linear regression analysis for performance improvement in bedside cardiac ultrasound.

Variables	ß	95% CI	p-value
YOG	9.4	-0.8 to 19.6	0.066
Age	-2.6	-4.8 to -0.4	0.027
period difference, week	0.5	-0.1 to 1.1	0.1
Score of Early phase	-0.4	-0.7 to -0.1	0.014
Number of Cardiac Ultrasound Examinations	0.4	0.2 to 0.5	<0.001
Prior Answers for Question about Confidence	-11.0	-18.6 to -3.4	0.009

CI, Confidence Interval; YOG, Year of graduate

Discussion

We found statistically significant improvement of cardiac ultrasound performance by emergency medicine residents following structured education and rotation programs. Furthermore, attitudes about and confidence in emergency PoCUS application improved. Additional analysis showed that younger age, lower previous performance level and confidence were associated with marked improvement.

Our study used a paired t-test, which has been successfully used to evaluate various programs for resident education, to evaluate the effect of the program in each participant [20, 21]. Recent research on focused cardiac ultrasound in surgical intensive care units used similar statistical methods and standardized scoresheets [22]. Our scoresheet included similar categories to those of previous studies but had more detailed components and evaluation instructions, which were designed by the PoCUS faculty. The correlation between the two emergency specialists revealed good agreement on total score and most of the sensitivity analysis [23].

Multivariable linear regression analysis was conducted to evaluate the association between the characteristics of participants and potential improvement. Variables could be retrieved only from the clinical database since it was a retrospective observation study, and the results were reasonable and explainable. We may consider focusing on residents with low performance levels and confidence, which would be more effective.

Previous studies have been predominantly based on education programs, and evaluations have not been blinded to both raters and participants. Our evaluation was conducted based on a clinical practice database in an emergency department. We assume that all ultrasound procedures were performed independently for clinical purposes without additional consideration, which would guarantee more objective outcomes than evaluating on via the education program.

There has been little evidence of an effective curriculum and evaluation method for ultrasound training. In focused abdominal sonography for trauma (FAST), previous studies usually focused on requirements for certification or competency [24-26]. The results of this research demonstrated an effective way to increase the performance and favorable attitudes of cardiac PoCUS in residents. Anstey et al. demonstrated an effective PoCUS curriculum for internal medicine residents that produced long-term gains in knowledge and high confidence achievements [27]. Emergency medicine residents struggle in overcrowded emergency departments to manage various patients with severe needs, which results in a stressed and dissatisfied training period [28, 29]. It often leads residents to overload themselves to complete the mandatory cardiac PoCUS training sessions off duty. On the other hand, procedural training without real clinical practice can blur the effect of education programs. Furthermore, a recent study showed that doctors recognize their need for sufficient practice with certain procedures to build competency [30]. We can infer that less education and more practice would help improve both performance and attitude on cardiac PoCUS.

Our research focused on performance improvement of cardiac PoCUS with real practice, but the clinical benefit has not been well studied. We are planning an additional study to evaluate the association between the implementation of cardiac PoCUS protocol for specific population and improvement of clinical outcomes. Additionally, after operating the PoCUS program for a few more years, other components of the PoCUS program, including pediatric abdomen sonography and lung ultrasound, are thought to be analyzed with sufficient case volume.

Our study has several limitations. First, we screened all residents for the study population and included 23 participants based on compatibility for analysis rather than the number of examinations they performed. A previous study demonstrated a cutoff for improvement in ultrasound administration [31]. In our study, residents conducted cardiac ultrasound when needed to evaluate patients, but they did not log all examinations in the database, especially in urgent situations. Accordingly, the number of cases with usable data varied from 10 to over 200 for each participant but confining the study population based on the number of examinations completed may be inappropriate for study purpose of evaluating all participants in the program. Also, even assuming that the data would have been saved with well-performed ultrasound, the missing ultrasound data could lead to bias, which is one of the major limitations of the study. Second, since the yearly schedule is different for each participant, including the timing of the adult emergency department rotation, the number of program openings per educational period and the number of PoCUS rotations available would differ. The participation period varied from 1 month to 5 months. We assume that if the residents recognize the clinical importance of PoCUS, they will practice it in the course of their regular duties even if they do not need to. Next, scoring system used in this study has not been validated for various environment. Though it was developed based on the standard guidelines, further research for validation is needed. Fourth, evaluation of performance was conducted by emergency medicine specialists. In United States, certification for PoCUS competency in emergency practice has been developed for various institutes. We consider certification of assessor as appropriate with more than 200 cardiac PoCUS experienced, and qualification for PoCUS instructor in Seoul National University Hospital Department of Emergency Medicine. Detailed components to prove the practitioner’s credentials may not match perfectly due to lack of official PoCUS certificate programs in South Korea, which can be another limitation. Fifth, this study was limited to one tertiary academic hospital emergency department, and thus cannot be generalized directly to other environments. We are trying to construct a multicenter PoCUS program, which may overcome this limitation. Next, concrete description or analysis for diastolic function and valve function was not conducted. Though some description for them was in interpretation, we considered competency of participants is insufficient, which including diastolic function and valve function seems inappropriate. Last, this is a retrospective observational study, which has limitations in proving the effect of experience.

Conclusion

We found performance improvements in bedside cardiac ultrasound administration and favorable attitudes toward PoCUS in emergency medicine residents who were enrolled in a structured education and rotation program.

(XLSX)

Click here for additional data file.

8 Jan 2021

PONE-D-20-30438

Operating bedside cardiac ultrasound program in emergency medicine residency: a retrospective observation study from the perspective of performance improvement

PLOS ONE

Dear Dr. Jung,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

This is an interesting and generally well-done paper in an area with little formal research. As such, I believe it will make an important contribution to the medical literature, provided the recommended changes can be made.

In addition to the reviewer comments, can you please state whether the scores were normally distributed and thus appropriate for analysis with a t-test. i also agree that lack of cardiologist involvement is not a limitation and should be removed.

Please submit your revised manuscript by Jan 31 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Robert Ehrman, MD, MS

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability.

Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

We will update your Data Availability statement to reflect the information you provide in your cover letter.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Thanks for the opportunity to review this.

As you say, it shows objective improvement in competence and subjective confidence following a structured POCUS training programme.

I have a couple of queries

- why does the mean time in program (3 months) and the mean period between early and late phases (20 weeks) not match? You state early and late phase are at the beginning and end of the programme?

- I would value some more detail / clarity about the qualifications or experience of the faculty / assessors. it is not clear how experienced or suitable they are at assessing the quality of images (I am sure they are perfectly qualified but to allow reproducibility we need more details of their qualifications)

Reviewer #2: The article is an interesting analysis on the Operating bedside cardiac ultrasound program in emergency medicine residency. The article is clear and the main results well represented. To make the article more appealing for readers would be of interest to know also attitude in the use of color doppler and pulse wave doppler for assessment of diastolic function and have a dedicated issue on assessment of valve disease.

Reviewer #3: Thank you for the opportunity to review this manuscript on a retrospective analysis of the impact of a dedicated POCUS on attitudes and output of a POCUS program in an EM residency. Overall i found the data contained in the manuscript to be informative and very useful for readers who play a role in an EM based POCUS program. Strengths of the manuscript include:

-clearly stated statement of objective

-an detailed enough description of what was done that a similar researcher could reproduce this study in their own EM program

-utilizing independent research coordinators to randomly select the cases from the two periods that were evaluated by the study team

-detailed explanation of the items that were evaluated in each of the cardiac ultrasound views

-employing a sensitivity analysis

THe following items are things that i believe should be addressed to improve the manuscript:

-at the end of the 1st paragraph in the Introduction you should state that the ACGME RRC EM requires/mandates training in POCUS and cite their documents. it would be a more powerful and accurate statement to highlight the importance of POCUS for the general readership

-although the objective statement at the conclusion of the Intro is explicitly stated, i do take issue with the use of the term "perception". It is not clearly defined anywhere in the manuscript and b/c you are including it in your objective statement i think you should either revise the verbiage of the objective statement, or include in the methods section how you intend to measure "perception"

-the use of the term "screening" in the Methods section seems misplaced. It almost seems like you simply excluded residents who didn't follow the parameters of the POCUS rotation, rather than screening. I think you should eliminate the use of the word screening here and simply describe precisely how you went about selecting the participants

-your scoring system, although appearing to be quite robust, is certainly novel. Did you do anything to internally validate this measure?

-going back to the "screening" issue, by my interpretation of the manuscript there were 33 eligible participants amongst the 2nd-4th year residents, and ultimately 22 of them were included b/c 6 refused to complete the questionnaire and 5 were excluded b/c they didn't perform or document appropriately their POCUS exams. If this is indeed true, i think a flow diagram outlining this would be a much better way to describe this then the way it is currently described in the manuscript using the "screening" terminology

- on lines 229-230 of the Results section you state that younger age had a negative association with performance improvement, however on lines 243-244 of the Discussion section you state that younger age had an association with marked improvement. Please resolve this discrepancy.

-since this study was essentially only looking at cardiac POCUS, i would recommend rewording much of the paragraph on lines 263-276

-lines 277-283 seem to be discussing practice of POCUS that occurs "on shift" in an ED, although this is not explicitly stated, i would rephrase to include this term if this is indeed what you are discussing here

-Line 300 referring to the lack of a cardiologist performed evaluation is not a Limitation. Cardiologists are not content experts in cardiac POCUS and need not be the ones performing evaluations of POCUS exams in the ED

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

14 Jan 2021

We sincerely appreciate your comment. I would like to notify you that scores found to be normally distributed by shaprio-wilks test and thus considered to be appropriate for analysis with a t-test.

Submitted filename: Answer to comment.docx

Click here for additional data file.

17 Feb 2021

PONE-D-20-30438R1

Operating bedside cardiac ultrasound program in emergency medicine residency: a retrospective observation study from the perspective of performance improvement

PLOS ONE

Dear Dr. Jung,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

==============================

Thank you for your revised submission, it is overall markedly improved. However, there are still a few areas that require your attention. Please review and explicitly address the concerns raised by reviewer #3. I agree that proper understanding of how these issues were handled is critical to the interpretation and application of the results of your study.

==============================

Please submit your revised manuscript by Apr 03 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Robert Ehrman, MD, MS

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: All comments have been addressed

Reviewer #3: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #2: Yes

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: Yes

Reviewer #3: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #2: Yes

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #2: I'm satisfied with revision and aswers provided by authors to the main comments...............................................................................................................................................................................................................................................................

Reviewer #3: Thank you for taking the time to revise your manuscript. I do however have one overarching issue that remains in spite of the revisions, and that is how you handled incomplete data sets. Two specific areas of the revised manuscript In the Methods section, lines 91-93. How did you treat the scenario where a resident saved some of the images of the exams they did, but not all of them? Were they excluded, or did you only exclude residents who did not save any of their images? Selection bias is possible (likely) given the manner in which most residents save some, but not all of their images from exams they do on ultrasound rotations.

The other location in the manuscript where this comes up is in the Results section, lines 194-198. 35 pairs of questionnaires from 22 inhabitants means there was a lot of participants who only completed 1 survey…and also, the 22 participants is different from the 28 participants identified in the first sentence. What happened to the 6 residents w/o surveys? The next sentence says 5 were excluded so that still leaves you at 23 and not 22. The old manuscript says 6 participants refused to complete the questionnaire, why was this info excluded from the revision?

Without more detail on how incomplete data was handled, it is not possible to reproduce this study, and this is why i cannot recommend it for publication in its presents state.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: No

Reviewer #3: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

18 Feb 2021

Reviewer #3:

Two specific areas of the revised manuscript In the Methods section, lines 91-93. How did you treat the scenario where a resident saved some of the images of the exams they did, but not all of them? Were they excluded, or did you only exclude residents who did not save any of their images? Selection bias is possible (likely) given the manner in which most residents save some, but not all of their images from exams they do on ultrasound rotations.

(ANSWER) Thank you for the review. Participants were excluded who did not saved the whole basic view of ultrasound. The entire set of basic view were introduced and educated to participants at the beginning of POCUS program. We considered that evaluating the best performance at each point is thought to be proper in comparing between phases. We clarified this at the manuscript more clearly.

(REVISION: Methods) Participants who did not save the whole basic views of ultrasound imaging in the picture archiving communication system or document their interpretations in medical records were excluded from the analysis.

(REVISION: Methods) All residents had to take a comprehensive ultrasound workshop for basic echocardiography, including basic views, by a cardiologist early in their 2nd years.

The other location in the manuscript where this comes up is in the Results section, lines 194-198. 35 pairs of questionnaires from 22 inhabitants means there was a lot of participants who only completed 1 survey…and also, the 22 participants is different from the 28 participants identified in the first sentence. What happened to the 6 residents w/o surveys? The next sentence says 5 were excluded so that still leaves you at 23 and not 22. The old manuscript says 6 participants refused to complete the questionnaire, why was this info excluded from the revision?

(ANSWER) Thank you for your valuable comments. First, 6 residents denied or neglect conducting surveys, as you indicated. 5 participants were excluded from performance analysis, not survey analysis. Total achieved questionnaires were 35 pairs, 1 pair for Early-phase and Late-phase. Which means there were 70 questionnaires for 22participants. There were 13 residents, who participated yearly POCUS program in succession (as described in manuscript, 2nd ~ 4th year of residents participate the program), conducted questionnaire in each year. We clarified this in the manuscript more clearly.

(REVISION: Results) From April 2018 to February 2019, a total of 28 emergency medicine residents participated in the PoCUS program. 46 ultrasound data from 23 participants were analyzed in performance evaluation, since 5 participants were excluded who did not save the whole basic ultrasound views or interpretations. In survey analysis, 6 participants refused to fill out questionnaire and 22 residents conducted 35 pairs (35 early-phase and 35 late phase) of questionnaires, since 13 residents participated yearly POCUS program in succession for study period.

Submitted filename: Answer to comment.docx

Click here for additional data file.

23 Feb 2021

PONE-D-20-30438R2

Operating bedside cardiac ultrasound program in emergency medicine residency: a retrospective observation study from the perspective of performance improvement

PLOS ONE

Dear Dr. Jung,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

==============================

Thank you for clarifying the issue regarding missing US exams. However, beyond just simply stating that it exists, I believe that it is important to add some interpretation of the ramifications of the missing data. Does it introduce bias? How does it impact results, etc?

==============================

Please submit your revised manuscript by Apr 09 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Robert Ehrman, MD, MS

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

23 Feb 2021

Thank you for clarifying the issue regarding missing US exams. However, beyond just simply stating that it exists, I believe that it is important to add some interpretation of the ramifications of the missing data. Does it introduce bias? How does it impact results, etc?

(ANSWER) Thank you for the review. As your valuable comments, missing ultrasound data is one of limitation of this research. We would like to describe the assumption and potential bias of that in the discussion section as below.

(REVISION: Discussion) Also, even assuming that the data would have been saved with well-performed ultrasound, the missing ultrasound data could lead to bias, which is one of the major limitations of the study.

4 Mar 2021

Operating bedside cardiac ultrasound program in emergency medicine residency: a retrospective observation study from the perspective of performance improvement

PONE-D-20-30438R3

Dear Dr. Jung,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Robert Ehrman, MD, MS

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

24 Mar 2021

PONE-D-20-30438R3

Operating bedside cardiac ultrasound program in emergency medicine residency: a retrospective observation study from the perspective of performance improvement

Dear Dr. Jung:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Robert Ehrman

Academic Editor

PLOS ONE