Reliability of focused cardiac ultrasound performed by first-year internal medicine residents at a community hospital after a short training.

The use of bedside ultrasound over the past few decades has created a new wave of options for visualizing pathological processes allowing for faster and better detection of disease. We aimed to evaluate the reliability of focused cardiac ultrasound (FCU) performed by first-year internal medicine residents at a community hospital after a short period of training. They received a two-hour lecture and initially performed a supervised FCU followed by ten unsupervised/independent FCUs each. The four parameters that were assessed were left systolic ventricular function, right systolic ventricular function, presence of pericardial effusion, and presence of IVC dilation. Interpretation and analysis of ultrasound images were then carried out by both the residents and an attending physician with expertise in FCU analysis and interpretation. Cohen's Kappa values were obtained comparing the results found by the interns versus the attending. Our findings indicate that more training is required for reliable analysis of FCU by first-year medical residents. Our results also emphasize the need to carefully evaluate the medical residents' FCU skills after the training.

Introduction

First introduced in 1816 by French physician Laennec, the stethoscope has without a doubt changed the face of cardiac and pulmonary physical exams [. However, with the advent of more powerful non-invasive diagnostic tools in the later part of the 20th century, multiple studies have shown that the accuracy of clinical diagnosis at bedside based on cardiac auscultation using a stethoscope alone has become less reliable [2-4]. Clinical decision making in this fast-evolving era of medicine, when supplemented with more advanced diagnostic tools such as the focused cardiac ultrasound (FCU) has better clinical outcomes in both acute and chronic settings [5-9].

The 2013 American society of echocardiography (ASE) guidelines have a very clear distinction between focused cardiac ultrasound and a limited echocardiogram [10]. A focused cardiac ultrasound (FCU) refers to a point of care ultrasound examination that is goal-oriented in a specific clinical setting to supplement the physical examination, whereas a limited echocardiogram refers to the performance of a limited number of views with otherwise full echocardiographic capabilities. In addition to this, the ASE has recommended certain guidelines in didactic, hands-on, and image interpretation components.

The aim of this study was to evaluate the ability of first-year internal medicine residents (medical interns) at a community hospital, with no prior knowledge of cardiac ultrasound, to quickly learn how to perform a FCU and assess their ability to match those opinions of a cardiologist. This study may shed light on the minimum amount of training needed in residency programs to allow medical interns to independently and effectively perform a FCU.

Methods

We conducted a prospective study that compared paired diagnostic findings between medical interns and a teaching internist who is board-certified in internal medicine in the U.S. and is also board certified in cardiology in Brazil, whose diagnostic determination was considered to be the gold standard of accurate assessment of patients. The four parameters that were assessed included left systolic ventricular function, right systolic ventricular function, presence of pericardial effusion and presence of IVC dilation. Only medical interns that had never received any ultrasound equipment training were allowed to participate in the study. From a total of fourteen medical interns, seven interns were enrolled in the study. The training started with a two-hour introduction to ultrasound equipment, knobology, and medical ultrasound safety. The ultrasound machine used was a Sonosite Edge 2 and the training was provided by the teaching internist(mentioned above).

After the initial lecture, medical interns were trained in small groups (up to three interns per group) over the next few weeks. During this training period demonstration of a FCU at the bedside with a real patient was done. The diagnoses of pericardial effusion and the evaluation of the systolic ventricular functions were done with the parasternal views (long and short axis) and four chamber apical view. The ventricular function was visually estimated and/or calculated by the ultrasound machine program after the ventricular dimensions were measured. The measurement of the size of the inferior vena cava during inspiration and expiration was done with a 2D-guided M-mode subcostal view. During these practical sessions, the interns were able to perform supervised FCU after daily demonstrations. During the training period, the medical interns were also introduced to applications and websites, which contained tutorials and clips of normal and pathological echocardiograms. After the initial training period, each medical intern did ten unsupervised FCUs on a random patient population with no separation to age, sex, body mass index throughout the hospital. Consequently, a total of seventy FCUs were done in this project. The medical interns filled a form with the findings of the FCU exam and the pictures and clips were stored for the academic internist/cardiologist’ interpretation.

Statistical analysis

To identify rates of inter-rater agreement between the medical interns’ ratings and those of the academic internist/cardiologist, Cohen’s Kappa values were calculated. Both un-weighted and weighted Cohen’s kappa values were calculated. Cohen’s kappa is used when two raters are compared to each other as described in multiple studies [11-13].

Conceptually this method considers the total agreement by the raters in comparison to the total agreement by chance to identify a kappa value. This

kappa value results in a value between −1 to 1, where −1 is very poor agreement worse than chance, 1 is complete agreement despite the likelihood by chance. We have generated a pivot table with the agreement values as a diagonal line in each table for the four parameters that were assessed in our study. The columns and rows represent interpretations of the same FCU by either the residents (columns) or the attending (rows).

Results

Table 1 shows the analysis of left ventricular systolic function as seen by both the interns as well as the academic internist/cardiologist. Cohen’s un-weighted kappa was 0.255 and weighted was 0.383. Both kappa values calculated indicate a fair agreement at kappa values between 0.2 and 0.4. The number of observed agreements was totaled to be 45 or 64%, and the number of agreements by chance was calculated to be 36.4 or 52%. This is considered to indicate partial agreement, and thus it is still questionable whether the amount of training provided was sufficient to assess left ventricular systolic function among the interns.

Table 1.

Left ventricular systolic function.

	Normal	Mild	Severe	Unknown	Total
Normal	39	10	1	1	51
Mild	5	3	2	0	10
Severe	2	2	3	2	9
Unknown	0	0	0	0	0
Total	46	15	6	3	70

Cohen Kappa (1 attending) = 0.255 Fair agreement.

Weighted Cohen Kappa = 0.383 Fair agreement.

Note: Columns represent residents, and rows represent attending physician.

Number of observed agreements = 45 (64%).

Number of agreements expected by chance = 36.4 (52%).

Table 2 shows the analysis of the right ventricular systolic function as seen by an academic internist/cardiologist. Cohen’s un-weighted kappa was 0.018 and weighted was −0.027. Both kappa values calculated indicate a poor agreement at kappa values between 0 and 0.2. The number of observed agreements was totaled to be 54 or 77%, and the number of agreements by chance was calculated to be 53.7 or 77%. This is considered to indicate poor agreement, and thus the amount of training provided was not sufficient to assess right ventricular systolic function among the interns.

Table 2.

Right ventricular systolic function.

	Normal	Mild	Severe	Unknown	Total
Normal	53	7	0	2	62
Mild	3	1	0	0	4
Severe	0	0	0	0	0
Unknown	4	0	0	0	4
Total	60	8	0	2	70

Cohen Kappa (1 attending) = 0.018 Poor agreement.

Weighted Cohen Kappa = −0.027 Poor agreement.

Note: Columns represent residents, and rows represent attending physician.

Number of observed agreements = 54 (77%).

Number of agreements expected by chance = 53.7 (77%).

Table 3 shows the analysis of pericardial effusion as seen by both the interns as well as the academic internist/cardiologist. Cohen’s un-weighted kappa was 0.159 and weighted was 0.132. Cohen’s kappa and weighted Cohen’s kappa values calculated indicate a poor agreement at kappa values between 0 and 0.2. The number of observed agreements was totaled to be 57 or 81%, and the number of agreements by chance was calculated to be 54.5 or 78%. Overall this is considered to be poor agreement, and thus the amount of training provided was not sufficient to assess pericardial effusion among the interns.

Table 3.

Pericardial effusion findings.

	Normal	Mild	Severe	Unknown	Total
Normal	55	2	0	1	58
Mild	10	2	0	0	12
Severe	0	0	0	0	0
Unknown	0	0	0	0	0
Total	65	4	0	1	70

Cohen Kappa (1 attending) = 0.159 Poor agreement.

Weighted Cohen Kappa = 0.132 Poor agreement.

Note: Columns represent residents, and rows represent attending physician.

Number of observed agreements = 57 (81%).

Number of agreements expected by chance = 54.5 (78%).

Table 4 shows the analysis of Inferior Vena Cava function as seen by both the interns as well as the academic internist/cardiologist. Cohen’s un-weighted kappa was 0.441 and weighted was 0.540. The number of observed agreements was totaled to be 55 or 79%, and the number of agreements by chance was calculated to be 43.1 or 62%. Overall, this is considered to indicate fair to moderate agreement, and thus the amount of training provided was sufficient to assess inferior vena cava function among the interns.

Table 4.

Inferior vena cava assessment.

	Normal	Mild	Severe	Unknown	Total
Normal	47	7	0	4	58
Mild	2	2	0	0	4
Severe	0	0	0	0	0
Unknown	1	1	0	6	8
Total	50	10	0	10	70

Cohen Kappa (1 attending) = 0.441 Moderate agreement.

Weighted Cohen Kappa = 0.540 Moderate agreement.

Note: Columns represent residents, and rows represent attending hysician.

Number of observed agreements = 55 (79%).

Number of agreements expected by chance = 43.1 (62%).

Discussion

Ultrasound units are becoming smaller and more affordable, and medical schools are increasingly incorporating ultrasound curricula into medical student training and medical residency training programs. Consequently, there has been an increase in studies, which have attempted to find a correct training regimen for internal medicine residents [9,14,15]. Some medical residencies have published curriculum guidelines to help incorporate FCU into medical residency training [16-19]. However, each of these studies that have analyzed methods to incorporate FCU training into the medical residency curriculum has their own set of findings, and so far there has been no general consensus in the literature. In these studies, the duration of training has also been shown to have a significant impact on the interpretation of images positively.

Multiple studies have been done to assess the right amount of training for medical students and residents to be able to perform FCU at the bedside and obtain reliable results [14,15,18-23]. Each of these studies used different methods of training including pre-test and post-test questionnaires, supervised followed by unsupervised bedside FCUs and/or dedicated lectures for FCU training. The time spent in training was variable from a minimum of 4 hours to 3 months. However, the exact amount of time needed for training residents still remains unclear. This could partly be due to the differences in protocols and periods of training.

Our current training protocol for FCU indicated that of the four parameters assessed, only the inferior vena cava assessment was found to have a moderate agreement between the medical interns and the academic internist/cardiologist. There was a mild agreement on the evaluation of the left ventricular systolic function but no significant agreement on the evaluation of the pericardial effusion and right ventricular systolic function. From these results, it can be inferred that the amount of training provided was insufficient for the assessment of all intended study parameters.

It is very likely that soon medical residents will need less training to perform ultrasound exams since the ultrasound training is being adopted in an increasing number of medical schools. However, our results emphasize that it is not only important to try to identify the appropriate amount of training necessary to perform reliable FCUs but, perhaps even more important, to develop tools to ensure accurate assessment of FCU results obtained by medical residents.

One limitation of our study was the small sample size of patients as well as residents. However, we did enroll every resident that did not have any previous ultrasound training. Another limitation was how the patients were selected to participate in the study. Each intern selected their own patients and this could have led to minor discrepancies and bias of the patient population. Detecting abnormalities in patients that do not exhibit significant disease characteristics is often challenging for new trainees and may have contributed to the results.

Conclusion

Despite some agreement between the medical interns and the attending in the evaluation of the IVC and the left ventricular function, we demonstrated that our proposed training protocol was not sufficient to appropriately prepare interns to do unsupervised FCU.

Further studies are needed to more precisely determine the minimum amount of training that is necessary for accurate FCU to be performed by interns in a community hospital. Most importantly, independent of the amount of training, it is necessary to assess the accuracy of the FCU performed by medical interns after the training and also assess if the knowledge gained through these training procedures is being retained in the future.