Implementation of a Longitudinal Critical Care Fellowship Ultrasound Curriculum.

Background: The use of point-of-care ultrasound as a diagnostic and interventional tool is rapidly becoming standard of care in critical care medicine; a standardized training curriculum is needed to ensure provider proficiency. Objective: This study aimed to describe a longitudinal critical care ultrasound (CCUS) curriculum in a pulmonary critical care medicine (PCCM) fellowship training program. It evaluated the curriculum's impact on fellows' knowledge, skills, and self-reported confidence and retention of these attributes.
Methods: We conducted a prospective observational study of a longitudinal CCUS training program within a single PCCM fellowship training program. Knowledge, skills, and confidence of 22 fellows were assessed at baseline; after initial training; and at 6, 12, and 18 months in five domains (ultrasound basics, vascular, lung/pleural, abdomen, and cardiac). We quantified changes in CCUS knowledge, confidence, and skills by fellowship class and assessed for longitudinal retention of these three attributes. The difference in scores between new first-year fellows undergoing formal training and second-year fellows with previous informal training was compared at matched time points.
Results: After the initial formal training, there was a significant increase in knowledge, skills, and confidence scores, which were maintained or continued to increase up to 18 months. Fellows with 1 year of formal training also had a higher level of knowledge and skills than fellows with 1 year of informal training, although they had similar levels of self-reported confidence in their skills.
Conclusion: A formal, longitudinal CCUS curriculum implemented in a PCCM fellowship program improves trainees' knowledge and skills in various ultrasound domains in addition to their confidence in using ultrasound for patient care. A longitudinal curriculum results in retention of all three attributes and appeared to be more effective than an informal training program based on teaching during rounds, but this needs to be replicated in a larger cohort.

Point-of-care ultrasound (POCUS) is widely perceived as a valuable diagnostic and interventional tool in the critical care setting. Although the use of POCUS has recently expanded, many fellowship programs lack longitudinal training curricula and defined competencies to achieve proficiency (1), which can lead to providers employing ultrasound techniques without the benefits of adequate training, expertise, and supervision (2). The American College of Physicians acknowledges the importance of POCUS in medicine and recognizes the need to establish guidelines for its use, to define a training curriculum, and to expand training availability, but further data are needed to achieve these goals.

Although almost all critical care fellowship programs have ultrasound systems for use in the intensive care unit (ICU), less than half of these programs report having a formal ultrasound curriculum (1). Cited barriers to implementing a curriculum include the lack of faculty, inadequate expertise, time constraints, and the absence of standard competencies (1–3). Common characteristics of existing training structures include a combination of lectures, case logs, hands-on mentorship, practical assessment, and simulation laboratory opportunities (1, 3, 4). However, there are currently no standardized training competency requirements. Programs vary widely in the number of scans required, the skills assessed, the methods of assessment, and the inclusion of procedural skills (4). Despite the diversity in instruction, individual workshops or a series of courses have proven to improve providers’ short-term knowledge (5–9), skills (10, 11), and confidence in ultrasound use (5, 11, 12). Training experience in ultrasound also increases the likelihood providers will use it as a decision-making tool in clinical practice (5, 9).

To date, there is a paucity of data on the impact that an integrated ultrasound curriculum during a critical care fellowship program has on knowledge and skill retention throughout fellowship, and there are no studies comparing an informal to a formal POCUS training program. We therefore designed a study to assess the impact of a longitudinal critical care ultrasound (CCUS) training program on fellows’ skills, knowledge, and self-reported confidence. We hypothesized that a formal longitudinal POCUS training program would result in significant improvements in knowledge, skills, and confidence levels compared with an informal POCUS training program incorporated into routine patient care rounds.

Methods

Study Design and Participant Selection

This study was approved by the Louisiana State University Health Sciences Center (LSUHSC) Institutional Review Board (#9515) with a waiver of informed consent. We conducted a prospective observational study of a longitudinal POCUS training program for first-, second-, and third-year pulmonary and critical care medicine (PCCM) fellows at LSUHSC from January 2017 until July 2018. Before January 2017, the LSUHSC PCCM fellowship did not have a formal POCUS training program, and ultrasound skills were taught informally only on inpatient teaching rounds. State-of-the-art ultrasound machines were available for clinical use in all of the training hospitals.

POCUS Training Curriculum

Beginning in January 2017, all PCCM fellows at LSUHSC participated in a formal POCUS curriculum based on prior literature and recommendations (11, 13). The course was taught by four faculty members, three of whom were PCCM attendings with POCUS experience (M.R.L., R.S.T., and B.P.D.), and one was the Emergency Medicine ultrasound training program director (C.B.). Each course was conducted on two consecutive Thursdays and consisted of a total of six 45-minute didactic lectures by the instructors on the following topics: ultrasound basics, vascular, thoracic, abdominal, cardiac basics, and cardiac pathology. On each training day, this was followed by a 2-hour skills session, in which each learner received personalized hands-on ultrasound acquisition teaching using healthy volunteers. To then align with the start of the fellowship year, these sessions were repeated in July 2017 and July 2018. In between each course, there was a weekly maintenance of skills session with one instructor (either M.R.L. or B.P.D.) and a small group of two to four fellows. Each session took place in the ICU and lasted for 1 hour. Ultrasound image acquisition and interpretation were reinforced for each of the POCUS domains, with an emphasis on relevant abnormal findings. Fellows attended if they were on an ambulatory, elective, or research rotation; on average, each fellow attended one session per month.

Training Assessments

Assessments of POCUS knowledge, skills, and confidence were conducted before and after training at baseline (Training Session 1), 6 months (Training Session 2), and 18 months (Training Session 3). Additionally, selected assessments were conducted at 12 months after Training Session 1, although there was no training course conducted at that time (Figure 1). All assessments were repeated for five domains: ultrasound basics, vascular, thoracic, abdominal, and cardiac. Knowledge assessments consisted of a written test comprising 30 multiple-choice or fill-in-the-blank questions, using both still images and ultrasound videos. Correct answers were not provided after the precourse test so as to not influence the postcourse test. Confidence in each of the five domains was self-rated by each learner, using a 5-point Likert scale (1 = low confidence; 5 = very high confidence). Thirty-five image acquisition and structure identification skills were conducted in the ICU with each learner, testing the same five domains. To obtain credit, each ultrasound view/question had to be obtained or answered within 2 minutes. A single instructor (M.R.L.) used a standardized interpretation checklist to assess the adequacy of image acquisition and structure identification; the checklist was developed using a previous consensus statement (13) and a prior single-center study (11). The full knowledge and skills assessments, Likert confidence scale, and interpretation checklist are provided in the data supplement.

Figure 1.

Training and assessment timeline. Training sessions and assessments occurred at baseline, 6 months, and 18 months. Additional assessments were conducted at 12 months. There were five cohorts of fellows; three cohorts (A–C) underwent the baseline training session, and two cohorts (D and E) joined fellowship during the longitudinal follow up. Arrows denote the ongoing period of fellowship training, “X” represents graduation or end of fellowship, and closed circles indicate the start of fellowship for individual cohorts. Solid lines represent periods of formal training, and dashed lines represent historical periods of informal training. The black closed squares and lines symbolize the time periods for comparison between the informal training program (cohort B at baseline) and the formal training program (cohort C at 12 months). See the Methods for more details.

Statistical Analysis

All statistical analyses were conducted using STATA 13 and Graph Pad Prism 5. Because of the limited sample size, nonparametric statistical tests were used for all analyses; data are displayed as median (interquartile range [IQR]).

Assessment scores were compared between cohorts A through C (Figure 1) before the first training session using Kruskal-Wallis with Dunn’s post hoc test. Spearman correlation was used for correlations between knowledge, confidence, and skills at baseline. Next, participants were grouped by their fellowship training levels (first, second, and third year). As an example of the grouping, cohort C at Training Session 1, cohort D at Training Session 2, and cohort E at training session 3 were all pooled together as first-year fellows. Assessments were compared immediately before and 1 week after the POCUS training sessions for each grouped training level using Wilcoxon signed rank test for paired data. To assess longitudinal retention of knowledge, confidence, and skills in cohort C, assessment scores were compared between the Training Session 1 (postcourse) and 12 months later using Wilcoxon signed rank test for paired data. In addition, assessment scores for cohort C were compared between the Training Session 1 baseline and after 12 and 18 months using a Friedman test for repeated measures. Lastly, we sought to compare 1 year of informal training to 1 year of our formal longitudinal POCUS training program. As shown in Figure 1, we compared the precourse assessments for cohort B (who received 12 months of informal POCUS training) to assessments for cohort C at 12 months of training using a Mann-Whitney U test.

Results

Cohort Description

Through the 18 months of follow up, a total of 22 fellows participated in the course; the number of training sessions for each fellow was variable based on when each entered and exited fellowship training (Figure 1). Four fellows had received prior formal POCUS training (emergency medicine residency [n = 1], courses offered by PCCM societies [n = 3]). Before Training Session 1, overall scores for POCUS knowledge, confidence, and skills were lowest in cohort C (who were first-year fellows) but similar between cohorts A and B; this pattern was also found for many of the individual domains (Table E1 in the data supplement). There were strong correlations between confidence and skills (r = 0.79; n = 13; P = 0.001), confidence and knowledge (r = 0.79; n = 13; P = 0.001), and skills and knowledge (r = 0.76; n = 13; P = 0.002; Figure E1).

Changes in POCUS Knowledge, Confidence, and Skills by Fellowship Class

Participants were grouped by their training levels (first-, second-, and third-year fellows), and assessments were compared before and after each POCUS training session. For knowledge, all three grouped training levels improved after the course (Figure 2A and Figure E2), with first-year fellows having the greatest increase (median increase of 7 [IQR, 3.5–10.5] points for first years, 1.5 [0.3–4.8] for second years, and 2 [1-3] for third years; overall P = 0.001). First-year fellows significantly improved their knowledge in all domains, whereas second- and third-year fellows improved in some but not all individual domains (Table E2). All three grouped training levels improved their self-rated confidence after the course (Figure 2B and Figure E2) (median increase of 6.5 [IQR, 1.8–8.0] points for first years, 4 [1-4] for second years, and 2 [1.5–3] for third years; overall P = 0.15). First-year fellows significantly improved their confidence in all domains except for vascular, whereas second- and third-year fellows improved in most but not all individual domains (Table E3). Similarly, all three grouped training levels improved their skills after the course (Figure 2C and Figure E2) (median increase of 9 [IQR, 7–11] points for first years, 9 [7-11] for second years, and 6 [4-9] for third years; overall P = 0.21). First- and second-year fellows significantly improved all individual skill domains except for ultrasound basics (Table E4).

Figure 2.

Change in (A) knowledge, (B) confidence, and (C) skills scores from precourse to immediately postcourse, grouped by fellowship training levels. Plots show the median with interquartile range together with data points for individual participants.

Longitudinal Retention

Improvements in POCUS knowledge, confidence, and skills that were gained after Training Session 1 appeared to be strongly maintained after 1 year of training. Although limited by a small sample size in Cohort C, the overall knowledge score increased from a baseline median of 16 (IQR, 13–18) to 27 (26–29) 12 months after initial training. Similar robust increases were seen in confidence (median, 12 [IQR, 11–14] at baseline versus 20 [18-21] 12 months after initial training) and skills (median, 18 [IQR, 17–21] versus 33 [32-33]). This trend was true for nearly all the individual domains (Table E5). When assessing precourse baseline, 12-month, and 18-month assessments for confidence and knowledge, confidence continued to increase, whereas knowledge increased at 12 months and then was maintained to 18 months (Figure 3). Skills were not reassessed at 18 months.

Figure 3.

(A) Knowledge and (B) confidence scores at precourse baseline and 12 months and 18 months of follow up for Cohort C. Plots show the median with interquartile range together with data points for individual participants. Maximum scores are denoted by the dotted line. P value is for Friedman’s test for repeated measures.

Formal Compared with Informal POCUS Training Program

As detailed in the Methods and Figure 1, before Training Session 1, all POCUS training was informal. We compared cohort C assessments after 1 year of training to cohort B before the formal course (i.e., assessing their knowledge, confidence, and skills gained through the prior year of informal POCUS training). Knowledge and skills scores were significantly higher in cohort C fellows exposed to 1 year of formal training compared with cohort B fellows exposed to 1 year of informal training; confidence was not significantly different between these groups (Figure 4 and Table E6).

Figure 4.

(A) Knowledge, (B) confidence, and (C) skills scores for Cohort C after 1 year of formal POCUS training compared with Cohort B at baseline after receiving 1 year of informal POCUS training (see Methods). Maximum scores are denoted by the dotted line.

Discussion

As the use of CCUS increases in clinical practice, there arises a need for standardized, effective education for future practitioners. This study shows that a formal, longitudinal CCUS curriculum incorporated into a PCCM fellowship program not only increases trainee knowledge, skills, and confidence but also results in retention of these attributes. This longitudinal curriculum applies principles of the spacing effect, the concept that repetition of material at set time intervals improves retention (14). In addition, the maintenance of skills sessions adhered to the principles of deliberate practice in that they focused on specific goals, provided immediate feedback, and built off of prior sessions to push the fellows’ skill sets to a higher level (15).

Consistent with previous research, this study supports that formal ultrasound training improves trainees’ knowledge (5, 7, 9, 11), skills (6, 9–11), and confidence (9, 11, 12). Our results also demonstrate that although the intervention resulted in positive outcomes for all fellows, it had the greatest impact on first-year trainees in terms of overall improvement in scores as well as improvement in the greatest number of domains.

These data additionally contribute novel insight into the long-term effects of a longitudinal curriculum by quantifying retention of knowledge, skills, and confidence. All three attributes showed improvement after 12 months of training in nearly all individual domains. Confidence continued to increase over an 18-month period, whereas knowledge was gained until 12 months and then maintained, implying that the longitudinal experience and clinical practice have positive impacts on fellows’ comfort with POCUS techniques. Similar to the present study, Kelm and colleagues found that first-year internal medicine residents exposed to a 6-month ultrasound curriculum improved their ability to correctly identify images 6 months after course completion, compared with the effects of a single stand-alone workshop (8). In contrast, Dinh and colleagues found that 3 months after a single, two-day course, there was decreased retention of previously learned knowledge and skills (9).

We also observed that as knowledge and skills scores improved, fellows’ confidence appropriately followed suit. Mellor and colleagues found that together with improving knowledge and confidence, a 1-year POCUS curriculum for internal medicine residents resulted in an increased number of trainees that intended to use ultrasound in clinical practice (12). It has also been shown that practitioners who increased their number of ultrasound assessments were more likely to make clinical decisions based on ultrasound findings after receiving a single formal training course (5). Previous research has also demonstrated that despite not achieving competency after a series of POCUS short courses, doctors continued to perform ultrasound to aid clinical diagnosis without oversight or supervision, emphasizing the need for appropriate training and evaluation (1).

In contrast to previous studies (5, 7, 9, 11), the present study also compared outcomes between an intervention and a control group. Our formal training curriculum, in comparison to informal training on inpatient teaching rounds (the most common approach to ultrasound education [3]), resulted in a higher acquisition of knowledge and skills. However, despite the difference in skills and knowledge, fellows with formal training reported similar levels of confidence in their ability to perform POCUS compared with fellows with only informal training. This suggests that although both groups have the same level of comfort implementing the technique into their practice, the trainees with formal training were better equipped to do so.

This study also disputes cited barriers to ultrasound training, including the perceived need for many expert faculty members and an abundance of dedicated time (1–3). Instead, our approach represents a practical intervention that affects longitudinal results. This course required only four faculty members, one of whom was trained in emergency medicine. We propose that preexisting elements of formal training already established in fellowship programs, such as didactic lectures, simulations, and hands-on skills training (3, 16, 17), can easily incorporate POCUS training to improve effectiveness. To alleviate the lack of faculty expertise (3), interprofessional education with surgery, anesthesia, emergency medicine, radiology, internal medicine, and/or pediatric colleagues should be considered (11).

This study obtained data from a single center, resulting in a limited sample size of 22 fellows. An additional limitation is the inability to assess the informal ultrasound training fellows received during inpatient teaching rounds. Because of logistical barriers in a busy clinical fellowship, there were some missing data, particularly for the skills testing, but we do not feel that this impacted our results. The skills and knowledge assessments were also generated by the investigators based on prior literature and recommendations (11, 13) but are not validated evaluations, as none exist for POCUS. In addition, because all fellows received the same comprehensive training program, we cannot evaluate which components (didactics, hands-on sessions, or weekly skills maintenance sessions) are most valuable. Finally, this study was done in 2017–2018, and it is unknown if the results would apply to a more contemporary cohort of fellows, who presumably now have more POCUS training before starting fellowship.

In conclusion, implementing a formal, longitudinal CCUS curriculum throughout fellowship training improves trainees’ knowledge and skills in various ultrasound domains in addition to their self-reported confidence. The longitudinal nature of the curriculum results in retention of all three attributes to facilitate use in future practice and appeared to be more effective than an informal training program based on teaching during rounds. Because of the increasing prevalence of CCUS and evidence supporting formal training incorporated into a fellowship program, future studies are needed to establish the minimum number of scans to obtain competency in individual domains, to externally validate our standardized curriculum, and to investigate trainee and faculty perception of the benefits and burden of such a training program.