Impact of Deliberate Practice on Point-of-Care Ultrasound Interpretation of Right Ventricle Pathology.

Background: Point-of-care (POC) ultrasound (POCUS) has become an essential tool in caring for critically ill patients in several specialties. Mastery in POCUS requires competency in image acquisition, image interpretation, and integration into clinical care. Deliberate practice is an effective method for performance improvement in many areas of medical education; however, it is not well described in the literature for POCUS training. Objective: To analyze differences in the effect of deliberate practice in POCUS image interpretation on performance improvement in groups with varying skill levels.
Methods: We recruited attending physicians and trainees with varying degrees of expertise in POCUS to complete a 50-item educational instrument on the interpretation of right ventricle size and function. The instrument incorporated deliberate practice for the task of correctly identifying right ventricle size and function as either normal or abnormal. Pulmonary critical care trainees obtained and interpreted POCUS images of patients with diagnosed acute pulmonary embolism, which were compared with gold-standard, two-dimensional echocardiographic scans obtained by an expert technician and interpreted by a cardiologist board-certified in echocardiography. We mapped learners' cumulative accuracy on a learning curve to assess their performance. In addition, we compared groups on the basis of prior experience with using POC echocardiography.
Results: Seventy-nine of 81 participants completed the survey and examination and were included in the analysis. Of the participants, 69 (87.3%) were trainees. The overall cumulative accuracy for the group was 72.9%. All groups demonstrated improvement in accuracy with repetitive practice.
Conclusion: Deliberate practice in POC echocardiograph interpretation is effective for improving performance in a wide range of learners. Further study is needed to define accuracy cutoffs for competency to help guide learning plans and program requirements and for application into a model for global POC echocardiography competence.

Point-of-care (POC) ultrasound (POCUS) is an essential tool in acute patient care for diagnostics, procedural guidance, clinical monitoring, and resuscitation. Clinicians use POCUS to rapidly and accurately diagnose the etiologies of acute, life-threatening conditions, including undifferentiated shock, acute respiratory failure, and cardiac arrest (1–3). Frontline clinicians who use POCUS obtain immediate and valuable clinical information that can be rapidly integrated into patient care. This is in direct contrast to the use of traditional ultrasonography, which often requires multiple steps and participants, often leading to delays in patient care (4–6).

The rapid growth of POCUS has appropriately created important questions regarding the training and competency of the performing physician. As ultrasound systems become more widely available, allowing for increased use, the importance of structured training to ensure competency becomes paramount for patient safety. The American College of Chest Physicians–La Société de Réanimation de Langue Française Statement on Competence in Critical Care Ultrasonography defines competence as the knowledge of image acquisition, interpretation, and clinical applications for each subset of POCUS (pleural, lung, abdominal, and vascular and critical care echocardiography) (7–9). Recommendations for training include longitudinal practice after initial training; however, there are limited data on how much and what type of POCUS practice is needed or effective for achieving basic competency. Given the broad use of POCUS and the emphasis by the Accreditation Council for Graduate Medical Education on competency assessments and developmental milestones throughout training (10), developing universal methods of tracking performance improvement toward competency in POCUS is increasingly important.

Mastery learning is an education model that includes four key elements: the pace of training is set by the individual learner’s progress; there are fixed, defined outcomes; deliberate practice (repetition with feedback) takes place; and there is increasing complexity at each progressive level. As an educational model, mastery learning has been well studied in clinical education and has been shown to be superior to traditional medical education in multiple skills, ranging from advanced cardiac life support to procedural skills (11–13). Deliberate practice is an education technique focused on purposeful and systematic repetitive practice with a goal of ongoing skill improvement. Deliberate practice encompasses multiple elements, including highly motivated learners; well-defined learning objectives; appropriate levels of difficulty; focused, repetitive practice; rigorous, reliable measurements; informative feedback from educational sources (i.e., simulator, instructor); the ability for error correction; and the ability to reach a mastery standard in which learning time is allowed to vary by learner (11). Learning curves, which plot performance against practice time, have been used in medical education to demonstrate the effectiveness of the learning approach and to define the amount of practice needed to achieve specific performance milestones (14, 15). The structure of a learning curve includes three key elements: first, the initial performance level at the y-intercept and the learner’s base knowledge; second, the slope, which describes the rate of learning; and third, the asymptote, denoting the maximum performance able to be achieved in the specific learning scenario (Figure 1) (16). A learning curve can describe the change in performance of an individual learner or of a group and can be a valuable tool for examining the effectiveness of an educational intervention (17).

Figure 1.

Learning curve example.

The aim of our study was to use learning curves to demonstrate and analyze performance improvement through deliberate practice in POCUS image interpretation. We developed a deliberate practice model for qualitative image interpretation of right ventricle (RV) size and function, given its importance in assessment of patients with acute pulmonary embolism and that it is achievable by trainees and frontline clinicians (6, 18, 19). The views expressed in this article do not communicate an official position of the author-affiliated institutions.

Methods

Participant Selection

We recruited physician learners from two sources; a regional ultrasound conference and academic conferences held in emergency medicine, internal medicine, and pulmonary/critical care at our institution. The learners included residents, fellows, and attending physicians with varying degrees of POCUS experience. We informed the learners of the purpose of the study. Anonymous survey and examination result data were collected from participants. The institutional review board at Mount Sinai Beth Israel approved this study for exemption (Health Sciences Institutional Review Board Number 17-00213, Grant and Contract Office Number : 17-0567).

Examination and Survey Development

Deidentified POC echocardiogram videos from patients enrolled in a single-center registry of patients with acute pulmonary embolism were used in the development of this instrument. Cases were included if they had at least three of the four adequate POC echocardiogram views, including the parasternal short-axis, parasternal long-axis, apical four-chamber, and/or subcostal four-chamber views. POC images were initially acquired and saved at the bedside by the pulmonary and critical care fellows or attending physicians within 24 hours of diagnosis of an acute pulmonary embolism. The gold standard for accuracy comparison consisted of images obtained on an official two-dimensional transthoracic echocardiography machine by an expert technician within 48 hours and reviewed by a board-certified cardiologist in echocardiography. Each of the POC echocardiogram videos was reviewed by two nationally recognized POCUS educators (S.A. and P.P.) who agreed on the interpretations as a prerequisite for inclusion in our slide set. Each examination item consisted of one case with multiple POC echocardiogram clips. Out of a total of 50 cases included in the examination, 32 (64%) had normal RV size and function, 17 (34%) had abnormal RV size and function, and 1 (2%) had abnormal size with normal function as confirmed by the official transthoracic echocardiogram. This distribution was based on population data from our institution and other studies reporting a 30–50% prevalence of RV dysfunction in patients with acute pulmonary embolism (6, 20–22). There is no consensus on the number of images a learner must review to obtain competency in RV assessment in POC echocardiography. We chose 50 cases, as we anticipated this would be well beyond the number of cases needed for learners to practice interpreting both normal and abnormal RVs in POC echocardiography.

A Microsoft PowerPoint presentation was developed with one slide per case, and videos were embedded in Windows Media Video format on a repeated loop for 30 seconds. Each case corresponded to an item on an answer sheet for RV size (abnormal or normal) and RV function (abnormal or normal). Each case slide was followed by immediate feedback with the correct answer (RV size = abnormal or normal and RV function = abnormal or normal). Participants were told that these cases were from patients presenting with an acute pulmonary embolism, and no further clinical history was provided. Cases were presented consecutively to participants in groups, and the order of cases was randomized to prevent bias on the part of the researchers and learners. Each participant answered individually on their answer sheet, and then all participants received the immediate standardized feedback as a group in real time.

A pretest survey and posttest survey were also developed and included demographics, ultrasonography experience, and interpretation confidence graded on a 5-point Likert scale (1 = minimal confidence and 5 = full confidence).

Deliberate Practice Procedure

Participants were given a linked pretest survey, 50-item examination, and posttest survey during a formal conference in a proctored setting. The examination was projected on a large screen. Each case was played for 30 seconds, and participants completed their answers during this time. After each case, immediate feedback with the correct answer was provided on the next slide for 5 seconds. No further instruction or explanation was provided during the examination.

Data Analysis

Each completed case was scored as one item by using dichotomous scoring (0 for incorrect or 1 for correct). Each case was scored as correct if the learner correctly identified both the RV size and the RV function. Accuracy was calculated for each learner, the overall group, and subgroups predefined on the basis of experience. Novices were defined as learners reporting performance of fewer than 25 POC echocardiography examinations. Intermediate experience was defined as learners reporting the performance of 25–300 POC echocardiography examinations. Experts were defined as learners reporting performance of more than 300 POC echocardiography examinations. This scale was chosen on the basis of our choice of gold-standard interpretation as that of a board-certified cardiologist in echocardiography and on the basis of the National Board of Echocardiography cardiology board requirements during the time of the study (23).

Learning Curve Assessment

The data were analyzed by using a cumulative moving average model that took each new data point (the score for each case completed) and incorporated it into the overall average (24). This can be best understood as assigning a new overall examination score every time a repetition is completed. Of note, until 5–10 cases have been completed, the overall score will be less indicative of overall achievement, as it will be highly sensitive to each new case submission. The cumulative average was calculated for each individual learner. The cumulative averages for each learner were averaged for an overall group learning curve. The learning curves were displayed in graphic format, with the x-axis representing the number of cases completed and the y-axis representing accuracy.

Comparison of Subgroups

To further evaluate the relationship of accuracy with the amount of practice and the impact of experience on accuracy, repeated-measure logistic regression was applied, with the dichotomous score for each case (correct/incorrect) serving as the endpoint. The cases completed (1–50), experience subgroup (novice, intermediate, expert), and their interaction were included as fixed effects, and the learners were included as random effects to account for correlations between repeated measures. Type 3 likelihood ratio tests were used to test the significance of each effect.

We performed all analyses by using SAS Studio 3.7 and Microsoft Excel 2013.

Results

We recruited 81 participants. Two participants (one attending physician and one resident) did not complete the examination and were excluded from analysis. Of the 79 remaining participants, 69 (87.3%) were trainees, 45 (57%) were residents, 24 (30.3%) were fellows, and 9 (11.5%) were attending physicians. Participants were from multiple specialties: 35 (45%) were from emergency medicine, 17 (22.1%) were from internal medicine, 17 (22.1%) were from pulmonary/critical care medicine, and 8 (10%) were from critical care medicine. Most participants (85.9%) reported some prior training in ultrasonography.

Most participants had performed POC echocardiography, with a median report of 28 examinations performed. The median pretest confidence score for POC echocardiography performance was 4 (interquartile range [IQR], 3–4) out of a 5-point Likert scale. The median pretest confidence scores for the interpretation of RV dilation and dysfunction were reported as 4 (IQR, 3–4) and 3 (IQR, 2–4), respectively. The median posttest confidence scores for the interpretation of RV dilation and dysfunction were reported as 4 (IQR, 3–4) and 3 (IQR, 2–4), respectively (Table 1).

Table 1.

Demographic and confidence data

Total participants	79
Experience,* n (%)
Trainee, n (%)	69 (87.3)
Resident	45 (57)
Fellow	24 (30.3)
Attending physician, n (%)	9 (11.5)
Years in practice, mean (SD)	8.69 (8.67)
Reported training curriculum,* n (%)
Emergency medicine	35 (45.5)
Internal medicine	17 (22.1)
Pulmonary and critical care	17 (22.1)
Critical care medicine	8 (10.4)
Prior ultrasound training	67 (85.9)
National course	4 (5)
Didactic learning	43 (54.4)
Journal club	22 (27.9)
Self-directed learning	42 (53.2)
Local/regional course	21 (26.6)
Informal hands-on learning	54 (68.4)
Formal hands-on learning	49 (62)
Prior ultrasound training content,* n (%)
Vascular guidance	60 (76.9)
Vascular diagnostic (DVT)	53 (68)
Pleural guidance	42 (53.9)
Lung ultrasound	59 (75.6)
Abdominal ultrasound	59 (75.6)
Basic echocardiography	61 (78.2)
Advanced echocardiography	14 (18)
Procedure experience,† median (IQR)
Thoracentesis	2 (0–35)
Internal jugular catheters	20 (3–55)
DVT studies	10 (0–50)
Basic echocardiography	28 (6–95)
Pretest confidence score,* median (IQR)
RV dilation	4 (3–4)
RV dysfunction	3 (2–4)
Posttest confidence score,‡ median (IQR)
RV dilation	4 (3–4)
RV dysfunction	3 (2–4)

Definition of abbreviations: DVT = deep vein thrombosis; IQR = interquartile range; RV = right ventricle.

Procedure experience is expressed as the median number performed. Confidence is reported as the median score on a 5-point Likert scale.

1–3 missing survey responses.

7–8 missing survey responses.

11 missing survey responses.

The cumulative accuracy for the overall group was graphically mapped (Figure 2). The graph is consistent with prior research of skill advancement with repetitive practice (14). Of note, the initial decline in cumulative accuracy can be attributed to the low number of cases completed (denominator), with an incorrect answer thus imposing a greater influence on the overall average. This effect is seen to have resolved as the cases progressed.

Figure 2.

Cumulative accuracy for all learners (mean). RV = right ventricle.

Subgroup Assessment

The novice group consisted of 33 learners, with 90% identifying as residents and 69.7% reporting some prior ultrasound training. Their median pretest confidence score for interpreting POC echocardiography scans was 2.5 out of 5. The intermediate group consisted of 34 learners, with the majority (52.9%) identifying as fellows and 97.1% reporting prior ultrasound training. The median pretest confidence score for interpreting POC echocardiography scans was 4 out of 5. The expert group consisted of five learners, with 80% being identified as attending physicians and 100% reporting prior ultrasound training. Their median pretest confidence score for interpreting POC echocardiography scans was 5 out of 5 (Table 2). Similar to the overall group assessment learning curves, these learning curves were able to be graphically displayed, demonstrating overall improvement with practice (Figure 3). From repeated-measure logistic regression, the estimated logit values of the accuracy of each case were measured by using −0.161 + 0.039x, 0.106 + 0.454x, and 1.303 + 0.026x for the novice, intermediate, and expert subgroups, respectively, where “x” is the measure of effort (cases completed). The estimated trendlines of accuracy as an inverse of the logit are displayed in Figure 4. The accuracy levels were significantly different among different experience subgroups (P = 0.0016), and the accuracy was significantly increased as more cases were completed (P < 0.0001). The improvement rates (regression slopes) among subgroups were not significantly different, as the interaction term did not reach statistical significance (P = 0.316), potentially because of the small sample size in the expert group.

Table 2.

Results of group analysis by experience

	All (n = 72; 7 Did Not Report Experience)	Novice (n = 33)	Intermediate (n = 34)	Expert (n = 5)	P Value
Training, n (%)
Resident	40 (55.6)	30 (90.9)	10 (29.4)	0	<0.0001
Fellow	22 (30.6)	3 (9.1)	18 (52.9)	1 (20)
Attending physician	9 (12.5)	0	5 (14.7)	4 (80)
Prior ultrasound training	61 (84.7); 1 did not report	23 (69.7)	33 (97.1)	5 (100)	0.0049
Pretest confidence score, median (IQR)
POC echocardiography	4 (3–4)	2.5 (1–4)	4 (4–4)	5 (5–5)	<0.0001
RV dysfunction	3 (2–4)	2 (1–3)	2 (1–3)	4 (4–4)	<0.0001
RV size	4 (3–4)	3 (2–3)	3 (2–3)	5 (4–5)	<0.0001
Posttest confidence score, median (IQR)
RV dysfunction	3 (2–4)	3 (2–3)	3 (2–3)	4 (4–4)	0.0032
RV size	4 (3–4)	3 (2–4)	3 (2–4)	4 (4–5)	0.0015

Definition of abbreviations: IQR = interquartile range; POC = point of care; RV = right ventricle.

Analysis of variance was used for continuous variables. A Fisher exact test was used for categorical variables.

Figure 3.

Cumulative accuracy for subgroups based on the number of point-of-care echocardiography examinations performed: novice (<25 performed, red line), intermediate (25 to <300 performed, blue line), and expert (>300 performed, black line). RV = right ventricle.

Figure 4.

Cumulative accuracy for subgroups obtained by using repeated-measure logistic regression. RV = right ventricle.

Discussion

In our study, we demonstrated the effectiveness of a deliberate practice instrument for RV ultrasound interpretation by using a learning curve model. Analysis of the slope of learning and cumulative accuracy by experience group showed an overall improvement in accuracy with repetition for all groups. These results have significant implications that expand our current understanding of skill development in POCUS. This deliberate practice exercise led to improvements in the accuracy of RV ultrasound assessment for learners at any skill level and is reproducible, easy to create and administer, and applicable to all inpatient training settings that are performing POCUS education. It represents a feasible and structured approach to autodidacticism, which has been traditionally advocated when a traditional curriculum or local mentors are not readily available (25).

The current guidelines from the American College of Chest Physicians and La Société de Réanimation de Langue Française for critical care ultrasound education include a framework of well-defined outcomes and milestones of increasing complexity. However, they do not include a practical, generalizable model for deliberate practice for all programs. This limitation lies in the inherent difference between POCUS education and education in traditional medicine. In traditional medicine, the standard is defined by increasing expertise with practitioner seniority. In ultrasound education, however, trainees tend to have more experience and expertise than senior faculty. In 2011, Eisen and colleagues (26) performed a survey of pulmonary/critical care medicine and critical care medicine programs in the United States, with 41% of programs that responded reporting that they lacked sufficient faculty trained in POCUS. In 2014, Mosier and colleagues (27) performed a similar survey of critical care program directors and reported that only 25% of pulmonary and critical care faculty were trained in ultrasonography. Most recently (2021) Brady and colleagues (28) surveyed pulmonary and critical care fellows regarding ultrasound education, with the greatest barrier reported as being a lack of trained faculty. Without faculty to consistently give feedback, a deliberate practice model is difficult to achieve. Without this appropriate training, there exists a deficiency in complete ultrasound training and education. These instruments and learning curves can be applied on the individual level to assess individual growth and provide formative feedback without adding an impractical burden on supervising faculty, thus filling in the void.

Limitations of this study include the narrow scope for each element of POCUS as it applies to overall POCUS competency. We were able to study and analyze learning and accuracy in one subset of POCUS: image interpretation. Challenges remain in how to define and assess global POCUS competency, which includes multiple dimensions of skill (image acquisition, image interpretation, and clinical reasoning). Further study is needed to provide evidence concerning the validity of the image bank, including presenting cases in a random order to account for differences in difficulty, evaluating how well the score reflects diagnostic accuracy, and determining whether the image bank included an adequate mix of real-world pathologies. We were further limited in the aspect of RV ultrasound assessment in that we combined the scoring for the learners’ assessment of both size and function under the assumption that both are commonly abnormal in acute pulmonary embolism; however, further understanding of real-world prevalence would be valuable. Our population was mainly physicians within internal medicine, critical care medicine, and emergency medicine backgrounds with an interest in POCUS, limiting generalizability to other learners. Further study is needed to determine the value of this instrument as part of a more comprehensive curriculum, whether learning curves can aid in summative evaluations, and the longitudinal value of the deliberate practice over 6–12 months from the initial practice.

Conclusions

A deliberate practice exercise leads to improvements in the accuracy of RV ultrasound interpretation in learners with variable prior experience. Learning curves are effective for defining the cumulative accuracy of POC echocardiographic RV image interpretation for groups of learners. We foresee a role of varying deliberate practice models in supporting mastery learning in POCUS education.