Validation of a Web-based Platform for Online Training in Point-of-Care Diaphragm Ultrasound.

Diaphragm dysfunction occurs frequently in patients on mechanical ventilation (MV) and is associated with long-term morbidity and mortality (1, 2). Transthoracic diaphragm ultrasound (DUS) has emerged as a feasible and reproducible noninvasive technique to assess diaphragm structure and function during MV (3). DUS measurements can inform prognosis and may facilitate diaphragm-protective ventilation (4). Technical training remains an important hurdle to widespread dissemination of the technique (5). Although online learning platforms are increasingly employed in medical education and might facilitate dissemination, it is uncertain whether competency in DUS can be acquired via this medium. We set out to determine whether an online training module with expert feedback provided remotely can achieve competency in the DUS technique in trainees with no prior experience with DUS (DUS naïve) or no prior experience with any form of ultrasound (ultrasound naïve).

Methods

A prospective observational study was conducted in a quaternary intensive care unit from May to October 2019 following institutional ethics approval (Institutional Review Board #18-1192). Five trainees (from 43 potential candidates) were voluntarily selected based on their response to a survey and placed in one of two categories: 1) DUS naïve (n = 3, postgraduate year 5–6 pulmonary and critical care medicine fellows with experience in general critical care ultrasound [>30 studies] but no prior experience with DUS) and 2) ultrasound naïve (n = 2, research coordinators with no prior ultrasound experience). Each trainee completed an online DUS course (https://michener.blackboard.com) consisting of a 90-minute didactic curriculum on diaphragm anatomy, physiology, and DUS technique, including probe positioning, and an online test with 38 questions requiring an 80% correct response rate to pass. The ultrasound-naïve group was also given a 1-hour hands-on in-person ultrasound knobology session by an expert sonographer (S.D.) to familiarize them with the ultrasound machine (GE Venue, General Electric), probe manipulation, image acquisition, and archiving. Next, each trainee independently performed 17 B-mode and M-mode examinations (2 in healthy volunteers and 15 in MV patients). No hands-on training was provided by experts on the performance of DUS. Each exam consisted of two tidal breaths and one maximal inspiratory effort breath with measurement of diaphragm thickness at end-expiration (Tdi,ee), diaphragm thickening fraction during inspiration (TFdi), and maximal thickening fraction (TFdi,max). Measurements were made in the eighth or ninth intercostal space between the anterior and midaxillary lines using M-mode according to a previously published technique (3). During the training program, trainees received feedback on the quality of the acquired images and identification of zone of apposition through the online platform from two experts in DUS (J.W. and E.C.G.). Each trainee completed the course over the allotted 3 months.

In the validation step, all trainees and experts (S.D. and E.C.G. together) measured Tdi,ee, TFdi, and TFdi,max in 10 MV patients after obtaining informed consent. Each observer repeated the measurement a few minutes after the initial measurement. The observer order was randomized for each patient, with experts last to perform the ultrasound. Observers were blinded to each other’s findings, and all measurements were performed independently. The location of probe was marked during the first examination in each patient, and this mark was used as a guide for probe positioning in the same patient by subsequent examiners, as this method has been shown to be necessary to attain adequate reproducibility (3). The competency of the trainees (and hence the effectiveness of the training methodology) was established by comparing measurement agreement between the reference standard (measurement obtained by S.D. and E.C.G.) and each trainee using the method of Bland and Altman (6, 7). Intraobserver repeatability coefficients were computed from random effect models. Agreement on the diagnosis of diaphragm dysfunction (TFdi,max < 20%) was assessed by the multirater kappa.

Assuming a standard deviation for the difference in Tdi,ee between observers of 0.2 mm (larger than that previously observed) (3), a sample size of 10 patients was computed to be required to achieve 95% confidence intervals of ±0.2 mm for limits of agreement for Tdi,ee (6). Analyses were performed using R statistical software and SAS 9.4 software (SAS Institute).

Results

All trainees successfully completed the online module and the online test and uploaded 17 examinations for online review and feedback. In the validation phase, 10 MV patients were enrolled with mean (±standard deviation) duration of MV of 25 (±43) days (Table 1). All trainees were able to obtain DUS measurements on all 10 patients. TFdi,max could not be measured in three patients, as they were unable to tolerate a spontaneous mode of ventilation to make vigorous respiratory efforts. Bias and limits of agreement between trainees and experts varied slightly between trainees but were generally acceptable and similar to previously reported values for reproducibility of Tdi,ee and TFdi (Table 2, Figure 1). The coefficient of repeatability suggested acceptable within-observer reproducibility. Results were similar for DUS-naïve and ultrasound-naïve groups. Limits of agreement for TFdi,max were wide and variable between trainees (Table 2). In a sensitivity analysis excluding ultrasound-naïve trainees, the reproducibility parameters were generally very similar for all measurements.

Table 1.

Characteristics of critically ill study population for validation phase

Factor	Statistics (N = 10)
Age, yr	59 ± 15
Body mass index, kg/m2	24.7 ± 4.4
Duration of mechanical ventilation, d	25 ± 43
Tidal volume, ml	427 ± 91
Positive end-expiratory pressure, cm of H2O	7 ± 3
Respiratory rate	19 ± 6
FiO2	0.37 ± 0.09
Sex
Female	2 (20%)
Male	8 (80%)
Spontaneous effort on mechanical ventilation
No	3 (30%)
Yes	7 (70%)
Tracheostomy
No	6 (60%)
Yes	4 (40%)
Pressure support
No	3 (30%)
Yes	7 (70%)

Statistics are presented as mean ± standard deviation or n (column %).

Table 2.

Agreement between trainees and the reference standard after completing the online diaphragm ultrasound course

Measurement	Mean (Standard Deviation) in Study Population (Mean of All Observers)	Range in Study Population (Mean of All Observers)	Comparison between Experts and Trainees (Range of Values across Trainees)						Coefficient of Repeatability within Trainees*		Coefficient of Reproducibility between Trainees†
			Bias		Lower Limit of Agreement		Upper Limit of Agreement
End-expiratory diaphragm thickness, mm	2.8 (0.9)	1.6 to 4.9	−0.1 to 0.2		−0.2 to -0.4		0.2 to 0.3		0.3		0.2
Tidal diaphragm thickening fraction, %	15 (9)	2 to 26	−1 to 2		−13 to -4		7 to 12		11		7
Maximal diaphragm thickening fraction, %	48 (29)	11 to 101	−1 to 6		−17 to -56		26 to 40		N/A		36

Definition of abbreviation: N/A = not applicable.

The repeatability coefficient is the largest difference one would observe in repeated measurements in the same patient by the same observer 95 times out of 100 repeated measurements.

The reproducibility coefficient is the largest difference one would observe in repeated measurements in the same patient between two different observers 95 times out of 100 paired measurements.

Figure 1.

Bland-Altman plot comparing the average end-expiratory diaphragm thickness measured by each trainee to the value obtained by the experts. DUS = diaphragm ultrasound; LOA = limit of agreement; US = ultrasound.

Discussion

We found that, upon completion of an online DUS training platform, trainees with varying prior ultrasound experience and no prior DUS experience obtained DUS measurements of Tdi,ee and TFdi with acceptable agreement to measurements obtained by experts. The observed measurement precision was sufficient to detect clinically relevant changes in diaphragm thickness (±0.2 mm) and diaphragm thickening fraction (±15%) based on thresholds established in previous outcome studies (2, 4). However, the reproducibility of TFdi,max measurements was comparatively inadequate to distinguish the presence or absence of diaphragm weakness (TFdi,max < 20–30%), which might result in misdiagnosis of diaphragm weakness (8). We conclude that competency in measurements of Tdi,ee and TFdi can be achieved using a web-based training platform and that further work is required to develop the platform to disseminate competency in TFdi,max measurements.

Several approaches have been used to evaluate online learning in bedside ultrasound. A combination of web-based didactic training with either hands-on training or self-guided assessment has been found to be noninferior to traditional training methods (9, 10). For example, a combined approach of video tutorials and expert-guided hands-on training in DUS was more effective in obtaining windows and performing DUS measurements than video tutorial alone (11). Competency is often defined qualitatively, such as the ability to perform a procedure or to obtain an ultrasound view (10). A strength of the present study was that competency was assessed quantitatively rather than qualitatively, providing evidence of competency to obtain measurements in clinical practice or research. The web-based training approach described in the present study provides a potential model for disseminating competency in other ultrasound techniques. Future work to improve dissemination of competency in measuring TFdi,max is required, possibly by increasing the number of measurements obtained by a single operator to reduce measurement variability.

Conclusions

Our findings suggest that the online DUS training platform can be deployed to efficiently disseminate competency in DUS measurements of diaphragm thickness and TFdi at a level sufficient for application in research and clinical practice. Future research is required to improve training in the measurement of TFdi,max through the platform.