Lung-ultrasound objective structured assessment of technical skills (LUS-OSAUS): utility in the assessment of lung-ultrasound trained medical undergraduates.

PURPOSE: Recently, some attempts have been made to integrate lung ultrasound (LUS) teaching into medical curricula. However, current education studies of LUS are extremely heterogeneous due to the lack of evidence-based guidelines on LUS education. In particular, the assessment of competencies is poorly standardized and mostly relies on non-validated scales. A new validated tool, the objective structured assessment of lung ultrasound skills (LUS-OSAUS), has the potential to overcome these limitations. Therefore, we adopted the LUS-OSAUS tool to assess the competencies of a group of LUS-trained undergraduates. Existing no prior practical applications of the LUS-OSAUS, our aim was to investigate the practical utility of this tool and its applicability in the evaluation of US-trained medical students.
METHODS: Eight undergraduates (two males, six females) were enrolled on a voluntary basis to receive a theoretical and practical training in LUS. Once completed their training, each student performed an LUS examination on a different patient hospitalized for respiratory symptoms. The same eight patients were also scanned by a senior resident in emergency medicine for a comparison with students' results. Students and the senior resident were tested by an examiner using the LUS-OSAUS tool. We compared the scores obtained by operators in all areas of competence of the LUS-OSAUS, the total scores, and the time needed to complete the sonographic task.
RESULTS: Median students' score in the single items of the scale was significantly lower than the ones obtained by the senior resident (4.0 [3.3-5.0] vs. 5.0 [5.0-5.0]; p < 0.0001). Students scored significantly lower than the senior resident in each item, except for B-line identification, choice of the correct transducer, and suggested focused questions. Median total score was also lower for students compared to the senior resident (70.5 [61.0-74.8] vs. 84.0 [83.5-84.3] (p = 0.0116). Median time required to complete the examination was significantly higher for students (14.1 [12.8-16.1] vs. 4.7 [3.9-5.2] min, p = 0.0117).
CONCLUSIONS: The LUS-OSAUS tool allowed for a standardized and comprehensive assessment of student's competencies in lung ultrasound, and helped to discriminate their level of expertise from that of a more experienced operator. The scale also specifically tests the theoretical knowledge of trainees, thus making redundant the use of questionnaires designed for this purpose.

Introduction

Lung ultrasound (LUS) is a fundamental tool for the assessment of patients with dyspnoea and respiratory failure, and is widely adopted by physicians working across several medical specialties, including intensive care, emergency, pulmonary, and internal medicine [1-7].

Recently, some attempts have been made to integrate LUS teaching into medical curricula, either as focused sonographic tasks, like the detection of pleural effusion or pneumothorax, or as part of a more comprehensive ultrasound curriculum [8-11]. These evidences suggest that LUS has the potential to improve clinical skills of medical students at the bedside. However, despite the diffusion of LUS in the clinical practice, there are no clear evidence-based guidelines on LUS education: in fact, a recent review demonstrates the extreme heterogeneity of current education studies in terms of setup, teaching methods, and assessment of competencies [12]. For the assessment of theoretical knowledge, educators usually rely on ad-hoc multiple-choice questionnaires or open questions, often administered as before–after tests, a setup that suffers from minimal internal and external validity and considered obsolete. Instead, for practical skills’ assessment, various checklists have been proposed, none of which has undergone a validation process [12]. Hence, there is a need of a standardized approach for the assessment of LUS skills. In this regard, a recently validated tool, i.e., the objective structured assessment of lung-ultrasound skills (LUS-OSAUS), has the potential to overcome all the above limitations. This tool consists of 17 items grouped into six areas of competencies, selected through a Delphi consensus and then underwent construct validity [13]. In our study, we used the LUS-OSAUS to assess the competencies of a group of LUS-trained undergraduates from the University of Pavia.

Aim of the study

Existing no prior practical applications of the LUS-OSAUS, our aim was to investigate the practical utility of this tool, and its applicability in the evaluation of US-trained medical students. We wanted to determine whether the LUS-OSAUS could help to differentiate the level of competence of our group of LUS-trained undergraduates with that of a more expert operator.

Methods

Population and operators

We enrolled on a voluntary basis eight undergraduates (two males, six females), who were attending the 5th year of medical school at the University of Pavia. Participants were recruited among a group of ten medical students who had previously taken part (18 months earlier) to another study, which investigated the learning curve of the sonographic assessment of inferior vena cava [14]. During that previous experience, students had received theoretical teaching on the basic principles of ultrasonography, with a focus on the utility and limitations of inferior vena cava sonographic evaluation, as a surrogate of central venous pressure measurement. Moreover, they had been taught the technical skills for subxiphoid inferior vena cava visualization and measurement of the caval index, which they performed on real patients using the hand-held sonographic device Vscan™, GE Vingmed Ultrasound AS, Horten, Norway.

In the current study, we further trained our group of students on lung ultrasonography. Initially, students attended a 3-h didactic lecture, which also included a hands-on session. The didactic lecture was hold by two senior residents in emergency medicine (EM) who practice LUS on a daily basis (MM, SDP), and by a consultant in internal medicine with certified ultrasound expertise in lung ultrasonography (TP). Following the theoretical training, students attended the San Matteo University Hospital, where they performed ten supervised LUS examinations on patients hospitalized in the Internal Medicine ward. Among all patients, we selected those who referred respiratory symptoms at admission, both as main or secondary complaint. The first 80 eligible patients accepting to take part to the study were enrolled as a convenience sample. Tutoring during the practical training was provided by the two same residents involved in theoretical teaching (MM, SDP). Once completed their training (i.e., after completing ten sonographic examinations of the lung), students were tested for their competencies: they were asked to perform an LUS examination on a patient hospitalized for respiratory symptoms. For this purpose, an additional sample of eight patients (one for each student) was enrolled. The same eight patients were also scanned by a more expert operator, an EM resident (SDP), for a comparison with student’s results. Performances of students and senior resident were assessed by a consultant in Internal Medicine with years of experience in LUS (FF), who adopted the LUS-OSAUS scale and measured the time needed to complete each scan. The examiner was only responsible of students’ evaluation and had no role in their training. All participants involved in the study, including patients, students, and physicians, provided written informed consent.

Ultrasound technique

For this study, we used a sonographer MyLab™Seven-Esaote provided with a convex transducer (2–5 MHz). All the operators (students and resident) were allowed to use only the B-mode setting; therefore, M-mode evaluation of the pleural sliding was not requested. When scanning a patient, operators were taught to divide each hemithorax in four quadrants, as described in the current international recommendations for point-of-care lung ultrasound [15]. In case of detection of pleural effusion, students were also instructed to move patients to the sitting position, and to scan it from the posterior costophrenic angle upwards: this allows a more accurate estimation of the effusion and a more reliable assessment of whether a thoracentesis would be safe or not (item 13–14 of scale; Table 1).

Table 1

Lung-ultrasound objective structured assessment of technical skills (LUS-OSAUS) [13]

		1	2	3	4	5
Indication
1	Evaluates the indication for lung ultrasound	None		Some		Sufficient
2	Suggests focused questions that can be examined by lung ultrasound	None		Some		Sufficient
Systematic lung-ultrasound examination
3	Performs lung ultrasound systematically	No systematic approach		Some systematic approach		Sufficient systematic approach
4	Performs lung ultrasounds on the basis of focused questions	No correlation between focused question and scanning		Some correlation between focused questions and scanning		Sufficient correlation between focused questions and scanning
Technical skills
5	Correct placement of patient (e.g., supine when scanning for pneumothorax)	Wrong placement (e.g., evaluating pleural effusion with patient in Trendelenburg)				Optimal placement (e.g., evaluating pleural effusion with patient seated)
6	Correct choice of transducer	Wrong choice of the transducer				Optimal choice of the transducer
7	Correct depth	Wrong depth setting				Optimal depth setting
8	Correct gain	Wrong setting of gain				Optimal setting of gain
9	Correct handling of transducer	Poor transducer handling				Optimal transducer handling
Findings
10	Correct assessment of pleura	Not able to assess correctly		Properly assessed sometimes		Properly assessed every time
11	Correct assessment of B lines	Not able to assess correctly		Properly assessed sometimes		Properly assessed every time
12	Correct assessment of consolidations	Not able to assess correctly		Properly assessed sometimes		Properly assessed every time
13	Correct assessment of pleural effusion	Not able to assess correctly		Not able to assess correctly		Not able to assess correctly
14	Correct assessment of whether US-guided thoracentesis is safe	Not able to assess correctly		Properly assessed sometimes		Properly assessed every time
Documentation
15	Documents findings in patient’s chart	None		Main findings are described		Described sufficiently
Conclusion
16	Able to make a diagnosis on the basis of lung-ultrasound findings	No diagnosis		Some diagnosis		Correct diagnosis
17	Able to integrate lung-ultrasound findings with patient’s history	No integration		Some integration		Optimal integration

For each of the 17 items, operators receive a 1–5 score

At the beginning of the examination for the assessment of competencies, operators were provided with a written note containing a brief description of patient’s history and complaints (e.g., “72-year-old female; history of heart failure; orthopnea since last week, swollen legs, and unusually high values of blood pressure”). Based on this short history, the operators were asked to evaluate the indication for LUS and to suggest focused questions (item 1–2 of the scale; Table 1). Moreover, patient’s history would also drive operator’s choice on how to position their patient and from where to start scanning, to provide answers to focused questions (item 4–5 of scale; Table 1).

The time needed to complete the task was measured from the first contact of the probe with patient’s skin until when the operator declared the scan complete. At the end of the scan, operators provided a written report of their sonographic findings on a blank paper sheet, which was evaluated by the examiner considering its completeness, accuracy, and use of correct technical jargon (item 15 of the LUS-OSAUS scale; Table 1).

When scoring operators with the LUS-OSAUS scale, that is a 1–5 point Likert scale, the examiner could also assign fraction of the entire number (e.g., 3.5 or 4.5).

Design

Student’s training started with a 3-h lecture that covered the fundamental theoretical knowledge behind the practice of LUS. In particular, the lecture was designed to teach all the skills described in the LUS-OSAUS, which includes 17 items grouped into six areas of expertise: indication, systematic lung-ultrasound examination, technical skills, findings, documentation, and conclusion (Table 1).

The scientific content of the presentation, which also comprised several images and clips of normal and pathological conditions, was based on the latest and most robust evidences on LUS [4, 15]. The lecture included a hands-on session, during which students scanned each other to familiarize with the ultrasound machine and to gain confidence with the LUS examination. As an adjunct to the didactic material used during the lecture, we provided trainees with an educational video to facilitate their learning process and retention of theoretical knowledge. The video was specifically created for the purpose of this study, and it reviews all topics covered in the classroom and at the bedside (see Supplementary Material). Following theoretical training, students attended for 2 consecutive days the San Matteo University Hospital, where each of them performed ten supervised LUS examinations on patients hospitalized in the Internal Medicine ward. Four weeks after their practical training, they returned to the same ward to undertake the test with the modalities discussed above. Students and the EM resident participating in the assessment of competencies could not communicate or help each other while scanning patients. The examiner was allowed to provide explanations and clarifications to trainees only once they had completed the assessment of their assigned patient.

We analyzed the obtained data to determine whether the LUS-OSAUS could help to differentiate the level of competence of our group of LUS-trained undergraduates with that of a more expert operator. In particular, we compared the scores obtained by operators in all areas of competence of the LUS-OSAUS, the total scores, and the time needed to complete the sonographic task.

Statistical analysis

In the absence of the previous practical applications of the LUS-OSAUS, a sample size computation was not performed, and this is a pilot study. Quantitative data are expressed in median and interquartile range, due to the limited size of the sample. The difference between students and resident performance was analyzed by a Wilcoxon rank sum test for paired data. All the analyses were performed by STATA 14 for Macintosh. The significance threshold was set at p < 0.05.

Results

Overall, median students’ score in the single items of the scale was significantly lower than the ones obtained by the senior resident (students: 4.0 [3.3–5.0] vs. senior resident: 5.0 [5.0–5.0]; p < 0.0001). This result was similar when considering the six areas of expertise separately (Table 2; Fig. 1). When analyzing each single item of the scale, the students’ scores were significantly lower than senior resident’s ones, except for B-line identification, choice of the correct transducer, and focused suggested questions (Table 2).

Table 2

Median scores obtained by the operators for each single item of the scale and in the six areas of expertise

	Students’ score	Senior resident’s score	p
Overall	4.0 [3.3–5.0]	5.0 [5.0–5.0]	< 0.0001
Areas of expertise
Indication	4.5 [4.0–5.0]	5.0 [5.0–5.0]	0.0052
Systematic LUS examination	4.0 [4.0–5.0]	5.0 [5.0–5.0]	0.0013
Technical skills	4.0 [4.0–5.0]	5.0 [5.0–5.0]	< 0.0001
Findings	4.0 [3.0–4.0]	5.0 [5.0–5.0]	< 0.0001
Documentation	4.0 [3.3–5.0]	5.0 [5.0–5.0]	0.0293
Conclusion	4.0 [3.5–5.0]	5.0 [5.0–5.0]	0.0033
Items
Evaluates indications for LUS	4.0 [4.0–5.0]	5.0 [5.0–5.0]	0.0253
Suggests focused questions	5.0 [3.0–5.0]	5.0 [5.0–5.0]	0.0833
Performs systematic LUS	4.0 [4.0–5.0]	5.0 [5.0–5.0]	0.0253
Performs LUS on focused questions	4.0 [3.0–4.8]	5.0 [5.0–5.0]	0.0186
Correct patient placement	4.0 [4.0–5.0]	5.0 [5.0–5.0]	0.0274
Correct of choice transducer	5.0 [4.3–5.0]	5.0 [5.0–5.0]	0.0854
Correct depth	4.0 [4.0–4.5]	5.0 [5.0–5.0]	0.0164
Correct gain	4.0 [4.0–4.0]	5.0 [5.0–5.0]	0.0102
Correct handling	3.5 [3.0–4.0]	5.0 [4.5–5.0]	0.0083
Correct assessment of pleura	4.0 [3.3.4.0]	5.0 [5.0–5.0]	0.0118
Correct assessment of B lines	4.0 [3.0–4.0]	5.0 [4.3–5.0]	0.0556
Correct assessment of consolidations	3.5 [3.0–4.3]	5.0 [4.5–5.0]	0.0416
Correct assessment of pleural effusion	3.0 [3.0–4.0]	5.0 [4.8–5.0]	0.0137
Thoracentesis safe	4.0 [3.5–4.5]	5.0 [5.0–5.0]	0.0178
Documents findings in patients’ charts	4.0 [3.3–5.0]	5.0 [5.0–5.0]	0.0293
Able to make diagnosis based on LUS	4.5 [3.3–5.0]	5.0 [5.0–5.0]	0.0487
Able to integrate LUS	4.0 [3.8–5.0]	5.0 [5.0–5.0]	0.0287

Differences between resident’s and students’ results were always statistically significant except for the items “suggests focused questions”, “correct choice of the transducer”, and “correct assessment of B lines”

Fig. 1

Box plot illustrates students’ and resident’s median scores obtained in the six areas of expertise

When considering cumulative scores, median total score for students was 70.5 [61.0–74.8] vs. 84.0 [83.5–84.3] for senior resident (p = 0.0116) (Fig. 2). This difference remained when considering the six areas of expertise separately (Fig. 3; Table 3).

Fig. 2

Plot shows the differences of median total scores among the operators

Fig. 3

Plot illustrates operators’ median cumulative scores in the six areas of expertise

Table 3

Median cumulative scores obtained by students and the senior resident

Cumulative scores	Students’ score	Resident’s score	p
Overall	70.5 [61.0–74.8]	84.0 [83.5–84.3]	0.0116
Category
Indication	9.0 [7.5–9.5]	10.0 [10.0–10.0]	0.0186
Systematic LUS examination	9.0 [7.0–9.0]	10.0 [10.0–10.0]	0.0103
Technical skills	21.0 [18.3–22.5]	25.0 [24.5–25.0]	0.0115
Findings	18.5 [17.0–20.5]	24.3 [23.5–25.0]	0.0139
Documentation	4.0 [3.3–5.0]	5.0 [5.0–5.0]	0.0293
Conclusion	8.5 [7.0–10.0]	10.0 [10.0–10.0]	0.0293

The difference between operators’ cumulative scores was always statistically significant

Median time required to perform a complete examination was significantly higher for students (student 14.1 [12.8–16.1] min vs. senior resident 4.7 [3.9–5.2], p = 0.0117) (Fig. 4).

Fig. 4

Box plot illustrates the median time that operators required to complete the LUS examination

Discussion

The performances of our EM resident, both in terms of scores and in terms of time needed to complete the scan, correspond to those of the expert group in the LUS-OSAUS validation study [13]. Hence, he acted as an expert operator although not being a consultant. As expected, student’s median times to complete the scan (14.1 [12.8–16.1]) min were in line with those of the intermediate group of the study (mean 16.4 min [95% CI, 8.9–23.9; p = 0.005]) [13], whereas their scores were slightly higher compared to that group. In this regard, it would be hazardous and possibly wrong to conclude that our students were more expert than the intermediate group of the validation study. The operators of the intermediate group were residents of various medical specialties in different years of training; however, the author does not specify what kind of education they had received in lung ultrasound. It is likely that this group had a quite heterogeneous background of LUS competencies and possibly some residents lacking a formal and structured training in lung ultrasound may have scored relatively low when they were assessed with the LUS-OSAUS. On the contrary, our student’s group was very homogeneous in terms of preliminary competencies (same year of training; same background in ultrasound), and most importantly, they received an LUS training that was “LUS-OSAUS-oriented”, i.e., they were specifically trained both in the classroom and at the bedside to master each item of the scale. This can help to explain why they scored higher than the intermediate group of the validation study, which was the level that we expected they would reach at the end of our training.

In our study, students’ median scores were significantly inferior to resident’s scores in all items of the LUS-OSAUS, except for B-line identification, choice of the correct transducer, and focused suggested questions. These results suggest that for these specific tasks a limited training can be sufficient to reach the level of competency of an expert operator, whereas other skills do require more training to be mastered.

The performances of our students also suggest the efficacy of our didactic, particularly if we consider that the practical examination took place 4 weeks after completion of the practical training and a few months after the didactic lecture. During this period of time, the students asked us for some study material to strengthen and maintain the theoretical knowledge acquired in the class. For this reason, we developed an educational video that reviews all the fundamental aspects of lung ultrasound, which could be consulted anytime by our students. Based on our teaching experience, a “tips and tricks” section has also been included in the video, to highlight what the most common technical mistakes of novices are and to suggest how to avoid them. Although we did not specifically measure the efficacy of the video, this tool has been very much appreciated by our trainees, and we believe that it can be beneficial to all practitioners who are approaching to lung ultrasound and to their educators. Therefore, we decided to make it available as supplementary material.

For what concerns the utility of the LUS-OSAUS, we found particularly practical the fact that several items also explore the theoretical knowledge of trainees: this allowed us to assess the theoretical competencies of our students during their practical examination, without the need to administer a separate questionnaire, which we deemed unnecessary. We believe that the adoption of this tool in future studies will probably make less and less common the use of ad-hoc multiple-choice questionnaires, often administered as before-after test [12]. Nevertheless, we also noted some weak points of the LUS-OSAUS. For instance, the item “correct assessment of pleura” does not specify exactly what skills the trainee should master; in our case, this was limited to B-mode visual evaluation of sliding/bat sign, lung pulse, and appearance of the pleura (regular vs irregular; thin vs thickened). However, other potential LUS trainees, such as ICU residents, would probably be requested to provide other information when assessing the pleura, such as a detailed description of aerated vs non-aerated areas of the lungs in a mechanically ventilated patient. The integration of LUS with clinical findings and conclusions will also differ according to trainees: for instance, the hypothetical ICU resident describing non-aerated lung bases in his mechanically ventilated patient will integrate this finding with the ventilator setting, for example concluding that the PEEP values should be modified and LUS repeated afterwards.

All these considerations are related to the fact that lung ultrasound, as any other application of ultrasonography, can be performed across different medical specialties and at different level of expertise. Therefore, we suggest using the LUS-OSAUS tool together with a supplementary table, which should describe in detail what skills trainees are supposed to demonstrate for each item. This set of skills should be defined a priori and should be individualized according to the specific needs of trainees. For the item “documentation”, it should be made clear whether the trainee is requested to use an open scheme or one of the several pre-defined schemes available, such as the one proposed by ACEP [16].

Concerning LUS training for undergraduates, in this experience, we intentionally restricted the technical abilities to be acquired by our students while focusing our didactic on sonographic semeiotics, integration of sonographic and clinical findings, and clinical reasoning. For example, to standardize and simplify trainee’s learning process, we restricted the use of probes to the convex one. However, students knew from their theoretical training all the advantages and disadvantages of linear and microconvex probes, which they were asked to discuss during their examination (evaluation of item 6 of LUS-OSAUS scale; see Table 1). In our opinion, an appropriate LUS training for students should include the minimum number of skills that can actually make a difference in their clinical performances at the bedside. Educators should consider that medical curricula are already very demanding, and a too much technical LUS course carries the potential risk of generating confusion rather than supporting clinical reasoning.

Limitations

Our study is certainly limited by the small number of participants. Moreover, the number of patients that each student scanned during the practical training was also small: therefore, some of our students were not exposed to some relevant clinical scenarios (e.g., pneumothorax). Finally, the student testing was done on a single patient and by a single evaluator and is, therefore, prone to bias.

Conclusions

In our experience, the LUS-OSAUS scale allowed for a standardized and comprehensive assessment of student’s competencies in lung ultrasound, and helped to discriminate their level of expertise from that of a more experienced operator. Some items of the scale specifically test the theoretical knowledge of trainees, thus making redundant the use of questionnaires designed for this purpose. To individualize the assessment of competencies of LUS-trained operators, we suggest to combine this tool with a scheme that defines a priori the skills that trainees are requested to demonstrate for each single item.

Below is the link to the electronic supplementary material.

Supplementary file1 (MPG 668464 kb)