Effects of 1 year of training on the performance of ultrasonographic image interpretation: A preliminary evaluation using images of Sjögren syndrome patients.

PURPOSE: This study investigated the effects of 1 year of training on imaging diagnosis, using static ultrasonography (US) salivary gland images of Sjögren syndrome patients.
MATERIALS AND METHODS: This study involved 3 inexperienced radiologists with different levels of experience, who received training 1 or 2 days a week under the supervision of experienced radiologists. The training program included collecting patient histories and performing physical and imaging examinations for various maxillofacial diseases. The 3 radiologists (observers A, B, and C) evaluated 400 static US images of salivary glands twice at a 1-year interval. To compare their performance, 2 experienced radiologists evaluated the same images. Diagnostic performance was compared between the 2 evaluations using the area under the receiver operating characteristic curve (AUC).
RESULTS: Observer A, who was participating in the training program for the second year, exhibited no significant difference in AUC between the first and second evaluations, with results consistently comparable to those of experienced radiologists. After 1 year of training, observer B showed significantly higher AUCs than before training. The diagnostic performance of observer B reached the level of experienced radiologists for parotid gland assessment, but differed for submandibular gland assessment. For observer C, who did not complete the training, there was no significant difference in the AUC between the first and second evaluations, both of which showed significant differences from those of the experienced radiologists.
CONCLUSION: These preliminary results suggest that the training program effectively helped inexperienced radiologists reach the level of experienced radiologists for US examinations.

Introduction

Ultrasonography (US) has a broad spectrum of indications in diagnosing different conditions, because it is a low-cost modality that does not involve radiation exposure. In the field of oral and maxillofacial radiology, the indications for US include the diagnosis of soft tissue lesions, such as cervical lymph node metastases from malignant tumors.1 However, US image interpretation is sometimes difficult, and the diagnostic accuracy usually depends on the examiner's experience. Previous studies of US have reported that the diagnostic accuracy of inexperienced radiologists is lower than that of experienced radiologists in the fields of breast, thyroid, and salivary gland imaging.23456 In a recent study, inexperienced radiologists were reported to show a lower diagnostic performance for classifying US images of Sjögren syndrome (SjS) than a deep learning artificial intelligence system.7

In the actual use of US devices in clinics, diagnoses should ideally be determined by the US examiners, because the examiners can create appropriate sectional images for diagnosis by setting appropriate US beam directions. In some cases, however, this is impractical and, in such instances, the examiner saves a typical static image for other interpreters. Without knowing the exact position of the probe in relation to the anatomical structures on a static US image, the assessment is difficult for an interpreter who did not actually examine the patient. This is especially the case for inexperienced interpreters. Therefore, an effective training or diagnostic support system, using computer-aided diagnosis (CAD) techniques, may be beneficial. However, 2 previous reports on US diagnosis have emphasized that although CAD systems can support inexperienced radiologists, their performance quality did not reach the level of experienced radiologists.34 Therefore, CAD support alone may be insufficient, indicating a need for training in the interpretation of US images.

Several studies have aimed to create technical improvements in US training, including the acquisition of high-quality images or the detection of target lesions using various training methods, such as lectures, hands-on sessions, and apprenticeships with an experienced radiologist.8910111213 However, no studies have addressed the problem of how to improve the evaluation of static US images, especially when the examiner and the interpreter are not the same person.

The authors' future goal is to determine the duration and type of training that an inexperienced radiologist needs to become an expert in US diagnosis. To make progress towards this goal, this preliminary study investigated the effects of a 1-year training program that was routinely performed at the authors' department for imaging diagnosis, using static US images of salivary glands in SjS patients.

Materials and Methods

The design of this preliminary study was approved by the Ethics Committee of Kyushu University, Japan, and written informed consent was obtained from all the patients (IRB serial number: 29-006).

Subjects

This study involved 3 inexperienced radiologists (observers A, B and C) who agreed to participate in this study and had different levels of experience in imaging diagnosis. These 3 inexperienced radiologists evaluated a total of 400 static US images of the parotid gland (PG) and the submandibular gland (SMG) twice at a 1-year interval, in April 2019 (the first evaluation) and April 2020 (the second evaluation). These US images were acquired at a different institution from the study location. The acquisition procedure is described below in detail. At the authors' department, where the study was carried out, novices receive 2 years of training in diagnostic imaging, including US examinations. Observer A had already completed the first-year training and had just started the second year when the first evaluation was performed. Observers B and C were beginning their first-year training at the time of the first evaluation.

To compare the observers' performance, 2 experienced radiologists with more than 10 years of experience evaluated the same images. In April 2019, the 3 inexperienced radiologists each independently evaluated 100 static US images of the PG obtained from patients with verified SjS and those without SjS (non-SjS). They also evaluated a total of 200 US images of the SMG in SjS patients (100 images) and non-SjS patients (100 images). All personal information and examination dates included in the obtained images were removed, and the images were randomly displayed in PNG format using Microsoft PowerPoint (Microsoft Corp., Redmond, WA, USA). Typical images are shown in Figures 1A–F. Figures 1A and 1D show US images of the PG and SMG of a non-SjS patient. The parenchyma was homogeneous and well-defined. Figures 1B, 1C, 1E, and 1F are US images of the PG and SMG of a patient with SjS. Figures 1B and 1E are typical examples characterized by multiple hypoechoic areas, and Figures 1C and 1F are typical examples characterized by multiple hyperechoic lines. The evaluations were classified with a 4-point scale of SjS probability: 1, definitely SjS; 2, probably SjS; 3, probably non-SjS; and 4, definitely non-SjS. After finishing the 1-year training, the same 3 radiologists evaluated the same images using the same method in April 2020. Their diagnostic performance at the first and second evaluations was compared in terms of the area under the receiver operating characteristic curve (AUC).

Fig. 1

Examples of ultrasonographic (US) images. A. Parotid gland (PG) of a patient without Sjögren syndrome (SjS). The parenchyma is characterized as homogeneous and well-defined. B and C. PG of a patient with SjS. B. The US image shows typical examples characterized by multiple hypoechoic areas. The glandular parenchyma is heterogeneous with multiple hypoechoic areas found within the parenchyma (white arrows). C. The US image shows typical examples characterized by multiple hyperechoic lines. The glandular parenchyma is heterogeneous and multiple hyperechoic lines are found within the parenchyma (white arrows). D. Submandibular gland (SMG) of a patient without SjS. The parenchyma is characterized as homogeneous and well-defined. E and F. SMG of a patient with SjS. E. The US image shows typical examples characterized by multiple hypoechoic areas. The glandular parenchyma is heterogeneous with multiple hypoechoic areas found within the parenchyma (white arrows). F. The US image shows typical examples characterized by multiple hyperechoic lines. The glandular parenchyma was heterogeneous and multiple hyperechoic lines are found within the parenchyma (white arrows).

Acquisition of US images

All patients complained of dry mouth and received SjS examinations between 2011 and 2018. Clinical examinations to diagnose SjS were performed in accordance with both Japanese criteria14 and American-European Consensus Group criteria15, and US examinations of PG and SMG were also performed. US examinations were performed using a diagnostic unit (Logiq 7: GE Healthcare, Tokyo, Japan) with a center frequency of 12 MHz. The PG was scanned with the coronal plane parallel to the posterior border of the ramus of the mandible. The SMG was scanned with the sagittal plane parallel to the inferior border of the mandible.

An experienced radiologist (MS) with more than 20 years of experience in US examinations selected the static US images used for evaluation. The selected images were downloaded from the hospital imaging database in Digital Imaging and Communications in Medicine (DICOM) format. Another radiologist (YK) then converted the images from DICOM format to PNG format. Consequently, 100 SjS images (13 men and 87 women with an average age of 56.6 years) and 100 non-SjS images (21 men and 79 women with an average age of 61.1 years) were available for this study.

Training program

The diagnostic imaging training was conducted 1 or 2 days a week under the instruction of an experienced radiologist at the dental hospital where the study was conducted. Four experienced radiologists with more than 10 years of experience were randomly paired with each inexperienced radiologist. The training program included participation during clinical examinations, collecting patient histories, physical examinations including palpation, and imaging examinations of the patients (Fig. 2). This program included not only US, but also computed tomography (CT), cone-beam CT for dental use (CBCT) and magnetic resonance imaging (MRI) examinations for various maxillofacial diseases, such as tumors, cysts, and inflammation. At the hospital, approximately 10 to 15 patients per day required these examinations, including 8–10 CT or CBCT examinations, 1–2 US examinations, and 1–2 MRI examinations. The inexperienced radiologists performed these examinations, including actual operation of the US machine, with support from an experienced radiologist. Approximately 10 diagnostic reports a week were generated based on discussions with an experienced radiologist, and 2 or 3 US reports were created in a week. Additionally, the participants attended a weekly conference involving all departmental staff, and presented and discussed important cases during the week. A total of 172 US examinations were performed and 97 cases were handled at the weekly conference during the 1-year program.

Fig. 2

Schematic illustration of the training program for the inexperienced radiologists at the dental hospital. CT: computed tomography, CBCT: cone-beam computed tomography, MRI: magnetic resonance imaging, US: ultrasonography.

Statistical analysis

The differences between the AUC values were tested using chi-square analysis in the JMP statistical software package (version 13.0.0; SAS Institute, Cary, NC, USA). The significance level was set to P<0.05.

Results

Observers A and B had completed the 1-year training at the time of the second evaluation. Therefore, the former completed 2 years of training and the latter finished 1 year of training. Observer C interrupted the training to take maternity leave for 8 months between August 2019 and March 2020.

A summary of the AUCs of the observers for PG and SMG is shown in Table 1. For observer A, who completed 2 years of training, no difference was found in the AUC for either PG or SMG between the first and second evaluations, and the values were similar to those of the experienced radiologists (Figs. 3A and 4A).

Table 1

The area under the curve of the first and second evaluations of ultrasound images for inexperienced radiologists (A, B, C) and experienced radiologists (D, E)

The 95% CI is shown in parentheses. *: P<0.05 between the first and second evaluation, †: P<0.05 compared with the experienced observers

Fig. 3

Receiver operating characteristic curves for the parotid gland at the first and second evaluations by the inexperienced radiologists (A. Two years of experience, B. One year of experience, C. Four months of experience) compared with the experienced radiologists.

Fig. 4

Receiver operating characteristic curves for the submandibular gland at the first and second evaluations by the inexperienced radiologists (A. Two years of experience, B. One year of experience, C. Four months of experience) compared with the experienced radiologists.

For observer B, who completed 1 year of training, the AUCs were significantly higher at the second evaluation than at the first evaluation for both PG and SMG (P=0.0463 and P=0.0005, respectively). The AUC reached the level of the experienced radiologists for PG assessment, but still showed a difference for SMG classification (P=0.0249) (Figs. 3B and 4B).

For observer C, who did not complete the training, the AUCs were not significantly different between the first and second evaluations for both PG and SMG, and were significantly different from those of the experienced radiologists (P<0.0001, P<0.0001) (Figs. 3C and 4C). This might indicate that a 4-month training period was insufficient.

Discussion

In this preliminary study, the authors evaluated the effects of a 1-year training program in diagnostic imaging using US images of SjS patients, and found that trainees' diagnostic performance significantly improved after the completion of training. This may have been because inexperienced radiologists paired up with experienced radiologists for examinations, diagnostic imaging, and discussions of cases during conferences. It is considered important that the inexperienced radiologists were placed in an environment where they could efficiently absorb the expertise of experienced radiologists.

Delzell et al.11 reported the effect of a comprehensive musculoskeletal US training program to improve accuracy in the diagnosis of rotator cuff tears in relatively inexperienced examiners. The training program consisted of lectures on US application software and scanning techniques for the musculoskeletal system, hands-on imaging sessions, and apprenticeships with an experienced radiologist, until the candidates gradually started performing examinations independently. Weekly ongoing medical education conferences were held to present and discuss all clinical cases from the previous week. In contrast, the training program did not include lectures or hands-on imaging sessions in advance, although trainees received real-time lectures and advice from experienced radiologists during patient examinations and discussion after the examination. The training program also included attendance at weekly conferences, which probably assisted in learning retention and are recommended for inclusion in the training program.

The training program analyzed in this study included not only US imaging, but also CT and MRI imaging, the collection of patients' medical history, and information about physical examinations. The addition of physical and anatomical information about the target tissue makes it possible to make a comprehensive diagnosis. This broad focus is thought to improve diagnostic performance to a greater extent than US training alone.

Observer A, who had already completed 1 year of training and was in the second year, reached the level of the experienced radiologists for both the first and second evaluations. For Observer B, who was training for the first time, PG assessment reached the level of the experienced radiologists, while SMG assessment reached a level close to that. However, for observer C, who discontinued training, the AUC values did not improve and remained significantly lower than those of the experienced radiologists. Although this study was limited to still images of SjS, its findings show that 1 year of training may be sufficient. The difference between the results for PG and SMG was presumably because the US images of the PG did not contain other structures (such as muscle, lymph nodes, and blood vessels) within the image, whereas the US images of the SMG often contained such structures. Therefore, US interpretation may be more difficult for the SMG than for the PG, possibly implying that inexperienced radiologists need more training in diagnosing diseases in the SMG. After training, the radiologists were able to identify the surrounding anatomical structures, and their diagnostic performance improved significantly.

This study used images of SjS. SjS is an autoimmune disease characterized by dry eyes and dry mouth. Dry mouth results from reduced secretion of saliva caused by deterioration of salivary gland function, and fatty degeneration is a characteristic finding in the parenchyma of affected glands.1617 US images of the PG and SMG in SjS patients are characterized by inhomogeneous parenchyma, multiple hypoechoic areas, and multiple hyperechoic lines.6 Cysts and tumors are usually located partially in the PG and SMG tissue, and their presence can be easily detected using US. However, because SjS causes degeneration of the entire gland, the diagnosis is difficult unless the interpreter has a clear understanding of the appearance of healthy glands and the surrounding anatomy. Therefore, the detection of SjS may be more dependent on experience than the detection of cysts or tumors, which was the reason for choosing SjS images to verify the effects of training.

The present study had some limitations. First, the number of radiologists was small and the study included radiologists with different durations of training experience at the first evaluation. One had 1 year of experience with US, and 2 were inexperienced. The significant effect of the training was confirmed by comparing the diagnostic performance of these 3 inexperienced radiologists. However, a further analysis will be needed with a larger number of radiologists. Second, the effect of training focused on the accuracy of image interpretation, and did not include the technique of image acquisition. In some cases, the examiner does not perform image diagnosis, and another doctor makes a diagnosis based on the image stored on the server. Since remote imagingbased diagnoses have become widespread in recent years, such situations occur increasingly frequently. Most of these images are static images rather than dynamic images due to the large volume of data involved and the storage capacity of the hard disk. It is difficult to interpret static images that are not taken by oneself, but this is a necessary ability for radiologists. Third, the only images evaluated by the radiologists were of salivary glands in patients with or without SjS. Future studies should evaluate other lesions and diseases. Finally, the training in this study is unique to the institution where the study was conducted, and the number of patients and the type of lesions depended on the patients presenting at that time, so bias may have occurred among the evaluators. For future research, multi-institutional training programs should be established and implemented in a unified manner.

In conclusion, these preliminary results indicate that the training program described herein could be effective in helping inexperienced radiologists to attain the level of experienced radiologists for US examinations.