Inter-rater reliability in the classification of supraspinatus tendon tears using 3D ultrasound - a question of experience?

BACKGROUND: Three-dimensional (3D) ultrasound of the shoulder is characterized by a comparable accuracy to two-dimensional (2D) ultrasound. No studies investigating 2D versus 3D inter-rater reliability in the detection of supraspinatus tendon tears taking into account the level of experience of the raters have been carried out so far.
OBJECTIVES: The aim of this study was to determine the inter-rater reliability in the analysis of 3D ultrasound image sets of the supraspinatus tendon between sonographer with different levels of experience. PATIENTS AND METHODS: Non-interventional, prospective, observational pilot study of 2309 images of 127 adult patients suffering from unilateral shoulder pain. 3D ultrasound image sets were scored by three raters independently. The intra-and interrater reliabilities were calculated.
RESULTS: There was an excellent intra-rater reliability of rater A in the overall classification of supraspinatus tendon tears (2D vs 3D κ = 0.892, pairwise reliability 93.81%, 3D scoring round 1 vs 3D scoring round 2 κ = 0.875, pairwise reliability 92.857%). The inter-rater reliability was only moderate compared to rater B on 3D (κ = 0.497, pairwise reliability 70.95%) and fair compared to rater C (κ = 0.238, pairwise reliability 42.38%).
CONCLUSIONS: The reliability of 3D ultrasound of the supraspinatus tendon depends on the level of experience of the sonographer. Experience in 2D ultrasound does not seem to be sufficient for the analysis of 3D ultrasound imaging sets. Therefore, for a 3D ultrasound analysis new diagnostic criteria have to be established and taught even to experienced 2D sonographers to improve reproducibility.

Background

Two-dimensional (2D) ultrasound (US) is an inexpensive, painless, non-invasive, and fast method of a dynamic examination of the shoulder in real-time. It is characterized by a similar accuracy to the MRI examination to evaluate superficial tendons and soft tissues with a high degree of resolution, including the rotator cuff and muscles(. Furthermore, US is not limited by MRI contraindications, such as the patient's body habitus, claustrophobia, inability to lie flat, or implanted devices not compatible with the magnetized environment of MRI. Yet, the major disadvantage of 2D US, including the sonography of the shoulder, is a high rater dependence(.

Three-dimensional (3D) US with the advantages of multiplanar imaging facility with three perpendicular planes observed in any position, plus direction, has been an established prenatal diagnostic technique since the early 90's(. The use of this technique in musculoskeletal diagnostics has been reported in the examination of the neonatal hip and in the diagnosis of meniscus lesions(. However, the role of 3D US in the diagnosis of shoulder pathologies is not clear with only few studies, and on small populations, investigating the rater dependency in 3D US(.

Objectives

The objective of this study was to determine the inter-rater reliability in the analysis of 3D US image sets of the supraspinatus tendon between sonographers with different levels of experience in musculoskeletal 2D US.

Patients and methods

We performed a non-interventional, prospective, observational pilot study involving 127 patients who were referred for the sonographic examination of a painful shoulder with clinical suspected or considered supraspinatus tendon pathology by the referring doctor. They were recruited in the period between January, 2014 and July, 2014 and were enrolled in a single outpatient clinic after signing an informed consent. The approval of the local ethics review board Basel was obtained. The study was performed in accordance with good clinical practice and carried out in compliance with the Helsinki Declaration.

Dynamic 2D US examination of both shoulders was conducted in accordance with the guidelines set by the Swiss Society of US Medicine (Schweizerische Gesellschaft für Ultraschall in der Medizin, SGUM), musculoskeletal section( which are compatible with the Musculoskeletal US Technical Guidelines from the European Society of Musculoskeletal Radiology (ESSR)( and the guidelines of the European League Against Rheumatism (EULAR).

2D US examination was performed by an advanced sonographer (EULAR Teacher/EFSUMB Level 3 = rater A). GE Logic E9 with ML6-15 probe was used and the scanning time for dynamic examination was 10 minutes.

Diagnostic criteria for 2D US findings of the supraspinatus tendon are listed in Tab. 1.

Tab. 1

Diagnostic criteria for the supraspinatus tendon in 2D US

2D US findings	Diagnostic criteria
Tendinosis	Thick swollen tendon with hypoechoic echotexture. Partial interruption may occur inside the tendon. Irregularity of fibrillar pattern, fragmentation, and focal hypoechoic or hyperechoic areas
Bursal sided partial-thickness tear	Focal hypoechoic or anechoic defect in the tendon, involving the bursal surface
Articular sided partial-thickness tear	Focal hypoechoic or anechoic defect in the tendon, involving the articular surface
Intra-substance sided partial-thickness tear	Focal hypoechoic or anechoic defect inside the tendon
Full thickness tear	Hypoechoic zone extending through the entire substance of the cuff or nonvisualization of the rotator cuff with “approximation” of the deltoid muscle to the surface of the humeral head

A volumetric 3D image set (6–16D probe on GE Logic E9 and 6–16D probe on GE Voluson E6, scanning time 2 minutes) with the visualization of the long biceps tendon at the level of the rotator cuff interval and the footprint of the supraspinatus tendon was acquired during the same examination. The arm was held in a modified Crass position. Afterwards, a set of transverse, longitudinal, axial and rendered volumetric 3D images of the supraspinatus tendon insertion was created automatically.

Based on the assumption that the accuracy of ultrasound is comparable to MRI in the evaluation of the rotator cuff lesions(, we decided not to perform additional MRI examination. We selected 3 raters with different experience level in musculoskeletal ultrasound.

Rater A initially evaluated all 2D images and the 3D image sets. Afterwards, the blinded 3D image sets were scored independently by two other raters (one advanced sonographer EULAR Teacher/EFSUMB Level 2 = rater B and one fellow sonographer EULAR intermediate level = rater C) and with an interval of 3 months again by rater A. The most advanced rater A with 13 years of experience in ultrasound imaging set as “gold standard”. The findings of rater A were then compared to those of rater B and C.

The possible findings with corresponding diagnostic criteria were the same as for 2D US (cf. Tab. 1). Figure 1 shows the examples of the 3D US findings.

Fig. 1

Examples of 3D US images (coronar and 3D rendered views)

To assess the inter-rater reliability between different raters with regard to the findings in the 3D US image sets we used κ statistic. For rater A intra-rater reliability analyses were conducted between the findings in 2D US and 3D image sets. Overall interpretations of the κ statistic were based on the criteria described by Landis and Koch(. The level of reliability was defined as follows: κ values of 0.81 to 0.99 were considered to represent almost a perfect reliability; 0.61 to 0.80 – substantial reliability; 0.41 to 0.60 – moderate reliability; 0.21 to 0.40 – fair reliability; and 0.01 to 0.20 – slight reliability.

Results

We obtained 2309 images (210 3D image sets) of 127 adult patients suffering from unilateral shoulder pain. In our study, the 3D image set acquisition took about 2 minutes, compared to 10 minutes in the 2D US examination.

Baseline characteristics are shown in Tab. 2 and the US findings for the 2D US examinations are shown in Tab. 3.

Tab. 2

Baseline characteristics of the patients enrolled in the study (n = 127)

Characteristic	Patients (n = 127)
Male patients – no. (%)	51 (40)
Total shoulders scanned – no.	210
Left shoulders scanned – no.	103
Right shoulders scanned – no.	107
Mean age – yrs	59.67 (range 18-86 yrs)

Tab. 3

2D US findings (n = 210)

Finding	Shoulders (n = 210)
No pathology	73 (34.8%)
Tendinosis	11 (5.2%)
Bursal sided partial-thickness tear	11 (5.2%)
Articular sided partial-thickness tear	69 (32.9%)
Intrasubstance sided partial-thickness tear	3 (1.4%)
Full thickness tear	44 (20.9%)

There was almost a perfect intra-rater reliability of rater A in the overall classification of supraspinatus tendon tears (2D vs 3D κ = 0.892, pairwise reliability 93.81%, 3D scoring round 1 vs 3D scoring round 2 κ = 0.875, pairwise reliability 92.857%). There was almost a perfect overall reliability in the classification of full thickness tears between 2D and 3D (κ = 0.810), and a substantial reliability in the classification of partial thickness tears (κ = 0.667).

The inter-rater reliability between rater A and B was only moderate (κ = 0.497, pairwise reliability 70.95%) and fair between rater A and C (κ = 0.238, pairwise reliability 42.38%) comparing findings in the 3D US imaging sets.

Discussion

3D US of the shoulder is as accurate as 2D US when compared to MRI for the diagnosis of full- and partial-thickness supraspinatus tendon tears, and the 3D US examination significantly reduced the time between the initial scan and interpretation by a radiologist, which ultimately improved the efficiency of the workplace(. However, the time spent on the interpretation of data obtained from 3D US was not measured in our study, therefore one could assume that the overall time will remain the same, except that the 3D US data are interpreted from a third party.

The inter-rater reliability of 2D US of the shoulder had been investigated in the past with experienced sonographer(. The corresponding κ values were only fair-to-moderate. The conclusion was that further consensus regarding the standardisation of scanning technique and diagnostic criteria is necessary to improve the reproducibility of musculoskeletal ultrasonography even for an experienced sonographer.

Given different levels of experience in 2D US, sonography has a high level of inter-rater reliability for full-thickness rotator cuff tears and a lower inter-rater reliability for partial-thickness and intratendinous rotator cuff tears(.

Studies have even shown that with 3D US, rotator cuff tears were more often correctly diagnosed than with a conventional 2D sonography(. However, 3D US of the shoulder seems to have a poorer inter- and intra-rater reliability compared to 2D sonography as investigated by Hayter et al.( Using MRI for the classification of rotator cuff tears, fellowship-trained, experienced orthopaedic surgeons had a good reliability for predicting full-thickness rotator cuff tears and the number of tendons involved, and a moderate reliability in predicting the involved side of the partial-thickness rotator cuff tear, but a poor reliability in predicting the grade of the partial-thickness tear(.

In our study the reliability for predicting full thickness tears was almost perfect (κ = 0.810) and substantial (κ = 0.667) for predicting the partial-thickness tear. According to Hayter et al.(, the reason behind is the difficulty in the interpretation of small partial-thickness tears at the footprint on 3D images, a common area affected by anisotropy. Therefore, a standardized 3D US image acquisition with minimal areas of anisotropy should be defined. Standardized cross-sectional images (like those made automatically with the 3D probe) in multiple section planes would be necessary with repositioning of the transducer.

On the other hand, the 2D US examination compared to 3D US still has the additional advantage to collect information (e.g. sonopalpation, dynamic examination) in the differentiation of articular sided partial thickness tears versus tendinosis or anisotropy. The 2D US examination is known to be an examiner-dependent method. However, the superior advantage of ultrasound compared to all other imaging modalities (including 3D US) is the possibility to create static images as well as dynamic images, which provides a unique functional information. One weakness of our study is that this dynamic examination option was only given to rater A, who defined the standard sets, but not to rater B and C, who had to deal with sets of static 3D images.

Conclusions

In conclusion, the reliability of 3D US of the supraspinatus tendon depends on the level of experience of the sonographer, according to our data and experience. Experience in 2D US dose not seem to be sufficient for the analysis of the 3D US imaging sets. Therefore, we propose to develop 3D diagnostic criteria for the examination of supraspinatus tendon pathologies prior to further 3D ultrasound studies of this shoulder area.