Detection of Rotator Cuff Tears by Ultrasound: How Many Scans Do Novices Need to Be Competent?

BACKGROUND: Ultrasound is commonly used for evaluating rotator cuff tears. However, little training in ultrasound imaging is provided during orthopedic residents' training period. Therefore, we performed this study to determine how many ultrasound scans are required for orthopedic residents to be competent and self-confident in the diagnosis of supraspinatus tendon tears and to investigate whether senior residents outperformed junior residents.
METHODS: We studied two third-year residents who had no previous experience of shoulder ultrasound and evaluated their ability to detect rotator cuff pathologies. Their learning curves were plotted using a cumulative summation analysis with a 20% acceptable failure rate compared to arthroscopic findings. Downward, upward, and horizontal cumulative summation trends indicated incompetence, exceptional competence, and competence, respectively. The diagnostic accuracy of third-year residents was compared with that of second-year residents and the number of cases required to gain self-confidence was evaluated.
RESULTS: Cumulative summation analysis showed that after 26-28 scans, residents achieved the competence to correctly diagnose supraspinatus tears: an upward trend was observed from the beginning for full-thickness tears and a downward trend was observed for partial-thickness tears. Sensitivity and specificity were 0.95 and 0.79, respectively, for third-year residents and 0.91 and 0.58, respectively, for second-year residents. Residents reported self-confidence after 30 ultrasound scans for the detection of rotator cuff tears.
CONCLUSIONS: The number of scans that novices needed to be competent for detecting rotator cuff tears was approximately 30 cases, and the diagnostic accuracy of third-year residents was significantly higher than that of second-year residents.

Ultrasound (US) is commonly used for evaluating shoulder injuries and has been utilized as a reliable and accurate imaging modality for identifying rotator cuff tears.1234) When US is performed by experienced radiologists, US and magnetic resonance imaging (MRI) results are comparable regarding the identification and quantification of rotator cuff pathologies.12567) Furthermore, advances in US technology have led to the more widespread use of this imaging method, even among general physicians. More recently, US technology has produced more affordable, portable, and higher-resolution scans. Although more general physicians now utilize this modality to evaluate pathologies and administer injections, these physicians lack specific training in US imaging. It has resulted in some debate surrounding the training and level of competence in general physicians’ use of US.

The American Institute of Ultrasound in Medicine and the European Federation of Societies for Ultrasound in Medicine and Biology announced training guidelines for the musculoskeletal US for general physicians.89) These guidelines state that a minimum of 150 to 300 scans must be read under the supervision of a radiologist to obtain proficiency. However, a few studies have reported that orthopedic surgeons can evaluate rotator cuff tears using US approximately as well as musculoskeletal radiologists.101112) Unfortunately, more knowledge is needed to determine the required amount of background knowledge to achieve US proficiency. No previous study has evaluated the learning curve for the interpretation of US images; such knowledge is particularly required by orthopedic surgeons and orthopedic residents because the specialists’ background knowledge may significantly shorten the training period required to reach competency for the diagnosis of rotator cuff pathologies using US imaging. Furthermore, orthopedic surgeons efficiently learn US because they can combine image, history taking, physical examination, and arthroscopic findings. However, little training in US imaging is provided during orthopedic residents’ four-year training period.

Therefore, we performed this study to determine how many US scans are required for orthopedic residents to be competent and self-confident in the diagnosis of supraspinatus tendon tears and to investigate whether senior residents outperformed junior residents. We hypothesized that senior residents would need a smaller number of US scans to be competent and self-confident than junior residents in the diagnosis of supraspinatus tendon tears.

METHODS

All procedures performed in studies involving human participants were in accordance with the ethical standards of the Institutional and National Research Committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. The study was approved by the Institutional Review Board of Seoul Medical Center (IRB No. SMC 2013-017). Informed consent was obtained from all patients and residents before the start of this study.

Patients

From July 2013 to February 2014, 53 consecutive patients underwent arthroscopic subacromial decompression with or without rotator cuff repair. Those patients with a history of prior fracture around the affected shoulder, revision surgery, accompanying calcific tendinitis, or infection were excluded from this study. After the additional exclusion of 3 patients with incomplete data and 3 who refused to participate, 47 patients were analyzed. The included patients were 18 men and 29 women with a mean age of 59 years (range, 41–79 years).

Methods for US Examination

Two third-year and 2 second-year residents without previous experience in shoulder US participated in this study. All residents completed a hands-on workshop provided by the Korean Orthopaedic Ultrasound Society. Additionally, the in-hospital training of US scans was performed. In March 2013, all trainees were taught shoulder anatomy through a shoulder anatomy seminar. Subsequently, they attended US lectures conducted by a shoulder surgeon (KHK). Supervised scanning of 5 normal shoulders was also provided by the shoulder surgeon (KHK). Each resident’s ability to detect rotator cuff pathologies was evaluated. A high standard of patient care was ensured by the principal investigator (KHK), who had more than 10 years of experience in shoulder US, examining all participants at an outpatient clinic. The residents performed US scans (HD 11 XE; Philips, Korea) one day before arthroscopic surgery without any prior knowledge of the patient condition and without performing any physical examination.

US Assessment

We reviewed the literature on US for rotator cuff tears and developed a case report form to standardize US examination results and reports of pathologies in the supraspinatus tendon.13) The following items were recorded in this report after each scan: (1) discontinuity, (2) focal thinning or loss of convexity in the bursal surface, (3) cartilage interface sign, (4) cortical irregularity, and (5) intra-articular or bursal fluid collection. A final diagnosis of the pathology of the supraspinatus tendon was recorded as a normal, partial tear, or full-thickness tear at the end of the examination. Additionally, the residents’ level of self-confidence regarding diagnostic accuracy was documented.

The diagnostic accuracy of US imaging with the arthroscopic diagnosis used as the reference standard and each scan’s time-to-completion were calculated for the 2 third-year residents. Learning curves were plotted using cumulative summation (CUSUM) analysis. The second part of this study involved evaluating the differences between residents in different years of training. Senior residents have more background knowledge about the anatomy of the shoulder and rotator cuff tears, so we hypothesized that senior residents would outperform junior residents at diagnosing rotator cuff pathologies. The data for 2 second-year residents were available for 40 US scans, and the number of cases required to achieve self-confidence was evaluated for these 2 second-year residents, and diagnostic accuracy was compared between the second- and third-year residents.

Statistical Analysis

Learning curves were plotted using CUSUM analysis1415) with a 20% acceptable failure rate compared to the arthroscopic findings. We set 20% as the failure rate by referring to a similar study on spinal US.14) If a resident successfully performed US, 0.2 points were awarded; if a resident failed to perform US successfully, (1–0.2) = 0.8 points were deducted. The CUSUM curve was plotted with the CUSUM value representing continuously accumulated scores on the Y-axis and the number of consecutive attempts on the X-axis. Downward, upward, and horizontal CUSUM trends indicated a lack of competence, exceptional competence, and competence, respectively. Continuous variables were compared by Student t-test, and categorical data were compared by chi-square test and Fisher’s exact test. Statistical analyses were performed with IBM SPSS ver. 22.0 (IBM Corp., Armonk, NY, USA) with a significance level of 0.05 for all analyses.

RESULTS

The mean US scan time-to-completion was 14 minutes (range, 9–23 minutes; standard deviation, 3 minutes). The diagnostic accuracy of supraspinatus tears is shown in Tables 1 and 2 for the second- and third-year residents, respectively. Diagnostic parameters (sensitivity, specificity, and positive and negative predictive values) for supraspinatus tear are summarized in Table 3. The third-year residents’ diagnostic parameters were higher than those of the second-year residents. A CUSUM plot revealed that 26–28 scans were required to achieve competence in detecting supraspinatus tears (Fig. 1). When we divided the supraspinatus tears into partial tears and full-thickness tears, the CUSUM graph showed an upward trend from the beginning of the examinations in the diagnosis of full-thickness tears and a downward trend in the diagnosis of partial-thickness tears (Fig. 2).

Table 1

Diagnostic Accuracy of Examiners 1 and 2 (2 Third-Year Residents)

US finding		Arthroscopic finding			Total
		Normal	Partial tear	Full tear
Examiner 1
	Normal	5	2	0	7
	Partial tear	2	7	1	10
	Full tear	0	2	28	30
	Total	7	11	29	47
Examiner 2
	Normal	6	2	0	8
	Partial tear	1	8	3	12
	Full tear	0	1	26	27
	Total	7	11	29	47

US: ultrasound.

Table 2

Diagnostic Accuracy of Examiners 3 and 4 (2 Second-Year Residents)

US finding		Arthroscopic finding			Total
		Normal	Partial tear	Full tear
Examiner 3
	Normal	4	4	0	8
	Partial tear	2	3	3	8
	Full tear	0	2	22	24
	Total	6	9	25	40
Examiner 4
	Normal	3	2	0	5
	Partial tear	3	3	5	11
	Full tear	0	4	20	24
	Total	6	9	25	40

US: ultrasound.

Table 3

Diagnostic Parameters of Both Third-Year and Second-Year Residents

Variable	Sensitivity	Specificity	Positive predictive value	Negative predictive value
Third-year resident	0.95	0.79	0.96	0.73
Second-year resident	0.91	0.58	0.93	0.54

Fig. 1

Cumulative summation graph showing that 26–28 scans are needed to achieve competency in detecting supraspinatus tears.

Fig. 2

(A) Cumulative summation graph of full-thickness tears showing an upward trend from the beginning of examinations. (B) Cumulative summation graph of partial-thickness tears, revealing a downward trend until the last examination.

Following the completion of all 47 consecutive US scans, the 2 third-year residents reported that it took approximately 30 cases to gain self-confidence in detecting rotator cuff tears. Although Fig. 3 shows that roughly 20 cases were sufficient to have self-confidence, those 2 third-year residents answered that it took 30 cases, probably because of the subjective nature of self-report. The number 30 was smaller than the self-reported number for one of the second-year residents, who was uncertain about his abilities until he had read 30 consecutive scans; the other second-year resident reported self-confidence after reading 30 scans. The self-confidence level was documented at the end of each examination, and the resultant graphs show that third-year residents were confident about their diagnosis after approximately 20 scans, whereas second-year residents reported self-confidence after approximately 30 scans (Fig. 3). The overall diagnostic accuracy of second-year residents was lower than that of third-year residents.

Fig. 3

(A) Two second-year residents exhibited self-confidence after about 30 scans. (B) Two third-year residents showed self-confidence after a smaller number of scans. Upper dots indicate an ultrasound scan in which the examiner exhibited self-confidence about the ultrasound diagnosis for rotator cuff pathology; lower dots indicate a lack of confidence.

DISCUSSION

Our study reveals that the learning curve for orthopedic residents who are novices in US imaging is very steep: only about 30 scans were necessary for the second- and third-year residents to achieve competence. In particular, orthopedic residents were remarkably competent in the detection of full-thickness supraspinatus tears from the beginning. These results differ from the guidelines published by the American Institute of Ultrasound in Medicine and the European Federation of Societies for Ultrasound in Medicine and Biology, which state that 150–300 scans are necessary for general physicians who are not radiologists.89)

We also reported a difference in diagnostic accuracy and self-confidence between second- and third-year residents. Orthopedic residents in this study who had more background knowledge and were familiar with surgical anatomy seemed to have comparable or shorter learning curves than those previously reported for general physicians for US imaging. Similarly, the third-year residents with greater knowledge and experience in orthopedic disease showed better diagnostic accuracy and earlier self-confidence in their diagnoses than the second-year residents.

Improvements in US equipment technology have led to the more widespread use of US in almost every medical and surgical department, including general surgery, gynecology, anesthesiology, urology, and internal medicine. Although US has traditionally been considered an examination that must be performed by a radiologist or orthopedic surgeon, general physicians have started using this imaging method, particularly in cases of the musculoskeletal disease. Studies on the amount of experience required to gain US competency are rare.16171819) Two studies reported results from the training of inexperienced non-orthopedic trainees.2021) One recent study by Alavekios et al.22) evaluated US imaging competency by two inexperienced orthopedic surgeons and reported that the majority of learning to detect full-thickness supraspinatus tears occurs within the first 50 scans; this study also reported that peaks in sensitivity and specificity were achieved at around 30 scans. Similarly, we report that 2 third-year residents required approximately 30 scans to obtain competency; these results imply that orthopedic surgeons (even third-year trainees) who have more background knowledge and are familiar with surgical anatomy have shortened learning curves compared to non-orthopedic specialists. Thus, US imaging training would be a valuable addition to the training period for orthopedic surgeons.

In contrast, there appears to be a longer learning curve for novices to reach competency at detecting partial-thickness rotator cuff tears using US. Previous studies have reported that US shows comparable diagnostic accuracy to MRI for both partial- and full-thickness rotator cuff tears.23) In this study, inexperienced orthopedic residents performing US examinations showed a downward trend in CUSUM analysis although there were few scanned partial-thickness rotator cuff tears. Residents who were involved in this study showed a trend of mistaking partial-thickness rotator cuff tears for normal or full-thickness rotator cuff tears, which is similar to the results of a study by Murphy et al.24) The mistaken identification of a partial-thickness tear as either normal or as a full-thickness tear led to decreased diagnostic accuracy in US examination among novice examiners in our study. During the evaluation of shoulders using US, indirect signs such as cartilage interface signs or the cortical irregularity of greater tuberosity provide clues for the detection of rotator cuff tears.25) Particularly in partial-thickness tears, detecting these signs might require more experience and a longer learning curve than the detection of full-thickness tears. Although many cases of full-thickness tears with persistent symptoms necessitate surgical treatment, the determination of surgical indication in a partial-thickness tear depends on the tear’s degree and clinical manifestations, and thus requires more clinical experience. A sufficient number of US scans should be done for the proper detection of a partial-thickness rotator cuff tear using US imaging.

All residents involved in this study had completed a hands-on workshop on US imaging and had in-hospital training with US with supervision and feedback from the principal investigator. Although orthopedic residents have more background knowledge of surgical anatomy and clinical presentation, training sessions focused on US imaging should be included before starting US imaging practice. Furthermore, standardized training in US imaging should be included in orthopedic residents’ training regimen.

One of the limitations of our study is that true blind evaluation was impossible. Because the residents knew that all the shoulders they examined were scheduled for arthroscopic surgery, the residents might have tried to find any pathologic findings in the US scans, which could lead to a high false-positive rate. However, they did not know the detailed clinical presentations and diagnoses and did not have the ability to perform a clinical examination that could have helped them in determining which structure was damaged. These facts are rather the strengths of this study. A second limitation is that the number of normal rotator cuff cases was small due to the study design. We thought it to be most accurate to evaluate the diagnostic accuracy of US through the arthroscopic operation. It was indeed impossible to decide whether to perform the operation according to the sonographic finding by the resident. Therefore, we had to carry out this study with patients who had already decided to undergo surgery, which led to a small number of normal rotator cuff patients. Although there was a small number of cases of the normal rotator cuff, the residents showed a satisfactory level of diagnostic accuracy. A third limitation is that there may be differences in the learning curve between examiners. A fourth limitation is that we could not determine the number of scans required to develop competence in detecting partial tears because of the small number of patients. Additionally, we could not obtain a CUSUM plot for those second-year residents because CUSUM analysis requires consecutive datasets. Indeed, it was difficult to perform four US scans with the same patient. Also, we could not derive intraclass coefficient because of the missing patients in the second-year resident group. Finally, our definition of self-confidence was somewhat arbitrary and not validated.

The number of scans that novices needed to be competent for detecting rotator cuff tears was approximately 30 cases, and the diagnostic accuracy of third-year residents was significantly higher than that of second-year residents.