Interobserver agreement of contrast-enhanced harmonic endoscopic ultrasonography in the evaluation of solid pancreatic lesions.

BACKGROUND AND STUDY AIMS: Previous reports assessing the reproducibility of contrast-enhanced harmonic endoscopic ultrasonography (CH-EUS) in the evaluation of solid pancreatic lesions (SPLs) involved mainly experienced endosonographers. We aimed to assess the interobserver agreement (IOA) of CH-EUS in the evaluation of SPLs by endoscopists with different levels of experience in EUS and CH-EUS. PARTICIPANTS AND METHODS: A cross-sectional observational multicenter study was designed and included 11 endoscopists who were divided into four groups according to their experience in EUS and CH-EUS: group A (long experience in EUS and CH-EUS); group B (short experience in EUS and CH-EUS); group C (long experience in EUS and no experience in CH-EUS); and group D (no experience in EUS or CH-EUS). The observers independently classified the patterns of 60 CH-EUS video sequences of 60 SPLs after a 20-minute training session. For each group, we calculated the IOA (kappa statistic, κ) of CH-EUS and the accuracy of CH-EUS for the diagnosis of pancreatic adenocarcinoma by comparing the pattern of CH-EUS indicative of pancreatic adenocarcinoma (hypo-enhanced contrast pattern) with the final diagnosis.
RESULTS: The overall IOA for CH-EUS was fair (κ = 0.32; 95 %CI 0.22 - 0.41). Group A (κ = 0.63; 95 %CI 0.45 - 0.85) had the highest IOA, followed by group C (κ = 0.54; 95 %CI 0.39 - 0.71), group B (κ = 0.38; 95 %CI 0.22 - 0.55), and group D (κ = 0.21; 95 %CI 0.07 - 0.36). The IOA of groups A and C was significantly higher than that of group D. No significant difference was seen between groups A, B, and C or between groups B and D in terms of IOA. Group A (area under the curve under summary receiver operating characteristic [AUROC] = 0.67; 95 %CI 0.58 - 0.75) had the highest accuracy for the diagnosis of pancreatic adenocarcinoma, followed by group C (AUROC = 0.58; 95 %CI 0.50 - 0.65), group B (AUROC = 0.55; 95 %CI 0.48 - 0.63), and group D (AUROC = 0.51; 95 %CI 0.43 - 0.58). The diagnostic accuracy of group A was not significantly different from that of group C, but it was significantly higher than that of groups B and D. No significant difference was seen between groups B, C, and D in terms of diagnostic accuracy.
CONCLUSIONS: CH-EUS is reproducible in the evaluation of SPLs, even between endoscopists with no or limited experience in EUS and/or CH-EUS. Long experience in EUS is a major contributor to the IOA and diagnostic accuracy of CH-EUS.

Introduction

Endoscopic ultrasonography (EUS) provides high resolution images of the pancreas, and it is considered one of the most accurate methods for the diagnosis and staging of chronic inflammatory, cystic, and neoplastic pancreatic diseases 1 2. The differential diagnosis of solid pancreatic masses, however, remains a challenge 3. EUS can guide fine-needle aspiration (EUS-FNA) for obtaining cytologic samples of pancreatic lesions, thus making a pathologic diagnosis possible 4 5. EUS-FNA, however, may be technically demanding, and multiple punctures of pancreatic lesions may be needed to obtain adequate material for cytologic or microhistologic evaluation. Furthermore, EUS-FNA of the pancreas is associated with a small, but not insignificant, morbidity 6 7. In addition, the sensitivity of cytology for malignancy is limited, and false-negative results are obtained in up to 20 % to 40 % of cases 8 9. In an attempt to overcome these limitations of EUS-FNA, techniques of image enhancement are currently under active technical development. Contrast-enhanced harmonic endoscopic ultrasonography (CH-EUS) is one of the most promising in this context 10.

Contrast-enhanced harmonic ultrasonography is a method used for the real-time evaluation of tissue perfusion, without Doppler-related artifacts, and has improved the depiction and characterization of digestive diseases by trans-abdominal ultrasonography 11 12. It is based on the detection of intravenous contrast agents by means of a dedicated harmonic. Recently, it was found that contrast-enhanced harmonic imaging could be generated by using an echo-endoscope with a wide-band transducer 13. Recent studies have shown that CH-EUS may improve the evaluation of solid pancreatic lesions (SPLs) and that the microvascular pattern closely correlates with the histologic features of the lesion 14 15. In a recent meta-analysis 16, the pooled sensitivity of CH-EUS for the differential diagnosis of pancreatic adenocarcinoma was 94 % (95 % confidence interval [95 %CI] 0.91 – 0.95), and the specificity was 89 % (95 %CI 0.85 – 0.92). The area under the curve under summary receiver operating characteristic (AUROC) was 0.9732. The pooled positive likelihood ratio was 8.09 (95 %CI 4.47 – 14.64), and the negative likelihood ratio was 0.08 (95 %CI 0.06 – 0.10).

Although CH-EUS seems to be promising for the evaluation of SPLs, it is not clear whether the interpretation of CH-EUS is reproducible among different endosonographers. Previous reports assessing the reproductibility of CH-EUS for the evaluation of SPLs involved mainly experienced endosonographers. The main aim of this study was to assess the interobserver agreement (IOA) of CH-EUS in the evaluation of SPLs by endoscopists with different levels of experience in EUS and CH-EUS. We additionally evaluated the accuracy of CH-EUS for the diagnosis of pancreatic adenocarcinoma by endoscopists with different levels of experience in EUS and CH-EUS.

Participants and methods

Design of the study and selection of patients

This was a cross-sectional observational study with two aims. The primary aim was to assess the IOA of CH-EUS in the evaluation of SPLs by endoscopists with different levels of experience in EUS and CH-EUS. The secondary aim was to assess the accuracy of CH-EUS for the diagnosis of pancreatic adenocarcinoma by endoscopists with different levels of experience in EUS and CH-EUS.

A total of 60 patients with SPLs who underwent routine EUS at the Department of Gastroenterology, University Hospital of Santiago de Compostela (Spain), during 2011 were consecutively included in this study after giving informed consent for EUS. A final diagnosis of malignant or benign tumor was defined according to the following reference methods: (1) histologic findings of surgical specimens in patients undergoing surgery; (2) cytologic findings definitely positive for malignancy, together with compatible EUS and computed tomographic (CT) findings for a final diagnosis of malignant disease, in patients with unresectable tumors; and (3) EUS and CT findings at entry, clinical presentation, and a minimum follow-up period of 6 months, including EUS-FNA and CT, for a final diagnosis of benign disease in patients with benign cytologic findings. All of the material provided for the study was anonymous, and in no instance was a patient’s identity revealed. The study was approved by the local institutional review board and conducted in accordance with the Declaration of Helsinki and its amendments, as well as Good Clinical Practice guidelines.

Technique of endoscopic ultrasonography and selection of videos

In each of the 60 patients, the SPL was evaluated with standard EUS imaging and CH-EUS. EUS was performed with a linear EUS probe (EG3830UTK; Pentax Europe GmbH, Hamburg, Germany) attached to a Preirus platform (Hitachi Medical Systems GmbH, Wiesbaden, Germany), which includes the harmonic module. All procedures were done by two experienced endosonographers (J. I.-G. and J. L.-N.). SonoVue (sulfur hexafluoride MBs; Bracco International BV, Amsterdam, the Netherlands) was the contrast agent used for CH-EUS in all cases. The complete description of the technique of CH-EUS has been reported elsewhere 10. For each patient, one video sequence was recorded for 2 minutes, starting at the time of SonoVue administration (Video 1). Each video sequence also included a B-mode standard EUS image of the lesion of interest. Each video sequence was labeled with a random number by an endosonographer who had not participated in the EUS procedure and was blinded to the clinical history and the pathologic diagnosis. The observers were provided with a pen drive containing the 60 video sequences and were allowed unlimited time to review the videos. On the other hand, the observers were blinded to the clinical history and the pathologic diagnosis and to each other’s evaluation. No prior selection was made based on the quality of recorded images to avoid inducing any bias in the IOA evaluation.

Video sequence of contrast-enhanced harmonic endoscopic ultrasonography (left panel). The video sequence also included a B-mode standard endoscopic ultrasound image of the lesion of interest (right panel).

Video 1

Selection of observers and evaluation of videos

A total of 11 endoscopists from six EUS centers (Santiago de Compostela, Spain; Braga, Portugal; Porto, Portugal; Viana do Castelo, Portugal; Guimarães, Portugal; Gothenburg, Sweden) participated in this study. They were divided into four groups according to their experience in EUS and CH-EUS.

Group A included two endosonographers (J. I.-G. and J. L.-N) with long experience in EUS (> 1000 procedures) and CH- EUS (> 200 procedures).

Group B included three endosonographers (B. L., L. L., and J.-B. S.) with 3 months of experience in EUS (> 100 procedures) and CH-EUS (> 20 procedures).

Group C included three endosonographers (P. M., P. P.-N., and P. B.) with long experience in EUS (> 1000 procedures) but no experience in CH-EUS.

Group D included three endoscopists (A. F., A. C. C., and B. G.) with no experience in EUS or CH- EUS.

A kickoff session of 20 minutes was undertaken to share the principles of the techniques and to make everybody acquainted with the parameters of CH-EUS under evaluation.

Observers were asked to classify the lesion of interest, based on its overall degree of enhancement in comparison with the surrounding structures, as one of three types: 1, hypo-enhancement; 2, iso-enhancement; 3, hyper-enhancement. The degree of enhancement was evaluated during the arterial phase starting from the first arrival of contrast (usually in 10 – 20 seconds) until approximately 30 to 45 seconds 17.

We also calculated the diagnostic accuracy for pancreatic adenocarcinoma by comparing the pattern of CH-EUS indicative of pancreatic adenocarcinoma with the final diagnosis as previously described. We considered a hypo-enhanced contrast pattern in CH-EUS as indicative of pancreatic adenocarcinoma.

Statistical analysis

The Fleiss kappa (κ) statistic was used to evaluate the IOA among observers. An individual κ value for each group of observers, as well as an overall κ value, was determined for CH-EUS. The κ values were interpreted according to the guidelines proposed by Landis and Koch 18. The κ statistic allocates a score of 0 if the agreement is no better than would be expected by chance, whereas perfect agreement is indicated by a κ value of 1. Scores can also be negative if there is consistent disagreement. In detail, κ values of 0.00 to 0.19 represent slight agreement, 0.20 to 0.39 fair agreement, 0.40 to 0.59 moderate agreement, and 0.60 to 0.79 substantial agreement; a value of more than 0.80 is considered almost perfect agreement. The κ values were considered statistically significant when the 95 %CI of the κ values was superior to 0. Bootstrap resampling was used to calculate the 95 %CI of the κ values. Statistical comparison of κ values between groups was done with the κ analysis extension for ArcView 3.2. We also evaluated the sensitivity, specificity, positive predictive value, negative predictive value, and AUROC of each group for the final diagnosis of pancreatic adenocarcinoma by using the hypo-enhanced pattern in CH-EUS as indicative of the presence of pancreatic adenocarcinoma 16.

With the exception of the comparison of κ values (see above), all statistical analyses were performed with SPSS 18.0 software (Chicago, Illinois, USA). Differences with a P value of < 0.05 were considered significant.

Results

Patients’ characteristics

A total of 60 patients (17 women and 43 men with a mean age of 64 ± 15 years) were included in the study (Table 1). The mean size of the pancreatic masses was 36.5 ± 15.9 mm. The lesions were located mostly in the pancreatic head. The diagnosis was based on EUS-FNA in 43 patients, on EUS-FNB (endoscopic ultrasonography-guided fine-needle biopsy) in 14 patients, on surgery in 1 patient, and on follow-up in 2 patients. As determined according to the reference methods, the final diagnoses were as follows: pancreatic adenocarcinoma (45 patients), inflammatory mass in the context of chronic pancreatitis (10 patients), pancreatic neuroendocrine tumor (3 patients), autoimmune pancreatitis (1 patient), and metastatic colon cancer metastasis (1 patient).

Characteristics of patients and pancreatic lesions included in an analysis of interobserver agreement of contrast-enhanced harmonic endoscopic ultrasonography in the evaluation of solid pancreatic lesions.

Age of patients, mean± SD, y	64 ± 15
Gender of patients (females : males)	17 : 43
Lesion size, mean ± SD, mm	36.5 ± 15.9
Location of lesions in pancreas (head/body/tail)	43/15/2
Final diagnosis Pancreatic adenocarcinoma Inflammatory mass in the context of chronic pancreatitis Pancreatic neuroendocrine tumor Autoimmune pancreatitis Colon cancer metastasis	4510 3 1 1

SD, standard deviation.

Interobserver agreement

The IOA evaluation data are presented in Table 2. The overall IOA for CH-EUS was fair. Group A had the highest IOA, followed by groups C, B, and D. The IOA of groups A and C was significantly higher than that of group D. No significant difference was seen between groups A, B, and C or between groups B and D in terms of IOA.

Results of interobserver agreement for each group of endosonographers.

Group (number of observers)	κ (95 %CI)	Interpretation of agreement
A (2 observers with long experience in EUS and CH-EUS)	0.63 (0.45 – 0.85)1	Substantial
B (3 observers with short experience in EUS and CH-EUS)	0.38 (0.22 – 0.55)	Fair
C (3 observers with long experience in EUS and no experience in CH-EUS)	0.54 (0.39 – 0.71)1	Moderate
D (3 observers with no experience in EUS or CH-EUS)	0.21 (0.07 – 0.36)	Fair
Overall (11 observers)	0.32 (0.22 – 0.41)	Fair

κ, kappa value; 95 %CI, 95 % confidence interval; EUS, endoscopic ultrasonography; CH-EUS, contrast-enhanced harmonic endoscopic ultrasonography.Data are shown as κ (95 %CI).

P  < 0.05 vs. group D.

Diagnostic accuracy for pancreatic adenocarcinoma

The diagnostic accuracy data for pancreatic adenocarcinoma are presented in Table 3. Group A had the highest diagnostic accuracy, followed by groups C, B, and D. The diagnostic accuracy of group A was not significantly different from that of group C, but it was significantly higher than that of groups B and D. No significant difference was seen between groups B, C, and D in terms of diagnostic accuracy.

Diagnostic accuracy of each group of endosonographers for pancreatic adenocarcinoma.

	Group A	Group B	Group C	Group D	Overall
Sensitivity (95 %CI)	79.6 (70.3 – 87.1)	61.9 (53.5 – 69.8)	64.0 (55.6 – 71.7)	58.5 (50.1 – 66.6)	64.6 (60.6 – 68.8)
Specificity (95 %CI)	54.6 (32.2 – 75.6)	48.5 (30.8 – 66.4)	51.5 (33.6 – 69.2)	42.4 (25.5 – 60.8)	48.8 (39.6 – 58.0)
PPV (95 %CI)	88.6 (80.1 – 94.4)	84.3 (76.0 – 90.6)	85.5 (77.5 – 91.4)	81.9 (73.2 – 88.7)	84.9 (81.1 – 88.2)
NPV (95 %CI)	37.5 (21.1 – 56.3)	22.2 (13.3 – 33.6)	24.3 (14.8 – 36.0)	18.7 (10.6 – 29.3)	23.7 (18.5 – 29.5)
AUROC (95 %CI)	0.67 (0.58 – 0.75)	0.55 (0.48 – 0.63)1	0.58 (0.50 – 0.65)	0.51 (0.43 – 0.58)1	0.57 (0.53 – 0.61)

95 %CI, 95 % confidence interval; PPV, positive predictive value; NPV, negative predictive value; AUROC, area under the curve under summary receiver operating characteristic.Results of the evaluation of 45 pancreatic adenocarcinoma and 15 non-pancreatic adenocarcinoma lesions (inflammatory mass in the context of chronic pancreatitis, n = 10; neuroendocrine tumor, n = 3; autoimmune pancreatitis, n = 1; metastasis, n = 1) were included for analysis. Pancreatic adenocarcinoma was defined by the presence of a hypo-enhanced contrast pattern in contrast-enhanced harmonic endoscopic ultrasonography (CH-EUS).

P < 0.05 vs. group A.

Discussion

This is the first study evaluating the IOA of CH-EUS in the evaluation of SPLs by endoscopists with different levels of experience in EUS and CH-EUS. Our data suggest that CH-EUS is reproducible in the evaluation of SPLs, even between endoscopists with no or limited experience in EUS and/or CH-EUS. Our data also suggest that experience in both EUS and CH-EUS influences the IOA. This finding is based on the observation that group A (observers with long experience in EUS and CH-EUS) had the highest IOA for CH-EUS (κ = 0.63; 95 %CI 0.45 – 0.85), whereas group D (observers with no experience in EUS and CH-EUS) had the lowest IOA for CH-EUS (κ = 0.21; 95 %CI 0.07 – 0.36; P < 0.05). The fact that group B (observers with short experience in EUS and CH-EUS) and group C (observers with long experience in EUS but no experience in CH-EUS) had similar IOA for CH-EUS (κ = 0.38; 95 %CI 0.22 – 0.55 vs. κ = 0.54; 95 %CI 0.39 – 0.71; P > 0.05) suggests that long experience in EUS may influence the IOA of CH-EUS and compensate for the lack of experience in CH-EUS.

Some studies have reported IOA in the evaluation of SPLs by CH-EUS, but only two studies compared endosonographers with different levels of experience in EUS 19 20. Fusaroli et al. evaluated the IOA of CH-EUS in 40 SPLs conducted by eight endosonographers who were experienced in EUS and seven endosonographers who were not experienced in EUS 19. They reported a κ value for contrast uptake of 0.56 for all endosonographers, 0.56 for the experienced endosonographers, and 0.55 for the endosonographers who were not experienced. Gincul et al. evaluated the IOA of CH-EUS in 100 SPLs conducted by five senior and two junior endosonographers 20, reporting a κ value of 0.66 for all endosonographers, 0.65 for the senior endosonographers, and 0.76 for the junior endosonographers. Kitano et al. evaluated the IOA of CH-EUS in 277 SPLs conducted by two endosonographers experienced in EUS and CH-EUS 21, reporting a κ value for contrast uptake of 0.95. Even between experienced endosonographers, these studies show a large range of κ values (0.56 – 0.95) for the evaluation of contrast uptake by SPLs. In our study, the κ value (0.63) for the evaluation of contrast uptake by SPLs by experienced endosonographers fits within the range of previously reported κ values. On the other hand, contrary to the studies of Fusaroli et al. and Gincul et al., who reported no difference between endosonographers who were experienced and those who were not experienced, our study found a difference between endosonographers with different levels of experience in EUS and CH-EUS.

This is also the first study comparing the diagnostic accuracy of CH-EUS for pancreatic adenocarcinoma by endoscopists with different levels of experience in EUS and CH-EUS. The results of the diagnostic accuracy evaluation were very similar to those of the IOA evaluation. As for IAO, our data suggest that experience in EUS or CH-EUS influences the diagnostic accuracy. This is based on the observation that diagnostic accuracy of CH-EUS was highest in group A (AUROC = 0.67; 95 %CI 0.58 – 0.75) and lowest in group D (AUROC = 0.51; 95 %CI 0.43 – 0.58; P < 0.05). The fact that group B (observers with short experience in EUS and CH-EUS) and group C (observers with long experience in EUS but no experience in CH-EUS) had similar diagnostic accuracy for CH-EUS (AUROC = 0.55; 95 %CI 0.48 – 0.63 vs. AUROC = 0.58; 95 %CI 0.50 – 0.65; P > 0.05) suggests that long experience in EUS may influences the diagnostic accuracy of CH-EUS and compensate for the lack of experience in CH-EUS.

Only one previous study has compared the diagnostic accuracy of CH-EUS for pancreatic adenocarcinoma by observers with different levels of experience in EUS 20. In this study, the observer accuracies for the diagnosis of pancreatic adenocarcinoma by CH-EUS ranged from 81 % to 91 %, with no statistically significant difference found between the senior and junior endosonographers 20. However, in this study, no clear definition of senior and junior endosonographer is provided. Thus, the difference in the experience of the senior and junior endosonographers is not clear. This would be important to compare our data with the data of that study.

Previous long experience in EUS seems to have a significant influence on both the IOA and diagnostic accuracy of CH-EUS. This is based on the observation that the IOA and diagnostic accuracy of CH-EUS in group C were similar to those of groups A and B, even though group C had no experience in CH-EUS.

A limitation of CH-EUS is that in the qualitative image analysis performed, the findings are amenable to subjective interpretation. Thus, methods for the quantitative assessment of CH-EUS have been recently developed, such as the contrast uptake ratio index and the time–intensity curve 16 22 23. Although these methods have already proved to be helpful, there are still some limitations. All are based on the computed automated analysis of regions of interest that are selected subjectively, thus allowing the generation of selection bias. Moreover, they have not yet been proved to be superior to the qualitative analysis of CH-EUS 16 22 23.

This study has some weaknesses. First, the small sample size, the low number of observers per group, and the low rate of non-neoplastic lesions (although similar to that of clinical practice) may have influenced our data. In fact, a major limitation of the study is the smaller number of observers in group A (two observers) than in the other groups (three observers per group). This was the result of the low number of available experts in CH-EUS and may have led to overestimation of the IOA of group A, compromising the comparison between this group and the other groups. Even so, this does not compromise either the overall data for the IOA of CH-EUS or the comparison of the other three groups, and thus does not influence the major conclusions of the study. Second, the procedures were done by the same experts who later evaluated the lesions (group A). To reduce this bias, each video sequence was labeled with a random number, and the experts blindly evaluated the lesions. Third, each video sequence also included a B-mode standard EUS image. Although this is similar to the clinical practice of CH-EUS, the evaluation of a B-mode standard EUS image could influence the evaluation of CH-EUS images, thus possibly contributing to overestimation of the IOA and diagnostic accuracy of observers with long experience in EUS (groups A and C). It has been shown that B-mode standard EUS may show changes strongly consistent with malignant tumor in 30 % of SPLs 24. Finally, as we did not assess the intra-observer agreement of CH-EUS, we cannot make any conclusions about intra-observer reproducibility of CH-EUS.

The IOA and diagnostic accuracy of CH-EUS can be influenced by the contrast agent that is used. Besides SonoVue (phospholipid-stabilized microbubbles of sulfur hexafluoride), which was used in our study, Sonazoid (microbubbles of perfluorobutane with a lipid membrane encapsulation; Daiichi Pharmaceutical, Tokyo, Japan, and GE Healthcare, Chalfont St. Giles, United Kingdom) is also a commonly used ultrasound contrast agents for CH-EUS 13 25. It has been shown that the two ultrasound contrast agents differ in terms of intensity and duration of signaling after infusion. CH-EUS images obtained with SonoVue disappear within 60 seconds, which limits the duration of observation. In contrast, after infusion of Sonazoid, parenchymal perfusion can be observed throughout the pancreas for at least 90 seconds. Thus, the longer-lasting effect of Sonazoid may improve the observation of the pancreas by CH-EUS. Nonetheless, no study has directly compared the two contrast agents in the evaluation of SPLs. Moreover, in our study, we evaluated only the arterial phase, starting from the first arrival of contrast (usually in 10 – 20 seconds) until approximately 30 to 45 seconds, so it is very unlikely that our results would be different with Sonazoid.

In conclusion, we present the first multicentric study of the IOA and diagnostic accuracy of CH-EUS in the evaluation of SPLs by endoscopists with different levels of experience in EUS and CH-EUS. Our data suggest that CH-EUS is reproducible in the evaluation of SPLs, even in groups with no or limited experience in EUS and/or CH-EUS. The study also hints that long experience in EUS is a major contributor to the IOA and diagnostic accuracy of CH-EUS. Nonetheless, these data should undergo external validation in larger studies.