First interobserver agreement of optical coherence tomography in the bile duct: A multicenter collaborative study.

Background and study aims  Optical coherence tomography (OCT) is a new technology available for evaluation of indeterminate biliary strictures. It allows under-the-surface visualization and preliminary studies have confirmed standardized characteristics associated with malignancy. The aim of this study is to evaluate the first interobserver agreement in identifying previously agreed upon OCT criteria and diagnosing of malignant versus benign disease. Patients and methods  Fourteen endoscopists were asked to review an atlas of reference clips and images of eight criteria derived from expert consensus A total of 35 de-identified video clips were then evaluated for presence of the eight criteria and for final diagnosis of malignant versus benign using the atlas as reference Intraclass correlation (ICC) analysis was done to evaluate interrater agreement. Results  Clips of 23 malignant lesions and 12 benign lesions were scored. Excellent interobserver agreement was seen with dilated hypo-reflective structures (0.85) and layering effacement (0.89); hyper-glandular mucosa (0.76), intact layering (0.81), and onion-skin layering (0.77); fair agreement was seen with scalloping (0.58), and thickened epithelium (0.4); poor agreement was seen with hyper-reflective surface (0.36). The diagnostic ICC for both neoplastic (0.8) and non-neoplastic (0.8) was excellent interobserver agreement. The overall diagnostic accuracy was 51 %, ranging from 43 % to 60 %. Conclusions  Biliary OCT is a promising new modality for evaluation of indeterminate biliary strictures. Interobserver agreement ranged from fair to almost perfect on eight previously identified criteria. Interobserver agreement for malignancy diagnosis was substantial (0.8). Further studies are needed to validate this data. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/).

Introduction

Accurate and timely diagnosis of biliary strictures remains a challenge. Current diagnostic modalities include cytology brushing, confocal endomicroscopy, and cholangioscopy. Tissue sampling with brushing and biopsy during endoscopic retrograde cholangiopancreatography (ERCP) has been the standard for diagnosis, however this method is not adequately sensitive, with reported sensitivities ranging from 18 % to 60 % 1 2 . The addition of confocal endomicroscopy increases accuracy and sensitivity, but reported sensitives are still below 90 % 3 . Direct visualization of the pancreaticobiliary mucosa and targeted biopsies is achieved by cholangioscopy, but this is limited by visualization of the surface epithelium only 4 5 6 .

Optical coherence tomography (OCT) is an optical imaging modality introduced in 1991 that can perform high-resolution, cross-sectional tomographic imaging that can be interpreted in vivo. OCT measures back-scattered or back-reflected infrared light to image tissue microstructure. Multiple studies, both in-vitro and in vivo, have demonstrated the ability of OCT to visualize and differentiate between the multiple layers of the gastrointestinal and pancreatobiliary wall structure as well as various microscopic structures such as blood vessels and submucosal glands 7 8 9 10 11 . Due to differences in light backscattering processes, OCT has also been shown to differentiate between neoplastic and non-neoplastic tissue 12 13 14 .

While early studies showed promising ability of OCT to delineate mucosal layers and tissue microstructures, widespread use was limited by shallow penetration depths and suboptimal resolution 9 10 12 15 16 17 18 19 20 . However, newer versions of this technology, such as the NVision Volumetric Laser Endomicroscopy (VLE) (Ninepoint, Bedford, Massachusetts, United States), allow for higher quality images 21 22 23 . These high-quality images and visualization of microstructures and layers of the pancreatobiliary wall make OCT a potential complementary technique to previously established modalities of diagnosing indeterminate biliary strictures such as cytology brushing and cholangioscopy.

OCT has previously been demonstrated to be both safe and feasible, and preliminary studies have confirmed standardized characteristics associated with benign and malignant disease 21 22 23 . The aim of this study is to evaluate the first interobserver agreement in identifying previously agreed upon OCT criteria and making a diagnosis of malignant versus benign disease.

Patients and methods

Criteria

Tyberg et al. identified nine specific criteria based on recurring characteristics described by individual operators; these criteria were then condensed to eight criteria by expert opinion during an invited meeting of experts in OCT technology 23 . These criteria set included presence of: 1) dilated hypo-reflective structures; 2) hyper-glandular mucosa ( > 3 per frame); 3) hyper-reflective surface; 4) intact layering; 5) layering effacement; 6) onion-skin layering; 7) thickened epithelium; and 8) scalloping ( Fig. 1 )

Fig. 1 a 

Dilated hypo-reflective structures. b Hyper-glandular mucosa (> 3 per frame). c Hyper-reflective surface. d Intact layering. e Layering effacement. f Onion-skin layering. g Thickened epithelium. h Scalloping.

Video 1  Visualization of scalloping criteria in the bile duct.

Video clips and images

Three tertiary-care centers contributed a total of 35 de-identified video recording clips in mp4 (MPEG-4 Part 14) format for the IOA scoring; as well as 10 reference images and clips for a new reference atlas ( Video 1 ). None of these recordings and images were used in any previous studies or publications.

The OCT images used in the atlas and the video clips were obtained from adult patients ( Table 1 ) evaluated for indeterminate biliary strictures between January 2017 to May 2018 using the NVision VLE Imaging System (Ninepoint Medical). Obtained images contained an axial resolution of seven microns, lateral resolution of 40 microns, focal length of 2.4 mm and depth of image of 3 mm. Images were captured via 90-second scan encompassing a length of 6 cm.

Patient characteristics and clinical diagnoses.

Case	Age	Gender	Presenting symptoms and cross-sectional imaging	Final diagnosis
1	72	Male	Jaundice, distal biliary stricture	Pancreatic cancer
2	65	Female	Jaundice, hilar stricture	Cholangiocarcinoma
3	58	Male	Elevated LFTs, distal biliary stricture	Pancreatic cancer
4	65	Male	Elevated LFTs, CHD stricture	Post-surgical stricture
5	66	Female	Jaundice, CHD stricture	Benign biliary stricture
6	59	Male	Jaundice, distal biliary stricture	Benign biliary stricture
7	72	Male	Elevated LFTs, distal biliary stricture	Benign biliary stricture
8	29	Female	Elevated LFTs, distal biliary stricture	Pancreatic cancer
9	73	Male	Elevated LFTs, hilar stricture	Cholangiocarcinoma
10	69	Female	Elevated LFTs, distal biliary stricture	Benign biliary stricture
11	62	Female	Jaundice, distal biliary stricture	Pancreatic cancer
12	72	Male	Jaundice, distal biliary stricture	Pancreatic cancer
13	68	Female	Elevated LFTs, hilar stricture	Cholangiocarcinoma
14	75	Male	Elevated LFTs, distal biliary stricture	Pancreatic cancer
15	81	Female	Jaundice, hilar stricture	Cholangiocarcinoma
16	67	Male	Jaundice, distal biliary stricture	Benign Biliary stricture
17	75	Female	Elevated LFTs, distal biliary stricture	Pancreatic cancer
18	59	Male	Jaundice, distal biliary stricture	Pancreatic cancer
19	76	Female	Elevated LFTs, distal biliary stricture	Benign biliary stricture
20	75	Female	Jaundice, distal biliary stricture	Pancreatic cancer
21	66	Female	Jaundice, hilar stricture	Cholangiocarcinoma
22	82	Female	Jaundice, distal biliary stricture	Pancreatic cancer
23	75	Male	Jaundice, distal biliary stricture	Pancreatic cancer
24	71	Male	Elevated LFTs, hilar stricture	Cholangiocarcinoma
25	59	Female	Jaundice, distal biliary stricture	Pancreatic cancer
26	68	Female	Jaundice, distal biliary stricture	Benign Biliary stricture
27	56	Male	Jaundice, CHD stricture	Benign Biliary stricture
28	74	Male	Elevated LFTs, hilar stricture	Cholangiocarcinoma
29	58	Female	Jaundice, distal biliary stricture	Pancreatic cancer
30	63	Male	Jaundice, distal biliary stricture	Benign Biliary stricture
31	65	Female	Jaundice, CHD stricture	Benign Biliary stricture
32	58	Male	Jaundice, Hilar stricture	Cholangiocarcinoma
33	82	Female	Jaundice, distal biliary stricture	Pancreatic cancer
34	76	Female	Jaundice, distal biliary stricture	Pancreatic cancer
35	52	Male	Jaundice, CHD stricture	Benign biliary stricture

LFT, liver function test; CHD, common hepatic duct.

None of these patients had any history of primary sclerosing cholangitis. A final diagnosis of malignancy was obtained from histopathology results and confirmed on follow-up. For those patients with benign lesions on OCT and histology, a minimum of 6-month follow-up was required before confirming a benign etiology.

Each clip was at least 20 seconds long in duration and were selected based on image resolution, presence of at least one criterion, and confirmed final diagnosis. The median duration of the 35 clips was 25 seconds. The video clips focused on only the visualization of the affected bile duct. There were no mid-clip edits or any other videography/software enhancements to alter resolution.

After the reference video atlas was created, it was reviewed and confirmed by expert interventional endoscopists with experience in biliary OCT.

The participating 14 interventional endoscopists were first instructed to familiarize themselves with the criteria published in Tyberg et al. 11 .

Then they were asked to view the new reference video and images atlas that displayed these criteria.

Upon completion of the review of the reference atlas, 35 de-identified OCT video clips and corresponding scoring sheet were sent to the endoscopists via an online survey module.

Expert interventional endoscopists view each clip and mark the presence or absence of each criterion visualized in each clip on the scoring sheet. No medical history, clinical data, histological data, final diagnosis, fluoroscopic images, or other imaging was provided to these endoscopists.

The endoscopists then classified the clips as neoplastic or non-neoplastic based on criteria they visualized in each clip.

A majority of interventional endoscopists had performed a minimum of 100 cholangioscopies and minimum of 20 OCT procedures at the time of the study. At the time of the study, all but one endoscopist had conducted at least five biliary OCT procedures. One endoscopist had performed 15. Endoscopists whose centers contributed clips to the study or reference atlas were excluded from scoring the video clips.

Statistical analysis

Intraclass correlation (ICC) analysis was conducted to evaluate interrater agreement for the eight criteria and final diagnosis based on a mean-rating (k = 14), consistency and absolute agreement, 2-way mixed-effects model. The coefficient interpretations were based on the Cicchetti (1994) guidelines: less than 0.40—poor; between 0.40 and 0.59—fair; between 0.60 and 0.74—good; and between 0.75 and 1.00—excellent.

Categorial variables were described as frequencies and percentages, and quantitative variables as mean and percentage. Accuracy to detect malignancy on indeterminate biliary strictures using OCT were calculated for each endoscopist. Chi square test and Fishers exact test were conducted for proportions. P  < 0.05 was considered statistically significant. All descriptive and statistical analyses were conducted using MedCalc V18.9 (MedCalc Software, Ostend, Belgium).

This study was deemed exempt trial by WCG IRB (NCT03951324).

Results

A total of 14 interventional endoscopists completed the scoring. Clips of 23 malignant lesions and 12 benign lesions were scored. The mean time of the clip was 19.8 seconds (range 7–47). The ICC using criteria ranged from poor to excellent (range 0.36 to 0.9) ( Table 2 ).

Intraclass correlation of previously determined OCT criteria to evaluate biliary strictures.

Criteria	Coefficient	Intraclass correlation	95 % confidence interval
Presence of dilated hypo-reflective structures	0.85	Excellent	0.76 to 0.91
Presence of hyper-glandular mucosa (> 3 per frame)	0.76	Excellent	0.63 to 0.86
Presence of hyper-reflective surface	0.36	Poor	0.004to 0.63
Presence of intact layering	0.81	Excellent	0.70 to 0.89
Presence of layering effacement	0.89	Excellent	0.73 to 0.94
Presence of onion-skin layering	0.77	Excellent	0.65 to 0.87
Presence of thickened epithelium	0.40	Fair	0.06 to 0.66
Presence of scalloping	0.58	Fair	0.34 to 0.76
Neoplastic diagnosis	0.79	Excellent	0.67 to 0.88
Non-neoplastic diagnosis	0.80	Excellent	0.68 to 0.88

Excellent interobserver agreement was seen with dilated hypo-reflective structures (0.85), layering effacement (0.89); hyper-glandular mucosa (0.76), intact layering (0.81), and onion-skin layering (0.77); fair agreement was seen with scalloping (0.58), and thickened epithelium (0.4); poor agreement was seen with hyper-reflective surface (0.36). The diagnostic ICC for both neoplastic (0.8) and non-neoplastic (0.8) had excellent interobserver agreement. The overall diagnostic accuracy was 51 %, ranging from 43 % to 60 %.

Of 10 variables, seven had excellent interobserver agreement.

Criteria frequency

Neoplastic clips

The most frequent criteria visualized in neoplastic clips were: dilated hypo-reflection structures (51 %), thickened epithelium (57 %), layering effacement (49 %) and hyper-reflective surface (49 %) ( Table 3 ). The majority of the neoplastic clips (83 %) featured at least three of these criteria (17/23).

Criteria frequency

Feature	Overall	Neoplastic	Non-neoplastic
Presence of dilated hypo-reflective structures	18/35 (51 %)	14/23 (61 %)	4/12 (33 %)
Presence of hyper-glandular mucosa (> 3 per frame)	11/35 (31 %)	7/23 (30 %)	4/12 (33 %)
Presence of hyper-reflective surface	17/35 (49 %)	11/23 (48 %)	6/12 (50 %)
Presence of intact layering	14/35 (40 %)	7/23 (30 %)	7/12 (58 %)
Presence of layering effacement	17/35 (49 %)	12/23 (52 %)	5/12 (42 %)
Presence of onion-skin layering	8/35 (23 %)	7/23 (30 %)	1/12 (8 %)
Presence of thickened epithelium	20/35 (57 %)	13/23 (56.5 %)	7/12 (58 %)
Presence of scalloping	10/35 (29 %)	5/23 (22 %)	5/12 (42 %)

P value compares frequency of criteria visualized in malignant clips vs benign clips.

At least six criteria were present in two of 23 clips, at least four or more criteria in eight of 23 clips (46 %), at least three or more criteria in 17 of 23 clips (83 %) and at least two or more criteria in 23 of 23 clips (100 %). Hence, in this study, based on endoscopist visualization, 83 % of the cases had a moderate to slightly high risk of being diagnosed as neoplastic if five or more criteria were visualized during DSOC.

Non-neoplastic clips : Presence of intact layering and presence of thickened epithelium (58 %, P  = 0.08) were the only criteria most frequently visualized in non-neoplastic clips (7/12). At least two or more criteria were visualized in all clips (12/12; 100 %) and five criteria were visualized in two of 12 clips. All non-neoplastic clips featured at least two criteria.

Discussion

OCT evaluation of the biliary tree is a promising new modality for the evaluation of indeterminate biliary strictures. Current standard of care of evaluating these strictures includes cytology brushing during ERCP, confocal endomicroscopy, and cholangioscopy. However, while the addition of confocal endomicroscopy and cholangioscopy increase sensitivity of traditional cytology brushing, sensitivity of these methods still remains inadequate for diagnosing indeterminate strictures 1 2 3 4 .

Multiple ex vivo studies have demonstrated that OCT can delineate tissue microarchitecture in the pancreatic duct and the bile duct, and these studies also show high concordance with histology in benign and malignant disease 9 12 18 . Furthermore, in vivo studies have confirmed the ability of OCT to define biliary architecture while also demonstrating safety and feasibility of this method 15 16 17 19 20 . OCT seems to be an overall safe addition to procedures; adding OCT to ERCP does not appear to increase procedure risk and reported adverse events rates have been low 21 23 . Technical success rates of OCT have also been reported to be as high as 100 % 23 .

OCT notably has many advantages over other diagnostic modalities for indeterminate biliary strictures 23 . While the surface mucosa can be examined with cholangioscopy, OCT allows for visualization of structures below the surface mucosa which may show lesions that do not extend transmurally into the duct. In addition, the OCT probe has a smaller diameter than a cholangioscope, which allows for easier passage and visualization of tight strictures. Finally, while confocal endomicroscopy can evaluate one small area of tissue at a time, OCT has the ability to evaluate 6 cm of tissue in 90 seconds 23 .

The previously demonstrated safety, feasibility, and advantages of OCT in evaluating indeterminate biliary strictures is promising, However, to better utilize this tool, criteria predictive of benign or malignant disease need to be determined. Accurate diagnosis of indeterminate biliary strictures is critical, as management is dependent on whether the stricture is benign or malignant. The normal wall of the bile duct has an inner hypo-reflective layer corresponding to the epithelium, a hyper-reflective layer corresponding to the fibromuscular tissue, and a hypo-reflective layer representing the connective tissue 9 . Within a biliary stricture, the fibromuscular layer is larger, but the three layers of the wall are maintained, whereas a malignant lesion will lose organization of the bile duct wall layers and have large non-reflective areas which may be due to underlying tumor vessels 25 . In this study, a reference video atlas was created that contained clips and images of eight previously agreed upon criteria based on expert opinion and prior studies. These clips were then scored by 14 expert interventional endoscopists and ICC estimates were calculated. ( Table 4 )

Interobserver diagnostic accuracy.

		Interobserver (IOV) diagnostic accuracy, yes or no
Clip no.	Actual diagnosis	IOV 1	IOV 2	IOV 3	IOV 4	IOV 5	IOV 6	IOV 7	IOV 8	IOV 9	IOV 10	IOV 11	IOV 12	IOV 13	IOV 14
Clip 1	Neoplastic	No	No	No	No	No	Yes	No	No	Yes	Yes	Yes	No	No	No
Clip 2	Neoplastic	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes
Clip 3	Neoplastic	Yes	No	No	No	No	No	No	No	Yes	No	No	No	No	No
Clip 4	Non-neoplastic	Yes	No	Yes	No	No	No	No	No	No	Yes	No	Yes	No	No
Clip 5	Non-neoplastic	Yes	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No	Yes
Clip 6	Non-neoplastic	Yes	Yes	Yes	Yes	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Clip 7	Non-neoplastic	Yes	Yes	Yes	Yes	Yes	Yes	No	Yes	No	Yes	Yes	Yes	Yes	Yes
Clip 8	Neoplastic	No	No	No	No	No	Yes	Yes	No	No	No	Yes	No	Yes	No
Clip 9	Neoplastic	No	Yes	Yes	Yes	No	No	Yes	Yes	Yes	Yes	Yes	No	Yes	Yes
Clip 10	Non-neoplastic	Yes	Yes	No	No	No	No	No	No	No	No	No	Yes	No	No
Clip 11	Neoplastic	Yes	Yes	No	Yes	No	No	Yes	Yes	Yes	No	Yes	No	Yes	No
Clip 12	Neoplastic	No	No	Yes	Yes	No	No	No	No	Yes	Yes	No	No	No	No
Clip 13	Neoplastic	No	Yes	No	No	No	Yes	No	No	Yes	Yes	Yes	No	No	No
Clip 14	Neoplastic	No	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No	Yes	Yes	Yes	Yes
Clip 15	Neoplastic	No	Yes	Yes	Yes	Yes	No	Yes	Yes	Yes	No	Yes	No	Yes	Yes
Clip 16	Non-neoplastic	Yes	Yes	Yes	No	Yes	No	No	No	No	Yes	No	No	Yes	Yes
Clip 17	Neoplastic	No	Yes	No	No	No	No	No	No	No	No	No	No	No	No
Clip 18	Neoplastic	Yes	Yes	No	No	No	Yes	No	No	No	Yes	Yes	No	No	No
Clip 19	Non-neoplastic	Yes	No	No	No	No	No	No	No	No	Yes	No	Yes	No	No
Clip 20	Neoplastic	Yes	No	Yes	No	No	Yes	No	No	Yes	No	Yes	Yes	No	No
Clip 21	Neoplastic	Yes	No	No	No	No	Yes	No	No	No	No	Yes	No	No	No
Clip 22	Neoplastic	Yes	No	No	No	No	Yes	Yes	No	No	No	Yes	No	No	No
Clip 23	Neoplastic	No	Yes	Yes	Yes	No	No	No	Yes	Yes	Yes	No	No	No	No
Clip 24	Neoplastic	No	No	Yes	Yes	Yes	Yes	Yes	No	Yes	No	Yes	Yes	No	No
Clip 25	Neoplastic	No	Yes	No	Yes	No	Yes	Yes	Yes	Yes	No	No	No	No	No
Clip 26	Non-neoplastic	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No	Yes	No	Yes	Yes	Yes
Clip 27	Non-neoplastic	Yes	Yes	Yes	Yes	Yes	No	Yes	Yes	Yes	Yes	No	Yes	Yes	Yes
Clip 28	Neoplastic	Yes	No	Yes	No	Yes	Yes	Yes	Yes	No	No	Yes	Yes	Yes	Yes
Clip 29	Neoplastic	Yes	No	Yes	Yes	No	No	Yes	No	Yes	Yes	Yes	No	Yes	No
Clip 30	Non-neoplastic	Yes	Yes	No	No	Yes	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Clip 31	Non-neoplastic	Yes	Yes	Yes	No	Yes	No	Yes	Yes	No	No	Yes	Yes	No	Yes
Clip 32	Neoplastic	No	No	No	No	No	No	No	No	No	No	No	No	No	No
Clip 33	Neoplastic	No	Yes	No	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No	No	Yes
Clip 34	Neoplastic	No	Yes	No	No	No	Yes	Yes	Yes	Yes	No	No	No	Yes	Yes
Clip 35	Non-neoplastic	Yes	No	No	Yes	Yes	No	No	Yes	No	Yes	No	No	Yes	Yes
	Total accuracy (n)	20	19	18	16	15	18	18	18	19	19	21	15	16	16
	Total accuracy (%)	57 %	54 %	51 %	46 %	43 %	51 %	51 %	51 %	54 %	54 %	60 %	43 %	46 %	46 %

Excellent interobserver agreement was seen with dilated hypo-reflective structures (0.85) and layering effacement (0.89), which have previously been identified as a criteria suggestive of malignancy 20 . Excellent agreement was also seen onion-skin layering (0.77) and intact layering (0.81), which are features predictive of benign disease. Other criteria suggestive of malignancy, such as hyper-reflective surface and thickened epithelium showed poor to fair interobserver agreement, 0.36 and 0.4 respectively 21 . Overall, the diagnostic accuracy was fair at 51 % with a range of 43 % to 60 %. However, there was high interobserver agreement for the classification of neoplastic (0.8) versus non-neoplastic (0.8) lesions.

Limitations of this study include small sample size (35 clips) with only 14 interventional endoscopists scoring the films. At the time of the study (2018–2019), OCT was not available widely and only select centers were doing OCT for biliary indications. While diagnostic accuracy in this study was suboptimal, the majority of interventional endoscopists had performed minimum five OCT procedures at the time of the study. In addition, the high interobserver agreement for diagnosis of neoplastic vs non-neoplastic disease is promising. Utility of OCT is likely operator-dependent, and we expect that as endoscopist experience with OCT increases, accuracy will also increase.

Conclusions

This study provides promising data supporting OCT as an accurate modality to diagnose indeterminate biliary strictures and is the first study to demonstrate interobserver agreement. Further studies are needed as endoscopist experience increases to validate these criteria.