Training Effect on the Inter-observer Agreement in Endoscopic Diagnosis and Grading of Atrophic Gastritis according to Level of Endoscopic Experience.

BACKGROUND: Endoscopic diagnosis of atrophic gastritis can contribute to risk stratification and thereby tailored screening for gastric cancer. We aimed to evaluate the effect of training on inter-observer agreement in diagnosis and grading of endoscopic atrophic gastritis (EAG) according to the level of endoscopists' experience.
METHODS: Twelve endoscopists (six less-experienced and six experienced) participated in this prospective study. The training session consisted of 1) four interventions with two-week intervals, and 2) a follow-up period (two follow-up assessments without feedback). EAG was categorized as C1 to O3 according to the Kimura-Takemoto classification. Kappa statistics were used to calculate inter-observer agreement.
RESULTS: At baseline, kappa indexes were 0.18 in the less-experienced group and 0.32 in the experienced group, respectively. After four interventions with feedback, the kappa index improved in both groups and was sustained during the follow-up period. Overall diagnostic yields of EAG were 43.1% ± 10.7% in pre-intervention and 46.8% ± 5.9% in post-intervention. Variability in the rate of diagnosis of EAG significantly decreased in the less-experienced group (r = 0.04, P = 0.003).
CONCLUSION: Irrespective of experience level, inter-observer agreement for diagnosis and grading of EAG improved after training and remained stable after intervention.

INTRODUCTION

Presence of mucosal atrophy induced by the Helicobacter pylori infection in the background of gastric carcinogenesis has been widely recognized. Epidemiologic studies demonstrated the relationship between H. pylori associated atrophic gastritis (AG) and the development of gastric cancer.12 In addition, the extensive degree of endoscopic atrophic gastritis (EAG) was associated with high risk for gastric cancer, including synchronous and metachronous multiple gastric neoplasms.3 In the Eastern countries, including Korea and Japan, the prevalence of AG, as well as gastric cancer is higher than that in the Western countries.45 Therefore, it is important to evaluate the degree and extent of gastric mucosal atrophy during endoscopy screening to identify individuals at high risk for gastric cancer in high-risk countries. The endoscopic scoring system used to categorize the extent of AG has been proposed by Kimura and Takemoto6 in Japan since 1969. The system used the endoscopic atrophic border (EAB), marking the transition between non-atrophic normal mucosa and atrophic gastritis. EAB is regarded as the boundary between the pyloric and fundic glandular territories, which is endoscopically recognized by the differences in color and height of the gastric mucosa. The atrophic mucosal area is a grossly pale-yellowish color with transparent blood vessels, while normal mucosa is homogeneously reddish-colored and smooth. The close relationship between EAB and the location of gastric cancer was reported; the intestinal type of gastric cancer usually occurs on the distal side of the EAB, where severe atrophy was present.7 For an endoscopic classification system to be useful, it should be reproducible in practice, so that reliable outcomes can be obtained based on its application. However, little evidence is available regarding inter-observer agreement in endoscopic evaluation of AG. In the present study, we aimed to evaluate the effect of training on inter-observer agreement in the diagnosis and grading of EAG according to experience level.

METHODS

Study design and subjects

We conducted a prospective intervention study at the gastrointestinal department of the Seoul National University Hospital Healthcare System Gangnam Center in Korea from January 2015 to May 2016. This study consisted of three stages and entailed the following (Fig. 1): 1) pre-seminar for education about EAG classification; 2) intervention period, i.e., four times assessment with feedback; 3) follow-up period, i.e., two times assessment without feedback. We reviewed the endoscopic reports for two months before and after the training and compared the actual diagnostic rates of atrophic gastritis in clinical practice. Twelve board-certificated gastroenterologists were enrolled. They were divided into two groups according to the level of endoscopic experience as follows: six experienced endoscopists with over 10 years of experience and six less-experienced endoscopists with less than 10 years of experience. All upper gastrointestinal endoscopies were carried out with a conventional white-light videoendoscope (GIF-H260/H290 series endoscopes, Olympus Optical Co., Ltd., Tokyo, Japan) as a part of comprehensive health check-up program.

Fig. 1

Schedule diagram during the study period. Training was conducted in a systematic manner during 36 weeks of daily clinical practice. The training session consisted four times interventions and two follow-up assessments after intervention.

EAG = endoscopic atrophic gastritis.

Definition and endoscopic grading for gastric mucosal atrophy

The grade of gastric atrophy was estimated according to the location of the EAB, which is recognized by distinctive differences in visible capillary network, color, and height of the gastric mucosa in a non-overdistended stomach, as reported by Kimura and Takemoto6 and Liu et al.8 (Fig. 2). The extent of the EAG was categorized into six grades (C1 to O3) according to the Kimura-Takemoto (K-T) classification; the EAB is limited to the antrum in C1, the angle or lower corpus in C2, and the upper corpus in C3. The EAB is parallel to the vertical axis of stomach, and is on the lesser curvature in O1, the anterior and posterior wall in O2, and the greater curvature in O3. These six categories were grouped into two types: a closed type (C-type, C1–C3) and an open type (O-type, O1–O3).

Fig. 2

Representative illustration of EAB and K-T classification. The EAB is the boundary between the pyloric and fundic territories which is differences in visible capillary network, color, and height of the gastric mucosa in a non-overdistended stomach.

EAB = endoscopic atrophic border, K-T = Kimura-Takemoto.

Pre-intervention and training module

Before the training session started, a preliminary investigation was conducted to identify baseline diagnostic yield of EAG. Two investigators reviewed all the electronic endoscopic records and endoscopic images from January to February 2015. These investigators were board-certificated and had experienced more than a mean of 15,000 cases of upper endoscopy over a mean 6.5 years. They identified the rate of baseline diagnosis for EAG according to each endoscopist, and selected representative cases of EAG for a training module. A total of 66 endoscopy cases were randomly selected that consisted of 11 cases per grade (C1 to O3), who showed H. pylori serology positivity but were never treated with H. pylori eradication.

Subsequently, we held a pre-seminar to build conception of the K-T classification and to establish consensus on assessment for the extent and grade of EAG, thereby, to achieve a learning curve. All participating endoscopists gathered and shared opinions to determine the EAB using selected representative endoscopic images with various grades of EAG. After several discussions about grading EAG according to the K-T classification, the participating endoscopists built a consensus for determination of EAB and grading of EAG.

Intervention period for endoscopic assessment of atrophic gastritis

Training was conducted for 36 weeks in a systematic manner. All endoscopists recorded the grade of the EAG, according to the K-T classification, into the electronic endoscopic database for all screening endoscopy cases in daily clinical practice. During the intervention period, a total of four assessments with feedback was done every two weeks between March and April 2015. For the assessment, the investigators sent the selected cases to twelve endoscopists by email; 12-12-12-10 cases. Each case consisted of 6 photos: antrum, angle, lesser and greater curvatures of the lower body, greater curvature of the upper body, and cardia of the stomach. Each endoscopist received the cases, assessed the EAG grade (C1–O3), and returned the results by email (Supplementary Fig. 1). All endoscopists were blinded to any clinical data or results of histopathological examinations of the selected cases. The investigators collected the individual endoscopists' assessments and opened the final results to all participating endoscopists by email. Subsequently, the participants gathered and discussed the cases with the concurrent cases projected onto a screen; they reviewed the cases again and achieved a consensus on the grade of EAG in each case. The final grade of the EAG diagnosis was calculated based on the highest degree of consensus among endoscopists if there was a discrepancy of the EAG grade between the two investigators and the participant endoscopists.

Follow-up period and post-intervention

During the follow-up period, two assessments occurred without feedback 8 weeks and 28 weeks after the intervention period. The investigators sent the 10 selected cases for each follow-up and the participating endoscopists received their assessments in the same manner as the intervention. They collected data with no release of the results on EAG grade and no more discussion was performed between endoscopists. After the completion of the training and follow-up sessions, the investigators reviewed all the endoscopic records from April 2016 to May 2016 to evaluate the change of diagnostic yield for EAG between pre- and post-intervention periods. One endoscopist in the experienced group was excluded from diagnostic yield analysis before and after the intervention because the endoscopist did not perform endoscopy during this study period.

Statistical analysis

The inter-observer agreements were assessed using a weighted kappa value and its 95% confidence interval (CI) using SAS statistical software (SAS system for Windows, version 9.3; SAS Institute, Cary, NC, USA). A kappa value ≤ 0.20 was considered poor, 0.21–0.40 was fair, 0.41–0.60 was moderate, 0.61–0.80 was good, and > 0.80 was excellent.9 If the kappa value was greater than 0.4, an acceptable degree of concordance was considered to be present. We collected the data and analyzed the overall inter-observer agreements for each of 6-type (C1 to O3) and 2-type (C-type and O-type) K-M classification. Inter-observer agreement for 6- and 2-type categories were compared using the Wilcoxon signed rank test. Linear regression analyses were used to examine the training effect on inter-observer agreement during the intervention period. In addition, a sub-analysis was performed based on the experience level of the endoscopists. The post-intervention diagnostic yield for EAG was compared to the pre-intervention yields. Variability of diagnostic yields among the endoscopists in the pre- and post-intervention periods was calculated by the F-test. A two-tailed P value of < 0.05 was considered statistically significant.

Ethics statement

The study protocol conformed to the ethical guidelines of the 1975 Declaration of Helsinki and its subsequent revisions, and was approved by the Institutional Review Board of Seoul National University Hospital (approval No. 1504-021-662). Written informed consent was obtained from all participating endoscopists.

RESULTS

Table 1 shows the years of experience for the participating endoscopists. The median experience for endoscopists was 5 years (range, 3–9 years) for the less-experienced group, and 11.5 years (range, 11–15 years) for the experienced group. Across the study period, the inter-observer EAG grading agreement was consistently lower in the six-category analysis than that in the two-type analysis by the Wilcoxon signed rank test (P = 0.015) (Supplementary Fig. 2).

Table 1

Characteristics of the enrolled endoscopists

Endoscopists		No.	Median experience in endoscopy, yr
All		12	10 (3–15)
	Experienced	6	11.5 (11–15)
	Less-experienced	6	5 (3–9)

The overall inter-observer agreement significantly increased after the intervention period, i.e., four assessments with feedback every two weeks; the kappa value for EAG grading significantly increased from 0.27 to 0.67 in two-category analysis (coefficient beta = 11.36, P = 0.029). The kappa values increased in both groups; from 0.32 to 0.59 in the experienced group and 0.18 to 0.76 in the less-experienced group (Fig. 3). According to the extent of the EAG, the agreement rate increased in both the closed and open types (Table 2). Although grading the open type of EAG showed a tendency to increase inter-observer agreement rather than the closed type of EAG, it was not statistically significant (P = 0.058, data not shown).

Fig. 3

The kappa coefficient of reliability on grading of EAGa according to the level of experience.

EAG = endoscopic atrophic gastritis.

aEAG was sub-classified into two-type: closed type (C-type, C1 to C3) and open type (O-type, O1 to O3); bExperienced group was defined by their endoscopic experience over 10 years; cLess-experienced group was defined by their endoscopic experience less than 10 years. Kappa values (K) ≤ 0.20 denoted poor, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 good and > 0.80 excellent.

Table 2

Inter-observer agreements in EAG according to the level of endoscopic experience

Groups		Agreement rate, %
		Intervention period				Follow-up period
		I	II	III	IV	V	VI
Overall (n = 12)
	Close type (C1–C3)	24.2	51.3	33.3	66.4	51.9	64.1
	Open type (O1–O3)	32.0	63.6	59.1	73.0	71.2	63.6
Experienced groupa (n = 6)
	Close type (C1–C3)	28.6	56.4	37.0	56.0	41.0	74.6
	Open type (O1–O3)	36.5	71.1	64.9	69.2	59.3	73.2
Less-experienced groupb (n = 6)
	Close type (C1–C3)	15.7	41.5	27.7	75.9	64.0	71.2
	Open type (O1–O3)	23.0	53.2	51.4	75.9	82.2	72.0

EAG = endoscopic atrophic gastritis.

aExperienced group was defined by their endoscopic experience over 10 years; bLess-experienced group was defined by their endoscopic experience less than 10 years.

After the intervention period, two follow-up assessments were provided without feedback. A good level of inter-observer agreement was maintained during the follow-up period without additional education and feedback, regardless of the experience level. Overall kappa values were 0.59 at the first follow-up (at 8 weeks) and 0.69 at the last follow-up (at 28 weeks), as shown in Fig. 3. In the experienced group, kappa values were 0.54 at the first follow-up and 0.78 at the last follow-up. In the less-experienced group, kappa values were 0.68 at the first follow-up and 0.76 at the last follow-up. There were no significant differences in kappa coefficient values related to the level of the endoscopists' experience during the follow-up period.

A total of 1,152 patients underwent upper endoscopy screening and were reviewed for analysis of baseline diagnostic yield of EAG, and 496 (43.1%) patients showed EAG. The rate of EAG was 28.0% to 64.7%, depending on the endoscopists (Table 3). There was no difference in the diagnostic rate of EAG between the experienced group and less-experienced group (45.2% ± 5.5% vs. 41.3% ± 14.1%, P = 0.545). After the training session (i.e., intervention and follow-up periods), the diagnostic yield was reassessed (46.8%, range 39.3% to 60.9%) among 1,886 upper endoscopy screenings. In comparison with pre-investigation, discrepancy between the inter-observers decreased, but not statistically significant at post-intervention (ratio of variance = 0.31, P = 0.075). However, the less-experienced group showed significantly decreased inter-observer variation (46.6% ± 2.8%, ratio of variance = 0.04, P = 0.003).

Table 3

Diagnostic yield of EAG in each endoscopist at the pre- and post-intervention

Groups		Pre-intervention	Post-intervention	Ratio of variancea	P value
Overall (mean ± SD, %)		43.1 ± 10.7	46.8 ± 5.93	0.31	0.075
Experienced groupb		45.2 ± 5.5	47.1 ± 8.8	2.62	0.374
	A	38.2	50.3
	B	50.5	44.4
	C	40.8	40.6
	D	47.2	39.3
	E	49.5	60.9
Less-experienced groupc		41.3 ± 14.1	46.6 ± 2.8	0.04	0.003
	F	33.9	44.3
	G	51.9	43.2
	H	33.0	46.2
	I	28.0	48.1
	J	64.7	51.1
	K	36.1	46.8

EAG = endoscopic atrophic gastritis, SD = standard deviation.

aVariability of diagnostic yield was calculated by F-test. Ratio of variance showed the diagnostic variability in each endoscopist at the post-intervention compared with pre-intervention; bExperienced group was defined by their endoscopic experience over 10 years; cLess-experienced group was defined by their endoscopic experience less than 10 years.

DISCUSSION

In this study, inter-observer agreement for the diagnosis and grading of EAG improved after systematized training and remained stable after the follow-up period. Moreover, a good level of inter-observer agreement in EAG grading was sustained during follow-up without feedback in both the less-experienced and the experienced groups.

Patients are diagnosed with AG based on endoscopic findings, serum pepsinogen, and pathological analyses.10 Operative Link on Gastritis Assessment (OLGA), is the gold standard for diagnosis of AG.11 Further, it is also well known that pathological diagnosis through multiple biopsies is impossible for examining the entire gastric mucosa.12 Other non-invasive methods to diagnose patients with AG include serum pepsinogen, which is a marker of gastric mucosal status and can be used to identify mucosal atrophy.13 In a previous study, the authors reported that pepsinogen was not reliable in asymptomatic subjects for predicting AG.14 A significant correlation was also shown between EAG and histologic atrophic grade in the antrum and body.13 In particular, moderate-to-severe EAG was highly consistent with pathologic atrophic grades.151617 Thus, endoscopy may be advantageous because it can reveal the distribution and extent of atrophic gastritis, and is easy to apply in actual clinical practice.

However, the diagnostic criteria of EAG are not generally used in Western countries and a wide variation in the rate of EAG diagnoses was reported among endoscopists in a real-time clinical setting. The prevalence rate of EAG was reported about 50% in Western countries and up to 80% in Eastern Asia.18 The prevalence of EAG rates varies widely between populations and even within population groups.12 Not only ethnic disparity, but also inter-observer variation may affect the difference in prevalence and diagnostic rate of EAG.

Previous studies showed that autofluorescence imaging or magnifying imaging could be helpful for diagnosing patients with EAG.1219 Conventional white-light endoscopy has low sensitivity (48%) and high specificity (87%).20 However, autofluorescence imaging for EAG has high sensitivity (100%), but low specificity (63%), thus, autofluorescence imaging is highly likely to show false-positive results.12 In addition, less-experienced endoscopists are not familiar with these methods and hence, sufficient time is necessary to acquire the ability to distinguish between normal and atrophic mucosa.

A recent study reported an inter-observer agreement of H. pylori infection on endoscopic images improvement after two years of training.21 Appropriate education and repetitive training is required to increase inter-observer agreement for consistent endoscopic diagnoses.22 Well-defined and easily applicable endoscopic classification criteria are needed for EAG diagnoses. Nevertheless, there have been no interventional studies designed to increase inter-observer agreement and reproducibility in EAG diagnoses. To the best of our knowledge, this is the first attempt to increase inter-observer agreement using structured learning with feedback for EAG.

At the beginning of this study, the level of overall inter-observer agreement for EAG was fair. After four interventions, the inter-observer agreement significantly increased in both the experienced and less-experienced groups. Our results suggested that systematic training and education for endoscopists could be effective to increase inter-observer agreement, regardless of endoscopic experience. Even though the endoscopists did not receive feedback during the follow-up period, a good level of inter-observer agreement (κ = 0.66) was sustained for more than 28 weeks irrespective of the experience level. The effectiveness of education may continue without additional intervention.

To investigate the effect in real practice, the actual diagnostic rate for EAG before and after the intervention was analyzed. Before and after the intervention, diagnostic discrepancy among inter-observer agreement decreased, but was not statistically significant (43.1% ± 10.7% in pre-investigation versus 46.8% ± 5.93% in post-investigation). However, the wide variation significantly reduced especially in the less-experienced group (ratio of variance = 0.04, P = 0.003). We could postulate that teaching a standardized system to diagnose EAG influenced the diagnosis and grading of EAG in actual practice, and thereby, could help with tailored screening and surveillance. Education might be more effective to the less-experienced group because they easily accept new endoscopic criteria and feedback.

We acknowledge that this study has several limitations. First, the diagnosis of EAG cases, which we used as a training module, was not supported by serum pepsinogen or pathology.10 Although the representative cases of EAG were selected with the consensus reading of two investigators, EAG diagnosis is liable to subjective and wide inter-observer variability. Therefore, the result of this study should be cautiously interpreted in that the final diagnosis of EAG is uncertain. Second, an expert endoscopist was excluded from diagnostic yield analysis before and after the intervention because she did not perform endoscopy during this study period. This may affect the results of the experienced group, which showed no change in the discrepancy of diagnoses before and after the intervention versus the less-experienced group. Third, we assessed gastric atrophy using conventional white-light videoendoscopy. Previous studies have shown that autofluorescence imaging or magnifying imaging could be helpful in evaluating the extent of gastric atrophy. Fourth, we assessed EAG grade using still photos, not video images. Therefore, the whole stomach's atrophic status was not observed. Fifth, as the study was conducted at a single health care center and included only twelve endoscopists, the sample size was relatively small. Additionally, this study did not include any beginners.

In conclusion, this study revealed that the inter-observer agreement and diagnostic yield for EAG improved though intervention, regardless of the endoscopists' experience. Educational intervention could help to decrease discordance in the diagnosis of EAG and improve communication between endoscopists. The present results may help extract the gastric cancer risk and provide early detection of precursor atrophy. Further, a large-scaled multicenter study is needed to evaluate the effect of this educational intervention.