A Systematic Review of Interpathologist Agreement in Histologic Classification of Lupus Nephritis.

INTRODUCTION: Lupus nephritis (LN) is one of the most severe manifestations of systemic lupus erythematosus (SLE), resulting in increased morbidity and mortality. The gold standard for diagnosis of LN is a renal biopsy. Considering the importance of the biopsy in determining long-term prognostication and treatment decisions, it is crucial to assess renal histopathology with utmost accuracy and precision. This review represents a systematic search of published literature to estimate the degree of interpathologist reproducibility in current assessment of LN.
METHODS: Using the PubMed and Google Scholar search engines, studies analyzing the agreement of 4 or more pathologists assessing LN slides using the ISN/Renal Pathology Society (RPS) classification, activity index, and chronicity index were selected for analysis in this systematic review.
RESULTS: In reviewing 6 qualifying studies (those analyzing the agreement of 4 or more pathologists using the ISN/RPS classification, activity index, and chronicity index) for the assignment of ISN/RPS class was 0.325 (interquartile range [IQR] 0.2405-0.425), which is "poor." The median interpathologist concordance values for the assigned activity index and chronicity index were "moderate": 0.52 (IQR 0.51-0.69) and 0.49 (IQR 0.36-0.58), respectively.
CONCLUSION: Thus, the current scoring using the ISN/RPS classification system and activity and chronicity indices for LN exhibits poor interpathologist agreement, which limits its use in clinical practice. Given that this can have severe repercussions on a patient's treatment and prognosis, efforts to update pathology assessment guidelines, objectively measurable biomarkers, and deep learning approaches are strongly warranted.

LN is one of the most severe manifestations of SLE, resulting in increased morbidity and mortality. Up to 60% of adults and 80% of children with SLE develop LN, with 10%–30% progressing to end-stage renal disease (ESRD) within 15 years despite aggressive treatment.2, 3, 4 The gold standard for diagnosis of LN is a renal biopsy. Histologic parameters, as defined by the ISN/RPS 2003 classification, remain the best predictors of ESRD and are utilized to assign the severity6, 7, 8, 9, 10, 11, 12, 13, 14, 15 of LN to classes I–VI. The renal biopsy is also scored for its activity index (AI) based on 6 specific histologic features, and its chronicity index (CI) based on 4 specific histologic features.16, 17 These indices, along with the assigned LN class, provide prognostic value and treatment guidance. In general, ISN/RPS class III or class IV LN, high renal AI (score >7), high renal CI (score >4), and the presence of subepithelial and subendothelial deposits are associated with poor renal survival.6, 7, 8, 9, 10, 11, 12, 13, 14, 15

According to current guidelines, a diagnosis of either class III or class IV LN is an indication for immunosuppressive therapy.18, 19, 20 Additionally, whereas a high chronicity index is associated with refractoriness to aggressive therapy, lesions with a high activity index are potentially reversible.7, 21, 22, 23 As suggested by these observations, histologic features play an important role in guiding the intensity of therapy.8, 23, 24, 25 It has also been reported that early, accurate detection and prompt treatment can significantly reduce morbidity and mortality in LN.25, 26, 27, 28, 29

Considering the ISN/RPS classification system’s role in determining long-term prognostication and treatment decisions, it is crucial to assess renal histopathology in LN with utmost accuracy and precision. This also extends to the scoring of renal pathology, AI, and CI. Certain histologic parameters that constitute the AI, such as cellular crescents and fibrinoid necrosis, have been documented to correlate with renal failure and ESRD7, 10; as a result, these features are weighted twice in the calculation of a patient’s AI score. Additionally, every individual chronicity index feature (glomerular sclerosis, fibrous crescents, interstitial fibrosis, and tubular atrophy) has been associated with an increased risk of ESRD, and renal failure rates are significantly elevated in patients with high CI scores.7, 10, 30, 31 Given that an improper assignment of LN class, AI score, and CI score may have significant prognostic and therapeutic repercussions, these metrics need to be assessed with high accuracy.

Despite being the gold standard, histologic assessment of LN using a renal biopsy is fraught with several pitfalls and high interpathologist variation.30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52 Thus far, 6 interobserver renal pathology assessment studies have been reported in which 4 or more pathologists assessed a test set of LN slides.30, 31, 32, 33, 34, 35 The metrics used to assess interpathologist correlation/concordance, intraclass correlation coefficient (ICC), and kappa factor (κ)-value, are comparable; a concordance value of 0 represents no agreement, and 1 represents complete agreement between the pathologists. In these 6 studies,30, 31, 32, 33, 34, 35 the ICCs or κ-values for most histologic parameters of LN fell below 50%. The ICC or κ-values for fibrinoid necrosis and cellular crescents, both of which are indicators of poor patient prognosis, were30, 31, 32, 33, 34, 35 below 0.6. Given that both of these parameters are weighted doubly in the calculation of the AI score, the lack of interpathologist agreement is compounded. The concordance values for the CI, which is also an indicator of poor prognosis, were30, 31, 32, 33, 34, 35 as low as 0.35. Given how important histologic assessment of LN is for guiding treatment, this is clearly unacceptable.

The purpose of this review is to conduct a systematic search of published literature to estimate the degree of interpathologist reproducibility in using the current ISN/RPS classification for LN, as well as the renal pathology activity and chronicity index metrics.

Methods

PubMed and Google Scholar were used to search for relevant literature between June and August 2018. The search criteria used were [systemic lupus erythematosus OR lupus] AND [variability OR concordance OR reproducibility] AND [nephritis OR renal OR kidney]. Only studies analyzing the agreement of 4 or more pathologists assessing LN slides using the ISN/RPS classification, activity index, and chronicity index are included in this systematic review (see Supplementary Figure S1 for a flow diagram of the study selection process). In all studies, the degree of concordance between the pathologists was measured using the ICC or the closely related value κ. The ICC most resembles the Pearson correlation coefficient. With ICC, the data are centered and scaled using a pooled mean and SD, whereas with the Pearson correlation, each variable is centered and scaled by its own mean and SD. ICCs and κ-values are commonly used for the assessment of consistency or reproducibility of quantitative measurements made by different observers measuring the same study subject(s). For each selected study, the following data were independently extracted: number of pathologists, number of LN slides, ICC or κ-values for the ISN/RPS classification, ICC or κ-values for the activity index and associated histologic features, and ICC or κ-values for the chronicity index and associated histologic features.

Results

Of the 308 studies screened, 9 studies discussing interpathologist concordance were selected. Six studies met the specified criteria of assessing the interpathologist concordance among 4 or more pathologists.30, 31, 32, 33, 34, 35 Not included were studies with 3 or fewer pathologists,36, 37, 38 as they did not meet the predetermined criteria. Table 1 details the evaluation of these 6 studies, including for each study the number of pathologists, the number of LN slides, ICC or κ-values for the ISN/RPS classification, ICC or κ-values for the activity index and associated histologic features, and ICC or κ-values for the chronicity index and associated histologic features. A summary of this table is displayed in Figure 1. The concordance level of the assigned LN class based on the 2003 ISN/RPS Classification (or the previous consensus criteria if conducted before 2004) was analyzed in 4 of the 6 studies,32, 33, 34 whereas 5 of the 6 studies30, 31, 32, 33, 35 examined the concordance level of the assigned activity and chronicity indices as scored by 4 or more pathologists. Two studies analyzed only renal biopsy slides depicting proliferative forms of LN,31, 33 and one study analyzed only pediatric LN patients. Two of the 6 studies included renal biopsy slides of patients without LN.31, 34 The biopsy interpreters were nephropathologists in 4 studies,30, 32, 33, 34 and in 2 studies,31, 35 they were general pathologists experienced in reading renal biopsies.

Table 1

Extracted data from 6 articles to estimate the degree of interpathologist concordance in LN histology assessment

	Furness and Taub32 (2006)	Grootscholten et al.33 (2008)	Oni et al.35 (2017)	Schwartz et al.30 (1993)	Wernick et al.31 (1993)	Wilhelmus et al.34 (2015)
No. of pathologists	31 nephropathologists	5 nephropathologists (3 evaluated each biopsy)	4 histopathologists	4 nephropathologists	5 pathologists experienced in reading renal biopsies	34 nephropathologists
No. of LN slides	20 renal biopsies from SLE	87 biopsies with proliferative LN	55 slides from pediatric LN	83 slides from LN	25 slides with proliferative LN	30 biopsies with class III or class IV LN
ISN/RPS class concordance	0.53 (κ)	0.182 (ICC)	0.26 ± 0.12 (κ)			0.39 (ICC; for class III/IV vs. class I/II/V)
Activity index concordance	0.39 (κ)	0.72 (ICC)	0.69 ± 0.06 (ICC)	0.44–0.63 (ICC; between 2 pathologists)	0.51 (ICC)
Endocapillary hypercellularity		0.65 (ICC)	0.62 ± 0.07 (ICC)		0.10 (ICC)	0.46 (ICC)
Leukocyte infiltration			0.30 ± 0.13 (ICC)		0.27 (ICC)	0.32 (ICC)
Subendothelial hyaline deposits			0.39 ± 0.09 (ICC)		0.53 (ICC)
Fibrinoid necrosis/karyorrhexis		0.48 (ICC)	0.45 ± 0.09 (ICC)		0.26 (ICC)
Cellular crescents			0.55 ± 0.07 (ICC)		0.50 (ICC)
Interstitial inflammation		0.51 (ICC)	0.83 ± 0.05 (ICC)
Chronicity index concordance	0.35 (κ)	0.49 (ICC)	0.36 ± 0.09 (ICC)	0.60–0.76 (ICC; between 2 pathologists)	0.58 (ICC)
Glomerular sclerosis		0.82 (ICC)	0.40 ± 0.09 (ICC)		0.34 (ICC)
Fibrous crescents			0.25 ± 0.09 (ICC)		0.58 (ICC)
Interstitial fibrosis		0.42 (ICC)	0.10 ± 0.10 (ICC)		0.44 (ICC)
Tubular atrophy		0.51 (ICC)	0.07 ± 0.10 (ICC)		0.44 (ICC)

ICC, intraclass correlation coefficient; LN, lupus nephritis; RPS, Renal Pathology Society; SLE, systemic lupus erythematosus.

An ICC or κ-value of 0 indicates no agreement, and a value of 1 indicates complete agreement.

The following additional metrics were also computed in isolated studies: wire loops: 0.50 ICC (Grootscholten et al. [2008]), 0.35 ICC (Wilhelmus et al. [2015]); swelling of endothelial cells: 0.46 ICC (Wilhelmus et al. [2015]); extracapillary proliferation: 0.64 ICC (Grootscholten et al. [2008]), 0.57 ICC (Wilhelmus et al. [2015]); mononuclear infiltration: 0.46 ICC (Wernick et al. [1993]); tubular cell necrosis: 0.16 ± 0.09 ICC (Oni et al. [2017]).

Figure 1

Interpathologist concordance of lupus nephritis (LN) histology assessment in 6 studies. ICC, intraclass correlation coefficient.

Although the biopsy interpreters were all trained pathologists, and most were nephropathologists, their concordance values (as assessed using ICC or κ-value) in assessing LN were generally modest. The median interpathologist concordance values for the reported pathology indices and component histologic features detailed in Figure 1 ranged from 0.30 to 0.83. The median interpathologist concordance value for the assignment of ISN/RPS class was 0.325 (IQR 0.2405–0.425), which is “poor.” The median interpathologist concordance values for the assigned AI and CI were “moderate”: 0.52 (IQR 0.51–0.69) and 0.49 (IQR 0.36–0.58), respectively.

In these studies, an ICC or κ-value of <0.4, 0.4–0.6, 0.6–0.8, or >0.8 is considered to reflect poor, moderate, good, or excellent agreement, respectively.32, 33, 34 The histologic features with poor agreement (based on median concordance values) were leukocyte infiltration and glomerular sclerosis. Assessment of a majority of histologic features exhibited “moderate” interpathologist agreement (based on median concordance values): these include fibrous crescents, interstitial fibrosis, tubular atrophy, fibrinoid necrosis/karyorrhexis, subendothelial hyaline deposits, cellular crescents, and endo-capillary hypercellularity. No feature exhibited a “good” median interpathologist concordance value, whereas the only feature with “excellent” interpathologist agreement was interstitial inflammation (Table 1). However, only one study calculated an ICC or κ-value for interstitial inflammation, and this needs to be investigated in further interpathologist comparison studies in the future.

Conclusion

Accurate assessment of LN renal pathology, with interpathologist concordance values exceeding 0.9 would be ideal if clinical decisions are guided by pathology, as recently discussed. However, the present review shows that the interpathologist agreement in assessing LN ISN/RPS class, AI, and CI is “poor” to “moderate” overall. Even though all biopsy interpreters were trained pathologists, and most were expert nephrologists, all but one histologic feature (interstitial inflammation) exhibited a median ICC or κ-value below 0.6.

The renal pathology metrics assessed with the poorest concordance values (i.e., ISN/RPS class, leukocyte infiltration, and glomerular sclerosis) have important implications for patient prognosis and treatment plans. For example, an incorrect classification of the renal biopsy into either class III or IV instead of class I or II for LN will subject the patient to unnecessary immunosuppressive therapy. This can have detrimental effects, including an increased risk for infections, cancers, and nephrotoxicity, resulting in increased morbidity and mortality.53, 54 As another example, patients with glomerular sclerosis have increased risk for ESRD,7, 10 and the failure to recognize this feature in renal biopsies may delay appropriate treatment.

Some authors have hypothesized potential causes for low interpathologist agreement. Wernick et al. cited the level of experience of the pathologist, which is supported by the finding by Wilhemus et al. that highly experienced nephropathologists exhibit higher interpathologist agreement compared with less-experienced pathologists. This suggests that interpathologist agreement may improve by better educating and training pathologists and nephropathologists. Another reason for low interpathologist agreement may be the classification system itself.16, 34 For example, multiple publications have claimed that the definition of “endocapillary proliferation” is unclear,16, 34 and therefore, different pathologists may score the renal biopsy differently for that feature. Furthermore, Wilhelmus et al. have suggested that more than one pathologist should review the same renal biopsy results before clinical decisions are made. Indeed, efforts are in progress to significantly improve the classification criteria for LN, as very recently recommended.

In interpreting these studies, a couple of caveats should be considered. Two studies reviewed30, 31 were conducted in 1993, which is prior to the publication of the ISN/RPS classification in 2003. As a result, ISN/RPS class concordance values are not reported for these studies. In 3 of the studies assessed in this review (Schwartz et al., Furness and Taub, Wilhelmus et al.), the concordance values for the individual histologic features that constitute AI and CI are not reported, making it difficult to understand which lesions contributed to the overall discordance noted in the AI and CI scores. As a result, the median interpathologist concordance value is determined in only 2 studies. Additionally, in the 3 publications by Wernick et al., Furness and Taub, and Wilhelmus et al., the number of LN slides studied fall on the lower side (25, 20, and 30 slides, respectively) to allow for definitive conclusions from these studies. Lastly, the studies by Wernick et al. and Oni et al. calculated the concordance score among general pathologists, who may have less expertise in evaluating LN slides than trained nephropathologists. These shortcomings in some of the included studies call for more consistent, systematic assessment of interpathologist concordance in ISN/RPS classification, the activity index, and its component histologic features, and the chronicity index and its associated histologic features, involving significantly larger datasets.

Though the pathologist assessment of a renal biopsy is considered the gold standard diagnostic procedure for LN, it is highly subjective, and the result depends on the training of the interpreter. Our results support the necessity of central pathology reviews involving highly experienced nephropathologists analyzing digitized slides for the diagnosis of LN. In addition, the use of machine learning algorithms could facilitate more accurate diagnosis of the LN class, AI score, and CI score. Previous studies have shown that machine learning can be utilized for accurate prediction of chronic kidney damage in SLE patients, and renal survival in patients with chronic kidney disease, if the neural network is trained with a large and diverse set of renal biopsy images. To place this in perspective, we believe that this kind of discordance is not limited to just the morphologic classification of LN but is ubiquitous in any morphologic classification, ranging from the Banff criteria for organ transplant rejection to other diseases such as atypical/dysplastic lesions of the breast, the gastrointestinal tract, and other organs. Clearly, there may be exceptions to this statement. For example, careful and consistent reviews by expert pathologists have been effective and reproducible in assessing renal pathology in the VALIGA study for IgA nephropathy and the assessment of vasculitis-associated renal pathology by the RENHIS (renal histology) group in the European Vasculitis Study Group (EUVAS).

In summary, the current scoring of the ISN/RPS classification system and the activity and chronicity indices for LN exhibit poor interpathologist agreement, which limits their use in clinical practice. Given that low interpathologist agreement can have severe repercussions regarding a patient’s treatment and prognosis, efforts to update pathology assessment guidelines, objectively measurable biomarkers, and deep learning approaches are strongly warranted.

Disclosure

All the authors declared no competing interests.