Impact of Consensus Definitions on Identification of Glomerular Lesions by Light and Electron Microscopy.

INTRODUCTION: In 2020, a working group of 13 renal pathologists published consensus definitions for 47 individual glomerular lesions found on light microscopy (LM) and 47 glomerular lesions and 9 normal structures found on electron microscopy (EM).
METHODS: To test the impact of these definitions on identification of these lesions and structures, 2 surveys were circulated to all members of the Renal Pathology Society (RPS), each having 32 images (19 LM, 13 EM) and accompanying questions with 5 multiple-choice answers, one being the consensus choice of the working group. The first survey (survey 1 [S1]), answered by 297 RPS members, was sent in September 2020, before publication of the consensus definitions. The second (survey 2 [S2]), with images of the same lesions and structures (but not the same images) and the same questions and multiple choices in different order, was sent in April 2020, 5 months after the publication of the definitions.
RESULTS: S2 was taken by 181 RPS members; 64% also took S1 and 61% reported having read the definitions paper (def. paper). Mean agreement with the consensus answers increased modestly between the 2 surveys (65.2% vs. 72.0%, P = 0.097); the increase was greater and significant when only respondents to S2 who read the def. paper were considered (65.2% vs. 74.8%, P = 0.026). Furthermore, in S2 agreement with consensus answers was greater among respondents who read this paper versus those who did not (66.9% vs. 74.8%, P < 0.0001).
CONCLUSIONS: Publication of the consensus definitions modestly improved interobserver agreement in identification of glomerular lesions.

In the past 2 decades, there has emerged the development of new and updated histologic scoring systems for multiple glomerular diseases1, 2, 3, 4, 5, 6, 7, 8 and consortia for the clinicopathologic and pathophysiological study of glomerular diseases.10, 11, 12, 9 Although each consortium has specific criteria for enrollment of patients based on clearly stated clinical and pathologic features, an important limitation faced by different consortia and others in applying pathologic scoring systems has been a lack of uniformity in the definitions for the lesions on which the scoring systems are based. Furthermore, the use of different definitions and thresholds for individual lesions by pathologists at different centers would be expected to contribute to higher levels of interobserver variability both in overall diagnosis and in applying any histologic classification schema in which all included lesions are not specifically defined. In response to this, the RPS charged a working group of 13 pathologists to review and evaluate the published definitions for key glomerular lesions by LM and EM and to ultimately develop consensus definitions for each that can be applied across the full spectrum of glomerular diseases in both research and clinical settings. This effort resulted in the publication of definitions for 47 individual glomerular lesions found on LM and 47 glomerular lesions and 9 normal structures found on EM.

To evaluate whether the availability of these consensus definitions resulted in an improvement in the ability of RPS members to identify glomerular lesions and structures encountered in their routine renal biopsy practices, we distributed 2 surveys to the >700 members of the RPS on 6 continents (excluding the committee members), each consisting of a set of 32 images (19 LM, 13 EM) with accompanying multiple-choice questions pertaining to the lesion(s) found in each. S1 was sent approximately 2 months before the publication of the consensus definitions and S2 approximately 7 months after. The same lesions and structures (but not the same images) and the same multiple-choice questions and answers, albeit in a different order, were included in both surveys. This paper summarizes and compares the results of these surveys.

Methods

Survey Development

After submission of the def. paper for publication, the chair of the working group (MH) began assembling LM and EM images collected from the working group members, excluding those chosen as figures in the latter paper, for use in the first of 2 planned surveys to be sent to the full RPS members. A total of 30 images were initially selected and multiple-choice questions written for each; these were then circulated to the working group members. Some of the initial questions written by the chair had >1 correct choice among the answers with one answer such as “all of the above” or “a and c” to account for this; however, most of the working group members preferred only a single correct choice for each question and the questions and answers were rewritten accordingly. The working group members also submitted additional images, some to replace images initially included that were felt to be ambiguous and others accompanied by multiple-choice questions to be added to the survey. Modifications to some questions were also suggested. The working group chair then modified and finalized the survey images and questions, the result being a survey consisting of a total of 32 images (19 LM, 13 EM), each with an accompanying multiple-choice question and 5 choices for answers. This version of the survey was again circulated to the working group members with requests for answers to each question and comments where the correct answer was not felt to be clear. This resulted in a final set of modifications to the survey that was sent to the working group with consensus answers from the previous version. The group approved this final version and consensus answers.

The survey was then sent to the information technology consultant to the RPS (RS), together with instructions on completing the survey and additional questions asking the respondent’s field of specialization (pathology, nephrology, basic science, other), country of practice, and whether their center performed EM on all or most native kidney biopsies. The images and questions were then loaded into Survey Monkey and distributed to the full RPS membership in early September 2020. The RPS members were given 4 weeks to complete the survey.

After the 4-week period, the responses to each question were compiled by the RPS information technology consultant and send to the working group chair.

In February 2021, 2 months after the publication of the def. paper, the working group chair once again solicited new images from the group members of the same 32 lesions/structures represented in the initial survey (S1). The chair then selected 1 image representing each of the 32 lesions/structures that was felt to closely but not exactly replicate the corresponding image from S1, and these were circulated to the working group members. This resulted in the replacement of 3 images with others of the same lesion that were felt to more closely resemble the S1 image. The working group chair then incorporated these 32 images into a new survey (S2), using the same questions and answer choices as in S1 but changing the order of the questions and of the multiple-choice answers for each.

The information technology consultant then circulated S2 to the full RPS membership in early April 2021, with 2 additional questions asking each respondent if they completed S1 and if they had read the November 2020 def. paper. Again, the RPS members were given 4 weeks to complete the survey, after which time, the information technology consultant compiled the results and sent these to the working group chair.

Analysis of Survey Results

A total of 297 RPS members completed S1 and 181 completed S2. The working group chair entered the complete results of each survey (both complete sets of responses and analysis of subgroups for S2 including respondents who did (S1—yes) and did not (S1—no) complete S1 and those who did (def. paper—yes) and did not (def. paper—no) read the def. paper) for statistical analysis. Comparisons of agreement of respondents with the consensus answer for each question were done by paired analysis for S1 versus S2, S1 versus S2 (def. paper—yes), S2 (S1—yes) versus S2 (S1—no), S2 (def. paper—yes) versus S2 (def. paper—no), and subgroups of the latter two (S1—yes and S1—no). Results of the paired analyses were analyzed by t test for paired samples; very similar results (with no difference in significance) were also found using Wilcoxon’s rank-sum test. Comparisons between summary data for S1 versus S2 (field of specialization, region of practice) were done using Fisher’s exact text. All tests were 2 tailed, and P < 0.05 was considered statistically significant. SAS version 9.4 (SAS Institute, Cary, NC) was used for statistical calculations.

Results

The first survey was answered by 297 RPS members and the second by 181. Table 1 summarizes the characteristics of the respondents to each survey; these did not differ significantly with respect to medical specialty, geographic distribution, and the fraction of respondents from centers routinely performing EM on native kidney biopsies. Nearly two-thirds of the respondents to S2 completed both surveys, and 61% of the S2 respondents reported reading the November 2020 def. paper before completing this survey. Among the 111 respondents to S2 who read the def. paper, 77 (69.4%) took S1 compared with 34 (52.3%) who did not read this paper, although this difference did not reach statistical significance (P = 0.08 by Fisher’s exact test).

Table 1

Characteristics of survey respondents

Characteristics	Survey 1	Survey 2	P valuea
Number (%) of respondents	297	181	0.076
Pathologist	261 (88)	170 (94)
Nephrologist	26 (9)	7 (4)
Other/no response	10 (3)	4 (2)
Geographic distribution			0.88
US and Canada	114 (38)	81 (45)
Western Europe including UK	56 (19)	29 (16)
Eastern Europe	19 (6)	11 (6)
East Asia	16 (5)	11 (6)
India/Pakistan/Bangladesh	21 (7)	16 (9)
Latin America and Caribbean	19 (6)	16 (9)
Middle East and Africa	26 (9)	15 (8)
Australia and New Zealand	5 (2)	2 (1)
No answer	21 (7)	0
Perform EM on all/most biopsies			0.68
Yes	204 (69)	134 (74)
No	74 (25)	41 (23)
Other response	19 (6)	6 (3)
Completed survey 1
Yes		116 (64)
No		65 (36)
Read 2020 definitions paper
Yes		111 (61)
No		70 (39)

EM, electron microscopy; UK, United Kingdom; US, United States.

By χ2 test.

Table 2 lists the 32 lesions and structures depicted in the surveys in descending order based on the percent agreement with the consensus diagnosis in S1; this same order is used for the horizontal axes of the figures. Figure 1 compares the percent agreement of the respondents with the consensus answer of the working group members for each lesion or structure in S1 versus S2. Overall agreement was modestly better in S2 although this did not quite reach statistical significance by paired analysis; the mean (±SD) level of agreement for the 32 images was 65.2 ± 21.6% in S1 versus 72.0 ± 17.2% in S2, P = 0.097 by paired t test. Similarly, modest and not statistically significant differences between S2 and S1 by paired analysis were found when considering only the 19 LM images (P = 0.24; means 69.0% ± 19.5% and 61.6% ± 25.5%, respectively) and the 13 EM images (P = 0.17; means 76.3% ± 12.8% and 70.6% ± 13.3%, respectively). Within each survey, there was no significant difference between agreement with the consensus answers for LM versus EM images (P = 0.40 and P = 0.31 for S1 and S2, respectively, by Wilcoxon rank-sum test). Nevertheless, when just those 111 respondents to S2 who read the def. paper were considered, agreement with the consensus answers improved (Figure 1; mean 74.8% ± 17.0%) and the difference in agreement between these results and those of S1 became significant (P = 0.026 by paired t test).

Table 2

Lesions and structures in the 32 survey images

Image
1. Capsular hyaline drop—LM (S1 96.4%, S2 97.6%)
2. Wire loops and hyaline pseudothrombi—LM (S1 96.0%, S2 68.9%)
3. Lamellation of the lamina densa—EM (S1 94.4%, S2 67.1%)
4. Mesangial and segmental endocapillary hypercellularity—LM (S1 90.8%, S2 50.6%)
5. Mesangial matrix expansion—LM (S1 89.1%, S2 67.1%)
6. Electron-lucent intramembranous immune deposits—EM (S1 86.0%, S2 94.4%)
7. Glomerular basement membrane duplication—LM (S1 85.6%. S2 92.0%)
8. An adhesion—LM (S1 82.3%, S2 75.5%)
9. Fibrillary deposits—EM (S1 82.2%, S2 77.9%)
10. An immature glomerulus—LM (S1 78.8%, S2 73.6%)
11. Endothelial honeycombing–EM (S1 75.6%, S2 80.7%)
12. Subendothelial widening—EM (S1 75.2%, S2 80.9%)
13. Elongated intramembranous deposits—EM (S1 75.0%, S2 86.9%)
14. Foot process effacement with cytoskeletal condensation—EM (S1 74.5%, S2 76.1%)
15. Ischemic-type capillary collapse—LM (S1 73.5%, S2 77.9%)
16. Microtubular deposits—EM (S1 70.2%, S2 86.2%)
17. Lipoprotein thrombi—LM (S1 68.9%, S2 82.2%)
18. Mesangial hypercellularity and a capillary microaneurysm—LM (S1 66.8%, S2 33.3%)
19. A pseudocrescent—LM (S1 64.4%, S2 28.1%)
20. Mesangial interposition—EM (S1 64.2%, S2 86.7%)
21. Endotheliosis—EM (S1 61.3%, S2 77.1%)
22. Subepithelial remodeling with intact lamina densa—EM (S1 58.9%, S2 45.2%)
23. Endocapillary hypercellularity and a cellular crescent—LM (S1 52.2%, S2 49.4%)
24. Mesangial “waist” deposit—EM (S1 51.1%, S2 64.1%)
25. Segmental sclerosis and hyalinosis—LM (S1 49.4%, S2 93.8%)
26. Glomerular basement membrane rupture—EM (S1 49.1%, S2 68.2%)
27. Mesangiolysis—LM (S1 49.0%, S2 61.6%)
28. Fibrinoid necrosis—LM (S1 47.2%, S2 92.7%)
29. Glomerular basement membrane lucencies (craters)—LM (S1 35.3%, S2 86.6%)
30. Membranoproliferative pattern—LM (S1 35.3%, S2 50.0%)
31. Segmental endocapillary hypercellularity—LM (S1 23.2%, S2 58.5%)
32. An intracapillary thrombus with GBM duplication—LM (S1 21.8%, S2 35.2%)

EM, electron microscopy; GBM, glomerular basement membrane; LM, light microscopy; S1, survey 1; S2, survey 2.

Number in parentheses is percent agreement with the consensus diagnosis for S1 and S2. Diagnosis numbers (1–32) are listed based on the % agreement with the consensus diagnosis in S1, in decreasing order, and correspond to the numbers on the horizontal axes in Figures 1 and 2.

Figure 1

Comparison of findings of S1 versus S2. The horizontal axis lists the 32 lesions and structures depicted in the surveys in descending order based on the percent agreement with the consensus diagnosis in S1. The lines represent the percent of all respondents (n = 297 for S1, n = 181 for S2, and n = 111 for S2 respondents who read the def. paper) agreeing with the consensus answer for each of the 32 images, with the means of the 32 values represented by the points at the far right. P values, determined by paired t tests, were 0.097 for S1 versus S2 and 0.026 for S1 versus S2 respondents who read the def. paper. def. paper, definitions paper; S1, survey 1; S2, survey 2.

Figure 2

Comparison of findings within S2 in respondents who did (n = 114) or did not (n = 67) read the def. paper. The order in which the 32 images depicted in the survey are plotted is the same as used in Figure 1. (a) For all respondents to S2, comparison of the fractions agreeing with the consensus answer for each of the 32 images. (b) For respondents to S2 who completed S1, comparison of the fractions agreeing with the consensus answer for all 32 questions among those who did (n = 77) and did not (n = 39) read the def. paper. (c) For respondents to S2 who did not complete S1, comparison of the fractions agreeing with the consensus answer for all 32 questions among those who did (n = 34) and did not (n = 31) read the def. paper. For each figure, the means of the 32 values for each survey represented by the points at the far right and the P value found were determined by t test for paired samples (n = 32). def. paper, definitions paper; S1, survey 1; S2, survey 2.

Paired analysis of percent agreement with the 32 consensus answers in S2 was significantly better among those respondents who read the November 2020 def. paper than in those who did not (Figure 2a; P < 0.0001 by paired t test, means 74.8% ± 17.0% and 66.9% ± 18.7%, respectively). In addition, agreement with the consensus answers was greater among respondents who read the paper for 29 of the 32 survey images. Nevertheless, as noted previously, the fraction of respondents who read this paper that took S1 was greater than that of respondents not reading the paper, and percent agreement with the consensus answers in S2 was also significantly better among respondents who completed S1 than in those who did not (P < 0.001 by paired t test, means 73.9% ± 16.8% and 68.9% ± 18.7%, respectively). We therefore evaluated the impact of whether or not respondents read the def. paper separately in those respondents who did and did not complete S1. By paired analysis, percent agreement with consensus answers in S2 was significantly greater in respondents who read the def. paper, whether they did (P < 0.0001 by paired t test, means 76.3% ± 16.2% and 68.6% ± 19.3%, respectively) or did not (P < 0.0001 by paired t test, means 72.5% ± 20.7% and 64.8% ± 19.6%, respectively) complete S1 (Figure 2b and c).

Discussion

Kidney biopsy, with examination of tissue by LM, immunofluorescence/immunohistochemistry, and EM, is an essential diagnostic tool in the field of nephrology, especially pertaining to glomerular diseases. Furthermore, integration of biopsy findings with clinical data, such as renal function, severity of proteinuria, and blood pressure, often adds prognostic information beyond that which can be obtained from clinical parameters alone, as is well documented for IgA nephropathy,, and lupus nephritis., Specific ultrastructural changes may also be markers for molecular phenotypes; for example, Royal et al. revealed that more than 1100 genes were differentially expressed between podocytopathies with and without significant endothelial cell damage by EM. Nevertheless, the value of morphologic changes in predicting clinical outcomes and guiding therapy is directly related to our ability to accurately and reproducibly identify such changes, and the current literature indicates that such reproducibility is often lacking.,,19, 2, 20, 21, 22 A contributing factor to this lack of reproducibility has been an absence of uniformity in definitions for individual glomerular lesions, as evidenced from our review of such definitions listed in the papers detailing the scoring systems for different renal diseases and glomerular disease consortia1, 10, 11, 12, 2, 3, 4, 5, 6, 7, 8, 9 and major textbooks of renal pathology., In addition, it is not unusual when clinicopathologic studies are designed for new or newly modified definitions for individual lesions to be used that have only been internally validated by the participating pathologist(s).

Because of the lack of international standardization of terms and definitions for glomerular lesions, the RPS charged a working group to develop consensus definitions for individual glomerular lesions found by LM and EM that can be applied across the full spectrum of glomerular diseases in both research and clinical settings. This effort resulted in the publication of definitions for 47 individual glomerular lesions found by LM and 47 glomerular lesions and 9 normal structures found by EM. The underlying hypothesis behind this effort is that adoption of the consensus definitions by the renal pathology community would facilitate harmonization of nomenclature of the individual lesions comprising different glomerular disease classifications and potentially improve correlations of pathologic lesions with emerging genomic, transcriptomic, proteomic, and other pathophysiological parameters to better understand the pathogenesis of glomerular diseases and help with identification of novel therapeutic targets.

As an initial test of the impact of the consensus definitions on the ability of RPS members (mainly pathologists) to identify individual glomerular lesions found by LM and EM, 2 surveys were circulated to the full membership of the RPS, each having 32 images (19 LM, 13 EM) and accompanying questions with 5 multiple-choice answers, one the consensus choice of the 13-member definition working group. The surveys were sent 7 months apart, the first (S1) 2 months before publication of the consensus definitions and the second (S2), with images of the same lesions and structures (but not the same images) and the same questions and multiple choices in different order, 7 months later.

Our findings suggest that publication of the consensus definitions has positively affected interobserver agreement in identifying glomerular lesions, although this improvement was modest. Although the overall percent agreement of respondents to the surveys with the consensus answers of the working group did not increase significantly from S1 to S2, this increase became greater and statistically significant when the S2 results were limited to those respondents who reported reading the def. paper. Furthermore, among the respondents to S2, agreement with the consensus answers was significantly greater overall among those who read the paper than those who did not and was also greater for 29 of the 32 images in the survey. As a greater fraction of S2 respondents who read the paper completed S1 than among those who did not read the paper, we considered the possibility that the apparent impact of the paper might really reflect in large part familiarity with the survey format owing to taking the previous survey, and it was indeed found that overall agreement with the consensus answers in S2 was significantly higher in those respondents who completed S1 than in those who did not. Nevertheless, agreement with consensus answers in S2 was significantly greater in respondents who read the def. paper, whether they did or did not complete S1, with the mean agreement for the 32 questions being 7.7% greater among respondents who read the paper for both those who did and did not take S1.

The comparison of findings on 2 surveys as a means for evaluating the impact of publication of the consensus definitions clearly has important limitations. As noted previously, familiarity with the survey format and possibly the lesions included might have contributed to the better agreement with the consensus answers found in S2. In addition, although it is tempting to attribute the better results in S2 among respondents who read the def. paper mainly or entirely to their having learned and been able to apply the consensus definitions, there are additional factors that may have contributed to these results. For example, this group of respondents may as a whole be more diligent in their overall reading of the literature and/or more careful in analyzing the survey images and questions. More notably, although the lesions and structures and the multiple-choice questions and answers were the same in the 2 surveys, the images themselves were not. This almost certainly contributed to some questions having widely different levels of agreement with the consensus answers in the 2 surveys. Specifically, there were 7 questions with a disparity in the percent agreement between the 2 surveys of >30%; in 3 (questions 4, 18, and 19 in Table 2) agreement was better in S1 and in 4 (questions 25, 28, 29, and 31) agreement was better in S2. In question 4, the respondents were shown a glomerulus with mesangial and segmental endocapillary hypercellularity, and one of the alternative answers was just mesangial hypercellularity. Differences in the extent of endocapillary hypercellularity in the 2 images likely contributed to the large difference in agreement with the consensus answer in the 2 surveys. The images for question 31 clearly revealed endocapillary hypercellularity, but also a segmental cellular crescent. One of the choices was endocapillary hypercellularity, whereas another was a fibrocellular crescent, and in S1, most of the respondents chose the latter, perhaps because the cellular crescent contained a minor (but <25%) component of matrix. In S2, there was considerable (35%) improvement, possibly because the consensus definitions clarified the cutoff values for cells and matrix in cellular, fibrocellular, and fibrous crescents (an important point of emphasis in the consensus definitions), but possibly also because the crescent in S2 may have been easier to distinguish as cellular versus fibrocellular. Despite our best efforts, there can be little doubt that some of the images were easier to identify in one versus the other survey, and although the data for survey 2 alone do reveal that respondents who read the def. paper had better overall agreement with the consensus answers than those who did not, the poor agreement for some S2 images even among the former group of respondents strongly suggests that these images were likely to have been suboptimal.

There were also some lesions in which agreement with the consensus answer was suboptimal in both surveys, most notably the membranoproliferative pattern (question 30; Figure 3a and b) and an intracapillary thrombus with glomerular basement membrane duplication (question 32; Figure 3c and d). Nevertheless, for these images, it seems to have been one of the alternative choices in the multiple-choice questions that created difficulty; nodular glomerulosclerosis and an intracapillary thrombus with glomerular basement membrane duplication and visceral epithelial cell hyperplasia, respectively; the latter choice was the most often selected for question 32 in both surveys. What this seems to indicate is two-fold: first, there are some definitions that require modification (e.g., that nodular-type mesangial expansion may be found with a membranoproliferative pattern, and better distinction between visceral epithelial hyperplasia and hypertrophy), and second, pathologists need to be more vigilant in identifying multiple lesions within the same glomerulus.

Figure 3

Images from questions for which agreement of respondents to both surveys with the consensus answers was ≤50%: (a, b) membranoproliferative pattern and (c, d) an intracapillary thrombus with glomerular basement membrane duplication. Images a and c are from survey 1 and b and d are from survey 2. Note that the images in a and b also reveal a nodular-like pattern of mesangial expansion and that those in c and d reveal visceral epithelial cell hypertrophy without true hyperplasia. Original magnification of all images ×400; a: periodic acid–Schiff stain, b–d: Jones methenamine silver stain; bars in each image = 70 μm.

Those lesions for which agreement with the consensus answer remained suboptimal even among respondents who read the def. paper indicate that although it is possible to reduce interobserver variability in renal pathology diagnosis, even under the best of circumstances, this is likely to remain a significant limitation in optimizing patient care and therapeutic trials. This latter conclusion is reinforced by the following 2 additional points: first, the rather modest improvement in overall agreement with consensus answers from S1 to S2, even among respondents who read the def. paper, and second, that the most ambiguous images that were intentionally excluded from the surveys are in fact present in real-world practice. It is largely because of these limitations of conventional renal biopsy interpretation that interest in application of computational image analysis to renal pathology has grown considerably in recent years., Such analysis is now being applied to whole slide images of renal biopsies and holds promise not only for reducing interobserver variability but also for more complex processes, such as prediction of disease progression and therapeutic response. Currently, there are considerable limitations in the application of computational image analysis to renal pathology, some of which relate to available hardware and software that are likely to improve over time, but others relate to standardization of tissue analysis which is where having accepted consensus definitions for individual lesions and disease processes can prove very helpful.

Finally, even in the best case scenario, which would likely include presenting multiple and better examples of each lesion to the renal pathology community, the impact of using consensus definitions to improve interobserver agreement in identification of morphologic lesions still greatly depends on how widespread the acceptance of these definitions becomes within this community. From our experience with establishing uniformity of definitions in more limited settings, such as the Oxford classification of IgA nephropathy,1, 2, 3 updates to the ISN/RPS classification of lupus nephritis,, and within revisions to the Banff classification for kidney transplant rejection, this general acceptance will likely be gradual. The finding that more than 60% of the respondents to S2 reported reading the def. paper indicates significant awareness of these definitions within the renal pathology community. Still, it is hoped that the results of the surveys will encourage acceptance of the consensus definitions, and we are optimistic that future development and application of glomerular disease classification and scoring systems will ultimately benefit.

Disclosure

All the authors declared no competing interests.