Interobserver Agreement Using Histological Scoring of the Canine Liver.

BACKGROUND: Grading schemes for the assessment of hepatic fibrosis and necroinflammatory activity in humans previously have been applied to dogs with chronic hepatitis. Interobserver agreement is a desirable characteristic for any histological scoring scheme. HYPOTHESIS/
OBJECTIVES: To assess interobserver agreement associated with pathologists using a previously published histological scoring scheme to assess hepatic fibrosis and necroinflammatory activity in dogs and to compare fibrosis scores assigned to serial sections stained with hematoxylin & eosin (H&E) and picrosirius red. ANIMALS: Histological sections of liver from 50 dogs with variable degrees of hepatic fibrosis and necroinflammatory activity were selected from institutional tissue archives.
METHODS: Six board-certified veterinary anatomic pathologists assigned fibrosis and necroinflammatory activity scores to the histological sections. The multiuser kappa statistic was calculated to assess interobserver agreement. Fibrosis stage assigned to serial sections stained with picrosirius red and H&E was compared using the Wilcoxon signed-rank test.
RESULTS: Multiuser kappa statistics for assessment of fibrosis and necroinflammatory activity from H&E-stained sections were 0.35 and 0.16, respectively. There was no difference in median fibrosis scores assigned to serial section stained with H&E and picrosirius red (P = .248). CONCLUSIONS AND CLINICAL IMPORTANCE: There was fair interobserver agreement when pathologists assessed fibrosis and poor agreement when they assessed necroinflammatory activity. This suboptimal agreement must be taken into account by clinicians making decisions based on histology reports of the liver and in the design of studies evaluating these findings. To decrease this variability, ideally >1 pathologist should evaluate each section.

central–central

hematoxylin and eosin

not applicable

portal–central

portal–portal

Cohen's weighted kappa statistic

Cohen's kappa statistic

Several histological grading systems have been developed for assessment of hepatic necroinflammatory activity and fibrosis in human patients with chronic hepatitis, including the Knodell,1 Ishak,2 and METAVIR3 schemes. Necroinflammatory activity grade encompasses various patterns of hepatocellular necrosis and inflammation, providing information regarding activity of the disease process, whereas the fibrosis stage gives an indication of disease chronicity.4 Despite use of histological scoring schemes in human patients with liver disease, currently there is no widely accepted scheme for use in dogs. A semiquantitative scoring system encompassing necroinflammatory changes, apoptosis, and fibrosis previously was developed for use in dogs with primary hepatitis.5 More recently, several studies in dogs with chronic hepatitis6, 7 have used a histological scoring scheme for necroinflammatory grade and fibrosis stage that was adapted from the Ishak scheme.8

It has been proposed that histological scoring schemes fulfill several criteria, including interobserver agreement, intra‐observer agreement, and clinical relevance.2, 4 Suboptimal agreement among pathologists evaluating histological sections prepared from intestinal biopsy specimens from dogs has previously been documented,9, 10 and 1 study found a lack of interobserver agreement in the morphologic diagnosis assigned to needle and wedge liver biopsy specimens from dogs.11 Other studies involving a single pathologist suggest that the histopathologic interpretation of a canine liver sample is unlikely to vary if the specimen contains at least 3–12 portal triads, regardless of the biopsy technique,12 and that the likelihood of obtaining a representative sample is increased when multiple liver lobes are sampled.13 To our knowledge, interobserver agreement associated with the histological scoring of fibrosis and necroinflammatory activity from hepatic biopsy specimens collected from dogs has not been reported previously.

Cohen's kappa statistic (κ) frequently is used to estimate agreement of observers for data on nominal scales.14 A κ of zero represents no agreement beyond that due to chance, and a κ of 1.0 represents complete agreement. For nominal scoring systems, partial agreement between users may be taken into account.15 A weighted kappa statistic (κ′) accounts for partial agreement by assigning weights to different levels of disagreement.14

The primary objective of our study was to assess interobserver agreement associated with the use of a scoring scheme to evaluate hepatic fibrosis in dogs. The secondary objectives were to assess interobserver agreement associated with the use of a scoring scheme to evaluate hepatic necroinflammatory activity and to compare fibrosis scores assigned to serial sections of the same biopsy specimen stained with hematoxylin and eosin (H&E) and picrosirius red. We hypothesized that pathologists assign higher fibrosis scores to serial sections of canine liver stained with picrosirius red to those stained with H&E.

Materials and Methods

Case Material

Fifty paraffin‐embedded specimens of canine liver with variable degrees of fibrosis and necroinflammatory activity were selected from tissue archives. The sections were selected primarily to represent the full severity range of hepatic fibrosis by a clinician (JAL) who read the preexisting histopathology reports for these cases. This was done to avoid having poorly represented or unrepresented lesion severities, which could lead to falsely lower kappa values or make it impossible assess agreement for lesions of those severities, respectively. Selected case material has been used in other agreement studies.16, 17 Overall, 36 dogs had chronic hepatitis, 11 were considered to be free from liver disease, and 3 had hepatic changes associated with congenial portosystemic shunts. No dogs were euthanized or underwent liver biopsy for study related reasons. Seventeen wedge biopsy specimens were collected at necropsy, 17 were collected during laparotomy, 11 were collected during laparoscopy with forceps (typically 4–6), and 5 were collected during laparotomy using skin biopsy punches. We opted to use wedge biopsy samples collected at necropsy, laparoscopically collected samples, and those collected during laparotomy so that interobserver agreement could be assessed when evaluating specimens of adequate size. Two serial 4–5 μm sections of liver were cut from the paraffin‐embedded tissue and mounted onto separate microscope slides. One section was stained manually with picrosirius red1 and counter‐stained with Weigert's iron hematoxylin.2 The sections were stained in as few batches as possible (3) to maintain consistency. The other section was stained routinely with H&E using an automated slide stainer.

Histological Assessment

A number from 1 to 100 was randomly assigned to each section (50 stained with H&E and 50 stained with picrosirius red). The sections then were relabeled with this number as their only identifier. A digital image of each whole section was captured with a slide‐scanning microscope3 using the 40× objective lens. Six board‐certified veterinary anatomic pathologists evaluated the images of the scanned sections (3) or the slides (3), depending on their location. At the time, the study was performed 3 of the pathologists had ≥20 years of post‐residency experience, 1 had 12 years of post‐residency experience, and 2 had 5 years of post‐residency experience. One of the pathologists helped to develop the scoring scheme. The remaining 5 pathologists were unfamiliar with the scheme and did not receive specific training before this study. For the sections of liver stained with H&E, the pathologists evaluated the stage of fibrosis and grade of necroinflammatory activity using a histological scoring scheme that was adapted from the human Ishak scheme.8 For the sections stained with picrosirius red, the pathologists scored stage of fibrosis only. According to this scoring scheme, necroinflammatory activity is graded as 0: absent, 1: slight, 2: mild, 3: moderate, 4: marked, or 5: very marked, and fibrosis is graded 0: absent, 1: mild, 2: moderate, 3: marked, or 4: very marked (Appendix 1). During the scoring process, the pathologists were not aware of the identity of the sections they were assessing, the scores assigned by the other pathologists, or the results of image analysis.

Statistical Analysis

To evaluate interobserver agreement, κ for each pair of observers was calculated. Kappa and κ′ for multiple observers was calculated for each scoring category. Kappa was weighted as follows: 1 − (difference in rating between 2 raters/(maximum number of possible ratings − 1)). Kappa and κ′ values were interpreted using the following guidelines: poor agreement <0.20, fair agreement 0.21−0.40, moderate agreement 0.41−0.60, good agreement 0.61−0.80, and very good agreement 0.81−1.00.14 Interobserver agreement also was summarized by calculating the frequency of scores assigned by each of the 15 possible pairs of pathologists. This analysis was performed using a commercially available software package.4 The median fibrosis and necroinflammatory scores assigned by the 6 pathologists for each section were compared using Friedman's test, followed by Dunn's post‐test as appropriate. The median fibrosis stage assigned to each case for the sections stained with picrosirius red and H&E was compared using the Wilcoxon signed‐rank test. This analysis was performed using another software package.5 Statistical significance level was set at α < 0.05.

Results

All 100 sections were deemed to be of adequate to be size and quality for analysis by the 6 veterinary pathologists.

Fibrosis (Stage)

Agreement between the pairs of pathologists assigning scores for hepatic fibrosis to sections stained with H&E is presented in Table 1. The median (minimum−maximum) κ for each pair was 0.41 (0.14−0.56). Multiuser κ (95% confidence interval [CI]) was 0.35 (0.26−0.44), and multiuser κ′ was 0.59 (0.50−0.70). Assignment of fibrosis scores by the 15 possible pairings of pathologists to sections stained with H&E is presented in Table 2. The pairs of pathologists were in complete agreement 48.8% of the time, differed by 1 score level 40.5% of the time, and differed by 1 score level 10.7% of the time. There was a significant difference in fibrosis scores assigned to H&E‐stained sections by the 6 pathologists (P < .0001), and significant differences were found between scores for 5 of 15 possible pathologist pairings.

Table 1

Pairwise comparison of kappa statistics for the assessment of fibrosis from hematoxylin and eosin‐stained sections

Observer	1	2	3	4	5	6
1	NA	0.47	0.48	0.28	0.41	0.41
2	0.47	NA	0.56	0.17	0.42	0.35
3	0.48	0.56	NA	0.14	0.42	0.48
4	0.28	0.17	0.14	NA	0.18	0.17
5	0.41	0.42	0.42	0.18	NA	0.44
6	0.41	0.35	0.48	0.17	0.44	NA

NA, not applicable.

Table 2

Assignment of fibrosis scores by pathologist pairings for hematoxylin and eosin‐stained sections

Score	Absent	Mild	Moderate	Marked	Very Marked
Absent	9.9%	NA	NA	NA	NA
Mild	12.9%	7.2%	NA	NA	NA
Moderate	5.3%	9.7%	14.7%	NA	NA
Marked	0.7%	2.4%	11.9%	3.2%	NA
Very marked	0.0%	0.5%	1.7%	6.0%	13.9%

NA, not applicable.

The percentages represent the frequency at which the 15 possible pathologist pairings assigned hepatic fibrosis scores to the sections. The pairs of pathologists were in complete agreement 48.8% of the time (dark gray), differed by 1 score level 40.5% of the time (light gray), and differed by >1 score level 10.7% of the time (white).

Agreement between the pairs of pathologists assigning scores for hepatic fibrosis to sections stained with picrosirius red is presented in Table 3. The median (minimum−maximum) κ for each pair was 0.40 (0.22−0.56). Multiuser κ (95% CI) was 0.39 (0.30−0.49), and multiuser κ′ was 0.64 (0.55−0.73). Assignment of fibrosis scores by the 15 possible pairings of pathologists to sections stained with picrosirius red is presented in Table 4. The pairs of pathologists were in complete agreement 53.0% of the time, differed by 1 score level 42.3% of the time, and differed by >1 score level 4.7% of the time. There was a significant difference in fibrosis scores assigned to picrosirius red‐stained sections by the 6 pathologists (P < .0001), and significant differences were found between scores for 2 of 15 possible pathologist pairings.

Table 3

Pairwise comparison of kappa statistics for the assessment of fibrosis from picrosirius red‐stained sections

Observer	1	2	3	4	5	6
1	NA	0.50	0.40	0.24	0.29	0.48
2	0.50	NA	0.45	0.29	0.43	0.40
3	0.40	0.45	NA	0.49	0.39	0.56
4	0.24	0.29	0.49	NA	0.22	0.27
5	0.29	0.43	0.39	0.22	NA	0.56
6	0.48	0.40	0.56	0.27	0.56	NA

NA, not applicable.

Table 4

Assignment of fibrosis scores by pathologist pairings for picrosirius red‐stained sections

Score	Absent	Mild	Moderate	Marked	Very Marked
Absent	5.2%	NA	NA	NA	NA
Mild	9.6%	9.7%	NA	NA	NA
Moderate	1.7%	14.0%	18.7%	NA	NA
Marked	0.3%	1.6%	11.2%	3.7%	NA
Very Marked	0.0%	0.0%	1.1%	7.5%	15.7%

NA, not applicable.

The percentages represent the frequency at which the 15 possible pathologist pairings assigned fibrosis scores to the sections. The pairs of pathologists were in complete agreement 53.0% of the time (dark gray), differed by 1 score level 42.3% of the time (light gray), and differed by >1 score level 4.7% of the time (white).

There was no significant difference median scores assigned by the 6 pathologists for fibrosis between contiguous H&E and picrosirius red‐stained sections (P = .248).

Necroinflammatory Activity (Grade)

Agreement between the pairs of pathologists assigning scores for necroinflammatory activity is presented in Table 5. The median (minimum−maximum) κ for each pair was 0.19 (−0.03−0.40). Multiuser κ (95% CI) for assessment of necroinflammatory activity was 0.16 (0.10−0.23), and multiuser κ′ was 0.43 (0.32−0.55). Assignment of necroinflammatory scores by the 15 possible pairings of pathologists is presented in Table 6. The pairs of pathologists were in complete agreement 34.1% of the time, differed by 1 score level 46.7% of the time, and differed by >1 score level 19.2% of the time. There was a significant difference in necroinflammatory scores assigned to H&E sections by the 6 pathologists (P < .0001), and significant differences were found between scores for 7 of 15 possible pathologist pairings.

Table 5

Pairwise comparison of kappa statistics for the assessment of necroinflammatory activity

Observer	1	2	3	4	5	6
1	NA	−0.03	0.23	0.14	0.19	0.13
2	−0.03	NA	0.06	0.26	0.08	0.06
3	0.23	0.06	NA	0.11	0.22	0.26
4	0.14	0.26	0.11	NA	0.20	0.37
5	0.19	0.08	0.22	0.20	NA	0.40
6	0.13	0.06	0.26	0.37	0.40	NA

NA, not applicable.

Table 6

Assignment of necroinflammatory scores by pathologist pairings

Score	Absent	Slight	Mild	Moderate	Marked	Very Marked
Absent	13.2%	NA	NA	NA	NA	NA
Slight	19.3%	9.1%	NA	NA	NA	NA
Mild	6.1%	11.1%	4.9%	NA	NA	NA
Moderate	2.7%	3.6%	8.7%	3.6%	NA	NA
Marked	0.5%	0.7%	2.3%	4.7%	1.5%	NA
Very marked	0.3%	0.5%	0.7%	1.9%	2.9%	1.9%

NA, not applicable.

The percentages represent the frequency at which the 15 possible pathologist pairings assigned scores to the sections. The pairs of pathologists were in complete agreement 34.1% of the time (dark gray), differed by 1 score level 46.7% of the time (light gray), and differed by >1 score level 19.2% of the time.

Discussion

We found fair agreement among veterinary pathologists using a previously published scheme to score hepatic fibrosis in dogs using H&E and picrosirius red‐stained sections with κ of 0.35 and 0.39, respectively. These findings are comparable to results of a study using the Ishak scheme, in which pairwise κ ranged from 0.26 to 0.47, indicating fair to moderate agreement.15 Although the results from our study and those of the study in humans were similar, they cannot be compared directly because we used a 5‐point scheme rather than the original 7‐point scheme. When the pathologists did not completely agree, they often assigned scores that deviated only by 1 level, and they deviated only by >1 level 15% and 5% of the time for H&E and picrosirius red‐stained sections, respectively. This partial agreement was apparent when κ′ was calculated. Weighted kappa statistics for H&E and picrosirius red‐stained sections were 0.59 and 0.64, indicating moderate and good agreement, respectively. In the aforementioned study using the Ishak scheme to score hepatic fibrosis in human patients, pairwise κ′ ranged from 0.57 to 0.69, indicating moderate to good agreement.15 Again, the results are similar to those from our study, but direct comparisons cannot be made.

Interobserver agreement using this scoring scheme to assess hepatic fibrosis in dogs was suboptimal, but a high level of partial agreement was found. There was a significant difference in the fibrosis scores assigned to both the H&E and picrosirius red‐stained sections by the 6 pathologists. This suggests that some of the disagreement observed was caused by systematic differences in the way the individual pathologists assigned scores. Therefore, it may be possible to improve the level of agreement among pathologists if the descriptions for each score are clarified to encompass differences in interpretation (ie, by defining the subjective descriptors mild, moderate and marked that this scheme uses to define some scores), if the pathologists receive more training on how to apply the scoring system, or both. Agreement as assessed by κ tends to be higher for histological scoring systems with fewer levels.4 For example, in a study of humans evaluating fibrosis using the 4‐level METAVIR scoring scheme, κ was reported to be 0.59, indicating moderate agreement.18 Thus, it may be advantageous to simplify the system we evaluated to a 4‐level scale (absent, mild, moderate, and marked).

A multiuser κ of 0.19 indicates poor agreement among pathologists scoring necroinflammatory activity. In human medicine, it also has proven more difficult to develop a grading scheme for hepatic necroinflammatory activity that has acceptable interobserver agreement than 1 for fibrosis.4, 15 This may be because of the complexity and subjectivity of the histological grading schemes, which must take into consideration a diverse and difficult to standardize set of features, including interface hepatitis, focal necrosis, confluent necrosis, and portal inflammation. The multiuser κ′ for necroinflammatory activity was 0.43, indicating moderate interobserver agreement. This reflected the finding that there was often partial agreement between observers. Indeed, the scores assigned by pairs of pathologists only deviated by >1 level 19% of the time. In a study of humans using the Ishak scheme, pairwise κ for the different components of necroinflammatory activity was reported to range from 0.11 to 0.41, indicating poor to moderate agreement, whereas κ′ ranged from 0.19 to 0.53, indicating poor to moderate agreement.15 However, because the scheme we used in this study was a modification of the original Ishak scheme, direct comparisons cannot be made.

Because of this suboptimal interobserver agreement, before use in a clinical setting it would be beneficial to simplify the hepatic necroinflammatory activity scoring scheme that we used. One way to do this would be by collapsing it to 4 levels such as in the METAVIR scheme used to assess chronic hepatitis in humans.18 It also may be beneficial to score the components of hepatic necrosis and inflammation separately. A significant difference was found in the necroinflammatory scores assigned by the 6 pathologists. As discussed for the scoring of fibrosis, it may be possible to improve the level of agreement among pathologists if the descriptions for each score are clarified, if the pathologists receive more training on how to apply the scoring system, or both.

We hypothesized that fibrosis would be easier to detect on picrosirius red‐stained sections and that these sections would be assigned a higher fibrosis score, but there was no significant difference in median fibrosis scores assigned to H&E and picrosirius red‐stained sections. The agreement associated with the evaluation of fibrosis from sections stained with picrosirius red was slightly higher than that from H&E‐stained sections, κ statistics of 0.35 (0.26−0.44) and 0.39 (0.22−0.56), respectively. This difference should be interpreted cautiously because there was considerable overlap between CI. Taken together, these findings did not indicate a clear objective benefit of staining sections with picrosirius red. However, some of the coauthors still prefer this stain for the assessment of hepatic fibrosis.

Our study had several limitations. The kappa statistic is commonly used to assess interobserver agreement in biomedical research, but some authors have criticized its use. One concern is that κ and κ′ are dependent upon the distribution of severity among the cases.19 The cases in this study were selected primarily to represent a wide range of fibrosis scores, and although a wide range of severity of necroinflammatory activity scores was represented, the distribution among median scores was not even. Therefore, the absolute values of κ and κ′ for the assessment of interobserver agreement associated with histological scoring of this variable should be interpreted cautiously. Interpretation guidelines and weighting schemes used to calculate the κ′ also have been criticized as being too subjective.19 Because of the limitations of κ, we also expressed results as the frequency of different levels of agreement between pathologists. Another potential limitation is the use of material from healthy dogs and dogs with congenital portosystemic shunts. This was done to ensure that patients with no or mild fibrosis were included, and we believe doing so is unlikely to have influenced our main conclusion that there was disagreement among pathologists. Only 1 of the 6 pathologists had previously used a grading scheme for assessment of hepatic biopsy samples and introducing pictorial templates and example cases may help to decrease the interobserver variability documented in our study. As previously discussed, a limitation of the scoring scheme that we used was the use of subjective descriptors such as mild to define some of its score levels. Although 3 pathologists evaluated biopsy samples based upon whole‐slide digital images and 3 used conventional light microscopy, this was not thought to affect the results because previous studies indicated excellent agreement between the 2 modalities.20, 21, 22, 23 The effect of pathologist experience and specialization on interobserver agreement has been found to be important in previous studies,16 but our study did not evaluate this factor or consider differences in biopsy technique, both of which are worthy of further investigation. It also would be useful to determine the intra‐observer agreement for pathologists using this scheme.

In conclusion, use of this scoring scheme resulted in fair interobserver agreement when pathologists assessed hepatic fibrosis in dogs and poor agreement when they assessed hepatic necroinflammatory activity. This suboptimal agreement is concerning and to decrease this variability ideally >1 pathologist should evaluate each section. A simplified scoring scheme with fewer, more clearly defined levels may improve interobserver agreement. Additionally, fibrosis scores and interobserver agreement were similar for serial sections of liver stained with H&E and picrosirius red.

Necroinflammatory Activity (Grade)
Grade	Periportal or Periseptal Interface Hepatitis	Focal Lytic Necrosis	Confluent Necrosis
Absent (0)	Absent	Absent	Absent
Slight (1)	Very mild	1 focus per 10× obj.	Absent
Mild (2)	Mild	2–4 foci per 10× obj.	Absent
Moderate (3)	Moderate	5–10 foci per 10× obj.	Absent
Marked (4)	Marked	>10 foci per 10× obj. and/or →	Confluent or bridging necrosis
Very marked (5)	Marked	>10 foci per 10× obj. and/or →	Bridging or panacinar/multiacinar necrosis

Degree of Fibrosis (Stage)
Stage	Fibrosis	Bridging Fibrosis	Bridging Fibrosis with Nodule Formation
Absent (0)	Absent	Absent	Absent
Mild (1)	Mild fibrous expansion (periportal or central)	Absent	Absent
Moderate (2)	Moderate fibrous expansion	Some bridging fibrosis (PP, CC, or PC)	Absent
Marked (3)	Marked fibrous expansion	Marked bridging fibrosis (PP, CC, or PC)	Absent
Very marked (4)	Marked fibrous expansion	Marked bridging fibrosis (PP, CC, or PC)	Present