Inter- and intra-operator variability in the reading of indirect immunofluorescence assays for the serological diagnosis of scrub typhus and murine typhus.

Inter- and intra-observer variation was examined among six microscopists who read 50 scrub typhus (ST) and murine typhus (MT) indirect immunofluorescence assay (IFA) immunoglobulin M (IgM) slides. Inter-observer agreement was moderate (κ = 0.45) for MT and fair (κ = 0.32) for ST, and was significantly correlated with experience (P = 0.03 and P = 0.004, respectively); κ-scores for intra-observer agreement between morning and afternoon readings (range = 0.35-0.86) were not correlated between years of experience for ST and MT IFAs (Spearman's ρ = 0.31, P = 0.54 and P = 0.14, respectively; P = 0.78). Storage at 4°C for 2 days showed a change from positive to negative in 20-32% of slides. Although the titers did not dramatically change after 14 days of storage, the final interpretation (positive to negative) did change in 36-50% of samples, and it, therefore, recommended that slides should be read as soon as possible after processing.

Introduction

Rickettsial diseases infecting humans are caused by obligate intracellular organisms of the genera Rickettsia and Orientia transmitted by arthropod vectors such as ticks, fleas, lice, and mites. The most common clinical manifestations are fever, headache, and myalgia, making clinical diagnosis difficult because of similarities to other undifferentiated fevers, such as malaria, dengue, and leptospirosis.1 The laboratory diagnosis of rickettsioses conventionally relies on serological tests, such as enzyme-linked immunosorbent assay (ELISA) and indirect microimmunofluorescence assay (IFA), and less commonly relies on blood and eschar polymerase chain reaction (PCR) and in vitro culture. Serological tests require paired admission and convalescence samples to assess a dynamic change in titer (more than or equal to fourfold rise is conventionally regarded as positive). IFA is considered to be the gold standard1,2 for the serological diagnosis of murine typhus and scrub typhus infections. However, most studies have based their interpretation on a cutoff titer on an admission serum without clear justification, rather than a fourfold (or greater) rise in titer between paired sera.3 This interpretation, coupled with other sources of variation (antigen selection and antibody isotype), causes a lack of consistency in the results.4,5 We are not aware of any studies examining inter-microscopist variability in the reading of IFA slides for the diagnosis of any infectious disease or examining the consequences of IFA slide storage on results.6 We therefore assessed inter-observer and intra-observer variability in the reading of scrub typhus and murine typhus immunoglobulin M (IgM) antibody IFAs plus the consequences on the results of keeping the prepared slides for 14 days at 4°C.

Materials and Methods

Study site and patient samples.

This study was performed on sera from patients admitted to the Adult Infectious Disease Ward at Mahosot Hospital, Vientiane, Lao People's Democratic Republic (Laos) between January and April of 2009; 50 adult patients with suspected typhus (fever, headache, and/or myalgia; ages ≥ 15 years) were recruited if they gave informed written consent. The study was approved by the National Ethics Committee for Health Research of the Lao People's Democratic Republic and the Oxford Tropical Research Ethics Committee, United Kingdom.

Indirect IFAs were used as previous described.7 Briefly, 4 μL serum were diluted to 1:25 in a microtitration plate with sterile phosphate-buffered saline (PBS) plus 3% skimmed milk powder. These sera were serially diluted twofold from 1:25 to 1:3,200. A 2-μL aliquot of each serum dilution was aspirated from the wells (being careful to prevent cross-contamination), added to IFA slides coated with antigen from O. tsutsugamushi (scrub typhus) strains Karp, Kato, and Gilliam serotypes or R. typhi (murine typhus) strain Wilmington (Australian Rickettsial Reference Laboratory, Geelong, Victoria, Australia), and incubated in a moist chamber at 37°C for 1 hour. Slides were then washed three times (5 minutes per wash) with sterile PBS. After washing and drying, the slides were treated with fluorescein isothiocyanate-conjugated goat anti-human IgM or IgG (Sigma Aldrich, Munich, Germany), incubated for 30 minutes at 37°C, washed three times (5 minutes per wash) with sterile PBS, and mounted in buffered glycerol (90% glycerol and 10% PBS). The end point of each IFA titer was defined as the lowest serum concentration showing definite fluorescence. A positive result was defined as an IgM or IgG titer ≥ 1:400 or a fourfold increase in titer.8

Inter-observer variation.

The 50 scrub typhus IgM slides and 50 murine typhus IgM slides were read by six operators of differing experience who were blinded to the results of the other operators. Operator experience ranged from 0.1 (estimated < 100 slides) to 10 years (estimated ≥ 10,000 slides; microscopist A: 0.1 years [inexperienced]; microscopist B: 1 year [trainee]; microscopist C: 2 years [experienced]; microscopist D: 3 years [experienced]; microscopist E: 5 years [expert]; microscopist F: 10 years [expert]). Paired slide readings could be interpreted into one of five possible classes: less than fourfold increase in titer, fourfold increase in titer, eightfold increase in titer, 16-fold increase in titer, or 32-fold increase in titer.

Intra-observer variation.

Each of the six operators read the 50 scrub typhus IgM slides and 50 murine typhus IgM slides in the morning and afternoon of the same day. The slides were relabeled by an independent investigator between the morning and afternoon sessions to blind the operators to the identity of the slides. A κ-statistic was calculated for each operator's 50 paired readings for scrub typhus and 50 paired readings for murine typhus with a weighting matrix described below.

Determination of the effects of 4°C IFA slide storage.

To determine the effect of slide storage at ∼4°C on IFA results, scrub typhus and murine typhus IgM and IgG slides were prepared from the admission and convalescence sera of 50 patients (i.e., 200 slides total). These patients were the same patients as described for the observer variation study. The slides were read immediately after processing (day 0 [D0]), stored in a refrigerator at ∼4°C, and reread after 2 (D2) and 14 days (D14) of storage by the same six microscopists as above. At each reading time point, the slides were relabeled and randomized by a person who was not involved in slide reading to blind the operators to slide identity.

Statistical analysis.

All analyses were performed using STATA version 10 (Stata Corp., College Station, TX). Agreement among the six operators was calculated using the κ-statistic, which was interpreted as follows9: κ ≤ 0.20 (poor), 0.21 ≤ κ ≤ 0.40 (fair), 0.41 ≤ κ ≤ 0.60 (moderate), 0.61 ≤ κ ≤ 0.80 (good), and κ > 0.80 (very good).

For the κ-value calculation, a weighting matrix was used, where one indicated perfect agreement (when paired readings were assigned the same titer) and a weight of 0.60 meant two-thirds agreement (used when paired readings differed by only ± one titer). All other paired readings were in complete disagreement (i.e., paired readings differed by ± two or more titers) and given a weight of zero.

To determine whether there was any correlation between κ-score and years of experience, each pair-wise combination of the six microscopists (i.e., 15 pairs) was ranked into four groups (i.e., inexperienced, trainee, experienced, and expert). Any pair that included the operator with less than 1 year experience was considered inexperienced, and the microscopists E and F pair had the highest expert rank. Any pair that included the operator with 1 year of experience was trainee, and all other pairs were ranked experienced; κ-scores were calculated for each pair-wise combination of the six microscopists, and these scores were compared against the experience ranking using a Spearman's rank correlation coefficient.

Results

IgM IFA results read by an expert microscopist.

The results from the microscopist with most experience (10 years; expert microscopist) were used to give the true status of the samples. In summary, for murine typhus IgM IFA results, 25 (50%) slides had titers of ≤ 1:400 titer, and 25 (50%) slides had titers of ≥ 1:400 titer (1:400 [4%], 1:800 [16%], 1:1,600 [16%], and ≥ 1:3,200 [14%]); for scrub typhus IgM IFA results, 17 (34%) slides gave titers of ≤ 1:400, 33 (66%) slides gave titers of ≥ 1:400 (1:400 [14%], 1:800 [16%], 1:1,600 [8%], and ≥ 1:3,200 [28%]).

Inter-operator agreement.

κ-analysis overall agreement between the six microscopists was moderate (κ = 0.45) for murine typhus IgM IFA titers and fair (κ = 0.32) for scrub typhus IgM IFA titers (Table 1). The range of κ-scores for murine typhus IgM IFA titers was 0.39 (microscopists A and D) and 0.76 (microscopists B and C), and the range of κ-scores for scrub typhus IgM IFAs was 0.20 (microscopists A and D) and 0.75 (microscopists D and E). Years of experience and intra-operator κ-score were significantly correlated for scrub typhus (Spearman's ρ = 0.70, P = 0.004) and murine typhus (Spearman's ρ = 0.55, P = 0.03).

Table 1

Inter-observer κ-scores of scrub typhus and murine typhus IgM IFA results from six microscopists of varying experience for 50 admission sera

	Microscopist							Overall κ-score	Spearman's ρ (P value)
	Years	A	B	C	D	E	F
Murine typhus
A	0.1	1
B	1	0.49 (0.09)	1
C	2	0.54 (0.09)	0.76 (0.10)	1
D	3	0.39 (0.08)	0.66 (0.10)	0.53 (0.09)	1				0.55 (P = 0.03)
E	5	0.45 (0.09)	0.75 (0.10)	0.69 (0.09)	0.66 (0.10)	1
F	10	0.49 (0.09)	0.66 (0.10)	0.68 (0.10)	0.71 (0.10)	0.64 (0.10)	1	0.45
Scrub typhus
A	0.1	1
B	1	0.55 (0.09)	1
C	2	0.28 (0.07)	0.45 (0.08)	1				0.32	0.70 (P = 0.004)
D	3	0.20 (0.07)	0.43 (0.09)	0.65 (0.09)	1
E	5	0.36 (0.08)	0.50 (0.09)	0.57 (0.09)	0.75 (0.10)	1
F	10	0.33 (0.08)	0.54 (0.09)	0.60 (0.09)	0.46 (0.09)	0.51 (0.10)	1

SE of the κ-score is presented in parentheses.

Intra-operator variation.

Overall, the κ-values and percent agreement between morning and afternoon readings for each operator were high (Table 2), with κ-values indicating fair to very good agreement (0.35–0.86) for the murine typhus slides. For scrub typhus slides, agreement between the morning and afternoon readings ranged from moderate to very good (0.52–0.86) for all six microscopists. No correlation between years of experience and intra-operator κ-score was evident for scrub typhus (Spearman's ρ = 0.31, P = 0.54) or murine typhus (Spearman's ρ = 0.14, P = 0.78).

Table 2

Intra-observer agreement and κ-values for six microscopists of varying experience for the reading of 50 scrub and murine typhus IgM IFA slides in the morning and afternoon.

Microscopist	Years of experience	Experience status	Murine typhus			Scrub typhus
			Percentage agreement	κ-value	Spearman's ρ (P value)	Percentage agreement	κ-value	Spearman's ρ (P value)
A	0.1	Inexperienced	88.0	0.76	0.14 (P = 0.78)	88.0	0.62	0.31 (P = 0.54)
B	1	Trainee	71.6	0.35		92.4	0.83
C	2	Experienced	92.4	0.86		88.8	0.75
D	3	Experienced	82.8	0.67		81.6	0.52
E	5	Expert	92.8	0.86		86.8	0.7
F	10	Expert	88.8	0.71		91.6	0.85

Effect of storage at ∼4°C.

The slides were stored in a refrigerator for 14 days with a mean daily temperature of 4.7°C (95% confidence interval = 3.8–5.6; minimum = 1.4°C, maximum = 9.2°C). IgM and IgG antibody titers for both murine and scrub typhus IFAs showed a general reduction with increasing storage time (Tables 3–5). Murine typhus IFA results tended to be more stable (no change in titer; range = 56–80% of readings) (Table 4) than scrub typhus IFA results (36–64% of readings) (Table 3). Total percentage of IFA titer decreases at day 2 of storage between admission and convalescent samples was reasonably consistent for scrub typhus IgM (30% and 32%, for admission and convalescent sera, respectively) and IgG (both 40%) and murine typhus IgM (28% and 30%) and IgG (28 and 14%). Decreases between admission and convalescent samples were noted at day 14 of storage for scrub typhus IgM (38% and 32%) and IgG (64% and 38%) and murine typhus IgM (30% and 28%) and IgG (34% and 16%). Small increases in titers were also noted, with the largest increase being for scrub typhus IgM admission and murine typhus IgM convalescent titers. Despite the reasonably stable overall percentage change between 2 and 14 days of storage, the magnitude of the change did increase with storage duration (most notably with scrub typhus IFA titers and to a lesser extent, with murine typhus IFA titers), which affected the final interpretation of the result. Storage at 4°C for 2 days showed a change from positive to negative in 20% (murine typhus IgG and scrub typhus IgM) to 32% (murine typhus IgM) of slides (Table 5), and whereas the titers did not dramatically change after 14 days of storage, the final interpretation (positive to negative) did change in 36% (murine typhus IgM/IgG and scrub typhus IgM) to 50% (scrub typhus IgG) of samples.

Table 3

Scrub typhus IFA results for IgM and IgG antibody titers for 50 patient samples (admission and convalescent samples) stored at ∼4°C and tested on days 2 and 14 compared with day 0 results

Antibody	Storage	Scrub typhus IFA (N = 50)
		No change	Increased titer (%)				Decreased titer (%)
			+1	+2	> +2	Total	−1	−2	> −2	Total
Admission
IgM	Day 2	28 (56)	2 (4)	3 (6)	2 (4)	7 (14)	11 (22)	2 (4)	2 (4)	15 (30)
IgM	Day 14	27 (54)	2 (4)	2 (4)	0	4 (8)	12 (24)	6 (12)	1 (2)	19 (38)
IgG	Day 2	28 (56)	2 (4)	0	0	2 (4)	13 (26)	4 (8)	3 (6)	20 (40)
IgG	Day 14	18 (36)	0	0	0	0	14 (28)	14 (28)	4 (8)	32 (64)
Convalescent
IgM	Day 2	30 (60)	3 (6)	0	1 (2)	4 (8)	13 (26)	3 (6)	0	16 (32)
IgM	Day 14	32 (64)	1 (2)	1 (2)	0	2 (4)	8 (16)	6 (12)	2 (4)	16 (32)
IgG	Day 2	26 (52)	4 (8)	0	0	4 (8)	11 (22)	4 (8)	5 (10)	20 (40)
IgG	Day 14	29 (58)	2 (4)	0	0	2 (4)	10 (20)	6 (12)	3 (6)	19 (38)

Table 5

Scrub typhus and murine typhus IFA results for IgM and IgG antibody titers for 50 patient samples (admission and convalescent sera) stored at ∼4°C and tested on days 2 and 14 compared with day 0 results showing the change in diagnostic status

Combined admission and convalescent	N = 100	Murine typhus (N = 50)		Scrub typhus (N = 50)
		IgM	IgG	IgM	IgG
Day 2	Percent change to positive	7 (14)	1 (2)	6 (12)	4 (8)
Day 2	Percent change to negative	16 (32)	10 (20)	10 (20)	15 (30)
Day 14	Percent change to positive	1 (2)	2 (4)	3 (6)	1 (2)
Day 14	Percent change to negative	18 (36)	18 (36)	18 (36)	25 (50)

Table 4

Murine typhus IFA results for IgM and IgG antibody titers for 50 patient samples (admission and convalescent sera) stored at ∼4°C and tested on days 2 and 14 compared with day 0 result

Antibody	Storage at 4°C	Murine typhus (N = 50)
		No change	Increased titer (%)				Decreased titer (%)
			+1	+2	> +2	Total	−1	−2	> −2	Total
Admission
IgM	Day 2	34 (68)	2 (4)	0	0	2 (4)	7 (14)	7 (14)	0	14 (28)
IgM	Day 14	35 (70)	0	0	0	0	6 (12)	7 (14)	2 (4)	15 (30)
IgG	Day 2	36 (72)	0	0	0	0	9 (18)	5 (10)	0	14 (28)
IgG	Day 14	33 (66)	0	0	0	0	9 (18)	8 (16)	0	17 (34)
Convalescent
IgM	Day 2	28 (56)	5 (10)	2 (4)	0	7 (14)	12 (24)	1 (2)	2 (4)	15 (30)
IgM	Day 14	33 (66)	3 (6)	0	0	3 (6)	11 (22)	3 (6)	0	14 (28)
IgG	Day 2	40 (80)	2 (4)	0	1 (2)	3 (6)	6 (12)	1 (2)	0	7 (14)
IgG	Day 14	40 (80)	1 (2)	1 (2)	1 (2)	3 (6)	5 (10)	3 (6)	0	8 (16)

Discussion

IFA interpretation requires microscopists to use their judgment to determine the fluorescence endpoint, and therefore, it is inherently subjective.10 In this study, we found that overall inter-microscopist agreement using κ-scores was fair (scrub typhus IgM IFA) to moderate (murine typhus IgM IFA) (Table 1), with a κ-score range of 0.20 (poor) to 0.76 (good). The results cast doubt on the comparability of results between different studies. Furthermore, they highlight the need for using as few microscopists as possible when reading IFAs titers in a large study. Similar subjectivity has been noted in other studies requiring the interpretations of individual operators. In a study of inter-observer and intra-observer variability among pathologists in lymph node assessments, 15 slides wereassessed by 10 pathologists, and significant disagreement on the size of the smallest countable node was noted.11 An analysis of thin-preparation Papanicolaou tests among 19 cytotechnologists from three different laboratories also showed significant variability.6

Microscopist inexperience plays a role in the lack of agreement between IFA microscopists, with significant correlation between κ-scores and years of experience for both scrub and murine typhus IFAs. The poorest agreement was between the least experienced operator (microscopist A) and the other readers; however, generally higher agreement was noted between all other microscopists of various levels of experience. These results suggest that IFA microscopists need a minimum of 12 months training before their IFA results may be considered reliable or that they must buddy with other trainee or experienced microscopists during a period of probation.

Intra-operator κ-scores and agreement between morning and afternoon IFA readings showed variation, but it was generally less than the variation for inter-operator comparison with the same samples. Nevertheless, it is concerning that reproducibility of the titers cannot be guaranteed between two separate readings on the same day. Interestingly, the intra-operator results were not significantly influenced by the experience of the reader, with even the inexperienced reader having good (0.62) to very good (0.76) κ-scores.

Overall κ-scores were generally higher for murine typhus IFA slide reading than scores for scrub typhus IFA slide reading. These results were unexpected, because murine typhus IFA slides have previously been considered (Blacksell SD, unpublished results) to be more difficult to read; the extracellular nature of the R. typhi organism on the IFA slides has a fine gold dust-like appearance and may be confused with artifact. Scrub typhus IFA slides, however, tend to be simpler for the inexperienced microscopist to interpret, because the O. tsutsugamushi fluorescence is intercellular and more defined. One explanation may be that microscopists spend more time reading the murine typhus slides because of their perceived difficulty of assessment, and this investment of time gives a more reliable result. Antibody titers can influence the κ-scores. The dilution series in this study varied from low titers (< 1:400) to high titers (≥ 1:3,200). Negative sera or sera approaching the positive/negative threshold give more variable results because of the subjective nature of reading. This subjectivity may have contributed to the low κ-scores because of the relatively high number of negative and low to moderate number of positive samples examined.

This study also showed that IFA slides should be read as soon as possible after processing. A change in final diagnostic interpretation of scrub and murine typhus IgM/IgG titers was noted for admission and convalescent sample IFAs stored for 2 days at 4°C, and although the titers did not dramatically change at 14 days of storage at 4°C, the final interpretation (from positive to negative) using the Coleman criteria5 changed for 30–48% of samples. Interestingly, up to 8% of the samples also showed increases in titers during the storage period, presumably because of intra-operator variation.

In conclusion, the subjective nature of reading IFA slides for titer endpoints, which is compounded by issues of inter- and intra-microscopist variation, microscopist experience, interpretation difficulties when approaching the diagnostic threshold, and storage issues (especially when large numbers of samples are processed), makes the diagnosis of rickettsial illness using IFA variable at best and at worst, unreliable. It is imperative that alternative objective and reliable means of serological gold standard diagnosis are developed and validated without delay. The results also emphasize the importance of internal and external quality assurance schemes for rickettsial IFA slide reading.