Low interrater reliability of brachial plexus MRI in chronic inflammatory neuropathies.

INTRODUCTION: Magnetic resonance imaging of the brachial plexus shows nerve thickening in approximately half of the patients with chronic inflammatory demyelinating polyneuropathy (CIDP) and multifocal motor neuropathy (MMN). The reliability of qualitative evaluation of brachial plexus MRI has not been studied previously.
METHODS: We performed an interrater study in a retrospective cohort of 19 patients with CIDP, 17 patients with MMN, and 14 controls. The objective was to assess interrater variability between radiologists by using a predefined scoring system that allowed the distinction of no, possible, or definite nerve thickening.
RESULTS: Raters agreed in 26 of 50 (52%) brachial plexus images; κ-coefficient was 0.30 (SE 0.08, 95% confidence interval 0.14-0.46, P < .0005). DISCUSSION: Our results provide evidence that interrater reliability of qualitative evaluation of brachial plexus MRI is low. Objective criteria for abnormality are required to optimize the diagnostic value of MRI for inflammatory neuropathies.

confidence interval

chronic inflammatory demyelinating polyneuropathy

multifocal motor neuropathy

nerve conduction study

Tesla

University Medical Center

INTRODUCTION

Magnetic resonance imaging (MRI) of the brachial plexus can be helpful to diagnose inflammatory neuropathies such as chronic inflammatory demyelinating polyneuropathy (CIDP) and multifocal motor neuropathy (MMN). The diagnostic challenge in CIDP and MMN is to distinguish these disorders from those that do not respond to immunomodulatory treatment.

Diagnostic criteria for CIDP and MMN primarily rely on clinical phenotype and specific nerve conduction study (NCS) abnormalities.1, 2 Brachial plexus MRI can be of diagnostic value when NCS is inconclusive despite high clinical suspicion. Magnetic resonance imaging abnormalities associated with CIDP and MMN are thickening of roots, plexus, and nerves, often combined with T2 hyperintensity.3, 4, 5 Previous MRI studies were exclusively qualitative and lacked clear definitions of abnormality.6, 7, 8 To clarify the value of brachial plexus MRI in the diagnostic workup of inflammatory neuropathies, we assessed interrater variability.

MATERIALS AND METHODS

Study design

We performed an interrater study in a retrospective cohort of patients with CIDP and MMN and controls.

Participants

Patients aged 18 to 85 years with CIDP and MMN according to the European Federation of Neurological Societies/Peripheral Nerve Society criteria1, 2 who were seen at the University Medical Center (UMC) Utrecht and who underwent brachial plexus MRI between September 2016 and September 2018 were selected for this study. Brachial plexus MRI from patients with other causes of peripheral motor deficits were used as controls.

Clinical data

We obtained clinical data from electronic patient records including age; sex; and time from onset of symptoms to diagnosis in months, defined as disease duration. All patients gave informed consent. This study was approved by the Medical Ethical Committee of the UMC Utrecht.

Magnetic resonance imaging protocol and assessment

Brachial plexus MRI was performed for diagnostic purposes and was reassessed for this study. Magnetic resonance imaging was performed on 1.5 and 3.0 Tesla (T) scanners (Philips, Best, the Netherlands). The MRI protocol consisted of a fat‐suppressed coronal three‐dimensional T2‐weigthed short‐τ inversion recovery with the following parameters: field of view = 250 × 320 × 170 mm3, matrix size = 208 × 269, voxel size = 0.6 × 0.6 × 1 mm3, echo time = 259 ms, repetition time = 2200 ms, turbo spin echo factor = 95, acquisition time = 06:16 minutes. In postprocessing, a coronal slab maximum intensity projection was created.

We developed a scoring system with categories of abnormality of nerve thickening (Figure S1). Scans were scored with a 3‐point scale (1 = no nerve thickening, 2 = possible nerve thickening, 3 = definite nerve thickening). Examples of abnormality were selected from a subset of all included patients by two experienced (2 years and > 30 years) neuroradiologists (A.G., T.D.W.) via discussion and consensus. The examples were saved with the Teaching Tool in PACS IDS7 19.3.12 (Sectra AB, Linköping, Sweden). With this tool, raters can scroll through images and compare them with the target image. The radiologists scored all images in PACS IDS7. The degree of abnormality was assessed by the overall impression of the entire brachial plexus. Images were presented in the same order and on screens with similar resolution to raters who were blinded to clinical status of the participants.

Statistical analysis

All statistical analysis was performed in SPSS Statistics (version 25; IBM, Armonk, New York). To analyze patient characteristics, we used independent samples t tests. The interrater variability of qualitative assessments of brachial plexus MRI was determined by Cohen's κ as coefficient for measure of agreement because we evaluated categorized data with limited categories. We interpreted a κ value of 0 to 0.20 as no agreement, 0.21 to 0.39 as minimal, 0.40 to 0.59 as weak, 0.60 to 0.79 as moderate, 0.80 to 0.90 as strong, and > 0.90 as almost perfect agreement.9 P < .05 was considered significant. We calculated sensitivity and specificity per rater using receiver operating characteristic curves.

RESULTS

We identified 36 patients with a chronic inflammatory neuropathy (CIDP = 19, MMN = 17) and 14 disease controls (motor neuron disease = 4, Hirayama disease = 3, ulnar neuropathy = 1, neurogenic thoracic outlet syndrome = 1, polyneuropathy in Sjögren disease = 1, brachial plexopathy caused by alcohol abuse = 1, cervical myelopathy = 1, lumbar polyradiculopathy = 1, chronic idiopathic axonal polyneuropathy = 1). Patient characteristics are presented in Table 1. Data were acquired for 26 participants by using 1.5 T MRI scanners and for 24 participants by using 3.0 T MRI scanners.

Table 1

Patient characteristics

	Inflammatory neuropathy			Controls
Patient characteristics	Total	CIDP	MMN	Total	Level of significance
No. of participants	36	19	17	14
Age, y (SD)	59.7 (14.9)	69.9 (9.0)	48.3 (11.7)	55.2 (16.5)	NS
Men (%)	26 (72)	12 (63)	14 (82)	7 (50)	NS
Disease duration, mo (SD)	43.3 (47.7)	45.0 (48.7)	41.5 (48.0)	39.1 (37.0)	NS

Abbreviations: CIDP, chronic inflammatory demyelinating polyneuropathy; MMN, multifocal motor neuropathy; NS, not significant.

Interrater variability

Raters agreed in 26 of 50 (52%) brachial plexus images when using three categories for abnormality (Table 2). Using the dichotomy normal‐abnormal (ie, category 1 vs categories 2 and 3), raters agreed in 36 of 50 (72%) cases. The κ coefficient was 0.44 (SE 0.13, 95% confidence interval [CI] 0.19–0.67, P = .002); κ coefficients for both methods indicate a minimal to weak level of agreement between raters.

Table 2

Assessment of brachial plexus MRI by two raters

Raters and cases	Category 1	Category 2	Category 3	Total (%)	Cohen's κ (SE)	95% CI	P value
Rater 1 No. of cases	25	17	8	50
Rater 2 No. of cases	21	4	25	50
Cases of agreement	16	2	8	26 (52)	0.30 (0.08)	0.14–0.46	<.00001

Abbreviation: CI, confidence interval.

In 15 of 50 (30%) cases, rater 1 scored “possible nerve thickening” while rater 2 scored “no nerve thickening” or “definite nerve thickening.” Discrepancies between raters seem, therefore, to be caused mostly by the appreciation and distinction of subtle abnormalities (Figure 1). Sensitivity was 61% and 75% and specificity was 79% and 86% for rater 1 and rater 2, respectively. Area under the curve was 0.698 (95% CI 0.539–0.858) for rater 1 and 0.804 (95% CI 0.667–0.940) for rater 2 (Figure S2).

Figure 1

Examples of interrater disagreement. Subject A was rated as “possible nerve thickening” by rater 1 and as “no nerve thickening” by rater 2. Subjects B and C were rated as “possible nerve thickening” by rater 1 and as “definite nerve thickening” by rater 2. This figure illustrates the difficulties in classifying subtle nerve thickening as either normal or thickening. Each subject represents an example of a maximum intensity projection (MIP) and three‐dimensional short‐tau inversion recovery (STIR) image

The κ coefficients for assessment of images performed with 1.5 T and 3.0 T scanners were 0.26 (SE 0.13, 95% CI 0.00–0.56, P = .032) and 0.26 (SE 0.10, 95% CI 0.07–0.49, P = .012), respectively.

DISCUSSION

In this study, raters agreed in 26 of 50 (52%) images, indicating poor reliability. Although agreement was better when data were dichotomized (normal vs abnormal), our results provide evidence that difficulties are related mostly to distinguishing more subtle cases of nerve thickening. Objective criteria for abnormality are required to avoid false positive and false negative results and to optimize the diagnostic value of MRI for inflammatory neuropathies.

The poor agreement may have several explanations. We cannot exclude the possibility that the difference in radiological experience between raters underlies the poor reliability. However, the assessors work in the same department and had comparable training in neuroradiology. This may be an indication that interrater variability could have been even higher had we selected radiologists from different hospitals and training backgrounds. Furthermore, the gap in experience represents current clinical practice. Second, assessors of brachial plexus MRI may lack clear reference points, in particular when abnormalities are two‐sided, which may have caused best‐guessing, particularly in cases from category 2. Third, three categories in the scoring model may have been one category too many. Analysis of dichotomized data led to a slightly higher κ but still indicated a poor level of agreement. Fourth, differences between scanners may partially explain the poor level of agreement. However, despite small groups, our results showed similar agreement with overlapping CIs for assessment of images performed on 1.5 T and 3.0 T scanners.

Earlier studies in chronic inflammatory neuropathies and plexus MRI focused on characteristics, distribution, and prevalence of abnormalities.3, 4, 5 This study addresses the reproducibility of assessment of such abnormalities. Quantification of these abnormalities represents an obvious approach to improve reproducibility and reliability of assessment. Quantification of nerve size allows the identification of patients with CIDP or MMN with high sensitivity and reasonable specificity, as shown in ultrasound studies.10 Few studies have explored the use of quantitative MRI in chronic inflammatory neuropathies. One report described cutoff values of 5.0 mm for roots C6, C7, and C8 to distinguish patients with CIDP (n = 14) from controls (n = 10).11 Sensitivity and specificity were not reported, probably because of the small sample. Researchers in another recent study used the diameter of the ganglia and nerve roots of C5 to T112 and found these to be significantly larger in patients with CIDP (n = 14) than in controls (n = 9), providing evidence to support the feasibility of this approach. However, sensitivity of ganglia measurements was only 48%, despite a specificity of 92%. Sensitivity of root measurements was slightly better at 62%, with 82% specificity. Interrater agreement was good for both ganglia and root measurements. One study did not find any differences in cervical nerve root diameter between patients with CIDP (n = 15) and controls (n = 29).13 Three‐dimensional volume measurements may be another approach. Researchers in a recent study showed increased volume of peripheral nerves in patients with CIDP (n = 13) compared with controls (n = 12) using MRI with diffusion‐weighted whole‐body imaging with background body signal suppression.14 Combined, these studies provide evidence of the potential of a quantitative approach to improve diagnostic reliability.

The potential of imaging techniques for diagnosis of CIDP and MMN has been demonstrated in several recent studies.15, 16, 17, 18 In one of these studies, MRI was abnormal in 22 of 38 (58%) patients with CIDP without definite electrodiagnostic criteria, which led to an adjustment of final diagnosis to definite CIDP in seven patients.15 Additional studies are required to determine reproducible and reliable quantification techniques with optimal sensitivity and specificity to ensure proper diagnosis of treatment‐responsive polyneuropathies.

CONFLICT OF INTEREST

The authors declare no conflicts of interest.

ETHICAL PUBLICATION STATEMENT

We confirm that we have read the Journal's position on issues involved in ethical publication and affirm that this report is consistent with those guidelines.

Supplemental figure 1 The brachial plexus scoring system

Representative images of the scoring system used to define categories of abnormality, i.e. category 1 = no nerve thickening, 2 = possible nerve thickening, 3 = definite nerve thickening. Each category represents an example of a Maximum Intensity Projection (MIP) and 3D short‐tau inversion recovery (STIR) image.

Click here for additional data file.

Supplemental figure 2 ROC curves per rater

Receiver operating characteristic (ROC) curve of rater 1 (red line) and rater 2 (blue line) with an area under the curve (AUC) of 0.698 (95% CI 0.539–0.858) and 0.804 (95% CI 0.667–0.940) respectively. Reference line in green.

Click here for additional data file.