Interobserver agreement for the direct and indirect signs of pulmonary embolism evaluated using contrast enhanced magnetic angiography.

BACKGROUND: Accurate diagnosis of pulmonary embolism (PE) using contrast enhanced MRA (CE-MRA) requires awareness of both the direct and indirect findings of PE.
PURPOSE: To evaluate reader agreement of the direct and indirect findings of PE on CE-MRA.
METHODS: We evaluated pulmonary artery diameter, right ventricle/left ventricle ratio, and clot/vessel lumen signal intensity ratio. Also, eight direct and eight indirect findings of PE were interpreted twice by two radiologists with different experience levels. The prevalence, and intra- and inter-reader agreement for the direct and indirect findings of PE were recorded. Statistical analysis of the measurements was assessed using intraclass correlation while Cohen's kappa test determined inter- and intra-reader agreement.
RESULTS: We reviewed 66 positive CE-MRA exams, 10 of which cases were used for training. The largest PE for each of the remaining 56 cases (40 woman) were included in this analysis (38.9 ± 19.7 (mean age (years) ± S.D.)). The highest interobserver agreement for the direct findings were vessel cutoff (κ = 0.52, 95 % CI = (0.30, 0.74), p < .0001) and bright clot (κ = 0.51, 95 % CI = (0.26, 0.78), p = .0001). The highest interobserver agreement for the indirect findings were for atelectasis (κ = 0.67, 95 % CI = (0.49, 0.87), p < .0001), pleural effusions (κ = 0.56, 95 % CI = (0.32, 0.79), p = 0001) and blank slate sing (κ = 0.56, 95 % CI = (0.18, 0.94), p < .0001).
CONCLUSION: The indirect findings of atelectasis and pleural effusion had better interobserver reproducibility than the direct findings of vessel cutoff and bright clot. The intraobserver reproducibility of the direct and indirect findings is dependent on experience level. SUMMARY STATEMENT: Using contrast enhanced magnetic resonance angiography for the diagnosis of pulmonary embolism, the indirect findings of atelectasis and pleural effusion had better interobserver reproducibility than the direct findings of vessel cutoff and bright clot.

Introduction

Pulmonary embolism (PE) is a leading cause of acute cardiovascular death in the world, following only myocardial infarction and stroke in incidence [1,2]. PE is fatal in up to 30 % of affected patients [3,4]. The mortality rate can be decreased to 8% with proper treatment [5,6]. For these reasons, prompt diagnosis and treatment is critical for optimal patient outcomes.

Computed tomography pulmonary angiography (CTPA) is currently the standard of care for the diagnosis of PE [7]. CTPA can be performed and interpreted rapidly and has resulted in its widespread use in the emergency setting. However, some authors have shown that the use of CTPA has resulted in “over diagnosis” of this disorder [8]. Other limitations of CTPA include increasing concerns about the effects of ionizing radiation, particularly for younger women [[9], [10], [11], [12]]. Further, many patients have contraindications to iodinated contrast, including those with renal failure (eGFR < 30) or a history of allergic/anaphylactoid reaction to iodinated contrast material.

Contrast enhanced magnetic resonance angiography (CE-MRA) is increasingly used for the diagnosis of PE and requires neither iodinated contrast nor ionizing radiation. Over the last ten years, we have performed over 10,000 of these examinations for the diagnosis of PE. Accurate interpretation requires that the radiologist be aware of the many direct and indirect findings of PE as well as artifacts associated with these acquisitions. Many of these findings have a correlate on CTPA exams, while others are unique to CE-MRA (See online supplement).

Therefore, the purpose of this study was to determine (a) the reproducibility of routine measurements on CE-MRA exams in patients with PE, (b) differences between readers of varying experience and (c) which direct and indirect signs of PE had the highest intra- and interobserver agreement.

Materials and methods

Study design

This was a HIPAA compliant and IRB-approved retrospective study. We studied consecutive of patients who underwent CE-MRA for the diagnosis of PE from May 2008–May 2014. All patients had CE-MRA performed for the diagnosis of PE were included in the study. These participants were also evaluated as part of a separate publication on outcomes [13], although the current analysis focused on reader agreement. An initial listing of the patient exams for this study was obtained through a search of the radiology picture archiving and communication system (PACS) and radiology information system (RIS). The search included a query for all thoracic CE-MRA scans and their corresponding final reports within the study period. All CE-MRA exams were reported by sub-specialty trained cardiovascular radiologists all with more than six years of experience with CE-MRA interpretation for the diagnosis of PE (SKN, TMG, CJF, SBR, and MLS). All of the cases used in this study were re-reviewed by consensus (MLS, Reader 1, and Reader 2) to confirm the initial clinical report of a positive PE. All exams that were determined to be positive for PE were included in this analysis. (Fig. 1- Flow chart)

Fig. 1

Participant flow chart for this study. The readers had a joint training session for all of the direct and indirect findings of pulmonary embolism that were enumerated along with practicing the methodology for the measurements of the right ventricle/ left ventricle ratio, main pulmonary artery and the modified method for separating a non-occlusive filling defect from Gibbs’ artifact. (Abbreviations: CE-MRA- contrast enhanced magnetic resonance angiography, PE- pulmonary embolism).

MRA protocol

The MRA imaging protocol used in this study has been previously published [14]. Briefly this included the following pulse acquisitions: (A) localizer 3-plane single-shot fast spin-echo; (B) pre-contrast, pulmonary arterial phase, immediate delayed-phase, and a low flip angle delayed-phase 3D contrast-enhanced T1 weighted MRA with near isotropic spatial resolution and full chest coverage with an interpolated voxel size of 0.7 × 0.7 × 1.0 mm3; and (C) 2D post-contrast fat-saturated T1-weighted spoiled gradient recalled (SGRE) echo images. For intravenous contrast, a weight-based dose of 0.1 mmol/kg of gadobenate dimeglumine (Multihance™, Bracco Diagnostics, Princeton, NJ) diluted to a total volume of 30 mL with saline and power-injected at 1.5 mL/s followed in 20 mL of normal saline injected at the same rate, was used. This protocol is very simple and requires six separate breath holds of about 17 s with 10–20 second rest periods for a total table time of less than 10 min.

Analysis

All CE-MRA exams known to have PE, were evaluated independently by two readers. The readers first had a joint training session consisting of ten cases that was supervised by a proctor (MLS) for each of the measurement methods and the various direct and indirect signs of PE on CE-MRA. A pre-evaluation training session for Reader 2 included 20 training cases: 10 CE-MRA positive for PE and 10 negative for PE. Each reader independently assessed each study twice for the direct and indirect findings in order to assess inter and intra reader agreement using intra class correlation (ICC) (Fig. 1). The reading sessions were separated by two weeks apart to minimize any recall bias. Reader 1 had two years of experience interpreting CE-MRA for the presence of PE. Reader 2 had one month of experience interpreting these exams.

First, each reader assessed measurements that are typically made in the setting of PE stratification for severity of disease burden an indirectly the presence of pulmonary hypertension. These included the following: the clot/vessel lumen signal intensity ratio separating PE from the Gibbs artifact [15], the pulmonary artery (PA) diameter and right ventricle/ left ventricle (RV/LV) short axis ratio (See online supplement). ICC was employed to determine intra and inter reader agreement for these measurable variables.

Secondly, each study was assessed for the presence of PE with false positive exams excluded from further analysis. The exam was re-evaluated by the two readers along with an additional more experienced reader (MLS) to reach a consensus about the presence or absence of the PE. The largest PE for each exam determined by consensus was included in the study and assessed for the presence of each of the direct and indirect findings [16] (See online supplement).

The following direct findings of PE were tallied by each reader on two separate occasions (See online supplement): (a) non occlusive filling defect within the enhanced pulmonary artery lumen, (b) occlusive filling defect completely obstructing the pulmonary artery, (c) vessel cutoff on thin-slab maximum intensity projection images and/or on direct coronal CE-MRA exams, (d) double bronchus sign, (e) central dot sign, (f) ghost vessel sign, (g) bright clot sign and (h) a filling defect within the pulmonary artery found on the post gadolinium fat saturated SGRE images (this sequence is employed to survey the chest for any ancillary findings) [16] (Table 1, Fig. 2). The following indirect findings were included (see online supplement): (a) pulmonary venous stasis, (b) atelectasis, (c) perfusion defect, (d) pleural effusion, (e) visceral pleural enhancement, (f) white-black-white sign, (g) pulmonary infarction and (h) blank slate sign [16] (Table 1, Fig. 3). The direct findings of PE only found on MRI/MRA include: double bronchus sign, central dot sign, ghost vessel sign, and bright clot sign [16]. There are also unique indirect findings of PE at CE-MRA include: visceral pleural enhancement, white-black-white sign, and blank slate sign [16]. Thus, each reader reviewed 56 patients (mean age (yrs.) ± (SD) = 37.2 ± 17.8 (40 woman), 43.3 ± 12.8 (16 man)) with 56 pulmonary emboli for the presence or absence of eight indirect and eight direct findings of PE on two separate occasions. In total, for the two readers reading each case twice, there were 3584 data entry fields (56 PE * (8 indirect +8 direct findings) * 2 readers* two reading sessions = 3584 data fields) that were manually entered onto spread sheets.

Table 1

Definition of direct and indirect findings of pulmonary embolism on MRA.

Direct findings of PE on MRA	Definition
Filling defect in the pulmonary arteries	Low signal intensity clot in the pulmonary arteries after contrast enhancement
Vessel cutoff sign	Vessel completely obstructed and amputated on Maximum Intensity Projection images
Double bronchus signa	Two low signal intensity structures in cross-section- one representing a bronchus and the other representing the occlusive thrombus
Central dot signa	Central high intensity in the clot
Ghost vessel signa	Enhancement of a vessel wall surrounding an obstructing embolus on delayed contrast enhances MRA images
Bright clot signa	High T1 signal intensity from intraluminal methemoglobin on a pre-contrast image
Indirect findings of PE on MRA
Pulmonary venous stasis	Higher signal intensity vein related to slow flow and delayed timing of the peak enhancement time course of contrast through this vein
Atelectasis	Strong enhanced lung reflects collapsed lung
Perfusion defects	Wedge shaped areas of low signal intensity due to loss of lung parenchymal enhancement
Visceral pleural enhancementa	liner enhancement along lung surface
White-black-white signa	A perfusion defect surrounded by enhancing lung
Pulmonary infarction	High T2 weighted wedge shaped area of lung parenchyma without enhancement
Blank slate signa	A large area of completely black lung that has no signal intensity with in it

Abbreviations: PE-pulmonary embolism, MRA- magnetic resonance angiography.

Findings unique for contrast enhanced MRA.

Fig. 2

Direct findings of pulmonary embolism (PE) at contrast enhanced pulmonary magnetic resonance angiography (CE-MRA): (A) Axial CE-MRA showing an occlusive filling defect in right pulmonary artery (arrow) and non-occlusive filling defect in left pulmonary artery(dotted arrow); (B) Coronal MRA at 1.25 mm showing vessel cutoff of the truncus anterior (upper arrow) and interlobar (lower arrow) pulmonary arteries by PE; (C) Axial post contrast fat saturated T1-wieghted spoiled gradient recalled echo images showing the “double bronchus sign” -neighboring two low signal intensity oval structures with the medial one (straight arrow) being an occlusive PE in the right pulmonary artery and the lateral one (dotted arrow) the right main bronchus; (D) Axial CE-MRA showing the “central dot” sign high intensity in clot which helps to distinguish clot from a Gibbs’ artifact; (E) Axial delayed phase CE-MRA showing the “ghost vessel” sign wherein there is delayed enhancement of the wall surrounding the occlusive clot, which is likely related to inflammation within the vessel wall secondary to the presence of the thrombus; (F) Coronal pre contrast T1 weighted MRA showing the “bright clot sign”-high intensity PEs (arrow) from methemoglobin of the clot in the left and right lower lobe pulmonary arteries.

Fig. 3

Indirect signs of pulmonary embolism (PE) found on contrast enhanced pulmonary magnetic resonance angiography (CE-MRA): (A) Coronal CE-MRA showing a large perfusion defect in the right lower lobe as areas of very low signal intensity with a lack of vasculature (arrow, blank slate sign) and a smaller one in the left lower lobe with vessel enhancement (dashed arrow, simple perfusion defect); (B) Coronal CE-MRA showing the “white-black-white” sign of a perfusion defect (within the circle “B”) surrounded on both sides by normally perfused lung (within the circle “W”); (C) Axial post contrast fat saturated spoiled gradient echo image showing a pleural effusion (dashed arrow), compressive atelectasis (straight arrow) and a non-occlusive PE (arrow-PE); (D) Axial CE-MRA showing the enhancement of the right parietal pleural surface (dashed arrow) and PE in the right lower love pulmonary artery; (E) Axial T2-weighted image showing pulmonary infarction detected as a high signal intensity area (arrow); (F) Axial CE MRA of the respective pulmonary vein draining the right lower lobe posterior segment with high signal within the draining vein (dashed arrow) due to a slower transit time when compared to the contralateral left lower lobe pulmonary vein (arrow) without a PE.

Statistical analysis

Statistical analysis was conducted by a statistician (CL). For the continuous measurements, agreement was assessed via the ICC using the ‘psych’ package in R (V 3.6.1) [17,18]. For the discrete measurements, agreement was assessed via an unweighted kappa statistic using the ‘irr’ package [19]. All point estimates were accompanied by parametrically estimated 95 % confidence intervals (95 % CI) and p-values. Due to the exploratory nature of this study, no multiple testing corrections were applied to calculated p-values. A p-value less than 0.05 was considered statistically significant.

Results

Between May 2008 and May 2014 there were 902 CE-MRA studies performed to assess for the presence of PE. Of these studies, 69 were initially interpreted as positive for PE. The image quality was limited for 3 of these exams (4%). Ten studies were used to create a training set and the remaining 56 studies were included in the analysis. Upon consensus review, 29 patients had a single PE, 17 had two PEs and the remaining 10 patients had three or more PEs. There was no false positive PE exam in the 56 studies. Only the largest PE, as determined by consensus for each patient, was analyzed for direct and indirect findings. There were three studies without pre-contrast T1 weighted images, one study without a delayed-phase contrast-enhanced T1 weighted MRA and 19 studies without a post-contrast fat-saturated T1-weighted SGRE acquisition.

Continuous variables

The ICCs for the three continuous measurable variables described above (clot/vessel lumen signal intensity ratio, RV/LV short axis ratio and the main PA diameter) are shown in Table 2. Bland-Altman plots for reader variability of these variables are shown in Fig. 4. The inter-observer agreement for the RV/LV ratio (ICC = 0.65, 95 % CI = (0.48, 0.78), p < .0001) and main PA diameters (ICC = 0.66, 95 % CI = (0.49, 0.79), p < .0001) were very good with high intra-observer agreement as well.

Table 2

Intra and Interobserver agreement between two separate readings made by the experienced and novice observer for the presence of three continuous variables in contrast enhanced pulmonary angiography exams know to be positive for pulmonary embolism.

CE-MRA Imaging Measurement	Number of embolimeasured	ICC	95 % C.I.	p value
Clot/vessel ratio (R1)	54	0.38	0.12−0.58	0.0022
Clot/vessel ratio (R2)	54	0.31	0.06−0.54	0.0093
Clot/vessel ratio (Interobserver)	54	0.40	0.15−0.60	0.0011
Main PA (R1)	56	0.88	0.81−0.93	<.00001
Main PA (R2)	56	0.81	0.70−0.89	<.00001
Main PA(Interobserver)	56	0.66	0.49−0.79	<.00001
RV/LV Ratio (R1)	56	0.88	0.80−0.93	<.00001
RV/LV Ratio (R2)	56	0.78	0.65−0.86	<.00001
RV/LV Ratio(Interobserver)	56	0.65	0.48−0.78	<.00001

Abbreviations: R1- Reader one, R2- Reader 2, Intraobserver- Interobserver variability, Clot/vessel ratio- clot/vessel lumen signal intensity ratio, Main PA- transverse measurement of the main pulmonary artery maximum diameter, RV/LV ratio- the modified 4 chamber (straight axial) measurement of right ventricle/left ventricle minor axis ratio, ICC- intraclass correlation coefficient, 95 % C.I.−95% confidence interval about estimated ICC, p value – p-value from subsequent hypothesis test.

Fig. 4

Bland-Altman plot of reader variability for (a) clot/vessel lumen signal intensity ratio, (b) main pulmonary artery, (c) Right ventricle /Left ventricle ratio. The bias is the average pairwise difference between readers, the p-value comes from a paired t-test, and the range is the difference between the upper and lower confidence limits.

Direct findings

The Cohen kappa values for the inter- and intra-observer agreement for the eight direct findings of PE at CE-MRA (see Table 1, Fig. 2 and online supplement) for the two readers are shown in Table 3 along with the prevalence of each finding. The two direct findings of PE with the highest inter-observer agreement in the direct findings of PE were vessel cutoff on MRA (κ = 0.52, 95 % CI = (0.30, 0.74), p = .0001) and bright clot sign (κ = 0.51, 95 % CI = (0.26, 0.78), p = .0001). These findings also had a very high intra-reader agreement (vessel cutoff: κ = 0.44, 95 % CI = (0.20, 0.68), p < .001, κ = 0.48, 95 % CI = (0.25, 0.70), p < .0002, respectively) (bright clot sign: κ = 0.67, 95 % CI = (0.45, 0.89), p < .0001, κ = 0.64, 95 % CI = (0.41, 0.86), p < .0001, respectively). The two direct findings with the highest prevalence were the occlusive filling defect (N = 43) and vessel cutoff (N = 42).

Table 3

Experienced and novice observer agreement for the direct findings of pulmonary embolism found on contrast enhanced pulmonary magnetic resonance angiography.

Direct Finding of PE at CE-MRA	ConsensusNumber of findings	ReaderPrevalencetime 1, time 2	N	kappa	95 % C.I.	P value
Non-occlusive, R1	12	19,20	56	0.88	0.75−1.00	<0.0001
Non-occlusive, R2	12	14,16	56	0.46	0.19−0.72	0.0006
Non-occlusive, Interobserver	12		56	0.36	0.10−0.62	0.0056
Occlusive, R1	43	36,35	56	0.88	0.76−1.00	<0.0001
Occlusive, R2	43	38,39	56	0.54	0.31−0.78	<0.0001
Occlusive, Interobserver	43		56	0.44	0.20−0.69	0.0009
Vessel Cutoff, R1	42	36,33	56	0.44	0.20−0.68	0.001
Vessel Cutoff, R2	42	31,39	56	0.48	0.25−0.70	0.0002
Vessel Cutoff, Interobserver	42		56	0.52	0.30−0.74	0.0001
Double Bronchus, R1	26	24,27	56	0.53	0.31−0.75	0.0001
Double Bronchus, R2	26	10,13	56	0.77	0.51−0.95	<0.0001
Double Bronchus, Interobserver	26		56	0.45	0.24−0.65	0.0001
Central Dot, R1	34	21,16	56	0.32	0.07−0.58	0.01
Central Dot, R2	34	32,25	56	0.54	0.33−0.75	<0.005
Central Dot, Interobserver	34		56	0.34	0.12−0.57	<0.0001
Ghost Vessel, R1	31	23,17	55	0.54	0.31−0.76	<0.0001
Ghost Vessel, R2	31	34,36	55	0.53	0.39−0.83	<0.0001
Ghost Vessel, Interobserver	31		55	0.27	0.27−0.69	0.0001
Bright clot, R1	18	16,13	53	0.67	0.45−0.89	<0.0001
Bright clot, R2	18	12,18	53	0.64	0.41−0.86	<0.0001
Bright clot, Interobserver	18		53	0.51	0.26−0.78	0.0001
Filling Defect on FS-SGRE, R1	28	28,31	34	0.62	0.24−1.00	0.0001
Filling Defect on FS-SGRE, R2	28	28,28	34	0.60	0.24−0.95	0.0005
Filling Defect, Interobserver	28		34	0.19	−0.20−0.58	0.3

Abbreviations: PE- pulmonary embolism, CE-MRA- contrast enhanced magnetic resonance angiography, C.I.- confidence interval,R1- reader 1 with 2 years of experience reading CE-MRA, R2- reader 2 with one month of experience reading CE-MRA. FS-SGRE – Fat saturated spoiled gradient echo sequence.

Indirect findings

The inter- and intra-observer agreement results for the eight indirect findings of PE at CE-MRA (see Table 1, Fig. 3 and online supplement) are summarized in Table 4. The indirect finding with the highest inter-observer agreement was the presence of atelectasis (κ = 0.67, 95 % CI = (0.49, 0.87), p < .0001). The two findings with the next highest inter-observer agreement were the presence of pleural effusion (κ = 0.56, 95 % CI = (0.33, 0.79), p < .0001) and the blank slate sign (κ = 0.56, 95 % CI = (0.18, 0.94), p < .0001). Intra-observer agreement for these variables was also high for these variables (atelectasis: κ = 0.54, 95 % CI = (0.32, 0.76), p < .0001, κ = 0.82, 95% CI = (0.66, 0.97), p < .0001, respectively) (pleural effusion: κ = 0.79, 95% CI = (0.62, 0.97), p < .0001, κ = 0.88, 95% CI = (0.75, 1.0), p < .0001, respectively) (blank slate sign: κ = 0.64, 95% CI = (0.26, 1.0), p < .0001, κ = 1.0, 95% CI = (1.0, 1.0), p < .0001, respectively). The two indirect findings with the highest prevalence were perfusion defects (N = 42) and the white-black-white sign (N = 34).

Table 4

Experienced and novice observer agreement for the indirect findings of PE found on contrast enhanced pulmonary magnetic resonance angiography.

Indirect Finding of PE at MRA	Consensus number of findings	Prevalence	N	kappa	95 % C.I.	P value
Pulmonary venous stasis, R1	10	11,12	56	0.40	0.11−0.69	0.003
Pulmonary venous stasis, R2	10	3,2	56	0.37	−0.18−0.93	0.004
Pulmonary venous stasisInter-reader	10		56	0.06	−0.18−0.31	0.5
Atelectasis, R1	21	27,32	56	0.54	0.32−0.76	<0.0001
Atelectasis, R1	21	24,23	56	0.81	0.66−0.97	<0.0001
AtelectasisInter-reader	21		56	0.67	0.49−0.87	<0.0001
Perfusion Defect, R1	42	39,37	56	0.59	0.37−0.82	<0.0001
Perfusion Defect, R2	42	37,41	56	0.66	0.45−0.87	<0.0001
Perfusion Defect,Inter-reader	42		56	0.51	0.27−0.75	0.0001
Effusion, R1	21	17,18	56	0.79	0.62−0.97	<0.0001
Effusion, R2	21	20,19	56	0.88	0.75−1.0	<0.0001
Effusion,Inter-reader	21		56	0.56	0.32−0.79	<0.0001
Pleural Enhancement, R1	22	10,14	56	0.47	0.20−0.75	0.0003
Pleural Enhancement, R2	22	23,19	56	0.70	0.51−0.88	<0.0001
Pleural Enhancement,Inter-reader	22		56	0.31	0.09−0.54	0.0058
White-Black-White, R1	34	22,14	56	0.60	0.39−0.81	<0.0001
White-Black-White, R2	34	20,25	56	0.67	0.47−0.86	<0.0001
White-Black-White,Inter-reader	34		56	0.47	0.23−0.71	0.0005
Pulmonary Infarction, R1	12	11,12	54	0.94	0.84−1.0	<0.0001
Pulmonary Infarction, R2	12	6,7	54	0.74	0.46−1.0	<0.0001
Pulmonary Infarction,Inter-reader	12		54	0.52	0.22−0.82	<0.0001
Blank Slate, R1	5	4,5	56	0.64	0.26−1.0	<0.0001
Blank Slate, R2	5	6,6	56	1.0	1.0−1.0	<0.0001
Blank Slate,Inter-reader	5		56	0.56	0.18−0.94	<0.0001

Abbreviations: PE- pulmonary embolism, C.I.- confidence interval, R1- reader 1 with 2 years of experience reading CE-MRA, R2- reader 2 with one month of experience reading CE-MRA.

Discussion

In this work we investigated that the high reproducible observations of pulmonary embolism on contrast enhanced magnetic resonance angiography were the presence of a vessel cutoff and bright clot sign for the direct findings, and the presence of atelectasis, pleural effusion and blank slate sign for the indirect findings. The interobserver reproducibility of the indirect findings were better than the direct findings.

For the continuous variables that are well defined CE-MRA measurements, we found that there was good inter- and intra-observer agreement for the main PA diameter and the RV/LV ratio [20]. The clot/vessel lumen signal intensity ratio for separating PE from the Gibbs artifact did not perform quite as well but was still a reproducible metric. This method was originally described by Bannas et al [15], and there is no reproducibility study for this method. For the nominative observations that were studied, high reproducible inter observations associated with PE on CE MRA includes both common findings with CTPA (a vessel cutoff and atelectasis and pleural effusion) and unique findings on CE MRA, with no direct correlate on CTPA (bright clot sing and blank slate sign). Recognition of these new signs may be helpful for the interpreting radiologist, as pulmonary emboli may be very difficult to directly observe at CE-MRA. The intra observer agreement for each direct and indirect finding was different by the experience level of readers. It is not surprising that there is variability in the interpretation of CE-MRA exams for the direct and indirect signs for PE. With respect to prior work in determining reader agreement, Bannas et al showed that in an animal model of PE, there was not the same efficacy [21]. In that study, Reader 1 had a sensitivity of 100% (11/11) and a specificity of 100% (13/13) for PE detection on a per-animal basis for both MR angiography and ultrashort TE [21]. While Reader 2 had a sensitivity of 91% and specificity of and 92% [21]. There is variation in the diagnosis of PE at CTPA as well [22,23]. The correct interpretation of pulmonary MRA examinations for the diagnosis of PE is dependent on experience level and knowledge of the direct and indirect findings unique to this modality. The combined use of direct and indirect findings is key to observing the presence of emboli on this exam. These data confirm the need for appropriate training for accurate interpretation of PE using the CE-MRA.

There are limitations to this study. First, the 10 training cases were chosen randomly and may not have provided clear cut examples of each finding making it more challenging for the novice reader to become comfortable with less familiar findings. Second, additional training of the novice reader prior to beginning this observational study may have been helpful. Third, the ground truth for this study was the final radiology report and a consensus read, however, there was not a second confirmatory imaging exam showing the presence of PE. Related to this, it is also likely that some patient with a true PE, but a negative CE-MRA exam were not included in this study. Fourth, the readers were instructed to choose the largest embolus when there was more than one present on an exam. It is possible that in some cases the two readers did not pick the same embolus for their observations and clot/vessel lumen signal intensity ratio measurement. Fifth, additional readers would have been helpful to more completely determine the intra- and inter-reader variabilities for these findings.

In summary, (a) The reproducibility of the measurable quantities (main PA diameter, RV/LV ratio and clot/vessel lumen signal intensity ratio) on contrast enhanced magnetic resonance angiography exams in patients with pulmonary embolism was good, (b) The intraobserver agreement of the direct and indirect findings was dependent on experience level. and (c) The high interobserver agreements were shown in vessel cutoff and bright clot for the direct findings, atelectasis, pleural effusion and blank slate sign for the indirect findings.