Variability of radiological grading of blunt cerebrovascular injuries in trauma patients.

CONTEXT: Blunt cerebrovascular injury (BCVI) occurs in 1%-2% of all blunt trauma patients. Computed tomographic angiography of the neck (CTAn) is commonly used for the diagnosis and grading of BCVIs. Grade of injury dictates treatment, and there remains a lack in understanding the inter-reader reliability of these interpretations. AIMS: The aim of this study is to determine the extent of variability in BCVIs among specialized neuroradiologist interpretation of CTAn. SETTINGS AND
DESIGN: Retrospective review of trauma patients admitted to a level one trauma center with a BCVI from January 2012 to December 2017. Patients were randomly assigned for CTAn re-evaluation by two of three blinded, neuroradiologists. STATISTICAL ANALYSIS USED: The variability in BCVI grades was measured using the coefficient of unalikeability (u), and inter-reader reliability was calculated using weighted Cohen's kappa (k).
RESULTS: Two hundred and twenty-eight BCVIs were analyzed with initial grades of 71 (31%) grade one, 74 (32%) grade two, 26 (11%) grade three, 57 (25%) grade four, and 0 grade five. Variability was present in 93 (41%) of all BCVIs. Grade one injuries had the lowest occurrence of total agreement (31%) followed by grade three (61%), grade two (63%), and grade four (92%). Total variability of grade interpretations (u = 100) occurred most frequently with grade three BCVIs (21%). Weighted Cohen's k calculations had a mean of 0.07, indicating poor reader agreement.
CONCLUSIONS: This novel study demonstrated the BCVI variability of radiological grade interpretation occurs in more than a third of patients. The reliability of CTAn interpretation of BCVI grades is not uniform, potentially leading to undertreatment and overtreatment. Copyright:

INTRODUCTION

Traumatic blunt cerebrovascular injuries (BCVIs) have a risk of stroke that increases as a function of injury grade.[12] The pathophysiology of injury relates to endothelial damage-causing ischemic neurologic sequela from an embolic source; treatment ranges from antiplatelet therapy to anticoagulation based on grade and algorithm.[13456]

BCVI screening criteria include high-risk mechanisms and computed tomographic angiography of the neck (CTAn) is the preferred modality, although the reliability of grade is not understood [Table 1].[7891011] We hypothesized there would be poor agreement among radiologists interpretation of BCVI grades; thus, changes in treatment recommendations could ensue.

Table 1

Blunt cerebrovascular injury grading defined by angiographic findings

	Grade 1	Grade 2	Grade 3	Grade 4	Grade 5
Intimal injury	Irregularity	Intramural hematoma	Pseudoaneurysm	Occlusion	Transection, extravasation
Luminal narrowing (%)	<25	>25	-	100	-

METHODS

This was a retrospective study of patients at an urban level one trauma center from January 1, 2012, to December 31, 2017. The study was approved by the OhioHealth institutional review board before initiation. All adult trauma patients that suffered a blunt mechanism and obtained a CTAn study were included in the analysis. Patients with no CTAn evaluation, who were prisoners, who were pregnant, or who were <16 years of age were excluded from the study. Data were obtained from a combination of electronic medical records and institutional trauma databases.

In all cases, CTAn was performed with multidetector scans (three total in hospital computed tomographys [CTs]: One Cannon 320 and two GE Medical Systems Optima CT64) with at least three detectors and a low radiation dose. CT acquisition parameters included 120 kV and 85–560 effective mAs (milliampere-seconds/pitch). Slice thickness is 0.625 mm on each machine with a reconstructed slice thickness of 2.0 mm. Maximum intensity projections were also performed in the sagittal and coronal planes. Isovue-370 intravenous contrast was the contrast used in all patients; 75 ml of contrast is injected and image capturing begins once the contrast reaches the carotid arteries for all CTA imaging that includes a CTAn.

Patients who met inclusion criteria and were originally diagnosed with a BCVI (all grades included) were divided randomly yet evenly into three BCVI groupings (A, B, C). Three neuroradiologists were assigned two of the groupings (AB, BC, and AC) so that each CTAn was independently interpreted by two neuroradiologists. All neuroradiologists were blinded to the original BCVI grades assigned at the initial clinical interpretation which was recorded from the medical record. The three neuroradiologists used only the patients' CTAn as patients did not receive conventional angiography or magnetic resonance angiography (MRA) in this study.

We compared the BCVI grades reported from the re-evaluations to both each other and the initial evaluation and analyzed the degree and frequency of grade variability. Given that BCVI grades are ordinal values, variability was quantified using the coefficient of unalikeability (u) discussed by Kader and Perry.[12] The scale was adjusted from 0 to 1–100 where the lower the value, the more alike the data. The data were entered into an Excel spreadsheet (2013; Microsoft, Redmond, Washington), and u was calculated for each CTAn read that met inclusion criteria. Coefficient values of 1, 67, or 100 were calculated which represent no variability, some variability and total variability, respectively, for each BCVI grade. The frequency of each coefficient was reported as a percentage for all BCVIs lesions, isolated carotid vessel injuries, and isolated vertebral artery injuries.

Significant BCVI grade differences were defined as those that would directly change recommended clinical treatment based on institution treatment guidelines [Appendix A]. The significant grade differences were categorized as: (1) grade zero (no identified BCVI) versus any grade; (2) grades one or two versus grades three, four, or five; and (3) grades one, two, three versus grades 4 or 5. These significant grade differences were not analyzed in relation to specific vessel involvement (carotid vs. vertebral) or vertebral level, as neither specification would affect clinical management recommendations.

Inter-reader agreement (AB, BC, and AC) was calculated using weighted Cohen's kappa (k). We used value characterization as defined by Landis and Koch: <0 = no agreement, 0–0.20 = slight agreement, 0.21–0.40 = fair agreement, 0.41–0.60 = moderate agreement, 0.61–0.80 = substantial agreement, and 0.81–1.0 = near perfect agreement.[1314]

RESULTS

Our study started with 222 patients who originally met inclusion criteria. Five patients were excluded due to a combination of chronic lesions and the lack of additional radiologist interpretation. Therefore, we had a total of 228 BCVIs in 217 patients. The majority of the BCVIs diagnosed occurred after a motor vehicle collision (48.6%), followed by falls (26.4%) and assaults (6.0%). Other mechanisms included motorcycle collisions, pedestrian versus auto, ATV accidents, bicycle accidents, and encounters with animals. The majority of our patients were male (63.9%) and Caucasian (84.3%). The mean age was 49 years old ± 21 years. When examined by trauma acuity, 84.2% of patients had full trauma activation and 14.8% were trauma consultations. The initial grades consisted of 71 (31%) grade one, 74 (32%) grade two, 26 (11%) grade three, 57 (25%) grade four, and zero grade five. Seventy-six (33%) involved only the carotid vessels, 144 (63%) involved only the vertebral vessels, and eight (4%) involved both vessels.

Among all the patient groups (AB, BC, and AC), read variability was the highest among grade one lesions with agreement (u = 1) occurring an average of only 31.0% ± 10.5% [Figure 1]. Also for grade one injuries, incomplete agreement occurred an average of 55.7% ± 5.68% and complete disagreement an average of 13.3% ± 7.73%. Grade two and three lesions had similar variability with complete agreement occurring an average of 62.8% ± 8.79% and 61.4% ± 11.0%, respectively. Grade three lesions also had a complete disagreement (u = 100) an average of 20.5% ±12.8. Grade four lesions had the least amount of variability, with complete agreement occurring an average of 91.6% ± 4.50%. The average frequencies of BCVI grade variability for the carotid and vertebral injuries did not significantly differ with one exception [Figure 2]. There was a significant difference in complete agreement among grade 4 lesions (carotid 65.0% vs. vertebral 97.2%; P < 0.01). Among both vessels, variability continued to be highest among grade one lesions.

Figure 1

Mean frequency of blunt cerebrovascular injury grade variability for carotid and vertebral vessels combined

Figure 2

Frequency averages of blunt cerebrovascular injury grade variability for carotid vessels (top) and vertebral vessels (bottom).u= coefficient of unalikeability

Analysis of BCVI grade differences revealed a total of 65 (30%) injury interpretations when the difference would have changed clinical management. Grade zero versus any other grade occurred among 39 (17%) of the reads, grades one or two versus grades three or four occurred among 25 (11%) of the reads, and grades one, two, or three versus grade four occurred among nine (4%) of the interpretations. These differences would have changed treatment in 30% of patients, with treatment scope downgraded in 22% of patients and upgraded in 8%.

Weighted Cohen's k for each BCVI grade among each patient group (AB, BC, AC) was calculated and showed a very poor overall inter-reader agreement. As shown in Figure 3, the mean k was 0.19 for group AB. This indicated slight agreement and was the strongest degree of agreement among all the groups. The mean k was − 0.04 for group AC, indicating no agreement, and the mean k was 0.06 for group BC, indicating slight agreement.

Figure 3

Inter-reader agreement for blunt cerebrovascular injury grades

DISCUSSION

The results of this study show significant inter-reader variability exists among neuroradiologists interpreting BCVI using CTAn in trauma patients. The degree of disagreement was an indirect function of BCVI grade with initial grade one injuries having the largest amount of inter-reader variability. This was reflected in both the weighted Cohen's k analysis and the coefficient of u. Measuring both the weighted Cohen's k and coefficient of u strengthens our overall findings in two ways. First, the coefficient of u analyzed the incidence of difference among these groups but did not account for the grade differences. By using the second form of analysis weighted Cohen's k, we were able to evaluate for consistency and account for the grade differences amongst the BCVI reads.

Grade four injuries to both carotid and vertebral arteries had the most amount of agreement which is important as the treatment of these injuries includes full anticoagulation at this institution. For grades one and two, the large negative weight Cohen's k value would suggest the reads differed more than what would be read by chance. In fact, some of the disagreement found original grade one or two injuries were interpreted as grade zero on the reevaluations, indicating no BCVI was present. When deciding to treat these injuries, knowledge that the interpretation is not consistent should impact decision-making discussions. Our data on the high amount of disagreement among low-grade BVCIs zero to two should make the clinician skeptical about the diagnosis of these injuries. The clinician not only has to weight the risks of not treating a BCVI (future stroke) but also the risk of treating the BCVI (hemorrhage, coagulopathy, and bleeding risk).

Downgrading or upgrading a BCVI could mean clinically the patient's treatment recommendation would change. The study showed that 36% of patients could have been downgraded to a grade zero (no BCVI) based on the initial CTAn reevaluation and subsequently receive no treatment. For grade four injuries, 7% of these patients could be downgraded, which means 7% of these patients may have been fully anticoagulated without additional benefit. For grade three injuries, 7.8% of patients could have been upgraded to a grade four, meaning they may have benefitted from full anticoagulation. For grade two BCVIs, 13.5% of patients could have been upgraded to a grade three or four, meaning these patients may have benefitted from additional surgical subspecialty input or full anticoagulation. These significant upgrades and downgrades raise many questions about the reliability of our radiological interpretation of BCVIs.

Although this study of variability among radiologist interpretation of BCVI is not found in the current literature, some studies have examined interobserver agreement in other types of radiologic imaging. A study published in Radiology in 2013 examined the interobserver variability among the diagnosis of honeycombing on chest CTs. Watadani et al. found a weighted Cohen's k value range of 0.4 up to 0.58 for expert chest radiologist.[15] Interpretation of honeycombing on chest CT has similarities to the diagnosis of BCVIs in that both are diagnosed based on grading systems: Fleischner society guidelines for honeycombing and Denver grading for BCVIs. Comparing this to our BCVI study, only one grade four BCVI group had the degree of agreement published in the above study.

Chun et al. examined inter-reader variability of CT diagnosis of traumatic brain injury (TBI). This study evaluated the head CTs of 50 trauma patients with blunt mechanisms and graded their TBI injuries using both the Marshall CT classification and Rotterdam CT classification. The average Cohen's k score for TBI classification was 0.568, indicating moderate agreement.[16] Once again compared to our study of agreement in BCVIs, grade four injuries had an agreement of 0.4 at best. This study was similar in nature to our study as the diagnosis of the injury is solely based on radiological findings. All other grades of BCVIs had less than moderate agreement in our study. Although the two cited studies above do not directly examine BCVI diagnosis, both have an inter-reader agreement greater than what was found in our study, indicating that we are less certain of our BCVI grading than other conditions diagnosed by CT evaluation.

Our study has several limitations. As a single institution study, the data represents the imaging and radiological interpretation of one radiologist group. Secondary to its retrospective design, this study also does not address the actual clinical impact of inter-reader variability. Our other limitations related to the CT scanners and contrast timing. At our institution, we have two different CT scanners. Although all the reconstructed slice thicknesses were the same, the actual scanners are slightly different. Each of our patients in this study received the same contrast, but we were unable to control for the timing of this contrast; as the contrast timing and scanner type can affect image quality. CT scans are also affected by motion, bone, and metallic artifact as well as possible pathologies involving atherosclerotic plaques.[17] Another area of limitation is the use of only one type of imaging. MRA and Doppler ultrasound have been shown to have beneficial secondary roles in patients with BCVI diagnosis.[1718] CTAn does not account for the active arterial flow as Doppler ultrasound does, addition of US or MRA could be a future area of study.[18] Future directions of this study could include prospective analysis to analyze optimal timing of contrast to minimize variability as well as investigate the clinical impact of BCVI read variability. Another future study would be to compare variability in BCVI diagnosis between CTAn and conventional angiography or MRA.

CONCLUSIONS

BCVIs are more common than originally thought and with the liberalized screening and the use of CTAn, these injuries are more often identified.[8910] Clinical decision-making about BCVIs for trauma patients is multifactorial and often complicated in patients with multiple injuries. This study illustrates the variability and frequency among the radiologic interpretation of BCVIs. This study is meant to highlight problems with diagnosing and grading BCVIs that may have a significant clinical impact. This study raises questions related to additional imaging to confirm the injury before treatment, especially in grade one and grade four carotid BCVIs.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.

Research quality and ethics statement

This study was approved by the Institutional Review Board / Ethics Committee. The authors followed applicable EQUATOR Network (http://www.equator-network.org/) guidelines during the conduct of this research project.