Assessment of cervical spine CT scans by emergency physicians: A comparative diagnostic accuracy study in a non-clinical setting.

OBJECTIVES: To determine and compare the diagnostic accuracy of assessing injuries on cervical spine computed tomography (CT) scans by trained emergency physicians and radiologists, both in a non-clinical setting.
METHODS: In this comparative diagnostic accuracy study, 411 cervical spine CT scans, of which 120 contained injuries (fractures and/or dislocations), were divided into 8 subsets. Eight emergency physicians received focused training and assessed 1 subset each before and after training. Four radiologists assessed 2 subsets each. Diagnostic accuracy between both groups was compared. The reference standard used was a multiverified data set, assessed by radiologists, neurosurgeons, and emergency physicians. The neurosurgeons also classified whether an "injury in need of stabilizing therapy" (IST) was present.
RESULTS: Posttraining, the emergency physicians demonstrated increased sensitivity and specificity for identifying cervical spine injuries compared to pretraining: sensitivity 88% (95% confidence interval [CI] 80% to 93%) versus 80% (95% CI 72% to 87%) and specificity 89% (95% CI 85% to 93%) versus 86% (95% CI 81% to 89%). When comparing the trained emergency physicians to the group of radiologists, no difference in sensitivity was found, 88% (95% CI 80% to 83%); however, the radiologists showed a significantly higher specificity (P < 0.01): 99% (95% CI 96% to 100%). In the 12% (15 scans) with missed injuries, emergency physicians missed more ISTs than radiologists, 6 versus 4 scans; however, this difference was not significant (P = 0.45).
CONCLUSION: After focused training and in a non-clinical setting, no significant difference was found between emergency physicians and radiologists in ruling out cervical spine injuries; however, the radiologists achieved a significantly higher specificity.

INTRODUCTION

Background

Cervical spine injuries occur in 2% to 6% of blunt trauma patients. , , Because computed tomography (CT) has a higher sensitivity compared to conventional radiography in identifying cervical spine injury (98% vs 54%) , current international guidelines recommend a cervical spine CT as primary imaging modality for assessing cervical spine injury. , ,

Importance

The objective of trauma care is to decrease morbidity and mortality, which is expected to be achieved by fast, systematic assessment and treatment of the injured patient. In general, emergency physicians determine whether imaging of the cervical spine in a trauma patient is required; subsequently, a radiologist assesses the cervical spine CT scan. However, availability of radiological expertise varies per country and even per hospital; from a 24/7 presence of a radiologist at the emergency department to after‐hours coverage through remote consults with a radiologist on call for multiple hospitals. Additionally, more and more radiology services have been outsourced to external teleradiology services, because of an international shortage of radiologists. ,

Over the past 2 decades, the number of cervical spine CT scans increased considerably , as a result of increased availability of CT scanners, implementing a cervical spine CT as primary imaging modality, and centralizing trauma care to designated trauma centers.

Involvement of emergency physicians in the primary assessment of cervical spine CTs may have potential benefits, such as a faster evaluation of trauma patients in centers where scans are not interpreted by radiologists in real‐time. This could lead to greater autonomy and flexibility for the emergency physician to direct patient flow. Furthermore, because they have examined the patient, emergency physicians are aware of the trauma mechanism and clinical presentation. Assessment of cervical spine CTs by emergency physicians could stimulate collaboration and joint clinical decision‐making between emergency physicians, radiologists, and neurosurgeons.

In order to implement this, emergency physicians must be able to safely rule out cervical spine injuries and display a sensitivity comparable to radiologists. Two previous studies showed that the sensitivity of emergency physicians in ruling out cervical spine injuries on CT scans ranged from 87% to 94%. , However, these studies used radiologists as a reference standard or did not directly compare the diagnostic accuracy of emergency physicians and radiologists.

Goals of this investigation

The primary goal of this study was to determine and compare the diagnostic accuracy of trained emergency physicians with the diagnostic accuracy of radiologists in ruling out cervical spine injuries in CT scans of trauma patients.

MATERIALS AND METHODS

Study design and setting

This single‐center prospective comparative diagnostic accuracy study was performed in a non‐academic level 1 trauma center in the Netherlands, with an annual census of 45,000 visits. In this center, radiological expertise is available 24/7, but routinely the radiologist is not physically present in the ED. The study was approved by the local institutional review board (NWMO 15.08140) and was reported according to the Standards for Reporting of Diagnostic Accuracy Studies guidelines.

Selection of participants

Eight emergency physicians (of whom 3 were residents) and 4 radiologists participated in this study. Average postgraduate experience for emergency physicians was 5 years (range 2–10 years) and for radiologists 15 years (range 7–21 years). On average, emergency medicine residents had been in training for 2 years (range 1–3 years).

Clinical reference data set

A retrospective data set was used, containing 1991 consecutive trauma patients of all ages assessed for cervical spine injuries using CT between 2007 and 2012. During this time period, indication for imaging was determined based on the National Emergency X‐Radiography Utilization Study criteria. When imaging was indicated, conventional radiographs of the cervical spine were acquired first and assessed by an emergency physician. When the radiographs did not suffice, when a (potential) injury was detected, or when in spite of negative radiographs a possible occult fracture was suspected based on clinical examination, an additional CT was performed and assessed by a radiologist.

Of the 1991 patients in the data set, 170 were diagnosed with a cervical spine injury (fracture and/or dislocation). From this database, a data set consisting of 417 scans was compiled (Figure 1). The injury prevalence was artificially increased to 30% to ensure that the data set encompassed a sufficient number of injuries without consuming too much time from the participants. To achieve this, 125 scans with injuries were randomly selected from the database by an independent researcher, and the data set was supplemented with randomly chosen scans without injuries. This data set of 417 scans was subsequently divided into 8 subsets (1 subset per emergency physician). Similar to an earlier study by van Zyl et al., a set size of 50 scans per subset was chosen, with a few additional scans per subset to cover for potential missing data. Scans with and without injury were distributed randomly over the subsets, resulting in 8 different subsets with varying injury prevalence (Appendix 1). After exclusion of erroneous CTs (scans without imaging of the cervical spine and scans with pathological fractures), the entire data set consisted of 411 scans, 120 with and 291 without injuries. Eight scans showed dislocations and 117 showed fractures of the cervical spine. Five scans contained fracture‐dislocations (Table 1, Appendix 2).

FIGURE 1

Overview of scan selection process and assessment process. Abbreviations: CT, computed tomography; T0, assessment at baseline, pretraining; T1, assessment posttraining

TABLE 1

Patient and injury characteristics of the clinical reference data set

Patient and scan characteristics		P‐values
Number of females (%)		0.72
Injured group	39 (32.5%)
Non‐injured group	100 (34.4%)
Median age in years (range)		<0.001
Injured group	55 (19‐99)
Non‐injured group	45 (0‐91)
Number of injuries (%)		N.A.
Fractures	117 (97.5%)
One	81 (67.5%)
Two or more	36 (32.5%) a
Dislocations	8 (6.7%)
Fracture‐dislocations	5 (4.2%)
Number of fractures per vertebral level		N.A.
C0	2
C1	14
C2	35
C3	10
C4	14
C5	24
C6	32
C7	30

Maximum number of injuries in 1 patient was 4 fractures in 4 different vertebrae.

Abbreviation: N.A., not applicable.

In this experimental diagnostic accuracy study of 411 cervical computed tomography scans, trained emergency physicians showed a similar, albeit not perfect, sensitivity (88%) as radiologists in ruling out injuries. As missed injuries can result in significant morbidity, further studies in clinical setting should be considered, including collaborative strategies and embedded artificial intelligence (AI) decision support.

To construct the clinical reference data set, CT scans were first selected based on the original radiologist report. Subsequently, all CT scans with injuries were provided with a complete description of the affected vertebral levels and anatomical structures by an independent radiologist following a structured format (Appendix 3). To further optimize the quality of the data set, 3 neurosurgeons assessed the scans with injuries.

When the interpretation of 2 or more participants (emergency physicians and radiologists) differed from the reference interpretation during assessment of the subsets, the scan was checked by 2 radiologists and an adjusted annotation "positive" or "negative" for injury was determined by consensus. With this multiverified clinical reference data set as reference standard, the results of the CT assessment by emergency physicians and radiologists were compared.

The clinical reference data set also contained information on which injuries qualified for stabilizing therapy (injury in need of stabilizing therapy [IST]), based on the opinion of the neurosurgeons and the actually provided therapy. As such, the reference standard also encompassed information on clinically important injuries. Stabilizing therapy was defined as either non‐invasive stabilizing treatment (hard collar or Halo‐frame) or invasive surgical stabilization.

Intervention and measurements

Each of the 8 emergency physicians assessed 1 of the 8 subsets at baseline (T0, Figure 1). Subsequently, the emergency physicians received training in assessing cervical spine CT scans for traumatic injuries. This training consisted of self‐study of proposed literature, 2 hours of classical training by a board‐certified musculoskeletal radiologist, and a hands‐on workshop in assessment of cervical spine CT scans (with scans that were different from the data set used in this study) (Appendix 4). Within 2 weeks posttraining, all emergency physicians assessed a new subset of cervical spine CTs (T1). The 4 radiologists each assessed 2 different subsets of CT scans once.

During the assessments, no clinical information was provided and participants were blinded to the CT report. Participants were informed that multiple injuries in 1 scan were possible but were not aware of the injury prevalence. The assessments were performed in a non‐clinical setting: participants were off duty and in a silent room with dimmed lights. CT scans were displayed on a diagnostic display. A researcher documented the answers regarding assessment of the scan digitally and in the same structured format as the clinical reference data set (Appendix 3).

Outcomes

The primary outcome was to assess and compare the diagnostic accuracy of trained emergency physicians and radiologists in detecting cervical spine injuries on CT. For this, the assessment of emergency physicians and radiologists (injury present/absent) served as index tests, and the clinical reference data set as the reference standard.

The secondary outcomes were the diagnostic accuracy of emergency physicians before and after training and the number of missed ISTs. Because the participants included both postgraduate emergency physicians and emergency medicine residents, we also compared diagnostic accuracy posttraining between these 2 groups.

Analysis

Statistical analysis was performed with SPSS Statistics version 24 (IBM, Armonk, NY, USA), MedCalc Statistical software version 19.05 (MedCalc software, Ostend, Belgium), and R (R version 4.0.2, R Foundation for Statistical Computing, Vienna, Austria). Normality tests were used to determine the distribution of continuous data. Outcomes were presented either as mean with standard deviation (SD) or range, median with (interquartile) range, or as proportion with 95% confidence interval (95% CI).

Two‐by‐two contingency tables were constructed, from which diagnostic accuracy measures were calculated (sensitivity, specificity, positive, and negative likelihood ratios with 95% CI). Positive and negative predictive values, as well as overall diagnostic accuracy, were not calculated because these diagnostic accuracy measures are dependent on the injury prevalence, which was artificially increased in the data set used in this study.

Means of continuous data, when distributed normally, were compared using the independent Student's t test. Proportionsof paired data (ie, sensitivity and specificity) were compared using McNemar's test. Non‐paired proportions were compared using chi‐square or, when expected counts were below five, Fisher's exact test. For comparing likelihood ratios, a regression model approach was used. A P value < 0.05 was considered statistically significant.

The effect of training on correctly assessing a cervical spine CT (ie, identification of an injury on scans positive for injuries and no injury on scans negative for injuries) was determined by calculating the odds ratio (OR) with 95% CI, using logistic multilevel analysis with correction for potential individual differences. For this, a 2‐level structure was used; CT assessments were clustered per emergency physician.

RESULTS

Primary outcome

Posttraining (T1) emergency physicians and radiologists both correctly identified 105 out of 120 scans with injuries, resulting in a sensitivity of 88% (95% CI 80% to 93%) for both groups (Table 2, Appendices 5 and 6). Ten of 15 cases were missed by both groups (Appendix 7).

TABLE 2

Diagnostic accuracy of emergency physicians (before and after training, T0 and T1) and radiologists in assessing cervical spine CT scans

Diagnostic accuracy measure	Emergency physicians (T0)	Emergency physicians (T1)	Radiologists
Sensitivity (%, 95% CI)	80 (72–87)	88 (80–93)	88 (80–93)
Specificity (%, 95% CI)	86 (81–89)	89 (85–93)	99 (96–100)
Negative likelihood ratio (95% CI)	0.23 (0.16–0.33)	0.14 (0.09–0.22)	0.13 (0.08–0.20)
Positive likelihood ratio (95% CI)	6 (4–7)	8 (6–12)	64 (24–169)

Abbreviations: CT, computed tomography; CI, confidence interval.

Trained emergency physicians had a specificity of 89% (95% CI 85% to 93%) versus 99% (95% CI 96% to 100%) for radiologists (P < 0.001). The negative likelihood ratio was not significantly different between the 2 groups (0.14 [95% CI 0.09–0.22] vs 0.13 [95% CI 0.08–0.20], P = 0.64). The positive likelihood ratio for emergency physicians and radiologists was 8 (95% CI 6–12) versus 64 (95% CI 24–169), respectively (P < 0.001, Table 2).

Secondary outcomes

Effect of training

At baseline (T0), emergency physicians assigned 96/120 scans with injuries correctly and incorrectly assigned 42/291 negative scans as positive for injuries, resulting in a sensitivity of 80% (95% CI 72% to 87%) and a specificity of 86% (95% CI 81% to 89%). Posttraining (T1), the sensitivity and specificity increased to 88% (95% CI 80% to 93%) and 89% (95% CI 85% to 93%), respectively (P = 0.08 and P = 0.15). The negative likelihood ratio improved from 0.23 (95% CI 0.16–0.33) to 0.14 (95% CI 0.09–0.22), P = 0.03. The positive likelihood ratio improved from 6 (95% CI 4–7) to 8 (95% CI 6–12), P = 0.05 (Table 2). Overall, training resulted in a significant improvement in correct assessment of the CT scans (OR 1.53; 95% CI 1.01–2.30, P = 0.04).

Injury in need of stabilizing therapy

Of the 120 scans with injuries, 87 scans were defined as containing an IST. Six of the 15 scans with missed injuries at T1 by emergency physicians were ISTs (40% of missed injuries, 1.5% of all reviewed scans). For radiologists, 4 of the 15 scans with missed injuries were ISTs (26.7% of missed injuries, 1% of all reviewed scans; P = 0.45).

Of the 5 missed cases that differed in both groups, 3 were ISTs (Appendix 7).

Postgraduate emergency physicians versus emergency medicine residents

No statistically significant difference was found between sensitivity of postgraduate emergency physicians and emergency medicine residents posttraining; 87% (95% CI 76% to 93%) versus 89% (95% CI 75% to 96%), respectively, P = 0.73. Specificity was also not significantly different; 88% (95% CI 82% to 92%) vs 92% (95% CI 85% to 96%), respectively, P = 0.32.

LIMITATIONS

We performed a single‐center study in a non‐clinical setting. It is uncertain whether the results of our study can be generalized to clinical settings and to other hospitals. In daily practice, emergency physicians experience a busier and noisier environment with increased time pressure, which could negatively influence the CT assessment. However, in daily practice clinical information is available that may aid focused assessment, and only one scan at a time is assessed instead of a large number of scans consecutively as in this test setting. Furthermore, the emergency physicians assessed the CTs within 2 weeks posttraining, bypassing the potential disadvantageous effect of knowledge degradation over time but also the potential beneficial effect of gaining experience in reading CTs in daily practice.

For the construction of the reference standard, initially only the positive scans were reassessed by a radiologist and 3 neurosurgeons independently. Although discrepant interpretations of participants were reevaluated by 2 radiologists, it cannot be excluded that this may have affected the accuracy of the reference standard.

It should be noted that since the CT scans used in this study were collected, the image quality has increased through improved hardware (new CT scanner) and software (automatic dose modulation and reconstruction algorithm). This quality improvement could enhance diagnostic accuracy if it were to be tested with current technology.

In our data set we included patients of all ages; however, there was only a small number of patients below the age of 18 years (n = 39). No injuries were present in this age group, reflecting the low prevalence of cervical spine injury in children (∼1%). , Therefore, our study results should be extrapolated with caution to young patients.

DISCUSSION

After focused training, no significant difference between emergency physicians and radiologists in ruling out cervical spine injury was found. Radiologists performed better in distinguishing between actual injuries and anatomical variants that mimic an injury on a CT scan, resulting in a significantly higher specificity and positive likelihood ratio. Emergency physicians and radiologists missed the same number of injuries; two‐thirds of these missed cases corresponded between both groups.

With regard to the secondary aims, we observed an improvement in sensitivity of 7.5% after the structured training of emergency physicians. Considering the fact that this concerns a diagnostic test with important therapeutic consequences, we judge this improvement to be of clinical relevance. Moreover, multilevel analysis showed that training resulted in significantly higher odds on overall correct assessment, as such supporting the benefit of training. Among the missed injuries were ISTs– a derived indicator for clinically important injury. Of the 15 scans with missed injuries, emergency physicians missed more ISTs than radiologists (6 vs 4 scans); however, this difference was not significant. Furthermore, there were no significant differences in diagnostic accuracy measures between postgraduate emergency physicians and emergency medicine residents after the training intervention.

In a Turkish study where emergency physicians performed the primary assessment of cervical spine injuries on CT scans in conjunction with radiologists in their normal workflow, a sensitivity of 94% and 100% and a specificity of 100% and 97% by emergency physicians and radiologists, respectively, was reported. They used the diagnosis of a 'final result team' as reference standard, a team that consisted of an emergency physician and an attending radiologist who reviewed the initial report, CT scan, and clinical data. This study was set in daily practice, and their clinical practice might differ from ours. More important, this study contained only 18 cervical spine fractures in 483 trauma patients (injury prevalence: 3.7%). This low number of injuries might make it impossible to draw conclusions about diagnostic accuracy.

In this study, trained emergency physicians reached a similar sensitivity as those in the study of van Zyl et al. (87%). The specificity in our study was notably higher (76% in van Zyl et al.). This may be because of the more extensive training received by the emergency physicians in our study compared to the 2‐hour review lectures in the study of van Zyl et al. Moreover, in their study the assessment of 2 radiologists served as reference standard; consequently, the diagnostic accuracy of those radiologists could not be evaluated. The present study used a comprehensive, finalized data set as reference standard, including unified neurosurgical expertise. Van Zyl et al. concluded that the sensitivity of emergency physicians was insufficient to accurately exclude any clinically important injuries, as they did not reach the predetermined acceptable sensitivity threshold of 95%. In our study, neither emergency physicians nor radiologists reached this threshold.

The sensitivity of the radiologists to detect injuries in our study corresponds with the diagnostic performance of single‐pass assessment of radiologists in assessing total body CT scans for traumatic injuries in a study of Stengel et al. In that study, a sensitivity of 85% for head and neck injuries was found. The high miss rate of the radiologists in our study may be explained by the assessment of a large number of successive scans (∼100 scans in 2 sessions) in a non‐clinical setting. The scans were interpreted without clinical information, such as the trauma mechanism. Information on trauma mechanism can be essential as it may direct assessment to a specific injury or to subtle findings of the injury. The high false‐positive rate of emergency physicians could be explained by the tendency to appraise non‐specific abnormalities as an injury, in order to not miss an injury. This might have led to a significantly lower specificity of the emergency physicians, compared to radiologists. Ideally, emergency physicians would have a high specificity; however, to assess cervical spine CTs safely, a high sensitivity can be at the expense of a high specificity.

In this study, 12,5% of all injuries were missed by both emergency physicians and radiologists. For the patient the priority would be to improve diagnostic accuracy, in particular sensitivity. As one third of the missed injuries differed between the groups, assessment by both an emergency physician and a radiologist might increase sensitivity. This is also our experience in daily practice, where radiologists perform the primary assessment of cervical spine CT, but emergency physicians routinely assess the scans as well. In our opinion, this promotes patient care and stimulates mutual collaboration.

For now, based on our findings, a scenario where emergency physicians perform the primary assessment of cervical spine CTs, when no real‐time radiological expertise is available, is not attainable at this moment. Yet, we observed a large improvement in diagnostic accuracy of emergency physicians after training, which provides opportunities for future research. This research should focus on the assessment of cervical spine CTs in a clinical setting by emergency physicians and radiologists, with a real‐world prevalence. It would also be important to assess the long‐term effects of training and to consider ways to improve knowledge retention, for example, through refresher courses. Moreover, e‐learning in conjunction with the current training program might further improve image interpretation skills. , In addition to developing medical training programs, improvement of assessment of scans might be achieved by investing in artificial intelligence (AI) that, when proven to have a high sensitivity, can further support assessment of cervical spine CT by emergency physicians. When emergency physicians reach sufficient diagnostic accuracy (with or without AI), it would yield opportunities and flexibility to advance clinical decision‐making before the final radiologist report becomes available.

In conclusion, in this study, no significant difference was found between emergency physicians and radiologists in ruling out cervical spine injury after a thorough training program in a non‐clinical setting. Radiologists are pivotal in filtering out excess false‐positive results as they demonstrated a significantly better specificity and higher positive likelihood ratio. Because the setting of this study deviates from daily practice in several aspects, a prospective, clinical study is required, before adapting the primary assessment of cervical spine CT scans by emergency physicians. And finally, because emergency physicians and radiologists missed different injuries, it might warrant joining efforts and letting both assess cervical spine CT scans.

MEETINGS

An abstract of this study was presented at the Dutch North Sea Emergency Medicine Conference on June 6, 2019, Egmond aan Zee, The Netherlands.

CONFLICTS OF INTEREST

All authors declare no conflict of interest.

AUTHOR CONTRIBUTIONS

Brigitta Y.M. van der Kolk, Niek Warringa, and Martijn F. Boomsma conceived the study and designed the trial. Boudewijn A.A.M. van Hasselt provided the detailed description of the CT scans in the clinical reference data set. Brigitta Y.M. van der Kolk and Niek Warringa supervised conduct of the trial and managed data collection. Brigitta Y.M. van der Kolk and Gabriella J. van den Wittenboer undertook database management and a part of the statistical analysis. Ingrid M. Nijholt provided methodological and statistical advice, checked the performed statistical analysis and provided additional statistical analyses. Brigitta Y.M. van der Kolk and Gabriella J. van den Wittenboer performed the literature search and drafted the manuscript. Ingrid M. Nijholt, Lonneke N. Buijteweg, Niels W.L. Schep, Mario Maas, and Martijn F. Boomsma critically revised the manuscript. All authors contributed substantially to writing, editing, revising, and finalizing the manuscript before submission. Brigitta Y.M. van der Kolk and Martijn F. Boomsma take responsibility for the paper as a whole.

Appendices

Click here for additional data file.