External validation of computed tomography decision rules for minor head injury: prospective, multicentre cohort study in the Netherlands.

OBJECTIVE: To externally validate four commonly used rules in computed tomography (CT) for minor head injury.
DESIGN: Prospective, multicentre cohort study.
SETTING: Three university and six non-university hospitals in the Netherlands. PARTICIPANTS: Consecutive adult patients aged 16 years and over who presented with minor head injury at the emergency department with a Glasgow coma scale score of 13-15 between March 2015 and December 2016. MAIN OUTCOME MEASURES: The primary outcome was any intracranial traumatic finding on CT; the secondary outcome was a potential neurosurgical lesion on CT, which was defined as an intracranial traumatic finding on CT that could lead to a neurosurgical intervention or death. The sensitivity, specificity, and clinical usefulness (defined as net proportional benefit, a weighted sum of true positive classifications) of the four CT decision rules. The rules included the CT in head injury patients (CHIP) rule, New Orleans criteria (NOC), Canadian CT head rule (CCHR), and National Institute for Health and Care Excellence (NICE) guideline for head injury.
RESULTS: For the primary analysis, only six centres that included patients with and without CT were selected. Of 4557 eligible patients who presented with minor head injury, 3742 (82%) received a CT scan; 384 (8%) had a intracranial traumatic finding on CT, and 74 (2%) had a potential neurosurgical lesion. The sensitivity for any intracranial traumatic finding on CT ranged from 73% (NICE) to 99% (NOC); specificity ranged from 4% (NOC) to 61% (NICE). Sensitivity for a potential neurosurgical lesion ranged between 85% (NICE) and 100% (NOC); specificity from 4% (NOC) to 59% (NICE). Clinical usefulness depended on thresholds for performing CT scanning: the NOC rule was preferable at a low threshold, the NICE rule was preferable at a higher threshold, whereas the CHIP rule was preferable for an intermediate threshold.
CONCLUSIONS: Application of the CHIP, NOC, CCHR, or NICE decision rules can lead to a wide variation in CT scanning among patients with minor head injury, resulting in many unnecessary CT scans and some missed intracranial traumatic findings. Until an existing decision rule has been updated, any of the four rules can be used for patients presenting minor head injuries at the emergency department. Use of the CHIP rule is recommended because it leads to a substantial reduction in CT scans while missing few potential neurosurgical lesions. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

Introduction

Minor head injury or mild traumatic brain injury is a common injury increasingly seen in emergency departments.1 2 Possible causes for this increase are ageing of the population and increased awareness of the potential intracranial complications of minor head injury among general practitioners and paramedics.3 4 Although the risk of intracranial complications after minor head injury is low, the consequences are important because these patients need close observation and sometimes even neurosurgical intervention.5 Several clinical decision rules exist that aim to identify those patients with minor head injuries who are at high risk for intracranial complications and need computed tomography (CT) of the head. Examples of frequently used decision rules are: the New Orleans criteria (NOC), Canadian CT head rule (CCHR), and the National Institute for Health and Care Excellence (NICE) guideline for head injury (appendix 1).6 7 8

The purpose of these rules is to detect all relevant intracranial traumatic lesions while minimising the number of unnecessary CT scans. Relevant lesions are those that need neurosurgical intervention or prolonged clinical observation because of a risk of neurological deterioration. Although the number of patients that present at the emergency departments with minor head injury has increased substantially, the overall incidence of disease specific mortality after head injury has remained fairly stable.9 An increased number of patients leads to more CT scans, longer waiting times at the emergency department, burden for the patients, radiation risks, and higher costs.10 The need for reliable CT decision rules for minor head injury to reduce unnecessary CT scans is therefore even more apparent.

Two of the decision rules were developed for patients who had had blunt trauma to the head, had a Glasgow coma scale score of 13-15 at presentation, and had experienced loss of consciousness or post-traumatic amnesia.6 7 However, these two rules could not be applied to patients who had not experienced loss of consciousness or post-traumatic amnesia.11 12 Therefore, a new decision rule was developed, the CT in head injury patients (CHIP) rule, which includes patients with and without loss of consciousness or post-traumatic amnesia.13 The potential reduction of CT scans by use of the CHIP rule was estimated at 23% compared with the scanning of all patients.13

The NOC, CCHR, and NICE guidelines were externally validated in previous studies, but there has been no external validation of the CHIP rule, even though this is necessary to determine whether the rule is generally applicable.14 15 16 17 18 19 20 21 Our aim was to perform an external validation of frequently used CT decision rules for minor head injury (CHIP, NOC, CCHR, and NICE) and compare their performance in a multicentre study in the Netherlands in university and non-university hospitals.

Methods

Study design

We conducted a prospective, multicentre cohort study between March 2015 and December 2016 in the Netherlands. Three university emergency departments (all level 1 trauma centres) and six non-university emergency departments (trauma level 1 (two centres), trauma level 2 (two centres), and trauma level 3 (two centres)) participated in this study. The emergency departments were all situated at an urban location. Institutional ethics and research board approval was obtained and informed consent was waived.

Inclusion criteria were age 16 years and over, presentation within 24 h after blunt trauma to the head, and a Glasgow coma scale score of 13-15 at presentation at the emergency department. Patients with and without loss of consciousness or post-traumatic amnesia were included. We excluded all patients with a Glasgow coma scale score of less than 13, patients younger than 16 years, transferred from other hospitals, or with any contraindication for CT.

Definition of risk factors

Clinical data concerning risk factors for intracranial complications used in the CCHR, NOC, NICE, and CHIP decision rules were collected.6 7 8 13 These clinical risk factors were:

Age

History of coagulopathy

Use of anticoagulants

Dangerous trauma mechanism (pedestrian/cyclist v vehicle, ejected from vehicle, fall from elevation (>1 m or 5 stairs), or an equivalent mechanism)

Fall from any elevation

Loss of consciousness reported by patient or witness

Retrograde amnesia

Post-traumatic amnesia

Headache

Vomiting

Intoxication with drugs or alcohol (history or suggestive findings on examination)

Post-traumatic seizure

Glasgow coma scale score on presentation

Significant injury above clavicles

Suspected open or depressed skull fracture

Contusion of skull

Clinical signs of skull base fracture (eg, raccoon eyes, battle sign, haemotympanum, cerebrospinal fluid otorrhea, cerebrospinal fluid rhinorrhea, palpable discontinuity, or bleeding from ear)

Neurological deficit (paresis, dysphasia, or other such as cranial nerve damage including diplopia, changes in sensibility, asymmetrical reflexes or pathological reflexes, coordination problems and ataxia)

Deterioration in Glasgow coma scale 1 h after presentation.

Main outcome measures

The primary outcome was any (intra)cranial traumatic finding on CT, defined as a subdural haematoma, epidural haematoma, subarachnoid haemorrhage, cerebral lesions (haemorrhagic contusion, non-haemorrhagic contusion, diffuse axonal injury), intraventricular haemorrhage, and skull fracture. The secondary outcome was any potential neurosurgical lesion, which was defined as an intracranial traumatic finding on CT that could lead to a neurosurgical intervention or death. Examples of potential neurosurgical lesions are an epidural haematoma, large acute subdural haematoma (mass), large contusion(s) (mass), depressed skull fracture, and any lesion with a midline shift or herniation. To compare our findings with previous studies, we also assessed the performance of decision rules for detecting neurosurgical interventions. All outcome measures were chosen a priori.

Study procedures

During patient inclusion in the study, neurologists (in training) and emergency physicians (in training) followed their local guideline for CT scanning in patients with minor head injury. Most participating centres used the same national guideline based on the CHIP rule, two centres followed a slightly adapted guideline (appendix 2).

Eligible patients were consecutively included by trained researcher physicians, who did not personally interview the patients. Clinical data were collected before diagnostic tests as far as possible by using forms the clinicians could fill in for each patient. The head CT scans were performed according to a routine trauma protocol at each hospital. The scans were interpreted by (neuro)radiologists who were aware of the patient’s history and clinical findings, but they were not aware of the actual score of the CT decision rules.

The clinical risk factors were collected by taking the patient’s history or information from a witness or family member. Characteristics such as injury severity score were also collected. All patients’ details about hospital admission, neurosurgical intervention, and moment of discharge were collected. If the patient was scanned, details about CT findings were recorded. The electronic health records were reviewed 30 days after the injury to assess follow-up information about a neurosurgical intervention. All data were entered by researcher physicians in the case report forms of the web based data management system OpenClinica (LCC, version 3.12.2).

Data management

After patient inclusion and data entering, two authors (KAF and CLvdB) checked the database for correct patient inclusion and completeness of data using IBM statistical package for social sciences (SPSS) version 21. Missing data were assumed to be missing at random; so to avoid bias, missing data were imputed on the basis of all the risk factors mentioned above, using multiple imputation (n=5) with the “multivariate imputation by chained equations” function in R, version 3.3.2 (R foundation for statistical computing).

Data analysis

The study population was described in terms of demographic characteristics, risk factors, admission to the hospital, and neurosurgical intervention. In patients with a CT scan, we also evaluated any intracranial traumatic findings and potential neurosurgical lesions on CT. Continuous variables were described as mean and interquartile range, categorical variables as frequencies and percentages.

The diagnostic performance of the CHIP, NOC, CCHR, and NICE decision rules for detecting intracranial traumatic findings and potential neurosurgical lesions were compared. Because the NOC and CCHR rules were developed in a specific patient population, we performed the analysis in our entire study population, as well as in a subset of the study population (based on the inclusion/exclusion criteria of the development studies of the NOC and CCHR; referred to as original NOC and original CCHR), and in our entire study population with adjustment of the rules. In the adjusted rules, the exclusion criteria of the NOC and CCHR rules were added as additional risk factors (referred to as adjusted NOC and adjusted CCHR). For the NOC rule, a Glasgow coma scale score of 13 or 14 and presence of neurological deficit were added. Finally, for the CCHR rule, use of anticoagulation, post-traumatic seizure, and presence of neurological deficit were added. All patients who had a risk factor according to the NOC or CCHR rules scored positive on these rules, indicating that they needed a CT scan.

The sensitivity, specificity, and proportion of patients needing a CT scan (with 95% confidence intervals) were assessed for each of the four decision rules. Sensitivity was calculated by dividing the number of patients in whom the outcome measure was present and the decision rule was positive, by the total number of patients in whom the outcome measure was present. Specificity was calculated by dividing the number of patients in whom the outcome measure was absent and the decision rule was negative, by the total number of patients in whom the outcome measure was absent. The Cochran’s Q test was used to directly compare the sensitivities and specificities between the four decision rules, but it should be noted that results of this test do not automatically imply that any one rule is better than the other.22 The proportion of patients needing a CT scan was calculated by dividing the number of patients in whom the decision rule was positive by the total number of patients. Confidence intervals were calculated by a bootstrapping method in R, which analyses the performance for each rule 500 times and derived the confidence intervals from the results.

In patients without a CT scan, the outcomes could not be observed. In these patients, the expected outcomes (any intracranial traumatic finding and potential neurosurgical lesion) were imputed on the basis of their risk factors with multiple imputation, in order to avoid selection bias and thus yield unbiased estimates of sensitivity and specificity.23 This imputation was possible for patients from six of the nine centres, because the other three centres had not included patients without a CT scan. The patients with and without CT scans (with imputed outcomes) from these six centres were used for the primary analysis. In addition, we analysed all patients with a CT scan from all the centres in a secondary (sensitivity) analysis, which in theory would lead to an overestimation of sensitivity and underestimation of specificity of all the rules.

In this decision problem, avoiding false negatives was more important than avoiding false positives: a false negative result leads to not performing a CT scan and thus potentially misses a lesion, whereas a false positive result leads to performing an unnecessary CT scan. The decision rule should identify all patients with potential neurosurgical lesions and most with intracranial traumatic findings, because of the severe clinical consequences (intracranial surgery, neurological sequelae, death).

Net proportional benefit has been proposed to incorporate such weighting in calculation of clinical usefulness of decision rules.24 25 For each rule, we expressed the net proportional benefit using the formula: (true positives/total number) − weight×(false positives/total number). Over a range of different weights, the net proportional benefit was calculated and compared with the scanning of all patients. The weight in this formula expresses the ratio of harmful consequences due to a false positive divided by the harmful consequences of a false negative, and it is equivalent to the odds of a lesion above which one would perform a CT scan. At a low threshold for performing CT, we would avoid false negatives of the decision rule (that is, maximise true positives) at the cost of performing many CT scans: if the threshold is 1%, this level implies performing 100 CT scans to avoid one missed lesion. At a higher threshold for performing CT, we would avoid false positives of the decision rule: if the threshold is 10%, this level implies performing 10 CT scans to avoid one missed lesion. We considered an intermediate range of thresholds (4-6% for any traumatic finding and 0.5%-1% for potential neurosurgical lesion) acceptable from a clinical point of view.24 26 Net proportional benefit expresses the true positives, and the decision rule with the highest net benefit at the intermediate thresholds has the highest clinical value.24 All statistical analyses were performed with R software, version 3.3.2 (R foundation for statistical computing, Vienna, Austria).

Patient involvement

No patients were involved in setting the research question or the outcome measures, nor were they involved in developing plans for design or implementation of the study. No patients were asked to advise on interpretation or writing up of results. There are plans to disseminate the results of the research to the relevant patient community.

Results

Between March 2015 and December 2016, 5839 consecutive patients with minor head injury were entered in the database in the participating centres (fig 1). After checking the inclusion and exclusion criteria, 322 patients were excluded from the study (Glasgow coma scale score <13, age <16 years, or no blunt head injury). In three of the nine centres, only patients with a CT scan were included (n=960). The remaining six centres included patients with and without a CT scan (n=4557).

Fig 1

Study flow diagram. *Six centres=one university centre (trauma level 1) and five non-university centres (trauma levels 1 (two centres), 2 (one), 3 (two)), including patients with and without CT scans; three centres=two university centres (both trauma level 1) and one non-university centre (trauma level 2), including only patients with a CT scan. CT=computed tomography

For the primary analysis, 4557 patients from six centres were included; 3742 (82.1%) received a CT scan and 815 (17.9%) did not. Compared with patients who received a CT scan, more patients without a scan had a Glasgow coma scale score of 15 (n=3109 (83.1%) v n=805 (98.8%)), and fewer patients experienced loss of consciousness (n=1136 (30.3%) v n=56 (6.8%)) or post-traumatic amnesia (n=1075 (28.7%) v n=29 (3.5%); table 1). Some data were unknown to the including physician, which was most frequently the case for retrograde amnesia (n=675, 14.8%), loss of consciousness (n=651, 14.3%), post-traumatic amnesia (n=502, 11%), and headache (n=630, 13.8%; table 1).

Table 1

Baseline characteristics of 4557 study patients from six centres*

Characteristic	All patients (n=4557)	Missing	Patients with CT (n=3742)	Patients without CT (n=815)
Age (years; mean (range))	53.1 (16-101)	—	56.9 (16-101)	35.7 (16-96)
Male sex	2656 (58.3)	—	2145 (57.3)	511 (62.7)
Glasgow coma scale score at presentation
13	143 (3.1)	—	141 (3.8)	2 (0.2)
14	500 (11.0)		492 (13.1)	8 (1.0)
15	3914 (85.9)		3109 (83.1)	805 (98.8)
Use of anticoagulation
None	4045 (88.8)	29 (0.6)	3233 (86.4)	812 (99.6)
Coumarin	418 (9.2)		418 (11.2)	—
Direct oral anticoagulants	54 (1.2)		53 (1.4)	1 (0.1)
Use of thrombocyte aggregation inhibitors	615 (13.5)	33 (0.7)	577 (15.4)	38 (4.7)
Bleeding disorder	44 (1)	33 (0.7)	41 (1.1)	3 (0.4)
Mechanism of injury
Pedestrian in road traffic accident	64 (1.4)	47(1.0)	57 (1.5)	7 (0.9)
Cyclist in road traffic accident	162 (3.6)		152 (4.1)	10 (1.2)
Fall from height	574 (12.6)		532 (14.2)	42 (5.2)
Other†	3710 (81.4)		2955 (79.0)	755 (92.6)
Ejected from vehicle	150 (3.3)	56 (1.2)	135 (3.6)	15 (1.8)
Loss of consciousness
None	2714 (59.6)	651 (14.3)	1968 (52.6)	746 (91.5)
≤15 min	1160 (25.5)		1105 (29.5)	55 (6.7)
>15 min	32 (0.7)		31 (0.8)	1 (0.1)
Retrograde amnesia
None	3425 (75.2)	675 (14.8)	2637 (70.5)	788 (96.7)
≤30 min	312 (6.8)		303 (8.1)	9 (1.1)
>30 min	145 (3.2)		144 (3.8)	1 (0.1)
Post-traumatic amnesia
None	2951 (64.8)	502 (11)	2185 (58.4)	766 (94.0)
≤2 h	976 (21.4)		948 (25.3)	28 (3.4)
2-4 h	69 (1.5)		68 (1.8)	1 (0.1)
>4 h	59 (1.3)		59 (1.6)	—
Intoxication with drugs or alcohol‡	1031 (22.6)	85 (1.9)	922 (24.6)	109 (13.4)
Post-traumatic seizure	36 (0.8)	68 (1.5)	33 (0.9)	3 (0.4)
Headache	1410 (30.9)	630 (13.8)	1208 (32.3)	202 (24.8)
Vomiting
Once	158 (3.5)	50 (1.1)	148 (4.0)	10 (1.2)
Twice or more	144 (3.2)		142 (3.8)	2 (0.2)
Deterioration in Glasgow coma scale (1 h after presentation)
1 point	38 (0.8)	23 (0.5)	38 (1.0)	—
≥2 points	12 (0.3)		12 (0.3)	—
Neurological deficit	130 (2.9)	141 (3.1)	128 (3.4)	2 (0.2)
Signs of skull base fracture	144 (3.2)	25 (0.5)	139 (3.7)	5 (0.6)
Visible injury of the head	2564 (56.3)	19 (0.4)	2208 (59)	356 (43.7)
Visible injury of the face	1631 (35.8)	22 (0.5)	1315 (35.1)	316 (38.8)
Suspicion of open fracture	11 (0.2)	40 (0.9)	11 (0.3)	—
Injury severity score (mean (range))	6.5 (0-75)	—	7.1 (0-75)	3.5 (0-29)

Data are number (%) of patients unless stated otherwise. CT=computed tomography.

These centres refer to those on the left hand side of figure 1, for the primary analysis.

Includes patients with mild head injury such as a bumped head against an object.

History or suggestive findings on examination (eg, nystagmus, abnormal walking).

In 384 patients (8.4%), CT showed an intracranial traumatic finding, mostly consisting of traumatic subarachnoid haemorrhages (n=182, 4.0%) and skull fractures (n=150, 3.3%; table 2). Of 74 (1.6%) patients with a potential neurosurgical lesion, 18 (0.4%) underwent a neurosurgical intervention for head injury within 30 days after the injury.

Table 2

Traumatic CT findings in 3742 patients with a CT scan from six centres*

CT finding	No (%)†
Total	384 (8.4)
Skull fracture	150 (3.3)
Depressed fracture	19 (0.5)
Linear fracture	66 1.4)
Skull base fracture	68 (1.5)
Subarachnoid haemorrhage	182 (4.0)
Contusion
Small	115 (2.5)
Large (mass)	10 (0.2)
Subdural haematoma
Small	126 (2.8)
Large (mass)	22 (0.5)
Epidural haematoma
Small	30 (0.7)
Large (mass)	5 (0.1)
Suspicion of diffuse axonal injury on CT	13 (0.3)
Basal cisterns compressed or obliterated	11 (0.2)
CT shift
0-4 mm	16 (0.4)
≥5 mm	9 (0.2)

CT=computed tomography.

These centres refer to those on the left hand side of figure 1, for the primary analysis.

Some patients had more than one CT finding.

In 116 of 3742 patients without loss of consciousness and in 117 of 3742 patients without post-traumatic amnesia, an intracranial traumatic finding was found (table 3). In total, 20 patients without loss of consciousness had a potential neurosurgical lesion and four patients underwent a neurosurgical intervention. In patients without post-traumatic amnesia, 14 had a potential neurosurgical lesion and three underwent a neurosurgical intervention.

Table 3

Baseline characteristics of 3742 patients with a CT scan from six centres*, according to status of CT findings

Characteristic	Patients with normal CT findings (n=3358)	Patients with abnormal CT findings (n=384)	All patients with a CT scan (n=3742)
Age (years; mean (range))	56.6 (16-101)	59.1 (17-98)	56.9 (16-101)
Male sex	1901 (56.6)	244 (63.5)	2145 (57.3)
Glasgow coma scale score at presentation
13	94 (2.8)	47 (12.2)	141 (3.8)
14	401 (11.9)	91 (23.7)	492 (13.1)
15	2863 (85.3)	246 (64.1)	3109 (83.1)
Use of anticoagulation
None	2886 (85.9)	347 (90.4)	3233 (86.4)
Coumarin	387 (11.5)	31 (8.1)	418 (11.2)
Direct oral anticoagulants	50 (1.5)	3 (0.8)	53 (1.4)
Use of thrombocyte aggregation inhibitors	502 (15.0)	75 (19.5)	577 (15.4)
Bleeding disorder	39 (1.2)	2 (0.5)	41 (1.1)
Mechanism of injury
Pedestrian in road traffic accident	48 (1.4)	9 (2.3)	57 (1.5)
Cyclist in road traffic accident	127 (3.8)	25 (6.5)	152 (4.1)
Fall from height	451 (13.4)	81 (21.1)	532 (14.2)
Other†	2691 (80.1)	264 (68.8)	2955 (79)
Ejected from vehicle	120 (3.6)	15 (3.9)	135 (3.6)
Loss of consciousness
None	1852 (55.2)	116 (30.2)	1968 (52.6)
≤15 min	943 (28.1)	162 (42.2)	1105 (29.5)
>15 min	21 (0.6)	10 (2.6)	31 (0.8)
Retrograde amnesia
None	2443 (72.8)	194 (50.5)	2637 (70.5)
≤30 min	251 (7.5)	52 (13.5)	303 (8.1)
>30 min	102 (3.0)	42 (10.9)	144 (3.8)
Post-traumatic amnesia
None	2068 (61.6)	117 (30.5)	2185 (58.4)
≤2 h	776 (23.1)	172 (44.8)	948 (25.3)
2-4 h	54 (1.6)	14 (3.6)	68 (1.8)
>4 h	38 (1.1)	21 (5.5)	59 (1.6)
Intoxication with drugs or alcohol‡	836 (24.9)	86 (22.4)	922 (24.6)
Post-traumatic seizure	26 (0.8)	7 (1.8)	33 (0.9)
Headache	1086 (32.3)	122 (31.8)	1208 (32.3)
Vomiting
Once	131 (3.9)	17 (4.4)	148 (4.0)
Twice or more	119 (3.5)	23 (6.0)	142 (3.8)
Deterioration in Glasgow coma scale (1 h after presentation)
1 point	33 (1.0)	5 (1.3)	38 (1.0)
≥2 points	6 (0.2)	6 (1.6)	12 (0.3)
Neurological deficit	100 (3.0)	28 (7.3)	128 (3.4)
Signs of skull base fracture	89 (2.7)	50 (13.0)	139 (3.7)
Visible injury of the head	1945 (57.9)	263 (68.5)	2208 (59)
Visible injury of the face	1181 (35.2)	134 (34.9)	1315 (35.1)
Suspicion of open fracture	6 (0.2)	5 (1.3)	11 (0.3)
Injury severity score (mean (range))	6.2 (0-54)	15.2 (1-75)	7.1 (0-75)

Data are number (%) of patients unless stated otherwise. CT=computed tomography.

These centres refer to those on the left hand side of figure 1, for the primary analysis.

Includes patients with mild head injury such as a bumped head against an object.

History or suggestive findings on examination (eg, nystagmus, abnormal walking).

In a subgroup analysis of the 3914 patients with a Glasgow coma scale score of 15, more than half the patients (n=2465, 63%) had no loss of consciousness and no post-traumatic amnesia. Ninety three (3.8%) patients had any intracranial traumatic finding, seven (0.3%) had a potential neurosurgical lesion, and one underwent a neurosurgical intervention.

Of all 4557 patients, 1511 (33.2%) were admitted to the hospital for head injury and other reasons. Of the admitted patients, 226 (5.0%) were admitted for two nights or longer because of head injury; 52 (1.1%) had neurological deterioration during admission, and six (0.1%) were intubated for longer than 24 h. Eleven (0.2%) patients died as a result of head injury, and 21 (0.5%) died as a result of a different illness or trauma.

Performance of the decision rules

After imputation of outcomes in patients without a CT scan, 23 of 815 patients had any intracranial traumatic finding and no patient had a potential neurosurgical lesion. None of these 815 patients without a CT scan had undergone a neurosurgical intervention in 30 days after injury. The sensitivity for identifying patients with any intracranial traumatic finding on CT ranged from 72.5% for the NICE criteria to 98.8% for the NOC rule (table 4; appendix 3).

Table 4

Performance of the four decision rules* used for CT in 4557 patients with minor head injury presenting at six centres†

CT decision rule	Positive outcome (No)	Negative outcome (No)	Sensitivity (%; 95% CI)	Specificity(%; 95% CI)	Positive likelihood ratio (95% CI)	Negative likelihood ratio (95% CI)
CHIP (n=4557)
Any traumatic finding on CT	—	—	94.1 (91.5 to 96.3)	21.6 (20.4 to 22.9)	1.20 (1.16 to 1.23)	0.27 (0.17 to 0.40)
CHIP, positive	383	3253	—	—	—	—
CHIP, negative	24	897	—	—	—	—
Potential neurosurgical lesion	—	—	97.3 (93.1 to 100)	20.5 (19.4 to 21.7)	1.22 (1.17 to 1.26)	0.13 (0 to 0.34)
CHIP, positive	72	3564	—	—	—	—
CHIP, negative	2	919	—	—	—	—
NICE (n=4557)
Any traumatic finding on CT	—	—	72.5 (67.8 to 77.2)	60.9 (59.3 to 62.5)	1.85 (1.72 to 2.0)	0.45 (0.37 to 0.53)
NICE, positive	295	1624	—	—	—	—
NICE, negative	112	2526	—	—	—	—
Potential neurosurgical lesion	—	—	85.1 (76.4 to 92.9)	58.6 (57.1 to 60.1)	2.06 (1.84 to 2.27)	0.25 (0.12 to 0.40)
NICE, positive	63	1856	—	—	—	—
NICE, negative	11	2627	—	—	—	—
NOC (n=4557)
Any traumatic finding on CT	—	—	98.8 (97.6 to 99.8)	4.4 (3.8 to 5.1)	1.03 (1.02 to 1.05)	0.28 (0.06 to 0.53)
NOC, positive	402	3966	—	—	—	—
NOC, negative	5	184	—	—	—	—
Potential neurosurgical lesion	—	—	100 (100 to 100)	4.2 (3.6 to 4.8)	1.04 (1.04 to 1.05)	0 (0 to 0)
NOC, positive	74	4294	—	—	—	—
NOC, negative	0	189	—	—	—	—
CCHR (n=4557)
Any traumatic finding on CT	—	—	80.3 (76.1 to 84.2)	44.2 (42.7 to 45.9)	1.44 (1.35 to 1.52)	0.44 (0.36 to 0.55)
CCHR, positive	327	2314	—	—	—	—
CCHR, negative	80	1836	—	—	—	—
Potential neurosurgical lesion	—	—	87.8 (79.7 to 94.9)	42.5 (41.0 to 44.1)	1.53 (1.40 to 1.66)	0.29 (0.12 to 0.47)
CCHR, positive	65	2576	—	—	—	—
CCHR, negative	9	1907	—	—	—	—

CT=computed tomography.

CHIP=CT in head injury patient rule; NICE=National Institute for Health and Care Excellence guideline for head injury; NOC=New Orleans criteria; CCHR=Canadian CT head rule.

These centres refer to those on the left hand side of figure 1, for the primary analysis.

The sensitivity for identifying patients with potential neurosurgical lesions was 100% for NOC, the NICE criteria had the lowest sensitivity (85.1%) for identifying potential neurosurgical lesions (table 4). The NICE criteria would have missed 11 of 74 patients with potential neurosurgical lesions (appendix 4). The CHIP criteria would have missed two patients with potential neurosurgical lesions, who both had a small epidural haematoma, which did not need neurosurgical treatment. Of these two missed patients, one had surgery to repair a depressed skull fracture (appendix 4).

The specificity for identifying any intracranial traumatic finding was lowest for the NOC rule (4.4%) and highest for the NICE criteria (60.9%). The specificity for potential neurosurgical lesions ranged from 4.2% (NOC) to 58.6% (NICE criteria). The sensitivity and specificity differed significantly between all the rules (Cochran’s Q P<0.001). Sensitivity and specificity for the original CCHR and NOC groups were slightly different from the adjusted versions (see the methods section for definition of the original and adjusted groups; appendix 5). For the outcome of neurosurgical intervention, the NOC rule had the highest sensitivity (100%) and the NICE criteria the highest specificity (58.1%; appendix 6a).

Clinical usefulness

The decision curve of the NOC rule was almost identical to CT scanning all patients in both study outcomes (fig 2). When using a low threshold for performing CT (to avoid false negatives of the decision rule), we found that the NOC rule and the scanning of all patients had the highest net proportional benefit. When using a high threshold for performing CT (to avoid false positives), we found that the NICE criteria had the highest net proportional benefit (fig 2). Over a narrow range of intermediate thresholds, the CHIP criteria had the highest net proportional benefit (0.038-0.054 for intracranial traumatic findings and 0.008-0.012 for potential neurosurgical lesions). For the neurosurgical intervention outcome, the differences in net proportional benefit were small (appendix 6b).

Fig 2

Decision curves for study outcomes showing net proportional benefit per CT decision rule. CT=computed tomography; CHIP=CT in head injury patient rule; NICE=National Institute for Health and Care Excellence guideline for head injury; NOC=New Orleans criteria; CCHR=Canadian CT head rule; scan all=scanning of all patients; scan none=scanning no patients. For each rule, the net proportional benefit was calculated with the formula: (true positives/total number) − weight×(false positives/total number)

Proportion of patients needing CT

According to the different decision rules, the proportion of the study population needing CT was 95.9% (95% confidence interval 95.3% to 96.5%) with the NOC rule, 79.8% (78.6% to 80.9%) with the CHIP criteria, 58.0% (56.4% to 59.4%) with the CCHR rule, and 42.1% (40.6% to 43.6%) with the NICE criteria. To increase the sensitivity of the CHIP criteria to the level of the NOC rule, 733 more CT scans would have been needed to identify 19 more patients with intracranial traumatic findings and identify two more patients with a potential neurosurgical lesion.

Secondary (sensitivity) analysis in all patients receiving CT scans

In all included centres, 4702 patients received a CT scan (fig 1). Most of these patients had a Glasgow coma scale score of 15 at presentation (n=3798; 80.8%), 1511 (32.1%) experienced loss of consciousness, and 1480 (31.5%) had post-traumatic amnesia (appendix 7a). We found that 528 (11.2%) patients had an intracranial traumatic finding on CT (appendix 7b). Although the sensitivity of all rules was higher and the specificity lower, their ordering was the same. The NOC rule had the highest sensitivity (99.1%) and lowest specificity (3.1%) for any intracranial traumatic finding, whereas the NICE guideline had the highest specificity (50.3%) and lowest sensitivity (77.5%; appendix 7c). Net proportional benefit analysis showed the same pattern as in the primary analysis (appendix 7c).

Discussion

Principal findings

In this large, multicentre, external validation study of CT decision rules for minor head injury patients, the NOC rule had the highest sensitivity and was the only rule with a 100% sensitivity for potential neurosurgical lesions. Nevertheless, the high sensitivity of the NOC rule comes at the cost of an extremely low specificity, with a consequence that nearly all patients would need a CT scan. The NICE guideline had the highest specificity and the lowest proportion of patients who needed a CT scan, but at the cost of a low sensitivity. The sensitivity of the CHIP criteria was high (97% for potential neurosurgical lesions) with an acceptable specificity and a substantial reduction in the proportion requiring CT. The sensitivity for the identification of patients with any intracranial traumatic finding on CT was less than 100% for all decision rules.

Which decision rule is the best for the situation depends on several factors. It depends not only on its characteristics but also on how many CT scans the physician is willing to perform to identify one patient with an intracranial traumatic finding or potential neurosurgical lesion. Because a potential neurosurgical lesion could have serious consequences, such as a neurosurgical intervention or even death, most professionals would agree that the sensitivity of the decision rule should be 100%.27 However, it is less easy to agree on the desired sensitivity for finding any intracranial traumatic lesion, because not all small intracranial traumatic findings have clinical consequences. If a CT decision rule gives a false positive result, the patient receives an unnecessary CT scan and will be discharged after spending a few hours in the emergency department. If the rule gives a false negative result, the patient will be discharged without a CT scan and an intracranial traumatic finding will be missed. If this intracranial traumatic finding was a potential neurosurgical lesion and adequate treatment was omitted or was given too late, this missed scan could have serious consequences.27

The net proportional benefit analysis might help in finding the best decision rule for different thresholds, but interpretation of the curves can be challenging.24 If a low threshold is chosen, the best rule to use in order to identify all patients with any lesion is the NOC rule, but this choice would imply that practically all patients undergo CT. At a high threshold, use of the NICE criteria avoids unnecessary scans and has the highest net proportional benefit, but important lesions might be missed. For the outcome of potential neurosurgical lesions, a very low net proportional benefit threshold and 100% sensitivity is desired. For intermediate thresholds, use of the CHIP criteria makes a trade-off between avoiding missed lesions and achieving a substantial reduction in CT scans of 21%. For the outcome of intracranial traumatic findings, the threshold can be higher, because it is not necessary that all findings are identified. From a societal perspective, not only clinical usefulness but also cost effectiveness is important. A cost effectiveness study showed that a prediction rule needs a sensitivity of at least 97% for identifying potential neurosurgical lesions in order to be cost effective, otherwise performing CT in all patients with minor head injury is more cost effective.26 In our study, only the NOC and CHIP rules fulfilled this criterion.

Comparison with other studies

Several other studies have validated and compared the sensitivity and specificity of CT decision rules for adult patients with minor head injury, but only the NOC, CCHR and NICE decision rules have been externally validated.13 14 15 16 17 28 Our study adds the CHIP rule to externally validated decision rules and compares it head-to-head with the other rules. Validation studies vary in design and in outcome measures (eg, clinically significant findings on CT are not uniformly defined), and are therefore difficult to compare. In addition, the case mix of our study is different from previous validation studies because we included all patients with blunt traumatic minor head injury, including those without risk factors. Our study is in line with earlier findings that the NOC rule has a high sensitivity but leads to a high scan rate, whereas the CCHR rule and NICE guideline can reduce the number of CT scans substantially, but at the cost of a lower sensitivity. However, the potential reduction in CT scans has not been proved in clinical practice yet. In terms of sensitivity and specificity, the CHIP rule lies between the NOC and CCHR rules.

All the decision rules in this study have been designed for an emergency department population. Although only the NICE and CHIP criteria have been designed to apply to all patients with minor head injury, in daily practice the NOC and CCHR rules probably apply to these patients as well. Therefore, we also investigated adjusted versions of the NOC and CCHR rules, which are applicable to all patients with minor head injury. The sensitivity and specificity of these two adjusted rules were comparable to the sensitivity and specificity of their original versions.

Our study population had a mean age of 53.1 years; by comparison, patients in the development studies for the NOC, CCHR, and CHIP rules had a mean age of 36-41 years. This difference is probably indicative of ageing of the population, as well as other factors such as changes in referral patterns or the increasing incidence of fall accidents.9 The percentage of patients with any intracranial traumatic finding (8.4%) was comparable with most other studies (6.9-12.1%).6 7 13 The percentage of patients who underwent a neurosurgical intervention within 30 days after injury in our study (0.4%) was low compared with most other studies (0.4%-1.5%). This difference might be because the indication for neurosurgery not only depends on clinical factors, but also differs from country to country and from neurosurgeon to neurosurgeon and could have changed over time.29 We therefore believe that instead of actual neurosurgical interventions, it is better to use “potential neurosurgical lesions” as an outcome measure. The confidence intervals for neurosurgical intervention were wide (sensitivity 71-100%) because of the low prevalence of this outcome.

Patients with minor head injury presenting at the emergency department not only reflect the ageing of the population but also the result of the decision rules themselves. In the Netherlands, use of anticoagulants (coumarines or direct oral anticoagulants) is considered a risk factor for intracranial complications and a reason for referral to the emergency department in both the ambulance and general practitioner protocols.30 The percentage of patients using anticoagulants in our study was higher than in the CHIP rule development cohort (9.2% v 6.9%).15

Limitations of the study

A limitation of our study was that not all consecutive patients with minor head injury were scanned. Following the guidelines for CT scanning at the participating centres resulted in patients with 0-1 minor criteria who did not undergo a CT scan. Therefore, patients who did not receive a CT scan but had intracranial traumatic findings (that is, those with false negative results) could have been missed. To detect this patient subgroup and precisely estimate their relative frequency among unscreened patients would need many thousands of individuals, which was not feasible. Missing patients without a CT scan could have led to a slight overestimation of the sensitivity and an underestimation of the specificity. We therefore performed the primary analysis on data from six centres which also collected data for patients without a CT scan. For all the rules, the new calculated sensitivities were a little lower and the specificities higher, as expected. The fact that most centres in our study used CT guidelines based on the CHIP rule could have introduced a bias in favour of the CHIP rule, owing to possible missed lesions (because the patient was not scanned according to the local guideline) that would have been detected by the other rules. However, by imputing the outcomes of the patients without a CT scan, we were able to keep this bias to a minimum.

Because most physicians used the CHIP rule on a regular basis, they were more likely to apply it correctly. However, many risk factors are the same for all rules and the validation was performed based on the scored risk factors, not on the physicians’ judgment of a rule being positive or negative. In addition, in our centres, it is clinical practice to assess not only risk factors from the CHIP rule, but also other risk factors such as headache and retrograde amnesia. In our study, it was unclear how quickly patients proceeded to CT and whether lesions appeared after this time. However, af Geijerstam and colleagues concluded in a literature review that the risk of a patient developing an intracranial lesion after an early normal CT scan is very low.31

Another limitation was the possibility that we missed patients undergoing a neurosurgical intervention in a different hospital. However, because the participating centres were all the primary neurosurgery centres of the area, this potential bias is highly unlikely. Furthermore, because we used potential neurosurgical lesions as a secondary outcome instead of neurosurgical intervention, our main findings would not have been affected. In the development studies of the four decision rules, potential neurosurgical lesions were not used as an outcome measure.

Conclusions and policy implications

Application of the CHIP, NOC, CCHR, or NICE decision rules leads to a wide variation in CT scanning among patients with minor head injury, resulting in unnecessary CT scans and missed intracranial traumatic findings. Only the NOC rule did not miss potential neurosurgical lesions, but this was at the cost of having to scan nearly all patients. Although the NICE guideline had the highest reduction of CT scans (58%), missing 15% of patients with potential neurosurgical lesions would be unacceptable to most physicians in the emergency department, because it would mean that for every 200 patients not be scanned according to the NICE criteria, one patient would turn out to have a potential neurosurgical lesion.

Of the four investigated rules, the CHIP rule performed the best with an acceptable sensitivity of 97% for potential neurosurgical lesions according to previous cost effectiveness analysis, the highest net proportional benefit at intermediate thresholds, and a substantial reduction of CT scans of 21% compared with the scanning of all patients. Updating an existing decision rule might increase the sensitivity and specificity for detecting potential neurosurgical lesions. Until this update is conducted, it is justified to use any of the four rules for patients with minor head injury presenting at the emergency department. We recommend use of the CHIP rule because it leads to a substantial reduction of CT scans and misses very few potential neurosurgical lesions.

Several decision rules in computed tomography (CT) have been developed to identify patients with a higher risk of intracranial complications after sustaining a minor head injury

The New Orleans criteria have a high sensitivity but would lead to a high scan rate, whereas criteria from the Canadian CT head rule and the National Institute for Health and Care Excellence guidance for head injury reduce the number of CT scans substantially but with a lower sensitivity

Commonly encountered risk factors such as loss of consciousness and anticoagulation in these patients have not been addressed with current decision rules; the CT in head injury patients (CHIP) rule includes these risk factors

This external validation study directly compared the CHIP rule with the other three frequently used and validated rules for CT scanning of patients presenting with minor head injury

The CHIP rule showed an acceptable sensitivity for potential neurosurgical lesions, the highest net proportional benefit at intermediate thresholds of performing CT scans, and a substantial reduction of scans compared with the scanning of all patients