Post-Concussion Symptoms Rule: Derivation and Validation of a Clinical Decision Rule for Early Prediction of Persistent Symptoms after a Mild Traumatic Brain Injury.

Mild traumatic brain injury (mTBI) is a common problem. Depending on diagnostic criteria, 13 to 62% of those patients develop persistent post-concussion symptoms (PPCS). The main objective of this prospective multi-center study is to derive and validate a clinical decision rule (CDR) for the early prediction of PPCS. Patients aged ≥14 years were included if they presented to one of our seven participating emergency departments (EDs) within 24 h of an mTBI. Clinical data were collected in the ED, and symptom evolution was assessed at 7, 30 and 90 days post-injury using the Rivermead Post-Concussion Questionnaire (RPQ). The primary outcome was PPCS at 90 days after mTBI. A predictive model called the Post-Concussion Symptoms Rule (PoCS Rule) was developed using the methodological standards for CDR. Of the 1083 analyzed patients (471 and 612 for the derivation and validation cohorts, respectively), 15.6% had PPCS. The final model included the following factors assessed in the ED: age, sex, history of prior TBI or mental health disorder, headache in ED, cervical sprain and hemorrhage on computed tomography. The 7-day follow-up identified additional risk factors: headaches, sleep disturbance, fatigue, sensitivity to light, and RPQ ≥21. The PoCS Rule had a sensitivity of 91.4% and 89.6%, a specificity of 53.8% and 44.7% and a negative predictive value of 97.2% and 95.8% in the derivation and validation cohorts, respectively. The PoCS Rule will help emergency physicians quickly stratify the risk of PPCS in mTBI patients and better plan post-discharge resources.

Introduction

Mild traumatic brain injury (mTBI), commonly known as concussion, is a frequent problem. The Public Health Agency of Canada recently reported more than 5 million emergency department (ED) visits for head injuries between 2002 and 2017, 70% of which were due to concussions or mTBI.[1] This situation is similar elsewhere; in the United States, approximately 2 million people suffer a mTBI each year[2-4] and worldwide, mTBI represents approximately 70-80% of all TBIs.[5] Patients with mTBI often suffer from post-concussion symptoms such as headache, dizziness, and concentration problems, which can impede the return to normal function.[6]

Although post-concussion symptoms are usually transient, prevalence rates of persistent post-concussion symptoms (PPCS) have been reported in between 13 and 62% of patients, depending on diagnostic criteria.[2,3,7-21] Prolonged symptoms lead to time lost from work, social costs, and adverse impact on quality of life.[20,22] Early interventions, individualized neuropsychological interventions, and interdisciplinary interventions seem to reduce the frequency, intensity, and duration of PPCS[23-25] and may facilitate the recovery of daily activities and return to work.[26-29] Unfortunately, predicting long-term mTBI symptoms is challenging in the ED since emergency physicians’ diagnostic accuracy for predicting full-recovery or PPCS at 3 months post-injury is no better than chance.[30]

Prior studies have identified some specific organic and psychogenic factors that seem to be associated with PPCS. The most frequently reported characteristics are sex (female),[2,10,31-41] prior TBI,[3,10,31,32,35,42] history of psychiatric disorders,[8,10,11,14-16,34,40,43-45] level of education,[10,32] comorbidities,[32,46,47] dizziness in ED,[15,48] headaches in ED,[11,12,15,16,32] concomitant injuries,[3,49] alcohol intoxication,[16,40] hemorrhage on computed tomography (CT) scan,[11,12,32,38,50,51] cervical sprain or strain,[39,52,53] and litigation.[19,54] Some symptoms have also been associated with PPCS during early follow-up[32]: light sensitivity,[3,55] fatigue,[32,56] sleep disturbance,[8,57,58] severe symptoms on the Rivermead Post-Concussion Questionnaire (RPQ),[32,59] pain,[32,60] depressive symptoms,[59,61,62] anxious symptoms,[45] difficulty concentrating,[55] and abnormal eye movements.[63,64] It seems that patients who sustained sport-related mTBI may experience less post-concussion symptoms.[65] However, the association with some factors remains controversial with contradictory results in the literature, such as age,[8,15,17,32-36,38,45] loss of consciousness,[2,14,15,18,29,33,42,61,66] post-traumatic amnesia[2,33,42,45,55,59,61,67] and Glasgow Coma Scale <15 on arrival in ED.[29,32,39,40] Some biomarkers, combined with clinical factors, may be useful to predict the diagnosis and prognosis of mTBI. It has also been reported that S100ß, neuron-specific enolase (NSE), glial fibrillary acidic protein (GFAP), and cleaved-Tau (C-Tau) could be associated with brain damage.[68] However, their ability to predict PPCS is still unclear.[69] Finally, even if other biomarkers like diffusion tensor imaging[70] could be potentially predictive, they are expensive and difficult to measure among all mTBI patients in the ED.

To better identify patients who would most benefit from follow-up, some predictive models have been proposed,[10,11,13,32,39,71-73] but to our knowledge, none were externally prospectively validated and no evidence-based guidelines have been proposed. Consequently, there is a pressing need for ED clinicians to have access to a fast and reliable clinical decision rule (CDR) to predict PPCS in adults who sustained a mTBI.

The main objective of this study was to derive and validate a CDR, called the PoCS Rule (Post-Concussion Symptoms Rule), to predict PPCS 90 days after a mTBI. The secondary objectives were to validate the PoCS Rule for the prediction of PPCS, using other definitions and non-return to work/school at 90 days.

Methods

Study design and setting

This prospective multi-center cohort study was conducted across seven Canadian academic EDs (Level I and Level II trauma centers). Patients in the derivation cohort were recruited between July 2013 and October 2016, and those in the validation cohort were recruited between February 2017 and September 2018. This study was approved by each participating institution's research ethics board.

Study population

Patients were included if they had a documented mTBI within 24 h of ED visit and were age ≥14 years. We defined mTBI using the World Health Organization Task Force criteria[74]: a patient with a head trauma and a Glasgow Coma Scale (GCS) score of 13-15 ≥ 30 min post-injury and at least one of the following: confusion or disorientation, loss of consciousness ≤30 min, post-traumatic amnesia ≤24 h, and/or other transient neurological abnormalities such as focal signs, seizure, and intracranial lesion not requiring surgery. Patients were excluded if they were hospitalized, were unable to consent, or were not able to communicate in English or French.

Data collection

Assessment in the ED

Eligible patients were identified by emergency physicians and research assistants at each participating site. During the initial ED visit, the emergency physicians used a standardized data collection form (Supplementary File S1) to collect relevant sociodemographic and clinical data, including the time of trauma. Informed consent was provided during the initial visit before the blood samples were taken. The blood samples were taken for all patients in our cohort for research purposes. The decision to perform a head CT scan was left to the emergency physician, and patients were included whether they had a CT or not.

Patient follow-up

Trained research assistants administered phone interviews at 7, 30, and 90 days post-mTBI (Supplementary File S2). During the 7-day interview, research assistants collected detailed information about the trauma and the medical history of patients to optimize data completeness and accuracy (Supplementary File S3). The Rivermead Post-Concussion Symptoms Questionnaire (RPQ), which is frequently used in the clinical and research settings,[2,3,6,9,13,16,21,52,75,76] was used to evaluate the patient's symptoms at each interview. This simple validated questionnaire assesses the severity of PCS by having patients rate 16 symptoms on a scale of 0 (not experienced at all) to 4 (a severe problem), with the total score being the sum of all RPQ items.[77-79]

Blood sampling and analysis

The exact time of blood sample collection was recorded for each participant. These samples were taken by the ED nurse ≤24 h after the trauma in most of the study's recruiting sites (4/7). This was done only on weekdays by a research nurse for the other participating sites. The samples were sent to the laboratories of each hospital to be centrifuged, put into three small aliquots and frozen at -20°C until they were sent on dry ice to the study's central laboratory (CHU de Québec-Université Laval, Quebec City), where they were stored at -80°C until batch tested.

The blood concentrations of S100ß, NSE, GFAP, and C-Tau were analyzed by enzyme-linked immunosorbent assay (ELISA). For each biomarker, different cut-offs were tested during our statistical analyses, including those most often described in the literature: S100B ≥ 0.10 mcg/L[80] and ≥0.20 mcg/L,[81] NSE ≥1.0 mcg/L,[82] and C-Tau ≥1.5 mcg/L.[83] For GFAP, no cut-off point has been established related to PPCS,[84] we have therefore tested ≥0.045 mcg/L (detection limit of our ELISA assays) and ≥0.1 mcg/L (another limit of detection described in the literature).[85]

Primary outcome measure

Until recently,[86] there was no consensus in the literature regarding the diagnostic criteria of PPCS.[87] Therefore, we used a nominal croup technique to define our primary outcome, PPCS at 90 days. This technique allows participants to give their opinions during a structured group interaction and comprises four key stages: silent generation, round-robin, clarification, and voting (ranking or rating).[88] In our study, PPCS was defined as the presence of spontaneously described, severe, persistent symptoms that have an impact on patient's life. Symptoms were considered spontaneously described if patients mentioned symptoms following a standardized interview prompt (“I would like to know whether you still have any symptoms from your head trauma?’’ The research assistant completed the RPQ as per standard questionnaire administration guidelines if patients answered positively. If patients responded negatively, their symptoms were not considered spontaneously described. Symptoms were considered severe if patients answered that they had at least one symptom of 4 points (severe problem) or two symptoms of 3 points (moderate problems), as scored on the RPQ. Symptoms were considered persistent if they lasted at least 90 days. To ensure that the symptoms were not newly present at 90 days, patients had to spontaneously answer that they had symptoms at 7 and 30 days.

As no specific measure of impact was available in the derivation cohort, symptoms were considered as having an impact if patients who were working/in school at the time of trauma did not return to work/studies as before the trauma. In the validation cohort, the measure of the impact on the patient's life was obtained by asking the following questions: “Did you return to your normal activities just as before the head trauma? If not, is this due to your trauma to the head?” A positive answer to the latter question was considered as having an impact on the patient's life.

Secondary outcome measures

Secondary outcome measures were PPCS at 90 days according to four other sets of criteria used in the literature:

Delphi definition: “Any post-concussion symptom appeared within hours of mTBI which is still present every day 90 days after the trauma and has an impact on at least one sphere of life.”[86]

PPCS with a documented impact: impact on normal activities due to mTBI, according to the patient.

Non-return to work: patients working/in school at the time of trauma and did not return to work/studies as before trauma.

Moderate-to-severe symptoms: the presence of at least three symptoms of 3 or 4 points on the RPQ, indicating a moderate-to-severe intensity level.

Sample size

Derivation cohort

Based on the literature, the proportion of patients with PPCS at 90 days was estimated to be around 13-62%.[2,3,7-21] Assuming a middle range of 35%, 16 potential candidate predictors, a precision of 5% and at least 10 events per covariable, we estimated that 468 observations were required to develop the model.

Validation cohort

As the prevalence of PPCS at 90 days was 15.6% in our derivation cohort, for an accurate estimation of regression coefficients, assuming 10 events per covariable and a final model including eight predictors in the ED, we estimated that a minimum of 550 observations were required to validate the model. For the second step, five predictors at follow-up), assuming a prevalence of 18% of PPCS in that sub-group, a precision of 5% and at least 10 events per covariable, we estimated that a minimum of 275 observations were required.

Statistical analysis

We used the methodological standards[89,90] to develop and validate our CDR. Descriptive statistics were computed for baseline characteristics. The prevalence of PPCS has been reported as the proportion of participants corresponding to each definition. Univariate analyses were performed to measure the association between each potentially predictive factor and the primary outcome (odds ratio [OR] with 95% confidence intervals [CIs]). Multiple imputation was used to handle missing data for potential predictors in the ED: the chained equations method was used to simulate missing values, and 20 imputed datasets were generated for the following variables: cervical sprain, history of mental health disorders, prior multiple TBI and prior TBI <1 year.

A two-step bootstrap sampling with replacement was used to select variables of interest.[91] For each step, 500 random samples were generated for each imputation and variables that were statistically significant in more than 40% of samples (alpha = 0.157) were considered relevant to be tested in the model. Backward selection was used to validate candidate predictors before using them in the final logistic regression model (alpha = 0.157). Selected predictors were screened for multi-collinearity. Coefficient rating was standardized using the smaller value of the coefficient.[92] Thirty-day follow-up data were used if they were missing for patients who were not reached at the 7-day follow-up (simple imputation). Sensitivity analyses were performed to test different categories for the following variables: age, number of prescribed medications, headache intensity (out of 10), loss of consciousness duration, amnesia duration, RPQ, and different blood levels for all four biomarkers. Some symptoms on the RPQ at the 7-day follow-up were group tested using subtypes classification suggested by some authors[93,94]: cognitive symptoms, oculomotor symptoms, headache/migraine, vestibular symptoms, and anxiety/mood symptoms.[94] Risk groups were defined using the most appropriate thresholds from the classification performance for the following two steps: the ED evaluation and the follow-up evaluation 7 days after the trauma. Model performance was evaluated by measuring sensitivity, specificity, negative predictive value (NPV), and positive predictive value (PPV). Discrimination was evaluated for each step and is presented using area under the curve (AUC). For each step of the model, a calibration curve of expected versus observed risk was assessed. Variance Inflation and Condition Index were used to evaluate the possibility of collinearity. All analyses were performed with the Statistical Analysis System (SAS Institute, Cary, NC, version 9.4).

Patient and public involvement

Patients were involved as research partners during our nominal group technique, particularly when discussing the outcome definition.

Results

We enrolled 1339 patients, 563 in the derivation cohort and 776 in the validation cohort. Of those, 1083 patients were included in our analyses: 471 and 612 in the derivation and validation cohorts, respectively (Fig. 1). We found a 15.6% prevalence of PPCS (15.1% and 15.9% in the derivation and validation cohorts, respectively). Table 1 shows the characteristics of the study population.

FIG. 1.

Study flowchart.

Table 1.

Characteristics of the Study Population: ED Evaluation

	Derivation cohort N = 471		Validation cohort N = 612
	n	(%)	n	(%)
Sociodemographic variables
Age, median (Q1-Q3)	37	(23-57)	43	(25-60)
14-24	142	(30.1)	151	(24.7)
25-34	81	(17.2)	95	(15.5)
35-44	49	(10.4)	73	(11.9)
45-54	67	(14.2)	89	(14.5)
55-64	76	(16.1)	81	(13.2)
65-74	39	(8.3)	77	(12.6)
75-84	15	(3.2)	32	(5.2)
85+	2	(0.4)	14	(2.3)
Sex (M)	306	(65.0)	346	(56.5)
Past medical/injury history
History of TBI[*]	166	(35.4)	249	(40.7)
Prior TBI <1 year	23	(5.0)	36	(5.9)
Prior multiple TBI	49	(10.7)	71	(11.6)
Prior moderate/severe TBI	19	(4.5)	38	(6.4)
History of mental health disorder[†]	131	(28.0)	126	(20.6)
At least 3 different prescribed medications[¶]	104	(22.4)	136	(22.2)
Evaluation in ED
Mechanism of injury&
Sport	109	(23.5)	101	(16.5)
Fall from their own height	97	(20.9)	114	(18.6)
Motor vehicle accident	76	(16.4)	87	(14.2)
Bicycle accident	73	(15.7)	76	(12.4)
Fall more than own height	34	(7.3)	46	(7.5)
Others	75	(16.2)	188	(30.7)
Documented loss of consciousness[#]	228	(48.7)	289	(47.3)
Unknown duration	78	(38.5)	111	(18.2)
< 5 min	124	(26.5)	150	(24.6)
≥ 5 min	26	(5.6)	28	(4.6)
Unknown information	115	(24.6)	100	(16.4)
Documented post-traumatic amnesia¥	238	(50.5)	225	(36.8)
< 30 min	189	(40.1)	167	(27.6)
≥ 30 min and <3 h	37	(7.9)	47	(7.8)
≥ 3 h	12	(2.5)	11	(1.8)
Post-traumatic confusion§	261	(57.9)	320	(53.2)
Documented headache in ED[‡]	313	(66.5)	433	(70.8)
≤ 3	166	(35.2)	154	(25.2)
4-7	104	(22.1)	202	(33.0)
8-10	43	(9.1)	77	(12.6)
Glasgow Coma Scale score at ED arrival
15	409	(86.8)	545	(89.1)
14	56	(11.9)	61	(10.0)
13	6	(1.3)	6	(1.0)
Concomitant injuries	278	(60.2)	374	(61.1)
Cervical sprain	130	(31.9)	158	(25.8)
Hemorrhage on CT	31	(6.6)	46	(7.5)

Missing data for derivation cohort: history of TBI (n = 2), prior TBI <1 year (n = 7), prior multiple TBI (n = 13), prior moderate/severe TBI (n = 45). Validation cohort prior moderate/severe TBI (n = 21).

Any pre-injury mental health disorder, including depression, anxiety, bipolar disorder, schizophrenia, attention-deficit/hyperactivity disorder (ADHD), personality disorders, obsessive-compulsive disorder, post-traumatic stress disorder (PTSD), and other psychiatric disorders (missing data for derivation cohort: n = 3).

Missing data for derivation cohort (n = 7).

Missing data for derivation cohort (n = 7).

Missing data for derivation cohort (n = 3), validation cohort (n = 1). Unknown information means that the patient was unsure if they lost consciousness and there was no witness.

Missing data for validation cohort (n = 7).

Missing data for derivation cohort (n = 20), validation cohort (n = 11).

Verbal rating scale, 0-10.

ED, emergency department; Q, quarter; M, male; TBI, traumatic brain injury; CT, computed tomography.

The potential predictive factors of PPCS in ED are found in Table 2. The following factors were statistically associated with PPCS: age, sex (female), history of TBI or mental health disorder, taking prescribed medication, having sustained a fall, headache in ED and cervical sprain. There was no association between PPCS and age when treated as a continuous variable. However, we found a U-shaped association between PPCS and age when treated as a categorical variable (Fig. 2). Alcohol consumption, level of education, loss of consciousness, post-traumatic amnesia or confusion, hemorrhage on CT and the four biomarkers tested were not significantly associated with PPCS. Wearing a helmet (for sports or activities where applicable) seems to decrease the risk of PPCS.

Table 2.

Univariate Correlation between Potential Predictive Factors in ED and PPCS

Potential predictive factors	Odds ratio	95% CI[§]
Sociodemographic variables
Age (continuous)	1.01	1.00-1.02
Age (category)
14-24 or ≥65 (reference category)	1.00
25-34	2.38	1.10-5.13
35-44	4.11	1.82-9.27
45-54	3.00	1.38-6.55
55-64	2.57	1.19-5.56
Sex (F)	2.02	1.21-3.36
Level of education	1.13	0.56-2.27
Past medical/injury history
History of TBI	1.32	0.78-2.20
Prior TBI <1 year	3.20	1.30-7.86
Prior multiple TBI	2.19	1.09-4.37
History of mental health disorder	2.14	1.27-3.62
Alcohol consumption	0.68	0.20-2.32
Drug consumption	0.98	0.28-3.42
Prescribed medication (at least one)	3.11	1.70-5.68
Evaluation in the ED
Mechanism of injury
Sport	0.56	0.28-1.11
Fall	1.77	1.01-3.10
Motor vehicle accident	0.97	0.51-1.86
Helmet wearing (if applicable)	0.43	0.19-0.97
Loss of consciousness	0.88	0.47-1.63
Post-traumatic amnesia	0.91	0.55-1.52
Post-traumatic confusion	0.90	0.52-1.57
Vomiting after trauma	0.58	0.22-1.52
Headache in ED	2.08	1.13-3.81
GCS on arrival (13-14 vs. 15)	0.83	0.09-8.10
Suspected intoxication in ED	0.40	0.12-1.34
Other diagnosis
Any concomitant injury	1.66	0.96-2.88
Cervical Sprain	2.53	1.51-4.24
Multiple body injuries	1.36	0.73-2.51
Hemorrhage on CT α	1.71	0.71-4.14
Biomarkers[†]
S100ß ≥ 0.02 mcg/L (≥ 20 pg/mL)[‡] ≥ 0.10 mcg/L (≥ 100 pg/mL) ≥ 0.20 mcg/L (≥ 200 pg/mL)	0.950.740.46	0.55-1.650.28-1.960.06-3.61
NSE ≥ 0.1 mcg/L (≥ 100 pg/mL) ≥ 0.2 mcg/L (≥ 200 pg/mL) ≥ 1.0 mcg/L (≥ 1000 pg/mL)[‡]	1.381.100.94	0.82-2.330.62-1.940.20-4.28
GFAP ≥ 0.045 mcg/L (≥ 45 pg/mL) ≥ 0.1 mcg/L (≥ 100 pg/mL)[‡]	0.640.69	0.31-1.310.34-1.37
C-Tau ≥ 0.156 mcg/L (≥ 156 pg/mL) ≥ 1.5 mcg/L (≥ 1500 pg/mL)[‡]	0.640.90	0.32-1.280.52-1.57

αAny traumatic hemorrhage on CT.

S100ß, S100ß protein; NSE, neuron specific enolase; GFAP, glial fibrillary acidic protein; C-Tau, cleaved-Tau.

The most documented threshold in the literature.

95% confidence interval.

ED, emergency department; PPCS, persistent post-concussion symptoms; CI, confidence interval; F, female; TBI, traumatic brain injury; CT, computed tomography.

FIG. 2.

Association between age and persistent post-concussion symptoms (PPCS).

In the derivation cohort, multiple imputation was applied to handle missing ED data in 13.4% of patients with cervical sprain, 0.6% of those with a history of mental health disorders, 2.8% of patients with prior multiple TBI, and 1.5% for prior TBI <1 year.

Table 3 shows the univariate correlations between symptoms at the 7-day follow-up and PPCS. All RPQ symptoms were significantly associated with PPCS except restlessness. A RPQ score ≥21 at 7 days had a 10.52 OR (95% CI: 5.49-20.14) to predict PPCS 90 days post-trauma. For the derivation and validation cohorts, data from the 30-day follow-up were used to replace the 7-day missing data in 13% and 8% of cases, respectively. In the validation cohort, the receiver operating characteristic (ROC) curve for the RPQ score versus the impact on the patient's activities had an AUC = 0.90.

Table 3.

Univariate Correlation between Symptoms at 7-Days Follow-Up and PPCS

Variable	Odds ratio	95% CI[§]
Rivermead Post-concussion Questionnaire (RPQ) symptoms[α]
Headaches	8.03	3.66-17.64
Feelings of dizziness	4.21	2.28-7.75
Nausea and/or vomiting	3.62	1.77-7.42
Noise sensitivity, easily upset by loud noise	3.83	2.10-6.99
Sleep disturbance	3.75	2.06-6.81
Fatigue, tiring more easily	6.48	2.69-15.65
Being irritable, easily angered	4.05	2.22-7.37
Feeling depressed or tearful	4.27	2.32-7.86
Feeling frustrated or impatient	3.89	2.14-7.08
Forgetfulness, poor memory	4.67	2.53-8.62
Poor concentration	5.54	2.91-10.55
Taking longer to think	4.28	2.32-7.89
Blurred vision	4.46	2.35-8.49
Light sensitivity, easily upset by bright light	6.35	3.42-11.82
Double vision	8.57	3.10-23.68
Restlessness	1.97	0.83-4.69
Rivermead score ≥21	10.52	5.49-20.14
Number of symptoms spontaneously identified by the patient[†]	2.42	1.83-3.18
Number of symptoms (≥ 2 points) on the RPQ[‡]	1.40	1.28-1.53
Number of symptoms (≥ 3 points) on RPQ[‡]	1.45	1.32-1.60
Cervical pain	4.33	2.15-8.73
Limb pain	2.46	1.36-4.47
Cognitive symptoms[*]	5.57	2.62-12.61

At least 2 points.

Before asking questions from the Rivermead Questionnaire.

Rivermead Post-Concussion Symptoms Questionnaire.

Cognitive symptoms are defined by the following symptoms: forgetfulness/poor memory, poor concentration, taking longer to think.

95% Confidence interval.

PPCS, persistent post-concussion symptoms.

The PoCS Rule is shown in Figure 3, with full model specification in Supplementary File S4. During assessment in the ED, the total score helps the clinician better identify which mTBI patients are at higher risk of PPCS and may benefit from a follow-up. Otherwise, a phone follow-up should be done for medium-risk patients (2-6 points at ED evaluation) to identify another sub-group of high-risk patients.

FIG. 3.

Post-Concussion Symptoms Rule (PoCS Rule).

None of the biomarkers studied helped improve the final model, irrespective of the threshold value. Table 4 and Supplementary File S5 describe the proportions of patients with PPCS for all predictors and risk categories. According to the rule, only 2.7% of low-risk ED patients will experience PPCS compared with 30% of high-risk patients. Supplementary File S6 presents the classification performance of the PoCS Rule for the evaluation in the ED and for the 7-day follow-up. With an ED score threshold ≤1 for low risk and ≥7 for high risk, and a 7-day follow-up score ≥2 for high risk, the rule has a sensitivity of 91.4% and 89.6%, a specificity of 53.8% and 44.7% and a NPV of 97.2% and 95.8% in the derivation and validation cohorts, respectively (Table 5). Data are also displayed using a ≥3-point threshold. There was no multi-collinearity between predictors, and discrimination was good, with an AUC of 0.75 for ED evaluation and 0.85 for the 7-day follow-up. The calibration of the model was excellent (Fig. 4). Based on our validation cohort, the unnecessary follow-up rate with and without the PoCS Rule would be 66.3% and 84.3%, respectively (absolute value: 84.3-66.3 = 18% reduction). We estimate that using our rule would have reduced follow-ups by 21% (relative value: (516-406)/516 = 21%) compared with systematic follow-up post-ED visit.

Table 4.

Prediction of PPCS According to the PoCS Rule Risk Categories (Validation Cohort)

	Number of points	n with PPCS (%)
Total number of patients with PPCS		96 (15.9)
Step 1: Assessment in ED (n = 612)
ED low risk (n = 113)	0-1	3 (2.7)
ED medium risk (n = 466)	2-6	83 (17.8)
ED high risk (n = 33)	≥ 7	10 (30.3)
Step 2‡: Follow-up evaluation for ED medium risk patients (n = 456)[†]
Follow-up low risk (n = 123)	0-1	7 (5.7)
Follow-up high risk (n = 333)	≥ 2	76 (22.8)
PoCS Rule final prediction
Advised to consult a healthcare provider PRN (n = 236)		10 (4.2)
Follow-up/interventions (n = 366)		86 (23.5)
Step 2[§]: Follow-up evaluation for ED medium risk patients (n = 456)[†]
Follow-up low risk (n = 184)	0-2	10 (5.4)
Follow-up high risk (n = 272)	≥ 3	73 (26.8)
PoCS Rule final prediction
Advised to consult a healthcare provider PRN (n = 297)		13 (4.4)
Follow-up/interventions (n = 305)		83 (27.2)

Using threshold ≥2 points 7 days after trauma.

For 10 medium risk patients, data were missing for at least one predictor at follow-up evaluation.

Using threshold ≥3 points 7 days after trauma.

PPCS, persistent post-concussion symptoms; PoCS, Post-Concussion Symptoms Rule; ED, emerge0ncy department.

Table 5.

PoCS Rule Performance for Prediction of PPCS (90 Days)

	Derivation cohort (n = 471)		Validation cohort (n = 602)[†]
Post-concussion symptoms, n (%)	71	(15.1)	96	(15.9)
Using threshold ≥2 points 7 days after trauma	%	95% CI[§]	%	95% CI[§]
Sensitivity	91.4	(84.9-98.0)	89.6	(83.5-95.7)
Specificity	53.8	(48.7-58.9)	44.7	(40.3-49.0)
Negative predictive value	97.2	(94.9-99.4)	95.8	(93.2-98.3)
Positive predictive value	26.6	(21.0-32.1)	23.5	(19.2-27.8)
Using threshold ≥3 points 7 days after trauma
Sensitivity	87.1	(79.3-95.0)	86.5	(79.6-93.3)
Specificity	66.7	(61.9-71.5)	56.1	(51.8-60.5)
Negative predictive value	96.6	(94.4-98.8)	95.6	(93.3-97.9)
Positive predictive value	32.4	(25.7-39.1)	27.2	(22.2-32.2)

95% Confidence interval.

For 10 medium risk patients, data were missing for at least one predictor at follow-up evaluation.

PoCS, Post-Concussion Symptoms Rule; PPCS, persistent post-concussion symptoms; CI, confidence interval.

FIG. 4.

Calibration plots for derivation and validation cohorts.

Table 6 shows the PoCS Rule performance for all secondary outcomes. The rule has an excellent sensitivity (97.3%) and NPV (99.1%) for predicting moderate to severe symptoms.

TABLE 6.

PoCS Rule Performance for Prediction of Secondary Outcomes (90 Days), Validation Cohort

	Delphi definition outcome[β] (n = 602)		Impact on normal activities[†] (n = 599)		Non-return to work/school[‡] (n = 302)		Moderate/severe symptoms[§] (n = 599)
PPCS, n (%)	113	(18.8)	58	(9.7)	58	(19.2)	74	(12.4)
Using threshold ≥2 points 7 days after trauma	%	95% CI §	%	95% CI §	%	95% CI §	%	95% CI §
Sensitivityα	88.5	(82.6-94.4)	91.4	(84.2-98.6)	84.5	(75.2-93.8)	97.3	(93.6-100.0)
Specificityα	45.6	(41.2-50.0)	42.7	(38.5-46.9)	34.4	(28.5-40.4)	44.4	(40.1-48.6)
Negative Predictive Valueα	94.5	(91.6-97.4)	97.9	(96.0-99.7)	90.3	(84.3-96.3)	99.1	(98.0-100.0)
Positive Predictive Valueα	27.3	(22.8-31.9)	14.6	(11.0-18.2)	23.4	(17.7-29.2)	19.8	(15.7-23.9)
Using threshold ≥3 points 7 days after trauma
Sensitivityα	85.8	(79.4-92.3)	91.4	(84.2-98.6)	82.8	(73.0-92.5)	95.9	(91.5-100.0)
Specificityα	57.5	(53.5-61.8)	54.0	(49.8-58.2)	42.6	(36.4-48.8)	55.8	(51.6-60.1)
Negative predictive valueα	94.6	(92.0-97.2)	98.3	(96.9-99.8)	91.2	(86.0-96.4)	99.0	(97.8-100.0)
Positive predictive valueα	31.8	(26.6-37.0)	17.5	(13.3-21.8)	25.5	(19.3-31.8)	23.4	(18.7-28.2)

With 95% confidence interval.

As per Lagacé-Legendre and colleagues.[86] J Head Trauma Rehabil (2021).

PPCS with an impact on normal activities due to head trauma, according to patient.

For those who were worker or student at the time of the mTBI.

3 symptoms of 3-4 points using the Rivermead Post-concussion Questionnaire.

POCS, Post-Concussion Symptoms Rule; PPCS, persistent post-concussion symptoms.

Discussion

We developed an ED-friendly CDR for the early stratification of PPCS after an mTBI. The lack of multivariable prognostic models for patients with mTBI has been highlighted in a 2015 systematic review, which concluded that none of the 26 included studies found a multivariable prognostic model able to predict individual mTBI patient outcomes adequately.[72] Indeed, most frameworks were based on small cohorts with inadequate sample size or low event per predictor ratio,[63,95,96] were only internally validated[32] or the model calibration was not reported or was poor.

Since then, additional models have been proposed,[3,10,11,39,73] but to our knowledge, none of them have been externally validated. Two studies had only conducted internal bootstrap validation.[11,39] Moreover, some of them used retrospective data,[3] had a high rate of loss to follow-up[10] or recruited very small cohorts.[73] Otherwise, Lingsma and colleagues tried to validate some existing models using the Transforming Research and Clinical Knowledge in Traumatic Brain Injury Pilot (TRACK-TBI) dataset,[71] but they concluded that the models performed unsatisfactorily. More recently, Mikolic and colleagues tried to externally validate three predictive models[13] using the Collaborative European NeuroTrauma Effectiveness Research in Traumatic Brain Injury (CENTER-TBI) Study dataset. Still, the authors concluded that none of these models has good calibration and discrimination for early prediction of PPCS.[13]

Although the diagnostic criteria for mTBI and PPCS were different in these studies, some common factors were found to be strongly associated with PPCS. The most frequently reported predictors were sex,[2,10,31-41] prior TBI,[3,10,31,32,35,42] history of mental health disorder,[8,10,11,14-16,34,40,43-45] headache in ED,[11,12,15,16,32] and patient-reported symptoms 1-2 weeks after trauma,[3,8,32,55-59] and selected variables in the PoCS Rule are concordant with these known risk factors. Moreover, the PoCS Rule includes two well-known concussion-associated conditions: sleep disturbance[8,57,58] and cervical sprain.[39,52,53] Even if the level of education was not associated with PPCS in our univariate analyses, it was tested based on suggestions in the literature, but not kept in the final model, as it did not demonstrate significance after backwards regression analyses. Headache[11,12,15,16,32] in the ED was tested using a 10-point severity scale, but this had no added value compared with using headache as a dichotomized variable. Despite testing different thresholds, the four biomarkers sampled did not improve the model. This is consistent with other recent publications.[68,97,98]

Our study has limitations. First, we were unable to recruit all consecutive patients. However, four sites we able to perform 24/7 patient recruitment. Even if some patients (19%) were lost to follow-up or had missing outcome data, we were able to compare the characteristics of patients included in our analysis[99] to those lost to follow-up (Supplementary Files S7–S10). Therefore, we believe that our results can be generalized to non-hospitalized ED patients presenting during the acute phase after a mTBI. With regard to the outcome measurement tool, the RPQ has been criticized, but it is widely used in clinical settings and was found to be as useful as an objective neuropsychological assessment to help target patients who would most benefit from further intervention following their trauma.[52,100] It is an easy tool to use in the context of limited resources.

As the sample size was initially estimated using a higher prevalence than that ultimately observed in our sample, the final model should include no more than seven variables. Nevertheless, the event per predictor ratio was adequate (12:1 and 16:1 for step 1 and step 2, respectively) for the validation cohort, and the performance was still good when the rule was applied to this prospective cohort. No impact measure was explicitly obtained in the derivation cohort. Still, a specific impact measure was available for the validation cohort, and this measure was strongly associated with the RPQ score at 90 days (AUC = 0.90).

Because Canada has public health care coverage, litigation was not considered. The type of insurance coverage would have been challenging to collect and assess. This would have defeated the purpose of creating an easy-to-use rule that would focus on clinical and biomarker data. Even though coverage may differ in some provinces/countries, the most common type of compensable mTBI would be motor vehicle accidents. This type of trauma mechanism concerned 16.1% and 14.2% of our derivation and validation cohorts, respectively. As shown in Table 2, we found no correlation between this injury mechanism and PPCS (OR: 0.97, 95% CI: 0.51-1.86) and therefore, even though this may be considered a limit to our study, we believe this had little impact on our results. Some authors did find an association between litigation and the number of PPCS[19] and psychological distress.[54] However, even though this factor could play a role in the recovery of some patients, several authors agree that the presence of persistent symptoms is likely the result of a complex interaction between neuropathological, psychological, and social factors.[52] Secondary monetary gain would be a rare phenomenon and would not be the only reason for which a person could amplify or invent symptoms.[101] Generally, most patients who sought compensation were not seeking /receiving compensation by 3 months.[102]

Finally, we also used information from the 7-day interview as a proxy for the Delphi definition of PPCS.[86] This may have slightly impacted our results as we cannot confirm that the symptoms appeared within hours of the trauma.

As recommended, we used multiple imputation and bootstrap models selection to handle missing data among ED predictors to add stability to the prognostic models.[91] We also performed a simple imputation using 30-day follow-up data for patients who were not reached at the 7-day follow-up (13% and 8% in the derivation and validation cohorts, respectively). Performing imputation with data collected closer to the outcome in time could artificially increase the strength of the predictors. However, this strategy is representative of standard follow-up practice, with new telephone attempts when the patient has not been contacted. Moreover, since data were missing for all variables when the patients could not be reached, no predictor benefited from this method.

Age was analyzed in categories, reflecting the inverted “U curve” of its association with PPCS. In contrast, most studies that found no association between those factors consider age as a continuous variable.[15,33,34,45] Our analyses ensure a better adjustment for this variable, and our results are consistent with that of others who found that older age may even be slightly protective.[11,103]

Because the literature on PPCS diagnostic criteria is heterogeneous, the PoCS Rule was validated with different outcome measure definitions, including non-return to work.[35] If patients at risk of PPCS are identified early, some rehabilitation interventions to control “negative perceptions of mTBI”[104] and illness perceptions[61,105] have been described as beneficial. Further, some data support interventions like cervical[53] and vestibular rehabilitation and multidisciplinary care.[106] The scientific community has been attempting to better understand the bio-psycho-social factors that influence the prognosis of patients who sustained an mTBI. The PoCS Rule takes these concepts into account.[107,108]

In summary, our simple CDR may improve our health system by achieving three main goals: 1) the early identification of patients at higher risk of PPCS (≥ 90 days), who may benefit from tailored post-ED interventions; 2) the early identification of patients at low risk of PPCS, who may not need clinical follow-ups; and 3) the catalyzation of research and clinical efforts to enhance the quality of care and prevent complications. The PoCS Rule will be clinically useful in helping emergency physicians quickly stratify the risk of PPCS in mTBI patients and better plan post-discharge resources, which may result in a better use of human and financial resources, while improving the quality of care.