Prediction of risk of prolonged post-concussion symptoms: Derivation and validation of the TRICORDRR (Toronto Rehabilitation Institute Concussion Outcome Determination and Rehab Recommendations) score.

BACKGROUND: Approximately 10% to 20% of people with concussion experience prolonged post-concussion symptoms (PPCS). There is limited information identifying risk factors for PPCS in adult populations. This study aimed to derive a risk score for PPCS by determining which demographic factors, premorbid health conditions, and healthcare utilization patterns are associated with need for prolonged concussion care among a large cohort of adults with concussion. METHODS AND
FINDINGS: Data from a cohort study (Ontario Concussion Cohort study, 2008 to 2016; n = 1,330,336) including all adults with a concussion diagnosis by either primary care physician (ICD-9 code 850) or in emergency department (ICD-10 code S06) and 2 years of healthcare tracking postinjury (2008 to 2014, n = 587,057) were used in a retrospective analysis. Approximately 42.4% of the cohort was female, and adults between 18 and 30 years was the largest age group (31.0%). PPCS was defined as 2 or more specialist visits for concussion-related symptoms more than 6 months after injury index date. Approximately 13% (73,122) of the cohort had PPCS. Total cohort was divided into Derivation (2009 to 2013, n = 417,335) and Validation cohorts (2009 and 2014, n = 169,722) based upon injury index year. Variables selected a priori such as psychiatric disorders, migraines, sleep disorders, demographic factors, and pre-injury healthcare patterns were entered into multivariable logistic regression and CART modeling in the Derivation Cohort to calculate PPCS estimates and forward selection logistic regression model in the Validation Cohort. Variables with the highest probability of PPCS derived in the Derivation Cohort were: Age >61 years ([Formula: see text] = 0.54), bipolar disorder ([Formula: see text] = 0.52), high pre-injury primary care visits per year ([Formula: see text] = 0.46), personality disorders ([Formula: see text] = 0.45), and anxiety and depression ([Formula: see text] = 0.33). The area under the curve (AUC) was 0.79 for the derivation model, 0.79 for bootstrap internal validation of the Derivation Cohort, and 0.64 for the Validation model. A limitation of this study was ability to track healthcare usage only to healthcare providers that submit to Ontario Health Insurance Plan (OHIP); thus, some patients seeking treatment for prolonged symptoms may not be captured in this analysis.
CONCLUSIONS: In this study, we observed that premorbid psychiatric conditions, pre-injury health system usage, and older age were associated with increased risk of a prolonged recovery from concussion. This risk score allows clinicians to calculate an individual's risk of requiring treatment more than 6 months post-concussion.

Introduction

Concussions are common, affecting almost 2% of the population at any time [1]. Most symptoms resolve spontaneously in the first few weeks following injury; however, there are highly variable estimates (15% to 40%) [2-4] of the proportion that experience prolonged post-concussion symptoms (PPCS) in the months to years following injury. Numerous factors including, but not limited to, age, sex, headache history, history of mental health problems, loss of consciousness (LOC), and mechanism of injury have been proposed to influence recovery from concussion [5-17]. However, the literature is conflicting and controversial as to which predictive factors affect the development of PPCS in adulthood [18]. The lack of consensus may be due to a number of limitations with previous research including small sample sizes, specific populations studied (e.g., athletes or children), limited duration of follow up, and reliance on self-report measures. In a pediatric concussion population, the Predicting and Preventing Post-Concussion Problems in Pediatrics (5P) study followed individuals presenting to an emergency department (ED) in Canada to determine an absolute risk calculation for persistent concussion-related symptoms at 1 month after injury [19], but no risk score has been developed for adults with concussion.

Health data for each of the more than 13 million residents [20] of Ontario, Canada can be tracked through their unique Ontario Health Insurance Plan (OHIP) number. By linking administrative health databases at the Institute of Clinical and Evaluative Sciences (ICES), cohorts can be created using diagnostic codes, examine healthcare utilization (doctor visits, emergency visits, hospitalizations, wait times, etc.), perform retrospective analyses, and determination of risk [21,22], on the entire province’s population. ICES is an independent, nonprofit research institute funded by an annual grant from the Ontario Ministry of Health and Long-Term Care (MOHLTC). As a prescribed entity under Ontario’s privacy legislation, ICES is authorized to collect and use healthcare data for the purposes of health system analysis, evaluation, and decision support. Secure access to these data is governed by policies and procedures that are approved by the Information and Privacy Commissioner of Ontario.

The objective of this study was to derive a risk score for PPCS by determining which demographic factors, premorbid health conditions, and healthcare utilization patterns are associated with PPCS using a large cohort of adults with concussion identified using administrative health databases housed at ICES. We hypothesized that mental health issues, headache, and neurological issues would be associated with increased incidence of PPCS and that it would be feasible to derive a PPCS risk score using available information.

Methods

Ethics statement

This study was approved by the Research Ethics Board at University Health Network (ID# 15–9532). The data were anonymized, reported in aggregate, and cells with less than 6 were suppressed as per ICES Privacy Policy. Consent was waived as ICES is a prescribed entity under section 45 of Ontario’s Personal Health Information Protection Act. Section 45 authorizes ICES to collect personal health information, without consent, for the purpose of analysis or compiling statistical information with respect to the management of, evaluation or monitoring of, the allocation of resources to or planning for all or part of the health system. This study is reported as per the Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) statement (S1 Checklist).

Setting

Ontario had a population of 13,448,949 in the 2016 Census [20] and is the most populous province in Canada. It has the second largest geographical area. Approximately 15% of the population resides in areas classified as rural (defined by Statistics Canada as not residing in a metropolitan area or its surrounding area). The average age of residents of Ontario in the 2016 Census was 41.0 (no standard deviation was provided), and the median was 41.3 years. Females comprised 50.6% of the population. Visible minorities, as defined by Statistics Canada as “persons, other than Aboriginal peoples, who are non-white in colour,” accounted for 29.3% of the population. English is the most commonly spoken language (97.3%).

Data sources

Records were identified in ICES through the (1) National Ambulatory Care Reporting System (NACRS), for people diagnosed in an ED using ICD-10 diagnostic codes, or (2) the OHIP Physician Billing Database, for records containing diagnoses by a primary care physician using ICD-9 diagnostic codes. OHIP is a taxpayer-funded health insurance plan for all eligible residents of Ontario. Physicians, laboratories, and diagnostic facilities submit the patient’s OHIP identifier, type of visit, diagnosis, tests performed, etc. to OHIP to receive payment. All follow-up visits were linked using the individuals’ unique OHIP identifier and abstracted from the physician billing database. The premorbid health retrospective review was conducted using the relevant ICD-9 diagnostic codes (S1 Table) 5 years prior in the OHIP Physician Billing Database for each linked OHIP identifier and, additionally, included the Ontario Mental Health Reporting System (OMHRS) database for psychiatric hospital admissions. These datasets were linked using unique encoded identifiers and analyzed at ICES.

Study cohorts

The cohort consisted of adults ≥18 years of age with a valid OHIP number and a physician-diagnosed concussion between 2008 and 2014. Cases with injury index dates in 2015 and 2016 were excluded due to delay in OHIP reporting to ICES that did not permit 2 years healthcare tracking after index date. ICD-9 code 850 was used to identify concussions diagnosed by primary care physicians, and ICD-10 S06 was used to identify those diagnosed in the ED. Duplicate records, such as those with all variables identical or if multiple records with the same diagnostic billing code per person per physician per day (only 1 record will be counted), were removed. Records were excluded if there were incomplete demographic data, the individual was not a resident of Ontario, age was greater than 105 years, or if a date of death was listed before the index event date. There was a 365-day washout period for previous traumatic brain injuries (TBIs) to ensure that the cohort would not have duplication of identified index cases. To ensure cohort data contained only mild traumatic brain injuries (mTBIs), records of anyone with admission to hospital or death within 30 days of the index event were excluded (Fig 1). This concussion cohort was tracked for 2 years after their index date of injury (the date recorded by the physician at the initial point of access to the healthcare system associated with concussion diagnosis code) using their unique OHIP identifier for concussion-related follow-up care in the OHIP Physician Billing Database. Specialists identified for providing relevant care were as follows: neurosurgery, neurology, physical medicine, otolaryngology, ophthalmology, and psychiatry. Next, a 5-year retrospective review of frequency of premorbid healthcare system utilization and relevant premorbid diagnoses was performed. Diagnoses of interest were selected based on a literature review of the following predictive factors for prolonged concussion recovery: depression and anxiety, personality disorders, bipolar disorder, migraine, headache without migraine, sleep disorders, pain disorders, neurological disorders (multiple sclerosis, dementia, stroke, etc.), vestibular disorders (i.e., ataxia, gait disorders, dizziness not caused by a primary disease like hypotension or a neurological disorder), temporomandibular joint (TMJ) disorders, schizophrenia, alcohol misuse disorder and illicit drug use disorders, and prior TBIs, using the OHIP Physician Billing Database and OMHRS for schizophrenia, personality disorders, and bipolar disorder (S1 Fig). The diagnostic billing codes used are available in S1 Table.

Fig 1

Creation of the General Adult Cohort, Derivation Cohort, Validation Cohort, and the PPCS Cohort.

ED, emergency department; GP, general practice; ICES, Institute of Clinical and Evaluative Sciences; OHIP, Ontario Health Insurance Plan; PPCS, prolonged post-concussion symptoms.

PPCS does not have a validated or unanimously agreed upon definition for diagnosis or even accepted terminology. The World Health Organization (WHO) [23] criteria for post-concussion syndrome is 3 out of 8 possible symptoms (i.e., headache, dizziness, fatigue, irritability, insomnia, concentration or memory problems, intolerance of stress, emotion, or alcohol) within 30 days of a head injury sufficient to result in LOC. Criteria from the Diagnostic and Statistical Manual of Mental Health Disorders (DSM-IV) [24] included neuropsychological impairments as well as symptoms prolonged more than 3 months post-injury that were not present pre-injury or worsened post-injury. For the purpose of this analysis, PPCS was defined as individuals with concussion requiring assessment by 2 or more specialists appropriate for concussion-related symptoms (neurology, otolaryngology, psychiatry, neurosurgery, etc.) more than 182 days (6 months) after their injury index date. Six months was selected to allow patients with prolonged symptoms to access a primary care physician in order to obtain a referral for a specialist physician, which may have long wait times for a first visit.

The “Derivation Cohort” was composed of all adults diagnosed with a concussion between the years 2009 and 2013, and all analyses to create the predictive model were performed on this cohort. The “Validation Cohort” included all adults diagnosed with a concussion in either 2008 or 2014 and was used to validate the predictive model. The year 2008 was selected as it was assumed to have lower rates of referrals due to lower physician awareness on the topic of concussion. The year 2014 was selected due to increased awareness in the medical community due to increased research as well as high-profile injuries to athletes and military personnel that we believe would have resulted in higher rates of referrals to specialist physicians. Adults with an index date in either 2015 or 2016 did not have full 2 years of specialist healthcare tracking and were not included in the risk score analysis.

Analyses

An analysis plan was developed as part of the Dataset Creation Plan prior to initiating the study and included the univariate logistic regressions used to determine the variables entered into a stepwise multivariate regression model and then a multivariate logistic regression model in the Derivation Cohort, forward selection multivariate logistic regression model in the Validation Cohort, and bootstrapping of the Derivation Cohort. The Classification and Regression Tree (CART) and derivation of a risk score were prompted by the data analysis after the multivariate logistic regression model. Secondary descriptive analyses comparing the Derivation and Validation Cohorts were requested by reviewers. (S1 Text) All statistics were performed on SAS 9.4 (SAS Institute, Cary, North Carolina, USA) for Windows. Nineteen variables were selected a priori for analysis. Potential covariates were sex, age group, premorbid health conditions, frequency of primary care use in the year previous to the injury index date, and rurality (not residing in a metropolitan area). Univariate logistic regressions were performed between each premorbid condition, sex, and age group and the number of specialist visits in the persistent post-concussion symptoms cohort. Linear regressions were performed between the number of primary care visits the year before the injury and specialist follow-up visits. Chi-squared analysis of rurality classification of the record and inclusion in the Prolonged Cohort was performed. Variables with statistically significant (p < 0.05) odds ratio (OR) or significant linear regressions were entered into a stepwise regression model. Variables from the univariate analysis were retained in the model if they had a p-value less than 0.01 and underwent stepwise regression. Only variables that had a significant p-value less than 0.001 in the stepwise regression were retained for the multivariate logistic regression model. The variables that remained significant were entered into a CART analysis using the derivation concussion cohort with a diagnosis made between 2009 and 2013. Forward validation of the model was performed on the validation cohort with an index date in 2008 or 2014 with the same variables retained from the multiple covariate logistic regression in the derivation cohort. Risk score calculations were made for the significant nodes of the CART analysis using the methods described by Sullivan and colleagues [25] for deriving risk scores using the results from a multivariate logistic regression model on a binary outcome. The final model using variables retained from the multivariate logistic regression was validated internally in the Derivation Cohort (2009 to 2013) using bootstrap resampling of 1,000 randomly selected samples.

Results

The records of 1,567,508 individuals were initially identified. A total of 197,489 duplicate records were excluded, and 39,683 records were excluded for insufficient demographic data or potentially more severe injuries. The final Ontario Concussion Cohort totaled 1,330,336 records. This cohort was restricted to those with 2 years post-injury follow-up of healthcare surveillance (index years 2008 to 2014) and were 18 years of age or older at the index date resulting in 587,057 cases. Of this, 73,122 people (12.5%) met our criteria for PPCS (The Prolonged Cohort). The demographics are presented in Table 1. The “Derivation Cohort” was composed of 417,335 records. The “Validation Cohort” contained 169,722 records. Due to the high rates of psychiatric healthcare use prior to their injury, schizophrenia, and illicit drug use were not further analyzed in this model. Prevalence of the premorbid conditions is presented in Table 2.

Table 1

Demographics of the overall Adult Concussion Cohort (2008–2014) and subcohorts of Prolonged Symptoms (2008–2014), the Derivation Cohort (2009–2013), and the Validation Cohort (2008 and 2014).

		2008–2014 Adult Concussion Cohort	Adult Prolonged Symptoms Cohort	Derivation Cohort (2009–2012)	Validation Cohort (2008 and 2014)
Total		587,057	73,122	417,335	169,722
Sex (%)	Female	248,765 (42.4%)	35,344 (48.3%)	175,962 (42.2%)	72,803 (42.9%)
	Male	338,292 (57.6%)	37,778 (51.7%)	241,373 (57.8%)	96,919 (57.1%)
Age group (%)	18–30	181,811 (31.0%)	13,544 (7.4%)	130,202 (31.2%	51,609 (30.4%)
	31–40	84,587 (14.4%)	8,530 (10.0%)	59,903 (14.4%)	24,684 (14.5%)
	41–50	88,463 (15.1%)	11,157 (12.6%)	62,937 (15.1%)	25,526 (15.0%)
	51–60	75,741 (12.9%)	11,483 (15.2%)	53,396 (12.8%)	22,345 (13.2%)
	61–80	100,703 (17.2%)	19,979 (19.8%)	71,399 (17.1%)	29,304 (17.3%)
	>81	55,752 (9.5%)	8,429 (15.1%)	39,498 (9.5%)	16,254 (9.6%)
Diagnosed (%)	ED	483,051 (82.3%)	59,172 (80.9%)	343,084 (82.4%)	139,084 (82.0%)
	GP physician	104,006 (17.7%)	13,950 (19.1%)	73,368 (17.6)	30,638 (18.0%)
Mean GP visits 1 year before index date (SD)		14.0 (SD 18.7)	19.1 (SD 20.6)	13.9 (SD 18.6)	14.2 (SD 18.7)
Median GP visits 1 year before index date (range)		9 (0–365)	13 (0–365)	9 (0–365)	9 (0–365)

ED, emergency department; GP, general practice.

Table 2

Prevalence of premorbid conditions in the Adult Concussion Cohort and the Adult Prolonged Symptoms, Derivation Cohort (2009–2013), and the Validation Cohort (2008 and 2014) subcohorts.

Premorbid factor		2008–2014 Adult Concussion Cohort (% prevalence in cohort)	Derivation Cohort (2009–2013)	Validation Cohort (2008 and 2014)	Adult Prolonged Symptoms Cohort (% prevalence in cohort)	Percent with prolonged symptoms
Mental health	Anxiety and depression	244,045 (41.6%)	172,521 (41.3%)	71,524 (42.1%)	34,856 (47.7%)	14.3%
	Personality disorders	12,204 (2.1%)	8,557 (2.1%)	3,647 (2.2%)	1,856 (2.5%)	15.2%
	Bipolar disorder	1,627 (0.27%)	1,098 (0.26%)	529 (0.31%)	233 (0.32%)	14.3%
	Schizophrenia	12,801 (2.2%)	9,091 (2.2%)	3,170 (2.2%)	1,772 (2.4%)	13.8%
	Illicit drug use	34,443 (5.9%)	24,386 (5.8%)	10,057 (5.9%)	4,980 (6.8%)	14.5%
	Other	74,036 (12.6%)	52,405 (12.6%)	21,631 (12.7%)	11,321 (15.5%)	15.3%
Neurological disorders		22,082 (3.8%)	15,545 (3.7%)	6,537 (3.9%)	3,312 (4.5%)	15%
Sleep disorders		179,310 (30.5%)	126,844 (30.4%)	52,466 (30.9%)	26,788 (36.6%)	15%
Pain disorders		146,168 (24.9%)	103,323 (24.8%)	42,845 (25.2%)	21,759 (29.8%)	14.9%
Migraine		36,722 (6.3%)	25,405 (6.2%)	10,767 (6.3%)	5,401 (7.4%)	14.7%
Headache without migraine		40,083 (6.8%)	28,372 (6.8%)	11,711 (6.9%)	5,810 (7.9%)	14.5%
Vestibular/balance disorders		348,185 (59.3%)	246,986 (59.2%)	101,199 (59.6%)	49,012 (67.0%)	14.1%
TMJD		4,371 (0.7%)	3,059 (0.7%)	1,312 (0.8%)	637 (0.9%)	14.6%
Prior concussion/TBI		25,520 (4.3%)	17,349 (4.2%)	8,171 (4.8%)	3,273 (4.5%)	12.8%

TBI, traumatic brain injury; TMJD, temporomandibular joint dysfunction.

Overall, there were 248,765 (42.4%) females and 338,292 (57.6%) males included in the analysis. There were 35,344 (48.3%) females and 37,778 (51.7%) males in the Prolonged Cohort. Approximately 80% had a concussion diagnosed by an ED physician.

The results from the univariate logistic regressions are presented in S2 Table. There was a significant relationship (p < 0.0001, unadjusted r2 = 0.265) between the number of primary care visits the year prior to the index concussion and requiring specialist visits for PPCS. The mean number of primary care visits for the Prolonged Cohort in the year prior to concussion was 14.0 visits with a standard deviation of 18.7; the median was 9 visits (range 0 to 365). Quartiles for primary physician visits in the Derivation Cohort were 0 to 4 visits (low usage), 5 to 8 visits (average usage), 9 to 14 visits (above average usage), and >15 visits the previous year (high usage); the categories were used for all later analyses instead of continuous value. There was no relationship between living in either a rural or urban/suburban area of Ontario and meeting prolonged cohort criteria (χ2 = 0.16, p = 0.69); however, those with a rural classification had a longer wait to access care (rural mean 213.5 (SD 196.4) days after index, nonrural mean 202.3 (SD 197.0) days after index, p < 0.0001). The location of diagnosis (ED or primary care) was not significantly associated with the number or type of specialist visits for prolonged concussion-related symptoms (OR 1.00, CI 0.996 to 1.006, p = 0.83). All covariates except for rurality and ED/ primary care diagnosis were entered into the stepwise model (S2 Table). Prior TBI (p = 0.21) and headache without migraine (p = 0.86) were not retained in the stepwise model. TMJ disorders were not retained in the multivariate logistic regression model (p = 0.001). Sex was also excluded due to small effect size (effect 0.001) despite reaching the significance threshold (p < 0.0001) (Table 3 and Fig 2). The intercept of the multivariate logistic regression model was −0.92. The area under the curve (AUC) for this model was 0.79 (95% CI 0.78 to 0.80) (Fig 3), and when the model was internally validated by bootstrap resampling, the AUC was 0.79 (95% CI 0.77 to 0.81).

Table 3

Multivariate logistic regression in the Derivation Cohort (2009–2013).

		OR	95% confidence interval	p-value
Mental health	Bipolar disorder	4.28	3.86–4.76	<0.0001
	Personality disorder	2.40	2.31–2.50	<0.0001
	Anxiety/depression	1.32	1.31–1.34	<0.0001
	Other	1.34	1.31–1.37	<0.0001
Number of primary care visits year prior	>15	5.51	5.41–5.62	<0.0001
	8–14	3.04	2.99–3.10	<0.0001
	5–7	1.92	1.89–1.97	<0.0001
	0–4 (reference group)
Sex (female)		0.95	0.94–0.97	<0.0001
Age group	18–30	0.30	0.29–0.36	<0.0001
	31–40	0.35	0.35–0.36	<0.0001
	41–50	0.49	0.48–0.51	<0.0001
	51–60	0.69	0.68–0.71	0.00002
	61–80	1.63	1.62–1.72	<0.0001
Neurological disorders		1.76	1.70–1.81	<0.0001
Pain disorders		1.26	1.21–1.30	<0.0001
Migraine		1.26	1.23–1.29	<0.0001
Sleep disorders		0.95	0.92–0.99	0.0001
Vestibular disorders		1.47	1.45–1.49	<0.0001
TMJD		1.14	1.05–1.19	0.001 (NS)

NS, not significant; OR, odds ratio; TMJD, temporomandibular joint dysfunction.

Fig 2

Multivariate logistic regression results in Derivation Cohort with OR and 95% confidence intervals.

The reference group for primary care visits comparisons was 0–4 primary care visits. Value are also presented in Table 2. OR, odds ratio; TMJ, temporomandibular joint disorders.

Fig 3

AUC of Derivation Cohort 0.79 (95% CI 0.78–0.80).

It has 75% sensitivity and 70% specificity. AUC, area under the curve; ROC, receiver operating characteristic.

Variables entered into the CART analysis with outcome of number of specialist visits included sex, age group, healthcare use frequency, personality disorders, bipolar disorder, depression and anxiety, migraine, pain disorders, neurological disorders, sleep disorders, vestibular disorders, and other mental health disorders. Personality disorders were the primary node in this analysis; however, given the low prevalence of personality disorders in the general population and difficulty of reliable diagnosis, personality disorders and bipolar disorder were removed and a second CART was performed (Fig 4). The significant nodes (99.5%) of the CART were bipolar disorder, personality disorder, high number of primary care appointments the year prior, premorbid diagnosis of depression and anxiety, and age group. Those with premorbid pain or neurological disorders also had the greatest average number of specialist visits (9) greater than 6 months after their injury, although these nodes were not significant for prediction. Forward model validation in the Validation Cohort included retained variables primary care usage, age group, vestibular disorders, personality disorders, anxiety and depression, neurological disorders, other mental health disorders, migraine, and bipolar disorders and accounted for 99.6% of the model (S2 Fig) with an AUC of 0.64 (95% CI 0.63 to 0.65) (Fig 5).

Fig 4

Decision rule tree from Derivation Cohort CART.

Terminal nodes with the highest risk of PPCS are for individuals with personality disorders or bipolar disorder or adults over 61 years with high levels of primary care usage in the year before injury with a history of anxiety and/or depression. CART, Classification and Regression Tree; PPCS, prolonged post-concussion symptoms.

Fig 5

AUC of Validation Cohort 0.64 (95% CI 0.63–0.65).

The validated model has a sensitivity of 75% and a specificity of 47%. AUC, area under the curve; ROC, receiver operating characteristic.

Risk modifiers were bipolar disorder (52% risk), personality disorder (45%), more frequent than monthly primary care use (46%), 9 to 12 primary care visits in the year prior (34%), 5 to 8 primary care visits the year prior (26%), 0 to 4 primary care visits the year prior (20%), preexisting anxiety or depression (33%), 61-plus years (55%), 51 to 60 years (23%), 41 to 50 years (20%), 31 to 40 years (17%), and 18 to 30 years (15%). For ease of use by clinicians, the raw estimates for each variable were converted to a point value, and a risk point system to estimate risk score for adults requiring 2 or more specialist visits more than 6 months after their concussion in the Derivation Cohort was determined using the method detailed by Sullivan and colleagues [25] (Table 4). The Toronto Rehabilitation Institute Concussion Outcome Risk Determination and Rehab Recommendations (TRICORDRR) risk calculator tool is available in S1 Tool TRICORDRR and online (https://kite-uhn.com/tricordrr). A hypothetical patient aged 50 to 60 years (0 point) with 6 primary care visits in the year prior (0 point), history of depression (1 point) but no other psychiatric history (0 point) would have a total point score of 1 and corresponding risk of 34%. When risk for PPCS is calculated using the equation containing the raw values for each variable instead of the point system, the risk is 31.6%. A 20-year-old (−3) with no primary care visits the year before (−1) and no mental health history would have a point of −4, corresponding risk of 9%, and a raw risk of 11.8%. A patient in their 60s (3) with more than monthly primary care visits (4) and diagnosis of anxiety (1) would have a point score of 8 corresponding to a risk of 78% and a raw calculated risk of 76.2%.

Table 4

Factor specific risk scores of the Derivation Cohort and estimated risk of the combined point for an individual.

		Point value
Bipolar disorder		4
Personality disorder		3
Anxiety and depression		1
Primary care visits year prior to injury	>15	3
	9–14	1
	5–8	0
	<4	−1
Age group	18–30	−3
	31–40	−2
	41–50	−1
	51–60	0
	>61	4
Combined point score		Estimated risk
−4		9%
−3		14%
−2		19%
−1		23%
0		28%
1		34%
2		44%
3		47%
4		54%
5		61%
6		67%
7		73%
8		78%
9		82%
10		86%
11		89%
≥12		91%

This study was approved by the Research Ethics Board at the University Health Network.

The proportion of cases that had PPCS for each point score strata was identified using the Validation Cohort (Fig 6). The proportion that actually met criteria was determined by dividing the PPCS cases by total number of patients who had that specific point score. Each risk stratum is comprised of multiple subgroups (i.e., individuals can be assigned to that strata based on different combinations of points representing their individual premorbid and demographic factors). More than 51% of the Validation Cohort scored a combined risk score of 0 or lower on the TRICORDRR.

Fig 6

Identification of cases that met PPCS criteria in the Validation Cohort. Median risk strata was 0. PPCS, prolonged post-concussion symptoms.

Discussion

This study observed that 12.5% of adults diagnosed with a concussion would require specialized medical care related to their injury 6 months or longer post-injury. Older age (over 61 years), premorbid psychiatric and mental health history, and high usage of primary care in the year before injury were associated with greater risk of PPCS. To the authors’ knowledge, this is the largest study to date, with almost 600,000 adults, examining predictive factors for PPCS requiring specialized healthcare. The Ontario Concussion Cohort is also the second largest concussion cohort globally; only the CDC’s TBI-related ED visits, hospitalizations, and deaths (EDHDs) cohort is larger at 2.87 million people [26], although it also included moderate to severe TBIs. The size of this study enabled both the estimate of the frequency of PPCS and development of a risk stratification model.

Prolonged symptoms cohort

Approximately 50.3% of all cases included in this analysis visited a specialist in the 2 years after the index injury date for either a concussion or concussion-related symptom. The Prolonged Symptom Cohort consisted of 73,122 cases, 12.5% of the overall cohort, which met the criteria of >2 specialist visits >6 months after the index injury (12.4% of the Derivation Cohort met criteria, and 12.7% of the Validation Cohort met criteria). This is consistent with estimates suggesting that 10% to 20% of adults with a concussion continue to experience symptoms more than 3 months post-injury [2,3]. This is based on the presence of a diagnosis of concussion or concussion-related symptoms (e.g., headache, migraine, poor mental health) by physicians in Ontario (although there is no billing code in Ontario for “prolonged post-concussion symptoms”). We identified people who were symptomatic enough to require a specialist more than 6 months post-injury from the prolonged symptom cohort. When criteria for PPCS was relaxed to identify those with specialist visits 3 months after index date, 175,135 cases (29.8%) were identified, which is also consistent with estimates of PPCS. However, only 72.2% of cases had follow-up by a primary care physician within 3 months of injury, and some of the visits may have been for pre-injury referrals depending upon the wait time(s) to see that particular specialist. The 12.5% PPCS is a more conservative estimate that most likely reflects those cases with true prolonged symptoms.

Risks associated with mental health conditions

The most predictive factors in developing PPCS were a prior diagnosis of mental health problems, especially depression, anxiety, bipolar and personality disorders, high frequency of pre-injury primary healthcare use, and age. Consistent with the literature, mental health and psychiatric premorbid diagnoses were associated with greater healthcare use for PPCS [14-17]. Personality disorders and bipolar disorders have the greatest predictive risk for developing PPCS. People with these diagnoses were identified in OMHRS such that the diagnosis was made by either a psychiatrist or a psychologist, and these individuals would have received some form of inpatient treatment for their mental health. Personality disorders in particular have been found to be associated with increased healthcare utilization independent of physical health [27,28]. Bipolar disorder was the most significant predictive factor for developing PPCS, although it is extremely rare, difficult to diagnose, has high similarity with other psychiatric disorders, and most clinicians will probably encounter few in their practice. Furthermore, OHIP billing code for bipolar disorder does not differentiate between subcategories of the disorder. Those with these diagnoses typically have frequent primary care visits and comorbid conditions like illicit drug use disorders and other mental health conditions [29].

Anxiety and depression, both premorbid and after injury, have been consistently identified as predictive factors of PPCS in adult and pediatric populations. Anxiety in particular has been shown to influence severity of symptoms of acute concussion and has mediating effects on post-injury catastrophization [30], somatoform disorders, and other post-concussion symptoms like sleep disorder, pain, and headache.

Prior healthcare utilization

High frequency of primary care utilization in the year prior to injury as a predictor of prolonged symptoms is a novel finding for concussion but has been found in other conditions such as spinal pain [31]. There are a number of potential reasons including premorbid health concerns or healthcare-seeking behaviors, which may be mediated by factors such as sex, education, health literacy, premorbid coping strategies, a tendency to catastrophize, and somatoform disorders [32]. Interestingly, the mean number of primary care visits associated with premorbid health conditions like mental health and migraine did not differ statistically significantly from the overall cohort, so this finding may indeed be a result of healthcare-seeking behaviors, somatization, and innate coping strategies.

Age-related risks

Recovery from concussion has been shown to vary based upon the individual’s age [7,8]; however, older adults are rarely studied in concussion research, making the increased risk of PPCS in seniors a potentially novel finding. This study suggests that younger adults are more likely than their older counterparts (age >61) to recover post-concussion. This may relate to more prevalent premorbid health conditions in the older group (as shown with a significantly higher (p < 0.0001) number of primary care visits) than in younger age groups. It might also relate to reduced neurocognitive reserve capacity in the older group. Neurocognitive reserve capacity is a proposed protective mechanism that may enable some individual’s brain to adapt to or recover from pathology associated with neurological diseases and/or psychiatric disorders and is believed to decrease with the aging process [33]. Indeed, there has been some preliminary evidence to suggest that lower cognitive reserve is associated with worse cognitive outcome post-mTBI [34] as well as risk for persistent post-concussive symptoms [35].

Model fit

The model fit in all models was moderate though lower in the Validation Cohort. This may be due to the years selected a priori for the Validation Cohort. The years included for the Validation Cohort were selected for clinical reasons. Concussion awareness was assumed to be lower in 2008 than in 2014 [1] among both healthcare providers and the general public because there were more peer-reviewed papers published in 2014 than 2008 as indexed by PubMed; the release of treatment guidelines (e.g., Ontario Neurotrauma Foundation Guidelines for Concussion/Mild Traumatic Brain Injury and Persistent Symptoms in 2013 [36]; updates on the Consensus Statement on Concussion in Sport that were published in 2008 [37] and 2012 [38]) was assumed to have increased adherence to best practices in 2014, and, also, there were high-profile concussions among professional athletes. The 2 years (2008 and 2014) were adopted to balance the low and high awareness of concussion at either end of the cohort timeline. The proportion of cases that were referred for additional healthcare for PPCS was higher in 2014 than 2008 (13.1% in 2014 cf. 12.3% in 2008). Furthermore, there were fewer diagnosed concussions in 2014 than 2008, and the decrease in cases was not offset by the increase in people meeting the PPCS criteria, particularly in age groups under 51 years as the Validation Cohort had slightly greater proportions of adults in older age groups than the Derivation Cohort, but these differences were not statistically significant (range p = 0.45 to 0.62). This discrepancy may have affected accuracy of the risk estimates, particularly in lower-risk younger age groups without a psychiatric history. There was less concordance between the observed prevalence for each point estimate between the Derivation and Validation Cohorts particularly of risk score of 1 to 5 points, and this may due to the multiple possible combinations of risk factors (i.e., age group, frequency of primary care usage, and mental health status that may generate that point score).

Other potential risk modifiers

The branches on the CART analysis for premorbid neurological disorders were below the threshold for significance, and, therefore, neurological disorders were not included in the formal risk score; however, these conditions are important clinically as they have the highest number of visits with specialist physicians compared with other premorbid conditions. This may be due to decreased neurocognitive reserve associated with the neuropathology of these disorders. The additional neurological burden resulting from the concussions may further exasperate premorbid symptoms associated with their neurological disorders.

The literature is inconsistent for the role of confounding variables like migraine history, sex, headache, and sleep disorders in developing PPCS. In this model, although some retained significance through the multivariate regression step, they were not retained in the CART analysis and were not included in the final risk score calculation. There are other prospective cohort studies that are also analyzing possible factors for risk of developing PPCS such as TRACK-TBI or the NCAA cohort. A Nelson and colleagues study [39] included all people with mild TBI at level 1 trauma centers including some who were CT scan positive. They acknowledge that this may not reflect the broader mTBI population; however, they clearly identify that a significant proportion 53% were having difficulties at 12 months. A study by Stein and colleagues [40] also confirmed our findings that those with history of preexisting mental health issues were at higher risk of developing complications from their mild TBI. This study also included 66% who were severe enough for admission to hospital and 24.5% admitted to ICU. This study involves much more severe TBI than our cohort. Risk factors for probable depression at 6 months after mTBI included less education (adjusted OR, 0.89; 95% CI, 0.82 to 0.97 per year), being Black, self-reported psychiatric history, and injury resulting from assault or other violence. Unfortunately, no race-based or individual educational data were available. The Stein group did not calculate a risk score, although they clearly identified similar risk factors to our study; however, their more severely injured population precludes direct comparison to our study group.

Future directions

The TRICORDRR tool is currently being prospectively validated by staff physicians at the Hull-Ellis Concussion and Research Clinic at Toronto Rehabilitation Institute–University Health Network.

Limitations

We acknowledge some limitations. For example, we were unable to verify the diagnoses of the included cases in this study in the medical charts and are reliant upon doctors recording the correct diagnostic billing code. As there is also no currently accepted definitive diagnostic criteria or billing code for “prolonged post-concussion symptoms,” we had to create our own criteria to identify those most likely to be seeking medical treatment for PPCS. Furthermore, sports medicine physicians are classified as primary care physicians by their OHIP specialty code, and, therefore, the patients they treated are not included in this analysis of specialist care, although they are a frequent provider of concussion-related healthcare. Although we tried to reduce the likelihood of contamination of this cohort with individuals requiring pre-injury referrals to specialized physicians, the date the referral was made to the specialist was unknown, and, thus, some referrals may be unrelated to a concussion. This analysis also was unable to include injury-specific confounding variables such as duration and depth of LOC, post-traumatic amnesia (PTA), effects of medico-legal issues (e.g., workplace-related injuries or litigation from motor vehicle collisions), mechanism of injury, initial symptom presentation, or factitious disorders. The decision to use index years 2008 and 2014 for the Validation Cohort was made prior to initiation of the study and may have affected the performance of the validation model. Knowledge of concussions and their appropriate treatment in 2008 was considerably less than in subsequent years included in this study, and this may have influenced the validation.

Conclusions

The results of the current study should allow the first primary care or emergency medicine providers who see an individual with concussion to quickly and easily determine a patient’s risk of developing PPCS while still in the acute stage of injury. This, in turn, should result in earlier tailoring of treatment with concussion education and reassurance of high likelihood of a good outcome to low-risk individuals, while higher-risk patients might receive earlier psychological intervention, prescription for subthreshold aerobic exercise [41], and referrals to specialist physicians and allied healthcare professionals as needed.

(DOCX)

Click here for additional data file.

Premorbid health conditions diagnostic codes.

(DOC)

Click here for additional data file.

Univariate logistic regressions in the Prolonged Concussion Symptom Cohort (2008–2014).

(DOC)

Click here for additional data file.

Methodology for deriving risk factors for persisting post-concussion symptoms using the Ontario Concussion Cohort.

ED, emergency department; ENT, otolaryngology; NACRS; National Ambulatory Care Reporting System; OHIP, Ontario Health Insurance Plan; TBI, traumatic brain injury; TMJ, temporomandibular joint disorder.

(TIF)

Click here for additional data file.

Forward logistic regression modeling in Validation Cohort with selection stop step indicated (vertical line after bipolar disorder).

SBC, Schwarz Bayesian information Criterion for model goodness of fit; lower SBC better fit.

(TIF)

Click here for additional data file.

TRICORDRR (Toronto Rehabilitation Institute Concussion Outcome Risk Determination and Rehab Recommendations) calculator tool.

(DOC)

Click here for additional data file.

Dataset Creation Plan (includes Analysis Plan).

(PDF)

Click here for additional data file.

10 Nov 2020

Dear Dr Langer,

Thank you for submitting your manuscript entitled "Predicting Prolonged Post-Concussion Symptoms using the TRICORDRR (Toronto Rehabilitation Institute Concussion Outcome Risk Determination & Rehab Recommendations)" for consideration by PLOS Medicine.

Your manuscript has now been evaluated by the PLOS Medicine editorial staff as well as by an academic editor with relevant expertise, and I am writing to let you know that we would like to send your submission out for external peer review.

However, before we can send your manuscript to reviewers, we need you to complete your submission by providing the metadata that is required for full assessment. To this end, please login to Editorial Manager where you will find the paper in the 'Submissions Needing Revisions' folder on your homepage. Please click 'Revise Submission' from the Action Links and complete all additional questions in the submission questionnaire.

Please re-submit your manuscript within two working days, i.e. by .

Login to Editorial Manager here: https://www.editorialmanager.com/pmedicine

Once your full submission is complete, your paper will undergo a series of checks in preparation for peer review. Once your manuscript has passed all checks it will be sent out for review.

Feel free to email us at plosmedicine@plos.org if you have any queries relating to your submission.

Kind regards,

Caitlin Moyer, Ph.D.,

Associate Editor

PLOS Medicine

7 Jan 2021

Dear Dr. Langer,

Thank you very much for submitting your manuscript "Predicting Prolonged Post-Concussion Symptoms using the TRICORDRR (Toronto Rehabilitation Institute Concussion Outcome Determination and Rehab Recommendations)" (PMEDICINE-D-20-05392R1) for consideration at PLOS Medicine.

Your paper was evaluated by a senior editor and discussed among all the editors here. It was also discussed with an academic editor with relevant expertise, and sent to three independent reviewers, including a statistical reviewer. The reviews are appended at the bottom of this email and any accompanying reviewer attachments can be seen via the link below:

[LINK]

In light of these reviews, I am afraid that we will not be able to accept the manuscript for publication in the journal in its current form, but we would like to consider a revised version that addresses the reviewers' and editors' comments. Obviously we cannot make any decision about publication until we have seen the revised manuscript and your response, and we plan to seek re-review by one or more of the reviewers.

In revising the manuscript for further consideration, your revisions should address the specific points made by each reviewer and the editors. Please also check the guidelines for revised papers at http://journals.plos.org/plosmedicine/s/revising-your-manuscript for any that apply to your paper. In your rebuttal letter you should indicate your response to the reviewers' and editors' comments, the changes you have made in the manuscript, and include either an excerpt of the revised text or the location (eg: page and line number) where each change can be found. Please submit a clean version of the paper as the main article file; a version with changes marked should be uploaded as a marked up manuscript.

In addition, we request that you upload any figures associated with your paper as individual TIF or EPS files with 300dpi resolution at resubmission; please read our figure guidelines for more information on our requirements: http://journals.plos.org/plosmedicine/s/figures. While revising your submission, please upload your figure files to the PACE digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at PLOSMedicine@plos.org.

We expect to receive your revised manuscript by Jan 28 2021 11:59PM. Please email us (plosmedicine@plos.org) if you have any questions or concerns.

***Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.***

We ask every co-author listed on the manuscript to fill in a contributing author statement, making sure to declare all competing interests. If any of the co-authors have not filled in the statement, we will remind them to do so when the paper is revised. If all statements are not completed in a timely fashion this could hold up the re-review process. If new competing interests are declared later in the revision process, this may also hold up the submission. Should there be a problem getting one of your co-authors to fill in a statement we will be in contact. YOU MUST NOT ADD OR REMOVE AUTHORS UNLESS YOU HAVE ALERTED THE EDITOR HANDLING THE MANUSCRIPT TO THE CHANGE AND THEY SPECIFICALLY HAVE AGREED TO IT. You can see our competing interests policy here: http://journals.plos.org/plosmedicine/s/competing-interests.

Please use the following link to submit the revised manuscript:

https://www.editorialmanager.com/pmedicine/

Your article can be found in the "Submissions Needing Revision" folder.

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see http://journals.plos.org/plosmedicine/s/submission-guidelines#loc-methods.

Please ensure that the paper adheres to the PLOS Data Availability Policy (see http://journals.plos.org/plosmedicine/s/data-availability), which requires that all data underlying the study's findings be provided in a repository or as Supporting Information. For data residing with a third party, authors are required to provide instructions with contact information for obtaining the data. PLOS journals do not allow statements supported by "data not shown" or "unpublished results." For such statements, authors must provide supporting data or cite public sources that include it.

We look forward to receiving your revised manuscript.

Sincerely,

Caitlin Moyer, Ph.D.

Associate Editor

PLOS Medicine

plosmedicine.org

-----------------------------------------------------------

Requests from the editors:

1. If possible, please compare the definition for PCS against more conventional criteria in a subset of randomly selected patients (e.g. a few hundred) where the latter are available as an embedded validation of the approach, or at least to show how it performs in comparison to more conventional approaches.

2.Title: Please revise your title according to PLOS Medicine's style. Your title must be nondeclarative and not a question. It should begin with the main concept if possible. "Effect of" should be used only if causality can be inferred, i.e., for an RCT. Please place the study design ("A randomized controlled trial," "A retrospective study," "A modelling study," etc.) in the subtitle (ie, after a colon).

3.Data Availability statement: Unfortunately, the link (www.ices.on.ca/DAS) did not seem to work when tried. For the ICES dataset, please provide a specific DOI/accession number needed to identify the relevant dataset, and if possible provide a contact email address (note this cannot be a study author) to request access. For the dataset creation plan and analytic code, please either include these as supporting information files, or provide a link where they can be accessed (the contact point for these cannot be one of the authors).

4.Abstract: Methods and Findings: Please start the first sentence of this section: “Data from a cohort study (Ontario Concussion Cohort study, 2008-2016; n=1,330,336) including all adults with a concussion diagnosis by either primary care physician… were used in a retrospective analysis” or similar.

5.Abstract: Methods and Findings: Please include some background demographic information (e.g. population and setting, ages, etc).

6.Abstract: Methods and Findings: At line 43: “The total cohort was divided into derivation…” Please also note the derivation and validation cohorts were not randomly split but divided based on year of index event.

7.Abstract: Methods and Findings: For the risk estimates for PPCS in the derivation cohort, please include the actual numbers underlying the percentages. At Line 49-50, please also describe the variables included in the models for the validation and derivation cohorts.

8.Abstract: Methods and Findings: In the last sentence of the Abstract Methods and Findings section, please describe the main limitation(s) of the study's methodology.

9.Abstract: Conclusions: We suggest beginning the paragraph with “In this study, we observed that premorbid psychiatric conditions…” or similar.

10.Author Summary: Why was this study done?: Please revise the third point to: “The goal of the study was to create a risk calculation for every adult…”

11.Author Summary: What did the researchers do and find?: Please change 587.057 to 587,057

12.Author Summary: What do these findings mean?: We suggest combining the last two bullet points, and adjusting the language to avoid implying causality “Most adults with a concussion recover by 6 months following injury; however, our results suggest that pre-exisiting disorders are associated with increased likelihood of prolonged post-concussion symptoms.”

13. Methods: Ethics section: Please mention whether the requirement for informed consent was waived by the ethics board that approved the study.

14.Methods: Prospective analysis plan: Did your study have a prospective protocol or analysis plan? Please state this (either way) early in the Methods section.

a) If a prospective analysis plan (from your funding proposal, IRB or other ethics committee submission, study protocol, or other planning document written before analyzing the data) was used in designing the study, please include the relevant prospectively written document with your revised manuscript as a Supporting Information file to be published alongside your study, and cite it in the Methods section. A legend for this file should be included at the end of your manuscript.

b) If no such document exists, please make sure that the Methods section transparently describes when analyses were planned, and when/why any data-driven changes to analyses took place.

c) In either case, changes in the analysis-- including those made in response to peer review comments-- should be identified as such in the Methods section of the paper, with rationale.

15. Results: Please provide ORs, 95% CIs and p values for all relevant results presented in the main text. If adjusted analyses are presented, please also present the results from unadjusted models (these may be in a separate or supporting information table, and it should be noted in the text or figure legend when adjusted analyses are presented and which factors were included in the adjustment).

16. Discussion: Please revise the first sentence to begin: “To the best of our knowledge, this is the largest study to date…”

17. Discussion: Line 287: Please use square brackets for in text citations throughout.

18. Discussion: Line 324: Please adjust the reference within the bracket to [7,8].

19. Discussion: Line 349: Please clarify what is meant by the sentence: “This may be due to decreased neurocognitive reserve.”

20. Discussion: Line 374: Please clarify “effects of medico=legal issues”

21. References: Please use the "Vancouver" style for reference formatting, and see our website for other reference guidelines https://journals.plos.org/plosmedicine/s/submission-guidelines#loc-references

22. Table on Page 15: This appears to be the results of the Multivariate Logistic regression model in table format (from Figure 2). Please make this more clear (either incorporate into Figure 2 as a series of columns, or include a separate Table legend.

23. Table 1 on page 32: Please use consistent terms “Concussion Cohort” vs. Adult Cohort vs. Adult Concussion Cohort. Please define the abbreviation for GP, TBI, and TMD. Please indicate values that are N (%) vs. value (SD), for example.

24. As mentioned by reviewer 2, please do present the demographics for the derivation vs. validation cohort, either in the main table or a supporting information file.

24. Table 2: Please indicate “Primary Care” reflects numbers of visits.

25. Figure 2: Please provide an x axis label, and please provide a descriptive legend for the figure. Please define the abbreviation for TMJ.

26. Supplemental Figure 2: Please note in the legend that the stop step is indicated by the vertical line. Please define “SBC” and include a general X axis label for the figure.

27. Supporting information file TRICORDRR risk tool: Please spell out or define the abbreviation for OCD.

28. Supporting information file: Thank you for including the full text of your article “Increasing Incidence of Concussion: True Epidemic or Better Recognition?” however, it is not necessary to include as it is cited in the reference list.

29. Checklist: Please ensure that the study is reported according to the Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) or the most appropriate reporting guideline for your study, and include the completed TRIPOD checklist as Supporting Information. Please add the following statement, or similar, to the Methods: "This study is reported as per the Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) statement (S1 Checklist)."

The TRIPOD statement can be found here: https://www.equator-network.org/reporting-guidelines/tripod-statement/

When completing the checklist, please use section and paragraph numbers, rather than page numbers.

Comments from the reviewers:

Reviewer #1: This is a very interesting and important paper that seeks to employ a large existing diagnostic health data system to determine factors associated with prolong post concussive symptoms.

The finings that behavioral health is associated with prolonged symptoms is not necessarily novel yet this paper expands this fining in a large population based setting. The limitations of the study are related to confirmation of diagnosis and unrecorded information as well as not using an a priori definition for post concussive symptoms.

1. It would be helpful for the authors to consider the fact that those with behavioral health co morbidities could be expressed as a series of somatic complaints and concerns. Might there be a population of people that had many of the ill defined PPCS symptoms preinjury or ascribed everything many of the concerns to the concussion that had existed previously

2. The authors may wish to add additional co morbid that could be linked to health complaints- OSA and overall BMI for example

3. The recent work of the TRACK TBO group may be a jumping off point for further discussion ie Nelson et al and Stein et al. ie the role of behavioral health and lack of resolution of symptoms in a prospectively collected cohort. In addition, Seabury et al speaks to the lack of ED follow up in a large group of those with brain injury. This paper creates an important point to consider is that the work in this paper may help to consider a " at risk cohort" that deserves clinical follow up early after injury

3a.The authors may to also construct a low risk predictor given the above Seabury et al data

Reviewer #2: "Predicting Prolonged Post-Concussion Symptoms using 1 the TRICORDRR (Toronto Rehabilitation Institute Concussion Outcome Determination and Rehab Recommendations)" describes the development and validation of a risk score to predict risk of needing prolonged concussion care. A cohort from the Ontario Concussion Cohort Study (2008-2016) was used, with the derivation cohort being from 2009-2013, and the validation cohort being from 2008 & 2014. Certain risk factors such as advanced age, bipolar disordor, high level of primary care visits per year, etc were identified. An AUC of 0.64 was achieved for the model applied to the validation cohort, and AUC=0.79 for the derivation cohort with bootstrap resampling.

The results suggest moderate efficacy of the risk score model towards concussion care prediction. Additionally, some issues might be addressed:

1. In Lines 50 to 52, the sentence "The area under the curve (AUC) for the derivation model was 0.79, Validation model was 0.64, and the bootstrap was 0.79" seems slightly awkward. It appears that the same final model was validated in all cases; if so it might be phrased more similarly to "the AUC the final model was 0.79 on the derivation dataset... validation dataset...", etc. Moreover, from line 197, it seems that the derivation cohort and bootstrap are actually one and the same. This might be clarified.

2. In Line 182, the details of the univariate logistic regressions etc. involved in variable selection, as well as the details of the bootstrap resampling procedure, might be presented - possibly in supplementary material. This is especially if the derivation may not follow exactly from Sullivan et al. (2004).

3. For the CART tree presented in Figure 4, the predictions/outcomes at each node of the tree might also be presented (as referred to in Line 196, with reference to Sullivan et al., 2004), as appears to be standard practice.

4. In line 204, "duplicate records" might be defined in more detail. If it refers to individuals with multiple diagnoses of concussion (e.g. first concussion in 2008, then in 2013), was there a particular reason for exclusion instead of including the first occasion? If not, the treatment for such patients with multiple diagnoses might be described further, especially as they are then possibly represented in both the derivation and validation dataset.

5. The choice of having the validation cohort being comprised of both diagnoses before (2008) and after (2014) the derivation cohort, might be explained further. While this seems that it would mitigate confounders (e.g. treatment conditions) across the study period to an extent, there remains a potential concern of validation on data prior to that used for derivation. As such, it might be considered to report the outcomes for 2014 separately from 2008 as well.

6. The demographics odds ratio table on Page 15 does not appear to be captioned.

7. For the corresponding demographics tables (Tables 1a & 1b), it might be considered to also present the figures separated according to derivation and validation cohorts, to determine as to whether they share similar distributions.

8. The derivation of the risk point system (Line 260, Table 2) might be described in greater detail, possibly in the supplementary material. If possible, the performance difference between a (more complicated logistic) model estimate based on raw odds ratios, and the approximated risk point system, might be presented.

9. In Line 268, Figure 6 is described as "the number of cases that had PPCS..."; might this be the proportion of cases instead?

10. The numerical details of the percentages/number of patients involved for each risk score strata in Figure 6 might be included in the supplementary material.

Reviewer #3: This manuscript reports the results of the study focused on analysis for predicting risk of prolonged postconcussion symptoms (PPCS). Specifically, the study uses the Toronto Rehabilitation Institute Concussion Outcome Determination and Rehab Recommendations (TRICORDRR) in their prediction model. Overall, the studies well done and likely to make an important contribution to the literature. The following comments are intended to help the authors improve the overall quality and impact of their manuscript.

1. The definition of PPC as used in this study, based on two or more specialist visits for concussion related symptoms more than six months after injury, is novel and unorthodox. Please provide justification for this definition.

2. It would be helpful to understand how accurate the premorbid health retrospective review was for this study, relying on relevant ICD-9 diagnostic codes. Obviously, premorbid conditions emerged as important predictor variables in the model, so the reliability of the original data is critical. Please comment

3. The greatest concern about this study has to do with the methodology used to calculate the rate of PPCS. That is, the calculation of the rate of prolonged postconcussion symptoms as 12.5% appears to be based on the denominator of only those patients that had two years of healthcare system tracking, not the original cohort of patients with concussion (n=1,330,336). My main concern is that this methodology artificially inflates the reported rate of PPCS. That is, many of the original concussed cohort patients may have had a spontaneous and complete recovery, which negated their need for any follow-up in the healthcare system after their concussion. If so, then the current estimated rate of 12.5% excludes them in his face only on the percentage of patients who pursue follow-up after their injury and continue to have prolonged symptoms at least six months after injury. The persisting cohort of 73,122 patients with persistent symptoms represents only 5.4% of the original concussed cohort of 1,330,336 patients. The different methodologies represent more than a twofold difference, 5.4% versus 12.5%. This issue is of major importance, as it has important bearing on the reported rate of PPCS from this study, which is likely to be cited frequently after publication. Please provide verification and justification for the methodology used.

4. With respect to the results of your production model, please comment on how these findings from your very large study are novel or highly consistent with the existing literature. Specifically, comment on novel findings from this study after the existing literature on age and premorbid psychiatric factors predicting risk of prolonged postconcussion symptoms after mild traumatic brain injury.

5. The modeling statistics, results, and graphic illustration of the findings are of good quality.

Any attachments provided with reviews can be seen via the following link:

[LINK]

5 Feb 2021

Submitted filename: PLoS Med Response to Reviewers.docx

Click here for additional data file.

30 Mar 2021

Dear Dr. Langer,

Thank you very much for re-submitting your manuscript "The TRICORDRR (Toronto Rehabilitation Institute Concussion Outcome Determination and Rehab Recommendations): Targeting those at most risk for prolonged post-concussion treatment

Short Title: Risk Score for Prolonged Post-Concussion Symptoms" (PMEDICINE-D-20-05392R2) for review by PLOS Medicine.

I have discussed the paper with my colleagues and the academic editor and it was also seen again by two reviewers. Provided the remaining editorial and production issues are dealt with we are planning to accept the paper for publication in the journal.

The remaining issues that need to be addressed are listed at the end of this email. Any accompanying reviewer attachments can be seen via the link below. Please take these into account before resubmitting your manuscript:

[LINK]

***Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.***

In revising the manuscript for further consideration here, please ensure you address the specific points made by each reviewer and the editors. In your rebuttal letter you should indicate your response to the reviewers' and editors' comments and the changes you have made in the manuscript. Please submit a clean version of the paper as the main article file. A version with changes marked must also be uploaded as a marked up manuscript file.

Please also check the guidelines for revised papers at http://journals.plos.org/plosmedicine/s/revising-your-manuscript for any that apply to your paper. If you haven't already, we ask that you provide a short, non-technical Author Summary of your research to make findings accessible to a wide audience that includes both scientists and non-scientists. The Author Summary should immediately follow the Abstract in your revised manuscript. This text is subject to editorial change and should be distinct from the scientific abstract.

We expect to receive your revised manuscript within 1 week. Please email us (plosmedicine@plos.org) if you have any questions or concerns.

We ask every co-author listed on the manuscript to fill in a contributing author statement. If any of the co-authors have not filled in the statement, we will remind them to do so when the paper is revised. If all statements are not completed in a timely fashion this could hold up the re-review process. Should there be a problem getting one of your co-authors to fill in a statement we will be in contact. YOU MUST NOT ADD OR REMOVE AUTHORS UNLESS YOU HAVE ALERTED THE EDITOR HANDLING THE MANUSCRIPT TO THE CHANGE AND THEY SPECIFICALLY HAVE AGREED TO IT.

Please ensure that the paper adheres to the PLOS Data Availability Policy (see http://journals.plos.org/plosmedicine/s/data-availability), which requires that all data underlying the study's findings be provided in a repository or as Supporting Information. For data residing with a third party, authors are required to provide instructions with contact information for obtaining the data. PLOS journals do not allow statements supported by "data not shown" or "unpublished results." For such statements, authors must provide supporting data or cite public sources that include it.

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript.

Please note, when your manuscript is accepted, an uncorrected proof of your manuscript will be published online ahead of the final version, unless you've already opted out via the online submission form. If, for any reason, you do not want an earlier version of your manuscript published online or are unsure if you have already indicated as such, please let the journal staff know immediately at plosmedicine@plos.org.

If you have any questions in the meantime, please contact me or the journal staff on plosmedicine@plos.org.

We look forward to receiving the revised manuscript by .

Sincerely,

Caitlin Moyer, Ph.D.

Associate Editor

PLOS Medicine

plosmedicine.org

------------------------------------------------------------

Requests from Editors:

1. Please fully address the remaining reviewer comments.

2. Title: Please revise your title according to PLOS Medicine's style. Your title must be nondeclarative and not a question. It should begin with main concept if possible. "Effect of" should be used only if causality can be inferred, i.e., for an RCT. Please place the study design ("A randomized controlled trial," "A retrospective study," "A modelling study," etc.) in the subtitle (ie, after a colon). We suggest Prediction of risk of prolonged post-concussion symptoms: Derivation and validation of the TRICORDRR (Toronto Rehabilitation Institute Concussion Outcome Determination and Rehab Recommendations) score” or something similar.

3. Data availability statement: Thank you for clarifying this, and please also include the reference number in the statement, alongside the ICES contact information (reference TRIM#201609760310000) if relevant for data requests.

4. Prospective analysis plan: Thank you for including the analysis plan as a supporting information file. Please reference the file in the Methods at Line 201 (S1_Analysis Plan). From looking at the analysis plan, it does not seem to describe in detail the specific analyses outlined in the study. In the Methods section, please fully describe when specific analyses were planned, and when/why any data-driven changes to analyses took place, including changes from peer review comments (these are described very generally at Lines 202-204).

5. Abstract: Line 41: Should this be “retrospective analysis” rather than retroactive?

6. Abstract: Please double check grammar/sentence structure throughout- for example: “The total cohort was divided…” at line 44, and “Variables were selected a priori... “ at line 46.

7. Abstract: Methods and Findings: Please clarify the values given in parentheses at lines 50-52: “Highest risk estimates for PPCS derived in the Derivation Cohort were: >61 years (0.54), bipolar disorder (0.52), high level of primary care visits per year (0.46), personality disorders (0.45), and anxiety and depression (0.33).”

8. Abstract: Line 54: Please define the abbreviation OHIP at first use.

9. Abstract: Conclusions: Please revise to avoid causal language: “In this study, we observed that premorbid psychiatric conditions were associated with increased risk of a prolonged recovery from concussion. Increased health care system usage and older age also were significantly associated with increased risk.”

10. Author summary: Please use sentence structure/punctuation for bullet points.

11. Author summary: We suggest revising under “What do these findings mean” to read: “The risk score may aid physicians treating adults with a concussion tp quickly assess the patient’s absolute risk of a good outcome or a prolonged recovery and to be able to tailor the treatment plan as appropriate, such as encouraging return to aerobic exercise, education about concussion, timely referrals for specialized psychological care, etc.” if relevant, to reflect the conclusion ties back to the derivation of the risk score.

12. Author summary: We suggest revising the final bullet point to avoid causal implications: “...psychiatric disorders and health care utilization are associated with increased risk of likelihood…”

13. Methods: Line 128-129: Can you please clarify this sentence: “Visible minorities accounted for 29.3% of the population.”

14. Results: Line 263-264: Please provide the p value for this, in addition to the 95% CIs, and please revise for causal language: The location of diagnosis (ED or primary care) was not significantly associated with the number or type of specialist visits for persisting symptoms (OR 1.00, CI 0.996 – 1.006).

15. Results: Line 310: Please clarify to “...hypothetical patient aged 50-60 years…”

16. Results: Line 324: If it has not been previously spelled out in the text to this point, please fully define the abbreviation for TRICORDRR here at first use.

17. Discussion: We suggest opening with a discussion of the main findings of the study.

18. Discussion: Line 394: Please include a space between “aging” and the reference bracket. (Please also correct at line 402, and double check throughout).

19. Discussion: Line 419-420: Please revise to “...it has the highest number of visits with specialist physicians compared with other premorbid conditions…” Please also elaborate on the “decreased neurocognitive reserve” here.

20. Discussion: Line 434: Please capitalize Black in this sentence.

21. Discussion: Future Directions: Please clarify that the TRICORDRR is “the tool” indicated, and also it would be helpful to include a sentence putting into context what these ongoing studies hope to inform/accomplish.

22. References: Please double check all formatting. Please use the "Vancouver" style for reference formatting, and see our website for other reference guidelines https://journals.plos.org/plosmedicine/s/submission-guidelines#loc-references

For example, please check journal name and punctuation in 11, 12, 13 (throughout).

23. Supplemental Figure 1: Please define all abbreviations used in the legend. If possible, please provide a short description in the legend to help illustrate the main point.

24. TRIPOD checklist: Please include the locations in the text where each item can be found, rather than checking the points off the list. Please use section and paragraphs, rather than page numbers, to refer to locations. Please include a separate supporting information file Title/label for this (e.g. S1_Checklist).

25. Figure 2: In the X axis label, please change to “Odds Ratio”

26. Figure 3: Please provide a descriptive legend for the figure, making it slightly more clear what point is illustrated.

27. Figure 4: Please provide a descriptive legend for this figure.

28. Figure 5: Please provide a descriptive legend for the figure, making it slightly more clear what point is illustrated.

Comments from Reviewers:

Reviewer #2: We thank the authors for considering our previous comments. However, some concerns remain:

1. Given that the motivation for selecting 2008 and 2014 for validation was specifically that "...they were expected to be the best and worst years and did not want it to influence the derivation cohort", it could be appropriate to (also) present their analyses separately, possibly in supplementary material. This is since neither were expected to represent typical performance, and it is uncertain as to whether their combination into a single validation set would represent the typical cohort either. It might be noted that this choice could have contributed to the reduced validation performance of the model, which might be considered as a limitation.

Minor issues:

2. The sentence beginning in Line 52 might be "The area under the curve (AUC) was 0.79 for the derivation model, 0.79 for bootstrap internal validation of the Derivation Cohort, and 0.64 for the Validation model."

3. For the CART figure, the explanation pertaining to SAS graphical limitations are acknowledged, but it would remain recommended to extract the figures corresponding to the nodes if at all possible (e.g. by writing the values of the corresponding variables to the output log)

Reviewer #3: I appreciate the authors' response to reviewer comments and their respective manuscript revisions. These modifications have significantly improved the overall quality and impact of the manuscript.

Any attachments provided with reviews can be seen via the following link:

[LINK]

6 Apr 2021

Submitted filename: Reviewer Responses April 2021.docx

Click here for additional data file.

7 May 2021

Dear Dr Langer,

On behalf of my colleagues and the Academic Editor, David Menon, I am pleased to inform you that we have agreed to publish your manuscript "Prediction of risk of prolonged post-concussion symptoms: Derivation and validation of the TRICORDRR (Toronto Rehabilitation Institute Concussion Outcome Determination and Rehab Recommendations) score" (PMEDICINE-D-20-05392R3) in PLOS Medicine.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. Please be aware that it may take several days for you to receive this email; during this time no action is required by you. Once you have received these formatting requests, please note that your manuscript will not be scheduled for publication until you have made the required changes.

Also, please address the following editorial points:

-Author summary: Line 81: Please revise "concussed adults" to "adults with concussion" or similar.

-Methods: Line 131: Please avoid the use of the Statistics Canada quotation if possible. If necessary to use, please include a reference. We suggest avoiding the term "Caucasian" and instead use "...persons, other than Aboriginal peoples, who are non-White..." or similar.

-TRIPOD Checklist: Please revise the Checklist. Please do not use page or line numbers to refer to locations in the text. Please use sections and paragraph numbers to refer to locations in the text for each item (for example "Methods, paragraph 2").

-Reference list: Please check formatting of references 21, 22, 27, 30, 31, 35, 41 for correct journal abbreviations and punctuation.

In the meantime, please log into Editorial Manager at http://www.editorialmanager.com/pmedicine/, click the "Update My Information" link at the top of the page, and update your user information to ensure an efficient production process.

PRESS

We frequently collaborate with press offices. If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximise its impact. If the press office is planning to promote your findings, we would be grateful if they could coordinate with medicinepress@plos.org. If you have not yet opted out of the early version process, we ask that you notify us immediately of any press plans so that we may do so on your behalf.

We also ask that you take this opportunity to read our Embargo Policy regarding the discussion, promotion and media coverage of work that is yet to be published by PLOS. As your manuscript is not yet published, it is bound by the conditions of our Embargo Policy. Please be aware that this policy is in place both to ensure that any press coverage of your article is fully substantiated and to provide a direct link between such coverage and the published work. For full details of our Embargo Policy, please visit http://www.plos.org/about/media-inquiries/embargo-policy/.

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

Thank you again for submitting to PLOS Medicine. We look forward to publishing your paper.

Sincerely,

Caitlin Moyer, Ph.D.

Associate Editor

PLOS Medicine