Brain Injury Screening Tool (BIST): test-retest reliability in a community adult sample.

OBJECTIVE: To determine the test-retest reliability of the Brain Injury Screening Tool (BIST), which was designed to support the initial assessment of mild traumatic brain injury (mTBI) across a variety of contexts, including primary and secondary care.
DESIGN: Test-retest design over a 2-week period.
SETTING: Community based. PARTICIPANTS: Sixty-eight adults (aged 18-58 years) who had not experienced an mTBI within the last 5 years and completed the BIST on two different occasions. MEASURES: Participants were invited to complete the 15-item BIST symptom scale and the Depression, Anxiety and Stress Scale (DASS-21) online at two time-points (baseline and 2 weeks later). To account for large variations in mood affecting symptom reporting, change scores on the subscales of the DASS-21 were calculated, and outliers were removed from the analysis.
RESULTS: The BIST total symptom score and subscale scores (physical-emotional, cognitive and vestibular) demonstrated moderate to good test-retest reliability with intraclass correlation coefficients ranging between 0.51 and 0.83. There were no meaningful differences between symptom reporting on the total scale or subscales of the BIST between time1 and time2 at the p<0.05 level when calculated using related samples Wilcoxon signed-rank tests.
CONCLUSION: The BIST showed evidence of good stability of symptom reporting within a non-injured, community adult sample. This increases confidence that changes observed in symptom reporting in an injured sample are related to actual symptom change rather than measurement error and supports the use of the symptom scale to monitor recovery over time. Further research is needed to explore reliability of the BIST within those aged <16 years. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The Brain Injury Screening Tool (BIST) demonstrated moderate to good test–retest reliability for the total and subscale scores.

The sample had high ecological validity showing reliability within a highly diverse community sample including those with high anxiety/depression/stress, comorbidities and low health ratings.

The study did not include those aged <16 years.

It remains unclear if mode of administration of the BIST affects symptom reporting.

Introduction

Mild traumatic brain injuries (mTBIs, including concussion) can occur when there is an impact to the body or head causing the brain to move within the skull, which leads to cell damage, neural pathway dysfunction and altered brain function. This alteration in brain function is experienced by the person as feeling dazed and confused, not remembering what happened or in some cases losing consciousness.1

Considerable attention has been placed on the assessment of mTBI within the sports context.2 The Sports Concussion Assessment Tool (SCAT-5 with child and adult versions) was specifically designed for the assessment of sports related mTBIs by a trained medical professional. However, only one in five mTBIs are due to sports or recreational activities, and the applicability of the SCAT-5 outside of sport is currently under question. Initial feasibility studies have revealed that the SCAT-5 needs to be substantially shortened for use in busy clinical environments such as the emergency department or general practice.3–5 The language used in the adult version of the SCAT-5 has been found to be too complex,4 with many respondents interpreting the items differently than what was intended.6 There is evidence that the symptom scale lacks unidimensionality, with the inclusion of several overlapping or redundant items, and not supporting use of a total score indicative of overall symptom burden.

While many people recover well from mTBIs, others can experience chronic symptoms and functioning deficits without treatment.7 8 In the general practice and emergency department contexts, the key consideration is therefore to identify who can be safely discharged and who may need early referral to specialist services. Consequently, there is a need for a tool with strong psychometric properties to support identification of mTBI, facilitate decision making, as well as to monitor recovery over time.9

The Brain Injury Screening Tool (BIST; https://tbin.aut.ac.nz/support-and-resources/brain-injury-screening-tool-bist)10 was designed to be a screening tool to support identification of injury and to assess symptom experience and impact of injury over time. The tool was designed to be brief and easy to administer without the need for specific training on how to use it. A key advantage of the BIST is that the tool identifies the presence or absence of clinical risk indicators for prolonged recovery, such as previous TBI and mental health history.11 12 This is followed by a 15-item symptom scale including symptoms and symptom clusters known to be important in the identification and diagnosis of mTBI and indicators of recovery.10 Responses on the BIST can provide guidance on the level and type of rehabilitation services that may be required, ranging from educational advice and follow-up from a general practitioner (GP, to referral to multidisciplinary specialist team input (eg, concussion clinic). Both static (pdf) and automated versions of the BIST are available. The automated version has been integrated into patient management systems in New Zealand to facilitate clinical use.

Previous research in people who have experienced an mTBI has reported that the BIST fits well with the RASCH model13 and that the factor structure matched the proposed symptom subscale clusters and support use of total symptom score.10 The BIST symptom scale has shown evidence of excellent internal consistency and high concurrent validity with existing symptom measures such as the SCAT-5.10 However, the stability of symptom reporting on the BIST symptom scale needs to be demonstrated. This is necessary in order to increase confidence that any changes observed in symptoms within an mTBI population are likely to reflect changes in the actual symptom experience rather than measurement error. Reliability of symptom reporting is difficult to demonstrate with an mTBI population as recovery can occur for many months to years after an mTBI.14 A good test–retest study aims to create a sampling frame where symptom variation is likely to be as low as possible. As the symptoms in the symptom scale are non-specific to TBI (eg, headaches and fatigue), this lends itself to exploring symptom reporting in a non-TBI injured population where symptom presentation would be expected to be more stable.15 16

However, previous evidence of symptom reporting has revealed a link between how a person is feeling and their symptom reporting. For example, in non-clinical populations, symptom reporting has also been shown to be higher in those with higher depression and anxiety.17 18 It is normal for mood to vary over the course of the day as well over different days. However, in some unique instances, mood can vary considerably for many different reasons including hormonal fluctuations or dysfunction (eg, premenstrual dysphoric disorder, perimenopause and menopause), life events, illness, weather patterns and tiredness. Consequently, participants with high variations in mood could affect symptom reporting. The aim of test–retest reliability studies is to explore whether a given measure is reliable in a sample where it is believed that the construct being measured does not change over time. Therefore, some factors need to be controlled to prevent low correlations from being found due to external variation rather than precision of the measure. Because a symptom scale needs to be able to demonstrate utility in a general population, maintaining ecological validity was deemed to be important including those with high anxiety and depression. However, extreme variations in mood may be problematic. This study was undertaken to determine the test–retest stability of the BIST symptom scale in a non-TBI injured adult community sample controlling for the influence of atypical mood variation.

Methods

Cohort description

A cross-sectional study of healthy adults (>16 years) was undertaken with participants who did not have a TBI in the past 5 years. A priori sample size calculation using GPower (V.3.1.9.7; https://stats.idre.ucla.edu/other/gpower/) revealed that to detect a correlation of 0.5 (cut-off for moderate test–retest reliability) between the two timepoints, with 90% power at the p=0.05 level, a minimum sample of 28 participants would be needed. Our participant recruitment from May to August 2021 was conducted via social media, adverts placed in community settings (eg, cafes, sports centres and libraries) and via word of mouth.

Participants were included if they were over the age of 16 years at the time of the first assessment, had not experienced a mild TBI in the past 5 years and were able to provide informed consent. People who reported comorbidities were included in the study due to high prevalence of comorbidities in the general population (eg, asthma). Those who reported experiencing a moderate or severe TBI in their lifetime were excluded from the analysis. Adults interested in taking part were asked to contact the research team directly via phone, text or email. Following initial contact, the research was discussed, including what would be involved, and eligibility was assessed. If the person was interested, and eligible, they were sent a written information sheet and consent form for them to sign and return via post or email. On receipt of the signed consent form, participants were sent a weblink to the online survey (time1) through the REDcap online database. A second survey link was sent 2 weeks later following completion of the first survey. This was done as the recommended interval for test–retest studies.19 Automatic reminders were set up within the system to prompt the participant to fill in the survey every 3 days after the 2-week interval, until completion. Following completion of both surveys, participants were sent a thank you letter and a $30 fuel voucher in recognition of their time and contribution. Participants’ contact details were stored separately within the REDcap system to protect participant privacy.

Patient and public involvement

Patients were involved in the initial concept of developing the screening tool. No patients were included in the specific design, recruitment procedures or conduct of this study. A summary of the study findings was sent to all participants on completion of data analysis.

Sociodemographic measures

At the first assessment, data regarding the person’s age, gender, ethnicity, education and physical and mental health comorbidities were collected to enable description of the sample and assess influence of these sociodemographic on symptom reporting.

Brain Injury Screening Tool

The first component of the BIST10 aims to identify those at high risk of poor recovery or medical complications from an mTBI and may require referral to hospital, physiotherapy or specialist rehabilitation services. As this study was being completed in a non-clinical sample, this component was not included. The second component of the BIST includes a 15-item symptom self-report scale. For this study, participants were asked to rate how much they experience the symptoms listed on a scale of 0 (not at all) to 10 (severe) at the time of completing the assessment, as opposed to the instruction for the clinical sample which states ‘Compared with before the accident, please rate how much you experience the following right now (at this point in time)’. A total symptom score and three subscale (physical-emotional, cognitive and vestibular) scores can be calculated based on participant responses. Higher scores on the symptoms are indicative of higher symptom burden.

To assess and control for mood variation, a 21-item scale, the Depression, Anxiety and Stress Scale (DASS-21),20 was used at both timepoints. The DASS-21 consists of three subscales (each with seven items) assessing levels of depression, anxiety and stress. Change scores on each of the three subscales were calculated between the two timepoints to identify any outliers (>1.5× IQR) on at least one of the three DASS-21 subscales (deemed cases with atypically high mood variation). This cut-off excluded those with a mood change score of ±12 points on one subscale or more.

Statistical analysis

Data were extracted from REDcap into IBM SPSS V.25. Frequencies and percentages were used to describe participant characteristics on categorical variables such as gender and education. Distribution of the data was determined to be non-normal based on skewness and kurtosis >3.29.21 The median and IQR values were used to describe ordinal level data, due to the non-normal distribution of these variables. Intraclass correlation coefficients (ICCs) were used to determine test–retest reliability for the total and cluster scores of the BIST with 95% CIs. ICCs were interpretated as <0.50 as poor, between 0.51 and 0.75 as moderate, between 0.76 and 0.90 as good and >0.91 as excellent.22 Wilcoxon signed ranks tests were used to determine if there were any meaningful differences in the measurement between time1 and time2. A p value of 0.05 was used to determine statistical significance, and effect sizes are also presented. Individual change scores for each symptom item were calculated by subtracting the time2 response from time1 response to show individual variation in symptom reporting.

Results

Of the 82 adults who consented to take part in the study, 78 (94.1%) completed both questionnaires at baseline and 2 weeks later. Ten participants (12.1%) were excluded from the analysis due to reporting atypically high variation in mood symptoms. Consequently, data for 68 participants who completed the BIST at the time1 and time2 were analysed to explore the test–retest reliability of the BIST. The sample ranged in age from 18 to 58 years of age, with a median age of 27.5 years (IQR=15.0) (see table 1). There was high variability in health ratings on the 0–100 health rating scale (which ranged between 30 and 100) with a median health rating of 78.5 (IQR=17.0). Comorbidities included attention deficit hyperactivity disorder, arthritis and asthma.

Table 1

Participant characteristics for the 68 non-TBI injured adults

	n (%)
Sex
Male	16 (23.5)
Female	52 (76.5)
Ethnicity
European	25 (36.8)
Māori/Pasifika	8 (11.8)
Asian	12 (17.6)
Not specified	23 (33.8)
Employment
Full-time or part-time employed	39 (57.3)
Student	25 (36.8)
Homemaker	2 (2.9)
Other	2 (2.9)
Highest level of education
Secondary school	6 (8.8)
College/professional training	10 (14.7)
University	52 (76.5)
Living situation
Living alone	9 (13.2)
Live with others	58 (85.3)
Other	1 (1.5)
Comorbidity
Yes	9 (13.2)
No	59 (86.8)

TBI, traumatic brain injury.

The median and IQR for time1 and time2 on the BIST total score and subscale scores are shown in table 2. There were no significant differences between the scores at the two timepoints for the total and three subscales. The effect size estimates for the BIST total and subscale scores ranged between 0.02 and 0.11. ICCs for the BIST total score, physical-emotional, cognitive and vestibular subscales were all classified as ‘moderate to good’ ranging between (0.51 and 0.83).

Table 2

Test–retest reliability of the BIST for non-TBI injured community adult sample

	Time 1Median (IQR)	Time 2Median (IQR)	Wilcoxon signed-rank test	P value(sig.)	Effect size (D)	ICC	95% CI
							Lower	Upper
BIST total	23.00 (27.00)	26.00 (26.00)	1164.5	0.87	0.02	0.79	0.66	0.87
BIST physical-emotional	18.00 (25.25)	19.00 (23.50)	941.5	0.39	0.05	0.83	0.73	0.90
BIST cognitive	3.50 (12.00)	6.00 (8.75)	1000.0	0.05	0.11	0.72	0.55	0.83
BIST vestibular	1.00 (5.00)	2.50 (5.00)	614.0	0.42	0.04	0.51	0.20	0.70

BIST, Brain Injury Screening Tool; ICC, intraclass correlation coefficient; TBI, traumatic brain injury.

Individual change scores for each symptom over the two timepoints are shown in online supplemental table 1. It is evident that while in most cases scores only changed slightly, there were some instances of high variation in symptom reporting. The most commonly rated symptoms were ‘feeling tired during the day’ (88.2%), ‘I get angry or irritated more easily’ (79.4%), ‘feeling restless’ and ‘needing to sleep more’ (both 76.5%). The least common symptoms reported were ‘I feel dizzy’ (36.7%) and ‘when I close my eyes, I feel like I am at sea’ (26.5%).

Correlations between sociodemographic variables, DASS-21 subscale scores and BIST total scores are demonstrated in table 3. Younger age, being of female gender and overall health rating and mood ratings (depression, anxiety and stress) were all significantly correlated with the BIST symptom total score at the p<0.01 level. Other sociodemographic factors such as ethnicity, education, living situation and comorbidities were not.

Table 3

Correlations between sociodemographic variables and BIST total symptom score at time1

	Age	Sex	Ethnicity	Health rating	Education	Living with others	Comorbidities	DASS-21 anxiety	DASS-21 stress	DASS-21 depression
Sex	0.37**
Ethnicity	0.03	0.05
Health rating	0.45**	0.32**	0.08
Education	0.21	0.16	0.29*	0.17
Living with others	−0.03	0.03	−0.15	−0.06	−0.05
Comorbidities	−0.12	−0.22	−0.11	−0.42**	0.01	0.01
DASS-21 Anxiety	−0.47**	−0.30*	−0.10	−0.54**	−0.23	0.05	0.31**
DASS-21 Stress	−0.42**	−0.30*	−0.03	−0.47**	−0.24	−0.05	0.19	0.64**
DASS-21 Depression	−0.30**	−0.34**	−0.11	−0.58**	−0.11	0.04	0.40**	0.73**	0.74**
BIST TotalSymptom Score	−0.43**	−0.32**	−0.16	−0.52**	−0.19	0.07	0.23	0.66**	0.67**	0.69**

*p<0.05, **p<0.01

BIST, Brain Injury Screening Tool; DASS-21, 21-item Depression, Anxiety and Stress Scale.

Discussion

This study determined the test–retest reliability of the BIST within a community non-TBI injured adult sample over a 2-week interval. ICCs and tests of difference for symptoms reflected moderate to good test–retest reliability for both the overall and symptom cluster subscale scores. Younger age, female gender, lower health rating and higher anxiety, depression and stress were all significantly correlated with higher symptom scores in this population.

One of the challenges in measuring test–retest reliability is testing the symptom experience within a sample where the construct is stable, but one that has relevance to the population within which the tool will be used. For the purposes of this study, we chose to explore test–retest reliability in a non-TBI injured sample. This is due to evidence indicating that symptoms can continue to fluctuate for a long time after injury as people return to everyday activities, take on new challenges or experience changing life demands. The recovery process can also continue for many years.14 Consequently, a general population sample was deemed to be the most suitable for determining test–retest reliability of the BIST. We acknowledge that future studies could explore symptom stability in a ‘recovered’ sample of mTBI participants; however, this is problematic until there is a sound definition of what ‘recovered’ means and how it can be reliably assessed. In most cases, symptom reporting was relatively stable in this sample of non-TBI injured participants; however, there were some cases where there was significant variation in symptom reporting particularly in relation to cognitive symptoms between baseline and follow-up. These changes were not statistically significant which was also supported by small effect sizes (Cohen’s d <0.2) for total and subscale score between two timepoints. While the reason behind this variation is unclear, it is suspected that this may be related to environmental, social or biological influences participants may have been subject to between the two timepoints. Participants were asked for details of other medical comorbidities, but they were not asked if they had experienced any negative life events, illnesses or other factors potentially influencing symptoms during the 2-week period, which could have identified the potential influences.

Comorbid conditions are common in the general population and therefore participants with comorbidities were included in the sample. Indeed, ratings of health status varied from quite low to very high (30/100). The inclusion of people with poor physical health and comorbidities may have affected symptom reporting. For example, even within this non-TBI injured population, some symptoms were reported as being experienced within the severe range. This finding does support previous research that symptoms of concussion are also frequently reported in non-TBI injured community samples and are not specific to TBI.15 Fatigue and sleep have previously been found to be common problems in the general population with up to 28.6% of the population reporting sleeping problems and 35% reporting fatigue.15 While this may suggest these symptoms may not be specific in identifying concussion, poor sleep whether directly related to the mTBI or not, has been found to be a significant predictor of longer term persistent symptoms.23 Consequently, it is important to assess sleep quality in the acute period post-mTBI as a risk factor for poorer longer term recovery.24 Inclusion of people with diagnosed comorbidities did not appear to have had a meaningful impact on the test–retest reliability of the BIST in itself, as the presence of comorbidities were not notably associated with the BIST symptom score. However, lower health status was related to higher symptoms suggesting that the impact of the comorbidity(ies) may indeed have affected symptom reporting.

Depression, anxiety and stress were all considerably correlated with the total BIST symptom score. It was felt that it was important to include those reporting high scores for anxiety and depression, as approximately 20% of the New Zealand general population report symptoms of anxiety or depression.15 We tried to minimise the influence of mood on symptom reporting by excluding outliers revealing high variation in mood across the two timepoints yet retaining those with more stable extreme mood in the sample for ecological validity. Test–retest reliability may be less robust among those known or suspected to have high variations in mood.

It was evident that symptom reporting was more stable on the vestibular items (excluding problems with eyesight where some high variation was evident). These were also the items where there were lower levels of reporting of these symptoms in the general population. However, test–retest reliability was only moderate for this subscale, which may reflect the more notable variation in symptom reporting on the item ‘I have difficulty with my vision (eyesight)’.

Due to the lack of evidence of a reliable, specific and sensitive biomarker or radiological imaging for concussion,25 identification and diagnosis of concussion is reliant on subjective symptom reporting and description of the mechanism of injury. While in many cases symptom reporting was relatively stable in this non-TBI injured population, there were some cases where symptom reporting by individuals varied considerably between the two timepoints and in some cases by ±10 points. The reasons behind the large variation remain unclear as the study questionnaire did not ask if there had been any major life events, a recent illness, hormonal changes or to clarify if the symptoms were linked to a comorbid condition that may have caused a fluctuation in symptom reporting. Further understanding of the experience and influences on symptom reporting in the general population is needed. Additional studies are also needed to determine stability of the factor structure over time.

Limitations

This study had several limitations. First, there was the risk of self-selection bias. There were a high proportion of students and those with higher levels of education in this sample. Our sample also had higher number of female participants. While females have identified to be at higher risk for concussion and have higher symptom severity,26 further studies with more representative samples are warranted. In this study, participants completed the BIST symptom scale online; however, in clinical practice, the BIST may be completed by being delivered by a GP or nurse. Studies in sports concussion have shown mode of administration can affect symptom reporting, with lower symptoms reported in an interview format. However, it was found that the gender of the interviewer also affected symptom reporting.27 It remains unclear if mode of administration of the BIST symptom scale affects symptom reporting or not. Due to the need for participants to be able to provide their own informed consent to complete the online questionnaire, those aged <16 years were not included. However, the BIST was designed for use for those aged 8 years and upwards. Further research is needed to determine the test–retest reliability for those aged 8–16 years. Currently, the BIST is only available in the English language, and further work is needed to explore translation and clinical utility of the tool in other languages. A 2-week test–retest design was used based on the recommended follow-up of mTBI patients 10 days after injury and the recommended timeframe for test–retest reliability analysis. However, it is acknowledged that in some clinical environments, a follow-up assessment may not occur until 1 month after injury. Further research could consider exploring test–retest reliability over longer timeframes.

Conclusion

The BIST showed evidence of good stability of symptom reporting within a non-injured community adult sample. This increases confidence that changes observed in symptom reporting in an injured sample are more likely related to actual symptom change rather than measurement error. Further research is needed to explore reliability of the BIST within those aged <16 years.