Performance of college athletes on the 10-item word list of SCAT5.

OBJECTIVES: The Sport Concussion Assessment Tool-5 (SCAT5) was published in 2017; however, normative performance within the college athlete population on the optional 10-item word list has not been described. This study reports normative values for immediate memory trials, total immediate memory score and delayed recall of the 10-item word list.
METHODS: The SCAT5 was administered as part of the preparticipation medical testing to 514 collegiate student-athletes, aged 17-23 (M=19.65, SD=1.40; 64% male) prior to the 2017-2018 athletic season.
RESULTS: On the SCAT5's optional 10-item word list, with a total possible immediate memory score of 30, participants recalled an average of 20.57 (SD=3.22) words over three learning trials, with an average for trial 3 of 8.13 (SD=1.32). The average delayed memory score was 6.59 (SD=1.85). Small but significant demographic comparisons were found. Women scored higher on both immediate and delayed recall, non-native speakers scored higher on delayed recall, and Black/African-American athletes scored lower than White athletes on immediate, and lower than White and Hispanic/Latino athletes on delayed recall.
CONCLUSION: The 10-item word list on the SCAT5 eliminates the ceiling effect observed on the five-item word list of the SCAT3, therefore, increasing its clinical utility in the diagnosis of sports-related concussions. Significant demographic differences suggest use of category-specific norms for sex, native language and race/ethnicity.

The average total immediate memory score on the Sport Concussion Assessment Tool-5 (SCAT5) was 20.68 (SD=3.21) words out of 30 over three learning trials.

The average delayed memory score on the SCAT5 was 6.62 (SD=1.87) from the 10-item word list.

By implementing the optional 10-item word list of SCAT5, the problematic ceiling effects of the SCAT3 five-item word list were eliminated.

Introduction

The Sport Concussion Assessment Tool-5 (SCAT5) was a product of the recent consensus meeting of the Concussion in Sport Group in Berlin, and represents the most recent revision of the 12-year-old SCAT.1 2 The SCAT has proven to be an effective tool in assisting with diagnosing sports-related concussions, although its role in tracking postconcussion recovery and return-to-play decision-making remains unclear.3 The various versions of the SCAT have found use in baseline assessment, as well as a stand-alone test following head injury.3

The primary neurocognitive portion of the SCAT has been the 30 items of the Standardized Assessment of Concussion (SAC),4 which includes measures of orientation, concentration, immediate memory and delayed recall. Deficits in learning and memory have been implicated historically as among the most sensitive components of diagnostic tests for sports-related concussions within 24–48 hours after the injury.5 6 The primary measure of learning and memory on the SAC is a five-item word list over three learning trials. McCrea et al 7 reported overall SAC scores for recently concussed athletes to be significantly lower than the mean score for athletes who underwent baseline testing. Specifically, regarding immediate memory, athletes tested directly following a sports-related concussion demonstrated significantly lower immediate memory subtest scores compared with non-injured athletes during baseline testing. Delayed recall scores remained consistent with the trend observed for immediate memory on the SAC, as non-injured athletes demonstrated significantly higher delayed recall abilities when compared with athletes who recently sustained a concussion.7 McCrea et al 4 7 and Echemendia and Julian8 have shown that baseline performance on the SAC word list typically results in medians of 14 correct out of 15 on total immediate recall and four words correct out of five on delayed recall.

As Echemendia et al 3 noted, a limitation of previous versions of the SCAT (ie, prior to the SCAT5) is the ceiling effect observed in the immediate and delayed recall trials. Assessments with a significant ceiling effect may obscure actual ability in either of two ways. First, noting that an individual scores at the top of a test conveys an interpretation that the individual’s ability is very high. Second, scoring at the ceiling prevents persons of greater ability the opportunity to demonstrate that ability. In the current instance with the SCAT, someone of high ability may score at the ceiling both on baseline and post-trauma, and neither of those scores is a true representation. As described by Echemendia et al,3 the SAC ceiling effect likely has limited the clinical utility of previous versions of the SCAT, as it is not sensitive to subtle verbal learning and memory changes following a concussion. The problematic nature of the ceiling effect is evidenced by the SAC immediate memory subtest mean of 14.51 out of 15, which suggests that perfect scores are common within this population.8 To further evidence the ceiling effect of the five-item word list, Shehata et al 9 reported that all athletes in their study (n=249) were able to recall at least four out of five words during the immediate recall trial (the original SCAT had only one learning trial), with the majority of participants able to recall all five words. During the delayed five-word recall, nearly 37% of athletes were able to recall all five words.9 In order to avoid such a ceiling effect in assessing this critical domain, the SCAT5 authors introduced an optional 10-item word list. The current study is the first to present normative performance of college athletes on the 10-item word list of the SCAT5.

Method

During the concussion education session at the preseason collection of baseline information, all potential participants consented to allow their deidentified information to be used for research purposes.

Participants

Five hundred and twenty-three collegiate athletes participated. Athletes older than the age of 23 were removed from the analysis to produce a more traditional college sample, resulting in a final sample size of 514 participants (327 men; 187 women). The ages of the athletes ranged from 17 to 23 (M=19.65, SD=1.40), with a mean education level of 13.56 (SD=1.23). The median number of previous concussions was zero; 101 reported previous concussions. The athletes represented 13 contact and non-contact sports comprising 23 teams, which included baseball, cheerleading, cross-country, football, golf, lacrosse, rowing, soccer, softball, swimming, tennis, track and field and volleyball. The athletes represented 49 countries of origin and 29 languages other than English were noted as first language.

Summary data from the SCAT3, used for comparison with the SCAT5, were collected in preseason baseline testing prior to the start of the 2016–2017 athletic season. That sample included 573 participants (360 men; 213 women) ranging in age from 17 to 23 (M=19.70, SD=1.49), with a mean education level of 13.57 (SD=1.34). The median number of previous concussions was zero, with 137 reporting prior concussions. The athletes represented the same sports and teams as mentioned above.

Materials

The SCAT310 and the SCAT5 are standardised instruments for evaluating suspected concussion and can be used in athletes aged 13 years and older. The revisions from the SCAT3 to the SCAT5 included an optional 10-item word list for the learning and memory component, compared with the original five-item word list. In each test version, the word list is presented in three consecutive trials, and delayed recall is measured following a 5 min delay.1

Procedure

All athletes were administered a baseline neurocognitive battery as part of their preparticipation medical testing before the start of the 2017–2018 (SCAT5) and 2016–2017 (SCAT3) playing seasons. Prior to the evaluation, athletes first attended a concussion education session that lasted between 30 and 40 min and consented to the research. All athletes were examined individually by trained clinical psychology doctoral students in small testing rooms. Delayed recall for both the SCAT3 and SCAT5 was assessed after a minimum of a 5 min delay, during which the concentration, neurological screen and modified Balance Error Scoring System portions of the SCAT were administered.

Statistical analyses

The data were analysed using the IBM Statistical Package for the Social Sciences (SPSS) V.24. Frequency and descriptive statistics were used for comparison of the mean totals for the SCAT3 and SCAT5 immediate and delayed memory recall, as well as independent sample t-tests and an analysis of variance (ANOVA) for demographic comparisons of sex, non-native versus native English speakers, race/ethnicity and US versus international athletes.

Results

Average SCAT5 scores for immediate trials, total immediate and delayed recall are shown in table 1. Similar to other list learning tasks, the number of words recalled increased in each of the three successive trials, with an average of 8 recalled in the ultimate trials. Figure 1 displays the frequency of words recalled across all three immediate memory trials, with means increasing as participants learnt across trials. Despite the quantitative limitations on normality, the scores display peaks with tails in both directions, and certainly approximate a normal curve. The average total immediate memory score was 20.57 out of a possible 30 words, while the average delayed memory score was 6.59 out of 10 words. A Shapiro-Wilk test showed that none of the recall scores were normally distributed; however, the value of skewness fell within the range of −0.5 to 0.5, indicating a symmetric distribution, with the exception of trial 3 memory, with a skewness of −0.59.

Table 1

Descriptive statistics of SCAT5 (10 words) scores

	n	Range	M	SD	Skewness		Shapiro-Wilk
Statistic					SE	Statistic	Pvalues
Trial 1 immediate	514	1–9	5.33	1.23	0.21	0.108	0.938	<0.001
Trial 2 immediate	514	2–10	7.11	1.40	−0.27	0.108	0.951	<0.001
Trial 3 immediate	514	3–10	8.13	1.32	−0.59	0.108	0.919	<0.001
Immediate memory total	514	9–29	20.57	3.22	−0.27	0.108	0.987	<0.001
Delayed memory	514	1–10	6.59	1.85	−0.22	0.108	0.965	<0.001

SCAT5, Sport Concussion Assessment Tool-5.

Figure 1

Frequency by serial position of words recalled across the three immediate memory trials.

Figure 2A,B shows graphically the distribution of score frequencies for immediate and delayed recall on the SCAT5, respectively. Despite the quantitative limitations on normality, the scores do show peaks with tails in both directions, and certainly approximate a normal curve. This contrasts markedly with distributions of immediate and delayed recall scores on the SCAT3, with its five-item list (figure 2). The ceiling effect is readily apparent with SCAT3 but not with SCAT5. Speaking directly to a ceiling effect, no participants earned a perfect 30 on total immediate recall of the SCAT5 words, and only 25 individuals (5%) earned a perfect 10 on the delayed recall. That was contrasted with 344 (60%) earning a perfect 15 of 15 immediate recall on the SCAT3, and 227 (40%) remembering all five words in the delayed recall.

Figure 2

Scorefrequencies for immediate and delayed recall on the SCAT5 contrasted with normative performance on the SCAT3. SCAT, Sport Concussion Assessment Tool.

Table 2 shows SCAT5 performance isolating the following demographic comparisons: sex, non-native versus native English speakers, race/ethnicity and US versus international origin. Independent sample t-tests were conducted to compare the dichotomous factors, with an ANOVA used to compare race/ethnicity. With respect to sex, there was a significant difference in the scores for both immediate and delayed memory with women performing significantly higher than men. Regarding non-native versus native English speakers, there was a significant difference in the scores for delayed memory, with non-native English speakers performing higher than English speakers. A two-factor ANOVA with sex and first language as factors did not produce a significant interaction.

Table 2

SCAT5 normative scores on 10-Item word list for demographic categories

	n	Immediate memory			Delayed memory
M		SD	Test statistic	M	SD	Test statistic
Sex
Male	327	20.15	3.23	t=−3.94,	6.36	1.88	t=−3.74,
Female	187	21.29	3.08	p<0.001	6.98	1.72	p<0.001
Race/ethnicity
White	333	20.73*	3.32	F=2.91,	6.70†	1.84	F=5.37,
Black/African American	80	16.61*	2.91	p<0.05	5.83†‡	1.87	p=0.001
Hispanic/Latino	45	20.80	3.04		6.87‡	1.69
Multiracial	41	20.93	2.81		6.61	1.88
First language
Native English	420	20.44	3.26	t=1.84,	6.49	1.87	t=2.60,
Non-Native English	94	21.12	2.95	p=0.066	7.03	1.66	p=0.009

SCAT5, Sport Concussion Assessment Tool-5.

*P=0.32.

†P=0.001.

‡P=0.014.

Both immediate and delayed recall differed significantly as a function of race/ethnicity (also table 2; the latest race and ethnic category definitions from Office of Management and Budget were used).11 Bonferroni pairwise comparisons determined that White athletes scored significantly higher than Black/African-American athletes in both immediate and delayed recall. Hispanic/Latino athletes scored significantly higher than Black/African-American athletes only on delayed recall. It is important to note that athletes falling within the Native Hawaiian/Other Pacific Islander, Asian, and American Indian/Alaska Native categories were not included from this analysis due to very small group sizes.

Discussion

For the measurement of immediate and delayed memory, the recently released SCAT5 includes the option to use a 10-item word list instead of a five-item word list as used in previous versions of the SCAT. As discussed by the authors of the SCAT5, the rationale for the 10-item list was to minimise ceiling effects.1 The present data clearly show those authors accomplished their objective. Performance of a large group of college athletes on the new 10-item list approximated a normal distribution, with means of 5.33 words learnt for trial 1 immediate, 7.11 for trial 2 and 8.13 for trial 3. As expected, participants recalled more words as the trials progressed. Participants also achieved a mean of 20.57 correct for the total score on immediate recall and 6.59 words correct on delayed recall. No participant recalled all 10 words on the three learning trials for immediate recall, and only 25 (4.9%) recalled all 10 words in the delayed recall.

The performance observed in the demographic comparisons is noteworthy with respect to the actual quantitative differences, and whether they might affect interpretation when a post-trauma test was compared with norms rather to a baseline. For example, in comparing performance by sex, women recalled an entire point more than men in the total immediate recall, and 0.6 of a point in delayed recall. This outcome was not surprising since previous research has shown that women generally perform better on list learning tasks in comparison to men.12–14 As expected, comparison of the male or female average scores with the whole sample average scores produced less dramatic differences, particularly in delayed recall. It is unlikely that the magnitude of these sex differences is sufficiently large to cause an error in determining that a concussion has occurred.

The most prominent finding regarding racial/ethnic differences in performance was that the Black/African-American athletes scored significantly lower than White athletes in total immediate recall, and significantly lower than Whites and Hispanics in delayed recall. The differences here appear to be of a magnitude that could contribute to inaccurate assessment if compared with the group values. That is, a difference of 0.96 in immediate recall and 0.76 in delayed recall would cause a loss of two points independent of any actual trauma effects. Historic appraisals of post-trauma changes from baseline SAC scores concluded that decreases of one to three points were meaningful for diagnosis of concussion.15 There is less likelihood that comparisons to normative data on any one part of a test would result in such a decision, but it is important for clinicians to be aware of these demographic differences.

The data regarding performance of athletes whose first language was not English were somewhat surprising. Although the non-native English speakers did not differ from native English speakers in immediate recall, they recalled significantly more in the delayed recall condition than did the native English speakers. Previous work would have suggested either no difference in free recall on such a serial learning test,16 or an advantage in the native language performance by bilingual speakers.17 One possible explanation for this outcome was that the percentage of women among the non-native speakers was higher than it was in the native English-speaking cohort (43% vs 35%), and women scored higher than men overall. However, no interaction was found between the factors of sex and first language. A second possibility was time studying in English. Specifically, only 18.9% of the non-native English speakers were new students; the others had been enrolled for 1–3 years. Previous research has shown that greater experience with and use of the second language does result in bilingual persons’ performance mirroring native speakers’ performance on semantic information recall tasks.18 The actual differences observed, though significant, were less than one point. That difference is unimportant clinically. However, it should serve to alert clinicians who interpret SCAT5 post-trauma tests that low delayed recall scores of non-native English speakers should not be discounted as due to language status versus actual injury effects.

Limitations to this study primarily related to demographic comparisons where group size was small, such as the racial/ethnic comparisons where only White, Black/African American, and Hispanic groups were large enough for valid comparison. Similarly, other comparisons were not made because of very small group size, specifically attention deficit hyperactivity disorder (n=35) and learning disability (n=9) groups. Results from previous studies would suggest that those premorbid conditions would be associated with lower recall scores.19 Given our results, more definitive conclusions about better performance by bilinguals whose first language is not English are needed. Furthermore, the clinical significance of these findings remains to be determined in larger cohorts, in both injured and non-injured athletes.

Following the standardised instructions for administering the SCAT5 resulted in small variations in the actual delay time intervening between immediate recall trials and the delayed recall. Future studies of the SCAT5 might examine directly the extent to which delay time impacts recall.