Development and Validation of a Short Version of the Anterior Cruciate Ligament Return to Sport After Injury (ACL-RSI) Scale.

BACKGROUND: The Anterior Cruciate Ligament Return to Sport After Injury (ACL-RSI) scale was developed to measure an athlete's psychological readiness to return to sport after anterior cruciate ligament (ACL) injury and reconstruction surgery. The scale is being used with increasing frequency in both research and clinical settings.
PURPOSE: To generate and validate a short version of the ACL-RSI scale. STUDY
DESIGN: Cohort study (diagnosis); Level of evidence, 2.
METHODS: The ACL-RSI scale was administered to 535 patients who had undergone ACL reconstruction surgery. Reliability (Cronbach alpha) was determined and factor analysis of the full scale was undertaken along with a process of item selection and elimination. A second group of 250 patients participated in a predictive validation analysis. This group completed the ACL-RSI scale at 6 months and reported return-to-sport outcomes 12 months following ACL reconstruction surgery. The predictive validity of both scales (full and short versions) was assessed by use of receiver operating characteristic (ROC) curve statistics.
RESULTS: The scale was found to have high internal consistency (Cronbach alpha, 0.96), which suggested that item redundancy was present. After an item selection process, the scale was reduced to a 6-item format. Cronbach alpha for the short version was 0.92, and factor analysis confirmed the presence of 1 factor accounting for 71% of the total variance. Scores for the short version were significantly different between patients who had and those who had not returned to sport. Six-month ACL-RSI scores for both the full and short versions had fair to good predictive ability for 12-month return-to-sport outcomes (full version: area under ROC curve, 0.77 [95% CI, 0.7-0.8]; short version: area under ROC curve, 0.75 [95% CI, 0.7-0.8]).
CONCLUSION: A 6-item short version of the ACL-RSI scale was developed from a large cohort of patients undergoing ACL reconstruction. The short version appears to be as robust as the full version for discriminating between and predicting return-to-sport outcomes. The short version of the ACL-RSI may be of use in busy clinical settings to help identify athletes who may find return to sport challenging.

Most athletes who undergo anterior cruciate ligament (ACL) reconstruction surgery aim to return to their preinjury sport. Although surgery addresses impairments in knee function, many athletes do not return to sport after surgery, and psychological responses have been identified as a potentially modifiable factor associated with this outome.[3,6,8,24] Indeed, over the past decade it has become increasingly clear that return to sport following ACL reconstruction is associated with a significant psychological response; consequently, increased efforts have been made to better understand the psychological recovery from this serious injury.[1]

One of the most commonly discussed psychological factors in terms of returning to sport is “psychological readiness.” Although multifactorial, psychological readiness is largely influenced by an athlete’s emotions and confidence.[1] Confidence is thought to be derived from 2 elements: confidence in the injured body part and confidence in the ability to perform well.[13,21] Developing confidence in both these aspects may provide a buffer from injury-related fear and anxiety, placing the athlete in a “psychologically ready” state to resume sport participation. In line with this, a recent systematic review concluded that an athlete’s psychological readiness to return to play is a combination of confidence in performing well and remaining uninjured as well as the lack of fear and anxiety.[9]

Although several measures are available to assess psychological readiness to return to sport, the Anterior Cruciate Ligament Return to Sport After Injury (ACL-RSI) scale[25] was developed and published in 2008 as a tool to specifically measure psychological readiness to return to sport after ACL injury and reconstruction surgery. The ACL-RSI is a unidimensional 12-item scale that measures 3 types of responses believed to be associated with the resumption of sport following athletic injury: emotions (5 items), confidence in performance (5 items), and risk appraisal (2 items). The scale is being used with increasing frequency and has been translated from English to Swedish,[14] French,[4] German,[19] Dutch,[23] Turkish,[11] Portuguese,[22] and Chinese,[5] with other translations currently underway. In a busy clinical environment, a short version of the ACL-RSI may be desirable. As such, the objective of this study was to reduce the length of the ACL-RSI and determine the divergent and predictive validity of the short version compared with the original and longer form. It was hypothesized that a short version could be developed that was as robust as the full version for discriminating between and predicting return-to-sport outcomes.

Methods

Participants

A total of 535 patients (347 male, 188 female; mean ± SD age, 26.8 ± 9 years) who had undergone primary ACL reconstruction surgery with a hamstring tendon autograft participated in the scale reduction component of the study. A second group of 250 patients (157 male, 93 female; mean age, 27.0 ± 9 years) participated in the predictive validation component of the study. For both groups, patients were eligible if this was their first ACL injury and they had participated in sport on a weekly basis prior to the injury. Patients were excluded if they had undergone further surgery within the first 12 months following the reconstruction or had another medical reason for not being able to participate in their preinjury sport (eg, pregnancy).

The rehabilitation protocol encouraged immediate full knee extension and the restoration of quadriceps function as soon as possible. Weightbearing was allowed as tolerated from the first postoperative day. Braces were not used. Patients were allowed to run starting at 12 weeks, provided they had no effusion and were deemed to have adequate quadriceps strength and a satisfactory range of flexion. Timing of return to sport varied considerably, frequently based on patient preference. However, return to noncontact training was not allowed before 6 months and required the patient to have good control of a single-leg squat and normal running and landing patterns. Return to play was not allowed before 9 months and required the patient to have participated in unrestricted full contact training for at least 4 weeks.

Procedures and Analysis

Institutional ethics approval was obtained, and patients were recruited from a private orthopaedics clinic when they attended for a routine review.

Item Eligibility Assessment

Before initiating the scale reduction process, we asked an additional 30 patients who had undergone ACL reconstruction surgery within the previous 12 months to rate each item on the ACL-RSI on a scale from 1 to 3 with respect to the item’s importance (1 = unimportant, 2 = somewhat important, 3 = very important). These patients were statistically similar to the cohorts used for item reduction and validation. The rating data were used to calculate mean relevance scores for each item, and items were considered to be eligible for inclusion in the reduced ACL-RSI scale if they had a mean relevance score of at least 2.0, with at least two-thirds (67%) of patients rating the item as at least somewhat important, as per previous published methods.[16,17]

Item Reduction and Validation Process

Participants in the scale reduction sample completed the 12-item ACL-RSI at 12 months after surgery and answered a question that specifically asked whether they had returned to their preinjury level of sport. The response options were no, training only, yes but at a lower level, and yes at the same or higher level.

From the participant responses, reliability (Cronbach alpha) was determined and factor analysis of the full 12-item scale was undertaken along with a process of item selection and elimination. Item selection was carried out by first grouping items into their 3 domains: emotions, confidence in performance, and risk appraisal. Within each group, items were chosen to remain on the scale if their relevance score was high (items not meeting the relevance threshold of 2.0 were removed), their means were close to the center of the possible range (ie, 50), and they had a large standard deviation. As described by DeVellis,[7] items with means too close to the extreme of the response range will have low variances, while those that vary over a narrow range will correlate poorly with other items. Therefore, items with means closer to the center of the possible range and high variance are highly desirable.[7] An interitem correlation matrix was also used to compare the relationship between items. Any items making the scale redundant were considered for deletion.

Reliability and factor analysis was then carried out for the reduced ACL-RSI scale. Using 1-way analysis of variance (ANOVA) with post hoc testing, we assessed divergent validity of both the full scale and the proposed reduced scale by comparing the scores of patients who had returned to sport and those who had not.

Predictive Validation Process

Participants in the predictive validation sample completed the 12-item ACL-RSI scale at 6 months after surgery and then answered the same return-to-sport question as the scale reduction sample at 12 months. The reduced version of the ACL-RSI was subsequently extracted from the full version, and the predictive validity of both scales (full and short versions) was assessed by use of receiver operating characteristic (ROC) curve statistics and the Youden index, using the 6-month ACL-RSI score and 12-month return-to-sport status of the patients in the predictive validation group. Two predictive analyses were conducted. First, the predictive validity for returning to the same or higher preinjury level of sport was determined. For this analysis, return to sport was dichotomized as patients who had returned to their preinjury level of sport or a higher level versus those who had not (no sport, training only, or lower level). Second, the predictive validity for not returning to sport was determined. For this analysis, return to sport was dichotomized as patients who had not returned to any form of sport compared with those who had (training only, lower than preinjury level, or the same as preinjury level or higher).

Scoring and Statistical Analysis

Each item of the ACL-RSI is graded on a scale of 1 to 100 given in 10-point increments. Higher scores reflect a more positive psychological response.

All statistical analyses described above were performed with SPSS software version 22 (SPSS Inc), and statistical significance for group comparisons was set at P < .05.

Results

Scale Reduction and Validation

The ACL-RSI was found to have high internal consistency (Cronbach alpha = 0.96), which suggested that item redundancy was present. Factor analysis also confirmed the presence of 1 underlying factor accounting for 68% of the total variance in ACL-RSI scores.

The mean (SD) and range of scores for each ACL-RSI item are shown in Table 1, along with item relevance scores. Of the 5 items examining the emotions associated with return to sport, both items 4 and 5 refer to fear of reinjury. Although initially it was assumed that including the wording “accidentally injuring your knee” may have elicited a different response from some patients when completing the scale, the mean and standard deviation for these items established that patients responded very similarly to each item. Thus, we decided that item 4 would be retained in the reduced scale, since it had a higher relevance score as well as a mean score slightly closer to 50 and a slightly larger standard deviation. The means of item 1 (nervous) and item 2 (frustrated) were closest to 50, and both were considered to be important emotions in the return-to-sport context and were therefore retained, whereas item 3 (relaxed) was omitted.

TABLE 1

Scores for Individual Items on the Anterior Cruciate Ligament Return to Sport After Injury (ACL-RSI) Scale (N = 535)

Scale Item	Mean (SD)	Range	Relevance Scoreb
Emotions
1. Are you nervous about playing your sport?	58.07 (31.4)	0-100	2.2
2. Do you find it frustrating to have to consider your knee with respect to your sport?	48.26 (35.7)	0-100	2.2
3. Do you feel relaxed about playing your sport?	67.29 (28.6)	0-100	2.2
4. Are you fearful of reinjuring your knee by playing your sport?	53.78 (31.5)	0-100	2.4
5. Are you afraid of accidentally injuring your knee by playing your sport?	56.09 (29.8)	0-100	2.1
Confidence in performance
6. Are you confident that your knee will not give way by playing your sport?	68.93 (28.2)	0-100	2.4
7. Are you confident that you could play your sport without concern for your knee?	63.63 (30.5)	0-100	2.7
8. Are your confident about your knee holding up under pressure?	71.18 (25.6)	0-100	2.6
9. Are you confident that you can perform at your previous level of sport participation?	71.96 (29.5)	0-100	2.9
10. Are you confident about your ability to perform well at your sport?	72.47 (27.1)	0-100	2.7
Risk appraisal
11. Do you think you are likely to reinjure your knee by participating in your sport?	63.91 (27.8)	0-100	2.2
12. Do thoughts of having to go through surgery and rehabilitation again prevent you from playing your sport?	70.11 (31.9)	0-100	1.9

Items retained in the reduced version of the scale are shown in bold.

Patients (n = 30) were asked to provide a rating with respect to the item's importance (1 = unimportant, 2 = somewhat important, 3 = very important), and mean relevance scores were calculated from their responses.

The items measuring confidence in performance had been previously categorized to include both confidence in knee function and confidence in performing well at one’s sport. Items 6, 7, and 8 are related to confidence in knee function while playing sport. Results for these 3 items were reviewed and revealed quite similar means and standard deviations. Item 7 was chosen to represent this construct because it had the highest relevance score and a mean closer to 50. It also had high between-item correlations with item 6 (r = 0.85) and item 8 (r = 0.77). Items 9 and 10 are related to the confidence athletes have in their ability to perform well at their sport. Means and standard deviations were also quite similar for these 2 items. Item 9 was retained because it had a higher relevance score and a mean closer to 50. It also specifically mentions returning to preinjury level of participation, which has been established as an important outcome of the rehabilitation phase following ACL reconstruction.

Of the 2 items in the risk appraisal category, item 12 had relevance score of less than 2.0, and only 63% of participants rated this item as at least “somewhat important”; it was therefore removed, and item 11 was retained.

The ACL-RSI was therefore reduced to create a 6-item scale (Appendix). Scores for the 6 items were summed and averaged to provide a single score for the scale ranging from 0 to 100, with higher scores reflecting greater psychological readiness. Cronbach alpha for the 6-item scale was 0.92. Although this value was still high, we believed that the 6 items in the reduced scale were important areas that were needed to represent the 3 domains. Factor analysis of the short version confirmed the presence of 1 factor accounting for 70.7% of the total variance in ACL-RSI scores. The short version was also highly correlated with the full version (r = 0.98).

To demonstrate divergent validity, scores for both the full and short versions were significantly higher for patients who had returned to their preinjury level of activity compared with those who had not (P < .0001) (Table 2). Conversely, patients who had not returned to any sport scored significantly lower than those who had (P < .0001). No difference was found between patients who had returned to training and those who had returned to a lower level of their preinjury sport (P = .6).

TABLE 2

Comparative Scores Between the Full Anterior Cruciate Ligament Return to Sport After Injury (ACL-RSI) Scale and a Short Version According to Sport Status

	ACL-RSI, Mean (SD)	ACL-RSI (Short Version), Mean (SD)
No return (n = 239)	51.7 (25)a,b,c	47.9 (26)a,b,c
Return to training (n = 129)	69.6 (19)a,d,e	65.6 (21)a,d,e
Return to lower level (n = 59)	68.1 (20)b,d,f	63.7 (24)b,d,f
Return to same preinjury level (n = 108)	81.4 (15)c,e,f	77.8 (18)c,e,f

< .0001, no return vs return to training.

< .0001, no return vs return to lower level.

< .0001, no return vs return to same preinjury level.

< .6, return to training vs return to lower level.

< .0001, return to training vs return to same preinjury level.

< .0001, return to lower level vs return to same preinjury level.

Predictive Ability for Returning to Preinjury Level of Sport or Higher

Of the 250 patients in the predictive validation component, 72 (29%) returned to their preinjury level of sport at 12 months. Six-month ACL-RSI scores for both the full and the short versions had fair to good predictive ability for a return to preinjury level at 12 months (full version: area under ROC curve = 0.77; 95% CI, 0.7-0.8; short version: area under ROC curve = 0.75; 95% CI, 0.7-0.8) (Figure 1). For the full version, a Youden index of 0.39 was observed at a score of 62 points, corresponding to a sensitivity of 57% and specificity of 81%. For the short version, a Youden index of 0.34 was observed at a score of 60 points, corresponding to a sensitivity of 50% and specificity of 84%.

Figure 1.

Receiver operating characteristic (ROC) curve for (A) full 12-item Anterior Cruciate Ligament Return to Sport After Injury (ACL-RSI) scale and (B) reduced 6-item ACL-RSI scale for predicting return to preinjury level of sport or higher.

Predictive Ability for Not Returning to Sport

Of the 250 patients, 79 (32%) had not attempted any sport at 12 months. Six-month ACL-RSI scores for both the full and short versions had fair to good predictive ability for predicting nonreturners at 12 months (full version: area under ROC curve = 0.75; 95% CI, 0.7-0.8; short version: area under ROC curve = 0.72; 95% CI, 0.63-0.8) (Figure 2). For the full version, a Youden index of 0.35 was observed at a score of 42 points, corresponding to a sensitivity of 57% and specificity of 78%. For the short version, a Youden index of 0.34 was observed at a score of 39 points, corresponding to a sensitivity of 60% and specificity of 74%.

Figure 2.

Receiver operating characteristic (ROC) curve for (A) full 12-item Anterior Cruciate Ligament Return to Sport After Injury (ACL-RSI) scale and (B) reduced 6-item ACL-RSI scale for predicting not returning to sport.

Discussion

In the current study, we successfully shortened the ACL-RSI scale to a brief 6-item version. The process of item selection involved eliminating items in each of the 3 domains of emotions (2 items removed), confidence in performance (3 items removed), and risk appraisal (1 item removed). The short version had high internal consistency, and although the scale length was reduced by half, reliability was not sacrificed. The short version was also found to discriminate between athletes who had and had not returned to sport in a similar way to the full version. Overall, the short version was found to be a robust substitute for the full version of the ACL-RSI scale.

Although it could be argued that the full scale is not overly long, item redundancy was clearly present, suggesting that a short version could be developed. The primary advantage of a short version is that it reduces patient burden, which may be a significant consideration given that patients often are asked to complete a large number of self-report outcome measures. Indeed, researchers and clinicians must decide how many and which patient-reported outcomes they can reasonably use. Although it is important to include a comprehensive set of outcomes, the time taken to complete the final questionnaire also needs to be considered. As psychological variables have become recognized as playing a role in the recovery from ACL injury, their measurement has been the topic of discussion. For example, the ACL-RSI was recently included in a test battery to enhance the safe return to sport after ACL reconstruction.[10] It is hoped that the short version of the ACL-RSI scale will enable clinicians to measure this aspect of recovery with minimal additional burden on the patient.

The ability of the ACL-RSI scale to predict patients who would return to their preinjury level of sport or higher by 12 months was fair to good and was similar between both the full and short versions. A cutoff score of 62 points was determined for the full version and 60 points for the short version, both having high specificity and moderate sensitivity. Work is ongoing to determine what the ACL-RSI scores mean clinically and what cutoff might be useful for categorizing athletes as psychologically ready to return to sport. A cutoff score of 56 points was initially reported[2] based on ACL-RSI scores at 4 months and return-to-sport status at 12 months after surgery. The current study used ACL-RSI scores at 6 months. As scores on this scale have been shown to increase with time, the corresponding cutoff values were also higher.[15] A prospective study reported a cutoff score of 51 points, with moderate specificity (0.63) and high sensitivity (0.97), based on 6-month measures in 40 athletes.[18] The current study extends previous work by calculating a cutoff score for patients who do not return to sport. This analysis showed scores of 42 for the full version and 39 for the short version of the ACL-RSI scale, both having high specificity and moderate sensitivity.

Given the current data, it is reasonable to suggest that athletes who score above the 60-point cutoff for the short version at 6 months after surgery are highly likely to return to sport, whereas athletes who score below 39 for the short version are likely to not return by 12 months. Implications for those who score between these values are less clear. This is not unexpected, as there will always be athletes who have low scores on psychological readiness and yet still return to sport for a variety of reasons, and vice versa. It is also reasonable to hypothesize that some level of anxiety and fear is a normal response when contemplating a return to sport. In fact, investigators have suggested that some degree of caution may be protective, if it means that an athlete does not prematurely resume sport without full consideration of his or her knee function.[1] The clinical utility of such cutoff scores is that they provide clinicians with a means to identify which athlete groups may struggle to return to sport from a psychological perspective. This is important, given the typically limited resources available for psychological assistance during recovery from injury.

A relevant feature of the ACL-RSI (short version) is that some of the items that are more knee specific (such as symptoms of giving way or the knee's ability to hold up under pressure) have been removed, making the scale less knee specific and potentially more adaptable to other injuries or diseases.

A strength of the ACL-RSI scale is that it was developed from a theoretical perspective, and the domains used in the scale have received support from qualitative data.[20] Although the scale was designed around 3 domains, these domains are highly related, and factor analysis confirmed the presence of a single underlying factor, which has been termed “psychological readiness.” Given this, it may be possible to have a single score to rate psychological readiness, similar to the currently used Single Assessment Numerical Evaluation (SANE) rating.[26,27] A recent systematic review concluded that an athlete’s psychological readiness to return to play is a combination of confidence in performing well and remaining uninjured in addition to the lack of fear and anxiety.[9] The items included in both the full and the short versions of the ACL-RSI scale clearly address these psychological aspects. It has been frequently stated that psychological factors are potentially modifiable and should therefore be considered during rehabilitation in order to improve return-to-sport rates and the return-to-sport experience for athletes.[2] It is hoped that the ACL-RSI (short version) will help in the achievement of these aims.

The large sample sizes used in both the scale reduction and the predictive validation parts of this study are a strength. However, the patients were a relatively homogenous group. Only patients who had undergone primary ACL reconstruction were included, and thus we did not address the issue of injury experience. In previous studies, first-time injured athletes have been shown to be less confident and to find rehabilitation more stressful than those having been injured multiple times.[12] Although the extent of any psychological differences between athletes with a single versus multiple ACL injuries is unclear, both the full and short versions of the ACL-RSI have not been specifically validated for patients with multiple ACL injuries. As well, our sample included more male than female patients, which is representative of our clinical population but may not be for others.

In conclusion, a short version of the ACL-RSI scale was developed that has psychometric properties equal to those of the full version. Although half the length, the short version is as robust as the full version for discriminating between and predicting return-to-sport outcomes and therefore can be used in place of the full version in the setting of primary ACL reconstruction. The ACL-RSI (short version) can be added to patient-reported outcome tools with minimal burden. It can be used to identify athletes who will find return to sport difficult and to better understand the psychological sequelae of ACL injury.

1. Are you confident that you can perform at your previous level of sport participation?

Not at all confident	0	10	20	30	40	50	60	70	80	90	100	Fully confident
	□	□	□	□	□	□	□	□	□	□	□

2. Do you think you are likely to reinjure your knee by participating in your sport?

Extremely likely	0	10	20	30	40	50	60	70	80	90	100	Not likely at all
	□	□	□	□	□	□	□	□	□	□	□

3. Are you nervous about playing your sport?

Extremely nervous	0	10	20	30	40	50	60	70	80	90	100	Not nervous at all
	□	□	□	□	□	□	□	□	□	□	□

4. Are you confident that you could play your sport without concern for your knee?

Not at all confident	0	10	20	30	40	50	60	70	80	90	100	Fully confident
	□	□	□	□	□	□	□	□	□	□	□

5. Do you find it frustrating to have to consider your knee with respect to your sport?

Extremely frustrating	0	10	20	30	40	50	60	70	80	90	100	Not at all frustrating
	□	□	□	□	□	□	□	□	□	□	□

6. Are you fearful of reinjuring your knee by playing your sport?

Extremely fearful	0	10	20	30	40	50	60	70	80	90	100	No fear at all
	□	□	□	□	□	□	□	□	□	□	□