I spy with my little eye … a knee about to go 'pop'? Can coaches and sports medicine professionals predict who is at greater risk of ACL rupture?

BACKGROUND: The vertical drop jump (VDJ) test is widely used for clinical assessment of ACL injury risk, but it is not clear whether such assessments are valid. AIM: To examine if sports medicine professionals and coaches are able to identify players at risk of sustaining an ACL injury by visually assessing player performance during a VDJ test.
METHODS: 102 video clips of elite female handball and football players performing a baseline VDJ test were randomly extracted from a 738-person prospective cohort study that tracked ACL injuries. Of the sample, 20 of 102 went on to suffer an ACL injury. These 102 videos were uploaded to an online survey. Sports medicine professionals and coaches were invited to assess athlete performance and rate each clip with a number between 1 and 10 (1 representing low risk of sustaining an ACL injury and 10 representing high risk). Receiver operating characteristic analyses were used to assess classification accuracy and between-group differences were analysed using one-way analysis of variance.
RESULTS: 237 assessors completed the survey. Area under the curve values ranged from 0.36 to 0.60, with a mean score of 0.47, which is similar to random guessing. There were no significant differences in classification accuracy between groups (physicians, coaches, certified athletic trainers, researchers or physical therapists).
CONCLUSION: Assessors have poor predictive ability (no better than chance), indicating that visual assessment of a VDJ test is a poor test for assessing ACL injury risk in elite female handball and football players. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Introduction

Observational assessment of movement quality is believed to represent a critical component in effective ACL injury prevention programme,1 2 and the vertical drop jump (VDJ) test is widely used clinically to assess ACL injury risk. Since simple observation requires no tools, is fast and inexpensive, it can be useful for large-scale application. However, it is not clear whether such assessments are valid. Previous studies have shown that the assessment of movement patterns by experienced personnel during a drop jump corresponds well with that of sophisticated three-dimensional-analyses.3–9 Nonetheless, no studies have investigated whether simple observation can be used to prospectively identify elite athletes at increased risk for ACL injury.

Previous analyses of injury situations10–13 show that distinct movement patterns are associated with ACL injury, such as greater internal hip rotation and knee valgus. It is assumed that similar movement patterns, hypothesised to predispose the athlete for increased risk of injury, may be revealed in jump landings,14 in particular frontal plane knee motion.3 6 15 16 Therefore, the VDJ has been used extensively as a screening test for assessing ACL injury risk in the clinic and in scientific studies.17–19

In an early study using three-dimensional motion analysis techniques, Hewett et al 17 reported that increased knee abduction moment and ground reaction force (GRF) were associated with increased injury risk in young, female adolescent athletes. Later, Leppänen et al 19 reported that GRF, but not frontal plane variables, was associated with injury in a similarly aged cohort, while Krosshaug et al 18 found that neither GRF nor knee abduction moment was associated with ACL injury in their cohort of elite female athletes. However, although three-dimensional motion analysis is considered a gold standard, it is well known that substantial errors may occur. These errors may originate from, for example, identification of bony landmarks and skin artefacts. Thus, simple visual assessment could provide different results. In three-dimensional analysis, the data are typically reduced from continuous movement patterns to peak values, that may or may not be critical for the injury event, and thus vital information may be lost. Moreover, from machine learning, we know that even advanced algorithms, based on extensive sensory input, such as for example, three-dimensional analysis systems, may not evaluate situations as well as humans.20 It is therefore possible that the human eye and brain can detect and assess critical movement characteristics associated with increased risk for injury better than those based on discrete data points.

The purpose of this study was to examine whether sports medicine professionals and coaches could visually identify female elite football and handball players with increased risk for ACL injury based on their performance on a VDJ test, and to investigate which cues that are used in the assessment. Furthermore, we examined whether professional groups within sport and sports medicine (eg, physiotherapists, coaches) differed in their ability to assess ACL injury.

Methods

Study design

Sports medicine professionals (including students) and coaches all over the world were invited through email and social media platforms to rate the risk of a sample of Norwegian female handball and football premier league players. Video clips of 102 players, including 20 players who sustained an ACL injury during the study period (subsequent to the video recording) were made available online, allowing anyone with access to the link to participate anonymously. The face of the athletes was blurred to protect their anonymity. The current study did not involve patients.

The videos and injury information had been collected previously, through the Oslo Sports Trauma Research Center ACL risk factor study, where all teams in the Norwegian female handball and football premier league were invited to a preseason baseline screening including videotaping of three valid VDJs.18 21 22 The VDJ tests were recorded from a frontal plane view only.

Data material

We used video material from 2009 to 2013 as the video quality was uniform (the picture quality was poorer in the previous years). Seven hundred and thirty-eight female players were tested between 2009 and 2013 (474 football players, 264 handball players). Seventy-three of these players were excluded due to previous ACL injury. Among the remaining 665 players, we recorded 31 new injuries during the study period. From these, we excluded one direct contact injury. This left us with 633 uninjured and 30 injured players available for randomisation.

We decided in advance to limit the number of video clips to approximately 100, to prevent drop-out by assessors. The ACL injury prevalence in this cohort is approximately 10%,23 hence, 10 injury cases and 90 non-injury cases would represent the natural ratio between injured and non-injured. However, we decided to include a random sample of 20 injury cases in order to ensure sufficient power for the signal detection analysis. Thus, the number of uninjured players was reduced from 90 to 82, resulting in a total sample of 102 players. To ensure validity, a sensitivity analysis was conducted to explore how manipulating the injury ratio influenced statistical outcomes.

The 20 players (n=16 football, n=4 handball) who sustained an ACL injury during the follow-up period were 20±3 years, weighed 66±16 kg and measured 170±38 cm. The 82 uninjured players (age 21±4 years, weight 63±14 kg, height 169±37 cm; 51 football; 31 handball) were randomly selected from the sample of non-injured players.

Players signed a written consent form before inclusion and the Regional Committee for Medical Research Ethics, the South Eastern Norway Regional Health Authority and the Norwegian Social Science Data Services approved the study. All complete ACL injuries among the tested players were recorded throughout March 2014. Moreover, all injuries were verified by MRI and/or arthroscopy.

Recruitment of assessors

Recruitment of assessors took place in the period between 22 March and 27 April 2017. Invitations were sent through email with a direct link to the online survey. Email addresses were collected through email lists from different academies, conferences, universities, workplaces, courses and so on. The survey link was also distributed on the social media platforms Facebook and Twitter. Colleagues worldwide helped recruit assessors. We aimed to invite various groups within sport and sports medicine, including coaches, strength and conditioning coaches, athletic trainers, physicians, physiotherapists and researchers both nationally and internationally. Participants identified their profession in the questionnaire. No incentives for participation were given.

Online survey

The videos were embedded within Qualtrics (2017 Qualtrics LCC) in March 2017. Instructions on how to conduct the survey, followed by one test clip, were given initially. The assessors subsequently watched each clip following a 3-2-1 countdown. One clip lasted for two to three seconds (excluding the countdown), and the assessors could watch each clip as many times as needed. Assessors were asked to rate the clip with a number between 1 and 10, where 1 represented very low risk and 10 represented very high risk (figure 1). The assessors were not given any specific cues in advance. After completing all video assessments, they were asked to report what they based their assessment on, in a free text field before rating the importance of different predefined cues with a number between 1 and 10. Moreover, they were asked to rate their confidence level for the risk rating assessment with a number between 1 and 10. Finally, we recorded gender, age, region, current occupation, education level, work experience level, if they were working with athletes and within which sport and whether or not they had assessed the VDJ test before.

Figure 1

Screen shot of a risk rating assessment in the survey.

Statistical analysis

Receiver operating characteristic (ROC) curves and subsequent areas under the curve (AUC) with 95% CIs were calculated to describe the diagnostic ability of each assessor to identify players with increased ACL injury risk. We report the AUC as the mean with 95% CI for each of the occupational groups. For missing data in the risk rating assessment, the mean value for the clip was inserted. The level of accuracy was classified as outstanding (AUC 0.9–1), excellent (AUC 0.8–0.9), acceptable (AUC 0.7–0.8), poor (AUC 0.6–0.7) or no discrimination (AUC 0.5–0.6).24 Differences between groups were analysed using one-way analysis of variance with Bonferroni post hoc corrections.

In addition to ROC analyses, we used independent sample t-tests to compare aggregate ratings for the injured/non-injured athletes. This ‘wisdom of crowds’ type of analysis combined the ratings of multiple assessors to determine whether the collective may be more accurate than individual assessment. Finally, and due to the large sample size, latent variable modelling techniques were used to assess inter-rater reliability.25 The significance level was set to p<0.05. Effect sizes are presented by Cohen’s d. When participants completed fewer than 95% of the cases we did not process their survey (approximately 1500 responses). Where assessors reported being both a physical therapist and a researcher (n=2), they were counted as being a physical therapist for our analysis.

Results

Assessor characteristics

The survey was open from 22 March to 1 May. Over 1700 people opened the survey using the distributed link, but the majority did not complete it. Two hundred and thirty-seven assessors (32.5% women and 67.5% men, mean age 36±11 years) completed the online survey. About one half of the assessors were physiotherapists (n=110, 46%). Seventy-one per cent of the assessors reported that they currently were working with athletes, and 55% of the assessors reported that they have assessed performance of the VDJ test before (table 1).

Table 1

Demographic data

Occupation	n=237	Gender(n=236)		Age(n=237)	Working with athletes (%)(n=233)				Years of experience in current occupation (%)(n=236)					Assessed VDJ test before (%) (n=237)
		Male	Female	Mean(SD)	Yes	Within the last 5 years	Within the last 10 years	No	0–2	3–5	5–8	8–12	12+	Yes	No
Physician*	20	13	7	43.8 (11.9)	65	5	0	10	10	25	10	10	45	25	75
Certified athletic trainer	15	11	4	40.2 (8.3)	87	13	0	0	0	20	0	27	47	53	47
Coach	20	14	6	40.8 (11.4)	85	0	5	10	5	25	30	10	30	45	55
Strength/conditioning coach	12	10	2	31.5 (8.0)	10	2	0	0	8	33	33	17	8	75	25
Physical therapist†	110	71	38	35.3 (9.8)	76	11	1	12	11	27	22	14	26	66	34
Researcher‡	34	24	10	38.0 (10.7)	47	41	6	6	18	12	21	15	35	53	47
Student	19	11	8	25.4 (4.3)	47	37	5	11	63	21	16	0	0	21	79
Other	7	5	2	33.9 (9.9)	86	0	0	14	29	14	14	29	14	71	29
Total	237	159	77	36.2	168	38	5	22	36	56	47	32	66	131	106

*Working within general practice (n=4), sports medicine (n=12) and orthopaedics (n=4).

†Working in private clinic (n=72), hospital inpatient/outpatient (n=16), municipality physiotherapist (n=5), rehabilitation clinic (n=15).

‡Working with ACL-related research: n=14.

VDJ, vertical drop jump.

Risk classification accuracy

Receiver operating curves are presented in figure 2. In 12 video clips, one score out of 237 was missing. Individual AUC values ranged from 0.36 to 0.60. Average AUC for each group ranged from 0.45 to 0.47 (table 2). There were no significant differences between groups (p=0.67). The mean rating score of the injured players was 4.5/10 (±1.7 SD) whereas the mean score of the non-injured players was 4.8/10 (±1.7 SD) (p=0.52, d=0.16).

Figure 2

Individual (grey shaded lines) and average (thick black line) receiver operating characteristic curve for all 237 assessors. The thin black line represent the diagonal line.

Table 2

Mean area under the curve values for each occupation with 95% CI

Occupation	N	Mean	95% CI
Physician	20	0.47	0.44 to 0.49
Certified athletic trainer	15	0.45	0.43 to 0.47
Coach	20	0.46	0.44 to 0.48
Strength and conditioning coach	12	0.45	0.43 to 0.48
Physical therapist	110	0.47	0.46 to 0.48
Researcher	34	0.47	0.46 to 0.49
Student	19	0.47	0.45 to 0.49
Other	7	0.46	0.43 to 0.50
Average	237	0.47	0.46 to 0.47

Sensitivity analysis of ROC curves calculated with adjusted ratio of injured and non-injured players showed no different results than the original ROC curve analysis (mean 0.43, 95% CI 0.40 to 0.47). The subanalysis of the crowd sourced data did not give any different results (table 3). Latent variable modelling showed high inter-rater reliability (r=0.75) between assessors when rating injury risk (figure 3). The assessors reported an overall confidence score of 6.0 (±1.9 SD) out of 10 for performing this injury risk assessment.

Table 3

Subanalysis for the crowd sourced data

Group	n	Injured rating		Non-injured rating		Number of injured correctly classified	P value	Cohen’s d
		Mean	SD	Mean	SD
Physical therapists	110	4.3	1.6	4.5	1.7	1 out of 20	0.58	0.15
High ACL-IQ (1SD above mean)*	29	4.7	2.1	5.0	2.1	2 out of 20	0.48	0.18
Entire sample	237	4.5	1.7	4.8	1.7	1 out of 20	0.52	0.16

See Petushek4 for psychometric characteristics of this tool.

*ACL Injury Risk Estimation Quiz (ACL-IQ) is a five-item video-based skill assessment to evaluate the ability of individuals to visually estimate risk for ACL injury using the drop vertical jump exercise.

Figure 3

Mean individual athlete risk rating across all raters. Error bars indicate SD. Injured players are marked in bold black.

Use of cues

Inward/outward knee motion, knee position in landing and landing symmetry were the three cues the assessors most frequently reported that they used for injury risk assessment. These cues were used by 99.2%, 99.6% and 98.7% of the assessors, respectively. Assessors also rated the importance of these factors when assessing ACL injury risk giving inward/outward knee motion a score of 8.8 (95% CI 8.7 to 9.0), knee position in landing 9.0 (95% CI 8.8 to 9.1) and landing symmetry 8.1 (95% CI 7.9 to 8.3) on a scale ranging from 1 to 10 (see figure 4). Jump alignment and landing stiffness were also factors the assessors frequently reported that they used in the assessment, 96.6% and 95.8% respectively.

Figure 4

Importance of cues (with 95% CIs).

Discussion

This study revealed that observational assessment of the VDJ test cannot be used to assess ACL injury risk in this population of elite female handball and football players. The results from ROC curve analysis showed that AUC values ranged from 0.36 to 0.60, meaning that the individual risk rating ability generally was no better than tossing a coin. No differences were identified between groups with different occupational backgrounds. Inward/outward knee motion, knee position in landing and landing symmetry were all cues used by nearly 100% of the assessors and rated as cues with high importance (over 8/10) for assessing ACL injury risk.

Despite of the possibility to assess all thinkable movement characteristics, this study indicates that visual assessment of the VDJ performance is no better than three-dimensional motion analyses for identifying players at risk of ACL injury. However, although a vast number of motion and person characteristics were reported to be assessed (figure 3), it seems that the main focus of the assessors was similar to that of previous two-dimensional video-based analyses.17–19 26 The assessors reported that they focused predominantly on frontal plane knee motion (knee placement in landing and inward/outward knee motion) in their assessment and considered it to be an important factor (9/10) for assessing ACL injury risk. High inter-rater reliability between assessors indicates that individuals were utilising similar cues, but that these cues do not provide useful information for assessing injury risk.

A critical question is therefore whether the VDJ test is challenging enough for these elite players to detect factors that potentially are associated with increased ACL injury risk. The test does not include changes in direction, which is commonly seen in the moment of injury. Moreover, the load is more or less equally distributed to both legs during landing in a VDJ test. In contrast, ACL injuries typically occur in side step cutting or one-legged landings.10 12 27 Moreover, the test is preplanned, meaning that players do not have to react and make sudden decisions based on outer interference, in contrast to a game play situation.

Targeting movement patterns is an important part of effective preventative exercise programme that will reduce the risk of sustaining an ACL injury,2 28 29 and to date there is strong evidence indicating that we should keep encouraging athletes to engage in preventative exercise programme providing feedback on knee alignment, soft landings, even if prediction of future injury with sufficient level of accuracy seems unlikely.30 Although the VDJ test may provide information about movement patterns and modifiable risk factors in some groups of athletes,17 19 the current study reinforces that frontal plane assessment of the VDJ test does not provide useful information to predict ACL injury in this cohort of female elite football and handball players.

Interestingly, despite the low accuracy, the assessors generally reported moderate to high levels of confidence in their risk rating assignment (mean 6.0±1.9 on a scale from 1 to 10 with 10 being high confidence). This indicates poor judgement calibration and adds data to support the unskilled and unaware phenomena,31 indicating that people tend to overestimate their abilities and lack the ability to evaluate how well they are performing. Based on our results it is important to acknowledge that individuals should not be confident when screening ACL injury risk through visual assessment of the VDJ test in this population.

The present study has limitations that should be taken into consideration when interpreting these results. First, the assessors only assessed the player’s movements from a frontal plane view. Less knee flexion10 11 16 19 32–35 during landing has been suggested to be associated with increased ACL injury risk, although not in all studies.18 This factor might have been easier to assess from a sagittal plane view. Nevertheless, landing stiffness can still be evaluated from a frontal plane view-based on landing depth.

Second, the picture quality and zoom factor differed slightly between the different test sessions. This may have prevented assessment of fine details in the movement patterns. However, it seems unlikely that assessment of the cues given in figure 4 would change significantly with higher image quality. Importantly, the assessors were given the opportunity to watch the clips as many times as they wanted, allowing an in-depth assessment.

Third, despite the limitations of three-dimensional motion analysis, visual inspection of joint kinematics in complex tasks has proven to be even less accurate.36 However, for the pure frontal plane assessment of a VDJ, the problem of for example, soft tissue artefacts is less of a problem in visual assessment. Moreover, a previous study3 reported good agreement between subjective assessments and three-dimensional motion-based measurements of frontal plane knee control.

Finally, it is possible that some of the players in the non-injury group would sustain an ACL injury after our observation period, thus misclassifying them. The players were followed prospectively until March 2014. However, as there is no difference in the risk assessment of injured and non-injured athletes, it seems highly unlikely that this would change our conclusion. It is still possible that the results of this study may have been different in a different population, for example, adolescent and male players.

Conclusion

The present study documents that sports medicine professionals and coaches cannot identify female elite handball and football players at increased risk of ACL injury by visually assessing their performance of the VDJ test. ROC curve analysis reveal a predictive ability no better than chance. Despite this fact, sports medicine professionals and coaches generally feel confident when assessing ACL injury risk in this population of elite female athletes.

Sports medicine professionals and coaches show high levels of inter-rater consistency when assessing ACL injury risk.

Sports medicine professionals and coaches have moderate to high level of confidence in ACL injury risk assessment.

However, observational rating of the vertical drop jump (VDJ) test cannot predict ACL injury in female elite athletes.

The VDJ test should not be used for identifying female elite football and handball players at risk of sustaining an ACL injury.

Sports medicine professionals and coaches should be cautious interpreting results from their visual assessment of the VDJ test.