Intra-rater Reliability of a Qualitative Landing Scale for the Single-Hop Test: A Pilot Study.

Background: The test battery classically used for return-to-sport (RTS) decision-making after anterior cruciate ligament (ACL) reconstruction (ACLR) may not be sufficient, as it does not include a qualitative analysis of movement. Therefore, the Landing Error Scoring System (LESS) scale was adapted to a primary functional test in the typical RTS test battery: the single leg hop for distance (SHD). Hypothesis/ Purpose: The aim of this study was to determine the intra-rater reliability of the LESS scale adapted to the SHD (SHD-LESS scale) in healthy young athletes. Study Design: Reliability analysis.
Methods: Nineteen healthy individuals (14 men, 5 women; mean age: 22.4 years) participated in the study. Participants performed the SHD tasks on both limbs (dominant and non-dominant) using a standardized protocol in two sessions that were one week apart (single reviewer; 2-dimensional video). Intra-class correlation coefficients (ICC2,1) were used to measure the reproducibility of the scale in the dominant (dom) and non-dominant (nondom) limbs. Additionally, limb data (dom and nondom) were pooled and evaluated collectively with intra-class correlation coefficients. The Kappa coefficient was used to assess the reproducibility of each individual item of SHD-LESS scale.
Results: The intra-rater reliability was good (ICCdom = 0.77; ICCnondom = 0.87; ICCpooled = 0.87) for the overall SHD-LESS scale scores. Agreement of SHD-LESS individual items ranged from 62% to 100%. Dorsiflexion at initial contact (97% agreement; kappa value=0.79) and knee valgus after landing (88% agreement; kappa value=0.65) had excellent agreement and kappa values.
Conclusion: The newly-adapted SHD-LESS scale showed good intra-rater reliability overall. Further studies should evaluate the impact of using the SHD-LESS scale within the RTS test battery on outcomes in patients after ACLR. Level of Evidence: 3.

INTRODUCTION

After an anterior cruciate ligament (ACL) injury, the aim of ACL reconstruction is to restore functional stability of the knee. Following surgery, a primary end goal of rehabilitation is to prepare the patient to resume sports-related activities, defined in a recent consensus statement on return-to-sport (RTS) as being a part of a continuum: return-to-participation, return-to-sport, and return-to-performance. During late-stage rehabilitation, sport-specific tasks such as pivoting and rapid change of directions are typically resumed. To start the transition toward RTS at the completion of rehabilitation, clinical practice guidelines recommend that athletes complete a RTS testing battery in order to evaluate readiness for the return to the field without restriction and with the lowest possible risk of reinjury. However, recent meta-analysis data demonstrated that the ACL reinjury rate (ipsilateral and contralateral) in athletes under 25 years of age following RTS was 23%, suggesting that currently-used RTS testing may not be comprehensive and may be insufficient to identify those at risk of poor outcomes or reinjury. While RTS testing batteries often include measures of muscle strength and knee-related functional performance, such as one-leg hop for distance tests, a key measure missing from most testing batteries used in RTS decision-making is the assessment of movement patterns or movement quality.

One-leg hop for distance tests, such as the single leg hop for distance (SHD), are the most commonly-used lower limb functional tests after ACLR. The SHD remains the most used test because it is simple and does not require any equipment. In addition, the SHD allows for the evaluation of the ability of the knee to absorb load during landing. The Limb symmetry Index of SHD, a ratio of the performance of the operated limb to the healthy limb, is widely used in return to sport test batteries. But SHD performance/distance symmetry analysis (LSI of SHD) has previously been questioned due to the potential risk of masking information about poor or altered movement strategies to achieve symmetric distance. For this reason, we sought to combine the objective measurement of distance with a qualitative analysis of landing.

The Landing Error Scoring System (LESS) was originally developed for a double-leg landing task by Padua and colleagues, and has a good interrater and intra-rater reliability (.95 and .96 respectively). Recent work by O’Connor has adapted this scale for a single-leg drop-landing task. However, in the scientific literature, no previous work has utilized a quality of movement scoring system with the most commonly performed functional performance task in the RTS test battery: the SHD. Additionally, because the SHD assesses unilateral characteristics of the lower limb, its use as the functional task of interest may be better suited for an assessment of movement quality in those with unilateral injuries compared to a bilateral jump-landing-rebound task (such as the original LESS). The aim of this study was to evaluate the intra-rater reliability of the LESS adapted to the SHD (SHD-LESS) in healthy young athletes. The hypothesis of this study was that the reproducibility of the SHD-LESS would be good or excellent (ICC > 0.75).

METHODS

Study Design

This study was an intra-rater reliability study and followed the “Guidelines for Reporting Reliability and Agreement Studies” (GRRAS) and “Quality Appraisal of Reliability Studies” (QAREL) during the planning and implementation of this study. Prior to the study, a declaration of conformity for the protection of data has been made (MR4 - 2214186v0) to CNIL (Commission Nationale de l’Informatique et des Libertés). All subjects signed written informed consent prior to participating in any study-related procedures.

Participants

Nineteen healthy young individuals (14 men and 5 women; age: 22.4 ± 0.25 years; height: 175.0 ± 1.5 cm; weight: 68.2 ± 2.0 kg) were recruited between February and May 2019 and participated in the study. The inclusion criteria specified that participants were non-professional athletes, participated in a cutting/pivoting sport (soccer, basketball, handball, rugby, budo…) with a minimum frequency of two training sessions/week and one match or game/week, and were between 18 and 25 years old. All participants were enrolled and tested during their sports season. Exclusion criteria were a history of knee injury (ACL or other ligaments of the knee joint) or a history of lower limb musculoskeletal injury less than three months prior to enrollment and testing.

Testing Procedures

Participants completed the SHD task (Figure 1) in two sessions separated by six to 10 days. The general protocol consisted of four consistent steps: 1) introduction to the testing session and tasks, 2) warm-up, 3) test set-up, and 4) task performance and data collection. Following a standardized warm-up program, study participants were shown a video demonstration of the correct SHD technique with the following instructions: 1) begin standing on a lower limb behind the start line with arms crossed over the chest; 2) attempt to jump as far forward as possible; and 3) land on the same lower limb while keeping the arms crossed to avoid upper extremity momentum assisting in jump performance (described previously). Participants were informed that the test would be invalidated if they lost balance (inability to maintain balance upon jump landing for at least 3 seconds) or could not control the landing. Participants were then allowed to ask questions to clarify their understanding of what was required in order to correctly perform the SHD and were provided three practice trials. Following practice trials, participants performed three trials on both the dominant limb and the non-dominant limb, as described by Reid et al. The dominant limb was defined as “the primary foot used to kick a ball”.

Figure 1. Single leg hop for distance (SHD) task and 2D camera configuration.

SHD-LESS scale

An adapted LESS scale was created to evaluate the SHD landing: the SHD-LESS scale (Table 1). The adapted SHD-LESS scale was developed using the same procedures as the single-leg drop-landing LESS scale recently described by O’Connor. All items on the O’Connor LESS scale for falling and landing on one leg were analyzed for their applicability to the SHD assessment. As a result of this analysis, the items (items 1, 2, 3, 8 and 9) were modified to be relevant to the SHD task (Table 1).

Table 1. Single leg hop for distance – landing error scoring system (SHD-LESS) scale.

			Score
Sagittal View	1. Forward Trunk Flexion at IC Error: flexion < 5°.	Yes = 0 No = 1
	2. Knee Flexion at IC Error: flexion <5°.	Yes = 0 No = 1
	3. Ankle Dorsiflexion at IC Error: the landing is not done with the heel.	Yes = 0 No = 1
	4. Forward Trunk Flexion Displacement Error: Trunk DOES NOT flex more than at IC	Yes = 0 No = 1
	5. Knee Flexion Displacement Error: <30° more flexion after IC; F(IC)-F(displacement).	Yes = 0 No = 1
	6. Ankle Dorsiflexion Displacement Error: Heel DOES NOT touch the ground	Yes = 0 No = 1
Frontal View	7. Knee Valgus at IC Error: the vertical line passing through the center of the patella is medial to or passes through the big toe.	Yes = 2 No = 0
	8. Lateral Trunk Flexion at IC Error: the trunk is tilted by more than 10°.	Yes = 1 No = 0
	9. Knee Valgus DisplacementA. Error: the vertical line through the big toe is outside/lateral or on the axis of the patella on the key frame. If not, B. Error: the vertical line through the big toe is outside or on the axis of the patella among all images between the IC and after displacement.	Yes = 2 No = 0 if not, Yes = 1 No = 0
	10. Contralateral Pelvic Drop Displacement Error: Contralateral pelvic drops below ipsilateral pelvic.	Yes = 1 No = 0
	11. Tibial Rotation Displacement Error: there is tibial rotation (more than 2°, included).	Yes = 1 No = 0
Others	12. Distance covered > 64% height Error: the distance is less than 64% of the subject size.	Yes = 0 No = 1
	13. Overall impression Error if the movement performed by the subject is "rigid/poor" at the hip, knee and ipsilateral ankle.	Excellent=0 Average=1 Poor=2
OVERALL SCORE (0-16)

IC: initial contact; Displacement: Maximal flexion knee

In addition, two items have been added to the SHD-LESS. The first item added (item 12; Table 1) included an evaluation of the jump performance (distance, cm). In completing the SHD, participants were instructed to jump as far as possible, and this external focus allowed participants to concentrate on the jumping performance and not the explicit execution of the movement/landing. Because it was possible that participants could purposefully jump a shorter distance to improve the overall landing movement quality which might affect the results, the jump distance criterion was defined as a minimum distance equal to 64% of the participant’s height. If the participant was unable to achieve this distance, a penalty of one point was imposed (Table 1). The second item added (item 13; Table 1) included an overall impression of movement quality during the SHD landing. This item is the same as that of item 16 of the original double-leg landing LESS described by Padua and colleagues. Overall landing quality were evaluated as excellent (0 points), average (1 point), or poor (2 points).

Two-dimensional video analysis

As described by Everard et al., two cameras were used to film the jumps from the front and the side. Prior to completing the video evaluation and scoring, each SHD test video was verified to include the necessary scoring information (angles, distances, and key images). The cameras were placed on a tripod perpendicular to the frontal and sagittal plane, at a height of 60 cm and a distance of 3m (Figure 1). The video recordings were analyzed using Kinovea (Kinovea, www.kinovea.org, version 0.8.15).

Markers were placed on the great trochanter, the lateral epicondyle of the knee, the lateral malleolus, the center of the patella (estimation of the center of the knee; equidistant from the epicondyles of the femur), and the center of the line passing through the two malleoli (estimation of the center of the ankle joint).

The video evaluations were scored by a single rater (AR; physiotherapist with 20 years of experience). After collecting all data, analyses were performed first for Session 1 and then Session 2, consecutively. The video evaluation to score the SHD-LESS was completed in two parts. First, the evaluation of the jumping performance (item 12; distance) was performed, and this defined the trial that would be evaluated for qualitative performance (the trial with the furthest distance among the 3 trials). Using this trial, the remaining items from the SHD-LESS were scored (items 1-11; 13).

Statistical Analysis

In order to determine intra-rater reliability of the scale, the overall score of the SHD-LESS was compared between Session 1 and Session 2 individually for each lower limb (dominant and non-dominant) using intraclass correlation coefficients (ICC2,1) with statistical adapted tests to the distribution of variables (Pearson test or Spearman test). Following research by Van Melick and al., the dominant and non-dominant limbs were pooled and evaluated between Session 1 and Session 2. ICC2,1 values were interpreted as poor (<0.50), moderate (0.50-0.74), good (0.75-0.89), or excellent (0.90-1.00), according to Portney and Watkins.

SHD distance was compared between Sessions 1 and 2 with statistical adapted tests to the distribution of variables (paired simple t test or Wilcoxon test). Statistical significance was set at p < 0.05. To determine intra-rater reliability of each individual item of the SHD-LESS scale, kappa coefficients were calculated. The kappa-values were interpreted as follows values ≤ 0 as indicating no agreement and 0.01–0.20 as very poor agreement, 0.21–0.40 as poor agreement, 0.41– 0.60 as moderate agreement, 0.61–0.80 as strong agreement, and 0.81–1.00 as near perfect agreement and =1, perfect agreement. Statistical analyses were performed using JASP (JASP Team [2018] Version 0.9.2.0) and R software (version 3.4.4 [2018-03-15]).

RESULTS

The final sample included 17 participants that completed the SHD. Two outliers were excluded from analyses for the SHD test (SHD performance/distance difference between Sessions 1 and 2 >2.5 SD). For the overall SHD-LESS scale scores, intra-rater reliability ICC values were between 0.77 and 0.87 (Table 2).

Table 2. Intra-rater reliability of qualitative (Landing Error Scoring System scale adapted to the single leg hop for distance (SHD): SHD-LESS score) and quantitative (jump distance) measures between sessions 1 and 2. Data are shown as ICC value (95% confidence interval); ICC: intraclass correlation coefficient; dom: dominant lower limb; nondom: non-dominant lower limb.

	Qualitative: Quality of landing score Test: SHD (SHD-LESS score)	Quantitative: Jump Performance Test: SHD (jump distance)
ICCdom	0.77 (0.46 - 0.91)	0.90 (0.73 - 0.96)
ICCnondom	0.87 (0.67 - 0.95)	0.94 (0.83 - 0.98)
ICCpooled	0.87 (0.75 - 0.93)	0.92 (0.85 - 0.96)

For individual items within the SHD-LESS scale, kappa coefficients ranged from very low to almost perfect, depending on the individual item evaluated (Table 3). For jumping performance (quantitative measure), intra-rater reliability of jumping performance ranged from 0.90 to 0.94, depending on the lower limb (dominant vs. non-dominant vs. pooled) (Table 3). No statistical differences were found between Sessions 1 and 2 for the of quality of landing scores (pdom=0.41 ; pnondom=0.07 ; ppooled=0.05) or for SHD jump distance (pdom=0.55 ; pnondom=0.53 ; ppooled=0.73).

Table 3. Percent agreement and Kappa coefficients for SHD-LESS (Landing Error Scoring System scale adapted to the single leg hop for distance) individual items.

		SHD-LESS
Items	Item description	% Agreement	Kappa value	Interpretation
1	Forward Trunk Flexion at IC (degrees)	94%	NC	NC
2	Knee Flexion at IC (degrees)	100%	NC	NC
3	Dorsiflexion at IC (position)	97%	0.79	Strong
4	Forward Trunk Flexion Displacement (degrees)	79%	0.34	Poor
5	Knee Flexion after Landing (degrees)	97%	0.00	Very poor
6	Ankle Dorsiflexion after Landing (position)	71%	0.12	Very poor
7	Knee Valgus at IC (position)	88%	- 0.06	Disagreement
8	Lateral Trunk Flexion at IC (degrees)	71%	0.30	Poor
9	Knee Valgus after Landing (position)	88%	0.65	Strong
10	Contralateral Pelvic Drop after Landing (position)	65%	0.02	Very poor
11	Tibial Rotation Displacement (degrees)	62%	0.24	Poor
12	Distance Covered > 64% Height (Distance)	97%	0.84	Almost perfect
13	General observation	76%	0.42	Moderate

IC: Initial Contact.

DISCUSSION

Main Results (Overall SHD-LESS scores)

The aim of this study was to determine the intra-rater reliability of using an adapted version of the LESS scale for a new task, the SHD. For the overall scores on the SHD-LESS scale, we found good intra-rater reliability (ICC=0.77 and ICC=0.87 for the dominant and non-dominant limb, respectively). Regarding the reproducibility of this newly-developed SHD-LESS scale, there are no previous studies with which to compare our results. In addition, we observed differences in ICC values between the dominant and non-dominant lower limbs. The dominant limb was defined as the primary foot used to kick a ball. However, unilateral tasks were evaluated in the current study, and it may have not been necessary to differentiate dominant and non-dominant limbs. The data could have been pooled initially. When analyzing the pooled data, good intra-rater reliability was found (ICCpooled = 0.87), utilizing a larger sample size (17 subjects with 2 legs; n=34).

Secondary Outcomes (Individual Item Analyses)

In this study, percentage of agreement for individual SHD-LESS items ranged from 62% to 100%. For tibial rotation (item 11), the rater (AR) of the study achieved only 62% agreement. Because tibial rotation is a transverse plane variable, it is likely that evaluating this item may be particularly difficult with cameras only in the sagittal and frontal planes, and may be inappropriate for a 2D assessment of movement. In calculating percent agreement values, Cohen suggests the possibility that there is the potential for false agreement when raters make random guesses, and Cohen’s kappa was developed to account for this concern. For individual item assessment of the SHD-LESS, the Kappa coefficients ranged from very poor to almost perfect for both scales. While overall scores for the SHD-LESS demonstrated good reliability, only three items (dorsiflexion of the ankle, knee flexion displacement, and knee valgus displacement) showed Kappa values interpreted as strong on the 13 items for this scale. Interpretation of the Kappa scores for some items does not seem possible, because no errors were found in the video analyses for some items (not allowing for calculation of the Kappa score). These items may have less relevance for a healthy population, but could be of great relevance for individuals following ACL reconstruction. Thus, future studies should evaluate the reliability (overall and individual item assessment) of the SHD-LESS in patients following ACL reconstruction.

To ensure that subjects performed a reasonably similar task between the two sessions, jumping performance on the SHD (quantified as hop distance) has been investigated and an excellent agreement for jump performance (distance; item 12; Table 2) on the SHD task has been obtained. These findings are consistent with previous work examining the reliability of the SHD for jump performance (distance), which found excellent reliability (ICC=0.92). While the jump distances between the two sessions were highly correlated, recent research has shown that individuals following ACL reconstruction with similar distance performance on the SHD and other dynamic tasks use varying movement strategies to accomplish the tasks. Thus, the landing movement strategies used with similar distance jumps between the 2 trials could not be really identical, which may affect the reliability of the qualitative landing assessment.

Further applications

The overall scores for the SHD-LESS scale demonstrated good reliability and may be an important component in future ACL reconstruction RTS testing batteries. The SHD-LESS scale contributes to an objective RTS evaluation, with an analysis of the quality of landing during the test most frequently used in the decision to RTS (i.e., the SHD). The SHD may also be the most relevant sport-related task to evaluate for quality of movement in individuals following ACLR. Specifically, unilateral tasks may be more relevant than bilateral tasks for qualitative assessment, because of the possibility of compensation on the healthy lower limb during a bilateral landing task like the drop vertical jump (DVJ). Additionally, the single-leg hop tests described originally by Noyes et al. have been used extensively and previously evaluated for intra- and inter-rater reproducibility. These functional tests therefore provide clinicians with important information in the follow-up and serial testing of their ACLR patients.

The single-leg drop landing task is also incorporated in RTS testing batteries following ACL reconstruction, and is associated with other important elements such as psychological factors and social-contextual factors. Also, a single-leg drop landing task may be more biomechanically-demanding compared to SHD and only appropriate in late-stage rehabilitation. The SHD is often performed earlier in rehabilitation following ACL reconstruction (approximately 3 months) to assist with return to running, and thus may have greater utility during early-to-mid rehabilitation progression.

Single-leg distance jump tests appear to have low sensitivity in predicting injury-free return to sport, but high specificity, and providing an objective assessment of the patient’s functional abilities after ACLR. It should be noted that functional tests, although providing valuable information, do not, on their own, indicate how well the patient is recovering from ACLR or how ready for return to sport they are.

Study limitations

The current study, although pilot in nature, is not without important limitations. First, the 2D video analysis time was particularly long to complete the scale scoring for each task (30 minutes for one limb). This may limit the clinical application of these scales in a real-time setting. Secondly, the current study included only a young population (20-25 years old), with more males than females, which does not reflect or apply to the general population. Finally, items of this newly-developed SHD-LESS scale were not previously validated, due to the pilot nature of this study. Future research should refine and validate these individual items for eventual applications in patient populations (such as those post-ACL reconstruction) for which movement quality assessment may be a critical part of a multifactorial functional assessment.

Finally, it would be interesting to evaluate inter-rater reproducibility as well as to establish a cut-off score to reduce the risk of (re)injury, especially after ACL reconstruction.

CONCLUSION

The newly-adapted SHD-LESS scale showed good intra-rater reliability, and could be used alongside the evaluation of SHD performance in RTS testing following ACL reconstruction. Further studies should evaluate the inter and intra-rater reliability of the SHD-LESS in individuals following ACL reconstruction as well as the impact of using the SHD-LESS scale within the RTS test battery on outcomes.

Conflicts of Interest

The authors declare that they have no conflicts of interest.