Psychometric evaluation of commonly used game-specific skills tests in rugby: A systematic review.

OBJECTIVES: To (1) give an overview of commonly used game-specific skills tests in rugby and (2) evaluate available psychometric information of these tests.
METHODS: The databases PubMed, MEDLINE CINAHL and Africa Wide information were systematically searched for articles published between January 1995 and March 2017. First, commonly used game-specific skills tests were identified. Second, the available psychometrics of these tests were evaluated and the methodological quality of the studies assessed using the Consensus-based Standards for the selection of health Measurement Instruments checklist. Studies included in the first step had to report detailed information on the construct and testing procedure of at least one game-specific skill, and studies included in the second step had additionally to report at least one psychometric property evaluating reliability, validity or responsiveness.
RESULTS: 287 articles were identified in the first step, of which 30 articles met the inclusion criteria and 64 articles were identified in the second step of which 10 articles were included. Reactive agility, tackling and simulated rugby games were the most commonly used tests. All 10 studies reporting psychometrics reported reliability outcomes, revealing mainly strong evidence. However, all studies scored poor or fair on methodological quality. Four studies reported validity outcomes in which mainly moderate evidence was indicated, but all articles had fair methodological quality.
CONCLUSION: Game-specific skills tests indicated mainly high reliability and validity evidence, but the studies lacked methodological quality. Reactive agility seems to be a promising domain, but the specific tests need further development. Future high methodological quality studies are required in order to develop valid and reliable test batteries for rugby talent identification. TRIAL REGISTRATION NUMBER: PROSPERO CRD42015029747.

Rugby players are required to have a set of favourable anthropometric properties and well-developed physiological qualities, combined with a range of game-specific skills, both offensive and defensive, to cope with the demands of the game.

Anthropometric and physiological characteristics, like body mass, maximal leg strength and lower body muscular power, have been shown to differentiate between different rugby playing levels.

Game-specific skills tests in rugby seem to have predicting value in discriminating talented and less talented players, but a review investigating the underlying psychometrics of these tests is lacking.

Reactive agility, tackling skills and simulated rugby games are the most investigated game-specific skills.

For game-specific skills, there is mainly moderate to strong evidence for reliability and validity, but the studies lack methodological quality.

Reactive agility tests seem to be very promising, based on the preliminary evidence on validity, reliability and feasibility.

Future research should be focused on detailed and standardised studies of game-specific skills tests. These studies should include detailed reports of the study procedures, critical evaluation of test design and adequate sample sizes of n≥50 or a sample size calculation.

These findings can impact the current practice in the way that researchers, scouts and coaches will critically evaluate the tests they use to identify talented players, start focusing on new game-specific skill domains and will come up with new tests in methodological rigorous studies to have an objective and psychometric sound backbone for their talent identification.

Introduction

Rugby is a diverse collision sport played all over the world on amateur, semiprofessional and professional levels.1 Worldwide, it is played by >7 million players and the numbers are increasing.2 The >2.47 million tickets sold during the World Cup Rugby 2015 and the estimated 120 million people watching the final highlighted the popularity of the sport.3 During the game, players participate in frequent bouts of intense activity, separated by short bouts of low-intensity activities.4 The high-intensity activities include sprinting, physical collisions and tackles and low-intensity activities include walking and jogging.4–6 During the course of a game, each team will perform on average 300 tackles, with semiprofessional rugby matches being played at a mean intensity of 81% VO2-max.7 8

To cope with these demands, rugby players are required to have a set of favourable anthropometric properties and well-developed physiological qualities, combined with a range of game-specific skills, both offensive and defensive.7–9 Identifying young players who are promising to develop these prerequisites of the game requires rugby clubs to have efficient talent identification programmes.10 11 Fastened by the increase in popularity and professionalism in rugby, there is even more need to efficiently identify talent, so they come up with the most talented players and have the best teams in their category. Additionally, it is hard to predict the future potential of a player in the perspective of their development potential and factors surrounding the player. However, talent identification still remains mainly based on subjective assessments of scouts and coaches. To complement and increase the quality of this process, it would be beneficial to add a degree of objectivity by including sport science contributions on early talent predictors.11 12 This can be done by including specific tests in the talent identification process, which should be chosen based on proven potential to predict talent in a methodological responsible way.

Several studies investigated factors that potentially predict talent. In these studies, variables such as body mass, maximal leg strength and lower body muscular power were shown to differentiate between first and second division rugby league players.8 13 However, it remains questionable if the physiological qualities and anthropometric characteristics are the factors that discriminate between playing levels since higher levels of strength, power or body mass do not directly relate to better performance during the game. Not every amateur rugby player that increases levels of strength, power or body mass becomes an elite rugby player. The well-developed physical qualities must be translated into improved playing performance to have practical significance.9 Based on this, one presumes there should be additional, possibly even more important factors, which discriminate between talented and less talented players. In line with this, Gabbett et al14 found that skill-related characteristics discriminated most between successful and less successful rugby league players, while physiological or anthropometric characteristics did not. These findings suggest an important role for game-specific skills in rugby, and specific tests can potentially identify talented players who are able to translate the physical qualities to improved playing performance.

The inclusion of objective talent predicting assessments requires multidimensional test batteries on the unique set of anthropometric characteristics and physiological and game-specific skills qualities that are required.15 Different tests on talent predictors are available and used, but there are so far no overviews of the psychometric information underlying these tests and there is no gold standard yet. However, to be of added value in the process of talent identification, inclusion of tests should be based on objectivity, feasibility, validity and reliability. Additionally, a player should be measured at a young age and followed up for several years to investigate the progress of that specific variable in relation to the actual playing level, to judge if a certain variable has proven to be a talent predictor. However, studying talent predictors in this way requires longitudinal studies, which are generally expensive and time consuming and therefore often known groups validity approaches are used. In known groups validity approaches, groups that are known to differ or logically should differ are used and the test performance is compared over these groups.16

With this taken into account, the addition of sport science contributions could only be beneficial to the talent identification process when tests with good reliability and validity are being developed, so they can be used as an objective backbone for the subjective assessment of scouts and coaches. Therefore, it is required to have a clear overview of the available psychometric information of the commonly used tests in rugby. An overview of these tests can guide researchers, scouts and coaches in the process of development or inclusion of objective tests in talent identification programmes. In this review, we specifically focus on game-specific skills tests in rugby as these tests have not been investigated in detail, specifically not the underlying psychometrics. Besides the review about skills outcomes in different sports by Robertson et al,15 there are, to our knowledge, no available reviews about psychometrics of game-specific skills tests in rugby. Therefore, the aim of this review was to systematically evaluate the level of evidence on psychometric properties of measures of game-specific skills, used for males in rugby. Information from this review will guide selection of measurement instruments for future studies and assist scouts and coaches in knowledge about common usage of game-specific skills in literature and their psychometric properties. The review is divided into two steps. First, the commonly used game-specific skills will be investigated to get an overview of the current situation in game-specific skills testing in rugby. Second, the available psychometrics of these tests and methodological quality of the studies will be analysed. The second step about the psychometrics will be the main focus of this review, while the first step will be used to put the psychometric information into perspective of what is commonly used in rugby.

Methods

Literature search

A systematic review of all published literature was undertaken in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.17 A literature search for both steps was conducted on the databases PubMed, MEDLINE, CINAHL and Africa-Wide Information. Additionally, a hand search on the reference lists of included articles for the second step was done. The two steps varied in included articles because of an additional category in the search terms for the second step. The search terms for the first step included keywords of (1) game-specific skills, (2) age categories, (3) sports and (4) assessment tools. With these keywords game-specific skills tests could be found, without taking into account psychometric information. To find studies reporting psychometric information on these tests, an additional category of keywords was included in the second step. This was an extra category including keywords about (5) psychometric properties. The specific keywords of both categories can be found in the online supplementary appendix.

Selection criteria

English articles published between January 1995 and March 2017 were included. Only studies containing male subjects playing rugby league or rugby union were included. Opposed to the initial criteria set in the protocol on PROSPERO, we did to not include Australian Football players. The gameplay and required game-specific skills for these sports differ too much to make fair comparisons between these sports. No limitations were made for study design and age category. However, editorials, book chapters, poster and oral conference abstracts, unpublished theses, dissertations and case studies were excluded. Furthermore, studies published in non-English language and studies involving rugby participants living with disabilities were excluded.

Studies in the first step, about commonly used game-specific skills tests, had to report detailed information about the construct and testing procedure of at least one game-specific skill. Game-specific skills were defined as skills that directly relate to better performance during the game. These include, but are not limited to, ground skills, side-stepping, reactive agility, aerial and ground kick, passing for distance, passing for accuracy, kicking and catching (while moving).18 Studies included in the second step, about psychometric information, should have reported also on at least one psychometric property evaluating reliability, validity or responsiveness.

Selection process

Two reviewers (SO and MC) independently screened titles, abstracts and, if necessary, full texts to assess inclusion or exclusion based on the predefined criteria. In case of disagreement, the reviewers discussed until consensus was reached. Otherwise disagreements were resolved through discussion or referral to a third reviewer (BSE). The full texts of titles and abstracts deemed potentially relevant were retrieved and reviewed for inclusion.

Data synthesis

The following data were extracted: publication details (first author, year of publication), title, purpose of the study, age of the participants, test groups, rugby code, game-specific skill construct evaluated, test(s) used to measure the construct, measurement properties assessed (reliability, validity and responsiveness), evidence level and Consensus-based Standards for the selection of health Measurement Instruments (COSMIN) rating.

For the reliability results, strong evidence was indicated with intraclass correlation coefficient (ICC) ≥0.70, Pearson’s correlation coefficient (PCC) ≥0.80 or Cronbach’s alpha ≥0.70. For the validity results, the low effect size (ES) were indicated as weak evidence and high ES with strong evidence. Weak evidence was determined with ES 0.2–0.49, moderate evidence with ES 0.5–0.79 and strong evidence with ES ≥0.8. Studies reporting validity results in ω2 were assessed based on criteria set in that specific study as no assessment of the ω2 is available in the COSMIN.

Risk of bias assessment

The methodological quality of the selected articles in the second step was assessed using the COSMIN checklist by two independent reviewers (SO and MC).19 This checklist was developed to rate the methodological quality of a study on one or more measurement properties. The rating is based on a point rating scale (poor, fair, good or excellent) on different items of design requirements and expected statistical methods. The overall rating for the methodological quality is indicated by the lowest score of these items.

Results

Search results

In the initial procedure of the first step, 287 articles were retrieved. After removal of duplicates, screening the records and assessing full texts of potential studies, 30 articles were included in this step. The initial procedure in the second step delivered 64 articles. After removal of duplicates, screening the records and assessing full texts of potential studies, 10 articles were included. The flow diagram for search results and study selection is illustrated in figure 1.

Figure 1

Flow diagram for search results and study selection.

Step 1: commonly used tests

An overview of the indicated game-specific skills and subjects used is shown in table 1. Reactive agility and tackling skills were the most investigated skills, assessed both in seven articles. Three articles tested multiple skills. Articles assessed game-specific skills both in training setting and during match play. Two articles in training setting reported specifically on multiple skill-based tests, which were reactive passing and tackling under fatigue.20 21 Of the 30 articles, 18 articles used rugby league players, 10 articles used rugby union players and 2 did not specify the rugby code.

Table 1

Overview of game-specific skills found in studies included in the first step

	Skill tested	Rugby code
Green25	Reactive agility	Union
Serpell26	Reactive agility	League
Serpell29	Reactive agility	League
Gabbett24	Reactive agility	League
Gabbett33	Reactive agility	League
Gabbett9	Reactive agility, pattern prediction/recognition, tackling, catching, passing	League
Gabbett34	Reactive agility, pattern prediction/recognition, tackling, catching, passing	League
Pavely20	Reactive passing	Union
Usman35	Tackling	Union
Gabbett31	Tackling	League
Gabbett21	Tackling under fatigue	League
Waldron36	Tackling, catching and passing	League
Lacome37	Tackling and passing during match	Union
Wu38	Scrummage	Not specified
Pienaar10	Catching and passing	Union
Vaz39	Simulated rugby games	Union
Higgings40	Simulated rugby games	Union
Stuart22	Simulated rugby games	Union
Gabbett41	Simulated rugby games	League
Sirotic42	Game-specific skills per minute during match	League
Sirotic27	Game-specific skills per minute during match	League
Kempton28	Game-specific skills per minute during match	League
Bennett23	Skill involvement during match	League
Lacome43	Skill involvement during match	Union
Linthorne44	Rugby kick	Not specified
Cockcroft45	Rugby kick	Union
Gabbett46	Two-on-one attacking drill	League
Gabbett47	Two-on-one attacking drill	League
Gabbett14	Playing ability assessed by coach	League
Gabbett48	Playing ability assessed by coach	League

Step 2: psychometrics

Description of included studies

The 10 included articles in the second step reported in total on 49 variables of psychometrics. Seven articles assessed rugby league players and three articles rugby union players. Four studies assessed reactive agility, whereby Gabbett et al9 also assessed other technical and perceptual skills. Also four studies assessed game-specific skills during a match. The other three studies focused on other technical or game-specific skills, like passing and tackling. Between articles reporting on the same skill, test designs differed in set-up, assessment and subjects used. Studies ranged from 9 to 218 subjects, with reported age ranging from 16.5±1.0 to 25±4 years.22 23 Subjects varied from first to second grade, elite senior to elite junior and experienced to non-experienced players.

Reactive agility

Gabbett et al 24 and Gabbett et al 9 used the same test design, with players required to follow direction of tester initiated movement to left or right through finish gate. Green et al.25 used the design where players were required to sprint to 45° change of direction point and change direction to left or right flashing finish gate. And Serpell et al26 used a completely different design and asked players to sprint towards a screen displaying action of sport-specific movement and change direction as playing defender.

Game-specific skills during match

The game-specific skills during match were all analysed after the match by an expert analyst and tested on inter-rater or intra-rater reliability. Gabbett et al.14 let two expert coaches assess playing skills (general, evasion, tackling/defensive, offensive) on standardised criteria. Kempton et al, Sirotic et al and Bennett et al all coded specific events.23 27 28 Kempton et al28 coded kicks received, passes, ‘dummy-half’ pass, ball caries and kicks (attacking and for territory). Sirotic et al27 assessed ball caries, support runs, touches of the ball, play-the-balls and tackles made. And Bennett et al23 coded the ball caries, support runs, offensive misses, line breaks, line break assists and tackles completed or not completer.

Simulated rugby games and catching and passing

Furthermore, Stuart et al22 assessed simulated rugby games, focused on different forms of sprinting, kicking and passing. Pienaar et al10 analysed passing for distance, passing for accuracy over 4 and 7 m and catching while running.

An overview of all characteristics of the included studies is illustrated in table 2.

Table 2

Study characteristics and test design regarding studies included in the second step

Author(s)	Rugby code	Skill tested	Subjects	Age (mean±SD)	Test design
Green25	Union	Reactive agility	Academy (n=17)	19±1.67	Sprint to 45° change of direction point and change direction to left or right flashing finish gate
			Club (n=11)	19±1.30
Gabbett24	League	Reactive agility	First grade (n=12)	23.6±2.3	Follow direction of tester (four scenarios) initiated movement to left or right through finish gate
			Second grade (n=30)
Serpell26	League	Reactive agility	Elite (n=15)	Not reported	Sprint towards screen displaying action of sport-specific movement (48 scenarios) and change direction as playing defender
			Elite junior (n=15)
Gabbett9	League	Technical and perceptual skills	First grade (n=58)	23.8±3.8	Reactive agility: follow direction of tester (four scenarios) initiated movement to left or right through finish gate
					Pattern recall: recall player positions at time of video footage occlusion
					Pattern prediction: predict next action at time of video footage occlusion
					Tackling: 1-on-1 tackling drill assessed on standardised criteria by sport scientist
					Draw and pass: 2-on-1 drill in single and dual-task conditions assessed by sport scientist
Pienaar10	Union	Catching and passing	Experienced (n=173)	Not reported	Passing for distance: passing as far as possible
			Non-experienced (n=45)		Passing for accuracy: passing over 4 and 7 m towards target
					Running and catching: catching the ball while running
Stuart22	Union	Simulated rugby games	Elite (n=9)	25±4	Kick for distance: kicking as far as possible
					Kick-off for distance: kicking-off as far as possible
					Offensive sprint: forward run with ball while swerving
					Defensive sprint: running forward and backward three arcs
					Tackle sprint: tackle on tackle bag, running backward with ball, making another tackle and run forward
					Passing accuracy: repeatedly pass ball as fast as possible towards target
Gabbett14	League	Game-specific skills during match	First grade (n=86)	22.5±4.9	Two expert coaches assessing playing skills (general, evasion, tackling/defensive, offensive) on standardised criteria
Kempton28	League	Game-specific skills during match	Elite (n=5)	Not reported	2× first halves of 10 subjects coded on game-specific skills analysed on standardised criteria by trained operator from recordings of matches
			Elite junior (n=5)
Sirotic27	League	Game-specific skills during match	Elite (17)	24.8±3.1	2× first halves of nine matches analysed on game-specific skills per minute of playing time
			Semi-elite (22)	22.1±2.4
Bennett23	League	Game-specific skills during match	Elite junior (45)	16.5±1.0	2× first halves of eight matches analysed on skill involvement during match

Reliability and validity outcomes were the only risk of bias assessment data that were described in detail in the included articles in the second step. No details on other risk of bias assessment variables were given. Therefore, only outcomes on these two factors of the risk of bias assessment were evaluated.

Reliability outcomes

All 10 articles included in the second step reported reliability outcomes. Five articles reported test–retest reliability, four articles intra-rater reliability and one article both inter-rater and intra-rater reliability. Also, 27 of the 33 reliability outcomes indicated strong evidence, with the highest ICCs for intra-rater reliability of 1.00 on line break assists and tackles completed during a match.23 Only Serpell et al26 indicated two ICCs of ≤0.5 on perception and response time and confidence rating for test–retest reliability in a reactive agility test. The ICC was the most used analysis, PCC and Cronbach’s alpha were both used once. Stuart et al29 reported coefficient of variation (CoV), which indicated the level of dispersion around the mean. However, there is no COSMIN guideline for assessing CoV. In two studies, CoVs were <10%, and in two studies they found CoVs >10%. Five of the articles were found to be of poor and five of fair methodological quality. The main reasons for this were missing information about percentage and handling of missing items, small sample size and methodological flaws. Methodological flaws included missing information on the participants, unclear study design and missing information on the retest situation in test–retest setting or about the second rater in inter-rater setting. The reliability outcomes are illustrated in table 3.

Table 3

Reliability outcomes of studies included in the second step

Author(s)	Rugby code	Skill tested	Subjects (N)	Reliability	Variable	Result	Evidence	COSMIN
Green25	Union	Reactive agility	11	Test–retest	Reactive agility speed (s)	ICC=0.88, SEM=0.09	+	Poor
Gabbett24	League	Reactive agility	42	Test–retest	Movement time (s)	ICC=0.92, TE=2.1%	+	Fair
					Decision time (ms)	ICC=0.95, TE=7.8%	+
					Response accuracy	ICC=0.93, TE=3.9%	+
Serpell26	League	Reactive agility	15	Test–retest	Total agility time (s)	ICC=0.82, SEM=0.01	+	Fair
					Perception and response time (s)	ICC=0.31, SEM=0.01	–
					Participants’ confidence about decision made (%)	ICC=0.50, SEM=2.12	–
Gabbett9	League	Technical and perceptual skills	58	Test–retest	Tackling assessment	ICC=0.83, TE=3.3%	+	Poor
					Draw and pass assessment	ICC=0.86, TE=5.3%	+
					Reactive agility, decision accuracy	ICC=0.93, TE=3.9%	+
					Reactive agility, decision time	ICC=0.95, TE=7.8%	+
					Pattern recall	ICC=0.80, TE=9.3%	+
					Pattern prediction	ICC=0.85, TE=8.7%	+
				Intra-rater	Playing performance	ICC>0.80, TE<5.0%	+
Pienaar10	Not specified	Catching and passing	36	Test–retest	Passing for distance	PCC=0.74	–	Fair
					Passing for accuracy, 7 m	PCC=0.66	–
					Passing for accuracy, 4 m	PCC=0.39	–
					Running and catching	PCC=0.53	–
Stuart22	Union	Simulated rugby games	9	Between-subjects CoV	Offensive sprint	CoV=13%		Poor
					Defensive sprint	CoV=9.2%
					Tackle sprint	CoV=9.8%
					Passing accuracy	CoV=20%
Gabbett14	League	Game-specific skills during match	86	Intra-rater		ICC=0.85–0.98, TE=5.1%–5.3%	+	Poor
				Inter-rater		ICC=0.84–0.94, TE=7.0%–9.0%	+
Kempton28	League	Game-specific skills during match	10	Intra-rater	Attempted tackle	α=0.81	+	Poor
					Kick receive
					Pass
					‘Dummy-half’ pass
					Ball carry
					Kick (attacking)
					Kick (territory)
Sirotic27	League	Game-specific skills during match	39	Intra-rater	Ball carries	ICC=0.996, TEM=0.008	+	Fair
					Support runs	ICC=0.986, TEM=0.010	+
					Touches of the ball	ICC=0.997, TEM=0.022	+
					Play-the-balls	ICC=0.997, TEM=0.008	+
					Tackles made	ICC=0.991, TEM=0.015	+
Bennett23	League	Game-specific skills during match	45	Intra-rater	Ball carry	ICC=0.98, TEM=0.053	+	Fair
					Support run	ICC=0.86, TEM=0.077	+
					Offensive miss	ICC=0.71, TEM=0.061	+
					Line break	ICC=0.86, TEM=0.007	+
					Line break assist	ICC=1.00, TEM=0.000	+
					Tackle completed	ICC=1.00, TEM=0.064	+
					Tackle not completed	ICC=0.99, TEM=0.000	+

‘+’, strong evidence; ‘–‘, limited evidence; α, Cronbach’s alpha; COSMIN, Consensus-based Standards for the selection of health Measurement Instruments; CoV, coefficient of variation; ICC, intraclass correlation coefficient; PCC Pearson’s correlation coefficient, SEM, standard error of measurement; TE, typical error of measurement; TEM, technical error of measurement.

Validity outcomes

Four articles reported construct validity outcomes based on known groups evaluations. Different groups were compared; from experienced to non-experienced and elite to elite youth. Three articles assessed reactive agility, reporting Cohen’s d ES, and one article assessed passing, running, catching and kicking skills, reporting ω2. Practical significance was determined at ω2 ≥14% on the basis of assessment by Pienaar et al.10 Four of the six ES on different variables of the reactive agility tests indicated moderate evidence and one ES indicated strong evidence. Highest was an ES of 1.14 found by Green et al25 on reactive agility speed compared between academy and club rugby union players. Lowest ES (0.34) indicated weak evidence, which was on response accuracy in a reactive agility test compared between first-grade and second-grade players, by Gabbett et al.24 The ω2 values varied from 10.6% for the passing for accuracy test on 4 m to 50.7% on the passing for accuracy test on 7 m. Furthermore, all studies indicated fair methodological quality according to the COSMIN checklist. Main reasons were missing information on percentage and handling of missing items, low sample sizes and methodological flaws. Methodological flaws included missing information on the participants, unclear study design and missing information on how the validity results were determined.

Discussion

The present systematic review investigated existing game-specific skills tests and evaluated available psychometrics of these tests. The main finding was that studies on these tests mainly indicated moderate to high evidence for reliability and validity. However, articles assessing psychometrics often lacked methodological quality (tables 3 and 4). Therefore, it still needs to be confirmed if the tests are really valid and reliable in more methodologically rigorous studies specifically designed to evaluate psychometric properties. Since the findings of the present review can be influenced by the procedures followed in the included studies (eg, incomplete description of the procedure, low sample sizes, data handling), the reported reliability and validity results should be interpreted with caution. The results of this systematic review can help researchers, scouts and coaches in the process of development or inclusion of objective game-specific skills tests in talent identifying programmes in rugby clubs.

Game-specific skills

In the first step of this systematic review, 30 articles reporting on game-specific skills were included. Reactive agility, tackling skills and simulated rugby games were the most studied skills. Reactive agility is defined as ‘a rapid whole body movement with change of velocity or direction in response to a sports specific stimuli’.29 Tackling and simulated rugby games were tested using multiple different set-ups, assessments and analyses. In the second step, 10 articles were included, in which reactive agility also was the most assessed skill. In these studies, mainly test–retest and intra-rater reliability were assessed. The majority of these studies indicated strong reliability evidence, meaning that the tests were relatively consistent over different occasions or raters. Also, 4 of the 10 articles reported validity outcomes with ES, using known groups comparisons. In these studies, moderate to strong evidence was found for different reactive agility tests. None of the studies reported on comparison with some kind of gold standard, and for most game-specific skill domains there is no gold standard yet. With only four studies reporting validity there is a clear need for validity analyses of game-specific skills tests. Before tests can be of benefit in the talent identification process, their predictive validity should be known, otherwise the implication of a specific score on these tests do not give meaningful insights with practical validity.

A strength of this systematic review was the use of the COSMIN checklist for the assessment of methodological quality of the articles. This assessment for articles reporting on reliability resulted in five articles indicated with poor and five articles with fair methodological quality. All articles reporting on validity scored fair on methodological quality, according to the COSMIN. These low scores were mainly caused by low sample sizes, methodological flaws that were not properly addressed and lack of information about percentage and handling of missing items. The low methodological quality could be the result of the finding that most studies were not specifically designed to assess psychometrics. The studies primarily focused on relations between test outcomes and playing groups. Assessing psychometrics seemed to be a side issue, causing minimal details on testing procedures. Therefore, we recommend researchers to come up with detailed and standardised studies of game-specific skills tests in rugby. These studies should include detailed reports of the study procedures, critical evaluation of test design and adequate sample sizes of n≥50, according to the COSMIN guidelines, or a sample size calculation. These recommendations are challenging to apply in the real world, but are necessary to come up with studies with high methodological quality and be able to work towards a better underlayment of the game-specific skills tests.

Psychometrics

The included articles varied in test design, subjects and statistical analyses used. Therefore, comparison of psychometrics over different studies was difficult. For example, three different protocols were used in four reactive agility tests. These studies used different cues to react on, like flashing timing gates, directions of a tester or a screen displaying sport-specific actions. Arguably, different outcomes are found when different test designs or groups are compared. Therefore, standardisation and detailed reporting is required. Furthermore, objective guidelines for interpretation of outcomes should be developed, taking into account the specific age and playing position groups. These guidelines have to be based on game requirements for the playing groups, which can be followed by developing levels of minimal requirements for the different groups. For determining these guidelines, promising techniques like Global Positioning System tracking and video analyses can be used.

Some interesting findings of the studies included in the second step were that the passing tests of Pienaar et al10 and Stuart et al22 seemed to have difficulties to establish good reproducibility, based on the low PCCs of 0.39 and 0.74 of Pienaar et al and the high CoV of 20% for passing accuracy in the study of Stuart et al.10 22 Also differences were found between studies using reactive agility tests. Serpell et al26 indicated weak correlations between test and retest measurement in elite youth rugby league players for perception and response time and confidence rating (ICC=0.31 and 0.50, respectively). Although, a strong correlation (ICC=0.82) on the test and retest measurement for total reactive agility time was found. This could be the result of players reacting differently on the cues in the test and retest, while keeping their total reactive agility time relatively constant. However, Gabbett et al contradict this finding because they found strong correlations between the test and retest of their reactive agility test for both decision time and response accuracy, with ICCs of 0.95 and 0.93, respectively.24 Furthermore, Pienaar et al10 found large differences between the performance of experienced and non-experienced players on passing for accuracy over 4 and 7 m, respectively, ω² of 10.6% and 50.7%.10 The difference in the passing test was only 3 m, but the passing test over 7 m had the ability to better discriminate between experienced and non-experienced players than the passing test over 4 m. However, it remains unclear if this difference in discriminant power between the 4 and 7 m passing test of Pienaar et al10 resulted from different test design, subject selection, technique or other reasons.

Comparison of the reliability and validity outcomes resulted in some relatively low reliability and high validity outcomes found on the same test. This happened mainly on reactive agility and passing tests. For example, Pienaar et al10 found in their passing for accuracy test over 7 m a relatively low PCC of 0.66 for the test–retest situation, but a relatively high ω² of 50.7% for the discriminative validity between experienced and non-experienced players. This indicated the ability to discriminate between different groups, but groups showed inconsistent patterns over different test occasions. Therefore, stricter procedures could be developed to decrease variability. Moreover, it would be of interest to gain information about decisions made by the same player in the same situation or about decisions made by different players in a similar situation. This ‘inter-player’ and ‘intra-player’ reliability could be important, especially in reactive agility tests to evaluate decision-making patterns of specific players. Possibly this decision-making ability differs between playing levels and can partly explain the difference between talented and less talented players.

Reactive agility

Of all the different game-specific skills that were assessed, reactive agility seems to be the most promising skill to discriminate between different playing groups. Outcomes of studies analysing reactive agility revealed that higher-skilled players have faster decision-making and change of direction compared with lower-skilled players. Serpell et al26 argued that high-performance players are able to identify key sport-specific kinematic cues earlier than low-performance players and therefore can react quicker. These predictive features in highly skilled players make it more difficult to tackle them in attacking play or make it easier for them to adapt to opponent movements in defensive play. Besides the requirement of high methodological quality studies on this skill, all forms of validity and reliability, and the underlying principle of the skill should be further investigated. With critical evaluation and expert consensus, a gold standard for reactive agility can be developed, which can be used as reference test for new developments in this game-specific skill. Ecological validity should be taken into account in these tests, with Serpell et al20 being promising by using videos of real-time game situations as cues to react on. The lack of ecological valid tests is also highlighted by the fact that no studies assessed validity in match play situations. These factors should be considered when new tests are developed or existing tests are adapted.

Limitations

Like all systematic reviews, the quality of this review is dependent on articles identified and included. MeSH terms were not used for all possible search terms in the search strategy of this review, which could have led to missed information. On this specific topic there were limited data available, which made it hard to draw strong conclusions. It can be argued that studies on sports comparable to rugby, like American and Australian football, should have been included, but inclusion of these sports would have implied unfair comparisons, based on significant differences in rules and gameplay. Furthermore, it should be considered that most articles assessed rugby league players. This can partly be explained by the relatively high amount of studies of Gabbett et al, which studied rugby league only. Possibly, this could have biased the results towards tests more focused on rugby league, without taking into account the minor differences between both rugby codes, like the contest after a tackle, points scored with specific actions and the ‘six tackle rule’. Another limitation is the inclusion of articles with athletes from all age ranges as this could have led to unfair comparisons. This was due to the limited articles available and the focus of this review to analyse all the literature available on this topic. Readers or coaches should note that when using information about psychometric data from game-specific tests in this review it needs to be established whether the participants involved in such reported studies share similar characteristics to the populations the results are applied to. As a last point, it should be considered for future studies if the COSMIN rating as being used in this review is the best way to assess the methodological quality of the studies. Potentially other, more correct analyses or definitions are available.

Future directions

For future studies, we recommend researchers to differentiate between playing positions (groups) and to study game-specific skills in situations that mimic actual game situations. Now most skills are assessed in isolated and standardised settings. However, based on the gameplay of rugby it is suggested that additional skills are important, for example, skill performance while being fatigued. Logically, the closed skill of, for example, passing is useful, but potentially passing skills while fatigued is even more important. As fatigue has shown to influence tackling ability, with fatigue resulting in progressive reductions in tackling technique, this could also be the case for passing skills.21 30 During most game phases, and especially the end of the second half, there is a level of fatigue among the rugby player passing the ball. A suggestion for testing these skills with and without the influence of fatigue could be to perform a passing test, both before and after a maximal endurance test. Furthermore, an additional skill that seems to be important is the ability to ‘read the game’. A player that reads the game well is better in avoiding situations with risk on being tackled, can better create attacking opportunities and can reduce situations of making passing or kicking mistakes. Moreover, a player can have an excellent reactive agility score in an isolated testing situation, but if the player makes wrong decisions during the game about where to run and constantly runs into ‘traffic’, the player has no benefit of his excellent reactive agility score. Potentially, a player with a lower reactive agility score, but with better ability to read the game, is more successful during a game. This is in line with the findings from Gabbett et al31, which found that rugby league players with poor reactive agility scores had a lower risk of injury. They suggested that these players might inadvertently avoid the heavy collisions that result in injury, or at best result in partial contact that does not result in exposure to the full force of a tackle, because of their low reactive agility. It can be that this was also due to the better ability to read the game, which makes this an interesting variable to assess in further studies.

With these findings taken into consideration, we recommend a change of direction in game-specific skills testing in rugby before tests are further developed. Because there are no high methodological quality studies, which analyse validity and reliability, there is a need for more methodologically rigorous studies, specifically designed to evaluate psychometric properties of game-specific skills tests. Otherwise it is unclear to what extent the findings of studies assessing game-specific skills are influenced by the test procedures followed. Because of the limited studies on validity found, there should be more studies focusing on the validity of these tests to be able to place the results in the context of predictive value for talent identification. Furthermore, it should be analysed if the currently used game-specific skills tests are clear reflections of the requirements of the game. Potentially a new view on testing and new categories of testing should be developed. Only after addressing these features in game-specific skills testing we recommend to go on with analysing game-specific skills in the way it is currently done. After this consideration, game-specific skills tests can be included as objective tests to complement talent-identifying programmes in rugby clubs.

Conclusion

Articles assessing psychometrics of game-specific skills tests mainly indicated moderate to high evidence on reliability and validity measures, but the studies lacked methodological quality. Future research should focus on high methodological quality studies to indicate valid and reliable game-specific skills tests in rugby, preferably focused on specific playing positions. Reactive agility is the most studied and promising skill, and should be further investigated to find the best testing procedure and create a gold standard. This systematic review can help researchers, scouts and coaches in the process of development or inclusion of objective game-specific skills tests in talent identifying programmes in rugby clubs, which can function as an objective backbone for talent identification by scouts and coaches.

Table 4

Validity outcomes of studies included in the second step

Author	Sport	Domain	Subjects (N)	Known groups	Variable	Result	Evidence	COSMIN
Green25	Rugby union	Reactive agility	28	Academy vs club	Reactive agility speed (s)	ES=1.14	++	Fair
Gabbett24	Rugby league	Reactive agility	42	First grade vs second grade	Movement time (s)	ES=0.73	+	Fair
					Decision time (ms)	ES=0.54	+
					Response accuracy	ES=0.34	–
Serpell26	Rugby league	Reactive agility	30	Elite players vs elite youth players	Total agility time (s)	ES=0.56	+	Fair
					Perception and response time (s)	ES=0.68	+
Pienaar10	Rugby	Rugby skills	218	Experienced vs non-experienced	Passing for distance (m)	ω²=32.4	+	Fair
					Passing for accuracy, 7 m	ω²=50.7	+
					Passing for accuracy, 4 m	ω²=10.6	–
					Running and catching	ω²=23.3	+
					Kick for distance (m)	ω²=29.4	+
					Kick-off for distance (m)	ω²=13.9	–

‘++’, strong evidence; ‘+’, moderate evidence; ‘–, weak evidence; COSMIN, Consensus-based Standards for the selection of health Measurement Instruments; ES, effect size.