Investigating the test-retest reliability of Illinois Agility Test for wheelchair users.

The Illinois Agility Test (IAT) is a standard agility course used to assess and train able-bodied athletes as well as wheelchair-sport athletes and rehabilitation of wheelchair users. It has been shown to be a reliable and valid tool to assess the able-bodied population, but the reliability of this test for assessing wheelchair propulsion has never been shown. The purpose of this study is to investigate the test-retest reliability of IAT to assess wheelchair propulsion. In this paper, the test-retest reliability of using IAT for wheelchair users is found for peak and average velocity, acceleration, tangential and total force of the push, each for the left and the right wheel. Each of these variables was found for thirty-two decisive points throughout the IAT path. The Intra-class Correlation Coefficient (ICC) was found to be very strong for 15 out of 16 variables. The average ICC was 89% and the average 95% confidence interval was [44% 96%]. In addition, thirty-seven other significant propulsion parameters were found that are clinically important, such as the number of pushes participants take to go around cones on the right, relative to turning around the cones on the left. Also, all thirty-seven variables were compared between the two sessions using four separate MANOVAs; the results showed no significant difference between IAT performed in the two sessions which were at least one week apart. This, in turn, backs the reliability of IAT for wheelchair users. These results are sufficient evidence to show that IAT is a reliable tool to test wheelchair agility for fifteen variables tested for non-wheelchair users. Since experienced wheelchair users are much more consistent in wheelchair propulsion compared to non-wheelchair-users, the results of this study show that IAT can be used as a reliable tool to assess and train wheelchair users, both for clinical and athletic applications.

Introduction

Wheelchair propulsion requires an intense effort that wheelchair users repeatably undergo every day. The intensity of effort is firstly because the efficiency of wheelchair propulsion is almost half the efficiency of walking (~14% [1] vs 26 to 27% [2]), and secondly because wheelchair users use their arms instead of their legs to ambulate which are not evolved for that purpose [3]. The shoulder joint is therefore vulnerable when wheelchairs are used as a primary method of ambulation. In fact, about 20% of wheelchair users develop secondary injuries in their upper extremities around five years after starting to use a wheelchair [4]. This percentage increases to 46% about 20 years after the injury [4, 5]. This mainly happens because of the excessive forces exerted on the structures around the shoulder during wheelchair propulsion over rough terrains and also manoeuvring [6].

Manoeuvring is performed extensively for indoor wheelchair use [7] and is responsible for a greater portion of the excessive loading on upper extremities [8] and yet it has seldom been studied in the literature. Having standard and reliable tests for measuring wheelchair manoeuvres will help to develop a more sophisticated understanding of the biomechanics of manoeuvring.

The Illinois Agility Test (IAT) is a standard agility test that has been used for both training and assessment of able-bodied athletes for many years [9, 10]. It was introduced for measuring multidirectional agility for different sports [9, 11]. For instance, IAT has been used to test the effect of strengthening exercises and also to test athletes’ agility in soccer [12]. However, some researchers have used IAT for wheelchair athletes as well. William [13] tried to find physiologic determinants to assess the training of wheelchair basketball players [14] and Usma-Alvarez, et al. have considered developing some outcomes for IAT that are suitable in assessing wheelchair rugby.

We could not find any publications that have shown the reliability of an agility test for wheelchair users, although there are a few papers that have investigated the reliability of different agility tests for the able-bodied population. Namely, within-day reliability of IAT for 66 able-bodied semi-professional rugby players has been shown to be ICC = 86% [15]. In another study, the reliability of IAT was found on a sample of 97 able-bodied young military service men: ICC = 99% inter-rater reliability and ICC = 68% test-retest reliability [9]. Also, an ICC of 96% was reported elsewhere for IAT for a sample of 89 able-bodied sportsmen from football, handball, and rugby players when repeating the test on different days [11].

Williams [14] has investigated the construct validity of three different agility tests, including IAT, for athlete wheelchair basketball players. They compared the timed performance of a group of elite athletes to a group of competitive players and concluded the test was valid as there was a significant difference in the time to finish the test. They also found excellent test-retest reliability with correlations above 0.9 between 2 trials performed in one session. However, they only measured the time to finish the test and not other important biomechanical factors. This is important because when using an assessment tool for measuring an outcome we need to be confident that this tool will create similar results if used sometime later [16].

An important factor when testing agility is how fast the participant could finish the agility path. This is the reason that the primary output of all agility tests is the time needed to finish the task. However, a good agility test should be designed in a way to measure the acceleration of the participant, as changes in the direction, which are the main parts of agility tests, are more correlated with acceleration than speed [17]. Although completion time is not a great concern when studying wheelchair maneuvering, the acceleration assessment makes agility tests suitable for the purpose of this study, as it is closely related to maneuverability for wheelchair users. The purpose of this study was to investigate the test-retest reliability of IAT in order to introduce a reliable standard agility test that can be used not only for studying wheelchair maneuvering but also for training athlete wheelchair users.

Methods

Sample size calculation

To calculate a sample size that provides an acceptable power for finding the reliability of IAT, we used Eq 1 that is introduced by Shoukri et al. [18]: Where K is sample size, z is the value of the standard normal distribution corresponding to the level of significance (α) of this study, z is the value of the standard normal distribution corresponding to the power of this study (1-β), μ is mean of the distribution of reliability, ρ0 is the minimum reliability coefficient we can tolerate, and ρ1 is the estimated reliability coefficient of the measure. This equation calculates the needed sample size when the number of repetitions of the test per subject is two.

μ(ρ) is calculated using Eq 2: The minimal acceptable reliability was set as 0.6, as it is the minimum acceptable reliability coefficient for many clinical investigations [18]. So, for having the significance level of α = 0.05, β = 0.2 (power of 80%), ρ0 = 0.6 and ρ = 0.9, Eq 1 would give 11.63. Thus, the sample size needed for this study is obtained as 11 persons.

Subjects

Eleven healthy able-bodied subjects volunteered to participate in the study (6 women and 5 men: (mean ± SD) age = 27.9 ± 4.74 years, weight = 63.5 ± 10.96 Kg, and height = 1.69 ± 0.1 m). The ethics application for this study was approved by University of Alberta Research Ethics Board, Project Name “Can simulation be used to accurately represent real-life wheelchair propulsion?”, No. Pro00003315 _AME5, Date: 9/25/2015. Subjects signed a written informed consent form, a video release form, as well as a “PAR_Q and You” (Physical Activity Readiness Questionnaire) prior to participating. participants were excluded if they put “yes” for any of the questions in “PAR_Q and You” or if they had a musculoskeletal injury that affects normal wheelchair use, exercise-induced asthma, or heart disease. Participants were a convenience sample recruited from graduate or visiting students of the corresponding author’s university. One participant was Korean, two were Hispanic, 5 were White, and 3 were Persian.

Experiment content

Subject participation was according to an experimental protocol that was approved by the Human Research Ethics Board of the University of Alberta (ID: Pro00003315_AME6). Each subject had two sessions of testing that were at least one week apart (mean± SD = 10±4.1 days); In the first session, they performed four trials and at the second session they performed eight trials of IAT (Fig 1). Subjects were given enough time between the trials (at least three minutes) in both sessions to rest. The next trial did not start until the participants declared they had had sufficient rest and their heart rate returned to their resting heart rate.

Fig 1

Flow chart of the experiment including the measured variables.

IAT is an obstacle course consisting of 8 cones that are positioned as shown in Fig 2. Participants were asked to run the path (shown in Fig 2) as quickly as they could. IAT was originally designed to start from the prone position; the person performing the test was to lay down in the prone position and at the beginning of IAT he/she would quickly stand up and start running [9]. However, IAT sometimes has been used without the prone start [15].

Fig 2

Illinois Agility Test [9].

For this study, a standardized version of IAT was used [9], which is overall a 10 by 5 meters court; the cones on the horizontal lines are placed 2.5 meters in between, and on the vertical line they are positioned 3.3 meters apart (Fig 2). Since completion time is not a great concern for wheelchair users when accomplishing wheelchair manoeuvres, participants were instructed to propel their wheelchair through the IAT path at their comfortable speed. They were asked to read their heart rate just before and just after finishing the test, using a Fitbit Charge HR (wireless heart rate and activity wristband). The wheelchair that was used for the experiments was Quickle GP and was equipped with two 24-inch SMARTWheels, the validity of which for measuring propulsion forces has been shown by Asato et al. [19]. The subjects were video-captured while performing the test.

Data analysis

Statistical analyses were performed using SPSS (IBM® SPSS® Statistics Premium GradPack 24 for Windows) and the alpha level (critical significance) was set to 0.05.

At each trial, velocity and force information were recorded using SMARTWheel. Then, using a custom code the authors wrote in MATLAB, two categories of parameters were derived for each trial: data series and single-value parameters.

Data-series parameters

Sixteen variables were derived at thirty-two measurement points (Fig 3) throughout the IAT path: peak and average magnitude of tangential and total force, velocity, and acceleration, both for the left and the right side (Fig 1). Twenty-nine measurement points are shown in Fig 3A. Three other points (points number 3,7, and 28) pertained to the last push in the Straight-Line Propulsion (SLP) parts of IAT, just before turning pushes start (Fig 3C and 3D). Since participants could start turning anywhere between the two neighboring measurement points, e.g. 2 and 4, depiction of these points (3, 7, and 28) was not feasible. The guidelines used to find the measurement points are presented in Table 1.

Fig 3

The measurement points considered throughout the IAT path; (a) shows the full path, (b) shows the manoeuvring part of the path, and (c) and (d) specify the straight-line propulsion part.

Table 1

Method of finding the location of measurement points.

Point	Method of finding the coordinates
1	The first data set
4	At the time the yaw angle of body passes -7 deg*
2	When y equals to average of y coordinates of point 1 and 4
5	At the time the yaw angle of body passes -173 deg
8	At the time the yaw angle of body returns to -173 deg and passes it
6	When y equals to average of y coordinates of point 5 and 8
9,11,13,15,17	When x equals to 0**
10,12,14,16	At the extrema of y

Only information for half of the points are provided here; the other half of the points use similar methods, due to symmetry. For point numbers please refer to Fig 3A.

* (-7) degrees is the inverse tangent of ((2.5m/2)/10m) and is the threshold after that we are sure turning has been started.

** The origin of the coordinate system is placed at the middle cone at the starting and finish line.

To find the measurement points, we could either use Optitrack (motion analysis) or regular cameras to visually check the video recordings. Optitrack (NaturalPoint, Inc., USA, ARENA Motion Capture- Software: Motive_Tracker- Hardware: Prime 17W and S250e cameras) is a very high precision motion capture system and provides very accurate data, but due to prolonged nature of the tests and the number of repetitions of the tests, using Optitrack required an unrealistic effort to track each test frame by frame and to run custom algorithms to find the measurement points. Whereas, using video recordings it was possible to reduce the analysis time by a factor of 5. Even better, it prevented possible misinterpretations of participant’s movement., e.g. when the participant makes a substantial turn but is still on SLP and has not yet started the real turn.

To confirm that the video recording method was sufficiently accurate, for one subject, the test was recorded using both Optitrack and regular cameras. Then, the measurement points were found using the trajectory obtained from the Optitrack data and the guidelines presented in Table 1. These points were also obtained visually by reviewing the video record of the test frame by frame (frame rate of 30 fps) using Avidemux 2.6.13. The points found using each method were then analyzed using SPSS to find the agreement ICC and Pearson Correlation Coefficient (PCC). The data obtained using the two techniques are shown in Fig 4. The PCC and ICC were found to be 100% between points obtained from Optitrack and camera (Confidence Interval (CI) = 99.8%-100%). Therefore, the video-tape method was selected in order to reduce analysis time.

Fig 4

The time one subject passed the measurement points obtained from Optitrack and camera records.

After finding all the measurement points for all variables and all participants-trials, variables pertaining to each trial were averaged over all subjects. Then, the data for 32 points of measurement for each variable were averaged over all trials in each session. As a result, 32 averaged measurement points were achieved for each session-variable.

Data were checked for normality first, using Shapiro-Wilk test. Data analysis was performed using SPSS and Shapiro-Wilk test is recommended for testing normal distribution in SPSS [20]. Since parametric methods have higher research power than non-paramedic methods, where data did not follow normal distribution, we attempted normalization using data transformation. Considering the relatively small sample size here, we did not want to compromise the power of the study by using non-parametric methods. Therefore, in the event a variable had a non-normal distribution, the Two-Step [21] method was used to transform the data of that variable for both sessions. In the Two-Step method, we first transform the original values to fit uniform distribution and then analyzed the normalized data.

The test-retest reliability of the 16 variables was found using absolute agreement Intra-Class Correlation coefficient (ICC model (2,4)) between the scores of the first and second sessions. Also, the Standard Error of Measurements (SEM) was determined for each variable, using this formula [22]:

Where SD is the pooled standard deviation of all data for each variable and ICC is the intra-class correlation coefficient obtained for that variable.

Single-value parameters

Thirty-seven variables were derived as single-values for each trial-subject, which are significant parameters in the biomechanics of wheelchair propulsion, and decisive indices of similarity of the tasks repeated over different sessions. These variables are classified into four groups:

Group I: time used to finish:

The whole trial (t_Tot (s))

The straight-line part (t_SLP (s))

The manoeuvring part (t_Mnv (s))

Group II: Participant’s average cadence (frequency of pushing) when performing one IAT trial, for the right and the left wheel (Cadence (1/s).

Group III: Push parameters for the right and the left side:

Push starting angle (Starting Angle (Deg))

Push length (Push Length (Deg))

Number of pushes used to perform the first U-turn in IAT (U-TURNS1)

Number of pushes used to perform the second U-turn in IAT (U-TURNS2)

Number of pushes used to perform the SLP part of IAT (ALL-SLP)

Number of pushes used to go around the cones on the left (GOING-AROUND-CONES-ON-LEFT)

Number of pushes used to go around the cones on the right (GOING-AROUND-CONES-ON-RIGHT)

Group IV: Sum of forces applied by the wheelchair user on the right and the left side:

Sum of tangential forces applied on the rim during the whole path (SigmaFt_(TOT) (N))

Sum of tangential forces applied on the rim during the SLP part (SigmaFt_(SLP) (N))

Sum of tangential forces applied on the rim during the manoeuvring part (SigmaFt_(MNV) (N))

Sum of absolute tangential forces applied on the rim during the whole path (SigmaFt_ABS(TOT) (N))

Sum of absolute tangential forces applied on the rim during the SLP part (SigmaFt_ABS(SLP) (N))

Sum of absolute tangential forces applied on the rim during the manoeuvring part (SigmaFt_ABS(MNV) (N))

Sum of total forces applied on the rim during the whole path (SigmaFtot_(TOT) (N))

Sum of total forces applied on the rim during the SLP part (SigmaFtot_(SLP) (N))

Sum of total forces applied on the rim during the manoeuvring part (SigmaFtot_(MNV) (N))

Data analysis for the single-value parameters was also performed using SPSS. Firstly, for each subject, all variables were averaged over all trials in each session. Secondly, these averaged data were checked for normality. Thirdly, variables that did not have normal distribution were transformed using the Two-Step method [21]. Fourthly, one one-way MANOVA (Multivariate analysis of variance) was performed for variables belonging to each group (four MANOVAs in total). To check the assumption of homogeneity of variances (a necessary condition for performing MANOVA [23]), when computing Box's Test of equality of covariance matrices was not possible because of non-singular cell covariance matrices in the data (being caused by occasional missing data), the inter-item covariance matrix of all variables belonging to that group was developed and variances were visually checked for having close values.

Results

Data series parameters

For data series variables, 10 out of 32 (16 variables for each session) needed a transformation to comply with normality. Since some of these non-normal variables belonged to only one session and some of them were repeated in both sessions, 14 variables were transformed in total. To check the agreement of data, the comparison was made between the two data series which were based on the same construct: either the raw value or the transformed function of it.

Table 2 details the results achieved for the data series parameters. The average ICC of these variables was 89%. Also, the average 95% confidence interval was (44%-96%) (See Table 2 for details).

Table 2

ICC, lower and upper band of confidence interval, as well as standard error of measurement (SEM) for the sixteen variables of data series parameters: Average and peak values of velocity, acceleration, tangential and total force, each for the left and right.

	Mean	SD	ICC	CI_LB	CI_UB	SEM
Avg Vel_L (m/s)	0.77	0.025	0.88	-0.07	0.97	0.01 m/s
Peak Vel_L (m/s)	0.86	0.02	0.83	-0.07	0.96	0.01 m/s
Avg Ft_L (N)	6.16	32.09	0.96	0.82	0.99	6.42 (N)
Peak Ft_L (N)	29.09	146.14	0.96	0.71	0.99	32.68 (N)
Avg Ftot_L (N)	16.07	8.28	0.89	0.69	0.96	2.75 (N)
Peak Ftot_L (N)	39.82	56.83	0.94	0.54	0.98	15.04 (N)
Avg Acc_L (m/s2)	-0.006	0.009	0.98	0.97	0.99	0.001 m/s2
Peak Acc_L (m/s2)	0.295	0.016	0.93	0.04	0.98	0.006 m/s2
Avg Vel_R (m/s)	0.724	0.026	0.96	0.19	0.99	0.01 m/s
Peak Vel_R (m/s)	0.82	0.019	0.91	-0.06	0.98	0.01 m/s
Avg Ft_R (N)	2.43	35.18	0.98	0.95	0.99	6.09 (N)
Peak Ft_R (N)	22.45	121.32	0.95	0.77	0.98	29.72 (N)
Avg Ftot_R (N)	14.93	5.81	0.42	-0.22	0.76	4.43 (N)
Peak Ftot_R (N)	37.02	71.01	0.73	-0.15	0.93	37.58 (N)
Avg Acc_R (m/s2)	-0.019	0.013	0.98	0.97	0.99	0.002 m/s2
Peak Acc_R (m/s2)	0.26	0.019	0.97	0.93	0.98	0.004 m/s2

Single-value parameters

The results of single-value parameters of eleven subjects are presented in this part. After checking for compliance with normal distribution, 15 out of 37 variables (38%) were shown to need variable transformation, which was done using the Two-Step method [21]. The results of each group of single-value parameters are presented under the respective headings.

Group I: Time

The statistical details of results for the time taken for subjects to finish the whole IAT path and the SLP and manoeuvring part of it are presented in Table 3.

Table 3

Detailed statistical results for the time taken to finish different tasks.

	Mean (SD) (Sec)		F	Effect size (η2)	P-Value
	Session 1	Session 2
Whole path	100.37 (12.41)	90.14 (21.57)	1.584	0.081	0.224
Manoeuvring part	46.54 (7.07)	41.57 (9.72)	1.636	0.083	0.217
SLP part	46.58 (16.66)	39.75 (10.12)	2.287	0.113	0.148

The significance of Box's test of equality of covariance matrices for these variables was obtained as 0.307 and this does not provide enough evidence to reject the null hypothesis of Box’s test which says the covariance of matrices are equal, so this MANOVA assumption is also met. Df2 for this group of parameters was 19.

The significance of MANOVA results was 0.441 (Pillai's Trace method) which cannot reject the null hypothesis that says the time taken for the subjects to finish different parts of IAT is statistically the same at the two sessions (similarity was not rejected- favorable for our study).

Group II: Cadence

The mean and standard deviation of the left and right cadence is presented in Table 4. Box's test of equality of covariance matrices was not significant (sig = 0.564) which means the assumption of homogeneity of variances and covariances is met. Df2 for this group of parameters was 18. Pillai's Trace significance was 0.428 which again shows the MANOVA results for cadence were not significant (P-value<0.05) and therefore the similarity of the push frequencies of the left and right side was not rejected statistically between session 1 and session 2.

Table 4

Detailed statistical results for push frequency (cadence) for the left and right wheel.

	Mean (SD) (1/Sec)		F	Effect size (η2)	P-Value
	Session 1	Session 2
Left Cadence	0.86 (0.040)	0.87 (0.041)	0.269	0.014	0.610
Right Cadence	0.85 (0.035)	0.84 (0.038)	0.083	0.004	0.776

Group III: Push parameters

To find the main results obtained for push parameters during IAT task performed by eleven able-bodied wheelchair users at two different days see Table 5. For this group of parameters, the Box's test of equality of covariance matrices returned no results due to non-singular cell covariance matrices in the data which are caused by missing data of one person in the first session. Thus, the inter-item covariance matrix of all variables belonging to that group was visually checked for having close values. It was observed that in 85% of all cases the data were fairly close and therefore it was concluded that the assumption of homogeneity of variances and covariances is met. Df2 for this group of parameters was 17.

Table 5

Detailed statistical results for push parameters of the left and right wheel.

	Mean (SD)		F	P-Value	Effect size (η2)
	Session 1	Session 2
Left Starting Angle* (Deg)	-9.72 (0.23)	-9.29 (0.8)	1.709	0.209	0.091
Right Starting Angle* (Deg)	-9.6 (0.26)	-9.5 (0.39)	0.424	0.524	0.024
Left Push Length (Deg)	18.63 (3.74)	21.12 (4.66)	2.394	0.140	0.123
Right Push Length (Deg)	17.59 (3.96)	20.87 (5.11)	3.028	0.100	0.151
Left U-turn 1	4.9 (0.76)	4.2 (0.91)	4.930	0.040	0.225
Right U-turn 1	4.2 (1.3)	3.7 (0.96)	1.742	0.204	0.093
Left U-turn 2	5.1 (1.53)	4.6 (1.06)	0.740	0.402	0.042
Right U-turn 2	4.9 (1.84)	3.8 (1.54)	4.817	0.042	0.221
Left ALL-SLP	37 (9.3)	31.5 (8.85)	2.826	0.111	0.143
Right ALL-SLP	36 (8.5)	31.6 (8.89)	2.236	0.153	0.116
Left wheel-going around cones on left side	4.6 (1.08)	3.8 (0.92)	3.965	0.063	0.189
Right wheel-going around cones on left side	4.4 (0.97)	3.9 (0.96)	1.307	0.269	0.071
Left wheel-going around cones on right side	4.7 (1.12)	4.2 (1.08)	1.242	0.281	0.068
Right wheel-going around cones on right side	4.5 (0.86)	3.9 (0.98)	2.470	0.134	0.127

*-0 degree corresponds to the person’s longitudinal axis and negative angles represent the participants extending their hands backward.

Regarding the MANOVA results for this group, Pillai's Trace significance was obtained as 0.378 which means, here again, there is no statistically significant difference between push parameters in session one compared to session 2.

Group IV: Sum of forces

Table 6 details the main outcomes obtained regarding the sum of forces applies on the right and left wheel when performing IAT by non-experienced wheelchair users.

Table 6

Detailed statistical results for push parameters of the left and right wheel.

	Mean (SD) (KN)		F	P-Value	Effect size (η2)
	Session 1	Session 2
Left SigmaFt_Tot	170.84 (101.54)	111.39 (39.21)	0.265	0.614	0.017
Right SigmaFt_Tot	55.86 (14.05)	51.77 (17.28)	0.134	0.720	0.009
Left SigmaFt_Mnv	67.51 (46.59)	39.12 (21.45)	0.492	0.494	0.032
Right SigmaFt_Mnv	36.07 (5.87)	34.13 (7.53)	0.251	0.623	0.016
Left SigmaFt_SLP	77.23 (42.3)	55.95 (15.3)	0.081	0.780	0.005
Right SigmaFt_SLP	30.71 (5.2)	30.73 (11.65)	0.000	0.983	0.000
Left SigmaFt_ABS(Tot)	252.92 (69.18)	224.67 (28.16)	0.521	0.482	0.034
Right SigmaFt_ABS(Tot)	192.56 (26.4)	196.86 (33.39)	0.030	0.866	0.002
Left SigmaFt_ABS(Mnv)	124.76 (27.41)	115.6 (17.03)	0.183	0.675	0.012
Right SigmaFt_ABS(Mnv)	93.26 (15.61)	91.5 (14.48)	0.074	0.789	0.005
Left SigmaFt_ABS(SLP)	93.18 (35.15)	79.51 (14.54)	0.079	0.782	0.005
Right SigmaFt_ABS(SLP)	62.82 (6.81)	66.15 (10.54)	0.217	0.648	0.014
Left SigmaFtot_Tot	419.33 (200.95)	323.1 (199.26)	0.101	0.756	0.007
Right SigmaFtot_Tot	350.81 (205.76)	393.4 (108.01)	0.027	0.872	0.002
Left SigmaFtot_Mnv	206.21 (86.46)	164.16 (92.51)	0.000	0.999	0.000
Right SigmaFtot_Mnv	164.79 (93.91)	181.55 (49.65)	0.316	0.582	0.021
Left SigmaFtot_SLP	161.26 (96.53)	118.77 (80.89)	0.178	0.679	0.012
Right SigmaFtot_SLP	141.18 (83.56)	155.15 (50.64)	0.000	0.998	0.000

Similar to parameters of group 3, the Box's test of equality of covariance matrices returned no results, and checking the inter-item covariance matrix of all variables showed that 93.2% of them had relatively close values. Therefore, it was concluded that the assumption of homogeneity of variances and covariances was met. Df2 for this group of parameters was 15.

Also, in the MANOVA results for this group, Pillai's Trace significance was 0.356 which means the two sessions were not statistically different in terms of the sum of forces applied on the wheels.

Discussion

In this study, we recruited eleven able-bodied subjects to participate in two sessions of wheelchair manoeuvring with the aim of studying the reliability of a standard agility test adapted for wheelchair users. A comprehensive assessment was performed considering fifty-three parameters in total, out of which, sixteen were obtained as data series throughout the IAT. Considering ICC = 0.6 as the minimum acceptable ICC [18], all the data series variables studied were shown to be reliable when performing IAT by wheelchair users, except for the average total force of the right side. Hence, 94% of these variables were shown to have a good to excellent reliability. The average total force on the right wheel was the one with the lowest reliability (42%), while the same variable for the left wheel had very good repeatability (89%) and other force variables also had good to excellent repeatability. Therefore, we suspect that there may have been fundamental differences in the force application methods when participants repeated the test over two visits. Since the devices were calibrated before starting the experiments, we do not believe that there was an instrumentation issue. Furthermore, performing MANOVA on each group of single-value parameters showed that there were no statistically meaningful differences between all of these thirty-seven variables between sessions one and two, which, in turn, backs the reliability of IAT for wheelchair users.

It is worth mentioning that although the number of variables tested here is large, there is no need to use the Bonferroni correction, as this is used when we are worried that the “changes (differences) detected” are not real and could have occurred by chance. Whereas in this study, we were interested to show “no differences” among the two sessions. In fact, not considering the Bonferroni correction adds to the power of study here, as the null hypotheses are maintained by Alpha level of 0.05. Usually, the Bonferroni correction is used, to adjust the alpha level, so that a P-value (0.04, for example), does not result in claiming a significant change. Whereas in our case, the case is the opposite, and rejection by a lower alpha level is favorable, not restricting. Thus, using Bonferroni Correction is pointless, and even worse, it decreases the study power.

In this study, we recruited able-bodied subjects instead of experienced wheelchair users. This helped with the recruitment, but more importantly, supported the main purpose of this study which was studying the reliability of IAT rather than the wheelchair users themselves. Experienced wheelchair users are much more consistent in the way they individually propel their wheelchair compared to non-wheelchair-users. We believed that if we could show that non-experienced wheelchair users were very consistent in performing a wheelchair manoeuvring task when repeating it 10 days later (on average), this would provide stronger evidence that this test (IAT) can be used for both experienced and non-wheelchair users, as a reliable assessment tool.

Experienced wheelchair users are more efficient than non-wheelchair-users [24]. They tend to use longer pushes with lower peaks both to prevent overuse injuries and to increase efficiency [25]. Would this potentially increase inconsistency in the propulsion technic? It is unlikely, but further research is warranted recruiting real wheelchair users to be conclusive about the reliability of IAT for real wheelchair users.

Differences in wheelchair manoeuvring that may be associated with clinical factors (level of spinal injury, upper extremity injury) could be thought of as a possible source of propulsion variance the biomechanical parameters of interest during manoevering. However, these differences between participants do not violate our assumption as we believe within-subject consistency is higher among this population. Although the between-subject variance could be higher in wheelchair users, it does not harm our assumption that recruiting non-wheelchair-users provides stronger evidence for the reliability of IAT for wheelchair users.

The single-value parameters measured and analyzed for this study were selected based on clinical and practical relevance and importance. These measurements are frequently needed by both clinicians and engineers who work with or for wheelchair users. For instance, cadence and push length are important parameters that clinicians work on with newly injured wheelchair users [26], as they work effectively toward propulsion efficiency [27] which, in turn, plays a very important role in secondary injuries occurred to wheelchair users, as propulsions with higher efficiency would incorporate less total force to accomplish a given task [25], and that works against overuse of shoulder. Also, tangential and total forces applied on the rim are both effective on propulsion efficiency [25, 28] and determinative when measuring activities of wheelchair users [29]. Moreover, the starting angle is an important factor when adjusting the wheelchair axle for the stability of the wheelchair, and preventing secondary injuries [30]. The factors reported in this paper are therefore both clinically and technically informative and we observed that the comfortable speed when manoeuvring with a wheelchair was 0.85 m/s which is considerably lower than 1.1 m/s [1, 31] or 1.4 to 1.5 m/s (equal to 83.4 to 90.7 m/min) [32] which is the comfortable speed that has been reported before for performing straight-line wheelchair propulsion. Table 4 is also particularly useful as it displays valuable push parameters when performing IAT. One interesting observation in this table, for instance, is that when performing a rather complex manoeuvring task, participants did not show any indication of having more pushes on their dominant hand. However, although statistically the number of pushes in completing different parts of IAT was not different in session one compared to session two, we can see a consistent decrease in the number of pushes in session two relative to session one which could be clinically meaningful and a sign that participants used fewer pushes as they began to master the task.

Furthermore, Table 2 shows the average velocity, acceleration, and forces participants used in one average push. However, we have also reported the actual and absolute sum of forces applied by the participants for finishing different parts of IAT. Note that this is the summation of forces at all frames in each part that is being studied (240 Hz). The clinical importance of these parameters is that eventually, we are interested in the amount of force a wheelchair user needs to apply when performing different tasks. These data can be used as a reference for future works where having an estimate for forces applied during wheelchair propulsion is necessary, such as developing assistive devices, activity monitors, or injury prevention apparatus for wheelchair users. They can also be used where clinicians need to know which activities require more force and hence increase the risk of overuse injuries.

Conclusion

We were able to assess the reliability of a standard agility test, IAT, so that now it can be used in research studies focusing on wheelchair manoeuvring and also in training assessments of wheelchair-sport athletes.

We showed that 94% of the important biomechanical parameters are statistically reliable when wheelchair users perform IAT, and we furthermore showed that 37 other biomechanical variables do not statistically differ when the test is repeated after a reasonably lengthy period of time.

Other clinical or practical implications were also provided, such as the average speed of wheelchair manoeuvring which was 0.85, about 57% to77% of the average speed of SLP wheelchair propulsion reported elsewhere [1, 31, 32].

Future work on wheelchair manoeuvre can use findings of this study on the reliability of IAT. Also, the single-value parameters, in particular, can be helpful to clinicians and engineers in developing injury-prevention strategies and devices.

14 May 2020

PONE-D-20-08803

Investigating the test-retest reliability of Illinois Agility Test for wheelchair users

PLOS ONE

Dear Dr. Salimi,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

We would appreciate receiving your revised manuscript by Jun 28 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Yih-Kuen Jan, PhD

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. In your Methods section, please state where the participants were recruited for your study.

3. Thank you for stating the following financial disclosure:

'No- The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.'

At this time, please address the following queries:

Please clarify the sources of funding (financial or material support) for your study. List the grants or organizations that supported your study, including funding received from your institution.

State what role the funders took in the study. If the funders had no role in your study, please state: “The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.”

If any authors received a salary from any of your funders, please state which authors and which funders.

If you did not receive any funding for this study, please state: “The authors received no specific funding for this work.”

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

4. We noted in your submission details that a portion of your manuscript may have been presented or published elsewhere:

'This paper is prepared from the results of my PhD thesis and is included as Chapter 7 in my dissertation (Salimi, 2017).'

Please clarify whether this publication was peer-reviewed and formally published. If this work was previously peer-reviewed and published, in the cover letter please provide the reason that this work does not constitute dual publication and should be included in the current manuscript.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: No

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

Reviewer #2: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: No

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: General Comments:

•This paper investigated maneuverability in manual wheelchair users using a validated agility (IAT). This test has been validated in able-bodied athletes, and one study validated it time to finish in manual wheelchair athletes. This study should clarify its intent to validate biomechanical factors used within the test; not specifically the test, itself.

•Numerous grammatical, citation, and reading flow issues throughout. Several instances throughout the text where sections can be shortened and more “to the point.”

•Authors should reevaluate the statistical tests employed in this study and the interpretations that follow.

Introduction:

•Line 52: Sentences with only one supporting reference should acknowledge the reference at the END of the sentence rather than in the middle of the sentence.

•Lines 50-52: Reference is needed for these statements.

•Lines 55-57: Consider revising to read “…excessive forces exerted on the structures around the shoulder during wheelchair propulsion…” Reference is needed for this statement.

•Line 71, “We could not find any publications that have shown the reliability of an agility test for wheelchair users” is contradicted by Lines 79-84. Williams et al. did investigate the validity of IAT within the context of time to finish, and deemed it validated. While it may be true that they did not investigate the validity of “other important biomechanical factors,” it is important to remember why the IAT was created and how it is used: agility (presumably determined by time to finish as mentioned in Lines 88-90). It appears that this paper aims to validate “other important biomechanical factors” using the IAT; this should be clearer in the Title, Abstract, and Introduction as the main outcome of the IAT (i.e. agility) has already been validated.

•Line 93: Specify “the latter characteristic”

•Lines 94-97: Consider revising the purpose statement based on the fourth comment above.

Methods:

•Lines 125-127: Is the Fitbit Charge HR a validated tool for measuring heart rate? If so, this tool may be useful in the able-bodied manual wheelchair users in this study; however, heart rate is not a validated metric in other manual wheelchair users, such as individuals with spinal cord injury. Translation of this methodology to manual wheelchair users with spinal cord injury/disease may not be appropriate.

•Lines 135-138: Error message should not likely be included in text (“Error! Reference source not found”)

•Line 139: Numbers 3,7, and 28 are not pictured in Figure 3. Is there a reason for this?

•Lines 148-172: Explanation and justification for using standard camera video over Optitrack should be much shorter. Additionally, this decision was based off one subject’s data. Why were additional subjects’ data not included to make this decision more robust?

•Lines 177-179, 226-228: How were data assessed for normality? If data did not fit a normal distribution, why were data transformed to fit a normal distribution instead of using non-parametric statistical tests? Briefly describe the Two-Step method for readers who are not familiar with this test.

•The same study subjects performed the IAT at two timepoints roughly one week apart. Why was a MANOVA test conducted instead of a repeated measures MANOVA?

•No mention of significance criteria; alpha/p-values, effect sizes, etc.

Results:

•Line 238: Sentence should not start with a number.

•Line 242: Consider using “raw value” in place of “real variable.”

•The Results section should be written in a more technical manner using p-values, F-statistics, degrees of freedom, etc.

•The Results section should focus on the statistical outputs and interpretation of these outputs should be reserved for the Discussion section.

Discussion:

•Line 308: Sentence should not start with a number.

•Line 317: Careful using definitive terminology like “confirms” when interpreting results.

•Lines 318-328: Authors should reexamine their interpretation of a Bonferroni correction. Lines 318-319 suggest it IS NOT advantageous for this study to use a Bonferroni correction; however, in Lines 325-326, the authors claim IT IS advantageous for this study to use a Bonferroni correction. A Bonferroni correction adjusts the p-value used to determine significance based on the number of dependent variables.

•The methods of this paper are designed to control for Type 1 Error (rejecting a true null hypothesis); however, it does not control for Type II Error (accepting a false null hypothesis). This should be addressed.

•Lines 331-337: Research has found that pediatric and adult manual wheelchair users with pediatric-onset spinal cord injury employ highly variable upper extremity movement. While the exact mechanism for this is unknown, some attribute it to adaptation to prevent overuse injury during wheelchair propulsion. The authors’ interpretation that experienced manual wheelchair users use “more consistent” mechanics based on experience may not always be true.

•Line 349: Many of these metrics are determined as “clinically meaningful.” How so? Provide rationale or references that mention the metrics used in this study as “clinically meaningful.”

•Do the authors have any directions for future work based on these results?

Conclusion:

•This section should “wrap-up” the manuscript and highlight the results of this study. Consider adding a few more sentences to bring the manuscript to a succinct ending.

Reviewer #2: It is inappropriate to use Null Hypothesis Significance Testing to argue for equivalence or equality. As it is typically used (i.e., when the researcher hopes to reject the null hypothesis), NHST is designed to make it difficult for the researcher to get what they want. When the researcher is hoping to 'prove' the null, however, as is the case here, achieving the desired result using NHST is quite easy: just design a poor and/or underpowered study. Failing to reject the null hypothesis means either: 1) that the null is true (in the case of this study, the repeated measurements are equal), or 2) the null is false (the repeated measurements are not equal) but the experiment doesn't have enough power to reject it. In order to arrive at the first conclusion (equality) the authors must provide evidence against the second (an underpowered study). Unfortunately, the authors provide no evidence of high statistical power. In fact, using only 11 subjects in the study makes it likely that the study is underpowered. In order for the results of the study to be meaningful, I strongly suggest that the authors forgo the use of NHST-based procedures such as MANOVA to determine equivalence in favor of more appropriate methods such as Bayesian methods, confidence intervals (Loftus, 1996; Aberson, 2002), or noninferiority testing (Streiner, 2003). If the authors insist on using NHST to claim evidence of equality, I view it as critical to demonstrate that the study as ran had high power. In addition, effect size measures, F statistics, and their associated degrees of freedom should be included for all MANOVAs.

Aberson, C. (2002). Interpreting null results: Improving presentation and conclusions with confidence intervals. https://www.jasnh.com/a6.htm

Loftus, G. L. (1996). Psychology will be a much better science when we change the way we analyze data. Current Directions in Psychological Science, 5, 161-171.

Streiner, D. L. (2003). Unicorns do exist: A tutorial on "proving" the null hypothesis. Canadian Journal of Psychiatry, 48, 756-761.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Brooke A. Slavens, PhD

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

28 Jun 2020

Editor:

1 1. PLOS ONE's style requirements, including those for file naming Will do.

2 2. In your Methods section, please state where the participants were recruited for your study. Added.

3 3-a. Please clarify the sources of funding (financial or material support) for your study. List the grants or organizations that supported your study, including funding received from your institution. Canadian Foundation for Innovation (CFI 30852 Leading Edge Fund 2012) infrastructure grant

4 3-b. State what role the funders took in the study. If the funders had no role in your study, please state: “The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.” The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

5 3.c. If any authors received a salary from any of your funders, please state which authors and which funders. No salary provided to any authors from this grant

6 3-d. If you did not receive any funding for this study, please state: “The authors received no specific funding for this work.” Please include your amended statements within your cover letter. The authors received no specific funding for this work. The CFI funding provided the capital equipment for the project.

7 4. We noted in your submission details that a portion of your manuscript may have been presented or published elsewhere. Please clarify whether this publication was peer-reviewed and formally published. If this work was previously peer-reviewed and published, in the cover letter please provide the reason that this work does not constitute dual publication and should be included in the current manuscript. My thesis chapter was not formally peer reviewed; it was examined for the purposes of meeting the requirements of a PhD. This paper is based on a chapter from my PhD thesis, and although it was examined, this did not constitute peer review and no specific recommendations were made at that time.

Reviewer #1:

8 This paper investigated maneuverability in manual wheelchair users using a validated agility (IAT). This test has been validated in able-bodied athletes, and one study validated it time to finish in manual wheelchair athletes. This study should clarify its intent to validate biomechanical factors used within the test; not specifically the test, itself. With all due respect, this study aims in assessing the “reliability” of IAT, rather than validity. The paper you mentioned had checked the validity, which is out of scope of this study.

9 •Numerous grammatical, citation, and reading flow issues throughout. Several instances throughout the text where sections can be shortened and more “to the point.” Revised.

10 •Authors should reevaluate the statistical tests employed in this study and the interpretations that follow. Applied. Details are provided below, where questions point to details.

Introduction:

11 •Line 52: Sentences with only one supporting reference should acknowledge the reference at the END of the sentence rather than in the middle of the sentence. Displaced.

12 •Lines 50-52: Reference is needed for these statements. Added.

13 •Lines 55-57: Consider revising to read “…excessive forces exerted on the structures around the shoulder during wheelchair propulsion…” Reference is needed for this statement. Done.

14 •Line 71, “We could not find any publications that have shown the reliability of an agility test for wheelchair users” is contradicted by Lines 79-84. Williams et al. did investigate the validity of IAT within the context of time to finish, and deemed it validated. While it may be true that they did not investigate the validity of “other important biomechanical factors,” it is important to remember why the IAT was created and how it is used: agility (presumably determined by time to finish as mentioned in Lines 88-90). It appears that this paper aims to validate “other important biomechanical factors” using the IAT; this should be clearer in the Title, Abstract, and Introduction as the main outcome of the IAT (i.e. agility) has already been validated. Please note that Williams et al. demonstrated “validity” of IAT, whereas this paper aims assessing the “reliability” of it, for which we have stated that no publications were found.

15 •Line 93: Specify “the latter characteristic” Specified.

16 •Lines 94-97: Consider revising the purpose statement based on the fourth comment above. We respectfully clarified the misunderstanding in your fourth comment.

Methods:

17 •Lines 125-127: Is the Fitbit Charge HR a validated tool for measuring heart rate? If so, this tool may be useful in the able-bodied manual wheelchair users in this study; however, heart rate is not a validated metric in other manual wheelchair users, such as individuals with spinal cord injury. Translation of this methodology to manual wheelchair users with spinal cord injury/disease may not be appropriate. The heart rate measurement in this study was merely to ensure that the participants had enough rest time between the trials and was not included in the analyses. Since it was in addition to giving at least three minutes of rest time between the trials, it does not look essential to demonstrate the validity of Fitbit Charge HR.

18 •Lines 135-138: Error message should not likely be included in text (“Error! Reference source not found”) This has happened when converting Word to Pdf. I will make sure it does not happen when submitting the revised version.

19 •Line 139: Numbers 3,7, and 28 are not pictured in Figure 3. Is there a reason for this? As it is explained in Line 140, points number 3,7, and 28 pertained to the last push in the straight-line propulsion parts of IAT, just before turning pushes were started. Since participants could start turning anywhere between the two neighboring measurement points, e.g. 2 and 4, depiction of these points (3, 7, and 28) was not feasible.

20 •Lines 148-172: Explanation and justification for using standard camera video over Optitrack should be much shorter. Additionally, this decision was based off one subject’s data. Why were additional subjects’ data not included to make this decision more robust? The explanation was shortened.

Regarding using data of one subject, as it was mentioned in the manuscript, analyzing Optitrack data for 120 frame/sec and an average of 120 sec/trial, if it is used for all 11 subjects, will take days, even months, to finish. The rational for comparing the video results to Optitrack results was to show the validity of video data, by testing a portion of data, which was confirmed by a very high correlation.

21 •Lines 177-179, 226-228: How were data assessed for normality? If data did not fit a normal distribution, why were data transformed to fit a normal distribution instead of using non-parametric statistical tests? Briefly describe the Two-Step method for readers who are not familiar with this test. Normality was checked using Shapiro-Wilk test which is highly recommended for testing normal distribution in SPSS (Ghasemi & Zahediasl, 2012). Where data did not follow normal distribution, we tried our best to get them follow normal distribution through data transformation because parametric methods have higher research powers and considering the relatively small sample size here, we could not compromise power reduction.

In the Two-Step method, we first transform the original values to fit uniform distribution and then to normal distribution. This is a roughly old statistical method that had kept outsider to many fields of science, but through the paper by Gary Templeton, 2011, it was re-introduced to scientific community to use it to their advantage.

These explanations are added to the main text for extra clarification.

22 •The same study subjects performed the IAT at two timepoints roughly one week apart. Why was a MANOVA test conducted instead of a repeated measures MANOVA? One of the assumptions of MANOVA is independency observations; Repeated Measures MANOVA should be used when there are repetitions of a measurement to adjust for dependency between measurements of the dependent variables. In this study, we wanted to show that dependent variables were similar from one session to the next one. So, we were better to first assume that they were different and thus independent, and then try to reject it. Therefore, we used MANOVA.

23 •No mention of significance criteria; alpha/p-values, effect sizes, etc. You are right. They are added to the manuscript now.

Results:

24 •Line 238: Sentence should not start with a number. Applied.

25 •Line 242: Consider using “raw value” in place of “real variable.” Applied.

26 •The Results section should be written in a more technical manner using p-values, F-statistics, degrees of freedom, etc. Added.

27 •The Results section should focus on the statistical outputs and interpretation of these outputs should be reserved for the Discussion section. Revised.

Discussion:

28 •Line 308: Sentence should not start with a number. Applied.

29 •Line 317: Careful using definitive terminology like “confirms” when interpreting results. Applied.

30 •Lines 318-328: Authors should reexamine their interpretation of a Bonferroni correction. Lines 318-319 suggest it IS NOT advantageous for this study to use a Bonferroni correction; however, in Lines 325-326, the authors claim IT IS advantageous for this study to use a Bonferroni correction. A Bonferroni correction adjusts the p-value used to determine significance based on the number of dependent variables. We are afraid you have not grasped the point raised in that paragraph. What we were trying to say was that usually the case is Bonferroni correction is used to adjust the alpha level, so that a P-value of 0.04 (for example) does not result in a claim for detecting a significant change. Whereas in our manuscript, the case is the opposite - a rejection by a lower alpha level is favorable, not restricting. Thus, using Bonferroni Correction is pointless and even worse, it decreases the study power.

We reworded the paragraph to be clearer.

31 •The methods of this paper are designed to control for Type 1 Error (rejecting a true null hypothesis); however, it does not control for Type II Error (accepting a false null hypothesis). This should be addressed. The method of sample size calculations for maintaining an acceptable research power, and thus acceptable chance of Type II error, is now added to the manuscript.

32 •Lines 331-337: Research has found that pediatric and adult manual wheelchair users with pediatric-onset spinal cord injury employ highly variable upper extremity movement. While the exact mechanism for this is unknown, some attribute it to adaptation to prevent overuse injury during wheelchair propulsion. The authors’ interpretation that experienced manual wheelchair users use “more consistent” mechanics based on experience may not always be true. Right. But as it was explained in Lines 336-345, note that this consistency comparison is between real experienced wheelchair users and non-wheelchair users. While within-subject variability for real wheelchair users could be relatively high, we need to ask how is the between-subject variability when they are compared to non-wheelchair-users. It seems self-evident that people who have no experience using a very different method of ambulation will be less consistent in wheelchair propulsion, overall.

33 •Line 349: Many of these metrics are determined as “clinically meaningful.” How so? Provide rationale or references that mention the metrics used in this study as “clinically meaningful.” The single-value parameters measured and analyzed for this study were selected based on clinical and practical relevance and importance. These are metrics that are frequently needed by both clinicians and engineers who work with or for wheelchair users. For instance, cadence and push length are important parameters that clinicians work on with newly injured wheelchair users, as they work effectively toward propulsion efficiency which, in turn, plays a very important role in secondary injuries occurred to wheelchair users. Also, tangential and total force applied on the rim is both effective on propulsion efficiency and determinative when measuring activities of wheelchair users. Moreover, starting angle is an effectual factor in adjusting wheelchair axle for stability of the wheelchair, and preventing secondary injuries. All of these show the importance of the factors reported in this paper as why they can be very informative.

One example is the average speed of wheelchair manoeuvring which found to be 0.85, about 57% to 77% of the average speed of SLP wheelchair propulsion reported elsewhere (references in the manuscript). When a factor like speed become nearly half in value, the meaningfulness of it is self-evident.

34 •Do the authors have any directions for future work based on these results? Some sentences were added on this subject.

Conclusion:

35 •This section should “wrap-up” the manuscript and highlight the results of this study. Consider adding a few more sentences to bring the manuscript to a succinct ending. Added.

Reviewer #2:

36 It is inappropriate to use Null Hypothesis Significance Testing to argue for equivalence or equality. As it is typically used (i.e., when the researcher hopes to reject the null hypothesis), NHST is designed to make it difficult for the researcher to get what they want. When the researcher is hoping to 'prove' the null, however, as is the case here, achieving the desired result using NHST is quite easy: just design a poor and/or underpowered study. Failing to reject the null hypothesis means either: 1) that the null is true (in the case of this study, the repeated measurements are equal), or 2) the null is false (the repeated measurements are not equal) but the experiment doesn't have enough power to reject it. In order to arrive at the first conclusion (equality) the authors must provide evidence against the second (an underpowered study). Unfortunately, the authors provide no evidence of high statistical power. In fact, using only 11 subjects in the study makes it likely that the study is underpowered. In order for the results of the study to be meaningful, I strongly suggest that the authors forgo the use of NHST-based procedures such as MANOVA to determine equivalence in favor of more appropriate methods such as Bayesian methods, confidence intervals (Loftus, 1996; Aberson, 2002), or noninferiority testing (Streiner, 2003 The reviewer is correct in pointing out that NHST is not a suitable way to prove similarity. We agree and respectfully argue that the main evidence we are presenting to show the similarity here is the ICC between 16 variables on 32 measurement points, which is a standard statistical approach for proving reliability. The MANOVA analyses that the reviewer is pointing to, however, has been used to only confirm the proven idea by adding extra supporting material, and we made sure that we are very careful in the terminology we used for that in the manuscript to be clearer. For the MANOVA results, we just concluded that they were not different, not that they were the same. Again, the point raised by the reviewer is quite correct, but we need to clear up that this was not used as the main analysis in this study; only as supporting information.

37 If the authors insist on using NHST to claim evidence of equality, I view it as critical to demonstrate that the study as ran had high power. The sample size calculation based on 80% research power is added to the manuscript.

38 Effect size measures, F statistics, and their associated degrees of freedom should be included for all MANOVAs. Sure. They are added to the manuscript now.

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

17 Aug 2020

PONE-D-20-08803R1

Investigating the test-retest reliability of Illinois Agility Test for wheelchair users

PLOS ONE

Dear Dr. Salimi,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Oct 01 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Yih-Kuen Jan, PhD

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: (No Response)

Reviewer #2: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: General Comments:

• The reviewer thanks the authors for their diligent revisions. The manuscript flows well with a clear direction.

• Minor grammatical errors throughout.

• Concerns remain for MANOVA statistical testing and interpretation.

• Further discussion is needed to address how results obtained from able-bodied individuals in wheelchairs are reasonably translatable to wheelchair users with disabilities.

Introduction:

• Lines 94-96: Minor wording. Consider moving “wheelchair maneuvering” before “training athlete wheelchair users” as maneuvering is the primary focus of the introduction and manuscript as a whole.

Methods:

• Lines 115 and 116: Using “average” in demographics is unnecessary after the authors indicate data listed are mean ± SD.

• Line 127: Consider using mean ± SD for consistency.

• Line 142: Justification is needed for why participants performed an agility test at a comfortable speed (rather than done as quickly as possible [Line 134])

Results:

• The ICCs shows good to excellent reliability, which is the main outcome of the study. If ICCs are high for most variables, this should be the sole justification for reliability. Concerns remain for the interpretation of MANOVA tests. Why supplement statistical testing with MANOVAs that fail to reject the null hypothesis? This would also eliminate any verbiage needed regarding Bonferroni corrections.

Discussion:

• Lines 380-386: References are needed for these statements.

• Future research is likely needed with non-able-bodied wheelchair users (i.e., participants with disabilities) to ensure repeatability of the IAT, especially if authors suggest that IAT can be used for future wheelchair maneuvering research.

Reviewer #2: Thank you for addressing my comments. I have no problems with the statistical procedures as they are presented in the revision, with one very minor exception. By my calculations, the power analysis conducted on page 5 suggests a sample size of 11.69 to achieve power of 0.8, so technically you should have used 12 subjects to achieve 80% power, not 11. As I said, though, I consider this a minor point.

As an aside, a few grammatical errors remain in the manuscript.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

5 Sep 2020

Comments Responses:

Reviewer #1:

Introduction:

1 Lines 94-96: Minor wording. Consider moving “wheelchair maneuvering” before “training athlete wheelchair users” as maneuvering is the primary focus of the introduction and manuscript as a whole.

A: Done.

Methods:

2 Lines 115 and 116: Using “average” in demographics is unnecessary after the authors indicate data listed are mean ± SD.

A: Very right. Corrected.

3 Line 127: Consider using mean ± SD for consistency.

A: Sure. Corrected.

4 Justification is needed for why participants performed an agility test at a comfortable speed (rather than done as quickly as possible [Line 134])

A: Since completion time is not a great concern for wheelchair users when accomplishing wheelchair manoeuvres. Rather, other factors like how well they can finish the task without overusing their shoulders, are of more concern to them. An explanation is added to the manuscript [lines 141-142].

Results:

5 The ICCs shows good to excellent reliability, which is the main outcome of the study. If ICCs are high for most variables, this should be the sole justification for reliability. Concerns remain for the interpretation of MANOVA tests. Why supplement statistical testing with MANOVAs that fail to reject the null hypothesis? This would also eliminate any verbiage needed regarding Bonferroni corrections.

A: As the reviewer pointed out, the main evidence we are presenting to show the similarity here is the ICC between 16 variables, which is a standard statistical approach for proving reliability. The MANOVA analyses, however, have been used to only confirm the proven idea by adding extra supporting material. Fortunately, the MANOVA results for ALL 37 variables were favorable and backed the main hypothesis of the study. From the MANOVA results, we concluded that the parameters were not different. We need to emphasize that “not rejecting the null hypothesis” was favorable for our study. This should clear up the misunderstanding and show that using MANOVAs not only did not weaken but also strengthen the results of the study.

Discussion:

6 Lines 380-386: References are needed for these statements.

A: Six references were added.

7 Future research is likely needed with non-able-bodied wheelchair users (i.e., participants with disabilities) to ensure repeatability of the IAT, especially if authors suggest that IAT can be used for future wheelchair maneuvering research.

A: A paragraph is added regarding this issue on lines 371 to 376.

Reviewer #2:

8 I have no problems with the statistical procedures as they are presented in the revision, with one very minor exception. By my calculations, the power analysis conducted on page 5 suggests a sample size of 11.69 to achieve power of 0.8, so technically you should have used 12 subjects to achieve 80% power, not 11. As I said, though, I consider this a minor point.

A: You are right. Rounding it to 12 was a better practice. For greater clarity, I denoted in the manuscript that the exact result of the formula was 11.63 (according to this:

1.5+((1.64+.84)/((ln((1+.6)/(1-.6))/2)-(ln((1+.9)/(1-.9))/2)))^2=11.63).

9 A few grammatical errors remain in the manuscript.

A: We implemented another round of grammatical check and also applied the grammatical points raised by the first reviewer.

Editor:

10 Upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool

A: Done.

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

15 Oct 2020

Investigating the test-retest reliability of Illinois Agility Test for wheelchair users

PONE-D-20-08803R2

Dear Dr. Salimi,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Yih-Kuen Jan, PhD, University of Illinois at Urbana-Champaign

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: (No Response)

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: (No Response)

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: (No Response)

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: (No Response)

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The reviewer would like to thank the authors for their diligent revisions. All sections of the manuscript are thorough and provide great detail. Few minor grammatical errors and alpha-numeric inconsistencies (Example: Line 325 - 1 vs. one) remain.

Reviewer #2: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

19 Oct 2020

PONE-D-20-08803R2

Investigating the test-retest reliability of Illinois Agility Test for wheelchair users

Dear Dr. Salimi:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Yih-Kuen Jan

Academic Editor

PLOS ONE