Concurrent Validity and Reliability of the Transfer Assessment Instrument Questionnaire as a Self-Assessment Measure.

OBJECTIVES: To evaluate the psychometric properties of the Transfer Assessment Instrument Questionnaire (TAI-Q), a self-assessment measure to evaluate transfer quality compared with clinician-reported measures.
DESIGN: Participants self-assessed transfers from their wheelchair to a mat table using the TAI-Q. For session 1, participants self-assessed their transfer both before and after reviewing a video of themselves completing the transfer (session 1). Self-assessment was completed for another transfer after a 10-minute delay (session 2, intrarater reliability) and after a 1- to 2-day delay (session 3, test-retest reliability). Self-assessment was compared with a criterion standard of an experienced clinician scoring the same transfers with the Transfer Assessment Instrument (TAI) version 4.0 (concurrent validity).
SETTING: 2017 National Veterans Wheelchair Games. PARTICIPANTS: Convenience sample of full-time wheelchair users (N=44).
INTERVENTIONS: Not applicable. MAIN OUTCOME MEASURES: TAI-Q and TAI.
RESULTS: After video review of their transfer, acceptable levels of reliability were demonstrated for total TAI-Q score for intrarater (intraclass correlation [ICC], 0.627) and test-retest reliability (ICC, 0.705). Moderate to acceptable concurrent validity was demonstrated with the TAI (ICC, 0.554-0.740). Participants tended to underestimate the quality of their transfer (reported more deficient items) compared with the TAI. However, this deficit decreased and reliability improved from pre-video review to post-video review and from session 1 to session 2. The minimum detectable change indicated that a change of 1.63 to 2.21 in the TAI-Q total score is needed to detect a significant difference in transfer skills.
CONCLUSIONS: When paired with video review, the TAI-Q demonstrates moderate to acceptable levels of reliability and validity for the total score. Self-assessment was completed quickly (<5min) and could help to potentially screen for deficiencies in transfer quality and opportunities for intervention.

There are currently approximately 5.47 million individuals in the United States who use a wheelchair as their primary method of transportation. An international population of wheelchair users participated in a survey that identified transfers as one of the most important skills for wheelchair users to be able to perform. In a study conducted among individuals with spinal cord injuries, 58.6% reported that they consistently experience upper extremity pain. Of this group, 65% reported that the pain was strongly associated with and affected their ability to perform a successful wheelchair transfer.

Owing to the forces placed on the shoulder during transfers, wheelchair users are at a greater risk for upper extremity injury and pathology.4, 5, 6 Overuse injuries compounded with an unnecessary amount of force owing to improper transfer techniques exponentially increases the frequency and severity of injuries. In addition, the frequency of transfers puts wheelchair users at an increased risk for additional injuries. On average, wheelchair users transfer 15 to 20 times per day, which can cause overuse injuries. The TAI is an objective measure of transfer quality with demonstrated reliability and validity. Individuals who use ergonomic transfer techniques, as determined by higher scores on the Transfer Assessment Instrument (TAI), demonstrate decreased force on their shoulders and fewer ultrasonographic markers of shoulder pathology.,

To date, the TAI has only been used as a clinician-lead objective measure to assess and score the quality of a transfer. There are several barriers individuals with disabilities face when trying to access clinics where their wheelchair transfers could be evaluated. In a study of 153 community dwelling wheelchair users, when asked about their ability to visit locations outside of their home, those with more limitations also reported less participation in medical and nonmedical related activities. In addition, transportation and financial burdens negatively affect a wheelchair user’s ability to go to a medical clinic, which would thus increase the difficulty for them to receive an in-clinic transfer assessment.,

To address these barriers, the most recent version of the TAI also allows for assessment by nonclinicians and those without training. In an effort to increase accessibility to both clinicians and wheelchair users, this version of the TAI is written in plain language and includes illustrative pictures and diagrams. There is also a version in which the rater is blinded to scoring to allow for self-assessment, known as the Transfer Assessment Questionnaire (TAI-Q). In the TAI-Q version, respondents have answer options but associated scores are shown (supplemental appendix S1, available online only at http://www.archives-pmr.org/). As such, the TAI-Q could be used for remote assessment to leverage telehealth opportunities or to complement current in-person clinical care.

Self-assessment methods are beneficial in that they do not require in-person contact with a trained clinician and may better reflect what one does in everyday life, compared with a clinical assessment at a given time point. In a study assessing the accuracy and validity of a self-report measure for wheelchair skills, subjective self-assessments were highly correlated with the objective version of the test administered by a clinician. However, self-assessment scores were generally higher, indicating that the user reported a higher level skill., One reason for this may be because wheelchair users overestimate their ability through self-report. However, it could also be because the clinician assessment is often performed in an unfamiliar and potentially anxiety-invoking setting, which could cause a wheelchair user to be unable to perform a task they typically can perform.,

The primary goal of this study was to evaluate the psychometric properties of the TAI-Q including intrarater and test-retest reliability and concurrent validity with an in-person TAI assessment performed by a trained physical therapist. We hypothesized that TAI-Q self-assessment reliability would meet acceptable levels (intraclass correlation coefficient [ICC], ≥0.6) and that scores would improve after participants watched a video of their transfer performance compared with scoring from memory. We anticipated an improvement after watching the video because participants might not be used to paying attention to some of the aspects of their transfer assessed by the TAI-Q. Furthermore, some of the items on the TAI-Q may be better assessed through observation in a video (ie, lean) than from recall. In addition, item level agreement between the TAI and TAI-Q would meet or exceed 75% for all items.

Methods

Participants

A convenience sample of participants was recruited from the 2017 National Veterans Wheelchair Games (NVWG) in Cincinnati, OH. Participants signed consent forms approved by the Institutional Review Boards at the University of Pittsburgh, Northwestern University, Kessler Foundation, and University of Miami. The study inclusion criteria were as follows: age older than18 years, uses a wheelchair for their primary means of mobility (≥40h/wk), and speaks English. Because we were investigating independent transfers, participants had to be able to independently transfer to and from a wheelchair surface within 30 seconds. They could use a transfer board. Participants were excluded if they had a recent (within the past 3mo) or current history of pressure ulcers, stood to transfer, had a neurologic condition that could impair learning, or had arm pain that limited their ability to transfer or bear weight through their arms. The VA Pittsburgh Healthcare System Institutional Review Board approved this work (Pro00001261).

Sample size

Sample size estimations were based on a study of the TAI, which was evaluated with 41 participants with ICCs ranging from 0.81 to 0.85 for interrater reliability and 0.74 to 0.88 for intrarater reliability. Using a cutoff of 0.8 and assumption of achieving an ICC of at least 0.87, a sample size of 40 participants was expected to achieve 80% power using an F test with a significance level of 0.05.,

Testing protocol

General demographic information was collected from participants at baseline and included age, sex, and diagnosis. Before the start of the transfer, participants were reminded that they would be completing a self-assessment of their transfer performance immediately after completion of the transfer. Participants were instructed to set up their wheelchair and body as they normally would for a transfer from their wheelchair to a portable mat table. The height of the mat table was set to 0.56 m (22 inches) before each participant’s first transfer, and participants were instructed that the height of the mat table was adjustable and could be changed at their request. After the participant had set up his or her wheelchair for the transfer, the distance between the front corner of the wheelchair (TAI-Q item 1), angle between the wheelchair and mat table (TAI-Q item 2), and the difference in height between the cushion and mat table (TAI-Q item 6) were measured and read aloud by study staff (supplemental fig S1, available online only at http://www.archives-pmr.org/). Participants were then asked to complete their transfer to the mat (session 1). Participants then transferred back to their wheelchair but this transfer was not assessed. Additional transfers (sessions 2 and 3) were completed using the same procedures. Session 2 was performed on the same day, after at least a 10-minute delay to assess intrarater reliability. Session 3 was performed 1 to 2 days later to assess test-retest reliability. Each transfer was recorded by video to capture a view of the participant’s transfer from the sagittal plane.

During each transfer, a clinician with 13 years of experience working with wheelchair users independently completed the TAI. During session 1, participants were immediately asked to complete the TAI-Q for the transfer they completed to the mat table, scoring from memory. Study staff was present to transcribe participant responses if limited hand function affected their ability to record responses. After scoring the TAI-Q from memory, participants were then instructed to watch the video of themselves completing the transfer and were asked to complete a second TAI-Q for the same transfer. The video was played on a computer screen connected to a hand-held camera that recorded the transfer in a loop so it could be viewed as many times as the participant desired. For sessions 2 and 3, the video was presented for participant review before completion of the TAI-Q.

TAI-Q and TAI scoring

The TAI is an objective measure of transfer quality with demonstrated reliability and validity. For each item, a score of “1” indicates perfect technique and “0” indicates very poor technique. Partial credit (“0.5”) is allowed for some items. Some items include an answer option of “not applicable,” in which case the item is not included in the total score. All item scores are added together, multiplied by 10, and averaged, resulting in a score from 0 to 10 points:

Subscores are calculated in a similar manner for the 3 phases of a transfer: wheelchair setup, body setup, and flight and landing. For concurrent validity, the TAI was used as a reference standard for comparison.

The TAI-Q mirrors the TAI in content and scoring with the exception that scores are not displayed for each item as they are for TAI so as to not bias the user’s responses (supplemental fig S2, available online only at http://www.archives-pmr.org/). A scoring pane is embedded in the tool so scoring can be complete after self-assessment is complete. For each session, only one transfer was assessed, so item 16 (alternating arm) was not included in subscores or total scores.

Data analysis

Data analysis was performed using SPSS 24.0 software.a Descriptive statistics were calculated for TAI and TAI-Q total, subscores, and item scores. Participant demographics between completers and noncompleters were evaluated using chi square or independent t tests. ICCs were used to assess total and subscore (wheelchair setup, body setup, flight and landing) concurrent validity (TAI vs TAI-Q), intrarater reliability (TAI-Q session 1 post-video review vs session 2), and test-retest reliability (TAI-Q session 1 post-video review vs session 3)., Based on previous studies, we set cutoffs a priori of 0.8 or greater for strong, 0.60 to 0.79 for acceptable, 0.40 to 0.59 for moderate, and 0.39 or less for poor reliability.,20, 21, 22 Pairwise t tests with a Holm correction for multiple comparisons were used determine whether differences existed between TAI and TAI-Q total scores or between sessions. Bland-Altman plots were used to evaluate systematic bias between the TAI and TAI-Q scores, with good agreement indicated by an even spread of points within the limits of agreement and a mean difference close to zero.

TAI-Q items are targeted to specific components of the transfer to identify deficit areas for training and intervention. To evaluate consistency at the item level, percentage agreement between the TAI and TAI-Q (before and after video review) was evaluated with a cutoff of 75% indicating clinical agreement., Similarly, Cohen’s kappa was also run with agreement indicated as follows: substantial (≥0.61), moderate (0.41-0.6), fair (0.21-0.40), and slight (<0.20).

The standard error of measurement (SEM) was calculated for each session as SEM=SD × [1–r]1/2 where SD is the SD of the dataset and r is the intrarater reliability coefficient. The minimal detectable change (MDC) was calculated as MDC=1.96×21/2×SEM.

Results

Forty-four individuals completed session 1, 43 completed sessions 1 and 2, and 29 completed all sessions. Missing data were the result of participants not returning for follow-up (session 2, n=1; session 3, n=7) or sessions 1 and 2 being held on the last day of the NVWG with no remaining days for follow-up (n=7). Participant demographics are described in table 1. The majority of participants were men, had a spinal cord injury, and were manual wheelchair users. On average, participants were 56 years old and 17 years since injury. No significant differences were found between those who completed the study and those who did not return for follow-up (supplemental table S1, available online only at http://www.archives-pmr.org/). Self-assessment using the TAI-Q was completed in less than 5 minutes for all participants.

Table 1

Participant demographics

Variable	n (% of sample)
Sex
Men	35 (83.3)
Diagnostic category
Paraplegia	20 (45)
Tetraplegia	2 (5)
SCI unspecified	8 (18)
MS	1 (2)
Transverse myelitis	1 (2)
Amputee	5 (11)
Guillian-Barre	1 (2)
Stroke	1 (2)
Lower motor neuron	3 (7)
Unknown	2 (5)
Type of wheelchair
Manual	33 (75)
Power	11 (25)
Use of assistive device (sliding board)	2 (5)

TAI-Q and TAI scores

For session 1, the mean ± SD TAI total score was 7.7±1.1; this was higher than the TAI-Q scores for session 1, both before (7.1±1.0; P=.001) and after (7.3±1.0; P=.010) video review (fig 1). TAI scores were also higher than TAI-Q scores for session 2 (7.7±1.2 vs 7.3±1.1; P=.005). There was no difference in TAI and TAI-Q scores for session 3 (7.7±1.1 vs 7.6±1.4; P=.582). TAI-Q total scores were significantly higher in session 3 than session 1 post-video review (P=.012).

Fig 1

Transfer Assessment Instrument (TAI) and Transfer Assessment Instrument Questionnaire (TAI-Q) scores for sessions 1 through 3. ∗ indicates significant difference based on paired comparison.

Concurrent validity and reliability

Table 2 shows concurrent validity, intrarater reliability, and test-retest reliability for total and subscores of the TAI and TAI-Q. Agreement between the TAI and TAI-Q (concurrent validity) was poor-to-moderate for session 1 but improved to moderate-to-acceptable after video review and to acceptable-to-strong for session 2. Subscores for body setup and flight and landing did not meet acceptable levels for intrarater or test-retest reliability.

Table 2

Concurrent validity with the TAI and intrarater and test-retest reliability for the TAI-Q total and subscores

Score	Concurrent Validity (TAI-Q vs TAI)			Intrarater (TAI-Q)	Test-Retest (TAI-Q)
	Session 1Pre-Video	Session 1Post-Video	Session 2Post-Video	Session 1 Post-Video vsSession 2 Post-Video	Session 1 Post-Video vsSession 3 Post-Video
Total score	0.411∗	0.554∗	0.740†	0.627†	0.705†
Wheelchair setup	0.508∗	0.604†	0.657†	0.643†	0.668†
Body setup	0.457∗	0.676†	0.836‡	0.775†	0.549∗
Flight/landing	0.289	0.495∗	0.669†	0.533∗	0.380

NOTE. Session 1 is the only session self-scoring completed before video review. In the remainder of sessions, all self-scoring was completed after video review.

Moderate reliability.

Acceptable reliability.

Strong reliability.

Bland-Altman plots indicate that systematic bias decreased with each testing session, indicated by means of differences closer to zero (supplemental figs S3-S6, available online only at http://www.archives-pmr.org/). This is consistent with significantly higher TAI scores found for sessions 1 and 2 with paired comparisons. Overall, we found no trends in proportional error or error related to the measurement of the error.

Item analysis

Only 2 users used a transfer board, so agreement is not reported in item-level statistics for items 17 and 18. Percentage agreement between the TAI-Q and TAI met the 75% cutoff for all items except for angle (item 2) and lean (item 13) (table 3). Agreement levels were either maintained or improved after video review for all items. In cases of disagreement, participants were more likely to underestimate the quality of their transfer with higher incidence of a TAI-Q item score less than the TAI score, compared with greater than the TAI score.

Table 3

Item level percentage agreement and Cohen’s kappa between TAI and TAI-Q scoring before and after video review by participants for session 1

Phase of Transfer	Item		TAI-Q<TAI		TAI-Q=TAI		TAI-Q>TAI		Cohen’s Kappa
			Pre-Video Review, %	Post-Video Review, %	Pre-Video Review, %	Post-Video Review, %	Pre-Video Review, %	Post-Video Review, %	Pre-Video Review	Post-Video Review
Wheelchair setup	1	Distance	7	2	91∗	93∗	2	5	0.804‡	0.846‡
	2	Angle	25	23	64	73	11	5	0.281	0.471†
	3	Brakes	0	0	91∗	91∗	9	9	0.541†	0.463†
	4	Armrest	0	6	94∗	94∗	6	0	0.824‡	0.852‡
	5	Sideguard	0	0	100∗	100∗	0	0	1.000‡	1.000‡
	6	Level	9	7	86∗	91∗	5	2	0.613‡	0.742‡
Body setup	7	Feet	5	2	90∗	95∗	5	2	0.767‡	0.883‡
	8	Scoot	14	9	77∗	82∗	9	9	0.304	0.389
	9	Trail distance	0	0	100∗	100∗	0	0	1.000‡	1.000‡
	10	Push grip	16	12	77∗	86∗	7	2	0.453†	0.652‡
	11	Lead grip	16	9	84∗	89∗	0	2	0.462†	0.689‡
	12	Lead distance	20	11	77∗	86∗	2	2	0.098	0.195
Flight/landing	13	Lean	20	11	70	80∗	9	9	0.247	0.396
	14	Between surfaces	5	7	89∗	89∗	7	5	-0.035	-0.058
	15	Landing	7	7	89∗	89∗	5	5	-0.058	-0.058
Average			10	7	85	89	5	4
SD			9	6	11	8	4	3

Met 75% threshold for clinical significance. TAI-Q < TAI indicates that the self-assessment score was less than the clinician rating. TAI-Q=TAI indicates that the self-assessment score was in agreement with the clinician rating. TAI-Q > TAI indicates that the self-assessment score was greater than the clinician rating.

Moderate agreement (0.60 ≥ κ ≥ 0.41).

Substantial agreement (κ ≥0.61).

Further analysis was performed to examine item level agreement using Cohen’s kappa (see table 3). The majority of scores indicated either moderate or substantial agreement between the TAI and TAI-Q, both before and after video review. For most items, the agreement between the clinician (TAI) and participant (TAI-Q) improved after the participant watched the video. Three items resulted in a fair agreement: angle (item 2), scooting (item 8), and amount of lean (use of the head/hips relationship, item 13). Items with slight to no agreement were lead hand distance (item 12), smoothness of the movement between surfaces (item 14), and stability of landing (item 15).

SEM and MDC

The SEMs were 0.80, 0.71, and 0.59 for session 1 pre-video review, session 1 post-video review, and session 2, respectively. The MDCs were 2.21, 1.97, and 1.63 for session 1 pre-video review, session 1 post-video review, and session 2, respectively.

Discussion

Moderate to acceptable levels of reliability were demonstrated for total TAI-Q score for intrarater reliability with acceptable levels for test-retest reliability and concurrent validity with the TAI. Participants tended to underestimate the quality of their transfer (reported more deficient items) compared with the TAI. However, this deficit decreased and reliability improved from pre-video review to post-video review and from session 1 to session 2. This may indicate that psychometric properties of the tool would be optimized if participants were provided the opportunity to review and practice scoring with the tool before transferring and if scoring is paired with video review. Participants were shown a simple 1-camera view of their transfer, which could be emulated through recording capabilities available with most smartphones. These results are consistent with a previous study of the validity of a end-user’s self-assessment of manual wheelchair skills as compared with an objective clinical assessment. The aforementioned study found that a questionnaire version of the Wheelchair Skills Test was able to provide a valid assessment of wheelchair user’s manual wheelchair skills. Contrary to our findings, they did not find that providing visual aids to participants, including pictures and a wheelchair model, substantially improved the validity of the tool. However, they did not provide individualized pictures of the participants performing the skills, only generic photos to further explain the questions. The use of individualized visual aids may be an important factor to increase reliability.

Reliability was lower for the flight and landing subscore than the wheelchair or body setup subscales. Of note, compared with the other 2 subscales, there are substantially fewer items in the flight and landing section (3 items). In addition, the items relating to movement, which may be more difficult to evaluate, make up the flight and landing subsections exclusively, and the items assessed in the wheelchair or body setup subsections relate to setup (static) positions. Upon examination of interrater reliability of the TAI, higher levels of reliability were also found among the wheelchair set up subsection compared with the body set and flight and landing subsections.

In the wheelchair set up section, the only item that resulted in a fair level of agreement was angle between the wheelchair and transfer surface (item 2). After viewing the video, agreement increased to a moderate level. However, the item did not reach clinically acceptable levels. The video recorded a sagittal view of the transfer, which does not capture wheelchair angle well. Further prompting for the wheelchair user to score this item before moving their chair might assist with scoring for this item in the future. Alternatively, this item may need to be revised to improve accuracy of user responses.

Regarding body set up, scooting forward in the chair prior to the transfer (item 8) was found to have fair agreement, and the distance of the lead arm from the hip at the end of the transfer (item 12) had slight agreement. However, agreement on both items improved after watching the video. Both items 8 and 12 were modified when the TAI 4.0 was developed to improve reliability. Further revision may be necessary to improve the reliability of these items.

Finally, although clinical significance was achieved for all items in flight and landing after video review, individual item agreement was reported as either fair (lean [item 13]) or none (quality of movement between surfaces [item 14] and stability during landing [item 15]) for Cohen’s kappa. This difference in results is likely driven by skewed responses (ie, for item 14, only 4 respondents did not score a “1” on the item). Only leaning (item 13) improved after video review. However, percent agreement improved for the flight and landing subscore with subsequent scoring (session 1 to session 2). These nuanced assessments related to the quality of the movement and body positioning may be challenging for an end user to self-evaluate. Further instructions on how to assess these challenging items and practice may help to improve the level of agreement.

In the past, the TAI has been used as a training tool to identify deficient areas suitable for intervention.,, Clinically acceptable levels of item agreement support use of the TAI-Q for similar use in this domain. Because users were more likely to report deficits in cases of disagreement (ie, TAI-Q < TAI), areas in need of intervention are not likely to be missed by the tool.

The MDC indicated that a change of 1.63 to 2.21 in the TAI-Q total score is needed to detect a significant difference in transfer skills. Factoring in potential items with a “not possible” response, this would equate to an improvement in 1 to 4 items. Because the MDC was lower after repeated testing, this may support allowing an individual to practice scoring or review the TAI-Q before transfer to improve sensitivity of the tool. The established MDC for the TAI was 1.3 points, which was similar to the TAI-Q after repeated testing. Future studies are warranted that investigate minimal clinically important differences in the tool. An established minimal clinically important difference would provide both end users and clinicians an important benchmark to further interpret changes in transfer skills after engagement in transfer training.

Limitations

As this study was conducted at the NVWG, participants may be more active than the general population. A previous study used a TAI score cutoff of 7.36 or greater to identify individuals who would benefit from transfer training. The mean TAI indicates that this was an overall skilled group. Future studies would benefit from a more diverse range of transfers abilities to better represent those with poorer transfer skills. Participants were instructed to watch the video of themselves transferring to inform their scoring on the TAI-Q. However, not all participants paid equal attention to the video, and this may have influenced reliability. Study staff measured and read aloud items 1.2 and 6 related to wheelchair setup. In a true self-assessment, the wheelchair user would have taken these measurements him- or herself. Although the responses are increments that could likely be estimated without a ruler or goniometer (eg, <3in, 3-5in, >5in), future studies should considering evaluating reliability of these items either without the measurements read aloud or with the wheelchair user completing measurements him- or herself. Similar to the TAI, the TAI-Q is designed to evaluate transfers to many different surface types (bed, commode, etc.), but these types of transfers were not evaluated in this study. Additional studies are also warranted to explore the responsiveness of the tool to training to determine a clinically meaningful difference.

Conclusions

When paired with video review, the TAI-Q demonstrates moderate to acceptable levels of reliability and validity for the total score. Self-assessment was completed quickly (<5min) and could help to potentially screen for deficiencies in transfer quality and opportunities for intervention.

Supplier

SPSS 24.0; SPSS, Inc.