Evaluating the reproducibility of the short version of the Western Ontario Rotator Cuff Index (Short-WORC) prospectively.

BACKGROUND: Recently, a shorter version of the Western Ontario Rotator Cuff Index (Short-WORC) was created to reduce patient response burden. However, it has yet to be evaluated prospectively for reproducibility (reliability and agreement) and floor and ceiling effects.
METHODS: Patients (N = 162) with rotator cuff disorders completed the Short-WORC at baseline. From this cohort, 47 patients underwent measurement of test-retest reliability within 2 to 7 days. We used the Cronbach α to determine internal consistency and the intraclass correlation coefficient (ICC2,1) to assess test-retest reliability. To evaluate parameters of agreement, the standard error of measurement, minimal detectable change (based on a 90% confidence interval), and Bland-Altman plots were used.
RESULTS: The Cronbach α was 0.82 at baseline, and the intraclass correlation coefficient (ICC2,1) was 0.87. The agreement parameter was 8.4 for the standard error of measurement of agreement, and the limits of agreement fell within the range of -22.9 to 23.8. The Short-WORC is reliable over time and reflective of a patient's true score after an intervention.
CONCLUSIONS: The Short-WORC demonstrated strong reproducibility parameters and can be used for patients with rotator cuff disorders. The Short-WORC indicated no systematic bias and was reflective of the true score of both individual patients and groups of patients at 2 time points.

Rotator cuff disorders (RCDs) are a common cause of impairment and activity limitation, resulting in a loss of quality of life (QoL). The prevalence of partial- and full-thickness rotator cuff tears is greater than 60% in symptomatic patients older than 60 years., Therefore, the primary goal of both surgery and rehabilitation is to improve function and QoL in patients with RCDs.,

Recently, a shorter version of the Western Ontario Rotator Cuff Index (Short-WORC) was adapted from its original format to evaluate QoL in patients with RCDs. Through theoretical and clinical principles supported by a factor analysis, the Western Ontario Rotator Cuff Index (WORC) was reduced from 21 items to 7 items from the domains of work and lifestyle.,, The Short-WORC consists of a smaller number of items that focus on activity limitations and generates a single summary score without the 5 domain scores generated by the original version of the WORC. In 2012, Razmjou et al found strong psychometric properties for the Short-WORC and suggested that it reduces response burden. Shortly thereafter, Dewan et al found excellent reliability, validity, and responsiveness when extracting scores from the full WORC., This collection of work suggests that the Short-WORC has excellent psychometric properties compared with the full WORC and other patient-reported outcome (PRO) measures. However, no studies have prospectively evaluated the reproducibility (reliability and agreement) of the Short-WORC.,,

Reproducibility measures the extent to which similar results are obtained from repeated assessments. Furthermore, “reproducibility” is a broad term that incorporates the parameters of both reliability and agreement.,, Reliability focuses on the degree to which test scores are consistent, dependable, repeatable, and to a degree, free of measurement error. Reliability can be further investigated through internal consistency (cross-sectional reliability) and test-retest reliability (longitudinal reliability).,, In addition, the property of agreement focuses on measurement error and evaluates the proximity of scores derived from repeated measurements. Agreement is investigated through absolute reliability coefficients (standard error of measurement [SEM], minimal detectable change [MDC]) and Bland-Altman (BA) plots.,

PRO measures must demonstrate both reliability for discriminative applications and agreement to discern real change from error., Therefore, it is critical to examine both reliability and agreement in outcome measures. Thus, the purpose of this study was to evaluate the reproducibility (internal consistency, test-retest reliability, and agreement) of the Short-WORC in a prospective patient population with RCDs.

Methods

Study design

The reproducibility (internal consistency, test-retest reliability, and agreement) of the Short-WORC was assessed through a prospective cohort of patients undergoing treatment at the Roth McFarlane Hand and Upper Limb Centre at St Joseph's Health Care London, London, Ontario, Canada.

Participants

Patients 18 years or older, and received a diagnosis of RCD were advised to undergo (surgical or rehabilitation) at the Hand and Upper Limb Centre were eligible for the study. Patients with upper-extremity fractures, adhesive capsulitis, shoulder instability, infection, tumors, and/or labral, cartilage, and ligamentous tears were excluded from the study. Among the patients (N = 162) who completed the individual items on the Short-WORC at baseline, a subset of 47 participants in stable condition were retested within 7 days to determine test-retest reliability.

We expected to obtain test-retest reliability and internal consistency (intraclass correlation coefficient [ICC]) of 0.90, as shown in previous studies., Therefore, we calculated a sample size required to determine whether the reliability of the Short-WORC exceeds a 0.95 confidence interval (CI) around a power of 0.80.

Outcome measures

The WORC was originally shortened to the 7-item Short-WORC from the domains of work and lifestyle and validated by Razmjou et al. The Short-WORC total score can range from 0 (best possible score) to 700 (worst possible score). The percentage score is obtained from the sum of the raw item scores, divided by 700, and multiplied by 100. This generates a score between 0% (poor QoL) and 100% (high QoL). The Short-WORC cannot be scored if items are missing.,

Statistical analysis

Data were assessed for completeness, the percentage of missing data, the presence of outliers, and floor and ceiling effects. The data set was tested for normality but was shown to be non-normal. However, according to the central limit theorem, the distribution of means from any non-normal distribution can still be considered approximately normal as long as the sample size (n) is larger than 30 participants. Therefore, we used parametric statistics for our analysis as our sample size was greater than 30 participants.

SPSS software (version 24; IBM, Armonk, NY, USA) was used for data analysis, and P < .05 was considered statistically significant.

Floor and ceiling effects

The floor and ceiling effects were calculated as the percentage of patients whose total score fell between 0 and 10 (minimal scores) and between 90 and 100 (maximal scores). As suggested by McHorney and Ware, floor and ceiling effects are defined by using a cutoff of 15%. Therefore, floor and ceiling effects were considered to exist if more than 15% of participants reported minimal or maximal total scores.

Reliability

Internal consistency (using the ICC) is defined as the degree to which items on a questionnaire are correlated with each other when assessed at 1 point in time. The, Cronbach α was calculated with a 95% CI to assess internal consistency at baseline and at 3 months' follow-up. An α of 0.70 to 0.90 was deemed to indicate excellent internal consistency.

Test-retest reliability (longitudinal reliability) measures the extent to which consistent results are obtained at test and retest occasions in subjects in stable condition. A value of 0.70 to 0.80 is deemed appropriate for comparison in research, and a value over 0.90 is deemed appropriate for clinical interpretation.,, Test-retest scores were analyzed using a 2-way mixed model with absolute agreement to produce an intraclass correlation (ICC2,1) with a 95% CI for a single measure., An ICC of 0.80 was considered the minimum standard for good reliability in this study.,

Statistical hypothesis

We expected that the Short-WORC would demonstrate excellent internal consistency (Cronbach α) and test-retest reliability (ICC2,1), defined as values of 0.80 or greater and 0.90 or greater, respectively.

Agreement parameters

Absolute reliability was assessed by calculating the following statistics: SEM and minimal detectable change based on a 90% confidence interval (MDC90). The SEM of agreement (SEMagreement) was calculated using the following equation:

Assuming that our data verify the 2 required assumptions for estimation of MDC90 (ie, normally distributed data and no systematic bias), we used the SEM to calculate the MDC90 with the following equation,:

The SEM provides the estimate of measurement error in the same units as the original measurement, and MDC90 is the minimum amount of change required to be 90% confident that a change has occurred over a period without measurement error. The 95% CI for MDC90 was calculated as d ± MDC90, in which d is the mean difference.,

To calculate the real change over time between groups of patients, we calculated the minimal detectable change for a group given a 90% confidence (MDC90) using the following formula: MDCgroup = MDC90/√n × 1.64, in which n is the sample size of the group., Smaller SEM and MDC values indicate smaller measurement errors.

BA plots

BA plots were used for plotting the limits of agreement (LOAs). The LOA is the difference between scores at time 1 and time 2 of the test-retest period against the mean score for the 2 points. The BA plots produce an image of the results that can be used to evaluate systematic variability (bias), the presence of outliers, and homoscedasticity.,,

Results

The demographic characteristics of the study population are presented in Table I.

Table I

Patient baseline characteristics (N = 162)

Variable	Data
Age, mean ± SD, yr	61.2 ± 16.3 (162 [100%])
Sex, n (%)
Male	79 (48.8)
Female	90 (55.5)
Affected shoulder, n (%)
Left	54 (33.3)
Right	94 (58.1)
Bilateral	14 (8.6)
Occupational information, n (%)
Employed	80 (49.4)
LOA	12 (7.4)
Unemployed	10 (6.2)
Retired	60 (37.1)

SD, standard deviation.

Reliability

There were no floor or ceiling effects; therefore, the Short-WORC is appropriate for rotator cuff pathology. Internal consistency (Cronbach α [95% CI]) was excellent at the baseline assessment, with a value of 0.82 (N = 162), and at 3 months postoperatively, with a value of 0.87 (n = 51). Therefore, the Short-WORC is stable and reflects the true score of the patient. Test-retest reliability was excellent (ICC2,1 = 0.87) (Table II).

Table II

Longitudinal reliability of Short-WORC

	Test-retest reliability
Test mean (SD)	45.6 (23.9)
Retest mean (SD)	45.1 (23.1)
d (SD)	0.5 (11.9)
95% CI for d	–2.8 to 3.4
95% LOAs	–22.9 to 23.8
ICC (95% CI)	0.87 (0.79-0.92)

Short-WORC, short version of Western Ontario Rotator Cuff Index; SD, standard deviation; d, mean difference; CI, confidence interval; LOAs, limits of agreement; ICC, intraclass correlation coefficient.

Agreement parameters

Reported values for SEMagreement (8.4), MDC90 (19.5), and MDC90group (1.7) are shown in Table III. These findings indicate that the Short-WORC is an excellent measure of change within a group of patients.

Table III

Reproducibility: agreement parameters of Short-WORC

	Value
SEMagreement (95% CI)	8.4
MDC90individual	19.5 (–19 to 20)
MDC90group	1.7

Short-WORC, short version of Western Ontario Rotator Cuff Index; SEM, standard error of measurement of agreement; CI, confidence interval; MDC, the minimal detectable change for a group given a 90% confidence.

BA plots

The 95% LOAs for test-retest scores are presented in Table II. Visual inspection shows the random scatter of most points to be within the 95% LOAs and represents negligible systematic bias (error) between scores for the Short-WORC (Fig. 1). Short-WORC scores are stable between time intervals and are reflective of a patient's true score of an intervention.

Figure 1

Bland-Altman limits-of-agreement plot between test and retest scores on short version of Western Ontario Rotator Cuff Index (Short-WORC) (n = 51). The middle line represents the mean of the individual differences (d), and upper and lower lines represent the 95% limits of agreement. Differences lie between d ± 1.96 standard deviation (SD) of the mean difference.

Discussion

This study demonstrated excellent reliability and agreement properties for the Short-WORC when administered to a group of patients with RCDs. Our findings provide strong evidence to support the findings of previous studies that assessed these properties retrospectively., Together, this collection of studies suggests that the Short-WORC is sufficiently reproducible such that clinicians can have confidence in the stability of patient scores,, when making decisions about patient QoL and changes in QoL.,

In this study, we did not observe floor or ceiling effects, which is also consistent with previously published work,,, suggesting that the Short-WORC is well suited to detect both improvement and worsening in the RCD population. The internal consistency (0.82) was acceptable and similar to that reported by Razmjou et al (0.89) and Dewan et al (0.84) at baseline, and was comparable to the Cronbach α (0.85-0.92) of the original WORC depending on the translation.,, Because it has been suggested that values exceeding 0.90 indicate redundancy, the Short-WORC may be more efficient than the WORC.,

An ICC of 0.90 or greater can be difficult to obtain; however, previous literature considered a measure reliable if the point estimate exceeded 0.75.,, The ICCs found in our study were similar to those of previously published work and the WORC.,,, The ICC2,1 value (0.87) exceeded the benchmark of 0.75. Thus, our study provides strong evidence that the Short-WORC has excellent reliability across multiple contexts. On the basis of our narrow CI, we can be confident that our estimate is precise and exceeds minimum expectations.

The SEMagreement value of 8.4 for the Short-WORC found in this study indicates that there is a 68% chance (1 ± SEM) that the true score on the Short-WORC for an individual assessed at a single point in time lies within 8.4 points of the measured score. We used the ICC2,1 absolute agreement value to calculate the SEM instead of the Cronbach α and did not choose to use the Cronbach α to estimate the SEM. Instead, we used the SEMagreement value to compute MDC,,, as it expresses the measurement error through the systematic difference between test and retest scores, which is otherwise ignored with the SEM consistency value.

The MDC90 of the Short-WORC implies that if the individual's score on the Short-WORC has changed by at least 19.5 points, the clinician can be confident that true change (over and above questionnaire error) has occurred. In comparison to the WORC (17.8), we observed that the MDC value is higher for the Short-WORC (19.5). This could be a result of fewer items on the Short-WORC, therefore producing greater variability.

The low MDCgroup value indicates that the Short-WORC is an excellent measure of change within a group of patients. When both MDC90 values (individual vs. group) are compared, the Short-WORC is better at measuring change for a group of patients than for an individual patient. As shown in the literature, a smaller value for MDC90group than for MDC90individual aligns with agreement parameters reported for other PRO measures. This is an expected finding because the formula for MDC90group is dependent on the square root of the sample size, unlike that for MDC90individual, which is dependent on the square root of 2 and the error band around the mean difference of 2 measurements. This is further evident as the group effect will always average out any differences that would normally be highlighted in the individual effect. Therefore, the variability will always be higher for MDC90individual compared with MDC90group. However, measuring both groups of patients and individual patients is important to ensure that the measure is reliable when assessing an individual patient over a test-retest interval and over a period between groups of patients after an intervention.,

The LOAs on BA plots are known to represent the interval within which repeated measures would be expected to fall 95% of the time. The wide 95% LOAs (–22.9 to 23.8) reported in our study reflect large within-individual variability and hence limited usefulness of measures for individual comparisons. We used the retest assessment period of 2 to 7 days as a stable period for patients because it is long enough to prevent recall bias but short enough to expect that no clinical change has occurred given that RCD is a chronic condition. This interval was sufficient according to previous literature but can allow some potential for circumstances to destabilize a patient’s condition. Our assumption of considering 1 week as the time interval was supported by the results of the BA plot, indicating a stable time frame.

The negligible mean difference and acceptable agreement of the Short-WORC reported in this study suggest that the Short-WORC can replace the 21-item WORC for both clinical and research applications. Although we found high values for the LOAs (–22.9 to 23.8), they are similar to those of the WORC (–22.7 to 20.1) and those in our previously published work (–26.5 to 22.3). The agreement parameters are also in accordance with our previously published work.,,

Overall, our findings are consistent with values obtained when the Short-WORC was extracted from its full parent version. Lower internal consistency and reliability with wider variations between test-retest scores can be expected when using abbreviated questionnaires. The goal of shortening a questionnaire is to reduce patient and/or administrative burden while retaining the conceptual linkage to the intended construct and sufficient psychometric equivalence., We assumed that patients required less time to complete the 7 items of the Short-WORC than the 21 items of the original WORC, although we did not directly measure time. In this study, only certain psychometric properties of the Short-WORC were assessed. Therefore, future studies should evaluate comprehension and construct clarity of the Short-WORC through qualitative assessments and should include longitudinal evaluations of responsiveness. Although our previous work supported the responsiveness of the Short-WORC, it was conducted with data collected using the original version of the WORC. Therefore, it is important to understand whether the equivalence between the extracted and isolated versions of the Short-WORC is consistent. In addition, all of the studies to date have been conducted at specialty shoulder surgery clinics; therefore, assessment of populations in different contexts or that include a broader spectrum of RCDs would clarify whether these measurement properties exist in multiple contexts of the disorder.

Conclusion

The Short-WORC had an absence of ceiling and floor effects and showed acceptable internal consistency, as well as excellent reliability, for group comparisons; it showed suitable but imperfect confidence in the test-retest reliability of scores at the level of the individual patient with an RCD. Although reproducibility data are essential, data to evaluate the validity and responsiveness of the Short-WORC are still required.

Disclaimer

Joy C. MacDermid was supported by a Canadian Institute for Health Research Chair in Gender, Work and Health and the Dr. James Roth Research Chair in Musculoskeletal Measurement and Knowledge Translation.

The authors, their immediate families, and any research foundations with which they are affiliated have not received any financial payments or other benefits from any commercial entity related to the subject of this article.