Validation of PROMIS Physical Function for Evaluating Outcome After Acute Achilles Tendon Rupture.

BACKGROUND: There is increased demand for valid, reliable, and responsive patient-reported outcome measures (PROMs) to evaluate treatment for Achilles tendon rupture, but not all PROMs currently in use are reliable and responsive for this condition.
PURPOSE: To evaluate the measurement properties of the Patient-Reported Outcomes Measurement Information System Physical Function (PROMIS PF) compared with other PROMs used after treatment for acute Achilles tendon rupture. STUDY
DESIGN: Cohort study (diagnosis); Level of evidence, 2.
METHODS: A retrospective cohort study with a follow-up questionnaire was performed. All adult patients with an acute Achilles tendon rupture between June 2016 and June 2018 with a minimum 12-month follow-up were eligible for inclusion. Functional outcome was assessed using the PROMIS PF computerized adaptive test (CAT), Foot and Ankle Ability Measure (FAAM) Activities of Daily Living (ADL), FAAM-Sports, and Achilles Tendon Total Rupture Score (ATRS). Pearson correlation (r) was used to assess the correlations between PROMs. Absolute and relative floor and ceiling effects were calculated.
RESULTS: In total, 103 patients were included (mean age, 44.7 years; 74% male); 82 patients (79.6%) underwent operative repair, while 21 patients (20.4%) underwent nonoperative management. The mean time between treatment and collection of PROMs was 25.3 months (range, 15-36 months). The mean scores were 55.4 ± 9.2 (PROMIS PF), 92.9 ± 12.2 (FAAM-ADL), 77.7 ± 22.9 (FAAM-Sports), and 83.0 ± 19.4 (ATRS). The ATRS was correlated with FAAM-ADL (r = 0.80; 95% CI, 0.72-0.86; P < .001) and FAAM-Sports (r = 0.86; 95% CI, 0.80-0.90; P < .001). The PROMIS PF was correlated with the FAAM-ADL (r = 0.66; 95% CI, 0.53-0.75; P < .001), FAAM-Sports (r = 0.65; 95% CI, 0.53-0.75; P < .001), and ATRS (r = 0.69; 95% CI, 0.58-0.78; P < .001). The PROMIS PF did not show absolute floor or ceiling effects (0%). The FAAM-ADL (35.9%), FAAM-Sports (15.8%), and ATRS (20.4%) had substantial absolute ceiling effects.
CONCLUSION: The PROMIS PF, FAAM-ADL, and FAAM-Sports all showed a moderate to high mutual correlation with the ATRS. Only the PROMIS PF avoided substantial floor and ceiling effects. The results suggest that the PROMIS PF CAT is a valid, reliable, and perhaps the most responsive tool to evaluate patient outcomes after treatment for an Achilles tendon rupture.

The incidence of Achilles tendon ruptures is rising not only among young patients but also among an increasingly aging, but active, population. The role of operative versus nonoperative management remains controversial, but determining the most effective solution for any given patient depends on patient-reported outcome measure (PROM) tools that are able to reliably evaluate the success of a chosen clinical treatment strategy.

The Achilles Tendon Total Rupture Score (ATRS) is the most commonly used PROM to evaluate outcomes after the treatment of an acute Achilles rupture because it was the first validated, injury-specific PROM. The Foot and Ankle Ability Measure (FAAM) is used to evaluate a myriad of lower extremity disorders, and it has also been shown to have substantial content relevance to patients with Achilles tendon disorders. The more recently developed Patient-Reported Outcomes Measurement Information System (PROMIS) provides a comprehensive set of questionnaires, and critically, items can be administered as a computerized adaptive test (CAT) to limit the number of questions that a patient must answer to attain a score. The PROMIS Physical Function (PROMIS PF) CAT has been shown to be an excellent method for measuring outcomes for patients with foot and ankle injuries. While all the aforementioned instruments are currently employed to evaluate treatment of lower extremity conditions, the correlation between the validated ATRS, FAAM, and PROMIS PF CAT scores in patients with Achilles tendon ruptures has not been evaluated.

The primary aim of this retrospective study was to evaluate the validity, reliability, and responsiveness of the PROMIS PF compared with the FAAM and ATRS measurement tools in patients with an acute Achilles tendon rupture.

Methods

Study Design

All adult patients who were evaluated at 2 academic medical centers, Massachusetts General Hospital and Brigham and Women's Hospital, with an acute Achille tendon rupture between 2016 and 2018 were eligible for inclusion. Eligible patients were identified by searching for Current Procedural Terminology codes and International Classification of Diseases codes in the institution’s Research Patient Data Registry. Inclusion criteria were (1) acute Achilles tendon rupture, (2) age 18 years or older, and (3) minimum of 12 months of follow-up. Exclusion criteria were (1) treatment for Achilles rerupture, (2) cognitive impairment, (3) and the patient spoke a language other than English. Data collection was performed by reviewing electronic medical records, and after institutional review board approval, eligible patients were invited to participate in the study by a recruitment letter. Questionnaires were collected online and managed using Research Electronic Data Capture (REDCap).

Patient and Treatment Characteristics

Electronic medical records and collected REDCap questionnaires were reviewed to collect baseline patient characteristics regarding age, sex, smoking status, other surgery on the affected leg since initial Achilles treatment, trauma mechanism, Charlson Comorbidity Index (CCI), operative treatment method, nonoperative treatment method, and time from treatment to questionnaire. Tobacco smoking status was subdivided into current, former, or never smoker. The CCI is a method of categorizing and indexing multiple comorbidities. Operative treatment included open and minimally invasive/percutaneous surgery. The operative stitch technique was recorded if noted in the operative report and included Bunnell, Kessler, Krackow, end-to-end, Lindholm/Ma-Griffith, and Kessler/percutaneous. Immobilization methods included the use of a cast, boot, or splint. The time from initiation of treatment to the start of rehabilitation was collected. Full weightbearing status was stratified into <4 weeks and ≥4 weeks. The use of a functional rehabilitation protocol was recorded (eg, gradual reduction of plantarflexion, self-administered exercise program, or formal physical therapy) as well as the use of an accelerated rehabilitation protocol (start early range of motion at <3 weeks).

Patient-Reported Outcome Measures

The collection of PROMs was performed electronically and included the PROMIS PF Version 2.0 CAT, FAAM–Activities of Daily Living (ADL), FAAM–Sports, and ATRS. The PROMIS questionnaires evaluate the limitations of daily activities, pain, and physical activities, with scores ranging from 0 to 100, with higher scores representing higher function and a mean score of 50 for the general population of the United States. The PROMIS PF CAT was developed using the item-response theory to maximize efficient administration; from a calibrated item bank of 124 questions, a minimum number of 4 items must be answered in order to receive a score. The minimal clinically important difference (MCID) of the PROMIS PF CAT is 16 points.

The FAAM was developed to assess physical functioning for individuals with foot- and ankle-related disabilities; items are scored on a 5-point Likert scale from “no difficulty at all” to “unable to do.” Scores are transformed to percentage scores, with higher scores representing higher levels of functioning. The scores for FAAM-ADL and FAAM–Sports are regarded as valid and are generated when patients complete 90% or more of the items. The MCIDs of the FAAM-ADL and FAAM–Sports have been reported to be 8 and 9 points, respectively. The ATRS is an instrument developed specifically for measuring outcome after treatment for Achilles tendon ruptures, with items graded on a 11-point Likert scale according to level of limitations and/or difficulties, from “major limitations” to “no limitations,” with a score of 100 indicating no symptoms and full function. The scores for the ATRS are regarded as valid and generated when patients complete 80% or more of the items. The ATRS has a reported MCID of 10 points. Currently, the ATRS has been identified as the most appropriate PROM to evaluate the management of Achilles tendon ruptures, and thus, it is considered to be the primary comparator. The time from treatment to questionnaire in months was available for all PROMs. Patients completed all the PROM questionnaires electronically at the same time, and they completed the minimum valid answers required to compute the scores.

Statistical Analysis

Descriptive results were presented as mean values with standard deviation and range, median values with interquartile range (IQR), or absolute numbers and percentages. Pearson correlation (r), with 95% CI, was used to assess the relationship between the PROMIS PF, FAAM-ADL, FAAM–Sports, and ATRS. Correlation coefficients of 0.3 or less were considered weak; 0.31 to 0.39, moderate-weak; 0.40 to 0.60, moderate; 0.61 to 0.69, moderate-high; and greater than 0.70, high. Additionally, floor and ceiling effects were assessed for all PROMs. The absolute floor was defined as the percentage of patients with the absolute lowest possible PROM score, and the absolute ceiling as the percentage with the absolute highest possible PROM score. The relative floor was defined as the percentage of patients who reported the lowest PROM score in the cohort, and the relative ceiling as the percentage with the highest PROM score reported in the cohort. Floor or ceiling effects are considered to be substantial if more than 15% of patients achieve the lowest or highest possible score, respectively. The required sample size for studies assessing measurement properties has been advocated to be a sample size of at least 50 patients. The significance level was defined as a P value <.05. All analyses were performed in R Version 3.6.1 (R Development Core Team).

Results

Study Population

In total, 305 patients met the inclusion criteria, of whom 179 patients (58.7%) did not respond, and 23 patients (7.5%) refused participation. This led to the final inclusion of 103 patients (overall response rate, 34%). The different PROM questionnaires were completed by patients at the same time point. The mean time from treatment to PROM completion was 25.3 months (range, 15-36 months). The patient characteristics, stratified by treatment method, are presented in Table 1.

Table 1

Characteristics of the Study Patients

	Overall (N = 103)	Operative Treatment (n = 82)	Nonoperative Treatment (n = 21)
Age at injury, y	44.7 ± 14.6 (19-77)	42.3 ± 12.9 (19-74)	54.1 ± 17.2 (25-77)
Sex
Male	76 (73.8)	63 (76.8)	13 (61.9)
Female	27 (26.2)	19 (23.2)	8 (38.1)
Smoking
Current	8 (7.8)	6 (7.3)	2 (9.5)
Former	14 (13.6)	11 (13.4)	3 (14.3)
Never	81 (78.6)	65 (79.3)	16 (76.2)
Other surgery on leg since Achilles treatment	7 (6.8)	6 (7.3)	1 (4.8)
Trauma mechanism (%)
Sports related	89 (86.4)	73 (89.0)	16 (76.2)
Ground-level fall	5 (4.9)	4 (4.9)	1 (4.8)
Fall from height	1 (1.0)	0 (0.0)	1 (4.8)
Twisting motion	5 (4.9)	3 (3.7)	2 (9.5)
Other	3 (2.9)	2 (2.4)	1 (4.8)
CCI index overall	2.0 ± 1.3 (1-7)	1.8 ± 1.0 (1-5)	2.7 ± 1.8 (1-7)
Time from treatment to PROMs, mo	25.3 ± 5.7 (15-36)	25.9 ± 5.7 (15-36)	23.2 ± 5.1 (16-31)

Data are reported as mean ± SD (range) or n (%). CCI, Charlson Comorbidity Index; PROM, patient-reported outcome measure.

Treatment Method

In total, 82 patients (79.6%) underwent operative repair. The treatment characteristics are shown in Table 2. Open surgery was performed in 69 patients (84.1%), with the Krackow stitch as most used stitch technique (41%). The median duration of operative treatment to start of rehabilitation was 2.0 weeks (IQR, 2.0-5.5 weeks). Nonoperative treatment was performed in 21 patients (20.4%), with use of a boot (48%) as most common method. The mean duration of nonoperative treatment to start of rehabilitation was 4.5 weeks (IQR, 3.0-9.3 weeks). The treatment characteristics are shown in Table 2.

Table 2

Treatment Characteristics of the Study Groups

	Value
Operative treatment (n = 82)
Treatment method
Open surgery	69 (84.1)
Minimally invasive/percutaneous	11 (13.4)
Operative stitch technique (n = 59)
Bunnell	1 (1.7)
Kessler	3 (5.1)
Krackow	24 (40.7)
End-to-end	1 (1.7)
Lindholm/Ma-Griffith	20 (33.9)
Kessler/percutaneous	9 (15.3)
Other	1 (1.7)
Immobilization method
Cast	3 (3.7)
Boot	0 (0.0)
Splint	79 (96.3)
Full weightbearing status (n = 80)
<4 wk	11 (13.8)
≥4 wk	69 (86.2)
Time from treatment to rehabilitation, wk (n = 79)	2.0 (2.0-5.5)
Functional rehabilitation protocol (n = 81)	79 (97.5)
Accelerated rehabilitation protocol (n = 80)	45 (56.2)
Nonoperative treatment (n = 21)
Treatment method
Cast	4 (19.0)
Boot	10 (47.6)
Splint	7 (33.3)
Full weightbearing status (n = 18)
<4 wk	3 (16.7)
≥4 wk	15 (83.3)
Time from treatment to rehabilitation, wk (n = 16)	4.5 (3.0-9.3)
Functional rehabilitation protocol (n = 17)	17 (100.0)
Accelerated rehabilitation protocol (n = 16)	4 (25.0)

Data are presented as n (%) or median (interquartile range).

PROMs: Measurement Properties

The overall mean PROM results were 55.4 ± 9.2 (PROMIS PF), 92.9 ± 12.2 (FAAM-ADL), 77.7 ± 22.9 (FAAM–Sports), and 83.0 ± 19.4 (ATRS). The PROMs stratified by treatment method are shown in Table 3, and the mutual correlations between the different PROMs are presented in Table 4. The ATRS showed a high correlation with the FAAM-ADL (r = 0.80; 95% CI, 0.72-0.86; P < .001) and the FAAM–Sports (r = 0.86; 95% CI, 0.80-0.90; P < .001). PROMIS PF showed a moderate-high correlation with the FAAM-ADL (r = 0.66; 95% CI, 0.53-0.75; P < .001), FAAM–Sports (r = 0.65; 95% CI, 0.53-0.75; P < .001), and ATRS (r = 0.69; 95% CI, 0.58-0.78; P < .001). The floor and ceiling effects for the PROMs are presented in Table 5. The PROMIS PF did not show absolute floor or ceiling effects (0%). The FAAM-ADL (35.9%), FAAM–Sports (15.8%), and ATRS (20.4%) had significant absolute ceiling effects. There were no substantial changes in relative floor and ceiling effects compared with the absolute floor and ceiling effects.

Table 3

Patient-Reported Outcome Measures

	Overall (N = 103)	Operative (n = 82)	Nonoperative (n = 21)
PROMIS PF	55.4 ± 9.2	56.4 ± 9.1	51.5 ± 8.7
FAAM-ADL	92.9 ± 12.2	93.6 ± 11.6	90.3 ± 14.5
FAAM–Sports	77.7 ± 22.9	78.7 ± 22.6	73.5 ± 24.1
ATRS	83.0 ± 19.4	83.9 ± 19.5	79.6 ± 19.5

Data are presented as mean ± SD. ADL, Activities of Daily Living; ATRS, Achilles Tendon Total Rupture Score; FAAM, Foot and Ankle Ability Measure; PF, Physical Function; PROMIS, Patient-Reported Outcomes Measurement Information System.

Table 4

Correlations Between Patient-Reported Outcome Measures

	PROMIS PF	FAAM-ADL	FAAM–Sports	ATRS
PROMIS PF	—	0.66 (0.53-0.75)	0.65 (0.53-0.75)	0.69 (0.58-0.78)
FAAM-ADL		—	0.68 (0.56-0.77)	0.80 (0.72-0.86)
FAAM–Sports			—	0.86 (0.80-0.90)
ATRS				—

Data are presented as Pearson r (95% CI). For all correlations, P < .001. ADL, Activities of Daily Living; ATRS, Achilles Tendon Total Rupture Score; FAAM, Foot and Ankle Ability Measure; PF, Physical Function; PROMIS, Patient-Reported Outcomes Measurement Information System.

Table 5

PROM Floor and Ceiling Effects

	Absolute Floor (%)	Absolute Ceiling (%)	Relative Floor (%)	Relative Ceiling (%)
PROMIS PF	0	0	1.0	2.9
FAAM-ADL	0	35.9	1.0	35.9
FAAM–Sports	0	15.8	1.0	15.8
ATRS	0	20.4	1.0	20.4

Absolute floor/ceiling effect is defined as percentage with absolute lowest/highest possible patient-reported outcome measure (PROM) score; relative floor/ceiling is defined as percentage with lowest/highest PROM score reported in the cohort. ADL, Activities of Daily Living; ATRS, Achilles Tendon Total Rupture Score; FAAM, Foot and Ankle Ability Measure; PF, Physical Function; PROMIS, Patient-Reported Outcomes Measurement Information System.

Discussion

In this cohort study of both the operative and nonoperative functional treatment outcome of Achilles tendon ruptures, the FAAM-ADL, FAAM–Sports, and PROMIS PF all showed a moderate to high mutual correlation with the ATRS. Of these measures, however, it should be noted that only the PROMIS PF CAT avoided substantial floor and ceiling effects.

The overall PROM results from this study demonstrate good to excellent long-term functional outcome after Achilles tendon treatment. The correlation between the ATRS, FAAM, and PROMIS scores in patients with an Achilles tendon rupture has not been previously evaluated. In this study, the ATRS showed a moderate to high correlation with the FAAM-ADL, FAAM–Sports, and PROMIS PF. The ATRS is an injury-specific PROM, which has been evaluated and found to be valid, reliable, and responsive. It has also been confirmed and validated in several languages, and many currently consider it the most appropriate PROM to evaluate the management of Achilles tendon ruptures. Ganestam et al reported the ATRS in 90 patients with a follow-up between 2 and 24 months and showed a moderately strong criterion validity, with a ceiling effect of 8%. However, the test-retest variability showed poor reliability, raising questions regarding the use of the ATRS for repeated assessments of individual patients. Kearney et al evaluated 64 patients and reported that the ATRS demonstrated high internal consistency and responsiveness, with a celling effect of 11% at 9 months of follow-up. The ceiling effect of the ATRS (20.4%) in this study was higher compared with previous reports, which could be because of the shorter follow-up in these studies. Functional outcome is likely to continue to improve with longer follow-up, and the previously reported ceiling effects might have underestimated the actual ceiling effects at long-term follow-up.

The FAAM is used to evaluate a variety of lower extremity disorders and also demonstrates substantial content relevance to patients with Achilles tendon disoders. While the FAAM has been shown to be a reliable, responsive, and valid measure of physical functioning in various lower extremity disorders, it was validated in 164 individuals with a broad range of musculoskeletal disorders, with only 2 patients sustaining an Achilles tendon rupture. Reb et al evaluated the relevance of the FAAM specifically in 75 patients with Achilles tendon disease after a mean of 4 months (range, 0-24 months) and found a substantial content relevance; however, ceiling effects were apparent for the Sports subscale (42.7%). Subgroup analysis was performed based on treatment groups with ceiling effects for the Sports subscale among nonoperative patients (22%) and ceiling effects for the ADL (21%) and Sports (54%) subscales among operative patients. These ceiling effects are similar to the results presented in the present study.

The PROMIS has been shown to be an excellent method for measuring outcomes for patients who have undergone foot and ankle surgery. The PROMIS PF CAT was developed using item response theory to maximize efficient administration from a calibrated item bank of 124 questions and has been shown to result in equally high reliability and less ceiling effects in the assessment of general orthopaedic trauma patients. Hung et al evaluated the performance of the PROMIS PF CAT specifically for adult patients with common disorders of the foot and ankle and found it to be an excellent method for measuring outcomes, with good coverage (floor effect, 0%; ceiling effect, 0.32%) and an average test administration time of 47 seconds. Hung et al recently also reported the responsiveness of both the PROMIS CAT and FAAM–Sports instruments in the orthopaedic foot and ankle population, which included 785 patients and found both to be sensitive and responsive to changes in patient-reported functional health. However, the study included a variety of 43 different disorders without further specifying whether they included Achilles tendon ruptures. They stated that further assessment of the responsiveness of the PROMIS and FAAM–Sports instruments within specific conditions and across different populations is recommended.

The PROMIS questionnaires were developed with the goal of providing standardized, valid, and flexible PROM-collection tools with features that lower response burden and make it possible to seamlessly incorporate them into patients’ medical records. Papuga et al recently explored the implementation of PROMIS CAT tools with 23,813 patients during outpatient clinic visits and reported an average time to completion of 3.5 minutes. There was no significant change in registration times for new patients, showing the implementation of PROMIS to be effective; results, moreover, could be imported directly into the electronic medical record in real time for use during the clinical visit. Ho et al assessed whether preoperative PROMIS PF CAT scores were predictive of functional improvement after operative treatment in foot and ankle patients. They found that patients with scores below 29.7 were likely to improve with surgery, whereas patients with scores above 42 were unlikely to improve. Cutoff values like these could help guide surgeons regarding the most appropriate treatment option for each individual patient. Future research could focus on reporting the prognostic cutoff values of the PROMIS PF CAT scores for Achilles tendon ruptures. The PROMIS instruments are already being used in the evaluation of Achilles tendon disorders.

PROMs are increasingly used in orthopaedic trauma care to evaluate patient-oriented health status in clinical care and research, to assess cost-effectiveness, and, more recently, to influence reimbursement decisions. Today, however, a number of different outcome measures are still used in different Achilles tendon studies. Despite early promising results, PROMIS has not been adopted in most orthopaedic literature. The performance of the PROMIS compared with various legacy “traditional” outcome measures has been evaluated across various conditions that have shown the PROMIS to correlate well with traditional outcome measures used in orthopaedic studies. Validity, reliability, and responsiveness are properties that define the clinical relevance of any outcome instrument, and establishing usefulness is an ongoing process meant to substantiate utility under various conditions and populations. These properties may differ across settings and patient populations. It is therefore important to continue to evaluate different available PROMs in specific patient populations. The correlations found in this study suggest that perhaps PROMIS PF CAT ought to be considered the most useful measure for evaluating patients with an Achilles tendon rupture—particularly when compared with the use of FAAM or ATRS. The PROMIS PF CAT tool did not show substantial floor or ceiling effects, while both the FAAM and the ATRS showed substantial (>15%) ceiling effects. Such effects suggest that extreme items are missing in the upper end of the FAAM and ATRS outcome instruments, indicating limited content validity. Therefore, patients with the highest possible score cannot be distinguished, thus reducing reliability. Furthermore, responsiveness is limited as functional changes cannot be measured in these patients. This might present challenges in studies that explore and evaluate improvement in the more athletic patients with Achilles tendon ruptures.

Limitations

Potential limitations in this study need to be acknowledged. First, the study is limited by the nature of the injuries in a trauma setting; hence, we were unable to collect PROMs at the time of injury to allow for baseline comparison. Second, only 103 (34%) of patients contacted ultimately participated in this study, potentially creating selection bias among those who did agree. Third, the study might be limited by the order in which the PROMs were administered to patients during the electronic collection. Randomization of the order of the PROM questionnaires may have eliminated the potential effects of survey fatigue. Finally, the different PROMs were only evaluated in the English language.

Conclusion

The PROMIS PF, FAAM-ADL, and FAAM–Sports all showed a moderate to high mutual correlation with the ATRS. However, of these measures, only the PROMIS PF CAT avoided substantial floor or ceiling effects. The results of this study strongly suggest that PROMIS PF CAT be considered perhaps the most valid and responsive tool for evaluating function after Achilles tendon rupture—regardless of whether patients are treated operatively or nonoperatively.