Preoperative PROMIS Scores Predict Postoperative Outcomes After Primary ACL Reconstruction.

BACKGROUND: PROMIS (Patient-Reported Outcomes Measurement Information System) scores in patients undergoing anterior cruciate ligament (ACL) reconstruction have not been fully described in the literature to date. The ability of preoperative patient-reported outcome scores to directly predict postoperative outcomes in patients who undergo primary ACL reconstruction is unknown. HYPOTHESIS: Postoperative PROMIS physical function (PF), pain interference (PI), and depression (D) scores in patients who undergo ACL reconstruction will show improvement when compared with preoperative scores. Additionally, preoperative PROMIS PF, PI, and D scores will predict which patients will not achieve a minimal clinically important difference (MCID) postoperatively. STUDY
DESIGN: Cohort study; Level of evidence, 3.
METHODS: A total of 233 patients who underwent primary ACL reconstruction between 2015 and 2016 and had completed PROMIS measures both preoperatively (within 60 days of surgery) and postoperatively (100-240 days after surgery) were included in this study. PROMIS PF, PI, and D scores were compared. Accuracy analyses were performed to determine whether preoperative PROMIS scores from each domain could predict postoperative achievement of MCID in the same domain. Cutoff scores were then calculated.
RESULTS: PROMIS PF, PI, and D scores all showed a significant improvement after ACL reconstruction (all P < .001). Preoperative scores from all 3 PROMIS domains showed a strong ability to predict clinically meaningful improvement, as defined by MCID, with areas under the receiver operating characteristic curve from 0.72 to 0.84. Optimal cutoffs for preoperative PROMIS scores showed that patients with a PF score of <42.5, PI score of >56.2, or D score of >44.8 were more likely to achieve MCID.
CONCLUSION: PROMIS PF, PI, and D scores improved significantly in patients who underwent primary ACL reconstruction. Preoperative PROMIS PF, PI, and D scores were highly predictive of outcome in the early postoperative period. The reported cutoff scores showed high probability in predicting which patients would and would not achieve a clinically meaningful improvement.

Patient-reported outcomes (PROs) are rapidly being incorporated into electronic medical records throughout orthopaedic clinical practices. PROs are becoming not only a research tool but also a standard instrument to capture patient-oriented indicators,[5] and there has been increasing interest in to determining the ability of PROs to predict which patients are the best candidates for certain orthopaedic treatments. Anterior cruciate ligament (ACL) reconstruction is one of the most common orthopaedic interventions, with more than 120,000 surgeries performed each year in the United States.[15] Although outcomes after ACL reconstruction are generally thought to be excellent, outcome studies have produced variable results.[4,6,8,9,23]

Previous research has shown that PROs can be an effective tool to identify patient factors associated with poor outcomes after ACL reconstruction. Obesity, smoking status, education, age, and ethnicity have all been extracted from PRO data and identified as possible predictors of worse outcomes.[16,21] However, none of these studies has investigated whether baseline PRO scores can directly predict postoperative patient outcomes. As well, these PROs were administered via lengthy paper forms, a model that may prove too burdensome for routine clinical use. Therefore, the use of a higher throughput PRO measure in patients with ACL reconstruction is needed to provide broadly applicable and more clinically relevant data. PROs with predictive capabilities could be used as powerful tools to stratify patients and predict those most and least likely to have successful outcomes. This would provide valuable information for counseling to help patients make well-informed decisions. Patients who are identified to be at high risk for poor outcomes could receive supplementary preoperative and postoperative interventions to maximize their chance for a satisfactory outcome.

The Patient-Reported Outcomes Measurement Information System (PROMIS) is a dynamic instrument that reliably and validly measures PROs through the use of an item response theory (IRT)–based computer-adaptive testing (CAT) program.[13] The PROMIS CAT data are captured entirely electronically, eliminating the need for data entry, and the program has been shown to have greatly reduced assessment times compared with paper forms.[14] Papuga et al[18] demonstrated that PROMIS could detect decreases in physical function (PF) at 3 to 10 weeks after ACL reconstruction, which the current gold standard International Knee Documentation Committee (IKDC) scale was unable to detect. Preoperative PROMIS scores have been shown to predict postoperative PROMIS scores in patients with foot and ankle abnormalities and in patients undergoing arthroplasty.[2,3,12,24] The growing body of literature supports the strength of the PROMIS as an orthopaedic PRO measurement.[5,19] However, the ability of preoperative PROMIS scores to predict postoperative PROMIS scores in patients who have undergone ACL reconstruction is currently unknown.

The primary purpose of this study was to compare PROMIS PF, pain interference (PI), and depression (D) scores collected within the 60 days prior to surgery with those taken 100 to 240 days after surgery in patients who underwent primary ACL reconstruction. The secondary purpose was to determine the accuracy of preoperative PROMIS scores in predicting postoperative outcomes in these patients. Our first hypothesis was that all PROMIS scores would improve from preoperative to postoperative assessment for the patient group as a whole. Our second hypothesis was that preoperative PROMIS PF, PI, and D scores would accurately predict which patients would attain a minimal clinically important difference (MCID) when we compared preoperative and postoperative scores.

Methods

A retrospective analysis of perioperative PROMIS scores in patients who underwent primary ACL reconstruction was performed by use of protocols approved by our institution’s research subjects review board prior to implementation. Patients who underwent primary ACL reconstruction between February 2015 and August 2016 at our institution were identified through Current Procedural Terminology (CPT) code 29888. All patients underwent routine, standard-of-care, preoperative and postoperative treatment and rehabilitation.[1] Inclusion criteria included primary ACL reconstruction and completed pre- and postoperative PROMIS PF, PI, and D measures. Exclusion criteria included CPT codes documenting concomitant infections and fractures.

The first time interval of within 60 days prior to surgery was selected to maximize the chance of capturing a patient’s last preoperative visit. If a patient had more than one visit during this time frame, the scores from the date closest to surgery were used. The second time interval of 100 to 240 days after surgery was chosen to best capture the return-to-play phase of ACL rehabilitation, which in our practice involves bicycling, unbraced walking, and jogging progressions at around 3 to 4 months postoperatively, then initiation of return to sport-specific training at around 5 to 6 months. Papuga et al[18] previously demonstrated that the period of greatest improvement in PROMIS scores occurred around 20 weeks (4.5 months), with less substantial improvements in PROMIS scores after this time. Our selected second time frame captures patients in the early postoperative phase during this period of maximal improvement; PROMIS scores from this time closely resemble findings at further postoperative visits. If a patient had more than 1 visit during this time frame, the scores from the time point farthest from surgery were used. The 2 sets of scores were compared to determine improvement in each PROMIS domain after operative intervention.

Patients completed the 3 PROMIS domains on an iPad (Apple Inc) during their clinic visits as part of our standard of care. Clinic staff were available for technical support and to answer any questions during the completion process. The PROMIS PF, PI, and D domains have previously demonstrated concurrent validity with other legacy scales.[7,14,18] Custom software was used with Assessment Center (http://www.assessmentcenter.net) application program interface, which was run on the university servers behind the firewall, reducing data privacy concerns. Domains for PF (version 1.2), PI (version 1.1), and D (version 1.0) were included.

Given the CAT nature of PROMIS, each patient’s exact questions, sequence of questions, and number of questions differs based on his or her responses. The PROMIS CAT algorithm for each scale produces standardized T scores based on a normative population within the United States (US). The normalized mean T score for all domains is 50, with a standard deviation of 10. Of note, higher PF scores indicate increased function, higher PI scores indicate increased pain, and higher D scores indicate increased depression.

Statistical Analysis

Individual comparisons of PROMIS scores based on sex, age, and time were analyzed via a 1-way analysis of variance (ANOVA). Tukey post hoc analysis was then used to determine specific differences. It was determined that to detect an effect size equivalent to MCID (the value 4 was used based on MCID calculations) with an SD of 10, the study needed N = 98 to achieve 80% power. All available data were used. Associations among pre- and postoperative scores as well as interdomain associations were quantified by use of the Pearson correlation coefficient.

All PROMIS scores were assessed for discriminatory ability in predicting a postoperative improvement (of the same domain) that reached the threshold of MCID, as defined by one-half SD of the preoperative sample,[17,20,25] through use of receiver operating characteristic (ROC) curve analyses, with overall accuracy summarized using the area under the ROC curve (AUC). MCID was calculated to be a postoperative increase in PF score by 4.6, a postoperative decrease in PI score by 4.0, and a postoperative decrease in D score by 4.9. Corresponding P values for testing the hypothesis that the AUC is 0.5 (no discriminatory ability) were also computed. An AUC of 0.70 or higher represented a statistically valid predictive model. All analyses used a significance level of 5%. SPSS software (IBM) was used for all statistical analyses.

Results

From a review of 3204 patient visits identified via CPT code 29888, we identified 619 unique individuals who had undergone primary ACL reconstruction surgery during our study period. From that group, 233 patients had completed the requisite PROMIS questionnaires and met the inclusion criteria for the study. The cohort included 118 (51%) male and 115 (49%) female patients, with an average ± SD age of 26.5 ± 12.5 years. The average preoperative visit was 20.1 ± 12.5 days before surgery, and the average postoperative visit was 137.0 ± 27.2 days after surgery. None of the PROMIS domains had a significant difference regarding change in T score from preoperative to postoperative time points based on patient age or sex (P values = .320-.990). We also found no significant time effect on the T score of any PROMIS domain during the preoperative (P = .541) or postoperative time frames chosen (P values = .255-.891). The mean pre- and postoperative PROMIS domain scores are provided in Table 1.

TABLE 1

Overall PROMIS Scores

PROMIS Domain	Preoperative	Postoperative
Physical function	41.1 (9.2)	46.6 (7.0)
Pain interference	55.3 (7.9)	47.3 (8.8)
Depression	45.3 (9.8)	40.0 (8.2)

Values expressed as mean (SD). PROMIS, Patient-Reported Outcomes Measurement Information System.

On average, patients had statistically significant preoperative to postoperative improvements of 5.5 in PF, −8.0 in PI, and −5.3 in D (all P < .001). Each of these average changes reached the level of MCID calculated for each separate domain (PF MCID = 4.6; PI MCID = –4.0; D MCID = –4.9). Scatter plots for each PROMIS domain showing the relationship between the preoperative T score and the change in T score after surgery are seen in Figures 1 through 3. Linear regression analysis showed R 2 values of 0.490, 0.177, and 0.392 for PROMIS PF, PI, and D, respectively.

Figure 1.

Scatter plot of preoperative Patient-Reported Outcomes Measurement Information System (PROMIS) physical function T scores versus the change in T score from preoperative to postoperative assessment. A linear regression line yielded R 2 = 0.490. The solid black line represents the optimal cutoff score (42.5). Patients to the right of the solid line have a 77.3% risk of no minimal clinically important difference (MCID), and those to the right of the dotted line (46.7) have a 95% chance of no MCID. Patients to the left of the solid line have a 23.5% risk of no MCID, and those to the left of the dashed line (35.3) have a 95% chance of achieving MCID.

Figure 2.

Scatter plot of preoperative Patient-Reported Outcomes Measurement Information System (PROMIS) pain interference T scores versus the change in T score from preoperative to postoperative assessment. A linear regression line yielded R 2 = 0.177. The solid black line represents the optimal cutoff score (56.2). Patients to the left of the solid line have a 51.2% risk of no minimal clinically important difference (MCID), and those to the left of the dotted line (46.3) have a 95% chance of no MCID. Patients to the right of the solid line have an 18.9% risk of no MCID, and those to the right of the dashed line (63.0) have a 95% chance of achieving MCID.

Figure 3.

Scatter plot of preoperative Patient-Reported Outcomes Measurement Information System (PROMIS) depression T scores versus the change in T score from preoperative to postoperative assessment. A linear regression line yielded R 2 = 0.392. The solid black line represents the optimal cutoff score (44.8). Patients to the left of the solid line have an 84.1% risk of no minimal clinically important difference (MCID), and those to the left of the dotted line (39.5) have a 95% chance of no MCID. Patients to the right of the solid line have a 35.0% risk of no MCID, and those to the right of the dashed line (56.9) have a 95% chance of achieving MCID.

A post hoc analysis was conducted to predict the percentage of patients who met the MCID for each PROMIS scale; 54% of patients met the MCID for PF, 64% met the MCID for PI, and 41% met the MCID for D. The accuracy analysis of the preoperative PROMIS scores suggested that the PF, PI, and D domains significantly predicted clinically meaningful improvement, as defined by MCID. ROC analysis for PROMIS PF, PI, and D yielded significant AUCs of 0.826, 0.715, and 0.836, respectively (all P < .001). Next, 95% specificity cutoffs for achieving and failing to achieve MCID were determined for each domain (Table 2). As an example, for PROMIS PF, patients with a preoperative score of 35.3 or below had a 95% chance of achieving MCID, while those with a score of 46.7 or above had a 95% chance of failing to achieve MCID.

TABLE 2

Accuracy Analysis of Predictive Abilities of Preoperative PROMIS Scores

PROMIS Domain	AUC	P	95% Likelihood of MCID	95% Likelihood of No MCID
Physical function	0.826	<.001	≤35.3	≥46.7
Pain interference	0.715	<.001	≥63.0	≤46.3
Depression	0.836	<.001	≥56.9	≤39.5

AUC, area under the receiver operating characteristic curve; MCID, minimal clinically important difference; PROMIS, Patient-Reported Outcomes Measurement Information System.

Optimal cutoff scores were calculated for PROMIS PF, PI, and D.[11] The optimal cutoffs were used to calculate risks and odds of failing to achieve MCID and are documented in Table 3. As an example, patients who had a preoperative PROMIS PF score below 42.5 had a 23.5% risk, or 0.31 odds, of not reaching MCID. Patients above the 42.5 cutoff had a 77.3% risk, or 3.4 odds, of not achieving MCID. This correlates to a 3.3 times higher risk and 11.1 times higher odds of not achieving MCID for patients with a PROMIS PF above 42.5.

TABLE 3

Optimal Cutoffs for Preoperative PROMIS Scores and Associated Risks and Odds

PROMIS Domain	Optimal Cutoff	Low Risk of No MCIDb	High Risk of No MCIDc	Low Odds of No MCIDb	High Odds of No MCIDc	Increased Riskd	Increased Oddsd
Physical function	42.5	23.5%	77.3%	0.31	3.4	3.3	11.1
Pain interference	56.2	18.9%	51.2%	0.23	1.0	2.7	4.5
Depression	44.8	35.0%	84.1%	0.54	5.3	2.4	9.8

MCID, minimal clinically important difference; PROMIS, Patient-Reported Outcomes Measurement Information System.

Low risk of no MCID and low odds of no MCID are for patients with a score below the physical function (PF) optimal cutoff, above the pain interference (PI) cutoff, or above the depression (D) cutoff.

High risk of no MCID and high odds of no MCID are for patients with a score above the PF optimal cutoff, below the PI cutoff, or below the D cutoff.

Increased risk = High risk of no MCID ÷ Low risk of no MCID. Increased odds = High odds of no MCID ÷ Low odds of no MCID.

Discussion

PRO measures are being implemented throughout orthopaedic clinical practices, giving patients a voice in their health care. PROMIS is a high-throughput PRO that has proven to be valid and accurate in multiple orthopaedic patient populations. Specifically, PROMIS PF has been previously validated in patients who have undergone ACL reconstruction.[18] The current study found that preoperative PROMIS scores for PF, PI, and D were predictive of postoperative outcomes in patients who underwent primary ACL reconstruction, validating the original hypothesis.

Patients in our study showed significant postoperative improvements in PROMIS PF (mean, 5.5), PI (mean, –8.0), and D (mean, –5.3) scores. Postoperative PROMIS PF scores demonstrated that patients continued to have decreased PF (46.6) when compared with the normal US population, with only 54% of patients meeting PF = MCID. These findings are most likely attributable to the postoperative follow-up period of 4.5 months (mean, 137 days). However, the improvement in PROMIS PF following ACL reconstruction as seen in this study is similar to previously described findings. Papuga et al[18] showed that the postoperative time frame captured by this study reflected the time of greatest improvement in PF. Therefore, patients’ PROMIS PF scores should continue to improve at longer follow-up times, but the increases will be smaller in magnitude.

Postoperative PROMIS PI scores showed that patients had less PI (47.3) compared with the US population, a substantial change from the preoperative PI of 55.3. Clinically, an ACL injury is acutely painful for the patient, but with immobilization and bracing, pain often recedes. The improvement in PI seen in our patients postoperatively could be due to timely clinic visits, with preoperative PROMIS scores captured in the very early postinjury phase. Furthermore, the PROMIS PI questionnaire is not designed to quantify the exact amount of pain a patient is having; instead, the score captures a patient’s perceived interference in daily activities secondary to pain. Given this distinction, it is possible that some patients may feel limited in certain activities preoperatively out of fear that an activity may be painful, without actually experiencing pain, and they may answer the PROMIS questionnaire in a way that increases their PI scores.

Patients continued to have less D than the normal population postoperatively (D scores improved from 45.3 to 40.0). The significant improvement in D scores postoperatively is similar to findings in a recent study, although different PRO measures were used.[10] To our knowledge, this is the first study to describe perioperative PROMIS PI and D scores in patients who underwent ACL reconstruction.

Preoperative PROMIS PF, PI, and D scores showed significant accuracy in predicting which patients would reach MCID, with AUCs of 0.72 to 0.84. Although the predictive ability of PROs has not been studied before in ACL patients, similar studies have been performed in other orthopaedic patient populations. Berliner et al[2,3] found that the 12-Item Short-Form Health Survey (SF-12) physical component score as well as the Knee injury and Osteoarthritis Outcome Score (KOOS) and Hip Disability and Osteoarthritis Outcome Score (HOOS) predicted the attainment of MCID in arthroplasty patients, with an AUC of 0.62 to 0.83. Ho et al,[12] studying patients who had foot and ankle surgery, found that PROMIS PF and PI scores predicted attainment of MCID with an AUC of 0.73 to 0.83. The current study shows a predictive ability that matches the most accurate predictive abilities described previously.

Our cutoff points can be applied directly to patient care. The 95% specificity cutoffs and the optimal cutoff can be used in unison to interpret each distinct PROMIS score and stratify patients based on risk of failure to reach MCID. For example, imagine that a patient with an ACL tear is evaluated at a clinic and scores 45 for PF, 65 for PI, and 50 for D. A PF score of 45 is above the optimal cutoff of 42.5 but below the 95% specificity cutoff of 46.7. Therefore, using Table 3, we find that this patient has a 77.3% chance of failing to achieve MCID. Compared with someone who has a PF score below 42.5, this patient has a 3 times higher risk and 11 times higher odds of failing to reach MCID. However, the patient’s PI score is above both the optimal cutoff (56.2) and the 95% specificity cutoff (63.0). Therefore, the patient has a 95% chance of achieving MCID in terms of PI. Last, the patient’s D score falls between the optimal cutoff of 44.8 and the 95% specificity cutoff of 56.9. Therefore, this patient has a 35% chance of failing to reach MCID.

Taking these findings together, we can predict that postoperatively, this patient has a low likelihood of achieving significant improvement in PF, a very high likelihood of having improved pain, and a strong likelihood of having improved mental health. For this patient, preoperative discussion would emphasize operative intervention for relief of symptoms such as pain and depression, while cautiously preparing the patient for the possibility that his or her function may not dramatically improve. This example illustrates that the predictive ability of preoperative PROMIS scores and the subsequent cutoff scores are very powerful tools that clinicians can use to characterize each individual patient’s chances of achieving successful outcomes. This new information can be used in conjunction with the previously established risk factors for poor outcomes following ACL reconstruction[16,22] to give patients more accurate expectations postoperatively and to guide the decision-making process.

The current study has limitations, including potential selection bias, retrospective nature, and length of follow-up. A selection bias is possible because patients who are doing well are less likely to return to clinic. Since only patients who completed the PROMIS survey at both time points were included, our study population may have been skewed to include a higher proportion of those who were having poorer outcomes. However, given that our second time point was 100 to 240 days postoperatively and many patients were still in the ACL rehabilitation phase during that time and therefore more likely to appear in clinic, this potential bias was, we hope, minimal. Patient participation in completing the PROMIS questionnaires may also be subject to technological bias, as patients who were uncomfortable with the electronic survey format may have opted to decline participation even with staff assistance. Furthermore, this was a retrospective study and therefore was weakened by the typical, inherent biases of studies of this nature. Finally, our postoperative time point was only 100 to 240 days after surgery. Our rationale for choosing this time frame was previously discussed. Although longer follow-up is desirable, as previously stated, prior work has shown that this time frame represents the phase of most rapid improvement after ACL reconstruction.[18] Therefore, our data best describe this period of swift improvement directly following surgery rather than longer term outcomes. Further work specifically investigating long-term PROMIS scores after ACL reconstruction is a natural next step following this study.

Conclusion

PROMIS PF, PI, and D scores showed significant pre- to postoperative improvements in patients who underwent primary ACL reconstruction, and preoperative PROMIS PF, PI and D scores were highly predictive of postoperative outcome. The reported optimal preoperative PROMIS cutoff values provided high probability for predicting those patients who would achieve a clinically meaningful improvement. Using these cutoff values, surgeons can stratify patients and appropriately counsel those who are less likely to achieve excellent outcomes. This powerful information can help both the surgeon and the patient in maintaining reasonable expectations and can maximize patient satisfaction.