Preoperative Patient-Reported Outcomes Measurement Information System Computerized Adaptive Testing (PROMIS CAT) Scores Predict Achievement of Minimum Clinically Important Difference Following Anterior Cruciate Ligament Reconstruction Using an Anchor-Based Methodology.

PURPOSE: To determine the change in Patient-Reported Outcomes Measurement Information System Computerized Adaptive Testing (PROMIS CAT) scores for physical function, pain interference, and depression that constitute minimum clinically important difference (MCID) using an anchor-based technique and to identify pre-operative clinical thresholds in anchor-based MCID that predict likelihood of achieving MCID following anterior cruciate ligament (ACL) reconstruction.
METHODS: Adult patients aged 18 years or older undergoing ACL reconstruction that completed both preoperative and postoperative PROMIS CAT assessments and an anchor-based questionnaire were identified over a 23-month period. Anchor-based MCID was determined for PROMIS CAT forms for physical function (PROMIS PF CAT), pain interference (PROMIS PI CAT), and depression (PROMIS D CAT).
RESULTS: A total of 137 patients were included for statistical analysis, with pre-operative PROMIS CAT forms completed 27.9 ± 31.2 days before surgery and 492.5 ± 219.9 days postoperatively on average. Statistically significant improvements were observed for all PROMIS CAT domains. PROMIS PF CAT improved from 39.5 ± 8.2 to 55.0 ± 9.7 (P < .0005), PROMIS PI CAT from 59.8 ± 7.2 to 48.2 ± 8.3 (P < .0005), and PROMIS D CAT from 47.9 ± 8.8 to 41.5 ± 8.6 (P < .0005). Anchor-based MCID for each PROMIS CAT form was calculated to be +4.5, -5.4, and -4.1 for PROMIS PF CAT, PROMIS PI CAT, and PROMIS D CAT, respectively. Mean difference between preoperative and postoperative PROMIS CAT scores exceeded MCID for all domains. The percentage of patients achieving MCID for PROMIS PF CAT, PROMIS PI CAT, and PROMIS D CAT was 85%, 72%, and 55%, respectively. After introduction of 95% specificity cutoffs, the percentage of patients achieving MCID for PROMIS PF CAT, PROMIS PI CAT, and PROMIS D CAT increased to 100% (<35.6 cutoff score), 92% (>65.7 cutoff score), and 83% (>57.5 cutoff score), respectively.
CONCLUSIONS: According to anchor-based analysis of PROMIS CAT MCID, ACL reconstruction is effective in improving physical function, pain interference, and depression symptoms. In addition, preoperative PROMIS CAT scores can predict the likelihood of achieving MCID postoperatively. LEVEL OF EVIDENCE: Level IV, prognostic case series.

Patient-reported outcome measures (PROMs) are used in orthopaedic surgery to assess clinical outcomes from the patient’s perspective. The National Institutes of Health Patient-Reported Outcomes Measurement System (PROMIS) is a standardized PROM that has emerged as an increasingly popular tool for measuring patient reported outcomes in response to treatment. In particular, the computer adaptive test (CAT) version of these forms has demonstrated favorable survey and psychometric properties when compared with “legacy” PROMs, such as the Health Assessment Questionnaire or Medical Outcome Study Short Form-36 Survey, in orthopaedic sports medicine patient populations due to efficient data collection involving fewer questions.2, 3, 4, 5 These CAT questionnaires use item response theory to customize question delivery based on real-time patient answers, leading to 4- to 12-item forms that correlate strongly with other PROMs, while remaining highly reliable., In addition, PROMIS CAT health domains allow for a multifaceted evaluation of patient outcomes by interpreting pain interference (PROMIS PI CAT), physical function (PROMIS PF CAT), and depression (PROMIS D CAT) separately.,,

Recently, PROMIS CAT measures have demonstrated utility in measuring responsiveness to treatment following various orthopedic interventions.8, 9, 10 However, postoperative statistical score improvements may not indicate clinical improvement. To understand whether a patient has improved clinically (as opposed to statistically), the score change correlating to the minimal clinically important difference (MCID) must be identified. There are 2 techniques commonly used to identify the score change needed for achieving MCID. The first technique, called the distribution-based (DB) technique, defines the MCID as one half of the standard deviation of the preoperative scores from a cohort. The second technique, called the anchor-based (AB) technique, correlates score changes with anchor questions that denote clinical improvement from the patient’s perspective., The AB technique is considered more precise, as score changes are correlated to actual subjective metrics of symptom improvement, as opposed to statistical correlations in the DB technique.,, Despite this recognition, the AB technique is less frequently reported on when compared with the DB technique due to the increased challenge of collecting postoperative anchor assessments.

In patients undergoing anterior cruciate ligament reconstruction (ACLR), the MCID for PROMIS CAT measures have been identified previously using a DB technique. Chen et al. demonstrated that PROMIS PF CAT, PROMIS PI CAT, and PROMIS D CAT scores improved significantly in patients who underwent ACLR, and that preoperative scores were highly predictive of postoperative outcomes. In addition, preoperative score ranges for PROMIS CAT assessments have been correlated to likelihood of achieving MCID.17, 18, 19 This knowledge can be used in preoperative shared decision-making with patients. However, the MCID for PROMIS CAT measures using an AB technique has not been studied for patients undergoing ACLR, and neither have the score ranges that correlate with likelihood of achieving MCID. The AB method improves upon the DB method by associating the pre- and postoperative difference in outcome measure score to a meaningful external anchor question, which represents the patient’s perspective, thereby optimally representing clinical changes from a patient’s perspective.,,,

The purposes of this study were to determine the change in PROMIS CAT scores (for PF, PI, and D) that constitute MCID using an AB technique and to identify preoperative clinical thresholds in AB MCID that predict likelihood of achieving MCID following ACL reconstruction. Given that the defined standard deviation of PROMIS CAT forms are defined to be 10 points, we hypothesized that AB MCIDs would similarly approximate 5 points. Moreover, we hypothesized that patients would be more likely to achieve MCID for physical function if they exhibit low preoperative PROMIS PF CAT scores, whereas those with high preoperative PROMIS PI CAT and PROMIS D CAT scores would be more likely to achieve MCID for those respective domains.

Methods

This investigation was a retrospective case series study that received institutional review board approval from Henry Ford Hospital. Patients undergoing primary ACLR by 1 of 2 fellowship-trained sports surgeons were identified over a 23-month period from July 2017 to June 2019. Three PROMIS CAT forms (PROMIS PF CAT, PROMIS PI CAT, PROMIS D CAT) were prospectively collected from all patients. PROMIS CAT forms were administered on an iPad (Apple Inc., Cupertino, CA) or via e-mail message by using REDCap, an electronic, secure platform for recording PROMIS CAT forms (REDCap, Vanderbilt University, Nashville, TN). Inclusion criteria were adult patients aged 18 years or older who completed all preoperative PROMIS forms as well as postoperative PROMIS forms and anchor questionnaire items. Patients were excluded if they were unable to answer PROMs (e.g., language barrier), did not complete preoperative PROMIS forms, or did not have an available e-mail address for postoperative PROMIS form and anchor questionnaire administration. In addition, a chart review was conducted to record demographic factors such as age, ethnicity, sex, and smoking status, as well as surgical factors such as side of surgery, graft type, meniscal injury, and concomitant procedures.

PROMIS CAT forms were collected preoperatively as well as postoperatively. Three clinical anchor questions inquiring about patient perception of improvement in physical function, pain, and mental health relative to presurgery were administered electronically with each postoperative PROMIS CAT survey. The anchor questionnaire used was adapted from previously validated anchor questions in the literature and included question items that respectively assessed physical function, pain, and depression as displayed in Fig 1, with 2 separate groups representing “significant improvement” and “no significant improvement” determined for each question item based on responses., According to PROMIS CAT scoring, a score of 50 represents the mean score of the reference population within any given health domain, with 10 points denoting one standard deviation. All PROMIS CAT forms were administered by research assistants in the institution’s Department of Orthopedics who were either current medical students or full-time employees currently taking a gap year from medical school to conduct research. Preoperative PROMIS forms considered were administered at the patient’s clinic visit closest to the date of their surgery, whereas postoperative PROMIS forms and anchor questionnaire items were collected either at clinic visits or remotely via e-mail with the most recent forms being considered.

Fig 1

Overview of anchor questions used for minimal clinically important difference (MCID) patient groupings. For calculation of MCID, differences between the “no significant improvement group” and the “significant improvement group” were assessed.

Statistical Analysis

Paired samples t tests were used to compare preoperative and postoperative PROMIS CAT scores. MCID was calculated according to AB methodology. AB MCID was determined using independent samples t tests on anchor question responses to determine statistical differences between responses indicating “no significant improvement” and “significant improvement.” Once confirmed, respective mean PROMIS CAT domain changes were identified as the MCID. In addition, all PROMIS CAT scores were assessed for discriminatory ability in predicting a postoperative improvement (of the same domain) that reached the threshold of MCID through use of receiver operating characteristic curve analyses, with overall accuracy summarized using the area under the ROC curve (area under the curve [AUC]). AUC values were interpreted as follows: weak (0.5-0.599), moderate (0.6-0.699), strong (0.7-0.799), and excellent (>0.8). Prognostic cutoffs for achievement, or failure to achieve clinically significant outcomes, were set using 95% specificity. All analyses used a significance level of 5%. SPSS software was used for all statistical analyses (IBM Corp. Released 2017. IBM SPSS Statistics for Windows, Version 25.0. IBM Corp., Armonk, NY).

Results

One hundred eighty-nine patients who underwent ACLR and fully completed preoperative PROMIS CAT forms who were available at follow-up were assessed for inclusion. A total of 137 patients completed all preoperative PROMIS forms as well as postoperative PROMIS forms and anchor questionnaire items and were included for statistical analysis. Fig 2 displays the included and excluded patients undergoing primary ACLR during the study period. The patient population consisted of 82 male subjects (60%) and 55 female subjects (40%), with an average age of 30.3 ± 11.4 years. Preoperative PROMIS CAT forms were completed 27.9 ± 31.2 days before surgery and 492.5 ± 219.9 days post-operatively on average. Complete demographic characteristics are summarized in Table 1.

Fig 2

Enrolled patient cohort with inclusion and exclusion criteria. (ACL, anterior cruciate ligament; PROMIS D CAT, Patient-Reported Outcomes Measurement Information System Computer Adaptive Test forms for Depression; PROMIS PF CAT, Patient-Reported Outcomes Measurement Information System Computer Adaptive Test form for Physical Function; PROMIS PI CAT, Patient-Reported Outcomes Measurement Information System Computer Adaptive Test forms for Pain Interference.

Table 1

ACLR Cohort Demographics

Average age, y	30.3 ± 11.4
Sex	Total	%
Male	82	60
Ethnicity	Total	%
Not Hispanic/Latino	120	87.6
Hispanic/Latino	8	5.8
Declined to answer/unknown	9	6.6
Smoker	Total	%
Never	115	83.9
Current	12	8.8
Former	10	7.3
Side of surgery	Total	%
Bone–tendon–patellar–bone	102	74.5
Hamstring	31	22.6
Quadriceps	3	2.2
Unspecified	1	0.7
Graft Source	Total	%
Allograft	9	6.6
Concomitant procedure	Total	%
Medial meniscectomy	17	12.4
Lateral meniscal repair	9	6.6
Medial meniscal repair	16	11.7
Meniscal debridement	1	0.7
Medial cruciate ligament repair	3	2.2
Posterior cruciate ligament repair	1	0.7
Lateral cruciate ligament repair	2	1.5
Allograft augment	1	0.7

ACLR, anterior cruciate ligament reconstruction.

Across the entire cohort, statistically significant improvements were observed for all analyzed PROMIS CAT domains. PROMIS PF CAT improved from 39.5 ± 8.2 to 55.0 ± 9.7 (P < .0005), PROMIS PI CAT improved from 59.8 ± 7.2 to 48.2 ± 8.3 (P < .0005), and PROMIS D CAT improved from 47.9 ± 8.8 to 41.5 ± 8.6 (P < .0005). With AB methodology, the MCID for each PROMIS CAT form was calculated to be +4.5, –5.4, and –4.1, for PROMIS PF CAT, PROMIS PI CAT, and PROMIS D CAT, respectively. Average change in PROMIS CAT scores before and after surgery exceeded MCID for all domains (Table 2). AUC values were 0.756 (PROMIS PF CAT), 0.728 (PROMIS PI CAT), and 0.761 (PROMIS D CAT), indicating strong predictive ability.

Table 2

PROMIS CAT Scores and AB MCID

PROMIS CAT Domain	Average Scores			AB MCID
	Preoperative	Postoperative	Change	MCID	ROC
PROMIS PF CAT	39.5 ± 8.2	55 ± 9.7	+15.5	+4.5	.756
PROMIS PI CAT	59.8 ± 7.2	48.2 ± 8.3	–11.6	–5.4	.728
PROMIS D CAT	47.9 ± 8.8	41.5 ± 8.6	–6.4	–4.1	.761

AB MCID, minimum clinically important difference calculated according to anchor-based methodology; PROMIS D CAT, Patient-Reported Outcomes Measurement Information System Computer Adaptive Test forms for Depression; PROMIS PF CAT, Patient-Reported Outcomes Measurement Information System Computer Adaptive Test form for Physical Function; PROMIS PI CAT, Patient-Reported Outcomes Measurement Information System Computer Adaptive Test forms for Pain Interference; ROC, receiver operating characteristic.

Overall, the percent of patients who met MCID for PROMIS PF CAT, PROMIS PI CAT, and PROMIS D CAT was 85%, 72%, and 55%, respectively. Post hoc analysis was performed in order to determine prognostic cutoff thresholds for achieving or failing to achieve MCID using 95% specificity. Application of prognostic cutoffs to presurgical PROMIS CAT scores improved the accuracy of predicting patients that would achieve or fail to achieve MCID, with results summarized in Table 3. For example, in PROMIS PI CAT, a pre-operative score >65.7 resulted in 92% of patients achieving MCID, whereas a score of <51.7 resulted in 67% of patients failing to achieve MCID. In addition, for PROMIS PF CAT, a preoperative score of <35.6 resulted in 100% of patients achieving MCID, while a score of >50.7 resulted in 40% of patients failing to achieve MCID.

Table 3

Prognostic Cutoffs for PROMIS CAT Domains

PROMIS CAT Domain	Achievement of AB MCID		Failure to Achieve AB MCID
	Pre-Cutoff	Post-Cutoff	Pre-Cutoff	Post-Cutoff
PROMIS PF CAT	85%	100% (<35.6)	15%	40% (>50.7)
PROMIS PI CAT	72%	92% (>65.7)	28%	67% (<51.7)
PROMIS D CAT	55%	83% (>57.5)	45%	83% (<40.5)

AB MCID, minimum clinically important difference calculated according to anchor-based methodology; PROMIS D CAT, Patient-Reported Outcomes Measurement Information System Computer Adaptive Test forms for Depression; PROMIS PF CAT, Patient-Reported Outcomes Measurement Information System Computer Adaptive Test form for Physical Function; PROMIS PI CAT, Patient-Reported Outcomes Measurement Information System Computer Adaptive Test forms for Pain Interference.

Discussion

ACLR resulted in significant improvements in PROMIS PF CAT, PROMIS PI CAT, and PROMIS D CAT scores. The majority of patients exceeded AB MCID in PROMIS PF CAT, PROMIS PI CAT, and PROMIS D CAT following surgery. Using prognostic cutoffs, preoperative scores were able to accurately predict the postoperative response to surgery.

PROMs are essential tools for evaluating patient response to intervention and providing objective measurements of the patient experience. PROMIS has been validated as a reliable patient reported outcome metric compared to other PROMs used in knee surgery, and its increasing use is largely attributed to not only this favorable correlation but also decreased survey burden for patients.26, 27, 28, 29, 30 Furthermore, PROMIS PF CAT, PROMIS PI CAT, and PROMIS D CAT have all been validated in assessing outcomes in ACLR patient populations in particular.31, 32, 33 The present investigation was able to define the magnitude of MCID for three relevant PROMIS CAT domains in patients undergoing ACLR by using an AB method. In addition, average PROMIS CAT scores had statistically significant improvement following ACLR with improvements exceeding AB MCID. Similar findings were demonstrated in a previous study by Chen et al., which assessed PROMIS CAT MCID in ACLR patients according to the DB methodology. The authors also found that the average change in PROMIS CAT scores exceeded the MCID following their procedures. While the investigation by Chen et al. provides essential information regarding patient response to ACLR, a weakness of the DB method is the focus on statistical significance and consequential failure to incorporate the patient’s perspective of clinically important change., The AB method addresses the shortcomings of the DB method by linking the pre- and postoperative difference in outcome measure score to a meaningful external anchor question, which represents the patient’s perspective., Although DB MCID is a valuable method for predicting clinically meaningful improvement, AB MCID is shown to better represent patient-centric clinical changes.,

While many patients experience positive surgical outcomes following ACLR, there remains a subset of patients who do not return to preinjury levels of physical function. Previous investigations by Ardern et al. and Rodríguez-Roize et al. indicate that only 52% and 65% of athletes undergoing ACLR return to their preinjury performance levels, respectively. Thus, it is important to identify patients who are at risk for poor outcomes to optimize treatment. One such strategy for optimizing outcomes is use of preoperative PROMs to gauge patients’ degree of physical and mental impairment. The present investigation revealed that applying preoperative score cutoffs in all three PROMIS CAT domains enhanced the ability to predict patients who would achieve or fail to achieve MCID following surgery. This has also been evaluated in orthopaedic foot and ankle patients, as Ho et al. used similar methodology to provide clinical cutoffs that aid in preoperative decision making in a variety of foot and ankle conditions. The aforementioned study by Chen et al. also found that preoperative clinical cutoffs were beneficial in predicting post-operative outcomes following ACLR using the DB MCID. As in our study, both of these investigations also noted that worse baseline physical function and pain scores portended to a higher likelihood of achieving MCID following surgery. Still, it is important to recognize how these patients with lower baseline function did not necessarily achieve similar post-operative levels of function to their counterparts with higher initial function, even though they were more likely to achieve MCID.

The practical application of PROMs to clinical decision making remains a challenge. PROMIS CAT has been demonstrated to have strong responsiveness and minimal floor and ceiling effects in patients with knee pathology., Thus, the use of PROMIS CAT forms as well as the findings of the present investigation can be beneficial in the shared decision-making process with patients who have suffered from ACL tears. For example, a moderately active 50-year-old female with an ACL tear who has pre-operative PROMIS CAT scores consistent with high physical function (PROMIS PF CAT >50.7) and low pain interference (PROMIS PI CAT <51.7) based on the calculated prognostic cutoffs may not receive a significant clinical benefit following ACLR and resultantly can be educated on alternative treatment options to surgical intervention. Conversely, a patient with preoperative scores that fall below the cutoff for PROMIS PF CAT (<35.6) and above in PROMIS PI CAT (>65.7) may have a very high preoperative likelihood of exceeding MCID after surgery. While PROMs should not be the primary driver in surgical decision making, they offer an important added measure when making shared decisions with our patients while considering other important variables such as age and activity level. Moreover, pre-operative PROM assessment in this capacity allows surgeons an opportunity to evaluate surgical appropriateness based on the individual patient.

Limitations

There were limitations to this investigation. All enrolled subjects were English-speaking patients at a metropolitan academic center, thereby potentially limiting the external validity of findings given the lack of representation across different healthcare modalities. However, the general diversity of the enrolled patient population in terms of age, sex, and ethnicity mitigates this limitation to a significant degree. In addition, only patients who completed all PROMIS CAT and anchor questionnaire forms were included in final analysis, thus limiting the total population based on follow-up and willingness to complete forms at each visit. Incomplete patient participation potentially leads to an enrolled cohort that may not be fully representative of the general clinic population. Furthermore, there were differences in ACLR surgical technique and post-operative rehabilitation within the enrolled, both of which are variable factors that have been shown to potentially influence postoperative outcomes that our study did not account for. Regardless, previous literature supports how significant variation exists in post-operative rehabilitation protocols, supporting how our findings are reflective of real-life patient populations. Lastly, there are some inherent limitations to the clinical interpretation of MCID as calculated in this study. For certain orthopaedic procedures such as hip arthroscopy, there are low levels of patient satisfaction between 25 and 30% as reflected by the patient acceptable symptom state (PASS), despite at least 90% of patients achieving MCID in various outcome measures and thus achieving improvement in health status. Such findings indicate how achieving MCID may not represent true satisfaction with clinical improvement, which may also be a discrepancy that applies to our study given that we do not assess PASS to show otherwise. Regardless, we still feel that our study is valuable given how MCID is a reliable indicator of clinical improvement in its own right, and how future follow-up studies can focus on evaluating PASS.

Conclusions

According to AB analysis of PROMIS CAT MCID, ACL reconstruction is effective in improving physical function, pain interference, and depression symptoms. In addition, preoperative PROMIS CAT scores can predict the likelihood of achieving MCID postoperatively.