Comparing the responsiveness of a generic and a musculoskeletal specific functional outcome measure in orthopaedic patients with operative fixation of pelvic ring, acetabulum, or tibia fractures: a comparison between single injury and multiply injured patients.

OBJECTIVES: This study compares the responsiveness, or the ability to detect clinical change in a disease, between the generic Short Form-36 (SF-36) and musculoskeletal specific Short Musculoskeletal Functional Assessment (SMFA) patient-reported outcome measures (PROMs) in the orthopaedic trauma population. Stratified analysis was performed to compare whether responsiveness differs between patients with single or multiple orthopaedic injuries.
DESIGN: Prospective case series.
SETTING: Level 1 Trauma Center. PATIENTS: A total of 659 patients with orthopaedic trauma injuries to the pelvis, acetabulum, or tibia were included for analysis. There were 485 patients with a single isolated injury and 174 patients with multiple orthopaedic injuries. INTERVENTION: None. MAIN OUTCOME MEASUREMENTS: Responsiveness was calculated through the standard response mean (SRM), the proportion meeting a minimal clinically important difference, and floor and ceiling effects.
RESULTS: Between baseline and 6 months the magnitude of the SRM for SF-36 was consistently greater than that of SMFA in patients with single (P < .01) and multiple injuries (P < .01). Between 6 and 12 months, there were no differences in SRM across all cohorts. The proportion of patients who achieved minimal clinically important difference was consistently higher when assessed with SF-36 compared with SMFA between baseline and 6 months (81.8% vs 68.1%, P < .0001) and between 6 and 12 months (63.3% vs 55.4%, P = .01).A ceiling effect was only observed at baseline for the SMFA with 16.6% of patients achieving the maximal level of functioning detectable. No floor effects were seen in either PROM.
CONCLUSION: This study demonstrates that SF-36 has superior responsiveness versus SMFA in both polytrauma and isolated injury patients and supports the collection of SF-36 as the primary PROM in prospective orthopaedic trauma studies irrespective of whether the patient has an isolated injury or multiple injuries.

Introduction

Patient-reported outcome measures (PROMs) have rapidly become the gold standard for assessing the outcomes of care when seen from a clinical, research, or healthcare funding perspective.[ The Short Form 36 (SF-36) is a self-administered general health survey that has been validated across numerous patient populations.[ The SF-36 Physical Component Score (PCS) has frequently been utilized in the orthopaedic literature.[ While the use of a generic outcome measure, such as the SF-36, allows for comparison of outcomes across differing disease states, disease or anatomic specific outcome measures are believed to be capable of capturing smaller clinical changes in patients they are designed for.[ The Short Musculoskeletal Function Assessment (SMFA) is a musculoskeletal specific PROM that has been demonstrated to be valid and reliable in the orthopaedic population.[

Typically, patient-reported outcome questionnaires are long and time consuming to complete. Nonetheless, researchers often use multiple PROMs in an effort to fully assess the outcomes of patient treatments. This imposes a significant burden on patients and clinicians at each assessment time point and is likely to impact both recruitment to studies and follow-up rates.

While numerous studies have been conducted on validity and reliability of PROMs in the orthopaedic trauma population, far fewer studies have formally assessed responsiveness, or the ability to detect small but clinically significant changes.[ This is likely because formally assessing responsiveness requires conducting multiple outcome measurements at numerous timepoints. This study aims to compare the responsiveness of 2 validated and reliable PROMs, the generic SF-36 and the musculoskeletal specific SMFA, in orthopaedic trauma patients with pelvic ring, acetabulum, or tibia fractures. The secondary aim is to determine whether the responsiveness of these outcome measures changes depending on whether patients have single or multiple orthopaedic injuries.

Methods

This prospective observational study enrolled patients between 2008 and 2015 at a Level 1 Trauma Center. Patients over the age of 18 with operatively treated pelvic ring, acetabular, tibial plateau, tibial shaft, and tibial plafond fractures were eligible. Exclusion criteria included subjects who were unable to complete the questionnaire due to language barrier, injury (head trauma, prolonged intubation), or would not complete follow-up with the institution. The study was approved by the Institutional Ethics Committee of the University of British Columbia. Informed consent was obtained from all patients before participation.

Outcome measures

The SF-36 is a generic outcome measure that measures 8 health domains: physical function, limitations in activities because of physical health problems, limitation in activities because of emotional problems, bodily pain, general health perceptions, vitality, social function, and mental health.[ A scaled score from 0 to 100 is calculated for the various domains, with lower scores indicating a poorer function. The scaled scores are standardized to z scores based on the US population, with a mean of 50 and standard deviation of 10.[ These are then typically summated into a PCS and mental component summary scores.

The SMFA was developed from the longer Musculoskeletal Functional Assessment as a way of assessing functional outcome across patients with different musculoskeletal conditions.[ This PROM is comprised of 46 questions divided into 4 categories (daily activities, emotional status, arm and hand function, and mobility). Alternatively, the SMFA can be summarized by an index score: the dysfunction index, which has 34 items for the assessment of patient function, and the bother index, which has 12 items for the assessment of how much patients are bothered by functional problems. Both SMFA indices are standardized to a score from 0 to 100 based on the AAOS mean score and standard deviations. In a normal population, the mean SMFA is 50 with higher scores indicating poorer function.[

This study specifically used the PCS of the SF36 and the dysfunction index of the SMFA as these measures are conceptually similar and ideal for comparison. Furthermore, these measures are commonly utilized in the orthopaedic literature. In addition, to reduce responder burden and survey fatigue, only questions used in the scoring of the selected component or indices of the PROM were utilized.[

Data collection

The PROMs were administered by trained research personnel using paper surveys. Surveys were administered in the English language. PROMs were collected prior to discharge from initial hospital admission, and during their clinic follow-up visits at 6 and 12 months. Patients rated their preinjury (baseline) status during their initial hospitalization. Previous literature suggests patients can accurately recall their preoperative quality of life, function, and general health up to 6 weeks postsurgery.[

Data analysis

Only patients who completed both SF-36 and SMFA questionnaires at each time point were included in the study. Responsiveness was assessed for 3 patient cohorts: isolated single injury, polytrauma, and all patients. Patients were divided into those with isolated or multiple injuries as defined by their Injury Severity Score, with those with Injury Severity Score >9 being categorized as polytrauma. The distribution of outcome scores was plotted at each time point. The correlation between SF-36 and SMFA scores was calculated between time points using Pearson correlation coefficients (r).

Responsiveness

Responsiveness was assessed by calculating the standardized response mean (SRM), the proportion of patients exceeding the minimal clinically important difference (MCID), and observed floor and ceiling effects.[ Each of these assessments was carried out for patients with isolated injuries, multiple injuries, and then for the combined group.

The SRM is a measure of the effect size of the changes in score and is measured using the mean score change divided by the standard deviation of the score change between each time period.[ A larger SRM indicates greater responsiveness. The SRM was calculated between baseline and 6 months, as well as between 6 and 12 months for the SF-36 and SMFA. Differences in SRM between outcome measures were assessed using the paired t test. The MCID is defined as the smallest difference in a score that patients perceive as being of benefit, and therefore might consider a change in their treatment. There is no established MCID for the SF36 or SMFA specific to the orthopaedic trauma population. A previous review has demonstrated that in most circumstances, a change of one half of a standard deviation in the score in the affected population constitutes an MCID.[ We calculated the MCID from our data at the 6 month time point when patients were maximally affected by the disease. The proportion of patients achieving MCID for SF-36 and SMFA between assessments was compared using the McNemar test. Outcome scores at each time point were also evaluated for a floor (scores reflecting the lowest level of functioning) and ceiling (scores reflecting maximal level of functioning) effects. An instrument was considered to have floor or ceiling effects if more than 10% of the scores were at the lowest or highest level of functioning.

All statistical analyses were carried out using the R statistical computing environment (R Foundation for Statistical Computing, Vienna, Austria), with P values < .05 considered statistically significant.

Results

A total of 660 patients were analyzed with complete data at all time points. This included 430 males and 230 females. One hundred seventy-five patients had multiple injuries and 485 had isolated injuries. Summary details of the included patients are in Table 1.

Table 1

Study patient summary

	Patients with complete data (n = 660)
Injury
Pelvic ring fractures	305 (46.2%)
Tibial plateau fractures	116 (17.6%)
Tibial shaft fractures	185 (28.0%)
Tibial plafond fractures	54 (8.2%)
Polytrauma	175 (26.5%)
Gender
Female	230 (34.8%)
Male	430 (65.2%)
Age
Mean (SD)	43.7 (16.3)
Injury severity score (ISS)
Mean (SD)	12.1 (7.2)
ISS >9 (%)	175 (26.5%)
ISS >18 (%)	86 (13.0%)

The distribution of scores for SF-36 and SMFA at each time point are displayed in Figure 1. The correlation between scores was assessed at each time point to ensure that the outcome scores were measuring similar attributes. The correlation was lowest at baseline (r = −0.55) and highest at 12 months (r = −0.84). There was a statistically significant negative correlation at all time points which was consistent when assessing isolated injury, multiple injury patients, and the whole cohort (P < .0001).

Figure 1

Distribution of SF-36 (PCS) and SMFA (DI) scores for all patients. DI = Disability Index, PCS = physical component score, SF-36 = short form-36, SMFA = short musculoskeletal functional assessment.

Between baseline and 6 months, the SRM of SF-36 was consistently greater than the SRM of SMFA for both polytrauma (P < .001) and isolated injury patient cohorts (P < .001). Between 6 and 12 months, there were no significant differences in SRM between the 2 PROMs (Table 2).

Table 2

The magnitude of the standardized response mean for isolated trauma, polytrauma, and combined cohorts

	Isolated trauma			Polytrauma			Combined
	SF-36 PCS	SMFA DI	P value	SF-36 PCS	SMFA DI	P value	SF-36 PCS	SMFA DI	P value
Baseline to 6 months	1.42	1.03	<.001	1.71	1.33	<.001	1.47	1.08	<.001
Six to 12 months	0.58	0.50	.07	0.60	0358	.81	0.59	0.52	.10

DI = Disability Index, PCS = physical component score; SF-36 = short form-36; SMFA = short musculoskeletal functional assessment.

The MCID was calculated for each PROM using one-half of the standard deviation of outcome scores at 6 months. Applying this to our data, 81.8% of patients achieved an MCID between baseline and 6 months using the SF-36 compared with 68.1% of patients assessed with the SMFA (P < .001). This was again demonstrated between 6 and 12 months where 63.3% of patients improved by the MCID when measured using the SF-36 compared with 55.4% when measured using the SMFA (P = .01). Of note, this trend was observed in all cohorts although it did not reach statistical significance in isolated trauma between 6 and 12 month time points (Table 3).

Table 3

Proportion of patients achieving minimal clinically important difference (MCID) between timepoints

	SF-36 PCS	SMFA DI
	Number achieving MCID (%)	Number achieving MCID (%)	P value
Baseline to 6 months
Isolated trauma	395 (81.4)	323 (66.7)	<.001
Polytrauma	144 (82.8)	134 (77.0)	.02
Combined	539 (81.8)	449 (68.1)	<.001
Six to 12 months
Isolated trauma	304 (62.7)	278 (57.3)	.07
Polytrauma	111 (63.8)	97 (55.8)	.01
Combined	417 (63.3)	365 (55.4)	.01

PCS = physical component score; SF-36 = short form-36; SMFA = short musculoskeletal functional assessment.

When assessing for a floor effect, or the proportion of patients with the lowest level of functioning that can be assessed by each outcome measure, a threshold of 10% was determined to be significant. There was not a significant floor effect for either outcome measure at any timepoint. However, a significant ceiling effect was observed at baseline for the SMFA scores with 16.6% of patients obtaining the highest level of function measurable by the SMFA (15.9% of patients with single isolated injuries and 26.5% of polytrauma patients). Of note, for the SMFA, a lower score indicates a greater level of function, which is inverse to the SF-36 where a higher score indicates greater function (Table 4).

Table 4

Ceiling effects: number of patients (%) achieving the greatest measurable function for each PROM

	Isolated injuries
	Baseline	6 months	12 months
SF-36 PCS
Isolated injuries	1 (0.1%)	1 (0.2%)	1 (0.2%)
Polytrauma	1 (0.4%)	1 (0.5%)	1 (0.5%)
Combined	1 (0.1%)	1 (0.1%)	1 (0.1%)
SMFA DI∗
Isolated injuries	105 (15.9%)	6 (0.9%)	18 (2.7%)
Polytrauma	68 (26.5%)	2 (0.9%)	3 (1.5%)
Combined	162 (16.6%)	10 (0.2%)	20 (2.7%)

PCS = physical component score; SF-36 = short form-36; SMFA = short musculoskeletal functional assessment.

For the SMFA DI, a lower score corresponds to greater functional outcomes.

Discussion

Over the recent past there has been a gradual shift in the emphasis away from clinician-reported outcome measures, such as radiographic outcomes, to patient-reported outcomes. This is acknowledged as being an important step in clinical and translational research as patient-reported outcomes capture what is valued by the patient without interpretation or bias from the clinician or other stakeholder.[ Increasingly, PROMs are required by funders and publishers of research to be the primary outcome measures in studies. It is essential that the selected PROM have undergone appropriate psychometric testing specific to the study population. Although it is believed that musculoskeletal-specific scores are more responsive to capturing important clinical changes in the orthopaedic population, this study demonstrates limitations with regards to responsiveness in the SMFA, particularly when evaluating preinjury trauma patient populations. There was a significant ceiling effect at baseline, with 16.6% of patients obtaining the highest score or function measurable by the SMFA. This is demonstrated in Figure 1 with clustering of patients at the lower limits of SMFA scoring. This likely played a role in the superior responsiveness in the SF36 specifically between baseline and 6 months as measured by the SRM and proportion meeting MCID. Furthermore, stratified analysis in our study revealed that the SF-36 maintained superior responsiveness for both populations with isolated injury and polytrauma.

Our findings are consistent with prior studies evaluating the SF-36 and SMFA specifically in the orthopaedic trauma population. For fractures surrounding the tibia, there were no observable advantages in the responsiveness of SMFA compared with SF-36.[ A significant ceiling effect was observed specifically for musculoskeletal and injury-specific outcome measures not seen in generic outcome scores.[ The SF-36 was designed to remain relevant and valid across numerous disease states which may explain its ability to assess healthier patient populations. In the orthopaedic trauma population, the “preinjury” patient population is frequently healthy without functional limitation. In fact, in this study, the mean baseline SF36 score was 55.5, which is greater than the population mean which is standardized to 50 (Fig. 1). In studies assessing nontrauma orthopaedic diseases where there is no “healthy” population, the benefits of SF-36 over SMFA with regards to responsiveness have not been reproduced. Kirschner et al[ demonstrated that the SMFA had larger effect sizes than the SF-36 in patients undergoing total knee arthroplasty for primary osteoarthritis. Studies have also noted that the SF-36 performs a limited assessment specific to the upper extremity function which negatively impacts responsiveness in these patient populations.[ This is important to note given our study excluded upper extremity trauma.

It is important to minimize the number of PROMs collected in any research study as each additional measure that is collected adds an additional burden in terms of time and acceptability to both patients and clinicians. This is particularly important when assessing the orthopaedic trauma population who have demonstrated significant loss to follow-up in prior literature, ranging from 8.5% to 40%.[ The newly developed Patient-Reported Outcomes Measurement Instrumentation System (PROMIS) is a series of standardized outcome measures administered using computer adaptive technology aimed at reducing patient burden while maintaining validity and responsiveness.[ This universal assessment score is intended to be used across disease groups. Furthermore, specific item banks can be selected for assessment of specific functional measures, such as the “PROMIS physical function” versus the “PROMIS mobility” item banks. The surveys are tailored to the patients’ prior answers, limiting irrelevant survey questions and thus minimizing the time required for patient completion.[ Early investigation into the utilization of PROMIS on the orthopaedic trauma population has demonstrated excellent validity with minimal ceiling effect.[

We acknowledge that our study does have some limitations. First, this study only enrolled patients with operatively treated pelvic ring and lower limb injuries. Our study results may not be generalizable to patient populations outside of this sample. It is possible that the granularity of SF-36 alone may not be sufficient for assessing patients treated nonoperatively where there may be smaller changes in health status. Furthermore, as noted earlier, there have been concerns regarding the SF-36 effectively evaluating patient outcomes with regards to the upper limb. In addition, as the prospective collection of preinjury functional outcome scoring is not possible in a trauma population, our baseline assessment was based on patient recall postoperatively and prior to discharge. This potentially could introduce recall bias. However, previous studies have suggested that patients can accurately recall their preoperative state up to 6 weeks postoperatively, albeit in the elective surgery setting.[

In conclusion, this study demonstrates limitations in the responsiveness of the SMFA in orthopaedic trauma populations, particularly regarding baseline or preinjury assessment. Because of these findings, we advocate that the SF-36 is a responsive and previously validated tool in measuring patient outcomes in prospective orthopaedic trauma studies evaluating pelvic ring and lower limb injuries, and that the simultaneous collection of the SMFA offers little added benefit. This is irrespective of whether the patient has sustained an isolated injury or multiple injuries. It will be prudent for future studies to undertake the same rigorous psychometric analysis of newly developed outcome scores, such as the PROMIS, with regards to the disease or population of interest.