Sociodemographic Factors Are Associated with Patient-Reported Outcome Measure Completion in Orthopaedic Surgery: An Analysis of Completion Rates and Determinants Among New Patients.

Patient-reported outcome measures (PROMs) and, specifically, the Patient-Reported Outcomes Measurement Information System (PROMIS), are increasingly utilized for clinical research, clinical care, and health-care policy. However, completion of these outcome measures can be inconsistent and challenging. We hypothesized that sociodemographic variables are associated with the completion of PROM questionnaires. The purposes of the present study were to calculate the completion rate of assigned PROM forms and to identify sociodemographic and other variables associated with completion to help guide improved collection efforts.
Methods: All new orthopaedic patients at a single academic medical center were identified from 2016 to 2020. On the basis of subspecialty and presenting condition, patients were assigned certain PROMIS forms and legacy PROMs. Demographic and clinical information was abstracted from the electronic medical record. Bivariate analyses were performed to compare characteristics among those who completed assigned PROMs and those who did not. A multivariable logistic regression model was created to determine which variables were associated with successful completion of assigned PROMs.
Results: Of the 219,891 new patients, 88,052 (40%) completed all assigned PROMs. Patients who did not activate their internet-based patient portal had a 62% increased likelihood of not completing assigned PROMs (odds ratio [OR], 1.62; 95% confidence interval [CI], 1.58 to 1.66; p < 0.001). Non-English-speaking patients had a 90% (OR, 1.90; 95% CI, 1.82 to 2.00; p < 0.001) increased likelihood of not completing assigned PROMs at presentation. Older patients (≥65 years of age) and patients of Black race had a 23% (OR, 1.23; 95% CI, 1.19 to 1.27; p < 0.001) and 24% (OR, 1.24; 95% CI, 1.19 to 1.30; p < 0.001) increased likelihood of not completing assigned PROMs, respectively. Conclusions: The rate of completion of PROMs varies according to sociodemographic variables. This variability could bias clinical outcomes research in orthopaedic surgery. The present study highlights the need to uniformly increase completion rates so that outcomes research incorporates truly representative cohorts of patients treated. Furthermore, the use of these PROMs to guide health-care policy decisions necessitates a representative patient distribution to avoid bias in the health-care system. Level of Evidence: Prognostic Level III. See Instructions for Authors for a complete description of levels of evidence.

Patient-reported outcome measures (PROMs) provide a means to relay current symptoms and health state in a concise, validated manner[1]. One set of PROMs is the Patient-Reported Outcomes Measurement Information System (PROMIS), which was developed and validated with support from the National Institutes of Health (NIH) to provide a generalizable PROM across medical conditions and specialties[2]. Since its introduction, the PROMIS has demonstrated responsiveness across varying pathologies and interventions[3,4]. Within orthopaedic surgery, it performs as well as, or better than, historical instruments designed for subspecialty niches[5].

The advent of the electronic medical record (EMR) facilitated the integration of PROMs into patient care[6]. The generalizability of the PROMIS positions it as an ideal instrument for health-care systems to uniformly apply to an EMR framework. Beyond clinical care, payers are financially incentivizing health-care systems to collect PROM scores[7]. With a trove of outcomes data, one can expect payers to begin quantifying the quality variable in the health-care value equation by using PROMs as a key metric[8]. Beyond mere collection, clinical research incorporating PROMs has focused on determining minimum clinically important difference (MCID) thresholds—or the change in PROM scores reflecting a true clinical change appreciated by the patient—for orthopaedic surgery interventions[9,10]. The results of those studies can serve as preliminary benchmarks for clinical care and health-care policy[11]. However, the denominator of patients eligible for PROMIS score measurement is often unknown or not reported, which may be problematic and may bias findings.

It is incumbent on orthopaedic surgeons to conduct the research necessary to guide appropriate health policy and care innovations that will affect orthopaedic surgery. This research includes understanding the proportion of patients completing assigned questionnaires and the variables associated with successful completion to ensure appropriate outcome measurement. We hypothesized that sociodemographic variables are associated with completion of PROM questionnaires. Thus, we had 2 aims: (1) to determine the proportion of patients successfully completing all assigned PROMs at new patient visits and (2) to assess patient factors associated with successful completion of assigned PROMs at new patient visits.

Materials and Methods

Patient Sample

This retrospective, observational study was approved by the institutional review board. All new patients who were scheduled for an orthopaedic clinic encounter at a single academic medical center from June 15, 2016, to December 31, 2020, were identified. Patients were defined as new if they had not seen the surgeon before or if it had been >3 years since the last visit. Patients who cancelled, did not present, or left before the clinic encounter was completed were excluded. Our data are not granular enough to assess the number of patients in each exclusion criteria subset. Overall, 221,635 (65%) of 340,288 new patient encounters were identified on the basis of these criteria.

Patient-Reported Outcome Measures

The primary outcome was defined as completion of a PROM set (i.e., answering all questions of each required PROM to produce an accurate score) associated with a new patient clinical encounter. As part of routine clinical care at our institution, patients across all orthopaedic subspecialties are requested to complete a predetermined set of PROMs (including both PROMIS and legacy assessments) based on the subspecialty for which they are presenting for care. All patients were asked to fill out certain PROMIS domains, which included Physical Function-10a (PF-10a) or PROMIS Upper Extremity Version 1.2 (UE-v1.2) and/or Global-10 Mental Health subscore (G-10 MH) and Global-10 Physical Health subscore (G-10 PH). Additional general PROMs evolved over time by orthopaedic subspecialty and included Pain Interference-4a (PI-4a), Anxiety-4a (A-4a), Pain Intensity-3a (I-3a), and/or Depression-4a (D-4a). In addition, patients are asked to complete forms specific to the presenting pathology/anatomic region of interest. Patients presenting for upper-extremity evaluation are also asked to complete the QuickDASH, an abbreviated version of the Disabilities of the Arm, Shoulder and Hand (DASH) questionnaire. Patients presenting for hip or knee evaluation are asked to complete the Hip disability and Osteoarthritis Outcome Score-Physical Function Short form (HOOS-PS) or the Knee injury and Osteoarthritis Outcome Score-Physical Function Short form (KOOS-PS), respectively. Patients presenting for foot or ankle evaluation are asked to complete the Foot and Ankle Ability Measure activities of daily living (FAAM ADL). Spine patients are asked to complete neck, low back, and leg pain scores that are measured on a Likert scale from 0 to 10.

Patients are asked to complete their assigned PROMs, prior to their clinical visit, via the EMR patient portal. If not completed in this manner, patients are asked to complete them in person at the clinical encounter via a tablet computer. Patients could refuse to complete the PROMs, or the clinical visit could begin prior to completion of the PROMs. Successful completion or lack of completion of all assigned PROMs is automatically recorded in our institution’s database.

The following characteristics were included in this study: activation of the EMR patient portal (i.e., Patient Gateway) prior to the clinical encounter (yes or no), year of visit (2016, 2017, 2018, 2019, or 2020), payer (commercial, Medicaid, Medicare, Workers’ Compensation, or other), sex (man or woman), self-reported race (White, Black, Asian, or other), marital status (married, single, divorced, widowed, or other), age (<40 years, 40 to 64 years, ≥65 years), primary language (English or non-English), and orthopaedic subspecialty (hand, foot and ankle, trauma, arthroplasty, oncology, spine, or other). Unknown or unreported sex (n = 1 [0.0005% of 221,635]) and those without a known primary language (n = 1,743 [0.79% of 221,635]) were excluded from all analyses. This left a final sample of 219,891 new patient encounters for use in the final analyses.

Descriptive statistics were calculated across all variables. Bivariate analyses compared characteristics among patients who completed all assigned PROMs with those of patients who did not. Chi-square tests were utilized for categorical variables. Mann-Whitney U tests were performed for continuous variables, which were not normally distributed. A multivariable logistic regression model was created to determine which variables were associated with successful completion of all assigned PROMs. For all analyses, the level of significance was set at p < 0.05.

Source of Funding

No funding was obtained for the work represented in this manuscript.

Results

Of the 219,891 new patients, 88,052 (40%) completed all assigned PROMs (Table I). Of those who completed all assigned PROMs, 75% (n = 66,141) completed them in the clinic. Nearly three-quarters of all new patients had registered on the Patient Gateway (n = 155,685 [71%]). A majority of patients were primarily English-speaking (n = 205,209 [93%]), and approximately 4 of every 5 patients self-identified as White (n = 178,001 [81%]). Just over one-half of included patients were married (n = 112,815 [51%]), and the most common insurance type was commercial insurance (n = 121,918 [55%]).

There was a significant difference across all clinical and patient characteristics between those who completed all assigned PROMs and those who did not (Table I).

When we accounted for possible confounders, patients who had not activated the Patient Gateway had a 62% increased likelihood of not completing assigned PROMs (odds ratio [OR], 1.62; 95% confidence interval [CI], 1.58 to 1.66; p < 0.001) (Table II). Sociodemographically, patients of Black race, patients who were ≥65 years of age, and patients who were non-English-speaking had a 24% (OR, 1.24; 95% CI, 1.19 to 1.30; p < 0.001), 23% (OR, 1.23; 95% CI, 1.19 to 1.27; p < 0.001), and 90% (OR, 1.90; 95% CI, 1.82 to 2.00; p < 0.001) increased likelihood of not completing assigned PROMs. Patients with insurance classified as other (self-pay or uninsured) had a 17% (OR, 1.17; 95% CI, 1.10 to 1.25; p < 0.001) increased likelihood of not completing assigned PROMs. Patients who were evaluated for a spine-related condition had a 96% decreased likelihood of not completing assigned PROMs (OR, 0.04; 95% CI, 0.04 to 0.04; p < 0.001). A complete listing of all characteristics, including sociodemographic variables and respective ORs, is found in Table II.

Discussion

In this retrospective analysis of 219,891 new orthopaedic patients, we found that 88,052 (40%) completed the assigned PROMs. Further analysis demonstrated that lack of online EMR portal activation (i.e., Patient Gateway), Black race, Medicare coverage, non-married status, and non-English-speaking status were associated with non-completion. Additionally, there was significant variation of completion rates between orthopaedic subspecialties. Such findings suggest that PROM scores do not reflect the orthopaedic patient population seeking care, which could negatively impact certain patient subgroups if policy is enacted on the basis of only the data that are presently available.

Importantly, the findings of the present study are similar to those reported by Schamber et al., who identified variables that affect PROM completion rates in patients managed with arthroplasty[12]. However, the present study is different in that it includes the spectrum of pathological conditions treated with orthopaedic surgery. Additionally, our focus on PROMIS scores is a notable strength as the call for the PROMIS to be a routine PROM in orthopaedic surgery is growing and is being incorporated into the American Board of Orthopaedic Surgery (ABOS) Part II Oral Examination[13].

To our knowledge, the present study represents the largest analysis of PROM scores in the orthopaedic literature. At our institution, orthopaedic surgery represents the largest individual contributor to PROM completion[14]. The time period of our analysis included the implementation of different institutional efforts to increase completion rates, but it did not include financial incentives to the department or individual surgeons. The trend of improved collection rates year-over-year (Table II) is attributable to the implementation of a variety of non-punitive efforts, including hiring a dedicated staff for PROM collection and increasing targeted advertising.

It is difficult to infer a cause for the association of individual variables with non-completion of PROMs. The activation of an online portal can be associated with technology fluency but also assumes the financial means for a computing platform and internet connectivity at home. However, it is also possible that activation of an online EMR portal may be a proxy for patient activation or a patient’s ability and willingness to engage with and manage their own health[15]. Self-reported race, insurance type, marital status, and primary language are all surrogates for socioeconomic status and are known social determinants of health and health literacy. Prior research in the orthopaedic surgery literature suggests a correlation between baseline PROM scores and sociodemographic health indices and factors[16-18]. Our work builds on that research by showing that both (1) PROM scores and (2) the completion rate of PROMs differ on the basis of sociodemographic factors. While prior research also has demonstrated the relationship between certain sociodemographic factors and PROM completion, our findings reinforce the key factors associated with PROM completion with use of a larger and more diverse sample, which improves generalizability[12,19]. However, given the pseudo-R2 of 0.17, other factors that may impact PROM completion have yet to be identified.

A higher completion rate of PROMs is needed to better understand the population served by orthopaedic surgeons. This need is underscored by an observed downward trend in completion rates of PROMs for registry participants[20]. The completion rate of PROMs is high in the clinical trial setting, but that setting represents a sliver of the overall care provided[21]. Retrospective research with use of PROM scores potentially excludes a majority of eligible patients. Future investigators using PROM scores should be cognizant of this pervasive potential limitation. Furthermore, this limitation may open the possibility for statistical techniques to impute missing data. These findings highlight the need to increase completion rates so that research incorporates truly representative cohorts of the patients treated. Although payers have yet to risk-adjust based on PROMs at the individual surgeon level, there are payment models that vary according to sociodemographic factors[22]. The accumulation of representative data allows a more accurate understanding of baseline PROM scores, calculation of MCID thresholds, and quality measurement, which is critical as clinical practice guidelines and payment models adapt to incorporate more PROMs, including the PROMIS.

Sisodia et al. suggested that the most effective method for improving completion rates is through surgeon engagement and administrative support[14]. Importantly, in that study, mandated collection was not associated with improved completion rates. Other investigations on strategies to improve completion rates have recognized the importance of increased staff resources to provide reminders, help enroll patients in online portals, and monitor completion[23,24]. It also has been documented that a physician champion is needed to truly drive improved collection of PROMs[25]. Additionally, PROMIS computerized adaptive test (CAT) forms will help to decrease the time required for completion and may improve completion rates[26]. However, as is the case for the short forms, it is vital for the CAT forms to be available in a wide array of languages, as evidenced by the importance of primary language in the PROM completion rate noted in the present study. Last, if the PROMIS continues to show psychometric equivalence to legacy measures, it may continue to supplant certain PROMs across most orthopaedic subspecialties and further decrease the time burden for completion across patients.

The present study had limitations that should be considered. First, the data were gathered at a single, urban academic orthopaedic surgery department and therefore the results may not be generalizable to other geographic regions or health-care settings. However, our patient sample was large and diverse across a number of characteristics; thus, it is likely to be as generalizable as one can expect in such a single-institution study. Second, hospital-wide efforts were made to increase completion rates during the study time period, including notifications to health-care professionals and patients encouraging PROM completion. These efforts were not directly targeted at certain patient subgroups or populations; still, certain surgeons may place greater importance on, and encourage, the completion of PROMs. Furthermore, the variables that we considered for completion rates were those that were available in the medical record, and there may be unmeasured confounding. Third, the availability of certain foreign-language versions of the PROMs that were utilized increased over the study period, which may lead to overestimation of the association between this variable and decreased PROM completion. It is also possible that family members and/or translators assisted in PROM completion, although how often this occurred is challenging to know. Fourth, at present, our institution utilizes PROMIS Short forms, not PROMIS CATs, because of a lack of widespread foreign-language validation of the CAT forms. The use of multiple short forms and static legacy PROMs introduces the potential for measurement bias in the form of response fatigue. Fifth, there are potential unmeasured influences on completion rates, including differences in health literacy, social determinants of health, and presenting etiology (e.g., traumatic versus elective). Sixth, there may be insights that can be gained by assessing not only patients who do or do not complete requested PROMs but also those who begin but do not fully complete assigned PROMs. In our sample, only 3,726 patients (1.7%) began but did not complete their PROMs, limiting any analysis of this group. CAT forms may improve completion rates for this group of patients as well. Seventh, the patients in our sample were considered “new” if they had never been seen before or had not been seen in the past 3 years, which is standard billing practice. It is possible that some patients received care >3 years ago and returned as a “new” patient. Additionally, the present study investigated patients presenting to the hospital’s main clinic and not to the community clinics within the hospital system. This variable, in addition to measurement bias, may explain the variation in completion rates between more centralized subspecialties (e.g., spine) and outpatient disciplines (e.g., foot and ankle). Therefore, orthopaedic surgeons should consider the variation that exists between physical clinic locations when analyzing completion rates. Finally, the number of significant results in our work is likely indicative of our very large sample size; thus, any small variation may be significant, but it would not necessarily be clinically relevant. As such, we have sought to highlight the variables with the strongest associations, and with most clinical relevance.

Despite its limitations, the present study provides important insight into the likely set of patients used for PROM research in orthopaedic surgery. Unfortunately, our work suggests that a large proportion of patients, especially minorities and those commonly facing disparities in care, are not completing PROMs at new patient visits. It will be critical to see if our findings hold among patients seeking follow-up care as well. Overall, it is important to increase PROM completion rates to capture an accurate representation of the patients being treated. In turn, this will improve the accuracy of clinical research and health-care policy using PROM scores for measurement.

TABLE I

Descriptive Characteristics and Comparison of Patient Characteristics by PROM Completion Status*

Characteristic	Total (N = 219,891)	Completed (N = 88,052)	Not Completed (N = 131,839)	P Value
PROM completion
Yes	88,052 (40%)	—	—
No	131,839 (60%)	—	—
Year				<0.001
2016	26,747 (12%)	5,116 (5.8%)	21,631 (16%)
2017	50,718 (23%)	16,982 (19%)	33,736 (26%)
2018	53,515 (24%)	23,087 (26%)	30,428 (23%)
2019	54,908 (25%)	27,203 (31%)	27,705 (21%)
2020	34,003 (16%)	15,664 (18%)	18,339 (14%)
EMR patient gateway activated				<0.001
Yes	155,685 (71%)	67,005 (76%)	88,680 (67%)
No	64,206 (29%)	21,047 (24%)	43,159 (33%)
Age				<0.001
<40 yr	41,393 (19%)	16,922 (19%)	24,471 (19%)
40-64 yr	90,405 (41%)	36,252 (41%)	54,153 (41%)
≥65 yr	88,093 (40%)	34,878 (40%)	53,215 (40%)
Sex				<0.001
Female	115,446 (53%)	46,734 (53%)	68,712 (52%)
Male	104,445 (48%)	41,318 (47%)	63,127 (48%)
Race				<0.001
White	178,001 (81%)	74,234 (84%)	103,767 (79%)
Black	11,999 (5.5%)	3,498 (4.0%)	8,501 (6.4%)
Asian	8,528 (3.9%)	3,163 (3.6%)	5,365 (4.1%)
Other	21,363 (9.7%)	7,157 (8.1%)	14,206 (11%)
Language				<0.001
English	205,209 (93%)	84,335 (96%)	120,874 (92%)
Non-English	14,682 (6.7%)	3,717 (4.2%)	10,965 (8.3%)
Marital status				<0.001
Married	112,815 (51%)	47,456 (54%)	65,359 (50%)
Single	69,358 (32%)	26,149 (30%)	43,209 (33%)
Divorced	16,064 (7.3%)	6,292 (7.1%)	9,772 (7.4%)
Widowed	13,403 (6.1%)	4,998 (5.7%)	8,405 (6.4%)
Other	8,251 (3.8%)	3,157 (3.6%)	5,094 (3.9%)
Payer				<0.001
Commercial	121,918 (55%)	50,184 (57%)	71,734 (54%)
Medicare	70,686 (32%)	27,489 (31%)	43,197 (33%)
Medicaid	15,209 (6.9%)	5,556 (6.3%)	9,653 (7.3%)
Workers’ Compensation	7,292 (3.3%)	3,002 (3.4%)	4,290 (3.3%)
Other	4,786 (2.2%)	1,821 (2.1%)	2,965 (2.2%)
Orthopaedic subspecialty				<0.001
Hand	72,789 (33%)	25,230 (29%)	47,559 (36%)
Foot & ankle	48,741 (22%)	6,461 (7.3%)	42,280 (32%)
Trauma	27,132 (12%)	17,672 (20%)	9,460 (7.2%)
Arthroplasty	26,047 (12%)	11,382 (13%)	14,665 (11%)
Oncology	25,887 (12%)	12,673 (14%)	13,214 (10%)
Spine	18,160 (8.3%)	13,934 (16%)	4,226 (3.2%)
Other	1,135 (0.5%)	700 (0.8%)	435 (0.3%)

The values are given as the number of patients, with the percentage in parentheses.

TABLE II

Factors Associated with Failure to Complete PROMs at New Patient Presentation (N = 219,891)*

Characteristic	Adjusted Odds Ratio	95% Confidence Interval	P Value
EMR patient gateway activated
No	1.62	1.58 to 1.66	<0.001
Year of visit
2020	Reference
2019	0.62	0.60 to 0.64	<0.001
2018	0.77	0.75 to 0.80	<0.001
2017	1.18	1.14 to 1.22	<0.001
2016	2.88	2.76 to 2.99	<0.001
Age
<40 yr	Reference
40-64 yr	1.15	1.12 to 1.18	<0.001
≥65 yr	1.23	1.19 to 1.27	<0.001
Sex
Male	1.12	1.10 to 1.14	<0.001
Race
White	Reference
Black	1.24	1.19 to 1.30	<0.001
Asian	1.04	0.99 to 1.10	0.10
Other	1.07	1.03 to 1.11	<0.001
Language
Non-English	1.90	1.82 to 2.00	<0.001
Marital status
Married	Reference
Single	1.07	1.05 to 1.10	<0.001
Divorced	1.10	1.06 to 1.15	<0.001
Widowed	1.20	1.14 to 1.25	<0.001
Other	1.12	1.06 to 1.18	<0.001
Payer
Commercial	Reference
Medicare	1.06	1.03 to 1.09	<0.001
Medicaid	1.12	1.07 to 1.16	<0.001
Workers’ Compensation	1.06	1.00 to 1.11	0.048
Other	1.17	1.10 to 1.25	<0.001
Orthopaedic subspecialty
Foot & ankle	Reference
Hand	0.28	0.27 to 0.29	<0.001
Spine	0.04	0.04 to 0.04	<0.001
Trauma	0.07	0.06 to 0.07	<0.001
Arthroplasty	0.17	0.17 to 0.18	<0.001
Oncology	0.14	0.14 to 0.15	<0.001
Other	0.12	0.10 to 0.13	<0.001

Pseudo-R2 = 0.17.