Improving the response rate of patient-reported outcome measures in an Australian tertiary metropolitan hospital.

Background: The benefits of collecting patient-reported outcome measures (PROMs) for clinical care are widely accepted; however, the collection and response rate remain a significant barrier. Objective: The objective of this study was to assess predictors of successful PROM response rate in an orthopedic outpatient setting at a public tertiary hospital. Method: A prospective cohort study was conducted at a metropolitan hospital assessing the response rate after a number of interventions in the collection of PROMs in the orthopedic outpatient setting. All patients were invited to complete a PROM relevant to their presenting condition. Eight cohorts were studied, all different in the process of collection, the timing of collection and the physical environment of collection. Analysis was completed in Stata v14.1 with collin commands used to assess collinearity. A multiple logistic regression model and a mixed effect logistic regression model were performed and compared. The significance level of p<0.05 was used.
Results: During the trial period 2,338 patients were seen. Response rates as high as 81% were seen, which was significantly improved compared with the earliest six cohorts (p<0.01). Being younger; being a new patient; having a longer wait time; having an English-speaking background and being a pre- or post-operative patient were all associated with an increased response rate of PROMs. Gender, the patient's pathology and the type of PROM did not significantly affect the response rate. Conclusions: The method employed to invite and inform patients of the PROM collection, and the environment in which it is undertaken, significantly alter the response rate in the completion of PROMs. Being younger, being a new patient, having a longer wait time, being English speaking, being a pre-op or post-op patient were all associated with an increase in response rate.

Key points for decision makers

Routine collection of patient-reported outcome measures (PROMs) is clinically important for the ongoing care of patients.

The low response rate in the collection of PROMs is one of the major hurdles toachieving this.

Response rate is highly variable but can be dramatically improved through implement-ing a variety of techniques.

Introduction

PROMs are objective questionnaires used to measure quality of life data.1–4 Many PROMs are now considered the gold standard in measuring outcomes in orthopedics5,6 and their use improves both patient satisfaction and clinical practice.4,7 New Zealand, Sweden, and the UK have been leading the world in the collection of routine PROMs for joint registries, paving the way for quality of life data to be analyzed nationally.1,8–10 Despite the benefits associated with the collection of PROMs,11,12,21 its uptake into routine clinical practice around the rest of the world remains slow.1

There are many challenges to the routine collection of PROMs in the public hospital system1,2,10 including the time, effort and cost required.13 Although the development of a web-based repository for PROMs can help tackle some of these challenges,14,15 the utilization of this method remains limited within the orthopedic patient population.1,16,17 Another challenge is the low response rate, which is often fueled by disinterest, lack of time or inability to comprehend the questions.18 Certain characteristics that hinder the successful completion of PROMs include age greater than 75 years, social economic status, living alone, poorer pre-operative health, Hispanic or Black ethnicity, private health insurance and participants who have had previous surgeries.11,18

There is currently a lack of literature on overcoming the ongoing challenge of low response rate to patient outcome collection. The purpose of this study is to assess the predictors of completion of PROMs in the orthopedic outpatient clinic at an Australian public hospital and determine what strategies lead to an improvement in PROM response.

Methods

A prospective cohort study was conducted in an orthopedic outpatient setting of a metropolitan tertiary hospital assessing the response rate of PROM collection. Data collection occurred from February to June 2016. All patients attending the outpatient clinic were invited to complete a PROM relevant to their presenting condition via paper or digital format. All patients were asked to complete two generic wellbeing and condition-specific PROMs relevant to their presenting condition and operation (Table 1).

Table 1

Condition-specific PROMs for common operations

PROM	Operation
Disability of the Arm Shoulder and Hand (DASH)29	All shoulder operations
Oxford Knee Score (OKS)5	Total knee replacement
	Knee arthroscopy
Oxford Hip Score (OHS)5	Total hip replacement
Oxford Shoulder Score (OSS)30	Total shoulder replacement
International Hip Outcome Tool (IHOT)31	Hip arthroscopy
Western Ontario Rotator Cuff (WORC)32	Acromioplasty
	Rotator cuff repair
Western Ontario Shoulder Instability Index (WOSI)33	Labral repair and stabilization Latarjet procedure
Manchester-Oxford Foot Questionnaire (MOXFQ)34	Forefoot, midfoot and hindfoot procedures
Visual Analog Scale (VAS)35	Generic wellbeing PROM
EuroQol five dimensions (EQ5D)36	Generic wellbeing PROM

The primary outcome was the response rate, which was defined as fully completing and returning the PROMs back to the investigators. Data collected included patient demographics (age, sex, primary language spoken) and visit outcome. All patients were able to opt out of filling in a PROM and had their reason recorded for analysis. During the study period, eight different methods or environmental variations were employed and the response rate recorded (Table 2).

Table 2

The eight different interventions trialed throughout the study period

	Consult timing	Approached	Instruction sheet	Routine care	Opt in/Opt out	Surgeon assessment	Research desk	Directed to desk
Cohort A	Before	No	No	No	Opt in	No	No	No
Cohort B	After	No	No	No	Opt in	No	No	No
Cohort C	After	Yes	No	No	Opt in	No	No	No
Cohort D	After	Yes	Yes	No	Opt in	No	No	No
Cohort E	After	Yes	Yes	Yes	Opt out	No	No	No
Cohort F	After	Yes	Yes	Yes	Opt out	Yes	No	No
Cohort G	After	Yes	Yes	Yes	Opt out	No	Yes	No
Cohort H	After	Yes	Yes	Yes	Opt out	No	Yes	Yes
Summary
Cohort A	Verbally invited to complete a PROM after seeing their surgeon.
Cohort B	Verbally invited to complete a PROM prior to seeing their surgeon in the waiting room.
Cohort C	Approached and verbally invited to complete a PROM prior to seeing their surgeon.
Cohort D	Approached, given an instruction sheet that explained the research and verbally invited to complete a PROM prior to seeing their surgeon.
Cohort E	Received an instruction sheet, asking to complete a PROM as part of routine care prior to seeing their surgeon
Cohort F	Received an instruction sheet, asking to complete a PROM as part of routine care prior to seeing their surgeon and instructed that the consulting surgeon would be assessing the form.
Cohort G	A registration desk was set up where patients were given an instruction sheet, asked to complete the PROM as part of routine care and return afterwards.
Cohort H	Instructed to line up at the registration desk where they were given an instruction sheet, asked to complete the PROM as part of routine care and return afterwards.

Data analysis

A bivariate analysis was performed on the demographic data (Table 3). Age and time spent were reviewed as continuous variables; sex (male vs female), language (English vs non-English speaking), PROM pathology (upper vs lower limb) and the PROM type (visual analog scales vs Likert scales) as binary variables; and visit outcome as nominal categories (pre-, post-, non-op). Continuous variables were presented as means/standard deviations, and analyzed with simple logistic regression analysis. Categorical variables were presented as absolute numbers/frequency and analyzed using a chi-squared/Fisher’s exact analysis. Those that had significance of p<0.05 were retained for model building. Collinearity was assessed between variables and one factor (PROM questionnaire) was found to be strongly correlated R ≥0.5 so removed from the multivariate analysis. A manual backward stepwise regression was used to develop a multivariate logistic regression model of predictors of response rate of PROMs. A mixed effect analysis was used to assess for any unmeasured influences by repeat observations of Cohort H over 9 weeks of intervention, and the effect applied to the other cohorts. Only factors significant at the α =0.05 level were retained in the final model, except age and sex which were considered as potential confounders. The results of the multivariate analysis (Table 4) were reported with ORs and 95% CIs. Analysis was completed in Stata v14.1 with collin commands used to assess collinearity.

Table 3

Characteristic	N	%	95% CI	P-value
Sex				0.866
Male	900	47.07	44.84 - 49.31
Female	1,012	52.93	50.69 - 55.16
Language				<0.001
Non-English	387	20.45	18.7 - 22.33
English	1,505	79.55	77.67 - 81.3
Outcome				<0.001
Non-op	461	44.9	40.41 - 49.48
Pre-op	689	68.8	65.23 - 72.15
Post-op	774	64.21	60.76 - 67.52
Pathology				0.93
Upper limb	609	61.08	57.14 - 64.88
Lower limb	1,315	61.29	58.63 - 63.89
PROM measurement				0.506
Likert scale	1,551	60.86	58.41 - 63.27
Visual Analog Scale	373	62.73	57.7 - 67.51
Visit type				<0.001
New	629	67.57	63.8 - 71.11
Review	1,258	58.35	55.6 - 61.05
PROM filled				0.122
DASH	231	58.44	52.07 - 64.81
IHOT	55	67.27	54.75 - 79.8
MOXFQ	337	64.39	59.27 - 69.51
OHS	279	60.57	54.83 - 66.32
OKS	644	59.47	55.67 - 63.27
OSS	60	66.67	54.63 - 78.7
WORC	246	58.13	51.95 - 64.31
WOSI	72	75	64.92 - 85.08
Cohort				<0.001
A	150	13.33	8.75 - 19.79
B	90	21.11	13.85 - 30.81
C	57	36.94	29.73 - 44.79
D	272	56.25	50.28 - 62.05
E	129	62.79	54.11 - 70.72
F	184	54.89	47.63 - 61.95
G	110	65.45	56.07 - 73.77
H	832	81.01	78.2 - 83.54

Notes: Cohort A: verbally invited to complete a PROM after seeing their surgeon. Cohort B: verbally invited to complete a PROM prior to seeing their surgeon in the waiting room. Cohort C: approached and verbally invited to complete a PROM prior to seeing their surgeon. Cohort D: approached, given an instruction sheet that explained the research and verbally invited to complete a PROM prior to seeing their surgeon. Cohort E: received an instruction sheet, asking to complete a PROM as part of routine care prior to seeing their surgeon. Cohort F: received an instruction sheet, asking to complete a PROM as part of routine care prior to seeing their surgeon and instructed that the consulting surgeon would be assessing the form. Cohort G: a registration desk was set up where patients were given an instruction sheet, asked to complete the PROM as part of routine care and return afterwards. Cohort H: instructed to line up at the registration desk where they were given an instruction sheet, asked to complete the PROM as part of routine care and return afterwards.

Abbreviations: op, operation; Disability of the Arm Shoulder and Hand (DASH); Oxford Knee Score (OKS); Oxford Hip Score (OHS); Oxford Hip Score (OHS); Oxford Shoulder Score (OSS); International Hip Outcome Tool (IHOT); Western Ontario Rotator Cuff (WORC); Western Ontario Shoulder Instability Index (WOSI); Manchester-Oxford Foot Questionnaire (MOXFQ); Visual Analog Scale (VAS); EuroQol five dimensions (EQ5D).

Table 4

Multivariate analysis

Characteristics	Naive logistic regression model				Mixed effect logistic regression model
OR	Standard error	P-value	95% CI	OR	Standard error	P-value	95% CI
Age	0.99	0.00324	0.002	0.984 - 0.996	0.99	0.00324	0.002	0.984 - 0.996
Sex
Male
Female	1.063	0.1252	0.602	0.844 - 1.339	1.063	0.1252	0.602	0.844 - 1.339
Wait time	1.001	0.0006	0.042	1 - 1.002	1.001	0.0006	0.042	1 - 1.002
Language
Non-English
English	3.839	0.5419	<0.001	2.911 - 5.063	3.839	0.5419	<0.001	2.911 - 5.063
Outcome
Non-op
Pre-op	1.964	0.30,342	<0.001	1.451 - 2.658	1.964	0.30,342	<0.001	1.451 - 2.658
Post-op	1.927	0.29,429	<0.001	1.429 - 2.6	1.927	0.29,429	<0.001	1.429 - 2.6
Pathology
Upper limb
Lower limb	1.198	0.18,126	0.232	0.891 - 1.612	1.198	0.18,126	0.232	0.891 - 1.612
PROM type
Likert scale
Visual Analog scale	1.224	0.21,947	0.261	0.861 - 1.739	1.224	0.21,947	0.261	0.861 - 1.739
Visit type
New
Review	0.649	0.09549	0.003	0.486 - 0.866	0.649	0.09549	0.003	0.486 - 0.866
Cohort
H
G	0.662	0.18,093	0.132	0.388 - 1.131	0.662	0.18,093	0.132	0.388 - 1.131
F	0.241	0.0446	<0.001	0.168 - 0.347	0.241	0.0446	<0.001	0.168 - 0.347
E	0.403	0.09157	<0.001	0.258 - 0.629	0.403	0.09157	<0.001	0.258 - 0.629
D	0.317	0.05405	<0.001	0.227 - 0.442	0.317	0.05405	<0.001	0.227 - 0.442
C	0.115	0.02343	<0.001	0.077 - 0.172	0.115	0.02343	<0.001	0.077 - 0.172
B	0.058	0.01719	<0.001	0.032 - 0.103	0.058	0.01719	<0.001	0.032 - 0.103
A	0.03	0.00806	<0.001	0.018 - 0.051	0.03	0.00806	<0.001	0.018 - 0.051

Notes: Cohort A: verbally invited to complete a PROM after seeing their surgeon. Cohort B: verbally invited to complete a PROM prior to seeing their surgeon in the waiting room. Cohort C: approached and verbally invited to complete a PROM prior to seeing their surgeon. Cohort D: approached, given an instruction sheet that explained the research and verbally invited to complete a PROM prior to seeing their surgeon. Cohort E: received an instruction sheet, asking to complete a PROM as part of routine care prior to seeing their surgeon. Cohort F: received an instruction sheet, asking to complete a PROM as part of routine care prior to seeing their surgeon and instructed that the consulting surgeon would be assessing the form. Cohort G: a registration desk was set up where patients were given an instruction sheet, asked to complete the PROM as part of routine care and return afterwards. Cohort H: instructed to line up at the registration desk where they were given an instruction sheet, asked to complete the PROM as part of routine care and return afterwards.

Abbreviation: op, operation.

Results

Bivariate analysis

Over the course of the research period, 2,338 patients were seen; of these, 414 were excluded and 1,924 patients used for analysis. Of the analyzed patients, 47.1% were males (mean age 51.16, s.d. 18.79) and 52.9% females (mean age 58.14, s.d. 17.29) (Figure 2). Gender was not found to have a significant effect on response rates between males and females (χ<0.03, p=0.87). Increasing age was found to adversely affect rate of response with a significance of (p=0.002, OR 0.99, 95% CI: 0.987, 0.997) with a significant drop in response between patients aged 60–69 compared with 50–59 (p=0.011, OR 0.69, 95% CI: 0.514, 0.918). Of the 40 different languages spoken by patients attending clinic (Table 5), English-speaking patients completed a PROM 67.18% of the time, compared with non-English speaking patients who completed it 40.05% of the time (χ2=95.78, p<0.01). New referral patients had a response rate of 68.8%, review patients with non-operative management were least likely at a rate of 44.9% and review patients who were already on a surgical waiting list had a response rate of 64.21%. The differences in response rate were found to be significant between these groups (χ=71.28, p<0.01). There was no significant difference in response rate between patients who had an upper limb pathology (61.08%) and a lower limb pathology (61.29%) (χ=0.008, p=0.93). There was no difference found in response rate between PROMs utilizing Likert scales (60.86%) and those utilizing a visual analog scale (62.73%) (χ=0.44, p=0.506). The response rate based on specific PROMs was found in patients completing the Western Ontario Shoulder Instability Index (75%), International Hip Outcome Tool (67.27%), Oxford Shoulder Score (66.67%), Manchester-Oxford Foot Score (64.39%), Oxford Hip Score (60.57%), Oxford Knee Score (59.47%), Disabilities of the Arm, Shoulder and Hand Score (58.44%) and Western Ontario Rotator Cuff Index (58.13%). However, these differences were not statistically significant (χ=11.4, p=0.122).

Figure 2

Gender age distribution.

Table 5

List of languages spoken by patients

List of languages	Number
English	2,180
Vietnamese	80
Macedonian	45
Greek	40
Italian	27
Arabic (including Lebanese)	25
Spanish	23
Croatian	21
Persian (excluding Dari)	18
Cantonese	17
Mandarin	13
Tamil	11
Dinka	10
Serbian	7
Serbo-Croatian/Yugoslavian	7
Turkish	7
Burmese	6
Dari	6
Bosnian	5
French	5
Maltese	5
Polish	5
Punjabi	4
Karen	3
Portuguese	3
Albanian	2
Hakka	2
Hindi	2
Pashto	2
Samoan	2
Slovene	2
Somali	2
Tigrinya	2
Urdu	2
African Languages	1
Assyrian Neo-Aramaic	1
Auslan	1
Azeri	1
Chaldean Neo-Aramaic	1
Eastern Anmatyerr	1
Hazaraghi	1
Hungarian	1
Lao	1
Nepali	1
Oromo	1
Romanian	1
Sinhalese	1
Slovak	1
Swahili	1
Telugu	1
Thai	1
Tibetan	1
Tigre	1

We found that the highest response rate was in Cohort H (81.01%); this was followed by Cohort G (65.45%), Cohort E (62.79%), Cohort D (56.25%), Cohort F (54.89%), Cohort C (36.94%), Cohort B (21.11%) and Cohort A (13.33%) (Figure 1). The difference in response rate between Cohort H and the other variables were significant (χ=389, p<0.01).

Figure 1

Response rates for all eight cohorts.Notes: Cohort A: verbally invited to complete a PROM after seeing their surgeon. Cohort B: verbally invited to complete a PROM prior to seeing their surgeon in the waiting room. Cohort C: approached and verbally invited to complete a PROM prior to seeing their surgeon. Cohort D: approached, given an instruction sheet that explained the research and verbally invited to complete a PROM prior to seeing their surgeon. Cohort E: received an instruction sheet, asking to complete a PROM as part of routine care prior to seeing their surgeon. Cohort F: received an instruction sheet, asking to complete a PROM as part of routine care prior to seeing their surgeon and instructed that the consulting surgeon would be assessing the form. Cohort G: a registration desk was set up where patients were given an instruction sheet, asked to complete the PROM as part of routine care and return afterwards. Cohort H: instructed to line up at the registration desk where they were given an instruction sheet, asked to complete the PROM as part of routine care and return afterwards.

There were multiple reasons why a patient did not complete a PROM. These included patients leaving the waiting room to (30%), requiring an interpreter (27%), recently completing the same PROM (19%), patients who had other physical or medical impairment not related to their orthopedic pathology that would confound the PROM results (8%), refusal (4%), missed patients (2%), vision impairment (1%), poor literacy (0.8%), incomplete PROMs (0.2%), extreme pain (0.1%), and being in handcuffs (0.1%).

Multivariate analysis

There was no difference found between the naive model and the mixed effect model (Table 4). Overall the mixed-effect logistic regression model found that there was a significant difference (p<0.01) between the eight cohorts after including the independent variables into the model. Of these variables, there were multiple predictors that were found to be associated with a better response rate. These predictors were: being younger (OR=0.99, 95% CI: 0.984, 0.996); having a longer wait time (OR=1.001, 95% CI: 1.00, 1.002); being of English-speaking background (OR=3.758, 95% CI: 2.854, 4.949); being a new patient (OR=2.227, 95% CI: 1.662, 2.984) or post-operative patient (OR=1.738, 95% CI: 1.299, 2.324) compared with non-operative patients; and having the intervention performed on Cohort H compared with Cohort A (OR=0.0296, 95% CI: 0.018, 0.05), Cohort B (OR=0.0.058, 95% CI: 0.033, 0.105), Cohort C (OR=0.113, 95% CI: 0.076, 0.168), Cohort D (OR=0.334, 95% CI: 0.240, 0.465), Cohort E (OR=0.392, 95% CI: 0.251, 0.611) and Cohort F (OR=0.235, 95% CI: 0.164, 0.337).

Discussion

The benefits of routine collection of PROMs are well documented.1,12,19 They improve patient satisfaction, enable early screening of functional decline and are extremely informative for clinicians.4,7,20,21 PROMs improve patient consultation and can be used for national quality control and population-based research.8,22,23 However, poor response rate continues to be a challenge that hinders its routine collection.11,18

The findings of this study show that the response rate can be significantly improved through the use of different methods or environmental variations. The most successful approach was found in Cohort H utilizing strategies including collecting PROMs prior to clinic, approaching patients individually, directing patients through an administrative area, distributing an information sheet and informing patients that the collection of PROMs was part of routine care. This approach had a response rate of 81.35%, which compares favorably with other reported PROM collection registries – UK National Joint Registry (64%), the Swedish Hip Arthroplasty Registry (79%), and the New Zealand Joint Registry (70%).8,10,24–26 We found that making PROM collection a part of routine care and the development of a specific PROM administration desk yielded the greatest improvement in response rate. We found that the response rate dropped when patients were asked to take their completed PROM into the consulting room to be assessed by the surgeon (p=0.08, OR 0.64, 95% CI 0.389, 1.058). This finding reflects the logistical difficulty of PROM collection when a multitude of users are required to participate in the collection process.

Currently, there are multiple known predictors to poor response rates described in the literature.1,11,18,27 Our study reinforces some of these predictors, including age and language.18 Other predictors such as a shorter length of time in the waiting room and review patients for non-operative treatment were also found to reduce the response rate to PROMs. Visual analog scales have been demonstrated as the most accurate measure for disabilities.25 Our study also found no difference in response rates between PROMs which utilized visual analog scales and Likert scales.

Overall the results of our study have identified factors that can be broadly split into modifiable and non-modifiable factors. The modifiable factors, such as the environmental collection modifications demonstrated in our study, were found to sequentially improve the collection response rate. Non-modifiable factors include age, language, literacy and computer skills, current stage within the surgical journey, vision impairment and patient time constraints. Measures to overcome these factors include the availability of paper format, using multi-lingual PROMs, staff availability to assist with vision impairment, literacy and computer skills, and providing information regarding the purpose of PROM collection and utilizing the patients waiting time in outpatients to collect PROMs.

This study aims to describe steps to improve the significant challenge of response rate in PROM collection. Future routine collection of PROMs will enhance the patient experience as well as for population-level quality assurance and research.2,28

Limitations

There are a number of limitations that can be identified with this study. Although this was a prospective cohort study, there was no randomization. Allocation of patients into cohorts presents potential confounding variables, including temporal bias, which could be overcome by repeating each collection method over multiple clinics or randomization of patients to different methods of collection during a clinic. However, this would be logistically infeasible at our institution. Participants of this study are a small subset of orthopedic patients at a large metropolitan public hospital with particular socioeconomic, education and language variables and as such the results may not be generalized to dissimilar populations.

The future

The future of the collection of PROMs is promising and following completion of this study, our goal is to implement the routine collection of PROMs with the lessons learned from our research. Cost and efficiency can be improved by the use of a web-based repository system to automate much of this process, requiring only the application and maintenance of the program.14 Such a system has been developed at our institution14 and successful implementation is the next major challenge that we face.

Conclusion

One of the major barriers to the routine collection of PROMs in the outpatient setting is the low patient response rate, and this study identifies several predictors that affect successful completion. Known predictors such as age and language were reconfirmed and others such as waiting time in clinic, new referral patients, and pre-operative patients were identified. This study also demonstrates that a high response rate can be achieved by alterations in the method of collection.

Table 3

Bivariate analysis

Characteristic	N	Mean	SD	P-value
Age				0.002
Not respond	741	56.5	18.11
Respond	1,171	53.81	18.42
Wait time				0.012
Not respond	688	147.63	95.05
Respond	1,144	159.61	100.85