Adding telephone follow-up can improve representativeness of surveys of seriously ill people.

INTRODUCTION

A growing population of seriously ill people in the United States receives care from community‐based serious illness programs. Surveys of patients' experiences with these programs can help identify areas for improvement and compare care quality across programs. However, survey respondents may be systematically different from nonrespondents in ways related to their experiences of care, which could bias the results by underrepresenting the views of underserved patient groups such as those with Medicaid coverage.

To test whether mixed mode survey administration could improve single‐mode response rates and representativeness for seriously ill patients, we conducted an experiment in which sampled patients from home‐based serious illness programs were randomized to mixed mode (mail with telephone follow‐up) or mail‐only survey administration.

METHODS

We surveyed patients aged 18+ receiving care from home‐based U.S. serious illness programs. The survey was available in English and Spanish and designed to be completed by the patient or proxy respondent (i.e., family caregiver). The mail‐only mode consisted of a pre‐notification letter, followed by a mail survey 1 week later, and an additional mail survey 3 weeks later. The mixed mode consisted of a pre‐notification letter, followed by a mail survey 1 week later, and up to five calls to complete the survey by phone if the mail survey was not returned after 3 weeks.

The survey instrument contained 56 items assessing communication, care coordination, help for symptoms, planning for care, and support for family and friends. Final, more concise versions of the survey are available online.

Program administrative data included the patient's sex, age, residential setting (home vs. assisted living facility), primary payer for care, and primary diagnosis, and number of in‐person visits.

We predicted survey response from patient characteristics and mode of administration using multivariate logistic regression models.

To determine whether mixed mode particularly benefits some patient groups in a way that affects survey representativeness, we added interactions between mode and all patient characteristics. We also conducted a stratified analysis, fitting the main effects regression model separately for sampled cases assigned to the mail‐only and mixed modes.

RESULTS

There were 2263 eligible respondents (overall response rate 36.4%; 30.4% in mail‐only mode, 42.5% in mixed mode).

Response rates increased with age. Those who lived in an assisted living facility had less than half the odds of responding as those in a private home. Patients with Medicaid as the primary payer for care were less likely to respond (odds ratio [OR]: 0.69) than those with Medicare as the primary payer. The odds of response were higher for those with more in‐person visits.

Mode was one of the strongest predictors of response, with mixed mode yielding 74% greater odds of responding than mail‐only (OR: 1.74; Table 1). Tests for heterogeneity of the mixed mode advantage by patient characteristics found significant differences only for the primary payer (p = 0.01). These results suggested that mixed mode may be most beneficial for patients with Medicaid as the primary payer (OR: 3.50 for mixed vs. mail mode), with response rates of 43.8% in mixed mode and 17.9% in mail‐only mode. In stratified regression models within the mail‐only and mixed mode strata, patients in the mail‐only mode with Medicaid as their primary payer were substantially less likely to respond than patients with Medicare as the primary payer (OR: 0.43 for Medicaid relative to Medicare; p < 0.01 for payer). However, in the mixed mode stratum, odds of response did not differ significantly across primary payer types (OR: 0.89 for Medicaid relative to Medicare, p = 0.85 for payer). Figure 1 shows that mixed mode response rates are consistently higher than mail‐only for all ages, response rates are higher for older age groups, and that mixed mode is especially beneficial in increasing response rates for those on Medicaid.

TABLE 1

Characteristics of patients sampled and responding to serious illness care survey

Characteristic	Eligible sampled (N = 6210) (%)	Mail‐only (N = 3102) RR in category (%)	Mixed (N = 3108) RR in category (%)	Adjusted OR of responding by mixed mode versus mail only (p‐value) a
Any	100.0	30.4	42.5	1.74 (p < 0.01) b
Sex
Female (reference)	59.7	29.2	42.4	1.90 (p < 0.01)
Male	40.3	32.1	42.8	1.59 (p < 0.01)
Age
18–54	5.9	16.9	35.9	2.54 (p < 0.01)
55–64	10.3	25.9	40.5	1.84 (p < 0.01)
65–69	7.0	30.0	43.3	1.87 (p < 0.01)
70–74	10.9	29.2	42.6	1.89 (p < 0.01)
75–79	13.1	29.7	39.4	1.67 (p < 0.01)
80–84	16.3	31.7	45.1	1.90 (p < 0.01)
85–89	17.6	36.1	45.2	1.54 (p < 0.01)
90 or older (reference)	19.0	31.5	42.9	1.63 (p < 0.01)
Residential setting
Unknown	23.3	23.9	40.7	2.22 (p = 0.13)
Facility	8.1	19.5	26.8	1.76 (p = 0.04)
Home (reference)	68.6	33.9	44.9	1.64 (p < 0.01)
Primary diagnosis
Cancer	15.6	25.4	37.9	1.85 (p < 0.01)
Alzheimer's or dementia	10.2	27.2	40.2	2.07 (p < 0.01)
Other (reference)	74.3	31.9	43.7	1.72 (p < 0.01)
Number of in‐person visits
Unknown	24.7	23.9	40.3	1.63 (p = 0.17)
1–2 times	15.4	28.5	31.5	1.36 (p = 0.09)
3–4 times	12.2	27.2	41.4	2.11 (p < 0.01)
5–6 times	9.8	33.0	43.4	1.98 (p < 0.01)
7 or more times (reference)	37.9	35.9	48.2	1.91 (p < 0.01)
Primary payer
Medicare (reference)	48.1	32.1	40.6	1.55 (p < 0.01)
Medicaid	8.5	17.9	43.8	3.50 (p < 0.01)
Private	18.9	40.5	49.6	1.57 (p = 0.01)
Other or unknown	24.5	23.5	40.3	2.00 (p = 0.10)

Note: Percentages were calculated excluding missing values for all variables that had negligible missingness (<4%). For the three variables with non‐negligible missingness (residential setting, number of in‐person visits, and primary payer), we included missing/unknown as a separate category. p‐values from joint significance tests for interactions with mode were 0.12 (sex), 0.81 (age), 0.87 (residential setting), 0.59 (primary diagnosis), 0.25 (number of in‐person visits), and 0.01 (primary payer). We imputed missing patient‐level characteristics with the program mean for that variable, except for the three variables with non‐negligible missingness (residential setting, number of in‐person visits, and primary payer), for which we included missing/unknown as a separate category. If the variable was missing for the entire program, it was imputed with the overall mean. To ensure results were not sensitive to treatment of missing values, we conducted a sensitivity analysis that removed the small number of programs (4 of 32) that had an entire variable missing, and a complete case analysis. Results from main effects models as well as models that added interaction terms were not sensitive to treatment of missing values.

Abbreviations: OR, odds ratio; RR, response rate.

OR represent odds of responding by mixed mode, relative to mail‐only mode, for patients having the specified characteristic. p‐values correspond to the p‐value for the effect of mixed mode within the stratum, holding all other variables at their mean values. Results are from a logistic regression model for the probability of response, including all characteristics listed in the table, mode, and interaction terms between mode and all characteristics. The model includes program fixed effects.

OR represents odds of responding by mixed mode, relative to mail‐only mode, from a logistic regression model for the probability of response including only main effects for the characteristics listed in the table and program fixed effects.

FIGURE 1

Response rates by mode of survey administration, payer, and age group. Mail‐only mode response rates are shown in blue, mixed mode in red, and primary payer is distinguished by solid (all payers), dotted (Medicaid), and dashed line (non‐Medicaid) shaded bars. Sample sizes are based on eligible sampled cases. The four response rates corresponding to Medicaid payer crossed with mode within the 65–74 and 75+ age groups (mail‐only, Medicaid, 65–74; mail‐only, Medicaid, 75+; mixed, Medicaid 65–74; mixed, Medicaid, 75+) have low precision as they are based on small sample sizes

DISCUSSION

Seriously ill patients with low socioeconomic status are at greater risk of undertreatment and lower quality care6, 7; consequently, it is important to ensure that their experiences are included in assessments of care, and that Medicaid enrollees are adequately represented in patient survey responses used to measure care quality. Our results demonstrate the value of mixed mode survey administration in ensuring that the voices of these seriously ill individuals are heard.

Consistent with the findings for other patient experience surveys, we found that mixed mode of survey administration yielded a much higher response rate (42.5%) than mail only (30.4%), and that mixed mode was especially effective in increasing response rates among patients with Medicaid as the primary payer, more than doubling their response rates. This hard‐to‐reach group had lower response rates overall, and thus using a mixed mode of survey administration increases representativeness of survey respondents relative to the sample from which they are drawn.

Our results support the use of survey mixed mode administration to achieve broad representation of seriously ill people when evaluating quality of care.

CONFLICT OF INTEREST

The authors have no conflicts of interest.

AUTHOR CONTRIBUTIONS

All authors have read and approved the submission of this manuscript. Maria DeYoreo, Rebecca Anhang Price, Paul D. Cleary, Joan M. Teno, and Marc N. Elliott contributed substantially to conception and design; Melissa A. Bradley and Danielle Schlang to data acquisition; and all authors to analysis and interpretation of data. Maria DeYoreo and Rebecca Anhang Price drafted the article, and all authors contributed to critical revisions of the article for important intellectual content. Maria DeYoreo, Cheryl K. Montemayor, Anagha Tolpadi, and Marc N. Elliott were responsible for statistical analysis. Rebecca Anhang Price obtained funding and provided study supervision.

SPONSOR'S ROLE

The Moore Foundation funded this work but played no role in the design, methods, subject recruitment, data collections, analysis, and preparation of paper.