Comparative Usability Study of a Newly Created Patient-Centered Tool and Medicare.gov Plan Finder to Help Medicare Beneficiaries Choose Prescription Drug Plans.

INTRODUCTION: In response to reported difficulties in selecting a Medicare Part D prescription drug plan, we designed a patient-centered online Part D plan selection tool (CHOICE1.0) to simplify the selection process and to provide personalized, expert recommendations.
METHODS: This ethnographic comparative usability study observed 44 patients using the first version of the tool during Medicare 2016 Open Enrollment. Participants were observed as they chose their drug plan using Medicare.gov and 1 of 3 versions of CHOICE1.0 that varied in amount of expert guidance. Descriptive statistics were used to analyze exit survey data. The observations were video-recorded, and field notes were analyzed thematically.
RESULTS: Participants were significantly more satisfied with CHOICE1.0 for choosing a plan, understanding information, and ease of use compared to Medicare.gov. Those using expert versions of CHOICE1.0 were more likely to indicate their intention to switch plans than those using Medicare.gov, though they wanted to know the source and content.
CONCLUSION: The more patient-centered prescription drug choice tool improved user experience and enabled users to choose plans more consistent with expert recommendations.

Introduction

Choosing a Medicare Part D prescription drug plan is complicated. Complex plan features, including formularies, deductibles, coinsurance rates, out-of-pocket maximums, and complex cost sharing arrangements (“donut hole”), are difficult for many Americans—particularly older adults—to understand (1). In addition, people eligible for Part D coverage usually must choose from upward of 30 different plans (2), and many studies have documented that decision-making quality declines as choice set increases (3 –6). Finally, the complexity of the choice is compounded by the varying health needs of each individual, so the plan that best covers one person’s needs may differ from that of that person’s spouse or friend.

Correspondingly, many Medicare beneficiaries find choosing a prescription drug plan frustrating and confusing (3 -6). In a nationally representative survey of older adults, 62% of those who did not enroll in a Part D plan and more than half of those who did enroll agreed with the statement that “I had difficulty understanding how Medicare Part D works and what savings it would provide (4).” Reports from recent focus groups suggest that while more beneficiaries are using the Internet to research and compare plan options than in previous years, the annual open enrollment process for selecting or changing plans still seems to be confusing (7). These decisions are particularly difficult for less numerate Medicare beneficiaries (8,9). As such, patients are not selecting the lowest cost plan for their needs (10,11), which could potentially save them several hundred dollars a year, a significant amount for retirees on a fixed income. Previous research has shown that even small “nudges” can affect plan selection. For example, a simple letter moved people to lower cost plans, resulting in potential savings of $US100 a year (12).

Older adults utilize a wide range of resources to assist them in selecting a Medicare Part D plan. Some turn to experts, such as the Health Insurance Counseling and Advocacy Program and insurance brokers, while others refer to books, brochures, or the Internet for guidance (13). One focus group study regarding the experiences of beneficiaries selecting a Medicare Part D plan found that the most frequently reported website was the Medicare.gov Plan Finder tool. Hosted by the Centers for Medicare and Medicaid Services (CMS), this interactive online tool has beneficiaries enter their zip code and current medications and then provides a list of all the available plans in the region, as well as their estimated costs and coverage. While some participants found the CMS tool to be helpful, others described it as confusing and prone to glitches (13).

To alleviate the reported difficulties with the CMS online tool and in response to evidence that older adults were more likely to switch plans when provided with easy-to-access evidence on the costs associated with different plans (12), we developed a new online Medicare Part D selection tool, CHOICE1.0, with a multidisciplinary team of researchers at Palo Alto Medical Foundation Research Institute and Stanford University, as well as a software developer, user experience consultant, pharmacist stakeholder, and health IT executive at Palo Alto Medical Foundation (PAMF) Community Health Resource Center and patient stakeholders. Building on findings from our focus group research (13) and input from stakeholders, CHOICE1.0 incorporates user-centered design concepts by having a more streamlined interface and by directly importing beneficiaries’ medications that are listed in the electronic health record (EHR) of a large, multispecialty group practice, PAMF.

In our focus group, we also found that older adults wanted assistance in choosing a plan (13). As a result, we developed 3 versions of CHOICE1.0, each of which varied based on the extent to which and how they provide the user with personalized expert recommendations. In all versions, descriptions of the plan features were provided and patients were able to compare up to 3 plans at a time. In version 1 (Expert Recommendation), we highlighted 3 expert recommended plans for each patient based on their likely annual out-of-pocket spending, including plan premiums and spending on prescription drugs, and the Medicare star ratings (a measure of customer satisfaction). In version 2 (Guided Analysis), instead of explicitly identifying the recommended plan, we provided patients with a score for each plan based on the algorithm underlying the expert recommendation. In version 3 (Individual Analysis), we provided information on each of the features underlying the expert recommendation, but not the recommendation itself. The expert recommendation was based on individual cost estimates, including both premium and expected out-of-pocket spending for prescription drugs, and a CMS measure of plan quality.

Our objective was to compare the new CHOICE1.0 tool with CMS’s Medicare.gov Plan Finder, and our hypothesis was that CHOICE1.0 would make the Medicare Part D prescription drug plan selection process easier for older adults and direct them to plans that would lower their overall spending on prescription drugs.

Methods

Participants were patients of PAMF, a large, multispecialty medical foundation affiliate of Sutter Health, serving over 1 million patients in Northern California. We used survey and ethnographic methods to compare the experiences of older adult participants using CHOICE1.0 and the CMS tool during Medicare’s 2016 Open Enrollment Period (October 15, 2015, to December 7, 2015). Direct observation is an ideal method for gaining insight into and understanding of user needs and is commonly used by innovative product developers. By observing and documenting the tasks and procedures used by participants, we were able to gain insight into what worked and what needed to be improved.

Participants were recruited through active and passive means. Active recruitment occurred by members of the research team at multiple events and locations, including an “Introduction to Medicare” lecture, an event at a local older adult community, blood pressure clinics, and laboratories within PAMF clinics. Posters and flyers were placed in the waiting rooms of family medicine, internal medicine, and geriatric medicine departments at several PAMF clinics; an article seeking participants was included in the PAMF patient newsletter; a request for participants was included on the online patient portal (My Health Online) homepage; and an e-mail was sent to the electronic mailing lists of groups helping older adults (Avenidas Village, a local aging-in-place organization and a group discussing “Successful Aging”).

Potential participants were screened by a research team member either in person or over the phone for the following eligibility criteria: were of age 66 to 85 at the time of the study, were a PAMF patient, and had a Medicare Part D plan (not a Medicare Advantage Plan). If they met all of the eligibility criteria, they provided the research team member with their demographic information, which was then used to extract their medication data from the EpicCare EHR and input into CHOICE1.0. Participants were asked to bring a copy of their current medications and dosages to the observation session to verify the information in CHOICE1.0 and to enter in the CMS tool. Informed consent was obtained prior to participation.

Participants sat in front of a computer screen with full control of the mouse and keyboard while a research team member observed as they interacted with the tools. Participants were instructed by the researcher to verbally explain their thoughts and reactions to the tools as they navigated the different options. The researcher probed for further details and clarifications about participant thoughts and navigation choices.

Each participant sequentially used both CMS and 1 version of CHOICE1.0. Participants were randomly assigned both the order in which they used the tools (either CMS or CHOICE1.0 first) and which of the 3 versions of CHOICE1.0 they used. Each participant used 1 tool and completed a survey about their experience with that tool and then used the other tool and completed the same survey. All participants then completed a comparative survey about their use of the 2 tools (see Figure 1). Survey data were entered into WorldApp.

Figure 1.

Diagram and randomization of ethnographic comparative usability study.

Participants did not actually enroll in a plan during the comparative observation period—instead, they reported which plan they were likely to choose (their “intended choice”). Participants could print out lists of and information about potential plans for later use. They were not given a time limit to review the information. Instead, they indicated to the research team member when they were finished, generally between 1 to 2 hours. Upon completion, they received a $US100 gift card as a token of appreciation. The protocol was approved by the institutional review boards at Sutter Health and Stanford University.

Basic descriptive statistics were used to analyze participant demographics and survey responses. Statistical significance tests were performed to determine differences across versions of CHOICE1.0 and CMS. The first 3 outcome measures reflected potential changes in enrollment: “plan switching,” “enrollment in expert recommended plan,” and “enrollment in lowest cost plan” (see Table 1). For each of these measures, we calculated the percentages, that is, the percentage of participants who chose a different plan than their original one, the percentage of participants who enrolled in the expert recommended plan, and the percentage of participants who enrolled in the lowest cost plan, across CMS and CHOICE1.0 (overall and version 2 versus version 1 and version 3 versus version 1). To determine “change in coverage generosity,” we calculated the difference between the cost of the participant’s original plan, given their current set of drugs and considering both premium and out-of-pocket spending for drugs given the plan’s cost sharing structure, and that of the plan the participant chose on the day of the observation (Table 1). The other outcome measures reflect aspects of the participants’ experience in choosing a Medicare Part D prescription drug plan including how satisfied the patient was with the chosen plan, how confident the patient was in choosing a plan, how satisfied the patient was with the process of choosing a plan, how much the patient enjoyed choosing a plan, and whether the patient would have liked more or less information. Each of these measures was on a scale of 1 to 4, with 1 being the highest and 4 being the lowest. For example, in “How satisfied with plan,” 1 corresponds to being extremely satisfied and 4 corresponds to being not at all satisfied. Given that this was a small pilot study, we chose a minimum P value of 0.25 to infer the statistical significance of intervention effects, instead of relying on the traditional statistical significance levels (14).

Table 1.

CMS and CHOICE1.0 Survey Results.a

Variable	CMS	CHOICE1.0
		All CHOICE1.0 versionsb	Version 1: Expert Recommendation	Version 2: Guided Analysisc	Version 3: Individual Analysisc
Plan switching, %	60	67	71	67	62
Enrollment in expert recommended plan, %	49	71d	86	87	38d
Enrollment in lowest cost plan, %	20	40e	57	47	15d
Change in coverage generosity, mean (SD) ($US)	−129 (588)	−672e (1,839)	−886 (2,152)	−1015 (2,316)	−86e (125)
How satisfied with plan, mean (SD)f	1.76 (0.69)	1.81 (0.76)	1.93 (0.96)	1.87 (0.64)	1.62 (0.65)
How confident, mean (SD)f	1.95 (0.76)	2.10 (0.91)	2.14 (0.95)	2.13 (0.99)	2.00 (0.82)
How satisfied with process, mean (SD)f	2.60 (1.30)	2.00c (1.11)	1.80 (1.08)	2.14 (1.17)	2.07 (1.14)
How much enjoyment, mean (SD)f	2.85 (1.05)	2.20g (1.05)	2.40 (1.18)	2.00 (1.07)	2.21 (0.89)
More or less information, mean (SD)f	1.95 (0.79)	1.83 (0.66)	1.93 (0.73)	1.87 (0.74)	1.69 (0.48)
Number of participants	44	44	15	15	14

a Source: Medicare Part D Comparative Usability Study Questionnaires.

b Compared all versions of CHOICE1.0 with CMS Plan Finder.

c CHOICE1.0 versions 2 and 3 was compared to CHOICE1.0 version 1, respectively.

d P < .05.

e P < .25.

f Scale of 1 to 4, where 1 = more and 4 = less.

g P < .01.

Qualitative Observations

Sessions were video-recorded capturing only the computer screen and participants’ verbal comments (no faces), and this was compared with notes taken by the research team members. A qualitative researcher coded the field notes through open coding to identify emergent themes from the data and for suggestions on how to improve CHOICE1.0.

Results

A total of 44 participants consented and completed the study. Participants were mostly married, Caucasian/white, educated (completed college or master’s degree/higher), and retired (see Table 2).

Table 2.

Participant Characteristics.a

Characteristic	Number	Percent
Age
66-69	12	27
70-79	25	57
80-84	4	9
Age not listed	2	5
Missing	1	2
Gender
Male	16	36
Female	24	55
Missing	4	9
Marital status
Divorced/separated	7	16
Married/living as married	34	77
Widowed	3	7
Ethnic background
Hispanic/Latino	1	2
Caucasian/white	36	82
Asian/Pacific Islander	6	14
Preferred not to answer	1	2
Education level
College	20	45
Master’s degree or higher	24	55
Employment status
Unemployed	2	5
Retired	33	75
Employed part-time	7	16
Employed full-time	1	2
Missing	1	2
Household income
$US50 000 or less	3	7
$US50 001-$US75 000	8	18
$US75 001 or more	23	52
Preferred not to answer	9	21
Missing	1	2
First thought about choosing a plan
4-6 months ago	4	9
2-3 months ago	10	23
A month ago	25	57
Less than a week ago	4	9
Missing	1	2

a Source: Medicare Part D Comparative Usability Study Questionnaires.

Participants did not differ significantly in the likelihood that they would switch plan after using CHOICE1.0 (67%) compared to after using CMS (60%). There was also no statistically significant difference in plan switching probability across the 3 versions of CHOICE1.0. After using CHOICE1.0, 71% of participants would enroll in an expert recommended plan when compared to after using CMS (49%; P < .05; Table 1). Compared to 86% of participants who would enroll in an expert recommended plan after using version 1 of CHOICE1.0, 87% would do so after using version 2, which was not statistically different from version 1. However, only 38% of those who used version 3 would enroll in an expert recommended plan, which was significantly lower than the probability among those after using version 1 (86%; P < .05). Furthermore, 40% of participants stated an intention to enroll in the lowest cost plan after using CHOICE1.0, compared to 20% after using CMS (P < .25; Table 1). Although the probabilities of enrolling in the lowest cost plan did not differ statistically significantly between versions 1 (57%) and 2 (47%), the probability was significantly lower in version 3 (15%) than in version 1 (57%)(P < .05).

Changes in coverage generosity differed between CMS and CHOICE1.0 as well as between versions of CHOICE1.0. Compared to what their expenditure would have been if they had stayed in their original plan, participants would save $US129 after using the CMS tool compared to an estimated saving of $US672 after using CHOICE1.0 (P < .25; Table 1). Potential savings of $US886 from version 1 also differed from potential savings of $US86 from using version 3 of CHOICE1.0 (P < .25).

Participants were significantly more satisfied with the process of choosing a Medicare Part D prescription drug plan after using CHOICE1.0 (mean = 2.0, standard deviation [SD] = 1.11) when compared to after using CMS (mean = 2.6, SD = 1.03; P < .05; Table 1). Participants reported having greater enjoyment in the process of choosing a Medicare Part D prescription drug plan after using CHOICE1.0 (mean = 2.2, SD = 1.05) when compared to after using CMS (mean = 2.9, SD = 1.05; P < .01; Table 1). Participants did not differ in their opinion on whether they wanted more or less information after using each tool (Table 1). There were also no statistically significant differences in these measures across the 3 versions of CHOICE1.0.

Many participants reported how difficult and confusing CMS was as it just had too much information for them to process. One said that it was “mystifying” and another said “I feel like I should take a college course in how they [all the parts of Medicare Part D] interrelate.” A third participant said “I must be dumb” for not understanding one of the terms (“Original Medicare”) on the CMS Plan Finder.

A participant reported that “the experimental tool [CHOICE1.0] is better for the technologically inept.” Yet he, and a few other participants, preferred all of the details and nuanced information provided by CMS: “I want to see where the numbers come from—I want to see where the costs are coming from.”

Participants also made comments regarding the 3 versions of CHOICE1.0. In terms of the expert recommendation (version 1) and guided analysis (version 2), many participants wanted to know who the “expert” was, as well as more information about the components involved in making the recommendations and how the number was calculated. A few participants in version 1 mentioned that they like how it presented them with just a few recommended options so that they did “not have an endless list of choices.”

Discussion

In comparing CHOICE1.0 and CMS’s Plan Finder, participants had greater satisfaction with the selection process in CHOICE1.0 and found it easy to use. Many user-centered features of CHOICE1.0 improved patient experience, which enabled participants to choose plans more consistent with expert recommendations. A design consultant, reviewing the 2 tools, likened CHOICE1.0 to an Apple product, meaning that the focus was on ease of use, and CMS to a PC product, requiring greater technical capability on the part of the user. CHOICE1.0 resulted in more than $US500 of potential savings. The usability test provided findings similar to a previous behavioral economic experiment (12) that using choice architecture(15) where small modifications in how the information is presented (eg, providing some specific recommendations) can have a significant impact on plan selection and user experience.

A main limitation of this study is the demographics of the participant population (highly educated and high income). The choice of our study population was driven both by the types of patients who chose to participate in the study and the characteristics of the population for which we had potential access to electronic information on prescription drug usage for use in the online tool. This highly educated and relatively technology savvy population potentially has more experience navigating online websites than the average older adult and was well positioned to identify areas in CHOICE1.0 that needed improvement to make CHOICE2.0 even more user-friendly. Future research should examine the attractiveness and effects of expert recommendations and online decision-making tools for more heterogeneous users based on characteristics such as race/ethnicity, gender, education, and income to determine whether expert recommendations have differing impacts on different segments of the population.

The findings from this comparative usability study were used to guide further refinement of the tool CHOICE2.0 for a 3-arm randomized control trial that was completed during the 2017 Open Enrollment period (October 15, 2016, to December 7, 2016). Given the similar enrollment levels and change in coverage generosity between expert recommended and guided analysis, we decided that, while patients appeared to value the expert recommendation, the study did not provide evidence of which version of the recommendation they preferred. Thus, in our trial, we decided to merge the 2 expert recommendation arms, providing both an explicit plan recommendation and the expert scores. The results of the trial will be reported separately.