Beneficiary survey-based feedback on new Medicare informational materials.

In response to the Balanced Budget Act (BBA) of 1997, the Center for Medicare & Medicaid Services (CMS) initiated a massive information and education campaign to promote effective health plan decision-making. Early results suggest that the pilot version of the Medicare & You handbook and other new Medicare informational materials were viewed favorably overall. Despite their limitations, most beneficiaries found the information useful. The longer, more comprehensive materials were not perceived to be more useful than the shorter, less complicated version. Additional research is needed to determine which subgroups of beneficiaries may need more and, possibly less, information.

Introduction and Background

BBA 1997 requires that comparative information be provided to Medicare beneficiaries on an annual basis to inform them about their health insurance options. In response to the BBA, CMS initiated the National Medicare Education Program (NMEP) to enable beneficiaries to make more informed health plan decisions. The objectives of the program are to ensure that beneficiaries have access to accurate and reliable information, and that they are aware of the different health plan choices available to them, understand the consequences of choosing different plans, and are able to use the information provided to them when making decisions. CMS would also like beneficiaries to view the Medicare program and its private sector partners as trusted and credible sources of information (Goldstein, 1999).

In addition to several newly created print materials, the education program includes telephone helplines, an Internet information data base called Medicare Compare, training and support for intermediaries, enhanced beneficiary counseling services, and State and community-based outreach and education efforts. The Medicare & You handbook (formerly the Medicare Handbook) is the primary print medium that CMS developed as part of the NMEP. It contains an overview of the Medicare program and basic benefits, a description of the different plan choices, information on how to get assistance and beneficiary rights and protections, a question and answer section, and definitions of important terms used throughout the handbook. It also contains a section comparing the costs and benefits of the five local Medicare health maintenance organizations (HMOs). The handbook was pilot tested in five States (Arizona, Florida, Ohio, Oregon, and Washington State) and the Kansas City metropolitan statistical area (MSA) in fall 1998. The Kansas City handbook was 52 pages long.

That fall, the remainder of the Nation received an abbreviated version of the handbook in the form of an eight-page tri-fold Medicare & You 1999 bulletin which contained most of the key messages contained in the handbook as well as a description of the different plan options and a tear-out telephone list of reference numbers. None of the information in the bulletin was tailored to different geographic regions, and thus it contained no comparative information.

CMS commissioned RTI to evaluate the 1999 pilot version of the Medicare & You materials in the Kansas City MSA. The study included an outcomes survey and focus groups with Medicare beneficiaries. As part of the study, RTI also evaluated the Consumer Assessment of Health Plans Study (CAHPS®) survey report that provided Medicare beneficiaries in the Kansas City MSA information comparing the quality of care provided by the five local Medicare HMOs. CMS has adopted the CAHPS® survey as its standard for measuring assessments of health plans by Medicare beneficiaries. The CAHPS® survey report provides information about beneficiaries' experiences in different health plans that can be used by other consumers who are considering joining a plan. Kansas City was selected because it was one of the CAHPS® field test sites in which demonstration evaluations were being conducted at the time CMS initiated the Medicare & You 1999 evaluation.

In this article we present primarily results from the outcomes survey, including descriptive statistics on beneficiaries' assessment of the new Medicare materials and logistic regressions of the perceived usefulness of the materials controlling for other factors. After reporting the most common sources of Medicare information used by beneficiaries, we show how much time they spent with the new materials, how helpful beneficiaries felt these materials were, how easy they were to understand, and how they were used by beneficiaries. We discuss the findings in light of concerns about educational, cognitive, and literacy levels of the older adult population.

Data and Methods

Both new (defined as those just aging into the Medicare program at the time of the interview) and experienced (those age 65 or over) beneficiaries residing in the Kansas City MSA were included in the study. Sampling frames were provided by CMS (n = 1,855 new beneficiaries and n = 170,062 experienced beneficiaries). No explicit stratification was performed on the samples, but we sorted the files by age, sex, race, and Zip Code prior to randomly assigning sample members to a control group or one of three treatment groups, to ensure a proportional draw on these characteristics. We attempted to survey all new beneficiaries, but sampled 3,573 experienced beneficiaries, over-sampling those who said they were interested in getting information about the Medicare program so that we would have a sufficient number of responses about the interventions to analyze. However, the sample did include beneficiaries who said they were not interested in receiving such information so that we could generalize the results to all Medicare beneficiaries in Kansas City.

The first treatment group received the Medicare & You bulletin; the second treatment group received the Medicare & You 1999 handbook; and the third treatment group received both the handbook and the Medicare CAHPS® survey report. Treatment group members were interviewed immediately after CMS mailed the Medicare & You materials in fall 1998. Control group beneficiaries received no information as part of the study and were interviewed before the CMS mailing. However, both control and treatment group members could have received information through traditional means not as part of the study. In fact, most control group members reported receiving Medicare-related information outside the study. The samples excluded beneficiaries who were dually eligible for Medicare and Medicaid, whose original reason for entitlement was a disability, had end-stage renal disease, who were institutionalized or receiving hospice care, or who were Medicare qualified government employees. These groups were excluded because we expected them to be less interested in the materials. We further screened out beneficiaries during the data collection process who were away during the study period or had died, those who were physically or mentally incapable of participating or could not read, and those who did not speak English.

We completed computer-assisted telephone interviews with 951 new and 1,156 experienced beneficiaries, distributed about equally across the control and three treatment groups. The survey was conducted between September 1998 and January 1999 and had a mean response rate across the two populations of 60 percent. This response rate was comparable with those obtained in several other studies of the Medicare population conducted around the same time (Carman et al., 1999). Experienced beneficiary respondents were more likely to be younger, male, and white. relative to non-respondents. New beneficiary respondents were more likely to be white than non-respondents, and new beneficiary respondents were more likely to be female and white relative to all 65-year olds in Kansas City.

We conducted descriptive statistical analyses of the data testing differences between subgroups using chi-square statistics (Mantel-Haenszel tests for ordinal data). Unless otherwise noted, differences were tested at the alpha less than 0.05 criterion level. Logistic regression analysis was used to estimate how useful beneficiaries felt the materials were controlling for other factors. This approach was chosen because the dependent variable is binary (somewhat or very useful relative to not very or not at all useful). Predicted probabilities were also calculated for the logit models to estimate the likelihood that different subgroups of beneficiaries, i.e., control versus treatment groups, found the information useful controlling simultaneously for other factors. All estimates were weighted to address non-response and take the complex sampling procedures into account.

Potential limitations of the study are associated with the fact that we asked treatment group members to look at the intervention materials, which may result in overstating some of the results, particularly those related to time spent with the materials.

Results

Descriptive Statistics

Information-Seeking Behavior and Sources of Information

Approximately 90 percent of new and experienced Medicare beneficiaries in the Kansas City MSA feel that choosing a health insurance plan is a very or extremely important decision, and moreover, many feel that it is a difficult decision to make. Specifically, 60 percent of both study populations said the decision is either hard or very hard for them. College educated beneficiaries found the decision easiest. Yet only 18 percent of experienced beneficiaries and 26 percent of new beneficiaries have ever sought out help from another person or organization when choosing a Medicare health plan. The majority of beneficiaries were not aware of State- and federally- funded health insurance counseling services, known collectively as State Health Insurance Assistance Programs, that provide free and unbiased information and counseling about Medicare to beneficiaries and their families. However, about 60 percent of beneficiaries said they would use this type of service if it existed. This supports earlier research indicating that these programs are underutilized, need to perform more outreach, and require additional funding (McCormack, et al., 1996). Recent efforts by CMS to enhance the role of these programs may increase utilization by beneficiaries.

Beneficiaries receive information about their Medicare health plan choices from several different sources (Table 1). Seventy-one percent of experienced and 88 percent of new beneficiaries reported receiving some type of information in the last 6 months about the different types of Medicare health plans. Surprisingly, treatment and control group members were equally likely to report having received information even though no information was sent to controls as part of the study. The most common source for both groups of Medicare-related information was overwhelmingly insurance companies, agents, and/or health plans. Fifty-one percent of experienced beneficiaries and 33 percent of new beneficiaries reported receiving information from this source in the last 6 months. This was also reported to be the most informative source according to both new and experienced beneficiaries. Other common sources included radio and television, direct mail and telephone solicitations, AARP, and the Medicare program. This is consistent with research indicating that television is the most frequently utilized information medium for those age 65 or over (Brown et al., 2000).

Table 1

Sources of Medicare-Related Information Received

Source	Beneficiary
	Experienced	New
	Percent
Any Medicare Information Received During Last 6 Months	71.0	87.6
Medicare-Related Information Received
AARP	7.1	14.2
Church, Synagogue, or Mosque	0.2	0.1
Counseling Agency	0.7	0.7
Doctor or Other Health Care Professional	1.9	2.0
Employer	1.5	3.0
Insurance Company, Agent, or Health Plan	132.6	150.5
Mail or Telephone	314.2	223.2
Medicare Program	4.2	13.2
Radio/Television	225.8	319.8
Social Security Administration	1.4	7.9
State Department of Insurance or Health	1.9	2.2
Other/Don't Know	2.1	2.8

Indicates the most informative source according to beneficiaries.

Indicates the second most informative source according to beneficiaries.

Indicates the third most informative source according to beneficiaries.

NOTE: Includes experienced beneficiaries aged 65 or over and new beneficiaries just aging into the Medicare program.

SOURCE: Survey of new and experienced Medicare beneficiaries in the Kansas City metropolitan statistical area conducted by Research Triangle Institute between September 1998 and January 1999.

About 1 in 10 beneficiaries have ever used the Medicare Web site to obtain information. Use of this information source is likely to grow as the baby-boomer generation ages into Medicare. These data confirm that the intervention materials provided as part of this study were only one of the many places in which beneficiaries could turn for information about Medicare and related health plan options.

Evaluation of the Medicare & You Materials

We asked beneficiaries to look at the intervention materials in order to participate in the study, but gave no guidance on how long they should spend on them. Beneficiaries who received only the handbook spent the most time looking at the information they received (Table 2). About 10 percent of beneficiaries spent less than 15 minutes with the handbook, and almost 30 percent spent more than 1 hour looking at it.

Table 2

Time Spent Looking at the Materials

Time Spent	Beneficiary
	Experienced			New
	Bulletin	Handbook	Handbook and CAHPS®	Bulletin	Handbook	Handbook and CAHPS®
Less than 15 Minutes	27.6	11.0	32.1	34.5	9.8	32.7
15-30 Minutes	50.7	29.8	38.9	47.2	31.2	41.7
15-60 Minutes	12.9	30.5	19.8	14.8	31.5	14.3
More than 60 Minutes	8.8	28.7	9.2	3.5	27.4	11.3

NOTES: CAHPS® is Consumer Assessment of Health Plans Study. Time spent is shown for those who received each intervention. Includes experienced beneficiaries aged 65 or over and new beneficiaries just aging into the Medicare program.

SOURCE: Survey of new and experienced Medicare beneficiaries in the Kansas City metropolitan statistical area conducted by Research Triangle Institute between September 1998 and January 1999.

The Medicare & You materials have been helpful to most beneficiaries in understanding their health plan choices, with about 80 percent of beneficiaries rating them as good, very good, or excellent at helping them understand the advantages and disadvantages of the different Medicare options. Perceived helpfulness varied significantly with beneficiaries' level of education, with higher educated beneficiaries finding the materials more helpful (Table 3). For example, 46 percent of college-educated experienced beneficiary respondents compared with 27 percent of those with less than a high school education rating the materials as very good or excellent.

Table 3

Helpfulness of the Materials in Understanding Advantages and Disadvantages of Different Medicare Health Insurance Options

Education	Helpfulness
	Poor/Fair	Good	Very Good/Excellent
	Percent
Less than 12 Years	16.9	56.6	26.5
High School Graduate	19.0	44.2	36.8
Some College/Technical School	14.3	42.8	43.0
College Graduate	7.5	46.9	45.7

Includes only experienced beneficiaries 65 years of age or over.

NOTE: Differences between educational groups are significant at the 0.05 level.

SOURCE: Survey of new and experienced Medicare beneficiaries in the Kansas City metropolitan statistical area conducted by Research Triangle Institute between September 1998 and January 1999.

About 4 in 10 beneficiaries found the Medicare & You materials easier to understand relative to information they have received in the past, nearly one-half found them to be about the same level of difficulty as other materials, and 5 percent found them more difficult. Higher educated beneficiaries found the materials easier to understand, but the differences by education level were only significant for the experienced beneficiary group.

Of those who received the Medicare & You materials, nearly one-half of beneficiaries said they learned something new, with treatment group members being slightly more likely to have learned something (p = 0.09 for experienced beneficiaries). The following notes from telephone interviewers reflect some of the more commonly mentioned issues beneficiaries said they learned about:

Where to go to learn about additional benefits.

About supplements, choices, savings account, etc.

He is pleased with what he has and doesn't want to change.

Made the right choice in choosing a new HMO.

If I don't like it [plan] I can go back to the old plan.

About 40 percent of beneficiaries said they have used or will use the information in the materials to help them make a decision about their Medicare health plan. For those who had made a decision, most had used it to confirm a health plan choice they had already made. Treatment group members were more likely than control group members to use the materials they received to confirm a health plan choice that they had already made (McCormack et al., 2001). When asked whether they preferred to receive the Medicare & You information annually without requesting it or to return a postcard requesting the information on an as-needed basis, more wanted to receive it automatically, particularly new beneficiaries.

Evaluation of the Medicare CAHPS® Survey Report

Beneficiaries spent less time with the handbook and CAHPS® survey report combination relative to the Medicare & You handbook alone, with new beneficiaries spending slightly less time than experienced beneficiaries (Table 2). Three-quarters of respondents found the CAHPS® survey report very or somewhat easy to understand, while just over 10 percent found it very or somewhat hard, and the remainder saying it was neither hard nor easy. Although the data suggest that higher education was associated with increased comprehension of the CAHPS® survey report (not shown), the differences by education were not statistically significant.

Self-reported understanding of how to read the bar and star charts was relatively high in that over 80 percent of respondents indicated that they could tell which health plans were rated the best from reading the charts. However, this result may be somewhat overestimated due to social desirability bias, in which respondents give the answer they believe an interviewer wants to hear. About 95 percent of beneficiaries correctly reported that the information in the survey report told them at least a little about how quality of care differs across health plans. Just over 90 percent of survey respondents reported that the plans were rated on performance measures they care about. There was no strong preference for presenting the information using the star chart versus the bar graphs in either population. Nearly all beneficiaries said they kept a copy of the CAHPS® survey report to share with others or refer to later.

Multivariate Analysis

Perceived Usefulness of the Materials

We asked control and treatment group members to rate the usefulness of the informational materials they received about the Medicare program during the last 6 months. Because most control and treatment group members received at least some information, we were able to make comparisons between all of the groups. However, because not all beneficiaries received information, we restricted our analysis to those who did receive the materials. This included 81 percent of new and 63 percent of experienced beneficiary respondents. Substantive findings did not differ when the models were run with and without those who did not receive any information during the last 6 months.

Using logistic regression analysis, we found that beneficiaries in all three treatment groups were significantly more likely to find the information they received useful compared with control group members who only received information outside the study (Table 4). This finding was further supported by predicted probabilities which showed that 61 percent of experienced beneficiary control group members found the materials useful in contrast to between 71 and 75 percent of treatment group members (Table 5). The pattern was similar for new beneficiaries, except a higher proportion of control group members (72 percent) found the information they received useful.

Table 4

Logistic Regression Results Predicting the Probability of Finding the Materials Useful

Independent Variable	Beneficiaries
	Experienced1 beta(se)(n = 732)	New2 beta(se)(n = 779)
Intercept	−1.8856(1.1580)	0.6047(1.4898)
Study Group
Bulletin	***0.7485(0.2498)	***0.8689 (0.2825)
Handbook	***0.5392(0.2517)	***0.7775 (0.2759)
Handbook and CAHPS®	***0.6765(0.2470)	*0.5099 (0.2675)
Age
65-74 Years	0.0524(0.0322)
75 Years or Over	**−0.0722(0.0332)	NA
Sex
Male	NS	*0.4009 (0.2138)
Income
More than $30,000 a Year	**−0.7640 (0.3617)	NS
Supplemental Insurance
Individually-Purchased	NS	***0.6822 (0.2230)
Spouse's Choice
Affects Respondent's Choice	**0.3987(0.2011)	*0.3955 (0.2270)
Physicians Visits
3 or More Physicians Visits in the Last 3 Months	* −0.4366(0.2606)	NS
Regular Source of Care
Have a Usual Source of Care	NS	*−0.7729(0.4401)
Exposure to Other Information Sources (Range=0-11)	** 0.2223(0.1033)	***0.2800 (0.1142)
Exposure to Quality of Care Information	* 0.5206(0.3125)	NA
15-ltem Knowledge Index	***1.9875(0.5785)	***3.4982 (0.5828)
Negative Attitude about HMOs	**−0.4475(0.1844	**−0.5367 (0.2134)

p <0.01.

0.01 p <0.05.

0.05 p <0.10.

n=732.

n=779.

NOTES: CAHPS® is Consumer Assessment of Health Plans Study. NA is not applicable. NS is not significant. HMOs is health maintenance organizations. The regression models also included variables reflecting beneficiary race, ethnicity, education, whether the beneficiary lives alone, was hospitalized in the last year, and health status as measured by the Standard Form-12©. Includes experienced beneficiaries aged 65 and over and new beneficiaries just aging into the Medicare program.

SOURCE: Survey of new and experienced Medicare beneficiaries in the Kansas City metropolitan statistical area conducted by Research Triangle Institute between September 1998 and January 1999.

Table 5

Predicted Probability of Finding the Intervention Materials Useful, by Study Group

Study Group	Beneficiary
	Experienced	New
Control	0.605	0.716
Bulletin	0.751	0.842
Handbook	0.714	0.831
Handbook and CAHPS®	0.739	0.796

NOTES: CAHPS® is Consumer Assessment of Health Plans Study. Includes experienced beneficiaries aged 65 or over and new beneficiaries just aging into the Medicare program.

SOURCE: Survey of new and experienced Medicare beneficiaries in the Kansas City metropolitan statistical area conducted by Research Triangle Institute between September 1998 and January 1999.

Among experienced beneficiaries, those who were age 75 or over, had higher incomes, and had 3 or more physician visits in the last 3 months all found the information less useful (Table 4). Those who reported being exposed to quality of care plan performance information were more likely to find the materials useful. Among new beneficiaries, male respondents and those with individually-purchased supplemental insurance, i.e., medigap, were more likely to find the materials useful than those with a regular source of medical care who found the materials less useful. Education level was not a significant variable in either the new or experienced beneficiary models.

Four variables were significant across the new and experienced beneficiary logit models. The greater a beneficiaries' knowledge of the Medicare program was (as measured by a 15-item knowledge index developed previously as part of the study [McCormack et al., forthcoming 2002]), the more likely they were to find the materials useful. Beneficiary knowledge was the factor that had the greatest effect on the probability of finding the materials useful as indicated by the odds ratio (not shown). For a 1-percent increase in beneficiary knowledge, the weighted odds of finding the materials useful was 6 percent for experienced beneficiaries and 32 percent for new beneficiaries.

Similarly, the more exposure beneficiaries had to other sources of information, the more useful they found the pilot materials. Surprisingly, those who reported that their spouse's choice of insurance affected their choice were more likely to find the materials useful. Those who reported having a negative attitude about HMOs found the materials less useful, controlling for other factors.

Discussion

These early results suggest that the NMEP has moved toward its goal of providing beneficiaries with access to information. Overall, the new Medicare materials were viewed favorably by beneficiaries. However, there is room for improvement as nearly one-half of beneficiaries found the Medicare & You materials as difficult to understand as information they have received in the past. It is important to keep in mind, however, that the 1999 pilot version of the handbook evaluated in this study has since been revised, and is continuously being reviewed and updated by CMS. Medicare & You 2000 was mailed nationally for the first time in fall 1999. It was tailored to 26 different geographic regions of the country and provided local-level quality of care information for competing health plans.

Despite the limitations of the materials, beneficiaries found them useful. It is interesting to note, however, that the magnitude of the effect of the different materials—the bulletin, handbook, and the handbook and CAHPS® survey report combination—varied very little. The statistical interpretation of this finding is that the longer and more detailed handbook and the handbook/CAHPS® survey report combination was not viewed as being more useful than the shorter bulletin. Given that a great majority of beneficiaries reported receiving Medicare information during the last 6 months, this could imply that beneficiaries are simply being saturated with information, and that more information has been not necessarily better. A possible policy implication of these findings is that some beneficiaries may be satisfied with only the amount of information in the bulletin, while others value the handbook, particularly as a reference tool as indicated by focus group research (Harris-Kojetin et al., 2001). Additional research is needed to determine which subgroups of beneficiaries may need more and, possibly less, information.

About one-third of the current Medicare population has less than a high school education, and another one-third has a high school diploma but no college according to the 1998 Medicare Current Beneficiary Survey. Although level of education was not directly associated with perceived usefulness according to the multivariate analysis, beneficiaries' knowledge did affect perceived usefulness. Earlier research has shown that level of education is an important predictor of beneficiary knowledge, therefore knowledge might be serving as an intervening variable between education and usefulness.

Beneficiary literacy and cognition are critical issues that should be taken into consideration when developing informational materials for an older adult population. Problems associated with limited literacy have been found to be the greatest in the area of document literacy, which includes filling out forms, reading and following directions, and using schedules which are skills needed to navigate the new Medicare materials. Hibbard and colleagues (2001) in a recent study found that over one-half of the Medicare population has difficulty using comparative informational materials, including interpreting charts and tables. Thus, the materials should be reviewed with these limitations in mind. As is suggested by other research, transmission strategies beside print materials are probably needed. These avenues may include television, newspapers, and radio, which are popular among older adults, as well as the use of counselors or other intermediaries that CMS is already pursuing.

We still do not know whether beneficiaries have sufficient understanding and adequate decisionmaking skills to make informed choices. This is a major concern as beneficiaries are asked to make complex choices among an expanded array of options and are given a greater volume of information to support those choices. Improving decision support systems for beneficiaries is an enormous challenge and better informational materials are one step in that direction.