Medicare beneficiary knowledge: measurement implications from a qualitative study.

Medicare beneficiary knowledge about fee-for-service (FFS) Medicare versus managed care alternatives (MCA) has been studied extensively. However, these efforts might be compromised by lack of familiarity with common Medicare terminology. We used qualitative methods to examine beneficiaries' familiarity with Medicare Programs (FFS and MCA) and terminology. Twenty-one indepth, semi-structured beneficiary interview transcripts were analyzed through iterative review. Across sex, race/ethnicity, and benefits programs, participants found interview questions with Medicare terminology difficult to answer, potentially causing missing, incorrect, and inaccurate responses to interview questions. Assessment of beneficiary knowledge may be fundamentally impacted by absence of basic familiarity with Medicare Programs terminology.

Introduction

Many studies examining Medicare utilization barriers have studied beneficiary decisions about their Medicare benefits, including factors influencing benefit program or plan enrollment and disenrollment (Harris-Kojetin et al., 2002; Lied et al., 2003; Riley, Ingber, and Tudor, 1997; Rossiter et al., 1989), effective benefit program information (Edgman-Levitan and Cleary, 1996; Harris-Kojetin et al., 2001; Hibbard et al., 2001; McCormack et al., 2001a; 2001b), and information influence on specific health coverage decisions (Bann, Berkman, and Kuo, 2004; Farley et al., 2002; Gazmararian et al., 1999; LaTour, Friedman, and Hughes, 1986; Sofaer et al., 2001). Beneficiary knowledge has been assessed extensively, usually with mail or telephone surveys (Bann, Berkman, and Kuo, 2004; Cafferata, 1984; Fyock et al., 2001; Hibbard et al., 1998; Levesque et al., 2001; McCormack et al., 2001c; McCormack et al., 2002; McCormack and Uhrig, 2003; Sing and Stevens, 2005). These assessments have focused primarily on beneficiary understanding of traditional FFS versus MCA, such as Medicare Advantage and its predecessor Medicare+Choice (M+C), and they will likely be expanded to address the Medicare prescription drug plans that were implemented in January 2006. Knowledge assessment items typically evaluate perceived knowledge (what beneficiaries think they know about Medicare) or actual knowledge (factual questions for which there are correct answers). The Medicare Current Beneficiary Survey (MCBS), for example, has used both types of knowledge items in its annual assessment of beneficiaries. (Bann and Berkman, 2002; Bann et al., 2003). Both types of items rely on basic Medicare terminology to cue beneficiaries.

This study was motivated by results from our pilot examination of a draft survey for a national mailed survey of Medicare beneficiaries (Morgan, 2006). The national mailed survey targeted random samples of elderly, male and female, White, Black, and Hispanic Medicare beneficiaries in both metropolitan and non-metropolitan areas. Beneficiaries enrolled in traditional FFS and MCA plans were equally sampled. The broad purposes of the survey were to identify factors affecting enrollment in MCA plans, how enrollment subsequently affected both real and perceived access to care, and how enrollment decisions and subsequent access varied among the three race/ethnic groups sampled. To pilot test our survey, we conducted three focus groups of 27 White, Black, and Hispanic beneficiaries from community centers and a retirement home. Participants completed the draft survey prior to the focus group discussion. During the focus group, the moderator assessed items that were left blank or were difficult to answer, with probes regarding sources of non-response (such as unfamiliar words, words with multiple meanings, and the appropriateness of available responses). These focus groups suggested that participants did not understand questions that relied on Medicare terminology common to most efforts to measure Medicare knowledge. Based on the focus group observations, we developed an additional qualitative pilot study using indepth semi-structured interviews (Carbone, Campbell, and Honess-Morreale, 2002; Collins, 2003; Drennan, 2003; Jobe and Mingay, 1990) to examine the nature of and reasons for poor comprehension of Medicare benefits. This article presents the methodology and findings from those semi-structured interviews.

Methods

Recruitment and Participation

Recruitment for interview participants was coordinated through community centers and health coalitions. An Institutional Review Board (IRB)-approved information sheet was distributed, and interested participants contacted the research staff by telephone. In an attempt to have as broad a representation as the sample in our national survey, callers were purposefully screened for eligibility and selected for inclusion in the pilot study based on their sex, race/ethnicity, and Medicare benefits program. Eligible participants were interviewed at a location convenient to the participant. In accordance with IRB approval, written consent was obtained from each participant before the interview began. Each participant received $20 on the completion of the interview.

Interviews were conducted with 21 participants (Table 1) with an average age of 77.3 (SD = 6.4). Of these 11 (52.3 percent) were female, 7 (33.3 percent) were Black, 6 (28.6 percent) were Hispanic, and 8 (38.1 percent) were White. Eight (38.1 percent) had achieved a college degree, and 11 (52.4 percent) reported annual incomes of $20,000 or less. For our analyses, we combined 8 beneficiaries who were currently utilizing a Medicare MCA with 2 others who had previously used MCA for 10 (47.6 percent) total beneficiaries experienced with MCA. The remainder (11, 52.4 percent) were currently utilizing traditional FFS. When asked “How would you rate your health during the last four weeks?” 71.5 percent (15) of the participants reported being in good, very good, or excellent health. There were no differences in health ratings by race/ethnicity or Medicare benefits program, though females reported being in better health than males (t (19) = -2.67, p = 0.015). As expected, those experienced with MCA reported being more familiar with Medicare managed care than those in FFS (t (19) = -5.584, p < 0.001). There were no differences in ratings of familiarity with Medicare managed care by race/ethnicity or sex. There were no differences in ratings of familiarity with FFS Medicare by Medicare benefit program, race/ethnicity, or sex.

Table 1

Number of Medicare Beneficiaries Interviewed, by Sex, Benefit Program, and Race/Ethnicity

Sex	Benefit Program	Race/Ethnicity			Total
		White	African American	Hispanic
Female	FFS	1	1	2	4
	MCA	3	3	1	7
Total		4	4	3	11
Male	FFS	2	3	2	7
	MCA	2	0	1	3
Total		4	3	3	10
Race/Ethnicity Total		8	7	6	21

NOTES: FFS is fee-for-service. MCA is managed care alternative.

SOURCES: Teal, C.R., Murphy, C.L., Morgan, R.O., Baylor College of Medicine and the Michael E. Debakey VA Medical Center, Houston; John, D.A., University of Washington; and Paterniti, D.A., University of California, Davis.

Description of Interviews

The 60 to 90 minute interviews were conducted by three trained staff in English using a facilitator's guide to ensure consistency. The interviews were designed to assess sources of knowledge and confusion in the beneficiaries' understandings of their Medicare benefits. What did they know, and how had they come to learn it? What didn't they know? How did their knowledge or confusion relate to their difficulty answering questions about their benefits and benefit choices? What factors influenced understanding of common Medicare terminology?

Each interview began with 14 close-ended items selected from the draft survey. These questions, asked verbally by the interviewer, assessed health status, delay in receiving health care, types of health care coverage, satisfaction with Medicare benefits, familiarity with MCA and FFS Medicare, and comparative perceptions of the value of MCA and FFS programs. (At the time these interviews were conducted, CMS, MCA plans, and prescription drug plans had not yet begun disseminating information about the Medicare prescription drug program. Thus, in this pilot study, we did not assess knowledge specific to the prescription drug program.) For the six questions utilizing specific Medicare terminology, participants were asked about the information that drove their decisions and responses. In the second part of the interview, participants were asked to provide definitions of basic Medicare terms (e.g., health care plan, original Medicare, supplemental insurance, Medicare health maintenance organization (HMO), M+C, and Medicare Advantage). In the final section of the interview, participants also addressed questions about their health care coverage. These included aspects of their coverage that they found confusing, decisions they had made regarding their coverage, factors that influenced those decisions, how they came to be informed about their coverage, and what kinds of information they found most helpful.

Five interviews were conducted in which the responses to close-ended questions and requests for definitions were not audio-recorded. These interview sections were expected to be straightforward and intended to inform the interviewer's understanding of the open-ended questions in the latter part of the interview. The interviewers recorded their observations, but not the verbatim responses of the interviewees. A central theme of these observations quickly became evident: Participants were very unclear about specific terminology used to reference benefit programs, even when their conceptual understanding of the benefit programs appeared somewhat strong. Consequently, we decided to audio-tape record these two closed-ended interview sections and include these responses in the transcripts of the final 16 interviews.

Data Analysis

Interviewer observations were recorded for each interview. A transcript was prepared for each of the 21 audio-taped interviews. Data from each section of the interview was analyzed. Five members of the research team, including a medical sociologist and two others experienced in qualitative analysis, conducted independent reviews of the transcripts. Two team members independently coded (1) the definitions of Medicare terms for participant recognition and understanding of the term, and (2) assessments of whether participants could conceptually distinguish between FFS and MCA programs, or between MCA benefits and supplemental insurance. Coding categories included yes (to indicate the participant could recognize and define the term, or conceptually distinguish between programs/plans), no, and could not be evaluated. Inter-rater reliability was assessed for these codes. (Cohen, 1960). An iterative group review process was utilized to develop thematic codes from the remaining interview text. Dealing directly with transcript content (direct coding), team members reviewed and coded each transcript individually; the team then met to systematically review coded transcripts. Team members agreed on positive and negative cases and example text segments for each code, resolving disagreements through extensive discussion, further exemplification, and consensus (Corbin and Strauss, 1998).

Results

Most participants were unable to distinguish between FFS and MCA program terminology, or between the terms for MCA and various forms of supplemental insurance. For example, when one beneficiary was asked about his familiarity with the term M+C plan, the beneficiary replied, “I'm not familiar with that. So I imagine its Medicare plus any other plan, a supplement or closes in the gaps.” Other samples of participant statements that illustrate familiarity with Medicare terminology are available from the authors. As Table 2 demonstrates, there were multiple forms of evidence for this finding, and more generally, for the inability to recognize or understand terms across each of the sections of interview questions.

Table 2

Interview Sections and Associated Types of Evidence for Results

Interview Section	Evidence
Close-ended items from draft survey	Inability to select health coverage Inability to meaningfully compare fee-for-service (FFS) and managed care alternatives (MCA) benefits programs Ratings of familiarity substantiated by open-ended questions
Probes for information used in selecting responses to close-ended items	Limited understanding of FFS or MCA benefit terms Reliance on familiar wording as cues Wording was familiar from their experiences receiving care Preference for current benefit program enrollment
Definitions of common Medicare terms	Unclear or inaccurate descriptions Attempts to induce meaning from term Attempts to clarify while answering Acknowledgment that the term was unfamiliar when directly asked
Open-ended questions about the interviewee's health care coverage	Little participant-initiated use of terminology Focus on specifics of the process by which they received care For MCA, focus on specific plan

SOURCES: Teal, C.R., Murphy, C.L., Morgan, R.O., Baylor College of Medicine and the Michael E. Debakey VA Medical Center, Houston; John, D.A., University of Washington; and Paterniti, D.A., University of California, Davis.

Confusion was first indicated by an inability to correctly choose their types of health coverage from a list of possible options provided during the close-ended portion of the interview. When asked to describe their own health coverage, most interviewees focused on where they received care, who provided it, or the process by which the bills were paid. An unedited quote illustrates this. When asked about original Medicare, one participant indicated (correctly), “And uh, you pay the doctor, I don't know how much, but you pay the doctor and Medicare will absorb some of the payment and you have to pay a certain percentage.” Probing questions regarding benefits or insurance revealed little understanding of either FFS or MCA program terminology. Interviewees seldom used terms such as original Medicare, Medicare Advantage, or Medicare HMO to indicate plan types.

In general, participants had difficulty answering questions that relied on common but program-specific Medicare terminology. This was evident in beneficiaries' inability to accurately define several common Medicare terms. As Table 3 demonstrates, most participants were unfamiliar with the terms original Medicare, Medicare HMO, Medigap, and Medicare Advantage. Less specific terms such as health care plan and supplemental insurance were more familiar, though interviewees sometimes induced their meanings from the words themselves. This was sometimes helpful, as in the case of the term supplemental insurance, in which the induced meaning was usually correct. Attempts to induce other terms, such as retiree health coverage and M+C led to incorrect assumptions. For example, when asked about retiree health coverage, one participant replied: “I would say that's Medicare.” Another described M+C in this way—“It gives you uh, choice of getting your insurance that you want.” Many examples of this inability to recognize or define terms were available among our transcripts. One detailed, unedited quote is offered here as an exemplar of the prototypical beneficiary's response. Some knowledge of Medicare Programs is evident; however, the beneficiary could not link his correct conceptual knowledge to individual program terms. Such inability has considerable implications for how this beneficiary's knowledge could be assessed with our current measurement methods.

Table 3

Interview Outcomes and Inter-Rater Reliability for Coding of Participant Responses (n = 21)

Term Recognition and Definition	Number of Participants Who Could Recognize and Define Term (%)	Number of Participants Who Could Not Recognize and Define Term (%)	Number of Participants With Responses That Could Not Be Evaluated (%)	Inter-Rater Reliability(Cohen's Kappa)
					Health Care Plan	18 (85.7)	2 (9.5)	1 (4.8)	0.71
					Original Medicare	9 (42.9)	10 (47.6)	2 (9.5)	0.72
					Fee-for-Service (FFS) Medicare	1 (4.8)	15 (71.4)	5 (23.8)	0.77
Supplemental Insurance	15 (71.4)	4 (19.1)	2 (9.5)	0.89
Retiree Health Coverage	2 (9.5)	15 (71.4)	4 (19.1)	0.72
Medigap Supplemental Insurance	5 (23.8)	12 (57.1)	4 (19.1)	0.92
HMO	9 (42.9)	11 (52.4)	1 (4.8)	0.82
Medicare HMO	10 (47.6)	11 (52.4)	0 (0)	0.72
Medicare + Choice	3 (14.3)	17 (81.0)	1 (4.8)	1.00
Medicare Advantage	0 (0)	17 (81.0)	4 (19.1)	1.00
Medicaid	7 (33.3)	11 (52.4)	3 (14.3)	0.84
Conceptual Understanding of Benefit Programs
Distinguish between FFS and MCA programs	15 (71.4)	5 (23.8)	1 (4.8)	0.77
Distinguish between MCA and supplemental insurance	9 (42.9)	9 (42.9)	3 (14.3)	0.84
Understand whether their specific health plan was an MCA plan or a supplemental insurance plan1	10 (71.4)	4 (28.6)	NA	1.00

Applicable to only those with either an MCA or supplemental plan (n=14).

NOTES: MCA is managed care alternatives. HMO is health maintenance organization. NA is not applicable.

SOURCES: Teal, C.R., Murphy C.L., Morgan, R.O., Baylor College of Medicine and the Michael E. Debakey VA Medical Center, Houston; John, D.A., University of Washington; and Paterniti, D.A., University of California, Davis.

[Interviewer: So you think Fee for Service Medicare is something like the HMO?] Yes. [What about supplemental insurance?] I would think that that is the insurance you would need to supplement your Medicare cost. [What about retiree healthcare coverage?] I would think that would be your Medicare. [You mean the same as the Original Medicare?] The Original Medicare. [Do you think that's the same as some people refer to as the employer or union coverage?] No, no, I think that would be government coverage. [You think it's different?] Yes. [Okay, so retiree health coverage is more like Medicare and employer, union coverage is more like the—how would you explain it?] I would think that it would be something that you would pay over the years with your company or your business. [What about Medigap supplemental insurance?] Well, that's who—your supplemental insurance would cover. The gap—the payment that Medicare actually furnishes and what the actual doctor or hospital bill is. […What about HMO?] HMO is who I would think is like a supplemental insurance. You know like a stock gap. With your social security benefits. […so if I understood you right, you described it as a form of supplemental insurance? Something you get with your social security benefit?] Right. [Okay, what about Medicare HMO?] “That would be about the same thing that would be the supplemental insurance. [And you see Medicare HMO as a type of supplemental insurance?] Right. [Do you see them the same thing or do you see them as two different supplemental insurances?] I would see them as same thing. [Okay, what about a Medicare+Choice health plan?] What I see the choice I got is a choice supplemental insurance. [So would you put it in the same plan—is it the same as a Medicare HMO or a Medigap supplemental insurance….] …that's exactly what I would say. [To you they are all the same thing.] They are all the same thing as far as what they are supposed to achieve.

The confusion and lack of familiarity with program-specific Medicare terminology affected question comprehension and responses. Six questions referenced a specific Medicare benefits program type (e.g., FFS, original Medicare, Medicare Advantage, Medicare HMO), and probing questions explored information used by the interviewees to select their responses. The responses to these questions often indicated that the selected responses were inaccurate, or that the participant was indicating a preference for their current Medicare benefit program rather than an informed understanding of the differences in the two programs. For example, one question asked the beneficiary to compare MCA and FFS and included the option “I do not have enough information.” Despite this option, many beneficiaries still endorsed their current plan, even if they admitted not having enough information to compare. This is illustrated in this unedited exchange between interviewer and one participant.

[Interviewer: In your opinion which option is more likely to provide care when you need it? You can choose from these options. Original Medicare, Medicare+Choice, there's no difference, I do not have enough information.] I do not have enough information. [Okay, which provides more choices of physicians?] Uhm, I do not have enough information for that either. [Okay, which offers more generous benefits?] The same, I don't uh, I can't compare. [And which offers higher quality of care?] I think mine does. [Okay and why do you think it does?] Because when you go to see the doctor, right away you don't have to form a line outside to see your doctor. She's very good, the doctor that I have.

Finally, unfamiliar terms were typically clarified by participants before answering the question. These findings suggested that beneficiaries' responses were partly driven by a desire to appear knowledgeable, even when answers to probing questions demonstrated otherwise.

Despite this confusion over terminology, many participants did indicate conceptual understanding of benefit program and plan differences. As shown in Table 3, only 31 percent could not describe conceptual differences in the benefit delivery mechanisms. One-half (50 percent) could distinguish conceptually between MCA and supplemental plans. This proved helpful to the interviewer. Once an interviewer ascertained that the interviewee did, in fact, understand the meaning of the term, she routinely utilized the language spoken by the interviewee (e.g., plan name in addition to Medicare managed care plan) to reference that term. This was a successful strategy for cuing the respondent to specifically what we were asking about.

These findings were consistent with our previous focus group findings. Sex, race/ethnicity, or Medicare benefit program did not appear to have an impact on the pervasive confusion and lack of familiarity with common Medicare Program terms.

Discussion

These findings illustrate the difficulties experienced by Medicare beneficiaries in understanding what is often their primary source of health care coverage. One frustrated beneficiary stated, “I know they use these [terms] to differentiate, but how do they expect a regular person to know what that means?” Another summarizes, “… there is a lot of terminology you would not encounter [everyday]…” Beneficiaries often attempted to induce a term's meaning and respond to questions, with mixed results. Even when allowed the option to indicate that they do not have enough information, participants responded definitively, and often, inaccurately. Our research suggests that Medicare beneficiaries likely understand their coverage through actual participation in the health care process, not through Medicare Program particulars or even its most basic terminology. These findings are consistent with previous studies demonstrating poor beneficiary knowledge (Bann et al., 2003; Cafferata, 1984; Edgman-Levitan and Cleary, 1996; Frederick Schnieders Research, 1995; Fyock et al., 2001; Hibbard et al., 1998; LaTour, Friedman, and Hughes, 1986; McCall, Rice, and Sangl, 1986).

However, our findings also raise critical questions about current methods of beneficiary assessment. Basic measurement principles indicate that question wording should be simple, familiar and not have multiple meanings, and that socially desirable responses should be minimized (Converse and Presser, 1986; Fowler, 1995; Sudman and Bradburn, 1982; Tourangeau, Rips, and Rasinski, 2000). Our results suggest that using Medicare benefit program terms in knowledge assessments introduces unfamiliarity, and may generate measurement error related to respondents' attempts to induce meaning in order to appear knowledgeable. Surveys of beneficiary knowledge routinely rely on terminology with which beneficiaries lack basic familiarity. Studies of Medicare plan preferences or decisions regarding program enrollment necessarily rely on a similar understanding of Medicare terms, as will assessments of beneficiary knowledge regarding the Medicare prescription drug program. If beneficiaries are not familiar with these terms, assessments of these kinds may be fundamentally flawed. In our pilot study, this was illustrated by the discrepancies in Table 3 between the numbers of participants who could recognize or correctly define specific terms and the higher numbers of participants who had conceptually correct knowledge unconnected to specific terms. Future research should focus on testing terminology that can be routinely understood by beneficiaries. Further, researchers should explore beneficiaries' experiences of receiving care through the various Medicare benefit programs, with a goal of identifying common terms or phrases to cue the beneficiary about which plan is being referenced.

In addition, possible strategies emerge from our interviews that are consistent with existing general measurement principles that could be applied in specific beneficiary assessment situations. Many of these solutions have been examined as methods for educating beneficiaries about or marketing various Medicare Programs (AARP, 2005; Fyock et al., 2001) and could be adapted beneficiary assessment. For example, the use of vignettes could be helpful for examining decisionmaking about specific benefit programs (Converse and Presser, 1986). Because beneficiaries rely on their experiences with health care to understand their Medicare benefits, such vignettes could be effective if they were based on common depictions of elderly health needs and experiences with care, incorporated variables such as physician selection and payment mechanisms, and examined the tradeoffs of selecting one benefit program over another. Another suggestion for improving beneficiary knowledge assessment is using random open-ended followups to closed-ended knowledge questions (Converse and Presser, 1986). This strategy worked well in our face-to-face interviews, particularly for providing information regarding the quality of responses that were being received. The use of respondent wording in telephone surveys (rather than or in addition to the survey's specific term) could enhance beneficiary assessments by providing accurate cuing (Sudman and Bradburn, 1982). One example of this is using specific MCA plan names instead of or in addition to terms such as Medicare Advantage, similar to the MCBS (Centers for Medicare & Medicaid Services, 2006), although using specific plan terminology can change the nature of the question being asked. We utilized a variant of this method in our national mail survey examining benefit enrollment and disenrollment patterns among beneficiaries. We utilized the wording offered by interview participants to craft correct definitions of FFS and MCA benefit programs, which emphasized critical elements of the health care process that differentiated the benefit programs. Finally, the use of self-administered forms rather than telephone or interviewer based forms (Fowler, 1995) could prevent beneficiaries from feeling compelled to answer questions in which they do not know all the terms or the correct answer. These forms would, of course, be improved through careful pilot testing of question wording and response options, as well as providing respondents with the option to indicate when they do not know how to answer.

Limitations

Our participants purposefully included members of three different race/ethnic groups, both males and females, and beneficiaries in both the MCA and FFS benefits programs. However, this qualitative pilot study reflects the understanding of only a small number of English-speaking beneficiaries in a southern urban region with specific Medicare Program benefits availability and plans. As such, the results may not be generalizable to national Medicare population. Further, these results do not address the assessment of knowledge among those who are dually eligible for Medicare and Medicaid. We were successful in the interviews in using some of the suggested solutions to bridge the gap between participants' terminology and conceptual understanding of the benefit programs, and were able to apply some of these findings to our national survey. Future research is necessary to formally test the efficacy of these solutions, as well as to examine others that lend themselves to other modes of data collection, such as mail or telephone surveys.

Conclusions

Although our data were collected prior to the advent of the Medicare prescription drug benefit program, virtually all of the terms we examined will remain in common use in the coming years. If the field currently has difficulty assessing beneficiary knowledge without heavily relying on little understood jargon, we will only compound the problem as the Medicare Program changes and becomes more complex. The implementation of the Medicare prescription drug benefit program offers an excellent opportunity for qualitative and psychometric Medicare researchers to explore new methods for beneficiary assessment and how the experiences of care could be utilized to improve assessment. Such work could help the Medicare Program itself simplify the program's terminology so that it is more consistent with the ways beneficiaries understand their health care and insurance experiences.