Psychometric properties of an instrument to assess Medicare beneficiaries' prescription drug plan experiences.

Using data from 335,249 Medicare beneficiaries who responded to the 2007 Medicare Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey, along with data from 22 cognitive interviews, we investigated the reliability and validity of an instrument designed to assess beneficiaries' experiences with their prescription drug plans. Composite measures derived from the instrument had acceptable internal consistency and sufficient plan-level reliability to inform consumer choice, quality improvement, and payor oversight. These measures were positively associated with members' overall rating of the plan and their willingness to recommend the plan. Moreover, each was independently useful in predicting beneficiaries' global ratings of their plan. This instrument can be an important tool for helping beneficiaries to choose a plan that best meets their needs.

Introduction

The CAHPS® surveys, which ask health care consumers about their experiences with and evaluations of health care they have received, are among the most widely used measures of the quality of health care in the U.S. Results from these surveys are reported publicly to provide consumers with comparative data on health care quality and to provide health care professionals and plans with feedback for setting quality improvement goals. Since 1997, CMS has used the Medicare CAHPS® survey to evaluate the quality of care and services provided by Medicare Advantage (MA, the managed care form of Medicare) health plans. Beneficiaries who obtain care through original fee-for-service (FFS) Medicare have been surveyed since 2000 (Goldstein et al., 2001). Each year, a national probability sample of Medicare beneficiaries is asked to complete a CAHPS® Medicare survey.

Medicare Part D, the Federal program to provide prescription drug coverage for Medicare beneficiaries, was introduced in the Medicare Prescription Drug, Improvement, and Modernization Act of 2003 and implemented in January 2006. The initial Part D enrollment period was from November 15, 2005 to May 15, 2006. Unlike coverage in Medicare Parts A and B, Part D coverage is not provided within the traditional Medicare Program. Instead, beneficiaries must enroll in one of many Part D plans offered by private companies either as a MA plan that covers both medical services and prescription drugs (MA-PD) or a stand-alone (also called “free-standing”) prescription drug plan that provides drug coverage only (stand-alone PDP) to traditional Medicare (FFS) beneficiaries or to MA beneficiaries whose plans do not offer PD options. These plans, which are reimbursed directly by CMS, control the cost of prescription drugs by establishing a system of tiered formularies (lists of covered drugs for which they will make payment) that are tied to the monthly premiums paid by beneficiaries.

In this article, we describe the development of measures to assess beneficiaries' experiences with their prescription drug coverage. While many surveys exist to capture experiences with health care, the Medicare drug plan survey is the first instrument designed to measure experiences with drug benefits on a large scale. Given the multitude of Medicare drug plan choices, the lack of standardized drug coverage, and the complexity of tiered pricing schemes, selecting the appropriate domains and questions to help guide beneficiary decisionmaking proved to be a significant challenge.

The prescription drug plan instrument captured beneficiaries' experiences with prescription drug coverage during the initial year of Part D implementation (2006) and was included in the 2007 CAHPS® Medicare Survey. We first conducted focus groups with Medicare beneficiaries to identify salient themes regarding drug coverage. To investigate the reliability and validity of the prescription drug coverage measures, we used data from 132,960 MA beneficiaries and 202,289 stand-alone PDP beneficiaries who responded to the 2007 Medicare CAHPS® survey. Finally, we used data from 22 cognitive interviews to assess Medicare beneficiaries' understanding of the content and meaning of the prescription drug measures, as well as descriptive labels for those measures.

Methods

Instrument Development

As part of the development of measures of beneficiary experience with prescription drug coverage, we conducted 12 focus groups with Medicare beneficiaries in three sites to identify aspects of prescription drug coverage that were salient to beneficiaries. These focus groups were conducted separately for MA and FFS enrollees and for those who filled fewer than three versus three or more prescriptions in the previous 6 months. Results from these focus groups indicated that beneficiaries were especially concerned with (1) the ease of getting help and information from the plan, (2) understanding the coverage each plan provides and the steps required to get needed prescription drugs, (3) how easy it is to get prescription drugs under each plan, (4) the costs under each plan for obtaining needed prescription drugs, (5) the timeliness of reimbursements, and (6) the drug coverage rights of Medicare beneficiaries. This list of topics guided our development of a draft instrument.

In additional formative work, we conducted nine cognitive interviews with Medicare beneficiaries with prescription drug coverage to refine the wording of items (e.g., “pharmacy” versus “drug store,” “medicine” versus “prescription medicine,” “drug plan” versus “prescription drug plan”) in the draft instrument and narrow the item set. Participants in these interviews were FFS and MA enrollees between the ages of 65 and 85 years; all had prescription drug coverage for less than 6 months. These nine interviews were intended to identify problems with the phrasing, interpretation, and applicability of items in the draft instrument. The items that emerged from this formative work (Figure 1) covered three main aspects of prescription drug coverage (getting needed prescription drugs, getting needed information from the plan, and customer service) and included two global rating items (an overall rating of the plan and a question about plan members' willingness to recommend the plan to others). Additional cognitive interviews, described in detail below, were conducted to evaluate composite measures (multi-item scales) based on these items and labels for these composites.

Figure 1

Full Set of Prescription Drug Items Included in the 2007 Medicare CAHPS® Survey

Sample

The 2007 CAHPS® Medicare survey assessed the experiences of Medicare beneficiaries covered by MA, FFS Medicare without prescription drug coverage, and FFS Medicare with a stand-alone PDP. Respondents to the survey were selected from 288 MA contracts and 66 FFS contracts between Medicare and health service providers (i.e., plans). To be eligible for inclusion in the sample, individual enrollees had to be members of a contract plan in both October 2006 and January 2007. The primary objective of the sampling plan was to achieve adequate sample size to compare MA, MA-PD, and standalone PDP contracts, omitting only those contracts with enrollment too low to yield 300 completed surveys. An additional objective was to investigate the possibility of geographic heterogeneity in plan quality. Conditional on those two aims, sampling was otherwise proportionate to the size of each plan so that national estimates of plan quality would be as precise as possible. For cases in which a contract had a large number of enrollees and/or covered a wide geographic area (large multi-State plans), the enrollment population covered by the contract was split into smaller sampling strata that were defined by location (States or combinations of adjoining States). This investigation uses data from the 132,960 MA enrollees with Part D coverage and the 202,289 stand-alone PDP enrollees who completed the 2007 survey.

Survey Procedures

Sample beneficiaries were mailed a cover letter and the survey between April 3 and July 6, 2007. Beneficiaries who did not return surveys by mail were contacted up to four times to complete a telephone interview. Seventy percent of the surveys were completed by mail; the other 30 percent, by telephone. The overall response rate for the 2007 survey, and that for those with PD coverage, was 49 percent. Subgroups with significantly lower than average response rates include beneficiaries under age 45 with rates that range from 26-33 percent, Asian beneficiaries at 29 percent, and Hispanic beneficiaries at 33 percent (data not shown).

Analyses

To help select items for further evaluation, we calculated item-to-scale corrected correlations involving all of the non-screener (Yes/No) items in Figure 1 separately for MA-PD and PDP respondents. Scales were calculated by summing the responses to items in each domain indicated in Figure 1. We then dropped items that had low item-to-scale correlations and/or that correlated more highly with a scale other than the one intended. Using the reduced set of items, we estimated the reliability of composite measures (scales) with Cronbach's coefficient alpha, a measure of internal consistency, and calculated item-scale correlations, corrected for item overlap. Alpha coefficients greater than 0.70 are usually considered to indicate adequate reliability for group comparisons on psychometric constructs (Nunnally and Bernstein, 1994). We used confirmatory factor analysis to test the factor structure of the prescription drug coverage items included in composite measures. We judged the overall fit of this model using the comparative fit index (CFI) (Bentler, 1990) and the root mean square error of approximation (RMSEA) (Steiger, 1998). A CFI value greater than 0.95 (Hu and Bentler, 1999) and a RMSEA less than 0.06 (Browne and Cudeck, 1993; Yu and Muthén, 2002) suggest good fit. Although its sensitivity to sample size makes the model chi-square value a poor indicator of overall model fit in samples of this size (Marsh, Balla, and McDonald, 1988), we also include this commonly reported statistic, although we do not use it to assess the adequacy of fit of our models.

We calculated the plan-level reliability of composite measures and global rating items (Hays et al., 1999; Shrout and Fleiss, 1979; Solomon et al., 2002) to assess how well scores on the CAHPS® items and composites from multiple plan members distinguish among plans. This estimate of reliability represents the ratio of the variance in ratings between plans over the sum of the between-plan variance plus sampling error (Hays et al., 1999; Solomon et al., 2002). We also used the Spearman-Brown prophecy formula to estimate the sample sizes necessary to achieve reliabilities of 0.80 for composites and global rating items.

Finally, we used linear regression to assess the construct validity of the composite measures by examining their association with the overall rating of the drug coverage provided by the plan (in the MA-PD case; overall rating of the plan itself in the stand-alone PDP case) and respondents' willingness to recommend the plan for drug coverage (in the MA-PD case; willingness to recommend the plan itself in the stand-alone PDP case).

Cognitive Interviews

We created draft labels for the composite measures that emerged from the psychometric analyses, and evaluated beneficiaries' understanding of those labels, as well as the content and meaning of the composites, in a series of cognitive interviews. Twenty-two individual interviews, each typically lasting 65-75 minutes, were conducted with Medicare beneficiaries in Alexandria, Virginia and Baltimore, Maryland. Twelve participants were female, 12 were non-Hispanic Black, 9 were non-Hispanic White, and 1 was Native American. Participants' median age fell within the range of 65-74. Sixteen of the 22 participants were enrolled in traditional FFS Medicare (as opposed to MA plans); all but two were taking daily prescription medications to treat a chronic condition.

Participants completed one of two sets of procedures. In the first set of procedures, participants were provided with the items that constitute each of the composite measures and were asked to provide a label to describe what the items have in common. In the second set of procedures, participants were provided with all of the items that make up the composite measures (randomly ordered), and were asked to match each item to the label with which it seemed to best fit. At the end of both sets of procedures, participants were presented with each of the labels that the researchers had developed for the composite measures and were asked questions to probe their understanding of those labels (e.g., “Here is a phrase that some people have suggested would be a good summary phrase for this group of items. Do you think the phrase does a good job of describing these items? Do you think it describes all of the items in the group? Is there any item that doesn't seem to belong with this phrase? Do you think there is a better phrase for describing the items?”). Results of these cognitive interviews, described below, helped us to refine our composite labels and provided important insight into people's understanding of the measures.

Results

2007 Medicare CAHPS® Part D Surveys

Respondents were 37 percent male, 10 percent African American, and 2-3 percent each of Asian, Hispanic, and other non-White race. The median age of respondents was between 65 and 74 years, with 3 percent under age 45 and 9 percent over age 84. Twenty-five percent of respondents did not graduate from high school; 15 percent had a 4-year college degree. Seventeen percent of respondents were dually eligible for Medicare and Medicaid, and 7 percent received financial assistance from the Federal Government to cover part of their prescription drug costs (i.e., a low-income subsidy). Approximately onesixth were eligible for Medicare because of disability.

Psychometric Testing

Item-scale correlations among the full set of test items indicated that one question (about whether the plan told the member how to request that a drug be added to the plan's formulary) did not correlate positively with the other items in the hypothesized composites (getting needed information from the plan). We therefore excluded this item from further analyses.

The item-scale correlations also indicated that a hypothesized customer service item (about getting information/help regarding prescription drugs) correlated more highly with other items about getting needed information from the plan than it did with the question, “How often did your health (drug) plan's customer service treat you with courtesy and respect?” Given the conceptual and empirical fit of the information-focused customer service item with the other items about getting needed information from the plan, we decided to include this item in a composite measure of how easy it is to get needed information from the plan.

This left a single-item measure of members' customer service experience, an item that was not entirely comparable across the MA-PD and FS-PD surveys. That is, although the courtesy and respect item was included in a section of the MA-PD version of the survey labeled, “Your Prescription Drug Plan,” it did not specifically ask about customer service regarding prescription drugs. Thus, we decided not to test further a measure of customer service experience from the 2007 Medicare CAHPS® survey.

Table 1 presents the remaining items in the two areas and the two global ratings. The composite labels are not the draft labels that we tested in the cognitive interviews. The labels tested in those interviews were, “How Easy Is It to Find Out How the Plan Works?” and “How Easy Is It to Get the Drugs You Need?” The labels shown in Table 1 were chosen after the cognitive interviews revealed that the question format used in the draft labels caused some confusion.

Table 1

Composites and Global Rating Items Subjected to Full Psychometric Testing

Composite Measures Getting Information from the Plan about Prescription Drug Coverage and Cost In the last 6 months, how often did your plan give you all the information you needed about which medicines were covered? In the last 6 months, how often did your plan give you all the information you needed about how much you would have to pay for your prescription medicine? In the last 6 months, how often did your plan's customer service give you all the information you needed about prescription drugs? Getting Needed Prescription Drugs In the last 6 months, how often was it easy to use your plan to get the medicines your doctor prescribed? In the last 6 months, how often was it easy to use your plan to fill a prescription at your local pharmacy? In the last 6 months, how often was it easy to use your plan to fill a prescription by mail? Global Rating Items Using any number from 0 to 10, where 0 is the worst prescription drug plan possible and 10 is the best prescription drug plan possible, what number would you use to rate your [health plan for coverage of prescription drugs / prescription drug plan]? Would you recommend your [health plan for coverage of prescription drugs / prescription drug plan] to other people like yourself? (1=Definitely no to 4=Definitely yes)

NOTE: Unless otherwise noted, the response scale is: 0=never, 1=sometimes, 2=usually, 3=always.

SOURCE: Martino, S.C., Elliott, M.N., Kanouse, D.E., and Brown, J.A., RAND; Cleary, P.D., Yale School of Public Health; Spritzer, K.L. and Hays, R.D, University of California, Los Angeles; and Heller, A., UMHA Health and Retirement Funds, 2009.

Respondent-Level Reliability

Respondent-level reliability for the Getting Information from the Plan composite was high (α = 0.87) for both MA-PDs and the stand-alone PDPs. Internal consistency reliability for the Getting Needed Prescription Drugs composite fell just short of the 0.70 standard in both cases. However, corrected item-scale correlations (which indicate the relationship between a particular scale item and the sum of the other items in the scale) exceeded the conventional cutoff value of 0.40 (Nunnally and Bernstein, 1994) for all items (Table 2).

Table 2

Scale Properties of Items and Composites

	Correlation of Item with Sum of Other Items in Scale(MA-PD/stand-alone PDP)	Cronbach's Alpha for Scale(MA-PD/stand-alone PDP)
Getting Information from the Plan about Prescription Drug Coverage and Cost	—	0.87/0.87
How often did your plan give you all the information you needed about which medicines were covered?	0.79/0.79	—
How often did your plan give you all the information you needed about how much you would have to pay for your prescription medicine?	0.75/0.76	—
How often did your plan's customer service give you all the information you needed about prescription drugs?	0.71/0.72	—
Getting Needed Prescription Drugs	—	0.67/0.65
How often was it easy to use your plan to get the medicine your doctor prescribed?	0.53/0.52	—
How often was it easy to use your plan to fill a prescription at your local pharmacy?	0.46/0.44	—
How often was it easy to use your plan to fill a prescription by mail?	0.44/0.41	—

SOURCE: Data from the 2007 Medicare CAHPS® survey.

Factor Structure

We specified and tested a two-level (members clustered within plans) confirmatory factor model of the six items that the Getting Information from the Plan and Getting Needed Prescription Drugs composites comprise. The three items that dealt with getting needed information from the plan were constrained to load on one factor only and the three items that dealt with getting needed prescription drugs were constrained to load on a second factor only. The correlation between the two factors was freely estimated. A two-level analysis was necessary to generate robust standard errors for accurately testing the measurement model. The six items, which were measured on a 4-point scale with responses ranging from 0=never to 4=always, were treated as categorical in the analysis. We present results based on MA-PD data only. As Table 3 shows, a two-factor model fit the data well (χ2 (9) = 232.15, p<0.001; CFI = 0.997; RMSEA = 0.015) and factor loadings were uniformly high. The estimated correlation between the Getting Information from the Plan and the Getting Needed Prescription Drugs factors was 0.72.

Table 3

Factor Loadings from 2-Level Confirmatory Factor Analysis of PD Items from MA Survey

	Factor 1	Factor 2
How often did your plan give you all the information you needed about which medicines were covered?	0.93	—
How often did your plan give you all the information you needed about how much you would have to pay for your prescription medicine?	0.89	—
How often did your health plan's customer service give you the information or help you needed about prescription drugs?	0.85	—
How often was it easy to use your plan to get the medicine your doctor prescribed	—	0.92
How often was it easy to use your plan to fill a prescription at a local pharmacy?	—	0.80
How often was it easy to use your plan to fill a prescription by mail?	—	0.70

NOTES: Standardized coefficients are shown. Responses were measured on a 4-point scale with endpoints of 0=Never and 4=Always. Items were treated as categorical in the analysis.

χ2 (9) = 232.15, p<0.001; CFI = 0.997; RMSEA = 0.015.

SOURCE: Data from the 2007 Medicare CAHPS® survey.

Plan-Level Reliability

MA-PD plan-level reliability estimates for the composite and global ratings ranged from 0.82 to 0.92; stand-alone PDP plan-level reliability estimates ranged from 0.90 to 0.98 (Table 4). In both the MA-PD and the stand-alone PDP case, each of the composites and global ratings significantly discriminated among plans (F's ranged from 5.49 to 41.74, all p's <0.001). These results suggest that reliabilities of 0.80 or greater can be achieved with sample sizes ranging from 66 to 90 for MA-PD plans and from 124 to 249 for stand-alone PDP plans.

Table 4

Plan-Level Reliability and Validity of Composites and Global Rating Items

Measure	F-Statistic	Observed Plan-Level Reliabilitya	Avg. N Per Planb	N Per Plan for Reliability = 0.80c	Correlation with Overall Rating of Pland, e	Correlation with Willingness to Recommend Pland, f
Composites
Getting Information from the Plan about Prescription Drug Coverage and Cost	5.49†	0.82	128.7	66.9	0.44	0.42
	9.80†	0.90	902.6	239.0	0.51	0.47
Getting Needed Prescription Drugs	11.41†	0.91	368.8	82.7	0.47	0.41
	41.74†	0.98	2168.4	124.2	0.50	0.44
Global Rating Items
Overall Rating of Plan	12.52†	0.92	361.7	77.3	—	—
	23.66†	0.96	2194.5	226.0	—	—
Willingness to Recommend Plan	10.97†	0.91	382.8	89.9	0.64	—
	25.09†	0.96	2196.8	212.8	0.67	—

= p < 0.0001.

Plan-level reliability = (F-1)/F using the average number of respondents obtained per plan.

Within MA-PD and stand-alone PDP, variation in N's has to do largely with the fact that whereas all survey respondents were asked to provide global ratings of their plans, only those who indicated that they had sought information from their plans in the preceding 6 months were asked about their experiences in getting information from their plans and only those who indicated that they had used their plans to fill prescriptions in the preceding 6 months were asked about how easy it was to do so.

Calculated with the Spearman-Brown prophecy formula.

Pearson product-moment correlations.

MA-PD beneficiaries were asked to give an overall rating of the prescription drug coverage provided by their plan. Stand-alone PDP beneficiaries were asked to give an overall rating of their drug plan.

MA-PD beneficiaries were asked about their willingness to recommend their plan for prescription drug coverage. Stand-alone PDP beneficiaries were asked about their willingness to recommend their drug plan.

NOTE: Entries in the top of each row are for MA-PD; entries in the bottom are for stand-alone PDP.

SOURCE: Data from the 2007 Medicare CAHPS® survey.

Validity

Linear regression analyses indicate that both composites are strongly and independently predictive of both criterion variables among both MA-PD and stand-alone PDP plan members (Table 5). In all cases, the amount of variance in the criterion explained by the composites is around 30 percent. Thus a substantial portion of the variance in members' overall ratings of their plan and willingness to recommend their plan is driven by things other than their experiences getting needed prescription drugs and getting information from the plan.

Table 5

Regressions of Global Rating of the Plan and Willingness to Recommend the Plan on the Prescription Drug Composites

Composite	Overall Rating		Willingness to Recommend Plan
	b	SE	b	SE
(Constant)	24.489†	0.815	21.454†	0.950
	21.433†	0.896	17.654†	0.968
Getting Information from the Plan about Prescription Drug Coverage and Costa	0.222†	0.007	0.275†	0.008
	0.266†	0.004	0.307†	0.007
Getting Needed Prescription Drugsa	0.409†	0.008	0.405†	0.010
	0.404†	0.009	0.410†	0.009
	R2 = 0.298		R2 = 0.250
	0.340		0.280

= p < 0.0001.

Transformed to a 0-100 scale from the original 4-point scale.

NOTES: Entries are unstandardized beta coefficients and their standard errors. Top entries in each cell are for MA-PD; bottom entries are for stand-alone PDP.

SOURCE: Data from the 2007 Medicare CAHPS® survey.

Participants in the cognitive interviews who were provided with the composite labels (i.e., “How Easy Is It to Find Out How the Plan Works?” and “How Easy Is It to Get the Drugs You Need?”) and asked to match survey items to the labels generally did so in a manner that is consistent with the composites shown in Table 1. Moreover, participants who were shown the labels and asked about their meaning generally associated the labels with content that is consistent with their underlying items. Cognitive interview participants almost universally disliked the question format of the composite labels and suggested that we develop labels in a non-question format instead. When probed about this issue, several participants explained that the labels-as-questions were too easily confused with actual questions that might appear in the survey, obscuring the notion that these were summary measures. Thus, we switched to the labels that appear in all of the tables of this article.

Discussion

In 2008, a total of 1,824 prescription drug plans were available across the nation (Kaiser Family Foundation, 2008). Although not all plans are available to each Medicare beneficiary (beneficiaries may choose only from the plans available in their CMS-designated region of the country), beneficiaries in all but three States have at least 50 stand-alone plans and multiple MA-PD plans from which to choose (Kaiser Family Foundation, 2008). The number of plans, diversity of plan types, and variety of cost sharing and formulary designs have made choosing a plan a challenging, and sometimes frustrating, experience for many beneficiaries (Heaton, Carino, and Dix, 2006; Hoadley, 2008; Medicare Payment Advisory Commission, 2006; Neuman et al., 2007; Uhrig et al., 2006). Our aim in developing measures of consumer-rated quality of prescription drug plans is to facilitate informed decisionmaking by providing beneficiaries with easy-to-understand, reliable, and valid information about aspects of prescription drug plans that are important to them.

The current CAHPS® standard for sample size is 300 completed surveys per plan (CAHPS). Our analysis indicates that this sample size would provide more than enough information for detecting differences among plans on the dimensions that we assessed: ease of getting information from the plan about coverage and costs, ease of getting needed prescription drugs, and overall rating of the plan and willingness to recommend it. Although the getting information composite had strong internal consistency, Cronbach's alpha for the getting needed prescription drugs composite fell somewhat short of the 0.70 level often defined as the minimum acceptable level for making group comparisons. In reporting of CAHPS® data, composites are used as convenient summary measures of items that cover a broad domain. Greater breadth of item content than would be desirable when measuring pure psychometric constructs is in fact desirable in CAHPS® composites because it allows for the assessment of a broad range of patient experience. Thus, slightly weaker internal consistency is acceptable for CAHPS® composites. Moreover, given that CAHPS® composites are used to evaluate characteristics of plans rather than characteristics of enrollees, the most important reliability estimate for the measures is plan-level reliability; each of the composite measures that we tested had sufficient plan-level reliability to inform consumer choice, quality improvement, and payor oversight (including pay-for-performance).

Both composite measures were positively associated with members' overall rating of their plan and their willingness to recommend their plan (for prescription drug coverage). Moreover, although the correlation between the composites was high, each was independently useful in predicting members' global ratings of their plan. Together, scores on the two composites accounted for between 25 and 35 percent of the variation in members' overall evaluations of their plans, suggesting that consumers rely considerably on their experiences getting information from plans and getting the medicines they need when evaluating the quality of their drug plans. This percentage is comparable to the 40 percent of variance in patients' overall evaluations of medical groups that is accounted for by patients' scores on CAHPS® composite measures of clinician access, office staff service, and patient-clinician communication (Solomon et al., 2002). It is not as high as the 54 percent of variance in patients' overall evaluations of their health plans that is accounted for by patients' scores on CAHPS® composite measures of patient-clinician communication, patients' ability to get the care they need when they need it, perceptions of office staff, and experiences with customer service (Hargraves, Hays, and Cleary, 2003). We note, however, that it would be very difficult to attain the same level of explanatory power with two measures as has been attained with five.

The one item that appears to be problematic is, “Did your (drug) plan give you information about how to ask the (drug) plan to cover a medicine.” The problem with this question, as written, is that responses were not clearly interpretable. In replying, “Yes,” plan members may have been indicating both that they were in the (unfortunate) situation of needing to appeal to the plan to cover their medication and that the plan provided the necessary information. In replying, “No,” plan members may have been indicating either that they were not in the situation of needing to appeal to the plan to cover a medication or that they were in such a situation and could not get the necessary information from their plan. In fact, this item had a weak negative correlation with the other items about getting needed information from the plan, suggesting that the former interpretation of a “No” response is more often correct than the latter interpretation. Because of this ambiguity, this item was not used in the public reporting of 2007 Medicare CAHPS® data.

Although we attempted to develop a composite measure of Medicare beneficiaries' experiences with their prescription drug plan's customer service staff (or, in the case of MA-PD enrollees, their health plan's customer service regarding issues of prescription drugs), one of the items had a better fit empirically with items about getting information from the plan and the other was written in a way that responses were not clearly interpretable. New customer service items were developed for the 2008 Medicare CAHPS® survey. Psychometric testing of these items will be conducted when data from that survey become available.

Our study demonstrates the importance of supplementing empirical data on the reliability and validity of CAHPS® measures with qualitative data on consumers' understanding and perception of the measures. From our cognitive interviews we learned that Medicare beneficiaries' understanding of the composite measures was consistent with the intended content of the items that they comprise. We also learned that although beneficiaries understood the labels we had generated initially for the composites, they were not in favor of labels that were written as questions, e.g., “How Easy Is It to Get the Drugs You Need?” Thus, we revised the labels to eliminate potential confusion related to the question phrasing. It is a limitation of this study that we did not test the new composite labels with beneficiaries, something that we plan to do in future research. Moreover, while the number of cognitive interview we conducted exceeds the 5 to 15 that are typical (Willis, 2005), more extensive testing might have revealed additional problems (Blair et al., 2006).

Conclusion

Without a clear understanding of their many prescription drug plan choices, Medicare beneficiaries are likely to find it difficult to make an informed decision about prescription drug coverage (Dulio, Perry, and Cubanski, 2008). Part of this understanding comes from knowing how other beneficiaries evaluate prescription drug plans based on their experiences with them. The instrument we have developed and continue to refine provides a reliable and valid means of assessing beneficiaries' experiences with their prescription drug plans. Further research is needed to investigate the extent to which providing data collected via this instrument helps beneficiaries to choose a plan that best meets their needs.