Randomized trial of stage-based interventions for informed Medicare choices.

A randomized trial involving 1,351 new Medicare enrollees was conducted to assess the efficacy of a transtheoretical model (TTM) based manual and multimedia expert system program that delivered guidance and feedback matched to individual stage of readiness to compare Medicare health plans. At 6 months post-intervention, compared to enrollees in the control group, those receiving the manual plus expert system intervention or the manual alone exhibited greater increases in Medicare knowledge. The TTM-based interventions also increased use of and satisfaction with traditional Medicare education materials among most enrollees. The interventions' impact on stage of change for comparing plans was observed only among treatment group participants who had examined and evaluated the materials. The challenges to increasing informed choice and possible dissemination channels for stage-based materials are discussed.

RCT Entities:   Population Interventions Outcomes

Background

The 1997 Balanced Budget Act (BBA) authorized new health plan options under the Medicare+Choice program, providing Medicare beneficiaries with a range of choices that could potentially offer more benefits at lower cost than traditional fee-for-service Medicare. The 2003 Medicare Prescription Drug, Improvement, and Modernization Act (MMA) added a prescription drug benefit and expanded plan options. Those changes and others introduced with the MMA dramatically increase the complexity of the Medicare Program and decisionmaking process for beneficiaries and new enrollees.

To help meet Medicare beneficiaries' need for information and decision support, CMS has focused on information infrastructure, public-private partnerships, its Web site, and 1-800-Medicare toll-free help line. However, improvements in these areas cannot ensure that beneficiaries will carefully consider the range of health plan options available. A recent survey found that although 52 percent of Internet users age 65 or over had heard of the Medicare. gov Web site, only 11 percent had ever visited it, and although 55 percent of seniors had heard of the toll-free Medicare help line, only 13 percent had ever called it (Kaiser Family Foundation, 2006).

The TTM offers an empirically validated framework to help beneficiaries increase their information-seeking behavior and involvement in choice. Research on the TTM has found that behavior change involves progress, over time, through a series of stages that represent ordered categories along a continuum of motivational readiness: precontemplation, contemplation, preparation, action, and maintenance (Prochaska and DiClemente, 1983). The model includes additional dimensions central to change:

Decisional Balance—The pros and cons associated with a behavior's consequences (Janis and Mann, 1977).

Self-Efficacy—Confidence to make and sustain changes in difficult situations, and temptation to slip back into old patterns (Bandura, 1977).

Processes of Change—10 cognitive, affective, and behavioral activities that facilitate progress through the stages of change (Prochaska and DiClemente, 1985).

Over 25 years of research on a variety of health behaviors and in a variety of populations have identified the principles and processes of change that work best in each stage to facilitate progress. This research served as a foundation on which to build stage-matched interventions to increase participation in informed health plan choice among Medicare beneficiaries.

In Phase I of a program of research applying the TTM to informed choice in the Medicare population, CMS' Center for Beneficiary Choices contracted with Pro-Change Behavior Systems to: (1) develop an operational definition of informed choice, (2) develop and validate measures of the core constructs of the TTM, and (3) establish decision rules and cutoff scores necessary for the development of TTM stage-based interventions. Phase I measures and findings demonstrating the applicability of the TTM to informed choice are described elsewhere (Levesque et al., 2006).

In Phase II, Pro-Change developed TTM-based interventions to increase informed choice among new enrollees by accelerating progress through the stages of change for comparing plans, operationally defined as:

Finding out what your Medicare health plan choices are.

Gathering information on the different Medicare health plans.

Comparing the advantages and disadvantages of your choices, such as cost, benefits covered, doctors and hospitals you can use, rules you must follow to get care, and the quality of the health care provided.

Using this information to choose the plan that best meets your needs given your health and financial situation.

For a detailed description of the TTMbased intervention materials refer to Levesque and Cummins (2002).

In Phase III, a randomized trial was conducted to assess the efficacy of the TTM intervention materials among individuals newly eligible for Medicare benefits. Additional information on approvals from administrative agencies and organizations, the structure and content of the intervention materials, and study eligibility criteria and recruitment procedures are available in Levesque and Cummins (2004).

Method

Intervention Materials

Expert System

The multimedia computer-administered expert system intervention, entitled “Making Good Choices about Medicare Health Plans”, was intended for new enrollees who had just received their initial enrollment package from CMS 3 months prior to reaching age 65. The program was disseminated over the Internet with multimedia components residing on a CD-ROM. During a 20-minute interactive session, the expert system compiled text paragraphs, audio files, and images as it administered a TTM assessment and provided immediate individualized feedback on the participants' stage of change, decisional balance, processes of change, self-efficacy, and strategies for taking small steps to progress to the next stage. The expert system also referred participants to sections of the manual. At the end of the session, enrollees could print out a report containing their feedback. The final screens in the interactive session and the printed report included links to parts of the Medicare Web site that were most stage appropriate. For example, participants in the contemplation stage were linked to “Helpful Contacts,” and those in action were linked to “Medicare Compare.”

Stage-Matched Manual

The 30-page manual is based on TTM research on new Medicare enrollees. Designed to be used alone or with the expert system, the manual provides a definition of comparing plans, asks readers to assess their own readiness to compare plans, and guides them to sections containing stage-matched information and exercises based on TTM principles and processes of change. The manual integrates CMS educational information, support tools, and services in a stage-appropriate fashion, and includes a plan comparison worksheet.

Participants

Recruitment and Random Assignment

The efficacy of the manual plus expert system intervention was tested in a sample of 451 new Medicare enrollees who had access to a computer with a CD-ROM and Internet connection (Subsample One); the efficacy of the manual alone was tested in a sample of 900 new enrollees, most of whom did not have computer access (Subsample Two). New enrollees were randomly selected from CMS' Initial Enrollment File, a repository of data for persons who were going to turn age 65 in the next 3 months and become eligible for Medicare. A telematch service located telephone numbers for 48 percent of selected enrollees, whose contact information was then forwarded to a survey research center.

At baseline, the Making Good Choices about Medicare Health Plans Survey was administered using a mixed-mode methodology involving an initial mail survey with a telephone followup for non-responders. At the start of the survey, a question assessed access to a computer with a CD-ROM and Internet connection. Participants with computer access were assigned to Subsample One, and those without were assigned to Subsample Two. Once recruitment for the first subsample was completed, individuals with computer access were included in the second subsample until the target sample size was reached. Next, individuals in Subsample One were randomly assigned to an intervention group that received the manual plus expert system intervention, or to a control group that did not; individuals in Subsample Two were randomly assigned to the intervention group that received the manual alone, or to a control group that received no materials.

Participant Characteristics

The baseline survey response rate was 70 percent. On average, respondents completed the baseline survey 7.1 weeks (standard deviation [SD]=1.8) before age 65. Forty-seven percent had access to a computer with a CD-ROM and Internet connection—100 percent of Subsample One and 19 percent of Subsample Two. Eighty-three percent of the sample was White non-Hispanic, 7 percent Black non-Hispanic, 3 percent Hispanic, and 7 percent other. Fifty-seven percent were female, 48 percent had annual household incomes above $30,000, and 70 percent were married. Sixteen percent had not completed high school, 38 percent had completed high school or earned a GED, 28 percent had attended some college, and 18 percent had a college degree. Fifty-six percent of participants reported that they had already chosen a Medicare health plan; 6 percent of the sample reported that they will be covered by Medicaid in addition to Medicare. At baseline, 36 percent of the sample were in the precontemplation stage for comparing plans, 6 percent were in contemplation, 14 percent in preparation, and 44 percent in action. Among participants who had already chosen a Medicare health plan, 34 percent were in precontemplation, 3 percent contemplation, 6 percent preparation, and 57 percent action. Presumably, individuals in the preaction stages who had already chosen a health plan had done so without comparing their options.

Control and treatment group characteristics for Subsamples One and Two at baseline are presented in Table 1. Measures assessed at baseline revealed no significant differences between the Subsample One control and treatment groups, or between the Subsample Two control and treatment groups for demographics, health status, Medicaid eligibility, or other study variables. However, there were significant differences (p<0.01) between Subsamples One and Two on all measures (Table 1). For example, individuals in Subsample One had more education and higher incomes, and scored higher on objective and subjective measures of Medicare knowledge. These subsample differences precluded any direct comparison of the manual plus expert system intervention versus the manual alone.

Table 1

Control and Treatment Group Characteristics for Subsamples One and Two at Baseline

Characteristic	TotalN	Subsample One		Subsample Two
		Control(n=225)	Treatment(n=226)	Control(n=451)	Treatment(n=449)
		Percent
Race/Ethnicity
White, non-Hispanic	1,116	90.2	87.6	81.7	78.5
Black, non-Hispanic	92	2.2	3.6	7.6	10.1
Hispanic	37	2.2	3.1	3.1	2.5
Other	99	5.4	5.8	7.6	8.9
Sex
Male	584	52.0	44.2	40.8	40.8
Female	767	48.0	55.8	59.2	59.2
Marital Status
Married	930	76.7	80.8	64.8	65.1
Widowed	173	9.0	7.6	16.3	14.3
Divorced/Separated	185	11.2	10.3	14.5	16.3
Never Married	49	3.1	1.3	4.5	4.3
Income
<$10,000	116	2.5	3.0	13.9	13.9
$10,000 - $19,999	265	17.5	11.2	25.8	29.4
$20,000 - $29,000	214	19.0	14.7	20.0	19.0
$30,000 - $49,999	279	24.5	34.0	20.5	22.7
$50,000	277	36.5	37.1	19.7	15.0
Education
Less than High School	211	4.9	4.9	21.2	21.3
High School or GED	512	30.0	28.4	42.0	43.7
Some College or 2-Year Degree	378	31.4	38.2	26.3	23.5
Four-Year College Degree	237	33.6	28.4	10.5	11.5
Health Status
Excellent	200	18.4	20.0	13.1	12.4
Very good	457	37.2	38.7	29.3	35.0
Good	454	33.6	28.4	37.6	33.0
Fair	188	9.0	10.7	15.8	16.5
Poor	42	1.8	2.2	4.2	3.2
Eligible for Medicaid
Yes	69	1.6	2.2	7.7	9.5
No	1,019	98.4	97.8	92.3	90.5
Had Chosen Medicare Health Plan at Baseline
Yes	757	67.0	64.2	53.2	49.9
No	589	33.0	35.8	46.8	50.1
Stage of Change at Baseline
Precontemplation	476	26.0	31.8	38.8	39.6
Contemplation	84	4.9	4.5	7.7	6.6
Preparation	180	10.8	13.5	15.3	13.1
Action	591	58.3	50.2	38.1	40.7
		Mean (SD)
Knowledge at Baseline
Medicare	1,297	4.4 (1.7)	4.1 (1.8)	3.8 (1.9)	3.7 (1.8)
Perceived	1,342	2.9 (1.1)	3.0 (1.1)	2.6 (1.1)	2.6 (1.1)

NOTE: Numbers in parentheses are standard deviations.

SOURCE: Making Good Choices about Medicare Health Plans Survey, 2002-2003.

Intervention Trial

After completing the baseline survey, intervention materials were mailed to treatment group participants. One month later, a brief survey was mailed to assess whether treatment group participants had used the materials, and whether they found them to be understandable, easy to use, attractive, and helpful. One hundred and thirty-two (29 percent) of the 449 new enrollees who received the manual alone and 51 (23 percent) of the 226 new enrollees who received the manual plus expert system intervention returned the acceptability survey. A majority of respondents found the manual to be easy to use and understand, attractive, and informative; 87 percent still had the manual, and 85 percent reported that they would recommend it to a friend. Acceptability ratings for the expert system intervention were similar to those for the manual. A more detailed description of the acceptability findings is available in Levesque and Cummins (2004).

At 6-months' followup, the Making Good Choices about Medicare Health Plans Final Survey assessing outcomes was completed by 1,087 of 1,351 study participants (81 percent). Compared to respondents, non-respondents had significantly lower baseline knowledge scores, were less likely to have attended some college, and were more likely to be of Hispanic origin or descent, or separated. On average, participants completed the followup survey 18.6 weeks (SD=3.3) after reaching age 65.

Measures

The baseline survey assessed demographics and health status. The outcome measures were assessed at baseline and followup, unless otherwise noted.

Medicare Knowledge—A 7-item knowledge quiz taken from Rounds 23 and 26 of the Medicare Current Beneficiary Survey (MCBS) assessed knowledge about the Medicare Program. A scale score was computed as the total number of correct responses (Bann et al., 2003). Cronbach's (1951) Alpha for the scale was 0.64 in the current study.

Perceived Knowledge—Three questions from Rounds 24 and 26 of MCBS assessed how much new enrollees felt they knew about: (1) what medical services Medicare covers, (2) supplemental or Medigap insurance, and (3) the availability and benefits of Medicare managed care plans. Response options ranged from 1 (just about everything you need to know) to 5 (almost none of what you need to know). The response scale was reversed and a scale score was calculated by taking the mean of the three items (Bann et al., 2003). Cronbach's (1951) Alpha was 0.81 in the current study.

Use of Traditional Medicare Information—Questions adapted from the 2003 CMS Survey of New Medicare Beneficiaries assessed information seeking. Respondents were asked to place a checkmark next to each information source they used during the last 12 months. For each source used, respondents indicated how useful they found the information in understanding their health plan choices. Response options ranged from 1 (not useful at all) to 4 (very useful). We examined use of and satisfaction with two formal information sources: the Medicare & You handbook and the Medicare Web site.

Satisfaction with Medicare Information—At followup only, two questions taken from Rounds 20 and 23 of MCBS assessed satisfaction with Medicare information: (1) “In general, do you think the Medicare program is understandable?” and (2) “How satisfied are you in general with the availability of information about the Medicare program when you need it?” Response options for the first question were “yes,” “no,” and “I don't know;” response options for the second were “very satisfied,” “satisfied,” “very unsatisfied, “not applicable,” and “I don't know.”

Health Plan Choice—Participants were asked whether they had chosen a Medicare health plan and, if so, to indicate the general type of plan they had chosen. Response options were “the Original Medicare plan (with or without supplemental Medigap insurance),” “a Medicare managed care plan (HMO),” “a Medicare Private Fee-for-Service Plan,” and “I am not sure what I chose.”

Stage of Change for Comparing Plans—Participants were presented with the operational definition of comparing plans previously presented, and then asked: (1) “Have you compared different Medicare health plans in the last 6 months?” (2) “Do you intend to compare different health plans in the next 3 months?” and (3) “Do you intend to compare different plans in the next 30 days?” Individuals who had compared plans in the last 6 months were classified in the action stage. Among individuals who had not compared plans, those who had no intention of doing so in the next 3 months were classified in precontemplation, those intended to do so in the next 3 months were classified in contemplation, and those who intended to do so in the next 30 days were classified in preparation. Validation research found that individuals in later stages were more knowledgeable about the Medicare Program, engaged in more information seeking, and were more likely to have chosen a Medicare health plan (Levesque et al., 2006).

Data Analysis

Medicare Knowledge and Perceived Knowledge

Two 3-way repeated measures analyses of variance assessed the effect of the TTM interventions on change in Medicare knowledge and perceived knowledge, the dependent variables. The between-subjects independent variables were intervention condition (treatment/control) and subsample (One/Two), and the within-subjects independent variable was assessment timepoint (baseline/followup). For continuous measures, η2 (eta square) was used as the measure of effect size. For η2, 0.010 generally indicates a small effect, 0.059 a medium effect, and 0.138 a large effect (Cohen, 1988). However, for large-scale population-based intervention trials, which tend to find effect sizes in the 0.010 range, small, medium, and large effects can be defined as 0.005, 0.010, and 0.015, respectively (Rossi, 2003). On a population basis, small effects can have enormous practical importance and overall impact.

Use of and Satisfaction with Medicare Information

Among study participants who had not chosen a Medicare health plan at baseline, one-tailed 2×2 Chi Square tests examined the main effect of intervention condition (treatment/control) on the following 6-month measures: (1) reading the Medicare & You handbook; (2) perceived helpfulness of the handbook among users; (3) use of Medicare Web site; (4) perceived helpfulness of the Web site among users; (5) whether Medicare information was understandable; and (6) satisfaction with Medicare information. We then assessed condition × subsample interaction effects by calculating the mean of the effect sizes across the information use and satisfaction measures in Subsample One, and then in Subsample Two. A t-test of the difference between Subsamples One and Two effect size means assessed whether there was a differential impact of the manual plus expert system intervention versus the manual alone in their respective subsamples. For dichotomous measures, Cohen's h was used as the measure of effect size. Generally, h=0.20 indicates a small effect, 0.50 a medium effect, and 0.80 a large effect (Cohen, 1988). However, for large-scale population-based intervention trials, small, medium and large effects can be defined as h=0.15, 0.20, and 0.25, respectively (Rossi, 2003).

Health Plan Choice

Among participants who had not chosen a Medicare health plan at baseline, one-tailed 2×2 Chi Square tests examined the main effect of intervention condition (treatment/control) on whether participants had chosen a plan at followup and could indicate the general type of plan chosen. Participants who were not sure what type of plan they had chosen were grouped with participants who had not chosen a plan, since their choice was unlikely to have been an informed one. Condition × subsample interaction effects were examined by calculating and comparing intervention effect sizes for Subsamples One and Two.

Stage Progression and Regression

Among participants who had not chosen a Medicare health plan and were in a preaction stage at baseline, stage progression was defined as movement to a more advanced stage at followup. Among study participants who had not chosen a plan and were in the contemplation, preparation, or action stage at baseline, stage regression was defined as movement to an earlier stage at followup. Two one-tailed 2×2 Chi Square tests examined the overall or main effect of treatment condition (treatment/control) on stage progression and regression. Condition × subsample interaction effects were examined by calculating and comparing intervention effect sizes for Subsamples One and Two.

Results

Medicare Knowledge and Perceived Knowledge

Figure 1 shows Medicare knowledge scores at baseline and followup for Sample One and Sample Two treatment and control groups. There was a within-subjects effect for time (mean knowledge scores increased from 4.0 (SD=1.9) at baseline to 4.4 (SD=1.8) at followup, F(1,1024)=46.9, p<0.001, η2=0.044), and a between-subjects effect for subsample (knowledge scores were higher in Subsample One, F(1,1024)=24.3, p<0.001, η2=0.023). There was an effect for condition × time (F(1,1024)=12.3, p<0.001, η2=0.012), indicating that treatment group participants showed greater gains in Medicare knowledge over time than control group participants. No effect was found for condition × subsample × time (F(1,1024)=0.8, p=0.38, η2=0.001), indicating that the pattern of treatment-control group differences in Subsample One was not different from the pattern in Subsample Two.

Figure 1

Medicare Knowledge for Two Treatment and Control Groups at Baseline and 6 Months Followup

Results for perceived knowledge show a similar pattern of improvement (Figure 2), with a within-subjects effect for time (in the whole sample, scores increased from 2.8 (SD=1.1) to 3.0 (SD=1.0), F(1,1070)=47.5, p<0.001, η2=0.043), a between-subjects effect for subsample (F(1,1070)=39.6, p<0.001, η2=0.036), an interaction effect for condition × time (F(1,1070)=4.3, p<0.05, η2=0.004), and no interaction effect for condition × subsample × time (F(1,1070)=0.0, p=0.912, η2=0.000).

Figure 2

Perceived Knowledge for Two Treatment and Control Groups at Baseline and 6 Months Followup

Use of and Satisfaction with Medicare Information

First, examining the overall sample, Table 2 shows rates of use of and satisfaction with Medicare information among treatment and control group participants who had not chosen a Medicare health plan at baseline. Findings on the individual outcome measures were in the predicted direction, but did not always reach statistical significance. Across all six measures, the mean effect size was 0.183 (95 percent confidence interval (CI)=0.115 to 0.250), which was significantly different from 0.

Table 2

Rates of Use and Satisfaction with Medicare Information Among Treatment and Control Group Participants Who Did Not Choose a Health Plan Baseline

Measure	Overall Sample					Subsample One Effects Sizes1	Subsample Two Effect Sizes1
	N	Treatment Percent	Control Percent	χ2 (df=1)	Effect Sizes*
Read Medicare & You Handbook	589	45.1	38.2	2.8	0.140	0.299	0.078
Found Handbook Somewhat or Very Useful	249	85.5	75.7	*3.9	0.250	0.336	0.219
Used Medicare Web Site	589	10.2	7.4	1.5	0.099	0.313	—
Found Web Site Somewhat or Very Useful	53	84.4	71.4	1.3	0.316	0.657	—
Found Medicare Information Understandable	457	57.0	48.2	*3.5	0.176	0.357	0.108
Satisfied or Very Satisfied with Medicare Information	439	69.3	63.9	1.4	0.115	0.341	0.035
Mean Effect Size	—	—	—	—	0.183	0.384	0.110

Cohens h. Generally, h=0.20 indicates a small effect, 0.50 a medium effect, and 0.80 a large effect (Cohen, J.: Statistical Power Analysis for the Behavioral Sciences. Second Edition. Erlbaum. Hillsdale, NJ. 1988). However, for large-scale population-based intervention trials, small, medium and large effects can be defined as h=0.15, 0.20, and 0.25, respectively (Rossi, J.S.: Comparison of the Use of Significance Testing and Effect Sizes in Theory-Based Health Promotion Research. Paper Presented at the 43rd Annual Meeting of the Society for Multivariate Experimental Psychology. Keystone, CO. September 2003).

p<0.05.

NOTE: Given the under-representation of individuals with computer access in Subsample Two, use of and perceived helpfulness of the Medicare Web site were not examined in that group.

SOURCE: Making Good Choices about Medicare Health Plans Final Survey, 2003.

Next, we examined the effect of the TTM interventions on the same six outcome measures in the separate subsamples. In all cases, findings were in the predicted direction: individuals in Subsamples One and Two treatment groups were more likely than their peers in the control groups to: (1) have read their Medicare & You handbook (57 versus 42 percent in Subsample One, 41 versus 37 percent in Subsample Two); (2) have found their handbook somewhat or very helpful if they had read it (85 versus 71 percent in Subsample One, 86 versus 78 percent in Subsample Two); (3) have used the Medicare Web site (31 versus 18 percent in Subsample One); (4) have found the Web site somewhat or very helpful if they had used it (92 versus 67 percent in Subsample One); (5) find Medicare information understandable (67 versus 49 percent in Subsample One, 53 versus 48 percent in Subsample Two); and (6) be satisfied or very satisfied with Medicare information (81 versus 66 percent in Subsample One, 65 versus 63 percent in Subsample Two).

Table 2 shows effect sizes for the manual plus expert system intervention in Subsample One, and for the manual alone in Subsample Two. The mean effect size in Subsample One was about 3½ times greater than the mean effect size in Subsample Two (0.384, 95 percent CI=0.276 to 0.492 versus 0.110, CI=0.033 to 0.187). The mean effect sizes for the two subsamples were both significantly greater than 0, and were significantly different from each other (t(8)=3.6, p<0.01).

A post hoc test was conducted to examine whether differences in the effect sizes for the two subsamples might be explained, at least in part, by education, one of the major differences between the two subsamples, rather than by any differences between the two interventions. Previous analyses were expanded by assessing condition × subsample × education interaction effects among participants with low (high school or less) and high (at least some college) education. Table 3 shows effect sizes for the Medicare information use and satisfaction measures among Subsamples One and Two participants with the two levels of education. The mean effect size for the manual plus expert system intervention among Subsample One participants with low and high education were 0.360 (CI=0.004 to 0.715) and 0.423 (CI=0.334 to 0.512), respectively; the mean effect size for the manual alone among Subsample Two participants with low and high education were 0.018 (CI=-0.074 to 0.110) and 0.270 (CI=0.183 to 0.357), respectively. With the exception of the Sample Two low education group, mean intervention effects for all groups were statistically different from 0, quite large for a population-based intervention trial, and roughly equivalent to each other. Multiple comparisons found that the mean effect size for the Sample Two low education group was significantly lower than the mean effects for the Sample One and Sample Two high education groups (t(8)=6.0, p<0.001 and t(6)=3.9, p<0.01, respectively).

Table 3

Rates of Use of and Satisfaction With Medicare Information Among Treatment and Control Group Participants With Two Education Levels

Measure	Subsample One				Subsample Two
	High School or Less		Some College		High School or Less		Some College
	Effect Sizes1	n	Effect Sizes1	n	Effect Sizes1	n	Effect Sizes1	n
Read Medicare & You Handbook	0.107	56	0.360	99	0.045	294	0.167	135
Found Handbook Somewhat or Very Useful	0.284	20	0.390	57	0.102	106	0.337	64
Used Medicare Web Site	0.050	56	0.376	99	—	—	—	—
Found Web Site Somewhat or Very Useful	1.231	6	0.618	31	—	—	—	—
Found Medicare Information Understandable	0.110	42	0.483	83	0.040	222	0.224	106
Satisfied or Very Satisfied with Medicare Information	0.377	40	0.309	79	-0.117	212	0.350	104
Mean Effect Size	0.360	—	0.423	—	0.018	—	0.270	—

Cohen's h. Generally, h=0.20 indicates a small effect, 0.50 a medium effect, and 0.80 a large effect (Cohen, J.: Statistical Power Analysis for the Behavioral Sciences. Second Edition. Erlbaum. Hillsdale, NJ. 1988). However, for large-scale population-based intervention trials, small, medium and large effects can be defined as h=0.15, 0.20, and 0.25, respectively (Rossi, J.S.: Comparison of the Use of Significance Testing and Effect Sizes in Theory-Based Health Promotion Research. Paper Presented at the 43rd Annual Meeting of the Society for Multivariate Experimental Psychology. Keystone, CO. September 2003).

SOURCE: Making Good Choices about Medicare Health Plans Final Survey, 2003.

Health Plan Choice

Among study participants who had not chosen a Medicare health plan at baseline, 59 percent of treatment group participants had chosen a plan at followup and could indicate the type of plan chosen, compared to 50 percent of control group participants (χ2 (1,N=446)=3.3, p<0.05, h=0.173). Effect sizes in Subsamples One (h=0.174) and Two (h=0.170) were virtually equivalent.

Stage Progression and Regression

At baseline, 39 percent of participants who had not chosen a Medicare health plan were in the precontemplation stage for comparing plans, 10 percent were in contemplation, 23 percent preparation, and 28 percent action. Only 30 percent of these participants progressed one or more stages during the 6-month followup period, and 43 percent regressed. For the treatment and control groups, stage progression rates were identical, at 30 percent, and stage regression rates were 41 versus 45 percent, respectively. Neither effect was statistically significant.

Results of Chi Square tests and intervention effect sizes are reported in Table 4. For the separate subsamples, rates of stage progression for the treatment and control groups were 26 versus 28 percent, respectively, in Subsample One, and 32 versus 31 percent in Subsample Two. Rates of stage regression for the treatment and control groups were 41 versus 44 percent in Subsample One and 40 versus 45 percent in Subsample Two. Effect sizes for the two subsamples were not statistically different from each other.

Table 4

Rates of Progression and Regression in Stage of Change for Treatment and Control Group Participants Comparing Medicare Health Plans

Measure	Overall Sample					Subsample One Effects Sizes1	Subsample Two Effect Sizes1
	N	Treatment Percent	Control Percent	χ2 (df=1)	Effect Sizes1
Stage Progression	310	30.3	30.3	0.0	0.002	-0.034	0.015
Stage Regression	288	40.5	45.0	0.6	0.091	0.059	0.105
Treatment Group Limited to known Intervention Users
Stage Progression	190	44.7	30.3	2.9	0.299	0.069	0.212
Stage Regression	173	27.3	45.0	*3.5	0.371	0.301	0.412

Cohen's h. Generally, h=0.20 indicates a small effect, 0.50 a medium effect, and 0.80 a large effect (Cohen, J.: Statistical Power Analysis for the Behavioral Sciences. Second Edition. Erlbaum. Hillsdale, NJ. 1988). However, for large-scale population-based intervention trials, small, medium and large effects can be defined as h=0.15, 0.20, and 0.25, respectively (Rossi, J.S.: Comparison of the Use of Significance Testing and Effect Sizes in Theory-Based Health Promotion Research. Paper Presented at the 43rd Annual Meeting of the Society for Multivariate Experimental Psychology. Keystone, CO. September 2003).

p<0.05.

SOURCES: Making Good Choices about Medicare Health Plans Survey, 2002-2003, and Making Good Choices about Medicare Health Plans Final Survey, 2003.

These null findings regarding the effect of the TTM materials on stage progression and regression were surprising, given that the intervention materials are associated with greater increases in Medicare knowledge, a greater likelihood of using traditional Medicare materials, and a greater likelihood of making a health plan choice, all presumably requisites for comparing Medicare health plans. In addition, earlier work found these same dimensions to be related as predicted with stage of change for comparing plans (Levesque et al., 2006).

To confirm the negative findings, we repeated the analyses and included in the treatment group only those participants who were known to have at least examined the intervention materials: that is, those who had returned the acceptability survey (Table 4). Compared to control group participants, individuals who examined the intervention materials showed a non-significant trend toward higher rates of stage progression (45 versus 30 percent, p=0.07), and were significantly less likely to regress to an earlier stage of change for comparing health plans (27 versus 45 percent, p<0.05).

For the separate subsamples, rates of stage progression for the treatment and control groups were 33 versus 28 percent, respectively, in Subsample One, and 48 versus 31 percent in Subsample Two. Rates of stage regression were 33 versus 44 percent in Subsample One and 25 versus 45 percent in Subsample Two. Effect sizes for the two subsamples were not statistically different from each other.

Discussion

While both the treatment and control groups showed significant gains in Medicare knowledge during the study period—which would be expected during a critical period when enrollees were being asked to choose a plan and were being exposed to new information from CMS, health plans, and other sources—knowledge gains were significantly greater among participants who received the TTM manual plus expert system intervention or the manual alone. Among most enrollee groups, the TTM intervention increased use of and satisfaction with traditional Medicare information, so it is possible that at least a portion of the treatment groups' differential gain in knowledge was due to increased use of—or ability to benefit from—existing CMS educational materials. However, among less-educated and less-advantaged enrollees, the TTM manual alone did not increase use of and satisfaction with traditional Medicare materials, but generated significant increases in knowledge nonetheless, perhaps by filling a critical information void. The manual contains a great deal of CMS educational information, a plan comparison chart, and recommended CMS support tools and services, but presents them in a gradual, stage-appropriate, and supportive fashion that may be more likely to engage and less likely to intimidate users who may have lower literacy or self-efficacy, or who may have skill deficits.

Among participants who still needed to choose a plan at baseline, we found a discouragingly low rate of stage progression (30 percent) and high rate of stage regression (43 percent) during the followup period. Although the TTM interventions had a significant positive impact on three important dimensions of informed choice—Medicare knowledge, use of CMS educational materials, and choosing a plan—their impact on stage of change for comparing plans was observed only among treatment group participants who had evaluated them for the acceptability survey. Compared to control group participants, treatment group participants who examined the TTM materials were about 50 percent more likely to progress to a later stage of change for comparing plans, and 60 percent less likely to regress to an earlier stage. Comparing plans, as operationally defined, requires far more than Medicare knowledge and information seeking; it also requires an integrated, sustained effort to see the plan comparison process to the end. Through the operational definition of comparing plans, the TTM expert system and manual outline the steps involved in making an informed choice. With the stage-based guidance and feedback, they provide an individualized roadmap to help users progress through the stages and prevent regression to an earlier stage when the task becomes difficult. The materials are unlikely to achieve these aims if enrollees do not use them, or use them just as a reference guide.

This study has several limitations. First, in the baseline survey, administered an average of 7 weeks before study participants reach age 65, 56 percent reported that they had already chosen a Medicare health plan. As a result, over one-half of the intervention group received their materials too late to be maximally useful (e.g., 43 percent of study participants who had already chosen a plan were still in a preaction stage, and thus, had made their choice without carefully weighing their options). We took this into account in some analyses. Ideally, the intervention materials would have been delivered several months earlier, to give users enough time to use them and progress through the stages of change before needing to make a health plan choice.

It is possible that the acceptability survey served as a prompt for new enrollees to use the TTM materials or seek other Medicare information, and thus, may account for a portion of the intervention effect—especially the effect for stage progression and regression observed among individuals who returned the acceptability survey. In real-world applications, new enrollees and beneficiaries may similarly benefit from additional prompts, reminders, and encouragement to use health plan information to make an informed choice.

Some disadvantaged segments of the Medicare population were underrepresented in the study sample, including enrollees without telephones, and enrollees who could not read or write English. In addition, enrollees who dropped out of the study at followup tended to be less knowledgeable about the Medicare Program, less educated, and more likely to be Hispanic than those who completed the followup survey, limiting the generalizeability of the findings.

With these limitations in mind, results suggest that providing individualized guidance and feedback matched to stage of change for comparing plans could be helpful in increasing new enrollees' Medicare knowledge and participation in informed choice. The intervention materials have been revised to incorporate recent changes to the Medicare Program and the increased complexity of the decisions beneficiaries will need to make under the MMA. The challenge is to identify channels for dissemination. One possibility is to integrate stage-based messages into traditional Medicare materials, or to post the manual on the Medicare.gov Web site. Stage-based materials could also be disseminated to older workers in the workplace well before they reach age 65.