Consumer information development and use.

The availability of informational materials to aid consumer health care purchasing decisions is increasing. Organizations developing and disseminating materials include public- and private-sector employers, providers, purchasing cooperatives, State agencies, counseling programs, and accreditation bodies. Based on case study interviews with 24 organizations, we learned that 10 included consumer satisfaction ratings and performance measures based on medical records. An additional four organizations developed materials with consumer satisfaction ratings exclusively. Printed materials were the most common medium used to convey information to consumers. However, other mechanisms for conveying the information were also employed. On the whole, the materials have not been rigorously evaluated. Evaluations are needed to determine if consumers find the information useful and how different individuals prefer to receive the information.

Background

Purpose of the Study

The use of competitive, market-based approaches to delivering health care is increasingly common in the United States. It is estimated that 67 percent of employed Americans were enrolled in some form of managed care in 1995, according to a recent employer survey (Jenson et al., 1996). If this market is to operate successfully, consumers–individual beneficiaries and their group health plan managers-need information about the performance of competing suppliers–health plans and their participating providers.

Like employment-based health insurance, the Medicare and Medicaid programs are relying increasingly on managed care organizations to deliver health care services to their beneficiaries. In 1996, 11 percent of Medicare beneficiaries (nearly 4 million) were enrolled in managed care organizations; enrollment increased 67 percent since 1993. Enrollment of Medicaid beneficiaries in managed care plans is even more substantial. By mid-1995, 32 percent of Medicaid eligibles (11.6 million) were enrolled in managed care plans; enrollment increased 140 percent between 1993 and 1995 (Health Care Financing Administration, 1996).

As the nation's largest purchaser of managed care services, HCFA, which administers the Medicare and Medicaid programs, is moving toward a performance measurement system intended to help beneficiaries choose among competing health plans (Armstead, Elstein, and Gorman, 1995). As part of this effort, HCFA commissioned a study of beneficiaries' information needs and how to develop materials that respond to these needs. The goal of the study was to learn what types of information consumers need to select a health plan and use the plan effectively. The study included three major components: (1) focus groups asking consumers what information they would like to have for health plan decisionmaking; (2) case studies of organizations developing consumer informational materials to learn about current activities; and (3) development and testing of prototype materials for the Medicare and Medicaid population.

This article, which focuses only on the case study component, describes the the state of the art in the design and presentation of information to assist health insurance beneficiaries in choosing among competing health plans. The information about successful approaches already in operation was meant to guide the choice of data items, media, and format in developing prototype materials for Medicare and Medicaid beneficiaries.

Our case study results portray the status of information interventions as of late 1995. The experiences of the case study organizations are valuable for others who are engaged in the rapidly growing field of health plan performance measurement. Many of the organizations with which we met are at the forefront of this effort and their experiences can inform those at earlier stages of implementation.

Organizations developing and disseminating performance information include private sector employers, insurers, providers, purchasing cooperatives, State agencies, counseling programs, and accreditation bodies, among others. The evolution of information materials can be generally traced to employers' demands for accountability from the health plans with which they contract. It is hoped that providing health plan beneficiaries with this type of information will allow them to compare competing health plans and make more informed choices, thus also playing a pivotal role in reforming the health care system (Hibbard, Sofaer, and Jewett, 1996).

The efforts to develop useful information for consumers about the performance of competing health plans have raised a variety of questions: What constitutes performance? What are the dimensions of performance that should be the basis for choice and how should each dimension be measured? What measures are available? If not currently available, what measures are feasible to collect? What information do consumers want? What information will they use? How can consumers be informed about the meaning and value of the measures that professionals think are important? How do the information needs of different types of consumers (e.g., beneficiaries, health plan managers) differ? How can the information be conveyed to consumers in a way that will enable them to use it effectively? Through the case studies, we sought to identify the ways in which leaders in the field have addressed these questions.

Data and Methods

Data Collection Method

The case studies involved on-site, in-depth interviews with key members of organizations that were actively involved in information development. Through the case studies, we identified the process by which the organizations determined consumers' information needs, the methods they used to transmit information to health care consumers, and some information about the effectiveness of their approaches. One overarching goal of the case studies was to identify a list of candidate performance measures, or “quality indicators,” for inclusion in the project's prototype materials and a list of potential communication media.

During the site visit interviews, respondents were asked to describe the process of developing their materials; the evolution of content, materials, and media that were used and those that were not used along with the rationale for these choices; any research or theoretical assumptions that served as the basis for their choices; and any evaluation of the materials that they had conducted. The materials developed by the organizations were collected and reviewed.

Because we felt it would be important to validate the usefulness of the materials developed by these organizations, we also interviewed some of the actual consumers of the materials. As part of two site visits, we asked a small group (n=8) of employees to provide feedback on the materials that their employer had developed. These meetings were held in an informal focus group fashion; participants were recruited with assistance from the employer or by posting flyers. Topics of discussion centered on their level of interest and perceived usefulness of the materials made available to them.

Selection of Case Study Sites and Organizations

Candidate organizations for the case studies were identified through a literature review (Research Triangle Institute, Health Economics Research, Inc., and Benova, 1994), the personal knowledge and contacts of project staff, and the advice of a technical advisory panel convened by HCFA. It was not our intention to create a comprehensive inventory of organizations providing information for health plan choice. Rather, we focused on developments at a few visibly active organizations.

Case study participants were selected by focusing on locations with active managed health care markets, in which several organizations were developing or providing consumer choice information. When feasible, organizations that presented their constituents with a choice of competing health plans were selected. A total of 24 organizations in five regions were interviewed. Table 1 lists the names of the case study organizations as well as the type of organization. We met with four health care providers, three large group purchasing organizations (cooperatives or private employers), three counseling programs, two State agencies, and a melange of other groups including two publishing organizations (Consumers Union and Health Pages) which produce periodic health care magazines. By including many different types of organizations–including both purchasers and providers of care–we obtained multiple perspectives on the value and success of different kinds of content and media in conveying useful concepts to consumers.

Table 1

Case Study Organizations

State	Organization	Organization Type
California	The California Public Employees' Retirement System	Public Employer
	Pacific Business Group on Health	Business Coalition
	Kaiser Permanente of Northern California	Provider
	Managed Risk Medical Insurance Board	Purchasing Cooperative
	Medi-Cal	State Medicaid Agency
	Health Choice, Inc.	Enrollment and Information Contractor
	California Health Information Counseling and Assistance Program	Counseling Program
Minnesota	State of Minnesota Department of Employee Relations	Public Employer
	Park Nicollet Medical Center	Provider/Evaluator
	Business Health Care Action Group	Purchasing Cooperative
	Minnesota Health Data Institute	Health Care Data Institute
	Minnesota Health Insurance Counseling Program	Counseling Program
	HealthPartners	Provider
	United Healthcare	Provider
New York	Health Pages	Consumer Health Magazine
	Consumers Union	Consumer Organization
	Xerox Corporation	Private Employer
Ohio	Cleveland Health Quality Choice	Health Care Coalition
	Cleveland Clinic Foundation	Provider
Washington, DC	National Committee for Quality Assurance	Accreditation Organization
	Federal Employees Health Benefits Program	Public Employer
	United Senior Health Cooperative	Consumer Organization
	Health Insurance Counseling Project	Counseling Program
	Maryland Health Care Access and Cost Commission	State Health Care Agency

SOURCE: Original data collected by Research Triangle Institute and Health Economics Research, Inc., 1995.

Findings

Our findings are discussed in the following format. First, we briefly review the process organizations undertook to learn about consumers' information needs. Next, we provide an overview of the types of performance measures found in the consumer materials, highlighting the most frequently presented measures and some of the very active organizations. This is followed by a discussion of the media used to transmit the information to consumers.

Eight of the 24 organizations used focus groups to ask consumers directly what information they want when choosing a health plan and how they would like the information presented. A handful of organizations have learned about consumer information needs through readership or employee surveys following the dissemination of their materials. However, the surveys did not typically use scientific sampling methods and have low response rates. Organizations that directly counsel individuals, e.g., Medicare Information, Counseling, and Assistance (ICA) programs, generally disseminate information that responds to the major issues raised by beneficiaries. On the whole, organizations learned about what information consumers need to choose a health plan through years of personal and organizational experience. Some did not conduct any formal research to make this determination or confirm their impressions.

Fifteen of the 24 case study organizations interviewed had already produced information materials in some format (e.g., print, seminar, computer-based). The majority of the performance reports we encountered were specifically intended to be used by employees or individual consumers for health plan choice. A handful were used primarily for internal quality assurance or were oriented toward employers or large group purchasers.

Clearly, information that characterizes basic health plan structure–such as the benefits offered and the premiums charged–was the most common type of information included in consumer information materials. These data are traditionally found in summary plan descriptions and premium rate charts. Distribution of this information was essentially universal among health plans and purchasing cooperatives because they want their constituents to understand what benefits the plan(s) cover and the consumer out-of-pocket responsibility.

Because this type of information provides a necessary foundation for understanding health plan choices, it is not surprising that most organizations opted to include it. Xerox Corporation, however, decided to discontinue distribution of benefit coverage summaries beginning in 1996 because it was believed that employees were viewing them as binding contracts, when they were intended to simply summarize plan coverage. Xerox employees must now obtain benefit coverage information directly from the health plan.

Performance Measures

We classified the performance measures found in the materials into two major categories: (1) consumer satisfaction ratings and (2) process and outcome measures.

Consumer Satisfaction Ratings

Consumer satisfaction ratings are generated from surveys of health plan members who are queried about their perceptions and level of satisfaction with the health plan to which they belong or from which they recently disenrolled. The ratings are based on subjective impressions of individuals' health care experiences and expectations, and can reflect the values that individuals place on health care and the health care system. This is often the only type of information available about the interpersonal process of health care delivery and is recognized as an important measure of quality of care (Davies and Ware, 1988). We found that the dimensions most commonly addressed by the surveys included ratings of access and quality of care, communication or interpersonal skills of the providers and staff, experiences with the physician/hospital, preventive and other services, and overall satisfaction with the plan.

Of the 24 case study organizations, 14 included data from consumer surveys in their information materials. These organizations include the California Public Employees' Retirement System (CalPERS), Pacific Business Group on Health (PBGH), Kaiser Permanente of Northern California, the Minnesota Department of Employee Relations, the Minnesota Health Data Institute (MHDI), Health Partners, United HealthCare, Health Pages, Consumers Union, Xerox Corporation, Cleveland Health Care Quality Choice (CHQC), the Cleveland Clinic Foundation (CCF), National Committee on Quality Assurance (NCQA), and the Federal Employees Health Benefit Plan (FEHBP).

CalPERS administers the health benefit program for nearly 1 million State employees, retirees, and their families. In 1991, CalPERS' members began receiving information regarding how other members rated their current health plan based on an internal survey of CalPERS members. By 1995, members received both a general heath plan guide as well as a health plan quality report that rated 17 health plans on consumer satisfaction and technical quality of care. Satisfaction ratings for the 1995 report were based on a survey of a representative sample of 27,000 CalPERS members, of whom 48 percent responded. The survey was conducted as part of a multi-employer group effort led by the PBGH. Plans were ranked based on a score of 0 to 100 and were presented in quartiles in the quality report.

The FEHBP, managed by the U.S. Office of Personnel Management (OPM), is the largest employer-sponsored health insurance program in the world. In 1994, OPM cooperated with the Center for Study of Services to conduct the first satisfaction survey of Federal employees who were enrolled in FEHBP health plans. In total, 261 plans participated, with an overall response rate of 62 percent. Separate surveys for members of prepaid and fee-for-service (FFS) plans were conducted. The following dimensions of care were rated: access to medical care; quality of care; doctors available through the plan; coverage; and information provided by the plan, customer service, and simplicity of paperwork. Results of the satisfaction survey were made available to Federal employees in the form of an eight-page booklet intended to help them compare health plans nationwide.

Table 2 lists the consumer satisfaction measures we encountered and the number of case study organizations that were using them. The most commonly reported survey-based performance measures were:

Table 2

Consumer Satisfaction Ratings Used in Information Materials

Measure	Number of Organizations That Included the Measure (n=14)
Access to Care
Waiting Time for Appointment1	4
Waiting Time at a Physician's Office1	4
Convenience of Physician Services	2
Choice of Doctors	3
Access to Care1	4
Access to After Hours, Urgent Care, Emergency Care	3
Ability to See a Personal Physician When Care Is Needed	2
Communication/Interpersonal Skills
Personal Treatment During Physician Services1	4
Doctor's or Nurse Practitioner's Personal Interest in the Patient and Medical Condition (for Pediatrics and Maternity Care)	2
Experience With the Physician/Hospital
Overall Satisfaction With Provider1	4
Would Recommend Hospital to Family and Friends	3
Overall Satisfaction With Therapist	2
Quality
Quality of Physician Care	3
Overall Satisfaction With Health Care1	7
Overall Satisfaction With Hospital	3
Technical Quality of Adults' Care	2
Technical Quality of Children's Care	2
Quality of Service at Primary Care Office or Clinic	3
Hospitalized Patients Experience at Hospital	2
Overall Experience as Obstetrics Patient	2
Experience With the Plan
Range of Services Covered1	5
Overall Satisfaction With Insurance Plan or HMO1	8
Would Recommend Insurance Plan to Others	3
Quality of Customer Service at the Health Plan	3
Satisfaction With HMO's Responsiveness to Questions and Complaints	3
Information Provided by the Plan, Customer Service, and Simplicity of Paperwork	2
Would Renew Their Membership	2
Courtesy and Helpfulness of Non-Medical Staff	2

Most commonly used measures.

NOTES: Only measures used by at least two organizations are included. HMO is health maintenance organization.

SOURCE: Original data collected by Research Triangle Institute and Health Economics Research, 1995.

Satisfaction with waiting time for an appointment and in the physician's office.

Satisfaction with access to care.

Satisfaction with personal treatment during physician services.

Overall satisfaction with the provider.

Overall satisfaction with health care.

Satisfaction with the range of services covered.

Overall satisfaction with the insurance plan or health maintenance organization (HMO).

The majority of organizations used 6 or fewer survey measures in their materials; 14 was the maximum number of measures included into one set of materials. Performance reports designed for large group purchasers tend to use more measures. In many cases, the measures were composites of several other measures used to create an overall rating. For example, the PBGH report card included a measure representing the convenience of the plan which is an average of the satisfaction ratings for the location of the doctor's office, the hours when appointments are available, and the length of time spent waiting for an appointment. Composite measures are used to minimize the number of data elements reported, to keep the information simple, and to avoid overwhelming the audience.

No standard survey instrument was being used by the case study organizations. Rather, a variety of very different surveys was used to collect data on plan satisfaction. Some of the instruments were based on surveys that have been under development for years. For example, the Minnesota Department of Employee Relations (DOER) relied on the (former) Group Health Association of America instrument as a starting point for their survey. The DOER negotiates and manages employee benefits for over 60,000 government-related employees, including 44,000 State employees. It has been conducting bi-annual consumer satisfaction surveys of its employees since 1991 in order to provide more information during the annual open enrollment period. The most recent survey was administered over the telephone to 400 respondents in each of the 6 plans. Separate quality of care ratings for adults' primary care, children's primary care, and specialty care were collected. Results were presented in an eight page color foldout using bar graphs to display satisfaction levels.

Different sampling designs and survey methods were used by the different organizations in the study. Therefore, even if similar questions were asked, comparisons across organizations' materials were not possible. However, two State governments have made advances in providing comparable data on a Statewide basis. The first was Minnesota. In addition, the State of Maryland passed legislation in 1993 charging the Health Care Access and Cost Commission (HCACC), a new independent commission functioning independently within the Department of Health and Hygiene, to implement a system to evaluate and compare the quality of care outcomes and performance measures of HMOenrollees in the State and to disseminate this information to State residents. The State developed a standard survey instrument to collect satisfaction information from HMO enrollees in the State on an annual mandatory basis. Interestingly, they have also developed a standard survey instrument to collect data from Maryland physicians about their perceptions and level of satisfaction with Maryland HMOs. Beginning in 1997, enrollee satisfaction information will be included in a report card to assist consumers in comparing Maryland HMOs (Lubalin et al., 1996).

Process and Outcome Measures

While consumer satisfaction ratings reflect the interpersonal quality of care, process and outcome measures reflect the technical quality of care provided. Technical quality “depends on the knowledge and judgment used in arriving at the appropriate strategies of care and on skill in implementing those strategies (Donabedian, 1988). Performance measures reflecting the process and outcomes of health care delivery are derived from automated medical records, claims data, encounter reports, and accounting data. We encountered Screening/Preventive Care, Utilization, and Outcomes measures.

Ten of the case study organizations used this type of measure in their materials. They include CalPERS, PBGH, Kaiser Permanente of Northern California, Health Partners, United Health Care, Health Pages, Xerox Corporation, CCF, CHQC, and NCQA.

Kaiser Permanente of Northern California distributed its first health care report card in 1993 and its second in 1995. More than 100 performance measures were presented, including satisfaction ratings based on a survey of over 6,000 Kaiser members and Health Employer Data Information Set (HEDIS) indicators. The 1993 report was divided into seven sections organized around the following parts of the life cycle or common diseases: childhood health, maternal care, cardiovascular disease, cancer, common surgical procedures, other adult health, mental health, and substance abuse. Two new categories–diabetes and HIV–were added in the 1995 report. Benchmarks, many based on the disease prevention and health promotion goals of Healthy People 2000 and State health department studies and reports, were included in the report card where applicable.

Health Pages magazine, which is targeted at a diverse readership, seeks to provide people with the tools to become educated consumers. The magazine is weighted toward information about cost and benefit comparisons of different local health plans, but has stories on more regularly occurring topics such as managed care. Process and outcome measures can be found in some editions. By fall 1995, Health Pages magazine was published in nine metropolitan areas.

The most common type of process and outcome measures found were those that comprise the HEDIS data set developed by NCQA. Because HEDIS is weighted toward screening/preventive care measures, it was not surprising that this type of indicator was found in most of the materials. As shown in Table 3, the most common screening/preventive care measures included:

Table 3

Process and Outcome Measures Used in Information Materials

Measure	Number of Organizations That Included the Measure (n=10)
Screening/Preventive Care
Percent of Children Immunized1	9
Percent of Adults Who Had Their Cholesterol Screened1	8
Percent of Women Who Received a Mammogram1	9
Percent of Women Who Received a PAP Smear1	9
Utilization
Percent of Enrollees Who Visited a Practitioner	4
Percent of Pregnant Women Who Received Prenatal Care in the First Trimester1	7
Asthma Inpatient Admission and Emergency Room Rates1	6
Percent of Enrollees Who Received a Diabetic Retinal Exam1	9
Hysterectomy Rate	2
Inpatient Discharge Rates for AMI, CABG, CVA, Transentistemic Heart Attack, Diabetes, Pneumonia/Pleurisy, and Asthma/Bronchitis	2
General Utilization Measures (Inpatient Admissions per 1,000, Inpatient Days per 1,000, Length of Stay, Emergency Room Rate per 1,000)	4
Percent With a Follow-up Visit After Hospitalization for a Major Affective Disorder	5
Average Length of Stay for Selected General Medical/General Surgical Diagnosis	1
Ambulatory Care Visit Rate	2
Utilization Rates for Selected Procedures	2
Percent of Women with Caesarean Section Following Vaginal Births1	6
Caesarean Section Delivery Rate	4
Chemical Dependency Readmission Rate	3
Average Length of Stay for Intensive Care Patients	2
Inpatient Utilization-General Hospital/Acute Care	3
Inpatient Discharge and Emergency Room Rates for Children	2
Number of Inpatient Days	4
Average Length of Stay for Deliveries	2
Average Length of Stay for Laminectomies	2
Readmission Rates for Major Affective Disorder	2
Complex Newborn Rate	2
Inpatient Hospital Mortality Rate for Low-Birth-Weight Babies	2
Outcomes
Percent of Low-Birth-Weight Births (of Live Births)1	7
Mortality Rate for AMI, CABG, and CVA Patients (Within 3 Days of Admission for AMI and CABG)	4
Heart Disease Mortality Rate	2
Hypertension Treatment Effectiveness (Normal Blood Pressure After 1 Year)	2
Inpatient Rate for AMI Patient	2

Most commonly used measures.

NOTES: AMI is acute mycardial infarction. CABG is coronary artery bypass graft. CVA is cerebrovascular accident.

SOURCE: Original data collected by Research Triangle Institute and Health Economics Research, 1995.

Percent of children immunized.

Percent of women who received a mammogram.

Percent of adults who had their cholesterol level checked.

The most common utilization measures included in the case study materials were the “percent of diabetics who received a retinal exam” and the “percent of pregnant women who delivered through cesarean section following a vaginal birth.”

While utilization indicators measure contacts with the health care system, the outcome indicators reflect the health status consequences of those contacts. The “percent of low-birth-weight births” was the most common outcome measure used. However, outcome indicators were generally sparse in the consumer materials, mainly as a result of the lack of agreement on a risk-adjustment methodology to account for underlying characteristics in the population of the health plan. Without appropriate risk-adjustment, measures derived from medical records may distort plan performance (Luft, 1996). Most case study organizations acknowledged this limitation and plan to adjust for patient risk when there is greater agreement in the health policy community regarding the method (s) to use.

We found two organizations that were already risk-adjusting their ratings: CHQC and MHDI. CHQC risk adjusts their indicators of medical/surgical and obstetric outcomes using its own risk-adjustment system known as Cleveland Hospital Outcome Indicators. MHDI is a public-private partnership created by the Minnesota legislature in 1993 to design and implement a statewide health care data system to support the needs of consumers, purchasers, providers, and policymakers. MHDI's 1995 consumer satisfaction survey interviewed over 17,500 enrollees in 46 health plans in both the public and private sector. MHDI statistically adjusted their survey responses for enrollee age, sex, education, self-reported health status, and region and weighted comparisons to account for differences in enrollment size across individual health plans.

Finally, several of the organizations we studied focused their efforts on materials to specifically assist consumers in using the health care system, i.e., choosing providers and therapies, rather than on materials to assist in the choice among competing health plans. Four were particularly noteworthy: CHQC, CCF, Health Partners, and Health Pages. CHQC developed and disseminated comparative information on the quality of care and patient satisfaction in the Cleveland area hospitals; CCF developed condition-specific “consumer guides” to support patient use of quality indicators in selecting a hospital; Health Pages published physician and hospital directories in different markets; and Health Partners developed interactive software to assist members in choosing a physician. Their computer model, which was introduced in September 1995 and is called the Consumer Choice SystemSM, includes information on facility location and hours, physician profiles, member satisfaction survey results, and plan-level HEDIS measures.

Modes of Communication

There are several different options for presenting health plan information to consumers, in terms of the medium (e.g., printed report, video, computer model, in-person seminar) used as well as how the comparative data are presented (e.g., in charts, graphs, text, or some combination of these). Some consumer materials include information on only one health plan (with and without comparisons to external standards or benchmarks), while others provide comparative data on multiple health plans. Often the same materials are prepared in multiple languages.

Focus groups of both privately and publicly insured individuals have indicated a general preference toward personal communication of the information, such as individual counseling sessions (Gibbs, 1995). The desire for this type of approach is undoubtedly a function of the need for feedback on one's personal health insurance situation, which may be more useful than general information. However, this mode of communication is costly, time consuming, and reaches relatively few people. Videos have the advantage of portability and mass exposure. Computer-based programs offer interactive opportunities for users to follow a specific information path based on their interests and priorities. Although counseling, videos, and computer presentations have their appeal, consumers generally want information in print format, so that they can take it home to share with family and review it at their leisure (Gibbs, 1995).

The printed report card was the most popular format (used by 9 of the 15 organizations that produced materials) for the consumer information materials we reviewed (Table 4). Only one organization, Health Partners, was using interactive computer software to help current members use the health plan. It operates on a laptop computer that uses touch-screen technology. Although it was designed for non-Medicare members, Health Partners expects to adapt it to the Medicare population once the prototype has been sufficiently developed. Consumer Reports information, produced by Consumers Union, can also be accessed on-line through the Internet.

Table 4

Modes of Communication Used by Case Study Organizations

Mode	Organization
Written Report Card	California Public Employees Retirement System
	Pacific Business Group on Health
	Kaiser Permanente of Northern California
	Department of Employee Relations, State of Minnesota
	Minnesota Health Data Institute
	United Healthcare
	Cleveland Health Quality Choice
	Xerox Corporation
	Office of Personnel Management
Presentation/Seminar	MediCal
	Health Choice, Inc.
Interactive Computer Software	Health Partners
Magazines	Consumers Union
	Health Pages
Internet	Consumers Union
Other	Minnesota Health Insurance Counseling Program (HICAP)1
	Cleveland Clinic Foundation (CCF)2

HICAP developed a plan comparison chart and articles in newspaper format.

CCF developed written quality guides bound as 6.5″ × 8.5″ booklets ranging from 24-28 pages in length.

SOURCE: Original data collected by Research Triangle Institute and Health Economics Research Inc., 1995.

MediCal and Health Choice, Inc. were using live presentations based on predetermined scripts to bring information to consumers. Because MediCal's and Health Choice's approach was oriented toward Medicaid enrollees with a low-literacy level, it was believed that a combination of a verbal presentation and printed materials would provide the best means for communicating health plan choice information. Health Choice, Inc. also operates a telephone call-in center.

For most case study organizations, the mode of communication was determined by the organization's staff who generally rely on journalistic and publishing experience with general consumer information to design the format and presentation for health care information. Developers struggled with the issue of how much information they should include without overwhelming the reader. This was a particular concern for organizations using print materials. A related issue is how to present individual performance measures in an understandable fashion. The dominant approach was a combination of text and graphics to illustrate the information, followed by a combination of text with percentages reflecting rating scores in a table format.

Evaluation Research

Overall, the materials developed have not undergone rigorous evaluations; most have not been evaluated at all (in part because they are so new in many cases). A handful of the organizations were considering evaluations in the future, however. For those that were evaluated, focus groups was the most common method used. Limited cognitive testing–which applies methods developed by cognitive scientists to understand how people react to draft materials–was completed. Cognitive testing is especially effective in exploring comprehension, memory, decisionmaking, and motivation. One or two organizations changed their existing materials based on the information they had received during this process. For example, Xerox substituted their prenatal care indicator for an indicator showing the frequency of cesarean sections because it was viewed as more salient to their employees.

As part of the case study process, we held two informal focus groups with employees in two organizations (CalPERS and FEHBP) to discuss the information materials developed by their employers. Both employers' materials were viewed favorably, but were more pertinent to those who were considering changing plans or who were new to the organization. Nearly all participants said that they would refer to the materials if they were considering a change.

The level of interest in the specific information was mixed. The group of Federal employees displayed considerable skepticism about the survey-based measures and wanted to know who collected and reported the data. A few voiced concern that there were only small differences in the ratings across plans. The satisfaction ratings were somewhat less popular among the group of State-level employees; one participant stated that he did not care what other people thought about the health plan because “choosing a health plan is a very personal decision.”

It was evident that the groups needed some additional explanation of how to interpret the charts and graphs. The difficulty with interpretation may result from a lack of understanding about what the indicators mean (e.g., “Is a high rating good or bad?”) and what they are supposed to measure. It could also be related to the particular formats used, since several participants offered suggestions on how to change the materials so they would be easier to read. Consumers may also have difficulty comprehending the role of the health plan in keeping its members healthy.

Lessons Learned

These case studies demonstrate that, although the actual use of health plan performance reports is not yet widespread, several organizations are committed to them and have made significant advances in choosing data, media, and formats. Since formal evaluations were few and fairly limited, we identified little in the way of empirical results about the success or failure of these information interventions. Nevertheless, we did learn several anecdotal lessons about developing consumer information materials.

Complexity

Most beneficiaries are still unfamiliar with health plan performance measures and their meaning. One implication of this is that performance reports should be kept simple initially and only be expanded to include more measures after consumers become familiar with the purpose and interpretation of the initial information. This is the way most performance reports have evolved.

Format

The volume of information and the degree of detail that beneficiaries can understand and make use of will vary considerably within any population. The “layering” of information is viewed as the best solution to the problem of how much information to provide. Layering involves presenting the information in two or three different ways, some with progressively more detail. The decision to layer information makes it even more important to limit the number of measures included in the report, because layering can multiply the amount of information.

Using composite measures is one form of layering. This is consistent with the objective of using few measures and keeping the presentation simple. Some report cards that use composite measures provide a short list of the data items that were included in the composite statistic without presenting the actual data for those underlying items. This seems to be a useful compromise that minimizes the amount of data beneficiaries have to process, yet provides additional explanation to those who want it.

Choice of Measures

In comparing performance measures among organizations, we discovered no consistent pattern that clearly points to a set of the best measures. Not only do the reported measures themselves vary, but composite measures that appear to be similar in name are often developed from different underlying dimensions. Some organizations reported that their constituents preferred satisfaction data from consumer surveys to HEDIS-type performance measures. The Business Health Care Action Group and Health Partners in Minnesota indicated that consumers had no interest in HEDIS measures, based on their experience to date. Recent research by Hibbard, Sofaer, and Jewett (1996) also indicates that patient ratings of quality and satisfaction were viewed as most useful. However, satisfaction could simply be more salient because it is a more familiar concept.

To date, there is no clear indication of which measures should be included in consumer materials and that are used for actual choice. Without this knowledge, materials may continue to include measures which are not helpful. For example, the low-birth-weight indicator was found to be least useful according to focus groups and a consumer survey conducted by Hibbard, Sofaer, and Jewett (1996), but it was found in the majority of materials. The authors believed that the measure is poorly understood in terms of its basic definition or interpretation, i.e., comprehending the meaning of a rate.

Explanation of Measures

Consumers' limited experience with performance reports also implies that the presentation of data needs to be accompanied by simple explanations of the meaning and interpretation of the data. While some consumers may be able to use the materials independently, it should be expected that a large proportion of users will need assistance in understanding and interpreting the information provided. Larger corporate purchasers of health care are generally a more sophisticated audience and probably need less explanatory information than individual consumers. Many developers hope that consumers will become more skilled at interpreting this information as their experience with it grows.

Media

Printed materials are the essential starting point for performance reports. They are comparatively simple and inexpensive. Even Xerox, a high technology firm, rejected a computer-based system in favor of a simple printed report because many employees do not have access to personal computers. Videos and seminars can be used as companions to the printed materials for complementing and reinforcing the message. Computer-based interventions could also be used to complement written material, although computers will provide the ability for more sophisticated decision support as computer-literacy spreads and the sophistication of decision support systems improves.

For some individuals, interactive systems, such as those under development at HealthPartners and under several Small Business Innovative Research (SBIR) grants from the Agency for Health Care Policy and Research (AHCPR), may be particularly valuable. The value is not in the technology per se, but in the format that enables users to specify preferences and come out at the end with a designated selection or at least a list of a few leading candidates.

Computer technology facilitates this format, but it can also be presented using mechanical, printed devices (e.g., worksheets). For the Medicare and Medicaid populations, to whom computers could be unfamiliar, consideration should be given to the printed alternative as well as to the computer-based system. If computer-based decision support systems are used, then it would be useful to build them around a supplemental videotape presentation, since older or less educated persons may be more familiar with video than with computers.

Future of Consumer Information

Most of the consumer information activities were at an early stage of development at the time of our case studies. Progress and collaborative efforts continue, but we are several years away from widespread development and use of information materials. The activities that are currently underway will serve as the foundation of future materials if they are deemed useful by consumers. It is paramount that we learn which types of information are of interest and of use to different types of consumers by evaluating their effectiveness.

The use of performance measures and consumer information materials is being pioneered for the privately insured population under 65 years of age, although there are some activities for Medicare and Medicaid beneficiaries. The materials available to these groups generally include only basic descriptive information, such as benefits and premiums, without quality-of-care information, in part because the need for risk adjustment of utilization and outcome measures is even greater with the populations served by these programs. As more Medicaid programs adopt managed care, they are also incorporating quality assurance programs, many of which include collecting encounter data and calculating HEDIS measures. The Medicare and Medicaid populations also require special consideration with respect to presentation style, such as reading comprehension level and print size. As managed care enrollment for the publicly insured grows, the need for comparative health plan information will increase so that consumers can make informed decisions.

Several challenges face the future of consumer materials development and use. The major challenge for both consumer satisfaction ratings and process and outcome measures is the lack of standardization in the performance measures selected for inclusion, the algorithms used to define the measures, and the data collection methods. Survey sampling methods, data collection modes, survey questions, and analytic methods need to be standard to make comparisons among health plans useful. This standardization requires a significant commitment of financial resources to surveys with samples and response rates that are large enough to make reliable estimates separately for each health plan. It also requires significant cooperation from health plans. Still, unless the survey data are collected by an independent organization, consumers may remain skeptical about their validity.

The difficulty in obtaining resources and cooperation from an ever growing number of health plans present significant potential impediments to the growth of performance reports for health plan choice. Where there is standardization of methods currently, it is primarily at the local or purchasing organization level. However, two States–Minnesota and Maryland–have become leaders in designing and conducting standard consumer surveys. This suggests that regulation mandating standards and/or the commitment of public agencies to the surveys may be needed to achieve the necessary standardization.

The Consumer Assessment of Health Plans Study (CAHPS), funded by AHCPR, is another step in this direction. CAHPS is developing and testing methods for measuring consumers' satisfaction with their health plans and ways to communicate the results to consumers. For process and outcome measures, NCQA's accreditation process offers some degree of assurance that specific algorithms are used to calculate indicators, yet these data are also not currently audited. Finally, the prototypes developed for HCFA during the project (handbooks and videos) include performance measures reflecting quality of care information for Medicare and Medicaid beneficiaries.

There is likely to be a role for the government in meeting some of these challenges. The key will be to strike an acceptable balance between public and private involvement. This will probably not occur until there is greater consensus on which performance measures consumers want and need and a standard consumer satisfaction survey is well-accepted. Financial resources will also be needed not only to collect standard data but for evaluation of the choices of data items, media, and format and whether or not providing this kind of information actually influences health plan choice. As the materials and consumers become more sophisticated, it will be even more imperative to evaluate their impact on health care decisionmaking. The ultimate test of effectiveness is whether consumers will accept and incorporate the materials as part of their health care decisionmaking process.