Assessing process of care under capitated and fee-for-service Medicare.

In 1986 more than 770,000 Medicare beneficiaries were enrolled in risk-based health plans that had very little experience in providing medical care to elderly patients. This article addresses one major facet of the National Medicare Competition Evaluation of capitated versus fee-for-service delivery of Medicare benefits, sponsored by the Health Care Financing Administration: the assessment of the quality of the process of care delivered to Medicare enrollees. The rationale, design, and analysis plans for this subpart of the larger evaluation and its interrelationships with the other components of the project are described.

Introduction

The National Medicare Competition Evaluation (NMCE) is a comprehensive study of Medicare risk-based plans sponsored by the Health Care Financing Administration (HCFA). The NMC Evaluation began in 1983 when the first set of risk plans, usually called health maintenance organizations (HMO's), were enrolling Medicare beneficiaries under demonstration contracts with HCFA. The study continued with several modifications when the demonstrations were phased out and the plans converted to the new Tax Equity and Fiscal Responsibility Act of 1982 (TEFRA) regulations in Spring 1985. The goal of the NMCE, is to measure the effects of risk contracting on several different aspects of health services for Medicare beneficiaries, including:

Use and cost of services delivered.

Beneficiary choice of capitated versus traditional delivery system.

Biased selection in patterns of beneficiary enrollment.

Quality of care delivered.

Because of the significant health care needs of the Medicare population and program incentives to retain any cost "savings," quality of care was identified as an important issue. The design of this component of the NMCE follows the conceptualization of quality of care assessment introduced by Donabedian (1969), focusing on structural, process, and outcome dimensions. The evaluation element addressing the structural basis of quality of care focuses primarily on a comparative description of the quality assurance programs in each HMO. Process of care is assessed through several means, including medical record reviews. Beneficiary satisfaction with care and health status changes are examined as indicators of outcomes of care. Together the three elements of the quality of care component provide a comprehensive comparison of the quality of care received by Medicare beneficiaries under HMO and fee-for-service (FFS) delivery systems.

This article provides an overview of the design for the more complex element of the quality evaluation: the process of care assessment. In the sections that follow, we discuss several key issues in evaluating the quality of the process of care in Medicare HMO's relative to FFS alternatives, the conceptual framework that was developed to guide the study, the design of the data collection, and major issues in data analysis.

Managed care and the elderly

Based on our knowledge of HMO's (for reviews, see Luft, 1981; and Wolinsky, 1980) and HCFA's early experience with Medicare risk-based demonstrations, normal service use patterns are expected to change for those enrolling in Medicare HMO's. Previous studies have shown lower utilization rates, particularly inpatient use, in HMO's. Research also suggests that HMO's generally provide care that is at least of equal quality to that provided in the FFS setting (Cunningham and Williamson, 1980; Francis, Polissar, and Lorenz, 1984; Yelin, Shearn, and Epstein, 1986). These findings, however, reflect HMO experience that is primarily based on employed enrollees and their dependents, including relatively few Medicare beneficiaries.

HMO's are managed-care systems providing a specified set of services to an enrolled population for a fixed fee per period of time. The HMO, rather than a separate insurer (in this case the Medicare program) assumes the risk of providing needed care for the premium paid. HMO's have, therefore, an incentive to deliver care in a cost-effective manner, providing no more than professional standards require and the least intensive services that will manage the health condition while minimizing future expenditures (Luft, 1981). To this end, HMO's use various means to control the cost of care while maintaining quality, including procedures to channel access to care and financial incentives affecting the patient and the provider (Brown, 1983).

Use of these methods to manage the care delivered to Medicare beneficiaries could enhance or reduce the quality of care relative to that found in the FFS setting. More organized relationships among providers may contribute to the continuity and coordination needed for chronic care. Moreover, unnecessary diagnostic testing could be reduced by better communication among providers, thus benefiting the elderly who are often at greater risk of adverse effects of medical interventions (Bonanno and Wetle, 1984; Kennie, 1983). Alternatively, transfer of too much risk to the providers may create adverse incentives for underservice, particularly in the area of expensive inpatient care (General Accounting Office, 1986). Overly restrictive controls or adverse financial incentives to limit referrals to specialists may result in primary care providers managing care beyond their scope of practice.

The risk of adverse effects from HMO care-management methods applied to the Medicare population may be greater than for the younger, healthier populations previously served by HMO's. A managed-care system is new to most Medicare enrollees and may be difficult for them to learn and use. Studies of real-life problem-solving abilities (Denny and Palmer, 1981; Milligan et al., 1984) and the physician-patient relationship (Greene et al., 1986) suggest that older, sicker users of services may be at a disadvantage in managing a new, structured system of medical care.

The substitution of ambulatory for inpatient care may be more difficult to accomplish in a Medicare population. The disabilities of many elderly patients are exacerbated when they become ill, and the social resources on which they depend to compensate for these disabilities may be strained by home care needs. The controlled environment that can be achieved for younger, healthier patients with substantial support systems may not be possible for frail, elderly patients. The complexities of caring for the multiple conditions often found in older patients may also defy ambulatory management. A recent analysis of Medicare data provides support for this possibility, showing an increasing rate of utilization of inpatient services with age, but an inverted, J-shaped pattern of use of physician visits, with the decline in ambulatory use beginning at the age of approximately 80 years (Wolinsky, Mosely, and Coe, 1986).

The few studies of Medicare beneficiaries enrolled in HMO's are consistent with the larger body of research, suggesting that HMO's provide more continuity (i.e., seeing the same provider, having a scheduled return appointment, and being seen in the past 6 months) (German, Skinner, and Shapiro, 1976); incur fewer inpatient days; and have lower costs to the Medicare program than FFS providers (Weil, 1976; Greenlick et al., 1983).

Research in one large HMO suggests, however, that there are significant differences in the determinants of utilization among patients over the age of 65 years compared with younger enrollees (Hibbard and Pope, 1986). Although medical need variables appear equally important, the age groups respond differently to other factors. Attitudes about health and health care are more important to the use patterns of younger HMO members, while characteristics of the HMO explain more about use by older members. For example, the elderly were more sensitive than younger patients to seeing the same provider at each visit and to travel time to the plan.

The recent finding of poorer health outcomes for low-income, initially sick enrollees in an HMO versus an FFS setting (Ware et al., 1986) may also be relevant. Although this finding is not based on elderly individuals, the same factors noted related to income may also apply to the elderly. For example, it has been suggested that both the poor and the elderly are less able to manage the constraints on access often faced in HMO's (Luft, 1981). Ware and his colleagues also found that Medicaid participants using an HMO were less likely to report seeing the same physician at each visit than were participants using FFS care. As noted above, seeing the same provider may be an important determinant of utilization for older HMO enrollees.

Such differences in the determinants of utilization and health outcomes for subgroups of HMO enrollees in combination with the changes in financial incentives created by the new risk-based contracts reinforce the need to study the effects of HMO methods on an elderly population.

Evaluation design

Research on the quality of the process of care in Medicare HMO's must take into account the major sources of variation that could affect quality. These sources include the characteristics of managed-care settings versus FFS settings, the range of health conditions that burden the elderly, and the kinds of medical care that are provided to Medicare beneficiaries.

Delivery system characteristics

The delivery systems to be studied are highly variable. Some of the sources of variation may be of interest to policymakers as objects of control through regulation; others, though not amenable to manipulation, are important as potential confounders of effects that could be controlled. Given the scale of this study and the existing diversity, only the most important sources of variation are addressed.

Geographic region

Medicare HMO's are being implemented nationwide, thus any evaluation of the quality of their care must address the geographic diversity. Research has demonstrated that there is significant variation in medical practice from one region to another. (A recent review can be found in Health Affairs, Summer 1984, Vol. 3, No. 2). Although regional variation is not a likely subject of policy control, it is a major source of influence that could cloud our ability to detect real differences that might exist between the managed-care settings and those of FFS. We must, therefore, capture the regional diversity in a manner that will allow its removal from the effects of interest. This is accomplished in the NMCE by sampling HMO's and FFS practices equally from all major regions of the country.

Reimbursement arrangement

Not only are there significant differences in the reimbursement arrangements and practice incentives between the FFS sector and the risk-based plans, but also there is substantial variation in these arrangements among and within types of HMO's. Because the focus of the evaluation and much current policymaking is on the new risk-based plans, variation within HMO's rather than within the FFS setting is more important. Although FFS care is examined in comparison to the Medicare HMO's, only regional differences in the former are controlled by design. Information on providers, e.g., specialty, board certification, and the organizational arrangement of practice, are collected to compare with similar characteristics in the HMO's.

The one variable that captures the most variation among HMO's is organizational form. Relationships among providers depend on the organization of the plan. Evidence suggests that staff and group models involve greater peer interaction and may be more effective in reducing utilization than models that depend on physicians operating in relative isolation from one another (Held and Reinhardt, 1979; Wolinsky and Marder, 1982). Models with greater peer interaction may also provide more support for professional standards in the face of incentives to reduce care giving (LoGerfo et al., 1979; Rhee, 1983).

Plans reimburse participating physicians using a variety of means involving degrees of risk sharing from essentially none to most of the risk being transferred from the plan to the physician. Maintenance of more traditional forms of payment to physicians, e.g., FFS, may not provide significant incentives to alter practice patterns (Martin, Ehreth, and Geving, 1985; Wolinsky and Marder, 1982). Alternatively, transfer of too much risk to individual providers may create adverse incentives to underserve. Risk is affected not only by the kinds and amount of care that are covered by a fixed payment, but also by the size of the caseload being managed under the fee structure. Risk can become excessive and incentives to underserve may ensue if the panel of enrollees is too small to balance the effects of a few heavy users (General Accounting Office, 1986). The financial incentives for physicians tend to vary by HMO organizational type. Staff and some group models pay their physicians on a salaried basis. It is primarily in the independent practice associations (IPA's) and the networks with large IPA components that significant transfer of risk to the physician is likely to take place. It is also in the IPA's that one may find payment structures that are more traditional.

Although the major organizational models display significant within-group diversity in both professional relationships among providers and in financial arrangements, they do represent important distinctions. Group and staff models both tend to have more highly structured and frequent contacts among participating providers than do IPA and network models. The former are also more likely to share similar financial arrangements with one another than with the other models. IPA's and networks have a wide range of financial arrangements, from little risk-sharing with a modified FFS system to a significant transfer of risk to the physician with capitation for both primary and specialty care. Aggregating staff and group models versus IPA and network models into two major categories of HMO captures major differences. These organizational differences are potentially related to practice patterns that could affect the quality of care delivered to Medicare beneficiaries and that are also amenable to policy control. Their control is important as a major design element of the study.

Rate of growth

Some Medicare plans have grown very slowly, adding fewer than 100 enrollees per month. Others have recruited as many as 1,500 enrollees per month. Rapid rates of growth may strain the organization's ability to manage care effectively for new Medicare enrollees. Rate of growth and size of Medicare enrollment could be regulated if found to have significant effects on quality of care. Information on growth is available in advance of sampling to facilitate its use as a major design factor.

Although it might be expected that IPA and network model plans could and would grow at generally faster rates than staff and group models, the enrollment of the plans initially participating in Medicare risk contracts from start-up through 1984 shows a different pattern. The range of growth rates for both pairs of models is from fewer than 20 new enrollees per month to approximately 1,600 new enrollees each month. The median rate for IPA's and networks is 177 new enrollees per month, while that for staff and group models is 260 new enrollees per month. Faster growth for the purposes of stratifying the plan sampling frame is defined as greater than 200 new Medicare members per month for IPA and network models and greater than 300 new enrollees per month for staff and group model plans. Plans of each type growing at less than these respective rates are designated as slower growing.

Quality assurance programming

Quality assurance programs are a traditional method used to maintain professional standards where large numbers of providers are working together in an organized manner. Such procedures can also serve to counter influences of any financial incentive to underserve. Underscoring the policy relevance of quality assurance program structures, HCFA has proposed that review of Medicare HMO activities by professional review organizations be more or less intensive, depending on the design of the HMO quality assurance systems. Because another major element of the quality of care component of the NMCE is a description of the quality assurance activities under way in the plans, it is possible to incorporate that information in the analysis of the quality of the process of care. There is little evidence that traditional quality assurance procedures are effective, despite assumptions that they are useful and despite regulatory and professional requirements that such programs be in place (Hetherington, 1982). The availability of information on the types of utilization and quality control methods being used in each of the sampled plans permits direct assessment of the differential effects of the various quality assurance approaches observed. Work is under way to develop one or more indexes to capture the variation in quality assurance approaches used. The ability to establish a link between quality of care and quality assurance activities is heavily dependent on being able to measure the key elements and the interrelatedness of the elements in a quality assurance program.

Variations in health care needs

Medical care for the elderly must address a variety of health needs, including age-appropriate health monitoring and prevention, management of chronic diseases, attention to intercurrent acute illnesses, and treatment of the rarer conditions requiring resource-intense intervention such as hospitalization, expensive diagnostic evaluation, and/or surgical treatment. A study of the process of care giving for these conditions must also address the responsiveness of the delivery system to patient requests for care as well as the physician-managed portion of the care process. This issue is often examined under the rubric of access rather than process of care. It is, however, an essential first step in the process of care.

A variety of approaches to assessing the quality of the process of care has been developed since no one method appears to be appropriate for the range of conditions and types of care delivered to most populations. Most methods involve examining medical records, the primary documentation of the care given. The methodologic approaches used in the NMCE are discussed below in terms of the medical conditions and types of care that must be addressed in a comprehensive assessment of the process of care delivered to Medicare beneficiaries.

Relatively rare diagnosed conditions

An important facet of health care for the Medicare population is management of cases of life-threatening conditions that may require major diagnostic and therapeutic resources. These conditions do not occur with great frequency, yet they represent situations in which, if appropriate care resources are not expended, a less-than-optimal outcome is almost inevitable. If adverse incentives to reduce costs of care exist, it is in these cases that often require hospitalization and specialty referral that the effects of these incentives are most likely to be observed.

Certain medical record review methods rely on the selection of cases of specific health conditions or "tracers" that are expected to reflect how treatment is routinely provided (Kessner and Kalk, 1973). The management of each case is assessed for specific technical responses to the selected condition. Care is measured against explicit criteria so that one can say that what was done met professional standards or did not. Criteria sets for ambulatory and inpatient management of several conditions common among the Medicare population have been developed and used in previous studies. For example, Payne and others (1984) used this approach to examine ambulatory adult care in a variety of settings.

Such diagnosis-based methods can be used where the condition of interest is relatively rare, because information systems are commonly available to permit sampling on the basis of diagnoses. This approach limits one, however, to looking at care for conditions that have been identified by the providers. These methods are not likely to be useful in detecting the occurrence of undiagnosed disease that might arise from inadequate access or incentives for underservice. Also, a separate sample must be created for each condition to be studied.

Causes of hospitalization among the elderly were examined for diagnoses that occur with sufficient frequency to produce an adequate number of cases for analysis among the risk-based plan enrollees and for which explicit quality criteria sets were available. Two diagnoses requiring resource-intense management were targeted for study using diagnosis-based review methods. One of these conditions—colorectal cancer—is most often surgically managed; the other— congestive heart failure—is typically medically managed. Approximately 172,000 patients, 65 years of age or over, were discharged in 1983 with a diagnosis of colorectal cancer; almost 1.3 million patients, 65 years of age or over, were discharged with a diagnosis of congestive heart failure (Kozak and Moien, 1985).

Prevalent conditions and health monitoring

Basic health assessment and monitoring, followup of unexpected findings, preventive care, and management of high-frequency conditions that are typically handled on an ambulatory basis form the bulk of health care for Medicare beneficiaries. If quality of care is not adequate in these areas, large numbers of patients would be affected. In comprehensive case review, one examines the records of a random sample of patients against criteria that cover a fairly wide variety of kinds of care, e.g., basic health monitoring, prevention, and diagnosis and treatment of several specific conditions. As with diagnosis-based approaches, observed care is compared with explicit professional standards. This approach was used by Palmer and others (1984) and Goldberg and others (1983) in their assessments of quality of care in group practice settings. Certain of the criteria developed in those studies are appropriate for adaptation in the NMCE.

Comprehensive medical record review methods allow determination of whether a provider follows up appropriately on unexpected findings—an aspect of the care process that is not readily examined when the cases must be selected by diagnosis. Comprehensive review is also feasible for assessment of aspects of the care process and health conditions that occur with sufficiently high frequency to obtain an adequate number of observations in an unselected sample of records.

After examination of utilization rates for health conditions among the elderly and the availability of developed care criteria, two common chronic conditions—hypertension and diabetes—were selected for inclusion with basic health assessment, preventive care, and followup of unexpected findings in a comprehensive case review. Hypertension is the leading cause of ambulatory visits among the elderly population, accounting for over 15 million office visits, 8 percent of all visits in the group age 65 years or over. Diabetes accounts for 2 percent of all office visits by elderly patients, or over 3.7 million ambulatory visits (Cypress, 1981). Together with the care for colorectal cancer, congestive heart failure, routine monitoring, and followup on unexpected clinical findings, ambulatory care for hypertension and diabetes are expected to reflect the general quality of care provided to the Medicare beneficiary.

Access

Neither diagnosis-based nor comprehensive case review methods are appropriate for assessing the responsiveness of the provider to patients' requests for care, because these methods rely on the medical record as the primary data source. If access is impaired, there may be no entry in the record or no record at all. In order to assess the responsiveness to patients' care-seeking of the Medicare HMO's relative to the FFS sector, one must take an approach that does not rely on the physician's records as the base of information.

Because most episodes of care begin with a patient's request for service, an approach that goes directly to the patient for information on access appears appropriate. Medicare beneficiaries were surveyed directly in the NMCE regarding several aspects of their decisions to join a risk-based plan. Survey respondents were asked to report their recent symptom experiences, requests for service based on these symptoms, and the responsiveness of the providers to their requests. Symptom-use and service request-use ratios will be examined as a measure of system responsiveness to patient-initiated requests.

In summary, the conceptual framework for this evaluation must address two major elements: the diversity of kinds of health care required to meet the needs of the Medicare beneficiary and the major sources of variation in the organizational settings in which the care is delivered. The approach must incorporate an examination of access to care, health monitoring, care for routine causes of morbidity, and resource-intense management of life-threatening conditions. It must also address key policy-relevant variables that may influence the quality of care delivered, including the general organization of the plan, the rate of growth, and quality assurance activities. Finally, because risk-based contracting for Medicare was implemented nationally and there are large variations in practice patterns from one region of the country to another, the study must attend to regional differences in order to separate them from any differences in the quality of care between Medicare HMO's and the FFS alternative.

Objectives of the analysis

The major objective of this analysis is to determine whether the quality of the process of care is different in the risk-based plans than in the FFS settings used by Medicare beneficiaries. Specific subobjectives include:

To determine whether beneficiaries' attempts to access their care providers for perceived symptoms are responded to differently by risk-based plan providers than by FFS providers.

To determine whether the process of care for Medicare beneficiaries meets professional quality criteria equally often in risk-based versus FFS settings for routine ambulatory care (including management of hypertension and diabetes, and followup of unexpected clinical findings) and resource-intense care (including care of colorectal cancer and congestive heart failure).

To determine whether the following several plan characteristics affect the quality of the process of care delivered to Medicare enrollees: rate of growth, organizational form (i.e., staff or group versus network or IPA model) and quality assurance program structure.

Limitations

Uncertainty about the effects of medical care on the health of patients limits one's ability to measure quality of care and, hence, to compare the quality of care in one setting with that in another setting. These uncertainties in the health care-health outcome relationship make it very difficult to create unequivocal standards of quality. Process of care measures are not absolutely associated with quality, because professional standards cannot address all clinical circumstances. Outcomes may be affected by conditions that precede the patient-physician interaction or are otherwise beyond the control of the delivery system. These problems of definition and measurement are inherent in any study of the quality of health care. Other limitations follow from the specific needs of this evaluation.

Because we wish to evaluate the quality of care provided to all Medicare patients in a variety of organizational entities across the United States, our resources are particularly stretched by the magnitude of the quality measurement problems. We cannot focus on the details of care and clinical status of a relatively few patients or providers. Nor can we examine the total care of large numbers of patients or assess the overall quality of care provided by large numbers of care givers. The analysis is designed to provide a sampling of the range of care delivered to the Medicare population. We are limited by the degree to which the sampled elements of care actually represent the entirety of care given.

The analysis of tracer conditions begins with patients with a particular diagnosis. Although one can determine whether appropriate steps were taken to make that diagnosis, one cannot readily determine how much of the condition remains undiagnosed. Because a question of underservice exists in comparing risk-based plans with the FFS sector, this issue presents an additional limitation. To the extent that diagnosed patients from the two settings represent vastly different proportions of the true population with the condition, tracer methods may provide a biased view of any existing differences in care.

This limitation is not severe because of two study design elements. First, the analysis of access is driven by patient requests for care. If underservice is a major problem, this part of the analysis should detect it. Secondly, the analysis of the process of care for hypertension and diabetes is based on a random sample, rather than a diagnosis-specific sample. Moreover, application of the care criteria is initiated by evidence in the record of relevant signs and symptoms in addition to a provider label or diagnosis. This limitation does, however, apply to the analysis of the two resource-intense conditions, because they exist in such relatively small numbers that a diagnosis-based sampling strategy is required.

Finally, this study is to evaluate the quality of care in risk-based plans generally. We have not attempted to study plans with sufficient precision to reach conclusions about the quality of care in any individual plan. We examine differences among plans in organizational arrangements, rate of growth, and structure of quality assurance programming to determine the overall relationships between these policy-relevant factors and quality of care.

Data

Three sources of data are used for this analysis: beneficiary surveys, case study reports, and reviews of medical records. A description of each source of data follows.

Beneficiary surveys

Data for the analysis of access are from baseline and follow-up surveys of approximately 2,000 enrollees in 17 Medicare HMO's in 10 sites, and of 1,000 beneficiaries in 10 comparison sites with no Medicare HMO activity. Telephone interviews were carried out in Spring and Summer 1985 and 1986. The follow-up survey includes information on respondents' recent experiences of certain symptoms that might require assessment by a health professional, the respondents' attempts, if any, to seek medical attention for each symptom, and their reports of success in being seen for each symptom for which care was sought. Table 1 provides the list of symptoms and the questions asked for each symptom reported.

Table 1

Specific data elements from the beneficiary surveys included in analysis of access to care

List of symptoms

In the last 6 months, have you had or been bothered with: Chest pain with exercise or exertion? A cough without fever that lasted at least 3 weeks? Severe loss of eyesight? Stiffness, pain, or swelling of joints lasting more than 2 weeks? Bad stomach cramps or pain? Loose bowels or diarrhea? Any loss of consciousness, fainting spells, or passing out? Any problems with bleeding, other than nosebleed, not caused by an accident or injury? Shortness of breath with light exercise or light work? Weight loss of more than 10 pounds unless you were dieting?

For each "Yes" response, the following sequence of questions was asked. Did you seek care from (comparison respondent: a medical person; enrollee: name of HMO/CMP) for this problem? If "Yes": Did you actually see (a medical person/someone at HMO/CMP) about your (name of symptom)? If "No" to (a) or (b) above: Why not? Responses: Couldn't get an appointment. Problem went away or wasn't important. Talked with caregiver over the phone. Chronic problem, no further contact necessary. For enrollee only: Sought help somewhere else. Other specified.

NOTES: HMO is health maintenance organization. CMP is competitive medical plan.

Case studies

Information on quality assurance programs was gathered during sites visits to 20 early Medicare HMO's. Plans were visited by project staff members approximately 6 months after each HMO began enrolling Medicare-risk members. Interviews were conducted with HMO staff, including the executive and medical directors, person in charge of quality assurance, and chief financial officer.

Medical record reviews

The major source of data for the evaluation of the quality of care given in the Medicare HMO's versus the FFS sector is the medical record maintained by the plans and FFS physicians and hospitals. Three sets of explicit quality of care criteria—comprehensive ambulatory care, including hypertension and diabetes, congestive heart failure, and colorectal cancer—were developed by expert clinicial panels based on criteria written for other studies. The criteria and data collection instruments were structured for use by registered nurse abstractors.

Criteria and instrument development

Physician panels were formed to develop the clinical care criteria and advise on the appropriate interpretation of findings from the study of medical records. Nominations for the panels were solicited from professional and governmental agencies and societies, including the Gerontological Society of America, Health Care Financing Administration, National Heart, Lung, and Blood Institute, Institute of Medicine, American Geriatrics Society, American College of Cardiology, National Society of Clinical Oncology, American Society of Internal Medicine, Group Health Association of America, and American Medical Review Organization.

Panel members were selected to provide balance among needed clinical specialties and knowledge of risk-based delivery systems as well as FFS settings. Three panels with overlapping membership were established: cardiology, oncology, and basic care. The cardiology panel worked on criteria for hypertension and congestive heart failure, the oncology panel designed criteria for colorectal cancer; and the basic care panel developed criteria for the comprehensive record review and diabetes. Table 2 provides the names and affiliations of the panelists and the composition of each of the panels.

Table 2

Clinical advisory panels

Paul Ertel, M. D. Applied Medical Data Ann Arbor, Michigan	Barbara Hulka, M.D. University of North Carolina Chapel Hill, North Carolina
Stanley Kilty, M.D. Capital Area Community Health Plan Latham, New York	Suzanne Knoebel, M.D. Indiana University Indianapolis, Indiana
Raymond Lenhard, M.D. Johns Hopkins University Baltimore, Maryland	Richard Lindsey, M.D. University of Virginia Charlottesville, Virginia
Gregory Pawlson, M.D. George Washington University Washington, D.C.	Charles Rackley, M.D. Georgetown University Washington, D.C.
Knight Steel, M.D. Boston University Boston, Massachusetts	Jeff Weiner, M.D. U.S. Health Care Systems Blue Bell, Pennsylvania

Basic Care Panel: Ertel, Lindsey, Steel, Weiner

Cardiology Panel: Ertel, Kilty, Knoebel, Steel, Rackley

Oncology Panel: Hulka, Lenhard, Lindsey, Pawlson

Draft criteria sets were developed based on existing protocols and sent to the panel members for review prior to each panel meeting. Each panel met for approximately a day and a half to finalize the selection of specific tracer conditions and to review and modify the draft criteria sets. Discussion of each criteria set focused on the following:

The definition of a case, including specific inclusion and exclusion criteria for sampling.

The specific record notations that must be present 01 absent to meet each criterion;

The time period in the record that must be reviewed for each criterion.

The specific record or part of a record, if appropriate, that should be abstracted for each criterion.

Criteria sets were structured in a uniform way, beginning with items of history, then addressing, in order, specific elements of physical examination, diagnostic studies (including laboratory, radiological, and other procedures), treatment, followup, and indications for hospitalization.

Discussion among panel members was managed by a designated member within the general outline of an agenda set by the project staff. After consensus was reached on which conditions were appropriate tracers, face-to-face interaction focused first on an item-by-item consideration of the draft criteria set, with emphasis on what additional items might be added and which existing items might be deleted. A second item-by-item review focused on additional discussion and consensus formation. Panel members voted openly for or against each proposed item.

Following the panel meeting, study staff redrafted the criteria based on the panel deliberations. Each member was then asked to reaffirm the criteria set by voting for or against each specific item in a mail ballot. In several cases members offered modified criteria for consideration, or split votes emerged. These were resolved through telephone conferences and, where significant changes were apparent from the previous draft, a third draft criteria set was mailed out for a final ballot. In no case were more than two mailed ballots needed to reach consensus.

Specific protocols and data collection instruments for use by registered nurse abstractors were developed from the final criteria sets. The fit between the criteria and the abstracting protocols was reviewed by an outside physician consultant and the panel members to assure that the meaning behind the criteria had been adequately captured in the instruments and protocols. The instruments were field tested on ambulatory and inpatient records, revised, retested, and finalized.

Abstractor training

Ten registered nurses with research experience were recruited from the areas where data collection would take place. They were trained in one 5-day session that included presentation of the instruments and procedures, item-by-item discussion of the instruments, and practice in abstracting actual records. Field manuals that included explicit instructions on procedures for contacting sample practices and plans and for gathering each bit of information to be recorded were developed for the use of the abstracting staff.

Sample frame

As noted above, the sample of HMO's should reflect the geographic distribution, variation in organization, and differences in rates of growth observed. Very small plans were excluded because they could not support sampling of sufficient numbers of enrollees. The remaining early plans were distributed within a two-by-two design, with two categories of organizational form—staff and group models versus IPA and network models—and two categories of rate of growth. Faster growing IPA and network models grew at rates of 200 new enrollees per month or more. Faster growing staff and network models grew at rates of 300 new enrollees per month or more. Two plans were selected for each cell of the table so that four major regions of the country were represented. Table 3 illustrates the resulting sample of plans. A sample of 100 records for comprehensive review and 25 records for each of the two tracer conditions from each plan (total n = 800 records for comprehensive review and 200 for each resource-intense condition) and a comparision FFS sample of equal size were selected.

Table 3

Sampling frame for Medicare HMO's

Rate of growth	Organizational form
	IPA/network	Staff/group
Faster growing	Plan A	Plan C
	Plan B	Plan D
Slower growing	Plan E	Plan G
	Plan F	Plan H

NOTES: The following geographic areas are represented: Southern California, South Florida, Central Massachusetts, Southeastern Michigan. HMO's is health maintenance organizations.

Sampling plan records

Records were selected separately for comprehensive and diagnosis-based review. Certain attributes of the patients to be sampled were determined by the criteria established by the physician panels. For example, the panels determined that standards of practice and expectations of medical recordkeeping often differ for the management of new and regular patients, with criteria being very difficult to establish for patients who have been known to the provider for some time. For this reason and because most Medicare HMO enrollees are new to their plans, the criteria sets for comprehensive review and congestive heart failure were designed by the panels to cover only new patients. Records from these two facets of the review were selected from new HMO enrollees. Enrollees with colorectal cancer were selected from those with an incident cancer, omitting recurrences, because standards for diagnosis and management of recurrent colorectal cancer were deemed not sufficiently established by the physician panel for that stage of the disease to be used as a tracer condition. Because registries of ambulatory as well as inpatient diagnoses are not consistently available, patients with resource-intense conditions were selected from hospitalized cases.

Sampling fee-for-service records

FFS records were sampled from the same communities represented by the HMO's, in numbers equal to those from plans, to maintain a comparable geographic distribution. A two-stage sampling approach was used to identify records for comprehensive review, first selecting physicians in each area who provide FFS care to Medicare beneficiaries and then from each physician's Medicare caseload. To increase the probability of finding providers of primary care to Medicare beneficiaries, the list in each area was stratified by specialty. The sample was selected from those specialties accounting for the majority of ambulatory visits from patients 65 years of age or over, i.e., general and family practice, internal medicine, and cardiology. Records were sampled from patients first seen by the provider within the same time frame as the enrollment in area Medicare HMO's. Balancing costs of collecting data from a large number of providers with the need to represent FFS care to Medicare patients generally, 7 to 10 records were selected from each physician's caseload.

To create a sampling frame for the FFS resource-intense conditions, Medicare Part A claims data were used to select hospitalizations from the same time interval as the HMO hospitalizations for the two tracer conditions. The resource-intense condition cases were also selected using a two-stage approach. Ten hospitals were sampled in each geographic area, with the probability of selection of each institution equal to its share of the cases of the tracer conditions in the area. Random sampling of claims then took place within those in each selected hospital.

Unit of analysis

The above discussion leaves open the question of what constitutes a "record." The comprehensive review criteria were drafted to be applied to the ambulatory record of a patient's primary care provider. The unit of analysis is, therefore, the medical record maintained by the HMO provider unit or the physician who believes he or she is the primary caregiver for a particular FFS patient. Although in the FFS setting a provider may not be aware of the actual proportion of care for a given patient that he or she manages, contacting patients to inquire about their major sources of care is considered logistically too difficult.

The diagnosis-based review criteria were developed to be applied to the record of one hospital stay combined with the ambulatory record of the patient's primary care provider. The major provider of care was identified from information in the hosptial record. Although a different provider may manage a patient before and after hospitalization, the size of the study limits the number of physicians who can be recruited. Typical referral and consultation procedures usually result in the communication of essential information about a patient's status and management to the source of referral, in this case assumed to be the primary provider before hospitalization. In the event that such communication does not take place, the criteria were developed to emphasize prehospital diagnosis and treatment. Moreover, information can be collected on only those criteria reflected in the available record, indicating that the remainder of the record is truncated. In such a case, analysis is performed on the applicable criteria rather than the entire set.

Statistical analysis

The analysis is done in several major phases. The access analysis includes two parts: a descriptive analysis and hypothesis testing. The analysis of the medical record data will be conducted in four stages:

Descriptive analysis to determine how the characteristics of the providers, patients, and records sampled in the plans and the FFS sector compare with one another.

Comparison of compliance across each criteria set of FFS versus plan care.

Comparison of compliance with each criteria set for faster versus slower growing plans, staff and group model plans versus network and IPA model plans, and major dimensions of quality assurance programming.

Special analyses to clarify or elaborate conflicting or confusing findings from the first three stages.

Analysis of access

Descriptive analysis compares HMO enrollees and comparison site beneficiaries based on characteristics thought to be most directly related to the process of accessing care, e.g., need for care and factors enabling or inhibiting care seeking. Need-for-care indicators include reports of bed and restricted activity days, dependencies in activities of daily living, perceived need for health care or hospitalization, and self-reported health status. Enabling factors include social support systems (e.g., marital status, employment status, educational attainment), income, and attitudes about health care.

Analyzing beneficiary reports of access involves comparing rates of care seeking and care receipt between enrollees and comparison site beneficiaries. The major analysis is carried out on an aggregate of all symptoms, rather than each one individually. Only where individual symptoms occur with sufficient frequency are they analyzed separately. The same technique is used to compare careseeking results among the risk-based plans by organizational arrangement, rate of growth, and quality assurance program characteristics.

If differences are found in the health status or level of symptom reporting of one group of beneficiaries that might affect the delivery system response, the effects of these differences on the rate of response can be examined and controlled using statistical techniques. Similarly, if differences between plan and FFS users are found in enabling factors, their effects on system responsiveness will be examined. The experiences of different income groups of Medicare enrollees in accessing HMO's versus the FFS system (Ware et. al., 1986) can also be addressed.

Analysis of process of care

Analysis of the medical record data begins with describing the characteristics of the providers, patients, and records surveyed. The second step asks whether compliance with quality criteria is the same in the plans as the FFS sector. Step three includes the analysis of quality among plans with different characteristics. Finally, further analytic studies may be carried out to clarify or extend findings on observed differences.

Descriptive analysis

Data collected from the medical record reviews include a number of characteristics of the providers, patients, and records being studied. These attributes are examined to determine whether the characteristics of providers, patients, and records surveyed are different in the FFS sector versus the plans and among plans of one type. Knowledge of such differences may be useful in understanding the findings from subsequent analyses.

Compliance with quality criteria

A major methodologic issue in this analysis is the degree and kind of aggregation that is appropriate in the quality of care criteria. Although some criteria may be of interest individually, most are useful as they contribute to a pattern of compliance. Each criterion may best be considered one item in an index that, taken as a whole, measures the quality of care provided. There is also a statistical issue involved. Multiple tests of significance increase the probability of chance occurrences that cloud real differences. Given these issues—no real interest in the individual criteria, the need to construct the best indicator of quality of care possible, and the need to preserve the interpretability of statistical tests—some aggregation appears appropriate. Too much aggregation, however, may hide important areas of discrepancy that can be used to interpret differences.

The analysis is carried out using two levels of aggregation to capture as much information as possible from the data. At the greatest level of aggregation, we examine compliance with all criteria in a given set, i.e., basic care, hypertension, diabetes, congestive heart failure and colorectal cancer. A second level of aggregation is also examined to determine whether there are different patterns of compliance in subparts of the criteria. The criteria sets were developed with a standard structure—history, physical examination, diagnostic tests, treatment, criteria for hospitalization, and followup—although not all sets have criteria in all categories. Average percent compliance within each section of the criteria is compared between care settings.

Compliance by plan characteristics

The third stage in this phase of analysis includes determining whether quality of care is different among plans of various types. The analysis takes the same form as that described above for comparing FFS with risk-based plan records. The analysis focuses on a comparison of plans by organizational form, rate of growth, and quality assurance procedures, with regard to average compliance with the various aggregates of the criteria sets.

Special analytic studies

Further analyses are carried out to clarify patterns found in the above analyses. An issue inherent in studies using medical record data is the degree to which the quality of recordkeeping limits one's ability to determine what actually occurred. The quality of care criteria were developed with consideration for what would likely be found in most medical records. If a particular element of care was not expected to be recorded in most care settings, that element was excluded from the criteria set irrespective of its relevance to the care process. Thus the criteria themselves should control for much of the variation in recordkeeping. The descriptive analysis may lay to rest this concern, if no significant differences in record characteristics are found. On the other hand, if differences are found, it is important to explore the relationship of the recording differences to any differences in compliance with quality criteria found, to assess the degree to which record characteristics may actually explain apparent differences in quality of care.

A similar special study relates to the health status of patients seen in one delivery setting versus another. Compliance with quality criteria might vary by the health status of the patient population. For example, physicians might know from experience with healthy patients that standard procedures or tests reflected in the criteria would not be useful. In contrast, where patients are more likely to have complex conditions, physicians might be using different procedures to manage complicated cases. If the descriptive analysis of patient characteristics indicates differences in health status in various care settings, understanding the degree to which health status might explain apparent differences in quality of care is then important to the analysis of the major hypotheses of the study.

The important question of whether one can observe a relationship between quality of process of care and patient outcomes may also be addressed in the context of the NMCE. Data on self-reported satisfaction with their care system and changes in health status over a 1-year period are available on independent samples of aproximately 120 enrollees in each of the plans sampled for the process of care study from the beneficiary surveys described above. Although one cannot link the quality of care of an individual with that same individual's reported satisfaction or health changes—the samples of enrollees for the two elements of the study are independent of each other— satisfaction and health changes might be related to an estimate of the general quality of care in a given plan. The feasibility of such an analysis depends on whether an index of overall quality can be constructed. Combining compliance with the several criteria sets and the measures of access into one index is not otherwise needed in the analysis as discussed above. Relative weights to be applied to each criteria set and to access would have to be addressed in any combined measure. Given the several types of care and conditions to be examined in each plan, however, it would appear possible to build an overall index of quality of care if the patterns of compliance observed within individual criteria sets and across criteria sets are interpretable and meaningful.

Finally, additional studies to clarify the interpretation of observed compliance with quality criteria may be suggested by the physician advisory panels. A major purpose of the panel review is to advise on the interpretation of the clinical criteria. Panels are scheduled to meet after the initial steps of data analysis in order to review preliminary findings.

Conclusion

This study of the process of care delivered by a sample of early entrants into the Medicare risk market has been conceptualized in terms of the fit between HMO care-management methods and the needs of a special population, the elderly. Process of care is viewed broadly as encompassing both the patient's access to the provider and care managed by the provider. Study design and analysis are focused on capturing major sources of variation in practice patterns across care settings—both between fee-for-service and HMO and between major types of HMO's—and across the range of major health conditions and types of care delivered to the elderly. Access to care is estimated by examining rates of beneficiary-reported care system responses to their requests for attention to health symptoms. Quality of clinical management is estimated by examining ambulatory and inpatient medical records for compliance with professional care criteria.

The limitations of this study derive primarily from the inherent problems of definition and measurement of quality of care. Its strengths derive from several factors. It is one element of a comprehensive evaluation of Medicare HMO's involving studies of plan structure, implementation and operation, biased selection, utilization, costs, and other aspects of quality of care. Because the plans and comparison sites used in the study of quality of care are a subset of those in other parts of the study, an unusually rich data resource is available to enhance interpretation of findings. Because the study is being conducted on a national basis and encompasses a variety of kinds of HMO's and FFS providers, it promises to be more generalizable than the typical study of one or a few plans, sites, or types of providers. The study also relies on multiple methodologic approaches to create as complete and unbiased a picture as possible of the quality of care provided. It includes assessments from the perspective of both the Medicare beneficiary and the provider. It uses comprehensive and diagnosis-based record review methods on samples of ambulatory and combined ambulatory and inpatient records. In summary, although the issues to be addressed are very complex and the findings will require careful interpretation, this study has the potential for providing significant information to policymakers responsible for the Medicare program, Medicare HMO's, and those with more academic interests in quality of care, health services for the elderly, and managed-care systems.