Risk adjustment and public reporting on home health care.

Risk adjustment is a critical tool in public reporting of quality measures. Its aim is to level the playing field so that providers serving different patients can be meaningfully compared. We used a theory and evidence-based approach to develop risk-adjustment models for the 10 publicly reported home health quality measures and compared their performance with current models developed using a data-driven stepwise approach. Overall, the quality ratings for most agencies were similar regardless of approach. Theory and evidence-based models have the potential to simplify risk adjustment, and thereby improve provider and consumer understanding and confidence in public reporting.

Introduction

Medicare is the single largest payer for home health care with expenditures exceeding $10 billion in 2003 (Centers for Medicare & Medicaid Services, 2005). The benefit covers part-time or intermittent skilled nursing and therapy services provided to eligible homebound beneficiaries under the direction of a physician. It also covers, for individuals qualifying for the home health benefit, part-time or intermittent home health aide services that are needed to maintain the beneficiary's health or to facilitate treatment. Over 7,000 home health agencies (HHAs) were certified to participate in the Medicare Program in 2003 with 7.5 percent of all beneficiaries (2.7 million persons) receiving home health services. These individuals were predominantly female, over age 75, and had a wide range of acute and chronic conditions (Centers for Medicare & Medicaid Services, 2005).

DHHS has two key initiatives developed and implemented by CMS to assess, improve, and report on the quality of home health care. The Outcome-Based Quality Improvement (OBQI) program provides reports to all Medicare-certified HHAs on their patients' outcomes so that agencies can identify potential quality problems and devise appropriate strategies to address them (Shaughnessy et al., 2002). The Home Health Quality Initiative (HHQI) uses a subset of the OBQI quality measures for public reporting (U.S. Department of Health and Human Services, 2003a). The purpose of HHQI is to provide useful information for potential home health consumers to make informed decisions when choosing a HHA, and to provide an incentive for HHAs to improve the quality of their care.

There are 41 home health quality measures in the context of the OBQI framework. They include clinical outcomes in several domains (i.e., physical functioning, physiologic measures, emotional/behavioral problems, and cognitive functioning) as well as health care utilization outcomes. Ten of the 41 OBQI quality indicators currently are publicly reported as part of HHQI: improvement in bathing, transferring, ambulation, management of oral medications, pain interfering with activity, dyspnea, and urinary incontinence; and acute care hospitalization, discharge to the community, and emergent care (Table 1).

Table 1

Outcome Based Quality Improvement (OBQI) Quality Indicators

Functional: Activities of Daily Living Improved in: Bathing1 Dressing Upper Body Dressing Lower Body Grooming Toileting Transferring1 Eating Ambulation/Locomotion1 Stabilized in: Bathing Grooming Transferring

Functional: Instrumental Activities of Daily Living Improved in: Management of Oral Medications1 Light Meal Preparation Laundry Housekeeping Shopping Telephone Use Stabilized in: Management of Oral Medications Light Meal Preparation Laundry Housekeeping Shopping Telephone Use

Physiologic Improved in: Pain Interfering with Activity1 Number of Surgical Wounds Status of Surgical Wounds Dyspnea1 Urinary Tract Infection Urinary Incontinence1 Bowel Incontinence Speech or Language Stabilized in: Speech or Language

Emotional Behavioral Improved in: Anxiety level Behavioral Problem Frequency Stabilized in: Anxiety level

Cognitive Improved in: Confusion Frequency Cognitive Functioning Stabilized in: Cognitive Functioning

Utilization Outcome Measures Acute Care Hospitalization1 Discharge to Community1 Emergent Care1

The 10 OBQI measures that are publicly reported as part of the Home Health Quality Initiative.

NOTES: Outcomes labeled as Improved in are binary indicators of whether status at discharge is better than at start of the episode on that outcome. Episodes that start at the ceiling of the outcome measure (i.e., those that could not improve because they are already at the top) are excluded from the denominator for Improvement outcomes.

Outcomes labeled as Stabilized in are binary indicators of whether status at discharge is the same or better at discharge as compared to the start of the episode for that outcome. Episodes that start at the floor of the outcome measure (i.e., those that could not get worse because they start at the worst level), are excluded from the denominator for Stabilization outcomes.

SOURCE: Shaughnessy, P.W., Crisler, K.S., Hittle, D.F., et al.: Summary of the Report on OASIS and Outcome-Based Quality Improvement in Home Health Care: Research and Demonstration Findings, Policy Implications, and Considerations for Future Change. Center for Health Services Research. University of Colorado Health Sciences Center. 2002. Internet address: http://www.cms.hhs.gov/apps/hha/Summary-WebSite.pdf (Accessed 2007.)

The Outcome and Assessment Information Set (OASIS) is the source of the data used in OBQI and HHQI. Since July 1999, HHAs participating in the Medicare or Medicaid Programs have been required to collect OASIS on all persons age 18 or over admitted to certified HHAs. The two exceptions are persons receiving pre- or postpartum maternity services and those receiving only personal care, chore, or housekeeping services. Data are collected on admission, every 60 days thereafter and at discharge, as well as when patients have a significant change in health status, transfer to an inpatient setting, and resume home health care following an inpatient stay. Baseline and discharge values of clinical measures are compared to determine a patient's health status outcomes, while utilization outcomes (e.g., hospitalization) are based on transfer and discharge data.

Risk adjustment is a critical tool in public reporting of quality measures. Its aim is to level the playing field so that providers serving different patients can be meaningfully compared (Johnson, 2003). HHAs need to be confident that risk adjustment allows a fair comparison regardless of the types of patients served, and consumers need to be confident that quality of care is accurately reported.

All 10 HHQI quality indicators currently are risk adjusted using a data-driven step-wise approach with a separate set of risk factors included in the risk-adjustment model for each outcome. The stepwise approach tends to maximize the variance in the outcome measure explained by risk adjusters available in the particular data set being modeled. It also risks over-fitting the data with risk-adjustment models not performing as well when applied to new data (e.g., subsequent years of agency OASIS data). To at least partially address this problem, the current stepwise risk-adjustment models first were estimated using a developmental sample and then evaluated using a validation sample. The stepwise approach to risk adjustment, however, still is difficult for agencies to understand and to implement in their own settings.

The purpose of our project was to develop and test a theory and evidence-based approach to risk adjusting all 41 OBQI quality indicators including the 10 publicly reported quality measures that are the focus of this article. Advantages of a theory and evidence-based approach include simplicity, understandability, and stability of the risk-adjustment models over time. There also is the potential for greater parsimony in data elements needed for risk adjustment given the large number of outcomes in the OBQI program. Specific aims were to:

Identify a core set of risk adjusters for inclusion in all risk-adjustment models as well as a small number of supplemental risk adjusters plausibly influencing individual outcomes.

Examine the contribution of the supplemental risk adjusters to the explanatory power of outcome models that already include core risk adjusters.

Compare the number of risk adjusters used and performance of current and alternative risk-adjustment models.

Compare agency quality ratings based on current and alternative risk-adjustment approaches.

Refinements in risk adjustment ultimately should lead to more effective methods of assessing and improving the quality of home health care (Murtaugh et al., 2006).

Framework

The alternative models were developed within the framework of the OASIS in place at the time of the study. We relied solely on OASIS, which is the source of current risk adjusters and contains only patient data, because a project goal was to develop alternative risk-adjustment models that could be implemented using existing data sources. Market characteristics were not considered. Although they theoretically could influence patient outcomes, public reporting under the HHQI program is designed to report outcomes for agencies in the same geographic area, primarily the same county or ZIP Code, although outcomes can be generated for all agencies in a State (U.S. Department of Health and Human Services, 2003b). Market characteristics do not need to be considered as potential risk adjusters for home health outcomes given the design of public reporting.

The framework for selecting the core set of OASIS risk adjusters is based on Iezzoni's (2003) conceptual approach to risk adjustment of health care outcomes, prior empirical research on home health outcomes (Schlenker, Powell, and Goodrich, 2005; Murtaugh et al., 2005; Feldman et al., 2005; Peng, Navaie-Waliser, and Feldman, 2003; Shaughnessy and Hittle, 2002; Cheh and Black, 2003; Fortinsky and Madigan, 1997), and the input of a Technical Advisory Group (TAG) convened to provide feedback on preliminary project findings. The specific domains (and subdomains) of potential risk adjusters in our framework are:

Demographic characteristics.

Socioeconomic factors (educational attainment, health insurance coverage, familial characteristics, and assistance in the home).

Clinical status prior to home care admission (historical use of health services, prior medical conditions, and prior physical functioning).

Clinical status at home care admission (physical functioning, physiologic measures, primary diagnosis and comorbidities, cognitive status, mental health, and clinical therapies).

Health-related behaviors and activities.

Methods

Source of Data

The data analyzed in this project were obtained from the University of Colorado Health Sciences Center, the CMS OBQI contractor at the time of the study. They drew the data from the OASIS National Repository at CMS to create discrete episodes of home health care during calendar year 2001. OBQI episodes in most cases represent the period from admission to discharge with OASIS data collected at these two time points determining the 41 outcomes. However, if the patient is transferred to an inpatient facility (e.g., an acute care hospital) but not formally discharged, the initial OBQI episode is ended and a new OBQI episode begins when the patient returns to the HHA. The OASIS assessment conducted when care resumes is the source of baseline data for evaluating subsequent outcomes.

Approximately 1,500,000 OBQI episodes are present in the overall data set. The file includes all episodes of care that both begin and end within the calendar year. The University of Colorado randomly assigned about one-third of the episodes to the developmental sample for initial estimation of risk-adjustment models for most outcomes. The remaining 1,000,000 were used to validate the final models derived from analysis of the developmental sample.

The data set contains the 41 OBQI outcome indicators and 143 potential risk adjusters derived from OASIS, with detailed technical specifications publicly available online (U.S. Department of Health and Human Services, 2003c). All of the data needed to replicate the risk-adjustment models employed in OBQI and HHQI at the time of the study were included on the files.

The project estimated preliminary models using the 143 candidate risk adjusters. Following the TAG meeting, a small number of potentially important risk adjusters available on raw data files edited by the CMS contractor were requested by the project team and provided by the University of Colorado.

Analytic Methods for Developing Alternative Models

Preliminary data analyses included replication of current risk-adjustment models for the 11 HHQI outcomes at the time the study began and development of alternative models for these outcomes. Logistic regression is the statistical method currently used to risk adjust OBQI outcomes. We also used logistic regression when estimating alternative models since the purpose of the project was to replicate the existing approach and compare it with a theory and evidence-based approach to selecting risk adjusters.

Estimation of the preliminary theory and evidence-based models proceeded sequentially. A total of six models was estimated for each of the initial 11 HHQI outcomes. We began with a model limited to a core set of clinically relevant risk adjusters, which included the baseline value of the outcome measure if it was not already among the core variables. We then added risk adjusters in groups to the core model in the following order:

Supplemental Clinical Measures—Addition of other baseline clinically relevant patient characteristics plausibly influencing the specific outcome.

OASIS Prior Items—Addition of prior health status variables (e.g., physical functioning 14 days prior to admission). The rationale for examining prior health status variables separately is because of questions regarding their reliability and possible elimination from the OASIS instrument.

Clinical Therapies—Addition of indicators of whether the patient was receiving specific therapies at baseline (i.e., oxygen therapy, intravenous/infusion therapy, enteral/parenteral nutrition, and ventilator). The rationale for examining therapies separately from other clinically relevant risk adjusters is that they are qualitatively different from the demographic and clinical characteristics of individuals. In addition, these therapies are used to determine the case mix adjusted Medicare home health payment rate and might seem to be subject to HHA gaming. Clinical and industry experts agree, however, that these services are invasive and would not be initiated without very clear clinical indications and medical orders.

Social Support—Addition of the living arrangement and social support indicators as risk adjusters.

Length of Stay (LOS)—The sole purpose for including LOS was to allow comparison of model statistics and parameter estimates with the University of Colorado risk-adjustment models that include LOS.

Several statistics were estimated to compare and evaluate the performance of current and alternative risk-adjustment models: (1) number of OASIS items that are the basis for the risk adjusters included in the model, (2) number of OASIS elements that are the basis for the risk adjusters included in the model (some OASIS items include multiple elements with each element separately assessed and marked), (3) R2 statistic (technically, a pseudo R2 statistic employed by the University of Colorado that is the squared correlation between the observed and predicted value of the dependent variable), and (4) c statistic (a measure of how well the risk adjusters in the model correctly classify the outcome examined; a model that performed no better than chance would have a c statistic of 0.5, while a completely accurate model would have a c statistic of 1.0).

The TAG meeting then was conducted with experts in home health care and risk adjustment as well as policymakers and provider representatives. The TAG provided input on our initial approach based on the results of the preliminary data analyses. They suggested that a small number of additional OASIS items be obtained and preliminary risk adjusters refined prior to finalizing the set of core risk adjusters to be included in all alternative models.

The analyses conducted after receipt of additional OASIS data from the University of Colorado included examination of alternative specifications of the detailed living arrangement and informal support data collected in OASIS. The additional data and respecification of the living situation and informal support measures did not substantially alter their contribution to the explanatory power of the alternative risk-adjustment models that already included demographic, payer, and clinical measures. The one exception is the risk-adjustment model for improvement in medication management. When the living arrangement and social support measures were added to a model with demographic, payer, and clinical measures, the R2 statistic increased from 15.7 to 16.7 percent. These conceptually important measures were excluded from the alternative models because of the limited contribution to the explanatory power of the risk-adjustment models.

Table 2 lists the final set of core risk adjusters in the alternative models along with their specification. A total of 43 OASIS items was used to construct the core risk adjusters. The one core risk adjuster that varies from model to model is the baseline value of the outcome indicator. The baseline value, specified as a categorical variable, tends to make a relatively large contribution to the explanatory power of risk-adjustment models. It appears to be adjusting for differences in the probability of improving related to the number of levels of the OASIS item.

Table 2

Final Set of Core Risk Adjusters in All Alternative Risk-Adjustment Models of Home Health Quality Initiative Outcomes

Core Variable		OASIS Items		Specification
Demographics
Age		M0066		<6565-74 (Reference Category) 75-8485+
Sex		M0069		FemaleMale (Reference Category)
Socioeconomic Factors
Current Payer		M0150		Any MedicaidMedicare Health Maintenance Organization Medicare Fee-for-Service and Other (Reference Category)
Prior Service Use
Discharged Past 14 Days
Discharge from HospitalDischarge from Rehabilitation FacilityDischarge from Nursing Home		M0175		YesNo (Reference Category)
Clinical Factors
Baseline Value of Outcome Indicator		Varies Depending on Outcome Indicator		—
Prognoses
Overall Prognosis		M0260		Poor (Reference Category) Good/Fair
Rehabilitation Prognosis		M0270		Guarded (Reference Category) Good
Diagnoses
Diabetes (PPS Group)Neurological (PPS Group)Orthopedic (PPS Group)Wound/Burn (PPS Group)DementiaHypertensionIschemiaArrhythmiaHeart FailureCOPDSkin UlcerOrthopedic (Other than PPS)IncontinenceCancerMental ConditionSigns, Symptoms, and Ill-Defined Conditions		M0230, M0240		YesNo (Reference Category)(Note: Diagnoses Are Not Mutually Exclusive
Diagnosis Severity
Number of Severity Ratings ≥ 2		M0230S, M0240S		Integer Count (Range 0 to 6)
Sensory Status
Vision		M0390		Normal (Reference Category) Partially ImpairedSeverely Impaired
Speech/Language		M0410		No Impairment (Reference Category) Minimal DifficultyModerate DifficultySevere Difficulty (Categories 3-5)
Integumentary Status
Surgical Wound Present		M0482		YesNo (Reference Category)
Stage of Most Problematic Pressure Ulcer		M0440, M0445, M0460		Scale 0-4 (0=No Pressure Ulcer)
Status of Most Problematic Stasis Ulcer		M0440, M0468, M0476		Scale 0-3 (0=No Stasis Ulcer)
Physical Functioning
ADL/IADL Summary Score		M0640 through M0780		Integer Count 0-14 (0=No Impairment)
Elimination Status
Urinary Incontinence Severity		M0520, M0530		No Incontinence (Reference Category) Timed Voiding Deters IncontinenceNight OnlyDay and Night
Urinary Catheter		M0520		YesNo (Reference Category)
Bowel Incontinence		M0540		Rarely or Never (Reference Category) Less than Once Weekly1-3 Times Weekly4-6 Times WeeklyDaily or More Often (Categories 4-5)
Ostomy for Bowel Elimination		M0550		Yes (Categories 1-2) No (Reference Category)
Neuro/Emotional/Behavioral Status
Cognitive Functioning		M0560		No Impairment (Reference Category) Requires PromptingRequires Assistance and Some DirectionRequires Considerable AssistanceTotally Dependent
Confusion Frequency		M0570		Never (Reference Category) New or Complex Situations OnlyOn Awakening or At NightDay and Evening, Not ConstantlyConstantly
Anxiety Frequency		M0580		None (Reference Category)Less Often Than Daily Daily, But Not Constantly All of the Time
Verbal Disruption at Least Once Weekly		M0610_3		Yes No (Reference Category)
Symptoms of Depression		M0590		None (Reference Category) Depressed Mood Any Other Symptoms

NOTES: OASIS is Outcome and Assessment Information Set. PPS is prospective payment system. COPD is chronic obstructive pulmonary disease. ADL is activity of daily living. IADL is instrumental activity of daily living.

SOURCE: Murtaugh, C., Peng, T., and Maduro, G., et al., Visiting Nurse Service of New York, Aykan, H., Department of Health and Human Services, 2006.

Risk adjusters specific to each outcome, including measures of health status prior to admission and selected clinical therapies, are listed in Table 3. Some items are common to all risk-adjustment models within a domain. For example, obesity is included in the risk-adjustment models of all activity of daily living (ADL) outcomes. Other items are specific to a single outcome. For example, whether a patient smokes is specific to the improvement in dyspnea risk-adjustment model. Measures of clinical status prior to home health admission are listed after the baseline measures. As previously noted, these OASIS items were examined separately from other outcome-specific risk adjusters because of questions about their reliability and possible elimination from the OASIS instrument. There are no directly related, conceptually important prior health status risk adjusters for four HHQI outcomes (i.e., improvement in dyspnea and the three utilization outcomes).

Table 3

Outcome-Specific Risk Adjusters in Alternative Risk-Adjustment Models of the Home Health Quality Initiative Outcomes

Variable	OASIS Items	Specification
Improvement in Bathing
Obesity	M0290_2	YesNo (Reference Category)
Frequency of Pain Interfering with Activity	M0420	No Pain, or Does Not Interfere with Activity (Reference Category)Less Often Than DailyDaily But Not ConstantlyAll of the Time
Bathing Prior to Admission	M0670_P	Independent (Reference Category)Able with Use of DevicesAble with Partial AssistanceRequires AssistanceUnable-Bathed in Bed/Chair Totally Dependent
Improvement in Transferring
Obesity	M0290_2	Yes No (Reference Category)
Frequency of Pain Interfering with Activity	M0420	No Pain, or Does Not Interfere with Activity (Reference Category)Less Often Than DailyDaily But Not ConstantlyAll of the Time
Current Ambulation	M0700	Walks Independently (Reference Category)Requires Use of DeviceAble to Walk Only with SupervisionChairfast, Able to Wheel SelfChairfast, Unable to Wheel SelfBedfast, Unable to Ambulate
Transferring Prior to Admission	M0690_P	Independent (Reference Category)Able with Minimal AssistanceUnable But Can Pivot SelfNeeds AssistanceBedfast
Improvement in Ambulation
Obesity	M0290_2	YesNo (Reference Category)
Frequency of Pain Interfering with Activity	M0420	No Pain, or Does Not Interfere With Activity (Reference Category)Less Often Than DailyDaily But Not ConstantlyAll of the Time
Current Transferring	M0700	Transfers Independently (Reference Category)Transfers with Minimal Assistance or DeviceUnable to Transfer; Can Bear WeightUnable to Transfer; Unable to Bear WeightBedfast, Able to Turn and Position Self
Ambulation Prior to Admission	M0700_P	Independent (Reference Category) Needs Device to Walk Needs Assistance to WalkChairfast-Able to Wheel SelfChairfast-Unable to Wheel SelfBedfast
Improvement in Management of Oral Medication
Behaviors Demonstrated at Least Once a Week
Memory Deficit	M0610_1	YesNo (Reference Category)
Impaired Decisionmaking	M0610_2	YesNo (Reference Category)
Conditions Prior to Medical Regimen Change or Inpatient Stay
Impaired Decisionmaking	M0220_4	YesNo (Reference Category)
Memory Loss Requiring Supervision	M0220_6	YesNo (Reference Category)
Medication Management Prior to Admission	M0780_P	Independent (Reference Category) Able If Prepared by Another PersonTotally Dependent
Improvement in Pain Interfering with Activity
Intractable Pain	M0430	YesNo (Reference Category)
Conditions Prior to Medical Regimen Change or Inpatient Stay
Intractable Pain	M0220_3	YesNo (Reference Category)
Improvement in Dyspnea
Obesity	M0290_2	YesNo (Reference Category)
Smoking	M0290_1	YesNo (Reference Category)
Respiratory Treatments
Oxygen	M0500_1	YesNo (Reference Category)
Ventilator	M0500_2	YesNo (Reference Category)
Improvement in Urinary Incontinenc
Obesity	M0290_2	YesNo (Reference Category)
Current Toileting	M0680	Able to Get to and from the Toilet Independently (Reference Category)Able to Get to and from Toilet When Reminded, Assisted, or SupervisedUnable to Get to Toilet, but Can Use Bedside CommodeUnable to Get to Toilet or Use Bedside Commode, But Can Use BedpanTotally Dependent in Toileting
Toileting Prior to Admission	M0680_P	Independent (Reference Category) Able When SupervisedUses Bedside CommodeUses Bedpan Independently/Totally Dependent (Levels 3, 4)
Treated for Urinary Tract Infection in Past 14 Days	M0510	YesNo (reference category)
Conditions Prior to Medical Regimen Change or Inpatient Stay
Urinary Incontinence	M0220_1	YesNo (Reference Category)
Intradwelling/Suprapubic Catheter	M0220_2	YesNo (Reference Category)
Impaired Decisionmaking	M0220_4	YesNo (Reference Category)
Memory Loss Requiring Supervision	M0220_6	YesNo (Reference Category)
Acute Care Hospitalization, Discharge to the Community and Emergent Care
Dyspnea	M0490	Never, Patient Is Not Short of Breath (Reference Category) Walking More than 20 Feet, or on StairsWith Moderate ExertionWith Minimal ExertionAt Rest
Therapy Received in Home
Intravenous/Infusion	M0250_1	YesNo (Reference Category)
Respiratory Treatments
Ventilator	M0500_2	YesNo (Reference Category)

NOTE: OASIS is Outcome and Assessment Information Set.

SOURCE: Murtaugh, C., Peng, T., and Maduro, G., et al., Visiting Nurse Service of New York, Aykan, H., Department of Health and Human Services, 2006.

Analytic Methods for examining Agency impacts

An agency-level analysis was conducted to examine how alternative approaches to risk adjustment of the OBQI quality indicators affect an agency's quality ratings. The agency-level analysis employed the validation data set provided by the University of Colorado with approximately 5,000 agencies included on the calendar year 2001 files. Three adjusted agency outcome rates were calculated for each OBQI outcome: (1) the adjusted outcome based on the current risk-adjustment model, (2) the adjusted outcome based on the alternative risk-adjustment model including only core variables, and (3) the adjusted outcome based on the alternative model including outcome-specific risk-adjusters in addition to the core variables. Not all agencies have estimates for all outcomes. We followed the current HHQI approach which is not to estimate or report an outcome for an agency if it has fewer than 20 episodes with the potential to have an outcome.

There were five steps in the calculation of the adjusted agency outcome:

Identify episodes where patients have the potential to have an outcome; for the seven health status measures reported in HHQI, this is the subset of episodes ending with discharge to the community where patients can improve their status from baseline (i.e., they do not begin the episode in the best category of the measure).

Determine the observed percent with the outcome at each agency with at least 20 episodes with the potential to have the outcome.

Estimate the predicted probability of the outcome at the individual episode level using the current risk-adjustment model and then the alternative models.

Calculate the average predicted probability of the outcome at each agency when the current risk-adjustment model is used, and then when the alternative models are used.

Adjust the agency mean so that agencies can be compared to the national average for an outcome using the CMS formula for HHQI reports, as developed by the University of Colorado (U.S. Department of Health and Human Services, 2003d):

The following statistics then were estimated for each of the 10 HHQI outcomes:

Number and percent of agencies with the outcome (i.e., agencies with 20 or more episodes where the patient had the potential to have an outcome).

Mean and standard deviation of the absolute difference in the adjusted percent of patients at each agency with the outcome.

Percentage point difference at the 5th percentile of the distribution of differences in the adjusted percent of patients at each agency with the outcome.

Percentage point difference at the 95th percentile of the distribution of differences in the adjusted percent of patients at each agency with the outcome.

Rank of an agency based on the current risk-adjustment model (an integer number with 1 representing the best rank among all agencies).

Rank of an agency based on each of the alternative risk-adjustment models.

Spearman's rank correlation test of the association between the ranking of agency performance as calculated using the current versus alternative risk-adjustment models.

Results

Comparison of Current and Alternative Models

A summary of the model statistics for the current as well as core and full alternative models is reported in Table 4. The alternative models typically have slightly lower explanatory power than the current risk-adjustment models. Specifically, the R2 statistic for the full model tends to be within 1 to 2 percentage points of the R2 statistic for the model developed by the University of Colorado. There is a similar pattern for the c statistic. The slightly lower explanatory power of all but one of the full models is not surprising since, as previously noted, the stepwise approach tends to maximize the variance in the outcome measure explained by risk adjusters available in the particular data set being modeled.

Table 4

Summary Statistics for Current and Alternative Risk-Adjustment Models for the Home Health Quality Initiative Outcomes: 2001

Variable	Current Model	Alternative Model
		Core	Full
Improvement in Bathing
Percent Who Could Improve: 62.2
Percent Improving Among Those Who Could: 57.0
Number of OASIS Items	52	41	44
Number of OASIS Elements	72	59	64
R2 Statistic	0.192	0.167	0.190
c Statistic	0.755	0.738	0.753
Improvement in Transferring
Percent Who Could Improve: 46.3
Percent Improving Among Those Who Could: 49.8
Number of OASIS Items	601	41	44
Number of OASIS Elements	871	59	64
R2 Statistic	0.1371	0.102	0.129
c Statistic	0.7111	0.681	0.705
Improvement in Ambulation
Percent Who Could Improve: 59.9
Percent Improving Among Those Who Could: 34.1
Number of OASIS Items	38	41	44
Number of OASIS Elements	53	59	64
R2 Statistic	0.180	0.213	0.244
c Statistic	0.755	0.768	0.788
Improvement in Medication Management
Percent Who Could Improve: 38.7
Percent Improving Among Those Who Could: 34.8
Number of OASIS Items	48	41	44
Number of OASIS Elements	76	59	66
R2 Statistic	0.180	0.132	0.157
c Statistic	0.754	0.718	0.737
Improvement in Pain
Percent Who Could Improve: 44.4
Percent Improving Among Those Who Could: 56.2
Number of OASIS Items	401	42	45
Number of OASIS Elements	651	60	64
R2 Statistic	0.0651	0.053	0.058
c Statistic	0.6431	0.630	0.635
Improvement in Dyspnea
Percent Who Could Improve: 44.2
Percent Improving Among Those Who Could: 53.3
Number of OASIS Items	571	42	44
Number of OASIS Elements	851	60	66
R2 Statistic	0.1141	0.098	0.110
c Statistic	0.6951	0.680	0.690
Improvement in Urinary Incontinence
Percent Who Could Improve: 20.7
Percent Improving Among Those Who Could: 49.0
Number of OASIS Items	531	41	46
Number of OASIS Elements	831	59	72
R2 Statistic	0.1191	0.088	0.103
c Statistic	0.6961	0.667	0.682
Acute Care Hospitalization
Percent Who Could Be Hospitalized: 100.0
Percent Hospitalized: 28.2
Number of OASIS Items	49	41	44
Number of OASIS Elements	75	59	62
R2 Statistic	0.152	0.119	0.125
c Statistic	0.740	0.714	0.719
Discharged to the Community
Percent Who Could Be Discharged to Community: 99.5
Percent Discharged to the Community: 68.1
Number of OASIS Items	53	41	44
Number of OASIS Elements	79	59	62
R2 Statistic	0.185	0.147	0.153
c Statistic	0.753	0.728	0.732
Emergent Care
Percent Who Could Have Emergent Care: 97.6
Percent with Emergent Care: 22.7
Number of OASIS Items	44	41	44
Number of OASIS Elements	69	59	62
R2 Statistic	0.100	0.072	0.075
c statistic	0.710	0.679	0.683
Summary for All 10 HHQI Outcomes
Total Number of OASIS Items	87	43	55
Total Number of OASIS Elements	133	61	84

Indicates that current model statistics are for multiple submodels; we report the number of unique OASIS items and elements across all submodels.

NOTES: OASIS is Outcome and Assessment Information Set. Percent Who Could Improve calculated using all home health episodes, not just those discharged to the community.

SOURCE: Centers for Medicare & Medicaid Services' Contractor at the University of Colorado: National Data from OASIS, Calendar Year 2001.

The alternative models that include only core risk adjusters do not predict outcomes as well as the full alternative models for the health status outcomes. Measures of physical functioning prior to home health admission are particularly significant in the risk-adjustment models of ADL and instrumental activity of daily living (IADL) improvement (not shown). In contrast, the outcome-specific risk adjusters contribute relatively little to the explanatory power of the utilization outcomes. Figure 1 graphically compares the explanatory power of the current and alternative HHQI risk-adjustment models (both the core and full models) as measured by the pseudo R2 statistic.

Figure 1

Performance of Alternative and Current Risk-Adjustment Models, by Home Health Quality Initiative (HHQI) Outcome: 2001

The total number of OASIS items and elements employed to risk adjust all 10 HHQI indicators is reported in Table 4. The current risk-adjustment models require a greater number of OASIS items and elements than the alternative models (87 items and 133 elements for the current models; 43 items and 61 elements for the 10 core models; and 55 items and 84 elements for the 10 full alternative models). The current model for risk adjusting improvement in transferring employs the greatest number of OASIS items and elements (i.e., 60 items and 87 elements). Among the alternative models, the largest number is used in the full risk-adjustment model of improvement in urinary incontinence (i.e., 46 items and 72 elements).

Agency Analyses

Overall, the quality ratings for most agencies and most outcomes are similar regardless of whether the current or alternative models are used to risk adjust HHQI outcomes. The difference between the risk-adjusted percent of an agency's patients with each outcome (e.g., the percent of an agency's patients improving in bathing) tends to be minimal. The average difference is no more than 1 to 2 percentage points when current and full alternative model results are compared. For a small share of agencies (i.e., those below the 5th or above the 95th percentile of the distribution), however, differences exceed 4 percentage points for three of the HHQI measures: improvement in ambulation, acute care hospitalization, and discharge to the community (not shown). The basic pattern of impacts is the same when quality ratings for the current and core alternative models are compared. The difference in risk-adjusted outcomes increases, however, to between 1 and 3 percentage points for most agencies on most outcomes (not shown).

While the magnitude of the difference between outcome estimates using current and alternative risk-adjustment approaches is important, it is the ranking of each agency relative to others that is likely to be of most concern to providers. Spearman's rank correlation coefficient between an agency's ranking using the current and alternative risk-adjustment models for each of the 10 HHQI outcomes is presented in Figure 2.

Figure 2

Spearman's Rank Correlations for Agency Outcomes Using Current Versus Alternative Outcomes Risk Adjustment Models, by Home Health Quality Initiative (HHQI) Outcome: 2001

The correlation coefficients for agency rankings based on the current and full alternative models approach 1 (i.e., they are above 0.950) for most outcomes indicating that the rankings are close to the same. The lowest correlation coefficient is 0.925 for improvement in ambulation. The correlation coefficients decline slightly when agency rankings are based on the current and core alternative models. However, only the correlation coefficient for improvement in ambulation falls below 0.900.

Agency quality rankings for health status outcomes differ the most where there is the greatest difference in the explanatory power of the current and alternative risk-adjustment models. In the case of improvement in ambulation, the alternative risk-adjustment models explain considerably more of the variation in the outcome than the current model. It is the reverse for the improvement in medication management outcome indicator. The correlation in quality rankings for the utilization outcomes, on the other hand, remain well above 0.900 despite the exclusion of LOS from the alternative risk-adjustment models which contributes to their substantially lower explanatory power compared to the current risk-adjustment models.

Discussion

There are important tradeoffs and differences between the current and alternative approaches to risk adjusting HHQI indicators studied here. The first is the generally higher explanatory power of the current models versus the relative simplicity of the alternative models and their overall reliance on a smaller number of OASIS items and elements. That current models generally have slightly better explanatory power than the alternative models is not surprising since the stepwise approach is likely to result in models with close to the best explanatory power possible for the data set analyzed. At the same time, however, it leads to the selection of a large number of risk factors when all outcome measures are considered. In addition, because the stepwise approach fits models to the data on which they are developed, the explanatory power of these models is likely to decline when they are applied to new data sets.

A second tradeoff is between the full alternative models that include the outcome-specific risk adjusters and alternative models with only the core set of risk adjusters. The latter models are easier to explain and for providers and consumers to understand. These models, however, tend not to predict outcomes as well as the full alternative models. Measures of physical functioning prior to home health admission are particularly significant in the risk-adjustment models of ADL and IADL improvement. The prior OASIS items, however, are more difficult than many other items for HHAs to collect and some are less reliable than other clinical measures (Neder, Rosati, and Huang, 2005). Should they be dropped from the OASIS instrument, the explanatory power of the risk-adjustment models for most ADL and IADL improvement outcomes would be reduced by roughly 2 percentage points.

The decision to exclude home health LOS from the alternative models, in addition, has a significant impact on the explanatory power of the risk-adjustment models for the three utilization outcomes. LOS was excluded because it can be affected by problems in the care process that also affect outcomes (i.e., low-quality care can cause a longer stay as well as worse outcomes). If included, it could mask poor quality of care. A possible methodological solution with data burden and simplicity implications is to collect information on the timing of utilization outcomes (e.g., hospitalization) and estimate hazard models that take into account the time to the outcome of interest.

An agency-level analysis was conducted to examine how alternative approaches to risk adjustment of the HHQI indicators affect an agency's quality ratings, with two main findings. First, for most agencies and most outcomes, the adjusted proportion of patients with an outcome is similar regardless of whether the current or alternative models are used to risk adjust outcomes. Second, the relative ranking of agencies using current risk-adjustment models and the ranking using the full alternative risk-adjustment models are in close agreement for most outcomes. With only the core risk adjusters included in the alternative risk-adjustment models, the basic pattern of impacts is the same but, as expected, the difference in risk-adjusted outcomes and agency rankings increases. We also compared agency rankings on each of the 10 outcomes without any risk adjustment, with agency rankings using current and alternative risk-adjustment models. Not surprisingly, the correlation of unadjusted and adjusted rankings tended to be highest for outcomes where the explanatory power of risk-adjustment models is lowest, and vice versa (e.g., around 0.95 for improvement in pain, and 0.76 for improvement in bathing).

The results suggest that the relatively small reduction in explanatory power of most of the alternative risk-adjustment models for the HHQI indicators is unlikely to have a substantial effect on the quality ratings of the majority of agencies. A theory and evidence-based modeling approach, then, has the potential to simplify and provide a consistent and stable basis for risk adjustment relative to the current approach. This should make it more understandable to providers and encourage individual agencies to risk adjust their own outcomes. The reliance on a smaller number of OASIS data elements, in addition, would contribute to ongoing efforts to streamline the OASIS instrument.

We examined an alternative approach to risk adjustment of home health quality indicators in the context of the current OBQI and HHQI programs. Whether the 41 OBQI outcome measures and method of determining agency performance adequately capture the quality of care provided to the full range of patients served by all certified HHAs is the focus of other work sponsored by the DHHS. This includes the extent to which long-stay patients are underrepresented because outcomes are estimated only for OBQI episodes beginning and ending within a 12-month period. How to increase information on the quality of care provided by small agencies with too few episodes for public reporting of OBQI outcomes also is being examined. Risk adjustment will remain, however, an important issue regardless of the specific measures and procedures for assessing agency performance.

The generally limited diffusion of risk-adjustment methods for assessing the quality of health care may be due to the multiple dimensions of quality, cost of appropriate data, and technical complexity of risk-adjustment methods. Blumenthal and colleagues (2005) argue that greater attention needs to be paid to simplicity, practicality, and the intuitive appeal of risk-adjustment methods to increase diffusion and the effective use of this tool. The theory and evidence-based risk-adjustment approach examined in our project is consistent with these recommendations and shows promise as a method for improving the effectiveness of publicly reported home health quality measures.