Population health measurement: applying performance measurement concepts in population health settings.

INTRODUCTION: Whether the focus of population-health improvement efforts, the measurement of health outcomes, risk factors, and interventions to improve them are central to achieving collective impact in the population health perspective. And because of the importance of a shared measurement system, appropriate measures can help to ensure the accountability of and ultimately integrate the efforts of public health, the health care delivery sector, and other public and private entities in the community to improve population health. Yet despite its importance, population health measurement efforts in the United States are poorly developed and uncoordinated. COLLABORATIVE MEASUREMENT DEVELOPMENT: To achieve the potential of the population health perspective, public health officials, health system leaders, and others must work together to develop sets of population health measures that are suitable for different purposes yet are harmonized so that together they can help to improve a community's health. This begins with clearly defining the purpose of a set of measures, distinguishing between outcomes for which all share responsibility and actions to improve health for which the health care sector, public health agencies, and others should be held accountable. FRAMEWORK FOR POPULATION HEALTH MEASUREMENT: Depending on the purpose of the analysis, then, measurement systems should clearly specify what to measure-in particular the population served (the denominator), what the critical health dimensions are in a measurement framework, and how the measures can be used to ensure accountability. Building on a clear understanding of the purpose and dimensions of population health that must be measured, developers can then choose specific measures using existing data or developing new data sources if necessary, with established validity, reliability, and other scientific characteristics. Rather than indiscriminately choosing among the proliferating data streams, this systematic approach to measure development can yield measurement systems that are more appropriate and useful for improving population health.

Although the phrase “population health” has multiple meanings, there are a number of commonalities in what might be called the “population health perspective.” It considers a broad set of determinants (environmental, social, economic, cultural, behavioral, biological, as well as clinical services) in improving the distribution of health and well-being outcomes (well-being, and functioning; and also death, disease, and injury). This perspective recognizes that responsibility for population health outcomes is shared but accountability is diffuse. The shared responsibility arises from the many upstream factors that influence population health and the opportunities to address them. The diffuse accountability, on the other hand, reflects the reality that although there are many possibilities for upstream interventions, the entities that take them on vary from community to community. To improve population health, communities must establish and nurture partnerships in a system designed to achieve collective impact. Although conceptually similar to the “public health perspective,” this definition differs in at least two respects: it is less directly tied to governmental health departments; and it explicitly includes the health care delivery system, which is sometimes seen as separate from or even in opposition to governmental public health.1 Indeed, as described in more detail below, some of the current applications of population health thinking apply primarily to the health care delivery setting rather than to public health.

However defined, measurement is critical in the population health perspective. For instance, recognizing the challenges of improving health outcomes that are the results of a complex set of factors—many of which are outside the health care system per se—a new report from the Institute of Medicine’s (IOM) Roundtable on Value & Science-Driven Health Care stresses the importance of population health measurement to ensure accountability in order to improve the quality of health care and population health outcomes, and also documents the inadequacy of current measurement systems:

Without a strong measurement capability, the nation cannot learn what initiatives and programs work best, resources cannot be guided toward the most promising strategies, and there is little ability to promote accountability in results… Current measurement initiatives focus on health care quality as it affects individuals, often on narrow or technical aspects of care, which encourages improvement only on those areas being measured. Yet the goals of the health system are broader, including health outcomes at the individual and population level, the quality of care that is delivered, cost and resource use by the system, and engagement of patients and the public…These areas are interconnected, and changes to any particular area would likely have effects on the others. Furthermore, there are multiple factors that influence a person’s health, many of which lie outside the traditional health system.2

Similarly, a series of reports from the IOM’s Roundtable on Population Health Improvement, building on population health payment models and delivery system reforms in the Patient Protection and Affordable Care Act (ACA), all call for new approaches to population health measurement.3,4,5 The IOM also notes that an unprecedented wealth of health data is providing new opportunities to understand and address community level concerns, and that the sharing and collaborative use of data and analysis is essential for the integration of primary care and public health in the interest of population health.6

From a theoretical perspective, population health measurement is important because a “shared measurement system” is one of the five conditions that Kania and Kramer7 conclude—in their synthesis of effective means of achieving “collective impact”—are necessary for large-scale social change. The other four are the following: a common agenda, mutually reinforcing activities, continuous communication, and backbone support organizations—all of which rely to some extent at least on a shared measurement system. Kania and Kramer8 write that agreement on a common agenda is illusory without agreement on the ways success will be measured and reported. Rather, collecting data and measuring results consistently on a short list of indicators at the community level and across all participating organizations not only ensures that all efforts remain aligned, it also enables the participants to hold each other accountable and learn from each other’s successes and failures.

However, despite these and many similar calls for better measurement from both the health care delivery sector and public health, there is no consensus on how to measure population health. For example, in their analysis of 12 successful partnerships between hospitals and public health, Prybil and colleagues found that these partnerships continue to be challenged in developing objectives and metrics and in demonstrating their linkages with the overall measures of population health on which they have chosen to focus.9 Thus, the goal of this paper is to summarize the current status of population health measurement and suggest a number of ways to advance the national dialogue on this critical issue. There is no single answer to this question—rather, different measurement approaches are necessary depending on purpose and context. We can assume, however, that there is value in harmonizing these approaches around central concepts of population health.

Consistent with best practices in health sector performance measurement, as reflected for instance in the criteria used by the National Quality Forum (NQF) to evaluate health care quality measures,10 this paper begins by identifying the major goals and objectives of population health measurement. These range from efforts focused on the health care delivery system to initiatives such as Community Health Needs Assessments (CHNA) that are explicitly designed to coordinate the efforts of health care-, public health-, and other community organizations. The next sections of this paper looks at which aspects of population health to measure and how to measure them. The final section addresses the validity, reliability, and other scientific characteristics of population health measures.

Goals and Objectives of Population Health Measurement

Beyond the ACA’s most prominent provisions to improve access to health care, there are many that aim to improve population health,11 each of which has implications for kinds of population health measures that are needed and how they will be used. This paper first looks at two approaches focused on the health care delivery system: Accountable Care Organizations (ACOs), and “population health management.” The paper then discusses a series of initiatives that can be categorized as “value-based purchasing” and a new requirement of nonprofit hospitals to prepare CHNA to address the needs of the total population of the geographic areas they serve.

These examples represent different steps in which populations are defined along a continuum that ranges from where they receive their health care to where they live. For measurement purposes, clarity about the “denominator” is important, and is addressed below. There are also important differences in the substantive issues stressed at different points on this spectrum, with much more focus on quality of care and value at the population health management end and on disparities at the total population end. But the most important distinction between the population health and the public health perspectives is that the former explicitly includes the health care delivery system as an important factor in improving population health rather than seeing it in opposition to governmental public health.12 To reflect this, we first look at the similarities rather than stress the differences.

Accountable Care Organizations (ACOs)

One of the most prominent ways that the ACA seeks to improve population health is through changes in the health care delivery system that incentivize providers to take responsibility for population health outcomes. For example, ACOs are groups of physicians, hospitals, and other health care providers that agree to assume responsibility for the care of a clearly defined population of Medicare beneficiaries. ACOs that succeed in both delivering high-quality care and reducing the cost of that care share in the savings they achieve for Medicare. Managing this “shared savings” program, therefore requires a set of measures of the quality of care provided and the health outcomes achieved in the ACO population.13

The first set of ACO performance measures (Table 1) was issued by the Centers for Medicare and Medicaid Services (CMS) in 2011, and includes 33 measures in four domains: patient and caregiver experience (7 measures), care coordination and patient safety (6 measures), preventive health (8 measures), and at-risk populations and frail elderly health (12 measures).14 According to Berwick,15 the final measures—chosen from a list of measures that was almost twice as long as the final list of measures—represent a compromise between the optimal and the feasible. Indeed, commenting on the ACA’s accountable care provisions, Fisher and Shortell16 had written that the limitations of current approaches to performance measurement are well recognized. Existing measures often assess individual clinicians and silos of care, focus largely on processes of questionable importance, are imposed as an add-on to current work, and require burdensome chart reviews and auditing or reliance on out-of-date administrative claims data. The result, they conclude, is a performance measurement system that often provides little useful information to patients or clinicians, reinforces the fragmentation that pervades the United States health care system, and reinforces physicians’ perception that measurement is a threat.

Table 1.

Accountable Care Organization (ACO) Quality Measures

Domain	Sample Measures
Patient/caregiver experience (7 measures)	*CAHPS: Getting Timely Care, Appointments, and Information
	CAHPS: Patients’ Rating of Doctor
	CAHPS: How Well Your Doctors Communicate
Care coordination/patient safety (6 measures)	Risk-Standardized, All Condition Readmission
	Ambulatory Sensitive Conditions Admissions: Chronic Obstructive Pulmonary Disease or Asthma in Older Adults
	Ambulatory Sensitive Conditions Admissions: Congestive Heart Failure
Preventive health (8 measures)	Influenza Immunization
	Mammography Screening
	Screening for High Blood Pressure
At-risk population:
Diabetes (6 measures)	Hemoglobin A1c Control (<8 percent)
	Blood Pressure <140/90
Hypertension (1 measure)	Controlling High Blood Pressure
Ischemic Vascular Disease (2 measures)	Complete Lipid Panel and LDL Control (<100 mg/dL)
Heart Failure (1 measure)	Beta-Blocker Therapy for Left Ventricular Systolic Dysfunction (LVSD)
Coronary Artery Disease (2 measures)	Drug Therapy for Lowering LDL-Cholesterol

Note:

CAHPS is Consumer Assessment of Healthcare Providers and Systems. Source: CMS13

To address these issues, Fisher and Shortell17 propose an alternative measurement system based on advances in the science of improvement and progress in health information technology that would build on different levels of ACOs based on different payment models, which would require differing levels of organizational structure. For example, level 1 ACOs—those without electronic health records (EHRs) or well-established patient registries—could rely in the near term on the meaningful measures that can be ascertained from claims data (cancer screening and diabetes testing, for example). These ACOs would be expected to progress rapidly to report on a more advanced set of measures, like selected health outcomes such as blood pressure control, patient-reported care experience measures (e.g., after-hours access), and total costs of care. Level 2 ACOs—those with site-specific EHRs and registries—might be expected to add more advanced measures such as patient-reported health outcomes for selected conditions. Level 3 ACOs—those with comprehensive EHRs across all sites of care—could be required to test and implement measurement systems that support practice improvement and accountability in such difficult areas as informed patient choice and health outcomes for a broad array of conditions.

Population Health Management

Whether they are part of an ACO or not, many health care delivery systems are shifting from a focus on the diagnosis and treatment of disease to a population health management approach that emphasizes wellness and that views acute care as only one component in a delivery system designed to provide value over a patient’s lifespan and across targeted populations. According to the Institute for Health Technology Transformation,18 the principles and best practices of population health management include data collection, storage, and management; population monitoring and stratification; patient engagement; team-based interventions; and outcomes measurement.

“Health reform is all about practicing population-based medicine. And the only way we’re going to bend the cost curve is by keeping people out of the hospital, reducing unnecessary utilization,” said David Nash, dean of the Jefferson School of Population Health.19 In this use, population health improvement emphasizes the central role of the primary care provider, a fully engaged and activated patient, and care coordination.20 Populations can be defined in terms of age, income, geography, community, employer, insurance coverage, health status, and by combinations of these factors. McAlearney21 notes that specifying a population allows a health system to design a management program that meets the needs of the group. To achieve efficiencies, population health management approaches often focus on patients with one or more chronic diseases.

Blue Cross and Blue Shield of Louisiana’s (BCBSLA) Quality Blue program illustrates the key attributes of a population management system. As shown in Figure 1, BCBSLA employs “Quality Navigators” who act as the communication hub to facilitate a variety of patient services. Both the Quality Navigators and the practice are equipped with MDinsight, a cloud-based data aggregation tool that pulls together multiple data sources in near real time so both parties can view and act on the same information. This includes patient-level data needed to facilitate clinician decision-making at the point of care. In addition, MDinsight uses the same data to create population-level performance measures for each practice, as illustrated in . Practices are measured and benchmarked at quarterly intervals against other BCBSLA network practices, regional practices in the southeastern United States, and national evidence-based standards in order to improve the quality of care for the population served by each practice.22

Figure 1.

Blue Cross and Blue Shield of Louisiana (BCBSLA) Quality Blue Program Design

Source: Carmouche22

According to Jonathan M. Niloff, vice president and executive medical director of Population Health for the McKesson Corporation, population health management requires access to relevant data and using that data to understand and manage the population.23 Data are needed for two purposes: to manage the care of individual patients, which is often accomplished with an integrated electronic medical or health record; and data analytics to identify high-risk patients and to manage the care that the system provides to these groups of patients. More than half of the organizations McKesson surveyed recently will be making investments to integrate clinical data and add data analytics capabilities. Too often ambitious plans for population health cannot be achieved with the limited data that is typically available, Niloff said.24

Communitywide Approaches to Value-based Purchasing

Beyond ACOs per se, Hester25 noted that the United States health care system is transitioning from a payment system driven by the volume of health care services provided to a payment system based on value defined in terms of the Institute for Health Care Improvement’s “Triple Aim” goals: (1) improved health of populations, (2) improved patient experience for those who need care, and (3) reduced trends in total per capita health care expenditures.

Shortell26 notes that the key to this transition will be changes in payment; if the goal is to improve population health, then we must pay for it. He suggests that CMS can start by paying selected communities for meeting population-health improvement objectives. The core idea is to offer a risk-adjusted, communitywide population-health budget to an accountable entity for achieving predetermined quality and health status targets for, initially, a defined set of conditions. Hacker and Walker27 suggest that to fully meet the Triple Aim goals, including improving the health of a population, ACOs must embrace a broad community definition of population health and take steps to work collaboratively with community and public health agencies. As health care moves toward alternative and global payment arrangements, the need to understand the epidemiology of the patient population is imperative. Keeping the population healthy will require enhancing the capacity to assess, monitor, and prioritize lifestyle risk factors and social determinants of health that unduly affect health outcomes. Future financing and value-based purchasing, they recommend, should reward collaborations that result in population health improvements at the community level.

Hester28 writes that both private and public payers are testing new payment models at scale, and payers are learning to align their financial models with each other in order to accelerate the transformation of the system. The complexity and relative weakness of key building blocks of population health payment models, however, create the threat that population health will not be integrated into the new payment system in a meaningful way. One problem is that the existing set of population health measures and data sets for process improvement, accountability, and payment are neither well developed nor implemented to provide timely data with the needed granularity. There is significant confusion about the distinction between measuring quality of care and measuring population health, even though they are two very different dimensions of performance. Moreover, the population health measures incorporated in current payment models focus on clinical preventive services. A more robust set of population health measures would track progress in upstream determinants of health, intermediate outcomes in disease burden and patient-reported quality of life, and final outcomes in quality-adjusted life expectancy.29

Auerbach and colleagues30 see the ACA’s State Innovation Models (SIMs) as an opportunity to test new alignments, payments, and incentives that focus our current delivery system on achieving health for all. For these approaches to work, measures of population health that focus not only on clinical preventive services but also on upstream or population-level determinants of health and health outcomes are needed for a communities as defined by a geographic region.

According to Auerbach and colleagues,31 the major SIMs currently being tested focus primarily on controlling total costs of care delivery and improving the patient experience, and do not significantly reward improvements in population health. They include measures of population health that focus on clinical preventive services but that do not track upstream or higher-level determinants of health, such as school days missed, patient-reported health statuses, or health outcomes for a community as defined by a geographic region. What is needed, they say, is for the states receiving CMS funding to test and implement SIMs to conduct pilots and experiments that are focused on improving population health. These pilots should be structured with goals and actions at the community level and should integrate clinical services, public health programs, and community-based initiatives targeting the upstream determinants of health. They should include the implementation of a core set of metrics for tracking changes in population health for both program improvement and accountability, and should also include aligned payment models for key stakeholders that reward and incentivize demonstrated improvements in the health of the community. An optimal approach would involve a portfolio of measures paired with financial incentives that are substantively balanced to meet the prioritized needs of the community, designed to capture and link both clinical and communitywide measures for process and outcome, and intended to produce both short- and long-term impacts.

For example, a balanced portfolio might include both practice-and communitywide measures and intentionally seek ones with relatively quick positive and measurable health benefits and cost-saving outcomes, such as effective prevention interventions (e.g., influenza vaccinations, alcohol screening, and brief counseling), asthma intervention measures (which decrease emergency room visits and hospitalizations), and behaviors responsive to city- or statewide interventions (e.g., tobacco use levels). Mental health measures could be included (e.g., Patient Health Questionaire-9 for depression, which can be used for screening and follow-up). Alternatively, there might be complementary metrics for which significant benefits may be seen over a longer period, such as the prevalence of risk factors (e.g., obesity) and illness (e.g., diabetes, HIV), and summary measures of population health such as the Centers for Disease Control and Prevention’s (CDC’s) healthy days.32

Community Health Needs Assessments (CHNA)

Another ACA approach to improving population health stands out as having the potential for coordinating the efforts of the health care delivery sector, public health agencies, and other community organizations to improve population health outcomes. Intended to leverage the community benefits that hospitals are required to spend (estimated at $12.6 billion in 2008) to improve population health, all nonprofit hospitals are now required to work with public health agencies and other community organizations to conduct a CHNA at least every three years and to also adopt an implementation strategy describing how identified needs will be addressed. Under Internal Revenue Service (IRS) regulations, both the CHNA and the associated implementation plan are expected to include population health measures and be available to the public.33 These developments have the potential to leverage the strengths and resources of both the health care and public health systems to create healthier communities.34

States such as New York, Massachusetts, Wisconsin, and North Carolina have had CHNA requirements—and data systems to support them—for some time. These initiatives, however, have generally applied to public health rather than to hospitals or the health care delivery system. In this sense, the ACA’s imposition of a mandate that hospitals prepare CHNAs and implementation strategies creates a unique opportunity to align public health and health care efforts to improve population health.35

Triggered in large part by the new IRS CHNA requirements, community-level data are increasingly available through programs such as the County Health Rankings,36 the Healthy Communities Institute,37 CHNA.org,38 Dignity Health’s Community Need Index,39 as well as organizations such as the Association for Community Health Improvement.40 In addition, Community Commons41 and the Healthy Communities Institute42 have developed CHNA toolkits or models. However, despite such guidance, implementation of this mandate varies markedly, and there is a strong need to further develop and refine methods to use CHNAs to catalyze and coordinate the community health improvement activities of hospitals, public health agencies, and other organizations.43 Indeed, in their analysis of successful partnerships between hospitals and public health, some of which specifically capitalized on CHNAs, Prybil and colleagues conclude that to assess a partnership’s progress toward its goals and fulfill its accountability to stakeholders the partnership leaders must adopt measures (intermediate and long-term); implement evidence-based strategies; compile pertinent data; and conduct sound, objective evaluations. Moreover, they recommend that to enable objective, evidence-based evaluation of a partnership’s progress in achieving its mission and goals and to fulfill its accountability to key stakeholders, the partnership’s leadership should specify the community health measures they want to address, the particular objectives and targets they intend to achieve, and the metrics and tools they will use to track and monitor progress.44

The IRS requirements call for population health measures to serve two purposes. First, tax-exempt hospitals must conduct a CHNA once every three years. A “CHNA” is defined as “a written document developed for a hospital including a description of community served by the hospital, a statement of existing health care resources within the community available to meet community health needs, and a list of the prioritized health needs identified through the process.” Second, IRS also requires the development of an implementation strategy—possibly developed in collaboration with other organizations—to meet the community health needs identified through the CHNA.45 In preparing these reports, hospitals are expected to take into account input from persons who represent the broad interests of the community served, including those with special knowledge of or expertise in public health. At a minimum, hospitals must consult with “at least one state, local, tribal or regional governmental public health department with knowledge, information, or expertise relevant to the health needs of the community.”46

Similarly, Public Health Accreditation Board (PHAB) standard 1.1 calls on health departments seeking accreditation to participate in or conduct a collaborative process resulting in a comprehensive Community Health Assessment (CHA). The purpose of this process is to learn about and describe the health status of the population, to identify areas for health improvement, to determine factors that contribute to health issues, and to identify assets and resources that can be mobilized to address population health improvement. Less distinctly, other PHAB standards also require health departments to conduct a comprehensive planning process resulting in a community health improvement plan, to assess health care service capacity and access to health care services, to identify and implement strategies to improve access to health care services, and to use a performance management system to monitor achievement of organizational objectives.47

The challenge of managing a shared responsibility for the community’s health, however, is that given the broad range of factors that determine health, no single entity can be held accountable for health outcomes. Indeed, identifying accountability for specific actions is an essential component of both the Community Health

Improvement Plan required by the IRS regulations and the comprehensive planning process in the PHAB standards. Although they use different terminology, both the IRS requirements and the PHAB standards call for two sets of population health measures: (1) measures of population health outcomes for which health care providers, public health agencies, and many other community stakeholders share responsibility; and (2) performance measures capable of holding these same entities accountable for their contributions to population health goals. This same distinction was articulated in Improving Health in the Community,48 where the community health profile—another term for a CHNA or CHA—is highlighted in blue in Figure 3 in the upper “Problem Identification and Prioritization Cycle” in a Community Health Improvement Process (CHIP) schematic. The central idea is that these sets of community health indicators are intended to summarize a community’s overall health status, for which health care providers, public health agencies, and many other community stakeholders share responsibility. Such profiles should reflect the diversity of the determinants of health, and can be used to set priorities among issues in a community’s health improvement plan.

Figure 3.

Institute of Medicine (IOM) Community Health Improvement Process (CHIP)

Source: Adapted from IOM48

This fundamental distinction between population health outcome measures and accountability-oriented performance measures also appears in the Mobilizing for Action through Planning and Partnership (MAPP) model49 cited by the public health officials in their comments on the IRS regulations. In particular, this approach calls for a community health status assessment and three other assessments before identifying strategic issues, formulating goals and strategies, and developing a plan-implement-evaluate action cycle that includes relevant performance measures. Similarly, the Association for Community Health Improvement’s (ACHI) Community Health Assessment Toolkit includes as one of its six steps “Planning for Action & Monitoring Progress.”50 Data on community health outcomes are collected (step 3) and priorities for action are set (step 4); and step 6 includes defining achievable goals, objectives, and strategies; collecting information on existing community programs and efforts; identifying evidence about effective interventions, and developing an action plan and evaluation strategy.

For instance, in Montgomery County, Maryland, the Department of Health and Human Services, all five not-for-profit hospitals, the Primary Care Coalition of Montgomery County (representing safety net clinics), and other health care providers, government agencies (including the school system, land-use planning agency, and recreation department), and community organizations all participate in Healthy Montgomery, an ongoing community-driven process to identify and address key priority areas. Six priorities have been identified (behavioral health, obesity, diabetes, cardiovascular disease, cancers, and maternal and infant health). The Healthy Montgomery steering committee adopted 37 core measures that can be monitored over time and disaggregated to the relevant social units, can include behaviors and other health determinants as well as health outcomes, and can address the concerns of the hospitals and existing Healthy Montgomery priority areas.51

The hospitals in Montgomery County, in turn, build their own CHNAs on Healthy Montgomery. Holy Cross Hospital, for instance, serves a sizable immigrant population, and the hospital’s community benefits primarily target access for underserved populations—financial, geographic, and ethnic. As a result, the hospital shifted its community benefits focus over the past few years, steering away from general programs like health fairs toward more targeted approaches directed at ethnic and elderly populations. Holy Cross differentiates itself from other hospitals, however, by developing a comprehensive and detailed community benefit strategy with specific quarterly deliverables, just as it does for personnel, infrastructure, and financial planning. For example, to address Healthy Montgomery’s Maternal and Infant Health priority, Holy Cross chose to focus on outreach efforts that improve health status and access for underserved, vulnerable mothers. Holy Cross evaluates success by monitoring the number of admissions to the Maternity Partnership, the number of perinatal class encounters, the percentage of low birth weight infants, and the reduction in the infant mortality rate.52

Both improvement plans and their associated performance measures must be tailored to a community’s health needs, the resources that are available, and the actions that health care providers, health departments, and other entities are willing to take and be accountable for. As discussed below, the IOM’s Improving Health in the Community53 proposes sample performance measure sets that communities can adopt for this purpose. As discussed in the “accountability” section below, For the Public’s Health: The Role of Measurement in Action and Accountability54 lays out a very useful “Framework for Accountability” and suggests specific measures and the stakeholders (or accountable entities) associated with them.

Developing Population Health Measures

The world of health care is awash with data. Whether one reads Computerworld (e.g., “How big data will save your life”),55 Healthcare Executive (e.g., “The power of analytics: Harnessing big data to improve the quality of care”),56 or Health Affairs (e.g., “Creating value in health care through big data”),57 most people now accept the idea that electronic clinical data and other health records can be used to manage and improve the processes, outcomes, and the quality of health care. But although the potential of these data to improve population health is frequently cited, the creation of population health measures—for any of the purposes described above—is neither automatic nor straightforward. To advance the use of these data, this section addresses a number of technical issues that must be considered in the development of population health measures. This discussion begins with the denominator, the most important question to address in specifying population health measures. It then describes the need for a measurement framework, including the careful specification of measures and the target population that clarify accountability for expected actions. Then the data are considered that are currently available in the health care delivery sector and for geographically defined populations, concluding with a discussion of validity and reliability in population health measures.

Population and Denominator

The focus on the improving the distribution of health and well-being and their determinants in defined populations is the essential feature of the population health perspective. Careful consideration of the best ways to improve health outcomes often requires that health care providers look beyond patients who seek care for existing conditions to the populations from which they are drawn, sometimes implementing community-based interventions. Indeed, one of the major long-term benefits of the increasing adoption of population health perspective in the ACA and in practice is an increased focus on upstream factors, promoting health, and preventing disease before it occurs.

This common focus unites a variety of population health approaches that fall along a spectrum based on how that population is defined. At one extreme, the concern is for health outcomes in populations defined by geography, what Jacobson and Teutsch call “total population health.”58 The County Health Rankings and most community health assessments conducted by state and local health departments typically take this approach. Population health management approaches that focus on patients in a health care system with specific chronic diseases are at the other end of the spectrum. Not surprisingly, Noble and colleagues59 found that health professionals who work for ACOs most often viewed “population health” as referring to a defined group of their organization’s patients, while public health agency staff were more likely to consider population health from a geographical perspective.

Approaches focusing on accountability for health outcomes in populations defined by health care delivery systems such as ACOs or the Institute for Healthcare Improvement’s (IHI’s) Triple Aim model represent intermediate positions along this spectrum. The communitywide approaches to value-based purchasing described above can be seen as a way to broaden the denominator for population health from patients defined by their disease status and where they receive care to entire communities. Focusing on implementing the Triple Aim in ACOs, for instance, Hacker and Walker60 call for a broader “community health” definition that could improve relationships between clinical-delivery and public health systems and health outcomes for communities. Addressing similar issues, Gourevitch and colleagues61 suggest potential innovations that could allow urban ACOs to accept accountability, and rewards, for measurably improving population health.

The current variation in the definition of “population” means that measurement systems must begin by determining the denominator that best describes the population whose health is being monitored. The choice of the denominator also has important implications for accountability that may not be obvious. Consider something as simple as the coverage rate for influenza immunization, an important preventive service. The NQF62 recommends that the denominator for this rate be defined as the number of persons of the appropriate age for the vaccine (currently over 18 months of age) in one of two ways: (1) in a facility, agency, or practice with an encounter between October 1 and March 31; or (2) for health plan measures, enrolled with a plan between October 1 and March 31.

The difference between these two is that health plans are responsible for ensuring that everyone in the plan during the period when immunization is appropriate is included in the denominator. Hospitals or physician practices, on the other hand, are only responsible for ensuring that patients who are seen during that period are immunized. It is interesting to note that the initial ACO performance measures issued by CMS63 adopt the first of the two NQF options—the denominator for the measures is patients seen for a visit between October 1 and March 31—rather than all of the patients for whom the ACO is nominally “accountable.” Shouldn’t ACO’s be responsible for ensuring that all of their enrolled patients are vaccinated, not just those who see a provider between October 1 and March 31?

Moreover, despite the ACA and other factors leading the health care delivery system toward a total population health approach, different and overlapping definitions of a population pose a major challenge. Hospital and ACO service areas generally do not correspond to county or other geopolitical boundaries. The lack of overlap goes in both directions; in some metropolitan areas, health care service areas include multiple counties, and simultaneously only parts of some counties. For instance, as can be seen in Figure 4, the primary service area of Holy Cross Hospital in Silver Spring, Maryland includes parts of both Montgomery and Prince George’s Counties, but the hospital also has referrals from the remainder of those counties, as well as neighboring Washington, D.C. Finding ways to bridge these mismatched jurisdictions is one of the primary practical challenges of population health measurement.64 Having data available by ZIP code or other small geographical areas is a step toward addressing these issues.

Figure 4.

Holy Cross Hospital Service Areas

Source: Holy Cross Hospital52

Measurement Framework

Considering the range of uses and applications of population health measures discussed in the previous section, two critical issues emerge. First, population health measures must take a broad view of health and its determinants. The IHI “composite model”65 reproduced in Figure 5 represents this perspective well, with its emphasis on health promotion and disease prevention as well as interventions focusing on upstream factors rather than outcomes. Second, the measures should have the capacity to bring together public health, health care, and other stakeholders. In this regard the IHI composite model’s explicit recognition of the role of health care and of personal preventive services as part of the population-health production system is especially important. Similarly, Friedman and Parrish66 provide a conceptual description of the “population health record” assimilating information and statistics from diverse data sets and sources. If based upon an explicit population health framework, this record would provide a comprehensive view of population health to support exploratory and other analyses of health and factors that influence it.

Figure 5.

Institute for Healthcare Improvement (IHI) Composite Model of Population Health

Source: Adapted from IHI74

These considerations suggest that, to be maximally useful, population health measures require a scientifically valid measurement framework. For instance, summarizing the work of the “social indicators movement,” Andrews has identified the key characteristics of a set health indicators as follows: “a limited yet comprehensive set of coherent and significant indicators which can be monitored over time, and which can be disaggregated to the level of the relevant social unit.”67 All of the limitations of current population health measures discussed in the previous section can be summarized in these terms.

Obviously, there are a number of tensions in these criteria. The set of measures must be limited in number—otherwise users lose sight of the big picture—but yet comprehensive enough to cover all of the important issues, including the determinants as well as health outcomes. Composite measures such as indicators of preventable chronic disease mortality can be useful in this context. The individual measures in the set must be coherent so that they work together to tell the community’s health story, yet be significant enough to gain policymakers’ attention. Stoto68 describes how age can be used as an organizing structure, reflecting the relative ease in identifying priority issues within each age group. If the measures cannot be monitored over time, they are not very useful for tracking progress so adjustments can be made in population health improvement plans. Census data that are available for counties may not be useful for tracking population health improvements. On the other hand, as discussed in more detail below, much of the available population health data—whether based on sample surveys, case reports, or mortality statistics—are not reliable for small geographical areas.

For population health measures, “disaggregated to the level of the relevant social unit” has many meanings. For health outcomes, policymakers are interested in addressing disparities among groups defined by race and ethnicity, but also social and economic status, gender, and geography. In principle, disparities can be identified by comparing the same health outcomes measures for subpopulations defined by race, ethnicity, socioeconomic status, and so. This allows analysts to identify absolute and relative differences in rates. If time series data are available, one can also see whether gaps are being closed or not.69 In practice, however, the ability to calculate disaggregated rates is limited by the availability of racial, ethnic, socioeconomic, and other identifiers in the data, and by small sample sizes (addressed further below).

Geographical variation is also important, whether it reflects inequities in population health factors or “hot spots” where access to health care is poor. And to the extent that population health measures are needed to drive improvement plans, the data must be disaggregated to reflect the health determinants, from individual health care providers to neighborhood factors. Dignity Health’s Community Health Need Index (CNI) is a useful indicator of the severity of the socioeconomic barriers to health care access in a given community. This index is based on evidence about the economic and structural barriers related to income, culture and language, education, insurance, and housing that affect overall health.70

The indicators chosen by the IOM71 for the State of the USA (SUSA) Health Indicators website (Table 2), provide a good illustration of the use of a measurement framework. In order to keep the measures manageable by limiting the number to 20, each indicator was required to demonstrate that it has the following characteristics:

Table 2.

Institute of Medicine’s (IOM’s) State of the USA Health Indicators

Health Outcomes

Life expectancy at birth (number of years that a newborn is expected to live if current mortality rates continue to apply)

Infant mortality (deaths of infants ages under 1 year per 1,000 live births)

Life expectancy at age 65 (number of years of life remaining to a person at age 65 if current mortality rates continue to apply)

Injury related mortality (age-adjusted mortality rates due to intentional and unintentional injuries)

Self-reported health status (percentage of adults reporting fair or poor health)

Unhealthy days—physical and mental (mean number of physically or mentally unhealthy days in past 30 days)

Chronic disease prevalence (percentage of adults reporting one or more of 6 chronic diseases [diabetes, cardiovascular disease, chronic obstructive pulmonary disease, asthma, cancer, and arthritis])

Serious psychological distress (percentage of adults with serious psychological distress as indicated by a score of > 13 on the K6 scale, with scores ranging from 0–24)

Health-Related Behaviors

Smoking (percentage of adults who have smoked > 100 cigarettes in their lifetime and who currently smoke some days or every day)

Physical activity (percentage of adults meeting the recommendation for moderate physical activity [at least 5 days a week for 30 minutes a day of moderate intensity activity, or at least 3 days a week for 20 minutes a day of vigorous intensity activity])

Excessive drinking (percentage of adults consuming 4 [women], 5 [men], or more drinks on one occasion; consuming more than an average of 1 [women] or 2 [men] drinks per day during the past 30 days)

Nutrition (percentage of adults with a good diet [conformance to federal dietary guidance] as indicated by a score of > 80 on the Healthy Eating Index)

Obesity (percentage of adults with a body mass index > 30)

Condom use (proportion of youth in grades 9–12 who are sexually active and do not use condoms, placing them at risk for sexually transmitted infections)

Health Systems

Health care expenditures (per capita health care spending)

Insurance coverage (percentage of adults without health coverage via insurance or entitlement)

Unmet medical, dental, and prescription drug needs (percentage of [noninstitutionalized] people who did not receive or delayed receiving needed medical services, dental services, or prescription drugs during the previous year)

Preventive services (percentage of adults who are up-to-date with age-appropriate screening services and flu vaccination)

Childhood immunization (percentage of children aged 19–35 months who are up-to-date with recommended immunizations)

Preventable hospitalizations (hospitalization rate for ambulatory care-sensitive conditions)

Source: Adapted from IOM71

A clear importance to health or health care;

The availability of reliable, high quality data to measure change in the indicators over time;

The potential to be measured with federally collected data; and

The capability to be broken down by geography, populations subgroups including race and ethnicity, and socioeconomic status.

Based on these criteria, the IOM noted that health outcomes were chosen because they reflect the well-being of the population. Health-related behaviors such as smoking and nutrition were chosen because of their importance in determining health outcomes—for example, behavioral patterns are responsible for 40 percent of the premature deaths in the United States. Finally, the category “health system performance”—including the health care and public health systems—was selected because access to available services is critical to the treatment and prevention of disease and illness. The IOM committee also considered characteristics of the social and physical environment, such as income and air quality, but decided not to include them in its list of 20 health indicators because they would be covered in other SUSA domains. These factors were, however, included in the County Health Rankings (see below).

On the delivery sector side, Bankowitz and colleagues72 note that many of today’s measures are inadequate to the task of assessing and paying for value as Medicare as well as many private sector insurers, providers, and employers transition to ACOs and other value-based payment mechanisms. Current measures focus on process and clinical outcomes, as opposed to health status, for instance, and most measures are add-ons to current work rather than an integral part of the care process, requiring manual chart reviews and retrospective data analysis. These inadequacies create opportunities to implement new measures that will be more meaningful to consumers, clinicians, purchasers, and policymakers. To avoid a proliferation of measures that are inconsistent or questionable in terms of assessing value, they propose a framework to define specific measures for each component of value—health outcomes, patient experience, and per capita cost.

Addressing similar issues at the geographical level, the Health Policy Institute of Ohio (HPIO) notes that improving health care value—looking at the relationship between health outcomes and health costs—is critical to evaluating efforts to improve health, but that current tracking efforts are too narrow in scope and do not factor in determinants outside of the health care system such as social and economic factors. Consequently, HPIO is developing a concise and comprehensive dashboard of health outcome and cost measures to track Ohio’s progress in improving health value. This includes tracking population health outcomes, health costs, health care system performance, public health system performance, and health access and evaluating Ohio’s social, economic, and physical environment.73

Example: IHI Triple Aim

The IHI’s Guide to Measuring the Triple Aim74 provides a good example of population health measurement framework focused on the health care delivery sector. The choice of measures is based on four basic principles:

The frame for the Triple Aim is a population; and the measures, especially for population health and per capita cost, require a population denominator. Populations served by a Triple Aim initiative might be either a total population of a geopolitical area or a subpopulation defined as those served by a particular health system.

Tracking data over time helps to distinguish between common cause and special cause variation, to gain insight into the relationship between interventions and effects, and to better understand time lags between cause and effect.

Measurement for the Triple Aim can be constructed hierarchically, with top-level population-outcome measures for each dimension of the Triple Aim, and with related outcome and process measures for projects that support each dimension.

While data tracked and plotted over time help to measure improvement, benchmark or comparison data enable comparisons with other systems. Benchmarking is easier if the measures selected are standardized and in the public domain.

The proposed measures based on these principles are summarized in Table 3, and are described in more detail and illustrated in the Guide to Measuring the Triple Aim. In particular, the population health component includes measures of health outcomes such as mortality, health and functional status, and healthy life expectancy; disease burden, including the incidence and prevalence of major chronic conditions; and behavioral and physiological factors.75

Table 3.

Institute for Healthcare Improvement (IHI) Triple Aim Population Health Measures

Dimension of the IHI Triple Aim	Outcome Measures
Population Health	Health Outcomes: Mortality: Years of potential life lost; life expectancy; standardized mortality ratio Health and Functional Status: Single-question assessment or multidomain assessment Healthy Life Expectancy: Combines life expectancy and health status into a single measure, reflecting remaining years of life in good health
	Disease Burden:Incidence (yearly rate of onset, average age of onset) and prevalence of major chronic conditions
	Behavioral and Physiological Factors: Behavioral factors include smoking, alcohol consumption, physical activity, and diet Physiological factors include blood pressure, body mass index (BMI), cholesterol, and blood glucose A composite health risk assessment score
Experience of Care	Standard questions from patient surveys such as the Consumer Assessment of Healthcare Providers and Systems questions on likelihood to recommend to others
	Set of measures based on key dimensions such as the Institute of Medicine’s (IOM’s) six aims for improvement: safe, effective, timely, efficient, equitable, and patient-centered
Per Capita Cost	Total cost per member of the population per month
	Hospital and emergency department utilization rate and cost

Source: Adapted from IHI74

Example: County Health Rankings

The Robert Wood Johnson Foundation/University of Wisconsin’s County Health Rankings76 is good example of a useful population health framework. Because the County Health Rankings measures are based on the latest publically available data for every county in the United States, this model also provides a practical framework for CHNAs required of hospitals by the IRS or CHAs required by the PHAB.

The rankings framework involves 33 specific measures that collectively describe a community’s health in terms of health outcomes and four categories of health determinants (see Figure 6). The health outcomes include one mortality measure (premature death, i.e. years of potential life lost before 75 years of age) and four morbidity measures (percentage of participants reporting fair or poor health, average number of physically and mentally unhealthy days in the past month, and low birth weight). The health factors cover health behaviors (9 measures addressing tobacco use, diet and exercise, alcohol and drug use, and sexual activity), clinical care (6 measures covering both access to and quality of care), social and economic (8 measures addressing education, employment, income, family and social support, and community safety), and the physical environment (5 measures covering air and water quality as well as housing and transport).

Figure 6.

County Health Rankings Population Health Measurement Framework

Source: University of Wisconsin Population Health Institute36

In order to rank the counties in each state, an overall Health Outcomes summary score is created as an equally weighted composite of the mortality and morbidity measures. An overall health factors summary score is a weighted composite of four components: health behaviors (30 percent), clinical care (20 percent), social and economic factors (40 percent), and physical environment (10 percent). The County Health Rankings designers note that there is no one correct formula or true set of weights that perfectly represents the health of a community. Rather they have used information from a wide variety of sources—scientific research, available data, expert opinion, and statistical analysis—to arrive at a set of easy to understand weights that reasonably reflect the different components and determinants of health. These weights are not perfect, but are regarded as reasonable estimates supported by the best available evidence balanced with the availability of health data and interpretability. Users more interested in specific aspects of a community’s health, perhaps because they are using the data as part of a CHNA, can use the 6 summary measures, or even the 33 underlying specific measures, in their analysis of their community’s health.

Accountability

Careful consideration of all of the factors that influence, and are effected by, health reminds us that population health must be seen as the shared responsibility of health care providers, governmental public health agencies, and many other community institutions (Figure 7).77 The challenge of managing a shared responsibility for the community’s health, however, is that given the broad range of factors that determine health, no single entity can be held accountable for health outcomes.

Figure 7.

Institute for Healthcare Improvement (IOM) Population Health Model

Source: IOM54

In this context, the IOM has noted that the absence of a robust and available set of health indicators makes it difficult to hold the health system accountable for improving population health. This problem is worsened because the roles and responsibilities of different parts of the health system—from governmental public health agencies to schools and hospitals, from transportation networks to local zoning departments, from community-based organizations to local and national businesses—are not clear. Indeed, the health care delivery sector, public health agencies, and other public and private organizations that can contribute to population health, are often—and correctly—characterized as silos. To address this problem, the IOM describes and makes recommendations for a measurement framework that provides the clear accountability needed to enable communities and policymakers to understand, monitor, and improve the contributions of various partners in the health system.78

Drawn from this IOM report, Table 4 illustrates how population health measures can be used to align efforts among health system stakeholders. Measurement of execution and outcomes of the agreed-on plans (strategies, interventions, policies, and processes) can clarify the shared responsibility for population outcomes, reveal the levels of effort and achievement needed to reach shared objectives, hold implementing agencies or stakeholders accountable for execution, and help identify necessary revisions to action plan. Distal health outcomes (death and diseases) are not useful in the context of accountability but can help the system to assess overall progress and to know where it stands. Jacobson and Teutsch79 make a similar point in distinguishing between measures of population health outcome and of population health improvement activities.

Table 4.

Examples of Measures of Common Agreed-on Strategies

Sample Measure	Accountable Entity
Number of employers who have voluntarily adopted and complied with smoke-free workplace policies	Business, nonprofits
Number of (nonchain) restaurants voluntarily posting or complying with requirements for disclosure of nutritional information	Business (retail)
School adherence to nutritional guidelines, including removal of some vending machine products	Schools
Planning and zoning decisions consistent with local needs	Planning department
Small-business compliance with smoking bans (something intermediate to) high school graduation rates	Schools, community-services agencies
Percentage of community housing that is affordable (give parameters)	Planning department, local government, developers
Percentage of community housing that is safe and livable (give parameters)	Police, planning, local government, community groups, faith-based organizations
Percentage of poor children (specify percentage of federal poverty level) who receive early-childhood interventions (from public health and other social-service agencies)	Public health agency, social services, nonprofit organizations, including advocacy groups and philanthropic organizations
Percentage of medical insurance plans that implement health-literacy education; percentage of medical insurance plans or medical providers that adopt health-literacy strategies and implement steps to increase cultural competence of their staff; measures of health literacy in adolescents	Clinical care, schools
Percentage of employers that provide wellness services to employees	Business, employers
Percentage of employers who adopt policies supportive of breastfeeding mothers (including dedicated, acceptable space and time to pump)	Business, employers
Percentage of baby-friendly (that is, breastfeeding-supportive) hospitals (specific parameters have been described elsewhere)	Clinical care

Source: Adapted from IOM54

Example: Performance Measures for Tobacco and Health

For example, consider the following sample-performance indicator set, drawn from IOM.80 It starts from the point where a community has chosen to focus on tobacco and health issues. Figure 8 is a simplified “driver diagram” that illustrates the primary and secondary drivers of the main outcome—tobacco-related mortality—as well as the process changes or interventions needed to bring these forces into alignment in a community. The corresponding measures are summarized in Table 5.

Figure 8.

Tobacco and Health Driver Diagram

Source: Adapted from Institute of Medicine Improvement (IOM)53

Table 5.

Sample Performance Indicators for Tobacco and Health

Sample Measure	Accountable Entity
Deaths from tobacco-related conditions	Shared responsibility
Smoking-related residential fires	Shared responsibility
Initiation of smoking among youth	Shared responsibility
Prevalence of smoking in adults	Shared responsibility
Ordinances to control environmental tobacco smoke	Local lawmakers
Local enforcement of laws on tobacco sales to youth	Local merchants and law enforcement
Tobacco prevention curricula in schools	Board of Education
Counseling by health care providers	Health care providers
Availability of cessation programs	Nongovernmental agencies, local health departments, employers, etc.
Health plan coverage for cessation programs	Employers that determine health plan coverage

Source: Adapted from IOM53

A community taking this approach would want to monitor tobacco-attributable mortality, which can be estimated even in small communities using CDC Smoking-Attributable Mortality, Morbidity, and Economic Costs (SAMMEC) program data.81 Tobacco-related mortality is useful to monitor to remind the community of the importance of the problem, but because of a long latency for cancer and other outcomes, this measure is not a good indicator of the impact of current prevention initiatives. Monitoring smoking-related residential fires can complement this as timelier outcome. Although the number of such fires in any community will be small, they can call attention to the problem by serving as sentinel events. As intermediate outcomes, or as drivers of tobacco-related mortality, it would also be important to measure both the initiation of smoking among youth and the prevalence of smoking in adults (to assess the cumulative effect of cessation, which is more difficult to measure directly).

Taken together, these measures monitor the community’s shared responsibility for tobacco-related outcomes. The following set of measures serves to assess the contributions of specific stakeholders. A measure of the prevalence or strength of ordinances to control environmental tobacco smoke, for instance, can hold local lawmakers responsible for passing legislation that, over the long run, can influence both smoking initiation and cessation in adults. Similarly, since national laws control tobacco sales to youth, measures of local enforcement of laws on tobacco sales to youth can represent the efforts of local merchants and law enforcement officials to address the problem in the community. Measures of the existence or the extent of tobacco prevention curricula in schools can serve as an indication of the school board’s commitment to addressing tobacco in the community.

More directly addressing the health care community, a measure of the extent to which providers counsel their patients about smoking cessation can serve to hold providers accountable for this effort. In any given community, groups such as the American Cancer Society or the American Lung Association, local health departments, hospitals, employers, or others can commit to providing smoking cessation programs. A measure of the availability of these programs in the community reflects those commitments. Finally, the set includes an indicator of health plan coverage for cessation programs that measures the commitment of those who purchase health insurance—primarily employers—to ensure that these programs are included. With the mandatory inclusion of prevention programs under the Affordable Care Act, this measure is no longer necessary.

Choice of Individual Measures

Whatever denominator and measurement framework are chosen, adopting the population health perspective surfaces a number of important conceptual issues in measurement that must be addressed and clarified. Organizations such as the Agency for Healthcare Research and Quality (AHRQ) and the NQF have well-developed concepts of quality and performance measurement for health care service providers, but efforts to translate this approach to population health settings are just beginning. As discussed above, one aspect of this involves identifying the relevant denominator—for instance, going from patient encounters in a fixed period to enrolled populations to geographically defined populations. More conceptually, in the context of shared responsibility for population health outcomes, measures that clarify accountability for actions are needed.82,83 In measuring health outcomes associated with health care organizations, methods for and the appropriateness of risk adjustment are reasonably well understood. How do these issues play out in the context of a shared responsibility for population health?

While a discussion of general measurement principles is beyond the scope of this paper, the principles used by the NQF for evaluation of health care quality measures84 are worth noting and apply well to population health measurement, with some minor adjustments. NQF criteria apply to both process measures, which describe the process of health care, and the associated health outcomes. There are four basic criteria: importance, scientific acceptability, usability, and feasibility.

For process measures, the first question is whether there is evidence connecting the process in question with the desired health outcome. One must know, for instance, that mammographic screening reduces breast cancer in order to make the case that screening rates are an important measure. Beyond that, the criteria ask whether there is a gap in performance, i.e., evidence of disparities among groups in terms of process or outcome measures, and a potential for improvement. In the population health setting in particular, the availability of data for other populations is essential for identifying gaps in performance and the possibility of improvement.

Establishing the reliability, validity, and precision of population health measures can be challenging. The more one limits attention to populations such as the patients served by a particular health care provider or living in a defined geographical area—as is necessary to make indicators actionable—measures based on sample surveys, or even based on all reported cases of a particular condition, become less reliable. This is discussed further, below. With regard to validity, a common question regarding population health outcomes measures relates to accountability: Given the multiplicity of the determinants of population health, can the health system that is the subject of the measures reasonably be responsible for that outcome? This too is addressed below.

For population health, a key question is whether the measures are actionable. First, is there agreement on whether smaller or larger values represent progress? Are the health outcomes susceptible to improvement through enhanced treatment, prevention, or other upstream activities? Are process measures within the control of the health care system or other community organization to which the measures apply?

In health care, feasibility often depends on whether measurement data can be obtained within the normal flow of patient care. For population health, one must ask whether the measure can be implemented without undue burden using existing data systems.85

In addition, Bilheimer86 and Pestronk87 both offer useful sets of criteria for population health measures, as summarized in Table 6.

Table 6.

Measurement Principles

Evaluating Metrics to Improve Population Health*

Are the measures actionable? Are the measures sensitive to interventions? Are the measures affected by population migration? Are the measures easily understood by collaborating organizations, policymakers, and the public? Is the meaning of an increase or decrease in a measure unambiguous? Do the measures stand alone or are they aggregated into an index or summary measure? Are the measures uniform across communities? To what extent do measures address disparities as well as overall burden? Can unintended consequences be tracked?

Using Metrics to Improve Population Health**

Simple, sensitive, robust, credible, impartial, actionable, and reflective of community values Valid and reliable, easily understood, and accepted by those using them and being measured by them Useful over time and for specific geographic, membership, or demographically defined populations Verifiable independently from the entity being measured Politically acceptable Sensitive to change in response to factors that may influence population health during the time that inducement is offered Sensitive to the level and distribution of health in a population Responsive to demands for evidence of population health improvement by measuring large sample size

Sources:

Bilheimer,86

Pestronk87

Data Availability and New Sources of Information to Measure Population Health

Although some analysts begin by examining the available data, it is usually better to start with the population and measurement framework, as described above. Starting with the data doesn’t automatically force the careful consideration of the appropriate denominator or what is important to measure. Eventually, though, data availability must be considered.

On the health care delivery system side, the American Recovery and Reinvestment Act of 2009 has enabled health care providers to adopt EHRs on a massive scale, and the act’s “meaningful use” provisions include the required reporting of quality measures for the populations they serve. According to Friedman, Wong, and Blumenthal,88 use of these data beyond their original purpose of supporting the health care of individual patients can speed the progression of knowledge from the laboratory bench to the patient’s bedside and provide a cornerstone for health care reform. Consistent with this, McKesson reported in 2013 that two-thirds of survey respondents expect to invest in integrating clinical data across the care continuum, and more than half expect to add to their data analytics capability. Access to data is a necessary component of population health management, according to Earl Steinberg, executive vice president, innovation and dissemination, for the Geisinger Health System.89

Okun and colleagues90 demonstrate how electronic health data can help to improve disease monitoring and tracking; can better target medical services for improved health outcomes and cost savings; help inform both patients and clinicians to improve how they make decisions during clinical visits; avoid harm to patients and unnecessary costs associated with repeat testing and delivery of unsuccessful treatments; and accelerate and improve the use of research in routine medical care to answer medical questions more effectively and efficiently. Existing electronic clinical data is also essential for the Center for Medicare and Medicaid Innovation’s new, rapid-cycle approach to evaluation, which aims to deliver frequent feedback to providers in support of continuous quality improvement, while rigorously evaluating the outcomes of each model tested.91

Also focusing on the CHNAs that are an essential part of the arsenal of local health policymakers and are required of nonprofit hospitals, Gresenz92 describes how large administrative data sources such as hospital discharge data, health insurance claims and encounter data, and EMR data can be used to monitor population health. Discharge data can be used to construct a widely used indicator of health systems performance—ambulatory care sensitive hospitalization rates—for local populations and for demographic subgroups within localities. Health insurance claims and encounter data have been a key source of information for tracking health and access to health care, and recent developments by federal and state government agencies and by private organizations has dramatically increased. This is likely to continue to augment the availability of public payer encounter data, private insurance claims data, and integrated public and private claims data. With heavy investment by the federal government, EMR data are likely to be increasingly useful in the future, and especially so if the data are integrated across health care providers, hospitals, and health systems. Finally, other big data sources such as data from third party intermediaries that go between providers and insurers in the billing process, and aggregated data from pharmaceutical retailers, hold additional promise for contributing to a more comprehensive understanding of health and health care among a local population.

For geographically defined populations, organizations such as Community Commons93 and the Healthy Communities Institute,94 as well as the CDC95 have developed CHNA data resources. The County Health Rankings96—prepared for the Robert Wood Johnson Foundation by the University of Wisconsin Population Health Institute—which are based primarily on data from the Census Bureau, vital statistics, and the CDC’s Behavioral Risk Factor Surveillance System (BRFSS), use similar data to rank counties within states in terms of health outcomes and the factors that determine them (see below for more details). The County Health Rankings are intended to illustrate that much of what affects health occurs outside of the doctor’s office, and they help counties understand what influences how healthy residents are. Ultimately, the goal is to use these data to garner support among government agencies, health care providers, community organizations, business leaders, policymakers, and the public for local health improvement initiatives.97

Tomines and colleagues98 suggest that electronic health information systems can reshape the practice of public health and improve surveillance, disease and injury investigation and control programs, decision-making, quality assurance, and policy development. However, while these opportunities are potentially transformative, and the meaningful use provisions have included important public health components, significant barriers remain. Unlike incentives in the clinical care system, scant funding is available to public health departments to develop the necessary information infrastructure and workforce capacity to capitalize on EHRs or personal health records. Current EHR systems are primarily built to serve clinical systems and practice, and are generally not structured for public health use. And there are policy issues concerning how broadly public health officials can use the data.

In New York City, the Department of Health and Mental Hygiene is working to bridge the gap between clinical and population health with the NYC Macroscope, a population health surveillance system that uses EHRs to track conditions managed by primary care practices that are important to public health. The goal is to monitor in real time the prevalence of chronic conditions, such as obesity, diabetes, and hypertension, as well as smoking rates and flu vaccine uptake. Noting that EHRs are rapidly becoming the standard of care for office-based medical practices as a result of federal incentive programs encouraging their uptake by health care professionals and institutions, the NYC Macroscope relies on data from the Primary Care Information Project. This project helps ambulatory providers in underserved areas adopt EHRs with population management tools to improve the quality of health care for the most vulnerable New Yorkers. These EHRs have been constructed with population-health management goals in mind, and can complement and expand the capacity of existing surveillance systems by capturing care events economically and with relative completeness. The NYC Macroscope will be validated by comparing ambulatory EHR data with data from the 2013 NYC Health and Nutrition Examination Survey—a gold standard, population-based health survey.99

Despite their potential, however, Gresenz100 describes a number of key challenges to using EHR data for population health purposes, including the level of resources required for their procurement and the need for rigorous independent analyses of data quality and representativeness. In particular, many commonly used survey-data sources regarding health or health care are engineered to provide national- or state-level estimates, but sample sizes available for more localized areas are often too small to support statistical estimation with precision. Geographic identifiers are sometimes unavailable or only available for use in restricted ways. And the lag time between data collection and data availability can be as long as a year to several years, limiting the ability of such data to inform timely and time-sensitive health policymaking. Area-specific surveys are another option, but are expensive.

Similarly, recognition of a shared interest in population health, and the encouragement provided by the Office of the National Coordinator for Health IT’s “meaningful use” standards, often requires merging existing electronic data in different formats from providers and public health sources.101 This in turn reveals a number of complex data governance issues (ownership, privacy, confidentiality, etc.) that must be addressed.

Friedman and colleagues102 also agree that EHRs could contribute to improving population health by providing a better understanding of the level and distribution of disease, function, and well-being within populations. They note, however, that realizing this potential will require understanding what EHRs can realistically offer and the requirements for obtaining useful information from them: improved population coverage, standardized content and reporting methods, and adequate legal authority for using EHRs for population health.

Scientific Properties of Population Health Measures

As noted above, scientific acceptability is one of the four major NQF criteria performance measures,103 but ensuring the reliability and validity of population health measures can be challenging, and one must often address trade-offs between them to improve the measures’ precision and sensitivity to change. The primary questions are whether the available measures can track changes in population health over time and between jurisdictions and subpopulations, and whether observed changes and differences in these measures reflect real differences in population health, as opposed to measurement error.

A measure’s reliability is the degree to which it consistently measures the concept it is supposed to represent. For population health measures, the primary challenge to reliability comes from the small numbers of events in the numerator of a rate and limited sample sizes. For instance, the development of community-health profiles reminds us about the limitations of population-based data for “small areas,” which include most governmental units below the state level: counties, cities and towns, hospital service areas, neighborhoods, and so on. Disaggregation by racial and ethnic groups, as well as socioeconomic variables, which are needed to study health disparities and inequities, further reduces the sample size.

Population health data based on rates frequently exhibit large relative random variation due to rare outcomes in the numerator. The infant mortality rate, for instance, is a widely used population health measure, but in most communities the number of infant deaths is small and statistical fluctuations from year to year can be relatively large. Similarly, medical errors and hospital-acquired infections—while disturbingly common on a national basis—are relatively rare for any given health care provider. Indicators based on survey data are also problematic. Most health surveys are designed to make estimates at the national or state level, and incorporate a sample size of thousands of respondents at these levels. The number of observations for counties or smaller areas, however, is substantially less, and often too small to provide reliable estimates.

The ACA specifies 5,000 “Medicare lives” as the minimum size for ACOs, primarily to ensure a large enough population base to spread risk effectively.104 For preventive health and other performance measures that relate to an entire ACO population, this number is also likely to be sufficient to ensure reliability. However, for measures focused on at-risk populations (e.g., those with heart failure or coronary artery disease), there may not be a sufficient base to ensure reliable estimates.

Because performance measures are typically followed over time to assess progress, “sensitivity to change”—an indicator’s ability to measure change—is particularly important. Some changes simply reflect chance fluctuations in epidemiological rates. For most communities, for instance, infant mortality rates fluctuate substantially from year to year simply because the numerator, the number of infant deaths, is small. Statistical tests can be used to assess the degree to which the indicator changes if and only if the concept being measured also changes.

The most common way to deal with the problem of small numbers is aggregation. Rather than presenting the infant mortality rate for 2014, the average rates over the last three, five, or perhaps seven years are reported. Survey data are often available for the “White,” “Black,” and “Other” populations, essentially ignoring any differences within these groups. Community health profiles are calculated for counties, but not for smaller geographic areas or subgroups within counties. Minimum numbers of patients are needed to form an ACO. (In addition to reliability, another common rationale for aggregation is to preserve confidentiality.) The problem with this approach is that differences within the aggregated groups are invisible. Infant mortality rates based on five years of data make it difficult to see a sharp rise in recent years. Rates for the Other population mask differences among the groups that fall into this category. And county-level data make it impossible to see either socioeconomic or geographical disparities within the county.

Alternatively, proxy measures might be developed. In some circumstances, the proportion of children born with low birth weight (with a larger numerator) might be a more stable summary measure of infant health than the infant mortality rate.105,106

There are also a variety of statistical methods that “borrow strength”—that is, they use information from jurisdictions that are temporally or geographically proximate or similar in other ways. One such alternative is to smooth the data by averaging over time (using time series methods and weighted moving averages, for instance) or over space using spatial analogs of these methods.107 One such model was used in the preparation of the Atlas of United States Mortality.108 Model-based estimates use hierarchical statistical models to combine information from one survey with other more geographically detailed data to increase reliability. Zhang and colleagues109 have used this approach, combining data from the BRFSS with data from the U.S. Census and Department of Agriculture to estimate the county-level prevalence of obesity in Mississippi. Raghunathan and colleagues110 have combined data from the BRFSS and the National Health Interview Survey (NHIS), which have complementary strengths and weakness, with county-level demographic and socioeconomic variables to estimate county-level prevalence rates of cancer risk factors and screening.

Statistical models of this sort, it must be acknowledged, are better for some purposes than for others. The complex statistical models and data demands lead to problems in timeliness of the results and perhaps the understandability of the results. Depending on the variability in the data for the target area and the desired degree of smoothing, more or less weight is put on a community’s own area data. The resulting estimates are not, therefore, unbiased. Because of the assumptions that underlie them, these models are not good for looking for outliers or differences between adjacent or similar communities; depending on the degree of smoothing and the statistical model, differences of this sort will be minimized. These techniques can be very useful, however, in seeing the “big picture” in geographical and other patterns.

Validity, on the other hand, is an indicator’s capacity to measure the intended concept. Risk adjustment to account for differences in patient severity is commonplace in reporting hospital mortality.111 However, while methods for risk adjustment have long been evaluated for clinical care measures, these approaches may not directly translate to the context of shared accountability. One must ask whether the measure is really within the control of the entity whose performance is being assessed, and consider risk adjustment to control for differences in populations being served. If used inappropriately, risk adjustment can obscure disparities, and the NQF has had a long-standing policy against adjusting quality measures for sociodemographic factors. Recently, however, an NQF panel recommended measure by measure determination of the appropriateness of sociodemographic adjustment based on two criteria: (1) the existence of a conceptual relationship between one or more sociodemographic factors and an outcome or process of care; and (2) the existence of empirical evidence that sociodemographic factors affect the measure.112

Changes in the process of measurement, and the distinction between real and artifactual changes in mortality statistics and self-reported health conditions, also must be considered. Does an increase in the proportion who report being told that they have diabetes reflect the success of a screening program, or the failure of primary prevention? A similar problem arises when health service utilization records are used to assess changing disease burdens. Does a decrease in emergency department visits for asthma indicate the success of primary care in the community, or measures to restrict access to individuals without insurance?

Because assessing a measure’s validity can be complex, it helps to review the “theory of the measure”—the measure’s purpose and intended use, and precisely how and why a rate with a particular numerator and denominator is expected to reflect a certain aspect of a population’s health or the quality of health care a population receives.

For example, consider the Prevention Quality Indicators (PQIs), a family of measures developed by AHRQ based on hospital inpatient discharge data to identify quality of care for ambulatory care sensitive conditions. These are conditions for which good outpatient care can potentially prevent the need for hospitalization or for which early intervention can prevent complications or more severe disease. Patients with diabetes, for example, may be hospitalized for diabetic complications if their conditions are not adequately monitored or if they do not receive the patient education needed for appropriate self-management. The denominator for these “area measures” is the adult population in a metropolitan area or county. The PQIs are population based and adjusted for covariates.113 Even though these indicators are calculated with standardized software using hospital inpatient data, they are intended to provide insight into the community health care system or services outside the hospital setting rather than the hospital per se. The validity of this family of measures thus depends on the following: (1) evidence that the hospital admissions that form the basis of the measure actually are sensitive to ambulatory care, or more generally care received in the community; and (2) understanding that the purpose of the measure is to assess the quality of and access to care in the population rather than the quality of the care that the hospital serves. Some have suggested that the denominator be restricted to the population with diabetes to ensure fair comparisons, but this is not done because accurate data on the prevalence of diabetes at the county level is not available.

Compromises must generally be made among validity, reliability, data availability, and sensitivity to change. In the area of prenatal care, for instance, evaluators often use the receipt of prenatal care in first trimester, rather than more complex measures based on official recommendations of the U. S. Public Health Service for the frequency and content of prenatal care, because the former measure is available on birth certificates and the latter is not. This is a case of trading validity for increased data availability. In many communities, annual infant mortality rates are not reliable due to the small number of infant deaths. Instead of annual rates, therefore, epidemiologists commonly calculate running averages by average infant mortality rate over three or five years. This is a case in which reliability is gained at the expense of timeliness and responsiveness to change. Another approach that is frequently used to deal with sparse data is to use proxy measures that reflect trends and differences. The percentage of infants born at low birth weight, for example, is used rather than infant mortality because low birth weight has been shown to be strongly associated with infant mortality. This is a case of trading validity for reliability.

Example: Promoting Successful Birth Outcomes

The IOM’s Access to Health Care in America report114 provides a good example of the trade-off between reliability and validity, and how a family of measures can overcome the limitations of the individual measures. This report proposes five major health care access objectives—one of them being successful birth outcomes—and sets of utilization and outcome measures for each.

The first indicator is based on the utilization of health care, specifically prenatal care. Using information on birth certificates, one can calculate the percentage of pregnant women obtaining “adequate” prenatal care. The strength of this indicator is that it is a direct measure of health care access. There are, however, a number of weaknesses. First, adequate prenatal care is measured only by what is available on the birth certificate, which is usually limited to the time of initiation and the frequency of care. There is nothing on the distribution of visits over time or their content. Indeed a large number of visits could represent a problem pregnancy. More complex indices of the adequacy of care are available, but their calculation can be confounded by missing or incomplete data, and there might be problems in recalling the number of timing of visits when the birth certificate is being filled out.

The second indicator proposed by the IOM is an outcome measure: the infant mortality rate. This is a commonly used measure of access and is available even for small communities from vital statistics. The infant mortality rate, however, provides little information about access barriers, and includes causes of death that cannot be affected by the health care system. Moreover, because infant deaths are uncommon, there is substantial variability in infant deaths in many areas simply due to small numbers.

As an alternative, the third indicator proposed is the low birth weight rate, the proportion of infants weighing less than 2,500 g at birth. The strengths of this indicator are that it is known to be specific to adequate prenatal care and access to nutrition services, and is an important predictor of infant survival. Moreover, since the numerator is not as rare as infant deaths, the rate is more stable. As with the infant mortality rate, however, the low birth weight rate provides little information about access barriers.

The final indicator is the rate of congenital syphilis. This is a reportable condition in most states, so the data are available, and the condition is very specific to lack of or inadequate prenatal care. The weaknesses are that reporting may be incomplete, and since syphilis is rare in most states, the measure often lacks reliability.

Conclusions

Whether the focus of population-health improvement efforts the measurement of health outcomes, risk factors, and interventions to improve them are central to achieving collective impact in the population health perspective. And because of the importance of a shared measurement system, appropriate measures can help to ensure the accountability of and ultimately integrate the efforts of public health, the health care delivery sector, and other public and private entities in the community to improve population health. Yet despite its importance, population health measurement efforts in the United States are poorly developed and uncoordinated.

To achieve the potential of the population health perspective, public health officials, health system leaders, and others must work together to develop sets of population health measures that are suitable for different purposes yet are harmonized so that together they can help to improve a community’s health. This begins with clearly defining the purpose of a set of measures, distinguishing between outcomes for which all share responsibility and actions to improve health for which the health care sector, public health agencies, and others should be held accountable.

Depending on the purpose of the analysis, then, measurement systems should clearly specify what to measure—in particular the population served (the denominator), what the critical health dimensions are in a measurement framework, and how the measures can be used to ensure accountability. Building on a clear understanding of the purpose and dimensions of population health that must be measured, developers can then choose specific measures using existing data or developing new data sources if necessary, with established validity, reliability, and other scientific characteristics. Rather than indiscriminately choosing among the proliferating data streams, this systematic approach to measure development can yield measurement systems that are more appropriate and useful for improving population health.