The use of quality indicators to promote accountability in health care: the good, the bad, and the ugly.

In Canada, we place significant cultural and financial value on our healthcare system.  As such, it is imperative we measure its quality.  In this commentary, we highlight some of the potential benefits and harms associated with measuring quality using currently available indicators, such as hospital mortality rates, emergency department length of stays, and readmission rates.  These measures tend to focus on provider and process issues rather than patient outcomes and also reflect what we can measure rather than what we should measure.  We argue that the current approaches are inadequate and recommend a better understanding of the limitations of current indicators and more provider engagement.  To meet these recommendations the health system needs to increased investment in performance measurement systems.

Health care is a top policy priority for Canada’s federal and provincial governments,1 and many Canadians believe that publicly funded health care is a defining feature of their nation.1 Twelve percent of Canada’s gross domestic product and between 33% and 46% of provincial budgetary allocations are directed toward the health care system.2 In view of the societal and financial importance of Canada’s health care system, it is imperative that individuals and organizations are held accountable for the role they play within it.

One way to promote accountability in health care is to measure the quality of services provided. In recent years, funders have been under increasing pressure to direct health care payments on the basis of so-called “big dot” quality indicators such as hospital mortality rates, emergency department length of stay, and all-cause readmission rates.3 In addition, many jurisdictions have started to publicly report a variety of quality indicators.

Although the concept of holding people accountable by means of quality performance metrics makes sense, the technical challenges that this presents are significant. Even the concept of quality is difficult to define in an operationally meaningful way. One of the most influential thinkers in this field, Avedis Donabedian, defined high quality of care as “that kind of care which is expected to maximize an inclusive measure of patient welfare, after one has taken account of the balance of expected gains and losses that attend the process of care in all its parts.”4 Although this definition makes conceptual sense, it certainly challenges those responsible for evaluation. To make it easier, Donabedian suggested the need to measure attributes within “care domains,” including structures, processes, and outcomes. Others, such as the US Institute of Medicine, have built on this work by defining each domain in terms of care-related attributes such as timeliness, effectiveness, safety, and patient-centredness.5

Although numerous quality domains and conceptual frameworks exist, we find it helpful to anchor any measurement or accountability exercise in Donabedian’s classic definition. From a very pragmatic perspective, one often ends up measuring what can be measured and then tries to reverse-engineer these measures into whatever construct happens to be in fashion. As a result, most current health care indicators focus on provider and process issues (e.g,. emergency department length of stay) rather than patient issues (e.g., did the care provided optimize the patient’s health status?). These issues legitimize concerns regarding the fairness of current quality measures for discriminating between high and low performers.

In this commentary, we hope to stimulate innovation in the field of health care performance measurement. We will discuss several considerations regarding the use of three quality indicators that are commonly used to improve accountability in the Canadian context. Specifically, we will focus on the hospital standardized mortality ratio (HSMR), all-cause urgent readmissions, and emergency department length of stay (ED-LOS; see Textbox 1). We discuss “the good,” “the bad,” and “the ugly” to illustrate both positive and negative consequences related to measurement. We conclude with specific recommendations regarding investments to improve quality measurement.

Textbox 1

Quality indicators

The good

Making people and organizations accountable for specific quality indicators promotes at least two positive behaviours. First, it directs individuals and organizations toward priorities that are important to health care payers. For example, governments may focus on readmission rates to reflect the public’s desire to improve “continuity of patient care,” a factor that contributes to readmissions.6 By targeting ED-LOS and HSMR, governments are reflecting, respectively, the public’s desire to focus on access to acute care and patient safety. While the democratic process is not perfect, government policy often reflects the concerns of everyday citizens; it would be difficult to argue that, from the perspective of the Canadian public, access to acute care and patient safety are not top priorities. Thus, setting targets on particular issues allows governments to stimulate action in areas they feel reflect societal values without being prescriptive about the specific interventions that are used.

Second, setting targets promotes collaboration. Organizations often create multidisciplinary teams to address quality problems, since no single group can adequately address performance issues. For example, reducing ED-LOS requires physicians to work with nurses and other hospital employees (such as laboratory and diagnostic imaging technicians) to ensure maximum efficiency.7 Also, efforts to reduce HSMR involve different disciplines—for example, though the establishment of multidisciplinary rapid response teams in hospitals.8 Government attention to indicators also promotes productive collaboration across sectors and organizations, as can be seen with interventions designed to reduce readmissions such as the ARC (Avoidable Readmissions through Collaboration) initiative.

The bad

Quality indicators can also be a counterproductive means of ensuring accountability. Their use is premised on the assumption that the indicators conceptually and statistically reflect an attribute of health care excellence.9 Unfortunately, this premise is often false or at least unproven: most indicators are chosen on the basis of what can be measured, rather than what should be measured.

Hospital readmission rates serve as an example. Many health systems use this indicator because it is measurable and because some readmissions reflect imperfect care. However, we have shown that the vast majority of readmissions are attributable to “unavoidable” issues such as disease progression or social factors.10 We have also demonstrated that differences between hospitals with respect to urgent readmission rates do not in fact reflect differences in “avoidable” readmissions.10 Although risk-adjustment methods can be used to remove the influence of factors not related to quality of care, these approaches are imperfect and can result in unreliable performance rankings.11

This concern has at least three significant consequences for ensuring accountability. Most importantly, indicator-based incentives will lead to unjust resource distribution if they do not reflect “true” quality. Second, investments to improve performance will be misspent if they are directed by inappropriate indicators rather than “true” quality measures. Third, a finding of “poor” performance based on such indicators is demoralizing to individuals and organizations who legitimately work toward solving “true” quality problems.

The ugly

Finally, the use of quality indicators to promote increased accountability could lead to behaviours that do not translate into better care of patients. For example, the use of indicators to guide funding decisions could bias data capture by health care providers. Most quality indicators are derived from data based on subjective assessments. For example, assigning diagnostic codes to hospital encounters relies on doctors documenting diagnoses and health records analysts assigning codes, and both of these processes are determined to a certain extent by subjective criteria. Providers who are cognizant of the impact that coding has on indicator performance could ensure that data capture is conducted in a way likely to favour their institution. For example, the most efficient way to reduce a hospital’s risk-adjusted mortality is to “up-code” the factors that contribute to the risk of death in hospital. In fact, the results of several studies suggest that this phenomenon explains many improvements in indicator performance over time.13 Such behaviour does nothing to directly improve patient care.

Alternatively, undue pressure to achieve targets might make clinicians or administrators react in a way that could actually increase patient risk. For example, hospital administrators might inadvertently put pressure on providers to prematurely discharge patients to avoid exceeding an arbitrary emergency department wait-time target. An administrator or clinician who lacks a holistic view of quality might feel that such decisions are justified, given the negative consequences (for the institution) associated with poor performance on the indicator. Such actions would, paradoxically, adversely affect patient care in an effort to improve indicators designed to measure quality.

Conclusion

The health system is currently in a difficult situation: “true" quality measures are needed to hold people accountable, but the current approach is arguably inadequate. Although avoiding measurement or waiting for perfect measures are clearly not reasonable options, we should acknowledge that current approaches cannot fully meet the public’s expectation, that providers should, with fair methods, be held accountable. For this reason, we suggest several actions.

First, we need a better understanding of the limitations of current indicators. This can come only through critical analyses of the association of current indicators with “true” quality attributes. For example, it is necessary to determine how accurately statistics such as the HSMR or readmission rates classify the performance of organizations as high or low. Such analyses will maximize the fairness of any incentives and minimize the unintended consequences of the measure.

To achieve this goal, the health sector will need to make modest investments in research on quality indicators; this would cost significantly less than many other interventions in the health care system (such as that for a new drug being added to provincial formularies). To avoid perceived conflicts of interest, this investment should be directed toward experts who are able to critically assess quality indicators but are not themselves responsible for holding people accountable.

Second, providers need to be fully engaged in the measurement, analysis, and interpretation of indicator data. Providers know their business and should demonstrate the leadership required to measure its quality. Approaches in which governments and academics independently define standards are doomed to fail, as their understanding of what drives performance often does not take into account important nuances at the level of the individual patient, clinic, or hospital.

To achieve this goal, we recommend a gradual transition in funding methodology such that providers would be more accountable for their actual performance. Increasing the accountability of individuals who are responsible for care provision—and, therefore, their stewardship of resources—will likely increase their engagement in determining how they are actually monitored. In this setting, “providers” can include an individual practitioner in a solo practice setting, a director and medical lead in a group practice setting, or a chief of staff and chief executive officer within institutions. Although it is beyond the scope of this article to fully explore how to change funding methods, it is imperative to attain clarity in the question of specifically who is accountable to ensure fair and effective governance. Given that our current evaluation methods are likely inappropriate, the transition to new funding methods must be gradual. In addition, the changes will need to occur in tandem with training programs to teach the specific skills and knowledge related to performance management.14 Creating new expectations without an ability to meet them could be interpreted as unfair.

Third, the entire health care system needs to increase investment in performance measurement systems. The current approach—measuring what we can instead of what we should—is inadequate, especially in light of our large investments in health care services. Several thoughtful people and organizations have developed conceptual frameworks for measuring performance.4,14,15 These frameworks describe a need to measure overall incremental health benefits (i.e., the positive effects derived from the treatment minus the negative effects from adverse events) within specific disease states. They also include a requirement to accurately evaluate the costs of providing care at the level of the individual patient.

Rather than developing more frameworks or indicators per se, we should be investing more heavily in the infrastructure of measurement. Specifically, funding should focus on electronic data systems designed to support clinical workflow, data systems to integrate the data derived from operational systems, and the training of highly qualified people who are able to analyze and interpret health care data. Textbox 1 provides examples of how these investments could translate into more effective measures.

These investments need to be performed in a coordinated and sustained manner for the health care system to improve. For example, building a computerized physician order entry system to support doctors ordering drugs will not be useful from the measurement perspective unless prescription data are linked to information on patient diagnoses and outcomes. Furthermore, there is a need for experts who can analyze the data to determine whether drug utilization was appropriate. There are few shortcuts or “quick wins” in this domain.

Collectively, Canadians greatly value their health care system. During times of fiscal restraint, it may seem incongruous to suggest increasing investment into performance measurement and management. However, in relative terms, the required investment to improve performance measurement systems is a small fraction of our overall health care investment. Furthermore, such investments would complement the high-quality work performed by organizations such as the Canadian Institute for Health Information, the Organization for Economic Co-operation and Development, and the World Health Organization, who are leading efforts to improve measurement of quality and safety within health care through the development of standardized approaches to ensuring comparability within and across jurisdictions. Moreover, the return on this investment may be sufficient to preserve or even increase overall quality while simultaneously reducing overall spending.