Medical practice variation: public reporting a first necessary step to spark change.

From previous work, we know that medical practice varies widely, and that unwarranted variation signals low value for patients and society. We also know that public reporting helps to create awareness of the need for quality improvement. Despite the availability of rich data, most Western countries have no routine surveillance of the geographic distribution of utilization, costs, and outcomes of healthcare, including trends in variation over time. This paper highlights the role of transparent public reporting as a necessary first step to spark change and reduce unwarranted variation. Two recent examples of public reporting are presented to illustrate possible ways to reduce unwarranted variation and improve care. We conclude by introducing the Value Improvement Cycle, which underscores that reporting is only a necessary first step, and suggests a path toward developing a multi-stakeholder approach to change.

Introduction

For over 40 years, health services research has shown that clinical practice is highly variable across regions, and that this variation is often unrelated to differences in population need. Practice variation is considered unwarranted when the variation is not explained by the incidence of illness, the constraints of medical science, or the preferences of individuals. Patterns of care are primarily reflective of local practice styles of doctors and health systems [1] and are often sensitive to the local availability of resources. Brownlee illustrates this in a study that found ‘an almost-perfect correlation between the availability of catheterization in a region and the propensity for patients to be given angioplasty.’ [2] Many more patients received stents where there were higher numbers of cardiologists performing angioplasty. At the clinician level, when doctors are faced with discretionary decisions in standardized vignettes describing a clinical case, their choices correlate with local health status adjusted utilization levels [3, 4]. Clinicians adhere to local standards of care, with the local supply of resources acting as one of the dominant drivers of utilization [5]. As pointed out by Mulley, ‘Clinicians become accustomed to standard clinical policies that have been shaped over time by local capacity and are often surprised to learn that they are radically different from those in other regions.’ [6] Mapping and reporting regional patterns and investigating geographic differences in medical utilization are essential for recognizing problems and developing policies and practices that increase the value of healthcare. In this era of austerity, the need for greater value is acute. ‘One doctor’s waste is another patient’s delay. Potentially, it could be that other patient’s lack of treatment’ [7].

Public reporting and mapping unwarranted healthcare variation

The findings from research into practice variation are rich and demonstrative of the challenges facing national healthcare systems [8, 9]. From previous work, we know that unwarranted variation can be harmful to patients and costly to society [1, 7]. We also know that patients who live in regions with high costs and high utilization rates do not reliably experience better care and outcomes: more is not always better [10, 11]. These findings have been partially replicated in a few countries, but in most countries, geographic data analysis is at the earliest stages, and very few have established systems for monitoring healthcare value metrics [9] with sufficient regional and provider specificity to guide change. There are a few noteworthy exceptions. In 2010, NHS England began publishing a series of Atlases [12]. According to the authors, ‘In many localities across England, the NHS Atlas of Variation in Healthcare series has been used as a stimulus to start a search for unwarranted variation, and as a springboard to releasing resources for re-investment in higher-value healthcare for local patients and populations.’ Another example is the Dartmouth Atlas of Health Care [13], which found marked variation in end-of-life care across hospitals in the USA. A few other countries showed progress in variation research in a recent Organisation for Economic Co-operation and Development study [9]. However, in general, most countries do not have clear, valid and reliable insights into the geographic distribution of the use, costs and outcomes of healthcare, or into trends over time.

Data statement: ‘Driving Without a Map’

At the fourth annual Wennberg International Collaborative meeting at Dartmouth College (2013), attendees recognized that the public, patients, providers, payers and politicians have a common need for information about healthcare at local and regional levels and acknowledged that robust data is an essential element of any successful clinical transformation initiative. Yet, many of the meeting attendees reported difficulties in accessing existing data, which restrained efforts to conduct healthcare surveillance in their countries. In most countries, access to anonymous patient utilization and cost data for independent research is highly restricted, fragmented across many data holders, or prohibitively expensive. The reluctance of the holders of administrative data to share valuable healthcare data with legitimate researchers and policy analysts diminishes the prospects for analyzing the performance of regional health systems and targeting public policy to spur improvement. To promote the improvement of healthcare through the responsible use of healthcare data, the Wennberg Collaborative members (n = 45, 23 countries) signed the statement ‘Driving Without a Map’ at the 2015 London meeting. Visit for signed text: www.wennbergcollaborative.org/data-statement.

Avoiding authentic facts

Why is there so often an unwillingness to permit legitimate researchers to access healthcare data and freely publish the results they observe? The most commonly offered reason is to protect patient privacy. This ignores that protecting patient data is a paramount concern for all medical and healthcare researchers. Health service researchers have an excellent track record in demonstrating good stewardship in securing data. The reluctance to release data is more likely because medical practice analysis often shows embarrassing gaps in quality and efficiency. Displaying the results at the area or hospital level is often uncomfortable for clinicians, troubling to patients, and concerning to politicians. At times, the resistance to measuring healthcare appears to stem from willful ignorance, a sentiment not lost on Berwick, who noted that the concept of unwarranted variation is provocative to clinicians and seen as a direct challenge to their clinical practice [14]. When data are available, pushback from providers and insurance plans has been a predictable response to publicly mapping health system performance.

As analyses move from describing variation toward research into its causes, intense and valuable discussion often ensues regarding the implications of the findings for public policy and clinical improvement. In this dialog, it is crucial that the metrics presented are accurate and that case-mix control between regions or centres is done in a valid and reliable way. Public reporting can only be effective when the data and corresponding interpretation are sound.

Does it change anything? And how?

Releasing maps depicting variation stimulates discussion, but the important question is whether and how change ensues? In the USA, public reporting has been associated with better healthcare [15], a review of literature by Agency for Healthcare Research and Quality concludes. ‘For most of the outcomes, the strength of the evidence available to assess the impact of public reporting was moderate.’ [15] ‘Studies of healthcare providers’ response to public reports suggest they engage in activities to improve quality when performance data are made public.’ [15]

Aylin and colleagues [16] showed reduced variation in surgical rates between Primary Care Trusts in England during the period from 1998 to 2003 but concluded that ‘the use of surgical procedures still varies widely.’ Appleby et al. [17] found no evidence of diminishing variation between 2005/2006 and 2009/2010 for hip replacement, cataract surgery, or tonsillectomy. ‘Variations in tonsillectomy rates may not be as high as the 20-fold variation found by Glover (1938) across London boroughs in the 1930s, but they nevertheless remain high.’ [21] All in all, the answer to the question of whether greater transparency leads to change is incomplete. In the next section, we present two recent projects from two countries (the Netherlands and New Zealand) that demonstrate that publishing data is prerequisite to improve care but needs to be accompanied by other actions.

Reducing regional variation in herniated disc surgery in the Netherlands

In the past decade, the effect of early spine surgery on outcomes such as disability scores, pain scores, and patient-reported recovery in patients with lumbosacral radicular syndrome has been debated. Randomized controlled trials have shown that early surgery, in comparison to medical treatment may result in better one-year outcomes, but not in better long-term outcomes [18, 19]. Despite convincing evidence that a conservative approach is good medical practice, surgery is often seen as the best option in daily practice, resulting in substantial geographical variation, as reported in both the USA and the Netherlands [20, 21].

In 2011, data regarding regional and hospital practice variation in herniated disc surgery in the Netherlands were made public for the first time. Figure 1a shows that there was substantial variation in the adjusted and standardized number of herniated disc surgeries in 2011. The data in Fig. 1a and b shows a decrease in this variation in surgery rates after publication of these reports. Figure 1a shows the two hospitals with the highest and lowest indicator scores (i.e. the age, sex and socio-economic status adjusted number of herniated disc surgeries per 100 000 persons), as well as the overall mean. There was a substantial drop in the maximum indicator score and a slight increase in the minimum indicator score between 2011 and 2014; this means that the surgery rate at the hospital with the most surgeries per 100 000 adherent persons in 2011 dropped in the years after, while the rate at the hospital with the lowest number increased slightly in the years after (Fig. 1a). The difference between the extremes got smaller. This trend is also reflected by the decreasing factor score, i.e. the ratio between the three highest scoring hospitals and the three lowest scoring hospitals over time. Between 2011 and 2014, the factor score decreased substantially, from 22.9, 23.2, 19.1 to 12.3. However, despite the decrease in variation between hospitals, the observed total volume was stable over time: 10.650 surgeries in 2011 and 10.175 in 2014. In other words, while the surgical load was redistributed, the total observed volume remained constant. Figure 1b also shows that, in 2012, a new provider (dotted line) entered the hospital market, taking a high share of the volume and affecting the redistribution.

Figure 1

(a) Trends in age and sex adjusted number of herniated disc surgeries per 100 000 persons, 2011–2014, The Netherlands. (b) Observed volumes of individual hospitals with high and low volume of herniated disc surgeries, 2011–2014, The Netherlands.

In 2013, the Netherlands adopted the Choosing Wisely (CW) campaign, and surgeons agreed on a set of ‘wise choices’ in the treatment of herniated disks that encourage a conservative approach. It is too early to assess the full impact of CW on volume and practice variation in herniated disc surgery, but the hypothesis is that it will lead to further reductions in unwarranted practice variation.

Reducing variation in opioid use in New Zealand

In New Zealand, dispensing rates of opioids, e.g. oxycodone, increased by 249% between 2007 and 2011 [22, 23], leading to increasing concern by 2014. The New Zealand Health Quality & Safety Commission (HQSC) released an Atlas of Health Care Variation that reported dispensing rates of community-dispensed opioids by District Health Boards (DHBs) in December 2014.

The crucial step in this process was the disclosure of dispensing rates for all DHBs. The Atlas showed that, in 2013, dispensing rates of opioids varied by a factor of 2.8 across DHBs (Fig. 2a). The DHB with the highest rate of strong opioid use was Wairarapa, a small rural region bordering the capital, Wellington where the lowest rate was observed. For oxycodone, the rate varied by a factor of 6.5 (see Fig. 2b). It is unlikely that the observed variation was attributable to underlying differences in the health status of the populations [22]. A draft version was sent to all DHBs before being publicly released in an effort to ensure all data was accurate.

Figure 2

(a) Strong opiod dispensing rates; total by year, rate by 1000, 2011–2015, New Zealand (red line = national average). (b) People dispensed oxycodone; total by year, rate by 1000, 2011–2015, New Zealand (red line = national average).

In response to the release of the Atlas, newspapers in the Wairarapa region immediately picked up on the high prescribing rates of opioids and published a front-page story. The findings created a ‘burning platform’ for the local Primary Health Organisation (PHO) to take action. The PHO formed a multi-disciplinary team and obtained data from the Ministry of Health, as well as local practice-level data, to better understand the population taking opioids: by practice, by prescriber, by condition, by age and by ethnicity. The team established systems and processes for prescribing opioids involving patients, general practitioners, and secondary providers. For example, the emergency department developed a policy that prescriptions would supply no more than two days of opioids. In the 2016 update of the Atlas, the Wairarapa DHB had significantly reduced rates of strong opioid dispensing, from 30.9 to 24.1 per 1000 inhabitants, and the variation between DHBs was reduced from a factor of 2.8 to 2.1 (Fig. 2a). Dispensing rates for oxycodone dropped from 18.8 to 2.7 per capita (Fig. 2b), below the national average rate. All in all, though the sector was already ‘primed for change’ regarding opioid prescriptions, the timely public reporting of data functioned as a firm nudge and helped to change behavior [24].

What spurs change?

In a seminal article in the Lancet in 1986, Domenighetti et al [25]. wrote that ‘Information on regional rates and on the need for hysterectomy given through the mass media to the general population can change professional practices.’ The annual frequency of hysterectomy rates in the Ticino Canton, Switzerland, was monitored between 1977 and 1986. ‘From February to October 1984, there was a public information campaign in the mass media about rates of and need for hysterectomy. After the start of the campaign and during the following year the annual rate of operations per 100 000 women of all ages dropped by 25.8%, whereas in the reference area (Canton Bern), where no information was given to the public, hysterectomy rates increased by 1%.’ The Domenighetti study is one of the very few examples in the medical literature showing that public reporting and presenting current practice styles to the public fuels debate and promotes change. This 30-year-old study is both inspiring and challenging for present times. First, it inspires because it shows the power of going public and giving a role to the mass media. Second, it confronts us with the fact that this approach is not used more often. This is partly due to how health services research works. Analyses of healthcare utilization data, beyond the national Atlases, are usually presented in academic journals and regrettably, stay there. Too frequently, meaningful data regarding the performance of health systems is not presented to the public, or even worse, not translated from technical terms into the day-to-day language of the public or politicians.

NHS RightCare claims that awareness is the first step, and that revealing variation is powerful, but it is only the end of the beginning. The key stages beyond that are what and how to change. The example from UK, through the RightCare programme, demonstrates how publishing data on variation can be a useful start to a wide-ranging conversation between clinicians, managers, and patient representative groups exploring why variation exists in the local system and attempting to understand its causes and potential remedies.

Changing persistent patterns of practice variation requires a debate among stakeholders and an ongoing four-stage process that we named the Value Improvement Cycle (VIC) (Fig. 3). VIC is similar to the Plan Do Study Act (PDSA) cycle [26] and to the RightCare approach, but as the figure shows, it is specific to the phenomenon of practice variation. The PDSA cycle is shorthand for testing a change by developing a plan to test the change (Plan), carrying out the test (Do), observing and learning from the consequences (Study), and determining what modifications should be made to the test (Act) [26].

Figure 3

The Value Improvement Cycle. How to reduce unwarranted variation?

Stage 1 of VIC is devoted to presenting the existing variation (MAP), using valid and reliable data, adjusting for case mix, and feeding the data back to the actors responsible for producing it. In the case of herniated disc surgery, this would mean the surgeons. Ideally, their response to the data would be ‘we have a problem to work on.’ Unfortunately, the most common responses are ‘the data isn’t right,’ ‘my patients are sicker,’ or ‘it may be a problem, but it’s not mine,’ usually in that order. These so-called ‘stages of grief’ (of losing professional autonomy) must be processed, one by one. Stage 2—‘can we understand the pattern?’—will only begin when the first step is finished in a satisfying way.

Stage 2 includes an in-depth analysis by health services researchers and a few influential medical professionals about the meaning of the data, what causes the variation, and how to understand the observed pattern in the data (ANALYZE): where are the high and low use areas, and what variables are associated with utilization differences? An important issue here is the assessment of balancing measures to ensure that, as utilization patterns change, new problems are not developing elsewhere. For instance, reducing opioid dispensing rates may cause an increase in return ER visits, or an apparent reduction in the number of herniated disc surgeries by a high-volume provider may be partly caused by a change in diagnostic coding practice (e.g. spinal stenosis surgery).

In Stage 3, other stakeholders are involved: referring physicians (e.g. general practitioners or neurologists in the case of a herniated disc), payers (e.g. insurers), and patients (SHARE). In this stage, barriers need to be tackled and facilitators activated in order to make change possible. What and whose value is at stake? This stage ends in a collectively agreed-upon action, an intervention that differs between high and low use areas: Stage 4 (CHANGE).

The example from New Zealand provides an excellent model where all VIC stages were passed. VIC ends and restarts at Stage 1, where the improvement (reduction of unwarranted practice variation) is verified. New data from t + 1 (after the intervention) are compared to the data at t0. In most cases, this leads to a new round.

Conclusions

In this paper, we point out the fact that, in many countries, there is limited or no information about unwarranted variation in healthcare utilization, costs, quality or outcomes, and that additional effort is needed to convince governments and policymakers that measuring, and reporting healthcare data is necessary to reduce unwarranted variation. There is little evidence that public reporting of healthcare variation improves care without the engagement of health system leadership and clinicians. The current paper shows two examples of how this can be done.

Finally, the paper points out reducing unwarranted variation and improving care are not always linked. Greater public reporting and multi-stakeholder interventions may stimulate improvement in evidence-based care, but at a different pace across regions: healthcare does improve, but variation persists. The sustained surveillance of healthcare and unwarranted variation provides a powerful reminder of the need for continual improvement.