Trends in Canadian hospital standardised mortality ratios and palliative care coding 2004-2010: a retrospective database analysis.

BACKGROUND: The hospital standardised mortality ratio (HSMR), anchored at an average score of 100, is a controversial macromeasure of hospital quality. The measure may be dependent on differences in patient coding, particularly since cases labelled as palliative are typically excluded.
OBJECTIVE: To determine whether palliative coding in Canada has changed since the 2007 national introduction of publicly released HSMRs, and how such changes may have affected results.
DESIGN: Retrospective database analysis.
SETTING: Inpatients in Canadian hospitals from April 2004 to March 2010. PATIENTS: 12 593 329 hospital discharges recorded in the Canadian Institute for Health Information (CIHI) Discharge Abstract Database from April 2004 to March 2010. MEASUREMENTS: Crude mortality and palliative care coding rates. HSMRs calculated with the same methodology as CIHI. A derived hospital standardised palliative ratio (HSPR) adjusted to a baseline average of 100 in 2004-2005. Recalculated HSMRs that included palliative cases under varying scenarios.
RESULTS: Crude mortality and palliative care coding rates have been increasing over time (p<0.001), in keeping with the nation's advancing overall morbidity. HSMRs in 2008-2010 were significantly lower than in 2004-2006 by 8.55 points (p<0.001). The corresponding HSPR rises dramatically between these two time periods by 48.83 points (p<0.001). Under various HSMR scenarios that included palliative cases, the HSMR would have at most decreased by 6.35 points, and may have even increased slightly. LIMITATIONS: Inability to calculate a definitively comparable HSMR that include palliative cases and to account for closely timed changes in national palliative care coding guidelines.
CONCLUSIONS: Palliative coding rates in Canadian hospitals have increased dramatically since the public release of HSMR results. This change may have partially contributed to the observed national decline in HSMR.

The hospital standardised mortality ratio (HSMR), a ratio of a hospital's actual to the expected number of deaths, is a popular but controversial method of gauging hospital quality.

Because patients labelled as palliative are typically excluded from the calculation, the HSMR is potentially sensitive to changes in coding practices.

This study assessed whether the release of public HSMR data in Canada was correlated with changes in palliative care coding, and how such changes might have affected HSMR results.

After public release of HSMR data in Canada, national HSMR declined while rates of palliative care coding increased dramatically.

If palliative cases had been included, rates of HSMR decline would have been at least modestly attenuated.

Changes in palliative care coding practices likely contributed to the observed decline in HSMR.

This study is based on over 12.5 million cases across Canada over 6 years.

The usual weaknesses of a retrospective analysis on database information apply.

Measuring the quality of a hospital is an important but exceedingly difficult task. Different methods of capturing quality have been devised, including composite scores of compliance with various quality indicators1 to the adoption of balanced scorecard techniques from the business world.2 No universally accepted gold standard exists.

The hospital standardised mortality ratio (HSMR) is a controversial macrolevel tool for measuring the quality of hospital care. The HSMR is a conceptually simple ratio of the observed deaths to expected deaths, multiplied by 100. An institution's number of expected deaths is calculated with a regression model using national data;3 having an HSMR greater than 100 implies having a mortality rate greater than expected for the types of patients admitted. Proponents argue that the metric is easy to understand, useful for tracking the effect of quality improvement initiatives and encourages hospitals to explore processes that affect patient safety.4 5 Opponents dissent that the HSMR has too many methodological weaknesses and has not been well-validated as a useful measure.6 7 Thus, while some countries have endorsed and publicly released HSMR results, other jurisdictions have felt that it is not a suitable tool for monitoring quality improvement.8

One major concern of the HSMR is the potential for ‘gaming’ in which the rules of the calculation are exploited to achieve the semblance of a good outcome without actually changing care processes. Because calculating the expected number of deaths is dependent on how patients have been coded, an opportunity exists for a hospital to alter how sick their patients appear to be.9 In the UK, there has been evidence that systematic coding differences may affect HSMR results.10 Specifically, recoding a patient as palliative can be a simple way to exclude deaths that would otherwise have been included in the calculation. Since publication of the HSMR in the UK, there has been a noted increase in palliative coding,11 with two trusts increasing the number of deaths labelled as palliative to about 50%.12 In particular, a public inquiry into one UK hospital trust highlighted how abrupt changes in palliative care coding can quickly alter HSMRs.13

In 2007, the Canadian Institute for Health Information (CIHI) announced plans to release HSMRs for all large hospitals in Canada (excluding the province of Quebec) participating in its Discharge Abstract Database (DAD) programme.14 Results were ultimately released in November 2007, and the public relations ramifications for hospitals with high HSMRs were severe.15 16 Similar to what occurred in the UK,13 defensive claims were made that poor rates were due to misunderstandings in palliative coding,17 although CIHI had prepared and released palliative care coding guidelines in 2006.18

Since 2007, HSMRs across Canada have been declining, which has been relayed to the public as a sign that releasing the data has had a salutary effect.19 20 The purpose of this study was to explore how palliative coding in Canadian hospitals has changed since announcing plans to release the HSMR, and how that adjustment may have affected subsequent results.

Methods

Ethics approval and funding

This study was approved by both the Lakeridge Health and Mount Sinai Hospital Research Ethics Boards. Funding was provided by unrestricted continuing medical education funds from Lakeridge Health.

Data

A request was made to CIHI to access its record-level DAD from Fiscal Years 2004–2009 with anonymity for hospital and patient identifiers. This database includes 12 593 329 discharges across 606 hospitals in Canada (excluding the province of Quebec) from April 2004 to March 2010, which constitutes our entire study population. The DAD includes a ‘most responsible diagnosis’ which is the principal condition responsible for the patient's stay in hospital and up to 24 other discharge diagnoses typed as comorbidity, secondary, transfer or other diagnoses.21 Details on the DAD are presented elsewhere.22

The authors independently recalculated monthly HSMRs using the methodology released by CIHI and the same inclusion and exclusion criteria.3 The CIHI HSMR includes inpatient deaths only. A palliative diagnosis was identified with International Classification of Diseases-10-CA code Z51.5; CIHI excludes cases when Z51.5 is the most responsible diagnosis code. Total number of diagnoses/case was calculated as the sum of coded discharge diagnoses/case. Total number of interventions/case was calculated as the sum of Canadian Classification of Health Interventions codes/case. Crude percentages were calculated as the variable divided by the total number of cases each month. Charlson comorbidity scores were calculated using the methods described by CIHI.3

While expected HSMR is based on all hospitals, CIHI only releases the HSMRs of hospitals which have at least 2500 annual cases meeting inclusion criteria. As hospitals were de-identified, we labelled institutions as having publicly released data as those with at least 15 000 qualifying HSMR cases over the six study years (ie, an average of at least 2500/year), which was 84 hospitals.

Statistical analysis

Data were analysed using SPSS V.19.

Variables visually exhibiting seasonality on plots were adjusted using the SPSS multiplicative algorithm function, which yields seasonal component factors proportional to the overall series level.

To construct a hospital standardised palliative ratio (HSPR), we used the same approach applied to building the HSMR.12 Using data from 2004 to 2005, we constructed a binary logistic regression model to define variables and coefficients that would predict the expected number of cases with palliative code Z51.5 as the most responsible discharge diagnosis (the CIHI HSMR is also based on 2004–2005 data). Inclusion criteria were all elective or urgent admissions to an acute care institution; cases with a discharge disposition of stillborn or cadaver were excluded. Final included variables were any diagnosis of metastatic cancer, length of stay, age, number of interventions and being a medical versus surgical case. Details on this model are found in the appendix. The HSPR is interpreted as the observed number of cases with palliation as the most responsible diagnosis/expected such number×100.

To calculate an HSMR which included palliative cases (HSMR+palliative), we needed to estimate a coefficient value for these records. In the standard CIHI HSMR, the most responsible discharge diagnosis is given a coefficient, but no similar value was derived in cases identifying palliation as the main diagnosis. For many cases with palliation as the most responsible diagnosis we found that no further diagnoses were provided; when further diagnoses were coded, it was unclear which to consider the next most responsible condition. Therefore, we used three approaches to estimate HSMRs that included palliative cases.  First, we ran a sensitivity analysis in which the palliative coefficient would range from the minimum to the maximum possible values of the normally included discharge diagnoses. The purpose of this approach was to capture the entire probable range of HSMRs if another diagnosis had been entered as the main diagnosis.  Second, we estimated a palliative coefficient based on the expected mortality of patients in Canadian palliative units from other studies,23 24 which is detailed in the appendix. In the absence of primary data, this approach provided an indirect estimate of what the palliative coefficient should be based on existing literature.  Third, we reconstructed the usual CIHI HSMR binary logistic regression with all variables in the same categorisation, but with the addition of palliative care as one of the main diagnosis groups. This approach provides a coefficient for palliative cases within the existing CIHI statistical model. The usual CIHI HSMR regression was also reconstructed with complete elimination of the main diagnosis group as a variable. This approach used the existing CIHI regression without the single variable that differentiated palliative and non-palliative cases. Details of these three approaches are provided in the appendix.

We defined April 2004 to March 2006 as the time prior to changes in palliative coding and HSMR publication, April 2006 to March 2008 as the time during and April 2008 to March 2010 as the time post. The t-tests were used to compare the first and last time intervals. Corrections for multiple comparisons were not needed as comparisons among other different time periods were not examined. Ordinary least-squares linear regression was used to analyse trends in clinical and demographics variables over time.

Results

Clinical and demographic characteristics of study population and crude mortality and palliative rates

Table 1 outlines the baseline characteristics of the study population and their trends over the study period. Inpatients in Canada have been gradually getting older with increasing comorbidities. Mean age, mean number of diagnoses/patient, mean number of interventions/patient and Charlson scores have all been increasing. As expected, the crude proportion of inpatient deaths and the crude proportion of inpatients with a palliative code as either a most responsible or any discharge diagnosis have correspondingly been increasing as well (table 1 and figure 1).

Table 1

Descriptive statistics of clinical–demographic characteristics and crude mortality and palliative percentages for Canadian inpatients, April 2004 to March 2010 (n=12 593 329).

	Descriptive statistics			Linear regression
	April 2004 to March 2006 (n=4 280 732)	April 2006 to March 2008 (n=4 158 014)	April 2008 to March 2010 (n=4 154 603)	Change/month*	p Value for change/month
Mean age, years (95% CI)	51.17 (52.14 to 52.19)	52.65 (52.63 to 52.68)	53.00 (52.98 to 53.02)	0.017 (0.014 to 0.019)	<0.001
Percentage of female (95% CI)	57.96 (57.92 to 58.01)	58.26 (58.22 to 58.31)	58.34 (58.29 to 58.39)	0.007 (0.005 to 0.009)	<0.001
Percentage of urgent admission	67.90 (67.85 to 67.94)	66.73 (66.69 to 66.78)	66.43 (66.38 to 66.47)	−0.029 (−0.034 to −0.024)	<0.001
Mean length of stay, days	6.88 (6.87 to 6.89)	7.03 (7.02 to 7.04)	7.28 (7.26 to 7.30)	0.008 (0.007 to 0.009)	<0.001
Mean n discharge diagnoses	3.55 (3.55 to 3.55)	3.79 (3.78 to 3.79)	3.99 (3.98 to 3.99)	0.008 (0.008 to 0.009)	<0.001
Percentage of Charlson score ≥3	2.87 (2.86 to 2.89)	3.31 (3.30 to 3.33)	3.27 (3.26 to 3.29)	0.006 (0.003 to 0.009)	<0.001
Mean n interventions	1.16 (1.15 to 1.16)	1.23 (1.22 to 1.24)	1.28 (1.27 to 1.29)	0.002 (0.002 to 0.003)	<0.001
Crude percentage of death	3.75 (3.73 to 3.77)	4.00 (3.98 to 4.02)	4.05 (4.03 to 4.07)	0.006 (0.004 to 0.007)	<0.001
Crude percentage with palliative code as any discharge diagnosis	1.71 (1.70 to 1.72)	2.38 (2.36 to 2.39)	3.32 (3.30 to 3.33)	0.033 (0.031 to 0.035)	<0.001
Crude percentage with palliative code as most responsible diagnosis	0.79 (0.78 to 0.81)	0.97 (0.94 to 1.00)	1.21 (1.18 to 1.25)	0.009 (0.008 to 0.009)	<0.001

*Based on β-coefficient from linear regressions of seasonally adjusted monthly data.

Figure 1

Crude monthly percentage of deaths and percentage of cases with palliative care as any diagnosis of Canadian inpatients, April 2004 to March 2010 (n=12 593 329).

A comparison of the percentage of inpatients with any palliative discharge diagnosis between hospitals with and without publicly released HSMR data is shown in figure 2. The visual trend of eventually more marked palliative coding in hospitals with publicly released data is supported by descriptive statistics. In 2004–2006, the mean percentage of cases coded with a palliative diagnosis was 1.67% in hospitals with publicly released data and 1.78% in hospitals without, for a mean difference of 0.11% (95% CI of difference 0.05 to 0.17, p=0.001). In 2008–2010, this pattern is reversed. Hospitals with publicly released data now have a higher percentage of cases coded with palliation at 3.40% compared to 3.18% in hospitals without (95% CI of difference 0.10 to 0.35, p=0.001).

Figure 2

Crude monthly percentage of cases with palliative care as any diagnosis, grouped by inpatients in hospitals that do (n patients=4 791 714) and do not (n patients=7 801 635) have publicly released Canadian HSMR results, April 2004 to March 2012.

Trends in standardised mortality and standardised palliative coding rates

Seasonally-adjusted nation-wide monthly HSMR rates are illustrated in figure 3 and the corresponding rates in the time periods before, during and after HSMR publication are shown in table 2. HSMR rates are significantly lower in 2008–2010 than in 2004–2006 by 8.55 points (p<0.001).

Figure 3

Seasonally adjusted HSMR (n=3 816 181) and HSPR (n=12 593 172) rates for Canadian inpatients, April 2004 to March 2010.

Table 2

Mean monthly HSMR and HSPR for Canadian inpatients, April 2004 to March 2010

	April 2004 to March 2006 (95% CI)	April 2006 to March 2008 (95% CI)	April 2008 to March 2010 (95% CI)
HSMR as calculated by CIHI (total n=3 816 181)	99.60 (98.22 to 100.99)	97.96 (96.25 to 99.67)	91.06 (89.84 to 92.28)
HSPR (total n=12 593 172)	100.72 (99.20 to 102.25)	121.74 (118.38 to 125.11)	149.56 (145.32 to 153.79)
HSMR including palliative cases (total n=3 940 586)
Substituting range of usual main diagnosis coefficient values for palliative cases
Minimum value	119.64 (118.10 to 121.18)	122.28 (120.70 to 123.86)	120.98 (119.55 to 122.42)
Mean value	115.06 (113.61 to 116.51)	116.71 (115.21 to 118.22)	113.77 (112.52 to 115.03)
Maximum value	100.43 (99.16 to 101.69)	99.69 (98.28 to 101.10)	94.24 (93.17 to 95.31)
Using an estimated palliative coefficient based on the literature	106.80 (105.47 to 108.13)	106.96 (105.53 to 108.39)	102.22 (101.12 to 103.33)
Reconstructed models using the same variables as the original HSMR except:
Include palliation as a main diagnosis	100.28 (99.00 to 101.56)	99.52 (98.09 to 100.96)	93.93 (92.83 to 95.04)
Exclude any main diagnosis as variable	102.85 (101.40 to 104.31)	106.42 (104.87 to 107.98)	105.16 (103.80 to 106.52)

The time period April 2006 to March 2008 is counted as the intervention period when there were national changes to palliative care coding and public release of HSMR results.

To compare with the HSMR, we constructed and calculated an HSPR expressing the ratio of observed cases with a most responsible discharge diagnosis of palliation to the expected number based on 2004–2005 data. As shown in figure 2 and table 2, the HSPR has been increasing, and has jumped 48.83 points in 2008–2010 compared with 2006–2008(p<0.001).

The HSMR strongly correlates negatively with the HSPR (Pearson r=−0.862, p<0.001).

Estimating the effect of including palliative cases on HSMR

In the time period 2008–2010, under three palliative care scenarios, the HSMR would have decreased by at most 6.35 points (p<0.001) and may have potentially even increased by 2.30 points (p=0.028) (table 2). This is in contrast to the observed HSMR decrease of 8.55. While we would caution that the differing calculations make the HSMR+palliative statistic not directly comparable to the HSMR, one could roughly use these differences to estimate that the increase in palliative coding contributed to at least one quarter of the observed HSMR decline.

In the final fiscal year 2009, of 84 hospitals with publicly released data over the study period, 6 had HSMR+palliative scores (when simply adding palliative care as a main diagnosis group) that were 10 or more points worse than the usual HSMR. The HSPR scores for these six hospitals were significantly higher than the remaining hospitals (347.4 vs 140.1, p<0.001).

Discussion

Across Canada, we demonstrate that since publication of the HSMR, hospitals have dramatically increased the number of patients labelled as palliative, a trend that may be more marked at sites with publicly released HSMR results. This change strongly correlates with the witnessed decrease in HSMR. Our results are in keeping with the UK experience, in which palliative care coding doubled since palliative exclusions began.12 The findings are also consistent with prior Canadian data suggesting that inclusion or exclusion of palliative cases can significantly alter HSMRs.25 In our study, including palliative cases would have at least modestly attenuated the rate of HSMR decline. Similarly, in the UK, including palliative cases would have changed whether some hospitals were labelled as high versus low performing.12 We suggest our results demonstrate that publicly releasing a quality indicator induces hospitals to improve their scores on that metric at least partially by modifying coding practices.

Accurate coding is an essential part of administrative medicine and public health management. Any form of coding involves subjectivity, but ethically coding a patient as palliative can be a particularly difficult task requiring clinical judgment and a standardised, consistent approach.9 In a public inquiry to one UK trust, much debate was held on the intentions behind palliative care coding changes, although no formal finding of fact was made.13 Regrettably, the close timing between CIHI's release of new palliative care coding guidelines in mid-2006 and then plans to release the HSMR make it difficult to distinguish between the relative contributions of these two effects. For example, in the UK, the 2007 broadening and subsequent 2010 renarrowing of the palliative definition was associated with a commensurate expansion and retraction in the percent of deaths coded under palliation.13 Changes in national coding guidelines would be expected to change institutional coding practices. This is an important limitation to any pre- and post-based study, when two closely spaced interventions cannot adequately be separated. Also, it is challenging to determine how much of the observed increase in palliative coding is due to improved awareness of the importance of proper labelling and thus is appropriate and ultimately beneficial. Indeed, over the past decade there may truly have been an increase in palliative care resource utilisation as has been observed elsewhere.26 If our observed increased palliative coding rate is due to such a trend, then one might hypothesise the rise should correspond with an equally proportionate uptake in palliative care services. Such a study is beyond this paper's scope, but we speculate it would be difficult for the phenomenal rise in HSPR to be entirely accounted for by these effects.

While our findings suggest that a component of gaming may exist, it is very important to note that even with including palliative cases, HSMR rates are probably still declining since 2007, albeit at a slower rate. Debate continues over the value and consequences of publicly reporting any quality metric.27 Publicly releasing the HSMR may very well have fostered adoption of evidence-based practices that improve patient safety and the quality of care, as anecdotally reported.19 20 However, because the HSMR is dependent on the subjective coding process, we feel the tool does not accurately quantify changes in hospital quality. As such, we agree with others who claim that as is, the HSMR is not a particularly robust instrument.28

Strengths of this study include the size of the study population that spans across a nation with a high number of pre- and postimplementation observation points. Weaknesses include those shared with all non-randomised, retrospective observational database studies29 and the aforementioned inherent weakness of having a competing intervention (changes in coding guidelines) in our analysis. Specific weaknesses for our study include being unable to definitively label which hospitals had publicly released data. As well, we could not provide a definitive coefficient for palliative cases when calculating HSMR+palliative scores that would fit into the original model. In particular, the literature-based coefficient is based on data that is heavily biased towards patients with cancer. Our study also focussed solely on palliative care coding; evaluating other methodological issues such as the effects of possible shifts in comorbidity coding,10 inclusion of readmitted patients,30 or shifts towards out-of-hospital or other-facility deaths31 32 on HSMR was beyond the scope of this study; each may have had an effect on the observed decline. Also, it is important to note that in contrast to other countries, Canada is unusual in allowing palliative care to be positioned as the primary diagnosis; such a difference may limit extrapolation of our findings to other nations.

Society demands transparency and accountability from their public hospitals, but identifying methods that openly encourage and accurately monitor quality remains a major challenge. We contend our findings indicate that to at least some degree, when publicly pressured to show improvement, administrators will seek ways to alter metrics in order to account for perceived local idiosyncrasies that they feel bias against their institution's performance. However, ‘real’ quality improvement more in the spirit intended may still occur. Recently, the English Department of Health has adopted use of the Summary Hospital-level Mortality Indicator which includes all palliative care cases and deaths occurring within 30 days of discharge;33 and HSMRs with palliative case adjustments can be calculated.12 Further experience should reveal whether such evolutions will lead to a truly accurate big-dot measure immune to variations in coding practice.