Variations in the hospital standardized mortality ratios in Korea.

OBJECTIVES: The hospital standardized mortality ratio (HSMR) has been widely used because it allows for robust risk adjustment using administrative data and is important for improving the quality of patient care.
METHODS: All inpatients discharged from hospitals with more than 700 beds (66 hospitals) in 2008 were eligible for inclusion. Using the claims data, 29 most responsible diagnosis (MRDx), accounting for 80% of all inpatient deaths among these hospitals, were identified, and inpatients with those MRDx were selected. The final study population included 703 571 inpatients including 27 718 (3.9% of all inpatients) in-hospital deaths. Using logistic regression, risk-adjusted models for predicting in-hospital mortality were created for each MRDx. The HSMR of individual hospitals was calculated for each MRDx using the model coefficients. The models included age, gender, income level, urgency of admission, diagnosis codes, disease-specific risk factors, and comorbidities. The Elixhauser comorbidity index was used to adjust for comorbidities.
RESULTS: For 26 out of 29 MRDx, the c-statistics of these mortality prediction models were higher than 0.8 indicating excellent discriminative power. The HSMR greatly varied across hospitals and disease groups. The academic status of the hospital was the only factor significantly associated with the HSMR.
CONCLUSIONS: We found a large variation in HSMR among hospitals; therefore, efforts to reduce these variations including continuous monitoring and regular disclosure of the HSMR are required.
OBJECTIVES: The hospital standardized mortality ratio (HSMR) has been widely used because it allows for robust risk adjustment using administrative data and is important for improving the quality of patient care.
METHODS: All inpatients discharged from hospitals with more than 700 beds (66 hospitals) in 2008 were eligible for inclusion. Using the claims data, 29 most responsible diagnosis (MRDx), accounting for 80% of all inpatient deaths among these hospitals, were identified, and inpatients with those MRDx were selected. The final study population included 703 571 inpatients including 27 718 (3.9% of all inpatients) in-hospital deaths. Using logistic regression, risk-adjusted models for predicting in-hospital mortality were created for each MRDx. The HSMR of individual hospitals was calculated for each MRDx using the model coefficients. The models included age, gender, income level, urgency of admission, diagnosis codes, disease-specific risk factors, and comorbidities. The Elixhauser comorbidity index was used to adjust for comorbidities.
RESULTS: For 26 out of 29 MRDx, the c-statistics of these mortality prediction models were higher than 0.8 indicating excellent discriminative power. The HSMR greatly varied across hospitals and disease groups. The academic status of the hospital was the only factor significantly associated with the HSMR.
CONCLUSIONS: We found a large variation in HSMR among hospitals; therefore, efforts to reduce these variations including continuous monitoring and regular disclosure of the HSMR are required.

INTRODUCTION

Measuring the quality of healthcare is a critical prerequisite for improvements to be achieved [1-3]. Among the three types of healthcare quality measures, outcome measures are of great intrinsic interest because they reflect all of the aspects of the services provided, unlike structure or process indicators [4]. Mortality is one of the most commonly used outcome measures because the data is readily available and very useful, but robust risk adjustment methodologies are required if mortality is used as a quality measure. Disease-specific risk adjustment models for mortality prediction have been developed and are available for various diseases such as cardiovascular diseases and cancers. Although disease-specific risk adjustment is useful for quality improvement in certain clinical areas, generic risk-adjustment methodology that encompasses all clinical conditions and reflects the overall quality of a hospital is more useful for hospital-wide quality improvement than disease-specific risk adjustment is.

Several generic risk-adjustment measures such as the MedisGroups and Disease Staging have been available for decades [5-7]. However, these generic measures require extensive efforts to collect detailed clinical data, thus are not widely used. In 1999, the hospital standardized mortality ratio (HSMR) was developed and is a generic risk-adjustment measure that can be calculated from administrative data [8]. After its introduction in the UK, the HSMR has been widely used in many countries including the US, Canada, and the Netherlands [9].

In Korea, disease-specific risk-adjusted mortality has been evaluated nationally for various clinical conditions such acute myocardial infarction and major cancers since 2005. However, to the best of our knowledge, hospital-wide overall mortality has never been assessed using adequate risk-adjustment methodology. Thus, we sought to measure and compare hospital inpatient mortality among Korean hospitals using HSMR methodology. Specifically, the aim of this study was to assess variations in hospital HSMRs overall and to compare hospital HSMRs across hospital characteristics and disease groups.

METHODS

Study Population

All inpatients who were discharged from any tertiary or general hospital in Korea with more than 700 beds between January 1 and December 31 in 2008 were eligible for this study. We chose 700-bed hospitals as the cutoff point for inclusion to increase the accuracy of the measured mortality rates using an adequate sample size. Sixty-six hospitals fit this description, and more than half of all Koreans admitted to a hospital have visited one of these hospitals.

Using the inpatient claims data from the National Health Insurance and Medical Aid from 2008, inpatients with conditions accounting for 80% of all inpatient deaths at these hospitals were selected. Our final study population included 703 571 patients, which includes 27 718 (3.94% of all patients) in-hospital deaths nested in 66 hospitals (Table 1). We excluded deaths due to respiratory failure, cardiac arrest, and ventricular fibrillation from the analysis because these deaths cannot be attributed to poor quality health care rather than disease characteristics. We also excluded patients who were transferred in and/or out of the hospital, those who were more than 100 years old, those who were less than 28 days old (neonates), and those who had a length of stay of less than one day or more than 365 days [9-16]. Patients admitted for palliative therapy were also excluded because their mortality correlates poorly with the quality of care received at the hospital. Patients who had inpatient claims within 7 days before or after the index admission were defined as transferred patients. We excluded transferred patients because they tended to be sicker than non-transferred patients were. According to these exclusion criteria, 2884 patients were excluded from the study population (Supplemental Figure 1).

Table 1.

Number of cases, fatalities, and the crude mortality rate for each MRDx

MRDx group	MRDx	Cases (n)	Deaths (n)	Crude mortality rate (%)
Certain infectious and parasitic diseases (A00-B99)	Septicemia	6720	1832	27.26
	Cancer of the breast	46 669	541	1.16
	Cancer of the bronchus, lung	64 328	3436	5.34
	Cancer of the colon	48 853	874	1.79
	Cancer of the esophagus	7674	327	4.26
	Cancer of the head and neck	11 405	286	2.51
	Cancer of the liver and intrahepatic bile duct	54 146	3443	6.36
Neoplasms (C00-D48)	Cancer of another GI organ, peritoneum	13 128	890	6.78
	Cancer of the pancreas	11 848	906	7.65
	Cancer of the rectum & anus	34 417	557	1.62
	Cancer of the stomach	74 002	2292	3.10
	Leukemia	16 051	951	5.92
	Non-Hodgkin’s lymphoma	16 355	549	3.36
	Secondary malignancy	5794	271	4.68
	Aspiration pneumonia	2561	401	15.66
Diseases of the respiratory system (J00-J99)	Chronic obstructive pulmonary disease	12 832	443	3.45
	Other lower respiratory disease	8310	351	4.22
	Pneumonia	53 149	1447	2.72
	Acute cerebrovascular disease	46 938	2675	5.70
Diseases of the circulatory system (I00-I99)	Acute myocardial infarction	14 240	883	6.20
	Congestive heart failure	8021	402	5.01
	Coronary atherosclerosis	70 351	294	0.42
Diseases of the digestive system (K00-K93)	Alcohol-related liver disease	8194	412	5.03
	Gastrointestinal bleeding	13 974	400	2.86
	Non-alcoholic liver disease	16 031	794	4.95
Diseases of the genitourinary system (N00-N99)	Acute renal failure	3275	342	10.44
	Chronic renal failure	20 045	640	3.19
Injury, poisoning, and other consequences of external causes (S00-T98)	Intracranial injury	10 178	636	6.25
	Poisoning	4082	443	10.85
	Total	703 571	27 718	3.94

MRDx, most responsible diagnosis; GI, gastrointestinal.

Variables

The patient outcome (alive and dead) and dates of death (if relevant), were obtained from the 2009 National Health Insurance registry. The patient characteristics required for calculating HSMRs were derived from the National Health Insurance and Medical Aid inpatient claims data. The claims data include patient demographic information such as age, gender, principal diagnosis, comorbidities, surgical procedures, and the whether admission was elective or emergency. Patient comorbidities were abstracted from secondary diagnoses codes. In addition, hospital characteristics such as academic status (academic vs. non-academic), ownership (public vs. private), bed size, and geographic location (metropolitan vs. non-metropolitan) were identified.

Hospital Standardized Mortality Ratio

The final outcome variable in this study was the HSMR, which represents hospital-wide inpatient mortality. The HSMR is the ratio of the actual number of in-hospital deaths to the expected number of in-hospital deaths, for the most responsible diagnosis (MRDx) accounting for 80% of inpatient mortality [16]. Patients were grouped according to their MRDx category using the HSMR methodology, and the Korean Classification of Diseases, 6th revision was used to map their principal diagnoses to MRDx. Among 258 MRDx of all inpatient, 29 MRDx accounted for 80% of all deaths in this study.

Using logistic regression, risk-adjusted mortality prediction models were built for each MRDx. Then, using model coefficients, the number of expected deaths was determined based on the sum of the probabilities of in-hospital death for patients from each hospital. The logistic regression models incorporated the factors affecting mortality, including age, gender, urgency of admission, income level (National Health Insurance vs. Medical Aid), and comorbidities. The Elixhauser comorbidity index was used to adjust for comorbidities [17-20]. The diagnosis related group (DRG) screening method embedded in the Elixhauser comorbidity index was used to differentiate comorbid conditions present on admission from complications that occurred during hospitalization. If the MRDx was a malignant neoplasm, variables reflecting cancer stage such as undergoing surgery, chemotherapy, and/or radiation were also included in the models. We also included principal diagnoses codes and disease-specific mortality risk factors for further risk adjustment. The predictive power of the model was assessed using the c-statistic and the fitness of the model was assessed using the Hosmer-Lemeshow (H-L) statistic [21]. To avoid over-fitting the model, the bootstrapping technique was applied to each model. In addition, 95% confidence intervals of the HSMRs were calculated using Byar’s approximation [22]. Further information about our risk-adjustment model can be found elsewhere [23].

Statistical Analyses

Variations in the hospital HSMRs across the included hospitals were assessed and their statistical significance was compared with the average HSMR, which is 100. In addition, multiple regression models were used to measure associations with hospital characteristics. To evaluate variations in hospital HSMR across different clinical conditions, we categorized hospitals into tertiles based on their overall HSMRs and the HSMRs of each MRDx group. Then, the HSMR distribution for each MRDx group across each tertile was analyzed within the tertile of the overall HSMR. Using the 6th edition of the Korean Classification of Diseases system, 29 MRDx were categorized into the following seven MRDx groups: particular infectious and parasitic diseases (A00-B99), neoplasms (C00-D48), and diseases of the circulatory system (I00-I99), respiratory system (J00-J99), digestive system (K00-K93), and genitourinary system (N00-N99) as well as injury, poisoning, and certain other consequences of external events (S00-T98). The institutional review board of Seoul National University College of Medicine (H-0910-015-296) approved this study. All data analyses were performed using SAS version 9.3 (SAS Inc., Cary, NC, USA).

RESULTS

Validity of Risk-adjusted Mortality Prediction Model

Table 2 shows the c-statistics and H-L statistics from the logistic regression models across the 29 MRDx. Except for the mortality prediction models of three MRDx, all of the models had c-statistics higher than 0.8. The c statistic was over 0.9 for five MRDx (17.2%) that included colon cancer, breast cancer, and pneumonia; between 0.9 and 0.8 in 21 MRDx (72.4%) that included acute cerebrovascular disease, acute myocardial infarction, and acute renal failure; and less than 0.8 in the remaining three MRDx (10.3%) that included poisoning, aspiration pneumonia, and congestive heart failure. However, in 14 out of 29 MRDx, the H-L statistics of these models were significant (p<0.05), which suggests poor model fitness.

Table 2.

C-statistics and H-L statistics for each MRDx

C-statistic range	MRDx	Cases (n)	-2logL	C-statistic	95% CI of c-statistic		H-L Pr>ChiSq
					Lower	Upper
C≥0.9	Cancer of the colon	48 853	5590.687	0.938	0.93	0.94	0.001
	Cancer of the breast	46 669	3739.008	0.936	0.93	0.94	<0.001
	Pneumonia	53 149	8989.760	0.921	0.92	0.93	0.001
	Cancer of the rectum and anus	34 417	3990.594	0.915	0.90	0.93	0.05
	Cancer of the stomach	74 002	14 416.941	0.903	0.88	0.93	<0.001
	Gastrointestinal bleeding	13 974	2416.709	0.900	0.88	0.91	0.47
0.8≤C<0.9	Cancer of the bronchus, or lung	64 328	20 960.620	0.891	0.87	0.91	<0.001
	Cancer of the head and neck	11 405	2004.884	0.888	0.86	0.91	0.14
	Acute myocardial infarction	14 240	4559.415	0.886	0.85	0.92	<0.001
	Other lower respiratory tract disease	8310	2092.522	0.879	0.86	0.90	0.09
	Coronary atherosclerosis	70 351	2906.599	0.875	0.84	0.91	0.08
	Septicemia	6720	5082.639	0.872	0.86	0.88	0.03
	Non-Hodgkin's lymphoma	16 355	3618.164	0.869	0.85	0.88	<0.001
	Cancer of the liver or the intrahepatic bile duct	54 146	19 035.828	0.868	0.86	0.87	<0.001
	Cancer of the esophagus	7674	2102.728	0.854	0.83	0.87	0.11
	Non-alcoholic liver disease	16 031	4881.563	0.845	0.83	0.86	0.10
	Leukemia	16 051	5619.912	0.843	0.79	0.90	0.001
	Chronic obstructive pulmonary disease	12 832	3107.190	0.842	0.83	0.86	0.04
	Cancer of the pancreas	11 848	5026.615	0.836	0.82	0.85	0.06
	Secondary malignancy	5794	1747.959	0.831	0.81	0.87	0.06
	Acute renal failure	3275	1727.085	0.829	0.80	0.85	0.02
	Chronic renal failure	20 045	4692.210	0.828	0.81	0.84	0.002
	Intracranial injury	10 178	3855.706	0.821	0.80	0.84	0.13
	Alcohol-related liver disease	8194	2708.256	0.820	0.80	0.84	0.24
	Cancer of another GI organ, or the peritoneum	13 128	5286.592	0.819	0.81	0.83	0.20
	Acute cerebrovascular disease	46 938	17 024.397	0.809	0.80	0.82	0.01
C<0.8	Poisoning	4082	2361.975	0.781	0.75	0.81	0.36
	Aspiration pneumonia	2561	1965.775	0.732	0.71	0.76	0.98
	Congestive heart failure	8021	2947.031	0.706	0.67	0.74	0.47

H-L, Hosmer-Lemeshow; MRDx, most responsible diagnosis; CI, confidence interval; GI, gastrointestinal.

Overall Variation Among Each Hospital’s Hospital Standardized Mortality Ratio

The HSMR widely varied across the hospitals included in this study (Figure 1). Of all of the hospitals, 25 (37.9%) had a significantly low HSMR and 17 hospitals (25.8%) had a significantly high HSMR. The remaining 24 hospitals (36.4%) showed an average HSMR. The highest hospital HSMR (189.0) was 4.1 times higher than the lowest HSMR was (46.0).

Figure 1.

Caterpillar plot of the overall hospital standardized mortality ratio (HSMR) for large hospitals in 2008. Each hospital's HSMR is graphed as the upper and lower 95% confidence limits. ■17 Hospitals had a significantly high HSMR, ▲24 hospitals had a non-significant HSMR, ●25 hospitals had significantly low HSMR.

Variation Across Hospital Characteristics

The HSMR for different sub-groups of hospitals are presented in Table 3 and Supplemental Table 1. The academic status of the hospital and Medical Aid insurance status of the patients were significantly associated with HSMR; however, the other hospital characteristics such as hospital location, the number of doctors, having older age patients, having an emergency admission, having undergone surgical procedures, and having cancer patients were not related. Moreover, large variations were found among non-academic hospitals (vs. academic), public hospitals (vs. private hospitals), and hospitals with less than 800 beds (vs. those with 800 beds or more).

Table 3.

The hospital standardized mortality ratio for various hospital characteristics

Variables	n	Mean	SD	Minimum	Maximum	p-value1
Total, all hospitals	66	108.5	27.1	46	189	-
Academic status						-
Academic	56	104	23.8	46	154.9	0.001
Non-academic	10	133.7	32	73.4	189	-
Ownership						-
Public	12	106	36	59.1	189	0.73
Private	54	109.1	25.1	46	167.5	-
Beds						-
<800	18	114	33.2	73.4	189	0.20
800-1000	29	111.2	22.7	70.9	154.9	-
≥1000	19	99.2	26.2	46	151.1	-
Hospital location						-
Metropolitan	42	109.6	27.8	46	189	0.66
Non-metropolitan	24	106.6	26.5	66.5	154.9	-
No. of doctors per 100 beds						-
<20	4	133.3	48.6	73.4	189	0.05
20-30	10	114.7	32.2	75.6	167.5	-
30-40	27	101.4	23.5	46	154.9	-
40-50	17	116.7	21.1	83.5	151.1	-
≥50	8	95.3	21.1	59.1	127.5	-
Nursing grade						-
1	4	81.2	35.1	46	128.4	0.13
2	14	108.4	27.1	59.1	154.9	-
3	30	107.4	24.1	73.4	167.5	-
4+	18	116.5	28.3	84.8	189	-

All p-values were calculated using t-tests and ANOVA with the Bonferroni correction for multiple comparisons.

Variation Across Most Responsible Diagnosis Groups

To evaluate agreement between overall HSMR and disease-specific HSMR among hospitals, hospitals were categorized into tertiles according to overall HSMR and HSMR by MRDx groups. The agreement between overall HSMR and disease-specific HSMR was limited (Table 4). In four out of seven MRDx groups, the MRDx group-specific HSMR showed fair agreement (kappa, 0.21 to 0.40) with the overall HSMR regarding the strength of the agreement. The level of agreement between the overall HSMR and HSMR by MRDx was fair and was highest among those with neoplasms (kappa, 0.682), yet lowest among those with injury, poisoning, and other external causes (kappa, 0.156).

Table 4.

The number of hospitals across the tertiles of the standardized mortality ratio for each MRDx group

MRDx group	HSMR tertiles1	Overall HSMR tertiles1			Agreement (%)	Kappa
		High (115.6-189.0)	Middle (95.0-115.1)	Low (46.0-94.6)
All		22	22	22
Certain infectious and parasitic diseases (A00-B99)	High (109.0-167.8)	12	6	4	47.0	0.205*
	Middle (89.5-108.5)	5	9	8
	Low (47.7-89.3)	5	7	10
Neoplasms (C00-D48)	High (130.6-259.9)	18	4	0	78.8	0.682***
	Middle (97.2-130.4)	4	15	3
	Low (44.1-93.3)	0	3	19
Diseases of the circulatory system (I00-I99)	High (114.2-168.5)	13	6	3	54.5	0.318***
	Middle (93.8-114.0)	8	9	5
	Low (43.7-92.9)	1	7	14
Diseases of the respiratory system (J00-J99)	High (120.1-200.9)	14	5	3	54.5	0.318***
	Middle (93.2-118.4)	7	9	6
	Low (32.5-91.9)	1	8	13
Diseases of the digestive system (K00-K93)	High (120.2-242.3)	14	5	3	62.1	0.432***
	Middle (85.8-119.8)	6	12	4
	Low (41.0 -84.9)	2	5	15
Diseases of the genitourinary system (N00-N99)	High (118.1-295.0)	13	8	1	49.2	0.239**
	Middle (82.9-115.7)	6	7	9
	Low2(11.4 -80.6)	3	6	12
Injury, poisoning and other consequences of external causes (S00-T98)	High (110.6-206.6)	12	6	4	43.8	0.156
	Middle (79.5-109.8)	5	8	9
	Low3 (12.3-78.3)	4	8	8

MRDx, most responsible diagnosis; HSMR, hospital standardized mortality ratio.

Number in parenthesis refers to the HSMR range for each HSMR tertile.

One hospital experienced no deaths and was excluded from the HSMR calculation.

Two hospitals experienced no deaths and were excluded from the HSMR calculation.

p<0.05,

p<0.01,

p<0.001.

DISCUSSION

We found that the hospital HSMR among large hospitals in Korea varied greatly. The mortality of the highest HSMR was 4.1 times higher than that of the lowest HSMR. Out of the 66 hospitals, about two-thirds of them showed either significantly high or low mortality while only about one-third showed average mortality. The magnitude of variations in hospital HSMR was larger than those of previous studies were. The range of the HSMR in this study was larger (46 to 189) than that in the UK (53 to 137 in 1999), Canada (56 to 142 from 2004 to 2007), and the Netherlands (68 to 132 in 2009) [10,15,24,25]. We considered the HSMR of large hospitals in Korea only; therefore, the HSMR variation among Korean hospitals could be larger if all hospitals are compared. Hospitals with a high mortality rate should make efforts to reduce it, and at the national level, efforts should be made to decrease the large variation in the mortality rates of all hospitals.

Among hospital characteristics included in the analysis, the academic status of the hospital and Medical Aid insurance status of the patients were associated with hospital HSMR. This finding was consistent with those of previous studies. Teaching hospitals showed lower hospital mortality than non-teaching hospitals did [26-28]. In addition, Medical Aid patients generally have higher mortality rates than non-Medical Aid patients do [29,30]. However, hospital location, number of doctors, having older age patients, emergency admission, undergoing a surgical procedure, and having cancer patients were not associated with hospital HSMR. Nevertheless, hospital characteristics might differ in their association with hospital HSMR if small hospitals are added to the analysis. The hospital HSMR also showed a fair level of agreement in most MRDx groups; therefore, a hospital showing a low mortality for certain clinical conditions did not necessarily have low mortalities for other clinical conditions. For example, hospitals that had a low mortality for neoplasm tended to have a high mortality in other conditions such as digestive diseases. This finding indicates that there are disease-specific factors determining mortality in addition to generic factors and that disease-specific hospital mortality should be evaluated with overall hospital mortality. The limited agreement of hospital HSMR across the MRDx groups also suggests that disease group-specific mortality should be disclosed to facilitate targeted quality improvement effort in hospitals and informed choices among patients.

The c statistic and H-L statistic are generally used to evaluate the predictive power and robustness of risk-adjusted models. The predictive power of our mortality prediction models was high. For most models, the c statistic had excellent (0.8 to 0.9) or outstanding (≥0.9) discriminative value while only three models showed acceptable discriminative value (0.7 to 0.8). The prediction models of this study also had a better range of c statistics (0.706 to 0.938) than previous studies did. In the UK and the Netherlands, the range was 0.668 to 0.939 [31] and 0.677 to 0.957 [32], respectively. Approximately half of all mortality prediction models indicated poor fitness according to the H-L statistic. However, when mortality prediction models were rebuilt using a subset of the study population for each MRDx, the H-L statistics of the models were no longer significant except for three MRDx models. This finding suggests that the significant H-L statistics of these models might have resulted from our large sample size rather than from poor fitness.

This study has several limitations regarding use of claims data for risk adjustment. First, claims data may not be accurate enough for risk-adjusted mortality prediction because code creep in claims data has been known to be prevalent. Thus, to evaluate the accuracy of our claims data, we chose two MRDx (pneumonia and colon cancer), and then compared comorbidities obtained from the medical records of all inpatients with these diseases who were admitted at four randomly selected hospitals in our study. We found that only 6% of comorbid conditions that appeared in the claims data were not found in the medical records. Second, the DRG screening method embedded in the Elixhauser comorbidity index may not accurately differentiate comorbid conditions from complications that occurred during hospitalization. Thus, to evaluate the accuracy of the DRG screening method, we compared the results of the DRG screening with that of the medical records we reviewed using the inpatient episode to evaluate the accuracy of the claims data. We found that only 6% of comorbid conditions based on DRG screening were judged as complications according to the medical records. The rate of disagreement was 4.1% for pneumonia and 23.4% for colon cancer. However, to differentiate comorbid conditions from complications accurately, the diagnosis coding system for claims data should include flags to discern whether a disease developed before or after hospitalization [33]. Third, the predictive power of mortality prediction models using claims data may not be as robust as those made using clinical data are. Thus, to compare predictive power of the claim data based models and clinical data based models, c-statistics of the claim data based model and clinical based model (using the inpatient data) were calculated. The model based on clinical data included comprehensive risk factors such as vital signs (temperature, blood pressure, and respiration rate) and laboratory results (pH, hemoglobin, and creatinine). The c-statistic of the model based on clinical data for pneumonia was 0.005 higher than that of the model based on claims data (0.940 vs. 0.945). Because of the small sample size for this analysis, a model based on clinical data that incorporated all variables of the model based on claims data could not have been built.

This study also has several methodological limitations, which are common in studies using HSMR methodology. First, coding variations among hospitals can result in inappropriate risk adjustment [10,14,33]. Because the comorbidity index and emergency admission episodes are two important variables for risk adjustment in HSMR methodology, variations in coding practice for these variables may have a potentially large impact on the HSMR of individual hospitals. When we included the interaction term of the Elixhauser comorbidity index for each hospital and individual hospital in our mortality prediction models, the interaction terms were significant only in 20 hospitals out of the 66 total hospitals. In addition, the odds ratios were not large (0.713 to 1.382) across these hospitals with significant interaction terms. Hospitals that tend to treat emergency patients with mild symptoms may have a lower HSMR than those that tend to treat more severe emergency patients, regardless of their quality of care [14]. In this study, the emergency admission and number of transfer episodes with a 1-day length of stay had a low mortality of 2.19% compared to the overall mortality. Therefore, we included only patients who visited the emergency room if they were charged a fee for emergency medical service and had a length of stay longer than 1 day. By clearly those who visited the emergency room, we tried to minimize the potential impact of different case-mix of these emergency patients on the HSMR calculation of individual hospitals. Second, applying different mortality timeframes may result in discrepancies between the HSMR of individual hospitals, and previous studies have reported inconsistent findings regarding the consequences of applying different mortality timeframes on HSMR [34,35]. We used in-hospital mortality as the mortality timeframe because other countries have reported national HSMR using in-hospital mortality as the timeframe [9], and the c-statistics of the mortality prediction models used in calculating the in-hospital mortality timeframe were higher than those employing a different mortality timeframe were. Moreover, a patient’s cause of death post-discharge could have been unrelated to the patient’s reason for previous admission. Third, the exclusion criteria of the HSMR methodology might create a selection bias [33]. We excluded transferred patients, admissions with an extremely long length of stay, and patients older than 100 years old from the analysis. However, in our study, excluding these patients did not seem to cause a substantial selection bias because it required only approximately 0.4% of the study population (2884 out of 706 455) to be excluded from the analysis.

Variables	β	t	p-value
Intercept	193.637	4.31	< 0.001
Academic status
Academic	-27.093	-2.14	0.04
Non-academic	0	-	-
Hospital location
Metropolitan	3.83	0.56	0.58
Non-metropolitan	0	-	-
No. of doctors per 100 beds
< 20	-22.17	-1.05	0.30
20-30	-3.758	-0.26	0.79
30-40	-6.231	-0.58	0.56
40-50	17.254	1.56	0.12
≥50	0	-	-
Over 65 years old (%)	-0.494	-0.7	0.49
Recipient of Medical Aid (%)	1.863	2.02	0.05
Admitted via the ER (%)	-0.653	-1.59	0.12
Received surgery (%)	-0.775	-1.31	0.19
Cancer diagnosis (%)	-0.456	-1.73	0.09
R2		0.380

ER, emergency room.