Use of risk-adjusted CUSUM charts to monitor 30-day mortality in Danish hospitals.

BACKGROUND: Monitoring hospital outcomes and clinical processes as a measure of clinical performance is an integral part of modern health care. The risk-adjusted cumulative sum (CUSUM) chart is a frequently used sequential analysis technique that can be implemented to monitor a wide range of different types of outcomes.
OBJECTIVE: The aim of this study was to describe how risk-adjusted CUSUM charts based on population-based nationwide medical registers were used to monitor 30-day mortality in Danish hospitals and to give an example on how alarms of increased hospital mortality from the charts can guide further in-depth analyses.
MATERIALS AND METHODS: We used routinely collected administrative data from the Danish National Patient Registry and the Danish Civil Registration System to create risk-adjusted CUSUM charts. We monitored 30-day mortality after hospital admission with one of 77 selected diagnoses in 24 hospital units in Denmark in 2015. The charts were set to detect a 50% increase in 30-day mortality, and control limits were determined by simulations.
RESULTS: Among 1,085,576 hospital admissions, 441,352 admissions had one of the 77 selected diagnoses as their primary diagnosis and were included in the risk-adjusted CUSUM charts. The charts yielded a total of eight alarms of increased mortality. The median of the hospitals' estimated average time to detect a 50% increase in 30-day mortality was 50 days (interquartile interval, 43;54). In the selected example of an alarm, descriptive analyses indicated performance problems with 30-day mortality following hip fracture surgery and diagnosis of chronic obstructive pulmonary disease.
CONCLUSION: The presented implementation of risk-adjusted CUSUM charts can detect significant increases in 30-day mortality within 2 months, on average, in most Danish hospitals. Together with descriptive analyses, it was possible to use an alarm from a risk-adjusted CUSUM chart to identify potential performance problems.

Introduction

Monitoring hospital outcomes and clinical processes has become an essential part of modern health care.1–5 Hospital-wide mortality measures may be monitored as an indicator of the clinical quality. Compared with metrics based on a single diagnosis, hospital-wide mortality can potentially uncover cross-cutting concerns such as infection control, handoff of patient care between shifts, and care coordination. However, only a proportion of hospital-related deaths will be preventable and changes in hospital mortality are in general weakly associated with hospital performance and offer no explanation of the underlying cause.1 Nonetheless, monitoring hospital mortality using risk-adjusted cumulative sum (CUSUM) charts6,7 may still provide valuable insight at a low cost for the hospital. When a risk-adjusted CUSUM chart signals significant changes in hospital mortality, additional descriptive analyses can be used to compare mortality in patient subgroups before and after the alarm, which can be used to guide further in-depth analyses directed specifically at the relevant time period and patient group.

The risk-adjusted CUSUM chart is a sequential monitoring scheme, which excels at detecting small persistent changes in a process parameter of interest, while adjusting for case-mix.8 The method has been implemented in many different medical settings, eg, to monitor the performance of surgeons and general practitioners, to analyze surgery learning curves, and to monitor in-hospital outcomes.2,6,9 The risk-adjusted CUSUM chart is a well-established method,6,7,10,11 but the literature lacks detailed examples on how the method has been implemented in practice in hospital settings. Here, we describe how risk-adjusted CUSUM charts were implemented in Denmark to monitor 30-day mortality after admission at a public hospital, using 2015 as an example. In addition, we provide an example on how an alarm of increased hospital mortality from the charts could guide further in-depth analyses at the hospital.

Materials and methods

Materials

In this study, we used routinely collected data from Danish population-based nationwide registries. Denmark has ~5.6 million citizens.12 The Danish National Health Service provides universal tax-supported health care, guaranteeing unfettered access to general practitioners and hospitals for all residents.13 Unambiguous linkage of all Danish registries at individual level is possible using the unique 10-digit Central Personal Register number assigned to each Danish citizen at birth and to residents upon immigration.12 Each month, we obtained data from the Danish National Patient Registry (DNPR)14 and the Danish Civil Registration System (DCRS)12 to get information on all inpatient hospital admissions.

Danish National Patient Registry

The DNPR is a nationwide administrative registry that has collected data from all Danish hospitals since 1977 with complete nationwide coverage since 1978.14 In 1995, the DNPR was expanded to cover outpatient, emergency room, and psychiatric hospital contacts, and in 1999, hospital examinations and treatments were included. Reporting to the DNPR at least once a month is required by law, but in practice, most hospitals do so weekly or, in some cases, daily. We obtained data on type of admission (acute/elective), hospital of admission, admission and discharge dates, and primary and secondary diagnoses as recorded at discharge (coded according to the International Classification of Diseases [ICD]-10 since 1994) from the DNPR. The primary diagnosis was categorized into groups determined by the first three digits of the ICD-10 code. If a patient was discharged and readmitted to another hospital/department within 24 hours, the readmission was interpreted as a hospital/department transfer and the two admissions were combined into one. Similarly, if the patient was discharged and readmitted to the same hospital department in <24 hours, the two admissions were combined into one. All public hospitals in Denmark were grouped into 24 units corresponding to administrative hospital units in Denmark. We ascribed admissions to the last hospital unit of the combined admission if the combined stay was <30 days and to the hospital unit where the patient was hospitalized at day 30, in case of hospitalizations of ≥30 days. We classified comorbidity according to the Charlson Comorbidity Index15 (CCI) using data from all public hospital admissions since 1977 (visits at emergency departments excluded). The CCI was categorized into the following three groups: 0 (no comorbidity), 1–2 (medium), and 3+ (high). Denmark consists of five geographical regions. An interregional transferal was defined as an admission with a transfer between hospital units in different regions during the first 30 days of admission.

Danish Civil Registration System

The DCRS is a nationwide administrative registry established in 1968 that includes individual-level information on all persons residing in Denmark. We obtained data on vital status, date of birth, sex, and marital status from the DCRS. Information on vital status is updated on a daily basis, and the data are virtually complete and have high accuracy.12 We determined vital status 30 days after hospital admission, and both in and out of hospital deaths were considered. Age at hospital admission was categorized into 10-year age groups, with patients older than 90 years concatenated into one category. We defined marital status at admission as married, never married, divorced, or widowed. If marital status was unknown at the time of admission, we excluded the admission (~10 admissions nationwide each month).

Implementation of risk-adjusted CUSUM charts

The risk-adjusted CUSUM chart plots the function: where W is a weight assigned to each value of t. In this study, the risk-adjusted CUSUM charts were updated on a patient-to-patient basis, ie, each value of t corresponds to a new admitted patient. Consequently, the weights W are given by

Here, Y is the outcome of patient t (death within 30 days after admission yes/no) and p is the expected probability of the outcome estimated from a prediction model based on data from a reference period. Finally, R > 1 is a specified OR increase in the outcome rate, as compared to the reference period, that the risk-adjusted CUSUM chart is set to detect. The weight W is positive if the patient has the outcome, and negative if not, and the absolute value of the weight is large if the outcome is unexpected. Thus, in our study, if more patients died than predicted, the CUSUM function would increase. If the CUSUM function crosses a specified control limit h, sufficient evidence has accumulated that the outcome rate has changed by an OR of at least R as compared to the outcome rate in the reference period. In this study, the control limits were determined using simulations so that false alarms of increased 30-day hospital mortality should only happen once every 25 years, on average, for each hospital (Supplementary materials). We used R = 1.5 in this study. This choice was motivated as a compromise between the desire of being able to detect clinically relevant increases in 30-day mortality, at the same time avoiding that the CUSUM charts would signal an alarm in case of small temporary changes in the mortality rate. After the control limits had been determined, we did further simulations to estimate the average time it would take the risk-adjusted CUSUM charts to signal an alarm when the mortality rate had increased by an OR of R = 1.5 (Supplementary materials). Table S1 illustrates the CUSUM method calculations using a dummy data set.

Factors such as diagnosis coding and referral patterns are ever-changing in hospitals. To reduce the potential problem that predictions made from past data would become inappropriate due to changes in these factors, we used a recent period, 2014, as the reference period from which we used hospital admissions to build our prediction model. Using all inpatient hospital admissions to the 24 hospital units in 2014, we determined the number of admissions leading to death within 30 days of admission for each primary diagnosis (three-digit ICD-10 codes) and ranked them according to the number of deaths. The diagnoses with the largest number of deaths accounting for ~80% of all deaths were determined, and only admissions with one of these diagnoses were used to build the prediction model and included in the risk-adjusted CUSUM charts in 2015.3 In the process of identifying these diagnoses, we excluded hospital admissions with ICD-10 codes Z99 (no specified cause of death) and Z38 (live births). We will henceforth refer to admissions with one of these diagnoses as “CUSUM eligible admissions”.

To estimate the risk of 30-day mortality for CUSUM eligible hospital admissions in 2015, we used a logistic regression model based on the CUSUM eligible hospital admissions from 2014 including the following covariates: hospital code, admission type (acute/elective), primary diagnosis (three-digit ICD-10 code), quarter of admission, interregional transfer during the first 30 days of hospitalization (yes/no), CCI score (0, 1–2, and 3+), sex, age group (10-year age groups, 90+ in one category), and marital status (single, divorced, married, and widow). The prediction model was validated on all CUSUM eligible hospital admissions from 2013. The discriminative ability of the model was assessed using the c-statistic, and calibration of the prediction model was assessed by the calibration-in-the-large and the calibration slope from the framework as originally proposed by Cox.16–18

When using a risk-adjusted CUSUM chart, a starting point X0 of the CUSUM function has to be chosen (the chart needs to be initialized). Using the starting point X0 = 0 implicitly assumes that the process being monitored is under control when the monitoring begins, ie, that the outcome rate at the beginning of the monitoring period corresponds to the rate in the chosen reference period. It was uncertain if this was a reasonable assumption in our study. Consequently, we used an alternative approach when initializing the risk-adjusted CUSUM charts; to initialize a risk-adjusted CUSUM chart for a hospital on January 1, 2015, the CUSUM function was initialized at X0 = 0 on January 1, 2014, and then ran through the hospital-specific CUSUM eligible admissions from that year. The value of the CUSUM function on December 31, 2014, was then used as the initialization value of the risk-adjusted CUSUM chart on January 1, 2015.

While all risk-adjusted CUSUM functions were updated on a patient-to-patient basis, plotting a CUSUM function graphically on a patient-to-patient basis was judged infeasible due to the high number of daily inpatient admissions to the hospitals. Consequently, we only plotted the value of the risk-adjusted CUSUM function after the last admission each day. The risk-adjusted CUSUM charts were distributed to hospitals once each calendar month, witĥ3 months delay. The 3-month delay allowed time for the outcomes to be observed and to acquire and analyze the data. When a risk-adjusted CUSUM chart signaled an alarm of increased mortality in a hospital, we conducted a descriptive analysis using data from the period giving rise to the alarm. For comparison, we also did a corresponding descriptive analysis in an earlier period with no alarms. In the descriptive analyses, we tabulated patient characteristics and listed the diagnosis groups with the highest differences between observed and expected mortality and compared the results for the two periods. Depending on the results, additional subgroup analyses were performed. The descriptive analyses were conducted in collaboration with hospital representatives. Further in-depth analysis, such as medical record reviews, was decided exclusively by the respective hospital and performed without any obligation to report their findings. The risk-adjusted CUSUM chart was not reset after the alarm (or after the analysis of the alarm).

All data management and statistical analyses were performed using the SAS software, Version 9.2, with the exception of control limit simulations, which were performed on a dedicated computer using the SAS software, Version 9.4.

According to Danish legislation, no approvals from Institutional Review Board or ethics committees are needed for purely registry-based research. The study was approved by the Danish Data Protection Agency (Jr Number: AU 392 2007-41-0357).

Results

Among 1,091,660 admissions to the 24 hospital units in 2014, 77 diagnoses accounted for ~80% of all deaths within 30 days of hospital admission (Table S2). There were 437,647 admissions in 2014 with one of the 77 included diagnoses as the primary diagnosis, and we used these admissions to build the prediction model. Of the 1,090,461 admissions in 2013, 425,697 admissions had one of the 77 selected diagnoses as the primary diagnosis and we used these admissions to validate the prediction model. Among 1,085,576 admissions in 2015, 441,352 were CUSUM eligible admissions and were included in the risk-adjusted CUSUM charts. Patient characteristics of CUSUM eligible admissions for each year are shown in Table S3.

On the validation data, the prediction model had a c-statistic of 0.846. The calibration-in-the-large was 0.005 (95% CI, −0.025;0.035), and the calibration slope was 1.029 (95% CI, 1.016;1.042).

The size of the hospital units varied from 3,396 (of the 7,382 annual admissions) to 38,940 (of the 98,188 annual admissions) CUSUM eligible admissions in 2015. When considering only CUSUM eligible admissions, the crude 30-day mortality varied between 3.60% and 7.54% in 2015 among hospital units. Risk-adjusted CUSUM control charts for all the hospital units covering all of 2015 are shown in Figure 1. In 2015, the risk-adjusted CUSUM charts yielded a total of eight alarms of increased mortality. The median of the hospitals’ estimated average time to detect a 50% increase in 30-day mortality was 50 days (interquartile interval, 43;54), with a minimum and maximum of 40 and 92 days, respectively.

Figure 1

Risk-adjusted CUSUM charts for all 24 hospital units (A-X) in 2015.

Notes: The horizontal line is the control limit. The x-axis is the date, and the plotted values of the chart are the values of the risk-adjusted CUSUM function after the last admission each day.

Abbreviation: CUSUM, cumulative sum.

Management of an alarm—an example

In late February 2015, the risk-adjusted CUSUM chart for hospital unit H signaled an increase in 30-day mortality (Figure 1). The alarm was analyzed in June when data from March became available. The chart signaled in February, and after the signal, additional evidence kept accumulating for about a month. Afterward, the chart quickly dropped below the control limit, indicating that the increase in the mortality rate was temporary and followed by a period with a decrease in the mortality rate as compared to the mortality rate in 2014. For the initial descriptive analysis, we compared two periods of admission data from the hospital. We identified the period February 10 to March 7, 2015, as the period with possible performance problems, corresponding to the period when the CUSUM function was rapidly increasing. The CUSUM function seemed stable in the period before this, indicating a mortality rate comparable to that in 2014, and thus, we used the period January 13 to February 7, 2015, as the comparison period for the descriptive analysis, a period with the same distribution of weekdays. The observed and expected 30-day mortality risks among CUSUM eligible admissions in the first period were 8.11% and 6.74%, respectively, and those in the second period were 9.64% and 7.36%, respectively, showing that there was an overall increase in both observed and expected 30-day mortalities in the period up to the time of the alarm. In both time periods, we noticed excess mortality among the specific diagnoses “pneumonia, unspecified organism” (J18), “other chronic obstructive pulmonary disease (COPD)” (J44), and “fracture of femur” (S72) (Table 1). We interpreted this as an indication of a general quality problem among these patient groups. Later, in 2016, other measures of clinical quality similarly indicated that 30-day mortality following hip fracture surgery and 30-day mortality following a COPD diagnosis were higher than in other hospitals in 2015 and actions such as medical record reviews were undertaken.

Table 1

Descriptive analysis of the alarm from hospital unit H

Period	Diagnosis	Number of admissions	Number of observed deaths	Number of expected deaths	Observed–expected difference
January 13, 2015 to February 7, 2015 (alarm period)	Other chronic obstructive pulmonary diseases	71	12	5.53	6.47
	Pneumonia, unspecified organism	126	21	15.08	5.92
	Fracture of femur	76	12	7.16	4.84
	Secondary malignant neoplasm of respiratory and digestive organs	17	6	1.63	4.37
	Respiratory failure, not elsewhere classified	14	6	3.55	2.45
	Fever of other and unknown origins	58	5	2.62	2.38
	Atrial fibrillation and flutter	123	4	1.64	2.36
	Other symptoms and signs involving cognitive functions and awareness	9	3	0.79	2.21
	Other sepsis	31	8	5.84	2.16
	Malignant neoplasm of colon	17	3	0.91	2.09
February 10, 2015 to March 7, 2015	Other chronic obstructive pulmonary diseases	77	11	5.23	5.77
	Fracture of femur	76	12	8.20	3.80
	Pain, unspecified	41	6	2.86	3.14
	Pneumonia, unspecified organism	70	10	7.20	2.80
	Abnormalities of breathing	60	9	6.22	2.78
	Other symptoms and signs involving cognitive functions and awareness	20	5	2.22	2.78
	Volume depletion	25	5	2.31	2.69
	Vascular disorders of intestine	5	4	1.93	2.07
	Cerebral infarction	57	6	4.00	2.00
	Fever of other and unknown origins	45	4	2.05	1.95

Notes: Top 10 diagnoses with increased mortality in the following two different periods: a time period where the mortality rate caused the risk-adjusted CUSUM chart to signal an alarm (February 10, 2015 to March 7, 2015) and a previous time period (January 13, 2015 to February 7, 2015) where the mortality rate did not result in an alarm.

Abbreviation: CUSUM, cumulative sum.

Discussion

Routinely collected medical data can be used to create risk-adjusted CUSUM charts monitoring hospital mortality. Although an alarm from the risk-adjusted CUSUM chart does not explain the cause of the change in the mortality rate in itself, it allows the identification of a time interval in which the change occurred, which together with descriptive analyses may guide further in-depth analyses at the hospital. Under optimal conditions where the risk-adjusted CUSUM charts are updated as soon as 30-day mortality can be assessed, the presented implementation of risk-adjusted CUSUM charts in this study can detect significant increases in 30-day mortality within 2 months, on average, in most Danish hospitals. Together with descriptive analyses, it was possible to use an alarm from a risk-adjusted CUSUM chart to identify potential performance problems well in advance of other already implemented monitoring systems measuring selected clinical indicators.

In a previous study describing the implementation of risk-adjusted CUSUM charts to monitor hospital-wide outcomes in England,2 one of the key strengths of the implementation was that it incorporated an interactive presentation of the results via a web-based front end, which enabled users to monitor the CUSUM charts and analyze alarms themselves. A limitation of our implementation was that no such tool was available to the users and instead they had to rely on collaboration with the provider to obtain information on alarms related to their hospital. In connection with this, another limitation of our implementation was that the risk-adjusted CUSUM charts were only distributed on a monthly basis and with delay (in addition to the 30 days, it took to assess the 30-day mortality outcome). Using only the hospital’s own data to predict 30-day mortality would ensure that differences in diagnosis coding and so on between hospitals would not influence the predictions. As such, it is a limitation in our implementation that we built a single combined prediction model using data from all the different hospitals, instead of separate prediction models for each hospital, since this result in predictions being based on an “average” hospital, not the hospital itself. In contrast, having a combined prediction model also had its advantages, since it eliminated sparse data problems, and facilitated the practical implementation. The prediction model was judged to be well calibrated and have good discrimination. As such, we expect that the prediction model performed well on hospitals with “average” hospital coding, referral patterns, and so on, but we cannot rule out that predictions from hospitals with differences in these factors might be inappropriate. A strength of our implementation was that it used routinely collected administrative data from Danish population-based registries. Information on vital status from the DCRS is updated on a daily basis, the data are virtually complete and have high accuracy, and many diagnosis codes from the DNPR have been shown to have acceptable predictive values.12,14

Commonly, clinical indicators on hospital-wide mortality are reported quarterly or yearly.3,19,20 Our choice of implementation of risk-adjusted CUSUM charts made it possible to detect significant increases in 30-day mortality in ~2 months, on average, for most hospitals in Denmark. As shown in our example, the information on an alarm from a risk-adjusted CUSUM chart also has the potential of being more informative than traditional clinical indications of an increase in hospital-wide mortality over a long time period, since the descriptive analyses might reveal that the increased mortality is concentrated on specific subgroup of patients in a specific time period.

Monitoring decreases in hospital mortality may be important as well, since a decrease in hospital-wide mortality can be an indicator of improvements in hospital performance. For simplicity, we have focused on the implementation of risk-adjusted CUSUM charts to monitor increases in 30-day mortality in this study, but separate risk-adjusted CUSUM charts monitoring decreases in 30-day mortality were also implemented.7

A nonstandard method was used to initialize the risk-adjusted CUSUM charts. A more conventional method to initialize a risk-adjusted CUSUM chart would be to initiate the charts halfway between zero and the chosen control limit or to simply initiate the chart at zero.6 The method we used was purely heuristic, and using the more conventional methods may be considered in other settings. We also chose not to reset the charts after an alarm because of the time lag between the actual time of the alarm and the time it was discovered and analyzed. Resetting risk-adjusted CUSUM charts makes more sense in the traditional use of the CUSUM methods where an alarm is investigated with little or no time lag.

The importance of different aspects of the implementation of risk-adjusted CUSUM charts depends on the monitoring period. In our implementation, the decision not to reset the risk-adjusted CUSUM charts after an alarm and the method with which we initialized the charts were influenced by the fact that we planned to update the prediction model and reset the charts each January. When the monitoring period is short, proper initialization is important, or alarms in the start of the monitoring period might be delayed or missed entirely. In contrast, resetting the charts after an alarm is not that important in our implementation, since large increases in hospital mortality are likely to be temporary. When the monitoring period is long, the method with which the charts are initialized are probably of little importance, but resetting the charts after an alarm might be appropriate, if the aim is to detect a lower increase in mortality.

Conclusion

The presented implementation of risk-adjusted CUSUM charts has the potential to detect significant increases in 30-day mortality within 2 months, on average, in most Danish hospitals. Together with descriptive analyses, alarms from risk-adjusted CUSUM charts may identify potential performance problems in advance of other previously implemented clinical quality indicators.

Supplementary materials

Simulation of control limits

For each risk-adjusted CUSUM chart, we determined control limits from simulations corresponding to an average run length under control (ARL0) of 25 years, ie, we determined control limits so that each hospital should only get a false alarm every 25 years, on average. After the control limit corresponding to the desired ARL0 had been determined for a hospital, we used that control limit to simulate the corresponding average run length when the process was not in control (ARL), ie, the average time it would take for the risk-adjusted CUSUM chart to signal an alarm when the mortality rate had increased with an OR of R = 1.5.

Estimation of the control limit for each hospital corresponding to an ARL0 of 25 years, and the corresponding ARL, was done using the CUSUM eligible admission data from 2014, restricted to admissions assigned to that specific hospital. Since the risk-adjusted CUSUM charts in our study is updated on a patient-to-patient basis, ARLs are not naturally expressed in terms of time but in number of admitted patients. Therefore, the hospital-specific admissions were used to estimate the average number of daily CUSUM eligible admissions in each hospital and we used this as the number of simulated admissions that corresponded to a time period of 1 day.

We performed simulations of ARLs using the approach described in Figure S1.1 Patient covariate patterns were drawn at random, with replacement, from the specific hospital’s CUSUM eligible admissions from 2014 in step 1. When estimating the ARL0, the monitored process is assumed to be under control and estimation of 30-day mortality in step 2 can be done using the prediction model. We simulated 500 years of hospital admissions for each hospital in step 4 to ensure that the risk-adjusted CUSUM chart would signal an alarm before running out of simulated data, regardless of the value of the control limit that was considered (all values from 3.0 to 9.0 with 0.1 steps). We simulated the 500 years of data 1,000 times in step 5. The control limit corresponding to an ARL0 of 25 years was then estimated by linear interpolation of the two control limits corresponding to ARL0 values just >25 and <25 years (step 6). Having estimated a control limit corresponding to the desired ARL0, we repeated steps 1–5 in the simulation scheme, using only that specific control limit, to estimate the corresponding ARL. When estimating the ARL, the monitored process is assumed to be out of control, and risk estimation of 30-day mortality in step 2 can be modified using the following formula:1 where p is the estimated probability of 30-day mortality using the prediction model.

In real life, 30-day mortality cannot be assessed before 30 days have passed. This unavoidable delay can potentially make the estimate of the ARL misleading. Because of this, we added 30 days to the estimated ARL to get an estimate of the average time it would take a CUSUM chart to detect an increase in mortality under optimal conditions in a real-life setting where data can be acquired and processed immediately when available.

Simulation scheme for estimating average run lengths.

Table S1

Illustration of the CUSUM method calculations using a dummy data set

Patient ID (t)	Observed outcome (Yt)	Predicted outcome (pt)	Wt	Xt−1 + Wt	Xt
0					0.00
1	1	0.7	0.11	0.11	0.11
2	0	0.2	−0.10	0.01	0.01
3	0	0.1	−0.05	−0.04	0.00
4	0	0.3	−0.14	−0.14	0.00
5	1	0.5	0.18	0.18	0.18
6	1	0.1	0.36	0.54	0.54
7	0	0.8	−0.34	0.20	0.20
8	1	0.7	0.11	0.31	0.31
9	0	0.4	−0.18	0.13	0.13

Note: In this example, the CUSUM chart has been initialized at X0 = 0.

Abbreviation: CUSUM, cumulative sum.

Table S2

Primary hospital diagnoses in 2014 with the highest number of deaths

Diagnosis group	Number of deaths	Cumulative %	Description	Diagnosis group	Number of deaths	Cumulative %	Description
J18	1,822	6.29	Pneumonia, unspecified organism	R09	170	66.50	Other symptoms and signs involving the circulatory and respiratory system
J96	1,574	11.72	Respiratory failure, not elsewhere classified	I26	169	67.08	Pulmonary embolism
A41	1,407	16.57	Other sepsis	C79	163	67.64	Secondary malignant neoplasm of other and unspecified sites
E86	950	19.85	Volume depletion	I64	157	68.18	Stroke
Z03	855	22.80	Encounter for medical observation for suspected diseases and conditions ruled out	R50	152	68.71	Fever of other and unknown origins
J44	824	25.64	Other chronic obstructive pulmonary diseases	R41	146	69.21	Other symptoms and signs involving cognitive functions and awareness
R06	745	28.21	Abnormalities of breathing	C50	139	69.69	Malignant neoplasm of breast
J15	716	30.68	Bacterial pneumonia, not elsewhere classified	N17	132	70.15	Acute kidney failure
S72	709	33.12	Fracture of femur	K59	129	70.59	Other functional intestinal disorders
C34	594	35.17	Malignant neoplasm of bronchus and lung	I35	126	71.03	Nonrheumatic aortic valve disorders
I46	588	37.20	Cardiac arrest	R11	125	71.46	Nausea and vomiting
R10	587	39.23	Abdominal and pelvic pain	J90	124	71.88	Pleural effusion, not elsewhere classified
I50	551	41.13	Heart failure	I70	121	72.30	Atherosclerosis
I63	467	42.74	Cerebral infarction	I60	120	72.72	Nontraumatic subarachnoid hemorrhage
I61	466	44.35	Nontraumatic intracerebral hemorrhage	R07	112	73.10	Pain in throat and chest
Z51	448	45.89	Encounter for other aftercare	R29	107	73.47	Other symptoms and signs involving the nervous and musculoskeletal systems
I21	439	47.41	STEMI and NSTEMI	T88	104	73.83	Other complications of surgical and medical care, not elsewhere classified
R53	416	48.84	Malaise and fatigue	C67	101	74.18	Malignant neoplasm of bladder
N39	323	49.96	Other disorders of urinary system	D63	100	74.52	Anemia in chronic diseases classified elsewhere
K92	321	51.06	Other diseases of digestive system	K62	98	74.86	Other diseases of anus and rectum
R52	313	52.14	Pain, unspecified	R55	97	75.20	Syncope and collapse
K56	276	53.09	Paralytic ileus and intestinal obstruction without hernia	K72	96	75.53	Hepatic failure, not elsewhere classified
A49	261	53.99	Bacterial infection of unspecified site	C83	92	75.85	Nonfollicular lymphoma
I71	261	54.90	Aortic aneurysm and dissection	I25	90	76.16	Chronic ischemic heart disease
R57	261	55.80	Shock, not elsewhere classified	J81	90	76.47	Pulmonary edema
C78	243	56.63	Secondary malignant neoplasm of respiratory and digestive organs	C16	89	76.77	Malignant neoplasm of stomach
I48	242	57.47	Atrial fibrillation and flutter	R17	89	77.08	Unspecified jaundice
C18	241	58.30	Malignant neoplasm of colon	F05	84	77.37	Delirium due to known physiological condition
N30	238	59.12	Cystitis	C20	81	77.65	Malignant neoplasm of rectum
D64	223	59.89	Other anemias	C92	75	77.91	Myeloid leukemia
C25	220	60.65	Malignant neoplasm of pancreas	K83	75	78.17	Other diseases of biliary tract
S06	215	61.39	Intracranial injury	C15	74	78.42	Malignant neoplasm of esophagus
J69	205	62.10	Pneumonitis due to solids and liquids	K55	74	78.68	Vascular disorders of intestine
E87	193	62.76	Other disorders of fluid, electrolyte, and acid–base balance	M54	73	78.93	Dorsalgia
R18	192	63.43	Ascites	K25	72	79.18	Gastric ulcer
N18	189	64.08	CKD	J22	70	79.42	Unspecified acute lower respiratory infection
C61	181	64.70	Malignant neoplasm of prostate	R31	70	79.66	Hematuria
Z50	180	65.32	Care involving use of rehabilitation procedures	C90	69	79.90	Multiple myeloma and malignant plasma cell neoplasms
K70	170	65.91	Alcoholic liver disease

Abbreviations: CKD, chronic kidney disease; NSTEMI, non-ST elevation myocardial infarction; STEMI, ST elevation myocardial infarction.

Table S3

Patient characteristics of all CUSUM eligible admissions in 2013 (used to validate the prediction model), 2014 (used to build the prediction model), and 2015 (used in the risk-adjusted CUSUM charts)

Characteristic	2013		2014		2015
	n	%	n	%	n	%
Admission less than 24 hours
No	269,779	63.37	272,694	62.31	273,083	61.87
Yes	155,918	36.63	164,953	37.69	168,269	38.13
Admission type
Acute	346,778	81.46	356,682	81.50	363,318	82.32
Elective	78,919	18.54	80,965	18.50	78,034	17.68
Age group (years)
0–10	23,075	5.42	21,720	4.96	19,408	4.40
10–20	13,668	3.21	13,933	3.18	14,138	3.20
20–30	16,937	3.98	18,230	4.17	18,679	4.23
30–40	20,145	4.73	20,690	4.73	20,202	4.58
40–50	32,976	7.75	33,958	7.76	33,389	7.57
50–60	51,570	12.11	53,253	12.17	53,656	12.16
60–70	87,513	20.56	88,655	20.26	86,265	19.55
70–80	91,656	21.53	97,407	22.26	102,496	23.22
80–90	69,525	16.33	70,523	16.11	72,592	16.45
90+	18,632	4.38	19,278	4.40	20,527	4.65
Marital status
Widow	79,992	18.79	80,635	18.42	80,893	18.33
Divorced	60,181	14.14	64,522	14.74	65,736	14.89
Married	188,900	44.37	192,996	44.10	194,873	44.15
Never married	96,624	22.70	99,494	22.73	99,850	22.62
Interregional transfer during admission
No	422,046	99.14	434,109	99.19	437,768	99.19
Yes	3,651	0.86	3,538	0.81	3,584	0.81
Sex
Female	212,804	49.99	219,962	50.26	221,349	50.15
Male	212,893	50.01	217,685	49.74	220,003	49.85
Charlson Comorbidity Index
0	186,868	43.90	189,419	43.28	187,699	42.53
1–2	138,806	32.61	143,179	32.72	145,262	32.91
3+	100,023	23.50	105,049	24.00	108,391	24.56

Abbreviation: CUSUM, cumulative sum.