Building a Patient-Specific Risk Score with a Large Database of Discharge Summary Reports.

BACKGROUND There is increasing interest in clinical research with electronic medical data, but it often faces the challenges of heterogeneity between hospitals. Our objective was to develop a single numerical score for characterizing such heterogeneity via computing inpatient mortality in treating acute myocardial infarction (AMI) patients based on diagnostic information recorded in the database of Discharge Summary Reports (DSR). MATERIAL AND METHODS Using 4 216 135 DSRs of 49 tertiary hospitals from 2006 to 2010 in Beijing, more than 200 secondary diagnoses were identified to develop a risk score for AMI (n=50 531). This risk score was independently validated with 21 571 DSRs from 65 tertiary hospitals in 2012. The c-statistics of new risk score was computed as a measure of discrimination and was compared with the Charlson comorbidity index (CCI) and its adaptions for further validation. RESULTS We finally identified and weighted 22 secondary diagnoses using a logistic regression model. In the external validation, the novel risk score performed better than the widely used CCI in predicting in-hospital mortality of AMI patients (c-statistics: 0.829, 0.832, 0.824 vs. 0.775, 0.773, and 0.710 in training, testing, and validating dataset, respectively). CONCLUSIONS The new risk score developed from DSRs outperform the existing administrative data when applied to healthcare data from China. This risk score can be used for adjusting heterogeneity between hospitals when clinical data from multiple hospitals are included.

Background

In the “big data” era [1] there is an increasing use of large healthcare administrative databases to conduct clinical and healthcare research, besides learning assessing delivery efficiency and effectiveness [2]. Among many interesting healthcare research topics facilitated by big data is how to evaluate and compare healthcare delivery quality across different healthcare providers [3]. Another interesting topic is use of multiple hospital administrative datasets in performing meta-data analyses [4]. Regardless of which applications are considered, one fundamental challenge is that administrative data are highly heterogeneous from hospital to hospital. Besides differences due to distributions of medical specialties and their delivery processes, the key source is that patient populations in hospitals are highly variable, and are not randomly distributed, which is a main source for heterogeneity between hospitals. The presence of such hospital heterogeneity creates a challenge to evaluating hospital performance or to pooling multiple hospital datasets in clinical studies, unless such heterogeneity is adjusted.

One analytic strategy is to develop risk scores (RS), quantifying characteristics of patients within hospitals, and to adjust risk scores as a way of controlling heterogeneity between hospitals [5]. There are many risk score calculation methods and risk scores developed for various medical conditions [6-8]. Among them, probably the most common is the Charlson comorbidity index (CCI).

Being stimulated by Chinese economic development, the Chinese government increases the investment into the healthcare system while emphasizing evidence-based evaluation to improve healthcare delivery quality and compare performance across hospitals. Besides the evaluation of hospital-wide performance, there is also increasing interest in evaluating disease-specific performance across hospitals. A major disease attracting much attention in the Chinese government is acute myocardial infarction (AMI), because AMI incidence has been rapidly increasing in recent years and is associated with high fatality. To the best of our knowledge, no AMI study in China, using multiple hospital datasets, has considered the adjustment for hospital heterogeneity. Similarly, there is no commonly accepted method used to adjust heterogeneity between hospitals in evaluating and comparing hospital-wide performance.

In this study, our objective was to develop a single numerical score for characterizing inpatient mortality associated with treating acute myocardial infarction (AMI) patients based on comorbidity information generally recorded in the routinely collected administrative data in China.

Material and Methods

The data sources

Two large databases were used for this study. The first database (database A) is the Discharge Summary Reports (DSRs) from 49 tertiary hospitals in Beijing from 2006 to 2010. The second database (database B) consists of DSRs from 65 tertiary hospitals from various cities and provinces in China in 2012, all of which are directly regulated by the National Health and Family Planning Commission (NHFPC) and are different from those 49 hospitals in the first database. In China, all hospitals are classified into 1 of 3 tiers – primary, secondary, and tertiary hospitals – in which tertiary hospitals are the best. Typically, most tertiary hospitals are owned and managed directly by local government agencies that are responsible for public health. Some selected tertiary hospitals are directly managed by NHFPC. By government mandate, each hospital is required to submit DSRs on each hospitalization to their local health authority, forming local administrative databases. Health authority uses such an administrative database to assess the status of healthcare delivery through annual reports from local health authorities to the NHFPC.

Early on, NHFPC has designed a DSR form, and has required all hospitals to use the same DSR form, so that DSR databases are interoperable throughout the country. In general, key clinical information in DSR includes, per hospitalization, the basic demographics of every patient, admission and discharge dates, primary and secondary diagnoses (up to 7 secondary diagnoses), surgical procedures, primary physicians, outcome of hospitalizations, and costs of each hospitalization. From 2006 to 2010, the basic structure of DSRs was stable, with minor modifications made by adding more information or by adopting international coding schemes.

Study population

Among all 4 216 135 DSRs in database A, all DSRs with the primary diagnosis codes I21 and I22 in ICD-10 were included as AMI-related discharges. Excluding hospitals without cardiovascular patient admissions leads to 34 hospitals with 50 531 DSRs of AMI patients, and all were used for the development and validation of the risk score. All DSRs were randomly divided into the training dataset (n=25 204) to develop the score, and the testing dataset (n=25 327) to conduct the internal validation test. For external validation, we used database B, with 3 110 566 total DSRs, and it includes 21 571 AMI records from 35 hospitals.

Statistical analyses

In the data analysis, we used a conservative strategy to remove all DSRs that appear to have unimportant and unclear disease codes, or possible coding errors (Figure 1). The demographic variables include sex (male or female) and age in years at the time of admission. Other variables include the number of diagnoses, the number of procedures, CCI, length of stay (within hospital), and total cost during the hospitalization. Inpatient mortality was defined as death at discharge (i.e., death during hospitalization).

Figure 1

Flow chart of the diagnosis codes selection. PDC – primary diagnosis code.

To ensure the robustness of risk score calculations with all diagnostic codes, we identified all primary diagnosis ICD-10 codes on DSRs that reported death at discharge within each calendar year in database A. Then, we included those diagnostic codes only if they appeared at least once in a calendar year from 2006 to 2010.

Applying the logistic regression model, we regress the discharge vital status on indicators of all diagnostic codes in addition to several demographic variables. After fitting, we use the logistic probability function [9] to denote the predicted probability of death at discharge given the dichotomous independent variables to denote presence or absence of high-risk secondary diagnosis indicators. Those indicators are retained in the model only if it is statistically significant at the significance level of 0.05. From logistic models, we estimate odds ratios (OR) of all secondary diagnoses, and round them to their integers as resulting weights associated with corresponding secondary diagnoses. An individual’s risk score is computed as the weighted sum of all secondary diagnoses at discharge present at the time of admission, with estimated ORs.

To compare across different risk scores, we computed the c-statistic as a measure of discrimination and compared these values to those obtained using CCI and its common adaptions, including Deyo [10], D’hoore [11], Gahli [12], and Quan CCI [13]. All statistical analyses were performed using SAS®, version 9.13 (SAS Institute, Cary, NC).

Results

Characteristics of patients in datasets

In the training set of 25 327 discharges, the mean age of the patients was 63.4±13.2 years, and 73.2% were male. Similarly, there were 25 204 discharges in the testing dataset, patients had a mean age of 63.3±13.2 years, and 73.6% were male. In validating dataset B, the discharge number was found to be 21 571, the mean age was 62.3±12.8 years, and 75.2% were male. Other patient characteristics, including number of diagnoses, number of procedures, CCI, length of stay, total cost, and in-hospital mortality, are summarized in Table 1.

Table 1

Population characterization of AMI patients.

	Training datasetN=25327	Testing dataetN=25204	Validation datasetN=21571
Age, mean ±SD (year)	63.4±13.2	63.3±13.2	62.3±12.8
Male (%)	73.2	73.6	75.2
Number of diagnosis, mean ±SD	5.3±2.0	5.3±2.0	5.2±2.0
Number of procedures, mean ±SD	2.4±1.9	2.4±1.9	1.5±1.3
CCI, mean ±SD	1.8±1.0	1.8±1.0	1.7±1.0
Length of stay, median (day)	10	10	8
Total cost, median (CNY)	44,284	44,449	45,461
In-hospital mortality (%)	7.7	7.7	4.7

CCI – Charlson comorbidity index.

Compared to the training and testing dataset, the external validating dataset had fewer procedures performed, shorter length of stay, and lower in-hospital mortality. Because the training and testing datasets were from 2006 through 2010 and the validating dataset was from 2012, such differences may be expected in light of continuous improvement of AMI treatment and healthcare quality in China every year. This assumption was supported by the fact that from 2006 to 2010, the AMI causes of in-hospital mortality decreased from 8.97% to 6.91% in the Beijing hospitals in database A.

Development of a risk score

After integrating the high-risk diagnosis codes for the 5 years, we removed 149 codes on the basis of coding because of relatively low occurrences in at least 1 year. Additionally, 67 codes were not selected because their associations with inpatient death was not statistically significant. A total of 22 diagnosis codes were eventually identified to associate with inpatient mortality of AMI patients (Figure 1). Their ORs and corresponding weights in CCI are shown in Table 2. The high-risk secondary diagnoses were mainly composed of circulatory system diseases, such as cardiac arrhythmias, cerebral infarctions, and hemorrhaging. For each of the possible conditions that a given patient had, the combined risk score was calculated by summing the weights given above.

Table 2

Odds ratios of covariates in comorbidity selection model.

Codes	Description	OR	[95% CI]	Weight/presence in CCI*
I60	Subarachnoid hemorrhage	26.1	2.1, 326.7	1
C11	Malignant neoplasm of nasopharynx	24.9	2.0, 318.1	2
C81	Hodgkin’s Disease	21.3	1.1, 396.2	2
I46	Cardiac arrest	19.4	12.8, 29.3	×
C32	Malignant neoplasm of larynx	17.9	1.3, 250.1	2
C16	Malignant neoplasm of Stomach	15.9	2.0, 127.9	2
R57	Shock, not elsewhere classified	13.9	10.6, 18.2	×
Q25	Congenital malformations of great arteries	12.4	1.8, 85.4	×
R53	Malaise and fatigue	12.0	4.4, 32.8	×
Q23	Congenital malformations of aortic and mitral valves	11.8	1.4, 100.1	×
S06	Intracranial injury	11.0	1.2, 105.7	×
L89	Decubitus ulcer	10.4	1.3, 83.7	×
C90	Multiple myeloma and malignant plasma cell neoplasms	9.9	1.9, 53.1	2
I61	Intracerebral haemorrhage	8.0	2.7, 23.9	1
R04	Haemorrhage from respiratory passages	7.4	1.2, 7.6	×
K56	Paralytic ileus and intestinal obstruction without hernia	6.7	2.1, 21.5	×
N17	Acute renal failure	6.5	3.8, 11.1	×
I66	Occlusion and stenosis of cerebral arteries, not resulting in cerebral infarction	4.4	1.0, 19.0	1
J47	Bronchiectasis	2.9	1.1, 8.3	1
N18	Chronic renal failure	2.8	2.1, 3.7	2
I63	Cerebral infarction	1.8	1.2, 2.5	1
I50	Heart failure	1.3	1.1, 1.6	1

The number represents the corresponding weights in CCI, ‘×’ means not included in CCI.

Validation of the risk score

We used c-statistics to evaluate the logistic regression models in this study (Table 3) through 3 separate comparisons: using the testing dataset as an internal validation, an external validating dataset, and a technical comparison with the widely used CCI. On computed risk score, the c-statistics in training, testing, and validating datasets were 0.829, 0.832, and 0.824, respectively. These estimates indicate that the risk scores were quite stable.

Table 3

C-statistics in internal retrospective and external prospective validations.

	Training dataset	Testing dataset	Validation dataset
Demographics	0.768	0.765	0.689
+CCI	0.775	0.773	0.710
+Deyo	0.782	0.779	0.784
+D’hoore	0.783	0.779	0.784
+Gahli	0.780	0.778	0.784
+Quan	0.781	0.780	0.786
+RS	0.829	0.832	0.834

CCI – Charlson comorbidity index; RS – risk score.

When using the widely used CCI as the comorbidity index, the c-statistics of the logistic regression model for predicting AMI patient mortality were 0.775, 0.773, and 0.710 in training, testing, and validating datasets, respectively. The c-statistics for Deyo were 0.782, 0.779, and 0.784; 0.783, 0.779, and 0.784 for D’Hoore; 0.780, 0.778, and 0.784 for Gahli; and 0.781, 0.780, and 0.786 for Quan CCI. These results were similar to those of CCI, but lower than our novel risk score, indicating our risk score model was more suitable for AMI patients in China.

We also computed the c-statistics of the logistic regression model with only the demographic variables. They were 0.768, 0.765, and 0.689 for training, testing, and validating datasets, respectively, indicating that high-risk secondary diagnosis and demographic characteristic-composed risk scores can improve the predictive ability of the model.

Application of the risk score to hospital evaluation

As part of healthcare reform, there is growing interest in hospital evaluation. A government-endorsed criterion in evaluating hospital quality is via inpatient mortality. However, the major pushback to using inpatient mortality as an index to evaluate hospital quality is that hospitals are very different from each other because their patient populations are different (i.e., hospital-to-hospital heterogeneity). To correct this heterogeneity, we propose to use the risk score as a basis to compute the relative mortality ratio for each hospital. Computationally, we use the risk score to calculate the expected death probabilities of AMI patients. Then, aggregating all of death probabilities within individual hospital leads to the estimated expected inpatient mortality for that hospital. In other words, if the average disease severity for patients in one hospital is much higher than that for patients in another hospital, the expected inpatient mortality is expected be higher. Correcting this source of heterogeneity, we compute the relative mortality ratios, defined as the observed inpatient mortality over the expected inpatient mortality, for each hospital.

In the current application, we use the risk score established on the training set to compute the predicted death probabilities for all individuals in the external validation dataset. Averaging these computed risk scores within hospitals results in hospital-specific relative mortality ratios (Table 4). Naively basing hospital ranking on observed inpatient mortality (column 4 of Table 4), the hospitals are ordered from 1 to 35 as their ranks for these hospitals under considerations. However, based on relative mortality ratios, the hospital ranks are revised (column 5 of Table 4). Interestingly, some of the top hospitals remain as top hospitals. However, ranks for some other hospitals are changed substantially.

Table 4

Hospital ranking change after adjusted using the novel risk score.

Hospital No.	Observed mortality (%)	Expected mortality (%)	Relative risk ratio (O/E)	Unadjusted rank	Risk-adjusted rank
Hospital 01	0.76	0.09	0.12	1	1
Hospital 02	0.85	0.22	0.26	2	2
Hospital 03	1.56	0.42	0.27	3	4
Hospital 04	1.66	0.43	0.26	4	3
Hospital 05	2.27	0.96	0.42	5	6
Hospital 06	2.32	1.02	0.44	6	7
Hospital 07	2.65	1.40	0.53	7	9
Hospital 08	2.95	2.91	0.99	8	20
Hospital 09	3.33	2.05	0.61	9	11
Hospital 10	3.53	2.74	0.78	10	15
Hospital 11	3.82	2.19	0.57	11	10
Hospital 12	4.60	1.46	0.32	12	5
Hospital 13	4.93	5.17	1.05	13	21
Hospital 14	5.12	3.22	0.63	14	12
Hospital 15	5.26	3.40	0.65	15	13
Hospital 16	6.09	4.95	0.81	16	16
Hospital 17	6.19	2.94	0.47	17	8
Hospital 18	6.22	8.62	1.39	18	28
Hospital 19	6.57	6.48	0.99	19	19
Hospital 20	6.96	7.59	1.09	20	23
Hospital 21	7.03	7.45	1.06	21	22
Hospital 22	7.06	9.09	1.29	22	26
Hospital 23	7.29	8.89	1.22	23	24
Hospital 24	7.88	10.53	1.34	24	27
Hospital 25	7.95	7.16	0.90	25	18
Hospital 26	9.26	6.95	0.75	26	14
Hospital 27	9.36	11.54	1.23	27	25
Hospital 28	9.51	15.52	1.63	28	29
Hospital 29	9.79	17.37	1.77	29	31
Hospital 30	10.00	8.78	0.88	30	17
Hospital 31	10.27	17.80	1.73	31	30
Hospital 32	10.40	21.53	2.07	32	33
Hospital 33	10.45	21.82	2.09	33	35
Hospital 34	20.93	43.37	2.07	34	34
Hospital 35	25.00	49.89	2.00	35	32

Since discussion of hospital ranks is not of primary interest, we selected 3 hospitals to examine why their ranks changed substantially. Table 5 lists detailed characteristics of these 3 hospitals. We took hospitals A, B, and C as examples. Without disclosing their identities, it suffices to state that all 3 are well-known tertiary hospitals in China. In particular, hospital C is known to be the best hospital in treating AMI patients, while hospitals A and B are excellent in clinical areas other than AMI. In this case, use of unadjusted inpatient mortality would provide misleading ranks of hospitals. Indeed, after accounting for disease severity across hospitals, the new ranks of hospitals are much more consistent with the actual clinical assessment.

Table 5

An example of adjusting hospital heterogeneity in evaluation of AMI healthcare quality using the novel risk score.

	Hospital A	Hospital B	Hospital C
N	482	215	87
Age, mean ± SEM (year)	62.9±12.2	63.8±13.3	64.7±12.7
RS	2.52±3.28	4.25±7.32	6.54±7.19
Observed mortality (%)	3.53	5.12	4.60
Expected mortality (%)	4.55	8.12	14.46
Relative risk ratio (O/E)	0.78	0.63	0.32
Unadjusted rank	10	14	12
Risk-adjusted rank	15	12	5

AMI – acute myocardial infraction; RS – risk score.

Discussion

Using over 4 million DSRs, we have developed a comorbidity risk score that characterizes inpatient mortality of treating acute myocardial infarction (AMI) patients in these tertiary hospitals in China. Besides demographic variables, key predictors are multiple secondary diagnostic codes reported on DSR. After building a probability model for risk score calculation on a training dataset, we have checked their performance on an internal validation set. It appears that performance, measured by c-statistic, is satisfactory. Taking this predictive model to an external validation dataset, we were able to show again that the c-statistic remains quite impressively over 0.8 for predicting all inpatient morality of AMI patients. In conclusion, the novel risk score can be used for predicting disease severity for patients, and can thus be used for adjusting heterogeneity between hospitals. Such an adjustment is necessary for many clinical studies on clinical data extracted from multiple hospitals, such as health service research or evaluation of healthcare quality.

From the methodological point of view, the current risk score calculation synthesizes multiple variables for many different underlying diseases through using the logistic regression model, and becomes quite stable as a single index for application. When working with clinical diagnostic variables, some diagnostic variables are very sparse. To ensure stable estimation, the logistic regression needs to have a sufficient frequency [14]. For example, Peduzzi and Harrell reported that, when associated frequencies are less than 10, the regression coefficients in the logistic regression could be biased and estimated standard errors could be inflated [15,16]. Through appropriate variable selections, our risk score can be readily used for applications with desired robustness.

In comparison to CCI and its adaption to using administrative data, both indices are used to characterize disease severity of patients within hospitals. However, the CCI index is used to approximate the comorbidity burden, and is not designed for Chinese administrative data. As we have shown, its performance is not as good as our risk score.

Conceptually, the risk score is not just for AMI. In fact, our risk score shares the same motivation as other risk scores such as GRACE, CADILLAC, TIMI, and PAMI [6]. The key differentiator for our risk score is that it uses administrative data, as opposed to detailed information from clinics directly or electronic medical records. For developing countries like China, it is easier to develop risk scores for other clinical specialties

Risk scores have many applications. For example, clinicians can use our risk score to predict inpatient mortality based on baseline information collected when patients are admitted to hospitals [17]. Such a risk score, once validated, can help clinicians design effective patient management. If a patient is at high risk of inpatient death, the clinician will pay much more attention and will institute more active treatment. The second example is that our risk score can be readily applied to hospital evaluations, which is the application used in the present study. The third application is to optimize use by healthcare delivery organizations, by estimating disease burdens across hospitals.

It is also important to acknowledge the limitations in our development of risk scores. Because DSRs are the primary data sources for computing risk scores, distinguishing between pre-existing conditions and complications during hospitalization is not entirely unambiguous. In the current study, we attempted to minimize the influence of complications through choosing an appropriate set of high-risk diagnosis codes. Further, it is estimated that 92% to 94% of secondary diagnoses were pre-existing conditions [18].

Conclusions

The new risk score developed from DSRs outperform the existing administrative data-based indices when they are used in China. The primary use of this risk score is to adjust for heterogeneity between hospitals when clinical data from multiple hospitals are used. Another application is to predict inpatient mortality based on information at admission, and the predictive information may help clinicians to manage patients and treatments more efficiently.