Are medical record front page data suitable for risk adjustment in hospital performance measurement? Development and validation of a risk model of in-hospital mortality after acute myocardial infarction.

OBJECTIVES: To develop a model of in-hospital mortality using medical record front page (MRFP) data and assess its validity in case-mix standardisation by comparison with a model developed using the complete medical record data.
DESIGN: A nationally representative retrospective study.
SETTING: Representative hospitals in China, covering 161 hospitals in modelling cohort and 156 hospitals in validation cohort. PARTICIPANTS: Representative patients admitted for acute myocardial infarction. 8370 patients in modelling cohort and 9704 patients in validation cohort. PRIMARY OUTCOME MEASURES: In-hospital mortality, which was defined explicitly as death that occurred during hospitalisation, and the hospital-level risk standardised mortality rate (RSMR).
RESULTS: A total of 14 variables were included in the model predicting in-hospital mortality based on MRFP data, with the area under receiver operating characteristic curve of 0.78 among modelling cohort and 0.79 among validation cohort. The median of absolute difference between the hospital RSMR predicted by hierarchical generalised linear models established based on MRFP data and complete medical record data, which was built as 'reference model', was 0.08% (10th and 90th percentiles: -1.8% and 1.6%). In the regression model comparing the RSMR between two models, the slope and intercept of the regression equation is 0.90 and 0.007 in modelling cohort, while 0.85 and 0.010 in validation cohort, which indicated that the evaluation capability from two models were very similar.
CONCLUSIONS: The models based on MRFP data showed good discrimination and calibration capability, as well as similar risk prediction effect in comparison with the model based on complete medical record data, which proved that MRFP data could be suitable for risk adjustment in hospital performance measurement. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The analysis was based on a nationally representative cohort of hospitals in China, from which random samples of patients admitted with acute myocardial infarction was drawn to represent the heterogeneity in outcome of care.

We used hierarchical generalised linear models that fully considered the patient clustering in hospitals and were able to distinguish the differences within and between hospitals, which suits the purpose to adjust for case-mix in hospital performance comparison.

We validated the finding that concise data extracted from medical record front page are good enough to reflect patients’ risk profile using the data from a closer year.

External validations that include more diverse hospitals and among other diseases will be needed in the future.

Introduction

Equal access to high-quality healthcare is one of the major aims in China’s recent public hospital reform.1 2 To continuously improve quality of care and mitigate its disparities across regions or hospitals, sustainable monitoring of hospital performance, particularly patient outcomes, is first required.3 4 The Ministry of Health (named as ‘National Health Commission’ now) of China established the Hospital Quality Monitoring System (HQMS) in 2011 to collect key information of all hospitalisations, including patients’ diagnosis and outcomes recorded in the medical record front page (MRFP) using a standardised form (online supplemental table S1).5 6 Although the MRFP lack of detailed information on treatment process such as lab test results or medications, with structured records on diagnosis, procedure and outcome, it could be used as a unique nationwide data source of outcome quality assessment (ie, in-hospital mortality).

Assessing quality of care between hospitals needs to take into account patients’ different demographic and clinical characteristics of patients between hospitals, like most of the prior studies have done based on a broad array of information from complete medical record.7–9 However, it is still unclear whether the MRFP data collected in HQMS can act as good surrogates for complete medical record model in estimation of risk-standardised mortality.

In China Patient-centred Evaluative Assessment of Cardiac Events (PEACE) retrospective study, we built a nationally representative sample of patients hospitalised for acute myocardial infarction (AMI) and extracted high-quality data from their complete medical records (including MRFPs), which provided an ideal condition to assess disparities in quality of care.10 We aim to develop a model of in-hospital mortality using their MRFP data, then assess its effect in case-mix standardisation by comparing with a model developed using the complete medical record data of the same patient cohort.

Methods

Patient and public involvement

No patient involved.

Study design and population

The design of China PEACE-Retrospective AMI study has been published previously.11 In brief, the study used a stratified two-stage random sampling method to select representative hospitals and patients admitted for AMI nationwide during 2001, 2006 and 2011. In addition, the study also included a more recent sample of patients admitted in 2015 using the same random sampling process. First, five regions (Eastern cities, Central and Western cities, Eastern villages, Central villages and Western villages) were used for representative hospital selection by simple random sampling method. Second, AMI cases (identified accoding to International Classification of Diseases—Clinical Modification codes, versions 9 (410.xx) and 10 (I21.xx), or key words from discharge diagnosis) were randomly selected from all patients who met the inclusion criteria in each selected hospital by random sampling method. Trained personnel at the national coordinating centres abstracted data from the medical records using standardised data definitions. Data abstraction quality was monitored by random audits that ensured that the overall variable accuracy exceeded 98%.11

As a retrospective study, written informed consent of patients were not required.

In this study, patients from year 2011 were regarded as the modelling cohort, and patients from the year 2015 were regarded as the validation cohort. Patients who transferred out to another hospital were excluded since we could not get their outcomes. A total of 8370 patients from 161 hospitals (96 secondary hospitals and 65 tertiary hospitals) were included as modelling cohort, and another 9704 patients from 156 hospitals (93 secondary hospitals and 63 tertiary hospitals) were included as validation cohort. In addition, if a hospital had less than 10 eligible patients per year, it would be further excluded from the hospital-level analysis. A total of 8269 patients (137 hospitals, 73 secondary hospitals and 64 tertiary hospitals) from modelling cohort and 9583 patients (132 hospitals, 71 secondary hospitals and 61 tertiary hospitals) from validation cohort were included in the further analysis (online supplemental figure S1).

Statistical analysis

According to study aim, we need to develop and evaluate a model predicting in-hospital outcome at patient level based on MRFP data from modelling cohort first. If the model performed well, then another model used to evaluate hospital quality of care would be built based on prior model. The validation cohort was used to conduct external evaluation of models. Hospital-level model would be built based on complete medical record data, which could be considered as ‘the best reference’. By comparing the difference and association of the indicators evaluated by the MRFP model and the complete medical record model, we could explore whether the model based on MRFP data had similar efficiency with that based on complete medical record data. The analysis roadmap was demonstrated in figure 1.

Figure 1

Analysis roadmap. HGLM, hierarchical generalised linear model; RSMR, risk standardised mortality rate.

Candidate predictors and outcome

Patient characteristics were selected as candidate predictors, according to previous AMI predictive models such as GRACE, TIMI and ACTION-GWTG.7–9 12–17 For the model based on MRFP data, the candidate predictors included demographic characteristics (gender, age, medical insurance status, ethnicity and marital status), admission department, diagnosis at admission (cardiac arrest) and at discharge (acute ST-segment elevation myocardial infarction (STEMI), infarction position, hypertension, diabetes, dyslipidaemia, cardiogenic shock, heart failure, stroke and renal failure), which was available from MRFP data. For the model based on complete medical record, we additionally include patients’ symptoms, vital signs and lab test results at admission.

In-hospital mortality, as the outcome variable in the models, was defined explicitly as death that occurred during hospitalisation, which was recorded both on the MRFP and elsewhere such as discharge record. For the accuracy of analysis, we used complete medical record as data source. We did not include patients who withdraw treatment as outcome since we could not get ‘withdraw’ information from MRFP data, though plenty of these patients might die soon after giving up treatment.

Patient-level model development and evaluation

A logistic regression model was built based on MRFP data from the modelling cohort. Area under receiver operating characteristic curve (AUC) and observed rates in deciles determined by model estimating value were used to evaluate the discrimination. Slope and intercept of regression equation between the observed and the predicted mortality was used to evaluate the calibration ability. To assess the overfitting of the model, we used the coefficients estimated from the logistic model to predict the probability of mortality in the validation cohort by multiplying coefficients by the observed risk factors variables and summing over for each subject. Then another logistic regression model was built, in which the dependent variable was observed mortality and independent variables were the predicted mortality generated as above. The slope different from 1 and the intercept different from 0 indicated overfitting.

Furthermore, we re-estimated the logistic regression model in the validation cohort used selected predictors above. If the estimated coefficients of new model were similar to prior, the selected predictors were considered to be stable. Discrimination and calibration were also evaluated in the re-established logistic model.

Complete medical record model was developed and validated based on the data from complete medical records, using the same method mentioned above. Additionally, we compared the performance of our complete medical record model and MRFP model with the GRACE in-hospital mortality model7 among development and validation cohorts by calculating the difference of AUC and the integrated discrimination improvement (IDI) (online supplemental appendix A).

Hospital-level model development and comparison

Hierarchical generalised linear models (HGLMs) were established among modelling and validation cohort separately using above selected covariates and hospitals as random effects. HGLM considered the patient clustering in hospitals and could be used to distinguish the differences of outcome within and between hospitals.

Hospital-level risk standardised mortality rate (RSMR) was used as an indicator to evaluating hospital quality of care in this study. The RSMR of each hospital could be calculated from HGLM as the ratio of predicted and expected mortality of the hospital, multiplied by the unadjusted rate of all hospitals. The expected mortality is the mortality rate of the hospital if patients in each hospital were treated in a ‘reference’ hospital; the predicted mortality accounted for the characteristics of a hospital (the hospital-level random effects of the model).8 18

We use two methods to compare the RSMR derived from the HGLMs based on MRFP and the complete medical record data: (1) absolute differences of RSMR from two models were calculated, and the distribution of differences was described using mean, median and maximum. (2) A linear regression model was built, with RSMR from the complete medical record data as the dependent variable and RSMR from the MRFP data as the independent variable. The slope of the model approaching 1 and the intercept approaching 0 indicated that the predicted probabilities from the two models were very similar. All above calculation and comparison would be conducted among the modelling and validation cohort separately.

All statistical inferences were performed on two-tailed test, and p<0.05 was considered statistically significant. The statistical software used is SAS V.9.4 (SAS Institute Inc, Cary, North Carolina, USA).

Results

Study population and characteristics

In the modelling cohort, the average age was 65.4±12.8 years, and 2519 (30.1%) patients were female. About half of the patients were admitted to cardiovascular department at admission. A percentage of 65.8% were diagnosed with STEMI, while 46.5%, 19.7% and 10.0% had comorbidities of hypertension, diabetes and dyslipidaemia, respectively. Cardiogenic shock occurred in 4.8% of the patients, and 0.1% of patients had cardiac arrest before admission (table 1). A total of 621 patients died during hospitalisation, accounting for 7.4% of the modelling cohort.

Table 1

Patients’ characteristics from MRFP data and in-hospital mortality in modelling cohort and validation cohort

	Modelling cohort (year 2011)	Validation cohort (year 2015)	P value
	N=8370	N=9704
In-hospital mortality	621 (7.4)	687 (7.1)	0.3793
Female	2519 (30.1)	3121 (32.2)	0.0028
Age (years), mean (SD)	65.4 (12.8)	65.9 (12.7)	0.0081
<40	195 (2.3)	213 (2.2)	<0.0001
40–49	910 (10.9)	891 (9.2)
50–59	1600 (19.1)	1816 (18.7)
60–69	2090 (25.0)	2674 (27.6)
70–79	2431 (29.0)	2590 (26.7)
≥80	1144 (13.7)	1520 (15.7)
Han	7701 (92.0)	9285 (95.7)	<0.0001
Married	7460 (89.1)	8740 (90.1)	0.0391
Having medical insurance	5126 (61.2)	7507 (77.4)	<0.0001
Admission at cardiology department	4087 (48.8)	6532 (67.3)	<0.0001
Admission diagnosis
Cardiac arrest	6 (0.1)	18 (0.2)	0.0362
Discharge diagnosis
STEMI	5509 (65.8)	4753 (49.0)	<0.0001
Acute extensive anterior MI	967 (11.6)	769 (7.9)	<0.0001
Acute anterior MI	1504 (18.0)	1310 (13.5)	<0.0001
Acute anterior septal MI	587 (7.0)	408 (4.2)	<0.0001
Acute inferior MI	2558 (30.6)	2214 (22.8)	<0.0001
Acute lateral MI	359 (4.3)	311 (3.2)	0.0001
Acute posterior MI	699 (8.4)	502 (5.2)	<0.0001
Acute right ventricular infarction	615 (7.3)	418 (4.3)	<0.0001
Hypertension	3894 (46.5)	5080 (52.3)	<0.0001
Diabetes mellitus	1650 (19.7)	2345 (24.2)	<0.0001
Dyslipidaemia	836 (10.0)	1434 (14.8)	<0.0001
Cardiogenic shock	403 (4.8)	510 (5.3)	0.1773
Heart failure	2853 (34.1)	3793 (39.1)	<0.0001
Stroke	655 (7.8)	1389 (14.3)	<0.0001
Renal failure	259 (3.1)	684 (7.0)	<0.0001

MI, myocardial infarction; MRFP, medical record front page; STEMI, ST-segment elevation myocardial infarction.

Compared with modelling cohort, patients in the validation cohort had a higher proportion of patients with medical insurance and admission in cardiovascular departments (p<0.001). Less proportion (49.0%) of patients were diagnosed with STEMI (p<0.001), while a greater proportion of patients had hypertension, diabetes, dyslipidaemia, heart failure and renal failure (p<0.05) (table 1). Six hundred and eighty-nine patients died during hospitalisation, accounting for 7.1% of the validation cohort, which was not significantly different from the modelling cohort (p=0.41).

Development and validation of patient-level model

A total of 14 risk factors were included in the MRFP model based on modelling cohort (figure 2A). Model discrimination was good, with the AUC of 0.78, and observed mortality rate ranging from 0.83% in the lowest decile of the predicted mortality rate to 26.88% in the highest decile. The slope of the calibration curve was 0.91, and the intercept was −0.007, which showed the good calibration ability of this model (table 2). The overfitting statistics were within an acceptable range (slope=1.01, intercept=−0.07), indicating that no overfitting exist.

Figure 2

Odds ratios (ORs) of MRFP model and complete medical record model based on modelling and validation cohorts. (A) MRFP model. Note: blue for modelling sample, orange for validation sample. Estimate of between-hospital variance: 0.592 (SE=0.125) in derivation sample and 0.773 (SE=0.147) in variation sample. (B) Complete medical record model. Note: blue for derivation, orange for validation sample. Estimate of between-hospital variance: 0.558 (SE=0.121) in derivation sample and 0.719 (SE=0.139) in validation sample. MI, myocardial infarction; MRFP, medical record front page.

Table 2

Performance of the MRFP model and the complete medical record model

Model	N	Discrimination		Calibration
		Area under ROC curve	Predictive ability* (mean rate of lowest/highest decile)	Calibration indices (slope, intercept)
MRFP model
Year 2011 (modelling cohort)	8370	0.776	0.83%–26.88%	(0.909, 0.007)
Year 2015 (validation cohort)	9704	0.794	1.00%–29.72%	(0.933, 0.005)
Complete medical record model
Year 2011 (modelling cohort)	8370	0.790	0.51%–27.96%	(0.940, 0.004)
Year 2015 (validation cohort)	9704	0.798	0.92%–28.69%	(0.927, 0.005)

*Observed rates in deciles determined by estimated model.

ROC, receiver operating characteristic; MRFP, medical record front page.

The predictors included previously were applied to the validation cohort to reconstruct the model, which showed that the effect direction and size were still similar (figure 2A). In the validation cohort, the AUC was 0.79, with observed mortality rate ranging from 1.00% to 29.72%, and the slope and intercept of the calibration curve was 0.93 and 0.005 (online supplemental figure S2 and table 2).

Using the same method, a complete medical record model was built, in which a total of 13 risk factors were included (figure 2B). The AUC of the model was 0.79, and observed mortality rate ranged from 0.51% in the lowest decile to 27.96% in the highest decile. The slope of the calibration curve was 0.94, and the intercept was 0.004 (online supplemental figure S2 and table 2). Similar with the MRFP model, the complete medical record model had good discrimination and calibration, as well as relatively stable coefficients when validated among the validation cohort (figure 2B, online supplemental figure S2 and table 2). Additional analysis showed that both our two patient risk prediction model had better AUC (all p value <0.001) and positive IDI among development and validation cohorts compared with the GRACE prediction model (online supplemental appendix A).

Development and comparison of hospital-level model

A total of 8269 patients (137 hospitals, 73 secondary hospitals and 64 tertiary hospitals) from modelling cohort and 9583 patients (132 hospitals, 71 secondary hospitals and 61 tertiary hospitals) from validation cohort were included in estimating the hospital-level HGLMs.

In the modelling cohort, the median hospital-level RSMR was 7.4% (IQR: 5.2%–10.1%). The median of absolute difference between the RSMR predicted by the complete medical record data and MRFP data was 0.08% (IQR: −0.67% to 0.53%), and the 10th and 90th percentiles were −1.8% and 1.6%, with no statistical significance (p=0.499). In the validation cohort, the median RSMR was 6.4% (IQR: 4.5%–10.4%), and the median of absolute difference was 0.05%, with 10th and 90th percentiles of −2.8% and 1.9% (online supplemental figure S3). For the regression model comparing the RSMR between the MRFP data and complete medical record data, the slope (intercept) was 0.90 (0.007) in the modelling cohort, while 0.85 (0.010) in the validation cohort (figure 3). The correlations among secondary hospitals were better than among tertiary hospitals.

Figure 3

Correlation of risk standardised mortality rate estimated by MRFP model and complete medical record model. (A) Modelling cohort and (B) validation cohort. MRFP, medical record front page.

Discussion

This study developed patient- and hospital- level MRFP models of in-hospital mortality of AMI, and took into account the patient case-mix in the hospital-level disparity analysis. These models based on MRFP data showed good discrimination and calibration capability, as well as similar risk prediction effect in comparison with the model based on complete medical record data, which proved that MRFP data could be suitable for risk adjustment in hospital performance measurement in China.

To our knowledge, the current study extended literatures in several ways. First, this is the first in-hospital mortality risk model based only on MRFP data in China. Currently, in China, it is still difficult to obtain detailed complete medical records data nationwide for quality monitoring due to the fragmentation in development and deployment of Hospital Information Systems and Electronic Medical Record Systems. In the USA, which faces similar challenges, several risk models have been developed using concise administrative claims data and successfully applied as substitute of complete medical record models.8 9 The key value of this model is to demonstrate how MRFP data from HQMS can serve as a solution for national quality assessment, rather than to identify coefficients of specific risk characteristics.

Second, the methods we chose for model development had advantages in managing patient case-mix and obtaining the hospital-level risk-standardised rates. We first selected an array of patient characteristics which influenced their risk profile significantly using backward logistic regression, and confirmed the stability of this array in the validation cohort. Then, we established a HGLM with these characteristics as covariates, which could take the correlation of patients in the same hospital into account and avoid underestimating the SE of other risk factors,18 19 thus it reflected the nature that patients were clustered within individual hospitals. This method has been well-tested in previous studies on hospital-level comparisons.7–9

Third, the MRFP model was robust when we repeated the analysis in validation cohorts, and more importantly, comparable to the model derived from complete medical records data with similar validities. Even though there is no real golden standard of risk standardisation, medical record data enable the most complete characteristics of patients’ demographic and clinical profile. The China PEACE retrospective study provided a unique opportunity to compare the MRFP model against the complete medical record model, because scanning copies of sampled medical records were collected and detailed information on patient characteristics had been centrally extracted from the front page and all other parts of medical records.

The feasibility of MRFP model has significant policy implications for China, as the government emphasised the importance of hospital performance monitoring.20 China needs a nationwide data platform, which supports timely, accurate and sustainable outcome measurement, since the outcomes of care such as mortality provide a global assessment of quality and have the most relevance to patients. However, outcome measurement is challenging because of variation among hospitals in patients’ risk profile; meanwhile, extracting data from electronic medical records is infeasible in most hospitals. Our study first proved that concise MRFP data that are available in the HQMS can sufficiently reflect patients’ risk profile, which makes it suitable to generate risk-standardised mortality rates at hospital level. Thus, this existing platform covering 1800 (73%) tertiary hospitals and 2300 (26%) secondary hospitals can serve as a base for national hospital performance measurement similar to the US Centres for Medicare & Medicaid Services’ use of administrative claims data.19 20 Moreover, some challenges should to be addressed. First, the quality of MRFP data across hospitals, particularly the completeness of comorbidity documentation and accuracy of diagnosis coding in diagnosis, needs to be improved.21 Second, for chronic conditions with low in-hospital mortality rates, data on postdischarge outcomes (eg, 30-day readmission rates) data need to be obtained from clinical registries, insurance claims and other sources.

Limitations of the study

There are some limitations in this study. First, weaker correlation in tertiary hospitals between RSMRs generated from the two risk models indicated a relatively poorer performance of current MRFP model applied in tertiary hospitals. However, this could be improved if the model development and disparity assessment were conducted within subgroups of hospitals separately. Second, although this study was based on nationally representative cohorts with model development and validation using data from different years, external validations that include more diverse hospitals will be needed in the future.

Conclusion

In conclusion, the MRFP model of in-hospital mortality supported that HQMS data could act as reasonable substitute for complete medical record data in risk adjustment between hospitals across the nation. The lessons from AMI treatment could serve as a model to nationwide assessment on quality of care in other clinical fields.