Validation of Acute Myocardial Infarction and Heart Failure Diagnoses in Hospitalized Patients With the Nationwide Claim-Based JROAD-DPC Database.

Background: Big data systems such as diagnosis procedure combination (DPC) datasets have recently been used for research purposes. However, there have been few validation studies to determine the accuracy of diagnoses. The aim of this study was to validate and evaluate 2 diagnoses, namely acute myocardial infarction (AMI) and heart failure (HF), using International Classification of Diseases, 10th revision (ICD-10) codes in the Japanese Registry Of All cardiac and vascular Disease (JROAD)-DPC database. Methods and 
Results: ICD-10 codes I21.0-I21.9 and I50.0-I50.9 were used to identify AMI and HF, respectively, in the JROAD-DPC database. Diagnoses of AMI and HF were validated in clinical datasets assessing sensitivity and positive predictive value (PPV). Over 1-2 years, 742 patients hospitalized for AMI and 1,368 patients hospitalized for HF were identified in the DPC dataset. Sensitivity and PPV for AMI were 78.9% and 78.8%, respectively. When emergency hospitalization was included as a criterion, PPV increased to 84.9%. For HF, sensitivity and PPV were 84.7% and 57.0%, respectively. When emergency hospitalization and acute HF were included as criteria, PPV increased to 83.0%. Conclusions: Using ICD-10 codes for AMI and HF diagnoses among hospitalized patients, the DPC dataset showed acceptable concordance with clinical datasets. PPV increased when any conditions of hospitalization were included, especially in HF.

A lump-sum payment system based on diagnosis procedure combinations (DPCs) was introduced in acute care hospitals throughout Japan in 2003. In fact, the Ministry of Health, Labour and Welfare (MHLW) and its affiliated research institution have started research on the feasibility of using a casemix classification system as a tool for standardizing medical profile and payment data. The use of a DPC dataset for clinical research can provide enormous amounts of longitudinal data at low cost so that researchers can conduct studies effectively. This DPC system was started in 82 Japanese hospitals with specific functions. By 2018, the system had expanded to 1,730 hospitals, covering approximately 83% of acute care hospitals in Japan.,

The Japanese Registry Of All cardiac and vascular Disease (JROAD) was launched in 2004 to assess the clinical activity of each Japanese institution with respect to cardiovascular beds and to provide adequate feedback to teaching hospitals to improve patient care. JROAD only includes institution-level information, not individual patient data. Therefore, in 2014, in collaboration with the National Cerebral and Cardiovascular Center (NCVC), the Japanese Circulation Society (JCS) started developing the JROAD-DPC database, which includes a unique hospital identifier, age, sex, main diagnosis, comorbidities, length of stay, in-hospital medications, and discharge status. This dataset extracts only records from the MHLW’s DPC dataset that contain cardiovascular diseases in major diagnosis categories. The JROAD-DPC database was designed to used DPC resources to analyze, interpret, and advance the quality of medical care. The JROAD/JROAD-DPC database has been available to JCS members for research purposes.– Each research topic is reviewed by the JCS IT and Database committees for approval.

Although several research studies have been conducted to validate the accuracy of the DPC dataset with medical records in Japan,,– no validation studies have been conducted to date for the JROAD-DPC database. The aims of this study were to validate 2 major diagnoses, namely acute myocardial infarction (AMI) and heart failure (HF), using International Classification of Diseases, 10th revision (ICD-10) codes along with other information, and to identify diagnoses of AMI and HF in the JROAD-DPC database.

Methods

Study Population

The details of the DPC dataset have been described elsewhere.,, In the DPC dataset, all procedures and prescriptions performed during hospitalization are recorded according to the Japanese fee schedule for reimbursement. In addition, patient information in the DPC dataset can only be linked when the patient is admitted in the same hospital because different identifiers are assigned to patients by different hospitals within the database. Furthermore, the DPC dataset includes neither the cause of death nor laboratory data.

We conducted a validation study with 5 institutions (Kumamoto University Hospital, Nara Medical University Hospital, NCVC Hospital, Yokohama City University Medical Center Hospital, and Sapporo Medical University Hospital). All participating hospitals were certified as advanced medical centers by the MHLW of Japan. The period of DPC data collection for each hospital was 1 year, either between April 1, 2012 and March 31, 2013 or between April 1, 2013 and March 31, 2014, depending on the availability of the DPC data. The period of DPC data collection for AMI from the NCVC was 2 years.

This study was approved by the Ethics Committee of the NCVC (Authorization no. M23-051-14).

Diagnostic Criteria

Diagnoses were classified using ICD-10 codes I21.0, I21.1, I21.2, I21.3, I21.4, and I21.9 for AMI and I50.0, I50.1, and I50.9 for HF. Four categories of diagnoses can be recorded in the DPC system: main diagnosis, admission-precipitating diagnosis, most resource-consuming diagnosis, and second most resource-consuming diagnosis. There are 2 additional diagnostic categories for comorbidities: conditions present at the time of admission and conditions arising after admission. We identified records for AMI and HF from DPC data when the relevant ICD-10 codes appeared in the main diagnosis, admission-precipitating diagnosis, or most resource-consuming diagnosis categories.

Cardiovascular specialists in each hospital performed a retrospective chart review to extract their institution’s records for AMI and HF for its clinical dataset. Criteria for AMI consisted of the following 3 components: (1) elevation of cardiac troponins in peripheral blood; (2) electrocardiographic findings such as ST elevation or ST depression; and (3) symptoms such as ischemic chest pain, dyspnea, nausea, unexplained weakness, or a combination of these symptoms. Criteria for HF included the following information in a medical record: clinical syndrome consisting of dyspnea, malaise, swelling, or decreased exercise capacity due to the loss of compensation for cardiac pump function secondary to structural or functional abnormalities of the heart. A comparison of records from the JROAD-DPC database and the clinical dataset revealed that some clinical records were missing. Therefore, we asked cardiovascular specialists in each hospital to recheck the records. This revealed that these records could not be found because they had been from another department. Consequently, their diagnoses were confirmed.

Diagnostic Variables and Conditions

In order to improve validation accuracy, we combined ICD-10 codes with information from examinations, diagnostic imaging, injections, medications, and therapeutic procedures for fee calculations in the DPC dataset (). We defined hospitalized patients who underwent surgery, intervention, or inpatient drug treatment using specific DPC variables. We also determined emergency hospitalization other than planned or scheduled hospitalization using a specific DPC variable. We added diagnostic or therapeutic procedures for hospitalized patients such as percutaneous coronary intervention (PCI), electrocardiography (ECG), echocardiography (UCG), or B-type natriuretic peptide (BNP) measurement. An emergency examination was defined as any examination performed within 48 h of hospitalization. We also evaluated having a troponin or creatine kinase measurement more than once during hospitalization in combination with ICD-10 codes (twice). We defined UCG examinations performed within 1 week of admission as early examinations. In HF, we included subanalysis stratified by New York Heart Association (NYHA) functional classification. Furthermore, acute HF in the JROAD-DPC dataset was defined as the following 2 conditions: (1) an ICD-10 code of most resource-consuming diagnosis of I50; and (2) additional disease codes of 30101 or 30102 (acute exacerbation of acute/chronic HF) attached in the record.

Figure.

Diagnostic criteria for (A) acute myocardial infarction (AMI) and (B) heart failure (HF) in the diagnosis procedure combination (DPC) dataset. Acute HF is defined as 2 conditions: an International Classification of Diseases, 10th Revision (ICD-10) code for the most resource-consuming diagnosis of I50 and additional disease codes of 30101 or 30102 attached in the record. BNP, B-type natriuretic peptide; CK, creatine kinase; CK-MB, creatine kinase-myocardial band; ECG, electrocardiography; NT-proBNP, N-terminal pro BNP; PCI, percutaneous coronary intervention; Tn, myocardial troponin.

Statistical Analysis

The frequencies of diagnoses and other variables were assessed with clinical data and DPC data. Sensitivity and positive predictive value (PPV) of the DPC data were calculated, with the clinical data from chart reviews considered to be the gold standard. Sensitivity was defined as the proportion of patients with a diagnosis based on chart reviews that was correctly identified as having the same diagnosis based on DPC data. PPV was defined as the proportion of patients with a diagnosis based on DPC data that had the same diagnosis based on chart reviews. In addition, 95% confidence intervals (CIs) were calculated using CI for a proportion in 1 sample formula. Data cleaning and analysis were performed using SAS version 9.4 (SAS Institute, Cary, NC, USA).

Results

In the DPC dataset, 742 patients were identified as having AMI and 1,386 patients were identified as having HF based on ICD-10 codes. Among patients with AMI, the mean (±SD) age was 69.2±12.5 years and 75.0% were male. Among patients with HF, the mean (±SD) age was 73.6±12.5 years and 62.7% were male. During the same period, the clinical dataset based on chart review identified 741 patients with AMI and 933 patients with HF. The sensitivity and PPV for AMI diagnosed based on ICD-10 codes only were 78.9% and 78.8%, respectively (). The original number of AMI patients before comparing records from the JROAD-DPC database with records from the clinical dataset was 612. The sensitivity and PPV of the original dataset for AMI diagnosed based on ICD-10 codes only were 74.5% and 61.5%, respectively. In all, 157 records were classified as false positive. Most recorded comorbidities were angina or HF. It is possible to diagnosis these diseases as AMI. In addition, 156 records were classified as false negative. Diagnoses in false-negative records were similar diseases to AMI, such as unstable angina or recent myocardial infarction (MI) or old MI. The ICD-10 codes for these diseases are I20 or I24 or I25. When patients with angina pectoris, unspecified (ICD-10 code I21.9) were removed, the sensitivity and PPV for AMI diagnosed based on ICD-10 codes only were 68.0% and 81.2%, respectively. When emergency ECG examination was combined with ICD-10 codes, sensitivity increased to 96.4% and PPV decreased to 5.0%. Sensitivity of PCI plus ICD-10 codes was similar to sensitivity with ICD-10 codes only, but PPV decreased to 69.9%. When repeated troponin or creatine kinase measurements at the time of hospitalization were added to ICD-10 codes, sensitivity and PPV remained similar. When hospitalization for treatment only or emergency hospitalization only was considered along with ICD-10 codes, PPV was slightly increased compared with PPV for ICD-10 codes only.

Table 1.

Validity Indices for the Diagnosis Procedure Combination Data-Based Diagnosis Identification of Acute Myocardial Infarction

Diagnostic criteria	Sensitivity (%) (95% CI)	PPV (%) (95% CI)
ICD-10 code only	78.9 (78.3–79.6)	78.8 (78.2–79.5)
ICD-10 code or UCG (emergency)	96.4 (96.1–96.6)	5.0 (4.7–5.4)
ICD-10 code or Tn/CK/CK-MB (twice)	78.9 (78.3–79.6)	78.8 (78.2–79.5)
ICD-10 code or PCI	79.8 (79.2–80.4)	69.9 (69.2–70.5)
ICD-10 code or UCG (emergency) and PCI	79.5 (78.9–80.1)	71.9 (71.2–72.6)
ICD-10 code + hospitalization for treatment only	78.5 (77.9–79.2)	81.7 (81.2–82.3)
ICD-10 code + emergency hospitalization only	75.3 (74.7–76.0)	84.9 (84.4–85.5)

CI, confidence interval; CK, creatine kinase; CK-MB, creatine kinase myocardial band; ICD-10, International Classification of Diseases, 10th revision; PCI, percutaneous coronary intervention; PPV, positive predictive value; Tn, myocardial troponin; UCG, echocardiography.

Sensitivity and PPV for HF diagnosed based on ICD-10 codes only were 84.7% and 57.0%, respectively (). The original number of HF patients before comparing records from the JROAD-DPC database with records from the clinical dataset was 754. The sensitivity and PPV of the original dataset for HF diagnosed based on ICD-10 codes only were 81.0% and 44.1%, respectively. In all, 596 records were classified as false positive. Most recorded comorbidities were atrial fibrillation, angina, or chronic renal failure. These conditions have similar symptoms and physical findings as HF, which may cause suspicious diagnoses. Further, 143 records were classified as false negative. The diagnoses in false negative records were mainly underlying diseases, such as cardiomyopathy, valvular disease, acute coronary syndrome, and arrhythmia. When patients with HF, unspecified (ICD-10 code I50.9) were removed, the sensitivity and PPV for HF diagnosed based on ICD-10 codes only were 81.5% and 57.4%, respectively. When early UCG was combined with ICD-10 codes, sensitivity increased to 93.0% and PPV decreased to 15%. Similarly, when emergency BNP/N-terminal pro BNP (NT-proBNP) measurements were combined with ICD-10 codes, sensitivity increased to 93.0% and PPV decreased to 14.6%. Sensitivity and PPV remained similar after hospitalization for treatment was combined with ICD-10 codes (83.2% and 58.2%, respectively). When emergency hospitalization was combined with ICD-10 codes, sensitivity decreased to 66.5% and PPV increased to 69.6%. When considering acute HF, PPV increased to 77.7%. In addition, when emergency hospitalization was combined with acute HF, PPV increased to 83.0%. Sensitivity for comorbidities ranged from 43.8% to 71.1% and PPV ranged from 69.7% to 87.7%. shows sensitivity by NYHA functional classification. Under most conditions, sensitivity remained similar or decreased slightly. Emergency hospitalization and ICD-10 codes increased sensitivity in patients with NYHA Class IV disease to 72.7%, compared with 65.9% in patients with NYHA Class ≥II disease.

Table 2.

Validity Indices for the Diagnosis Procedure Combination Data-Based Diagnosis Identification of HF

Diagnostic criteria	Sensitivity (%) (95% CI)	PPV (%) (95% CI)
ICD-10 code only	84.7 (84.0–85.3)	57.0 (56.1–57.9)
ICD-10 code or UCG (early)	93.0 (92.6–93.5)	14.6 (14.0–15.3)
ICD-10 code or BNP/NT-proBNP (emergency)	93.1 (92.7–93.6)	12.7 (12.1–13.3)
ICD-10 code or UCG (early) and BNP/NT-proBNP (emergency)	91.4 (90.9–91.9)	20.6 (19.9–21.4)
ICD-10 code + hospitalization for treatment only	83.2 (82.5–83.8)	58.2 (57.3–59.0)
ICD-10 code + emergency hospitalization only	66.5 (65.6–67.3)	69.6 (68.8–70.4)
Acute HFA	45.7 (44.8–46.5)	77.7 (77.0–78.5)
Acute HFA + emergency hospitalization only	33.4 (32.6–34.3)	83.0 (82.3–83.6)

AAcute heart failure (HF) is defined as 2 conditions: an ICD-10 code for the most resource-consuming diagnosis of I50 and additional disease codes of 30101 or 30102 attached in the record. BNP, B-type natriuretic peptide; NT-proBNP, N-terminal pro BNP. Other abbreviations as in Table 1.

Table 3.

Validity Indices for the Diagnosis Procedure Combination Data-Based Diagnosis Identification of HF Stratified by NYHA Classification

Diagnostic criteria	Sensitivity (%) (95% CI)
	NYHA ≥II	NYHA ≥III	NYHA IV
ICD-10 code only	84.2 (83.6–84.9)	82.8 (82.1–83.5)	77.6 (76.9–78.4)
ICD-10 code or UCG (early)	92.8 (92.4–93.3)	93.1 (92.6–93.5)	92.7 (92.2–93.2)
ICD-10 code or BNP/NT-proBNP (emergency)	92.9 (92.5–93.4)	93.2 (92.8–93.7)	92.2 (91.8–92.7)
ICD-10 code or UCG (early) and BNP/NT-proBNP (emergency)	91.2 (90.7–91.7)	91.1 (90.6–91.6)	89.6 (89.1–90.2)
ICD-10 code + hospitalization for treatment only	82.9 (82.2–83.6)	82.1 (81.4–82.8)	77.6 (76.9–78.4)
ICD-10 code + emergency hospitalization only	65.9 (65.1–66.7)	68.8 (68.0–69.6)	72.7 (71.9–73.5)

NYHA, New York Heart Association. Other abbreviations as in Tables 1,2.

At each hospital, sensitivity for the diagnosis of AMI based on ICD-10 codes only ranged from 74.3% to 91.9% and PPV ranged from 76.0% to 88.3% (). For the diagnosis of HF, sensitivity ranged from 55.3% to 95.8%, and PPV ranged from 45.2% to 67.0%. There was no consistency between diagnostic performance for AMI and HF in each hospital.

Discussion

We performed a validation study of diagnoses in the DPC dataset using hospital-based clinical chart data from 5 hospitals. Using ICD-10 codes only, the sensitivity and PPV for AMI were 78.9% and 78.8%, respectively, and the sensitivity and PPV for HF were 84.7% and 57.0%, respectively. When emergency hospitalization was added, PPV for AMI and HF increased to 84.9% and 69.6%, respectively. In HF, when emergency hospitalization was combined with acute HF, PPV increased further to 83.0%.

Not many validation studies for AMI and HF have been conducted in Japan., A recent single-center study found that PPV for AMI was 82.5%. That study also showed that PPV was even higher (93.8%) when restricted to patients with an AMI code in DPC claims, particularly when the condition was recorded as 1 of the 3 major types of “Class B condition codes”, which defines the same condition as in the present study. However, that study was limited by a single-center design. In the present study, significant variations in diagnostic accuracy across facilities were observed, suggesting that validation in multiple facilities is necessary. In another study, Yamana et al validated the diagnosis of MI and congestive HF in 4 mid-sized acute care hospitals that implemented the DPC-based payment system and Standardized Structured Medical Information eXchange (SS-MIX) storage. Yamana et al reported that sensitivity and PPV for MI were 52.2% and 92.3%, respectively. Even though the PPV was higher than in the present study, sensitivity was lower. The values of sensitivity and PPV were balanced in the present study. In the study of Yamana et al, sensitivity and PPV for congestive HF were 68.8% and 75.9%, respectively. Sensitivity was lower and PPV was higher than in our study. The reason for these differences is not known, but it may be associated with the small number of the patients with MI and congestive HF in the study of Yamana et al (23 and 32, respectively). In addition, Yamana et al only focused on the ICD-10 code I50.0, whereas we used codes I50.0, I50.1, and I50.9 for HF. This difference may have decreased sensitivity and increased PPV. Furthermore, several research groups have performed validation studies in the emergency department setting using administrative datasets.– However, they have not studied the diagnosis of AMI or HF.

ICD-10 codes are widely accepted and their use in DPC claims is reported to be useful for clinical research. To further improve the accuracy of diagnoses in the DPC dataset, it is essential to narrow down the study population by including clinical characteristics. For example, in previous JROAD studies, in addition to ICD-10 code, conditions such as patient age <20 years or discharge on the day of hospitalization were used to exclude patients from the analysis dataset to reduce analysis bias for patients without AMI or HF.– In the present study, although combining ICD-10 codes with other information did not improve either sensitivity or PPV, PPV increased without decreasing sensitivity when hospitalization for treatment was included for AMI. The PPV for ICD-10 codes only in HF was inferior to that in AMI. This may be explained by the following: the diagnosis of HF is essentially based on non-specific symptoms and is difficult in clinical situations. Elderly patients are often misdiagnosed as having HF. In contrast, elderly patients with HF have many comorbidities, and the diagnosis of HF may be overlooked. However, by focusing on emergency hospitalization with acute HF diagnosis, we were able to increase the PPV to 83.0%. Further studies are needed to further improve the accuracy of diagnoses in the DPC dataset.

This study had several limitations. First, inter-rater agreement could not be assessed because chart review was conducted by a single cardiovascular specialist at each hospital. Second, because our original study design was just to extract the case group only, records of true negatives were not available in the present study. Thus, we were not able to calculate specificity or negative predictive value. Finally, the present study was performed based on data obtained from 5 different hospitals. However, the period of data extraction was different, which may have affected the results. In addition, because these hospitals were not general hospitals, the generalizability of the present findings may be limited.

Conclusions

In conclusion, we evaluated the validity of using condition codes to identify AMI and HF diagnoses in the JROAD-DPC dataset. ICD-10 codes showed acceptable concordance between DPC and clinical data for the diagnosis of AMI or HF among hospitalized patients. PPV increased when information was added regarding hospitalization, especially for HF.

Sources of Funding

This research was supported by an Intramural Research Fund(27-4-7) and AMED under Grant Number JP19ek0210089.

Disclosures

K. Tsujita, Y.M., and S.Y. are members of Circulation Reports’ Editorial Team. The remaining authors have no conflicts of interest to declare.

IRB Information

This study was approved by the Ethics Committee of the NCVC (Authorization no. M23-051-14).

Supplementary Table. Validity indices for the DPC Data-Based Diagnosis Identification Stratified by the Hospital