Validation of chief complaints, medical history, medications, and physician diagnoses structured with an integrated emergency department information system in Japan: the Next Stage ER system.

AIM: Emergency department information systems (EDIS) facilitate free-text data use for clinical research; however, no study has validated whether the Next Stage ER system (NSER), an EDIS used in Japan, accurately translates electronic medical records (EMRs) into structured data.
METHODS: This is a retrospective cohort study using data from the emergency department (ED) of a tertiary care hospital from 2018 to 2019. We used EMRs of 500 random samples from 27,000 ED visits during the study period. Through the NSER system, chief complaints were translated into 231 chief complaint categories based on the Japan Triage and Acuity Scale. Medical history and physician's diagnoses were encoded using the International Classification of Diseases, 10th Revision; medications were encoded as Anatomical Therapeutic Chemical Classification System codes. Two reviewers independently reviewed 20 items (e.g., presence of fever) for each study component (e.g., chief complaints). We calculated association measures of the structured data by the NSER system, using the chart review results as the gold standard.
RESULTS: Sensitivities were very high (>90%) in 17 chief complaints. Positive predictive values were high for 14 chief complaints (≥80%). Negative predictive values were ≥96% for all chief complaints. For medical history and medications, most of the association measures were very high (>90%). For physicians' ED diagnoses, sensitivities were very high (>93%) in 16 diagnoses; specificities and negative predictive values were very high (>97%).
CONCLUSIONS: Chief complaints, medical history, medications, and physician's ED diagnoses in EMRs were well-translated into existing categories or coding by the NSER system.

Introduction

Electronic medical records (EMRs) have been widely used for clinical research in various areas including emergency medicine. , Electronic medical records are mainly comprised of two types of data: structured and unstructured free‐text data. Structured data includes billing diagnoses, prescriptions, and laboratory test results. In particular, claims data from these structured data are the major sources of information for research and policy development. , By contrast, the use of free‐text data has been discouraged due to the clear limitations in scalability and its use being time‐consuming and labor‐intensive.

Over the last decade, the emergence of integrated emergency department information systems (EDISs) has changed emergency department (ED) administration and data management. , Multiple studies have reported the advantages of EDIS, such as improvement of usability, study recruitment, visual representations, and data collection. Several studies have utilized artificial intelligent algorithms to automatically extract structured information from clinical notes in EMRs. , In Japan, the Next Stage ER system (NSER; TXP Medical Co. Ltd, Tokyo, Japan) has been increasingly implemented in EDs of university and tertiary care hospitals. The NSER system supports not only physician decision making, clinician workflow, and communication but also EMR standardization. Through the prespecified forms for each component of medical records (e.g., chief complaints), the NSER system automatically translates recorded free‐text EMRs into structured data regardless of the EMR software. This is a merit for physicians as writing a medical record that is filled in with multiple choices in a structured template is time‐consuming and impractical. In the NSER system, text‐based chief complaints are categorized into 231 categories based on the Japan Triage and Acuity Scale (JTAS); medical histories and physicians’ diagnoses in the ED are encoded using the corresponding International Classification of Diseases, 10th Revision (ICD‐10) codes. Although structured free‐text data can be used for ED‐based clinical research, no study has validated the accuracy of the NSER system translations of EMRs into corresponding categories or codes. As with the claims data, the validity of the accuracy of each data component (e.g., chief complaints) is critical to undertaking high‐quality study.

To address the knowledge gap in published reports, we investigated whether the NSER system accurately translated EMRs into corresponding categories or codes using chart review results as the gold standard. The assessed data components for each case were chief complaints, medical history, medications, and physician’s diagnoses at the ED.

Methods

Study design and setting

This is a retrospective cohort study using data from the ED of the Hitachi General Hospital (Hitachi, Japan) from 1 April 2018 through 31 September 2019. The Hitachi General Hospital is a tertiary care center that covers a population of approximately 3 million with approximately 27,000 annual ED visits. The study protocol was approved by the Ethics Committee of Hitachi General Hospital and they waived informed consent due to the nature of the retrospective design.

Study population

We used EMRs of 500 random samples from 27,000 ED visits during the study period. The sample size of 500 was determined based on previous validation studies and the feasibility of manual chart review.

Structured data with the NSER system

The development of the NSER system and implemented natural language processing algorisms are reported as previously described (see Methods S1). In short, by using the NSER system as the user‐interface on EMR software, practitioners (physician or nurse) use the prespecified fields in the interface to record chief complaints, present illness, medical history, physical assessment, and physician’s ED diagnoses for each patient. For chief complaints, practitioners can choose chief complaints from the predefined list of general chief complaints (e.g. fever, headache). Practitioners can also use the free‐text area when there are no appropriate chief complaints listed. The recorded chief complaints are automatically translated into 231 chief complaints’ categories based on the JTAS, which was developed based on the Canadian Triage and Acuity System. For the medical history and physician’s diagnoses, practitioners record these data to each prespecified field, and the NSER system abstracts the information and applies ICD‐10 codes. For the medication data, the NSER system encodes using the WHO’s Anatomical Therapeutic Chemical Classification System (ATC). The NSER does not need legal authorization from the Pharmaceutical and Medical Devices Agency as a medical device. The NSER system has been developed to assist physicians in writing medical records, structuring the recorded data, and developing a database for clinical research. Therefore, the NSER system does not suggest any potential diagnosis or provide any information for clinical decision making. The developers consulted with the Tokyo Metropolitan Pharmaceutical Affairs Bureau before implementation and confirmed that the NSER system does not need legal authorization.

Data components for the validation

The following data components were chosen for validation: chief complaints, medical history, medications, and physician’s diagnoses. For chief complaints and physician’s diagnoses, we selected the 20 most frequent chief complaints and diagnoses. For medical history, we identified 20 comorbidities based on their frequency and the Charlson comorbidity index or Elixhauser comorbidity index. For medications, the 20 most frequently prescribed drugs (ingredient name) were identified, but we excluded a drug if its effects were similar to other drugs (e.g., nifedipine and amlodipine as a calcium channel blocker). Additionally, any antihypertensives that were noted but unidentified were categorized as “unidentified – antihypertensive” (i.e., patients who were treated with any antihypertensives but the details were unclear). The physician’s diagnoses were defined using ICD‐10 codes (Tables S1 and S2).

Chart reviews

Two reviewers independently reviewed EMRs of 500 random samples. The reviewers were board‐certified emergency physicians who were not involved in the planning of this study nor the development of the NSER system, to maintain the credibility of the results. The reviewers identified whether patients had each item (e.g., fever) in the corresponding prespecified form (e.g., chief complaints) or not. When discrepancies arose, an additional reviewer independently reviewed EMRs and resolved them. The inter‐reviewer agreements in chart reviews between the two main reviewers were evaluated using kappa coefficients and categorized as near‐perfect (0.81–1.00), substantial (0.61–0.80), moderate (0.41–0.60), fair (0.21–0.40), and poor (0.00–0.20).

Statistical analysis

We calculated four association measures including sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of the structured data by NSER, using the results of chart review as the gold standard. As the NSER system facilitates the use of a predefined list for chief complaints, some chief complaints were not provided by physicians as free‐text data. Therefore, we also used manually extracted chief complaints from the present illness as the gold standard (e.g., if all chief complaints were recorded with the use of the predefined list, all association measures should be 100%). This assessment could provide further information on whether the extracted chief complaints are clinically consistent or not. Statistical analyses were carried out using Stata version 14.1 (StataCorp, College Station, TX, USA).

Results

Among the 500 random samples, the median age was 74 (interquartile range, 62–83) years, 229 (45.8%) cases were women, and 283 (56.6%) were transported by ambulance.

Chief complaints

When using the prespecified field for chief complaints, the inter‐reviewer agreements were as follows: 15 were near‐perfect, 3 were substantial, and 2 were moderate (Table 1). The inter‐reviewer agreements were relatively low in ambiguous chief complaints (e.g., general malaise, vertigo). Sensitivities were very high (>90%) except for syncope (64.3%), headache (61.1%), and leg pain (35.7%). The PPVs were high for 14 chief complaints (≥80%) except for low back pain (66.7%), abnormal test results (63.6%), and general malaise (54.5%). The NPVs were ≥96% for all chief complaints.

Table 1

Measures of association between structured chief complaints in the Next Stage ER system and physician‐identified chief complaints using the prespecified chief complaint field (n = 500)

Chief complaint	Frequency	Kappa coefficient	Sensitivity (%)	Specificity (%)	Positive predictive value (%)	Negative predictive value (%)
Fever	61	0.94	96.6	98.9	91.8	99.5
Shortness of breath	53	0.92	86.7	99.8	98.1	98.2
Altered mental status	50	0.88	75.0	98.2	84.0	96.9
Abdominal pain	29	0.92	90.6	100.0	100.0	99.4
Chest pain	24	0.94	77.8	99.4	87.5	98.7
Nausea, vomiting	24	0.87	76.7	99.8	95.8	98.5
General malaise	22	0.59	92.3	97.9	54.5	99.8
Paralysis	18	0.88	93.3	99.2	77.8	99.8
Low back pain	15	0.91	90.9	99.0	66.7	99.8
Headache	14	0.95	61.1	99.4	78.6	98.6
Vertigo	14	0.52	100.0	99.6	85.7	100.0
Arthralgia	12	0.67	100.0	99.6	83.3	100.0
Hemodiarrhea	11	1.00	100.0	99.8	90.9	100.0
Fall	11	0.74	100.0	99.6	81.8	100.0
Syncope	11	0.78	64.3	99.6	81.8	99.0
Abnormal test results	11	0.87	87.5	99.2	63.6	99.8
Chest discomfort	10	0.95	80.0	99.6	80.0	99.6
Rash	9	0.80	81.8	100.0	100.0	99.6
Palpitation	9	1.00	100.0	99.8	88.9	100.0
Leg pain	7	0.93	35.7	99.6	71.4	98.2

When using chief complaints from present illness as the reference, the inter‐reviewer agreements were near‐perfect in six, substantial in eight, moderate in four, and fair or poor in two (Table S3). Sensitivities were high (>80%) for six complaints, moderate (60%–80%) for eight complaints, but fair to poor for headache (45.0%), fall (26.2%), syncope (18.2%), abnormal test results (33.3%), and leg pain (41.7%). Specificities were all ≥96%. The PPVs were high for eight complaints, and fair to poor for general malaise (36.4%) and syncope (40.0%). The NPVs were >93% for all chief complaints.

Medical history

For medical history, the inter‐reviewer agreements for all the most frequent 20 comorbid conditions were near‐perfect (Table 2). The association measures were high. The sensitivity ranged from 70% (chronic obstructive pulmonary disease) to 100%, and the specificity ranged from 99.3% to 100%.

Table 2

Measures of association between structured medical history in the Next Stage ER system and physician‐identified medical history (n = 500)

Past medical history	Frequency	Kappa coefficient	Sensitivity (%)	Specificity (%)	Positive predictive value (%)	Negative predictive value (%)
Hypertension	217	0.97	96.8	99.3	99.1	97.5
Diabetes	121	0.96	97.6	100.0	100.0	99.2
Ischemic stroke	79	0.98	96.3	99.8	98.7	99.3
Dyslipidemia	59	0.97	98.3	100.0	100.0	99.8
Atrial fibrillation	38	0.96	97.0	98.7	84.2	99.8
Chronic renal failure	38	0.83	100.0	99.6	94.7	100.0
Heart failure	31	0.94	100.0	100.0	100.0	100.0
Asthma	31	0.97	87.9	99.6	93.5	99.1
Myocardial infarction	26	0.94	83.3	99.8	96.2	98.9
Cataract	26	0.98	96.3	100.0	100.0	99.8
Dementia	26	0.96	96.2	99.8	96.2	99.8
Appendicitis	22	0.97	95.7	100.0	100.0	99.8
Gastric cancer	20	0.92	90.9	100.0	100.0	99.6
Prostatic hypertrophy	19	1.00	100.0	99.8	94.7	100.0
Angina	18	0.97	89.5	99.8	94.4	99.6
COPD	14	0.95	70.0	100.0	100.0	98.8
Gastric ulcer	12	0.96	100.0	100.0	100.0	100.0
Osteoporosis	11	0.96	83.3	99.8	90.9	99.6
Depression	11	1.00	100.0	100.0	100.0	100.0
Rheumatoid arthritis	9	0.89	100.0	100.0	100.0	100.0

COPD, chronic obstructive pulmonary disease.

Medications

For medications, the inter‐reviewer agreements were near‐perfect in 7, substantial in 11, and fair to moderate in the remaining 2 (Table 3). The primary reason for the fair to moderate kappa coefficients was the differences in the counts of combination tablets. Except for unidentified hypertensive and acetaminophen, the association measures were high.

Table 3

Measures of association between structured medications in the Next Stage ER system and physician‐identified medications (n = 500)

Drug name (generic name)	Frequency	Kappa coefficient	Sensitivity (%)	Specificity (%)	Positive predictive value (%)	Negative predictive value (%)
Amlodipine	73	0.72	94.7	99.8	98.6	99.1
Magnesium	59	0.98	92.1	99.8	98.3	98.9
Aspirin	56	0.78	98.2	99.8	98.2	99.8
Furosemide	45	0.99	100.0	100.0	100.0	100.0
Lansoprazole	41	0.66	90.9	99.8	97.6	99.1
Rosuvastatin	32	0.68	91.2	99.8	96.9	99.4
Rebamipide	32	0.72	96.4	98.9	84.4	99.8
Loxoprofen	31	0.78	96.9	100.0	100.0	99.8
Atorvastatin	30	0.84	100.0	100.0	100.0	100.0
Acetaminophen	29	0.73	95.5	98.3	72.4	99.8
Febuxostat	25	0.72	100.0	99.8	96.0	100.0
Warfarin	24	0.98	96.0	100.0	100.0	99.8
Prednisolone	24	0.87	88.9	100.0	100.0	99.4
Mecobalamin	20	0.82	82.6	99.8	95.0	99.2
Olmesartan	19	0.52	94.7	99.8	94.7	99.8
Clopidogrel	17	0.40	94.1	99.8	94.1	99.8
Pregabalin	17	0.76	94.4	100.0	100.0	99.8
Linagliptin	15	0.61	100.0	100.0	100.0	100.0
Rivaroxaban	14	0.76	100.0	100.0	100.0	100.0
Antihypertensive (unknown)	9	0.94	64.3	100.0	100.0	99.0

Physician’s diagnoses at the ED

For physician’s diagnoses at the ED, the inter‐reviewer agreements were near‐perfect in 16 and substantial in 4 (Table 4). Sensitivities were very high (>93%) in 16 physician’s diagnoses at the ED but were low for influenza (66.7%), sepsis (57.1%), and head trauma (22.2%). Specificities for all physician’s diagnoses at the ED were very high (>97%). The PPVs were high (>80%) in eight diagnoses, moderate in eight diagnoses (60%–80%), and low in four diagnoses including syncope, femoral neck fracture, bruise, and gastric bleeding. The NPVs were all ≥98%.

Table 4

Measures of association between structured diagnoses in the Next Stage ER system and physician‐identified diagnoses in the emergency department (n = 500)

Medical history	Frequency	Kappa coefficient	Sensitivity (%)	Specificity (%)	Positive predictive value (%)	Negative predictive value (%)
Ischemic stroke	26	0.92	100.0	99.4	89.7	99.8
Pneumonia	24	0.98	100.0	97.3	64.9	100.0
Urinary tract infection	18	0.92	94.4	98.8	73.9	99.8
Acute heart failure	15	0.89	93.3	98.4	63.6	99.8
Acute respiratory tract infection	14	0.81	100.0	99.6	87.5	100.0
Benign vertigo	11	0.85	100.0	100.0	100.0	100.0
Syncope	9	1.00	100.0	98.4	52.9	100.0
Head trauma	9	0.85	22.2	100.0	100.0	98.6
Asthma	8	0.77	100.0	98.0	88.9	100.0
Bruise	8	0.66	87.5	98.4	46.7	99.8
Hemorrhagic stroke	7	0.92	100.0	100.0	100.0	100.0
Acute diarrhea	7	0.75	100.0	99.4	70.0	100.0
Hyponatremia	7	1.00	100.0	99.6	77.8	100.0
Sepsis/septic shock	7	1.00	57.1	99.6	66.7	99.4
Influenza	6	0.91	66.7	100.0	100.0	99.6
Gastric bleeding	6	0.83	100.0	98.0	37.5	100.0
Acute myocardial infarction	5	0.77	100.0	99.4	62.5	100.0
Aspiration pneumonia	4	1.00	100.0	99.8	80.0	100.0
Femoral neck fracture	4	1.00	100.0	99.2	50.0	100.0
Urolithiasis	3	0.86	100.0	99.6	60.0	100.0

Discussion

In this validation study using chart review results as the gold standard, chief complaints, medical history, medications, and physician’s ED diagnoses in EMRs were well‐translated into existing categories or coding (JTAS chief complaints category, ICD‐10 codes, and ATC codes) by the NSER system.

In agreement with previous reports on symptom documentation using EDIS, , , the association of the measures of the NSER system to the structured chief complaints was high and had the capability of incorporating new free‐text chief complaints data. Although the importance of chief complaints is well‐known, sparse data exist on the presenting chief complaints of ED patients due to the ambiguity and complexity of medical language. Yet, this free‐text information with EDIS is an important element of ED triage, initial assessment, determination of patient flow, and generation of clinical hypotheses. , , Accordingly, well‐structured data are imperative to developing on‐time decision‐support tools, alerting systems for misdiagnosis, and optimizing resource allocation. Although efforts have been made to create structured EMRs (e.g., using multiple choice for each EMR component), physicians tend to prefer free‐fill text. Therefore, the advantage of the NSER system is that it easily extracts the described data as structured data even when physicians describe EMRs with free text, as they prefer. The observed, acceptable performance of the NSER system, an integrated EDIS, could contribute to future studies on these topics. Although the performance using chief complaints from the present illness was relatively lower than that using a prespecified chief complaint field, this disparity is largely attributable to the quality of physicians’ documentation and the reviewer’s decision.

Similar to chief complaints, structured information on medical history and medications was also important. Medical history and medications have been used in clinical research mainly using administrative data as they are recorded in the claims for each admission and can be obtained from past hospitalizations and prescription data, respectively. Nevertheless, EDs frequently receive new patients (who have not used the department at this hospital before), and their medical histories and current medications are generally not recorded on the claims. Additionally, the accuracy of medical history in the claims data is suboptimal. Thus, structured medical history and medications should facilitate further research, such as studies on polypharmacy, drug interactions, inappropriate management, and acute on chronic conditions in the ED.

One may surmise that diagnoses made by physicians in EDs are similar to those indicated by the ICD‐10 codes for hospitalized patients. Yet limited data are available as to whether the diagnosis made in the ED differs from the final diagnosis at hospitalization or discharge. Additionally, physicians’ diagnoses should reflect patients’ conditions more accurately compared to conditions indicated by claims codes. As one of the ED principles is to reduce the risk of misdiagnosis for patient safety, knowing the physicians’ diagnoses made in the ED is especially important for providing relevant feedback and retrospective assessment. Although the current findings suggest that the structured data with the NSER system have high accuracy, except for certain conditions, the accuracy could be, at least partially, improved by the use of combined procedure and/or prescription codes if available.

Limitations

Our study has several potential limitations. First, the documentation of EMRs depends on each institution and the educational level of the staff using the system; therefore, the findings of this single‐center study might have limited generalizability. Yet, the use of prespecified fields (chief complaints, medical history, medications, and physician’s diagnoses at the ED) could support precise documentation, resulting in the accurate structuring process. Second, the inter‐reviewer agreement varied across EMRs and could require further validation study. Finally, the association measures for medical history and physician’s diagnoses at the ED were affected by the definition (i.e., which ICD‐10 codes were included). However, these codes were widely used and consistent with previous reports.

Conclusions

Chief complaints, medical history, medications, and ED diagnoses in EMRs were well‐translated into existing categories or coding (JTAS chief complaints category, ICD‐10 codes, and ATC codes) by the NSER system. The observed findings could be important as the basis not only for future ED‐based clinical research but also for informing precise initial assessment, decision making, and optimal resource allocation in the ED.

Disclosure

Approval of the research protocol: The study protocol was approved by the Ethics Committee of Hitachi General Hospital.

Informed consent: The Ethics Committee of Hitachi General Hospital waived informed consent due to the nature of the study’s retrospective design.

Registry and the registration no. of the study/trial: N/A.

Animal studies: N/A.

Conflict of interest: Dr. Goto is the Chief Scientific Officer, TXP Medical Co. Ltd. Dr. Sonoo is the Chief Executive Officer, TXP Medical Co. Ltd. Dr. Hara and Dr. Shirakawa are employees of TXP Medical Co. Ltd. None of the authors received any financial incentives for this work from TXP Medical Co. Ltd. TXP Medical may gain potential profit from the manuscript acceptance, such as the use of results to introduce the NSER system.

Table S1. Medical history and corresponding International Classification of Diseases, 10th Revision codes.

Table S2. Emergency department diagnoses and corresponding International Classification of Diseases, 10th Revision codes.

Table S3. Measure of association between structured chief complaints and physician‐identified chief complaints from present illness (n = 500).

Click here for additional data file.

Methods S1. The details of the NSER system.

Click here for additional data file.