Validity of Self-reported Stroke and Myocardial Infarction in Korea: The Health Examinees (HEXA) Study.

OBJECTIVES: Self-reported disease history is often used in epidemiological studies. In this study, we acquired the hospital records of subjects who self-reported stroke or myocardial infarction (MI) and evaluated the validity of the participants' self-reported disease history. We also determined the level of agreement between specialists and non-specialists.
METHODS: Among the participants in the Health Examinees study, 1488 subjects self-reported stroke or MI during 2012-2017, and medical records were acquired for the 429 subjects (28.8%) who agreed to share their medical information. Each record was independently assigned to 2 medical doctors for review. The records were classified as 'definite,' 'possible,' or 'not' stroke or MI. If the doctors did not agree, a third doctor made the final decision. The positive predictive value (PPV) of self-reporting was calculated with the doctors' review as the gold standard. Kappa statistics were used to compare the results between general doctors and neurologists or cardiologists.
RESULTS: Medical records from 208 patients with self-reported stroke and 221 patients with self-reported MI were reviewed. The PPV of self-reported disease history was 51.4% for stroke and 32.6% for MI. If cases classified as 'possible' were counted as positive diagnoses, the PPV was 59.1% for stroke and 33.5% for MI. Kappa statistics showed moderate levels of agreement between specialists and non-specialists for both stroke and MI.
CONCLUSIONS: The validity of self-reported disease was lower than expected, especially in those who reported having been diagnosed with MI. Proper consideration is needed when using these self-reported data in further studies.

INTRODUCTION

Self-reported disease history is often used in large-scale epidemiological studies, especially if no other data with higher validity are available. Although self-reported data can be useful, assessments of the reliability of self-reported data are inconsistent across studies, making researchers reluctant to utilize self-reported data sources. According to a study conducted in USA [1] assessing the validity of self-reported diagnoses in the first wave of the National Health and Nutritional Examination Survey Epidemiologic Follow-up Study, the positive predictive values (PPVs) for stroke and myocardial infarction (MI) were 67% and 83%, respectively. In contrast, lower validity was reported in a study conducted by Spanish researchers that investigated the Spanish cohorts of the European Prospective Investigation into Cancer and Nutrition study [2], which reported PPVs of 22.2% and 60.7% for stroke and MI, respectively. Moreover, a validation study for questionnaires answered by nurses [3] reported PPVs of 79% and 68% for cerebrovascular accidents and MI, which was not ideal considering the profession of the participants; this finding underscores the need for studies to validate each individual cohort. Several other studies from various countries showed PPVs for stroke and MI in the ranges of 29-79% and 43-81%, respectively.

In this study, we acquired hospital records from participants in a large-scale cohort study, the Health Examinees (HEXA) study, who self-reported stroke or MI and evaluated the validity of those participants’ self-reported disease history. In addition, we determined the level of agreement between specialists and non-specialists who reviewed the same medical records, in order to establish whether a specialist should review medical records for certain types of diseases.

METHODS

The HEXA study, one of the Korean Genome and Epidemiology Studies conducted by the Korean Centers for Disease Control and Prevention and known as KOGES_HEXA, is a population-based cohort study that aims to investigate the associations between epidemiological characteristics and various diseases, including stroke and MI [4]. A total of 173 357 participants were prospectively recruited between 2004 and 2013 at 38 health examination centers and training hospitals located in 8 different regions in Korea using a standardized study protocol [5]. Follow-up of the participants was started in 2007, comprising self-reports of being diagnosed with various diseases and changes in risk factors such as lifestyle habits. The validation of self-reported diseases through medical record review targeted participants who were followed during the period of 2012-2017. Details of the main objectives, rationale, study design, and baseline characteristics of the HEXA study have been published elsewhere [4].

To validate self-reported cases of disease, participants’ hospital records were reviewed and compared. For that purpose, written consent and delegation for acquiring hospital records, as well as a photocopy of an identification card, were collected from those who agreed to participate in this study. Six hired agents were assigned to make contact with participants and to acquire their hospital records, and were dispatched to different administrative regions throughout the country.

Among the followed participants in the HEXA cohort, 1488 subjects self-reported having been diagnosed with stroke or MI. We successfully made contact with 1338 of those subjects (89.9%) and asked whether each subject reconfirmed his or her prior self-report of disease and was willing to participate in this study. The numbers of subjects who reported no disease were 51 (11.8%) and 575 (54.5%) for stroke and MI, respectively. In total, 440 subjects reconfirmed their self-report, of whom 429 (28.8%) agreed to share their medical information. Those participants’ hospital records were acquired from the medical facility where the diagnosis was made.

Every acquired record was independently assigned to 2 medical doctors for review, and after that process, each record was labeled as ‘definite,’ ‘probable’, or ‘not’ stroke or MI. Only the records with specific test results indicating stroke or MI were labeled as ‘definite,’ such as patients with an abrupt onset of unilateral motor impairment with confirmatory magnetic resonance imaging results or patients with typically elevated ST segments on electrocardiography with elevated cardiac biomarkers. However, if the descriptions of the diagnostic tests did not clearly support the diagnosis, the reviewers looked for other definite evidence, such as hospital discharge records with a stroke or MI diagnosis and a referral for rehabilitation treatment. Medical records without definite evidence of the diagnosis of stroke or MI that nonetheless contained reasonable evidence that the patients might have been previously diagnosed with those conditions were labeled as ‘probable’; this category included patients who were transferred from another hospital after diagnosis and had several records of National Institutes of Health Stroke Scale scores or a history of rehabilitation. Those without any sign of either disease were classified as ‘not’ having stroke or MI. When the 2 doctors did not agree, a third doctor made the final judgement. Two PPVs were calculated: one only counted ‘definite’ disease cases as positive, and the other both included ‘probable’ and ‘definite’ disease cases as positive.

To derive proper guidelines for whether a specialist should review the medical records of participants in this cohort, the level of agreement between different doctors regarding the same medical records was observed. Thirty randomly selected cases of stroke and MI were given to a neurologist and a cardiologist, respectively, currently working at Seoul National University Hospital. Cohen’s kappa (κ) and the sensitivity and specificity were calculated, with the results from the neurologist and cardiologist used as the gold standard.

Ethics Statement

This study was approved by the Institutional Review Board of Seoul National University Hospital (C-0608-018-179).

RESULTS

The baseline characteristics of the participants in this study are shown in Table 1. Out of the 429 medical records that were reviewed, 57.2% and 69.7% were from male participants who self-reported stroke or MI, respectively. Most of the records (97.9% and 96.6%) were obtained from those who were 50 years old or older at the time of follow-up, but no participant was more than 80 years old. The participants were predominantly diagnosed in hospitals located in metropolitan areas such as Seoul, Busan, and Daegu.

Table 1.

Characteristics of participants who self-reported stroke or MI

Characteristics	Stroke (n = 208)	MI (n = 221)
Sex
Male	119 (57.2)	154 (69.7)
Female	89 (42.8)	67 (30.3)
Age (y)
Mean±SD	64.4±7.0	64.3±7.1
40-49	3 (2.1)	6 (3.4)
50-59	41 (28.5)	39 (22.0)
60-69	64 (44.4)	90 (50.9)
70-79	36 (25.0)	42 (23.7)
Location of hospital of diagnosis
Seoul	34 (16.4)	38 (17.2)
Incheon	13 (6.3)	7 (3.2)
Daejeon	0 (0.0)	1 (0.5)
Daegu	36 (17.3)	23 (10.4)
Busan	36 (17.3)	57 (25.8)
Ulsan	8 (3.9)	13 (5.9)
Gwangju	16 (7.7)	20 (9.1)
Gyeonggi Province	6 (2.9)	12 (5.4)
Chungnam Province	16 (7.7)	9 (4.1)
Gyeongnam Province	13 (6.3)	21 (9.5)
Jeonnam Province	1 (0.5)	0 (0.0)
Jeonbuk Province	3 (1.4)	2 (0.9)
Gangwon Province	26 (12.5)	18 (8.1)

Values are presented as number (%).

MI, myocardial infarction; SD, standard deviation.

Table 2 shows the results of the medical record review by 2 different medical doctors regarding self-reported stroke and MI, and the agreement between them is represented as the Cohen’s κ statistic. Of the 208 self-reported stroke cases and 221 MI cases, 154 (92 definite stroke, 4 probable stroke, and 58 not stroke) and 194 review results (57 definite MI, 1 probable MI, and 135 not MI) showed agreement between the 2 doctors. A total of 53 stroke and 28 MI records showed disagreement between the different doctors, and another medical doctor made the final judgement. The agreement between doctors, calculated as κ (95% confidence interval [CI]), was 0.54 (95% CI, 0.45 to 0.64) for stroke and 0.74 (95% CI, 0.66 to 0.82) for MI, which are moderate and substantial levels of agreement, respectively [6].

Table 2.

Agreement between 2 medical doctors when reviewing self-reported stroke and MI

Review results of doctor 2	Stroke (n = 208)			MI (n = 221)
	Review results of doctor 1			Review results of doctor 1
	Definite	Probable	Not stroke	Definite	Probable	Not MI
Definite	92 (44.2)	11 (5.3)	13 (6.3)	57 (25.8)	7 (3.2)	4 (1.8)
Probable	4 (1.9)	4 (1.9)	3 (1.4)	6 (2.7)	1 (0.5)	3 (1.4)
Not stroke	11 (5.3)	12 (5.8)	58 (27.9)	2 (0.9)	6 (2.7)	135 (61.1)
κ (95% CI)		0.54 (0.45, 0.64)			0.74 (0.66 ,0.82)

Values are presented as number (%).

MI, myocardial infarction; CI, confidence interval.

Figure 1 shows the results of the medical record review of self-reported stroke and MI. Out of the 208 self-reported stroke cases, 107 were determined to be definite stroke, 16 to be probable stroke, and 85 to be not stroke. Out of the 221 self-reported MI cases, 72 were classified as definite MI, 2 as probable MI, and 147 as not MI.

Figure. 1.

Results of medical record review of self-reported stroke and myocardial infarction (MI).

As shown in Table 3, the PPVs (95% CI) of self-reported stroke were 59.1% (95% CI, 52.3 to 65.6) and 51.4% (95% CI, 44.7 to 58.1) with and without the inclusion of probable strokes, respectively. The PPVs (95% CI) of self-reported MI were 33.5% (95% CI, 27.6 to 39.9) and 32.6% (95% CI, 26.7 to 39.0) with and without the inclusion of probable MI, respectively. The PPVs of males were slightly higher than those of females, but were not statistically significantly different. The PPVs for stroke and MI showed contrasting outcomes in different age groups, as the PPV for stroke decreased as age at follow-up increased, while the PPV for MI increased. When we stratified the results by the region of the hospital of diagnosis, no noticeable patterns emerged. Gyeongsang and Jeolla Provinces showed the highest PPVs for stroke, and Gangwon and Chungcheong Provinces showed the highest PPVs for MI.

Table 3.

Results of medical record review of self-reported stroke and MI by sex, age group, and region of hospital of diagnosis

Characteristics	Stroke								MI
	Confirmed/self-report[1]	PPV (%)[1]	95% CI		Confirmed/self-report[2]	PPV (%)[2]	95% CI		Confirmed/self-report[1]	PPV (%)[1]	95% CI		Confirmed/self-report[2]	PPV (%)[2]	95% CI
			LL	UL			LL	UL			LL	UL			LL	UL
Overall	107/208	51.4	44.7	58.1	123/208	59.1	52.3	65.6	72/221	32.6	26.7	39.0	74/221	33.5	27.6	39.9
Sex
Male	66/119	55.5	46.5	64.1	75/119	63.0	54.1	71.2	51/154	33.1	26.2	40.9	51/154	33.1	26.2	40.9
Female	41/89	46.1	36.1	56.4	48/89	53.9	43.6	63.9	21/67	31.3	21.5	43.2	23/67	34.3	24.1	46.3
Age (y)
<60	61/108	56.5	47.1	65.4	68/108	63.0	53.6	71.5	12/45	26.7	16.0	41.0	12/45	26.7	16.0	41.0
60-69	35/64	54.7	42.6	66.3	40/64	62.5	50.3	73.3	25/90	27.8	19.6	37.8	27/90	30.0	21.5	40.1
70-79	11/36	30.6	18.0	46.9	15/36	41.7	27.1	57.8	16/42	38.1	25.0	53.2	16/42	38.1	25.0	53.2
Region
Seoul, Gyeonggi	22/40	55.0	39.8	69.3	24/40	60.0	44.6	73.7	16/50	32.0	20.8	45.8	16/50	32.0	25.0	53.2
Metropolitan area[3]	71/143	49.7	41.6	57.7	82/143	57.3	49.1	65.2	53/159	33.3	26.5	41.0	55/159	34.6	27.6	42.3
Province
Chungcheong	8/16	50.0	28.0	72.0	8/16	50.0	28.0	72.0	4/9	44.4	18.9	73.3	4/9	44.4	18.9	73.3
Gyeongsang	9/13	69.2	42.4	87.3	10/13	76.9	49.7	91.8	8/21	38.1	20.8	59.1	8/21	38.1	20.8	59.1
Jeolla	4/4	100.0	25.0	84.2	4/4	100.0	25.0	84.2	0/2	0.0	0.0	65.8	0/2	0.0	0.0	65.8
Gangwon	13/26	50.0	32.1	67.9	17/26	65.4	46.2	80.6	10/18	55.6	33.7	75.4	10/18	55.6	33.7	75.4

MI, myocardial infarction; PPV, positive predictive value; CI, confidence interval.

Only ‘definite’ was considered as disease.

Both ‘definite’ and ‘probable’ were considered as disease.

The category ‘metropolitan area’ includes Seoul, Incheon, Daejeon, Daegu, Busan, Ulsan, and Gwangju.

Table 4 shows the agreement between different reviewers regarding 30 randomly selected medical records for each disease. For this process, we asked the doctors only to differentiate ‘disease’ from ‘not disease.’ Therefore, the ‘confirmed’ records would be equivalent to ‘definite’ only. The κ statistics (95% CI) for stroke and MI were 0.49 (95% CI, 0.18 to 0.81) and 0.46 (95% CI, 0.15 to 0.78), which both could be interpreted as moderate levels of agreement, while the sensitivity and specificity were both in the range of 71-78% [6].

Table 4.

Agreement of medical record review results between specialists and general doctors

Variables	Self-reported disease
	Stroke (n=30)		MI (n=30)
	Reviewed by general doctors		Reviewed by general doctors
	Confirmed	Not confirmed	Confirmed	Not confirmed
Reviewed by specialist, n (%)[1]
Confirmed	16 (53.3)	5 (16.7)	12 (40.0)	4 (13.3)
Not confirmed	2 (6.7)	7 (23.3)	4 (13.3)	10 (33.3)
κ (95% CI)	0.49 (0.18, 0.81)		0.46 (0.15, 0.78)
Sensitivity, % (95% CI)[2]	76.2 (52.8, 91.8)		75.0 (47.6, 92.7)
Specificity, % (95% CI)[2]	77.8 (40.0, 97.2)		71.4 (41.9, 91.6)

MI, myocardial infarction; CI, confidence interval.

Specialist: neurologist for stroke and cardiologist for MI.

Sensitivity and specificity were calculated using the results of the specialist as the gold standard.

DISCUSSION

We calculated PPVs to assess the validity of self-reported stroke and MI in questionnaires for the follow-up study of the HEXA cohort. The PPVs of our study participants were 51-59% for self-reported stroke and 33-34% for self-reported MI. For stroke, our result falls in the range reported in similar studies with large study populations previously performed in Europe and the USA [1,7-14], and was significantly higher than the PPVs reported in studies done in Spain and New Zealand [2,15]. However, the PPV for MI in our study was far lower than those in other similar studies [1-3,7,8,10,12,13,16-19]. The only study that showed a similar PPV for MI (43%) was a study conducted in Japan among the Japan Public Health Center-based (JPHC) prospective study cohort.

An important fact worth mentioning that could explain the low PPV for MI is that the follow-up questionnaire participants in HEXA between 2012 and 2016 were asked about either MI or angina. Because MI was our only outcome of interest in this study, we asked the participants whether they had been diagnosed with MI and only included those who reconfirmed the diagnosis. This resulted in a large proportion of subjects who reported no MI, despite previously self-reporting MI on follow-up. However, we included participants who were not sure whether they were diagnosed with MI or angina when re-contact was made as candidates for medical record review. This possibility may be supported by the fact that a large proportion of false-positive cases turned out to be angina. Moreover, a few of the acquired hospital records were not related to the diseases of interest. Those records were therefore classified as not confirming the disease, resulting in a lower PPV; this may have taken place because study participants cannot always precisely recall the date and place of their first diagnosis.

Nonetheless, the higher PPV observed for stroke than for MI could have reflected the distinct characteristics of these diseases, as stroke is typically accompanied by very noticeable and memorable symptoms such as motor impairment or prolonged dysphasia, while MI and other cardiovascular diseases share symptoms such as pain or discomfort in the chest. Moreover, the follow-up questionnaire for stroke only asked about stroke, and no irrelevant cases were included in the candidates for medical record review.

The finding of higher PPVs in males than in females in our study is consistent with the findings of other studies from the USA and Japan [14,19], despite its statistical insignificance. The observation that the PPV for stroke was inversely related with age at follow-up is also in accordance with the findings reported for the JPHC cohort; this trend could be the result of cognitive decline or cortical dysfunction in elderly individuals [14]. However, a previous study performed in the Netherlands among only individuals aged 85 and over showed one of the highest PPVs for stroke (78%) of the various studies that have investigated this issue, even though the total number of subjects was small (n=9) [17].

In other words, only half of the patients correctly gave their stroke diagnosis on questionnaires during follow-up and in regard to MI, only one-third of the responses were true in our study, regardless of whether the results included possible disease. Such a high rate of false positives for patients’ outcomes in a cohort not only distorts the descriptive statistics in this population, but also poses the risk of greatly misleading subsequent studies that utilize this dataset. Therefore, self-reported disease history in HEXA should be supported by linkage to other reliable data, such as National Health Insurance claims data, and should not be used alone.

In terms of the agreement between different reviewers and between general doctors and specialists, the calculated κ values were 0.49 and 0.46 for cases of stroke and MI, both showing moderate levels of agreement. The characteristics of the 30 cases each used for the comparisons between specialists and general doctors were not statistically significant different from those of the overall subjects. Although it is undoubtedly best for a specialist, such as a neurologist, to review hospital records for relevant conditions, doing so is not always possible, nor is it always the most efficient approach. According to our results, only 2 and 4 cases of stroke and MI, respectively, were classified by specialists as ‘not disease’ despite being classified as ‘disease’ by non-specialist doctors, whereas it was more common for specialists to reclassify cases that were initially categorized as ‘not disease’ by non-specialist doctors. This discrepancy implies that only the negative results of medical record reviews performed by non-specialists need revision by a specialist. Furthermore, to improve the accuracy and consistency of the identification of disease, a standardized protocol for acquiring and reviewing medical records is recommended, including which subcategory to review in a record and which diagnostic criteria to use.

The importance of this study lies in the fact that it is the first study to investigate the validity of self-reported stroke and MI in the Korean population, as the validity of self-reporting differs widely according cultural characteristics, such as language. Moreover, we provided meaningful results regarding the agreement between doctors of different specialties who reviewed the same medical records and the possibility of implementing more efficient and better processes when reviewing medical records in future studies.

A few limitations of this study should be noted. First, not all the hospital records were acquired from patients who self-reported stroke or MI during follow-up, making it impossible to measure the sensitivity or specificity of self-reporting, such that the results were restricted to PPVs. Second, information on important factors that may affect the validity of self-reported diseases, such as socioeconomic status or education level, was not available, leaving unresolved questions about which variables may be correlated with validity. In addition, only 429 cases (28.8%) were reviewed for validity out of 1488 self-reported cases. It was frequently the case either that it was not possible to contact the subject or that the subject was not willing to participate and refused to provide consent for the acquisition of his or her medical records. The limited number of cases reviewed could have resulted in selection bias because the characteristics of the cases that were reviewed or not reviewed could have been different. Nevertheless, we could not address these possible differences because we had limited information on the study participants who refused to provide consent for medical record review.

In conclusion, the PPVs of self-reported stroke and MI were 59.1% and 33.5%, respectively, which were lower than expected, and inter-reviewer agreement was moderate to substantial. Proper review of hospital data is needed to validate self-reported disease history prior to utilizing it in further studies.