Validity of self-reported healthcare utilization data in the Community Health Survey in Korea.

To evaluate the sensitivity and specificity of Community Health Survey (CHS), we analyzed data from 11,217 participants aged ≥ 19 yr, in 13 cities and counties in 2008. Three healthcare utilization indices (admission, outpatient visits, dental visits) as comparative variables and the insurance benefit claim data of the Health Insurance Review & Assessment Service as the gold-standard were used. The sensitivities of admission, outpatient visits, and dental visits in CHS were 54.8%, 52.1%, and 61.0%, respectively. The specificities were 96.4%, 85.6%, and 82.7%, respectively. This is the first study to evaluate the validity of nationwide health statistics resulting from questionnaire surveys and shows that CHS needs a lot of efforts to reflect the true health status, health behavior, and healthcare utilization of the population.

INTRODUCTION

Identifying the current status of health indices sets the foundation for a series of processes in health programs, including selection of program priorities, the establishment of purposes and goals, and the development of strategy, program implementation, and evaluation. Efforts to identify the health status of the nation were initiated with the 'National Health Survey' in 1962 and the 'National Nutrition Survey' in 1969. Currently the 5th 'Korean National Health and Nutrition Examination Survey (KNHANES)' is in progress. As KNHANES considers only 16 wide-area self-governing districts, this nationwide sampling design has failed to cover the actual current status of 253 basic regional self-governing districts in which health programs are executed (1).

To overcome this limitation and to create health indices that are comparable among regions, the Community Health Survey (CHS) has been conducted annually since 2008 in individuals aged ≥ 19 yr in 253 regions around the country (2). The CHS has a two-stage sampling process. The first sampling stage is the process of selecting a sample area (tong/ban/ri) that is a primary sample unit, selected according to the number of households in dong/eup/myeon (the smallest administrative unit) using a probability proportional to the sampling method. In the second sampling stage, as a process of selecting sample households, the number of households in the selected sample tong/ban/ri is identified to create a household directory, and sample households are selected through systematic sampling methods. These methods are used to ensure the sample units are representative.

To evaluate reliability of CHS, a third organization conducts a telephone survey in 5% of the respondents 2 weeks after the interview. This identified how well the responses match. In fact, the kappa values were, on average, 0.68 (0.51 to 0.90) in 2008 and averaged 0.80 (0.70 to 0.90) in 2009 (3, 4).

However, the validity of the CHS has not yet been assessed. We evaluated the accuracy of self-reported CHS obtained by analysis of both sensitivity and specificity for some healthcare utilization.

MATERIAL AND METHODS

Study subjects

The CHS of Chungbuk-do was performed on 12,449 participants aged ≥ 19 yr, in 13 cities and counties in 2008. This study used data of 11,217 participants, excluding 1,206 participants who did not allow the use of their data for national statistics connected with other organizations (e.g., health insurance, death data) and 26 who agreed but provided incorrect residency registration numbers. The CHS of Chungbuk-do was conducted from September 1st to November 30th, 2008.

Selection of questionnaire items for validity evaluation

The questionnaire items regarding healthcare utilization for the validity test have to have sufficient responses in CHS and gold-standard tests to be objectively compared. This study selected the following three questions; 1) Have you been hospitalized in the past year? Yes/No (admission), 2) Have you been examined as an outpatient in the past 2 weeks? Yes/No (outpatient visit), 3) Have you visited a dental clinic in the past year? Yes/No (dental visit). The actual survey question used to assess the dental visits was; when was the last time you visited the dentist? ① within the last 6 months, ② over 6 months-within the last year, ③ over 1 yr-within the last 3 yr, ④ over 3 yr, ⑤ no visit, and ⑥ unknown/cannot remember. The response based on 'within the last year or not' in this study was changed by categorizing the answers into two Yes/No answers, those who had visited in the last year (①, ②) and those who had not (③-⑤).

Gold-standard for validity tests

The insurance benefit claim data of the Health Insurance Review & Assessment Service (HIRA) was used as the gold-standard for the validity test of the three questions investigated in this study.

Merging of the two data sets

Resident registration numbers offered by respondents were used to merge the CHS data and the HIRA's insurance benefit claim data. The two data sets were merged by HIRA and a new identification number was assigned to the merged data. Resident registration numbers were deleted to protect the private information of the participants.

Data fabrication

The period of the respondent's healthcare utilization in the HIRA claim was within the last year for admission and dental visits, and within the last 2 weeks for outpatient visits, based on the survey date of the CHS. The respondent's healthcare utilization for admission or outpatient visits was defined by admission or outpatient visit cases that were claimed by all healthcare institutions (including dental clinics) for insurance benefits to the HIRA during the assigned period. In the case of dental visits, it was defined with the cases (admission or outpatient visits) that were claimed by only dental healthcare institutions.

Validity tests

The sensitivity of each question is the rate of those who used a healthcare organization among those whose data from the HIRA identifies their healthcare utilization in the CHS. The specificity of each question is the rate of those who claimed not to use a healthcare organization among those whose data from the HIRA identifies their non-use of healthcare in the CHS.

Data analysis

The frequency of healthcare utilization was initially calculated from the two data sets. The sensitivity and specificity of the three questions were then assessed. Additionally, the sensitivity and specificity of the detailed variables were assessed to test the significance between the sensitivity and specificity of each variable. A P value < 0.05 (two tailed test) was considered to indicate statistical significance.

Ethics statement

This study was reviewed and approved by the institutional review boards of Konkuk University Hospital (Approval number: KUH1230005). We received informed consents from all of the participants in the interview survey.

RESULTS

Respondent's healthcare utilization in HIRA and CHS

The rates of admission and outpatient visits were 13.4% and 36.6% in HIRA and 10.4% and 28.2% in CHS, respectively. The rate of dental visits was 44.7% from HIRA and 30.6% from CHS (Table 1).

Table 1

Healthcare utilization of study population in HIRA and CHS

Difference of value is due to missing, for example 11 for admission, 16 for outpatient visit, and 679 for dental service in total. HIRA, Health Insurance Review & Assessment Service; CHS, community health survey.

Sensitivity of healthcare utilization

The sensitivities of admission, outpatient visits, and dental visits in CHS were 54.8%, 52.1%, and 61.0%, respectively (Table 2). No significant difference was evident from the sensitivity of admissions according to gender or marital status. But the values were significantly lower for those participants aged in their 20s (38.1%) and 30s (45.3%), those with 10-12 yr (50.3%) and more than 13 yr (45.3%) of education, and those who were health insurance holders (54.0%; P < 0.01). The sensitivity for outpatient visits was significantly lower for males (47.8%), those aged in their 20s (30.0%), 30s (44.0%), 40s (46.1%), and 50s (46.7%), those who were single (35.7%) and married (50.6%), those with 10-12 (42.4%) and 13 yr (37.0%) of education, and those who were health insurance holders (51.2%; P < 0.001). The sensitivity for dental visits displayed no significant difference by gender, and health insurance type, but was significantly lower for those aged in their 20s (51.6%), and 70s (50.6%), those who were single (53.5%), and those who possessed less than 6 yr of education (54.4%; P < 0.01).

Table 2

Sensitivity for self-reported of healthcare utilization

Specificity of healthcare utilization

The specificities of admission, outpatient visits, and dental visits in CHS were 96.4%, 85.6%, and 82.7%, respectively (Table 3). The specificity of admissions displayed no significant difference by gender, but showed the highest values in those who were aged in their 20s (98.2%), were single (98.0%), those who were educated more than 13 yr (97.8%), and who were health insurance holders (96.6%; P < 0.01). The specificity of the outpatient visits showed the highest values in males (87.7%), those aged in their 20s (93.7%), those who were single (93.4%), those who had more than 13 yr of education (91.8%), and those who were health insurance holders (86.1%; P < 0.001). The specificity of admissions displayed no significant difference by gender, but showed the highest values in those who were aged in their 70s (88.3%), were married but not having spouse (86.5%), those who were educated less than 6 yr (86.5%), and those who were public health aid holders (87.0%; P < 0.05).

Table 3

Specificity for self-reported of healthcare utilization

DISCUSSION

The sensitivities for admission, outpatient visits, and dental visit were considerably lower than the specificities. Additionally, the sensitivity of admissions and outpatient visits were similar, despite the different utilization periods. Maybe, it seems likely that the respondents feel pressured to provide direct information regarding their medical care use, and not because they failed to remember. Although the sensitivities showed significant differences according to gender, age, marital status, and education in individual question, there were no variables with consistent pattern on sensitivity.

However, the sensitivity of dental visits was higher than those of admission and outpatient visits and the specificity was lower than those of admission and outpatient visits. The reason for the higher sensitivity appears that the respondents remembered the dental visits as a more specialized event than admission or outpatient visits. This may be also due to the methods of obtaining the answers, which differed, with the admission and outpatient answers being only yes/no, versus the dental visits, which had six options for the visit period. On the contrary, reasons for the lower specificity appear to be the dental clinic covers more non-benefit items than other general medical areas, meaning the dental clinic cannot claim insurance benefits for all their treatments. In addition, the respondents can use all healthcare institutions with their Industrial Accident Compensation Insurance, car-insurance, and non-insurance so this could lead to a higher false positive value for outpatient visits and dental visits.

A questionnaire survey is one of the data sources used to identify health behavior, health status, and healthcare utilization in a population, but its accuracy is known to be relatively low, when compared to either medical record review or insurance claim data (5-7). The accuracy of self-reported data in healthcare utilization depends, among other factors, on the subjects' knowledge and understanding of the relevant information, their ability to recall information, and their willingness to report it (8).

The range of sensitivity was 86-96% and the range of specificity was 77%-89% with a 6-48 month follow-up period in previous prospective studies that investigated the validity of data on self-reported dental visits in the Florida Dental Care Study (9). Among elderly aged ≥ 70 yr, the kappa value of hospital episodes and physician visits was 0.767 and 0.255, respectively, within the past year (10). Using the data from the 2001 Taiwan National Health Interview and the National Health Insurance, the crude kappa values for inpatients (within the past year), outpatients (within the past month), and dental visits (within the past month) were 0.75, 0.64, and 0.62, respectively (11). These values are higher than those reported in this study and may be explained by the discrepancies between the study design, questionnaire content, and the method of results presentation. In fact, the kappa values on admission, outpatients visits and dental visits in this study were 0.57, 0.40 and 0.44, respectively, although agreement rate of those were 90.9%, 73.4%, and 76.1% respectively.

Consequently, based on these study results, increasing the sensitivity of the respondents requires effort in the process of CHS. First, the recall time can be subdivided into periods of 1 month, 2-6 months, and 7 months to 1 yr for admissions, and 1 week, 2 weeks, and 3 4 weeks for outpatient visits, and reply methods can be changed from healthcare institute utilization (Yes/No) to the utilization period, like those used for the dental visits. Indeed, it has been reported that an optimal recall period for self-reported surveys occurs at 6 months or less (6). Second, when healthcare utilization questions are placed at the end of a long survey, the participant may want to finish the survey and not invest the time and effort needed for an accurate recall (12). Third, applying the correct probing methods for the respondents is required when obtaining information from the past to the present. For example, it has been reported that a 6-2 month approach, where the past 6-month utilization questions are asked initially, is more accurate than a 2-6 month approach (13).

The study has some limitations. First, is it right to use the insurance benefit claim data of the HIRA as the gold standard for validity test? Although NHI did not cover all healthcare utilization in Korea, NHI is a social insurance whereby all Korean take out a policy for medical security as a form of health insurance (96.7%) or medical aid (3.3%). In addition, 89.2% of an insured person visited the healthcare institution as admission or outpatient more than once in 2009 (14). That is, the gap which is about 10.8% included some people who did not use the healthcare institution, some people who used the healthcare institution by other insurance such as life-insurance, car-insurance, Industrial Accident Compensation Insurance, etc or some people who used the healthcare institution is not covered by any insurance. But we did not know exact amount of coverage of other insurance or non-insurance. Anyway, considering the amount of visit to the healthcare institution covered by NHI and access to data which we used as gold standard, we thought that using the insurance benefit claim data of the HIRA is the proper approach. Second, why did we use only 3 items, admission, outpatient visit, and dental visit for validity evaluation of CHS? Why did not we use more detailed healthcare utilization question such as outpatient visit by hypertension, admission by stroke, etc. In fact, we can evaluate the validity of specific utilization of CHS and compare CHS with HIRA about the accuracy of diagnosis. But they are the topics of the next study. In this study, we just want to evaluate the validity of CHS irrespective of specific disease, from a conservative point of view. Third, this study was a validity test conducted only for one broad area (a self-governing district); as a result, we did not analyze the regional difference of sensitivities and specificities and the study results should not be generalized regarding the validity of the entire CHS. This limitation could be overcome with validity testing for all respondents of the 2008 CHS (about 220,000 participants). In fact, the sensitivities and specificities of admission, outpatient visits and dental visits among 13 cities and counties of Chungbuk-do were very diverse. These were as follows: range of sensitivity and specificity of admission was 38.9%-67.7% and 94.1%-97.5%, range of sensitivity and specificity of outpatient visits was 31.2%-66.8% and 78.0%-92.0%, and range of sensitivity and specificity of admission was 39.2%-74.5% and 75.6%-88.0%. It needs the further task which contains a multilevel study to explain the regional variation of CHS validity.

In conclusion, despite some limitations, the present study is the first to evaluate the validity of nationwide health statistics resulting from questionnaire surveys and shows that CHS needs a lot of efforts to reflect the true health status, health behavior, and healthcare utilization of the population.