Validity assessment of self-reported medication use by comparing to pharmacy insurance claims.

OBJECTIVES: In Japan, an annual health check-up and health promotion guidance programme was established in 2008 in accordance with the Act on Assurance of Medical Care for the Elderly. A self-reported questionnaire on medication use is a required item in this programme and has been used widely, but its validity has not been assessed. The aim of this study was to evaluate the validity of this questionnaire by comparing self-reported usage to pharmacy insurance claims.
SETTING: This is a population-based validation study. Self-reported medication use for hypertension, diabetes and dyslipidaemia is the evaluated measurement. Data on pharmacy insurance claims are used as a reference standard. PARTICIPANTS: Participants were 54,712 beneficiaries of the National Health Insurance of Chiba City. PRIMARY AND SECONDARY OUTCOME MEASURES: Sensitivity, specificity and κ statistics of the self-reported medication-use questionnaire for predicting actual prescriptions during 1 month (that of the check-up) and 3 months (that of the check-up and the previous 2 months) were calculated.
RESULTS: Sensitivity and specificity scores of questionnaire data for predicting insurance claims covering 3 months were, respectively, 92.4% (95% CI 91.9 to 92.8) and 86.4% (95% CI 86.0 to 86.7) for hypertension, 82.6% (95% CI 81.1 to 84.0) and 98.5% (95% CI 98.4 to 98.6) for diabetes, and 86.2% (95% CI 85.5 to 86.8) and 91.0% (95% CI 90.8 to 91.3) for dyslipidaemia. Corresponding κ statistics were 70.9% (95% CI 70.1 to 71.7), 77.1% (95% CI 76.2 to 77.9) and 69.8% (95% CI 68.9 to 70.6). The specificity was significantly higher for questionnaire data covering 3 months compared with data covering 1 month for all 3 conditions.
CONCLUSIONS: Self-reported questionnaire data on medication use had sufficiently high validity for further analyses. Item responses showed close agreement with actual prescriptions, particularly those covering 3 months. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/

We evaluated, for the first time, the validity of a self-reported questionnaire on medication use in the annual health check-up system of Japan by comparing self-reported usage to a log of pharmacy insurance claims, a record that is free of recall bias and regarded as a ‘gold standard’.

A large population-based sample was used.

Specificity might be underestimated due to incomplete data on pharmacy insurance claims.

Introduction

Self-reported questionnaires have been employed to determine drug usage in many epidemiological studies.1–3 However, the accuracy of the information obtained by such questionnaires is limited by recall bias.4–13 A substantial amount of inaccurate data could result in ‘misclassification bias’, leading to incorrect estimates of disease risk and/or prevalence.14 To date, a few studies have evaluated the validity of self-reported medication use but the results have been inconsistent, with some finding high validity4 10 13 and others finding relatively low validity.5 9 11 This inconsistency could result from differences in data collection method, type of drug, age and/or nationality of the target population, and healthcare system.

In Japan, an annual health check-up and health promotion guidance programme was started in April 2008 in accordance with the Act on Assurance of Medical Care for the Elderly by the Ministry of Health, Labour and Welfare (MHLW).15 Medical insurers are obliged to provide this programme to all their beneficiaries aged 40–74 years. During the period from 2010 to 2014, a total of around 112 million people used the programme. The programme mainly targets individuals with metabolic syndrome. A self-reported questionnaire on medication use for hypertension, hyperglycaemia and hypercholesterolaemia, is one of the required items, and the collected data are used to identify individuals in need of further guidance. When a recipient of a health check-up answers ‘yes’ to the question on medication use, he or she is automatically excluded from the target population for the health guidance programme. Consequently, misclassification of medication use can lead to recipients with metabolic syndrome losing the opportunity to receive appropriate guidance. In addition, the questionnaire has been used to detect untreated individuals so that they can be advised to see a doctor when their laboratory data strongly indicate hypertension, diabetes and/or dyslipidaemia. If all health insurance claims were to be computerised and integrated with health check-up data, a self-reported questionnaire would no longer be necessary for public health researchers. In fact, the Japan National Database (NDB) project led by the MHLW was started for this purpose. At the moment, however, the linkage rate between health insurance claims and health check-up data in the NDB is very low,16 meaning that researchers must use self-reported questionnaire data as an alternative. Thus, validation of the data is crucial for practical as well as for research applications.

The aim of this study was to evaluate the validity of the self-reported questionnaire on the use of drugs for hypertension, diabetes and dyslipidaemia that is conducted as part of the annual health check-up in Japan's health guidance programme. To do this, we compared self-reported usage to a log of pharmacy insurance claims, a record that is free of recall bias and regarded as a ‘gold standard’.

Methods

Participants

The participants of this study were beneficiaries of National Health Insurance (NHI) of Chiba City. Japan has a universal healthcare insurance system that covers all citizens.17 There are two types of coverage for individuals younger than 75 years of age, Employees’ Health Insurance and NHI; the latter is managed by municipalities and covers the self-employed, farmers, retirees and the unemployed. The participants in this study consisted of 54 760 beneficiaries aged 40−74 years who received a health check-up from 1 May 2012 to 28 February 2013. Of these individuals, 48 with missing data were excluded, for a final total of 54 712 beneficiaries (22 242 men and 32 470 women). Health check-up data and pharmacy insurance claims data were integrated for comparison using the values of household number, birth month and sex.

Ethics statement

Consent was not obtained from participants because this study was performed using only existing data. To ensure anonymity, personal identifiers (eg, name, address and telephone number) were removed from the records, date of birth was changed to the first of the month, and personal number and household number administered by Chiba City Hall were converted to random numbers prior to release. The study was conducted in accordance with the Declaration of Helsinki.

Definition of self-reported medication users

The self-reported questionnaire, which is required in the health check-up, includes the following item.

Are you currently taking the following medications?

Medication for hypertension (yes or no).

Insulin injection or oral medication for hyperglycaemia (yes or no).

Medication for hypercholesterolaemia (yes or no).

A participant who answers ‘yes’ is defined as a self-reported user of drugs to treat hypertension, diabetes and/or dyslipidaemia.

Definition of true medication users

‘True medication users’ were determined by pharmacy claims submitted from April 2012 to March 2013. Unfortunately, pharmacy insurance claims provided by Chiba City for this study include only prescriptions dispensed outside hospitals. Nonetheless, pharmacy insurance claims are available for 71.4% of all prescriptions filled in Chiba Prefecture during fiscal year 2012.18 Although the pharmacy claim data in this study were not perfect, they were the best data currently available for determining medication users in our study. Thus, we used the obtained pharmacy claim data as a tentative ‘gold standard’. Generic names of medications for the three conditions are listed in online supplementary files 1–3. For detecting appropriate drugs, the Database of Drugs in Japan was used.19 Codes of the Anatomical Therapeutic Chemical Classification System (ATC codes) provided by the WHO20 have not been assigned to every drug used in Japan, but we list as many as possible in the online supplementary files.

Initially, we used two different definitions for the true medication users detected by pharmacy insurance claims: one for participants prescribed during the same month as the health check-up (1 month); and the other for participants prescribed during the same month as the check-up or in the previous 2 months (3 months). In Japan, the law limiting prescriptions to 2 weeks was repealed in April 2002 to allow long-term prescriptions (with some exceptions). Thus, even if the participants did not receive a prescribed medication during a survey month, they might have already received one during the previous month. To overcome this possible omission error, we decided to analyse the month of the check-up and the month of the check-up plus the previous 2 months, separately.

Equivalent household income

Individual annual income from 1 January to 31 December 2011 was obtained from tax records at Chiba City Hall. Number of people per household was obtained by counting the persons with the same household number. People per household included persons insured by NHI in Chiba City and other householders regardless of whether they were beneficiaries. Household income was calculated by summing the incomes of all household members, as aforementioned. An equivalent household income was calculated as household income divided by the square of the number of household members.

Statistical analysis

The proportions of the individual medication users as determined by the self-reported questionnaire and by pharmacy insurance claims were compared by McNemar's test. For assessing the validity of self-reported medication use for hypertension, diabetes and dyslipidaemia, medication use as detected by pharmacy insurance claims was assumed to be accurate (as the gold standard). Sensitivity was calculated as the proportion of participants with self-reported medication use among the participants with pharmacy claims, and specificity was calculated as the proportion of participants without self-reported medication use among the participants without pharmacy claims. In addition, κ statistics were also calculated for each medical condition. The κ statistic measure of agreement is scaled to be 0 when the agreement is what would be expected by chance and to 1 when there is perfect agreement. Landis and Koch21 defined values of 0.00−0.20 as slight, 0.21−0.40 as fair, 0.41−0.60 as moderate, 0.61−0.80 as substantial and 0.81−1.00 as almost perfect agreement. All comparisons were planned and all tests were two tailed. A p value <0.05 was considered statistically significant. All statistical analyses were performed using the STATA13 software package (Stata Corp, College Station, Texas, USA).

Results

Participant demographics, clinical characteristics and medication use, as determined by the self-reported questionnaire and insurance claims, are shown in table 1. Means and SDs of age and body mass index were 65.5±7.8 years and 22.9±3.3 kg/m2, respectively. Median equivalent income was 1 170 000 yen (as of 30 June 2015, US$1 was equivalent to 122.72 yen). The proportions of participants prescribed medications for hypertension, diabetes and dyslipidaemia, as indicated by insurance claims during the check-up month (20.3% for hypertension, 4.0% for diabetes and 14.3% for dyslipidaemia) and within 3 months (25.0% for hypertension, 5.0% for diabetes and 17.9% for dyslipidaemia) were all significantly lower than as indicated from self-reports (33.4% for hypertension, 5.6% for diabetes and 22.8% for dyslipidaemia).

Table 1

Demographic and clinical characteristics, including medication use as determined by the self-reported questionnaire and insurance claims

	Mean (SD)	Median (25th–75th centile)	p Value*
Number	54 712
Sex (men), N (%)†	22 242 (40.7)
Age, years	65.5 (7.8)	68 (63−71)
Body mass index, kg/m2	22.9 (3.3)	22.6 (20.7−27.8)
Equivalent income, 10 000 yen	154 (242)	117 (61−186)
Self-reported medication user, N (%)
Hypertension†	18 246 (33.4)
Diabetes†	3040 (5.6)
Dyslipidaemia†	12 489 (22.8)
Participants prescribed during 1 month, N (%)‡
Hypertension†	11 078 (20.3)		<0.001
Diabetes†	2201 (4.0)		<0.001
Dyslipidaemia†	7835 (14.3)		<0.001
Participants prescribed during 3 months, N (%)§
Hypertension†	13 693 (25.0)		<0.001
Diabetes†	2737 (5.0)		<0.001
Dyslipidaemia†	9817 (17.9)		<0.001

As of 30 June 2015, US$1 was equivalent to 122.72 yen.

*Comparison of the proportions of medication users as determined by insurance claims versus those from the self-reported questionnaire, using McNemar's test.

†Number (%) shown.

‡Participants prescribed drugs during the same month as the health check-up according to insurance claims.

§Participants prescribed drugs during the month of check-up or the previous 2 months according to insurance claims.

Table 2 presents the results of the validity analysis of self-reported medication use in all participants. In general, the self-reported questionnaire predicted actual prescriptions (ie, according to pharmacy insurance claims) with high sensitivity and specificity for the month of check-up and for 3 months. Specificity was uniformly higher for predicting prescriptions within 3 months for all three drug classes. The κ values were also higher for predicting prescriptions within 3 months compared with 1 month. Thus, the self-reported questionnaire more accurately represented medication use for 3 months than for 1 month, the only exception being sensitivity for hypertension medication use (p=0.001). Only those results for predicting actual prescriptions over 3 months were used for subgroup analyses (table 3).

Table 2

Validity of self-reported medication use

	Predicting actual prescriptions during 1 month*			Predicting actual prescriptions during 3 months†
	Hypertension	Diabetes	Dyslipidaemia	Hypertension	p Value‡	Diabetes	p Value‡	Dyslipidaemia	p Value‡
True-positive, N	10 356	1802	6795	12 652		2261		8458
True-negative, N	35 744	51 273	41 183	35 425		51 196		40 864
False-positive, N	7890	1238	5694	5594		779		4031
False-negative, N	722	399	1040	1041		476		1359
Sensitivity (95% CI), %	93.5 (93.0 to 93.9)	81.9 (80.2 to 83.5)	86.7(86.0 to 87.5)	92.4 (91.9 to 92.8)	0.001	82.6 (81.1 to 84.0)	0.500	86.2 (85.5 to 86.8)	0.273
Specificity (95% CI), %	81.9 (81.6 to 82.3)	97.6 (97.5 to 97.8)	87.9 (87.6 to 88.2)	86.4 (86.0 to 86.7)	<0.001	98.5 (98.4 to 98.6)	<0.001	91.0 (90.8 to 91.3)	<0.001
κ Statistic (95% CI), %	60.7 (59.9 to 61.5)	67.2 (66.4 to 68.1)	59.8 (59.0 to 60.6)	70.9 (70.1 to 71.7)	–	77.1 (76.2 to 77.9)	–	69.8 (68.9 to 70.6)	–

*Check-up month.

†Check-up month and previous 2 months.

‡One month versus 3 months (χ2 test).

Table 3

Validity of self-reported medication, among subgroups, for predicting actual prescriptions during 3 months

	Sex		Age		Income*
	Men	Women	40−64 years	65–74 years	<Median	≥Median
Hypertension
True-positive, N	5952	6700	2472	10 180	6491	6161
True-negative, N	13 180	22 245	13 375	22 050	17 467	17 958
False-positive, N	2652	2942	1082	4512	2857	2737
False-negative, N	458	583	240	801	541	500
Sensitivity (95% CI), %	92.9 (92.2 to 93.5)	92.0 (91.3 to 92.6)	91.2 (90.0 to 92.2)	92.7 (92.2 to 93.2)	92.3 (91.7 to 92.9)	92.5 (91.8 to 93.1)
Specificity (95%CI), %	83.2 (82.7 to 83.8)	88.3 (87.9 to 88.7)	92.5 (92.1 to 92.9)	83.0 (82.6 to 83.5)	85.9 (85.5 to 86.4)	86.8 (86.3 to 87.2)
κ Statistic (95%CI), %	69.1 (67.8 to 70.4)	72.0 (71.0 to 73.1)	74.3 (72.8 to 75.8)	68.9 (67.9 to 69.9)	70.6 (69.5 to 71.8)	71.2 (70.0 to 72.3)
Diabetes
True-positive, N	1324	937	462	1799	1141	1120
True-negative, N	20 168	31 028	16 465	34 731	25 553	25 643
False-positive, N	458	321	153	626	406	373
False-negative, N	292	184	89	387	256	220
Sensitivity (95% CI), %	81.9 (80.0 to 83.8)	83.6 (81.3 to 85.7)	83.8 (80.5 to 86.8)	82.3 (80.6 to 83.9)	81.7 (79.5 to 83.7)	83.6 (81.5 to 85.5)
Specificity (95%CI), %	97.8 (97.6 to 98.0)	99.0 (98.9 to 99.1)	99.1 (98.9 to 99.2)	98.2 (98.1 to 98.4)	98.4 (98.3 to 98.6)	98.6 (98.4 to 98.7)
κ Statistic (95%CI), %	76.1 (74.8 to 77.4)	78.0 (76.9 to 79.1)	78.5 (77.0 to 80.0)	76.6 (75.6 to 77.6)	76.2 (75.1 to 77.4)	77.9 (76.7 to 79.1)
Dyslipidaemia
True-positive, N	2644	5814	1651	6807	4391	4067
True-negative, N	17 736	23 128	14 407	26 457	20 161	20 703
False-positive, N	1290	2741	781	3250	2083	1948
False-negative, N	572	787	330	1029	721	638
Sensitivity (95% CI), %	82.2 (80.8 to 83.5)	88.1 (87.3 to 88.8)	83.3 (81.6 to 85.0)	86.9 (86.1 to 87.6)	85.9 (84.9 to 86.8)	86.4 (85.4 to 87.4)
Specificity (95%CI), %	93.2 (92.9 to 93.6)	89.4 (89.0 to 89.8)	94.9 (94.5 to 95.2)	89.1 (88.7 to 89.4)	90.6 (90.2 to 91.0)	91.4 (91.0 to 91.8)
κ Statistic (95%CI), %	69.0 (67.7 to 70.3)	69.8 (68.7 to 70.9)	71.2 (69.7 to 72.6)	68.8 (67.8 to 69.8)	69.4 (68.2 to 70.6)	70.1 (68.9 to 71.3)

*Median equivalent income was 1 170 000 yen (as of 30 June 2015, US$1 was equivalent to 122.72 yen).

Analyses of subgroups divided by sex, age range and income are shown in table 3. In all subgroups, sensitivity and specificity were >80% and κ statistic >60%. Thus, the validity of the self-report questionnaire was high regardless of sex, age and income.

Discussion

The self-reported medication-use questionnaire for the annual health check-up programme overseen by the Japanese MHLW was found to have high validity for predicting actual prescriptions for drugs used to treat hypertension, diabetes and dyslipidaemia.

Although pharmacy insurance claims data inherently include comprehensive prescription information, making the data suitable as a ‘gold standard’, only prescriptions dispensed outside hospitals were available in this study, which accounted for 71.4% of all prescriptions in fiscal year 2012.18 Indeed, the proportions of participants with actual pharmacy claims were significantly lower than the proportions based on self-reports for all three conditions. Accordingly, we should consider how this drawback influences the accuracy of the sensitivity and specificity values calculated. For calculation of sensitivity, we determined the proportion of participants self-reporting use only among those with external (outside of hospital) prescriptions. However, if the responses to the self-reported questionnaire by participants with external prescriptions and those with in-hospital prescriptions are assumed to be the same, the sensitivity should be close to the true figure. On the other hand, for calculation of specificity, we used the number of participants without actual prescription claims as the denominator, which includes those actually prescribed in-hospital, making the value lower than the true data. Despite this influence, however, values of specificity and sensitivity were satisfactory (>80%) for all three diseases.

To date, only two studies, to our knowledge, have conducted a validity assessment of self-reported medication use for hypertension, diabetes and/or dyslipidaemia. One assessed self-reported drug use for hypertension and diabetes in 17 191 participants of different ethnicities in British Columbia, Canada.10 This study found high specificity for hypertension (99−100% in each ethnicity) and diabetes (99−100% in each ethnicity), but relatively low sensitivity for hypertension (60−76% in each ethnicity). Here, the two-step methodology may have influenced the result. In the first step, only those participants who answered ‘yes’ to ‘Do you have this condition?’ were extracted. Then, the selected participants were asked, ‘In the past 12 months, have you taken any medicine for this condition?’. It is known that this two-step method increases specificity but decreases sensitivity.14 In contrast, the self-reported questionnaire item for medication use analysed in the present study asks participants a single question for each condition (‘Are you currently taking the following medications?’), and high validity was observed, indicating that this mode of questioning is appropriate and reliable.

The second study, from Washington State, reported high validity for self-reported medication use for hypertension and dyslipidaemia (statins) in 403 participants of a population-based, case–control study of breast cancer in women aged 65−79 years.4 Sensitivity in cases and controls were 92% and 92% for hypertensive medication, and 83% and 98% for statins, respectively. Specificity was 91% and 93% for hypertensive medication, and 98% and 98% for statins.4 This result is quite similar to ours.

In contrast to medication for hypertension, diabetes and dyslipidaemia, these studies reported lower sensitivity of self-reported medication use for asthma (32−52% in each ethnicity10) and depression (64% and 66% in cases and controls4), suggesting that the validity of self-reported medication use depends on the specific medical condition. So et al10 suggested that the difference in self-report validity among medication types is likely related to the frequency of use. For example, self-report validity for medications used to treat acute symptoms, such as asthma, tends to be lower than that for medications taken routinely for chronic conditions such as hypertension, diabetes and dyslipidaemia. Further studies are needed to confirm whether this is also the case for the health check-up self-report questionnaire.

We found that validity of the self-reported questionnaire was higher (more accurately reflected medication use) during 3 months than during 1 month. Patients in Japan can now obtain longer term prescriptions (after the law limiting prescriptions to 2 weeks was repealed). Thus, we assumed that many patients currently taking medications for hypertension, diabetes and/or dyslipidaemia would have prescriptions filled in previous months before the annual check-up and simultaneous completion of the self-reported questionnaire. In general, prescriptions for chronic diseases are renewed, at most, every 3 months,22 so there should be relatively few respondents, with no recent claims over this period, answering ‘yes’. Even so, to control for the influence of longer term prescriptions, we evaluated the validity for predicting actual prescriptions over 10 months in 20 529 participants who received the health check-up from 1 December 2012 to 28 February 2013 (data not shown). The κ (95% CI) values for hypertension, diabetes and dyslipidaemia medication use were 72.2% (70.9% to 73.6%), 78.4% (77.0% to 79.7%) and 70.5% (69.2% to 71.9%), respectively; similar to or even slightly higher than the values for 3 months. Thus, we suggest that the self-reported questionnaire is better for predicting medium-term and long-term medication use than for predicting short-term use.

We found high validity of the self-reported questionnaire in this study, indicating that healthcare workers and public health researchers can both use these data for practical and research purposes. For example, these data can be used in research toward the NDB's goal of replacing self-reported data once integration of data between annual health check-ups and insurance claims becomes reliable.

Conclusion

We found that the self-reported questionnaire of medication use for hypertension, diabetes and dyslipidaemia that is conducted as part of the annual health check-up in Japan's health guidance programme is a valid measure of true medication use. Accuracy appears better for predicting prescriptions filled within 3 months compared with those filled within 1 month.