Is E-Version Transition of the Medication Adherence Scale Feasible for CKD Management? A Pilot Study.

BACKGROUND: To transfer a paper-version Chinese and Western medication adherence scale for CKD into an electronic scale, and evaluate its validity, internal consistency and clinical implementation, and assess whether the transition is feasible in clinic.
METHODS: We built an e-version Chinese and Western medication adherence scale based on the Wen-JuanXing platform. CKD subjects' responses were applied to test the scale's validity and internal consistency. We retested some of the participants two weeks later randomly. We also tested the clinical application.
RESULTS: Of the 434 recruited patients, 228 responded. In exploratory factor analysis (EFA), the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy = 0.8 and Bartlett's approx. Chi-Square = 1340.0 (df = 105, p < 0.001). We extracted four common factors which could explain 61.47% of the variance. However, Item 15 "Have you changed a traditional Chinese medicine prescription yourself within the past month?" had factor loading = 0.3 and measure of sampling adequacy (MSA) = 0.5, meaning we could not enter it into the factor analysis. The internal consistency reliability for medication adherence was 0.9, with a Guttman split-half coefficient = 0.5 and a Spearman-Brown coefficient = 0.6. Cronbach's α was 0.9, 0.4 and 0.5 for the knowledge, belief and behavior domains, respectively. The correlation coefficient r of the test-retest reliability was -0.8 and was -0.8, 0.4, -0.3 in the knowledge, belief and behavior domains, respectively. Patients with comorbidities were more likely to respond. We detected no other significant differences in the clinical profiles between respondents and non-respondents.
CONCLUSION: The e-version Chinese and Western medication adherence scales have undesirable construct validity and internal consistency. Thus, caution is needed in transitioning the paper-version scale into an e-version.

Introduction

Chronic kidney disease (CKD) affects 9.1% of adults worldwide and 10.8% of adults in China.1,2 Costs related to CKD and end-stage renal disease (the terminal manifestation of CKD) exert an enormous burden on both individuals and healthcare systems, making it a growing public health problem worldwide. Furthermore, its chronic, progressive and non-communicable characteristics are usually associated with adverse metabolic effects, anemia, comorbidities and psychological problems, such as anxiety disorders.3 Medical treatment is generally lifelong, and ideal medication adherence, defined as “the extent to which the patient’s behaviour matches agreed recommendation from the prescriber”,4 is a crucial factor in thwarting CKD progression. A systematic review and meta-analysis reveal that patients with advanced CKD, misconceptions about medication, lack of perceived self-efficacy in medication use, polypharmacy, loss of confidence in the physician, poor social support and lower education levels perform lower medication adherence.5 Complex disease characteristics, frequent follow-up, various dosage forms, pre-existing conditions, side effects, financial disparities, and Chinese medicine prescriptions (eg, oral solutions, decoctions, as well as their taste and temperature) present new challenges to medication adherence in CKD.

As such, reliable medication adherence assessment tools are needed, including direct measures (eg, drug assays of blood and/or urine), indirect measures (eg, pill counting, electronic monitoring and measures based on big data) and patient-reported outcome measures (PROMs) (eg, the Morisky, Green and Levine scale (MGL scale), Hill-Bone Compliance Scale, and Medication Adherence Rating Scale (MARS)).6 PROMs are used by many researchers due to their advantages over other methods, including simplicity, cost-effectiveness, applicability across settings, and the ability to provide immediate feedback at the point of care and detect potential factors influencing adherence.7 However, PROMs may be sub-optimal in evaluating herbal medication and CKD-specific medication adherence. Additionally, they are limited to a single aspect of behavior or cognitive modification.

Hence, we developed a paper-version Chinese and Western medication adherence scale for CKD (version-1) in 2017 on the basis of the MGL scale, Knowledge-Attitude-Belief Practice (KABP) theory and items analysis theory.8 This provided physicians with feedback on medication adherence for people with chronic diseases taking medication long term. Considering the shortcomings of version-1, we devised a Chinese and Western medication adherence scale for CKD patients (version-2) in 2019, based on the Samejima’s GRM from Item Response Theory (IRT).9 The version-2 scale, which tests have shown has desirable reliability and validity, accounts for herbal medication and CKD medication characteristics, and has been applied in many clinical settings.

Great progress has been made in telemedicine, and this has been accentuated by the COVID-19 epidemic. Traditional paper-version scales have disadvantages, such as inadequate reliability, poor distribution, laborious data collection and entry, and costly resources. As such, the electronic administration of a scale is appealing in medication adherence assessment, as it may reduce administrative work and improve extendibility. Based on the advantages of e-version scales, we have transferred a paper-version medication adherence scale into an e-version. We improve our existing CKD-specific medication adherence questionnaire based on smartphone use, to explore 1) the validity and reliability of an e-version medication adherence questionnaire, and 2) medication adherence response rate facilitating or hindering factors so that we could determine whether the transition is feasible in clinic.

Materials and Methods

Study Setting

In October 2020, we conducted a single-center, exploratory study at the Guangdong Provincial Hospital of Chinese Medicine’s Renal Chronic Disease Management Department. The study was approved by the Guangdong Provincial Hospital of Chinese Medicine’s Ethics Committee (B2016-93) and conducted in accordance with the Declaration of Helsinki.

Study Participants

All patients meeting the following criteria were eligible for inclusion in the pilot study: 1) diagnosed as non-dialysis stage 1–5 CKD according to the 2012 KDIGO guidelines;10 2) taking a Chinese medicine prescription; 3) able to sign informed consent; 4) judged mentally and physically able to participate by our medical team staff; 5) had regularly visited the Guangdong Hospital of Chinese Medicine’s Chronic Disease Management Department (in the Renal Division); 6) able to use WeChat via smartphone. Two researchers notified the eligible patients using the Hospital Information System (HIS) and chronic disease management system. Eligible patients would give a written signed informed consent.

Sample sizes were estimated as at least 150 patients for this 15-item Likert Scale based on a recommended 5:1 patient-item ratio11 and a 50% estimated response rate for the electronic scale.12 For test–retest, given that Park et al13 reported that 40% of studies set a sample-item ratio from 1:1 to 1:4 in order to test the external reliability, we calculated the required sample size as at least 15 patients based on a 1:1 ratio.

Study Procedures

The 15-item Likert Scale which consists of 15 items and 3 dimensions, was input, and possible omitted or incorrect responses were checked in Wen-JuanXing by two trained interviewers just prior to the first version of the questionnaire. Questions were same as paper-version scale. It contains both positively worded (PW) and reverse worded (RW) items (item 10, 11), with ordinal scores of 5-1 or 1–5 points, as sequential options for A–E. The reverse worded scores were pre-designed when typing the e-version. To guide each patient, text clarifying the study details were written in advance. Eligible patients received an electronic invitation around noon on October 13, 14 and 15, 2020, via a smartphone-based WeChat application, by four interviewers. Patients who agreed to participate started the survey by clicking the attached website link or quick response (QR) code (2-dimensional bar code) independently. Additional virtual guidance related to the administration of the survey was offered via WeChat or phone by individual team members, if necessary. The test–retest was performed among randomly selected patients from those who had responded to the questionnaire in the first round by the same group of interviewers two weeks later.13

Information including the name, HIS code, date, survey time, and score, was collected by Wen-JuanXing. All requested data was treated confidentially and used internally for research purposes only. Demographic characteristics (age, sex, marital status, education level, employment status and comorbidity) and relevant indicators of renal function (serum creatinine (SCr) level, estimated glomerular filtration rate (eGFR), and CKD stage) were acquired in the HIS system and chronic disease system (Figure 1).

Figure 1

Issuing and design of the electronic Chinese and western medication adherence questionnaire.

Statistical Analysis

We conducted statistical analysis with IBM SPSS version 23.0. Values were considered statistically significant if P < 0.05 upon two-tailed analysis. Descriptive statistics were expressed using means (M), standard deviation (SD), medians, and interquartile ranges (IQR) for quantitative variables. We employed frequency and percentages for qualitative variables.

Validity is the accuracy of measuring the degree of psychological or behavioral traits. It includes content validity (logical validity), criterion-related validity (empirical validity), and construct validity. We conducted an exploratory factor analysis (EFA) to evaluate the construct validity.

Internal consistency means that all items contribute positively towards measuring the same construct. To evaluate internal consistency, we used Cronbach’s α, split-half reliability, and test–retest reliability in reliability analysis. For test–retest reliability (coefficient of stability), a Pearson correlation coefficient of >0.7 indicated a high correlation with total scores.

Clinical Application

To evaluate the feasibility of the e-questionnaire, we compared the clinical profiles of those patients who had responded and those who had not. We analyzed continuous variables with an independent Student’s t-test or a Mann–Whitney U-test, while binary data were handled with a Chi-Square test. We used a Mann–Whitney U-test to compare ordinal data.

Results

We screened 474 patients in the HIS and chronic disease system; two were on Western prescription medication only, twenty-nine were missing demographic data, one was below 18 years of age, and eight were undergoing dialysis. Finally, 434 patients were enrolled and provided e-version scale. We received valid information (e-version questionnaires and additional clinical documents) for 228 copies, corresponding to a 52.5% response rate. The participants who responded spent a mean time of 348.7 (164.0, 375.8) seconds completing the questionnaire with an average total score of 55.3 (49.3, 61.0) points. There was no other missing data in any of the input information.

Basic information and the available data from the assigned patients can be found in Table 1. The median participant age was 49.0 years, with a 25th percentile below 37.0 and a 75th percentile above 61.0. 51.2% were males. 85.3% were married, 39.4% worked full time and 36.9% were retired. 25.3% had completed junior high school, and 30.2% had completed senior high school/junior technical school. For relevant indicators of renal function, the median SCr and eGFR levels of the participants were 102.5 (79.8, 160.3) μmol/L and 63.3 (35.3,91.1) mL/min/1.73m2, and there were 108 stage 1 (24.9%), 109 stage 2 (25.1%), 141 stage 3 (32.5%), 46 stage 4 (10.6%), and 30 stage 5 (6.9%) CKD patients. Additionally, 52.1% of the participants had chronic diseases other than CKD, and 42.4% had hypertension.

Table 1

Clinical Profiles of the 434 Patients Invited to Participate

Characteristics		Participants (n=434)
Age, years		49.00 (37.0, 61.0)
Sex	Male	222 (51.2%)
	Female	212 (48.8%)
Marital status	Unmarried	57 (13.1%)
	Married	370 (85.3%)
	Other	7 (1.6%)
SCr, μmol/L		102.5 (79.8, 160.3)
eGFR, mL/min/1.73 m2		63.3 (35.3, 91.1)
CKD stage	1	108 (24.9%)
	2	109 (25.1%)
	3	141 (32.5%)
	4	46 (10.6%)
	5	30 (6.9%)
Education level	Primary school	39 (9.0%)
	Junior high school	110 (25.3%)
	Senior high school/junior technical school	131 (30.2%)
	Senior technical college	80 (18.4%)
	Bachelor’s	66 (15.2%)
	Master’s or above	8 (1.8%)
Employment status	Full-time	171 (39.4%)
	Part-time	19 (4.4%)
	Retired	160 (36.9%)
	Laid-off	10 (2.3%)
	Unemployed	24 (5.5%)
	Student	14 (3.2%)
	Other	36 (8.3%)
Comorbidity	Hypertension	184 (42.4%)
	Hyperlipidemia	41 (9.4%)
	Hyperuricaemia	121 (27.9%)
	Diabetes	58 (13.4%)
	Anemia	26 (6.0%)
	Other chronic disease	226 (52.1%)

Note: “Other” means “the patient refused to disclose their social status”.

Validity

Development and refinement in versions 1 and 2 maintained the scale’s acceptable content validity.5,6 Table 2 provides results of the Kaiser–Meyer–Olkin (KMO) measure of sampling adequacy (0.8), and detected an approx. Chi-Square of 1340.0. This means it is applicable for the EFA. We extracted four distinct factors in Principle Component Analysis (PCA). This model could explain 61.5% of the total variance (Table 3). The factor loading on all items was >0.4, except for Item 15. Item 15 also had a Measure of Sampling Adequacy (MSA) of 0.5 (Table 4). Thus, Item 15 was unfit for the factor analysis process. Therefore, the construct validity of the Chinese and Western Adherence Scale’s e-version still needs improvement.

Table 2

Kaiser–Meyer–Olkin and Bartlett’s Test of Sphericity

		Statistics
KMO Measure of Sampling Adequacy		0.8
Bartlett’s Test of Sphericity	Approx. Chi-Square	1340.0
	df	105.0
	Sig.	0.0

Abbreviation: KMO, Kaiser–Meyer–Olkin.

Table 3

Total Variance Explained Using Exploratory Factor Analysis

Factor	Extraction Sums of Squared Loadings			Rotation Sums of Squared Loadings
	Total	% of Variance	Cumulative %	Total	% of Variance	Cumulative %
1	5.1	34.0	34.0	5.0	33.0	33.0
2	1.6	10.8	44.8	1.7	11.1	44.2
3	1.5	9.7	54.4	1.4	9.6	53.8
4	1.1	7.1	61.5	1.2	7.7	61.5

Table 4

Measure of Sampling Adequacy and Factor Loading

	Mean±SD	MSA	C2	Factor Loading
				1	2		3		4
1. Do you know the therapeutic effect of each drug you are taking?	3.6±1.0	0.8	0.6	0.6	0.2		−0.2		0.5
2. Do you know how to use the medicine you are taking?	4.1±1.0	0.8	0.7	0.6	0.3		−0.2		0.6
3. Do you know what utensils should be used for Chinese medicine decoction?	4.1±1.0	0.9	0.6	0.7	0.1		0.2		0.2
4. Did you understand how to properly soak the herbal drugs before decocting?	3.7±1.2	0.8	0.6	0.8	0.1		0.3		0.1
5. Do you understand the following situations that require special handling in traditional Chinese medicine decoction process (ie which herb should be decocted first, which should be decocted later, wrap-boiling, melting, separate decoction, and infusion)?	3.5±1.2	0.9	0.6	0.8	0.0		0.1		0.0
6. Do you know the best temperature for taking traditional Chinese medicine?	3.0±1.3	0.9	0.7	0.8	−0.1		0.0		−0.1
7. Do you know the best time to take traditional Chinese medicine?	2.8±1.2	0.9	0.7	0.8	−0.0		−0.1		−0.1
8. Do you know the dosage of traditional Chinese medicine?	3.1±1.1	0.9	0.7	0.8	−0.0		0.1		−0.2
9. Do you know the dietary contraindications while taking Chinese medicine?	2.9±1.1	0.9	0.6	0.7	0.1		0.1		−0.1
10. I think the taste of traditional Chinese medicine is acceptable	3.8±0.7	0.6	0.6	0.1	0.0		0.7		0.0
11. I think there is no difficulty in decocting and taking Chinese medicine by oneself for a long time.	3.5±0.9	0.7	0.7	0.2	0.1		0.8		−0.0
12. I think it is normal to have all kinds of side effects after taking drugs.	3.0±0.7	0.6	0.6	0.2	0.2		−0.1		−0.7
13. Have you stopped taking medication at any point during the past month because you felt your symptoms had improved?	4.5±0.8	0.5	0.7	0.1	0.8		0.0		−0.0
14. Have you stopped taking medication at any point during the past month because you felt worse?	4.7±0.7	0.8	0.7	0.1	0.8		0.0		−0.2
15. Have you changed a traditional Chinese medicine prescription yourself within the past month?	4.9±0.4	0.5	0.3	−0.1	0.4		0.1		0.3

Note: The grey shading indicates items with factor loading >0.4 .

Internal Consistency

The internal consistency of medication adherence was desirable with a Cronbach’s α of 0.9, but was inadequate with a Guttman split-half coefficient of 0.5 and a Spearman–Brown coefficient of 0.6. The Cronbach’s α was 0.9, 0.4 and 0.5 in the knowledge, belief and behavior domains, respectively. For test–retest reliability, results of the collected 35 copies highlighted a correlation coefficient r for the test–retest reliability of −0.8, and coefficients of −0.8, 0.4, −0.3 for the knowledge, belief and behavior domains, respectively (Table 5).

Table 5

Results of Internal Consistency

Item	Medication Adherence	Knowledge	Belief	Behavior
Cronbach’s Alpha	0.9	0.9	0.4	0.5
Spearman–Brown coefficient	0.6	0.8	0.1	0.3
Guttman coefficient	0.5	0.8	0.1	0.2
Pearson correlation coefficient	−0.8**	−0.8**	0.4*	−0.3
p value	<0.001	<0.001	0.0	0.1

Note: **p<0.01. *p<0.05.

The Relationship Between Patient Characteristics and Scale Response

Of the 434 recruited patients, 228 responded while the other 206 were non-respondents. Patients with comorbidities (Hypertension: Pearson x2=17.5, P < 0.001, Hyperlipidemia: Pearson x2=414.7, P < 0.001, Hyperuricaemia: Pearson x2=129.2, P < 0.001, Diabetes: Pearson x2=347.0, P < 0.001, Anemia: Pearson x2=506.1, P < 0.001, other chronic disease: Pearson x2=56.2, P < 0.001) were more likely to respond. The comparison of other clinical profiles, including age (z =−0.4, P = 0.7), sex (Pearson χ2 = 0.6, P = 0.4), marital status (z = −1.1, P=0.3), SCr (z = −1.1, P = 0.3), eGFR (z =−1.0, P = 0.3), education level (z = −0.4, P=0.7) and working status (z = −1.5, P=0.1) between the respondent and non-respondent group indicated no significant differences (Table 6).

Table 6

Association Between Patient Characteristics and e-Questionnaire Response

Characteristics		Respondent (N=228)	Nonrespondent (N=206)		P value
Age, years		(38.3, 62.0)	(35.8, 61.0)	z =−0.4	0.7
Sex	Male	114 (50%)	108 (52.4%)	Pearson x2=0.6	0.4
	Female	114 (50%)	98 (47.6%)
Marital status	Unmarried	26 (11.4%)	31 (15.0%)	z = −1.1	0.3
	Married	198 (86.8%)	172 (83.5%)
	Other	4 (1.8%)	3 (1.5%)
SCr, μmol/L		(77.3, 146.8)	(81.0, 162.3)	z = −1.1	0.3
eGFR, mL/min/1.73 m2		(34.7, 93.1)	(35.39, 90.3)	z =−1.0	0.3
CKD stage	1	56 (24.6%)	52 (25.2%)	z = −0.3	0.8
	2	58 (25.4%)	51 (24.8%)
	3	78 (34.2%)	63 (30.6%)
	4	22 (9.6%)	24 (11.7%)
	5	14 (6.1%)	16 (7.8%)
Education level	Primary school	19 (8.3%)	20 (9.7%)	z = −0.4	0.7
	Junior high school	57 (25.0%)	53 (25.7%)
	Senior high school/junior technical school	68 (29.8%)	63 (30.6%)
	Senior technical college	48 (21.1%)	32 (15.5%)
	Bachelor’s	33 (14.5%)	33 (16.0%)
	Master’s or above	3 (1.3%)	5 (2.4%)
Working status	Full-time	95 (41.7%)	76 (36.9%)	z = −1.5	0.1
	Part-time	9 (3.9%)	10 (4.9%)
	Retired	86 (37.7%)	74 (35.9%)
	Laid-off	5 (2.2%)	5 (2.4%)
	Unemployed	14 (6.1%)	10 (4.9%)
	Student	7 (3.1%)	7 (3.4%)
	Other	12 (5.3%)	24 (11.7%)
Comorbidity	Hypertension	103 (45.2%)	81 (39.3%)	Pearson x2=17.5	<0.001
	Hyperlipidemia	23 (10.1%)	18 (8.7%)	Pearson x2=414.7	<0.001
	Hyperuricaemia	62 (27.2%)	59 (28.6%)	Pearson x2=129.2	<0.001
	Diabetes	25 (11.0%)	33 (16.0%)	Pearson x2=347.0	<0.001
	Anemia	11 (4.8%)	15 (7.3%)	Pearson x2=506.1	<0.001
	Other chronic disease	122 (53.5%)	104 (50.5%)	Pearson x2=56.2	<0.001

Discussion

After being transferred into the e-version, EFA revealed that the CKD Chinese and Western Medication Adherence Scale is four-dimensional, unlike the paper-version. The MSA and factor loadings indicated that Item 15 needed to be deleted. This indicated poor construct validity in the e-version scale. The Cronbach’s α were 0.9 and 0.9 in medication adherence and medication knowledge, and 0.4 and 0.5 in belief and behavior. Split-half coefficients were worse than Cronbach’s α. The Pearson correlation coefficients were −0.8 in medication adherence and −0.8, 0.4, −0.3 in knowledge, belief and behavior, respectively. This indicated that the e-version scale had undesirable internal consistency reliability, both overall, and among its subscales. In clinical application, individuals with comorbidities were found to be more likely to respond. Based on the validity and reliability analysis, we implemented second factor analysis. Common factor analysis removed Item 15, and extracted four components which explained 65.3% of the variance. The Cronbach’s Alpha was the same as in the original e-version (0.9), but with a higher Spearman–Brown coefficient (0.7) and Guttman coefficient (0.7). However, the Pearson correlation coefficient showed little improvement.

These results were confusing because the paper-version had a Cronbach’s α of 0.84 for medication adherence and 0.9, 0.6, 0.7 for the knowledge, belief and behavior subscales, respectively,6 the latter three having superior quality and balance. It also produced a Pearson correlation coefficient of 0.9.6 It meant that there was a research gap between paper-version and e-version transition. We considered five potential reasons for these undesirable results in e-version scale: 1) there may have been a ceiling or floor effect in some of the items. We found that at least 15% of the participants chose 5 points on Items 1–6 (Knowledge Part) and 13–15 (Behavior Part), and chose 1 point on Item 7 (Knowledge Part). This explained why the knowledge part outperformed the others in validity and reliability. However, this begs the question: why was this not a problem in the paper-version study. This could mean that our participants were regular hospital visitors, and had received CKD medicine education in clinic. As such, they would have been familiar with fundamentals of medicine and would have been supervised by clinic staff. 2)Other potential reasons for the phenomenon may be connected to that fact that most participants had comorbidities. As such, they had attached great importance to these conditions and had exceptional adherence to treatment processes. This demonstrates the achievements in CKD management, in terms of improving medication adherence. On the negative side, our participants were overconfident. 3) Another potential reason for the undesirable results is that technical issues may have impaired response patience, rates and accuracy. Answers to the e-version questionnaire were limited by internet speeds and device limitations. The layout of the questionnaire requested that users slide to navigate the scale forwards and backwards, and font size might have visually influenced questionnaire administration. 4) Environmental factors also may have impaired response concentration. Some participants finished the survey carelessly, without the supervision of medical personnel, or interrupted by external factors, especially when they proceeded to the last item.14 Moreover, for the convenience of the electronic version, completion time was limited to 24 hours, which was too long to reveal actual completion times. 5) Also, inter-patient difference impaired reliability. Course of disease, acquisition of medical fundamentals, degree of recognition and self-management ability varied from participant to participant. Comorbidities encouraged patients to pay more attention to their medication, and investigate their own medication statuses.

The electronic version of the questionnaire had advantages such as its convenience, unlimited time and setting, and rapid and accurate data administration. Plus, it was paperless. However, this study revealed that adapting a paper-version medication adherence scale into an e-version was challenging. Considering the difficulties, we offer several suggestions: For electronic devices, an offline e-version questionnaire less dependent on internet access might prevent network latency. For example, the payment processor Alipay can generate offline QR codes and unique identities for each user as seed data so that their customers can complete transactions even when their devices are operating with a poor signal. Moreover, remote monitoring of time limits could prevent intermittent and/or careless completion. For e-version questionnaires, personalized design is necessary to improve visual perception. Further improvement in content elevates acceptable reliability and validity, which is more suitable to participants. For users, disease management education needs improvement to narrow inter-patient knowledge disparities. In sum, more research on other commonly used PROMs tools for evaluating medication adherence is needed to narrow the transition gap between paper-version and e-version scales.

Limitations

This study has several limitations. Our population had accepted CKD self-management education, which may limit this study’s generalizability to other CKD populations. Moreover, we did not compare the medication adherence questionnaire to either a paper medication adherence questionnaire, or a different computer-administered adherence questionnaire within the same study population. Furthermore, because of data deficiency, we did not analyze the correlation between the medication adherence questionnaire and other biochemical indexes.

Conclusion

We evaluated the e-version Chinese and Western medication adherence scale for CKD, and obtained unacceptable reliability and validity. Methods to monitor scale completion are essential in e-version scale implementation. Caution is needed in transitioning paper-version scales into e-versions. Further research into this would provide the basis for further improvement of computer-administered medication adherence questionnaires.