Instrument for Assessing Patients' Desirability, Acceptability, and Adherence to Telemedicine in Diabetes: Development, Validity, and Reliability.

PURPOSE: To develop, test, and validate an instrument for assessing patients' desirability, acceptability, and adherence to telemedicine in diabetes. PATIENTS AND METHODS: A group of six experts defined the components that needed to be assessed when developing telemedicine platforms aimed at improving the care of patients with diabetes. The resulting instrument was tested for reliability and construct validity of 114 patients with diabetes and re-tested for reproducibility and consistency on a sub-group of 34 patients. Based on the analysis, the questionnaire's syntax, phrasing, and flow were improved to obtain optimal assessment results.
RESULTS: The resulting questionnaire has three major sections: one for the main evaluated component (acceptability, desirability, and adherence to telemedicine in diabetes), one for collecting socio-economic and demographical information, and one for diabetes history-related data, respectively. The corresponding data for these sections are collected using 29 questions. The developed instrument has a good reliability (Cronbach's alpha = 0.890), reproducibility (Total score 164 vs 166 points at re-test vs test; Mann-Whitney U p-value = 0.394) and external validity (Spearman's rho = -0.580; p < 0.001 for correlation with Patient's Health Questionnaire -9; and Spearman's rho = -0.516; p < 0.001 for correlation with Generalized Anxiety Disorder - 7).
CONCLUSION: The developed questionnaire is a valid and reliable instrument in assessing the patients' acceptability, desirability, and adherence to telemedicine use in diabetes care. In addition, the use of this instrument in the development of telemedicine platforms may improve and optimize their usage by providing valuable information regarding patients' preferences and specific needs.

Introduction

Diabetes Mellitus (DM) is one of the major public health issues nowadays and was estimated to affect approximately 463 million individuals worldwide in 2019, generating major direct and indirect costs to society and healthcare systems. Furthermore, the incidence trend of DM points to a continuous increase over time. The prevalence estimates for persons living with DM in 2045 is 700 million individuals.1

A major burden in diabetes-care is the lifelong nature of the disease. To achieve improvements in disease specific as well as overall prognosis, the disease management interventions consisting in lifestyle optimization, medication, and interventions aiming to alleviate risk factors should be continuous during the patient’s life after the diagnosis is made.2 Since in addition to indirect and intangible costs, the diabetes-related direct healthcare costs rise to approximately 760 billion USD with a consistent over-time increase according to the International Diabetes Federation, the necessity of improving the cost-efficiency of health care solutions emerges as a sine qua non condition to be able to provide further quality care for patients with DM.1

According to the World Health Organization, telemedicine represents “the use of information and communications technology to improve patient outcomes by increasing access to care and medical information”.3 Besides overcoming distance barriers, the use of telemedicine is acknowledged to optimize the diagnosis and treatment flow in patients’ care, reducing the cost and time-related burden of healthcare. Considering that DM is a chronic disease, in which patients are periodically undergoing medical visits, usually at the same healthcare provider, and considering that during the disease’s course, the patients’ health status is usually predictable, telemedicine may become a valuable tool in enhancing diabetes care by optimizing the patient – healthcare provider interaction.3 Several optimized health-related interventions are becoming feasible regarding the care of the patient with DM: healthcare professionals may be able to provide diabetes care advice remotely, patients may be able to send DM-related data automatically (ie, data from glucose-meters, continuous glucose monitoring sensors, or insulin-pump charts), and providers may send medical prescriptions. Since usage of technology emerges nowadays in the modern paradigm of diabetes care, it is to be expected that the role of telemedicine in diabetes care will increase in the following years.4

Besides this, the role of telemedicine may be of paramount importance in continuing the care provided to patients with DM. For example, during the global COVID-19 pandemic which started in 2020, DM has been recognized to be one of the most important risk factors for developing moderate or severe forms of COVID-19. This emphasized the need for patients with DM to implement social distancing measures as well as to avoid any unnecessary movements outside their housing, including hospital visits except for medical emergencies. In this scenario, telemedicine played a crucial role in continuing the care of patients with DM.5

Since telemedicine has a great potential in improving the care of patients by enhancing the traditional medical consult and knowing that a key role in the success of its implementation is played by its acceptance by the patients with DM, our aim was to develop, test, and validate an instrument for assessing patient’s acceptability, preference, and desirability regarding telemedicine solutions used in diabetes care.

Materials and Methods

Questionnaire Development

A group of 6 experts (3 medical experts, one social sciences expert, one linguistic and semantics specialist, and one biostatistician) were involved in the development of the questionnaire. The development process was divided according to the stepwise algorithm proposed by Stone in the British Medical Journal, as follows:6

Decide what data you need: the questionnaire aimed to assess three major components regarding the use of telemedicine as complementary care in patients with diabetes: desirability, acceptability, and adherence.

Select items for inclusion: the three main components under evaluation were sub-divided into multiple clustered questions, aiming to assess all the desirability, acceptability, and adherence aspects regarding the use of telemedicine as a complementary healthcare instrument, focusing on the diabetes-specific patient to healthcare provider interaction. Furthermore, relevant demographic, anthropometric, and clinical diabetes-related data were added to the questionnaire.

Design of the individual questions: each data collection item was analyzed, and based on the optimal collecting variable type, the answer type was accordingly designed: for nominal and scale variables the response was collected using a free-text form, for ordinal variables the response was collected using a Likert scale, and for dichotomous variables the response was collected using a yes/no field, respectively.

Compose wording: question and answer wording for each item was done using a sequential backwards feedback procedure, first through the medical experts and afterwards validated by the social sciences and linguistic experts.

Design the layout and presentation: after reaching the alpha-version of the question bank of the questionnaire, the layout and presentation were developed for both printed and electronic data collection form.

Pilot and evaluate: the instrument developed was tested for internal consistency, construct validity, external reliability, and test/re-test consistency. Based on the resulting statistics, the phrasing, question order, and flow of the questionnaire were optimized.

Patients and Study Enrollment

Patients who previously used the diabetes-care telemedicine platform (telediabet.ro), developed by the “Pius Brinzeu” Emergency Hospital from Timisoara and who provided consent for further contact, were invited to participate in the study via an email message provided at the time of first access to the platform during a 2-month timeframe (May–June 2021). One hundred and fourteen patients agreed to participate in completing the main questionnaire as well as the external reliability instrument, and 35 of them accepted to participate in the test/re-test validation, which consisted in both completion of the main questionnaire at the initial moment and the external reliability instrument, as well as a re-test after one month, by again completing the main questionnaire in order to evaluate the response reproducibility.

The responders’ demographic and clinical characteristics, as evaluated at the initiation of the study, are presented in Table 1.

Table 1

Demographic and Clinical Characteristics of Study Participants

Studied Parameter	Value
Gender
Men	30 (26.3%)
Women	84 (73.7%)
Residence
Urban	88 (77.2%)
Rural	26 (22.8%)
Occupation
Student	4 (3.5%)
Employee	70 (61.4%)
Freelancer	8 (7.0%)
Unemployed	3 (2.6%)
Retired	29 (25.4%)
Diagnosis
Type 1 Diabetes	71 (62.2%)
Type 2 Diabetes	43 (37.8%)
Education
Primary school	3 (2.6%)
High school	33 (28.9%)
University degree	40 (35.1%)
Master or PhD	38 (33.3%)
Internet access
Intermittent access to the internet	4 (3.5%)
Permanent access to the internet	110 (96.5%)
Most frequently used device for internet access
Smartphone	82 (71.9%)
Laptop or desktop computer	29 (25.4%)
Tablet	3 (2.6%)
Duration of diabetes
Less than one year	5 (4.4%)
1–5 years	39 (34.2%)
6–10 years	19 (16.7%)
More than 10 years	51 (44.7%)

Ethics Approval

The study protocol and informed consent template were approved by the Ethics Committee of the “Pius Brinzeu” Emergency Hospital, Timisoara, Romania (decision number 200/12 August 2020). Informed consent was obtained for all participants, being the first step in the electronic data collection form. The study was conducted according to the guidelines of the declaration of Helsinki

Statistical Analysis

Data were collected using a dedicated electronic collection form and analyzed using the Statistical Package for Social Sciences v.28 (IBM Corp., Armonk, N.Y., USA).

For descriptive statistics, the results are presented as mean ± standard deviation (numerical variables with Gaussian distribution), median and [interquartile distance] (numerical variables with non-parametric distribution), and absolute frequencies and percentage from the sub-group’s total for nominal or dichotomous variables, respectively.

The internal consistency of the instrument was evaluated using the Cronbach’s alpha method. For the purpose of this study, a Cronbach’s alpha coefficient higher than 0.7 was considered an indicator of a good internal consistency.

The construct and convergent validity of the questionnaire were evaluated by being handed out in parallel with the Romanian version of the Patient’s Health Questionnaire – 9 and Generalized Anxiety Disorder – 7 questionnaires, instruments previously validated in the Romanian population. To evaluate the association between instruments, the Spearman’s r correlation coefficient was calculated; a higher in modulo Spearman’s r correlation coefficient is associated with improved construct and convergent validity.

The reproducibility of the instrument was evaluated via re-assessment using the questionnaire a month after the initial assessment in patients who provided consent to participate in the follow-up of the initial study. Thirty-five patients agreed and performed the entire test/re-test procedure. Two methods were used to evaluate the reproducibility: correlation analysis using the Spearman correlation coefficient, as well as assessing the statistical significance of the provided results between re-test vs test using the paired Wilcoxon signed-rank test; a higher in modulo Spearman’s r coefficient, respectively, non-significant (p ≥ 0.05) differences were considered to be associated with a good degree of reproducibility of the instrument.

Based on the variable’s distribution obtained after the pilot of the study, we estimated that a sample size of 110 individuals was needed for the primary objective (internal consistency and construct validity), respectively, 30 individuals were needed for the secondary objective (instrument reproducibility) to obtain a statistical power of 80% in parallel with an α value of 0.95.

In this study, a p-value of 0.05 was considered the threshold of statistical significance.

Results

Questionnaire Design

After identifying the aspects of interest, the questionnaire was divided into three major sections: one for the main evaluated component (acceptability, desirability, and adherence), one for the collection of socio-economic and demographical information, and one for diabetes history-related data. The corresponding data for these sections were collected using 29 questions (Q1-Q9 evaluated the main component, Q10-Q19 evaluated socio-economic and demographical data, and Q20-Q29 evaluated diabetes-related information). Q6, Q7, Q8, and Q9 were clustered Likert-scale questions, each main question containing 7, 8, 9, and 6 sub-questions, respectively. To obtain the score, each Likert question or sub-question was scored between 1 and 5 points (with Q6 being scored on a reverse scale). Dichotomous responses were scored with 1 point for “no” responses and 5 points for “yes” responses. Multiple-choice answers regarding the acceptability of use for several telemedicine components were scored with one point for each accepted component.

The main outcome of the questionnaire is the global telemedicine desirability adherence and acceptance score, QTelemeDiab. The QTelemeDiab score is the sum of Q1 – Q9 individual scores. The telemedicine desirability sub-score is the sum of Q6, Q7, Q8 (minimum score: 24, maximum score: 120). The telemedicine acceptability sub-score is the sum of Q1, Q2, Q4, Q5 (minimum score: 2, maximum score: 25). The telemedicine adherence sub-score is the sum of Q3, Q9 (minimum score: 8, maximum score: 40). The total score ranges from 34 to 185. The sub-scores are not proportional. Thus, to compare sub-scales, the percentage from the scale’s maximum should be used, not the absolute score obtained.

The resulting questionnaire is presented in and may be freely used for research purposes with proper citation of the source.

Reliability Analysis

Reliability Analysis

The built instrument had a good internal consistency, as defined by the obtained Cronbach’s alpha coefficient in both overall and clustered analyses. For the overall component, including all questionnaire’s items, a Cronbach’s alpha value of 0.890 [0.859 to 0.917] 95% CI. The scale was consistent between items. When items were stepwise deleted from the item-total statistics, the overall alpha value ranged from 0.882 (Q8 being the most consistent item, due to a decrease in the score when deleted) to 0.902 (Q5 being the least consistent item, due to an increase in the score when deleted).

Among the analyzed sub-scales, the components assessing desirability (Cronbach’s alpha = 0.915) and adherence (Cronbach’s alpha = 0.691) had better internal consistency in contrast to the components assessing acceptability (Cronbach’s alpha = 0.328). Q5 was the item least correlated with the other items evaluating acceptability (r = 0.054). However, it was kept in the instrument’s design due to its importance in providing health-policies insights and development of further telemedicine platforms. The internal consistency of the components of the instrument is presented in Table 2.

Table 2

Internal Consistency of the Questionnaire

Component	Cronbach’s Alpha	95% Confidence Interval	p-value
Overall	0.890	0.859 to 0.917	<0.001
Desirability	0.915	0.890 to 0.936	<0.001
Adherence	0.691	0.597 to 0.770	<0.001
Acceptability	0.328	0.082 to 0.516	0.006

Reproducibility

No significant differences were observed between test and re-test regarding the total score or regarding the sub-components of the instrument, thus demonstrating a good reproducibility in evaluating both the main and sub-components (Figure 1, Table 3).

Figure 1

Differences between initial assessment and re-test regarding the evaluated components.

Table 3

Test vs Re-Test Results

Evaluated Component	Test		Re-Test		p-value
	Median Score	Interquartile Distance	Median Score	Interquartile Distance
Total score	166	18	164	15	0.394
Desirability	116	11	113	8	0.073
Adherence	35	9	35	5	0.241
Acceptability	17	6	17	5	0.595

The re-test results were correlated significantly regarding both the total questionnaire score (Spearman’s rho = 0.531; p = 0.001) and for the evaluated sub-components: adherence (Spearman’s rho = 0.573; p < 0.001), desirability (Spearman’s rho = 0.447; p = 0.007) and acceptability (Spearman’s rho = 0.900; p < 0.001).

The inter-item correlations are presented in Figures 2 and 3.

Figure 2

Correlation matrix between the initial assessment and re-test.

Figure 3

Correlations between test and re-test scores.

External Validity

The score obtained by the developed instrument was strongly, reversely, and significantly correlated with the scores obtained using two previously validated instruments for the Romanian population with diabetes: PHQ-9 (Spearman’s rho = - 0.524; p < 0.001; Figure 4) and GAD-7 (Spearman’s rho = −0.447; p < 0.001; Figure 5).

Figure 4

Correlations between total score and PHQ-9.

Figure 5

Correlations between total score and GAD-7.

Good associations with the previous PHQ-9 and GAD-7 scores were observed both for the total score and for the score’s sub-components (Table 4).

Table 4

Associations Between PHQ-9, GAD-7, and the Instrument Sub-Component Scores

Correlation Matrix
		Total Score	Desirability	Acceptability	Adherence
PHQ-9	Spearman’s rho	−0.580	−0.630	−0.095	−0.410
	p-value	<0.001	<0.001	0.317	<0.001
	N	114	114	114	114
GAD-7	Spearman’s rho	−0.516	−0.586	−0.055	−0.332
	p-value	<0.001	<0.001	0.564	<0.001
	N	114	114	114	114

Discussion

This questionnaire is the first comprehensive instrument developed to date, which assesses all three dimensions regarding the patients’ usage of telemedicine systems in diabetes: acceptability, desirability, and adherence. The prior assessment of these components is of a paramount importance in defining, developing, and implementing the usage of telemedicine systems in diabetes, by identifying and adapting the platforms to be developed to particular patients’ preferences, which may be influenced by socio-economic, cultural, or educational factors among different populations.7 It is known that the first step in the success of these interventions is represented by patients’ adoption of these instruments. Thus, the tailored development of telemedicine platforms is a key component in implementing telemedicine solutions.8 Considering the predictions for an increased incidence of diabetes over the next 25 years, in parallel with the limited resources available for diabetes care, designing telemedicine systems to augment the care of diabetes may be an appropriate solution in order to provide adequate high-quality diabetes care.9–11 The instrument developed in this project is a valid tool in assessing the patients’ acceptability, desirability, and adherence to telemedicine systems in diabetes care, providing reliable, consistent and reproducible data to the user. Furthermore, this instrument allows the provider to identify patients’ particular preferences regarding telemedicine use based on socio-economic, demographic, disease history or occupational characteristics, thus opening the door for the development of targeted platforms for specific patient populations.12

Since diabetes care guidelines are pointing to similar interventions regardless geographical or socio-economical differences between countries and since the diabetes management using telemedicine interventions does not have major particularities between different countries, this instrument may be used to evaluate patient’s desirability, acceptability and adherence to telemedicine interventions in any public health system.

The main drawback of the study is the fact that all participants were previously users of telemedicine platforms for diabetes care – this being the main recruitment criteria. However, this fact does not influence the development, reliability, and construct validity-related results; it may only generate a value distribution skewed to the left, impacting only the raw results of the scoring instrument. Also, since most of our telemedicine platform users were patients with Type 1 Diabetes Mellitus (T1DM), most of the population on which the questionnaire was validated consisted of a higher proportion of patients with T1DM vs T2DM compared to the general population of patients with DM. Neither of these facts is an issue as they do not affect the instrument’s validation as they can only impact the raw score.

Conclusion

The questionnaire developed in this study is a valid instrument for assessing patients’ desirability, adherence, and acceptability towards the use of telemedicine in diabetes care. The developed instrument had good internal reliability, external validity, and result reproducibility. The use of the instruments to assess patients’ desirability, acceptability, and adherence in the development of telemedicine services for diabetes care may improve and optimize the outcomes of telemedicine use by providing valuable information regarding patients’ specific needs and preferences, by prioritizing the areas of intervention according to patients’ preferences and by defining efficient public health policies regarding telemedicine interventions in diabetes care.