The reliability and validity of the Turkish version of the Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ) and Telemedicine Patient Questionnaire (TPQ) in individuals with multiple sclerosis.

OBJECTIVE: The study was aimed to translate and adapt the Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ) and Telemedicine Patient Questionnaire (TPQ) into Turkish, thereafter to analyze the psychometric properties of both questionnaires.
METHODS: A total of 149 multiple sclerosis (MS) patients were recruited in the study. For 4 years, all patients were supervised by a department clinician using telemedicine. Cronbach's alpha coefficient was used to assess internal consistency. By evaluating the scores of 41 retested patients 1 week later, the test-retest reliability was determined using the intraclass correlation coefficient (ICC). Pearson's correlation coefficient was used to assess the construct validity (r).
RESULTS: A total of 149 patients (103 women, 46 men) with a mean age of 40.9±10.9 years were included in the study. MS duration of the patients was 9.15±6.24 years. Internal consistency of all items and the total score of the TSUQ were excellent (>0.80; ranged 0.971-0.974). On the other hand, the internal consistency of all items and total score of the TPQ was excellent, either (>0.80; ranged 0.878-0.890). The ICC of the TSUQ's and TPQ's total score was excellent (>0.80). The correlation between TSUQ and TPQ was strong (r=0.734, p<0.01). In addition, there was moderate relationship between the TSUQ and the Beck Depression Scale (BDS) (r=-0.363, p<0.01). On the other hand, there was low correlation between TPQ and BDS (r=-0.217, p<0.05).
CONCLUSION: The Turkish version of the TSUQ and the TPQ is valid and reliable in individuals with MS.

Introduction

Telehealth services are common in neurological assessment and rehabilitation, including stroke, multiple sclerosis (MS), Parkinson’s disease, and epilepsy [1-4]. MS patients need specific healthcare services due to different neurological deficits. Individuals with MS living in rural areas with low socioeconomic levels and mobility problems experience difficulties receiving neurological care. It has been reported that at least 30% of individuals with MS do not have access to healthcare. In addition, MS patients who receive neurological care have more opportunities to receive specific health services such as physiotherapy and occupational therapy [5].

Disease-modifying therapies and neurorehabilitation are essential in the effective management of MS and are provided by neurorehabilitation specialists with proper coordination [6]. Robb et al. compared telemedicine and face-to-face visits in individuals with MS. It has been stated that the telemedicine service provided in the virtual environment for individuals with MS is low cost and saves time. Moreover, it has been reported that it is feasible and preferable for the clinician and that individuals with MS have a high level of satisfaction [7]. In both telemedicine and face-to-face interviews, patient and caregiver satisfaction has been reported to be similar. Individuals suffering from neurological diseases can receive fast and accessible healthcare services with telemedicine service and reduce the cost of healthcare services [8].

The services provided within the scope of telemedicine service in individuals with MS have been reported as functional assessment, treatment follow-up, treatment compliance, cognitive assessment, cognitive rehabilitation, mindfulness, and telerehabilitation [7, 9–13]. In a systematic review on this subject, evidence about the positive effects of telerehabilitation on fatigue, function, and quality of life of individuals with MS has been presented [14]. Moreover, it was stated that, with telerehabilitation, people with MS increased their participation in exercise and their level of satisfaction was good [2].

In order to get the maximum benefit from telehealth technologies, the telehealth application system can be used effectively for clinicians and patients [15]. In addition to the quality and cost of telemedicine services, the acceptance of this service by individuals, its availability, and satisfaction level should be evaluated [16]. The availability of telemedicine services can be made by observation, questionnaire, interview, and different recording methods [17, 18]. Clinical results of telemedicine services can be measured with questionnaires, quality of life, and different tools specific to the disease [19].

In the literature, the expectations, satisfaction, and usability of individuals who receive telemedicine services are evaluated with different surveys: Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ) and Telemedicine Patient Questionnaire (TPQ) [20, 21]. The psychometric properties of the Turkish versions of both questionnaires were not investigated. Besides, there is no other tool that shows the availability and satisfaction of Turkish-speaking individuals’ telehealth services. Thus, it is not possible to control the usability of telehealth applications and receive patient feedback in terms of satisfaction in a standard way. The present study aims to translate and adapt the TSUQ and the TPQ into Turkish, and to analyze the psychometric properties of both questionnaires.

Materials and methods

Translation and adaptation process

Permission was obtained from Bakken and Demiris, who developed the Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ) and Telemedicine Patient Questionnaire (TPQ), respectively. The cultural adaptation of both questionnaires was carried out by taking into account the internationally accepted translation procedures in order to carry out the language translation stages [22, 23]. During the advanced translation phase, the original English was translated into Turkish by four academicians, native speakers of TSUQ and TPQ, who have a good command of English. Expert committee members in the field identified and recorded linguistic and cultural problems. In the second phase, the expert committee members created correction notes of TSUQ and TPQ for translations and discussed. The English translations of both questionnaires were synthesized and finalized. In the third stage, both questionnaires were independently translated into Turkish by two native English translators. Original English versions of both questionnaires were rejected and compared with the English versions in the fourth step, regardless of whether the final Turkish versions were conceptually and linguistically correct. In the last stage, draft versions of both questionnaires were created for use in pilot testing. At the last stage, a pilot study was conducted to question whether the Turkish versions of TSUQ and TPQ are suitable for understandability. A pretest was conducted on a 5-point Likert-type scale with 30 Turkish-speaking elderly individuals. Understandability was excellent in the pilot study. Therefore, no additional changes have been made. The latest Turkish versions of TSUQ and TPQ have been created.

Sample size estimation

The sample size was determined using statistical suggestions and power analysis. Fayers and Machin proposed that in cross-cultural adaptation experiments, the sample size should be at least 100 and, preferably, seven times the number of questionnaires [24]. TSUQ and TPQ have 21 and 17 items, respectively. The psychometric study required a total of 147 participants due to the outnumbered TSUQ items. The minimum sample for the reproducibility was calculated by G* power 3 software (Heinrich-Heine-Universität, Düsseldorf, Germany) with an effect size of 0.40, a probability of error (α) = 0.05, and the power (1-ß) of 0.80. Thirty-four patients were required for the reproducibility. As a result, 149 MS patients were tested in the first round, and 41 patients were refilled the TSUQ and TPQ, 1 week later for retest reliability [25].

Study design

The study was conducted as a prospective cross-sectional design by “X” University Neurology Department on individuals with MS. Inclusion criteria of the study: Individuals with MS who are older than 18 years old and have been monitored with telemedicine for at least 4 years, individuals who have not had an MS attack in the last month, individuals with MS who can read and understand Turkish. The study’s exclusion criteria are individuals who have had an MS attack the last month. Telemedicine services (e.g., medication modifications, monitoring of symptoms, exercise management, follow-up of relapses) conducted to the patients were carried out through several operators or through web-based applications, phone calls, messaging, teleconference, within the needs and possibilities of the patients. Socio-demographic and physical characteristics of individuals with MS were recorded. Expanded Disability Status Scale (EDSS) of the patients was scored by the neurologist [26]. In the first evaluation, 149 individuals with MS completed by thoroughly answering all the items of the Turkish versions of TSUQ, TPQ, and Beck Depression Scale (BDS) [20, 21, 27]. Due to the psychological criterion of pleasure was questioned, convergent validity was investigated by comparing it to BDS. In addition, since the expectations and benefits of the patients from the telemedicine service were also questioned, it was thought that it would have positive effects on their psychological state. A retest evaluation was carried out for individuals with MS 1 week later. In the retest evaluation, 41 individuals with MS answered the Turkish versions of TSUQ and TPQ.

Telemedicine Satisfaction and Usefulness Questionnaire

TSUQ was developed by Bakken et al. in 2006. It consists of 21 items. With this questionnaire, the satisfaction levels and usability of individuals using telemedicine service (from the software or system from which they receive treatment or other therapy services) are evaluated. A 5-point Likert scale is used in TSUQ evaluation (1 = strongly disagree, 5 = strongly agree). The total score ranges from 21 to 105 [20].

Telemedicine Patient Questionnaire

TPQ was developed by Demiris et al. It consists of 17 items in total. A 5-point Likert-type scale is used for TPQ evaluation. It evaluates individuals’ telemedicine service, expectation, satisfaction, and usability. Items 2, 3, 7, 15, and 17 are scored by reversing. The total score is scored between 17 and 85 [21].

Beck Depression Scale

The Turkish version of the scale was proved for validity and reliability. BDS evaluates depression severity. BDS consists of 21 items. Each item is pointed between 0 and 3. It is assessed over the sum score. A higher score shows high depression. The highest score is 63 [27].

Statistical analysis

SPSS for Windows v25.0 was used for statistic interpretations in all of the analyses (SPSS Inc, Chicago, IL, USA). The quantitative data was assigned with a mean and a standard deviation. For the qualitative results, percentages were used. To detect whether the data in the sample had a homogeneous distribution, the Shapiro-Wilk test was used. The correlation coefficients were agreed with 95% confidence intervals.

Reliability

The Cronbach alpha coefficient was calculated to determine the TSUQ and TPQ’s internal consistency. TSUQ and TPQ, which have a higher alpha coefficient, suggest more consistent results. An alpha of ≥0.6 was deemed acceptable; ≥0.8 was ranked excellent consistency [28]. The two-way random-effect model single-measure reliability study of Shrout and Fleiss was chosen. Reproducibility was measured using the intraclass correlation coefficient (ICC, 95% confidence interval). ICC was calculated for the total score and items of the TSUQ and TPQ. An ICC ≥0.8 shows excellent test-retest reliability [29].

Validity

The construct validity of TSUQ and TPQ was tested by Pearson’s correlation coefficient (r). The total of the TSUQ, TPQ, and BDS was compared with each other. A high correlation coefficient was presumed in terms of convergent validity. In addition, low correlation was considered divergent validity. If the coefficient was greater than 0.5, it was considered strong; if it was between 0.5 and 0.35, it was considered medium; and if it was less than 0.35, it was considered low [30].

Results

A total of 149 patients (103 women, 46 men) with a mean age of 40.9±10.9 years were recruited in the study. MS duration of the patients was 9.15±6.24 years. Table 1 shows the absolute values of the patients. TSUQ and TPQ were excellently understandable in linguistic and cultural concepts in a pilot study conducted with 30 Turkish-speaking individuals. As a result, after the pretest, no additional changes were required (Table 2).

Table 1

The characteristics of the patients

n: 149	Total
Age (years, mean±SD)	40.9±10.9
BMI (kg/m2)	25.1±4.3
Gender (n, %)
Female	103 (69.1)
Male	40 (30.9)
MS duration (years, mean±SD)	9.15±6.24
Education
Primary school (n, %)	17 (11.4)
Middle school (n, %)	14 (9.4)
High school (n, %)	48 (32.2)
University (n, %)	64 (43.0)
Postgraduate (n, %)	6 (4.0)

SD standard deviation, n number of patients, BMI body mass index, MS multiple sclerosis

Table 2

The clinical assessment values of the patients

n: 149	Total (mean±SD)	Range
TSUQ	85.2±21.0	21-105
TPQ	62.2±9.9	39-85
BDS	11.7±10.4	0-41
EDSS	2.0±1.7	0-7

SD standard deviation, n number of patients, BMI body mass index, MS multiple sclerosis, TSUQ Telemedicine Satisfaction and Usefulness Questionnaire, TPQ Telemedicine Patient Questionnaire, BDS Beck Depression Scale, EDSS Expanded Disability Status Scale

Reliability and validity

The study’s reliability findings are summarized in Tables 3 and 4. Cronbach’s alpha coefficient was determined for the TSUQ and TPQ items and overall ratings. Internal consistency of all items and the total score of the TSUQ were excellent (>0.80; ranged 0.971–0.974). On the other hand, the internal consistency of all items and total score of the TPQ was excellent, either (>0.80; ranged 0.878–0.890). The ICC of the TSUQ total score was excellent (>0.80). The ICC score of items for TSUQ was ranged from 0.581 to 0.889. The reproducibility of the total score for the TSUQ was excellent (>0.80). The test-retest reliability of the items of the TPQ was ranged from 0.214 to 0.873. The validity results are presented in Table 5. The correlation between TSUQ and TPQ was strong (r=0.734, p<0.01). In addition, there was moderate relationship between the TSUQ and the total score of the BDS (r=−0.363, p<0.01). On the other hand, there was low correlation between TPQ and BDS (r=−0.217, p<0.05).

Table 3

Test-retest reliability and internal consistency for the TSUQ

	Test (Mean±SD)	Retest (Mean±SD)	ICC (95% CI)	α
Item 1	4.28±1.10	4.02±1.15	0.878 (0.77–0.93)	0.972
Item 2	3.32±1.47	3.29±1.40	0.581 (0.21–0.77)	0.974
Item 3	3.71±1.40	3.73±1.28	0.625 (0.29–0.80)	0.972
Item 4	3.97±1.25	3.85±1.29	0.817 (0.65–0.90)	0.972
Item 5	4.14±1.21	3.95±1.28	0.788 (0.60–0.88)	0.971
Item 6	3.97±1.25	3.56±1.46	0.821 (0.66–0.90)	0.972
Item 7	3.80±1.33	3.56±1.44	0.889 (0.79–0.94)	0.972
Item 8	4.18±1.17	3.85±1.37	0.817 (0.65–0.90)	0.971
Item 9	4.18±1.15	3.97±1.27	0.721 (0.47–0.85)	0.971
Item 10	4.18±1.22	3.92±1.33	0.741 (0.51–0.86)	0.971
Item 11	3.85±1.35	3.29±1.20	0.719 (0.47–0.85)	0.972
Item 12	3.85±1.30	3.60±1.26	0.786 (0.59–0.88)	0.972
Item 13	4.28±1.18	4.31±1.01	0.738 (0.50–0.86)	0.972
Item 14	4.15±1.21	3.75±1.19	0.786 (0.59–0.88)	0.972
Item 15	4.02±1.26	3.78±1.42	0.727 (0.48–0.85)	0.972
Item 16	4.00±1.24	3.85±1.35	0.804 (0.63–0.89)	0.972
Item 17	4.21±1.18	3.97±1.33	0.763 (0.55–0.87)	0.971
Item 18	4.38±1.30	3.90±1.37	0.772 (0.57–0.87)	0.971
Item 19	4.36±1.13	4.39±0.97	0.618 (0.23–0.79)	0.971
Item 20	4.36±1.10	4.43±0.86	0.600 (0.24–0.78)	0.971
Item 21	4.33±1.11	4.21±1.06	0.665 (0.37–0.82)	0.972
Total score	85.2±21.0	81.26±20.74	0.850 (0.71–0.92)	0.973

n number of patients, ICC intraclass correlation coefficient, CI confidence interval, α Cronbach’s alpha, TSUQ Telemedicine Satisfaction and Usefulness Questionnaire

Table 4

Test-retest reliability and internal consistency for the TPQ

	Test (Mean±SD)	Retest (Mean±SD)	ICC (95% CI)	α
Item 1	3.56±0.96	3.51±0.97	0.513 (0.08–0.74)	0.884
Item 2	3.84±0.89	3.68±0.81	0.573 (0.19–0.77)	0.886
Item 3	4.08±1.05	3.75±1.42	0.632 (0.30–0.80)	0.890
Item 4	3.57±1.09	3.53±1.09	0.686 (0.41–0.83)	0.879
Item 5	3.42±1.02	3.82±0.86	0.566 (0.18–0.76)	0.880
Item 6	4.14±0.77	4.26±0.70	0.652 (0.34–0.81)	0.882
Item 7	3.40±1.23	3.21±1.40	0.214 (–0.47–0.58)	0.890
Item 8	3.89±0.84	3.87±1.00	0.813 (0.64–0.90)	0.878
Item 9	3.11±1.15	3.29±1.00	0.627 (0.30–0.80)	0.888
Item 10	3.73±0.93	3.80±0.98	0.751 (0.53–0.86)	0.878
Item 11	4.08±0.83	4.17±0.73	0.657 (0.35–0.81)	0.879
Item 12	3.78±0.99	3.82±1.18	0.784 (0.59–0.88)	0.880
Item 13	4.08±0.74	4.24±0.73	0.789 (0.60–0.88)	0.884
Item 14	4.04±0.77	4.07±0.87	0.634 (0.31–0.80)	0.884
Item 15	2.62±1.19	2.80±1.16	0.873 (0.76–0.93)	0.882
Item 16	3.88±0.79	3.90±0.94	0.755 (0.54–0.86)	0.882
Item 17	2.99±1.02	3.17±1.04	0.729 (0.49–0.85)	0.886
Total score	62.2±9.9	62.9±11.3	0.876 (0.76–0.93)	0.889

n number of patients, ICC intraclass correlation coefficient, CI confidence interval, α Cronbach’s alpha, TPQ Telemedicine Patient Questionnaire

Table 5

Correlation between TSUQ, TPQ, and BDS

n: 149	r	p
TSUQ-TPQ	0.734	<0.01
TSUQ-BDS	−0.363	<0.01
TPQ-BDS	−0.217	<0.05

TSUQ Telemedicine Satisfaction and Usefulness Questionnaire, TPQ Telemedicine Patient Questionnaire, BDS Beck Depression Scale, r Pearson’s correlation coefficient, p significance level

Discussion

This research aimed to demonstrate the TSUQ’s and TPQ’s translation, cross-cultural adaptation, reliability, and validity in MS patients who were monitored with telemedicine by the neurology department for 4 years. As a result, the Turkish versions of the TSUQ and TPQ have been proven reliable and valid patient-reported outcome measures for testing Turkish-speaking, multiple sclerosis cases. To adapt the questionnaires to Turkish patients, a scientific, textual, and cultural adaptation procedure was followed [22, 23]. Psychometric properties of the TSUQ and TPQ were also successfully presented. Since there are no Turkish screening methods for telehealth providers, both questionnaires are a one-of-a-kind assessment tool for Turkish-speaking patients.

It is evident that the importance of telemedicine applications during the COVID-19 era further increased [31]. The importance of telemedicine in terms of providing care and increasing accessibility in patients with MS during the COVID-19 period has been emphasized in a recent study [32]. In addition, it has been stated that telemedicine has vital importance in pandemic situations due to possible limited staff, limited outpatient working hours, and increasing outpatient needs [33]. In this study, psychometric properties of TSUQ and TPQ were analyzed in individuals who had been undergoing a telerehabilitation program for 4 years, but more intensively for the last 6 months, by the neurology outpatient clinic. In this way, it is proposed to provide an analysis to use these patients reported outcome measures in future studies safely. TSUQ is one of the most popular questionnaires frequently used in telemedicine studies. English and Spanish versions of the TSUQ are available [20]. On the other hand, TPQ is available only in its original English version [21]. The Turkish version of both questionnaires could greatly lead to clinical practice and other case-control or intervention trials, given the dramatic rise in telemedicine and telerehabilitation in recent years and the role of remote health services in the COVID-19 process [34-36]. The consistency of the telemedicine program and level of satisfaction could be compiled and tracked in a more detailed and systematic way using the quantitative data derived from patient reviews. Both questionnaires found to be practical, easy to use, and thus can be completed in a limited period of time.

The phrase “telemedicine program” was changed as “method” in the TSUQ and TPQ translation adaptation processes. The translation committee anticipated this technological revolution, speculating that telemedicine programs would no longer be restricted to the “system.” Furthermore, the term “clinician” was favored over “healthcare professional” in TPQ. The translation committee introduced this modification because the “clinician” identified a possible telemedicine provider in all health sciences to improve conformity with the questionnaire’s interprofessional mode. The TSUQ and TPQ were found to be appropriate in terms of comprehensibility in a pilot analysis. As a result, no other changes to the pretest were needed. The internal consistency for the total score and item scores of the TSUQ and TPQ was excellent (α>0.80). TSUQ and TPQ are consistently assessing the usefulness and satisfaction of telemedicine systems, according to the findings. Besides, the items of both questionnaires are compatible within themselves. Only Demiris and colleagues analyzed the Cronbach alpha for TPQ. The results of the alpha are above 0.80 [21]. Their calculation results were conformed to our study (α> 0.80). Cronbach’s alpha scores were not presented in the TSUQ development study [20].

The test-retest reliability for the total scores of the TSUQ and TPQ was excellent (ICC>0.80). In addition, all items (except item 2) of the TSUQ have moderate to excellent reproducibility (>0.60). The ICC value of only the second items of TSUQ was less than 0.6. In this article, the answer to the question given to the sentence “My health is better than it was before I used the technology” is sought. In the answer given by the patients to this question, we emphasized the possibility of not understanding telemedicine service with the word technology. In addition, the ICC score in items 1, 2, 5, and 7 in TPQ was below 0.60. However, despite this, having the ICC score of the total score above 0.80 is sufficient for the questionnaire’s reproducibility. The ICC was not mentioned in the development work of both questionnaires. Our study is also unique in this respect.

Construct validity was analyzed by comparing the TSUQ, TPQ, and BDS scores. The correlation between TSUQ and TPQ was strong (r=0.734, p<0.01). In addition, there was moderate relationship between the TSUQ and the total score of the BDS (r=−0.363, p<0.01). On the other hand, there was low relationship between the TPQ and the total score of the BDS (r=−0.217, p<0.05). In particular, the authors used BDS mainly within the scope of divergent validity. In other words, we were predicting that there would be a low to moderate correlation of BDS with TSUQ and TPQ. According to the validation results of our study, both questionnaires were found to be valid.

It is also necessary to note the study’s limitations. The current research did not look at responsiveness analysis. Furthermore, the “System Usability Scale” was not used to determine validity [37]. The construct validity could be further clarified using a 10-item System Usability Questionnaire. However, it could be time consuming to ask the participants more questions. It can also obstruct the processing of specific data. Furthermore, in both surveys, testing and validating more psychometric properties in various case types would be more helpful to clinical practice.

Conclusions

Ultimately, the Turkish versions of the TSUQ and TPQ are determined to be reliable and valid for measuring usability and satisfaction in MS cases. Both questionnaires are accurate patient-reported outcome measures for patients with multiple sclerosis. Both questionnaires also would contribute significantly to both science and clinical practice, given that telemedicine is a trendy era due to the rapid development in healthcare technologies.