Evaluation of Psychometric Properties of the Acceptability of ICT Use for Mental Health Care Questionnaire.

The adoption of e-mental health strategies is determined by the acceptability of Information and Communication Technologies (ICT) to professionals, an issue barely explored in Mexico. The objective was to evaluate the psychometric properties of the Acceptability of ICT Use for Mental Health Care Questionnaire. A convenience sample of 492 Mexican mental health professionals answered the questionnaire on ICT use for mental health care. Internal consistency analyses, exploratory, and confirmatory factor analyses were performed. Around 73.4% of the sample are women, aged between 21 and 30 (32.1%), around 50% has completed graduate studies, and 88.3% is practicing psychologists. The questionnaire comprised 11 items grouped into three factors: perceived usefulness, risk perception, and subjective norm. The CFA indicated a good fit χ2 SB/gl = 443.38/206 = 2.15; CFI = 0.930; RMSEA = 0.068 (95% CI [0.059-0.077]). The instrument has adequate psychometric properties for evaluating the acceptability of ICT use for mental health care.

The health sector has incorporated the use of the Internet and other Information and Communication Technologies (ICT) for the provision and administration of care services and human resource training, which benefits both health professionals and sector users (Edejer, 2000). These changes in health service provision are known as e-Health, a relatively new concept that refers to the use of ICT for the set of functions that affect the health sector (Commission of the European Communities, [CEC], 2001).

It should be noted that mental health care is one of the areas capable of implementing e-Health strategies through the design, evaluation, and implementation of interventions via the Internet using various applications and technological devices. This field of application is known as e-mental health (Blankers 2011). There is a growing body of literature that demonstrates the usefulness of this alternative for serving people with limited access to traditional treatment services, or who live in remote rural villages. E-mental health is an acceptable option for people wishing to avoid stigma or who prefer to take an active role in their recovery (Wells et al., 2006). Likewise, ICT-based interventions have the potential to substantially reduce individual distress and the disease burden (Christensen & Hickie, 2010).

ICT-based interventions encompass different care modalities such as internet-based interventions, mobile-based interventions, social media-based interventions, virtual reality, and serious games (Mohr et al., 2014).

Various randomized clinical trials, systematic reviews, and meta-analyses show that this type of intervention is efficient and effective in the treatment of a wide range of disorders such as depression, anxiety, panic attacks, eating disorders, and substance use, among many other conditions (Andersson & Cuijpers, 2009; Richards & Richardson, 2012; Riper et al., 2011; Robinson & Serfaty, 2008).

In Latin America, both the number of publications on the subject and the supply of care services through ICT are low. However, a significant growth of web-based interventions is predicted for the coming years due to the rapid increase in Internet use. The Mexican Internet Association (AMIPCI) reported that in 2016 there were approximately 65 million Internet users, which increased to 82.7 million by 2019, with an annual growth rate of 6.0% vs. 4.3%. Connection time also increased during that period, from 7:14 h a day to an average of 8:20 h. The main online activities include connecting to social networks (79% vs. 82%), sending and receiving email (70% vs. 77%), sending and receiving instant messages (68% vs. 78%), and information seeking (64% vs. 76%) (AMIPCI, 2016, 2019).

Despite this growth, the information available on the supply and demand of e-mental health services in Mexico is limited. A first step towards the dissemination of the latter is to understand the factors that can influence the acceptability and adoption of ICT in the daily lives of mental health professionals and users of these services (Lal & Adair, 2014). This will facilitate the digital literacy process, through the significant appropriation of the intellectual, social, and ethical skills required to interact with information and recreate it in a critical, independent way. The goal of literacy is to develop in each person the ability to act and participate in an autonomous, cultured, and critical way in cyberspace. This is a right and a need of all citizens in the information society (Area and Pessoa, 2012).

Some studies suggest that low ICT acceptability is linked to the fact that users and professionals perceive greater complexity in the use of e-mental health resources compared to traditional resources; this may be because of the limited information on the matter and low frequency of use despite the interest in the subject (Carper et al., 2013; Molfenter et al., 2018).

Various studies have explored the acceptability of ICT use in mental health care. For example, researchers have assessed the acceptability of web-based cognitive-behavioral therapy (CBT) (Apolinário-Hagen et al., 2017; McCall et al., 2020; Friesen et al., 2014), computer-based CBT (Stallard et al., 2010; Fleming & Merry, 2013; Vigerland et al., 2014), and the use of various technologies applied to patient care (Entenberg et al, 2019; Schuster et al., 2020).

There are various theoretical models explaining why a person does or does not use a certain technology, such as the Theory of Reasoned Action (TRA, Fishbein & Ajzen, 1980), the Technology Acceptance Model (TAM, Davis, 1989), the Extension of the Technology Acceptance Model (EMAT, Venkatesh & Davis, 2000), the Diffusion of Innovation Model (DIM, Rogers, 1983), and the Unified Theory of Acceptance and Use of Technology (UTAUT, Venkatesh, et al., 2003); however, most studies that have addressed this issue fail to specify the theoretical underpinning or model on which they rest (Curioso et al., 2011; Entenberg et al., 2019; Friesen et al., 2014; Fleming & Merry, 2013; Stallard et al., 2010; Vigerland et al., 2014).

However, some studies evaluate the acceptability of different ICT-based interventions with instruments based on a theory of technology acceptance, such as the one by Kohnke and et al., (2014), in which the acceptability of telemedicine is explored through a questionnaire undertaken under UTAUT. The study reports eight factors: (1) performance expectation, (2) effort expectation, (3) social influences, (4) facilitating conditions, (5) attitude, (6) anxiety, (7) self-efficacy, and (8) behavioral intention. Another finding reported by the authors is that when physicians have a positive attitude, a high perception of self-efficacy, and intend to use the equipment, there is a high probability of promoting the technology. Saigi-Rubió et al. (2016) evaluated the acceptability of telemedicine through the components of the TAM. This study examined five factors: (1) perceived usefulness, (2) ease of use, (3) security and confidentiality, (4) ICT use profile, and (5) subjective norm. However, the psychometric properties of the instrument used were not reported. Békés and Aafjes-van Doorn (2020) recently evaluated psychotherapists’ attitudes towards web-based interventions during the COVID-19 pandemic using the UTAUT.

As can be seen, there are very few instruments on the acceptability of ICT in clinical practice to mental health professionals. Against this background, the objective of the study was to evaluate the psychometric properties of the Acceptability of ICT Use for Mental Health Care Questionnaire.

Hypothesis 1: The instrument will have medium to high internal consistency in each factor (Cronbach’s alpha > 0.70).

Hypothesis 2: As a function of the factorial structure, the instrument will have factor loads greater than 0.40 in each item and its main fitness indices will be sufficient (root mean square error of approximation [RMSEA] < 0.08, comparative fit index [CFI] > 0.90, and the Tucker Lewis index [TLI] > 0.90).

Method

Study Design

A non-experimental, descriptive, exploratory study was carried out to assess the internal consistency and validity of an instrument on the acceptability of ICT use in clinical practice among mental health professionals in Mexico.

Participants

A convenience sample was formed of mental health professionals (including psychologists, doctors, and social workers) who were contacted through mental health care institutions in Mexico and through social networks, who voluntarily agreed to participate and are currently employed, either at the institutions that agreed to collaborated in the administration of the instrument or independently in the public or private sector and were over 18 years old.

Instruments

A structured questionnaire including three sections was administered:

Sociodemographic data: It comprises nine questions that inquire about characteristics such as age, sex, education, profession, and marital status.

Internet use: This section was expressly created for this study. It comprises 16 questions that explore Internet use habits, such as place of connection, ease of access, and activities undertaken on the Internet, which has been used in other research (Do you know how to use the Internet? Do you use the Internet regularly? Where do you connect to the Internet? How often do you carry out each of the following activities on the Internet?) (Tiburcio et al., 2018).

Acceptability of ICT Use in Mental Health Care Questionnaire: Items in the instrument were developed based on a review of the corresponding literature of the TRA, MAT, and UTAUT. The questionnaire comprises 39 items with Likert-type response options ranging from 1 = totally disagree to 5 = totally agree (Mondragón, 2017).

Procedure

Participants were contacted through two procedures, first through various institutions that provide mental health care (following approval by the directors of each institution and inviting professionals to participate in person) and second through social networks (Facebook and LinkedIn, where subjects were invited by email to find out about the research and to answer the questionnaire). In both cases, the objectives of the project were presented before the answers were obtained. Those interested in participating voluntarily provided their consent in writing or digitally and subsequently answered the questionnaire on their own. This process took an average of 15 min.

Statistical Analysis

A frequency analysis was performed to describe the sociodemographic data of the participants. To evaluate the discrimination of each of the items, the distribution of responses, bias and kurtosis of each item were analyzed, and the discrimination power of extreme groups was also analyzed.

The Bartlett sphericity test and the Kaiser-Meyer Olkin (KMO) test of adequacy were performed to determine whether the data structure is adequate for the factor analysis. The values for sample adequacy in Barlett’s sphericity test must be less than 0.05 and the KMO greater than 0.80.

The sample was randomly divided into two to determine the factor structure of the instrument. An exploratory factor analysis (EFA) of maximum likelihood with VARIMAX rotation was performed with the first half (n = 242). Items with factor loads of less than 0.4 or more than 0.4 in two or more factors were eliminated. The analyses were performed using SPSS v.21.

Confirmatory factor analysis (CFA) was performed with the second half of the sample (n = 250) in STATA v. 14. The maximum likelihood estimation method was used, the goodness of fit indicators being: χ2/df ≤ 5 acceptable fit, ≤ 3 perfect fit, root mean square error of approximation (RMSEA) ≤ 0.10 weak fit, ≤ 0.08 good fit, ≤ 0.05 perfect fit, comparative fit index (CFI) ≥ 0.90 acceptable fit, ≥ 0.95 good fit, ≥ 0.97 perfect fit, Tucker Lewis index TLI (NNFI) ≥ 0.90 acceptable fit, ≥ 0.95 good fit (Hu & Bentler, 1999).

Cronbach’s alpha tests were used to analyze total internal consistency and that of each factor.

Results

Participant Characteristics

Around 73.4% of the respondents were women; the majority were aged between 21 and 30, and psychologists (88.2%) with graduate studies (50.4%) (see Table 1).

Table 1

Sociodemographic characteristics of health professionals

Data	Total n = 492%	Sample 1 n = 242%	Sample 2 n = 250%	X2
Sex
Male	26.6	28.5	24.8	0.867
Female	73.4	71.5	75.2	p = 0.352
Profession
Physician	4.7	3.3	6	2.715 p = 0.438
Psychologist	88.2	89.7	86.8
Social worker	2.8	3.3	2.4
Other	4.3	3.7	4.8
Academic
Undergraduate	48.8	47.5	50	1.351 p = 0.509
Graduate	50.4	52.1	48.8
Other	0.8	0.4	1.2
Marital status
Married	34.8	36.4	33.2	3.608 p = 0.546
Single	42.5	40.9	44
Divorced/separated	7.9	7	8.8
Partnered	13.4	14.9	12
Widowed	1.4	0.8	2
Age
21 to 30	32.1	31.8	32.4	0.848 p = 0.932
31 to 40	31.7	31	32.4
41 to 50	21.1	20.7	21.6
51 to 60	8.3	9.1	7.6
Over 61	6.7	7.4	6
Frequency of Internet use
Every day	93.3	92. 6	94	0.477 p = 0.788
2 to 3 times a week	5.7	6.2	5.2
Once a week	1	1.2	0.8
Connection time
Less than an hour	4.5	4.1	4.8	2.215 p = 0.696
An hour	8.7	7	10.4
2 to 3 h	28.3	28.5	28
4 to 6 h	28.5	28.5	28.4
Over 6 h	30.1	31.8	28.4
Use of the Internet as a resource
No	23.2	21.1	25.2	1.176
Yes	76.8	78.9	74.8	p = 0.278

Internet Use

Around 93.3% of the subjects reported connecting to the Internet daily, the majority (30.1%) for over 6 h a day; 76.8% mentioned using the Internet as a resource to communicate with their patients (see Table 1).

Exploratory Factor Analysis

Four items were eliminated from the discrimination analysis of each of the items (3. I do not think there is a direct interaction when I use ICT to provide service for my patients. 7. I have doubts about the professionalism of the health experts who use ICT in their work. 25. I have doubts about using ICT for dealing with my patients in case I make mistakes I cannot correct. 30. The lack of technological resources prevents me from using ICT in my work.) The value of the KMO test was 0.937, while that of the Bartlett sphericity test was 5596.992 (gL = 592, p < 0.001). These values indicated that the data structure was adequate to perform exploratory factor analysis.

In the first exploratory factor analysis, 10 items were eliminated because they had factor loads of over 0.40 in two factors (1. I think it would be helpful to use ICT in my work as a health professional 2. The use of ICT in my work enables me to serve a larger population 9. I think it is easy to use ICT in my work 12. I currently use ICT at work 14. I think using ICT in my work is easy 16. Using ICT in my work is justified, 17. I think I will incorporate the use of ICT into my work soon19. In general, in the place where I work, ICT are regarded as useful tools for professional practice 28. I think my colleagues and workmates would support my using ICT in my clinical practice 39. I think my boss would approve of me using ICT in my job). Two items were eliminated because they did not have factor loads of over 0.40 in any of the factors (15. There might be technical flaws that could interrupt the use of ICT in my work, 22. I need to learn to use ICT to incorporate them into my work, and lastly one item was eliminated because it did not form a factor 4. I have the necessary knowledge to use ICT in my professional practice). Twenty-two items were kept.

In the second exploratory factor analysis performed with 22 items, a factor structure of three factors was obtained: factor 1, Perceived usefulness (5, 6, 8, 10, 24, 26, 29, 31, 35, 36, and 38); factor 2, Perception risk (11, 18, 20, 23, 27, 32, and 37); and factor 3, Subjective work norm (13, 21, 33, and 34) that explained 56.23% of the total variance (see Table 2).

Table 2

AFE factor loads

	F1	F2	F3
Using ICT in health care is effective	0.737
6. I like working with ICT in my job	0.684
8. I think using ICT to treat mental health problems enhances patients’ recovery	0.724
10. I think using ICT in health care is innovative	0.717
24. I can use ICT anywhere to do my job	0.546
26. I use ICT in my daily life	0.522
29. I think ICT lets me work faster	0.690
31. Using ICT makes my job interesting	0.715
35. I think using ICT in mental health care is interesting	0.859
36. I would use ICT at work	0.768
38. I think using ICT in my job increases by productivity as a health professional	0.770
11. I feel insecure when I use ICT at work		0.696
18. I think that when you use ICT, there is no honesty between patients and the health professional		0.795
20. I think communication with my patients is poor when I use ICT at work		0.760
23. I think using ICT in health care is not very useful		0.880
27. I think that patient care through ICT is incomplete		0.607
32. I would feel anxious if I used ICT at work		0.719
37. I find using ICT at work rather intimidating		0.712
13. My boss uses ICT at work			0.650
21. My colleagues use ICT in their professional practice			0.775
33. I have all the necessary resources to use ICT at work			0.455
34. My colleagues use ICT at work			0.859

Confirmatory Factor Analysis and Reliability

The final evaluation of the model indicated the following goodness of fit measures:

χ2/gL = 443.38/206 = 2.15 (p = 0.001), CFI = 0.930, TLI = 0.921, and RMSEA = 0.068 (95% CI [0.059–0.077]). Factor one, perceived usefulness, included items 5, 6, 8, 10, 24, 26, 29, 31, 35, 36, and 38; factor two, perceived risk, comprised items 11, 18, 20, 23, 27, 32, and 37; while factor 3, subjective work norm, consisted of items 13, 21, 33, and 34 (see Fig. 1).

Fig. 1

CFA model

The internal consistency analysis indicated α = 0.94 for the total scale, α = 0.92 for the first factor, α = 0.87 for the second factor, and α = 0.81 for the third factor.

Discussion

The objective of the study was to validate an instrument to explore the acceptability of ICT use in clinical practice to a group of mental health professionals in Mexico. The result is an easily administered questionnaire with adequate psychometric properties, which provides valuable information for the development of awareness-raising strategies to encourage the use of ICT and confirms previous findings on the importance of this topic (Ebert et al., 2017).

It was found that the factor structure of the instrument showed congruence between the obtained factors (perceived utility, subjective norm, and risk perception) and the various determinants of ICT use proposed by the theories on which the study was based (Davis, 1989; Fishbein & Ajzen, 1980; Venkatesh et al., 2003). In other studies, considering MAT in health professionals, the main factors for implementing telemedicine were workplace support, perceived utility, and ease of use (Pereyra-Rodriguez et al., 2018). Acceptance of ICT in both users and doctors is a determining factor for the use of telemedicine as a care option, with the variables of perceived utility, ease of use, and attitude being the predominant variables (Tabares-Tabares et al., 2020). These characteristics are compatible with what was found in this research.

When Clough et al. (2019) developed an instrument to measure factors related to the behavior of health professionals regarding electronic interventions (eTAP-T), four factors were found in 12 items related to as follows: the opinion of their colleagues on the interventions, perceived control related to the use of electronic interventions in clinical practice, attitudes and beliefs of professionals towards electronic interventions, and finally the intentions of health professionals to implement online interventions. These findings are similar to the three factors developed in this instrument on perceived utility, use at work, and risk perception.

It is important to mention that the documentary research conducted to develop the questionnaire showed that one of the most widely used ways of evaluating the acceptability of ICT in the area of mental health is through qualitative methods (semi-structured interviews or focus groups) or surveys in which the theoretical framework of reference is not specified (Banna et al., 2010; Fleming & Merry, 2013; Friesen et al, 2014; Mira et al., 2009; Musiat et al., 2014; Stallard et al., 2010; Valenzuela et al., 2009; Vigerland et al, 2014). Although these types of studies provide relevant information to identify the issues that must be addressed in training programs, they require a great deal of time to process and analyze the data. The Acceptability of ICT Use Questionnaire therefore represents a cost-effective alternative for evaluating perceptions of this topic.

There is very little research that uses validated questionnaires as an evaluation method and is based on various theories of acceptance of technology as an explanatory reference for the intention of using telemedicine, information systems, telerehabilitation, cognitive-behavioral therapy through the Internet, online interventions, mHealth, video games, bibliotherapy, and computerized interventions (Kohnke et al, 2014; Madera et al., 2012, de Veer et al, 2015, Ernstmann et al., 2009; Jung & Loria, 2010; Saigi-Rubió et al., 2016; Zhang et al., 2015; Apolinário-Hagen et al., 2017). This research belongs to this last group and is relevant because it systematically investigates the acceptability of ICT use in mental health in Mexico, a topic that has not been explored in depth. It is therefore worth considering the specific barriers of health professionals to ICT use and viable options for reducing these obstacles.

Continuing this line of research is particularly important given that in Mexico, health professionals are already incorporating elements of technology into their clinical practice, either for monitoring the progress of their patients through electronic clinical files (Vázquez, Martínez, Blázquez & Castañeda, 2011) or by using online programs recently developed in Mexico such as Help for Depression (Lara et al., 2014), the Help for Drug Abuse and Depression Program (Tiburcio et al., 2016, 2018), Drink Less (Schaub et al., 2018), and Help for Relatives of Alcohol and Drug Users (Natera et al., 2016). This study is also relevant since due to the COVID-19 pandemic, the use of psychological therapies through the Internet has increased and it is important to incorporate an element that studies acceptability to improve the implementation of these resources.

One of the limitations of the study is the difficulty of generalizing results because the sample is not representative. Future research should include other groups of professionals with different levels of experience in the use of ICT as a treatment and prevention option.

Another limitation is that it failed to consider the degree of digital literacy of the subjects, since, although the data might suggest that a large sector of the Mexican population has access to the Internet, this does not mean that everyone uses it correctly, or that they have the digital skills required to provide psychological care through the Internet.

Conclusion

It would be useful to expand this area of research through the study of the acceptability of the use of e-mental health strategies such as computerized cognitive behavioral therapy, evaluation questionnaires, screening and diagnosis, social support, behavioral activation, psychoeducation, self-control, activation monitoring, and sleep management.