Development of the eTAP-T: A measure of mental health professionals' attitudes and process towards e-interventions.

BACKGROUND: The development of technological applications within psychotherapy has opened up new opportunities for mental health professionals (MHPs) to address client need. Despite the clinical efficacy and utility of evidence-based electronic interventions, MHPs' engagement with these interventions remains poorly understood.
OBJECTIVE: The aim of the current study was to develop and conduct a preliminary psychometric investigation of the measurement properties of the electronic-therapy attitudes and process questionnaire - therapist version (eTAP-T). Based upon the theory of planned behaviour (TPB), the eTAP-T measures factors related to MHPs' engagement with e-interventions for clients' mental health concerns.
METHODS: Participants were 222 practicing MHPs who reported being in direct contact with clients. Participants completed the eTAP-T and related measures with a subsample of 40 participants completing a two-week follow up questionnaire.
RESULTS: Exploratory factor analysis with item reduction resulted in a 12-item eTAP-T, with four factors accounting for 82% of variance. The four factors (subjective norms, perceived behavioural control, attitudes and intentions) were consistent with the four TPB domains. The eTAP-T demonstrated satisfactory validity and reliability as per the consensus-based standards for the selection of health measurement instruments.
CONCLUSIONS: The development and preliminary psychometric investigation supported the validity and reliability of the eTAP-T. Further research is required for confirmatory analyses. The eTAP-T may be useful in identifying the training needs of MHPs and evaluating training programs. Specific areas for intervention, such as attitudes or perceived credibility may be identified and targetted, with the measure then also used to evaluate change across these domains. It is anticipated that the eTAP-T may useful tool in improving uptake of digital interventions by MHPs.

Introduction

Mental health professionals (MHPs) are increasingly being encouraged to utilise novel methods to enhance, complement, and increase dissemination of psychological approaches to prevention, assessment, intervention, and psychoeducation (Lal and Adair, 2014; Sucala et al., 2012; WHO, 2018). Certain electronic-interventions (e-interventions) are evidenced to overcome barriers to mental healthcare by providing clients with accessible, equitable and flexible options for care that are tailored to clinical presentations (Casey et al., 2013; Iacono et al., 2016). e-Interventions have the capacity to overcome stigma associated with mental health treatment, thus increasing help seeking behaviours (Perle et al., 2013), as well as the dissemination and efficacy of existing interventions (Arnberg et al., 2014; Casey et al., 2014; Weisel et al., 2018). Despite considerable research supporting the efficacy and advantages of such approaches, considerably less research has focused on understanding factors influencing therapist uptake of e-interventions.

Numerous terminologies exist within the field of e-interventions. For the purposes of the current research, e-interventions will be defined as any electronic technology or medium used to aid in either the diagnosis, treatment, monitoring and/or management of mental health related problems (Lal and Adair, 2014; Sucala et al., 2013). Previous research has demonstrated the clinical efficacy of e-interventions across a range of disorders (Biagianti et al., 2017; Fairburn and Patel, 2017; Hedman et al., 2012; Musiat and Tarrier, 2014) and with varying degrees of therapist support. Treatment effects for guided versus unguided support for e-interventions range from large (therapist-supported d = 0.78), to moderate (administrative supported d = 0.58), to small (studies that had no support for computer-based treatments d = 0.36) (Andersson et al., 2014; Cuijpers et al., 2010). Research into consumer attitudes also supports the readiness and willingness of consumers to engage with e-interventions (Berle et al., 2015; Boschen and Casey, 2008; Casey et al., 2013; Casey et al., 2014; Clough and Casey, 2011; Shalom et al., 2015). Despite the efficacy and advantages of e-interventions, substantial barriers exist that often limit MHPs' referral, implementation and dissemination of e-interventions (Carper et al., 2013; Mohr et al., 2017; Mohr et al., 2015; Reynolds et al., 2015). This is problematic to the field given the success of many e-interventions is reliant upon MHPs' involvement and engagement. The clinical efficacy and utility of these interventions can be influenced by the ability of MHPs' to locate and assess the evidence-base of e-interventions, engage in ongoing education and training in e-interventions, and demonstrate professional practice competencies with e-interventions. This is particularly the case for the adjunctive use of e-interventions, as is recommended by stepped care approaches to treatment. Furthermore, only limited research has quantitatively examined the factors that influence MHPs engagement with e-interventions in clinical practice (Godin et al., 2008; Lustgarten & Elhai, 2018).

MHPs engagement with e-interventions

Current evidence indicates that MHPs typically perceive face-to-face therapy as being superior to e-interventions, despite effect sizes for some e-interventions rivalling that of traditional therapies (Andersson et al., 2014; Cuijpers et al., 2010; Meurk et al., 2016). MHPs are more likely to endorse e-interventions as an adjunct to face-to-face interventions than for standalone treatment (Mora et al., 2008; Sinclair et al., 2013; Vigerland et al., 2014), and support its application for prevention and intervention with mild-to-moderate mental illnesses, as opposed to severe presentations (Schröder et al., 2017; Sinclair et al., 2013; Stallard et al., 2010; Vigerland et al., 2014). Barriers to MHPs' enagagement consist of: knowledge deficits and limited awareness of e-interventions (Donovan et al., 2015); misconceptions about cost, clinical efficacy and clinical utility of e-interventions (Ashurst et al., 2012; Mora et al., 2008; Sucala et al., 2013); the perceived inability to form a working alliance with clients when using e-interventions (Cook and Doyle, 2002; Kiluk et al., 2014); and deficits in confidence and competence in using e-interventions (Bennett-Levy et al., 2017; Sucala et al., 2013). In addition to these factors, MHPs may have concerns about the appropriate and ethical use of e-interventions, such as concerns regarding ongoing monitoring of clients or use for specific disorders or symptom severities (e.g., Meurk et al., 2016; Wells et al., 2007; Yellowlees et al., 2012). It is certainly the case that e-interventions are not appropriate for all client presentations, and as such one of the challenges within this area may concern therapists' confidence and capacity to identify clients who are appropriate for these interventions, and then selecting interventions to meet the client's presentation and monitoring needs.

It is difficult to ascertain a broad view of MHPs' intentions to engage with e-interventions due to inconsistent findings in the few studies available within the literature. Previous studies focus mostly on consumers (Casey et al., 2014; Clough et al., 2019a; Klein & Cook, 2010), are qualitative (Berry, Bucci, & Lobban, 2017; Schneider, Bolier, de Vries, & van Osch, 2016), and have been argued to lack a sound theoretical approach (Clough et al., 2016). Few quantitative studies have investigated MHPs' intentions to engage with e-interventions, with most focusing solely on attitudes (Bruno & Abbott, 2015; Donovan et al., 2015; Perle et al., 2013; Schröder et al., 2015). Also common to research in this field is the lack of validated tools to understand MHPs' behaviours, intentions, and associated constructs (De Grood et al., 2016; Lal and Adair, 2014; Meurk et al., 2016). Existing scales that measure factors related to MHPs' engagement behaviours with e-interventions are limited, atheoretical, and lack appropriate psychometric investigation and validation (Clough and Casey, 2015a, Clough and Casey, 2015b; Mohr et al., 2017; Mohr et al., 2015).

The theory of planned behaviour

The theory of planned behaviour (TPB) has been posited as a useful model for understanding patient engagement with e-interventions and may also prove useful in understanding MHP engagement (Clough and Casey, 2011; Wilson et al., 2013). Ajzen's (1991) TPB is a conceptual model for understanding behaviour and is an extension of the theory of reasoned action (Fishbein and Ajzen, 1977). The TPB (Fig. 1) maps decision making by explaining the complex relationships between beliefs, attitudes and behaviours (Ajzen, 2002), with behavioural enactment best predicted by behavioural intentions. The more positive an individual's intentions are, the more likely the behaviour to occur. Application of the TPB in measuring MHPs' behaviours towards e-interventions would therefore be gauged by a behavioural intent (e.g., ‘I intend to use e-interventions with my clients’).

Fig. 1

Theory of planned behaviour (Ajzen, 1991).

Behavioural intentions are influenced by three constructs; subjective norms (perceptions of the opinions and views of others), perceived behavioural control (PBC; self-efficacy or confidence) (Ajzen, 1991, Ajzen, 2002, Ajzen, 2011), and behavioural attitudes (positive or negative evaluations). Beliefs are the foundation of each underlying predictor of intention. That is, individuals' behavioural beliefs are comprised of attitudes towards a behaviour, which together with their subjective norms, and PBC, form behavioural intentions. Together these predict whether individuals will perform the target behaviour. Godin et al.'s (2008) systematic review of 56 studies found that the TPB was the most widely applied theory to explain health professionals' behaviour. Further, the TPB was significantly better at predicting health professionals' behaviour than other theories, such as the technology acceptance model and the theory of interpersonal behaviour (Z = 6.085, p < .001) (Godin et al., 2008).

Measures of engagement with e-interventions

Beyond a theory driven approach there is also a need for validated measures of the factors that influence engagement with e-interventions. Existing measures consist of Wangberg et al.'s (2007) attitude towards e-Therapy scale, Davis, 1989, Davis, 1993 perceived usefulness and perceived ease of use scales, Schröder et al.'s (2017) attitudes towards psychological online interventions questionnaire in health care professionals, and Clough et al.'s (2019), electronic-therapy attitudes and process questionnaire (eTAP). As previously mentioned, these measures focus primarily on attitudes and are atheoretical (Schröder et al., 2017; Schröder et al., 2015; Wangberg et al., 2007). Theoretically grounded measures of engagement are scarce, however Davis, 1989, Davis, 1993 perceived usefulness and perceived ease of use scales are grounded in the technology acceptance model while Clough et al.'s (2019) measure is grounded in the TPB. Davis, 1989, Davis, 1993 measures are limited in that the technology acceptance model focuses exclusively on attitudinal aspects in the prediction of behavioural intentions. In the context of the current study, it is argued that the TPB provides a more suitable theoretical model in measuring MHPs' engagement behaviours as it includes social variables associated with behavioural control and differentiates between internal (e.g. skill) and external (e.g. time) control factors. As the TPB can tap into control variables for each independent situation, it may be more likely than the TAM to capture situation-specific factors.

The TPB also provides a broad model to understand the process-based factors that may influence a MHP's decision to initiate and continue use of e-interventions in their practice. Similar to those relevant in client focussed research, early findings in this area suggest that process variables such as therapists' perceptions of treatment credibility, outcome expectations, concerns regarding working alliance, and self-efficacy to engage in therapeutic activities/modalities may be important determinants of engagement in this field (Greene et al., 2010; Perle et al., 2013; Schröder et al., 2015). These process variables may be able to be conceptualised as belonging to the TPB factors of attitude, PBC, and subjective norms, although a theoretical approach to understanding MHP engagement is lacking within the field.

Study aim and hypotheses

The aim of the current study was to develop and conduct a preliminary psychometric investigation of a brief measure of factors related to MHPs' engagement with e-interventions. Given a TPB-based client measure (eTAP) has previously demonstrated strong psychometric properties pertaining to structural, convergent, divergent, and predictive validity (Clough et al., 2019), it was decided that this measure would form an appropriate basis for adaptation and development for use among MHPs.

The electronic-therapy attitudes and process questionnaire – therapist version (eTAP-T) was developed in accordance with COSMIN guidelines (Mokkink et al., 2010). Items were developed based on a comprehensive review of the literature, feedback from a panel of experts within the field, and feedback from a panel of MHPs. Preliminary investigation of the scale's psychometric properties was conducted. Specifically, of interest was the scale's internal consistency, and structural, convergent, and divergent validity. It was hypothesized that: an exploratory factor analysis of the eTAP-T would reveal a four-factor structure with item loadings on factors that corresponded with the appropriate TPB domains; the eTAP-T would demonstrate satisfactory estimates of internal consistency and test-retest reliability and that this would be comparable with the original eTAP questionnaire; and that the eTAP-T subscales would demonstrate adequate convergent and divergent validity with measures assessing related constructs (Table 1).

Table 1

Validity predictions of eTAP-T subscales with related measures.

	Attitudes	Subjective norms	Perceived behavioural control	Intentions
Attitude towards e-Therapy scale (Wangberg et al., 2007)	Convergent	–	–	–
Perceived usefulness scale (Davis, 1989)	Convergent	–	–	–
Perceived ease of use scale (Davis, 1993)	Convergent	–	Convergent	–
Evidence-based practice attitudes scale (Aarons et al., 2010)	–	Convergent	–	Convergent
eHealth literacy scale (Norman and Skinner, 2006)	–	–	Convergent	–
Burnout measure – short version (Malach-Pines, 2005)	–	–	–	Convergent
Toronto mindfulness scale (Lau et al., 2006)	Divergent	Divergent	Divergent	Divergent
The social desirability scale – 17 (Stöber, 1999, Stöber, 2001)	Divergent	Divergent	Divergent	Divergent

Method

Participants and recruitment

Minimum sample size was determined using the best practice guidelines for the development of health-related scales outlined by the consensus-based standards for the selection of health measurement instruments (COSMIN; Mokkink et al., 2010). COSMIN guidelines directed the a priori determination of minimum sample size using a participant-to-item ratio of 7:1. This heuristic required a minimum sample size of 217 participants, based on seven times the number of eTAP-T items (7 × 31 = 217). Three hundred and ninety-three individuals responded to invitations to participate in the study. The final sample comprised of 222 participants. Participants were excluded if they had less than 1 h of direct contact with clients per week (n = 11), did not identify as a MHP (n = 9) (e.g. emergency nurse, speech pathologist, paramedic), or if sufficient data was not completed (n = 151) (e.g. participants that did not progress past the initial demographic questions). The final sample was mostly female (n = 180, 81.1%) and aged between 21 and 74 years (M = 48.1, SD = 13.3). The majority of respondents were psychologists (n = 62, 27.9%), working an average of 30 hours per week (M = 30.4, SD = 12.7), and were engaged in approximately 20 hour client contact per week (M = 19.9, SD = 10.4). Additional sample characteristics are displayed in Table 2.

Table 2

Participant characteristics.

Characteristics	N = 222
Age, mean (SD)	48.1 ± 13.3
Gender
Female	180 (81.1%)
Education
Postgraduate	163 (73.4%)
Undergraduate	51 (23.0%)
Diploma	8 (3.6%)
Profession
Psychologist	62 (27.9%)
Mental health nurse	44 (19.8%)
Counsellor	38 (17.1%)
Occupational therapist	29 (13.1%)
Social worker	19 (8.6%)
Music therapist	6 (2.7%)
Psychiatrist	3 (1.4%)
General practitioner	1 (0.5%)
Other	20 (9.1%)
Registration
Full	198 (89.2%)
Provisional	4 (1.8%)
No registration status	20 (9.0%)
Therapeutic approacha
Cognitive behavioural therapy	79 (35.6%)
Psychodynamic therapy	19 (8.6%)
Acceptance and commitment therapy	27 (12.1%)
Dialectical behaviour therapy	10 (4.5%)
Emotion focused therapy	14 (6.3%)
Humanistic psychotherapy	16 (7.2%)
Other	57 (25.7%)
Using digital interventions with clients	134 (60.4%)

Therapeutic approach most frequently employed by participants.

Participants were recruited through convenience and snowball sampling. Invitations to access an online survey were disseminated using profession-specific social media (e.g. Facebook), websites (e.g. Australian Psychological Society), and email (e.g. directories).

Materials and measures

Development of the eTAP-T

The eTAP-T was designed, developed and adapted from both the therapy attitudes and process questionnaire (TAP) (Clough et al., 2016) and eTAP (Clough et al., 2019). Both scales have demonstrated strong psychometric properties, including four factor structural validity (consistent with the underlying TPB model), high internal consistency (α ranging from 0.88 to 0.94, and 0.78 to 0.94 respectively), and good test-retest reliability (0.65 to 0.80, and 0.67 to 0.76 respectively). Consistent with the eTAP, the eTAP-T went through a staged item generation process.

The expert/target population panel was combined given members had experience and expertise with the TPB and were in current contact with their clients. The expert/target population panel was comprised of seven MHPs selected based on their profession and familiarity with the TPB. The experts consisted of a Mental Health Nurse, two general Psychologists, three Clinical Psychologists, and an Occupational Therapist. The questions posed in the expert/target population panel were based on guidelines for developing TPB questionnaires (Fishbein & Ajzen, 2011). The panel were asked to read and respond to the 38-items and provide qualitative feedback surrounding item wording, item modification and relevance to the TPB construct. Consistent with development procedures of the TAP and eTAP, experts independently ranked the top four items (from the adapted 31-item eTAP and 7 items sourced from the literature) they believed best addressed each of the TPB constructs. Consensus was reached by selecting the four highest expert ranked items within each domain (4 × 4 = 16). These items were included with the 16 modified eTAP items (total of 32-items). Prior to psychometrically investigating the 32-items, qualitative feedback elicited from the expert review supported removal of one item on the premise that it was ambiguous, double-barrelled, and was not associated with the intended TPB construct. Experts recommended the change in terminology from ‘online interventions’ in the eTAP to ‘digital interventions’ in the eTAP-T to ensure MHPs' clarity in recognizing the target construct. As such a 31-item initial eTAP-T was put forward for psychometric evaluation. In line with the study aim, several established scales with sound psychometric properties were included to test the psychometric properties of the eTAP-T.

Validity measures

The convergent and divergent validity of the eTAP-T were examined using relevant measures that were associated with each TPB construct. Attitudes towards e-interventions were measured using Davis' perceived ease of use and perceived usefulness scales (1989; 1993), (α = 0.98 and α = 0.94, respectively) and the attitude towards e-Therapy scale (α = 0.77) (Wangberg et al., 2007). The requirements subscale of the evidence-based practice attitudes scale (Aarons et al., 2010), was used to measure feelings towards evidence-based practice in the context of therapy, intervention, and treatment. Internal consistency of the subscales of the evidence-based practice attitudes scale ranges from 0.67 to 0.91 (total scale α = 0.78). The eHealth literacy scale (Norman and Skinner, 2006), was used to measure electronic health literacy and user knowledge, comfort, and skill in evaluating, sourcing, and applying electronic health information to health problems (α = 0.88). Burnout was measured using the short version of Malch-Pines' (2005) burnout measure (α = 0.89). The Toronto mindfulness scale (α = 0.95) was used as a measure of an individuals' capacity to invoke a mindful state (Lau et al., 2006). The social desirability scale – 17was used as a measure of frequent and infrequent socially desirable behaviours (α = 0.72) (Stöber, 2001).

Research design

A cross-sectional, within-participants design was employed, using both factor analytic and correlational approaches, with a focus on patterns of within-participants variance.

Procedure

Ethical clearance was obtained from the affiliated university prior to the commencement of the study. Information about the study was provided online and participants were informed that their consent would be implied should they progress past the cover page of the online survey. Participants responded to a series of demographic questions, the eTAP-T items, and then the additional measures in the order outlined in Table 1. The average time to complete the survey was 30 mins. Participants had the option to opt-in to be contacted for a follow-up survey two weeks after completing the main survey. Participants consenting to this process provided a contact email address in a separate survey to their main responses and were informed that their identifying information could not be linked to their responses. The follow-up survey was matched with the initial survey using a participant generated code. The follow up survey took approximately 5 minsto complete and consisted of demographic questions, the eTAP-T items, and questions relating to use of digital technologies in practice since completion of the initial survey.

Results

Data screening

Inspection of the data indicated small deviations from normality with several negatively skewed variables. These deviations were not deemed problematic given the robust nature of exploratory factor analysis (Tabachnick et al., 2013). Scatterplots were examined, with spot checks revealing linear relationships and absence of curvilinearity. No univariate outliers were identified. Further screening revealed 10 multivariate outliers with a Mahalanobis distance greater than the critical χ2 cut-off. Removal of the outliers produced no substantial differences in the results and appeared to not impact inferential decisions, and as such these cases were retained. Correlations of the variables suggested presence of multicollinearity. Further examination revealed one variable with a bivariate correlation of 0.94, a tolerance value below 0.10, and a condition index greater than 30. After considering the findings and guidelines suggested by Tabachnick et al. (2013), the problematic variable (“I find digital interventions for my clients' mental health to be?”) was removed as it unduly influenced the factorability of the data.

Data analysis

Given the early stages of development, an exploratory factor analysis (EFA) was performed on the 30-item eTAP-T, as a confirmatory analysis was deemed premature (Tabachnick et al., 2013).

Initial factor analysis

Principal axis factoring was selected due to its capacity to handle data that deviates from normality and contains moderate to high correlations between items (Pett et al., 2003; Tabachnick et al., 2013). Several well-recognised heuristics were applied to determine factorability of the eTAP-T. Inter-item correlations, communalities of items (greater than 0.30), Kaiser-Meyer-Olkins' (KMO) measure of sampling adequacy (0.94), and a significant Bartlett's Test of Sphericity (χ2 (435) = 6630.38, p < .001) suggested that the variables were related and factorable.

Factor extraction and retention

Debate and disagreement exist on how best to determine factor structure in EFA (Costello and Osborne, 2005; Mokkink et al., 2010). Given this lack of consensus, several strategies were sourced from the literature to determine factor retention in the initial principal axis factoring. Using the Kaiser criteria (K1), principal axis factoring extracted six factors with eigenvalues greater than 1.0. Catell's (1966) scree-test was conducted by inspecting the scree plot. The scree plot revealed a slight break after the fourth, fifth and sixth factors, with no obvious ‘break’ point. The ‘percentage of total variance explained rule’ suggests retaining factors based on a ‘cut-off’ where the next factor in the model contributes less than 5% of additional variance (Gaskin and Happell, 2014). This heuristic supported a four-factor solution for the eTAP-T. Each of the four, five, and six factor solutions were inspected. The four-factor solution presented the most parsimonious and theoretically meaningful solution, with fewest item cross-loadings, and was consistent with the total variance explained rule. As such, based on the pattern matrix, previous research, and the theoretical underpinnings of the eTAP-T, the decision was made to retain four factors. The four-factor solution explained a total of 65.97% of the total variance, and 49.99%, 6.94%, 4.94% and 4.10% of the variance respectively. The rotated four-factor solution presented a simple structure, with all but two variables loading to a single factor (see Appendix A). One item (item 6) had a communality less than 0.30. Four items (items 3, 4, 5 and 16) resulted in factor loadings greater than 1.00. These loadings were not considered problematic given the assumptions of the analysis were met and an oblique rotation applied (Jöreskog, 1999; Tabachnick et al., 2013).

Reducing the item pool

Construction of a shortened scale was considered justified to facilitate ease of administration with MHPs (Cho et al., 2013). Therefore, the three items with the highest loadings on each factor were selected to progress to final factor analysis and interpretation. Three items per factor was considered the briefest version of the scale possible, with less than three items per factor likely to adversely influence the stability of the eTAP-T (Tabachnick et al., 2013)

Final factor analysis

As a further check of factor structure, parallel analysis was performed for the final EFA (12 items). Parallel analysis confirmed presence of four factors, with the fifth factor containing an eigenvalue (λ = 0.03) that was less than the average eigenvalue (λ = 0.12) and the 95th percentile eigenvalue (λ = 0.17). The final scale was suitable to proceed to a final factor analysis based on sampling adequacy (KMO = 0.88), and Bartlett's Test of Sphericity (χ2 (66) = 2566.54, p < .001). Principal axis factoring with promax rotation revealed presence of a simple structure with emergence of four conceptually meaningful factors. The 12-item eTAP-T (Appendix B) explained 82.16% of the variance, with the factors one through four explaining 52.22%, 12.04%, 10.84% and 7.06% of variance, respectively. Item loadings and communalities are provided in Table 3. As shown in Table 4, all factors were significantly correlated either moderately or highly with the other factors.

Table 3

Promax rotated factor structure and communalities of the 12-item eTAP-T.

Item	F1	F2	F3	F4	Communalities
16. My peers would support my use of digital interventions with my clients:	0.97				0.85
17. My colleagues would support my use of digital interventions with clients:	0.93				0.92
19. My colleagues would approve of my use of digital interventions with my clients:	0.87				0.85
11. I am confident in my ability to use digital interventions with clients:		0.95			0.88
14. I am confident in my ability to use technology to deliver digital interventions to clients:		0.91			0.85
15. I possess the required technical knowledge to use digital interventions for my clients' mental health:		0.88			0.75
3. I find the use of digital interventions for clients' mental health to be: unpleasant/pleasant			0.83		0.76
4. I find the use of digital interventions for clients' mental health to be: harmful/beneficial			0.97		0.94
5. I find digital interventions for clients' mental health to be: not credible/credible			0.98		0.89
28. I will learn more about digital interventions:				0.91	0.83
29. I intend to make time to learn about digital interventions within the next week:				1.00	0.90
30. I intend to check with my clients whether they would like to use digital interventions to address their mental health concerns:				0.45	0.45

Table 4

Factors intercorrelations for the 12-item eTAP-T.

Factor	Subjective norm	PBC	Attitude	Intention
Subjective norm	1.00	0.49	0.60	0.36
PBC		1.00	0.55	0.42
Attitude			1.00	0.54
Intention				1.00

Factor interpretation

The four factors were theoretically aligned with the four constructs of the TPB (Ajzen, 1985). Factor 1, subjective norms, loaded three items that focused on MHPs' perceptions of what their peers or colleagues would think about their use of e-interventions. Factor 2, PBC, consisted of three items that focused on MHPs' perceived control over their e-intervention use in clinical practice. Factor 3, attitude, loaded three items that focused on MHPs' attitudes and beliefs towards e-interventions. Factor 4, intentions, loaded three items relating to MHPs' intentions to engage with e-interventions in their practice.

Reliability analyses

Internal consistency

Excellent internal consistency for the total scale (α = 0.91) and good to excellent internal consistency for the subscales were found (Subject Norm α = 0.95, PBC α = 0.93, Attitude α = 0.95, and Intention α = 0.86).

Test retest reliability and standard error of measurement (SEM)

Test retest reliability was analysed by way of Intraclass Correlation Coefficients (ICCA, 2), using the absolute agreement definition, and a 2-way random effects ANOVA model of participant responses by time point. The ICCA, 2 for the total scale was acceptable (0.72, SEM = 0.72), with the subscales ranging from acceptable to excellent (subjective norms = 0.51; attitudes = 0.61, SEM = 0.97; intentions = 0.64 SEM = 1.66; PBC = 0.85, SEM = 1.06).

Validity analyses

Convergent and divergent validity

Convergent validity was assessed for all scales of the eTAP-T separately. Support for the convergent validity of the eTAP-T scales was found with significant moderate positive relationships between: the attitudes subscale and the attitudes towards e-Therapy scale, and both of Davis, 1989, Davis, 1993, perceived usability and perceived ease of use scales; the evidence-based practice attitudes scale (Aarons et al., 2010) and subjective norms and intentions subscales; and the eHealth literacy scale (Norman and Skinner, 2006) and the perceived ease of use scales with the PBC subscale. The weak, divergent relationship between the Intentions subscale and the burnout measure – short version (Malach-Pines, 2005) was not significant. Each subscale of the eTAP-T demonstrated convergent validity with at least one other related measure. Divergent validity was also assessed for the eTAP-T and subscales separately. Support for the divergent validity of the eTAP-T scales was demonstrated in the weak and non-significant correlations between the eTAP-T scales with both the Toronto mindfulness scale (Lau et al., 2006) and the social desirability scale-17 (Stöber, 2001) (Table 5).

Table 5

Pearson correlations for the eTAP-T and subscales, with related measures (n = 181).

	Subjective norms	PBC	Attitudes	Intentions	eTAP-T total
Attitude towards e-Therapy scale	–	–	0.46⁎⁎	–	–
Perceived usefulness scale	–	–	0.55⁎⁎	–	–
Perceived ease of use scale	–	0.54⁎⁎	0.33⁎⁎	–	–
Evidence-based practice attitudes scale	0.31⁎⁎	–	–	0.30⁎⁎	–
eHealth literacy scale	–	0.42⁎⁎	–	–	–
Burnout measure – short version	–	–	–	0.01	–
Toronto mindfulness scale	−0.01	0.15	0.11	0.08	0.11
The social desirability scale – 17	0.06	0.10	−0.02	0.11	0.09⁎

p < .05.

p < .01.

Discussion

The current study aimed to develop and conduct a preliminary psychometric investigation of the properties of a brief measure of MHPs' engagement with e-interventions in clinical practice. To ensure a sound theoretical basis for the new measure, an existing client-based measure, developed according to the TPB, was adapted for a MHP sample. Clinical and research experts informed the adaptation and modification of items. Methodological rigor in development of the eTAP-T was further ensured by applying the best practice for recommendations for the development of health questionnaires (Francis et al., 2004). Overall, the results supported the validity and reliability of the newly developed eTAP-T.

Investigation of the psychometric properties revealed a four-factor structure that aligned with the four factors of the TPB (Ajzen, 1985). The factors explained a total of 82% of the variance in scores. Consistent with the TAP and eTAP, the eTAP-T revealed strong internal consistency, with subscales ranging from good to excellent. Correlational analyses showed that the total scale demonstrated acceptable test-retest reliability over time, however findings from the test-retest reliability of the four subscales were mixed. In light of conservative estimates of error for two-week test-retest reliability, the stability of the subscales over time ranged from adequate to excellent. Estimates of reliability were comparable to other self-report TPB-based measures of therapy engagement (Clough et al., 2019b; Clough et al., 2016).

Almost all predictions of convergent and divergent validity of the eTAP-T subscales were supported. No convergence was seen between professional burnout and the Intentions subscale. This finding was contrary to predictions and suggests that lower levels of burnout did not result in greater intentions to use e-interventions. The lack of association between burnout and e-interventions intentions could be explained by the lack of variance and spread in scores on the burnout measure. Few high scores on the scale suggested that burnout was not as prevalent within the sample (9.85% of participants), as previously seen in research within this field (Johnson et al., 2017; Kim et al., 2018; Shanafelt et al., 2017).

Limitations and future directions

While the current study has various strengths, these should be considered with reference to some limitations. Firstly, the COSMIN guidelines were applied to guide the development of the eTAP-T. However, not all criteria outlined in the COSMIN guidelines were investigated (Mokkink et al., 2010). Of the 10 criteria, the present study did not evaluate the predictive, responsiveness or cultural validity of the eTAP-T. Future research should seek to measure the predictive capacity and responsiveness of the eTAP-T with a diverse sample of MHPs. Secondly, parallel analysis was performed in an attempt to strengthen the structural validity of the eTAP-T by mitigating inaccurate factor retention decisions. While this technique is evidenced to be more accurate than others, and its application is widely supported within the literature (Costello and Osborne, 2005; Gaskin and Happell, 2014; Hayton et al., 2004; Tabachnick et al., 2013), future research should seek to confirm the factor structure of the eTAP-T with an independent sample. Finally, while the analyses of the eTAP-T were sufficiently powered, it could be argued that the online recruitment of participants may suggest presence of sampling bias. The current sample's high rate of MHPs' engagement with e-interventions was an unexpected finding and one that may offer promise for future implementation and dissemination research. The importance of this finding cannot be inferred though given the possibility that such rates were the result of participant self-selection to the study.

Future research should seek to overcome these limitations by confirming the factor structure and model fit of the eTAP-T using confirmatory analyses with an independent, representative and diverse sample of MHPs. Investigation of the suite of validity criteria is needed to ensure its applicability, efficacy, and alignment with the recently updated COSMIN guidelines (Mokkink et al., 2018). Extending on the current study's limitations, as is often needed by research within this field, future research needs to be intuitive, rapid and responsive to the climate of the e-interventions. Specifically, attention needs to be given to strategies that can support the needs of MHPs in making appropriate use of technologies.

Implications

The development of the eTAP-T and its associated findings provides several theoretical and practical implications. To the authors' knowledge, this is the first theoretically grounded and psychometrically sound instrument to measure factors related to MHPs' engagement with e-interventions. Barriers to engagement with adjunctive e-interventions (e.g. knowledge deficits in the clinical efficacy and utility of e-interventions) limit the ability of MHPs to fully meet the needs of clients (e.g. access to mental health services). The eTAP-T has the capacity to not only identify the training needs of MHPs, but may also serve as an evaluative measure for existing training programs. This capacity is particularly relevant given the growth of and funding given towards training programs in this area. One such initiative is the e-Mental Health in Practice (eMHPrac) training program, which is nationally funded in the Australian health context. We argue that the eTAP-T may prove useful in determining the needs of individuals prior to commencement of such training programs, but then also in evaluating individual level change and program efficacy over time.

The absence of appropriate measurement tools in this area has impeded the scientific assessment of training, education and competencies for practitioner use of e-interventions (Meurk et al., 2016; Reynolds et al., 2015). Extending on previous literature within this field, the eTAP-T suggests that MHPs' attitudinal beliefs and intentions to engage with e-interventions are different to that of traditional face-to-face therapies (Shalom et al., 2015; Sucala et al., 2013; Wentzel et al., 2016). The eTAP-T items were adapted and modified from existing TPB-based client measures of engagement with traditional interventions and e-interventions (Clough et al., 2019; Clough et al., 2016). However, none of the items sourced from the literature emerged in the final scale, suggesting that similar factors are relevant to both clients' and MHPs' engagement with e-interventions.

Conclusions

Measurement of the factors that influence MHPs' engagement with e-interventions is vital to informing and promoting a better understanding of the barriers and enablers to the uptake, dissemination, evaluation and implementation of evidence-based e-interventions. The eTAP-T provides a significant step towards addressing this need and may prove to be a valuable resource for clinicians, educators, and policy makers. We expect that it will have utility in these populations in the measurement of training needs, development of training programs, and evaluation of such programs. Ensuring that MHPs are adequately trained in the use of e-interventions will ensure appropriate support and referral is provided to clients for these tools, facilitating uptake and engagement in this emerging area of clinical practice.

Item	F1	F2	F3	F4	Communalities
1. I find the impact of digital interventions on clients' mental health to be: negative/positive	0.90				0.64
2. I find digital interventions for clients' mental health to be: bad/good	0.96				0.82
3. I find the use of digital interventions for clients' mental health to be: unpleasant/pleasant	1.00				0.81
4. I find the use of digital interventions for clients' mental health to be: harmful/beneficial	1.02				0.87
5. I find digital interventions for clients' mental health to be: not credible/credible	1.03				0.84
6. When using digital interventions, I believe that the personal information provided by clients is mostly: insecure/secure					0.27
7. Regardless of the severity of symptoms, digital interventions are typically: ineffective/effective	0.55				0.34
8. When using digital interventions with clients, a strong working alliance is: not achievable/achievable	0.51				0.43
9. When digital interventions are combined with face-to-face therapies, they can be: disadvantageous/advantageous	0.60				0.42
13. I think I can use digital interventions for clients' mental health:	0.35		0.40		0.69
23. I intend to continue using digital interventions with clients:	0.76				0.77
24. I intend to use digital interventions with clients in the next week:	0.50				0.60
25. I will use digital interventions with my clients:	0.69				0.80
26. It is likely that I will use digital interventions for my clients' mental health:	0.63				0.76
27. I will ensure that I have the resources to use digital interventions with my clients:	0.32				0.65
10. I have complete control over whether I use digital interventions with clients:		0.50			0.38
12. It is mostly up to me whether I decide to use digital interventions with clients:		0.41			0.28
16. My peers would support my use of digital interventions with my clients:		1.02			0.81
17. My colleagues would support my use of digital interventions with clients:		0.92			0.86
18. My professional body would approve of my use of digital interventions within my practice:		0.67			0.56
19. My colleagues would approve of my use of digital interventions with my clients:		0.98			0.89
20. My colleagues would support my use of digital interventions in therapy:		0.88			0.84
21. My colleagues think digital interventions are effective in treating mental health concerns:		0.64			0.67
22. My colleagues think that compared to face-to-face therapy, digital interventions are equally effective in treating mental health concerns:		0.31		0.41	0.31
11. I am confident in my ability to use digital interventions with clients:			0.94		0.81
14. I am confident in my ability to use technology to deliver digital interventions to clients:			0.96		0.86
15. I possess the required technical knowledge to use digital interventions for my clients' mental health:			0.89		0.72
28. I will learn more about digital interventions:				0.90	0.79
29. I intend to make time to learn about digital interventions within the next week:				0.94	0.81
30. I intend to check with my clients whether they would like to use digital interventions to address their mental health concerns:				0.43	0.46