Content validity and perceived utility of a self-help online acceptance and commitment therapy program focused on repetitive negative thinking.

Telehealth strategies have become essential for responding to the sanitary emergency due to the COVID-19 pandemic. In this context, developing online psychological interventions (OPIs) that can treat and prevent psychological difficulties is gaining more relevance. This article describes an acceptance and commitment therapy (ACT) based OPI focused on repetitive negative thinking (RNT). This OPI is called Disentangled, Aware, and Committed (DAC) and represents an attempt to adapt previous RNT-focused ACT protocols to this format type. Study 1 evaluated content validity through the assessment of four experts in ACT regarding clarity, utility, pertinence, conceptual adjustment, and therapeutic goal fulfillment for each component of the DAC program. In Study 2, forty-one undergraduate clinical psychology trainees rated the components of the DAC program according to its clarity and perceived utility. These participants were enrolled in a randomized controlled trial that analyzed the efficacy of the DAC to prevent the usually observed increase in emotional symptoms and RNT among this population. Expert reviewers in Study 1 rated all DAC components as clear and useful for potential users, following a logical order, theoretically coherent with the ACT model, and successfully fulfilling its stated goals. In Study 2, the participants also rated the DAC components as clear and potentially useful for their lives. In conclusion, the DAC appears to be a feasible transdiagnostic OPI for treating and preventing emotional symptoms, which warrants further studies analyzing its efficacy.

According to the World Health Organization (WHO), psychological disorders are among the leading causes of disability globally, having an immense burden that is systematically under-appreciated and unmet (Kessler et al., 2009; Ngui et al., 2010; World Health Organization, 2008). Only one-third of the global population experiencing psychological disorders receive mental health intervention. Patients in high-income countries typically have easier and more frequent access to mental health services (World Health Organization, 2017). Additionally, the proportion of treatment is much higher among severe rather than moderate or mild cases (World Health Organization, 2017). Unfortunately, the dissemination of prevention interventions for the development of psychological disorders is also scarce, especially in developing countries (Naslund et al., 2017). In summary, most individuals treated in mental health services suffer from severe psychological disorders, and prevention and treatment interventions only reach a minority of people suffering from mild to moderate psychological difficulties.

Online psychological intervention

New models for delivering psychological interventions are needed to prevent and treat psychological disorders (Kazdin & Rabbitt, 2013). Online psychological intervention (OPI) is an intervention model especially well-suited to scaling up mental health services.

Online psychological intervention (OPI) is an emerging field consisting of the delivery of self-help, efficient, ethical, high-quality, evidence-based, empowering, and encouraging psychological intervention through information and communication technologies (Eysenbach, 2001; Peñate et al., 2014; Sierra et al., 2018). Evidence suggests that OPIs are a legitimate and flexible alternative to traditional psychological interventions. Additionally, OPIs show significant advantages in terms of less time investment for both parties, higher mental health coverage, brief interventions that foster the patient's autonomy and represent lower economic costs (Amichai-Hamburger et al., 2014; Barak et al., 2008; Parikh & Huniewicz, 2015; Rochlen et al., 2004).

The development of transdiagnostic evidence-based OPIs can scale up psychological interventions to a larger proportion of the population suffering from psychological disorders or at risk of developing clinically-relevant psychological difficulties. Indeed, transdiagnostic therapies have been adapted to the OPI's format. Most of the existing studies feature online adaptations of acceptance and commitment therapy (ACT; Hayes et al., 2011; Wilson & Luciano, 2002) or other acceptance-based interventions. These OPIs are usually self-help-oriented and arranged in short modules of multimedia content (Brown et al., 2016; Cavanagh et al., 2014; Kelson et al., 2019; Spijkerman et al., 2016; Twomey et al., 2017). However, to our knowledge, there are no adaptations of ACT that can be used for broad treatment and prevention purposes in Spanish-speaking Latinos. In this regard, brief ACT interventions focused on dismantling unconstructive repetitive negative thinking (RNT) have recently shown promising results in treating multiple disorders among Latinos such as depression, generalized anxiety disorder, panic disorder, complicated grief, and fibromyalgia (López-Palomo et al., 2022; Ruiz et al., 2016, 2018, 2019b, 2020a, 2020b; Salazar et al., 2020; Toquica-Orjuela et al., 2022), and in improving interpersonal skills (Bernal-Manrique et al., 2020). This study presents the adaptation of these RNT-focused ACT protocols into an OPI format.

Acceptance and commitment therapy

ACT is a transdiagnostic, process-based, contextual behavioral therapy. It is philosophically and theoretically rooted in functional contextualism (Hayes, 1993) and relational frame theory (RFT; Hayes et al., 2001), respectively. The ACT model suggests that psychological inflexibility is at the core of psychopathology and consists of a functional class of behavior in which ongoing private experiences dominate responding over chosen values (Bond et al., 2011; Luciano et al., 2022; Törneke et al., 2016).

There are at least three central processes involved in psychological inflexibility: cognitive fusion, experiential avoidance, and the lack of values clarity. Cognitive fusion (or fused behavior) entails responding in accordance with the immediate functions of private experiences without realizing that these experiences are only transitory events (Hayes, 2004; Ruiz et al., 2022). Fused behavior with the aversive functions of private experiences leads to engaging in experiential avoidance, which is a verbal regulation strategy consisting of deliberate efforts to reduce these aversive functions (Hayes et al., 1996). Experiential avoidance usually entails multiple topographies of behavior such as thought suppression, distraction, overeating, substance consumption, worry, rumination, etc. The engagement in experiential avoidance strategies tends to obstruct values-based behaviors. Additionally, the lack of values clarity precludes behavior controlled by personally relevant, long-term abstract reinforcers (i.e., hierarchical positive reinforcers). This makes it more probable to orient behavior toward short-term contingencies and engaging in experiential avoidance strategies when experiencing aversive private experiences.

ACT interventions aim to foster psychological flexibility, which is the skill to nonjudgmentally contact ongoing private experiences while orienting behavior toward valued ends. According to Törneke et al. (2016), the clinical work in ACT can be conceptualized through three strategies that help the client: (a) to identify and contact the counterproductive consequences of the inflexible pattern of behavior, (b) to develop an observation point of thoughts in a way that allows contacting with functions linked to values, and (c) to clarify values and behaviors inserted in them.

ACT and repetitive negative thinking

RNT is a term that encompasses processes consisting of perseverative thinking, such as worry and rumination (Ehring & Watkins, 2008). RNT is considered a core transdiagnostic process due to the strong empirical evidence suggesting that it significantly predicts the onset and severity of emotional symptomatology in clinical and nonclinical samples across several contexts and cultures (Nolen-Hoeksema, 1991, 2000; Nolen-Hoeksema et al., 2008; Segerstrom et al., 2000; Sorg et al., 2012).

From a functional-analytic standpoint, RNT can be conceptualized as an experiential avoidance strategy (Ruiz et al., 2016, 2020a). Importantly, RNT tends to become a predominant experiential avoidance strategy because it: (a) commonly is the first reaction to fear, unattained goals, and incoherence, (b) tends to prolong negative affect as it focuses on negative private experiences, and (c) usually leads to engaging in additional experiential avoidance strategies (Caselli et al., 2013; Newman & Llera, 2011; Nolen-Hoeksema et al., 2007; Ruiz et al., 2016; Wells, 2009).

A recent approach has explicitly integrated RNT into ACT through a therapeutic approach called RNT-focused ACT (Ruiz et al., 2016, 2018). RNT-focused ACT incorporates recent RFT conceptualizations of psychological flexibility (Luciano et al., 2022; Törneke et al., 2016), values and triggers for RNT as hierarchical networks of positive and negative reinforcers (Gil-Luciano et al., 2019), and the counterproductive effects produced by RNT (Ruiz, Luciano, et al., 2020).

The efficacy of brief, RNT-focused ACT interventions has been tested in the last few years through several single-case experimental designs (SCED) and randomized controlled trials (RCT) in Colombian participants. Through a multiple-baseline design, Ruiz et al. (2016) showed that a 1-session protocol produced large effects in reducing RNT for participants with mild to moderate emotional symptoms. Following these initial encouraging results, Ruiz et al. (2018) evaluated the effect of a 2-session protocol in treating moderate emotional disturbances through another SCED. Ninety percent of participants showed clinically significant changes in emotional symptoms, and the effect sizes were very large (d = 2.44 and 2.68). Subsequent SCEDs found similar results in participants diagnosed with generalized anxiety disorder (GAD) with the couple relationship as the main worry domain (Ruiz, García-Beltrán, et al., 2019), chronic and comorbid depression and GAD (Ruiz, Luciano, et al., 2020), child depression (Salazar et al., 2020), panic disorder (Toquica-Orjuela et al., 2022), fibromyalgia (López-Palomo et al., 2022), and complicated breakup grief (Medina-Reina & Ruiz, 2022). An RCT compared the efficacy of a 2-session, RNT-focused ACT protocol versus a waitlist control condition in treating depression and GAD (Ruiz, Peña-Vargas, et al., 2020). The results showed that the RNT-focused ACT intervention significantly reduced emotional symptoms (d = 2.42), cognitive fusion, experiential avoidance, and RNT (d = 2.26 to 2.73). Lastly, two RCTs showed that RNT-focused ACT interventions were effective in improving interpersonal skills in adolescents with problems with school and social adaptation (Bernal-Manrique et al., 2020) and in decreasing RNT and emotional symptoms in clinical psychology trainees (Dereix-Calonge et al., 2019).

Aims of this study

This study has two main objectives. First, we present a detailed description of the adaptation of RNT-focused ACT protocols into an OPI format. This adaptation is called Disentangled, Aware, and Committed (DAC; Sierra & Ruiz, 2018, July) and is an online psychological self-help intervention program. Second, we analyze the content validity and perceived utility of this OPI throughout two studies. In Study 1, four ACT experts were asked to review the whole program and evaluate its components regarding their perceived clarity, utility, pertinence, adjustment to the ACT model, and therapeutic goal fulfillment. In Study 2, a group of 41 clinical psychology trainees was asked to rate the DAC components in terms of perceived clarity and utility. These ratings were obtained during the trainees' participation in a randomized controlled trial that analyzed the effect of the DAC program components in preventing the common increase of emotional symptoms in novice clinical psychology trainees (Ruiz, Dereix-Calonge, & Sierra, 2019). The DAC program was administered through six group sessions or workshops (Dereix-Calonge et al., 2019).

The studies presented in this article were conducted before implementing the DAC program in its final web platform. The rationale for doing so was to evaluate the adequacy and potential utility of the DAC material irrespective of the usability of the web platform. Thus, as an initial step in the research program, we wanted to explore the quality of the DAC's materials. This is similar to the development of other OPIs that included some sort of evaluation by experts on the therapeutic model or the users themselves regarding the clarity and utility of the intervention contents (Farrer et al., 2019; Firestone et al., 2019; Krüsche et al., 2013; Levin et al., 2017; Rahmadiana et al., 2019; Simpson et al., 2015; Weisel et al., 2019; Witlox et al., 2018).

Online RNT-focused ACT program

The DAC program consists of three modules with a total of 15 videos, 9 audios, and 15 written activities and experiential exercises. The program's average duration is 6 h (approximately 2 h per module), although duration depends on the pace and detail the participant chooses to take.

A diagram is used as graphical support for the individual throughout the intervention (see Fig. 1 ). It consists of four sections that the participant fills with their own information regarding their hierarchical network of triggers for RNT, experiential avoidance strategies, valued directions, and actions that would help the participant move towards them.

Fig. 1

Self-clinical conceptualization diagram.

Most of the examples presented in DAC follow the cases of three hypothetical characters currently facing a difficult time in their lives. None of the characters show specific symptoms related to a particular psychological disorder to keep the intervention's focus on treatment and prevention. Contrarily, the examples always focused on how the characters reacted to their own thoughts and what choices they were making at every moment.

The first character is Pedro, an office worker whose biggest dream is to be a successful professional, and his hierarchical trigger for RNT is related to being a failure. Pedro's most common experiential avoidance strategy to stop the engagement in RNT is drinking alcohol. His other valued directions are improving his relationship with his family, especially his father, who is currently suffering from cancer, and getting closer to his spiritual life.

The second character is Andrea, a single mother and music student whose highest hopes are becoming a loving and caring mother, fixing her relationship with her parents, and retaking her career as a professional musician. Andrea's hierarchical trigger is related to failing as a mother. Her experiential avoidance strategies are watching TV shows, chatting and talking to her best friend (ruminating in her company), and complaining about the life she has to live.

Mariana, a young artist looking for a romantic relationship based on understanding and love, is the last character. Mariana's hierarchical trigger for RNT is failing to develop this type of romantic relationship and being the kind of partner she wants to be. Mariana's experiential avoidance strategies involve medication abuse, arguing with her boyfriend about her insecurities, and canceling their plans when she doubts or feels jealous. Her other valued directions are improving the relationship with her artist colleagues and feeling beautiful and healthy by improving her lifestyle.

Module I. Knowing the problem and finding solutions

The first module has three primary objectives. The first objective is to conceptualize the participants' case by assessing their hierarchical triggers for RNT, experiential avoidance strategies, valued directions, and possible committed actions. The second aim is to help the participants identify the ineffectiveness of the previous coping strategies. Lastly, the third aim is to propose an alternative behavioral pattern consisting of disengaging from RNT, being aware of their values, and acting towards them. A more detailed description of the content of Module I can be seen in Table 1 .

Table 1

Module I multimedia contents and activities with their therapeutics goals.

Module I: Knowing the problem and finding solutions
Aim 1. Introducing the rationale of the intervention	❖ Illustrating how the person is the overarching context for his/her thoughts and emotions (Video 1 + Activity 1). ❖ Illustrating how RNT works and its consequences when displayed in a counterproductive way (Video 2 + Activity 2) ❖ Illustrating that thoughts and emotions are transitory experiences and that the person can choose between letting them go or struggling with them (Video 3)
Aim 2. Identifying the hierarchy of triggers for RNT and core aspects of the RNT process	❖ Describing how triggers of RNT are usually organized in a hierarchical network (Video 4) ❖ Identifying the trigger at the top of the hierarchical network for the participant and core features of the participant's process of RNT (when, how long, bodily sensations associated, emotional consequences, and affectation in valued living) (Activity 3)
Aim 3. Identifying experiential avoidance strategies associated with the RNT process	❖ Describing the process of engaging in additional experiential avoidance strategies as a way to stop the RNT process and alleviate its emotional consequences (Video 5) ❖ Identifying the participant's main experiential avoidance strategies (Activity 4)
Aim 4. Introducing and identifying values	❖ Describing what is a value (or valued direction) and how it orients the participant toward his or her desired life (Video 6) ❖ Identifying the participant's most important values (Activity 5)
Aim 5. Introducing and identifying valued actions	❖ Describing valued actions as a necessary step to advance toward valued directions (Video 7). ❖ Identifying valued actions that would help the participant to advance towards his or her values (Activity 6)
Aim 6. Introducing learning to choose between engaging in RNT or valued actions as the alternative	❖ Describing the flexible repertoire of choosing in which direction the person is advancing at every moment as a healthier alternative to the previous strategies used by the participant (Video 8) ❖ Identifying three situations in which the participant has engaged in RNT, how the process began, the time the participant dedicated to it, valued actions that he or she could not do in the meantime, and the alternative ways of behaving (Activity 7) ❖ Amplifying the cost and consequences of engaging in the inflexible versus the flexible pattern of responding to the own private experiences (Video 9)
Aim 7. Experiential discrimination of the RNT process	❖ Practicing a defusion exercise in which thoughts are observed, and the participant chooses to let them go and engage in RNT voluntarily in response to some triggers (Audio 1)

Module II. Developing an observation point for thoughts

This module aims to train the participants to discriminate the start of RNT, understand its characteristics, and disengage from it. This module introduces several defusion exercises that illustrate how some aspects of human thinking might be pervasive, emphasizing that the participants can choose how they relate to their private events. The activities also show the participants how being mindful of what is important and useful for them in the present moment can help them disengage from RNT and advance towards their valued directions. Table 2 details the content of Module II.

Table 2

Module II multimedia contents and activities with their therapeutics goals.

Module II. Developing an observation point of thoughts
Aim 1. Illustrating and identifying the RNT process in greater detail.	❖ Illustrating in greater detail the difference between responding to private experiences flexibly or inflexibly and the RNT process (Video 10) ❖ Identifying two personal examples of the last week: (a) The chain of thoughts involved in the process of RNT, (b) How the thoughts involved in the RNT process were related to the trigger at the top of the hierarchical network, (c) the duration of the RNT episode, and (d) alternative valued behavior that the person could engage instead of engaging in RNT (Activity 8)
Aim 2. Practicing the difference between judging and engaging in RNT in response to external events versus taking a nonjudgmental stance towards them.	❖ Illustrating the difference between judging an external event by engaging in RNT and just noticing the thoughts and feelings provoked by the event and adopting a nonjudgmental stance towards it (Video 11) ❖ Practicing the differentiation between judging an external event and engaging in RNT versus adopting a nonjudgmental stance with personal examples (Audio 2)
Aim 3. Illustrating how the context influences private events and that engaging in RNT can be under voluntary control.	❖ Practicing observing the thoughts generated in a free association exercise nonjudgmentally and not engaging in RNT (Audio 3). ❖ Practicing fantasizing and worrying consciously while discriminating the process (Audio 4 and Activity 9).
Aim 4. Practicing the skill of noticing the flow of thoughts and focusing the attention on a valued behavior.	❖ Practicing an exercise in which the person has to focus his/her attention on a specific picture point while noticing the impulse to observe other parts of the picture and the flow of thoughts (Video 12). ❖ Practicing an exercise where the person has to imagine a concert and observe the flow of images and thoughts without trying to control the process (Audio 5).
Aim 5. Promoting a transcendental and coherent perspective of the self.	❖ Practicing a modified version of the “observer exercise” (Hayes et al., 2001) that includes multiple hierarchical cues between the self and the private events and emphasizes the coherence of the personal history, including triggers for RNT and values (Audio 6). ❖ Writing about the experience and conclusions of the “observer exercise” (Activity 10).

Module III. Focusing on what really matters

This module aims to help the participants clarify their values. Some activities show the long-term consequences of the past coping strategies and how those proposed throughout the intervention can help them advance towards the meaningful life they want. Other activities propose establishing concrete and realistic goals related to these valued directions and how they can overcome potential barriers that might arise during their efforts to advance (see Table 3 ).

Table 3

Module III multimedia contents and activities with their therapeutics goals.

Module III: Focusing on what really matters
Aim 1. Illustrate that values can change and evolve through time in the context of a transcendent self.	❖ Practicing the “autobiography of dreams exercise.” This exercise explores how values evolved throughout a person's life and explore events of success and failure (Audio 7) ❖ Exploring in detail two of the participant's values by answering why they are important, how moving towards and away from these valued directions makes him or her feel, and what opportunities of advancing towards them the person has lost (Activity 11)
Aim 2. Exploring the long-term consequences of rigidly engaging in RNT versus behaving with flexibility towards the own values.	❖ Practicing “the legacy” exercise. This exercise explores how the 80th birthday would feel after dedicating life to engaging in RNT and experiential avoidance versus engaging in valued behaviors (Audio 8)
Aim 3. Establishing goals and objectives that permit advance towards own values.	❖ Illustrating the importance of establishing value-related goals to advance towards the own values (Video 13) ❖ Establishing realistic, concrete, and verifiable medium-term goals related to the participants' values (Activity 12)
Aim 4. Exploring time management and establishing specific daily activities towards values.	❖ Illustrating how engaging in RNT impedes engaging in valued behaviors in daily life related to the established goals and objectives (Video 14) ❖ Analyzing in detail the participant's daily activities during a business day and a weekend day to help him or her to realize how is he or she managing time and under which function is his or her behavior at those times (Activity 13)
Aim 5. Identifying the psychological barriers to advancing toward values and establishing an agenda for several days focused on advancing towards previously defined goals.	❖ Illustrating psychological barriers (i.e., aversive private events that may arise when advancing toward values) and how the person can react to them (Video 15) ❖ Identifying the participant's psychological barriers when they begin to move towards values in daily life (Activity 14) ❖ Planning daily activities by focusing on what is the goal of each activity, what barriers may arise when moving towards values, and what could they do to keep going even in their presence (Activity 15)
Aim 6. Summarizing the main contents of the program.	❖ Summarizing the contents of the program and closing the intervention by empowering the participants into using their newly acquired abilities to make significant changes in their life (Audio 9)

Study 1. Analyzing the content validity of the DAC according to expert reviewers

Participants

Four reviewers with formal training in ACT participated in the study. Reviewer 1 was a Ph.D. in Psychology who received a master's degree in contextual therapies with the main focus on ACT. She has also published about ten scientific papers related to ACT. Reviewer 2 was another Ph.D. in Psychology who had previously conducted a clinical study on child depression using an RNT-focused ACT protocol. Reviewer 3 was also a Ph.D. in Psychology who had previously developed a psychometric instrument and tested the efficacy of an RNT-focused ACT intervention for psychological distress. Finally, Reviewer 4 was a licensed clinical psychologist with extensive ACT training and previous research experience who was conducting an institutional internship.

These experts were not involved in the development of the program. However, they were invited to assess the content validity of the multimedia content and experiential exercises featured in DAC.

Instruments

Experts Validity Evaluation (EVE). The EVE is a custom instrument developed for this study that evaluates the content validity of the DAC according to five criteria: clarity, perceived utility, pertinence, adjustment to the ACT model, and therapeutic goal fulfillment. It consists of five items rated on a 5-point Likert-type scale (5 = completely agree; 1 = completely disagree). Clarity refers to how clear is the content of a particular media piece or activity. The item of the EVE evaluating Clarity was “The content of this part of the program is clear and easy to understand.” Utility inquires how useful the content could be for the participant's daily life with the following item “The content of this part of the program will be useful for the participants.” Pertinence means how adequate the content is at that time of the intervention with the item “The presentation of this content is pertinent in this moment of the intervention program.” Adjustment to the ACT model analyzes how conceptually coherent the content of the activity is with the ACT model. The item for this variable was “The content of this part of the program was conceptually coherent with the ACT model.” Lastly, therapeutic goal fulfillment evaluates how successful the activity is towards fulfilling the therapeutic goal described by the authors. The item that evaluated this variable was “The content of this part of the program accomplished the therapeutic aims planned by the authors."

Procedure

The procedures followed in the research reported in the manuscript were approved by the Bioethics Committee of Fundación Universitaria Konrad Lorenz (2016-021B). The reviewers were given access to DAC through the Moodle platform provided by the institution for its initial development and assessment. On the start screen of the platform, the reviewers found a few instructions regarding navigating the program and successfully assessing its content validity through the EVE. The EVE was programmed to appear in the 23 moments presented in Table 4 . The reviewers had to go through the whole program as if they were participants but evaluate the content at the end of each video, audio, or activity.

Table 4

Contents evaluated on each Experts Validity Evaluation and User Quality Survey item.

EVE/UQS	Module	Contents
1	I	Video 1 + Activity 1
2	I	Video 2 + Activity 2
3	I	Video 3
4	I	Video 4 + Activity 3 + SCCD
5	I	Video 5 + Activity 4 + SCCD
6	I	Video 6 + Activity 5 + SCCD
7	I	Video 7 + SCCD
8	I	Video 8 + Activity 6
9	I	Video 9
10	I	Audio 1
11	II	Video 10 + Activity 7
12	II	Video 11 + Audio 2
13	II	Audio 3
14	II	Audio 4 + Activity 8
15	II	Video 12
16	II	Audio 5
17	II	Audio 6 + Activity 9
18	III	Audio 7 + Activity 10
19	III	Audio 8
20	III	Video 13 + Activity 11
21	III	Video 14 + Activity 12
22	III	Video 15 + Activity 13 + Activity 14
23	III	Audio 9

Note. SCCD = Self-Clinical Conceptualization Diagram.

Data analysis

The raw data of this study can be accessed at https://osf.io/54qpn/. Descriptive statistical analyses were conducted for the EVE responses of the reviewers. In addition, to assess the DAC's content validity, Aiken's V (Aiken, 1980, 1985) and its 95% confidence intervals (CI) were also computed for each variable (i.e., Clarity, Utility, Pertinence, Conceptual Adjustment, and Therapeutic Goal Fulfillment) and component of the DAC program. Coefficient V takes values between 0 and 1, with 1 indicating a perfect agreement among judges regarding the higher score of the evaluated contents. This analysis was computed with the Microsoft Excel calculator provided by Cordón (2017), based on Merino-Soto and Livia-Segovia (2009). Charter (2003) suggested that V values should be statistically significantly higher than 0.70, which can be tested by observing whether the 95% CI includes values lower than 0.70. This guideline was adopted to consider component adequacy.

Results

Table 5 shows that the reviewers rated all aspects of the DAC program with high scores. Specifically, the mean scores for Clarity, Utility, and Goal Fulfillment ranged from 4.75 to 5, whereas the mean scores for Pertinence and Conceptual Adjustment were 5 in all cases. Regarding Aiken's V, Clarity, Utility, and Goal Fulfillment ranged from V = 0.93 (95% CI [0.71, 0.98]) to V = 1 (95% CI [0.80, 1.00]). Pertinence and Conceptual Adjustment had perfect agreement scores of V = 1 (95% CI [0.80, 1.00]) in all cases.

Table 5

Descriptive statistics and Aiken's V of the Expert Validity Evaluation (EVE).

	Clarity					Utility					Pertinence					Conceptual Adjustment					Therapeutic Goal Fulfillment
	M	SD	V	95% CI		M	SD	V	95% CI		M	SD	V	95% CI		M	SD	V	95% CI		M	SD	V	95% CI
				Lower	Upper				Lower	Upper				Lower	Upper				Lower	Upper				Lower	Upper
EVE-1	4.75	0.5	.93	.71	.98	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	4.75	0.5	.93	.71	.98
EVE-2	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	4.75	0.5	.93	.71	.98
EVE-3	4.75	0.5	.93	.71	.98	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1
EVE-4	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1
EVE-5	5	0	1	.8	1	4.75	0.5	.93	.71	.98	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1
EVE-6	4.75	0.5	.93	.71	.98	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	4.75	0.5	.93	.71	.98
EVE-7	4.75	0.5	.93	.71	.98	5	0	1	0.8	1	5	0	1	.8	1	5	0	1	.8	1	4.75	0.5	.93	.71	.98
EVE-8	4.75	0.5	.93	.71	.98	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1
EVE-9	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1
EVE-10	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1
EVE-11	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	4.75	0.5	.93	.71	.98
EVE-12	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1
EVE-13	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1
EVE-14	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1
EVE-15	4.75	0.5	.93	.71	.98	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1
EVE-16	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1
EVE-17	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1
EVE-18	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	4.75	0.5	.93	.71	.98
EVE-19	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1
EVE-20	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1
EVE-21	4.75	0.5	.93	.71	.98	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1
EVE-22	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1
EVE-23	4.75	0.5	.93	.71	.98	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1	5	0	1	.8	1

Conclusions

This study analyzed the clarity, utility, pertinence, conceptual adjustment, and goal fulfillment of the DAC program according to four expert reviewers. The reviewers rated the 23 components of the DAC program indicated in Table 4 with Aiken Vs ranging from 0.93 to 1.0. The 95% CI did not include scores lower than 0.70, which means that all scores were statistically significantly higher than the strict cutoff of V = 0.70 (Charter, 2003).

In conclusion, the 23 components of the DAC program showed a high degree of clarity, utility, pertinence, conceptual adjustment, and goal fulfillment according to the four experts in ACT who evaluated it.

Study 2: analyzing the content validity and perceived utility of the DAC according to users

Forty-one undergraduates in Psychology participated in this study. They were at the beginning of their clinical training in the Center of Clinical Psychology of a Colombian university. Unlike other countries, Colombian laws allow undergraduates in psychology to receive clinical psychology training and assist patients under the guidance of a supervising professional. All participants were in the ninth semester (out of ten), in which they had a 4-month, 28-h/weekly, compulsory clinical practice. The clinical psychology trainees could handle up to 5 patients simultaneously and had a weekly, 1.5-h supervision with a clinical psychologist.

This sample was recruited to participate in a randomized controlled trial that served as a preliminary evaluation of the effect of DAC's components on emotional symptoms and valued living among novice clinical psychology trainees (Dereix-Calonge et al., 2019). The rationale for piloting DAC's components with this population was based on previous studies that found that (a) similar novice clinical psychology trainees showed a significantly higher increase in emotional symptoms throughout the semester than a control cohort (Ruiz, Dereix-Calonge, & Sierra, 2019), and (b) RNT focused on the clinical practice longitudinally predicted the changes in their emotional symptoms (Dereix-Calonge et al., 2020).

User Quality Survey (UQS). The UQS is a custom instrument developed for this study that evaluates the clarity and perceived utility of the components of the DAC program. It consists of two items rated on a 5-point Likert-type scale (5 = completely agree; 1 = completely disagree). Clarity and utility were defined similarly to the EVE instrument. Specifically, the items for Clarity and Utility were, respectively, “The content of this part of the program has been clear and easy to understand,” and “The content of this part of the program will be useful in my daily life."

The procedures followed in the research reported in the manuscript were approved by the Bioethics Committee of Fundación Universitaria Konrad Lorenz (2016-021B). Written informed consent was obtained from all participants in this study. The study was presented to all undergraduate clinical psychology trainees (N = 94) during the general induction to the Center of Clinical Psychology in the first week of the semester. Eighty-five trainees agreed to participate and provided informed consent. Participants were then randomly allocated to the RNT-focused ACT condition (N = 43) or the waitlist control condition (N = 42). Forty-one of the 43 participants in the RNT-focused ACT condition agreed to provide feedback about the intervention. These were the participants of the current study.

The intervention began two weeks after recruitment and was conducted in six, 1-h, weekly sessions. The sessions were conducted after the training sessions in assessment and intervention organized by the center. We did not use the preliminary moodle platform facilitated by the institution to avoid contaminating the clarity and perceived utility scores with the usability of this tool. The intervention consisted of the application of the DAC program components in groups of approximately 15 participants. The sessions were led by two psychologists who only had a facilitator role that consisted of presenting the materials of the DAC program. The video and audio files were played for the whole group, and participants responded individually to the program's exercises. No group discussion or interaction with the facilitators was provided. The UQS was administered in pen-and-paper form after experiencing each of the 23 components of DAC.

Further details about the procedure and results of this randomized controlled trial can be found in Dereix-Calonge et al. (2019).

The raw data of this study can be accessed at https://osf.io/54qpn/. Data analysis for this study was identical to the one conducted in Study 1. Thus, we conducted descriptive analyses for the UQS responses by the participants and calculated the clarity and perceived utility of the DAC components Aiken's V.

Table 6 shows that the participants rated all aspects of the DAC program with mean scores higher than 4 out of 5 points. Specifically, the mean scores in Clarity ranged from 4.15 to 4.79, whereas for Utility ranged from 4.23 to 4.76. Regarding Aiken's V, values in Clarity ranged from V = 0.79 (95% CI [0.72, 0.85]) to V = 0.95 (95% CI [0.90, 0.97]) and for Utility from V = 0.81 (95% CI [0.74, 0.86]) to V = 0.94 (95% CI [0.89, 0.97]).

Table 6

Descriptive statistics and Aiken's V of the User Quality Survey.

	Clarity					Utility
	M	SD	V	95% CI		M	SD	V	95% CI
				Lower	Upper				Lower	Upper
UQS-1	4.58	0.78	.89	.84	.93	4.43	0.64	.86	.79	.90
UQS-2	4.78	0.47	.95	.90	.97	4.76	0.49	.94	.89	.97
UQS-3	4.68	0.47	.92	.87	.95	4.68	0.47	.92	.87	.95
UQS-4	4.76	0.43	.94	.89	.97	4.76	0.49	.94	.89	.97
UQS-5	4.79	0.41	.95	.90	.97	4.61	0.72	.90	.85	.94
UQS-6	4.70	0.52	.93	.87	.96	4.73	0.51	.93	.88	.96
UQS-7	4.73	0.51	.93	.88	.96	4.73	0.51	.93	.88	.96
UQS-8	4.74	0.44	.94	.89	.97	4.69	0.57	.92	.87	.96
UQS-9	4.77	0.44	.94	.90	.97	4.72	0.51	.93	.88	.96
UQS-10	4.51	0.76	.88	.82	.92	4.64	0.74	.91	.86	.95
UQS-11	4.64	0.58	.91	.86	.95	4.67	0.58	.92	.86	.95
UQS-12	4.70	0.46	.93	.87	.96	4.54	0.77	.89	.83	.93
UQS-13	4.59	0.68	.90	.84	.94	4.54	0.76	.88	.83	.93
UQS-14	4.62	0.78	.90	.85	.94	4.64	0.74	.91	.86	.95
UQS-15	4.15	1.20	.79	.72	.85	4.23	1.04	.81	.74	.86
UQS-16	4.62	0.54	.90	.85	.94	4.59	0.59	.90	.84	.94
UQS-17	4.47	0.69	.87	.81	.91	4.47	0.76	.87	.81	.91
UQS-18	4.58	0.64	.89	.84	.93	4.58	0.64	.89	.84	.93
UQS-19	4.34	1.10	.84	.77	.89	4.42	1.11	.86	.79	.90
UQS-20	4.53	0.86	.88	.82	.92	4.47	0.92	.87	.81	.91
UQS-21	4.43	0.74	.86	.80	.90	4.31	1.21	.83	.76	.88
UQS-22	4.62	0.55	.90	.85	.94	4.41	1.08	.85	.79	.86
UQS-23	4.67	0.60	.92	.86	.95	4.58	0.75	.89	.84	.93

This study analyzed the clarity and perceived utility of the DAC program according to users' ratings. The participants rated the 23 components of the DAC program with Aiken Vs ranging from 0.79 to 0.95. The 95% CI did not include scores lower than 0.70, which means that all scores were statistically significantly higher than the cutoff of V = 0.70 (Charter, 2003). In conclusion, the 23 components of the DAC program showed a high degree of clarity and utility according to the participants in Dereix-Calonge et al. (2019).

Discussion

This article presents an OPI called DAC, an adaptation of previous, successfully tested, RNT-focused ACT protocols designed for face-to-face therapy (Ruiz et al., 2016, 2018, 2019b, 2020a, 2020b; Salazar et al., 2020). As in the development of other OPIs (e.g., Farrer et al., 2019; Firestone et al., 2019; Krüsche et al., 2013; Levin et al., 2017; Rahmadiana et al., 2019; Simpson et al., 2015; Weisel et al., 2019; Witlox et al., 2018), two studies were conducted in which expert reviewers (Study 1) and users (Study 2) evaluated the DAC components using a custom instrument. Specifically, in Study 1, four ACT expert reviewers rated the DAC components as being clear and useful for the potential users, following a logical order, theoretically coherent with the ACT model, and successfully fulfilling their stated goals. In Study 2, participants in a randomized controlled trial (Dereix-Calonge et al., 2019) also rated the same DAC components as clear and useful for their lives.

The studies presented in this article suggest that the DAC components have content validity and perceived utility. Additionally, Dereix-Calonge et al. (2019) presented preliminary data regarding the efficacy of the DAC in preventing and reducing emotional symptoms in novice clinical psychology trainees. As previously commented, Dereix-Calonge et al. (2019) randomly allocated 85 clinical psychology trainees to the RNT-focused ACT or the waitlist control condition (WLC). The intervention in the RNT-focused ACT condition involved applying the DAC program in a face-to-face group setting and without interaction with the facilitators. The primary outcomes of the study were indicators of emotional symptoms and valued living, whereas indicators of RNT acted as process outcomes. The results revealed that the DAC program was efficacious in reducing emotional symptoms (d = 0.75), depression (d = 0.79), the frequency of behaviors obstructing valued living (d = 0.51), RNT focused on the clinical practice (d = 0.89), and general RNT (d = 0.62). Importantly, the intervention effects were longitudinally mediated by reducing RNT focused on the clinical practice.

Dereix-Calonge et al. (2019) also analyzed the results of the DAC program in the subgroup of participants showing high levels of emotional symptoms. These participants showed statistically significant reductions in emotional symptoms (d = 0.97 to 2.52) and frequency of behaviors obstructing valued living (d = 0.88), and an increase in values progress (d = 1.10). Additionally, 73.33% of participants obtained a reliable change in emotional symptoms, whereas 66.67% showed a clinically significant change. Conversely, only 7.14% of participants in the WLC showed reliable and clinically significant change.

The results obtained in Study 2 and in Dereix-Calonge et al. (2019) provide preliminary support for the clarity, perceived utility, and efficacy of DAC. However, the current study has some relevant limitations that should be highlighted. First, the EVE and UQS lacked reliability and validity evidence. These instruments were developed for the current study because we did not find standardized instruments appropriate for our aim of evaluating each DAC component's content validity and perceived utility. Second, users' content validity and perceived utility evidence were obtained in a different format from the one originally intended (i.e., six weekly, group, clinical-directed sessions instead of accessing the online platform itself). However, this was intentional on our part. Our rationale was to avoid contaminating the content validity and perceived utility judgments with the potential difficulties in the usability of the preliminary moodle platform facilitated by the institution. In this regard, further studies will need to explore the usability of the final platform in which the DAC program will be allocated and the DAC efficacy implemented in this format. Third, the content validity, perceived utility, and efficacy data of the DAC program have been obtained only from psychology undergraduates. This limits the generalizability of the results. For instance, psychology undergraduates might be more inclined to evaluate psychological interventions positively and could understand their components better than the general population due to their previous training. Subsequent studies should analyze the efficacy of the DAC in more diverse samples. Importantly, a recent feasibility study has been conducted exploring the efficacy of the DAC program, as implemented in its final web platform, in reducing emotional symptoms, RNT, and psychological inflexibility in Colombian young adults with promising results (Sierra & Ortiz, 2022).

Conclusions

Although the current COVID-19 global pandemic has turned telehealth strategies into part of the “new normal” (Hollander & Carr, 2020; Ohannessian et al., 2020; Smith et al., 2020), even high-income countries struggle to meet their current mental health demand and have not started yet to create and implement alternative strategies to synchronic virtual visits (Duan & Zhu, 2020). Therefore, the development of OPIs is fundamental to help mitigate the psychological burden of COVID-19 on the general population (Duan & Zhu, 2020; Zhou et al., 2020). The present article contributes to developing OPIs that could help mitigate this burden by analyzing the content validity of a transdiagnostic OPI intervention that might be used to treat and prevent emotional symptoms.

In conclusion, according to expert reviewers and potential users, this study has shown that the DAC components have content validity and perceived utility. This evidence warrants the conduction of further studies analyzing the efficacy of the DAC program in fostering psychological flexibility, reducing emotional symptoms, and improving quality of life.

Funding

This study was funded through the agreement 198 of 2017 between MinCiencias and Fundación Universitaria Konrad Lorenz that legally formalizes Call for Proposals #761 of 2016 by MinCiencias.

Declaration of competing interest

Given their role as an Editorial Board Member Ruiz F.J., had no involvement in the peer-review of this article and had no access to information regarding its peer-review. The other author has declared no conflicts of interest.