An Internet-delivered Cognitive-Behavioral Therapy (iCBT) for Prolonged Grief Disorder (PGD) in adults: A multiple-baseline single-case experimental design study.

The death of a loved one has physical, psychological, and social consequences. Between 9.8 and 21.5 % of people who lose a loved one develop Prolonged Grief Disorder (PGD). Internet- and computer-based interventions (i.e., Internet-delivered Cognitive-Behavioral Therapy, iCBT) are cost-effective and scalable alternatives that make it possible to reach more people with PGD. The main goal of the present investigation was to examine the effect and feasibility (usability and satisfaction) of an iCBT (GROw program) for adults with PGD. A secondary objective was to detect adherence to the app (Emotional Monitor) used to measure daily grief symptoms. The study had a single-case multiple-baseline AB design with six participants. The GROw program is organized sequentially in eight modules, and it is based on the dual-process model of coping with bereavement. Evaluations included a pre-to-post treatment assessment of depression, grief symptoms, and typical grief beliefs, along with daily measures of symptom frequency and intensity on the Emotional Monitor App. Treatment opinions and adherence to the App were also collected. Efficacy data were calculated using a Nonoverlap of All Pairs (NAP) analysis and Reliable Change Index (RCI). The mean age of the sample was 29.5 years (SD = 8.19). Two participants dropped out of the study. Adherence to the App varied across patients (4.8 % -77.8 %). Most participants (75 %) showed a clinically significant change (recovered) in depression, and 50 % obtained a clinically significant improvement (recovered) in symptoms of loss and typical beliefs in complicated grief. The participants reported high usability and satisfaction with the treatment content and format. In sum, the GROw program was very well accepted and generally feasible, and it has strong potential for treating PGD. The results support scaling up the treatment by using more complex designs with larger samples (i.e., randomized controlled trials comparing GROw with active conditions).

Introduction

The death of a loved one has physical, psychological, and social consequences (Shear, 2015a). In particular, it has been associated with the development of multiple psychological disorders and other physical and psychological symptoms, such as panic disorder, physical symptoms such as pain, tiredness, and oversensitivity to noise, sleep disturbances, increased use of medications, and social and work interference, to name a few (Keyes et al., 2014; Lancel et al., 2020; Simon et al., 2007; Stroebe et al., 2007). Despite this, during a grief period, intense feelings of regret and longing are considered natural and usually diminish over time (Jordan and Litz, 2014; Shear et al., 2011). When the symptoms associated with grief do not diminish over time, however, they can develop into Complicated Grief (CG: Shear et al., 2011; Prigerson et al., 2009), also known as Prolonged Grief Disorder (PGD), as defined in the International Classification of Diseases 11th Revision (ICD-11) (World Health Organization, 2019), or Persistent Complex Bereavement Disorder (PCBD), according to the Fifth Edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) (American Psychiatric Association, 2013).

In the ICD-11, PGD is defined as a persistent and pervasive grief response characterized by preoccupation about the deceased or longing for the deceased accompanied by intense emotional suffering. At least six months must elapse since the death of the loved one for this diagnosis to be made (WHO, 2019). Similarly, PCBD is defined as a severe and persistent grief and mourning reaction, and it is currently in the category of conditions for further study in the DSM-5 (American Psychiatric Association, 2013). We use the term PGD in this paper.

PGD is a frequent mental disorder among adults. The reported prevalence rates of PGD range between 9.8 % and 21.5 % across various samples and countries (Lundorff et al., 2017; Parro-Jiménez et al., 2021). This high diversity in the PGD rates might also be associated with the instrument used to assess PGD. The COVID-19 pandemic has made PGD an even more important public health concern worldwide (Eisma et al., 2020). The increase in the number of deaths as a result of the virus and the widespread implementation of social distancing and visitor restrictions in healthcare centers have complicated natural grief reactions (Diolaiuti et al., 2021; Nyatanga, 2020; Wallace et al., 2020). Moreover, other factors such as guilt for “not being there” for the loved one in the final days or having images of the loved one “struggling for life on some machine” have been strongly associated with adverse bereavement outcomes during the COVID-19 pandemic (Neimeyer and Lee, 2021). In addition, the pandemic situation not only affects people who have lost a loved one. It is also related to trauma- and stressor-related disorders. For example, health-related stressors, job loss, work-related stressors, and other stressors associated with the pandemic could increase the prevalence of adjustment disorders worldwide (Kazlauskas and Quero, 2020). In addition, the rates of post-traumatic stress disorder (PTSD) and subthreshold PTSD in patients hospitalized for COVID-19 are alarming (Tarsitani et al., 2021), and a recent meta-analysis shows that PTSD could affect up to 21.5 % of healthcare workers (Li et al., 2021).

Fortunately, several psychological treatments for PGD exist, using both individual (Boelen et al., 2007; Range et al., 2000; Shear et al., 2005; Shear et al., 2016) and group formats (Constantino et al., 2001; Lieberman and Yalom, 1992; Sikkema et al., 2004), and systematic reviews and meta-analyses show that psychological interventions are effective in reducing grief symptoms in bereaved adults (Bergman et al., 2017; Johannsen et al., 2019; Wittouck et al., 2011). However, research has also shown that the existence of effective psychological treatments does not ensure that these interventions reach the people who need them (Kazdin, 2014), and the majority of individuals who need psychotherapy do not receive professional support (Kazdin, 2016). Therefore, there is an urgent need for cost-effective, accessible, and scalable psychological treatments for PGD. In addition, the restrictions imposed due to the COVID-19 pandemic, forcing people to isolate and use social distancing, have increased the need for novel approaches to psychotherapy that can be implemented in these adverse conditions. These approaches include, for example, Internet-based PGD treatments (Eisma et al., 2020) and telecommunication-based alternatives (Wallace et al., 2020).

Internet- and computer-based psychotherapies can be good alternatives to reach people who need treatment, especially during a pandemic situation. These interventions are cost-effective and often cheaper than face-to-face approaches (Musiat and Tarrier, 2014). They also facilitate access to care and involve less stigma compared to visits to mental health clinics (Aboujaoude et al., 2015; Andersson et al., 2019). Moreover, research shows that Internet- and computer-based psychological interventions are clearly superior to waiting list conditions (no treatment) and generally as effective as face-to-face psychotherapy (Andersson et al., 2014; Andersson et al., 2019; Andrews et al., 2010; Carlbring et al., 2018; Lewis et al., 2019; Sijbrandij et al., 2016).

Internet-delivered Cognitive-Behavioral Therapy (iCBT) is the most widely used psychological treatment using the Internet (Andersson, 2009). More than 200 randomized controlled trials have been published using this psychological approach (Carlbring et al., 2018), and they have shown that iCBTs are clinically more effective than waiting list conditions (Andersson et al., 2017) and safer than no-treatment conditions (Karyotaki et al., 2018b). In this type of intervention, patients log into a secure website to access, read, and download online materials organized in a series of lessons or modules (Lange et al., 2003). They can often do this at their own pace, anywhere they choose, and at no cost or very low cost to the patient. In the specific case of PGD, several Internet- and computer-based psychological interventions have also been developed to treat this problem, with promising results (Tur et al., 2019; Wagner et al., 2020). Despite this, few studies have been developed for PGD, and none of them have been adapted for use in Spanish populations.

The main aim of this study was to examine the effect and feasibility (adherence and usability) of an iCBT (GROw program) for adults with PGD using a multiple-baseline single-case experimental design. This stepped approach is recommended to avoid testing novel approaches in large samples when efficacy reports and potential feasibility problems are unclear (Margolis and Giuliano, 2019; Smith, 2012). As a secondary objective, adherence to the App (Emotional Monitor) used to measure daily grief symptoms was examined. We expected that the GROw program would be feasible in terms of treatment satisfaction, usability, and, ultimately, usefulness in significantly reducing symptoms of prolonged grief.

Method

Study design

A single-case multiple-baseline AB design, where A refers to the baseline phase and B corresponds to the treatment phase, was used in the study to evaluate the treatment outcomes. Because at least four replications of the effect (i.e., four participants) are recommended to enhance the credibility of the findings (i.e., external validity) (Kratochwill and Levin, 2010), six participants were recruited. Following the multiple-baseline design approach, participants were assigned to different lengths of the baseline phase (at least three different lengths of the baseline phase to ensure three replications of the treatment effect, according to the guidelines) (Tate et al., 2016). As recommended in the guidelines (Kratochwill et al., 2010), the possible duration of the baseline phase was set at between 8 and 16 days to ensure that sufficient assessment points would be obtained to reach the minimum of five evaluations in each phase and ensure the reliability of the data (which justifies the minimum of eight days in the baseline phase) and minimize the time without treatment (which justifies the maximum of 16 days in the baseline phase). All the participants started the baseline evaluation on the same day, after signing the informed consent sent by e-mail, but the treatment was provided at different times. In the present study, the duration of the baseline was assigned by an independent researcher and ranged from 9 to 15 days (11, 9, 14, 12, 9, and 15) of baseline evaluation before treatment onset.

Although participants had access to the treatment on the assigned days following the baseline evaluation, not all the participants accessed the online treatment on the assigned day. Therefore, the final baseline was estimated from the start of the baseline until the participants accessed treatment for the first time, which ranged from 9 to 33 days (17, 9, 19, 12, 19, and 33 days). This prolonged baseline evaluation did not affect the analysis of the results because all the participants started the baseline phase on the same day, and the requirement of a minimum of five assessment points in the baseline and treatment phases was satisfied (Kratochwill et al., 2010). The only consequence was that the waiting times until they received the treatment were longer than initially planned for some patients (patients No. 1, 3, 5, and 6). Although baseline extension was not an issue in terms of analyzing the effectiveness of the intervention, it was relevant to treatment feasibility (i.e., patients' availability to start the treatment when planned).

The study was registered at clinicaltrials.gov: NCT04376385 and approved by the Ethics Committee of Universitat Jaume I (CD/002/2019). The study was conducted at the Emotional Disorders Clinic of Universitat Jaume I (Spain).

Procedure

Eligibility criteria

Inclusion criteria included: (1) age ≥ 18 years; (2) meeting diagnostic criteria for PGD according to the ICD-11 (World Health Organization, 2019); (3) ability to understand and read Spanish; (4) ability to use a computer, having access to the Internet, and having an e-mail address; and (5) signing the informed consent.

Exclusion criteria: (1) presence of another severe mental disorder (i.e., substance abuse, psychotic disorder, or borderline personality disorder); (2) presence of self-destructive behaviors or suicide risk; (3) presence of a medical condition that prevents study participation; (4) receiving another psychological treatment during the study; (5) an increase and/or change in medication during the study period, which was evaluated at baseline (t1) and after the intervention (t2), and (6) not having daily access to an Android smartphone.

Recruitment and screening

Recruitment was carried out using non-professional (i.e., Facebook and Instagram) and professional social networks (i.e., Linkedin). In addition, patients who attend the Emotional Disorders Clinic at Universitat Jaume I were screened for eligibility.

Interested participants who saw the announcement about the study online sent an enquiry by email, and an experienced clinician conducted a telephone assessment to determine whether they met the inclusion and exclusion criteria. During this call, potential participants were informed of the study conditions (duration, procedures, etc.). After accepting the terms and signing the informed consent, the participants were randomized to the different baselines by an independent researcher using the online platform randomizer.org. The participants did not receive financial compensation for participating.

Measures

Demographics and diagnostic measures

These data were collected by a therapist during a pre-treatment telephone interview. Demographic characteristics included age, sex, educational level, occupational status, and marital status. A semi-structured interview developed for this study was used to assess some inclusion/exclusion criteria (e.g., Internet access, e-mail, etc.). For the diagnosis, two PGD assessment instruments were used. First, we administered the Structured Clinical Interview for Complicated Grief (SCI-CG) (Bui et al., 2015a), which evaluates symptoms of prolonged grief in people who lost a loved one six or more months ago. This tool was adapted to Spanish for this study following a back-translation procedure. In addition, we administered the Inventory of Complicated Grief (ICG) (Prigerson et al., 1995a) adapted to Spanish (Limonero et al., 2009), which rates current feelings of grief and differentiates between normal and pathological grief. A total score on the ICG >25 was established in the present study because it indicates complicated grief.

Emotional Monitor App measures

The Emotional Monitor App was used for daily assessment. This App can be downloaded for free at the Google Play store and was developed in Android operating system (https://play.google.com/store/apps/details?id=monitoremocional.code&hl=ca&gl=US). The researchers gave participants an individual access code to use this App. When users accessed the App, the items appeared in a linear manner, always in the same order. The participants received daily reminders from the App to complete the assessment protocol.

Evaluation with the App took place once a day from the beginning of the baseline phase to the end of the treatment. The participants had a period of 2 h (8–10 pm) to access the app and answer 14 items related to their daily symptoms. A team of experienced clinicians adapted the items (see Table 1) from well-established measures, including the Inventory of Complicated Grief (ICG; Limonero Garcia et al., 2009; Prigerson et al., 1995b) (Item 1), Prolonged Grief Disorder (PG-13) (Estevan et al., 2019; Prigerson et al., 2009) (Items 2–7), and The Frustration Discomfort Scale (Harrington, 2005) (Items 8 and 9). A set of items were also used from an App developed and validated by the authors of the study in previous research with individuals with pain and mood problems (Suso-Ribera et al., 2018) (Items 9–14).

Table 1

App assessment items.

Item 1	How often have you thought TODAY that you can't believe your loved one is deceased?	0 = At no time1 = At some point in time2 = At various times of the day3 = Most of the time4 = All the time
Item 2	How often have you intensely wished your loved one were with you TODAY?
Item 3	How often have you remembered the absence of your loved one TODAY with enormous and deep sadness?
Item 4	How often have you remembered the absence of your loved one TODAY with enormous and deep anger?
Item 5	How often have you remembered the absence of your loved one TODAY with enormous and deep anxiety?
Item 6	How often have you remembered the absence of your loved one TODAY with enormous and deep guilt?
Item 7	How often have you had pleasant memories of your loved one TODAY?
Item 8	How often have you wanted to get rid of your unpleasant emotions related to your loved one TODAY?
Item 9	How often have you tried TODAY to get rid of unpleasant thoughts related to your loved one?
Item 10	Related to the death of your loved one; What intensity of sadness have you felt TODAY?	0 = None…..10 = Extremely high
Item 11	Related to the death of your loved one; What intensity of anger have you felt TODAY?
Item 12	Related to the death of your loved one; What intensity of anxiety have you felt TODAY?
Item 13	Related to the death of your loved one; What intensity of guilt have you felt TODAY?
Item 14	Related to the death of your loved one; What intensity of grief did you feel TODAY?

Other psychological and mental health outcomes

Three questionnaires related to depression and grief symptoms were self-administered using the web platform (https://psicologiaytecnologia.labpsitec.es) where the intervention was located. These evaluations took place immediately before and after the treatment:

Beck Depression Inventory - Second Edition (BDI-II) (Beck et al., 1996), validated in the Spanish population (Sanz et al., 2003): This is a widely used 21-item self-report measure for depression symptoms and characteristic attitudes associated with depression. The items are scored on a scale from 0 to 3. Cronbach's alphas for the BDI-II range from 0.76 to 0.95, and test-retest reliability estimates of around 0.8 have been obtained (Beck et al., 1996; Sanz et al., 2003).

Inventory of Complicated Grief (ICG) (Prigerson et al., 1995c), validated in the Spanish population (Limonero Garcia et al., 2009): This is a self-report, 19-item instrument that evaluates grief symptoms in adults using five response categories (Likert-type): 0 = never; 1 = rarely; 2 = sometimes; 3 = often; and 4 = forever. The ICG items measure the frequency of the emotional, cognitive, and behavioral grief symptoms. Cronbach's alphas of 0.88–0.94 have been reported in previous studies. The total ICG score ranges from 0 to 76, and a total score of >25 indicates complicated grief disorder.

Typical Beliefs Questionnaire (TBQ) (Skritskaya et al., 2017), translated into Spanish language for this study following a back-translation procedure: This is a self-applied, 25-item instrument that assesses maladaptive thinking common in people with complicated grief. It uses a Likert response format ranging from 0 (not at all) to 4 (very strongly). Psychometric properties include good internal consistency for the general scale (α = 0.83) and good test-retest reliability.

Patient satisfaction and usability rating

Intervention satisfaction and usability rating items were administered by a therapist (by phone call) immediately after treatment completion.

The Treatment satisfaction scale (adapted from Borkovec and Nau, 1972) was administered by phone at the end of the intervention. It includes four items rated from 0 to 10 (0 = not at all; 10 = very much), namely: 1) “How logical did this treatment seem to you?”; 2) “How much do you think the treatment has been useful to you?”; 3) “How much would you recommend this treatment to a grieving friend?”; and 4) “How much do you think this treatment has been aversive (unpleasant, annoying) for you?”

The usability of the clinical content of the GROw program was measured on a scale from 0 to 10 (0 = not at all; 10 = very much), using items developed specifically for this study. Patients were asked to evaluate and provide feedback on the content of the intervention: motivation for change, slow breathing technique, psychoeducation, behavioral activation, exposure hierarchy, loss diary (reconstructing the meaning of loss), cognitive reappraisal (imaginary conversation with a friend with grief), questions about positive and negative aspects and memories of the deceased, imaginary conversations with the deceased, self-care, guilt, and forgiveness, letter of projection towards the future, and relapse prevention.

We also asked participants to rate the weekly therapist calls, again using a scale from 0 to 10 (0 = not useful/not at all; 10 = very useful/very much). The questions were: 1) “How much did you like receiving a short weekly support phone call from the therapist?”; 2) “To what extent was the weekly support phone call useful?”; and 3) “To what extent did this weekly support phone call from the therapist help you to continue with the treatment?” Qualitative information on weekly calls was also recorded by asking “Why did you choose that score?” on every item.

Intervention

The individual, self-applied GROw program was accessible online via https://psicologiaytecnologia.labpsitec.es, a website designed by the LabPsiTec group (Laboratory of Psychology and Technology, Universitat Jaume I). Participants received their own username and password to access the GROw intervention. The treatment was organized sequentially in eight modules lasting approximately 60 min each (see Table 2). The treatment was intended to last between 8 and 10 weeks. However, if necessary, the treatment could be extended a few weeks due to the difficulty and intensity of some components. Any extension of the duration of the program was recorded and reported.

Table 2

Module names and therapeutic contents.

Module/session	Content
1. Welcome module: starting the program	General explanation of the treatment. Presentation of grief cases to be used as examples. Motivation for change. Slow breathing technique
2. Understanding reactions to loss	Psychoeducation. Behavioral activation. Grief self-monitoring diary
3. Coping with loss	Mindfulness. Exposure hierarchy
4. Loss integration and restoration: first steps	Giving a metaphorical meaning to loss. Loss Diary (reconstructing the meaning of loss): Chapter 1, life before loss
5. Deepening integration and restoration of loss	Loss Diary (reconstructing the meaning of loss): Chapter 2, reaction to the death. Cognitive Reappraisal. Questions about positive and negative aspects and memories of the deceased
6. Consolidating loss integration and restoration	Loss Diary (reconstructing the meaning of loss): Chapter 3, life after loss. Imaginary conversations with the deceased
7. Self-care, guilt, and forgiveness in the grieving process	Psychoeducation about compassion. The compassionate gesture and phrases. Compassionate coping with difficulties. Psychoeducation and strategies about guilt. Psychoeducation and exercise for forgiveness (optional)
8. Evaluating progress and looking to the future	Review of the therapeutic achievements. Action plan for high-risk situations. Action plan to face difficult dates. Letter of projection towards the future

The GROw program contains texts, videos, photographs, diagrams, interactive exercises, and downloadable pdfs and audios (see Fig. 1). Participants can log in at any time to review the content, see the calendar where the session record appears, and view their progress through visual graphs (i.e., measures of grief, anxiety, depression, and positive and negative affect). A weekly support call lasting approximately 10–15 min was made by a trained clinician to: 1) review and reinforce the participants' effort and achievements, 2) motivate them to continue to work on the program content, and 3) clarify doubts and questions about the use of the GROw intervention.

Fig. 1

“Screenshots” of the “Psychology and Technology” web platform.

The therapeutic content offered in this treatment was adapted from an original intervention protocol for complicated grief developed by the LabPsiTec group (Botella et al., 2008), based on Neimeyer's program for reconstructing the meaning of loss during complicated grief (Neimeyer, 2000; Neimeyer, 2001). When updating this treatment, we incorporated treatment components developed by Shear (2015a), based on the dual-process model of coping with bereavement (Stroebe and Schut, 1999). In addition, we included elements (i.e., cognitive reappraisal) of other computerized psychological treatments for grief (Hoffmann et al., 2018; Shear, 2015b; Wagner et al., 2006) and new mindfulness activities and compassion and self-compassion strategies (Campos et al., 2019; García-Campayo et al., 2016; García-Campayo et al., 2016; García-Campayo, 2018; Lopez-Montoyo et al., 2019; Navarro-Gil et al., 2020). More information about this treatment can be found in Tur et al. (2021).

Calculations and analyses

To quantify the effects of the intervention in this single-case experimental design, we calculated the Nonoverlap of All Pairs (NAP) and the Reliable Change Index (RCI).

The NAP is an index of data overlap between phases (baseline-to-posttreatment changes) that calculates the percentage of data that show an improvement or a deterioration, with a score from 0 to 100. High scores indicate greater treatment effectiveness (Parker and Vannest, 2009). In this study, the NAP was used to evaluate the results of the 14 daily-measured items included in the Emotional Monitor App. The NAP index for each item for each participant was calculated using the Single Case Research website (http://www.singlecaseresearch.org/calculators/nap). To provide a more specific interpretation of the results, cut-off points for each effect size have been proposed according to the study by Parker et al. (2011). These cut-off rates have been used in previous single-case studies (Fernandez-Alvarez et al., 2021; Gómez-Pérez et al., 2020). In the study by Parker et al. (2011), >60 articles that used an AB contrast were analyzed. The authors proposed a rank of 0–100 related to the NAP information, considering the percentile ranks of the evaluated studies. Based on these categories, it was established that NAP scores lower than 38 % would correspond to poor treatment effects (<25th percentile), NAP scores from 38 % to 68 % would reflect mild-to-moderate intervention effects (25th–50th percentile), NAP scores between 69 % and 96 % should be interpreted as moderate-to-large effects (50th–75th percentile), and scores above 96 % would correspond to very large treatment effects (> 75th percentile).

The calculation of RCI (Jacobson et al., 1986; Jacobson and Truax, 1991) requires estimates of a scale's internal consistency and the standard deviation for a given population, and it indicates patient change after an intervention. The clinically significant change occurs when a person with pathology returns to normal functioning, that is, when he/she becomes part of the functional population. First, we calculated the RCI for each participant and questionnaire.

If the RCI is |1.96| or greater, the difference from pre-treatment to post-treatment is considered statistically significant (1.96 corresponds to the 95 % confidence interval). According to the different types of populations described (functional population, clinical population, or both) in each instrument and the characteristics of its distribution, functionality cut-off points are established for each questionnaire (Jacobson and Truax, 1991; McGlinchey et al., 2002). Finally, to classify the different types of responses to the therapeutic process, the results are then usually divided into four categories (recovered, improved, no change, and deteriorated) following the recommendations of Kupler (1991). A person is considered ‘Recovered’ when the change shown is significantly reliable (ICF > |1.96|) and the final score is within the functional distribution. An individual is considered ‘Improved’ when the change shown is significantly reliable (ICF > |1.96|), but the functional level is not reached. A person is considered to present ‘No change’ when the change is not significantly reliable (ICF < |1.96|). Finally, an individual is considered ‘Impaired’ if the change is significantly reliable (ICF > |1.96|), but in the opposite direction.

Adherence to the app was calculated as the number of days the participants answered the questions in relation to the total number of days the questions were prompted (% of the daily response).

Results

Sample

The sample was composed of six individuals diagnosed with PGD: five females and one male. The mean age was 29.5 years (SD = 8.19), ranging from 19 to 44 years. Half (50 %) of the sample had lost a parent. Six to 12 months had passed since the death of their loved one for 50 % of the sample. More details are reported in Table 3.

Table 3

Characteristics of the sample.

Participant ID	Age	Sex	Educational Level	Occupational status	Time after loss (months)	Civil status	Relationship with the deceased
1	44	Male	Elementary	Student	6–12	Widower	Partner
2	31	Female	University	Employed	12–24	In a relationship	Mother
3	29	Female	Elementary	Employed	>48	In a relationship	Grandmother
4	27	Female	Elementary	Student	6–12	In a relationship	Mother
5	27	Female	University	Student	>48	Single	Sister
6	19	Female	Elementary	Employed	6–12	Single	Father

As Table 4 indicates, two participants did not complete the treatment and dropped out in Modules 1 (Participant 1) and 6 (Participant 4).

Table 4

Results of the NAP analyses and adherence to Emotional Monitor App and GROw.

Participant ID	Nonoverlap Index (%) Item														Compliance with the app	Modules completed (GROw)
	1	2	3	4	5	6	7	8	9	10	11	12	13	14	(%)
1	50.0	69.0a	65.9	35.7	66.7	61.1	45.2	58.0	54.8	89.7a	77.0a	94.4a	69.8a	79.4a	22.2	0
2	71.5a	76.6a	68.0	75.3a	53.4	54.7	35.0	50.0	62.0	83.6a	57.1	46.8	55.6	59.6	77.8	8
3	70.9a	79.4a	77.2a	72.2a	73.9a	72.1a	20.9	83.2a	71.4a	75.5a	71.7a	72.2a	67.0	32.8	37.6	8
4	43.8	53.2	55.9	66.1	55.5	56.3	64.6	62.0	75.6a	61.8	61.9	55.7	52.7	42.1	48.6	5
5	30.6	44.6	36.1	23.8	48.4	44.0	47.3	31.7	27.4	36.5	23.4	42.2	48.4	38.7	58.9	8
6	–	–	–	–	–	–	–	–	–	–	–	–	–	–	4.67	8

Moderate-to-large effects (NAP indices ≥69). Item 1: “can't believe the death of the deceased”; Item 2: “wishes to be with the deceased”; Item 3: sadness frequency; item 4: anger frequency; Item 5: anxiety frequency; Item 6: guilt frequency; Item 7: pleasant memories frequency; Item 8: unpleasant emotions frequency: Item 9: unpleasant thoughts frequency; Item 10: intensity of sadness; Item 11: intensity of anger; Item 12: intensity of anxiety; Item 13: intensity of guilt: Item 14; intensity of grief.

Nonoverlap analyses (NAP)

According to the NAP analyses (Table 4), most of the sample (except Participant 5), including those who prematurely abandoned the treatment (Participants 1 and 4), experienced a moderate-to-large treatment effect on at least one of the outcomes. For example, Participant 1 showed a moderate-to-large improvement on almost half of the 14 items after accessing only Module 1. In contrast, Participant 4 only obtained a moderate-to-large effect on Item 9 (“How often have you tried TODAY to get rid of unpleasant thoughts related to your loved one?”) after completing five treatment modules.

Participant 2 obtained a moderate-to-large improvement in four of 14 outcomes, specifically on the items “not being able to believe that the loved one has passed away” (Item 1), “intensely wishing that your loved one was with you” (Item 2), “remembering his/her absence with anger” (Item 4), and “intensity of sadness” (Item 10).

Participant 3 obtained a significant treatment effect on most of the outcomes (11/14). The largest effect was observed on “intensely wished your loved one were with you” (Item 2), “remembering the absence of the deceased with enormous and deep sadness” (Item 3), and “get rid of unpleasant emotions related to the loved one” (Item 8).

Participant 5 did not obtain a significant treatment effect on any outcome and even presented a deterioration (NAP <50), especially on “remembering his/her absence with anger” (Item 4), “trying to avoid unpleasant thoughts related to the loved one” (Item 9), and “intensity of anger” (Item 11).

A minimum of five responses are required in the NAP analysis to ensure the reliability of the data (Kratochwill et al., 2010). Therefore, it was not possible to analyze the results of Participant 6 due to poor adherence to the app assessments.

Considering the analyzed data of the participants who finished the treatment, the outcomes on which they obtained the largest treatment effect corresponded to Item 2 (“intensely wishing that your loved one was with you”) and Item 10 (“intensity of sadness”). A visual example of item 10, related to the intensity of sadness, can be seen in Fig. 2, Fig. 3. In this item, the participant 2 showed a significant improvement, while the participant 5 showed a deterioration.

Fig. 2

Participant 2: Evolution in item 10 (intensity of sadness). NAP: nonoverlap of all pairs.

Fig. 3

Participant 5: Evolution in item 10 (intensity of sadness). NAP: nonoverlap of all pairs.

Clinically significant change: Reliable Change Index (RCI)

RCI analyses were conducted to evaluate the individual change (before and after treatment) in depression, grief symptoms, and grief beliefs (Table 5).

Table 5

RCI of the Individual test value (before and after treatment) and response to the therapeutic process.

Participant ID	Depressive Symptoms			Symptoms of loss			Typical beliefs in complicated grief
	X1	X2	Treatment response	X1	X2	Treatment response	X1	X2	Treatment response
1	22	–	–	36	–	–	61	–	–
2	19	8	Recovered	34	25	No significant change	45	40	No significant change
3	42	10	Recovered	51	5	Recovered	82	5	Recovered
4	44	–	–	36	–	–	70	–	–
5	22	18	No significant change	51	44	No significant change	66	53	No significant change
6	34	2	Recovered	44	1	Recovered	70	8	Recovered

X1: Individual test value (before treatment); X2: Individual test value (after treatment).

The majority (75 %) of the participants who finished the treatment showed clinically significant changes in depression symptoms (BDI-II), and these participants, according to the classification proposed by Kupfer (1991), could be considered recovered (Table 5). Regarding the measures of bereavement (ICG and TBQ), 50 % of the participants who finished the treatment obtained a clinically significant change, also classifying them as recovered.

Adherence to the GROw intervention and the Emotional Monitor App

Although the duration of the treatment was initially estimated at between 8 and 10 weeks, it took the patients an average of 15.4 weeks to complete all the modules. Two-thirds (66 %) of the participants finished all the modules, whereas two of them dropped out in Modules 1 and 6, respectively. Regarding adherence to the Emotional Monitor App, the participants who finished the treatment completed between 37.6 % and 77.8 % of the assigned evaluations (average of 55.7 % of measures completed; Table 4).

Usability

We found high user satisfaction among study participants who completed the intervention. All (100 %) the respondents scored “10 = very much” on the item “I would recommend this treatment to a friend or relative who has lost a loved one”. The participants also scored between 8 and 10 on the items related to their satisfaction about the usefulness and logic of the treatment. Regarding the question about whether the treatment seemed aversive (unpleasant or annoying) to them, half (three) of the participants answered “yes” and gave unpleasantness scores of 8, 7, and 4, respectively (0–10 range, where 10 is the highest level of unpleasantness).

Regarding the perceived usefulness of each therapeutic content, which was measured on a scale from 0 to 10 (0 = not useful at all; 10 = very useful), the participants responded that they found all the contents useful, with a total mean of between 8 and 10 points. The most highly valued content was the slow breathing technique, which obtained a 10 from all the participants who completed the treatment (n = 4). The least valued technique in terms of usefulness was behavioral activation, which obtained an average score of 8 points.

Finally, all the participants who completed the treatment (n = 4) gave the maximum score to the weekly calls from a therapist (10 out of 10). The items were related to the usefulness of the calls (0 = not useful; 10 = very useful), how much they liked receiving the weekly call (0 = not at all; 10 = very much), and how much it helped them to continue with the treatment (0 = not at all; 10 = very much). In addition, their qualitative opinion of the calls was recorded. Comments given included: “It is very necessary that you feel that the therapist is going to call you to ask how you are, or to ask questions”; “In case of doubts, it is much easier to solve them by telephone, I also think it is a way to avoid leaving therapy due to lack of motivation or laziness”; “You feel that you are not doing it just for yourself but that there is someone with you, someone who supports you once a week and who is asking how you are doing. In the end, it's like a psychologist's therapy, but without going to the clinic”; “What I liked the most was being able to tell someone who doesn't judge everything that happens to me. The therapist gave a more global perspective”.

Discussion

The main goal of the study was to examine the effect and feasibility (usability and satisfaction) of an iCBT (GROw program) for adults with PGD and the adherence to the app (Emotional Monitor) used to measure daily grief symptoms.

Overall, we found that the GROw program was effective for some patients only. Half of the participants who finished the treatment obtained a clinically significant change on the bereavement measures (ICG and TBQ). Similarly, a moderate-to-large effect on four or more items measured with the App was observed in 50 % of the participants who finished the treatment (see Table 4). These results are frequent in the psychotherapy literature when the improvement is analyzed patient by patient and not at a global level, as approximately 40–60 % of patients respond poorly to psychological treatments (Cuijpers et al., 2014, Cuijpers et al., 2019; Ebert et al., 2013; Karyotaki et al., 2018a). These data show that the response rate to the treatment is similar in face-to-face and guided Internet-based interventions. However, Internet-based interventions have many advantages over traditional therapies. For example, they can be more accessible and associated with less stigma (no need to visit mental health clinics), and they can have lower financial costs for the patients (Aboujaoude et al., 2015; Andersson et al., 2019; Musiat and Tarrier, 2014).

To the best of our knowledge, no studies have evaluated Reliable Change in grief treatments, and so we cannot compare the data obtained in this investigation with similar studies of PGD. Even so, a systematic review and meta-analysis study on web-based bereavement care that analyzed outcomes of RCT studies concluded that web-based CBT for bereavement showed moderate to large effects on symptoms of grief and posttraumatic stress disorder (PTSD) and small effects on depression (Wagner et al., 2020). These data are consistent with those found in our research, where grief symptoms decreased significantly in half the sample. However, our results show that depression symptoms also decreased significantly in 75 % of the participants, and so the effect of this treatment on depression symptoms was not low. One of the reasons for this discrepancy could be that the GROw program is a comprehensive treatment that includes not only techniques specifically related to grief, but also techniques (e.g., behavioral activation and mindfulness) that have been shown to improve depression symptoms (Blanck et al., 2018; Reangsing et al., 2021; Stein et al., 2021).

The items measured on the app that generated a greater treatment effect were Item 2 (“intensely wishing that your loved one was with you”) and Item 10 (“intensity of sadness”). According to the literature, feelings of wishing that the deceased could be with you and deep sadness are associated with PGD (Boelen and Smid, 2017; Shear, 2015b). The components of the GROw program mainly focus on reconstructing the meaning of loss (see Table 2). This meaning reconstruction, along with the other elements of the treatment, could have helped the participants to remember their loved one with less sadness and decrease the feeling of wanting to be with the deceased.

Regarding adherence to treatment, 33 % of the sample (two participants) dropped out of the intervention. One of these participants obtained a moderate-to-large improvement in almost half the 14 items measured with the Emotional Monitor app at baseline and during the first module. The improvement in these variables could explain why the patient left the treatment because it may indicate that this patient perceived that he/she no longer needed the intervention. Overall, the dropout rate obtained in the current investigation is consistent with what was found in other similar studies using Internet-based interventions, where the dropout rates have been approximately 30 % (Rachyla et al., 2020; Van Ballegooijen et al., 2014).

Regarding adherence to the Emotional Monitor app, an average of 55.7 % of the daily monitoring measurements was completed. This percentage is slightly lower than those obtained in other studies with similar apps (Gómez-Pérez et al., 2020). The fact that the participants had to respond on a large number of days (the treatment lasted an average of 15.4 weeks) and that they only had a short period of time during the day to respond (8–10 pm) could explain these response rates.

An important characteristic associated with adherence to Internet-delivered therapy is the involvement of a human therapist. Providing online support and guidance from a therapist during online therapy has been found to improve participant adherence (Andersson, 2009; Christensen et al., 2009; Spek et al., 2007). In this study, all the participants received pre- and posttreatment assessments and a weekly telephone follow-up from an experienced therapist, which might explain why dropout rates were similar to those found in previous research.

Regarding quantitative analysis of treatment discontinuation, qualitative studies on experiences of non-adherence in Internet interventions have shown that participants' perception that therapists cared for them was important for adherence (Johansson et al., 2015). This is consistent with the qualitative results of our study, which showed that the participants perceived that the weekly telephone support was useful and reported that they liked it and it helped them continue with the treatment: (“…it is a way to avoid leaving therapy due to lack of motivation or laziness”; “you feel that you are not doing it just for yourself but that there is someone with you, someone who supports you…”). Similar to the satisfaction with the human support, all the participants who finished the intervention reported high usability of the therapeutic contents and format of the treatment.

Strengths

The Reliable Change Index and Non-overlap analysis were used to assess the effect of the GROw program. These analyses can measure the impact of the treatment on each participant, rather than providing a summary measure of the treatment effects. Group-based effect sizes (standardized mean differences at post-treatment) have dominated the literature on treatment effectiveness due to overreliance on randomized controlled trials. This type of analysis, however, has been criticized due to the focus on pooled group scores and the fact that it is not useful to estimate causal effects (Cummings, 2011), the clinical importance of the treatment, or the number of patients in remission or recovery (Cuijpers et al., 2014). A strength of the present study was that each patient's response was evaluated separately. This allowed us to provide a more idiographic conclusion, which is more consistent with the idea proposed by John Grimley Evans more than two decades ago: “Managers and trialists may be happy for treatments to work on average; patients expect doctors to do better than that.” (Evans, 1995).

Another advantage of the study design is related to collecting daily measurements with the Emotional Monitor app. This method allowed us to record data on each variable measured for each participant across time. Although Internet- and mobile-based interventions for many disorders have been well established, comprehensive knowledge about the impact of such treatments on underlying psychopathology processes is pending (Domhardt et al., 2020). Thanks to the use of the Emotional Monitor App, both outcome (e.g., severity of depression and anxiety levels) and process variables (e.g., avoidance of thoughts and emotions) were measured throughout the treatment process. Analyzing the results of this study may be an initial approximation to understanding the mechanisms of change involved in treatments for people with PGD, but it cannot provide solid conclusions in this regard. Even so, the design of this study, using the App as a daily measure, can serve as a precedent for future studies that analyze the mediators responsible for therapeutic change in Internet-based interventions for adults with PGD. Another important strength is that this study presents qualitative and quantitative data on the participants' satisfaction with the treatment. GROw is a comprehensive multi-component program with techniques that have been found to be effective in reducing grief symptoms and other associated psychological problems and it was created based on the most recent findings related to the treatment of PGD (e.g., Campos et al., 2019; Hoffmann et al., 2018; Shear et al., 2016). Lastly, this iCBT program can reach more people than a face-to-face intervention and, therefore, might be cost-effective. The GROw program is one of the few online self-applied treatments created for adults with PGD, and, as far as we know, it is the first iCBT for PGD created in the Spanish language. This program could reach many people in Spanish-speaking countries.

Limitations

One of the limitations of this study is related to the sample size because efficacy and satisfaction data were obtained from a small sample of four individuals. However, it has been argued that this number is sufficient to generate an adequate number of replications of the treatment effect (Dallery and Raiff, 2014). Another limitation is related to the reason for dropping out, given that this information could not be obtained despite several attempts to contact patients by mail and telephone. Qualitative information about the Emotional Monitor app was not reported either.

It should be noted that the diagnostic interview used in this study (The Structured Clinical Interview for Complicated Grief; SCI-GC) was published in 2015 (Bui et al., 2015b), whereas the PGD diagnostic category was introduced in the ICD-11 in 2019 (World Health Organization, 2019). Despite this, the SCI-GC was created based on the criteria proposed for the DSM-5 and the ICD-11 (Prigerson et al., 2009). d In fact, the SCI-GC is an evaluation instrument that follows the symptom (e.g., longing for the deceased) and time (at least 6 months from death) criteria proposed in the ICD-11.

Finally, efficacy results of this investigation were inconsistent in the participants on the different measured variables, and so more studies are needed. The results support scaling up the treatment using more complex designs with larger samples (i.e., randomized controlled trials comparing GROw with active conditions) that also include non-active control groups to evaluate the influence of the passage of time without intervention. Even though the implementation of different lengths of baseline assessments minimizes the threats to internal validity, we cannot absolutely guarantee that the changes are not related to spontaneous recovery in some of the cases.

In sum, despite the limitations, the present study clearly shows strong potential for this intervention as an alternative to face-to-face therapy for PGD. Patients reported high usability and satisfaction with the intervention, making it possible for more people in need to easily access an evidence-based treatment for PGD.

Funding

This work was supported by the 2018 Research Promotion Plan of the [UJI-2018-57]; . Programa Estatal I+D+I [RTI2018-100993-B-100]; Plan to promote research at for 2018 [PREDOC/2018/45]; and CIBEROBN, an initiative of the [CB06 03/0052].

Declaration of competing interest

The authors declare that they have no competing interests.