Evaluation of the implementation of an internet intervention in well-baby clinics: A pilot study.

PURPOSE: Despite promising results, internet interventions are not widely accessible or well-integrated in health services. The objective of this study was, therefore, to examine the implementation of an internet intervention ('Mamma Mia') for the prevention of perinatal depression in Norwegian well-baby clinics (i.e., primary care).
METHODS: Mamma Mia begins in mid-pregnancy and lasts up to 6 months after childbirth. It consists of 44 online sessions, supported by midwives and public health nurses at up to 5 contact points during this period, following principles of empathic communication. Well-baby clinics offer free, universal services to all pregnant women and children aged 0 to 5 years in Norway and were recruited via an intermediary organization for this study. Data were collected at pre- and post-training, but before the delivery of Mamma Mia in clinics, and at 3 and 6 months follow-up. Quantitative and qualitative data were used to evaluate the training in Mamma Mia, examine program implementation (i.e., number of pregnant women registered for the program), and identify barriers and enablers of implementation.
RESULTS: Twenty-four self-selected healthcare professionals from 14 well-baby clinics were recruited, for this study. Training increased participants' knowledge about Mamma Mia and exceeded their expectations. The program review and implementation plan were necessary training components. Implementation climate was related to the number of colleagues working with Mamma Mia and overall satisfaction with implementation, while characteristics of the intervention predicted the number of registered women at 6 months. Organizational re-structuring, leadership, and competing activities were identified as barriers to implementation that need to be considered further.
CONCLUSIONS: The dissemination and implementation of a health-service supported internet intervention appears to be promising but requires further research.

Introduction

It has long been known that internet interventions can be effective for mental health problems such as stress, anxiety, and depression in general or clinical populations (Heber et al., 2017; Karyotaki et al., 2017; Spek et al., 2007). In recent years, there has also been a greater interest in previously understudied sub-groups in research on internet interventions, such as pregnant and postpartum women. In the first systematic reviews on perinatal women, Ashford et al. (2016) concluded that computer and internet-based interventions seem promising for perinatal mental health, while Lee et al. (2016) conducted a meta-analysis where they found, more specifically, that therapist-supported internet-based cognitive behavioral therapy (iCBT) can improve depression among postpartum women. A similar meta-analysis, that included more and recent studies, confirmed these findings and found that iCBT could also improve stress and anxiety in new mothers (Lau et al., 2017).

All these systematic reviews concluded that this is still a developing field and point toward several future challenges for research such that there is a lack of studies on internet interventions based on principles other than CBT, of exclusive self-help programs, and that all of the interventions considered, were delivered after childbirth (Ashford et al., 2016; Lau et al., 2017; Lee et al., 2016). A recent study by Forsell et al. (2017) has indeed investigated the effect of iCBT among pregnant women and found positive results on antenatal depression, but there are still no studies of internet interventions that follow expectant mothers from pregnancy and through childbirth. This may be particularly important in view of the fact that maternal depression may occur before, during pregnancy, and after childbirth (Wisner et al., 2013), thereby making it important to prevent and intervene throughout the entire perinatal period.

Despite promising results, internet interventions are still not very accessible, widely disseminated or well-integrated in health services. There have been some discussions about various dissemination and implementation strategies in the literature such as service delivery models (Andrews and Williams, 2015; Batterham et al., 2015), and papers describing ‘lessons learned’ in the development and delivery of iCBT (see e.g., Hadjistavropoulos et al., 2011). However, a recent review of the existing research literature, showed that there are few, if any, primary implementation studies on internet interventions for depression (Drozd et al., 2016). It appears that the available knowledge of their implementation, in general, is very poor, and especially aspects related to leadership and organization. In fact, concerns about the integration of iCBT in the mental health care systems and their long-term sustainability were also shared and expressed by managers and clinical staff in a recent multi-national, comparative study on iCBT (Folker et al., 2018). However, questions about what activities must be carried out to implement and sustain an internet intervention in practice, remains unanswered. There is thus a gap between what we know about internet interventions (i.e., knowing) and how we translate this knowledge into practice (i.e., doing). To bridge this knowledge-action gap, we need to understand the barriers and enablers for the delivery and sustainability of internet interventions in practice.

The objective of this study was, therefore, to examine the implementation method for an internet intervention (‘Mamma Mia’) designed to prevent perinatal depression in a primary care setting (i.e., well-baby clinics). More specifically, we first aimed to evaluate knowledge gain, recommendations, and expectations of training, and identify which training components that were regarded as the most and least useful. Second, it was hypothesized that there would be a positive relationship between the number of pregnant women registered for the program, implementation climate, and participants' report of the integration of Mamma Mia in their well-baby clinics. Finally, we examined whether attitudes and program implementation fit predicted the number of pregnant women registered for Mamma Mia in well-baby clinics, and identified barriers and enablers of the implementation, which emerged during coaching.

Methods

Design

This study was a pilot evaluation of the implementation of a guided internet-based intervention in well-baby clinics. This means that the primary focus for the study was on the implementation process and outcomes, rather than the intervention objectives. The study was approved by the Norwegian Centre for Research Data (www.nsd.no; project number: 41361).

Internet-based intervention

‘Mamma Mia’ was originally developed as an unguided, self-help program available for PC/Mac, smartphones, and tablets. The goal of the program is to (1) prevent the onset or development of depression and (2) enhance subjective well-being from mid-pregnancy and up to 6 months after childbirth. The main program components include (1) identification and monitoring of depressive symptoms, (2) help and support for depressive symptoms, (3) maternal mental health and quality of life, (4) couple relationship and (5) the child's development. Mamma Mia is a universal preventive intervention that targets all expectant mothers, most of which will experience few symptoms of depression. Thus, the program emphasizes components 3–5 as described above (i.e., health promotion). However, all women are screened for symptoms of depression, using the Edinburgh Postnatal Depression Scale (EPDS; Cox et al., 1987), at 6 timepoints in the program. Women with an EPDS-score ≥ 10 (i.e., cut-off for depression) are provided with immediate, in-program help and support based on metacognitive therapy (Wells, 2009), and are encouraged to seek help by either calling a mental health hotline or contacting their midwife, public health nurse (PHN) or general practitioner (GP). Further details about the program components are also described elsewhere (Drozd et al., 2015; for a demonstration, see: Changetech, 2015).

Initial research examined the feasibility of the program (Haga et al., 2013), before the commencement of a randomized trial (for protocol, see: Drozd, 2013). In parallel to the trial, perinatal women were interviewed about the quality of the program and recommendations for future improvements. The results suggested that (optional) support or guidance could be beneficial, even for women low on depressive symptoms (Drozd et al., 2017b). Consequently, we developed clinical and implementation guidelines for use in well-baby clinics (Drozd et al., 2017a), including up to 5 contact points with midwives and PHNs in a blended care model, and a supplementary training for the delivery and implementation of Mamma Mia. The guidelines for clinical use were carefully designed to fit with existing national guidelines for pre- and postnatal care for well-baby clinics (Norwegian Directorate of Health, 2005, Norwegian Directorate of Health, 2013), whereas the implementation guidelines were designed to address the core components of implementation according to the Active Implementation Framework (see Procedures below; Fixsen et al., 2005).

The “blending” combines the Mamma Mia program, which is pre-structured protocol with some built-in tailoring, with face-to-face follow-up from midwives and PHNs. According to the clinical guidelines, midwives and PHNs are supposed to follow principles of empathic communication (Brudal, 2014; Rogers, 1951) by drawing attention to women's mental health, and to raise their affective consciousness, reflect on their experience, and help making sense of their joys and worries during the perinatal period. The important thing, is not to discuss the program per se, although face-to-face contact may improve adherence rates, but to allow midwives and PHNs to the woman's specific needs, discuss her thoughts and feelings, and assist with practical application or adaptations of program contents to her unique situation.

Participants and clinical settings

Participants were recruited via the Regional Center for Child and Adolescent Mental Health's website (RBUP; www.r-bup.no) from December 2014 to February 2015. RBUP is an intermediary organization devoted to aiding, strengthening, and developing primary and secondary child and adolescent mental health services. Their role is to offer supplementary education, research, and act as an intermediary between program developers and health services that want to adopt these, which implies that the health services initiate contact with RBUP for support and quality improvement of their services. Therefore, participants were recruited via RBUP based on self-selection.

The primary target group for the implementation of the current internet intervention, were midwives and PHNs in well-baby clinics. Midwives and PHNs are (a) registered and licensed in nursing, (b) with a 1- or 2-year post-graduate education, (c) have experience with perinatal women, and (d) providers of primary community services. Well-baby clinics offer free, universal services to all pregnant women and children aged 0 to 5 years in Norway. In 2016, about 85% of all pregnant women used the midwifery service offered at well-baby clinics and 98–100% of new parents attended the routine child-health examinations during the first year postpartum (Statistics Norway, 2017). The majority of the target group for the program can be reached by delivery in well-baby clinics, including mothers with or in high risk of depression (Glavin and Schaffer, 2013). Other professionals working in well-baby clinics or with specialized care of pregnant women in primary care settings such as community psychologists and GP, were also considered eligible. However, these were considered a secondary target group for delivery of the program for this study.

Procedures and materials

In the following, we describe the method for implementation related to the competency drivers (i.e., staff qualifications, training, and coaching), which are not covered in detail in the clinical and implementation guidelines (for further details about the guidelines, see Drozd et al., 2017a). The goal of the implementation method is that all participants should be able to deliver and use Mamma Mia in their line of work with women. Participation, therefore, assumes that the clinics develop a plan for the implementation of Mamma Mia, and requires that the clinics has an extended collaboration with other professionals working with pregnant and postpartum women in the municipal primary care setting.

Recruitment and staff selection

The recruitment and staff selection criteria for the delivery of Mamma Mia in well-baby clinics are described above (see Participants and clinical settings). Following the implementation guidelines, practitioners attending the training in Mamma Mia must develop a plan for involving key staff members that are necessary for operating and delivering the program in practice. This will, most often, involve leaders, but may also include administrative staff and other personnel that may be required to fulfill important functions or help to solve specific tasks (e.g., office workers to provide promotional materials to pregnant women or IT staff to upload program information on the clinic website).

Training

Training and coaching was primarily, but not exclusively, aimed at midwives and PHNs; that is, the practitioners that were to deliver Mamma Mia as part of their clinical practice, because practitioners were also responsible for the implementation of the program at their clinic. At first, practitioners attended a 2-days pre-service delivery training, covering: (a) an introduction to postpartum depression, child development, and internet interventions, (b) a review of the Mamma Mia program (incl. practical and experiential exercises and discussions), (c) skills training in the delivery of Mamma Mia to pregnant and postpartum women, and (d) preparation of a plan for the implementation of Mamma Mia for each local well-baby clinic.

During training (see below), clinical staff were provided with written educational materials, in addition to the program. Clinicians received an information brochure for pregnant women, mainly to be used as an orientation about Mamma Mia for women who visit their midwife before gestational week 17, which is too early to start the program, or women who need time to consider using the program. Clinicians were also given pamphlets to aid in their delivery of the program with pregnant and postpartum women, respectively. The pamphlets consist of a summary of the program contents up to the relevant contact point at the clinic, and a short version of the clinical guidelines, intended as a useful reminder and resource to be used in face-to-face nurse-woman interaction (Supplementary Fig. 1, Supplementary Fig. 2).

Coaching and fidelity

After pre-service training, participants retreated to work on the implementation and delivery of Mamma Mia to women at the clinic for 3 months. All practitioners were offered one monthly coaching session, lasting for about 1 h, for 2 months; mainly including implementation coaching, but also clinical coaching. The coaching consisted of (a) a review of implementation outcomes (e.g., number of women registered for the program), (b) discussion of barriers to implementation, and (c) a written summative performance feedback. The overall goal of coaching was to understand how outcomes and barriers relate to the implementation plan and identify next action steps in the implementation.

At 3 months after pre-service training, practitioners attended a 2–day maintenance seminar. The first day included (a) sharing and exchanging experiences with the delivery of Mamma Mia in their clinics, while the second day focused on (b) revising the local implementation plans. Practitioners were invited to submit their implementation plans for review within 6 months after pre-service training to obtain a certificate. The requirement for a certificate was documentation of work to shape professional practice toward Mamma Mia. In addition, some self-study was expected during the entire period (e.g., reading literature on Mamma Mia).

Data collection

Data were collected at pre- and post-training (i.e., both pre-service delivery), 3 and 6 months follow-up (i.e., post-service delivery).

At pre-training, participants completed a questionnaire with demographics such as age, gender, education, current profession, and previous experience with internet interventions, and information about their clinic such as the number of colleagues, intervention methods provided in the clinic, and number of births per year. Training was assessed in terms of knowledge gains from pre- to post-training (“How much do you know about Mamma Mia?”) and the degree to which participants would recommend Mamma Mia to pregnant women and health personnel, respectively. Both knowledge and recommendations were rated on a 10-point Likert-scale from “1 = very little/to a very small extent” to “10 = very much/to a very large extent”. They were also asked to assess their expectations to training, after the 2-days of pre-service delivery training. In total, five items were rated on a 3-point Likert-scale (−1 = “not good (must improve)”, 0 = “OK, as expected”, 1 = “very good (better than expected)”): What do you think about (1) the lectures, (2) group work and discussions, (3) skills training, (4) review of Mamma Mia, and (5) review of the implementation plan. Finally, participants were given two open-ended qualitative question where they were asked to identify the most and least useful components of training, respectively (“What has been the most/least useful for you during the training?”).

The Implementation Climate Questionnaire from the Measures of Implementation Components (MIC; Fixsen et al., 2008) was administered at 3 months follow-up. This questionnaire has 27-items that reflect the clinics' day-to-day operations and how staff members view their work in relation to the use of innovations (i.e., Mamma Mia) in their organization (e.g., “Mamma Mia is being implemented in the clinic as prescribed by RBUP”). The following 7 program implementation variables were also collected from the MIC: (1) “How many of your colleagues are working with Mamma Mia?”, (2) “How many of your colleagues are working with other evidence-based programs?”, (3) “Mamma Mia is well-integrated in my workplace” (1 = “completely disagree” to 5 = “completely agree”), (4) “How satisfied are you with the implementation of Mamma Mia?” (1 = “very dissatisfied” to 5 = “very satisfied”), (5) “How much time do you have set aside at work with Mamma Mia?” (1 = “0–20%” to 5 = “81–100%”), (6) “Do you have enough time to work with Mamma Mia?” (0 = “no”, 1 = “sometimes”, 2 = “yes”), and (7) “Sooner or later I am going to stop using Mamma Mia” (1 = “completely disagree” to 5 = “completely agree”).

Attitudes toward internet-based interventions were assessed pre-training by using the Computer-Assisted Therapy Attitudes Scale (CATAS; Becker and Jensen-Doss, 2013). CATAS has eight items that are rated on a 5-point Likert scale, where higher scores indicate more positive attitudes. It contains two sub-scales of which three items are related to the comfort sub-scale (e.g., “I feel apprehensive about using computers during therapy”) and five items refer to the efficacy sub-scale (e.g., “Using computers in therapy will interfere with rapport”). The terms “computers” and “therapy” were exchanged with “internet-based interventions” and “practice” for all items in this study.

The e-Health Implementation Toolkit (e-HIT; Murray et al., 2010) is a 23-item tool that was used at post-training to assess how well Mamma Mia seemed to fit with clinics organizational priorities and structures, the nature of the intervention, and the impact of the intervention on staffs' work. The e-HIT consists of three areas; (1) context (e.g., “The intervention is entirely at odds/compatible with all current and planned local policies”), (2) intervention (e.g., “The intervention will disrupt/facilitate health professional-client interactions”), and (3) workforce (e.g., “There is un-/likely to be a period of increased workload during the implementation”) that are rated on a scale from 0 to 10 where higher scores indicate more facilitating conditions.

The primary implementation outcome was the number of registered pregnant women for the Mamma Mia program collected by means of autogenerated log data in the program and extracted at 6 months follow-up. In addition, researchers' field notes from the coaching were used to identify any barriers and facilitators that occurred during implementation.

Data analysis

Descriptives were used to characterize the participant sample (i.e., individual-level data) and well-baby clinics (i.e., service-level data). Knowledge and recommendations from pre- to post-training was examined by means of paired-samples t-tests with an alpha-level of 0.05, and Cohen's d as a measure of effect size. Expectations at post-training were examined using a one-sample t-test and the deviance of the sample mean from the null value (i.e., as expected) in units of the sample standard deviation, as a measure of effect size. Participants' open-ended, qualitative responses to the usefulness of the training, and notes from coaching, were analyzed thematically (Braun and Clarke, 2006), to assess the usefulness of training, and barriers and facilitators of implementation discussed during coaching.

Pearson product-moment correlations were applied as a measure of the relationship between participants' report of the integration of Mamma Mia in the clinic, implementation climate, and the number women registered for the program. The coefficient of determination (i.e., r2) was estimated for significant correlations to indicate explained variance of one variable from the other variable. Finally, negative binomial regression analyses were conducted on aggregate scores of attitudes toward internet-based interventions and integration of Mamma Mia in each clinic to predict the number of women registered for Mamma Mia at the well-baby clinics. The number of births at each health clinic was taken into the regression models as an exposure variable (i.e., an exposure variable indicates the number of times an event could have happened).

Results

Participants and well-baby clinics

In the present study, 24 participants from 14 well-baby clinics consented to participate (for characteristics, see Table 1). Participants had either completed a bachelor's degree and supplementary education in public health or psychiatric nursing, a master's degree in midwifery, or a 6-year professional degree in clinical psychology (i.e., all had >3 years in college or university). It is also noted that more than one-third of participants had supplementary education in the Edinburgh-method, which is method for screening and providing supportive counselling to prevent severe postpartum depression, and that well-baby clinics varied in organizational size, as determined by number of colleagues and births per year.

Table 1

Sample characteristics.

Participant (n = 24)
Age (years, M ± SD)	52.6 (4.8)
Gender (n, %)
Female	23 (95.8)
Male	1 (4.2)
Current profession (n, %)a
Midwife	13 (54.2)
Public health nurse	8 (33.3)
Psychiatric nurse	1 (4.2)
Psychologist	1 (4.2)
Edinburgh-method (n, %)a
Yes	9 (37.5)
No	14 (60.9)
Previous experience with internet interventions (n, %)a
Yes	4 (16.7)
No	19 (79.2)
Workshop attendance (n, %)
4 days (training + maintenance seminar)	16 (66.7)
2 days	8 (33.3)

Numbers and percentages adjusted for missing values.

Aggregate mean scores.

Evaluation of training

The first aim of this study was to assess the training in Mamma Mia by investigating participants' gains in knowledge, recommendations, and expectations, and, thereafter, to examine what participants identified as the most and least useful training components. There was a large and significant change in participants' knowledge about Mamma Mia from before (mean (M) = 4.67, standard deviation (SD) = 2.61) to after training (M = 8.38, SD = 1.12; t(20) = −8.39, p < .001, Cohen's d = 1.47). The degree to which participants would recommend Mamma Mia to health personnel (M = 9.50, SD = 0.82) and pregnant women (M = 9.75, SD = 0.45) was already high at pre-training. Consequently, there were no changes in participants recommendations, neither for health personnel (M = 9.44, SD = 1.09; t(15) = 0.37, p = .718) nor pregnant women (M = 9.69, SD = 0.60; t(15) = 0.57, p = .580). However, training significantly exceeded participants' expectations to training (M = 1.76, SD = 1.79, 95% CI = 0.85–2.28; t(17) = 4.07, p = .001, Cohen's d = 0.99).

Qualitative assessment of training

A review of participants' qualitative responses, identified the walkthrough of the Mamma Mia program as a particularly useful component. It provided participants with insight and trust in the program contents for delivery among pregnant women. Participants explained that the program review made them excited as well as inspired by the content and seemed convincing in terms of the program's usefulness for women. A second theme, was the implementation training. Participants stated that learning about ‘how–to–implement’ Mamma Mia in their clinics was useful and that the implementation training would be especially advantageous in their continued work. The least useful training component, however, was information about perinatal depression. Some participants expressed that they already had extensive knowledge about perinatal mood disorders. Thus, they did not find that they learned anything new from the information given on this topic. This corresponds to the results in Table 1 above, where more than one-third of the participants had previously received training in methods for postpartum depression.

Correlations

The second aim was to investigate whether there would be a positive relationship between the number of pregnant women registered for the program, implementation climate, and the program implementation in well-baby clinics. Six months after training, the well-baby clinics had recruited and registered 310 pregnant women in total, as measured by log data. The number of pregnant women did not, however, correlate significantly with any of the program implementation variables from the MIC questionnaire (see Table 2). In contrast, implementation climate did correlate positively with the number of colleagues working with Mamma Mia at the clinic (r2 = 0.397) and satisfaction with the implementation of Mamma Mia (r2 = 0.518).

Table 2

Correlations between the number of pregnant women recruited at the well-baby clinic and program implementation variables.

Implementation variables	1	2	3	4	5	6	7	8	9	M	SD
1. How many of your colleagues work with MM?	–									2.5	1.5
2. How many of your colleagues work with other EBPs?	0.53	–								8.3	5.9
3. MM is well integrated in my workplace	0.37	0.26	–							3.1	1.3
4. How satisfied are you with the implementation of MM?	0.51	0.00	0.44	–						3.2	1.3
5. How much time do you have set aside at work for MM?	−0.12	−0.32	0.12	0.22	–					1.1	0.3
6. Do you have enough time to work with MM?	−0.01	−0.34	0.06	0.54	0.08	–				1.2	1.4
7. Sooner or later I am going to stop using MM	−0.21	−0.54	−0.54	−0.12	−0.13	−0.31	–			1.8	0.7
8. Implementation climate for MM at clinic	0.63⁎	0.54	0.60	0.76⁎	0.18	0.47	−0.50	–		26.5	6.8
9. Number of women recruited at clinic	0.39	−0.17	−0.34	0.27	−0.16	0.56	−0.02	0.34	–	22.1	23.1

Note. MM = Mamma Mia. EBP = evidence-based practice.

p < .05 level (two-tailed).

Negative binomial regression analysis

The final aim was to test whether clinical staffs' attitudes toward internet-based interventions and program implementation fit, would predict the number of pregnant women recruited and registered for Mamma Mia in well-baby clinics. Thereafter, to identify other barriers and enablers of the implementation during coaching. Results from the negative binominal regression models indicated that the e-HIT intervention sub-scale was the only significant predictor of how many women a well-baby clinic recruited to Mamma Mia, offset by the number of births (Table 3). The rate ratio indicated that when a clinic's e-HIT intervention mean score increased by one point, there was a 27% increase in women recruited.

Table 3

Negative binomial regressions on the number of participants recruited for Mamma Mia, offset by number of births at the well-baby clinics.

Variable	Rate ratio	95% CI		p
		Lower	Upper
CATAS-comfort	0.75	0.38	1.48	.404
CATAS-efficacy	1.23	0.82	1.86	.318
e-HIT context	1.03	0.97	1.09	.317
e-HIT intervention	1.27	1.05	1.54	.015
e-HIT workplace	1.03	0.92	1.16	.563

Qualitative assessment of implementation coaching

Of 14 well-baby clinics, 9 (64.3%) received implementation coaching and 7 (50.0%) clinics were certified by submitting complete implementation plans that were approved by the implementation coach. The first theme during implementation coaching, were challenges related to organization and leadership. For example, two of the non-certified clinics could not make the implementation of Mamma Mia a priority at the time due to major organizational changes or having a period with high turnover among leaders and employees. Some clinics also experienced competing activities (e.g., introduction of other methods at the same time) and others strived with engaging and involving leaders to a sufficient degree in the work with Mamma Mia. Nevertheless, participants at all clinics indicated a positive attitude toward using Mamma Mia in their clinics and several received positive feedback from pregnant women.

A second theme in implementation coaching was concerns with the extensiveness of the implementation plan, although participants recognized that the implementation guideline and planning was necessary. These participants were mainly worried that it would require too much additional work and organization and were concerned with carrying the responsibility for the implementation. However, participants who had worked the most with the implementation plan, expressed surprise at how easy the implementation work was when they first got started and how little time-consuming it was. Thus, the implementation plan had a mixed reception. On the one hand, it was perceived as comprehensive and a bit overwhelming to use for some (i.e., a barrier). On the other hand, it turned out very valuable for clinics that managed to utilize the plan in their implementation work (i.e., a facilitator).

Discussion

The aims of this study were to evaluate the training in Mamma Mia, examine the relationship between program implementation variables, and investigate barriers and enablers of the implementation. The results showed that training increased participants knowledge about Mamma Mia, maintained their willingness to recommend Mamma Mia, and exceeded their expectations about training. The program review and review of the implementation plan were considered the most useful training components, while information about perinatal depression was least useful. The number of women recruited and registered for Mamma Mia, did not correlate significantly with any of the implementation variables. However, there were moderate-to-strong positive correlations between implementation climate and the number of colleagues working with the program and overall satisfaction with the implementation. Furthermore, characteristics of the intervention itself was the only predictor of the number of women registered for the program. During implementation coaching, a few additional barriers and facilitators emerged, that are important to consider and examine more closely in future research.

Despite a modest number of participants and clinics, this study shows that a 4-day training and maintenance seminar, where clinics can acquire new knowledge and competency, share and discuss their experiences, can reach a number of clinics and recruit a substantial number of users for an internet intervention, within a period of 6 months. The findings indicated that participants and clinics had «bought into» (i.e., would highly recommend) the program. Their willingness to recommend Mamma Mia was high at the outset (i.e., ceiling effect), which makes it encouraging that their willingness to recommend the program was maintained after training, and after learning about all of the work required for its implementation.

The findings also indicated that participants, generally, had positive attitudes toward internet interventions and the implementation of Mamma Mia. It is important to remember that these participants and clinics were self-selected, which may be different under other circumstances where, for example, implementation is mandatory and/or professionals are unfamiliar with internet interventions. In a study, among psychologists, Nordgreen and Havik (2011) found that those that were not familiar with or did not use internet- or computer-based programs, would need more information and training before beginning to use such programs. Furthermore, although some previous studies suggest that mental health professionals hold positive attitudes toward internet interventions (Becker and Jensen-Doss, 2013; Gun et al., 2011; Vigerland et al., 2014), other studies show that there is still considerable skepticism toward applying iCBT (Folker et al., 2018; Musiat et al., 2014; Topooco et al., 2017; Waller and Gilbody, 2009) As an example, a Norwegian study by Wilhelmsen et al. (2014), found that despite GPs' positive attitudes, the usage of iCBT was low even after training. It is, therefore, important to consider issues with acceptability and attitudes toward internet interventions, although this study shows that this may be less of a challenge under the voluntary settings described herein.

It was interesting that aspects of the intervention itself such as its' perceived effectiveness, ease of use, security, and ability to facilitate practitioner-client interaction (i.e., the e-HIT intervention sub-scale), was related to the number of pregnant women recruited at the clinics. This suggests that the design of the intervention and the way the intervention is perceived and conveyed to the participants is important, as shown in much previous research (Cugelman et al., 2011; Kelders et al., 2012; van der Vaart et al., 2014). It was more surprising, however, that the impact of the implementation on participants' and clinics' work was unrelated to recruitment. Experiences from the implementation coaching, nevertheless, indicate that not all clinics were able to utilize the implementation plan as intended and it was perceived as being comprehensive. This may be because (organizational or implementation) planning is not a routine task or part of clinical staffs' professional roles or job instructions. Once started, however, clinical staff expressed surprise at how easy it was to work with the implementation plan and that it was considered necessary part of the implementation.

Clinics that did not manage to fully develop implementation plans underwent major re-organization, experienced leadership barriers, and/or competing activities. These experiences point toward several barriers to implementation that are relevant for interventions or evidence-based practices more generally and suggests that these factors also need to be considered for health-service supported internet interventions. Practically, it may suggest the need for a systematic (organizational) assessment of readiness for change and barriers and facilitators prior to implementation (e.g., Shea et al., 2014), and that certain admission criteria or requirements must be fulfilled in the clinic, before full-scale implementation can occur. During or post-implementation, it may also be necessary with either ongoing, continued implementation coaching or for as long as-needed basis. This is, most often, provided for therapists delivering internet interventions (see e.g., Drozd et al., 2016; Hadjistavropoulos et al., 2011), but may also apply to clinics and other health services as well.

Limitations

This study has some limitations, such as the small sample size, which requires that the quantitative results must be interpreted with caution. In general, however, pilot studies such as this study, are useful to generate data for future sample size estimations where there are no data from previous studies to inform this process (Thabane et al., 2010), and must be considered accordingly. As an example, some of the correlations and rate ratios were rather high, despite being non-significant. This demonstrates that the study was underpowered, but also that very large sample sizes may not be needed in future studies. This limitation does not, however, necessarily encompass the qualitative evaluation of the training and implementation.

Another limitation is that we did not include any measures of clinical or implementation fidelity. This was, of course, not the aim of this study and it does not mean to suggest that complete fidelity is required for successful outcomes or that adaptations to local knowledge, clinics, and systems, are not allowable. In fact, it appears that flexibility within fidelity may be the preferred perspective (see e.g., Kendall and Beidas, 2007). But it is important that clinical staff follow protocols to ensure a certain standard for quality of care and delivery of internet interventions, both for clinical and research purposes.

These findings may not be applicable for implementation under mandatory settings. The participants in this study were positive toward internet-based interventions and Mamma Mia. This may be the result of a voluntary, self-selection among participants, which may have produced a positivity bias among those who attended the training. Thus, in future studies, it may still be important to evaluate clinical staffs' ratings of recommendation, attitudes, and other proxy measures of acceptability, especially among those that may not have attended training or are unfamiliar with a given program.

Finally, the correlational approach in this study, makes it impossible to determine the (causal) relationship between implementation climate, number of colleagues working with Mamma Mia and satisfaction with implementation. Currently, it is not possible to know whether health services with a good implementation climate makes it easier to involve colleagues in the implementation of new methods or whether more colleagues working together creates a good implementation climate.

Conclusion

The results from the present study suggest that it is fully possible to disseminate and implement a health-service supported internet intervention in primary care settings. It can provide staff in well-baby clinics with a new method to prevent perinatal depression and support women during pregnancy and after childbirth. Furthermore, it has enabled us to revise the training and implementation by identifying important training components and implementation barriers and facilitators and allows us to prepare for future studies and implementation. It is also one of the few primary studies on the implementation of internet interventions and, thus, serves to inform other implementation studies.

The following are the supplementary data related to this article.

Supplementary Fig. 1

Mamma Mia for practitioners - parents and children.

Supplementary Fig. 2

Mamma Mia for practitioners – pregnancy.

Funding

This research was supported by the Norwegian Women's Public Health Association [project number H3/2013] and the Research Council of Norway [project number 213737].

Declarations of interest

None.

Compliance with ethical standards

All procedures involving human participants were in accordance with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. Informed consent was obtained from all individual participants included in the study. A formal consent was not required for aggregate and service-level implementation outcomes as these contained no directly or indirectly personally identifiable data and, hence, were anonymous. The study was approved by the Norwegian Centre for Research Data [reference number 41361].