Effectiveness of a neonatal COVID-19 response project: A mixed-methods evaluation using the Donabedian model.

Objective: This article outlines notable findings of a service evaluation of a COVID-19 response project, the Nurture Project (July 2020-March 2021). Method: The Donabedian structure-process-outcome model was used. Mixed-methods online surveys and organisational data were analysed using reflexive thematic analysis and statistical analysis methods.
Results: Most staff and service users were satisfied with the project, reporting positive benefits to mental health, child development, and wellbeing. However, project outcome measures (Generalised Anxiety Disorder Scale GAD-7 and the Patient Health Questionnaire PHQ-9) were statistically non-significant.
Conclusion: Although the project was considered successful, recommendations for future service evaluation methods, outcome measurement, and future research are provided.

Introduction

The first cases of COVID-19, a coronavirus, were reported in the United Kingdom (UK) in January 2020 (Public Health England, 2020c). Social distancing measures were introduced (Cabinet Office, 2020; Public Health England, 2020b, d, e), along with additional guidance for individuals considered clinically vulnerable to COVID-19 (Office for National Statistics, 2020; Public Health England, 2020a; Royal College of Paediatrics and Child Health et al., 2020a, b).

Critical changes were made to National Health Service (NHS) provision, including maternity services (The Royal College of Midwives et al., 2020; Royal College of Obstetricians and Gynaecologists, 2020; The Royal College of Midwives and Royal College of Obstetricians and Gynaecologists, 2020; The Royal College of Midwives et al., 2020); and neonatal and paediatric services, including the restriction of parental access to neonatal units (NNU) (British Association of Perinatal Medicine, 2018; Royal College of Paediatrics and Child Health, 2020; Royal College of Paediatrics and Child Health et al., 2020c, d).

Bliss (2020) acknowledged the need for restrictions but critically stated “parents are not visitors”. The organisation highlighted the variability of hospital-implemented guidance, with some policies restricting parental access to one parent, with time restrictions of as little as 2 hours per day. Restrictions led to consequences for both infant and family outcomes, parental mental health, confidence, and bonding (Bliss, 2020).

The subject of evaluation is Leo's Neonatal, a charity founded in 2018, that supports families who have experienced a NNU admission in the North-East of England and Cumbria (Leo's, 2021). The organisation launched the Nurture Project, in July 2020. The project aimed to provide support to families who had experienced a NNU admission during the COVID-19 pandemic, or who had experienced a prior NNU admission and were affected by the pandemic, supplementing NHS provision. The swiftly-created project was designed in response to a 337% increase in demand for the organisation's services, during the initial months of the pandemic (Leo's, 2021).

The Nurture Project offered a wide-ranging amalgamation of both group and individual service options to support mental health, parent-infant attachment, and child development. Delivered by clinicians, registered practitioners, and peer support staff, services included peer support (parent-to-parent support), counseling, cognitive behavioural therapy (CBT), eye movement desensitization and reprocessing, clinical psychology, and video interaction guidance and occupational therapy. Group services provided peer support and clinical provision to support parent-infant attachment, play, child development (physical, social, emotional, and sensory development), de-medicalisation from the NNU, and general social and emotional wellbeing. Opportunities were provided to families for individual swimming sessions at community venues.

This paper reports on the Nurture Project pilot service evaluation. The third sector faces an increasing demand for evaluation engagement (McCoy et al., 2014). Evaluation is needed to measure the impact of performance assessment in a complex funding landscape (Grant Thornton, 2018), to provide evidence of performance to stakeholders and funders, and to identify areas of improvement (National Council for Voluntary Organisations, 2021). Organisations face mounting pressure to demonstrate the appropriateness of resource use and efficient and effective service delivery (Department of Health, 2012).

Service evaluation is beneficial to gather intervention data such as appropriateness of outcome measurement, and attendance figures, and act as a precursor to future research studies (Polley et al., 2016), in addition to solving or mitigating problems and enhancing service quality (Owen, 2006). However, evaluation can be challenging due to frequently changeable, innovative, and complex service provision (McCoy et al., 2014). This evaluation must be considered pragmatically, due to the ‘real-world’, reactive, third-sector service provision during a global pandemic.

The primary aim of the service evaluation was to evaluate the effectiveness of the Nurture Project. Secondary aims were to investigate the characteristics of those who engaged in the Nurture Project to determine implications for accessibility; and to provide preliminary estimates of change (improvement or deterioration), over time, during the Nurture Project and its interventions (for example, including clinical difference and statistical significance).

Knowledge will contribute to future service decision-making, development, and partner collaborations between the third sector, and statutory services (Gentry et al., 2018).

Methods

The Donabedian structure-process-outcome (SPO) model was applied to structure the evaluation (Donabedian, 1966, 2005). The model is commonly used as a framework for healthcare evaluation and audit (Gardner et al., 2014; Gentry et al., 2018; Kobayashi et al., 2011; Rai and Wood, 2018) and is recommended by NHS England and NHS Improvement for service improvement and evaluation projects (NHS England and NHS Improvement, 2021).

Data collection

Due to feasibility and resource constraints, the mixed-methods evaluation was conducted by secondary data analysis via a charity-academic partnership. Data sources included routinely collected organisational project data (n = 82 adults, n = 95 children), in addition to two, online, post-service, cross-sectional surveys of staff (n = 17) and service users (n = 52) using Survey Monkey (Momentive, 2021). Participants responded to questions presented as 5-point Likert-type response items (Field, 2016), multiple-choice, and free-text response options. A purposive sampling method was used by the organisation to gather feedback from staff, volunteers, and service users who participated in the Nurture Project (July 19, 2020 - March 31, 2021). Organisational data was collected by Leo's Neonatal staff and management team, anonymised, and provided to K.C. on a pre-designed spreadsheet.

Participants provided consent for their data to be included in organisational data analysis and during survey completion. Two eligible service users did not consent to participation and were therefore excluded from analysis.

Data analysis

All quantitative data analysis was carried out using the Statistical Package for the Social Sciences (SPSS) version 28 (IBM, 2021). The paired samples t-test was used to analyse outcome data (Christopher, 2017).

Reflexive thematic analysis (Braun and Clarke, 2019, 2021) was used to analyse qualitative data, using NVivo qualitative data analysis software version 12 (QSR International Pty Ltd, 2021). To ensure rigor, a steering group of charity representatives was routinely consulted to mutually discuss, reflexively consider, and reach a consensus on developed themes. This ensured accurate representation of findings and provided clarity regarding the field or organisational-specific language (Gentry et al., 2018). Verbatim quotes are used to illustrate findings.

Results

Aspects of the evaluation that are pertinent to COVID-19, third sector service delivery, and the evaluation of outcome.

Sample characteristics

Most service users were female (71, 86.59%), with a mean age of 32.76 years (SD 4.96), and from a white ethnic group (82, 100%). Most children requiring an NNU admission, were admitted after the January 1, 2018 (70, 85.37%).

Evaluation of structure

Findings of note included that all staff strongly agreed or agreed (17, 100%) that the project was a success, cost-effective, addressed the needs of families, inspired them as service providers, work was enjoyable and that service users were satisfied. Qualitative staff responses referred to positive aspects of the project in terms of person-centered individualised care; supporting child development and mental wellbeing; and organisationally in terms of iterative adaptation and flexibility of staff, services, and safety measures.

Peer supporters (n = 6) qualitatively responded that they felt that they were “giving back” and viewed the work as a “positive experience” due to “supporting families at a time when they felt very alone and isolated”. However, one participant highlighted the personal challenge of “having to juggle” the role when shielding their own child, describing it as “incredibly draining”.

Mode of delivery of services

Due to COVID-19 restrictions, most services were required to adapt to provide remote provision (for example by video communication, telephone, email, or text message). When asked, several staff cited government guidance and safety as themes relating to the mode of service delivery. Staff reported that “flexibility was key”, but that guidance regarding personal protective equipment, reduced numbers due to social distancing, and requesting clients to bring their own equipment, was challenging.

Staff reported advantages of remote provision including “improved attendance” due to “better availability” of client and staff members, improved “geographical reach”, along with benefits of “convenience, anonymity” and enabling greater “confidence in sharing” difficult material.

Whilst some participants felt that they would continue to offer the flexibility of online services, others felt that face-to-face provision remained preferable, offering a “more effective therapeutic relationship”. Participants felt in-person services enabled better observation and understanding of “the presentation of the family” and “non-verbal behavioural cues” of infants. One participant raised concerns, reporting that “safeguarding has been difficult to manage”. One participant felt the adaptation to community support provision, rather than support cot-side in the NNU was advantageous: “I felt that being able to speak to parents away from the cot-side allowed them to be more open and honest about their mental health. …” (peer support staff member).

Evaluation of process

Pertinent findings included that most service users self-referred to the project (68, 82.93%), with only 8 (9.76%) referrals received from a health professional. Most service users (14, 27.45%) were previously known to the charity, with 13 (25.49%) being made aware of the project by a health professional. Other awareness methods included advertising booklets, word of mouth from other sources, social media, and advertising materials.

56.10% (46) of service users were assessed within 10 days of receipt of referral with 19.51% (16) of service users receiving an assessment within 24 hours of referral receipt. Most service users were satisfied with the waiting time for initial and subsequent appointments, the convenience of appointment times, and the perceived competency and interpersonal skills of staff.

Evaluation of outcome

Satisfaction with modes of provision

Echoing the staff findings, service-user themes relating to the mode of delivery of services included access to services (flexibility), depth of interaction, and individual circumstances. Some participants felt remote provision enabled shorter appointment times and service contacts, better scheduling, convenience, and flexibility when considering childcare.

Several participants felt video communication was successful, bridging the gap between remote services and face-to-face delivery, and supported relationship development by being able to see the therapist/staff member, providing “human contact”. In contrast, one participant felt more comfortable using the telephone:

“ …. able to be more open and not have to put a brave face on” (service user).

Some participants felt that their personal circumstances at the time of the appointment affected their preferences such as childcare, in addition to COVID-19 safety requirements, appreciating available service options, and choice.

Impact of COVID-19

Participants reported a deterioration in their mental health, isolation, loneliness, and a loss of support systems, including childcare (Table 1 ). Several participants highlighted the impact of hospital restrictions such as attendance at medical appointments, having a birth partner present, only one parent permitted in the NNU, and visiting restrictions. Many participants expressed concerns for the health and wellbeing of their family including the impact on mental health, concern of separation if their child required hospital admission and being able to care for their child if they contracted COVID-19 themselves. Participants reported delayed hospital appointments, a lack of NHS face-to-face appointments, and little or no support for child development. These findings were echoed by staff members.

Table 1

The impact of COVID-19 on service users.

Theme	Sub-theme	Example Service User Quote
The impact of COVID-19restrictions	Support system and social isolation	… I was unable to receive support from family and friends, most couldn't attend [my child's] funeral … I was feeling isolated and alone
	Hospital restrictions	… we were only allowed to visit him for 2 hours. This caused unbelievable heartache, anxiety, stress, and low mood, feeling physically separated …
Concern for the health and wellbeing of the family	Concern of contracting COVID-19	COVID-19 not only caused me to deliver our son 6 weeks prematurely due to being in intensive care with bilateral COVID pneumonia … we were separated, isolated, critically unwell, and missed out …
	Delayed appointments and medical care	… missed postnatal depression diagnosis

Service user satisfaction

The themes developed highlighted the importance of interaction with staff and other service users with lived experience (Table 2 ). Multiple participants recognised the importance of purposeful pairing of peer-supporter and service user based upon the similarity of lived experience. Services were reported to aid mental health and wellbeing, reduce social isolation, and to provide reassurance and open communication.

Table 2

Service-user satisfaction.

Theme	Sub-theme	Example Service User Quote
Mental health and the wellbeing of the family	Lived experience and shared understanding	… I like that there's someone to talk to who has been through what you have …
	Mental health and wellbeing	I loved being heard, voicing concerns, and having thoughts validated …
	Parental role and relationships	… If you have ever struggled in identifying yourself as a parent, a mam, a caregiver, and not a HCP, nurse, doctor, then this is definitely for you!
	Isolation and social interaction	… alone …
	Open communication (reassurance and support)	… I feel like I've come a long way. It's refreshing knowing you can open-up and let all fears, emotions, and stresses out …
Service delivery and organisational processes of the NurtureProject	Not reported in this article.

Outcome measurement

Adult users were assessed on entry to and exit from the Nurture Project using two self-report measures, the Generalised Anxiety Disorder Scale 7 (GAD-7) which focuses on symptoms of anxiety, and the Patient Health Questionnaire (PHQ-9) which focuses on symptoms of depression (Gilbody et al., 2007; Hinz et al., 2016). The commonly recommended clinical cut-off for diagnostic acceptability, alerting clinicians to monitoring or intervention is ≥ 8 (GAD-7) and ≥10 (PHQ-9) (National Health Service and National Collaborating Centre for Mental Health et al., 2020).

The paired-samples t-test was used to compare means. Analysis of the full Nurture Project sample (n = 48) scores showed that prior to the project, the mean PHQ-9 score was 8.52 (SD 6.12) which decreased to 8.23 (SD 5.76), and the GAD-7 score was 9.85 (SD 5.89), which decreased to 9.65 (SD 6.03). Both were non-significant (p > 0.05), and neither achieved the minimal clinically important difference (MCID) (PHQ-9 = 3 and GAD-7 = 4). However, with clinical thresholds applied (n = 19 ≥ 10 PHQ-9; n = 27 ≥ 8 GAD-7), before the project, the mean PHQ-9 score was 14.95 (SD 3.61) which significantly decreased to 11.21 (SD 5.49) (p = 0.008), meeting the MCID. However, the mean GAD-7 score was 14.22 (SD 3.90) which was reduced to 12.07 (SD 5.25) and remained non-significant (p = 0.48). Further analysis of these data could be completed, such as regression to the mean (Barnett et al., 2005), to explore the impact of the absence of a control group for comparison and use of a sub-sample. However, this is beyond the scope of this paper.

Discussion

This evaluation was of a rapid, newly developed service during the COVID-19 pandemic. To the best of our knowledge, this evaluation is the first UK service evaluation of a third-sector neonatal COVID-19 response project. Due to speed and responsivity, the service was implemented before consideration of data collection tools, leading to missing data (Hibberd et al., 2021).

The evaluation was not pre-registered and did not have pre-existing criteria (Taylor et al., 2021). Due to no prior baseline outcome measurement scores, a comparison could not be made between the Nurture Project and the prior provision (Johnston et al., 2018). Caution with an interpretation of results is needed due to the small sample size, lack of control group, and regression to the mean effects alongside natural recovery trajectories (Taylor et al., 2021).

The absence of a control group limits findings as it is plausible that the mental health of the cohort may have deteriorated during the pandemic. There is potential for an increase in the value of the project results, due to the benefits of maintenance of mental health and wellbeing.

Conversely, the mental health of the cohort may have naturally improved at a significant level. Future research would benefit from the inclusion of a control group (Polley et al., 2016), and collaboration with academics and statisticians when designing such studies and future evaluations to avoid design pitfalls (Mellis, 2017). The inclusion of a cost-benefit economic analysis would be preferable (Thirthalli and Rajkumar, 2009).

The PHQ-9 and GAD-7 did not capture all project aims or reported service-user benefits. The data collected did not precisely address the outcomes (Gentry et al., 2018). Similarly, Carnes et al. (2017), reported non-significant findings relating to health, wellbeing, anxiety, or depression which contrasted with positive qualitative findings. The authors discussed the participant complexity and suggested that better outcomes may be related to both quality of life and general mental wellbeing, highlighting that the outcome measures may not have adequately captured these aspects.

A hybrid service delivery model of both remote and in-person service delivery should be considered to give families a choice of providers. Recording of group or online session content would allow access to families with unpredictable schedules (such as hospital appointments) and employment commitments. Remote-adapted services should be piloted to ensure accessibility with consideration of visual and audio content. The reported demographic profile of service users suggests that service user characteristics should be further examined to ensure accessibility of services. The organisation should continue to match peer support staff with service users based on the similarity of experience and implement a formal appointment process for peer support.

Whilst the findings of the project were generally positive, the reasons for the limited number of referrals from healthcare professionals remain unclear. The referral process could be enhanced by improved collaboration with the NHS and private sectors. Exploration of the views of skeptics, stakeholders, and referrers may improve communication, and referral and provide direction to the organisation (Gardner et al., 2014; NHS England and NHS Improvement, 2021).

Conclusions

The evaluation found that despite non-significant statistical results, the positive impact of the Nurture Project was captured qualitatively and quantitively within survey data, and the appropriateness of outcome measures should be further considered. Future evaluations would benefit from pre-planned consultation. Consultation and collaboration with other service providers (NHS and private sector) would aid referral and increased awareness of third-sector projects as awareness are vital for project success (Rai and Wood, 2018). Although strengths and barriers of remote service provision (modes of delivery) were identified, more research is needed in this area, including exploration of the effectiveness, accessibility, and satisfaction with modes of delivery.

Ethics statement

The study design and procedures were approved by the Teesside University School of Health and Life Sciences Ethics Committee (approval number 5581).