Evaluating the impact of a maternity and neonatal emergencies education programme in Australian regional and rural health services on clinician knowledge and confidence: a pre-test post-test study.

INTRODUCTION: Almost 78 000 women gave birth in the state of Victoria, Australia, in 2019. While most births occurred in metropolitan Melbourne and large regional centres, a significant proportion of women birthed in rural services. In late 2016, to support clinicians to recognise and respond to clinical deterioration, the Victorian government mandated provision of an emergency training programme, called Maternity and Newborn Emergencies (MANE), to rural and regional maternity services across the state. This paper describes the evaluation of MANE. DESIGN AND
SETTING: A quasi-experimental study design was used; the Kirkpatrick Evaluation Model provided the framework. PARTICIPANTS: Participants came from the 17 rural and regional Victorian maternity services who received MANE in 2018 and/or 2019. OUTCOME MEASURES: Baseline data were collected from MANE attendees before MANE delivery, and at four time points up to 12 months post-delivery. Clinicians' knowledge of the MANE learning objectives, and confidence ratings regarding the emergencies covered in MANE were evaluated. The Safety Attitudes Questionnaire (SAQ) assessed safety climate pre-MANE and 6 months post-MANE among all maternity providers at the sites.
RESULTS: Immediately post-MANE, most attendees reported increased confidence to escalate clinical concerns (n=251/259). Knowledge in the non-technical and practical aspects of the programme increased. Management of perinatal emergencies was viewed as equally stressful pre-MANE and post-MANE, but confidence to manage these emergencies increased post-delivery. Pre-MANE SAQ scores showed consistently strong and poor performing services. Six months post-MANE, some services showed improvements in SAQ scores indicative of improved safety climate.
CONCLUSION: MANE delivery resulted in both short-term and sustained improvements in knowledge of, and confidence in, maternity emergencies. Further investigation of the SAQ across Victoria may facilitate identification of services with a poor safety climate who could benefit from frequent targeted interventions (such as the MANE programme) at these sites. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

A strength of this study was our use of the Kirkpatrick Evaluation Framework, which has been used to evaluate obstetric programmes in the past; and that Maternity and Newborn Emergencies (MANE) attendees were followed up to 12 months post-programme delivery to assess knowledge of perinatal emergencies covered, confidence to manage these emergencies and whether learning outcomes from MANE were sustained over time.

The study investigated the impact of MANE training on teamwork and safety culture using the Safety Attitudes Questionnaire and, where data were available, there were improvements across most domains for five services 6 months post-MANE delivery, although the response rate was low.

This was a pre-test post-test study, without the inclusion of a control group, relying on survey-based self-report rather than skills-based assessment to assess changes in teamwork and responsiveness to perinatal emergencies.

Given the small sample size among this cohort (approximately 3700 births per year across the maternity service providers that received MANE), obstetric clinical outcome data pre-MANE and post-MANE delivery were not collected for this study, limiting the ability to assess Kirkpatrick level 4. Instead, data were collected via interviews with maternity managers to explore perceived changes to clinical practice as a consequence of MANE.

Introduction

Safety and quality of maternity care during pregnancy, intrapartum and postpartum is a priority to prevent avoidable mortality and morbidity, and in many contexts globally, maternity providers implement multidisciplinary obstetric emergency training programmes. This paper presents an evaluation of a newly developed programme implemented in Victoria, Australia, the Maternity and Newborn Emergencies (MANE) programme, that aimed to reduce adverse perinatal outcomes for women birthing in rural areas of the state by providing maternity emergency training, improving multidisciplinary teamwork and communication and addressing clinical governance and safety culture issues.1

Context

Maternity services in Australia include public (ie, government-funded) and private maternity providers. Within the state of Victoria, the Department of Health has responsibility for managing healthcare provision, including maternity care. Approximately 78 000 births occurred in Victoria in 2019, with approximately 75% of women birthing in public hospitals.2 3 Approximately 70% of public hospital births occur in the state capital, Melbourne.4 Of the public hospital births that occur in regional and rural Victoria, around 70% occur in either large regional maternity service providers or in subregional health services, with the remainder in local rural health services.4 Therefore, a significant proportion of Victorian women birth at local maternity services, often some distance from Melbourne, the state capital.

In Victoria, as in the whole of Australia, serious complications during labour and birth are rare, as are maternal or neonatal deaths: in Victoria in 2019, there were nine maternal deaths, and the perinatal mortality rate was 2.3 per 1000 live births.3 Although unexpected, a proportion of perinatal deaths each year are linked to preventable factors such as a lack of communication in the maternity team environment, or inadequate knowledge and skills of the staff providing maternity care, particularly during maternity emergency situations.3 The MANE programme is one of a number of simulation training programmes delivered to rural and regional public maternity services who provide pregnancy, intrapartum and postnatal local care for healthy women and their babies (defined as low-to-medium medical risk). MANE was introduced in addition to the other existing programmes to meet the perceived unmet need of linking an understanding of safety culture and clinical governance to the obstetric emergencies training programme, and to offer the opportunity for organisations to have independent external providers facilitate the training, and provide ‘external eyes’ to assess to assess to teams' skills ad interaction, and teamwork.

Safety culture

Safety culture was a core construct underpinning the MANE programme given its association with obstetric outcomes and the link between a positive safety climate and an increase in safety behaviours.5 Safety culture is defined as ‘a product of individual and group values, attitudes, perceptions, competencies and patterns of behaviour that determine the commitment to, and the style and proficiency of an organisation’s health and safety management’.6 The terms safety culture and safety climate are often used interchangeably; however it is generally agreed that the safety climate is the ‘perceived value placed on safety in an organisation at a particular point in time’.7 The safety climate of a health service is measurable and focuses on staff perceptions about the way in which safety is managed in their health service.8 Measuring the safety climate of a health service provides a snapshot of the safety culture (attitudes and perceptions towards safety) of a health service at one point in time.9

Evaluation of obstetric emergency training programmes

A number of studies have evaluated the effectiveness of multidisciplinary obstetric emergency training programmes similar to MANE. A recent systematic review assessed the effectiveness of training in emergency obstetric care using Kirkpatrick’s Evaluation Model at four levels: participant reaction (level 1); knowledge and skills (level 2); change in behaviour and clinical practice (level 3); and availability of quality emergency obstetric care and health outcomes (level 4).10 The included studies were conducted in a number of countries, but none in Australia. Overall, the review found that participants’ reactions to the obstetric emergencies training was positive (level 1), and that the training led to increased knowledge and skills (level 2) and improved clinical practice (level 3). There is less evidence to support whether obstetric emergency training improves health outcomes (level 4).10

In this paper we aim to present participants’ perceptions of the MANE programme, data on whether participation in MANE led to increases in knowledge and skills, and to changes to clinical practice, and an exploration of safety culture that was undertaken to ascertain whether the programme led to positive effects on the perceived safety climate.

Method

Description of the MANE programme

The MANE programme combines obstetric emergency simulation training with education on teamwork principles, effective communication, leadership and delegation, clinical governance and risk management, appropriate escalation and situational awareness. MANE is delivered on-site by expert external facilitators (n=6). Attendance is multidisciplinary, and delivery is tailored to the individual health service requirements. The programme is a mix of didactic presentations, two simulations and clinical workstations. The simulations always include neonatal resuscitation, and a maternity emergency simulation is chosen by the service. During the evaluation data collection period, health services chose postpartum haemorrhage (n=5); shoulder dystocia (n=3); pre-eclampsia (n=2); eclampsia (n=4); vaginal breech (n=3); maternal collapse (n=2); and cord prolapse (n=1). A detailed description of the programme is provided elsewhere.1

Context of MANE delivery

There are a variety of other maternity services emergency education programmes currently delivered across Victoria, but the most widespread is the PRactical Obstetric Multi-Professional Training (PROMPT) programme.11 PROMPT is an obstetric training package that provides clinicians with education in the effective management of obstetric emergencies,11 and is delivered at most maternity services across Victoria. Depending on the needs of the service, courses can be run as often as is required. MANE was designed to complement PROMPT, with one of the key differences being that MANE is delivered by external facilitators, whereas PROMPT is delivered internally, using a ‘train the trainer’ model.

Study design

A quasi-experimental study design was used, and the Kirkpatrick Evaluation Model served as a framework to underpin the evaluation.12 This four-level evaluation model, measures reaction (participants’ reactions to and attitudes towards MANE); learning (the degree to which participants acquired the intended knowledge, skills, attitudes, confidence and commitment based on their participation in MANE); behaviour (change in the ‘on the job’ behaviours among clinicians, and at an organisational level); and results (the degree targeted outcomes occurred as a result of the training event and subsequent reinforcement). This model has been used to evaluate and review training programmes in obstetrics in various locations in the past.13–16

Study population and recruitment

MANE was delivered to 18 public maternity services across Victoria in 2018 and 2019, all of which were low- or medium-obstetric risk services, with births per year ranging from approximately 20 to 900.17 Seventeen of these services were invited to participate in this evaluation. One health service was excluded as there were additional interventions underway there aimed at educating clinicians and improving safety. Services could choose to ‘opt out’ if they did not wish to take part in the evaluation (no service took this option).

Patient and public involvement

There was no direct patient or public involvement in this study as this was an evaluation of a maternity education programme delivered to maternity and newborn care clinicians.

Data collection

Online supplemental figure 1 highlights data collected at each time point. Data were collected from maternity and newborn care clinicians at each of the 17 sites that participated in the MANE programme in 2018 or 2019. Attendees were midwives, nurses and medical staff including general practitioner obstetricians, anaesthetists and paediatricians. Survey data were collected from attendees immediately before and after MANE delivery, and from maternity and newborn care clinicians at the service (irrespective of MANE attendance) 6 months and 12 months post-MANE. Semi-structured interviews were conducted with midwifery unit managers and/or clinical midwifery educators at each service 4 months after programme delivery. Where a maternity service received MANE in both 2018 and 2019, data were collected from the service for the 2018 programme only. Each maternity service included in the analysis was randomly assigned a number (health service 1 to health service 17).

Survey data collection

Baseline data were collected from MANE attendees via a paper-based survey tool completed immediately before MANE delivery. This voluntary anonymous questionnaire collected basic demographic data and assessed clinicians’ perceptions of safety climate using the Safety Attitudes Questionnaire (SAQ),6 18 (used in a previous obstetric training evaluation in Victoria, Australia)14. Surveys of hospital staff are the most common way of measuring patient safety culture.19 The Australian Commission on Safety and Quality in Health Care cites the SAQ (short version) as a valid tool for measuring patient safety culture in Australian health services.20 There are six domains in the SAQ: teamwork climate (items 1–6); safety climate (items 7–13); job satisfaction (items 15–19); stress recognition (items 20–23); perceptions of management (items 24–28); and working conditions (items 29–32). Each item is answered on a 5-point Likert-type scale where the respondents indicate their level of agreement with the statement provided (ranging from ‘Disagree strongly’ to ‘Agree strongly’). For each domain, a 100-point score was calculated, with a mean domain score generated for each health service with a higher score indicating a stronger safety climate.6 Clinicians were also asked to rate their knowledge of the MANE learning objectives, and confidence ratings regarding the maternity and neonatal emergencies covered in the MANE programme prior to the training.21

Immediately following completion of MANE, attendees were again invited to complete a paper-based survey tool assessing their satisfaction with MANE, their perception of the relevance and usefulness of the programme and their knowledge and confidence in managing the emergencies covered during the programme.21 The commitment of clinicians to apply what they had learnt during MANE was assessed and demographic data collected.1 This survey addressed the first and second levels of the Kirkpatrick Evaluation Model, reaction and learning, by investigating participants’ satisfaction with the programme and their knowledge of key programme components.

Six months and 12 months after completion of MANE, all maternity and newborn care clinicians at that health service (irrespective of their attendance at MANE) were invited to complete an online questionnaire incorporating the SAQ6 18 and exploring knowledge of the MANE learning objectives, and confidence ratings relating to the simulations covered in the MANE programme.21 The commitment of clinicians that attended MANE to apply what they learnt during the programme to their clinical practice was also assessed. Demographic data were collected. These components primarily addressed the third level of the Kirkpatrick Evaluation Model (behaviour) which measures the efficiency of training at, or 6 months post-training.22

Semi-structured interview data collection

A semi-structured telephone interview was conducted 4 months post-training with the maternity unit manager and/or clinical midwife educator to assess the impact of MANE on behaviour change at each service (Kirkpatrick Evaluation Model, levels 3 and 4). Interviews explored clinician confidence to manage perinatal emergencies; skill acquisition as a result of MANE; staff teamwork and collaboration; and changes to clinical practice as a result of MANE (reported elsewhere).

Data management and analysis

Survey data were entered directly into Research Electronic Data Capture,23 24 then transferred to Stata V.1425 for cleaning and analysis. Data cleaning included checks for missing data, range and logic checks. Any discrepancies in the data were checked, with the outcome agreed by two members of the research team. Where questions had pre-coded response options, simple descriptive analyses including frequencies, means/medians and SD are presented. Pearson’s χ2 test or Fisher’s exact test were used to determine the difference between the two groups.

Results

Data were collected from 17 rural and regional maternity service providers across Victoria between March 2018 and December 2019 inclusive. Only clinicians who attended the MANE programme completed the pre-MANE (84%; 294/350) and post-MANE survey (81%; 282/350). All maternity and newborn care clinicians working at the participating maternity services were invited to complete the survey 6 and 12 months post-MANE delivery regardless of MANE attendance. The mean response rate for the 6-month survey was 22% (range 3%–63%); the mean response rate for the 12-month survey was 21% (range 9%–42%). Fifty-six per cent (n=67) and 63% (n=57) of 6-month and 12-month respondents, respectively, attended MANE.

Demographic characteristics

Demographic characteristics of survey respondents across these four time points are shown in table 1. Most survey respondents were midwives and/or nursing staff, woman and working on a part-time basis.

Table 1

Demographic characteristics of survey respondents pre-MANE, immediately post-MANE, 6 months and 12 months post-MANE

Characteristic	Pre-MANE(n=294)	Post-MANE (n=282)	6 months post-MANE (n=120)	12 months post-MANE (n=90)
	n (%)	n (%)	n (%)	n (%)
Position within health service (n=294, 267, 105, 90)
Medical	74 (25.2)	58 (20.6)	16 (13.3)	4 (4.4)
Midwifery and nursing	212 (72.1)	198 (70.2)	88 (73.3)	66 (73.3)
Allied health	0 (0)	0 (0)	1 (0.8)	0 (0)
Others*	7 (2.4)	11 (3.9)	0 (0)	1 (1.1)
Not stated/unknown	1 (0.3)	15 (5.3)	0 (0)	0 (0)
Gender (n=291, 264, 103, 70)
Male	254 (87.3)	233 (88.3)	92 (89.3)	66 (94.3)
Female	36 (12.4)	31 (11.7)	9 (8.7)	3 (4.3)
Prefer not to say	1 (0.3)	0 (0)	2 (1.9)	1 (1.4)
Age (years), mean (SD) (n=284, 253, 98, 64)	42.4 (12.2)	42.6 (12.2)	46.5 (12.5)	46.7 (11.8)
Years of experience in profession, median, mean (SD) (n=282, 251, 102, 65)	12, 15.4 (12.4)	11, 15.5 (12.7)	16.5, 19.2 (13.2)	20, 20.1 (12.4)
Usual shift (n=281, 257, 103, 69)
Morning	36 (12.8)	33 (12.8)	19 (18.5)	13 (18.8)
Afternoon	10 (3.6)	7 (2.7)	7 (6.8)	2 (2.9)
Night	11 (3.9)	10 (3.9)	9 (8.7)	4 (5.8)
Variable	223 (79.4)	204 (79.4)	64 (62.1)	50 (72.5)
On-call	1 (0.4)	3 (1.2)	4 (3.9)	0 (0)
Usual job status (n=287, 257, 104, 67)
Full-time	89 (31.0)	76 (29.6)	19 (18.3)	10 (14.9)
Part-time	181 (63.1)	166 (64.6)	80 (76.9)	54 (80.6)
Casual	16 (5.6)	14 (5.5)	4 (3.9)	3 (4.5)
Locum	1 (0.4)	1 (0.4)	1 (1.0)	0 (0)
English first language (n=291, 258, 103, 69)	266 (91.4)	240 (93.0)	100 (97.1)	69 (100)

*Other includes paramedics, diabetes educators, international board certified lactation consultants, maternal and child health nurses.

MANE, Maternity and Newborn Emergencies.

Baseline SAQ data

The SAQ formed part of the questionnaires administered to MANE attendees prior to their attendance at MANE, 6 months post-MANE and 12 months post-MANE. Online supplemental table 1 presents the SAQ Short Form raw scores from each health service pre-MANE. As shown in figure 1, across five of the six domains (perception of management, work conditions, job satisfaction, safety and teamwork), the trend was for health service 7 to consistently receive the highest scores (ie, most positive attitudes across these domains) and health service 16 to consistently receive the lowest SAQ scores (ie, least positive). The stress recognition domain was an outlier in our analysis, with raw scores not showing a pattern similar to the other domains.

Figure 1

Heat maptable of pre-MANE SAQ scores from health services receiving MANE. The number in each cell represents the health service ranking for that domain. A ranking of 1 indicates the best performer for that domain; a ranking of 17 indicates the poorest performer for that domain. The colour represents the ranking, and changes from green (strongest performer) to red (poorest performer).

Kirkpatrick Evaluation Model—level 1

The first level of the Kirkpatrick Model measures participant satisfaction with MANE.12 This component was measured using 5-point Likert-type scaled questions assessing the perception of usefulness, relevance and satisfaction with MANE where respondents indicated their level of agreement with a statement (ranging from ‘Disagree strongly’ to ‘Agree strongly’). Across all services, participants responded favourably to the programme: over 99% (277/279) ‘Agreed’ or ‘Strongly agreed’ that MANE was relevant to their practice; all respondents ‘Agreed’ or ‘Strongly agreed’ that the programme was useful to their practice (279/279); and that the programme was a worthwhile investment of their time (278/278). The facilitators were viewed as engaging, and participants ‘Agreed’ or ‘Strongly agreed’ that their learning was enhanced by the knowledge of the facilitators (259/260).

Kirkpatrick Evaluation Model—level 2

Level 2 of the Kirkpatrick Model (learning) refers to the effectiveness of training in providing immediate benefits for clinicians.12 Changes in knowledge were investigated by assessing participants’ perception of their understanding of key programme components before and after MANE delivery. Perception of their knowledge of the core learning outcomes (ie, teamwork principles and effective communication; leadership and delegation; clinical governance and risk management; appropriate escalation; and situational awareness) in addition to topics covered in simulations at each site were assessed for this level. Overall, survey respondents reported an increase in knowledge immediately after programme delivery for both the technical and non-technical components covered during MANE (online supplemental figure 2; figure 2). Where adequate response rates were achieved (ie, for newborn resuscitation, postpartum haemorrhage management and eclampsia management), confidence scores for managing the emergencies included in the programme improved post-MANE delivery, however MANE attendees still viewed management of these perinatal emergencies as equally stressful pre-MANE and post-MANE delivery (figure 3A, B).

Figure 2

MANE attendees’ knowledge of non-technical and practical components of the MANE rogramme pre-MANE, immediately post-MANE, 6 months and 12 months post-MANE delivery. For all components apart from postpartum haemorrhage management, there was a significant difference (p<0.05) in responses pre-MANE and post MANE; pre-MANE and 6 months post-MANE; and pre-MANE and 12 months post-MANE. For postpartum haemorrhage management, there was a significant difference (p<0.05) in responses pre-MANE and post MANE only.

Figure 3

(A) MANE attendees’ confidence to manage selected perinatal emergencies covered during the MANE rogramme pre-MANE, immediately post-MANE, 6 months and 12 months post-MANE delivery. For all components there was a significant difference (p<0.05) in responses pre-MANE and post MANE; pre-MANE and 6 months post-MANE; and pre-MANE and 12 months post-MANE. (B) MANE attendees’ perceived management of perinatal emergencies as stressful pre-MANE, immediately post-MANE, 6 months and 12 months post-MANE delivery. For all components there was no significant differences in responses pre-MANE and post MANE; pre-MANE and 6 months post-MANE; and pre-MANE and 12 months post-MANE.

Kirkpatrick Evaluation Model—level 3

Kirkpatrick level 3 measures change in behaviour resulting from training received. Staff surveys administered 6 and 12 months post-MANE explored skill and knowledge retention and changes in workplace behaviour. Data reported by MANE attendees are shown in figures 2 and 3. The trend was for knowledge of all MANE learning outcomes to be retained 6 months and 12 months post-MANE delivery among MANE attendees (figure 2). Confidence to manage these obstetric emergencies among MANE attendees also remained high up to 12 months post-MANE delivery (figure 3A), although management of these emergencies was still viewed as stressful at this time point (figure 3B). Knowledge of MANE learning outcomes for clinicians that did not attend MANE was also high 6 and 12 months post-MANE delivery (online supplemental figure 3).

Changes in workplace behaviour up to 12 months post-MANE were assessed by exploring MANE attendees’ confidence to escalate clinical concerns; their commitment to applying skills covered during MANE to their practice; and their overall confidence to manage perinatal emergencies. Over 90% of the respondents who attended MANE ‘Agreed’ or ‘Strongly agreed’ with each of these three statements up to 12 months after MANE delivery (online supplemental figure 4).

The SAQ further addressed Kirkpatrick level 3 by assessing whether there were changes in the safety climate within each organisation. Changes in SAQ mean scores for each domain were assessed 6 months post-MANE for services where more than five responses were received (nine health services; figure 4). Six months after MANE delivery, excluding the stress recognition domain, five services showed improvements across most domains, including health service 16, the poorest performer pre-MANE delivery. The four remaining services had a reduction or no change in SAQ scores across most domains 6 months after MANE (figure 4).

Figure 4

Change in SAQ mean score pre-MANE and 6 months post-MANE. A score of 8.1 in the teamwork domain for health service two indicates an increase of 8.1 percentage points in this domain in the 6 months after MANE delivery; a score of −11.3 in the teamwork domain for health service eight indicates a reduction of 11.3 percentage points in this domain in the 6 months post-MANE delivery.

Kirkpatrick Evaluation Model—level 4

Kirkpatrick level 4 measures the degree to which there were sustained improvements in clinical practice attributable to MANE. Given overall birth numbers were insufficient to make comparisons of clinical outcomes, this was assessed through interviews with maternity unit managers and/or clinical educators 4 months post-MANE. Most interview participants (n=13) agreed that MANE had resulted in defined changes to clinical practice in their unit, with four providing a neutral response. Interview participants were also asked whether they thought MANE had an impact on attendees’ awareness and understanding of clinical governance, another aim of the programme. Participants in all but two health services reported that MANE increased clinicians’ awareness and understanding of clinical governance. Participants also reported defined changes that were implemented as a direct result of MANE. These included facilitating further education sessions based on the needs of clinicians, education to increase midwifery scope of practice, the purchase or rental of new equipment for the service and the refinement of documentation tools and clinical protocols.

Discussion

Main findings

The MANE programme evaluation was conducted using the Kirkpatrick Evaluation Model as a framework to guide data collection and analysis.12 22 Kirkpatrick’s model assesses the effectiveness of training programmes at four levels: reaction; learning; behaviour; and results.12 Our evaluation demonstrated improved knowledge of all learning outcomes, and increased confidence to manage emergencies, sustained up to 12 months post-programme delivery, fulfilling levels 1–3 of Kirkpatrick’s model. Although clinical outcome data were not collected in this study, interviews conducted with maternity managers and clinical midwifery educators highlighted several changes to clinical practice as a consequence of MANE delivery, thus addressing level 4 of Kirkpatrick’s model. The safety climate of each health service was measured pre-MANE and 6 months post-MANE delivery using the SAQ; five services showed improvements in most domains, while four showed either a reduction or no change.

Sustained changes following MANE

Immediately post-MANE, as has been noted in other studies,26–28 participants reported increased knowledge of all learning components, which translated into increased confidence to manage these emergencies. However, programme attendees still viewed management of these emergencies as stressful. This is in contrast to a study by Sørensen and colleagues who found that after a multidisciplinary simulation-based training programme, participants considered management of the perinatal emergencies covered as less stressful and less unpleasant to perform.21 There may be several reasons for this: maternity services in our evaluation are low and medium obstetric risk services located in rural Victoria, often some distance from larger services in the regional and metropolitan settings. Therefore, clinicians at these services would have relatively infrequent exposure to the perinatal emergencies. Despite this, their confidence to manage these emergencies including neonatal resuscitation remained high up to 12 months post-training.

Participants’ self-report of their knowledge of MANE leaning components up to 12 months post-MANE was also comparable to post-MANE levels, as has also been shown previously in studies evaluating similar obstetric emergency training programmes.29 30 It must be noted, however, that reported knowledge of MANE learnings, and confidence to manage perinatal emergencies were also high 6 and 12 months post-MANE among clinicians that did not attend the programme. It may be that all clinicians at these services may benefit from programmes such as MANE because of the diffusion of programme learnings. However, the response rate for surveys administered 6 months and 12 months post-MANE training was low, and respondents at these time points were more likely to report their profession as nursing/midwifery, work part-time, be older and have more experience in their profession than pre-MANE and post-MANE respondents. This increased professional experience may impact their perceived confidence to manage perinatal emergencies. Kumar and colleagues found that knowledge of, and confidence to manage neonatal resuscitation was higher for midwives than the medical staff in their evaluation of the PROMPT obstetric programme,16 which may also account for sustained knowledge and confidence in our study given more respondents were nursing and midwifery professionals. Further, our evaluation relied on self-report rather than skills-based evaluation of knowledge and confidence, which may have impacted the results at all time points.

The final level of the Kirkpatrick Evaluation Model assesses results, the degree to which defined outcomes occurred as a result of MANE. Sustained improvements to clinical practice were explored during follow-up interviews with maternity managers 4 months after MANE, with several changes evident at all health services. Some of these included facilitating further education sessions to increase midwifery scope of practice; the purchase or rental of new equipment for the service; and the refinement of documentation tools. For this level, an investigation of perinatal outcomes was not conducted: given the small number of births (overall, approximately 3700 births per year for all services that received MANE in 2018 and 2019) we reasoned that the rare outcomes of importance were too infrequent to allow potential changes to be measured meaningfully. Several other studies have investigated the impact of obstetric training programmes on clinical outcomes including postpartum haemorrhage,14 16 31 Apgar scores,31–33 neonatal hypoxic ischaemic encephalopathy31 32 and trauma from shoulder dystocia.16 31 34 A recent systematic review and meta-analysis has found weak evidence to support an effect on brachial plexus injury; and a positive but non-significant effect on Apgar scores below 7 at 5 min; the evidence supporting other clinical outcomes was less clear.35 Indeed, emergency obstetric training programme evaluations routinely demonstrate changes in the first three levels of the Kirkpatrick Evaluation Model: a systematic literature review assessed the evidence for the effectiveness of training in emergency obstetric care, and included 101 studies.10 Changes in level 1 and/or level 2 were investigated in 68 of these, with level 3 assessed in 51 studies, and level 4 in 21.10 Currently, there are few high-quality studies providing a causal link between obstetric training and a reduction in maternal and neonatal mortality. In accordance with the weight of published material, this study has also demonstrated changes in level 1 and level 2, and has also confirmed sustained knowledge and confidence increases up to 12 months post-MANE delivery.

Impact on safety culture

This evaluation investigated the impact of MANE training on teamwork and safety culture using the SAQ.6 As has been previously reported, the stress recognition domain of this tool was an outlier in our analysis.36 Without the inclusion of this domain, pre-MANE SAQ scores showed consistent strong and poor performing health services across all other domains, indicative of the safety climate at that service. Although the SAQ response rate was low, 6 months post-MANE delivery, there were improvements across most domains (teamwork; safety climate; job satisfaction; perception of management and working conditions) for five services. Health service 2, health service 12 and health service 16, which were among the poorest performers pre-MANE, showed improvements 6 months post-MANE delivery, while health service 8 had reductions across all domains of the SAQ. While changes in the safety culture cannot conclusively be attributed to MANE, assessments of safety culture and teamwork using the SAQ have previously shown an association with patient harm and hospital mortality,37 38 and a recent systematic review has highlighted the role of teamwork and communication training interventions on improving safety culture in emergency department settings.39 The SAQ has previously been used in an evaluation of the PROMPT obstetric training programme in Victoria, where increases in the teamwork, perception of management and safety climate domains were seen among PROMPT attendees post-programme delivery.14 However, as was demonstrated by Shoushtarian and colleagues,14 we did not see an overall increase in SAQ scores across the teamwork, perception of management and safety climate domains at all services up to 6 months post-MANE delivery. This may be due to our low response rate 6 months post-MANE, or be a consequence of other factors such as management changes or overall staff attrition that have not been captured during this study. Indeed, one service showing a reduction in the SAQ shows the biggest decreases in the perception of management domain.

Strengths and limitations

A strength of this evaluation is that it used the Kirkpatrick Model, which has been used to evaluate obstetric programmes in the past. Although this study relied on self-report, MANE attendees were followed-up 6 and 12 months post-programme delivery. All health services that received MANE participated in the evaluation, with high survey response rates among MANE attendees pre-MANE and immediately post-MANE.

There are several limitations to our study: this was a before and after design, without the inclusion of a control group. It is therefore difficult to ascribe changes in our exposures of interest definitively to the MANE programme. Our evaluation was a pragmatic one; the MANE programme was mandated by the Victorian government to be delivered to all rural and regional maternity services across Victoria, and so conducting a randomised controlled trial was not feasible. The sustained increased in knowledge and confidence to manage perinatal emergencies, or changes in safety climate also cannot be ascribed to MANE alone: it may be that factors independent of, or in addition to, MANE also play a role, including other education programmes such as PROMPT, and other safety initiatives underway across Victoria. MANE may be one component in a whole suite of initiatives that contribute to these outcomes. Further, the low response rate at 6 months and 12 months post-MANE, along with the characteristics of survey respondents may have affected responses provided at these time points, and the lack of clinical outcome data further limits the interpretation of this study. It is worth noting, however, that the evidence supporting clinical outcomes following obstetric training programmes is unclear,35 or the length of time they are sustained.40

Conclusions and implications for maternity service providers

Overall, this study has demonstrated improved knowledge of all learning outcomes; increased confidence to manage emergencies, sustained up to 12 months post-programme delivery; and highlighted several changes to clinical practice as a result of MANE. Given the fact that MANE is delivered every 12 months at best, regular delivery of other education programmes such as PROMPT may sustain these knowledge and confidence levels in the months post-MANE delivery. SAQ data showed strong and poor performers at each time point. Given this variation, a long-term follow-up using this tool conducted at health services that received MANE in 2018 and 2019 would further assess safety climate. Indeed, provided adequate response rates can be achieved, a state-wide SAQ for maternity staff across Victoria may facilitate early identification of services with a poor safety climate who would benefit from more frequent targeted interventions, such as the MANE programme, at these sites.