Using the Kirkpatrick Model to evaluate the Maternity and Neonatal Emergencies (MANE) programme: Background and study protocol.

INTRODUCTION: Over 310 000 women gave birth in Australia in 2016, with approximately 80 000 births in the state of Victoria. While most of these births occur in metropolitan Melbourne and other large regional centres, a significant proportion of Victorian women birth in local rural health services. The Victorian state government recently mandated the provision of a maternal and neonatal emergency training programme, called Maternal and Newborn Emergencies (MANE), to rural and regional maternity service providers across the state. MANE aims to educate maternity and newborn care clinicians about recognising and responding to clinical deterioration in an effort to improve clinical outcomes. This paper describes the protocol for an evaluation of the MANE programme. METHODS AND ANALYSIS: This study will evaluate the effectiveness of MANE in relation to: clinician confidence, skills and knowledge; changes in teamwork and collaboration; and consumer experience and satisfaction, and will explore and describe any governance changes within the organisations after MANE implementation. The Kirkpatrick Evaluation Model will provide a framework for the evaluation. The participants of MANE, 27 rural and regional Victorian health services ranging in size from approximately 20 to 1000 births per year, will be invited to participate. Baseline data will be collected from maternity service staff and consumers at each health service before MANE delivery, and at four time-points post-MANE delivery. There will be four components to data collection: a survey of maternity services staff; follow-up interviews with Maternity Managers at health services 4 months after MANE delivery; consumer feedback from all health services collected through the Victorian Healthcare Experience Survey; case studies with five regional or rural health service providers. ETHICS AND DISSEMINATION: This evaluation has been approved by the La Trobe University Science, Health and Engineering College Human Ethics Sub-Committee. Findings will be presented to project stakeholders in a deidentified report, and disseminated through peer-reviewed publications and conference presentations. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

A strength of this evaluation is that it will explore the effect of Maternal and Newborn Emergencies (MANE) on overall health service governance, organisational and clinician behaviour change, clinician education, and consumer experience and satisfaction with quality of care, in addition to providing recommendations on programme sustainability over the long term.

This is a multi-methods study collecting both quantitative and qualitative data: quantitative data will assess changes in knowledge and skills, and confidence to manage perinatal emergencies; qualitative data collected from health service executive and clinical staff will assess individual behaviour change and changes in teamwork and collaboration within the maternity setting.

The Kirkpatrick Evaluation Framework, which has been used to evaluate obstetric programmes in the past, will be used to assess the effectiveness of MANE training, assessing participant reaction to the MANE programme, knowledge acquisition, behaviour change, organisational change and patient experience.

Given the small sample size in this cohort (approximately 7500 births per year across all health services), clinical outcome data will not be collected, as the important rare outcomes of interest are too infrequent to allow changes to be measured.

To our knowledge, no studies have been conducted using the Kirkpatrick Model to investigate specific changes in relation to consumer experience and satisfaction with quality of care following an obstetric emergency training programme.

Introduction

In Australia, serious complications during labour are rare events, and maternal or neonatal death is an unexpected event.1 In the state of Victoria, the Consultative Council on Obstetric and Paediatric Mortality and Morbidity reported seven maternal deaths in 2017, three of which were directly related to the pregnancy,2 and Victoria’s perinatal mortality rate (which includes stillbirths and deaths of live-born babies within the first 28 days after birth) is among the lowest in Australia (8.8 per 1000 births) and comparable with other countries of similar socioeconomic status.2 Although maternal or neonatal deaths are unexpected in Australia, a proportion of these deaths have been linked to preventable factors such as a lack of communication in an obstetric or midwifery team environment, or inadequate knowledge and skills of the midwives and doctors providing maternity care, especially emergency obstetric care.1 Therefore, education and training programmes to improve clinician skills in managing obstetric emergencies have been introduced in Australia, as in other countries, in the hope of reducing maternal and neonatal morbidity and mortality.

Australia’s healthcare system is made up of both government-funded (public) and private organisations. Almost all births in Australia occur in public or private hospitals. In 2016, 97% of mothers birthed in hospitals, in birth centres (1.8%), at home (0.3%) or in other settings including births before arrival at hospital (0.4%).3 Of the mothers who birth in hospitals, the majority (74%) do so in public hospitals, with the remaining 26% birthing in private health services.3

Over 310 000 women gave birth in Australia in 2016,3 with over 80 000 births occurring in the state of Victoria.4 In Victoria, as in the broader Australian context, the majority of births occur in public hospitals. Approximately 70% of births in Victoria occur in metropolitan Melbourne.5 Of the approximately 30% of public hospital births that occur in regional and rural areas, around 70% occur in either large regional centres or in subregional health services, with the remainder in local rural health services.5

The Capability Frameworks for Victorian Maternity and Newborn Services (2018)6 outline the role of each public maternity and newborn service in metropolitan, regional and rural Victoria. These frameworks define the care provided by health services into six levels, describing the services required at each level of care, and the relationships between maternity and newborn services at the Victorian state-wide level (see table 1 for 2017–2018 Maternity Capability Levels). Maternity services operate in a networked system across these six levels of care. Level 1, 2 and 3 maternity service providers are defined as primary maternity care providers and have the capacity to provide local care for low-risk pregnancies and newborns. These facilities have capacity to provide care for women in labour from 37 weeks gestation and infants born at 37 weeks gestation or later. Level 4 services are also equipped to provide local care for women and babies at low risk, in addition to providing women and babies with some (medium) risk requiring additional care, such as women with hypertensive disorders of pregnancy. Level 4 facilities are equipped to provide care for women in labour from 34 weeks of gestation. Level 5 and 6 services provide local care for all women and babies, in addition to regional or statewide care for those deemed at moderate and high risk.6 Level 5 services provide labour and birth facilities for moderate-risk pregnancies from 31 weeks gestation. Level 6 services provide maternity care for the highest risk women from across the state, as well as providing care to women of any risk level living in the catchment area of that service. Level 6 services are equipped with neonatal intensive care units, and therefore care for the earliest and sickest newborns in the state. Level 2, 3 and 4 health service providers are located mainly in regional and rural Victoria.7

Table 1

2017–2018 Victorian maternity and newborn capability levels*

Health service	Campus	Maternity capabilitylevel	Newborn capabilitylevel†
Maternity Services Level 6
Mercy Public Hospitals Inc.	Mercy Hospital for Women	6	6a
Monash Health	Clayton	6	6b
The Royal Women’s Hospital	Parkville	6	6a
Maternity Services Level 5
Albury Wodonga Health	Wodonga	5	4
Ballarat Health Services	Base	5	4
Barwon Health	Geelong	5	5
Bendigo Health Care Group		5	4
Eastern Health	Box Hill	5	4
Goulburn Valley Health	Shepparton	5	4
Latrobe Regional Hospital	Traralgon	5	4
Northern Health	Northern	5	5
Peninsula Health	Frankston	5	To be confirmed
Western Health	Sunshine	5	5
Maternity Services Level 4
Central Gippsland Health Service	Sale	4	3
Eastern Health	Angliss	4	3
Mercy Public Hospitals Inc.	Werribee Mercy	4	4
Mildura Base Hospital	Mildura	4	3
Monash Health	Casey	4	To be confirmed
	Dandenong	4	To be confirmed
Northeast Health Wangaratta		4	3
South West Healthcare	Warrnambool	4	3
The Royal Women’s Hospital	Sandringham	4	3
West Gippsland Healthcare Group	Warragul	4	3
Wimmera Health Care Group	Horsham	4	3
Maternity Services Level 3
Bairnsdale Regional Health Services		3	2
Bass Coast Health		3	2
Benalla Health		3	2
Colac Area Health		3	2
Djerriwarrh Health Services	Bacchus Marsh	3	2
East Grampians Health Service	Ararat	3	2
Echuca Regional Health		3	2
Gippsland Southern Health Service	Leongatha	3	2
Kilmore and District Hospital		3	2
Mansfield District Hospital		3	2
South Gippsland Hospital		3	2
South West Healthcare	Camperdown	3	2
Swan Hill	Swan Hill	3	2
Western District Health Service	Hamilton	3	2
Maternity Services Level 2
Castlemaine Health		2	2
Cohuna District Hospital		2	2
Kyneton District Health Service		2	2
Maryborough District Health Service	Maryborough	2	2
Portland District Health		2	2
Yarrawonga District Health Service		2	2
Terang and Mortlake Health Service	Terang	2	2
Maternity Services Level 1
Alpine Health	Bright	1	1
	Mount Beauty	1	1
	Myrtleford	1	1
Casterton Memorial Hospital		1	0
Kerang and District Health		TBC	0
Kyabram and District Health Services		1	0
Numurkah District Health Service		1	0
Orbost Regional Health		1	1
West Wimmera Health Service		1	0
Newborn services only
The Royal Children’s Hospital		Not applicable	6b

*Table taken from the Department of Health and Human Services.7

†Newborn capability levels 1 and 2 provide primary newborn services, caring for low-risk, uncomplicated newborns; levels 3, 4 and 5 provide secondary newborn services, caring for moderate to selected high-risk newborns in a special care nursery; and level 6 (a and b) provides tertiary newborn services, caring for newborns requiring continuous life support and comprehensive multidisciplinary care in a neonatal intensive care unit. Level 6b also provides surgical services.33

The Maternity Services Education Program (MSEP) is a Victorian Department of Health and Human Services (DHHS)–funded initiative auspiced by the Royal Women’s Hospital, Melbourne. MSEP has facilitated pregnancy care and maternity emergency education across regional and rural Victoria for over 10 years. A description of the original maternity emergency component of MSEP has been published elsewhere.8 The MSEP vision is to improve clinical outcomes for women and their babies through the provision of interdisciplinary education, supportive partnerships and capacity building in rural and regional maternity services.9 In recent years, MSEP has re-structured programme delivery to ensure level 2, 3 and 4 services in rural and regional Victoria are provided with regular, high-quality maternity emergency training by offering a locally delivered Maternity and Newborn Emergencies (MANE) MSEP programme. This restructured service has maintained a strong focus on recognising and responding to clinical deterioration, in addition to educating clinicians about clinical governance and risk management principals. Course content relevant to governance is guided by the five domains of robust clinical governance systems introduced by Safer Care Victoria (the state’s healthcare quality and safety improvement agency) in June 2017 (ie, leadership and culture; consumer partnerships; workforce; risk management and clinical practice).10 The programme also combines maternity emergency education with Paediatric Infant Perinatal Emergency Retrieval (PIPER) training, supporting rural and regional providers who have limited resources to their own team-based maternity emergency training. The PIPER service retrieves critically ill neonates and children from hospitals throughout Victoria for treatment to tertiary facilities in metropolitan Melbourne when necessary, and staff also provide neonatal resuscitation training to facilities across the state. All rural and regional level 2, 3 and 4 health Victorian health service providers, who have births ranging from 20 to 1003 births per year,11 are now required by the Victorian DHHS to participate in this restructured MANE programme. Table 2 provides a list of these health services.

Table 2

List of Victorian regional and rural health services receiving MANE

Campus	Births per year*
Maternity S ervices: Level 4
Central Gippsland Health Service	Sale	461
Mildura Base Hospital	Mildura	891
Northeast Health	Wangaratta	594
South West Healthcare	Warrnambool	728
West Gippsland Healthcare Group	Warragul	1003
Wimmera Health Care Group	Horsham	381
Maternity Services: Level 3
Bairnsdale Regional Health Services	Bairnsdale	339
Bass Coast Health	Wonthaggi	170
Benalla Health	Benalla	103
Colac Area Health	Colac	185
Djerriwarrh Health Services	Bacchus Marsh	747
East Grampians Health Service	Ararat	119
Echuca Regional Health	Echuca	338
Gippsland Southern Health Service	Leongatha	201
Kilmore and District Hospital	Kilmore	223
Mansfield District Hospital	Mansfield	77
South Gippsland Hospital	Foster	76
South West Healthcare	Camperdown	36
Swan Hill	Swan Hill	256
Western District Health Service	Hamilton	199
Maternity Services: Level 2
Castlemaine Health	Castlemaine	76
Cohuna District Hospital	Cohuna	53
Kyneton District Health Service	Kyneton	38
Maryborough District Health Service	Maryborough	78
Portland District Health	Portland	87
Terang and Mortlake Health Service	Terang	20
Yarrawonga District Health Service	Yarrawonga	60

*Adapted from Department of Health and Human Services, Safer Care Victoria.11

The MANE programme is delivered over 2 days, facilitated by both MSEP and PIPER. Registration is open to midwives, nurses, interns, registrars, obstetricians, paediatricians, general practitioners, paramedics and students. The delivery of the MANE programme is tailored to the individual requirements of the health service based on consultation between MSEP staff and key stakeholders within the health service in the months leading up to the workshops to develop a programme agenda suitable for attendees at that service. Programme delivery includes core components and elective modules, chosen by the health service. Core modules are didactic presentations describing the changing landscape of Victorian maternity services (clinical governance module); the effective team (ie, team resource management); newborn resuscitation; time, escalation, decision; and Aboriginal and Torres Strait Islander health. In addition, the health service can choose to focus on one of the following elective modules: clinical governance (further information on clinical governance, in addition to the core clinical governance module); risk management; obesity in pregnancy and labour; pre-eclampsia and hypertensive disorders; induction of labour; maternal and newborn sepsis; postpartum haemorrhage and third-stage management. Two simulation modules are also included which can reflect the topics discussed in the elective modules. These include: postpartum haemorrhage; breech vaginal birth; shoulder dystocia; cord prolapse; preterm birth; newborn resuscitation; eclampsia and maternal collapse. Three clinical workstations are also offered encompassing obstetric, maternity and neonatal resuscitation skills updates. A core component of the programme is a feedback session where attendees have the opportunity to highlight clinical and governance concerns to facilitators. These are collated and fed back anonymously to executive and management staff within the service. Therefore, while simulations and work-stations to improve care remain core components of the programme, there is also an increased focus on site-specific initiatives to improve clinical governance, quality and safety, clinical care, teamwork and communication during an emergency incorporated into the restructured programme.12

Although evidence exists for a positive impact of training in obstetric emergencies,13–18 much of the available evidence on the effectiveness of training focuses on the evaluation of participants’ confidence, skills and knowledge post-training, rather than clinical or health service outcomes.19 Several reviews have highlighted the active components of effective emergency obstetric training including: institution-level incentives to training; regular, multiprofessional, mandatory, ‘in house’ training; teamwork training integrated with clinical teaching; and the use of on-site high-fidelity simulation models,20 21 but in some studies, training for obstetric emergencies is not always found to be effective in terms of measurable change in clinical outcomes.22 23 However, a recent systematic review investigating the effectiveness of training in emergency obstetric care found strong evidence for improved clinical practice in areas such as adherence to protocols, resuscitation technique, communication and teamwork, as well as improved neonatal outcomes such as reduced trauma after shoulder dystocia.24 It did not, however, find strong evidence for a reduction in other outcomes such as postpartum haemorrhage, maternal death and stillbirth rates.24 Further, a large randomised controlled trial of PROMPT (PRactical Obstetric Multi-Professional Training) which included over 87 000 births in Scotland, published after the systematic review, found no improvement in neonatal outcomes (measured by the number of infants with an Apgar score of less than 7 at 5 minutes).25

To our knowledge, no studies have been conducted investigating specific changes in relation to clinical governance or organisational culture following maternal and newborn emergency training programmes, yet these are potentially strong influences on clinical outcomes. Therefore, to investigate individual and institutional behaviour change, and to inform the ongoing development of the content, design and delivery of the MANE programme, an evaluation of MANE is underway. This paper describes the protocol for the evaluation of the MANE programme and demonstrates how the Kirkpatrick Evaluation Model could be used to effectively explore the implementation of a multidisciplinary education programme being undertaken in a complex setting.

Methods and analysis

Aims

The aim of the evaluation is to determine the effectiveness of MANE in relation to: governance changes at the health service; organisational behaviour change; clinician behaviour change; multidisciplinary education, teamwork and collaboration across teams and disciplines; individual clinician education and practical use of skills; and consumer experience and satisfaction with quality of care.

Study design

The Kirkpatrick Evaluation Model will be used to explore the effect of MANE training.26 This model is a widely used approach to training evaluation and has previously been used to evaluate and review training programmes in obstetrics.16 19 27 It is a four-level training evaluation model, measuring:

Reaction (measures participants’ reactions to and attitudes towards MANE);

Learning (measures the degree to which participants acquired the intended knowledge, skills, attitudes, confidence and commitment based on their participation in MANE);

Behaviour (measures change in the ‘on the job’ behaviours among clinicians, and at an organisational level as a result of MANE training); and

Results (to what degree targeted outcomes occur as a result of the training event and subsequent reinforcement).

Bergh and colleagues reviewed evidence of change in healthcare provider behaviour and maternal and neonatal outcomes as a result of emergency obstetric and neonatal care training.20 This review proposed a refined version of the Kirkpatrick classification for programme evaluation, with increased focus on levels 3 (behaviour) and 4 (results). This refined model will underpin our evaluation, with a greater focus on behaviour change and the impact of training. Figure 1 provides a research plan for our evaluation, incorporating the essential components of the Kirkpatrick Evaluation Model.

Figure 1

Research plan for the MANE evaluation based on the Kirkpatrick evaluation Model.20 SAQ, Safety Attitudes Questionnaire.

Study population

All eligible health services in Victoria, Australia will be invited to participate in the evaluation. Data will also be collected from health service consumers who receive antenatal, intrapartum or postnatal care at eligible health services.

All Victorian public level 2, 3 and 4 rural and regional health services participating in the MANE programme in 2018 and 2019 including:

Maternity and newborn care clinicians (MANE attendees and non-attendees) at Victorian level 2, 3 and 4 rural and regional health services participating in MANE in 2018 and 2019;

Key stakeholders (eg, health service Chief Executive Officers, Executive Staff, Maternity and Quality Managers) at level 2, 3 and 4 rural and regional health services receiving MANE in 2018 and 2019; and

Maternity consumers receiving care at these health services in 2017, 2018 and 2019 who complete the Victorian Healthcare Experience Survey (VHES).

Victorian public health service providers not participating in the MANE programme; and

Victorian private maternity care providers.

Recruitment

Twenty-seven rural and regional health services in Victoria are required by the Victorian DHHS to participate in MANE (see table 2) and will be invited to participate in the evaluation. These health services, who range in size from approximately 20 to 1000 births per year, will have the option of opting out if they do not wish to take part in the evaluation. Maternity and newborn care clinicians at these health services will have the option of not completing questionnaires or participating in focus groups or interviews if they do not wish to. Data will only be collected from health services receiving MANE during the data collection period (2018 and 2019).

Data collection

Data will be collected from all Victorian public level 2, 3 and 4 rural and regional health services participating in the MANE programme in 2018 and 2019. These data will be collected over a 12-month period depending on timing of MANE programme delivery at the organisation. Baseline data will be collected from maternity service staff and consumers at each health service immediately before MANE delivery, and at four time-points post-MANE delivery, in accordance with the Kirkpatrick Model Framework (see figure 1). There will be four components to data collection:

Survey of maternity and newborn care clinicians (all health service providers) at four time-points: before, immediately post, 6 months post and 12 months post-MANE delivery);

Follow-up interviews with key personnel and participants in the MANE programme in the health services, 4 months post-MANE delivery;

Consumer views from all health services (collected through the VHES) at two time-points: before and 12 months post-MANE delivery; and

Case studies with five regional or rural health service providers, one from each Victorian regional boundary.

Therefore, the data collection period will encompass 2018, 2019 and 2020. Because perinatal mortality and morbidity are rare events in the state of Victoria, and there are inadequate birth numbers across all participating health services, clinical outcome data will not be collected part of this evaluation.

Baseline data

Baseline data will be collected from maternity and newborn care clinicians via a survey tool completed immediately before MANE delivery. This voluntary survey will be completed by MANE attendees and assess safety attitudes, teamwork, stress and perceptions of management using the Safety Attitude Questionnaire (SAQ).28 29 The SAQ assesses staff attitudes through six climate scales: teamwork; safety; job satisfaction; perceptions of management; working conditions and stress recognition,28 and is used as a tool to measure caregiver attitudes, to prompt interventions to improve safety attitudes and to measure the effectiveness of these interventions.28 It has been successfully used in a previous obstetric training evaluation in Victoria.16 Knowledge and skills ratings of the MANE learning objectives will be evaluated, and confidence ratings of the maternal and neonatal emergencies covered in the MANE programme will also be assessed pre-delivery as per Sørensen and colleagues.30 Basic demographic data will be collected including: position at health service (medical/midwifery/nursing/allied health/other); participant age (years),; gender (male/female/other/prefer not to say); years employed at health service (years); planned length of time in chosen profession (years); usual work hours (morning/afternoon/night/varied shifts); and job status (full-time/part-time/casual staff/agency staff/locum).

Outcome data

Component 1: survey of maternity and newborn care clinicians

Immediately following completion of MANE, attendees will be asked to complete an evaluation tool assessing their knowledge, perception of the relevance, usefulness and confidence in addressing the maternal and neonatal emergencies covered in the services’ programme.30 Likert-type scales will be included in all questionnaires pre-MANE and post-MANE delivery to assess the learnings from the programme more broadly. Participants will be asked to rate their knowledge and skills of the emergencies covered during MANE, and also of the core components of the MANE programme (ie, teamwork and communication; effective leadership and delegation delegate; situational awareness; escalation; knowledge of clinical governance and risk management). This will provide a snapshot of learnings from MANE up to 12 months post-delivery. Clinicians will be asked to list one key message that they have taken from MANE and describe how they will apply this in their clinical practice. The commitment of clinicians to apply what they have learnt during MANE will also be assessed (rated on a 5-point Likert-type scale). Feedback on the simulations, clinical discussions, debriefing sessions, presentations, and review and response session will be sought from attendees, as well as their views on the ability of facilitators to keep the audience engaged during MANE. Demographic information will be collected, as above. This survey addresses the first and second level of the Kirkpatrick Evaluation Model, reaction and learning.

Six months after completion of MANE, all maternity and newborn care clinicians at participating health services will be invited to complete a brief online questionnaire, assessing staff attitudes through the SAQ.28 29 Knowledge and skills ratings of the MANE learning objectives will again be evaluated, and confidence ratings of the maternal and neonatal emergencies covered in the MANE programme will be assessed.30 Again, the commitment of clinicians to apply what they have learnt during MANE will be assessed, and demographic data will be collected. Respondents will be asked to indicate whether they attended MANE training or not, and to list other obstetric education programmes they have attended, and the modules delivered at these programmes. This component addresses the third level of the Kirkpatrick evaluation model, behaviour, which measures the efficiency of training at or 6 months post-training.20 This online questionnaire will again be distributed to all maternity and newborn care clinicians 12 months post-training, addressing the fourth level of the Kirkpatrick model (results), which assesses behaviour change at least 12 months after programme delivery.20

Component 2: follow-up interviews with health services (all health service providers)

A follow-up semistructured interview will be conducted 4 months post-training to assess behaviour change at each health service. This will be conducted in person or by telephone, with two members of the research team, and the Director of Nursing, Maternity Unit Manager and/or Clinical Midwife Educator at the health service. It is anticipated that the interview will take approximately 1 hour to complete and will be audio-recorded for later analysis. This interview will explore several themes including: decision(s) to attend MANE; perceptions of MANE; confidence and skill acquisition; teamwork and collaboration; governance and organisational change; MANE and other educational programmes underway in Victoria and MANE in the future. This component addresses the third level of the Kirkpatrick Evaluation Model, behaviour.

Component 3: consumer views from all health services (collected through the VHES)

On behalf of the Victorian DHHS, the VHES collects anonymous data from a range of healthcare users of Victorian public health services. Each month, eligible consumers of health service settings are randomly selected to receive a questionnaire. The survey features specialised questionnaires for maternity clients to collect information on antenatal, labour and birth and postnatal care received at Victorian health services. Maternity consumer data from the 6 months pre-MANE delivery and 6–12 months post-MANE delivery will be provided to the research team from the Victorian DHHS and used to investigate whether there are differences in consumer experiences at each health centre following MANE training. Data items collected from the VHES include questions relating to staff in a caring role during the birth and overall perceptions of care. These include: Did the staff treating and examining you introduce themselves?; Were you left alone by midwives or doctors at a time when it worried you?; If you raised a concern during labour and birth, did you feel that it was taken seriously?; If you needed assistance, were you able to get a staff member to help you within a reasonable time frame?; How often did the doctors, midwives and other health professionals caring for you explain things in a way you could understand during your labour and birth?; Did you have confidence and trust in the staff caring for you during your labour and birth?; While you were in hospital, did hospital staff talk about you as if you weren’t there?; Overall, did you feel you were treated with respect and dignity?; Throughout your maternity experience, did you feel listened to and understood?; and Overall, how would you rate the care you received during your pregnancy, labour, birth and after your baby was born? This component addresses the fourth level of the Kirkpatrick Evaluation Model, results, assessing behaviour change at least 12 months post-training.20

Component 4: case studies with five regional or rural health service providers

Outside of metropolitan Melbourne, the Victorian DHHS defines five public hospital regional boundaries within the state (ie, the Barwon South Western, Grampians, Loddon Mallee, Hume and Gippsland regions). Five health service providers, one from each regional boundary, will be invited to participate in case studies for the final component of the evaluation. These facilities will be chosen purposively and at random (depending on MANE delivery at these sites), with ethics approval sought from the health service provider where required. Two level 2 health services, one level 3 service and two level 4 services will be chosen as case study sites across these regional boundaries.

A pre-MANE semi-structured interview will be conducted with the Maternity Manager and/or the Clinical Midwife Educator at these sites to explore their perception of MANE before programme delivery. Approximately 1 month post-MANE delivery, focus groups and key informant interviews will be undertaken to obtain more information on the attitudes of maternity and newborn care clinicians and executive staff about MANE and other maternity emergency programmes in place, and suggestions to improve programme content and delivery, as well as providing a better understanding of the barriers to uptake (if any). Topic areas will mirror the follow-up interviews with all health services 4 months after MANE. An audit of incident reports and referral patterns (transfer from the health service during labour) in the 12 months pre-MANE and post-MANE delivery will also be conducted at these sites to investigate whether MANE has had an impact on these outcomes. The case-study component of the evaluation addresses the third and fourth level of the Kirkpatrick Evaluation Model: behaviour and results.

Sample size considerations

We aim to include all health services who are required to receive MANE training in 2018 and 2019. These health services will have numbers of births ranging from 20 to 1000 per year. No sample size calculations have been undertaken—the aim is to include all eligible consenting respondents in all relevant outcome measures and to provide response rates for each component. Clinical outcome data will not be collected as discussed earlier given the sample size is too small to measure meaningful changes in clinical outcomes.

Patient and public involvement

As this is an evaluation of a maternity education programme delivered to maternity and newborn care clinicians, patient and public involvement has not been included in the research design. However, we will collect anonymous maternity consumer data to investigate whether there are any changes in consumer experience and satisfaction with quality of care following delivery of this training programme.

Data analysis

Survey data will be entered directly into REDCap (Research Electronic Data Capture), a secure web application for building and managing online surveys and databases,31 and transferred to Stata V.1532 for analysis. Where questions have pre-coded response options, analysis will be undertaken using descriptive statistics, including frequencies, proportions and means/medians. Open-ended responses will be coded and collapsed into meaningful themes. In many instances, data will be stratified by service capability and location.

Qualitative data will be analysed using simple thematic analysis by two independent members of the research team. Semistructured interviews and focus groups with Maternity Managers and other key informants will be documented using detailed notes. All potentially identifying information about individuals and individual hospitals will be removed. The research team will use the survey, interview and focus group data to provide a summary of the current situation; to formulate suggestions/solutions on streamlining a state-wide system of maternity service education provision across Victoria; and to inform the development of recommendations on how to implement and sustain such a system over the long term.

Ethics and dissemination

The research team will inform eligible health service sites about the evaluation with the option for health services to opt out if they do not wish to participate. Maternity and newborn care clinicians can choose not to complete staff questionnaires or participate in focus groups or interviews if they so wish. Case study sites, when chosen, will be required to provide written informed consent to participate. Those taking part in focus group and key informant interviews will also provide written consent prior to participation. Verbal consent will be obtained prior to all telephone interviews conducted with health service providers.

The findings of this study will be presented to stakeholders at the Royal Women’s Hospital and the Victorian government in a deidentified confidential report. The report may then be disseminated to various stakeholders within and external to the Victorian DHHS as appropriate. Findings will also be presented in academic journals, at national and international conferences, and presented as part of a PhD thesis. All identifying material will be removed from reports, publications and presentations arising from this study.