The identification of key indicators to drive quality improvement in obstetric anaesthesia: results of the Obstetric Anaesthetists' Association/National Perinatal Epidemiology Unit collaborative Delphi project.

A three-stage Delphi survey process was undertaken to identify the quality indicators considered the most relevant to obstetric anaesthesia. The initial quality indicators assessed were derived from national peer-reviewed publications and were divided into service provision, service quality and clinical outcomes. A range of stakeholders were invited to participate and divided into three panels: obstetric anaesthetists; other maternity care health professionals; and women who had used maternity services. In total, 133 stakeholders registered to participate with 80% completing all three phases of the survey process. Participants ranked indicators for their relative importance using the grading of recommendations assessment, development and evaluation scale. From an initial list of 31 quality indicators, 11 indicators were rated as extremely important by > 90% of participants in at least two panels. These 11 indicators were presented to stakeholders; they were asked to vote for the five indicators they considered most relevant and useful for assessing and benchmarking the quality of obstetric anaesthesia provided. The indicators chosen were: the percentage of women who had an epidural/combined spinal-epidural for labour analgesia with accidental dural puncture; the presence of guidelines for the referral of patients to an anaesthetist for antenatal review; whether there are dedicated elective caesarean section lists; the availability of point-of-care testing for estimation of haemoglobin concentration; and the percentage of epidurals for labour analgesia that provided adequate pain relief within 45 min of the start of epidural insertion. These indicators may be used for quality improvement and national benchmarking to support the implementation of quality standards in obstetric anaesthesia.

Recommendations

A core set of five quality indicators in obstetric anaesthesia, which were derived from national published standards, are recommended as a pragmatic set of priority measures to be used for quality improvement. This core set of indicators can be used locally by maternity services to support quality improvement activity and feed into hospital quality accounts. This indicator set will also assist the development of national benchmarking to support the implementation of quality standards in obstetric anaesthesia. These five quality indicators are

The percentage of women who had an epidural (or a combined spinal epidural) for labour analgesia who had an accidental dural puncture;

Whether there are guidelines for the referral of patients to an anaesthetist for an antenatal review;

Whether there are elective caesarean section lists with dedicated (i.e. not expected to cover emergency work) obstetric, anaesthetic and theatre staff;

Whether point‐of‐care testing is available for estimation of haemoglobin (e.g. Hemocue®, Hemocue AB, Ängelholm, Sweden or blood gas analyser);

The percentage of epidurals for labour analgesia that provided adequate pain relief within 45 min of placement (from the start of epidural insertion).

It is anticipated that the core indicator set will continue to evolve with time to reflect changing practice and quality improvement needs. There is a need to further explore service users’ priorities and expectations for a high‐quality obstetric anaesthetic service.

Introduction

In 2008, the UK Department of Health published ‘High‐quality care for all’ 1, a report that aimed to shift the focus of healthcare delivery from being target driven, to a system with the ability and capacity to deliver higher quality and more effective care. The report proposed that all healthcare providers should produce publicly available annual reports (quality accounts) describing their performance in a series of quality indicators related to patient safety, experience and outcomes. Internationally, there has been a similar focus on driving improvements in the quality of maternity care with regular reporting of metrics. An example is the work of the California Maternal Quality Care Collaborative, a multistakeholder organisation committed to ending preventable morbidity, mortality and racial disparities in maternity care 2. Using a multipronged approach, including quality improvement toolkits and large‐scale data collection to improve health outcomes for women and their babies, California has seen maternal mortality decline between 2006 and 2013, while the maternal mortality rate in the United States continues to rise 3.

Quality indicators are measures that reflect the quality of care or processes linked to improved outcomes 4. They can be used to help identify areas requiring action and to support the development and monitoring of local performance improvement schemes. Quality indicators allow aspects of healthcare to be benchmarked against national data. Indicators can be related to the structure, process or outcomes of healthcare and should be based on evidenced‐based standards of care 5. Different stakeholders in healthcare may have different perspectives on the relative importance of an indicator as a measurement of healthcare quality 6.

A significant amount of work has been undertaken in the UK to set standards for anaesthetic care. The Royal College of Anaesthetists (RCoA) ‘Guidelines for the provision of anaesthetic services (GPAS)’ 7 uses a National Institute for Health and Care Excellence (NICE)‐accredited process to develop recommendations for anaesthetic services in the UK. These recommendations form part of the Anaesthesia Clinical Services Accreditation scheme run by the RCoA to support quality improvement through peer review 8. The RCoA has also published a compendium of audit recipes for continuous quality improvement in anaesthesia 9. Together, these documents contain a large number of standards for service provision and clinical outcomes in obstetric anaesthesia that could be used as quality indicators. Although all these standards have merit, in view of their large number there is a need to consider how these standards could be prioritised. Identification of a core set of prioritised quality indicators in obstetric anaesthesia would facilitate national benchmarking and offer a pragmatic solution to local maternity services seeking to drive service quality improvement.

The Obstetric Anaesthetists’ Association (OAA), in collaboration with National Perinatal Epidemiology Unit (NPEU), undertook a Delphi survey of key stakeholders to prioritise published national standards of care for obstetric anaesthesia. The goal was to identify a core set of standards that were considered by consensus, to be the most important and useful as quality indicators for an obstetric anaesthetic service.

The Delphi method was developed by the RAND Corporation in the Cold War era to assist with forecasting the impact of technology on warfare 10. It is now widely used in various disciplines, including healthcare, to form an expert consensus on a series of topics or questions 11. The Delphi method was judged the most appropriate way to identify an agreed set of obstetric anaesthesia quality indicators. This paper describes the OAA/NPEU Collaborative Delphi Project and resulting indicator set.

Methods

The process followed is summarised in Fig. 1.

Figure 1

Flow chart illustrating the stages of the Delphi Project. QWOG (Quality and Outcomes Working Group).

In an initial consultation process, relevant stakeholder organisations were approached for advice and support and were invited to enlist participants for the Delphi survey. Organisations consulted included NHS England, NHS Improvement, the Royal College of Obstetricians and Gynaecologists, the Royal College of Midwives, the British Maternal and Fetal Medicine Society, Strategic Clinical Networks (Maternity) and the National Maternal Perinatal Audit. Participation of service users was coordinated by the OAA executive committee lay representative.

Indicators were selected for review by identifying existing published standards of care for obstetric anaesthesia 7, 8, 9. These were then reviewed by the OAA's Quality and Outcomes Working Group (QOWG). The QOWG consisted of four consultant anaesthetists who were OAA executive committee members, two consultant anaesthetist members of the OAA, two trainee anaesthetists, a consultant obstetrician, a health services researcher and a lay representative. As well as drawing up a long list of indicators from existing publications, QOWG members also suggested additional items for consideration. Using a 5‐point scoring system (Table 1), each of these indicators was scored independently by each QOWG member for importance as an indicator for measuring and driving improvements in quality of service provision and service delivery in obstetric anaesthesia. The scores were collated, and the results of the scoring were discussed at a meeting of the QOWG. Indicators that were similar were combined. Based on the scoring exercise and group discussions, a list of indicators was produced for inclusion in the first phase of the Delphi survey.

Table 1

Scoring criteria used to select indicators for first phase of the survey

Importance	Score
Very important	4
Probably important	3
Probably not important	2
Not important	1
Don't know	0

Invitations to register to participate in the Delphi Survey were sent out between November and December 2017. These invitations were sent to individuals identified by stakeholder organisations as having leadership roles and interest in quality improvement. Invitees included all lead obstetric anaesthetists in the UK known to the OAA secretariat, from the approximately 220 maternity units that may be expected to have a lead consultant anaesthetist; obstetricians, invited through the Royal College of Obstetricians and Gynaecologists, the British Maternal and Fetal Medicine Society, Clinical Networks and by direct invitation; midwives, maternity managers and policymakers, invited through NHS England, NHS Improvement, Royal College of Midwives and Maternity Clinical Networks; lay members/patient representatives as service users, invited through the OAA; and contact with known third sector organisations working in the maternity sphere.

The participants were separated into three panels consisting of obstetric anaesthetists (referred to as ‘anaesthetists’); other maternity healthcare professionals (obstetricians, midwives and service managers, referred to as ‘other healthcare professionals’); and women/third sector organisations (referred to as ‘service users’). The panels allowed each group of stakeholders to score independently by consensus, in the first two phases of the Delphi survey without influence by the scores of the other groups.

For ease of presentation and scoring, the indicators chosen for the survey were grouped into three domains: service provision (‘what is done’); service quality (‘how it is done’); and clinical outcome (‘how well it is done’). An explanatory description of each indicator was provided to assist participants unfamiliar with particular terminology associated with an indicator. The QOWG lay representative assisted with the drafting of the descriptions to ensure the language used was accessible to all participants.

The grading of recommendations assessment, development and evaluation (GRADE) scale of measurement was chosen for use in scoring the indicators based on recommendations from the core outcome measures in effectiveness trials (COMET) initiative 12 (Table 2).

Table 2

The grading of recommendations assessment, development and evaluation (GRADE) scoring scale

Importance	Score
Extremely important	9
	8
	7
Important	6
	5
	4
Not important at all	3
	2
	1

An online system was used to conduct a three‐phase Delphi survey. The three phases of the Delphi survey were conducted between January and June 2018. A scheduled period of 6 weeks allowed for the completion of each phase. If registered participants had not completed the online survey within 1 week of the phase closing date, a reminder email was sent. Participants who did not complete the online survey by the closure date were deemed not to have completed that phase, thereby to have withdrawn from the survey process and were not sent an invitation to access the subsequent phase. To minimise attrition of participants between phases, participants were presented with a statement at the beginning of each phase to emphasise the importance of completing the whole survey process as recommended by Sinha et al. 13.

In first phase, participants were invited to suggest additional indicators to be included in subsequent phases of the Delphi survey. The QOWG, in liaison with the project team, independently assessed any new indicators proposed by the participants to determine if they represented de novo indicators or were already listed in first phase. De novo indicators agreed by the QOWG were taken forward to second phase of the Delphi Survey. Scores from first phase were analysed separately for each panel. Descriptive analytics were produced which presented as a number and bar chart the percentage of panel participants who gave a particular score for each indicator.

For second phase, participants were shown their previous scores together with the descriptive analysis of the scoring by their panel in first phase and asked to rescore each indicator. After second phase, the QOWG reviewed the scores from first and second phases, and the lowest scoring indicators were removed before third phase. For third phase, the participants were asked to rescore each indicator again, having been supplied with their own scores from second phase and the scores provided by all three panels from second phase.

Following the completion of third phase of the Delphi survey, there was an analysis of the scores and those indicators that scored at least 7–9 (‘extremely important’) amongst 90% of at least two panels were taken forward for presentation and discussion at a consensus meeting in October 2018 in order to determine the core indicator set.

All participants in the Delphi survey were invited to attend the consensus meeting, which was held as a session of an OAA educational meeting. Other participants attending the educational meeting were also invited to contribute to the voting. There was an open group discussion on each of the high scoring indictors. Following discussion, participants were asked to vote to accept or reject each indicator for inclusion in a final core set of five judged to be the most important, useful and measurable quality indicators for obstetric anaesthesia. The voting was electronic and anonymous.

Results

Pre‐survey preparation involved the QOWG compiling an initial ‘long list’ of 42 indicators from ‘Guidelines for the provision of anaesthesia services (GPAS) Guidance on the Provision of Obstetric Anaesthesia Services, Royal College of Anaesthetists, 2015’ 7, ‘Anaesthesia Clinical Services Accreditation, Royal College of Anaesthetists, 2016’ 8 and ‘Raising the Standard: a compendium of audit recipes 3rd Edition Section 8: Obstetrics, Royal College of Anaesthetists 2012’ 9. After removal of duplicates and consideration of additional suggestions from QOWG members, 31 indicators across the 3 domains of ‘service provision’ (13 indicators), ‘service quality’ (eight indicators) and ‘clinical outcome’ (10 indicators) were included for evaluation in first phase of the Delphi process.

Details of participants at each stage of the Delphi survey (phases 1–3) are shown in Table 3. There were 133 participants initially registered for the Delphi Survey, with 98 (74%) participating in first phase. Of those who took part in first phase, 70 (71%) were obstetric anaesthetists. Of those who participated in first phase, 78 continued to participate in phases 2 and 3, resulting in 80% retention rate across the three phases. Voting across the three phases was as follows: for first phase, 31 indicators were scored by 98 participants; for second phase, 32 indicators were scored by 87 participants (this included the 31 indicators from first phase that were rescoed by participants in light of the first round of scoring, plus an additional indicator added following QOWG review of suggestions by participants for additional indicators for inclusion); and for third phase, eight of the lowest scoring indicators from second phase were excluded. The remaining 24 indicators were scored by 78 participants.

Table 3

Participants in the Delphi survey with proportion completing each stage. Values are number completed/number eligible

Participants	Completed first phase	Completed second phase	Completed third phase	Retention between first and third phases
Anaesthetists	70/91	63/70	58/63	58/70
Obstetricians	10/12	9/10	8/9	8/10
Midwives	5/11	4/5	3/4	3/5
Managers	3/3	2/3	2/2	2/3
Service users	10/16	9/10	7/9	7/10
Total	98/133	87/98	78/87	78/98

The indicators scored during the three phases are shown in Table 4. Table 4 also shows the percentage of participants who gave each indicator a score 7–9 at each phase. The 11 indicators that scored 7–9 amongst 90% of at least two panels in third phase are summarised in Table 5. These indicators were presented to the consensus meeting for review.

Table 4

Proportion of all participants in each Delphi phase who scored an indicator between 7 and 9 (‘extremely important’)

Indicator	First phase	Second phase	Third phase
Whether a rapid‐infuser device is available	63%	68%	75%
Whether a blood gas analyser is available to measure serum lactate in adults	74%	84%	91%
Whether point‐of‐care testing is available for estimation of haemoglobin	80%	90%	91%
Whether there is at least one fully equipped and staffed obstetric theatre within the delivery unit	97%	100%	100%
Whether there are elective caesarean section lists with dedicated (i.e. not expected to cover emergency work) obstetric, anaesthetic and theatre staff	72%	83%	91%
Whether there is patient information available to women on how to access medical advice if they experience a postnatal complication that may be related to their intrapartum anaesthetic care	–	71%	81%
Whether there is ultrasound imaging equipment available for use for anaesthetic procedures (e.g. central vascular access, transversus abdominis plane blocks, epidural/spinal cannulation)	40%	44%	55%
Whether there is O rhesus‐negative blood immediately (within 5 min) available at all times for emergency use	91%	97%	96%
Whether there is intralipid immediately available if needed to treat local anaesthetic toxicity?	82%	94%	–
Whether there is sugammadex immediately available for reversal of rocuronium‐induced neuromuscular paralysis	63%	72%	–
Whether there is equipment available to enable bed‐side estimation of coagulation (e.g. thromboelastography or thromboelastometry)	28%	24%	–
Whether there is additional separate consultant anaesthetist cover for elective caesarean section lists	69%	75%	81%
Whether there is a separate consultant anaesthetist on‐call (out of hours) rota for obstetrics without other on‐call commitments (e.g. intensive care, general theatres)	31%	36%	–
Whether there is an anaesthetic antenatal clinic	69%	82%	90%
Whether there is a process for identifying and/or follow‐up of women left with long‐term complications due to a peripartum anaesthetic procedure	73%	83%	92%
Whether there are guidelines for the referral of patients to an anaesthetist for an antenatal review	78%	91%	93%
Whether there is the facility to offer a cell salvage service at any time that it is needed	43%	42%	–
The number of hours (i.e. timetabled primary duty) of consultant anaesthetic presence on the delivery unit per week?	68%	78%	86%
The percentage of caesarean sections carried out with regional anaesthesia categorised by caesarean section urgency	62%	76%	88%
The percentage of elective caesarean section operations that had to be rescheduled on the day of admission due to the need to give priority to emergency operations	55%	63%	76%
The percentage of women who had labour epidural analgesia who were attended by an anaesthetist within 30 min of the anaesthetist being informed of the request for labour regional analgesia	43%	41%	47%
The percentage of women who had labour epidural analgesia who were attended by an anaesthetist within 60 min of the anaesthetist being informed of the request for labour regional analgesia	61%	70%	–
The percentage of women who had regional anaesthesia for caesarean section who are changed to general anaesthesia after the commencement of surgery	65%	77%	86%
The percentage of women who had general anaesthesia for caesarean section who report having awareness during general anaesthesia	91%	92%	94%
The percentage of epidurals for labour analgesia that provided adequate pain relief within 45 min of placement (from the start of epidural insertion)	58%	77%	87%
The percentage of epidurals inserted for labour analgesia that were replaced at any time during labour	53%	60%	–
The percentage of women who had an epidural (or a combined spinal‐epidural) for labour analgesia who had an accidental dural puncture	76%	91%	92%
The percentage of women who had spinal analgesia who had a postnatal post‐dural puncture headache	58%	65%	–
The percentage of women who had epidural labour analgesia who had a postnatal post‐dural puncture headache	74%	82%	87%
The percentage of women who had epidural (or a combined spinal‐epidural) labour analgesia who required an epidural blood patch	70%	80%	81%
The percentage of women who have postnatal follow‐up after receiving an intrapartum anaesthetic intervention	57%	68%	80%
The percentage of women given a general anaesthesia for caesarean section who had a failed tracheal intubation after induction	81%	93%	91%

Table 5

Final indicator list presented at the consensus meeting categorised by domain (SP, service provision; SQ, service quality; CO, clinical outcome)

Domain	Indicator	Panel			Vote yes
		OA	OHP	SU
SP	Whether a blood gas analyser is available to measure serum lactate in adults		✔	✔	45%
SP	Whether point‐of‐care testing is available for estimation of haemoglobin	✔	✔		56%
SP	Whether there is at least one fully equipped and staffed obstetric theatre within the delivery unit	✔	✔	✔	56%
SP	Whether there are elective caesarean section lists with dedicated (i.e. not expected to cover emergency work) obstetric, anaesthetic and theatre staff	✔	✔	✔	57%
SP	Whether there is O rhesus‐negative blood immediately (within 5 min) available at all times for emergency use.	✔	✔		71%
SP	Whether there is an anaesthetic antenatal clinic	✔	✔		38%
SQ	Whether there is a process for identifying and/or follow‐up of women left with long‐term complications due to a peripartum anaesthetic procedure	✔	✔	✔	47%
SQ	Whether there are guidelines for the referral of patients to an anaesthetist for an antenatal review	✔	✔		60%
CO	The percentage of women who had general anaesthesia for caesarean section who report having awareness during general anaesthesia	✔	✔	✔	8%
CO	The percentage of epidurals for labour analgesia that provided adequate pain relief within 45 min of placement (from the start of epidural insertion)		✔	✔	48%
CO	The percentage of women who had an epidural (or a combined spinal‐epidural) for labour analgesia who had an accidental dural puncture	✔	✔		72%

✔ represents which panels (OA, obstetric anaesthetists; OHP, other health professionals; SU, service users) had 90% of panel members give a 7–9 score in Third phase. The ‘Vote Yes’ is the percentage of consensus meeting participants who voted ‘Yes’ for inclusion of the indicator as one of the five most important quality indicator in the list. The shaded rows represent the indicators selected for inclusion in the core set.

During the consensus meeting, 6 out of the 11 indicators were initially voted for inclusion in the core set by over 50% of meeting participants with two indicators having a tied vote. On discussion, a majority of meeting participants considered that at least two of the proposed indicators “Whether there is O rhesus‐negative blood immediately (within 5 min) available at all times for emergency use” and “Whether there is at least one fully equipped and fully staffed obstetric theatre within the delivery unit” were already attained by nearly all units in the UK. It was the consensus view that these two indicators should be excluded from the core set.

There was one indicator “The percentage of epidurals for labour analgesia that provided adequate pain relief within 45 min of placement (from the start of epidural insertion)” that had been scored highly by both the ‘service user’ and ‘other healthcare professional’ panels but had not passed the scoring threshold (i.e. scored 7–9 by at least 90% of panel participants) in the ‘anaesthetists’ panel. The indicator came close to the voting threshold for inclusion in the final core set but had not passed it. Several meeting participants commented that they would be more comfortable with the indicator set if this metric was included, especially as it was recognised that service users were under‐represented at the consensus meeting. Although it was also highlighted that this indicator might be difficult to measure, it was agreed to include this indicator in the final core set of five indicators. The final core set are shown as shaded rows in Table 5.

Discussion

Our Delphi survey canvassed service users, managers and clinicians to identify five core indicators for driving quality improvement in obstetric anaesthesia. These indicators covered the domains of service provision, service quality and clinical outcomes. Two clinical outcome indicators were included in the final core set.

The selection of participants is a crucial factor in the success of the Delphi technique. There is a necessity to have participants who have specialised knowledge, for example anaesthetists, but others may be ‘expert’ by experience, for example, service users. A varied panel may also be best in achieving a credible consensus with a broad perspective 14. It is essential for processes involving judgement on the quality of healthcare that the perspective of the service user is sought. Our process involved anaesthetists, obstetricians, midwives, service managers and service users. Obstetric anaesthetists were by far the biggest group among participants making up 71% of those completing first phase, compared with only 10% who were service users. The use of separate voting panels helped to ensure that the views of service users were represented. It is disappointing that less than 50% of senior midwives who had registered their interest to be a participant were able to contribute to the Delphi process, and that the number of service managers who participated was also low. The reasons for this are not known.

It is recommended that a Delphi process should involve three rounds within a 4‐month period 11. The number of rounds is a balance between allowing enough opportunities for participants to come to a consensus and the inevitable attrition as participants fall away from the process. Our process involved three rounds within a 5‐month period, and the total number of completions in third phase was 80% of those who completed first phase. This attrition rate of 20% is within the interquartile range found in a systematic review of other similar Delphi technique surveys 15 and exceeded the recommendation of maintaining a 70% response rate for a Delphi survey 11.

Our process used a 9‐point Likert scale for participants to rate an indicator. The method for selecting those indicators that achieved consensus was a median threshold score selected by a defined proportion of panel participants, which was also the most common method used by other similar studies identified in a systematic review 15.

Indicators that are used for quality improvement need to be important, valid, feasible and meaningful 16. The publication of national quality standards that have undergone a peer review process provides a valuable source for deriving indicators that can be used. The indicators chosen should reflect the domains of healthcare as described by Donabedian, that is, structure, process and outcome 17. These domains were represented in our Delphi survey by the quality indicators that reflected service provision (‘what is done’), service quality (‘how it is done’) and clinical outcome (‘how well it is done’). Participants were also given the opportunity to make suggestions for additional indicators. Only one additional indicator was added to the survey as a result of the participant feedback “Whether there is patient information available to women on how to access medical advice if they experience a postnatal complication that may be related to their intrapartum anaesthetic care”.

A high proportion of the indicators in the final phase and final core set are process indicators. This may reflect that for many participants, the provision of service was a priority compared with the quality or outcome of a service. However, one of the primary considerations at the consensus meeting was the measurability of the indicators, and the predominance of process indicators in the final core set may also indicate the lack of availability of obstetric anaesthetic outcomes amongst routinely collected data. If the quality of obstetric anaesthetic services, as an essential part of obstetric services, is to continue to improve, such outcomes must be included in mandated data collection.

The difficulty anaesthetists have in obtaining useful quality data on clinical outcomes, even if these outcomes are considered to be important, was highlighted by the comments received from participants during the Delphi Survey. A recent survey of lead anaesthetists in the UK found that 30% of respondents reported having insufficient resources to analyse relevant clinical data 18. Despite these challenges, 97% of survey respondents were enthusiastic about being able to benchmark local data against national peer data 18.

A limitation of this project was the relatively low number of service users who participated in the final consensus meeting, which may have influenced the final choices of quality indicators. There is a need to further explore what ‘service users’ rate most highly as indicators of a high‐quality obstetric anaesthetic service. We anticipate that the core indicator set will continue to evolve with time to reflect changing practice and quality improvement needs.

This process has identified a core set of five quality indicators in obstetric anaesthesia to be used as a pragmatic set of priority measures for quality improvement. Maternity services can use this core set of indicators to support local quality improvement activity and feed into hospital quality accounts. The intention is that the indicator set will also assist the development of national benchmarking to support the implementation of quality standards in obstetric anaesthesia. The OAA will be working with its members and other national organisations to seek the adoption of these core quality indicators within national audit processes, and to develop a process to keep under review which core quality indicators remain the most useful for driving quality improvement.