WHO Standards-based questionnaire to measure health workers' perspective on the quality of care around the time of childbirth in the WHO European region: development and mixed-methods validation in six countries.

OBJECTIVES: Develop and validate a WHO Standards-based online questionnaire to measure the quality of maternal and newborn care (QMNC) around the time of childbirth from the health workers' perspective.
DESIGN: Mixed-methods study.
SETTING: Six countries of the WHO European Region. PARTICIPANTS AND METHODS: The questionnaire is based on lessons learnt in previous studies, and was developed in three sequential phases: (1) WHO Quality Measures were prioritised and content, construct and face validity were assessed through a Delphi involving a multidisciplinary board of experts from 11 countries of the WHO European Region; (2) translation/back translation of the English version was conducted following The Professional Society for Health Economics and Outcomes Research guidelines; (3) internal consistency, intrarater reliability and acceptability were assessed among 600 health workers in six countries.
RESULTS: The questionnaire included 40 items based on WHO Standards Quality Measures, equally divided into four domains: provision of care, experience of care, availability of human and physical resources, organisational changes due to COVID-19; and its organised in six sections. It was translated/back translated in 12 languages: Bosnian, Croatian, French, German, Italian, Norwegian, Portuguese, Romanian, Russian, Slovenian, Spanish and Swedish. The Cronbach's alpha values were ≥0.70 for each questionnaire section where questions were hypothesised to be interrelated, indicating good internal consistence. Cohen K or Gwet's AC1 values were ≥0.60, suggesting good intrarater reliability, except for one question. Acceptability was good with only 1.70% of health workers requesting minimal changes in question wording.
CONCLUSIONS: Findings suggest that the questionnaire has good content, construct, face validity, internal consistency, intrarater reliability and acceptability in six countries of the WHO European Region. Future studies may further explore the questionnaire's use in other countries, and how to translate evidence generated by this tool into policies to improve the QMNC. TRAIL REGISTRATION NUMBER: NCT04847336. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This is a multicountry study on the development and validation of a WHO Standards-based questionnaire for conducting online surveys on quality of maternal and newborn care, from the perspective of health workers in WHO European region.

The major strength of the questionnaire is the multiphase approach used for its development: the questionnaire was based on lessons learnt and adapted from a pilot study; content, construct and face validity were assessed throughout a Delphi study among a multidisciplinary group of experts; the tool was then translated and back translated following the Professional Society for Health Economics and Outcomes Research guidance.

Internal consistency, intrarater reliability and acceptability were assessed among a large sample of health workers from six countries of WHO European Region.

One disadvantage is that the process of validation can be quite lengthy, and, as such especially in the context of a pandemic such as COVID-19.

Introduction

High-quality respectful care around the time of childbirth is a fundamental aspect of human rights and, according to recent global estimates, could prevent more than 100 000 maternal deaths and 1.3 million neonatal deaths annually.1–3 Despite some maternal and newborn health indicators in high-income countries being better in comparison to low-income and middle-income countries, existing evidence shows that improvements are needed in the quality of care provided to women and newborns in every country.4–9

The COVID-19 pandemic has challenged health systems worldwide and increased pre-existing fragilities such as the shortage of skilled professionals and equipment, potentially exacerbating health inequities and increasing social and economic disparities, both among and within countries.10–13 Rapid changes in the workplace and in the procedures of delivering care have constrained the quality of maternal and newborn care (QMNC) and has increased stress among health workers.14 15 Global maternal and fetal outcomes have worsened during the COVID-19 pandemic, with an increase in maternal deaths, stillbirths, and maternal depression.16–18 The pandemic has also amplified the need to improve data collection systems, to enhance the monitoring of key indicators, and to better manage the public health response to current and future emergencies.11 12 19–21

In 2016, the WHO developed a framework22 and a list of Standards23 for improving the QMNC. The WHO Standards23 define a set of 318 Quality Measures, divided into three key domains—experience of care, provision of care and availability of resources—which can be used by hospital managers to assess the QMNC. Many of these WHO Quality Measures—such as those related to the availability of equipment, training opportunities and quality improvement initiatives—should be assessed by utilising health workers as one of the sources of data. Both service providers’ and services users’ perspectives are critical to assess QMNC and get important suggestions for health system improvement.24 Health workers are the cornerstone of any health system, having a key role in contributing to health services preparedness and response to emergencies, but often they are poorly involved in designing the quality improvement mechanisms.24–27 Exploring health workers’ perspectives on key aspects of provision of care, experience of care, availability of resources and the reorganisation of the health services will provide critical information on the QMNC, but also has the potential, if properly implemented through a participatory quality improvement approach, to increase staff ownership on critical aspects of QMNC, to improve working conditions and to increase motivation of workforces.24–26

There are a lack of WHO Standards-based validated tools for collecting data on health workers perspectives of the QMNC.13 28–31 A unified comprehensive approach to measure QMNC as defined by the WHO Standards, through validated tools, would allow for comparisons of data across settings and over time, and allow for efficient monitoring.22–25 In particular, in the context of the COVID-19 pandemic, innovative methods enabling rapid data collection from multiple countries is essential.

Since 2016, the WHO Collaborating Centre for Maternal and Child Health Burlo Garofolo, Trieste, Italy, has worked on developing and validating tools to collect data on priority WHO Quality Measures. Pilot studies were conducted in Italy between 2016 and 201929–33 and were scaled up in a multicountry project in the WHO European Region, called IMAgiNE (Improving Maternal Newborn CarE).34

Through the IMAgiNE study research network, two complementary questionnaires were conceived: one questionnaire to collect key WHO Quality Measures from the perspective of key service users (ie, mothers) and one from the perspective of health workers, each including 40 priority WHO Quality Measures. Results on the development, validation and use of the questionnaire from the perspective of service users has been reported elsewhere.30 This paper describes the process of development and validation of the second tool, the questionnaire for health workers.

Methods

Pilot studies

The IMAgiNE health workers’ questionnaire was developed based on pilot studies conducted between 2016 and 2019; these studies have been described elsewhere.29 Briefly, development of the tools in the pilot studies included an extensive literature review (online supplemental table 1) and a Delphi study with a multidisciplinary group of international experts to assess content validity, construct coherence and face validity.29 Thereafter, the questionnaire was field tested to further assess face validity and evaluate acceptability, and the tool was improved at each stage.29 Finally, it was used in one tertiary facility in Italy, showing good acceptability. In all, 105/136 (77.2%) of the health workers answered the questionnaire, with good utility: data were reported to be used by 35 decision-makers for developing written recommendations for improving QMNC in their facilities.29 This version of the questionnaire included 117 multiple choice questions, plus three questions on sociodemographic variables and one open-ended question.

Development of the questionnaire for the WHO European Region

During July 2020–June 2021, the questionnaire was further optimised and adapted for the IMAgiNE project among countries of the WHO European Region, and updated to collect data on QMNC during the COVID-19 pandemic. The process included three phases (figure 1), based on recommendations for development of health-related questionnaires35–40 and previous pilot studies.29–33 The characteristics of the questionnaire are synthesised in table 1. Considering logistic limitations of printed questionnaires during the COVID-19 pandemic and the possibility of using diverse recruitment strategies (ie, institutional emails and websites) according to the professional profile of the study population41–43 an online format was chosen. Online questionnaires can be fielded quicker, and less expensively, than traditional mail questionnaires.43 It was predefined that the questionnaire had to collect a set of 40 prioritised WHO Quality Measures,23 ten for each of the four domains of the tool: provision of care, experience of care, availability of human and physical resources, and organisational changes related to the COVID-19 pandemic response.

Figure 1

Phases of questionnaire development and multicountry validation.

Table 1

Characteristics of the IMAgiNE questionnaire for health workers

Expected use	Collect useful data to improve the QMNC during childbirth at facility level in the WHO European Region
Phenomena of Interest	QMNC as for a set of 40 prioritised WHO Quality Measures23
Domains	Four domains: Provision of care, as defined by WHO Standards23 Experience of care, as defined by WHO Standards23 Availability of Physical and Human Resources, as defined by WHO Standards23 Organisational changes related to the COVID-19 pandemic response
Responders	Health workers are defined as professionals routinely working in maternal and neonatal care around the time of childbirth at facility level for at least 1 year
Context	WHO European Region
Administration format	Self-administered, online, anonymous and voluntary Informed consent required
Other characteristics	Multi-item instrument including different dimensions of QMNC Collects information of the key WHO Quality Measures for which using health workers as source of data is appropriate and important Complementary to a tool collecting information on service users’ perception of QMNC30 Content heavily informed by health workers (ie, end users of the questionnaire) Content sufficiently comprehensive, retaining acceptability by health workers Clear, specific, and concise questions, structured in logical sequence Comparable pathways for maternal and neonatal area Includes open-ended questions to collect suggestions from health workers and additional feedback Socio demographic country-adjustable questions under requests Good psychometric properties according to literature Allowance for scoring of QMNC with a single quantitative index

QMNC, Quality of maternal and neonatal care.

Since the process was carried forward over 12 months, and the research network of the IMAgiNE study was growing in the meanwhile, during subsequent phases an increasing number of partners from different countries was involved.

Phase 1: content, construct and face validity

First, the questionnaire developed for the pilot study in Italy29 was translated and back translated into English, by native speakers, and was then adapted for an online survey.

Second, additional items related to the implementation of appropriate procedures and resources for the COVID-19 pandemic were developed in order to assess the health facilities’ preparedness and response during the COVID-19 pandemic. During March to July 2020, relevant WHO guidelines, professional association statements, protocols released by European countries, and studies were searched and selected for review by three experts (EPV, BC and ML). Additionally, reference lists of selected studies were handsearched, grey literature related to respectful care and health services preparedness to COVID-19 pandemic was reviewed using Google, and experts in the field (from WHO and from other networks) were consulted.13 44–64

Third, 40 of the existing WHO Quality Measures were prioritised through a Delphi process, involving two rounds. The Delphi process35–40 was carried out among a multidisciplinary group of 26 experts (psychologists, physicians, midwives, lactation consultants, and reproductive rights advocates) from 11 countries of the WHO European Region. Experts were asked to: (1) prioritise the 40 Quality Measures, ten for each of the four domains of the questionnaire; (2) assess comprehensiveness and clarity of the questions and suggest rewording; (3) suggest additional relevant questions and (4) provide any other comments on content, construct, structure, face validity and expected acceptability of the questionnaire. An ad hoc template was used for this purpose and criteria to reach consensus were predefined.38 Questions were developed in two different and parallel pathways: one for health workers providing care in the maternal area and one for health workers providing care in neonatal area. The questionnaire structure was developed based on the principle of increasing comprehension and acceptability from respondents: interrelated questions were numbered in a logical sequence,35 and were organised in six sections (eg, sections A, B, C…).

In addition, a QMNC index, to be used as a complementary measure of QMNC, in line with other studies,30 31 65–67 was developed through the Delphi process. Further testing of this index is ongoing.

Phase 2: translation and back translation

Translation and back translations from English to other languages were performed by native speakers that are experts in healthcare and/or health research, and project partners, following the steps of Professional Society for Health Economics and Outcomes Research Task Force for Translation and Cultural Adaptation Principles of Good Practice, which include: (1) preparation; (2) forward translation; (3) reconciliation; (4) back translation; (5) back translation review; (6) harmonisation; (7) cognitive debriefing; (8) review of cognitive debriefing results; (9) proofreading and (10) final report.40 An ad hoc template was used, allowing the translation and back translation of each question separately (online supplemental table 2).

Phase 3: internal consistency, intrarater reliability and acceptability in six countries

This phase was performed using data from the following European regions and countries: South Europe (Italy and Portugal), Scandinavia (Norway and Sweden) and East Europe (Croatia and Romania). Data were recorded using REDCap V.8.5.21 —2021 Vanderbilt University, via a centralised platform.

Internal consistency was analysed using Cronbach’s alpha correlation (alpha) for all sections where questions were meant to be interrelated (ie, sections C, D and E). For the three European regions, a required sample size of 104 respondents for analysis of work organisation, data management and communication (section C), 106 for Quality of care and practices performed (section D), and 104 for COVID-19 preparedness and response (section E) was calculated. In the null hypothesis an alpha of 0.55, in the alternative hypothesis an alpha of at least 0.70, 80% power was assumed, several items equal to 8, 12, and 16 for sections C, D and E, respectively, and a significance level of 2.5% with a one-tailed test. Internal consistency was considered good whenever Cronbach’s alpha ≥0.70.68

Intrarater reliability was analysed on all questions on Quality Measures, using the Cohen’s kappa (K) statistic69 by inviting volunteer health workers to answer the questionnaire twice (test–retest responses with a maximum time gap of 7 days between the two responses). The estimated minimum sample size was 89 health workers, assuming in the null hypothesis a K value of 0.35, in the alternative hypothesis a K of at least 0.60, 80% power, a significance level of 2.5% with a one-tailed test and an anticipated proportion of the three possible answers of 0.10, 0.30 and 0.60. As additional parameter of intra-rater reliability, the Gwet AC1 was calculated to consider the possibility of Cohen’s Kappa paradox (ie, low kappa values in presence of a high degree of agreement due to substantial imbalance in the table’s marginal totals).70 71 For values of K or Gwet AC1 >0.60, the intra-rater reliability was considered good.72 Due to the limited sample enrolment, for this analysis data for all countries were considered together. Data were analysed using SAS (Statistical Analysis Software V.9.4, SAS Institute) and R V.3.6.1.

Acceptability was evaluated by analysing responses to one open-ended question in the questionnaire, which explicitly asked health workers to comment on the quality of the questionnaire and provide practical suggestions on how to improve it. All comments were analysed in their national language by native speakers that are experts in healthcare and/or health research and project partners.

Findings of all steps above were used for the final questionnaire optimisation, following consensus agreement among all partners of the IMAgiNE project, which at this stage included a multidisciplinary group of 58 partners from 19 countries in the WHO European Region.

Before participating, consent was requested and all participants were informed about the objectives and methods of the study, including their rights in declining participation (a complete privacy policy was available for download). Anonymity was ensured by not collecting any information that could disclose participants’ identity.

Patient and public involvement statement

Health workers from several countries participated in the development, content and construct validation, assessment of face validity, internal consistency, intrarater reliability and acceptability of the questionnaire. Inputs received were used to optimise the questionnaire.

Results

Phase 1: content, construct and face validity

As a result of the first step, the online English questionnaire was made available.

Second, 22 Quality Measures were generated for the COVID-19 preparedness and response section of the questionnaire.

The Delphi process with international experts prioritised 40 Quality Measures and defined a core set of 13 sociodemographic variables. It also optimised both the wording and the structure of the questionnaire, and added additional open-ended questions. The final questionnaire structure included six sections (online supplemental table 3). Table 2 shows the list of 40 key Quality Measures by domain.

Table 2

Quality measures of the IMAgiNE questionnaire for health workers

Provision of care*	Experience of care*	Availability of resources*	Organisational changes due to COVID-19 pandemic response
1.Availability of sufficient quantities of equipment and supplies for care of both healthy women/newborns	1.Adequate handover	1.Adequate continuity of care infrastructures for continuity of care of both healthy women/newborns	1.Existence of dedicated paths for patients with suspected/confirmed COVID-19
2.Availability of guidelines and protocols for case management of healthy women/newborns	2.Effective communication with users	2.Adequate infrastructure for essential care during emergencies	2.Regular distribution of HW personal protective equipment in sufficient number
3.Effective training on case management of both healthy women/newborns†‡	3.Availability of education materials for users	3.Availability of appropriate and functioning equipment and supplies during emergencies	3.Appropriate number of functioning and accessible hand hygiene stations
4.Effective in-service supportive supervision on case management of healthy women/newborns	4.Effective training on communication with women/families and counseling†	4.Existence of effective tutoring organised during emergencies	4.Availability of updated guidelines based on international recommendations
5.Availability of guidelines and protocols for emergencies	5.Labour companionship guaranteed	5.Sufficient staff number to ensure adequate care	5.Sufficient COVID-19 nasopharyngeal swabs
6.Effective training on case management of emergencies†§	6.Effective training in providing emotional support†	6.Clear definition of roles and responsibilities	6.Adequate information and training for HW on key procedures related to COVID19
7.Functional referral system for emergencies	7.Adequate infrastructures to ensure users’ privacy	7.Existence of clinical data collection system	7.Closure of healthcare facilities or routine services reduction due to COVID-19 reorganisation changes
8.Existence of systems to routinely monitor quality of care	8.Availability of consent request material aids	8.Existence of protocols to guarantee privacy	8.Sufficient number of health workers for essential care
9.Weekly clinical meetings	9.Effective training on informed consent†	9.Existence of a quality of care improving dedicated team	9.Silensing (censorship) of staff to avoid reporting of inadequate practices
10.Existence of maternal and/or neonatal deaths audits	10.Effective training on pain relief practices†	10.Effective training covering rights of women/newborns†	10.Critical changes in the provision of care due to COVID 19 pandemic¶

*Based on WHO standards.

†At least one training event in the last 3 years.

‡Only for maternal area path: Partogram, fetal well-being, unnecessary caesarean section—only for neonatal area path: breastfeeding promotion, skin-to-skin, standards precautions.

§Only for maternal area path: postpartum haemorrhage, eclampsia, shoulder dystocia, pregnant woman cardiovascular arrest—only neonatal area path: newborn resuscitation.

¶Increase medicalisation and/or limitations on companionship, labour movements, pain relief, rooming-in, breastfeeding, skin to skin in absence of clear medical indications.

In addition, a QMNC Index was developed. A predefined score (eg, 0-5-10 points) was attributed to each possible answer of each one of the 40 questions on Quality Measures of the IMAgiNE questionnaire for health workers. Higher scores indicating higher adherence to WHO Standards. The sum of all points in one specific domain could range from 0 to 100, while the total QMNC Index could range from 0 top 400 considering all domains (online supplemental table 4).

Phase 2: translation and cultural adaptation

The IMAgiNE questionnaire for health workers was translated and back translated into the following 12 languages: (1) Bosnian, (2) Croatian, (3) French, (4) German, (5) Italian, (6) Norwegian, (7) Portuguese, (8) Romanian, (9) Russian, (10) Slovenian, (11) Spanish and (12) Swedish.

Phase 3: internal consistency, intrarater reliability and acceptability in six countries

A total of 600 health workers participated in this phase; the sample included a heterogeneous group of professionals with different ages, genders, professional roles and experience. More than half of health workers had more than 10 years of experience in maternal and neonatal health (54.3%) with midwives representing 48.5% of the total sample. Detailed characteristics are presented in table 3.

Table 3

Health workers’ characteristics

Health workers	Totaln (%)(N=600)	South Europe		Scandinavia		East Europe
		Italyn (%)(N=190)	Portugaln (%)(N=89)	Norwayn (%)(N=91)	Swedenn (%)(N=93)	Croatian (%)(N=44)	Romanian (%)(N=93)
Age (range, years)
20–29	58 (9.7)	25 (4.2)	9 (1.5)	10 (1.7)	2 (0.3)	6 (1.0)	6 (1.0)
30–39	181 (30.2)	60 (10.0)	26 (4.3)	27 (4.5)	36 (6.0)	9 (1.5)	23 (3.8)
40–49	158 (26.3)	59 (9.8)	11 (1.8)	24 (4.0)	20 (3.3)	12 (2.0)	32 (5.3)
50–59	123 (20.5)	34 (5.7)	8 (1.3)	16 (2.7)	24 (4.0)	8 (1.3)	26 (4.3)
60–69	31 (5.2)	4 (0.7)	7 (1.2)	8 (1.3)	7 (1.2)	1 (0.8)	4 (0.7)
≥70	2 (0.3)	0	0	2 (0.3)	0	0	0
Missing	47 (7.8)	8 (1.3)	21 (3.5)	4 (0.7)	4 (0.7)	8 (1.3)	2 (0.3)
Gender (self-described)
Male	36 (6.0)	11 (1.8)	8 (1.3)	0	6 (1.0)	5 (0.8)	6 (1.0)
Female	511 (85.2)	166 (27.7)	60 (10.0)	87 (14.5)	83 (13.8)	31 (5.2)	84 (14.0)
Other	0	0	0	0	0	0	0
Nonbinary/genderfluid/agender	0	0	0	0	0	0	0
Preferred not to answer	6 (1.0)	5 (0.8)	0	0	0	0	1 (0.8)
Missing	47 (7.8)	8 (1.3)	21 (3.5)	4 (0.7)	4 (0.7)	8 (1.3)	2 (0.3)
Professional qualification
General physician	8 (1.3)	4 (0.7)	1 (0.2)	0	0	0	3 (0.5)
Working in maternal care	3 (0.5)	1 (0.2)	0	0	0	0	2 (0.3)
Working in neonatal care	5 (0.8)	3 (0.5)	1 (0.2)	0	0	0	1 (0.2)
Midwife	291 (48.5)	100 (16.7)	15 (2.5)	88 (14.7)	57 (9.5)	19 (3.8)	12 (2.0)
Working in maternal care	244 (40.7)	86 (14.3)	13 (2.2)	83 (13.8)	39 (6.5)	11 (1.8)	12 (2.0)
Working in neonatal care	4 (0.7)	0	0	0	3 (0.5)	1 (0.2)	0
Working in both areas of care	40 (6.7)	14 (2.3)	2 (0.3)	4 (0.7)	13 (2.8)	7 (1.2)	0
Nurse	139 (23.2)	37 (6.2)	29 (4.8)	3 (0.5)	3 (0.5)	11 (1.8)	56 (9.3)
Working in maternal care	54 (9.0)	10 (1.7)	1 (0.2)	3 (0.5)	1 (0.2)	8 (1.3)	31 (5.2)
Working in neonatal care	85 (14.2)	27 (4.5)	28 (4.7)	0	2 (0.3)	3 (0.5)	25 (4.2)
Neonatology physician	60 (10.0)	15 (2.5)	30 (5.0)	0	2 (0.3)	2 (0.3)	11 (1.8)
Ob&gyn physician	72 (12.0)	34 (5.7)	9 (1.5)	0	11 (1.8)	10 (1.7)	8 (1.3)
Registrar/medical resident	28 (4.7)	0	4 (0.7)	0	19 (3.2)	2 (0.3)	3 (0.5)
Obstetrics and gynaecology	23 (3.8)	0	1 (0.2)	0	17 (2.8)	2 (0.3)	3 (0.5)
Neonatology	5 (0.8)	0	3 (0.5)	0	2 (0.3)	0	0
Years of work in MNH area
<5 years	118 (19.7)	40 (6.7)	16 (2.7)	18 (3.0)	23 (3.8)	8 (1.3)	13 (2.2)
5–10 years	110 (18.3)	39 (6.5)	13 (2.2)	19 (3.2)	17 (2.8)	4 (0.7)	18 (3.0)
>10 years	326 (54.3)	104 (17.3)	39 (6.5)	50 (8.3)	49 (8.2)	24 (4.0)	60 (10.0)
Missing	46 (7.7)	7 (1.2)	21 (3.5)	4 (0.7)	4 (0.7)	8 (1.3)	2 (0.3)
Type of facility
Public	575 (95.8)	175 (92.1)	87 (97.8)	91 (100)	93 (100)	44 (100)	85 (91.4)
Private	25 (4.2)	15 (7.9)	2 (2.2)	0 *	0 †	0	8 (8.6)

*There are no private facilities in Norway.

†There is only one private facility in Sweden.

MNH, maternal and/or neonatal health; Ob&gyn, obstetrics and gynaecology.

The Cronbach’s alpha values were ≥0.70, showing good internal consistency for all sections analysed. It is presented in online supplemental table 5.

Findings on intrarater reliability are reported in online supplemental table 6. Overall, 164 health workers answered the questionnaire twice (test–retest), thus resulting in a power of 0.97. All K values or Gwet’s AC1 (in case of Kappa paradox) were equal or above the required value of 0.60, except for the question D5.1, that was edited to increase clarity.

Regarding acceptability, only 10 (1.7%) respondents suggested improvement to the questionnaire wording, with all languages of the questionnaire available for validation receiving only one comment each, except for Swedish (four comments) and Norwegian (three comments).

Online supplemental tables 7–13 present the final English version of IMAgiNE questionnaire for health workers in English, Italian, Portuguese, Norwegian, Swedish, Croatian and Romanian.

Discussion

Collecting the perspectives of health workers providing care to mothers and newborns during facility-based childbirth is essential for improving several aspects of the quality of care, in particular during challenging situations like the COVID-19 pandemic. This paper presents the results of the development and validation of a WHO Standards23-based online questionnaire on the QMNC in the WHO European Region, from the perspective of health workers. To our knowledge, no other similar online tool, explicitly based on the WHO Maternal and Newborn Quality of Care Standards,23 has been developed for health workers. This questionnaire complements an existing WHO Standards23-based questionnaire dedicated to collect service users’ (mothers’) perspectives on the QMNC.34 The availability of unified comprehensive approaches to measure QMNC as defined by the WHO Quality Measures, through validated tools, allows comparisons of data across settings and over time, allows triangulation with routinely collected official data and may support decision makers on designing and implementing future quality improvement initiatives that might improve health outcomes.

Findings suggest that the questionnaire has good content and construct validity, face validity, internal consistency, intrarater reliability and acceptability in several countries of the WHO European Region. These relevant psychometric properties of the tool allow its utilisation in similar settings. Even though the small sample size by country/language did not allow to perform the exploratory and confirmatory factor analysis, useful to evaluate the underlying structure among variables,73 the cross-cultural careful planning and comprehensive methodological approaches74 used for this study ensure the strength of the validation process. Further results will be reported separately in coming publications.

The process of developing this questionnaire was based on existing guidance32–37 and had several strengths. The questionnaire was based on previous pilot studies.29–33 75 The characteristics of the questionnaire were defined in advance, based on previous experience developing measurement tools.29–33 74 Both international experts and health workers of different nationalities and with different backgrounds were involved in the development process at different phases, including the assessment of content and construct validity, face validity, internal consistency, intrarater reliability and acceptability.74 Other questionnaires recently used for collecting multicountry health workers’ perspectives during COVID-19 pandemic did not go through a similar formal validation process.13 76 77 As a lesson learnt from this experience, we acknowledge that the process of validation can be quite lengthy, and, and may not be the most rapid in a pandemic.

The number of Quality Measures collected by the tool (40 Quality Measures) may be seen as a limitation; however, this questionnaire should be seen as complementary to an already existing tool investigating maternal perspectives on the QMNC, also including 40 WHO Standards-based Quality Measures.30 When developing questionnaires, consideration has to be given to the length of the tool, not to decrease acceptability and to assure feasibility.35 41–43 During the COVID19-pandemic, health workers have seen an increase in their workload and an increase in requests to participate in many different surveys, thus critical attention should be given to avoid lengthy surveys, which may result in a low response rate.

Another potential limitation of the questionnaire is that it only collects data on the QMNC from the health worker’s perspectives. Health workers may not fully be aware of their institutions’ policies and/or personal attitudes might have influenced answers. However, the fact that only health workers directly involved in maternal or neonatal care for at least 1 year should participate in the validation process should have minimised this risk. Thus, we suggest to collect data from health professionals with a minimum experience of 1 year of clinical work.

In projects aiming at changing behaviours and improving quality of care, gathering information about opinions and view of key actors is essential.78 79 Opinions of both service users and service providers should not be dismissed. To get a fuller picture, data should ideally be collected, if feasible, from multiple data sources, including service users, service providers, from official data sources and from direct observation.31 75 78 79

The QMNC index is intended as a complementary (not substitutive) way to quantitatively measure QMNC in a synthetic format and should always be interpreted looking at detailed results of the whole list of Quality Measures collected. Responsiveness and other properties of the QMNC index shall be further evaluated and published in future studies.

Both the maternal and the health workers’ questionnaires will be used among partners of the IMAgiNE study networks, and research findings from individual countries or specific subgroup analysis (eg, data health professionals in the maternal area) will be reported in future publications. With this multicountry survey we have the possibility to explore a variety of local practices during the different phases of the COVID-19 pandemic, and to identify relevant influencing factors on the quality of care provided around childbirth (ie, healthcare policies, etc). This data may allow for domains relevant to QMNC over time and across countries comparison.

The ultimate objective of the tool described in this paper is to help stakeholders, department directors and policy-makers understand at a glance what works well and what needs to be changed or improved in the health facilities where women give birth, and babies are born, to ensure the QMNC. Future research shall further explore how better use the findings from this questionnaire across different settings and which can be the most effective strategies for translating quality of care evidence into policies in the best interest of mothers, newborns and health workers.

Conclusions

Findings suggest that the online health workers’ IMAgiNE questionnaire, based on WHO Standards, has good content, construct validity, face validity, internal consistency, intrarater reliability and acceptability in several countries of the WHO European Region. Further research may explore in depth the use of this questionnaire in other countries, documenting the responsiveness of the QMNC index, and test approaches for translating data generated into quality improvement policies across settings.