Assessing the quality of care for children attending health facilities: a systematic review of assessment tools.

INTRODUCTION: Assessing quality of healthcare is integral in determining progress towards equitable health outcomes worldwide. Using the WHO 'Standards for improving quality of care for children and young adolescents in health facilities' as a reference standard, we aimed to evaluate existing tools that assess quality of care for children.
METHODS: We undertook a systematic literature review of publications/reports between 2008 and 2020 that reported use of quality of care assessment tools for children (<15 years) in health facilities. Identified tools were reviewed against the 40 quality statements and 510 quality measures from the WHO Standards to determine the extent each tool was consistent with the WHO Standards. The protocol was registered in PROSPERO ID: CRD42020175652.
RESULTS: Nine assessment tools met inclusion criteria. Two hospital care tools developed by WHO-Europe and WHO-South-East Asia Offices had the most consistency with the WHO Standards, assessing 291 (57·1%) and 208 (40·8%) of the 510 quality measures, respectively. Remaining tools included between 33 (6·5%) and 206 (40·4%) of the 510 quality measures. The WHO-Europe tool was the only tool to assess all 40 quality statements. The most common quality measures absent were related to experience of care, particularly provision of educational, emotional and psychosocial support to children and families, and fulfilment of children's rights during care.
CONCLUSION: Quality of care assessment tools for children in health facilities are missing some key elements highlighted by the WHO Standards. The WHO Standards are, however, extensive and applying all the quality measures in every setting may not be feasible. A consensus of key indicators to monitor the WHO Standards is required. Existing tools could be modified to include priority indicators to strengthen progress reporting towards delivering quality health services for children. In doing so, a balance between comprehensiveness and practical utility is needed. PROSPERO REGISTRATION NUMBER: CRD42020175652. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

There are no universally agreed indicators to assess quality of health care.

Previous reviews on quality of health care for children in low-income and middle-income countries (LMICs) tend to concentrate on system input measures such as physical infrastructure, availability of essential medicines, equipment and human resources.

There has been no systematic review of existing assessment tools for quality of health care for children in health facilities.

This is the first systematic review to compare existing quality of care assessment tools against the WHO ‘Standards for improving the quality of care for children and young adolescents in health facilities’, and found that they do not adequately assess the WHO Standards in its current format.

Most assessment tools were more comprehensive in assessing provision of care and available human and physical resources, but deficient in assessing experience of care.

Most assessment tools focused more on input and process measures than outcome measures.

There is no existing assessment tool that can comprehensively assess all the indicators in the ‘WHO Standards’, however, the indicators are extensive and may not be feasible for LMICs to comprehensively assess.

Future endeavours should focus on identifying and obtaining consensus on a selection of key indicators in the assessment of quality of health care for children in health facilities. Harmonisation of key indicators embedded within existing assessment tools will enable regular monitoring and comparable data in order to report progress in the quality of health care for children at local and national levels.

Introduction

Ending preventable child deaths by 2030 is a major focus for the Sustainable Development Goals (SDGs).1 A crucial factor to achieving this is Universal Health Coverage (UHC) which ensures that all people, including children, have access to quality essential healthcare services without being pushed into financial hardship. Quality healthcare is defined by WHO as health services which are ‘effective, efficient, accessible, patient centred, equitable and safe’.2 To further reduce child deaths, many countries will need to find ways to increase UHC with quality healthcare for children.

Determining progress in quality of healthcare delivery for children requires monitoring and tracking of measurable indicators. However, there are no universally agreed indicators for quality of care (QoC). To better understand the complex multidimensional nature of quality healthcare, WHO developed a framework to identify domains to assess, improve and monitor the quality of paediatric care in health facilities, an extension of the earlier framework for improving maternal and newborn care in health facilities.3 4 The framework encompasses three broad categories of QoC: (A) provision of care—evidence-based practices, effective information systems and referral pathways; (B) experience of care—effective communication, recognition of child rights and appropriate emotional and psychological support; and (C) available human and physical resources to meet the best interests of children. The broad categories are subdivided into eight domains to provide a structured approach when addressing QoC at all levels of the health system (online supplemental appendix A). These eight domains reflect the eight quality standards (QSd) in the WHO ‘Standards for improving the QoC for children and young adolescents in health facilities’ which are further detailed in 40 priority statements and 510 measurable indicators.4 The WHO Standards can, therefore, be used as a standard point of reference when assessing QoC for children in a healthcare facility.

Historically, various tools have been developed to assess QoC for children. We sought to understand if these tools adequately assess all aspects of QoC as outlined by the WHO Standards for children and young adolescents. A recent review identified and compared five existing assessment tools to the WHO ‘Standards for improving quality of maternal and newborn care in health facilities’.5 6 The percentage of indicators outlined in the WHO Standards that the five tools were able to assess ranged from 12% to 62%.6 There has been no systematic review of existing assessment tools for QoC for children <15 years. There is an urgent need to better understand the capacity of readily available tools to assess the QoC for children, in order to meet the SDG targets for child health.

The aim of this systematic literature review is to identify existing tools used to assess QoC for children and young adolescents in health facilities and assess the extent to which they represent the domains in the WHO QSd.

Methods

Search strategy and selection criteria

A systematic review of the literature was undertaken in August 2020 using Preferred Reporting Items for Systematic Reviews and Meta-Analyses reporting guidelines, to identify assessment tools available globally that evaluate QoC for children attending health facilities.7 MEDLINE (Ovid) database was searched using Medical Subject Heading terms and/or keywords. PubMed was searched using keywords, to retrieve items not indexed on MEDLINE. The PubMed search strategy was adapted for use in Global Health (Commonwealth Agricultural Bureaux direct) database and the International Journal for Quality in Health Care. Additional peer-reviewed publications were identified through handsearching of reference lists of key articles. Grey literature was identified by conducting a keyword search using the World Bank and WHO library databases. The search strategies and results yielded are available from online supplemental appendix B.

The inclusion criteria for eligibility included publications/reports that: reported the use of an assessment tool to evaluate QoC in a primary, secondary or tertiary level healthcare facility. The assessment tool was deemed eligible if used by more than one country; and included at least one module/component evaluating QoC in children. A child was defined as aged 0–14 years, to align with the definition of ‘birth up to 15 years’ used in the WHO Standards. To identify tools more likely to be in current use and available globally, the search period was limited to ten years preceding the publication of the WHO Standards to present time (2008–2020), and to those published in English language. Exclusion criteria included publications/reports of assessment tools that: evaluated only newborns (<1 month old) or only adolescents (10–19 years old); were developed only for research purposes; evaluated QoC for a specific disease; or a niche component of QoC (eg, antimicrobial prescribing practice). For any assessment tools not publicly available, authors and/or the original developers of the tool were contacted.

The screening process was performed independently by two reviewers using Covidence systematic review software.8 Titles and abstracts were screened and excluded if inclusion/exclusion criteria were not met. Full texts of remaining publications/reports were assessed for eligibility. The assessment tools from full texts were retrieved and further assessed to ensure that they met eligibility criteria. The assessment tools that were identified in multiple reports were grouped together as one unique tool for analysis. Conflicts in determining whether an article/assessment tool met eligibility criteria were resolved by discussion between the two reviewers. If consensus was not reached, a third reviewer was consulted.

Data analysis

Each quality assessment tool included was compared against the WHO ‘Standards for improving QoC for children and young adolescents in health facilities’.4 The WHO Standards comprises of eight overarching ‘QSd’, each one correlating to one domain of the framework for improving quality of paediatric care. Each QSd is composed of priorities or ‘Quality Statements (QSt)’ (total of 40) for improving QoC for children. The QSt further subdivided into 510 ‘Quality Measures (QM)’, comprised of 235 input, 169 process/output and 106 outcome measures (figure 1). A full list of the QSd, QSt and QM are listed in online supplemental appendix C.

Figure 1

Structure of the WHO ‘Standards for improving the quality of care for children and youngadolescents in health facilities’.

For each assessment tool, we used the most updated version in English in its generic format. Each tool was composed of various modules to assess quality (eg, direct clinical observations, health worker interviews, inventory checklists). We evaluated all modules and excluded those not relevant to our review (eg, antenatal care services). For the remaining modules, we reviewed each question/statement and matched them if applicable to the relevant QM in the WHO Standards. Two paediatricians (AQ and ST) performed the matching process independently to decrease the risk of bias. Any conflicts were discussed between the two reviewers and a third reviewer was consulted for any unresolved conflicts.

We used a similar scoring system as the review of facility assessment tools on maternal and newborn QoC in health facilities performed by Brizuela et al.6 A question/statement from the tool was considered a match to a WHO QM if any component of the QM was included. If a WHO QM was matched, a score of 1 was allocated. Although multiple questions could match the same QM, each QM could only score a maximum of 1. For a QM that consisted of more than one subcomponent, a question/statement from the assessment tool only had to fulfil one subcomponent to be considered a match. For example, QM 1.1.1: ‘health facility maintains an up-to-date 24 hours staff duty roster, with a functioning contact mechanism for finding additional support, which ensures that staff responsible for paediatric triage are available at all times’; would be matched by a question asking if the health facility has a 24-hour staff roster.4 Conversely, a single question/statement could also be matched to more than one QM. For example, an assessment tool with a checklist of available antibiotics in the health facility would match the WHO QM detailing adequate supplies of antibiotics to treat pneumonia (QM 1.3.3), sepsis (QM 1.5.4), neonatal infections (QM 1.2.2); while also matching the QM detailing adequate stocks of essential medicines (QM 8.4.4).

Descriptive statistics were used to calculate the percentage of matched QM, QSt and QSd for each assessment tool. The assessment tools were ranked according to the total percentage of WHO QM assessable. Assessment tools were also categorised according to whether they were able to completely assess (100%), partially assess (1%–49% and 50%–99%), or not assess (0%) any of the QSd and QSt.

Patient and public involvement

No patients or public were involved in this study.

Results

The search strategy identified 1180 publications/reports, after duplicates were removed (figure 2). The screening process excluded 1035 publications/reports. The remaining 145 full-text manuscripts were assessed for eligibility, with 39 publications/reports being deemed eligible. These publications/reports were further evaluated to collate duplications of assessment tools, with 10 unique tools identified as eligible. One tool was not accessible from the author, leaving nine unique assessment tools for analysis (figure 2).

Figure 2

PRISMA flow diagram for the selection of assessment tools. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

Table 1 summarises the nine assessment tools and the modules that were evaluated as part of our analysis. All tools were developed for use in low-income and middle-income countries (LMICs), except for the Child Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS).9 All tools were available in English, but could be adapted/translated to local context. All tools were structured questionnaires/interviews with checklist style questions, but varied in length and composition of modules. The shortest tool was the HCAHPS with 62 multichoice questions. Others such as the Service Provision Assessment (SPA) and the WHO Hospital Care assessment tools were detailed with over 100 pages, had multiple modules, with over 100 questions/checklist items per module.10–12

Table 1

Summary of assessment tools reviewed for quality of care for children in health facilities

Tool	Development/source	Type of health facility tool designed for	Population and services tool designed for	Key topics assessed by tool	Countries that have used tool*	Version assessed	Modules included in assessment
Service Provision Assessment10	USAID - MEASURE Evaluation	All health facility types	All agesFamily planning, maternal and child health services, communicable and non-communicable diseases, basic surgery	Availability of health services Services readiness of health facilities Extent services follow accepted standards of care Satisfaction levels of clients and service providers	Afghanistan, Bangladesh, Congo DR, Egypt, Ethiopia, Ghana, Guatemala, Guyana, Haiti, Kenya, Malawi, Namibia, Nepal, Rwanda, Senegal, Tanzania, Uganda, Zambia	2012	Inventory Clinical observation—sick child Exit interview—caretaker of sick child Health worker interview
Rapid Health Facility Assessment32	USAID-MEASURE Evaluation	Primary healthcare facilities	Women and childrenMaternal, neonatal and child health services	Assessment, diagnosis and treatment of common childhood illnesses Service readiness of health facilities Quality of management processes in health facilities	Nigeria, South Sudan	2008	Clinical observation—sick child Exit interview—caretaker of sick child Health worker interview and record review Health facility checklist
Service Availability and Readiness Assessment19	WHO	All health facility types	All agesMaternal and child health services, communicable and non-communicable diseases, surgical care	Service availability-infrastructure, core health personnel, service utilisation General service readiness-basic equipment, amenities, infection prevention, diagnostic capacity, essential medicines Service-specific readiness	Bangladesh, Benin, Burkina Faso, Cambodia, Congo DR, Cote D’Ivoire, Djibouti, Ghana, Haiti, India, Kenya, Lao PDR, Libya Madagascar, Mauritania, Myanmar, Namibia, Niger, Rwanda, Senegal, Sierra Leone, Somalia, South Sudan Tanzania, Togo, Uganda Vietnam, Zambia, Zimbabwe	2015	Staffing Infrastructure Available services Diagnostics Medicines and commodities
Health Facility Survey-using Integrated Management Childhood Illness as clinical guidelines33	WHO	Primary level outpatient health facilities	All childrenOutpatient care for sick children	Quality of case management received by sick children Quality of counselling of caretakers Availability of key heath system supports-edicines, equipment, supervision, referral system Barriers to effective integrated case management	Botswana, Cambodia, Guatemala, Kenya, Zambia, Afghanistan, Rwanda, China, Malawi, Mongolia, Nigeria, Papua New Guinea, Philippines, Vietnam, Kiribati, Lao PDR, Solomon Islands	2003	Clinical observation—sick child Exit interview—caretaker of sick child Re-examination of sick child Equipment and supply checklist
Assessment Tool for Hospital Care (WHO-SE Asia Office)12	WHO-SE Asia Office	All hospitals	Women and childrenMaternal, neonatal and child health services	Service availability and readiness-infrastructure, hospital support systems, staffing, essential medicines and diagnostics Quality of medical care provided for maternal, neonatal and child health services Satisfaction levels of clients and service providers	Indonesia, Nepal	2016	General hospital information Paediatric care Caregiver interview Health worker interview Postal questionnaire
Hospital care for children: quality assessment and improvement tool (WHO-Europe Office)11	WHO-Europe Office	All hospitals	All children, excluding newbornsPaediatric services, excluding neonatal healthcare	Service availability and readiness-infrastructure, hospital support systems, health management information systems, staffing, essential medicines and equipment Quality of medical care provided for common childhood illnesses and emergency presentations Hospital polices and guidelines-infection prevention, training, access, patient’s rights Satisfaction levels of clients and service providers	Kyrgystan, Tajikistan, Ethiopia, Angola, Malawi, Mozambique	2015	Hospital support services Case management Policies and organisation of services Interviews
Health Resources Availability Mapping System34	WHO	Humanitarian and Emergency Response settings	All agesEssential trauma care, child health and nutrition, communicable diseases, mental health, sexual and reproductive health	Availability of health facilities and services Resources for service delivery-infrastructure, human resources, communications, infection control Reasons for gaps in service availability	Sudan, Uganda	2017	Hospitals assessment tool Health centres assessment tool
Health Results Based Financing impact evaluation toolkit35	World Bank	All healthcare facilities(Programme evaluating the impact of health-related results-based financing incentives)	All ages with focus on women and childrenMaternal, neonatal and child health, communicable diseases	Quantity of health services delivered Quality of health services provided Health outcomes Resource management at health facilities Impact on non-results based financing services delivered Disaggregation of impact by provider and population characteristics	Tajikistan, Burkina Faso, Congo DR, Rwanda, Benin, Lesotho, Kyrgyzstan, Cameroon, Central African Republic, The Gambia,	2012	Health facility assessment Health worker individual Exit interview-caregiver for child under 5
Hospital Consumer Assessment of Healthcare Providers and Systems-Child Version9	Boston Children’s Hospital	All hospitals	All childrenAll paediatric inpatient health services	Quality of communication with caregivers and patients Attention to safety and comfort Hospital environment	Belgium, Canada, USA, Argentina	2014	Entire survey (62 items)

*Only includes countries with reports/data publicly available.

Table 2 summarises the percentages of WHO QM within each QSd assessable by each tool. Overall, QM related to the domains of provision of care and available human and physical resources were more widely assessable than experience of care. QSd 1: ‘Every child receives evidence-based care and management of illness according to WHO guidelines’ was most comprehensively assessable across all tools, apart from the Child HCAHPS which did not assess it at all.4 9 QSd 6: ‘All children and their families are provided with educational, emotional and psychosocial support that is sensitive to their needs and strengthens their capability’ was absent from five of the nine included tools.4 All assessment tools were able to partially assess QSd 7 (staff availability) and QSd 8 (health facility physical infrastructure, waste management, supplies and equipment).

Table 2

Percentage of WHO Quality Standards assessable by each quality assessment tool

	Percentage of Quality Measures assessable for each Quality Standard
	WHO-Europe	WHO-SE Asia	SPA	HRBF	HeRAMS	r-HFA	SARA	HFS-IMCI	HCAHPS-Child
Standard 1: Every child receives evidence-based care and management of illness according to WHO guidelines. (n=222)	59%	55%	49%	27%	39%	24%	23%	18%	0%
Standard 2: The health information system ensures the collection, analysis and use of data to ensure early, appropriate action to improve the care of every child. (n=29)	52%	52%	48%	38%	0%	24%	0%	10%	10%
Standard 3: Every child with condition(s) that cannot be treated effectively with the available resources receives appropriate, timely referral, with seamless continuity of care. (n=28)	39%	18%	36%	39%	43%	7%	7%	18%	0%
Standard 4: Communication with children and their families is effective, with meaningful participation and responds to their needs and preferences. (n=45)	53%	22%	29%	24%	9%	7%	0%	16%	27%
Standard 5: Every child’s rights are respected, protected and fulfilled at all times during care, without discrimination. (n=44)	45%	11%	20%	14%	7%	5%	5%	2%	9%
Standard 6: All children and their families are provided with educational, emotional and psychosocial support that is sensitive to their needs and strengthens their capability. (n=32)	66%	9%	3%	0%	0%	0%	0%	0%	19%
Standard 7: For every child, competent, motivated, empathic staff are consistently available to provide routine care and management of common childhood illnesses. (n=44)	70%	34%	52%	59%	2%	25%	7%	16%	7%
Standard 8: The health facility has an appropriate, child friendly physical environment, with adequate water, sanitation, waste management, energy supply, medicines, medical supplies and equipment for routine care and management of common childhood illnesses. (n=66)	56%	50%	41%	39%	35%	20%	32%	3%	8%

Red boxes indicate that the assessment tool did not assess any; orange boxes indicate less than half and yellow boxes indicate equal to or more than half of the quality measures were assessable in the associated Quality Standard.

HCAHPS, Hospital Consumer Assessment of Healthcare Providers and Systems; HeRAMS, Health Resources Availability Mapping System; HFS-IMCI, Health Facility Survey—using Integrated Management of Childhood Illness clinical guidelines; HRBF, Health Results Based Financing impact evaluation toolkit; n, number of Quality Measures within the associated Quality Standard; r-HFA, rapid Health Facility Assessment; SARA, Service Availability and Readiness Assessment; SPA, Service Provision Assessment.;

Overall, the tools were more comprehensive at assessing the input QM (median 32·8%, IQR (16·4%–45·3%)) and process/output QM (median 21·3%, IQR (12·4–41·7%)) compared with the outcome QM (median 16·0%, IQR (3·8%–28·3%)). Table 3 summarises the proportion of QSt that had at least one each of input, process and outcome QM within each QSd. No single tool was able to fulfil this for all 40 WHO QSt. The percentages of QM that was assessable within each of the 40 QSt are summarised in figure 3. The WHO (Europe regional office) ‘Hospital care for children: quality assessment and improvement tool’ was the only tool able to partially assess every QSt. Of the remaining tools, 5–27 of 40 QSt were not assessable at all. The SPA and the Health Resources Available Mapping System (HeRAMS) were the only tools able to assess 100% of any one QSt. The HeRAMS, however, failed to assess any QM of 15 other QSt.

Table 3

Proportion of WHO Quality Statements with at least one each of input, process and output Quality Measure assessable by each quality assessment tool

	Percentage of Quality Measures assessable for each Quality Standard
	WHO-Europe	WHO-SE Asia	SPA	HRBF	HeRAMS	r-HFA	SARA	HFS-IMCI	HCAHPS-Child
Standard 1: Every child receives evidence-based care and management of illness according to WHO guidelines. (QSt=15)	7/15	7/15	6/15	2/15	1/15	0/15	0/15	1/15	0/15
Standard 2: The health information system ensures the collection, analysis and use of data to ensure early, appropriate action to improve the care of every child. (QSt=3)	2/3	3/3	2/3	2/3	0/3	1/3	0/3	0/3	0/3
Standard 3: Every child with condition(s) that cannot be treated effectively with the available resources receives appropriate, timely referral, with seamless continuity of care. (QSt=3)	1/3	0/3	1/3	2/3	1/3	0/3	0/3	0/3	0/3
Standard 4: Communication with children and their families is effective, with meaningful participation, and responds to their needs and preferences (QSt=4)	2/4	1/4	1/4	2/4	0/4	0/4	0/4	0/4	1/4
Standard 5: Every child’s rights are respected, protected and fulfilled at all times during care, without discrimination. (QSt=5)	2/5	0/5	1/5	0/5	0/5	0/5	0/5	0/5	0/5
Standard 6: All children and their families are provided with educational, emotional and psychosocial support that is sensitive to their needs and strengthens their capability. (QSt=3)	3/3	0/3	0/3	0/3	0/3	0/3	0/3	0/3	0/3
Standard 7: For every child, competent, motivated, empathic staff are consistently available to provide routine care and management of common childhood illnesses. (QSt=3)	3/3	2/3	2/3	2/3	0/3	1/3	0/3	1/3	0/3
Standard 8: The health facility has an appropriate, child friendly physical environment, with adequate water, sanitation, waste management, energy supply, medicines, medical supplies and equipment for routine care and management of common childhood illnesses. (QSt=4)	1/4	1/4	2/4	2/4	0/4	1/4	1/4	0/4	0/4
Total Quality Statements=40	21/40	14/40	15/40	12/40	2/40	3/40	1/40	2/40	1/40

Red boxes indicate that the assessment tool did not include any; orange boxes indicate less than half; yellow boxes indicate equal to or more than half and green boxes indicate that all of the Quality Statements within the associated Quality Standard had at least one each of input, process and outcome Quality Measure assessable.

HCAHPS, Hospital Consumer Assessment of Healthcare Providers and Systems; HeRAMS, Health Resources Availability Mapping System; HFS-IMCI, Health Facility Survey—using Integrated Management of Childhood Illness clinical guidelines; HRBF, Health Results Based Financing impact evaluation toolkit; QSt, number of Quality Statements within the associated Quality Standard; r-HFA, rapid Health Facility Assessment; SARA, Service Availability and Readiness Assessment; SPA, Service Provision Assessment.;

Figure 3

Percentage of WHO Quality Statements* assessable by each quality assessment tool. *‘Quality Statements’ are 40 concise statements of the priorities for improving quality of care for children as documented in the WHO Standards. Each quality statement contains from 6 to 22 quality measures.4 Not assessable = the assessment tool did not assess any quality measures in the quality statement. Partially assessable = the assessment tool assessed at least one of the quality measures in the quality statement. Completely assessable = the assessment tool assessed all of the quality measures in the quality statement. HRBF, Health Results Based Financing impact evaluation toolkit; HeRAMS, Health Resources Availability Mapping System; HFS-IMCI, Health Facility Survey—using Integrated Management of Childhood Illness clinical guidelines; HCAHPS, Hospital Consumer Assessment of Healthcare Providers and Systems; r-HFA, rapid Health Facility Assessment; SPA, Service Provision Assessment. SARA, Service Availability and Readiness Assessment.

Figure 4 shows the overall percentage of QM assessable by each tool. The WHO-Europe tool was the most comprehensive with 291 (57·1%) of the 510 WHO QM being assessable.11 The remaining tools varied from 6·5% to 40·8% in their capacities to assess the QM. Table 4 shows the percentage of assessable QM in each QSt. Most tools assessed less than half the QM in any single QSt. The WHO-Europe tool and the WHO-SE Asia were the most comprehensive, able to assess more than half the QM in 23 and 13 of the 40 QSt, respectively. The HCAHPS-Child tool had the largest number of gaps, leaving 27 QSt completely unassessed, but was able to assess 9 (69%) of the 13 QM for QSt 4.1, that is, effective communication given to children and their carers, which is a key objective of the tool.

Figure 4

Overall percentage of WHO Quality Measures* assessable by each quality assessment tool. HRBF, Health Results Based Financing impact evaluation toolkit; HeRAMS, Health Resources Availability Mapping System; HFS-IMCI, Health Facility Survey—using Integrated Management of Childhood Illness clinical guidelines; HCAHPS, Hospital Consumer Assessment of Healthcare Providers and Systems; r-HFA, rapid Health Facility Assessment; SPA, Service Provision Assessment; SARA, Service Availability and Readiness Assessment.

Table 4

Percentage of WHO Quality Statements assessable by each quality assessment tool

	Percentage of Quality Measures assessable in each Quality Statement
	WHO-Europe	WHO-SE Asia	SPA	HRBF	HeRAMS	r-HFA	SARA	HFS-IMCI	HCAHPS-Child
Standard 1: Every child receives evidence-based care and management of illness according to WHO guidelines
Statement 1.1 All children are triaged and promptly assessed for emergency and priority signs to determine whether they require resuscitation and receive appropriate care according to WHO guidelines. (n=19)	63%	58%	68%	16%	63%	26%	16%	16%	0%
Statement 1.2 All sick infants, especially small newborns, are thoroughly assessed for serious bacterial infection and receive appropriate care according to WHO guidelines (n=19)	37%	79%	42%	11%	11%	16%	16%	11%	0%
Statement 1.3 All children with cough or difficult breathing are correctly assessed, classified and investigated and receive appropriate care and/or antibiotics for pneumonia, according to WHO guidelines. (n=18)	83%	83%	61%	17%	61%	39%	28%	33%	0%
Statement 1.4 All children with diarrhoea are correctly assessed and classified and receive appropriate rehydration and care, including continued feeding, according to WHO guidelines. (n=19)	74%	68%	68%	32%	74%	53%	16%	42%	0%
Statement 1.5 All children with fever are correctly assessed, classified and investigated and receive appropriate care according to WHO guidelines.(n=19)	58%	68%	58%	37%	47%	42%	32%	26%	0%
Statement 1.6 All infants and young children are assessed for growth, breastfeeding and nutrition, and their carers receive appropriate support and counselling, according to WHO guidelines. (n=12)	83%	75%	50%	33%	17%	25%	17%	25%	0%
Statement 1.7 All children at risk for acute malnutrition and anaemia are correctly assessed and classified and receive appropriate care according to WHO guidelines. (n=16)	75%	88%	44%	38%	56%	25%	31%	25%	0%
Statement 1.8 All children at risk for TB and/or HIV infection are correctly assessed and investigated and receive appropriate management according to WHO guidelines. (n=17)	47%	29%	47%	65%	41%	18%	41%	0%	0%
Statement 1.9 All children are assessed and checked for immunisation status and receive appropriate vaccinations according to the guidelines of the WHO expanded programme on immunisation. (n=12)	17%	58%	100%	75%	42%	50%	58%	58%	0%
Statement 1.10: All children with chronic conditions receive appropriate care, and they and their families are sufficiently informed about their condition(s) and are supported to optimise their health, development and quality of life. (n=12)	42%	0%	33%	8%	17%	0%	25%	0%	0%
Statement 1.11 All children are screened for evidence of maltreatment, including neglect and violence, and receive appropriate care. (n=7)	29%	0%	0%	0%	29%	0%	0%	0%	0%
Statement 1.12: All children with surgical conditions are screened for surgical emergencies and injuries and receive appropriate surgical care. (n=12)	33%	17%	17%	8%	25%	8%	17%	0%	0%
Statement 1.13: All sick children, especially those who are most seriously ill, are adequately monitored, reassessed periodically and receive supportive care according to WHO guidelines. (n=14)	93%	50%	36%	0%	21%	7%	7%	0%	0%
Statement 1.14: All children receive care with standard precautions to prevent health care-associated infections. (n=14)	64%	50%	50%	57%	43%	21%	36%	7%	0%
Statement 1.15: All children are protected from unnecessary or harmful practices during their care. (n=12)	67%	33%	17%	0%	0%	0%	0%	8%	0%
Standard 2: The health information system ensures the collection, analysis and use of data to ensure early, appropriate action to improve the care of every child
Statement 2.1: Every child has a complete, accurate, standardised, up-to-date medical record, which is accessible throughout their care, on discharge and on follow-up. (n=13)	62%	38%	23%	15%	0%	23%	0%	8%	0%
Statement 2.2: Every health facility has a functional mechanism for data collection, analysis and use as part of its activities for monitoring performance and quality improvement. (n=9)	44%	67%	78%	56%	0%	44%	0%	0%	0%
Statement 2.3: Every health facility has a mechanism for collecting, analysing and providing feedback on the services provided and the perception of children and their families of the care received. (n=7)	43%	57%	57%	57%	0%	0%	0%	29%	43%
Standard 3: Every child with condition(s) that cannot be treated effectively with the available resources receives appropriate, timely referral, with seamless continuity of care
Statement 3.1: Every child who requires referral receives appropriate prereferral care, and the decision to refer is made without delay. (n=10)	30%	30%	30%	50%	40%	0%	0%	20%	0%
Statement 3.2: Every child who requires referral receives seamless, coordinated care and referral according to a plan that ensures timeliness. (n=12)	33%	0%	17%	42%	17%	8%	8%	25%	0%
Statement 3.3: For every child referred or counter-referred within or among health facilities, there is appropriate information exchange and feedback to relevant healthcare staff. (n=6)	67%	33%	83%	17%	100%	17%	17%	0%	0%
Standard 4: Communication with children and their families is effective, with meaningful participation and responds to their needs and preferences
Statement 4.1 All children and their carers are given information about the child’s illness and care effectively, so that they understand and cope with the condition and the necessary treatment. (n=13)	85%	23%	38%	31%	0%	15%	0%	23%	69%
Statement 4.2 All children and their carers experience coordinated care, with clear, accurate information exchange among relevant health and social care professionals and other staff. (n=9)	22%	11%	11%	11%	0%	0%	0%	0%	0%
Statement 4.3 All children and their carers are enabled to participate actively in the child’s care, in decision making, in exercising the right to informed consent and in making choices, in accordance with their evolving capacity. (n=10)	80%	10%	10%	20%	0%	0%	0%	0%	20%
Statement 4.4 All children and their carers receive appropriate counselling and health education, according to their capacity, about the current illness and promotion of the child’s health and well-being. (n=13)	23%	38%	46%	31%	31%	8%	0%	31%	8%
Standard 5: Every child’s rights are respected, protected and fulfilled at all times during care, without discrimination
Statement 5.1 All children have the right to access healthcare and services, with no discrimination of any kind. (n=9)	78%	11%	44%	22%	11%	0%	0%	11%	0%
Statement 5.2 All children and their carers are made aware of and given information about children’s rights to health and healthcare. (n=9)	44%	0%	0%	0%	0%	0%	0%	0%	0%
Statement 5.3 All children and their carers are treated with respect and dignity, and their right to privacy and confidentiality is respected. (n=7)	57%	14%	43%	43%	0%	14%	14%	0%	57%
Statement 5.4 All children are protected from any violation of their human rights, physical or mental violence, injury, abuse, neglect or any other form of maltreatment. (n=9)	11%	0%	0%	0%	22%	0%	0%	0%	0%
Statement 5.5 All children have access to safe, adequate nutrition that is appropriate for both their age and their health condition during their care in a facility. (n=10)	40%	30%	20%	10%	0%	10%	10%	0%	0%
Standard 6: All children and their families are provided with educational, emotional and psychosocial support that is sensitive to their needs and strengthens their capability
Statement 6.1 All children are allowed to be with their carers, and the role of carers is recognised and supported at all times during care, including rooming-in during the child’s hospitalisation. (n=9)	78%	22%	0%	0%	0%	0%	0%	0%	11%
Statement 6.2 All children and their families are given emotional support that is sensitive to their needs, with opportunities for play and learning that stimulate and strengthen their capability. (n=10)	60%	10%	0%	0%	0%	0%	0%	0%	20%
Statement 6.3 Every child is assessed routinely for pain or symptoms of distress and receives appropriate management according to WHO guidelines.(n=13)	62%	0%	8%	0%	0%	0%	0%	0%	23%
Standard 7: For every child, competent, motivated, empathic staff are consistently available to provide routine care and management of common childhood illnesses
Statement 7.1 All children and their families have access at all times to sufficient health professionals and support staff for routine care and management of childhood illnesses. (n=15)	67%	27%	40%	53%	0%	13%	0%	13%	7%
Statement 7.2 Health professionals and support staff have the appropriate skills to fulfil the health, psychological, developmental, communication and cultural needs of children. (n=17)	76%	47%	65%	71%	6%	35%	18%	24%	6%
Statement 7.3 Every health facility has managerial leadership that collectively develops, implements and monitors appropriate policies and legal entitlements that foster an environment for continuous quality improvement. (n=12)	67%	25%	50%	50%	0%	17%	0%	8%	8%
Standard 8: The health facility has an appropriate, child friendly physical environment, with adequate water, sanitation, waste management, energy supply, medicines, medical supplies and equipment for routine care and management of common childhood illnesses.
Statement 8.1 Children are cared for in a well-maintained, safe, secure physical environment with an adequate energy supply and which is appropriately designed, furnished and decorated to meet their needs, preferences and developmental age. (n=15)	40%	47%	40%	33%	20%	13%	33%	0%	20%
Statement 8.2 Child-friendly water, sanitation, hand hygiene and waste disposal facilities are easily accessible, functional, reliable, safe and sufficient to meet the needs of children, their carers and staff. (n=22)	68%	59%	41%	45%	41%	18%	18%	0%	0%
Statement 8.3 Child-friendly, age-appropriate equipment designed to meet children’s needs in medical care, learning, recreation and play are available at all times. (n=15)	40%	27%	27%	27%	33%	7%	33%	0%	13%
Statement 8.4 Adequate stocks of child-friendly medicines and medical supplies are available for the routine care and management of acute and chronic childhood illnesses and conditions. (n=14)	71%	64%	57%	50%	43%	43%	50%	14%	0%

Red boxes indicate that the assessment tool did not assess any; orange boxes indicate less than half; yellow boxes indicate equal to or more than half and green boxes indicate all of the quality measures were assessable in the associated quality statement.

HCAHPS, Hospital Consumer Assessment of Healthcare Providers and Systems; HeRAMS, Health Resources Availability Mapping System; HFS-IMCI, Health Facility Survey—using Integrated Management of Childhood Illness clinical guidelines; HRBF, Health Results Based Financing impact evaluation toolkit; n, number of Quality Measures within the associated Quality Statement; r-HFA, rapid Health Facility Assessment; SARA, Service Availability and Readiness Assessment; SPA, Service Provision Assessment.;

Discussion

This is the first systematic review to compare QoC assessment tools against the current WHO Standards for children and young adolescents in health facilities. Three of the nine assessment tools included questions from all eight of the WHO QSd, but only one (WHO-Europe Hospital Care assessment tool) was able to address all 40 QSt. Despite being the most comprehensive tool, the WHO-Europe Hospital Care assessment tool still only included about half of all QM. QSd that included evidence-based management, staffing and physical infrastructure and resources (QSd 1,7 and 8) were more widely covered across the tools than those which encompassed health information systems (HIS), referral processes, communication, psychosocial support and child rights (QSd 2–6).

Previous assessments of QoC in LMICs have often centred on input measures as these are seen to be more tangible, objective measures.6 13–15 This was reflected in this review with most tools assessing more input measures and less process or outcome measures, and no tool assessing at least one input, process and outcome measure in all 40 QSt. Mortality data were largely absent from almost all tools. The Donabedian quality framework describes the relationship between the three components on input/structure, process and outcomes.16 Although structural, input measures are important to healthcare delivery, the flow-on effects to processes such as appropriate care delivery and adequate communication, and outcome measures such as morbidity and mortality data, and patient satisfaction, determine how successful a health facility is. Including all three components is therefore crucial for assessing the level of QoC that is being provided for children in hospitals.

The WHO Standards for improving QoC for children and young adolescents were used in this review as a reference standard for assessing QoC for children attending health facilities. They are comprehensive and include all three components of the Donabedian quality framework, while aligning with the SDG emphasis on equity. They have been developed as a resource for healthcare professionals and managers at the health facility level through to government bodies and technical partners responsible for policy and programme development at the national level, to support quality improvement practices for children.4 How the WHO Standards are implemented in practice and what tools to use when assessing quality of healthcare for children, are to be decided within the local context. However, it is unlikely that a single tool could encompass all 510 QM and be feasible in most settings in LMICs.

The purpose and context of the assessment tools needs to be considered when examining how comprehensive they are in comparison to the WHO Standards. The WHO Hospital Care assessment tools were found to be more comprehensive in their ability to capture the WHO Standards. These tools were first developed in 2001 to provide government and stakeholders guidance in performing evaluation of quality of healthcare practices in order to identify key areas to improve on.17 They have since been revised multiple times and adapted to multiple settings with the most recent European version including indicators for child rights, communication and alignment with evidence-based practice as outlined in the WHO Pocket Book of Hospital Care for Children.18 The WHO Standards for improving QoC have been modelled on similar frameworks, which may explain the overlap of indicators between the WHO Hospital Care assessment tools and the WHO Standards.

Of the three broad arms of QoC in the WHO Standards: provision of care (QSd 1–3) and available human and physical resources (QSd 7–8), were more widely covered by the tools than experience of care (QSd 4–6). This may be because provision of care and system inputs have more definitive QM, which make them amenable to be assessed through checklist style questionnaires. The SPA and Service Availability and Readiness Assessment were developed to provide nationwide data on the capacities of health facilities to provide quality services and have been used in over 30 countries.10 19 These surveys are designed to be repeated at periodic intervals, every 1–5 years, to monitor progress and inform development of national health policies and programmes. It is therefore not surprising that their strengths lie in assessing availability of concrete measures such as physical infrastructure and human resources, with less emphasis on subjective components such as communication and emotional support. The HeRAMS similarly was developed to collect information on availability of health resources and services. Designed to be implemented in humanitarian and emergency response settings, rapid reporting is essential to its purpose in order to obtain supplies and resources required for basic healthcare needs.

The child-specific assessment tools such as the WHO hospital tools, HFS-IMCI and Child-HCAHPS, had relative strengths in assessing experience of care when compared with the remaining tools. This may be in recognition that communicating with parents/carers is a large component of paediatric healthcare. The Child-HCAHPS survey was developed for the sole purpose of obtaining parent/guardian feedback on their experience of care of their child in hospital. Health service delivery has traditionally revolved around disease diagnosis and treatment. However, there has been a gradual global shift towards integrated people-centred health services, where people and communities are seen as active participants as well as beneficiaries of their responsive health systems. In 2016, the WHO adopted the ‘Framework on integrated people-centred health services’ to help drive change in national policies on health services delivery to include cross-sectoral collaboration and community involvement and empowerment in decision-making processes.20 Involving people in their own care, especially marginalised subpopulations, is considered essential to achieving equitable access and QoC towards UHC. So, although tools such as the Child-HCAHPS may not be a suitable tool to assess all aspects of QoC for children, it can be a useful adjunctive tool to assess communication skills and patient experiences of care, which may otherwise be lacking in existing quality assessment frameworks.

Feasibility was not formally assessed as part of this review. The more comprehensive WHO Hospital Care Assessment tools are extensive and labour intensive and yet only cover about half the QM in the WHO Standards. Our search only identified two original peer-reviewed publications that used the WHO Hospital Care Assessment Tools.21 22 It is possible that the tools are used for auditing and quality improvement practices with information only being disseminated within country. However, there is little anecdotal evidence of this occurring. Without clear quality frameworks in place and limited resources, health facilities would likely find it challenging to implement these assessment tools effectively. In order for LMICs to effectively implement quality improvement practices and to sustainably assess and monitor them, key indicators need to be clear and manageable. We recommend that the QM in the WHO Standards be simplified and that key indicators to monitor in each QSd be highlighted. Key indicators should obtain global consensus and adhere to a measurement framework to ensure that they are relevant, acceptable, achievable and robust. Existing core assessment tools could then be combined and simplified, to incorporate the key indicators, with flexibility for other QM to be included as prioritised by individual health facilities. This will be more achievable and constructive at the local level, while assisting in reporting of national progress of uniform child health indicators in LMICs.

An alternative to using explicit QoC assessment tools to evaluate and monitor quality improvement practices, would be to have key indicators embedded in other routine data collection systems. In the USA, there are existing monitoring systems of QMs embedded in HIS. Data are collated from multiple databases by such agencies as the Agency for Healthcare Research and Quality to produce annual National Healthcare Quality and Disparities Reports which include indicators overlapping with many of the WHO Standards.23 In LMICs, paper-based surveys and medical records have traditionally been the main way to collect health data. Routine HIS which could potentially monitor key indicators for QoC can be variable in levels of data recording and quality, and are seldom used to evaluate programme interventions and policy changes often due to lack of capacity.24 25 The introduction of electronic health records and platforms are emerging as more reliable sources of data management and analyses in LMICs.26 27 However, a range of challenges has meant that electronic-HIS have not yet replaced paper-based records or survey tools in most LMIC settings.28–31 Until routine HIS become more robust and reliable, using existing, comprehensive, low-cost tools to assess QoC, will continue to more feasible.

Our review focused on assessing QoC for children and young adolescents. However, it is important to recognise that quality healthcare is required throughout the continuum of the life course, including the antenatal and perinatal periods, to ensure improved outcomes for all children. The WHO Standards for children and adolescents are an extension of the WHO Standards for maternal and newborn care in health facilities, with both developed using the same framework. A previous review comparing existing assessment tools with the WHO Standards for maternal and newborn care drew similar conclusions to our assessment—that current tools had gaps in assessing experience of care and that there should be global consensus on core data to be collected.6 Although each life stage has its own unique healthcare needs (eg, obstetric care for women; immunisations for children), comparable themes for quality healthcare practices are applicable across the life course. Having similar frameworks to assess and monitor quality of healthcare across the life course would make quality improvement practices and assessment tools easier to develop. It would also foster collaboration between health sectors in the development of common goals towards achieving better health outcomes for all.

Our systematic review had several limitations. In the matching process, clinical judgement determined whether questions/items from assessment tools matched the QM in the WHO Standards. This subjective process could have led to bias in some indicators. Our selection criteria were aimed at identifying stand-alone assessment tools. This may have inadvertently excluded quality assessment tools integrated within other health data collection activities. We also only included English publications, which may have excluded other existing tools used in non-English speaking countries. Our review only evaluated the survey instruments and did not assess feasibility of implementation, which would include preplanning, training, supervision, evaluation and feedback of data. These pragmatic factors would affect the capacity of a tool to reliably assess the WHO Standards and would need to be considered in future activities assessing QoC.

Conclusion

This review found that although the WHO Standards are comprehensive, no single tool can adequately assess all the QM in its current format. Furthermore, operational use of extensive assessment tools is seldom seen due to lack of resources and organisational frameworks. Existing tools tend to emphasise input measures and few tools adequately assess experience of care. Consensus and harmonisation of select key indicators from the WHO Standards, integrated into simplified assessment tools would make them more achievable in LMICs. Comparable data on key indicators for monitoring within and between countries will also assist in national and global reporting on progress of child health outcomes. Further research into the feasibility of modified tools with key indicators to assess QoC and the impact on health outcomes, is, therefore, an important next step in establishing equitable access to quality healthcare.