Set of Quality Indicators of Pediatric Intensive Care in Spain: Delphi Method Selection.

INTRODUCTION: This study objective was to identify, select, and define a basic set of quality indicators for pediatric intensive care in Spain.
METHODS: (1) Review of the literature to identify quality indicators and their defining elements and (2) selection of indicators by consensus of a group of experts using basic Delphi methodology (2 rounds) and forms distributed by email among experts from the Spanish society of pediatric intensive care.
RESULTS: We selected quality indicators according to their relevance and feasibility and the experts' agreement on their incorporation in the final set. We included only those indicators whose assessment was within the highest tertile and greater than or equal to 70% evaluator agreement in the final selection. Starting from an initially proposed set of 136 indicators, 31 experts first selected 43 indicators for inclusion in the second round. Twenty indicators were selected for the final set. This "top 20" set comprised 9 process indicators, 9 of results (especially treatment-associated adverse effects), and 2 indicators of structure. Several of them are classical indicators in intensive care medicine (rates of hospital-acquired infections, pressure ulcers, etc.), whereas others are specifically pediatric (eg, unrestricted parent visitation or training the parents of technology-dependent children).
CONCLUSIONS: We reached a consensus on a set of 20 essential quality indicators for pediatric intensive care in Spain. A significant subset reflects the peculiarities of pediatric care. We consider this subset as a starting point for future projects of network collaboration between pediatric intensive care units in Spain.

The quality of healthcare management requires monitoring of (1) key results of adequate assistance (clinical, professional, financial, and user satisfaction) and (2) adverse events arising from patient care. Monitoring usually involves the use of so-called indicators of quality.[1]

By means of measurement, publication, and recognition, such metrics can help reduce the variability of healthcare quality.[2] Professionals and managers of pediatric critical care units can use quality indicators and their standard values as a guide to set their units’ objectives and design their improvement projects. The so-called composite indicators (combining several individual indexes) summarize in a single measurement several critical aspects of the process and outcome of care, considered benchmarks of the performance, and quality of healthcare units and institutions. Health organizations also use composite indicators as a tool for external benchmarking and internal tracking of improvements.[3]

Several medical societies have established sets of indicators and standards of quality in their field.[4] Some intensive care medicine organizations have performed exhaustive reviews of possible indicators, leaving the final decision about their use as tools for improvement to the respective health unit.[5] However, the most recent initiatives focus on a few indicators with a high degree of expert agreement.[1,6]

In this work, we show the results of the project “QuaCIP” (Quality, Cuidados Intensivos Pediátricos) of the Working Group on Safety of the Spanish Society of Pediatric Intensive Care (SECIP). Our goal was to identify, select, and define a set of quality and safety indicators for use in pediatric intensive care units (PICUs) with the greatest possible consensus of experts from our scientific society.

METHODS

The present study was performed in stages as follows:

Identification of potential quality indicators (from October to December 2014). An initial set of indicators was drawn up based on (1) existing metrics of the Spanish Society of Intensive Care Medicine of adults,[5] (2) a literature search in Medline (keywords: “ pediatric critical care,” “quality markers,” and “quality indicators”), and (3) contributions of SECIP working groups through their coordinators. We also requested the collaboration of members who explicitly expressed interest in participating in the subsequent stages of the study.

Indicator assessment (from January to May 2015). This stage was carried out after basic Delphi methodology in 2 rounds.

The first round involved 31 evaluators randomly selected from members of SECIP working groups, all physicians with current clinical activity, who agreed to collaborate. Evaluators used Excel (Microsoft Office, Microsoft Corporation, Redmond, WA) spreadsheets to record their evaluation of the relevance and feasibility of each indicator, defined as follows: (1) relevance: the indicators regarded as most relevant were those that reflect either very prevalent aspects in PICUs, those associated with high morbidity and mortality, or those closely related to clinically relevant outcomes (ie, with impact on patient health). These indicators are important for a variety of stakeholders (patients and/or relatives, PICU staff, and hospital managers).[2] Feasibility: the easier the measurement according to resource availability, the more useful the indicator. Evaluators used Likert ordinal scales (from 1 to 9) to evaluate relevance and feasibility, and they were encouraged to add comments and observations about each indicator.

We expressed the results of evaluations as average values and ranges. Only those indicators whose average relevance and feasibility values fell within the highest tertile in this first round were carried forward to the second round.

In the second round, 24 of the 31 initial evaluators with an exclusive dedication to a PICU repeated their relevance and feasibility scoring of the remaining indicators after the first round. The anonymized comments made by the evaluators in the first round were distributed to the entire group of experts to facilitate consensus. The evaluators were also asked to indicate their degree of agreement with the final selection of each potential indicator (using a 5-point ordinal scale, from “absolute disagreement” to “absolute agreement”). Only those indicators with an average value higher than 7 in relevance and feasibility, favorable vote from 70% of evaluators and no more than 2 unfavorable opinions, were positively selected from this round.

Final definition (from May to July 2015). In this stage, we defined the components of the final 20 indicators. We emphasized the importance of establishing mean values and standard ranges for each indicator, the potential barriers to measurement, and published evidence to support the selection of the indicator. When such evidence was scarce or not available, the indicator-defining element was decided by consensus.

RESULTS

The initial list of indicators comprised the contributions of 23 members of SECIP. Of the 300 members of SECIP, 60 (20%) explicitly agreed to collaborate in the subsequent stages of the project. All 31 selected evaluators were attending physicians working in PICUs located in 11 different regions of Spain. All evaluators involved in the project completed their evaluations in a timely manner.

The initial list included 136 potential indicators. This list was reduced to 43 in the first round and finally to 20 indicators after the second round (Table 1). The top 20 indicators showed a mean relevance value of 7.8 (range, 7.3–8.5) and a mean feasibility value of 7.7 (range, 6.6–8.3), of a maximum score of 9. There was unanimous consensus on only 2 indicators (rate of central venous catheter–associated bacteremia and rate of mechanical ventilation–associated respiratory infection).

Table 1.

Indicators Selected after the Second Round, Classified According to the Proportion of Experts in Favor of Their Inclusion in the Final Set, Based on Average Relevance and Feasibility (Ordinal Scale from 1 to 9)

The top 20 set comprised 9 process indicators, 9 outcome metrics (especially treatment-related adverse effects), and 2 structural indicators (eg, availability of a system for notification of adverse events and nursing protocols for drug administration). Seven indicators required group consensus to propose a standard, especially some process indicators. Specific evidence-based support proved difficult to obtain for only 1 result indicator (rate of pressure ulcers [PUs] associated with noninvasive ventilation). Table 2 summarizes the main characteristics of the final top 20 indicators and bibliographic references.

Table 2.

Quality Indicators Included in the Final Set (Top 20 Indicators); Main Characteristics and Published Evidence

DISCUSSION

In the present work, we identified, defined, and selected a set of readily measurable quality indicators for pediatric intensive care in Spain. This set was based on the consensus of experts belonging to working groups in our medical society.

Several groups have previously attempted to define the characteristics of ideal quality indicators in intensive care medicine, such as sufficient variability to distinguish between good and poor performers, relative insensitivity to severity adjustment, and the ability to capture what providers do rather than patients’ characteristics.[26] The main conclusion is that even traditional indicators such as standardized mortality rates or mean hospital length of stay are not ideal indicators. In short, there is no gold standard in this field. In this context, the opinion of attending physicians themselves becomes more relevant. If attending physicians regard a given indicator either as unimportant or as nonmeasurable, then it is empirically useless. For this reason, we selected our indicators by expert consensus using the Delphi methodology, as described by others.[1,4,6]

We did not select the evaluators according to their expertise in quality management or their professional prestige in our society, as in other studies. We gave priority to the fact that all of them were interested in participating in the study, were active members of working groups of our society, and were physicians with current clinical activity. In summary, they were the same type of professionals who eventually will use the tool we are proposing.

Most of our top 20 indicators already seem as quality indicators in intensive care medicine (eg, rates of bacteremia, infection associated with mechanical ventilation, PUs, etc.).[1,6] However, indicators of clinical efficiency, such as length of stay, were not prioritized in the final selection. This exclusion may reflect the different level of bed demand and case mix among the Spanish PICUs. Finally, a subset within our top 20 group represents a healthcare priority focused on children and their families, such as the parental training in cases of children with significant risk because of technological dependence or the unlimited parent visitation with their child during hospitalization.[27]

Many quality indicators reflect healthcare complications, and in general, their current standards are amply supported by published reports. However, we found it difficult to propose a standard for some of them: that is, the rate of PUs associated with noninvasive mechanical ventilation , which is not frequently referred to in published studies. Probably, its incidence is buried within the overall incidence of PU, with around 30% of PU in pediatrics.[28] Others, such as the rate of infection associated with mechanical ventilation, are currently decreasing because of improvement projects (“pneumonia zero”), so that any standard we propose now will become obsolete in the short term.

The present work is the first consensus-based study of quality indicators in pediatric intensive care in our country, but it has limitations. First, it did not explicitly consider the criteria of other stakeholders, such as nursing staff or parents, as some previous studies have done.[29,30] Second, none of the final set of indicators relates to certain complex healthcare processes (eg, cardiac surgery and organ transplantation) and techniques (eg, extracorporeal membrane oxygenation). Instead, we wished to identify general quality indicators for use in any PICU.

We believe that our results can serve as the cornerstone of a nationwide network of collaborating PICUs to monitor results, identify benchmarks, and initiate a national system of continuous improvement. There is currently no nationwide infrastructure to systematically improve pediatric intensive care in Spain. A pediatric critical care registry would be an excellent tool to drive improvement in our field, as it has been in other fields.[31] In any case, our scientific society seems to be the ideal framework to collaborate with government agencies, scientific societies, or private institutions in this mission. The first step is probably to test the utility of our top 20 indicators and resolve practical problems of implementation in operational PICUs, which may even lead to further simplification of the list of quality indicators.

CONCLUDING SUMMARY

With broad participation of experts from our scientific society and using a collaborative methodology, we reached consensus on a set of 20 essential quality indicators for pediatric intensive care in Spain. Most are traditional indicators of intensive care medicine, but others reflect the peculiarities of pediatric intensive care. This work is a starting point for future projects of network collaboration between PICUs in Spain.

ACKNOWLEDGMENTS

We thank the evaluators who participated in this study: Amalia Martínez, Fernando Gómez, Mónica Balaguer, Ángela Ferrer, Juan Luis Pérez, César Pérez-Caballero, Marta Brezmes, Antonio Morales, Corsino Rey, Juan I. Sánchez, Silvia Belda, Arturo Hernández, Andrés Concha, Fco. José Cambra, Antonio Rodríguez, Ramón Hernández, Julio López, Francisco Fernández, Montserrat Pujol, Pedro Gómez, Mª José Salmerón, José D. López, Elvira Morteruel, Ricardo López, David Ruano, Alberto García, Iria González, Irene García, César Villa, Victoria García, and Juan C. de Carlos. We also thank Mr. Michael McClean for his help in the translation of this article.

DISCLOSURE

The authors have no financial interest to declare in relation to the content of this article.