A systematic approach to develop a core set of parameters for boards of directors to govern quality of care in the ICU.

OBJECTIVE: Hospital boards are legally responsible for the quality of care delivered by healthcare professionals in their hospitals, but experience difficulties in overseeing quality and safety risks. This study aimed to select a core set of parameters for boards to govern quality of care in the intensive care unit (ICU).
DESIGN: Two-round Delphi study.
SETTING: Two university hospitals in the Netherlands. PARTICIPANTS: An expert panel of 12 former ICU patients or their family members, 12 ICU nurses, 12 ICU physicians and 12 members of boards of directors and quality managers. MAIN OUTCOME MEASURES: Participants indicated the relevance of existing parameters for assessing the quality of ICU care for governance purposes (round 1) and selected 10 quality parameters that together provide boards of directors with a good representation of quality of care in their ICU (round 2).
RESULTS: We identified 122 quality parameters related to care in the ICU, which we limited to a short list to present to participants in round 1. The response rate was 94% in round 1 and 85% in round 2. The final set consisted of the 10 most frequently selected quality parameters per hospital. Five parameters were included in both sets; all related to patient safety and continuous quality improvement.
CONCLUSIONS: Parameters in the core set were mostly qualitative and generic, rather than quantitative and ICU-specific in nature. To engage in a true dialog about quality of care, boards are more interested in the story behind the numbers than in just the numbers themselves.

Introduction

Hospital boards have a legal and moral responsibility to ensure high quality of care delivered by healthcare professionals in their hospitals [1]. They are the ones held accountable by government and insurers but experience difficulties in overseeing quality and safety risks [2-5]. Problems such as insufficient resources, gaps in board members’ experience and skills and difficulties to oversee the quality of care in the entire hospital make it difficult for boards to govern the quality of healthcare in their hospital [3].

Currently, hospitals measure and collect hundreds of quality parameters [6]. Their large number however limits their use: it is difficult to obtain an overview and to recognize signals, meaning an important opportunity to continuously improve care is missed. For boards of directors, especially, many of these parameters are of limited use, due to their detailed and specific nature.

Professionals experience a growing pressure to provide requested information, which compromises the internal use of those quality indicators [7]. To improve quality and fight the waste of energy, enthusiasm and (financial) resources, it is paramount that the generated information is actually used by healthcare professionals, managers and directors [8].

This project was aimed at providing boards of directors with insight into quality and safety of care in the intensive care unit (ICU), in order to timely recognize possible quality problems. We selected the ICU as the focus of the project because it is one of the hospital’s core departments, in which critically ill patients are cared for with high-risk interventions. Suboptimal quality and safety of care have a tremendous impact on this patient group as well as for the hospital, in the form of critical incidents and legal claims. The ICU cares for many different types of patients and is therefore an important cog in the hospital’s machine. When quality of ICU care is good (or bad), this will affect other departments in the hospital.

Because of the complexity of ICU care, the importance of a core set of key quality parameters for boards of directors is clear. However, it is unknown which quality information and parameters are suitable for boards of directors to govern quality of ICU care. To assist boards, we systematically asked an expert panel of boards members, patients and their family members and healthcare professionals to determine which quality information is relevant for good governance. The aim of this study was to select a core set of parameters for boards of directors to govern quality of care in the ICU.

Methods

To select a core set of quality parameters, we used the modified Delphi method. This systematic, iterative methodology is used to collect and distill knowledge on a specific topic from a panel of experts. In multiple rounds, experts are confronted with each other’s ideas and viewpoints. This method has important advantages: it can be carried out via questionnaires, meaning it is both time efficient as well as anonymous, which avoids the possible negative impact of power imbalances and participant dominance [9, 10]. We performed separate Delphi studies in two Dutch academic hospitals between March and June 2016. The Delphi studies were conducted separately to allow for tailoring of the core set to the local setting and the needs and desires of each hospital’s stakeholders. Our modified Delphi study consisted of four steps, described below.

Ethical approval was sought from the Research Ethics Committee of the Radboud University Nijmegen Medical Centre (registration number: 2016/2525); the committee judged that ethical approval was not required under Dutch National Law. All participants received written information about the project and its aims and were subsequently invited to participate.

Step 1. Extraction of quality parameters from the available quality information

The basis for the selection of the core set was all available quality information pertaining to the ICU. From January to March 2016, two researchers (AO and MZ) systematically inventoried all quality information registered in both hospitals. Quality information is generated for different information-requesting stakeholders, with different goals (quality improvement, governance and accountability), and is varied in nature (structure, outcome, or process indicators).

From this inventory, two researchers (AO and MZ) extracted the quality parameters that could be used to govern quality of care in the ICU. In consultation with the head of each participating ICU, we limited the long list by removing doubles (parameters registered for separate stakeholders, with (partially) overlapping definitions) and quality parameters very likely to be considered ‘not relevant’. The resulting list of quality parameters was divided into seven domains:

organization of ICU care;

effectiveness of ICU treatment;

incidence and prevention of complications and iatrogenic injury;

learning from complications and incidents;

functioning of individual healthcare professionals and teams;

experiences of patients and family;

patient outcomes and functional status after discharge.

The domain names, quality parameter names, parameter descriptions and the domain allocation were reviewed by each ICU department head and modified if necessary. The resulting list served as the basis for the questionnaire of Delphi round 1.

Step 2. Delphi round 1: relevance of quality parameters

The Delphi panels consisted of 24 experts in each hospital:

6 former ICU patients and family members of former ICU patients;

6 ICU nurses;

6 intensivists;

6 managers or board members (including ICU department head, quality managers, hospital board of directors).

The former patients and family members were recruited through a post-ICU care polyclinic (which sees patients 3 months (or longer) after ICU discharge for a follow-up appointment) and through the foundation for Family and patient Centered Intensive Care, meaning patients had not necessarily been admitted to the ICUs of hospital A or B. The physicians and nurses were selected on the basis of their proven interest or expertize in quality of care (for example, membership of a quality assurance committee). All experts invited agreed to participate.

In the first round, the expert panel received a questionnaire with the list of quality parameters with a brief description, divided into seven domains. The participating professionals received a link to an online version of the questionnaire via e-mail. The patients and family members were sent a Word document or printed version and were guided through the process via telephone (by AO).

The experts were instructed to individually rate each quality parameter on a nine-point Likert scale (ranging from ‘not at all relevant’ [1] to ‘very relevant’ [9]) by asking: ‘For each parameter, indicate how relevant you think this information would be for a board member to determine quality of care in the ICU.’

Based on the relevance scores, the parameters were divided into three categories:

Non-exclusion

A convincing majority of participants considered the parameter relevant: at least 70% of participants scored 7, 8 or 9 and the median was at least 8.

Equivocal

Extremely skewed distribution: at least 30% of participants scored 1, 2 or 3 and at least 30% of participants scored 7, 8 or 9.

Or

Somewhat skewed distribution: at least 70% of participants scored 7, 8 or 9 and the median was 7 or lower.

Exclusion (all other cases)

Parameters in the ‘non-exclusion’ or ‘equivocal’ categories were included in the questionnaire of round 2 [11, 12].

Step 3. Delphi round 2: selection of a core set of quality parameters

In this second round, the experts received the questionnaire via e-mail. If so desired, patients and family members were again guided through the completion process by phone. They were presented with the remaining quality parameters divided into the same domains as in round 1, and were asked the following: ‘In the questionnaire we invite you to select 10 quality parameters from the remaining parameters. Ten quality parameters that provide a board member with sufficient means to enter into a dialog with healthcare professionals about the quality of care in the ICU.’

For each quality parameter in the questionnaire we presented the distribution of round 1 participant scores across the Likert scale, the percentage of participants that scored the parameter in the highest tertile and the median score. (For a fragment from the second round questionnaire, see the online Supplemental Material.)

The 10 most frequently selected parameters in each hospital formed the two core sets.

Step 4. Feasibility study

To evaluate the usefulness of the identified core set of quality parameters, we performed a feasibility study in both hospitals. To this purpose, we filled each core set with data from the previous 2 years. Where relevant, we displayed trends through time or a comparison with national averages. Per parameter, we added a concise description of the most notable observations or the main problem points, with planned improvement actions and current state of affairs, where present. If possible, we displayed information in figures or tables, with the use of signal colors green, orange and red. We discussed draft versions with each ICU department head and subsequently modified the core set where needed.

In both hospitals, the complete core set was used in a conversation between ICU management (department head and financial manager/nursing manager) and a member of the board of directors. A researcher (AO) observed this conversation and asked short evaluation questions afterwards.

Results

Step 1. Quality information inventory

The inventory of quality information yielded 122 individual quality parameters in total in both hospitals. A vast majority of information (72%) was collected for quality improvement purposes. Other indicators were collected for external accountability (17%) and internal governance (11%).

We can distinguish four parameter levels: structure indicators (12%, for example, the 24 h availability of a specialized ICU physician), process indicators (40%, for example, adherence to hand hygiene guidelines), outcome indicators (39%, for example, the number of complications) and patient-reported experiences (9%).

For the first Delphi round, we shortened the long list of 122 parameters to a list of 54 quality parameters in hospital A and 47 quality parameters in hospital B. For the number of parameters per domain in both rounds, see Table 1.

Table 1

Number of quality parameters per domain

Domain	Examples of quality parameters	Number of parameters per domain in round 1		Number of parameters per domain in round 2
		Hospital A	Hospital B	Hospital A	Hospital B
Organization of ICU care	Nurse–patient ratio Days with full bed occupancy Recommendations and points of improvements based on internal audit resultsa	6	5	6	5
Effectiveness of ICU treatment	ICU readmissions Duration of mechanical ventilation Standardized Mortality Ratio (SMR)	6	6	1	0
Incidence and prevention of complications and iatrogenic injury	Number of patients with severe sepsis Compliance with delirium diagnosis guidelines Incidence of airway-related issues	21	18	12	13
Learning from complications and incidents	Preventable adverse events and deaths Complaints Critical incidents	8	8	8	7
Functioning of individual care professionals and team	Team Climate Inventory Annual performance reviews with specialist physicians Compliance with Crew Resource Management guidelines	3	3	2	3
Patient and family experiences	Experiences of ex-ICU patients and their recommendations Experiences of family members of patients (based on questionnaire) Experiences of family members and their recommendations (based on conversations)	3	3	3	3
Patient outcomes and functional status after discharge	Quality of life Fatigue Anxiety and depression	7	4	2	4

aAn internal audit is a multiple-source method that evaluates whether standards and regulations are being followed. The audit involves site visits, interviews, document analysis, surveys and observations. Auditors are colleagues from a different department than the department being audited. The internal audit results in recommendations for quality improvements.

Step 2. Delphi round 1

The response rates for round 1 were 96% (23 experts) and 92% (22 experts) for hospital A and B, respectively. Based on the results, 20 (hospital A) and 12 (hospital B) parameters were excluded. The distribution of the parameters across the three categories is shown in Table 2.

Table 2

Results round 1

Category	Hospital A (n = 54 parameters)	Hospital B (n = 47 parameters)
Exclusion	20 (37%)	12 (26%)
Equivocal	7 (13%)	10 (21%)
Non-exclusion	27 (50%)	25 (53%)

Step 3. Delphi round 2

In hospital A, 22 experts (92%) participated in round 2, compared with 19 experts (79%) in hospital B.

The 10 quality parameters most frequently selected by the participants from hospital A and hospital B can be found in Tables 3 and 4.

Table 3

Core set selected in hospital A

Rank	Quality parameter	Frequency (n = 22)	Rank in hospital B
1–3	Standardized Mortality Ratio (SMR)	17
	Recommendations and points of improvements based on internal audit results	17	7–10
	Preventable adverse events and deaths	17
4	Experiences of post-ICU-patients (from interviews during post-ICU polyclinic visits)	15
5	Team Climate Inventory	14	4–5
6–7	Conclusions, points of advice and recommendations from the quality visitation of the professional organization of Dutch intensivists	13	2–3
	Incident reports	13	7–10
8–9	Critical incidents reported to the Inspectorate	12
	Intensivist presence and availability	12
10	Multidisciplinary discussions of complications	11	7–10

Table 4

Core set selected in hospital B

Rank	Quality parameter	Frequency (n = 19)	Rank in hospital A
1	Compliance with policy for preventing medication errors	14
2–3	Nurse–patient ratio	13
	Conclusions, points of advice and recommendations from the quality visitation of the professional organization of Dutch intensivists	13	6–7
4–5	Days with full bed occupancy	12
	Team Climate Inventory	12	5
6	Experiences of relatives of ICU patients	11
7–10	Recommendations and points of improvement based on internal audit results	10	1–3
	ICU discussions of complications	10
	Multidisciplinary discussions of complications	10	10
	Incident reports	10	6–7

The parameters that were selected for the core sets can roughly be divided into three categories:

parameters as signals for possible safety problems (for example, critical incidents and discussions of complications)

parameters that show whether a department is proceeding through the plan-do-check-act (PDCA) cycle (for example, follow-up after audit results, and team climate);

organizational preconditions for quality and safety of care (for example, presence and availability of an intensivist and nurse–patient ratio).

Additionally, the experiences of former patients or their family members were selected in both hospitals. With regard to the preference for information regarding safety and improvement culture, there were no major differences between the four participant categories (physicians, nurses, patients and managers and board members).

The core sets included mostly qualitative, aggregate information (such as the results of visitations and audits) rather than quantitative indicators (such as the percentage of infections or pressure sores, and protocol adherence rates). The 2 core sets of 10 quality parameters showed overlap: 5 of the 10 parameters were selected in both hospitals. Strikingly, the SMR most often frequently selected in hospital A is not present in hospital B’s core set. Conversely, a parameter with respect to medication error policy scores highest in hospital B, while going unselected in hospital A. (For a categorization of the core set parameters, see Table 5.)

Table 5

Categorization of core set parameters

Quality parameter	Donabedian categorization	Domain	Core set hospital 1?	Core set hospital 2?
Recommendations and points of improvements based on internal audit results	Structure	Organization of ICU care	Y	Y
Conclusions, points of advice and recommendations from the quality visitation of the professional organization of Dutch intensivists	Structure	Organization of ICU care	Y	Y
Nurse–patient ratio	Structure	Organization of ICU care		Y
Days with full bed occupancy	Structure	Organization of ICU care		Y
Intensivist presence and availability	Structure	Organization of ICU care	Y
Standardized Mortality Ratio (SMR)	Outcome	Effectiveness of ICU treatment	Y
Critical incidents reported to the Inspectorate	Outcome	Learning from complications and incidents	Y
Preventable adverse events and deaths	Outcome	Learning from complications and incidents	Y
Compliance with policy for preventing medication errors	Structure	Learning from complications and incidents		Y
Incident reports	Outcome	Learning from complications and incidents	Y	Y
ICU discussions of complications	Outcome	Learning from complications and incidents		Y
Multidisciplinary discussions of complications	Outcome	Learning from complications and incidents	Y	Y
Team Climate Inventory	Process	Functioning of individual care professionals and team	Y	Y
Experiences of post-ICU-patients (from interviews during post-ICU polyclinic visits)	Outcome/ process	Patient and family experiences	Y
Experiences of relatives of ICU patients (survey)	Outcome/ process	Patient and family experiences		Y

The raw data set is available from the authors by request.

Both from this observation as well as from brief evaluation with the participants afterwards, we conclude that there was a true dialog about quality of care based on the presentation of the results of the parameters included in the core set: more so than is usually the case in these meetings, as indicated by the participants. The chosen format with aggregated information about quality and safety issues, improvement actions taken, and current status regarding those actions provided an excellent basis for the conversation.

After the conversation, one board member said the following: ‘What I like about this is that it leads to a true conversation and the was the intent. That you reach the essence of what it’s about. It isn’t about a checklist or a scoring list, it’s about the conversation about where are the challenges, where are the worries, where are the bottlenecks.’

Discussion

This project used the Delphi method to compile two core sets of ICU quality parameters. Boards of directors are the end users of the information produced. Their role in quality improvement is different from that of healthcare professionals, and this study showed that the information boards need to govern quality of care is therefore different in nature than the information healthcare professionals need to continuously improve quality of care.

In the literature on quality dashboards, however, the focus appears to be on quality information for healthcare professionals: real-time information which is continually displayed [13, 14]. The parameters selected for both core sets, however, contain few parameters that could fruitfully be used for such a dashboard. From our conversations with board members, we conclude that board members would rather receive an overview of the most important problem areas, improvement actions and current state of affairs: an overview that can serve as a basis to carry out a dialog with the head of the department about quality of care. They are interested in the story behind the numbers rather than in just the numbers themselves. This is not an isolated finding; the Netherlands has recently seen growing criticism of the dominance of numbers in different sectors [15-17].

Most parameters in the core sets are generic rather than specific to the ICU setting, such as audit results, SMR, and critical incidents. In addition to organisational preconditions and patient experiences, the core sets mostly contains information about patient safety and information that indicates to what extent departments or professionals are in improvement mode. Is a department continuously working to optimize quality of care and does it have a true improvement culture? Is a department in a PDCA cycle and if so, in which phase?

This project was not aimed at producing one generic set of parameters to fit all hospitals and departments. Rather, we considered whether the Delphi method would be a useful tool in compiling a tailor-made set for a specific department or hospital. A department that uses this method would not do away with the other established quality monitoring and improvement methods, such as monitoring by the healthcare inspectorate and automatic registering of quality indicators for internal quality improvement processes. This tailor-made core set would be used as a means to instigate a dialog about quality of care between department and board of directors.

When we started this project, we expected the Delphi process to yield mostly quantitative indicators that could be aggregated and visualized on a quality dashboard. One of our most striking observations, therefore, is the fact that the expert panels selected mostly qualitative parameters.

Furthermore, the overlap between both core sets is noticeable: half of the parameters is the same. Future research will have to show whether this is an incidental finding or points to a trend. If this method is conducted in different departments in the same hospital, this overlap might provide the means to set several general, generic parameters and add several department-, specialty- or disease-specific parameters.

The modified Delphi method to generate the core set has several important advantages. Characteristic of this method is the purposeful selection of experts, meaning a limited number of participants is sufficient to generate valuable information. The inclusion of key figures in the project group make participant recruitment relatively easy and ensured a high response rate. Representation of the most important stakeholders—both board members, healthcare professionals as well as patients and their family members—led to a broadly supported core set.

Because this method makes use of quality information that is already being collected in hospitals, for internal quality improvement or external accountability, no new quality parameters have to be generated and the amount of information to be registered does not increase. In addition, it means that the core set is truly tailored to the hospital which uses it; in both generated core sets, we recognize characteristic accents of both hospitals.

Many generally used quality parameters have not been scientifically evaluated: information about reliability and validity is lacking. This is also the case for the parameters in our core sets. Therefore, we cannot say with certainty whether the core sets give a reliable and valid image of the quality of care in both ICUs. As a conversation starter about quality of care, however, the core set appears to be very well-suited.