A Systematic Review of Quality Indicators for Appropriate Antibiotic Use in Hospitalized Adult Patients.

Many quality indicators for appropriate antibiotic use have been developed. We aimed to make a systematic inventory, including the development methodology and validation procedures, of currently available quality indicators (QIs) for appropriate antibiotic use in hospitalized adult patients. We performed a literature search in the Pubmed interface. From the included articles we abstracted i) the indicators developed ii) the type of infection the QIs applied to iii) study design used for the development of the QIs iv) relation of the QIs to outcome measures v) whether the QIs were validated and vi) the characteristics of the validation cohort. Fourteen studies were included, in which 200 QIs were developed. The most frequently mentioned indicators concerned empirical antibiotic therapy according to the guideline (71% of studies), followed by switch from IV to oral therapy (64% of studies), followed by drawing at least two sets of blood cultures and change to pathogen-directed therapy based on culture results (57% of studies). Most QIs were specifically developed for lower respiratory tract infection, urinary tract infection or sepsis. A RAND-modified Delphi procedure was used in the majority of studies (57%). Six studies took outcome measures into consideration during the procedure. Five out of fourteen studies (36%) tested the clinimetric properties of the QIs and 65% of the tested QIs were considered valid. Many studies report the development of quality indicators for appropriate antibiotic use in hospitalized adult patients. However, only a small number of studies validated the developed QIs. Future validation of QIs is needed if we want to implement them in daily practice.

Competing interest statement

Conflict of interest: the authors declare no potential conflict of interest.

Background

Today antibiotics are indispensable in practically all health care systems.[1] However, the extensive use of antibiotics is also the main driving force in the emergence of resistant microorganisms.[2] Worldwide, antibiotic consumption and antibiotic resistance are still on the rise, which, together with the decline in the discovery of new antibiotics, creates one of the greatest current threats to human health.[2-5]

To curb the rise of antibiotic resistance of medically important bacteria, better use of current agents is warranted and a decrease of inappropriate antibiotic use is imperative.[3] Antibiotic stewardship programs are developed to optimize the appropriateness of antibiotic use, in order to maximize the chance of clinical cure or prevention of infection.[6] At the same time, they aim to limit the unintended consequences of antibiotic use, such as the emergence of resistance, adverse drug events, and costs.[6] Antibiotic stewardship programs (APSs) have shown to be effective and financially self-supporting.[7-9] Multidisciplinary local stewardship teams are now established across the world, with the task to design programs in their own hospitals.

A requirement for an effective stewardship program is the ability to measure the appropriateness of antibiotic use in individual patients. Quality indicators (QIs) are measurable elements of practice performance for which there is evidence or consensus that they can be used to assess the quality of antibiotic care provided.[10] A well-known classification to categorize QIs is: structure-, process- and outcome indicators.[11]

For an optimal and reliable use of the developed QIs, their clinimetric properties have first to be tested in clinical practice. Registration of data is different in every country and varies over time, which has an effect on validity and reliability of data collection. It is mandatory to locally test the clinimetric properties of the QIs, in order to discriminate between indicators that are feasible, valid and reliable in a specific setting and those that are not. There are several criteria to consider when assessing the QIs, including measurability,[12-14] applicability (the indicator should be applicable to a substantial proportion of the reviewed patient records),[14,15] inter-observer reliability,[12-14,16-18] room for improvement,[10,12-14] and case mix stability.[12-14,17]

During the past decade many quality indicators for appropriate antibiotic use have been developed.[19] Providing information on the development and validation processes of available QIs can support healthcare professionals to select QIs that are considered reliable in their healthcare setting. In this systematic review we aimed to make an inventory, including the development methodology and validation procedures, of currently available quality indicators for appropriate antibiotic use in hospitalized adult patients.

Materials and Methods

To create an overview of the existing QIs for appropriate antibiotic use in hospitalized adult patients, two authors (MCK and JMP) performed a systematic literature search in the Pubmed interface. The search strategy is provided in Figure 1. Antibiotics were defined as antibacterial agents. Quality indicators were defined as quality measures, metrics or criteria. Limitations of the search included humans and English language. We screened title and abstract in order to identify studies reporting QIs for antibiotic use. Articles were excluded if they did not concern antibiotic use, did not concern quality measures, did not apply to adults, or concerned the outpatient setting. Duplicate studies were removed. We reviewed potentially relevant articles in full-text format. Articles were excluded if no full text was available or if the minority (<33%) of developed QIs in an article were related to antibiotic use, i.e. QIs as part of a general quality of care set. Finally, we selected the articles describing the consensus or evidence-based development of QIs for appropriate antibiotic use in hospitalized adult patients. The Drive-AB report, a recently performed but not yet published modified Delphi study to identify QIs for antibiotic use, was added to the final list with selected articles.

Figure 1.

Search strategy in Pubmed. Limits: humans, English.

From the included articles we abstracted i) the indicators developed, ii) type of infection the QIs applied to iii) the study design used for the development of the QIs iv) relation of the QIs to outcome measures v) whether the QIs were validated and vi) the characteristics of the validation cohort. See Table 1 for an overview.[19-30]

Table 1.

Overview of studies reporting quality indicators for appropriate antibiotic use.

Author	Year	Study setting	No. QIs	Target infection	Study design	Proven relation with outcome?	Validated?	Validation cohort
Nathwani et al.[20]	2002	EU	6	Pneumonia	Literature review	Yes, as evidenced by a literature and guideline review	No	-
Schouten et a1.[13]	2005	EU	19	LRT1	Four step RAND-modified Delphi procedure: national, multidisciplinary panel of 11 experts from multiple hospitals	Yes, assessed in Delphi study	Yes. 15 QIs validated: 9 out of 12 QIs validated for CAP and 6 out of 7 QIs validated for AECB	899 hospitalized patients with LRT1 in the Netherlands
Berenholtz et al.[21]	2007	US	10	Sepsis	Two step modified Delphi procedure: national, multidisciplinary panel of experts from multiple hospitals	Yes, assessed in Delphi study	No
Hermanides et al.[12]	2008	EU	13	UT1	Three step RAND-modified Delphi procedure: national, multidisciplinary panel of 13 experts from multiple hospitals	Yes, assessed in Delphi study and additional trial	Yes. 9 QIs validated for internal medicine dept. and 4 QIs validated for urology dept.	341 patients with complicated UTIs in the Netherlands
Pulcini et al.[18]	2008	EU	5	NS	Literature review	No	Partially. 4 QIs showed high or moderate inter-rater reliability	Forty medical notes of patients admitted to an ID ward in the UK
Fry et al.[22]	2008	US	3	SSI	Literature review	No	No	-
Zarb et al.[23]	2011	EU	9	NS	Based on results of a PPS	No	No	-
Zarb et al.[24]	2011	EU	2	NS	Based on a retrospective audit	No	No	-
Coll et al.[25]	2012	EU	30	NS	Based on agreement of a multidisciplinary team in one hospital, with reference to the evidence base, national strategy and local policy	No	No
Thern et al.[26]	2014	EU	21	CAP,HAP, UTI,BSI, all infections	Three step RAND/UCLA- modified Delphi procedure: national, multidisciplinary panel of experts from multiple hospitals	No	No
van den Bosch et al.[27]	2014	EU	5	Sepsis	Five step RAND-modified Delphi procedure: national, multidisciplinary panel of 14 experts from multiple hospitals	No	No
Farida et al.[28]	2015	Asia	15	CAP	Two step Delphi procedure: national, multidisciplinary panel of 18 experts from multiple hospitals	Yes, assessed in Delphi study	Yes. 6 QIs validated	128 patients hospitalized with CAP in Indonesia
van den Bosch et al.[14,29]	2015	EU	11	NS	Four step RAND-modified Delphi procedure: international, multidisciplinary panel of 17 experts	Yes, assessed in Delphi study and additional trial	Yes. 7 QIs validated in the Netherlands	1890 patients from 22 hospitals
DRIVE-AB[30]	2016	EU	51	NS	Four step RAND-modified Delphi procedure: international, multidisciplinary panel of 51 experts	No	No	-

ABS, antibiotic stewardship; LRTI, lower respiratory tract infection; UTI, urinary tract infection; CAP, community- acquired pneumonia; HAP, hospital- acquired pneumonia; BSI, blood stream infection; SSI, surgical site infection; PPS, point prevalence survey; HDU, High Dependency Unit; EU, Europe; US, United States of America, NS, not specified.

Results

The systematic literature search resulted in 606 potentially relevant articles. After screening of titles and abstracts, 58 potentially relevant articles were selected for full-text screening. After the full text review another 47 articles were excluded, based on the previously mentioned criteria. Two additional articles were selected from literature references and added to the final list, together with the Drive AB report. Finally, fourteen articles describing the development of QIs regarding appropriate antibiotic use in hospitalized adult patients were included (Figure 2). Details of these studies are given in Table 1.[19-30]

Figure 2.

Procedure of article selection

The fourteen included articles described 200 QIs: 17 structure and 183 process indicators. Most QIs were specifically developed for lower respiratory tract infection, urinary tract infection or sepsis. See Appendix 1 for the complete list of indicators. The most frequently mentioned indicators concerned empirical antibiotic therapy according to the guideline (71% of studies), followed by switch from IV to oral therapy (64% of studies), followed by drawing at least two sets of blood cultures and change to pathogen-directed therapy based on culture results (57% of studies) (Table 2).

Table 2.

Description of the top 10 retrieved quality indicators.

Developed indicators	Number of articles mentioning the indicator / total number of articles	Percentage of articles mentioning the indicator
Prescribe empirical antibiotic therapy according to (local or national) guidelines	10/14	71
Switch from intravenous to oral therapy	9/14	64
Perform at least two sets of blood cultures	8/14	57
Change to pathogen-directed therapy when culture results become available	8/14	57
Timely initiation of antibiotic therapy	7/14	50
Adapt dose and dosing interval of antibiotics to renal function	7/14	50
Documentation of antibiotic plan in medical record	7/14	50
Perform a site culture	6/14	43
Discontinue antibiotic therapy if infection not confirmed	6/14	43
Duration of antibiotic therapy	6/14	43

A (RAND)-modified Delphi consensus procedure was used in the majority of studies (57%). All Delphi studies were performed with a multidisciplinary team of experts, working in different hospitals. Two studies, van den Bosch et al.[14,29] and Drive AB,[30] performed the consensus procedure with an international team of experts. The panel size varied from 11 to 51 experts.

Six studies took outcome measures, like mortality, morbidity, length of hospitalization, or cost-effectiveness, into consideration during the procedure. For two sets of indicators the relation between adherence to the QIs, and length of hospital stay was investigated in a subsequent observational multicenter trial.[14,31]

Five out of fourteen studies (36%) tested the clinimetric properties of the QIs. 41 of 63 tested QIs (65%) were considered valid for use in the clinical setting. The most common reasons why QIs were not considered valid were low feasibility of data abstraction from the patient files and lack of room for improvement.

Discussion and Conclusions

In this systematic review we provided an overview, including the development methodology and validation procedures, of all reported quality indicators for appropriate antibiotic use in hospitalized adult patients. Fourteen studies described 200 QIs: 17 structure and 183 process indicators. Five studies (36%) tested the clinimetric properties of the QIs. 41 of 63 tested QIs (65%) were considered valid for use in the clinical setting.

We performed a literature search in order to include all available studies describing the development of quality indicators in adults. Furthermore we extracted from the included studies the methods used to develop the quality indicators, the relation of the QIs with outcome measures, and the validation process of the QIs. To our knowledge such a detailed inventory of QIs for inpatient antibiotic use, including the development methodology and validation procedures, has not been done before. Recently, a systematic review on QIs for diagnosis and antibiotic treatment in primary care was reported.[32] The majority (72%) of the 130 QIs focused on choice of antibiotics, 22% concerned the decision to prescribe antibiotics, and few (6%) concerned the diagnostic process. Most QIs were either related to respiratory tract infections or not related to any type of infection. Therefore, the QIs developed for this setting clearly differ from those developed for the inpatient setting.

Our study has several limitations. First, for pragmatic reasons the search was performed only in the Pubmed interface. There is a possibility that articles were overlooked for this reason. However we used a wide range of search terms regarding quality measurements and antibiotics and therefore we assume that the terms should identify those studies reporting the development of quality indicators. Second, we included only the articles that developed QIs exclusively related to antibiotic use. If the minority of QIs (<33%) in an article referred to antibiotic use, we did not select the article. Therefore, we might have missed QIs on antibiotic use. Finally, we excluded articles concerning QIs for pediatric care.

The question remains what the implications of these data are. To develop QIs, a systematic or non-systemic method can be used. Systematic methods rely on available scientific evidence complemented when necessary with expert opinion.[33] A Delphi consensus procedure is a systematic process where QIs are developed based on scientific evidence combined with expert opinions.[10,12-14,26,27] and at this moment the most rigorous way to develop QIs. In our study, the most frequently applied method indeed was a modified consensus Delphi procedure (57%). All Delphi panels consisted of a multidisciplinary team of (international) experts, usually working in different hospitals. Non-systematic methods, i.e. a point prevalence survey, were also used to develop the QIs.[23,34]

In a minority of studies (43%) the relation of quality indicators with an outcome measure, like mortality, length of stay or costs, was taken into account during the development process, and for only two sets of indicators the relation between adherence to the QIs, and length of hospital stay was investigated in a subsequent observational multicenter trial. Most of the developed QIs applied only to a specific patient group, for example patients with UTI, CAP or sepsis. Van den Bosch et al. is the only study so far to have developed a generic set of QIs of which the relation to outcome measures was assessed during a Delphi study and in an additional trial.[14] As was stated before, a requirement for an effective stewardship program is the ability to measure the appropriateness of antibiotic use using QIs. For an optimal use of the developed QIs it is mandatory to test their clinimetric properties in daily practice, in order to discriminate between indicators that are feasible, valid and reliable in a specific setting and those that are not. In this study we showed that only 65% of the tested QIs were considered valid, implicating that one third of all developed QIs is possibly not eligible for use in clinical practice. Therefore, we recommend to locally test the applicability of these QIs in a pilot or controlled trial before implementing them. Only few study groups have tested the clinimetric properties of their QIs in daily clinical practice.

In our opinion, the set of QIs developed by the Drive AB group,[30] is the most comprehensive set of QIs, as it is developed in a high-quality consensus procedure and is based on the most recent literature covering antibiotic use in the inpatient and outpatient setting. However, validation of these QIs is necessary in order to implement them in daily practice. So far, only van den Bosch et al.[14,29] managed to develop a high quality, generic and valid set of QIs, and we recommend in comparable settings to apply this set of QIs in stewardship programs.