Discrepancies in Control Group Mortality Rates Within Studies Assessing Topical Antibiotic Strategies to Prevent Ventilator-Associated Pneumonia: An Umbrella Review.

OBJECTIVES: To test the postulate that concurrent control patients within ICUs studying topical oropharyngeal antibiotics to prevent ventilator-associated pneumonia and mortality would experience spillover effects from the intervention. DATA SOURCES: Studies cited in 15 systematic reviews of various topical antibiotic and other infection prevention interventions among ICU patients. STUDY SELECTION: Studies of topical antibiotics, stratified into concurrent control versus nonconcurrent control designs. Studies of nondecontamination-based infection prevention interventions provide additional points of reference. Studies with no infection prevention intervention provide the mortality benchmark. Data from additional studies and data reported as intention to treat were used within sensitivity tests. DATA EXTRACTION: Mortality incidence proportion data, mortality census, study characteristics, group mean age, ICU type, and study publication year. DATA SYNTHESIS: Two-hundred six studies were included. The summary effect sizes for ventilator-associated pneumonia and mortality prevention derived in the 15 systematic reviews were replicated. The mean ICU mortality incidence for concurrent control groups of topical antibiotic studies (28.5%; 95% CI, 25.0-32.3; n = 41) is higher versus the benchmark (23.7%; 19.2-28.5%; n = 34), versus nonconcurrent control groups (23.5%; 19.3-28.3; n = 14), and versus intervention groups (24.4%; 22.1-26.9; n = 62) of topical antibiotic studies. In meta-regression models adjusted for group-level characteristics such as group mean age and publication year, concurrent control group membership within a topical antibiotic study remains associated with higher mortality (p = 0.027), whereas other group memberships, including membership within an antiseptic study, are each neutral (p = not significant).
CONCLUSIONS: Within topical antibiotic studies, the concurrent control group mortality incidence proportions are inexplicably high, whereas the intervention group mortality proportions are paradoxically similar to a literature-derived benchmark. The unexplained ventilator-associated pneumonia and mortality excess in the concurrent control groups implicates spillover effects within studies of topical antibiotics. The apparent ventilator-associated pneumonia and mortality prevention effects require cautious interpretation.

The incidence of ventilator-associated pneumonia (VAP) among patients requiring prolonged (> 24 hr) ICU stay and methods for its prevention have been extensively studied (1–19).

Oropharyngeal applications of topical antibiotics, with (selective digestive decontamination [SDD]) or without (selective oropharyngeal decontamination) protocolized parenteral antibiotic prophylaxis (PPAP), appear highly effective, with apparent reductions typically being ~50% for VAP and ~10–20% for mortality (10–16). By contrast, nondecontamination-based methods achieve less than 50% reductions in VAP and nonsignificant effects on mortality (4–10), whereas oropharyngeal applications of topical chlorhexidine may be associated with increased mortality among ICU patients (10).

In the first SDD study, Stoutenbeek et al (19) postulated that the results from concurrent control (CC) versus nonconcurrent control (NCC) designed studies of topical antibiotics would differ. Specifically, SDD would reduce the infection risk among CC control group patients but no benefit could occur for NCC control group patients (Fig. ). This contextual effect is akin to the spillover, or herd, effect of vaccination wherein the unvaccinated minority within a population derives benefit (or risk) from the vaccinated majority (20–26).

Stoutenbeek et al (19) predicted that selective digestive decontamination (SDD) given to intervention group patients (black) would have direct effects on the intervention group patients and indirect or spill over (contextual) effects on the CC control group patients (dotted rectangle) in the same ICU (ICU 1, red color) but not on NCC control group patients (dotted rectangle) in a separate ICU (ICU 2, blue color). Hence, the results of SDD studies with CC patients would differ from studies with NCC patients. A physical (brick wall) or temporal (time) barrier separates ICU 1 and ICU 2. CC = concurrent control, NCC = nonconcurrent control.

The two objectives here are first, to benchmark the mortality incidence within component (control and intervention) groups among studies of a broad range of methods of VAP prevention such as various nondecontamination, topical antiseptic, and topical antibiotic-based methods of VAP prevention as cited within 15 systematic reviews. Second, to attempt to reconcile the mortality incidence within a group-level analysis in an attempt to identify any spillover or contextual effect of topical antibiotics in these studies.

MATERIALS AND METHODS

Being an analysis of published work, ethics committee review of this study was not required.

Study Selection and Decant of Groups

The literature search and study decant used here is in four steps as detailed in Figure S1 (Supplemental Digital Content 1, http://links.lww.com/CCX/A129).

The inclusion criteria were study citation in one of 15 systematic reviews or meta-analyses with incidence data for mortality extractable as an incidence proportion with the denominator being the numbers of mechanically ventilated (MV) patients with an ICU stay of at least 24 hours. Additional more recent studies were obtained using the “Related articles” function within Google Scholar to use as tests of sensitivity. The exclusion criteria were as follows: studies restricted to specific patient populations such as studies limited to patients with the acute respiratory distress syndrome, pancreatitis, cardiothoracic or liver transplantation surgery, and studies of pediatric populations. Also, studies in which less than 50% of patients received MV, studies with fewer than 30 patients overall, studies which reported zero mortality, studies limited to parenteral antibiotic prophylaxis as the study intervention, and case-control studies were excluded.

The studies were classified on the basis of whether the study intervention was a topical oropharyngeal antibiotic regimen, a topical oropharyngeal antiseptic regimen, a nondecontamination intervention or no intervention (observational studies). The studies of nondecontamination methods of infection prevention, including VAP prevention, encompass a broad range of methods delivered via the gastric route, the airway route, or the oral care route. Within the topical antibiotic studies, any group receiving topical oropharyngeal antibiotic was regarded as an intervention group and all other groups were regarded as a control group regardless of other interventions including receipt of PPAP. The control groups from these studies were stratified into NCC and CC groups.

Outcomes of Interest

The VAP or ICU mortality incidence was calculated as the number of patients with respectively VAP or ICU mortality per 100 patients. Each incidence was expressed as a proportion using the total number of patients as the denominator. In addition, the following were also extracted where available: whether less than 90% (an arbitrary threshold) of patients received MV, whether the ICU was a trauma ICU, being defined here as having greater than 50% of patients admitted for trauma, the group mean (or median) age, year of study publication, and whether the component group was exposed to PPAP. Late mortality, defined here as hospital mortality or mortality later than day 21, was used in a sensitivity analysis for studies without ICU mortality data.

Study Effect Sizes

The study specific and overall summary effect sizes and associated 95% CI for each category of intervention were calculated using the DerSimonian and Laird random-effect methods of meta-analysis using the “metan” command in Stata 15.1 (Stata Corp., College Station, TX).

Benchmarking: Visual

Caterpillar plots were generated to facilitate a visual benchmark of the mortality incidence proportions. These were generated as follows: the data for mortality were logit transformed to generate caterpillar plots using the “metan” command in Stata as previously. For mortality, this transformation proceeds as follows: with the number of patients as the denominator (D), the number of patients who decease as the numerator (N), and R being the mortality proportion (N/D), the logit(mortality) is log(N/[D–N]) and its variance is 1/(D × R × [1–R]). Note that for any group with a zero event rate (N = 0), the addition of the continuity correction (i.e., N + 0.5) is required to avoid indeterminate transformations of the logit proportion and its variance. The visual benchmark is the summary incidence for ICU mortality as derived using the observational studies. This visual benchmark was then used in the respective caterpillar plots of the component groups from the VAP prevention studies as a reference line. Dot plots were used to provide an “at a glance” summary of the entire evidence base. These were derived as above for caterpillar plots but without the confidence limits.

Benchmarking: Meta-Regression

Group-level regression models of VAP and mortality proportions were developed using meta-regression methods using DerSimonian and Laird random-effects methods using the “metareg” command in Stata (27). In these regression models, the category of observational groups acts as the reference (benchmark) category in each model. The following group-level factors were entered into the meta-regression model: component group type; PPAP use; greater than 90% receiving MV; trauma ICU; group mean (or median) age; and year of study publication. In addition, mode of VAP diagnosis and mortality census time were entered into the VAP and mortality models, respectively. All factors were entered into the meta-regression models without any preselection step.

The regression models were repeated using generalized estimating equation methods (“xtgee” command in Stata). There were three sensitivity tests of the findings. The meta-regression was repeated using intention to treat (ITT) data from eight studies of topical antibiotic-based methods as recorded within the most recent Cochrane review (13). The meta-regression was repeated with and without seven studies found outside of the 15 systematic reviews. The meta-regression was repeated with the additional inclusion of studies, which only reported late mortality data.

RESULTS

Characteristics of the Studies

There are 206 included studies cited within 15 systematic reviews or meta-analyses (1–15). These, together with seven studies providing data used in sensitivity testing and 45 studies meeting exclusion criteria, are listed in Tables S1–S4 (Supplemental Digital Content 1, http://links.lww.com/CCX/A129). Most studies were published between 1990 and 2010 and a minority contained patient groups for which fewer than 90% received MV or originated from trauma ICUs (Table ).

Component groups were decanted from observational studies (Table S1, Supplemental Digital Content 1, http://links.lww.com/CCX/A129), studies of various nondecontamination methods of VAP prevention (Table S2, Supplemental Digital Content 1, http://links.lww.com/CCX/A129), studies of topical antiseptic-based methods (Table S3, Supplemental Digital Content 1, http://links.lww.com/CCX/A129), and studies of topical antibiotic-based methods (Table S4, Supplemental Digital Content 1, http://links.lww.com/CCX/A129). The nondecontamination interventions (Table S2, Supplemental Digital Content 1, http://links.lww.com/CCX/A129) include the following: use or nonuse of different gastric acid inhibitors; feeding by the gastric versus the small bowel route; open versus closed tracheal suction; kinetic bed therapy versus not; subglottic secretion drainage; vitamin D supplementation; and various ventilation strategies. There were 32 different topical antibiotic regimens and four different types of topical antiseptic interventions. Group mean age ranged between 27 and 72 years and was similar across all four categories of study.

Effect Sizes

Of the three categories of intervention, the largest summary effect size for VAP prevention was that associated with topical antibiotic methods (Table 1; Figs. S2–S4 and Table S6, Supplemental Digital Content 1, http://links.lww.com/CCX/A129). A significant mortality prevention effect size was apparent only for the category of topical antibiotic-based interventions and not for the other two intervention categories (Table S7, Supplemental Digital Content 1, http://links.lww.com/CCX/A129). The mortality prevention effect size estimates were in each case comparable to various estimates in the 15 systematic reviews from which the studies had been drawn (Table 1; Table S7 and Figs. S6–S8, Supplemental Digital Content 1, http://links.lww.com/CCX/A129).

TABLE 1.

Characteristics of Studies

Visual Benchmarking

The mean VAP incidence proportion derived from studies without an intervention (observational studies) was 26.1 per 100 patients. There was a difference of more than eight percentage points between the mean VAP incidence derived among the four control group and observational group categories (Table 1; Fig. S5, Supplemental Digital Content 1, http://links.lww.com/CCX/A129). By contrast, the mean VAP incidence derived from each of the three categories of intervention group differed from each other by less than four percentage points.

The ICU mortality incidence proportion is displayed for individual study groups (Figs. ; Figs. S9–S14, Supplemental Digital Content 1, http://links.lww.com/CCX/A129) and in summary (Table 1). Of note, the mean mortality proportions for CC control groups of topical antibiotic studies and for topical antiseptic intervention groups are each more than four percentage points higher versus the ICU mortality benchmark derived from the observational groups, whereas that for each other category differed in each case nonsignificantly from the benchmark by less than three percentage points. By contrast, the mean ICU mortality incidence proportion for intervention groups of topical antibiotic studies and also NCC control groups were each within one percentage point of the benchmark.

Figure 2.

The ICU mortality incidence for the component (C = control, CC = concurrent control, I = intervention, NCC = nonconcurrent control) groups of studies of nondecontamination, topical antiseptic, or topical antibiotic-based methods versus the benchmark being the summary mean (central vertical line) derived from the observation studies (Ob = observational) together with the 95% confidence limits (horizontal error bars) associated with the summary incidence. Shown are incidences from all intervention groups (solid triangles), control groups and observational group incidences (open circles), and control groups that received protocolized parenteral antibiotic prophylaxis (PPAP) (solid squares). The ICU mortality incidence data are displayed in more detail as caterpillar plots (Figs. S9–S14, Supplemental Digital Content 1, http://links.lww.com/CCX/A129). SDD = selective digestive decontamination, SOD = selective oropharyngeal decontamination.

ICU mortality incidence among intervention groups from studies of topical antibiotic methods stratified by whether protocolized parenteral antibiotic prophylaxis (PPAP) was included in the intervention or not. Caterpillar plots of the group-specific (open diamonds) and summary (central broken line and bottom open diamond) mortality incidence and 95% CI. Groups and studies are listed in Table S4 (Supplemental Digital Content 1, http://links.lww.com/CCX/A129). Note that the x-axis is a logit scale. The central solid line is the ICU mortality incidence benchmark from Figure S9 (Supplemental Digital Content 1, http://links.lww.com/CCX/A129) as displayed in Figure 2.

A linear regression line of ICU mortality proportion versus group mean age was derived using the observational groups (Fig. S15, Supplemental Digital Content 1, http://links.lww.com/CCX/A129).The control and intervention groups from all study categories, with one exception, are distributed symmetrically versus this regression line. By contrast, CC groups from studies of antibiotic-based methods are mostly above this regression line (Figs. S15–S17, Supplemental Digital Content 1, http://links.lww.com/CCX/A129).

Statistical Benchmarking

In meta-regression models for VAP and for mortality (Table 2), membership of each category of intervention group was associated with a lower VAP incidence (p < 0.02), whereas no intervention group category was associated with lower mortality (p = not significant).

TABLE 2.

Meta-Regression Modelsa,b,c

Strikingly, membership of a CC group of a topical antibiotic study was associated with higher mortality (p = 0.027). The size of this association exceeded that associated with 10 years difference in group mean age. By contrast, membership of no other category of component group was associated a significant effect in the mortality model.

ITT data reporting an additional 103 deaths among 263 patients (39%) from eight CC design studies of topical antibiotics as reported within one systematic review (13) are listed in Table S5 (Supplemental Digital Content 1, http://links.lww.com/CCX/A129). The ITT mortality incidence proportions among the control and intervention groups are between 2 and 20 percentage points higher than the published data (as in Table S4, Supplemental Digital Content 1, http://links.lww.com/CCX/A129) of these studies. Using these ITT data in the meta-regression fails to change the findings (data not shown). Two other sensitivity tests as listed in the methods gave similar findings to the base analysis (Table 2).

DISCUSSION

This analysis benchmarks the ICU mortality incidences in the component groups of studies, drawn mostly from 15 systematic reviews (1–15), for three broad categories of VAP prevention methods versus a benchmark derived using the ICU mortality data from observational studies of MV patients. Each of the three broad categories of intervention appears effective in preventing VAP among patients receiving MV, whereas a significant mortality prevention effect was apparent only for the topical antibiotic-based methods. This analysis is undertaken to address the question of possible spillover effects associated with the use of topical antibiotics as infection prevention, as originally postulated (19).

Of note, the mortality effect sizes estimated here are each broadly comparable to those reported in the systematic reviews from which the studies were sourced (Table S7, Supplemental Digital Content 1, http://links.lww.com/CCX/A129). However, in addition to the discrepancy between the results of topical antibiotics studies using CC design versus studies of topical antibiotics using NCC design, there are three observations which suggest spillover effects from the topical oropharyngeal antibiotic intervention used within these VAP prevention studies.

First, there is a nearly five percentage point excess mortality in the CC control groups of the topical antibiotic studies versus the benchmark, whereas the overall mean mortality in the categories of intervention and NCC control groups of the topical antibiotic studies are each within one percentage point of the benchmark. Furthermore, the mean ICU mortality incidence for all other categories of component group, with the exception of topical antiseptic intervention groups (see below), differs from the benchmark by less than three percentage points.

Second, the ICU mortality incidence for CC groups is generally higher than expected (Fig. 3) and higher than expected versus group mean age (Figs. S16 and S17, Supplemental Digital Content 1, http://links.lww.com/CCX/A129). Furthermore, the meta-regression models developed using several additional group-level predictors of mortality incidence fail to account for this mortality excess (Table 2). In these models, membership of a CC control group of a topical antibiotic study remained a significant and substantial (positive) predictor of mortality incidence, whereas no other intervention group category was associated with a significant coefficient, positive or negative, in the model. By contrast, membership of the category of topical antiseptic intervention groups, which was associated with a significant four percentage point excess in mortality, was not significant in the mortality meta-regression model.

Figure 3.

ICU mortality incidence among control groups from NCC (top) and CC (bottom) studies of topical antibiotic methods. Caterpillar plots of the group-specific (small diamonds) and summary (central broken line and large open diamond) mortality incidence and 95% CI. Groups and studies are listed in Table S4 (Supplemental Digital Content 1, http://links.lww.com/CCX/A129). Note that the x-axis is a logit scale. The central solid line is the ICU mortality incidence benchmark from Figure S9 (Supplemental Digital Content 1, http://links.lww.com/CCX/A129) as displayed in Figure 2. CC = concurrent control, NCC = nonconcurrent control.

Third, in general, studies of SDD having a CC design provide stronger evidence of prevention effect versus studies of SDD with an NCC design (25). On the one hand, several recent meta-analyses of mostly CC design studies of topical antibiotic methods show significant reductions in mortality of up to 20% (10–16); on the other hand, two large multicenter European NCC studies of these interventions found neutral results for mortality (17), which, in one (18), became significant only in an adjusted analysis. Apart from the discrepancy, the pattern of the discrepancy is contrary to the expectation, as originally postulated (19), that SDD might reduce infection risk among CC control patients as a spillover effect and bias CC study results toward the null versus studies with NCC control patients. The pattern of the discrepancy is also contrary in two ways to the general pattern for results of CC and NCC design studies of the same topic (28). In general, in the investigation of new therapies, trials that have used historic controls are more likely to report a benefit than are randomized trials (28). Furthermore, this difference can be attributed to inequalities in the prognostic factors among the historic control groups in comparison with randomized control groups with a worse outcome for the historic control groups as a consequence. By contrast, the conflicting results of CC versus NCC studies of SDD are doubly discrepant to these patterns.

There are six key limitations to this analysis, the first being that, intentionally, there was considerable heterogeneity in the interventions, populations, mean group age, and study designs among the studies included here. This heterogeneity reflects the broad mix of patient populations within the studies of the 15 systematic reviews. Hence, the summary effect sizes derived here as displayed in Figs. S2–S4, S6–S8, and Tables S6 and S7 (Supplemental Digital Content 1, http://links.lww.com/CCX/A129) are indicative only.

Second, the studies have been published over a period of 30 years. However, the mortality incidence varied nonsignificantly with year of publication in the meta-regression model.

Third, the findings are inherently observational and relate to the group rather than the patient level of analysis. Of note, this is not an individual patient-level analysis. The meta-regression models include only a limited number of key group-level factors, and there was neither attempt nor ability to adjust for the underlying patient-level risk or variation in treatment practices between studies within the analysis.

The fourth limitation is that in estimating ICU mortality incidence, competing risks such as ICU discharge and patient exclusion due to early mortality, would likely cause underestimation of group mortality. For example, the additional ITT data cited in one systematic review (13) arose from several studies of topical antibiotics, which excluded patients usually because the patient died before completing the course of topical antibiotic prophylaxis (or matching control). Although the overall impact of these competing risk biases is uncertain, of note, the inclusion or not of mortality data from studies which have only late mortality data, and also inclusion or not of ITT data from eight studies, gave similar findings within the meta-regression models (Table 2 and data not shown).

Furthermore, this analysis was mostly limited to studies listed within 15 systematic reviews. Systematic reviews typically exclude those studies with only NCC-group patients due to their low-quality scores. Hence, to minimize the risk of missing some NCC studies, these and other difficult-to-locate studies were actively sought for sensitivity testing the regression model. Also, systematic reviews of studies not in the controlled trial format, such as studies of nondrug interventions, have not been included here. For example, a systematic review of 13 “before and after” studies of the impact of ventilator care bundles toward reducing ICU mortality was not included here (29).

Finally, the observational studies, from which the mortality benchmark is derived, are key to this analysis. It is presumed that their mortality experience is generalizable to the prevention studies. Furthermore, none of these observational studies explicitly stated that decontamination strategies were used. It remains possible that decontamination use was a part of the units’ ventilation bundle. This, however, is unlikely as published surveys indicate that SDD was used in less than 5% of European ICU’s over this period of time (30, 31).

A strength of this analysis is that group mean age was available from greater than 200 studies to test as a group-level predictor of ICU mortality risk. Furthermore, the data and results for each individual study are traceable in the Online Data Supplement (Supplemental Digital Content 1, http://links.lww.com/CCX/A129). By contrast, a substantial variation in 28-day group mortality rate was found among 65 trials of sepsis therapies undertaken in the critical care context (32). However, the group mean age varied little across these 65 studies (sd, 3.8 yr), and hence a correlation with 28-day group mortality rate was not demonstrable (32).

Where an intervention appears to prevent ICU-acquired infection within a CC trial, it remains crucial to clarify how much of the effect is due to a direct effect of the intervention on the intervention group patients versus how much may result from any spillover effect of the intervention on the CC control group patients. This question is unanswerable for any one study or even any one systematic review examined in isolation. To clarify the paradoxical observations among the studies of topical antibiotic-based methods noted here would require a large, purpose-designed, cluster randomized trial of topical antibiotics in ICU patients. Such a study would present unique logistic and ethical challenges (33, 34).

The analysis here takes advantage of the NCC versus CC designs of the topical antibiotic studies together with reference to an external benchmark as a type of natural experiment (22, 34) of the postulate that topical antibiotics might influence events among nonintervention (i.e., CC) patients within the study ICU as a contextual or spillover effect. Contextual effects, which are of great interest within population-based prevention studies of communicable diseases, are inapparent at the individual patient level of analysis and cannot otherwise be estimated (34).

The paradoxical higher mortality noted here for the CC control groups of studies of topical antibiotics aligns with other observations of higher and otherwise unexplained control group incidence of VAP overall (Fig S5, Supplemental Digital Content 1, http://links.lww.com/CCX/A129), VAP associated with Pseudomonas (23), Candida and Acinetobacter (20) and bacteremia overall, Pseudomonas bacteremia (35), Enterococcus bacteremia (22), as well as candidemia (21) and patterns of colonization (36). In each case, the endpoint incidence is higher among CC groups of randomized controlled trials of SDD versus the respective benchmarks and versus the component groups of other studies of MV patients.

It remains to speculate on the potential mechanism of the observations noted here. The generally higher incidence of a range of infection endpoints may indicate inapparent cross-infection within ICUs using topical antibiotics to prevent infection resulting in a dysbiosis of the ICU microbiome extending to patients exposed to the context of topical antibiotic use within the ICU.

CONCLUSIONS

The incidence of VAP and mortality within the CC groups within studies of topical antibiotics used to prevent ICU-acquired infection are unusually high versus comparable groups in the literature and remains unexplained. This is consistent with negative spillover effects within CC studies causing the apparent prevention effect to be spurious. The inference that topical antibiotics prevent mortality requires cautious interpretation.