An International Prospective Cohort Study To Validate 2 Prediction Rules for Infections Caused by Third-generation Cephalosporin-resistant Enterobacterales.

BACKGROUND: The possibility of bloodstream infections caused by third-generation cephalosporin-resistant Enterobacterales (3GC-R-BSI) leads to a trade-off between empiric inappropriate treatment (IAT) and unnecessary carbapenem use (UCU). Accurately predicting 3GC-R-BSI could reduce IAT and UCU. We externally validate 2 previously derived prediction rules for community-onset (CO) and hospital-onset (HO) suspected bloodstream infections.
METHODS: In 33 hospitals in 13 countries we prospectively enrolled 200 patients per hospital in whom blood cultures were obtained and intravenous antibiotics with coverage for Enterobacterales were empirically started. Cases were defined as 3GC-R-BSI or 3GC-R gram-negative infection (3GC-R-GNI) (analysis 2); all other outcomes served as a comparator. Model discrimination and calibration were assessed. Impact on carbapenem use was assessed at several cutoff points.
RESULTS: 4650 CO infection episodes were included and the prevalence of 3GC-R-BSI was 2.1% (n = 97). IAT occurred in 69 of 97 (71.1%) 3GC-R-BSI and UCU in 398 of 4553 non-3GC-R-BSI patients (8.7%). Model calibration was good, and the AUC was .79 (95% CI, .75-.83) for 3GC-R-BSI. The prediction rule potentially reduced IAT to 62% (60/97) while keeping UCU comparable at 8.4% or could reduce UCU to 6.3% (287/4553) while keeping IAT equal. IAT and UCU in all 3GC-R-GNIs (analysis 2) improved at similar percentages. 1683 HO infection episodes were included and the prevalence of 3GC-R-BSI was 4.9% (n = 83). Here model calibration was insufficient.
CONCLUSIONS: A prediction rule for CO 3GC-R infection was validated in an international cohort and could improve empirical antibiotic use. Validation of the HO rule yielded suboptimal performance.

Choosing empiric antibiotic therapy is increasingly troublesome due to the increase in antibiotic resistance. Inappropriate empiric treatment of infections caused by third-generation cephalosporin-resistant Enterobacterales (3GC-R-EB) may lead to worse outcomes for individual patients, while liberal use of carbapenems increases selective pressure for carbapenem resistance [1]. Therefore, better prediction of a patient’s risk of infection with 3GC-R-EB would improve clinical decision making in this trade-off between inappropriate empiric antibiotics and unnecessary use of broad-spectrum antibiotics, such as carbapenems.

Previous studies have focused on prediction of bloodstream infections (BSIs) caused by 3GC-R-EB (3GC-R-BSI) [2-4]. However, in most studies, patients with 3GC-R-BSI were compared with patients with documented BSI, caused by non–3GC-R gram-negatives, ignoring patients who are empirically treated for gram-negative infection in whom blood cultures remain or do not yield gram-negative bacteria. To be useful for selecting empiric antibiotics, a prediction rule must be derived from all patients in that domain, which includes all patients treated for presumed gram-negative infection, even when the blood culture yields gram-positive pathogens or no pathogens at all.

Rottier et al [5] previously developed 2 prediction rules for 3GC-R-BSI, one for community-onset and one for hospital-onset suspected BSIs, based on a retrospective nested case-control study in 8 Dutch hospitals. Both rules more accurately predicted the risk of 3GC-R-BSI compared with the presence of 2 risk factors: having prior isolation of 3GC-R-EB in microbiological testing or prior fluoroquinolone/cephalosporin use. These risk factors are currently recommended in the Dutch guideline for considering coverage of these bacteria in empirical treatment. Yet, before implementation of a new prediction rule, external validation is required. In this study, we validated both prediction rules in an international prospective cohort study.

METHODS

Settings and Patients

Between February 2017 and June 2019, we performed a prospective cohort study in 33 centers in 13 countries, of which 26 were university hospitals. In every hospital, the goal was to collect 200 consecutive infection episodes where a causative role of gram-negative pathogens was suspected. Such episodes were defined as follows: (1) the obtainment of a blood culture and (2) the start of intravenous antibiotics that cover gram-negative pathogens within a period of 2 hours before through 12 hours after blood culture obtainment and (3) the patient being 18 years or older. (Details on inclusion criteria are specified in Supplementary Materials A.)

The outcome of interest was 3GC-R-BSI with all other blood culture results (including negative) serving as a comparator (analysis 1). In a second (utility) analysis, 3GC-R gram-negative infections (3GC-R-GNIs) (which included 3GC-R-BSI) were considered as the outcome. 3GC-R-BSI was defined as an infection episode in which 3GC-R-EB was cultured in at least 1 blood culture bottle (drawn from any site). 3GC-R-GNI was defined as an infection episode in which 3GC-R-EB was cultured from a clinical culture and/or blood culture. Multiple sets of blood cultures obtained on the same calendar day were considered part of the same infection episode.

Analysis 2 serves to assess the effect of using the prediction rule use for all 3GC-R infections, when using the prediction rule derived for predicting the risk of 3GC-R-BSI. To clarify: in analysis 2 we only redefine the outcome to incorporate non-BSI but do not reassess model performance. The study population (suspected BSI) remains the same.

Infection episodes were categorized as community onset if blood culture obtainment occurred at day 0, 1, or 2 of hospital admission. All other episodes were categorized as hospital-onset infections. Species identification and susceptibility testing were based on local standard procedures. All hospitals except for one used EUCAST (European Committee on Antimicrobial Susceptibility Testing) criteria for determination of antibiotic susceptibilities [6]. The other hospital used CLSI (Clinical and Laboratory Standards Institute) criteria [7].

Data Collection

Data were entered in an electronic case record form (eCRF) in 2 steps. First, local investigators screened consecutive blood cultures and determined study eligibility, followed by entering culture and admission dates and relevant predictors for either community- or hospital-onset infection. Second, blood culture results were entered at a later stage in a separate eCRF to avoid information bias. Additional data collection included empiric antimicrobial treatment, the ward on which the blood culture was obtained, and the presence of any other clinical cultures yielding 3GC-R-EB at the day of blood culture obtainment. Data entry was based on chart review.

Prediction Rules and Definitions

For the 2 prediction rules, see Box 1. For definition of individual predictors, see Supplementary Table A.

Box 1.

Prediction Rules

We defined empiric use of carbapenems as appropriate if prescribed in patients with 3GC-R-BSI (primary analysis, focusing on 3GC-R-BSI, and in patients with 3GC-R-GNI, analysis 2). Unnecessary carbapenem use was defined as any carbapenem prescription for infection episodes without 3GC-R-BSI (analysis 1) or without 3GC-R-GNI (analysis 2). Inappropriate therapy was defined as not prescribing carbapenems for 3GC-R-BSI (analysis 1) or for 3GC-R-GNI (analysis 2).

Sample Size

We used the statistical rule of thumb of 100 cases (3GC-R-BSI) per prediction rule [8]. For feasibility reasons, it was decided to end data collection in June 2019 with 97 and 83 3GC-R-BSI cases of community-onset and hospital-onset infections, respectively.

Missing Data

Three sites that started data collection were excluded: one only included positive blood cultures, one completed data for 2 patients only and then stopped, and one only entered culture dates without predictor or outcome data.

Included patients without outcome data (culture results) or date of birth were removed from the dataset.

Missing individual predictors were approached pragmatically as to reflect clinical practice, where information about predictors is not always available. Thus, whenever a predictor was missing, this was considered to be “no.” See Supplementary Material for numbers of missing individual predictors.

Statistical Analysis

Data are reported as medians, means, or percentages, where appropriate. The prediction rules were validated by assessing discrimination and calibration of the prediction models. We calculated c-statistics and corresponding 95% confidence intervals (CIs) to evaluate the discrimination, which is a measure of how well the model can separate cases and noncases, and visually assessed the calibration plots. We recalibrated the models to take into account the higher incidence of 3GC-R-BSI in an international population by recalibrating (only) the intercept. To assess the potential impact of the model on “eligibility for carbapenem prescription,” we report predictive performance (sensitivity, specificity, positive-predictive value, negative-predictive value) for different cutoffs, as well as potential changes in carbapenem use with those cutoffs. See Supplementary Material B for a more elaborate description of statistical methods. We performed 1 sensitivity analysis, dividing the community-onset cohort in low– and high–3GC-R-BSI countries.

Ethics

The ethics committees of the University Medical Center Utrecht and participating hospitals granted a waiver for informed consent because of the observational nature of the study. Patient data were anonymized at the respective study sites.

RESULTS

We included 6576 infection episodes from 33 hospitals in 13 countries, of which 235 were considered noneligible (Figure 1). There were 4650 community-onset and 1683 hospital-onset infection episodes. Most clinical sites were in Italy (n = 7), followed by the Netherlands (n = 6) and Spain (n = 3). Most 3GC-R-BSI episodes came from Italy (94 out of 180, 52%), followed by Turkey (n = 22, 12.2%). In the community-onset infections, 55% of episodes had clinical predictors entered within 3 days or less from blood culture obtainment. In the hospital-onset cohort, this held for 49% of episodes (Supplementary Material C).

Figure 1.

Flow chart of the study. There are 2 separate cohorts, one for the community-onset prediction rule and one for the hospital-onset prediction rule, where different data were collected. Abbreviations: 3GC-R-BSI, third-generation cephalosporin-resistant bloodstream infection; 3GC-R GNI, third-generation cephalosporin-resistant gram-negative infection.

Community-onset Cohort

Among the 4650 community-onset infections, there were 97 3GC-R-BSI episodes (2.05%), of which 9 with pathogens were co-resistant to carbapenems (0.2%; 9% of BSIs caused by 3GC-R-EB). In the remaining 4553 episodes, blood cultures were either negative (n = 3680) or yielded other pathogens (n = 873) (Table 1).

Table 1.

Predictors and Baseline Variables of the Community-onset Rule

	3GC-R EB BSI (n = 97)	Without 3GC-R EB BSI (4553)
Predictors
Age, years	73.1 ± 12.9	66.5 ± 17.9
Suspected source of infection
Urinary tract	52 (53.6)	933 (20.5)
Respiratory tract	14 (14.4)	1615 (35.5)
Immunocompromised	30 (30.9)	1095 (24.1)
Prior culture with 3GC-R EB (<1 year)	27 (27.8)	255 (5.6)
Prior antibiotic use (<2 months)	65 (67.0)	1710 (37.6)
Other descriptive variables
Male sex	61 (62.9)	2611 (57.3)
Culture obtained where
Internal medicine	68 (70.1)	3326 (73.1)
Surgery	20 (20.6)	773 (17.0)
ICU	8 (8.2)	333 (7.3)
other	1 (1.0)	121 (2.6)
Suspected source of infection
Intra-abdominal	19 (19.6)	593 (13.0)
Other	12 (12.4)	1615 (35.5)
Other 3GC-R positive cultures	51 (52.6)	181 (4.0)
Of which include urine culture	44 (45.4)	101 (2.2)
Cultured pathogens
Escherichia coli	66 (68.0)	268 (5.9)
Klebsiella spp.	16 (16.5)	54 (1.2)
Other Enterobacterales	15 (15.5)	39 (0.9)
No growth	…	3680 (80.8)
Staphylococcus aureus	…	98 (2.2)
Other gram-positive	…	300 (6.6)
Other species	…	95 (2.1)
Nonfermenters	…	24 (0.5)
Empiric antibiotics
Amoxicillin	0 (0.0)	112 (2.5)
Co-amoxiclav	7 (7.2)	767 (16.8)
First-generation cephalosporins	0 (0.0)	13 (0.3)
Second-generation cephalosporins	1 (1.0)	306 (6.7)
Third-generation cephalosporins	30 (30.9)	1736 (38.1)
Fourth/fifth-generation cephalosporins	2 (2.1)	28 (0.6)
Piperacillin/tazobactam	26 (26.8)	902 (19.8)
Fluoroquinolones	12 (12.4)	592 (13.0)
Aminoglycosides	4 (4.1)	187 (4.1)
Carbapenems	28 (28.9)	398 (8.7)
Sulfamethoxazol/trimethoprim	0 (0.0)	35 (0.8)

Data are presented as n (%) unless otherwise indicated.

Abbreviations: BSI, bloodstream infection; ICU, intensive care unit; 3GC-R, third-generation cephalosporin-resistant.

Patients with 3GC-R-BSI were older, more frequently colonized with 3GC-R EB in the prior year (27.8% vs 5.6%), more often used antibiotics in the 2 months prior to infection (67.0% vs 37.6%), and more often had a clinical suspicion of urinary tract infection (53.6% vs 20.5%) and less often had a suspected respiratory tract infection than comparators (14.4% vs 35.5%) (Table 1). (See Supplementary Material D for a comparison with the derivation study and Supplementary Material E for a baseline table per country.)

Third-generation cephalosporins were the most frequently prescribed antibiotics (in 30.9% and 38.1% of patients with and without 3GC-R-BSI, respectively). In total, 426 patients (9.2%) received carbapenems, which included 28 of the 97 patients with 3GC-R-BSI, as did 398 of 4553 patients without 3GC-R-BSI. Consequently, undertreatment occurred in 71.1% of the patients with 3GC-R-BSI and overtreatment in 8.7% of the patients without 3GC-R EB BSI.

Among patients who did not have 3GC-R-BSI, 181 (4.0%) had 3GC-R EB isolated from samples other than blood cultures, mainly from urine cultures (n = 101; 56%). Therefore, in analysis, 2278 patients were categorized as 3GC-R-GNI. In this analysis, carbapenems were prescribed to 65 of 278 patients with (23.4%) and to 361 of 4372 patients without (8.2%) 3GC-R-GNI.

Model Performance

The community-onset prediction rule had good discrimination, with a c-statistic of .79 (95% CI, .75–.83). Regression models are shown in Supplementary Material F. The calibration plot of the original model shows structural underprediction (Figure 2A). After recalibration of the intercept, thereby updating the model to reflect the higher incidence of 3GC-R-BSI in the validation cohort, calibration was improved (Figure 2B). (See Supplementary Material F for further recalibration steps, receiver operating characteristic curves, and precision-recall curves.)

Figure 2.

Calibration plots: community-onset model (A); original model (B). Recalibration of the intercept. Systematic underprediction occurs in the original model, which disappears after updating the intercept to account for the higher incidence of 3GC-R-BSI in the validation cohort. Abbreviation: 3GC-R-BSI, third-generation cephalosporin-resistant bloodstream infection.

Clinical Utility

At a 5% 3GC-R-BSI risk cutoff, 6.8% of all patients in the cohort would be “test-positive” for 3GC-R-BSI, and thus eligible for empiric treatment with a carbapenem (Table 3). At this cutoff, the prediction rule has a sensitivity of 28.9% and a positive-predictive value of 8.9%. If all patients would be treated accordingly, 28 patients with 3GC-R-BSI would be empirically treated with carbapenems, similar to what is observed, but it would reduce carbapenem in non–3GC-R-BSI patients from 398 to 287 (6.3%, 28% reduction of overtreatment). With a higher cutoff, 6.6%, overtreatment would be reduced by 53% (from 398 to 190) at the expense of the number of patients with 3GC-R-BSI appropriately receiving carbapenems being reduced from 28 to 25 patients. For other cutoffs, see Supplementary Table G.

Table 3.

Clinical Impact on Carbapenem Use, Community Onset

3GC-R-BSI as Outcome					3GC-R-GNI as Outcome
Cutoff	4.3%	5%	6.6%	Current Cohort	4.3%	5%	6.6%	Current Cohort
Proportion of cohort, %	9.1	6.8	4.6	…	9.1	6.8	4.6	…
Sensitivity, %	38.1	28.9	25.8	…	35.0	29.2	24.8	…
Specificity, %	91.6	93.7	95.8	…	92.6	94.7	96.7	…
PPV, %	8.8	8.9	11.6	…	23.2	26.0	33.4	…
NPV, %	98.6	98.4	98.4	…	95.6	95.4	95.3	…
Appropriate carbapenems, n/N (%)	37/97 (38)	28/97 (29)	25/97 (26)	28/97 (29)	98/278 (35)	82/278 (29)	72/278 (26)	65/278 (24)
Potential appropriate 3GC-R-EB  coveragea	…	…	…	56/97 (57.8)	…	…	…	140/278 (50.4)
Unnecessary carbapenems, n/N (%)	384/4553 (8.4)	287/4553 (6.3)	190/4553 (4.2)	398 (8.7)	323/4372 (7.4)	233/4372 (5.3)	143 /4372 (3.2)	361 (8.2)

The cutoff reflects the predicted 3GC-R-BSI risk for an individual patient. For instance, at the 5% cutoff, 6.8% of all patients in the cohort have a risk of at least 5% for 3GC-R-BSI (proportion of cohort). At this cutoff the prediction rule has a sensitivity of 28.9% and a PPV of 8.9%. If all patients would be treated accordingly, 28 patients with 3GC-R-BSI would be empirically treated with carbapenems (which resembles the observed number of patients with 3GC-R-BSI who received carbapenems—thus, a 0% change). Another 287 patients without 3GC-R-BSI would also receive carbapenem, which reflects a 28% relative reduction (from 8.7% to 6.3%). Increasing the cutoff to 6.6% reduces both the number of appropriately treated patients (11% reduction), but also the number with unnecessary carbapenem use (53% reduction).

On the right, the analysis repeated with all 3GC-R-GNIs as outcome. The net carbapenem use is the same, but when considering a carbapenem appropriate for a 3GC-R–positive clinical culture, more patients are treated appropriately.

Abbreviations: NPV, negative-predictive value; PPV, positive-predictive value; 3GC-R, third-generation cephalosporin-resistant; 3GC-R-BSI, third-generation cephalosporin-resistant bloodstream infection; 3GC-R-GNI, third-generation cephalosporin-resistant gram-negative infection.

aIncludes cefepime, piperacillin/tazobactam, colistin, and aminoglycosides. Of note, resistance data on these antibiotics have not been collected; thus, they may inappropriate.

When using the 5% cutoff from the previous paragraph in analysis 2 (using all 3GC-R-GNI episodes as outcome), the proportion of patients with 3GC-R-GNI appropriately treated with carbapenems would increase from 24% to 29%, whereas overtreatment would be reduced by 35.4% (from 8.2% to 5.3%) (Table 3). (See Supplementary Material H for sensitivity analysis of low and high 3GC-R-BSI incidence countries.)

Hospital-onset Cohort

Among the 1683 hospital-onset infections there were 83 3GC-R-BSI episodes (4.9%), of which 35 with pathogens co-resistant to carbapenems (2.1%; 42% of BSIs caused by Enterobacterales). In the remaining 1600 episodes blood cultures were either negative (n = 1112) or yielded other pathogens (n = 488). Patients with 3GC-R-BSI more often had renal disease, malignancy, central catheter, and signs of hypoperfusion and were more often colonized with 3GC-R EB (43.4% vs 12.5%) than the comparators.

The most frequently prescribed antibiotic in the non–3GC-R-BSI group was piperacillin/tazobactam (31.4%). Carbapenems were the predominantly prescribed class in the 3GC-R-BSI group (n = 40; 48.2%), followed by piperacillin/tazobactam (25.3%) (Table 2).

Table 2.

Predictor and Baseline Variables of the Hospital-onset Cohort

	3GC-R EB BSI (n = 83)	Without 3GC-R-BSI (n = 1600)
Predictors
Suspected source of infection
Respiratory tract	7 (8.4)	391 (24.3)
Central venous catheter	50 (60.2)	691 (43.0)
Solid malignancy	26 (31.3)	417 (26.1)
Prior culture with 3GC-R EB	36 (43.4)	200 (12.5)
Prior cephalosporin use (<2 months)	39 (47.0)	451 (28.2)
Surgery in last month	40 (48.2)	498 (31.1)
Renal disease	22 (26.5)	270 (16.9)
Pre-infection length of stay in days (IQR)	17 (9–33.5)	11 (5–21)
Hypoperfusion	19 (22.9)	273 (17.1)
Other descriptive variables
Age, years	63.2 ± 16.0	64.6 ± 16.7
Male sex	52 (62.7)	967 (60.4)
Suspected source of infection
Intra-abdominal	43 (51.8)	278 (17.4)
Urinary tract	16 (19.3)	230 (14.3)
Other	17 (20.5)	709 (44.1)
Culture ward
Internal medicine	38 (45.8)	969 (60.6)
Surgery	27 (32.5)	393 (24.6)
ICU	17 (20.5)	251 (15.7)
Carbapenem resistance	35 (42.2)	…
Other 3GC-R EB culture	24 (28.9)	116 (7.3)
Of which include urine	17 (20.5)	69 (4.3)
Cultured pathogens
Escherichia coli	28 (33.7)	66 (4.1)
Klebsiella spp.	45 (54.2)	34 (2.1)
Other Enterobacterales	10 (12.0)	33 (2.1)
No growth	…	1133 (70.8)
Staphylococcus aureus	…	75 (4.7)
Other gram-positve	…	162 (10.1)
Other species	…	52 (3.2)
Nonfermenters	…	47 (2.9)
Empiric antibiotics
Amoxicillin/penicillin	0 (0.0)	6 (0.4)
Co-amoxiclav	3 (3.6)	143 (8.9)
First/second-generation cephalosporins	0 (0.0)	64 (4.0)
Third-generation cephalosporins	12 (14.5)	395 (24.7)
Fourth/fifth-generation cephalosporins	1 (1.2)	32 (2.0)
Piperacillin/tazobactam	21 (25.3)	502 (31.4)
Fluoroquinolones	6 (7.2)	138 (8.6)
Aminoglycosides	6 (7.2)	67 (4.2)
Carbapenems	40 (48.2)	363 (22.7)
Sulfamethoxazol/trimethoprim	1 (1.2)	22 (1.4)
Colistin, tigecycline, other	11 (13.3)	17 (1.1)

Data are presented as n (%) unless otherwise indicated.

Abbreviations: BSI, bloodstream infection; ICU, intensive care unit; IQR, interquartile range; 3GC-R, third-generation cephalosporin-resistant; 3GC-R-BSI, third-generation cephalosporin-resistant bloodstream infection.

The c-statistic of the hospital-onset prediction rule was .75 (95% CI, .70–.80). Regression models are shown in Supplementary Material F. Calibration of the original model was poor (Figure 3A), with major underprediction from risks between 0 and 0.2, and then overprediction. Recalibration of the intercept improved the calibration, but still had significant and inconsistent deviations from the ideal line, especially in predicted risks between 0% and 10%, involving the majority of patients (Figure 3B). This is reflected in limited clinical utility, with a reduction of 13% in unnecessary carbapenem use while keeping a similar rate of inappropriate therapy. (See Supplementary Table G for cutoff values.)

Figure 3.

Calibration plots of the hospital-onset original model (A), and recalibration in the large (B).

DISCUSSION

We externally validated 2 prediction rules for 3GC-R infections in patients with a clinical infection in whom empiric antibiotic treatment covering gram-negative bacteria was initiated. The rule for community-onset infections showed good discrimination and calibration and has the potential to safely reduce unnecessary carbapenem use. The hospital-onset rule had poor calibration, and its clinical utility was therefore limited.

The prediction rules illustrate the challenging trade-off between inappropriate therapy and overtreatment with broad- spectrum antibiotics. Prediction rules have been considered one of the options to mitigate the increase of broad-spectrum antibiotic prescriptions by aiding risk assessment of patients [9]. While other researchers previously developed prediction tools for Extended spectrum beta-lactamase (ESBL)-BSI/3GC-R-BSI [2–4, 10, 11], prediction rules have either not been validated or validation yielded poor performance [12]. Moreover, since these prediction rules (except for Fröding et al [11]) were derived from comparisons of 3GC-R-BSI with other gram-negative BSIs, they are unsuitable for aiding in initiation of empiric treatment. Most of these studies do not report on calibration, arguably a more important performance metric than discrimination for clinical prediction rules, since accurately predicted risks are the basis for classifying patients in treatment groups [13, 14]. We consider this international validation an important step forward in using prediction rules to improve antibiotic use.

Several predictors, such as prior isolation of 3GC-R isolates from diagnostic cultures and recent antibiotic use, were already included in guideline recommendations for selecting empiric therapy. However, we observed that 9% of community-onset infections were empirically treated with carbapenems, while only 2.1% of all infection episodes involved 3GC-R-BSI and 6.0% involved 3GC-R infections [15]. Reducing the eligibility for carbapenems therefore seems to offer the biggest advantage of the prediction rule. Importantly, these improvements are extended to other infections caused by 3GC-R-EB (eg, blood culture negative with positive urine culture) when used in patients with suspected BSI (the study population).

The community-onset prediction rule was generalizable from a low-resistance country (the Netherlands) to a validation cohort with a 5 times higher incidence of 3GC-R-BSI. This is explained by a similar distribution of individual predictors compared with the derivation cohort (see Supplementary Material for direct comparisons and predictor distribution per country). The c-statistic of .79 was very similar to that of .78 (95% CI, .71–.84) in the derivation study. Prior isolation of 3GC-R EB from samples in individual patients was not more prevalent in countries with a higher prevalence of 3GC-R-BSI, implying it serves as a proxy of prior healthcare exposure and a higher risk of infection with resistant bacteria.

The prediction rule for hospital-onset infections performed markedly worse than the community-onset infection rule, primarily in calibration and impact on eligibility for carbapenem use. There may be several reasons for this difference. First, the predictors among hospitalized patients are less likely to be universal than in community-dwelling patients. For example, length of stay depends on organization of healthcare, and median length of stay prior to infection onset per country varied from 6 to 20 days, without a concurrent increase in the incidence of 3GC-R-BSI. Second, renal disease was a major predictor in the derivation study, but not in this validation study. Third, although we used the same definitions for the predictors in the current validation cohort, the hospital-onset rule had more context-dependent predictors and interrater variability may have had a larger impact on model performance.

Implementation of the community-onset prediction rule requires consideration of how clinicians will incorporate the rule in clinical practice. It can give an absolute predicted risk to aid decision making or give a treatment recommendation based on predefined cutoff values. This could mean different cutoffs for patients with and without septic shock. The rule should be adapted to account for local incidences of 3GC-R-BSI by updating the intercept of the model. Ideally, a prediction rule would be implemented in the electronic patient management system. To what extent real-world use of this prediction rule changes the appropriateness of empiric antibiotic therapy requires a diagnostic trial [16]. Furthermore, cultural and sociological aspects (eg, risk-averseness) often limit implementation in different situations and should be considered, preferably by involving experts in these fields.

Several study limitations are important to mention. First, data collection was intended to be prospective, but this was unfeasible in some centers; approximately 50% of patients were included retrospectively. Retrospective data entry occurred more frequently in countries with a higher prevalence of resistance. Overall, the distribution of predictors was similar among prospectively and retrospectively collected data, and there were no differences in incidences of 3GC-R-BSI between centers with prospective and retrospective data collection from the same region. Finally, time logging allowed us to determine whether the outcome eCRF (with culture results) had been completed before the predictor eCRF and this occurred in 1% of the cohort. We therefore consider the impact of retrospectively collected data on the study validity to be limited. Additionally, data collection was based on chart review, which may have resulted in more missing or unknown variables than when directly asking the patient when including them in the study. However, we think the chart review approach more accurately reflects clinical practice, in which patients often do not know what happened and clinicians do not have the time to elaborately investigate prior antibiotic use or other parts of the medical history.

Second, we simplified the clinical problem to 3GC-R in Enterobacterales, while clinicians also may have to consider other potential pathogens, like Pseudomonas aeruginosa and methicillin-resistant Staphylococcus aureus [17]. However, these pathogens were rare in community-onset infections in our cohort.

Third, we only collected susceptibility data of EB isolates for cephalosporins and carbapenems. In the community-onset group, 30 patients with 3GC-R-BSI received piperacillin/tazobactam or cephalosporins plus aminoglycosides for empiric treatment. Some of these patients may have received appropriate treatment. However, aminoglycosides may not be an ideal strategy for empiric treatment [18]. Additionally, co-resistance to aminoglycosides is common in ESBL-producing pathogens, particularly in Klebsiella species [19].

Fourth, we included patients with a respiratory tract infection, where the a priori risk of bacteremia is low. However, 14% of the 3GC-R-BSI population had a suspected respiratory tract infection according to local physicians. This high percentage may have resulted from the selection of more severe cases in patients with community-onset pneumonia. In these patients, obtaining blood cultures is not always standard care and may be associated with higher disease severity, and thus a higher a priori risk of 3GC-R-BSI [20].

In conclusion, we externally validated a prediction rule in community-onset infections caused by 3GC-R E that may reduce unnecessary broad-spectrum antibiotic prescriptions, which can now be implemented and tested for clinical utility.

Supplementary Data

Supplementary materials are available at Clinical Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.

Click here for additional data file.

Click here for additional data file.