External validity of a randomised controlled trial on the treatment of severe infections caused by MRSA.

OBJECTIVES: To assess the external validity of a pragmatic, investigator-initiated RCT on treatment of severe infections caused by methicillin-resistant Staphylococcus aureus (MRSA), we compared patient characteristics and treatment effect estimates for patients included in the RCT versus those excluded. PARTICIPANTS AND OUTCOMES: The RCT included hospitalised patients with documented or highly-probable invasive MRSA infections who were randomised to vancomycin versus trimethoprim-sulfamethoxazole (TMP-SMX) treatment, between 2007 and 2014. A concomitant observational study prospectively included all consecutive patients, between 2008 and 2011, who were excluded from the RCT due to no consent, meningitis, left-sided endocarditis, severe neutropaenia, chronic renal dialysis or treatment with study medications for longer than 48 h. The primary outcomes were clinical failure at day 7 and 30-day mortality for both studies. We compared baseline and infection characteristics, outcome rates and treatment effect estimates for included versus excluded patients.
RESULTS: The RCT included 252 patients who were compared with 220 excluded patients who were observed. Inability to provide informed consent was the main reason for patient exclusion. Excluded patients' functional and cognitive performance was significantly poorer than that of included patients. Sepsis was more severe among excluded patients (higher rates of mechanical ventilation, indwelling catheters, septic shock and organ failure). Clinical failure occurred in 83/252 (32.9%) versus 175/220 (79.5%) and deaths in 32 (12.7%) versus 64 (29.1%) for included versus excluded patients, p<0.001 for both comparisons. Comparing vancomycin to TMP-SMX, in the RCT mortality, was non-significantly lower with vancomycin (OR 0.76, 95% CIs 0.36 to 1.62), while in the observational analysis of excluded patients, mortality was significantly higher with vancomycin (OR 2.63, 1.04 to 6.65), p=0.04 for the difference.
CONCLUSIONS: Patient characteristics, outcome event rates and treatment effects differed significantly in the setting of a RCT, despite its pragmatic design, compared to patients treated outside the trial settings. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

RCT Entities:   Population Interventions Outcomes

Identical data prospectively collected for patients included and excluded from a randomised controlled trial (RCT) allowed the analysis of comparative baseline patient characteristics and outcomes.

There were very few medical exclusion criteria from the RCT, as this was a pragmatic trial targeting all hospitalised patients treated for methicillin-resistant Staphylococcus aureus infections in clinical practice.

Results are affected by local clinical practice in the observational analysis.

National informed consent regulations vary, thus conclusions might not be applicable globally.

Background

Randomised controlled trials (RCTs), if conducted rigorously, allow for unbiased treatment comparisons. Internal validity is defined as a study's ability to correctly establish that the intervention caused the variation in the outcome. External validity is defined as the extent to which the results of a study can be generalised beyond the study sample. Issues that may affect the external validity of a trial include the setting, patient selection and differences between the trial interventions or protocol and clinical practice.1 While much attention has been drawn to the assessment and reporting of the internal validity of trials, less attention is given to their external validity. Thus, RCTs are routinely appraised for their internal validity, but not for their external validity.

Few studies have reviewed the exclusion criteria of RCTs in infectious diseases, inferring poor external validity as regards patient selection.2 3 A study focusing on outcome incidence showed significantly different incidence rates of ventilator-associated pneumonia (VAP) in observational studies versus control arms of RCTs on antimicrobial-based methods of VAP prevention.4 We showed that patients included in RCTs on the treatment of community-acquired pneumonia (CAP) are significantly younger than patients included in observational studies.5 We are not aware of studies comparing the characteristics of patients included in RCTs versus those treated in clinical practice or treatment effect estimates in RCTs versus analyses based on observational data, in infectious diseases.

We compared baseline characteristics, outcomes and treatment effect estimates of patients included in a RCT6 comparing trimethoprim-sulfamethoxazole (TMP-SMX) versus vancomycin for the treatment of invasive MRSA infections versus those fulfilling medical inclusion criteria but not included in the RCT. The RCT was investigator-initiated and eligibility criteria were designed to reflect, as closely as possible, the population of patients treated with glycopeptides in clinical practice.

Methods

The RCT methods have been previously described.6 Briefly, this was an open-label RCT conducted in four medical centres in Israel, between July 2007 and April 2014. hospitalised patients were included with microbiologically-documented MRSA infections or highly probable MRSA infections, as previously described.6 We excluded patients with meningitis, left-sided endocarditis, neutropaenic patients with acute leucaemia or undergoing stem-cell transplantation, patients on chronic renal dialysis and previously included patients. All clinical exclusion criteria were based on conditions that we were not confident of vis-a-vis the effectiveness or safety of TMP-SMX7–9 and that we opted to exclude considering clinical equipoise in trial settings. We did not define exclusion criteria based on other comorbidities, cognitive or functional status, sepsis severity or target organ dysfunction. Further excluded from the RCT were patients who did not or could not provide informed consent and lacked a consenting legal guardian; patients treated with study medications for longer than 48 h prior to recruitment; patients with a staphylococcal infection resistant to cotrimoxazole or vancomycin; pregnant or lactating women; and those participating in another trial. Patients were randomised to treatment with TMP-SMX versus vancomycin and the primary outcomes were treatment failure at day 7 and 30-day mortality. Failure was a composite of death, persistent sepsis or organ failure, haemodynamic instability and microbiological failure.6

Between June 2008 and July 2011, patients excluded from the trial were included in a prospective observational study. The observational study included all consecutive patients in one of the study centres (recruiting 76% of the patients in the RCT) that fulfilled the RCT infection inclusion criteria (documented or highly-probably MRSA infections) but were not included, without treatment limitations. Thus, the observational study included patients who had one or more of the RCT exclusion criteria, could not provide or refused informed consent. Patients were identified at the screening visit of the RCT and were treated according to physicians’ discretion. The same data were collected in the observational study as were collected in the RCT, including baseline patient characteristics, sepsis presentation, infection characteristics and outcomes, except that missing data could not be completed and data were recorded anonymously. The observational study's sample size was calculated to enable the detection of 20% absolute difference in the primary outcome rate between the observation cohort and RCT (50% vs 30% success rate estimated, power >90%, α=0.05).

Ethics committee approval was obtained for both studies, with waiver of informed consent in the observational study. We compared patient characteristics, infection characteristics and outcomes for patients included in the RCT versus those excluded. Categorical data were compared using a χ2 test. A Kolmogorov-Smirnov test was carried out in order to determine whether the distributions of continuous variables were normal. Normally distributed values were described as means with SDs and compared using the t test. Skewed variables were described as medians with 25–75% IQR and compared using the Mann-Whitney U test. Unadjusted ORs for the comparison between TMP-SMX and vancomycin in each cohort were computed with 95% CIs and compared using the Breslow-Day test. Analyses were performed using SPSS V.21.

Results

The RCT included 252 patients and the observational study, 220 patients. The most common reasons for exclusion from the RCT were inability or refusal to provide informed consent and treatment with study drugs for longer than 48 h, together accounting for more than 70% of exclusions, table 1.

Table 1

Reason for exclusion of patients from the RCT (and inclusion in the observational study)

Reason for exclusion	N (%)(Total N=220)
Treatment with study drugs >48 h prior to identification	73 (33.2%)
Refusal to sign an informed consent	44 (20%)
Inability to provide informed consent and no legal guardian	40 (18.2%)
Chronic dialysis	29 (13.2%)
Resistance to one of the study antibiotics	14 (6.4%)
Left-side endocarditis	8 (3.6%)
Acute leucaemia with neutropaenia	7 (3.2%)
Hypersensitivity to one of the antibiotics in the trial	2 (0.9%)
Meningitis	2 (0.9%)
Participation in other trial	1 (0.5%)

RCT, randomised controlled trial.

Baseline patient characteristics

Excluded patients differed from included patients in their functional and cognitive status, conditions affecting the ability to provide informed consent, including past cerebrovascular events (table 2). As expected from the exclusion criteria of the RCT, patients with chronic renal failure were significantly less represented in the RCT and malignancy was slightly less common. Otherwise, there were no significant differences regarding baseline comorbidities. The total Charlson score was significantly higher among excluded patients.

Table 2

Baseline patient characteristics

	RCT includedN=252	ExcludedN=220	p Value
Age, years (mean±SD)	65.8±17	67.9±17.2	0.192
Female sex	86 (34.1%)	90 (40.9%)	0.129
Admission from home	194 (77%)	145 (65.9%)	0.008
Functional capacity—bedridden	53 (21%)	115 (52.3%)	<0.001
Dementia	12 (4.8%)	41 (18.6%)	<0.001
Congestive heart failure	50 (19.8%)	40 (18.2%)	0.647
Ischaemic heart disease	80 (31.7%)	63 (28.6%)	0.463
Cerebrovascular accident in the past	44 (17.5%)	57 (25.9%)	0.026
Chronic lung disease	35 (13.9%)	27 (12.3%)	0.604
Diabetes mellitus	102 (40.5%)	88 (40%)	0.916
Chronic renal failure	6 (2.4%)	39 (17.7%)	<0.001
Manifest malignancy	49 (19.4%)	58 (26.4%)	0.073
McCabe score—no fatal disease	196 (77.8%)	166 (75.5%)	0.551
Charlson score (median, percentile)	2 (1–4)	3 (2–4)	0.008

RCT, randomised controlled trial.

Infection characteristics and management

Differences in infection characteristics reflected the inability to recruit critically-ill patients at baseline and those with severe sepsis. Thus, excluded patients were significantly more ventilated and carried more catheters than those included in the RCT (table 3). They had more septic shock at onset and a higher Sequential Organ Failure Assessment score. This led to more patients with skin, soft tissue, bone and joint infections in the RCT (with a higher rate of bone implants and prior surgery) and less CVC-related or primary infections. While bacteraemia occurrence was similar among included and excluded patients, there were more patients with highly probably non-microbiologically documented MRSA infections among the excluded patients.

Table 3

Infection characteristics and management*

	RCT includedN=252	ExcludedN=220	p Value
Predisposition
Hospital-acquired infection†	173 (68.7%)	138 (62.7%)	0.176
Nasogastric tube prior to infection	26 (10.3%)	80 (36.4%)	<0.001
Urine catheter prior to infection	80 (31.7%)	138 (62.7%)	<0.001
Central venous catheter prior to infection	32 (12.7%)	104 (47.2%)	<0.001
Foreign body prior to infection‡	84 (33.3%)	26 (11.8%)	<0.001
Surgery 30 days prior to infection	121 (48%)	77 (35%)	0.004
Mechanical ventilation at onset	27 (10.7%)	98 (44.5%)	<0.001
Infection characteristics and presentation
Bacteraemia	91 (36.1%)	91 (41.4%)	0.242
Any microbiologically (MRSA)-documented infection	245 (97.2%)	167 (75.9%)	<0.001
Source of infection			<0.001
Central venous catheter-related	16 (6.3%)	53 (24.1%)
Other endovascular	9 (3.6%)	9 (4.1%)
Pneumonia	27 (10.7%)	30 (13.6%)
Skin, soft tissue, bone or joint	168 (66.7%)	54 (24.5%)
Other documented source	17 (6.7%)	4 (1.8%)
Primary, unknown source	15 (6%)	70 (31.8%)
Leucocytes, k/mL‡ (median, IQR)	9.7 (7.4–13.8), N=251	10.9 (7.4–15.1)	0.358
Haematocrit, % (median, IQR)	31.3 (27.5–34.6), N=251	29.5 (26.3–33.65)	0.006
Platelets, k/mL‡ (median, IQR)	279 (192–403), N=251	218 (123–331)	<0.001
Urea, mg/dL (median, IQR)	38 (25–63), N=251	55 (34–101)	<0.001
Albumin, g/dL (median, IQR)	2.8 (2.4–3.3), N=250	2.7 (2.1–3.3), N=183	0.059
Septic shock at onset	6 (2.4%)	23 (10.5%)	<0.001
SOFA score at onset (median, IQR)	2 (1–4)	3 (2–4)	<0.001
Infection management
Antibiotic treatment			<0.001
Vancomycin	117 (46.4%)	167 (75.9%)
TMP-SMX	135 (53.6%)	39 (17.7%)
Other§	0	14 (6.4%)
Vancomycin levels measured¶	97/117 (82.9%)	95/167 (56.9%)	<0.001
Mean vancomycin trough levels (median, IQR)¶	14.9 (10.4–21), N=97	14 (11–22.8), N=95	0.778
Vancomycin trough levels >10 mg/dL attained	80/97 (82.5%)	79/95 (83.2%)	0.9
Total treatment duration, days (median, IQR)**	15 (11–28)	11 (5–18)	<0.001
Treatment duration in 30-day survivors, days (median, IQR)**	17 (12–30), N=220	12 (7–22) N=156	<0.001

*Numbers apply to all patients in the group unless stated otherwise.

†All other infections were healthcare-associated.

‡Foreign bodies included prosthetic joints or implants, vascular prostheses, prosthetic valve.

§Clindamycin seven patients, non-covering antibiotics six patients and chloramphenicol one patient. Vancomycin and TMP-SMX were covering in all cases.

¶Of patients treated with vancomycin.

**Total duration of antibiotic therapy covering the MRSA isolate up to 30 days’ follow-up. Excluded patients not receiving covering treatment coded as 0 days.

MRSA, methicillin-resistant Staphylococcus aureus; RCT, randomised controlled trial; SOFA, Sequential Organ Failure Assessment score; TMP-SMX, trimethoprim-sulfamethoxazole.

While patients in the RCT were randomised to vancomycin versus TMP-SMX, patients in the observational study were mostly treated with vancomycin (167/220, 75.9%), reflecting the standard of care for MRSA infections in the study centre. Of those treated with vancomycin, levels were measured significantly more frequently for patients included in the RCT (97/117, 82.9% among included vs 95/167, 56.9% among excluded, p<0.001). When measured, vancomycin mean levels and attainment of therapeutic trough levels were similar. Antibiotic treatment duration was significantly longer among patients included in the RCT, both among all patients and among 30-day survivors (table 3).

Outcomes and treatment effects

There was a very large difference in outcome events rates between the cohorts. Clinical failure was documented in 83/252 (32.9%) patients in the RCT versus 175/220 (79.5%) among excluded patients (OR 7.94, 95% CI 5.21 to 12.05, p<0.001). The failure rate was 33/40 (82.5%) among patients unable to provide informed consent, 34/44 (77.3%) among patients refusing to provide informed consent and 108/136 (79.4%) among all other excluded patients. The differences between each subgroup and RCT were significant (p<0.001), while differences between subgroups of excluded patients were not statistically significant. All-cause mortality at 30 days occurred in 32 (12.7%) patients in the RCT versus 64 (29.1%) excluded patients (OR 2.82, 95% CI 1.76 to 4.52, p<001). Mortality rates were, relatively, 23/40 (57.5%), 9/44 (20.5%) and 32/136 (23.5%) among patients unable to provide consent, those refusing to provide informed consent and others who were excluded (p<0.001 for patients unable to provide informed consent vs both excluded subgroups).

Moreover, treatment effects for vancomycin versus TMP-SMX differed, reaching a statistically significant difference for mortality. In the RCT, mortality was non-significantly lower with vancomycin, while in the observational study, mortality was significantly lower with TMP-SMX (p=0.04 for the difference between treatment effect in the RCT vs observational cohort), table 4. Restricting the analysis to the one centre conducting the observational study did not alter any results (data not shown).

Table 4

Treatment effects

	RCT includedN=252		ExcludedN=220		p Value*
	Vancomycin	TMP-SMX	Vancomycin	TMP-SMX
Clinical failure	32/117 (27.4%)	51/135 (37.8%)	137/167 (82%)	31/39 (79.5%)
	OR 0.62 (0.36–1.06)		OR 1.18 (0.49–2.82)		0.216
30-day all-cause mortality	13/117 (11.1%)	19/135 (14.1%)	54/167 (32.3%)	6/39 (15.4%)
	0.76 (0.36–1.62)		2.63 (1.04–6.65)		0.04

*p Values comparing the ORs of vancomycin versus TMP-SMX among included and excluded patients.

RCT, randomised controlled trial; TMP-SMX, trimethoprim-sulfamethoxazole.

Discussion

We found large differences between patients included in an investigator-initiated RCT comparing TMP-SMX versus vancomycin for the treatment of invasive MRSA infections and those excluded. The RCT was specifically designed to reflect the patient population seen in clinical practice as closely as possible. Exclusion criteria frequently listed in RCTs in infectious diseases (eg, renal failure, septic shock) were not used for this study. Using a previous classification of exclusion criteria in RCTs,10 there were no poorly justified exclusion criteria in the RCT. Yet, excluded patients had significantly more comorbidities and more severe infections than those included in the RCT. Included and excluded populations also differed with respect to the types of infections, as we encountered difficulty in recruiting patients empirically (for highly probable MRSA infections) into the RCT. Patient management might have been better within the RCT, as reflected by covering therapy for all patients, vancomycin level monitoring and longer treatment duration. Failure and mortality rates were significantly lower in the RCT. Interestingly, this was observed also in patients able to provide informed consent but refusing. Moreover, TMP-SMX was superior to vancomycin in the observational study but not in the RCT, reflecting selection bias in the observational study due to clinicians’ tendency to prescribe TMP-SMX to less ill patients.

Unlike the extensive literature published on the internal validity of RCTs, very little has been described regarding external validity of RCTs in general, and specifically regarding infectious diseases. External validity of trials in infectious diseases has several unique dimensions. External validity is affected by the very fact that it is difficult to recruit patients with the severest infections into RCTs due to the systemic and cognitive effects of severe sepsis resulting in inability to provide informed consent or refusal. This is a differential bias, as certain patient subgroups, such as the elderly, are more strongly affected. Industry-initiated RCTs have many further exclusion criteria that dissociate patients assessed in RCTs versus those treated in practice. In a review of RCTs comparing antibiotics, frequent exclusion criteria included immunosuppression, many comorbidities, renal and liver failure, use of concomitant medications and many more.3 Our RCT attempted maximal external validity regarding patients and interventions, but achieved poor similarity to all patients cared for in clinical practice.

The major impediment to patient recruitment in our trial was the need for informed consent. We compared treatment regimens used alternatively in our centre in clinical practice. Patients were baffled by our request for informed consent at a time of severe distress (bacteraemia or other invasive infection) when we explained that in any case they were to receive either of the trial treatments. The most common reason for consent refusal was ‘who am I to decide’ or ‘I trust you to decide’, and such refusal was more common in the sicker patients. Furthermore, as expected, we were unsuccessful in recruiting many patients who could not provide informed consent due to poor cognition or poor functional status, and critically-ill patients; conditions frequently triggered by the acute infection, per se. In our trial, surrogates were less likely to provide informed consent than patients (data not presented). In our view, an appropriate balance between obtaining evidence on standard treatment effectiveness and safety for the most vulnerable patients versus patients’ rights and autonomy has yet to be achieved. A case has been made for relinquishing informed consent in comparative effectiveness research.11 The conditions proposed by Faden et al, where such a relaxation might be permissible, included the testing of commonly used interventions where practice varies and patients in the clinical setting are not consulted before treatment selection; settings with rigorous oversight of the trial and inclusion of laypersons in the decision to waive informed consent; and when there is a societal understanding that academic centres conduct trials for care improvement. Delayed consent was encouraged for an intervention where attempting to obtain patients’ consent before the intervention would have increased the risk of harming them by delaying treatment and could also have affected recruitment.12 This situation is true for most acute infections, as delaying antibiotic treatment is associated with mortality.13 Two recent large surveys in the USA showed that a significant percentage of people are agreeable to procedures other than written informed consent, such as a general notification or verbal consent for pragmatic RCTs (28.5% to 39.7% in one study14 and 47.6% to 52.8% in another15). The scenarios presented to the public concerned interventions for chronic conditions (hypertension). The public's attitudes towards informed consent in effectiveness research with acute infections and research conducted outside the USA should be evaluated.

In summary, patients recruited into an investigator-initiated trial on the treatment of invasive MRSA infections and outcome event rates were significantly different from patients not included and treated in usual clinical practice. Treatment effects shown in the RCT6 were different from those estimated in an observational analysis of this cohort and a previous cohort in one of the study centres,16 attesting to the importance of assessing treatment effectiveness in RCTs. This occurred despite the trial having minimal exclusion criteria. This raises a question on the generalisability of evidence garnered from RCTs in infectious diseases. We need a formal assessment of the external validity of RCTs in general, and these will have to be tailored to trials in infectious diseases because of the unique considerations in the external validity of these studies. To improve the transparency of RCTs in infectious diseases, we suggest that RCTs in infectious diseases include a third arm of excluded patients. Comparison of baseline patient characteristics and outcomes will allow better appreciation of the external validity of the RCT regarding patients and infections.