Applicability of Vasopressor Trials in Adult Critical Care: A Prospective Multicentre Meta-Epidemiologic Cohort Study.

Objective: To assess the applicability of evidence from landmark randomized controlled trials (RCTs) of vasopressor treatment in critically ill adults. Study Design and Setting: This prospective, multi-center cohort study was conducted at five medical and surgical intensive care units at three tertiary care centers. Consecutive cases of newly initiated vasopressor treatment were included. The primary end point was the proportion of patients (≥18 years) who met the eligibility criteria of 25 RCTs of vasopressor therapy in critically ill adults included in the most recent Cochrane review. Multilevel Poisson regression was used to estimate the eligibility proportions with 95% confidence intervals for each trial. Secondary end points included the eligibility criteria that contributed most to trial ineligibility, and the relationship between eligibility proportions and (i) the Pragmatic-Explanatory Continuum Indicator Summary-2 (PRECIS-2) score, and (ii) the recruitment-to-screening ratio of each RCT. The PRECIS-2 score was used to assess the degree of pragmatism of each trial.
Results: Between January 1, 2017, and January 1, 2019, a total of 1189 cases of newly initiated vasopressor therapy were included. The median proportion of cases meeting eligibility criteria for all 25 RCTs ranged from 1.3% to 6.0%. The eligibility criteria contributing most to trial ineligibility were the exceedance of a specific norepinephrine dose, the presence of a particular shock type, and the drop below a particular blood pressure value. Eligibility proportions increased with the PRECIS-2 score but not with the recruitment-to-screening ratio of the trials.
Conclusion: The applicability of evidence from available trials on vasopressor treatment in critically ill adults to patients receiving vasopressors in daily practice is limited. Applicability increases with the degree of study pragmatism but is not reflected in a high recruitment-to-screening ratio. Our findings may help researchers design vasopressor trials and promote standardized assessment and reporting of the degree of pragmatism achieved.

Plain Language Summary

Why was the study done?

Blood pressure-raising drugs are widely used in critical care. The evidence base underlying this practice has not been previously assessed.

What did the researchers do?

We prospectively enrolled 1189 consecutive cases of critically ill patients with newly initiated blood-pressure raising drug treatment in five intensive care units in Austria, Europe. We examined the extent to which study patients are represented by 25 landmark clinical trials on the use of blood-pressure raising drugs in intensive care. We assessed (1) the proportion of patients eligible for each trial, (2) the inclusion/exclusion criteria that contributed most to patients being ineligible for a trial, and (3) the relationship between eligibility of study patients and the design of a trial. A tool assessing nine areas of study design was used to evaluate the pragmatism of each trial.

What did the researchers find?

The percentage of eligible patients was low (between 1.3–6%), which was due to specific inclusion criteria of the trials, such as exceeding a certain dose of blood pressure-raising drugs or the presence of a certain type of shock. The percentage of eligible patients was higher when a trial had a pragmatic study design.

What do these results mean?

Few critically ill patients receiving blood pressure-raising medications in daily critical care are represented in the available landmark trials on this topic. Researchers should look for standardized assessment and reporting of how pragmatic a clinical trial is.

Introduction

Randomized controlled trials (RCTs) are the cornerstone of evidence-based medicine and the foundation of daily clinical practice.1 Ideally, clinical trial populations should represent the population of interest to gain broadly applicable results. Sample populations, however, are often restrictive and may reflect only a minority of individuals treated in daily practice, limiting the applicability of evidence obtained from clinical trials.

Vasopressors are often administered to critically ill patients with hypotensive shock who do not respond to volume resuscitation to increase mean arterial pressure (MAP) and organ perfusion.2 The available evidence for their broad application, however, has rarely been assessed. A hallmark trial investigating the effect of vasopressin versus noradrenaline on septic shock excluded almost 90% of a total of 6229 patients evaluated for trial eligibility due to extensive exclusion criteria.3 The criteria included patient characteristics frequently observed in daily practice, including chronic heart disease, hyponatremia <130 mmol/l, and a history of rheumatologic disorders or malignancy.3 Aside from adopting overly restrictive eligibility requirements, excluding patients from study participation may be based on the treating physician’s discretion, which may likewise cause selection bias.

The eventual generalizability and applicability of trial results depend on the study design employed. A pragmatic design broadens the applicability of trial results. Several tools, including the Pragmatic-Explanatory Continuum Indicator Summary-2 (PRECIS-2) score, have been developed to appraise a trial’s degree of pragmatism and help readers assess its applicability.4 Unfortunately, details on the level of pragmatism are scarcely published, making it difficult to determine a given trial’s value for clinical practice.

We hypothesized that the available evidence from RCTs on vasopressor treatment in critically ill adults applies only to a minority of patients receiving vasopressor therapy in daily practice. We tested this hypothesis by assessing the proportion of cases of newly initiated vasopressor treatment presenting at five mixed critical care units, which matched the eligibility criteria of 25 pivotal vasopressor RCTs included in a Cochrane review,5 and we assessed the PRECIS-2 score of each RCT.

Materials and Methods

Ethics Approval and Consent to Participate

The study was approved by the local ethics committees of each participating centre (Ethics Committee of the Medical University of Vienna: 1842/2016; Niederösterreich Ethikkommission [St. Pölten and Baden]: GS1-EK-4/463-2017) and conducted in accordance with the latest version of the Helsinki declaration. In view of the anonymized data collection and processing, the Ethics Committees waived the requirement to obtain individual informed consent in accordance with Austrian law.

Study Setting and Cohort

This multicentre prospective cohort study included all consecutive cases of newly initiated vasopressor treatment presenting over 12 months at three medical and two mixed medical/surgical critical care units of three tertiary care centres in Austria, Europe.

Inclusion criteria included patients for whom vasopressor treatment was initiated or for whom an established vasopressor was switched to another, allowing for more than one inclusion per patient if several vasopressors were administered. Vasopressors adopted included norepinephrine, epinephrine, phenylephrine, dopamine, vasopressin, and terlipressin alone or in combination with dobutamine.

In all recruiting centers, the local principal investigator manually reviewed medical records daily during the 12-month recruitment period. In centers with electronic medical records and electronic documentation of inpatient medications prescriptions (all centers except the Department of Emergency Medicine, Medical University of Vienna), electronic extraction was performed simultaneously. When patients received a vasopressor and thus met the eligibility criteria of the study, two of the investigators (N.B., R.L., J.M., F.K., and G.S.) extracted the variables of interest () from the electronic medical records. In case of discrepancies, a third investigator (M.S.) reviewed the medical records.

Reference Trials Included in the Cochrane Review and PRECIS-2 Score Assessment

Twenty-five RCTs included in the most recent update of a Cochrane review5 were used as a reference for available high-quality evidence on vasopressor treatment for hypotensive shock in critical care. The meta-analysis included 28 RCTs published until June 2015, involving 3497 critically ill patients with 1773 mortality outcomes. The analysis compared the use of six different vasopressors, including norepinephrine, epinephrine, phenylephrine, dopamine, vasopressin, and terlipressin, given alone or in combination with dobutamine or dopexamine.

The current study included 25 of these RCTs to assess trial eligibility.3,6–29 We excluded three trials from the analysis, including two paediatric trials30,31 and one trial available only in the Chinese language.32 A database was created that included the eligibility criteria of all reference RCTs and the data from case report forms collected for each vasopressor therapy initiated during the study period. Three authors (N.B., M.S., and C.S.) independently extracted the trials’ inclusion and exclusion criteria and assessed their PRECIS-2 scores and the eligibility proportions of the study cohort. Disagreements between assessors were resolved through arbitration by a fourth author (H.H.).

The PRECIS-2 score was applied to determine the pragmatic/explanatory level of a clinical trial study design. The assessment is based on nine domains, including eligibility, recruitment, setting, organization, flexibility: delivery, flexibility: adherence, follow-up, primary outcome, and primary analysis. To be classified as pragmatic, a trial should score ≥ 4 points in all 9 domains (ie, score 36–45 points overall).33 However, the domain flexibility: adherence assesses patient compliance in an outpatient setting and was therefore not applicable to the reference trials. Thus, we ranged the PRECIS-2 score of reference trials from 8 points, indicating a very explanatory trial, to 40 points, indicating a very pragmatic trial.

Statistical Methods

We present categorical data as total number (n) and relative frequency (%) and continuous data as median and interquartile range (IQR) or mean and standard deviation (SD), as indicated. The primary endpoint was the proportion of newly initiated vasopressor treatment cases matching the eligibility criteria of 25 RCTs for vasopressor treatment in adult critically ill patients included in a Cochrane review.

The unit of analysis was individual vasopressor treatment. Receiving more than one vasopressor during the study period was treated as multiple units of analysis. Allowing for the design-related three-level structure (cases nested in participants nested in centres), we used multilevel Poisson regression without covariates to estimate the eligibility proportions with 95% confidence intervals (95% CI) for each study separately.

Sample size calculation was precision based. Based on the numbers of patients included in a previous large RCT,3 we expected an eligibility proportion of 12%. We considered a sample size of 1000 cases sufficient to yield a 95% CI of 10–14%. We expected a recruitment period of one year adequate enough to include 1000 cases. A database comparing eligibility criteria of the 25 Cochrane RCTs and data of prospectively surveyed cases allowed us to generate a matrix comparing eligibility criteria and case characteristics. Using the matrix, we calculated the eligibility proportions of cases for each trial.

The secondary endpoints included the proportions of cases eligible for reference RCTs, the eligibility criteria contributing most to trial ineligibility and the relation of eligibility proportions to the number of eligibility criteria, the PRECIS-2 score, and the recruitment-to-screening ratios of each RCT. Associations between eligibility proportions, the PRECIS-2 score, and the recruitment-to-screening ratio were assessed using Spearman correlation with 95% CIs. Eligibility proportions of cases were plotted against the PRECIS-2 scores and the recruitment-to-screening ratios of the RCTs. Data analysis was performed using STATA Release 17 (Stata Corp, College Station, Texas) and Prism 9 for macOS Version 9.2.0.

Results

A total of 1189 cases in 744 individuals of newly initiated vasopressor treatment were included for trial eligibility assessment between January 1, 2017 and January 1, 2019. The inclusion period for each centre was 12 months. Sixty-three percent (n=468) of patients were male, with a median age of 68 years (IQR 56–76). The median systolic and MAP levels before vasopressor therapy were 100 mmHg (IQR 85–116) and mean (SD) 69 (24), respectively. The demographic and clinical characteristics of the study cohort at the time of inclusion are shown in the table below (Table 1).

Table 1

Characteristics of the Study Cohort

Variable
Demographics		Patients (n=744)
	Age, years, n=736	68 (56–76)
	Male sex, n (%), n=744	468 (63)
	BMI, kg/m2, n=664	26.12 (23.86–29.56)
Type of critical care		Cases (n=1112)
	Medical	846 (76)
	Surgical	266 (24)
	Emergency	180 (16)
	Elective	86 (8)
Indication for vasopressor treatment		Cases (n=1189)
Shock	Shock type
	Distributive	366 (31)
	Septic	341 (29)
	Anaphylactic	3 (0)
	Neurogenic	22 (2)
	Cardiogenic	376 (32)
	Obstructive	19 (2)
	Hypovolemic	36 (3)
	Intoxication	4 (0)
	Mixed or unknown	311 (26)
Surgery		77 (7)
Vasopressor		Cases (n=1202)*
	Norepinephrine	788 (66)
	Epinephrine	89 (7)
	Phenylephrine	33 (3)
	Dopamine	1 (0)
	Vasopressin	144 (12)
	Terlipressin	0 (0)
Inotropic agent	Dobutamine	147 (12)

Notes: *In 13 cases, two vasoactive agents were started simultaneously and therefore counted as single time points.

The median proportion of cases matching the eligibility criteria for all 25 RCTs was 2.5% (95% CI 1.3–6.0) (Figure 1). A total of 881 (74,1%) cases matched at least one RCT, 125 (10,51%) cases matched ≥5 RCTs, and 68 (5,72%) cases matched ≥10 RCTs. None of the cases matched ≥20 of the 25 RCTs. For 92% (n=23) of the RCTs, there were <20% (n<238) matching cases.

Figure 1

Proportions with 95% confidence intervals of matching study cases and proportions of cases eligible for reference RCTs.

The inclusion and exclusion criteria contributing most to the trial ineligibility of the study cases were the exceedance of a specific norepinephrine dose, the drop below a particular blood pressure value, or the presence of a specific shock type, which led to trial ineligibility on average in 77% (n=916), 67% (n=797) and 62% (n=737) of cases, respectively (Figure 2).

Figure 2

Eligibility criteria contributing most to the trial ineligibility of study cases.

The median number of eligibility criteria used in the 25 RCTs was 9 (IQR 8–11). There was no correlation between the proportions of matching cases and the number of eligibility criteria (r=0.06, 95% CI −0.35 to 0.46; p=0.77). The median PRECIS-2 score of the 25 RCTs was 29 points (IQR 27–31) (Figure 3A). A higher PRECIS-2 score was associated with higher trial eligibility (Figure 3B): The highest proportions of matching cases (49% and 48%) were found for the trials with the highest PRECIS-2 scores (35 points).

Figure 3

Median PRECIS-2 scores of the RCTs included in the Cochrane review and their relation to the eligibility proportions of the study cases.

Figure 4 shows the relation between the eligibility proportions of the study cases and the ratios of recruited to screened patient numbers reported by 13 of the 25 RCTs. The proportions of matching cases did not increase with a higher recruitment-to-screening ratio (r= 0.13, 95% CI −0.47 to 0.65; p= 0.67).

Figure 4

Eligibility proportions of study cases vs recruitment-to-screening ratios of 13 RCTs.

Discussion

Although vasopressors are among the most frequently used interventions in critical care, we found a substantial limitation in the applicability of evidence from available vasopressor RCTs in critically ill adults. The median eligibility proportion of study cases was 2.5%, and for 92% of all trials, less than 20% of all cases matched.

The effect of vasopressors on patient-relevant outcomes is still controversial. Recommendations for their use are mainly based on expert opinion, animal data, and studies that use surrogate endpoints as a proxy for mortality.5 No vasopressor RCT has yet conclusively proven a survival benefit of using one vasopressor over another despite apparent effects on haemodynamics.3,22 The current study suggests that the available evidence for vasopressor treatment applies to only a fraction of patients receiving vasopressors in daily practice.

The limited applicability of evidence to clinical practice likely applies to all medical research to some extent. In a previous study, more than 50% of all patients admitted to an ICU did not meet the enrolment criteria for any of 15 selected major RCTs in critical care, most often due to a lack of study-specific inclusion conditions.34 In the field of perioperative beta-blocker therapy, the applicability of evidence from trials may be likewise limited despite the availability of high-quality studies. We recently found that only 6.5% of 2241 patients assessed for trial eligibility matched the inclusion and exclusion criteria of landmark randomized trials evaluating perioperative ß-blocker therapy in noncardiac surgery.35 A commonly explanatory study design, indicated by an overall low median PRECIS-2 score, explained the observation.

The results of a clinical trial are most pragmatic when they can be applied to a large number of patients under real-world conditions.36 Diverse study populations and settings make trial results externally valid but may dilute effect estimates. Homogenous samples, on the other hand, increase internal validity and the likelihood of detecting a causal effect relationship between treatments and outcomes. However, overly restrictive eligibility criteria for clinical trial enrolment may preclude a substantial proportion of patients from trial participation.34 Assessing the applicability of evidence in a given medical field to a real population may help improve sample selection, safety, and efficacy when designing future trials.

In the context of the current study, Gamper et al encourage researchers to conduct large-scale vasopressor trials with a simple, pragmatic study design and based on patient-centred outcomes, including survival and long-term health-related quality of life.5 In particular, blood pressure values used as endpoints of vasopressor trials are identified as of concern because goals may vary individually, do not equal the reversal of the underlying pathology and are frequently not reached in trial settings of RCTs.37 We found specific blood pressure values to exclude more than two-thirds of all study cases from trial eligibility. The heterogeneity in clinical practice regarding shock definition and vasopressor initiation based on individual blood pressure values limits the broad applicability of trial results to individual centres or settings.

Shock type was identified as another major criterion causing trial ineligibility in our study cohort. The majority of the 25 RCTs included only patients with presumed septic shock. Sepsis is the most common cause of shock in ICUs.9,38 The local distribution of shock types, however, depends on the population served by a given institution. In our two medical and three mixed critical care study units, sepsis accounted for only one-third of all shock cases in addition to cardiogenic (31%) and cryptogenic shock (30%). Both latter shock types are poorly represented by the available evidence. A higher level of trial pragmatism may broaden the applicability of trial findings across various shock types. In this context, the median PRECIS-2 score of the 25 RCTs was 29 points, which may be interpreted as moderate level of pragmatism. This is comprehensible given the shortage of RCTs that have shown improved patient-centred outcomes with the use of vasopressors. A high level of pragmatism may reduce the effect size of an investigational intervention and the probability of demonstrating a beneficial effect.39 However, this allows for determining an intervention’s effectiveness in real settings. A recent example of the success of pragmatism used in large-scale clinical research is reflected by the RECOVERY trial.40 An easy-to-follow protocol and the use of hard endpoints available from data linkages rapidly increased the number of participating centres to more than 170, allowing for an impressive recruitment of more than 40,000 patients. Given its pragmatic study design, the results of the RECOVERY trial have been rapidly dovetailed in international guidelines and have changed daily practice in the treatment of COVID-19 patients. This example highlights the strength of pragmatism in clinical trials and the improvement of national and international networking capacities, which may be a paragon for future trials in the field of critical care research.

The representativeness and external validity of a given trial further depend on the population screened for inclusion (Consolidated Standards of Reporting Trials [CONSORT] 2010). Both STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) and CONSORT suggest a standardized reporting of the screening phase. Only 13 of the 25 RCTs, however, report numbers of screened patients in the main text. We previously reported a similar finding for RCTs on perioperative beta-blocker therapy in noncardiac surgery,35 indicating that the nonreporting of the screening phase is not restricted to the field of critical care medicine and underlining the need for standardized reporting on the screening population, procedures, and failures in general. A high recruitment-to-screening ratio may incorrectly suggest a pragmatic study approach and thus broad representativeness of the analyzed sample as long as the screening population is unknown.

Future studies should extend the results of our study to a multinational level and investigate the eligibility of patients for RCTs who have not been considered so far. Future studies of an established intervention ideally would be more pragmatic, but this trend needs to be assessed for each topic. At least awareness of the potential limitations of extrapolating evidence to subgroups of individuals who have not been studied in RCTs is a key future need. With the advancement of observational trial methodology such studies could augment our understanding of clinical effects in such subgroups. Moreover, it may be useful to implement reporting the degree of pragmatism as a standard item in reporting tools such as CONSORT statement. Accurate information about the degree of pragmatism allows investigators and readers assess the extent to which study findings are relevant to real-world settings.

Limitations

We used the latest Cochrane review published by Gamper et al in 2016 as a reference for the current evidence on the effect of vasopressors on critically ill adults. Randomized trials published since this work have not been considered in the assessment of trial eligibility. In addition, manual data extraction from medical records can be prone to error, although we attempted to minimize this source of error by having two investigators extract the data independently. Furthermore, it should be noted that we did not assess potential differences between the RCT samples analysed and hypothetical samples as defined by the eligibility criteria. This information, however, may be considered when interpreting the applicability of trial findings. Finally, it should be considered that the present study was conducted at several medical centers including a mixture of academic centers and standard hospitals in Austria, which represents a middle European perspective. Our study may therefore not necessarily be generalisable to countries outside Europe, and estimates may also vary by economic status of the countries.

Conclusion

The eligibility criteria of available RCTs for vasopressor treatment in critically ill adults are commonly restrictive and only applicable to a minority of patients receiving vasopressors in daily practice. Our study findings may aid researchers in designing future vasopressor trials and emphasize a standardized assessment and reporting of the level of pragmatism achieved, the screening phase, and screening failures.