Efficacy and safety of antimicrobial stewardship prospective audit and feedback in patients hospitalized with COVID-19: A protocol for a pragmatic clinical trial.

BACKGROUND: The use of broad-spectrum antibiotics is widespread in patients with COVID-19 despite a low prevalence of bacterial co-infection, raising concerns for the accelerated development of antimicrobial resistance. Antimicrobial stewardship (AMS) is vital but there are limited randomized clinical trial data supporting AMS interventions such as prospective audit and feedback (PAF). High quality data to demonstrate safety and efficacy of AMS PAF in hospitalized COVID-19 patients are needed. METHODS AND
DESIGN: This is a prospective, multi-center, non-inferiority, pragmatic randomized clinical trial evaluating AMS PAF intervention plus standard of care (SOC) versus SOC alone. We include patients with microbiologically confirmed SARS-CoV-2 infection requiring hospital admission for severe COVID-19 pneumonia. Eligible ward beds and critical care unit beds will be randomized prior to study commencement at each participating site by computer-generated allocation sequence stratified by intensive care unit versus conventional ward in a 1:1 fashion. PAF intervention consists of real time review of antibacterial prescriptions and immediate written and verbal feedback to attending teams, performed by site-based AMS teams comprised of an AMS pharmacist and physician. The primary outcome is clinical status at post-admission day 15 measured using a 7-point ordinal scale. Patients will be followed for secondary outcomes out to 30 days. A total of 530 patients are needed to show a statistically significant non-inferiority, with 80% power and 2.5% one-sided alpha assuming standard deviation of 2 and the non-inferiority margin of 0.5. DISCUSSION: This study protocol presents a pragmatic clinical trial design with small unit cluster randomization for AMS intervention in hospitalized COVID-19 that will provide high-level evidence and may be adopted in other clinical situations. TRIAL REGISTRATION: This study is being performed at the University of Alberta and is registered at ClinicalTrials.gov (NCT04896866) on May 17, 2021.

Introduction

COVID-19 is the disease caused by the severe acute respiratory coronavirus 2 (SARS-CoV-2), a novel coronavirus responsible for a global pandemic. The case burden and death toll is the highest of any respiratory virus outbreak in the modern antibiotic era [1]. Bacterial co-infections are known complications of viral pneumonia [2, 3] and it is estimated that 4–8% of hospitalized patients with COVID-19 will develop bacterial co-infection [4, 5]. The majority of COVID-19 management guidelines recommend judicious use of antimicrobials in patients presenting with pneumonia owing to the lack of benefit [6] and risks of Clostridioides difficile infection and other antimicrobial-associated adverse events [7, 8]. Despite this, significant and often broad-spectrum antibiotic use in hospitalized patients with COVID-19 is reported in the literature [9].

The COVID-19 pandemic is a significant source of unnecessary antibacterial therapy and is driving the often overlooked antimicrobial resistance (AMR) pandemic [10]. Many have highlighted the crucial role of formal antimicrobial stewardship program (ASP) involvement in managing the COVID-19 pandemic [11]. ASP goals are to combat AMR, reduce antimicrobial related complications, improve patient outcomes and maximize healthcare system efficiencies [5, 12, 13]. One core strategy is prospective audit and feedback (PAF) where antimicrobial stewardship (AMS) teams review patients’ charts and provide real-time feedback to attending teams to optimize an antimicrobial prescription. This is a collaborative post-prescription strategy that course-corrects suboptimal prescribing. It serves as a clinical service that provides education and recommendations based on an individual patient’s clinical context without providing direct clinical care.

While the benefit and safety of AMS PAF is described in settings such as community acquired pneumonia and viral acute respiratory infections, their cohort or quasi-experimental designs limit the ability to draw firm conclusions [14-18]. Given the significant antibiotic utilization in patients with COVID-19, this population offers a unique opportunity to study AMS PAF and produce high quality data using a robust randomized clinical trial design. There are no published studies to our knowledge that evaluate PAF in patients hospitalized with COVID-19 and specifically to determine the safety of rationalizing antibacterial therapy in those initiated empirically.

The objective of this study is to evaluate the safety and efficacy of PAF intervention plus standard of care (SOC) versus SOC alone in patients hospitalized with COVID-19 using clinical outcomes and a unique randomized pragmatic clinical trial design.

Materials and methods

This is a prospective, multi-center, non-inferiority, pragmatic randomized clinical trial of PAF + SOC versus SOC alone in patients with microbiologically confirmed SARS-CoV-2 infection in the preceding 2 weeks of hospitalization due to COVID-19. This study was reviewed and approved by the University of Alberta Research Ethics Board (Pro00105598) on January 27, 2021, and the Covenant Health Research Center on February 9, 2021. A waiver of individual patient consent was granted. This study is being performed at and sponsored by the University of Alberta and is registered at ClinicalTrials.gov (identifier, NCT04896866) on May 17, 2021 (protocol version 6.8, May 15, 2021). Study enrollment commenced in March 2021. We intend to publish the final results in a peer-reviewed medical journal. The study protocol adheres to the SPIRIT 2013 guidelines for clinical trials (Fig 1). Any modifications to the protocol will be reported to the University of Alberta Research Ethics Board and ClinicalTrials.gov, and will be stated in the final manuscript. We plan to grant public access to the full protocol, participant-level data, and the statistical code if required.

Fig 1

SPIRIT schedule of enrollment, interventions, and assessments.

Randomization

Each participating hospital will generate a line list of all hospital beds in adult COVID units and critical care units prior to enrollment. The line list will include the beds in each room on the unit and additional theoretical surge beds in the case of hospital surge and overcapacity during the pandemic. Randomization of beds will be stratified by COVID unit and critical care unit beds, and will be based on computer randomization with generation of allocation sequence in a 1:1 fashion to PAF + SOC or SOC alone.

If additional COVID or critical care units at a participating hospital are opened in the event of pandemic surge, the beds within the newly opened units will be included in the line list and the entire bed line list of the participating hospital will be subsequently re-randomized using the same randomization rules, to ensure appropriate stratification to COVID units and critical care units.

Participating centers

Participating centers include:

University of Alberta Hospital, Edmonton, Alberta, Canada

Grey Nuns Community Hospital, Edmonton, Alberta, Canada

Misericordia Community Hospital, Edmonton, Alberta, Canada

At our study hospitals, physicians attend to patients geographically located on a single hospital ward rather than an assigned roster. Patients are frequently transferred between units for a variety of reasons such as changes in required level of care or infection control purposes. Furthermore, transient providers such as clinical associates, resident physicians, or physician extenders often provide overnight coverage. Patients are expected to have numerous physicians providing care through the course in hospital.

Study population

The study target population is patients with SARS-CoV-2 infection requiring hospital admission for severe COVID-19 pneumonia. Patients with nosocomial-acquired SARS-CoV-2 infection were not included unless they were re-admitted to hospital from the community for severe COVID-19 pneumonia.

Inclusion criteria

All hospital beds in designated conventional units and critical care units accepting adult patients with microbiologically confirmed COVID-19 will be randomized. Patients are eligible for enrollment in the study if they meet all of the following inclusion criteria:

Age ≥ 18 years at the time of hospital admission.

Confirmed SARS-CoV-2 infection by nucleic acid testing or point-of-care antigen testing in the preceding 14 days of hospital admission.

Admitted from the community (including continuing care facilities).

Admitted to a hospital bed designated in the study.

A SpO2 ≤ 94% on room air, require supplemental oxygen, or have chest imaging findings compatible with COVID-19 pneumonia.

Exclusion criteria

All hospital beds outside of designated COVID units and critical care units will be excluded from randomization. A patient will be excluded from the study if:

The patient is enrolled in another clinical trial that involves antibacterial therapy.

The patient’s goals of care are anticipated to be designated “total compassionate care” or palliative within 48 hours of admission.

The patient’s progression to death is anticipated to be imminent and inevitable within 48 hours of admission.

The patient was attended by any member of the research team within 30 days of enrollment.

The patient is transferred from another acute care center.

Participant recruitment

Patients hospitalized with COVID-19 will be identified by the Alberta Health Services (AHS) Tableau dashboard, direct notification from site-based Infection Prevention & Control Programs, or direct screening of COVID units and critical care units. The AHS Tableau dashboard is a restricted access, secure dashboard developed by AHS analytics and reports positive COVID-19 cases that have been admitted to the study site along with date of admission and confirmatory SARS-CoV-2 test. The study team will screen patients every weekday (less statutory holiday) for study eligibility and enroll patients. The attending team and patients are blinded to the randomization sequence. Due to the nature of the PAF intervention, blinding of the ASP team will not be possible.

Intervention

The intervention employed will be AMS PAF. PAF consists of an unsolicited review of active antimicrobial prescriptions with real time feedback to attending teams. Audits are performed prospectively, on weekdays less statutory holidays, by members of the ASP team consisting of infectious disease or AMS pharmacists and physicians. Verbal and written feedback will be provided in real time to attending team members (most often the attending physician) if the ASP team is making a specific recommendation. The attending physician is considered the most responsible physician and will make the final decision to accept or reject PAF recommendations, and is therefore minimal risk to patient safety. ASP teams do not conduct interviews or perform physical examinations with patients.

The initial PAF will occur on the day an eligible patient is identified and enrolled in the study. Follow-up audits will then occur weekly (+/-3 days to account for weekends or statutory holidays) and ad-hoc if a new antibacterial is prescribed, until the primary end-point at post-admission day 15. Appropriateness in antimicrobial prescribing will be assessed based on local clinical practice guidelines (AHS COVID-19 Scientific Advisory Group recommendations) [19]. If no such guidelines exist, then appropriateness is defined by expert opinion of the AMS team member(s) performing the audit. The focus of ASP recommendations will be to discontinue therapy where bacterial co-infection is not suspected or confirmed, and to optimize the duration and spectrum of antimicrobial therapy when antibiotics are warranted.

Only antibacterials will be audited and included in the analysis. Antimycobacterial, antiviral, antifungal, and antiparasitic agents will not be audited or included in the analysis. Prescriptions will be excluded from PAF if they are single doses or discontinued prior to PAF. Prescriptions will also be excluded from PAF and the final analysis if being used for surgical or medical prophylaxis.

Patients will be followed and analyzed in the arm they were assigned to regardless of transfers or movements through the hospitalization period. Patients will be followed out to 15 days where the primary endpoint will be assessed, and then up to 30 days for assessment of secondary endpoints.

Primary outcome

The primary outcome will be the clinical status of the patient on post-admission day 15 measured using a 7-point ordinal scale, as presented in Table 1.

Table 1

Primary outcome 7-point ordinal scale.

Clinical Outcome	Points
Not hospitalized, able to resume normal daily activities	1
Not hospitalized, unable to resume normal daily activities	2
Hospitalized, not on supplemental oxygen	3
Hospitalized, on supplemental oxygen	4
Hospitalized, on high flow oxygen therapy or non-invasive mechanical ventilation	5
Hospitalized, on ECMO or invasive mechanical ventilation	6
Death	7

Secondary outcomes

Clinical endpoints

Hospital length of stay, in-hospital and 30-day mortality, C. difficile associated mortality, and 30-day re-admission rates will be examined.

Antimicrobial stewardship endpoints

Antimicrobial utilization will be measured by days of therapy and length of therapy normalized by patient-days for the duration of hospitalization (capped at 30 days). Furthermore, the number of audits, types of recommendations, and rate of acceptance will be determined.

Microbiologic endpoints

The 30-day multi-drug resistant (MDR) infection rates and 30-day C. difficile infection rate will be examined. The definition of MDR will be the lack of susceptibility to 1 or more agents in 3 or more antimicrobial categories active against the isolated bacteria [20]. In the case of Staphylococcus aureus and Enterococcus species, methicillin and vancomycin resistance, respectively, defines the strain as MDR regardless of resistance to other antimicrobials [20].

Adverse events and complications

The 30-day rates of neutropenia and acute kidney injury, diagnosed and staged according to Kidney Disease Improving Global Outcomes definitions, will be examined.

Data collection and management

The research team will assess, collect, and record all research data to a bespoke AHS REDcap® database in accordance with the protocol. All data access is controlled by unique usernames and passwords for individual study staff. Study staff will have access restricted to the functionality and data that are appropriate for their role in the study. Multiple education sessions were held to train the study staff regarding data integrity and quality. All study staff undergo mandatory AHS Information & Privacy education and training. Only investigators will have access to the final trial dataset. Research records will be kept for a minimum of 5 years in concordance with the University of Alberta Research Records Stewardship Guidance Procedure. This study does not involve the collection of biologic specimens.

Sample size estimation

A total of 530 patients (265 per arm) are needed to show a statistically significant non-inferiority, with 80% power and 2.5% one-sided alpha assuming standard deviation of 2 and the non-inferiority margin of 0.5. The non-inferiority margin was estimated based on previous published data comparing tocilizumab plus SOC versus SOC alone, that included a primary end-point based on the 7-level ordinal scale measured at day 15 [21]. In this study, the estimated mean score at day 15 in both groups was 3.04 and standard deviation 2.24. We opted for a conservative approach to meet the non-inferiority criteria selecting a non-inferiority margin less than 20%. A non-inferiority margin of 0.5 of the predicted mean score at day 15 (3.04) fulfilled the requirements without leading to an unreasonably large sample size. Accounting for a 5% drop out rate, 279 participants will be recruited in each arm (558 participants total). When 250 participants from the control arm reach 15 days follow up or when 260 patients are recruited in the control arm (whichever is earlier) a non-comparative sample size reassessment will be performed. The standard deviation will be calculated for the control group and will be used to recalculate the sample size.

Statistical analysis

All analyses will adhere to the principle of intention-to-treat (ITT). The ITT population will include all participants who were randomized in the trial. Additional analyses will be conducted on the per-protocol (PP) populations. The PP population will include all patients who completed the study as described in the protocol. This additional analysis will only be presented if there is a substantial difference in this populations compared to the ITT population.

The primary outcome will be a two-sample comparison of scores between the treatment and control arm. We will assess whether the scores in the treatment arm are not worse than in the control arm using the Mann-Whitney U test. A one-sided level of 0.025 will be used to declare significance for the non-inferiority. Results will be reported along with 95% confidence intervals.

Binary outcomes will be analyzed by a two-sample comparison of proportions using chi-square test. Continuous variables will be tested either by the Student’s t-test or by the Wilcoxon rank sum test depending on whether assumptions for the t-test are satisfied. Fisher’s exact test will be used to determine the statistical significance of difference with respect to the incidence of serious adverse events between the treatment and control arms. Baseline characteristics will be presented by the appropriate descriptive statistics: continuous variables will be summarized by mean, standard deviation, median, quartiles, minimum and maximum. Categorical data will be presented by absolute and relative frequencies (n and %).

All subgroup analyses will be considered exploratory. A time dependent and case intensity analyses will also be performed to determine if the ASP intervention has downstream effects on the prescribers. Comparison of the outcomes by sex (male/female), age group (by median age), and comorbidities will be performed. This analysis will be planned and described in statistical analysis plan. Primary and secondary outcomes will be adjusted for covariates. All adjusted analyses will be exploratory. Co-variates of interest will be included based on clinical relevance and will be specified in the statistical analysis plan. Adjustment will be performed by adding covariates to the original models. The senior biostatistician will be unblinded.

Discussion

The Infectious Diseases Society of America (IDSA) guidelines for implementing an ASP recommends PAF as a core component of antimicrobial stewardship programs [22]. However, the evidence for AMS intervention, including PAF, is heterogeneous and generally of low quality. Conclusions are drawn from mostly cohort or quasi-experimental studies [16-18]. Furthermore, a systematic review of AMS interventional studies between 1950 and 2017 concluded that the quality of evidence has not improved with time and that limitations should inform the design of future stewardship studies [23].

The IDSA guidelines also implies PAF is performed on a daily basis. The guidelines recommend that if not feasible, limited PAF 3 times per week can still offer benefit. However many studies define PAF interventions as providing delayed feedback such as with performance report cards [24]. While this strategy is valid, heterogeneity amongst PAF studies make it challenging to draw conclusions. Our study uses the daily PAF strategy as inferred by the IDSA guidelines as the intervention.

Antimicrobial stewardship programs identify various goals including to reduce rates of AMR, adverse events, and healthcare costs. Most published stewardship literature focuses on process outcomes rather than patient outcomes. Relatively few AMS studies report on mortality [18, 25, 26]. However, patient clinical outcomes including mortality should be emphasized more in stewardship research [27, 28]. Our study uses a 7-point ordinal scale of patient outcomes, including mortality, similar to other COVID-19 therapeutic trials as the primary outcome [29]. Furthermore, safety was prioritized as the primary outcome. In the setting of the first small-unit randomized clinical trial of prospective audit and feedback (PAF) in patients with COVID-19 to our knowledge, we believe it is critical to first demonstrate that rationalizing antibacterial therapy does not cause undue harm.

Few studies have examined the impact of AMS PAF in hospitalized patients using randomization. A Cochrane review of 221 studies identified only 58 RCTs by design [24]. Of only four studies with enablement plus feedback, only one intervened with real-time feedback. In another systematic review of 37 included stewardship interventional studies in hospital settings, only 3 of 14 studies evaluating PAF were RCT in design [30]. There are limitations to quasi-experimental and cohort study designs, the greatest is the limited ability to establish causal relationships due to multiple confounders that can influence antibiotic prescribing [23]. Robust randomization in AMS studies is desired but often not feasible resulting in studies using large unit cluster randomization such as by program, ward, or hospital. Traditional therapeutic trials intervene and study the outcomes in the same population with randomization at the individual participant level. In PAF studies, the intervention is on the prescriber whereas the outcomes of interest are in patients. At our 3 hospital study sites, the attending physician attends to a cohort of patients geographically located on a single ward. Therefore, randomizing by prescriber will effectively result in cluster randomization by ward. This may risk imbalanced arms as units have differences in patient acuity and medical complexity. Furthermore, differing stewardship practices in providers at baseline may also result in imbalanced arms. We therefore performed cluster randomization at the level of the hospital bed. To our knowledge, this is the lowest defined cluster reported in any AMS study that effectively minimizes contamination and baseline imbalances [26]. A robust homogeneous sample of patients by prescriber on every eligible COVID-19 patient is also achieved. More importantly, randomizing by hospital bed allows the same prescriber to potentially be in both arms of study which allows the evaluation of the effect of PAF independent of the prescriber.

High impact RCTs, especially studies using cluster randomization, have been performed without informed consent [31-36]. A waiver of consent was requested on the basis that our research involves no more than minimal risk to participants, alteration to consent requirements is unlikely to adversely affect the welfare of participants, and it is impossible or impracticable to carry out the research and to address the research question properly, given the research design, if the prior consent of participants is required. PAF functions as a reminder service to guide prescribers to follow patient-specific prescribing suggestions based on institutional guideline recommendations. PAF recommendations are reviewed and co-signed by the most responsible attending physician before the recommendations are live and executed. For this reason, PAF is considered no more than minimal risk. Furthermore, a waiver of consent minimizes selection bias of prescribers which is a threat to the validity of the research. A waiver of consent was granted by the University of Alberta Research Ethics Board pursuant to Article 3.7 of the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans–TCPS 2 (2018).

Given the open-label nature of our study, the Hawthorne effect is an unavoidable limitation. The Hawthorne effect is a phenomenon where prescribers may alter their antibiotic prescribing behavior once they are aware their prescribing is being monitored. The waiver of consent, a critical element of study design, may serve to minimize any Hawthorne effect. Additionally, physician learnings derived from the PAF + SOC arm may be applied to other patient prescriptions, including those patients in beds randomized to receive SOC. This is a reflection of the real world setting where the impact of PAF often extends beyond that individual encounter and begins to permeate a physician’s regular prescribing and speaks to the pragmatic element of our study. We hypothesize this effect may be diluted by rapid staff turnover as many additional physicians have been brought in to manage the pandemic surge, providing care on a weekly rotational basis (often with a separate day and night team). Furthermore, there will be an emphasis on patient-specific feedback based on institutional COVID-19 management guidelines as opposed to generic recommendations. Only study team members will have knowledge of the randomization sequence such that physicians cannot anticipate which study arm they have been assigned.

Antimicrobial stewardship is fundamental in the COVID-19 pandemic response [11, 37]. However, there is emerging evidence that AMS resources are being diverted away and ASPs are impacted with reduced productivity [38]. This study aims, as a secondary objective, to demonstrate the value of AMS intervention in viral pandemic management.

Conclusions

This study protocol describes a prospective, multi-centered, small unit cluster randomized, pragmatic clinical trial evaluating an antimicrobial stewardship intervention (prospective audit and feedback) in patients hospitalized with COVID-19 using a clinical primary outcome. The study design will provide high-level evidence and may be adopted in other clinical situations.

SPIRIT 2013 checklist: Recommended items to address in a clinical trial protocol and related documents.

(DOC)

Click here for additional data file.

COVASP protocol approved by the University of Alberta Research Ethics Board (Pro00105598).

(DOCX)

Click here for additional data file.

11 Jan 2022

Efficacy and Safety of antimicrobial stewardship prospective audit and feedback in patients hospitalized with COVID-19: a protocol for a pragmatic clinical trial

PLOS ONE

Dear Dr. Chen,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

The reviewers provided insougthful comments regarding the manuscript. Please provide answers for each of them.

Please submit your revised manuscript by 18-Feb-22. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Dafna Yahav

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We have noted that the estimated number of participants is reported as 558 while in the protocol this is calculated at 530. Please could you clarify this discrepancy.

3. In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability.

Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

We will update your Data Availability statement to reflect the information you provide in your cover letter.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Does the manuscript provide a valid rationale for the proposed study, with clearly identified and justified research questions?

The research question outlined is expected to address a valid academic problem or topic and contribute to the base of knowledge in the field.

Reviewer #1: Yes

Reviewer #2: Partly

**********

2. Is the protocol technically sound and planned in a manner that will lead to a meaningful outcome and allow testing the stated hypotheses?

The manuscript should describe the methods in sufficient detail to prevent undisclosed flexibility in the experimental procedure or analysis pipeline, including sufficient outcome-neutral conditions (e.g. necessary controls, absence of floor or ceiling effects) to test the proposed hypotheses and a statistical power analysis where applicable. As there may be aspects of the methodology and analysis which can only be refined once the work is undertaken, authors should outline potential assumptions and explicitly describe what aspects of the proposed analyses, if any, are exploratory.

Reviewer #1: Yes

Reviewer #2: Partly

**********

3. Is the methodology feasible and described in sufficient detail to allow the work to be replicable?

Descriptions of methods and materials in the protocol should be reported in sufficient detail for another researcher to reproduce all experiments and analyses. The protocol should describe the appropriate controls, sample size calculations, and replication needed to ensure that the data are robust and reproducible.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors described where all data underlying the findings will be made available when the study is complete?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception, at the time of publication. The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above and, if applicable, provide comments about issues authors must address before this protocol can be accepted for publication. You may also include additional comments for the author, including concerns about research or publication ethics.

You may also provide optional suggestions and comments to authors that they might find helpful in planning their study.

(Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Very nicely designed study and described in a clear manner, the authors address the challenges of such a study and examine two types of endopoints, both for the patient and the antibiotic stewardship outcomes.

my only major concern with this study is what the authors refer to as the Hawhtorne effect.

in my opinion it will be more severe than just changing of practice once one knows that he is being monitored.

the way the study is conducted the same physician can be randomized into the study in both arms at the same time on two different patients. ASP impacts clinical behavior and is never isolated to a single patient, clinicians learn and change their practice from case to case and i am sure that if a teaching point is successful in impacting a clinician to modify his behavior he will modify his behavior moving forward on other cases (if the ASP was successful) .

i fear that in the current study the investigators will not be able to assess this effect.

analysis should be performed not only per patient / bed but per clinician and in a time dependent manner to see if the ASP intervention has downstream effects on the prescriber.

Reviewer #2: The authors present a protocol for a pragmatic randomized controlled trial evaluating antimicrobial stewardship prospective audit and feedback in the context of COVID-19. Although there is growing evidence to support ASP PAF in general, more high quality data are needed, and COVID-19 is an optimal context to evaluate this important strategy. This is a much needed study but the authors should strengthen the argument for why it is needed, why they've selected a primary outcome that links more with safety (given that we already know not using antibiotics in the context of COVID-19 without co-infection is safe), and how they selected the non-inferiority margin. Additional suggestions below:

1. Abstract mentions confirmed SARS-CoV-2 prior to hospitalization, but will patients with nosocomial COVID-19 be included?

2. Please clarify in abstract - is randomization at the ward or patient bed level?

3. Abstract background can be shortened slightly in favour of additional detail on the intervention, e.g. who is providing feedback (interdisciplinary w/ pharmacist and physician?, how will feedback be provided?)

4. Abstract - what is the primary outcome and the non-inferiority margin?

5. Introduction - please cite examples of cohort and quasi experimental ASP studies in COVID-19.

6. More explanation is needed as to why this research is important. There are existing PAF/ASP studies in respiratory tract infections/CAP. Please explain why COVID-19 would be unique.

7. Lines 55-57. It seems that the authors have used secondary infection and co-infection interchangeably. Consider distinguishing the two in terms of their risk for bacterial infection.

8. "There is no interaction with the patient before, during, or after the intervention." I do not believe this is true for all PAF strategies, consider rephrasing.

9. Methods - are patients already being enrolled, at what date was the first patient enrolled?

10. How will "contamination" within providers be addressed? Presumably a provider could care for patients in SOC or SOC+PAF beds? A goal of PAF should be to empower providers to be stewards without the intervention from ASP/ID experts when the experts are not around. So if the intervention is done well, there should be a lot of within provider contamination over time. Cluster randomization at the provider level stratified by prescriber service would be ideal. This may need to be further addressed as a limitation to be mitigated.

11. Please provide more detail on the use of the primary outcome ordinal scale. Will it be based on change from baseline or simply the status of the patient at day 15?

12. Please define multi-drug resistant infection rates.

13. How was the non-inferiority margin of 0.5 selected?

14. Why was safety selected as a primary outcome? There doesn't seem to be any need to show that discontinuing antibiotics in COVID-19 is safe. There is already a Cochrane review on this topic. It may be more informative to make antibiotic utilization a primary outcome, to show that PAF is effective in the context of COVID-19. It is admirable that the authors select a clinical outcome as the primary outcome but ideally would want one that PAF can have a direct positive impact on (e.g., antibiotic-related harms, length of stay).

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

15 Feb 2022

Dear Dr. Yahav,

RE: PLOS ONE Decision: Revision required [PONE-D-21-21326] - [EMID:77fa8cea8ed2289f]

We thank PLOS ONE and the reviewers for reviewing this manuscript, providing thoughtful feedback and allowing the opportunity to submit revisions to improve the quality of the product. We have reviewed the reviewers’ comments carefully and have revised the manuscript to answer the reviewers’ questions and to incorporate suggestions. Please find our responses below. We have also resubmitted the revised manuscript with and without track changes as requested. Please note references to lines are to the revised, unmarked version of the manuscript without track changes.

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

We have carefully reviewed the style requirements. Revisions include the formatting of the title, authors, affiliations, figure titles, headings, in-text citations, and references. The rebuttal letter, marked, and unmarked revised manuscript files are named according to journal requirements.

2. We have noted that the estimated number of participants is reported as 558 while in the protocol this is calculated at 530. Please could you clarify this discrepancy.

A total of 530 participants (265 per arm) with the completed data for the final analysis are needed to show a statistically significant non-inferiority, with 80% power and 2.5% one-sided alpha assuming standard deviation of 2 and the non-inferiority margin of 0.5. We increased the sample size by 5% (558 participants) to account for the risk of participant exclusions or missing data.

3. In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability.

Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

We will update your Data Availability statement to reflect the information you provide in your cover letter.

The protocol manuscript does not contain data and the data availability policy is not applicable to our article.

Reviewer #1: Very nicely designed study and described in a clear manner, the authors address the challenges of such a study and examine two types of endopoints, both for the patient and the antibiotic stewardship outcomes.

my only major concern with this study is what the authors refer to as the Hawhtorne effect.

in my opinion it will be more severe than just changing of practice once one knows that he is being monitored.

the way the study is conducted the same physician can be randomized into the study in both arms at the same time on two different patients. ASP impacts clinical behavior and is never isolated to a single patient, clinicians learn and change their practice from case to case and i am sure that if a teaching point is successful in impacting a clinician to modify his behavior he will modify his behavior moving forward on other cases (if the ASP was successful) .

i fear that in the current study the investigators will not be able to assess this effect.

analysis should be performed not only per patient / bed but per clinician and in a time dependent manner to see if the ASP intervention has downstream effects on the prescriber.

Thank you for your feedback. Your point regarding learned behaviors contaminating the standard-of-care arm is well taken and has been acknowledged in the Discussion (lines 324-333). We believe this effect is likely diluted based on how our study 3 hospitals function. At our study hospitals, physicians attend to patients geographically located on a single hospital ward rather than an assigned roster. Patients are frequently transferred between units for a variety of reasons such as changes in required level of care or infection control purposes. Furthermore, transient providers such as clinical associates, resident physicians, or physician extenders often provide overnight coverage. For these reasons, patients are expected to have numerous physicians providing care through the course in hospital. This has been clarified in Methods (lines 114-118). Any behavior modifications derived from ASP effect in one attending physician is unlikely to impact the next attending physician.

We will perform time dependent and case intensity analyses to see if the ASP intervention has downstream effects on the prescribers was added to the Methods (lines 254-255). Antimicrobial utilization (secondary outcome) analysis will also provide insight.

Reviewer #2: The authors present a protocol for a pragmatic randomized controlled trial evaluating antimicrobial stewardship prospective audit and feedback in the context of COVID-19. Although there is growing evidence to support ASP PAF in general, more high quality data are needed, and COVID-19 is an optimal context to evaluate this important strategy. This is a much needed study but the authors should strengthen the argument for why it is needed, why they've selected a primary outcome that links more with safety (given that we already know not using antibiotics in the context of COVID-19 without co-infection is safe), and how they selected the non-inferiority margin. Additional suggestions below:

Thank you for your review and thoughtful feedback. We believe the specific concerns are addressed below in suggestion #6, 14, and 13 respectively.

1. Abstract mentions confirmed SARS-CoV-2 prior to hospitalization, but will patients with nosocomial COVID-19 be included?

We include patients with microbiologically confirmed SARS-CoV-2 infection requiring hospital admission for severe COVID-19 pneumonia. Patients with nosocomial-acquired SARS-CoV-2 infection were not included unless they were re-admitted to hospital from the community for severe COVID-19 pneumonia. We have added this clarification in the Methods (lines 121-123).

2. Please clarify in abstract - is randomization at the ward or patient bed level?

Randomization is at the patient bed level. We have changed “Eligible ward and critical care unit beds will be randomized…” to “Eligible ward beds and critical care unit beds will be randomized…” to provide clarity (lines 29-31).

3. Abstract background can be shortened slightly in favour of additional detail on the intervention, e.g. who is providing feedback (interdisciplinary w/ pharmacist and physician?, how will feedback be provided?)

The Abstract background has been shortened in favor of adding the following additional detail: “PAF intervention consists of real time review of antibacterial prescriptions and immediate written and verbal feedback to attending teams, performed by site-based AMS teams comprised of an AMS pharmacist and physician” (lines 31-33).

4. Abstract - what is the primary outcome and the non-inferiority margin?

The Abstract has been revised to incorporate the primary outcome (lines 33-34) and non-inferiority margin (line 37).

5. Introduction - please cite examples of cohort and quasi experimental ASP studies in COVID-19.

The Introduction has been revised to include 2 citations (citations 14 and 15) referencing ASP studies using cohort and quasi-experimental designs (line 70). To our knowledge, there is no published primary literature describing ASP prospective audit and feedback of antibacterials in patients with COVID-19.

6. More explanation is needed as to why this research is important. There are existing PAF/ASP studies in respiratory tract infections/CAP. Please explain why COVID-19 would be unique.

While the benefit and safety of ASP/PAF studies in community acquired pneumonia and viral acute respiratory infections has been studied, their cohort or quasi-experimental designs limit the ability to draw firm conclusions. Currently, the COVID-19 pandemic is a significant source of unnecessary antibacterial therapy and is driving the antimicrobial resistance pandemic. Antimicrobial stewardship intervention, supported by high quality clinical trial data, is warranted to control the pandemic within a pandemic. The Introduction has been extensively revised to incorporate detail and explanation (lines 47-78):

COVID-19 is the disease caused by the severe acute respiratory coronavirus 2 (SARS-CoV-2), a novel coronavirus responsible for a global pandemic. The case burden and death toll is the highest of any respiratory virus outbreak in the modern antibiotic era. Bacterial co-infections are known complications of viral pneumonia. It is estimated that 4-8% of hospitalized patients with COVID-19 will develop bacterial co-infection. The majority of COVID-19 management guidelines recommend judicious use of antimicrobials in patients presenting with pneumonia owing to the lack of benefit and risks of Clostridioides difficile infection and other antimicrobial-associated adverse events. Despite this, significant and often broad-spectrum antibiotic use in hospitalized patients with COVID-19 is reported in the literature.

The COVID-19 pandemic is a significant source of unnecessary antibacterial therapy and is driving the often overlooked antimicrobial resistance (AMR) pandemic. Many have highlighted the crucial role of formal antimicrobial stewardship program (ASP) involvement in managing the COVID-19 pandemic. ASP goals are to combat AMR, reduce antimicrobial related complications, improve patient outcomes and maximize healthcare system efficiencies. One core strategy is prospective audit and feedback (PAF) where antimicrobial stewardship (AMS) teams review patients' charts and provide real-time feedback to attending teams to optimize an antimicrobial prescription. This is a collaborative post-prescription strategy that course-corrects suboptimal prescribing. It serves as a clinical service that provides education and recommendations based on an individual patient’s clinical context without providing direct clinical care.

While the benefit and safety of AMS PAF is described in settings such as community acquired pneumonia and viral acute respiratory infections, their cohort or quasi-experimental designs limit the ability to draw firm conclusions. Given the significant antibiotic utilization in patients with COVID-19, this population is an opportunity to the study of AMS PAF to produce high quality data using a robust randomized clinical trial design. There are no published studies to our knowledge that evaluate PAF in patients hospitalized with COVID-19 and specifically to determine the safety of rationalizing antibacterial therapy in those initiated empirically.

The objective of this study is to evaluate the safety and efficacy of PAF intervention plus standard of care (SOC) versus SOC alone in patients hospitalized with COVID-19 using clinical outcomes and a unique randomized pragmatic clinical trial design.

7. Lines 55-57. It seems that the authors have used secondary infection and co-infection interchangeably. Consider distinguishing the two in terms of their risk for bacterial infection.

The use of “secondary bacterial infection” has been revised to “bacterial co-infection” in the manuscript (line 49 and line 172).

8. "There is no interaction with the patient before, during, or after the intervention." I do not believe this is true for all PAF strategies, consider rephrasing.

Thank you for pointing this out. We removed the line "There is no interaction with the patient before, during, or after the intervention" from the Introduction.

At our 3 study sites, the antimicrobial stewardship teams perform chart reviews and provide written and verbal feedback to the attending team. The antimicrobial stewardship team do not conduct an interview or physical examination with the patient. In the Methods, we revised the text “There is no interaction with the patient before, during, or after the intervention” to “ASP teams do not conduct interviews or perform physical examinations with patients” (line 164).

9. Methods - are patients already being enrolled, at what date was the first patient enrolled?

We have added “Study enrollment commenced in March 2021.” to the Methods (lines 87-88).

10. How will "contamination" within providers be addressed? Presumably a provider could care for patients in SOC or SOC+PAF beds? A goal of PAF should be to empower providers to be stewards without the intervention from ASP/ID experts when the experts are not around. So if the intervention is done well, there should be a lot of within provider contamination over time. Cluster randomization at the provider level stratified by prescriber service would be ideal. This may need to be further addressed as a limitation to be mitigated.

We had considered cluster randomization at the provider level during study conception and design. However, at all 3 of our study hospitals, one physician attends a geographic a unit (average 20 beds per unit) for 1 week at a time. By randomizing at the provider level, cluster randomization at the unit level would have inadvertently occurred which may risk imbalanced arms as units have differences in acuity and medical complexity. Furthermore, differing stewardship practices in providers at baseline may also result in imbalanced arms. We therefore performed cluster randomization at the level of the hospital bed. To our knowledge, this is the lowest defined cluster reported in any AMS study that effectively minimizes contamination and baseline imbalances. This has been clarified in the Discussion (lines 297-303).

Your point regarding learned behaviors contaminating the standard-of-care arm is well taken and has been acknowledged in the Discussion (lines 324-333). We believe this effect is likely diluted based on how our study 3 hospitals function. At our study hospitals, physicians attend to patients geographically located on a single hospital ward rather than an assigned roster. Patients are frequently transferred between units for a variety of reasons such as changes in required level of care or infection control purposes. Furthermore, transient providers such as clinical associates, resident physicians, or physician extenders often provide overnight coverage. For these reasons, patients are expected to have numerous physicians providing care through the course in hospital. This has been clarified in Methods (lines 114-118). Any behavior modifications derived from ASP effect in one attending physician is unlikely to impact the next attending physician.

We will perform time dependent and case intensity analyses to see if the ASP intervention has downstream effects on the prescribers was added to the Methods (lines 254-255). Antimicrobial utilization (secondary outcome) analysis will also provide insight.

11. Please provide more detail on the use of the primary outcome ordinal scale. Will it be based on change from baseline or simply the status of the patient at day 15?

The primary outcome will be the clinical status of the patient on post-admission day 15 measured using a 7-point ordinal scale. The Methods has been revised to provide clarity (lines 186-187).

12. Please define multi-drug resistant infection rates.

We will use the definition of multi-drug resistance (MDR) as the lack of susceptibility to one or more agents in three or more antimicrobial categories active against the isolated bacteria [1]. In the case of Staphylococcus aureus and Enterococcus species, methicillin and vancomycin resistance, respectively, defines the strain as MDR regardless of resistance to other antimicrobials [1].

This has been added to the Methods (lines 201-204).

13. How was the non-inferiority margin of 0.5 selected?

The following has been added to the Methods (lines 222-227): “The non-inferiority margin was estimated based on previous published data comparing tocilizumab plus standard of care versus standard of care alone, that included a primary end-point based on the 7-level ordinal scale measured at day 15 [2]. In this study, the estimated mean score at day 15 in both groups was 3.04 and standard deviation 2.24. We opted for a conservative approach to meet the non-inferiority criteria selecting a non-inferiority margin less than 20%. A non-inferiority margin of 0.5 of the predicted score at day 15 (3.04) fulfilled the requirements without leading to an unreasonably large sample size.”

14. Why was safety selected as a primary outcome? There doesn't seem to be any need to show that discontinuing antibiotics in COVID-19 is safe. There is already a Cochrane review on this topic. It may be more informative to make antibiotic utilization a primary outcome, to show that PAF is effective in the context of COVID-19. It is admirable that the authors select a clinical outcome as the primary outcome but ideally would want one that PAF can have a direct positive impact on (e.g., antibiotic-related harms, length of stay).

There is a body of evidence demonstrating the incidence of bacterial co-infection in patients hospitalized with COVID-19 is low, suggesting routine empiric antibacterial therapy targeting bacterial coinfection is likely not necessary unless there is strong suspicion or evidence that one is present. The Cochrane review includes 11 randomized clinical studies of over 11 thousand patients investigating antibiotics compared to placebo, standard of care alone, or another antibiotic for treatment of COVID-19 [3]. Azithromycin was the only antimicrobial studied. This Cochrane review concludes that 28 day mortality is not reduced with azithromycin treatment. It however does not examine discontinuation of antibacterial therapy initiated by a prescriber.

Empiric, broad-spectrum antibiotics remain commonly prescribed in patients with COVID-19 upon hospitalization. A number of these patients are likely to go on to “complete a course” of antibiotics despite not exhibiting any features of bacterial co-infection. Clinical practice often does not follow guideline recommendations. There are no high quality evidence to draw firm conclusions that antibacterial therapy can be safely discontinued in those initiated empirically.

In this context, and in the setting of the first small unit randomized clinical trial of prospective audit and feedback (PAF) in patients with COVID-19 to our knowledge, we believe it is critical to first demonstrate that rationalizing antibacterial therapy does not cause undue harm in patients. We have added this to the Discussion (lines 282-285).

Other outcomes that PAF can have a direct positive impact on, such as antibiotic utilization, antibiotic-related harms, and length of stay, remain important and are examined as secondary outcomes.

On behalf of all the authors, we would again like to thank PLOS ONE and the reviewers for taking the time to review our manuscript.

3 Mar 2022

Efficacy and Safety of antimicrobial stewardship prospective audit and feedback in patients hospitalized with COVID-19: a protocol for a pragmatic clinical trial

PONE-D-21-21326R1

Dear Dr. Chen,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Dafna Yahav

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

14 Mar 2022

PONE-D-21-21326R1

Efficacy and Safety of antimicrobial stewardship prospective audit and feedback in patients hospitalized with COVID-19: a protocol for a pragmatic clinical trial

Dear Dr. Chen:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Dafna Yahav

Academic Editor

PLOS ONE