STROBE-AMS: recommendations to optimise reporting of epidemiological studies on antimicrobial resistance and informing improvement in antimicrobial stewardship.

OBJECTIVES: To explore the accuracy of application of the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) tool in epidemiological studies focused on the evaluation of the role of antibiotics in selecting resistance, and to derive and test an extension of STROBE to improve the suitability of the tool in evaluating the quality of reporting in these area.
METHODS: A three-step study was performed. First, a systematic review of the literature analysing the association between antimicrobial exposure and acquisition of methicillin-resistant Staphylococcus aureus and/or multidrug-resistant Acinetobacter baumannii was performed. Second, articles were reviewed according to the STROBE checklist for epidemiological studies. Third, a set of potential new items focused on antimicrobial-resistance quality indicators was derived through an expert two-round RAND-modified Delphi procedure and tested on the articles selected through the literature review.
RESULTS: The literature search identified 78 studies. Overall, the quality of reporting appeared to be poor in most areas. Five STROBE items, comprising statistical analysis and study objectives, were satisfactory in <25% of the studies. Informative abstract, reporting of bias, control of confounding, generalisability and description of study size were missing in more than half the articles. A set of 21 new items was developed and tested. The new items focused particularly on the study setting, antimicrobial usage indicators, and patients epidemiological and clinical characteristics. The performance of the new items in included studies was very low (<25%).
CONCLUSIONS: Our paper reveals that reporting in epidemiological papers analysing the association between antimicrobial usage and development of resistance is poor. The implementation of the newly developed STROBE for antimicrobial stewardship (AMS) tool should enhance appropriate study design and reporting, and therefore contribute to the improvement of evidence to be used for AMS programme development and assessment. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/

This study developed a revised version of Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) to increase appropriateness of reporting for epidemiological studies focused on the link between antimicrobial-resistant bacteria and antibiotic usage.

STROBE for antimicrobial stewardship (STROBE-AMS) was developed through a 2-round Delphi experts approach. The final set included 21 new variables and four areas were graded as the most relevant: definition of infection and/or colonisation; setting; antibiotic usage at patient’s and hospital’s level; molecular resistance mechanisms; and infection control.

The STROBE-AMS application was tested only in epidemiological studies on methicillin-resistant Staphylococcus aureus and multidrug-resistant Acinetobacter baumannii. Since the postdevelopment test was performed only for observational reports, no specific bias was introduced.

Introduction

The rate of infections caused by antimicrobial-resistant microorganisms is seen increasingly by the public and healthcare inspection organisations as an indicator of quality of healthcare and patient safety.1 Several studies have explored risk factors associated with infection or colonisation due to antimicrobial-resistant microorganisms.2 3 Among them, antimicrobial exposure is a well-known risk factor for subsequent infections due to methicillin-resistant Staphylococcus aureus (MRSA), vancomycin-resistant enterococci (VRE) and multidrug-resistant (MDR) Gram-negative bacteria.4–6 Antimicrobial stewardship (AMS), included in a multifaceted approach, represents an essential tool to combat the spread of resistance in the healthcare setting.7 8 An effective AMS programme should include the appropriate choice of empirical therapy (including decision on no antimicrobial treatment) as well as duration and dosage of the drug in order to provide the best outcome for the patient, to reduce adverse effects, and to prevent and control the rate of antimicrobial resistance.9 As with intervention measures, AMS should be based on evidence. However, although randomised controlled trials (RCTs) are considered to be the gold standard of epidemiological research, analyses of risk factors for antimicrobial-resistant infections, especially those in which we can intervene, are usually not tested by RCTs, and rely mainly on observational studies. The major limitations of epidemiological studies on antimicrobial-resistant infections are the retrospective design, poor controlling for confounding factors (ie, duration and dosage of antibiotics, knowledge of carrier status, combination and sequential antibiotic therapy) and heterogeneity in several definitions (ie, definition of infection and of antimicrobial-resistant microorganisms, time-at-risk period for previous antimicrobial exposure, selection of control group).10–13 A further limitation affects external validity, since prevalence of resistance changes in different locations and in time. In an era of increasing antimicrobial resistance, reducing these flaws and increasing appropriate reporting is critical to the application of findings to AMS programmes.

The ‘Strengthening the Reporting of Observational Studies in Epidemiology’ (STROBE) was an initiative to improve the quality of reporting evidence.14 It aimed to provide useful guidance by establishing a checklist consisting of 22 items for the transparent and complete publication of observational studies, facilitating their critical evaluation.14 Recently, an extension of the STROBE statement (STROME-ID) established recommendations to support good scientific reporting of molecular epidemiological studies.15 However, the peculiarity of the epidemiology of antimicrobial-resistant infections being strongly connected to patients’ characteristics as well as to different settings (hospital, community and healthcare centres) as well as surveillance methods of antibiotic usage (ie, defined daily dose or packets or prescriptions) cannot be properly covered through the items currently included in STROBE.

Therefore, we performed a study with two main objectives: (1) to explore the accuracy of application of STROBE in epidemiological studies focused on the link between antibiotics usage and development of resistance; and (2) to derive and test an extension of STROBE that could improve the suitability of the tool in evaluating the quality of reporting of epidemiological studies in this area. Studies addressing MRSA and the MDR-Acinetobacter baumannii group were selected for the frequency with which they have been used worldwide as indicators of healthcare associated infections.2 16

Study design

Methods

There were three-steps to the study design. First, to explore the STROBE application, we searched the literature (1976–2013) for articles analysing the association between antimicrobial exposure, and acquisition of MRSA and/or MDR-A. baumannii. Second, all the articles were reviewed according to the STROBE checklist.14 Third, a set of potential new items focused on antimicrobial-resistance quality indicators for proper reporting was derived through a two-round RAND-modified Delphi procedure, involving experts in the field of antimicrobial prescribing.17 The extension of STROBE for AMS (STROBE-AMS) was then tested on the articles selected in the first study step.

Selection of articles

Published human studies concerning the role of previous antimicrobial therapy as a risk factor for developing MRSA and MDR-A. baumannii colonisation or infection in hospitalised patients were identified through computerised literature searches using MEDLINE (National Library of Medicine, Bethesda, Maryland, USA) and EMBASE (Excerpta Medica Database), and by reviewing the references of the retrieved articles. Index search terms included the following Medical Subject Headings: ‘risk factors’, ‘resistance’, ‘antimicrobial therapy’, ‘Acinetobacter’, ‘outbreak’, ‘Staphylococcus aureus’ and ‘methicillin-resistant’. The search was carried out with no language restriction and followed PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) guidelines. Studies were considered eligible if they included adult patients (>16 years old) and presented data pertaining to the relationship between antimicrobial use and the development of MRSA or MDR-A. baumannii colonisation or infection. Authors were contacted for missing information.

Data extraction

Data extraction was performed by four independent researchers in two centres. This process involved infectious disease specialists as well as epidemiologists. In case of disagreement among the reviewers, a senior reviewer was consulted. Reviews, letters, editorials and case reports were excluded. Data extraction included also the impact factor (IF) of the journal (2013 Journal Citation Report, Thomson Institute for Scientific Information), which is a measure of average citation frequency for all the articles over a given period of time.

Reporting assessment

The assessment of the quality of reporting was performed according to the STROBE statement.14 Data from each study were entered into standardised forms, verified for consistency and accuracy, and then entered into a computerised database. Entries for each item of the STROBE statement were categorised as ‘Yes’, if they were completely in agreement with the STROBE statement explanation and elaboration documentation, ‘No’, if this was not satisfied, and ‘Partly’, if evident only in part of the text. Absolute and relative frequencies were used to describe the results of these judgments of quality of reporting according to the STROBE statement.

New items development

The new items development was carried out through a two-round Delphi approach. A list of experts in the field of antimicrobial prescribing was collated from the network of the European Society of Infectious Diseases and Clinical Microbiology (ESCMID) and the Impact of Specific Antibiotic Therapies on the prevalence of hUman host ResisteNt bacteria (SATURN) Project on antimicrobial usage and selection of resistance in hospitalised patients (EU-7th FP7-241796). To develop a first set of quality indicators, the literature was reviewed to define the major limitations of current research on the association between antimicrobial usage and antimicrobial resistance,11 18 19 and a first set of indicators, developed from the authors, was sent out in December 2013. Respondents were asked to rate each new item against two continuous 1–9 integer scales and provide their comments. The second round was performed in May 2014, according to the same indications. Ethic consent was not required because no patient data were used and the study was based on literature review.

Test of the new items

The new developed items were then applied to the papers selected for the systematic review. The same reporting assessment used for the STROBE evaluation was applied.

Statistical analysis

Scores were analysed using a non-parametric test (Fischer's test). Analysis was performed with STATA V.12.1 (Stata Corporation, Texas, USA).

Results

Identification of relevant articles

The search identified 1008 potentially relevant studies; 311 were excluded because they were reviews, case reports or letters; 233 were excluded because they did not investigate previous antibiotic therapy. Among the remaining 464, 386 articles were excluded because they did not analyse the relationship between antibiotic usage and development of resistance, were duplicates, or their authors could not provide missing information. Overall, 78 studies (57 on MRSA and 21 on MDR-Acinetobacter) fulfilled inclusion criteria for the association between antimicrobial usage and antimicrobial resistance: 39 (51%) case–control, 28 (35%) cohort and 11 (14%) cross-sectional studies. The list of reviewed articles and the flow chart are reported in online supplementary annex 1—figure 1 and table 1.

The results of the assessment of quality of reporting through the STROBE statement of these 78 studies are shown in table 1. Overall, the quality of reporting appeared to be poor in most areas. Best performing areas of the STROBE tool were seven items including the description of setting, participants and generic review of statistical methods that were well satisfied in >75% of the studies. However, five items, comprising statistical details for the analysis of quantitative variables, subgroup analysis, sample size calculation, addressing potential sources of biases and, most notably, the description of main study objectives, were satisfactory in <25% of the studies.

Table 1

STROBE item checklist for 78 epidemiological studies analysing antimicrobial usage and development of colonisation/infections due to MRSA and MDR-Acinetobacter baumannii

Level of satisfaction	STROBE statement number	Section
<25% of studies	3. State specific objectives, including any prespecified hypotheses.	Introduction
	9. Describe any efforts to address potential sources of bias10. Explain how the study size was arrived at11. Explain how quantitative variables were handled	Methods
	17. Report other analyses performed—eg, analyses of subgroups	Results
≥25% to <50% of studies	1a. Indicate the study design with a commonly used term in the title or the abstract1b. Provide an informative and balanced summary in the abstract	Title and Abstract
	19. Discuss limitations of the study, taking into account sources of potential bias or imprecision21. Discuss the generalisability	Discussion
	22. Give the source of funding	Funding
≥50% to <75% of studies	2. Explain the scientific background and rationale for the investigation being reported.	Introduction
	4. Present key elements of study design early in the paper7. Clearly define all outcomes, exposures, predictors, potential confounders, and effect modifiers8. For each variable of interest, give sources of data and details of methods of assessment	Methods
	13. Report the numbers of individuals at each stage and reason for non-participation16. Give unadjusted estimates and, if applicable, confounder-adjusted estimates. Make clear which counfounders were adjusted for and why they were included	Results
≥75% of studies	5. Describe the setting, locations and relevant dates6. Description of participants—eligibility criteria, sources and methods12. Describe all statistical methods, including those used to control for confounding. Explain how missing data, lost to follow up and sensitivity analyses were addressed	Methods
	14. Give the characteristics of study participants15. Report outcome data	Results
	18. Summarise key results with reference to study objectives20. Give a cautious overall interpretation of results	Discussion

Bold typeface indicates main variables included in the STROBE tool.

MDR, multidrug-resistant; MRSA, methicillin-resistant Staphylococcus aureus; STROBE, Strengthening the Reporting of Observational Studies in Epidemiology.

Table 2 describes the association between the value of the IF and the grade of satisfaction of the STROBE statement's items. Informative abstract, explanation of objectives, reporting of bias, statistical methods, control of confounding, generalisability, definition of quantitative variables, giving of estimates and description of study size, were missing or incomplete in more than half the articles published in 13 journals with IF greater than four. The cut-off of four was selected based on the IF median distribution. There was no association between type of journal (general medicine vs clinical infectious diseases and microbiology-dedicated journal) and year of study publication (before and after 2007, when the STROBE tool was introduced).

Table 2

Association between satisfaction of the STROBE items and journal's IF

Item	IF>4N=13 (%)	IF≤4N=53 (%)	No IFN=12 (%)
Study design reported in title/abstract	7 (54)	28 (53)	2 (17)
Informative abstract included	5 (38)	22 (42)	2 (17)
Background/rationale explained	10 (77)	40 (75)	3 (25)
Objectives explained	4 (31)	11 (21)	2 (17)
Key elements of study design defined	10 (77)	28 (53)	3 (25)
Hospital/community setting explained	13 (100)	44 (83)	8 (67)
Eligibility criteria defined	12 (92)	47 (89)	7 (58)
Epidemiological variables described	10 (77)	35 (66)	6 (50)
Data sources/measurement defined	9 (69)	34 (64)	4 (33)
Analysis of study bias included	1 (8)	0	0
Justification of study sample size reported	1 (8)	4 (8)	0
Quantitative variables defined	3 (23)*	2 (4)*	0
Statistical methods and control of confounding reported	5 (38)*	5 (9)*	0
Number of participants reported	9 (69)	34 (64)	5 (42)
Characteristics of participants reported	13 (100)	52 (98)	9 (75)
Outcome data defined	13 (100)	51 (96)	8 (67)
Unadjusted and adjusted estimates given	6 (46)	18 (34)	2 (17)
Key results summarised	13 (100)	50 (94)	8 (67)
Limitations of the study considered	8 (62)	27 (51)	2 (17)
Interpretation of results provided	13 (100)	45 (85)	7 (58)
Generalisability of study reported	6 (46)	20 (38)	4 (33)

*p≤0.05 based on two-sided Fisher’s exact test comparing the proportions among articles with journal IF>4 vs IF≤4.

IF, impact factor; STROBE, Strengthening the Reporting of Observational Studies in Epidemiology.

Development of new items

In the first round, 16 new items were sent to the experts. No indicators were discarded between rounds and five new items were added. In the second round, participants were provided with the frequency of distribution of scores and qualitative comments. The final set included 21 new variables, presented in table 3. The main new items referred to the description of the study setting and participants. In particular, the following four areas were graded as the most relevant for studies reporting on antimicrobial usage and resistance development: definition of infection and/or colonisation, and evidence of robustness of the new definition (if not a validated reference); definition of setting (epidemic or endemic); definition of antibiotic usage at patient’s level including type, dosage, duration, route of administration and combinations; description of antimicrobial formulary at the hospital level and measurement of antibiotic usage (defined as daily dosage, packet daily dosage, treatments, units), definition of how antibiotic consumption data were obtained (pharmacy, patients’ charts, etc) and if it was actually used or purchased/dispensed. The remaining three items focused on infection control measures applied at the study location: definition of resistance, cross-resistance and molecular resistance mechanisms, for example, plasmids, chromosomes, integrons, transposons; and statistical methods including controlling for confounders, rationale for grouping of antimicrobials and time at risk for antimicrobial exposure. Among the confounders, items reporting on previous stay in long term care facilities, nursing home and other hospitals, were added.

Table 3

New checklist items proposed to be included in the STROBE statement for deepening the assessment of epidemiological studies analysing the impact of antimicrobial usage on the development of antimicrobial-resistant infections

Item	Item number	STROBE recommendation	STROBE-AMS new items
Introduction
Background/rationale	2	Explain the scientific background and rationale for the investigation being reported	2.1 Report previous clinical in vivo and in vitro studies
Objectives	3	State specific objectives, including any prespecified hypotheses
Methods
Setting	5	Describe the setting, locations, relevant dates, including periods of recruitment, exposure, follow-up and data collection	5.1 Describe if setting is epidemic or endemic (high, low, medium) for the study outcome5.2 Specify type of hospital or unit and characteristics of population served by the healthcare setting5.3 Describe antimicrobial formulary in use at the study location related to the analysed antibiotics5.4 Describe infection control measures dedicated to the target resistant bacteria applied at the study location
Participants	6	(a) Cohort study—Give the eligibility criteria, the sources and methods of selection of participants. Describe methods of follow-up	6.1 Define unit analysed (person, department or other)
		Case–control study—Give the eligibility criteria, the sources, methods of case ascertainment and control selection. Give the rationale for the choice of cases and controlsCross-sectional study—Give the eligibility criteria, the sources and methods of selection of participants(b) Cohort study—For matched studies, give matching criteria, the number of exposed and unexposedCase–control study—For matched studies, give matching criteria and the number of controls per case	6.2 Provide reasons (epidemiological and clinical) for choosing matching criteria
Variables	7	Clearly define all outcomes, exposures, predictors, potential confounders and effect modifiers. Give diagnostic criteria, if applicable	7.1 Specify antimicrobial usage according to: type, dosage, duration and route of administration7.2 Provide information using defined daily dosages (DDDs) and, in addition, other definitions closer to local reality (packages, prescriptions). Provide justification for the measurement presented7.3 Address antimicrobial combinations7.4 Explain rationale for grouping of antimicrobials7.5 Define time at risk for antimicrobial exposure and for resistance development7.6 Include description of potential confounders (other than epidemiological variables)7.7 Provide definition of resistance, multidrug resistance, including pattern of co-resistance; whether studies performed to identify location or resistance eg, plasmid, chromosome, integron, transposon7.8 Definition of infection and/or colonisation. If not a validated reference, provide evidence of robustness of the new definition
Data sources/measurement	8	For each variable of interest, give sources of data and details of methods of assessment (measurement). Describe comparability of assessment methods if there is more than one group	8.1 Describe how antimicrobial consumption data were obtained (pharmacy, patients’ charts, etc) and if it was actually used or purchased/dispensed
Quantitative variables	11	Explain how quantitative variables were handled in the analyses. If applicable, describe which groupings were chosen, and why	11.1 Provide subgroup analyses for immunocompromised, surgical/medical patients and patients in intensive care units, if applicable
Results
Descriptive data	14	(a) Give characteristics of study participants (eg, demographic, clinical, social) and information on exposures and potential confounders	14.1 Specify among the exposure: previous stay in long-term care facilities, nursing home and other healthcare settings
		(b) Indicate number of participants with missing data for each variable of interest
		(c) Cohort study—Summarise follow-up time (eg, average and total amount)
Other analyses	17	Report other analyses performed—eg, analyses of subgroups and interactions, and sensitivity analyses	17.1 Report subgroup analysis by type of patients and type of microorganism, if applicable
Discussion
Limitations	19	Discuss limitations of the study, taking into account sources of potential bias or imprecision. Discuss both direction and magnitude of any potential bias	19.1 Provide description of sources of selection bias, including infection control measures, audit and confounding
Generalisability	21	Discuss the generalisability (external validity) of the study results	21.1 Discuss study setting, type of hospital, local epidemiology for the generalisability
Other information
Funding	22	Give the source of funding, the role of the funders for the present study and, if applicable, for the original study on which the present article is based

Bold typeface indicates main variables included in the STROBE tool.

STROBE, Strengthening the Reporting of Observational Studies in Epidemiology; STROBE-AMS, STROBE for antimicrobial stewardship.

Test of new items

The performance of reviewed articles to the new items was very poor. Overall, the new items were satisfied in <25% of the papers reviewed (item 5.3, 6.2, 7.1–4, 8, 11, 17 and 21).

Discussion

With increasing reporting of antibiotic-resistant infections worldwide, epidemiological investigations on risk factors for resistance require rather special attention regarding designing and reporting in order to maximise their ability to inform AMS programmes. Our paper reveals that current reporting in epidemiological studies focusing on the association between antibiotic usage and development of resistance is very poor. Although some items including the description of setting and generic review of statistical methods were well satisfied in >75% of the studies, more than half the 22 domains included in the STROBE checklist were not satisfied in the majority of studies analysing the role of antibiotics in selecting MRSA and MDR-A. baumannii.

The major consequence of this inaccuracy is that a significant part of the evidence contributing to the development of AMS programmes is currently of a very low level. Improvement of reporting could be achieved at two different levels: on the one side, it is essential to increase the awareness of researchers and publishers on the importance of the appropriate reporting for epidemiological studies on antimicrobial resistance, and on the other, availability of items specifically developed for this field could play a pivotal role, and increase feasibility and implementation. The new items developed by a group of experts through a Delphi approach, which combined evidence and expert opinion, mainly focuses on the methods section and includes specific epidemiological items (ie, definition of the epidemiological setting in terms of endemicity or epidemicity) as well as a set definitions for antimicrobial usage (ie, local formulary, antibiotic measurements, presence or absence of any AMS, etc). To further underline the innovation of the items and the importance of introducing the new STROBE-AMS, our results showed that the new items were not reported in more than two-thirds of 78 already published studies.

Interestingly, although a significant correlation was observed between the IF of the journal and satisfactory compliance with the STROBE statement criteria, there were very important areas such as an informative abstract, explanation of objectives, reporting of statistical methods and generalisability, that were missing or incomplete in >50% of articles published even in higher quality journals with IF factors >4. This result underlines the need for involvement of major scientific journals on clinical infectious diseases in a re-evaluation process concerning the review process of such papers. Indeed, RCTs are not accepted for publication in high-ranked journals if they do not comply with the CONSORT guidelines (Consolidated Standards for Reporting Trials).20 Similarly, systematic reviews and meta-analyses need to comply with the PRISMA guidelines.21 We would advocate that, for observational trials on antibiotic-resistant infections, where there is a greater potential for bias, this adaptation of the STROBE tool must be an essential review requirement, significantly contributing to the global efforts to combat the spread of antimicrobial-resistant microorganisms with improvement of the quality of evidence of the reports. Implicit in this requirement is that the studies will have had to consider these checklists at the design stage as well.

Our study has limitations. The analysis of STROBE application in the field of antimicrobial-resistant microorganisms was limited to epidemiological studies on MRSA and MDR-A. baumannii. These microorganisms were selected as their occurrence is very common, the majority of published epidemiological studies are focused on these bacteria, and their prevention and control are important potential indicators of the success, or otherwise, of healthcare associated interventions. The testing of the new items was performed on the same group of articles tested for satisfying the STROBE. However, we believe that, since the postdevelopment test was performed only for observational reports, no specific bias was introduced. Generalisability of our study applies therefore only to epidemiological studies exploring the association between previous antibiotic therapy and development of infection and/or colonisation due to antimicrobial-resistant strains.

Our study has shown, through an expert's Delphi approach, that the current version of the STROBE tool does not describe all the components considered essential to define the association between antibiotic usage and resistance. That these studies should be conducted and reported effectively has become an imperative, given the global threat imposed by antimicrobial-resistant microorganisms, for future effective therapies.1 18 19 Increasing proper reporting will reduce heterogeneity between papers, and assist in the evaluation of the evidence through systematic reviews and peer reviewing for journal and grant proposals. The implementation of the STROBE-AMS will also impact on other aspects of antimicrobial-resistant research, including study design, extraction of data and generalisability of results. We do believe that the introduction of STROBE-AMS will increase the quality of available evidence to policymakers, relevant healthcare workers and the research community, and ultimately ensure that there are sustained improvements to AMS throughout the world.