Reporting quality of stepped wedge design randomized trials: a systematic review protocol.

BACKGROUND: Stepped wedge design (SWD) is a cluster randomized controlled trial (RCT) design that sequentially rolls out intervention to all clusters at varying time points. Being a relatively new design method, reporting quality has yet to be explored, and this review will seek to fill this gap in knowledge.
OBJECTIVES: The objectives of this review are: 1) to assess the quality of SWD trial reports based on the CONSORT guidelines or CONSORT extension to cluster RCTs; 2) to assess the completeness of reporting of SWD trial abstracts using the CONSORT extension for abstracts; 3) to assess the reporting of sample size details in SWD trial reports or protocols; 4) to assess the completeness of reporting of SWD trial protocols according to SPIRIT guidelines; 5) to assess the consistency between the trial registration information and final SWD trial reports; and 6) to assess the consistency of what is reported in the abstracts and main text of the SWD trial reports. We will also explore factors that are associated with the completeness of reporting.
METHODS: We will search MEDLINE, EMBASE, Web of Science, CINAHL, and PsycINFO for all randomized controlled trials utilizing SWD. Details from eligible papers will be extracted in duplicate. Demographic statistics obtained from the data extraction will be analyzed to answer the primary objectives pertaining to the reporting quality of several aspects of a published paper, as well as to explore possible temporal trends and consistency between abstracts, trial registration information, and final published articles. DISCUSSION: Findings from this review will establish the reporting quality of SWD trials and inform academics and clinicians on their completeness and consistency. Results of this review will influence future trials and improve the overall quality and reporting of SWD trials.

Background

Our reliance on the medical literature to inform practice and recommendations put forward in clinical guidelines demands the need for high-quality reporting of randomized trials. The introduction of checklists, such as the CONSORT statement which was introduced in 1996 and revised in 2010, promotes completeness and transparency in randomized controlled trials (RCTs).1 However, although endorsed by prominent medical journals such as The Lancet and New England Journal of Medicine, authors still fail to give full disclosure on their RCTs through incomplete reporting in the abstracts or in main text of the final reports, which has potential to result in harm to the patients.2 Furthermore, many clinicians in developing countries may not have full access to academic journals and the papers within them, and may rely on publically accessible abstracts to assist their medical judgments.3

Selective reporting of outcomes is a problem that not only affects the reliability of the published study but also reviews that include them.4 One study found that 65% of harm outcomes in 122 published journal articles were incompletely reported.4 In order to have complete, reliable reporting in all trials, consistency between abstracts, trial registration information, and the full published article is paramount. This study seeks to identify the gaps in reporting consistency and evaluate the relationship between complete reporting of the published article and consistency.

A systematic scoping review of reviews on the adherence of reporting guidelines found that 86.0% of the 50 studies reviewed reported suboptimal levels of adherence to reporting guidelines.5 The study emphasizes the need for action on the adherence of reporting guidelines. Furthermore, a paper on the reporting quality of alcohol outcome studies found that journals that had adopted the CONSORT statement showed significant improvements in the reporting of key elements of alcohol outcome studies.6 However, previous work focused primarily on standard RCTs, and the quality of reporting of stepped wedge design (SWD) trials is largely left unreported.

SWD is a cluster RCT design that sequentially rolls out intervention to all clusters at varying time points.7 At the end of a SWD study, all groups will have received intervention. This study design provides strategic advantage by preventing ethical complications that coincide with RCTs where the treatment is considered to be beneficial, leaving one arm of the study to continue receiving suboptimal therapy. This design also allows for unique statistical analysis, as the duration of intervention for each group is variable.7 However, being that it is a newer trial design method, it has not received the volume of a real-world experience that other commonly used RCT designs have been exposed to, which creates possibility for not only methodological errors but also reporting inadequacies. In this study, we will systematically review published trials and protocols utilizing SWD, and assess and report our findings on the quality of reporting of these SWD trial reports.

A summary of all SWD trials was previously published in systematic reviews by Beard et al8 and Mdege et al,9 both of which provide comprehensive reviews of any SWD randomized trials performed since the inception of this novel methodological approach. Each review provides a detailed summary of the trials included, interventions used, reasoning behind the use of SWD, acceptability of the intervention, as well as variations in analysis techniques. While reporting quality is described in both reviews, neither has taken a systematic approach to examining 1) reporting quality using adapted CONSORT guidelines for cluster RCT; 2) reporting quality among protocols and SWD trial abstracts; 3) the areas impacting completeness of reporting; 4) consistencies between final published trials with original protocols or trial registry profiles; and 5) a specific focus on the information provided for sample size estimation. These are the key gaps we aim to fill with our systematic review. Reviews by Beard et al8 and Mdege et al9 emphasize on the interventions, acceptability of the study design, and current medical fields uptaking the SWD. Neither review has explored the standards of transparency and consistencies in the reporting of such trials.

Objectives

The objectives of this methodological systematic review are the following: 1) to assess the reporting quality of SWD trial reports based on the CONSORT guidelines1 or CONSORT extension to cluster RCTs;10 2) to assess the completeness of reporting of SWD trial abstracts using the CONSORT extension for abstracts; 3) to assess the reporting of sample size details in SWD trial reports or protocols; 4) to assess the completeness of reporting of SWD trial protocols according to SPIRIT guideline;4 5) to assess the consistency between the trial registration information and what is reported in the final SWD trial reports; and 6) to assess the consistency of what is reported in the abstracts and main text of the SWD trial reports. We will also explore factors that are associated with the completeness of reporting. These factors will include: whether or not the journal has endorsed corresponding guideline, funding source for the trial (industry vs nonindustry), high impact factor, multicentered study, number of authors, reporting of allocation concealment, reporting method of sequence generation, sample size, trial quality, type of intervention (pharmacological vs nonpharmacological), year of publication (before or after CONSORT statement publication in 1996), and origin of report (country of origin).10

Methodology

Eligibility criteria

Literature search will involve all RCTs employing SWD since 1987 (ie, the date of publication of the first SWD trial). We are interested in completed reports or protocols of RCTs utilizing SWD that are written in English and have been published in a peer-reviewed journal.

Search strategy

An extensive search for SWD trials and protocols will be performed by the lead author on the following electronic databases: MEDLINE, Web of Science, CINAHL, EMBASE, and PsycINFO. The specific search strategies tailored to each database, outlined in Table 1, will be used to identify potential trial reports or protocols that meet the eligibility criteria. Cited materials found in participating articles relevant to the systematic review will be reviewed for potential participation as well. A single reviewer will process the articles in the first stage of the review involving searching the chosen databases using the keywords described previously. After the removal of any duplicates from the preliminary search, the titles, abstracts, and full texts generated will then be reviewed by two independent reviewers to determine study eligibility. When differences arise, reviewers will discuss and come to a consensus; if an agreement cannot be reached between both reviewers, a third party will help provide consensus as to the article’s eligibility. The systematic review will be conducted with adherence to the Preferred Reporting Items for Systematic reviews and Meta-Analysis (PRISMA) guidelines to ensure adequate reporting quality.12 As a methodological systematic survey, this study is not registered with PROS-PERO. Please refer to Figure 1 for the PRISMA flow diagram used to guide article selection.

Table 1

Search strategy of electronic databases

Database	Search terms
MEDLINESearch details = __________	1. “stepped wedge design”[All Fields] OR “stepped wedge”[All Fields] OR “wedge design”[All Fields] OR (stepped[All Fields] OR “experimentally staged introduction”[All Fields] AND wedge[All Fields] AND design[All Fields] AND protocol[All Fields]) OR (stepped[All Fields] AND wedge[All Fields] AND protocol[All Fields]) OR (stepped[All Fields] AND wedge[All Fields] AND design[All Fields] AND (“clinical trials as topic”[MeSH Terms] OR (“clinical”[All Fields] AND “trials”[All Fields] AND “topic”[All Fields]) OR “clinical trials as topic”[All Fields] OR “trial”[All Fields])) OR (wedge[All Fields] AND design[All Fields] AND (“clinical trials as topic”[MeSH Terms] OR (“clinical”[All Fields] AND “trials”[All Fields] AND “topic”[All Fields]) OR “clinical trials as topic”[All Fields] OR “trial”[All Fields])) OR “stepped wedge trial”[All Fields] OR (stepped[All Fields] AND wedge[All Fields] AND design[All Fields] AND abstract[All Fields]) OR (stepped[All Fields] AND wedge[All Fields] AND (“clinical trials as topic”[MeSH Terms] OR (“clinical”[All Fields] AND “trials”[All Fields] AND “topic”[All Fields]) OR “clinical trials as topic”[All Fields] OR “trial”[All Fields]) AND abstract[All Fields]) AND (“1987/01/01”[PDAT] : “3000/12/31”[PDAT])
Web of ScienceSearch = __________	1. TOPIC: (“stepped wedge design” OR TOPIC: [“stepped wedge”] OR TOPIC: [“experimentally staged introduction”] OR TOPIC: [“stepped wedge design trial”] OR TOPIC: [“stepped wedge design protocol”] OR TOPIC: [“stepped wedge trial”] OR TOPIC: [“stepped wedge protocol”] OR TOPIC: [“stepped wedge design abstract”] OR TOPIC: [“wedge design”] OR TOPIC: [“wedge design trial”] OR TOPIC: [“wedge design protocol”])
	2. Timespan: 1987–2015
	3. Indexes: SCI-EXPANDED, SSCI, A&HCI, CPCI-S, CPCI-SSH
CINAHLSearch = __________	1. Suggest subject terms: stepped wedge design OR stepped wedge OR experimentally staged introduction OR stepped wedge trial OR stepped wedge proctor OR stepped wedge design trial OR stepped wedge design protocol OR wedge design OR wedge design trial OR wedge design protocol OR stepped wedge design abstract OR wedge design abstract OR stepped wedge abstract
EMBASE (Ovid interface)Search = __________	1. “stepped wedge design” OR “stepped wedge” or “stepped wedge design trial” or “stepped wedge design protocol” or “stepped wedge design abstract” “experimentally staged introduction” or “stepped wedge trial” or “stepped wedge protocol” or “stepped wedge abstract” or “wedge design” or “wedge design trial” or “wedge design protocol” or “wedge design abstract”
	2. Limit to human
	3. Limit to publishing year =1987
PsycINFO (Ovid interface)Search = __________	1. “Stepped wedge design” or “stepped wedge” or “stepped wedge design trial” or “experimentally staged introduction” or “stepped wedge design protocol” or “stepped wedge design abstract” or “stepped wedge trial” or “stepped wedge protocol” or “stepped wedge abstract” or “wedge design” or “wedge design trial” or “wedge design protocol” or “wedge design abstract”
	2. Limit to human
	3. Limit to publishing year =1987

Figure 1

The Preferred Reporting Items for Systematic reviews and Meta-Analysis (PRISMA) flow diagram for article selection.

Abbreviation: SWD, stepped wedge design.

Data collection and outcomes

Data extraction forms will be created from the CONSORT statement and extensions (abstract, cluster RCT), SPIRIT guidelines (protocol), as well as a checklist for what we propose for the transparent reporting of sample size details (Supplementary material; Additional File 1). We will pilot test screening and data extraction forms. Discussions will be held with all parties involved (reviewers and methodologists included in this review) to determine whether the description of tasks and items is clear to reviewers carrying out screening and extraction. A kappa statistic calculation will be performed for each stage of screening to determine the agreement between the independent reviewers. Data from each article will be extracted in duplicate and inputted into an Excel spreadsheet that will contain all the extracted data. In specific, we will be extracting the author name, title of article, article type (trial report, protocol), and all applicable criteria pertaining to the CONSORT and SPIRIT guidelines for the corresponding extraction form. The primary outcomes of the data abstraction and analysis will be to assess and quantify the areas of reporting quality in SWD trials, protocols, and abstracts, and make a conclusion as to whether the reporting of these articles, according to CONSORT and SPIRIT guidelines, is of adequate quality. Consistency between trial registration information and abstracts to the final manuscript will also be assessed and reported as a percentage and count (eg, number of articles with inconsistencies between trial registration and final published paper for X variable).

Additional items identified as both important to SWD but not commonly disclosed according a recent review by Davey et al13 will be added to our data extraction forms. These items include: detailed cluster/patient retention rate, specification of the time trend utilized in analytical approach, and discussion effect of variance over cluster and time point evaluation. Transparency in the reporting of sample size–related items has been largely left out of previous reviews evaluating SWD. We aim to focus on many of these items including sample size criterion for statistical significance, desired level of statistical significance, desired power, as well as assumptions made about distribution or variability of the outcome. Please refer to Additional File 1 for a complete list of sample size–related items.

We will contact all corresponding authors for articles identified as eligible to ensure we have abstracted data from their studies correctly. We will provide each author an excel file detailing results from the data extraction of their SWD trial.

Statistical analyses

Descriptive statistical calculations (mean, median, standard deviation, range), as well as percentile calculations of the data, will examine and determine the proportion of SWD articles that are reported suboptimally and will reveal areas of reporting in need of quality improvement. Calculations by year will also determine the mean number of checklist items reported using estimate unadjusted and adjusted differences via two-sample t-test and estimation equations. Statistical adjustments will be made accordingly for each trial to account for potential clustering or similarity in articles published in the same journal. Kappa statistic calculations will be conducted to determine the agreement. We will consider kappa statistic values between 0.40 and 0.59 to represent fair agreement, values between 0.60 and 0.74 to represent good agreement, and values of 0.75 or above reflecting excellent agreement.14 We will also create summary tables for analysis, which will include objective, outcome, hypothesis, and method of analysis columns. All statistical analyses will be conducted using STATA version 13 (StataCorp LP, College Station, TX, USA).15 Comparisons to similar studies on reporting the quality of articles will provide a perspective on how the reporting quality of studies utilizing SWD compare to other design methods.

Quality assessment

As this is a review of methodological reporting quality, we will not be assessing trial quality, but rather we will specifically look at the reporting quality of the trials. Clarity and completeness will be assessed in conjunction with the checklists, as unclear or incomplete reporting may affect the reporting quality. Risk of biases will be calculated in duplicate using the Cochrane Collaboration’s tool for assessing the risk of bias in randomized trials.16 Risk of bias items specific to cluster trials will be evaluated, including whether individual studies accounted for 1) clustering during analysis; 2) clusters omitted during analysis due to center dropout, prognostic imbalance between clusters or individuals; or 3) recruitment bias. Any disagreements will be settled with by discussion among team members and in the event the discussion cannot lead to resolution, a third party will be asked to review the disagreement. Categorical scores of “high”, “low”, and “unclear” will be determined for each paper to decide their risk of bias.14

Discussion

Accurate and complete reporting of clinical trials is important in the interpretation of studies and the advancement of scientific and medical knowledge. This study expects that many aspects of reporting quality will be omitted as seen from other studies, but as more journals and editors begin to enforce guidelines, reporting quality will improve,17 and we expect to see that in our yearly data. Suboptimal reporting of clinical trials and protocols results in risk of biases and reduces their value for both clinicians and researchers alike.18 Once completed, the systematic review will offer information to both readers and the authors of SWD reports as to their reporting quality and will identify commonly underreported aspects of SWD trials. This review will offer unique insight and exposure into the design method and promote the transparency of future SWD articles. This insight will allow physicians and researchers over the world to make proper interpretations of SWD studies, and make appropriate medical and academic judgments of the paper.