A methodological survey on reporting of pilot and feasibility trials for physiotherapy interventions: a study protocol.

INTRODUCTION: Pilot and feasibility trials aim to test whether a full trial can be conducted or if any procedures must be changed for the full trial. Pilot trials must be reported in a transparent, accurate and complete way. In this report, we present a protocol for a methodological survey with the following aims: (1) to determine the percentage of physiotherapy trial reports which claim to be pilot or feasibility trials that evaluate feasibility, (2) to determine the aspect of feasibility evaluated in the primary objectives of the pilot or feasibility trials, (3) to describe the completeness of reporting of abstracts and full articles of pilot or feasibility trials using the Consolidated Standards of Reporting Trials (CONSORT) extension to randomised pilot and feasibility trials and (4) to investigate factors associated with completeness of reporting of pilot or feasibility trials. METHODS AND ANALYSIS: Reports of randomised controlled trials indexed in the Physiotherapy Evidence Database (PEDro) that claim to be pilot or feasibility trials and published in 2011-2017 will be included. Two independent reviewers will confirm eligibility and classify the aspect of feasibility being evaluated in the objectives of the included pilot or feasibility trials. Completeness of reporting of both the abstract and the full article will be evaluated using the CONSORT extension to randomised pilot and feasibility trials. The primary analysis will be a descriptive analysis about the reporting quality of abstracts and full texts of pilot and feasibility trials. We will use generalised estimating equation analysis to explore factors associated with completeness of reporting. ETHICS AND DISSEMINATION: The results of this study will be disseminated by presentation at conferences and will be submitted for publication in a peer-reviewed journal. Ethical approval is not necessary for this study. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This study will be the first to evaluate the quality of reporting of abstracts and full articles of pilot or feasibility trials in the field of physiotherapy using the Consolidated Standards of Reporting Trials statement extension for pilot and feasibility studies.

All data will be extracted by two independent reviewers in order to increase precision.

Findings from this study are restricted to pilot and feasibility trials published between 2011 and 2017 indexed in Physiotherapy Evidence Database. Therefore, the results of this study cannot be generalised to all existing pilot and feasibility trials in physiotherapy.

Introduction

Pilot and feasibility trials are exploratory studies that aim to investigate whether the crucial components of planning or preparing for a larger and definitive randomised controlled trial will be viable.1 2 They are intended to provide useful information with regard to planning complex interventions (eg, identifying the optimal dose and testing safety), testing study procedures (eg, the form of randomisation, estimation of the recruitment rate and the plausibility of multicentre collaborations), investigating surrogate outcomes or estimating parameters to help perform sample size calculation.3 4 Although pilot and feasibility trials have slightly different definitions, both are designed to establish whether the main or definitive trial can and should be conducted in the future, and, if so, to determine how the main trial should be done.5 Pilot and feasibility trials are designed to ensure that the main trial will be achievable, rigorous and economically justifiable in order to avoid waste of resources.4 However, without a clear understanding of how the pilot or feasibility trial was conducted, researchers and clinicians would not be able to judge the methodological quality and to clinically appraise the published report of the trial.

Evaluations of published pilot and feasibility trials suggests that the trials may not actually be evaluating feasibility6 7 and are being poorly reported.7 In a small sample of 93 pilot and feasibility trials published in Indian biomedical journals, 68% of the trials performed between-group statistical comparisons and none reported feasibility objectives.7 In addition, an ad hoc list of trial characteristics was used to evaluate reporting, rather than a scale or a checklist. Another survey of 191 pilot and feasibility trials published in 1987–2015 in a single journal (ie, Clinical Rehabilitation)6 revealed that 110 (58%) trials actually tested feasibility for a future trial, with only 23 trials being followed by a definitive trial.6 This implies that the terms ‘pilot’ or ‘feasibility’ may be being incorrectly used to assist in the publication of small trials rather than to systematically test procedures to inform the conduct of a large definitive trial. Since those evaluations were published, the methods for evaluating the quality of reporting of pilot and feasibility trials have improved substantially with the introduction of an extension of the Consolidated Standards of Reporting Trials (CONSORT) statement specifically for randomised pilot and feasibility trials.8 9 This extension consists of a 40-item checklist for full articles and a 16-item checklist for abstracts.

The completeness of reporting of full published reports of randomised controlled trials10–15 and the abstracts of trials16–18 has been evaluated to be suboptimal across all areas of healthcare, including in physiotherapy.11 13–15 19 Factors that appear to be associated with improved reporting quality or methodological quality include publication in a journal with a high-impact factor,20–22 being a multicentre trial,21 23–25 a higher number of authors,21 26 publication in a journal that endorses the CONSORT statement,12 13 19 language of the publication,15 21 25 discipline of physiotherapy,27 year of publication,13 28 receiving grants from research funding agencies,14 sample size14 29 and evidence of clinical trial registration.13 19

The number of randomised controlled trials in physiotherapy has grown exponentially over time.30 Time and funding are resources that could be saved by conducting high-quality pilot and feasibility studies. To our knowledge, the reporting quality of pilot and feasibility trails of physiotherapy interventions using the new extension of the CONSORT statement for randomised pilot and feasibility trials has not yet been performed, and the factors associated with better reporting have not been identified.

Objectives

The purpose of this methodological survey is to describe the quality of reporting of abstracts and full articles of pilot or feasibility trials from a representative sample in the field of physiotherapy. Specifically, the first aim is to determine the percentage of trial reports indexed in the Physiotherapy Evidence Database (PEDro), which claim to be pilot or feasibility trials that evaluate feasibility. The second aim is to determine the aspect of feasibility evaluated in the primary objectives of the true pilot or feasibility trials. The third aim is to describe the completeness of reporting of abstracts and full articles using the CONSORT extension for randomised pilot and feasibility trials. The fourth aim is to investigate factors associated with completeness of reporting of pilot or feasibility trials.

Methods

Study design

This study is a methodological survey of completeness of reporting of abstracts and full articles of pilot or feasibility trials for physiotherapy interventions.

Eligibility criteria

We will include all reports of randomised controlled trials indexed in PEDro (http://www.pedro.org.au) that claim to be a pilot or feasibility trial. We will only include trials published in 2011–2017 that are fully indexed in PEDro (in-process trials, which have not had search terms and PEDro scores allocated, will not be included). We decided to only include trials published after 2010 because the International Committee of Medical Journal Editors stated that all trials started after July 2005 should be registered in a free, publicly available and electronically searchable register31 32 and also because the last update of the CONSORT statement was published in 2010. There will be no language restrictions.

We selected PEDro as the source of trial reports because PEDro is one of the most comprehensive indexes of reports of randomised controlled trials evaluating physiotherapy interventions.33 34 Moreover, all trials indexed on PEDro are rated for methodological quality and completeness of statistical reporting using the PEDro scale,35 and are coded for the area (or subdiscipline) of physiotherapy practice and type of intervention. To be eligible for inclusion on PEDro, trials must involve comparison of at least two interventions (or an intervention and control condition) applied to subjects who are representative of those whom the interventions might be applied to in the course of clinical practice, with at least one of the interventions under evaluation being part of physiotherapy practice. In addition, all trials included in PEDro must involve random (or intended-to-be-random) allocations of subjects into interventions and must be fully published in a peer-reviewed journal.36

Search strategy

To identify reports of pilot or feasibility trials, a search on PEDro will be conducted for the period from 2011 to 2017. We will use ’Clinical trial' in the Method field, combined with the following search terms in the Abstract and the Title field: Pilot* OR Feasibility* OR Vanguard* OR “Dress rehearsal”.

Study selection

Two independent reviewers will screen titles and abstracts to identify references that claim to be a pilot or feasibility trial. The title, abstract and, if necessary, full-text of these self-identified pilot or feasibility trials will be evaluated to identify the subset of articles that contain objectives linked to feasibility. Any disagreements between reviewers will be resolved by discussion or, if necessary, arbitration by a third reviewer. Figure 1 presents the flow diagram used to guide article selection.

Figure 1

Study flow diagram. PEDro, Physiotherapy Evidence Database.

Data extraction

Two independent reviewers will classify the reasons to conduct a pilot or feasibility for each included trial. There are several reasons for conducting pilot and feasibility studies. These reasons could be grouped under some broad classifications.37 A widely known tutorial4 aimed to provide a detailed examination of the key aspects of pilot studies suggested four categories to classify the rationale to conduct a pilot study. The four categories are (1) process (steps that need to take place as part of the main study), (2) resources (time and budget), (3) management (human and data optimisation) and (4) scientific (issues such as treatment safety).4 If more than one category is identified for an included trial, we will code for all relevant categories and indicate which category is linked to the primary objective of the trial. The number of subjects randomised and whether the pilot or feasibility trial recommends that a large-scale trial will be conducted will also be recorded.

The two independent reviewers will also complete the CONSORT pilot and feasibility trials checklist (40 items) and the CONSORT pilot and feasibility trials abstracts checklist (16 items) (note that the ‘author’ item was omitted as this relates to conference abstracts only) for each trial.8 9 The CONSORT checklist includes items related to the title, trial design, methods, results, conclusions, registration and funding. Each item will be rated as ‘reported’, ‘inadequately reported’, ‘not reported’ or ‘not applicable’. Summary scores for the CONSORT pilot and feasibility trials checklist (range 0–40) and the CONSORT pilot and feasibility trials abstracts checklist (range 0–16) will be calculated by tallying the items scored as reported.

The reviewers will independently extract the data using an electronic data extraction form designed for this survey. The data extraction form will be created using information from the CONSORT extension to randomised pilot and feasibility trials. We will pilot the data extraction forms on 10 randomly selected trials before proceeding with full data extraction to ensure all reviewers extract data consistently and to ensure the data extraction form is unambiguous and free from errors. Discrepancies between the two reviewers will be resolved by discussion and by consulting the published explanation of the CONSORT checklists. If necessary, arbitration by a third reviewer will help to provide consensus on the data extracted. In order to improve the clarity regarding inclusions and exclusions and to increase accuracy and consistency among the reviewers, between-reviewer agreements will be measured using the kappa coefficients using an initial trial run involving 10 articles per reviewer. If adequate reliability will be not achieved, additional training or improvement in the data extraction form will be undertaken.

PEDro scale scores, subdiscipline of physiotherapy, intervention, language of publication and year of publication will be downloaded from PEDro. The PEDro scale is an 11-item scale that measures the methodological quality and completeness of statistical reporting of reports of randomised controlled trials.35 The items are (1) eligibility criteria and source of subjects, (2) random allocation, (3) concealed allocation, (4) baseline comparability, (5) blinding of subjects, (6) blinding of therapists, (7) blinding of assessors, (8) >85% follow-up, (9) intention-to-treat analysis, (10) between-group statistical comparisons, and (11) reporting of point measures and measures of variability.35 Each item is rated as ‘yes’ (unambiguously achieved) or ‘no’, with the number of yes responses for items 2–11 tallied to obtain the total PEDro score (out of 10). Both the individual items (coded as ‘0’ for no or ‘1’ for yes) and the total PEDro score (range 0–10) will be downloaded. There is evidence that the PEDro scale has higher reliability for individual ratings and consensus ratings compared with the Cochrane risk of bias.35 38 Also, the PEDro scale is strongly correlated (r=0.83, 95% CI 0.76 to 0.88) with the Cochrane risk-of-bias scale.39 On the other hand, a meta-epidemiological study found discrepancies in terms of clinical trial quality using PEDro and the Cochrane risk-of-bias scale.40 The subdiscipline of physiotherapy codes are cardiothoracic, continence and women’s health, ergonomics and occupational health, gerontology, musculoskeletal, neurology, oncology, orthopaedics, paediatrics and sports, or no appropriate value. Each trial can be assigned up to three codes for subdiscipline; in our study, we will select the most applicable subdiscipline and this will be coded as a dummy variable. The language of publication will be coded to produce two different variables: '1' for English and '0' for languages other than English, and Chinese as ‘0’ and all other languages as ‘1’. The year of publication will be subtracted from 2017 to produce an ‘age’ (in years) for each trial.

One reviewer will determine if the trial was registered and if the journal of publication for each trial endorses the CONSORT statement. Clinical trial registration will be extracted from the full article or, if not reported in the full article, by searching the International Clinical Trials Registry Platform (http://apps.who.int/trialsearch/) and will be coded as 1 for yes or 0 for no. Journal endorsement of the CONSORT statement will be achieved by reviewing the list of journals on the CONSORT website41 and, if necessary, by visiting journal websites and reviewing the instructions for authors and other editorial policies.

One reviewer will collect the journal impact factor if available at the time of pilot trial publication (as a continuous variable) through a search at the Journal Citation Reports website (https://jcr.incites.thomsonreuters.com). Other variables, including number of authors (as a continuous variable), source of funding, declaration of conflict of interests and sample size (as a continuous variable), will be collected by one reviewer through the electronic data extraction form designed for this review.

Patient and public involvement

Patients and the public were not involved in this study.

Statistical analysis

First, we will calculate the number, percentage and 95% CI of trials indexed in PEDro that claim to be a feasibility or pilot trial that evaluates feasibility. The PEDro CI calculator will be used to calculate the 95% CI.42 We will also compute the aspect of feasibility evaluated in the primary objectives of the pilot or feasibility trials.

The primary analysis will be a descriptive analysis of completeness of reporting of the abstracts and full articles of the pilot or feasibility trials. The frequency that each item is scored as reported, inadequately reported, not reported and not applicable for the CONSORT pilot and feasibility trials checklist and the CONSORT pilot and feasibility trials abstracts checklist will be tabulated. The mean (SD) summary score will be calculated for each checklist.

In the secondary analysis, we will perform a Poisson regression analysis to determine which study characteristics are associated with greater completeness reporting. Two independent models will be built, one using the summary score for the CONSORT pilot and feasibility trials checklist (ie, for the full article) as the dependent variable and the second model using the summary score for the CONSORT pilot and feasibility trials abstracts checklist. Independent variables for both models will be (1) publication in a journal that endorses CONSORT12 13 (1 for yes or 0 for no), (2) trial funded14 (1 for yes or 0 for no), (3) sample size14 (as a continuous variable), (4) reported trial registration number (1 for yes and 0 for no), (5) total PEDro score (continuous variable, 0–10), (6) most applicable subdiscipline of physiotherapy27 (coded as dummy variables), (7) language of publication (1 for English and 0 for all other languages), (8) non-Chinese reports (1 for yes and 0 for ‘trials published in languages other than Chinese’), (9) number of authors (continuous variable) and (10) reporting allocation concealment (PEDro scale item 3, 1 for yes and 0 for no).

We will use generalised estimating equation (GEE) analysis, assuming an exchangeable correlation structure, to explore factors associated with completeness of reporting. GEE allows us to model a possible correlation or similarity of the papers published within the same journal. All analyses will be performed using SAS 9.2 (Cary, NC).

Discussion

This study will be the first to describe the completeness of reporting of pilot or feasibility trials from a representative sample in the field of physiotherapy using the CONSORT extension to randomised pilot and feasibility trials. This is important as good reporting, or transparency, will provide sufficient information about the methods and results of the trial to guide clinical practice and further research to both clinicians and researchers.

The transparency in reporting randomised controlled trials has improved since the introduction of the CONSORT statement.12 A number of other factors are also associated with better trial quality, including being funded,14 being prospectively registered,13 being published in English15 and having larger sample sizes.14 29 Whether these variables are also associated with better reporting quality of pilot or feasibility studies has not been rigorously investigated. To the best of our knowledge, only one study has evaluated quality of reporting of pilot studies.7 That study did not use a scale or checklist to evaluate reporting nor did it test for possible factors that could predict quality.7