A methodological approach for assessing the uptake of core outcome sets using ClinicalTrials.gov: findings from a review of randomised controlled trials of rheumatoid arthritis.

Objective To assess the uptake of the rheumatoid arthritis core outcome set using a new assessment method of calculating uptake from data in clinical trial registry entries.Design Review of randomised trials.Setting ClinicalTrials.gov.Subjects 273 randomised trials of drug interventions for the treatment of rheumatoid arthritis and registered in ClinicalTrials.gov between 2002 and 2016. Full publications were identified for completed studies from information in the trial registry or from an internet search using Google and the citation database Web of Science.Main outcome measure The percentage of trials reporting or planning to measure the rheumatoid arthritis core outcome set calculated from the information presented in the trial registry and compared with the percentage reporting the rheumatoid arthritis core outcome set in the resulting trial publications.Results The full rheumatoid arthritis core outcome set was reported in 81% (116/143) of trials identified on the registry as completed (or terminated) for which results were found in either the published literature or the registry. For trials identified on the registry as completed (or terminated), using information only available in the registry gives an estimate for uptake of 77% (145/189).Conclusions The uptake of the rheumatoid arthritis core outcome set in clinical trials has continued to increase over time. Using the information on outcomes listed for completed or terminated studies in a trial registry provides a reasonable estimate of the uptake of a core outcome set and is a more efficient and up-to-date approach than examining the outcomes in published trial reports. The method proposed may provide an efficient approach for an up-to-date assessment of the uptake of the 300 core outcome sets already published. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

Introduction

The selection of appropriate outcomes is crucial to the design of randomised trials. If the findings of a trial are to influence healthcare, the outcomes that are measured and reported need to be relevant to patients, healthcare professionals, and others making decisions about healthcare provision. A core outcome set has previously been defined as an agreed standardised set of outcomes that should be measured and reported, as a minimum, in all clinical trials in specific areas of health or healthcare.1 Core outcome sets can enhance the relevance of research by ensuring outcomes of importance to health service users and other people making choices about healthcare in a particular setting are measured routinely.2 The adoption of a core outcome set can reduce heterogeneity in reported outcomes between trials and reduce the risk of outcome reporting bias, since trial reports would always include a presentation of the findings of a core outcome set, as a minimum.1

The OMERACT (Outcome Measures in Rheumatology) Initiative advocates the use of a core outcome set and strives to improve outcome measures in musculoskeletal conditions through data driven multi-stakeholder consensus processes.3 A brief history of OMERACT is provided elsewhere.4 After the first OMERACT conference in 1992, the World Health Organization and International League of Associations for Rheumatology (ILAR) ratified a core outcome set for clinical trials of symptom modifying antirheumatic drugs in rheumatoid arthritis. The WHO-ILAR rheumatoid arthritis core outcome set was published in 1994 and consisted of seven outcomes (tender joints, swollen joints, pain, physician global assessment, patient global assessment, physical disability, and acute phase reactants), and one additional outcome (radiographs of the joints) for studies lasting one year or more.5

Assessing the uptake of a core outcome set allows the impact of research on the development of core outcome sets to be assessed. The uptake of the rheumatoid arthritis core outcome set has been previously assessed using a sample of 204 randomised trials of drug treatments identified from those included in 31 Cochrane Reviews (published on the Cochrane Library up to September 2012 issue) of interventions for rheumatoid arthritis.6 These reviews included trials that were published between 1955 and 2009. Over time there was an increase in the percentage of trials reporting the core outcome set items, with almost 70% measuring all these outcomes in trials that were published at the end of the first decade of the 21st century. However, assessing the uptake of a core outcome set in this way can be a lengthy process because each individual trial report needs to be found and examined. Moreover, many systematic reviews can be several years old, meaning that the most up-to-date trials might not be included in the assessment.

We investigated the use of trial registries as a more efficient approach and up-to-date resource for assessing the uptake of a core outcome set, using the rheumatoid arthritis core outcome set as our target example. We compared the uptake rates obtained by examining the trial registry entries with those obtained by checking the published reports of completed studies that had been registered in the registry, and examined whether the uptake of the rheumatoid arthritis core outcome set has improved since our previous study. With over 300 core outcome sets already published for different health and healthcare settings,7 the new methodological uptake approach proposed in this research article has relevance for those from rheumatology and non-rheumatology communities to evaluate the uptake of core outcome sets in their area. Evaluation of uptake is crucial to avoid core outcome sets being developed but never used, thus contributing to research waste,8 the very problem they are designed to tackle.

Methods

Assessment of trial registry entries

We searched the trials registry ClinicalTrials.gov on 6 October 2016 to identify all phase III/IV drug clinical trials of rheumatoid arthritis that had been registered on the site. To identify potentially relevant trials we applied the following filters: “conditions: rheumatoid arthritis”, “study type: interventional studies”, and “phase: 3 and 4”. The returned hits were exported into a spreadsheet and further filters were applied based on additional mandatory condition and study design fields recorded in the registry entries. We excluded trial registry entries if the trial was not exclusive to participants with rheumatoid arthritis (eg, also contained participants with osteoarthritis), did not consider efficacy as an endpoint (eg, were safety studies, or pharmacokinetic, pharmacodynamic, or immunology studies only), considered a non-drug intervention or device, were non-randomised studies, were diagnostic test accuracy studies, or were studies where all participants received the same intervention (single group assignment). We applied these exclusions because these types of studies were beyond the scope of the current rheumatoid arthritis core outcome set.

For each eligible trial registry entry, we extracted information on all planned trial outcomes and assessed whether the full rheumatoid arthritis core outcome set was listed. If trialists had registered a composite outcome, such as the American College of Rheumatology improvement criteria,9 we considered all the individual outcomes in the composite in the assessment, even if they were not listed separately. For example, if the American College of Rheumatology 20 criteria were specified and the trial was less than 52 weeks in duration, then we assumed the full rheumatoid arthritis core outcome set was assessed.

Assessment of trial reports

We searched for trial publications for all eligible trials that had been identified on the trial registry. We found relevant publications either directly from their listing in the trial registry entry, through a Google search for the clinical trial registry number (limited to the first three pages of Google hits), or through a search of the clinical trial registry number on a citation database, Web of Science. Publications that included the clinical trial registry number of a trial but did not report on the trial findings were excluded. An assessment of whether the full rheumatoid arthritis core outcome set was reported in each trial publication was carried out in the same way as for the trial registry entries. PRW checked a random sample of 10% of the trial registry entries and publications, which showed agreement with another independent assessor (JJK). One reviewer (JJK), with experience in the assessment of the uptake of the rheumatoid arthritis core outcome set then carried out the remaining assessments.6

Assessment of uptake of the rheumatoid arthritis core outcome set

Several measures of uptake were of interest, using data from either trial results, trial registry entries, or a combination.

The percentage of trials that reported data on the rheumatoid arthritis core outcome set for trials identified in the registry as completed (or terminated) where results were found either in a publication or in the trial registry. This is the gold standard approach and requires the most work to obtain the uptake estimate, as all publications and trial results from the registry need to be found and read.

The percentage of trials that reported or planned to measure data on the rheumatoid arthritis core outcome set for trials identified in the registry with results found in the registry (either in a publication listed in the registry entry or in the trial registry). If the results were not found in the registry entry, the information on planned outcomes to be measured is taken from the trial registry entry. This method uses only information from the registry and involves reading the publications identified in the registry. It allows all eligible trials identified from the registry to be included in the evaluation of uptake.

The percentage of trials that planned to measure data on the rheumatoid arthritis core outcome set for registered trials regardless of trial or publication status, based solely on the outcomes listed in the trial registry entry. This method allows all eligible trials identified from the registry to be included in the evaluation of uptake regardless of whether they are ongoing or completed (or terminated) and does not require the reading of any publications.

The percentage of trials that reported or planned to measure data on the rheumatoid arthritis core outcome set for trials identified in the registry as completed (or terminated), based on the information in the trial registry. The aim of this approach is to estimate uptake for completed (or terminated) trials using only the trial registry information and not from any wider search.

In our updated assessment of how the measurement of core outcomes had changed over time, we combined the data from the trial publications from the previous assessment (systematic review approach)6 with the data from the trial publications from this new assessment (trial registry entry approach). Any publications that were identified by both approaches contributed only once to the analyses. For the purposes of this assessment, data from the trial registry entry approach were used for those studies with a trial publication only. If no publication was found but results had been included on the trial registry entry, we excluded these as this was an extra source of data that was not considered in the previous assessment. We ordered the published trials by publication date, divided them into blocks of 10, and calculated an average of the percentage reporting the full rheumatoid arthritis core outcome set over the previous 10 years. For example, the average for 2016 was taken to be the average percentage of trials reporting on the full rheumatoid arthritis core outcome set from 2007 to 2016. Statistical analysis was carried out in Microsoft Excel 2010, and graphs were produced in R version 3.12.

Patient involvement

No patients were involved in setting the research question or the outcome measures, nor were they involved in the design and implementation of the study. There are no plans to involve patients in the dissemination of results.

Results

After applying the relevant filters, a total of 652 rheumatoid arthritis trials were identified on ClinicalTrials.gov, with registration dates from 9 May 2002 to 17 August 2016. After exporting the results and applying additional filters to meet the study inclusion criteria, 366 of the exported records were ineligible: 138 trials were not exclusive to rheumatoid arthritis, 35 did not consider efficacy as an endpoint, 17 did not consider a drug intervention, and 176 did not use an eligible study design for this assessment (fig 1). After a review of the outcome specifications within the registry entry, a further 13 records were excluded: 12 owing to poor outcome specification (eg, remission was specified as an outcome, but the criteria for remission were not defined) and one entry did not specify any outcomes (entry registered in 2002). This left 273 registry entries for this assessment (fig 1).

Fig 1 Flow diagram of rheumatoid arthritis trials registered on ClinicalTrials.gov and included in this study

Of the 273 eligible registry entries, the recruitment status of 171 (63%) was shown as completed in ClinicalTrials.gov while for the remaining 102 entries, recruitment was either ongoing, not started, or the study was on hold or terminated prematurely (table 1). Similar percentages of trials planned to follow participants for less than six months (44%; 120/273) and for at least 12 months (40%; 108/273). Most trials received commercial funding (table 1). About half the trials had a planned recruitment of between 100 and 500 participants (49%; 134/273), and just over a third planned for more than 500 participants (35%; 96/273). We found trial publications for nearly two thirds (65%; 122/189) of trials that were registered as completed (n=119) or terminated (n=3) (table 1). No trial publications were found for trials that were ongoing, suspended, or withdrawn. Publications were listed on ClinicalTrials.gov for 97 trials, and we found the remainder from our searches using Google and Web of Science. The median time from the trial start date (date that enrollment to the protocol began) recorded on the trial registry to the first recorded publication date (as recorded on the journal article) was about five years (table 1). Of the 67 trials registered as completed or terminated that had no trial publication, trial data were available on Clinicaltrials.gov for 21, whereas no trial data were available for the remaining 46 trials (table 1).

Table 1

Characteristics and publication status of included rheumatoid arthritis trials registered on ClinicalTrials.gov

Characteristics	No (%) of trials (n=273)
Recruitment status:
Completed	171 (63)
Terminated	18 (7)
Recruiting	44 (16)
Enrolling by invitation	1 (<1)
Suspended	4 (1)
Not yet recruiting	34 (12)
Withdrawn	1 (<1)
Trial duration (months):
<6	120 (44)
6-12	43 (16)
≥12	108 (40)
Not specified	2 (<1)
Funding:
Commercial	208 (76)
Non-commercial	51 (19)
Both	14 (5)
Planned sample size:
<100	43 (16)
100-500	134 (49)
>500	96 (35)
Primary trial publication status (n=189)†:
Trial published	122 (65)
Publication listed on ClinicalTrials.gov	97
Search for Clinical Trial Registry number using Google/Web of Science	25
No trial publication found but trial data published on ClinicalTrials.gov	21 (11)
Recruitment completed (results posted on ClinicalTrials.gov)	14
Study terminated (results posted on ClinicalTrials.gov)	7
No trial publication found (no trial data found)	46 (24)
Recruitment completed (no results available)	38
Study terminated (no results available)	8
Time to publication (n=122)‡:
Median (interquartile range) number of years	4 (3-5)
Median (interquartile range) number of days	354 (142-263)

*Recorded on ClinicalTrials.gov (6 October 2016).

†Recruitment status listed as either completed or terminated on ClinicalTrials.gov.

‡Taken from start date (date that enrollment to the protocol began, as recorded on ClinicalTrials.gov) to first recorded publication date (as recorded in the published article).

Methods for assessing uptake of full rheumatoid arthritis core outcome set

The four uptake measures listed can be computed using the data presented in the bottom half of figure 1:

The percentage reporting data on the full rheumatoid arthritis core outcome set was 81% (116/143) for trials identified on the registry as completed (or terminated) when the trial results were found from any source.

The percentage reporting or planning to measure data on the rheumatoid arthritis core outcome set was 70% (190/273) for trials where results were identified in the registry or where planned measurements were taken from the trial registry entry if results were not available.

The percentage planning to measure data on the rheumatoid arthritis core outcome set was 67% (184/273) for registered trials regardless of trial or publication status, based on the outcomes listed in the trial registry entry.

The percentage reporting or planning to measure data on the full rheumatoid arthritis core outcome set was 77% (145/189) for trials identified on the registry as completed (or terminated), where a publication or results were identified in the registry (ie, not from a wider search) or where planned measurements were taken from the trial registry entry if a publication or results were not available in the registry.

Uptake of the rheumatoid arthritis core outcome set over time

The reporting of the full rheumatoid arthritis core outcome set in trial publications over time is illustrated both for the previous approach of identifying trial publications from the inclusion of studies in systematic reviews (reported in6) and for the new approach of identifying trial publications from trial registry entries (fig 2). For 2006 to 2009, we found 20 trials that were published in the overlap period, 10 of which were included in the original evaluation and 10 of which were not included in our original evaluation. The original approach based on systematic reviews found 10 trials in the overlapping period, 8 (80%) of which reported the full core outcome set. The new method based on trial registry entries found 10 trials, 9 (90%) of which reported the full core outcome set. Figure 2 shows a continuation over time in the upward trend in the percentage of trials measuring the full rheumatoid arthritis core outcome set.

Fig 2 Percentage of trials measuring full rheumatoid arthritis core outcome set (RA COS) averaged over past 10 years. WHO=World Health Organization; ILAR=International League of Associations for Rheumatology

Discussion

This study has shown that the uptake of the rheumatoid arthritis core outcome set, which was published in 1994, has continued to increase over time. The increase in uptake was encouraging but the slighter increase in recent years perhaps suggests that further advances might be challenging, especially as some trialists do not measure the full rheumatoid arthritis core outcome set even though they are aware of its existence.6 In the previous assessment of the rheumatoid arthritis core outcome set,6 we noted that the introduction of regulatory guidance—for example, from the Food and Drug Administration 1996,10 and European Medicines Agency 199811—which were involved in ratifying and recommending the rheumatoid arthritis core outcome set, might have contributed to trials measuring these core outcomes. Uptake of the core outcome set also increased before the publication of the core outcome set (1994), which perhaps indicates that consensus may have been developing; this was formalised by publication. Over 80% of the trials in this updated assessment received some commercial funding and therefore their adherence to the EMA/FDA guidance in general may have resulted in trialists using the rheumatoid arthritis core outcome set. In 2007 a patient perspectives workshop at OMERACT 8 (Outcome Measures in Rheumatology) identified that fatigue was an important patient outcome for rheumatoid arthritis,12 as well as generic quality of life.13 Although this is an OMERACT recommendation, no update of the core set has yet been ratified. We found that 30 of the 203 trials received on the trial registry from 1 January 2008 planned to measure fatigue and 29 planned to measure quality of life, with 16 planning to measure both.

A review of the outcomes listed in the trial registry entries suggested that the uptake of the rheumatoid arthritis core outcome set across all trials would be 67%. Considering only those trials recorded as completed or terminated, the uptake rate based on trial registry information alone was 77%; this compared favourably to the uptake rate of 81% found through an assessment of trial results and publications. We suspect that the lower uptake statistic based on the trial registry entry data compared with that which combines information from the registry entries and publications is largely a result of the quality of information recorded within a trial registry.14 15 For example, only a single primary outcome was registered in four of the 12 trials where the full rheumatoid arthritis core outcome set was mentioned in the trial report. The information in a trial registry entry may also be subject to legitimate changes while a trial is ongoing, which means that uptake rates based on the registry entry for ongoing studies may be different from that for trials that are completed and published. Moreover, discrepancies in reported outcomes (in a trial report) that are not prespecified (in a trial registry) have previously been found to be common.16 Despite this difference in the number of trial registry entries listing the full rheumatoid arthritis core outcome set and the number of trial publications doing so, we found that the use of trial registry entries to assess uptake of a core outcome set was efficient and provides a more up-to-date method than identifying trials because of their inclusion in systematic reviews. It is also preferable to citation analysis, which is the only other method we have identified as having been used to assess the uptake of a core outcome set.17 That approach was also applied to the rheumatoid arthritis core outcome set, but it proved unreliable because few of the reports of the trials that measured the core outcome set cited the relevant publication.17

Strengths and limitations of this study

The strength of this study is that we considered all rheumatoid arthritis trials registered on ClinicalTrials.gov, which is one of the largest clinical study registries. While we acknowledge that more trials could have been identified if more primary registries were searched, such as all those registered with the World Health Organization International Clinical Trials Registry Platform (ICTRP), the trials identified on ClinicalTrials.gov are likely to be a representative sample of all trials that are registered in rheumatoid arthritis, given that trials entered onto the site are registered from across the world.18 Furthermore, since the International Committee of Medical Journal Editors (ICMJE) accepts registration in any registry that is a primary register of ICTRP or in ClinicalTrials.gov (a data provider for ICTRP), we do not anticipate that the trials registered in ClinicalTrials.gov will differ in quality given that all trial registries endorsed by ICMJE must meet the same criteria.19 One potential difference between a sample drawn from ClinicalTrials.gov and from other registries is that the percentage of commercially funded trials on this US based registry might be higher, which could lead to higher estimates of uptake of core outcome sets if such trials are more likely to use the core outcome set for regulatory reasons. With regard to practicalities when considering ways to assess the uptake of a core outcome set, we found that ClinicalTrials.gov had a user friendly interface, which helped make this an efficient source of the outcomes measured in studies. With this in mind, we suggest that similar assessments should be carried out for core outcome sets from other treatment areas and that our work provides a template for an efficient method to conduct such assessments.

A potential limitation of this study is that one reviewer (JJK) carried out most of the assessments. However, a second author (PRW) independently checked a sample of registry entries and reports, and no discrepancies were identified. When considering the outcomes reported in trial publications, we also relied heavily on trial authors listing their publications in their trial entry on Clinicaltrials.gov. Although we supplemented this with internet searches using Google and a citation database, we are likely to have missed some trial reports. The identification of the outcomes that are actually measured and reported in trials (as included in reports or datasets) compared with those that are planned to be measured (as included in registry entries) should become easier in the future—for example, as a result of US legislation (effective on Clinicaltrial.gov from January 2017), mandating the uploading of summary trial results within a certain time frame, independent of decisions made about journal publication.20 Improvements in automatic data linkage between published articles and trial registry entries will also improve the process. One final notable limitation that may affect the estimate of uptake based on the method proposed is that trials would not be identified if they are not registered. The uptake rate of 81% calculated from reported data from trials that were identified as completed on the trial registry can be taken as our ideal method (with the caveat that unregistered trials might be less likely to measure the core outcome set). If trials that are not registered are of lower quality in general and less likely to be aware of the core outcome set and the importance of using it, our “gold standard” result for reported uptake might be an overestimate when compared with all trials undertaken.

Relation to other studies and implications

In the broader context, a recently updated systematic review identified around 300 published and nearly 150 ongoing core outcome sets,7 and therefore the present report provides evidence to support the potential value of core outcome sets for improving the quality of research and reducing waste. The current report highlights the successful implementation of a well established core outcome set in rheumatoid arthritis. Although it appears to have taken over 20 years to reach a stable uptake rate for this particular core outcome set, the promotion of core outcome sets by the COMET (Core Outcome Measures in Effectiveness Trials).21 Initiative, and its referencing in guidelines for trialists,22 by funders,23 and from regulatory authorities24 should accelerate uptake in the future. Furthermore, greater awareness of the need to consider the use of a core outcome set and inclusion of links to the core outcome set in registry entries25 should also have a positive impact, bearing in mind that many of the queries received by trial registry providers relate to the outcomes section.26

Conclusions

The adoption of a core outcome set has the potential to increase consistency in outcomes measured across trials and ensure that trials are more likely to measure appropriate outcomes. The WHO-ILAR core outcome set for rheumatoid arthritis was first ratified in 1994, and recent trends suggest that there is a consistent increase in published trials of rheumatoid arthritis measuring it. This is the first study to assess the measurement of a core outcome set using trial registry information, which found that this was a more efficient and up-to-date approach than retrieving and assessing trial publications, and more reliable than citation analysis. The uptake rate estimated from trial registry information, which avoids the need to find and read trial publications that are not listed in the registry, seems to be reasonably reliable when based on those trials recorded as completed or terminated in the registry. Our recommended method for assessing uptake is therefore to identify trials in the relevant area of healthcare in the registry, select those that are completed or terminated, and then use the registry information (publication, results, or planned outcomes) to assess uptake of the core outcome set.

Core outcome sets can enhance the relevance of research by ensuring that a standardised set of outcomes are measured and reported in all trials for a specific clinical area

Assessing uptake allows the impact of development of core outcome sets to be evaluated, to improve implementation and ensure core outcome sets do not themselves contribute to waste in research by not being used

Previous methods used to estimate the uptake of core outcome sets have proved to be time consuming and inefficient

The reporting of the rheumatoid arthritis core set of outcomes in completed trials was found to be 81%, corresponding to an uptake rate of 77% estimated from the information on outcomes listed in the trial registry

Reviewing outcomes listed in trial registries provides a reasonable estimate of the uptake of a core outcome set, and is less time consuming than examining the outcomes in published reports of trials

The method proposed provides an approach for assessing the uptake of the 300 core outcome sets already published