Prospective registration and outcome-reporting bias in randomized controlled trials of eczema treatments: a systematic review.

We assessed completeness of trial registration and the extent of outcome-reporting bias in published randomized controlled trials (RCTs) of eczema (atopic dermatitis) treatments by surveying all relevant RCTs published from January 2007 to July 2011 located in a database called the Global Resource of Eczema Trials (GREAT). The GREAT database is compiled by searching six bibliographic databases, including EMBASE and MEDLINE. Out of 109 identified RCTs, only 37 (34%) had been registered on an approved trial register. Only 18 out of 109 trials (17%) had been registered "properly" in terms of submitting the registration before the trial end date and nominating a primary outcome. The proportion of "any registered" and "properly registered" RCTs increased from 19% and 10% in 2007 to 57% and 36% in 2011, respectively. Assessment of selective outcome-reporting bias was difficult even among the properly registered trials owing to unclear primary outcome description especially with regard to timing. Only 5 out of the 109 trials (5%) provided enough information for us to be confident that the outcomes reported in the published trial were consistent with the original registration. Adequate trial registration and description of primary outcomes for eczema RCTs is currently poor.

Introduction

Outcome-reporting bias in a clinical study is defined as selective reporting of a subset of the study findings based on the significance and direction of the results. Selective reporting of outcomes has been well recognized in general medicine trials where around 40–62% of studies have been found to have at least one primary outcome that was changed, introduced, or omitted (Dwan ). The effects of such distortion of what was planned in a study are potentially serious for clinical practice. In their review of the effects of outcome-reporting bias in systematic reviews, Kirkham found that of 42 meta-analyses with a statistically significant result, 8 (19%) became nonsignificant after adjusting for outcome-reporting bias and 11 (26%) would have overestimated the treatment effect by ⩾20%. The problem of outcome-reporting bias has not been studied among randomized controlled trials (RCTs) in the field of dermatology.

One of the main ways of reducing selective outcome-reporting bias is to require all trial investigators to register details of their trial in a publicly accessible register before the trial recruitment starts. Those reading the final trial report can then go back to the original trial registration to check whether what was highlighted as the primary outcome measure in the published report corresponded to that in the original registration. In 2004, the International Committee of Medical Journal Editors (ICMJE) initiated a policy requiring investigators to register their trials into a clinical trial registry before participant enrollment begins as a condition of publication in one of their journals (DeAngelis ). This policy came into effect in July 2005. A number of dermatology journals followed this policy, including the British Journal of Dermatology and the Journal of Investigative Dermatology in 2005 (Williams and Stern, 2005; Ormerod and Williams, 2005). A number of trial registers sprang up in the early 1990s, containing different core items, culminating in the World Health Organization (WHO) hosting an International Clinical Trial Registry Platform of approved trial registries (http://www.who.int/ictrp/en/, accessed 23 November 2011) that fulfilled key quality criteria and collected a minimum set of key study domains. Around the same time, dermatology journals also encouraged authors to report all of the important study features included in the Consolidated Standards of Reporting Trials Statement (http://www.consort-statement.org/, accessed 23 November 2011).

This study sought to assess the extent to which outcome-reporting bias is evident in the field of dermatology, using published RCTs of eczema treatments in people with eczema (atopic dermatitis) as an example. The study had three objectives:

To assess the proportion of eczema treatment RCTs registered.

To see whether the lack of registration was associated with differences in risk of bias, sample size, and funding source.

To assess the level of possible outcome-reporting bias of the primary outcome in eczema treatment trials.

Our hypotheses were that (1) less than half of eczema RCTs would have been registered; (2) quality of reporting of trials without preregistration would be poorer; and (3) even in the trials that were properly registered, selective outcome-reporting bias may still exist, i.e., there may be discrepancies between the primary outcome that was registered and the primary outcome reported in the published trial report.

Results

A total of 122 eczema RCTs were published between 1 January 2007 and 31 July 2011 and included in the Global Resource of Eczema Trials (GREAT) database (Figure 1). Twelve studies were excluded because they had published abstracts, but it was not possible to identify a full trial publication. Two publications came from the same study, with the latter publication being a follow-up of the original trial (van der Aa ). Only the first publication, describing the original trial, was included in this study.

Figure 1

Flowchart of trial selection. GREAT, Global Resource of Eczema Trials; RCT, randomized controlled trial.

Levels of registration of eczema treatment trials

Of the remaining 109 included studies, only 37 (34%) had been registered in any form as shown in the study flowchart in Figure 1. In 9 of these 37 trials, the trial registration was done more than a year after the study had been completed and published (the longest time interval was almost 3 years after study completion), and thus could not be defined as “properly registered”. Only 20 (54%) of the 37 registered studies indicated their trial registration number in the final trial publication.

Among the 37 studies with a registered protocol, 18 studies (49%) had “properly registered” according to our definition. Although there has been an increase in trial registration from 2007 onward, the proportion of “properly registered” trials was still low, reaching a maximum of 29% (4/14) in the first half of 2011. The increasing trend in the proportion of registered eczema treatment trials was significant (P=0.003) (Figure 2).

Figure 2

Proportion of eczema trials that were “registered” and “properly registered” among those published from January 2007 to July 2011. Total convenience sample=109 trials.

Differences between trials with and without registration

Although registered trials had a larger median sample size of 70 people (interquartile range 37.5–192) compared with 60 people (interquartile range 30–104.5) for trials without registration, this difference was not statistically significant (P=0.405). Nonregistered trials were significantly less likely to specify their funding source in the subsequent publication than registered trials (3% vs 30%, P=0.001), and a post-hoc analysis looking at choice of comparator showed that nonregistered trials were no more likely to include a placebo comparator than registered trials (70% vs 57%, P=0.188).

Overall, the differences in the estimated risk of bias were not statistically significant, with the exception of allocation concealment domain, where registered trials scored marginally better: 24% (9/37) were at low risk of bias compared with 8% (6/72) for nonregistered trials, P=0.04 (Figure 3).

Figure 3

Risk of bias among trials with and without registration (% at low, unclear, and high risk of bias).

Outcome reporting in properly registered trials and their registration records

The most common discrepancy was missing time-frame information for measurement of the primary outcome: in 8 out of 18 trials, registration records did not state the timing of the primary outcome measurement. In addition, the description of the primary outcome was often vague and unclear: this occurred in 5 out of the 18 “properly registered” trials (Table 1). For example, a primary outcome might be described as “efficacy in treating exacerbations” or “stable remission”, without further detail as to how this was to be measured or assessed.

Table 1

Trials that have been properly registered: comparison of primary outcome reporting in trial registration and published article

	Publication		Registration
Reference	Primary outcome	Time frame for primary outcome assessment	Primary outcome	Time frame for primary outcome assessment	Statistical significance of results	Comments
Hon et al., 2007	Changes in SCORAD and CDLQI	12 weeks	Changes in SCORAD	12 weeks	SCORAD—NSS CDLQI—SS (P=0.008)	Introduction of a new outcome that showed SS results
Murrell et al., 2007	Proportion of patients with IGA score=0 or 1 (clear or almost clear)	6 weeks	Changes in IGA scores	Not stated	SS (P<0.001)	Missing time frame in protocol
Peserico et al., 2008	Time to relapse of AD	16 weeks	Time to relapse of AD	16 weeks	SS (P<0.0001)
Huang et al., 2009	Change in EASI score	3 months	Change in EASI score	3 months	SS (P=0.004)
Moore et al., 2009	Change in SCORAD	4 weeks	Change in SCORAD	Not stated	SS (P<0.001)	Missing time frame in protocol
Ruer-Mulard et al., 2009	Time to relapse (confirmed by IGA and pruritus score)	16 weeks	Time to relapse confirmed by IGA and pruritus score	Not stated	NSS	Missing time frame in protocol
Wirén et al., 2009	Time to relapse of AD confirmed by ADSI	26 weeks	Compatibility of the skin with the new formulation; effect of maintenance therapy with an emollient cream on the possible recurrence of atopic eczema	Compatibility study: 3 weeks; maintenance study: up to 6 months	Time to relapse—SS (P<0.01)	Vague definition of primary outcome and time frame in protocol
Hoeger et al., 2009	Proportion of IGA=0 or 1 (clear or almost clear)	Day 43	Proportion of IGA=0 or 1 (clear or almost clear)	Not stated	SS (P<0.001)	Missing time frame in protocol
Gambichler et al., 2009	Change in SASSAD	6 weeks	Change in SCORAD	Not stated	NSS	Change in eczema score; missing time frame
van der Aa et al., 2010	Change in SCORAD	12 weeks	Decrease in SCORAD >25% from baseline	12 weeks	NSS
Tzaneva et al., 2010	Length of remission	12 months	Length of remission	12 months	SS (P=0.012)
Foelster Holst et al., 2010	Proportion of IGA=0 or 1 (clear or almost clear)	4 weeks	Measure efficacy in treating acute exacerbation	4 weeks	NSS	Vague definition of outcome in protocol
Schmitt et al., 2010	Proportion of patients with stable remission (SCORAD improvement ⩾50%) and no flare (SCORAD ⩾75% of baseline)	18 weeks	Stable remission in both treatment groups	Not stated	SS (P=0.031)	Vague definition of outcome in protocol; missing time frame
Brenninkmeijer et al., 2010	Change in PAIS	10 weeks treatment+6 months F-U	Change in PAIS	Not stated	At 10 weeks: NSS; F-U: SS (P <0.05)	Missing time frame
Armstrong et al., 2011	Change in POEM	12 weeks	Change in POEM	12 weeks	SS (P=0.0043)
Bangert et al., 2011	1. Reduction in EASI 2. Reduction in number of leukocytes in skin biopsies	3 weeks	Determining whether pimecrolimus cream has an effect on the cellular and molecular profile of atopic dermatitis skin	Not stated	1. EASI: P-value not provided 2. Number of CD45+ cells in biopsy SS difference (P=0.047)	Vague definition of primary outcome; missing time frame
Frankel et al., 2011	Change in IGA and TLSS	4 weeks	Improvement and maintenance of PGA, TLSS (target lesion symptoms score), and subjective eczema control	26 weeks	NSS	Vague definition of outcome; different time frame
Thomas et al., 2011	Change in SASSAD	12 weeks	Change in SASSAD	12 weeks	NS

Abbreviations: AD, atopic dermatitis; ADSI, Atopic Dermatitis Severity Index; CDLQI, Children Dermatology Life Quality Index; EASI, Eczema Area and Severity Index; IGA, Investigator Global Assessment; NSS, not statistically significant; PAIS, Physician Assessment of Individual Signs; PGA, Physician Global Assessment; POEM, Patient-Oriented Eczema Measure; SASSAD, Six Area, Six Sign Atopic Dermatitis Score; SCORAD, Scoring of Atopic Dermatitis; SS, statistically significant; TLSS, Target Lesion Symptoms Score.

Assessment of outcome-reporting bias among properly registered trials

The lack of detail and clarity of reporting for the outcomes of registered trials both from the registration record and from the published trial report makes the objective of assessing outcome-reporting bias in eczema treatment trials difficult. One study introduced a new statistically significant primary outcome that was not present in the registration record (original primary outcome was not significant) (Hon ), one study changed the eczema severity score from SCORAD (Scoring of Atopic Dermatitis) to SASSAD (Six Area Six Sign Atopic Dermatitis Score) (Gambichler ), and one study changed the time frame for assessment of the primary outcome from 26 weeks to 4 weeks (Frankel ). None of these latter changes resulted in a significant result being reported.

Discussion

Main findings

Our study has shown that the proportion of published eczema RCTs registered on an approved trial registry increased from 19% in 2007 to 57% in 2011. The fact that over half of the RCTs published in 2011 were registered is cause for some optimism. However, the proportion of trials that were “properly registered”, lodging details of the proposed trial design before the end of the study, or within 12 months of the recorded trial start date and indicating the primary outcome in the registration, was considerably lower. Taking into account that our definition of “properly registered” was a very generous one, this is a particularly sobering result. Overall, of the 109 eczema RCTs examined, only 5 trials were registered a priori, gave sufficient detail about the primary outcome to judge adherence to the original registration record, and reported the primary outcome in line with the original registration (or included an explanation as to why this had changed). That only 5% of the recent eczema trials were registered correctly and with enough detail to assess outcome-reporting bias for the primary outcome, coupled with the observation that nonregistered or incompletely registered studies fail to highlight the shortcomings of such omissions, is bad science and a potential waste of resources (Chalmers, 1990; Chalmers and Glasziou, 2009).

Even when investigators had preregistered their trial, 46% failed to include details of their trial registration in the published trial report, suggesting that investigators and journal editors do not yet appreciate the importance of such information in their trial reports. The revised version of the Consolidated Standards of Reporting Trials Statement statement for guidance on the reporting of RCTs now explicitly recommends that details of trial registration be reported fully, and this should improve trial reporting in the future (http://www.consort-statement.org/, accessed 23 November 2011).

The fact that some investigators chose to preregister their trial could be an indicator of trial quality, and this was explored using the four key domains known to be associated with a high risk of bias (Higgins and Altman, 2008). In this sample of eczema RCTs, there was a suggestion that trial quality might be improved in registered trials, but this was only significant for the domain of allocation concealment, and is possibly limited by the modest sample size of our survey.

With regard to the hypothesis that eczema trials are subject to outcome-reporting bias, we found some evidence of discrepancies between trial registration and trial reports. However, these were generally as a result of unclear and nonspecific trial registration information rather than clear signs of biased reporting. In general, the number of correctly registered trials was so low that firm conclusions are difficult. Greater efforts could certainly be made to provide more detail regarding the primary outcome and its timing within trial registry records. It is hoped that reviews such as this one will be helpful in informing trialists of the importance and relevance of detailed information being lodged in the trial registries. It is also important that changes to the registration record subsequent to trial commencement be fed into the trial registries in a timely manner, in order to maintain transparency throughout. The concept of threaded publications, enabling the tracking of clinical research studies from inception and the linking of all resulting publications, including the raw data where this is available, is another emerging tool to ensure consistency between what was planned and what was done (Faure and Hrynaszkiewicz, 2011).

Comparison with other studies

Overall, the rates for registration of eczema RCTs were lower than those reported for other surveys of general medical journals and larger specialities. Mathieu looked at RCTs indexed in 2008 in 10 general medicine and specialty journals (cardiology, rheumatology, and gastroenterology); of the 323 included trials, 45.5% were adequately registered, 13.9% were registered after the completion of study, and 12% were registered with no or unclear description of the primary outcome.

A systematic review of studies examining the impact of selective outcome reporting in RCTs identified five studies and compared the trial publication with trial registration record. It found that between 40 and 62% of trials had at least one primary outcome that had been changed, introduced, or omitted (Dwan ). These rates are higher than the rates found in the current study, which might be a reflection of improvements in trial reporting subsequent to the introduction of the ICMJE policy. Alternatively, it is possible that higher rates of outcome-reporting bias would have been found if more of the eczema trials had been registered, thus allowing a comparison of the published results with the trial registration. It is conceivable that to date only the better-quality eczema trials have been registered appropriately.

Strengths and limitations of our study

This study used the GREAT database to identify and assess all eczema trials published over the past four and a half years. Using this global collection of appraised eczema trials allowed for the speedy completion of this review and the easy identification of all relevant RCTs that had been identified through a sensitive search of several bibliographic databases. Including all trials on a particular topic, rather than just those reported in specific journals or as a random sample of published trials, meant that our study is less open to selection bias. Eczema is one of the most common skin conditions and is commonly researched. More than 250 RCTs of eczema treatments have been published in the past 10 years alone (www.greatdatabase.org.uk, accessed August 2011). It is possible that for other common skin conditions, the proportion of trials registered and possible outcome-reporting bias may be different, which makes the results of this study difficult to generalize to the field of dermatology research as a whole.

Our definition of a properly registered trial as one that was registered before the end of the study was a generous one, given that the ICMJE requires that all clinical trials need to be registered in an approved publicly accessible clinical trial register before patient recruitment begins. It is likely that the bottom-line figure of only 5% of all recent eczema trials that have been properly registered would have been even worse if this more stringent definition had been applied, although in many cases it was impossible for us to tell when recruitment began and when the study ended. On the point of timing, it is worth noting that some trialists registered their trials after the study had been completed, which defeats the whole purpose of prospective trial registration as a tool to prevent cheating. Our sample size was relatively small, which affected the overall precision of our estimates of “properly registered” RCTs.

Finally, this review limited its scope to an evaluation of the primary outcomes assessed in trials. Others have reported considerable outcome-reporting bias for secondary outcomes, particularly in relation to the preferential reporting of statistically significant secondary outcomes (Chan ).

Implications of our findings

The introduction of mandatory trial registration as required by the Journal of Investigative Dermatology is a great opportunity to improve the quality and truthfulness of trial reporting, which will in turn lead to better clinical decision-making by reducing the prevalence of spurious and misleading results and reduction in research wastage (Chalmers and Glasziou, 2009). Although the publication bias of whole studies is relatively well known, the phenomenon of selective outcome-reporting bias within studies is possibly less well understood within the dermatology clinical community (Chan and Altman, 2005).

Ideally, deviations from the trial registration record should be described in the published articles so that readers can interpret the results in full knowledge of the changes made. For the full benefits of trial registration to be realized, it is important that all investigators, funders, journal editors, peer reviewers, readers, and the public have an active role in making full use of trial registration information and highlighting the need for transparent trial reporting. Journal referees and readers of clinical trials especially should make more use of scrutinizing trial registries to note whether a clinical trial has truly been registered prospectively, and whether the outcomes reported in the paper are consistent with what was planned. Those conducting systematic reviews are also in a good position to check on and comment on trial registration of included trials, and to assess whether selective outcome-reporting bias was likely to have occurred.

Conclusion

Adequate trial registration for eczema RCTs is poor. Registration of all trials in a publicly accessible database is a critical step toward ensuring the transparent reporting of clinical trial results that affect health care.

Materials and Methods

A systematic search for RCTs of eczema treatments that had been published since January 2007 was conducted. For all identified trials, the WHO International Clinical Trial Registry Platform was searched for proof of approved trial registration.

Searching for published eczema trials

We used a convenience sample of all published eczema trials that have been captured in the GREAT database (http://www.greatdatabase.org.uk, accessed August 2011). This open-access resource brings together information on all RCTs of eczema treatments published since the beginning of 2000 in order to facilitate future methodological research and systematic reviews. The GREAT database uses the Cochrane Highly Sensitive Search Strategy for identifying randomized trials in MEDLINE: sensitivity-maximizing version (2008 revision); Ovid format (Lefebvre ) combined with the terms eczema, atopic eczema, atopic dermatitis, infantile eczema, childhood eczema, neurodermatitis, and Besnier's prurigo. Similar searches combined with the same list of eczema terms are used to search the EMBASE, CINAHL, AMED, and LILACS databases. The Cochrane Library (www.thecochranelibrary.co.uk) and the Cochrane Skin Group Specialised register are also searched using the same eczema terms. The search runs from the beginning of 2000 and is updated regularly.

Inclusion criteria for the present study

Allowing 1.5 years from the ICMJE policy of trial registration coming into effect, we included all eczema treatment RCTs contained in the GREAT database that had been published between 1 January 2007 and 31 July 2011. Whether or not these trials contained safety data was not relevant to this study and thus did not form part of the inclusion or exclusion criteria. We excluded RCTs that were published in abstract form only. If both an abstract and a full paper were found, only the full paper was included. Details of all trials included in this study are listed as an online table (Supplementary Table S1 online).

Identifying trial registration

If the trial registration ID number was stated in the final publication, we used this number to find the trial registration. Where this number was not stated, we tried to obtain the registration record using the WHO International Clinical Trial Registry Platform, which searches a number of different trial registries throughout the world (see Table 2).

Table 2

The list of data providers of the WHO ICTRP search portal1

Australian New Zealand Clinical Trials Registry (ANZCTR)

Chinese Clinical Trial Register (ChiCTR)

Clinical Research Information Service (CRiS), Republic of Korea

ClinicalTrials.gov

Clinical Trials Registry—India (CTRI)

Cuban Public Registry of Clinical Trials (RPCEC)

German Clinical Trials Register (DRKS)

Iranian Registry of Clinical Trials (IRCT)

ISRCTN.org

Japan Primary Registries Network (JPRN)

Pan African Clinical Trial Registry (PACTR)

Sri Lanka Clinical Trials Registry (SLCTR)

The Netherlands National Trial Register (NTR)

1Source: World Health Organization website http://www.who.int/ictrp/search/data_providers/en/index.html (last accessed 12 August 2011).

Two people searched for trial registration independently (AB and HN) using a combination of key words: “eczema” or “atopic dermatitis”, plus keywords describing the trial's interventions, such as “pimecrolimus” or “probiotics”. All registration records that met the search criteria were reviewed to find the one that matched the published trial using key information supplied in the record (e.g., the name of principal investigator, funding source, and trial design). As a rule, we did not spend more than 15–20 minutes looking for each record, on the grounds that trial registration should be easy and quick to identify.

Distinguishing between “registered” and “properly registered” trials

All eczema treatment trials in the sample from the GREAT database registered on the WHO International Clinical Trial Registry Platform were included in the number of “registered” trials. This provides the overall level of trial registration for eczema trials, regardless of whether or not they fulfilled the guidelines for trial registration correctly. In addition, we identified a “properly registered” subset of these trials, which could be used to assess outcome-reporting bias. We defined “properly registered” trials as those trials in which the primary outcome was explicitly stated in the registration document and in which the trial was registered not later than the end of the study (last patient visit). In cases in which the start of participant enrollment was stated, but an end date was not recorded, we considered a trial to be properly registered if the registration was lodged within 12 months of the study start date. Only “properly registered” trials were included in the evaluation of outcome-reporting bias.

To take into account the amendments and possible changes that could have taken place after initial trial registration but before data lock, we used the URL provided on the WHO trial registration webpage to go to the source record for each of the “properly registered” trials in the primary register and looked for additional information or updates regarding the trial.

Comparison between registered and nonregistered trials

We compared registered and nonregistered trials with respect to several characteristics, including the number of participants randomized, funding, and quality of reporting. The quality of reporting was assessed using the Cochrane Collaboration's risk of bias tool (Higgins and Altman, 2008) in four different domains: randomization method, allocation concealment, blinding (of participants, personnel, and outcome assessors), and completeness of outcome data. Each trial was assessed for high, low, or unclear risk of bias for each of these domains using the revised terms suggested in the updated version of the Cochrane Handbook (Higgins ) to improve interpretation of the risk of bias.

Comparison of primary outcome between publication and registration among properly registered trials

For each properly registered trial, we reviewed and compared primary outcomes reported in the publication with those stated in the registration. Both the primary outcome and the time frame for analysis were recorded. We chose to limit the review of outcome bias to the primary outcome for this first study, because of the importance of primary outcomes in determining the success or otherwise of a study and because detailed analysis of secondary outcomes was beyond the scope of our available resources.

We defined a discrepancy as follows: (1) when the primary outcome in the published report was different from that in the registration and (2) when the time frame for assessing the primary outcome in the published report was different from the registration.

Sample size calculation

Sample size was determined by the number of available trials that had been published since the introduction of the ICMJE guidelines. We also estimated that we would need around 100 studies in order to estimate the proportion of registered studies (which we hypothesized would be around 50%) to within 10 percentage points. All eczema RCTs published in the past four and a half years were included.

Data assessment

The following comparisons were made:

The proportion of trials registered.

The proportion of trials properly registered versus the proportion registered in any form.

The proportion of registered trials with the registration number stated in the final publication.

The average sample size of registered trials versus nonregistered trials.

The proportion of trials that declared a funding source.

The proportion of registered versus nonregistered trials with a placebo comparator.

The number of trials with a low/unclear/high risk of bias for each of the four domains for registered trials versus nonregistered trials.

The number of properly registered trials with a missing time frame for the primary outcome.

The number of properly registered trials with a vague and/or unclear primary outcome.

Statistical analysis

We used frequency and percentages for categorical variables, and mean with SD or median and interquartile ranges for continuous variables, as appropriate. Proportions were compared using χ2 and Fisher's exact tests where appropriate. Trends over time were assessed using χ2 for trend. Continuous variables were compared using t-test and Mann–Whitney U-test for nonparametric data. P<0.05 (two-tailed) was considered statistically significant. All statistics were calculated using SPSS version 16 (IBM, www.ibm.com) except χ2 for trend, which was calculated using StatsDirect (Stats Direct, www.statsdirect.com). The protocol for this study was posted on the Centre of Evidence Based Dermatology website (http://www.nottingham.ac.uk/dermatology) at the beginning of the study in September 2011 and before data analysis.