The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed.

OBJECTIVES: To examine the reporting characteristics and methodological details of randomised trials indexed in PubMed in 2000 and 2006 and assess whether the quality of reporting has improved after publication of the Consolidated Standards of Reporting Trials (CONSORT) Statement in 2001.
DESIGN: Comparison of two cross sectional investigations. Study sample All primary reports of randomised trials indexed in PubMed in December 2000 (n=519) and December 2006 (n=616), including parallel group, crossover, cluster, factorial, and split body study designs. MAIN OUTCOME MEASURES: The proportion of general and methodological items reported, stratified by year and study design. Risk ratios with 95% confidence intervals were calculated to represent changes in reporting between 2000 and 2006.
RESULTS: The majority of trials were two arm (379/519 (73%) in 2000 v 468/616 (76%) in 2006) parallel group studies (383/519 (74%) v 477/616 (78%)) published in specialty journals (482/519 (93%) v 555/616 (90%)). In both 2000 and 2006, a median of 80 participants were recruited per trial for parallel group trials. The proportion of articles that reported drug trials decreased between 2000 and 2006 (from 393/519 (76%) to 356/616 (58%)), whereas the proportion of surgery trials increased (51/519 (10%) v 128/616 (21%)). There was an increase between 2000 and 2006 in the proportion of trial reports that included details of the primary outcome (risk ratio (RR) 1.18, 95% CI 1.04 to 1.33), sample size calculation (RR 1.66, 95% CI 1.40 to 1.95), and the methods of random sequence generation (RR 1.62, 95% CI 1.32 to 1.97) and allocation concealment (RR 1.40, 95% CI 1.11 to 1.76). There was no difference in the proportion of trials that provided specific details on who was blinded (RR 0.91, 95% CI 0.75 to 1.10).
CONCLUSIONS: Reporting of several important aspects of trial methods improved between 2000 and 2006; however, the quality of reporting remains well below an acceptable level. Without complete and transparent reporting of how a trial was designed and conducted, it is difficult for readers to assess its conduct and validity.

Introduction

The Consolidated Standards of Reporting Trials (CONSORT) Statement, first published in 19961 and revised in 2001,2 provides recommendations for authors regarding how to prepare reports of trial findings. The aim of this 22 item checklist is to facilitate complete and transparent reporting of trial findings and aid their critical appraisal and interpretation. The statement has been endorsed by the World Association of Medical Editors, the International Committee of Medical Journal Editors, the Council of Science Editors, and a significant number of journals worldwide.3

More than twelve years have now passed since the CONSORT Statement was first published and more than eight years have passed since the statement was last revised. A study of randomised trials published in December 2000, before publication of the revised CONSORT Statement in 2001, showed that many items on the CONSORT checklist—such as details of sample size calculations, primary outcomes, random sequence generation, allocation concealment, and handling of attrition—were inadequately described in more than half of the publications assessed.4

With further revisions of the CONSORT Statement expected in 2010, it is timely to assess whether the recommendations have led to improvements in the quality of reports of randomised trials. In part one of this study, we examine the extent to which reports of randomised trials published in 2006 incorporate recommendations from the CONSORT Statement. In part two, we assess whether the quality of reports of randomised trials has improved since a similar assessment was conducted in 2000,4 before the publication of the revised CONSORT Statement in 2001.2

Methods

Study sample

We used the Cochrane highly sensitive search strategy (phase 1)5 to identify primary reports of randomised trials published in December 2006 and indexed in PubMed by 22 March 2007. This search approach was developed specifically to retrieve reports of controlled trials from Medline by using PubMed.5 One reviewer screened the titles and abstracts of all retrieved reports to exclude any obvious reports of non-eligible trials. A copy of the full article was then obtained for all non-excluded reports, and each full article was assessed by the same reviewer to determine if it met the inclusion criteria. Any additional material about the trial included as an appendix on the journal website was also obtained, if available.

We defined a randomised trial as a prospective study that assessed healthcare interventions in human participants who were randomly allocated to study groups. Studies of cost effectiveness or the accuracy of diagnostic tests, and non-English language reports, were excluded. Studies of parallel group, crossover, cluster, factorial, and split body design were included. We defined parallel group trials as studies where each participant is randomised to one of the intervention arms. Crossover trials were defined as trials where each participant was exposed to each intervention in a random sequence. Cluster trials were defined as trials in which clusters of individuals (for example, clinics and schools) were randomly allocated to different study arms. Factorial trials were defined as trials where participants were randomly assigned to individual interventions or a combination of interventions, and split body trials were defined as those where separate body parts within each participant (for example, eyes) were randomised. These definitions mirror those used in the original survey by Chan and Altman, which assessed 519 randomised trials published in December 2000 and indexed in PubMed by July 20024; however, their cohort also included one trial report published in French.

Data extraction

We extracted details on a number of general and methodological items from each of the included articles. General items were trial design, journal type and medical specialty, type of intervention, number of data collection sites, number of randomised groups, and sample size. Methodological items were use of the term “randomised” in the study title and specification of the primary outcome, sample size calculation, method of random sequence generation and allocation concealment, who was blinded, and how blinding was achieved. Publication of a participant flow diagram, details of loss to follow-up in each arm of the trial, whether the trial stated that an intention to treat analysis had been carried out, details of trial registration, access to the trial protocol, and funding sources were also analysed (box). We also assessed whether a trial report was published in a journal that endorses the CONSORT Statement on the basis of the journals’ instructions to authors (accessed June 2008).

Definitions used to assess reporting of methodological details in publications of randomised trials

Primary outcome

Explicit definition of primary or main outcome(s) or an outcome used in the sample size calculation, or explicit description of a main outcome in the primary study objectives

Sample size

Sample size calculation stated to have been undertaken

Random sequence generation

Description of method for allocating participants to study groups, including computer generated sequences, random number tables, and coin tosses

Allocation concealment

Description of method used to prevent individual participants from knowing or predicting the allocation sequence in advance, including central randomisation or sealed envelopes

Blinding—who was blinded

Study participants, outcome assessors, care providers, or investigators with no knowledge of the participants’ group allocation stated to be blinded; or the trial stated to be blinded, single blind, double blind, or triple blind

Blinding—how blinding was achieved

Similarities between interventions or procedures described, or the trial stated to be placebo controlled or unblinded

Attrition—loss to follow-up for each group

Losses to follow-up, with reasons, enumerated for all study groups

Intention to treat analysis

Randomised participants with available data stated as having been assessed in an intention to treat analysis

Data extraction was carried out by three reviewers. To ensure consistency in the data extraction process, the three reviewers first completed data extraction for ten articles and any disparities in the data obtained were discussed. Differences were primarily owing to differing interpretation of the data extraction form; thus the form was modified and the exercise repeated using a further ten articles until there was agreement. Once agreement on the interpretation of the data extraction form was reached, the three reviewers carried out single data extraction on all remaining articles. If there was uncertainty regarding a particular article, the data obtained were checked by a second member of the review team and resolved by discussion.

Data analysis

Descriptive summary statistics were calculated for each of the general and methodological items specified, and outcomes were stratified by study design. The descriptive summary statistics were then used to compare the quality of reporting for randomised trials published in December 2006 with the quality of reporting in trials published in December 2000 (that is, in those trials assessed by the original survey by Chan and Altman4). STATA (version 10) was used to calculate risk ratios (RR) and absolute differences (AD) with 95% confidence intervals (CI) to quantify changes in reporting between 2000 and 2006. Finally, for articles published in 2006 we also compared the quality of reporting for randomised trials published in journals endorsing the CONSORT Statement with the quality in journals that did not, recognising that there is a time lag from when the journal article was published and when the journal instructions to authors were accessed (June 2008).

Results

Survey of randomised trials indexed in PubMed in December 2006

The Cochrane highly sensitive search strategy identified 1735 citations that possibly linked to reports of randomised trials. After screening the titles and abstracts of all retrieved citations, we reviewed 879 full text articles and identified 616 primary reports of randomised trials for inclusion in our final cohort (fig 1). General and methodological characteristics were stratified by trial design. In 2006, 316 different journals published primary trial reports, with 25 journals publishing five or more randomised trials in the single month studied. The majority of trial reports were published in specialty journals (555/616 (90%)). The highest number of reports appeared in the American Journal of Cardiology (n=9), Anesthesia & Analgesia (n=9), Journal of the American College of Cardiology (n=9), New England Journal of Medicine (n=8), Journal of Clinical Oncology (n=7), and Journal of Infectious Diseases (n=7). All these journals are monthly apart from the New England Journal of Medicine, which is weekly.

Fig 1 Identification of randomised trials from PubMed citations indexed in December 2006

Table 1 provides information on general trial characteristics. Over three quarters (477/616 (78%)) of reports were of parallel group trials, 16% (100/616) were crossover trials, and the remaining 6% (39/616) were classified as “other”—more specifically, cluster randomised (13/39), factorial (10/39), or split body (16/39) trials. More than half (356/616 (58%)) of the trials investigated drugs as the primary intervention of interest, whereas 21% (128/616) assessed surgical or procedural interventions, 18% (113/616) assessed counselling or lifestyle interventions, and 3% (19/616) assessed equipment or devices. Forty per cent (248/616) of reports explicitly stated that the trial was conducted at a single centre and 28% (172/616) stated that the trial took place at multiple centres; the number of study centres was not explicitly defined in the remaining reports (196/616 (32%)). The median number of participants recruited per trial was 62 (10th to 90th percentile 19 to 392). As expected, parallel group trials were larger (median 80, 10th to 90th percentile 28 to 418) than crossover trials (median 20, 10th to 90th percentile 9 to 61).

Table 1

 General characteristics of randomised trials indexed in PubMed in December 2006

	All trials (n=616)	Trial type
		Parallel group (n=477)	Crossover (n=100)	Other* (n=39)
Journal type
Specialty	555 (90%)	424 (89%)	99 (99%)	32 (82%)
General medical	61 (10%)	53 (11%)	1 (1%)	7 (18%)
Top five speciality areas
	Cardiology 68 (11%)	Cardiology 57 (12%)	Physiology 30 (30%)	Dentistry 5 (13%)
	Psychiatry 57 (9%)	Psychiatry 48 (10%)	Respiratory medicine8 (8%)	Ophthalmology5 (13%)
	Physiology 53 (9%)	Surgery 33 (7%)	Cardiology 7 (7%)	Psychiatry5 (13%)
	Paediatrics 38 (6%)	Paediatrics 32 (7%)	Pharmacology6 (6%)	Infectious disease 4 (11%)
	Surgery 38 (6%)	Neurology 31 (6%)	Dentistry 6 (6%)	Cardiology4 (10%)
Journal stance on CONSORT Statement†
CONSORT endorsing journal	274 (44%)	231 (48%)	30 (30%)	13 (33%)
Non-endorsing journal	342 (56%)	246 (52%)	70 (70%)	26 (67%)
Intervention
Drug	356 (58%)	278 (58%)	66 (66%)	12 (31%)
Surgery or procedure	128 (21%)	102 (21%)	16 (16%)	10 (26%)
Counselling or lifestyle interventions	113 (18%)	87 (18%)	13 (13%)	13 (33%)
Equipment or devices	19 (3%)	10 (2%)	5 (5%)	4 (10%)
Study centres‡
Single	248 (40%)	184 (39%)	42 (42%)	22 (56%)
Multiple	172 (28%)	153 (32%)	9 (9%)	10 (26%)
Unclear	196 (32%)	140 (29%)	49 (49%)	7 (18%)
Number of study groups
Two	468 (76%)	374 (78%)	64 (64%)	30 (76%)
Three	97 (16%)	69 (15%)	27 (27%)	1 (3%)
Four	35 (5%)	21 (4%)	7 (7%)	7 (18%)
More than four	16 (3%)	13 (3%)	2 (2%)	1 (3%)
Sample size
Median per trial (10th to 90th percentile)	62 (19 to 392)	80 (28 to 418)	20 (9 to 61)	86 (20 to 1571)
Median per treatment group per trial (10th to 90th percentile)	36 (13 to 241)§	34 (12 to 187)	NA	376 (28 to 668)¶

*Includes cluster (n=13), factorial (n=10) and split body (n=16) trials.

†According to journal instructions to authors, assessed June 2008.

‡Explicitly stated as single centre or multicentre, otherwise defined as unclear.

§n=500 trials; excludes crossover and split body trials.

¶n=23 trials; excludes crossover and split body trials.

Table 2 provides information on the reporting of methodological items. A third (205/616 (33%)) of all trial reports stated that the study was randomised in the study title, 53% (324/616) defined the primary outcome, and 45% (279/616) stated that a sample size calculation had been undertaken. The method used to generate the random sequence for allocating participants to study groups was reported in just over a third (209/616 (34%)) of trial reports, and a quarter (156/616 (25%)) reported the method used to prevent the person enrolling participants from knowing or predicting the allocation sequence.

Table 2

 Reporting of methodological characteristics for randomised trials indexed in PubMed in December 2006

	All trials (n=616)	Trial type
		Parallel group (n=477)	Crossover (n=100)	Other* (n=39)
“Randomised” stated in title
Stated	205 (33%)	180 (38%)	11 (11%)	14 (36%)
Not stated	411 (67%)	297 (62%)	89 (89%)	25 (64%)
Primary outcome
Defined	324 (53%)	276 (58%)	29 (29%)	19 (49%)
Not defined	292 (47%)	201 (42%)	71 (71%)	20 (51%)
Sample size calculation
Stated	279 (45%)	232 (49%)	33 (33%)	14 (36%)
Not stated	337 (56%)	245 (51%)	67 (67%)	25 (64%)
Method of random sequence generation
Reported	209 (34%)	176 (37%)	17 (17%)	16 (41%)
Computer	169 (81%)	146 (83%)	13 (76%)	10 (62%)
Random number table	23 (11%)	18 (10%)	2 (12%)	3 (19%)
Coin toss	4 (2%)	2 (1%)	1 (6%)	1 (6%)
Other†	13 (6%)	10 (6%)	1 (6%)	2 (13%)
Not reported	407 (66%)	301 (63%)	83 (83%)	23 (59%)
Method of allocation concealment
Reported	156 (25%)	141 (30%)	9 (9%)	6 (15%)
Envelopes	78 (50%)	73 (52%)	5 (56%)	0
Central	52 (33%)	47 (33%)	1 (11%)	4 (67%)
Pharmacy	22 (14%)	17 (12%)	3 (33%)	2 (33%)
Other‡	4 (3%)	4 (3%)	0	0
Not reported	460 (75%)	336 (70%)	91 (91%)	33 (85%)
Blinding—who was blinded
Any blinding	362 (59%)	277 (58%)	63 (63%)	22 (56%)
Details reported§	160 (44%)	127 (46%)	18 (29%)	15 (68%)
Details not reported¶	202 (56%)	150 (54%)	45 (71%)	7 (32%)
Unblinded#	149 (24%)	117 (25%)	26 (26%)	6 (15%)
Unclear	105 (17%)	83 (17%)	11 (11%)	11 (28%)
Blinding—how blinding was achieved
Any blinding	254 (41%)	190 (40%)	50 (50%)	14 (36%)
Details reported**	113 (45%)	82 (43%)	22 (44%)	9 (64%)
Details not reported††	141 (55%)	108 (57%)	28 (56%)	5 (36%)
Unblinded	147 (24%)	115 (24%)	26 (26%)	6 (15%)
Not reported	215 (35%)	172 (36%)	24 (24%)	19 (49%)
Participant flow diagram
Included	172 (28%)	155 (32%)	8 (8%)	9 (23%)
Not included	444 (72%)	322 (68%)	92 (92%)	30 (77%)
Attrition—loss to follow-up for each group
Reported	422 (69%)	353 (74%)	45 (45%)	24 (62%)
Reason given‡‡	362 (86%)	301 (85%)	41 (91%)	20 (83%)
Reason not given	60 (14%)	52 (15%)	4 (9%)	4 (17%)
Not reported	194 (31%)	124 (26%)	55 (55%)	15 (38%)
Attrition—intention to treat analysis
Reported	188 (31%)	172 (36%)	6 (6%)	10 (26%)
Not reported	428 (69%)	305 (64%)	94 (94%)	29 (74%)
Trial registration
Reported	58 (9%)	53 (11%)	2 (2%)	3 (8%)
Not reported	558 (91%)	424 (89%)	98 (98%)	36 (92%)
Trial protocol
Reported	9 (1%)	6 (1%)	1 (1%)	2 (5%)
Not reported	607 (99%)	471 (99%)	99 (99%)	37 (95%)
Funding
Solely industry	107 (17%)	87 (18%)	16 (16%)	4 (10%)
Part industry	70 (11%)	48 (10%)	17 (17%)	5 (13%)
Non-industry	196 (32%)	147 (31%)	32 (32%)	17 (44%)
None	7 (1%)	7 (1%)	0	0
Unknown	236 (38%)	188 (39%)	35 (35%)	13 (33%)

*Includes cluster (n=13), factorial (n=10) and split body (n=16) trials.

†Includes alternation (n=2), army regiment number (n=1), telephone automation system (n=3), date of birth (inadequate method of sequence generation; n=1), drawing lots (n=2), odd and even numbers [inadequate] (n=1), shuffling cards (n=1), throwing dice (n=1), and web automation system (n=1).

‡Drugs dispensed by person independent to the trial (n=4).

§Article reports exactly who was blinded.

¶Report stated used the terms “blinded”, “single blind,” or “double blind,” or similar, without providing further details on who was blinded.

#Trial stated as unblinded if explicitly stated as such or blinding clearly not possible.

**Article reports similarities between interventions or procedures.

††Trial stated as placebo controlled, without further details on how placebo control was achieved.

‡‡65 trial reports stated that there was no loss to follow-up (parallel n=43, crossover n=19, other n=3).

Fifty-nine per cent (362/616) of publications reported details of any blinding. Of these, 44% (160/362) provided specific details on who was blinded after assignment to interventions (for example, study participants, outcome assessors, care providers), whereas the remaining 56% (202/362) simply used the terms “blinded,” “single blind,” or “double blind” without providing further details. In addition to reporting who was blinded, 41% (254/616) of trial reports provided information on how blinding was achieved; of these, 45% (113/254) specifically described any similarities between the interventions or procedures (for example, identical in size, colour, and taste), whereas the remaining 55% (141/254) simply used the term “placebo” without providing further details.

A participant flow diagram depicting, for each group, the number of participants who were randomly assigned, received the intended treatment, and were analysed for the primary outcome was included in 28% (172/616) of study reports. Sixty-nine per cent (422/616) of trials reported details of any loss of follow-up for each study group; of these, 86% (362/422) explicitly stated the reasons for attrition or reported that there was no loss to follow-up. A third (188/616 (31%)) of trial reports stated that an intention to treat analysis had been carried out. Very few trial reports provided details of trial registration (58/616 (9%)) or where the original trial protocol could be accessed (9/616 (1%)). Details of funding sources were provided in 62% (380/616) of trial reports: 17% (107/616) of trials were funded solely by industry, 11% (70/616) were part funded by industry, and 32% (196/616) were not industry funded. For all trials, the quality of reporting was generally poorer for crossover trials than for parallel group trials (table 2).

Comparison of randomised trials indexed in PubMed in 2000 with those indexed in 2006

In both 2000 and 2006, the majority of trials involved two study arms (379/519 (73%) in 2000 v 468/616 (76%) in 2006), had parallel group design (383/519 (74%) v 477/616 (78%)), with a median of 80 participants per trial, and were published in specialty journals (482/519 (93%) v 555/616 (90%)). The proportion of articles that reported drug trials decreased between 2000 and 2006 (from 393/519 (76%) to 356/616 (58%)), whereas the proportion of surgical trials increased (from 51/519 (10%) to 128/616 (21%); table 3).

Table 3

 Reporting of general characteristics for randomised trials indexed in PubMed in 20004 compared with those indexed in 2006

	PubMed 2000 (n=519)	PubMed 2006 (n=616)
Journal type
Specialty	482 (93%)	555 (90%)
General medical	37 (7%)	61 (10%)
Trial design
Parallel	383 (74%)	477 (78%)
Crossover	116 (22%)	100 (16%)
Other	20 (4%)	39 (6%)
Intervention
Drug	393 (76%)	356 (58%)
Surgery or procedure	51 (10%)	128 (21%)
Counselling or lifestyle intervention	55 (11%)	113 (18%)
Equipment or devices	20 (4%)	19 (3%)
Number of study groups
Two	379 (73%)	468 (76%)
Three to four	122 (24%)	132 (21%)
More than four	18 (3%)	16 (3%)
Sample size
Median per trial (10th to 90th percentile)	52 (12 to 310)	62 (19 to 392)
Median per treatment group per trial (10th to 90th percentile)	32 (12 to 159)*	36 (13 to 241)†

*n=393 trials; excludes crossover, split body trials, and n of 1 trials.

†n=500 trials; excludes crossover and split body trials.

We identified an increase between 2000 and 2006 in the proportion of trial reports that included details of the primary outcome (RR 1.18, 95% CI 1.04 to 1.33; AD 8%, 95% CI 2% to 14%), sample size calculation (RR 1.66, 95% CI 1.40 to 1.95; AD 18%, 95% CI 12% to 23%), and the methods of random sequence generation (RR 1.62, 95% CI 1.32 to 1.97; AD 13%, 95% CI 8% to 18%) and allocation concealment (RR 1.40, 95% CI 1.11 to 1.76; AD 7%, 95% CI 2% to 12%). There was no difference in the proportion of trials that provided specific details on who was blinded after assignment to interventions (RR 0.91, 95% CI 0.75 to 1.10; AD −3%, 95% CI −8% to 3%; fig 2).

Fig 2 Differences in reporting of methodological items between 2000 and 2006

Comparison of journals endorsing the CONSORT Statement with non-endorsing journals in 2006

We also compared the quality of reporting for randomised trials indexed in journals that endorse the CONSORT Statement with those in non-endorsing journals for reports published in December 2006. We identified a significantly higher rate of reporting of key methodological items in CONSORT endorsing journals (fig 3). Interestingly, we observed that the proportion of trial reports that provided specific details on who was blinded after assignment to interventions was higher in CONSORT endorsing journals (RR 1.53, 95% CI 1.17 to 1.99; AD 11%, 95% CI 4% to 18%), and a reverse scenario was true for those reports that used the terms “blinded,” “single blind,” or “double blind” (RR 0.93; 95% CI 0.74 to 1.16; AD −3%, 95% CI −10% to 5%).

Fig 3 Differences in reporting of methodological items between CONSORT endorsing and non-endorsing journals in 2006

Discussion

Principal findings of study

Our study provides a comprehensive assessment of the design and reporting characteristics of a cohort of PubMed indexed randomised trials published in December 2006. It also provides an important comparison with a similar cohort of randomised trials published in December 2000 that allows us to assess whether the quality of reporting has improved over time and, importantly, following the publication of the revised CONSORT Statement in 2001 and before its revision, which will be published in early 2010 (BMJ in press). To our knowledge, our study and that by Chan and Altman4 are the only studies to provide a comprehensive assessment of randomised trials that covers a specified time period and is not restricted by disease or journal type. Our sample is representative of trials accessible in PubMed, which is the primary information resource for clinicians and researchers.

Interestingly, we identified an increase in the number of reports of randomised trials in 2006 (n=616) compared with 2000 (n=519). This difference may reflect a true increase in the number of randomised trials being published or could reflect an increase in the number of journals being indexed on PubMed.

The majority of randomised trials reported in 2006 were two arm parallel group trials published in specialty journals. The proportion of surgical trials in 2006 was higher than in 2000, whereas the proportion of drugs trials decreased—a trend that has also been reported elsewhere.6 The worrying relatively small sample sizes seen in our cohort may be an indication that many trials are inadequately powered to detect clinically important treatment benefits. It is important to note that we only assessed reports of randomised trials included in the PubMed database; therefore, our findings might not be representative of all published reports of trials.

Despite clear evidence of improved reporting of several important aspects of trial methods, the quality of reporting remains well below an acceptable level. In 2006, more than half of all trial reports defined the primary outcome and provided information on blinding, loss to follow-up, and funding source; however, just under half of all reports detailed sample size calculations. Only a third or less provided information on the method of random sequence generation or allocation concealment, or whether an intention to treat analysis had been carried out. Reporting of trial registration details and access to the trial protocol was rare. In our study, we did not assess whether an intention to treat analysis had been carried out correctly, simply that the term was stated in the trial report; however, several studies have shown that this terminology is often used inappropriately.7 8 9

Comparison with other studies

Recent studies that have assessed the quality of reporting of randomised trials have generally been restricted to trials in specific disease areas10 11 12 13 14 15 or journal types.16 17 18 19 These studies show that considerable variability may exist in the reporting of some aspects of randomised trials, depending on the area assessed. For example, a systematic review of 122 surgical trials published in 2000 to 2003 found that no reports detailed how the randomisation process was implemented, only 24% reported details of the primary outcome, and 26% reported details of blinding.10 A separate study of 67 ophthalmology trials published in 2005 found that details of sequence generation, allocation concealment, participant flow diagrams, and sample size calculations were reported for less than half of the randomised trials assessed.11

The situation may be worse in specific journals. For example, a study of 7422 randomised trials published in Chinese medical journals found that important methodological components such as sample size calculation, randomisation sequence, and allocation concealment were documented in only a small minority of trial reports (1%, 8%, and 0.3%, respectively).16 More encouraging is a study of 253 randomised trials published in five leading medical journals between 2002 and 2003, which found that 80% of trials reported sequence generation, 48% reported allocation concealment, 55% reported the blinding status of study participants, 83% reported the sample size justification, and 86% reported a participant flow diagram.17 These findings may well be an indication of more rigorous reporting requirements in these journals.

There is considerable evidence to show that reports of randomised trials that have inadequate or unclear documentation of sequence generation, allocation concealment, and blinding lead to biased estimates of intervention effects. A meta-epidemiological study of 146 meta-analyses found that in trials with subjective outcomes, effect estimates were exaggerated when allocation concealment was inadequate or unclear (ratio of odds ratios 0.69, 95% CI 0.59 to 0.82) or blinding had not taken place (ratio of odds ratios 0.75, 95% CI 0.61 to 0.93).20 Another study found a trend towards larger estimates of treatment effects in publications with inadequate or unclear reporting of random sequence generation compared with those adequately reporting methods (ratio of odds ratios 0.81, 95% CI 0.60 to 1.09).21 Further details on the rationale and importance of these items can be found in the CONSORT explanation and exploration paper.22

When assessing the reporting of key methodological items in CONSORT endorsing as opposed to non-endorsing journals, we identified significantly better reporting in CONSORT endorsing journals. These results should be viewed with a certain amount of caution, however, because there is a time lag in our study between when journal articles were published and when the journal instructions to authors were accessed (June 2008). Nevertheless, these findings mirror those of a recent systematic review, which concluded that journal adoption of the CONSORT Statement is associated with improved reporting of randomised trials, although poor reporting is common.23 Our study provides important baseline data against which future revisions of the CONSORT Statement can be measured.

Limitations of study

Our study has several limitations. We carried out only single data extraction, and although we did our best to minimise any inconsistency in the reviewers’ interpretation, it is possible that errors may have accrued. In addition, data extraction in 2000 and that in 2006 was carried out by different teams of reviewers; however, all reviewers conferred to try to ensure consistency in the interpretation of data extraction items.

We also used slightly different sampling techniques between 2000 and 2006. The 2000 sample included all reports of randomised trials published in December 2000 and indexed in PubMed by July 2002 (to account for the time lag in PubMed indexing). For pragmatic reasons, the 2006 sample included trials published in PubMed in December 2006 and indexed as of March 2007. This approach meant that we were able to capture our sample of trials within one search, but we may have missed a small number of eligible trials that were indexed in PubMed after March 2007.

Finally, it is important to recognise that poor reporting does not necessarily mean poor methods. Several studies24 25 that compared the content of published reports with their protocol found that the methodological techniques in the actual trial were of better quality than reported in the final publication. Conversely, other studies26 27 have shown that important information about the trial methodology is often missing in the protocol but is included in the published report. Such studies recommend reviewing research protocols and contacting trialists for more information when assessing trial quality; we did not do this in our study.

Conclusions

Without complete and transparent reporting of how a trial was designed and conducted, it is difficult for readers to assess its conduct and validity or to differentiate trials with unbiased results from those with questionable results.22 We believe that the CONSORT Statement, and its extensions, is a key tool through which adequate reporting can be achieved. More journals should endorse the CONSORT Statement, and, most importantly, they should do more to ensure adherence. For example, journals could incorporate the checklist and flow diagram into their review processes and indicate this requirement in their published instructions to authors.28 Without wide endorsement, the CONSORT Statement cannot fully yield the benefits it was intended to produce.

The CONSORT Statement provides recommendations for authors about how to prepare articles reporting the findings of randomised trials, thereby aiding their critical appraisal and interpretation

Previous studies show that before revision of the CONSORT Statement in 2001, important methodological details such as sample size, primary outcome, randomisation process, blinding, and attrition were inadequately described in more than half of trial reports

Reporting of methodological details in publications of randomised trials has improved somewhat between 2000 and 2006

Many trial reports still omit important information about trial conduct; thus it is difficult to gauge the validity of the trial findings

This study provides important baseline data against which future revisions to the CONSORT Statement, such as that due to be published in early 2010, can be measured