Outcome reporting bias in nephrology randomized clinical trials: Examining outcomes represented by graphical illustrations.

Background: Outcome reporting bias (ORB) is widely reported in the medical literature, but the contribution from published graphical illustrations is unknown. The aim of this study was to investigate the occurrence of ORB in contemporary nephrology clinical trials relating to the choice of outcomes reported through graphical illustrations.
Methods: An observational study was conducted using nephrology clinical trials searched from five high-impact medical journals from 2015 to 2020. Eligible trials reported a phase 2, 3 or 4 trial, contained at least one published outcome graphical illustration and were registered on a clinical trial registry. The primary outcome was the occurrence of ORB based on the choice of graphical illustrations in published trial manuscripts, deemed to be present if a graphical illustration displayed a secondary or unregistered outcome ahead of a trial's primary outcome, or if any unregistered trial outcome was presented as a graphical illustration.
Results: In 75 eligible clinical trials, the primary outcome for ORB was present in 60% of the trials (n = 45). Occurrence of the primary outcome did not differ significantly based on trial sample size, funding model, trial phase, individual medical journal or publication year. An unregistered trial outcome was graphically illustrated in 93% (n = 42) of those clinical trials with ORB present.
Conclusion: Outcome reporting bias based on the choice of graphical illustration is common, driven primarily by graphical illustration of unregistered trial outcomes. More appropriate choice of outcomes for graphical illustrations by authors, coupled with both increased enforcement of CONSORT guidelines by medical journals and specific guidelines for graphical illustrations choice, are desirable to address these findings.

outcome reporting bias

randomized controlled trial

primary outcome

secondary outcome

unspecified outcome

Consolidated Standards of Reporting Trials

Standard Protocol Items: Recommendations for Interventional Trials

Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols

inter-quartile range

Instrument for the reporting of Planned Endpoints in Clinical Trials

Introduction

Outcome reporting bias (ORB) is widely reported in the medical literature [[1], [2], [3]]. Potentially leading to bias in the interpretation of trial results, ORB can misinform clinical decision-making and ultimately affect patient outcomes. There are several forms of ORB, including selective outcome reporting, reporting of unregistered outcomes, altering the assessment period of outcomes or changing the hierarchy of primary and secondary outcomes [3,4]. Even in high-impact journals, only a small minority of clinical trials are truly faithful in reporting all registered outcomes, with a high prevalence of omitting outcomes or reporting de novo unregistered outcomes [5].

Several studies of ORB have focused on mapping the outcomes in the published manuscript text to those pre-specified at trial registration [[6], [7], [8]]. The extent to which the choice of graphical illustrations (figures and graphs) might represent ORB has not been previously reported. The CONSORT guidelines recommend that each clinical trial primary and secondary outcome be reported with an estimate effect size and its precision [9]. However, the manner of portrayal of outcomes, including the use of graphical illustrations, remains at the authors’ discretion, and there is a distinct paucity of trial methodological research to guide researchers. The limited evidence available suggests that, although graphical illustrations are very commonly utilized in medical journals, the choice, content and configuration of these visual aids are often sub-optimal [[10], [11], [12], [13]].

Historical research has demonstrated that visual data, compared to text alone, was associated with superior reader comprehension and persuasion [14]. Miller and Barnett reported that a combination of visual and text reporting was superior to text alone regarding reader comprehension [15]. Marketing research claims that visual data is highly efficacious at attracting reader attention, increasing reader understanding and retention, and increasing circulation of data on digital platforms, although rigorous research methodology is generally not evident to support these claims [16,17]. Overall, graphical illustrations represent a potentially impactful method of conveying research outcomes, and hence could be an important source of ORB.

The aim of this study was to investigate whether ORB was evident in the graphical illustrations of contemporary nephrology randomized control trials (RCTs), when compared to the registered outcomes on trial registries. This study will give clinical researchers in all disciplines an insight into the current role of graphical illustrations in portraying trial outcomes, and the degree to which they contribute to ORB.

Methodology

Study design

A retrospective review of RCTs published in the New England Journal of Medicine (NEJM), the Lancet and the three leading clinical nephrology journals by impact factor rating (the Journal of the American Society of Nephrology, Kidney International and the Clinical Journal of the American Society of Nephrology) was conducted. In order to represent the contemporary era of trial reporting, the study included trials published in a 6-year period from Jan 1st, 2015 through Dec 31st, 2020. The STROBE reporting guidelines for observational studies were adhered to. Ethical approval was not required for the study.

Search strategy and trial eligibility

Using PubMed, all RCTs published relating to nephrology practice from 2015 to 2020 were identified through the following search strategy: ((“randomized clinical trial"[All Fields] AND “hypertension"[All Fields]) OR “diabetic kidney disease"[All Fields] OR “renal replacement therapies"[All Fields] OR “kidney stone"[All Fields] OR “vasculitis"[All Fields] OR “glomerulonephritis"[All Fields] OR (“hemodialysis"[All Fields] OR “renal dialysis"[All Fields] OR (“renal"[All Fields] AND “dialysis"[All Fields]) OR “renal dialysis"[All Fields] OR “hemodialysis"[All Fields]) OR “peritoneal dialysis"[All Fields] OR “kidney transplantation"[All Fields]) AND ((fha[Filter]) AND (randomized controlled trial[Filter]) AND (2015:2020[pdat])).

Inclusion criteria were: manuscripts reporting a single phase 2, 3 or 4 trial; outcomes reported for chronic kidney disease, acute kidney injury, nephrolithiasis, glomerular disease, hypertension, dialysis therapies or transplantation; trials that contained at least one outcome illustration (graph or figure) in the published manuscript; and registration with a clinical trial registry. Exclusion criteria were: RCTs that presented secondary analyses of previously reported trials without a new trial registration; studies that presented post-hoc analyses of previously published trials; and studies that lacked a designation of trial phase in their registration.

Study outcomes

The primary outcome of this study was the presence of ORB, which was deemed to be present if either a graphical illustration was used in a clinical trial to present a secondary outcome (SO) or previously unregistered outcome (UO) ahead of the trial's primary outcome (PO), or if any UO was graphical illustrated in a trial manuscript.

The secondary outcomes of this study were concerning the presentation of clinical trial outcomes in the text of its published abstract. The outcomes assessed were: the number and proportion of a clinical trial registered POs published in the abstract; the number and proportion of a clinical trial registered SOs published in the abstract; and whether any UOs were published in a clinical trial's abstract.

Data extraction

Eligible trials were entered into the citation manager Zotero. Trial abstracts and clinical registration status was screened to ensure suitability for inclusion. Trials excluded were recorded, along with reason for exclusion, and presented in the study flow diagram (Fig. 1).

Fig. 1

Study inclusion flow diagram.

Full text review of suitable manuscripts was conducted for extraction of the publication year, randomized sample size, study phase, funding model of the trial (public vs industry) and the number and nature (primary, secondary or unregistered) of outcomes published in graphical illustrations and in the abstract. The clinical trial registration database pertaining to each trial was reviewed to identify the number and list of pre-specified primary outcomes and secondary outcomes. Outcomes labelled as “other” in registration were considered to be secondary outcomes for the purposes of this study.

Analysis plan

For each included trial, the outcome graphical illustrations were mapped to the registered study outcomes, and categorized as primary, secondary, unregistered or a combination of these outcomes. Graphical illustrations pertaining to study flow diagrams or study operational diagrams, such as graphs demonstrating separation between intervention and control groups regarding the intervention, were not included in the analysis. Adverse events reported by graphical illustrations were included in the analysis. Outcome reporting bias that favoured the reporting of statistically significant results was recorded, adjudged to have occurred if a statistically significant SO or UO was graphically illustrated ahead of a PO, or if any statistically significant UO was graphically illustrated.

Categorical data were presented as absolute numbers and percentage of total, and compared using the Chi-square test (χ2). Continuous data were presented as median values with interquartile range, and compared by student's t-test. Subgroup analyses were conducted regarding the prevalence of outcome reporting bias in relation to the source of funding of the trial, the year of the study, the phase of the RCT and the number of pre-registered outcomes. An alpha level of 0.05 was considered statistically significant for this study.

Results

The search strategy revealed 2,056 clinical trials, of which 75 were suitable for inclusion (Fig. 1). The median sample size was 162 participants (IQR 99–417), with industry funding reported in 55% of the clinical trials (n = 41) and 42% designated as phase 3 clinical trials (n = 31) (Table 1). The majority of trials had a single registered primary outcome (77%, n = 58). Fifty-two percent of the trials (n = 39) had >4 registered secondary outcomes. Publication of two outcome graphical illustrations in a trial manuscript was the most frequent practice, occurring in 39% (n = 29) of trials, followed by 23% of trials having either one (n = 17) or three (n = 17) published.

Table 1

Characteristics of Included Clinical Trials, stratified by Journal. Reported number (n) represent the number of clinical trials in each category.

	NEJM	Lancet	KI	JASN	CJASN	Combined
No. of RCTs	12	10	11	28	14	75
Sample Size(median, IQR)	454(158–1459)	185(112–1777)	118(106–265)	198(54–299)	110.5(59–312)	162(99–417)
Publication Year201520162017201820192020	1–1154	111421	11–54–	538453	33143–	1181118198
FundingPublicIndustry	66	37	56	1216	86	3441
Trial Phase234	–102	442	641	9109	833	273117
Registered Primary Outcomes12≥3	84–	9–1	10–1	2341	833	58116
Registered Secondary Outcomes≤45–8≥9	345	532	614	1675	653	362019
No. of Published Outcome Graphical Illustrations12345	651––	2521–	2423–	510832	25421	17291793

Abbreviations: RCT = randomized clinical trials, NEJM = New England Journal of Medicine, KI = Kidney International, JASN = Journal of the American Society of Nephrology, CJASN = Clinical Journal of the American Society of Nephrology.

Mapping of the graphical illustrations to the registered trial outcomes revealed that the first outcome graphical illustration was used to present a clinical trial PO in 47% of cases (n = 35) and a combination of a PO/SO, a PO/UO and a PO/SO/UO in 21% (n = 16), 8% (n = 6) and 8% (n = 6), respectively (Table 2). The first outcome graphical illustration was used to portray results that did not include a PO in 16% of trials (n = 12), a finding which was not significantly associated with any one specific journal (χ2 = 2.2, p = 0.68), trial phase (χ2 = 2.4, p = 0.29), funding model (χ2 = 0.7, p = 0.39), sample size (t = −1.4, p = 0.16) or the number of registered secondary outcomes (t = −1.07, p = 0.28). A clinical trial PO was not displayed in any graphical illustration in 7% of the clinical trials (n = 5), and 40% of these unpresented POs (n = 2) were not statistically significant. In trials that included a second published graphical illustrations (n = 58), the outcomes portrayed were predominantly a SO (n = 18, 31%) or an UO (n = 16, 28%).

Table 2

Distribution of reported trial outcomes by order of published graphical illustration.

Order of Published Graphical Illustration	Firstn (%)	Secondn (%)	Thirdn (%)	Fourthn (%)	Fifthn (%)
Eligible RCTs	75	58	29	12	3
Trial Outcome Reported	35 (47)	11 (19)	3 (10)	–	–
PO	4 (5)	18 (31)	12 (41)	7 (58)	1 (33)
SO	6 (8)	16 (28)	13 (45)	4 (33)	1 (33)
UO	16 (21)	6 (10)	–	–	–
Combination of PO/SO	6 (8)	5 (9)	1 (4)	1 (9)	1 (33)
Combination of PO/UO	2 (3)	2 (3)	–	–	–
Combination of SO/UO	6 (8)	–	–	–	–
Combination of PO/SO/UO

Abbreviations: RCTs = randomized clinical trials; PO = primary outcome, SO = secondary outcome, UO = unregistered outcome.

n = number of RCTs.

Primary outcome

Outcome reporting bias based on the choice of graphical illustration was present in 60% of the clinical trials (n = 45) (Fig. 2). In these 45 clinical trials, a SO or UO had been graphically illustrated prior to the PO in 16% (n = 7) of the trials, while an UO had been graphically illustrated in 96% (n = 43). Both of these criteria were present in 11% (n = 5) of the trials.

Fig. 2

Primary outcome occurrence in eligible RCTs, by individual medical journal and the combined cohort of RCTs.

The occurrence of ORB did not differ significantly between the journals (χ2 = 6.1, p = 0.14), with year of publication (χ2 = 5.2, p = 0.39) or with the trial sample size (t = −0.59, p = 0.55). Outcome reporting bias occurred in 54% of industry-funded trials (n = 22) compared to 53% of public-funded trials (χ2 = 1.29, p = 0.25), and in 63% (n = 17), 58% (n = 18) and 58% (n = 10) of phase 2, 3 and 4 trials, respectively, which was not statistically significantly different (χ2 = 0.15, p = 0.92). In the 45 clinical trials with ORB bias present, the offending graphical illustrations represented a statistically significant or positive findings regarding the intervention in 82% of cases (n = 37). Publication of a statistically significant versus non-significant UO by graphical illustration was not associated with trial sample size (t = −0.9, p = 0.36), the number of registered secondary outcomes (t = −1.1, p = 0.23), the trial phase (χ2 = 0.19, p = 0.9) or the funding model (χ2 = 0.2, p = 0.64).

Secondary outcomes

All clinical trials reported at least one PO in their abstract. Ninety-two percent (n = 69) of trials reported all registered POs in their abstract. All registered SOs were reported in 32% (n = 24) of the trial abstracts, whereas 24% (n = 18) of trials did not report any SOs in their abstract. The median number of SOs reported in the abstracts was 2 (IQR 1–3). An UO was reported in 63% (n = 47) of trial abstracts. This did not differ significantly between industry or publicly funded clinical trials (χ2 = 0.65, p = 0.41). Of those trials which reported an UO in their abstract, the reported UO was a statistically significant or positive result towards the trial intervention in 64% of occurrences (n = 30).

Discussion

In contemporary nephrology RCTs there was a high prevalence of ORB based on the choice of outcome graphical illustrations. This primarily resulted from the publication of unregistered outcomes through graphical illustrations. The occurrence of ORB was ubiquitous throughout the trials, and was not found to significantly differ based on the trial funding model, the trial phase, the randomized sample size or between individual journals. In a large majority of cases, the graphical illustrations leading to ORB represented outcomes that were statistically significant or positive results relating to the trial intervention.

Previous research has shown a similar high prevalence of ORB in RCTs, but no previous study has focused on the contribution of graphical illustrations to ORB [[6], [7], [8],[18], [19], [20]]. This is very surprising, given the key role that data visualization has assumed in the medical literature since the first principles were historically presented by authors such as Tufte and Cleveland, and particularly with the advent and growth of social media and electronic publishing [21,22]. There are many reasons why graphical illustration represent an important source of ORB. The human brain is designed to rapidly process visual data and the majority of the population are visual learners [23,24]. Data visualization is more user-friendly, provides superior reader satisfaction compared to text and may be more effective in terms of imparting knowledge [25,26]. The evolution of data visualization relates not only to the diversity of graphical illustrations that modern statistical packages can produce, but also developments such as the growing use of visual abstract infographics [[27], [28], [29]]. Infographics have become ubiquitous with the aid of user-friendly software, and present data in a logical and concise sequence. These visual abstracts capture a reader's attention and can deliver results in a persuasive manner, although their outright superiority over plain text still remains to be proven and is an area of active research [25,30,31]. Infographics can lead to significantly increased dissemination of research on electronic platforms compared to traditional media [32]. The logical assertion is that graphical illustrations can effectively draw and focus a reader on selected outcomes, hence facilitating ORB if used inappropriately. However, there remains no empiric methodological research examining how medical journal readers approach a trial manuscript, or what relative weight of importance are attributed to graphical illustrations compared to the text or abstract sections, which certainly warrants future research.

Despite the pre-requisite for clinical trial registration, there was a high level of unregistered outcomes being graphically illustrated. Since 2005, the International Committee of Medical Journal Editors has required mandatory trial registration, in an attempt to achieve full transparency and high quality in clinical trial reporting [33]. This has been further supported by design guidelines such as SPIRIT and PRISMA-P [34,35]. However, these measures have not proven to be the intended panacea, with non-registration of published trials and ORB remaining commonplace [6,7,[36], [37], [38], [39], [40]]. This inertia has been attributed to a lack of researcher awareness, but stricter enforcement by editorial teams is required, particularly moving away from the publication of un-registered trials or trials registered after outcome collection [[41], [42], [43]]. Despite the CONSORT guidelines requiring full disclosure of outcome measures altered during a trial, there is clear evidence that non-disclosure of changes, as well as other major deviations from CONSORT, have not proven to be a barrier to publication [44,45]. In this study, it is particularly poignant as the included studies were from five high-impact journals in the field of nephrology. In the future, the onus will need to be on the key stakeholders to be more rigorous in their adoption of CONSORT, with one suggestion that using an abbreviated form of the CONSORT checklist at clinical trial submission, followed by a full checklist submission at the time of acceptance, might improve overall compliance with CONSORT [46]. For journals that have endorsed the CONSORT guidelines, appropriate quality control must be in place to prevent low quality publications [[47], [48], [49]]. While the proposed introduction of core outcome sets will define what outcomes should be reported, the upcoming InsPECT/CONSORT-outcome extension will define how those outcomes should be reported [50,51]. Whether there will be any specific recommendation regarding the use of graphical illustrations is unknown, but the group's working documents suggest that authors will have to specifically describe how data will be presented [52]. Overall, through reduced reporting of “unimportant” outcomes, it will be hoped to avoid the lamentable occurrence of clinical trials that fail to translate into any benefits for patients [53].

Outcome reporting bias in this study typically presented statistically significant or “positive” results, which is consistent with previous reports [6,[54], [55], [56]]. The trial abstract, similar to graphical illustrations, is a highly visible component of the manuscript, and hence could be an important source of ORB. In our secondary outcome analysis, we examined the trial abstracts and found that the majority contained un-registered outcomes, with most also representing statistically significant findings when reporting their outcomes. This is consistent with previous studies of abstract reporting bias which have shown poor adherence with the CONSORT abstract extension, although a temporal trend towards improved abstract quality correlating to the extension guideline has been reported [45,[57], [58], [59]]. Combining this propensity for abstract reporting bias with the afore-mentioned use of visual abstract infographics in many of the high impact medicals journals, the trial abstract has the potential to become an even more potent contributor to ORB, unless stakeholders are to become more rigorous in their application of CONSORT.

As a unique study examining the role of graphical illustrations in ORB, these findings highlight a novel consideration for future researchers. The study also begins to address the current paucity of clinical trial methodological evidence regarding the optimal use of graphical illustrations. Although the study is limited in its size and scope, by focusing on contemporary nephrology trials in high impact journals and including only trials which have been registered, the results give a pragmatic insight into the current role of graphical illustrations in outcome reporting bias. The findings were consistent with previous findings relating to ORB in plain text.

Conclusion

Our findings suggest that the choice of graphical illustrations is an important source of ORB in contemporary nephrology clinical trials. Despite the increasing importance of graphical illustrations and visual data in clinical trials, there remains an absence of evidence in clinical trial methodological research to guide optimal use. While more rigorous engagement with the CONSORT guidelines by all stakeholders will be critical in the future to eliminate ORB, clear recommendations to direct the choice of published graphical illustration are desirable.

Author competing interest statement

All authors declare no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work; no other relationships or activities that could appear to have influenced the submitted work.

Funding

This study was not supported by any funding or grant

Transparency declaration

The lead author (FW) affirms that the manuscript is an honest, accurate and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as originally planned (and, if relevant, registered) have been explained.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.