The impact of mandatory conflict of interest disclosures on editors' manuscript acceptance decisions: A cross-sectional observational study.

Objective: Although current ethical standards mandate conflict of interest (COI) disclosure by authors of peer-reviewed publications, it is unknown whether disclosure affects a manuscript's fate. Our objective was to identify associations between author COI disclosure and editorial decision to publish.
Methods: We performed a cross-sectional observational study of editorial decisions for original research and brief research report manuscripts submitted to Annals of Emergency Medicine between June 2014 and January 2018 using data from the journal's editorial decision software and data from a prior study that characterized author COI for the same manuscripts. Outcomes of interest included final editor decision to publish (primary), initial editor decision, and number of revisions. We compared outcomes for manuscripts with COI versus those without and by type of COI (commercial/government/other).
Results: Out of 1312 manuscripts in the sample, 65.1% had no COI declarations, and 34.9% had one or more. Overall likelihood of editorial decision to publish was 13.5% (115/854) for articles without COI and 26.9% (123/458) for those with COI. Overall likelihood of editorial decision to publish was 19.8% (19/96) for articles with commercial COI only versus 33.3% (35/105) for those with government COI only. Conclusions: Articles with author-reported COI were more likely to be published than those without such a declaration. Additionally, results suggest that reports of government COI are associated with improved chance of publication. Authorities should consider relaxing COI requirements temporarily to allow investigators to perform larger scale, randomized controlled studies of the impact of mandated COI disclosure.

BACKGROUND

Leading authorities currently require conflict of interest (COI) disclosures for all peer‐reviewed articles. More than 5400 journals mandate reporting COI details using the International Committee of Medical Journal Editors (ICJME) format, which prompts authors to report potential COI or attest that they have no COI to declare. , According to the ICJME, the intent underlying COI disclosure is for the author to demonstrate a “commitment to transparency and help maintain trust in the scientific process.”

Importance

How COI disclosures affect a manuscript's likelihood of publication is not currently known. Our research group recently reported the results of a randomized controlled trial evaluating the impact of COI disclosure on journal reviewers’ ratings of papers they reviewed. In that study, COI disclosures had no impact on reviewers’ rating scores in a sample of 1480 manuscripts with 838 unique reviewers. However, the argument can be made that, although reviewers evaluate manuscripts, they do not ultimately make the decision to accept or reject because that is the role of the editor. To date, an analysis of the impact of COI disclosures on editorial decisions has not been reported.

Goals of this investigation

Our primary objective was to determine whether there was any association between author report of a COI and the likelihood of editorial decision to publish. In other words, we sought to examine the impact of COI declarations on editors by comparing the likelihood of publication for original research manuscripts that included COI disclosures versus those that revealed no disclosures.

METHODS

Study design and setting

This study was an additional evaluation of data reported in a related manuscript. In the original study, the influence of COI at the journal reviewer level was tested in a randomized controlled trial. In the current manuscript, we instead analyze the relationship between author COI disclosure and editors’ decision‐making. Note that the current study is an observational study (and not a randomized controlled trial) because editors were routinely provided COI disclosure information for every manuscript submission according to ICJME recommendations. , This is the standard procedure at most journals, and editors are not allowed to make thorough decisions without knowing the COI declarations.

The influence of conflict of interest (COI) reporting upon medical journal manuscript acceptance is unclear. In this analysis of 1312 manuscripts submitted to the Annals of Emergency Medicine, manuscripts with reported COI were more likely to be accepted than those without reported COI (26.9% vs 13.5%). The causality of this relationship is unclear. These findings suggest that further research is warranted to evaluate causality and inform future medical journal publishing practices related to COI reporting.

The main data source includes all editorial decisions for original research and brief research report manuscripts submitted to Annals of Emergency Medicine, a peer‐reviewed journal with an impact factor of 5.721, between June 2014 and January 2018. All data were collected automatically in real time via journal manuscript database software (Editorial Manager, Aries Systems, North Andover, MA, USA) at the time of submission. A subset of data was subsequently extracted from the database and analyzed in this study based on the following criteria: brief report or original research article; sent out for peer review (ie, not “desk‐rejected,” which is a rejection by an editor without peer review); and received a final decision of accept or reject.

Selection of participants

Data describing all editors’ decisions for manuscripts submitted during the study period were included. Decisions made by editors with less than 1 year of experience making publication decisions about original research studies (as compared to other editorial tasks) were excluded.

Measurements

Editorial decision data were obtained from the editorial manager database. Per ICJME disclosure rules, a COI disclosure was defined as any declaration of a conflict of interest for a given manuscript submission, including (1) grants, consulting fees, or honoraria; or (2) potentially influential financial relationships. COI disclosures were obtained through author self‐report, according to ICJME guidance. The determination of COI category was made by 2 coders who were blinded to the study intent. These 2 coders categorized disclosed COIs as government (eg, National Institutes of Health), commercial (eg, for‐profit company), university, non‐profit, or other by using information accessible on sponsor websites. Intercoder agreement was 95.8%. (See the appendix of the related manuscript for a comprehensive presentation of the coding process. ) Because a given manuscript may have included more than 1 type of COI, we lumped COI types together into the following categories that are conceptually similar: no COI, commercial COI only, government COI only, and other COI.

Outcomes

The primary outcome of interest was final editor decision to publish (accept or reject). Other outcomes of interest included initial editor decision (accept with revision, reject with reviews, and revise and resubmit) and number of formal revisions required by editor (0–4).

Analysis

We calculated rates of final editor decision to accept, initial editor decision, and mean number of revisions. In terms of statistical testing, we used a chi‐square test and Fisher's exact test where appropriate (alpha < 0.05) to determine whether there were statistically significant differences in outcome measures by subgroups (any COI vs no COI; commercial COI, government COI, other COI (including multiple COI types), and no COI).

Ethics

This study is an additional analysis of data that was collected after approval by the institutional review board at Harvard University (IRB: 24032). Our reporting conforms to reporting guidelines for observational research: The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE).

RESULTS

Characteristics of study sample

Data for 1312 articles (158 brief research reports and 1154 original research articles) were included in the analysis. Grants, consulting, fees, or honoraria were reported for 29.1% (382/1312) of submitted manuscripts, and authors for 13.5% (177/1312) of manuscripts reported other potentially influential financial relationships COI relationships in the prior 36 months. In total, 34.9% (458/1312) of manuscripts had at least 1 COI. For simplicity, all further discussions focus on combined COI related to submitted manuscripts. See Figure 1 for a flow chart showing inclusion and exclusion of articles in the sample.

FIGURE 1

Flow chart depicting how manuscripts were deemed eligible for inclusion as well as the primary outcome measure by subgroups. aOther COI includes multiple types of COI. COI, conflict of interest

Eighteen percent (238/1312) of manuscripts were ultimately accepted for publication after full peer review. For the entire sample, the median number of days under editorial review was 28, and the frequency of initial editorial decisions was reject with reviews (78.2% = 1026/1312), revise and resubmit (20.5% = 269/1312), and accept with revision (1.3% = 17/1312). See Table 1 for more detail.

TABLE 1

Primary and secondary measures for articles without COI versus those with 1 or more COI are displayed in the Table

		One or more COI
	No COI n = 854	All articles with COI n = 458	Commercial COI only n = 96	Government COI only n = 105	Other COI a n = 257
Final editor decision
Accept—n (%)	115 (13)	123 (27) c	19 (20)	35 (33)	69 (27)
Reject b —n (%)	739 (87)	335 (73)	77 (80)	70 (67)	188 (73)
Initial editor decision
Accept with revision—n (%)	7 (1)	10 (2)	2 (2)	2 (2)	6 (2)
Reject with reviews—n (%)	709 (83)	317 (69)	76 (79)	63 (60)	178 (69)
Revise and resubmit—n (%)	138 (16)	131 (28)	18 (19)	40 (38)	73 (28)
Number of formal revisions required by editor—n (%)
0 (Includes rejected)	713 (83)	320 (70)	76 (79)	64 (61)	181 (70)
1	49 (6)	56 (12)	8 (8)	16 (15)	32 (12)
2	63 (7)	68 (15)	11 (11)	20 (19)	36 (14)
3	24 (3)	11 (2)	1 (1)	3 (3)	7 (3)
4	5 (1)	3 (1)	0 (0)	2 (2)	1 (0)

Note: In the non‐shaded panels, primary and secondary measures are shown for articles that included COI acknowledgments, subdivided by COI type Abbreviation: COI, conflict of Interest.

Other COI includes multiple types of COI.

Reject includes a small number of articles that were withdrawn by authors before publication.

Odds ratio for acceptance of a manuscript with 1 or more COI = 2.4 (95% confidence interval 1.8–3.2).

Main results

Manuscripts with any COI declarations (as compared to those declaring no COI) were more likely to be accepted for publication (acceptance rate 26.9% [123/458] vs 13.5% [115/854], absolute difference 13.4% [P < 0.001]). Rates of acceptance by COI type were 19.8% (19/96) for commercial COI only, 33.3% (35/105) for government COI only, and 26.8% (69/257) for other COI.

LIMITATIONS

This study is limited by several aspects of its design. First, the study is an additional analysis of existing data from a study of the same manuscripts that focused on the impact of COI disclosures on reviewers rather than editors. Second, because the study was designed to measure the impact of COI disclosure on reviewers, our sample does not include so‐called “desk rejections,” which refers to when an editor decides the manuscript does not warrant additional peer review. Desk rejections are very common, and the decision is left entirely to a single editor, so future work should evaluate the relationship between COI disclosure and desk rejections. Third, COI disclosure data were self‐reported by manuscript authors (as required by ICMJE), so they may have been incomplete or erroneous. COI disclosures could be inaccurate for unintentional or intentional reasons, and it is possible that some authors choose to conceal certain COI to avoid perception of coercion. Fourth, there may be an interaction between author ability to produce high‐quality work and receipt of sponsor funding; this interaction makes it more difficult to understand what actions stakeholders should take in response to our findings. Fifth, we were unable to evaluate the relationship between academic appointments and rate of acceptance. Future work should be done to investigate this further. Sixth, our study design did not allow us to draw any inferences about causality. Finally, the study was done using data from only 1 journal; although its review and editorial practices are common, generalizability to other journals is unclear.

DISCUSSION

In this study of 1312 peer‐reviewed research articles that were reviewed by decision editors at Annals of Emergency Medicine, we found a positive association between the declaration of any COI disclosure and the chance that a manuscript would be accepted for publication. Additional analyses of the association between COI type and likelihood of acceptance suggest that disclosure of certain COI types, such as government COI, is associated with a higher chance of success. From this study, we cannot determine whether this is a causal effect or simply an association, because authors with COI disclosures might be more experienced or more highly trained on average than those without COI. We believe that having COI to disclose could be a surrogate marker for “better” research, increased availability of resources to conduct research, and/or increased pressure to publish research when extramural funding exists. However, we cannot rule out the possibility that editors are more likely to accept research because authors report certain COI. More research is warranted to further investigate these findings.

Some researchers have questioned the ability of peer reviewers and/or physician readers to account for COI in the appraisal of scientific literature. For example, physician subjects in Silverman's vignette‐based study frequently reported that they would “discount” the credibility of study results when a significant COI was reported; however, in practice, physicians were able to effectively perform discounting only in a simple direct comparison between 2 studies. Silverman concluded: “This brings into question the effectiveness of merely disclosing the funding sources of published studies.” Similarly, in our original study of the impact of COI disclosure on peer reviewers, we found no evidence that COI disclosure affected numerical ratings by real peer reviewers. This suggests that mandating submission of author COI information has little impact.

On the other hand, some researchers have concluded that providing COI information could unfairly inhibit adoption of new evidence funded by industry sponsors. In Kesselheim's vignette‐based study of physician reactions to hypothetical clinical trial abstracts, physicians reported that they would be less likely to prescribe new medications in the presence of industry funding for the trial. In fact, disclosure of industry funding led physicians to downgrade the rigor of a trial, confidence in the results, and willingness to prescribe drugs—even for drug trials with high methodologic rigor. His team's conclusion was that industry sponsorship negatively influences the perception of methodological quality and reduces willingness to believe and act on trial findings, which could inhibit translation of clinical research into practice. However, in general, vignette studies have limited external validity, because they are not evaluating real‐world behavior. The implications of the Kesselheim et al. study are further limited because subjects read only abstracts rather than complete manuscripts, which affects both internal and external validity of the results. Further research is warranted to determine the impact of COI on readers’ interpretations of study results. We recommend that a randomized controlled trial be done to test the impact of COI disclosures on readers’, peer reviewers’, and editors’ trust and willingness to act on study results. Such a study could also involve presenting fabricated studies with and without COI of various types in the normal editorial queue for 1 or more journals.

Under current policy conditions, our ability to study the impact of COI disclosures is hampered by contemporary best practices: It is considered unethical to perform a randomized controlled trial on editorial decisions because declaration of COI is mandated for all manuscripts. As a result, all editors in our study were provided COI declarations for all the papers they were assigned, per journal standard practice. Unlike peer reviewers in the original study who were randomly blinded to COI, there was no group of editors who were randomized to be deprived of COI information. Within the current decision structure at Annals of Emergency Medicine (which is very typical of most journals), there is no way to do a true randomized controlled trial without substantially changing the way that decisions are handled. For the time being, this means we must continue to use a model that has not demonstrated efficacy in actual practice.

Moving forward, we recommend that journals continue to require COI disclosures from authors until our understanding of their impact is better. Meanwhile, researchers should further investigate how reviewers and editors handle manuscripts differently in the presence of COI. To enable well‐designed research that can definitively evaluate the effectiveness of COI disclosure authorities such as ICJME should consider temporarily relaxing COI disclosure rules. Without rigorous studies, it will be impossible to discern whether current COI disclosure rules have the intended effect of providing actionable information that can prevent publication of unduly biased research.

CONFLICT OF INTEREST

Michael L. Callaham served as Editor‐in‐Chief of Annals of Emergency Medicine, the journal that was studied in this article, during the time of data collection and analysis. He has received salary support from the journal.

AUTHOR CONTRIBUTIONS

Michael L. Callaham conceived and designed the study, implemented and supervised data collection, and interpreted the results. Newton Addo assisted with analysis of data and statistical testing. Carl T. Berdahl interpreted the results and drafted the manuscript. All authors contributed substantially to editing the final manuscript. Carl T. Berdahl attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. Carl T. Berdahl and Michael L. Callaham take responsibility for the accuracy of the final product.