Possible Bias in the Publication Trends of High Impact Factor Anesthesiology and Gastroenterology Journals -An Analysis of 5 Years' Data.

BACKGROUND: We hypothesize that being an editorial board member (EBM) in a high impact factor specialty medical journal increases the chances of publishing in the same journal.
MATERIALS AND METHODS: The publication trends of the first five EBMs in the five highest impact factor Anesthesiology and Gastroenterology journals were analyzed. Preceding 5 years' publications appearing on PubMed were grouped into as follows: number of publications in the journal in which the EBM serves (N1), number of publications by the same author in the other four highest impact factor (IF) journals (N2) and number of publications in all the other journals (N3). We evaluated the probability of the observed distribution of publications in the five highest IF journals happening by chance alone, assuming that all the EBMs had the same opportunity of publishing in any of these journals. The probability of publishing in their own journal was assumed to be one fifth.
RESULTS: The EBMs published their manuscripts in their own journal at a very high frequency. Encompassing all ten journals, the calculated P value for such a distribution was <0.001. In two journals, Anesthesia and Analgesia and Anaesthesia, the EBMs' publications in their journal were more than twice the cumulative total in the remaining four journals. In three of the five gastroenterology journals analyzed, combined publications of the five EBMs were greater in their own journal than the remaining four journals combined.
CONCLUSIONS: Despite proclaimed fair peer review process, EBMs seem to get preference in their own journals.

BACKGROUND AND GOAL OF STUDY

In a recent commentary, Richard Horton, the editor in chief of Lancet began by saying “a lot of what is published is incorrect,” and ended with “the good news is that science is beginning to take some of its worst failings very seriously. The bad news is that nobody is ready to take the first step to clean up the system.”[1]

He was referring to the unreliability of considerable information related to scientific publications. Most of his observations are well known, yet never discussed except in closed corridors. However, there is another practice that relates to publication bias (again well known, but never studied) which plagues many journals, especially those with high impact factors.

The aim of every researcher is to publish their work in high IF journals. Medical researchers are no exception. Considering the small pool of high impact medical specialty journals, the space available for researchers working in these fields is even smaller. Generally, many of these journals claim an acceptance rate of 20%–25% on their websites. However, the experience of many authors is vastly different. Frequently, manuscripts with important findings are rejected for reasons difficult to comprehend or explainable by the reviewers' comments and vice versa.[2] It is well known that in 1920, Queen's University, Belfast, United Kingdom, rejected Sir Ivan Magill's MD thesis, which was based on endotracheal anesthesia as it considered this unlikely to be of much value.[3] It is also difficult to understand how the publication “Orthopaedic surgeons: as strong as an ox and almost twice as clever? Multicenter prospective comparative study” could find a place in British Medical Journal, even as a spoof.[4] Any careful observer would have noticed that certain author names appear more often than others and that many of them happen to be editorial board members (EBMs) of the same journal.

Based on these observations, we hypothesize that, being an EBM of high IF specialty medical journals, increases the chances of publishing in the same journal.

MATERIALS AND METHODS

We chose the top five journals in the field of anesthesiology and gastroenterology, as rated by Thomson Reuters Corporation generated IF. The anesthesiology journals included were-Anesthesiology (official Journal of the American Society of Anesthesiology), British Journal of Anesthesia (official journal of the Royal College of Anesthetists, London), Anaesthesia (official publication of the Association of the anesthetists of Great Britain and Ireland), Anesthesia and Analgesia (official journal of the International Anesthesia Research Society) and European Journal of Anesthesiology (official publication of the European Society of Anesthesiology). The gastroenterology journals analyzed were Gastroenterology (one of the three publications from the American Gastroenterological Association Institute), Gut (official journal of the British Society of Gastroenterology), American Journal of Gastroenterology, Nature Reviews Gastroenterology and Hepatology and Clinical Gastroenterology and Hepatology. We analyzed the sources of publications that appeared in the last 5 years and authored by the first five consecutively listed EBMs, as per the journals website. Only publications that appeared on the Pubmed and associated with the EBM's names were examined and entered on an excel spreadsheet. To minimize any errors, the names appearing on the author names section of the abstract on PubMed were compared with the names and affiliation appearing on the journal website. Rarely, the author's information was unavailable on Pubmed. In such an event, the journal's website was searched for the relevant publication. Occasionally, it was necessary to access the full text of the publication to get the authors information. All the data were collected during the last 6 months of 2015.

The publications included were from all the categories as follows: editorials, case reports, letter to the editor, opinions, original scientific studies, etc. As it was important to have the data available for independent verification, the “screenshot” of every publication was stored in a database.

The principal author is an anesthesiologist with a keen interest in gastroenterology had experienced rejection of many manuscripts from high IF anesthesiology and gastroenterology journals. Many of these manuscripts were subsequently published in journals with lesser impact factor and were extensively cited. He had observed that some of the author names appeared more often than others, and they happened to be EBMs. As a result, we decided to study the existence of EBM induced bias in the high impact factor journals from these two specialties.

RESULTS

The data obtained was analyzed using SPSS version 23 (IBM, Inc., USA). It is presented under three headings: number of publications in the journal in which the EBM serves (N1), number of publications by the same author in the other four highest IF anesthesiology and gastroenterology journals (N2) and number of publications in all the remaining anesthesiology and nonanesthesiology or gastroenterology and non-gastroenterology journals as the case may be (N3) [Tables 1 and 2]. The distribution of frequency data was analyzed by using single sample Chi-squared test after Yates' correction for every author and the journal. If the P value for the distribution was >0.025 (single-tailed), the distribution was attributed to chance.

Table 1

Publication trends of the first five editorial board members from the first five Anesthesiology journals (as rated by Thomson Reuters Corporation generated impact factor) over the preceding 5 years. The table also displays the cumulative publication trends

Table 2

Publication trends of the first five editorial board members from the first five Gastroenterology journals (as rated by Thomson Reuters Corporation generated impact factor) over the preceding 5 years. The table also displays the cumulative publication trends

We evaluated the probability of the observed distribution of the number of publications in the five highest IF journals happening by chance alone. To facilitate this, we assumed that all the EBMs had the same opportunity of publishing in any of the five highest IF journals and as a result, the probability of publishing in their own journal was calculated to be one fifth, i.e., 0.20. We acknowledge that this might not be a valid assumption. Three EBMs from anesthesiology and four from gastroenterology journals had a higher number of publications in their “own journals” in comparison to a cumulative total of all other journals (anesthesiology and nonanesthesiology or gastroenterology and nongastroenterology as the case may be) [Tables 1 and 2]. Two EBMs published exclusively in their own journals with no publications in any other journal over the study period. Both are EBMs of Nat Rev Gastro Hepat with 50 and 16 publications each.

For all the ten journals, the distribution of the cumulative total from all the EBMs was analyzed. A calculated P = 0.001 across all the journals indicates that the probability of the observed total number of publications of all EBMs in their own journal (in comparison to either the remaining four high IF specialty journals) happening by chance alone is <1 in 1000.

Figure 1 is a graphical representation of the cumulative publication trends of all the first five editorial EBMs in the field of Anesthesiology. It is clear that for every journal analyzed, combined publications of the five EBMs were greater in their own journal than the remaining four high IF journals combined. In the two journals, Anesthesia and Analgesia and Anaesthesia, the number of publications in own journal was more than four times and nearly thrice, respectively, than the combined total of the other four journals. Regarding percentage of publications in one's own journal to the total published in all five highest IF journals combined, Anesthesiology fared the best and Anesthesia and Analgesia fared the worst. In other words, “publication bias” was most in Anesthesia and Analgesia and least in Anesthesiology. EBMs of European Journal of Anesthesia published more manuscripts in journals other than the five highest IF anesthesiology journals compared to others. The sum total of Anesthesia and Analgesia and Anaesthesia EBMs was higher in their own journal than the cumulative total of all nonfive highest IF anesthesiology journals combined.

Figure 1

Bar graph showing the cumulative publication total of all five editorial board members for each high impact factor journal in the field of anesthesiology. Black- total number of publications in the journal where the author is the editorial board member, blue- total number of publications in the remaining four highest impact factor anesthesiology journals, Lavender- Number of publications in all other PubMed indexed journals except the five highest impact factor journals

Figure 2 is a graphical representation of the cumulative publication trends of the first five editorial EBMs in the field of Gastroenterology. In three of the five journals analyzed, combined publications of the five EBMs were greater in their own journal than the remaining four journals combined. Gastroenterology and Nature Reviews Gastroenterology displayed the most “publication bias” regarding publishing in their own journals. In Nature Reviews Gastroenterology, all the publications of the first five EBMs were in their own journal, with none in the remaining four high IF Gastroenterology journal. Similar to Anesthesiology as a specialty, four of the five Gastroenterology journals had a higher number of total EMBs publications in nonhighest five IF journals.

Figure 2

Bar graph showing the cumulative publication total of all five editorial board members for each high impact factor journal in the field of gastroenterology. Black- total number of publications in the journal where the author is the editorial board member, blue- total number of publications in the remaining four highest impact factor Anesthesiology journals, Lavender- Number of publications in all other PubMed indexed journals except the five highest impact factor journals

Table 3 provides the publication trends of individual editorial board members of high impact Anesthesiology and Gastroenterology journals.

Table 3

The publication trends of individual editorial board members of high impact Anesthesiology and Gastroenterology journals

DISCUSSION

In a book titled The Truth About the Drug Companies: How They Deceive Us and What to Do About It., Dr. Marcia Angell, a former editor in chief of the New England Journal of Medicine wrote.

“It is simply no longer possible to believe much of the clinical research that is published or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor of The New England Journal of Medicine.”[5]

Our findings add another element of uncertainty to the believability of what is published in high IF specialty journals. Obviously, all EBMs publications were peer-reviewed. Yet, the EBMs could publish the majority of their work in only the journal they were representing. It is difficult to believe that these EBMs did not consider the remaining four equally reputed journals in their specialty for submission. It is also problematic that only the journal they were involved in, regarded them as experts and invited to write editorials, opinions, or commentaries.

One way of explaining our findings are that a robust editorial policy and a fair peer review process is lacking. The responsibility for maintaining “fairness” in publishing ultimately rests with the editor in chief. He has to assure that no undue advantage is taken by or given to any single contributing author. An alternative explanation is that all the publications from the EBMs were of high caliber and worthy of publication in their own.

Nepotism in medical journal publishing is probably a closely guarded “secret.” We did not find any similar study examining the relationship between personal attributes of the contributing author and the success of manuscript acceptance. However, several writers have examined the role of editors, reviewers and the review process itself in medical publishing as discussed in the succeeding paragraphs.

The most commonly adopted method in almost all scientific publications is IMRaD (Introduction, Methods, Results, and Discussion), editorial review, peer review, and editorial decision.[6] Typically, prestigious journals put all submissions to a preliminary triage before deciding if they are worthy of peer review. If they are unworthy, a computer-generated letter promptly follows. In a study, Siler et al., examined the effectiveness of the peer review process in three major journals, namely, Annals of Internal Medicine, British Medical Journal, and The Lancet.[2] They concluded that the criterions used for rejecting the manuscript were never made clear to submitting authors. Although the papers that were not sent for peer review or received low reviewers scores received fewer citations when eventually published, many highly cited manuscripts were rejected by high impact journals. It is a failure on our part as medical professionals/specialists that this 300-year-old publication process has neither evolved nor been challenged. It is argued that even flawed peer refereeing is better than not being peer reviewed at all.[7] It is the biggest irony of our times that the most important element of the publication process has no evidence to support that it works and is open to misuse. Worse, all the evidence published is based on this flawed system. Perhaps, it is time to revisit the whole process.

Issues like ethics in medical publishing and conflicts of interest have been discussed before.[89] Unethical and unscientific behavior by the authors has led to many retractions, sometimes from high impact journals.[101112] Patients lives could have been put at risk because of the findings contained in retracted studies.[13] In this study, Steen evaluated 788 retracted English-language papers published from 2000 to 2010. Of the 788 studies examined, 180 were primary studies. The findings of the primary studies were followed by or involved in 851 secondary papers. In the primary studies, over 28,000 subjects were enrolled, and over 9000 patients were treated. Similarly, in the 851 follow-up studies >400,000 subjects were enrolled, and 70,501 patients were treated. The number of subjects that were potentially or actually harmed was probably more than this, as those that were treated based on the published findings and outside the scope of the research is unknown. If the disproportionate representation by the EBMs in their own journal publications is the result of favoritism, the results may be questionable. Considering that decisions such as evidence-based treatment, federal grants, promotions, and further studies are related in some way to the published evidence, these can be swayed by few publishers with influence on the journal. More importantly, people's lives could be endangered if any undue influence was exerted to publish questionable manuscripts in these high impact journals. Our findings certainly support a discussion in this area.

How do we address these concerns that might be present in the minds of our colleagues? A few suggestions are appropriate.

All journals send their manuscripts for peer review. In our experience, high IF journals do not conceal the names and institutions of the submitting authors. One publication group that conceals these details is “medknow,” a small publishing house.[14] Concealing the author's identity and their affiliation is known to positively impact the review quality.[15] Having the identifying information (name, institutional affiliations) of the authors can create bias in the minds of the reviewer. One wonders the need to identify the author and institution source, as the science should stand on its own strength. We recommend that everyone including the editor-in-chief and the manuscript handling editor should be blinded to this information.

The second concern is the practice of some journals mandating the authors to submit the names of potential reviewers. These names are supposedly unfamiliar to the submitting author! The problem is compounded by the practice of some of these journals that send the manuscript to the recommended reviewers. Although the practice helps the journals to increase the pool of potential reviewers, these names should be automatically excluded from the review process of the manuscript in question.

The third issue is the review process itself. Having reviewed many manuscripts over the years and studied the input from other reviewers, we think that the process is not standardized.[2] Frequently, excessive importance is attached to the language, grammar, and syntax than the methodology, results, and conclusions. The editor has an ultimate say in the decision to accept or reject a manuscript. We are not sure if this responsibility is used impartially. We suggest establishing a pool of international reviewers for all journals in one specialty to choose from, much like neutral umpiring in international sports. The manuscript should be sent to 5–6 random reviewers from the appropriate pool and the first 2–3 reviews should be considered while making the decision. The confidential message to the editor should be made away with. The reviews, reviewers, and the editorial board decision should be available for online viewing for the published manuscripts. If the reviewers know that their comments are available for all interested readers, they are likely to be more objective in their recommendations.

Declaration of conflict of interest is a common practice in all journals. Just being a member of the editorial board should be seen as a conflict of interest for submission in the same journal and should be declared.

At least in the United States of America, there are various “tracks” into which clinicians/physicians are employed at academic institutions. Some of these tracks have the dictum “publish or perish.” As a result, they may risk their job if unable to get promoted within a certain time limit. This might force many researchers and their mentors to solicit opportunities and platforms to publish. This might be a reason that we see guest editorials (that should be written by true experts in the field) frequently contributed by upcoming researchers. Over the years, there has been a major increase in the number of editorials (many issues will have 4–5). In addition, there are commentaries, etc., which are published regularly. These might potentially be perceived as an avenue created for increasing the total number of publications of certain favored faculty members for meeting promotion criteria. Maybe, it is time to delink the promotion from the total number of peer-reviewed publications. These should be connected with original scientific work and their impact on the practice of medicine. Encouraging questionable people to publish these “opinions” (editorials, commentary, etc.,) will only serve to occupy space in the servers (fortunately not the printed papers).

The publishers of all high IF journals are very likely to rebut any suggestion of unethical practice in this regard. Instead of mounting any defense, it is important to introspect with an aim to find solutions to end such dubious practices. We hope that the implementation of our suggestions might improve transparency and restore the trust of the medical community and patients.

Limitations of the study

It is possible that there is a requirement for EBMs to contribute editorials and commentaries on a regular basis. However, one overarching fact remains, that editorials and commentaries need not be contributed by EBMs on a frequent basis. As editorials and commentaries need to be authoritative, it might serve the scientific community better, if they are written by the experts in the field. If the EBMs are indeed experts in the field, it is hard to understand the reasons for their absence in other (top four) preeminent journals in the same field.

Our analysis presumed the probability of publishing in all top five journals to be equal. Most authors would like to publish in the highest IF journal and submit their manuscript to this journal first; thereby, increase the probability of publishing in that journal. Our assumption might be incorrect for two reasons. First, the EBMs loyalty might encourage them to submit to their own journal and the same factor might increase the chances of acceptance. The loyalty factor might also discourage them submitting to other high/higher IF journals that might be presumed as competitors. Second, every journal likes to have a niche and like to accept manuscripts in that narrow focus. Nevertheless, our findings show extremely skewed publication practices that cannot be explained by the above factors. It is even less likely as this trend is seen in almost all the EBMs analyzed.

Another possibility is that all the publications from the EBMs were of high quality that deserved a place in their own journals. It is beyond the scope of any small group like ours to assess the quality of these publications.

Nature Reviews Gastroenterology might be an exception considering that the journal stipulates the editor in chief to write an editorial every month. However, the fact remains that the journal did not make an effort to find other writers worthy of contributing for most articles.

We decided to analyze the publication trends of the first five EBMs. Choosing the members at random from the EMBs list was another approach we considered. However, an allegation might be made that we deliberately selected the biggest “defaulters” and falsely labeled them random. By choosing the first five members, we maintained transparency, consistency and probably eliminated any bias in selection. We have deliberately avoided mentioning the names of the EBMs; however, they can be easily deciphered by interested readers or journals. We are not sure if the results change by choosing the five EBMs at random or studying the publication trends of all the EBMs.

It was also suggested that the publication history of the EBM before taking their role should be analyzed. Such data were not uniformly available. Subgrouping the publications into various categories might yield different results. However, it was felt that sufficient expertise is available outside of the group of EBMs who can contribute editorials and commentaries, as a result, this is an indication of bias.

CONCLUSIONS

We conclude that the correlation between being an EBM in a high IF journal and the frequency of publishing their work in that journal is very strong and unlikely to be by chance alone. Further, it is possible that the EBM might have an undue advantage because of their position in the board that might have increased the chances of publication in that journal. The study was restricted to the top five journals rated by IF in the fields of gastroenterology and anesthesiology; however, the practice might easily be widespread. This practice might be deemed to violate publication ethics and probably requires an investigation by an independent federal agency. It is hoped that our findings are not just one symptom of a deeper malady.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.