Attitudes of editors of core clinical journals about whether systematic reviews are original research: a mixed-methods study.

OBJECTIVES: In 2009, not all journal editors considered systematic reviews (SRs) to be original research studies, and not all PubMed Core Clinical Journals published SRs. The aim of this study was to conduct a new analysis about editors' opinion regarding SRs as original research.
DESIGN: We conducted a survey and qualitative interview study of journal editors. PARTICIPANTS: All editors listed as editor-in chief of 118 PubMed Core Clinical Journals.
METHODS: We contacted editors via email and asked them whether they considered SRs original research, whether they published SRs in the journal and, if yes, in which section. We searched PubMed for any SRs (or meta-analyses) published in the included journals in 2017; if we did not find any, we hand-searched these journals. Editors were invited to participate in a follow-up qualitative interview study.
RESULTS: We received responses from 73 editors representing 72 (62%) journals. Fifty-two (80%) editors considered SRs original research, either for any type of SR (65%) or only for SRs with a meta-analysis (15%) and almost all (91%) of editors published SRs. Compared with the results of the 2009 study of Core Clinical Journals, a similar proportion of editors considered SRs to be original studies (71%), accepted SRs as original on certain condition such as presence of meta-analysis (14%) or published SRs (94%). Interviews with editors showed that they used various criteria to decide whether a SR is original research, including methodology, reproducibility, originality of idea and level of novelty.
CONCLUSION: The majority of editors of core clinical journals consider that SRs are original research. Among editors, there was no uniform approach to defining what makes a SR, or any study, original. This indicates that the concepts of originality of SRs and research are evolving and that this would be a relevant topic for further discussion. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This mixed-methods study combines an online survey with qualitative research methods.

This study had a high response rate of editors (62%).

The study provides more detailed data about editors’ reasoning about the originality of systematic reviews (SRs).

A limitation of this study is our use of a sample of journals indexed as PubMed Core Clinical Journals and the possibility that some journal editors were surveyed both in the earlier study and in this follow-up study, although none of the editors indicated that they participated in the earlier study.

The study relied on PubMed’s indexing to identify SRs and meta-analyses published by the included journals; hand-searching of all journals may have yielded some additional SRs.

Introduction

Global scientific output is growing exponentially,1 leaving users of research evidence to grapple with numerous individual studies, which may have different conclusions even if they cover the same topic. The development of research methodology for qualitative and quantitative synthesis of evidence led to the emergence of systematic reviews (SRs) as research synthesis method.2 This was also suggested in previous studies examining the status of SRs in research community.3 4

In 2012, Meerpohl et al published a survey of editors of core clinical journals regarding their attitudes about the value of SRs for journals.3 The study included 65 editors of the 118 journals, who were surveyed in April 2009. The editors were asked if they considered SRs to be original research, whether their journal published SRs and in which section SRs were published. The results indicated that 71% of the respondents considered SRs to be original studies and the majority of their journals published SRs.3

A study published in 2017 examined the acceptance of SRs as a doctoral thesis in European biomedical doctoral (PhD) programme.4 Almost half of the surveyed participants, who identified themselves as being in charge of doctoral programmes, reported that in their institutions a SR is an acceptable piece of research for an entire or at least part of a PhD thesis, but the majority of surveyed individuals did not have sufficient knowledge about basic concepts of SR methodology.4 However, more than a half of the participants indicated agreement with the statement that ‘systematic reviews do not produce enough new knowledge for a dissertation’. A third of the respondents indicated that there was a lack of appreciation for SR methodology among faculty members.4

In the years since the study of Meerpohl et al was conducted, the number of published SRs has increased, as well as their influence.5 On the other hand, it has been suggested that there is an overproduction of redundant, misleading and conflicted SRs and MAs in addition to the marked publication growth.6 The aim of this study was to follow up on the attitudes of journal editors towards SRs. We hypothesised that the proportion of editors who consider SR to be original research had increased since 2009 and that more of the surveyed journals now publish SRs.

Methods

Survey

Participants

We invited editors of 118 journals labelled as Core Clinical Journals by the National Library of Medicine, USA in February 2018.7 The list of journals that were considered Core Clinical Journals in 2009 (when the previous study was conducted)3 and in 2018 is available in online supplementary file 1. We retrieved editors’ names and contact details from journal web sites. Ten journals indicated that they had more than one editor-in-chief; eight of those 10 journals had two editors-in chief, one had three and one had four editors-in-chief. In total, there were 131 editors-in-chief listed for the 118 journals.

Survey

We surveyed editors via email, using the following four questions: (i) Do you consider a systematic review manuscript an original research project? (ii) Do you publish systematic reviews in the journal you edit? (iii) In which section of your journal would you publish a systematic review? (iv) Would you participate in a follow-up qualitative study via Skype?

We sent up to three email reminders approximately 1 week apart (MK, LP). We did not send reminders to the editors who responded that they do not wish to participate in the study.

Each study participant was assigned a code and coded responses were entered in to a spreadsheet with anonymised responses that were shared with other coauthors. One author initially evaluated and categorised responses (MK), in consultation with another author if necessary (LP).

Analysis of systematic reviews published in targeted journals

For all included journals, we assessed the types and characteristics of the published SRs. We performed a search on PubMed using journal name and limits for SRs and meta-analyses (MAs) and for articles published in 2017; search results were screened by two authors independently to verify that these publications were indeed SRs or MAs. Characteristics we analysed in those SRs are shown in box 1. We hand-searched the contents of journals published in 2017 if we did not find any SRs or MAs by searching PubMed. If by hand searching we did not find any SRs or MAs published in the journals included in 2017, we evaluated instructions for authors of journals that did not publish any SR to see if they had any guidance about submitting SRs and MAs.

Methods: We analysed the following characteristics of systematic reviews (SRs) published by target journals in 2017: (i) topic (therapeutic, epidemiology, diagnosis/prognosis or other), (ii) types of studies included, such as randomised controlled studies (RCTs), non-randomised studies (NRS), both RCTs and NRS, SRs or meta-analyses (MAs), SR/MA and primary research, SR and overview of SR (OSR) and those that did not define which studies were included, (iii) presence of meta-analysis (yes, no), (iv) update of a previous review (yes, no), (v) type of review (SR, OSRs, rapid review, scoping review).

Results: Detailed analysis of 1187 published SRs indicated that they most commonly addressed therapeutic interventions (n=585, 49%), epidemiology (n=281, 24%), diagnostic accuracy (n=105, 9%), prognostic issues (n=69, 6%) or other topics (n=147, 12%).

Regarding the type of included studies, the majority of SRs did not indicate in their methods what kind of studies were eligible for inclusion (n=301, 25%). There were 295 (25%) SRs that included both RCTs and NRS, 281 (24%) studies included only RCTs and 264 (22%) studies included only NRS. In 28 (2%) SRs, both primary research and SR/MA were included; 17 (1%) reviews included SR/MA and 1 review (0.1%) included both SR and OSR in the review.

Of all the analysed SRs, only 19 (2%) were an update of a previous SR. In 750 (63%) SRs, a meta-analysis was conducted. Most of the reviews were classic SRs that included only primary studies (n=1126, 95%); 28 (2%) SRs included both primary research and SR/MA so we defined them as SR/OSR; 19 (2%) OSRs, including one study that did not define which studies it included but called itself an OSR and one that included both SR/MA and OSR. Twelve (1%) SRs were scoping reviews and one was a rapid review (0.1%).

Statistics

We analysed data using frequencies, percentages and IQR to describe the results. Differences in proportions of key results between this study and the previous study3 were analysed with the χ² test. Proportions were reported with 95% CI. For analyses, we used MedCalc (MedCalc, Mariakerke, Belgium). Statistical significance was set at p<0.05.

Qualitative study

We contacted the editors who accepted to take part in the follow-up interview when responding to the survey. Participants were informed who will conduct the interview and that the authors intend to publish the study results.

Interviews

The qualitative study was conducted using semistructured interviews. Twenty-four editors volunteered to be interviewed. After 15 interviews, we stopped inviting further editors because we reached the level of saturation of identified themes which is usual methodology in qualitative research. The first interview was conducted on 6 July 2018 and the last on 7 September 2018. We used open-ended semistructured questions to enhance the discussion about originality of SRs (online supplementary file 2). Participants were not led to provide specifically any answer. We used the Consolidated Criteria for Reporting Qualitative Studies (COREQ) to guide the reporting of the study.8

Research team and reflexivity

One author conducted all interviews (LP). The author that conducted interviews did not personally know any editors that were invited to participate.

Conduct of interviews

Interviews were conducted individually via teleconferencing software (Skype or Zoom) or via telephone, based on the choice of participants. All conversations were recorded. After each interview, a transcript was made and analysed, to monitor for the point of information saturation. Transcripts and recordings of interviews were not sent to study participants for checking, commenting and/or correction. Instead, the final draft of the manuscript was sent to the editors for their insight into the collected results.

Data analysis and reporting

To ensure uniformity, all transcripts of interview recordings were made by one researcher (LP), and another member of the team checked all transcripts and analysed them (MK). For the purpose of reporting, the participants’ names were coded. The transcripts were analysed via conventional qualitative content analysis.9 10 The transcripts were first read as a whole to obtain general attitude of one editor, and afterwards the text was analysed word-by-word highlighting parts of the text that produced meaningful units, including complete sentences or parts of sentences, and then a code was attributed to those selected parts of text. Codes were sorted into categories related to editors’ attitude towards SRs., Two researchers did the coding; the results were then compared, and any discrepancies were resolved by discussion until reaching consensus. The codes were entered into an Excel spreadsheet for further quantitative analysis. Complete sentences were also copied from the interview transcripts to provide original thoughts of editors in the manuscript.

Results

Survey among editors of core clinical journals

Of 118 contacted journals, we received responses from 73 editors representing 72 (62%) journals. Interestingly, two editors from a single journal gave us opposite answers to our survey questions. Response rate based on the number of editors was 73 out of 131 (56%). Online supplementary file 1 shows a list of included Core Clinical Journals in 2009 and 2018. Since 2009, 13 new journals were added to the list of Core Clinical Journals, while the same number was dropped from the list. So the total number of Core Clinical Journals in both years was the same.

Editors’ responses

Of the 73 respondents, 8 (11%) did not answer the first question despite our repeated attempts to obtain missing answers. Among the remaining 65 editors who responded the first question, 42 (65%) considered SRs to be original studies, 13 (20%) did not and 10 (15%) indicated that only a SR with a MA was an original study. Overall, 52 (80%) editors considered SRs original research, either for any type of SR or only for SRs with a meta-analysis. This proportion was 71% (46 of 65) in the previous study;3 the difference was not statistically significant (p=0.23; χ2=1.412, df=1; 9% increase, 95% CI −5.8 to 23.3). The proportion of editors with conditional acceptance of SRs was 15% (10 of 65). In the previous study,3 it was 14% (9 of 65). The difference was not significantly different (p=0.87; χ2=0.026, df=1; 1% increase, 95% CI −11.4 to 13.4).

Four editors did not answer the second question. Of the remaining 69 editors, 63 (91%) responded that they had published SRs in their journals, 6 (9%) that they had not. In the previous study, 94% (60 of 64) of editors indicated they published SRs; there was no significant difference between these proportions (p=0.51; χ2=0.425, df=1; 3% increase, 95% CI −6.9 to 12.8).

Out of 63 editors that declared they had published SRs, 16 (25%) editors published SRs in a journal section devoted to original studies, 19 (30%) in the review section or separate section of the journal, 10 (16%) of them publish SRs in the section for original studies if they have a MA, and in the review section if they do not.

More than a quarter of editors (n=18, 29%) indicated that they did not have a particular section for publishing SRs in their journal and that they simply published them in the section that corresponds to the topic of the manuscript. Five (8%) editors did not answer the third survey question.

Even though 10 journals had more than one editor-in-chief, we received response from more than one editor from only one journal, for which two editors responded. From those two editors we received opposite responses on the first two questions. One did not consider SRs original and said that their journal generally does not publish SRs, and the other one considered SRs original and said SRs are published in their journal. Only the answer to the third question about the section in which SRs are published was the same as both answered SRs were published in no particular section of the journal.

Among the surveyed editors, 24 (33%) accepted to participate in the follow-up interview, 17 (23%) declined and 32 (44%) editors did not answer, even after a reminder.

Of the 24 editors who accepted to participate in the interview, 14 (58%) answered the first question that they considered SRs to be original studies, 6 (25%) answered that they did not, 3 (13%) answered that they considered SRs to be original if they had a MA incorporated and 1 (4%) accepted to participate in the interview without providing the answer to the first survey question.

Additionally, four editors who responded to the survey, but did not participate in the qualitative part of the study, offered their opinions regarding originality of SRs via email together with responses to the three survey questions. Two of them indicated that SR is original if it has methodological rigour; as one of them put it: ‘Cochrane-type review, with good scientific rigour combined with a thorough review of the literature, is original research.’ One of them remarked that a review can never be original research, with one exception: That exception is when a systematic exploration of the evidence (usually meta-analysis) is used to test a new hypothesis, one that had not previously been considered or addressed in the data analysis. Even then, I would not consider it completely original but an original application of the work of other investigators. The fourth elaborated that MA is considered new data and therefore only SR with MA would be considered original study.

Analysis of systematic reviews published in core clinical journals

Among the 118 included Core Clinical Journals, 110 published a median of 14 articles (range 1–528) indexed by PubMed as SR or MA, while 8 did not publish a single such manuscript.

Among the six editors that previously declared that they did not publish SRs, there were only two editors in whose journals our search did not find a SR or a MA published in 2017. Four stated that they did not publish but our search found SRs or MAs published in their journal in 2017.

The analysis of instructions for authors of eight journals that did not publish SR in analysed period showed that only one journal had guidance for authors regarding submission of SR or MA. Because one journal published 528 articles indexed as SR or MA by PubMed, we only analysed in detail the first 30% of these articles in order of publication (n=158). Altogether we analysed 2240 journal articles. Using those criteria, we found that 1187 (53%) were indeed a SR, scoping review, overview of systematic reviews (OSR) or a rapid review that used systematic searching methods and the others were not. For individual journals, the median per cent of articles that were SRs was 50% (IQR 33%–67%). We also found that two of the six journals for which editors responded that they did not publish SRs actually published at least one in 2017. A detailed analysis of 1187 published SRs is shown in box 1.

Fifteen (21%) editors participated in the follow-up interview. We stopped recruiting new editors after achieving thematic saturation. Interviews lasted between 6 and 25 min (median 12 min, IQR 9–16.5 min). The names of the interviewed editors were coded as ‘E1’-‘E15’.

Eight editors reported in the initial survey that SRs are original studies, three considered SR to be an original study only if it incorporated a MA and four did not consider SRs to be original studies.

Is any type of systematic review an original study?

Eight editors who considered SRs original studies were asked if they felt any different regarding the type of evidence synthesis, for example, SR with or without meta-analysis, scoping review, OSRs and any other type of SR or they thought any type of SR should be considered an original study.

Four editors indicated that all types of SR are original studies. One editor considered all SRs original if they passed peer-review scrutiny. One editor indicated that all SRs can be considered original studies but pointed out that he would still not publish a study which was not informative, such as an empty SR, that is, a review in which no eligible studies were found after literature search.11 One editor stated that a SR with a MA was still more novel than SR without MA. This indicates that certain editors equate originality of research with novelty and that certain types of studies, such as SRs, can have a whole spectrum of considerations:

E12: …from my point of view, the actual name or type of the systematic review is not that important. I think what I look is did they searched certain number of databases, did they do it in a systematic way, did they follow methods.

Three editors, who considered only SRs with MA as an original study, were asked if they thought meta-analysis was the only thing that makes a SR an original study. Two editors insisted that MA is essential for SRs to be considered original, while the third was more ambivalent:

E1: … I think that in the academic world of medicine we begin to recognise that not all science is discovery science, that there is also science of integrating results and so I think in that sense systematic reviews perhaps could be eventually consider original. But, in my research it is only a matter of in which section you put it, and we could as well put it in the section of original studies.

Four editors who did not consider SRs to be original studies had the same opinion about any type of SR and analytic methods used in such studies.

Two editors stated that there may be a continuum of considerations about originality of SRs. Editor E5 indicated that original SRs are those that bring new knowledge and, therefore, that there is no universal answer; that this should be judged individually for each SR:

E5: This should be judged on a case-by-case scenario. In my field, if you look at systematic reviews about influenza or vaccines, there have been multiple systematic reviews that did not bring any new knowledge. So, these studies should be judged individually.

Editor E15 was ambivalent about the response; this editor considered that SRs were original if they contain MA but acknowledged that SR authors may have planned to do a SR with MA, but whether MA is possible or not, this is often difficult to determine at the onset. MA may not be possible because of clinical or statistical heterogeneity:

E15: So it is not the fault of authors; they have done everything by the book. So I think there is some fluidity in this respect, and this division of original and non-original systematic review is artificial.

Elements of originality in systematic reviews

All editors were asked to define what makes a SR an original study or which elements are missing in SRs to be considered original. The answers are stated in table 1. Responses included specific elements of SR methodology, originality of idea and usefulness of SRs. Editor E9 opined that ‘original is bad descriptor for research’.

Table 1

Opinions of interviewed editors about elements of originality present or missing in systematic reviews

Editor	Is systematic review an original study?	Elements of originality present or missing in a systematic review; quotes
E2	Yes	New idea, analysis of bias, heterogeneity and level of evidence, provided summary and conclusion
E3	Yes	Done with high quality, using PRISMA guidelines
E4	Yes	Useful to improve or inform, either to advance knowledge or to improve and inform new research
E5	Yes	Meta-analysis helps in this respect, for a systematic review to be consider novel
E8	Yes	Different look at an old topic, something unique, probably in terms of search, novelty of the question, methods for searching
E11	Yes	Methodology involved in approach to search, careful process of filtering studies, looking at limitations of included studies, approaching a topic that requires some in-depth consideration and involving a thought process in summarizing data, reporting results, discussing them and providing conclusions approach to search strategy, analysis of results, discussion, limitations and making conclusions based on analyses
E12	Yes	Following the methodology, searching significant number of databases, they have to explain how they selected study they are going to review, what were the criteria, they have to talk about quality of the evidence, they have to summarize the results, it has to be a significant body of work, an element of quantity or magnitude
E13	Yes	Original question that hasn’t been answered before, new search strategy, a new methodology, a reinterpretation of the results
E1	Only with meta-analysis	Original thought
E7	Only with meta-analysis	Some kind of analysis, it does not have to be meta-analysis; it can be another type of analysis
E15	Only with meta-analysis	I would consider a systematic review without meta-analysis a semi-quantitative review and therefore not original study
E6	No	Original research starts with a data source that is in most of nursing a human and systematic reviews have data source that is secondary
E9	No	If you define original research as focused on discovery, then systematic reviews are not original in that sense. It does not have to do anything with methodology, but type of research. Primary studies that offer integration of existing research and synthesis are original research.
E10	No	Scientific method is different than in what I consider to be original
E14	No	In my opinion, if it does not touch the original data, it is not original

Four editors who did not consider SRs original studies were asked if they considered any secondary study to be original. Two indicated that only primary studies could be considered original, one did not provide an answer to this question while the fourth editor was ambiguous:

E10: Systematic reviews are taking data and analyzing them in different ways, and asking different questions, and so I do not value them as highly as primary or plain secondary research. …secondary studies could be considered original as long as they are not systematic reviews.

Editors’ opinion about value of systematic reviews

The editors provided the following responses to describe the main value of SRs: (i) synthesis of data (nine editors); (ii) providing answers to the posed clinical question (nine editors); (iii) data analysis (two editors); (iv) impartial and free of bias (three editors); (v) underlining guidelines (one editor); (vi) pointing where evidence is insufficient (one editor); (vii) independent and transparent methods using standardised assessment (one editor).

E4: independent assessment based on transparent methods of the evidence on a topic, including not only thorough search and systematic search of the evidence and the synthesis but also a standardized assessment of its quality and overall value … it’s systematic methods and their assessment is independent from conflict of interest in a broad sense…

Definition of original research

A question that emerged during one of the first interviews was the issue of a definition of original research. An editor asked interviewer to provide our definition of originality and explained that the answer depends on the definition of originality:

E9: This whole issue of originality depends on how one defines original research. If you do not define what original research is, then the question about whether systematic reviews are original studies is not fair.

Since there is no universally agreed definition of what constitutes an original research, we discussed this remark within our team and decided to ask all subsequent editors how they would define an original study. Ten editors that were interviewed subsequently provided their definition of original studies, which we divided into six categories (table 2). The original research was defined as: (i) one that brings new knowledge, data, information and conclusions; (ii) study that provides primary data analysis; (iii) definition according to the Boyer’s model of scholarship; (iv) study that has not been published before; (v) one which is replicable and done in accordance with criteria for the study in terms of scholarly inquiry and (vi) study where authors generate hypothesis, proceed in formal manner, choose methodology, share results, discuss limitations, provide conclusions and where people deserve authorship (table 2).

Table 2

Definitions of original research provided by editors

Editor	Is systematic review an original study?	Quote about definition of original research
E6	No	I define original research as that which involved individuals and their data and secondary research as that which doesn’t.
E7	Only with meta-analysis	Original is what has not been published before.
E8	Yes	To me, to be a study it has to be done in accordance with criteria for the study in terms of scholarly inquiry, so that it can be replicated, people can use the same search terms, etc.
E9	No	…if we look at it as a scholarship of discovery, and Boyer’s model, I would not consider any type of systematic review to be original research.
E10	No	A study that is hypothesis driven, that generates a new knowledge and applies appropriate methods to get there.
E11	Yes	A study where authors generate hypothesis, proceed in formal manner, choose methodology, share results, discuss limitations, and provide conclusions. Also, where people deserve authorship for what they have done.
E12	Yes	…original study is the study that will generate new conclusion, new data, new information and that requires significant intellectual effort on the part of investigators.
E13	Yes	I guess original research would be analyzing results and generating outcomes, or conclusions which haven’t necessarily been done by other people before.
E14	No	As an editor, an original study is a study in which someone produces data.
E15	Only with meta-analysis	A study that has new knowledge generated. New knowledge can also be generated through synthetic process of meta-analysis, but not qualitative data synthesis of systematic review without meta-analysis.

Using systematic reviews for making clinical decisions

All editors were asked if clinical decisions should be based on SRs or primary studies. Eight editors responded that clinical decision should be based both on SRs and primary studies, six editors gave some advantage to SRs for making such decisions and one editor stated that decisions should be made on high quality evidence, regardless of the study type.

Additional relevant themes

At the end of the interview, editors were asked to freely express anything else they would like to add regarding originality of SRs. Among the most common comments were questions whether it matters at all how SRs are categorised in terms of originality as long as they are useful. Editors gave various comments about production and publication of SRs, proliferation of SRs and their quality (table 3).

Table 3

Additional comments of editors regarding systematic reviews

Editor	Is systematic review an original study?	Additional comment
E1	Only with meta-analysis	I feel like we are in an evolution, you know. Even though it has been ten years since your first study, I do not think that people necessarily understand what a systematic review is… We do require systematic review authors to do a research checklist, we do require protocol registration, but we are not rigid about it, we don’t actually check if they did. We do not actually check if they followed it, which would be ideal to do, but that would just take a lot of people to do that, and we do not have resources for that.
E2	Yes	Editor who looks seriously at their impact factor will love to publish meta-analysis and systematic reviews that are good.
E3	Yes	…there are too many systematic reviews being submitted, and that is because it’s easy, you don’t have to leave the comfort of your home or office, you can collect data and write your manuscript. So the quality is not very good because the motivation is wrong. The motivation is to become published, the motivation is not to influence care.
E4	Yes	Well, as an editor, I find it difficult to find a balance between quality and useful message from a submitted systematic review.
E6	No	… I publish systematic reviews, I am a fan of systematic reviews. And I think they do more to move knowledge then a lot of original research does, because it takes the whole body of original research in that area and it elevates it.
E10	No	I think systematic reviews are scholarship. Boyer in 1990 or 1999 defined four different types of scholarship; original research is one, and synthesis is another. And I think it is incredibly important in terms of scholarship and academic advancement.Systematic reviews are maybe routinized in the way they are done, or the way data is collected, but they are hard to do a good one.
E14	No	People do systematic reviews because they have to build the CV and they don’t have access to their own data, or they are not able to generate their own data. And we see this coming from various parts of the world, where we know that investigative resources are thin. And there is considerable confusion in the scientific and clinical communities about what a systematic review is and how much significance should be attached to it.

Some of the practical aspects of this categorisation included eligibility of SRs as publications qualifying candidates for academic advancement or acceptance of SRs as a PhD thesis:

E1: I guess that depends on what you call original, and does it matter if you call it original or not—I don’t know does it matter. I guess it could matter to a PhD thesis committee or a promotion committee, but maybe they need to stop and think about what they are doing, and why they are doing it, and who it is that they are trying to train and what it means. In our institution we definitely have scholarship of synthesis, so meta-analyses and systematic reviews would get strong weight on our promotion committee. We have already moved in that direction.

Six editors indicated that they would consider SRs to be an eligible study design for a PhD thesis, one said that SRs should not be eligible for a PhD thesis and one editor considered that a candidate should prepare at least two SRs or a SR and a primary study if this study design will be considered for a PhD thesis:

E4: Somebody can oppose systematic review within a PhD thesis because they don’t know that people learn a lot by doing systematic reviews. They become better researchers if they do them.

Editor E6 indicated that there should not be restrictions regarding study designs while conducting academic thesis, because it is important to focus on learning outcomes and that the only goal of a thesis should not be putting hands on patients because we are moving towards electronic medical records anyway:

E6: We are not going to be doing individual data collection in 5 years or so. So we should be paying less attention to how we were doing things when we were younger, because then the only way to collect data was putting hands on a patient, or getting data from their chart, getting information by asking patient, observing, video-taping, etc., etc. But if now the data has already been collected, the old-fashioned way is very expensive method to gain knowledge. But my main point is whether that is our key competency, to collect an information directly from a patient.

Discussion

We followed up on the study of Meerpohl et al 3 and found a similar proportion of editors of Core Clinical Journals who considered SRs original studies. Most editors published SRs, and a quarter of them published them in the section of their journal devoted to original studies. Interviews with editors indicated that various elements are considered when deciding whether a SR is an original study.

Eighty per cent of the surveyed editors considered SRs per se or only SRs with MA original research. This number is slightly higher compared with 71% of editors who considered that SRs are original research projects in 2009,3 but the difference is not statistically significant. There was no difference between the proportion of editors who considered SRs to be original research compared with the earlier study.3 These results indicate minimal changes of editors’ attitudes between these two studies in terms of the premise that SRs are original research.

We went beyond the initial study by Meerpohl et al 3 with a qualitative study because we wanted to gather richer data, to give the editors an opportunity to explain their attitudes, and to get more details about what they consider original or non-original regarding SRs.

In the qualitative part of our study, many editors indicated that the concept of originality of SRs is still evolving and that there is a continuum of considerations to be made. One editor, who considered that SR is an original study, stated that a SR with a MA is still more novel than SR without MA. This indicates that certain editors equate originality of research with novelty. This also indicates that for some editors, there are no firm categories about novelty and originality of SRs. Instead, these are judged as a whole spectrum, where different characteristics or certain items of methodology can influence perception of a study.

Another editor, with the same opinion that SR with MA is an original study acknowledged that MA may not be justifiable and that this can be only determined after SR authors have already done a lot of work. In this case, the definition of originality of the study would depend on data that were found and not on the initial idea. This would make decisions about originality of a SR highly unpredictable; someone could embark on doing an original study and end up with a non-original study, depending on the results.

Furthermore, responses of editors during interviews indicate that there is a lot of fluidity in defining original research. There was no consensus among editors about what constitutes original research and what makes SRs original. Some editors quoted methodology and reproducibility and some originality of idea and level of novelty. Some insisted on analyses, presence of meta-analysis or another type of analysis, which would bring quantitative aspect to a qualitative summary in a SR. For one editor, SR without MA is semiquantitative and thus not original. However, if we duly take quantitative data as a paramount defining element of original research, then the whole field of qualitative studies would come under question as non-original.

Furthermore, some editors did not consider SRs or SRs without MA original research projects, because authors of such studies did not produce the data but relied on data collected by others. However, as other editors have mentioned, there are now a plethora of study designs in clinical medicine that include data which were not collected first hand. Studies that rely on electronic health records, or data collected via instruments, or any type of retrospective studies relying on data that somebody else collected, would then not be considered original according to these criteria.

Two editors cited Ernest Boyer’s model of scholarship as an argument why they do not consider SR to be original research.12 According to this model, introduced in 1990, there are four categories of scholarship, where original research is one category and the integration that involves synthesis of information is another category.12 However, one of the interviewed editors remarked that indeed there are different types of scholarship and therefore different types of original research.

Our survey also indicated that the majority (91%) of editors published SRs. The difference to the previously reported proportion (94%)3 is not significant. A quarter of editors published SRs in the original study section of the journal, which is a decline in comparison to earlier results that a third of editors published them in an original study section in 2009.3 However, it also has to be emphasised that some editors remarked in interviews that they find it completely irrelevant in which section a SR is published, because they have thematic sections where they publish manuscript of any study design, including SRs. Therefore, the finding that fewer SRs are published in sections devoted to original studies may simply be an indication that more journals organise manuscript in thematic sections and not according to the perceived originality of a contribution.

Few manuscripts so far were devoted to the considerations of originality of SRs. Aveyard and Sharp13 postulated that SRs are ‘original empirical research’ because they (quote): ‘review, evaluate and synthesise all the available primary data, which can be either quantitative or qualitative’. One editor in our qualitative study remarked that it does not matter whether a SR is considered original research, as long as it is valuable. This was also pointed out before by Biondi-Zoccai et al, who considered that the main criteria to judge a SR should be its novelty and usefulness and not whether a SR is original/primary or secondary research.14

There may be practical aspects in considerations whether a SR is an original study or not, such as during evaluating candidates for academic promotion, or allowing students to use SR as a study design in their academic thesis. Although these topics were not the subject of our study, several editors indicated that they would be in favour of recognising SRs both as studies that can be counted for academic promotion and for conducting PhD thesis. There were also discrepant opinions regarding SRs within PhD theses as well as considerations that acceptability of a SR for PhD thesis may depend on a research field. It has been shown recently that the topic of acceptability of a SR for PhD theses is debated in the academic community as well; about half of surveyed individuals in charge of European PhD programmes indicated that SRs are accepted as a study design in their schools.4

In addition to analysing editors' opinions, we also assessed SRs that the target journals published in 2017. Overwhelming majority of the analysed journals (93%) published at least one SR in 2017, and among the remaining eight journals, one had instructions for authors regarding submission of SRs. Six editors indicated that their journal does not publish SRs, but our analysis indicated that two of those six journals actually did publish SRs in year 2017.

Strengths and limitations

One strength of this study is its high response rate (62%). Furthermore, the proportion of editors with different answers to the question about whether SRs are original studies was very similar among editors who participated in the survey and editors who participated in follow-up interviews, indicating the editors were equally keen to participate in the interview regarding their consideration of SRs as original studies or not. Therefore, we can argue that interviews did not suffer from selection bias in terms of including only editors with uniform opinions regarding SRs being or not being original studies. Another strength of this study is the addition of a qualitative data collection to the study design as compared with Meerpohl et al.3 Through the interviews, we obtained more nuanced responses regarding editors’ opinion about whether SRs are original studies and what is an original study anyway. These additional data provide rich insight into the reasoning of editors and may provide inspiration for further studies and actions in this field. Certain tangible benefits can depend on whether a SR is regarded as original research or not. Consequently, it would be beneficial if relevant organisations, such as learnt societies and associations of editors, would address the issue and provide some guidance.

Our study had several limitations. We focused on the editors of PubMed Core Clinical Journals, which is a limited sample to begin with. It is possible that different responses would have been obtained if we had surveyed a broader sample of editors. Our decision to use this cohort of journals was guided by the preceding study by Meerpohl et al,3 to serve as a historical control.

Although 9 years have passed between the two studies, it is possible that some involved editors did not change over time and that they were included both in the first and in the second study. In our study, we did not include any questions to verify this. The list of Core Clinical Journals was similar between the two analysed years; there were 13 journals that were replaced by another one compared with the first study. However, none of the contacted editors mentioned that they participated in the first study.

We used PubMed to search for published SRs and MAs. If those were not indexed correctly in PubMed, we could have missed some. For this reason, we hand-searched all journals for which we did not find any SRs or MAs.

Another relevant consideration is the definition of a SR. We did not give editors any a priori definition of what could be considered a SR, and not a single editor asked which definition of a SR we used in the study. Presently, there is no consensus definition of a SR, and which methodology, or characteristics a study should have in order to be considered a SR. For example, it has been suggested that a study should not be called a SR if the authors searched only one database or if there is only one author.15

In conclusion, compared with results obtained in 2009, we found a similar proportion of editors of core clinical journals who consider that SRs are original research and who had conditional acceptance of SRs’ originality. Interviews with editors revealed that there is no uniform approach to defining what makes a SR (or any study) a piece of original research and that these concepts of originality of research are evolving. Editorial organisations, which set standards of publishing, should address this issue.