Background styles in systematic review articles are not related to the publication in high-impact-factor journals: A meta-epidemiological study.

OBJECTIVES: To clarify the styles used in background sections of systematic reviews (SR) and to identify which styles if any were related to the publication in high-impact-factor (HIF) medical journals.
METHOD: This was a cross-sectional study for original SR articles published in top 50 journals in MEDICINE, GENERAL & INTERNAL category in Journal Citation Reports 2018. We randomly included 90 articles from top 10 HIF journals and 90 from others, respectively. We conducted a content analysis to classify the background styles. We assessed the factors associated with the publication in HIF journals.
RESULTS: We found 6 categories. We defined 6 categories as follows: Update of prior SR, New in scope than prior SR, Higher quality than prior SR, Completely new SR, Limitations of primary studies only, and Not presenting unknown in prior SR or primary studies. All 6 categories were not related to the publication in HIF journals.
CONCLUSIONS: We found 6 categories of styles in background sections of SR, none of which however were related to publication in HIF journals. SR authors may wish to use any of these categories to communicate the importance of their research questions.

Introduction

The background section of a medical journal article has the important function to communicate the value of the research question to the readers. There are many textbooks and review articles reflecting expert opinions on how to write them.[ Almost all writers of academic articles find it difficult to write the background section.[

In addition, there are several analyses that examined its structure in medical research articles. The basic structure of the background section may be characterized as follows: “Presenting Background Information”, “Reviewing Related Research”, and “Presenting New Research”.[

However, little is known about how systematic review (SR) authors review related researches to emphasize the importance of their research question in their background section. This study aimed to examine the methods the authors use in order to delineate their novelty with regard to the existing literature and compare which methods were related to the publication in top medical journals. We have focused on SR because SR are the most important research design, in terms of practicing evidence-based medicine.[

Methods

Study design

This was a cross-sectional study for original SR articles. We published the study protocol before conducting this research.[ We adhered to the reporting guideline of meta-epidemiological study (Guidelines Checklist)[ where relevant.

Types of studies included

We included original SR articles published in top 50 journals in MEDICINE, GENERAL & INTERNAL category in Journal Citation Reports 2018.[ We included all SR articles irrespective of study designs of the included primary studies. We used the definition of SR by National Library of Medicine, which was “a scientific investigation that focuses on a specific question and uses explicit, prespecified scientific methods to identify, select, assess, and summarize the findings of similar but separate studies”.[

We excluded Cochrane Reviews, the U.S. Preventive Services Task Force review, or The Rational Clinical Examination articles because their backgrounds styles are prespecified by the respective organizations and different from usual original articles.[ We also excluded protocols or non-English articles.

Search methods

We searched PubMed on December 1, 2019. The search formula contained the following blocks: “date” AND “study type filter” AND “the name of journals”. The details of search formula are shown in Supplemental Digital Content (Table S1A, B).

Study selection

We prespecified in our protocol to randomly select 90 articles from the top 10 high-impact-factor (HIF) journals and 90 articles from the 11th to 50th journal as controls. In accordance with previous studies,[ we chose a cut-off of top 10 in our protocol. In our protocol we considered 9 potential confounding factors. Using the rule of thumb that we need cases that are 10 times the number of confounding factors to examine their influence,[ we chose 90 articles from the top 10 journals randomly and 90 controls from the top 12 to 50 journals randomly with a 1:1 match. One review author (YK) confirmed whether the articles were SR or not. We attempted to repeat the random sampling of SR to reach the prespecified number.

Data extraction and assessment

Exposures

Since there were no prior categories to refer to, we conducted a content analysis to classify the background style to show the novelty in SR. Content analysis is a method to analyze texts. This method enables researchers to evaluate texts and make meaningful and replicable interpretations.[ In generating categories, we used the lens of a position to guide the writing of systematic review articles for novices.

In the protocol we defined 4 categories. Using these categories, 5 authors (YK, ST, SYK, YT, or HY) who have prior experience of conducting systematic reviews independently assessed randomly selected 10 articles, then discussed the disagreements. Finally, we defined 6 categories as follows: Update of prior SR, New in scope than prior SR, Higher quality than prior SR, Completely new SR, Limitations of primary studies only, and not presenting unknown in prior SR or primary studies. The details of definitions are shown in Table 1. We evaluated the categories with an allowance for duplications.

Table 1

The definitions of background categories.

Category	Definition	Supporting codes
Update of prior SR∗	Mentioning prior SR of same research question, but the search date was new.	We aimed to update the information provided by our previous report (Siew)
New in scope than prior SR∗	There is a reference to prior SR of a similar research question and explains how this review is different from prior SR in scope (e.g. patient, intervention or outcomes reviewed)If there is no mention of any prior SR, this category is not applicable.	The most recent metaanalysis suggested (omitted). Previous reviews have also mainly reported results for (omitted) outcomes rather than (omitted) and omitted the growing body of evidence from observational studies on this topic, which often include large study populations in real world settings and longer follow-up. (Mathews)One quantitative review examined the effectiveness of psychological approaches for (omitted). We conducted a comprehensive systematic review and meta-analysis to evaluate the efficacy of CBT-based approaches for (Niknejad)
Higher quality than prior SR∗	There is a reference to prior SR of a similar research question but points out problems with their methods other than being outdated, or stating the need for new research methods, such as NMA.	This method allows comparison of all available (omitted) medications against placebo and between pharmacological agents despite the paucity of head-to-head comparisons of therapies in RCTs. (Gregori)
Completely new SR	Mentioning that it is a completely new SR.	To date, to our knowledge, no studies have comprehensively examined (Fellmeth)No clinical practice guidelines include (omitted). This systematic review focuses on this gap (Williums)
Limitations of primary studies only	No mention of prior SR, and only mention of limitations of primary studies.	Previous studies have largely been single centre trials with a small sample size, (omitted) not powered to assess the true effect of (omitted) and whether it interacts with important clinical subgroups. (Nath)
Not presenting unknown in prior SR or primary studies	No mention of prior SR, no mention of the limitations of primary studies, no reference to primary studies, or expresses uncertain points without any citations.	Consequently, there is interest among researchers, clinicians, and health policy leaders in (omitted) (Rosenstein)While most industrialized countries have introduced routine female (omitted) vaccination into their national immunization programs, routine vaccination of boys and men is currently implemented in only a few countries (omitted) (Harder)

After developing the 6 categories, 2 of the 5 authors (YK, ST, SYK, YT, or HY) independently assessed the background section of each of 170 remaining articles. We resolved the disagreements through discussion.

Other variables

One review author (YK) extracted variables related to the background. The background section included subheadings of background and aim or purpose. Word counts included citation symbols. The number of included studies was counted at the time of their synthesis of each article. One review author (YK) evaluated the study characteristics, and 1 of 4 authors confirmed (ST, SYK, YT, or HY). We resolved the disagreements through discussion.

Outcomes

The primary outcome was whether the article was published in HIF journals or not.

Data analysis

We conducted data analysis following a prespecified protocol.[ We used descriptive statistics to summarize. We used logistic regression models for univariate and multivariate analysis to explore the associations between study characteristics and the publication in HIF journals. Confounding factors were number of included articles in SR, number of tables and figures, number of authors, and presence of appendix. We used median as cut-off point to dichotomize the confounding variables. Two-tailed P values were considered statistically significant if less than .05. We used Stata ver. 16.1 (StataCorp LLC, College Station, Texas, United States of America).

Ethics

Ethics approval was not applicable because this study only used published articles.

Difference between the protocol and manuscript

We initially planned to investigate the presence of practice implications. It was difficult to reach agreement through discussion in the pilot review. For that reason, we dropped this feature investigation.

Patient and public involvement

We did not involve patients or the public in our study.

Results

Selection process

The search result and selection process are shown in Figure 1. We searched a total of 569 articles. We randomly included 112 articles from HIF journals and 106 from top 12 to 50 journals, respectively. After full-text screening, we included 90 articles from the former and 90 articles from the latter, respectively (Table S3).

Figure 1

Flowchart for the selection of reporting guidelines. ∗Top 11 journal was the Cochrane Database of Systematic Reviews (CDSR), we excluded CDSR because the backgrounds style is different from usual original articles. We randomly selected 112 and 106 potentially eligible articles from the search results. When the article was not eligible, we repeated to resample to reach 90 articles in each category. SR = systematic review, USPSTF = United States Preventive Services Task Force.

Study characteristics

Table 2 shows the study characteristics of included articles. The median number of paragraphs, the median number of cited articles and the word counts in background were fewer in HIF journals than in others.

Table 2

Characteristics of included articles.

	Top 10 (n = 90)	Top 12–50 (n = 90)	Total (n = 180)
Number of paragraphs in background	3 [3–4]	4 [3–5]	3 [3–5]
Number of cited articles in background	14 [10–19]	18 [12–23]	15 [11–21]
Word counts in background	388 [279–493]	465 [344–585]	433 [299–539]
Number of included articles in SR	36 [20–93]	18 [11–34]	24 [13–50]
Number of authors	8[6–11]	5 [4–7]	7 [5–9.5]
Number of tables	2 [1–3]	3 [2–4]	2 [1–3]
Number of figures	3 [2–5]	2 [1–3]	3 [1–4]
Number of citations	59 [44–84]	56 [42–80]	57 [43–82]
Presence of appendix	87 (97)	61 (68)	148 (82)
Study type∗
NMA (intervention)	9 (10)	3 (3)	12 (7)
intervention	40 (44)	37 (41)	77 (43)
DTA	2 (2.2)	3 (3)	5 (3)
prognosis	19 (21)	11 (12)	30 (17)
prevalence/incidence	11 (12)	8 (9)	19 (11)
Scoping review, qualitative synthesis	6 (7)	26 (29)	32 (18)
meta-epidemiology	3 (3)	2 (2)	5 (3)

We evaluated the inter-rater reliability by kappa statistics. Table 3 shows the proportion of background categories and inter-rater reliability. We did not add any categories for the 170 articles other than the 10 we initially reviewed.

Table 3

Categories of background style.

	Top 10 (n = 90)	Top 12–50 (n = 90)	Total (n = 180)	Kappa statistics∗
Update of prior SR	20 (22)	11 (12)	31 (17)	0.76
New in scope than prior SR	33 (37)	35 (39)	68 (38)	0.71
Higher quality than prior SR	18 (20)	13 (14)	31 (17)	0.68
Completely new SR	7 (8)	17 (19)	24 (13)	0.71
Limitations of primary studies only	20 (22)	15 (17)	35 (19)	0.52
Not presenting unknown in prior SR or primary studies	11 (12)	8 (9)	19 (11)	0.51

Primary outcome

Table 4 shows the association of factors and the publication in HIF journals. Unadjusted analysis showed that completely new SR were associated with the publication in HIF journals, others were not. Adjusted analysis showed that all 6 categories were not associated with the publication in HIF journals.

Table 4

Associations between study characteristics and the publication in top 10 high-impact-factor journals.

	OR (95%CI)	AOR (95%CI)
Number of included articles in SR (>24 vs ≤24)	3.45 (1.87 to 6.37)	2.74 (1.33 to 5.65)
Number of tables and figures (>5 vs ≤5)	1.26 (0.69 to 2.27)	1.31 (0.63 to 2.7)
Number of authors (>7 vs ≤7)	5.16 (2.7 to 9.86)	3.93 (1.86 to 8.29)
Presence of appendix	13.79 (4.02 to 47.31)	8.76 (2.33 to 32.91)
Update of prior SR	2.05 (0.92 to 4.58)	2.21 (0.5 to 9.69)
New in scope than prior SR	0.91 (0.5 to 1.66)	1.06 (0.24 to 4.61)
Higher quality than prior SR	1.48 (0.68 to 3.24)	1.08 (0.37 to 3.17)
Completely new SR	0.36 (0.14 to 0.92)	0.56 (0.09 to 3.4)
Limitations of primary studies only	1.43 (0.68 to 3.01)	2.27 (0.41 to 12.41)
Not presenting unknown in prior SR or primary studies	1.43 (0.55 to 3.73)	1.99 (0.31 to 12.82)

Discussion

Brief summary of main findings

We have identified 6 categories to review related researches in the background sections of systematic review articles published in general medical journals. None of the 6 categories was associated with publication in HIF journals, while the number of included articles, the number of authors, and presence of appendix were.

Results in relation to prior studies

This is the first study to reveal the “background styles” in SR. Nwogu clarified that in background section of medical articles authors review related research for 2 purposes, which were “Reference to previous research” and “Reference to limitations of previous research”.[ Five categories found in our research excluding “not presenting unknown in prior SR or primary studies” could be regarded as “Reference to limitations of previous research”. The existence of “not presenting unknown in prior SR or primary studies” may indicate that the research question that the review focused on was obvious for authors and editors to explain.

The amount of information included in SR would be related to the publication in HIF journals. Manriquez et al reported that number of authors and number of references cited were related to citations of skin disease SR.[ In our study, number of included articles, number of authors, and presence of appendix were related to the publication in HIF journals. These findings are consistent with a previous SR focused on the relationship between the feature of all biomedical articles and citations.[ Our results indicate that the “background style” is not an important determinant in submission by authors and/or acceptance by journal editors.

Clinical relevance and implications

Our findings may be useful for novice SR researchers when write their articles. As structured reporting with reporting guidelines helps to improve the quality of reporting,[ the effectiveness of teaching academic writing following with known structure in each professional academic community has been suggested.[ Further study to develop and evaluate an education program for writing background section in SR using the styles identified in this study would be warranted.

Limitations

There are several limitations in our study. First, there was some inconsistency in judgment in the classification of background styles. As far as possible, we classified the background based on the operational definition, but the kappa statistics of “limitations of primary studies only”, and “not presenting unknown in prior SR or primary studies” were only moderate. This may be partly due to the oversight of the researchers, but also due to the existence of some papers which fails to explain novelty for readers. Second, we used journal impact factor (JIF) as a proxy of the importance of individual articles. Usually, the distribution of citations are skewed and we should be careful about using JIF, which is a mean.[ However, we adopted the lens of a position to guide the writing of systematic review articles for novices. We believe that the JIF, which is determined by each journal, are the reflection of the editors preferences for deciding acceptance. Third, we only focused on general medical journals and a certain period. Our findings would not be applicable for journals that are only read by experts who have enough knowledge of the field. Future research is warranted to verify whether our findings would be applicable in other fields and periods.

Conclusions

We found 6 categories in background section of SR. SR authors would be able to use these categories to explain the importance of their research questions, but not necessary to care to choose the submission journal.

Acknowledgments

We would like to thank the members of the Research Group on Meta-epidemiology at The Kyoto University School of Public Health (Tomoko Fujii, Yusuke Tsutsumi, Aran Tajika, Kenji Omae, Edoardo G Ostinelli, and Yan Luo)

Author contributions

YK, ST, SYK, YT, HY and TAF contributed to the conception and design of the research. YK, ST, SYK, YT, HY retrieved the data. YK conducted statistical analyses. YK wrote the draft manuscript. TAF and all the other co-authors revised the manuscript critically for important intellectual content. All authors gave final approval of the manuscript before submission.

Conceptualization: Yuki Kataoka, Shunsuke Taito, Sachiko Yamamoto-Kataoka, Yasushi Tsujimoto, Hajime Yamazaki, Toshi A. Furukawa.

Data curation: Yuki Kataoka, Shunsuke Taito, Sachiko Yamamoto-Kataoka, Yasushi Tsujimoto, Hajime Yamazaki.

Formal analysis: Yuki Kataoka.

Investigation: Yuki Kataoka.

Methodology: Yuki Kataoka, Toshi A. Furukawa.

Project administration: Yuki Kataoka.

Resources: Yuki Kataoka.

Software: Yuki Kataoka.

Supervision: Toshi A. Furukawa.

Validation: Yuki Kataoka.

Visualization: Yuki Kataoka.

Writing – original draft: Yuki Kataoka.

Writing – review & editing: Shunsuke Taito, Sachiko Yamamoto-Kataoka, Yasushi Tsujimoto, Hajime Yamazaki, Toshi A. Furukawa.