The Content of Statistical Requirements for Authors in Biomedical Research Journals.

BACKGROUND: Robust statistical designing, sound statistical analysis, and standardized presentation are important to enhance the quality and transparency of biomedical research. This systematic review was conducted to summarize the statistical reporting requirements introduced by biomedical research journals with an impact factor of 10 or above so that researchers are able to give statistical issues' serious considerations not only at the stage of data analysis but also at the stage of methodological design.
METHODS: Detailed statistical instructions for authors were downloaded from the homepage of each of the included journals or obtained from the editors directly via email. Then, we described the types and numbers of statistical guidelines introduced by different press groups. Items of statistical reporting guideline as well as particular requirements were summarized in frequency, which were grouped into design, method of analysis, and presentation, respectively. Finally, updated statistical guidelines and particular requirements for improvement were summed up.
RESULTS: Totally, 21 of 23 press groups introduced at least one statistical guideline. More than half of press groups can update their statistical instruction for authors gradually relative to issues of new statistical reporting guidelines. In addition, 16 press groups, covering 44 journals, address particular statistical requirements. The most of the particular requirements focused on the performance of statistical analysis and transparency in statistical reporting, including "address issues relevant to research design, including participant flow diagram, eligibility criteria, and sample size estimation," and "statistical methods and the reasons."
CONCLUSIONS: Statistical requirements for authors are becoming increasingly perfected. Statistical requirements for authors remind researchers that they should make sufficient consideration not only in regards to statistical methods during the research design, but also standardized statistical reporting, which would be beneficial in providing stronger evidence and making a greater critical appraisal of evidence more accessible.

Introduction

Statistics is the study of the collection, analysis, interpretation, presentation, and organization of data.[1] Robust statistical designing, sound statistical analysis, and standardized presentation are important to enhance the quality and transparency of biomedical research. Therefore, most medical journals introduce their statistical reporting guideline so that researchers give serious considerations to statistical issues not only at the stage of data analysis but also at the stage of methodological design. In addition, statistical advice offered in the instructions for authors can help editors, peer reviewers, and readers evaluate the risk of bias and statistical validity.

The International Committee of Medical Journal Editors (ICMJEs) published a set of recommendations for the conduct, reporting, editing, and publication of scholarly work on manuscripts being prepared for submission to medical journals since 1979. However, the guideline for presenting and writing statistical issues was not included until 1988.[2] At the same time, Guidelines for Statistical Reporting in Articles for Medical Journals, Amplifications and Explanations was published.[2] Moreover, in 2013, the National Institutes of Health (NIH), as an organization of the nation's medical research agency, addressed Principles and Guidelines for Reporting Preclinical Research (NIHPG) to enhance the reproducibility of biomedical study.[34] In 2015, the guideline of Statistical Analyses and Methods in the Published Literature (SAMPL) was published. Adhering to basic principles for reporting statistical analyses addressed by ICMJE, SAMPL puts further emphasis on how to report basic statistical methods and results with more details.[5]

Moreover, statistical analyses and reporting are also closely related to the design and activities of the research itself. Besides specified guideline for statistical reporting, a series of guidelines were developed to enhance the quality and transparency of health research by EQUATOR network, mainly including the Consolidated Standards Of Reporting Trials (CONSORT), the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE), the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA), the Standards for Reporting of Diagnostic Accuracy (STARD), and the Reporting of In vivo Experiments for Animal Research (ARRIVE). These guidelines provide recommendations for reporting based on the design type of biomedical studies with intention to address all quality issues in medical publishing.

Furthermore, different biomedical journals have distinctive requirements for statistical reporting besides the above-authorized guidelines. However, more requirements result in more researchers not knowing the appropriate journal for submission. We aim to refine common issues among different requirements introduced by the biomedical journals by a systematic review so that the statistical reporting guidelines can be easier to understand and apply.

Methods

Included journals

Biomedical journals with impact factors (IFs) of 10 or above selected from the journal list issued by 2013 Journal Citation Reports® were included in the study. Only biomedical journals were included in the study. In addition, the review journals were excluded from the study due to few statistical requirements for authors in these journals. For those journals without clear identifier of statistical requirements, we decided whether it could be included after looking through its website or querying to its editor by email.

Searching statistical instruction for authors

To get the authority information of the detailed statistical instruction for authors, we linked into the homepage of each included journal and downloaded the instruction for authors. For some journals without clear statistical requirements in the instruction for authors, we contacted the editors by emails to make sure whether there were other requirements for authors about statistical reporting. To ensure that all of statistical instructions could be collected, the list of journals which used ICMJE, NIHPG, and CONSORT were checked by looking through the website of ICMJE (http://www.icmje.org/journals-following-the-icmje-recommendations/), NIHPG (http://www.nih.gov/about/endorsing-jounals.htm), and CONSORT (http://www.consort-statement.org/about-consort/consort-endosement/consort-endosers---journals/).

Data extraction

A data extract form was predesigned referring to Guidelines for Statistical Reporting in Articles for Medical Journals, Amplifications and Explanations published on Annals of Emergency Medicine, and SAMPL.[25] The last version of data abstract form was used by two reviewers (Tian-Yi Liu and Si-Yu Cai) independently after pilot trial. The extracted data included journal's IF, and whether the journal following the ICMJE,[6] NIHPG,[7] SAMPL,[5] CONSORT,[8] STROBE,[9] PRISMA,[10] STARD,[11] ARRIVE,[12] or any particular statistical requirement. Meanwhile, all details of statistical requirements presented in the instruction for authors were abstracted. Data management was performed using Microsoft Office Excel 2007 (Excel version 14, Microsoft Corp.).

Data analysis

First, we described the types and numbers of statistical guidelines introduced by different press groups. The improvement and update of statistical instructions for authors were summarized by years. After that, we also summarized the particular statistical requirements in frequency, and all of requirements were grouped into three categories, i.e., design, method of analysis, and presentation.[13]

Results

In the 2013 Journal Citation Reports®, there were 174 journals with IF of 10 and above. We excluded 71 journals, those were not biomedical journals (n = 103). Thirty-nine review journals were also excluded from this study and 64 medical research journals were included in this systematic review. In addition, Chinese Medical Journal was included considering as one of the most influential journals in China. In summary, 65 original journals, belonging to 23 press groups, were screened to abstract statistical requirements in instruction for authors [Supplementary Table 1]. Of 65 journals, 63 addressed statistical instructions. Only two biomedical journals did not introduce any statistical reporting guideline for authors [Supplementary Figure 1].

Supplementary Table 1

The statistical reporting guidelines and particular requirements for authors adopted by 23 press groups

Press groups	n	ICMJE	NIHPG	SAMPL	CONSORT	STROBE	PRISMA	STARD	ARRIVE	Particular requirements
Oxford University Press	9	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
American College Physicians	8	Yes	Yes	NA	Yes	Yes	Yes	Yes	Yes	Yes
American Society Clinical Investigation	8	Yes	Yes	NA	Yes	Yes	Yes	Yes	Yes	Yes
EMBO	8	Yes	Yes	NA	Yes	Yes	Yes	Yes	Yes	Yes
JAMA	8	Yes	Yes	NA	Yes	Yes	Yes	Yes	Yes	Yes
Science	8	Yes	Yes	NA	Yes	Yes	Yes	Yes	Yes	Yes
BMJ	7	Yes	Yes	NA	Yes	Yes	Yes	Yes	NA	Yes
Chinese Medical Association	7	Yes	NA	NA	Yes	Yes	Yes	Yes	Yes	Yes
Lancet	7	Yes	Yes	NA	Yes	Yes	Yes	Yes	NA	Yes
PLoS	7	NA	Yes	NA	Yes	Yes	Yes	Yes	Yes	Yes
Elsevier Inc.	5	Yes	Yes	NA	Yes	NA	NA	NA	Yes	Yes
American Thoracic Society	4	Yes	Yes	NA	Yes	NA	NA	NA	NA	Yes
American Society Clinical Oncology	4	Yes	Yes	NA	Yes	NA	NA	NA	NA	Yes
Massachusetts Medical Society-NEJM	4	Yes	Yes	NA	Yes	NA	NA	NA	NA	Yes
Wiley-Blackwell	4	Yes	Yes	NA	Yes	NA	NA	NA	NA	NA
Cell-press	3	NA	Yes	NA	NA	NA	NA	NA	Yrs	NA
Nature	3	NA	Yes	NA	Yes	NA	NA	NA	NA	Yes
Lippincott Williams and Wilkins	2	Yes	Yes	NA	NA	NA	NA	NA	NA	NA
American Association Cancer Research	2	Yes	Yes	NA	NA	NA	NA	NA	NA	NA
Landes Bioscience	2	Yes	Yes	NA	NA	NA	NA	NA	NA	NA
Cambridge University Press	1	NA	Yes	NA	NA	NA	NA	NA	NA	NA
Cold Spring Harbor Lab Press	0	NA	NA	NA	NA	NA	NA	NA	NA	NA
Clockss	0	NA	NA	NA	NA	NA	NA	NA	NA	NA

NA: Not Available; ICMJE: The International Committee of Medical Journal Editors recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly work in Medical Journals; NIHPG: Principles and Guidelines for Reporting Preclinical Research addressed by The National Institutes of Health; SAMPL: Statistical Analyses and Methods in the Published Literature; CONSORT: Consolidated Standards of Reporting Trials; STROBE: Strengthening the Reporting of Observational Studies in Epidemiology; PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-analyses; STARD: Standards for Reporting of Diagnostic Accuracy; ARRIVE: Animal Research: Reporting of In vivo Experiments.

Flow diagram of included journals.

Click here for additional data file.

Table 1 shows the common items of statistical requirements issued by different reporting guidelines for medical research. The most common issues included “outcomes and estimation (confidence interval [CI]),” “statistical methods explain,” “ancillary analyses methods (subgroup),” “research design explanation,” and “participants inclusion and exclusion criteria.” In addition, the emphasized items involved in specified design types were different in each reporting guideline. For example, item of “Blinding methods” was required by NIHPG, CONSORT (for randomized controlled trial [RCT]), and STARD (for diagnostic test).

Table 1

Common items of statistical requirements issued by different reporting guidelines for biomedical research

Common items	Total number	ICMJE	NIH	SAMPL	CONSORT	STROBE	PRISMA	STARD	ARRIVE
Outcomes and estimation (95% CI)	8	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Explanation of statistical methods	7	Yes	Yes	Yes	Yes	Yes	NA	Yes	Yes
Ancillary analyses methods (e.g., subgroup analysis)	6	Yes	NA	Yes	Yes	Yes	Yes	Yes	NA
Explanation of research design	5	Yes	NA	NA	Yes	Yes	NA	Yes	Yes
Inclusion and exclusion criteria of participants	5	NA	Yes	NA	Yes	Yes	Yes	Yes	NA
Estimation of sample size	4	NA	Yes	NA	Yes	Yes	NA	NA	Y
Definition of variables	4	NA	NA	Yes	Yes	Yes	Yes	NA	NA
Descriptive data (distribution, mean)	4	Yes	Yes	Yes	NA	Yes	NA	NA	NA
Participants recruitment	3	NA	NA	NA	Yes	NA	Yes	Yes	NA
Randomization methods	3	NA	Yes	NA	Yes	NA	NA	NA	Yes
Blinding methods	3	NA	Yes	NA	Yes	NA	NA	Yes	NA
Data sources and measurement	3	NA	NA	NA	NA	Yes	Yes	Yes	NA
Tables and figures	3	Yes	NA	Yes	NA	NA	NA	Yes	NA
Participant flow/numbers analyzed	3	NA	NA	NA	Yes	Y	NA	NA	Yes
Computer software	3	Yes	NA	Yes	NA	NA	Yes	NA	NA
Methods for test reproducibility	2	NA	Yes	NA	NA	NA	NA	Yes	NA
Baseline data	2	NA	NA	NA	Yes	NA	NA	NA	Yes
Demographic variables analysis	2	Yes	NA	Yes	NA	NA	NA	NA	NA
Terms and units standardization	2	Yes	NA	NA	NA	NA	NA	Yes	NA
Bias	2	NA	NA	NA	NA	Yes	Yes	NA	NA
Sampling	2	NA	NA	NA	NA	Yes	NA	Yes	NA
Alpha level setting	1	NA	NA	Yes	NA	NA	NA	NA	NA
Direction of hypothesis	1	NA	NA	Yes	NA	NA	NA	NA	NA
Adjustments for multiple comparisons	1	NA	NA	Yes	NA	NA	NA	NA	NA

NA: Not available; ICMJE: The International Committee of Medical Journal Editors recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly work in Medical Journals; NIH: Principles and Guidelines for Reporting Preclinical Research addressed by The National Institutes of Health; SAMPL: Statistical Analyses and Methods in the Published Literature; CONSORT: Consolidated Standards of Reporting Trials; STROBE: Strengthening the Reporting of Observational Studies in Epidemiology; PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-analyses; STARD: Standards for Reporting of Diagnostic Accuracy; ARRIVE: Animal Research: Reporting of In vivo Experiments; CI: Confidence interval.

The improvement and update of statistical instructions for authors are summarized in Figure 1. ICMJE statistical guideline published in 1988 listed 15 statements to provide basic reporting tips for preparing manuscripts, including “describe statistical methods, report exact P value and measurement error or uncertainty (CI), describe randomization and blinding,” and others. Considering the progress of clinical research, a series of reporting statements have been issued by EQUATOR network since 1996 which substantially expanded requirements referring to more commonly used methods, including subgroup analysis, adjusted analysis, sensitivity analysis, and dealing with missing data. At the same time, NIHPG, focusing on the performance of statistical analysis and transparency in reporting, was published to enhance the reproduction and internal validity of research. The latest published SAMPL included more detailed requirement according to the methods, such as association analysis, correlation analysis, survival analysis, and Bayesian analysis.

Figure 1

Improvements and supplements of statistical requirements addressed by different published guidelines.

The particular requirements addressed by 44 journals were summed up as frequency of issues relevant to design [Figure 2], method of analysis [Figure 3], and presentation [Figure 4]. When the particular requirement was addressed by a greater number of journals, it was deemed as more important. Figure 2 shows issues relevant to research design, including participant flow diagram, eligibility criteria, and sample size estimation. Figure 3 highlights that novel methods should be explained as possible as in detail and standard works reference when explaining the method of analysis. On the other hand, advanced analysis methods, including multivariable analysis, Bayesian analysis, longitudinal analysis, and Cox model, were emphasized by more journals compared with ICMJE guidelines. As shown in Figure 4, reporting actual outcomes and exact P value was emphasized by many journals.

Figure 2

Frequency of particular statistical issues relevant to research design.

Figure 3

Frequency of particular statistical issues relevant to method of analysis.

Figure 4

Frequency of particular statistical issues relevant to presentation.

Discussion

It was shown that complying with reporting guidelines strictly can improve the quality and reliability of the research effectively. Moher et al.[14] compared the quality score for reports of RCTs published in 1994 (pre-CONSORT) and in 1998 (post-CONSORT), which showed that the use of the CONSORT statement is associated with improvements in the quality of reports of RCTs. In 2014, Choi et al.[15] conducted a systematic review to assess the endorsement of reporting guidelines in traditional medicine journals in Korea. They suggested that it is necessary for traditional medicine journals to adopt reporting guidelines because better reporting is likely to influence the quality as well as the effect of future research.

A majority of biomedical journals introduce statistical reporting guideline issued by NIHPG for authors, which focuses on the interpretation and repetition of biomedical experiments. Compared with ICMJE and NIHPG, those focus on not only statistical reporting but also research design, SAMPL guideline issues statistical requirements in details referring to different statistical analysis methods to help authors understand the factors of better statistical reporting.

On the other hand, a series of reporting guidelines issued by EQUATOR network are based on different types of research. Therefore, specified statistical requirements in each research design are different.[16] For example, CONSORT statement focuses on sufficient randomization, estimation of sample size, dataset used in statistical analysis (i.e., intention-to-treat or per-protocol), and subgroup analysis. STROBE statement mentions statistical method used to control confounding factors, explain missing data, and solve matching strategy, and methods used to examine subgroups and interactions. STARD statement for diagnostic test emphasizes reporting methods for estimating or comparing diagnostic accuracy, dealing with indeterminate index test or reference standard results, any analysis of variability in diagnostic accuracy, and distinguishing the prespecified from exploratory.

The contents of guidelines have been revised again and again with adding explanation and elaborations to refine the usability as well as to amplify the range of application.[213] This study demonstrates that besides 7 statistical reporting guidelines issued by NIHPG, ICMJE, and EQUATOR, 68% of journals with IF of 10 or above still introduce some particular statistical requirements for authors. The most common statistical requirements remind us that authors should consider the statistical requirements at the beginning of research design rather than only at the stage of statistical analysis. As Ronald said, “to consult the statistician after an experiment is finished is often merely to ask him to conduct a postmortem examination. He can perhaps say what the experiment died of.”[17] It is worthy for authors to note that statistical requirements are required more with the improvements of research methods, especially the rapid development of clinical research.

Schriger et al.[18] analyzed the methodological and statistical general contents addressed in instructions for authors offered by better quality medical journals and analyzed the length of journals’ instructions for authors. Compared with the year 2006, there is prominent improvement referring to statistical requirements for authors at present. First, more guidelines have been published by different organizations and also are adopted by more biomedical journals up to now. In 2006, only 22 of 166 journals mentioned CONSORT, 4 of 166 addressed other guidelines such as Quality of Reporting of Meta-analyses (QUOROM) or STARD. Moreover, few journals provided requirements about statistical methods and reporting.[18] Nowadays, the types of reporting guidelines introduced by journals are increasing obviously. Second, some statements have been updated and refined timely. For example, the QUOROM statement was updated into PRISMA, which contains more details about reporting systematic review and meta-analysis. Third, a higher number of journals pay more attention to methodology issues and provide particular advice correspondingly to their instruction for authors. Compared with Schriger et al.'s study, we focused on specific content referring to methodological and statistical reporting requirements, which could be of benefit to authors to achieve completeness of statistical reporting.

Furthermore, the improvement and update of statistical requirements indicate the rapid growth of research method and statistical analysis. In recent decades, advanced statistical requirements are applied, especially in cohort study with large-size samples or observation studies to identify the relative risk, and to describe characteristics of population; for example, dealing with missing data and outliers, estimation of sample size, Bayesian analysis, multivariable analysis, Cox models, and longitudinal analyses. On the other hand, there also are timely supplements in terms of particular requirements.

As for Chinese Medical Journal, one of the most influential journals issued by the Chinese Medical Association covers a broad range of fields of biomedical research in China. It also recommends 6 kinds of statistical guidelines and mentions particular requirements. However, statistics errors in biomedical journals remained commonly despite that many editors and authors made a substantial effort.[19] Therefore, explanation and elaboration of statistical reporting guidelines, aiming to improve the compliance of reporting guidelines and to enhance the quality of researches sequentially, is imperative.[19202122]

However, the study still has some limitations. Considering the academic authority, we focused on the biomedical journals with IF of 10 or above. However, it was likely that some journals with higher impact on specific areas were missed. On the other hand, journals on public health, especially on epidemiology and biostatistics, those focusing on the breakthrough of methodology, were also missed. In fact, many press groups publish several journals, including journals on epidemiology and biostatistics with <10 of IF. Journals issued by one press group usually use the same reporting guideline for medical research.

In summary, statistical requirements for authors are becoming increasingly perfected. All these improvements could be contributed to the push of evidence-based medicine, which focuses on the critical appraisal quality of existing researches and the standardization of reporting. Furthermore, recent statistical requirements for authors remind that researchers should pay more attention to the research design and statistical reporting that would be benefit to provide stronger evidence for the development of biomedicine.

Supplementary information is linked to the online version of the paper on the Chinese Medical Journal website.

Financial support and sponsorship

This study was supported by the grant from the Beijing Natural Science Foundation (No. 7162061). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Conflicts of interest

There are no conflicts of interest.