Research Designs and Statistical Methods Trends in the Annals of Rehabilitation Medicine.

OBJECTIVE: To investigate trends of the research designs and statistical methods in the Annals of Rehabilitation Medicine (ARM) published from 2005 to 2015 through a comparison of articles with the Archives of Physical Medicine and Rehabilitation (APMR).
METHODS: The authors reviewed all articles published in ARM and APMR for the years 2005 and 2015 in order to determine their research designs as well as their statistical methods used in each article.
RESULTS: In ARM, randomized controlled trials increased from 4.5% in 2005 to 6.5% in 2015. In APMR, randomized controlled trials increased from 8.1% in 2005 to 14.0% in 2015, meta-analyses increased to 5.3%, and systematic reviews increased to 6%. The number of studies using statistical methods increased in ARM from 1.9 to 2.6 per article and in APMR, from 2.7 to 3.1. Use of advanced methods in ARM also showed an increase from 2005 to 2015.
CONCLUSION: This study concludes that there is a trend of increased awareness and attempts to use varied research approaches in ARM articles. There should also be more in-depth discussions and opportunities for researchers to share their experiences regarding statistical methods in the clinical field.

INTRODUCTION

Recently, there have been many approaches to research in the field of medicine from pathophysiological theories, individual clinical experiences, and expert opinions to evidence-based medicine. Evidence-based medicine, which employs fundamental objectives and methods, offers objective reasoning and scientific explanations for patient care and treatment. Due to this fact, it has a significant impact on current medical research trends. A number of articles and journals associated with evidence- based medicine are currently available and seem to play a major role in the medical education of many countries [12345678]. Due to this trend, statistics are used as an essential element for research designs, data collections, analyses, and interpretations in all stages and have become increasingly important in research objectives consisting of a hypotheses and their testing. To achieve a research goal, a reasonable research design is essential—along with an adequate statistical analysis—to draw logical and objective conclusions.

Annals of Rehabilitation Medicine (ARM) is an official journal of the Korean Academy of Rehabilitation Medicine and was first published in 1977. Initially, it was published under the name Journal of the Korean Academy of Rehabilitation Medicine, and in 2011, it was changed to Annals of Rehabilitation Medicine to broaden its scope to include the Asia-Pacific region. In 1998, Kim and Kim [9] published the article “Statistical Methods Used in the Journal of Korean Academy of Rehabilitation” to address the subject of statistical methods used in research for the first time but since then, there has been no additional publication on this subject. There are previous studies on the reviews and trends in research designs and statistical methods used in studies conducted both in and out of the country, and it has become necessary to review the status of these trends [1011121314151617]. Thus, the aim of this article is to investigate these trends in research designs and statistical methods through a comparison of published articles over the past 10 years, from 2005 to 2015, in ARM and the Archive of Physical Medicine and Rehabilitation (APMR), which is the most representative journal in the field of rehabilitation medicine.

MATERIALS AND METHODS

The subjects of this study are ARM and APMR articles published in 2005 and 2015. Among the articles, letters to the editor, replies, editorials, articles related to medical education, lectures, commentaries, special communications, and corrigenda were excluded. For four months, from January to April 2016, two doctors of rehabilitation medicine and one doctor of statistical philosophy, currently working for the clinical statistical section of a research institute, reviewed both journals and determined the research designs and statistical methods used in all articles. Three researchers compared the results after the review were complete and when the results differed, a final decision about the classification was made through a group discussion.

Research designs were divided into observational studies, experimental studies, and literature reviews. Observational studies were classified into cohort studies, case-control studies, cross-sectional studies, and other observational studies generally used in clinical research, while experimental studies were classified into randomized controlled trials and other clinical trials. Literature reviews were classified into meta-analyses, systematic reviews, and narrative reviews. Case studies/series and animal studies were classified separately. For statistical methods, every method used in each article was counted individually and repetitive usage of the same method in the same article was counted as one.

After all classification categories were determined, the research designs were recorded by year and journal name, while the number of statistical methods per article was classified as 0, 1, 2, 3, 4, 5, or more. Then, the statistical method was classified into either a basic or an advanced method (Table 1).

Table 1

Statistical method categories

Most lead authors in ARM were medical doctors, but lead authors of APMR consist of people working in various fields. There may be differences in research designs according to occupations and to examine them, APMR lead authors were classified into medical doctor and non-medical doctor groups.

Statistical analyses were performed using SAS ver. 9.3 (SAS Institute, Cary, NC, USA). The mean±standard deviation of the classifications for year and journal name was calculated to determine the average difference in the frequency of each statistical method per article. Regarding research designs classified by year and journal name, a chi-square test determined statistical differences. A p-value<0.05 was considered statistically significant.

RESULTS

Research designs

In 2005, research designs for ARM in the order of from the highest to the lowest were cross-sectional studies, case studies/series, and clinical trials. In 2015, the descending order was case studies/series, cross-sectional studies, and clinical trials (Table 2). Randomized controlled trials increased from 4.5% in 2005 to 6.5% in 2015, but this difference was not significant, and there were no meta-analyses or systematic reviews in 2005 or 2015 (Fig. 1A).

Table 2

Research design

Values are presented as number (%).

ARM, Annals of Rehabilitation Medicine; APMR, Archives of Physical Medicine and Rehabilitation.

Fig. 1

Research design differences by year between Annals of Rehabilitation Medicine (ARM, A) and Archives of Physical Medicine and Rehabilitation (APRM, B). *p<0.05.

In 2005, APMR published cross-sectional studies, clinical trials, and cohort studies, and in 2015 cross-sectional studies, randomized controlled trials, and clinical trials, all in descending order (Table 2). In 2005 and 2015, randomized controlled trials increased from 8.1% to 14.0%, meta-analyses from 0% to 5.3%, and systematic reviews from 0.5% to 6%. However, case studies/series decreased from 10.8% to 0.8% during this time frame (Fig. 1B).

Comparing ARM and APMR, there was a significant difference in the number of cohort and animal studies in 2005 and significant differences in the number of meta-analyses, systematic reviews, and randomized controlled trials in 2015 (Fig. 2).

Fig. 2

Research design differences by journal between 2005 (A) and 2015 (B). ARM, Annals of Rehabilitation Medicine; APMR, Archives of Physical Medicine and Rehabilitation. *p<0.05.

Statistical methods

We found that 20.9% of 2005 ARM studies did not use statistical methods and this percentage rose to 26.6% in 2015. Regarding APMR, in 2005, 12.7% of studies did not use statistical methods and this number decreased to 1.9% in 2015. In 2005 and 2015, the number of times statistical methods in ARM used per article increased from 1.9 to 2.6 and in APMR, from 2.7 to 3.1. For 2005 and 2015, the number of different statistical method types in ARM increased from 14 to 25, and increased from 32 to 38 in APMR (Table 3). According to the classifications displayed in Table 1, advanced methods increased in ARM from 2005 to 2015 (Table 4).

Table 3

Number of statistical methods per article

Values are presented as mean±standard deviation or number (%).

ARM, Annals of Rehabilitation Medicine; APMR, Archives of Physical Medicine and Rehabilitation.

Table 4

Differences in statistical method types

Values are presented as number (%).

ARM, Annals of Rehabilitation Medicine; APMR, Archives of Physical Medicine and Rehabilitation.

In APMR, for the medical doctor group, the number of clinical trials and case studies was higher, and lower for randomized controlled trials as well as meta-analyses compared to the non-medical doctor group (Table 5).

Table 5

Differences (%) between medical and non-medical doctors' research designs in APMR

APMR, Archives of Physical Medicine and Rehabilitation.

DISCUSSION

With computer and Internet technology developments, individuals in contemporary society are able to access an unlimited amount of information in any field and this phenomenon is also manifest in the medical field. Clinicians have the responsibility to select and apply scientific, treatment-assisting studies and significant quality information can be obtained from scholarly journals. A step required in the process of selection is known as testing and along with developments in the field of evidence-based medicine, there is a number of testing methods used to grade and classify studies [1819]. These methods exhibit subtle differences and there are diverse evaluation techniques, but when strictly appraising research designs, the most effective methods are meta-analyses, systematic reviews, and randomized controlled trials. According to the results of this research, randomized controlled trials in ARM increased from 4.5% in 2005 to 6.5% in 2015, but the difference was not statistically significant. In APMR, randomized control trials increased from 8.1% to 14.0%. For both journals, meta-analyses did not exist in 2005 but for APMR in 2015, 5.3% of all research designs were meta-analyses and 6.0% were systematic reviews. When ARM articles were analyzed from the perspective of evidence-based medicine, there are differences compared to APMR. Another conspicuous distinction is that case studies/series in ARM increased from 17.3% to 25.2%, while they decreased from 10.8% to 0.8% in APMR. This difference can be explained from various perspectives. First, ARM primarily consists of Korean researchers whereas APMR comprises investigators from all over the world. Additionally, there is a predilection for Science Citation Index journals, related to impact factors and research costs. Although high-quality evidence-based research is important, the fact that diverse research methods and experiences are necessary to form hypotheses should not be overlooked. Furthermore, when applying research to patients, various components from individual features to economic factors must be kept in mind, rather than depending solely on evidence-based research for the treatment of individuals.

Regarding the number of statistical methods used per article, in 2005, ARM numbers increased from 1.9 to 2.6, which indicates less frequent usage compared to APMR; however, this was still an internal increase for ARM. The number of articles that did not contain a statistical method increased from 20.9% to 26.6% between 2005 and 2015, but this result may be affected by a rise in the percentage of case studies/series. Both the frequency and the variety of statistical methods were lower in ARM as compared to APMR, but the use of advanced methods increased. The results above indicate changes about the view on statistics, more lectures associated with statistics in medical organizations, and enhanced access to information portals such as books, journals, and the Internet. Statistical methods do not only use differences or distributions in the process of formulating and testing a hypothesis, but open up processes to test the suitability of a sample. There is room for the usage of different statistical methods in the same study corresponding to different levels of understanding. Use of the appropriate statistical methods based on such understandings makes research more valuable and objective, which is more important than the number of statistical methods used, since various methods can be employed for different purposes. It is difficult to discuss the knowledge and understanding of statistics solely based on the results of this research but it can be stated that statistical approaches from various perspectives have increased from 2005 to 2015.

Changes in our views on statistics and the enhancement of various statistical approaches along with the development of evidence-based medicine allows for high-quality, systematic research and provides an adequate foundation for future clinical treatments. To that end, practitioners who actively consult statistics experts and exchange information during difficult times will be helpful for research. Moreover, it is hard for statisticians to fully grasp the clinical features or the contents frequently found in medical studies that use statistical methods and call for a clinician's or researcher's specific understanding and knowledge. Discussions related to this topic should continuously take place and opportunities for the acquisition of more knowledge and experience ought to be provided through conferences, education, and scholarly articles. Furthermore, a few suggestions regarding the research steps are given. First, research design methods, data collection, and statistical analysis methods should be precisely described. In APMR, they are clearly stated in the abstract and in terms of statistical methods, the steps are not simply listed but their purpose is accurately delineated. In ARM, a trend of stating the purpose of statistical methods more clearly can be witnessed, but even more improvements should be made, particularly since they are frequently omitted in research designs. Second, the extremely heavy weight assigned to case studies should be reconsidered. In order to draw more scientific and objective conclusions, the number of evidence- based studies should increase, which certainly will assist the journal development. Third, in order to choose proper statistical methods, the research purpose, design, and other factors must be considered, and post-examinations as well as reviews carried out. Fourth, conclusions should be drawn based on the ‘plan.’ For example, if any object was excluded or added to the original research plan, a related analysis should be performed. It is necessary to draw exact conclusions and to circulate information through a written explanation on the object change. In order to provide objective information without making any statistical errors, the results must be clearly based on the purpose. The selection of only significant values after simple statistical computations and their analysis should be avoided. In summary, it is beneficial to deeply think through the series of steps from setting the research purpose, research design, data collection, statistical analysis, conclusion to the discussion and to state them clearly in every research writing.

In this study, the frequency of appropriate applied methods was evaluated, but changes in research designs and statistical methods were not. An individual analysis for every study may be difficult but the errors and the suitability of statistical methods or designs such as in a prospective study or a case review ought to be certainly debated in future research. Additional topics related to research design and statistical method issues discussed in this study are definitely open for discussion, such as research techniques surrounding appropriate statistical usage, guidelines in accordance with research designs (CONSORT, STARD, STROBE, MOOSE, etc.), data size, and contents of the topic, but the amount of information is overwhelming [202122]. However, this is an illustrative case for the importance and necessity of further research. Most lead authors in ARM were medical doctor (97%); however, those of APMR consisted of people working in various fields—not only medical doctors, but also physical therapists, occupational therapists, nurses, science doctors, health scientists, and graduate students. Such occupational differences of the authors may indirectly reflect the differences in characteristics and purposes between ARMR and APMR. Since not all authors specified their occupations in a paper definitely, there may be some errors in the occupational classification. This paper can contribute to developments and it can suggest new directions for ARM, with most contributors being medical doctors. In future studies, analyzing the authors' occupations related to research subjects may produce interesting results, since this could point to their fields of interest. Additionally, a fragmentary comparison of journals might be misleading because the number of articles differs and the frequency of method use was a primary subject. One should not judge the quality of research or the level of the statistics presented in each journal solely based on frequency and type differences. It is our aspiration that these differences become motivating factors for more in-depth research, to forecast research trends, and to assist in research planning, as well as the development of systematic approaches, diverse perspectives, and the appropriate use of statistics. We also hope that statistical methods are perceived as tools that can provide powerful research evidence, as means to test oneself, to analyze treatment purposes in clinical applications, and to help accepting the research of others with more confidence. It is expected that ARM will become successful through its globalization and that it will play a pivotal role in the field of medical rehabilitation through such efforts and developments.