The quality of reporting of cohort, case-control studies in the korean journal of family medicine.

BACKGROUND: The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement was developed to improve the reporting of observational studies. We aimed to evaluate the quality of reporting in cohort studies and case-control studies among observational studies published in the Korean Journal of Family Medicine.
METHODS: We searched for cohort studies and case-control studies published as original articles in the Journal of the Korean Academy of Family Medicine during the period January 1992 through December 2009. The main outcome measures were the number and proportion of cohort studies and case-control studies that reported each of 22 checklist items of STROBE.
RESULTS: We identified a total of 84 articles, of which 46 articles were cohort studies and 38 were case-control studies. Concerning methods, study designs (10%), bias (13%), study size (0%), statistical methods (12-c and 12-e items, 0%; 12-d item, cohort study, 6%) have been poorly reported. Of results, participants (5-6%), descriptive data (14-b item, 5%), and funding (1%) among other information have been poorly reported.
CONCLUSION: The degree of adherence the STROBE recommendations was relatively low in cohort studies and case-control studies published in the Korean Journal of Family Medicine. An effort to improve the reporting of observational studies by application and recommendation of the STROBE statement is required.

INTRODUCTION

A great number of studies are being conducted in contemporary society, and researchers should present the processes and outcomes of their studies transparently so that the readers can screen out high-quality research and apply it to relevant areas. Reporting guidelines are created to help researchers present the readers the objectives and design of their studies, methods, analysis procedures, results, and the interpretation of the results, etc. in their theses.1) If the contents of research are not reported adequately, this makes it difficult for the readers to interpret and apply research results effectively.

Accordingly, several institutions established guidelines for academic reports by identifying items to be included in research by research design. By referring to such guidelines, authors writing such reports can keep themselves from omitting important information in their reports, and readers, editors and thesis reviewers can evaluate the quality of research. Representative guidelines recommended for report writing include Consolidated Standards of Reporting Trials (CONSORT) for randomized controlled trial, Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) for systematic review, Meta-analysis of Observational Studies in Epidemiology (MOOSE) for the meta-analysis of observational studies, Standards for Reporting of Diagnostic Accuracy (STARD) for diagnostic accuracy studies, and Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) for observational studies.2)

Among them, STROBE is a guideline that was established in September 2004 by a number of medical journal editors, epidemiological researchers, methodologist, statisticians, and clinical researchers for improving the quality of reports from observational studies (cohort studies, case-control studies, and cross-sectional study).3) In foreign journals, there is steadily increasing evidence that the introduction of CONSORT improves the quality of reports from randomized controlled trials.4) In addition, according to the results of evaluating how the abstracts of randomized controlled trials and observational studies satisfy the guidelines of CONSORT and STROBE, abstracts published in 2008 showed a much improved quality compared to those in 2005.5)

Until now there have been two studies on the current state and analysis of papers published in the Korean Journal of Family Medicine,6,7) and they showed the steady quantitative growth of papers from observational studies. However, no study has been conducted on the quality of reports from observational studies, which occupy a relatively large portion of papers published in the Korean Journal of Family Medicine. Thus, through this study, the authors purposed to assess how the STROBE guidelines are satisfied by the theses of cohort studies and case-control studies among observational studies published in the Korean Journal of Family Medicine and ultimately to contribute to improvement in the quality of the Korean Journal of Family Medicine publications.

METHODS

1. Subjects

This study analyzed 1,301 original articles among papers published in the Korean Journal of Family Medicine during the period from January 1992 (volume 13) to December 2009 (volume 30), excluding review articles, lectures, special contributions, case reports, seminars, training and education, summaries of domestic and foreign theses, research topic announcements, and posters. The reason for limiting the scope to those from volume 13 is that some of the contribution rules were changed in 1992.

2. Selection of Subject Theses

In consideration of the scale and effects of research, we included only cohort studies and case-control studies among observational studies. Two of the authors searched for theses separately and checked their titles and abstracts, and if information was not enough from the title and the abstract of a thesis we determined whether to include the thesis by checking its original text. If the two authors did not agree with each other, a third author joined the discussion and the three authors decided whether to include the thesis. The definitions of cohort study and case-control study followed what is explained in STROBE,8) and deficiencies were supplemented with other literature (National Health Service).9) Reports were classified using the flow chart of the classification tool developed by Kim et al.10) and those classified as cohort studies or case-control studies were used as the sample of this study.

3. Evaluation of Thesis Quality

For the theses selected as the sample of this study, we evaluated whether their reporting observed the methods recommended in the STROBE reporting guidelines. The evaluation was made by two of the authors. Before evaluation, the two authors obtained full understanding of the definition of each item in the STROBE checklist11) and went through a training course for evaluating cohort studies and case-control studies. When the two authors' opinions were different from each other, they were adjusted through consultation with a third author.

The STROBE checklist consists of 22 items. Among them, 18 are common items evaluated regardless of research design, and the other 4 items (no. 6, 12, 14, 15) are different according to research design so they are evaluated differently according to research design (Appendix 1). Some items (no. 8, 13, 14, 15) are applied, respectively, to the exposed group and the non-exposed group in cohort studies, and to the patient group and the control group in case-control studies. In case an item is divided into several sub-items (a)-(e) as in item no. 1, 6, 12, 13, 14, and 16, each sub-item was evaluated separately. As a result, a total of 34 items were evaluated.

In the evaluation of a paper, each item was marked 'yes' if the item was described well, 'partially' if described partially, and 'no' if described inadequately. For each thesis, we evaluated the degree in which the items of the STROBE checklist were reported and, at the same time, we counted the number and proportion of theses reporting each item 'yes,' 'partially,' or 'no.' Further, we divided cohort studies and case-control studies into 10 years' periods, counted the average number of items reported 'yes' in each thesis for each period, and compared the results in order to examine change in reporting patterns over time.

RESULTS

1. Quantitative Change in Observational Studies over Time

Among theses published in the Korean Journal of Family Medicine during the period from 1992 to 2009, 1,301 original articles were collected through a search and were classified based on their abstracts and original texts into 46 cohort studies (19 prospective cohort studies, 27 retrospective cohort studies), 38 case-control studies, and 942 cross-sectional studies (Figure 1). When cohort studies and case-control studies were divided into 5 years' periods and the quantity of published theses was compared among the periods, steady quantitative growth was observed since the period of 1992-1995 (Table 1).

Figure 1

Flow diagram of included studies.

Table 1

The quantitative change of observational studies according to the years.

2. Analysis by Item of the STROBE Checklist

Reporting on common items applicable regardless of research design was quite varied between 0-95%, and mainly information on methods, results and research fund support was inadequate (Table 2). Reporting on items applicable only in cohort studies was between 6-89%, and especially information on statistical methods (12-d) and technical data (14-c) was insufficient (Table 3). Reporting on items applicable only in case-control studies was between 59-92%, relatively satisfactory (Table 4). Among the items of the reporting guideline, those evaluated 'yes' in over 80% of the theses were scientific background/reasons in the introduction (2), participants (6-a) and data sources/measuring (8) in methods, technical data (14-a) and result data (15) in results, and key results (18) in discussion. Items evaluated 'yes' in over 50% of the theses were title and abstract (1-b), participants (6-b), variables (7), quantitative variables (11) and statistical methods (12-a, 12-b, and 12-d of case-control studies) in methods, major results (16) and other analyses (17) in results, and limitations (19), interpretation (20) and generalizability (21) in discussion. Items evaluated 'yes' in under 20% of the theses were title and abstract (1-a), objectives in the introduction, research design (4), biases (9) and statistical methods (12-d of cohort studies) in methods, participants (13) and technical data (14-b) in results, and research fund support (22) in other information. Items not reported in any of the theses were sample size (10) and statistical methods (12-c, 12-e).

Table 2

Reporting of common items of the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement in 84 articles published in the Korean Journal of Family Medicines.

*Give such information separately for cases and controls in case-control studies, and, if applicable, for exposed and unexposed groups in cohort. †As it is just consideration, give the number of articles reported except proportion.

Table 3

Reporting of items applied to cohort study in the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement (n = 46).

*Give such information for matched studies (n = 3). †If researcher follow up participants according to years, give such information (n = 33).

Table 4

Reporting of items applied to case-control study in the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement (n = 38).

*Give such information for matched studies (n = 22).

3. Change in Reporting over Time (Figure 2)

In cohort studies, the mean number of items evaluated 'yes' per thesis was 13.7 in 1992-2000 and 14.86 in 2001-2009, so it was 1.16 larger in the 2000s than in the 1990s. In case-control studies as well, the number was 13 in 1992-2000 and increased by 1.94 to 14.94 in 2001-2009.

DISCUSSION

When this study evaluated the reporting quality of cohort studies and case-control studies published in the Korean Journal of Family Medicine, the level satisfying the STROBE guidelines was quite varied among the papers. In particular, the proportion of theses evaluated 'yes' was low for items research design (4, 10%), biases (9, 13%) and sample size (10, 0%) in methods, missing data (12-c, 0%), loss to follow up (12-d of cohort studies, 6%) and sensitivity analysis (12-e, 0%) in statistical methods, participants (13, 5-6%) and technical data (14-b, 5%) in results, and research fund support (22, 1%) in other information.

Researchers should provide clear information on research design so that readers can determine the level of evidence for the conclusions of their research. According to our study, only 8 (10%) of the theses studied described research design and only 15 (18%) mentioned research design in the title or abstract. In a study that evaluated the reporting quality of observational studies published in foreign dermatological journals during the period from 2005 and 2007, the proportion of papers reporting research design was 70%, relatively high.12) Among observational studies published in the Korean Journal of Family Medicine during the same period, however, only 1 (11%) reported research design in the title or the abstract. Because the reliability of conclusions is different depending on the characteristic of research design, the key elements of research design should be stated clearly at the early stage of research.8)

On the other hand, researchers should decide the number of subjects with sufficient statistical power before starting research. This is because when research results are not significant, they should judge whether there is no significant difference or a significant difference was not detected due to a small sample size. Nevertheless, none of the theses examined in this study reported how their sample size had been decided. In a study in 2005 that evaluated the reporting quality of randomized controlled trials published in domestic medical magazines, only 8.9% of the papers calculated the sample size.13) This suggests that the importance of information on sample size is not recognized properly not only in observational studies but also in randomized controlled trials published in the Korean Journal of Family Medicine. Therefore, instead of presenting merely the number of participants, the authors should state the process of calculating the number of subjects clearly for their readers.

As missing data can influence the generalizability of results or cause biases,11) information on missing data should be included sufficiently for readers' understanding. Loss to follow-up and drop-out rate are also part of missing data. In the results of our study, reporting quality for missing data in statistical methods and results was very low (0-5%). This is because most of the theses did not state the number of subjects excluded at each stage and the reasons for the exclusion, the reasons for loss to follow-up and the drop-out rate, and whether to include censored data or not, and did not provide information on missing data for each variable of interest. As a high drop-out rate may cause selection biases, it is necessary to decide how to deal with drop-outs at the planning stage. In studies based on medical records, moreover, if data on participants and variables of interest are not sufficient or there is loss to follow up, it can enhance reporting quality to provide detailed information on missing data through additional surveys by phone or mail.

In our results, the proportion of theses reporting items related to participants (13) was also relatively low at 5-6%. This was because the theses described the sampling of participants, data sources, inclusion criteria, the number of subjects, etc., but they omitted details on the number of potentially valid participants at each stage, the number of subjects who had been followed up to the end and were included in the final analysis, and the reasons for the exclusion of non-participants. These types of information are very important because they provide grounds for identifying selection biases or confounders. Moreover, research results are often interpreted inadequately because the results do not represent the target population, and for this reason researchers should provide detailed information on the participants.

When change in reporting quality over time was examined, the number of items evaluated 'yes' per thesis was 1.16 higher in cohort studies and 1.94 higher in case-control studies during the 2000s than in those during the 1990s, showing that improvement was not so significant. The Korean Academy of Family Medicine added the observance of reporting guidelines to its information of authors in July 2008.2) There may be a limitation in comparing reporting quality between before and after the revision of information of authors because the number of cohort studies and case-control studies published in 2009 is small. Nevertheless, when reporting quality before the revision of contribution rules (1992-2008) was compared with that after (2009), the mean number of items evaluated 'yes' per thesis in the two periods was 14.3 and 17.5, respectively, for cohort studies and 13.81 and 16, respectively, for case-control studies. This shows that reporting quality was higher after the revision.

Langan et al.12) evaluated how much the guidelines of STROBE were satisfied by observational studies published in 5 dermatological journals during the period from January 2005 to December 2007. A total of 138 theses were analyzed, and each item was rated 'yes,' 'partially,' 'no,' 'unclear,' or 'not applicable.' Items showing high reporting quality (70-99%) were title and abstract, introduction, research design, setting, participants (6-a), variables, data sources/measuring, result data, key results, interpretation, and research fund support. However, reporting quality was low for biases (31%), sample size (7%), quantitative variables (31%) and statistical methods (6-58%) in methods, and participants (6-39%), technical data (8-26%), major results (4-53%), other analyses (27%), limitations (55%) and generalizability (33%) in results. In comparison with our results, the level of reporting quality was generally higher even for low-quality items.

Muller and Egger14) sampled 60 theses using by searching PubMed using keywords 'sexually transmitted infection' and 'cohort study' during the period from April 2004 to March 2008, and evaluated how properly they reported items related to methods and results among the reporting guidelines of STROBE. The proportion of theses showing high reporting quality was varied between 35-93.3% for method items but was between 40.0-53.3% for result items. Because they evaluated only some of method and result items selectively, their results are not comparable with ours.

Our study is meaningful in that it is the first attempt to evaluate the reporting quality of observational studies published in the Korean Journal of Family Medicine. Moreover, in the current situation that improvement in reporting quality is required in response to the increasing volume of observational studies, our results are expected to be helpful for qualitative improvement if researchers design their studies in consideration of the items whose reporting quality was found to be low in this study.

This study has a number of limitations. One is that its subjects were limited to theses published in a journal. Therefore, it is not clear whether the conclusions drawn from this study are limited to the Korean Journal of Family Medicine or represent problems in all domestic medical journals. In addition, further research is necessary to determine improvement in the reporting quality of studies after the introduction of the STROBE reporting guidelines compared to that before. Second, research on observational studies require much knowledge about epidemiologic concepts or methodologies, and the authors went through a training course for evaluation, but some of our evaluation results might have been different if they were reviewed by epidemiologists or statisticians. Third, this study classified the theses using a new classification tool developed in Korea and there can be disagreement in thesis classification between this tool and other classification tools. Fourth, the period after the introduction of STROBE was too short to evaluate improvement after the guidelines. Accordingly, we can find the value of this research as a base study to determine potential for further improvement rather than in evaluating how well the guidelines were observed by theses published in the Korean Journal of Family Medicine. Fifth, because STROBE was developed mainly in consideration of Western countries with a good research environment, it may be somewhat irrelevant to the domestic research environment. In case of research funds, for example, theses contributed to the Korean Academy of Family Medicine might not mention research funds because few of them had received financial support and this might be mistaken for a low reporting quality.

In conclusion, among cohort studies and case-control studies published in the Korean Journal of Family Medicine, the proportion of theses observing the reporting guidelines of STROBE was quite varied and, in particular, items related to methods and results showed a low reporting quality. This suggests deficiency in key elements for readers to determine the reliability of research results. Accordingly, researchers need to design research with full understanding of the STROBE reporting guidelines and full consideration of items whose reporting quality is low. Moreover, if medical journal editors and thesis examiners introduce, recommend and educate the guidelines of STROBE, they will make positive contributions to improvement in the reporting quality of observational studies published in the Korean Journal of Family Medicine.