Evaluation of Preclinical and Clinical Studies Published in Medical Journals of Bosnia and Herzegovina: Methodology Issues.

INTRODUCTION: Results of preclinical and clinical studies in medicine could be trusted only if their design and statistical analysis were appropriate. AIM: The aim of our study was to investigate whether preclinical and clinical studies published in medical journals of Bosnia and Herzegovina satisfy basic requirements for appropriate design and statistical interpretation of data.
METHODS: Preclinical and clinical studies published in medical journals of Bosnia and Herzegovina were retrieved from the PubMed database, and the sample for analysis was randomly chosen from the retrieved publications. Implementation rate of basic principles of experimental design (local control, randomization and replication) and rate of the most common errors in design of clinical/observational studies was established by careful reading of the sampled publications and their checking against predefined criteria.
RESULTS: Our study showed that only a minority of experimental preclinical studies had basic principles of design completely implemented (7%), while implementation rate of single aspects of appropriate experimental design varied from as low as 12% to as high as 77%. Only one of the clinical/observational studies had none of the errors searched for (2%), and specific errors rates varied from 10% to 89%. Average impact factor of the surveyed studies was around one, and average publication date recent, less than 5 years ago.
CONCLUSION: Prevalence of preclinical studies that did not follow completely basic principles of research design, and that of clinical/observational studies with errors are high, raising suspicion to validity of their results. If incorrect and not protected against bias, results of published studies may adversely influence future research.

INTRODUCTION

If not designed and conducted appropriately, both preclinical and clinical/observational studies in medicine will produce incorrect results and erroneous conclusions, misleading future researchers who will use them as basis for their own studies. (1). There are several ways to prevent errors in research design, e.g. establishing guidelines for preclinical and clinical/observational studies (2), teaching research design at under- or post-graduate studies (3), and publishing special methodological articles in medical journals. However, surveys showed that more than half of the studies published in medical journals had some errors in design making their conclusions questionable (4). While for preclinical studies it is important to satisfy three basic principles of design (having appropriate negative and positive controls, replicating experiments on independent experimental units and random assignment of a treatment and an alternative to experimental units) (5), clinical and observational studies should avoid measuring bias, should be conducted on a sample of sufficient size, their statistics should be based on correct assumptions and their limitations should be openly and honestly stated. Failure to acknowledge and implement these principles when designing a study will lead to production of incorrect or spurious results, and real truth about the object of study will remain hidden (6). In our recent research on representative sample of experimental studies indexed in PubMed we have revealed that only a handful of studies (7%) were methodologically excellent, while majority had serious problems with design (7).

AIM

The aim of our study was to investigate whether preclinical and clinical/observational studies published in medical journals of Bosnia and Herzegovina (B&H) satisfy basic requirements for appropriate design and statistical interpretation of data.

METHODS

The studies were retrieved for analysis from the PubMed database. For the preclinical studies, the following inclusion criteria defined the pool from which the study sample was extracted: journal article, published in a journal issued in Bosnia and Herzegovina, original experimental study, animal study, in vitro study and full text availability. The exclusion criteria were: review articles, clinical trials of phase I-IV, cohort studies, case control studies and cross-sectional studies. The following search strategy was used to implement inclusion and exclusion criteria and select the pool of preclinical studies for further analysis: ((“acta informatica medica”) AND ((animal study) OR (in vitro study))) OR ((“materia socio-medica”) AND ((animal study) OR (in vitro study))) OR ((“medical archives”) AND ((animal study) OR (in vitro study))) OR ((“acta medica academica”) AND ((animal study) OR (in vitro study))) OR ((“medicinski glasnik”) AND ((animal study) OR (in vitro study))) OR ((“bosnian journal of basic medical sciences”) AND ((animal study) OR (in vitro study))). Filter Free Full text.

For the clinical and observational studies, the following inclusion criteria defined the pool from which the study sample was extracted: journal article, published in a journal issued in Bosnia and Herzegovina, original clinical trial, original observational clinical study, and full text availability. The exclusion criteria were: review articles, animal studies, in vitro studies, modelling studies and case reports or case series. The following search strategy was used to implement inclusion and exclusion criteria and select the pool of clinical/observational studies for further analysis: ((“acta informatica medica”) AND ((observational study) OR (clinical trial))) OR ((“materia socio-medica”) AND ((observational study) OR (clinical trial))) OR ((“medical archives”) AND ((observational study) OR (clinical trial))) OR ((“acta medica academica”) AND ((observational study) OR (clinical trial))) OR ((“medicinski glasnik”) AND ((observational study) OR (clinical trial))) OR ((“bosnian journal of basic medical sciences”) AND ((observational study) OR (clinical trial))). Filters: Free full text.

Size of the study sample for both preclinical and clinical/observational studies (n=43) was calculated on the basis of the following assumptions: rate of inappropriate research design 0.5 (4) and width of the 95% confidence interval ± 0.15. The formula n = (1.96)2 x 4*p*(1-p)/d2 was used for the calculation, where „n“ is the sample size, „p“ probability of inappropriate research design and „d“ width of the confidence interval (8). Since the studies retrieved by the abovementioned search strategy were numbered orderly in the PubMed database, the study sample of 43 studies was extracted by simple randomization technique, activating for 43 times random number generator in Excel, using formula RANDBETWEEN(1;185) for preclinical, and RANDBETWEEN(1;149) for clinical/observational studies.

The extracted preclinical studies were analyzed for internal methodological validity, checking whether basic principles of correct experimental design (replication, control and randomization) were followed. For the purpose of preclinical studies analysis, the checklist with 8 questions was prepared (Table 1). The extracted clinical/observational studies were analyzed for common errors in design and statistics, as earlier described in the literature (measuring bias, lack of randomization, ignoring assumptions for statistical tests, ignoring intention-to-treat analysis, etc.) (9). For the purpose of clinical/observational studies analysis, the checklist with 9 questions was prepared (Table 2). The details of the analysis of preclinical and clinical studies are shown in the Tables 3 and 4, respectively. The results are tabulated and described by rates and percentages when categorical, and by means, standard deviations, medians and interquartile ranges, if continuous. Normality of the data distribution was checked by Kolmogorov-Smirnov test, and if not achieved, Spearman’s nonparametric correlation was used. Maximum acceptable probability of null hypothesis was set at 0.05. All calculations were performed by SPSS statistical program, version 18.

Table 1.

Results of the survey of the preclinical studies (n = 43).

Requirement	Satisfiedn (%)	Not satisfiedn (%)	Unclearn (%)	Not applicablen (%)
Sample size reported for the experiment?	31 (72%)	12 (28%)	-	-
Number of observations reported for the experiment?	33 (77%)	10 (23%)	-	-
Value of test statistics, exact p value and degrees of freedom reported?	5 (12%)	37 (86%)	-	1 (2%)
Error bars correspond to the analysis (i.e. standard error is based on number of independent observations)?	15 (35%)	5 (12%)	2 (5%)	21 (48%)
Only independent observations were taken into account for statistical tests?	31 (72%)	6 (14%)	6 (14%)	-
Is there negative control?	33 (77%)	4 (9%)	-	6 (14%)
Was positive control necessary, and if so, was it used?	29 (65%)	9 (21%)	-	6 (14%)
Were treatments randomly allocated to experimental units?	17 (40%)	21 (49%)	-	5 (11%)
Number of citations:mean, standard deviation, median, interquartile range	1.9 ± 2.1; 1.0; 3.0
Time passed from the publication (years): mean, standard deviation, median, interquartile range	4.7 ± 3.0; 4.0; 5.0
Number of satisfied criteria per study: mean, standard deviation, median, interquartile range	4.3 ± 1.8; 5.0; 2.0

Table 2.

Results of the survey of the clinical studies (n = 44).

Requirement	Non (%)	Yesn (%)	Unclearn (%)	Not applicablen (%)
Failure to specify the inclusion and exclusion criteria?	37 (84%)	7 (16%)	-	-
Failure to determine and report the error of your measurement methods?	4 (9%)	18 (41%)	-	22 (50%)
Failure to specify the exact statistical assumptions made in the analysis?	18 (41%)	26 (59%)	-	-
Failure to perform sample size analysis before the study begins?	5 (11%)	39 (89%)	-	-
Failure to implement adequate bias control measures?	11 (25%)	26 (59%)	-	7 (16%)
Failure to vigorously recruit and retain subjects?	38 (85%)	4 (10%)	-	2 (5%)
Failure to examine for normality of the data?	17 (38%)	23 (52%)	-	4 (10%)
Failure to report missing data, dropped subjects and use of an intention to treat analysis?	10 (23%)	32 (73%)	-	2 (5%)
Failure to point out the weaknesses of your own study?	17 (39%)	27 (61%)	-	-
Number of citations: mean, standard deviation, median, interquartile range	1.8 ± 1.9; 1.0; 2.8
Time passed from the publication (years): mean, standard deviation, median, interquartile range	4.7 ± 3.2; 4.0; 3.8
Number of satisfied criteria per study: mean, standard deviation, median, interquartile range	3.6 ± 1.8; 3.0; 3.0

RESULTS

In total 43 journal articles describing preclinical studies were retrieved randomly from pool of 185 articles in the PubMed database defined by the inclusion and exclusion criteria, and then analyzed according to predefined criteria of research design quality. Average impact factor of the journals with preclinical studies (for the years when the articles were published) was 1.17 ± 0.42, median impact factor was 1.46, and interquartile range 0.70. Forty-four journal articles describing clinical and observational studies were retrieved randomly from pool of 149 articles in the PubMed database defined by the inclusion and exclusion criteria, and then analyzed whether having the most frequent errors described in literature. Average impact factor of the journals with clinical/observational studies (for the years when the articles were published) was 0.94 ± 0.48, median impact factor was 0.93, and interquartile range 0.99. Compliance of the articles with the criteria, average number of citations per article and average time elapsed from the publication of the articles are shown in the Tables 1 and 2. Only three of the analyzed preclinical studies (7.0%) had all basic principles of experimental design completely implemented, and only one (2%) of the clinical/observational studies had none of the errors searched for. Details of analysis of the studies are shown in the Tables 3 and 4. Number of satisfied criteria per study was not correlated either with journal impact factor (Spearman’s rho = 0.281, p = 0.068) or with number of citations (Spearman’s rho = -0.079, p = 0.612). The time elapsed from the publication also was not correlated with the number of satisfied criteria per study (Spearman’s rho = 0.021, p = 0.894). Number of satisfied criteria per study was not correlated either with journal impact factor (Spearman’s rho = -0.168, p = 0.276) or with time elapsed from the publication (Spearman’s rho = -0.294, p = 0.053). The number of citations was correlated reversely with the number of satisfied criteria per study (Spearman’s rho = -0.318, p = 0.036).

DISCUSSION

In Bosnia and Herzegovina there are 9 indexed biomedical journals in the year 2019 deposited in Citation databases. In Web of Sciences is indexed only Bosnian Journal of Basic Medical Sciences (BJBMS) with IF-1,45. In database Scopus is indexed 7 biomedical journals: Medical Arhives (H-index is 19), BBJBMS (H-index is 18), Acta Informatica Medica (H-index is 14), Acta Medica Academica (H-index is 10), Medicinski Glasnik (H-index is 10), HealthMed (H-index is 9), Acta Medica Saliniana (H-index is 3) and Folia Medica Facultatis Medicine Universitatis Saraeviensis (H-index is 1).

In Pubmed database are indexed 6 journals: Medical Archives, Matera Socio-Medica, Acta Informatica Medica, BJBMS, Medicinski Glasnik and Acta Medica Academica (14). Authors of this article wanted to make quality assessment of the statistical analysis of the data from results of investigations presented in published papers, randomly taken from deposited issues on Pubmed (abstracts) and Pubmed Central (full texts) from active six biomedical journals in Bosnia and Herzegovina regarding used methodological issues during its analysis and interpretation to be presented in scientific journals.

Our study showed that minority of preclinical studies had basic principles of design completely followed (7%), and that just one of the analyzed clinical/observational studies (2%) was free from the most common errors in design. Average impact factor of the surveyed studies was about one (what is usual for medical journals in the region), and publication date relatively recent, which means that our findings are actual. Considering preclinical studies, our findings on articles published in journals from Bosnia and Herzegovina do not differ significantly from findings of our previous study on articles published in highly cited international medical journals with high impact factor (some of the parameters were even better), and from findings of other studies, suggesting global character of this phenomenon (10). Necessity of randomization and replication of experiments on independent units of observation is not understood by many researchers, who although employing expensive technology and sophisticated measuring methods but missing randomization and/or replication do not get reliable results (11). Biological variability between independent experimental units is high, and if the treatments are not randomly assigned, or the experiments are not repeated for sufficient number of times on independent units, individual characteristics on one or a handful of experimental units will be erroneously understood as representative for the whole population, or the other way around, true characteristics of the population will remain unknown (12-15).

Results of our analysis of clinical and observational studies were even worse, and absolute number of satisfied criteria (i.e. absence of errors) was lower than that of preclinical studies. Main errors made in the clinical and observational studies were ignoring measurement errors, investigator’s bias, not calculating necessary sample size and ignoring assumptions for statistical tests, especially normality of data distribution. In majority of studies the authors did not think of reliability of their measurement instruments and ignored possible measurement errors: even simple measurement of blood pressure could give misleading and incorrect results if not conducted properly and if the instrument for measuring blood pressure was not calibrated recently. Investigator’s bias was also rarely thought of, and besides randomization in clinical trials, no other measures were taken to control such bias. Key question in observational clinical studies is who is collecting the data, and how certain incomplete or corrupted data are interpreted; this process should be made objective as much as possible, at least by employing two or more investigators to work together on the same data source, or to implement strict control by chief investigator over work of co-investigators. Use of statistical tests can also be misleading if their assumptions are ignored. The most frequent error of this kind is use of parametric statistical tests on data that are not distributed normally; in such situation more sensitive parametric tests will make type 1 statistical error, finding a difference where it does not exist.

The articles we analyzed in this study were cited regardless of their methodological errors and possibly biased results, and this may mislead future researchers (16, 17). It is interesting that number of citations was not positively correlated with number of satisfied methodological criteria, suggesting that readers of the articles did not critically analyze methodological issues. Knowledge and skills of critically evaluating validity of published articles are essential for researchers, and should be given special emphasis in undergraduate and postgraduate programs in medicine (105-107).

Limitations of the study

The results of our study are limited to only one database (PubMed), and we missed to analyze other medical journals from Bosnia and Herzegovina with other kinds of editorial practices. Therefore, our results could underestimate or overestimate the problem of errors in study design, and should be interpreted with caution. The other limitation was that many published papers did not present sufficient data to allow for complete estimate of design issues.

CONCLUSIONS

Prevalence of preclinical and clinical/observational studies published in medical journals of Bosnia and Herzegovina that did not follow completely main principles of research design is high, but not higher than in influential international journals. Since incorrect results of published studies may adversely influence future research, it is necessary to make additional efforts through education and editorial practice to improve methodological quality of published studies. Earlier undergraduate and postgraduate statistical courses (both generic and specific) were mostly obsolete and incomplete, and the Bologna concept of teaching statistics also does not cover all essential methodological principles of both preclinical and clinical research.

Our previous study on a sample of preclinical studies published in journals with high impact factor and cited in WoS databases had already revealed this phenomenon, yet our intention in this study was to investigate with the same approach whether there are significant differences in methodological quality of articles published in less influential medical journals but indexed in the most comprehensive database of biomedical journals - PubMed. Our study showed that there are no significant differences between journals with high and relatively low impact factors in regard to the methodological quality, and this finding should be discussed further in the academic and scientific community.