Systematic review on the quality of randomized controlled trials in Saudi Arabia: Authors' response.

To the Editor,

We would like to thank the co-authors of the two correspondence (Vol. 16, December 2019) [1,2] for their interest in our paper entitled, “Systematic review on the quality of randomized controlled trials in Saudi Arabia” [3]. Herein is a response to the main issues that they raised in their letters —study rationale, registration, search strategy, and study duration and protocol.

The rationale for the systematic review was that the overall quantity and quality of randomized controlled trials (RCTs) conducted in Saudi Arabia in medical fields was unknown, as was stated in the paper. It was expected that the quantity would be low because some disease-specific (e.g., cardiovascular disease and type 2 diabetes) bibliometric studies showed that only a tiny minority of studies were trials (1–3%), and the overwhelming majority of studies were cross-sectional with major methodological weaknesses [4,5]. Additionally, regional bibliometric studies reported that Saudi Arabia is lagging behind not only western countries, but also behind regional countries like Turkey and Israel; for example, the number of publications in high-impact journals was 16 times higher, and the overall citation frequency was three times higher from Israel than from Saudi Arabia [[6], [7], [8]]. Actually, there is robust evidence to support the study rationale and to contradict the single study that was mentioned in the correspondence [9].

There are two issues at hand regarding Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA): registration and detailed data (e.g., risk of bias) for each included article. We considered registering this review with International Prospective Register of Systematic Reviews (PROSPERO); however, our experience with PROSPERO has been that they do not register systematic reviews that do not report on a specific patient or health outcome. Nonetheless, we adhered to and filled out the PRISMA checklist as rigorously as possible. In systematic reviews that focus on a particular disease outcome (such as Cochrane), the articles are typically stratified by risk of bias, and the associated statistical estimates are provided for the purpose of meta-analysis. In our review, we focused entirely on the methodology and reporting quality; we did not focus on a particular disease outcome, nor did we do any meta-analysis. Therefore, individual data estimates were not warranted.

The use of Medical Subject Headings (MeSH) is a recognized search strategy for systematic reviews. The MeSH system uses controlled language design, and articles are catalogued by Medline and not by authors. Because the MeSH system has a hierarchical design, small nuances in spelling (e.g., randomized or randomised) do not affect the cataloging. No search strategy can guarantee a perfect search result (i.e., 100% sensitivity and specificity). Authors can choose either a free-text or MeSH method as a search strategy. The first is more sensitive (i.e., higher number of records, more false positives, and more type 2 errors) as it allows for the use of more words (e.g., Saudi Arabia, KSA, Saudi), and the second strategy is less sensitive but more specific (i.e., lower number of records, more false negatives, and more type 1 errors) because it uses controlled language. Therefore, it is likely that our search missed a small number of eligible trials, but the critical question is whether the few missing trials could have changed the conclusions of the study. The answer is no because there is no reason to believe that a MeSH-based search systematically missed only the high-quality trials. One could maximize the sensitivity and specificity of the search results by using both methods [10] although the norm is to use one or the other.

The final assessment using the Cochrane Collaboration Risk of Bias Tool (CCRBT) was done during the months of March and April 2018 (2 months). Given that there were six co-authors and only 61 trials, one would agree that two months was sufficient to evaluate the articles. However, the preparation for this study (i.e., conceptualization, tool selection, search strategy determination, training, and pilot testing) started one year prior to the quality assessment (early 2017). In this period, the data abstractors received extensive training on CCRBT from the supervising authors, which included reading materials, group discussion, and mock extraction. These details about the study development and training period were unfortunately left out during the review and revision process of the manuscript in order to reduce the word count.

Our paper concluded that the majority of trials (61%) had an ‘unclear’ risk of bias. A proportion of these trials may well be high-quality studies in reality, but that could not be determined from the reporting in the manuscript. This perspective should not be missed just because no trial met the criteria for being low-risk (i.e., high-quality).

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Declaration of competing interest

The authors declare that they have no conflicts of interest related to this work.