The quality check of a systematic review on the quality of randomized controlled trials from Saudi Arabia.

TO THE EDITOR

We read with interest the paper by Rajab et al. describing the quality of randomised controlled trials in Saudi Arabia [1].

This systematic review included 61 trials and showed that the volume and quality of trials in Saudi Arabia were low. They reported that no trials had low risk, 39% had high risk, and 61% had an unclear risk of biases. The quality of such trails was evaluated from the authors in a short period of time from March to April 2018.

In addition to the limitations reported by the authors, there are additional points that need to be discussed further.

The authors started to justify their study with a risk of bias stating that ‘the trend is toward publishing observational studies in local journals with a low impact factor”. This was based on a paper published in a low impact factor journal that included studies published between 2008 and 2012 [2]. Such results appear particularly important in this review, which uses old data from a study with several acknowledged limitations. Another source of bias, the high-quality systematic review should have predefined criteria to minimise bias by prospectively registering the review protocol as recommended by Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [3].

The systematic review is considered the gold standard form of evidence for assessing the intervention effectiveness, however, to be reliable and generalisable it should have a comprehensive search strategy to find all the evidence around the outcome of interest. Their search strategy included a very limited number of search terms. For instance, they used “Saudi Arabia” and “Randomized”, not including “KSA”, “Randomised” and even the free text terms for such keywords which, consequently underestimate the total number of articles included. Their search strategy in each database was omitted with no details in the appendixes, which indicates this search strategy is not performed according to evidence-based guidelines for literature searching. Having instructions from an information specialist in this regard is really important to have a rigorous search strategy.

Regarding the risk of bias, the aim of any research is to have data that can be evaluated and reproduced. However, in their study, there was no table describing the risk of bias for each domain in The Cochrane Collaboration Risk of Bias Tool (CCRBT) for each study included in this review as recommended by Cochrane Group [4]. This table is a must to inform the readers about the studies with a high risk of bias and those with low risk of bias. The ultimate aim is to not overestimate the treatment effect by those at high risk of bias and also supporting other researchers to prioritise those with the lowest risk of bias. Since they were excluding non-published RCTs, their conclusion can not be extrapolated to all RCTs in Saudi Arabia.

Authors requested the local researchers to follow the Consolidated Standards of Reporting Trials (CONSORT) when they submit trials for publication, which is the minimum requirement to ensure more rigorous outcomes. However, they contradict their recommendation by not following the PRISMA––an evidence-based minimum set of items for reporting a systematic review. Authors claim that the local researchers are mostly engaged in explorative types of studies (e.g., cross-sectional). This is unquestionably incorrect and needs more reliable data to confirm such a statement, as an example, we have Saudi researchers had recently published an RCT in a journal with one of the highest impact factor [5].

In sum, although the authors had worked extensively in one month to finalise a quality check of 61 RCTs about a very sensitive and important topic, the value of their findings is diminished because of all the reasons stated above. With such methodological flaws, it was expected from the authors to write that our conclusion should be interpreted with caution. Indeed, this is just a hidden systematic review with no influence on the outcomes of the randomised control trials in Saudi Arabia.