Assessment of COVID-19-related meta-analysis reporting quality.

BACKGROUND: Meta-analysis of high-quality primary articles represents the top-quality evidence in medical literature. In this project, our aim was to assess the number and quality of COVID-related meta-analysis published since the beginning of the COVID-19 pandemic.
METHODS: The search included the period from January 1, 2020, when the beginning of primary articles on COVID-19, till October 31, 2020. We screened a total of 793 studies. We excluded non-meta-analytic non-COVID-19-related studies. We obtained different characteristics, and we determined the quality of reporting using the AMSTAR tool, an 11-items tool that assesses the content validity and methodological quality of systematic reviews and meta-analysis.
RESULTS: A total of 538 studies were included in our assessment. The first meta-analysis included was published in March, while the last one was on the 31st of October. Upon comparing the mean AMSTAR score for meta-analysis published during each month, we found a significant difference (p < 0.001, F = 4.139), where the mean score almost steadily increased since March.
CONCLUSION: The urge to publish during the COVID-19 period or any other surge in publishing should not be at the expense of quality.

Introduction

Meta-analysis is the analysis of primary studies, where primary studies can be randomized controlled trials for treatment-related studies, and can be cohort studies for diagnostic and prognostic studies [1]. While it is possible to conduct a meta-analysis with only two studies in principle, evidence extracted from such studies may not be well received as strong evidence by the research community. Moreover, the methods used for data extraction and the quality of reporting are of paramount importance to decide the quality of evidence that can be extracted from such studies.

Since the beginning of the recent Coronavirus disease 2019 (COVID-19), researchers rushed into publishing primary studies reporting characteristics of patients and effectiveness of treatments, where the number of primary publications were increasing since January 2020 [2], even before announcing the COVID-19 as a pandemic. However, the increase in publications usually associated with of quality issues, where proportions of such studies may also be falsified [3, 4]. An example of such studies was a paper that suggested a link between coronavirus spike protein and HIV; the paper was among the most commonly discussed papers, despite being withdrawn by its authors [5]. Moreover, a previous study suggested that most studies registered in clinicaltrials.gov would yield low-quality evidence [6]. Taking these considerations into account, systematic reviews and meta-analysis might be a safer approach to extract filtered evidence, which is why the quality of systematic reviews and meta-analysis as a source for information for scientists and policymakers should be continuously checked. Due to the urge to publish COVID-related research by both authors and journals [7], the quality of meta-analysis published and the primary data included are not yet well assessed. Where meta-analysis studies mostly depend on quality of reporting and methods used for data collection, this aim of this study was to assess the number and quality of COVID-19-related meta-analysis since the beginning of COVID-19 pandemic.

Methods

Search strategy

We conducted a search on the PubMed-derived Lit COVID database using the keyword “meta-analysis” and using the quotation marks to search for the exact word [8]. Lit COVID database was previously found to be sensitive for COVID-related articles than a search using conventional keywords in other databases, but not specific, so a manual search should be done within searched results to exclude redundant results [9]. Lit COVID uses the following keywords to retrieve results from PubMed:

“coronavirus"[All Fields] OR "ncov"[All Fields] OR "cov"[All Fields] OR "2019-nCoV"[All Fields] OR "COVID-19"[All Fields] OR "SARS-CoV-2"[All Fields].

We also performed a separate PubMed and Web of Science search using the same keywords above, which did not yield further results, further confirming its sensitivity.

The search was conducted on November 8, 2020, to include the period from January 1, 2020, when the beginning of primary articles on COVID-19, till October 31, 2020. From the searched results, we adopted the following exclusion criteria:

Non-meta-analytic studies, including primary studies, researcher letter, commentaries, review articles.

Meta-analysis not on COVID patients, including meta-analysis on general populations, biochemical studies, and even geographical studies.

Figure 1 details the number of retrieved studies, number of excluded studies, and the remained included studies.

Fig. 1

The number of retrieved studies, number of excluded studies, and the remained included studies

Characteristics assessed

For each study included, we obtained PMID number, title, first author affiliation, date of online publication, last date of literature review done for the meta-analysis, journal published in, and the main topic of the study. We also assessed the number of studies included and the total sample size assessed. Finally, we determined the quality of reporting using the AMSTAR tool [10], an 11-items tool that assesses the content validity and methodological quality of systematic reviews and meta-analysis.

We categorized meta-analysis purpose into either prognosis directed, diagnosis directed, and therapy directed studies, a classification adopted by the newer approaches to evidence-based medicine [11].

Bibliometric analysis

We used Web of Science database to perform a bibliometric analysis on the included studies. We first inputted the included studies through the advanced search function, by using the PMID number with the “OR” function, as previously detailed [12]. We analyzed the search in regard to publication country and organization. Moreover, we also analyzed the number of citations received by the included meta-analysis studies.

Statistical analysis

We used SPSS version 21.0 (Chicago, USA) in our analysis. We used mean (± standard deviation) to describe continuous variables, and we used count (frequency) to describe other nominal variables.

We performed one-way ANOVA to analyze the mean AMSTAR score, number of studies, and number of participants between each month. We also used one-way ANOVA to analyze the mean AMSTAR score, number of studies, and number of participants between each study category. We used Pearson correlation test to analyze the correlation AMSTAR score, number of studies, number of participants, and months. We used chi-square test with Bonferroni multiple testing correction to analyze the difference between AMSTAR questions and each month. All underlying assumptions were met. We adopted a p value of 0.05 as a significant threshold.

Results

A total of 538 studies were included in our assessment. The first meta-analysis included was published in March, while the last one was on October 31. Details regarding the number of meta-analytic studies published each month are shown in Fig. 2.

Fig. 2

The details regarding the number of meta-analysis studies published each month

Upon comparing the mean AMSTAR score for meta-analysis published during each month, we found a significant difference between mean AMSTAR score and the month of publishing (p < 0.001, F = 4.139), where the mean score almost steadily increased from March to October (Table 1). The lowest mean score was for studies published in May with 6.22 ± 1.64, and the highest mean score was for those published in October with 7.56 ± 1.56, with a mean difference of 1.34 and a 95% confidence interval between 0.51 and 2.17 (p < 0.001). Moreover, the Pearson correlation coefficient showed a significant positive correlation between months of meta-analytic studies published in and AMSTAR score (p < 0.001, r = 0.167). Despite having a statistically non-significant difference between number of studies included in the meta-analysis and months (p = 0.08, F = 1.825), the Pearson correlation coefficient showed a significant positive correlation (p = 0.005, r = 0.121). The number of participants included in the meta-analysis did not significantly differ between months of meta-analysis publishing (p = 0.905, F = 0.395), and did not correlate with months either (p = 0.737).

Table 1

The mean and standard deviation (SD) for AMSTAR score, number of studies included in the meta-analysis, and number of participants included in the meta-analysis for each month

		AMSTAR score		Number of studies		Number of participants
		Mean	SD	Mean	SD	Mean	SD
Month publishing	March	6.31	1.25	13.33	15.59	17,979.00	29,071.08
	April	6.36	1.77	21.84	18.21	7202.25	12,516.07
	May	6.22	1.64	19.77	18.37	39,191.74	236,831.52
	June	6.44	1.30	23.85	23.44	13,719.71	27,735.79
	July	6.57	1.46	25.01	20.32	20,786.19	82,081.37
	August	6.66	1.59	28.57	35.61	17,098.39	36,246.41
	September	6.47	1.59	23.33	26.33	18,404.27	59,692.92
	October	7.56	1.56	37.21	69.44	19,731.35	42,883.39

Upon performing a sub-group analysis with multiple testing correction to assess which specific AMSTAR questions significantly differed between months, we found that question five that assesses if a list of included and excluded studies provided (Was a list of studies (included and excluded) provided?), and question eight that assesses if scientific quality of the included studies used appropriately in formulating conclusions (Was the scientific quality of the included studies used appropriately in formulating conclusions?) significantly differed between months (p values were 0.002 and < 0.001, respectively). The percentage of studies compliant with including a list of studies improved from 0% in March to 12.5% in October, whereas studies compliant with assessing scientific quality improved from 7.7% in March to 36.4% in October (Table 2).

Table 2

The percentage of studies correctly fulfilling each AMSTAR question each month and p values of the difference in these percentages between months

	March	April	May	June	July	August	September	October	p value
AMSTAR 1	15.4%	3.0%	10.5%	19.4%	18.3%	23.2%	15.0%	21.1%	0.151
AMSTAR 2	92.3%	87.9%	80.3%	81.9%	88.5%	85.3%	76.3%	98.2%	0.024
AMSTAR 3	92.3%	90.9%	88.2%	94.4%	88.5%	84.2%	88.8%	94.6%	0.459
AMSTAR 4	7.7%	18.2%	27.6%	31.9%	36.5%	35.8%	25.3%	37.5%	0.136
AMSTAR 5	0.0%	0.0%	2.6%	0.0%	1.0%	3.2%	3.8%	12.5%	0.002
AMSTAR 6	92.3%	90.9%	94.7%	97.2%	97.1%	96.8%	96.2%	96.4%	0.797
AMSTAR 7	61.5%	63.6%	61.8%	66.7%	71.2%	65.3%	73.8%	87.5%	0.061
AMSTAR 8	7.7%	12.1%	3.9%	5.6%	7.7%	12.6%	12.8%	36.4%	< 0.001
AMSTAR 9	92.3%	93.9%	92.1%	93.1%	91.3%	95.8%	96.2%	94.6%	0.881
AMSTAR 10	69.2%	75.8%	61.8%	54.2%	56.7%	66.3%	65.8%	76.8%	0.102
AMSTAR 11	100.0%	100.0%	98.7%	100.0%	100.0%	97.9%	97.5%	98.2%	0.666

p values in bold are the significant p values after multiple testing correction

Most of the meta-analysis studies done for prognostic purposes (362, 67.3%), followed by diagnosis directed analysis (94, 17.5%), while therapy-directed meta-analysis were the least common (82, 15.2%). The proportion of therapy-directed meta-analysis has increased from only 8% in March to almost 19% in October, while the proportion of prognosis directed studies has decreased from 77 to 59% during the same period (Fig. 3).

Fig. 3

The change in proportion of prognosis, diagnosis, and therapy-based studies from March to October 2020

China was the country of highest number of meta-analysis published with 148 (27.5%), followed by USA with 73 (13.6%), and Italy with 38 (7.1%). Journal of Medical Virology has the highest number of publications, with 43 (8.0%), followed by Journal of Infection with 17 (3.2%), and both International Journal of Infectious Disease with 11 (2.0%) and Diabetes and Metabolic Syndrome with 11 (2.0%).

In regard to bibliometric analysis, 376 (69.89%) studies were indexed in Web of Science database. In regard to publishing area, 91 (24.2%) were internal medicine, 77 (20.48%) were categorized as infectious disease and virology, and 26 (6.92%) were categorized as cardiovascular diseases (Table 3). Lanzhou University published highest number of meta-analysis indexed with 15 publications, followed by both Harvard University and SiChuan University with 11 publications each. Indexed meta-analysis studies were cited 3503 times by a 2739 citing articles, with an average of 9.32 citation per document.

Table 3

Categories of research areas indexed meta-analysis were published in according to Web of Science categories

Area	Frequency	Percentage
General internal medicine	91	24.20%
Infectious disease and virology	77	20.48%
Cardiovascular system cardiology	26	6.92%
Public environmental occupational health	22	5.85%
Research experimental medicine	19	5.05%
Endocrinology metabolism	17	4.52%
Neurosciences neurology	15	3.99%
Pharmacology pharmacy	15	3.99%
Gastroenterology hepatology	14	3.72%
Oncology	13	3.46%

Discussion

Our analysis of around 800 meta-analyses published since the beginning of the COVID pandemic showed a linear increase in the quality of meta-analysis published since March 2020, when the first meta-analytic study was published. The most important quality measures that improved over the studied period were the use of scientific quality of the included studies in formulating conclusions and the inclusion of the included/excluded study list. After the medical community called for a need for better quality data [13], there has been a rise in the number of meta-analysis, as can be seen through our data, the number and more importantly the quality of research have been on the rise with the months following March 2020.

Among aspects commonly missed by meta-analysis, pre-registering the study (AMSTAR 1), considering the status of publication (AMSTAR 4), providing a list of included and excluded publications (AMSTAR 5), and considering the scientific quality of included studies in formulating the conclusions (AMSTAR 8). These missed quality measures are among the most important measures in quality of meta-analysis [10], and were also poorly considered in systematic reviews, as found by Yu et al. study [14], which found that methodological quality of most systematic reviews on COVID-19 are unsatisfactory [14]. Among these factors, providing a list of included and excluded studies and considering the scientific quality of included studies in formulating the conclusions were significantly improved during recent months’ meta-analysis compared to early ones.

Among the findings of our study was the high proportion of prognosis-directed meta-analysis, rather than diagnosis-directed or therapy-directed meta-analysis, an observation that was shifted with time toward therapy-directed meta-analysis. This finding is consistent with Pundi et al. study that observed a low proportion of studies that will yield strong evidence to change therapeutic and interventional approaches in dealing with COVID patients [6], and that even prognosis-directed studies do not focus on most important measures, like mortality [15]. Another study assessing the registered trials found a paucity in well-designed trials directed toward therapy-directed interventions [16].

To the best of our knowledge at the time of writing, our study included the largest number of meta-analysis published so far on COVID-19. However, the main limitation of our study was the relatively short period covered, while it yielded more than 500 meta-analytic studies published. Further prospective projects after the end of this pandemic should be carried out to confirm our results. In addition, other projects might compare the results of our study to previous pandemics.

In conclusion, our study assessed the quality of meta-analysis done since the beginning of COVID pandemic, where we observed an improvement in the quality of these meta-analysis. We also observed several key quality measures that commonly missed by meta-analysis, and prospective researchers should focus on considering them upon performing meta-analysis, especially those related to COVID-19. What we hope to send out from the current project is that the rush to publish should not overlook the quality of research, and we hope that this would be implicated in any following cause of publication surge in the future.