The quality of reports of medical and public health research from Palestinian institutions: a systematic review.

OBJECTIVE: Over the past decade, there has been an increase in reports of health research from Palestine, but no assessment of their quality. We have assessed the quality of reports of Palestinian health research and factors associated with it.
DESIGN: This is a systematic review. INCLUSION CRITERIA: We searched Medline and Scopus for reports of original research relevant to human health or healthcare authored by researchers affiliated with Palestinian institutions and published between January 2000 and August 2015 inclusive. OUTCOMES: We used international guidelines to assess report quality, classifying as adequate those with ≥50% of items completely addressed.
RESULTS: Of 2383 reports identified, 497 met our inclusion criteria. Just over half (264; 55%) of these were published after 2010. 354 (71%) of first authors were affiliated with Palestinian institutions; 261 (53%) reports had coauthors from outside Palestine. The majority of the reports in our study were inadequately reported (342; 69%), and none had adequately reported all items. Of 439 observational studies, 11 (2.5%) reports provided adequate descriptions of eligibility criteria and selection procedures; 35 (8%) reported efforts to address potential sources of bias; 50 (11.4%) reported the basis for the study sample size; and funding sources were mentioned in 74 reports (17%). Higher reporting quality was associated with international affiliation of the first author (prevalence ratio (PR) 1.6 (95% CI 1.2 to 2.1)), international collaboration (PR 2.9 (95% CI 1.7 to 5.0)), international funding (PR 1.9 (95% CI1.5 to 2.5)), publication after 2005 (PR 3.9 (95% CI 1.8 to 8.5)) and four or more coauthors (PR 1.5 (95% CI 1.1 to 2.1)).
CONCLUSION: Although the quality of reports of Palestinian research has improved in recent years, it remains well below an acceptable standard. International reporting guidelines should be used to guide research design and improve the quality of reports of research. TRIAL REGISTRATION NUMBER: The systematic review protocol was registered in the International Prospective Register of Systematic Reviews (PROSPERO) registery (registration number: CRD42015027553). © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2017. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

This is the first study to examine the quality of reporting of health research in Palestine or in any Arab country, using international reporting guidelines.

Citation screening and data extraction were conducted by two independent authors in duplicate.

We used internationally developed criteria from the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network.

We only searched two databases—Medline and Scopus—and that is only up to 2015 so we may have missed eligible studies not covered by these databases.

Introduction

Research reports are the most common way to communicate research findings for target readerships. Complete, accurate and transparent reporting of research studies facilitates dissemination, interpretation, translation and replication of research findings.1 Inadequate reporting has major consequences for clinicians, researchers, policy makers and ultimately patients.2 It impairs critical assessment of the validity, relevance and trustworthiness of research and so impedes its use in practice.3 It also limits the usability of study findings by other researchers conducting systematic reviews and meta-analyses and building on or replicating studies.4 In addition, inadequate reporting is one of the key contributors to avoidable waste in biomedical research.5 6 Researchers thus have an ethical obligation to research participants, funding organisations and society as a whole to report their findings in ways that are of use in practice and policymaking.

Several guidelines have been developed in attempts to improve the quality of the reporting of research. These include CONsolidated Standards of Reporting Trials (CONSORT),7 STrenghtening the Reporting of Observational studies in Epidemiology (STROBE),8 Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA),9 Standards for Reporting of Diagnostic Accuracy (STARD),10 CAse REports (CARE)11 and Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD).12 Evidence suggests that the adoption of these guidelines improves the quality of reporting of health research13–15 and the process and support of reviewers preparing quality reviews.16

Palestinian medical and public health institutions are relatively young and face many challenges, including scarce financial resources and limited research infrastructure. However, research is incentivised and encouraged at an institutional level and is linked to academic promotion.17 Furthermore, Palestinian academics, like those elsewhere, work in an atmosphere in which the maxim ‘publish or perish’ is accepted and promoted. As a result, researchers feel compelled to conduct research that they are often ill equipped to perform, with the consequence that it is done on too small a scale, over too short a time scale, and is subject to many possible biases.17

Health research in Palestine has been conducted initially in response to Palestinian community’s need to generate the evidence required to develop independent and informed health policies, with most of the resulting research studies ending up as unpublished reports.18 19 Later, publication of research reports increased and a recent analysis shows that there has been a promising rise in medical and health publications from Palestine over the previous decade.17 However, the quality of these studies has only been assessed using bibliometric analysis (such as journal citation data), which is an inadequate measure of the quality of research. In this study, we have assessed the quality of reports of medical and health research reported by authors affiliated with Palestinian institutions and explored the factors associated with the better reporting.

Methods

Study design

Systematic review: a protocol for this systematic review was devised in advance and registered in PROSPERO (registration no CRD42015027553). We followed PRISMA reporting guidance (see online supplementary material for PRISMA checklist).

Search sources and strategy

We searched the Ovid Medline and Scopus databases for reports of Palestinian human medical and public health studies published between 2000 and 2015 inclusive (search conducted on August 2015). We searched for the following terms in the author affiliation, title or abstract: ‘palestin*’, ‘jerusalem’, ‘west bank’, ‘gaza’, ‘oPt’, or ‘occupied Palestinian territories’ (for the complete search strategy, see online supplementary material).

Eligibility criteria

We included any report of original research authored and/or coauthored by researchers affiliated with Palestinian institutions. We included only research concerned with understanding anything that could have an effect on patients or other individuals and which had been conducted on individuals. We included quantitative studies, whether observational (cross-sectional, cohort, case–control studies and case series and reports), interventional (controlled trials) or systematic reviews and meta-analyses. Studies involving non-human animals were excluded. There were no language restrictions for inclusion in this review.

Study selection

Two reviewers (LA and MO) independently assessed the titles and abstracts of retrieved articles for definite or possible eligibility. We retrieved the full texts of all potentially eligible studies and assessed each full text independently by the same two reviewers. Disagreements about which studies to include were resolved by discussion and consensus. Reasons for exclusion were identified and documented.

Data extraction

For each included report, two reviewers (LA and MO independently used a data extraction form to record the following data: (1) citation details: authors, journal name, impact factor based on Journal Citation Report 2015,20 number of citations and year of publication; (2) number of authors, the first author’s affiliation and any national, regional (ie, the Middle East North Africa (MENA) region) or international collaboration; (3) description of disease or health domain evaluated, based on the groups used to study global burden of disease21; (4) type of study question (aetiological, diagnostic, prognostic and interventional) and study design (randomised controlled trial (RCT), non-randomised intervention study, cohort, case–control, cross-sectional and case report/series)22; and (5) source of funding (not reported/unclear, international/non-governmental organisations (NGOs), local government/or academic institution or industry).

Quality assessment of included studies

International Standard guidelines (STROBE,8 CONSORT,7 PRISMA,9 CARE,11 TRIPOD12 and STARD10 checklists) were used to assess the quality of reporting of each included study. We scored each item in each article as having been completely addressed (ie, when all the subitems were satisfactorily reported), partially addressed (ie, when some but not all the sub-items were satisfactorily reported) or when none of them had been addressed. One author (LA) assessed the quality of reporting of all included studies. A random sample of 10% of all included studies was independently assessed by a second author (KE). The level of agreement between authors was estimated using the kappa statistic (k=0.521; SE=0.033; 95% CI=0.455–0.586; moderate agreement). We have not attempted to assess the methodological quality (risk of bias) of included studies.

Reference to existing evidence

We also assessed whether the reports we included had referred to systematic reviews and/or meta-analyses of existing relevant research to justify the additional studies being reported, as recommended.23 Articles that have cited, described or discussed (as defined by Helfer et al 24) systematic reviews and/or meta-analysis were considered to have referred adequately to existing evidence.

Data analyses and interpretation

We reported the proportion of reports that completely, partially or did not address each item in the relevant reporting guideline. We calculated the percentage of completely addressed items for each report and assessed the quality of reporting by the percentage of completely addressed items, deeming ‘adequate’ to be reports with ≥50% of items completely addressed and ‘inadequate’<50%. We reported the total number and proportions of articles that completely reported ≥50% of the items, and then calculated the prevalence ratios (ie, the ratio of the proportion of the adequately reported studies with the examined factor over the proportion of the adequately reported studies without that factor, for example, publication year after 2005) comparing the following subgroups: author affiliation, collaboration, study question type, study design, funding source, number of authors per article, journal’s impact factor, number of citations and publication year). In addition, we reported mean±SD and the median and the IQR of the percentage of completely reported items per article for each subgroup. We used Fisher’s exact test (for categorical variables) and the Kruskal-Wallis test (for continuous variables) to investigate differences between groups.

Results

Out of 2383 potentially relevant reports that we identified, we excluded 1591 based on screening titles and abstracts. Of the remaining 792 full-text reports, we included 497 in our study. The reasons for excluding the others were (i) none of the authors/coauthors was affiliated to a Palestinian institution (105/295, 36%); (ii) it was a report of qualitative research (102/295, 35%); (iii) it had not been conducted on individual humans (eg, basic science research) (45/295, 15%); and (iv) full texts could not be retrieved (43/295; 14%) (figure 1).

Figure 1

Flow chart of the study selection process.

Characteristics of included studies

Publication details: just over half of the reports (265/497; 53%) were published after 2010, a third (164/497; 33%) between 2005 and 2010 and only 68 (14%) before 2005. About a third of the reports were published in journals with no impact factor. The mean impact factor of journals those published in which some of our study population had been published was 2.1±3.7. About two-thirds (354/497; 71%) of the reports had been cited less than five times or not at all.

Authorship affiliations: first authors of 354/497 (71%) reports were affiliated with Palestinian institutions, 29/497 (6%) with a regional institution and 114/497 (23%) with an international institution. Out of the 497 coauthors, 87 (18%) were affiliated with one Palestinian institution, 149 (30%) with more than one Palestinian institution, 30 (6%) with Palestinian and regional institutions and 231 (47%) with different Palestinian and international institutions. The number of authors per report ranged from 1 to 24. Most (361/497; 73%) of the reports were coauthored by four to six authors.

Study questions: three hundred and sixty (72%) of the reports presented prevalence/association studies, 60/497 (12%) therapeutic studies, 25/497 (5%) aetiological studies, 16/497 (3%) prognostic studies and 10/497 (2%) diagnostic studies. Three-quarters (372/497; 75%) of the studies reported were cross-sectional surveys. Only 16/497 (3.2%) were reports of controlled trials, and only 4/497 (0.8%) were systematic reviews. Sources of funding were not mentioned in about two-thirds (341/497; 69%) of the reports. The most commonly reported sources of funding were international (107/497; 22%) and local (49/497; 10%). Table 1 provides information on the general characteristics of the reports.

Table 1

Characteristics of included studies and the factors associated with adequate reporting of included studies (≥50% items were completely reported per article)

Factors	Studies (n (%*))	Adequately reported studies (n (%†))‡	Prevalence ratio (95% CI)	Percentage of completely reported items per article (median (IQR))	Refer to systematic reviews (n (%†))
Total	497 (100)	155 (31.0)		36 (23–55)	78 (15.7)
Affiliation(s) of the first author
Palestine	354 (71.2)	99 (28.0)	Reference	36 (22–50)	48 (13.6)
Regional	29 (5.80)	6 (20.7)	0.7 (0.4 to 1.5)	28 (23–46)	6 (20.7)
International	114 (22.9)	50 (43.9)	1.6 (1.2 to 2.0)	46 (27–64)	24 (21.1)
Collaborations
The same Palestinian institution	87 (17.5)	12 (13.8)	Reference	23 (14–36)	9 (10.3)
Different Palestinians institutions	149 (30.0)	43 (28.9)	2.1 (1.2 to 3.8)	36 (23–54)	15 (10.1)
Palestinian and regional institutions	30 (6.00)	9 (30.0)	2.2 (1.0 to 4.6)	41 (23–50)	5 (16.7)
Palestinian and international institutions	231 (46.5)	91 (39.4)	2.9 (1.7 to 5.0)	43 (27–59)	49 (21.2)
Type of the study question
Prevalence/association	360 (72.4)	125 (34.7)	Reference	41 (23–55)	55 (15.3)
Aetiology/risk factors	25 (5.0)	8 (32.0)	0.9 (0.5 to 1.7)	41 (36–54)	0 (0.00)
Diagnosis	10 (2.0)	0 (0.00)	–	20 (15–33)	3 (30.0)
Therapeutic/intervention	60 (12.1)	12 (20.0)	0.6 (0.3 to 0.97)	32 (18–41)	16 (26.7)
Prognosis	16 (3.2)	7 (43.8)	1.3 (0.7 to 2.2)	46 (41–56)	1 (6.3.0)
Others (case report)	26 (5.2)	3 (11.5)	0.3 (0.1 to 0.97)	35 (25–44)	3 (11.5)
Study design
Cross-sectional	372 (74.8)	121 (32.5)	Reference	36 (23–55)	60 (16.1)
Systematic review	4 (0.8)	2 (50.0)	1.5 (0.6 to 4.1)	44 (33–57)	0 (0.00)
RCTs	8 (1.6)	0 (0.00)	–	20 (19–29)	4 (50.0)
Non-RCTs	8 (1.6)	0 (0.00)	–	18 (12–25)	2 (25.0)
Cohort	36 (7.2)	15 (41.7)	1.3 (0.9 to 1.9)	46 (32–55)	7 (19.4)
Case–control	43 (8.7)	14 (32.6)	1.0 (0.6 to 1.6)	46 (25–50)	2 (4.7.0)
Case reports/series	26 (5.2)	3 (11.5)	0.4 (0.1 to 1.0)	35 (25–44)	3 (11.5)
Funding sources
Not reported/unclear	341 (68.6)	87 (25.5)	Reference	33 (23–50)	47 (13.8)
Local (governmental/ industry/ institutional)	49 (9.90)	16 (32.7)	1.3 (0.8 to 2.0)	41 (23–55)	11 (22.4)
International and NGOs	107 (21.5)	52 (48.6)	1.9 (1.5 to 2.5)	46 (30–66)	20 (18.7)
Reporting guidelines
STROBE	439 (88.3)	150 (34.2)	Reference	41 (23–55)	66 (15.0)
PRISMA	4 (0.80)	2 (50.0)	1.5 (0.54 to 3.9)	44 (33–57)	0 (0.00)
CONSORT	16 (3.20)	0 (0.00)	–	20 (12–28)	6 (37.5)
CARE	26 (5.20)	3 (11.5)	0.3 (0.1 to 0.99)	35 (25–44)	3 (11.5)
STARD	11 (2.20)	0 (0.00)	–	20 (17–33)	3 (27.3)
TRIPOD	1 (0.20)	0 (0.00)	–	32 (32–32)	0 (0.00)
Publication year
2005	68 (13.7)	6 (8.80)	Reference	23 (14–36)	4 (5.90)
2005–2010	164 (33.0)	49 (29.9)	3.4 (1.5 to 7.5)	36 (23–50)	22 (13.4)
2010	265 (53.3)	100 (37.7)	4.3 (2.0 to 9.3)	41 (23–55)	52 (19.6)
Number of authors per article
Single	51 (10.3)	7 (13.7)	Reference	27 (18–36)	6 (11.8)
2–3	134 (27.0)	36 (26.9)	2.0 (0.9 to 4.1)	36 (23–50)	14 (10.4)
4–6	227 (45.7)	75 (33.0)	2.4 (1.2 to 4.9)	41 (23–55)	35 (15.4)
7–10	57 (11.5)	23 (40.4)	2.9 (1.4 to 6.3)	46 (27–63)	15 (26.3)
More than 10	28 (5.60)	14 (50.0)	3.6 (1.7 to 8.0)	48 (30–73)	8 (28.6)
Impact factor of the journal
No impact factor	191 (38.4)	54 (28.3)	Reference	36 (23–50)	29 (15.2)
Less than 2	146 (29.4)	39 (26.7)	0.9 (0.7 to 1.3)	32 (18–50)	19 (13.0)
2 or more	160 (32.2)	62 (38.8)	1.4 (1.0 to 1.8)	42 (31–59)	30 (18.8)
Number of citations per article
None	164 (33.0)	46 (28.0)	Reference	36 (23–51)	28 (17.1)
Less than 5	190 (38.2)	59 (31.1)	1.1 (0.8 to 1.5)	36 (23–53)	34 (17.9)
5 or more	143 (28.8)	50 (35.0)	1.3 (0.9 to 1.7)	41 (23–55)	16 (11.2)
Referred to previous relevant systematic reviews
Yes	78 (15.7)	30 (38.5)	Reference	44 (31–67)	–
No	419 (84.3)	125 (29.8)	1.3 (0.9 to 1.8)	36 (23–50)	–

*The denominator of the percentage is the total included studies (n=497).

†The denominator of the percentage is the number of studies in each subgroup (row subgroup).

‡Adequately reported means≥50% items were completely reported per article.

Bold means significant (p value of Fisher’s exact test of statistical significance was <0.05).

CARE, CAse REport; CONSORT, CONsolidated Standards of Reporting Trials; NGO, non-governmental organisation; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses; RCT, randomised controlled trial; STARD, Standards for Reporting of Diagnostic Accuracy; STROBE, STrenghtening the Reporting of Observational studies in Epidemiology; TRIPOD, Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis.

Quality of reporting overall

We assessed about a third (155/497; 31%) of the reports as being adequate—at least 50% of all relevant items had been completely reported. However, no report had adequately covered all items. The median (first and third quantiles) percent of completely reported items per article was 36.4% (IQR 22.7%–54.6%), with a maximum of 95.5% of items completely reported in one article.

Completeness of reporting for the Introduction was generally high (68% of subitems per section), whereas title and abstract (40%), methods (32%), results (44%) and discussion (35% of subitems per section) were generally lower (see online supplementary figure S1). There was moderate inter-rater agreement between the two quality assessors (kappa statistics=0.52; 95% CI=0.46 to 0.59).

Of the 497 reports included, only 78 (15.7%) referred to systematic reviews of relevant existing evidence and the majority (419; 84.3%) did not cite at least one systematic review. Referring to systematic reviews or meta-analyses was associated with (i) collaboration among Palestinian and international institutions; (ii) studies of therapy/treatment; (iii) RCTs; and (iv) recent publication. However, higher quality of reporting was not associated with citation of relevant systematic reviews or meta-analyses (table 1).

Quality of reports of observational research

As the majority of the reports (439/497; 88%) were of studies of observational data, we assessed their quality using the STROBE checklist. Of the 439 observational studies assessed using STROBE, none completely addressed all 22 STROBE checklist items nor was any checklist item completely addressed in all 439 reports. Figure 2 shows the proportion of items assessed completely, partially or not at all. Of the 439 reports, 307 (69.9%) explained the scientific background and rationale for conducting the research and 297 (67.7%) its specific objectives. However, only 11/439 reports (2.5%) provided adequate information about eligibility criteria and sources and methods of selection of participants. Other items that were poorly reported were quantitative variables and how they were handled in the analyses (26/439; 5.9%); efforts to address potential sources of bias (35/439; 8%); sample size calculations (50/439; 11.4%); and funding sources (74/439; 17%). Only 180/439 (41%) reports identified the study design in the title or abstract and provided an informative, balanced summary of the study with the number of individuals at each stage of the study and use of flow diagrams (171/439; 39%). Only 151/439 (34%) reports mentioned key elements of the study design early in the text.

Figure 2

The frequency (and percentages of the total of observational studies) of observational studies that have been assessed completely, partially and did not reported each item in the Strengthening the Reporting of Observational studies in Epidemiology checklists (n=439).

The most frequent completely addressed item was reporting of outcome data, with 407/439 (93%) giving the number of outcome events or summary measures. Two hundred and ninety-five reports (67.2%) discussed key results in relation to study objectives (sources of the data and methods of assessment (measurement) (285/439; 64.9%), main results as unadjusted estimates and their precision (274/439; 62.4%) and settings, locations and relevant dates (267/439; 60.8%). Only 124/439 (28%) reports of observational studies mentioned the limitations of the study.

Factors associated with quality of reporting

Table 1 shows factors associated with higher quality of reporting. These were (1) first author affiliated to an international institution (PR1.6; 95% CI 1.2 to 2.0); (2) collaboration among different Palestinian institutions as well as Palestinian and international institutions (PR 2.9; 95% CI 1.7 to 5.0); (3) funding provided by international/NGO sources (PR 1.9; 95% CI 1. to 2.5); (4) recent publication after 2005 (PR 3.9; 95% CI 1.8 to 8.5); (5) authored by four or more co-authors (PR 1.5; 95% CI 1.1 to 2.1); and (6) published in a journal with an impact factor equal to or greater than 2 (PR 1.4; 95% CI 1.0 to 1.8). Case reports/series (PR 0.4; 95% CI 0.1 to 1.0) and studies of therapy/treatment (PR 0.58; 95% CI 0.34 to 0.97) were associated with lower quality of reporting.

Discussions

The majority of the reports in our study were inadequate as judged by international reporting guidelines, and we did not find any report that had reported all items. Outcome data (ie, number of outcome events or summary measures) were the most frequently satisfactorily reported items. Less than half of the reports provided adequate titles and abstracts, which are sometimes are the only parts of a report to be read. Study objectives were not clearly specified in about a third of reports. Only 2.5% of the reports of observational studies reported eligibility criteria and selection procedures completely, information which is essential in critical appraisal of reports.

Having international input, either in the form of first authorship, a collaboration, coauthorships or receiving funding from an international organisation are associated with higher quality of reporting. Research reports coauthored by four or more researchers were more likely to be of higher quality, as were reports published after 2005, or in journals with an impact factor of 2 or more. We intend to repeat our survey in 2–3 years’ time covering the period from September 2015, to assess whether there has been improvement in the quality of reporting.

Recent studies evaluating the quality of reporting of published reports of research have often been restricted to reports of studies with a specific study design (eg, RCTs25), disease area (eg, dermatology26) or to reports published in a limited number of journals.27 Despite the considerable heterogeneity in the quality of reporting documented in these studies, they make clear that reporting quality is generally poor.28 29 A systematic review of 50 studies (including >16 000 individual RCTs) evaluated the association between journal endorsement of the CONSORT statement and reporting quality found that there had been a slight improvement in the quality of reporting over recent years, but that it remained inadequate.30 Similarly, a survey of 456 observational studies found that the quality of reporting increased over time but remained unsatisfactory.31

Our findings were also comparable to findings from other developing countries. In a systematic review of all published reports of RCTs in China in 2004 (n=307), the quality of reporting was generally poor and lagged behind the quality of reports of RCTs in the developed world.32 Similarly, a survey of reports of medical research in Indonesia found that only a small proportion of reports were of high quality.33

It is worth mentioning that our findings reflect what was reported and not what was actually done. It is important to note that poor reporting does not necessarily reflect methodologically inadequate studies, and that apparently flawless reporting is no guarantee that a study was well conducted, or indeed that it was not completely fabricated.

Strengths and limitations

Strengths of our study included independent screening and data extraction from reports by two authors and our use of internationally developed criteria from the EQUATOR Network. One of its limitations is that we only searched two databases—Medline and Scopus—and that is only up to 2015 so we may have missed eligible studies not covered by these databases. In addition, only one author assessed reporting quality in all 497 included reports, although two authors independently assessed reporting quality in a 10% random sample of reports, with a moderate inter-rater agreement.

Conclusions

The quality of Palestinian medical and health reports has been improving but remains well below a satisfactory level. The inadequate reporting we have documented is contributing to waste in research in Palestine, where resources for health research are very limited. As elsewhere (www.rewardalliance.net), Palestine cannot afford waste in medical and health research. To judge by the reports we have studied, the Palestinian research institutions and researchers need to pay greater attention to ways of reducing waste and increasing value in research. International reporting guidelines should be endorsed and adopted by Palestinian institutions and academics, both to improve the reporting quality of research and to guarantee proper design and execution of research.