Single screen of citations with excluded terms: an approach to citation screening in systematic reviews.

Citation screening during the systematic review process can be time-consuming due to the exponentially increasing amount of research. This letter describes an approach to expediting the process by single screening citations that include terms in the abstract and/or keywords related to the exclusion criteria of the systematic review to quickly reject studies with a high likelihood of being excluded from the systematic review. This method can potentially improve the efficiency of the citation screening process while maintaining the quality of the systematic review; however, future research is needed to further validate this approach.

Background

Systematic reviews aim to identify all empirical evidence to answer a specific research question [1]. As part of the systematic review process, comprehensive literature searches of multiple databases are performed to identify all potentially relevant studies. The number of citations identified for a systematic review varies greatly based on the complexity of the systematic review and research questions (e.g., number of questions, inclusion criteria), the number of literature databases searched, the search strategies employed, and the range of search dates used. For some systematic reviews, thousands (or tens of thousands) of citations of potentially relevant studies are identified [2, 3]. As a result, systematic reviewers are exploring methods to improve the accuracy and efficiency of the citation screening process without compromising the quality of the systematic review [4-9]. This article describes an approach to expediting the citation screening process by single screening citations that include terms in the abstract and/or keywords related to the exclusion criteria of the systematic review to quickly reject studies that have a high likelihood of being excluded from the systematic review.

Approach

An initial literature search designed by a research librarian was conducted for a U.S. Preventive Services Task Force systematic review on the use of non-traditional risk factors in cardiovascular disease risk assessment [10]. The initial search yielded 18,360 unique citations; citations were managed in EndNote™ version 7.3.1 (Thomson Reuters, New York, N.Y.). The final research questions, protocol, and systematic evidence review [10], including the literature search strategies and study selection criteria, can be found at https://www.uspreventiveservicestaskforce.org.

After receiving the results of the initial search, but prior to initiating citation screening, a list of 83 terms (50 truncated terms) related to an excluded setting, population, or condition was developed based on the exclusion criteria of the systematic review (Table 1); from here on, these terms are labeled as “excluded terms.” In EndNote, these excluded terms were searched for in the abstract and/or keyword fields (using an asterisk for truncation) in groups of 10 (the maximum allowable number of terms in a search group) using the Boolean operator “OR.” Due to the limited number of groups of terms that can be combined into a “group of groups” (10 groups), one group of 10 excluded terms was only searched for in the abstract field. The nine groups of excluded terms were batched together using the Boolean operator “OR” and then separated from the group of citations that did not have these excluded terms in their abstract and/or keywords using the Boolean operator of “NOT” (reaching the limit of 10 groups in a “group of groups”). These citations were single-screened by the author for relevancy using the pre-specified inclusion and exclusion criteria of the systematic review, with single citations being either rejected or moved forward for dual abstract screening. Investigators screened citations in DistillerSR (Evidence Partners, Ottawa, Canada), an online-based systematic review software program.

Table 1

List of excluded terms

Exclusion criteria	Excluded termsd	Truncated excluded terma	No. of citations with term in abstractb	No. of citations with term in keywordsb
Condition:• Studies that exclusively include individuals with other pre-existing health conditions (e.g., HIV)	Arthritis, arthritic	Arthrita	267	183
	Autoimmune	Autoimmuna	93	21
	Cancer	Cancera	307	18
	Hepatitis, hepatic	Hepatia	162	24
	HIV	HIVa	197	135
	Infection, infectious	Infecta	736	349
	Kidney	Kidneya	813	789
	Lupus	Lupusa	127	112
	Pancreatic, pancreas	Pancreaa	35	25
	Renal	Renala	1135	781
	Rheumatology, rheumatic	Rheuma	336	223
	Sickle cell anemia	Sicklea	9	6
	Transplant, transplantation	Transplanta	281	255
	Prader-Willi syndrome	Willea	181	92
Country:• Studies not conducted in countries categorized as “Very High” on the 2014 Human Development Index (as defined by the United Nations Development Program)	Brazil, Brazilian	Braza	64	65
	China, Chinese	China	621	249
	Cuba, Cuban	Cubaa	51	1
	Egypt, Egyptian	Egypta	17	23
	Haiti, Haitian	Haitia	1	0
	India, Indian	Indiaa	136	90
	Iran, Iranian	Irana	53	28
	Iraq, Iraqi	Iraqa	3	1
	Kenya, Kenyan	Kenya	0	0
	Sri Lanka, Sri Lankan	Lankaa	3	1
	Libya, Libyan	Libyaa	0	0
	Mexico, Mexican	Mexica	33	25
	Nigeria, Niger, Nigerien, Nigerian	Nigera	8	10
	Peru, Peruvian	Perua	42	26
	Russia, Russian	Russa	37	21
	Sub-Sahara, sub-Saharan	Saharaa	11	1
	South Africa, South African	South Africaa	23	19
	Turkey, Turkish	Turka	37	95
	Ukraine, Ukrainian	Ukraina	1	0
Non-human:• Animal studies	Animal	Animala	221	--c
	Bovine	Bovina	6	--c
	Canine	Canina	9	--c
	Feline	Felina	1	--c
	Mammal, mammalian	Mammala	11	--c
	Mice	Micea	122	109
	Mouse	Mousea	49	--c
	Murine	Murinea	17	--c
	Pig	Piga	50	--c
	Primate	Primata	4	0
	Sprague-Dawley rat	Spraga	15	--c
	Swine	Swina	8	--c
Population:• Children age < 8 years• Pregnant women	Adolescent, adolescence	Adolesca	227	805
	Child, children	Childa	565	524
	Infant, infantile	Infanta	73	177
	Neonate, neonatal	Neonata	46	9
	Pregnant, pregnancy	Prega	253	134
Total	83 terms	50 truncated terms	7497	5426

aAsterisk indicates truncation of search term

bCitation counts are not mutually exclusive

cDue to limited number of terms and groups of terms in EndNote, these terms were only searched for in the abstract field

dPossible excluded terms based on truncation

A total of 6503 (35.4%) of the 18,360 citations from the initial search contained excluded terms in the abstract and/or keywords; these citations were screened by a single reviewer. The remaining 11,847 citations underwent traditional dual screening. Only 246 (3.8%) of 6503 citations containing excluded terms were moved forward for dual screening as they were deemed potentially relevant for inclusion, of which 39 (0.6% of 6503 citations with excluded terms) underwent dual full-text review. Among 39 articles reviewed at full-text, only 2 (0.03% of the 6503 citations with excluded terms) were included in the systematic review. The remaining 37 articles were excluded due to reporting on the wrong outcomes, being the wrong study design, or evaluating the wrong risk prediction base model. The two citations included in the systematic review did not influence the conclusions of the review. None of the 6257 citations with excluded terms—which did not undergo a second review—that were immediately rejected at single screening were brought forward for dual screening or full-text review based on checking the reference lists of included studies or existing systematic reviews. In total, 99.9% of the 6503 citations with excluded terms were excluded from the systematic review. Although these results only describe the initial search of the systematic review, these percentages are likely to not change with the inclusion of citations identified from bridge searches or after expert review and public comment.

Discussion

Single screening of citations with terms in the abstract and/or keywords that are related to the exclusion criteria of the systematic review is a potential method to increase the efficiency of the citation screening process as it reduces the number of citations requiring dual screening. It also has the potential to be reliable method as 99.9% of the 6503 citations with excluded terms were excluded from the systematic review. This approach would need to be reproduced in other systematic reviews to determine if it is a valid, reliable, accurate, and efficient method to citation screening. Single screening of citations with excluded terms was appropriate for this topic given its high literature yield and the a priori knowledge that most of the citations would not be relevant. This approach, however, may only work well (or be necessary) in topics that have a high literature yield and not those that have less restrictive inclusion criteria (e.g., any country). The amount of time spent on single and dual screening would also need to be compared to truly evaluate efficiency, which was not captured in this study. An optimal method for identifying and batching excluded terms is also needed. In this study, the selection of excluded terms was not systematic and could have been better informed by screening a few hundred citations to identify excluded terms that were more frequent or to omit those that might lead to false excludes. For example, “Kenya, Kenyan” and “Libya, Libyan” yielded zero citations, while “stent” (not an excluded term used) yielded 1480 citations. More excluded terms could also have been used if EndNote was not used to search for citations with excluded terms, as EndNote’s search function limited the number of terms to be searched and grouped. Excluded terms could be integrated into the original literature search strategies and imported separately into EndNote from those that did not have an excluded term in the abstract and/or keywords. And finally, an evaluation of whether the precautions to ensure all relevant studies are identified (i.e., examining reference lists of included studies and existing systematic reviews) is needed to determine that eligible studies with excluded terms were not wrongfully excluded during the single screen. There is still a possibility—even with reference mining—of a false negative as a result of excluding a relevant citation published since included studies or systematic reviews were completed.

Citations identified for a systematic review should at least be single-screened for eligibility, and citations with an excluded term should not be automatically rejected as the excluded term might be in the introduction or background section of the abstract. Due diligence should be taken to ensure no studies were wrongfully excluded by the single screener such as examining references lists of included studies and existing systematic reviews and querying experts for relevant citations. While more efficient processes are being developed and tested, such as machine learning, single screening citations that have a high likelihood of being excluded based on terms in the abstract and/or keywords related to the exclusion criteria of the systematic review can help reduce the number of citations requiring dual screening and thus improve efficiency while maintaining the quality of the systematic review. Additional research is needed to validate this approach.