Efficiency and contribution of strategies for finding randomized controlled trials: a case study from a systematic review on therapeutic interventions of chronic depression.

BACKGROUND: Identifying all existing evidence is a crucial aspect in conducting systematic reviews. Since the retrieval of electronic database searches alone is limited, guidelines recommend the use of additional search strategies. The aim of this investigation was to assess the efficiency and contribution of additional search strategies for identifying randomized controlled trials in conducting a systematic review on interventions after performing a sensitive electronic database search. DESIGN AND METHODS: Seven electronic databases, 3 journals and 11 systematic reviews were searched. All first authors of the included studies were contacted; citation tracking and a search in clinical trial registers were performed. A priori defined evaluation criteria were calculated for each search strategy.
RESULTS: A total of 358 full-text articles were identified; 50 studies were included in the systematic review, wherefrom 84.0% (42) were acquired by the sensitive electronic database search and 16.0% (8) through additional search strategies. Screening reference lists of related systematic reviews was the most beneficial additional search strategy, with an efficiency of 31.3% (5) and a contribution of 10.0% (5/50), whereas hand-searching and author contacts contributed two and one additional studies, respectively. Citation tracking and searching clinical trial registers did not lead to any further inclusion of primary studies.
CONCLUSIONS: Based on our findings, hand-searching contents of relevant journals and screening reference lists of related systematic reviews may be helpful additional strategies to identify an extensive body of evidence. In case of limited resources, a sensitive electronic database search may constitute an appropriate alternative for identifying relevant trials. Significance for public healthSystematic reviews provide the strongest form of evidence synthesis for therapeutic interventions and are of high relevance for decision makers in public health. Preparing high quality systematic reviews can be very time-consuming since all existing evidence should be identified, but the retrieval of electronic database searches is limited and therefore additional search strategies are recommended. However, the time needed for conducting full systematic reviews does often not address the need for urgent evidence. Thus, priorities of summarizing all available evidence and providing fast evidence-based recommendations may conflict. So far, no consensus exists regarding which additional search strategies are beneficial and sparsely time-consuming for conducting systematic reviews. We examined the efficiency and contribution of additional search strategies performed after a sensitive electronic database search. Our results provide highly relevant information for researchers conducting systematic reviews in various fields of public health research and for establishing guidelines for conducting rapid reviews.

Introduction

Systematic reviews provide the strongest evidence for therapeutic interventions according to the approach of Evidence Based Medicine.[1] Because systematic reviews of interventions should consider all existing relevant but unbiased data from clinical trials for achieving high validity, it is of particular importance to apply an optimal search strategy.[2] Furthermore, considering all existing evidence in the conduction of systematic reviews is of particular importance in the context of clinical decision making. This is especially the case, when many treatment options for a particular clinical question exist, only few studies in the field of a certain intervention have been conducted, and the existing evidence is conflicting.

Systematic search in electronic databases such as MEDLINE and Excerpta Medica Database (EMBASE) is considered as the gold standard in identifying relevant trials for conducting systematic reviews.[3] However, the limitations of solely an electronic database search are widely acknowledged.[24-6] The retrieval of applying an electronic database search strongly depends on the indexing of research topics within and across different sources.[4] Another factor that influences the retrieval of electronic database searches concerns the expertise of the individual conducting the search. An individual who is not trained in searching electronic databases might not be able to develop an optimal search strategy (e.g., not using MeSH terms adequately), which may result in missing relevant references.[4]

Accordingly, applying only an electronic database search entails the risk of missing a large amount of relevant literature, making it such that additional search strategies are necessary. Next to electronic database searching, additional search strategies constitute a valuable adjunct in identifying randomized controlled trials (RCTs) and controlled clinical trials (CCTs) for conducting systematic reviews.[2,4,5,7,8] However, no consensus exists regarding which additional search strategies should be applied. For example, the Cochrane Central Register of Controlled Trials (CENTRAL) as the first source to search is recommended when conducting systematic reviews,[2] whereas other authors recommend screening reference lists of related systematic reviews in addition to an electronic database search as the first step.[8] Apart from the diversity of recommendations as to which search strategy should be applied in addition to electronic database searching, little is known about the benefit of the various additional search strategies compared with each other. There is evidence that searching clinical trial registers as well as screening reference lists of related systematic reviews lead to the identification of more relevant literature than for example, hand-searching journal contents.[9]

Because additional search strategies are very time-consuming and project budgets are frequently limited, the question as to which search strategies are beneficial in addition to an electronic database search needs to be addressed. Thus, the very high time expenditure is described as a main disadvantage of hand-searching contents of relevant journals, screening reference lists of systematic reviews, as well as citation tracking. On the other hand, these search strategies contain the possibility of high accuracy, described as their main advantage.[10]

Furthermore, for many decision makers, the duration of six months or longer for conducting a full systematic review does not address the urgent need for evidence.[11] Thus, the priorities of meeting a high methodological quality when conducting a systematic review and providing fast evidence-based recommendations as demanded by policy makers and healthcare professionals results in conflict.[12] Rapid reviews aim to accelerate or streamline traditional systematic review processes to synthesize evidence within a shortened timeframe.[13] Although there appears to be a growing need of rapid reviews, methodological standards for conducting those reviews are still missing.[11] One substantial distinction in conducting rapid reviews in comparison to traditional systematic reviews consists of the abandonment of various particularly time-consuming search strategies, such as hand-searching contents of relevant journals and citation tracking. Likewise, studies not published in English and studies that are not available electronically are excluded.[11] In return, existing systematic reviews related to the research question play an important role. Additionally, extended systematic database searches in electronic resources such as MEDLINE, Cumulative Index to Nursing and Allied Health Literature (CINAHL), Cochrane Methodology Registry, and the Business Source Complete Database, as well as in open access journal databases are usually performed.[13] In another investigation that compared rapid reviews with systematic reviews the authors came to the conclusion that the essential conclusions of the rapid and full reviews did not differ extensively.[12] Thus, rapid reviews may constitute a useful alternative to full systematic reviews to deliver evidence in a timely manner and usable format, since there is evidence regarding the relative importance of additional search strategies, especially hand-search methods.[12] In this context, methodological research on the benefit of additional search strategies is of great relevance, as their role in rapid reviews (whether and which to perform) is still to be clarified.

The aim of this investigation was to assess the efficiency and contribution of additional search strategies for identifying RCTs in the context of conducting a systematic review on interventions after performing a sensitive electronic database search. Furthermore, within our investigation we tried to identify the additional search strategies that provide the highest benefit and are the least time-consuming at the same time. We aimed to gather information on which additional search strategies can be applied in case of limited resources.

Our case study deals with therapeutic interventions on chronic depression investigated in RCTs.[14] Search strategies focusing on identifying RCTs substantially differ from approaches to identify other forms of evidence (e.g., etiologic, prognostic, diagnostic).[15] In the present article we focus on systematic reviews of interventions and therefore on the following additional search strategies.

Hand-searching contents of relevant journals

Hand-searching journals comprises the systematic manual or electronically page-by-page examination of the contents of a journal issue or conference proceedings to identify eligible reports of trials and is described as a useful adjunct to searching electronic databases.[1,2] Relevant journals can be identified through electronic database searches on the basis of identified relevant records. Journals that publish most of the potentially relevant articles found can be considered.

Citation tracking

Forward citation tracking comprises the systematic examination of all references that cite any of the studies already included in the analysis, e.g., by using the Science Citation Index.

Backward citation tracking describes the systematic examination of reference lists of studies already included.

Screening reference lists of related systematic reviews

Because systematic reviews represent the most convenient sources of references to potentially relevant studies, reference lists of included and excluded studies of previous or similar reviews on the topic of interest can be screened for eligibility.[1] Special sources to identify existing reviews are for instance the Cochrane Database of Systematic Reviews, the Database of Abstracts of Reviews of Effects, and the Health Technology Assessment Database. Since reviews are also often indexed with corresponding keywords, they can be identified in electronic databases such as MEDLINE or EMBASE as well.

Searching clinical trial registers

Clinical trial registers constitute electronic databases that contain clinical trials only. Various national and international clinical trial registers exist, such as ClinicalTrials.gov and the International Clinical Trials Registry Platform (ICTRP).

Contacting first authors of included studies

All first authors of primary studies that are already included in the systematic review are contacted with an inquiry for further published or unpublished trials.

Design and Methods

We performed a secondary analysis of the systematic literature search in a project on the efficacy of psychotherapeutic, pharmacological and combined interventions in the treatment of chronic depression (MetaChron).[14,16,17]

Review eligibility criteria

Studies were included if they were conducted on adults with a diagnosis of chronic major depression, dysthymia, double depression, or recurrent depression without complete remission between episodes. Psychotherapeutic, pharmacological and combined interventions were considered. As comparator treatment no intervention, waitlist control groups, psychological or pharmacological placebos or other active interventions were regarded. The only studies included were RCTs that reported at least one outcome regarding the efficacy of the interventions. Further details on the review eligibility criteria are provided in the published study protocol.[14]

Electronic database search

First, a sensitive electronic database search targeting the occurrence of any of a large number of free-text terms at any point of the records was conducted in the following databases on January 18, 2010: CENTRAL, MEDLINE, EMBASE, ISI Web of Science, BIOSIS, PsycINFO and CINAHL. A disease component was combined (AND) with a design component focusing on RCTs for all searches. Within both components search items were combined with OR (e.g., MEDLINE): (((chron$ adj3 depress$) or dysthym$ or (double adj1 depress$) or (treatment adj1 resist$ adj1 depress$) or (non adj1 respon$ adj3 depress$) or (recurrent adj3 depress$)).ab,ti,sh.) AND ((random$ or rct).ab,ti. or random$.sh.).

Further detailed information regarding each particular electronic database search strategy is provided in Table 1.

Table 1.

Electronic database search strategies.

Database	Date of search	Search
Medline	18.01.2010	(((chron$ adj3 depress$) or dysthym$ or (double adj1 depress$) or (treatment adj1 resist$ adj1 depress$) or (non adj1 respon$ adj3 depress$) or (recurrent adj3 depress$)).ab,ti,sh.) AND ((random$ or rct).ab,ti. or random$.sh.)
Embase	18.01.2010	(((chron$ adj3 depress$) or dysthym$ or (double adj1 depress$) or (treatment adj1 resist$ adj1 depress$) or (non adj1 respon$ adj3 depress$) or (recurrent adj3 depress$)).ab,ti,sh.) AND ((random$ or rct).ab,ti. or random$.sh.)
PsycInfo	18.01.2010	(((chron$ adj3 depress$) or dysthym$ or (double adj1 depress$) or (treatment adj1 resist$ adj1 depress$) or (non adj1 respon$ adj3 depress$) or (recurrent adj3 depress$)).ab,ti,sh.) AND ((random$ or rct).ab,ti. or random$.sh.)
ISI Web of Science	19.01.2010	(TS=((“chron* depress*”) or dysthym* or (“double depress*”) or (“treatment resist* depress*”) or (“non respon* depress*”) or (“recurrent depress*”)) OR TI=((“chron* depress*”) or dysthym* or (“double depress*”) or (“treatment resist* depress*”) or (“non respon* depress*”) or (“recurrent depress*”))) AND (TS=(random$ or rct) OR TI=(random$ or rct)))
CINAHL	18.01.2010	((TI “chron* depress*” or TI dysthym* or TI “double depress*” or TI “treatment resist* depress*” or TI “non respon* depress*” or TI “recurrent depress*”) OR (AB “chron* depress*” or AB dysthym* or AB “double depress*” or AB “treatment resist* depress*” or AB “non respon* depress*” or AB “recurrent depress*”) OR (MW “chron* depress*” or MW dysthym* or MW “double depress*” or MW “treatment resist* depress*” or MW “non respon* depress*” or MW “recurrent depress*”)) AND (TI rct or AB rct or MW rct or TI random* or AB random* or MW random*)
BIOSIS	18.01.2010	((chron$ adj3 depress$) or dysthym$ or (double adj1 depress$) or (treatment adj1 resist$ adj1 depress$) or (non adj1 adj3 depress$) or (recurrent adj3 depress$)).ab,ti,sh. and ((random$ or rct).ab,ti. or random$.sh.) ((FT=random* OR FT=rct) AND PY=2005 to 2010 AND (LA=ENGLISH OR LA=GERMAN) AND pps=Mensch) AND
	25.01.2010	((((((FT=”chron* depress*” OR FT=dysthym*) OR FT=”double depress*”) OR FT=”treatment resist* depress*”) OR FT=”non respon* depress*”) OR FT=”recurrent depress*”) AND PY=2005 to 2010 AND (LA=ENGLISH OR LA=GERMAN) AND pps=Mensch)
Central	19.01.2010	((“chron* depress*”):ti,ab,kw or (dysthym*):ti,ab,kw or (“double depress*”):ti,ab,kw or (“treatment resist* depress*”):ti,ab,kw or (“non respon* depress*”):ti,ab,kw (“recurrent depress*”):ti,ab,kw in Clinical Trials) AND ((random*):ti,ab,kw or (rct):ti,ab,kw in Clinical Trials)

We preferred a sensitive electronic database search in contrast to a less extensive electronic database search to identify preferably all existing evidence. In this context, sensitivity is defined as the ratio of the number of relevant reports identified to the total number of relevant reports in existence.[1] Within the electronic database search, no language restrictions were applied, and all publications, including conference and dissertation abstracts from 1970 forward, were considered.

Additional search strategies

The following additional search strategies were conducted subsequent to the sensitive electronic database search and are listed in the order of their application. Potentially relevant studies as well as the studies that could be included in our systematic review subsequent to a detailed examination for eligibility were added to those already identified through the electronic database search in the order of the application of the additional search strategies.

Hand-searching contents of relevant journals

Relevant journals were identified and selected by considering the publishing journals of well-known potentially relevant primary studies as well as their impact factor. All volumes of the Archives of General Psychiatry, the Journal of Consulting and Clinical Psychology, and the Journal of Affective Disorders were searched by hand beginning with the year 1970.

Citation tracking

A cited reference search (forward citation tracking) was conducted in the Social Sciences and Science Citation Index. Backward citation tracking was performed by screening reference lists of all included studies.

Screening reference lists of related systematic reviews

A systematic search for systematic reviews has not been performed. Systematic reviews that appeared within the electronic database search as well as within searching the reference lists of all included studies and that focused on chronic depression and/or antidepressant interventions were screened for potentially relevant references.

Searching clinical trial registers

The following clinical trial registers were searched for on-going or non-published studies on February 15, 2010: ClinicalTrials.gov, the ICTRP.

Contacting first authors of included studies

The first authors of all included primary studies were contacted for further information regarding unpublished trials.

All titles and abstracts were screened by two independent ratters in order to identify potentially relevant studies. Likewise, all potentially relevant studies procured as full texts were checked for eligibility by two independent researchers. Disagreements regarding the inclusion or exclusion of particular studies were resolved through consulting a third researcher.

Statistical analysis

We defined two evaluation criteria to assess the benefits of the different search strategies: i) efficiency=a/b ; ii) contribution=a/c.

a=number of included studies identified by the search strategy

b=number of screened full-text articles identified by the search strategy

c=total number of included studies.

Accordingly, efficiency is defined as the ratio of included studies by the use of a certain search strategy to screened full texts identified through this search strategy. Contribution is defined as the ratio of included studies by the use of a certain search strategy to the total number of included studies in the systematic review.

Within our analyses records or references/papers were considered and counted first. Subsequently, the number of relevant studies was extracted (Figure 1).[18] In order to judge the value of the studies that could be identified through additional search strategies for our systematic review, further analyses regarding the risk of bias were performed applying the Cochrane Risk of Bias Tool,[1] including the following criteria: appropriate generation of allocation sequence, appropriate allocation concealment, appropriate blinding of participants, appropriate handling with incomplete outcome data, no selective outcome reporting, no other systematic errors within the study (Table 2). The global risk of bias was judged to be low when at least four of the six Cochrane criteria were met and there were no further indications of a high risk of bias. When three or more of the Cochrane criteria were not met and at least one of the unmet criteria suggested serious risk of bias, the global risk of bias of the included study was judged to be high. Subsequently, the methodological quality of the included studies identified through the sensitive electronic database search and those identified through additional search strategies was compared.

Figure 1.

Study flow chart (preferred reporting items for systematic reviews and meta-analyses).

Table 2.

Methodological quality of the included studies (risk of bias).

	1	2	3	4	5	6	Global judgment
Elkin/Agosti 1989*	Unclear	Unclear	Yes	Yes	Yes	Yes	Unclear
Aguglia 1995	Unclear	Unclear	Yes	Unclear	No	No	High
Amore 2001	Unclear	Unclear	Yes	Yes	Yes	Yes	Unclear
Anisman 1999	Yes	Unclear	Yes	Yes	Yes	Yes	Low
Baca 2003	Unclear	Unclear	No	Yes	Yes	Yes	High
Bakish 1993	Unclear	Unclear	Yes	Unclear	Unclear	Unclear	High
Barrett 2001/Williams 2000	Yes	Yes	Yes	Yes	Yes	Yes	Low
Bella/Fulgente 1990*	Unclear	Unclear	Unclear	Unclear	Yes	Unclear	High
Bellino 1997	Yes	Unclear	No	Yes	Yes	Yes	High
Bersani 1991	Unclear	Unclear	Yes	Unclear	Yes	Yes	Unclear
Bogetto 1997	Unclear	Unclear	Unclear	Unclear	Yes	Unclear	Unclear
Boyer 1996 A	Unclear	Unclear	Yes	Yes	Yes	Yes	Unclear
Boyer 1996 B*	Unclear	Unclear	Yes	Yes	Unclear	Yes	Unclear
Browne 2002	Yes	Yes	Yes	Yes	Yes	No	Low
deMello 2001	Unclear	Unclear	Yes	Unclear	Yes	No	Unclear
Devanand 2005	Yes	Yes	Yes	Yes	Yes	Yes	Low
Duarte 1996	Unclear	Unclear	Yes	Yes	Yes	Yes	Unclear
Dunner 1996	Unclear	Unclear	Yes	No	Yes	Yes	Unclear
Geisler 1992*	Unclear	Unclear	Yes	Yes	Yes	Yes	Unclear
Hellerstein 1993	Unclear	Unclear	Yes	Yes	Yes	Yes	Unclear
Hellerstein 1994/Rosenthal 1992*	Unclear	Unclear	No	No	Yes	Yes	High
Hellerstein 2010*	Unclear	Unclear	Yes	Unclear	Yes	Yes	Unclear
Katona 1999	Unclear	Unclear	Yes	Unclear	Unclear	Yes	Unclear
Keller 2000	Yes	Yes	Yes	Yes	Yes	Yes	Low
Kocsis 1988*	Unclear	Unclear	Yes	No	Unclear	Unclear	High
Kocsis 2009	Yes	Yes	Yes	Yes	No	Yes	Low
León 1994	Unclear	Unclear	Yes	Yes	Yes	Yes	Unclear
Markowitz 2005	Yes	Unclear	Yes	Yes	Yes	Yes	Low
Otero 1994	Yes	Unclear	No	Unclear	Yes	Yes	High
Randlov 2006	Unclear	Unclear	Yes	Unclear	Yes	Yes	Unclear
Ravindran 1999	Yes	Unclear	Yes	Yes	No	Yes	Unclear
Ravindran 2000	Unclear	Unclear	Yes	Yes	Yes	Yes	Unclear
Ravizza 1999	Unclear	Unclear	Yes	Yes	Yes	Yes	Unclear
Reyntjens 1986*	Unclear	Unclear	Unclear	No	Unclear	Unclear	High
Rocca 2002 A	Unclear	Unclear	No	Yes	Yes	Yes	High
Rocca 2002 B	Yes	Unclear	No	Yes	Yes	Yes	High
Keller 1998	Unclear	Unclear	Yes	Yes	Unclear	Yes	Unclear
Salzmann 1995	Yes	Unclear	Yes	Yes	Yes	Yes	Low
Schramm 2008	Yes	Unclear	Yes	Yes	Yes	Yes	Low
Schramm 2010	Yes	Yes	Yes	Yes	Yes	Yes	Low
Serrano-Blanco 2006	Yes	Yes	No	Yes	Yes	Yes	Unclear
Singh 1987	Unclear	Unclear	Unclear	Unclear	Unclear	Unclear	High
Smerladi 1998	Unclear	Unclear	Yes	Yes	Yes	Yes	Unclear
Stewart 1985	Unclear	Unclear	Yes	Unclear	Yes	Unclear	High
Thase 1996	Yes	Unclear	Yes	Yes	Yes	Unclear	Low
Tyrer 1988	Unclear	Yes	Yes	Yes	No	Yes	Unclear
Vallejo 1987	Unclear	Unclear	Yes	No	Unclear	Yes	Unclear
Vanelle 1997	Unclear	Unclear	Yes	Yes	Yes	Yes	Unclear
Versiani 1997	Unclear	Unclear	Yes	Yes	Yes	Yes	Unclear
Zanardi 2006	Unclear	Unclear	Yes	Yes	Yes	Unclear	Unclear
Total Yes	16/50	8/50	39/50	33/50	38/50	38/50

1: appropriate generation of allocation sequence; 2: appropriate allocation concealment; 3: appropriate blinding of participants; 4: appropriate handling with incomplete outcome data; 5: no selective outcome reporting; 6: no other systematic errors within the study;

*studies identified through additional search strategies.

The impact of the studies identified through additional search strategies on the total results of our systematic review was estimated by comparing the results of the meta-analyses with and without inclusion of these studies. Effectiveness measures including benefit ratios for response rates and odds ratios for dropout rates were calculated for three relevant comparisons: selective serotonin-reuptake inhibitors (SSRI) vs. placebo, tricyclic antidepressants (TCA) vs. placebo, and SSRI vs. TCA. Detailed information on statistical analysis of the regarding systematic review is provided in previous publications.[14,17]

Results

Fifty primary studies reported in 111 publications were included in the analysis. A total number of 358 full-text articles were screened for eligibility (Figure 1). Forty-two (84.0%) of the 50 (100%) included studies could be identified by the sensitive electronic database search and eight (16.0%) by additional search strategies (Table 3).

Table 3.

Efficiency and contribution of the applied search strategies.

Applied search strategy	Potentially relevant studies identified (b)	Studies included (a)	Efficiency (a/b), %	Contribution (a/c), %
Electronic database search	276	42	15.2	84.0
Additional search strategies	82	8	9.8	16.0
Hand-searching contents of relevant journals	33	2	6.1	4.0
Forward citation tracking	7	0	0.0	0.0
Backward citation tracking	15	0	0.0	0.0
Screening reference lists of related systematic reviews	16	5	31.3	10.0
Searching clinical trial registers	0	0	0.0	0.0
Contacting first authors of included studies	11	1	9.1	2.0
Total	358	50 (c)

Within the electronic database search a detailed screening of titles and abstracts of 2417 records lead to a total number of 276 potentially relevant studies that were screened for eligibility (Figure 1). A subsequent examination of all full texts led to the inclusion of 42 (84.0%) studies. The efficiency of this search strategy amounted to 15.2% (42/276), indicating that 15.2% of all screened full texts could be included in the systematic review (Table 1). The contribution of the electronic database search adds up to 84.0% (42/50) (Table 3).

Results of the additional search strategies

Within all applied additional search strategies a total number of 27,007 references were identified and screened for potentially relevant studies. The detailed screening lead to 82 potentially relevant studies that were checked for eligibility (Figure 1). From the total number of 27,007 references identified through additional search strategies screening contents of relevant journals provided 19,076 references, forward citation tracking 2979, backward citation tracking 1692 and screening reference lists of related systematic reviews 1191 additional references that needed to be screened for relevance. Searching clinical trial registers and contacting first authors pro vided 2053 and 16 further references (Figure 1).

Efficiency

Out of all full-text articles identified through additional search strategies, 9.8% (8/82) could be included in our systematic review. Among the full-texts assessed through hand-searching contents of relevant journals and those identified through forward and backward citation tracking 6.1% (2/33) and 0.0% [(0/7); (0/15)], respectively, were included in our systematic review (Table 3). Through screening reference lists of related systematic reviews 31.3% (5/16) of all included studies could be identified. Searching clinical trial registers, did not lead to any further study inclusion, indicating an efficiency of 0.0% (0/2). One additional study identified by contacting all first authors of the previously included studies was found to be eligible for our systematic review through screening, which corresponds to an efficiency of 9.1% (1/11) (Table 3).

Contribution

Four per cent (2/50) of the included studies were obtained through hand-searching contents of relevant journals, whereas forward and backward citation tracking did not contribute to the total amount of included studies [0.0% (0/50)]. Screening reference lists of related systematic reviews had the highest contribution to the total amount of included studies [10.0% (5/50)]. Searching clinical trial registers provided no contribution to the total amount of included studies [0.0% (0/50)], but through contacting all first authors of the already included studies, another 2.0% (1/50) could be identified (Table 3).

Evaluation of the methodological quality of the studies

The global risk of bias was evaluated as low for a total number 11 studies, as unclear for 26 studies and 13 studies were judged to have a high risk of bias. Regarding the eight studies resulting from additional search strategies, four were judged to have a high risk of bias and the remaining four studies were evaluated to have an unclear overall risk of bias. Concerning the 42 studies that were identified through the electronic database search nine were evaluated to have a high risk of bias and a total number of 22 studies to have an unclear overall risk of bias. The remaining 11 studies resulting from the electronic database search were judged to have a low risk of bias. The results of the methodological quality assessment are presented for individual studies in Table 2.

Impact of the studies identified through additional search strategies

When comparing the results of the meta-analyses for the three comparisons (SSRI vs. placebo; TCA vs. placebo and SSRI vs. TCA) we found no systematic differences regarding the response and dropout rates after exclusion of the additionally identified studies. In the sequence of the above defined comparisons, response rates altered from 1.49 (1.29-1.99) to 1.49 (1.29-1.72), 1.74 (1.50-2.02) to 1.75 (1.48-2.07), and 1.01 (0.84-1.21) to 0.99 (0.83-1.17) after exclusion of the additional studies and dropout rates changed from 0.81 (0.49-1.33) to 0.83 (0.49-1.39), 1.14 (0.74-1.78) to 1.12 (0.79-1.59), and 0.41 (0.19-0.86) to 0.53 (0.29-0.96), respectively. For none of the comparisons statistical significance was influenced by the exclusion of the additional identified studies.

Discussion

In summary, eight (16%) of a total number of 50 included studies were identified through additional search strategies. Within all additional search strategies the number of titles and abstracts screened for potentially relevant studies amounted to 27,007 in total. Depending on the previous experience of a researcher and his/her knowledge of the specific research field one may estimate the average time needed to screen a single title to calculate the overall work effort for the conduction of additional search strategies. Thus, identifying these studies through performing additional search strategies subsequent to a sensitive electronic database search required a high degree of time and human resources with at least two researchers for each search strategy. In addition, regarding the global judgement of the methodological quality of all studies that were included in our systematic review, half of the studies [4/8 (50%)] identified through additional search strategies were judged to have a high risk of bias and the remaining four studies were evaluated to be unclear. Furthermore, the exclusion of the additionally identified studies did not affect overall results of our meta-analyses on the effectiveness on different antidepressants for chronic depression.

In a retrospective qualitative examination of the studies that were not identified within the electronic database search, we found that a part of these studies were indexed correctly, but were nevertheless not identified through the first screening. This finding indicates the importance of sufficient training and knowledge of researchers conducting an electronic database search. If resources allow for, various additional search strategies are efficacious for identifying relevant trials in conducting systematic reviews of interventions,[2,4,5] such as screening reference lists of related systematic reviews, hand-searching contents of relevant journals,[2] as well as searching clinical trial registers.[2,4,9] In our case study screening reference lists of related systematic reviews yielded the largest amount of identified studies (5/8) that were not identified by applying a sensitive electronic database search. This finding is in accordance with previous research.[9] In contrast, our results concerning the benefit of specific additional search strategies partially differ from what was reported in previous research. Other authors who examined the same additional search strategies next to an electronic database search as we did, recommend searching clinical trial registers and screening reference lists of related systematic reviews.[9] In our case study searching clinical trial registers did contribute nothing to the total amount of studies included in our systematic review. One reason for this differing result may be that we considered the electronic resource CENTRAL as an electronic database whereas previous studies considered it as a clinical trial register. Another reason for the lack of contribution of clinical trial registers may result from the sequence of the applied additional search strategies. The search in clinical trial registers such as ClinicalTrials.gov was performed as a fourth step within all applied additional search strategies subsequent to hand-searching contents of relevant journals, citation tracking and screening reference lists of related systematic reviews. Thus, our results concerning the search of clinic trial registers may be influenced by the classification of CENTRAL as an electronic database rather than a clinical trial register and the sequence of the applied additional search strategies.

Furthermore, within our investigation hand-searching contents of relevant journals led to the inclusion of two additional studies, but considering invested time resources this search strategy was the most time-consuming additional search strategy, by far, so that the benefit of applying this search strategy is questionable.

To design an optimal search strategy for identifying relevant trials in the conduction of systematic reviews of interventions and in case of limited resources, a sensitive electronic database search or just searching the database CENTRAL that is specialized on clinical trials is recommended as first choice.[2,4,9] These findings correspond with our results, since the greatest amount of included studies in our systematic review could be identified through the sensitive electronic database search, although we did not calculate the retrieval of searching CENTRAL separately. Our finding, that the exclusion of the additionally identified studies did not affect overall results of our metaanalyses on the effectiveness on different antidepressants for chronic depression is in accordance with previous research that indicate missing underpowered studies does not compromise the results of a systematic review significantly.[18] As the additional identified studies were further more often rated to have a high risk of bias than the studies identified through electronic database search it might especially in case of limited resources be reasonable to rely on electronic database searches when no resources are available to adequately assess and control for the methodological quality of the identified studies. As previous studies have shown that it might be sufficient to include at least two adequately powered studies,[19] one can argue that additional search strategies might not be necessary in any case.

There is evidence that instead of an a priori decision on which search strategies to use, the application of saturation criteria and corresponding stopping rules may be a valuable option to guide the search in a flexible manner.[20,21] One further approach focuses on the acceleration of conducting systematic reviews through the advancement of the clinical trial register, ClinicalTrials.gov.[22] According to this approach, ClinicalTrials.gov constitutes a convenient resource for finding results of clinical trials but does not allow the download of relevant data in a format that can be immediately applied for quantitative analyses. This problem could be solved by developing a system that provides an interface to retrieve study results in a usable format that is ready for analysis. The realization of this approach could crucially contribute to the acceleration of conducting systematic and rapid reviews because the need for manual data extraction would be reduced.[22]

Limitations

Since our investigation is a single case study of a systematic review on interventions of chronic depression, the generalizability of our findings to other research topics may be limited. Above, the presented search strategies were applied in the research area of therapies that focus on chronic depression so that we could draw on a relatively large number of existing systematic reviews, 11 in total, that were rewarding for our analysis. Accordingly, in areas less researched, this beneficial additional search strategy cannot be applied.[23] Furthermore, we conducted a sensitive electronic database search which might have limited the gain of additional search strategies. In contrast, the use of only one database (e.g., MEDLINE) and a more restrictive search could have resulted in a significantly larger benefit through additional search strategies.

Another possible limitation of our findings may consist in the choice of the journals that were searched for relevant content since this selection may have influenced the number of identified studies through this strategy as well as the relative efficiency of other strategies conducted subsequently.

In addition, we did not consult an expert, e.g. a librarian, for designing the terms for the electronic database search, so we may have missed a few studies. However, it should be noted that several of the authors have extensive training and experience in performing systematic reviews and searches within them.

Above, the contribution of any additional search strategy strongly depends on the order of its application. In this case contents of relevant journals were searched at first. As we did not investigate any alternative sequence of additional search strategies and thus did not calculate the overlap between the applied additional search strategies, the generalizability of our findings may be limited. Although we applied many additional search strategies, the search for grey literature was limited. As the inclusion of unpublished trials in systematic reviews is a controversially discussed issue that is capable of both reducing and introducing bias, we decided not to search for additional unpublished trials.[1,24]

Conclusions

Implications for practice

We found that the performance of additional search strategies, especially screening reference lists of related systematic reviews and hand-searching contents of relevant journals, uncovers a substantial amount of relevant RCTs in order to conduct systematic reviews of interventions in the context of chronic depression, even if a sensitive electronic database search is performed. Thus, database searches alone may miss a crucial amount (in this case 16%) of relevant data. On the other hand we found no effect of the additional identified studies on the overall results of our meta-analyses on the effectiveness on different antidepressants for chronic depression. Further, the additional identified studies were more often judged to have a high risk of bias. Thus, when planning a search strategy for a systematic review costs and benefits of additional search strategies need to be carefully weighted depending on specific requirements of the review that is to be undertaken, such as the relevance of identifying preferably all studies or the availability of resources for dealing with methodological weak studies.

Based on our findings, since a great amount of relevant trials can be identified through conducting a sensitive electronic database search, including a search in the database CENTRAL, this search strategy may be the first choice if resources are limited. Since in controversial research fields where the evaluation of a certain treatment strategy may depend on singly studies it is of high relevance to find as much evidence as possible. In this case, we recommend additional search strategies to ensure the completeness of existing evidence. Based on our case study results’ we recommend screening reference lists of related systematic reviews as well as contacting experts as additional search strategies. If resources allow for, hand-searching contents of relevant journals can be applied. Since screening reference lists of related systematic reviews is not exceedingly time-consuming, it is absolutely worthwhile to apply this search strategy to identify relevant trials, even if resources are limited and a pool of potentially relevant systematic reviews can be easily accessed.

Additional search strategies such as hand-searching may be necessary for the conduction of systematic reviews of interventions, especially in research areas with conflicting evidence, because automated electronic literature retrieval systems are likely to be prone to error, either due to technical reasons, due to database content limitations or due to users’ misapplication.

Implications for research

Because the use of a restricted electronic database search in comparison to a sensitive one, as performed in our approach, increases the risk of missing relevant literature right at the beginning of the search,[4] further research is needed to estimate the benefits of additional search strategies after conducting a more restricted electronic database research. For example, more information is needed on sequence effects when performing different additional search strategies. This could provide crucial information about optimal search strategies in case of limited resources, as required within the field of rapid reviews. Above, to evaluate the benefit of additional search strategies, further research is needed to estimate the relative importance of trials according to their source of identification. The question, therefore, of whether trials reported in journals that are not indexed by electronic databases such as MEDLINE or EMBASE are of smaller power and thus less important for the results of rapid reviews needs to be addressed. These aspects can provide relevant information for the development of methodological standards in the process of conducting rapid reviews.