Reviewing retrieved references for inclusion in systematic reviews using EndNote.

An important task in conducting a systematic review is reading titles and abstracts of the retrieved references, which often number in the thousands, to determine which articles meet the predefined inclusion criteria. In the past, this was performed by manually scanning through large stacks of printed titles and abstracts, followed by face-to-face meetings to discuss which references should be included. Today, the workflow of the review process is more streamlined by using computer software.

Several specialized solutions for this process are available, most notably free or subscription-based online tools such as Covidence, DistillerSR, or Rayyan. The Cochrane collaboration uses its own tool called ReviewManager. A survey in 2013 showed that more than half of all systematic reviewers used EndNote software [1]. This dominance on the market is likely to increase as sales of the second most popular tool, Reference Manager, have ceased, and its website now advises users to switch to EndNote. Many reviewers use Microsoft Excel. Some libraries have even created specialized Excel workbooks to document the process in much detail [2]. A method for the inclusion process using EndNote is described by King et al. [3], but this process is rather complicated and time-consuming.

This paper describes the logistics of a method to perform the title and abstract screening, verdict assignment, and comparisons of results among multiple reviewers in EndNote. The process is blinded; all reviewers work in their own EndNote files, and after the individual inclusion and exclusion processes, the verdicts of the different reviewers are compared. The method described here can be performed much faster than comparable methods.

THE METHOD

The method consists of several steps. First, a custom style should be installed for easy abstract scanning. Second, a field should be added to show the reviewer’s name in the Library window. Third, for each systematic review, custom groups are made in the EndNote library for included and excluded references. Reviewers drag articles to the group corresponding to their verdicts. In the last steps for the final comparison of the verdicts, the included references of all reviewers are combined into one EndNote library and de-duplicated. References found as duplicates are included by both reviewers and are selected for full-text review, the non-duplicate references can then be discussed for inclusion or exclusion.

The steps below are written for EndNote X7 for Windows. EndNote is also available in a version for Apple Mac computers, but some of the menu items will appear on different places. The authors have added footnotes that guide Mac users as much as possible.

Step 1: Install the custom-made output style

A custom-made style (named _preview) facilitates easier reviewing of the titles and abstracts.

Visit http://bit.ly/emcendnote

Open the zip file

Double-click on the file_preview.ens (it will open in EndNote)*

In EndNote, click File > Save As

Remove the text “copy” from the file name, and click Save†

Close the style using the cross in the top right corner‡

To activate the _preview style, open the drop-down menu (called Select Bibliographic Output Style) in the top left part of the screen (Figure 1)§

Figure 1

Activating the _preview style

Click on Select another style, scroll to the top, select _preview, and click Choose

The abstract of the selected reference will be displayed in the preview tab in the Tab pane

Step 2: Change the settings of the library window

When comparing the included references between reviewers, a special field will be used to document the name of the reviewer who included a certain reference. To be able to view this in the Library window, this field has to be added to the preference settings.

Go to Edit > Preferences > Display Fields

In column 8 (or an alternative field that is not often used), under Field, select Custom 4

In the same column, under Heading, type Reviewer

Step 3: Create groups for inclusion and exclusion in the EndNote library containing the search results

In the first reviewing round, two reviewers independently read titles and abstracts to decide whether a reference is potentially relevant to the review. We propose the creation of two group sets: Includes and Excludes, each with, at this stage, just one subgroup (Figure 2).

Figure 2

Groups for Includes and Excludes

Right-click on My Groups, and select Create Group Set

Type Excludes, and hit Enter

Right-click on Excludes, and select Create Group

Type title/abstract, and hit Enter

Repeat the process for Group Set Includes with Includes group

After the groups for inclusion and exclusion have been made, two copies of these files are created (adding the name of the reviewer to the file name) and distributed to the reviewers. All reviewers will work in their own copies of the EndNote file.

Step 4: Title/abstract screening

The standard group named Unfiled contains references not yet assigned to one of the other groups. When starting the screening phase, the number of references in Unfiled will be equal to that in All References. Each reviewer reviews the relevance of references in Unfiled one by one, based on the title and/or abstract.

Broaden the title field in the Library window (Figure 2) by dragging the column break between the columns of Title and Journal to the right until the Title column reaches an appropriate width

Review the titles one by one in the Library window until a potentially relevant title is reached, without yet assigning references to the groups

Click on the relevant title, and read the abstract in the preview tab of the Tab pane (Figure 2)

If the abstract is irrelevant, continue reading titles in the Library window

If the abstract is relevant, select the article directly above that relevant article in the Library window, and press Ctrl+Shift+Home to select all references above it. Drag all of these articles into the Excludes > Title/Abstract group

Next, drag the top reference in the Library window (which is the reference to be included, as irrelevant references were removed from Unfiled) to the Includes > Includes group

Repeat this process until all references are filed and the Unfiled group is emptied

Go to the folder Includes in the EndNote library of reviewer 1, and click on one of the references

Go to Tools > Change/Move/Copy Fields

In Custom 4, select Insert after field’s text, type the first name of reviewer 1, and click on OK in three pop-up screens

Open the EndNote library of reviewer 2 without closing that of reviewer 1

Go to the Includes group in the library of reviewer 2; select all references in that group, right click on one of them, select Copy reference to, and select the file screened by reviewer 1

Go to Window, and select the file by reviewer 1

Go to the Unfiled group, and mark the records in that group with the name of reviewer 2 (as described above in steps 2–3)

Drag the references from Unfiled into the Includes group

Check the settings for de-duplication (Edit > Preferences > Duplicates); at least Author, Year, Title, and Secondary Title (Journal) should be selected

Go to the group Includes, select a random reference, and go to References > Find Duplicates; in the detailed comparison screen, click Cancel; then press Delete on the keyboard to remove the duplicate references

Right click in the group set Includes, select Create a group, and name it Definite Includes

Select all references in the Duplicate References group, and drag those to the Definite Includes group

Go to the Definite Includes group, and mark the records in that group with the name of reviewer 2 (as described above in steps 2–3)

Right click on the Includes group > Includes, and select delete group

Have the two reviewers discuss the articles currently in the Unfiled group

After consensus is reached, drag the references one by one to the appropriate group until Unfiled is empty

Step 6: Full-text reviewing

In the second round of screening, full texts of the included titles and abstracts need to be reviewed. Custom groups can be used to distinguish between various reasons for exclusion, and articles can be assigned to specific groups for certain sub-questions. All reviewers should again work in their own copies of this library. After reading all articles, each reference in the library should be discussed in detail; therefore, no automatic comparison should be used. The steps in this round are more laborious, differ per research topic, and can hardly be generalized and optimized. Therefore, we do not describe in detail how this process can be executed.

DISCUSSION

We advise against the use of a “Doubt” or “Maybe” group for articles for which it is not yet clear whether they should be included. We recommend that in cases of doubt, the article should be added to the folder Includes. If the second reviewer also has doubts about the relevance or decided to include the article, the full text should be used for final judgment. If the second reviewer excludes the article, it is an item for discussion.

The process of reading titles and abstracts for inclusion and exclusion is often considered time consuming, and the number of abstracts that can be read per hour is estimated at 120 by the Cochrane Handbook [4]. A recent study recorded the time needed for certain steps in the systematic review process [5]. Upon request, the authors informed us that, using specialized screening software, the median number of articles that could be reviewed per hour was 68. Another recent study estimated the time needed to screen 1 record based on title and/or abstract at 1 minute [6]. A survey among review authors at Erasmus MC, to whom we had sent an earlier draft of this article, reported that the median number of titles and/or abstracts reviewed per hour with the present method was 308, with a maximum of 675.

We found that the speed of the process increases when reviewers do not document the specific reasons for excluding references during the title/abstract screening phase. It is often clear that an article is not relevant to the topic, but to determine the exact reason (or very often multiple reasons) is very time consuming and unnecessary. According to PRISMA guidelines, reasons for exclusion, with the number of articles for each reason, should be given only in the full-text screening phase [7]. We found that researchers who meticulously documented the reasons for exclusion in the first round and those who had used other software such as Microsoft Excel reported much slower rates (median of sixty minutes) and often later regretted their decision to do so. Also, using specialized programs or online systematic review management systems such as Covidence or DistillerSR unnecessarily complicate and delay the process because each abstract has to be assigned to categories individually, in contrast to this method’s bulk assignment of nonrelevant articles.