Commentary to Gusenbauer and Haddaway 2020: Evaluating retrieval qualities of Google Scholar and PubMed.

We read with considerable interest the study by Gusenbauer and Haddaway (Gusenbauer and Haddaway, 2020, Research Synthesis Methods, doi:10.1002/jrsm.1378) comparing the systematic search qualities of 28 search systems, including Google Scholar (GS) and PubMed. Google Scholar and PubMed are the two most popular free academic search tools in biology and chemistry, with GS being the number one search tool in the world. Those academics using GS as their principal system for literature searches may be unaware of research which enumerates five critical features for scientific literature tools that greatly influenced Gusenbauer's 2020 study. Using this list as the framework for a targeted comparison between just GS and PubMed, we found stark differences which overwhelmingly favored PubMed. In this comment, we show that by comparing the characteristics of the two search tools, features that are particularly useful in one search tool, but are missing in the other, are strikingly spotlighted. One especially popular feature that ubiquitously appears in GS, but not in PubMed, is the forward citation search found under every citation as a clickable Cited by N link. We seek to improve the PubMed search experience using two approaches. First, we request that PubMed add Cited by N links, making them as omnipresent as the GS links. Second, we created an open-source command-line tool, pmidcite, which is used alongside PubMed to give information to researchers to help with the choice of the next paper to examine, analogous to how GS's Cited by N links help to guide users. Find pmidcite at https://github.com/dvklopfenstein/pmidcite.

What is already known?

Google Scholar is the most popular search system in the world.

Until Gusenbauer and Haddaway's 2020 paper, there was not a wide‐ranging, detailed study plus explicit advice for choosing a search system appropriate for systematic searches.

What is new?

A method to augment the list of citations returned from a PubMed query with the citation count and scientific influence data for each citation provided by NIH's Open Citation Collection (iCite).

A targeted comparison of Google Scholar and PubMed using the five search criteria recommended by Boeker with Google Scholar's documentation providing a description of their search support.

Potential impact for RSM readers outside the authors' field:

Draw attention to Gusenbauer and Haddaway's 27 search criteria, which is based on Boeker's five search criteria, and Gusenbauer and Haddaway's evaluation of 28 search tools so that a researcher may re‐evaluate their own specialized search systems and search methodology.

Make searches more effective and faster by informing researcher's how Boeker's five search criteria for evaluating search systems was applied using the example of PubMed vs Google Scholar.

INTRODUCTION

Modern literature reviews are primarily performed using online search engines. , The two most popular free academic search tools that are commonly used in health studies are PubMed and Google Scholar (GS). Researchers worldwide are drawn to GS as the most common starting point for literature searches , , , because of its intuitive and familiar search interface, , a forward citation search Cited by N link under every document result, a Cite link to download a document's citation to bibliographic management software such as EndNote for every document result, high citation counts, immense literature coverage, and researcher profile pages. GS's massive citation count, reflected in the “N” in their Cited by N links is due to their highly effective web crawlers and to agreements with publishing houses (Data S3, Figure 13).

But the GS search interface has severe deficiencies that make literature searches laborious and, most importantly, unreproducible. However, many researchers are unaware of the drawbacks of GS. For example, search results for a given query are dropped from one month to another, , , , with no documentation as to what has been dropped. Additionally, there is no way to download full search results in bulk, , resulting in the need to click and click and click to page through up to a maximum of 1000 search results, 10 or 20 results at a time (Data S3, Figure 12). And there is no direct access in GS to a paper's digital object identifier (DOI), which is a unique standardized persistent identifier.

It is important to call out the features and shortcomings of both PubMed and GS following two recent events. First, the National Institutes of Health (NIH) published a paper on October 11, 2019 announcing the NIH Open Citation Collection (NIH‐OCC), a free public citation database with citation data available for download in bulk. Citation records in the NIH‐OCC database are accessible through a set of web and Application Programming Interface (API) tools, collectively called “iCite.” Second, on November 18, 2019 the U.S. National Library of Medicine (NLM) announced that the new PubMed , was available. Highlights of the new PubMed include: a nimble mobile experience from a single responsive website for all screen sizes including mobile phones, tablets, and desktop computers; faster and more comprehensive search response; and advanced search features that GS simply lacks.

While we argue that PubMed is superior to GS in many ways, there is room to improve the literature search user experience in PubMed. We compare the “forward citation search” implementation in GS to that of PubMed, finding that the PubMed user experience can be improved by adding a GS feature to the PubMed Graphical User Interface (GUI). Alternatively, command‐line users can immediately augment their PubMed search results using the pmidcite scripts and library, which download citation data from the NIH‐OCC database using NIH's “iCite” API.

SCIENTIFIC SEARCH INTERFACE REQUIREMENTS

Many researchers are unaware that there is more than one type of search, , , , with each search type oriented to different user goals. All search tools are not appropriate for all search types. Three types of search include lookup tasks, exploratory search, and systematic search.

Lookup tasks are the most basic kind of search, usually involving a single query to obtain a well‐defined result. An example of a lookup task is searching for a specific paper by entering its title into the query box. GS excels at returning papers when provided with their title, even if there are errors in the query title text. Fellow researchers have complained that PubMed will sometimes not find a title if it is spelled incorrectly. Gehanno et al found that 100% of the 738 papers in their study were found using GS to search for each paper by entering its title into the query box. From this, they concluded that GS could be used in systematic reviews. This conclusion was quickly disputed by Giustini and Boulos whose paper is titled, “Google Scholar Is Not Enough to Be Used Alone for Systematic Reviews.”

Exploratory search and systematic search are useful in evidence synthesis. The goal of exploratory search is the acquisition of new knowledge and is considered to be demanding and potentially time‐consuming for the researcher. A researcher doing an exploratory search uses a number of queries to iteratively learn about a subject. The queries begin with a rudimentary understanding of the subject matter and become honed as the researcher's knowledge increases through the search process. Search tools like Google are frequently used in exploratory searches because they are made to be “user friendly” to increase user engagement, which benefits Google by making their market bigger. Google, with their user‐friendly interface nearly always returns search results, but the search results that are missing can not be known. Additionally, GS is not designed for systematic searches where researchers need control over the selection power of the query results.

Systematic search is profoundly different than exploratory search. The goal of systematic search is to catalyze an objective account of the cumulative state of evidence for a specific research question. An example research question is “What is the best treatment for lupus nephritis that was classified as stage IV on a renal biopsy ?” A well‐founded question addresses a clinical need where there is uncertainty regarding the effects of different interventions, which may vary in practice. The goal is to understand the costs and benefits of various treatments, so that together the doctor and patient can make an appropriate choice for their particular situation. Systematic reviews are an exacting evidence syntheses featuring numerous rigorous steps documented in method guidelines with the goal of providing an exhaustive synthesis of a well‐studied area of research.

Cochrane is one of the organizations that participate in systematic reviews. Steps in a Cochrane systematic review include creating the research question, building a team of people that includes those that have previously done a systematic review, writing or updating a protocol for the review, and having the protocol reviewed. Only after those steps, the systematic search using search tools begins by attempting to find all published and unpublished literature that may answer the research question. First and second authors work independently to remove irrelevant results and upon completion, compare their findings. Many other steps occur, which are all reviewed, under the umbrella of data synthesis and specialized plots, data interpretation, and data presentation. Finally, the review is written. To learn more, we recommend researchers read “How to write a Cochrane systematic review.”

High quality literature searches, both systematic and exploratory, are one of the important elements required for the creation of sound scientific evidence. In late October 2013, Boeker et al recommended that a scientific search interface contain five integrated search criteria. The 2013 Boeker guidance greatly influenced the Gusenbauer study, which expanded the Boeker list from five search criteria to 27 for their study of 28 search tools. The requirements for search interfaces are mandatory not only for structured scientific literature retrieval like systematic reviews, but also in any research that needs to provide a comprehensive literature review. We add “Forward citation search” to the Boeker list to evaluate the extremely popular GS implementation of this feature against the PubMed implementation and compare PubMed and GS's support for the search tools below using the 2013 foundational Boeker advice :

Reproducible search: A reproducible search is a critical quality measure of a systematic review in a well‐documented search process that allows others to replicate or update a published synthesis search. To be a reproducible search means that given a search, the same query returns the same results plus new results.

Export: Users should be able to export search results in full.

Search history: Histories are needed to create incremental search changes, which are used to selectively focus the search results.

Search strategy documentation: Documentation that instructs researchers how to create original search queries and how to iteratively develop new queries that build upon previous searches.

Search string builder. These include the use of numerous fields, such as author, title, journal, date, and abstract, and clinical query filters for categories including therapy and diagnosis.

Forward citation search. Tools that allow researchers to follow the chain of citing papers.

These six criteria synchronize well with other pertinent principals like the “The FAIR Guiding Principles for scientific data management and stewardship,” which emphasizes automating the discovery of researchers' work through software algorithms by applying a succinct and measurable set of principles to make the work FAIR—Findable, Accessible, Interoperable, Reusable.

The availability of fundamental search elements in the search interfaces of both PubMed and GS is summarized in Figure 1, showing that PubMed's search interface fully implements the five recommended Boeker et al search elements, while GS does not. However, GS's implementation of the forward citation search is much more popular than PubMed's implementation due to its heavy use of the Cited by N link.

FIGURE 1

Scientific search interface requirements. PubMed fully implement's Boeker et al's required list of characteristics for systematic search interfaces, while GS's implementation provides minimal support. But PubMed does not implement the extremely popular Cited by N links seen throughout Google Scholar

The next sections contrast PubMed and GS for each of the search requirements. When the GS documentation describes how they support the search interface requirements, it is featured in text boxes. Screen shots were taken of all GS documentation featured in this commentary (Data S3).

Reproducibility of search results

Repeating search queries in PubMed always produced the same previous content in the results, plus the expected steadily rising hits resulting from the increasing coverage of the database over time. But when a query is run month to month in GS, numerous researchers have observed sudden jumps, both rising, and falling, with large numbers of previous results lost. , ,

Search results can be exported in full

Search results in PubMed, even those not displayed on the screen, may be exported in full up to a maximum of 10 000 results. PubMed export formats include short summaries, files for import to citation software (like EndNote), PMIDs, abstracts, or a comma separated values (csv) file containing a list of data.

Searches in GS are limited to 1000 results maximum and cannot be exported in bulk (Box 1), as described in their help documentation (Data S3, Figure 1):

Can I see more than 1000 search results?

Sorry, we can only show up to 1000 results for any particular search query.

Try a different query to get more results.

How do I get bulk access to records in Google Scholar?

Sorry, we are unable to provide bulk access.

Researchers who write a script to download GS search results programmatically, quickly discover the downloaded results are halted (Box 2) upon reaching an unspecified limit and then find this on the help page (Data S3, Figure 1):

I wrote a program to download lots of search results, but you blocked my computer from accessing Google Scholar. Can you raise the limit?

Err, no, please respect our robots.txt when you access Google Scholar using automated software. As the wearers of crawler's shoes and webmaster's hat, we cannot recommend adherence to web standards highly enough.

GS explicitly states in their documentation that they will return a maximum of 1000 results for any search query (Box 1). The number of search results appears at the top of the list as “N result” in PubMed and “About N results” in GS. If “N” is less than 1000 results in GS, researchers may think they can copy and paste all “N” results, 20 citations at a time, by clicking, and clicking and clicking, advancing slowly through the search results. But researchers may be surprised to find that even if “N” is less than 1000 in GS, some of the “N” results may be missing.

PubMed can display 10, 20, 50, 100, or 200 results at a time on one page. GS can display 10 or 20 results. Clicking “Show more” in PubMed causes another set of results to be appended to the current results on the screen. The previous result sets are visible by scrolling up or pressing the browsers back button, which will cause the view to move to the previous divider and will not cause any results to disappear.

The “best match” relevance sort ordering in PubMed, described in a recent freely available peer‐reviewed research article, uses a modern machine learning algorithm that is trained with aggregated user searches. The “best match” algorithm uses dozens of features to sort a list of citations, but its developers find that the most important document features are publication year and past usage. Additionally, recently published papers are given extra elevation in the sort list so that they will not be missed.

Users can find GS search algorithm components described in a variety of locations including a 1999 “technical report” on the website for the Stanford Digital Library Technologies Project which ended in 2004, major updates as reported in a 2011 New York Times “Week in Review” piece, and numerous Google patents.

GS favors highly cited papers and ranks them at the top of the sort list so recent papers are more likely to be many pages back, making them harder to find. It is important to understand GS's sort practices to be able to estimate which results over the 1000 maximum were excluded.

Search history

PubMed records a history of every user search query and that user history is available as an interactive list where previous queries can be chained together and individual queries can be deleted to simplify the list. The full sequence of queries can be downloaded. GS has no similar search history.

Search strategy documentation

In addition to a comprehensive user guide, PubMed provides training in the form of tutorials, online training modules, quick tours, classes, and handouts. For further support, PubMed allows users to enter a question by clicking on the feedback link always shown at the bottom right corner of the page to bring up a contact form. A real person usually responds within the next business day or two. PubMed plans to move the feedback link to a “Contact Us” link located at the bottom of each web page now that “The New PubMed” is now the default link.

To access GS's contact form, click on questions like these (Box 3, Data S3, Figures 2 and 3):

I have noticed an error in a court opinion you are providing. What I can do to help fix it?

How do I remove a “Cached” (or “View as HTML”) link from your search results?

Search string builder

The link to PubMed's advanced search is immediately below the main search query box, making access straightforward, and efficient. The PubMed advanced search builder guides the user in building queries using more than 30 search fields, Boolean expressions (formed with AND, OR and NOT), and linking previous queries from the history. Additionally, users can customize the query entered in the query box.

The Boolean operators AND, OR, and NOT are required as it has been found that ranked retrieval alone, such as that found in GS, is not sufficient for a systematic search requiring high recall. High recall ensures that all the expected matches appear in the search results. These features give researchers the ability to fine‐tune which results are included and which are not.

GS's advanced search only offers access to three fields, “authored,” “published,” and “dated,” compared with PubMed's 30 search fields. There is no support for full Boolean search, and no ability to string together previous queries. The link to the GS advanced search documentation is still described as being located immediately to the right of the main GS search box (Data S3, Figure 6). But the link has been moved away from the right of the main search box to under a menu icon on the upper left‐hand corner of the GS web page (Data S3, Figure 7).

Forward citation search

PubMed has a forward citation search which can be accessed by opening the PubMed page for a single chosen article (Data S1, Figure 1). If the paper has citations, scrolling to the bottom of the page will show a “Cited by” section (Data S1, Figure 2, red 3) which lists the total number of citing papers in the section header and shows the first few papers in the section body. The full list of citing papers may be downloaded from PubMed in a variety of formats, including text or comma separated values (csv), by clicking the See all cited by articles link (Data S1, Figure 2, red 3a) and pressing the “Save” button (Data S1, Figure 5). But the web page showing the list of citing papers contains no citation count information for any articles on the page (Data S1, Figure 5). To see the citation count of each of the citing papers, the researcher must click on each citing paper one by one to open the individual paper's web page and scroll down to that paper's “Cited By” section, making choosing the next paper to explore a slow and laborious process (Data S1, Figures 1 and 2). We would like to see Cited by N links ubiquitously featured on all citations in a list (Data S1, Figure 6, red boxes). We rate the Forward citation search feature as “Good” rather than “Better” (Figure 1) because the Cited by N links do not appear (Data S1, Figure 5) in PubMed.

In GS, clicking the Cited by N link of a specific paper will open a web page with a list of papers citing the specific paper (Data S1, Figures 3 and 4). Each paper in the list has a Cited by N link (Data S1, Figure 4, red boxes), making it easier to compare the citing papers appearing in the list (Data S1, Figure 4, boxed in red). Unlike PubMed, there is no way to download the list of all citing papers in bulk. We rate this feature as “Better,” even though it is not possible to compare all search results in a single view, because of the usefulness, and popularity of the GS Cited by N link.

PubMed is missing the Cited by N link on each paper in a list papers which is prominently featured in GS (Data S1, Figure 6), causing researchers to be lured toward GS and away from PubMed despite a grueling literature search experience in GS.

Scientific search feature summary

The advanced features recommended in 2013 for an effective, exhaustive, and reproducible systematic review are fully implemented in PubMed, but was not implemented by GS in 2013, when Boeker did his study, and remains not implemented in 2020. , ,

And some GS features have made the search process more onerous. In 2008, GS search results could be displayed with 10 to 300 items per page. Today, it is restricted to either 10 or 20 items per page (Data S3, Figure 12). Featuring search results at a maximum of 20 per page rather than 300 per page makes literature search more time‐consuming, labor‐intensive, and reduces a researcher's ability to visualize the search results as a whole. In 2013, Boeker concluded that GS was not ready as a searching tool for tasks where structured retrieval methodology is compulsory. Almost a decade later, GS, still can not be considered for such tasks.

COVERAGE OF PUBMED AND GOOGLE SCHOLAR

The coverage of PubMed

PubMed is a search interface and toolset used to access databases like MEDLINE and PubMed Central (PMC) as well as additional content like books and articles published before the 1960s. Over 30.5 million article records are accessible through the PubMed interface (Figure 2). The databases, MEDLINE and PMC, are separate entities whose combined articles comprise 94% of all of the coverage indexed by PubMed (Data S2). MEDLINE is a highly selective database started in the 1960s. PMC, started in 2000, is an open‐access database for full‐text papers that are free of cost to the reader.

FIGURE 2

Most of the coverage of PubMed is indexed in the MEDLINE database and the PMC database. The coverage of PubMed is shown on the horizontal axis. The top two bars on the vertical axis show two overlapping databases indexed by PubMed. The bottom orange bar indicates PubMed citations not found in the two major databases. About 88.5% of the ∼30.5 million citations accessible through PubMed are in the MEDLINE database (top blue bars) or are about to be added to (top green bars) the MEDLINE database (top blue and green bars). The MEDLINE papers that are free full‐text and are also indexed in the PMC database (middle blue and green bars) comprise over 68% of papers indexed in PMC. About 5.5% of PubMed papers are only available in PMC (middle brown bar). Almost all of the remaining 6% of full‐text papers (bottom orange bar) are behind a paywall. We queried for and downloaded PubMed count data and created the figure with a script available in pmidcite [Colour figure can be viewed at wileyonlinelibrary.com]

The coverage of Google Scholar

While the coverage of GS is not known, it is estimated to exceed all other currently available search systems since GS aims to index all of scholarly information that is electronically available. This is a principal reason for its standard‐setting citation index, which is used to replace “N” with a number in GS's forward citation search via Cited by N. The size and scope of GS remains unknown despite having been the subject of sizable research efforts since its creation. ,

Journals covered

The GS documentation instructs researchers who want to know if a specific journal is covered to choose a “statistical sample” of articles published by the journal and search for each paper using its title in the search box (Box 4, Data S3, Figure 4):

Which specific journals do you cover?

Ahem, we index papers, not journals. You should also ask about our coverage of universities, research groups, proteins, seminal breakthroughs, and other dimensions that are of interest to users. All such questions are best answered by searching for a statistical sample of papers that has the property of interest – journal, author, protein, etc. Many coverage comparisons are available if you search for [allintitle:”Google Scholar”], but some of them are more statistically valid than others.

In contrast to the GS approach, researchers can download PubMed's complete list of journals currently indexed in MEDLINE and deposited in PMC by following the journals link found on PubMed's home page. If a journal is on MEDLINE's approved journals list, papers are automatically indexed by PubMed (Data S2).

Indexing procedure for individual manuscripts

If an individual author manuscript was accepted into a journal that is not on MEDLINE's approved journal list, the requirements to submit the manuscript for deposit into PMC and indexing by PubMed are as follows. The work must be funded by an approved agency, peer‐reviewed, accepted into a journal, and free to access electronically. If these criteria are met, the paper may be submitted to the NIH Manuscript Submission system (NIHMS) for potential indexing in PMC after the paper has been successfully vetted in NIHMS.

In GS, the requirements are the article must be contained in a pdf file whose contents include a title, list of authors, and bibliography, and uploaded to a website. The affect of GS's regularly crawled data and loose indexing policies is that GS indexes records that are non‐academic. For example, the GS policies for “author manuscripts” has resulted in a number of lunch menus that are stored as a pdf file online to be indexed as scholarly citations in GS with various food items being listed as authors (Data S3, Figure 10 and 11).

Additionally, some researchers have demonstrated that it is possible to deliberately trick the crawler and inflate the GS citation score.

FORWARD CITATION SEARCH

The NIH Open Citation Collection (NIH‐OCC), is a free public citation database, which liberates researchers from the constraints of citation data that were previously locked behind barriers, such as citation lists which were not downloadable in bulk. Having full access to citation data could allow researchers to perform more efficient literature searches and analyze publishing trends in biomedicine.

The NIH citation database differs from GS's citation database in coverage, usability, and content. The coverage and content of GS is huge and covers many disciplines, while the coverage of the NIH citation database is limited currently to about 30.5 million manuscripts that were assigned a PubMed ID (PMID). The usability of iCite citations is extremely high because they are accessible for free through the NIH “iCite” web site and downloadable in bulk through the NIH Application Programming Interface (API). Citations can not be downloaded in bulk in GS.

The citation counts in GS will be higher because their index is massive. We have not experienced that the differing citation counts when using pmidcite vs GS are a hindrance during exploratory search tasks because the data needed to decide the next paper to investigate is how one paper performs relative to another. If all citation counts are scaled down in NIH's “iCite” compared to GS, we still can successfully compare the performance of papers relative to one another.

Additionally, in NIH's “iCite,” new papers are easy to find and compare, even if they have few citations. Having the data, even if it is scaled down compared to GS, to choose the next paper will speed the exploratory literature search faster than having all of the citations that are available in GS, but not available in other search systems. Once the researcher has become familiar with the subject through their exploratory literature search, then they may choose to use GS to see what might have been missed.

We have tested the practical usage of a Cited by N link by creating a set of command‐line interface (CLI) scripts and a Python library, called pmidcite, which glue PubMed search results and NIH's “iCite” citation data together using PMIDs to provide functionality that is equivalent to having the Cited by N link. The results were so successful that we hope PubMed can expand the access to all biomedical researchers, even if they do not use a CLI by adding the links, Cited by N and N References (Data S1, Figure 6), to the PubMed GUI as soon as possible.

NIH's iCite

PubMed does not have a clickable Cited by N link for all the citations, making it difficult to choose the next paper to investigate (Data S1, Figures 5 and 6). But equivalent functionality can be had if, for a selected paper, the researcher downloads from PubMed the full list of citing papers as a list of PMIDs (Data S1, Figure 7) and uploads this PMID list of citations to iCite for analysis (Data S1, Figure 8). The list can then be sorted in the “citations” Table (Data S1, Figure 9, red 2) by Total Citations by clicking on the Total Citations column header under the OpenCites Table (Data S1, Figure 9, red 3).

But comparing papers only using its number of citations is problematic because papers in small niche fields may get considerably less citations than papers in large fields because both papers may be of relatively equal scientific influence in their respective communities. The NIH normalizes the number of citations that a paper receives by comparing it to the citation numbers of papers in its co‐citation networks. This measurement is called the Relative Citation Ratio (RCR) and can be used to sort a list of PMIDs. Only citation count is offered by GS and it cannot be used for sorting search results.

pmidcite

Functionality equivalent to having a Cited by N link can be had from the command‐line shown in the following example using a selected paper with a PMID of 25 505 874 by typing “icite 25505874 – verbose” and pressing the “Enter” button. This causes citation counts to be downloaded from NIH's iCite and a report to be written to the screen or to a file. In the report, the number of citations is seen under the “cit” column for the user‐requested paper (“TOP”), the full list of it's citing papers (“CIT”) and references (“REF”):

The NIH values based on a paper's RCR are also available in pmidcite, but are not shown here. For more information regarding pmidcite and to see options for sorting citing papers from the CLI, see Data S1 and https://github.com/dvklopfenstein/pmidcite.

CONCLUSIONS

We hope to raise awareness that there are various types of search, including lookup tasks, exploratory search, and systematic search. Each search type requires unique search system features. GS is the system used as the starting point of most searches, rather than specialized tools like PubMed, by most researchers due to its intuitive, accessible interface, fast response, and best in class coverage. GS excels for simple lookup tasks, like finding a paper by entering its title in the query box.

Both GS and PubMed can be used for exploratory searches, but we urge biomedical researchers to use PubMed rather than GS, because PubMed is one of the top recommended primary sources for literature searches of peer‐reviewed research in the biomedical sciences and has search feature criteria that GS has lacked since its inception. CLI users, especially, should consider using PubMed with search results annotated using NIH's “iCite” citation data because this functionality is available immediately through pmidcite.

Searching using the PubMed interface is a satisfying experience, even without the addition of the Cited by N link. But we hope that PubMed will soon add a clickable citation count link to every document entry in the search results list and to each paper listed in the document page sections, similar articles, cited by, references, and suggested reading so PubMed GUI users can enjoy similar benefits as CLI users.

GS fails to implement the required search criteria for systematic searches and should not be used as a primary search tool for systematic reviews. However, GS can be used as a secondary source.

Finally, we urge researchers to read the Gusenbauer and Haddaway paper to see how their own specialized search tool is or can be evaluated among the 28 extensively used academic search systems in the Gusenbauer study.

CONFLICT OF INTEREST

The authors declare no potential conflict of interests.

AUTHOR CONTRIBUTIONS

D. V. Klopfenstein wrote the manuscript and created the literature search methodology using citation data from NIH's iCite. D. V. Klopfenstein architected, designed, and implemented pmidcite, which allows researchers to use the literature search methodology (Data S1). Will Dampier reviewed the manuscript and provided crucial analytical feedback. All authors reviewed the final manuscript.

Data S1. The Cited by N link, the N References link, and how to use pmidcite.

Click here for additional data file.

Data S2. PubMed Coverage.

Click here for additional data file.

Data S3. Screenshots of GS containing content used in text boxes.

Click here for additional data file.