Reader engagement with medical content on Wikipedia.

Articles on Wikipedia about health and medicine are maintained by WikiProject Medicine (WPM), and are widely used by health professionals, students and others. We have compared these articles, and reader engagement with them, to other articles on Wikipedia. We found that WPM articles are longer, possess a greater density of external links, and are visited more often than other articles on Wikipedia. Readers of WPM articles are more likely to hover over and view footnotes than other readers, but are less likely to visit the hyperlinked sources in these footnotes. Our findings suggest that WPM readers appear to use links to external sources to verify and authorize Wikipedia content, rather than to examine the sources themselves.

Introduction

Wikipedia is a freely available online encyclopedia that intends to provide “every single person on the planet free access to the sum of all human knowledge” (Wikimedia Foundation, 2004). To meet this mission, thousands of volunteer editors have created almost six million English-language Wikipedia pages. Many of these pages include hyperlinked footnotes to the sources used to assemble and verify the content (Redi et al., 2018; Wikipedia Foundation, 2019a). These linked references do not only serve as sources of authority for Wikipedia content (Fallis, 2008), but also offer readers a gateway to further learning, with this opportunity enhanced by the growing degree of public access to research literature (Piwowar et al., 2018).

Wikipedia is proving to be a leading source of health information (Heilman and West, 2015; Laurent and Vickers, 2009), and reader engagement with a page’s references can provide opportunities to understand a diagnosis or inform a conversation with their physician. Wikipedia’s health-focused pages, which are maintained by WikiProject Medicine (WPM) editors, are thought to meet a high standard of quality and rigor (James, 2016; Maskalyk, 2014; Trevena, 2011). For example, WPM provides a series of resources for its editors, including “guidelines and policies” for reliable sources, and advice on how to avoid conflicts of interest. In its advice for reliable sources, WPM recommends using “review articles (especially systematic reviews) published in reputable medical journals” (Wikipedia Foundation, 2019b). Wikipedia then provides tools to standardize the bibliographic representation and linking to external sources.

These external sources are particularly relevant for students and practitioners in medicine and other health-related professions, who are active Wikipedia readers (Rössler et al., 2015; Scaffidi et al., 2017; Back et al., 2016; Egle et al., 2015; Allahwala et al., 2013). These readers are familiar with the research literature, and expected to engage in evidence-based medicine (Guyatt et al., 2015). To this end, faculty at several health professions schools, such as schools of medicine and pharmacy, teach courses on editing Wikipedia (Azzam et al., 2017; Joshi et al., 2019; Apollonio et al., 2018).

In a previous study, we investigated Wikipedia as a gateway to further reading through external links by tracking referrals from Wikipedia to research article DOIs (Digital Object Identifier) through Crossref, combined with pageview information from Wikimedia (Maggio et al., 2017). However, with the data available at the time, we could not determine if a click to an external reference with a DOI originated from a WPM page.

To learn more about reader engagement with Wikipedia’s external references, this study leverages Wikimedia’s newly created infrastructure for data collection to compare engagement with external links by readers of WPM pages with that of readers of the rest of Wikipedia. The study is guided by two research questions:

RQ 1: To what extent do WPM pages differ on average from the rest of Wikipedia pages?

RQ 2: To what extent do the behaviors of WPM readers with external links differ on average from readers’ behaviors with the rest of Wikipedia?

Methods

With the approval and support of the Wikimedia Foundation (WMF), we collected the data presented in this study between March 22nd – April 22nd, 2019 from Wikimedia’s Event Logging system and the production MediaWiki database. The data remained within that system, as required by WMF, for a year before being deleted (see Acknowledgements). We anonymized the aggregated data by removing IP addresses, any identifying browser information, and reader-sessions associated with page edits. While this data is not publicly available, data may be available upon request from the WMF research team. To gain access to the data, researchers should review the WMF Research team’s current procedures for data requests (https://www.mediawiki.org/wiki/wikimedia_research/research_and_data). The WMF Research Team will evaluate the request based on Wikimedia’s data access criteria. The code utilized to collect and analyze the data, however, is organized and made publicly available in a collated series of Jupyter notebooks in GitHub (Steinberg and Picardi, 2019; copy archived here).

Data collection

The data is drawn from English Wikipedia pages in the main namespace, a designation that contains the encyclopedia proper. Wikipedia pages or topics were identified as being part of two main groups: WikiProject Medicine pages (WPM) and the rest of Wikipedia (W). To be included in the study, the pages from each group had to have at least one external link in the externallinks table. The categorylinks table was used to define the WPM pages, with each possessing a Talk page bearing the category “All WikiProject Medicine articles.” Both the externallinks and categorylinks tables were queried twice (April 1st, 2019 and April 20th, 2019) during a 32 day study period (March 22nd – April 22nd, 2019).

For determining the number of pages, length of pages, the number of external links, and the number of “freely accessible” links added by editors as sources, a single day’s worth of database and XML dump files were captured from late in the study period (April 20th, 2019). As the database and XML dump files had only 0.5% more external links than on April 1st, 2019, the sample from April 20th was felt to be sufficiently representative to serve as the source for all static data counts. The external link count, which is based on MediaWiki’s externallinks table, does not include interwiki links, representing abbreviated forms of commonly-used internal and external links, which limits the accuracy of external link counts for both WPM and W. The event logging system this study relied on similarly omitted data from interwiki links, meaning the definition of an external link used across this study is consistent.

Pageview data was gathered from the wmf.pageview_hourly table. WMF employs methods to identify bot traffic in pageview data, which was excluded in our analysis. In reporting the data collected over a 32 day period (March 22nd – April 22nd, 2019), the raw counts were divided by 32 to create a count approximating a “daily average” for these counts, in light of this serving as a common measure of internet traffic.

Reader engagement with external links was gathered from Wikimedia’s Event Logging system using the CitationUsage schema, instrumented by Wikimedia’s programmers, following a month of piloting and refinement for this study. The CitationUsage schema collected all sessions with reader engagement, except for those involving anonymous Wikipedia editors (21 sessions out of 72,953,065 total, which translated into the removal of 34 citation events out of a total of 113,520,376). For the entire study period (March 22nd – April 22nd, 2019), the CitationUsage schema captured the following types of engagement: (a) clicking an external link; (b) clicking on a reference link listed among a page’s set of references; (c) clicking a footnote link leading to a reference on the page; (d) hovering over a reference link (defined as a reader’s cursor lingering over a link for 1000 milliseconds or more, revealing a rollover label); (e) time from pageview to event (Table 1). Additionally, the CitationUsage schema captured the clicking of links bearing the “freely accessible” icon.

Table 1.

Types of engagement with external links

Different events captured by the CitationUsage Schema that reflect how readers of Wikipedia pages engage with external links.

Engagement type	General description
External click	A click on a link located on a Wikipedia page leading to a web page outside of Wikipedia.
Hover	An event that occurs when a reader hovers over a link for at least 1000 milliseconds on a Wikipedia page.
Footnote click	A click of an internal footnote link – [1] – that takes the reader to the reference section at the bottom of the Wikipedia page.
Up click	A click of an arrow – ^ – that takes the reader from the reference at the bottom of the Wikipedia page back to the citation in the main text.

Analysis

Descriptive statistics were calculated using Excel (Redmond, WA). Inferential statistics were calculated to determine the significance in the difference between engagements in WPM and W using the Python library SciPy. In our analysis, we verified the statistical differences between WPM and W page for the following parameters: (a) time to first engagement with a link, (b) length of the articles, and (c) the number of page loads per event. For parameters (a) and (b) we used a normality test to determine the type of statistical test to apply. In both comparisons, after rejecting the normality hypothesis, we applied Mann-Whitney U-Test, a non-parametric test that does not require normality assumptions on the two distributions. For the third parameter (c), we compared the two groups over all possible combinations of access method and type of engagement with a two-tailed Fisher's exact test. To explore and visualize the data, we used respectively Spark/Pandas and the library Seaborn. The complete analysis is available in the project’s publicly accessible Github repository (Steinberg and Picardi, 2019).

Stakeholder engagement

This study was possible through collaboration with the WMF research team beginning in 2018, with our publishing of a research proposal to Wikimedia’s Meta-Wiki (Maggio et al., 2018). Members of the Wikipedia community, including members of WPM, were invited to provide feedback and pose questions. Over several months, we held teleconferences and in-person meetings with the WMF research team to understand Wikimedia’s infrastructure, parameters of data use (e.g., best practices for accessing the data securely; duration of data access), and the expectations of the Wikipedia community. Prior to data collection, a WMF research team member posted to Wikipedia’s Village Pump, a set of Wikipedia pages dedicated to discussing technical issues and policies, an announcement of the data collection (Redi, 2019). The announcement invited the Wikipedia community to post public comments and provided contact information for expressing concerns about the research.

Results

Wikipedia pages and external links

This study compares readers’ engagement with the pages curated by WikiProject Medicine (WPM) to their engagement with the rest of the English edition of Wikipedia (W). At the time of this study, WPM represented 34,324 pages (i.e., subject or topic entries), while the rest of W had 5,839,083 pages (Table 2). WPM pages possessed more than twice as many links to external references and other sites than the rest of Wikipedia. WPM pages were 13,084.9 characters in length on average (SD = 19,3780.4; median = 6,6280.0; IQR = 11,640), which was 70.1% longer than the average page in the rest of Wikipedia at 7,676.4 characters (SD = 13,6320.4; median = 3,8650.0; IQR = 5,789) (Figure 1; Table 6). WPM pages also demonstrated a greater “link density” with an external link appearing on pages with links every 450.3 characters, compared to a link every 657.3 characters for the rest of Wikipedia.

Table 2.

Factors that distinguish WPM pages from the rest of Wikipedia.

Differences between WPM pages and pages in the rest of Wikipedia (W) based on data collected on April 20th, 2019.

	WPM	W
Wikipedia pages	34,324	5,839,083
Pages with external links	32,609	5,210,746
External links	945,645	60,851,396
Links per page (with links)	29.0	11.7
Page length (characters)	13,084.9	7,676.4
Characters per link (pages with links)	450.3	657.3

Figure 1.

Page length of WikiProject Medicine (WPM) and other Wikipedia (W) pages.

Distribution of page lengths in characters for WikiProject Medicine (WPM) pages and the rest of Wikipedia (W) on April 20th, 2019. The difference between the two distributions is statistically significant according to Mann–Whitney U test (p<0.001, two-tailed).

The biomedical literature, including research reviews and studies, was a leading source of WPM external links. For example, the most common hostname was “www.ncbi.nlm.nih.gov,” accounting for 25.2% of the WPM’s external links. This indicates that research papers found through the PubMed database run by the National Institutes of Health (NIH) are likely to be cited. It was followed by the hostnames “www.worldcat.org” with 2.7%, and “www.google.com” with 1.2% of the external links. As for the hostnames that prevailed among external links in the rest of Wikipedia, the three leading hostnames were “tools.wmflabs.org” (where Wikimedia hosts tools developed to assist editors) at 3.5%, “www.google.com” at 2.9% and “books.google.com” at 2.0%, while “www.ncbi.nlm.nih.gov” still accounted for 1.8% of the external links outside of WPM.

Pageviews and events

Readers viewed 5,875,470.4 WPM pages a day on average, compared to 228,445,128.4 pages for readers of the rest of Wikipedia (Table 3). While the number of WPM pages is 0.6% of W pages, the number of pageviews that WPM received from readers was 2.6% of that for W. This suggests that WPM pages were viewed more than four times as frequently as the rest of Wikipedia. The majority of these views for both WPM and W took place on mobile devices, with WPM readers the heavier users of mobile devices in accessing Wikipedia.

Table 3.

Comparing pageviews between WikiProject Medicine (WPM) and other Wikipedia (W) pages.

The average number of pageviews WikiProject Medicine (WPM) and the rest of Wikipedia (W) received per day for different types of devices between March 22nd and April 22nd, 2019.

	WPM (%)	W (%)
Pageviews on desktop device	1,957,821.6 (33.3)	97,956,273.1 (42.9)
Pageviews on mobile device	3,917,648.8 (66.7)	130,488,855.3 (57.1)
Total number of pageviews	5,875,470.4 (100)	228,445,128.4 (100)

We also found that readers spent more time on a page before first clicking an external link after loading a WPM page. The WPM reader’s median time of 47.5 s (SD = 1.2; mean = 82.5; IQR = 147,306) was 44.8% longer than the median time of 32.8 s (SD = 5.6; mean = 39.1; IQR = 86,424) for W readers (Table 6). The difference between the two distributions is statistically significant according to Mann–Whitney U test (p<0.001, two-tailed).

Among the four types of events recorded, WPM readers were more likely to hover over a footnote or other link, especially on their desktop devices, and more likely to click on footnote links, compared to W readers (Table 4). On the other hand, W readers were more likely to click on external links than WPM readers, favoring their mobile devices in that regard. The WPM readers are interested, it appears, in seeing what evidence is being drawn upon in making the statements set out on a WPM page rather than trying to view the source compared to readers of the rest of Wikipedia.

Table 4.

Frequency of different types of link engagement per day.

The average number of times per day that readers of WPM and W pages engaged with external links using one of the event types captured from March 22nd to April 22nd, 2019.

Event type	WPM			W
	Total (%)	Desktop (%)	Mobile (%)	Total (%)	Desktop (%)	Mobile (%)
Hover over link	48,748.9 (46.9)	45,814.8 (60.3)	2,934.1 (10.5)	1,122,704.0 (32.7)	1,057,982.0 (47.2)	64,722.0 (5.4)
Footnote click	27,739.4 (26.7)	10,948.8 (14.4)	16,790.6 (60.3)	722,131.0 (21.0)	235,245.0 (10.5)	486,886.7 (40.7)
External click	25,811.9 (24.9)	17,792.3 (23.4)	8,019.7 (28.8)	1,557,125.0 (45.3)	915,445.1 (40.9)	641,676.4 (53.6)
Up click	1,539.5 (1.5)	1,422.8 (1.9)	116.4 (0.4)	34,738.0 (1.0)	31,230.1 (1.4)	3,508.1 (0.3)
All events	103,839.7 (100)	75,978.7 (100)	27,860.8 (100)	3,436,698.0 (100)	2,239,902.2 (100)	1,196,793.2 (100)

Among the external links in WPM, 22.1% of the research citations drawn from PubMed are labeled as “freely accessible”, and on desktop devices include a link bearing an open access icon (a green open lock, with the rollover label “freely accessible”), as shown in Figure 2. However, there was no evidence to suggest that links with these icons were clicked more frequently than other links. This may be, in part, because the typical research citation has three or four links leading to (a) the article on the publisher’s site (DOI link); (b) its PubMed entry (PMID link); and (c) PubMed Central (PMC link and article title link) if the article is open access.

Figure 2.

Open access reference displayed in Wikipedia.

A research study cited on Wikipedia’s “diabetes” page displayed on desktop device with “freely accessible” icon for PubMed Central (PMC) link and on a mobile device without the open access icon. There is an “up click”( ^ ) in front of the citation on the desktop device, which returns reader back to footnote 46 in the text.

To further compare event data (i.e. the type of engagement with external links) between WPM and W, the number of pageviews per event was calculated to determine how many pages readers viewed before engaging with an external link (Table 5). Overall, readers of WPM pages were more likely (56.6 pageviews per event) to engage with a link than W readers (66.5 pageviews per event). The differences in engagement were most pronounced with WPM readers hovering over a link on their desktop device. Yet, W readers more frequently clicked external links, with the difference especially notable when on their mobile device, where they were almost twice as likely to click an external link than WPM readers. This suggests that the external links in WPM may not be as mobile-friendly. The up click, in which a reader clicks the link from a footnote back (up) to the text, was not a function that appeared on the iOS and Android mobile devices we tested, as the footnote appears at the bottom of the screen when clicked and disappears on touching the text (Figure 2).

Table 5.

Number of pageviews per engagement event.

The frequency of each event per day was divided by the average number of daily pageviews for WPM and W pages from March 22nd to April 22nd, 2019. The lower the number of pageviews per event the greater the event frequency. Difference between each pair of WPM and W distributions is statistically significant as derived from Fisher’s exact test (p<0.001, two-tailed).

Pageviews/event	WPM			W
	Total	Desktop	Mobile	Total	Desktop	Mobile
Hover over link	120.5	42.7	1,335.2	203.5	92.6	2,016.1
Footnote click	211.8	178.8	233.3	316.3	416.4	268.0
External click	227.6	110.0	488.5	146.7	107.0	203.4
Up click	3,816.6	1,376.1	33,655.0	6,576.2	3,136.6	37,196.5
All events	56.6	25.8	140.6	66.5	43.7	109.0

Discussion

The Wikipedia pages maintained by WPM are longer, possess a greater density of external links (references), and are viewed considerably more often, on average, than the pages on the rest of Wikipedia (Table 6). The popularity of the pages also speaks to the valuable contribution that WPM makes to Wikipedia’s role as a source of health and medical information, as well as the level of trust that WPM readers have in Wikipedia’s coverage of health and medical topics.

Table 6.

Summary of data collected.

Summary statistics selected from the tables above, comparing WikiProject Medicine (WPM) pages and readers to the rest of Wikipedia (W) pages and readers. (a) The “4.4 more pageviews” reflects the ratio of WPM to W pages (0.6%) compared to the ratio of WPM to W pageviews/day (2.6%). (b)“WPM readers” and “W readers” refer to the behaviors of those reading a WPM and/or W page during the data collection period.

	WPM pages	W pages	WPM pages, compared to other W pages,...
Page length (characters)	13,085	7676	…are 70.5% longer by character count.
Characters/external link (on pages with links)	450.3	657.3	…possess a 31.5% greater external link density.
Pageviews/day	5,875,470.40	228,445,128.40	…receive 4.4 more pageviews per day.a
	WPM readers	W readers	WPM readers, compared to rest of W readers,...b
Time before engagement (sec)	47.5	32.8	…take 44.8% longer before engaging in a link activity.
Pageviews/link engagement	56.60	66.47	…engage 17.5% more often with links per pageview.
Pageviews/hover	120.5	203.5	…hover over links 68.9% more often per pageview.
Pageviews/footnote click	211.8	316.3	…click footnote numbers 49.3% more often per pageview.
Pageviews/external click	227.6	146.7	…click external links 35.5% less often per pageview.

In addition, the readers of WPM pages not only engage more often with external links, but spend longer on the page before engaging with the links, compared to readers of the rest of Wikipedia (Table 6). WPM readers are more likely to hover over references, causing them to appear in a window, and more likely to view the footnotes than other readers. But readers of WPM pages click on external links less often than readers of other parts of Wikipedia.

This behavior pattern suggests that readers are more interested in validating the external references. This interest may arise from readers who are relatively new to these pages and are in the process of reassuring themselves as to the scientific basis of the content they are consulting. But it may also arise from readers who are trained, or are in the process of being trained, in health professions where they are required to critically appraise and integrate health information into the care of their patients. These evidence-based practitioners are able to judge a lot about the quality of information from the bibliographic reference to the sources, which they view by hovering over or going to the footnote. In addition to being able to see at a glance (or a hover) whether the cited work is a review (which is recommend by Wikipedia as being the “ideal source”), readers are able to identify publication date, language, publication venue (i.e., journal name), and availability (via open access) (Figure 3A).

Figure 3.

Example of information displayed when hovering over footnotes in Wikipedia.

(A) An external link on the Hepatitis page, revealed by “hovering” over the footnote number, indicating that the source is a systematic review conducted by the Cochrane Collaboration (with a green icon PMC link signaling open access to the source). (B) Hovering over a different footnote number on the same page reveals a different external link for the 19th and relatively recent edition of a medical textbook from a leading publisher.

In addition, in some instances it is possible to obtain from a citation details on patient populations, medical interventions (e.g., treatments, diagnostic tests), and outcomes involved in the study or whether the source is a standard medical textbook in the field (Figure 3B). The greater attention that WPM readers are paying to citations could well be a function of their educational backgrounds, reflecting the demonstrated use of WPM by health professionals, students and practitioners. Although not the focus of this study, the educational value of internal links, in which terms on a WPM page are hyperlinked to other Wikipedia pages or to VideoWiki, should also be noted as a further strategy for maintaining the comprehensive informational and educational quality of WPM pages.

This still leaves the question of why WPM readers are less likely to click through to the external source than readers of the rest of Wikipedia. This question is particularly relevant given the interest demonstrated by WPM readers in viewing the citations (by hovering or clicking on footnote numbers), as well as by the studies (cited above) demonstrating that health professionals use WPM pages, which suggests a potential capacity for further learning through the external links. One possibility is that readers are deterred from clicking on external WPM links in light of our finding that only a fifth of the research cited in WPM pages are publicly accessible, with the rest placed by publishers behind a paywall for subscribers or credit card access. Earlier studies have found that it is common for readers, including physicians, who encounter a paywall to be reluctant to explore the literature further (Moorhead et al., 2015; Maggio et al., 2016). It is possible that if all the research literature was available through open access (and this fact was promoted among readers) health professionals may be more likely to pursue the greater educational advantage of the external links. Currently, there are very few links bearing an open access icon, and readers show no additional interest in viewing these links just because they are open. If readers start expecting that external links will be open (and there is some promotion of this fact), then the two studies just cited suggest that the chances of them pursuing an item of interest will become more likely.

Although this study does not provide a means of determining why readers hover rather than click on external links, this is something that could be addressed using smaller-scale research strategies like reader questionnaires or think-aloud protocols. Alternatively, design experiments, which experiment with external link format and context, could be conducted involving, (a) WPM pages in which all external links are clearly indicated as open access, (b) tested readers are given prior training in how to learn from cited works, and (c) how different mobile interfaces impact the amount research studies are clicked on for further reading.

Limitations

One of the limitations of this study is the way in which the count for “hovering” (defined as a reader’s cursor lingering over a link for 1000 milliseconds or more, revealing a rollover label) captures both intentional and unintentional acts. So, while the reported number of hovers is of limited value, there is no reason to believe that the incidental hovers would differ between WPM and W readers. As well, with the pageview data, various strategies were used to exclude pages visited by bots, but the limited effectiveness of these strategies is a known weakness in Wikipedia’s infrastructure. It, again, suggests a tempering of the pageview counts but not the ratios between WPM and W in this regard. This is similarly the case with the exclusion from this study of interwiki links that reduced external link counts, as described in the methods. Lastly, this study focuses only on the English-language version of Wikipedia and therefore our findings are limited to this version of the encyclopedia.

Conclusion

The study has identified a number of distinctions that set WPM pages apart from the rest of Wikipedia. These differences, such as page length and link density, reflect the quality of these pages as information sources. This finding was further supported by differences in reader behavior, such as time spent on page, and amount of external link engagement, particularly the rate readers hover or click on footnote numbers to examine the bibliographic information on the sources contained in the links. This suggests that WPM readers are more engaged in assessing the citation of the external link, which assists in validating and building trust in the content that makes up the WPM pages, than by the opportunity to visit the sources as a gateway for further learning. Subsequent studies of readers engagement with Wikipedia’s external links should investigate the role citations play in what readers learn about a topic and whether there is the potential, especially for readers in health professions, to increase the degree and depth of learning.

Disclaimer

The views expressed in this article are those of the authors and do not necessarily reflect the official policy or position of the Uniformed Services University of the Health Sciences, the U.S. Department of Defense, or the U.S. Government.

In the interests of transparency, eLife publishes the most substantive revision requests and the accompanying author responses.

Thank you for submitting your article "Reader engagement with Wikipedia's medical content" for consideration by eLife. Your article has been reviewed by three peer reviewers, and the evaluation has been overseen by two members of the eLife Features team (Julia Deathridge and Peter Rodgers). The following individuals involved in review of your submission have agreed to reveal their identity: James Heilman (Reviewer #1); Besnik Fetahu (Reviewer #2); Shani Evenstein (Reviewer #3).

The reviewers have discussed the reviews with one another and we have drafted this decision to help you prepare a revised submission. We hope you will be able to submit the revised version within two months.

Summary:

This work enriches our understanding of how readers of Wikipedia's medical pages engage with external links compared to readers of standard Wikipedia pages. However, there are a few major comments that need to be addressed.

Major comments to address:

a) The results reveal that in general the readers of the standard Wikipedia pages are more prone to actually clicking the external links. This is interesting given that one would think that the motivation for visiting such pages may reflect more personal reasons, i.e. checking on symptoms of a disease. Yet, the non-WPM pages seem to have better engagement with the external links. One possibility for this could be that the page visits are not normally distributed across all the pages. You have highly important pages in Wikipedia that gather a lot of attention, thus, their external links are more likely to be visited. I assume that normalizing the Wikipedia articles according to their page visits might paint a different picture here with respect to the page visit.

b) In the second research question, the motivation behind visiting such external links is postulated. While personally I agree with the points in the discussion as to why this may happen, there is no actual evidence for this in the current study. The authors should include a sentence or two discussing this limitation and what further work could be done to validate their hypotheses, such as carrying out a questionnaire with a diverse set of WPM readers that investigates their reasoning for clicking/not-clicking on external links.

c) As the article aims to review WPM content as a gateway to further learning and deeper understanding of a given topic, especially as it is used by both students, health professionals and the general public, one thing that was missing from this effort was an acknowledgement of the importance of both internal links and media files as a means of enhancing understanding, not only that of references and footnotes. Even if this research effort chooses to focus only on external links and footnotes, then an explanation is expected as to why specifically this focus. Especially in a world where imagery is so dominant, and videos have become a major means of education, further research is needed regarding how readers engage with media content on WPM articles, compared with the rest of Wikipedia. If the research team has any data related to this, not only to external links and footnotes, it could be of great contribution to mention it as well.

d) While research till 2016 (Heilman & West) showed that Wikipedia was the most popular site for medical content online, more than sites like WHO, NIH etc, WikiProject Medicine Foundation has shown that this has changed in the last couple of years. It seems that Wikipedia's medical content is no longer the most visited site in the world for medical content and is "losing" to sites that encourage users to ask different questions on the same topic and have multiple answer pages on that same topic. It would be of interest to the public if the research team considers this and discusses the results in this light. It might offer additional conclusions to the ones already presented.

[We repeat the reviewers’ points here in italic, and include our replies point by point, as well as a description of the changes made, in plain text]

Major comments to address:

a) The results reveal that in general the readers of the standard Wikipedia pages are more prone to actually clicking the external links. This is interesting given that one would think that the motivation for visiting such pages may reflect more personal reasons, i.e. checking on symptoms of a disease, yet, the non-WPM pages seem to have better engagement with the external links. One possibility for this could be that the page visits are not normally distributed across all the pages. You have highly important pages in Wikipedia that gather a lot of attention, thus, their external links are more likely to be visited. I assume that normalizing the Wikipedia articles according to their page visits might paint a different picture here with respect to the page visit.

We appreciate this question on our finding that WikiProject Medicine (WPM) readers clicked on external links less often per view than Wikipedia (W) readers. We investigated ways of normalizing for pageviews and page-length, as two factors that might be associated with this result. The most common approach would be to use matching methods, involving creating matched samples or datasets for, in this case, matching WPM and W for pageviews and for page length. This creates, in effect, random samples for these two factors which then allows one to determine the effect of an intervention, or in this case the extent to which a factor is associated with an external link behavior. But since we set out to determine if and how WPM pages and readers differ from the rest of W pages and readers, our goal has been to describe the pattern of the differences as a whole, rather than ask what would happen if pageviews or page lengths were the same for WPM and W. This helped us realize that the focus on the comprehensive pattern was not previously clear enough, and we have now, thanks to this reviewer’s comment, brought it to the fore in revisions to the research questions and discussion. The principal change is to have introduced a summary table (Table 6) in the discussion that presents the three key factors that distinguish WPM pages from the rest of W, and the five key factors that distinguish WPM readers from W readers. As a result of structuring the results this way, we believe it is easier to grasp consistencies that are to be expected with the reading behaviors of health sciences students and professionals (whose use of WPM has been well established, if not the proportion of WPM readers that they make up). We think that this points to the sort of future research helpfully suggested by a reviewer involving questioning readers either directly about their choices or indirectly through think-aloud protocols. All of that said, if the reviewers still feel that matching methods should be applied, we would, of course, be willing to conduct such an analysis. To reflect the above changes, we have also modified the abstract.

b) In the second research question, the motivation behind visiting such external links is postulated. While personally I agree with the points in the discussion as to why this may happen, there is no actual evidence for this in the current study. The authors should include a sentence or two discussing this limitation and what further work could be done to validate their hypotheses, such as carrying out a questionnaire with a diverse set of WPM readers that investigates their reasoning for clicking/not-clicking on external links.

We interpreted this important point as partly a valuable critique of our second research question. We have reframed the research questions in the introduction so that they follow the pattern of the study, dealing with differences between WPM and W pages and readers. This had led us to extend the analysis of the greater hovering and footnote-number clicking in the Discussion, in which WPM readers must be gaining additional reassurances from how the title and publication venue of the references can provide scientific characteristics of the study, using the “ideal sources” of systematic reviews and medical textbooks as examples. That said, we fully embrace and now include the reviewer’s point about follow-up qualitative studies among a diversity of users to determine what influences their choices around engagement with external links.

c) As the article aims to review WPM content as a gateway to further learning and deeper understanding of a given topic, especially as it is used by both students, health professionals and the general public, one thing that was missing from this effort - an acknowledgement of the importance of both internal links and media files as a means of enhancing understanding, not only that of references and footnotes. Even if this research effort chooses to focus only on external links and footnotes, then an explanation is expected as to why specifically this focus. Especially in a world where imagery is so dominant, and videos have become a major means of education, further research is needed regarding how readers engage with media content on WPM articles, compared with the rest of Wikipedia. If the research team has any data related to this, not only to external links and footnotes, it could be of great contribution to mention it as well.

We did fail to properly acknowledge the contribution of internal links for reader learning. And while our focus remains on the association between Wikipedia and the body of external research, that has now been made clear (in the Introduction), in the context of how richly cross-referenced Wikipedia is and how much of that lateral reading can contribute to the informational and educational quality of WPM pages.

d) While research till 2016 (Heilman & West) showed that Wikipedia was the most popular site for medical content online, more than sites like WHO, NIH etc, WikiProject Medicine Foundation has shown that this has changed in the last couple of years. It seems that Wikipedia's medical content is no longer the most visited site in the world for medical content and is "losing" to sites that encourage users to ask different questions on the same topic and have multiple answer pages on that same topic. It would be of interest to the public if the research team considers this and discusses the results in this light. It might offer additional conclusions to the ones already presented.

This comment raises interesting points about changes to the questions readers may be asking and how that is affecting the popularity of WPM pages. After some discussion among us, as well as searches for relevant literature on it, we wish to respectfully suggest that this comment is out of scope for this paper, given our focus on reader engagement with external links on WPM pages. Based on recent studies on medical and health provider use of Wikipedia (Scaffidi, 2017), we see the relevance of this site for health information is still substantial, and a particularly relevant source for the initial and continuing education of health professionals. We appreciate the reviewers’ encouragement to consider broader issues around such topics as reader interests and questions.