References that anyone can edit: review of Wikipedia citations in peer reviewed health science literature.

OBJECTIVES: To examine indexed health science journals to evaluate the prevalence of Wikipedia citations, identify the journals that publish articles with Wikipedia citations, and determine how Wikipedia is being cited.
DESIGN: Bibliometric analysis. STUDY SELECTION: Publications in the English language that included citations to Wikipedia were retrieved using the online databases Scopus and Web of Science. DATA SOURCES: To identify health science journals, results were refined using Ulrich's database, selecting for citations from journals indexed in Medline, PubMed, or Embase. Using Thomson Reuters Journal Citation Reports, 2011 impact factors were collected for all journals included in the search. DATA EXTRACTION: Resulting citations were thematically coded, and descriptive statistics were calculated.
RESULTS: 1433 full text articles from 1008 journals indexed in Medline, PubMed, or Embase with 2049 Wikipedia citations were accessed. The frequency of Wikipedia citations has increased over time; most citations occurred after December 2010. More than half of the citations were coded as definitions (n = 648; 31.6%) or descriptions (n=482; 23.5%). Citations were not limited to journals with a low or no impact factor; the search found Wikipedia citations in many journals with high impact factors.
CONCLUSIONS: Many publications are citing information from a tertiary source that can be edited by anyone, although permanent, evidence based sources are available. We encourage journal editors and reviewers to use caution when publishing articles that cite Wikipedia.

Introduction

Launched on 15 January 2001, Wikipedia is self described as “a free, collaboratively edited, and multilingual internet encyclopedia supported by the non-profit Wikimedia Foundation.”1 As of 2012 Wikipedia is the largest online reference site,2 3 and it is reported to be the most used online healthcare resource globally.4 However, the assessment of Wikipedia as a credible source for information has been debated since its origin.5 The impermanent nature of Wikipedia entries and concerns about quality have been raised as important matters.6

The literature on Wikipedia has concentrated on evaluating content and ensuring that users have access to appropriate information.5 Only 13% of Wikipedia articles had identifiable errors when assessed by academics.7 Giles and colleagues found that the number of factual errors, omissions, or misleading statements in Wikipedia articles was comparable to the Encyclopaedia Britannica.8 In general, Wikipedia articles reference academic literature,9 10 and medical articles are overseen by WikiProject Medicine, which is driven by editorial oversight.2 4 Several studies have confirmed the accuracy of health specific Wikipedia articles and discuss the potential value in education of patients.2 3 8 11 12 13 14 15 Wikipedia has offered an innovative way to provide free access to information for people around the world.2 4 5 To further improve the quality of these articles, medical professionals are being encouraged to contribute to Wikipedia.2 16 17 18 Studies have also examined groups that use Wikipedia, noting that, in addition to patients and nursing students,2 3 10 19 medical students and residents commonly access Wikipedia to acquire health information.20 21 22 23 Although physicians have been discouraged from relying on Wikipedia,16 one study showed that in practice use of Wikipedia is as high as 70% among junior physicians.22 In general, consultation of Wikipedia is a growing trend among academics. For example, among more than 1000 authors in Nature, 17% reported that they consulted Wikipedia on a weekly basis.8 As the public and healthcare professionals increasingly accept Wikipedia as a source of information, new questions emerge such as the appropriateness of citing Wikipedia in academic publications.

The controversial nature of citing Wikipedia as a reference source for academic information is threefold. Firstly, a theoretical concern exists that anyone with access to the internet can alter Wikipedia. This raises questions about the spread of unintentional misinformation, which is fuelled by acts of intentional vandalism that have gained media attention and have challenged the public’s perception of Wikipedia.24 25 26 However, the use of a wiki model addresses the fear that that the information on Wikipedia is not guaranteed to be correct. The concept of a wiki is based on the assumption that the majority is correct: if someone writes an erroneous statement on Wikipedia, the probability is considered to be high that someone else from the majority that is presumed to be correct will identify the error. This model has many advantages and allows Wikipedia to be a free source of a vast amount of collaborative information.2 4 5 However, the scientific community still has concerns about the academic integrity of a model that in theory could be edited by anyone.

Secondly, the changing nature of Wikipedia makes permanent versions difficult to access. Although Wikipedia maintains a detailed history of previous Wikipedia pages, current academic citation systems rarely require a citation to include time stamped access information detailed to the second. As Wikipedia pages are in constant change, unlike paper encyclopaedias in which readers can confidently find the permanent cited source, readers might find confirming that the Wikipedia reference is the exact version cited challenging, particularly if detailed time stamps are not included.27

Thirdly, citing tertiary sources such as Wikipedia, which are resources that compile or provide digests of secondary sources, has literary problems.28 Secondary sources are books, articles, or unpublished literature that provide an interpretation of primary sources, which include original data, manuscripts, records, or documents. International guidelines state that authors should provide direct references to original research sources,29 so citing Wikipedia or any other tertiary source in the academic literature opposes literary practice. To date, only one study has examined the frequency of citation of Wikipedia in the academic literature.30 To our knowledge, no one has examined the frequency of Wikipedia citations in the health science literature and the ways in which Wikipedia citations are used. We aimed to evaluate the prevalence of Wikipedia citations in indexed health science journals, identify the health science journals that publish articles with Wikipedia citations, and determine how Wikipedia is being cited.

Methods

Data collection

In February 2012 one investigator (LU) searched the ISI Web of Science and Scopus online databases to identify articles in the peer reviewed health science literature that had directly cited Wikipedia since its inception in 2001. She searched references of all articles in both databases on the same day for the word “Wikipedia” or any possible derivation to account for spelling errors. The word could appear anywhere in the reference, so not all articles necessarily referenced Wikipedia but could be articles about Wikipedia. To focus on indexed health science literature, we refined the results by using Ulrich’s Database to identify articles from journals indexed in at least one of Medline, PubMed, or Embase. We retrieved full text for the articles (including reviews, original research, editorials, letters to the editor, and case reports) identified in the search through the University of Ottawa library and associated Canadian inter-library loan services. We excluded citations when articles were not written in English or when full text was unavailable. Although citations originated in the health science literature, the topic of the citation could extend beyond health, so we included all Wikipedia pages that were cited in the academic literature. In the event that an article cited Wikipedia more than once, we considered all citations and created duplicate entries for that article.

We used Thomson Reuters Journal Citation Reports (JCR) to collected journals’ 2011 impact factors (a measure that reflects how frequently the average article in a journal has been cited in a particular period).31 32 JCR impact factors provide quantitative evidence about the position of one journal in relation to the competition and offer an approximation of the prestige of a journal. When interpreting the impact factor, one should consider only journals within the same subject category as the scores are relative.32 JCR coverage includes the world’s leading journals and offers a systematic, quantifiable means to critically evaluate them by measuring the influence and impact of research at the journal and category levels, as well as showing the relation between citing and cited journals. We did not recover other journal metrics because of the unique nature of JCR and its pre-eminence as a proxy to measure a journal’s importance within a field.

Data extraction/coding system

Using an iterative process, we developed a descriptive coding strategy. Three investigators (MDB, SB, ESH) reached consensus on the distinction between categories. After independently reviewing 15% of citations, the three investigators met as a team to discuss their coding strategies. They developed an initial coding strategy and recoded citations with the new guide. The team continued this iterative process of coding, meeting, and refining the guide before the three authors reached agreement and the coding strategy was finalised. Emergent sub-codes were integrated into the coding system as they arose. Using the coding system, one investigator (ESH) systematically coded all Wikipedia citations. To ensure coding reliability, another investigator (AEP) coded 100 random citations with the same coding system.

Analysis

We used SPSS 17.0 for statistical analysis. We calculated descriptive statistics for the frequency of Wikipedia citations by thematic code, the impact factor of journals citing Wikipedia (median, range), and the frequency of these statistics by year. We used the intra-class correlation coefficient to assess the inter-rater reliability.

Results

We recovered 2359 publications (2307 from Scopus and an additional 52 from Web of Science) in our search. After excluding duplicates, non-English journals, and articles that did not directly cite Wikipedia, we retrieved full text for 1433 articles from 1008 indexed journals, with 2049 Wikipedia citations. In total, 2011 impact factors were available for 1420 citations in 980 articles from 650 indexed journals and were found in the JCR database.

Table 1 shows the frequency of Wikipedia citations since 2001, the nature of citations (thematic code), and related impact factor information (median and range). The intra-class correlation coefficient for the two coders was 0.91 (P<0.001), indicating a high degree of inter-rater reliability. More than half of the citations were coded as definitions (n=648; 31.6%) or descriptions (n=482; 23.5%). For the purpose of this study, of the 13 categories, we considered citations (n=82; 4.0%) from only two categories (Citations about Wikipedia, and Wikipedia used in methods) to be the most appropriate uses of Wikipedia, as in these cases Wikipedia was the original source of information. Furthermore, we recovered 97 (4.8%) citations in which Wikipedia was cited in place of an original research study.

Table 1

 Frequency of Wikipedia citations since 2001, categorised by code and impact factor

Coding strategy		Total citations—No (%)	Median (range) impact factor
Definition	A statement defining a word, phrase, item, or other symbol. This includes definitions of chemical formulas and equations that are not otherwise supported by original research references. Example: Caregiver is an internationally accepted term for unpaid people who are caring for someone requiring support due to a disability, frailty, mental health problem, learning disability or old age (Wikipedia)	648 (31.6)	1.9 (0.1-31.2)
Descriptive statement	Description of a process, system, or event. This differs from a definition, in that a process is explained without necessarily defining a term. Example: The key component of clinical audit is that performance is reviewed (or audited) to ensure that what should be done is being done, and if not, it provides a framework to enable improvements to be made (Wikipedia)	482 (23.5)	1.9 (none*-31.2)
Historical	Description of a historical event. This can include the date, location, and a description of the events that took place. Example: In 1984 a beautiful 18-year-old coed, Libby Zion, died in a New York City hospital after being improperly treated by sleep-deprived residents (Wikipedia)	277 (13.5)	1.7 (0.1-36.3)
Statistics	Includes demographic and gross domestic product information (population size, including census information), geographical information, ratios, percentages, and averages. Example: West Bengal is an agriculture-dependent state in eastern India. It occupies only 2.7% of India’s land area, although it supports over 7.8% of the Indian population, and is the most densely populated state in India (Wikipedia)	161 (7.9)	2.0 (none*-16.1)
Original research	Reporting results from a research study when on the Wikipedia page the original research is referenced but in the article the authors referenced Wikipedia, or the authors imply something like “research suggests that. . .” with no support other than Wikipedia. Example: Epidemiologic studies have shown that individuals who consume high levels of b-carotene in their diet and those with high levels of b-carotene in their blood have significantly reduced risk of lung cancer (Wikipedia)	59 (2.9)	2.9 (0.7-8.8)
	Tools used, such as a laws, formulas, or scales that could have been referenced using an original source. Example: The paediatric GCS (Wikipedia) and PNG TB score 5,6 (Table 1) were recorded for all children	38 (1.9)	2.8 (0.6-14.1)
Biographical	Description of a person’s life or accomplishments. This may include dates. Example: In 1895, Wilhelm Conrad Roentgen reported the discovery of the x-ray (Wikipedia)	91 (4.4)	1.2 (0.4-13.7)
Citation about Wikipedia	Citation about the Wikipedia, describing what it is, how it is used, and so on.Example: A case in point is Wikipedia, the largest encyclopedia in the world. With over 3.5 million registered users (Wikipedia)	78 (3.8)	2.8 (0.6-13.8)
Web 2.0 citation	Describing what Web 2.0 is and examples; Web 2.0 are resources in which multiple users continually update and remix data.21 Example: During the first decade of the 21st century, we have witnessed the birth of Web 2.0, heralded by a set of powerful tools designed to enhance creativity, information sharing, collaboration, and functionality of the Web, and to transfer power to the end user (Wikipedia)	55 (2.7)	1.8 (0.5-5.7)
Images	Any images, including maps, people, structures, figures, and so on	50 (2.4)	1.8 (0.5-6.8)
Recommended reading	Author of article says to search Wikipedia for further details and references Wikipedia. Example: For a very interesting explanation of the impact factor and the debate around its use, go to Wikipedia’s “Impact Factor” entry (Wikipedia)	42 (2.0)	2.0 (0.5-7.8)
Unclear citation/ referencing	Unclear language, unable to understand what part of the citation is a reference to Wikipedia, reference number does not match citation, missing reference number, and so on. Example: http://en.wikipedia.org was listed in the reference list, but the matching reference number was absent from the text	37 (1.8)	2.1 (0.1-9.9)
Direct quotation	Replicating a direct quote from a person, movie, or book quoted in Wikipedia. Example: Ralph Waldo Emerson, an American philosopher, said: “Nothing great was ever achieved without enthusiasm” (Wikipedia)	27 (1.3)	1.8 (1.0-2.8)
Wikipedia in methods	Wikipedia is directly used in the research methods of the paper. Example: We compared the rise to fame of the most famous people of different eras. We took all 740,000 people with entries in Wikipedia, removed cases where several famous individuals share a name, and sorted the rest by birth date and frequency (Wikipedia)	4 (0.2)	4.1 (4.1-31.2)
Total		2049 (100)	2.0 (none*-36.3)

*Journal indexed in Journal Citation Reports does not have assigned impact factor.

The median impact factor of journals citing Wikipedia was 2.0 and has remained fairly consistent over time. However, the total number of Wikipedia citations has increased each year since 2004 except for between 2009 and 2010. The figure shows the number of articles that cite Wikipedia at least once by year up to 21 November 2013. Consistently since the inception of Wikipedia, journals with high impact factors have continued to publish references to Wikipedia. Table 2 illustrates the journals with the top 25 highest impact factors that emerged in our study. These journals accounted for 2.2% (n=3) of cases in which Wikipedia was the most appropriate citations (as previously defined). Table 3 specifies the titles and types of articles published in these journals, including the number of Wikipedia citations in each article. Journals with available impact factors that cited Wikipedia more than 10 times included Explore: The Journal of Science and Healing (n=53), Child’s Nervous System (n=26), BMC Bioinformatics (n=21), Theoretical Biology and Medical Modelling (n=20), Journal of Medical Internet Research (n=17), Health Information and Libraries Journal (n=13), BMJ (n=13), JALA—Journal of the Association for Laboratory Automation (n=12), and American Journal of Forensic Medicine and Pathology (n=11).

Number of articles that cite Wikipedia at least once by year since 2001. *Data from February 2012 to November 2013 were not included in detailed analysis

Table 2

 25 journals with highest impact factors that have cited Wikipedia since 2001

Periodical name	Impact factor	Total No (%) of Wikipedia citations	No (%) of citations for which Wikipedia is most appropriate source*
All recovered citations	—	2049 (100)	82 (100)
Nature	36.3	1 (0.05)	0 (0)
Science	31.2	3 (0.1)	1 (1.2)
Nature Reviews Neuroscience	30.4	1 (0.05)	0 (0)
Nature Medicine	22.5	1 (0.05)	0 (0)
Journal of Clinical Oncology	18.4	3 (0.1)	0 (0)
The Lancet Infectious Diseases	17.4	5 (0.2)	0 (0)
Annals of Internal Medicine	16.7	4 (0.2)	0 (0)
PLoS Medicine	16.3	5 (0.2)	0 (0)
Clinical Microbiology Reviews	16.1	2 (0.1)	0 (0)
Circulation	14.7	3 (0.1)	0 (0)
Journal of the American College of Cardiology	14.2	3 (0.1)	0 (0)
BMJ	14.1	13 (0.6)	0 (0)
Journal of the National Cancer Institute	13.8	2 (0.1)	1 (1.2)
Molecular Psychiatry	13.7	1 (0.05)	0 (0)
Nature Reviews Neurology	12.5	2 (0.1)	0 (0)
Gastroenterology	11.7	1 (0.05)	0 (0)
Journal of the American Chemical Society	9.9	5 (0.2)	0 (0)
Clinical Infectious Diseases	9.2	1 (0.05)	0 (0)
Genome Biology	9.0	2 (0.1)	1 (1.2)
Nature Reviews Cardiology	8.8	1 (0.05)	0 (0)
Schizophrenia Bulletin	8.8	5 (0.2)	0 (0)
Diabetes care	8.1	1 (0.05)	0 (0)
Nucleic Acids Research	8.0	1 (0.05)	0 (0)
Science Translational Medicine	7.8	4 (0.2)	0 (0)
Annals of Surgery	7.5	3 (0.1)	0 (0)
Total	—	73 (3.6)	3 (2.2)

*Citations for which Wikipedia is original source of information (citations about Wikipedia, and Wikipedia used in methods) were considered most appropriate citations of Wikipedia.

Table 3

 25 journals with highest impact factors: article titles, types, and number of Wikipedia citations

Journal	Article	Type of article	No of Wikipedia citations
Nature	Tracking the rupture of the Mw = 9.3 Sumatra earthquake over 1,150 km at teleseismic distance	Letter	1
Science	Challenges and opportunities in mining neuroscience data	Perspective	1
	On the future of genomic data	Perspective	1
	Quantitative analysis of culture using millions of digitized books	Research article	1
Nature Reviews Neuroscience	Gene therapy: can neural stem cells deliver?	Perspective	1
Nature Medicine	Perspectives on the properties of stem cells	Commentary	1
Journal of Clinical Oncology	Reply to R.I. Haddad et al	Correspondence	1
The Lancet Infectious Diseases	Emergence of medicine for mass gatherings: lessons from the Hajj	Series	4
	Non-communicable health risks during mass gatherings	Series	1
Annals of Internal Medicine	Akhenaten and the strange physiques of Egypt’s 18th dynasty	History of medicine	2
	The nomogram epidemic: resurgence of a medical relic	History of medicine	1
	Individualized guidelines: the potential for increasing quality and reducing costs	Medicine and public policy	1
PLoS Medicine	The global health system: linking knowledge with action—learning from malaria	Policy forum	1
	Motor vehicle crashes in diabetic patients with tight glycemic control: a population-based case control analysis	Research article	1
	Cardiovascular risk with non-steroidal anti-inflammatory drugs: systematic review of population-based controlled observational studies	Research article	1
	Measuring the true costs of war: consensus and controversy	Perspective	1
	Global health philanthropy and institutional relationships: how should conflicts of interest be addressed?	Policy forum	1
Clinical Microbiology Reviews	Bed bugs: clinical relevance and control options	Review	2
Circulation	There is a role for industry-sponsored education in cardiology	Controversies	2
	ACC/AHA/ACR/ASE/ASNC/HRS/NASCI/RSNA/SAIP/SCAI/SCCT/SC MR/SIR 2008 key data elements and definitions for cardiac imaging: a report of the American College of Cardiology/American Heart Association task force on clinical data standards (writing committee to develop clinical data standards for cardiac imaging)	Reports	1
Journal of the American College of Cardiology	ACC/AHA/ACR/ASE/ASNC/HRS/NASCI/RSNA/SAIP/SCAI/SCCT/SC MR/SIR 2008 key data elements and definitions for cardiac imaging: a report of the American College of Cardiology/American Heart Association task force on clinical data standards (writing committee to develop clinical data standards for cardiac imaging)	Reports	1
BMJ	How citation distortions create unfounded authority: analysis of a citation network	Research article	9
	Statistics notes: brackets (parentheses) in formulas	Research methods and reporting; Statistics notes	2
	The royal road to healing: a bit of a saga	Feature; Christmas 2011: Oral traditions	1
	CD4 cell count and viral load monitoring in patients undergoing antiretroviral therapy in Uganda: cost effectiveness study	Research article	1
Journal of the National Cancer Institute	Role of science in the treatment of breast cancer when tumor multicentricity is present	Commentary	1
	Effect of misclassified underlying cause of death on survival estimates of colon and rectal cancer	Brief communication	1
Molecular Psychiatry	Do reasons for major depression act as causes?	Original article	1
Nature Reviews Neurology	Review	2
Gastroenterology	Regenerative medicine and the gut	Reviews and perspectives	1
Journal of the American Chemical Society	Luminescent graphene quantum dots for organic photovoltaic devices	Communications	2
	White light emission and second harmonic generation from secondary group participation (SGP) in a coordination network	Article	1
	ENDOR/HYSCORE studies of the common intermediate trapped during nitrogenase reduction of N 2H 2, CH 3N 2H, and N 2H 4 support an alternating reaction pathway for N 2 reduction	Article	1
	Dielectric and thermal effects on the optical properties of natural dyes: a case study on solvated cyanin	Article	1
Clinical Infectious Diseases	A woman with knee pain and soft-tissue calcification	Photo quiz (answers)	1
Genome Biology	Proteopedia—a scientific ‘wiki’ bridging the rift between three-dimensional structure and function of biomacromolecules	Software	1
Nature Reviews Cardiology	Can we dramatically reduce the incidence of coronary heart disease?	Perspective	1
Schizophrenia Bulletin	Beating the odds—nothing is impossible, its just a road less traveled	Special feature	5
Diabetes Care	National standards for diabetes self-management education	National standards and review	1
Nucleic Acids Research	Rfam: Wikipedia, clans and the “decimal” release	Article	1
Science Translational Medicine	The NCGC pharmaceutical collection: a comprehensive resource of clinically approved drugs enabling repurposing and chemical genomics	Perspective	4
Annals of Surgery	A review of natural orifice translumenal endoscopic surgery (NOTES) for intra-abdominal surgery: experimental models, techniques, and applicability to the clinical setting	Review	3

Discussion

Our findings illustrate a relatively small but increasing frequency of citations of Wikipedia in indexed health sciences literature. We retrieved 2049 Wikipedia citations, in 1433 articles from 1008 journals indexed in Medline, PubMed, or Embase. Most of these citations occurred after December 2010 and were consistently found in journals with low impact factors and journals without impact factors, as well as in journals with high impact factors, including Nature, Science, and the BMJ. Using the descriptive coding strategy, we found a wide variety of uses for Wikipedia citations in the literature, the most common being definitions and descriptive statements.

Strengths and limitations

As the first study to describe the citation practices of Wikipedia (how, where, when, and so on) in indexed health science literature, our study adds insight into the role that Wikipedia may play in academic literature. We also provide information about the quality of the journals that cite Wikipedia, measured through impact factors from the Journal Citation Reports. The study searched citations from two well recognised databases (Scopus and Web of Science) and included all English language journals available in full text at Canadian university libraries, so some citations may have been excluded from the search because of the search criteria. However, the study used two comprehensive databases that probably captured most citations from journals with reaching influence as well as high impact factors. Although we report an increasing number of Wikipedia citations, the health science literature is expansive; therefore, papers that have cited Wikipedia remain a small minority of all published papers, albeit with increasing frequency.

Comparison with other studies

Active research on Wikipedia has examined the role of academic citations in Wikipedia, content validity, and interactions between Wikipedia and user groups.5 The only other study to examine Wikipedia in the scholarly references was completed by Park in 2011.30 In this bibliometric analysis, Park also used the ISI Web of Science and Scopus databases to identify the total number of studies (n=1746), as well as leading authors, their institutional affiliations, most frequent publication sources, main academic fields, and other statistics on the frequency with which scholarly articles cite Wikipedia. Our study took Park’s analysis further. Firstly, we used Ulrich’s database to refine search findings to be specific to health sciences literature. Secondly, we examined how Wikipedia is being cited. Both studies show a general increase in frequency of citations over time, suggesting that if Park’s data were to extend beyond 2010 they would continue to show increasing Wikipedia citations. While Park identified the publications that cited Wikipedia most often, our study also provides impact factors, adding information about journals’ impact.

Contextualisation and policy implications

Just because more researchers are citing Wikipedia does not necessarily justify it as a valid source of information for citation. Wikipedia itself has cautioned against using Wikipedia as a source,33 and some universities have gone so far as to ban students from citing Wikipedia.34 Recognising that learning modalities are changing and evolving increasingly towards online and e-resources,21 23 35 36 we believe that ensuring that peer reviewed academic literature aligns with the International Committee of Medical Journal Editors (ICMJE) guidelines is still important. Relevant to this study, the ICMJE guidelines state that “Readers should therefore be provided with direct references to original research sources whenever possible.”29 Although people have been argued that Wikipedia is comparable to an encyclopedia in terms of accuracy,8 37 this argument overlooks the literary problems associated with citing a tertiary source intended to direct researchers to an appropriate reference.34 We echo the findings of other researchers who believe that Wikipedia should not be cited when a more authoritative—that is, primary, permanent, peer reviewed, evidence based—source exists6; however, Wikipedia may be the most appropriate source to cite for a definition of Wikipedia and in situations in which Wikipedia is used as part of the scientific methods (for example, a search strategy). Outside of those rare instances, arguing that citation of Wikipedia in the academic literature is appropriate is difficult. Our “original research” category further illustrates how Wikipedia was often cited when the authors could have cited an original research study. For example, one article stated that “Some researchers propose that vitamin D supplementation may be beneficial in the treatment and prevention of some types of cancer,”38 and rather than citing the original sources provided on the Wikipedia page, the author instead cited Wikipedia.

Evidence based medicine requires that clinical decisions are made using professional expertise in conjunction with replicable evidence from systematic research.39 As Wikipedia entries are constantly changing, they may be difficult to access in the form that was originally cited by the time the work is published, reducing the reader’s ability to reproduce the author’s process.27 In a study by Peoples,6 most citations to Wikipedia did not reference the date and time at which the entry was visited. Although Wikipedia does have a history function that allows users to access previous versions of the page, this function may not be obvious to most users, and determining which version was accessed for citations that do not denote the date and time of the visit can be difficult. If Wikipedia is to be cited in the future, not only the date accessed but also the time should be included so that future researchers can access the entry as it was originally viewed. Thus, the process of peer review should include close attention to the references that appear in articles before recommending them for publication and potentially approving information that contradicts evidence based research.

Future directions

Although the proportion of retrieved Wikipedia citations in the indexed health science literature was relatively small, the increasing trend of citation is still important to note. The study of how Wikipedia is used in the academic literature is relatively young, and beginning an open discourse around the appropriateness of using non-permanent Wikipedia citations in the academic literature seems timely. The relationship between academic publication and Wikipedia remains largely understudied, and international guidelines such as those of the ICJME, World Association of Medical Editors, and Council of Science Editors lack editorial guidance on the subject. Although our study does not provide evidence of harm by these citations, even those that substitute for a permanent primary research source, we emphasise the need for a consistent voice on how Wikipedia should be used in the academic literature. This study begs the broader question of “what is an appropriate reference source?” Is it something that is generated by the scientific method, is replicable, and undergoes rigorous peer review or is it something that is collaboratively generated, readily open to editing, and broadly accepted? We call attention to the need to work with the Wikipedia community to establish guidelines not only for reviewers and editors but also for academics drafting publications.

Conclusion

An increasing number of peer reviewed academic papers in the health sciences are citing Wikipedia. The apparent increase in the frequency of citations of Wikipedia may suggest a lack of understanding by authors, reviewers, or editors of the mechanisms by which Wikipedia evolves. Although only a very small proportion of citations are of Wikipedia pages, the possibility for the spread of misinformation from an unverified source is at odds with the principles of robust scientific methodology and could potentially affect care of patients. We caution against this trend and suggest that editors and reviewers insist on citing primary sources of information where possible.

The use of Wikipedia as a source of academic information has been debated since its origin, but it is increasingly cited in peer reviewed health science literature

Although studies have examined the content and vailidity of Wikipedia, no evidence has been published about how this resource is being used in health science literature

Although a few instances exist that may warrant using Wikipedia as a reference, Wikipedia is often cited when permanent, evidence based sources are available

Authors, reviewers, and editors should use caution when publishing articles that include Wikipedia citations