Does anybody read "evidence-based" articles?

BACKGROUND: The electronic version of the British Medical Journal (eBMJ) has a unique feature in that it provides an electronic record of the number of times an article has been viewed ("hits") in the week after its publication. We sought to compare the relative popularity of primary research and "evidence-based" papers against that of narrative reviews and editorials. We surveyed four broad groupings of articles in 2001: Editorials, Clinical Reviews (which are narrative reviews), Education and Debate, and Papers (which are original research articles and systematic reviews). Clinical Reviews were the most frequently viewed articles, with an average of 4148 hits per article, while Papers were less popular (average of 1168 hits per article). Systematic reviews (23 articles, average of 1190 hits per article) were visited far less often than narrative reviews. Editorials (average of 2537 hits per article) were viewed much more frequently than Papers, even where the editorial was written as an accompanying piece with a direct link to the paper. DISCUSSION: Narrative reviews and editorials are accessed more frequently than primary research papers or systematic reviews in the first week after their publication. These findings may disappoint those who believe that it is important for readers to critically appraise the primary research data. Although the technical quality of journal articles may have been helped by recommendations on structured reporting, the readability of such articles has received little attention. Authors and journal editors must take steps to make research articles and systematic reviews more attractive to readers. This may involve using simpler language, as well as innovative use of web resources to produce shorter, snappier papers, with the methodological or technical details made available elsewhere.
CONCLUSION: Primary research and "evidence-based" papers seem to be less attractive to readers than narrative reviews and editorials in the first week after publication. Authors and editors should try to improve the early appeal of primary research papers.

Background

One of the benefits of publishing on the Internet is that the numbers of visitors to a particular article can be easily recorded and displayed. The electronic British Medical Journal (eBMJ) website has a unique "Hit Parade" section, with web access statistics for every online article dating back to 1999.[1] This "Hit Parade" records the number of electronic visits ("hits") in the first week of the article's publication.

We were disappointed to find, for instance, that the editorial accompanying a meta-analysis we published in 2000 had been viewed nearly 4000 times whereas the paper itself had received only 1236 visitors.[2,3] This prompted us to ask what people read when they visit the eBMJ. Do they prefer narrative-style articles to original research or evidence-based publications?

We assessed the viewing figures in the Hit Parade for all issues, except the Christmas edition, of the eBMJ published in the year 2001. We divided the journal articles into the following four broad categories and recorded the number of hits for each article:

· Editorials

· Papers (including articles listed under Primary Care)

· Clinical Reviews

· Education and Debate

Some articles are accompanied by editorials in the same issue. We reviewed these linked editorials and articles, and compared their numbers of hits.

Finally, we looked at review articles to see if they were narrative reviews or systematic reviews (including meta-analyses), and compared their numbers of hits.

The total number of hits for the electronic BMJ in the year 2001 was 5.38 million, giving a mean of 108 000 per issue (range 38 000 (Aug 4) to 141 000 (Oct 13)). Table 1 shows the distribution of hits within the four main sections of the journal, as well as the average number of hits per article in each area. Clinical Reviews were by far the most visited articles, and this would be true even if the comparison had been extended to cover all types of articles in the eBMJ. The top five articles for Clinical Reviews and Papers, by numbers of hits, are listed in Table 2 (see Additional file: 1 for reference list).

Table 1

Hits by category of article. The total number of hits for the BMJ in 2001 was 5.38 million.

Category	Number of hits (% of total for 2001)	Number of articles	Average hits/article
Editorial	652022 (12.1)	257	2537
Clinical Review	514352 (9.6)	124	4148
Education & Debate	262246 (4.9)	116	2261
Papers	353945 (6.6)	303	1168

Table 2

Top five articles in the Clinical Reviews and Papers section for 2001. References for the articles here are given in the additional file.

	Title of Article	Hits
Clinical Reviews
1	ABC of diseases of liver, pancreas, and biliary system: Gallstone disease.	21342
2	ABC of diseases of liver, pancreas, and biliary system: Investigation of liver and biliary disease.	19089
3	Recent advances: Geriatric medicine.	16917
4	ABC of diseases of liver, pancreas, and biliary system: Chronic viral hepatitis.	16517
5.	ABC of diseases of liver, pancreas, and biliary system: Acute hepatitis.	16255
Papers
1	Do doctors position defibrillation paddles correctly? Observational study.	7464
2	Ethical debate: Vaccination against mumps, measles, and rubella: is there a case for deepening the debate?	6001
3	10-minute consultation: Prostatic symptoms.	5048
4	Are "tomorrow's doctors" honest? Questionnaire study exploring medical students' attitudes and reported behaviour on academic misconduct.	3817
5	10-minute consultation: Dyspepsia.	3775

Systematic reviews compared with narrative reviews

During 2001, a total of 23 systematic reviews were published in the BMJ, all in the Papers section, with a total hit count of 27 390 and an average of 1190 per article. Clinical Review articles, which are all narrative reviews, had substantially greater number of visitors (average 4148 per article) than the systematic reviews. The average hit count for systematic reviews was much more similar to the overall count for Papers (average 1168 per article) than to the overall count for Clinical Reviews.

Articles and accompanying editorials

There were 69 articles (57 Papers, 5 Clinical Reviews, 7 Education and Debate) that had accompanying editorials. In almost all instances (54 of 57), articles in Papers were viewed less frequently than their accompanying editorials; the hit count differences ranged from 99 to 4787. Only three had a higher hit count than their linked editorial. On average, editorials were visited 2.9 times more frequently than the Papers articles to which they were linked.

Discussion

Visitors to the eBMJ prefer review articles and editorials to the new scientific work that is usually presented in the Papers section. One explanation may be that readers, who are pressured for time, prefer to read the evidence summarised in a review article or editorial. If this explanation were true, one would expect systematic reviews to have roughly the same number of viewers as narrative reviews. However, the most visited systematic review only managed 3179 hits, a figure that was easily surpassed by 50% of the narrative reviews. This may prove to be disappointing news for supporters of evidence-based medicine, such as the Cochrane Collaboration, who have often pointed out the many biases and weaknesses in traditional narrative reviews.[4]

While Editorials were, on average, not as popular as Clinical Reviews, they were still visited more often than articles in the Papers section. The most worrying aspect of this comes from those articles in Papers with linked editorials. Substantial numbers of readers viewed the editorial, but did not go on to visit the research article to which the editorial referred. This raises serious questions as to the role of editorials in the journal. Surely the aim of the accompanying editorial is not to act as a substitute for the research article, but to draw the attention of readers to the gems inside that particular issue?

Editorials also act as opinion pieces, providing a certain slant or interpretation (potentially either supportive or critical) of the data published in the research article. Readers who concentrate solely on the editorial, without consulting the original article, may well come away with an incomplete view of the research findings. Journal editors should bear this in mind when planning and accepting editorials for publication.

Limitations of this study

This is a retrospective, observational study looking at the numbers of times an article was electronically accessed in the first week of its publication. As such, it evaluates the initial appeal or impact of an article soon after publication. This appeal, at its most basic level, is what draws readers to click on a web link in that week's release of the eBMJ. Thus, it may only reflect the topical, immediate interest of readers, somewhat in the manner of a daily newspaper. We cannot assume that this bears any connection to the long-term impact or influence of the article, for example with regards to how many times it is cited by future work. This is something that, with the passage of time, we hope to be able to evaluate. It is also worth stating that we do not know how much (or how little) of the article was actually read after being "clicked" upon.

As this is an observational study, we must also consider the role of confounding factors that might have influenced the results. For example, it is conceivable that certain types of articles, such as systematic reviews, were less popular because they were not highlighted in "This Week in the BMJ"". In fact, "This Week in the BMJ" concentrates almost exclusively on promoting primary research articles, and all 23 systematic reviews were indeed listed there. In contrast, not one of the narrative reviews which made up the Top 5 most visited papers list (Table 2) was highlighted in "This Week in the BMJ".

One might also argue, from looking at the gastroenterological bent of the Clinical Reviews in 2001, that it may have been the content, rather than the type of article, that accounts for the relative popularity of Clinical Reviews. While there may be some truth in this argument, we note that a systematic review of interferon in chronic Hepatitis C (November 17 2001) scored only 1118 hits.[5] This contrast sharply with the linked Editorial (5839 hits), and the Clinical Review on treatment of chronic hepatitis (4575 hits) which were published in the same issue.[6,7]

There is other evidence to show that the appeal of the eBMJ's Review articles is not limited to specific topics. We looked at a larger sample of Clinical Reviews (from the year 2001 back to 2000) and found that a wide range of topics including obstetrics, intensive care, heart failure, palliative care, breast cancer, and vasculitis all managed to attract hit counts exceeding 10 000. Articles in the eBMJ, despite appearing different in subject matter, do actually share a common thread – the content of all the articles have been judged (by the editorial team) to be of interest to a general readership.

It is clear that the eBMJ (which is currently accessible without charge) aims to cater to a diverse audience, and our findings cannot be generalized to other journals, especially those which are specialized and/or subscription-based. For example, the readership probably includes a larger proportion of lay people than specialist journals, and this may be a major determining factor in the popularity of narrative reviews and editorials. Conversely, it is possible that primary research articles may prove relatively more popular in specialist journals that cater to a restricted audience. In some ways though, any debate about the applicability of these findings to other journals is moot. Web readers commendably have full access to eBMJ articles without charge, and the electronic version received over 5 million hits from all over the world in 2001 – it must in its own right be regarded as an important, internationally accessible source of medical information.

Why are narrative reviews and editorials more popular than original research papers or evidence-based articles?

Although our study does not formally evaluate the factor behind the popularity of certain articles, it is interesting to speculate on some of the possible reasons. We wonder if accessibility and readability may play an important part. The eBMJ is a general medical journal with an audience spanning across varying levels of medical knowledge. Busy readers browsing through the weekly eBMJ may not have the expertise, time or inclination to critically appraise and interpret the data from a particular research paper or systematic review. Editorials and narrative reviews are an attractive "quick fix" in that they can provide:

• "expert" appraisal on study strength and weaknesses

• succinct interpretation and summary of the data

• personal opinion or recommendation on what one should do in real-life clinical practice

In contrast, the strengths of evidence-based articles i.e. the methodological rigor and minimization of bias, may also happen to be their Achilles heel. Some readers might find such articles unappealing because of their rigid structure and the apparent detachment from everyday clinical practice. While systematic reviews might say "Here is the evidence", narrative reviews go one step further in satisfying the audience by saying "This is what you should do with it". Some would also argue that "evidence-based" articles have had every ounce of personality expunged from them in order to conform to recognized quality criteria. For example, reports of randomised controlled trials are bound by structured reporting criteria (CONSORT), which are designed to ensure that the reporting meets high technical standards.[8] Similarly, systematic reviews, in accordance with the published QUOROM statement, usually contain a wealth of methodological and statistical detail.[4] While this improves the technical merit of such articles, we wonder if it may also serve to discourage readers who find such immersion in detail a shock to their senses. The important question here is whether these methodological improvements in reporting are associated with any decrement in readability.

How can we make original papers and evidence-based articles more attractive to readers?

It is clearly in the scientist's interest to disseminate his or her results as widely as possible, and to ensure that what is published does become read. There are a number of steps that we believe would make a considerable difference:

Changing the style

High technical quality need not be incompatible with accessibility or readability. The "Summary for Patients" section published by the Annals of Internal Medicine is a model example of how complex technical and clinical details can be conveyed in an easily comprehensible fashion. Journals with dedicated editorial staff should be able to provide specific input into ensuring that articles are written, wherever possible, in "plain-English". For instance, the Cochrane Collaboration has published a "Style Guide" which advocates the use of plain English in their systematic reviews.[9] High-flying research doesn't have to be reported in an impenetrable manner – unless, of course, the authors believe that the accompanying mystique lends some additional credibility to their work.

Changing the format

The advent of web-based publishing opens up useful new possibilities in the formatting of the article. For instance, the eBMJ has recently introduced "abridged-text" versions which report on the key points of the article.[10] Readers who wish to explore further have the option of visiting the "full-text" version, as well as choosing to view (electronically) additional files that describe the method or results in greater depth. Such additional files may be used to store technical details (e.g. systematic review search strategies, reasons for patient withdrawal from trial) that are required in the quality checklist. It would be interesting to see if the readability (and hit count) of systematic reviews could be improved by using a shorter, snappier format, but with the relevant methodological data easily accessible from a web repository.

Finding out what readers want

The main questions here are:

• How do we make original research articles and systematic reviews more attractive to readers?

• What is it that puts them off?

• How can we encourage them to use both primary research articles and narrative reviews in tandem?

Unfortunately, the tremendous progress made in enhancing the quality of the evidence base may not have been accompanied by improvements in the readability of research articles. For instance, Weeks and Wallace evaluated 110 research papers (including 42 from the eBMJ in 2001) and concluded that almost all of them were extremely difficult to read.[11] The visual appeal of articles in academic journals could also be enhanced by the increased use of colour graphics and larger font sizes.[12] Perhaps we should be aiming, not just for higher quality, but also to make research articles more attractive to readers.

Finally, an important point to consider in the electronic era – the availability of web access opens up journals to a whole new group of readers, of varying scientific backgrounds. The Cochrane Collaboration, for one, is particularly keen on consumer involvement in its projects. In the past, a specialist journal may have attracted a limited, well-defined audience, but easily accessible electronic initiatives (such as Biomed Central) mean that increasing numbers of non-specialist readers will be taking a peek at articles that they would not have normally encountered. Perhaps we should ask these readers what they would prefer the style and format of research articles to be?

Here, we believe it is apt to close the debate with Laurance's sobering reminder – "No matter how great the discovery or how important the revelation, a piece is worthless unless it is presented in a way that makes the reader want to read it." [13]

Summary

Implications for authors

Although publication of original research articles is favoured by academics, authors who hope to quickly attract a large audience should concentrate on producing review articles and editorials.

Implications for journal editors

Narrative reviews are by far the most widely viewed articles in the first week after publication, and editors who are trying to boost the on-line popularity of their journal should consider publishing more such reviews, at the same time being aware of the potential weaknesses of this type of article. Editors should consider simplifying the style or formats of original research papers in order to attract a larger proportion of the audience.

Implications for supporters of evidence-based medicine

Steps need to be taken to make original research articles and systematic reviews more interesting and accessible to readers. There should be research on improving the readability of such articles, while at the same time retaining their technical merits.

Competing interests

None declared.

Authors' contributions

SD extracted and analysed data, and critically revised the manuscript. YL came up with the idea for the study, analysed the data and prepared the manuscript.

Both authors read and approved the final manuscript.

Pre-publication history

The pre-publication history for this paper can be accessed here:

Click here for file