Analysis of health stories in daily newspapers in the UK.

OBJECTIVES: To assess what makes a good piece of medical reporting in newspapers, to quantify what is being reported on in the major British newspapers, to identify the sources of the news threads, and to assess how these are reported. STUDY
DESIGN: Development and validation of a tool to assess the quality of reporting of health-related articles, and assessment of the quality of these articles in British newspapers.
METHODS: Eight national daily newspapers from the UK were reviewed for 20 days over a 2-month period. All articles reporting newly emerging research pertaining to health in humans were included and reviewed independently by two raters. A descriptive analysis was performed. Subsequently, a quality assessment tool for use by a non-expert was developed and validated to objectively assess the quality of a newspaper article on a health-related topic.
RESULTS: The quality assessment tool was found to have good internal consistency and inter-rater reliability. The Daily Mail published almost twice as many articles as its nearest rival, The Daily Express, and over eight times as many articles as The Guardian. Articles in The Times were, on average, more than twice as long as those in The Sun and The Daily Telegraph. The highest quality articles were in The Times and The Independent, with the lowest quality articles in The Sun. The quality scores of anonymous articles were significantly lower than those attributed to named journalists.
CONCLUSIONS: There are significant differences in the quality of reporting within and between major daily UK newspapers, with anonymous articles being the poorest quality, and widespread reliance on press releases from the major UK scientific journals.

Introduction

Although newspaper circulations are declining, they continue to be one of the leading sources of information for the public. Collectively and individually, they have the power to change public perceptions on health-related issues. Their coverage of these topics has been attracting increased scrutiny in recent years.2, 3 For example, Goldacre has drawn attention to what he suggests may be a project by The Daily Mail to ‘divide all the inanimate objects in the world into the ones that either cause or cure cancer’.

There have been a number of projects aimed at assessing the quality of medical reporting, such as the NHS ‘Behind the Headlines’ series. Poor medical reporting can have serious consequences; Hargreaves et al. criticized journalists for their contribution, at least in part, to the controversy over the measles–mumps–rubella vaccine for failing to challenge misleading claims.

Previous studies have examined what gets into the press, what types of material are covered,7, 8, 9 the sources from which newspaper articles were derived, the contrast between messages in the scientific and lay press, whether print media exaggerate scientific advances, the role of press releases,12, 13 and how journalists use the medical literature to produce newspaper articles. There are also a number of excellent resources describing the quality of reporting, including the Australian Media Doctor initiative (www.mediadoctor.org.au) and the US Health News Review (www.healthnewsreview.org).

This study aimed to assess a cross-section of all of the newspaper articles related to health published over a 2-month period in order to explore factors which may be associated with good and poor reporting, and to explore the newspapers' sources of their stories and the differing editorial styles. Additionally, the development of a novel instrument for objective assessment of the quality of newspaper reports of medical literature is described, together with its use to compare newspapers.

Methods

Cross-sectional analysis

Eight national daily newspapers were examined, comprising two ‘popular’ tabloid newspapers (The Sun, The Daily Mirror), two ‘mid-market’ tabloids (The Daily Mail, The Daily Express) and four broadsheets (The Times, The Daily Telegraph, The Guardian, The Independent). The newspapers with the leading circulation in each category were selected as defined by the Audit Bureau of Ciculations, and consistent with previous research on health journalism. For 20 weekdays over a 2-month period (6 December 2010–17 January 2011), print editions of the newspapers were collected. The 2-week period around Christmas was excluded to reduce any bias relating to seasonal news stories. The print edition of each newspaper was scanned by a single author from cover to cover, and each article pertaining to human health and medicine that reported newly emerging results was included. A selection of papers from each day was subsequently scanned by another author to ensure that any relevant articles were not missed. Features from the health supplements, often containing reviews, articles and plans for future research, were excluded.

The original papers on which the articles were based were then identified using an online database (PubMed) or, where insufficient information was given in the newspaper article, by searches on journal or university websites. Corresponding press releases were obtained from the EurekAlert! website (www.eurekalert.org) and, where necessary, from university or journal websites. The content of each newspaper article was compared with the original research paper.

Descriptive data about each newspaper article were also recorded, including the newspaper and reporter's name. The number of column inches of each article was calculated according to the industry standard (total number of square inches on the page, including headline and associated pictures and graphics).

Using the Centre for Evidence Based Medicine's levels of evidence (www.cebm.net), each newspaper article was also allocated to one of four ‘grades of recommendation’, with reference to the original journal article. If the original article could not be found (often indicating the article was derived from an unpublished report or conference press release), it was allocated to the lowest level of evidence.

The volume of newspaper coverage was compared with the burden of disease in the UK. To do this, each newspaper article was allocated to categories featured in the Tenth Revision of the International Classification of Diseases (ICD-10) according to the keywords in the headline. A large number of articles were not directly related to a disease process, but could still be classed as ‘health news’. The majority of these were related to the aesthetic effects of ageing or to general interest psychology, classified as either ‘age and beauty’ or ‘popular psychology’. The breakdown of stories by ICD-10 category was subsequently compared with mortality data from the Office of National Statistics, the Quality and Outcomes Framework database on disease prevalence, and the Health Survey for England 2010.

Development and validation of the quality assessment tool

The Guidelines on Health and Science Communication 2001, published by the Royal Institution, Royal Society and the Social Issues Research Centre, were used as a starting point for development of the assessment tool, providing both stylistic and substantive guidance.

The authors aimed to develop a tool that could discriminate between satisfactory, poor and excellent articles. Certain features were deemed to be essential (if the article did not meet these criteria, marks were deducted) and others desirable, whereby extra marks could be gained. A full breakdown of the marking scheme can be found in Appendix 1 (see supplementary online material).

The quality assessment tool was piloted on 15 newspaper articles from the selected newspapers over 2 consecutive days. A number of criteria were subdivided with new criteria added, finally yielding a 22-item tool grouped into seven broad categories: credibility of sources, context of the study, significance of the findings, communication of risk, anticipation of impact, editorial content, and specialist second opinion. From a baseline of zero, the maximum score achievable was 17 and the minimum was −12.

Two authors used the instrument independently to score each of the newspaper articles identified during the study period. Any significant divergence between the two authors was identified, the articles were discussed and consensus was reached. In order to test face validity, a third author independently evaluated 10% of the articles selected at random, making a global judgement on quality on a five-point scale, which was compared with the scores generated by the scoring instrument.

Statistical analysis

Statistical Package for the Social Sciences Version 19 (SPSS Inc., Chicago, IL, USA) was used for statistical analysis. For the cross-sectional study, univariate and two-way analysis of variance were performed, with pair-wise comparisons performed using Bonferroni's correction (α = 0.005). Multivariate regression was conducted to explore which factor was the best predictor of higher overall scores. A P-value <0.05 was taken to indicate significance.

For validation of the assessment tool, consistency was measured using Cronbach's alpha. The intraclass correlation coefficient of the scores of the two authors was calculated to test the agreement. Given the large number of categories (n = 22) and range of scores, this was felt to be a more appropriate test of agreement than using Cronbach's kappa. Face validity was assessed by comparing the agreed scores by the raters with the global scores of an independent rater, whereby 10% of the newspaper articles were chosen at random (using a random number generator) and rated on a global scale of 1–5, with 5 being the highest. Principal component analysis was conducted for all 22 items in the assessment tool using Varimax rotations, with items with factor loadings <0.50 removed. Further details of the statistical analysis are available on request from the corresponding author.

Results

Descriptive analysis

In total, 160 newspaper articles met the inclusion criteria, with an average of eight health stories published per day in all newspapers. As expected, the most prolific newspaper was The Daily Mail (53 individual articles), accounting for 32% of all articles. The Guardian published the fewest articles (six individual articles). Analysing the newspapers by type, the two mid-market papers published the most articles (n = 76), followed by the four broadsheets (n = 62) and the two popular tabloids (n = 21). Thirty-one different reporters contributed to the 160 articles, with an average of six articles per reporter. One of The Daily Mail reporters was the most productive, writing on average two articles per day. No reporter's name was given for 49 (30%) articles.

The number of column inches per article ranged from 1 to 60, with an overall mean of 14.1 [standard deviation (SD) 11.6]. The number of column inches dedicated to medical and health news differed significantly between the newspapers [F(7,158) 3.75, P = 0.001]. Post-hoc analysis showed that The Sun published relatively short articles (mean 8.12, SD 10.9). However, surprisingly, The Daily Telegraph also published articles with fewer column inches than the other newspapers (mean 9.61, SD 9.01). The Times dedicated the most column inches to health stories (mean 22.16, SD 18.04), although this was not significantly higher than the other newspapers. There was no significant difference between the mean column inches by type of newspaper. For both the mid-market and broadsheet newspapers, photographs, diagrams and illustrations contributed substantially to the total column inches.

Comparison with disease burden

There was no significant difference in the medical categories covered by the newspapers [F (7,153) = 0.590, P = 0.763]. The disease categories covered and their corresponding share of mortality in the UK are shown in Fig. 1 ; the most frequently reported ICD-10 category was cardiovascular disease, accounting for 16% of the articles published and 32% of deaths in UK. Fourteen percent of newspaper coverage was on conditions with no ICD-10 category (classified as ‘age and beauty’ and ‘popular psychology’). The Daily Mail published nine articles of this type, the same as the number of articles on cancer.

Fig. 1

Comparison of the top five ICD-10 categories by mortality (percentage of all deaths) and morbidity (percentage of total prevalence) with newspaper coverage (percentage of total column inches) in the UK. Based on data from the office of national statistics, quality outcomes framework (QOF) and the health survey for England 2001.

Only one article pertained to respiratory diseases, despite asthma affecting 8% of the population. Similarly, there was no newspaper coverage for musculoskeletal conditions, although it is estimated that up to 20% of adults will see their general practitioner about a musculoskeletal condition each year.

Level of evidence

Only nine of the 160 studies were judged to be ‘Grade A’ (derived from studies such as randomized controlled trials or systematic reviews). The majority (n = 104) could only be judged ‘Grade D’ (derived from expert opinion, bench research or first principles). These articles were found most frequently in The Daily Mail (n = 37) and The Daily Telegraph (n = 23). However, ‘Grade D’ articles also accounted for 83% of The Guardian's output (five out of six articles). The Daily Mail had the largest percentage of Grade A and B articles (43% of output), followed by The Daily Express with 38%. In comparison, only 6% of The Times articles were derived from articles judged as ‘Grade A or B’ evidence.

Sources of news stories

The source of the research paper (journal or conference) was cited in 69% of newspaper articles. Forty-three different journals were cited. The British Medical Journal was the most frequently cited source (15 separate newspaper articles), with The Lancet cited in 11 articles (Fig. 2 ). This corresponded to previous research finding that these leading British journals are the favoured source for health news by reporters. The third most frequently cited source was conference reports or press releases from individual laboratories or research groups (eight newspaper articles), in joint place with Science: Translational Medicine.

Fig. 2

Total number of times each journal was cited in the 161 newspaper articles. The 14 most-cited journals out of 42 are displayed.

A single journal article was often reported in multiple newspaper stories; of 160 newspaper articles in the study, 93 (58%) were derived from a journal article that was reported in at least one other newspaper. This finding is again consistent with previous research.2, 3

Quality analysis of the newspaper articles

The scores generated by the quality assessment tool were normally distributed (mean 5.53, SD 4.50). Eleven percent of articles scored <0 (predesignated as ‘unsatisfactory’) and 13% scored >10 (predesignated as high quality). The mean scores for each newspaper differed significantly [F(7,153) = 4.06, P = 0.004]. The newspaper with the lowest average score was The Sun [mean 1.5, standard error (SE) 1.209], and the newspaper with the highest average score was The Times (mean 8.67, SE 1.209) (P < 0.007) (Fig. 3 ).

Fig. 3

Boxplot showing the mean and range of scores, derived from the novel scoring instrument, for each newspaper.

Controlling for column inches, there was a significant difference in mean scores between the types of newspaper [F(2,81) = 3.34, P = 0.04], with a post-hoc test finding a significant difference between the broadsheets (mean 7.34, SE 0.489) and mid-market tabloids (mean 6.66, SE 0.476) compared with the popular tabloids (mean 2.75, SE 8.45).

Articles with a named reporter attained a significantly higher score (mean 5.54, SD 4.45) than anonymous articles (mean 0.75, SD 0.46) [t(159) = 5.43, P < 0.001]. Forty-six percent (n = 15) of The Daily Telegraph's articles and 35% (n = 19) of The Daily Mail's articles were anonymous, compared with no anonymous articles in The Independent and The Guardian, and just one anonymous article in The Times.

Articles attaining lower scores were often derived from specialist journals. A significant correlation was found between the impact factor of the journal from which the article was derived and the quality score [r(111) = 0.247, P < 0.01], with articles based on journals with a high impact factor achieving higher scores.

In a multivariate regression, the only significant predictor of a higher quality score was the length of the article (in2) [β = 0.49, t(112) = 5.14, P < 0.001; R 2 = 0.29, F(4,112) = 12.56, P < 0.001].

Validation of the quality assessment tool

The quality assessment tool was found to have adequate internal consistency (Cronbach's α = 0.76). The intraclass correlation coefficient (to assess the agreement of the raters) was α = 0.952 (P < 0.001), with an 87% match in the total scores for each rater. A significant correlation was found between the total scores and those made by an independent rater on a sample of articles (r = 0.74, P = 0.001), indicating good face validity.

Item reduction

Using a factor analysis, 12 items in the tool were excluded due to inadequate loading and poor internal consistency. A three-factor solution was found (Table 2 ).

Table 2

Theoretical groupings of quality assessment instrument criteria after item reduction and factor analysis.

Theoretical group	Criteria
Safety	Caveats explored, safety explored, specialist opinion, comparison of risk, context
Provenance	Citation of journal and author, background explored
Emotion	No undue anxiety caused, representative headline

The shortened scale (see Supplementary online material) had a maximum score of 9 and a minimum score of −7. Univariate analysis of variance produced results consistent with those using the long version of the scale, with a significant difference between scores for each newspaper [F(7,153) = 4.56, P < 0.001] (Table 1).

Table 1

Summary of the descriptive analysis, by newspaper.

Newspaper	Number of articles published	Column inches		No author listed	Quality assessment instrument score
		Mean	SD		Mean	SD
The Sun	12	8.13	10.95	6	1.5	4.99
The Daily Mirror	9	15.06	13.70	2	2.7	3.03
The Daily Mail	53	12.56	8.23	19	5.87	4.14
The Daily Express	24	20.88	13.93	6	6.13	3.91
The Guardian	6	12.75	7.22	0	7.08	4.48
The Independent	13	16.65	9.91	0	7.58	3.42
The Daily Telegraph	32	9.61	9.01	15	4.52	4.70
The Times	12	22.16	18.04	1	8.67	4.00
Total	161			49

SD, standard deviation.

Discussion

The Daily Mail published the most health-related articles which, given its very large circulation, gives it enormous potential to influence health behaviours. The Daily Mail publishes almost twice as many articles as its nearest rival, The Daily Express, and over eight times as many articles as The Guardian. Articles in The Times are, on average, more than twice as long as those in The Sun and The Daily Telegraph.

The volume and quality of reporting health research varies between British newspapers. Using a newly developed tool to assess the quality of reporting, this study found that the highest quality articles were in The Times and The Independent, both of which achieved consistently high scores. Among the broadsheets, The Daily Telegraph was more prolific, publishing three times as many articles as The Times or The Independent, but many of these were short, anonymous articles that seemed to have been taken from the newswires. Only The Independent and The Guardian had no anonymous articles. Unsurprisingly, short articles taken directly from press releases scored lower as there was rarely any background information, risks and caveats were not explored, and a second opinion was not included. The significant correlation between column inches and scores is intuitive; longer articles have more scope to meet the criteria, such as discussion of caveats and giving an adequate background. Thus, it was unsurprising that The Times had the highest average quality as it also had the longest articles on average.

The quality of articles correlated with the impact factor of the journals from which they emanated. This may, in part, reflect the study criteria; articles derived from journals with higher impact factors were more likely to report randomized controlled trials or large-scale studies, which will yield more information that will be captured by the tool. Another possibility is that the quality of the press release may be more informative in journals with a higher impact factor, thereby increasing the quality of derived articles.

While cardiovascular diseases and cancer made up the bulk of coverage, reflecting the overall mortality in the UK, there was relatively little coverage of conditions that affect many people but lead to few deaths. A particular problem arose where small-scale laboratory studies were taken out of context and transformed into outlandish claims about their implications for the general population.

Any conclusions in this area are subject to a number of caveats; only one measure of morbidity was used, which may not provide an accurate representation of the importance of the disease to an individual. Also, the newspapers may be reflecting the salience that particular diseases have with the general public, such as diseases of the newborn, which constitute a relatively low proportion of overall morbidity and mortality. There may, of course, be a cyclical relationship with what the public perceives as important and the type of articles that the newspapers are publishing. The newspapers could argue that diseases such as back pain and asthma do not provoke such strong feelings as those associated with cancer, and could argue that there have been few ‘ground-breaking’ discoveries recently.

Newspapers may also be reluctant to cover articles where a perception of fault or blame lies with the ‘sufferer’, such as with smoking-related diseases, and prefer the more emotive story of a presumably innocent person's ‘fight’ or ‘battle’ against an injudicious disease. However, the tabloid press do not shy away from ascribing fault or blame to a particular group of people when engaging in social commentary.

This is, to the authors' knowledge, the first attempt to compare, systematically, the volume and quality of health reporting in British newspapers. Consequently, the main findings summarized above add materially to what is already known about media coverage of health reporting. However, this study has also contributed to the methodology of assessing the quality of newspaper articles, operationalizing existing guidelines to create a novel instrument, demonstrating its reliability, and undertaking a process of item reduction to identify the core underlying constructs that can be used in future research (whether caveats are explored, whether the safety of a drug/intervention is described, whether there is a quotation from an independent specialist, whether the article is placed in context and whether risks are compared).

The study had a number of limitations. The raters were not blinded to the newspaper in which each article was published. One possible effect may have been to moderate the score of a poor article in a normally well-performing newspaper.

This study focused on print newspapers as they are static, making analysis easier. However, the Internet (e.g. the BBC news website) is becoming an increasingly popular source of news. During the course of the study, it was observed that the more bizarre headlines were moderated on the newspapers' websites. The articles were also modified throughout the day on the websites, presumably in response to criticism and further information. If the study encompassed internet news sites, it is likely that the overall standard would have been higher.

This study did not include the ‘free sheets’ that are read increasingly, with circulations higher than those of many national daily newspapers. After looking at a sample of these, the reports were found to be very close to the corresponding press releases, with little editorial input. The weekend papers were also excluded but, as these tend to feature more salacious news, it would be expected that the scores would generally be lower.

Although this study used column inches, consistent with previous research, word count offers an alternative measure. In some papers, diagrams added a great deal to the articles. In contrast, the large pictures attached to articles in the ‘mid-market’ and ‘popular’ newspapers often contributed little. No information was collected regarding the position of the articles in the paper (e.g. front page or later in the paper). This may have given more of an insight into the editorial reasoning behind the inclusion of certain articles.

This study was performed over a fairly short period during the winter. Given the seasonal nature of health problems, it cannot be assumed that the subjects covered would be generalizable to the whole year. However, the scale of effort required to cover an entire year would be enormous, and even then would be subject to the occurrence of important but one-off events, such as the emergence of severe acute respiratory syndrome (SARS).

All newspapers identified randomized controlled trials. However, on many occasions, they failed to differentiate between small and large trials. The instrument developed does include a criterion of ‘reporting results as a ‘breakthrough’ (when there was no strong evidence of one)’, although this did not prove to be significant in the factor analysis, perhaps because almost all articles were classed as such.

Finally, the quantitative approach clearly complements, rather than substitutes for, the more detailed qualitative assessments of specific stories cited above. In particular, there is scope to look in more detail at the genesis of stories and, in particular, hidden conflicts of interest, which receive more attention in the approach used by Health News Review.

Journals, researchers and academic institutions are increasingly reliant upon the mass media to raise the profile of their work. Newspapers are under increasing pressure due to declining sales and the growth of free sources of news, such as the Internet and ‘free sheets’. These findings suggest that medical journalism would be improved if newspapers concentrated on producing a smaller number of more detailed articles, and not be overly reliant on stories derived from press releases from specialist journals, taken out of context, many published as anonymous articles. This is likely to require dedicated health correspondents who are able to build relationships with scientists, gaining the trust of the readership and the scientific community, rather than trying to compete with the ever-increasing amount of ‘news’ available to the public.