How well do Canadian media outlets convey medical treatment information?: Initial findings from a year and a half of media monitoring by Media Doctor Canada.

The popular media play a crucial role in communicating information about health treatments. By informing the public about new research findings, they can affect how medical treatments are perceived1 and, in so doing, influence their use.2 Although some medical reporting is driven by public interest, many stories are prompted by companies, universities and research groups who are promoting their work and hope to get favourable coverage of it from major media outlets.3-5 A recent analysis by public relations specialists argues that the shift from traditional advertising to public relations is one of the most dramatic changes in the marketing field in decades and has come about because of the perceived superiority of this approach.3

Most recent studies of media coverage of medical treatments have found that many stories about pharmaceuticals lack complete information, particularly details on the quality of evidence and on the reported benefits, harms and costs of treatments.6-7 Research in this area in Canada has led to recent calls for more direct and honest reporting of the results of research.3, 8, 9 In the last three years, three different services — Media Doctor Australia, Media Doctor Canada and Health News Reviews in the United States — have monitored hundreds of health treatment stories using similar criteria, posting results to their websites.

Media Doctor Australia, with input from medical media researchers in Canada and the United States, developed 10 criteria with which to analyze media stories about treatments.10 Our group, Media Doctor Canada, adapted these into a similar tool, dropping a criterion about the presence or absence of press releases and adding one about the quantification of harms from treatments. Table 1 shows the 10 criteria that we ultimately used.

Table 1

Media Doctor Canada rating instrument for assessing the quality of news stories about medical treatments

We assembled a team of 12 reviewers with backgrounds in clinical epidemiology, clinical medicine, evidence-based medicine and media studies from personal contacts and people who had been involved in an earlier study of medical media reporting in Canada.6, 11 Over an 18-month period between 31 May 2005 and 1 November 2006, Media Doctor Canada analyzed 87 unique stories dealing with treatments that appeared in 10 major English-language daily newspapers, a medical news service directed toward health professionals, and two web-based news services that are linked to two major broadcasters in Canada (CBC and CTV). Our method of evaluating stories followed the procedure used in Australia: Each story was rated Satisfactory or Not Satisfactory for each of the 10 criteria by two reviewers, and any disagreements were resolved by consensus. We felt that media coverage may be different for stories dealing with pharmaceuticals versus other forms of treatment, so we examined stories in these two categories separately.

Table 2 shows the percentage of Satisfactory scores for each of the 10 different rating criteria for all 87 stories about pharmaceuticals and other forms of treatment. Nearly all stories dealing with pharmaceuticals and other treatments were rated Satisfactory on two criteria: Novelty of Treatment (93.8%) and Disease-Mongering (89.7%). This is to say that the stories did not exaggerate or misconstrue the novelty of the treatment (saying something was new when it was not) or misrepresent the natural history of the disease (disease-mongering). For all other criteria, fewer than half the stories in either category (pharmaceuticals or other therapies) were rated Satisfactory. The largest difference in the rate of Satisfactory scores between the two categories was for Harms of Treatment: 49.2% for pharmaceutical stories versus 30.4% for stories about other treatments.

Table 2

Ranking and percentage of “Satisfactory” scores on assessment criteria

Our results were consistent with those reported by Media Doctor Australia, which also found the highest percentage of Satisfactory scores for Novelty of Treatment and Disease-Mongering.10 In addition, the lowest-scoring criterion in the Australian study was Costs of Treatment, which similarly ranked near the bottom in our study.

Despite the consistency between these two studies, there are a few possible weaknesses to our approach. Although we trained our reviewers to apply the different criteria consistently, we recognize that there may have been individual biases that affected the scoring. Furthermore, we may have missed stories during our daily screening and selection, particularly because we looked only at stories from the largest English-language Canadian newspapers and two broadcast news agencies. The quality of stories appearing in other media, especially small media outlets where there are no journalists trained in health reporting, may be different.

Some journalists may consider our 10-criteria rating system to be difficult to implement, but none who communicated to us about our evaluations of their stories raised this concern, and we have had many favourable reviews of our site.8 We recognize that many elements that go into making a “good” story from the point of view of journalists may not have been considered, but do not believe that these preclude accurate and informed reporting. Similarly, there are arguments that short news stories cannot accommodate all of the criteria we measure. However, this assertion has not been supported by previous work in which we have shown that there are few significant differences in the overall quality between short news briefs and longer stories.11

The quality and comprehensiveness of the coverage of health technologies by the lay media is an important topic to study.1, 12Although journalism outlets strive13 to do a competent job while juggling competing interests, there are few mechanisms by which to provide them with feedback on the quality of their coverage. Since there is evidence that providing timely feedback of performance data can improve practice standards,6, 14, 15 auditing or monitoring the quality of health reporting in the lay press and feeding reviews back to news outlets may improve the informative value of these stories.

An obvious question then to ask is, Has Media Doctor Canada made a difference in the quality of Canadian medical reporting? We have recorded more than 59,000 unique hits on our site, but because they are anonymous we have no concrete evidence as to whether we are in fact contributing to the education of journalists. In the future, we intend to continue to build our database of articles, to send scores back to the editors and journalists whose stories we assess, to broaden and refine the scoring instruments and to periodically systematically analyze the collected data to see if there have been any changes to the averages of the scoring criteria. One measure of success could be a finding of better reporting of the criteria where journalists are currently missing the mark, such as Quantification of Harms of Treatment. If this were coupled with focus-group interviews with journalists indicating that our site had a positive impact on their reporting, we could be confident that our endeavour was having a positive effect.