Televised medical talk shows--what they recommend and the evidence to support their recommendations: a prospective observational study.

OBJECTIVE: To determine the quality of health recommendations and claims made on popular medical talk shows.
DESIGN: Prospective observational study.
SETTING: Mainstream television media. SOURCES: Internationally syndicated medical television talk shows that air daily (The Dr Oz Show and The Doctors).
INTERVENTIONS: Investigators randomly selected 40 episodes of each of The Dr Oz Show and The Doctors from early 2013 and identified and evaluated all recommendations made on each program. A group of experienced evidence reviewers independently searched for, and evaluated as a team, evidence to support 80 randomly selected recommendations from each show. MAIN OUTCOMES MEASURES: Percentage of recommendations that are supported by evidence as determined by a team of experienced evidence reviewers. Secondary outcomes included topics discussed, the number of recommendations made on the shows, and the types and details of recommendations that were made.
RESULTS: We could find at least a case study or better evidence to support 54% (95% confidence interval 47% to 62%) of the 160 recommendations (80 from each show). For recommendations in The Dr Oz Show, evidence supported 46%, contradicted 15%, and was not found for 39%. For recommendations in The Doctors, evidence supported 63%, contradicted 14%, and was not found for 24%. Believable or somewhat believable evidence supported 33% of the recommendations on The Dr Oz Show and 53% on The Doctors. On average, The Dr Oz Show had 12 recommendations per episode and The Doctors 11. The most common recommendation category on The Dr Oz Show was dietary advice (39%) and on The Doctors was to consult a healthcare provider (18%). A specific benefit was described for 43% and 41% of the recommendations made on the shows respectively. The magnitude of benefit was described for 17% of the recommendations on The Dr Oz Show and 11% on The Doctors. Disclosure of potential conflicts of interest accompanied 0.4% of recommendations.
CONCLUSIONS: Recommendations made on medical talk shows often lack adequate information on specific benefits or the magnitude of the effects of these benefits. Approximately half of the recommendations have either no evidence or are contradicted by the best available evidence. Potential conflicts of interest are rarely addressed. The public should be skeptical about recommendations made on medical talk shows. Additional details of methods used and changes made to study protocol. © Korownyk et al 2014.

Introduction

Mass media in the form of television, radio and printed material are frequently used to deliver medical information to the public. Research suggests that mass media can improve public knowledge1 and potentially improve health behaviors.2 Television is one of the most important mass media sources of health information.3 4 However, concerns have been raised about the quality, completeness and accuracy of medical information covered in the news media,5 6 7 8 and television news media is no exception.7 8 The quality of information outside of the news media has not been examined.

According to Nielsen’s report, American citizens spend an average of over five hours a day watching television.9 International health information programs, such as The Dr Oz Show and The Doctors have become a regular part of television broadcasting. In the 2012-13 season, The Dr Oz Show was consistently ranked in the top five talk shows in America with an average of 2.9 million viewers per day, while The Doctors had a high of 2.3 million viewers.10 11 In the 2012 Greatist report, Dr Mehmet Oz and Dr Travis Stork (one of the hosts of The Doctors) were both included in the top 100 health and fitness influencers.12

Popular television talk shows such as The Dr Oz Show often engender skepticism and criticism from medical professionals.13 14 15 However, no research has systematically examined the content of the medical information provided on these talk shows. Our objective was to review the most popular medical talk shows on television, to (1) determine the type of recommendations and claims given and the details provided, and (2) search for and evaluate the evidence behind these recommendations.

Methods

Programs

We searched for internationally syndicated medical or health television talk shows that aired daily (weekdays). Two internationally syndicated medical or health television talk shows (The Dr Oz Show and The Doctors) were identified by internet search. We prospectively recorded all episodes of The Doctors from 11 January to 1 May 2013 (79 episodes) and The Dr Oz Show from 7 January to 1 May 2013 (78 episodes).

Shows are typically divided into 3-4 topics (range about 1-5), each topic runs about 12-15 minutes, often divided by commercial breaks

Each topic may have 4-5 recommendations within it (range about 0-10)

Recommendations may come from the host, guest, or rarely audience members

Data collection and outcome measures

Three members of the research group (GMA, MRK, CK) reviewed two episodes of each show from the previous year (2012) to help inform the development of a data collection spreadsheet.

The primary outcome was the percentage of recommendations and claims (henceforth referred to as recommendations) supported by evidence as determined by a team of experienced evidence reviewers. Secondary outcomes included topics discussed, the number of recommendations made, the types and details of the recommendations, and general episode characteristics.

We had no previous information on which to base an estimation of the needed sample size to make a reasonable estimate of the number of recommendations that were evidence based. Based on direct observation of two of each of the shows, we hypothesized that 50% of the recommendations would be supported by evidence, then calculated that 158 recommendations would give an 80% chance that the 95% confidence interval would have a precision within ±10%. We estimated there would be at least two definitive or stronger recommendations per episode. Therefore, to attain 158 recommendations, we needed to review 80 episodes. Using a random number generator, 40 of the recorded episodes for each of The Dr Oz Show and The Doctors were randomly selected for analysis.

Being unfamiliar with broadcast health information, we were unaware of the non-specific nature of many statements and recommendations given on medical television talk shows. In addition, the content of the medical information provided on talk shows has not previously been assessed systematically. This required us to use an iterative approach to the study design, ultimately resulting in three small changes to the protocol and one larger change (see data supplement for full details). A flow chart of the final study is presented in figure 1.

Fig 1 Flow chart of study design

Classification and content of recommendations

Two reviewers (VL, KO) independently watched each randomly selected episode to document the topics discussed and the specific details surrounding recommendations. Further information on data extraction is included in the data supplement. To focus on stronger or clearer recommendations for the evidence review portion of the analysis, the reviewers attempted to delineate the more definitive recommendations. This was based on both the strength of the wording (for example, “Get your kids vaccinated” was considered a strong recommendation, whereas “Prescription retin-A helps with ear and other types of acne” was not) and the context in which the recommendation was made (for example, a recommendation may have been classified as strong if it was repeated multiple times). After independent review, data extraction was compared, disagreement of coding and classification was resolved by consensus or third party adjudication (CK, MRK, or GMA), and overall agreement was calculated. We found some of the initial classification of topics and recommendations included categories that were too broad. Therefore, two investigators (CK, JMcC) reclassified all topics and recommendations, again with a third investigator (GMA) resolving any uncertainties.

After preliminary review of the data, it was determined that more specific detail was needed to better quantify the type of information provided for all recommendations. Two new reviewers (CC, CF) independently watched the same randomly selected episodes and focused on the recommendations to identify if a benefit was described (such as “heart healthy”), if the benefit was specific (such as “reduces heart attacks”), if a magnitude of benefit was given (such as “reduced 5%”), if costs were mentioned (such as “this supplement costs $5 a month”), and if potential risks or harms were mentioned. In addition, the observers recorded any mention of potential conflict of interest with each recommendation or acknowledgement of potential conflicts of interest at any point during the show. After independent review, data extraction was compared, disagreement was resolved by consensus or third party adjudication (CK, MRK, or GMA), and overall agreement was calculated.

Development of searchable questions and the evidence behind the recommendations

We randomly selected 80 of the stronger recommendations from each show and developed a searchable question for each. Two healthcare providers from the group with medical literature searching skills (GMA, MRK, CK, JMcC, SG, AJL, HLB, DCS, RDT) independently searched for evidence supporting each recommendation. The searchable resources included the databases Pubmed/Medline, Embase, Cochrane Database of Systematic Reviews, Natural Standard Database, and Google. Maximum search time allotted was 1 hour per recommendation. If reviewers found high level evidence (a systematic review of randomized controlled trials or large randomized controlled trials) addressing the question or recommendation, the search was concluded. Original research that addressed patient oriented outcomes (such as mortality or quality of life) was assigned higher priority over surrogate markers or in vitro data. If several papers were available, reviewers selected the highest level evidence pertaining to the target audience of the shows (such as middle age adults versus pediatrics, and North American data). Reviewers recorded citations of all papers that were considered relevant and were encouraged to make notes regarding their search and evaluation of the evidence.

Originally each reviewer was also going to independently judge if the evidence was of sufficient consistency and believability to support the recommendations. However, challenges occurred with the non-specific nature of the recommendations and wide diversity of reviewer interpretation of the evidence (see data supplement). Therefore, the protocol was modified and four investigators (CK, MRK, JMcC, GMA) reviewed the evidence for each recommendation as a team. The investigators reviewed the recommendation and determined the ideal study design to answer the question but considered sources of evidence ranging from case reports to systematic reviews. While expert opinion is considered a level of evidence, these were excluded because all recommendations in the programs were, by definition, being made by experts. In cases where the answer remained unclear, an additional search was performed until the group was satisfied that a reasonable answer was found or that no evidence was readily available.

The evidence was evaluated for consistency and believability. Consistency was defined as both internal (within the study) and external (between studies). Believability was based on the quality, quantity, and type of evidence available. The group discussed each recommendation and voted on how believable (believable, somewhat believable, or not believable) the evidence was. When reviewing evidence, investigators used a relatively broad definition of support in an attempt to be as fair as possible (for example, see data supplement). The group also recorded how frequently “consult a healthcare provider” was mentioned in the 160 randomly selected stronger recommendations. All data are presented descriptively without statistical comparison. The 95% confidence intervals for the primary outcome were computed using the normal scores method.

Results

Program topics

Topic categorizations from the 80 randomly selected episodes (40 of The Dr Oz Show and 40 of The Doctors) are shown in figure 2. The most common show topic discussed on both The Dr Oz Show and The Doctors was general medical advice—32.0% (40/125) and 65.5% (74/113) respectively. The second most common topic on both shows, 24.8% (31/125) and 8.8% (10/113) respectively, was non-weight loss dietary advice (such as immune boosting diets). The combination of dietary advice and weight loss accounted for 43.2% (54/125) of all topics discussed on The Dr Oz Show and 16.8% (19/113) for The Doctors.

Fig 2 Prevalence of topic categories in 40 episodes of each of The Dr Oz Show (125 topics) and The Doctors (113 topics)

Program recommendations

We identified 479 recommendations for The Dr Oz Show (291 stronger recommendations) and 445 recommendations for The Doctors, (219 stronger recommendations). On average, each episode of The Dr Oz Show had 12 recommendations, while each episode of The Doctors had 11 recommendations.

Recommendation categories

The categories of the recommendations for both shows are shown in figure 3. The most common recommendation in The Dr Oz Show (39.2%, 188/479) was dietary advice (such as “Carb load your plate at breakfast”), while in The Doctors (17.8%, 79/445) it was to consult a healthcare provider (such as “Go to your primary care doctor or talk to their nurse before going to the ER to help relieve the load in the ER”). For the 160 recommendations that were randomized for evidence review, The Dr Oz Show recommended consulting a healthcare professional 9% (7/80) of the time compared with 33% (26/80) of the time on The Doctors.

Fig 3 Prevalence of recommendation categories in 40 episodes of each of The Dr Oz Show (479 recommendations) and The Doctors (445 recommendations).

Recommendation details

The details of benefits, harms, and costs around the recommendations are shown in table 1. The benefit of the recommendation was not specific 57.4-58.7% of the time. For example, a recommendation from The Dr Oz Show that vitamin E improves brainpower would be considered a benefit but not considered specific or measurable, and the magnitude of the increase in brainpower was not discussed. Magnitude of the potential benefit was mentioned in 11.0-16.5% of the recommendations, often in relative rather than absolute terms (for example, vitamin E in foods cuts risk of Alzheimer’s disease by 25-70%). Recommendations were made by the host(s) 26% (125/479) of the time on The Dr Oz Show and 65% (290/445) of the time on The Doctors. Guests made 65% (310/479) of the recommendations on The Dr Oz Show and 33% (146/445) on The Doctors.

Table 1

 Details of benefits, harms, and costs associated with each recommendation made in samples of medical television talk shows The Dr Oz Show and The Doctors

	No (%) of recommendations
	The Dr Oz Show (n=479)	The Doctors (n=445)
Benefit of recommendation mentioned	453 (94.6)	402 (90.3)
Benefit was specific	204 (42.6)	184 (41.3)
Magnitude of benefit mentioned	79 (16.5)	49 (11.0)
Possible harms mentioned	47 (9.8)	34 (7.6)
Cost mentioned	60 (12.5)	14 (3.1)
Potential conflict of interest declared or mentioned	1 time	3 times

Both the host(s) and guests made the recommendations 9% (44/479) of the time on The Dr Oz Show and 2% (7/445) of the time on The Doctors. Acknowledgment of a potential conflict of interest was identified four times over the 924 recommendations.

Evidence base for the stronger recommendations

Evidence assessment for 160 randomly selected recommendations is presented in table 2. Overall, we found that 87 of the 160 recommendations (54%, 95% confidence interval 47% to 62%) had some level of published evidence to support them. Believable or somewhat believable evidence supported 33% of the recommendations on The Dr Oz Show and 53% on The Doctors. We found believable or somewhat believable evidence against 11% and 13% of the recommendations on the The Dr Oz Show and The Doctors, respectively.

Table 2

 Evidence for 80 randomly selected recommendations from each of the medical television talk shows The Dr Oz Show and The Doctors

	Evidence believable	Number (%) of recommendations (n=80)
		By believability	Overall
The Dr Oz Show
Evidence agrees	Yes	17 (21.3)	47 (46.3)
	Intermediate	9 (11.3)
	No	11 (13.8)
Evidence disagrees	Yes	4 (5.0)	12 (15.0)
	Intermediate	5 (6.3)
	No	3 (3.8)
No evidence	—	—	31 (38.8)
The Doctors
Evidence agrees	Yes	26 (32.5)	50 (62.5)
	Intermediate	16 (20.0)
	No	8 (10.0)
Evidence disagrees	Yes	3 (3.8)	11 (13.8)
	Intermediate	7 (8.8)
	No	1 (1.3)
No evidence	—	—	19 (23.8)

Clinical outcomes (versus surrogate) were used for 85% of The Dr Oz Show and 98% of The Doctors recommendations. At times these recommendations were justified via an intermediate or surrogate outcome that in turn would result in clinical outcome. For example, flossing every day decreases inflammation of the gums, and this inflammation causes aging.

Discussion

Principal findings

In our review of the two most popular medical talk shows, we found that general medical advice was the most common topic addressed on both shows. This broad category includes advice that could not otherwise be categorized into dietary, weight loss, exercise, cosmetic or counseling. The most common recommendations differed between the shows. On The Dr Oz Show the most common recommendations were dietary advice, while the most common recommendation on The Doctors was to consult a healthcare professional. Dietary recommendations were almost four times more common (39% v 10%) on The Dr Oz Show than on The Doctors. Dietary recommendations on The Dr Oz Show were close to eight times more common (39% v 5%) than exercise recommendations. Similarly, dietary recommendations were twice as common as exercise recommendations (10% v 5%) on The Doctors.

For both shows, a specific benefit was mentioned for only about 40% of the recommendations. The magnitude of benefit (<20%), potential harms (<10%), and costs (<15%) were less commonly mentioned. Thus, anyone who followed the advice provided would be doing so on the basis of a trust in the host or guest rather than through a balanced explanation of benefits, harms, and costs. The near absence of potential conflict of interest reporting (<1%) further challenges viewers’ ability to balance the information provided.

Roughly a third of the recommendations on The Dr Oz Show and half of the recommendations on The Doctors were based on believable or somewhat believable evidence. Evidence was believable or somewhat believable against a recommendation for 1 in 8-10 recommendations. For slightly over 1 in 3 and 1 in 4 of the recommendations for The Dr Oz Show and The Doctors respectively, no evidence could be found. This is despite us being quite liberal in the type and amount of evidence we required. The percentage of medical practice in the real world that is evidence based is difficult to ascertain, although one review reported an average of 78% of medical interventions were based on some form of “compelling” evidence.16 Comparisons are difficult, however, because the types of evidence that was required to determine this is considerably varied.

Strengths and limitations of study

Limitations of this study include the inherent complexity of the shows, including the subjective nature of the recommendations such as distinguishing between what was said and what was implied. To our knowledge, a detailed analysis like this has not been attempted before, and thus determining the best approach to gathering and quantifying the information was iterative. Although the protocol was planned ahead, it was adapted due to unanticipated challenges in data collection and interpretation. While this was a limitation, it also represents a lack of pre-disposition bias.

Developing searchable questions for each recommendation was difficult because less than 50% of recommendations were associated with a specific benefit. Again, we were liberal with our question development and search for evidence. Our original approach of having two independent experienced evidence reviewers (out of a pool of nine) review and categorize the evidence for each recommendation led to too much inconsistency due to the variability of the recommendations and the type of evidence required for each recommendation. Our adapted approach of four reviewers examining and discussing the evidence for recommendations was also imperfect, but we believe this method was the best way to provide a consistent review of the evidence for these highly subjective recommendations. During evidence review, some of the recommendations may have been standard or commonly recommended medical care or public health advice (such as “sneezing into your elbow prevents the spread of germs”), but we could not find evidence to support the recommendation. Although these were uncommon, it increased somewhat the proportion of recommendations with no evidence.

Finally, the time period selected for recording may not be reflective of the shows over multiple years, although it is unlikely that sampling episodes over a different time would change the results meaningfully. This is the first study to assess recommendations made on popular medical television talk shows from an evidence based perspective, and the strengths of the study include dual review at all levels, detailed data extraction, committee discussion for evidence assessment, and an iterative methodology.

Conclusions

Consumers should be skeptical about any recommendations provided on television medical talk shows, as details are limited and only a third to one half of recommendations are based on believable or somewhat believable evidence. An interesting question is whether we should expect medical talk shows to provide more than entertainment. Future studies may be directed at determining what viewers hope to obtain from watching these shows, and if the airing of these shows results in behavior changes related to specific recommendations. If the shows are perceived as providing medical information or advice, viewers need to realize that the recommendations may not be supported by higher evidence or presented with enough balanced information to adequately inform decision making. Decisions around healthcare issues are often challenging and require much more than non-specific recommendations based on little or no evidence from media health professionals. Patients would do well to ask healthcare providers specific questions about the benefits and harms, along with the magnitude of the effect (in absolute numbers), and the costs and inconveniences of any recommendation.

Television is an important source of health information for the general public

Popular medical television talk shows frequently engender skepticism and criticism from medical professionals

No research has systematically examined the content of the medical information provided on these talk shows

Recommendations made on popular medical television talk shows often lack adequate information on specific benefits, magnitude of benefits, and harms, and do not facilitate informed decision making

Approximately half of the recommendations have either no evidence or are contradicted by the best available evidence. Potential conflicts of interest are rarely addressed

The public should be skeptical about recommendations made on medical talk shows