Physicians' ability to predict the risk of coronary heart disease.

BACKGROUND: Coronary heart disease (CHD) is the leading cause of death in the United States. Previous research examining physicians ability to estimate cardiovascular risk has shown that physicians' generally overestimate the absolute risk of CHD events. This question has, however, only studied risk prediction for a limited number of patient care scenarios. The aim of this study is to measure the ability of physicians to estimate the risk of CHD events in patients with no previous history of coronary heart disease.
METHODS: Twelve primary prevention scenarios with a 5-year risk of CHD events were developed. This questionnaire was surveyed at 3 university teaching hospitals where the participants were a convenience sample of internal medicine residents and fellows or attending physicians in general internal medicine or cardiology. For each scenario, physicians were asked to estimate the baseline 5-year risk of a coronary heart disease event and the revised risk if the patient were to receive lipid-lowering drug therapy. Estimates of the baseline 5-year risk were compared with values calculated from Framingham risk equations. Inaccurate responses were defined as those with a ratio of estimated to actual risk of more than 1.5 or less than 0.67. Physicians' estimates of the relative risk reduction with therapy were considered to be accurate if they were between 25% and 40%.
RESULTS: 79 physicians (53 residents, 8 fellows, 18 attending physicians) completed the survey. Only 24% of physicians' risk estimates were accurate. In most cases, physicians overestimated the absolute risk of cardiovascular events without therapy (proportion overestimating ranged from 32-92% for the 12 individual scenarios). Physicians made larger errors in patient scenarios involving patients with high total or LDL cholesterol levels. Physicians' estimates of the relative risk reduction from treatment were more accurate: 43% of estimates were between 25 and 40%. Over 85% of physicians recommended treatment in 10 of 12 scenarios.
CONCLUSIONS: Physicians overestimate the absolute risk of CHD events and the potential absolute benefit of drug therapy.

Background

Coronary heart disease is the leading cause of death in the United States.[1] Each year, more than 1,000,000 Americans experience new or recurrent myocardial infarction or fatal coronary heart disease (CHD). About one-third of these events will be fatal, and approximately 650,000 of these events will be a first myocardial infarction. The direct and indirect costs of CHD and stroke are estimated to be $130 billion for 2003.[2] Fortunately, there are a number of interventions that can reduce the risk of CHD events in patients with no previous history of CHD, including smoking cessation counseling, hypertension treatment, aspirin prophylaxis, and treatment of lipid disorders.[3-7]

Systematic reviews and recent trials have found that lipid-lowering therapy with HMG co-A reductase inhibitors (statins) can reduce the relative risk of future CHD events by approximately 30% and that this relative risk reduction appears to be similar for patients with different levels of CHD risk. [7]. These findings suggest that the absolute benefit from statin therapy is proportional to the patient's baseline level of CHD risk. Patients with a previous history of cardiovascular disease (CVD) are at high risk and generally warrant aggressive treatment; patients with no previous CVD history have a wide range of underlying CHD risk and hence require individualized decisions for optimal care. In order to make effective and efficient decisions about whether a patient without prior CVD should receive lipid-lowering drugs, providers must implicitly or explicitly consider the patient's risk of future CHD events.

Previous research examining providers' ability to estimate cardiovascular risk has shown that providers generally overestimate the absolute risk of CHD events but are more accurate in their assessment of relative risk reductions.[8-12] If providers overestimate the absolute risk of CHD events prior to treatment, their estimates of the absolute benefit of therapy may also be inaccurate. Previous studies examining this question have only studied risk prediction for a limited number of patient care scenarios, making it difficult to examine how specific patient characteristics or the degree of baseline risk (low, medium, or high risk) affect risk prediction and the amount or degree of overestimation.

We sought to examine physicians' ability to predict the risk of CHD events and the effect of pharmacological lipid-lowering therapy for a wide range of primary prevention scenarios by asking physicians at three academic medical centers to complete a survey of 12 hypothetical scenarios concerning patients with different levels of CHD risk.

Methods

Questionnaire Development

We used the Framingham risk equations [13] to develop 12 primary prevention scenarios with 5-year risk of CHD events (defined as new onset angina, non-fatal or fatal MI, or sudden death) from 1 to 22 per cent. The scenarios were balanced with regard to gender (6 male patients, 6 female patients) and included 4 low-risk scenarios (less than 5% risk of CHD events over 5 years), 4 medium risk scenarios (5-year risk of 5–9%), and 4 high-risk scenarios (5-year risk 10–22%). Six of the scenarios had one or no additional risk factors (not including age, gender, or lipid levels), while the other 6 scenarios had 2 or more additional risk factors.

The hypothetical patients were explicitly assumed: 1) to not be taking aspirin or lipid-lowering agents at baseline; 2) to not have responded to a trial of a low-fat diet; 3), to not be overweight; and 4) to not have a family history of early myocardial infarction. We did not specify the patient's ethnicity, as we did not wish to examine its effect in this study.

The questionnaire was pilot tested in two different groups of primary care providers and preventive medicine residents at one of the study sites and was revised based on their feedback.

Study Subjects and Administration

In the fall of 1998 we administered the survey to residents, fellows, and attending physicians at three university sites. Potential respondents were invited to a lunch meeting in which they would complete a survey on preventive care. They provided informed consent and then completed the survey over the course of the next 20–30 minutes. Subjects were not allowed to use decision aids or other external information. The protocol was considered exempt by the UNC Institutional Review Board.

Outcome Measures

For each scenario, we first asked subjects to estimate the 5-year risk of CHD events without any therapy, then to estimate the 5-year risk if the patient were taking a lipid-lowering agent. We used a 5-year time range, rather than the 10-year range used by some risk estimation tools, because trials examining the benefits of lipid-lowering therapy were performed over a mean of 5 years, making estimates of treatment effectiveness over this time period most robust. Finally, providers were asked to indicate on a 4-point Likert scale if they would recommend that the patient take lipid-lowering therapy, with responses ranging from "strongly recommend therapy" to "strongly recommend against therapy". For these analyses, strong or moderate recommendations for therapy were considered as positive.

Analysis

Data from completed surveys were entered and analyzed using STATA 6.0. (Stata Corporation, College Station, Tx) and Microsoft Excel (Microsoft, Redmond, WA). Univariate frequencies are presented as proportions, means, and medians. We measured the accuracy of CHD risk prediction by dividing the subjects' estimate of risk by the risk obtained from the Framingham model and considering results from 0.67 to 1.5 to be accurate. The accuracy of physicians' estimates of CHD risk after therapy with lipid-lowering drugs was examined by calculating the relative risk reduction (RRR, defined as the baseline risk – risk after therapy, divided by the baseline risk) and comparing it against a value of 30% derived from a meta-analysis of primary prevention trials. [7] Relative risk reductions between 25 and 40% were considered to be accurate. To examine patient factors that may be associated with the accuracy of risk estimation, we compared the mean relative over-estimations for scenarios using patients with the factor compared with scenarios of patients without the factor, using Wilcoxon rank sum tests for univariate analyses and linear regression for multivariate analysis.

We did not attempt to compare the risk prediction ability of residents (compared with attendings) or cardiologists (compared with generalists) because of the small numbers of participants in each sub-group.

Results

Seventy-nine respondents completed the survey (53 residents, 8 fellows, and 18 attending physicians). Mean age was 31 years and 65% were men. Of the 8 fellows and 18 attending physicians, 77% were general internists and 23% were cardiologists.

Accuracy of risk estimates

Across all 12 scenarios, risk estimates were accurate (a response between 0.67 and 1.5 times the actual value) for only 24% of the responses. Most physicians overestimated the 5-year risk of CHD: for all scenarios combined, 66% of responses were overestimates, defined as 1.5 times greater than the actual value. For the 12 individual questions, the proportion of physicians overestimating risk ranged from 33 to 99 %. Fewer providers underestimated risk: 10% responses were underestimates (less than 0.67 times the actual value). For the 12 individual scenarios, the proportion of physicians underestimating risk ranged from 0 to 29%. When an alternative definition of accuracy (estimates within 10 percentage points of the true value), similar patterns were noted.

Estimation of relative risk reduction

Physicians were more accurate in their ability to estimate the relative risk reduction that would be associated with lipid-lowering therapy: 43% of estimates were accurate. By scenario, the range of accurate calculated RRR was 29 to 53%.

Treatment recommendations

Most physicians (over 85% in scenarios # 3–12; 32% in scenario #1 and 67% in scenario #2) recommended treatment with statin drugs. The two scenarios with lower treatment recommendation rates were the two of the four lowest risk scenarios.

Patient factors affecting degree of risk over-estimation

The mean degree of over-estimation, expressed in relative terms, was larger for low-risk scenarios (mean physician estimate 7.8 times Framingham estimate), intermediate for medium risk scenarios (2.8 times), and smaller (1.5 times) for high-risk scenarios. Table 2 shows univariate analyses of the median degree of overestimation for scenarios with and without specific patient characteristics. Physicians were more likely to overestimate risk for patients with very high cholesterol levels (total cholesterol > 260 mg/dl) than for patients with more modest elevations. Scenarios in which the patient had other CHD risk factors produced smaller over-estimations than when those factors were absent. Multivariate linear regression analysis revealed similar patterns of results. (data not shown)

Table 2

The effect of patient characteristics on the relative estimation medians for all responses.

	Median Relative Estimation		Ranksum Test
Factor	Present	Absent	p value
Diabetes	1.8	2.3	0.0001
Hypertension	1.6	2.7	0.0000
Smoking	1.7	2.9	0.0000
Male sex	1.7	3.3	0.0000
Age >= 65	1.4	3.0	0.0000
Total cholesterol > 260 mg/dl	3.1	1.7	0.0000
HDL cholesterol <= 40 mg/dl	1.7	3.3	0.0000

Discussion

In this scenario-based study, academic physicians overestimated the risk of CHD events across a range of patient risk profiles encountered in primary prevention. Scenarios with young patients, female patients, and very high lipid levels led to the highest degrees of overestimation. Their estimates of the relative risk reduction associated with therapy using lipid-lowering drugs were more accurate, but their estimates of the absolute risk reductions were also inaccurate because of their overestimation of baseline CHD risk. Perhaps because of the over-estimation of risk and the potential benefits of treatment, most physicians recommended treatment in most scenarios, even those with relatively low risk.

Our study provides additional insight into the ways in which providers misestimate risk. It appears that providers are more likely to overestimate risk in young adults and women, after adjusting for other risk factors, and are less likely to make such overestimations when other risk factors are present. These findings suggest that providers' heuristics may be systematically inaccurate. Whether feedback of these particular errors could improve risk estimation has not been examined.

Although we did not study actual provider behavior, our study suggests that efforts to improve the low utilization of statin drugs in high-risk patients cannot rely solely on educational efforts aimed at increasing the awareness of risk or the benefit of treatment. Knowledge of risk, or the advantages of treatment, do not appear to be important barriers in the decision to prescribe lipid lowering agents among physicians at academic medical centers.

Our findings are consistent with previous research that has examined physicians' ability to predict the risk of CHD events and found that physicians misestimate absolute risk. Friedmann et al used a mail survey to measure 599 physicians' estimates of risk for three CHD risk scenarios, one of which involved lipid-lowering therapy. [9] They found that cardiologists were more accurate in their perceptions of the underlying risk of CHD than general internists and family physicians, but that estimates of relative risk reduction were generally accurate for all specialties. Cardiologists recommended treatment more often than generalists, even though they perceived the benefit to be smaller.

Grover surveyed 253 Canadian physicians about their estimates of 8-year risk of CHD in two hypothetical scenarios, one involving a 40 year old male, and the second involving a 70 year-old female patient. [8] The mean estimates of risk were higher than actual risk in both scenarios, but the degree of overestimation was larger for the 40 year-old male patient. They also overestimated the potential years of life saved for a series of CHD interventions.

Chatellier et al provided clinical information about 66 patients to 6 physicians and asked the physicians to estimate the patients' CHD risk. Physicians had widely varying estimates of risk in comparison with Framingham-derived estimates. The agreement between the Framingham estimates and physicians' estimates were only moderately better than chance for 5 of the 6 physicians. [10]

McManus examined the ability of practice nurses and general practitioners in central England to estimate risk using actual patients identified from computerized records. Providers' subjective assessments and risk calculations from 4 risk assessment tools were compared against Framingham estimates. The subjective assessments were less accurate than those estimates generated from properly completed risk tool calculations, with relatively equal number of overestimation and underestimation.[11]

Montgomery and colleagues evaluated the cardiovascular risk prediction ability of 69 general practitioners and 11 practice nurses in the United Kingdom. Providers were asked to estimate the risk of cardiovascular events for a series of older patients (ages 60–79) with hypertension who were being seen in the practice for a blood pressure check. [12] Estimates were compared to the Framingham-calculated risk. Estimates were accurate in 21% of cases; 63% were underestimates and 16% overestimates. The large number of underestimates stands in contrast to our study, and others [8,9] that mainly identified overestimation errors. This difference may result from the generally high risk of the patients included in the study by Montgomery and colleagues (60% had a 5 year risk greater than 15%). It is less likely, although plausible, that the use of real patients may produce systematically different results than the use of hypothetical vignettes.

Our study, like several previous studies in this area, is limited by the use of hypothetical scenarios completed outside of usual practice. Actual patterns of risk estimation and treatment recommendation may differ from these examples. However, studies suggest that vignettes provide accurate information about actual physician practice.[14,15] Our use of a convenience sample of physicians limits the generalizability of our results. To limit the effect of our sampling strategy on the validity of our results, we did not inform potential subjects of the specific nature of the study prior to their decision to participate. Our choice of a relative measure for degree of inaccuracy may have increased the apparent degree of inaccuracy for low-risk scenarios. However, alternate analysis using absolute measures found similar patterns. Our decision to use 5 year risk, rather than the more commonly used 10 year risk, could have led to inaccurate risk estimation. We attempted to limit this effect by reinforcing verbally and in writing that we were asking for 5 year estimates. Finally, we did not vary the order of the scenarios. Subjects may have become fatigued or may have learned from answering the earlier scenarios, but we could detect no such pattern.

Conclusion

Despite these limitations, our findings suggest that physicians overestimate the risk of CHD events, particularly when the "patient" has very high total or LDL cholesterol levels and in low-risk scenarios, particularly those involving women, younger patients, and those without other CHD risk factors. Providers' difficulties in estimating CHD risk should not be surprising, since accurate estimation requires integration of several different pieces of clinical information, some of which are categorical (smoking versus no smoking) and others of which are continuous (age, blood pressure). Findings from the field of cognitive psychology suggest that when faced with complex questions, people, including health care providers, utilize simpler "rules-of-thumb" or heuristics to make decisions.[16] In some cases these heuristics serve quite well; in other cases, they may lead to common, reproducible errors in judgment. Inaccurate estimates of CHD risk could lead to prescription of lipid lowering drugs to patients in whom their net benefit has not been established. [17]

Inaccurate risk estimation and over-prescription of risk-reducing therapies can also have economic consequences. Cost-effectiveness analyses suggest that the cost per life-year saved and total costs rise dramatically as treatment is extended to lower-risk patients. [18] Accurate risk assessment is crucial for making rationale treatment decisions. Fortunately, several easy to use and accurate CHD risk prediction tools are now available. [19] They can improve the accuracy of risk predictions [11] are becoming more widely disseminated, and may improve the quality of treatment decisions, although further research on their effect is required.

Competing interests

After the completion of this research, Dr. Michael Pignone has received licensing and consulting fees from Bayer, Inc. for the development of a CHD risk estimation tool.

Authors' Contributions

Michael Pignone conceived of the idea for the study. He supervised study design, recruited co-investigators, oversaw pilot testing and data collection at one site, and participated in data analysis. He drafted and revised the manuscript. Chris Phillips participated in study design, conducted data analysis, and created tables. He also participated in manuscript editing. Tom Elasy and Alicia Fernandez acted as site supervisors for data collection and revised the manuscript.

Pre-publication history

The pre-publication history for this paper can be accessed here:

Table 1

Provider estimates of risk and proportion of inaccurate estimates

#	Clinical history	CHD 5-year risk*	Mean/Median provider estimates of 5-year risk	% of providers overestimating risk by more than 1.5x	% of providers underestimating risk (ratio less than 0.67)	% of providers who overestimate risk by >= 10 percentage points	% of providers who underestimate risk by < 10 percentage points	% of providers recommending treatment with statin drug
1	49 yo man with TC = 240, LDL = 140, HDL = 50. Non-smoker, no history of diabetes; BP = 114/74.	3%	6.8% / 5%	67.1%	11.4%	34.2%	NA	31.6%
2	43 yo woman with TC = 278, LDL = 192, HDL = 48. Non-smoker, no history of diabetes; BP = 136/86.	1%	7.2% / 5%	89.9%	0.0%	34.2%	NA	67.1%
3	68 yo man with TC = 240, LDL = 150, HDL = 40. (+) smoker, (+) diabetes, BP = 110 / 70	22%	26.0% / 25%	32.9%	25.3%	84.8%	15.2%	97.5%
4	42 yo man with total cholesterol = 260, LDL = 192, HDL = 30. (+) smoker (+)diabetes, BP = 110 / 70	10%	18.2% / 15%	46.8%	12.7%	81.0%	NA	97.5%
5	42 yo woman with TC = 290, LDL = 200, HDL = 45. (+) smoker; (+) diabetes, BP = 114/ 74	3%	15.1% / 15%	92.4%	0.0%	73.4%	NA	96.2%
6	73 yo man with TC = 255, LDL = 165, HDL = 42. Non-smoker; no diabetes; BP = 150/88	18%	23.3% / 20%	27.8%	29.1%	83.5%	16.5%	89.9%
7	74 yo woman with TC = 270, LDL = 170, HDL = 40. Non-smoker; no diabetes; BP = 148/92	7%	19.6% / 15%	65.8%	5.1%	77.2%	NA	84.8%
8	40 yo man with TC = 235, LDL = 165, HDL = 40. (+) smoker; no diabetes; BP = 145/91	6%	14.6% / 10.5%	65.4%	7.7%	64.6%	NA	92.4%
9	68 yo woman with TC = 244, LDL = 155, HDL = 37. (+) smoker (+)diabetes, BP = 114 / 74	9%	22.6% / 20%	68.4%	8.9%	81.0%	19.0%	97.5%
10	52 yo man with TC = 320, LDL = 210, HDL = 45. Non-smoker, no history of diabetes; BP = 122/74	5%	17.9% / 15%	79.7%	0.0%	74.7%	NA	97.5%
11	54 yo woman with TC = 330, LDL = 235, HDL = 60. Non-smoker, no history of diabetes; BP = 114/74	1%	17.0% / 15%	98.7%	0.0%	68.4%	NA	93.7%
12	56 yo woman with TC = 264, LDL = 200, HDL = 34. (+) smoker; no diabetes; BP = 155/90	12%	23.5% / 20%	53.2%	3.8%	91.1%	8.9%	98.7%

Abbreviations: NA = not applicable, TC = serum total cholesterol, LDL = low density lipoprotein cholesterol, HDL = high density lipoprotein cholesterol, BP = blood pressure. Risk of myocardial infarction, sudden death, or new-onset angina as calculated from the Framingham risk equations