A tool for evaluating the potential for cost-effective outcomes measurement.

Cost related to higher-level outcomes measurement is often very high. However, the cost burden is felt even more by smaller, less well-funded continuing medical education (CME) programs. It is possible to overcome financial and participant-related barriers to measuring Level 6 outcomes, which are patient health outcomes. The Temple University School of Medicine's Office for Continuing Medical Education developed a sequential tool for attaining cost-effective outcomes measurement for determining the likelihood of a CME intervention to produce significant changes in physician performance. The appropriate selection of the CME topic and specific practice change indictors drive this tool. This tool walks providers through a simple YES or NO decision-making list that guides them toward an accurate prediction of potential programmatic outcomes. Factors considered during the decision-making process include whether: (a) the intended change(s) will have a substantial impact on current practice; (b) the intended practice change(s) are well supported by clinical data, specialty organization/government recommendations, expert opinion, etc; (c) the potential change(s) affects a large population; (d) external factors, such as system pressures, media pressures, financial pressures, patient pressures, safety pressures, etc, are driving this intended change in performance; (e) there is a strong motivation on the part of physicians to implement the intended change(s); and (f) the intended change(s) is relatively easy to implement within any system of practice. If each of these questions can be responded to positively, there is a higher likelihood that the intended practice-related change(s) will occur. Such change can be measured using a simpler and less costly methodology.

Introduction

Outcomes measurement is a required part of continuing medical education (CME). The Accreditation Council for Continuing Medical Education (ACCME) requires that all providers analyze and evaluate the effectiveness of their CME activities. Providers must evaluate the effectiveness of their CME activities in meeting identified educational needs. Educational effectiveness can be evaluated at multiple levels. Moore’s sevenlevel pyramid of outcomes measurement includes: participation, satisfaction, learning (declarative knowledge and procedural knowledge), competence, performance, patient health, and community health.1 CME providers are being called upon to demonstrate that their educational activities/programs result in better, more meaningful practice-based outcomes. Previously, measuring lower levels of outcomes – participation, satisfaction, or learning – was sufficient to meet ACCME expectations (Levels 1–3). ACCME now mandates that CME providers assess higher levels of outcomes; the impact of CME on physician competence (Level 4), performance (Level 5), and patient health (Level 6) must be measured. Providers must analyze changes in learners’ competence, performance, and patient outcomes achieved as a result of the overall program’s activities/educational interventions.

Achieving higher levels of outcomes is not only expensive, but it is also time consuming. The most widely used and accepted methodologies to measure the outcomes of CME programs is audits and/or chart reviews of patients and completing patient report forms. These two protocols are time consuming, labor intensive, and expensive for both CME providers and participating physicians. Physicians have to spend 8–10 hours completing patient report forms, and CME providers have to provide some kind of monetary incentive for physicians. Most providers give financial incentive to participating physicians to complete one patient report form. For 25 patient report forms, the cost ranges from $1250–$2475 per participating physician. This becomes financially burdensome for CME providers.

In mid-2007, the Temple University School of Medicine CME Program was researching potential topics for a “performance improvement” educational activity in the area of cardiology. A scan of various cardiology news sources and discussions with Temple-affiliated cardiologists established that the results of a major lipid trial (the Ezetimibe and Simvastatin in Hypercholes-terolemia Enhances Atherosclerosis Regression [ENHANCE] trial) were to be released during the upcoming American College of Cardiology (ACC) annual conference and these results were going to change the way physicians treated hyperlipidemia in patients using combination therapy. This seemed like a great opportunity for a performance improvement CME activity. A strategic plan was developed to design an Internet-based CME activity to be released soon after the ACC annual conference. A simple post survey methodology followed by telephone confirmation was selected to measure the change in the performance of physicians.

The CME program was designed to provide recommendations to primary care physicians for taking care of their hyperlipidemic patients. The results from the ENHANCE study indicated that in patients with familial hypercholesterolemia, combination therapy with a statin (simvastatin) and ezetimibe did not result in a significant difference in changes in intima-media thickness, as compared with simvastatin alone, despite a decrease in the levels of low-density lipoprotein cholesterol (LDL-C) cholesterol and C-reactive protein (CRP). Therefore, treatment plans for all patients on combination therapy need to be reassessed, with the emphasis on maximizing statin therapy and likely discontinuation of ezetimibe.2 An Internet-based CME program was designed, developed, and posted on Medscape 2 weeks after the ACC meeting to help practicing physicians manage their patients appropriately. The recommendations made by the faculty are:

LDL-C remains the primary treatment target

Statin therapy remains the first-line treatment for hyperlipidemia

Intensify statin therapy to achieve an LDL-C goal of <70 mg/mL or LDL-C reduction of >50% in high-risk patients

Titrate to maximize the tolerated dose of the statin

Switch to a more potent statin

If treatment with a statin alone is not sufficient to achieve the treatment goal or the use of high-dose statin therapy is not appropriate, initiate combination therapy; begin by adding a fibrate or niacin before the combination of a statin and ezetimibe

Use CRP as a surrogate marker to refine risk assessment of asymptomatic patients at low-to-intermediate cardiovascular disease risk who would benefit from a more aggressive risk-reduction strategy.

Material and methods

Study design

Over 8000 healthcare professionals participated in the activity and 1047 CME certificates were issued. Certain physician information, such as ethnicity, years in practice, and number of patients seen in a week, was not known. Of the 1047 physicians who received CME certificates, 200 primary care physicians were randomly selected for a post survey consisting of six questions. The survey was mailed to participants 120 days post activity. The participants who responded to the survey were called 60 days after receiving the survey results to reconfirm the responses.

The survey questions

Physicians were asked if they changed their behavior for each of six behaviors related to the CME initiative (Appendix 1).

Statistical methods

The responses were on a five-point scale where five equals “strongly agree” and one equals “strongly disagree”. The intended outcome was for the respondents to change their behavior; a “strongly agree” (coded as a 5) response indicates success for the program. Descriptive statistics (means and frequency summaries) were used to analyze the data. With reference to the means and item frequencies, a mean of ~4 (for item 1) on a five-point scale indicates success.

Pearson product moment correlations were calculated to measure the degree of relationship among the six behavioral outcomes represented by the item scores (see Table 2). For example, there is a relatively strong relationship between how the respondents answered item 1 and how they answered item 3.

Table 2

Correlations among the six items on the post survey

	Item 1	Item 2	Item 3	Item 4	Item 5	Item 6
Item 1	1.00	0.27	0.52**	0.06	−0.03	0.22
Item 2	0.27	1.00	0.27	0.15	0.29	0.18
Item 3	0.52**	0.27	1.00	0.17	0.20	0.34*
Item 4	0.06	0.15	0.17	1.00	0.28	0.41**
Item 5	−0.03	0.29	0.20	0.28	1.00	−0.09
Item 6	0.22	0.18	0.34*	0.41**	−0.09	1.00

Notes:

P < 0.05;

P < 0.01.

Results

The results of these studies are shown in Table 1. Forty-eight participants responded to the questions. For questions 1–4, the mean and item frequency ranged from 3.98–4.30. Forty-four participants out of 48 were using surrogate markers to identify patients at high risk for cardiovascular events who may benefit from intensive therapy. Question 2 was added to the analysis because, according to the Framingham Heart Study, cigarette smoking increases the risk of heart disease, an issue that was discussed by faculty.3 Thirty-nine out of 48 were using some kind of intervention for smoking cessation. It was interesting to note that 44 physicians out of 48 think it is essential to achieve a reasonable goal of <70 mg/dL LDL for patients with coronary heart disease (CHD) and diabetes. On the other hand, only 40 physicians out of 48 reported they would prescribe combination therapies to achieve LDL-C reductions of >50%. Thirty-eight physicians out of 48 had changed their performance as far as the use of a statin with ezetimibe as a preferred lipid-lowering combination therapy.

Table 1

Average agreement ratingwith concept and percentage of respondents who indicate “I have changed” or “I have not changed” my practice behavior in this area

Questions	Mean	Changed %	Not changed %
1. Use surrogate markers to identify patients at high risk of cardiovascular events who may benefit from intensive therapy	3.98	91.7	8.3
2. Insist that a patient with hypercholesterolemia who smokes enroll in a smoking cessation program	4.18	81.3	18.7
3. Help achieve a reasonable goal of <70 mg/dL LDL for patients with CHD and diabetes	4.30	91.7	8.3
4. Prescribe combination therapies to achieve LDL-C reductions of >50%	4.00	83.3	16.7
5. Use a statin with ezetimibe as a preferred lipid-lowering combination therapy	3.21	79.2	20.8
6. Use a statin with niacin as a preferred lipid-lowering combination therapy	3.75	81.3	18.7

Note:

The Likert scale values ranged from 1–5 where 1 indicates low agreement and 5 indicates high agreement.

Abbreviations: CHD, coronary heart disease; LDL-C, low-density lipoprotein cholesterol.

A significant correlation between questions on the post survey was observed using Pearson product moment correlations to measure the degree of relationship among the six behavioral outcomes represented by the item scores. There were three significant positive correlations between Questions 3 and 1, Questions 6 and 3, and Questions 6 and 4. As physicians reported that they used more “surrogate markers to identify patients at high risk of cardiovascular events who may benefit from intensive therapy,” they also reported that they “more frequently helped achieve a reasonable goal of <70 mg/dL LDL for patients with CHD and diabetes” (P < 0.01). The physicians who reported they “prescribed combination therapies to achieve LDL-C reductions of >50%,” also reported that they “more frequently prescribed use of a statin with niacin as a preferred lipid-lowering combination therapy” (P < 0.01). Similarly, physicians who reported they “more frequently helped achieve a reasonable goal of <70 mg/dL LDL for patients with CHD and diabetes,” also reported that they “more frequently prescribed use of a statin with niacin as a preferred lipid-lowering combination therapy” (P < 0.01) (see Table 2).

A paired t-test was conducted to determine whether there were significant differences in mean responses between questions on the post survey.

Table 3 below shows that on average, respondents indicated achieving a reasonable goal of <70 mg/dL LDL for patients with CHD and diabetes was more important than using surrogate markers to identify patients at high risk of cardiovascular events who may benefit from intensive therapy. On the other hand, it was more important to use surrogate markers to identify patients at high risk of cardiovascular events who may benefit from intensive therapy than to use a statin with ezetimibe as a preferred lipid-lowering combination therapy or use a statin with niacin as a preferred lipid-lowering combination therapy. Similarly, it was more important to insist that a patient with hypercholesterolemia who smokes enroll in a smoking cessation program than to use a statin with ezetimibe as a preferred lipid-lowering combination therapy or use a statin with Niacin as a preferred lipid-lowering combination therapy. Most interestingly, the use of a statin with ezetimibe as a preferred lipid-lowering combination therapy was lowest on the responders’ list.

Table 3

Preferences between six items on the post survey

Question pairs	Respondents	Mean difference	Std deviation difference	95% confidence interval of the difference		P values
				Lower	Upper
Q1, Q3	43	−0.33	0.61	−0.51	−0.14	P < 0.001
Q1, Q5	39	0.87	1.22	0.48	1.27	P < 0.001
Q1, Q6	39	0.28	0.79	0.02	0.54	P < 0.05
Q2, Q5	37	1.14	1.13	0.76	1.51	P < 0.001
Q2, Q6	37	0.51	0.93	0.20	0.82	P < 0.01
Q3, Q4	40	0.35	0.89	0.06	0.64	P < 0.05
Q3, Q5	38	1.13	1.12	0.76	1.50	P < 0.001
Q3, Q6	40	0.60	0.74	0.36	0.84	P < 0.001
Q4, Q5	38	0.79	1.12	0.42	1.16	P < 0.001
Q4, Q6	40	0.25	0.78	0.00	0.50	P < 0.05
Q5, Q6	37	−0.57	1.32	−1.01	−0.13	P < 0.05

Discussion

Achieving higher levels of outcomes measurement is not an easy task

Measuring higher levels of outcomes is complicated and usually requires a larger study population. In addition, it requires extended follow-up and more elaborate outcome methodologies, such as audits, chart reviews, electronic health records, health plan data, registries, and patient surveys. Patient level outcomes (changes in patient health status) require access to physicians’ actual patient data and may be limited by Health Insurance Portability and Accountability Act regulations. There are a number of studies showing that all formats of CME are effective in changing physicians’ behavior to some extent.4–20 However, in our experience, these changes are small and sometimes difficult to measure even using chart reviews.

In this study, we have demonstrated that even a single intervention can have a drastic effect on the behavior of physicians. As the face of CME is changing, there is tremendous pressure on educators to design CME activities around the problems in the health system and measure their effectiveness at the population or system level. The important thing to remember is that large numbers of physicians practice in communities; these physicians need good evidenced-based education, and as educators, we have to create education in all different formats to fulfill the needs of our community physicians. Based on our experience, we developed a sequential tool to determine the likelihood of a CME activity resulting in significant changes that can be measured using a simple and cost-effective methodology.

Description of a sequential tool for driving cost-effective outcomes measurement

It is possible to overcome financial and participant-related barriers to measuring Level 6 outcomes. Based on the results of this study, we developed a tool for projecting and evaluating variables related to cost-effective outcomes measurement. This tool describes the likelihood of a CME intervention to produce significant changes in physician performance using simple and cost-effective measurement tools. Appropriate selection of the CME topic and specific practice change indictors drive this tool (see Table 4). This evaluative tool starts with careful consideration of the topic and intended/needed/recommended changes in practice. If the answer to each of these questions is “yes”, then there is a higher likelihood that the intended practice-related change(s) will occur and such change can be measured using simpler and less costly methodology.

Table 4

Tool for evaluating the potential for cost-effective outcomes measurement

1. Will the intended change(s) have a substantial impact on current practice?	Yes	No
2. Is the intended practice change(s) well supported by clinical data, specialty organization/government recommendations, expert opinion, etc	Yes	No
3. Does this potential change affect a large population?	Yes	No
4. Are there external factors driving this intended change in performance?	Yes	No
5. Is there a strong motivation on the part of physicians to implement the intended change	Yes	No
6. Is the intended change(s) relatively easy to implement within any type of system of practice?	Yes	No

Will the intended change(s) have a substantial impact on current practice?

The selection of a CME topic is the most crucial step to study the changes in physician performance. When the change is small, one has to use both qualitative and quantitative methodologies to measure a clinically significant change. On the other hand, when there is a major shift in the paradigm due to negative clinical results, the change is significant and can be easily measured using low-cost measurement tools.

Is the intended practice change(s) well supported by clinical data, specialty organization/government recommendations, expert opinion, etc?

There should be a high level of consensus/agreement among health care providers/experts/regulatory agencies regarding the effectiveness and safety of therapeutic agents. In this particular case, it was crucial to use lipid-lowering strategies to reduce the risk of cardiovascular events. The use of lipid-lowering agents is considered the standard of care for a large segment of the US population to reduce both the risk of developing cardiovascular disease and the risk of a future event (primary and secondary prevention). These agents have well-understood mechanisms of action, well-documented efficacy, and minimal side effects. The use of a statin is supported by large clinical trials, and practice guidelines have been developed by associations and opinion leaders (ie, the American Heart Association, American College of Cardiology, National Cholesterol Education Program Adult Treatment Panel III). There is compelling data from large, well-designed, randomized, double-blinded, multicenter clinical trials reinforcing current treatment recommendations and best practices.

Does this potential change affect a large population?

The patient population potentially affected by the intended performance change should be large. Over 102 million American adults suffer from dyslipidemia (total cholesterol > 200 mg/dL).21 Over a third of US adults have LDL-C > 130 mg/dL, which is associated with a higher risk of CHD.21 According to the Centers for Disease Control and Prevention, only half of Americans with high LDL-C receive treatment.22 Antihyperlipidemic agents are the second most frequently prescribed class of medication.23 National Health Statistics Reports indicate that 200 million prescriptions for cholesterol drugs were written in 2007 and 2008.23 As many as 30 million Americans (10% of the population) are thought to be taking a statin to lower cholesterol. Statins are the most commonly used class of prescription drugs for US adults aged 60 years and over (nearly 45%).24 ENHANCE results directly affected ~20 million patients who were taking ezetimibe in combination with a statin.25

Are there external outside factors driving this intended change(s) in performance?

Negative clinical study results are a primary motivating factor to change practice behavior due to press reports, media attention, fear of being sued, and new practice guidelines released by experts, national associations, and societies. The format and timelines of the educational intervention are very important. In this case, the online format of education was selected and the expert treatment guidelines, along with performance recommendations, were posted on the Internet 2 weeks after the ACC meeting. The data that was shared was fresh and cutting edge.

Is there a strong motivation on the part of physicians to implement the intended change(s)?

Negative clinical outcomes studies frequently stimulate an urgency to implement practice changes. A large population was affected by the ENHANCE clinical data. Health care providers were eagerly waiting for clinical recommendations about how to manage these patient populations. In this study, a number of performance improvement indicators were defined, as shown in Table 1. These performance improvement indicators were also part of the recommendations made by the experts, as listed on the post activity survey. The experts and national associations recommended that lowering LDL-C is still the most important goal to reduce the risk of cardiovascular disease.

The behavior to encourage was that high-risk patients should be treated aggressively regardless of surrogate markers such as CRP. On the other hand, a CRP marker may be helpful to identify asymptomatic, low-to-intermediate risk patients who may benefit from more intensive therapy. Practicing physicians are looking for treatment guidelines to follow. Matching treatment programs to defined patient populations assists physicians in making the correct choices.

Is the intended change(s) relatively easy to implement within any type of system of practice?

This tool is based on criteria that physicians have no or limited perceived barriers. In this particular study, the physicians had a number of other options, such as (1) maximizing the dose of statins; (2) using other combination therapies; and (3) sending patients to therapists for smoking cessation or life style changes. Minimizing barriers has a direct relationship with physicians’ abilities to implement changes in practice.

Conclusion

The tool developed and presented here sets constraints that allow providers to systematically evaluate the potential results gleaned from the outcomes measurement component of their initiative. They can thus minimize potential financial and human resource expenditures before dealing with the outcomes of a CME activity. This is particularly pivotal for smaller academic CME providers with severely limited financial resources that are eager to comply with outcomes measurement requirements that are critical to the ACCME accreditation process. Selecting topics and initiatives that provide the most opportunity for rich outcome data helps conserve these scarce resources.

It is our hope that adoption of this evaluative tool will assist CME providers in meeting the challenges of fiscal responsibility and accreditation regulation in this competitive era within the CME industry. The authors of this research encourage other CME providers to self-reflect on their planning process to improve this tool or to facilitate the development of other tools to meet the growing needs of outcomes analysis of CME programs.

Annotation

There have been subsequent studies that have challenged the results of the ENHANCE clinical trial, Niacin, and CRP data. However, those studies have no impact on the outcomes of CME initiative.