| Literature DB >> 18341795 |
Marilyn Sitaker1, Jan Jernigan, Susan Ladd, Miriam Patanian.
Abstract
Logic models are graphic representations of the relationship between program activities and their intended effects and are used for both program planning and evaluation. Logic models can provide an important foundation for program evaluation by identifying evaluation questions that most appropriately assess program processes and outcomes and by guiding measurement decisions. We demonstrate how logic models can be used to plan program evaluation by describing the adoption of logic modeling by the Washington State Heart Disease and Stroke Prevention Program (WaHDSPP) and by specifying the changes in process and use of logic models since the program's initial funding. Our paper describes how a logic model was used in generating the program evaluation plan for the WaHDSPP, including the identification of evaluation questions and development of indicators to track progress effectively. We describe the use of evaluation results, as well as steps state programs can take to use logic models in program evaluation.Entities:
Mesh:
Year: 2008 PMID: 18341795 PMCID: PMC2396971
Source DB: PubMed Journal: Prev Chronic Dis ISSN: 1545-1151 Impact factor: 2.830
Figure 1.Washington State Specific Logic Model Framework for Heart Disease and Stroke Prevention Program. (HDSPP indicates Heart Disease and Stroke Prevention Program; CDC, Centers for Disease Control and Prevention; and HD&S, heart disease and stroke.)
Figure 2.Washington State Heart Disease and Stroke Prevention Program (WaHDSPP) Logic Evaluation Plan, 2005–2006. (WSC indicates Washington State Collaborative; CVD, cardiovascular disease; FHCQ, Foundation for Healthcare Quality; HBP, high blood pressure; BRFSS, Behavioral Risk Factor Surveillance System; HD, heart disease; PA, physical activity; and CHARS, Comprehensive Hospital Abstract Reporting System.)
Figure 3.Washington State Heart Disease and Stroke Prevention Program (WaHDSPP) Logic Evaluation Plan, 2007–2008. (HDSPP indicates Heart Disease and Stroke Prevention Program; CDC, Centers for Disease Control and Prevention; EMS TAC, Emergency Medical Service Technical Advisory Committee; NCQA, National Committee for Quality Assurance; EPHS, essential public health services; MIS, management information system; HD&S, Heart Disease and Stroke; MI, myocardial infarction.)
Evaluation Plana, Washington Heart Disease and Stroke Prevention Program (WaHDSPP)b, 2007–2008
| Questions | Indicators | Data Sources | Data Collected by | Data Analysis | Communicating Results | |
|---|---|---|---|---|---|---|
|
| ||||||
| To Whom? | How? | |||||
| What key lessons were learned from evaluation of the previous barber/hair stylist outreach program conducted by WaHDSPP? | Product = List of key lessons learned | Written summary | July 2007–August 2007 | Document review | CDC Steering Council | Progress reports to funding agency (CDC) |
| Was the hair stylist outreach program for community centers and faith-based organizations conducted as planned? | Process measures; feedback from stylists | Quarterly progress reports and stylist focus group | September 2007–February 2008 | Analysis of focus group data | ||
| What was the impact of the barber/hair stylist outreach program? | Of those whose screening results indicate high BP, the number who were treated by a health care provider | Interviews with sample of clients with high BP results | September 2007–February 2008 | Analysis of survey responses | ||
CDC indicates Centers for Disease Control and Prevention; BP, blood pressure.
Based on the CDC Division for Heart Disease and Stroke Prevention Evaluation Guide.
Program promotes awareness of signs and symptoms of heart disease and stroke and targets African Americans (supporting outcomes: increased public awareness of risk factors, signs and symptoms of heart disease and stroke, and calling 911; intermediate outcomes: controlled risk factors, increased recognition of symptoms, and decreased time between recognizing symptoms and calling 911).
| The figure begins with a series of boxes organized into three columns. The three boxes arranged on top of each of the columns read 1) Inputs, 2) Activities, and 3) Goals From State Plan (5 year MIS objectives). In the first column, entitled “Inputs,” are three boxes arranged from top to bottom. Inside the first box, entitled “HDSPP,” are the following bullet points: state program funding; expertise of professional staff and infrastructure; time commitment; internal partners and stakeholders; communication; and internal epidemiology support for surveillance and evaluation. Inside the second box, entitled “CDC,” are the following bullet points: funding; programmatic guidance; training; CDC project officer; conference calls, listserv; and technical assistance. Inside the third box, entitled, “Advisory Council Members and Organizations,” is the following information: investment of staff time for strategic planning and implementation of defined strategies. All three boxes in the first column are connected with a right-pointing bracket to the boxes in the second column. |
| The second column, entitled “Activities,” contains one large box beneath the title box. The large box contains four major sections, each with information listed beneath. The four major sections are: 1) Communication, 2) Systems Change, 3) Implement State Plan, and 4) Assessment and Evaluation. Underneath the “Communication” section is the following information: Go Red for Women campaign; and Signs of Stroke campaign. Underneath the “Systems Change” section are the following bullet points: employer-based health and productivity; Washington State Collaborative; Get with the Guidelines (AHA); NCQA recognition program (Puget Sound health alliance); and HD&S Systems of Care Improvement Project. Underneath the “Implement State Plan” section is the following information: Heart Disease and Stroke Prevention steering council meetings. Underneath the “Assessment and Evaluation” section is the following information: produce a) expanded county profiles, b) the surveillance notebook, c) other analyses; and inventory CVD data sources. |
| The third column, entitled “Goals From State Plan (5 year MIS objectives),” contains one large box beneath the title box. The large box contains a listing of eight goals, which are organized in the following order: 1) Goal 3: Washington residents know risk factors and warning signs of HD&S; 2) Goal 5: action plan and policies empower consumers, result in improved health status and quality; 3) Goal 2: surveillance used to transform operations (adopt evidence–based practices); 4) Goal 7: Washington maintains and coordinates HD&S systems of prevention, screening, diagnosis, treatment and rehabilitation; 5) Goal 8: education for HD&S prevention and management for licensed, registered, or certified health care providers; 6) Goal 4: steering council is convened; 7) Goal 9: evaluation of effectiveness of HD&S prevention system; and 8) Goal 1: comprehensive HD&S data. |
| All of the information in the large box in the second column is connected to the goals listed in the large box in the third column, using right-pointing arrows. “Go Red for Women Campaign” and “Signs of Stroke Campaign” under the “Communication” section of column two is connected to “Goal 3: Washington residents know risk factors and warning signs of HD&S” in column three. “Employer-based health and productivity” under the “Systems Change” section of column two is connected to “Goal 5: action plan and policies empower consumers, result in improved health status and quality” in column three. “Washington State Collaborative,” “Get with the Guidelines (AHA),” and “NCQA recognition program (Puget Sound health alliance)” under the “Systems Change” section of column two are connected to “Goal 2: Surveillance used to transform operations (adopt evidence–based practices)” in column three. “HD&S Systems of Care Improvement Project” under the “Systems Change” section of column two is connected to “Goal 7: Washington maintains and coordinates HD&S systems of prevention, screening, diagnosis, treatment and rehabilitation” and to “Goal 8: education for HD&S prevention and management for licensed, registered, or certified health care providers” in column three. “Heart Disease and Stroke Prevention steering council meetings” under the “Implement State Plan” section of column two is connected to “Goal 4: steering council is convened” in column three. “Produce a) expanded county profiles, b) the surveillance notebook, c) other analyses” under the “Assessment and Evaluation” section of column two is connected to “Goal 9: evaluation of effectiveness of HD&S prevention system” in column three. “Inventory CVD data sources” under the “Assessment and Evaluation” section of column two is connected to “Goal 1: comprehensive HD&S data” in column three. |
| To the right of column three are another series of three columns, each with a heading box above them. The first three columns already described are connected to the second series of three columns with a right-pointing bracket. The heading boxes above the second set of three columns read, “Short-term Outcomes,” “Medium-term Outcomes,” and “Long-term Outcomes.” Inside the first box of the second set of columns is the following information: partnerships and collaborations are established and Working; increased public awareness of the signs and symptoms of heart disease and stroke, knowledge to call 9-1-1 [noted as a CDC priority area]. Box one is connected with a right-pointing arrow to box two of the second set of columns. Inside the second box there are three sections. The first section reads, “Enhanced public health system infrastructure.” The second section, entitled “Improved quality of care,” contains the following two bullet points: a) improved emergency response to acute events across the state, and b) better management of chronic conditions. (The entire section of “Improved quality of care” is noted as a CDC priority area.) The third and last section of the second box, entitled “Individual Behavior Change,” contains the following three bullet points: a) better self-management of chronic disease; b) control of risk factors, especially blood pressure and cholesterol [noted as a CDC priority area]; and c) improved lifestyle. Box two is connected with a right-pointing arrow to two separate boxes in column three. These two smaller boxes are arranged vertically to form the final column. Inside the top box of column three is the following information: Reduce health disparities between general and priority populations. (The words “reduce health disparities” are noted as a CDC priority area.) Inside the bottom box of column three is the following information: Reduce deaths and disability due to heart disease and stroke. |
| Underneath the six columns is the following text: Washington State’s Contextual Influences: unique patterns of ethnic diversity and income inequities, high unemployment. This text has a series of seven arrows that point upwards toward a large bolded line that lies beneath the six columns. |
| This figure contains a series of boxes arranged into eight columns, each of which contains a title box: 1) Activities, 2) Process Measures, 3) Short-Term Outcome, 4) Short-Term Outcome Measure, 5) Medium-Term Outcome, 6) Medium-Term Outcome Measure, 7) Long-Term Outcome, and 8) Long-Term Outcome Measure. Underneath these eight heading boxes is one long, rectangular box that stretches the length of the figure. The following text is contained in the long box: Washington State Collaborative: By June 29, 2006, enroll 20 clinical practice teams in the WSC 5, focused on secondary prevention of heart disease and stroke, and hold at least two Learning Sessions (supports short-term outcome, “Widespread adoption of planned care model”). Underneath the heading box of the first column, entitled “Activities,” are the following six bullet points: 1) focus group with current CVD clinic teams to learn how to better market WSC 5; 2) develop marketing plan to enroll clinical practice teams; 3) develop marketing plan to enroll clinical practice teams; 4) site visits to at least 10 potential CVD clinical teams to join WSC 5; 5) work with clinical practice teams to identify baseline performance on established measures; and 6) conduct two learning sessions. |
| Underneath the heading box of the second column, entitled “Process Measures,” are six bullet points. The first bullet point says, “Target: Focus groups held. Measure: Written summary of focus group results.” The second bullet point says, “Target: Marketing plan. Measure: copy of marketing plan that is developed.” The third bullet point says, “Targets: Contract in place with FHCQ; scoring criteria are established, and grants are awarded and monitored by FHCQ. Measures: copy of contract; copy of scoring criteria, copy of grants awarded.” The fourth bullet point says, “Target: Completion of at least 10 site visits.” The fifth bullet point says, “Target: Clinics have baseline data on process/outcome measures. Measure: copy of baseline measures for each team.” The sixth bullet point says, “Target: at least 2 Learning Sessions held. Measure: copy of agenda for learning session.” The first two columns are connected to the third and fourth columns with a right-pointing bracket. |
| Underneath the heading box of the third column, entitled “Short-Term Outcome,” is the following information: Widespread adoption of the Planned Care Model. Underneath the heading box of the third column, entitled “Short-Term Outcome Measure,” is the following information: Number of clinical team using the Planned Care Model with CVD patients. The third and fourth columns are connected to the fifth and sixth columns with a right-pointing bracket. |
| Underneath the heading box of the fifth column, entitled “Medium-Term Outcome,” are five bullet points. The first bullet point says, “Improved quality of care for people with CVD.” The second bullet point says, “Better management of chronic conditions.” The third bullet point says, “Better self-management of chronic disease.” The fourth bullet point says, “Control CVD risk factors, HBP and high cholesterol.” The fifth bullet point says, “Improved lifestyle behaviors.” |
| Underneath the heading box of the sixth column, entitled “Medium-Term Outcome Measure,” are five bullet points. The first bullet point says, “Health care provider survey: Prevalence of implementation of Planned Care Model.” The second bullet point says, “Health care provider survey: Implementation of guidelines.” The third bullet point says, “BRFSS: Medications and knowledge of HD and stroke symptoms.” The fourth bullet point says, “BRFSS: High blood cholesterol, High blood pressure questions.” The fifth bullet point says, “BRFSS: Nutrition and PA questions.” The fifth and sixth columns are connected to the seventh and eighth columns with a right-pointing bracket. |
| Underneath the heading box of the seventh column, entitled “Long-Term Outcome,” are the following two bullet points: 1) Reduce health disparities between general and priority populations; and 2) Reduce deaths and disability due to heart disease and stroke. Underneath the heading box of the eighth column, entitled “Long-Term Outcome Measure,” are the following two bullet points: BRFSS, CHARS, and Vital Statistics data; and 2) CHARS and Vital Statistics datasets. |
| The figure begins with a series of boxes organized into three columns. The three boxes arranged on top of each of the columns read 1) Inputs, 2) Activities, and 3) State Plan strategic improvement goals for the 10EPHS*. In the first column, entitled “Inputs,” are three boxes arranged from top to bottom. Inside the first box, entitled “HDSPP,” are the following bullet points: State Program Funding; Expertise of professional staff and infrastructure; Time Commitment; Internal Partners and Stakeholders; Communication; and Internal epidemiology support for surveillance and evaluation. Inside the second box, entitled “CDC,” are the following bullet points: Funding; Programmatic Guidance; Training; CDC Project Officer; Conference calls, listserv; and Technical Assistance. Inside the third box, entitled, “Steering Council Members,” is the following information: Invest staff time to implement strategies articulated in the State Plan, to meet State Plan Goals. All three boxes in the first column are connected with a right-pointing bracket to the boxes in the second column. |
| The second column, entitled “Activities,” contains one large box beneath the title box. The large box contains three major sections, each with information listed beneath. The three major sections are: 1) Primary Prevention, 2) Event, and 3) Secondary Prevention. Under the “Primary Prevention” section, three major areas are listed: 1) Worksite Health and Productivity, 2) Awareness of Signs and Symptoms, and 3) Washington State Collaborative. The “Worksite Health and Productivity” section lists the following two items: 1) External Healthy Worksite Summit, and 2) Grant and Internal Worksite Training. The “Awareness of Signs and Symptoms” section lists the following three items: 1) Outreach, African Americans; 2) Outreach, Native Americans; and 3) Activation of EMS-Acute Events. Under the “Event” major section, the following two items are listed: 1) Implement EMS TAC recommendations, and 2) Get with the Guidelines. Under the “Secondary Prevention” major section, six subheadings are listed: 1) Assess Rehabilitation Capacity, 2) NCQA Recognition, 3) Steering Council meetings, 4) Communication, 5) Epidemiology, and 6) Evaluation. Underneath the “Epidemiology” subheading, the following three items are listed: 1) Ongoing data analysis, 2) Disparities assessment, and 3) GIS analysis. Underneath the “Evaluation” subheading, the following four items are listed: 1) Steering Council evaluation, 2) Monitor performance measures, 3) Intervention evaluation, and 4) Awareness evaluation. |
| The third column, entitled “State Plan strategic improvement goals for the 10EPHS*,” contains one large box beneath the title box. The large box lists the following nine goals: 1) Comprehensive HD&S data to support the State Plan; 2) Surveillance used to identify persons who have, or at risk for HD&S; 3) WA residents know risk factors & warning signs of HD&S; 4) Convene Steering Council to implement State Plan Goals; 5) Action plan and policies empower consumers, result in improved health status and quality; 6) Establish process to assess and monitor State laws and regulations and identify opportunities to support the implementation of the state plan; 7) WA maintains and coordinated HD&S systems of prevention, screening, diagnosis, treatment and rehabilitation; 8) Education for HD&S prevention and management for licensed, registered, or certified HC providers; and 9) Evaluate Accessibility, Effectiveness and Quality of HD&S Prevention system. |
| To the right of column three are another series of three columns, each with a heading box above them. The first three columns already described are connected to the second series of three columns with a right-pointing bracket. The heading boxes above the second set of three columns read, “Supporting Outcomes (1-3 years),” “Medium-term Outcomes (3-5 years),” and “Long-term Outcomes (5 years +).” Underneath the column entitled “Supporting Outcomes (1-3 years)” is a box that contains the heading, “Primary Prevention,” followed by three bullet points: 1) Increased public awareness of the signs and symptoms of heart disease and stroke, knowledge to call 9-1-1 [noted as a CDC priority area]; 2) Worksite support for screening and control of risk factors AND for awareness of signs and symptoms of HD&S; and 3) Improved quality of primary care regarding control risk factors. The first column is connected to the second column with a right-pointing bracket. |
| Underneath the column entitled “Medium-term Outcomes (3-5 years),” is a box that contains three headings: 1) Primary Prevention, 2) Event, and 3) Secondary Prevention. Underneath the “Primary Prevention” heading are the following three bullet points: 1) Control of risk factors (blood pressure and cholesterol) achieved [noted as a CDC priority area]; 2) Increased recognition of symptoms; shorter time between recognizing symptoms and calling 911; and 3) Improved quality of care in primary care settings [noted as a CDC priority area]. Underneath the “Event” heading are the following two bullet points: 1) improved emergency response to acute events across the state [noted as a CDC priority area]; and 2) increased hospital compliance with protocols for optimal response to HD&S event. Underneath the “Secondary Prevention” heading is the following bullet point: improved rehabilitation and management of chronic condition following a heart attack or stroke. The second column is connected to the first two boxes of the third column with a right-pointing bracket and two arrows. |
| Underneath the column entitled “Long-term Outcomes (5 years +)” are four boxes arranged vertically. The first box contain the following information: Fewer first events; better survival rates following a heart attack or stroke; fewer repeat MI or stroke. The second box contains the following information: Decrease in disability associated with heart attack or stroke. The second box is connected to the third box with a downward-pointing arrow. The third box contains the following information: Reduce health disparities [noted as a CDC priority area] between general and priority populations. The third box is connected to the fourth box with a downward-pointing arrow. The fourth box contains the following information: Reduce deaths due to heart disease and stroke. |
| Underneath the six columns is the following text: “Rationale: by working towards reaching our state plan goals for improving the 10 EPHS, we enhance the capacity of HD&S public health system to achieve the short-term, intermediate, and long-term outcomes.” |