Using concept mapping to develop a logic model for the Prevention Research Centers Program.

INTRODUCTION: Concept mapping is a structured conceptualization process that provides a visual representation of relationships among ideas. Concept mapping was used to develop a logic model for the Centers for Disease Control and Prevention's Prevention Research Centers Program, which has a large and diverse group of stakeholders throughout the United States. No published studies have used concept mapping to develop a logic model for a national program.
METHODS: Two logic models were constructed using the data from the concept mapping process and program documents: one for the national level and one for the local level. Concept mapping involved three phases: 1) developing questions to generate ideas about the program's purpose and function, 2) gathering input from 145 national stakeholders and 135 local stakeholders and sorting ideas into themes, and 3) using multivariate statistical analyses to generate concept maps. Logic models were refined using feedback received from stakeholders at regional meetings and conferences and from a structured feedback tool.
RESULTS: The national concept map consisted of 9 clusters with 88 statements; the local concept map consisted of 11 clusters with 75 statements. Clusters were categorized into three logic model components: inputs, activities, and outcomes. Based on feedback, two draft logic models were combined and finalized into one for the Prevention Research Centers Program.
CONCLUSION: Concept mapping provides a valuable data source, establishes a common view of a program, and identifies inputs, activities, and outcomes in a logic model. Our concept mapping process resulted in a logic model that is meaningful for stakeholders, incorporates input from the program's partners, and establishes important program expectations. Our methods may be beneficial for other programs that are developing logic models for evaluation planning.

Introduction

The Centers for Disease Control and Prevention's (CDC's) Framework for Program Evaluation in Public Health provides public health practitioners and evaluators with a practical, six-step approach for effective evaluation (1). The framework helps public health programs address increased accountability requirements, program improvement processes, and public health decision making (1,2). The two initial steps in the CDC's evaluation framework are describing the program and engaging stakeholders. The program description step includes developing a logic model that visually depicts the hypothesized relationships among program resources, program activities, and the results the program hopes to achieve — in other words, the program's underlying theory of change (3). The CDC evaluation framework and other models recommend engaging stakeholders during the logic model development to increase the usefulness and validity of the resulting model (1,4-6). The logic model can then be used as the basis for future program evaluation efforts.

Examples are available of public health programs that have used participatory methods to develop logic models (3,7-9), but the methods used by the programs to encourage stakeholder input are not the focus of those publications. In addition, participatory methods for developing logic models have typically involved small or single-site programs or engaged a small group of program representatives. In this article, we detail the efforts of the CDC's Prevention Research Centers (PRC) Program, in which concept mapping was used to develop a national program logic model.

Concept mapping can be used to identify key elements of a program and show their relationships to one another (10,11). Several projects have used concept mapping to set priorities, plan programs, and evaluate programs (12-15). Although the methodology has been used previously to develop a logic model for a single program (16), we could find no published studies that used concept mapping to develop a logic model for a national program. We describe the application of concept mapping in the PRC Program, a large, multisite program with national, state, and local stakeholders distributed throughout the United States. These methods should be beneficial for individuals involved in programs that are developing logic models for evaluation planning.

The PRC Program funds 33 university-based research centers to conduct community-based participatory research and training on chronic disease and health promotion issues facing communities today (17). The PRC Program is the CDC's largest extramural research program and encourages academic, community, and public health collaboration in conducting prevention research and applying research in practice and policies (18). Stakeholders in the PRC Program include researchers in schools of public health, schools of medicine, and other academic departments; community members; community-based organizations; tribal organizations; public health practitioners in state, county, and city health departments; other government agencies; school administrators and teachers; national advocacy organizations and public health associations; the CDC; Congress; and many others. During the first year of the logic model development project (2001), the PRC Program funded 26 centers in 24 states.

To address the increased emphasis on accountability and meet the recommendations made in the 1997 Institute of Medicine (IOM) review of the PRC Program (19), the program's leaders decided to initiate a national evaluation strategy. Using the CDC evaluation framework as a guide (1), an evaluation planning project was funded, with the goal of engaging stakeholders to develop an overall program description and logic model (steps 1 and 2 of the CDC evaluation framework). An external evaluation contractor was funded to facilitate a participatory process that would ensure the key stakeholders of the PRC Program had a role in developing the logic model.

Methods

The national logic model was developed in three stages. First, we constructed a logic model draft using data from the concept mapping process. Second, we refined the draft through regional meetings with PRC Program stakeholders. Third, we distributed the draft and written narrative to stakeholders and obtained suggestions through a structured feedback tool designed to help revise the model.

A collaborative evaluation design team (CEDT) comprising representatives from major stakeholder groups was formed and oversaw all aspects of the project. This group included experts in community-based participatory research, public health, disease prevention, and program evaluation who worked in various settings, including universities, state health departments, voluntary health agencies, and local organizations. The CEDT assisted with the concept mapping process and development of the PRC logic model, communicated with the constituency represented by each team member, and advised the evaluation contractor and the CDC on all aspects of project implementation.

Concept mapping

We used concept mapping to develop our program framework, or logic model. Concept mapping provides a visual representation of the complex relationships among ideas and results and integrates qualitative processes with quantitative methods (20). Unlike other qualitative methods such as focus groups, concept mapping provides a structured approach that allows participants to identify issues and participate in the actual interpretation of their group perceptions (21). Concept mapping also incorporates statistical tools that provide precise and credible data from qualitative information. The method was selected because it can elicit ideas from large and diverse groups about an issue or a topic within a short time and because its design enables it to overcome geographic barriers (20,22).

The concept mapping process had three phases: 1) project planning, which included developing the focus prompt (i.e., the type of input desired) and identifying participants (November 2001–January 2002); 2) idea generation and structuring (February–March 2002), and 3) analysis and interpretation of the concept maps (April–June 2002). During each step, we encouraged ongoing communication through committee meetings and conference calls to obtain stakeholder input and provide updates about each step of the concept mapping process.

Project planning phase

The evaluation contractor collaborated with the CEDT to develop the following two prompts to elicit ideas about the purpose and function of the PRC Program, with one focusing on the national level and one on the local level:

To ensure national excellence in prevention, a Prevention Research Center should have the following specific characteristic or function . . .

To successfully promote health in a community, an effective Prevention Research Center should have the following specific characteristic or skill . . .

We compiled a list of 175 PRC Program stakeholders to participate in the concept mapping process using the nationally focused prompt. Stakeholders included representatives from national organizations, such as Chronic Disease Directors, Directors of Health Promotion and Education, Association of Schools of Public Health, and Association of Teachers of Preventive Medicine; members selected from the IOM report review committee (19); CDC leaders familiar with the PRC Program; CDC program staff members; the PRC national community committee, which is composed of representatives from each PRC community committee, who advise the program, facilitate training of community members, and educate about prevention research (23); and PRC leaders, such as principal investigators, directors, administrators, and researchers from the PRCs. PRC leaders could invite other key stakeholders such as university leaders to participate in the brainstorming process at the national level.

We generated a similar list of 165 stakeholders to participate in the concept mapping process using the locally focused prompt. Participants were identified from the following groups: PRC community committees, research participants, health department partners, and PRC leaders such as principal investigators, directors, administrators, and researchers from the PRCs.  Because we knew that some stakeholders might not be able to respond online or by fax, and to ensure that the community's input was obtained, we selected a community liaison in each PRC who assisted community representatives in the concept mapping process. We invited some stakeholders who had national and local perspectives on the PRC Program to respond to both focus prompts.

Idea generation and structuring phase

We invited participants to submit up to 10 ideas in response to the focus prompt using a secure Web site or by mailing or faxing their ideas to the evaluation contractor. Because participants submitted their ideas anonymously, we could not calculate exact response rates or the average number of items submitted per respondent. However, based on unique identifiers, we estimated that 145 stakeholders (83%) responded to the nationally focused prompt, and 135 responded (82%) to the locally focused prompt.

Members of the CEDT reviewed the statements that had been generated for each prompt and eliminated repetitive statements, yielding 88 unique statements for the national responses and 75 unique statements for the local responses. The statements were sorted into themes (24). The national and local statements were then sorted independently by two subsets of participants who were selected for their familiarity with PRCs. For the national statements, 35 stakeholders were contacted, with 20 (57%) resulting participants. For the local statements, 30 stakeholders were contacted, with 17 (57%) resulting participants. The individuals were asked to sort the statements into categories, or themes, based on similarity of ideas. Participants either used the project's Web site to sort the statements into categories or manually sorted statements that had been printed on cards. Participants were asked to create their own categories; they were told that each statement could be placed into only one category, and the sorting process should result in more than one category but fewer categories than the total number of statements.

Analysis and interpretation phase

We used a software tool designed for multiple stakeholder input (Concept Systems, Inc, Ithaca, NY) to construct two separate concept maps (12). An expert in concept mapping conducted the analysis. First, a similarity matrix was constructed that represented the relative similarity of participants' sorting statements. Second, the total similarity matrix was analyzed using nonmetric multidimensional scaling analysis with a two-dimensional solution, which generated x and y coordinates in two-dimensional space for each statement based on its mathematical similarity to other statements. Configuring the multidimensional scaling of the statement points in two dimensions on a point map was the foundation for the final results. Third, statements were combined into clusters using a hierarchical cluster analysis. The results of the hierarchical cluster analysis were superimposed on the multidimensional scaling results to create a map displaying the points graphically within each group, with polygonal boundaries surrounding the points in each cluster group. A hierarchical cluster analysis yields all possible cluster solutions, from each statement in its own cluster to all statements in one cluster. A standardized, systematic process is applied to identify the most useful cluster number for each project. The appropriate number of clusters is determined by working with subject experts who consider the range of issues represented, the purpose and intended uses of the resulting map, and the observed coherence of clusters at different levels (21).

The CEDT reviewed the two PRC Program cluster maps and the statements associated with each cluster. The CEDT members then agreed on a theme and label for each cluster on both maps. These maps became the national- and local-level concept maps for the PRC Program.

Developing the logic models

We developed a draft logic model diagram, showing PRC Program inputs, activities, and outcomes and incorporating data from the concept mapping process. This information was supplemented by information from program documents. We presented the draft logic models at regional meetings in May and June 2002 and distributed the logic model with a written narrative in a structured feedback tool in September 2002. We used the feedback received through these mechanisms to make final revisions. The final logic model and narrative were then broadly distributed to the PRCs and other stakeholders.

Results

Concept maps

The national-level concept map had nine clusters (Figure 1):

Figure 1

National concept map showing 9 clusters and 88 statements. PRC indicates Prevention Research Center; CDC, Centers for Disease Control and Prevention.

Diversity and sensitivity

Community engagement

Research methods

Research agenda

Core expertise and resources

Active dissemination

Technical assistance

Training

Relationships and recognition

The local-level concept map had 11 clusters (Figure 2):

Figure 2

Local concept map showing 11 clusters and 75 statements. PRC indicates Prevention Research Center; CDC, Centers for Disease Control and Prevention.

Communication and dissemination

Outreach

Promotes community involvement

Responsive to community input

Builds community capacity

Committed community advisory group

Trust

Defining and measuring community outcomes

Training and mentoring

Human resources

Translation of research to practice

Development of the program logic model

We placed the concept map data into the appropriate columns of the logic models: program input, activity, or outcome (Tables 1 and 2). For example, the core expertise and resources cluster from the concept map (Figure 1) was placed in the input column of the draft national logic model (Table 1). Likewise, the community engagement cluster was placed in the activities column of the national logic model. We continued this process until all clusters from the national concept map had been categorized into the columns of the national logic model. Using the same process for the local logic model, we placed the committed community advisory board cluster from the local concept map (Figure 2) into the input column of the local logic model (Table 2) and the trust cluster from the concept map into the outcome column of the model. The remaining cluster information from the local concept map was placed into the appropriate columns of the local logic model. We reviewed program documents, such as the IOM report (19), authorizing legislation (25), and PRC guiding principles (17), to identify other activities and outcomes relevant to the program. Information from these documents augmented the concept mapping data.

Table 1

Draft Components of the National Logic Model for the Prevention Research Centers (PRCs): Inputs, Activities, and Outcomesa

Inputs	Activities	Outcomes
Committed community advisory groupb PRC capacity: Core expertise and resourcesc Faculty and staff diversityc Faculty and staff sensitivity to community issues Facility Communication and data systems Relationships with community partners, other PRCs, and the Centers for Disease Control and Preventionc	Community engagementc Establishment of research agendac Core and other research using sound research methodsc Testing of innovative strategiesb Active dissemination of research findingsc Trainingc Technical assistancec	Translation of research to practiceb Research and other publicationsb Widespread knowledge of effective interventionsb Relationships and recognitionc Trustb Widespread use of effective interventionsb

These draft components were shown to project participants as a full logic model diagram.

Additions that are based on information from other PRC Program documents and materials, such as the Institute of Medicine report (19), authorizing legislation (25), and PRC guiding principles (17).

Elements that are based on clusters from the national concept map (Figure 1).

Table 2

Draft Components of the Local Logic Model for the Prevention Research Centers (PRCs): Inputs, Activities, and Outcomesa

Inputs	Activities	Outcomes
Committed community advisory groupb PRC capacity: Human resourcesb Financial resourcesc Facilityc Communication and data systemsc Community partnersc	Responsiveness to community inputb Promotion of community involvementb Core and other researchc Testing of innovative strategiesc Communication and dissemination of research findingsb Community outreachb	Translation of research to practiceb Research and other publicationsc Knowledge of effective interventionsc Trustb Widespread use of effective interventionsc

These draft components were shown to project participants as a full logic model diagram.

Elements that are based on clusters from the local concept map (Figure 2).

Additions that are based on information from other PRC Program documents and materials, such as the Institute of Medicine report (19), authorizing legislation (25), and PRC guiding principles (17).

We presented the draft logic models at three regional meetings. The meetings were attended by 57 participants representing academic, community, and public health partners within the PRC Program. Based on comments received, we combined the two draft logic models into one logic model for the national PRC Program. Meeting participants agreed that the single PRC Program logic model should reflect the key clusters from the locally focused prompt that were not associated with the nationally focused prompt: community capacity building, trust, and translation of research to practice.

We distributed the single national logic model with a written narrative in a structured feedback tool. Representatives in 28 PRCs (rather than 26, because two additional PRCs had been funded) received the feedback tool, including members of the Chronic Disease Directors, the Directors of Health Promotion and Education, the PRC National Community Committee, and the CDC program staff. We asked each PRC to gather input from various respondents, including academic and community partners, and then provide a single response representing the individual PRC. The PRCs were asked to send their comments to the evaluation contractor; the response rate was 100%. As a result of the feedback, the logic model underwent minor revisions.

The PRC Program office at the CDC distributed the final logic model and accompanying narrative to program stakeholders and posted it on the PRC Program Web site (http://www.cdc.gov/prc/). We have presented the logic model at several national evaluation, public health, and health education conferences and meetings, such as the National Conference on Chronic Disease Prevention and Control and meetings of the American Public Health Association, American Evaluation Association, and Society of Public Health Educators.

Discussion

Concept mapping can be a useful tool for constructing a logic model for a national program. We identified several benefits from our experiences with the PRC Program. First, the most obvious benefit was that the logic model was based on a set of concepts that came directly from stakeholders. The concept map and underlying statements served as the foundation for the logic model refinement process. In addition, components of the final logic model were easily linked to the original concept mapping ideas submitted by stakeholders. Second, compared with an initially proposed logic model (available upon request) developed by a few CDC staff members and select partners, the logic model based on the concept mapping data was more comprehensive and representative of the processes and outcomes involved in prevention research. For the first time, community representatives could see themselves visually represented in a program's activities and outcomes. For example, their role in establishing a research agenda is clear, as is the intended outcome of enhanced community capacity for disease prevention.

Consistent with the CDC framework for evaluation recommendations, engaging stakeholders in the development of the program logic model was worth the investment of resources (1). Concept mapping encouraged participants to provide their opinions about the PRC Program anonymously during the idea-generation phase. The ability to provide anonymous input was important during the early project phases because trust was being established among the various stakeholder groups. Combining concept mapping with other methods for eliciting feedback throughout the project helped address the significant numbers of stakeholders who expressed differing views or general skepticism about the process, an issue that may be inherent in any large, multisite program. Overall, open discussions, compromise among people with conflicting views, transparent use of feedback and decision making, inclusion of stakeholder perspectives, and repeated explanations of the process were important methods for keeping all participants positively engaged and supportive of the final product. Our experiences and challenges were similar to those reported in other participatory evaluation process reports (26,27) and will be presented in another article.

Concept mapping has gained acceptance by researchers in the last 15 years; in the last 5 years, its use has been facilitated by Web applications for participant data collection and analysis. In addition, online data collection methods are more cost-effective and efficient than other participatory methods involving large groups. Another benefit of using a Web-based system is that the initial maps can be presented to stakeholders quickly. In our experience, the process allowed us to gather data from stakeholders in numerous geographic areas and then present the concept maps to PRC representatives 1 month after the idea generation and structuring were completed.

Concept mapping as a tool for developing a logic model does have some shortcomings. First, a logic model derived from a concept map is based on stakeholder perspectives; it is not a tested theory of how a program functions and arrives at intended outcomes. Therefore, it may not reflect some realities of program implementation and outcomes (4). Future evaluation efforts in the PRC Program will clarify the concepts and logic in the national model. Second, concept mapping was a new process for most stakeholders. Many who were not familiar with qualitative methods and terminology initially struggled to understand how the concept mapping activities would result in the construction of a logic model for the program. Finally, although many diverse perspectives are represented in the findings of the concept mapping process, they should not be interpreted as representing the views of all stakeholders.

Given the challenges faced during the project, we recommend using three of the strategies we found most helpful. First, program evaluation experts should be used to obtain the information from the concept mapping statements and other program documents to construct an initial logic model. Second, stakeholders should be fully informed about the concept mapping process and given concept mapping examples (such as this article) so that they can become familiar with the use of concept mapping as a tool for logic model development. Third, concept mapping data should be supplemented with program documents and stakeholder feedback, a strategy that is consistent with recommendations for using multiple methods for developing a program theory (6). Future evaluation project planners should consider using electronic methods for gathering feedback, such as Web-based conferencing and telephone focus groups.

Concept mapping is a valuable method for developing a logic model, particularly for a large program with a diverse group of stakeholders. Having a national logic model has permitted the PRC Program to identify its centers' outcomes and functions. The process and final logic model has incorporated the input of the program's national and community partners, engaged stakeholders, and provided the PRC Program with a platform on which to design and implement a national evaluation strategy.

Communication and dissemination cluster: associated statements Communication with the public using multiple different media; success at disseminating products of research with other PRCs; reports research findings back to community leaders; a strategic plan for disseminating products of research; success in disseminating research results back to the community in a useful form; effective communication with multiple audiences such as public health agencies, private health agencies, community groups, and the public; documented publicity of PRC events and activities; effective communication with policy makers and law makers; success in sharing evidence-based programs in addition to conducting research; develops research reports that are used by community leaders

Outreach cluster: associated statements Success in sharing the "how-to's" of community involvement; an effective system of communication among partners and the community; continuous tracking and reporting of health indicators to community health agencies; develops awareness in the community of the difference between the economic impacts of health care and the economic impacts of health promotion and disease prevention; PRC staff that have designated responsibility for community outreach.

Promotes community involvement cluster: associated statements Effective leadership that models and supports sensitive approaches to community partnerships; ability to advocate for community involvement in planning, implementing, and evaluating intervention research; demonstrated desire to share resources, power, and expertise with partners; success in implementing the principles of community work and knowing how these principles differ from clinical research; ability to link community members with relevant decision makers; knowledge of the community power structure and the decision-making systems in a community; capacity to share control with partners

Responsive to community input cluster: associated statements An agenda that is largely determined by the community; research that is driven primarily by community needs rather than funders; participation of community members in all aspects of research design and intervention; an ability to elicit and monitor community concerns; documented strategies for developing a sense of responsibility among the community groups who assist with research; documented input from community representatives regarding the accomplishments of the PRC; success at responding to input from the community even when it may not result in ideas that are fundable; prioritized community research needs based on community input; effective use of data from the community (e.g., needs assessment) to identify and address specific health problems; demonstrated understanding of the structure of the community; willingness to act on community recommendations; an equal partnership between the PRC and community members; demonstrated respect for contributions of all members of the community

Builds community capacity cluster: associated statements Demonstrated responsiveness for improving the health of the community; assessment of a community's capacity to implement health promotion and disease prevention among its community members; documented plans to allow time for the community to build ownership; programs that target the health needs of the community served; a demonstrated genuine care and concern for the health and well being of the community; established methods to enhance a community's capacity

Committed community advisory group cluster: associated statements An established community advisory group for the PRC; documented guidelines addressing the role of the community advisory group; a community advisory board with members knowledgeable about specific needs and assets of the community; a diverse community advisory group with respect to multiple criteria, such as ethnicity, organizations represented, and target health condition; a community advisory group for the core research project

Trust cluster: associated statements Recognition by members of the defined community that the PRC's activities benefit that community; recognized as a resource for the community; credibility within the community the PRC serves; considered as trustworthy within the community the center serves

Defining and measuring community outcomes cluster: associated statements Commitment to act on the issues identified by prior research; a broad definition of health incorporated into the PRC’s mission; a documented focus on social issues that will have a beneficial effect on multiple disease risk problems; measurable PRC objectives; a multidisciplinary approach that goes beyond a sole focus on individual behavior changes; measures to define the community; evaluation of progress and success with new initiatives

Training and mentoring cluster: associated statements Success at training students to be culturally and ethnically sensitive; ongoing training and learning opportunities for local public health practitioners; a mentorship or scholarship program that involves underrepresented groups or individuals

Human resources cluster: associated statements Incentives or a reward system for faculty who undertake community-based, participatory research; adequate facility and personnel to support projects; capacity to effectively address disparities in health care and outcomes; research staff with "real-world" experience working with communities — that is, staff who have worked outside of academia; staff with dissemination skills; a diverse team of staff and faculty; established skills in community organization and community action; a diverse PRC staff; a PRC staff that includes community members; demonstrated ability to sustain successful programs

Translation of research to practice cluster: associated statements Success in implementing community-based programs that have been shown to be effective; effective translation of research findings into practice; scientists who can communicate with nonscientists; translation of research findings into practice using culturally effective methods; capability to translate research into practical strategies to change practice and policy