Evaluating community engagement in an academic medical center.

From the perspective of academic medical centers (AMCs), community engagement is a collaborative process of working toward mutually defined goals to improve the community's health, and involves partnerships between AMCs, individuals, and entities representing the surrounding community. AMCs increasingly recognize the importance of community engagement, and recent programs such as Prevention Research Centers and Clinical and Translational Science Awards have highlighted community engagement activities. However, there is no standard or accepted metric for evaluating AMCs' performance and impact of community engagement activities.In this article, the authors present a framework for evaluating AMCs' community engagement activities. The framework includes broad goals and specific activities within each goal, wherein goals and activities are evaluated using a health services research framework consisting of structure, process, and outcome criteria. To illustrate how to use this community engagement evaluation framework, the authors present specific community engagement goals and activities of the University of Rochester Medical Center to (1) improve the health of the community served by the AMC; (2) increase the AMC's capacity for community engagement; and (3) increase generalizable knowledge and practices in community engagement and public health.Using a structure-process-outcomes framework, a multidisciplinary team should regularly evaluate an AMC's community engagement program with the purpose of measurably improving the performance of the AMC and the health of its surrounding community.

Although the United States has the highest per capita health expenditures in the world, it lags behind many developed nations in important measures of health.[1] Thus, there is within the United States an evolving appreciation for the principles of population health and a new focus on health promotion rather than disease treatment. Academic medical centers (AMCs) are under increasing pressure in all of their mission areas—education, research, and patient care—to demonstrate improvements in health,[2] health care quality, and cost by addressing the behavioral, social, and environmental determinants of health.[3] Accomplishing this requires collaboration with the communities that AMCs serve.

Community engagement is a relatively new activity within many AMCs. In 1997, the Centers for Disease Control and Prevention (CDC) defined community engagement as:

the process of working collaboratively with and through groups of people affiliated by geographic proximity, special interest, or similar situations to address issues affecting the well-being of those people.… It often involves partnerships and coalitions that help mobilize resources and influence systems, change relationships among partners, and serve as catalysts for changing policies, programs, and practices.[4]

Of note, some AMCs do have a history of engaging their surrounding communities to focus on consensus-based health problems. For example, in the 1920s, George Eastman (founder of the Eastman Kodak Company) funded a new medical school in Rochester, New York (now the University of Rochester School of Medicine and Dentistry) with the proviso that “the skills and talents be used to make Rochester the healthiest community in the world.”[5] However, most AMCs have until recently focused on principal missions of medical education, clinical care, and scientific research, while paying less attention to the health of their surrounding communities.

This appears to be changing. Over the past decade, a number of factors coalesced to increase AMCs’ attention on the importance of community engagement. First, a growing body of literature has highlighted the prominent role of behavioral, social, community, and environmental factors in the health of populations[1,6,7]—factors that influence the health of populations as much as or more than do strictly “medical causes.”[1] Investigators from multiple disciplines have demonstrated that collaboration between academic and community partners can enhance translation of scientific knowledge to clinical and community programs.[4,6,8] As the nation seeks to improve population health, transform the health care delivery system, and reduce costs, AMCs increasingly recognize the value of working with the community to identify critical health challenges and their potential solutions.[8,9] In addition to the moral and public health imperatives, financial imperatives force AMCs to address these issues as payment policies become more population- and outcomes based.[10]

Second, changing demographics and rising poverty in urban areas, together with the growth of hospital-based health care systems and the decline of the manufacturing sector, have elevated many AMCs to positions of economic and political leadership within their communities.[11] As a result, communities are increasingly demanding that AMCs serve the public good. In addition, new Affordable Care Act (ACA) and Internal Revenue Service regulations related to nonprofit organizations require institutions to demonstrate “community benefit”[12,13] by developing community service plans jointly with community partners.

Third, several major federal initiatives have focused AMCs’ attention on community engagement. The CDC’s Prevention Research Center program,[14] which currently funds 37 centers, requires community-based participatory research as a core component. More recently, the Clinical and Translational Science Awards (CTSAs),[15] which now fund 61 centers, encourage AMCs to include a community engagement component. In fact, many AMCs have made important strides in community-based participatory research and community engagement as demonstrated by recent examples of successes.[16-18]

Finally, health reform stimulated by the ACA has resulted in an explosion of new collaborations as AMCs have either formed accountable care organizations or partnered with other organizations to develop large health systems that are responsible for the health outcomes of populations.[19,20] In sum, converging factors have stimulated AMCs’ interest and investment in community engagement.

Community engagement at AMCs can occur within multiple contexts,[4,20] all of which are core functions of medical centers, including education, clinical activities linked with community-based organizations, research (particularly community-based participatory research or community-engaged research), health policy, and community service. Community engagement can involve many types of community-based partners: community and neighborhood organizations, individual community leaders, other institutions such as schools and workplaces, local government public health officials, and community-based coalitions that focus on key populations (e.g., youth, HIV, Latino, disabled) or priority issues within the community (e.g., lead poisoning, violence, obesity).

With the financial stresses that require all organizations to closely examine the effectiveness of their programs, AMCs are increasingly interested in evaluating the effectiveness and return on investment of their community engagement activities. In addition, the new funding cycle of CTSAs, unlike the prior cycles, no longer requires community engagement; thus, some AMCs may consider reducing or dropping these activities unless there is clear evidence of their value to institutions and communities alike.

Unfortunately, there is no standard or accepted metric for evaluating community engagement.[4] Unlike educational or research programs, which have standard guidelines, procedures, and metrics, AMCs’ community engagement activities vary widely and are often tailored to their specific communities. A recent joint publication from the Department of Health and Human Services, the National Institutes of Health, CDC, the Agency for Toxic Substances and Disease Registry, and the CTSAs[21] listed five types of evaluation for community engagement—(1) formative, (2) process, (3) summative, (4) outcome, and (5) impact—and stressed the importance of using both qualitative and quantitative methods for evaluation. This publication also provided successful case examples of community engagement, yet few of these programs displayed results of rigorous evaluations.

At the University of Rochester Medical Center (URMC), we have developed a framework for evaluating our community engagement activities in Rochester, New York. In 2006, the URMC established the Center for Community Health, which consolidates community engagement activities and provides a core infrastructure to link service, education, research, and policy programs with the community. The center has since grown to more than 60 faculty and staff and has developed a variety of community programs and research initiatives focused on population health improvement. In addition, the center supports and collaborates with programs from many other departments and schools at URMC that have successfully engaged the community. In this article, we describe the framework we used to evaluate the URMC’s community engagement activities, and provide specific local examples. Although not all AMCs will have the type of core infrastructure that URMC’s Center for Community Health provides, we hope that the evaluation framework can serve as a template for other AMCs in their quest to evaluate their own community engagement activities.

URMC’s Framework for Evaluating Community Engagement

At the URMC, we use a traditional health services research framework[22] to evaluate an AMC’s community engagement activities by assessing the structure, process, and outcomes of these community engagement activities. This framework was developed by a group of URMC faculty members interested in community engagement and community health improvement, in response to the concern that there were no commonly accepted metrics for community engagement and with the desire to codify our thinking as we developed and evaluated URMC’s community engagement activities. We have used components of this framework to define, drive, and evaluate our activities within URMC and in the community, and we plan to apply it more systematically in the future. With this framework, we can identify the key elements required for transformation through community engagement. These elements are reflected in the Community Health Strategic Plan and will provide a template for an ongoing evolution of a Prevention Strategic Plan under development by URMC.

In Chart 1, we show the key components of the framework—community engagement goals and activities—with the structure, process, and outcome elements that should be common to all robust community engagement efforts.

Chart 1

A Framework to Evaluate an Academic Medical Center’s (AMC) Community Engagement (CE) Activities

For community engagement goals, we suggest that AMCs develop three ambitious yet attainable 5- to 10-year goals: (1) to impact the surrounding community, (2) to impact the AMC itself, and (3) to impact population health through generalizable knowledge and practices. Of course, AMCs may tailor goals for their own communities. Within each goal, AMCs should have specific, focused objectives. For simplicity, these are not included in Chart 1, because they are likely to be specific to each AMC.

Major community engagement activities involve large-scale, multicomponent efforts designed to achieve each community engagement goal. These activities should span many disciplines and years, and include core functions of the AMC.

To assess progress, we propose three evaluation criteria: structure, process, and outcomes. We suggest that each AMC engage an evaluation team that includes health services researchers; community experts within and outside the AMC; community members; and education, policy, and administrative leaders. These teams, which optimally would be distinct from the individuals leading the community engagement activities, should regularly evaluate the community engagement activities and report to AMC leadership.

Community engagement activities evaluation framework

The structure-process-outcomes framework, which we use to evaluate community engagement activities, emanates from the seminal health services research framework first defined by Starfield and colleagues.[22]

Structure.

In the context of our framework, structure represents the administrative arrangements and committees that are developed, the new organizations established to enhance community engagement goals, any new facilities or space, and financial as well as nonfinancial arrangements regarding community engagement. Structural elements include percent effort of faculty and staff that are supported by the AMC for community engagement activities, and types of experts working on these activities. Each AMC should determine the extent of structural components needed to achieve its community engagement goals; one evaluation goal is to assess the adequacy of these structural elements.

Process.

A description of activities undertaken, process should include both qualitative and quantitative assessments of activities. In health services language, process assessments often ask two questions: (1) What services were delivered (i.e., were they appropriate and necessary)? and (2) How well were they delivered (i.e., were they delivered with fidelity and rigor, and in a timely, patient-centered, and culturally sensitive manner)? To evaluate community engagement, a process evaluation would measure the number and quality of community engagement activities and assess the perceptions of both the individuals delivering the interventions and the recipients. Process can be measured by (1) review of documents (e.g., activity logs, minutes), (2) interviews of key individuals, (3) quantitative surveys of constituents, and (4) observational and quasi-experimental research studies.

Outcomes.

Although they are the most challenging to measure and attain, outcomes are also the most important component in assessing community engagement. One key domain of outcomes involves health metrics within the community or catchment area surrounding the AMC. Truly successful community engagement should result in improved community health. Different communities will use different metrics to measure community health depending on their capacity to measure a variety of indicators, but in general, community-level measures should be consistent with national metrics such as those incorporated into Healthy People 2020.[23,24] Recent developments in electronic medical records and other data resources provide new opportunities to collect these outcome measures. A second domain of outcomes involves educational metrics, because many community engagement activities will include an educational component. We recommend Kirkpatrick’s Model of Education[25] to assess the knowledge, skills, and behavior of learners as well as the health outcomes of their patients or target audiences. For example, a long-term educational outcome of a community engagement training program could involve demonstration of improved health of a specific patient population, resulting from the community engagement activities performed by graduates who completed community engagement training within an AMC.

This evaluation framework is designed to assess key goals and activities of community engagement. We have not attempted to develop a composite scoring system to measure more precisely the potential return on investment for community engagement activities. In addition, return on investment accrues for both the AMC and the community itself, and is thus difficult to measure.

Evaluating an AMC’s Community Engagement Goals Using Examples from Rochester, New York

Our three overarching goals for community engagement at URMC—local community impact, AMC impact, and national and global impact—have their beginnings more than two decades ago when URMC leadership committed to the first goal: to measurably improve the health of our community. The second goal addresses impact on the institution itself: to increase institutional capacity for community engagement and thereby its value to the community, and community credibility/trust in the institution. The third goal seeks to have global impact by increasing generalizable knowledge and practice through research, education, collaborations, and advocacy. To address each of these goals, URMC has undertaken major activities. In the following section, we describe some examples using the structure-process-outcome framework. For a list of some of URMC’s community engagement goals and activities, as well as our specific findings for the activities, see Chart 2.

Chart 2

Evaluating Community Engagement (CE) Activities at the University of Rochester School of Medicine

Goal 1: Local community impact

Our first goal is to improve the health of the local community directly served by the URMC, with particular emphasis on Monroe County which has a population greater than 750,000 (including the city of Rochester, which has a population of about 250,000). Here, we provide an example of how we evaluated one activity—technical assistance in community health improvement—using the structure-process-outcomes framework.

Initiated in the early 1990s, Health Action is led by the Monroe County Department of Public Health. Its goal is to improve the health of the community in specific, measurable areas, through a process of community-wide process-improvement activities. Health Action provides no funding for its recommendations but, rather, creates a blueprint for the community to seek funding to support priority health interventions and research. URMC community engagement leaders and experts have worked closely with county department of health leaders to design and implement Health Action. Health Action has assisted community partners by providing community-level health assessments for key health measures (e.g., county-wide immunization rates or obesity rates), helping the county prioritize areas for support for upcoming years, implementing quality improvement strategies within health systems, and evaluating interventions.

The URMC made several structural enhancements to address these activities. Key community organizations such as the African-American Health Coalition have naturally forged coalitions with the URMC’s Center for Community Health; leaders of these community organizations serve on a very active Community Advisory Council at the Center for Community Health. The URMC revised its infrastructure to increase financial support for the Center for Community Health and elevate its director to senior-level leadership status within the URMC. The URMC improved its communication with the community through regular meetings, seminars, a newsletter, and direct bidirectional involvement in activities, many through the center but others emanating from URMC departments. Finally, URMC established both intramural and extramural funding streams for small grants to foster community-engaged research and service activities. Assessment of the strength and merit of collaborations is the primary factor in funding decisions.

Health Action is a process of community-wide continuous quality improvement. On a regular basis, local health leaders and community members from a variety of disciplines and organizations (including our AMC) review the health status for all age groups and set health priorities for children, adolescents, and adults through broad community engagement. Collaborating organizations, including URMC, then develop action plans and focus resources to address the highest priorities. URMC had a long-standing relationship with the Monroe County Department of Public Health preceding the creation of the Center for Community Health. Over the years, faculty and staff helped lead Health Action committees and participated in specific interventions. The URMC, through its own funds and sponsored research grants, has provided financial support for collaborating organizations of Health Action. These activities are detailed in Health Action Report Cards published by the health department.[26]

One of the Health Action priorities in the 1990s involved improving childhood immunization coverage, based on prior data showing low coverage overall and substantial disparities between immunization coverage rates among children residing in the city of Rochester compared with those in the suburbs of the county. The project began as a randomized clinical trial of a reminder/recall/outreach program (patient navigator program) to improve infant and childhood immunization rates. The trial improved rates by 20 percentage points. With leadership from URMC faculty, Health Action supported a community-wide collaborative that engaged all three of the community’s hospital-based health systems and the two largest insurers to implement the program throughout the city of Rochester and also to continuously evaluate the program by measuring immunization rates every three years county-wide. As a result, childhood immunization rates have improved, and preexisting disparities in immunization rates between city–suburb and across racial/ethnic groups have been virtually eliminated.[27-30] For a full description, see Box 1.

We implemented, evaluated, and then disseminated widely a patient reminder/recall/outreach (patient navigator) program designed to improve childhood immunization rates in the city of Rochester. We developed specific structure, process, and outcome indicators for URMC’s contribution toward Health Action.

Structure. Structural components included teams led by URMC faculty and resources provided by the URMC (as distinguished from external funding). Metrics included specific in-kind personnel and nonpersonnel support by pediatric and social work leaders, and substantial funding of outreach workers by the URMC through its “Community Services Plan.”

Process. These included (1) participation by URMC members in specific Health Action interventions, (2) surveys and focus groups facilitated by URMC faculty, and (3) participation in data analyses, writing, and dissemination of Health Action Report Cards. Project-specific process indicators included feedback from families as well as tracking of the procedures carried out—the number of mailed/telephone reminders and home visits, referrals, and specific actions performed by the patient navigators.

Outcomes. These included county-wide immunization metrics and disparities in these metrics by city/county and by race/ethnicity. A community-wide coalition led by URMC faculty has, every three years, measured and reported on county-wide childhood immunization rates and disparities in rates between the city and suburbs, both of which have improved markedly since the start of Health Action.[29] We have since expanded this model to serve adolescents and adults, respectively.[28,30]

Measuring the structure, process, and outcomes of Health Action illustrates a major challenge in evaluating the community engagement impact of an AMC. Because community engagement is collaborative by definition, credit for these collaborative activities is shared between the AMC and the community. In the case of Health Action, URMC faculty members collaborated with many community partners to carry out Health Action activities and assessments. Because the process is designed for equal rather than directed collaboration, we cannot clearly attribute specific efforts to specific results. Thus, it is not easy to measure the “return on investment” for one partner alone.

Goal 2: AMC impact

Our second goal reflects the impact of community engagement activities on the AMC itself—enhancing the quality of research, education, and service activities. We articulated five broad URMC community engagement activities, which are detailed in Chart 2. The following paragraphs summarize some of the structural, process, and outcome components that are involved in many of these five activities.

Educating faculty, staff, students, and others about community engagement and population health is a priority for URMC. Several URMC faculty members ensure that students at all education levels learn about community engagement through courses as well as experiential learning. Faculty and staff participate in a community-engaged faculty group, and leaders of experiential learning meet monthly to ensure institutional collaboration along a learning continuum. Regular communications through in-house listservs update and involve faculty and staff in community engagement activities within the AMC. Online community engagement learning modules are available to all faculty, staff, and students within URMC and incorporated into several graduate courses. Additional learning opportunities, including video recordings of public health grand rounds, can be accessed through the Center for Community Health’s Web site.

To evaluate the process by which community engagement affects the URMC, we are measuring the learning of faculty/staff, students, and community partners and the behavior and skills of faculty/staff and student activities that may foster sustainable contributions to community health improvements. Additional educational metrics include the number of students involved in community engagement projects during their education and their subsequent career trajectories after graduation. We use minutes, reports, and reference documents from various committees, as well as minutes and health status reports created by the coalitions, to qualitatively and quantitatively assess activities.

Community involvement in and awareness of the scope of URMC community engagement activities has effectively occurred through two major mechanisms. The URMC Community Advisory Council has provided advice and guidance to the medical center on all matters of community engagement since 2006. We as well as others[31] believe that community perspectives are critical. Second, community coalitions such as the African American and Latino Health Coalitions and the Deaf Health Community Committee, convened and staffed by a community organization, serve a vital role in advising URMC community-engaged researchers and other leaders. Short-term outcomes include the number of community members serving on the Community Advisory Council and community coalitions, the number and diversity of community organizations represented in these groups, and the number of community members trained in community-based participatory research and community engagement who are qualified to serve as community researchers. Additional outcomes include academic deliverables such as the Principles of Community Engaged Research[21] and a list of research priorities established through the Community Advisory Council process.[32] Although there are not separate measures of AMC culture change, the other short- and longer-term outcome measures represent key indicators of the AMC’s progress toward a focus on and investment in community health. For a specific example of how community engagement activities have affected the URMC itself—the establishment of the Healthy Living Center—see Box 2.

A response to a Health Action Priority, the development of the Healthy Living Center within URMC’s Center for Community Health is one example of the community having a major impact on URMC. Improving health behaviors was identified as one of two critical Health Action priorities.

Structure. The Healthy Living Center was established within the Center for Community Health in a community location, and funded by a supplement to the Clinical and Translational Science Award, for the purpose of translating basic behavioral science into interventions applicable in community and clinical settings. The triple aims of research, community intervention, and clinical preventive services address the community’s request for a serious institutional commitment to prevention. The Healthy Living Center is supported by the Center for Community Health infrastructure and directed by a team of clinicians and scientists. Services in the Healthy Living Center are paid for by a human resource contract for employee participants, insurer payments for referred patients, and extramural grant funds for community interventions. Research funding comes primarily from the National Institutes of Health, but increasingly from other federal agencies.

Process. The Healthy Living Center was established by a team of clinicians and investigators with expertise in health behavior change based on basic psychological research in the field of self-determination theory. The Healthy Living Center includes multiple, evidence-based programs translating behavioral science into community programs; a variety of studies addressing health behavior improvement; and individual treatment for obesity, tobacco use, stress, and chronic disease management.

Outcomes. The Healthy Living Center is an innovative and community-responsive approach to addressing critical health behaviors. Over 2,000 individuals have participated in its individual and group programs over the three years since its founding with consistent and significant improvements in health behaviors. Approximately 20,000 community members have participated in community-based education and health promotion activities, providing increased awareness and much-needed health education, as a first step to permanent health behavior change. The Healthy Living Center team has generated many grant proposals and publications and has become an important training facility for learners.

URMC’s National Center for Deaf Health Research is another example of our institution’s response to a community need. Rochester has one of the largest per capita deaf and hard-of-hearing populations in the world and is home to numerous community resources designed to serve deaf individuals. In 2003, a local community agency convened a Deaf Health Task Force of both deaf and hearing individuals who represent various community organizations serving the deaf. The Task Force Report recommended the study of the health of deaf and hard-of-hearing populations. Building on the work of the task force, in 2004, the URMC was awarded a five-year grant by the CDC to create an innovative Prevention Research Center (URMC’s National Center for Deaf Health Research) to establish baseline measures of health in the deaf and hard-of-hearing community and to develop research and programs to improve their health status.

The National Center for Deaf Health Research has cultivated and sustained meaningful partnerships with individuals and organizations in the deaf and hard-of-hearing community in Rochester and nationally.[33,34] Its first product was the development and deployment of a culturally and linguistically appropriate health risk assessment tool for the deaf, using computer-based American Sign Language video communication technologies. Data from this tool inform health priorities of the deaf population that are factored into the overall community health assessment and considered when determining community-wide health priorities for the URMC or community organizations. The National Center for Deaf Health Research, currently in its second five-year funding period, is focusing on four major goals: to eliminate health disparities between deaf and hard-of-hearing and other populations; to unite the National Center for Deaf Health Research, its partners, and the deaf and hard-of-hearing communities through enduring partnerships; to establish a rich, generalizable evidence base regarding health risks, determinants of health, and effective health promotion interventions in deaf and hard-of-hearing populations and ensure its dissemination; and to establish the National Center for Deaf Health Research as a leading organization for deaf and hard-of-hearing health research.

Goal 3: National and global impact

URMC’s third community engagement goal is to enhance its national and global reputation through community-engaged research, development of community engagement educational platforms, dissemination of community health improvement models (such as Health Action), and potentially through input on policies or guidelines that affect the health of national populations. URMC’s vision is to increase generalizable knowledge and practices.

To address this goal, URMC has undertaken several major activities, which are detailed in Chart 2. For an example of a community engagement activity with national impact—the Greater Rochester Practice-based Research Network—see Box 3.[35]

The GR-PBRN is a network of 85 primary care pediatric, family physician, internal medicine, and mixed-specialty (e.g., community health centers) primary care practices that have collaborated on their mutual interest in promoting practice-based research.[35]

Structure. The GR-PBRN is funded by the Clinical Translational Science Institute at the URMC. It consists of 85 primary care practices—internal medicine, family medicine, pediatrics, and medicine–pediatrics. Together, these practices serve >200,000 adults (30% of Monroe County) and >150,000 children (80% of Monroe County). A multidisciplinary executive committee and a largely community-based steering committee direct the GR-PBRN, and a core group led by a full-time senior health project coordinator perform the daily operations.

Process. The GR-PBRN has participated in 2012 in more than 50 practice-based research studies, most involving health care delivery, quality improvement, or comparative effectiveness research. We assess the number of providers and patients, degree of external funding, and content areas investigated, as well as the internal processes such as the stages in research during which the GR-PBRN assists the investigations.

Outcomes. We measure the number of completed projects and manuscripts resulting from these practice-based studies. We qualitatively measure the impact on practices through interviews and quantitatively assess impact (every few years) via surveys of practitioners. It is of course challenging to measure precisely the added value of the GR-PBRN on the research or the impact of the research for these projects, but we attempt to qualitatively assess these metrics through interviews of key informants including practitioners, community, and national scientific leaders.

A URMC-community partnership’s efforts to reduce lead exposure in Rochester resulted in significant improvement in the community’s health and developed national models for implementation of such efforts in other communities. Lead-poisoned children can have substantial long-term morbidity that often leads to difficulty with learning.[36,37] Geographic areas of Rochester had rates of children with elevated blood lead levels as high as 10 times the national average.

In 2001, a group of community leaders including experts from URMC created the Coalition to Prevent Lead Poisoning in response to community concern and included representatives of community residents, housing, business, philanthropy, local government, environmental health care, and public health.

URMC faculty and staff, in leadership positions with the coalition since its inception, performed much of the underlying research regarding lead poisoning[38-41] and have been instrumental in developing and implementing outreach, education, and screening programs as well as public policy regarding exposure.

These efforts culminated in the city’s passage of a historic lead ordinance that went into effect in July 2006. The impact of lead abatement efforts has been significant. More than 2,300 Rochester homes have been made “lead safe,” and the number of children in the Rochester area with elevated blood lead levels was reduced from 2,000 children in 1998 to 290 children in 2010.[42-45] Importantly, this ordinance has had significant national impact, becoming the national standard for policy change to reduce childhood lead poisoning. Several other municipalities nationwide have followed Rochester’s lead on lead policy, and the U.S. Environmental Protection Agency awarded the coalition an Environmental Justice Achievement Award in 2009 for its leadership in community-based efforts to prevent lead poisoning.

In Sum

In this article, we have presented a framework for evaluating the community engagement programs of an AMC. This framework follows a health services evaluation framework and includes identification of AMC community engagement goals (and objectives), delineation of multifaceted community engagement activities, and a systematic evaluation program with assessment of the structure, process, and outcomes of the community engagement efforts. We used URMC’s three community engagement goals as a template: (1) to improve the health of the community served by the AMC; (2) to increase the AMCs capacity for community engagement and its value to the community; and (3) to increase generalizable knowledge, practices, and policies to improve individual and population health. AMCs can use and modify the framework to tailor specific goals and activities to their needs, and apply a structure-process-outcomes approach to evaluation. In this era of fiscal constraint, it is imperative that AMCs rigorously evaluate their community engagement efforts, enhance some that need improvement, and focus efforts on the most productive and meaningful activities.

We acknowledge several limitations. First, we present this framework as a useful method to evaluate an AMC’s community engagement activities, but we do not have data that demonstrate that use of the framework in its entirety has improved an AMC’s community engagement performance. Instead, we presented examples of how URMC has used parts of the framework to evaluate specific community engagement activities. Research is needed, within multiple AMCs, to evaluate the usefulness of the framework in its entirety. Second, as mentioned, because by definition community engagement involves close collaboration with other organizations, it is not possible to distinguish precisely the unique impact of an AMC’s role in joint community-based activities or to measure, with precision, the AMC’s return on investment. Third, although we suggest that the three community engagement goals are applicable to all AMCs, some AMCs may wish to develop other goals. We believe the structure-process-outcome framework will be helpful in evaluating any AMC goal in community engagement.

Systematically evaluating an AMC’s community engagement efforts can have both short- and long-term paybacks. Short-term, rigorous evaluation can serve to highlight successful programs, point out those that may need modification, and elevate the rigor of community engagement activities. Long-term, successful community engagement activities can truly improve a community’s health and enhance the value of an AMC locally and its reputation nationally. Community engagement experts should help lead the way in self-evaluation, and AMC leaders should use the evaluation process to improve performance and not to cut costs. We believe that fundamentally, most AMC leaders have a strong desire to improve their local communities, but because community engagement is a relatively new endeavor, we currently lack the tools to evaluate an AMC’s community engagement activities in a standardized, validated way. By applying our framework, AMCs may be able to focus efforts in an organized manner and intensify successful activities that can lead to improved community health.

Acknowledgments: The authors wish to thank Thomas A. Pearson, MD, MPH, PhD, for reviewing a draft of the manuscript. We thank the numerous community partners who make community engagement possible.