Measuring Community-Engaged Research Contexts, Processes, and Outcomes: A Mapping Review.

Policy Points Community-engaged research (CEnR) engenders meaningful academic-community partnerships to improve research quality and health outcomes. CEnR has increasingly been adopted by health care systems, funders, and communities looking for solutions to intractable problems. It has been difficult to systematically measure CEnR's impact, as most evaluations focus on project-specific outcomes. Similarly, partners have struggled with identifying appropriate measures to assess outcomes of interest. To make a case for CEnR's value, we must demonstrate the impacts of CEnR over time. We compiled recent measures and developed an interactive data visualization to facilitate more consistent measurement of CEnR's theoretical domains. CONTEXT: Community-engaged research (CEnR) aims to engender meaningful academic-community partnerships to increase research quality and impact, improve individual and community health, and build capacity for uptake of evidence-based practices. Given the urgency to solve society's pressing public health problems and increasing competition for funding, it is important to demonstrate CEnR's value. Most evaluations focus on project-specific outcomes, making it difficult to demonstrate CEnR's broader impact. Moreover, it is challenging for partnerships to identify assessments of interest beyond process measures. We conducted a mapping review to help partnerships find and select measures to evaluate CEnR projects and to characterize areas where further development of measures is needed.
METHODS: We searched electronic bibliographic databases using relevant search terms from 2009 to 2018 and scanned CEnR projects to identify unpublished measures. Through review and reduction, we found 69 measures of CEnR's context, process, or outcomes that are potentially generalizable beyond a specific health condition or population. We abstracted data from descriptions of each measure to catalog purpose, aim (context, process, or outcome), and specific domains being measured.
FINDINGS: We identified 28 measures of the conditions under which CEnR is conducted and factors to support effective academic-community collaboration (context); 43 measures evaluating constructs such as group dynamics and trust (process); and 43 measures of impacts such as benefits and challenges of CEnR participation and system and capacity changes (outcomes).
CONCLUSIONS: We found substantial variation in how academic-community partnerships conceptualize and define even similar domains. Achieving more consistency in how partnerships evaluate key constructs could reduce measurement confusion apparent in the literature. A hybrid approach whereby partnerships discuss common metrics and develop locally important measures can address CEnR's multiple goals. Our accessible data visualization serves as a convenient resource to support partnerships' evaluation goals and may help to build the evidence base for CEnR through the use of common measures across studies.

While the traditional approach to health research treats individuals and populations as the subjects of inquiry, community‐engaged research (CEnR) involves establishing and maintaining authentic partnerships between researchers and those who are being researched, including local community members and organizations. , , , , CEnR is an umbrella term used to describe a range of activities and approaches (eg, stakeholder engagement, patient engagement, public involvement, participatory action research), with community‐based participatory research (CBPR) being the longest standing and best known of these related to health. Yet, all approaches to CEnR borrow from CBPR's emphasis on community members as equal partners in many aspects of the research process, from the identification and selection of priority topics and research questions, to developing data collection materials and analytical strategies, to drafting and disseminating the publication of findings. , , , , , In addition, CEnR often aims to build the future capacity of academic‐community partnerships by improving community members’ research literacy and researchers’ concurrent understanding of the community's history and needs. , , ,

CEnR is theorized to impact research evidence and community health outcomes through a number of different mechanisms. , , , , , , , , , , For example, including community input into research design is likely to result in evidence and subsequent interventions that are more applicable to the community's needs and thus, more readily accepted by the community. , , , Community input can also improve the translation of research findings to practice by providing evidence that is tailored to the setting and population of interest. , , , Similarly, community members are likely to greatly improve the translation and dissemination of research evidence by championing the findings within the community and suggesting alternatives to publication. , , Finally, the process of CEnR can build trust and mutual respect and facilitate future research participation, especially in populations that have traditionally been mistreated by or excluded from health research. , , , ,

The potential impacts of CEnR on population health outcomes are less established in the literature. Yet, engaged research is theorized to empower community members to become educated about their own health and activated to participate in their health care. Engagement in research may also lead community members to make better health decisions based on the knowledge uncovered, which can improve long‐term health outcomes. Finally, participation in engaged research may contribute to greater community acceptance of research, subsequently guiding community action.

Much of the literature on participatory approaches has relied on case studies, qualitative inquiry, or literature review to suggest conceptual models and best practices for conducting engaged research. , , , , , , Although many proximal and distal outcomes are assumed to be affected by community engagement, there is, in fact, little evidence in the literature to demonstrate these relationships. Recent systematic reviews have shown that popular evaluation techniques are to (a) count the number of participants involved in engaged projects or events, or (b) elicit researcher and community member impressions of the impact of participation through qualitative approaches such as interviews or focus groups, , , , , , which, while valuable, limits the ability to systematically measure change and compare outcomes across studies. Some CEnR projects have developed their own limited‐item scales to assess the short‐term health outcomes or processes most relevant to the particular researcher‐community partnership. , The selection of locally relevant tools is essential to the practice of CEnR as it allows researchers and community members to share control of all stages of the research process, including which outcomes are most relevant to the partnership. Nevertheless, the reliance on local tools impedes the comparison of outcomes across studies or the validation of broader CEnR constructs, which could help to strengthen the impact of the field and increase recognition of CEnR as a scientific endeavor. To achieve CEnR's broader goals of improved health and research, there may be value in partnerships utilizing common or broadly generalizable measures of engagement in projects alongside locally tailored tools. We encourage partnerships to consider the multiple aims of CEnR when discussing evaluation practices to assess their projects.

CEnR approaches have been increasingly adopted by health care systems, funders, and communities looking for solutions to seemingly intractable problems. , , , Although researchers and community members who have partnered can attest to the benefits of CEnR, there is a growing emphasis in health research on funding evidence‐based interventions. Thus, it is more important than ever for engaged partnerships to demonstrate evidence of the positive impacts of CEnR. To facilitate consistent measurement of the impact of engaged research across projects over time and contribute to a clearer understanding of the value of community engagement, our aims were to:

Identify and categorize evaluation tools or measures of community engagement contexts, processes, and outcomes that can be used across a variety of CEnR projects in conjunction with locally tailored measures

Characterize current gaps in measurement

Serve as a resource for future CEnR projects

Methods

Search Strategy

To achieve our stated aims, we opted to conduct a mapping review. Mapping reviews categorize existing literature according to a particular theoretical model, participant population, or setting. This allows the reviewers to identify gaps in the literature, which can inspire further reviews or primary research. , In addition, mapping reviews often present results in a user‐friendly visual format to support future work. ,

To ensure a comprehensive search for measures of engagement, we adopted multiple strategies. First, we scanned the peer‐reviewed literature using two popular electronic bibliographic databases: Web of Science (including Medline/PubMed), and Academic Search Premier (including PsycINFO). The Web of Science Core Collection includes citations across the social science disciplines, allowing for a search across an index of 148 sociology, 140 psychology, 90 anthropology, and 82 health policy/services peer‐reviewed journals as examples. Academic Search Premier similarly indexes across more than 3,900 peer‐reviewed journals across disciplines. We limited our search to include only English‐language articles published between January 2009 and December 2018, as other reviews have targeted earlier time frames. , , , , Four separate searches were conducted; search terms were entered into the databases using quotation marks and included (1) community‐engaged research AND evaluation, (2) stakeholder‐engaged research AND evaluation, (3) community‐based participatory research AND evaluation, and (4) participatory action research AND evaluation for a total of 2,708 abstracts to review (1,399 via Web of Science and 1,309 via Academic Search Premier).

The lead author (T.L.) reviewed each title and abstract to eliminate duplicate records, study protocols, and those that diverged from community member, stakeholder, patient, or public engagement in research (n = 1,666). At this stage, the following inclusion criteria were also applied: (a) the study must describe an assessment or evaluation (qualitative or quantitative) of an engaged project, thus excluding articles that only described conceptual models or practical steps for conducting engaged research (n = 482), and (b) the assessment or evaluation must measure the context, process, or outcome of engaged research. These evaluation targets were selected based on the theoretical and practical premise that understanding the context and process of engaged research is critical to interpreting the ultimate impact. , In addition, we aimed to uncover a body of “universal” measures that could be used by partnerships consistently across studies in addition to locally tailored tools. Thus, the lead author excluded articles where the only results reported were study‐specific and would not readily generalize to other engaged research settings. These included varied outcomes such as blood glucose level in a diabetes population, environmental health literacy in a sample of school children, or level of trauma awareness in a clinician group (to name a few) (n = 353). Although this decision likely excludes a number of specific health‐related items that could be useful for some engaged partnerships, it was not feasible to include and analyze the wide breadth of study‐specific outcomes in this particular review.

As a secondary approach, we conducted a scan of “community‐engaged research organizations” (via Google search and discussions with colleagues engaged in CEnR) to identify academic centers and working groups actively pursuing engaged research; we searched these organizations’ websites for additional evaluation resources and tools not captured by the bibliographic databases. This yielded a total of 25 additional white papers and instruments. Thus, the full text of 126 articles and white papers were obtained and reviewed to glean specific qualitative and quantitative assessments. After eliminating all papers that did not describe the measurement scales or items used (n = 57), this left a total of 69 final measures for data abstraction.

Data Abstraction, Validation, and Visualization

The lead author abstracted data from the individual study authors’ own descriptions of (a) the purpose of the measure, and (b) the domains measured. Additionally, the lead author classified the measures according to whether the study authors’ stated purpose or goal of the measure most aligned with a context evaluation (examining the conditions in which CEnR will be conducted), process evaluation (examining how the engagement is done), or outcome/impact evaluation (examining the intended effects). , Measures were also allowed to represent multiple goals—for example, a measure of how partnership conflict was resolved could be used during engagement to measure the process and postengagement to determine if a change has occurred—if indicated in the study author's stated purpose and the item descriptions of each subscale or subsection. Our aim in allowing measures to serve in multiple evaluation categories is for academic‐community partners to consider these measures according to what would be useful for their specific contexts.

If an article did not include a discussion of the measure's domains, the measure's items were summarized with key phrases to provide a cross‐item description. To validate the data abstraction, the other two authors (A.H. and G.T.) each independently coded half of the measures as described, so that all 69 measures were reviewed by two authors. All three authors met biweekly to arbitrate coding conflicts through consensus. The authors also maintained individual process memos to highlight patterns and themes within the data. Selected measures from each evaluation type (context, process, and outcome) can be found in Appendix Table 1. The full 69 measures are represented in Online Supplemental Table 1.

To serve as a resource for future CEnR projects, we utilized Tableau Public to produce interactive data visualizations of the discovered measures and the domains represented. Through this visual display, we hope to support researchers and community members to identify measures of engagement to use in their own CEnR projects, according to their own goals. We encourage readers to explore this resource at https://public.tableau.com/profile/tana.luger#!/vizhome/MeasuresofCommunityEngagement/AuthorsandDomains.

Results

Across the literature and measures reviewed, we found substantial variation in how researchers conceptualized and defined domains to measure. For example, the academic‐community partnership central to CEnR has been described in the literature with such disparate phrases as “reciprocal relationships,” “collaborative partnerships,” “mutually effective partnerships,” “mutual respect and acceptance of differences,” and “shared power and decision‐making,” , , among others. In addition, nearly half (47.8%, n = 33) of the included measures could be applicable to more than one domain (ie, context, process, or outcome evaluations).

We found the boundaries between core CEnR concepts were similarly unclear. For example, although many studies attempted to evaluate the role that community members had adopted within the engaged research, the scope of the evaluation varied significantly. Some studies asked community members to quantify the specific tasks in which they engaged. , Some focused on the community members’ roles in the research and communication processes that promote role clarity. , , , Still other studies focused on the community's perception of the impact that their specific activities had on the research project. , ,

In order to place our results within the context of the larger literature and characterize gaps, we present a brief table of the systematic reviews (n = 12) that were uncovered utilizing our specific search criteria (see Appendix Table 2). As our aim was not to conduct a review of reviews, but to discover measures of engagement, it is important to note that these 12 are not an exhaustive representation of the previously reviewed literature. Nevertheless, the authors of the included systematic reviews frequently call for the need to develop reliable and valid measures in order to better understand contextual factors, partnerships, and the impacts of engagement. Many of the systematic reviews also acknowledge the value of quantitative and qualitative measurement for better analyzing engagement processes.

Context Measures

We identified 28 measures that assess the context of engaged research (see Figure 1); these measures predominantly focus on the conditions under which the research will be conducted and considerations for effective collaborations between researchers and community members. The majority of context measures evaluate the community's capacity for engaged research (see Figure 2), including training and past CEnR experiences. For example, a quantitative needs assessment by Goytia and colleagues. inquires about the experiences of community organizations in research and evaluation, their interest in building specific research skills like survey development or literature review, and interest in partnering in the future. Similarly, the Canadian Foundation for Healthcare Improvement's tool encourages community organizations to self‐evaluate their capacity to utilize research evidence, a key skill for active participation in an engaged project. In addition, a survey by Rubin and colleagues asks community members to assess their level of confidence in working with researchers as well as motivations and attitudes toward research. Many context measures also assess the strengths and resources of the community that may be brought to bear in an engaged research project such as social capital and organizational linkages. A qualitative interview guide by Gibbons and colleagues encourages community members to reflect on the strengths within the community that may be drawn on for an academic‐community partnership. This includes a variety of community resources such as social groups, occupations, and sources of information. In addition, the Partnership River of Life participatory exercise published by Sanchez‐Youngman and Wallerstein encourages partners to develop a “communal narrative” about the origins and key events of their partnership in order to better understand the larger historical, social, political, and economic context in which the partnership is aiming to function.

Figure 1

Engagement Measures by Evaluation Type

For interactive data visualization, please visit https://public.tableau.com/profile/tana.luger#!/vizhome/MeasuresofCommunityEngagement/AuthorsandDomains.

Figure 2

Number of Domains Represented in Engagement Measures by Evaluation Type

For interactive data visualization, please visit https://public.tableau.com/profile/tana.luger#!/vizhome/MeasuresofCommunityEngagement/AuthorsandDomains.

Several measures evaluate the academic researcher's capacity for engaged research, such as researchers’ sufficient knowledge and understanding of the community or expectations and goals for engagement. The Engage for Equity Key Informant survey asks partners to rate whether the principal investigator of an engaged project is from a similar cultural background as the targets of the project and whether the project has provided any training or discussions about oppression and cultural sensitivity. Another survey, this one by DiGirolamo, asks researchers to assess their current skills and interests related to conducting CBPR (such as coalition‐building or how to obtain funding) for the purpose of uncovering needed academic infrastructure to support CBPR projects.

Finally, a few context measures encourage reflection on the specific logistics of an engaged project before it begins. For example, the Patient Engagement Workbook takes researchers through a number of important considerations for engaged research with patients, including the patient's role across the stages of research, methods to identify and recruit patient partners, institutional requirements, how patient concerns will be addressed, and processes for sharing data. As another example, the Community Priority Index allows researchers and community members to rate the importance and ease of change of community issues in order to quantify the partnership's priorities for engaged research projects.

Process Measures

We identified 43 measures that evaluate the process of engaged research—ie, aspects of how community engagement occurred (see Figure 1). Manyprocess measures examine group dynamics within the partnership, , , , , , , , , , typically adapted from a 2003 instrument by Schulz, Israel, and Lantz. For example, interview questions by Paige and colleagues elicit community members’ general experiences in the partnership to characterize the academic‐community collaboration.

Following the overarching goals of CEnR, many measures inquire about perceptions of influence and power, respect, trust, and communication across group interactions to ensure an equitable collaboration. , , , , , For example, the Youth‐led Participatory Action Research Process Template allows team members to rate the power‐sharing observed between teachers and students during an engaged project. The Partnership Assessment in Community‐based Research (PAIR) survey encourages stakeholders to assess the fundamental characteristics of a strong partnership, including open and honest communication, equitable collaboration, shared partnership values such as mutual respect and trust, and a plan for sustaining the partnership. In addition, quantitative measures by Braun and colleagues and Goodman and colleagues allow stakeholders to reflect whether the processes utilized within their partnership align with the goals of community‐based participatory research (CBPR). , The partnership is rated on the basis of how well or how often it “fosters capacity‐building for all partners” and “seeks and uses the input of community partners,” among others. These measures allow stakeholders to determine whether the engaged project is being conducted in the spirit of equitable, collaborative CEnR.

Another body of measures examine partnership synergy, or enhanced collaboration among partner members. For example, the CBPR Rating Scale by Pivik and Goelman asks partners to rate the importance of shared decision making, goals, and values to their engaged work. Similarly, the Engage for Equity Community Engaged survey encourages partners to assess the level at which they develop shared goals and strategies and respond to challenges and community needs.

Another process survey measures the practicalities of the engaged project, such as satisfaction with the organization and structure of project meetings. Other measures examine the leadership in place for the engaged research, assessing the effectiveness of leadership and outlining governance decisions. , , The Engage for Equity Community Engaged survey also asks respondents about project governance, such as who approved participation in the project on behalf of the community and who controlled project resources. Such measures would be useful for obtaining feedback to inform effective partnership strategies for future engaged research.

Outcome and Impact Measures

We identified 43 measures that evaluate the outcome or impact of engaged research or the intended effects of community engagement (see Figure 1). Although a number of measures ask open‐ended questions about the benefits and challenges of participating in engaged research, , , others more formally assess the perceived benefits of engaged research through survey methods in order to elicit reported impacts and costs within each academic‐community partnership. , , , Often, this involves asking stakeholders directly about their satisfaction with the partnership. ,

The majority of included outcome measures evaluate the systems and capacity changes produced by the engaged research, such as increased information and resource exchange among stakeholders, , , joint activities or events, ongoing or new funding for partnered work, and improvements to services and programs. Others focus especially on measuring improved community capacity for research, such as the knowledge and skills for future research engagement , , , as well as self‐efficacy and confidence in research participation. , , , ,

Similarly, many measures appraise the sustainability of the partnership by gauging stakeholder commitment to the work, , , , ongoing funding, , and enhanced research networks. Still others quantify the number of publications, policy revisions, and solicited funding opportunities to judge the effectiveness of the partnership's efforts. , , Yet, few measures in our sample directly assessed how engagement improved research practice or outcomes. The CBPR Model Visioning Guide encourages stakeholders to consider research productivity (ie, papers, grant applications, and awards) as an intermediate outcome, if applicable to their CBPR partnership.

Some measures examined more distal outcomes of the project such as policy changes , , , or community empowerment. For example, the Knowledge Ownership Social Network Analysis asks partners to rank stakeholders according to whether they held important information relating to the project. Meant to be conducted longitudinally, the measure allows partners to track whether knowledge is elicited and owned equitably among stakeholders, reflecting a more active, empowered role of the community. Similarly, the Ripple Effect Tool quantifies the social connectedness or enhanced relationships and resource sharing within an engaged project. Finally, the Engage for Equity Community Engaged survey asks about health outcomes to evaluate longer‐term changes to community health status or health outcomes of interest that may arise from the engaged project.

Discussion

Through a mapping review technique and multipronged search, we identified 28 measures appropriate for a context evaluation, 43 process measures, and 43 outcome or impact measures of CEnR that have been developed over the past decade. The majority of the measures assessed community capacity for engaged research, either as a context evaluation to understand the community's skills and resources for engaged work or as an outcome or goal of an engaged project. This is in contrast to two recent systematic reviews, , which found a focus on process measurement.

Our work also supports the assertion of many authors that there is a lack of consensus on the goals of CEnR (see Appendix Table 2); we found substantial variation in how study authors conceptualized and defined even similar domains to measure. Although all measurement is important to understanding the impacts of engaged research, achieving more consistency in the ways that partnerships evaluate key constructs could reduce some of the measurement confusion apparent in the literature. Although many authors have valiantly attempted to systematically organize this complex body of literature, it is still difficult for partnerships to identify measures to use in their engaged projects. In the context of CEnR, it is important to highlight the inherent tension between a researcher‐driven approach to selecting established and validated metrics and a community‐driven process of developing and/or adopting metrics that may be more trusted, relevant, and useful in the local setting. Based on our experiences conducting engaged research, discussions with colleagues and community members, and CEnR's overarching goal of an equitable partnership where no member is more or less important than another, , , it seems that a hybrid approach in which researchers and community members propose and discuss both measures of the broader domains of engaged research and the specific needs and goals of the community allows for both consistency and local tailoring. In addition, a hybrid approach that encourages partnerships to identify common metrics and develop locally important measures may represent a process whereby the desired outcomes of CEnR such as synergy, shared power, and shared ownership can flourish. In that spirit, we hope that the included data visualization will serve as a convenient resource to help academic‐community partnerships identify common measures to consistently include in their projects. Similarly, although many research groups and organizations have attempted to create models to guide the practice of CEnR, , , , more collaborative work needs to be done to promote a consistent framework for engaged research. A clear, guiding model built and tested across settings and data sources, such as the one proposed recently by Oetzel and colleagues, can encourage more systematic measurement of important domains.

In contrast to systematic reviews by Sandoval and colleagues and Esmail, Moore, and Rein, we discovered many context measures, which could indicate a growing interest in understanding the conditions in which CEnR will take place. As described earlier, these measures predominantly assessed the community's capacity for engaging in research, such as skills in literature review or experiences with academic researchers. This is a strong step forward in improving overall measurement of engaged research. As Sandoval and colleagues purport, “differences in context significantly influence processes, which form the core operational partnership features of CBPR. These differences … matter in terms of partnership success.” Thus, understanding the context in which engagement takes place is a critical step in understanding the academic‐community partnership and measuring the impact of engaged research. We recommend that partnerships include measures throughout the life cycle of the engaged project so that the context, processes used, and impacts are all assessed. Ultimately, the strength of the partnership and ability to identify project impacts may be facilitated by carefully made decisions at the project's start to include systematic measurement. Recent toolkits and planning exercises , , , can facilitate equitable participation among stakeholders in this critical stage.

Nevertheless, many of the measures that we uncovered can be used at multiple points in the engaged research; for example, a process measure administered at the start and end of the project may simultaneously represent an evaluation of the engagement process as well as an outcome or impact of the project (eg, change in the partnership or process). Although our goal is to point partnerships toward valid and reliable common measures that can be used across settings, we do not suggest that partnerships rely solely on these measures for evaluation purposes. It is critical for academic‐community partners to have the flexibility to adapt measures to best fit their needs and local settings. Many of the qualitative guides or workbooks that we include in this paper are designed to encourage partnerships to self‐reflect on the type of measurement that best fits their needs. , , , ,

Encouragingly, we identified a growing number of measures of the outcomes or impacts of engaged research, in contrast to the findings of previous systematic reviews. , , , , In our analysis, investigators chose to measure such outcomes as change in community capacity (eg, increased information exchange or resource sharing), perceptions of whether the project aligned with CBPR principles, the number of activities or goals that the partnership achieved, the perceived benefits of the project, and satisfaction with the project. Although these outcomes can be easily documented from project activities or elicited from participants, there is much more to be done to determine the impact of engaged research. Recent efforts in this direction provide valuable practical guidance for addressing the heterogeneity of impacts in the literature and the challenges of measuring across local settings and partnerships. Similar to multiple reviews, , , , we found a lack of attention to long‐term outcomes such as improved uptake of research or an effect on health outcomes. We recommend that partnerships include both measures of the immediate effect of the research process or project (eg, increased group cohesion) as well as consider follow‐up measurement of more distal outcomes over time.

As previously mentioned, many systematic reviews call for the need to develop psychometrically valid and reliable quantitative measures in order to further understanding of the mechanisms at work (see Appendix Table 2). We agree that it is critical for the field to move toward validation of existing quantitative measures as well as the continued production of new instruments in order to assure that engaged research models and measures are evidence‐based. Nevertheless, we affirm that there is value in including qualitative measurement in engaged research projects, following our choice to include interview and focus group guides in our mapping review. Hearing directly from stakeholders in their own words can lend insight that can lead to better conceptualization and operationalization of engagement. In turn, this can lead to more valid and reliable quantitative measures. Additionally, the nonspecific effects of engagement that are theorized to be influential, such as cohesive relationships or feelings of being valued, may not be fully captured by a 7‐point Likert scale. As Conklin and colleagues mention, “the emphasis placed on assessing outcomes/impact of public involvement risks missing the normative value of public involvement as intrinsically good because it is a deliberative democratic process.” There is likely inherent value in engaging community members in research beyond the targets of any intervention. Integrated qualitative and quantitative measurement may best capture these complex effects.

There are several limitations to our work. Although we relied on multiple strategies to ensure as comprehensive a search for measures as possible, we did not conduct a systematic review, due to the wide scope and ambiguity of the literature. As a result, we may have unintentionally missed a measure for inclusion. In addition, we reviewed only those measures whose individual items were described in the peer‐reviewed or online literature. It is possible that organizations have developed useful measures but have not identified an effective way to share them with other engaged groups. If readers have developed or identified measures for engaged work to recommend, we encourage them to contact the lead author for inclusion in the interactive data visualization.

Conclusion

Researchers and community members who have partnered on research together can attest to the perceived benefits of CEnR; many practical lessons learned and conceptual models can be found in the literature. Simultaneously, CEnR approaches and methods have been taken up by health care systems, funders, and communities looking for solutions to intractable problems, within the context of funding evidence‐based interventions. This means it is more important than ever for those conducting CEnR to be able to demonstrate impacts of CEnR over time, and for the field as a whole to make a case for the value of CEnR. Nevertheless, models and concepts of engaged research still remain muddy. We have compiled a body of recent measures and developed an interactive data visualization to facilitate more consistent measurement of the theoretical domains of CEnR. Along with previous systematic reviews, we hope that our work will support academic‐community partnerships to identify key domains to be measured within their projects and instruments with which to do so.

Supplemental Table 1. Measures of Community Engagement (Organized by Context, Process, or Outcome Evaluation based upon Study Authors’ Descriptions)

Click here for additional data file.

Appendix Table 1

Selected Measures of Community Engagement (Organized by Context, Process, or Outcome Evaluation Based on Study Authors’ Descriptions)

Instrument	Domains Measured	Definitions	Purpose of Measure	Number of Items	Validity/Reliability
Selected Context Measures
Community‐Based Participatory Research (CBPR) Skills and Training Needs 61	CBPR skill set CBPR training needs CBPR mentorship	Research, relationship building, community training [CBPR skill set] Interest in gaining additional skills/training related to CBPR [CBPR training needs] Willingness to mentor; interest in being mentored [CBPR mentorship]	Survey of research investigators’ skills, interest, and training needs in CBPR	20 closed‐ and open‐ended items	Piloted with academic medicine and public health faculty and then revised
Community Needs Assessment 55	Primary target population that organization serves Health issues that current programming addresses Research and evaluation experience Interest in partnering and training	History and experience with research collaboration [Research and evaluation experience] Interest in learning about specific research topics; preference for learning format and structure [Interest in partnering and training]	Assessment of local community‐based organizations’ research needs	23 closed‐ and open‐ended items	–
Group Level Assessment 97	Climate setting Generating Appreciating Reflecting Understanding Selecting Action	Building trust and encouraging participants to openly share what they know; icebreakers [Climate setting] Responding to project‐generated open‐ended prompts: “Our toughest problem to address is …” [Generating] Time to view the responses (collective data), discuss observations, and add additional comments [Appreciating] Initial observations/thoughts [Reflecting] Smaller group discussions of the responses; looking for common themes and analyzing the data [Understanding] Larger group discussion of themes; prioritization of themes by some voting/ranking process [Selecting] Smaller groups consider possible next steps/action plans for each prioritized theme [Action]	Formative evaluation method to assess needs and design a plan for future programs	7 participatory stages/activities	Implementation across more than 14 participatory evaluation settings
Community Priority Index 62	Importance of issue Changeability of issue	Ratings of importance and ease of change of issues identified by partnership/related to topic of interest	Method to quantify priorities (for engaged project) across stakeholders	Varies; issues derived a priori or through community focus groups	–
Cultural Identity Inventory 98	Cultural dimensions Manifestations Interactions Subordination Vantage point Connection to community practice	How does the participant identify across various cultural dimensions (eg, gender, race, sexual orientation, ability) [Cultural dimensions] Values, actions, or messages associated with the dimension [Manifestations] Intersectionalities; interactions with other dimensions [Interactions] Privileges afforded to member of dominant group; denials experienced by member of subordinate group [Subordination] Understanding of cultural aspect of oneself; how others see/view you [Vantage point] Strengths or challenges related to cultural identity and community practice [Connection to community practice]	Critical self‐reflection for community practitioners	8 domains for self‐reflection	–
Potential Partner Interview Guide 66	Community background Interest in partnering Capacity/fit Resources Key community members/organizations Referrals/point of contacts	Previous work in community; changes wanted/needed for target population [Community background] Anticipated time for coalition activities [Capacity/fit] Identification of community leaders, organizations that provide services to target population [Key community members/organizations]	Assessment of community organization and their interest in partnering	6 open‐ended questions	–
Community Health Initiative Interview Guide 58	Physical community assets Voluntary groups (associations) Paid groups (institutions) Economic assets Individual assets Community stories Communication	Neighborhood characteristics and trends; social connections; quality of life, safety, noise, city services [Physical community assets] Transportation; neighborhood/community associations; conflict management [Voluntary groups (associations)] Schools; locations to meet people and hang out; food locations; access to health care [Paid groups (institutions)] Sources of income, occupation, education, employment history, home ownership [Economic assets] Other personal skills; contact with political representatives; dependability of members of neighborhood [Individual assets] Stories about people/places around here; important events in location [Community stories] News sources; how news travels; most reliable source; first to know; health information [Communication]	Community assets assessment when initiating an academic‐community partnership	107 open‐ended questions	Interview guide refined and revised through community partner input and interviewing role‐play
Selected Context and Process Measures
Partnerships in Social Determinants of Health Interview Guide 45	Process for targeting social determinant of health (SDH) Community context Facilitators Barriers Shared power and influence	How did partners decide the specific SDH was meaningful to the partnering community? [Community context] Was the specific SDH of great importance and concern to the partnering community before working on this project?	Evaluation of academic‐community partnership processes Targeting SDH	8 open‐ended questions	–
Partnership Assessment in Community‐based Research (PAIR) 43	Communication Collaboration Partnership values Benefits Commitment to ongoing evaluation/ continuous improvement Sustainability	Open and honest dialogue between partners; resolution of conflicts through discussion [Communication] Equal partnership; shared leadership and resources; shared goals and decision making [Collaboration] Mutual trust and respect; valuing other perspectives; appreciation of partner; understanding culture of partner organization and community [Partnership values] Plans and resources for a sustainable partnership, including commitment from partners [Sustainability]	Evaluation of critical elements of academic‐community partnership	31 Likert‐type items and one open‐ended question	Community input sought at each step of measure development (generation of dimensions and items, item sorting and feedback, cognitive interviews and measure piloting)
Patient Engagement Workbook 49	Reasons for engagement/justification Cost of engagement Roles of partners across stages of research Logistics of engagement Identification/recruitment of patient partners, including compensation Institutional requirements for patient engagement Guidelines for screening participants Research team preparation for engagement Patient partner training for engagement Patient partner retention and feedback Data‐sharing processes Closure activities for project end	Goals; expectations and rules; training [Research team preparation for engagement] Tips for soliciting feedback and addressing partner concerns [Patient partner feedback]	Reflection guide for researchers considering engaging patients and documenting patient efforts	102 open‐ended questions	–
Selected Measures of Context, Process, and Outcome
Research for Improved Health (RIH) Partnership Interview Guide 85	Individual/project background Context for all communities Intervention research questions Policy research questions Partnership/group dynamics Individual‐level issues Partnership outcomes Research design	Origins of partnership [Background] Personal motivations; relevant knowledge of community; approval processes [Context] Role of community knowledge and experience; professional articles; evidence and best practices; local programs/agencies in project [Intervention research] Process of prioritizing policy changes; action steps; role of data and evidence; advocacy and personal testimony [Policy research] Successes in the partnership? Challenges? Power and trust? [Dynamics] Personal qualities useful for involvement in partnership; researchers share background with community? [Individual‐level issues] Successes of partnership; community member benefits [Partnership outcomes] Description of research approach; CBPR's influence on project? [Research design]	To describe the challenges and successes of the participatory process and outcomes that arose from the partnership	39 open‐ended questions	Developed and refined with relevant stakeholders
Rochester Suicide Prevention Training Institutes Evaluation 51	Personal knowledge and capabilities Partnership agency Partnership benefits	CBPR knowledge and research skills [Personal knowledge and capabilities] Quality of partnership interactions; community‐focused; clear roles and policies; community capacity for research [Partnership agency] Knowledge; improved research; improved partnership [Partnership benefits]	Follow‐up survey to assess knowledge and skills for engagement, partnership processes, and benefits or outcomes	58 closed‐ended items (Likert, yes/no, checklist)	Cronbach's α Personal knowledge = 0.55‐0.87 Partnership agency = 0.66‐0.93 Partnership benefits = 0.84‐0.92
Program for the Elimination of Cancer Disparities (PECaD) Collaboration Survey 52	Environmental characteristics of partnership Structural characteristics of partnership Group dynamics characteristics Intermediate partnership effectiveness Partnership effectiveness	Previous collaborations; diversity; organizational context [Environmental] Role within partnership [Structural] Communication; goals and vision; participatory decision making; shared power, influence, resources; mutual trust; collaborative evaluation and meetings [Group dynamics] Effectiveness in achieving goals; benefits of participation; satisfaction with role and influence; shared ownership; group empowerment [Intermediate] Collective impact; sustainability; enhanced networks/capacity [Effectiveness]	Assessing capacity, group dynamics, and effectiveness of achieving principles of CBPR in partnership	45‐60 closed‐ and open‐ended items (adaptation by two different community‐engaged project groups results in a varying number of items)	–
CBPR Model Visioning Guide 82	CBPR model for planning and visioning ○ Priority ○ Desired outcomes ○ Context for planning ○ Partnership processes ○ Interventions/research design	Priority issues [Planning priority] Desired intermediate system and capacity outcomes; long‐term community, health, and health equity outcomes [Desired outcomes] Review of contextual factors like socioeconomic, cultural, historical collaboration and trust, capacity [Context for planning] Review of partnership dynamics like individual characteristics, dialogue, conflict management, leadership, and partnership structures [Partnership processes] Integration of cultural or community knowledge for interventions matching community values, norms, practices; empowering processes; involvement of community members in all stages and impact on research design [Interventions]	Guidance for adapting a CBPR model to fit context, planning a new research project, evaluating partnership practices, and assessing the impact of practices	19‐page workbook/ facilitation guide	–
CBPR Model Visioning Guide 82 (cont.)	CBPR model for evaluation and reflection ○ Context evaluation ○ Partnership processes evaluation ○ Intervention/program and research evaluation ○ Outcomes evaluation	2‐3 important contextual issues to evaluate or consider; effects on ability to work together [Context evaluation] 2‐3 important partnership issues to evaluate or consider; impact on collective work [Partnership evaluation] 2‐3 important intervention/research issues to evaluate or consider; importance of integrating cultural/local knowledge; ability to work together/construct joint understanding of research; importance of including community in all steps of research [Intervention evaluation] 2‐3 intermediate and long‐term outcomes important to evaluate or consider [Outcomes evaluation]			–
Engage for Equity (E2) Key Informant Survey 99	Project features Populations and communities involved in project Community challenges Reflective practices Training topics Hiring and resource sharing Research integrity and governance practices Advisory boards Formal agreements Project outcomes	Initiator of study; community partners; type of partnership; description of meetings; number of members [Features] Focus of project; demographics of academic and community team members [Populations] Social, economic, or structural issues impacting the health of the community [Challenges] Extent that partnership engages in self‐evaluation or quality improvement [Reflective] Formal training/discussions about cultural sensitivity, CBPR, reflection [Training] Hiring; decisions around resources; distribution of funding [Hiring and resources] Human subjects training; confidentiality agreements; approvals on behalf of community [Research integrity] Advisory boards and their role [Advisory boards] Memorandums of understanding; verbal agreements; data ownership and sharing [Agreements] Publications; additional research/funding; evaluation instruments; revisions to Institutional Review Board (IRB) policies/practices [Project outcomes]	To describe engaged project structural features and processes	90 closed‐ and open‐ended items	Developed and refined through input with relevant stakeholders
Engage for Equity (E2) Community Engaged Research Survey 60	Community context and capacity Partnership capacity Bridging differences Missions and strategies Community involvement in research Influence in the partnership Quality of dialogue Reflexivity Leadership Resource use	Resources; experience with partnered projects [Community context and capacity] Partnership knowledge, skills, diverse members, legitimacy, and credibility in community [Partnership capacity] Skills to bridge differences; similar backgrounds among community and academic members [Bridging differences] Shared agreement for missions/strategies; clear understanding of goals [Missions and strategies] Level of involvement of community members in different stages of research process [Community involvement in research] Amount of voice and influence over decisions; commitment to decisions [Influence] Attitudes; listening; consensus; conflict/hostility among members [Quality of dialogue] Partnership discussions on activities; power; privilege; and improvements to collaboration [Reflexivity] Effectiveness of leadership in achieving active participation, respect, creativity [Leadership] Project's use of resources [Resource use]	Assessment of context, partnership processes, and research processes of engaged project Evaluation of intermediate and long‐term outcomes of engaged research	126 Likert‐type, yes/no, and open‐ended items	Refined through discussion with relevant stakeholders and psychometric testing 91
Engage for Equity (E2) Community Engaged Research Survey 60 (cont.)	Trust Community engagement principles Partnership synergy Agency outcomes/benefits Personal advantages Personal challenges Power relations in research Project sustainability Health outcomes Current community‐level, research, and policy outcomes Future community‐level, research, and policy outcomes Quality and satisfaction Time use	Trust in people and decisions of partnership; type of trust [Trust] Builds on resources and strengths in the community; facilitates equitable partnerships in all phases of the research [Community engagement principles] Goal setting and problem solving among partners [Partnership synergy] Enhanced reputation of community organization; use of expertise or services by others [Agency outcomes] Enhanced reputation; ability to affect policy; ability to seek education [Personal advantages] Challenges participating in partnership [Personal challenges] Change in power relations in research process and capacity [Power relations] Likelihood of project sustainability [Project sustainability] Improvement to community's health [Health outcomes] Changes to policy, practice, cultural identity/pride, social impacts, better research [Current outcomes] Desired changes to outcomes [Future outcomes] Quality and satisfaction with partnership [Quality and satisfaction] Time dedicated to project per week, both covered and not covered by salary [Time use]
Clinical and Translational Science Award (CTSA) Engagement Survey 76	Definition of community engagement Institutional partners Institutional transformation Measures being tracked Community advisory boards (CAB)	Best description of CTSA's engagement with community [Definition] Types of community stakeholders [Institutional partners] Number and intensity of local collaborations; institutional resources [Transformation] Metrics that CTSA's track, including community members with research training, projects that seek input from community members, publications and grant proposals [Measures] Presence of CAB associated with CTSA and larger institution; serving community members [Community]	Survey of academic perspectives on community engagement activities within clinical and translational science institutions	12 yes/no or forced‐choice items	–
Partnership River of Life 59	Start of partnership Important stages Tributaries Facilitators Obstacles Future vision Overall reflection	The group selects “where it's important to start” (eg, historical events before partnership; funding start) [Start of partnership Important or influential stages in the partnership's timeline [Important stages] Key “tributaries” (resources, mentors, members) that enter or leave the partnership [Tributaries] Factors that facilitated the partnership's work [Facilitators] Obstacles against the partnership's work [Obstacles] Where the partnership is headed [Future vision] Experience of collective process; general thoughts and feelings; facilitators and obstacles identified; important external factors [Overall reflection]	To facilitate partnership reflection on the history and influences of the members and the goals, processes, and results of partnership	Group exercise and 5 open‐ended questions for reflection	–
Selected Process Measures
Research for Improved Health (RIH) Focus Group Interview Guide 100	Current partnership activities Cultural issues CBPR approach Partnership/group dynamics	Change in partnership goals, resources [Current partnership] Impacts of cultural differences between academic researchers and community, influence of community knowledge [Cultural issues] Benefits and challenges of a CBPR approach; impact of CBPR approach on the project and community trust [CBPR approach] Power, conflict resolution, trust, future of partnership [Partnership/group dynamics]	To understand participant experiences within an academic‐community partnership	9 open‐ended questions	–
Partnership Trust Tool 69	Trust	Accessibility/approachability Dependability Good/clear communication Mutual benefit Openness Providing accurate information Relationship building Responsibility Shared power/decison making Supportive Truthful Value differences	Facilitate discussion about and enhance trust within the partnership	58 items: 30 Likert‐type items and 28 open‐ended written questions	–
Building Your Capacity Evaluation Survey 101	Community partner capacity for research after training (competencies)	Competencies such as conducting a literature review, recruiting participants into a study, designing a survey, or writing a research report	Assessment of community partner's research competencies post training	14 Likert‐type items	–
Community Engagement in Research Index 46	Perception of community partner's engagement in research (tasks)	Community as consultants or active partners across various research tasks	Assessment of community partner's level of participation in various research tasks during engaged project	12 closed‐ended items	Developed through qualitative interviews with community‐engaged project primary investigators
Youth‐Led Participatory Action Research (YPAR) Process Template 70	Training and practice of research skills Promoting strategic thinking Group work Opportunities for networking Communication skills Power sharing over major decisions Power sharing over class structure	Identifying research questions; developing tools; data collection [Training and practice] Considering how to influence rules and policy; alternative points of view [Promoting strategic thinking] Productive group processes; roles to further group goals [Group work] Student contact; teacher contact on behalf of students [Opportunities for networking] Practicing formal presentations; sharing ideas and perspectives aloud [Communication skills] How power is shared as decisions are made [Power over decisions] How power is shared in everyday class climate [Power over structure]	Classroom observational measure to assess the quality of YPAR implementation	25 qualitative codes	Intraclass correlation coefficients (ICCs) for interrater reliability: Training = 0.88 Promoting = 0.73 Group work = 0.97 Opportunities = 0.73 Communication = 0.76 Power over decisions = 0.66 Power over structure = 0.72
Selected Process and Outcome Measures
Participatory Evaluation Measure 102	Extent of involvement in evaluation Diversity of participants Control of the evaluation process	Extent of stakeholder involvement in the technical tasks of evaluation, such as questions and issues definition, data collection and analysis, interpretation and dissemination/reporting [Extent of involvement] Number of types of stakeholders involved, such as policymakers, frontline staff, target populations, or citizens [Diversity of participants] Authority to make decisions and other resources to bring to bear such as money, values, social skills	To assess participatory evaluation practices	3 coding schemes and weights	Face validity and apparent content validity, according to the authors
Participatory Impact Pathways Analysis 103	“Problem Tree” Network perspective Outcomes logic model Monitoring and evaluation	Linking problems that the project directly addresses with the conditions that it wishes to improve [“Problem Tree”] Envisioning what various stakeholders will do differently after the project [Network perspective] Changes to stakeholders’ specific knowledge, attitude, skills, and practice needed to achieve the project's vision [Outcomes logic model] Outcome targets (expected benefits) for the project and specific milestones toward achieving them [Monitoring and evaluation]	To allow researchers and stakeholders to jointly describe a project's “theories of action,” develop logic models, and use them for project planning and evaluation	4 stages of impact planning	–
Interorganizational Network Survey (ION) 75	Information exchange Joint activities Tangible resources Formal agreements	Frequency of interorganizational information exchange [Information exchange] Joint activities, trainings, events, or programs [Joint activities] Sharing of tangible resources [Tangible resources] Formal memorandums or contracts between organizations [Formal agreements]	Change in community capacity over the past 12 months	4 Likert‐type items and qualitative descriptions of linkages and resources	–
WeValueToolkit and List of Indicators 94	Justice Integrity Unity in diversity Empowerment Trust Care and respect	Policies are perceived as fair by all involved; people are treated equitably and with fairness [Justice] Decision‐making processes are ethical; processes and outcomes of decison making are transparent [Integrity] People explore their own ideas and/or reflect on their own individuality; people appreciate the differences in others [Unity in diversity] Organization's activities have a motivating effect on participants; people are encouraged to reach their potential [Empowerment] Trust partners to meet their commitments without the need for formal agreements; trust that others share a common vision [Trust] People do not talk negatively about others in their absence; people treat each other with kindness [Care and respect]	Assess intangible, values‐related processes and outcomes of organization's projects and activities	166 indicators	Evaluated via field trials 104
Community‐Based Participatory Research Rating Scale 71	Level of community involvement Relational issues Community access factors Mobilizing collaboration Training, educational and information support CBPR process methods Fostering collaboration	Importance of … community involvement in defining research objectives, developing the proposal, budget discussions, etc. [Level of community involvement] … sense of trust, mutual respect, commitment to power sharing, shared commitment [Relational issues] … remuneration for participation, reimbursement for travel/parking, flexible timelines and meeting places, technical access [Community access factors] … discussion about resources, clear roles, ground rules, common definition, strategic plan [Mobilizing collaboration] … educational materials, training, communication, information sharing [Training, education, information] … having regular meetings, use of consensus building, use of conflict resolution, regular assessments of partnership [CBPR process methods] … shared decision making, shared goals, public acknowledgement of community participation, common values [Fostering collaboration]	To assess the perceived importance of CBPR factors to the current study	33 Likert‐type items	Based on relevant social psychological and community organizing theory
Community Engagement Measure 65	Alignment with CBPR principles ○ Quality (how well) of community engagement ○ Quantity (how often) of community engagement	CBPR principles include: focus on local relevance and determinants of health; acknowledge the community; disseminate findings and knowledge gained to all partners; seek and use the input of community partners; involve a cyclical and iterative process in pursuit of objectives; foster co‐learning, capacity building, and co‐benefit for all partners; build on strengths and resources within the community; facilitate collaborative, equitable partnerships; integrate and achieve a balance of all partners; involve all partners in the dissemination process; plan for a long‐term process and commitments	Assessment of the quality and quantity of adherence to engagement principles	48 Likert‐type items	Cronbach's α Quality = 0.99 Quantity = 0.98
Peer Engagement Process Evaluation Framework 105	Supportive environment Equitable participation Capacity building and empowerment Improved programming and policy	Assess and address barriers and facilitators of engagement [Supportive environment] Ways to ensure that all participants’ experiences are respected and represented [Equitable participation] Individual and group abilities in terms of access, mobilization, interest, networks, opportunity, and literacy [Capacity building] Changes to program or policy; understanding of local risk environment; information synthesis; solution design [Improved programming]	To guide evaluation of primary data and project documents	4 domains, with assessment questions and sample constructs to measure	–
Ripple Effect Tool 83	Personal connections Professional connections Organizational connections New projects Changes in perceptions of health and lifestyle	Number of new personal connections since beginning of project; benefits to self, organization, community [Personal connections] Number of new professional connections since beginning of project; benefits [Professional connections] Number of new organizational connections since beginning of project; benefits [Organizational connections] New coalitions, committees, advisory boards, classes, or groups; changes in employment; articles, presentations, policy briefs, written or creative expression, news articles; new funding; policy or procedural changes as a result of the project [New projects] Change to understanding of health; lifestyle changes related to health; changes in people around you as a result of participation in project [Changes in perceptions]	To measure increased social connectedness and other benefits of community‐engaged research participation	12 open‐ended questions	–
Prevention Research Centers Cost Analysis Instrument 106	Labor Materials and consumables Travel Location	Salary and wage of each individual involved in the project [Labor] Cost of products or services regularly used; durable and capital goods [Materials and consumables] Travel expenses [Travel] Cost of primary project location and external places or settings used for project [Location]	To systematically collect budget year data on the costs related to a community‐engaged project	4 open‐ended items	Piloted with relevant stakeholders
Selected Outcome Measures
Student Learning Outcomes of Community‐Based Research (CBR) 107	CBR experiences Professional skills Civic engagement Educational experience Academic skills Personal growth	Type of CBR experience (course, internship, etc.); activities experienced in CBR course; course critique [CBR experience] Improved ability to run meetings, delegate, listen to others [Professional skills] Enhanced likelihood of participating in civic activities, voting [Civic engagement] Increased interest in college, in selected major [Educational experience] Improved analytical, academic writing, research skills [Academic skills] Ability to consider others’ perspectives; deepened understanding of self [Personal growth]	To measure student outcomes/benefits of participation in community‐based research coursework/projects	19 yes/no and Likert‐type items	Cronbach's α Overall CBR outcomes = 0.95 Professional skills = 0.91 Civic engagement = 0.86 Educational experience = 0.87 Academic skills = 0.80 Personal growth = 0.94. Convergent validity also tested
Social Network Analysis of Partnership Networks 108	Partner linkages Trust Benefits/drawbacks to partnership network Level of engagement	Shared information; shared resources; collaboration on community events; referrals; collaboration on grant proposals [Linkages] Strength of partnership relationship [Trust] Attendance at quarterly meetings [Engagement]	Measure the evolution/change in interorganizational relationships	19 survey items	–
Progress of Collaborative Action 47	Intensity of change—duration Intensity of change—reach Intensity of change—strategy Strategy employed	Assessment of (a) the activities and accomplishments facilitated by the partners and (b) the number of people reached or engaged in the activity	Measuring progress of partnership; intensity of change and strategy employed for change	Mixed methods coding scheme across 3 domains for partnership document review	96% inter‐observer agreement among 2 independent coders
Community Health Council Outcomes 109	Council development Community assessment Community action (coordination and leadership) Success stories Barriers	Number of presentations between community organizations and council; boards and committees [Council development] Times that emerging issues were discussed at council meetings [Community assessment] Current community priority; shared projects with other organizations; number of new/enhanced strategies, programs, services; new funding; new policy change initiatives [Community action] Success stories about outcomes [Success stories] Problems while carrying out council mission [Barriers]	Evaluate the effect of community health council actions on local health systems and health status outcomes	20 quantitative indicators with accompanying narrative (open‐ended) probes	Developed through qualitative document review and quantitative survey of relevant stakeholders. Stakeholder participation in analysis interpretation
Knowledge Ownership Social Network Analysis 74	Social network analysis of “knowledge ownership” (related to empowerment and self‐determination)	“Looking back to the beginning of the project, please rank the committee members [stakeholders] in order of who you would turn to for information relating to the project at that time”	Change in active involvement in knowledge creation	1 item	–
Critical Outcomes of Research Engagement 110	Patient‐centered Meaningful Team collaboration (“teamness”) Understandable Rigorous Integrity/adaptable Legitimate Feasible Ethical and transparent Timely Sustainable	Patient influence at each stage of research; outcomes compared to studies without patient engagement [Patient‐centered] Meaningful outcomes to patient community [Meaningful] Patient community's level of comfort with discussions and written materials; trust, respect among members [Team collaboration] Reading level and plain language of study materials and presentations; patient community training in research [Understandable] Realistic continuous improvement methods used? Discrete decisions for which partners were consulted at which level [Rigorous] Research question is clear and understood by all; partner contributions to ethical research design [Integrity/adaptable] Level of engagement at each stage of research; diversity of study population [Legitimate] Goals and methods are realistic and feasible [Feasible] Materials, study design, and data/privacy protection are transparent and fair [Ethical and transparent] Research conduct and sharing of information is timely [Timely] Mediums/channels used to disseminate/share findings; partners’ role in dissemination; use of findings outside the research study [Sustainable]	To assess the desired outcomes of engaged research	25 open‐ended questions	Developed through a workshop with patient partners

Appendix Table 2

Characteristics of Reviews (2009‐2018)

Review Authors	Years Represented	Sample Description	Key Domains Represented	Major Discussion Points
Bowen et al. 25	1973‐2014	68 studies containing quantified measures of stakeholder engagement	Level/extent of engagement Counts/records of participant behaviors, eg, referrals or attendance at events	Develop theory‐based, brief scales that are psychometrically tested Best scale will depend on context, goals, and relationship Qualitative data can improve understanding of how stakeholder's describe engagement and inform quantitative scales
Brett et al. 26	1995‐2012	65 studies on services user impact 35 studies on researcher impact 23 papers on wider patient community impact	Patient empowerment New patient knowledge and skills Mutual support and feeling part of a team Improved access to health information and problem‐solving skills Fresh insights into research issues and community needs Increased awareness of disease/conditions in community Challenges like lack of time and resources	Importance of studies to report context, process, and impact information Need for prospective studies to measure impact over time Valid and reliable tools needed to measure impact
Concannon et al. 40	2003‐2012	70 published articles on reports of stakeholder‐engaged research projects or programs	Type of stakeholders Stage and level of research involvement Improved relevance and adoption of research Increased trust Mutual learning Transparency of research process Challenges of engagement	Variable reporting/descriptions of stakeholder engagement in the CER and PCOR literature Quality and content of reporting needs improvement. The authors suggest 7 questions to consider (eg, types of stakeholders, methods for engagement, stage of engagement, impacts on research) Research needed: Descriptive studies of stakeholder engagement; evaluative research on the impact of engagement on research outcomes; and development and validation of engagement tools like training materials and recruitment tools
Conklin, Morris, and Nolte 41	2000‐2010	19 studies of outcomes of public involvement	Participant views/perceptions of project, changes, and satisfaction Participant knowledge, empowerment, and improved social ties Change to policy or practice Change to health care priority‐setting process	Public involvement is poorly specified and inconsistent/conceptual ambiguity Few studies explicitly state their intended outcomes; more robust evaluation is needed Yet, emphasizing measurement may miss “normative value of public involvement as intrinsically good because it is a deliberative democratic process”
Esmail, Moore, and Rein 16	2005‐2013	108 reviews and case studies of patient, public, and stakeholder engagement in health care or research	Appropriate funding/resources/time Attitudes toward engagement Availability of training Participation rates Diversity of participants Satisfaction Fairness and transparency Role clarity More relevant research More sensitive and ethical research methods Patient empowerment Changes to health outcomes	Few studies formally assessed or evaluated measures of engagement Little agreement on the major purposes of engagement and little empirical evidence to support its impact Process of engagement emphasized rather than impact Focus on immediate outcomes rather than long term Evaluations need to be included as component of research process
Jagosh et al. 39	Through 2009	276 peer‐reviewed and non‐peer‐reviewed publications and websites Descriptions of 23 participatory research partnerships	Partnership synergy Culturally appropriate research Community and researcher capacity Conflict and negotiation Sustainability System changes and new projects/activities	Synergy among partnerships can be both a context and outcome of engagement Long‐term impacts of engagement on health may arise from intermediate outcomes of capacity building, self‐empowerment, and infrastructure development New tools needed to understand and evaluate partnerships
Manafo et al. 111	2007‐2017	70 published articles addressing patient and public engagement in research priority setting	Level of engagement (deliberative vs. consultative) Types of data collection activities Processes used in engaging patients/the public in research priority setting Barriers and enablers for engagement in research prioritization	Realistic time frames, transparency about public contributions, and equal access to information creates an “ethically conscious” framework for engagement Limited evaluation of patient engagement in decision‐making activities, and effort should be made to solicit feedback from participants, sponsors, and leaders of engagement
Nitsch et al. 112	Through July 2011	42 published articles of participatory evaluation of health promotion projects/intervention	Level of stakeholder participation across evaluation phases Power/control over decisions Knowledge/experience to make decisions Executing a task in the evaluation process/operative performance	Participation in early phases of evaluation (eg, design and data collection) are higher than later phases (eg, reporting and dissemination of results) Program recipients are included most often in deliberations or discussions rather than having decision power, where they have a substantial say in determining the outcome Stakeholders should agree on a conceptualization of “participation” before deciding on a type of evaluation Authors of participatory evaluation studies need to describe stakeholder and participation processes in more detail to serve as a resource for the practice
Sandoval et al. 29	2002‐2008	46 instruments with 224 measures of CBPR context, processes, and outcomes	Community capacity Participatory decison making and negotiation Dialogue and mutual learning Leadership Task communication and action Self‐ and collective reflection Influence and power Empowerment and community capacity Health outcomes	Few measures reported reliability and validity information Context measures to assess community capacity for research, researcher capacity for engagement, and outcomes measures are needed Qualitative interviews and focus groups are necessary complement
Shen et al. 113	2005‐2015	10 articles examining parents as coresearchers in health research	Recruitment and group composition Level of engagement across research tasks Facilitators/enablers Success of parent engagement Benefits of engagement (for researchers and parents) Challenges	Parents were most often engaged in development, implementation, or evaluation of interventions, but less in conceptualizing the studies Lack of time and perceived control are challenges for parent participation Support, encouragement, and recognition of parents for their contributions are important, as topics are emotional and very personal to parents Plan for unpredictability
Tapp et al. 114	2003‐2013	17 articles of CBPR use in primary care settings	CBPR approach Level of engagement and tasks Outcomes Benefits of CBPR CBPR challenges	CBPR partnerships used to study access to primary care through needs assessments, developing surveys CBPR used to provide culturally tailored approaches to treatment for asthma CBPR used to identify culturally relevant behavioral and environmental factors influencing diabetes prevention CBPR used to understand barriers to engaging in exercise CBPR used to understand dissatisfaction with maternal health care and significant community stressors