The CI Index (CII): A New Instrument to Evaluate and Foster Collaborative Partnership in Public Health.

BACKGROUND OR
OBJECTIVES: Collective Impact (CI) is the sum of collaborative actions through coalition building to examine and address issues that threaten public health. We sought to develop an instrument to measure CI across diverse health-related organizations.
METHODS: The study was performed in 5 phases initiated with a comprehensive literature review through the generation of candidate items using a deductive instrument development approach. A matrix was then created to map assembled items onto related CI domains. An one-hundred and nine (n=109) item questionnaire covering multiple items per domain was administered to 200 health-related organizations selected randomly throughout the United States. Survey data were evaluated using Principal Component Analysis and Kaiser criterion or eigenvalue-greater-than-one rule was the factor retention method utilized.
RESULTS: Based on Kaiser's eigenvalue criteria and communality estimates, the number of items across the five domains of CI was reduced from 109 to 20, with 4 items populating each subdomain. All communality estimates in the final instrument had values > 0.6, which was sufficiently adequate as per Kaiser's criterion. CONCLUSION AND GLOBAL HEALTH IMPLICATIONS: This pilot study demonstrates CI represents a multi-factorial concept with domains that are multi-dimensional capturing diverse aspects of a construct. We developed the first measure of CI for public health practice. The index bears potential utility for assessing and monitoring areas of strengths and weaknesses within collaborative partnerships across the spectrum of population health.

1. Introduction

The healthcare environment is increasingly becoming a complex macrosystem that envelopes several inter-related factors which impact the health of individuals and communities at various levels.1,2 Many of these factors cannot be addressed or controlled through isolated efforts, but they demand the consolidation of diverse arrays of community assets, strengths, resources and know-how from each member in order to maximize CI.3,4 The CI (CI) framework proposed by Kania & Kramer5 represents a guiding framework for structuring collaborative efforts of coalitions and comprehensive community initiatives. The framework consists of five theoretical constructs or domains that indicate the degree of broad cross-sector coordination demonstrated by a particular collaborative group (multiple organizations partnering toward a common set of goals), which will ultimately result in the desired effects on the public health issues of interest. The five domains are: 1) common agenda, 2) shared measurement, 3) mutually reinforcing activities, 4) continuous communications, and 5) backbone organization (table 1).

Table 1

Core Pillars of CI

Common agenda	Members possess a shared vision for change that encompasses understanding the issue of interest and the need for a collaborative approach toward a solution
Shared Measurement systems	Gathering data and measuring stated outcomes uniformly across the collaborative partnerships to ensure alignment of efforts and accountability
Mutually reinforcing activities	Coordinating diverse partner activities through mutually reinforcing action plans
Continuing communications	Regular and open communication across partners to strengthen trust, buttress mutual objectives and generate common motivation
Backbone support	Creating and managing a supportive organization that is separate from the partnering collaborative entities with a focus on sustaining and coordinating the entire initiative

Source: Adapted from 13

The CI framework is based on social action premises such as: 1) no single organization is capable of effectively shouldering the responsibility for any major social problem, nor can any single institution address social problems effectively in silo; and 2) complex problems can be effectively solved only through cross-sector coalitions that engage those outside the sector including community stakeholders. The framework is gaining importance and acceptance as funders of health-related projects seek to maximize the impact of resource infusion at various levels.6 For instance, the Federal Healthy Start program, located and implemented throughout the United States (U.S.), is an example of a federally funded program with established effectiveness that promotes the adoption and integration of CI across the performance platforms of its grantees.7,8 The purpose is to create a culture that embraces coalition partnerships and collaborative work as integrative and essential elements of Federal Healthy Start programs with the expectation of enhanced efficiency and heightened impact. It is increasingly becoming clear that program accountability, an atmosphere of limited funding, and the need to be selective among competing programs of variable effectiveness constitute justifiable factors that warrant a CI index. In this context, a CI framework comes at a right time to meet a long-standing need to capture collaborative processes among community-based organizations partnering to improve maternal and child health (MCH) outcomes. Although the CI framework appears to be potentially useful in improving effectiveness of inter-organizational collaborations, currently, an effective instrument for measuring CI in the healthcare setting is lacking. To address this gap in public health practice, we conducted this study to create and pilot an instrument that measures CI across health-related organizations throughout the U.S. Based on the CI framework proposed by Kania & Kramer,5 we hypothesized as follows:

That CI will represent a multi-factorial concept with multi- dimensional domains that capture diverse aspects of a construct; and

That as a multi-dimensional concept encompassing multi-faceted constructs, an index scale rather than a traditional psychometric instrument will be more appropriate measure for overall CI in public health.

2. Methods

The study was conducted from January 2016 through November 2017 using survey data collected across the U.S. The initial step involved the generation of candidate items through a comprehensive literature review. We began with a deductive instrument development approach by identifying a theoretical definition of the constructs for “CI.” First, we searched PUBMED, CINHAL, and Scopus using the following search terms: CI, collective efficacy, collective action, collaborative action, community mobilization, comprehensive initiatives, collectivism, community coalition action, community engagement, community participation, inter-organizational collaboration, history of collaboration, shared leadership, and community readiness. We also conducted open internet searches for gray literature using Google, Bing, and Yahoo search engines. Our search revealed a definition frequently used for CI from a seminal gray document by Kania & Kramer.5 This definition had been operationalized into a distinctive framework encapsulating five domains or conditions for collective success: 1) Common Agenda, 2) Shared Measurement, 3) Mutually Reinforcing Activities, 4) Continuous Communication, and 5) Backbone Support. We then conducted a narrowly focused literature search to identify measures related to the conceptual domains of the referent framework and other previously used measures that attempted to capture related content areas for CI. To focus the search on measures or instruments, we ran secondary searches using the following combined terms “questionnaire” OR “inventory” OR “survey” OR “scale” OR “measure” OR “instrument” OR “assessment.” We also employed the same search terms combined with “Common Agenda” OR “Shared Measurement System” OR “Mutually Reinforcing Activities” OR “Continuous Communication” OR “Backbone Organization.”

2.1. Literature review and item generation

Articles abstracts and full texts were reviewed, and then categorized by relevance. To achieve understanding of the relevant literature and the CI construct, two faculty members with expertise in social science and community health reviewed full-texts of retrieved articles independently, with the goal of identifying topics that were related to CI. The findings were discussed weekly to achieve consensus. A matrix was developed that mapped the potential candidate-items onto related CI domains. Candidate items were derived from questions previously used by other researchers, as well as identified statements that could fit into specific CI domains. Next, based on examples reported in the literature, our research staff compiled items that corresponded to the conceptual map, which represented the first item pool. For this first set of preliminary items, we followed guidelines to ensure that the items were properly constructed. First, items were designed to address only a single issue (avoiding “double-barreled” items). We created items predominantly positive to keep consistency and reduce question burden. One or two negatively worded items were included per domain to control for agreement bias and social desirability. Second, items were ordered in a consistent manner in terms of sequential concepts to make sure that we did not mix items that assess one concept with items that assess other responses or outcomes of behaviors. Content redundancies were desirable at this stage, so we included similar items measuring similar ideas (but still differently worded). Finally, a list of multiple items per domain was framed and formed the template for evaluative analysis subsequently.

2.2. Content assessment of candidate items

To assess the adequacy of our initial item pool, two experts in community cognitive assessment & epidemiology, held a series of six-weekly virtual meetings to review the relevancy of items relating to CI. Through this collaborative effort, item wording was refined and some items dropped. The remaining item pool was then formatted into an online survey using the Qualtrics data collection software to gather additional expert insights. Federal Healthy Start and other community partners were invited to participate in online nominal groups (3 groups of 5-6 participants each) to assess face validity and content validity of the subdomain items. Participants were asked to assess the content adequacy (yes/no) and clarity of language (open-ended), as well as describe insights or experiences about CI within their organizational settings and assist in identifying aspects of CI not captured by the deductive approach. Any input regarding comprehension issues, needs for re-wording, and/or readability of items were noted and incorporated into the final survey. Further, comments/responses to open-ended statements were classified into several categories using thematic analysis. From these categorized responses, second-wave items were then derived and organized using a Likert Scale (5: Strongly Agree to 1: Strongly Disagree). The entire set of questions is presented in Supplement A.

2.3. Instrument administration

This initial scale was comprised of 109 items, and was administered to 200 healthcare or health-related organizations in the US. The organizations include community health centers, hospitals, public health departments, and non-profit health care organizations. Efforts were made to randomly select at least one organization from each state in the U.S. Individuals, including public health nurses, social workers, psychologists, administrators, and healthcare executives, were nominated by their organizations to respond to the survey. A trained staff interviewer contacted the organizations via telephone, provided a brief overview of the research project, and administered the questionnaire. Responses were voice-recorded and documented on paper. The collected data were then organized in a single Excel document for analysis.

2.4. Statistical analysis

We performed a principal component analysis (PCA) on the 109 items across the five CI domains. Initially, we employed both the scree plot and Kaiser’s rule9 of eigenvalue-greater-than-one criterion to select the number of components (i.e., number of factors to retain). The Kaiser rule is to drop all components with eigenvalues under 1.0 – this being the eigenvalue equal to the information accounted for by an average single item. With this approach, we retained and interpreted any component with an eigenvalue greater than 1.00. Our approach also involved striking a balance between “reduction” and “meaningfulness” of the correlations that existed within variable groups. We employed oblique rotation (promax) to optimize configuration on factors, allowing for maximum restriction of orthogonality (Delta = 0). The promax option allows for correlations of factors, and is computationally fast. When items loadings were >0.4 on more than one factor, one factor was selected based on clinical judgement regarding the most plausible solution. We then evaluated and compared the pattern and structure of matrices to determine the influence of shared variance. We applied the Barlett-test (test of sphericity) which assesses the following hypothesis: H0: Variables are uncorrelated; H1: Variables are correlated. The Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy (KMO) was computed to determine adequacy of sample size. Communality was calculated to show the proportion of variance explained by the extracted factors, with initial set at 1.00. The generated extraction communalities would then represent estimates of the variance in each variable accounted for by the components. The data were analyzed with the statistical package SPSS version 22.0 and by two statisticians working independently. Periodic meetings were held to discuss areas of discrepancies, and results were only adopted when the two statisticians concurred. Since this was a phone interview, verbal consent was recorded and archived. The study was considered exempt by Baylor College of Medicine Institutional Review Board.

3. Results

The instrument (Supplement A) is comprised of ordinally-framed variables on a 5-point Likert Scale that were recoded as numerical types (scores of 1 to 5) during analysis. In all, 109 items covering the five CI framework domains were administered with the following item distribution: 1) Common Agenda (A) = 22 items; 2) Shared Measurement Systems (B) = 23 items; 3) Mutually Reinforcing Activities (C) = 22 items; 4) Continuous Communication (D) = 24 items; and 5) Backbone Support (E) = 18 items. The mean score for each domain varied across the items. For Common Agenda, the highest mean score (Mean ± SD = 4.95 ±0.22) was observed for item A22 (Our partners’ individual work is increasingly aligned with the common agenda), while the lowest score (Mean ± SD = 3.90 ±0.49) was for item A19 (People in our community do not work together to solve their problems). For shared measurement systems, the highest score (Mean ± SD = 4.99 ±0.12) was for item B18 (Members of our collaborative group have clear expectations about the confidentiality of the data used by the collaborative), while the lowest score (Mean ± SD = 2.36 ±0.72) was observed for item B14 (There is sufficient funding to improve the information system for our collaborative group). Item C1 (In our collaborative, there is a process in place to share successes with each other and the larger community; Mean ± SD = 4.80 ±0.40) and C20 (The collaborative uses data as the basis to determine which activities should be stopped; Mean ± SD = 4.80 ±0.60) shared the greatest score for mutually reinforcing activities, while the lowest score was observed for item C21 (Funders are redirecting funds to support the collaborative goal; Mean ± SD = 2.36 ±0.74). A score of 4.96 (SD±0.20) and 2.75 (SD±0.94) represented the highest and lowest mean value for the domain assessing continuous communications respectively. The corresponding items were D6 (highest mean score; The collaborative group provides opportunities to share our different points of views) and D12 (lowest mean score; We have a common language of communication among partners in this collaborative). Finally, for backbone support, we observed the highest mean score of 4.93 (SD±0.36) for item E14 (This collaborative uses a supporting backbone infrastructure to build public will) and the lowest mean score of 1.70 (SD±0.75) on item E12 (The activities of this collaborative are established by a senior-level committee with ultimate decision-making power). It is noteworthy that the latter item scored (E12; see Supplement A) the lowest for all domains combined.

The Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy was greater than 0.40 for all domains, while the Barlett-test (test of sphericity) was significant for common agenda (p-value = 0.03) and marginally significant for shared measurement systems (0.06). This could be due to the assumption that each domain mirrored a specific dimension of CI. However, the significant correlations among fewer items might have been blunted by the presence of and dominance by uncorrelated variables. For this reason, we proceeded with a reduction procedure and generated the eigenvalues after rotation for each component in the data. The scree plots for all the domains exhibited a number of small inflexions. Under the criteria of eigenvalue of >1 or Kaiser’s criterion, 10, 11,10,12 and 9 components emerged from the sub-set of items belonging to common agenda, shared measurement systems, mutually reinforcing activities, continuous communications and backbone support respectively. Table 2 presents our results with the percentage of variance accounted for by the components.

Table 2

The CI components satisfying Kaiser’s criteria and the percent variance explained

% of variance explained	60%	63%	59%	65%	61%
CI domain	Common agenda (communalities)	Shared measurement systems (communalities)	Mutually reinforcing activities (communalities)	Continuous communications (communalities)	Backbone support (communalities)
	A3: Everyone in our collaborative group works on the same community issue	B14: There is sufficient funding to improve the information system for our collaborative group	C15: The leaders in our collaborative are responsive to change that will impact their organizations	D6: The collaborative group provides opportunities to share our different points of views	E5: For this collaborative group or coalition, an existing organization took on the lead in coordinating the strategies of the collaborative group
	A5: Together we developed a plan of action to outline the way in which the identified community problems will be addressed	B9: In our collaborative, we routinely participate in learning forums or continuous improvement discussions	C16: The efforts of our partners build upon existing activities in the community	D13: When communicating internally, member organizations use various channels to communicate more effectively, such as online platforms, facetoface meetings, and teleconferences	E16: Board members and other community leaders look to our backbone organization for initiative support, or strategic guidance and or leadership
	A7: If there were a problem that affected the whole or most of our community, agencies in our community would work together to deal with the problem	B16: Our collaborative group effectively uses webbased technology for data collection	C22: Our partners continue to explore new resources	D19: We involve community members to identify priority areas of needs	E2: In our collaborative, there is a leading organization that guides the vision of the collaborative
	A1: Community leaders who are part of our collaborative group are very committed to solve problems through collaboration	B2: Our results are measured using the same metrics or indicators	C3: Everyone supports the activities of others	D16: Leaders of the group take into account all members’ points of view when making important decisions	E1: As part of our collaborative, there is a separate organization dedicated to coordinating the various activities of the collaborative
	A22: Our partners’ individual work is increasingly aligned with the common agenda	B22: Our community partners lack the capacity to implement a shared data system	C10: People in this collaborative group know each other very well	D15: All partners are equally involved in the decisionmaking of the collaborative group	E3: We currently are supported by one funder who initiated the collaborative group (serves as planner, financier, or convener)
	A13: Participants in this collaborative group have a lack of commitment to work together	B10: We are able to use the data we collect to rapidly implement solutions to the problems found	C8: There are persons in this collaborative that others can look up to	D9: This collaborative provides a platform for concerns to be addressed	E15: In this collaborative, we have a supporting backbone infrastructure which mobilizes funding or seek new resources from public or private sources
	A4: Agencies in our community are all working together to achieve significant community change in our locality	B13: In our collaborative, there is strong leadership to improve the information systems of the group	C11: I feel my voice counts in this collaborative group	D8: We have a strong capacity to solve problems together	E10: As part of our backbone support we have a facilitator for meetings
	A17: People in our community can successfully stand up for their rights	B11: Our collaborative group is engaged in ongoing evaluation and quality improvement	C17: There is duplication of services among partners in our collaborative	D5: Yes, communication is one of our strengths	E18: There is a backbone organization that provides support and recognition to partners for their work and leadership
	A11: People in our collaborative group can mobilize many supporters in our community	B18: Members of our collaborative group have clear expectations about the confidentiality of the data used by the collaborative	C9: There are people in this collaborative that I can look up to	D10: Ideas discussed in our collaborative are transmitted throughout our organizations	E17: All partners coordinate efforts regularly with the backbone organization
	A21: All collaborators or partners who are part of our collaborative group publicly discuss/advocate for common agenda goals	B3: In our collaborative, we all have common outcomes for which we collect data	C4: It is very easy to initiate or coordinate any activities with the help of members of the collaborative	D1: As a member of this collaborative, I feel that my interests are valued and protected
		B12: If a member of our collaborative group needs help with data collection or analyses, there is ongoing support available		D2: People in this collaborative attend meetings to get to know each other better
				D21: Partners report increasing levels of trust with one another

Further analysis of the components demonstrated that there were generally minor differences in communalities when these components were arranged in a descending array ranging from the highest to the lowest estimates. For almost all the domains, a plateau was observable after extraction of the top four components (Table 3). This suggests that fewer components could offer the same information. Consequently, a decision was reached to retain the top four components (based on the communalities and relevance) as adequate items that could explain the same magnitude of component variance as the ones in Table 2. A reduction of the number of components using Kaiser’s eigenvalue criteria and communality estimates greatly enhanced the sub-domain correlations which now became statistically significant except for backbone support (Common Agenda, p = 0.045; Shared Measurement Systems, p = 0.038; Mutually Reinforcing Activities, p = 0.028; Continuous Communication, p = 0.020). A notable observation (Table 3) is that all the communalities had an individual value >0.6, which, as per Kaiser’s criterion, was indicative of sufficient communality.

Table 3

Communalities of extracted components (Initial Set at 1.00)

Common agenda	Communality (Sum of Squared Factor Loading)
A3: Everyone in our collaborative group works on the same community issue	0.69
A5: Together we developed a plan of action to outline the way in which the identified community problems will be addressed	0.69
A7: If there were a problem that affected the whole or most of our community, agencies in our community would work together to deal with the problem	0.68
A1: Community leaders who are part of our collaborative group are very committed to solve problems through collaboration	0.66
Shared measurement systems
B14: There is sufficient funding to improve the information system for our collaborative group	0.81
B9: In our collaborative, we routinely participate in learning forums or continuous improvement discussions	0.77
B16: Our collaborative group effectively uses webbased technology for data collection	0.75
B2: Our results are measured using the same metrics or indicators	0.69
Mutually reinforcing activities
C15: The leaders in our collaborative are responsive to change that will impact their organizations	0.75
C16: The efforts of our partners build upon existing activities in the community	0.75
C22: Our partners continue to explore new resources	0.71
C3: Everyone supports the activities of others	0.68
Continuous Communications
D6: The collaborative group provides opportunities to share our different points of views	0.76
D13: When communicating internally, member organizations use various channels to communicate more effectively, such as online platforms, facetoface meetings, and teleconferences	0.75
D19: We involve community members to identify priority areas of needs	0.74
D16: Leaders of the group take into account all members’ points of view when making important decisions	0.73
Backbone support
E5: For this collaborative group or coalition, an existing organization took on the lead in coordinating the strategies of the collaborative group	0.79
E16: Board members and other community leaders look to our backbone organization for initiative support, or strategic guidance and or leadership	0.73
E2: In our collaborative, there is a leading organization that guides the vision of the collaborative	0.69
E1: As part of our collaborative, there is a separate organization dedicated to coordinating the various activities of the collaborative	0.68

4. Discussion

Our study results demonstrated that the five tenets of the CI framework do not subscribe to a unidimensional interpretation. We observed that each domain embodies a lens that captures some specific construct of CI. Given this multi-dimensional structure, the instrument may not be suitable for use as a scale but rather should be used as an index. An index is a set of items (questions) that structures or focuses multiple yet distinctly related aspects of a dimension or theoretical domain of behavior, attitudes, or feelings into a single indicator or a score.6 A scale is typically a cluster of items (questions) that syncs into a single domain. Our results showed this was not the case for CI items. Hence, the findings in this study are in agreement with our first hypothesis which posits that CI represents a multi-factorial concept with domains that are multi-dimensional capturing diverse aspects of a construct. Considering that coalition building and successful collaborative partnerships are characterized by a variety of activities at multiple levels framed to maximize utilization of community assets to attain shared objectives, this finding is not surprising. Therefore, in support of our second hypothesis, the instrument we developed more appropriately represents an index rather than a traditional psychometric instrument that captures a simple construct.

The new instrument, referred to as “CI Index” or CII for short, is framed on a Likert scale from 5 to 1 with four items in each CI domain. All the scores are numerically summed across the five domains to yield a maximum score of 100 and a minimum score of 20. Subsequent studies should attempt to compute the subdomain scores to ascertain areas of strength and weakness. Subdomain scores will then potentially range from a maximum of 20 to a minimum of 4. At this pilot stage of instrument development, it is difficult to advise cut-offs correctly without information from future studies that will assess predictive validity of the index in diverse settings. Nonetheless, the following may be recommended for evaluators and practitioners interested in using the tool to assess the impact of collaborative endeavor. For subdomains, we recommend: ≥15 indicates strength; 10-14 indicates promise and <10 indicates weakness. The corresponding total score recommendations for CI are: ≥75 indicates strength; 50-74 indicates promise and <50 indicates weakness of the partnership. These values could change as utilization experiences are gathered through large-scale studies to test the performance and predictive ability of the CII.

5. Conclusion and Global Health Implications

A merit in our study is that we were able to demonstrate the feasibility of measuring CI based on a comprehensive process that entailed a rigorous literature review, expert consensus analysis, stakeholder input, nationwide data collection from a variety of health-related organizations and a meticulous data analysis and evaluation process. Although the CI framework has assumed popularity in health promotion through coalition building,10-12 we are unaware of an evaluated measurement instrument that coalition partners could easily utilize to assess baseline efforts and measure progress over time. Our instrument could be employed to identify specific segments of the collaborative that need more attention in order to attain desired goals.

A limitation of this study is that our search probably missed some non-English publications on the subject matter. However, this is very unlikely given that we employed the phrase “CI” as part of key phrases which could have captured relevant articles in other languages as well. Another limitation of the study is the likelihood that the US population may differ considerably from other settings in terms of the healthcare delivery system. It is therefore, to be recommended that this instrument be tested and validated in other settings.

Constructed as an index, the CII yields a summative score of collaborative efforts across the themes of a common agenda, shared measures, continuous communication and mutually reinforcing activities. Further, if the collaborative group has identified an organization that provides a cohesive leadership and administrative structure to the partnership, the instrument can be utilized to assess the functionality of the backbone organization as well. The inventory can be utilized to identify strengths and weaknesses in the collaborative group, or could be employed to provide an overall score of CI. The utility of the CII is vast, and potential end users include health-related organizations, social service providers, and other agencies with vested interest in public health.

CI is a multi-dimensional construct.

An index-based instrument could be employed to identify specific segments of a collaborative that need more attention in order to attain desired goals

The utility of the CI index is vast, and potential end users include health-related organizations, social service providers, and other agencies with vested interest in public health.