Evaluating a community-based participatory research project for elderly mental healthcare in rural America.

The purpose of this evaluation was to explore the collaborative nature of partners in a rural mental health program for the elderly, and to test an adapted method of assessing the collaborative process. Sixteen collaborative partners were interviewed to explore ratings of collaboration across 6 domains identified as critical to participatory research. Results indicate that the context of rural Missouri and uniqueness of the program necessitated an approach to collaboration that began with a top-down approach, but greater community responsibility developed over time. Partners recognized the efforts of the program's directors to seek input. Most were satisfied with their roles and the degree of success achieved by the program, although several wanted to have more input in the future in some domains, but not in others. Interviews revealed numerous barriers to achieving sustainability. Methods to improve the assessment of collaboration are discussed and areas for improvement are offered.

Introduction

Although studies suggest that the prevalence of mental illness nationally is not greater in rural areas, difficulties with access and quality of healthcare are many times greater (Norquist and Regier 1996). Reports suggest that across geographic regions, the estimated lifetime risk of any mental health disorder at age 75 years is 50.8%, slightly higher than the observed lifetime prevalence of 46.4% (Kessler et al 2005). Furthermore, the number of older adults affected by psychiatric disorders is expected to increase more than 3-fold to 15 million persons by 2030 (Patterson and Jeste 1999).

Problems with access and quality of mental healthcare in rural America have been found to originate both from the current structure of the healthcare system and from the stakeholders themselves. Factors specifically associated with rural settings include cultural (Valle 2005) and environmental factors, such as cost of services (Li et al 2005) and driving distance (McCarthy and Blow 2004). Older adults have been found to be less willing to utilize services despite having a greater need (Kessler et al 2005; Streiner et al 2006), with lack of knowledge about what counseling is and less openness to discussing personal problems cited as barriers (Smith et al 2002a). In addition, rural service providers themselves have been found to be less willing to collaborate with one another due to historical traditions and suspiciousness about loss of market share, as well as a lack of training in recognizing mental health problems in the elderly (Smith et al 2002b, 2004).

In order to address the growing mental health needs of older adults in rural Missouri, a consortium called ElderLynk was established through a federally funded university grant. The purpose of the present manuscript is not to focus on the patient outcomes of the ElderLynk program, as these will be reported elsewhere (McGovern et al 2008), but reserved to discuss an evaluation of the collaborative process between university researchers and their clinical community partners. Additionally, we will discuss methodological considerations for such an evaluation.

Participatory research and collaboration

In community-based participatory research (CBPR), success involves establishing whether the type and extent of collaboration between partners has resulted from mutually agreed upon sets of responsibilities and activities and, further, how this impacted the project outcomes and sustainability of the collaborative partnership. Accordingly, effective collaboration does not necessarily mean equal input from all parties, but achieving the degree of input the parties desire in a multi-stakeholder venture, which may be fluid and renegotiated over time. The ideal form of participatory research holds that all stakeholders (ie, academics, clinicians, community members) have equal input from the conceptualization of a relationship and any given research project through data collection, analysis, interpretation, and project dissemination and program quality improvement (see Jones and Wells 2007), but adopting this is often not common, desirable, or possible in every participatory venture (Viswanathan et al 2004; Israel et al 2005; Macaulay and Nutting 2006; Westfall et al 2006)—a tenent that will be reinforced by the results discussed in this paper. CBPR strives to achieve relationships that are completely egalitarian, but they must first develop over time and thus, we hold that an evaluation of the participatory process should most critically focus upon the concordance between desired and possible levels of collaboration and what was actually achieved.

From a health services research perspective, community consists of two different groups. The first group is composed of the service providers (formal and informal) providing front-line care and their associated business partners. Recipients or potential recipients of care and their social network members comprise the second group, which can sometimes be composed of multiple subcommunities that differ along important cultural dimensions. Although we will only briefly highlight the relationship between researchers and care recipients, for reasons that will be discussed below, the focus of this report is primarily concerning the collaborative partnership with the care provider group.

ElderLynk: Development and outcomes

Establishment of ElderLynk occurred after a university-generated grant application to HRSA was successfully funded in 1999; however, the partners initially began collaboration several years prior for a broader health-focused grant application and for the mandated development of a county mental health board. It was necessary to have this funding to initiate a participatory research model, with the goal of transitioning into full CBPR approach over several years. It is important to note that in the collaborative initiatives prior to ElderLynk, there are reports that the relationships were quite strained, with some dissention between community members; this was the platform upon which the present project began, building relationships and overcoming past difficulties. For ElderLynk, community stakeholders were involved in the development, implementation, and evaluation of program services. The following section will discuss the participatory approach adopted and discuss how that changed over time.

At notice of funding, a two-day retreat was held to identify a common vision for the program. The agreed upon vision was the development of an integrated system of care with “no wrong door” for entry into care. The five diverse healthcare education and service providers agreed on the joint goals established in the grant proposal for ElderLynk to: 1) link existing mental health and social service agencies with primary care providers, where the majority of the elderly enter the healthcare system (Levkoff et al 2004), and to 2) educate students in the health professions, practitioners, and the general public about senior mental healthcare and quality-of-life issues.

The initial two-day retreat included not only all the advisory board members, but also a broad array of community stakeholders, including leadership in public health and other civic leaders. An outside consultant was hired to serve as a neutral facilitator, and to assist in building consensus about needs and establishing a common vision. In addition, a collaborative care model for consumer assessment, treatment, and follow-up was unanimously agreed upon. This model was adopted to maximize the scarce community resources without causing unnecessary competition or duplication of services – issues that were significant barriers to collaboration for the partners in the past. Annual retreats were held with community representatives and invited consultants to serve as a steering committee and to continuously evaluate outcomes. Monthly advisory board meetings, which included consumer representation, were held for accountability and for continuous quality improvement through problem solving and planning. Meetings were initially led by the project director, with the project coordinator later taking on chairing the meetings. The project coordinator/case manager presented practice demographics and financial reports monthly for accountability. All major decisions were brought to the advisory committee, where issues were voted upon to reach majority consensus. Minutes and reports were distributed both electronically, prior to meetings, and by paper during meetings, with major documents, such as annual grant renewal reports, sent to partner agencies spiral bound by mail. Meeting discussions included strategic planning, such as inclusion of new partners and hiring of new personnel, and implementation issues, such as sites for services, marketing and navigating barriers. While discussions were subdued at the beginning development of the program, participation of partners at meetings increased significantly over time.

Over the course of the project, the five partners included a rural medical school, a system of federally qualified healthcare centers, a 164-bed medical center, a community mental health center, and an outpatient and residential mental healthcare provider. In 2003, a regional hospice provider was added and the original 8-county service area was expanded to 10 counties. Providers received annual contracts through the university, with those who provided clinical care receiving quarterly, mutually agreed upon stipends. All partners submitted monthly estimates of indirect contribution to the grant objectives, which were tracked by the project coordinator/case manager. The inclusion of the hospice as a new partner was decided by the original five partners, based on mutually beneficial dimensions of the project and the community that were thoroughly discussed in successive advisory board meetings.

The need for ElderLynk was first recognized by the University Research and Grants Department. The key grant writer was on the board of directors of a local community mental health center and intimately aware of the difficulty of providing mental health services in a rural setting, the gap in geriatric mental health services, and the difficulty in recruiting mental health providers to provide services. The incentive for all the partners to come together for ElderLynk was the potential improved ability to recruit, in particular, a psychiatrist who could serve multiple agencies. While the vision for ElderLynk changed after the two-day initial planning retreat to include central care coordination/case management and provide counseling in primary care, rather than focusing on psychiatric services, this initial work was the beginning of other collaborative projects. Subsequent to ElderLynk other successful partnering grants included Carelink, targeting the underinsured and uninsured, the Woman’s Care Connection, and, most recently, a telehealth grant funded by the Missouri Foundation for Health. In addition, the Northeast Missouri Rural Health Network, which predated ElderLynk and was fraught with political difficulties, maintained its viability. In fact, the Northeast Missouri Rural Health Network agreed to “adopt” ElderLynk and assist in nurturing its launching as an independent entity after initial funding ceased.

The organizational structure and general operation of ElderLynk remained consistent across the years of its development. Representatives from each partner agency formed an advisory board, in addition to a consumer representative. There was virtually no turnover in board members. Other than adding the hospice, the players remained the same, except when someone was replaced in their organizational position (this happened once). There was also little staff turnover. However, staff was continually added to enhance the mission of ElderLynk. For example, an associate pastor, who was a parish nurse, was hired to head up the Parrish Nurse program, as the role of the churches in aging mental health was identified as being important and needing more focus than current staff could provide.

As noted above, it was determined that a collaborative care model would be the most effective and efficient to accomplish project aims. The care coordinator/case manager was responsible for screening, assessment/risk management, care planning, implementing service arrangements, monitoring/evaluation, and advocacy for ElderLynk patients. Referrals were made to the case manager through a variety of mechanisms, including primary care physicians (9%), geriatricians (22%), nursing home care managers (46%), local area agencies on aging (3%), federally qualified health councils (5%), other clinics (4%), self/family members (8%), and other (not recorded, 1%). The case manager conducted a standardized intake assessment at the patient’s residence. Based on intake findings and in collaboration with the patient and family, a treatment plan was generated to be reviewed at the next monthly clinical review team meeting. The case manager also contacted appropriate referral sources and arranged a follow-up visit with the patient and family to assure that care needs were met. If appropriate and agreeable to the patient and family, the patient was referred for counseling. Monthly clinical review team meetings were conducted to review both care plans and progress made. Led by a geriatric nurse case manager, the ElderLynk team included a psychiatrist; social workers, who served as mental health counselors; and a psychologist. At a later point, a geriatrician and a parish nurse were added to the team. For each patient, patterns of ElderLynk service usage (number of hours of case management and number of counseling sessions) and scores on repeated assessments were tracked over time (at 6-month intervals) utilizing the Government Performance and Results Act instrument.

Personnel were housed in ways to enhance the goals of ElderLynk, and benefit both consumers and partner agencies. For example, the counselors, who were subcontracted by ElderLynk through the community mental health provider and the drug and alcohol service provider, were housed in the federally qualified healthcare provider, the best identified site of service for both consumers and for reimbursement/sustainability. The project coordinator/care manager had her operating office at the university medical school, but provided services wherever needed—at nursing homes, hospitals or in the consumer’s home. Essentially, the university medical school provided operating space, clerical and other services. The other partners provided clinical personnel and clinical space. The only clinical services provided at the university medical school were for initial screening and for clinical research projects. The clinical team did, however, meet monthly to review all cases collaboratively, including students from various disciplines for training in interdisciplinary care. While all the staff were responsible to their primary employers, each of whom had representation on the ElderLynk Advisory Board, monthly reports to the board reported ElderLynk clinical productivity for each clinical staff person.

ElderLynk clinical outcomes

More than 700 elders were served from 2000 to 2005 through a “linked” network of community providers who had not worked together in this way in the past. Outcomes for Elder-Lynk participants demonstrated a significant improvement in level of depression (effect size d = 0.38, p = 0.0005), life satisfaction (effect size d = 0.40, p = 0.001), and psychosocial functioning (effect size d = 0.33, p = 0.007) between first and last visits. Significant declines were found in mental status (effect size d = 0.28, p = 0.03), daily functioning (effect size d = 0.45, p = 0.001), and overall health (effect size d = 0.25, p = 0.02). More than 2000 contacts to deliver professional education in geriatric mental healthcare and more than 8000 contacts for community education, including chronic disease management, were made during the course of program funding. Consumer and family satisfaction ratings through surveys and focus groups reported that the community was “highly satisfied” with the services provided. Referring physicians reported somewhat less satisfaction than patients and families but, in general, were “satisfied.” At the termination of funding, ElderLynk was “adopted” by the rural health network and is currently applying for independent not-for-profit status.

Evaluation methodology

CBPR is not a new approach by any means, but it has gained increased popularity as funding agencies and research centers have increased attention to constructing, disseminating, and implementing effective and sustainable interventions, in addition to its adoption as a valid approach to research by disciplines outside of public health, where it initially began. CBPR can be used as an approach at any point in the research continuum from exploratory projects in new areas of research through efficacy and effectiveness studies, to translation and implementation efforts. However, the approach is especially critical beginning at the effectiveness stage of research, where interventions often have to begin to be tailored to fit within varying organizational structures and larger sociocultural and sociopolitical environments. Although CBPR can be adopted for a single study, it is more appropriately applied to ongoing collaborative relationships due to the extensive time required to build relationships and develop research interests that are mutually beneficial to all parties. As noted earlier, it is usually thought of as an ideal to reach, which takes much time to develop. ElderLynk was the first program to begin this developmental process around serving elder mental health needs.

Learning from the writings of such notable scholars as Rogers (2003), it has become unacceptable that evidence-based practices require decades to be communicated from effectiveness studies into implementation programs. Although ElderLynk, in its totality, was not an evidence-based practice, it adopted a collaborative care model, which is well grounded in the empirical literature. It also adopted evidence-based components into its treatment arms, such as cognitive-behavioral therapy for depression, an evidence-based drug and alcohol treatment model for the elderly and cognitive-behavioral treatment for sleep disorders in the elderly. The ElderLynk program was cited by HRSA as a promising practice in 2004.

It is important to recognize that CBPR is not a static or easily definable approach. Given the interests of participating stakeholders, their readiness for changes, and their ability to contribute to a research endeavor, there are varying levels at which CBPR would be appropriate and effective. For instance, at the outset of a CBPR program, community partners may not be interested or have the skills to meaningfully contribute to all of the research elements of a particular project. Determining the most appropriate degree of collaboration between researchers and community partners at a particular point in the relationship is highly dependent upon partners’ ability and desire to participate in the entire research process. Thus, to evaluate the CBPR process, one must choose a methodology that can respect the stage of development of a given community partnership.

Using a model of CBPR described by Naylor and colleagues (2002), collaboration was explored across six dimensions: identification of need, research activities, use of resources, evaluation methods, indicators of success, and sustainability. As ElderLynk’s primary funding originated from research grants, some elements of the program changed over time with funding source, but maintained a central focus of education and facilitating access to elder services in ten counties in northeast Missouri. This aim was further pursued through educational initiatives offered to community members and to providers.

In Naylor and colleagues’ (2002) method of evaluating collaboration, participants are asked to individually characterize the type and degree of collaboration across the six dimensions using four different categories, indicating the balance of responsibilities and participation on the project. Focus groups are then held with the partners at each participating site to discuss individual-level ratings with the goal of coming to a site-level consensus rating. The discussion that ensued was the primary focus of analysis. Due to the fact that most of the community partners in the ElderLynk initiative were led by individuals that later reported to the larger group of interested parties at a given organization, we decided that this process would have to be adapted with the approach described below.

Methods

Programmatically, the research questions for this evaluation were: 1) to what degree did the community partners perceive there to be a collaborative process with university affiliates; and 2) within each of Naylor’s six domains, how satisfied were participants with this degree of collaboration, what potential was there for changes in the future, and what suggestions could be offered to improve both the collaborative process and the project outcomes. Methodologically, the research focus was to explore the feasibility of the modified method of Naylor and colleagues (2006) to evaluate collaboration of a participatory research intervention.

Semi-structured questionnaires were developed to reflect each of the six domains of collaboration (see Table 1). Rating scales were developed for each of these areas and followed by probe questions to gain insight into the rationale used in making each rating. Each scale used a 4-point rating of the amount of collaboration in each domain and asked about the extent of input a given participant had in elements of each domain of the model, ranging from 1, None or Low to 4, Very High. Probe questions sought information that would allow for an explanation of how the partner decided on a particular rating of collaboration. Note that the rating scales only assessed the perceived degree of collaboration in a particular domain; these numerical values do not correspond to those of Naylor’s model.

Table 1

Evaluation scheme for ElderLynk

	Expert-driven			Participatory research
	Consultation from experts 1	Cooperation 2	Participation 3	Full control by community 4
Identification of need	Issue predetermined by experts who then “sell” program to clinicians	Clinicians offer advice and input but experts make decisions	Equal decision-making	Clinicians control decision- making, experts advise
Definition of actual research activities	Issue predetermined by experts who then “sell” program to clinicians	Clinicians offer advice and input but experts make decisions	Equal decision-making	Clinicians control decision- making, experts advise
Use of resources	Heavy influx of outside resources	Outside funding still most important but may include “in-kind” contributions	Balanced funding	Small amount of seed money stimulates
Evaluation methods	Tests, surveys, interviews designed by researchers and conducted using hypothesis testing and significance of results statistically determined	Tests, surveys, interviews designed by researchers, conducted by community, using hypothesis and significance of results statistically determined	Partnership in design and conduct using multi-methods of data collection in natural context	Advice from experts sought on design, 100% conducted by community using multi-methods in natural context
Indicators of success	Clinicians learn little, researchers have difficulty sharing power	Clinicians take only marginal responsibility and depend heavily on experts	Power is shared, but with great tensions	Providers learn new skills, researchers and providers both want to work together
Sustainability	Project dies at completion of research	Some “spinoffs” are produced	Program continues	Program continues and new programs initiated

Notes: Adapted and reprinted from Social Science & Medicine, S5, Naylor P, Wharf-Higgins, Blair L, et al. Evaluating the participatory process in a community-based heart health project, 1173-87. Copyright © 2002, with permission from Elsevier.

Triangulating the numerical ratings and the qualitative interview data, a final rating was assigned by the first author into one cell of Naylor’s matrix of collaboration across the 6 domains. A trained high-level research assistant reviewed the notes and confirmed these ratings. Only minor differences emerged, which were discussed to reach consensus. To pursue the second research question, transcribed notes from the interviews were analyzed qualitatively. Transcription occurred within one hour of the end of each call. The transcripts were segmented by each of the 6 domains of the Naylor Model. Several readings of text from all the interviews resulted in the development of themes within each domain to further understand the collaborative process. Due to the nature of the qualitative data and brevity of the phone calls, the focus of the analysis was to clarify the rationale underlying each domain rating. The data were segmented by domain and theme and re-read to ensure that secondary coding was not necessary. The transcripts were then reassembled and general themes were identified as they cross-cut each of Naylor’s domains.

Participants (n = 16) included advisory committee members, clinical staff, and administrators from university and community partners. Most of the participants had been involved with the project from the beginning; however, a few did join the advisory committee after the project was well under way. Partners who agreed to be interviewed signed an informed consent and were interviewed by phone for 30–45 minutes. Partners were provided with copies of the rating scales prior to their scheduled interview. Interviews were conducted and analyzed by an experienced qualitative researcher (DB) who was not associated with the ElderLynk program.

Results

Rating scales

The rating scales and qualitative data were triangulated to estimate how ElderLynk functioned as a collaborative partnership. These estimates appear as the shaded boxes in Table 1. Specifically, the responses of participants to the probe questions were compared to the descriptions of each level of participation noted by Naylor and colleagues (2002), in combination with the degree of participation indicated by the rating scales. Based upon this process, it was determined how to best characterize the respondent’s perceptions of collaboration in the ElderLynk program. Note that selecting the degree of collaboration should not be considered to be a process of assigning a value judgment, but merely describing how the program operated over time.

Across 11 ratings, some questions (mostly dealing with the research elements of the project) could not be answered by all participants depending upon the particular role they had in the program and one participant (the consumer participant) could not answer any of the questions. This is an important finding in and of itself as an indicator of the varying levels of involvement that different participants chose within the program and the degree of to which the ideal of CBPR had been reached. Ratings were averaged across questions and across participants in order to get a picture of the total perceived collaboration.

Figure 1 illustrates the mean ratings, adjusting for missing data, for each of the dimensions of the evaluation model. Based upon the mean rating of 3.2 (SD = 0.86) for the Identification of Need domain, it is clear that most respondents perceived a high need for ElderLynk’s mission at the outset of the project. In fact, many noted in follow-up questions that over the years of the project, they had become more conscientious about these issues and would rate the perceived need even higher at the time of the evaluation. Consistent with the structure of the program, most noted that their involvement in defining the research goals and activities at the outset was minimal, but noted that this increased over time (M = 2.2, SD = 0.99). A lower rating (M = 1.7, SD = 1.14) was given to the degree of collaboration pertaining to resources. While most respondents recognized the importance of the resources ElderLynk initially was able to provide their organizations, they noted that the resources decreased with time. Questions pertaining to the research elements of the program (ie, Methodology of the Evaluation) received a moderate rating (M = 2.1, SD = 1.02); however, several individuals could not answer these questions because they chose not be involved in this aspect of the project. Similarly, the indicators used to determine success were viewed as unchangeable elements of the grant requirements and participants felt they had little say in determining these (M = 1.9, SD = 1.68). Finally, the lowest rating concerned issues of sustainability (M = 1.4, SD = 1.50). It should be noted that the data for this evaluation were collected within two months before the end of program funding.

Figure 1

Naylor ratings of collaboration.

Most partners had been involved with ElderLynk for at least 2 years; the majority had been involved since the program’s inception or shortly thereafter with clearly defined roles. Few noted that they wanted greater involvement, especially in the research evaluation components. Overall satisfaction resulted in a mean rating of 3.3 (SD = 1.12). Thus, the respondents were quite satisfied with the amount of collaboration that had occurred; however, there were several suggestions noted through follow-up questioning. We now turn to these qualitative analyses to help clarify the apparent discrepancy between the collaboration ratings and satisfaction.

Qualitative analysis of interviews

Across the probe questions, there were several themes that surfaced quite frequently, regardless of the particular topic being discussed. The most important of these themes concerned the administration of the ElderLynk program and the role of the advisory board. ElderLynk was described by one participant as an octopus, with many tentacles, and all community members were primarily concerned with their well-defined, but narrow roles. An important comment noted by several participants was the ability of the program to bring together a group of providers that traditionally did not collaborate with one another and furthermore, now that it had been accomplished, that this could be built upon to initiate future efforts. The desire for greater involvement of community members in the actual development of grant applications was an important concern that was raised by several partners.

The strength of the program and its ability to affect the outcomes that were realized can largely be attributed to the particular individuals overseeing Elderlynk and those in the community that agreed to be involved in the project. Although most participants were willing to look past personal opinions of others to work towards a common goal and the historical tensions between partners and the university, it was widely believed that if there had been more turnover in the program, it would have likely failed. The principle investigator and program coordinator were positively regarded by most of the participants, but it was also noted that these individuals, while serving different roles, were almost entirely responsible for the creation and continued existence of ElderLynk. Although the funding for ElderLynk was about to expire at the time of this evaluation and the program’s future was in jeopardy, there was significant support for the need for the program in the community.

Additionally, buy-in on the part of primary care providers was noted as a challenge that was omnipresent across the years of the project. Nearly all of the participants acknowledged the importance of presence and time in the community to gain the trust necessary for a successful initiative such as ElderLynk. Specifically, participants noted that not enough time had passed for ElderLynk to be viewed as a trusted entity in most communities. Although a common weakness of grant-funded projects is the lack of long term funding to develop these relationships, this was expected to continue to be a barrier to outreach and developing buy-in from both consumers and providers. While one respondent felt the university affiliation was a strength, others noted that academia can be mistrusted by many providers as self-interested in short-term research grants. Rural consumers may also look at the university with much intimidation, and find it difficult to accept it and its faculty as members of their communities. However, respondents also noted there are likely to be continued concerns over competition between providers, particularly if any single agency appeared to assume leadership of the program.

Comments relevant to the linking function of the program between community providers via the collaborative care model were generally positive, as illustrated by one respondent: “In a rural area, ElderLynk has strengthened the collaborative history between partners. It has made other projects easier and sustainable.” Similarly, one person noted, “It is a fact that people that have collaborated successfully in the past are more likely to do so with others in the future.” Such comments suggest a significant potential for future collaboration and more mature relationships between partners.

Although some expressed a desire to become more involved in the setting of the goals, they also noted that the research elements were of little interest to them. There was wide recognition that because this was a grant-funded project, it was not possible to significantly alter the goals of the project. One participant noted, “I had no impact on the research elements. Any changes I would have wanted to make would cause problems with the grant.” When participants were asked about their understanding of what ElderLynk’s goals were, there were about a quarter who stated that they did not really know what they were. Nearly a third of participants felt that they did not truly understand what ElderLynk was at the outset, but that their understanding developed over time. For instance, one respondent stated, “Initially they just had to move forward until we knew all of our roles.” Despite this, it was commonly expressed that respondents were satisfied with the roles they had in the project and did not desire greater input or involvement. This was a curious finding given partner’s collaboration in the development of the original grant application.

Participants were also asked about the degree of input that consumers had into the program. There was almost universal agreement that they did not know if consumer input was sought out and utilized. However, several did state with some uncertainty that they thought satisfaction data was collected from consumers, but did not know how it was used. Only one person recognized that there was a member of the community on the advisory board. It is important to further note here that the consumer representative is the one participant who could not respond to any of the rating scales, which will be discussed further below.

There was a general consensus among most respondents that ElderLynk was successful in accomplishing its objectives, but these feelings were overshadowed by disappointments at the time of the evaluation in the possible discontinuation of programmatic funding at the conclusion of the grant. One participant commented, “The program has been very successful, except for not planning for sustaining it.” A number of respondents also felt that outreach into the communities to recruit more primary care providers would have significantly improved their perceptions of whether the program was successful or not. While some were unsure if the program had grown over time, several noted that referrals and caseloads did increase.

Sustainability of the program was a concern that was frequently noted by participants. This was important in both the recruitment of collaborative partners and the continued involvement of existing ones. Participants felt that many providers in the community were unwilling to become involved in a project that was grant funded, as there is a history of such services abruptly being discontinued at the conclusion of the funding period. Although the partners recognized the benefits of grant funding as seed money to begin new initiatives, there was also a feeling of disappointment that planning for sustainability did not appear to begin until the end of the project. From the perspective of the university affiliates, issues of sustainability had been addressed repeatedly from the inception of the program, but with little reported response from collaborative partners or funding agencies. The university affiliates believe that this was because the partners would not “step up to the plate” until the university turned the program over to the community to determine its fate, which is what happened shortly after this evaluation took place.

In summary, the themes that arose from the interviews described many strengths of the ElderLynk program and also highlight areas that may now be possible in future work in the community that were not present when the program first began. The most important concern of respondents was for the program to have a sustained presence in the community.

Discussion

The purpose of this manuscript was to report the results of an evaluation of the collaborative process used with the ElderLynk program. In this regard, we asked: 1) community partners to rate their perceived degree of collaboration with university affiliates; and 2) within each of Naylor’s six domains, how satisfied were participants with this degree of collaboration, what potential was there for changes in the future, and what suggestions could be offered to improve both the collaborative process and the project outcomes. A second and related purpose for this manuscript is to discuss the success of the methods employed to conduct this evaluation, noting strengths and weaknesses, and offering potential solutions for similar efforts in the future. Each of these foci will be discussed in turn in the following sections.

Evaluation of collaboration

ElderLynk was attempting to accomplish two goals: 1) link existing mental health and social service agencies with primary care providers, where the majority of the elderly enter the healthcare system (Levkoff et al 2004), and to 2) educate students in the health professions, practitioners, and the general public about senior mental healthcare and quality-of-life issues.

Partner membership had initially begun several years before ElderLynk was funded, but there was much dissention about resources and project goals on the earlier project. Focusing upon geriatric mental health issues was much more agreeable to partners and laid the groundwork for the success of the partnership. It was difficult to recruit consumer participation on the advisory board, as only one member maintained consistent commitment. Upon reflection, consumers were very satisfied with interactions with ElderLynk personnel (data not reported here), but were most concerned with issues of stigma and not having to pay for services out-of-pocket. It is believed that due to issues of confidentiality and stigma in rural settings, it was not only difficult to recruit consumers to advisory boards, but it is also true that it was challenging to secure participation of those that could adopt a larger community perspective, as opposed to immediate personal concerns. These also became issues with sustainability of the program. Further work needs to be done, however, to identify methods that could be used to enhance consumer desire and ability to serve in these important roles.

Respondents felt strongly that time and presence in the community were essential to future work, in order to develop greater trust and buy-in from providers and consumers. Several expressed concerns that there would be a need for the program to be recognized as a member of the local community, and not an academic entity, although the reputation and influence of the university affiliation was seen as having some benefits with regard to securing resources. This would be very helpful with not only recruiting more primary care practitioners, but also consumers.

The rating scales developed for this evaluation, while departing from the original methodology of Naylor’s model, reflected a mixed view of the collaboration achieved through the program. The highest rating for collaboration was for Identification of Need, which was surprising, given that the initial grant originated from the university with little community input. At the other extreme was the low rating for Sustainability. The other four domains pertained to the research elements of the project and all received similar ratings, including defining the research goals and activities, resources, program methodology, and indicators of success.

These findings need to be understood within the larger historical context of the project and geographical region, in addition to the insight gained through the analysis of the key informant interview data. ElderLynk was the first successful attempt in this part of Missouri to bring together a diverse array of facilities that address the mental health needs of older adults. Due to the fact that collaboration required securing of monetary resources, the extended timeline usually required for the development of a fully collaborative approach was not possible, as the project had to move forward to pursue the grant objectives. Additionally, as noted above, the majority of the community partners did not want to be more involved in the research elements of the program (which are central to Naylor’s and colleagues’ [2002] evaluation domains). Either in the absence of research expertise and/or desire for a more active role in these elements, it is important to consider the potential detrimental consequences to collaborative relationships to force greater participation in the technical aspects of a research endeavor if it is not desired. Most of the participants were very satisfied with not having to become involved in the research elements of the project. Does this negate considering this project CBPR? We hold that is does not. It is likely that this involvement will develop over time and if it does not, and is not desired by community partners, then the partnership should still be considered as potentially successful given other elements of this multifaceted concept. This will be discussed further below.

In explaining the results reported above, the Need rating likely reflected the fact that those in the community who did not perceive elder mental healthcare as an important area would not have joined the project. Additionally, it may have been the case that these partners were also making this rating judging the current perceived need, regardless of what was actually felt at the outset of the project. This was suggested in the qualitative results, where participants emphasized concerns over continuing the program to facilitate access to mental health services.

The low sustainability rating was not surprising given that this evaluation occurred while partners were dealing with the reality of the fact that the funding for the project was about to expire. Although university affiliates emphasized that they began discussing program sustainability at the outset of the initiative and continuously during implementation, these efforts were not recalled by partners during the interviews. It is interesting to note that subsequent to this evaluation, and perhaps precipitated by this evaluation, as mentioned earlier, ElderLynk was adopted by the Northeast Missouri Rural Health Network, a non-profit organization, run by some of the core partners originally involved in ElderLynk. Thus, although the program was not maintained in its original form, it maintained many of its core components and was sustained after grant funding had been discontinued, in part through carry over funds generated through clinical income.

Reconciling differences in memory and perspective posed unique challenges for the evaluation, especially with regard to sustainability, as information from both perspectives (university and community) were required and often conflicted. This posed challenges for both implementation of such a program, as well as its evaluation. Communication is critical to ensuring that all parties are “on the same page” with efforts being undertaken and how those efforts can be evaluated. Similarly, evaluating collaboration must include partners from both groups of stakeholders to examine potential differences in such perspectives. These differences could result from natural difficulties in communicating between the two groups (three if consumers are explicitly considered as well). It has often been noted that CBPR should call explicit attention to expected and likely differences in understanding due to the need to learn how to effectively communicate between partners (Fisher and Ball 2003).

Delving deeper into the interview data, it was apparent that regardless of the collaboration ratings, most participants felt ElderLynk was successful in achieving its mental health-care objectives and they were fully satisfied with the amount of collaboration that existed. Although some expressed that they would want to be more involved in certain elements of the program, these persons were a small minority.

Evaluating CBPR

The evaluation of the program resulted in a somewhat mixed view of the collaborative process if 1) only examining rating scales and 2) if one only adopts the “ideal” definition of CBPR. A more flexible conceptualization of CBPR should be considered. CBPR can be viewed as a fluid approach to accomplishing research collaboration between universities and community members. The importance of time and presence in the community for “outsiders” to gain the respect of providers, and hence conduct a successful research project, was noted by respondents. These elements are also true to forming a CBPR advisory board to guide the development of research agendas long before funding is even pursued.

CBPR is often a double-edged sword in that initiating and continuing a collaborative venture often requires resources that ironically needs grant funding even to begin in many settings. Thus, to pursue CBPR, ElderLynk’s university partners successfully acquired funding to bring together members of a community around broad goals that were translated into specific objectives as a collaborative venture. This latter outcome was viewed by respondents as a major accomplishment in rural Missouri; however, also noted was that they would not likely have been motivated to begin the project or continue active involvement if it was not for the resources that became available as a consequence of the grant funding. Researchers and clinicians alike need protected time to engage in CBPR initiatives, especially in their initial formation, which often requires significant time commitments.

Evaluating CBPR initiatives can be a particularly challenging task due to the need to consider multiple sources of information. One may approach this examining the outcomes sought after by a particular research project, but this does not indicate how well the participatory process was carried out. Other than theoretical guidance, there exists little in the literature to suggest methods to examine this participatory component. We adopted Naylor and colleagues’ (2002) model for the preset evaluation. This method, however, proved to be rather difficult to replicate with ElderLynk, given the structure of the program and the need to qualify the results with the developmental level of the relationships.

First, the categories of participation (see columns in Table 1) are somewhat difficult to distinguish from one another. This is partially the reason why we chose to construct rating scales indicating how much collaboration existed with each domain. By combining these rating with the comments of respondents it was possible to classify the participation level.

Second, the model does not allow for an indicator of the degree of satisfaction with the level of collaboration between partners. Most of the respondents were very satisfied with the participatory process, explicitly stating that they did not want a greater role in various domains (especially the research aspects of the project). We found this critical to the final judgments about the level of success achieved in working as a collaborative group.

Finally, this model does not assess the level of readiness to assume collaborative roles greater than what was achieved through the project. It does not provide a temporal component that would indicate the developing ability of both university and community partners for active input into all aspects of the endeavor. This component is critical and should be assessed on an ongoing basis throughout the implementation of CBPR.

Consequently, we recommend that while the domains of the Naylor et al (2002) model are valid, they should be more clearly distinguished from each other and the execution of an evaluation utilizing this model would be significantly enhanced by measures of readiness for change and satisfaction ratings with the level of collaboration attained. Finally, an assessment of the developmental process – across several evaluations of a relationship – would allow for theoretical models of how relationships of this type change over time, which would significantly contribute to the literature about how to evaluate CBPR ventures.

This evaluation has several limitations. First, through our alterations of the method of assessing collaboration, there may have been a reduction in the ability to compare the utility of the Naylor model to our use of it. We believe this was minimized by the use of several questions to assess those domains which were less clearly defined in the original model, and as a result of our use of qualitative key informant interviews for respondents to explain their ratings. Second, as with many implementation projects, issues of local history and culture mandate very unique applications to the execution of a program. This limits, in addition to the qualitative nature of the assessments, the ability to speak to potential generalizability of these findings to other rural collaborative partnerships; nonetheless, the principles used to define collaboration at the outset of the program and through its development, in addition to those used for the evaluation are consistent with those in the literature concerning CBPR.

Summary and conclusions

Results from this evaluation demonstrated considerable success of ElderLynk in establishing a CBPR project to improve mental healthcare in rural Missouri. Areas of collaboration that appeared to be less than optimal were in actuality at the level most participants desired at the outset of the project; those that were below expectation were areas where continued work with the partners were ripe for greater community involvement. Efforts to evaluate CBPR projects need to consider the degree of collaboration in conjunction with the ability and readiness of the partners to truly contribute to all aspects of a project and their desire to do so.