The Nursing Leadership Institute program evaluation: a critique.

A theory-driven program evaluation was conducted for a nursing leadership program, as a collaborative project between university faculty, the nurses' union, the provincial Ministry of Health, and its chief nursing officers. A collaborative logic model process was used to engage stakeholders, and mixed methods approaches were used to answer evaluation questions. Despite demonstrated, successful outcomes, the leadership program was not supported with continued funding. This paper examines what happened during the evaluation process: What factors failed to sustain this program?

Introduction

Frontline or clinically based nurse leaders are typically responsible for the management and coordination of patient care. Given the acuity of today’s patients and the complexity of our health care systems, leadership preparation is becoming increasingly important.1 Effective nurse leaders have been linked to improved quality and safe patient care delivery.2 Health leadership research has demonstrated that effective nurse leaders can create supportive work environments where staff are more productive and more engaged.3,4 Given the importance of effective leadership to health care organizations, a variety of postgraduate (ie, postbaccalaureate degree preparation) health leadership programs exist.5 Regardless of discipline, most health leadership program curricula are similar, covering leadership and management concepts such as effective leadership styles, resource utilization, communications and conflict resolution, and team-building.6 Despite multiple leadership development offerings that claim to improve leadership performance, there is a lack of theory-based evaluation and research related to leadership preparation.7

The purpose of this paper was to describe a theory-driven (TD) program evaluation of one nursing leadership development program for frontline nurse leaders that we, the authors (university faculty with evaluation/research expertise), conducted. We designed the course for novice nurse leaders. Although it was specifically designed for this population, the curriculum includes content that is commonly found in other health leadership programs that focus on novice or first-time leaders.8 We believe this case example will inform and guide others who are involved in health leadership development. In addition, we critique the effectiveness of our program evaluation strategy.

The NLI for frontline leaders

Effective nurse leadership has been recognized as an essential component of quality, safe practice environments.1 In this section, we provide a brief overview of a province-based nursing leadership development program that was created in response to a need for more effective frontline leadership across health care sectors (eg, acute care, community, and residential care). More detailed descriptions of the BC NLI and program outcomes can be found in the published literature.8–11 The concept of the BC NLI was based on recommendations from the Canadian Nursing Advisory Committee and a provincial needs assessment that included extensive interviews with midlevel and executive-level nurse leaders.8 These administrators identified a clear need for frontline leadership development. In nursing, there are typically three levels of formal leadership within health care organizations. Frontline leaders are usually union members who have clinical expertise and are charged with making unit-level staffing decisions and monitoring the flow of patients and resource needs. Midlevel leaders, also known as nurse managers or directors, are not union members, and their “excluded” status permits them to hire, fire, and oversee disciplinary activities. They are engaged in departmental operations (eg, pediatrics department) and have financial and budgetary responsibilities. Executive-level leaders are engaged in long-term planning and strategy, and they represent nursing with respect to organization-wide decision-making. Although these three levels of formal roles and responsibilities are equally vital, frontline leaders have closest contact with direct care staff, patients and their families, and they are a critical connection between direct care provision and the rest of the organization.3

Based on Canadian Nursing Advisory Committee (national) recommendations and the provincial needs assessment, the BC Ministry of Health Nursing Directorate collaborated with the BC health authorities’ Chief Nursing Officers (CNOs) and faculty from the University of British Columbia (UBC) School of Nursing to develop, implement, and evaluate a provincial NLI for novice frontline nurse leaders. An advisory council was organized with representation from key BC stakeholder groups: the nurses’ union, the CNOs, the Ministry of Health, and UBC School of Nursing. The advisory council helped craft the major program goals: a) to prepare effective frontline nurse leaders to meet practice environment challenges across BC; b) to provide necessary supports/resources to promote new leader retention; and c) to pave the way for ongoing leadership development (eg, from frontline to midlevel leadership).

The NLI program was offered three times annually over the course of 4 years (2006–2010). At each session, approximately five nominated nurse leaders attended the NLI from each of the seven health authorities. Over the funding period, over 400 novice frontline nurse leaders successfully completed the 1-year program. Participants were nominated by their direct supervisors and CNOs to attend, and the nominating criteria included less than or equal to 3 years of leadership experience, and leadership potential, as evidenced by interest in taking on leadership roles and responsibilities.

A unique aspect of the NLI was its use of empowerment theory to guide curricular content, delivery, and evaluation.9 Over two decades of research has been conducted on the nurse empowerment process, a process associated with enhanced outcomes for nursing staff and patients. Structural empowerment (SE) refers to staff access to critical information, supports, resources, and opportunities associated with quality, safe care delivery.12 Leaders’ formal authority enables them to grant staff access to important organizational empowerment structures. Leaders’ use of SE strategies has been linked to positive staff and patient outcomes.2,13,14 Leaders can be taught SE strategies, and they can also be taught how to use other leader-empowering behaviors (LEB) that are associated with positive staff outcomes. There are five leader-empowering behaviors categories that include sharing a common workplace vision and inspiring confidence in staff.15 The theoretical premise of the NLI was that empowerment strategies (SE and leader-empowering behaviors) taught via experiential learning activities would lead to positive staff outcomes (eg, increased organizational commitment).10

The leadership literature has demonstrated how active learning strategies work best with busy adult learners.16,17 The NLI learning activities included: a) interactive group exercises and individual critical reflection during a 4-day residential workshop; b) a 1-year innovation project of importance to the leader’s organization; c) mentoring and coaching supports from senior leaders within the leader’s organization; and d) online NLI networking via an electronic community of practice (housed on the UBC School of Nursing website).

TD program evaluation

TD program evaluation is becoming increasingly popular for programs and interventions based in complex, unpredictable, real-world situations, such as health care settings. TD evaluation is defined as “… an explicit theory or model of how the program causes the intended or observed outcomes and an evaluation that is at least partly guided by this model”.18 Traditional research approaches, such as randomized controlled trials, do not apply in contexts where there is “natural heterogeneity”.14 Researchers and program evaluators need to distinguish between low-variance and high-variance situations. Variance pertains to context, content, application, and potential outcomes.14 In high-variance situations (eg, health and social service programs), TD program evaluation is a well-recognized alternative (to traditional methods) for studying the impact of these programs on theoretically derived outcomes. Given the high variance associated with health leadership, we chose to use TD evaluation to study outcomes related to the NLI. A variety of resources are available to assist researchers and program evaluators who use the TD approach. We chose to use a program evaluation tool kit developed for the Ontario Ministry of Health and Long-Term Care.19 The tool kit evaluation process is depicted in Figure 1. This tool kit consists of five steps: 1) focusing the evaluation, 2) choosing the most appropriate methods for answering evaluation questions, 3) developing or modifying data collection tools, 4) gathering and analyzing data, and 5) using the answers to the evaluation question to help make decisions about the program. Each step has multiple components, and there are clear deliverables or outcomes at the end of each step.

Figure 1

The tool kit steps.19

As will be evidenced throughout this paper, collaboration between researchers and stakeholders was of paramount importance for the successful development, implementation, and evaluation of the NLI. TD evaluation lends itself well to collaborative projects because those resources associated with this approach can simplify and visually, schematically represent a complex evaluation process – providing diverse stakeholders with opportunities to witness and engage with the “steps” of evaluation. The following sections outline the key considerations associated with the major steps of the tool kit approach.

Step 1: focus the evaluation

TD evaluation is typically focused on the process of program delivery and the impact or outcomes related to the program. These different foci are characterized in the program evaluation literature as “program process theory” and “program impact theory”.20 The “theory” terminology accentuates the importance of using best evidence or theory throughout the design, implementation, and evaluation of a program. Researchers can use rich information from evidence-based sources, such as health services and organizational development literature, to help stakeholders determine how and why a program meets its goals. In cost-conscious times, this information helps ensure program accountability.15

Logic models are often used to visually present the process and impact components of program evaluation. They provide a concrete, comprehensive overview for stakeholders – demonstrating how all the pieces fit together to deliver a program with predicted deliverables and impact. Figure 2 is the NLI logic model that we collaboratively constructed with our stakeholders. The processes are depicted as inputs/resources and outputs/activities, while the outcomes are displayed as short-term outcomes and longer-term impact.

Figure 2

Collaborative logic model for the BC NLI.

Notes: (+) signifies an NLI enabler; (−) signifies an NLI barrier.

Abbreviations: BC, British Columbia; NLI, Nursing Leadership Institute.

There are many logic model formats available online, and we chose to use one from the University of Wisconsin–Extension.21 After a focused review of the leadership development literature, the UBC faculty (including authors) hosted a work day to collaboratively develop the NLI logic model, with advisory council support. This work day was also used to flesh out desired curricular content for the NLI.

A collaborative logic model process can be used to establish essential program features from the perspectives of different stakeholder groups.22,23 Our logic model format clearly explicates proposed causal links between inputs, outputs, and outcomes, and it includes program assumptions and external factors – important considerations that are not always explicitly represented by other logic model formats. The format also works well with “if … then” statements, and we used a series of these statements to further accentuate logic model links:

If we have certain resources needed to support the NLI, then we will be able to carry out specific activities for a given participant population.

If the participants receive the planned program activities, then they will gain certain competencies, particularly leader empowerment strategies and change management strategies.

If they effectively carry out these strategies, then the anticipated impact will be planned project deliverables, leader retention and evidence of succession planning, and increased positive staff attitudes.

The collaborative logic model process began with an advisory council discussion of key assumptions and external factors related to nurse leadership development. Logic model assumptions refer to stakeholder beliefs about a program and how they think it will work within specific contexts.21 During logic model construction, members of the advisory council had a difficult time clarifying their assumptions about the NLI; they had an abstract sense that the program would be a success if we used approaches associated with other successful leadership programs, but they had not thought about how successful leadership development would specifically impact their staff, patients, and health care organizations. The advisory council opted to use assumptions derived from the literature evidence, and these assumptions were generated by UBC faculty. Tourish found that organizations value leadership development but that they rarely consider how leadership will have a measurable impact on their organizational success.24 Tourish cites one US survey study where 78% of American businesses had no evaluation strategies in place to justify their investment of time and money on leadership development.

External factors are those environmental influences that potentially or actually influence program implementation. The advisory council identified some important external enablers and barriers with respect to NLI success. A notable enabler was previous provincial health care restructuring that had eliminated leadership positions, resulting in gaps in credible leadership. Restructuring had particularly affected the frontline level and in some instances, the midlevel managerial level. All the members of the advisory council were aware of significant leadership deficits due to recent restructuring and an ongoing lack of systematic succession planning. Stakeholders also acknowledged the complexity of health care delivery and the many challenges facing health care leaders and staff. They felt that the complexity of the health care environments was an enabler for leadership development: “You need savvy leaders to negotiate complex care delivery.” A barrier that dominated discussion was budgetary restrictions. From the outset of NLI planning, advisory council members were concerned about the expense of the program and NLI sustainability beyond the prescribed funding period: “It’s all well and good to invest in leaders now while there is money given for it, but will that continue?”

The advisory council reached consensus about identified inputs, outputs, and outcomes. As part of the formal NLI collaborative agreement, the Ministry of Health agreed to pay for residential workshop accommodations, travel, and learning materials. The CNOs and the health care organizations of selected NLI leaders agreed to cover costs related to leaders’ release time for residential workshop attendance, and for 1 work day per 2-week pay period, for project work. The leaders’ health care organizations also agreed to provide necessary project resources and a dedicated senior mentor to support novice leaders’ project work and overall professional development. The UBC faculty contributed their time for NLI development, implementation, and evaluation, and faculty conducted the residential workshop activities with assistance from senior leaders from the provincial health regions. The faculty also served as advisors for senior leader–novice leader mentor–mentee dyads, and they were responsible for managing the online NLI community of practice. Overall, there was excellent stakeholder collaboration related to resource-sharing and planned activities, (ie, inputs and outputs).

The UBC faculty identified short-term outcomes associated with NLI learning activities, such as empowerment and change management strategies. In the leadership literature, these learning outcomes are considered critical relational and operational competencies that contemporary leaders must possess.25 The UBC faculty also created long-term (impact) outcomes that could be operationalized and evaluated as determinants of NLI success. The advisory council members concurred with the content in these components of the logic model, but they did not play an active role in determining outcomes. After working through step 1 components of the toolkit, the key evaluation question of interest to the stakeholders was: “What inputs and leadership program activities result in the expected outcomes?”

Steps 2 and 3: selecting methods and developing tools

The first component of this step focuses on stakeholder expectations.19 Stakeholders expected that: a) resources would be committed by organizations and utilized as planned; b) activities would be meaningful to participants; c) only novice nurse leaders would be elected to attend the NLI; d) participants would use more empowerment strategies after attendance; e) participants would use the change management strategies in their innovation projects; f) participants would complete their innovation projects; g) participants would either stay in their frontline leader positions or advance to the next level; and h) staff of participants would express more positive attitudes about their work environment. We discussed staff turnover expectations, but stakeholders admitted that: “Turnover is so complex. Lots of other things are affecting turnover right now – not just what leaders do”.

According to Chen, “The need for program theory clarification and holistic assessment usually requires the use of mixed methods”.15 Sophisticated analytic techniques are often needed with TD evaluation.20 We selected a variety of methods (eg, interviews, surveys, etc) to answer the evaluation question: Do TD NLI inputs and outputs result in predicted outcomes (ie, short-term outcomes and impact outcomes)? We (evaluators/researchers) acquired research funding to supplement the original NLI evaluation budget. Our research methods and analytic approaches augmented our evaluation strategy: research rigor enabled us to more adequately and fully establish links between program resources, activities, and outcomes. Research funds, for instance, enabled us to conduct a longitudinal quasi-experimental study from 2006 to 2010: Participants who attended the NLI were compared with a similar population of novice nurse leaders who did not attend the NLI. The staff of NLI participant and nonparticipant leaders were also compared before and after the NLI program. We administered valid, reliable survey tools to leaders and staff before participants attended the NLI and 1 year after participants’ completion of the NLI program. We chose this 1-year interval (postprogram attendance) to better assess the true impact of the NLI. Our survey tools included demographic information, and they measured leaders’ self-assessments and staff assessments of the leaders with respect to levels of empowerment and use of empowerment strategies. We chose empowerment survey tools because the NLI curriculum was based on empowerment theory. Organizational commitment was chosen as a measure of positive staff outcomes because commitment is directly linked to turnover in the nursing research literature.26

Some evaluation/research methods were for NLI leaders only. A core component of the NLI curriculum was devoted to change management strategies – to solidify their change management skills, NLI participants were required to conduct innovation projects of significance to their organizations over a 1-year period of time. We conducted content analyses of NLI leaders’ project reports to determine applications of change management strategies and project completion successes. We developed a career tracking tool for NLI leaders to measure leader retention and evidence of succession planning (ie, advancement to other leadership positions within their organizations), and a satisfaction survey with Likertstyle and open-ended questions (eg, “What I liked most about the NLI …”). We also conducted purposeful interviews with NLI leaders to better understand the impact of the program on their leadership goals and career directions.

Step 4: gather and analyze data

Qualitative data from NLI leader interviews and project reports were analyzed using the NVivo software program, and quantitative data from leader and staff surveys were analyzed using Excel and SPSS software packages. Missing survey data were coded accordingly, and data were further cleaned when necessary. Content and thematic analyses were used for qualitative data, and descriptive (eg, frequencies) and inferential statistics were used for quantitative data analyses. More detailed descriptions of analytic approaches are in our published research papers.9–11,27–30

The findings

The majority of our research findings have been published, and these papers provide in-depth descriptions of our findings.8–11,27–30 We will summarize some of the highlights from our TD evaluation. The demographic data revealed that the intended leader population was recruited for the NLI: prior to attending the NLI, 87% of participants had received no formal leadership training. In addition, there was representation of novice leaders from every geographic region and every health sector (eg, acute care and community) per cohort.

There was a 74% return rate for participant satisfaction surveys that documented either “above average” or “excellent” ratings for the NLI experience. The mix of leaders per cohort from across the province was felt to be an important program asset: “It was so enlightening to learn with other leaders in BC – to see that you’re not alone and the same problems are everywhere”; “I acquired so many new ideas from people outside my own specialty program and health authority”.

In interviews with participants, NLI leaders indicated that release time to work on projects was needed, but “it rarely happened”. “I ended up doing most of the work off the side of my desk.” The same was true for project resources. Participants noted that project resources were often withdrawn or not available because of shifting organizational priorities. “It started out well, but then there was another priority.” Although some organizational supports were a recognized problem (ie, project release time and resources), the majority of NLI participants noted that their assigned senior mentors were dedicated to their professional development and would make time to meet with them on a regular basis (typically monthly), in-person or by phone. “It was always motivating – to have those little ‘pep’ talks.”

With respect to NLI activities, all the activities were highly valued except the online networking component of the program. Online networking was underutilized due to technical issues and time. “When I had time, I spent it on my project. Trying to navigate the site and find stuff was not helpful.”11 Conversely, participants stated that the in-person interactive sessions with peers were “amazing learning experiences”. Participants also appreciated the UBC faculty and senior leader presenters who reinforced the purpose and relevance of the conceptual empowerment framework at the beginning of the 4-day residential workshop and before and during every related learning exercise. Overall, the NLI participants valued the empowerment framework and recognized how it “helped things to ‘gel’ together for me”. Participants identified how empowerment-focused case scenarios, guided reflection questions, and workshop resources were “helpful to reinforce what empowerment ‘looks like’ and how to ‘do it’”. For all of the participants, the innovation project was the most valuable learning experience. “It was so important to do a project in my own organization with a senior leader. I learned so much about change management but especially, about my organization.” Each NLI participant completed a 6-month and a final 1-year project report as part of their attendance requirements. Content analyses of final project reports showed that all the participants were able to use change management strategies in their project work. Every participant completed an environmental scan, a stakeholder analysis, and an action plan with timeline and budget: 90% of innovation projects were successfully completed. We found that the majority of the projects were related to staff retention, staff professional development, team communications and collaboration, and care delivery model redesign. The project work mirrored health authorities’ priorities. By the end of the funding period, for instance, care delivery model redesign was taking place in more health care regions as a way to control costs and increase productivity and efficiency.28

The career tracking survey was sent out electronically at the annual anniversary dates from NLI program attendance. Over a 4-year period, approximately 80% of participants stayed in leadership positions, and of these 80%, almost one-third had moved into midlevel positions over the evaluation period. In these instances, participants reported that their senior mentors had groomed them for these position moves. Approximately 20% of participants had returned to direct care positions due to “burnout”, “very heavy workload”, “lack of organizational support”, and “unrealistic staff expectations”.

The pre- and post-NLI program leader and staff surveys used validated tools to assess leaders’ use of empowerment strategies and changes in staff organizational commitment. Leaders and their staff reported the use of more leader empowerment strategies after NLI attendance.10 There was a significant, positive increase in staff organizational commitment but only for those staff who indicated some level of commitment at the time of the pre-NLI survey administration. We surmised that if staff nurses were already cynical or burned out in their work environment, leader interventions would not matter.27

To summarize, mixed methods approaches helped us determine the answer to our key evaluation question: we were able to show what inputs and outputs were most related to expected outcomes. Novice leaders, for instance, benefited from having dedicated project release time and resources to conduct innovation work within their health care organizations, but the support of senior mentors was critical to their success. All the NLI activities created a learning synergy for participants except online networking. The project work, in particular, was most valued because participants, with mentor support, learned how to navigate within their own health care systems. In addition, the research study was able to demonstrate the importance of using an overarching empowerment framework to guide nurse leadership development.

Step 5: make decisions

We prepared an evaluation report for the advisory council based on the original stakeholder expectations. The findings were organized to highlight how expectations were met by the NLI. In addition, different business case options were presented for potential NLI delivery methods: a) the original structure with a 4-day residential workshop; b) delivery of content within each health region; or c) virtual content delivery (eg, Webinars, wikis, and blogs). These business case options were prepared at the request of the stakeholders, given their budgetary concerns. Despite clear indications that the NLI met stakeholder expectations, the program was not financially supported after the end of the contracted funding period. Since then, the curriculum has been shared with individual organizations who have adapted the content for in-house workshops, and UBC offers the curricular content and directed study project work in a graduate-level nursing program.

What happened?

Looking back at the process, from the very beginning, there were no explicit stakeholder assumptions or expectations about how the NLI would serve as an actual return on investment (ROI) within their health care organizations. As discussed by Tourish, “establishing measurable and repeatable ROI for leadership development is the wave of the future”.24 Health care administrators are expected to demonstrate concrete benefits from incurred costs, and leadership development is quickly pared from organizational agendas when it is conceptualized in abstract terms rather than measurable terms that can be analyzed and evaluated.31 With respect to the NLI, the advisory council settled for literature-based, faculty-driven assumptions and abstract expectations about the value of leadership development. Expectations did not follow the “specific, measurable, actionable, relevant, time-related” (SMART) approach stipulated in the Porteous et al program evaluation toolkit.19 Program success was based, for instance, on a statistically significant increase in leaders’ use of structural empowerment. We did not know, therefore, the specific amount of change that would satisfy stakeholders with respect to NLI outcomes.

We, the researchers, took over the responsibility for major steps in the evaluation process, relegating stakeholders to passive participants. Stakeholders acknowledged the many merits associated with the NLI, such as leader retention and advancement and increased staff organizational commitment, but these positive outcomes were not translated into concrete financial returns for stakeholders dealing with budgetary pressures. Stakeholders were eager to invest in leadership development when extra Ministry funds were available, but leadership development was seen as a cost versus an investment when funding ended. In our province, health care administrators are expected to provide business cases related to resources and activities for programs and services. To be current and competitive, leaders must know how to develop viable business plans.32,33 A cursory search of the health care literature details how to develop business cases for everything from electronic medication records 34 to management of patients with complex conditions,35 to employee wellness programs.36 We developed business case options for NLI delivery methods, but we did not collaboratively develop business cases for tangible program outcomes of importance to the stakeholders’ health care organizations. Working with stakeholders, we could have costed out the ROI related to leader retention and leaders’ successful completion of critical innovation projects. Ironically, the NLI curriculum included business case development. Ideally, UBC faculty should have included a cost-benefit analysis as part of NLI leaders’ project work. In hindsight, as researchers, we became absorbed in the theoretical significance of our findings (the research questions) our research focus overshadowed the significance of the evaluation question from our stakeholders’ perspectives. True collaboration and stakeholder engagement waned over time as we paid less attention to council needs and concerns. Formative process evaluations consisted of yearly summaries of participants’ satisfaction surveys – saying laudable things about the NLI. Advisory council members went through the motions of reviewing our NLI performance reports, but monthly meetings quickly shifted to quarterly and to annual meetings with little discussion or proactive planning for “next steps”. A better approach would have been to use developmental evaluation from the beginning of the NLI project. With developmental evaluation, key stakeholders are an integral part of using real-time data to inform ongoing decision-making and program adaptation.37 There was early evidence, for example, that NLI leaders and their mentors were encountering problems with cuts to project release time and resources. Rather than addressing these concerns in a timely fashion, we normalized troublesome evidence and did not share these concerning issues with advisory council members as a part of process evaluation.

We believe that our most critical mistake was the melding of evaluation with research. Although we required research funding to carry out a complex TD evaluation, we did not clearly differentiate between our evaluator and researcher roles and responsibilities – a common pitfall.38 Coryn et al describe similar instances of TD evaluation where conflict of interest arose among evaluators who were researchers interested in testing propositions derived from their own disciplines.20 We should have foregrounded potential evaluator/researcher conflict of interest throughout the steps of the program evaluation toolkit. For example, during step 2 in the tool kit, we had opportunities to collaboratively establish specific stakeholder expectations with the a) advisory council stakeholders (macro-level); and b) with the individual health care organizations (microlevel) that were responsible for resource investment in their leaders’ development.

At both levels, we should have worked with stakeholders and business analytics and human resource experts to create explicit ROI assumptions and expectations with respect to leadership development. An example is staff turnover. Due to the complexity involved in using “hard” human resources data, the advisory council agreed to use a more abstract measure – staff organizational commitment – to signify successful NLI outcomes. We influenced this decision because we had a standardized tool to measure organizational commitment, and we knew that in the leadership literature, there is a conceptual link between effective leadership and staff organizational commitment and, in turn, decreased staff turnover. Although this was an easy solution for us as researchers, the health economics literature indicates that it is possible to determine the cost of staff turnover to a health care organization. Turnover costs include recruitment, training, and replacement costs related to vacant positions.39 It would have been possible, therefore, to discuss staff turnover savings with respect to leadership development investment at the macrolevel and at individual organizational levels. The empowerment bar would have been raised by giving stakeholders the critical language and tools to demonstrate real ROI.

Given the diversity of workforce and organizational factors, it should be the responsibility of health care organizations to determine what effective leadership means in their own, local contexts. When organizations are tasked with determining their own ROI for leadership investment, they are more engaged, and the likelihood of a culture shift (valuing leadership) increases over time.40 When formal ROI analyses are not feasible because of the variables involved and the data to be collected, it is still important to provide a rich description of the organizational context before and after leader development; in these instances, in-depth case studies may be the next best substitute for a business case.41

As discussed by one program evaluator, going through the motions of modeling is not sufficient to truly engage stakeholders and gain their support.42 In our case, everything became researcher driven – from the start of the evaluation process. We assumed that literature evidence would support the importance of leadership development – the types of data we collected and reported confirmed the need for the NLI from our perspective. As a result, it was a disappointing surprise to discover no ongoing monetary support for the NLI, despite demonstrable, TD outcomes.

Conclusion

There are many ways that we could have promoted a more effective, collaborative evaluation process. From the beginning of the project, we subtly shifted into researcher mode and lost awareness of gradual stakeholder disengagement. There were many missed opportunities, particularly opportunities to create meaningful assumptions and expectations with stakeholders at the advisory council level and the individual health care organization level. Assumptions and expectations, nebulous from the start, were irrelevant to stakeholders faced with complex health care challenges. We missed opportunities to work with our stakeholders, human resource experts, and economists to develop macrolevel (advisory council) and microlevel (organizational) business cases – business cases that could have established ROI from leadership development. Troublesome signs were overlooked, such as lack of sufficient organizational support. External factors, particularly budgetary restrictions, overrode the apparent successful outcomes of the NLI. No doubt, the NLI was a valuable experience for those participants who attended, and the immediate impact of these leaders within their local work environments was positively perceived by staff. But the evaluation question remains as to whether these changes were seen as a cost or an investment by key stakeholders.