Evaluating the Implementation of Ontario's Organ and Tissue Donation Physician Leadership Model: Mapping a Way Forward.

BACKGROUND: The demand for solid organ transplantation has spurred countries around the world to search for innovative policies and practices to increase the supply of organs. Spain has become a global reference point for organ donation with the highest transplantation rates. In Ontario, Canada the Ontario Trillium Gift of Life (TGLN) has sought to replicate some of the successes in Spain. In particular, TGLN's implementation of the Physician Leadership Model has been viewed as a promising strategy to improve donation conversion rates.
OBJECTIVE: The objective of this study was to evaluate the implementation of TGLNs (TGLN) Physician Leadership Model by examining critical implementation process variables (education/training, communication, satisfaction, participation and reach).
METHODS: This mixed-method implementation evaluation included data from all members of the Physician Leadership Model including the Chief Medical Officer, five Regional Medical Leads (RMLs), and the 52 Hospital Donation Physicians (HDPs). Social Network Analysis (SNA) surveys were sent to all 52 HDPs and yielded an 85% rate. Analysis included constructing sociograms and qualitatively analyzing interviews.
RESULTS: TGLN's PLM was poised for success by utilizing the existing RMLs' network as a foundation. The social network analysis measures, particularly participation and reach, indicated the PLM was quite dense (ie, the degree to which members are connected) at baseline. HDPs reported communication to be facilitated by their connections to their RMLs. Early evaluative data indicated that lack of education and training was viewed by HDPs as a barrier, and thus more capacity would need to be directed to this issue. Overall, HDPs reported that various intended outcomes were being met.
CONCLUSION: We have demonstrated that an implementation evaluation helps us to understand which elements of the PLM were successful and which elements required immediate attention. This evaluation helped to highlight the successes and challenges in implementing the TGLN Physician Leadership Model in Ontario. Social network analysis of publicly funded capacity building systems has been identified as a promising area for health program evaluation to answer questions at a system level, such as identifying service provisions among information exchange networks and ultimately better health care.

Introduction

The demand for organ and tissue donation has become a growing concern worldwide. Globally, governments are searching for innovative methods to assist in improving donation rates.1,2 Currently, Spain occupies a privileged position as having the highest deceased donation rates ever recorded for a large country.3 To a large extent, successes can be attributed to the Spanish Model of Organ Donation and Transplantation.4,5 A core principle of the Spanish Model is a systematic and organized approach to the processes of deceased donation. Coordination of donation activities has been conceived and structured at three different but interlinked levels: national, regional, and hospital level. In general terms, the hospital level coordination is represented by a network of officially authorized procurement hospitals that are directly in charge of effectively developing the deceased donation process.3 The successes in Spain have since spurred other countries to adopt similar structures and policies.

In Canada, the regulation and administration of organ donation are a shared responsibility between federal and provincial and territorial governments. The safety of organs and tissues for transplantation is governed federally under the Safety of Human Cells, Tissues and Organs for Transplantation Regulations. Canadian provinces and territories have legislation governing the practices of organ and tissue donation.1 In 2010, the Trillium Gift of Life Network (TGLN), the agency charged with coordinating the donation and transplant system in Ontario, Canada, started to put the building blocks in place to create a network of donation physicians to enhance physician leadership. This process began with the Chief Medical Officer then later Regional Medical Leads (RMLs), and ultimately the addition of hospital-based donation physicians (HDPs). RMLs collaborate with the TGLN team and the CMO to support hospitals within their geographic region through the development of donation programs and the provision of education, whereas HDPs champion donation within their individual hospitals.6 The HDP role, broadly defined, is “to raise the profile of organ and tissue donation and to provide clinical leadership and champion the organ and tissue donation program”.7 A wide range of services and deliverables were outlined in the HDP service agreement including, but not limited to, ensuring effective implementation of TGLNs policies and procedures, ensuring integration of donation into end-of-life care practices and policies, liaise with TGLNs RMLs and CMO to address hospital concerns and issues, provide hospital-based leadership and education, improving hospital-specific performance indicators, and contribution to the hospital or Ontario donation program through effective leadership and engagement.7

Undeniably, the context in which public health programs operate has become more complex. Programs that work in some settings, fail in others because of differences in their fiscal, socioeconomic, demographic, interpersonal, and inter-organizational reality. Given the demands for greater accountability from policymakers and other stakeholders, and the changing landscape in which public health programs operate, strong program evaluation is now considered essential.8 There is no one “right” evaluation approach. Rather, a variety of questions – to assess whether the program is achieving its intended outcomes – will arise during the program implementation. Evaluation means paying attention to and answering these questions and then sharing this information with key stakeholders who are accountable for the program’s success (or failure) and its ongoing funding. Recent calls for greater accountability and better understanding of health delivery programs, especially their impacts, have been met by a growing emphasis on implementation evaluation.9–13 Program evaluation must be initiated in parallel with program implementation to rapidly identify and address issues that could fester if neglected.

This study adopted the implementation evaluation framework first developed by Nielson et al, 2016 then refined by Schelvis et al, 2016 which aims at describing the requirements for successful implementation of an educational intervention.13,14 The objective of this study is to systematically evaluate the implementation of TGLN’s Physician Leadership Model by examining critical implementation process variables (education/training, communication, satisfaction, and participation and reach).

Methods

The current implementation evaluation was performed alongside a larger Needs Assessment (NA) conducted for TGLN. Evaluation of education and training ascertained knowledge of HDPs, as well as their perceived readiness to engage TGLN’s overall goal of promoting a culture of donation across Ontario hospitals. Evaluation of communication ascertained whether HDPs know where to go for timely, expert information related to donation. Evaluation of physician’s satisfaction ascertained whether HDPs believed that the Physician Leadership Program impacted donation, and if so in what areas. Evaluation of participation and reach measured the degree to which HDPs participate in donation-related activities to determine the reach of the individual communication networks. To obtain a robust data set, we employed an integrative mixed-method design. This study was granted ethical approval by the Ottawa Hospital Research Ethics Board (OHREB).

Participants

This evaluation included data from all members of the Physician Leadership Model including the CMO (who also acted as an RML), five RMLs, and the 52 HDPs.

Data Collection

Data collection was conducted between December 2015 and March 2016. Surveys were sent to all 52 HDPs (see for survey instrument). Qualitative interviews were conducted with the CMO and RMLs.

Data Analysis

Sociograms were used to evaluate participation and reach and were constructed using NVivo 11 Plus for Windows using the following steps: gather data, import data, exploration and analysis, and visualization and sharing of data.15 To begin, cases are constructed for each member of the network. Thus, each RML and HDP who responds to a survey becomes a case. Case nodes hold all information related to individuals in the study (which appear as circles on the sociogram). Next, relationships are created for individual (case) which indicate their relationships with others and the direction of these relationships. For example, one physician may seek advice from others but does not provide advice. Once relationships have been created for each of the cases, a sociogram is constructed. Ego-centric and network sociograms enable further exploration and analysis. Ego-centric sociograms reveal an individual’s relationship with other members of the group whereas, a network sociogram depicts everyone in the group.

The RML was interviewed as part of a larger need’s assessment.16 Interviews were audiotaped and transcribed verbatim. Interviews were coded and entered into NVIVO software. Qualitative data analysis techniques were applied to the data, including inductive coding techniques.17 The analysis team consisted of three researchers who participated in coding training and meetings to develop the codebook. There were ongoing group discussions to reach consensus on all codes. The data was then re-examined from the perspective of the implementation evaluation.

Results

The overall social network survey response rate was 85%. All 44 respondents were intensivists with critical care experience levels ranging from 8 (<5 years), 14 (5–10 years), and 22 (>10 years) in practice. Twelve were from academic hospitals and 32 were from community hospitals.

The results are presented according to implementation variables: education and training, communication, implementation satisfaction, and participation and reach.

Education/Training

Respondents reported that the HDP roles were not clearly defined nor well communicated at the outset. They perceived this as a barrier to the program implementation. As illustrated in Table 1, HDPs provided a range of answers as to what they perceived their role to be. The most frequently described role was to monitor performance/quality improvement. When asked if HDPs received any training for their position, 42% reported not having received any training.

Table 1

HDP Understanding/Description of Role

Role Description	Number of Respondents	Percentage
Monitor performance/Quality improvement	12	29.3%
Physician lead	11	26.8%
Support/Resource	8	19.5%
Chair/Co-chair/Participate in OTD committee	8	19.5%
Education/Promote awareness	8	19.5%
Facilitate/Coordinate/Communicate process	7	17.1%
Administrative	6	14.6%
Hospital liaison with TGLN	5	12.2%
Implement and support best practices	5	12.2%
To practice critical care	4	9.8%
Do not understand question	2	4.9%
Unsure/Learning role	1	2.4%
Total	77	187.8%
Number of respondents with multiple answers	22	53.7%
Total number of respondents	41

Note: Respondents had the option of responding to more than one role descriptor.

The HDPs are just starting to grow into their roles. It will take time but they have the most potential to get work done at the local hospital level (RML)

To further probe into the HDPs’ understandings of their role, they were asked if they were aware of any performance metrics associated with the HDP role. Sixty-five percent indicated not being aware of any metrics.

Communication

I mean I think one of the biggest changes are just to have a physician clinician conduit back-and-forth both ways to the organization … clinicians now have some actionable authority to take thoughts and griefs and present them to all the physician leaders. You know, what seems to be a constant stick in the mud problem for you is actually relevant to the whole province. (RML)

As shown in Table 2, 65% of HDPs indicated that their RML was the first point of contact for advice and support concerning organ donation management.

Table 2

Communication Advice Networks

Who is the First Person You Contact for Support
My Regional Medical Lead	28	65.1%
Trillium Gift of Life Network	10	23.3%
Another physician at my hospital	2	4.7%
Another Regional Medical Lead	1	2.3%
Another HDP at a different hospital	0	0.0%
Someone else	2	4.7%
Total responses	43
Not answered	1

HDPs were asked if they had time to collaborate with their local organ and tissue donation coordinators (OTDC). Eighty-six percent of respondents felt they had sufficient time to discuss donation cases with their OTDC.

Satisfaction

HDPs were asked to indicate changes they had seen as a result of the program’s implementation see Table 3). The most highly rated items were: improved metrics (eg, increase in donation rates); increase in awareness of donation processes; better education; and improvements in clarification of roles, responsibilities, and processes during the management of donation cases.

Table 3

HDP Implementation Satisfaction

Changes Seen at Your Hospital as a Result of the Program	Number of Respondents	Percentage
Improved metrics (eg, donation conversion)	12	37.5%
Increased awareness/interest of donation process	7	21.9%
Better education	7	21.9%
Improved clarification of roles and responsibilities regarding donor management	5	15.6%
Improved communication	4	12.5%
Implementation of protocols	4	12.5%
Accountability translates into action	4	12.5%
Better support	3	9.4%
Organizational buy-in/increased profile	2	6.3%
Regular organ donation committee meetings	1	3.1%
Too early to tell	1	3.1%
None	1	3.1%
Total number of responders	32
Not answered	12

Respondents were asked about any unanticipated implementation outcomes (for example, did any issues come up during the implementation of the Physician Leadership Model that were unforeseen). Eighty-six percent of respondents (N=43) indicated having encountered unanticipated outcomes with Donation by Cardiocirculatory Determination (DCD) related to the low volume of DCD cases, barriers to developing a DCD process, and lack of follow-up and review of unanticipated events during DCD.

Finally, HDPs were given an opportunity via open text to indicate if they would do anything differently if given the opportunity to start over the program. A small number of respondents (N=10) provided answers relating to different needs: more clarity about the HDP role including defined responsibilities and expectations; more education about managing donors and; engagement with other HDPs to determine existing education resources at different hospitals. One respondent simply stated, “it’s too soon to tell.”

Participation and Reach

Social Network Analysis (SNA) provides baseline visual representation of the Physician Leadership Model at implementation (Figure 1). The Chief Medical Officer is illustrated in yellow, the RMLs are illustrated in green, and the HDPs are shown in purple. The CMO and RMLs are key information brokers within this network map as they play central roles in connecting the HDPs. The sociogram is dense meaning that there are already many connections between physicians as the program is being implemented.

Figure 1

TGLN’s Physician Leadership Model Sociogram at Baseline: Participation and Reach.

Figure 2 illustrates one individual RML network and depicts how SNA provides a more detailed examination of individual ego-centric network maps (eg, of RML network depicted in yellow). Table 4 is representative of the many HDPs who seek out this RML for advice on donation, and the frequency of their interactions (eg, weekly, bi-weekly, monthly or > monthly).

Figure 2

Examining Network Participation and Reach: RML Ego-centric Network.

Table 4

Individual RML Network Data (Example of Data Provided from Survey)

Frequency of Contacts				Number of Years Known
Weekly	Bi-weekly	Monthly	<Monthly	<5 years	6–10 years	>10 years
1	2	8	17	22	5	0

Discussion

The objective of this study was to systematically evaluate the implementation of the (TGLN) Physician Leadership model by utilizing a framework that examines critical implementation process variables (education/training, communication, satisfaction, and participation and reach). We have demonstrated that an implementation evaluation helps us to understand which elements of the Physician Leadership Model were successful and which ones required immediate attention.

As a result of program implementation, HDPs reported a greater awareness in donation processes and better education at their sites. HDPs also indicated they were very satisfied with the model as they reported increases in donation rates at their local hospitals. Social network mapping and analysis were utilized as an evaluative implementation measure to describe the physician network. The sociogram demonstrated a dense social network centered around the CMO and 5 RMLs (see again Figure 1); the CMO and the five RMLs (depicted in green) are key knowledge brokers within this network structure. They play central roles in connecting HDPs (depicted in purple), and perhaps most importantly, ensuring the timely flow of information throughout the network. Figure 1 demonstrates that information flows from RMLs to HDPs but not between HDPs, which may reflect the HDPs’ inexperience at the time of implementation. Consulting an experienced RML is no doubt much more informative than consulting other HDPs who are new at their work and lack training. The dense sociogram support the TGLN strategy of first creating the RML position, allowing it to mature, and then expanding the network to HDPs only once the RML position had been firmly established. Having performed a social network analysis during the implementation phase also enables stakeholders to examine the evolution of their network given the relative ease of repeated analysis.18

Perhaps more important to organizational leaders, this evaluation identified which elements of the Physician Leadership Model required more immediate attention. As the Physician Leadership Model was implemented across ON TGLN provided educational sessions for HDP. However, HDPs reported a need for more education and training. Thus, the issue was perhaps one of the awareness and uptake. Despite providing a listing of HDP role services and deliverables, the majority of HDPs were confused and lacked role clarity. Clearly defining roles and responsibilities improves the capacity to plan, guide, and control the work being performed.19 In contrast, role ambiguity, defined as the lack of clear, consistent information regarding the actions required in a particular position may produce anxiety, depression, lead to poor effort, and decrease work satisfaction.20–24 However, role ambiguity may also have positive effects in that, under certain conditions, it may enhance creativity. It appears that role ambiguity and creativity exist on a continuum. At one end, an excessive amount of role ambiguity may impede creative behaviour increasing stress and decreasing motivation while at the other end, a moderate level of role ambiguity may be conducive to creative behaviour.25,26 Wang et al, (2005) found that it is possible for organizational leaders to stimulate or stifle employee’s creative efforts by modifying or adjusting the level of role ambiguity.27 The hallmark of a learning organization is when leadership make a commitment to using evaluation data and make it synonymous with organizational improvement.28–30

In response to this evaluation, TGLN clinical donation leadership introduced many activities to improve the program, including clarifying the HDPs’ role and expectations, weekly regional teleconferences, webinars, audits, quarterly face-to-face meetings, support for participation in national Canadian Blood Services workshops, and the creation of a TGLN physician portal containing educational videos. Also, the Canadian Critical Care Forum now includes the Deceased Donation Symposium which has become an opportunity for strengthening the network, especially HDP to HDP relationships. In the near future, social media should enable secure discussion to improve HDP-HDP-RML conversation. In addition, the senior physicians (RMLs and CMO) hold weekly capacity building meetings that are universally well attended and monthly networking meetings with clinical donation leadership in the organization (personal communication, TGLN CMO). During difficult or challenging cases or policy discussions, social media (WhatsApp) can be employed to garner consensus within a few minutes on very difficult topics or judge the necessity of a team meeting. Recently, a goal-setting software solution allowing asynchronous communication between RMLs has been introduced. In ongoing efforts to refine and grow their network, TGLN developed a new performance metric called the eligible approach rate which measures the number of families approached to discuss the potential of organ donation for their loved one against the total number of cases where donation was a possibility and a family could have been approached.31 From April 1 to June 30, 2018, 55 Ontario hospital organizations achieved an eligible approach rate of 82%.31 Moreover, we have demonstrated that implementation evaluation assists organizations both in the short and long term. In the short term, it identifies individuals with expertise, fosters knowledge sharing across organizational and geographic boundaries, and improves the rate of implementation (eg, increases the uptake of evidence-based practices).32 In the long term, implementation evaluation assists organizations to increase capacity for knowledge development, support peer-based partnerships and collaborations, and leverage strategic planning. SNA can be used to target and improve group processes, provide feedback to organizational leaders, offer a systematic overview of the network, and help to increase public accountability.18

This study has strengths and limitations. Strengths include a high response rate for the social network questionnaire and the identification of elements that deserve immediate attention to ensure future sustainability and success. Organizational use of evaluation data ensures TGLN is poised for growth. Limitations include the collection of data at a single point in time, which provides a narrow look at the network and does not permit an examination of the network evolution. Due to feasibility constraints, this evaluation did not include all process-related implementation variables for investigation. For example, we did not purposefully examine context and culture as this was beyond the scope of our objective.

Conclusion

This implementation evaluation helped highlight the successes and challenges in implementing the TGLN Physician Leadership Model in Ontario. TGLN is well poised to share its lessons learned so that other Canadian provinces can implement similar leadership models. Social network mapping permits an ongoing evaluation of the network over time. It provides actionable information on the network density (degree to which its members are interconnected), identifies central information brokers and members who are less engaged, operate on the periphery of the network and deserve attention. For example, conducting another network analysis would reveal the degree to which HDP to HDP interaction increased. Social network analysis of publicly funded capacity building systems shows promise as an additional tool to be used for broader interdisciplinary health program implementation evaluation.