Evaluating an institutional health partnership using the ESTHER EFFECt tool: A case study of an evaluation of the institutional health partnership between Nigeria CDC and Public Health England.

Objectives: Bilateral Institutional Health Partnerships (IHPs) are a means of strengthening health systems and are becoming increasing prevalent in global health. Nigeria Centre for Disease Control (NCDC) and Public Health England (PHE) have engaged in one such IHP as part of Public Health England's International Health Regulations Strengthening project. Presently, there have been limited evaluations of IHPs resulting in limited evidence of their effectiveness in strengthening health systems despite the concept being used across the world. Study design: Qualitative, using a validated tool.
Methods: The ESTHER EFFECt tool was used to evaluate the IHP between NCDC and PHE. Senior leadership from both organisations participated in a two-day workshop where their perceptions of various elements of the partnership were evaluated. This was done through an initial quantitative survey followed by a facilitated discussion to further explore any arising issues.
Results: This evaluation is the first published evaluation of a bilateral global health partnership undertaken by NCDC and PHE. NCDC scores were consistently higher than PHE scores. Key strengths and weaknesses of the partnership were identified such as having wide ranging institutional engagement, however needing to improve dissemination mechanisms following key learning activity. Conclusions: There is a dearth of evidence measuring the effectiveness of international health partnerships; of the studies that exist, many are lacking in academic rigour. We used the ESTHER EFFECt tool as it is an established method of evaluating the progress of the partnership, with multiple previous peer-reviewed publications. This will hopefully encourage more organisations to publish evaluations of their international health partnerships and build the evidence base.

Introduction

Efforts to improve global health have included health systems strengthening, capacity building and technical collaboration in low and middle-income countries. One means of providing the education and professional development required to strengthen health systems is through bilateral institutional health partnerships (IHPs). The aim of such partnerships is to build capacity, offer peer to peer support, increase research and facilitate professional development in order to improve the capacity of the low and middle-income partner country (or regional organisation as the case may be). Improving health systems through such partnerships have gained new prominence on the aid agenda of donors [1]. Broad trends appear to demonstrate improved health outcomes as a consequence of health links [2]. A report by the European ESTHER Alliance found that IHPs are a valid, cost-effective and complementary form of technical cooperation [3]. However, a rapid evidence review of peer-reviewed and grey literature found that evidence for effectiveness of IHPs is thin in terms of quantity and academic rigour [4]. A need for evaluating IHPs as a whole, rather than focusing on the effectiveness of specific projects, was identified.

IHPs often use compliance with various international commitments, such as the World Health Organization’s International Health Regulations 2005 (IHR), as a framework around which the health systems strengthening work can be built around. The IHR are an agreement between 196 countries to work together for global health security. It commits countries to ensure that they are able to detect, assess and report global health threats. The Public Health England (PHE) International Health Regulations Strengthening Project is a strong example of IHP for strengthening IHR. This programme promotes IHR strengthening through exchanging knowledge, cultural insight, and professional experiences with other National Public Health Institutes (NPHIs) and supporting low and middle-income country (LMIC) NPHIs through providing technical expertise and some targeted resource investment. PHE is funded through the UK Aid budget to work in partnership with the WHO to improve IHR compliance globally and has bilateral relationships with Nigeria, Ethiopia, Pakistan, Sierra Leone, Myanmar and Zambia.

In November 2017, PHE and Nigeria Centre for Disease Control (NCDC) signed a memorandum of understanding to work in partnership and improve Nigeria’s compliance with IHR. Four broad objectives were agreed based on the outcome of Nigeria’s June 2017 Joint External Evaluation (JEE) monitoring Nigeria’s compliance with IHR:

Support the development of national and regional capacity for emergency preparedness, resilience and response.

Enhance national surveillance systems and public health laboratory networks.

Support the development of a skilled public health workforce.

Support the development of NCDC as the national NPHI and as a regional lead public health institution in West Africa.

In order to ensure that the PHE-NCDC partnership was meeting these objectives, in May 2019 an evaluation was conducted. This paper describes an evaluation of the different components and aspects of the partnership between NCDC and PHE to understand participant perceptions of whether the partnership objectives are being met.

Methods

Objectives of the evaluation

In exploring the partnership between NCDC and PHE we aimed to determine participants perceptions on:

The perceived engagement in and benefits from the partnership

The delivery of learning

The reach and delivery of capacity building activities

How the partnership could be improved moving forwards

In order to evaluate these perceptions, we selected the ESTHER EFFECt (Effective in Embedding Change) self-assessment tool1 designed to evaluate the effectiveness of interventions through institutional partnerships. The ESTHER EFFECt tool and questionnaire, as used during this evaluation, can be found in Appendix A. The tool is split up into four modules with between four and 11 questions per module. There are four response options for each question that are scored from 1 to 4. The higher the response number, the closer the perception is perceived as “best practice”.

The desired outcome of this exercise was for each institution to gain an understanding of the strength, value and impact of the partnership between NCDC and PHE through the IHR Strengthening Project partnership as well as to identify areas for further development. The evaluation was initiated in February 2019, following one day of training delivered to both PHE and NCDC staff.

ESTHER EFFECt evaluation steps/format

The evaluation exercise was conducted through two 3-h facilitated workshop sessions over two days, involving senior representatives from NCDC and the PHE IHR Strengthening Project.

The first session aimed to introduce the tool and questionnaire to the participants and was led by PHE Global Public Health Monitoring and Evaluation team. The opening of the first session stated the aims and objectives of the anticipated outcomes of the ESTHER EFFECt tool, and then a practical step-by-step of the process and the questionnaire. Each participant received a copy of the introductory presentation in their pack, together with a hard copy of the questionnaire. All participants then filled in the questionnaire individually. The facilitation team were on hand to answer questions if parts of the tool were unclear. The sessions were split over 2 days. This allowed participants to fully reflect on the questionnaire modules from day 1, enabling and empowering them to engage on day 2.

In the second session, two independent external consultants facilitated a discussion on the analysis of the questionnaire responses. Participants were guided to look at where there had been variation in responses, either between PHE and NCDC or within each partner organisation and encouraged to discuss why they thought different responses had been made. Participants were also asked to suggest action points for improving areas and to move the partnership to the more mature levels represented by higher scores.

This approach was intended to provide both partners with an idea of how the work is being perceived within the organisations, with the facilitated discussion intended to provide recommendations moving forward. All participants were aware of the study purpose, methods and dissemination. As a result, ethical approval was not needed or sought.

Results

Twelve [12] participants attended the workshop, seven from NCDC and five from PHE. Senior leadership including the NCDC Director, the PHE IHR project lead, managers and other directors across both organisations were present. A full list of attendees can be found in Appendix B. The questionnaire responses for the ESTHER EFFECt tool for each module of the tool is presented below, stratified by organisation (i.e. PHE or NCDC) response, alongside a summary of the facilitated discussion. Scores range from 1 to 4, with higher scores indicating better practice. NCDC and PHE scores were also compared by module using the unpaired t-test. A summary of the results can be found in Table 1 below:

Table 1

Summary of results by module for NCDC and PHE.

	Mean score NCDC	Mean score PHE	Number of components	p-value from unpaired t-test
Module 1	3.23	2.56	11	0.0006
Module 2A	2.04	1.25	4	0.002
Module 2B	2.87	2.7	4	0.41
Module 2C	2.78	2.57	6	0.30
Module 2D	2.57	2.48	5	0.73
Module 3	2.73	3.03	5	0.128
Overall	2.69	2.50	35	0.001

Module 1 and 2A show a statistically significant difference between NCDC and PHE scores. Overall, there is also a statistically significant difference between NCDC and PHE scores, with NCDC scores being higher in every module other than Module 3: Added benefits to your institution. This difference can be seen in Fig. 1 below, with NCDC scores consistently higher than PHE scores.

Fig. 1

Radar chart of scores for NCDC and PHE by module using the ESTHER EFFECt tool.

Module 1: Implementation best practice

For the Implementation best practice module (Fig. 2), participants were asked to score from 1 to 4 whether best practice had been implemented on a variety of topics ranging from needs assessments to dissemination of best practice. The average score for implementation best practice from Nigeria CDC was 3.23 whereas for PHE it was 2.56. Aside from one question, both agencies scored above ‘2’ in each question indicating a high level of satisfaction for most components. Dissemination (component 1.11) was scored the lowest by both NCDC (2.83) and PHE (1.4) indicating that this was perceived to be an area for improvement, whereas Beneficiary Partner Ownership (component 1.04) scored highest by participants from both partners.

Fig. 2

Summary of results for module 1 - Implementation best practice.

Facilitated discussion

The discussion highlighted a few factors that may have contributed to the high scores and provided some depth to the initial findings. The JEE was thought to form a solid basis for determining country needs (Component 1.01). There was also extensive discussion between PHE and NCDC, leading to agreement on the selection of activities for PHE to support.

At the beginning of the partnership, participants from both groups commented that the partnership seemed to involve fewer people who were generally, less engaged. As the partnership strengthened, both NCDC and PHE thought there was more engagement. Staff capacity has varied over time depending on issues faced in Nigeria and the UK but there was thought to be a good degree of flexibility to work around people’s availability, which was considered a key success factor. PHE was considered to be embedded as a partner within NCDC.

Engaged leadership was thought to be critical but had varied somewhat from the initial stated PHE IHR objective. It was suggested that lessons learned from stronger areas could be applied to weaker areas, possibly through more regular discussion and/or an annual leadership meeting.

The lessons learned from the partnership were disseminated organically. As a result, there was some divergence in scores between PHE and NCDC around dissemination because PHE was not aware of this work. This was considered to be an area for NCDC and PHE to improve on in order to demonstrate value for money. A formal dissemination plan was considered as an option for this.

Module 2A: Curriculum, learning and teaching development

For the Curriculum, learning and teaching development module (Fig. 3), participants were asked to score from 1 to 4 whether best practice had been implemented on a variety of topics ranging from curriculum coverage to learning and teaching methods. This was the module that received the lowest scores on average, from both NCDC (2.04) and PHE (1.25). Only 5 out of 12 participants (38%) replied to these questions. Of those who responded, the relevance of this module in relation to the NCDC and PHE partnership was thought to be low.

Fig. 3

Summary of results for module 2A - Curriculum, learning and teaching development.

During the discussion, it was highlighted that a knowledge management hub had been created within NCDC to address this module, although there may not be awareness of this throughout NCDC. Participants from both organisations suggested that there was a need to ensure that all NCDC staff were aware of this resource and that related departments from both organisations articulate and coordinate training needs.

Module 2B: Reach of capacity building activities

For the Reach of capacity building activities module (Fig. 4), participants were asked to score from 1 to 4 whether best practice had been implemented on a variety of topics ranging from critical mass to evidence-base. The average score across components was 2.88 for NCDC and 2.70 for PHE. The ability to deliver capacity building (component 2.B2) scored the lowest for both NCDC and PHE.

Fig. 4

Summary of results for module 2B: Reach of capacity building activities.

NCDC and PHE participants both agreed that capacity building activity appeared to be in progress to but there was still scope for improvement. PHE participants suggested that NCDC capacity could be further strengthened to deliver capacity building activities in areas with weaker engagement. Participants agreed that at present, most training is designed to facilitate a ‘training of trainers’ approach. Both sets of participants questioned whether this approach was working in cascading the training throughout NCDC. It was suggested that both partners could do with a better understanding of the actual effect training was happening.

The Evidence-base (component 2.B4) was one of the only components where PHE scores are larger than NCDC scores. It was thought that the high PHE scores reflect that its guidance has been adapted to the local context. Scores from NCDC participants were more mixed in this area, varying by NCDC department. It was proposed that this may be due to a difference in the needs of different areas in adapting PHE guidance to the local context. One example provided was that laboratory functions tend not to require much adaptation whereas anti-microbial resistance guidelines must be adapted to the local context. Expanding use of the NCDC knowledge hub was suggested as one way of increasing capacity in this area.

Module 2C: Improving practice through capacity building

For the Improving practice through capacity building module (Fig. 5), participants were asked to score from 1 to 4 whether best practice had been implemented on a variety of topics ranging from teams to advocacy. Improving practice through capacity building components were generally scored high, with averages of 2.78 for NCDC and 2.57 for PHE. The exception to this was Feedback (component 2.C4) which NCDC participants scored as a 2 and PHE participants scored as 1.2.

Fig. 5

Summary of results for module 2C: Improving practice through capacity building.

There was a general perception that higher scores for Changes in work practices (component 2.C3) were provided by more senior participants with a higher level and more strategic overview of organisation-wide performance. It was suggested that turnover in staff has restricted change in some departments. Focusing on this area was thought to be critical for sustainability. NCDC participants were positive that learning is being disseminated across departments leading to widespread change.

There was overall agreement that feedback could be improved, with it currently perceived as “ad hoc and reactionary”. Participants remarked that there was a need to systematise processes and suggested including a formal review process.

Component 2.C5, Access to Equipment/materials, was the only component in this module where PHE scores were higher than NCDC. There was some uncertainty around the commodities that PHE was able to offer as part of this project, which may have contributed to a lower NCDC score.

Module 2D: Whole institutional strengthening

For the Whole institutional strengthening module (Fig. 6), participants were asked to score from 1 to 4 whether best practice had been implemented on a variety of topics ranging from motivation for change to building institutional resilience. The average score across the module for NCDC participants was 2.57 and for PHE participants was 2.48. PHE participants scored Motivation for change (component 2.D1) and Systems thinking (component 2.D4) higher than NCDC participants.

Fig. 6

Summary of results for module 2D: Whole institutional strengthening.

NCDC responses were widely spread for Motivation for change (component 2.D1). The PHE team and some NCDC colleagues expressed surprise at this. The discussions suggested that there is general alignment in thinking and that the spread in scores may have been due to different interpretations of the question. Low NCDC scores related to systems thinking (2.D4) partly reflect that there is further work to do to ensure that NCDC is fully embedded/mainstreamed into the One Health sphere. This is however expected to improve over time as NCDC are considering opportunities to further coordinate partners in the IHR space, inclusive of entities operating in other sectors within the ‘One Health’ sphere.

Module 3: Added benefits to NCDC/PHE

For the Added Benefits to NCDC and PHE module (Fig. 7), the average score for NCDC participants was 2.73 and for PHE participants was 3.02. Questions for this module varied for both sets of participants. Though both sets of participants answered module 3.A, module 3.B was designed to explore NCDC opinions on peer support and spread/scale-up whereas module 3.C was designed to see if PHE opinions on whether there was any ‘reverse’ innovation. As such NCDC participants did not answer module 3.C and PHE participants did not answer module 3.B.

Fig. 7

Summary of results for module 3: Added benefits to PHE/NCDC.

In terms of Peer Support (component 3.B1) and Spread/Scale-up (component 3.B2), NCDC responses were generally positive with scores ranging from 2 to 4. Two PHE responses were positive about Reverse innovation (component 3.C1) whereas three said that it was too soon to judge this.

Scores related to Networking and partnerships (component 3.A1) were disparate between the two groups. While, PHE were generally more positive with an average score of three, NCDC participants had mixed perceptions, with two participants giving this component a score of one. The discussion revealed that the lower scores from NCDC reflected impressions from the early stage of the partnership, which was thought to have improved and become more joined-up over time. The technical nature of the partnerships was in general viewed very positively. While this is an area where further progress is required, efforts to engage with WHO and partners across sectors, as well as at a regional level, were acknowledged. PHE participants suggested that NCDC could consider further work in coordinating between the WHO, non-governmental organisations and other partner countries, in addition to the technical working groups already in place.

Staff Motivation (3.A2) and Empowerment (component 3.A3) scores were relatively high for both NCDC and PHE stakeholders. There was consensus that NCDC staff do feel empowered and more confident as a result of the technical assistance and opportunities that PHE support has provided; demonstrated by staff taking more responsibility for functions and championing new ideas for improvement. Scores related to Staff Retention (component 3.A4) were mixed for NCDC, where it was suggested that staff did not see the partnership as influencing retention; PHE, alternatively scored this high and remarked that staff engagement in the programme has increased motivation. PHE participants also felt that the partnership with NCDC had helped PHE systematise processes and technical content. This could then potentially be something that could be fed back to NCDC staff to learn from and build on.

It was discussed that this peer support and spread/scale-up is a core focus of the partnership. Participants expressed hope that this area of work can be improved over time by PHE ensuring training content is appropriate and NCDC disseminating this training across Nigeria and West Africa.

Module 4: Improved skills

For the Improved Skills module, participants were asked to pick the top three professional skills (Fig. 8) and then the top three management/comms skills (Fig. 9) that they felt they had gained.

Fig. 8

Frequency of professional skills gained by coordinators.

Fig. 9

Frequency of management and communication skills gained by coordinators.

Professional skills

The response with the highest frequency (i.e. participants that selected the skill the most) was ‘developing policies, protocols and guidance’ with the second highest being ‘emergency preparedness and building resilience’. There was some degree of variance between responses from NCDC and PHE. For instance, PHE participants did not choose ‘multidisciplinary team working’ as a top three skill gained, however 4 NCDC participants did.

Facilitated discussion

The above mentioned highest frequency responses were thought to reflect the participants desire to improve in the aim of the IHR programme competencies. The components with the highest disparate responses were multidisciplinary team working ability to take greater personal initiative, and exposure to different healthcare systems; these were felt to reflect each organisation’s prior experience in this type of working. For instance, NCDC participants expressed a desire for improving multidisciplinary team working where they may not have had much exposure, whereas PHE participants may be more used to working in such a fashion. In general, participants agreed that their skills and confidence have increased through programme engagement. Professional skills were thought to have been gained by participants in both organisations, potentially demonstrating the mutual benefit of the partnership.

Management and communication skills

NCDC participants reported they had gained leadership skills and problem-solving skills through this engagement whereas PHE participants were more likely to say that they no appreciable improvement to report or had gained skills in managing with limited resources.

Leadership skills were felt to have demonstrably improved in NCDC technical areas, with teams having strengthened capacity and working with greater confidence having engaged in the programme. Organisational skills and networking skills were also felt to have improved from state level engagement, which was thought to have been prioritised throughout the programme.

Recommendations/actions agreed by both institutions

As a result of the facilitated discussions, NCDC and PHE agreed upon the following recommendations and actions (Table 2). These recommendations are based on the overarching themes of the facilitated discussion.

Table 2

Recommendations and actions derived from the facilitated discussions following use of the ESTHER EFFECt tool.

Recommendation	Proposed Action
Leadership: Engaged leadership has been critical to implementation but has varied somewhat by PHE IHR pillar.	- Apply lessons learned from stronger areas to weaker areas through more regular discussion and/or an annual leadership meeting.
Dissemination and feedback: The lessons learned from the partnership between PHE and NCDC are disseminated organically, although PHE is not always aware of this dissemination and whether the approaches (e.g. the training of trainer’s approach to capacity building) adopted are working.	- Develop a formal feedback and dissemination plan to help both parties ensure that training content is appropriate and both entities are aware of the reach of activities. - Incorporate this plan into a broader dissemination/communications strategy.
Knowledge management: There is a need for a strengthened vision for knowledge management, including a training curriculum that all NCDC staff are aware of, have access to and mobilise around.	- A consistent and agreed approach to knowledge management. - Better understanding of and access to training curriculum.
Capacity: Further efforts should be invested into building NCDC capacity to deliver follow-on capacity building activities, particularly in the technical areas where capacity is weakest.	- Identify hurdles to capacity building. - Scope and plan how the two partners can ensure follow on capacity building is delivered, particularly in technical areas where capacity is weakest. - Review progress against the plan and evaluate effectiveness of the interventions.
Equipment and materials:PHE should clarify its position and ability to provide laboratory equipment and materials as soon as possible.	- PHE to update NCDC regularly regarding equipment. - Training to take account of availability of equipment and commodities.
Coordination: NCDC should consider opportunities to further coordinate partners in the IHR space, being inclusive of entities operating in other sectors within the ‘One Health’ sphere.	- Both partners to consider how opportunities can be worked into IHR deliverables. - Work plan to support NCDC to further coordinate with partners in the IHR space.

Discussion

Key findings

This evaluation is the first published evaluation of a bilateral global health partnership undertaken by NCDC and PHE. As a case study of the ongoing relationship between NCDC and PHE it demonstrates how the partnership perceptions across the organisations, key partnership strengths and areas for improvement. Opportunities for improving the delivery and usefulness of the tool have also been identified and include: increasing the number of participants, including people throughout the organisational hierarchy, not just senior leaders, ensuring all parts of the ESTHER EFFECt tool used are understood by participants, and re-phrasing some of the questions to ensure this. Additionally, whilst the tool allowed us to compare and contrast NCDC and PHE scores, this is the first instance of us using the ESTHER EFFECt tool—the responses from this initial exercise are useful as a baseline measurement of the partnership. Repeated administration of the tool may enable us to monitor progress.

Through administration of the ESTHER EFFECt tool a number of key partnership strengths were identified. There was a high level of NCDC ownership over proposed technical activity. Both partners agreed that there was wide ranging institutional engagement. The NCDC approved of PHE proposed capacity building activities. There was also a high level of motivation for change in NCDC. Areas for strengthening were also identified. There was a need to improve dissemination mechanisms following key learning activity. Strengthening curriculum awareness, use and access was thought to be needed to improve knowledge management. There was also a recognised need to evaluate how effective the capacity building activity has been. Part of this could including creating a plan for capturing and actively promoting ‘reverse’ innovation.

Our findings demonstrate that perceptions of an international health partnership and what these involve may differ but also converge depending on the element of the partnership being explored. A closer look at the various components of the partnership is needed to establish points of variation and convergence. The facilitated discussion highlighted that there may have been variation in responses dependent on management level, although we were unable to quantify this difference. Additionally, we found that a clear understanding within each organisation of what the partnership entails and what the aim of the partnership could strengthen the partnership and make it more productive. We also found that even within organisations there was variance in how the partnership was perceived. This may be due to differences in the level of engagement between corresponding staff and departments at both organisations and/or due to the partnership emphasising a specific topic more than others.

Both NCDC and PHE participants felt they had gained a number of skills through the partnership including both professional skills such as development of protocols, policies and guidance, and management skills such as managing with limited resources. Despite this, only one NCDC participant felt they had gained cultural competence through the partnership. Cultural competence is critical in a mutually beneficial global health partnership [5]. Across the board, scoring was generally positive suggesting that the partnership was viewed as having a constructive impact on organisational performance.

Strengths and weaknesses of the study

We used an established tool delivered by independent facilitators to understand the progress of the partnership and perceptions of the partnerships within both organisations. The tool was completed by senior leaders within both organisations and led to productive facilitated discussion resulting in a clear set of recommendations and actions. The application of the tool itself may have provided additional benefit, as it may have increased openness, cohesion and understanding between the two organisations through the facilitated discussion process. The qualitative approach enabled participants to voice their opinions in a substantive way, rather than simply responding to a quantitative survey without providing insight into why the scoring may be low or high.

PHE and NCDC have committed to repeat the ESTHER EFFECt tool in the near future at an appropriate point in the programme to review progress, and with repeat applications of the tool we will be able to understand how the partnership is progressing. The restricted number and variety of participants was a limitation as we could not ensure appropriate representation from all of the various stakeholders of the partnership at all relevant management levels. Another limitation is that several participants did not respond to all of the questions making our sample size even smaller. There may also have been variation in how some of the questions were interpreted by the participants leading to variation in scoring. This subjectivity and the reliance on participants answering honestly may have impacted the results. Piloting the tool beforehand with more detailed introductions to the tool and its modules could have mitigated some of the challenges above.

It also may have been too early in the partnership for participants to be able to answer some component questions (e.g. Reverse Innovation [component 3.C1]. Similarly, although the tool was administered by independent evaluators, the tool and its evaluators were funded by PHE. As a result, there may have been questions around their impartiality and subsequently this may have affected responses or participation in the facilitated discussion.

Interpretation and implications

The role of IHPs in implementing the IHR is clear. A WHO review of the role of the IHR in the West African Ebola epidemic identified the critical role that such partnerships play [6]. There is a dearth of evidence measuring the effectiveness of international health partnerships in improving public health systems; of the studies that exist, many are lacking in academic rigour [4] or focus on benefits to the donor country as in terms of ‘reverse’ innovation [7]. This is partly due to an inconsistency in the use of indicators and/or frameworks. We used the ESTHER EFFECt tool as it is an established method of evaluating the progress of the partnership, with multiple previous peer-reviewed publications [3]. We were unable to find similar evaluations of bilateral IHPs in peer-reviewed literature with most projects focusing on evaluating the outcomes of their partnerships rather than the strength of the partnership itself. Publishing this evaluation will hopefully encourage more organisations to publish evaluations of their international health partnerships and build the evidence base in order to better understand how to build effective IHPs.

Leadership plays a key role in the success of an IHP. The absence of high-level commitment and leadership has been identified as a significant challenge in strengthening health systems and improving IHR compliance [8]. Being able to administer the ESTHER EFFECt tool with senior leaders from both NCDC and PHE in itself demonstrates senior level buy in to the partnership and a willingness to improve and build upon the foundation already created. Increased engagement within the partnership also appears to lead to more positive perceptions of the partnership [9]. It was felt that as time progressed and more engagement occurred, many of the initial fears over the partnership were allayed and staff motivation increased. The importance of taking time over building and maintaining trusted relationships has been highlighted as key to success for IHPs [10].

Participants from both organisations gained in professional, management and communications skills highlighting the mutual benefit of the partnership. A review of potential benefits to developed country partners found that although there may be intangible benefits through international health partnerships, high-income countries can also benefit in ten key health areas: rural health service delivery; skills substitution; decentralisation of management; creative problem-solving; education in communicable disease control; innovation in mobile phone use; low technology simulation training; local product manufacture; health financing; and social entrepreneurship [11]. The relative lack of peer-reviewed evidence on the benefits of IHPs [4] and the focus on ‘reverse’ innovation could hint at where the publishing priorities of higher-income partner health institutions may lie. In order to ensure continued funding for such projects, evaluations tend to focus on activities and interventions rather than the impacts of building a long-term sustainable partnership or different health partnership models.

Conclusion

Bilateral global health partnerships are an important form of capacity building and system strengthening for all NPHIs—better resourced and well established NPHIs can be particularly integral in supporting the development of nascent institutions or NPHIs in low- and middle-income countries. Demonstrating the value that such partnerships bring and highlighting the perceived benefits to the partner countries is essential to ensure that there is political will to engage in such partnerships. Through using the ESTHER EFFECt tool, we have demonstrated one example of such an evaluation and how the partnership is seen in both NCDC and PHE. The application of the tool provided invaluable insight into how to improve partnership working and achieve the stated aims of the partnership. The recommendations and actions generated through application of the tool have served to further strengthen the IHP between NCDC and PHE. Future continued monitoring and evaluation will help ensure that such partnerships strengthen health systems, provide public health benefits to both partner institutes, increase value for money and strengthen the evidence base around what a good IHP looks like.

Author’s contributions

EO proposed the use of the tool and the approach to the evaluation. KH and GJ prepared and delivered the workshop with input from EO. AR produced a first draft of the paper which was subsequently reviewed by NE and EO. Further revisions were reviewed by the all of the authors. Additionally, EO, CI and OO were participants in the workshop.

Availability of data and material

All captured data is available within the paper and appendices.

Competing interests

None, there was no patient, public or commercial involvement in the study design or conduct of the study.

Ethical approval and consent to participate

Not applicable.

Funding

This work is funded by the through Official Development Assistance funds granted to through the in the 2015 spending review, for the delivery of the International Health Regulations Strengthening project.

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Competing interests – None, there was no patient, public or commercial involvement in the study design or conduct of the study.

Evaluators
Name	Organisation	Role
1. Dr Chikwe Ihekweazu	NCDC	Director General, Nigeria Centre for Disease Control
2. Dr John Oladejo	NCDC	Head of Department, Health Emergency Preparedness and Response Department
3. Mrs Olubunmi Ojo	NCDC	Director, Surveillance and Epidemiology Department
4. Mr Anthony Ahumibe	NCDC	Laboratory Senior Technical Advisor
5. Dr Joshua Obasanya	NCDC	Director, Prevention and Programmes Coordination Department
6. Ms Oyeronke Oyebanji	NCDC	Technical Assistant to the Director General, NCDC
7. Mrs Nwando Mba	NCDC	Director National Reference Laboratory
8. Dr Ebere Okereke	PHE	IHR Strengthening Project Lead
9. Dr James Elston (via Skype)	PHE	Consultant Epidemiologist, Field Epidemiology
10. Dr Colin Brown (via Skype)	PHE	Consultant Medical Microbiologist
11. Mr Paul Sutton	PHE	Director Emergency Response Department
12. Dr Olusola Aruna	PHE	Senior Public Health Advisor, IHR Strengthening Project Nigeria
Facilitators
1. Ms Katie Haddock	PHE	Monitoring and Evaluation Assurance Manager
2. Ms Anna Osei-Kofi	PHE	IHR Strengthening Project, Senior Project Officer
3. Dr Gurnam Johal	PHE	Monitoring and Evaluation Assurance Manager
4. Ms Cynthia Carlson	ITAD	Project Team Leader
Observers
5. Dr Jonathan Ashcroft	PHE	Microbiologist UK Public Health Rapid Support Team