The Joint External Evaluation of Taiwan: The External Evaluators' Perspective.

A team of experts from the Johns Hopkins Center for Health Security conducted an independent external assessment of Taiwan's capabilities under the International Health Regulations 2005 (IHR), using the IHR Joint External Evaluation (JEE) tool adopted by the World Health Organization and the Global Health Security Agenda. In this article we describe the methods and process of the assessment, identify lessons learned, and make recommendations for the government of Taiwan, the JEE process, and the JEE tool.

This article reports on an independent external assessment of Taiwan's capabilities under the International Health Regulations 2005 (IHR) using the IHR Joint External Evaluation (JEE) tool. The purpose of the evaluation was to guide Taiwan in its progress toward full development of IHR capabilities to prevent, detect, and respond to public health threats, whether they are naturally occurring, deliberate, or accidental. This process and tool have been adopted by the Global Health Security Agenda (GHSA) for the purpose of evaluating member countries.

Because of its unique international political situation, Taiwan is not a member of the United Nations, the World Health Organization (WHO), or the GHSA. However, because Taiwan wishes to be an active participant in the global community and supports the goals of the GHSA, it elected to undergo an external evaluation that mirrors the process being used in GHSA member countries that have volunteered to participate in the JEE process. At the time of this writing, only 7 other countries have completed and published an external assessment using the JEE tool; Taiwan is the 8th country to publish its results.

The JEE tool provides a standard metric by which countries can assess their current baseline capabilities and measure future progress. The JEE was used to assess Taiwan's collective capabilities, not just those of the Ministry of Health and Welfare (MOHW) or Taiwan Centers for Disease Control (Taiwan CDC). Many of the required capabilities involve other government agencies as well, such as agriculture, defense, border control, environmental protection, and nuclear power. Therefore, the external evaluation of Taiwan's capabilities emphasized cross-sectoral and interagency collaboration.

The Taiwan Evaluation

The multidisciplinary External Assessment Team consisted of 5 US subject matter experts from the Johns Hopkins Center for Health Security (Toner, Nuzzo, Cicero, Watson, Shearer), along with Ali Khan, MD, MPH, Dean of the College of Public Health of the University of Nebraska and former Director of the Office of Public Health Preparedness and Response at the US Centers for Disease Control and Prevention (CDC OPHPR). The team conducted the evaluation in collaboration with a multisectoral team of more than 80 officials from relevant agencies from the government of Taiwan. The evaluation involved 2 visits to Taipei, Taiwan, including a pre-assessment orientation visit in March 2016 and the evaluation mission, which took place from June 21 through July 1, 2016.

The GHSA evaluation process was evolving during the course of this project. Originally, 5 countries participated in a pilot project to test the feasibility of external assessments. A purpose-built tool was used in this phase that addressed 11 action packages. After the pilot phase, in conjunction with the WHO, the 11 action packages were merged with 8 relevant capabilities that are part of the IHR to create a new JEE tool. The 92-page tool consists of 48 indicators across 19 capabilities and hundreds of technical questions. Answers to these questions form the basis of individual scores for each of the indicators on a 5-point scale. The joint external evaluation of Taiwan used the then-current version of the JEE tool being used by GHSA for country evaluations.

The evaluation process as it has played out in various countries has involved some degree of country-specific customization. The “standard approach” includes some pre-evaluation education of both the External Assessment Team and the host country team, a self-evaluation by the host country using the JEE tool, a review of the self-evaluation prior to the team's visit, a 5-day visit to the host country, a wrap-up session on the final day of the visit, and scoring that is determined jointly by the host country and external team.

The Taiwan evaluation followed this template with several modifications. An introductory initial visit to the host country was added, the evaluation visit was extended to 2 weeks (10 working days, which included a site visit outside the capital), and the self-evaluation was presented in person during the external evaluation visit instead of in advance of the visit.

The Initial Visit

This 4-day visit by 3 of the External Assessment Team included an introductory plenary session chaired by the Director General of the Taiwan CDC, Hsu-Sung (Steve) Kuo, with many of the multisector participants in the self-assessment teams. Over the course of the remaining days, the External Assessment Team met individually with each of the 19 self-assessment teams and reviewed the JEE tool line-by-line to ensure that there was no confusion due to language and that the External Assessment Team and self-assessment team interpreted the questions similarly. This initial visit was viewed as very important by both host and external teams.

The Evaluation Visit

The external evaluation mission lasted 2 weeks (10 working days) and involved 6 external evaluators. Eight days were taken up with a series of 90-minute meetings with the 19 individual self-assessment teams during which they presented the self-evaluation, documentation, priorities for action, and possible scores. During these meetings, the External Assessment Team asked probing and clarifying questions and discussed strengths and gaps and scoring. The final day consisted of a concluding plenary session in which the External Assessment Team provided preliminary findings and tentative scores for each of the 48 indicators. Dr. Kuo and the self-assessment teams had the opportunity to comment on each of the findings.

Site Visit to Taichung

In the midst of the evaluation visit, the External Assessment Team conducted a full-day remote site visit to the city of Taichung, where they visited the District Health Center of the Mid-Western Region Bureau, the infectious disease unit of Taichung Hospital, the Taiwan CDC Central Regional Office, the regional public health laboratory, and the Taiwan FDA Central Regional Office. This site visit provided valuable insights into regional and local capabilities.

Post-Visit Process

After the end of the external evaluation mission, the External Assessment Team continued to collect and review documentation. Additional comments and further answers were provided by Taiwan agencies, and these were considered for inclusion in the final JEE. After some deliberation and consultation with the Taiwan CDC, it was decided that the final scores should reflect only the information available at the time of the external evaluation mission.

The JEE findings

The full report of the Taiwan JEE is available online.[1] For each of the 19 capabilities there are sections on targets, level of capabilities, strengths, and areas that need to be strengthened, along with a list of documentary evidence and a score. Scores are determined on a 5-point scale: 1 = no capacity, 2 = limited capacity, 3 = developed capacity, 4 = demonstrated capacity, and 5 = sustainable capacity. The scores are further color coded: 1 is red, 2 and 3 are yellow, and 4 and 5 are green. A summary of Taiwan's scores are provided in Table 1.

Table 1.

Summary of Scores

Element	Indicator	Score
National Legislation, Policy, and Financing	P.1.1 Legislation, laws, regulations, administrative requirements, policies, or other government instruments in place are sufficient for implementation of IHR.	4
	P.1.2 The state can demonstrate that it has adjusted and aligned its domestic legislation, policies, and administrative arrangements to enable compliance with the IHR (2005).	4
IHR Coordination, Communication, and Advocacy	P.2.1 A functional mechanism is established for the coordination and integration of relevant sectors in the implementation of IHR.	4
Antimicrobial Resistance	P.3.1 Antimicrobial resistance (AMR) detection	5
	P.3.2 Surveillance of infections caused by AMR pathogens	5
	P.3.3 Healthcare-associated infection (HCAI) prevention and control programs	4
	P.3.4 Antimicrobial stewardship activities	4
Zoonotic Disease	P.4.1 Surveillance systems in place for priority zoonotic diseases/pathogens	5
	P.4.2 Veterinary or animal health workforce	5
	P.4.3 Mechanisms for responding to infectious zoonoses and potential zoonoses are established and functional.	5
Food Safety	P.5.1 Mechanisms are established and functioning for detecting and responding to foodborne disease and food contamination.	3
Biosafety and Biosecurity	P.6.1 Whole-of-government biosafety and biosecurity system is in place for human, animal, and agriculture facilities.	3
	P.6.2 Biosafety and biosecurity training and practices	3
Immunization	P.7.1 Vaccine coverage (measles) as part of national program	5
	P.7.2 National vaccine access and delivery	5
National Laboratory System	D.1.1 Laboratory testing for detection of priority diseases	5
	D.1.2 Specimen referral and transport system	5
	D.1.3 Effective modern point-of-care and laboratory-based diagnostics	5
	D.1.4 Laboratory quality system	5
Real-Time Surveillance	D.2.1 Indicator and event-based surveillance systems	4
	D.2.2 Interoperable, interconnected, electronic real-time reporting system	4
	D.2.3 Analysis of surveillance data	5
	D.2.4 Syndromic surveillance systems	4
Reporting	D.3.1 System for efficient reporting to WHO, FAO, and OIE	5
	D.3.2 Reporting network and protocols in country	5
Workforce Development	D.4.1 Human resources are available to implement IHR core capacity requirements.	4
	D.4.2 Applied epidemiology training program in place such as FETP	4
	D.4.3 Workforce strategy	5
Preparedness	R.1.1 Multi-hazard national public health emergency preparedness and response plan is developed and implemented.	5
	R.1.2 Priority public health risks and resources are mapped and utilized.	5
Emergency Response Operations	R.2.1 Capacity to activate emergency operations	5
	R.2.2 Emergency operations center operating procedures and plans	5
	R.2.3 Emergency operations program	5
	R.2.4 Case management procedures are implemented for IHR-relevant hazards.	5
Linking Public Health and Security Authorities	R.3.1 Public health and security authorities (eg, law enforcement, border control, customs) are linked during a suspected or confirmed biological event.	4
Medical Countermeasures and Personnel Deployment	R.4.1 System is in place for sending and receiving medical countermeasures during a public health emergency.	4
	R.4.2 System is in place for sending and receiving health personnel during a public health emergency.	3
Risk Communication	R.5.1 Risk communication systems (plans, mechanisms, etc)	4
	R.5.2 Internal and partner communication and coordination	4
	R.5.3 Public communication	5
	R.5.4 Communication engagement with affected communities	4
	R.5.5 Dynamic listening and rumor management	4
Other IHR-Related Hazards and Points of Entry (PoEs)	PoE.1 Routine capacities are established at PoE.	5
	PoE.2 Effective public health response at points of entry	5
Chemical Events	CE.1 Mechanisms are established and functioning for detecting and responding to chemical events or emergencies.	3
	CE.2 Enabling environment is in place for management of chemical events.	5
Radiation Emergencies	RE.1 Mechanisms are established and functioning for detecting and responding to radiological and nuclear emergencies.	3
	RE.2 Enabling environment is in place for management of radiation emergencies.	5

As is evident from the scores and in the full report, Taiwan's robust strengths in public health were obvious. Taiwan is doing an excellent job of meeting most of the IHR goals. There is clear “sustainable capacity” (level 5) for many of the indicators, including points of entry and disease surveillance, and “demonstrated capacity” (level 4) for many others, such as the development of national policy and antimicrobial stewardship. For some indicators in which a lower capacity is evident, it is often only a small part of a criterion that is missing.

While Taiwan demonstrates considerable capacity in most of the assessed areas, it does face some challenges. These fall into 3 overarching themes that emerged during the evaluation. First, because of its unique international political status, Taiwan is not a full member state in the WHO and, therefore, cannot participate in some international programs that support IHR capabilities.

Second, like many countries, interagency and cross-sectoral collaboration in Taiwan is not optimal for fully achieving some IHR capabilities. For example, closer collaboration among the human public health, animal health, and food inspection sectors at both the local and national levels would enhance food safety and improve outbreak investigation capabilities in Taiwan.

Lastly, personnel and budgetary constraints and cutbacks limit some activities needed to fully achieve some IHR goals. Limited budgets also seem to hinder Taiwan's ability to be more engaged internationally. This inhibits Taiwan's ability to learn from bilateral or multilateral engagement and to share its considerable expertise with other countries.

Lessons Learned

In terms of the JEE process, we found that the initial visit was very important. The wording of some of the JEE questions is ambiguous, especially for users who are not native English speakers. Therefore, jointly reviewing the meaning of the questions in advance is important. Building trust was another important function of the initial visit. It is natural that some participants from the host country would feel defensive or would be reticent to share information that might reveal areas of relative weakness. Although the assessment process can be intrusive, the purpose of the JEE is intended to be entirely constructive. Through a friendly and collegial attitude, the external evaluators can demonstrate that they are trustworthy and draw out participants who would otherwise be reluctant to be open.

A highly committed host country champion or leader is essential to a successful evaluation. The self-evaluation is an arduous process involving detailed answers to hundreds of technical questions across multiple government agencies. Many government documents must be searched, referenced, and translated. More than 80 individuals were involved in the Taiwan self-assessment, which no doubt required extraordinary effort and many hundreds of hours of work time. Without a highly placed and motivated leader capable of commanding such effort, the self-evaluation would be less than optimal.

The leader must also provide clear guidance on the approach to the self-assessment scoring. On the one hand, a high score is naturally a source of pride; on the other hand, it could undermine ongoing funding. Conversely, a lower score could be used to substantiate a budget request for more funding but could also invite criticism from the public, press, or elected officials. In the end, strictly adhering to the scoring criteria seems to be the best approach.

Although this approach might tend to produce a lower score in some indicators if some minor scoring criteria are missing, it limits subjectivity. With this strict scoring approach, it is important to recognize that a less than perfect score does not necessarily mean that the host country is doing a poor job, just that certain specific criteria have not been met. Likewise, it is equally important to recognize that a perfect score on the tool does not necessarily equate to perfection in public health capabilities. There are many capabilities that the tool does not address.

Despite the effort needed and the limitations of the scoring system, the JEE process is valuable to the host country because it promotes interagency and cross-sectoral collaboration, uncovers gaps, and thus allows the host country to honestly reflect on priorities.

• An initial introductory visit to the host country by the External Assessment Team should be routine.

• Site visits to outlying public health departments should be encouraged.

• The evaluation visit should be lengthened.

• The tool should be reviewed for US-centric language and concepts to ensure cultural neutrality and clarity to non-Americans and non–native English speakers. Examples of such language include “dynamic listening,” “whole of society,” and “SWOT.”

• The number of questions and level of detail should be equalized across the indicators and redundancies should be eliminated. For example, “food safety” has 1 indicator with 8 technical questions; in contrast, “risk communications” has 5 indicators with 58 questions.

• Indicators related specifically to healthcare preparedness capabilities such as hospital surge capacity should added.

Conclusion

The JEE process has true value to the host country in that it helps to identify gaps, set priorities, document funding needs, and build cross-sectoral relationships. We conclude that all countries should conduct external evaluations, act on the findings of the JEE, and conduct periodic reevaluations to gauge progress. This project furthermore demonstrates that a JEE that meets the GHSA goals can be successfully accomplished outside the official WHO/GHSA process.