Implementation of the IHR Joint External Evaluation: Taiwan's Experiences.

In February 2016, the World Health Organization developed the Joint External Evaluation (JEE) tool to independently assess country capacity to prevent, detect, and respond to public health threats as part of the International Health Regulations (IHR) (2005) monitoring and evaluation framework. In light of this, the Taiwan government actively engaged at least 19 government agencies or institutions and voluntarily implemented the JEE. An External Assessment Team consisting of 6 US subject matter experts conducted the external evaluation, including site visits, from June 21 to July 1, 2016. The results, published on October 18, 2016, are useful and will be translated into actions and change in the system. Based on Taiwan's experiences, early stakeholder engagement and an experts' pre-JEE pilot visit would contribute to a successful JEE process.

The beginning of the 21st century has been marked by a series of epidemics and pandemics, including extensively drug-resistant tuberculosis (XDRTB) worldwide, anthrax in the United States, severe acute respiratory syndrome (SARS) in East Asia and Canada, avian influenza [A(H5N1) and A(H7N9)] in Southeast Asia and China, Middle East respiratory syndrome (MERS) in the Middle East and South Korea, Ebola virus disease in West Africa, and, most recently, Zika virus disease in the Americas. These emerging infectious diseases as a global health burden are thought to be driven, at least partly, by the emergence and spread of new pathogens; globalization of travel, food, and medicines; the rise of antimicrobial resistance; and intentional engineering or accidental release of biohazard agents.[1] The morbidity, mortality, and economic impact of such public health threats can be enormous. For example, the Ebola outbreak in West Africa during 2014-15 was estimated to have resulted in approximately 28,000 human cases, at least 11,000 deaths, and economic loss of more than $30 billion.[2,3]

In response to international public health emergencies such as SARS, the World Health Organization (WHO) established the International Health Regulations (IHR) (2005), which require that all countries have the capability to detect, assess, report, and respond—that is, detect potential threats through surveillance systems and laboratories, make decisions in public health emergencies, report specific diseases and any potential international public health emergencies, and respond to public health events.[4] However, the WHO reported that by 2012, fewer than 20% of the countries had met the IHR goals, and by 2014, only approximately 30% of the countries were fully prepared to detect and respond to an outbreak.[5] In the wake of the Ebola outbreak in West Africa, several high-level panel reports highlighted the pitfalls of relying solely on a government's voluntary self-assessment to measure core capacities.[6-8] The panels unanimously recommended that all countries should commit to participate in regular, independent peer review or external assessment of their core capacities. In the 68th World Health Assembly in 2015, the IHR Review Committee on Second Extensions for Establishing National Public Health Capacities and on IHR Implementation recommended that countries move from exclusive self-evaluation to approaches that combine self-evaluation, peer review, and voluntary external evaluations involving a combination of domestic and independent experts.[5]

In light of this, WHO, in collaboration with the Global Health Security Agenda (GHSA) launched in February 2014, developed the Joint External Evaluation (JEE) process and JEE tool in February 2016 as part of the IHR (2005) Monitoring and Evaluation framework.[9] The JEE tool integrated the GHSA assessment tool, which was intended to assess country capacity to prevent, detect, and respond to public health threats independently of whether they are naturally occurring, deliberate, or accidental. Countries are encouraged to request a JEE mission to help them identify the most urgent needs in their health system with the expectation that the JEE can help engage with stakeholders and partners to support a country's preparedness for outbreaks and health emergencies. Bangladesh, Ethiopia, Liberia, Mozambique, Pakistan, Tanzania, and the United States were among the first countries to have completed and published an external full assessment using the JEE tool.[10]

Taiwan's Commitment to Participation in JEE

Taiwan had a population of 23.5 million and a nominal gross domestic product (GDP) of $489.2 billion in 2013, when it was rated the world's 20th largest economy and the 5th largest economy in Asia. The 2003 SARS outbreak in East Asia, including Taiwan, and the subsequent public health threat from avian influenza, MERS, and Zika outbreaks in the region, have repeatedly tested Taiwan's preparedness for public health emergencies and illustrated the urgent need to establish a systematic approach to prevent, detect, and respond to human and animal disease threats, particularly through multilateral and multisectoral collaboration, including domestic and international partnerships.

Although not a member state of the WHO, Taiwan has demonstrated strong commitment to global health security activities. Examples include actively reporting cases to the WHO and sharing surveillance data with the relevant countries through the National IHR Focal Points since 2009, contributing funds and personal protective equipment to fight against Ebola in West Africa in 2014, and conducting Ebola, MERS, and Zika training courses for regional partners during 2015-16 in cooperation with the United States through the Taiwan-US Global Cooperation Training Framework (GCTF).[11]

In April 2015 (before the JEE tool was developed), the Taiwan Centers for Disease Control (CDC) officially announced that it would pursue an independent, external review of Taiwan's core capacities using the GHSA assessment tool. During the next 10 months, Taiwan CDC established a GHSA External Assessment Working Group in the Division of Planning and Coordination and an agency-level, interdepartmental GHSA External Assessment Task Force; secured funding and high-level support from the Executive Yuan and Ministry of Health and Welfare (MoHW); and requested expertise and guidance from the University of Pittsburgh Medical Center (UPMC) Center for Health Security (now the Johns Hopkins Center for Health Security) to prepare for the assessment. In February 2016, in light of the fresh release of the JEE tool, which already integrated the GHSA assessment tool, Taiwan CDC and the Center for Health Security promptly decided to switch to the JEE tool for this assessment.[12] Although the JEE Working Group (formerly named the GHSA External Assessment Working Group) remained in the division of planning and coordination of Taiwan CDC, because the JEE was used to assess Taiwan's collected capabilities, not just those of the MoHW or Taiwan CDC, the interdepartmental task force was expanded to a multisectoral team that involved other government agencies, such as authorities of agriculture, defense, border control, environmental protection, and nuclear power, to address elements that were not included in the GHSA assessment tool.

JEE Orientation

An External Assessment Team consisting of 5 US subject matter experts from the Center for Health Security and a former US CDC official conducted the evaluation in collaboration with the JEE working group and the multisectoral team of officials from relevant agencies from the government of Taiwan. The external evaluation process was planned to involve 2 separate visits to Taipei, Taiwan, including a pre–self-assessment orientation visit (pilot visit) in March 2016 and the evaluation mission, which took place in June 2016.

The 4-day pilot visit took place in Taiwan CDC's office building from March 29 to April 1, 2016. Three subject matter experts from the External Assessment Team were on the pilot visit. On day 1, the experts listened to briefings on Taiwan's response to Ebola and Zika threats, introduced the GHSA and JEE mission, and explained the proposed work plan for completing the JEE in Taiwan. The experts also discussed with Taiwan's JEE working group and multisectoral team the steps that would be required to conduct the self-assessment and the external evaluation using the JEE tool. On days 2 to 4, the experts met individual members of the multisectoral team who were responsible for specific JEE elements and discussed each of the technical and contextual questions contained in the respective sections of the JEE tool.

The indicators and questions listed in the JEE tool were in the original English text, not translated. The language used in the conversation was English, too, with occasional assistance from local staff in translating to Mandarin Chinese when needed. Before the discussion, individual members were instructed that, because the experts would talk them through each indicator and question to ensure full understanding, it was unnecessary for the members to answer or draft a response to the JEE questions in advance. The discussion of each element generally took at least 30 minutes, mostly involving clarification about what each question really means and how responses should be drafted. For example, elaborative answers that demonstrated real examples were preferred, rather than just yes or no answers. The members could also describe an example that came to mind and ask the experts if it was eligible for a response. Although most of the time the experts' answers would be “yes,” the Q&A process not only provided reassurance to the members who were not native English speakers but also helped engage the members in ownership of JEE implementation. Overall, the pilot visit was successful and well received.

At the end of the pilot visit, the 3 experts and Taiwan's JEE working group discussed next steps, jointly planned for work to be completed before the experts' next visit in June 2016, and developed a proposed agenda for the next visit.

JEE Self-Assessment

After the pilot visit, the JEE working group announced the launch of the JEE self-assessment on April 21. Members of the multisectoral team, supported by the respective authorities, were requested to submit point-by-point responses in English to the technical and contextual questions in the designated JEE element by May 16, 2016, to the JEE working group. Documents that served as attachments to the responses could be submitted in their original text and language, but each needed to be accompanied by a summary of content in English with the focus on its relevance to the specific JEE question. The team was asked not to provide a self-scoring result. Throughout the process, the JEE working group facilitated communication and coordination among all departments, forwarded additional queries that were raised by the multisectoral team members to the Center for Health Security for clarification, provided support with English translation if necessary, and compiled the self-assessment report. Four high-ranking Taiwan CDC officials, each responsible for 4 to 5 JEE elements, were asked to review the reports with the team members who drafted the responses to ensure appropriateness of the content and representativeness of the report at the national level. The experts of the Center for Health Security answered queries forwarded by the JEE working group to ensure consistent understanding of each JEE question.

The JEE Week

The external evaluation mission took place in Taiwan CDC's office building from June 21 to July 1, 2016, comprising 8 working days. During day 1 to day 7, the schedule was broken down into 19 JEE elements, with each session lasting 2 hours. The External Assessment Team reviewed the written self-assessment report and supporting documentation, interviewed the multisectoral team members who had drafted and presented the self-assessment results, and assigned scores for each of the indicators on a 5-point scale. In each 2-hour discussion, the External Assessment Team and Taiwan team members also developed consensus on Taiwan's strengths, limitations, recommended priority actions, and scores for the specific JEE element. To evaluate IHR capacities at different levels, the External Assessment Team conducted site visits to a regional health department and a regional teaching hospital in Taichung City, located in central Taiwan, and to the Taiwan CDC's Emergency Operation Center.

On the final day, the External Assessment Team presented to Taiwan's JEE working group and multisectoral team members a summary of key findings for each assessment area as well as a summary of needs for additional supporting documentation for certain indicators.

Publishing JEE Results

The results of the JEE for Taiwan were published on October 18, 2016, with the full text available at the websites of both the UPMC Center for Health Security and the Taiwan CDC.[13,14] On the same day, the findings of Taiwan's JEE were presented at an international public health symposium entitled “Assessing Countries' Global Health Security Capabilities,” convened by the Center for Health Security in Washington, DC.[15] Taiwan became the 8th country to publish the results of an external assessment using the JEE tool.

Lessons Learned

The JEE process in Taiwan in 2016 involved the participation of at least 19 government agencies or institutions. The key to success in achieving such large-scale multisectoral collaborations for JEE is the 10-month preparatory work beforehand: advocacy of the One Health concept among stakeholders, sensitization of high-level political leadership across levels and agencies, and the Taiwan government's strong commitment to transparency and accountability.

Previous self-assessment and external assessment experiences in Taiwan were useful in planning and advocating for the JEE. During 2011-12, Taiwan conducted a self-assessment of the national IHR core capacities using the WHO-issued assessment tool for core capacity requirements at airports, ports, and ground crossings and the protocol for assessing national surveillance and response capacities for the IHR (2005) that were released in October 2009 and in December 2010, respectively.[16] To commemorate the 10th anniversary of SARS, the government had further invited the UPMC Center for Biosecurity to conduct an external assessment of Taiwan's public health emergency preparedness in December 2012.[17] External assessment of IHR core capacity requirements at airports and seaports was conducted by invited Australian experts in March 2013.[16] These experiences not only helped to engage stakeholders in uptake of regular self and external review of relevant programs, but also helped build confidence and comfort among participants in honestly demonstrating both strengths and limitations to external reviewers without need for fear of being blamed or punished.

We also found the pilot visit of the external assessment experts to be extremely useful. Such face-to-face conversations substantially avoided confusion and misinterpretation of certain JEE questions and helped reduce anxiety among stakeholders, especially team members tasked to provide self-assessment results. The experts' reassuring emphasis on the importance of transparency, accountability, and ownership also echoed the values promoted by the Taiwan government and demonstrated the professionalism of the JEE process.

Prospects and Next Steps

The Taiwan government is dedicated to contributing to regional and global health security. The strengths recognized through the JEE might represent opportunities for Taiwan to help other countries prevent, detect, and respond to public health threats. On the other hand, limitations and challenges identified through the JEE highlight areas of core capacities that need prioritization for increased funding and other resource input for improvement.

Because the JEE is expected to be repeated every 4 to 5 years, the public health, animal health, and science and technology authorities have jointly proposed a 4-year project (2017-2020) with an annual budget of $7.5 million to address strengths and limitations identified through JEE with focus on elements such as antimicrobial resistance, zoonotic diseases, biosafety and biosecurity, the national laboratory system, real-time surveillance, and workforce development. Multisectoral collaborations will be substantially strengthened, including participation of nongovernmental organizations. As of November 2016, the proposal has been approved by the Executive Yuan, pending budget approval by the Legislative Yuan.

The leading agency for this JEE was Taiwan CDC. However, in light of the multisectoral nature of the JEE, Taiwan CDC plans to propose to the high-level political leadership that post-JEE monitoring mechanisms should be placed at an upper level, such as the Homeland Security Office of the Executive Yuan, which is authorized to oversee and coordinate all relevant ministries, agencies, and institutions. An annual self-assessment using the JEE tool is also recommended to help all stakeholders monitor the current status of progress of respective elements.

In conclusion, implementation of the JEE in Taiwan has led to findings that will be readily translated into actions and changes in the system. We encourage countries that have not conducted a JEE to consider and invest in this useful activity. Based on Taiwan's experiences, early stakeholder engagement and an experts' pre-JEE pilot visit would contribute to a successful JEE process.