Global Health Security Preparedness and Response: An Analysis of the Relationship between Joint External Evaluation Scores and COVID-19 Response Performance.

OBJECTIVES: The COVID-19 pandemic has highlighted the importance and complexity of a country's ability to effectively respond. The Joint External Evaluation (JEE) assessment was launched in 2016 to assess a country's ability to prevent, detect and respond to public health emergencies. We examined whether JEE indicators could be used to predict a country's COVID-19 response performance to tailor a country's support more effectively.
DESIGN: From April to August 2020, we conducted interviews with Centers for Disease Control and Prevention country offices that requested COVID-19 support and previously completed the JEE (version 1.0). We used an assessment tool, the 'Emergency Response Capacity Tool' (ERCT), to assess COVID-19 response performance. We analysed 28 ERCT indicators aligned with eight JEE indicators to assess concordance and discordance using strict agreement and weighted kappa statistics. Generalised estimating equation (GEE) models were used to generate predicted probabilities for ERCT scores using JEE scores as the independent model variable.
RESULTS: Twenty-three countries met inclusion criteria. Of the 163 indicators analysed, 42.3% of JEE and ERCT scores were in agreement (p value=0.02). The JEE indicator with the highest agreement (62%) was 'Emergency Operations Center (EOC) operating procedures and plans', while the lowest (16%) was 'capacity to activate emergency operations'. Findings were consistent with weighted kappa statistics. In the GEE model, EOC operating procedures and plans had the highest predicted probability (0.86), while indicators concerning response strategy and coordination had the lowest (≤0.5).
CONCLUSIONS: Overall, there was low agreement between JEE scores and COVID-19 response performance, with JEE scores often trending higher. JEE indicators concerning coordination and operations were least predictive of COVID-19 response performance, underscoring the importance of not inferring country response readiness from JEE scores alone. More in-depth country-specific investigations are likely needed to accurately estimate response capacity and tailor countries' global health security activities. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

To our knowledge, this is the first study to examine the Joint External Evaluation (JEE) in a systematic and methodical approach among multiple countries using an aligned scoring paradigm with another assessment.

This is also the first study to introduce a novel assessment tool specific to measuring a country’s COVID-19 emergency response capacity.

A limitation of our study is the alignment of two scoring systems (one with a five-point range and the other a three-point range), which impacted the accuracy of the newly aligned scoring system.

Another limitation of this study is bias from the Emergency Response Capacity Tool (ERCT) assessment because data collection was collected only from the perspective of Centers for Disease Control and Prevention country office staff.

Finally, the time gap between completion of the JEE assessment (2016–2018) and ERCT assessment (2020) and the alignment of the two for this specific study does not account for the socioeconomic and geopolitical events that may have occurred in between the time frames that may have affected response capacity.

Introduction

Since the emergence of SARS-COV-2, the virus that causes COVID-19, in December 2019, more than 216 million cases and approximately 4.5 million deaths have been reported globally as of August 2021.1 2 The COVID-19 pandemic highlights the importance of a country’s emergency response capacity to effectively control a novel public health threat.3 4 The pandemic has prompted local and regional lockdowns, varying levels of quarantine and social distancing measures and the redistribution of health resources from routine public health programmes to COVID-19 response efforts.5 Understanding a country’s level of preparedness can help support appropriate recommendations on resource allocation, establishment of policies and legislation, response planning and standard operating procedure development and personnel deployment.

In collaboration with the World Health Organization (WHO) and United Nations’ member states, the Global Health Security Agenda (GHSA) establishes a number of core capacities for preparing for and responding to global public health emergencies.6 In coordination with GHSA, the Joint External Evaluation (JEE) for health security was launched by WHO in 2016 as a voluntary, multisectoral, peer-to-peer evaluation. Using 49 indicators on a five-point scale, the JEE assesses a country’s ability to prevent, detect and respond to public health emergencies across 19 technical areas.7 8 The JEE assessment helps countries identify critical gaps within their public health systems by technical area, in order to prioritise actions to strengthen preparedness and response capacity.9 High JEE scores reflect intermediate to high capacity in responding to a public health emergency, and low JEE scores reflect low capacity in responding and weak or poor systems and processes.9 10

From April to August 2020, the United States Centers for Disease Control and Prevention (CDC) received requests from over 30 countries around the world for COVID-19 response capacity support. As countries responded and planned for ongoing SARS-COV-2 response activities, we questioned whether we could use existing assessments such as the JEE to inform critical areas that needed strengthening during the response. As JEE indicators are broad, often encompassing an amalgamation of multiple more detailed but critical components for emergency response capacity, to tailor specific technical support and interventions during COVID-19, CDC pursued the development of a new tool.7 Aligned to JEE indicators and scoring, CDC’s ‘Emergency Response Capacity Tool (ERCT)’ was developed for a systematic approach to assess and prioritise gaps in a country’s response capacity through examination of the country’s COVID-19 operational performance.11

As we used the tool, we wanted to assess whether we could have used the JEE, which is often conducted in peacetime, to predict how a country would respond to a public health event like COVID-19. To better understand this, we examined COVID-19 response performance in relation to specific JEE indicators to assess whether the JEE could be used to predict a country’s COVID-19 response capacity. We hypothesised that countries scoring lower in certain JEE indicators would continue to have challenges and deficits in responding to COVID-19, while countries with higher JEE scores would have responded more effectively to the current COVID-19 pandemic.

Methods

From April to August 2020, we used the ERCT to collect information on COVID-19 response performance in countries hosting a CDC country office meeting the following criteria: (1) requested CDC support for responding to the COVID-19 pandemic and (2) completed the JEE (version 1.0) between 2016 and 2018.7 The ERCT addresses and scores competencies in four technical areas: (1) public health systems integration; (2) multidisciplinary rapid response teams; (3) emergency operations centres (EOCs)/incident management system; and (4) risk communications and community engagement operations. The four competencies included a total of 28 indicators aimed at assessing a country’s emergency response systems, strategic planning, standard operating procedures and workforce capacity in responding to the COVID-19 pandemic.12–14 The ERCT scoring scale was 1–3. A score of ‘1’ indicated a country has no competency proficiency; ‘2’ indicated limited competency or proficiency; and ‘3’ indicated full competency or proficiency.

We requested CDC Country Office staff to complete the ERCT, scoring the country’s response performance according to the 28 indicators. During follow-up phone interviews, we reviewed provided scores with the CDC country office staff to ensure the indicator was interpreted correctly and the score accurately reflected the country’s response performance. Countries’ anonymity is maintained to protect disclosure of countries’ challenges and gaps in responding to the COVID-19 pandemic.

We obtained JEE scores from the JEE version 1.0 reports on the WHO’s website for the 23 countries included in this analysis.15 Country-specific scores for 49 JEE indicators were downloaded and merged into a single Microsoft Excel 2016 spreadsheet.16 The ERCT indicators were more specific and detailed with multiple ERCT indicators contributing to one JEE indicator. Four of the 49 JEE version 1.0 indicators aligned directly with the ERCT indicators, a ‘one-to-one’ alignment. For the remaining indicators, we calculated the mean ERCT score across the various detailed indicators that aligned to a single JEE indicator, a ‘grouped mean’ alignment (table 1).

Table 1

Alignment of the 28 detailed ERCT indicators with the eight JEE indicators using either a ‘one-to-one’ or a ‘grouped mean’ alignment mechanism

JEE Version 1.0 Indicator	ERCT Indicator	Alignment Mechanism
P.1.1 Legislation, laws, regulations, administrative requirements, policies or other government instruments in place are sufficient for implementation of the International Health Regulations.	1.1 Established national policies, directives regulatory documents and guidelines for information sharing.	One to one
R.1.2 Priority public health risks and resources are mapped and used.	1.13 Development of Threat and Hazard Identification and Risk Assessment to determine priority threats and hazards.	One to one
R.2.1 Capacity to activate emergency operations	1.4 Established working group with representation from key stakeholders to define the critical emergency management components and governance required to ensure coordination and information sharing during a response.	Grouped mean
	1.14 Identified external stakeholders and development of stakeholder engagement plan with linkages to the COVID-19 Strategic National Response Plan.	Grouped mean
	1.15 Determined capacity for involvement with partner and stakeholder agencies establishment of cross-sectional preparedness with key stakeholders and ministries.	Grouped mean
	3.2 Established terms of reference development for all Incident Management System (IMS) positions.	Grouped mean
	3.3 Identified staff to be rostered to fill and backup the above key IMS roles (or their equivalents) to support operations 24/7 if needed.	Grouped mean
	3.4 Rostered staff have received foundational training on emergency management and IMS.	Grouped mean
	3.5 Staff have been trained on core functions of IMS (operations, logistics, planning, finance and administrative, public information officer, liaison officer and safety officer).	Grouped mean
	3.6 Established structure for COVID-19 response is established and used for coordination.	Grouped mean
R.2.2 Emergency operations centre (EOC) operating procedures and plans	1.9 Development of functional reporting network and standard operating procedures (SOPs) in support to the EOC information flow.	Grouped mean
	4.1 Designated space for the public health EOC exists.	Grouped mean
	4.2 Public health EOC is equipped (eg, computers, telephones, etc) to function in a response.	Grouped mean
	4.3 Dedicated core team responsible for the operations of the public health EOC.	Grouped mean
	4.6 Development of concept of operations to articulate public health EOC relationship with other governmental sectors EOC.	Grouped mean
	4.7 Development of risk-based national health EOC plans and procedures.	Grouped mean
	4.9 Development of processes, procedures, protocols and SOPs for sharing information among IMS staff (notifications, reports, etc).	Grouped mean
R.2.3 Emergency operations programme	1.12 Established exercise and evaluation programme.	One to one
R.4.2 System is in place for sending and receiving health personnel during a public health emergency	2.1 Identified human resources to manage/support the rapid response team (RRT) in peacetime and response and its incorporation in the overall response coordination system.	Grouped mean
	2.4 Identified resources and mechanisms to ensure RRT safety, health and well-being including in RRT planning (eg, medical/disability/life insurance, medical care, mental healthcare, emergency evacuation etc) and inclusion in response plans.	Grouped mean
	2.5 Established RRT recruitment and onboarding standard operating procedures (eg, candidate inclusion/exclusion criteria, multisectoral/multidisciplinary candidate sources, database platform/variables, data collection, roster updates, mitigating roster attrition, etc).	Grouped mean
	2.7 Identified sustainable and trained multidisciplinary rapid response workforce/surge pool.	Grouped mean
	2.8 Development of predeployment standard operating procedures (eg, briefing, just-in-time training, equipment, etc).	Grouped mean
	2.9 Development of deployment standard operating procedures (eg, communication, reporting and team evolution, etc).	Grouped mean
	2.10 Development of postdeployment standard operating procedures (eg, demobilisation criteria, debriefs, etc).	Grouped mean
	3.3 Identified staff to be rostered to fill and backup the above key IMS roles (or their equivalents) in order to support operations 24/7 if needed.	Grouped mean
R.5.2 Internal and partner communication and coordination	1.7 Established universal information channels and coordination methods during a response	One to one
R.5.3 Public communication	1.10 Dedicated trained team responsible for risk communications and community mobilisation	One to one

ERCT, Emergency Response Capacity Tool; JEE, Joint External Evaluation.

Because the JEE score ranges from ‘1’ (indicating that implementation has not occurred) to ‘5’ (indicating that implementation has occurred, is tested, reviewed and exercised and that the country has a sustainable level of capability for the indicator) and the ERCT used scores of 1–3, we modified the scales to match for this analysis.15 A JEE score of 1 was matched to an ERCT score of 1, a JEE score of 2 and 3 was matched to an ERCT score of 2 and a JEE score of 4 and 5 was matched to an ERCT score of 3. To ensure accuracy in transforming the JEE indicator to the three-point ERCT scale, two authors independently examined the JEE and ERCT scoring criteria as well as interview qualitative data notes collected from the follow-up phone interviews. If a discrepancy was noted, then a third author would review the scoring and provide an adjudicated score. The final database included country-specific ‘transformed’ JEE scores and the ERCT score.

Data were captured and cleaned in Microsoft Excel 2016 and imported into R Studio for analysis.16 17 We conducted an agreement analysis to assess consistencies in JEE and ERCT indicator scores across the 23 countries. We initially calculated a strict agreement (transformed JEE score=ERCT score) analysis for all available indicators. Strict agreement is calculated as the percent of scores that were the same for the transformed JEE score and ERCT score of all possible scores. We additionally calculated weighted kappa statistics. The weighted kappa statistic accounts for random variability and closeness of agreement between ERCT and transformed JEE scores. A weighted kappa value above 0.2 generally reflects fair agreement; higher values suggesting stronger agreement.18 We calculated strict agreement and weighted kappa statistics for all indicator scores combined and then stratified by each JEE indicator, indicator score matching category (i.e., one to one or grouped mean) and the year the JEE was conducted.

We assumed that capacity should have remained the same or increased from the date of completing the JEE (2016–2018) and the date of implementing the ERCT in 2020. To control for this assumption, we created an additional binary variable, ‘J≤E’, defined as whether the JEE score≤ERCT score (yes/no). We used outcomes from generalised estimating equation (GEE) models to generate predictive probabilities of each JEE indicator score on the COVID-19 response capacity performance (ERCT scores), adjusting for possible correlations of country-specific scores across several indicators (ie, indicator scores are likely to be similar in a given country).19 We ran the GEE model including the ‘J≤E’ variable against each of the variables listed previously. From the GEE model estimated coefficients, we transformed them to predicted probabilities. This gave us predicted probabilities of concordance between JEE and ERCT scores for each JEE indicator. For the GEE model, we assigned the JEE indicator ‘EOC operating procedures and plans (R.2.2)’ as the reference indicator because R.2.2 was the only indicator with evidence of initial agreement.

As this research looked at a country’s overall performance during COVID-19 through the perspective of CDC country office staff, patients or the public were not involved in designing, conducting, reporting or the dissemination plans of our research.

Results

Twenty-three countries met the inclusion criteria; of these, 18 were regionally located in Africa, 4 in Southeast Asia and 1 in the Middle East. Countries had a combined average gross domestic product per capita in 2018 of US$2,332 (range: US$499–US$8,259).20

The final project database included 163 observations from the eight JEE indicators, which were successfully aligned to 28 ERCT indicators (figure 1). Of the 163 observations, agreement was highest (n=36) for JEE and ERCT indicator scores of ‘2’ (table 2). Agreement (n=28) was lowest for indicators with JEE scores of ‘2’ and ERCT scores of ‘1’.

Figure 1

Selection criteria for data included in final dataset.

Table 2

Agreement between the transformed (transformed Joint External Evaluation (JEE) scores is the alignment of the 28 detailed Emergency Response Capacity Tool (ERCT) indicators with the eight JEE indicators using either a ‘one-to-one’ or a ‘grouped mean’ alignment mechanism.) JEE and ERCT scores

		ERCT Score
		1	2	3	Total
JEE score	1	26	24	3	53
	2	28	36	9	73
	3	9	21	7	37
Total		63	81	19	163

Shaded regions indicate agreement or concordance between JEE and ERCT scores (explanation of scores: 1=no capacity, 2=partial capacity and 3=established capacity) (e.g., 1–1, 2–2, 3–3); unshaded areas reflect disagreement or discordance (e.g., 1–2,1-3, 2–1, 2–3, 3–1, 3–2)

The agreement between the JEE and ERCT scores across the 163 observations included in this analysis showed an agreement of 42.3% and a weighted kappa value of 0.134 (p-value=0.02) (table 3). On closer examination of the indicators, ‘EOC operating procedures and plans (R.2.2)’ had the highest strict agreement (61.9%) and weighted kappa (0.356, p-value=0.03), whereas ‘capacity to activate emergency operations (R.2.1)’ had the lowest strict agreement (15.8%) and weighted kappa (−0.125, p value=0.34). Of the discordant observations where JEE scores were not in strict agreement with ERCT scores (57.7% of all observations), 35.6% had a higher transformed JEE score and 22.1% had a higher ERCT score (0.134, p value=0.02) (table 3).

Table 3

Strict agreement and weighted kappa statistics for all observations, by transformed JEE indicator and ERCT scores, and JEE year.

	No. of Observations	Agreement ERCT=JEE (%)	Disagreement ERCT >JEE (%)	Disagreement ERCT <JEE (%)	Weighted Kappa (p-value)
Overall	163	42.3	22.1	35.6	0.134 (0.02)
R.2.2 EOC operating procedures and plans	21	61.9	23.8	14.3	0.356 (0.03)
P.1.1 Legislation, laws, regulations, administrative requirements, policies or other government instruments in place are sufficient for implementation of International Health Regulations (IHR)	20	50.0	5.0	45.0	0.141 (0.22)
R.5.3 Public communication	23	47.8	17.4	34.8	0.115 (0.46)
R.5.2 Internal and partner communication and coordination	22	40.9	9.1	50.0	−0.075 (0.42)
R.2.3 Emergency operations programme	20	40.0	10.0	50.0	0.103 (0.45)
R.4.2 System is in place for sending and receiving health personnel during a public health emergency	20	40.0	35.0	25.0	0.051 (0.76)
R.1.2 Priority public health risks and resources are mapped and used	18	38.9	16.7	44.4	0.201 (0.19)
R.2.1 Capacity to activate emergency operations	19	15.8	63.2	21.1	−0.125 (0.34)
One to one	103	43.7	11.7	44.7	0.161 (0.01)
Grouped mean	60	40.0	40.0	20.0	0.086 (0.37)
2016	59	44.1	15.3	40.7	0.136 (0.12)
2018	16	43.8	6.2	50.0	0.137 (0.33)
2017	88	40.9	29.5	29.5	0.139 (0.08)

EOC, emergency operations centre; ERCT, Emergency Response Capacity Tool; JEE, Joint External Evaluation.

In our stratified analysis, JEE indicator scores were generally higher than the corresponding ERCT scores (table 3). One-to-one matching had a concordance of 43.7% (p-value=0.01) and a higher transformed JEE score of 44.7% with a weighted kappa below 0.2 in comparison with the grouped mean. There was no statistical significance in variance in agreement among the years that the JEE was completed despite the slightly higher agreement (44.1%) but low weighted kappa statistic (0.14) in 2016.

In the GEE model, JEE indicators were the predictor, and J≤E was the outcome (figure 2). From the GEE model, the highest predicted probability of agreement was ‘EOC operating procedures and plans (R.2.2)’ with a predicted probability of 0.86. The lowest predicted probability of agreement was ‘internal and partner communication and coordination (R.5.2)’ and ‘emergency operations programme (R.2.3)’ with a predicted probability of 0.5.

Figure 2

The generalised estimating equation (GEE) model calculated the predicted probability for each Joint External Evaluation (JEE) indicator tested with a 95% CI. Explanation of labels on the x-axis: R.2.2: emergency operations centre (EOC) operating procedures and plans, R.2.1: capacity to activate emergency operations, R.4.2: system is in place for sending and receiving health personnel, R.5.3: public communication, R.1.2: priority public health risks and resources are mapped and used, P.1.1: legislation, laws, regulations, administrative requirements, policies or other government instruments in place are sufficient for implementation of International Health Regulations, R.5.2: internal and partner communication and coordination and R.2.3: emergency operations programme.

Discussion

Capacity to respond to the COVID-19 pandemic is multifactorial and complex—varying by context, existing resources, priority areas, and historical challenges. We developed and implemented the ERCT to assess several competencies related to response performance. Findings from the ERCT were often discordant with scores generated from previously conducted JEEs, where the transformed JEE scores (35.6%) were often higher than the ERCT scores (22.1%). With the 2–4 years from when the JEE was conducted, we expected there to be a similar or increase in capacity between the time of the completion of the JEE and when the ERCT assessment was conducted. However, the overall low agreement (42.3%) between the two assessments, JEE and ERCT, could have resulted from several factors. These could include: the data collection method of both assessments, the timeframe in which both were conducted, and the lack of congruency between the assessments.

For the ERCT, CDC country office staff completed the assessment tool and rated the country’s response performance, whereas the JEE is completed by a JEE team in-country composed of multisectoral external and internal subject matter experts. When examining the timeframe of both assessments, the ERCT was conducted during an active response between April to August of 2020, while the JEE was conducted at a specific point in time between 2016 and 2018, prior to the start of the COVID-19 pandemic. Furthermore, JEE indicators are quite broad with multiple emergency response operational factors included under one JEE indicator, which may have affected the specificity of the JEE scoring. Additionally, the ERCT scores assessed response performance specifically to COVID-19, whereas the JEE is not specific to one particular emergency and is not conducted during an active public health emergency event. Regardless of the underlying factors, this trend may indicate that a more detailed competency analysis, such as the ERCT, may be required for these particular indicators to provide a more accurate assessment of a country’s ability during a response, specifically in the context of COVID-19.

At the individual indicator level, the high agreement between specific indicators was notably in the capacity related to EOC operating procedures and plans, which showed the highest strict agreement and predicted probability between JEE and ERCT scores. These indicators are tangible and discrete (e.g., EOC plans exist or do not exist, EOC activation occurs or does not occur) and thus may lend themselves to be more easily assessed and measured prior to a large-scale response. Conversely, those indicators related to strategic planning (e.g., legal authority, policies, communication and partner coordination) were more discordant and generally received lower ERCT scores than transformed JEE scores. The identified trends and questions raised in this investigation highlight the importance of future studies to continue investigating this concordance to inform countries on how best to plan for global health security activities and prioritise their emergency response capacity development and implementation.

This initial investigation of the role of JEE indicators in predicting the ability to respond effectively to COVID-19 included several limitations. First, because of the higher specificity of the ERCT indicators, the JEE and ERCT scorings and indicators needed to be adjusted and aligned, respectively, which could have contributed to the low agreement between the scores. The alignment of two scoring systems (one with a five-point range and the other a three-point range) may impact the accuracy of the scoring system for the JEE and ERCT adjustments to account for a proper depiction of indicators. Furthermore, the more detailed ERCT indicators required the mean ERCT score to be taken across various indicators to align to a single JEE indicator. Second, the data collection for the ERCT assessment was from the perspective of CDC country office staff only. This potentially creates an external view bias, as well as a limited perspective compared with JEEs, which are scored based on multiple diverse subject matter experts and the government’s own assessment. Third, there was a gap in time between the JEE assessment (2016–2018) and ERCT assessment (2020). Although we tried to control for this with the development of the ‘J≤E’ variable, there may have been changes in capacity in that time frame (e.g., socioeconomic and geopolitical events) that we could not account for in this investigation. Finally, this analysis included only 23 countries selected through convenience sampling (i.e., countries requesting CDC assistance, and thus, these trends may not be representative of all countries).

Conclusion

This analysis offers a novel opportunity to examine COVID-19 predicted response capacity across several countries. Although limited in sample size to make conclusive statements, this analysis included geographically and economically diverse countries, which may indicate applicability beyond the countries sampled for this investigation. Despite the number of limitations highlighted in this study, especially due to operational research studies being difficult to translate capacity building efforts to transformed response operations, this is the first study to examine the JEE in a systematic and methodical approach among multiple countries using an aligned scoring paradigm with another assessment. The trend of ERCT scores being lower than JEE scores underscores the need for a country’s vigilance when inferring their strategic response readiness from JEE scores alone and in allocating resources for global health security initiatives. This trend, along with concordance variability among JEE indicators, warrants further investigation to assess response capacity and its relationship to response performance to better understand preparedness and capacity measures translated to broader public health outputs and outcomes. Additionally, this investigation may indicate the need to re-examine some of the JEE indicator’s specificity and accuracy in assessing a country’s capacity, especially concerning strategic response planning. As countries around the globe undergo the JEE process and use it to determine and prioritise their global health security activities, we believe understanding the relevance of the results during an active and specific public health event is of utmost importance from the country-level perspective and to the larger global health response community supporting the JEE initiative.