Clinical and epidemiological performance of WHO Ebola case definitions: a systematic review and meta-analysis.

BACKGROUND: Ebola virus disease case definition is a crucial surveillance tool to detect suspected cases for referral and as a screening tool for clinicians to support admission and laboratory testing decisions at Ebola health facilities. We aimed to assess the performance of the WHO Ebola virus disease case definitions and other screening scores.
METHODS: In this systematic review and meta-analysis, we searched PubMed, Scopus, Embase, and Web of Science for studies published in English between June 13, 1978, and Jan 14, 2020. We included studies that estimated the sensitivity and specificity of WHO Ebola virus disease case definitions, clinical and epidemiological characteristics (symptoms at admission and contact history), and predictive risk scores against the reference standard (laboratory-confirmed Ebola virus disease). Summary estimates of sensitivity and specificity were calculated using bivariate and hierarchical summary receiver operating characteristic (when four or more studies provided data) or random-effects meta-analysis (fewer than four studies provided data).
FINDINGS: We identified 2493 publications, of which 14 studies from four countries (Sierra Leone, Guinea, Liberia, and Angola) were included in the analysis. 12 021 people with suspected disease were included, of whom 4874 were confirmed as positive for Ebola virus infection. Six studies explored the performance of WHO case definitions in non-paediatric populations, and in all of these studies, suspected and probable cases were combined and could not be disaggregated for analysis. The pooled sensitivity of the WHO Ebola virus disease case definitions from these studies was 81·5% (95% CI 74·1-87·2) and pooled specificity was 35·7% (28·5-43·6). History of contact or epidemiological link was a key predictor for the WHO case definitions (seven studies) and for risk scores (six studies). The most sensitive symptom was intense fatigue (79·0% [95% CI 74·4-83·0]), assessed in seven studies, and the least sensitive symptom was pain behind the eyes (1·0% [0·0-7·0]), assessed in three studies. The performance of fever as a symptom varied depending on the cutoff used to define fever.
INTERPRETATION: WHO Ebola virus disease case definitions perform suboptimally to identify cases at both community level and during triage at Ebola health facilities. Inclusion of intense fatigue as a key symptom and contact history could improve the performance of case definitions, but implementation of these changes will require effective collaboration with, and trust of, affected communities. FUNDING: Médecins sans Frontières.

Introduction

Ebola virus disease case definition is a crucial surveillance tool to detect suspected cases for referral and as a screening tool for clinicians to support admission and laboratory testing decisions at Ebola health facilities. However, there have been long-standing concerns about the poor performance of the WHO Ebola virus disease case definitions, including the inability to distinguish Ebola virus disease from common diseases such as malaria and typhoid fever.1, 2, 3

The scale of the 2014–16 west African Ebola epidemic further challenged the operational use and validity of the WHO case definitions in detecting suspected cases at the community level and allocating patients appropriately to high-risk or low-risk wards for testing at specialised isolation centres. Consequently, during and since this epidemic, organisations involved in the Ebola virus disease response have estimated the sensitivity and specificity of the WHO case definitions and its constituent symptoms and signs, and developed alternative definitions and risk scores to identify clinical and epidemiological factors that could predict infection under outbreak conditions.5, 6 Discordance on the use of WHO Ebola virus disease case definitions with consequent delay on outbreak control and community disengagement have been reported in west Africa and, in the current outbreak, in the Democratic Republic of the Congo along with its bordering countries.7, 8, 9

However, the operational use and performance of those definitions and risk scores has not been rigorously evaluated. Such an evaluation is needed to guide communities and public health practitioners to improve the effectiveness and efficiency of identification and management of suspected cases during Ebola virus disease responses.

Evidence before this study

There have been long-standing concerns about the poor performance of WHO case definitions for Ebola virus disease, including their inability to distinguish Ebola virus infection from common tropical diseases. We did a systematic search of the scientific literature using PubMed, Scopus, Embase, and Web of Science, without regional restrictions, for research articles published in English between June 13, 1978, and Jan 14, 2020. We used the search terms “Ebola”, “EVD infection”, “case definition”, “admission symptoms”, “sensitivity”, “specificity”, “likelihood”, “score”, “classification”, “validity” and “performance”. We also contacted relevant experts. We found that different organisations have attempted to assess the performance of WHO Ebola case definitions and developed alternative definitions and risk scores. However, there has been no systematic and rigorous evaluation of those studies. Such an evaluation is needed to guide communities and public health practitioners to improve the effectiveness and efficiency of identification and management of suspected cases during an Ebola virus disease outbreak.

Added value of this study

To our knowledge, this study is the first systematic review and meta-analysis that assesses the performance of the WHO Ebola virus disease case definitions, and other clinical and epidemiological characteristics such as symptoms and signs at admission and contact history, against the reference standard (laboratory confirmation of Ebola virus infection). Our analysis provides the most comprehensive evidence on the limitations of WHO case definitions and its constituent symptoms and signs, and predictive risk scores. We show that the WHO case definitions perform suboptimally to identify cases at both the community level and during triage at general and specialist health facilities. The performance of fever as a symptom varied depending on the cutoff used to define fever. The most sensitive symptom was intense fatigue. History of contact was a key predictor for the WHO case definitions and for risk scores. This study identifies important gaps related to the paediatric and pregnant population and highlights the need to use consistent thresholds (eg, for fever) to explore viraemia and symptoms at admission, and to externally validate risk scores for Ebola virus infection.

Implications of all the available evidence

Inclusion of intense fatigue as a key symptom could improve the sensitivity, the primary requirement for community-based screening, of WHO and alternative case definitions. Inclusion of contact history will improve specificity, resulting in a lower number of false positives and thus a lower number of unnecessary admissions to Ebola health facilities. These improvements will contribute to reduced isolation from family, fear of being stigmatised, delay to appropriate care, and community mistrust in response activities.

We aimed to assess the performance of the WHO Ebola virus disease case definitions and other clinical and epidemiological characteristics, such as symptoms and signs at admission and contact history, as the index test or test under assessment, against the reference standard of laboratory-confirmed Ebola virus infection.

Methods

Search strategy and selection criteria

For this systematic review and meta-analysis, we searched PubMed, Scopus, Embase, and Web of Science, without regional restrictions, for studies in English published between June 13, 1978 (when the first Ebola virus disease outbreaks were reported on), and Jan 14, 2020.10, 11 We also endeavoured to capture data on the current outbreak of Ebola virus disease in the Democratic Republic of the Congo by contacting relevant people involved in the response.

The search terms included “Ebola”, “EVD infection”, “case definition”, “admission symptoms”, “sensitivity”, “specificity”, “likelihood”, “score”, “classification”, “validity”, and “performance” (appendix pp 5–6).

We included observational retrospective studies that estimated the sensitivity and specificity of WHO Ebola virus disease case definitions and other clinical and epidemiological characteristics (symptoms and signs at admission and contact history) against the reference standard (laboratory confirmation of Ebola virus infection), and studies that developed, or externally validated, predictive risk scores (based on a combination of symptoms and signs, and epidemiological information) to predict the risk of being positive for Ebola virus. We also included studies looking at sensitivity and specificity of WHO case definitions for Ebola or Marburg virus infections because they belong to the same family of viruses (Filoviridae) and share the same case definitions, and the reference standard is laboratory confirmation of infection. We excluded studies on the sensitivity and specificity of diagnostic tests, animal and vaccine studies, studies of survivors of Ebola virus disease, and studies on predictors of outcomes or severity of Ebola virus disease, community surveillance, and outbreak and clinical management. Studies specifically on frequency of symptoms at admission were also excluded as a previous review exists.

Two reviewers (GC and FT) independently screened all titles and abstracts to identify those meeting the selection criteria, and a third author (LI) arbitrated for studies without consensus. A full-text review was then done for these articles, and their bibliographies were assessed for other eligible studies. We extracted data on author, year of publication, country, virus, period of data collection, study design, study objective, outcomes measured, setting in which data were collected (eg, Ebola treatment centres), age of population included in the study, study size including number of patients who were negative and positive for Ebola virus, diagnostic method, limitation of individual studies, and performance of the WHO Ebola virus disease case definitions, and individual symptoms and signs, and epidemiological links or contact history with known patients with Ebola virus disease.

Performance data extracted included sensitivity, specificity, predictive values and risk score, and area under the receiver operating characteristic (ROC) curve (AUC). We developed a spreadsheet to compile extracted data based on the Cochrane data tool. The primary data extracted from each article were checked by a second researcher (FT). No protocol was developed for this study.

WHO Ebola virus disease case definitions were used to define suspected, probable, and confirmed cases, which varied by context and period of outbreak. In 2014 in Sierra Leone, WHO included miscarriage as an additional symptom (eg, abdominal pain) or sign (eg, vaginal bleeding) to the existing definitions.12, 15 For paediatric populations, the modified WHO case definition used in Sierra Leone was evaluated (figure 1 ).

Figure 1

WHO Ebola virus disease case definitions for all ages and the paediatric population

Data analysis

We derived the numbers of true positive, false negative, true negative, and false positive cases in each study using data provided in each article for each symptom and sign, and WHO Ebola virus disease case definition. Sensitivity and specificity are correlated, and univariate measures of heterogeneity, such as I 2, are not suitable to report heterogeneity in diagnostic test accuracy reviews. We used bivariate and hierarchical summary ROC (HSROC) models for meta-analysis.17, 18

The bivariate model provides estimation of a summary of sensitivity and specificity, whereas the HSROC model provides the estimation of a summary curve from studies that have used different thresholds, the 95% confidence region for the summary point, and the 95% prediction region. The prediction region graphically illustrates between-study heterogeneity as well as the bivariate relationship between sensitivity and specificity. Only studies that used comparable thresholds, symptoms and signs, or definitions were combined using these methods.

Given that HSROC models cannot be fitted when there are data from fewer than four studies, for some symptoms and signs we did a random-effects meta-analysis to calculate pooled estimates for sensitivity and specificity. Compared with bivariate and hierarchical models, pooled estimation from random-effects meta-analysis could slightly overestimate point estimation, so estimates from the random-effects model are provided for completeness.

We summarised, without any further re-analysis, studies that developed or externally validated risk scores for predicting Ebola virus infection. Scores were used to identify individuals with a higher or lower risk of Ebola virus infection during screening at Ebola health facilities. To obtain the risk scores, these studies used the regression coefficients of independent risks obtained by multivariable logistic regression against Ebola virus infection and then converted regression coefficients into an integer-based point-scoring system. Reviewed studies assigned positive and negative risk scores with calculated AUC to epidemiological, demographic, and clinical characteristics. Positive values indicated higher risk of Ebola virus infection and negative values indicated higher risk of another infection such as malaria or typhoid.

Values assigned to the risk score varied by study; therefore, a meta-analysis of risk scores was not done, but instead evidence was systematically reviewed. For comparability, we reclassified the risk scores reported in the included studies into categories, from very low risk to very high risk (appendix p 7). STATA 15 was used for statistical analysis.

PRISMA guidelines for Diagnostic Test Accuracy Studies (PRISMA-DTA) were followed (appendix pp 2–4).

Role of the funding source

GC, KL, AS, and JG were employed by the funder, and participated in planning the study, carrying out the research, and writing the report. The funder of the study had no further role in study design, data collection, data analysis, data interpretation, or writing of the report. The corresponding author had full access to all the data in the study and had final responsibility for the decision to submit for publication.

Results

Of the 2493 studies initially screened using the article title, 143 were deemed to be potentially eligible on the basis of the abstract, and their full-text articles were assessed. Of these studies, 18 met the inclusion criteria, but three were excluded because data on sensitivity and specificity could not be extrapolated (appendix p 8). One was excluded because it is yet unpublished (FG). Of the 14 included studies, 11 were full manuscripts,5, 6, 22, 24, 25, 27, 29, 30, 31, 32, 33 one a letter, one an oral plenary abstract, and one a conference poster (the author of this poster was also contacted and they provided an abstract with additional data [Kuehne A, Epicentre, Paris, France, personal communication]; table 1 ). 13 studies were published between 2015 and 2019 and assessed Ebola virus disease in the west Africa outbreak (seven in Sierra Leone,5, 6, 25, 26, 30, 32, 33 four in Guinea,24, 27, 28, 31 and two in Liberia23, 29). The remaining article was published in May, 2010, assessing Marburg virus in Angola.

Table 1

Overview of articles included in the systematic review and meta-analysis

	Country	Virus	Period of data collection	Design	Objective	Outcomes	Setting of data collection	Age of study population	Patients positive for Ebola virus/total patients	Method (reference standard) and timing of Ebola virus confirmatory testing	Limitations
Roddy et al (2010)22	Angola	Marburg	March–July, 2005	Observational retrospective study of data at admission	Evaluate the diagnostic validity of individual patient clinical and epidemiological characteristics and WHO-recommended case definitions for Marburg haemorrhagic fever, and develop a data-derived diagnostic algorithm for Marburg haemorrhagic fever that improves the WHO-recommended definitions	Sensitivity and specificity of WHO case definition, WHO case subdefinitions, symptoms at admission, and epidemiological link; and risk score	Screening at one hospital	All ages	41/102	Quantitative PCR on admission	Small sample; only saw patients at admission; data only captured Marburg haemorrhagic fever; hospital-based data collection; detailed data not available for all Marburg haemorrhagic fever cases; only presenting symptoms were recorded; highlights the necessity of collecting high-quality clinical and epidemiological data during outbreaks; over-representation of individuals with more serious symptoms that required hospital admission; no reported validation (external or internal)
Kuehne et al (2015)23	Liberia	Ebola	August, 2014–March, 2015	Observational retrospective study of data at admission and clinical results	Study the discriminative accuracy (sensitivity, attributable frequency, diagnostic test odds ratio, area under the receiver operating characteristic curve) of clinical signs, contact history, and combinations thereof	Sensitivity and specificity of WHO case subdefinitions, symptoms at admission, and epidemiological link; and risk score	One Ebola treatment centre	All ages	1235/1832	Quantitative PCR on admission	Reporting bias; poor data quality; conference poster and abstract data (Kuehne A, Epicentre, Paris, France, personal communication); no reported validation (external or internal)
Levine et al (2015)29	Liberia	Ebola	September, 2014–January, 2015	Observational retrospective study of data at admission	Develop a clinical prediction model that can help to guide care for patients with suspected Ebola virus disease, provide specific parameters for isolation and admission to treatment centres, and maximise resource use	Sensitivity and specificity of WHO case definition, symptoms at admission, and epidemiological link; and risk score	One Ebola treatment centre	All ages	160/382	Quantitative PCR on admission	Data collected only at admission, different stages of disease process; data might not be representative of all patients with Ebola virus disease; poor data quality; small sample; patients pre-screened by Ebola treatment units (ambulance travel); only assessed 14 variables; no reported external validation, only internal validation
Lado et al (2015)5	Sierra Leone	Ebola	May, 2014–December, 2014	Observational retrospective study of data at admission	Identify clinical characteristics that were predictive of Ebola virus disease diagnosis and assess the accuracy of suspected Ebola virus disease case definitions	Sensitivity and specificity of WHO case definition, WHO case subdefinition, symptoms at admission, and epidemiological link	One Ebola holding unit	All ages	464/724	Quantitative PCR on admission	Small sample; poor accuracy on reporting of symptoms and history; no access to patients who chose not to present to hospital or did not have access; no reported validation (external or internal)
Arranz et al (2016)30	Sierra Leone	Ebola	December, 2014–March, 2015	Observational retrospective study of data at admission	Compare the clinical characteristics of confirmed cases (patients with Ebola virus disease) and non-confirmed cases (patients without Ebola virus disease), assess the diagnostic validity of initial symptoms used in WHO case definition to diagnose Ebola virus disease in a low-incidence situation	Sensitivity and specificity of WHO case definition, WHO case subdefinition, symptoms at admission, and epidemiological link	One Ebola treatment centre	All ages	31/75	Quantitative PCR on admission	Only data at admission; poor data quality; retrospective design; small sample; no reported validation (external or internal)
Loubet et al (2016)31	Guinea	Ebola	December, 2014–February, 2015	Observational retrospective study of data at admission	Identify epidemiological, sociodemographic, and clinical variables associated with Ebola virus disease diagnosis and to create, based on these variables, a predictive score for identification of confirmed Ebola virus disease	Sensitivity and specificity of WHO case definition, WHO case subdefinition, symptoms at admission, and epidemiological link; and risk score	One Ebola treatment centre	All ages	76/145	Quantitative PCR on admission	Data collected only at admission; poor data quality; retrospective design; patients included might have been reluctant to come to the Ebola treatment centre, and thus were more likely to present severe clinical presentation with late symptoms; temperature taking might be affected by several factors; small sample size; anorexia and temperature (the factors that in that study were associated with an increased likelihood of Ebola virus disease) are not easy to measure and interpret; no reported external validation, only internal validation
Hartley et al (2017)32	Sierra Leone	Ebola	December, 2014–November, 2015	Observational retrospective study of data at admission	Construct an easy-to-use and highly accurate triage scoring system that discriminates Ebola virus infection risk in a malaria-sensitive manner and improve the predictive accuracy for Ebola virus disease and malaria	Risk score	One Ebola virus treatment centre	All ages	158/566	Quantitative PCR on admission; rapid diagnostic malaria test (histidine-rich protein-II antigen rapid diagnostic kits were used)	Only the most prevalent symptoms at admission were included in the score; poor data quality; did not fully cover all the malaria season because the Ebola treatment centre was opened from December to June; recall bias
Fitzgerald et al (2017)26	Sierra Leone	Ebola	August, 2014–March, 2015	Observational retrospective study of data at admission	Refine the case definition and describe outcomes of admitted children	Sensitivity and specificity of WHO case subdefinitions	11 Ebola holding units	Paediatric population (younger than 13 years)	309/1006	Quantitative PCR on admission	Only included children younger than 13 years; oral plenary abstract; no reported external validation, only internal validation
Ingelbeen et al (2017)27	Guinea	Ebola	March, 2014–September, 2015	Observational retrospective study of data at admission	Describe the burden of non-cases in relation to the phase of the outbreak; determine the duration of their stay at the Ebola treatment centre and (potential) subsequent nosocomial infections; compare characteristics, outcome, and risk factors for death in confirmed cases and non-cases to improve the selection, diagnosis, and care of people with suspected Ebola virus disease	Sensitivity and specificity of WHO case subdefinitions and symptoms on admission	One Ebola treatment centre	All ages	822/2362	Quantitative PCR on admission; Xpert Ebola Assay (Cepheid GeneXpert, Sunnyvale, CA USA) on admission	The Ebola treatment centre for part of the outbreak was located within one hospital but then moved to another area in July; could not assess possible drivers for the large proportion of non-cases; no reported validation (internal or external)
Oza et al (2017)33	Sierra Leone	Ebola	November, 2014–March, 2015	Observational retrospective study of data at admission	Develop two Ebola risk scores to supplement the broad WHO case definition by further separating triaged patients based on their likelihood of being positive for Ebola virus	Risk score	One Ebola treatment centre	All ages	252/424	Quantitative PCR on admission; biochemistry laboratory tests with the Piccolo Xpress (Abaxis, Union City, CA, USA) and i-STAT (Abbott Point of Care, Princeton, NJ, USA) device	Only one treatment centre; investigated 14 commonly recorded symptoms; small amount and poor quality of patient data; excluded exposure as a potential predictor because of large amount of missing data or poor data quality; patients might not be representative of the overall population of suspect Ebola cases; no reported external validation, only internal validation
Hsu et al (2018)24	Guinea	Ebola	March–October, 2014	Observational retrospective study of surveillance data	Assess the diagnostic performance of the WHO suspected case definition by using epidemiological surveillance and diagnostic test	Sensitivity and specificity of WHO case definition, WHO case subdefinition, symptoms at admission, and epidemiological link	National surveillance line list	All ages	1304/2847	Quantitative PCR (on admission and for deceased patients at the community level)	Unknown how representative the database was for all patients with Ebola virus disease; only 1412 patients had complete data to assess and analyse the WHO case definition; possible overestimation of performance of WHO definition because only common symptoms were recorded in the early stage of the outbreak; poor data quality; no reported validation (internal or external)
Fitzgerald et al (2018)25	Sierra Leone	Ebola	August, 2014–March, 2015	Observational retrospective study of data at admission	Develop a predictive score that could be used to tailor the paediatric case definition for suspected Ebola virus disease according to the clinical and epidemiological setting	Sensitivity and specificity of WHO case definition and risk score	11 Ebola holding units	Paediatric population (younger than 13 years)	309/1006	Quantitative PCR on admission	Only included children younger than 13 years; poor data quality; no data on the true Ebola status of people who did not meet the WHO case definition and were not admitted; no reported validation, only internal validation
Ingelbeen et al (2018)28	Guinea	Ebola	March, 2014–September, 2015	Observational retrospective study of data at admission	Validate risk score by Oza and colleagues13	Risk score	One Ebola treatment centre	All ages	805/2311	Quantitative PCR on admission; Xpert Ebola Assay (Cepheid GeneXpert) on admission	Did not propose another algorithm; letter; no reported external validation, only internal validation
Huizenga et al (2019)6	Sierra Leone	Ebola	September, 2014–November, 2015	Observational retrospective study of data at admission	Evaluate the pre-existing health-care infrastructure during the Ebola virus disease outbreak, and assess the provided health care and safeguard functionality of a health-care system for all patients not suspected to have or diagnosed with Ebola virus disease	Sensitivity and specificity of WHO case subdefinitions	Screening at one hospital	All ages	22/1556	Quantitative PCR on admission	Scant description of data; poor data quality; no reported validation (external or internal)

Overall, 12 021 people with suspected disease were included, of whom 4874 were confirmed as positive for Ebola virus infection. Study populations varied from 75 to about 2847 (table 1). All studies, apart from the national surveillance study, included patients who presented alive to health facilities for assessment. The national surveillance study included all cases (suspected, probable, and confirmed), including patients both alive and deceased, identified in both the community and health facilities. Eight studies' data were from single Ebola treatment centres,23, 27, 28, 29, 30, 31, 32, 33 with the remaining using a national surveillance list, three from Ebola holding units,5, 25, 26 and two from hospitals screening patients for Ebola virus disease while still functioning as general health facilities.6, 22 All studies covered distinct patient groups from different periods and geographical areas, except for two studies from Guinea.24, 27 Although these two studies covered overlapping patient groups, they reported on different clinical and epidemiological characteristics (WHO case definition performance vs symptom performance).24, 27

All selected manuscripts analysed all ages combined, except one author who assessed, in two different studies, the sensitivity and specificity of 2014 WHO Ebola case definitions and also developed a risk score specifically for the paediatric population (younger than 13 years).25, 26

Six studies explored the performance of a WHO case definition in non-paediatric populations.5, 22, 24, 29, 30, 31 In all of these studies, suspected and probable cases were combined and could not be disaggregated for analysis. The following results therefore apply to this combined group of suspected and probable cases. The pooled sensitivity was 81·5% (95% CI 74·1–87·2) and pooled specificity was 35·7% (28·5–43·6; figure 2 ). One study assessed WHO 2014 case definitions for a paediatric population (younger than 13 years old); the sensitivity was 98·0% (95% CI 95·0–99·0) and specificity was 5·0% (3·0–7·0).

Figure 2

HSROC summary of sensitivity and specificity

HSROC=hierarchical summary receiver operating characteristic.

When WHO subdefinitions were assessed, history of contact and symptoms had high specificity compared with clinical symptoms alone, ranging from 62·3% (95% CI 49·8–73·5) to 94·4% (95% CI not provided in original paper; table 2 ). The highest sensitivity (100·0%) was documented for the WHO subdefinitions in which fever was not mandatory. Among studies using clinical symptoms and signs alone, the definition including three or more symptoms (intense fatigue, confusion, conjunctivitis, hiccups, diarrhoea, and vomiting) had the highest specificity (79·1% [95% CI not provided in original paper]). Unexplained death had high specificity (92·8% [95% CI not provided in original paper]) but the lowest sensitivity (14·2% [95% CI not provided in original paper]; table 2).

Table 2

Sensitivity and specificity of WHO Ebola virus disease subdefinitions against reference standard of laboratory-confirmed Ebola virus infection, in decreasing order of sensitivity

	WHO subdefinition	Sensitivity (95% CI)	Specificity (95% CI)	Positive predictive value (95% CI)	Negative predictive value (95% CI)
Huizenga et al (2019)6	WHO definition, with the difference that fever with sudden onset is not a mandatory criterion	100·0%	42·5%*	2·4%*	100·0%
Fitzgerald et al (2017)26	Contact alone, fever (in children older than 2 years) OR fever and conjunctivitis (in children younger than 2 years)	94·0%*	35·0%*	Not provided	Not provided
Roddy et al (2010)22	Epidemiological link or a combination of myalgia or arthralgia and any haemorrhage	79·0% (64·0–91·0)	73·0% (60·0–84·0)	Not provided	Not provided
Loubet et al (2016)31	WHO subdefinition 2 (temperature ≥37·5°C plus risk factor†)	75·0% (63·5–83·9)	62·3% (49·8–73·5)	Not provided	Not provided
Roddy et al (2010)22	WHO case definition (clinical criteria only‡)	73·0% (57·0–86·0)	43·0% (30·0–56·0)	Not provided	Not provided
Roddy et al (2010)22	Fever plus three or more symptoms§	68·0% (52·0–82·0)	46·0% (33·0–59·0)	Not provided	Not provided
Loubet et al (2016)31	Temperature ≥38·5°C plus risk factor†	68·4% (56·6–78·3)	82·6% (71·2–90·3)	Not provided	Not provided
Arranz et al (2016)30	Contact and three symptoms§	67·7% (51·3–84·2)	81·8% (70·4–93·2)	72·4% (56·1–88·7)	78·3% (66·3–90·2)
Loubet et al (2016)31	WHO subdefinition 3 (temperature ≥37·5°C plus clinical symptoms§)	67·1% (55·2–77·2)	76·8% (64·8–85·8)	Not provided	Not provided
Loubet et al (2016)31	WHO subdefinition 1 (risk factor plus clinical symptoms§)	63·2% (51·3–73·7)	66·7% (54·2–77·3)	Not provided	Not provided
Lado et al (2015)5	Three or more major symptoms¶	57·8% (52·1–61·4)	70·8% (64·7–76·4)	77·9% (73·1–82·3)	47·5% (42·3–52·7)
Arranz et al (2016)30	Fever and three symptoms§	58·1% (40·7–75·4)	50·0% (35·2–64·8)	45·0% (29·6–60·4)	62·9% (46·8–78·9)
Hsu et al (2018)24	Clinical criteria§	57·2%*	62·0%*	66·4%*	52·5%*
Ingelbeen et al (2017)27	WHO case definition (clinical criteria only‖)	56·9%*	46·4%*	36·3%*	66·8%*
Roddy et al (2010)22	Epidemiological link and two or more general symptoms§	54·0% (37·0–70·0)	91·0% (80·0–97·0)	Not provided	Not provided
Roddy et al (2010)22	Epidemiological link and three or more general symptoms§	54·0% (37·0–70·0)	93·0% (83·0–98·0)	Not provided	Not provided
Arranz et al (2016)30	Contact plus fever	48·4% (30·8–66·0)	77·3% (64·9–89·7)	60·0% (40·8–79·2)	68·0% (55·1–80·9)
Roddy et al (2010)22	Fever plus haemorrhage	44·0% (28·0–60·0)	72·0% (59·0–83·0)	Not provided	Not provided
Ingelbeen et al (2017)27	Three major signs**	27·7%*	79·1%*	41·5%*	67·2%*
Fitzgerald et al (2017)26	Contact, fever, and conjunctivitis OR contact, fever, anorexia, and two of abdominal pain, diarrhoea, or male sex (older than 2 years)	23·0%*	97·0%*	Not provided	Not provided
Kuehne et al (2015)23	History of contact, gastrointestinal symptoms†† and illness duration of >3 days	20·0%*	94·4%*	Not provided	Not provided
Hsu et al (2018)24	Unexplained death	14·2%*	92·8%*	72·0%*	45·2%*

95% CI not provided in the original paper.

For example, being a health worker, have attended a funeral, and having contact with a relative suspect of having Ebola virus.

Fever plus three other symptoms or fever and haemorrhage.

Symptoms or criteria not specifed in original paper.

Three or more symptoms among the following: intense fatigue, confusion, conjunctivitis, hiccups, diarrhoea, or vomiting.

Acute fever and presenting three or more of the following: headache, anorexia or lack of appetite, lethargy, muscle or joint pain, breathing difficulties, vomiting, diarrhoea, stomach ache, difficulty swallowing, and hiccups; or any person with unexplained bleeding.

As proposed by Lado and colleagues.

Diarrhoea, vomiting, and anorexia or loss of appetite.

For children, the highest specificity (97·0% [95% CI not provided in original paper]) was with a case definition of contact, fever, and conjunctivitis, or contact, fever, anorexia, and two of abdominal pain, diarrhoea, or male sex (older than 2 years; table 2).

Seven articles developed a risk score,22, 23, 25, 29, 31, 32, 33 and among those five25, 29, 31, 32, 33 did an internal validation (using bootstrap or test and training methods) and one assessed a risk score according to outbreak prevalence in a paediatric population. An eighth study externally validated the score developed by Oza and colleagues without developing an alternative score. Of the 44 potential predictors of Ebola virus infection included across the seven studies that develped risk scores, 20 were found to be positive or negative predictors (figure 3 ). The score system ranged from very low to very high risk, with intermediate categories varying across studies (appendix p 7).

Figure 3

Overview of risk score by symptoms and epidemiological characteristics

Predictive scores (numeric or + symbol) are shown in shaded cells (blue indicates positive scores and light pink indicates negative scores). Y indicates that the characteristic was assessed, but not used. AUC=area under the receiver operating characteristic curve. NA=not assessed. ORL=otorhinolaryngology. *Diarrhoea, vomiting, or anorexia or loss of appetite. †95% CI is taken from Ingelbeen et al (2018) because, although Oza and colleages do not report 95% CIs in their manuscript, Ingelbeen and colleagues have externally validated Oza and colleagues' score and they do report the 95% CI. ‡95% CI, AUC, or both AUC and 95% CI not given in original paper.

One study created a malaria sensitive score aiming to discriminate between Ebola virus infection and malaria infection, which indicated a predictor power of 89·6% (95% CI 86–93) to discriminate Ebola virus positive versus negative, reaching a discrimination power of 98·5% (95% CI not provided in original paper) during the malaria season. The same study obtained similar results (AUC 76·8% [95% CI not provided in original paper] vs 75·0% [70·0–80·0]), when externally validating the scores developed by Levine and colleagues.29, 32

The study validating Oza and colleagues' algorithm found poorer performance in their cohort (AUC 58% [95% CI 56–61] vs 83·0% [79-86]).28, 33

The highest performing score was developed by Hartley and colleagues, a key difference being referral time (figure 3). For the adult population (six studies22, 23, 29, 31, 32, 33), a positive risk score for infection was associated in more than one study with each of the following five characteristics: epidemiological link (eg, history of contact), diarrhoea, conjunctivitis, unexplained bleeding, difficulty swallowing (also called dysphagia; figure 3). Fever was assessed at different thresholds (>38·0°C or ≥38·5°C), and inclusion of fever in the final predictive score was only reported by two studies31, 32 (figure 3). Discordant values were assigned across studies (either positive or negative) for anorexia or loss of appetite, muscle pain (also called myalgia), and abdominal pain.

For the paediatric population (one study), positive predictors were age (2 years or older), sex (male), epidemiological link, diarrhoea, conjunctivitis, fever (>38·0°C), anorexia or loss of appetite, and abdominal pain. Negative predictors were difficulty swallowing, rash, headache, and difficulty breathing (also called dyspnoea; figure 3). The same study compared two different time periods over the Ebola virus disease 2014–16 outbreak in Sierra Leone (high prevalence in October, 2014 [77% of suspected cases testing positive], and low prevalence in March, 2015 [4% of suspect cases testing positive]): a low cutoff for the risk score (with high sensitivity) performed better at periods of high prevalence transmission, and a high cutoff with high specificity performed better during low prevalence. Similarly, the positive predictive value decreased from 93% to 31%, and the negative predictive value increased from 23% to 90% when comparing high (early) to low (late) transmission periods in the Ebola virus disease outbreak in another study in Liberia in an all ages population.

Eight studies measured sensitivity and specificity of individual symptoms at admission, assessing a total of 35 symptoms.5, 22, 23, 24, 27, 29, 30, 31 The pooled sensitivity per symptom ranged from 79·0% (95% CI 74·4–83·0) for intense fatigue (seven studies) to 1·0% (0·0–7·0) for pain behind the eyes (three studies). By contrast, the pooled specificity ranged from 98·0% (95% CI 91·0–100·0) for pain behind the eyes to 32·3% (25·8–39·4) for intense fatigue (appendix p 9).

Haemorrhagic symptoms and signs were the most specific indicator of infection. Other symptoms and signs with high specificity included confusion, coma, hiccups, rash, and sore throat with specificity ranging from 92·0% (95% CI 91·0–94·0) for hiccups to 97·8% (95·2–99·0) for rash (appendix p 9). Performance of fever was assessed by seven studies, but each one used a different definition of fever.5, 22, 23, 27, 29, 30, 31 The optimal performance (definition that achieved best balance between maximising sensitivity vs maximising specificity) for fever was a threshold at ≥38·5°C (sensitivity 80·2% [95% CI 69·2–88·2]; specificity 82·6% [71·2–90·3]; table 3 ). In the random-effects analysis, a threshold at greater than 38·0°C (three studies22, 27, 29) gave a pooled sensitivity of 80·0% (95% CI 69·0–90·0) and specificity of 25·0% (17·0–33·0; table 3).

Table 3

Sensitivity and specificity of fever, epidemiological link, or contact history, ordered by optimal performance

	Variable	Sensitivity (95% CI)	Specificity (95% CI)
Fever cutoff
Loubet et al (2016)31	≥38·5°C	80·2% (69·2–88·2)	82·6% (71·2–90·3)
Loubet et al (2016)31	≥38·0°C	88·2% (78·2–94·1)	72·5% (60·2–82·2)
Loubet et al (2016)31	≥37·5°C	93·4% (84·7–97·5)	50·7% (38·5–62·9)
Kuehne et al (2015)23	History of fever	85·3%*	26·4%*
Lado et al (2015)5	≥37·5°C or referred	85·9% (82·4–89·0)	16·4% (12·0–21·6)
Arranz et al (2016)30	≥38·0°C or referred	61·3% (44·1–78·4)	29·5% (16·1–43·0)
Roddy et al (2010)22	>38·0°C	85·0% (71·0–94·0)	20·0% (11·0–32·0)
Levine et al (2015)29	>38·0°C	85·0% (79·0–91·0)	21·0% (16·0–27·0)
Ingelbeen et al (2017)27	>38·0°C	71·5%*	30·5%*
Pooled analysis†	>38·0°C	80·0% (69·0–90·0)	25·0% (17·0–33·0)
Epidemiological link
Hsu et al (2018)24	Contact with infected persons or body fluid, handling of bushmeat, attending the funeral of a patient with Ebola virus disease	74·7%*	67·1%*
Roddy et al (2010)22	Epidemiological link‡	67·0% (50·0–81·0)	86·0% (74·0–94·0)
Arranz et al (2016)30	History of contact with a person with confirmed Ebola virus disease	100·0%	59·0% (43·5–74·4)
Levine et al (2015)29	Sick contact§	65·0% (58·0–73·0)	61·0% (54·0–67·0)
Loubet et al (2016)31	Health worker or having had contact with a person with suspected Ebola virus disease or having attended funerals	81·5% (44·0–60·7)	29·0% (19·0–41·3)
Kuehne et al (2015)23	Contact to case	47·3%*	71·2%*
Lado et al (2015)5	Travel to an Ebola virus disease hotspot area, health-care work, funeral attendance, or contact with an ill family member or friend¶	21·6% (17·9–25·6)	84·6% (79·6–88·8)

Optimal performance is the definition that achieved best balance between maximising sensitivity versus maximising specificity.

95% CI not provided in original paper.

The pooled analysis was used for the studies that had the same cut-off for fever (>38°C).22, 27, 29

Epidemiological link was defined as direct contact with an individual potentially infected with Marburg haemorrhagic fever or his or her body fluids or direct contact during funeral practices.

Direct or indirect contact with a patient with suspected or confirmed Ebola virus disease in the previous 21 days, including living in the same household or providing direct care for the patient.

A contact is any person who comes into contact with a case or suspected case by sleeping in the same household within the past month; direct physical contact with the case (dead or alive); touching his or her linens or body fluid; or attendance at a funeral of a person with confirmed or suspected Ebola virus disease.

Seven studies assessed sensitivity and specificity of an epidemiological link.5, 22, 23, 24, 29, 30, 31 Across these studies, the sensitivity of an epidemiological link ranged from 21·6% (95% 17·9–25·6) to 100·0% and specificity ranged from 29·0% (95% CI 19·0–41·3) to 86·0% (74·0–94·0). The most sensitive definition was history of contact with a person with confirmed Ebola virus infection (100·0%; table 3). The most specific definition was direct contact with an individual potentially infected with Marburg virus or his or her body fluids, or direct contact during funeral practices.

Discussion

Our results indicate that, for all ages combined, the WHO case definitions have a sensitivity of 81·5% and a specificity of 35·7%. The sensitivity is not high enough to achieve acceptable false negative rates, particularly in low-prevalence settings, the primary requirement for community-based screening. The low specificity results in high numbers of false positives and thus potentially unnecessary admissions to Ebola treatment centres, with associated risk of nosocomial transmission and costs of managing suspected cases. As a consequence, a large number of people who do not have Ebola virus disease will experience unnecessary invasive procedures, risk of being infected with Ebola virus, isolation from family, fear of being stigmatised, and delay to appropriate care, and community mistrust in response activities will increase.

In our meta-analysis, fever had low specificity (25·0%), except for when defined as a threshold at 38·5°C or more (82·6%), and the WHO case subdefinition had 100% sensitivity only when fever was not a mandatory criterion. In the risk score systematic review, the association of fever with Ebola virus infection was not consistent across studies, with only two studies including it in the final predictive score. Presence of fever is likely to be related to the stage of infection at admission, with previous studies reporting absence of fever in a large proportion of suspected cases at admission. This finding is consistent with a recent Ebola seminar reporting that fever was absent in at least 10% of the cases in the west Africa outbreak.

Therefore, exclusion of fever from the case definition at the community level is likely to increase the sensitivity of the case definition. Intense fatigue was the most sensitive symptom (79·0%) that could be used at the community level to facilitate early referral of suspected cases and prevent community transmission.

The meta-analysis did not identify any individual symptom or sign having an optimal trade-off between sensitivity and specificity. Conjunctivitis, unexplained bleeding, difficulty swallowing, and diarrhoea were individual symptoms and signs with the best discriminatory accuracy in the studies that explored risk score for the all-age population and with the exception of diarrhoea all had high specificity (>80%) in the studies that explored their performance. However, these symptoms and signs could also be a proxy for late-stage disease when the virus infects endothelial cells, compromising vascular integrity, with massive tissue injury resulting in disseminated intravascular coagulopathy with risk of thrombosis, bleeding, and damage to the adrenal glands and gastrointestinal system.36, 37, 38 These symptoms and signs could enable health practitioners to prioritise patients for admission to an Ebola treatment centre when resources are scarce but are less useful at the community level because they appear at a late stage of the disease when transmission risk is the highest.

None of the studies assessed miscarriage, despite it being included in the December, 2014, WHO case definition. History of miscarriage and other associated pregnancy complications (eg, stillbirth) could help to identify cases that can be a major source of nosocomial transmission in general health facilities.

Although only one study focused on a paediatric population, this study used data from 11 Ebola holding units and included a large population of children (1006), providing useful guidance for this age group. The WHO paediatric definition had very high sensitivity (98·0%) but very poor specificity (5·0%). When the same authors assessed a WHO subdefinition (including contact, fever, and conjunctivitis, or contact, fever, anorexia, and two of abdominal pain, diarrhoea, or male sex [older than 2 years]), the sensitivity dropped markedly to 23·0% but the specificity improved to 97·0%. The optimal fever temperature cutoff for the paediatric population was not explored. However, in another study of a paediatric population of patients with confirmed Ebola virus disease admitted to one Ebola treatment centre in Sierra Leone, 25% of children aged 5 years and younger were afebrile. This difference might be due to several factors: how fever was assessed (either reported in their history or measured at admission), age groups included (younger than 13 years vs younger than 5 years), period of data collection (August–March, 2015, vs June–Dec, 2014) when seasonality of other febrile illnesses could have influenced fever prevalence, background Ebola virus transmission rates, and viraemia at admission and time since onset of symptoms.

The paediatric analysis did not explore sensitivity and specificity of individual symptoms and signs at admission for children. Alongside the fact that they might have different clinical presentations compared with adults, children are more likely to experience adverse outcomes from Ebola virus disease and are less able to report symptoms and history of contact.

Similarly, pregnant women with non-Ebola virus disease-related complications usually present with symptoms (such as bleeding and abdominal pain) that mimic Ebola virus infection. As suggested elsewhere, the paediatric and pregnant women populations might require adaptation of case definitions that take into account their specific characteristics.41, 42, 43 None of the selected manuscripts explored the performance of WHO Ebola case definitions among pregnant women. Therefore, further evidence specifically applicable to children and pregnant women is required to develop appropriate tools for screening for Ebola virus disease in these populations.

Reported history of contact was a strong predictor for paediatric and adult populations, often performing better than many of the clinical symptoms included in accepted case definitions, as also reported by other studies. However, it is likely that this is an underestimate of the potential performance of actual contact history in screening for Ebola virus disease.

Levels of disclosure of self-reported clinical information and contact history depend on community engagement with intervention strategies, including trust in the health-care provider. Therefore, to improve WHO case definition performance, effective and trusted collaboration with communities is essential to ensure reliable understanding and reporting of such crucial epidemiological information. Equally, it is the responsibility of response agencies to understand the underlying pattern of Ebola virus transmission, local traditions, coping mechanisms, and family dynamics in order to identify people at risk of infection. Genetic sequencing has also been put forward as a tool for identifying chains of transmission when contact history is unknown. One of the limitations in interpreting the results of this meta-analysis is that all the evidence reviewed, apart from the national surveillance study, came from patients triaged at health facilities or Ebola isolation centres. Thus, this meta-analysis might represent only cases with severe symptoms, limiting generalisability to the performance of these screening criteria at the community level and in early stages of disease. Second, there was significant heterogeneity between selected studies, and considerable variation in the quality of data on clinical symptoms and recollection of patients' history, with different variables and thresholds used in each study, and limited data on co-infection. For example, fever is a key symptom in the WHO case definitions, but different temperatures were used to define fever, which could explain the between-study heterogeneity. Inconsistency on thresholds for fever and the decision to include fever or not have been reported in the Democratic Republic of the Congo and in four neighbouring countries.

For the two studies with overlapping patient populations, performance of WHO case definitions was assessed only using national surveillance data, with Ebola treatment centre data for these patients being assessed for only individual symptoms or WHO subdefinitions. These two studies were therefore not included together in pooled estimations, so the cohort overlap would not have affected results. Individual studies mentioned small sample size and poor quality of data as part of their limitations.

A range of contextual factors related to study setting will affect the performance of Ebola virus disease case definitions, including seasonally occurring diseases such as malaria and Lassa fever, which have a similar clinical presentation to Ebola virus disease. Such factors will affect the generalisability of our findings to other settings. In addition, only two of the recommended risk scores were externally validated,28, 32 limiting the generalisability of those scores because performance appears to vary across outbreak periods and populations.

Finally, there is potential for publication language bias because we considered only studies in English. However, for Guinea, a French-speaking country, we included data from national surveillance and two major Ebola treatment centres; therefore, we consider that bias due to language restrictions was minimised in our results. We included peer-reviewed abstract and poster data to capture data on paediatric populations and additional evidence for all age cohorts, and we sought unpublished evidence from French-speaking countries.

This systematic review is relevant to inform public health practitioners in the current Ebola virus disease outbreak in the Democratic Republic of the Congo, in which only 8% of suspected cases isolated are confirmed, possibly because of inconsistent use of WHO case definition at community and health facility levels.

In conclusion, this first systematic review and meta-analysis of the strengths and limitations of the WHO Ebola virus disease case definitions highlights the need for further studies to assess consistent thresholds for fever, to explore viraemia and symptoms and signs at admission, and to externally validate risk scores for Ebola virus infection. The sensitivity and specificity of WHO Ebola case definitions could be improved by excluding fever and instead including both intense fatigue and history of contact. However, reliable disclosure of reported symptoms and history of contact requires effective collaboration with, and the trust of, affected communities. To achieve this trust and collaboration, responding organisations must recognise the paramount role of communities in controlling transmission and ending outbreaks. We also identified important gaps related to the paediatric and pregnant population, which must be addressed through future research.