Screening instruments for cognitive impairment in older patients in the Emergency Department: a systematic review and meta-analysis.

BACKGROUND: cognitive impairment is highly prevalent among older patients attending the Emergency Department (ED) and is associated with adverse outcomes.
METHODS: we conducted a systematic review and meta-analysis to evaluate the diagnostic accuracy of cognitive screening instruments to rule out cognitive impairment in older patients in the ED. A comprehensive literature search was performed in MEDLINE, EMBASE, CINAHL and CENTRAL. A risk of bias assessment using QUADAS-2 was performed.
RESULTS: 23 articles, examining 18 different index tests were included. Only seven index tests could be included in the meta-analysis. For ruling out cognitive impairment irrespective of aetiology, Ottawa 3 Day Year (O3DY) (pooled sensitivity 0.90; (95% CI) 0.71-0.97) had the highest sensitivity. Fourteen articles focused on screening for cognitive impairment specifically caused by delirium. For ruling out delirium, the 4 A's Test (4AT) showed highest sensitivity (pooled sensitivity 0.87, 95% confidence interval (95% CI) 0.74-0.94).
CONCLUSIONS: high clinical and methodological heterogeneity was found between included studies. Therefore, it is a challenge to recommend one diagnostic test for use as a screening instrument for cognitive impairment in the ED. The 4AT and O3DY seem most promising for ruling out cognitive impairment in older patients attending the ED.The review protocol was registered in PROSPERO (CRD42018082509).

Cognitive impairment in older patients attending the Emergency Department is associated with adverse outcomes.

Different screening tools for cognitive impairment have been developed and validated in the past decades.

Screening for cognitive impairment enables to apply interventions and care adjustment, to prevent adverse outcomes.

Introduction

Cognitive impairment is present in 26% of the older patients attending the Emergency Department (ED) and can be caused by delirium, dementia or both [1]. Delirium is an acute and fluctuating change in mental status, characterized by impaired cognition, an altered level of consciousness and inattention and is caused by an underlying medical illness [2]. Dementia is a chronic condition of impaired cognition, but patients with dementia can also develop delirium: delirium superimposed on dementia (DSD) [3]. Delirium and dementia are highly related syndromes: patients with dementia are at risk for delirium, but frequently dementia is not yet known, on the other hand patients with delirium can develop dementia. Recognising cognitive impairment irrespective of cause is of importance since the presence of cognitive impairment increases the risk of older patients to become hospitalised and once admitted to the hospital, they are at risk for longer hospital stay, progressive functional and cognitive decline, increased mortality and of being institutionalised, compared to patients without cognitive impairment [4-8]. Recognising delirium in the ED is of additional importance, because it is always caused by an underlying medical cause and this should be diagnosed and treated promptly. In this perspective, the altered mental status can be considered as a vital parameter [9]. Often, medical professionals are not aware of co-existing cognitive impairment and the provided (hospital) care is not suited to the demands of cognitive impaired patients [10]. Since most older patients enter the hospital through the ED, it would be favourable to recognise patients with cognitive impairment without delay in order to start interventions immediately to prevent adverse effects [7]. Unfortunately, cognitive impairment in the ED is frequently missed by health care professionals [11]. For instance, emergency physicians recognised cognitive impairment in only 38% of the older patients with either delirium or cognitive impairment without delirium [12]. An easy to apply screening tool for cognitive impairment irrespective of cause in the ED, preferably a single assessment, seems most useful [7]. Previous systematic reviews on screening instruments addressing the detection of cognitive impairment, aimed to identify delirium screening tools, focused on delirium or dementia instead of cognitive impairment in general, and others discussed a different patient population such as hospitalised patients instead of ED patients [13-15]. The aim of this systematic review and meta-analysis is to evaluate the diagnostic test accuracy of screening tools to rule out cognitive impairment in older patients in the ED. Identifying by ruling out results in a higher proportion of false positives, but simultaneously enhances the probability of patients with cognitive impairment receiving appropriate care to prevent adverse outcomes from happening.

Methods

This study was conducted according to the methods of the Cochrane Collaboration and reported according to PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses [16,17]. The review protocol was registered in PROSPERO (CRD42018082509).

Search strategy

A systematic literature review was conducted by searching the electronic databases of MEDLINE, EMBASE, CINAHL, and the Cochrane Central Register of Controlled Trials (CENTRAL), from inception to 3 March 2020. The search string was developed in collaboration with a library information specialist (S.W., see acknowledgements) and contained search terms related to cognitive impairment, delirium, older patient population and the ED setting (See Appendix 1 for complete search strategy). The references and citations of included studies and relevant reviews on this topic were screened for potential eligible studies.

Study selection

Studies were considered eligible for review when they met the following criteria:

Cohort study or case-control study

Study population consisted of patients with a mean or median age 65 years or older, visiting an ED. Studies conducted in a different environment than the ED were excluded.

The target condition was cognitive impairment irrespective of the aetiology. Ideally, the diagnosis was based on the Diagnostic and Statistical Manual of Mental Disorders (DSM) criteria (version III, IV, IV-R, V) made by a specialist in geriatric care [2,18-20]. The Confusion Assessment Method (CAM) and the Mini-Mental State Examination (MMSE) were accepted as a substitute gold standard because of their widely use in clinical practice [21,22].

The index test was an instrument to assess cognition in the ED.

The study provided sufficient data to construct a two-by-two table.

Three independent reviewers screened identified papers on title and abstract for eligibility (MP and HB screened the MEDLINE search results, AC and MP screened the results found in EMBASE, CINAHL and CENTRAL. The full text selection was performed by two reviewers independently (A.C., M.P.).

Data extraction and quality assessment

Two reviewers (A.C., M.P.) independently assessed the risk of bias and extracted data from the included studies with regard to study characteristics (design, in- and exclusion criteria); study population (age, proportion female, country); characteristics of the index test and reference standard (e.g. cut-off point) and outcome data (sensitivity, specificity, two-by-two table). In case the results of a study were reported in different articles, we treated the data as being independent when the test results of the participants, and not the participants themselves were the unit of analysis. Authors were contacted to obtain additional information in case of unreported data. The risk of bias was assessed using the Quality Assessment of Diagnostic Accuracy studies 2 (QUADAS-2) tool, which evaluates the risk of bias and applicability of diagnostic accuracy studies [23]. An additional item adapted from the QUADAS-1 criteria; “Was the reference standard independent of the index test (i.e., the index test did not form part of the reference standard)?” was added to the assessment [24]. Disagreements regarding the quality assessment were discussed in consensus meetings. The percentage of agreement and kappa statistic was calculated for the overall risk of bias.

Statistical analysis

Based on the chosen target condition of the different studies, cognitive impairment had to be categorised into caused by delirium and cognitive impairment irrespective of aetiology (i.e. delirium, dementia and DSD). The two-by-two table of each diagnostic test reported was used to calculate sensitivity, specificity, positive and negative predictive values, positive and negative likelihood ratios. In case an index test was conducted twice in a study by different assessors, we selected the diagnostic test which was conducted by a health care professional (e.g. physician or nurse). The results were graphically displayed by use of coupled forest plots, representing sensitivity and specificity.

Meta-analysis was conducted if data of two or more studies reporting on the same screening instrument (index test) for the same target condition were included. Data, sensitivities and specificities were pooled if the same screening instrument was used and all inclusion criteria were met. In case an index test was performed twice or more in the same study cohort, the test results (the unit of analysis) were included in the meta-analysis only once. For this meta-analysis, a bivariate random effects model was used [25]. In order to investigate heterogeneity, we used Cochran’s Q. In case the P-value of Cochran’s Q was below 0.05, heterogeneity was present. Sources of heterogeneity were explored by conducting subgroup analyses including used screening instrument (index test), the used reference standard (DSM criteria versus MMSE versus CAM), and risk of bias (per item). All statistical analyses were conducted in R, package “mada” [26].

Results

Included studies

Our literature search identified 7,112 articles. After screening the titles and abstracts of the identified articles, 95 articles were eligible for full text review (Figure 1), 74 of these articles were excluded. Main reasons for exclusion were different setting than the ED (n = 28) and type of publication (e.g. conference abstracts) (n = 27). Two studies regarding a serious game for cognitive assessment in the ED as an index test were excluded due to the chosen study methods which were not designed as diagnostic test accuracy studies. Data of these studies are provided in Appendix 2 [30-34]. Screening references of included articles and relevant reviews and citations of the included articles resulted in two additional eligible articles. In total, 23 articles were included in this systematic review. Five of these articles were part of one study design which investigated several screening instruments in the same study population [30-34]. Two other articles described two different screening instruments in the same study cohort [35,36]. One article investigated a screening instrument by use of a subanalysis on data of the same study cohort [37,38]. Nine articles evaluated >1 diagnostic test, and 37 data sets were available for the systematic review and meta-analysis.

Figure 1

PRISMA flow diagram.

Study characteristics

Twenty-three articles were included and described a total of 23 studies in 17 different study cohorts. The main characteristics of the included studies are presented in Table 1. The index tests were conducted by physicians in 12 studies. (52%) [30-34,39-44]. In other studies, a research assistant or nurse assessed the index test [45-52]. The reference standard was assessed by a psychiatrist or geriatrician in 13 studies (57%) [30-34,37,39,40,44-46,52]. In other studies, a research assistant conducted the reference standard [35,36,41-43,47-50,53]. In one study, an ED geriatric nurse conducted the reference standard [51].

Table 1

Characteristics of included studies

	Country	Included in analysis (N)	Period of recruitment	Age Mean (sd)/median (IQR)	Female (%)	Indextest, (treshold)	Reference standard, (treshold)
Cognitive impairment caused by delirium
Baten (2018) [44]	Germany	288	2016	78 (74–82)	55	bCAM (test+)	DSM-V
Bédard (2019) [35]	Canada	313	2016	76.8 (7.5)	53	O3DY (>0)	CAM
Fabbri (2001) [39]	Brazil	100	1996–1997	73.8 (0.8)	48	CAM (test+)	DSM-IV
Gagné (2018) [36]	Canada	313	2016	76.8 (7.5)	52	4AT (≥4)	CAM
Grossmann (2017) [38]	Switzerland	286	2015	79.9 (72.4–86.7)	59	mRASS (≠ 0)	DSM-IV-TR
Han (2013) [30]	USA	406	2009–2012	73.5 (69–80)	50	DTS (test+) bCAM (test+)	DSM-IV-TR
Han (2014) [31]	USA	406	2009–2012	73.5 (69–80)	50	CAM-ICU (test+)	DSM-IV-TR
Han (2015) [32]	USA	406	2009–2012	73.5 (69–80)	50	RASS (≠ 0)	DSM-IV-TR
Han (2018) [33]	USA	406	2009–2012	73.5 (69–80)	50	SQ patient SQ surrogate	DSM-IV-TR
Hasemann (2018) [45]	Switzerland	286	2015	80.0 (72.2–86.8)	59	mCAM-ED (test+)	DSM-IV-TR
Hasemann (2019) [37]	Switzerland	286	2015	80.0 (72.2–86,8)	59	MOYB (> 0 error)	DSM-IV-TR
Marra (2018) [34]	USA	235	2010–2012	74 (69–79)	46	MOYB (> 0 error)	DSM-IV-TR
Meeberg (2016) [40]	Netherlands	53	2012	78.5 (6.9)	49	CAM-ICU (test+)	DSM-IV-TR
Shenkin (2019) [52]	UK	395	2015–2016	81 (77–86)	56	4AT (≥4) CAM (test+)	DSM-IV
Cognitive impairment
Barbic (2018) [46]	Canada	117	2016	81.9 (5.7)	45	O3DY (> 0 error) SBT (>4)	MMSE (≤ 23)
Carpenter (2011) [47]	USA	163	2009–2010	78 (8)	61	O3DY (> 0 error) SBT (≥ 4) BAS (<26) cAD8 (≥2)	MMSE (≤ 23)
Carpenter (2011) [41]	USA	319	2008–2009	76 (NA)	58	SIS (≥2) AD8 (≥ 2)	MMSE (≤ 23)
Dyer (2016) [48]	Ireland	196	2014	78.5 (5.9)	46	AMT4 (> 0 error)	MMSE (≤ 26), CAM-ICU +
O’Sullivan (2017) [51]	Ireland	419	2015	77 (NA)	51	6-CIT (≥10) 4AT (1–3)	DSM-V
						6-CIT (≥10)4AT (≥4)	DSM-V
Schofield (2009) [49]	UK	520	2007	77 (NA)	NA	AMT10 AMT4	MMSE (≤ 23)
Wilber (2005) [42]	USA	75	2003	75.4 (6.6)	54	SIS (≤ 4) Mini-Cog (test+)	MMSE (≤ 23)
Wilber (2008) [43]	USA	352	2006–2007	77 (8)	63	SIS (≤ 4)	MMSE (≤ 23)
Wilding (2016) [50]	Canada	238	2010	81.9 (NA)	60	O3DY (≤ 3) AFT (<15)	MMSE (≤ 24)

NA: not available; bCAM: brief Confusion Assessment Method; DSM: Diagnostic and Statistical Manual of Mental Disorders; O3DY: Ottawa 3DY; CAM: Confusion Assessment Method; 4AT: 4 A’s Test; mRASS: modified Richmond Sedation Scale, DTS: Delirium Triage Screen; CAM-ICU: CAM-Intensive Care Unit; RASS: Richmond Sedation Scale; SQ: single question for delirium screen; mCAM-ED: modified CAM-Emergency Department; MOYB: Months Of the Year Backwards; SBT: Short Blessed Test; MMSE: Mini-Mental State Examination; BAS: Brief Alzheimer’s Screen; cAD8: caregiver Ascertain Dementia 8-item Questionnaire; SIS: Six-item Screener; AMT: Abbreviated Mental Test; TR: Text Revision;; AFT: Animal Fluency Test; 6-CIT: 6-Item Cognitive Impairment Test.

Delirium

Fourteen studies focused on cognitive impairment specifically caused by delirium, and one study investigated a screening instrument and categorised the results into chronic cognitive impairment and delirium [52]. Mean or median age of the study population included in the studies ranged from 73.5- to 81.0-years old. Forty-six to 59% of the participants were female.

Almost all studies used the DSM criteria for delirium as reference standard, two studies used the CAM. Various index tests were investigated (Table 1). Two articles reported on the brief CAM (bCAM) and a modified brief CAM (modified bCAM), but were performed in the same cohort [30,54]. Therefore, the data were used only once in the meta-analysis.

Cognitive impairment

Nine studies reported on a screening instrument for cognitive impairment irrespective the underlying aetiology (i.e. dementia, delirium, DSD, mild cognitive impairment), one of these studies investigated a screening instrument and categorised into chronic cognitive impairment and delirium. Mean or median age of the study population included in the studies ranged from 75.4- to 81.9-years old. The percentage of females participating in the studies ranged from 45% to 63%. Eight studies used the MMSE as reference standard with cut-off values varying from ≤26 to <24 points and one study used the DSM-criteria for dementia. Ten different index tests were investigated (Table 1). The 6-CIT test and SBT appeared to be identical tests [47,48,52].

Risk of bias assessment of included studies

Results of the risk of bias assessment of the included studies according to the QUADAS-2 criteria are presented in Appendices 3 and 4. In total, 16 of the 23 articles had a high or unclear assessment of risk of bias in at least one domain. Frequent shortcomings in the studies were the absence of a consecutive or random sample of patients (20 articles, 15 studies), the person conducting the index test was not blinded for the result of the reference standard (6 articles; 5 studies) or the assessor of the reference standard was not blinded for the result of the index test (8 articles; 7 studies). The inter-rater agreement on the risk of bias assessment was excellent with an overall agreement 87% (222/253 items); kappa statistic 0.69 (95% confidence interval [CI] 0.58–0.79).

Diagnostic test accuracy

Main results

The included tests for identifying cognitive impairment caused by delirium or any type of cognitive impairment in the ED show a wide variance in sensitivity (resp. 0.62–1.00 and 0.53–0.95) and specificity (resp. 0.55–0.96 and 0.39–0.97). In Figure 2, the sensitivity and specificity per test and per study with their coupled forest plots for detection of cognitive impairment are presented.

Figure 2

Coupled forest plots of diagnostic test accuracy. mRASS: modified Richmond Sedation Scale; DSM: Diagnostic and Statistical Manual of Mental Disorders; 4AT: 4 A’s Test; 6-CIT: 6-Item Cognitive Impairment Test; SBT: Short Blessed Test; CAM: Confusion Assessment Method; CAM-ICU: CAM-Intensive Care Unit; DTS: Delirium Triage Screen; MOYB: Months Of the Year Backwards; SQ: single question for delirium screen to patient/surrogate; bCAM: brief CAM; mCAM-ED: modified CAM-Emergency Department; MMSE: Mini-Mental State Examination; AFT: Animal Fluency Test; AMT: Abbreviated Mental Test; BAS: Brief Alzheimer’s Screen; O3DY: Ottawa 3 Day Year; SIS: Six-item Screener; cAD8: caregiver Ascertain Dementia 8-item Questionnaire.

Cognitive impairment caused by delirium

The summary estimate of the sensitivity and specificity of the 4 A’s Test (4AT) was 86.9% (95% CI 73.5–94.1) and 86.9% (95% CI 59.6–96.5), respectively. For (m)RASS, the summary estimate of the sensitivity was 76.7% (95% CI 58.4–88.5) and for the specificity was 89.7% (95% CI 78.6–95.4). Pooled sensitivity and specificity per index test are shown in Appendix 5. As a result of the wide variance of index tests of the include studies screening for cognitive, conducting a meta-analysis for the other index tests was not possible. High heterogeneity was found between the results of the studies evaluating the accuracy of assessing delirium (Cochran’s Q of 17,212; P < 0.001).

Cognitive impairment

The summary estimate of the sensitivity and specificity of the Ottawa 3 Day Year (O3DY) was 89.8% (95% CI 70.6–97.0) and 60.9% (95% CI 47.0–73.2), respectively. For respectively, the 6-CIT/SBT and SIS the summary estimate of the sensitivity was 89.1% (95% CI 78.2–94.9) and 71.5% (95% CI 58.5–81.8), and for the specificity was 67.2% (95% CI 55.8–76.9) and 79.2% (95% CI 75.1–82.8). Pooled sensitivity and specificity per index test are shown in Appendix 5. As a result of the wide variance of index tests of the included studies for cognitive impairment, conducting a meta-analysis for the other index tests was not possible. High heterogeneity was found between the results of the studies evaluating the accuracy of assessing cognitive impairment (Cochran’s Q of 5,235, P < 0.001).

Exploring heterogeneity

Heterogeneity of the included studies was explored with subgroup analyses. The choice of reference standard (DSM versus MMSE versus CAM) did affect the sensitivity and specificity of the used test (P < 0.05). Studies using the MMSE or CAM as a reference standard showed a higher sensitivity and lower specificity compared to studies using the DSM as a reference standard. The presence of a high or unclear risk of bias, based on the QUADAS-2 criteria, did not alter the results. Furthermore, the presence of a dependency between the index test and reference standard (item adapted from the QUADAS-1 criteria) did not influence the results (P = 0.87). For the index tests consisting of more than two observations, we constructed a summary receiver operating characteristic (SROC; Figure 3).

Figure 3

SROCs and estimates per index test; 6-CIT: 6-Item Cognitive Impairment Test; SBT: Short Blessed Test; O3DY: Ottawa 3 Day Year; SIS: Six-item Screener, (m)RASS: (modified) Richmond Sedation Scale, 4AT.

Discussion

In this review, a total of 23 articles on the diagnostic test accuracy of screening instruments for cognitive impairment in older patients in the ED have been evaluated. The ideal screening instrument has high sensitivity to rule out cognitive impairment and should be easy to integrate into daily ED practice [54]. For ruling out cognitive impairment as a global construct, the O3DY had highest sensitivity. From the studies focusing on cognitive impairment caused by delirium, the 4AT seems most promising. In a busy ED setting where time and resources are perceived as limited, preferably only one test should be assessed. Although it was not possible to include in the meta-analysis because of less than two studies reporting on the same screening instrument for the same target condition, O’Sullivan et al. described good sensitivity of the 4AT in chronic cognitive impairment (84.1%) and Bédard et al. of the use of the O3DY in delirium (84.2%) [35,51]. The 4AT and O3DY both seem to be robust enough to detect cognitive impairment, whether or not caused by delirium and since the 4AT and O3DY do not require extensive additional training of the assessor, the 4AT and O3DY seem to be the most practical screening instruments in an ED.

Some issues regarding the topic of this systematic review need to be addressed. First, most of the cognitive screening tools are designed to detect either delirium or chronic cognitive impairment such as dementia. As approximately half of the older people in the ED with dementia also have delirium, as delirium is often superimposed on dementia, perceiving this difference is not always easy, although relevant [7]. To be able to apply interventions, detection of cognitive impairment, whether it is caused by delirium, dementia or DSD, is of importance. Nonetheless, recognising delirium in the acute setting is of additional importance to detect and treat the possible causative medical disorders. As a first step, an easy to apply screening tool for cognitive impairment is useful, preferably one single assessment. An assessment such as the 4AT provides basic cognitive testing, aimed at detecting moderate–severe cognitive impairment, alongside assessment for delirium [51]. After recognising the presence of a cognitive disorder with help of a screening tool, further cognitive evaluation is needed to distinguish the aetiology at a later moment, for example within 24 h after hospitalisation. Second, in the included studies, the DSM criteria as well as the MMSE and CAM were used as reference standard. The DSM criteria are created to classify (neuro)cognitive disorders where the MMSE and CAM are created as cognitive screening instruments themselves. As shown in the subgroup analysis, the choice of reference standard did affect the results of the meta-analysis. Studies using the MMSE or CAM as a reference standard reported a higher sensitivity and lower specificity of the investigated index test compared to studies using the DSM as a reference standard. This can be explained by the fact that the index test is compared to a more extended screening instrument (i.e. MMSE or CAM), which are often developed for optimal sensitivity. We expected the more comprehensive assessment and often more trained assessors of the DSM would improve diagnostic test accuracy, but this is reflected only in improved specificity. To be able to compare and generalise results more easily, future studies should use the same reference standard, preferably the DSM criteria.

Third, all index tests included in this review serve the same purpose of screening for cognitive impairment and consist of similar items, such as assessment of orientation, attention and memory. In addition, index tests are often based on pre-existent disease criteria, such as the DSM-criteria or are based on a shorter version of an extensive, already validated, instrument (e.g. CAM–ICU). Thus additionally to the overlap between the different index tests, there is an overlap between index tests and the reference standard. We expected overlap between index test and reference standard would result in higher diagnostic test accuracy, because of similarity in the items being tested. To overcome this incorporation bias, the item adapted from the QUADAS-1 criteria was added as one of the factors in the analysis. Although we assumed this item: “Was the reference standard independent of the index test?” would be of influence, the effect was not significant when expressed in a p-value. This can be explained by the unequal distribution of the assessment itself. Of the total of the 37 analysed datasets from the 23 different studies, this item was adjudicated negative in 35 of the cases, meaning in 35 of the cases the reference standard was not completely independent of the index test. Therefore, we cannot confirm our assumption [16]. Fourth, in the risk of bias assessment, one of the QUADAS-2 domains includes the question: “Did the study avoid inappropriate exclusions?” Most of the studies used exclusion criteria such as “severe intellectual disability” or “not able to communicate” [38,51]. Although these criteria are justly for excluding patients from participation, they could lead to exclusion of patients with severe or end-stage cognitive impairment which would lead to an underestimation of the test results.

Fifth, in a hectic and busy environment such as the ED time is of importance. In this review, we focused on diagnostic test accuracy of the different index tests, although of similar importance to clinical applicability is the administration time. For future studies, it would be of additional value to add data on administration time of the investigated index test.

Strengths and limitations

This review used a comprehensive search strategy with additional hand search of references and cross-references of included studies. Two independent reviewers screened potential studies for inclusion and extracted data, reducing the potential risk of bias. Regarding the quality assessment of the included studies, an excellent agreement was reached between the two reviewers.

One of the limitations of this review is the heterogeneity caused by the use of multiple index tests. Most of the index tests were being used in one study, resulting in a single observation per index test. Meta-analyses on all included studies were therefore not performed.

Furthermore, diagnostic tests based on a continuous variable used a cut-off value to classify the results as being either positive or negative. A cut-off value is sometimes chosen arbitrarily and does not reflect the range of cognitive impairment possible. It could occur that cognitive impairment is of clinical relevance although not meeting the pre-set cut-off value.

Conclusion

Due to clinical and methodological heterogeneity of the included studies, it is challenging to determine one diagnostic test for use as a screening instrument for cognitive impairment. In this systematic review, the 4AT or O3DY seems to be most promising for identification of cognitive impairment irrespective of aetiology in the ED. To answer the main question of this review, an accuracy study similar to the study of Carpenter et al. comparing multiple screening instruments to a single reference standard in a single population would be preferable [47]. Unfortunately, a study with the aforementioned design including all index tests, will not be feasible due to overlap of items in the tests, high burden for participating patients and too time-consuming for an ED. Heterogeneity between studies is therefore not preventable because researchers, patients and setting will differ and a review provides the highest possible grade of evidence.

Click here for additional data file.