Accuracy of clinical stroke scores for distinguishing stroke subtypes in resource poor settings: A systematic review of diagnostic test accuracy.

BACKGROUND: Stroke is the second leading cause of death globally. Computerized tomography is used to distinguish between ischemic and hemorrhagic subtypes, but it is expensive and unavailable in low and middle income countries. Clinical stroke scores are proposed to differentiate between stroke subtypes but their reliability is unknown.
MATERIALS AND METHODS: We searched online databases for studies written in English and identified articles using predefined criteria. We considered studies in which the Siriraj, Guy's Hospital, Besson and Greek stroke scores were compared to computerized tomography as the reference standard. We calculated the pooled sensitivity and specificity of the clinical stroke scores using a bivariate mixed effects binomial regression model.
RESULTS: In meta-analysis, sensitivity and specificity for the Siriraj stroke score, were 0.69 (95% CI 0.62-0.75) and 0.83 (95% CI 0.75-0.88) for ischemic stroke and 0.65 (95% CI 0.56-0.73) and 0.88 (95% CI 0.83-0.91) for hemorrhagic stroke. For the Guy's hospital stroke score overall sensitivity and specificity were 0.70 (95% CI 0.53-0.83) and 0.79 (95% CI 0.68-0.87) for ischemic stroke and 0.54 (95% CI 0.42-0.66) and 0.89 (95% CI 0.83-0.94) for hemorrhagic stroke.
CONCLUSIONS: Clinical stroke scores are not accurate enough for use in clinical or epidemiological settings. Computerized tomography is recommended for differentiating stroke subtypes. Larger studies using different patient populations are required for validation of clinical stroke scores.

Introduction

Stroke is the second leading cause of death globally and is associated with up to 5.54 million deaths every year, two thirds of which occur in resource poor countries (RPC).[1] It has two main subtypes, ischemic and hemorrhagic. For optimal management, a distinction must be made between the subtypes since the therapy is different.[2] Ischemic stroke warrants institution of thrombolytic and/or antiplatelet therapy while in hemorrhagic stroke, hemostatic therapy may be given.[3] Ideally, either thrombolytic or hemostatic therapy should be given soon after the onset of stroke in order to improve outcome.[4]

Non-contrast computed tomography (CT) scan is the gold standard for distinguishing stroke sub-types.[56] It is cheaper than magnetic resonance imaging (MRI), but is still expensive and inaccessible for most resource poor settings. To overcome these difficulties and to enhance clinical bedside diagnosis, clinical stroke scores have been developed. The most commonly used ones include the Guy's hospital score (GHSS),[7] the Besson score,[8] the Greek stroke score[9] and the Siriraj stroke score (SSS).[10] In developing these scores, clinical variables that could potentially distinguish ischemia from hemorrhage in patients with acute stroke were used.

While these scores are not more accurate than neuro-imaging, they are simple, cheap and practical. However, their true accuracy and value in the diagnosis of stroke in resource poor settings remains unknown. We report a systematic review examining the evidence on the accuracy of clinical stroke scores in distinguishing between stroke subtypes, particularly within low and middle income countries (LAMICs).

Materials and Methods

Search strategy

We searched the following databases for both published and unpublished studies in the English language over the period 1983-2013: PubMed, EMBASE, Cochrane central register of controlled trials (CENTRAL), Cumulative Index to Nursing and Allied Health Literature (CINAHL), Web of Science, Virtual health library, System for Information on Grey literature in Europe (SIGLE), Scientific Electronic Library Online (SciELO), MedNar and ProQuest. The following terms were used to generate a search: Stroke, Acute stroke, Brain ischemia, Cerebral hemorrhage, Cerebral infarction, Siriraj stroke score, Clinical stroke score, Guy's hospital stroke score, Allen score, Besson score, Greek stroke score.

Inclusion and exclusion criteria

We considered studies of diagnostic test accuracy from LAMICs that included patients admitted to hospital with a diagnosis of acute stroke according to the WHO criteria[2] and in which the index tests (Siriraj, Guy's Hospital, Besson and Greek stroke scores) and reference test (CT-Scan) were interpreted independently of one another on the same group of participants. Table 1 provides more details on each score. Due to a difference in the prevalence of hemorrhagic stroke and stroke risk factors,[1112] we did not consider studies from high income countries (HICs). Details on the calculation of test scores have been described previously.[78910] Studies that evaluated two or more of these scores simultaneously were also included. Only studies that reported on the sensitivity and specificity of stroke scores compared to CT scan diagnosis were included however, studies that did not report on sensitivity and specificity but had sufficient information to calculate these were also considered.

Table 1

Comparison of clinical stroke scores

Assessment of methodological quality

Papers selected for review were assessed independently for methodological validity prior to inclusion in the review using the Quality Assessment of Diagnostic Accuracy Studies (QUADAS) criteria. Disagreements were resolved through discussion. We considered a representative spectrum of patients to be female or male patients of all ages presenting with mild, moderate or severe stroke symptoms, with or without a previous history of stroke. In the assessment of disease progression bias, 15 days was considered an appropriate period between onset of stroke and administration of the reference standard. This period helps to avoid interpreting resolving hemorrhages as infarcts and accommodates the lack of the sensitivity of CT scan in early ischemic stroke.

Data extraction

A data extraction form was designed and used to collect details from included studies. For each study, the following data were obtained: Author information, year of publication, study site, setting, study design, number and characteristics of patients (age, sex, ethnicity), reference standard, index test (s), information on clinicians who administered scores and clinicians who interpreted the reference standard i.e. background specialty. Sensitivity and specificity, number of patients with equivocal scores, true positive (TP), false positive (FP), true negative (TN) and false negative (FN) data for each test were taken directly from source papers. If this was not possible, they were calculated from provided data. Extracted data was then entered in a separate form and transferred to a spreadsheet.

Data synthesis

We derived indices of diagnostic performance from the data presented in each study for each index test. We constructed 2 × 2 contingency tables of true positive, false positive, false negative and true negative cases with CT scan as the independent variable and index test as the dependent variable and calculated sensitivity and specificity with 95% confidence interval (95% CI) for each stroke score in each study. However, since exclusion of indeterminate test results may result in overestimation of accuracy, we included equivocal test results in our calculations where these were reported. Because ischemic and hemorrhagic stroke have different management strategies we tabulated results for ischemic stroke separately from those for hemorrhagic stroke for each index test and generated separate forest plots of test performance for ischemic and hemorrhagic strokes.

Since included studies had a similar spectrum of patients and the threshold value used for the index tests was the same in all the studies, we performed a meta-analysis of the index test results using a bivariate mixed effects binomial regression model. Summary estimates for sensitivity, specificity, positive and negative likelihood ratios (LR) as well as diagnostic odds ratio were generated. Positive and negative predictive values were not calculated since they are dependent on prevalence of stroke subtypes which was different across included studies. We assessed heterogeneity graphically using forest plots and statistically using the quantity I2 . Analysis was performed on STATA v11.0 (Stata Corp., TX) using the ‘midas’ command and Review Manager Software version 5.[13]

Results

Study selection

A total of 115 relevant papers were identified in the literature search of which 39 were retrieved for further examination. Three papers were retrieved from additional sources. Of the 42 papers retrieved for detailed examination, 22 were excluded and 20 papers were included in the review. Figure 1 outlines the study selection process.

Figure 1

Flow diagram of study selection

Characteristics of included studies

Included studies were hospital-based diagnostic test accuracy studies conducted within a 20 year period (1991-2011) in Asia,[61415161718192021] Eastern Europe[2223] and Africa[242526272829303132] (including north Africa.[27] ) The aim in each study was evaluating the diagnostic performance of the index tests (SSS, GHSS and GSS) in distinguishing stroke sub-types among patients with acute stroke. CT scan was the documented reference standard in most studies. The SSS was evaluated in 18 studies while the GHSS was evaluated in 11 studies. The GSS was evaluated in three studies. No study evaluated the BS.

Two studies[2931] were retrospective in design while the rest were prospective studies. The patient spectrum was mainly adult patients with acute stroke. In a number of studies, patients were only included if they had a CT scan performed. Reasons for lack of CT scan in patients who did not have the investigation were not elaborated in most studies. Further, many studies did not report characteristics of excluded patients for comparison with those included.

Overall, a total of 3638 patients from LAMICs were evaluated. Participants were of age range 14-87 years. Only two studies reported the severity of stroke among included patients.[1425] Prevalence of hemorrhagic stroke (including SAH) ranged from 17.3% to 68.9% (median prevalence 37.85%; IQR 30.25%-48.76%) while prevalence of ischemic stroke ranged from 31% to 82.7% (median prevalence 58.75%; IQR 52.82%-68.5%). In five studies[1417222730] equivocal results were excluded from the estimation of diagnostic performance of the index tests.

The main outcomes reported in the studies were test sensitivity and specificity. However, positive and negative predictive values as well as likelihood ratios were reported in a number of studies. About half of the studies recommended the scores for use in the absence of CT scan for both clinical and epidemiological studies. Overall, the studies recommend larger and better designed studies for validation of the scores in addition to refining them. Table 2 presents a summary of the characteristics of included studies.

Table 2

Characteristics of included studies from low and middle income countries

Methodological quality of included studies

Overall, studies included in the review were of sound quality despite incomplete reporting in some of them. Spectrum bias, disease progression bias, partial verification bias, differential verification bias and incorporation bias were adequately minimized. Blinding of index test and reference standard results was poorly reported across studies. All studies reported indeterminate results although not all of them included these results in the assessment of test performance.

Meta-analysis results

Accuracy of the Siriraj stroke score

Sensitivity of the test for ischemic stroke ranged from 0.30 to 0.85 while specificity ranged from 0.36 to 0.97. Overall, sensitivity for the test was 0.69 (95% CI 0.62-0.75) and specificity was 0.83 (95% CI 0.75-0.88) for ischemic stroke. There was significant statistical heterogeneity observed (I2 = 90.25 for sensitivity and I2 = 86.30 for specificity). The positive likelihood ratio (+LR) was 4 (95% CI 2.7-5.8) while the negative likelihood ratio (-LR) was 0.38 (95% CI 0.31-0.47).The test's diagnostic odds ratio (DOR) was 10 (95% CI 6-17).

Sensitivity for hemorrhagic stroke ranged from 0.33 to 0.87 while specificity ranged from 0.65 to 0.99. Overall, sensitivity for the test was 0.65 (95% CI 0.56-0.73) and specificity was 0.88 (95% CI 0.83-0.91) for hemorrhagic stroke. There was substantial heterogeneity observed (I2 = 86.35 for sensitivity and I2 = 83.86 for specificity). The +LR was 5.2 (95% CI 3.5-7.7) while the –LR was 0.40 (95% CI 0.31-0.51).The test's DOR was 13 (95% CI 7-23). Figures 2 and 3 show forest plots of sensitivity and specificity for the SSS for both ischemic and hemorrhagic stroke respectively for included studies from LAMICs.

Figure 2

Coupled forest plots of sensitivity and specificity for stroke scores for studies from LAMICs (Ischemic stroke)

Figure 3

Coupled forest plots of sensitivity and specificity for stroke scores for studies from LAMICs (Hemorrhagic stroke)

Accuracy of the Guys hospital stroke score

Sensitivity for ischemic stroke ranged from 0.25 to 0.93 while specificity ranged from 0.50 to 0.97. Overall, sensitivity for the test was 0.7 (95% CI 0.53-0.83) and specificity was 0.79 (95% CI 0.68-0.87) for ischemic stroke. There was significant heterogeneity observed (I2 = 97.33 for sensitivity and I2 = 89.23 for specificity). The +LR was 3.4 (95% CI 2.4-4.7) while the -LR was 0.38 (95% CI 0.25-0.58). The DOR was 9 with a 95% CI of 5-15.

Sensitivity for hemorrhagic stroke ranged from 0.20 to 0.84 while specificity ranged from 0.48 to 1.00. Overall, sensitivity for the test was 0.54 (95% CI 0.42-0.66) and specificity was 0.89 specific (95% CI 0.83-0.94) for hemorrhagic stroke. There was significant heterogeneity observed (I2 = 86.69 for sensitivity and I2 = 90.92 for specificity). The + LR was 5.2 (95% CI 3.2-8.4) while the -LR was 0.51 (95% CI 0.40-0.66). The DOR was 10 (95% CI 5-19). Figures 2 and 3 show forest plots of sensitivity and specificity for the GHSS for both ischemic and hemorrhagic stroke for included studies from LAMICs.

Accuracy of the Greek stroke score

Statistical pooling of results was not undertaken due to the small number of studies evaluating this test. Sensitivity for ischemic stroke ranged from 0.39 to 0.64 while specificity ranged from 0.63 to 0.88. Sensitivity for hemorrhagic stroke ranged from 0.11 to 0.44 while specificity ranged from 0.63 to 0.96. There was graphical heterogeneity observed. Figures 2 and 3 show forest plots of sensitivity and specificity for the GSS for both ischemic and hemorrhagic stroke for included studies from LAMICs.

Discussion

Adequate management of acute stroke requires early distinction of ischemic stroke from hemorrhagic stroke. Before starting anticoagulant, antiplatelet or thrombolytic therapy for ischemic stroke, hemorrhagic stroke must be accurately ruled out. CT scan is sensitive for hemorrhagic stroke while MRI has been shown to have high sensitivity for early ischemic stroke.[3334] However, both are expensive and unavailable in resource poor settings.[11]

We identified 18 studies that validated the SSS in LAMICs from Africa, Asia and Eastern Europe. Only about half of these recommended the score for either clinical or epidemiological use. Overall, we found the test to have a sensitivity of 0.63 (95% CI 0.54-0.72) and specificity of 0.88 (95% CI 0.82-0.91) for hemorrhagic stroke. Since the exclusion of hemorrhage in patients with acute stroke is important before starting anticoagulant therapy for those with ischemic stroke, the SSS is not sufficiently sensitive to exclude hemorrhage. Also, the likelihood ratios for the test for hemorrhagic stroke show that it only very minimally changes the post-test probability of having hemorrhage. Further, in order to be sufficiently sure of the presence of ischemic stroke to guide on management, the clinical score needs to be specific enough to rule in ischemia. The SSS has a specificity of 0.82 (95% CI 0.74-0.88) for ischemic stroke and a positive likelihood ratio of 3.9 (95% CI 3.4-7.7). These estimates suggest that the score is not accurate enough for clinical or epidemiological use.

The poor accuracy of the SSS in LAMICs may be attributable to various factors. First, most studies had small sample sizes. Secondly, the SSS was developed in a country with a very high prevalence of hemorrhagic stroke relative to other countries and thus its accuracy may differ from its initial validation result. The inclusion of equivocal test results in our calculations may also explain the poor accuracy. However, this inclusion is necessary so as to give a true picture of the test's accuracy.

The Guy's hospital stroke score was developed on a predominantly young population which is known to have a higher prevalence of hemorrhagic stroke compared to an older population. The authors did not endorse its use for clinical management but suggested it may be used for epidemiological studies. The score has been validated in LAMICs with differing conclusions. In this review, 11 studies evaluated the GHSS. Specificity for ischemic stroke was 0.76 (95% CI 0.64-0.85) with a positive likelihood ratio of 2.9 (95% CI 2.2-3.9). These estimates are not sufficient enough to confirm ischemic stroke and commence anticoagulant therapy. Sensitivity for hemorrhagic stroke was 0.49 (95% CI 0.38-0.61) which shows poor accuracy for ruling out hemorrhage. In addition to its poor diagnostic accuracy, the GHSS is cumbersome and may not be practical in resource poor settings.

Three studies from LAMICs evaluated the Greek score and due to this small number, we did not include them in meta-analysis. The score was developed in Greece and is simple to use. It was found to be very accurate in its initial validation study. Overall, the score showed poor diagnostic performance in this review. It is probable that the higher prevalence of hemorrhagic stroke in resource poor settings affected its accuracy.

We found a high prevalence of hemorrhagic stroke among studies from LAMICs included in this review (median prevalence 37.85%; IQR 30.25%-48.76%). This is in keeping with other reports of higher prevalence of hemorrhagic stroke in patients from poorer countries.[1112] One author attributes this to inadequate control of blood pressure as well as a high occurrence of aneurysms and arteriovenous malformations in poorer countries.[11] Another author suggests that the dramatic presentation of hemorrhagic stroke inclines clinicians to request for CT scan hence the higher prevalence.[12] We propose that both scenarios may contribute.

Our review has various strengths. Firstly, the included studies had an appropriate spectrum of patients. Secondly, in all the included studies, the stroke scores and reference standards were performed on the same patient population and the whole sample received verification with the reference standard. Thus, partial verification bias and differential verification bias were adequately avoided. Thirdly, the time between administration of stroke scores and reference standards was short enough to avoid disease progression bias. In our estimation of diagnostic performance indices, we included equivocal test scores so as to avoid bias in the estimates.

There were a number of weaknesses. First, there was poor reporting of blinding across all studies, thus the bias attributable to this could not be determined. Secondly, included studies may have had selection bias as some studies excluded patients without CT scan results. While reasons for this exclusion were not elaborated in a majority of the reports, it is possible that in LAMICs, cost of CT scan was a hindrance to access as was shown in one of the studies.[26] Thirdly, a majority of studies did not report on the severity of stroke among included patients. This is of particular interest since the scores' performance may vary with stroke severity. Finally, some causes of stroke e.g. sickle cell disease and rheumatic heart disease are of significance in the pediatric age group.[11] However, no study assessed the utility of the scores in this group.

In conclusion, we do not find the SSS, GHSS and GSS to be of sufficient accuracy to warrant routine use (clinical or epidemiological). The GHSS is cumbersome and may not be practical in resource poor settings. While the GSS is simple to use, few studies from around the globe have evaluated it. The SSS has been widely validated but it may need to be modified to suit different populations. We recommend the continued use of CT scan for differentiating stroke subtypes in the absence of a sufficiently accurate clinical stroke score and advice on investment in neuroimaging equipment for use in resource poor settings. Further, larger studies in which the cost of CT scan does not influence patient selection are required for validation of the SSS, GHSS and GSS especially in resource poor countries. Newer scores using patient populations from each region may be warranted.