The Reliability of the Evaluation of Diffusion-Weighted Imaging in Suspected Stroke by Emergency Physicians.

Background Stroke is a leading cause of serious disability and death and its accurate and timely diagnosis is critical for better stroke outcomes. However, recent studies indicate that stroke is among the most common dangerous missed diagnoses. Diffusion-weighted imaging (DWI), which was first introduced in the early 1990s among clinical concerns, has become an invaluable tool in the evaluation processes of patients with suspected ischemic stroke. Objective Herein, it is aimed to investigate the reliability of the interpretation of DWI in the diagnosis of stroke by emergency physicians. Methods DWIs of the patients, evaluated in the Emergency Service of Yozgat City Hospital from April 1, 2017 to September 1, 2017, were retrospectively evaluated by four emergency physicians (specialists) separately in a blind and random fashion. The reliability of the evaluations was calculated using Kappa analyses on SPSS Statistics 20. Results DWI of 154 patients was enrolled in this study. Kappa values were determined to be as 0.911 (almost perfect), 0.909 (almost perfect), 0.897 (almost perfect), 0.779 (substantial), respectively. Discussion The results of this study may suggest that the reliability of DWI interpretation by emergency physicians was pretty high, even it can still be improved. Combining the literature data remarking a substantial number of misdiagnosed stroke patients in the emergency department (ED) and study results, it can be hypothesized that the major problem may be related with rather clinical evaluation processes as well as insufficient neurology consultation. Future reports are warranted to understand the basic problems of stroke evaluation processes in the ED. The results of these studies may aid to develop better solutions while constituting effective stroke programs and initiatives.

Introduction

In emergency practice, computed tomography (CT) of the brain is the most commonly used imaging methods in patients with a provisional diagnosis of stroke; however, it is known that CT fails to demonstrate the lesions in 30 to 60% of these patients in the acute phase. 1 On the other hand, diffusion-weighted imaging (DWI) which was first introduced in the early 1990s among clinical concerns has become an invaluable tool in the evaluation processes of patients with suspected ischemic stroke, 2 due to which, DWI has been reported to be efficient at a sensitivity of 88% and a specificity of 95% in detecting the ischemic lesion. 3

On the other hand, stroke is still a leading cause of severe disability and death, and its accurate and timely diagnosis is critical for better stroke outcomes. However, the diagnosis may be challenging in the emergency department (ED) as recent studies indicate that stroke is among the most common dangerous missed diagnoses. 4 5 6 As a result of these reports, some clinical remarks, such as being more attuned to the circumstances in suspected stroke in the younger female, and non-White patients, have been proposed. 4 Based on a similar, but distinct point of view we focused on the neuroimaging evaluation of these patients with a suspected stroke. Therefore, the reliability of the interpretation of DWI is investigated by emergency physicians. Remarkably, the reliability of DWI evaluation of patients with suspected stroke may be a critical remark as in many cases the initial evaluation of DWI is made by emergency physician, particularly in centers without adequate facilities and 24-hour radiology service. Therefore, we think that the results of our report may provide distinct perspectives to the issue of missed diagnosis of stroke in the ED in a substantial part of the centers.

Materials and Methods

This retrospective observational cross-sectional study was performed in Yozgat City Hospital. DWIs of the patients, who were admitted with acute neurological conditions and evaluated in the Emergency Service of Yozgat City Hospital from April 1, 2017 to September 1, 2017, were retrospectively evaluated by four emergency physicians (specialists) separately blindly and randomly. They were unaware of the official radiology report and patients’ clinical status at the time of imaging. The reference evaluation was performed by a neurologist (H.O.) who also had the data of radiology reports of the DWI and patients’ clinical status and follow-up data of the individuals, hospitalized after emergency service monitoring. The reviewers were asked to note, if diffusion restriction was present and where was the localization of restricted diffusion, if present. The reliability of the evaluations was calculated using Kappa analyses on SPSS Statistics 20. Also, all DWIs of false negative and positive results were re-evaluated. These specific lesion sites and false positive regions were demonstrated for better documentation and clarification of the major problems in DWI interpretation among emergency physicians. The necessary permission and approval was received from the research ethics committee of Bozok University (Registration number 2017-KAEK-189_2019.11.27_21).

MRI Protocol

All magnetic resonance imaging (MRI) scans were performed on a Siemens 1.5 Tesla Magnetom Amira MRI scanner using 5-mm slice thickness and apparent diffusion coefficient (ADC).

Statistical Analysis

Cohen’s Kappa statistics on SPSS 20 were used to investigate the reliability of the observers’ evaluations according to the reference evaluation. Other parameters including sensitivity, specificity, positive predictive value, negative predictive value, true positive–negative, false positive–negative diagnoses, and total discordance rates were evaluated using SPSS statistics 20 for all observers, separately. The interpretation of Kappa result was made as suggested by Cohen as follows: values≤ 0 as indicating no agreement and 0.01 to 0.20 as none to slight, 0.21 to 0.40 as fair, 0.41 to 0.60 as moderate, 0.61 to 0.80 as substantial, and 0.81 to 1.00 as almost perfect agreement.

Results

DWIs of 154 patients was enrolled in this study (Male/Female: 82/72). After accurate evaluation by neurologist, diffusion restriction was found to exist in 104 of them. Kappa values were determined to be as 0.911 (almost perfect), 0.909 (almost perfect), 0.897 (almost perfect), 0.779 (substantial), respectively. Total discordance rates, according to neurologist’s evaluation, were 3.8, 3.8, 4.5, 9.7%, respectively. The sensitivity rates of the evaluations ranged from 86 to 97%, and specificity rates ranged from 90 to 98% (other statistical analyses results are shown in Tables 1 2 . In total, 12 false positive evaluations (1.9%) were detected. The most common localization of false-positive results was temporal lobe ( n = 3). In a unique case with an epidermoid cyst, all the observers evaluated the image as ischemic lesion based on the diffusion restriction ( Fig. 1A ). In another patient, the high signal in adjacent to temporal lobe was misdiagnosed as diffusion restriction ( Fig. 1B ). On the other hand, in total, 22 (3.6%) false negative evaluations were detected. The most common localizations were brain stem ( n = 6), cerebellum ( n = 3), and thalamus ( n = 2), respectively. Small localizations of diffusion restriction were tended to be overlooked. Patient 1 and 2 with a diagnosis of ventrolateral pontine infarction were overlooked by one of the observers ( Fig. 2C D ). Patient 3, showing bilateral thalamic infarction compatible with occlusion of the artery of Percheron, was misdiagnosed by one of the observers ( Fig. 2E ). Patient 4 with DWI showing median, punctate medullar diffusion restriction, was overlooked by two of the observers ( Fig. 2B ).

Table 1

Sensitivity, specificity, positive predictive value, negative predictive value, K-value

Observers	Correct diagnosis		False diagnosis		K -value
	Sensitivity	Specificity	PPV	NPV
Abbreviations: NPV, negative predictive value; PPD, positive predictive value.
1.	97%	94%	97%	94%	0.911 (almost perfect)
2.	99%	90%	95%	98%	0.909 (almost perfect)
3.	96%	94%	97%	92%	0.897 (almost perfect)
4.	86%	98%	99%	78%	0.779 (substantial)

Table 2

True positive, true negative, false positive false negative, total discordance rates

Observers	Correct diagnosis		False diagnosis		Total discordance ratio
	True positive	True negative	False positive	False negative
1.	101 (65.6%)	47 (30.5%)	3 (1.9%)	3 (1.9%)	3.8%
2.	103 (66.9%)	45 (29.2%)	5 (3.2%)	1 (0.6%)	3.8%
3.	100 (64.9%)	47 (30.5)	3 (1.9%)	4 (2.6%)	4.5%
4.	90 (58.4%)	49 (31.8%)	1 (0.6%)	14 (9.1%)	9.7%

Fig. 1

( A–C ) Diffusion-weighted imaging samples of false-positive interpretations.

Fig. 2

( A–F ) Diffusion-weighted imaging samples of false-negative interpretations.

Discussion

The ED is a high-risk site for misdiagnosis and preventable errors as well. Remarkably, it has been emphasized that neurological conditions constitute a major group of diagnostic claims in the ED, 7 due to which, it has been found that acute stroke claims were more than twice as common as those for acute myocardial infarction. 7 Clinically, milder, transient, nonspecific symptoms, and younger and female patients in stroke carries the greatest risk for misdiagnosis. 4 7 As a result of their large-scale study, Toker et al found that there were 15,000 to 165,000 misdiagnosed cerebrovascular events annually in U.S. EDs those were prominent in cases presenting with a headache or dizziness. 4 Of note, the first medical contact of an acute stroke victim is generally a non-neurologist who is often a hospital emergency service physician. Hence, the diagnostic accuracy of stroke made in the initial evaluation by the emergency physician constitutes a critical point for timely approach and administration of the appropriate treatment. Based on this consideration, Ferro et al specifically investigated the accuracy of stroke diagnosis made by the emergency physician in which they found that in the majority of the cases, non-neurologists could make a correct diagnosis of acute stroke. 8 In another study including a larger group of patients ( n = 655) who were admitted for neurological reasons, the sensitivity of stroke diagnosis made by emergency doctors was found to be very high (97%), but the specificity and positive predictive value of that diagnosis were low. 9 Based on these results, the authors remarked that the potential significant number of erroneously diagnosed patients is due to an unnecessary risk of brain hemorrhage based on the diagnoses performed by emergency doctors, and they questioned the rationale of deciding on thrombolytic treatment in community hospitals without neurology evaluation. 9 At this point, neuroimaging certainly constitutes the critical stage for confirmation of stroke as well as making the proper differential diagnosis. Cranial CT, which is a more conventional method of imaging, may miss the early infarct, particularly the small infarct volumes. 10 However, since its introduction into clinical practice, DWI has formed an invaluable paraclinical method for the diagnosis of stroke, which has a sensitivity exceeding 90% for detecting acute ischemia. 11 On the other hand, the reliability of the interpretation of DWI may potentially vary among clinicians, and its difference even among radiologists, according to their level of training residency, has been reported previously. 12 In that report by Lee et al, in the ED, interobserver reliability of DWI among four radiology residents with different training years was investigated which was found to be good as the main conclusion. However, the level of training was also found to be positively associated with diagnostic performance. 12 From a distinct point of view, we think that the reliability of interpretation of DWI in the ED by emergency physicians may constitute a critical issue among clinical concerns as a substantial rate of the DWIs performed in the ED is evaluated by emergency physicians in centers without adequate facilities and 24-hour radiology service. 13 The interpretation results of the DWI performed in patients with suspected neurological clinics influence the decision of the clinicians (emergency physician) substantially, because of which we know that a substantial number of patients who are admitted to our ED and investigated with DWI, need not consult neurology doctor if DWI is interpreted as normal by the emergency physician. Ergo, we aimed to investigate the issue of stroke diagnosis in emergency service via a distinct method only canalizing the reliability of the interpretation of DWI by emergency physicians. Up to date, in only a single study, the reliability of DWI in screening for ischemic stroke was investigated among emergency physicians. 14 Using Fleiss’ K statistics, they found that the mean kappa values for DWI and ADC maps were substantial and moderate, respectively. In conclusion, they suggested DWI as a reliable tool that may be interpreted by emergency physicians in the ED. 14 As far as we know, our report is the second study conducted in this regard. The results of our analyses showed that the reliability rates of interpretation of DWI by the emergency physicians were pretty high (almost perfect, or substantial). Remarkably, we used a distinct method of Cohen’s Kappa statistics on SPSS 20 to investigate the reliability of the observers’ evaluations according to the reference evaluation. In our opinion, the examination of the diagnostic accuracy of these evaluations according to the reference value would be more rational in contrast to focusing on the interobserver reliability that was investigated by Oray et al. 14

Also, we also aimed to suggest some lessons to be kept in mind to develop the diagnostic success of stroke diagnosis in the ED. False negative results were determined in 22 (3.6%) of the overall evaluations. The most commonly missed localizations of diffusion restriction were brain stem ( n = 6), cerebellum ( n = 3), and thalamus ( n = 2), respectively. This result was also in accordance with the literature data emphasizing the posterior circulation ischemia as a major risk factor for negative diffusion stroke. 15 Besides, in 12 of the patients, false positive results were determined. In a unique case with an epidermoid cyst, all the observers evaluated the image as ischemic lesion based on the diffusion restriction. The lesion was hyperintense in DWI and hypointense in ADC suggesting ischemic lesion. It has been stated that epidermoid cysts demonstrate restricted diffusion secondary to the microstructure of parallel-layered keratin filaments and flakes leading to challenges in discrimination from an arterial stroke. 16 However, the lesion localization was extra-parenchymal and did not correspond to any vascular territory. Also, a slight mass effect was evident. In another patient, the high signal in adjacent to temporal lobe was misdiagnosed as diffusion restriction. When the region was re-investigated, it was evaluated as temporal lobe DWI artifact. However, pronounced artifacts at the anterior temporal lobes and around the pons due to mechanical vibration are known entities in DWI. 17 We think that a short training program about the DWI artifacts may furthermore increase the specificity rates which were already evaluated as high.

On the other hand, we know that there is a substantial ratio of patients with initial negative DWI, which may constitute another topic of discussion, 18 due to which, up to 21% rates of initial negative DWI have been reported in clinically diagnosed stroke patients. In conclusion of these studies, some risk factors, such as brain stem location, lacunar syndrome, early recoding time of DWI, have been emphasized as risk factors of negative initial DWI results. 15 18 19 Besides, it has been acknowledged that a negative scan is the most common DWI findings in patients with a definite transient ischemic attack which is often labeled as “mini-strokes.” 20 Ergo, it can be understood that stroke is not a diagnosis to be made by solely neuroimaging methods. Instead, it requires a multimodal approach of which clinical evaluation constitutes a vital component. Therefore, we would like to remark that the results of our study cannot be interpreted as that the evaluation of patients with suspected stroke by solely emergency physician is sufficient and reliable. In admissions of neurological presentations, neurology consultation should be done without hesitation, to avoid the missed diagnosis of particularly atypical scenarios of stroke, even if DWI is normal. However, the results of this study may rather suggest that the reliability of DWI interpretation by emergency physicians was pretty good, even it can still be improved. Combining the literature data remarking a substantial number of misdiagnosed stroke patients in the ED 4 and our study results, we can hypothesize that the significant problem may be related with rather clinical evaluation processes as well as insufficient neurology consultation. Future reports are warranted to understand the basic problems of stroke evaluation processes in the ED. The results of these studies may aid to develop better solutions while constituting effective stroke programs and initiatives.