Reliability and agreement of point-of-care carotid artery examinations by experts using hand-held ultrasound devices in patients with ischaemic stroke or transitory ischaemic attack.

OBJECTIVES: To investigate the reliability and agreement of hand-held ultrasound devices (HUDs) compared with conventional duplex ultrasound (HIGH) in examination for carotid stenosis in patients with suspected transitory ischaemic attack (TIA) or ischaemic stroke.
METHODS: Cardiologists, experienced in carotid ultrasound, examined patients admitted to a community hospital with suspected stroke or TIA. Patients were first examined by an HUD and second by HIGH as per usual care. Different operators performed HUD and HIGH blinded to each other. On clinical discretion, CT angiography (CTA) was performed, and analysed by a radiologist blinded to the results from the ultrasound.
RESULTS: Of 80 patients included, 9 (11%) were found to have >50% internal carotid artery (ICA) stenosis on reference examination. Agreement for classification of the degree of ICA stenosis was good for HUD versus HIGH (weighted Kappa 0.76) and HUD versus CTA (weighted Kappa 0.66). Agreement between HUD and HIGH examinations was excellent when ICA was classified as <50% diameter stenosis by HUD (99% agreement), but significantly lower when ICA diameter stenosis was classified as >50% by HUD (OR 0.15, 95% CI 0.06 to 0.42). Overall, HUD tended to overestimate the degree of carotid stenoses rather than underestimate (p=0.048).
CONCLUSION: Hand-held carotid ultrasound performed by experts demonstrated good agreement with conventional duplex ultrasound. The use of HUDs was reliable for ruling out significant carotid artery disease, but less reliable for ruling in significant disease. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Carotid stenosis is the underlying cause in approximately 15% of patients with stroke or transient ischaemic attack.

Carotid duplex ultrasound is the first line examination for carotid stenosis due to its non-invasive nature.

Hand-held ultrasound examinations have been shown able to provide accurate and reliable assessment of measures of cardiac size and function.

The agreement between hand-held carotid ultrasound examination and reference was overall good, and the agreement was excellent when the carotid artery stenosis was classified as less than 50% diameter stenosis by the hand-held ultrasound examination.

The agreement between hand-held carotid ultrasound examination and reference was modest in individuals with at least 50% diameter stenosis by hand-held ultrasound, and the examinations by hand-held ultrasound tended to overestimate the degree of carotid stenoses rather than underestimate.

Hand-held carotid ultrasound performed by experts was reliable for ruling out significant carotid artery disease, but less reliable for ruling in significant disease.

This supports the role of hand-held carotid ultrasound performed by adequately trained personnel to optimise the in-hospital workflow of patients with suspected stroke or transient ischaemic attack.

Introduction

Stroke remains one of the leading causes of death and disability worldwide,1 with carotid stenosis being one of few directly modifiable causes. Carotid endarterectomy has been proven to be effective in reducing future risk of stroke in patients with symptomatic carotid stenosis, but only when performed in the subacute phase after a stroke or transitory ischaemic attack (TIA).2–4 Physical examination and auscultation are unable to detect a significant carotid stenosis accurately and reliably. Thus, it is recommended to routinely examine all patients with stroke or TIA with diagnostic imaging.5

Carotid duplex ultrasound is recommended as the first line examination for carotid stenosis due to its non-invasive nature. Other modalities serve as confirmatory and discretionary tests due to (1) the resource and time demands of magnetic resonance angiography, (2) the use of potentially nephrotoxic intravenous contrast and radiation by CT angiography (CTA) and (3) the procedural risk of stroke (conventional angiography).6 Still, conventional high-end ultrasound requires expensive and relatively stationary equipment, dedicated labs and trained personnel. As carotid stenosis is the underlying cause in only approximately 15% of patients with stroke or TIA, the resources and time invested to identify the patients who could benefit from carotid endarterectomy is significant.7 Thus, improving the selection of patients in need for more comprehensive carotid imaging would improve in-hospital logistics, which is a major contributor to delayed referral to carotid endarterectomy.8

Point-of-care ultrasound in general and hand-held ultrasound devices (HUDs) specifically have been increasingly adopted in diverse fields as primary care,9 cardiology10–14 and emergency medicine,15 bridging the gap between physical examination and advanced imaging modalities due to their ease of use, steadily increasing affordability and accessibility.16 The potential for improvement in hospital logistics and use of resources is significant,17 and specifically the potential for rapid ruling-out individuals not in need for further imaging would be beneficial. Thus, there is a need to rigorously evaluate the reliability of these novel tools before their adoption into new clinical arenas and, to our knowledge, this has not been studied in patients admitted with stroke or TIA. Thus, we aimed to investigate the agreement of HUD compared with standard of care high-end vascular ultrasound (HIGH) performed by cardiologists experienced in carotid ultrasound. Second, we aimed to compare the agreement between HUD and reference according to the burden of carotid disease and lastly to evaluate potential predictors for the reliability of the HUD examinations.

Materials and methods

Study population

Patients referred for carotid ultrasound from the stroke unit at Levanger hospital, Norway admitted with suspected stroke or TIA were included. Inclusion was restricted to days where at least two out of four cardiologists experienced in carotid duplex ultrasound were present (to enable one cardiologist performing each of the examinations by HUD and HIGH, respectively). Exclusion criteria was lack of consent, or previous examination of the neck arteries within the last year. A sample size of 80 participants was estimated to yield 80% power to detect at least 7% using SamplePower (SPSS, Chicago, Illinois, USA)

The study was registered in the ClinicalTrial.gov database (unique ID: NCT02141932).

Examinations of the carotid arteries by HUDs

The methodology for this study has been described recently.17 In short, examinations with HUDs were performed bedside by a cardiologist (in the patients’ room on the ward) using a Vscan with Dual Probe (V.1.4; release 1 and 2, GE Ultrasound AS, Horten, Norway). Examinations followed a standardised protocol. The arteries imaged were the common carotid artery (CCA), the internal carotid artery (ICA) and the external carotid artery (ECA). The severity of carotid stenosis was graded on an ordinal scale from 1 through six based on visual assessment supported by linear measurements of dimension if needed (1, normal; 2, increased intima media thickness; 3, less than 50% stenosis; 4, 50%–69% stenosis; 5, 70%–99% stenosis; 6, near-occlusion and 7, occluded). The morphology of plaques was categorised as hypoechoic, isoechoic or hyperechoic. Image quality was classified as low, moderate or good by external review by one of the four cardiologists (HD).

The HUD, Vscan with Dual Probe, has both a phased array and a linear array probe and is capable of displaying and storing 2D grey scale images as well as colour doppler flow images. There is no available option for spectral Doppler measurements.

Reference imaging by high-end ultrasound

The reference examination was performed in a dedicated echocardiographic laboratory. A high-end ultrasound scanner (Vivid E9, GE Ultrasound AS) with phased array cardiac (M5S) and linear vascular (9 L) transducer was used. In each case, a cardiologist (another than the one who performed the HUD examination) performed the reference examination blinded to the results from the preceding HUD examination. The delay between the two carotid ultrasound examinations was sought to be less than 1 hour disregarding the results on HUD examination.

The classification of carotid disease within the vascular territories was graded from 1 through 6 as described above for HUD. Stenoses up to 50% diameter reduction were classified based on 2D measurements of dimensions and colour Doppler images. Higher degree stenoses were assessed using spectral Doppler velocities as per recommendations from American Society of Radiologists in Ultrasound, incorporating peak systolic velocity in the stenosis, its ratio to the peak systolic velocity in the CCA, and additionally the end-diastolic velocity through the stenosis.18 Plaque morphology was assessed as stated for examinations by HUD. Potential cardiac sources of emboli were assessed by transthoracic echocardiography, and baseline echocardiographic measurements were reported.

CTA was performed when judged clinically indicated by the treating clinician. A radiologist blinded to the results from the examinations by both HUD and HIGH assessed the vascular territories according to the European Carotid Surgery Trial convention, using the ratio of the residual lumen to the assumed original diameter of the vessel.19 They were subsequently categorised on the same ordinal scale as HUD and HIGH, with the exception of the intimal thickening category being dropped.

Sample size estimation

Sample size was based on inclusion of a minimum of five patients with ICA stenosis and the ability to detect a 50% reduction in need of advanced imaging by ruling out carotid artery disease by HUDs. With an estimated prevalence of minimum 7% with ICA stenosis we needed a sample of 72 patients, and by SamplePower (SPSS, Chicago, Illinois, USA) we needed a sample of 10 patients to detect a 50% reduction in need of advanced imaging. To account for some dropouts we wanted to include at least 80 patients.

Statistical analysis

Continuous variables following a normal distribution are presented as mean (SD), otherwise median (range). Categorical data are presented as proportions and numbers. Chance corrected agreement was assessed by weighted Cohen’s Kappa, with Fleiss-Cohen weights.20 Polychoric correlation, which estimates the correlation of assumed continuous latent variables underlying ordinal variables, was calculated as a measure of association.21 The influence of (1) disease severity by the grade of the stenosis evaluated by HUD, (2) image quality and (3) the HUD operator on the probability of agreement was explored using univariate logistic regression and likelihood ratio tests. A coefficient of rater bias was calculated as the ratio of overestimated carotid disease by HUD to the total number of misclassified stenoses, with values above 0.5 signifying systematic overestimation, and values below 0.5 signifying underestimation. The McNemar test was used to assess the significance of the bias. CI for proportions was calculated as Jeffrey’s intervals.22 P values less than 0.05 were considered significant. All statistical analyses were performed using R V.4.0.2.

Results

Table 1 highlights the baseline characteristics of the 80 patients (38 women, 42 men) admitted with suspected stroke or TIA included in the study. Figure 1 shows the flowchart of the study. These data have been recently published.17 In short, median age was 70 (23–93) and 22 patients (27.5%) had a previous history of cardiovascular ischaemic events (defined as ischaemic stroke, TIA or myocardial infarction) and 62 patients (77.5%) were finally discharged with a diagnosis of ischaemic stroke or TIA. The four operators had 6–25 years (median 14) of experience with carotid Doppler ultrasound.

Table 1

Basic characteristics of the 80 study participants

Variable
Women, n (%)	38 (47.5)
Age, median (range)	71.5 (23–93)
Body mass index, kg/m2	27.9±4
Systolic blood pressure, mm Hg	159±24
Diastolic blood pressure, mm Hg	84±16
Prior ischaemic stroke/haemorrhagic stroke/transitory ischaemic attack, n (%)	6 (7.5)/2 (2.5)/12 (15)
Known diabetes, n (%)	11 (14)
Treated for hypercholesterolemia, n (%)	15 (19)
HUD examinations per operator, n (SD)	20 (4.1%)
Significant ICA stenosis on reference exam, n (%)	9 (11)

*Values are mean±SD unless otherwise stated.

HUD, hand-held ultrasound device; ICA, internal carotid artery.

Figure 1

Flowchart of the study. The flow of the study participants is shown. Reference examination by carotid Doppler ultrasound was performed in all, while additionally CT angiography was performed only on clinical indication as decided by the treating physician (n=19, 24%). No patient was excluded from the analyses.

The agreement and reliability for grading of carotid stenoses by HUD

Internal carotid stenosis above 50% was found in 9 (7.5%) of ICAs, 3 (2%) of ECAs, but none of the common carotid arteries by reference ultrasound examinations. The distribution of stenoses by vessels and modality is shown in figure 2.

Figure 2

Distribution of stenosis classification by vessel and modality. Bar chart of number and proportion of vessels classified into the specific stenosis by modality and vascular territory. CCA, common carotid artery; CTA, CT angiography; ECA, external carotid artery; ICA, internal carotid artery; HIGH, high-end vascular ultrasound; HUD, hand-held ultrasound device.

As demonstrated in figure 3, of the 12 vessels graded as ≥50% diameter stenosis by HIGH, three were underestimated by HUD. Of these, two stenoses graded as 99% by HIGH were by HUD graded as 50%–69% and 70%–99%, respectively, and one 70%–99% stenosis was by HUD misclassified as <50%. Among the 22 ICA vessels categorised as >50% stenosis by HUD, 11 (50%) were reclassified as <50% by HIGH.

Figure 3

Contingency table for internal carotid artery grading of disease by examinations with hand-held ultrasound devices and reference carotid Doppler ultrasound. Marginal distribution is shown for both examination modes. Data in absolute numbers and percentages are provided per vessel. HIGH, high-end vascular ultrasound; HUD, hand-held ultrasound device.

Overall, the agreement between HUD and HIGH for classification of the degree of internal carotid stenosis was 69%. In table 2, the percentage agreement by categories of internal carotid stenoses is shown. Only two vessels showed 99% stenosis and only one ICA was occluded. Excluding these extremes, the agreement for correct grading on the 6-point category scale ranged 21%–84%. When dichotomising the degree of internal carotid stenosis to below or at least 50%, the agreement between HUD and HIGH for stenoses judged as <50% by HUD was 99.3%, with only one misclassification as stated above.

Table 2

Agreement between hand-held ultrasound assessment of the internal carotid artery grading compared with reference

Degree stenosis by HIGH	Number of vessels, n (%)	Correct classification by HUD, % (95% CI)
No plaque	22 (14)	73 (52 to 88)
Intimal thickening	29 (18)	21 (10 to 38)
<50%	97 (61)	84 (75 to 90)
50%–69%	5 (3)	80 (36 to 98)
70%–99%	4 (2)	50 (15 to 88)
99%	2 (1)	0 (0 to 67)
100%	1 (1)	100 (15 to 100)

The raw agreement of correctly classified stenoses by examinations with hand-held ultrasound devices (HUD) versus reference (HIGH) is shown in percentages by the categories of the stenoses by HIGH.

Figure 4 illustrates relevant predictors for the reliability of grading internal carotid stenosis by HUD. The sub categorisation of internal carotid stenosis by HUD was less likely to be in agreement with HIGH when the stenosis was judged as ≥50% (OR for agreement 0.15 (95% CI 0.06 to 0.42)) using <50% stenosis as reference. There was some numeric variation among the four cardiologists performing the examinations by HUD, with percentage agreement ranging from 60% to 76%. This difference was not significant (p=0.5). Neither image quality (p=0.56), nor plaque characteristics (p=0.57) were predictors for the agreement between sub categorisation of internal carotid disease between HUD and HIGH.

Figure 4

Predictors for the agreement for grading of internal carotid stenoses between hand-held ultrasound examinations and reference. Logit models for probability of agreement between for grading of internal carotid stenoses by hand-held ultrasound devices (HUD) and reference examinations by carotid Doppler ultrasound (HIGH). Data provided per vessel.

Weighted Kappa for agreement between HUD and HIGH was 0.76 (95% CI 0.66 to 0.85), while weighted Kappa for agreement between HUD and CTA was 0.66 (95% CI 0.47 to 0.86). Kappa for agreement on less than or at least 50% ICA stenosis between HUD and HIGH was 0.60 (95% CI 0.39 to 0.8).

The polychoric correlation coefficient for the categorisation of ICA disease between HUD and HIGH was 0.86 (95% CI 0.78 to 0.91) and the corresponding coefficient between HUD and CTA was 0.85 (95% CI 0.56 to 0.96), respectively.

Overall, there was evidence of systematic overestimation of ICA stenosis by HUD compared with HIGH demonstrated by a coefficient of rater bias of 0.64 (95% CI 0.5 to 0.76, p=0.048). Importantly, there was no evidence of systematic over or underestimation among the vessels graded as <50% stenosis by HUD with a rater bias of 0.54 (95% CI 0.38 to 0.70), compared with 0.87 (95% CI 0.64 to 0.97) for those rated as ≥50% stenosis (difference; p=0.02).

Discussion

In this study of 80 patients hospitalised with suspected stroke or TIA, we found a strong correlation between ICA stenosis grading by HUD and HIGH. For vessels classified as less than 50% stenosis with HUD the agreement with reference was almost perfect, while there was some overestimation in stenoses categorised as at least 50% by HUD.

The population consisted of patients admitted with suspected stroke or TIA, and finally the diagnosis was confirmed in almost four out of five patients. This study did not aim to intervene with the in-hospital logistics, and thus, patients were included on suspicion of stroke or TIA even before the diagnosis was confirmed. This may, however, have contributed to a population with lower prevalence of carotid stenosis than if restricted to a population with confirmed stroke or TIA. The operators of both HUD and HIGH were cardiologist experienced in carotid ultrasound, as the carotid Doppler ultrasound diagnostics at the hospital had been performed by the cardiologists for more than twenty years. The characteristics of the study population is in line with others,7 23 and we believe that our results may be broadly generalised to other users with the same level of expertise.

Overall, the agreement between HUD and HIGH was substantial to excellent. We present data showing that HUD was particularly reliable when the internal carotid disease severity was less than 50% diameter stenosis. Few publications have evaluated the reliability of HUD to rule out carotid stenosis. In a recent publication screening for carotid stenosis by residents in patients referred for coronary artery bypass grafting, the diagnostics using the same HUD was reliable.24 Previously, mobile ultrasound systems have proved to be reliable for assessment of intimal thickness,25 and for ruling out significant carotid artery disease.23 The HUD used in this study lacks the capacity of spectral Doppler measurements which is present in more advanced mobile ultrasound systems. As shown by the presented results the proportion of misclassifications was higher and biased towards overestimation when the stenosis was categorised as at least 50%. In cases with that category of stenoses, the use of velocity measurements is recommended to assess the degree of stenosis,18 and our results show the lower reliability by visual assessment alone. Importantly, the examinations by HUD could rule-out at least 50% stenosis in a reliable way with agreement close to 100%.

Comparing agreement between methods and studies is challenging. The Cohen’s kappa is commonly used as a summary statistic of agreement on categorical or ordinal scales. However, it is difficult to interpret and not least compare Kappa values across studies for several reasons since it is influenced by factors other than agreement. First, it is sensitive to the number of categories of the scale. Second, it is affected by prevalence in each category, especially when the prevalence in one category is much higher than others as in our study.21 Lastly, it has been criticised as a measure that does not discriminate between different components of disagreement, such as random error and systematic bias. Thus, we also assessed the data by polychoric correlation which assumes that the ordinal variables are categorisations of underlying normally distributed continuous variables and estimates their correlation coefficient. The results can be seen as an estimate of agreement when systematic error or bias is ignored. The coefficient of rater bias on the other hand only estimates the degree to which the methods systematically disagree, ignoring random error. The regression analyses showed that the agreement between HUD and HIGH was significantly better for less than 50% stenosis compared at least 50% stenosis, and the systematic bias was confined to those classified as over 50%. These analyses are to our knowledge not available in other materials assessing the reliability of HUDs for evaluation of carotid stenosis. Shortly, these methods underscore the high reliability for ruling out significant carotid disease by HUD (using a cut-off of 50% diameter stenosis). In the subpopulation with ≥70% ICA stenosis by HIGH, HUD underestimated the degree of ICA stenosis. Whether this is related to the diagnostic properties of HUD or the methodology used is not known. Thus, HUD may best serve as a tool to reliably rule out carotid stenosis, and not for subclassification of patients with at least 50% diameter stenosis. We could not detect any evidence of significant influence of neither the operators, plaque characteristics nor image quality. However, all operators were skilled in carotid ultrasound and the number of significant stenoses in the population was too small to rule out meaningful influence of these factors.

Strength and limitations

The main strength of the study is the detailed and reliable information of assessment by both HUD and HIGH on the individual carotid vessels. As anticipated most individuals did not have significant ICA disease. We did not compare HUD with an absolute gold standard, but with the current standard of care. Thus, we cannot decisively say that disagreement was due to error by HUD rather than HIGH. As the operators were experienced in carotid ultrasound and the use of HUDs, the findings are only externally valid when HUD examinations are performed by very experienced operators. Each patient was only examined once by each modality, as such we cannot assess interrater agreement of HUD.

Conclusion

Examination of the carotid arteries by hand-held ultrasound examinations in patients with suspected stroke or TIA showed good agreement with high-end carotid Doppler ultrasound when performed by trained cardiologists. The use of HUDs was reliable for ruling out significant carotid artery disease. However, as agreement was lower when carotid stenosis was categorised as at least 50%, examinations by and-held ultrasound devices should not be used for detailed classification of carotid disease.