Vascular ultrasound for the diagnosis of giant cell arteritis: a reliability and agreement study based on a standardised training programme.

OBJECTIVE: To evaluate the impact of a standardised training programme including equipment adjustment for experienced musculoskeletal ultrasonographers without previous experience in vascular ultrasound (US) on the reliability of US in the diagnosis of giant cell arteritis (GCA).
METHODS: In this prospective, non-interventional observational cohort study, patients suspected of GCA were evaluated by US by one of five rheumatologists with long-standing experience in musculoskeletal US (>8 years), trained using a standardised training programme including equipment adjustment. Images of cranial and large vessels were subsequently evaluated first by the performing ultrasonographer and thereafter by a blinded external expert (gold standard).
RESULTS: In three Danish centres, 112 patients suspected of GCA were included. According to the external expert, vasculitis changes were seen in 66 patients, in 45 of them with only cranial involvement, in 14 with both cranial and large vessel involvement, while in seven patients isolated large vessel vasculitis was found. The reliability was excellent between the local ultrasonographer and the US expert for the overall GCA diagnosis regarding the diagnosis of cranial and for large vessel GCA, with an interobserver agreement of 95-96%, mean kappa values of 0.88-0.92 (95% CI 0.78 to 0.99). Excellent reliability (mean kappa 0.86-1.00) was also found for the US examination of the individual arteries (temporal, facial, common carotid and axillary).
CONCLUSION: The US training programme resulted in excellent agreement between trainees and an expert in patients suspected of GCA and may thus be applicable for implementation of vascular US in clinical practice. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

INTRODUCTION

Giant cell arteritis (GCA) is a systemic vasculitis involving large and medium-sized vessels in individuals older than 50 years.[1] Early diagnosis and treatment of patients with GCA are important due to the risk of significant complications including blindness and stroke.[2] Because of a high level of evidence of good test performance, accessibility, minimal invasiveness, low cost and good overall performance, EULAR recommends vascular ultrasound (US) of the temporal and axillary arteries as the primary imaging test in patients suspected of cranial GCA (cGCA).[3]

The EULAR recommendations[3] highlight that US examination should be performed by a trained specialist using appropriate equipment, operational procedures and settings. The reliability of US examinations, which has often been a concern, can be improved by specific training. A proposal for the technical and operational parameters has been developed[4]; nevertheless, no specific training programmes exist, all the studies in a systematic literature review being conducted by expert groups.[5]

Vascular US examinations are regarded as strongly operator-dependent, and to date, very few studies have investigated US reliability. When evaluating preselected images/videos from patients with an established GCA diagnosis, high interobserver and intraobserver agreement has been demonstrated in two studies.[6 7] However, only moderate agreements were found in the Ultrasound Compared to Biopsy of Temporal Arteries in the Diagnosis and Treatment of Giant Cell Arteritis (TABUL) study.[8] The reliability in real-time patient assessment has been evaluated in one study in patients with an established GCA diagnosis[9] and in two studies in patients suspected of GCA, with excellent agreement among sonographers.[10 11] In these two studies, only two sonographers from the same centre participated.

In the TABUL study,[8] the sonographers were either experts or less experienced sonographers, who had successfully passed a training programme that included a test on individual competence in interpreting US videos and US examinations of 10 healthy persons as well as of one patient with active GCA. This was the first attempt in which a standardised vascular US training programme in the diagnosis of GCA had been evaluated as part of a trial. The diagnostic accuracy of US in this study was significantly lower than that in recently published studies and in the latest meta-analysis.[5] These results are probably related to the limited US experience of a significant number of sonographers, as documented by a 17% increase in US sensitivity for GCA diagnosis in US scans performed by operators after having performed >10 scans within the study.[8]

The primary aim of this study was to evaluate the impact of standardised training programme including equipment adjustments for musculoskeletal experienced ultrasonographers with no experience in vascular US before training initiation on the reliability of US in the diagnosis of GCA. The secondary aim was to assess the agreement on GCA-US diagnosis between the trained GCA-US ultrasonographers and an expert on US in GCA.[12]

METHODS

Study design and sample size

The study was designed and reported according to the recommendations that are given in the ‘Guidelines for Reporting Reliability and Agreement Studies’ statement.[13] The sample size was determined by the design of the main study, at least 100 patients being considered as a representative (and feasible) sample.

Sampling method

This prospective, non-interventional observational cohort study was performed during 3 years (from April 2014 to June 2017) in three Danish centres. At baseline, relevant clinical and paraclinical data were collected. Consecutive patients suspected of having GCA were included in the study using the following criteria: age ≥50 years with signs and symptoms which indicated the presence of GCA: new localised headache, jaw claudication, tenderness and/or reduced pulsation of the temporal artery, scalp tenderness, new-onset visual disturbances (anterior ischaemic optic neuropathy and amaurosis fugax/diplopia), elevated inflammatory parameters without other explanation (C reactive protein (CRP) and/or erythrocyte sedimentation rate) and/or polymyalgic symptoms. The US examination and temporal arterial biopsy were performed within 7 days after diagnostic initiation (ie, enrolment). Patients with previous GCA diagnosis, use of more than 20 mg prednisolone for more than 7 days before the US examination and tissue sampling, long-term use (>1 month) of less than 20 mg prednisolone daily until 3 months before study start, mental disease and/or alcohol or drug abuse that affected the patients’ ability to give informed consent were excluded from the study. All participating patients gave written consent according to the Declaration of Helsinki. The study was conducted according to Good Clinical Practice.

Ultrasound set-up

At inclusion, a vascular scanning of 12 vessels was performed: bilateral evaluation of the facial arteries (FA), common carotid arteries (CA),AA and three sites on the TA (common superficial artery and parietal and frontal branches).

All US examinations in each centre were conducted with the same equipment (Hitachi Preirus/Ascendus in Esbjerg/Silkeborg, GE Logic E9 in Glostrup) and settings were kept unchanged throughout the study (online supplemental file I). Equipment settings, scanning techniques and image analysis were performed according to the International Workshop on Ultrasound in Large Vessel Vasculitis & Polymyalgia Rheumatica standards[14] and were in line with the proposal by Terslev et al. [4] All arteries were examined in longitudinal and transversal views. In both projections, B-mode and colour Doppler examination was performed. A video clip of at least 3 seconds applying the compression technique[15] on the TA and FA was stored. The intima-media thicknesses (IMT) of the CA and AA were measured, and a colour Doppler video clip of at least 3 seconds was stored in an image database.

US images/videos were evaluated (online supplemental file II) by the performing ultrasonographer and subsequently by a rheumatologist (APD) with long-standing experience in GCA US (10 years of experience with over 3000 vascular US examinations of the TA and the large supra-aortic vessels),[12] who made the final US diagnosis (gold standard). The ultrasonographers and APD were blinded to all clinical, laboratory and biopsy data. A positive sign for vasculitis in the TA branches and in the FA was defined as a hypoechoic and increased IMT (halo sign) and a positive compression sign. Since the TA and FA have widely variable diameters, a specific IMT as the cut-off for vasculitis was not defined in these vessels. A homogeneous intima-media complex increased thickness in the AA of >1 mm and ≥1.5 mm in the CA was defined as vasculitis.[12]

Ultrasonographic training

Five rheumatologists from Denmark with more than 8 years of experience in the use of musculoskeletal US (Doppler included) but with no previous experience in vascular US were trained at the ‘International Workshop on Ultrasound in Large Vessel Vasculitis & Polymyalgia Rheumatica’ held by APD and a group of international experts on vascular US.[14] The course contained 5 hours of theoretical training and 10 hours of hands-on US examination of 12 healthy persons and eight patients with GCA pathologies (both cranial and large vessels) under supervision. Four months later, the training programme was followed by a 2-day workshop organised by the investigators in Denmark, with additional training and standardisation of the scanning technique and optimisation of equipment settings supervised by APD. The workshop included 6 hours of supervised hands-on training in four healthy subjects and four patients with GCA with both cranial and large vessel pathologies, followed by 1 hour of image evaluation. From the beginning of the vascular US training (March 2013) to the initiation of the study (April 2014), the five ultrasonographers had performed ≥50 vascular US examinations, with a minimum of 50% scans in hot cases or patients with an established GCA diagnosis (figure 1). APD was available for feedback and supervision of images/videos during that period.

Figure 1

Ultrasound training programme.

Statistical methods

For the quantification of interobserver agreement and reliability, we used absolute agreement and (unweighted) kappa. Kappa coefficients were interpreted according to Landis and Koch, with κ values of 0.0–0.2 considered poor, 0.2–0.4 fair, 0.4–0.6 moderate, 0.6–0.8 good and 0.8–1 excellent.[15] All statistical tests were based on a comparison between the ‘gold standard’ (APD) and each of the five experienced Danish ultrasonographers. The statistical analyses were based on the observed data, and no attempt to perform imputations for missing data in the primary analyses was done, that is, it was assumed that missing data were missing completely at random. All the statistical analyses were performed using SAS V.9.4 (SAS Studio).

RESULTS

Patient characteristics

From three Danish centres, 112 patients suspected of GCA were included as presented in the flow diagram (figure 2). The 112 patients had a mean age of 72.4 years±SD 7.9 years; 66 (59%) were women. The mean duration of symptoms before referral to the hospital was 5.9 weeks±SD 4.43 weeks, with initial mean CRP levels of 69.4±SD 61.5 mg/L, and a mean treatment duration prior to US examination of 0.91 days±SD 1.55 days. The majority of patients (92%) reported a newly emerged localised headache and 43.8% experienced symptoms of polymyalgia rheumatica at the time of debut (table 1).

Figure 2

Study flow diagram.

Table 1

Patient characteristics

	N	Mean	SD
Continuous variables
Age, years	112	72.5	7.9
Symptom duration, weeks	108	5.95	4.43
Treatment duration, days	112	0.91	1.55
C reactive protein, mg/L	112	69.4	61.5
Dichotomous variables	N	No.	(%)
Women, no. (%)	112	66	58.9
PMR symptoms, no. (%)	112	49	43.7
Newly occurred localised headaches, no. (%)	112	93	83
US positive for GCA*, no. (%)	112	66	58.9
US positive for cGCA*, no. (%)	112	59	52.6
US positive for LV-GCA*, no. (%)	112	21	18.7
Halo sign TA*, no. (%)	112	57	50.8
Compression sign TA*, no. (%)	107	51	47.6
Halo sign FA*, no. (%)	112	23	20.5
Compression sign FA*, no. (%)	107	17	15.8
Halo sign AA*, no. (%)	112	20	17.8
Halo sign CA*, no. (%)	112	6	4.4

*Assessed by the ‘gold standard’ assessor (APD).

AA, axillary artery; CA, common carotid artery; cGCA, cranial GCA; FA, facial artery; GCA, giant cell arteritis; LV-GCA, large vessel GCA; PMR, polymyalgia; TA, temporal artery; US, ultrasound.

US findings

In total, 1344 arteries/branches were evaluated by US. According to the external US expert, vasculitis changes were observed in 66 patients, in 45 cases with only cranial involvement, in 14 with both cranial and large vessel involvement, while in seven patients isolated large vessel vasculitis was found (table 1). In the 54 patients with cGCA in whom the compression sign was performed, a positive halo sign was also observed. FA involvement was observed in 23 out of 59 patients with cGCA (table 1). In patients with large vessel GCA (LV-GCA) (n=21), the AA was more often affected (90%) (table 1), and in 14 out of 19 cases, the findings were bilateral, with an IMT range from 1.1 mm to 2.5 mm (mean 1.72 mm±SD 0.47). AC involvement was observed in six patients, with a mean IMT of 1.97 mm±SD 0.44, one of whom exhibited isolated CA involvement (IMT 2.1 mm bilateral) without vasculitis changes in the AA.

Overall reliability and agreement

The reliability between the local ultrasonographers and the US expert was excellent for the final diagnosis of GCA. Of the total number of patients (112 patients), agreement on the final diagnosis was found in 108 patients. In 43 of these patients, the findings were normal, and in the remaining 65 patients, pathological findings were found. The mean agreement was 96%, mean kappa 0.92 (95% CI 0.85 to 099). Furthermore, the agreement was also excellent for the evaluation of cGCA and LV-GCA, kappa 0.89 (95% CI 0.81 to 0.98) in 95% and kappa 0.89 (95% CI 0.78 to 0.99) in 96% (table 2) Analysis of the reliability at the vessel level was also performed, evaluating the agreement between the US expert and the local ultrasonographer for the halo sign and the compression sign in the TA/FA and for halo sign in the AA/AC. The reliability was excellent in all vessels for both the halo sign and the compression sign, with an agreement between 94% and 100% and k coefficients from 0.86 to 1.0 (table 2).

Table 2

Overall inter-rater reliability and agreement

Variables	TotalN	Agreementanalogy*N	Interobserver agreement* (%)	Interobserver reliability*, kappa coefficient	95% confidence limits
Primary outcome
US positive for GCA	112	No-No=43Yes-Yes=65	96%	0.92	0.85–0.99
Key secondary outcomes
US positive for cGCA	112	No-No=49Yes-Yes=57	95%	0.89	0.81–0.98
US positive for LV-GCA	112	No-No=88Yes-Yes =20	96%	0.89	0.78–0.99
Other secondary outcomes
Halo sign temporal arteries, all segments	112	No-No=52Yes-Yes=55	96%	0.91	0.83–0.99
Compression sign temporal arteries, all segments	107	No-No=52Yes-Yes=49	94%	0.89	0.80–0.98
Halo sign facial arteries, all segments	112	No-No=86Yes-Yes=21	96%	0.87	0.75–0.98
Compression sign facial arteries, all segments	107	No-No=88Yes-Yes=15	96%	0.86	0.73–0.99
Hallo sign axillary arteries	112	No-No=90Yes-Yes=19	97%	0.91	0.81–1.00
Hallo sign common carotid artery	112	No-No=106Yes-Yes=6	100%	1.00	1.00–1.00

*Between the US expert and the ultrasonographer.

cGCA, cranial GCA; GCA, giant cell arteritis; LV-GCA, large vessel GCA; US, ultrasound.

The agreement for the overall diagnosis of GCA, cGCA and LV-GCA for each centre was excellent, with a mean agreement of 96%, 93% and 94%, respectively. The distribution of the US examinations varied between the performing ultrasonographers. Two of the operators performed under 10 US scanning (2 and 8), while the others evaluated 17, 22 and 63 patients. An additional analysis on the agreement between the US expert and the single ultrasonographer was performed for both the final diagnosis and on the vessel level, with excellent agreement (range from 88% to 100%). Furthermore, no differences regarding agreement were observed at the beginning of the study (first five scans of the four sonographers) comparing with the rest of the study.

DISCUSSION

This is the second multicentre study evaluating the impact of a standardised training programme and the only study to date where an extended training was used for musculoskeletal experienced ultrasonographers without previous experience in vascular US for obtaining the diagnosis GCA and for agreeing on vascular pathology. We found excellent US reliability for both the overall GCA diagnoses, but also on a vessel level using both the halo and the compression signs as elementary US vasculitis lesions.

Despite the growing body of evidence supporting the utility of US in GCA, standardised training programmes and their impact on reliability are lacking.[3] Previously, one study evaluated a standardised training programme and reliability exercise for rheumatologists without previous experience with vascular US[6] and found excellent interobserver agreements in line with our study. However, the impact of a training programme was only evaluated in preselected static images and not in patients in routine care with suspected GCA. In the TABUL study,[8] several of the participating ultrasonographers were not familiar with the vascular US and underwent a limited training programme, including a test on the individual competence in interpreting vascular US videos and performing vascular US examinations of 10 healthy persons as well as of one patient with active GCA. Previous US experience was not a prerequisite for participation. Only 53% of the participants passed the examination, only a few (16%) of whom passed the examination in the first attempt. In the TABUL study, the interobserver agreement based on blinded posthoc image analysis by 12 different sonographers was only moderate, illustrating the challenges presented in the education for the vascular US in GCA.

A study from 2012[15] demonstrated that the sensitivity of US in GCA decreases within the first days of steroid treatment, due to the effect of the treatment on the inflamed vessel wall swelling, with decreasing/disappearance of the halo sign. A recent study from the Outcome Measures in Rheumatology US subgroup on large vessel vasculitis evaluating the reliability of US in patients was performed in two steps.[9] The first step was an exercise in patients who turned out to have very little pathology due to long prednisolone treatment and long-standing disease, resulting in inconclusive results. Furthermore, the participants were not familiar with the US equipment used in the exercise. Subsequently, a patient-based exercise was carried out in patients with early disease and shorter duration of prednisolone treatment, with participants having more training and time with the US equipment and its settings. In the first exercise, inter-reader reliabilities were fair to moderate (light k 0.29–0.51), while in the main exercise, the inter-reader reliability was increased significantly (good to excellent, light k 0.76–0.86) for the overall diagnosis of GCA. In our study, the treatment duration of the patients before the US examination was very short (mean 0.91±SD 1.55 days), which may have facilitated the high reliability in vessel pathology in our study. Furthermore, the participants were well acquainted with the US equipment and settings.

In our study, we used halo and compression signs as indicators of vessel wall swelling. The reliability of the compression sign has been tested in one clinical study to date.[9] In this study, a reliability evaluation of the compression sign in US of the temporal arteries was performed, comparing the results of an experienced vascular specialist with those of a rheumatologist not familiar with vascular US, who had undergone a short training course with five supervised compression US examinations.[9] A high interobserver agreement was found regarding the compression sign in US of temporal arteries (98% agreement), in line with our results. However, the study did not provide information about the training protocol. We are also the first to present results on the reliability of the compression sign for the FA, which is as high as for the TA (table 2)

One limitation of our study is that one single ultrasonographer contributed to approximately half of the examinations (63 out of 112); however, no differences were observed regarding agreement on the ultrasonographer level. Another limitation is the lack of a real control group in the assessment of the impact of the training programme or a comparison of reliabilities before and after initiation of the training programme. In the TABUL study, the agreement with GCA findings was low between the performing ultrasonographer and the expert reviewer of images. In 47 out of 162 cases, the expert did not find any GCA changes on the images and videos evaluated as pathological by the performing ultrasonographer.[8] Using the TABUL study as a substitute for a control group, our study indicates that a more extended and structured training programme improves the reliability of vascular US examinations. Furthermore, no changes with regard to agreements were observed at the ultrasonographer level in the first five scans in the study compared with the rest of study, indicating that our training programme was sufficient, with no additional improvement being seen during the study.

CONCLUSION

The training programme together with previous musculoskeletal US experience resulted in excellent US reliability in patients suspected of GCA, both for the overall diagnosis and at the vessel level. The training programme may be used for the implementation of the vascular US in clinical practice.

Vascular ultrasound (US) is increasingly used for confirming or excluding a suspected diagnosis of giant cell arteritis (GCA) in clinical practice.

Vascular US is regarded as strongly operator-dependent, and US reliability for the diagnosis of giant cell arteritis may be improved by specific training.

This is the first multicentre study evaluating the impact of a standardised training programme for ultrasonographers without previous experience in vascular US.

Using a dedicated training programme with theoretical lectures and practical US examination of both healthy controls and patients with GCA pathology under supervision, a high level of reliability was achieved regarding vessel pathology and patient diagnosis.

Our training programme may be used for the implementation of vascular US in clinical practice.