Agreement Between International Radiologists on the Appropriateness and Urgency in Lumbar Spine MRI Referrals.

Purpose: To determine how radiologists across health-care jurisdictions internationally assess the appropriateness and urgency levels of lumbar spine Magnetic Resonance Imaging MRI referrals. Patients and
Methods: Clinical information was extracted from 203 lumbar spine MRI referrals. Texts were divided into 10 datasets and embedded into a software to facilitate the classification process. Participant radiologists were recruited at the Image Perception Lab, at the Radiological Society of North America Congress, 2019 and through the institution radiology network. Radiologists were asked if they use referral guidelines in their practices. Radiologists assigned appropriateness and urgency levels based on the referral text. Appropriateness level descriptors were: indicated, indicated but needs more information or not indicated. Urgency levels were categorized: urgent, semi-urgent, or not urgent. All cases containing neurological symptoms with/without red flags were extracted and exact agreement between radiologists' responses on the indication status was calculated.
Results: Seventy radiologists from 25 countries participated; 42% of participants indicated non-use of referral guidelines. Poor-moderate radiology agreements were recorded for appropriateness and referral urgency level decisions. 79.6% of responses indicated that cases containing neurological symptoms with/without red flags were indicated for scanning.
Conclusion: Despite referral guidelines promotion, nearly half of participants stated non-usage. Subsequently, a varied agreement levels were found in assigning the appropriateness of the referrals. Appropriateness of referrals with neurological symptoms (with/without red flags) recorded good agreement.

Introduction

Referral vetting is a fundamental principle for radiation protection and patient safety in radiology departments.1 It involves scrutiny of the referred requests by radiologists or nominated specialist imaging radiographers to ensure that the written indications for scanning are adherent to guidelines adopted by the department – often international guidelines, such as the American College of Radiology Appropriateness Criteria (ACR),2 the Royal College of Radiologists guidelines (RCR),3 or to national and local guidelines. Although these guidelines have been shown to minimize inappropriate scanning,4–7 some guidelines have a considerable level of ambiguity which causes variance in interpretation of guidelines.1,8 Several studies have reported various levels of agreement between radiologists or radiographers on the appropriateness for lumbar spine MRI. For example, a study in Spain9 reported substantial agreement (Kappa = 0.62) between radiologists in assigning the appropriateness of lumbar spine MRI referrals according to the ACR criteria.2 In Ireland,10 a fair agreement with kappa = 0.26 was found between three experienced MR radiographers who were recruited to assign the appropriateness of 1021 lumbar spine MRI referrals based on RCR guidelines (iRefer).3

Radiologists, or in a limited number of European countries MRI specialist radiographers, are responsible for vetting MRI requests to ensure referred requests’ appropriateness. Lumbar spine MRI examinations (LSMRI) referrals are of particular interest as the demand for LSMRI is rapidly growing, and inappropriate management of patient referrals can impact waiting lists and the appropriate prioritization of patients.11 In addition to avoidable delays in diagnosis and treatment for patients, alternative examinations might be required for those inappropriately referred for LSMRI, resulting in increased workloads, health-care costs, and low health-care outcomes.12 In contrast, appropriate referral vetting helps reduce the unjustified use of diagnostic imaging: improves healthcare quality, patient safety, and reduces cost and resource utility.13

However, the literature demonstrates variations between radiologists and radiographers from the same institution or country in assigning the referrals’ appropriateness, and there remains a paucity of literature related to how radiologists with different levels of experiences across different health-care jurisdictions internationally assess the appropriateness of lumbar spine MRI referrals.

This paper aims to investigate the extent of variation between radiologists from different countries in assigning the appropriateness and urgency level of a bank of 203 clinical LSMRI referral cases. The study also aims to identify the extent of variation in radiologists’ categorization of the appropriateness of the cases with neurological symptoms with/without red flags as stated in the ACR guidelines.2 This paper is novel in its approach as it is the first paper to investigate the variation in opinions at international level which could give insight into how international radiologists interpret the written indications and what factors influence their judges and agreements.

Materials and Methods

Referral Cases

Ethical approval was granted by the institutional ethics committee of University College Dublin and formal confirmation of the participating radiology department was attained as part of the ethics process. To develop a text referral case bank, a public hospital, which is an affiliated university teaching hospital, agreed to participate and facilitate the use of referral texts to be collated for use in the study. Clinical indications and patients’ demographics as well as referring departments were retrospectively extracted from 203 LSMRI referrals in the form of “referral texts” in an Excel sheet. Information pertaining to the local service and patients’ identities was edited to ensure referrals were anonymized. Information pertaining to clinical indications was not edited.

Referral Text

The extracted text data was randomly divided into ten datasets, each containing 20–21 referral texts. The number of the datasets was determined to be ten with 20–21 cases each to minimize time commitment from participants. The presents an example of dataset number 4. The datasets were inserted into a password-protected, web-based user interface (Ziltron Ltd., Dublin, Ireland). The data collection tool was accessed via four 4th generation Apple iPad tablet computers running IOS10.3.3 (Cupertino, CA, USA). The resultant data were saved in real-time in secure cloud-based storage.

Participating Radiologists

Radiologists from different countries and experience levels volunteered to assess the appropriateness of the collected LSMRIs referral texts as well as the examination urgency level at the Medical Image Perception Lab in the Radiology Society of North America conference RSNA 2019, which was funded by National Cancer Institute (NIC). Prior to commencing the research activity, the aim of the study was explained and verbal consent to proceed was obtained from all participants. The participants agreed to the use of their classifications and their information in the research, such as years of experience, country, and involvement in referrals vetting as part of the consent process. Advertisement of the Image Perception Lab research activity was approved by the RSNA scientific committee, and the research activity was promoted at the conference venue.

Each participant was assigned to classify one of the 10 datasets (20–21 cases) and was requested to provide the following demographic details: country of practice, number of years of experience post radiology qualification, and number of reported LSMRI cases per month. Participants were also asked if guidelines are used in their practices or not. Resident radiologists whilst having experience of MRI referral vetting were categorized as zero years to facilitate recognition of their early-stage professional status within the radiology participant cohorts. The participant then proceeded to view individual text referrals and used a drop-down menu to select whether the referral was indicated, indicated as routine but subject to additional information, or not indicated. If the referral was indicated, a further drop-down menu asked the participants to indicate the urgency level of the case, categorized as: urgent: scan within 48 hours, semi-urgent: scan within four weeks, or routine: scan can wait more than four weeks. The time intervals were chosen arbitrary based on the current practice in the hospital from where the data was extracted.

After the conference finished, the number of participants who had assessed each dataset was not equal, which caused some difficulties in testing the agreement between the radiologists in the whole stacked datasets as one score. Another recruitment round took place through the institution radiology network. We contacted 9 MSK radiologists directly, all of whom have not been involved in the previous texts review. Participants were given access to the Ziltron software in which the datasets were stored and directed to the number of the dataset that they are required to complete.

Analysis

All data analysis tests were performed using SPSS (version 26). Different inter-rater agreement tests were applied to calculate the agreement in different sets as follows:

Intra-class correlation coefficient ICC based on a mean-rating (k = 7), absolute agreement, and two-way random effects models were used to determine the agreement between participants in each dataset for both appropriateness and urgency level. Also, ICC with one-way random effects models was applied for the whole stacked referrals (n = 203) to measure the agreement between all radiologists in the appropriateness and urgency level of referrals. Levels of agreement were in accordance with the suggested levels by Koo and Li in which ICC less than 0.5 indicates poor agreement, 0.5–0.75 moderate, 0.75–0.9 good, and excellent if over 0.9.14

Agreement between experienced participants (board-certified participants) from the same countries in each dataset was calculated using ICC with the absolute agreement and two-way random-effects models. The reason is to investigate if experienced radiologists from the same countries and classified the same dataset have a higher agreement than agreement found between all participants in the same dataset or not.

Referrals containing neurological symptoms, and neurological symptoms with red flags were extracted, 17 referrals with both neurological symptoms and red flags (Set A), and 23 referrals with neurological symptom only (Set B). The percentage agreement was calculated for both Set A and B manually by dividing the number of exact agreements in observations (eg, indicated) by the total number of observations (all ratings including indicated, indicated but needs more information, and not indicated). The reason why the percentage agreement was selected is to separately calculate the agreement on each class (eg, indicated). Referrals in Set A were also categorized under ACR red flags categories to calculate percentages of exact agreement on class “indicated” and urgency level for each category.2

Results

A total of seventy radiologists (n = 70) were recruited for assigning referrals appropriateness and urgency level. Of those, 61 were recruited in the Medical Image Perception Lab and the remaining 9 were recruited after the conference. Participants originated from 25 countries, the majority originated from the United States, UK, Mexico, and Saudi Arabia. Figure 1 shows the number of participants from each country. Overall, 42% of the participants (n = 30) indicated no use of referral guidelines in their daily practice (Table 1). Agreement between radiologists in assigning the referral appropriateness varied: a poor agreement was found in 3 datasets, moderate in 6 datasets and only one dataset recorded good agreement (ICC in the ten datasets = −0.019 to 0.834). Agreement on referrals appropriateness for the whole datasets together (n = 203) was moderate with ICC = 0.621 and 95% CI [0.53–0.69]. Poor agreement was found in 6 datasets, and 4 datasets recorded moderate agreement (ICC = −0.410 to 0.699) for the urgency of referrals. Also, poor agreement was found between the radiologists in assigning the urgency level of the whole referrals (ICC = 0.464 with 95% CI [0.34–0.57]). Table 2 shows the agreement on the appropriateness and urgency level of LSMRIs in each data set and the whole stacked sets together, and Table 1 provides the participants’ demographics and the data set(s) completed.

Figure 1

A histogram showing the number of participants and their country of practice.

Table 1

Demographics of Radiologists Participated in Datasets 1–10

Radiologist Demographics
Set	Country of Work	Years Qualified	Use Guidelines Yes/No	Vet MRI Referrals YES/NO	Set	Country of Work	Years Qualified	Use Guidelines Yes/No	Vet MRI Referrals Yes/No
1	France	5	Yes	—	6	USA	4	No	—
	Taiwan	6	No	Yes		Australia	25	No	Yes
	Australia	5	No	Yes		USA	16	No	No
	France	7	Yes	Yes		USA	0	No	Yes
	Brazil	4	No	No		USA	15	No	No
	New Zealand	34	Yes	Yes		Germany	4	Yes	Yes
	Saudi	17	Yes	Yes		USA	0	No	Yes
2	Canada	20	Yes	Yes	7	Japan	22	No	Yes
	Mexico	0	No	Yes		Mexico	3	No	Yes
	USA	0	No	Yes		Singapore	1	No	Yes
	El Salvador	15	No	—		USA	0	Yes	Yes
	USA	0	Yes	Yes		France	6	Yes	Yes
	USA	0	No	Yes		USA	0	Yes	Yes
	Netherlands	26	No	Yes		Saudi	3	Yes	Yes
3	Canada	16	Yes	Yes	8	USA	2	Yes	Yes
	USA	0	Yes	–		Mexico	3	Yes	–
	Nigeria	13	Yes	No		UK	1	Yes	Yes
	Nigeria	14	Yes	–		Pakistan	8	Yes	Yes
	France	30	No	–		Saudi	3	Yes	Yes
	Mexico	1	No	–		Ireland	6	Yes	Yes
	Mexico	0	No	Yes		Egypt	15	Yes	Yes
4	Qatar	16	No	No	9	UK	30	Yes	Yes
	Brazil	2	Yes	No		Italy	4	Yes	Yes
	Brazil	2	Yes	Yes		USA	10	Yes	No
	Argentina	2	No	–		Argentina	22	Yes	Yes
	USA	0	Yes	Yes		Turkey	0	No	No
	USA	3	No	Yes		USA	0	No	No
	Saudi	17	Yes	Yes		Saudi	9	Yes	Yes
5	USA	15	Yes	–	10	UK	10	Yes	Yes
	USA	10	Yes	Yes		UK	10	Yes	Yes
	Egypt	1	No	Yes		USA	0	Yes	Yes
	Japan	20	Yes	–		India	8	No	Yes
	UK	14	No	Yes		UK	4	Yes	No
	Bahrain	2	No	Yes		USA	0	Yes	Yes
	Argentina	15	No	No		Ireland	6	Yes	Yes

Table 2

Inter-Rater Agreement (ICC) on the Appropriateness and Urgency Level of LSMRI Cases

Demographics				Inter-Rater Agreement (ICC) of Appropriateness of LSMRI Cases				Inter-Rater Agreement (ICC) of Urgency of LSMRI Cases
No. Set	No. Cases	No. Radiologists	No. Radiologists Not Using Guidelines	ICC	95% CI	P	Level of Agreement	ICC	95% CI	P	Level of Agreement
1	20	7	3	0.025	[−0.67, 0.53]	0.435	Poor	−0.410	[−1.24, 0.28]	0.853	Poor
2	20	7	5	0.690	[0.44, 0.85]	<0.001	Moderate	0.265	[−0.24, 0.64]	0.132	Poor
3	20	7	3	−0.019	[−0.79, 0.52]	0.490	Poor	0.184	[−0.36, 0.60]	0.221	Poor
4	20	7	3	0.834	[0.69, 0.92]	<0.001	Good	0.671	[0.41, 0.84]	<0.001	Moderate
5	20	7	4	0.482	[0.12, 0.75]	0.005	Poor	0.396	[−0.04, 0.71]	0.038	Poor
6	20	7	6	0.536	[0.20, 0.77]	0.002	Moderate	0.474	[0.04, 0.75]	0.017	Poor
7	21	7	3	0.699	[0.46, 0.85]	<0.001	Moderate	0.615	[0.33, 0.81]	<0.001	Moderate
8	21	7	0	0.747	[0.54, 0.88]	<0.001	Moderate	0.562	[0.22, 0.79]	0.002	Moderate
9	20	7	2	0.578	[0.27, 0.79]	<0.001	Moderate	0.359	[−0.09, 0.69]	0.055	Poor
10	21	7	1	0.688	[0.42, 0.85]	<0.001	Moderate	0.699	[0.46, 0.85]	<0.001	Moderate
All	203	70	30	0.621	[0.53, 0.69]	<0.001	Moderate	0.464	[0.34, 0.57]	<0.001	Poor

Abbreviations: ICC, interclass correlation coefficient; CI, confidence interval; P, p-value.

The agreement levels were higher between experienced radiologists from the same countries than agreements recorded for all experienced and non-experienced radiologists from different countries. Most of the agreements recorded moderate agreement with ICCs ranged from 0.547 to 0.628, and good agreement with ICC = 0.776 was recorded between pair of experienced participants from France. Table 3 shows the agreement results and participants’ nationalities in the datasets.

Table 3

Agreement Between Experienced Radiologists from the Same Countries

No. Set	Country	No. of Participants	Years of Experience	ICC of Appropriateness of LSMRI Cases	95% CI	P	Agreement Level
1	France	2	57	0.776	[0.44, 0.91]	<0.001	Good
4	Brazil	2	22	0.621	[0.01, 0.85]	0.023	Moderate
5	USA	2	1510	0.558	[−0.12, 0.82]	0.008	Moderate
6	USA	3	41615	0.547	[0.05, 0.80]	0.019	Moderate
10	UK	3	41010	0.628	[0.22, 0.83]	0.004	Moderate

Abbreviations: ICC, interclass correlation coefficient; CI, confidence interval; P, p-value.

Further analysis of Set A and Set B together (cases with neurological symptoms with/without red flags) recorded a percentage agreement of 79.6% for participant responses for indicated cases, 8.2% agreed referrals as indicated but need more information”, and 12.1% agreed that the referrals were not indicated despite cases of neurological symptoms with and without red flags being present. For referrals with red flags, Set A (n = 17), we found 91.5% of the participants agreed that the cases are indicated. A percent of 45.3% of the participants agreed that cases with red flags are urgent and need to be scanned within 48 hours. Referrals included key red flags that conformed with those presented by the ACR guidelines are categorized and displayed in Table 4.

Table 4

ACR Categories and Percent of Agreement on Indicated Judges and Urgency Levels for Referrals Included Neurological Symptoms and Red Flags

ACR Red Flags Categories	No. Referrals	No. Responses	Percentage (%) Deemed “Indicated”	Exact Agreement on Urgency Level
				Urgent Within 48 Hours	Semi-Urgent Within 4 weeks	Routine Scan (> Than a Month)	Not Indicated
Acute onset of urinary retention or overflow incontinence	5	35	100	71.4%	20%	11.4%	0%
Global or progressive motor weakness in the lower limbs	7	49	85.7	22.4%	42.8%	26.5%	8.1%
Back pain not resolved with conservative management	1	7	100	0%	100%	0%	0%
History of cancer	2	14	78.5	35.7%	28.5%	21.4%	14.2%
Loss of anal sphincter tone or fecal incontinence	2	14	100	92.8%	7.1%	0%	0%
Total	17	119	91.5	45.3%	33.6%	15.9%	5%

Discussion

This study investigated the variation amongst international radiologists in assigning the appropriateness and urgency level of LSMRI referrals. Varied agreement levels were found in assigning the appropriateness and urgency level of the referrals within the datasets. Moderate agreement on the referrals appropriateness on whole stacked datasets was found with ICC = 0.621 (95% CI [0.53, 0.69]) and poor for the urgency level with ICC = 0.464 (95% CI [0.34, 0.57]). We found good consensus between radiologists (79.6% of the responses were indicated) on the appropriateness of the 40 cases (Set A and B) that included neurological symptoms with and without red flags. However, a higher agreement with greater than 91% was recorded for the indication of cases with red flags in Set A (Table 4).

The overall moderate agreement of the referrals’ appropriateness for all datasets is similar to the agreement score reported in the study of Francisco et al 9 however, this study was conducted on a national scale (radiologists were from one country), whilst our study included radiologists from different countries. For the agreements within the datasets, most of the ten datasets recorded moderate or poor agreement in assigning the appropriateness of the LSMRI referrals, and only one set recorded good agreement. We speculate that this moderate-poor agreement (high-moderate variation) in radiologists’ opinions could be attributed to the differences in experience and countries between participants in each dataset (Table 1), the service that they come from (public or private), and their involvement in vetting MRI referrals.8,15 Moreover, we found 42% of the participants stated they did not use referral guidelines in their workplaces which also in line with previous studies16–20 and this lack of adherence to the guidelines might also influence the extent of variation among the radiologists’ decisions. Causes of non-adherence to the guidelines were not investigated as the study was not designed for this purpose; however, we might speculate some common reasons such as lack of awareness, lack of guidelines availability in workplaces.19 Another reason is that some radiologists rely on their clinical experience when they vet referrals without referring to the guidelines.10

To investigate the influence of factors such as experience and country of practice on the agreement variability found within the datasets, we measured the agreement between only experienced radiologists from the same countries as presented in Table 3. We found that all agreement levels improved to moderate and good agreement level after excluding the non-experienced participants and participants from different countries. This finding supports our speculation regarding the influence of difference in experience and counties of practice between participants on the agreements found within the datasets.

Regarding the good agreement between all participants (experienced and non-experienced) in the set 4, we identified that sentences in this set contained phrases clearly indicating the scans’ appropriateness. Randomization was applied using an Excel program; however, despite this process, dataset 4 contained texts that were more likely to achieve higher agreement than other data sets. For instance, referrals included phrases such as “normal exam”, “non-specific low back pain” which were classified as not indicated by most of the participants and other referrals with phrases indicating serious illness such as “for intervention”, “stenosis on previous MRI”, “reduced saddle sensation”, “urinary incontinence”, “progressively getting worse”, and “vertebral fractures” were categorized as indicated by all radiologists.

Good agreement was found on the appropriateness of referrals which contained clinical indications with neurological symptoms with and without red flag signs (79.6% of the responses agreed that these cases are indicated). Most of the disagreement in this dataset came from the cases with neurological symptoms but without red flags where there was a range of opinions regarding indication status. For example, “note degenerative lumbar spine on x-ray with positive nerve tension signs bilaterally, ? Lumbar spine foraminal stenosis” - this referral was classified as not indicated by 2 radiologists, indicated but needs additional information by 2 radiologists, and indicated by 3 radiologists. The lack of information provided in some referrals is also another cause of disagreement (16 out of 40 referrals), in which some radiologists agreed on the need for scanning, but they required more information relating to symptoms history, physiotherapy, side and location of pain, and imaging protocol. However, in all cases with red flags, such as urinary incontinence, bowel incontinence, back pain not resolved with conservative management, we found a consensus with 100% of these referrals deemed appropriate for scanning. Whereas referrals falling under other red flag categories such as “global or progressive motor weakness in lower limbs” and “history of cancer” recorded less consensus on that these referrals are indicated with percentages 85.7% and 78.5%, respectively.

Investigating how variant radiologists are in assigning the urgency level of the referrals has also resulted in poor agreement (high variation) in 6 datasets and moderate agreement in 4 datasets. Overall agreement for the whole urgency classifications of the 203 cases also showed poor agreement. The literature review revealed that no studies had been conducted to measure the variability between radiologists or specialist MRI radiographers in assigning the urgency level of the MRI or any other radiology referrals. However, it is possible that this variation is because there are currently no guidelines or clear definitions for urgency examination timings, and radiologists might be influenced by the difference in variables in their practices, such as the demand level on MRI scanning. For instance, in some non-acute hospital departments, urgent, the least urgent cases could be scanned within 48 hours, whilst in a department with high pressure, least urgent cases might take more than 4 weeks.21 In datasets with the moderate agreement (4-7-8-10), we noticed the abundance of negation phrases written within referrals which might lead most of the participants to select routine scan time, commenting included: “no red flags”, “no neuro”, “nil neurology”, “normal neuro”, “no motor features”, “no focal deficit”, and other instructions related to the time of the scan such as “for annual surveillance”, “repeat up to date MRI”, “interval MRI”, “to be done in 3 months”, “OPD in 4 months”. These datasets also contained referrals with the presence of some words that indicate the necessity of urgent examination, such as “vertebra fracture”, “CSF leak”, “reduced saddle sensation” and “urinary incontinence”, in which most of the participants selected urgent scan time.

With regard to the urgency level of the cases that included red flags, we found good agreement between participants on the urgency level of cases with new onset of urinary and fecal symptoms in the context of LBP (Table 4), in which most of the responses suggested urgent scan within 48 hours (71.4% agreement on cases with urinary incontinence and 92.8% agreement on cases with fecal incontinence). This finding is in accordance with recommendations reported by Bell, Collie and Statham in which the authors advise urgent scanning for patients present with both LBP and urinary symptoms.22 Other categories did show variant agreement between participants with no apparent trend, except that there was complete agreement between 7 radiologists that the case with “Back pain not resolved with conservative management” can be considered semi-urgent and wait up to 4 weeks. In general, most of the variations in determining urgency level were found within cases stating LBP with “history of cancer” or “global or progressive motor weakness in the lower limbs”.

This study has some limitations. First, only a few radiologists provided information regarding their opinions when selecting “indicated but subjected to additional information”, which would have assisted in understanding what information they deemed missing. Second, the causes of non-adherence to the guidelines were not investigated as we did not ask the reasons if the participant was not compliant. Third, it is acknowledged a greater proportion of radiologists originating from USA participated, this is to be expected at the RSNA venue; however, every effort was made to recruit participants from different countries of practice.

Conclusion

To conclude, we generally report poor to moderate agreement between radiologists from different counties and varying experiences in assigning the appropriateness and urgency level of LSMRI referrals. Overall moderate agreement on referrals appropriateness was recorded for the whole datasets together. Improved agreement scores were noticed between experienced radiologists from the same countries. High appropriateness agreement was found on referrals with red flag indications. These results and their interpretations highlight differences in radiologists’ opinions particularly across different healthcare environment and indicate the potential for improvement in the use of referral guidelines. Further research is warranted to investigate what are the causes of the variety between radiologists in assigning the appropriateness and urgency level of LSMRI and reasons for non-compliance with the guidelines and the subsequent potential impact on imaging resources and patient management.