An audit tool for assessing the appropriateness of carotid endarterectomy.

BACKGROUND: To update appropriateness ratings for carotid endarterectomy using the best clinical evidence and to develop a tool to audit the procedure's use.
METHODS: A nine-member expert panel drawn from all the Canadian Specialist societies that are involved in the care of patients with carotid artery disease, used the RAND Appropriateness Methodology to rate scenarios where carotid endarterectomy may be performed. A 9-point rating scale was used that permits the categorization of the use of carotid endarterectomy as appropriate, uncertain, or inappropriate. A descriptive analysis was undertaken of the final results of the panel meeting. A database and code were then developed to rate all carotid endarterectomies performed in a Western Canadian Health region from 1997 to 2001.
RESULTS: All scenarios for severe symptomatic stenosis (70-99%) were determined to be appropriate. The ratings for moderate symptomatic stenosis (50-69%) ranged from appropriate to inappropriate. It was never considered appropriate to perform endarterectomy for mild stenosis (0-49%) or for chronic occlusions. Endarterectomy for asymptomatic carotid disease was thought to be of uncertain benefit at best. The majority of indications for the combination of endarterectomy either prior to, or at time of coronary artery bypass grafting were inappropriate. The audit tool classified 98.0% of all cases.
CONCLUSIONS: These expert panel ratings, based on the best evidence currently available, provide a comprehensive and updated guide to appropriate use of carotid endarterectomy. The resulting audit tool can be downloaded by readers from the Internet and immediately used for hospital audits of carotid endarterectomy appropriateness.

Background

Evidence for efficacy of an intervention may be clearly established, but its expected effectiveness outside of a clinical trial setting, the values of decision makers, and resources available at a given point in time may all affect how that intervention is brought into generalized clinical care [1]. These may vary across different countries and decision-makers and this may lead to different interpretations of the evidence [2-5]. Hence different criteria, guidelines and appropriateness ratings often need to be developed in different countries, which may then guide the clinical audit of interventions and outcomes.

The first appropriateness study on the use of carotid endarterectomy (CE), prompted by concerns of its unnecessary use, suggested that only a third of CE procedures were appropriate [6]. It was published in 1988, prior to the large randomized controlled trials [7-10] that provide our current evidence base for the majority of indications for this procedure. In spite of the proliferation of the highest level of evidence available to guide patient management, controversy still exists as to what constitutes appropriate use of the procedure, especially in those with asymptomatic disease [11,12]. This may explain some of the marked variation in the rate of usage of CE between countries and regions [13,14].

The objectives of this paper were: firstly, to describe the development of contemporary appropriateness ratings for carotid endarterectomy according to a panel of Canadian experts; and secondly, to develop an audit tool for chart review based on these ratings. The ratings generated from this process are likely to be of particular relevance to health systems that resemble Canada's single payer system, and the tolls that we provide will facilitate the conduct of CE appropriateness audits as a means of ensuring optimal utilization of this procedure.

Methods

Appropriateness methodology

The RAND/ UCLA Appropriateness methodology has been described in depth elsewhere [1,2,15-18]. In brief, a comprehensive literature review is conducted collecting all available evidence for a given diagnostic or therapeutic intervention. A set of scenarios is developed covering all possible clinical situations under which that intervention might be performed. A panel of experts is asked to independently rate each scenario guided by the evidence presented. The rating scale ranges from 1 to 9; 1 to 3 indicating an inappropriate indication, 4 to 6 an uncertain indication, and 7 to 9 an appropriate indication. (For all definitions see Appendix) These initial ratings are collated. The panel is then convened and the participants provided with their initial rating and the median rating of the group for each scenario. Under the guidance of a chair, the group discusses each scenario with the relevant evidence and individually re-rates each scenario. The median of these final ratings is used to determine the appropriateness of each scenario according to the schema as presented above. Given that none of the scenarios, provoked disagreement among the panellists according to the RAND definition this played no role in the allocation of appropriateness.

The Medline database was used as the primary route for searching for relevant material for the literature review for this study. Evidence was rated as described elsewhere [19]. Where evidence was available for a given indication from randomized controlled trials, no further literature was presented. If this level of evidence was unavailable, literature from the next available evidence hierarchy was presented.

The nine panel members were selected from nominations made by presidents of the various Canadian national societies representing stroke neurology (two members), neurosurgery (two), vascular surgery (two), internal medicine (one), family medicine (one) and neuroradiology (one). The consensus conference proceedings were taped and transcribed.

Analyses

A descriptive analysis of the final appropriateness ratings of both symptomatic and asymptomatic stenoses by degree of internal carotid artery stenosis was performed. For all scenarios in this analysis, conventional angiography was the imaging modality assumed to have determined the degree of stenosis. The transcribed discussions of the panel were used to help to identify the 'key elements of reasoning' behind the panel's decisions. These 'key elements of reasoning' are presented in narrative form in the text of the Results section to accompany the quantitative reporting on panel ratings for the clinical scenarios.

A Microsoft Access® Database was developed to enable easy collection of the necessary clinical data to determine appropriateness from patient charts. Data can be entered directly into this database. In parallel, we developed an algorithm written in SAS version 8 for transforming the raw data gathered from chart review to final appropriateness ratings. We make both the Microsoft Access® Database and the SAS code available for download from the Internet [20].

A reviewer used the database to collect the clinical data by chart review on all carotid endarterectomies performed in the Calgary Health Region in the period of 1997 – 2001, a total of 801 cases. The data collected were then run with the SAS code and its ability to classify the cases tested.

Results and Discussion

Symptomatic carotid stenosis

The final appropriateness ratings for symptomatic internal carotid artery stenoses are summarised in Table 1. The ratings for severe (70 – 99%) symptomatic stenosis are the highest of any group. The panel felt that a majority of patients will benefit from treatment, because the magnitude of benefit conferred by surgery over medical treatment [2] has been shown to be so great that the procedure is likely to be of benefit even when performed by surgeons with a higher complication rate than in the trials.

Table 1

Final appropriateness ratings for carotid endarterectomy for symptomatic stenosis by degree of stenosis.

Degree of Stenosis	Indication	Comments	Appropriateness Rating
Severe (70 – 99%)	Hemispheric TIA	Any indication, surgery performed at any time up to 2 years	Appropriate
	Retinal TIA	Any indication, surgery performed at any time up to 2 years	Appropriate
	Mild Stroke	Any indication, surgery performed at any time up to 2 years	Appropriate
	Moderate Stroke	Any indication, surgery performed at any time up to 2 years	Appropriate
	Stroke in Evolution		Uncertain
Moderate (50 – 69%)	Hemispheric TIA	Single, multiple, crescendo events within the last 180 days with contralateral stenosis +/- ulcerated stenosis	Appropriate
		Single event with contralateral occlusion or event 6–24 months ago	Uncertain
	Retinal TIA	Multiple events, single event within 180 days with contralateral stenosis	Appropriate
		All other indications	Uncertain
	Mild Stroke	Any indication, surgery performed at any time up to 2 years	Appropriate
	Moderate Stroke	Any indication >3 weeks following event, <3 weeks only in presence of contralateral stenosis	Appropriate
		All other indications	Uncertain
	Stroke in Evolution		Inappropriate
Mild (0 – 49%)	Hemispheric TIA	Crescendo or single events associated with ulcerated stenosis	Uncertain
		All other indications	Inappropriate
	Hemispheric/ Retinal TIA where refractory to medical treatment	Multiple events associated with ulcerated stenosis	Uncertain
		All other indications	Inappropriate
	Retinal TIA	All indications	Inappropriate
	Mild Stroke	All indications	Inappropriate
	Moderate Stroke	All indications	Inappropriate
	Stroke in Evolution	All indications	Inappropriate
Acute Occlusion	Hemispheric TIA		Uncertain
	Retinal TIA	Multiple events	Uncertain
		Single event	Inappropriate
	Mild/ Moderate Stroke		Uncertain
Chronic Occlusion	Any Indication		Inappropriate
Vertebro-basilar	TIA	70% stenosis associated with severe incorrectable vertebrobasilar disease	Uncertain

Appropriateness ratings ranged from appropriate to inappropriate for moderate (50 – 69%) symptomatic stenosis. This reflected the reduced margin of benefit for CE when compared to severe stenosis in the trials [3,4], and the higher surgical complication rates seen in real-life practice [21]. Remote transient events, either hemispheric or retinal, and early operation after moderate stroke were considered to be of uncertain benefit. The presence of a contralateral occluded artery tended to lead to an indication being deemed less appropriate.

CE for those with mild (0–49%) stenoses was never rated as appropriate. Indications thought to be of uncertain benefit were multiple or crescendo hemispheric TIAs in association with an ulcerated plaque. The evidence showing that the presence of a symptomatic, irregular plaque was associated with a higher risk of recurrent stroke was a major factor that guided the panel in this decision [22]. However, there was a disparity of opinion among panellists, which did not meet the RAND definition of disagreement, as to rating of these particular indications, displaying the concerns raised by unplanned subanalyses of randomized controlled trials, which may be misleading [23].

Symptomatic acute occlusions were rated as borderline uncertain/ inappropriate, whereas operating on a chronic occlusion was always deemed inappropriate. The difference between these two indications relates to the limited evidence from case series showing the possibility of improvement in outcome from stroke with emergency endarterectomy for acute occlusion [24].

Asymptomatic carotid stenosis

Table 2 shows the final appropriateness ratings for asymptomatic internal carotid artery stenosis. For asymptomatic stenoses less than 60%, all agreed that CE was inappropriate. There was a wide range of opinion on the ratings for endarterectomy for stenosis greater than 60% and the scores changed considerably between the two rating sessions, moving from borderline appropriate to a final rating of uncertain benefit. Only the rating of severe asymptomatic stenosis with brain infarct present on brain imaging approached the definition but did not meet the RAND definition of disagreement. Otherwise only a single rater viewed these scenarios as an appropriate. This relates to concerns with the generalization of the results of the Asymptomatic Carotid Atherosclerosis Study (ACAS) [5]. The rate of stroke and/or death in the surgical group of that trial was 2.3%. Case series have been unable to match this exceptionally low complication rate, with rates ranging from 2.5% [25] to 5.6% [26]. Most of the panel felt that in real life the marginal benefit seen in the ACAS trial would not be seen in real life. However, in view of the positive randomized trial evidence, they felt that at worst this be rated as an uncertain indication. It was felt that the results of the then ongoing Asymptomatic Carotid Surgery Trial would help to resolve the place of carotid endarterectomy for asymptomatic stenosis.

Table 2

Final appropriateness ratings for carotid endarterectomy for asymptomatic stenosis by degree of stenosis.

Degree of Stenosis	Comments	Appropriateness Rating
80 – 99%	All indications	Uncertain
60 – 79%	All indications	Uncertain
<60%	All indications	Inappropriate

Carotid endarterectomy associated with coronary artery bypass grafting

Table 3 shows the final appropriateness ratings where carotid endarterectomy is performed either with or prior to coronary artery bypass grafting. The imperative to combine procedures is derived from anecdotal experience that those patients with asymptomatic carotid stenosis are at higher risk of stroke following coronary artery bypass grafting. However, case series in the literature both support and refute this [27-30]. The panel discussed a report that the combination of both procedures in a recent large case series appeared to put patients at an unacceptable risk of stroke or death (17.4%) [22]. Therefore, in all but the highest degree of asymptomatic stenosis, the panel thought it was an inappropriate indication for surgery, and even in that situation, an uncertain indication only.

Table 3

Final appropriateness ratings for carotid endarterectomy combined with coronary artery bypass grafting (CABG) for asymptomatic stenosis by degree of stenosis.

Degree of Stenosis	Combined Procedure or Prior to CABG	Infarct on Brain Imaging	Appropriateness Rating
80 – 99%	Combined	Present/ Absent	Uncertain
	Prior	Present	Uncertain
		Absent	Inappropriate
60 – 79%	Combined/ Prior	Present/ Absent	Inappropriate
<60%	Combined/ Prior	Present/ Absent	Inappropriate

Performance of the appropriateness rating algorithm

Of the 801 CE cases examined, 145 (18.1%) were performed for asymptomatic carotid stenosis. When these data were run with the Access database and SAS code that we developed, 541 were deemed appropriate (67.5%), 61 inappropriate (7.6%), 175 uncertain (21.9%). The most common reasons for inappropriate CE were as follows: asymptomatic stenosis less than 60% (28 cases), presence of concurrent medical co-morbidity (dementia, recent myocardial infarction) (18 cases), symptomatic stenosis <50% (five cases), and chronic occlusion (eight cases).

Only 24 cases (3%) were not classified. The reasons for the unclassified cases are as follows: no carotid artery imaging results available (thirteen cases), no information available regarding the degree of contralateral carotid artery stenosis in the presence of a moderate symptomatic stenosis (five cases), endarterectomy ipsilateral to ischaemic symptoms or side not identified (six cases).

Conclusions

The current study involved an all-Canadian panel of experts from the whole spectrum of specialties (Family Medicine, Internal Medicine, Neuroradiology, Neurology, Neurosurgery and Vascular Surgery) that are involved in the care of patients with carotid artery disease. Ratings of a balanced panel have been shown to reflect the opinion in the broader physician community [31] and to avoid the bias inherent in having only practitioners of the intervention doing the rating [12]. Each panellist was nominated by his/her respective specialty society and represented Canada geographically well with panellists from Nova Scotia to British Columbia. Their interpretation of the evidence could therefore be expected to have included a representative view of the current values and resource implications of the Canadian healthcare system.

These ratings identify ongoing areas of uncertainty that could be used to identify priority areas for research. The combination of coronary artery bypass grafting in association with carotid endarterectomy requires further close examination, especially given the high complication rates seen in the Multistate utilization study [21]. CE for asymptomatic carotid stenosis remains an area of uncertainty. Hopefully, the completed Asymptomatic Carotid Surgery Trial will shed further light on the efficacy of CE for asymptomatic disease, but it may be that further work is merited to risk stratify those for whom CE is appropriate. The ratings presented here, however, represent the Canadian viewpoint on the decision to perform CE for asymptomatic carotid disease until the Asymptomatic Carotid Surgery Trial is published. These will remain valid in determining the appropriateness of all CE performed prior to the date of its publication and for some time afterward, until clinical experts and guideline-producing bodies fully reappraise the role of CE in asymptomatic carotid disease after the new trial's publication.

This ongoing area of uncertainty with regard to CE for asymptomatic internal carotid artery stenosis led to the conservative ratings of this Canadian Panel, rating it an intervention of uncertain benefit at best. This is in keeping with international opinion leaders [12]. In contrast, this is discrepant from guidelines [32], appropriateness reviews [33-35] and practice patterns [20,36] from the United States, where the approach toward asymptomatic disease is more aggressive with surgical intervention being seen as standard of care. The synthesis of the evidence by this panel in the form of the appropriateness ratings may therefore be of considerable value to clinicians and policy-makers in health care systems that resemble Canada's single payer system, as opposed to the more procedurally aggressive American system.

Perhaps most importantly, this paper provides access to downloadable audit tools that readers can use to rapidly implement carotid endarterectomy appropriateness audits in their own clinical setting [20]. Both the data collection template and accompanying SAS code are easy to use, and will enable interested parties to perform such CE appropriateness work to compare their local standards of care with the latest Canadian interpretation of the available evidence.

Appendix

Definitions

Appropriate – where a procedure is worth doing if the expected medical benefit to the patient (health status, quality of life, longevity) exceeds the expected negative consequences to the patient (pain, disability, risk of death).

Inappropriate – where a procedure should not be performed if the expected negative consequences to the patient exceeds the expected medical benefit to the patient.

Uncertain – where the net result of a procedure's expected medical benefit and negative consequences are equivocal.

Agreement – scenarios where no more than two of the ratings are outside of the 3-point region containing the median result.

Disagreement – scenarios where three or more readings are in the 1-to-3 region and three or more readings are in the 7-to-9 regions.

Scenario definitions

Asymptomatic carotid artery stenosis – Carotid stenosis that has been identified through imaging, which on thorough history and examination has no ascribable symptoms.

Symptomatic carotid artery stenosis – Carotid artery stenosis, which has been identified following either transient monocular blindness (retinal TIA) or retinal infarction, or transient cerebral ischemia (hemispheric TIA) or stroke.

Crescendo TIA – recurrent transient cerebral or retinal ischaemia in the distribution of the carotid artery, characterized by a definite change in pattern, such as, (1) increased frequency, multiple episodes in a single day, over several days, or a cluster of spells over the course of several days, (2) increased duration, that is, spells lasting longer than the primary event, or several hours in duration, or (3) increased severity, that is, spread in the distribution of ischaemia with greater or new motor, sensory, speech, or visual defects.

Multiple TIA – multiple TIAs that do not match the definition of crescendo TIAs.

Moderate stroke – corresponds to a disability of 3,4 or 5 on the modified Rankin Scale.

Mild stroke – corresponds to a disability of 1 or 2 on the modified Rankin Scale.

Competing interests

None declared.

Authors' contributions

JK prepared evidence summaries for the appropriateness conference, analysed the data and wrote the initial manuscript. HQ contributed to planning the study and editing the manuscript. TEF wrote the application for funding, planned the study, participated in the analysis, and edited the manuscript. WAG was involved in writing the application for funding, planning the study, performing the data analysis and editing the manuscript. All authors read and approved the final manuscript.

Pre-publication history

The pre-publication history for this paper can be accessed here: