Marielle Ernst1, Albert J Yoo2, Levente Kriston2, Michael H Schönfeld2, Eik Vettorazzi2, Jens Fiehler2. 1. From the Department of Diagnostic and Interventional Neuroradiology (M.E., M.H.S., J.F.), Department of Medical Psychology (L.K.), and Department of Medical Biometry and Epidemiology (E.V.), University Medical Center Hamburg-Eppendorf, Hamburg, Germany; and Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston (A.J.Y.). m.ernst@uke.de. 2. From the Department of Diagnostic and Interventional Neuroradiology (M.E., M.H.S., J.F.), Department of Medical Psychology (L.K.), and Department of Medical Biometry and Epidemiology (E.V.), University Medical Center Hamburg-Eppendorf, Hamburg, Germany; and Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston (A.J.Y.).
Abstract
BACKGROUND AND PURPOSE: Angiographic occlusion as a surrogate marker of satisfactory aneurysm treatment is commonly used in clinical trials although some pitfalls have to be considered. To investigate the inter-rater reliability of visual rating of aneurysm occlusion as study end point, we performed a systematic review and meta-analysis. METHODS: Electronic databases (MEDLINE, EMBASE, PubMed, and the Cochrane Library) were searched up to June 2014. Assessment of risk for bias was based on the Quality Appraisal Tool for Studies of Diagnostic Reliability and the Guidelines for Reporting Reliability and Agreement studies. Inter-rater reliability estimates were pooled across studies using meta-analysis, and the influence of several factors (eg, imaging methods, grading scales, and occlusion rate) was tested with meta-regression. RESULTS: From 1193 titles, 644 abstracts and 87 full-text versions were reviewed. Twenty-six articles met the inclusion criteria and provided 77 reliability estimates. Twenty-one different rating scales were used, and statistical analysis varied. Mean inter-rater agreement of the pooled studies was substantial (κ=0.65; 95% confidence interval, 0.60-0.69). Reliability varied significantly as a function of imaging methods, grading scales, occlusion rates, and their interaction. Observer agreement substantially increased with increasing occlusion rate in digital subtraction angiography but not in MR angiography. Reliability was higher in studies using 2- or 3-value grading scales than in studies with 4-value grading scales. CONCLUSIONS: There is significant heterogeneity between studies evaluating the reliability of visual evaluation of aneurysm coiling. On the basis of our analysis, we found that the combination of magnetic resonance angiography, 3-value grading scale, and 2 trained raters seems most promising for usage as surrogate study end points.
BACKGROUND AND PURPOSE: Angiographic occlusion as a surrogate marker of satisfactory aneurysm treatment is commonly used in clinical trials although some pitfalls have to be considered. To investigate the inter-rater reliability of visual rating of aneurysm occlusion as study end point, we performed a systematic review and meta-analysis. METHODS: Electronic databases (MEDLINE, EMBASE, PubMed, and the Cochrane Library) were searched up to June 2014. Assessment of risk for bias was based on the Quality Appraisal Tool for Studies of Diagnostic Reliability and the Guidelines for Reporting Reliability and Agreement studies. Inter-rater reliability estimates were pooled across studies using meta-analysis, and the influence of several factors (eg, imaging methods, grading scales, and occlusion rate) was tested with meta-regression. RESULTS: From 1193 titles, 644 abstracts and 87 full-text versions were reviewed. Twenty-six articles met the inclusion criteria and provided 77 reliability estimates. Twenty-one different rating scales were used, and statistical analysis varied. Mean inter-rater agreement of the pooled studies was substantial (κ=0.65; 95% confidence interval, 0.60-0.69). Reliability varied significantly as a function of imaging methods, grading scales, occlusion rates, and their interaction. Observer agreement substantially increased with increasing occlusion rate in digital subtraction angiography but not in MR angiography. Reliability was higher in studies using 2- or 3-value grading scales than in studies with 4-value grading scales. CONCLUSIONS: There is significant heterogeneity between studies evaluating the reliability of visual evaluation of aneurysm coiling. On the basis of our analysis, we found that the combination of magnetic resonance angiography, 3-value grading scale, and 2 trained raters seems most promising for usage as surrogate study end points.
Authors: Anass Benomar; Behzad Farzin; David Volders; Guylaine Gevry; Justine Zehr; Robert Fahed; William Boisseau; Jean-Christophe Gentric; Elsa Magro; Lorena Nico; Daniel Roy; Alain Weill; Charbel Mounayer; François Guilbert; Laurent Létourneau-Guillon; Gregory Jacquin; Chiraz Chaalala; Marc Kotowski; Thanh N Nguyen; David Kallmes; Phil White; Tim E Darsaut; Jean Raymond Journal: Neuroradiology Date: 2021-02-24 Impact factor: 2.804
Authors: Min Soo Kim; Kyung Il Jo; Je Young Yeon; Jong Soo Kim; Keon Ha Kim; Pyoung Jeon; Seung Chyul Hong Journal: Neuroradiology Date: 2016-10-18 Impact factor: 2.804