Critical appraisal of the design and reporting of studies of imaging and measurement of carotid stenosis.

BACKGROUND AND
PURPOSE: Several hundred studies have been published over the last few years on imaging and measurement of carotid stenosis. Despite all this research, there is still no consensus about how best to image and measure stenosis. One possible explanation for this is that many of the studies have not been large enough or methodologically sound enough to allow useful conclusions to be drawn. We aimed to assess the design and methods of a random sample of published studies of imaging and measurement of carotid stenosis using 9 simple criteria.
METHODS: A formal literature search was performed for studies of imaging and measurement of carotid stenosis. Two subsets were randomly selected for detailed assessment: 20 studies published before 1991 and 20 published between 1993 and 1997 (some years after the initial publication of the ECST and NASCET trials). The criteria used to assess the selected studies were as follows: prospective rather than retrospective study design; patient selection based on a consecutive series or a random sample; adequate detail of study population; adequate detail of imaging techniques; inclusion of all investigations, ie, patients with poor-quality imaging were not excluded; blinded assessment of images; adequate detail of derivation of measurement of stenosis from images or data; adequate data on the reproducibility of measurements of stenosis; and study powered according to a sample-size calculation.
RESULTS: There were many basic methodological deficiencies in both subsets of studies, with relatively little evidence of improvement with time. For example, only 33% of studies were prospective, only 45% studied a consecutive or random selection of patients, and only 38% reported any data on the reproducibility of measurements. More than half of the studies satisfied < or =4 of the 9 quality criteria. However, there was considerable variation between studies, with 7 studies satisfying > or = 7 criteria and 10 studies satisfying < or =2. No study was based on a sample-size calculation. The number of patients studied was often small, particularly in the more recent studies: median sample size was 100 in the 1970-1990 studies and 58 in the 1993-1997 studies (P<0.0001).
CONCLUSIONS: The design and reporting of published studies of imaging and measurement of carotid stenosis are poor and have not improved much in recent years. The majority of published studies are not of a sufficient standard to enable the results to be used to inform clinical practice. The utility of future studies could be improved considerably by better adherence to 9 simple methodological guidelines.