Measurement of Plaque Characteristics Using Coronary Computed Tomography Angiography: Achieving High Interobserver Performance.

BACKGROUND: Coronary computed tomography angiography (CCTA) is used to assess plaque characteristics, remodelling, and progression and regression. Few papers address standard operating procedures that ensure achievement of high interobserver reproducibility. Moreover, assessment of coronary artery bypass grafts has not been reported.
METHODS: A training set of images was created of native coronary segments, spanning the full range of atheromatous disease from normal to severe, excluding totally occluded segments, and including segments with or without calcification (n = 24) and completely normal-appearing bypass grafts (n = 16). Three observers used a validated software program during a training phase to establish standard operating procedures and then to achieve high intraobserver performance based on Pearson's correlation coefficient. Subsequently, interobserver variability for the laboratory as a whole was determined with a focus on measures of plaque volume, low- attenuation plaque (LAP), mixed plaque (MP), and calcified plaque (CP).
RESULTS: We found no substantive differences in analytical issues between grafts and native vessels and emphasize the aggregated data. The range of mean total plaque percent was approximately 55% of total vessel volume with maximal interobserver mean absolute differences of 2% or less. Percent of LAP, MP, and CP demonstrated interobserver standard errors of 1% to 2% and interobserver mean absolute differences of 0% to 1%. Pearson's correlations were all highly significant and ranged from 0.969 to 0.999.
CONCLUSIONS: CCTA provides a rich diversity of measures of atherosclerosis in coronary and bypass segments that are highly reproducible with experience and adherence to standard operating procedures.

Coronary computed tomography angiography (CCTA) is increasingly being used to assess plaque characteristics as well as remodelling and progression and regression of atheroma. The noninvasive nature, the ability to see the entire coronary tree, and the ability to assess densities that reflect different components of plaque are features that are very attractive for longitudinal trials of vasculoactive interventions such as lipid modifying medications. The technique has been validated using comparisons of image-derived density measurements and histologic analyses that quantitate calcium, lipid-rich components, and other components that are either predominantly fibrous or fibrofatty. Comparisons with intravascular ultrasound have also been reported.3, 4, 5, 6, 7, 8 These different components of the plaque may change in differential fashion over time and in patients with different comorbidities such as diabetes. Interventions may also have differential effects on these diverse components.10, 11, 12, 13, 14, 15, 16 For example, lipid-rich components are more amenable to changes induced by statins than are the calcified components, which may actually increase over time.,, As a result, applications of CCTA for assessment of progression and regression must identify not solely overall changes in atheroma burden but also the components of the plaque most likely to undergo change. Low-attenuation plaque (LAP) components, reflecting the lipid-rich element of plaques, give information about the potential vulnerability and instability of plaques., Such plaques are more prone to rupture or erosion, thereby providing a substrate for clinical events such as acute coronary syndromes and providing mechanistic linkage between beneficial plaque remodelling and the reduction of cardiac events.,14, 15, 16,

The emerging importance of this method has motivated applications beyond the coronary tree. The Effect of Evolocumab on Saphenous Vein Graft Patency Following Coronary Artery Bypass Surgery (NEWTON-CABG) trial is an ongoing prospective randomized clinical trial assessing the role of PCSK9 inhibitors postcoronary artery bypass surgery for maintaining graft patency (ClinicalTrials.gov identifier: NCT03900026). An assessment of atheroma volume and components of this volume will also be undertaken. Limited information suggests that these CCTA-derived measurements are reproducible.21, 22, 23 The purpose of this study was to identify standard operating processes yielding high interobserver reproducibility measurements of plaque and its components to enhance the reliability of conclusions from clinical trials of either native coronary or coronary artery bypass graft segments.

Methods

Three highly experienced research associates in the Cardiovascular Imaging Research Core Laboratory (CIRCL; University of British Columbia, Vancouver, BC) underwent training using the Vitrea SurePlaque (Vital, Minnetonka, MN) software program for measurement of vessel and plaque volumes, including subcomponents of plaque. The initial step was to identify a training set of images from laboratory archives of completely normal appearing bypass grafts (n = 16) as well as coronary segments spanning the full range of atheromatous disease from normal to severe, excluding totally occluded segments, and including segments with or without calcification (n = 24). These were selected on the basis of good image quality and avoidance of segments with artifacts resulting from motion, misregistration, and implanted devices or metal artifacts that preclude tissue characterization. The resulting 40 segments were selected from 14 patients in the laboratory archive of CCTA scans. Training was iterative and included identification among the group of issues creating challenges for reproducible analyses. Following this phase, the operators were charged with independent analysis of the test set of images in an attempt to achieve excellent intraobserver reproducibility characteristics for plaque volume and component measurements based upon Pearson’s correlations.

In the absence of histologic or pathologic information about the segments that were studied, the reference values were considered to be the mean of the 2 trials for each of the 3 operators that showed excellent intraobserver results. The degree to which each operator matched these results formed the basis of the interobserver variability characteristics for the laboratory as a whole.

The ranges of Hounsfield units (HU) used were < 30 HU (LAP), 30 to 349 HU (mixed plaque [MP]), and ≥ 350 HU (calcified plaque [CP]).,, The aggregate term “mixed plaque" was used instead of additional subranges (eg, fibrofatty and fibrous plaque components) because each of these categories also contains normal vascular wall as a result of the HU conventions used in these algorithms and in studies of this nature. In addition, with respect to grafts, particularly arterial grafts selected based on completely normal appearance, it seemed inappropriate to imply specific measurement of fibrofatty or fibrous plaque.

Vessel and plaque features that were measured included linear measurements (segment length [mm], diameter stenosis measurement [%] at the point of minimum lumen diameter [mm] using the most normal lumen diameter as reference, remodelling index [ratio] at the point of minimum lumen diameter using the vessel diameter at that point as the numerator and the vessel diameter at the point of most normal vessel lumen as reference) and measurement of percent area stenosis (measured at the point of minimum lumen area [mm2] using the vessel area at that point as the reference); volume measurements in mm3 (total vessel volume, total plaque volume, LAP volume, MP volume, and CP volume); and measurements derived from volume measurements (total plaque percent using total vessel volume as the reference, and LAP percent, MP percent, and CP percent using total plaque as the reference for these plaque-specific measurements). Other measurements were also recorded (volume measurements normalized by length of segment and wall volume and lumen volume %) but are not reported further.

The results of each of the 3 operators (40 segments, measured twice) were compared with the overall mean results from all observations (40 segments, measured 6 times). The metrics assessed were the mean, standard error of the mean, mean of differences, mean of absolute differences and Pearson’s correlation coefficient. Bland-Altman plots were also constructed (Excel 2016; Microsoft, Redmond, WA).

Analyses were stratified by native coronary segments, graft segments, and all segments combined. Significance level of P < 0.05 was used for all measures of reproducibility. The range of interobserver performance was used to summarize overall performance characteristics of the laboratory.

Results

We found no substantive differences in analytical issues between grafts and native vessels and report primarily the aggregated data. More detailed comparisons among the vessel and graft types are provided in Supplemental Tables S1 and S2.

Scan parameters ranged as follows: kVp, 80 to 120 and mA 208 to 2198. Calcification percentages ranged from 0% to 2% in graft segments and from 0% to 28.1% in coronary segments.

Figure 1 provides a composite of screen shots that depict the analyses, and Figure 2 provides variable examples of differing plaques subjected to quantitative plaque analysis. During the training phase, the 3 operators identified potential pitfalls in measuring diverse plaque characteristics that formed the basis of a standard operating procedure (Table 1). Figure 3 illustrates an example of a pitfall resulting from inclusion of a variable portion of branch point vessels, leading to marked variability in vessel and plaque measurements. With these standard operating principles in mind, each of 3 observers achieved reproducible intraobserver results in 2 trials demonstrating highly significant Pearson’s correlation coefficients, which were 0.83 to 1.00 and with P < 0.000001 at a minimum. Supplemental Table S3 and Figure S1 provide a summary of individual intraobserver performance metrics and demonstrate the performance of each operator using correlation and Bland-Altman plots.

Figure 1

Plaque analysis. This figure shows a composite of screen shot outputs of the plaque analysis software. The lumen is depicted in green, low-attenuation plaque in red, mixed plaque in blue, and calcified plaque in yellow. (A) Shows a representative cross-section with volumetric parameters shown below. (B) Shows the linear reconstruction of the proximal right coronary artery from which the indexes are calculated. (C) Shows the location of key reference points for the measurements and the Hounsfield unit distribution plots.

Figure 2

Distribution of plaque components. The panel of 4 cross-section images illustrates the varying distribution of plaque component from 4 different coronary segments. The lumen is depicted in green, low-attenuation plaque in red, mixed plaque in blue, and calcified plaque in yellow.

Table 1

Suggested standard operating procedures for training to optimize reproducibility.

Vessel probe (This is Vitrea’s single-click tool to automatically define the centerline through the vessel lumen and perform automated detection of the inner lumen and vessel wallcontours from the ostium to the distal end of each artery instraightened multiplanar reformatted images. Similar functionality is provided in other software packages designed for assessing arteries and plaques.)	• After opening a CCTA case in the Vitrea’s Cardiac Analysis application, the program launches the “auto vessel probe” feature, which creates a centre line through the lumen of each coronary vessel and automatically segments and labels the main coronary arteries. A centre line serves as a reference point from which contours of the lumen and outer vessel wall are automatically defined. If the automatic vessel detection does not provide appropriate lumen and vessel contours, or when analyzing bypass grafts, manual vessel analysis and probing can be performed. • With situations in which a significant lesion is present, selection of a vessel location distal to the lesion to initiate the automatic definition of the lumen centre line and associated contours is recommended.
Lumen and vessel contours	• Review the lumen centerline and edit as necessary. Editing the centre line can improve the edge detection for the lumen and vessel contours. • If the automatically provided contours remain suboptimal, reprobe by selecting another point along the vessel. Reprobing is preferred to editing poorly tracked contours, as editing is difficult and time consuming. • Manual editing of vessel contours is difficult. The lack of visual cues to indicate the outer edge of the vessel make evaluating and repositioning the vessel contour extremely challenging, even with adjustment of the window and level display settings, as this has minimal effect on improving the ability to resolve the outer edges of the vessel wall.
Identifying segment landmarks	• Evaluate vessel anatomy with MPR, CPR, and SPR displays with a slice thickness setting of 5 to10 mm to aid the identification of branches. • Rotate the SPR view, and confirm the vessel segmentation landmark locations with the MPR and CPR views. (Note: A left button mouse selection on a location on the SPR display will automatically position the cross hairs on the corresponding location on the MPR views. Other software programs provide this ability as well.)
Start and end contours of the vessel segment (defining length)	• The first and last cross-sections of a vessel segment are positioned to the segment landmarks by adjusting the segment boundary markers on CPR and SPR displays. Ensure that the lumen and vessel contours of the first and last cross- sections are not affected by the segment landmarks such as branches (see Fig. 3).
Reference cross-section	• Using the lumen diameter histogram, identify the cross-section(s) with maximum lumen diameter. Cross-sections with calcium and overt vessel wall thickening should not be selected for the reference. • If there is more than 1 cross-section with the same maximum lumen diameter, the cross-section with the largest diameter and largest value for remodelling index can be used to determine the reference diameter.

CCTA, coronary computed tomography angiography; CPR, curved planar reformatting; MPR, multiplanar reformatting; SPR, straightened planar reformatting.

Figure 3

Effect of segment boundary selection. The series of sequential, cross-section images progress from left (1) to right (5) and illustrate an emerging branch at the 2 o’clock position, affecting the lumen and vessel contours. When identifying the last cross-section of a vessel segment, 1 and 2 would be included in the analysis segment but 3, 4, and 5 would not. In this example, the change in axis of the maximum lumen diameter (green line) aided in detecting the influence of the branch on the lumen contour.

Table 2 summarizes the interobserver reproducibility characteristic of the laboratory as a whole, based on volumetric measures of vessels and plaques. The interobserver range of mean vessel volumes was 494 to 498 mm3 for coronary segments, and 1813 to 1842 mm3 for graft segments, yielding an overall average volume of 1022 to 1025 mm3. The interobserver mean absolute differences were very small (maximum of 23 mm3), with mean differences distributed tightly around 0.0 mm3. The Pearson correlation was nearly 1.0 in all cases. The interobserver plaque-related measurements (total plaque, LAP, MP, and CP) showed similarly high reproducibility characteristics, with the highest mean absolute difference only 11 mm3 for total plaque volume. Table 3 shows the interobserver reproducibility characteristics of the laboratory as a whole, based on measures derived from the volume measurements in Table 2. The range of mean total plaque percent was approximately 55%, with maximal mean absolute differences of 2% or less. Similar results were noted for interobserver reproducibility measures of LAP percent, MP percent, and CP percent. Figure 4 summarizes the main findings. Supplemental Table S1 summarizes similarly favourable interobserver performance characteristics for segment length, remodelling index, percent of diameter stenosis, and percent of area stenosis.

Table 2

Interobserver variability measurements for volumetric measures of vessels and plaques

Parameter	Measurements	Coronary segments	Graft segments	All segments
Vessel volume (mm3)	Mean	494 to 498	1813 to 1842	1022 to 1025
	Standard error of the mean	53 to 55	247 to 248	146 to 146
	Mean of differences	–2.0 to 3.2	–12.6 to 16.2	–5.7 to 8.4
	Mean of absolute differences	7 to 12	13 to 23	9 to 16
	Pearson correlation	0.998 to 0.999	1.000 to 1.000	1.000 to 1.000
Total plaque (mm3)	Mean	281 to 282	933 to 941	542 to 545
	standard error of the mean	35 to 36	120 to 122	73 to 73
	Mean of differences	–0.8 to 1.1	–3.2 to 4.7	–1.7 to 1.7
	Mean of absolute differences	6 to 11	8 to 11	7 to 11
	Pearson correlation	0.996 to 0.999	1.000 to 1.000	0.999 to 1.000
Low-attenuation plaque (mm3)	Mean	47 to 48	207 to 210	111 to 113
	Standard error of the mean	5 to 6	28 to 28	17 to 17
	Mean of differences	–0.4 to 0.5	–1.2 to 2.3	–0.7 to 1.2
	Mean of absolute differences	1 to 2	2 to 3	2 to 2
	Pearson correlation	0.996 to 0.997	1.000 to 1.000	1.000 to 1.000
Mixed plaque (mm3)	Mean	195 to 196	731 to 736	409 to 411
	Standard error of the mean	24 to 24	98 to 100	59 to 59
	Mean of differences	–0.9 to 0.5	–3.3 to 4.4	–1.0 to 1.2
	Mean of absolute differences	5 to 9	8 to 10	6 to 10
	Pearson correlation	0.994 to 0.999	1.000 to 1.000	0.999 to 1.000
Calcified plaque (mm3)	Mean	37 to 39	1 to 2	23 to 24
	Standard error of the mean	10 to 12	1 to 1	7 to 7
	Mean of differences	–0.9 to 0.8	–0.3 to 0.3	–0.5 to 0.4
	Mean of absolute differences	2 to 3	0 to 0	1 to 2
	Pearson correlation	0.995 to 0.999	0.996 to 0.998	0.996 to 0.999

Table 3

Interobserver variability measurements for parameters derived from volumetric measurements of vessels and plaques

Parameter	Measurements	Coronary segments	Graft segments	All segments
Total plaque %	Mean	55 to 55	53 to 54	54 to 55
	Standard error of the mean	2 to 3	1 to 2	2 to 2
	Mean of differences	–0.5 to 0.3	–0.1 to 0.2	–0.4 to 0.3
	Mean of absolute differences	1 to 2	0 to 0	1 to 1
	Pearson correlation	0.978 to 0.993	0.997 to 0.999	0.986 to 0.995
Low-attenuation plaque %	Mean	18 to 18	22 to 23	18 to 20
	Standard error of the mean	1 to 1	1 to 1	1 to 1
	Mean of differences	–0.1 to 0.2	–0.1 to 0.1	–0.1 to 0.2
	Mean of absolute differences	1 to 1	0 to 0	1 to 1
	Pearson correlation	0.947 to 0.985	0.996 to 0.999	0.969 to 0.991
Mixed plaque %	Mean	71 to 72	77 to 77	71 to 74
	Standard error of the mean	2 to 2	1 to 1	2 to 2
	Mean of differences	–0.1 to 0.2	–0.1 to 0.1	–0.1 to 0.1
	Mean of absolute differences	1 to 1	0 to 0	0 to 1
	Pearson correlation	0.989 to 0.998	0.996 to 0.999	0.991 to 0.998
Calcified plaque %	Mean	10 to 11	0 to 0	6 to 6
	Standard error of the mean	2 to 2	0 to 0	2 to 2
	Mean of differences	–0.1 to 0.2	–0.03 to 0.03	–0.1 to 0.1
	Mean of absolute differences	0 to 1	0 to 0	0 to 0
	Pearson correlation	0.997 to 0.999	0.996 to 0.999	0.997 to 0.999

Figure 4

Interobserver variability for measures of plaque components. This illustration summarizes the key interobserver performance characteristics for the laboratory as a whole. On average, the set of test arterial segments and grafts displayed total plaque volume accounting for 55% of the total vessel volume. The lumen volume averaged 45%. With respect to the total plaque volume, mixed plaque was 74%, calcified plaque was 6%, and low-attenuation plaque (LAP) was 20%. The ranges of the key performance measurements are shown in tabular form.

Discussion

CCTA has evolved rapidly for the anatomic detection of coronary disease and has been exploited further to extract functional information, such as flow reserve indexes and features of vessel remodelling and plaque structure.1, 2, 3,6, 7, 8, This analysis focussed on advanced measures of plaque features and demonstrates that a standard operating procedure can promote high interobserver performance characteristics in a core imaging laboratory, thereby providing reliable measurements for randomized clinical trials. Even with such procedures, and even within a laboratory with experienced operators, months of training experience were required to achieve these performance characteristics. This suggests strongly that routine clinical applications and conclusions in serial studies of individual patients should be interpreted cautiously and should be performed by highly experienced personnel who are properly trained and who use well-developed standard operating procedures for these analyses.

It is axiomatic that the results are very dependent upon the adequacy of measurement of both the lumen and the outer wall of the vessel, which entails a certain degree of subjectivity and discretional editing of automatically determined contours. This is particularly challenging with respect to the outer vessel wall, for which there are few—if any—visual clues as to the adequacy of the automatically determined contour. Windowing and slice thickness selections to assist in this process are discretionary. Moreover, matching of length in analyses is critical for the total volume measurements and also for comparing serial measurements. Finally, although many centres purport to distinguish between fibrofatty and fibrous plaque, the reality is that as long as these are defined based upon HU, there is no ability to distinguish such subcomponents from normal vascular wall. This suggests, at least through experience—particularly with completely normal arterial grafts—that the less-specific term “mixed plaque” may be more appropriate. The term “noncalcified plaque,” which is also often reported, should be interpreted with this same caveat in mind. Thus, serial changes in such components must be interpreted cautiously.

Limitations

The main limitation of this study is that a histologic gold standard or an intravascular ultrasound correlate was not available or feasible for this analysis. Even so, the approach is not dissimilar to previous analyses of this nature, and this type of application in the field is dependent upon limited tissue validation studies. Despite the latter, use of CCTA for assessing plaque has become highly popular, thereby warranting our approach of encouraging extensive training experience and standard operating principles before reliable and reproducible results can be achieved. Although the number of native segments and grafts may appear limited, we have attempted to represent the full spectrum of disease that plaque burden analyses would need to accommodate. In addition, our library of grafts had limited examples of abnormality other than total occlusion, which precludes plaque burden analysis. Our library is anonymized to adhere to national and international policies for images in clinical studies; therefore, the demographic background is not provided. Even so, although demographic factors might indirectly affect image quality, there are no demographic factors that are used to dictate how the quantitative algorithms are applied or that would affect principles of image analysis. Finally, our analyses are purposely limited to images of high quality to understand and to develop principles of analysis that are within the control of the analysis operator. Indiscriminate application to images of poor quality should be discouraged, as interobserver performance will understandably be diminished in spite of rigourous training and adherence to standard operating procedures.

Our results complement and extend previous studies showing that plaque volume and component measures are reproducible.21, 22, 23, To our knowledge, analysis of grafts has not previously been attempted, and we show that analysis principles are similar and yield highly reproducible results. Inherent in these measurements are errors related not solely to operator issues but also calcium-related artifact or surgical clips (blooming and beam-hardening), especially when located at segment margins and when occurring within the plaque itself. It is generally thought that effects on measurements by factors such as contrast enhancement, contrast timing, and heart rate are likely to be small.

Conclusions

CCTA provides a rich diversity of measures of the atherosclerotic process in the coronaries and in coronary bypass grafts. Interventional trials using such measures as end points depend explicitly on demonstration of highly reproducible results. Such results can be achieved but only with extensive training and experience and with careful attention to best practices as incorporated in a standard operating procedure.

Funding Sources

No funding was received for this article.

Disclosures

The authors have no conflicts of interest to disclose.