Prediction model to estimate presence of coronary artery disease: retrospective pooled analysis of existing cohorts.

OBJECTIVES: To develop prediction models that better estimate the pretest probability of coronary artery disease in low prevalence populations.
DESIGN: Retrospective pooled analysis of individual patient data.
SETTING: 18 hospitals in Europe and the United States. PARTICIPANTS: Patients with stable chest pain without evidence for previous coronary artery disease, if they were referred for computed tomography (CT) based coronary angiography or catheter based coronary angiography (indicated as low and high prevalence settings, respectively). MAIN OUTCOME MEASURES: Obstructive coronary artery disease (≥ 50% diameter stenosis in at least one vessel found on catheter based coronary angiography). Multiple imputation accounted for missing predictors and outcomes, exploiting strong correlation between the two angiography procedures. Predictive models included a basic model (age, sex, symptoms, and setting), clinical model (basic model factors and diabetes, hypertension, dyslipidaemia, and smoking), and extended model (clinical model factors and use of the CT based coronary calcium score). We assessed discrimination (c statistic), calibration, and continuous net reclassification improvement by cross validation for the four largest low prevalence datasets separately and the smaller remaining low prevalence datasets combined.
RESULTS: We included 5677 patients (3283 men, 2394 women), of whom 1634 had obstructive coronary artery disease found on catheter based coronary angiography. All potential predictors were significantly associated with the presence of disease in univariable and multivariable analyses. The clinical model improved the prediction, compared with the basic model (cross validated c statistic improvement from 0.77 to 0.79, net reclassification improvement 35%); the coronary calcium score in the extended model was a major predictor (0.79 to 0.88, 102%). Calibration for low prevalence datasets was satisfactory.
CONCLUSIONS: Updated prediction models including age, sex, symptoms, and cardiovascular risk factors allow for accurate estimation of the pretest probability of coronary artery disease in low prevalence populations. Addition of coronary calcium scores to the prediction models improves the estimates.

Introduction

In the United States, about 10.2 million people have chest pain complaints each year,1 and more than 1.1 million diagnostic procedures of catheter based coronary angiography are performed on inpatients each year.2 In a recent report based on the national cardiovascular data registry of the American College of Cardiology,3 only 41% of patients undergoing elective procedures of catheter based coronary angiographies are diagnosed with obstructive coronary artery disease. The report’s authors concluded that better risk stratification was needed, underlined by decision analyses showing that the choice of further diagnostic investigation in patients with chest pain depends primarily on the pretest probability of coronary artery disease.4 5 6

The American College of Cardiology/American Heart Association,7 8 European Society of Cardiology,9 and United Kingdom10 currently recommend using the Diamond and Forrester model11 or the Duke clinical score12 13 to estimate the pretest probability of coronary artery disease in patients with chest pain. The Diamond and Forrester model tends to overestimate the probability of coronary artery disease (defined as ≥50% stenosis), and a revised version has recently been published.14 The Duke clinical score12 13 estimates the probability of coronary artery disease (≥75% stenosis) which, to our knowledge, has not been validated in populations outside the US. Although the American College of Cardiology/American Heart Association and European Society of Cardiology recommend exercise electrocardiography to select patients for further diagnostic investigation, UK guidelines recommend using the computed tomography (CT) based coronary calcium score in patients with a low to intermediate pretest probability (10-29%).

We perceived a need for an updated and stepwise approach to estimate the probability of coronary artery disease in patients with new onset of chest pain in a low prevalence population as clinical information and test results become available, in particular because implementation of the guidelines needs calculation of the pretest probability. Therefore, we aimed to estimate the probability of obstructive coronary artery disease on the basis of clinical presentation and cardiovascular risk factors, and to determine the incremental diagnostic value of exercise electrocardiography and the coronary calcium score.

Methods

Design overview

Researchers from Europe and the US formed a consortium. An existing database of at least 80 eligible patients was required for participation. Participation did not involve any financial incentives. All patients had to be enrolled in single centre studies, and local approval from the institutional review board for the original research objectives was required. The consortium is part of the European network for the assessment of imaging in medicine, which is an initiative of the European Institute of Biomedical Imaging Research.15 One of the network’s goals was to perform pooled analyses of existing prospectively collected data, improving power and increasing generalisability of results.

Participants

Patients were eligible for the analysis if they presented with stable chest pain and were referred for catheter based or CT based coronary angiography (≥64 slice). Patients were not eligible if they had acute coronary syndrome or unstable chest pain, had a history of myocardial infarction or previous revascularisation (percutaneous coronary intervention or coronary artery bypass graft surgery), or provided no informed consent. For the diagnosis of coronary artery disease, catheter based coronary angiography is regarded as the reference standard, which is an expensive and invasive procedure with a risk of complications. Non-invasive testing is generally recommended to select patients who might benefit from catheter based coronary angiography. CT based coronary angiography is a less invasive and less expensive test, with a high sensitivity and specificity for the detection of coronary artery disease on catheter based coronary angiography.16 17 18 A negative result from CT based coronary angiography virtually excludes the presence of obstructive coronary artery disease, whereas a positive result might need confirmation by the catheter based test.

Fourteen datasets consisted of consecutive patients enrolled in a prospective study for other research objectives. Four datasets consisted of patients retrospectively identified as eligible via electronic radiology reporting systems. Inclusion and exclusion criteria were evaluated by experienced doctors and missing information was obtained from patient records (web appendix table 1).19 20 21

Definitions

We collected data for age, sex, symptoms, cardiovascular risk factors, test results, and presence of coronary artery disease. Chest pain symptoms were classified as typical, atypical, or non-specific. Typical chest pain was defined as all of the following criteria: (1) substernal chest pain or discomfort that is (2) provoked by exertion or emotional stress and (3) relieved by rest or nitroglycerine (or both). We defined atypical chest pain as two of these criteria. If one or none of the criteria was present, symptoms were classified as non-specific.22

Definitions for hypertension, diabetes, dyslipidaemia, and smoking differed slightly across hospitals (web appendix table 1). Table 1 lists the most common definitions. We determined coronary calcium scores by the Agatston method23 and used log transformation to account for its skewed distribution.

Table 1

 Patient characteristics. Most commonly used risk factor definitions provided; characteristics data are no (%) of patients unless stated otherwise

Characteristic	Low prevalence setting (10 hospitals)			High prevalence setting (8 hospitals)
	Value	No (%) of patients with available data		Value	No (%) of patients with available data
No of patients per hospital
Mean (range)	443 (80-1241)	4426 (100)		156 (85-549)	1251 (100)
Age (years)
Mean (standard deviation)	57.2 (12)	4422 (99.9)		63.6 (10)	1251 (100)
Interquartile range	49-66			57-70
Range	18-92			18-93
Sex
Male	2406 (54)	4426 (100)		877 (46)	1251 (100)
Chest pain*
Typical	759 (17)	4424 (99.9)		656 (53)	1247 (99.7)
Atypical	2699 (61)			278 (22)
Non-specific	966 (22)			313 (25)
Other clinical characteristics
Diabetes†	622 (15)	4238 (96)		229 (18)	1250 (99.9)
Hypertension‡	2475 (58)	4300 (97)		840 (67)	1251 (100)
Dyslipidaemia§	2194 (52)	4255 (96)		801 (65)	1235 (99)
Smoking¶	1231 (29)	4273 (97)		454 (36)	1249 (99.8)
Body mass index (mean (median))**	28 (27)	4117 (93)		28 (27)	1206 (96)
History
Family history of coronary artery disease††	1720 (44)	3938 (89)		136 (51)	265 (21)
Previous cerebrovascular disease‡‡	78 (3)	2531 (57)		25 (9)	269 (22)
Previous renal artery disease	43 (1)	3351 (76)		2 (1)	316 (25)
Previous peripheral arterial disease	79 (2)	3356 (76)		10 (3)	369 (29)
Exercise electrocardiography
Normal	671 (42)	1612 (36)		166 (30)	547 (44)
Abnormal	443 (27)			336 (61)
Non-diagnostic	498 (31)			45 (8)
Coronary calcium score§§
Mean (standard deviation), median	160 (399), 4	4009 (91)		442 (643), 182	858 (69)
0	1777 (44)			155 (18)
0-<10	402 (10)			44 (5)
10-<100	749 (19)			154 (18)
100-<400	606 (15)			208 (24)
≥400	475 (12)			297 (35)
Results from CT based coronary angiography¶¶
No obstructive CAD	3232 (75)	4287 (97)		324 (36)	903 (72)
Moderate CAD	505 (12)			501 (55)
Severe CAD	550 (13)			78 (9)
Results from catheter based coronary angiography***
No obstructive CAD	406 (48)	848 (19)		480 (40)	1214 (97)
Moderate CAD	177 (21)			541 (45)
Severe CAD	265 (31)			193 (16)
Patients who underwent both CT based coronary angiography and catheter based coronary angiography
Results from CT based coronary angiography¶¶
No obstructive CAD	230 (31)	742 (17)		296 (34)	867 (69)
Moderate CAD	277 (37)			495 (57)
Severe CAD	235 (32)			76 (9)
Results from catheter based coronary angiography***
No obstructive CAD	356 (48)	742 (17)		339 (39)	867 (69)
Moderate CAD	172 (23)			436 (50)
Severe CAD	214 (29)			92 (11)

CAD=coronary artery disease; moderate CAD=50-70% stenosis; severe CAD=≥70% stenosis or ≥50% left main stenosis.

*According to traditional chest pain classification.19

†Defined as fasting glucose levels of ≥7 mmol/L or treatment with diet intervention, oral glucose lowering agent, or insulin.

‡Defined as blood pressure of ≥140/90 mm Hg or the use of antihypertensive drugs.

§Defined as total cholesterol concentration of ≥5.2 mmol/L or treatment with lipid lowering drugs.

¶Includes current or past smoking.

**Defined as weight (kg) divided by height (m2).

††Presence of CAD in a first degree female relative (age <65 years) or male relative (age <55 years).

‡‡History of carotid artery disease, stroke, or transient ischaemic attack.

§§Agatston score as measured by computed tomography.20

¶¶Some hospitals only compared obstructive CAD (≥50% stenosis) with no obstructive CAD (hospital numbers 2, 3, and 18 in web appendix table 1; for example, they did not consider the severe category for coronary computed tomography angiography). One hospital did not categorise patients with ≥50% left main stenosis in the severe CAD category (hospital number 6 in web appendix table 1).

***Two hospitals only compared obstructive CAD (≥50% stenosis) with no obstructive CAD (hospital numbers 2 and 18 in web appendix table 1; for example, they did not consider the severe category for catheter based coronary angiography).

Outcomes

Primary outcome was obstructive coronary artery disease, defined as at least one vessel with at least 50% diameter stenosis found on catheter based coronary angiography. Since we combined existing databases from different hospitals, CT based and catheter based coronary angiographies were performed at each institution according to local protocols; we allowed both visual assessment and quantitative assessment for the interpretation of results for these procedures.

Statistical analysis

We assumed missing data occurred at random, depending on the clinical variables and the results of CT based coronary angiography, and performed multiple imputations using chained equations.24 Missing values were predicted on the basis of all other predictors considered, the results of CT based coronary angiography, as well as the outcome.25 26 We created 20 datasets with identical known information, but with differences in imputed values reflecting the uncertainty associated with imputations. In total, 667 (2%) clinical data items were imputed.

In our study, only a minority of patients underwent catheter based coronary angiography. An analysis restricted to patients who underwent catheter based coronary angiography could have been influenced by verification bias.27 Therefore, we imputed data for catheter based coronary angiography by using the CT based procedure as an auxiliary variable, in addition to all other predictors.28 Results for the two procedures correlate well together, especially for negative results of CT based coronary angiography.16 17 18 This strong correlation was confirmed in the 1609 patients who underwent both procedures (Pearson r=0.72). Since its data were used for imputation, the CT based procedure was not included as a predictor in the prediction models. Our approach was similar to using the results of CT based coronary angiography as the outcome variable when the catheter based procedure was not performed (which was explored in a sensitivity analysis). However, this approach is more sophisticated because it also takes into account other predictors and the uncertainty surrounding the imputed values. We imputed 3615 (64%) outcome values for catheter based coronary angiography. Multiple imputations were performed using Stata/SE 11 (StataCorp).

External validation of the Duke clinical score

To evaluate the performance of the Duke clinical score, we calculated the predicted probability based on published coefficients13 (that is, prediction of ≥75% stenosis). Since patients with evidence of previous coronary artery disease were excluded, we assumed all had a normal resting electrocardiogram. If resting findings were available and taken into account, any overestimation would increase further. We used a calibration plot to compare predicted probability with the observed proportions of severe disease (that is, ≥70% stenosis or ≥50% left main stenosis) in a calibration plot.

Development of new prediction models

We defined three prediction models: a basic model including age, sex, symptoms, and setting; a clinical model including age, sex, symptoms, setting, diabetes, hypertension, dyslipidaemia, smoking, and body mass index; and an extended model including all clinical variables and the coronary calcium score. Since all clinical variables are known to be associated with coronary artery disease,29 all predictors were entered simultaneously in a multivariable, random effects, logistic regression model. We included hospital as a random effect to account for clustering of patients within hospitals. Availability of data for exercise electrocardiography was limited, and web appendix table 5 explores the variable’s incremental predictive value. We omitted non-significant predictors with small effects (that is, odds ratio <1.01).

Setting variable

To account for differences in patient selection across datasets (based on referrals to catheter based coronary angiography v CT based coronary angiography), we created a dummy variable for setting. This variable was coded “0” (low prevalence setting) if a patient came from a database that was created by selecting patients who underwent CT based coronary angiography (of whom only a proportion underwent the catheter based procedure in addition to the CT based procedure), and coded as “1” (high prevalence setting) if a patient came from a database that was created by selecting patients who underwent catheter based coronary angiography (of whom a proportion also underwent the CT based procedure).

We intended to apply our prediction models to patients in low prevalence populations, for whom the best diagnostic management should be determined based on an estimated pretest probability.10 By contrast, all patients in the high prevalence setting had a clinical indication for catheter based coronary angiography, in whom estimating the pretest probability would not be relevant to them. However, because it would be inefficient to derive a prediction model in a low prevalence population only (since most patients will not undergo the reference standard), we also included databases with patients referred for catheter based coronary angiography. These data provided valuable information on the correlations between clinical presentation, risk factors, and the two angiography procedures, which was essential for reliable imputation of covariables and outcomes in the low prevalence populations. By including the setting variable, we could derive the model using all available data, and adjust for differences in patient selection. When applying the model for new patients with chest pain, the setting variable was set to zero.

Predictor effects might differ across the low and high prevalence settings, and we tested these differences by using interaction terms between setting and all other variables. We also tested interactions between symptoms and sex, symptoms and age, and symptoms and diabetes. Linear effects of age and the log transformed coronary calcium score were tested by including a restricted cubic spline function with three knots (df=2).30 31

We quantified diagnostic performance by calculating the area under the receiver operating characteristics curve (c statistic). Reclassification was assessed by use of the continuous net reclassification improvement (web appendix table 2).32 We regarded P<0.05 to be statistically significant. Analyses were performed using Stata/SE 11 (StataCorp).

Validation

We assessed the validity of the clinical model in a cross validation procedure. The four largest low prevalence databases with sufficient numbers for reliable validation,33 and the remaining low prevalence databases combined, were each in turn removed from the model development sample. We then validated each model using the database that was omitted during model development. We calculated the c statistic and validated the model according to the steps in the box (the web appendix provides more detail).26 30 31

Box: Cross validation procedure

Calibration-in-the-large

When assessing the validity of a prediction model, the first step is to check whether the average prediction approximates the average observed outcome. This concept is referred to as “calibration-in-the-large”. In this study, we compared the mean observed frequency of coronary artery disease in the validation data with the mean predicted probability. A negative value would imply that the mean observed proportion in the validation data is lower than the mean predicted probability. Conversely, a positive value would indicate that the mean observed proportion in the validation data was higher than the mean predicted probability. A statistically significant result indicates significant miscalibration, whereas a non-significant result supports validity of the prediction model

Logistic recalibration

The second step is to test whether the overall effect of the predictors in the model is valid for the validation data. The overall predictor effects in the validation data were re-estimated and compared with the reference overall predictor effects in the prediction model (reference is set to zero). A negative value would imply that the overall effects of the predictors in the validation data were lower than those in the prediction model. Conversely, a positive value would indicate that the overall effects of the predictors in the validation data were stronger than those in the prediction model. A significant result would indicate significant miscalibration of the predictor effects, whereas a non-significant result would indicate no difference in predictor effects, supporting the validity of the model

Re-estimation

The third step is to re-estimate the predictor effects in the validation data and calculate the difference (∂ coefficients) with the estimated predictor effects in the prediction model (web appendix).26 Non-significant differences (high P values) indicate no difference in the predictor effect, supporting the validity of the model

Results

Data collection and study population

We retrieved databases from 18 hospitals (table 1 and web appendix table 2). The study population included 5677 patients (3283 men, 2394 women; mean age 58 and 60 years, respectively). Nearly all patients (5190, 91%) underwent CT based coronary angiography, which revealed obstructive coronary artery disease in 1634 (31%). Of these 1634 patients, 1083 (66%) underwent catheter based coronary angiography, which showed positive results in 886 (82%). Of the 3556 patients without obstructive disease on CT based coronary angiography, 526 (15%) underwent catheter based coronary angiography, which showed negative results in 498 (95%). Overall, 2062 (36%) patients underwent catheter based coronary angiography, with 1176 (57%) diagnosed with obstructive coronary artery disease. Missing values occurred in four (0.1%) patients for age, six (0.1%) for symptoms, 126 (2.2%) for hypertension, 189 (3.3%) for diabetes, 187 (3.3%) for dyslipidaemia, 155 (2.7%) for smoking, 354 (6.2%) for body mass index, and 810 (14%) for coronary calcium score.

Of the 3556 patients who did not have obstructive coronary artery disease revealed by CT based coronary angiography, 3030 (85%) did not undergo the catheter based procedure. Results for catheter based coronary angiography were imputed for these patients, and were mostly negative (range 97-98.4% across the multiple imputations), which accords with the high negative predictive value of the CT based procedure. Of the 1634 patients who had obstructive disease revealed by the CT based procedure, 551 (34%) did not undergo subsequent catheter based coronary angiography. For these patients, results for the catheter based procedure were imputed, and were mostly positive (range 65-77% across imputations), which accords with a reduced positive predictive value of the CT based procedure.

External validation of the Duke clinical score overestimated the probability of severe coronary artery disease, as observed in our dataset (fig 1 and web appendix table 3).

Fig 1 Calibration plot of the Duke clinical score, in low prevalence datasets (n=4426). Distribution of predicted probabilities shown separately for patients with and without severe coronary artery disease. Triangles indicate observed proportions of severe disease, by tenths of predicted probability; 95% CI=confidence interval

Table 2 summarises the results of the random effects analysis in logistic regression and the continuous net reclassification improvement (web appendix table 4 provides more detail). The prediction models are available as an online probability calculator (http://rcc.simpal.com/NpfpV5; web appendix fig A2). In the clinical model, all predictors except body mass index were significantly associated with obstructive coronary artery disease. The clinical model improved the prediction, compared with the basic model (cross validated c statistic improved from 0.77 to 0.79; table 2). Whereas an abnormal exercise electrocardiography had limited predictive value in the multivariable prediction model (web appendix table 5), the coronary calcium score was a major predictor, which increased the c statistic from 0.79 to 0.88 (table 2). Most predictor effects decreased after addition of the coronary calcium score; dyslipidaemia and smoking were no longer significant. We obtained similar results when using CT based coronary angiography as the outcome in patients who did not undergo catheter based coronary angiography.

Table 2

 Random effects logistic regression analysis* and cross validation, in the low prevalence setting

	Prediction model
	Basic	Clinical	Extended
Age (per 10 years)	1.89 (1.74 to 2.04)	1.85 (1.70 to 2.02)	1.11 (0.99 to 1.25)
Male sex	3.89 (3.24 to 4.66)	3.79 (3.13 to 4.58)	2.19 (1.75 to 2.75)
Chest pain (v non-specific chest pain)
Atypical	1.93 (1.48 to 2.52)	1.88 (1.44 to 2.46)	2.05 (1.50 to 2.80)
Typical, if diabetes is absent	7.21 (5.64 to 9.22)†	7.36 (5.64 to 9.61)	7.57 (5.56 to 10.3)
Typical, if diabetes is present	—	4.91 (3.16 to 7.63)	3.46 (2.12 to 5.63)
Diabetes	—	2.29 (1.72 to 3.04)	1.93 (1.41 to 2.65)
Hypertension	—	1.40 (1.18 to 1.67)	1.26 (1.04 to 1.54)
Dyslipidaemia	—	1.53 (1.25 to 1.86)	1.20 (0.95 to 1.53)
Smoking	—	1.59 (1.30 to 1.93)	1.23 (0.97 to 1.55)
Coronary calcium score
Log transformed (per standard deviation)	—	—	4.69 (3.76 to 5.84)
0-<10‡	—	—	2.23 (1.34 to 3.74)
≥10-<100‡	—	—	5.04 (3.38 to 7.52)
≥100-<400‡	—	—	15.3 (9.96 to 23.5)
≥400‡	—	—	35.9 (22.6 to 56.9)
Cross validation (mean§)
C statistic	0.77	0.79	0.88
Net reclassification improvement (%)¶	—	35	102

Data are odds ratio (95% confidence interval) unless stated otherwise. All odds ratios showed significant associations (P<0.05) apart from age, dyslipidaemia, and smoking in the extended model.

*Random effect for hospital included to account for clustering of patients within hospitals. Body mass index omitted from all analyses because odds ratio was less than 1.01 and was non-significant. Setting and the interaction between diabetes and typical chest pain, and between setting and coronary calcium score were predictive and were included in all models. All other interactions were not significant. Test for a non-linear effect of age was not significant. Evidence indicated additional non-linear effect of coronary calcium score beyond the log transformation, which was considered not clinically important, and omitted for simplicity.

†Irrespective of diabetic status, since basic model does not include diabetes.

‡Separate analysis using coronary calcium score as a categorical variable, adjusted for all other predictors in the model, reference category is score 0.

§Mean of the cross validation procedures calculated using the four largest low prevalence datasets and remaining low prevalence datasets combined.

¶Calculated by comparison with the next model on the left; defined as weighted sum of the increase in observed proportion among patients whose predicted probability goes up, and decrease in observed proportion among those whose predicted probability goes down (web appendix).29

Figure 2 and web appendix table 6 show the cross validation results for the clinical model. The c statistic ranged from 0.78 to 0.81. The continuous net reclassification improvement was a measure of the relative change in the observed proportion when the predicted probability changes (web appendix), which was most favourable (102%) for the extended model compared with the clinical model (table 2).

Fig 2 Validity of clinical model using the four largest low prevalence datasets and the smaller remaining low prevalence databases combined. Distribution of predicted probabilities shown separately for patients with and without obstructive coronary artery disease. Triangles indicate observed proportion of disease, by tenths of the predicted probability; 95% CI=95% confidence interval

Assessment of calibration-in-the-large showed a significant difference between the mean observed outcome and the predicted probability (clinical model), for Azienda Ospedaliero Universitaria Parma, Rotterdam, and the combined low prevalence hospitals (fig 2). Logistic recalibration showed no significant differences between the overall hospital specific effects of the predictors compared with the overall effects of the predictors in the clinical model (fig 2). When re-estimated in specific datasets, the predictor effects were not significantly different from the predictor effects in the clinical model, except for the effect of typical chest pain for Azienda Ospedaliero Universitaria Parma. The results indicated that predictor effects were similar across datasets (web appendix table 6).

Discussion

Summary of key findings

With recently collected data and modern statistical methods, we developed a prediction model that performed well in estimating the probability of coronary artery disease. The need for an updated model was evident by our results showing that the Duke clinical score significantly overestimated the probability of coronary artery disease. Age, sex, symptoms, and coronary calcium score were strong predictors for disease. The clinical model predicted probabilities between 2% for a 50 year old woman with non-specific chest pain without any risk factors, and 91% for an 80 year old man with typical chest pain and multiple risk factors.

Previous publications

In 1979, Diamond and Forrester11 showed the importance of age, sex, and symptoms in the prediction of coronary artery disease. Despite its limitations, current guidelines still recommend the use of Diamond and Forrester’s model.7 8 As previously shown, the model tends to overestimate the probability of coronary artery disease, mainly in women,14 and does not take into account cardiovascular risk factors associated with the presence of the disease.

In 1993, Pryor and colleagues12 13 developed the Duke clinical score in a large cohort of patients who underwent catheter based coronary angiography. The model predicted the presence of at least 75% stenosis, on the basis of age, sex, symptoms, history of myocardial infarction, smoking, dyslipidaemia, diabetes, and resting electrocardiography findings. In our study, we showed that the Duke clinical score also overestimated the probability of coronary artery disease.

Limitations of the study

The study population was derived from existing databases, some of which were designed for other research objectives (such as to investigate the diagnostic accuracy of CT based coronary angiography for coronary artery disease). In some studies, all patients underwent the reference standard test, whereas other studies selected patients for catheter based coronary angiography on the basis of results from the CT based procedure. When evaluating the diagnostic performance of CT based coronary angiography, selection on the basis of the test results could lead to verification bias or selective referral bias, which could affect the sensitivity and specificity estimates of such a test. However, in the current study, we did not assess the diagnostic value of the CT based procedure, thus making bias less likely. We also considered patients who underwent the CT based procedure only and not the catheter based procedure. We selected patients who underwent either angiography, or both procedures. However, not all patients presenting with chest pain will be referred for one procedure or the other, which limits the generalisability of our results.

To also consider patients who did not undergo catheter based coronary angiography, we multiply imputed these results on the basis of results from the CT based procedure and all other covariables. Although data for the catheter based procedure were imputed for a large proportion of patients, we believe that the high sensitivity and specificity of the CT based procedure justifies the imputation; we also did not consider the CT based procedure as an explanatory variable in the prediction models. However, the CT based procedure could overestimate the severity of disease, which in turn could have overestimated the imputed severity of disease from the catheter based procedure. If bias was present in our models, it would tend to advocate further diagnostic investigation rather than missing the diagnosis.

We combined existing data from several different hospitals. Since the current analysis was not the main purpose of the data collection, the selection of patients, availability of data, and predictor definitions differed across hospitals (web appendix table 1). Furthermore, some hospitals used quantitative coronary angiography to determine the degree of stenosis, whereas others used visual assessment. Overall, heterogeneity due to differences between protocols, level of physician experience, and guideline adherence across hospitals could have influenced our results. Despite these limitations, the models presented had generally good discrimination (via the c statistic) and calibration.

Because we intended to use our model in low prevalence populations, cross validation was performed using only the low prevalence datasets. The cross validation results in the data from Parma, Rotterdam, and the smaller hospitals combined, were less favourable in terms of calibration-in-the-large, possibly explained by heterogeneity. However, in general, calibration assessed graphically (fig 2) could be considered satisfactory, suggesting that the model is generalisable to other settings. Further external validation of our model in other populations is still needed.

Our study focused on the prediction of obstructive coronary artery disease according to age, sex, type of chest pain, cardiovascular risk factors, setting, and coronary calcium score. Unfortunately, no data were available to assess the predictive value of findings based on other imaging tests (such as nuclear perfusion imaging, perfusion magnetic resonance imaging, and echocardiography). Furthermore, we were unable to show incremental predictive value of exercise electrocardiography, which may be explained by the limited availability of data.

Finally, our analysis focused on the diagnostic prediction of the presence of coronary artery disease, defined as at least 50% diameter stenosis in at least one vessel, shown on catheter based coronary angiography. Predicting severe disease (for example, ≥70% stenosis in the left anterior descending coronary artery, three vessel disease, or left main coronary artery disease) would be helpful as a tool to select patients for revascularisation. However, the main purpose of the current analysis was the development of prediction models to help physicians select patients who would benefit from further testing.

Future directions

Other non-invasive imaging tests for the evaluation of patients with chest pain should also be considered as predictors for the presence of coronary artery disease. Furthermore, cost effectiveness analyses should be performed to establish appropriate thresholds for diagnostic testing, taking into account the long term benefits and harms.4 34

Clinical implications

We showed that the Duke clinical score overestimates the likelihood of coronary artery disease, and we believe that our model improves the estimate of the pretest probability. By contrast with the Duke clinical score, we developed and validated our model using data from different hospitals, settings, countries, and included hypertension as a predictor. Finally, our model does not need inclusion of resting electrocardiography findings, which could be convenient in primary practice.

A refined estimate of the probability of coronary artery disease allows doctors to make better decisions as to which diagnostic test is best in a particular patient, according to NICE guidelines, and to decide on further management based on the results of such tests. Our stepwise models can be used to evaluate the added value of performing coronary calcium score either before performing the test or after obtaining the calcium score. Our analysis shows that the coronary calcium score significantly improves the estimate of the probability of coronary artery disease, suggesting that the score should be considered for patients with chest pain. Use of the coronary calcium score is not routinely recommended by the guidelines from the American College of Cardiology/American Heart Association35 or the European Society of Cardiology,9 whereas the UK guidelines recommend use of the score if the pretest probability is 10-29%.10 However, triage strategies using the coronary calcium score have been proposed35 and are implemented in several centres. In this context, our prediction calculator could be used both to determine whether using the coronary calcium score is clinically useful (by checking whether the score alters the probability of coronary artery disease such that clinical management would change) and to determine the revised probability of coronary artery disease, based on the score’s result.

Our findings also suggest that the diagnostic value of exercise electrocardiography is limited (web appendix), which accords with its low sensitivity and specificity for detecting coronary artery disease,36 and with explicit recommendations in the UK not to use the test to diagnose or exclude disease in patients with chest pain.10 However, many physicians argue that the prognostic information obtained by exercise electrocardiography remains important for clinical practice. Alternatively, coronary calcium scores provide both diagnostic37 38 39 and prognostic information.40 41 42 43

Prediction tools are useful only when they are easily accessible at the point of care, which is why we designed an online calculator (web appendix fig A2). The calculator could be implemented in electronic patient records, electronic order entry systems, or smartphone or tablet applications. Overall, prediction models that include age, sex, symptoms, and risk factors allow for accurate estimation of the probability of coronary artery disease in low prevalence populations. The addition of using the coronary calcium score improved the prediction of the disease. Implementation of these models could improve clinical outcomes, but would need further evaluation.

Current guidelines recommend use of the Diamond and Forrester model or Duke clinical score to estimate the pretest probability of coronary artery disease in patients presenting with stable chest pain

The pretest probability of disease helps physicians identify which patients are most likely to benefit from further testing

The Duke clinical score overestimates the probability of coronary artery disease

An updated prediction model for low prevalence populations was developed and validated using individual patient data from 18 different hospitals

Use of the CT based coronary calcium score in the updated model improved the predicted probability of disease