Development and validation of a prediction rule for patients suspected of acute coronary syndrome in primary care: a cross-sectional study.

OBJECTIVE: To develop and validate a symptom-based prediction rule for early recognition of acute coronary syndrome (ACS) in patients with acute chest discomfort who call out-of-hours services for primary care (OHS-PC).
DESIGN: Cross-sectional study. A diagnostic prediction rule was developed with multivariable regression analyses. All models were validated with internal-external cross validation within seven OHS-PC locations. Both age and sex were analysed as statistical interaction terms, applying for age non-linear effects.
SETTING: Seven OHS-PC in the Netherlands. PARTICIPANTS: 2192 patients who called OHS-PC for acute chest discomfort (pain, pressure, tightness or discomfort) between 2014 and 2017. Backed up recordings of telephone triage conversations were analysed. PRIMARY AND SECONDARY OUTCOMES MEASURES: Diagnosis of ACS retrieved from the patient's medical records in general practice, including hospital specialists discharge letters. Performance of the prediction rules was calculated with the c-statistic and the final model was chosen based on net benefit analyses.
RESULTS: Among the 2192 patients who called the OHS-PC with acute chest discomfort, 8.3% females and 15.3% males had an ACS. The final diagnostic model included seven predictors (sex, age, acute onset of chest pain lasting less than 12 hours, a pressing/heavy character of the pain, radiation of the pain, sweating and calling at night). It had an adjusted c-statistic of 0.77 (95% CI 0.74 to 0.79) with good calibration.
CONCLUSION: The final prediction model for ACS has good discrimination and calibration and shows promise for replacing the existing telephone triage rules for patients with acute chest discomfort in general practice and OHS-PC. TRIAL REGISTRATION NUMBER: NTR7331. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

We could analyse the original and very first telephone conversation of patients with acute chest discomfort.

The developed prediction model can be well generalised to other out-of-hours services for primary care (OHS-PC) locations in the Netherlands, but also to similar OHS-PC settings in other countries or even emergency medical service settings.

A limitation is that a full external validation of the model in another OHS-PC was impossible because no other cohort similar to ours was available.

Another limitation is that the effects of the predictors were assumed to be similar for males and females while this is not exactly the case, but by incorporating a differential non-linear effect of age and interaction with sex in the analyses this effect is minimalised.

Introduction

Chest discomfort is among the top five reasons for telephone contact in out-of-hours services for primary care (OHS-PC) and concerns 5% of all cases at the emergency department (ED) in the USA.1 In the Netherlands, around 80% of patients with chest discomfort first call the general practitioner (GP) or OHS-PC, while 20% directly calls the emergency medical service (EMS, or ambulance dispatch centre) or are self-referrals to the ED.2 Adequate triage and early diagnosis in these patients is vital, because in case of an underlying acute coronary syndrome (ACS) early effective therapeutic interventions (‘time is muscle’) improve the patient’s outcome and prognosis.3 For the diagnosis of ACS, a 12-lead ECG and troponin testing is needed.3 However, before the patient is referred to an ED where these diagnostic tests can be done, patient selection is necessary based on symptom presentation retrieved by telephone triage.4 5 Symptom-based differentiation of ACS from other causes of chest discomfort is notoriously difficult.6 Symptom-based prediction rules for diagnosing ACS in general practice and other prehospital settings are—although highly needed—scarce.7–9 The efficiency and safety of telephone triage in OHS-PC was poor in a population with a prior chance of ACS of 8.3% in females and 15.3% in males; almost 50% of the males and females with chest discomfort received a high priority ambulance, while 11% diagnosed with an ACS did not receive a high urgency (ie, was seen within 1 hour).10

Most prediction rules for diagnosing ACS were developed in the ED setting and include results from ECG and troponin testing.11 Such prediction rules cannot be straightforward implemented for telephone triage in general practice because (1) in the latter setting these diagnostic tests are not available, (2) the prior chance of ACS is rather low and (3) on average disease severity is less than in those seen in the ED.12 13 The prevalence of ACS among patients with chest discomfort who call OHS-PC or EMS is about 10%–15%, and among those seen at the ED between 10% and 30%.10 11 13 14 Only one prediction rule was developed in primary care for diagnosing ACS; the modified Grijseels prediction rule, which had moderate discriminative ability (c-statistic of 0.66) after external validation.7 8 Five other primary care prediction rules were developed to predict CAD; for example, the Marburg Heart Score (MHS) and INTERCHEST prediction rule (International Working Group on Chest Pain in Primary Care).9 In these studies both patients with acute and non-acute chest discomfort were included and the prevalence of stable CAD showed to be 10.9%–12.6%, while that of ACS was only 1.5%–2.5%.9 Thus, these prediction rules have limited applicability for specifically diagnosing ACS.

In most OHS-PC and about half of the EMS in the Netherlands, triage nurses use the semi-automatic ‘Netherlands Triage Standard’ (NTS) as a decision support tool to classify the urgency of the patient’s condition. Triage nurses have to choose one out of 56 ‘main complaints’ and based on answers linked to the triage criteria, the NTS automatically proposes one out of six levels of urgency, that is, a certain time frame in which patients should be seen (U0-U5, online supplemental appendix table 1). The NTS is a modified and shortened version of the Manchester Triage Standard which was developed in the ED setting.15 Although the NTS was explicitly developed for telephone triage, it has not yet been validated against clinical outcomes even though it is already implemented on large scale. Recent research showed that the NTS had a poor sensitivity of 0.73 (95% CI 0.68 to 0.78) for telephone triage of patients with acute chest discomfort. The NTS recommended a low urgency (U3, U4 or U5) to 27% of patients who eventually showed to have an ACS or other life threatening event (LTE).10 The NTS’ specificity was also poor with 0.43 (95% CI 0.40 to 0.45); the NTS recommended a high urgency to 57% of the patients who eventually showed not to have an ACS or LTE. Given this poor safety and efficiency of the NTS, there is an urgent need for a better prediction model for patients with acute chest discomfort calling OHS-PC. In addition, there is a need for exploring sex-specificity of such a prediction rule as there is an ongoing debate on whether females differ from males in reporting symptoms of ACS.16 17

The aim of this study was to develop, and internal-external cross validate a symptom-based prediction rule for diagnosing ACS which is considerate to sex categories in male and female patients who call the OHS-PC for acute chest discomfort.

Methods

We performed a cross-sectional study among 2192 patients who called one out of seven participating OHS-PC in the Netherlands because of acute chest discomfort (pain, pressure, tightness or discomfort) during the period 2014–2017.18 These OHS-PC serve a total population of 1.5 million people and cover around 300 000 calls a year. We first selected calls based on the International Code for Primary Care (ICPC; a WHO world-wide code system for primary care) with ICPC-codes K01, K02, K03, K24, K74, K75, K76, K77, K93, L04, P74, R02, R98 and calls with keywords thoracic pain, chest pain, myocardial infarction, heart attack and their common abbreviations (figure 1). We included a broad variety of symptoms to capture the entire domain of patients that could be suspected of ACS. We listed all available calls of these patients and assigned random numbers with the Random Number Generator (RAND) function in Microsoft Excel to retrieve a random sample. Calls were excluded before relistening when the patients’ age was below 18 years or when the patient did not live in the surrounding area of the OHS-PC (in which case we could not retrieve the final diagnosis from the patient’s own GP). Calls were excluded during relistening when it did not concern a triage call (eg, intercollegial consultation) or when the recording was of poor quality (figure 1).

Figure 1

Flowchart of study population. ACS, acute coronary syndrome; ICPC, International Code for Primary Care; OHS-PC, out-of-hours services for primary care.

Candidate predictors

Research team members (LTCMW, DCAE) and medical students listened to the call recordings, blinded for the outcome, to collect data about symptoms, medical history and urgency allocation. Patient (age, sex) and call characteristics (call time, call duration) were collected from the OHS-PC electronic medical files of the patients. As candidate predictors we included age and sex, the NTS triage criteria (see online supplemental appendix table 2), the ACS predictors from the modified Grijseels prediction model (male sex, radiation, nausea, sweating and history of CAD), the ‘CAD predictors’ from MHS and INTERCHEST prediction models (age, pain feels like pressure, CAD history or cardiovascular (CV) risk factors, patient assumes cardiac origin of pain), and—based on a recent own study in OHS-PC—the predictor ‘calling at night between 00:00 and 09:00’.7–9 19

Outcome

The primary outcome was the diagnosis ACS. The final diagnoses were retrieved from the patient’s GP and based on the GP’s electronic medical files which include ED and cardiologist discharge letters and notes from the OHS-PC contact. The diagnosis ACS was nearly always made by a cardiologist (96.0%) and included information on levels of (high-sensitivity) troponin and electrocardiographic results. We used medical information up to 30 days following the contact with the OHS-PC to allow us to include diagnoses of ACS that were initially missed because the patient was not referred to the cardiologist the same day of the OHS-PC contact. However, in none of the patients in the study we had evidence of a missed diagnosis of ACS.

Sample size calculation

We relied on the minimal sample size criteria for prediction model development proposed by Riley et al using the ‘pmsampsize’ package in R.20 Based on an ACS prevalence of overall 11% and a Cox-Snell R2 of 0.075 (a conservative value based on a model with age and sex) and a total number of 2192 observations we were allowed to assess 19 candidate predictors.21 Based on sample size calculations we concluded that development of separate models for males and females would require a significantly larger sample, therefore we analysed sex as a statistical interaction term.20

Statistical analyses

We developed three diagnostic models using multivariable logistic regression analysis. First, we developed a base model with only age and sex as predictors, where age was modelled using a restricted cubic spline function (four knots) and an interaction with sex. This resulted in a base model with seven predictor parameters (excluding the intercept). Second, we fitted a full model with an additional 12 preselected binary predictors (having chest pain, acute chest pain shorter than 12 hours, shortness of breath, sweating, retrosternal located chest pain, radiation, pressing heavy pain, stabbing pain, history of cardiovascular disease (CVD), history of CAD, someone else calling, calling at night). Third, we applied backward elimination, with a cut-off p value <0.20 for including predictors (a higher cut-off value to lower the chances of overfitting).21

We applied internal-external cross validation (IECV) for model validation using the seven different OHS-PC locations (online supplemental appendix table 3 for patient characteristics of different OHS-PC locations).22 We evaluated the IECV performance in terms of the area under the receiver operating characteristic curve (c-statistic), the calibration slope and calibration in the large. The IECV estimates of performance were combined by using random-effect model (DerSimonian-Laird estimator). Based on the IECV we also constructed flexible calibration curves and decision curves. In the decision curve analyses we compared the final model with the currently used NTS triage model in OHS-PC in the Netherlands.23 Finally, we created an illustrative table of diagnostic test accuracy for various model-based risk thresholds of the final model, following the example in Wynants et al.23 IECV estimates for risk threshold specific sensitivity and specificity, and we applied a bivariate model commonly used for diagnostic test accuracy meta-analysis.24

Missing data

For missing data we carried out multiple imputation using the Multivariate Imputation via Chained Equation package in R, with 30 imputation rounds and 30 iterations.25 We pooled the results following Rubin’s rules.26 Predictors with over 50% missing were excluded from consideration in the models (online supplemental appendix table 4 for details about the missing data). Characteristics were compared between patients with and without information on the medical outcome—because some GPs refused diagnosis retrieval from their files—to allow for assessment of differences in characteristics between these patient groups (online supplemental appendix table 5). There were no clinically meaningful differences in symptoms and patient or call characteristics between the 2192 patients with information on the outcome, and the 1012 patients about whom knowledge of the medical outcome related to the OHS-PC contact because of acute chest discomfort was missing.

All analyses were done in R V.4.0.3. (10 October 2020) with the Regression Modelling Strategies (‘rms’) package in R.27 We reported our study in accordance to the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis criteria (online supplemental file).28

Patient and public involvement

No patients were involved in defining the research question or the outcome measures. Neither they were involved in developing plans for design. However, they participated in the discussion on implications and the implementation strategy. In addition, they were asked to advise on interpretation and writing up of the results. Results will be shared and discussed in more detail with representatives of the Dutch national patient community of cardiovascular diseases (‘Harteraad’).

Results

Among the 2192 callers with acute chest discomfort (mean age 59.1 (SD 19.5) years and 55.3% females) 251 (11.5%) had a final diagnosis of ACS; 101 (8.3%) females and 150 (15.3%) males (table 1). Patients with ACS were older than those without (mean age 69.7 (SD 13.4) vs 57.7 (SD 19.8) years) and females with ACS were on average older than men with ACS (73.8 (SD 13.5) years vs 67.0 (SD 12.6) years).

Table 1

Characteristics of 2192 patients who called OHS-PC with acute chest discomfort between 2014 and 2017, divided between females and males with and without ACS

Characteristics	1213 females (55.3%)		979 males (44.7%)
	ACS, n=101 (8.3%)	No ACS, n=1112 (91.7%)	ACS, n=150 (15.3%)	No ACS, n=829 (84.7%)
Patient characteristics
Mean age in years (SD) (n=2192)	73.8 (13.5)	58.0 (20.2)	67.0 (12.6)	57.2 (19.2)
Call characteristics
Median call duration in min (IQR) (n=2192)	5:27 (3:57–8:24)	6:59 (5:06–9:47)	6:04 (4:03–8:17)	6:56 (5:10–9:23)
Mean introduction time in min (IQR) (n=2192)	0:13 (0:09–0:18)	0:17 (0:11–0:26)	0:14 (0:09–0:21)	0:17 (0:11–0:25)
Call during the night (00:00–09:00) (n=2192)	34 (33.7)	304 (27.3)	62 (41.3)	188 (22.7)
Triage nurse consulted the GP (n=2192)	43 (42.6)	580 (52.2)	75 (50.0)	449 (54.2)
Someone else called on behalf of patient (n=2192)	69 (68.3)	515 (46.3)	98 (65.3)	432 (52.1)
The person who calls expressed concerns (n=988)	42 (95.5)	507 (90.5)	61 (96.8)	378 (87.1)
Medical history and risk factors
Cardiovascular disease or CV risk factors (n=1844)	70 (81.4)	552 (61.1)	106 (78.5)	464 (64.4)
History of coronary artery disease (n=1151)	23 (47.9)	131 (24.2)	54 (56.3)	181 (38.8)
Diabetes (n=893)	14 (42.4)	66 (14.3)	22 (39.3)	78 (22.0)
Hypertension (n=894)	26 (72.2)	162 (34.0)	22 (51.2)	113 (33.3)
Hypercholesterolaemia/statin use (n=826)	10 (40.0)	96 (22.6)	27 (50.0)	79 (24.5)
Cardiac arrhythmia (n=905)	4 (14.8)	125 (26.2)	12 (25.0)	89 (25.2)
Symptoms
Chest pain (n=2116)	95 (96.9)	1007 (94.1)	139 (93.3)	758 (94.9)
Shortness of breath (n=1696)	57 (71.3)	559 (65.4)	63 (61.2)	415 (63.1)
Chest pain duration <12 hours (n=1910)	74 (86.0)	703 (72.3)	113 (82.5)	510 (71.3)
Severe pain (Numeric Rating Scale>7, range 1–10) (n=917)	19 (61.3)	184 (39.6)	18 (25.4)	116 (33.0)
Pressing/heavy chest pain* (n=1625)	58 (81.7)	525 (62.5)	95 (81.2)	345 (57.7)
Stabbing chest pain* (n=1625)	8 (11.3)	190 (22.6)	9 (7.7)	159 (26.6)
Retrosternal chest pain † (n=1565)	36 (54.5)	326 (40.0)	52 (53.1)	227 (38.7)
Chest pain left or right of thorax† (n=1566)	19 (28.8)	318 (39.0)	28 (28.6)	262 (44.6)
Radiation of chest pain to any location (n=1678)	74 (86.0)	575 (67.8)	83 (65.4)	347 (56.2)
Radiation to the arm‡ (n=1677)	37 (43.0)	218 (25.7)	54 (42.5)	143 (23.2)
Radiation to the shoulder blades‡ (n=1678)	14 (16.3)	190 (22.4)	19 (15.0)	103 (16.7)
Radiation to the jaws‡ (n=1678)	10 (11.6)	77 (9.1)	4 (3.1)	33 (5.3)
Sweating (n=1366)	36 (52.9)	279 (42.0)	54 (51.9)	190 (35.8)
Nausea or vomiting (n=987)	24 (52.2)	295 (56.6)	31 (43.1)	139 (39.9)
Pallor or ashen skin (n=673)	22 (59.5)	139 (44.3)	36 (64.3)	125 (46.8)
(Near) fainting (n=1951)	8 (9.5)	76 (7.7)	9 (6.7)	50 (6.7)
Palpitations (n=162)	10 (100.0)	183 (84.7)	8 (50.0)	83 (75.5)
Patient recognises symptoms from previous cardiac event (n=915)	17 (35.4)	100 (22.0)	30 (46.9)	103 (29.5)
Urgency allocation
High urgency (U1 or U2) (n=2192)	89 (88.1)	740 (66.5)	133 (88.7)	534 (64.4)
U1	75 (74.3)	443 (39.8)	106 (70.7)	350 (42.2)
U2	14 (13.9)	297 (26.7)	27 (18.0)	184 (22.2)
Low urgency (U3 or U4 or U5)	12 (11.9)	372 (33.5)	17 (11.3)	295 (35.6)

*Pain described by patient. Pressing heavy pain: pressing, heavy or tightening pain vs other types of pain (stabbing, burning, cramping, tearing). Stabbing pain: stabbing versus other types of pain (pressing, heavy, tightening, burning, cramping).

†Retrosternal location versus other pain locations. Left or right side of the thorax versus other pain locations.

‡Radiation location versus no radiation and radiation other pain.

ACS, acute coronary syndrome; CV, cardiovascular; GP, general practitioner; OHS-PC, out-of-hours services for primary care.

Over two-thirds of all callers (68.3%) received a high urgency allocation (seen within 1 hour; U1 or U2) and among the 251 patients who showed to have an ACS, 88.4% received a high urgency allocation. Calls of patients who had an ACS were shorter than calls in those without ACS (median call duration 6:34 (SD 3:38) vs 7:42 (SD 3:48) min).

Medical history and symptoms

Females and males with ACS had more often a history of CVD or CV risk factors than those without (females with ACS 81.4% vs females without ACS 61.1%, males with ACS 78.5% vs males without ACS 64.4%) (table 1). The majority of both females and males had chest pain (94.5%) and this was similar among those with and without ACS. Overall, presented symptoms among males and females calling the OHS-PC for chest discomfort were quite similar. Symptoms associated with ACS in both sexes were pressing/heavy chest pain (females with ACS 81.7% vs females without ACS 62.5%, males 81.2% vs 57.7%, respectively), retrosternal located chest pain (females 54.5% vs 40.0%, males 53.1% vs 38.7%), radiation of pain (females 86.0% vs 67.8%, males 65.4% vs 56.2%) and sweating (females 52.9% vs 42.0%, males 51.9% vs 35.8%).

Diagnoses

In total 251 patients were diagnosed with ACS and 65 with other LTEs, and of clinical relevance is that both critical events occurred significantly more in males than females (15.3% vs 8.3%, p<0.001 for ACS, and 3.8% vs 2.3%, p=0.04 for other LTEs, respectively). Of the 101 females with ACS, 22.8% had an ST-segment elevation myocardial infarction (STEMI), 46.5% a non-ST-segment elevation myocardial infarction (NSTEMI), 19.8% unstable angina pectoris (UAP) and 10.9% non-classified ACS. In 150 males with ACS, 33.3% had an STEMI, 36.7% an NSTEMI, 26.0% UAP and 4.0% non-classified ACS (table 2).

Table 2

Diagnoses of 2192 males and females who contacted the OHS-PC for acute chest discomfort between 2014 and 2017, by sex

Diagnosis, n (%)	Females n=1213	Males n=979	P value
Acute coronary syndrome*	101 (8.3)	150 (15.3)	<0.001
STEMI	23 (22.8)	50 (33.3)	0.071
NSTEMI	47 (46.5)	55 (36.7)	0.119
UAP	20 (19.8)	39 (26.0)	0.256
Non-classified ACS	11 (10.9)	6 (4.0)	0.033
Life threatening events (LTEs)	28 (2.3)	37 (3.8)	0.043
Pulmonary embolism	8 (28.6)	10 (27.0)	0.890
Acute abdominal aneurysm	2 (7.1)	3 (8.1)	0.885
Thoracic aortic dissection	1 (3.6)	4 (10.8)	0.278
Other†	17 (60.7)	20 (54.1)	0.591
Non-urgent cardiovascular diseases‡	223 (18.4)	191 (19.5)	0.069
Musculoskeletal pain	245 (20.2)	148 (15.2)	0.039
Non-cardiac chest pain, not further specified*§	191 (15.7)	179 (18.3)	0.012
Psychogenic disorders	165 (13.6)	85 (8.7)	0.005
Gastrointestinal tract disorders	89 (7.3)	68 (6.9)	0.776
Respiratory tract disorders	61 (5.0)	56 (5.7)	0.203
Other non-urgent diagnoses¶	110 (9.1)	65 (6.6)	0.152

*Almost all patients (96.0%) were diagnosed by a cardiologist. Ten (4.0%) ACS patients were not diagnosed by a cardiologist; four died before arrival of the ambulance, one patient died after resuscitation at the ED (all these five were classified as acute cardiac death due to ACS), and in five patients the ACS diagnosis was solely based on the GP’s interpretation in patients who were not referred to the hospital after shared decision because of a short life expectancy due to cancer in a palliative stage.

†Stroke, severe COPD exacerbation, acute severe heart failure, sepsis, hypokalaemia, diabetic ketoacidosis, epileptic insult, bleeding from oesophageal varices, ovarian torsion, ventricular fibrillation.

‡Stable angina pectoris (including atypical chest pain), stable heart failure, arrhythmias, hypertension.

§Cardiac pathology unlikely after cardiologist’s or GP’s diagnostic work-up, but without differential diagnosis.

¶Among others: anaemia, carcinoma, vasovagal collapse, side effects medication, dermatological diseases (eg, herpes zoster infection).

ACS, acute coronary syndrome; COPD, chronic obstructive pulmonary disease; ED, emergency department; GP, general practitioner; NSTEMI, non-ST-segment elevation myocardial infarction; OHS-PC, out-of-hours services for primary care; STEMI, ST-segment elevation myocardial infarction; UAP, unstable angina pectoris.

Twenty-eight (2.3%) females and 37 males (3.8%) had another life-threatening event than ACS (eg, pulmonary embolism, thoracic aortic dissection, acute abdominal aneurysm). All other patients (85.6%) had non-urgent medical conditions such as non-urgent cardiovascular disease (18.9%), musculoskeletal pain (17.9%), non-cardiac chest pain (not further specified) (16.9%), psychogenic disorder (11.4%), gastrointestinal disorders (7.2%), respiratory disorders (5.3%) and other non-urgent diagnoses (8.0%).

Model development, performance and validation

The base model with sex and age had an apparent c-statistic of 0.72 (95% CI 0.70 to 0.75), and an internal-external validation based c-statistic of 0.72 (95% CI 0.68 to 0.75) (online supplemental appendix table 6). The basic model shows that the risk of ACS increases with age for both sexes, with a notable peak risk for men at an age near 60 years and a more gradual increase in risk of ACS for women (figure 2). The full model including all candidate predictors had an apparent c-statistic of 0.79 (95% CI 0.76 to 0.81) and an internal-external validation based c-statistic of 0.77 (95% CI 0.74 to 0.80) (online supplemental appendix table 7). The full model had optimal calibration (flexible line close to the 45 degree reference line) up to a predicted probability of ACS of approximately 0.2 (online supplemental appendix figure 1). Risks higher than 0.2 tended to be overestimated by the model, however, since any plausible risk threshold will be lower than 0.2 in the primary care setting, we find the calibration in the relevant range to be satisfactory. After backward elimination, the backward model had an apparent c-statistic of 0.79 (95% CI 0.76 to 0.81), and the internal-external validation c-statistic was 0.77 (95% CI 0.74 to 0.79) (Table 3) It had very similar calibration to the full model (Table 3, figure 3).

Figure 2

Base model with age and sex for predicting diagnosis acute coronary syndrome. ACS, acute coronary syndrome.

Table 3

Final model for predicting the diagnosis acute coronary syndrome

Predictors	Regression coefficients (SE)
Intercept	−16.246 (3.527)
Age	0.293 (0.081)
Age′	−0.391 (0.125)
Age’″	1.063 (0.395)
Female gender	2.504 (5.512)
Age*female gender	−0.096 (0.126)
Age′*female gender	0.189 (0.195)
Age″*female gender	−0.556 (0.605)
Acute chest pain shorter than 12 hours	0.290 (0.198)
Sweating	0.457 (0.178)
Radiation of chest pain	0.609 (0.176)
Pressing heavy pain	0.747 (0.200)
Call during the night (00:00–09:00)	0.504 (0.151)
Apparent c-statistic 0.79 (95% CI 0.76 to 0.81)Adjusted c-statistic 0.77 (95% CI 0.74 to 0.79)Calibration slope 0.826 (95% CI 0.658 to 0.994)Calibration −0.224 (95% CI −0.604 to 0.157)R2 0.106Knots for cubic spline functions placed at 5, 35, 65 and 95

Figure 3

Callibration of the final model with internal external validation.

Decision curve analyses and risk thresholds

Both the full and backward model showed a high net benefit as compared with the currently used NTS model for telephone triage in OHS-PC (figure 4). There was no difference in net benefit between the full model and backward model across plausible risk thresholds. Based on this analysis we decided to choose the backward as the final triage tool model because; (1) with fewer predictors the prediction of ACS remained similar accurate and (2) no valuable time is lost during telephone triage by asking the patient about symptoms that do not contribute to a better prediction. The final model included besides age and sex, the five following predictors; (i) acute onset of chest pain lasting <12 hours, (ii) a pressing/heavy character, (iii) radiation of pain, (iv) sweating, (v) calling at night between 00.00 and 09.00. Finally, we evaluated the diagnostic performance of the final prediction model across risk thresholds that may be chosen to apply in clinical practice (table 4, figure 5).

Figure 4

Decision curve analyses comparing the full and final models versus the currently used model and versus treat all patients. NTS, Netherlands Triage Standard.

Table 4

Diagnostic accuracy for a range of risk thresholds of the final model

Risk threshold	Sensitivity (95% CI)	Specificity (95% CI)	Positive predictive value	Negative predictive value
0.001	0.98 (0.95 to 0.99)	0.21 (0.18 to 0.24)	0.14	0.99
0.010	0.98 (0.95 to 0.99)	0.42 (0.40 to 0.45)	0.18	0.99
0.020	0.97 (0.94 to 0.99)	0.50 (0.47 to 0.54)	0.20	0.99
0.050	0.93 (0.87 to 0.96)	0.63 (0.59 to 0.67)	0.25	0.99
0.075	0.88 (0.81 to 0.92)	0.72 (0.68 to 0.76)	0.29	0.98
0.100	0.81 (0.7 to 30.87)	0.79 (0.76 to 0.82)	0.33	0.97
0.115 (prevalence)	0.76 (0.67 to 0.83)	0.82 (0.79 to 0.85)	0.36	0.96
0.150	0.64 (0.56 to 0.73)	0.88 (0.85 to 0.90)	0.41	0.95
0.200	0.46 (0.38 to 0.55)	0.93 (0.91 to 0.94)	0.46	0.93

Figure 5

Runway plot of diagnostic accuracy measures of the final model.

Discussion

This is the first study that developed and internal-external validated a symptom-based prediction rule for telephone triage of ACS in male and female patients who contact OHS-PC for acute chest discomfort (pain, pressure, tightness or discomfort). ACS was present in 8.3% of the females and 15.3% of the males. The prediction rule is applicable for triage in the OHS-PC setting and consists of sex and age as statistical interaction terms, and five other symptom-based predictors. It had a good discriminative ability (adjusted c-statistic 0.77 (95% CI 0.74 to 0.79)) and was well calibrated up to an ACS risk of 0.2.

Strengths and limitations

The major strength of this study is that we analysed the original and very first conversations of patients with acute chest discomfort with primary healthcare providers and assessed these talks without knowledge of the diagnosis; the assessment of symptoms was therefore not affected by hindsight bias caused by knowledge of the final diagnosis.29 Furthermore, we could analyse a large sample (N=2192) of patients which allowed us to evaluate up to nineteen candidate predictors. We assessed the risk of selection due to missing outcome data, and our data suggest that this missingness was unlikely to bias our findings. Because we used data from seven different OHS-PC our results will be well generalisable to other OHS-PC in the Netherlands, but we anticipate the model might be applicable to similar OHS-PC setting in, for example, the UK and Scandinavian countries. Our results may also be generalisable to some EMS settings, because the prior chance of having an ACS among those calling for acute chest discomfort is rather similar in the EMS setting as in the OHS-PC setting.14 30

A limitation of the study is that a full external validation of the final model is not done yet. However, at the time of executing the study hardly any primary care research data were available to perform such validation. Importantly, the internal-external validation showed very good calibration up to an ACS risk of 0.2. Although we performed extensive internal-external validation making use of the datasets of nine sites with substantial differences in case mix, we will strive for formal external validation before it can be widely applied in everyday primary care practice. Another limitation is that the effects of the predictors were assumed to be similar for male and female patients while that might not be optimal for the predictions. However, development of separate models for males and females would require a significantly larger sample size than was available. Importantly, a differential non-linear effect of age was incorporated using a spline function and interaction with sex was incorporated, and the final internal-external validated model did have good overall performance.

Comparison with other studies

Our prediction model had a higher discriminating ability for ACS than the NTS (c-statistic of 0.58) and modified Grijseels prediction rule (c-statistic of 0.66).7 This may largely be explained by the addition of age, the strongest predictor of ACS. This is in line with the notion that the prevalence of ACS increases with age.7 9 31 Importantly, in our study among people aged below 40, only one (0.4%) male patient had an ACS (UAP). For males to the age of 55 we found a peak risk of ACS of around 20% and remaining at this level with further age increase onwards. For females we found a gradual increase of risk with age with a maximum ACS risk of around 18% for those aged over 80 years. Similar to the modified Grijseels prediction rule our prediction model includes sweating and radiation of pain, however, the modified Grijseels rule combined nausea and sweating to a single predictor (ie, nausea or sweating).7 8 Age and sex were predictors in our model, but also in the MHS and INTERCHEST prediction models.32 33 Also the INTERCHEST rule included pressing heavy chest pain as predictor.32 A new predictor is calling at night (between 00.00 and 09.00).19 Previous studies in the ED setting also showed circadian variability with an early morning peak for ACS patients.34 Finally, symptoms associated with ACS were rather similar between females and males, which is in line with recent sex-stratified studies of patients with chest discomfort who called the OHS-PC, but is in contrast with the prevailing opinion.16 17 35

We performed decision curve analyses to investigate what would be the optimal threshold of ACS predicted probability to initiate treatment, where ‘treatment’ in prehospital setting refers to urgent referral for hospital admission (U1, ambulance within 15 min). However, there is a principal difference between diagnostic probability and categories of urgencies. Risk prediction provides a continuous value of the probability of disease, while urgency level categorisation is based on the interpretation of how risk probability can be translated in urgency, and time within a patient should be seen and treatment delivered. Context is needed to determine optimal thresholds, which concerns what percentage of missing ACS is considered as acceptable by healthcare providers, patients and policymakers. This percentage is expected to be very low because a missed ACS can result in permanent cardiac impairment, heart failure, life-threatening arrhythmias in the early phase and death.36 Furthermore missing an ACS is the most common reason for malpractice claims worldwide.37 A survey performed among 1029 ED doctors in the USA, New Zealand and Australia showed that they considered an average missing rate between 0.1% and 1% (range 0%–10%) as acceptable.38 When we apply a maximum of 1% missing with our prediction rules, the threshold has to be set at a predicted risk of ACS of 0.05 (negative predictive value of 0.99, table 4), which means based on our data that the majority of patients needs urgent referral. This would result in over-crowded EMS and EDs, and with the available resources being limited, this may result in exceeding target triage times, which could compromise patient safety in another way.39 A possible alternative to consider may be applying different ‘treatments’ per thresholds, that is, dispatching an ambulance (U1) for the high predicted risk patients, and GP visit within 1 hour (U2) for the low predicted risk patients. During GP visit more clinical parameters (blood pressure, heart rate, overall clinical impression) can be gathered to improve ACS risk prediction, and in the future, there might be room for applying point-of-care high-sensitivity troponin testing, as these are nowadays only available in the ED setting.40 In order to determine the ideal threshold, external validation will be needed combined with clinical and management considerations. The development of this diagnostic model is the necessary first step towards an implementation study in which this model is adapted to urgency levels that can be applied by triage nurses during telephone triage at the OHS-PC. The diagnostic model needs to be ‘translated’ in simple yes/no questions that can be incorporated in the existing NTS and a personalised risk prediction for age and gender is generated. Some older questions will then be substituted. We are aiming to do so in an implementation study applying action research.

Implications for clinical practice and future research

This symptom-based prediction model for ACS has good discrimination and calibration and could readily be applied for telephone triage of patients with acute chest discomfort in primary care, notably the OHS-PC setting. The results of the decision curve analysis showed a large net benefit over a range of plausible risk threshold as compared with the currently used NTS model in the OHS-PC in the Netherlands. For future research, full external validation in other OHS-PC or EMS populations could further optimise and update the model. Furthermore, sex-specific prediction models could be developed for ACS, but given the overlap in symptoms between men and women, this would not result in major changes in predictors.

Conclusion

The final prediction model for ACS has good discrimination and calibration and shows promise for replacing the existing telephone triage rules for patients with acute chest discomfort in general practice and OHS-PC. However, future research with an external validation is needed to provide insights into how the prediction model can be applied in practice.