Clinical Prediction Models for Valvular Heart Disease.

Background While many clinical prediction models (CPMs) exist to guide valvular heart disease treatment decisions, the relative performance of these CPMs is largely unknown. We systematically describe the CPMs available for patients with valvular heart disease with specific attention to performance in external validations. Methods and Results A systematic review identified 49 CPMs for patients with valvular heart disease treated with surgery (n=34), percutaneous interventions (n=12), or no intervention (n=3). There were 204 external validations of these CPMs. Only 35 (71%) CPMs have been externally validated. Sixty-five percent (n=133) of the external validations were performed on distantly related populations. There was substantial heterogeneity in model performance and a median percentage change in discrimination of -27.1% (interquartile range, -49.4%--5.7%). Nearly two-thirds of validations (n=129) demonstrate at least a 10% relative decline in discrimination. Discriminatory performance of EuroSCORE II and Society of Thoracic Surgeons (2009) models (accounting for 73% of external validations) varied widely: EuroSCORE II validation c-statistic range 0.50 to 0.95; Society of Thoracic Surgeons (2009) Models validation c-statistic range 0.50 to 0.86. These models performed well when tested on related populations (median related validation c-statistics: EuroSCORE II, 0.82 [0.76, 0.85]; Society of Thoracic Surgeons [2009], 0.72 [0.67, 0.79]). There remain few (n=9) external validations of transcatheter aortic valve replacement CPMs. Conclusions Many CPMs for patients with valvular heart disease have never been externally validated and isolated external validations appear insufficient to assess the trustworthiness of predictions. For surgical valve interventions, there are existing predictive models that perform reasonably well on related populations. For transcatheter aortic valve replacement (CPMs additional external validations are needed to broadly understand the trustworthiness of predictions.

Clinical Perspective

What Is New?

Risk prediction is central to decision making for patients with advanced valvular heart disease; however, the performance of clinical predictive models in external validations is often substantially worse than expected based on derivation data set performance.

What Are the Clinical Implications?

Isolated external validations appear insufficient to broadly understand the performance of valvular heart disease clinical predictive models.

There are clinical predictive models for surgical valvular heart disease interventions that perform well across multiple external validations.

The trustworthiness of transcatheter aortic valve replacement predictions is largely unknown as these models have not been widely tested in external validations.

Introduction

Treatments for patients with advanced valvular heart disease (VHD) are increasingly offered to patients with advanced age and elevated pre‐procedural risk.1, 2, 3 Clinical predictive models (CPMs) have assumed a central role in clinical decision making and current guidelines link VHD treatment decisions to predicted risk.4, 5, 6 CPMs can potentially enhance shared decision making,7, 8 when they perform well and are appropriately matched to the correct decisional context, though there remain major questions about how well CPMs for patients with VHD perform in external validations.

It is well recognized that many of the best known (and most widely used) CPMs for VHD were derived on patients receiving surgical interventions9, 10 and do not accurately predict outcomes for patients treated with percutaneous interventions,11 though they continue to be used for this purpose. While there are newer efforts to create CPMs specific to percutaneous valve interventions, the relative performance of these models and their performances in external validations remains largely unknown. Generally, the performance of CPMs has often been underreported and incompletely assessed12 and since CPMs that perform poorly can yield misleading predictions that motivate harmful decision making,13 it is essential that clinicians understand CPM performance before leveraging outputs to inform decisions. This is especially important for VHD treatment decisions, given the importance of these tools.

Here, using the Tufts PACE (Predictive Analytics and Comparative Effectiveness) CPM Registry, we describe the available CPMs for patients with VHD treated with percutaneous or surgical interventions. This analysis focuses on comparative model performance during external validations.

Methods

General Approach

The data that support the findings of this study are available from the corresponding author upon reasonable request. This study analyzed data from the Tufts PACE (Predictive Analytics and Comparative Effectiveness) CPM Registry, a database created to describe the CPM literature for patients at risk for and with known cardiovascular disease. The registry, which is free and available to the public at http://pace.tuftsmedicalcenter.org/cpm, encompasses a field synopsis of CPMs for patients with VHD. The methods have been previously reported.12 Briefly, we had previously searched PubMed for English‐language articles containing CPMs for cardiovascular disease published from January 1990 through 2015. We extended the search for VHD CPMs to January 2017 to include more recent CPM development (Table S1, Figure S1). Citations were reviewed to confirm completeness of our review. All citations and data fields were extracted in duplicate to ensure accuracy. Discrepancies were discussed until consensus was achieved.

For inclusion in the registry, articles had to meet the following criteria: (1) develop a CPM as a primary aim, (2) contain a model predicting the development of a specified clinical diagnosis (diagnostic models) or the probability of developing a clinical outcome (prognostic models), (3) contain at least 2 outcome predictors, and (4) present enough information to estimate the probability for an individual patient. Articles were excluded if they did not provide enough information to predict a patient's risk or if the described models predicted surrogate outcomes. We also excluded non‐English reports, pharmacology reports, cost‐effectiveness models, decision‐analysis models, systematic reviews, and editorials.

Model Selection

This report focuses on CPMs predicting outcomes for patients with VHD. CPMs predicting natural history outcomes and outcomes after surgical and percutaneous procedures were included. CPMs were grouped based on underlying valve pathology and procedure. CPMs were also included if they were derived on cardiac surgery cohorts where at least 50% of patients received treatment for VHD. CPMs derived exclusively on coronary artery bypass populations were excluded.

CPM Reporting

Information was extracted on CPM derivation and reporting. Collected fields included: index clinical condition, predicted outcome, timeframe of prediction, sample size, cohort size, and number of events. We calculated the events per variable (EPV) based on the number of variables included in the model. We also extracted information on modeling method and performance with specific attention to reporting of discrimination and calibration (Table S2).

Validation Search

Citations for each CPM article through September 2017 were identified using Scopus and reviewed for inclusion as external validations. An external validation was defined as any evaluation of CPM performance (assessment of either discrimination or calibration) on a data set distinct from the derivation data set.14 External validations included validations that were done on the same cohort but temporally or geographically distinct from the derivation cohort or on an entirely separate cohort. Each validation citation was reviewed by 2 investigators for inclusion and discrepancies were reviewed with an additional investigator to arrive at consensus.

Validation Reporting

Information on validation reporting was extracted, including sample size, continent of study, number of events, and reporting of measures of discrimination and calibration (Table S3). The validation performance analysis focused on whether CPM discrimination changed when compared with that seen in the derivation population. Because the c‐statistic ranges from 0.5 (no discriminatory ability) to 1.0 (perfect discrimination), it has been rescaled as Somer's D statistic15 (2×(c−0.5)) so that discrimination ranges from 0 (no discrimination) to 1.0 (perfect discrimination). We describe changes on this scale because it more intuitively reflects the true changes in discriminatory power. The percentage change in discrimination [(Validation AUC−0.5)−(Derivation AUC−0.5)/(Derivation AUC−0.5)×100] is presented. We also document whether validations include any assessment of CPM calibration. There is currently no literature standard for assessing calibration. Given this lack of consistency and interpretability, we have only reported on whether this dimension of performance was assessed. Calibration assessment included any comparison of observed versus expected outcomes. Examples include a Hosmer‐Lemeshow statistic or calibration plot. For this study we also included measures of global fit, where overall observed event rates are compared with predicted rates (ie, calibration‐in‐the‐large).

Relatedness

To assess the similarity between the derivation population and the validation population for each validation, we created a relatedness rubric to divide validations into 2 categories—”related” and “distantly related.” The rubric contained 3 domains: (1) type of intervention (ie, percutaneous or surgical), (2) percentage of the population undergoing isolated valve procedures (as opposed to valve procedures in combination with revascularization), and (3) calendar years of enrollment. We considered a validation population to be “related” if all of the following criteria were met: (1) same type of intervention (eg, both surgical populations), (2) ± 10% absolute difference in the proportion of isolated valve procedure (eg, derivation population was 100% isolated valve and validation population was 95% isolated valve), and (3) overlapping years of enrollment. Matches that did not meet all 3 criteria were deemed “distantly related.”

Results

VHD CPMs

We identified 49 CPMs predicting clinical outcomes for patients with VHD, which were cited a total of 1296 times (Table 1, Table S2).7, 9, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43 Thirty‐four (69%) predict outcomes following surgical interventions, 12 (24%) predict outcomes following percutaneous interventions, and 3 (6%) predict outcomes in the absence of intervention (Table 2). Overall, the most commonly predicted outcomes were 30‐day mortality (n=14, 29%) and in‐hospital mortality (n=14, 29%). Twenty‐four models (46%) were derived from patients in North America, followed by 12 (23%) from Europe and 8 (15%) from Asia (Figure 1). The median derivation sample size was 4510 (interquartile range [IQR], 1087–18 686), median event rate was 8.3% (IQR, 4.5%–14.8%), and median EPV was 40 (IQR, 20–92) (Table 2). The median number of covariates was 10 (IQR, 7–19).

Table 1

De Novo VHD CPMs Overview

Author, Model Name	Publication, y	Valve	Standardized Type of Intervention	Outcome	Model Method	C‐Statistic	Calibration Measure	Externally Validated?
Isolated valve
Edwards,16 STS (original) Isolated Valve	2001	Aortic/Mitral	Surgery	30 d operative mortality	Logistic regression	0.766	HL statistic, Calibration plot	Yes
Nowicki,17 NNE Aortic and Mitral Models	2004	Aortic	Surgery	In‐hospital mortality	Logistic regression, score	0.75	HL statistic	Yes
		Mitral	Surgery	In‐hospital mortality	Logistic regression, score	0.79	HL statistic	Yes
Kuduvalli,18 NWQIP	2007	Aortic	Surgery	In‐hospital mortality	Logistic regression, score	0.78	HL statistic	Yes
Cruz‐Gonzalez,19 PMV Score	2009	Mitral	Percutaneous	Procedural success	Logistic regression, score	NR	HL statistic	Yes
Monin20	2009	Aortic stenosis	Natural History	Composite (Non‐MACE)	Logistic regression, score	0.90	HL statistic	Yes
O'Brien,9 STS (2009)—Composite AEs	2009	Aortic/Mitral	Surgery	Composite (Non‐MACE)	Logistic regression	0.721	None	Yes
O'Brien,9 STS (2009)—DSWI	2009	Aortic/Mitral	Surgery	DSWI	Logistic regression	0.704	None	Yes
O'Brien,9 STS (2009)—Mortality	2009	Aortic/Mitral	Surgery	30 d mortality	Logistic regression	0.805	None	Yes
O'Brien,9 STS (2009)—Prolonged LOS	2009	Aortic/Mitral	Surgery	Prolonged LOS	Logistic regression	0.77	None	Yes
O'Brien,9 STS (2009)—Prolonged Ventilation	2009	Aortic/Mitral	Surgery	Prolonged ventilation	Logistic regression	0.77	None	Yes
O'Brien,9 STS (2009)—Renal Failure	2009	Aortic/Mitral	Surgery	Renal failure	Logistic regression	0.782	None	Yes
O'Brien,9 STS (2009)—Reoperation	2009	Aortic/Mitral	Surgery	Reoperation	Logistic regression	0.643	None	Yes
O'Brien,9 STS (2009)—Short LOS	2009	Aortic/Mitral	Surgery	Prolonged LOS	Logistic regression	0.738	None	No
O'Brien,9 STS (2009)—Stroke	2009	Aortic/Mitral	Surgery	Stroke	Logistic regression	0.694	None	Yes
Guaragna,21 GuaragnaSCORE	2010	Aortic/Mitral	Surgery	In‐hospital mortality	Logistic regression, score	0.82	HL statistic, Calibration plot	Yes
Guo22	2010	Aortic	Surgery	In‐hospital mortality	Logistic regression	NR	HL statistic	No
		Mitral	Surgery	In‐hospital mortality	Logistic regression	NR	HL statistic	No
Elmariah,23 CRRAC the AV Score	2011	Aortic	Percutaneous	30 d mortality	Cox regression, score	0.754	HL statistic	No
Bouleti24	2012	Mitral	Percutaneous	Composite (MACE)	Cox regression, score	0.74	Calibration plot	No
Cioffi25	2012	Aortic stenosis	Natural History	Composite (MACE)	Cox regression, score	NR	None	No
Holme,26 SEAS Score	2012	Aortic stenosis	Natural History	5 y mortality	Cox regression	0.722	HL statistic, Calibration plot, Brier score	No
Kötting,27 German Aortic Valve Score	2013	Aortic	Percutaneous	In‐Hospital Mortality	Logistic regression, score	0.808	HL statistic	Yes
Arnold,28 6 mo and 1 y Models	2014	Aortic stenosis	Percutaneous	Composite (Non‐MACE)	Logistic regression	0.66	HL statistics, Calibration plot	Yes
		Aortic stenosis	Percutaneous	Composite (Non‐MACE)	Logistic regression	0.66	HL statistics, Calibration plot	Yes
Capodanno,29  OBSERVANT Score	2014	Aortic stenosis	Percutaneous	30 d mortality	Logistic regression, score	0.73	HL statistic, Calibration plot, Brier score	Yes
D'Ascenzo,30 Survival Post‐TAVI (STT)—30 d and 1 y Models	2014	Aortic	Percutaneous	30 d mortality	Logistic regression, score	0.66	HL statistic	Yes
		Aortic	Percutaneous	1 y mortality	Logistic regression, score	0.68	HL statistic	Yes
Iung31	2014	Aortic	Percutaneous	30 d mortality	Logistic regression, score	0.67	HL statistic, Calibration in the large, Calibration plot	No
Debonnaire,32 TAVI2‐SCORe	2015	Aortic	Percutaneous	1 y mortality	Cox regression, score	0.715	HL statistic, Calibration in the large	Yes
Edwards7	2016	Aortic	Percutaneous	In‐hospital mortality	Logistic regression	0.67	HL statistics, Calibration in the large, Calibration plot	Yes
Isolated or multiple valve
Koplan33	2003	All	Surgery	Pacemaker placement	Logistic regression, score	NR	None	No
Ambler34	2005	Aortic, mitral	Surgery	In‐hospital mortality	Logistic regression, Score	0.77	HL statistic, Calibration plot	Yes
Xu35	2006	All	Surgery	Prolonged LOS	Logistic regression	0.81	Calibration in table form	No
Hannan36	2007	Aortic, mitral	Surgery	In‐hospital mortality	Logistic regression, Score	0.794	HL statistic, Calibration plot	Yes
Xu,37 Fuwai Score	2007	All	Surgery	Prolonged LOS	Logistic regression, Score	0.76	HL statistic, Calibration plot	Yes
Shi38	2010	Aortic, mitral	Surgery	In‐hospital mortality	Logistic regression	0.7358	None	No
Ariyaratne,39 Aus‐AVR Score	2011	Aortic, mitral	Surgery	30 d mortality	Logistic regression, Score	0.78	HL statistic, Calibration in the large	Yes
Nashef,10 EuroSCORE II	2012	All	Surgery	In‐hospital mortality	Logistic regression	0.8095	None	Yes
Hannan,40 NY Operative Mortality Risk Score	2013	Aortic, mitral	Surgery	30 d mortality	Logistic regression, Score	0.781	HL statistic	Yes
Wang41	2013	All	Surgery	Prolonged ventilation	Logistic regression	0.789	HL statistic	No
Zheng42	2013	Aortic, mitral	Surgery	In‐hospital mortality	Logistic regression, Score	0.76	HL statistic, Chi‐square statistic, Calibration plot	No
Multiple valve
Guo22	2010	Aortic, mitral	Surgery	In‐hospital mortality	Logistic regression	NR	HL statistic	No
Rankin,43 AM Preop	2013	Aortic, mitral	Surgery	30 d mortality	Logistic regression	NR	Calibration plot	Yes
Rankin,43 MT Preop	2013	mitral, tricuspid	Surgery	30 d mortality	Logistic regression	NR	Calibration plot	Yes
Rankin,43 AMT Preop	2013	Aortic, mitral, tricuspid	Surgery	30 d mortality	Logistic regression	NR	Calibration plot	Yes
Rankin,43 AM Preop + Intraop	2013	Aortic, mitral	Surgery	30 d mortality	Logistic regression	NR	Calibration plot	Yes
Rankin,43 MT Preop + Intraop	2013	Mitral, tricuspid	Surgery	30 d mortality	Logistic regression	NR	Calibration plot	Yes
Rankin,43 AMT Preop + Intraop	2013	Aortic, mitral, tricuspid	Surgery	30 d mortality	Logistic regression	NR	Calibration plot	Yes

AEs indicates adverse events; AM, aortic, mitral ; AMT, aortic, mitral, tricuspid; AV, aortic valvuloplasty; Aus‐AVR, Australian aortic valve replacement; CRRAC, critical status, renal dysfunction, right atrial pressure, and cardiac output; DSWI, deep sternal wound infections; EuroSCORE, European System for Cardiac Operative Risk Evaluation; HL, Hosmer‐Lemeshow; LOS, length of stay; MACE, major adverse cardiovascular events; MT, mitral, tricuspid; NNE, Northern New England; NR, not reported; NWQIP, North West Quality Improvement Programme in Cardiac Interventions; NY, New York; OBSERVANT, Observational Study of Appropriateness, Efficacy and Effectiveness of AVR‐TAVR Procedures for the Treatment of Severe Symptomatic Aortic Stenosis; PMV, percutaneous mitral valvuloplasty; SEAS, Simvastatin and Ezetimibe in Aortic Stenosis; STS, Society of Thoracic Surgeons; STT, Survival posT‐TAVI; TAVI, transcatheter aortic valve implantation; TAVR, Transcatheter Aortic Valve Replacement.

Table 2

Reported Characteristics of De Novo Valvular Heart Disease CPMs

Characteristica	Overall (n=49)	Surgical (n=34)	Percutaneous (n=12)	Natural History (n=3)
Publication rangeb	2001 to 2016	2001 to 2013	2009 to 2016	2009 to 2012
Age, y	69 (61–79)	65 (58–70)	82 (82–83)	68 (67–70)
Sample size	4510 (1087–18 686)	12 079 (3125–92 563)	1160 (752–2241)	772 (440–1169)
Event rate	0.08 (0.05–0.15)	0.07 (0.04–0.11)	0.14 (0.06–0.37)	0.35 (0.23–0.47)
Events per variable	40 (20–92)	46 (25–110)	42 (18–81)	11 (10–45)
C‐statistic	0.76 (0.72–0.78)	0.77 (0.75–0.79)	0.68 (0.67–0.74)	0.81 (0.77–0.86)
% Externally validated	71.4	73.5	83.3	33.3

CPM indicates clinical predictive models.

Values are reported as median (interquartile range), unless otherwise specified.

De novo CPM search spans January 1, 1990 to January 1, 2017.

Figure 1

Geography of derivation and validation cohorts. Country of origin for derivation (A) and validation (B) populations. Maps created in Tableau Public.

Among models that reported a c‐statistic (n=37, 76%), the overall median ROC was 0.76 (IQR, 0.72–0.78) (Table 2). When stratified by intervention type, the median c‐statistic was 0.77 (IQR, 0.75–0.79) for CPMs predicting outcomes following surgical interventions, 0.68 (IQR, 0.67–0.74) for CPMs for percutaneous interventions, and 0.81 (IQR, 0.77–0.86) for CPMs predicting outcomes in the absence of intervention (Table 2).

CPMs for Isolated Valve Disease

There are 31 CPMs for isolated valve disease. Sixteen (52%) predict outcomes following surgical intervention and 12 (39%) predict outcomes following percutaneous interventions (transcatheter aortic valve replacement [TAVR], balloon aortic valvuloplasty, and percutaneous mitral balloon valvuloplasty). Three CPMs (10%) predict outcomes for patients with aortic stenosis in the absence of intervention. The median derivation sample size was 2552 (IQR, 1064–108 410) and the median age was 70 (IQR, 64–82). The median number of events was 360 (IQR, 104–2021) and the median EPV was 55 (IQR, 18–112). The median event rate was 10% (IQR, 4.6%–18.3%). For the 27 (87%) models reporting discrimination, the median c‐statistic was 0.74 (IQR, 0.69–0.78).

CPMs for Isolated or Multiple Valve Disease

There are 11 CPMs that predict outcomes for patients undergoing either single or multiple valve surgical procedures. The median derivation sample size was 3544 (IQR, 2297–12 079) and the median age was 60 (IQR, 54–65). The median number of events was 303 (IQR, 139–507) and the median EPV was 26 (IQR, 20–40). The median event rate was 5.1% (IQR, 4.1%–9.5%). For the 10 (91%) models reporting discrimination, the median c‐statistic was 0.78 (IQR, 0.76–0.79).

CPMs for Multiple Valve Disease

There are 7 CPMs that predict outcomes specifically for multiple valve surgical interventions. These CPMs include the Society of Thoracic Surgeons (STS) Multi‐Valve Risk Models43 and the median derivation sample size was 18 686 (IQR, 4510–22 861). The median number of events was 1420 (IQR, 591–1981) and the median EPV was 71 (IQR, 48–92). Median age was 70 (IQR, 70–71). The median event rate was 9.4% (IQR, 7.6%–11.3%). Median number of covariates was 20 (IQR, 14–23).

CPMs for Percutaneous VHD Interventions

Since 2009, there have been 12 CPMs presented that predict outcomes following percutaneous VHD interventions. Two models predict outcomes following mitral percutaneous mitral balloon valvuloplasty.24, 44 There were 9 CPMs that predict outcomes following TAVR with a median derivation sample size of 2130 (IQR, 10 642 552). The median age of patients in the TAVR CPMs was 82 (IQR, 82–83). TAVR CPMs had a median number of events of 253 (IQR, 80–704) and the median EPV was 28 (IQR, 20–70). The median event rate was 9.9% (IQR, 5.6%–15.7%). All of the CPMs predicting outcomes following TAVR reported discrimination with a median c‐statistic of 0.67 (IQR, 0.66–0.72).

External Validations

Two hundred and four external validations of these CPMs were identified, of which 190 (93%) report a c‐statistic. Overall, 35 (71%) of the VHD CPMs have been externally validated and 20 (37%) have been externally validated more than once. External validations were most commonly done in cohorts of patients from Europe (n=93, 46%), Asia (n=38, 19%), and North America (n=37, 18%) (Figure 1). Fifty‐three (26%) validations were performed on populations from the same continent as the derivation population, with a median c‐statistic of 0.71 (IQR, 0.66–0.77). Seventy‐one (35%) were done on populations from a different continent, with a median c‐statistic of 0.68 (IQR, 0.64–0.73). External validations overall had a median c‐statistic of 0.71 (IQR, 0.65– 0.77) (Table 3). For the models that were externally validated, we noted an overall median percentage change in discrimination of −27.1% (IQR, −49.4–−5.7). Just under two‐thirds of validations (n=129) demonstrate at least a 10% relative decline in discriminatory power, and 18 (9%) showed a decline of >80%. Thirty‐three (16%) validations showed CPM discrimination at or above that seen in the derivation cohort.

Table 3

Reported Characteristics of Valvular Heart Disease External Validationsc

Characteristica	Overall (n=204)b	Surgical (n=131)	Percutaneous (n=70)
Sample size	450 (249–1495)	809 (407–3306)	304 (180–453)
Number of events	38 (15–95)	48 (14–119)	38 (15–56)
Event rate	0.06 (0.03–0.12)	0.04 (0.03–0.07)	0.11 (0.08–0.17)
% Men	53 (47–62)	57 (53–66)	47 (43–52)
C‐statistic	0.71 (0.65–0.77)	0.74 (0.70–0.79)	0.63 (0.57–0.68)

Values are reported as median (interquartile range).

Validations done on populations treated with surgical and percutaneous interventions that did not disaggregate results (n=2) are only included in the overall count.

Validation search includes citations through September 8, 2017.

The distribution of number of validations was skewed towards a small number of CPMs. Two CPMs (EuroSCORE II and STS [2009] Models9) accounted for 73% of the external validations. EuroSCORE II has been validated 78 times across 5 continents (Table 4). Validation c‐statistics ranged from 0.50 to 0.95 with a median percentage change of −23.4% (range −100%–+46.7%). For the STS (2009) Models, validation c‐statistics ranged from 0.50 to 0.86 (Table 5). The median percentage change was −31.8% and ranged from −100% to +18%. The STS (2009) Models have been validated 70 times across 5 continents.

Table 4

EuroSCORE II Population Compared With External Validation Populations, Stratified by Relatedness

Statistica	EuroSCORE II	Validation Populationsb, c
		Related	Distantly Related
Total patients (n)	16 828	14 382	98 744
Total validations (n)	NA	5	73
Age, y	Mean (SD): 64.6 (12.5)	63.4 (62.7–67.0)	67.1 (61.1–80.5)
Number of events (n)	656	123 (53–215)	27 (12–57)
Event rate, %	3.9	5.7 (5.7–6.1)	6.3 (3.0–10.5)
Sex reported, n (%)	NA	5 (100%)	50 (68%)
Men, %	69.1	65.2 (62.5–66.5)	52.5 (46.8–64.1)
Type of intervention, n (%)
Surgery	1 (100%)	5 (100%)	52 (71.2%)
Percutaneous	0 (0%)	0 (0%)	20 (27.4%)
Both	0 (0%)	0 (0%)	1 (1.4%)
Valve‐related, %	53.3	56 (54.6–56.1)	100 (100–100)
Enrollment, y (range)	2010	2005 to 2013	1999 to 2015
C‐statistic	0.8095	0.82 (0.76–0.85)	0.72 (0.67–0.78)
C‐statistic (range)	NA	0.737 to 0.861	0.50 to 0.95
Any calibration reported, n (%)	0 (0%)	4 (80%)	65 (89%)
Change in discrimination,d %	NA	2.6 (−16.0–13.1)	−28.9 (−45.3–−9.5)

EuroSCORE indicates European System for Cardiac Operative Risk Evaluation.

All values are reported as median (interquartile range) unless otherwise specified.

Validation data is reported at the population level only; patient‐level data was not available.

Validation population are “related” if it meets all of the following criteria: (1) same type of intervention (eg, both surgical populations), (2) ±10% absolute difference in the proportion of isolated valve procedure (eg, derivation population was 100% isolated valve and validation population was 95% isolated valve), and (3) overlapping years of enrollment. A validation population that does not meet all 3 criteria is “distantly related.”

Change in discrimination is calculated as [(Validation AUC−0.5)−(Derivation AUC−0.5)]/(Derivation AUC−0.5)×100.

Table 5

STS (2009) Population Compared With External Validation Populations, Stratified by Relatedness

Statistica	STS Models (n=9)	Validation Populationsb,c
		Related	Distantly Related
Total patients, n	109 759	37 395	49 530
Total validations, n	NA	33	37
Age, y	Not Reported	64.7 (56.6–73)	81.6 (74.5–83)
Number of events, n	9164 (3706–12 892)	29 (12–82)	38 (18–57)
Event rate, %	8.3 (3.4–11.7)	4.9 (2.7–12.6)	9.1 (3.7–11.7)
Men, %	55.4	56 (56.0–74.9)	47.8 (43.6–55.3)
Type of intervention, n (%)
Surgery	9 (100%)	33 (100%)	8 (21.6%)
Percutaneous	0 (0%)	0 (0%)	28 (75.7%)
Both	0 (0%)	0 (0%)	1 (2.7%)
Valve‐related, %	100	100 (100–100)	100 (100–100)
Enrollment, y (range)	2002–2006	1997–2014	1999–2015
C‐statistic, median, IQR	0.74 (0.70–0.77)	0.72 (0.67–0.79)	0.65 (0.6–0.71)
C‐statistic (range)	0.643 to 0.805	0.612 to 0.86	0.5 to 0.81
Calibration reported, n (%)	9 (100%)	18 (54.5%)	28 (75.7%)
Change in discrimination,d %	NA	−21.3 (−34.4–2.3)	−50.8 (−67.2–−25.1)

IQR indicates interquartile range; STS, Society of Thoracic Surgeons.

All values are reported as median (interquartile range) unless otherwise specified.

Validation data is reported at the population level only; patient‐level data was not available.

Validation population is “related” if it meets all of the following criteria: (1) same type of intervention (eg, both surgical populations), (2) ±10% absolute difference in the proportion of isolated valve procedure (eg, derivation population was 100% isolated valve and validation population was 95% isolated valve), and (3) overlapping years of enrollment. A validation population that does not meet all 3 criteria is “distantly related.”

Change in discrimination is calculated as [(Validation AUC−0.5)−(Derivation AUC−0.5)]/(Derivation AUC−0.5)×100.

Of CPMs that have been validated at least 2 times (n=17) in related populations, the highest median validation c‐statistic was seen for EuroSCORE II (0.82 [IQR, 0.76–0.85]), followed by the North West Quality Improvement Programme in Cardiac Interventions model (0.78 [IQR, 0.77–0.78]), and the Northern New England Aortic model (0.76 [IQR, 0.75–0.77]) (Table 6).

Table 6

CPMs that Have Been Validated ≥2 Times in Related Populations

De Novo CPM		External Validations in Related Populations (n)	Validation C‐statistic, median (IQR)	% Change in Discrimination,a Median (IQR)	Any Calibration Reported (%)
Pub., Y	Model Name
2001	STS (original): Isolated Valve	2	0.77 (0.77, 0.77)	2.6 (2.6–2.6)	100
2004	NNE Aortic	2	0.76 (0.76, 0.77)	4.0 (2.0–6.0)	100
2007	NWQIP	2	0.78 (0.77, 0.78)	−1.8 (−2.7–−0.9)	100
2005	Ambler	4	0.73 (0.72, 0.76)	−15.2 (−18.9–−2.2)	100
2009	STS: Mortality	19	0.74 (0.71, 0.79)	−21.3 (−31.5–−4.8)	95
2009	STS: Stroke	2	0.65 (0.65, 0.66)	−20.9 (−23.8–−17.9)	0
2009	STS: Prolonged Ventilation	2	0.72 (0.68, 0.75)	−20.2 (−33.8–−6.6)	0
2009	STS: Prolonged LOS	2	0.67 (0.65, 0.68)	−38.1 (−43.5–−32.8)	0
2009	STS: Renal Failure	2	0.76 (0.72, 0.79)	−9.6 (−22.5–3.4)	0
2009	STS: DSWI	2	0.68 (0.65, 0.70)	−13.7 (−24.8–−2.7)	0
2009	STS: Composite AEs	2	0.68 (0.65, 0.71)	−18.8 (−30.7–−6.9)	0
2009	STS: Reoperation	2	0.64 (0.63, 0.65)	−2.1 (−11.9–7.7)	0
2011	Aus‐AVR Score	3	0.72 (0.67, 0.72)	−22.9 (−40.4–−20.4)	100
2012	EuroSCORE II	5	0.82 (0.76, 0.85)	2.6 (−16.0–13.1)	80
2013	NY Operative Mortality Risk Score	3	0.73 (0.71, 0.75)	−18.1 (−26.5–−9.8)	66.7
2014	OBSERVANT Score	4	0.60 (0.58, 0.61)	−57.8 (−63.7–50.7)	50
2014	STT: 30 d	2	0.66	0	50

AEs indicates adverse events; Aus‐AVR, Australian aortic valve replacement; CPM indicates clinical predictive models; DSWI, deep sternal wound infections; EuroSCORE, European System for Cardiac Operative Risk Evaluation; IQR, interquartile range; LOS, length of stay; NNE, Northern New England; NWQIP, North West Quality Improvement Programme in Cardiac Interventions; NY, New York; OBSERVANT, Observational Study of Appropriateness, Efficacy and Effectiveness of AVR‐TAVR Procedures for the Treatment of Severe Symptomatic Aortic Stenosis; STS, Society of Thoracic Surgeons; STT, Survival posT‐TAVI; TAVI, transcatheter aortic valve implantation.

Change in discrimination is calculated as [(Validation AUC−0.5)−(Derivation AUC−0.5)]/(Derivation AUC−0.5)×100.

Forty‐five (22%) external validations did not report any measure of calibration. Of the 159 validations that did report calibration, 103 (65%) reported the Hosmer‐Lemeshow statistic, 87 (55%) reported calibration‐in‐the‐large, and 46 (29%) included a calibration plot. Median c‐statistic was 0.71 (IQR, 0.65–0.77) for validations that reported some measure of calibration and 0.68 (IQR, 0.63–0.74) for validations that did not report any calibration.

Clinical relatedness between the development and validation populations was assessed using a novel rubric. Seventy‐one validations (35%) were performed on related populations, while the remaining 133 (65%) were performed on distantly‐related populations. The median validation c‐statistic was 0.73 (IQR, 0.67–0.79) for related validations and 0.70 (IQR, 0.62–0.76) for distantly‐related validations (P=0.009). There was a significant difference in percentage change in discrimination: the median change in c‐statistic was −12.2% (IQR, −28.3%–+2.5%) for related validations and −32.1% (IQR, −54.9%–−12.8%) for distantly‐related validations (Figure 2, P<0.0001).

Figure 2

Percentage change in discrimination in external validations of valvular heart disease clinical prediction models, stratified by relatedness. Each bar represents a unique external validation that reports a c‐statistic (n=205). Society of Thoracic Surgeons (2009) Models. Percentage change in discrimination is calculated as ([validation c‐statistic–0.5]–[derivation c‐statistic–0.5])/[derivation c‐statistic–0.5]×100. STS indicates Society of Thoracic Surgeons

CPMs that were derived on percutaneously‐treated VHD populations and externally validated (n=9) underwent a total of 19 validations, almost all of which were on percutaneous populations. CPMs that were derived on surgical VHD populations and externally validated (n=25) underwent a total of 184 validations, of which 130 (71%) were on surgical populations, 52 (28%) were on percutaneous populations, and 2 (1%) were on populations including both surgical and percutaneous interventions. For validations of surgical VHD models discrimination was better when CPMs were tested on cohorts treated with surgical versus percutaneous interventions (median c‐statistic 0.74 versus 0.63, P<0.001).

Of the surgical VHD CPMs validated on percutaneous populations (n=52 validations), the CPM most often validated was the STS (2009) model predicting mortality (n=27, median c‐statistic 0.64 [IQR, 0.58–0.67]). EuroSCORE II (n=20) had the highest discrimination in this setting, with a median c‐statistic of 0.67 (IQR, 0.55–0.71).

Discussion

Here we show that there are many CPMs available for patients with VHD and that many of these CPMs have not been externally validated. For the CPMs that have been externally validated, models often perform substantially worse than expected based on performance in derivation data sets. Notably, isolated external validations of VHD CPMs appear insufficient for broadly understanding CPM performance in the context of specific clinical decisions as predictive models may have highly variable performance across various databases. For patients under consideration for surgical VHD interventions, there are CPMs that have been extensively validated. The fidelity of TAVR CPM predictions is largely unknown, as these models have not been widely tested in external validations.

Predicted risk is central to procedural decision making for patients with VHD, however. individual risk estimates using published CPMs for VHD appear more uncertain than originally thought, especially when prediction models are derived on patients who are not closely related to the patients being treated. CPM performance (specifically discrimination) substantially degrades from the derivation population to the validation population, particularly when populations are “distantly related” with respect to procedure type (percutaneous versus open surgical), therapeutic era, and the need for concurrent revascularization. Without attention to these patient‐level specifics, it is likely that there is widespread inappropriate use of CPMs that are informing treatment decisions for patients with VHD. While it is encouraging that newer models have been developed for TAVR patients, these CPMs have not been widely validated or integrated into contemporary guidelines, and have risk estimates that may become inaccurate as devices continue to improve and procedural techniques mature. The attenuated performance of these TAVR CPMs may also be related to the magnitude and significance of comorbid illnesses that are common for older treated adults and are rarely included as part of parsimonious modeling efforts.45 More work is needed to understand these risk factors.

The decrease in discrimination that is observed in this study may be attributable to model overfitting, differences in case mix (ie, narrower populations in the validation data set), and phenotypic heterogeneity. Ultimately, the relevant performance metrics for clinicians relate to the patients they are treating (with a specific intervention), not to performance measured at the time of CPM development. Rarely, discrimination appears to improve during validations. This is likely the result of differences between the derivation population and the validation population where some models are developed on more highly selected (narrow case‐mix) cohorts than they are testing on. The data presented here demonstrates that CPMs externally validated multiple times show substantial variation in performance. This strongly suggests that adequate performance demonstrated in a single external validation may be insufficient to assess the quality (and utility) of VHD CPMs and that a more tailored approach is needed to understand the trustworthiness of CPM predictions in specific settings.

There is increasing recognition of the central importance of CPM calibration. Surprisingly, calibration was reported in only 78% of the external validations of VHD CPMs. There is no agreed‐upon standard for reporting model calibration and no consensus on interpreting this metric. Moreover, there are well‐recognized limitations to the most commonly reported measure, the Hosmer‐Lemeshow statistic (eg, sample‐size dependence).46 Reporting of model calibration represents a major shortcoming in the assessment and overall trustworthiness of VHD CPMs, as adequate calibration can protect against harm.13 Ongoing efforts, including the Transparent Reporting of a Multivariable Model for Individual Prognosis or Diagnosis statement47 and the Prognosis Research Strategy,48 should lead to more consistent evaluation and better reporting of this important performance metric. For currently available CPMs, it is important for clinicians to know whether predictions over‐ or underestimate risk when compared with observed event rates in similar patients, especially when the decision for or against a particular intervention is informed by this output (as is the case for percutaneous VHD treatments). Poor calibration can often be overcome with a variety of updating techniques that can substantially improve accuracy of predictions.49

There is much that remains unknown about these predictive models. CPM performance has not been studied for patients treated in many areas of Eastern Europe, Asia, Central America, South America, and Africa. This is especially important considering the many regional differences in VHD etiology, access to technologies, care systems, and guidelines.50, 51, 52 The optimal number of validations required to adequately assess CPM performance remains unknown. Ideally, CPMs are serially validated and recalibrated (if necessary) to optimize performance for specific, local clinical decision making. Without addressing these limitations, clinical decisions that leverage CPM outputs may be inaccurate and lead to harmful decisions.

This analysis offers a structure to consider which CPMs are most accurate (discrimination and calibration) and trustworthy (consistent performance in multiple external validations). For patients being considered for surgical valve interventions, EuroSCORE II (median validation c‐statistic 0.82 [0.76, 0.85]), North West Quality Improvement Programme in Cardiac Interventions Model (median validation c‐statistic 0.78 [0.77, 0.78]), Northern New England Aortic Model (median validation c‐statistic 0.76 [0.76, 0.77]), Ambler (median validation c‐statistic 0.73 [0.72, 0.76]), and STS (2009) Mortality (median validation c‐statistic 0.74 [0.71, 0.79]) have reasonable discrimination and multiple assessments of discrimination and calibration in external data sets. There are no CPMs for patients treated with TAVR that demonstrate good performance across multiple related validation databases. The trustworthiness of these newer risk estimates for TAVR remains under‐studied.

There are several limitations to this work. Our review was limited to CPMs that provide enough information in the published report to calculate a risk prediction for a patient. Logistic regression models that did not report a full equation or intercept were not included. Cox regression models that did not report a point score or baseline hazard were excluded. The search for de novo VHD CPMs was last run in January 2017. While newer CPMs have been developed, there is often substantial delay before the publication of subsequent external validations. Notably, we present relative changes in discrimination to more accurately document changes on a clinically relevant scale, where small decreases in the C‐statistic can result in large changes in clinically relevant performance. Lastly, this study was limited in its examination of CPM calibration, which is an important measure of model performance, but often poorly reported and without a widely‐accepted summary measure.

While there are numerous available CPMs for patients with VHD, many have never been externally validated, and for those that have, discriminatory performance is often much worse than originally reported. We note that CPM performance is highly dependent on the cohort selected for study, suggesting that one‐off external validations may inadequately assess performance. Instead of new CPM development, robust external validations of established TAVR CPMs and an understanding of the updating techniques that best optimize model performance are needed.

Sources of Funding

This work was partially supported through a Patient‐Centered Outcomes Research Institute Methods Award (ME‐1606‐35555), as well as by the National Institutes of Health (R03 AG056447 GEMSSTAR Grant and K23 AG055667) and Bellows Foundation Grant from the American College of Cardiology.

Disclosures

None.

Table S1. Search strategy used to identify all CVD/Cerebrovascular and VHD‐specific CPMs

Table S2. De Novo Model Unabridged Overview

Table S3. External Validations Overview

Figure S1. Literature Search Overview.

Click here for additional data file.