Development and validation of a nomogram to predict recurrence and melanoma-specific mortality in patients with negative sentinel lymph nodes.

BACKGROUND: Patients with melanoma and negative sentinel nodes (SNs) have varying outcomes, dependent on several prognostic factors. Considering all these factors in a prediction model might aid in identifying patients who could benefit from a personalized treatment strategy. The objective was to construct and validate a nomogram for recurrence and melanoma-specific mortality (MSM) in patients with melanoma and negative SNs.
METHODS: A total of 3220 patients with negative SNs were identified from a cohort of 4124 patients from four EORTC Melanoma Group centres who underwent sentinel lymph node biopsy. Prognostic factors for recurrence and MSM were studied with Cox regression analysis. Significant factors were incorporated in the models. Performance was assessed by discrimination (c-index) and calibration in cross-validation across the four centres. A nomogram was developed for graphical presentation.
RESULTS: There were 3180 eligible patients. The final prediction model for recurrence and the calibrated model for MSM included three independent prognostic factors: ulceration, anatomical location and Breslow thickness. The c-index was 0·74 for recurrence and 0·76 for the calibrated MSM model. Cross-validation across the four centres showed reasonable model performance. A nomogram was developed based on these models. One-third of the patients had a 5-year recurrence probability of 8·2 per cent or less, and one-third had a recurrence probability of 23·0 per cent or more.
CONCLUSION: A nomogram for predicting recurrence and MSM in patients with melanoma and negative SNs was constructed and validated. It could provide personalized estimates useful for tailoring surveillance strategies (reduce or increase intensity), and selection of patients for adjuvant therapy or clinical trials.

Introduction

Sentinel lymph node biopsy (SLNB), introduced in 1991 as a staging procedure for cutaneous melanoma, evaluates the presence of lymph node involvement1. The impact of SLNB has been studied extensively and is one of the most important prognostic indicators for recurrence and survival in patients with melanoma2, 3. Consequently, sentinel node (SN) status has significant implications for treatment strategy. Patients with a positive SN usually had completion lymph node dissection (CLND), but the landmark DeCOG‐SLT trial4 and Multicenter Selective Lymphadenectomy Trial (MSLT) II trial5 concluded there was no significant survival benefit for CLND compared with nodal observation. In future, most patients with positive SNs will be offered routine adjuvant therapy, conceivably without preceding CLND4, 5, 6, 7, 8, 9. Patients with negative SNs have not been included in recent adjuvant therapy trials and are usually offered regular surveillance examinations instead.

Reported recurrence rates for patients with negative SNs vary between 6 and 29 per cent10, 11. When accounting for histological subtype and ulceration, the recurrence rate may increase up to 43 per cent, which, strikingly, approximates the recurrence rate in patients with positive SNs11. Perhaps these high‐risk patients with negative SNs might benefit from adjuvant therapy as well.

The eighth edition of the AJCC staging manual categorizes melanoma with a negative SN into stages IA–IIC based on ulceration and Breslow thickness (T category)12. Several other independent factors have been identified that contribute to risk of recurrence and/or melanoma‐specific mortality (MSM)13. Considering these additional clinicopathological factors in a prediction model might provide more accurate patient‐specific estimates that could be used for treatment strategy decision‐making. The objective of the present study was to identify independent prognostic factors in a large European melanoma population with negative SNs to develop and validate a prediction model for recurrence and MSM, presented in the form of a nomogram.

Methods

Cohort characteristics

A retrospective cohort collected and described previously14 was used for this study. The cohort contained 4124 patients who underwent a SLNB between 1997 and 2013 in one of four European Organization for Research and Treatment of Cancer (EORTC) Melanoma Group centres. The study was approved and performed in accordance with local ethics committee guidelines and national legislation. For purposes of the present study, a total of 3220 patients with a negative SN were identified from this cohort. Data on sex, age, diagnosis date, date of SLNB, primary tumour characteristics (Breslow thickness, ulceration), and details on recurrence and follow‐up were collected.

Procedures and follow‐up

Histopathological examination of an excision biopsy of the primary melanoma led to the diagnosis in all patients. The excision biopsy was performed with total thickness excision and a narrow circumferential margin15. Eligibility for SLNB in all centres was assessed according to international guideline criteria: Breslow thickness greater than 1·0 mm or presence of risk factors, including ulceration, Clark level IV or V according to the sixth edition of the AJCC staging manual up to 200916, and regression or mitosis greater than 1/mm2 according to the seventh edition of the AJCC staging manual from 200917. In general, a wide local excision was performed simultaneously with the SLNB, as described elsewhere1, 14. Histopathological analysis of the SN was conducted according to the EORTC Melanoma Group pathology protocol18. Follow‐up strategies in EORTC centres varied, but usually consisted of clinical examination two to four times per year for 5–10 years15, 19.

Outcomes

Outcomes of interest were recurrence and MSM, calculated from date of SLNB to date of first recurrence or death. When there was multisite first recurrence, the site with the worst prognosis was scored as the first site. Subsequently, recurrence was defined as new locoregional recurrence only: in‐transit metastasis or satellites, regional nodal recurrence in similar SN basin (with or without concurrent locoregional disease), or distant nodal or systemic recurrence (with or without concurrent regional nodal and/or locoregional disease). As the type of recurrence does not have clinical consequences at first (all patients with recurrence will undergo several diagnostic tests anyway) and to retain as much statistical power and as few methodological issues as possible, all recurrence was the outcome used for the prediction model. Median follow‐up from date of SLNB to date of last follow‐up was calculated, applying the reversed Kaplan–Meier method; deaths were censored. Disease‐free survival (DFS) was calculated from date of SLNB to date of first recurrence; lost to follow‐up or death was censored. Melanoma‐specific survival (MSS) was calculated from date of SLNB to date of MSM; lost to follow‐up or death from other causes was censored.

Statistical analysis

The checklist proposed by the AJCC was used for guidance in building a high‐quality prediction model20. Associations between possible prognostic factors and recurrence were studied with Cox regression analysis. The following nine variables were identified as possible prognostic factors based on clinical experience, literature review and availability of sufficient data: sex, age, ulceration, location, histology, Breslow thickness, level of invasion (Clark level), total number of SNs removed and multiple SN fields. To make efficient use of the available data an advanced multiple imputation of missing values strategy (5 imputations) was applied21. The possible non‐linearity of the continuous variables (age, Breslow thickness and total number of SNs removed) was modelled by logarithmic transformation. Independent prognostic factors were selected with multivariable backward selection. Linear predictor values (the sum of truncated predictor values times their predictor effects) were scaled and rounded to a risk score with integer values between 0 and 100. Because recurrence and MSM are strongly related, the final recurrence prediction model based on data from all four EORTC centres was used as a basis for predicting MSM, where the baseline hazard and the slope of the recurrence prediction model were calibrated to MSM22. The advantage of this approach is that it is possible to obtain a unique risk score for each patient that translates into probabilities of both outcomes of interest: recurrence and MSM. This is in contrast to developing two independent prediction models that result in two independent risk scores with corresponding probabilities. To test the validity of this approach, the performance of an independently developed MSM prediction model was compared with that of the calibrated MSM prediction model. The absolute risk prediction of each of the two outcomes was plotted against the risk score. To reduce the overestimation of events occurring in patients with extremely high scores, the score was truncated at an integer of 15, which corresponded to the 95th percentile of score distribution in the cohort. Model performance was assessed by examining discrimination and calibration. Discrimination was measured using the concordance index (c‐index); the closer the c‐index is to 1, the better the discrimination, and a value of 0·5 indicates that the model is no better than chance23. Calibration was assessed visually by plotting the predicted probability against the actual observed frequency in quintiles of predicted recurrence and melanoma‐specific mortality. A 45° line indicates perfect calibration (when the predictive value of the model perfectly matches the patient's actual risk). Any deviation above or below the 45° line indicates underprediction or overprediction respectively. To evaluate the generalizability of the model across different centres, an internal–external cross‐validation was performed in which the model was fitted using data from three centres and validated in the centre that was left out24. A nomogram was developed for graphical presentation of the models. All statistical tests were two‐sided with a statistical significance level set at P < 0·050. Statistic analyses were performed with IBM SPSS® 22.0 (IBM, Armonk, New York, USA) and R version 3.3.2 (R Foundation for Statistical Computing, Vienna, Austria).

Results

From the 3220 patients identified with melanoma and negative SNs, 3180 were eligible for inclusion in the present study. Patients were excluded due to duplicates (9), urogenital melanoma (8), in situ melanoma (7), SLNB for recurrent disease (2), missing data (4), or discrepancy between date of recurrence and date of diagnosis and/or SLNB (10). Baseline patient and tumour characteristics for all patients and per EORTC centre are shown in Table 1.

Table 1

Baseline patient and tumour characteristics by centre

	All (n = 3180)	EORTC centres
		Centre 1 (n = 398)	Centre 2 (n = 1082)	Centre 3 (n = 953)	Centre 4 (n = 747)
Age (years)*	55 (44–67)	51 (40–62)	63 (49–71)	51 (42–62)	55 (44–65)†
Sex
F	1668 (52·5)	211 (53·0)	478 (44·2)	589 (61·8)	390 (52·2)
M	1510 (47·5)	187 (47·0)	604 (55·8)	364 (38·2)	355 (47·5)
Missing	2 (0·1)	0 (0)	0 (0)	0 (0)	2 (0·3)
Anatomical site
Arm	556 (17·5)	74 (18·6)	187 (17·3)	180 (18·9)	115 (15·4)
Leg	996 (31·3)	146 (36·7)	255 (23·6)	369 (38·7)	226 (30·3)
Trunk	1360 (42·8)	162 (40·7)	517 (47·8)	390 (40·9)	291 (39·0)
Head and neck	259 (8·1)	16 (4·0)	123 (11·4)	13 (1·4)	107 (14·3)
Missing	9 (0·3)	0 (0)	0 (0)	1 (0·1)	8 (1·1)
Histological type
SSM	1739 (54·7)	204 (51·3)	762 (70·4)	307 (32·2)	466 (62·4)
NM	885 (27·8)	134 (33·7)	204 (18·9)	353 (37·0)	194 (26·0)
ALM	93 (2·9)	10 (2·5)	39 (3·6)	23 (2·4)	21 (2·8)
LMM	139 (4·4)	5 (1·3)	42 (3·9)	75 (7·9)	17 (2·3)
Other	46 (1·4)	9 (2·3)	1 (0·1)	4 (0·4)	32 (4·3)
Missing	278 (8·7)	36 (9·0)	34 (3·1)	191 (20·0)	17 (2·3)
	(n = 3125)	(n = 392)	(n = 1069)	(n = 926)	(n = 738)
Breslow thickness (mm)*	1·70 (1·10–3·00)	1·90 (1·40–2·80)	1·30 (0·88–2·40)	2·00 (1·00–4·00)	1·70 (1·20–2·70)
Clark level
I–II	271 (8·5)	13 (3·3)	60 (5·5)	180 (18·9)	18 (2·4)
III	1230 (38·7)	147 (36·9)	400 (37·0)	479 (50·3)	204 (27·3)
IV	1354 (42·6)	188 (47·2)	569 (52·6)	219 (23·0)	378 (50·6)
V	140 (4·4)	18 (4·5)	31 (2·9)	41 (4·3)	50 (6·7)
Missing	185 (5·8)	32 (8·0)	22 (2·0)	34 (3·6)	97 (13·0)
Ulceration
No	2264 (71·2)	242 (60·8)	874 (80·8)	604 (63·4)	544 (72·8)
Yes	788 (24·8)	92 (23·1)	182 (16·8)	339 (35·6)	175 (23·4)
Missing	128 (4·0)	64 (16·1)	26 (2·4)	10 (1·0)	28 (3·7)
Mitosis
No	39 (1·2)	11 (2·8)	0 (0)	0 (0)	28 (3·7)
Yes	112 (3·5)	59 (14·8)	0 (0)	0 (0)	53 (7·1)
Missing	3029 (95·3)	328 (82·4)	1082 (100)	953 (100)	666 (89·2)
	(n = 3039)	(n = 397)	(n = 1072)	(n = 823)	(n = 747)
Total no. of SNs*	1 (1–2)	2 (1–2)	1 (1–2)	1 (1–1)	2 (2–3)
Multiple SN fields
No	2768 (87·0)	337 (84·7)	918 (84·8)	953 (100)	560 (75·0)
Yes	412 (13·0)	61 (15·3)	164 (15·2)	0 (0)	187 (25·0)

Values in parentheses are percentages unless indicated otherwise;

values are median (i.q.r.).

Based on 741 patients. EORTC, European Organization for Research and Treatment of Cancer; SSM, superficial spreading melanoma; NM, nodular melanoma; ALM, acral lentiginous melanoma; LMM, lentigo maligna melanoma; SN, sentinel node.

Median duration of follow‐up for all survivors was 61 (i.q.r. 29–99) months. Recurrence occurred in 496 patients (15·6 per cent). The DFS rate at 5 and 10 years was 86·7(s.e. 0·7) and 72·8(1·3) per cent respectively. Some 277 patients (8·7 per cent) died from melanoma. The MSS rate at 5 and 10 years was 91·5(0·6) and 84·8(1·0) per cent respectively. Details of outcome and follow‐up for all patients and per EORTC centre are depicted in Table 2.

Table 2

Outcomes and follow‐up by centre

	All (n = 3180)	EORTC centres
		Centre 1 (n = 398)	Centre 2 (n = 1082)	Centre 3 (n = 953)	Centre 4 (n = 747)
Recurrence
Yes	496 (15·6)	91 (22·9)	94 (8·7)	191 (20·0)	120 (16·1)
No	2684 (84·4)	307 (77·1)	988 (91·3)	762 (80·0)	627 (83·9)
Recurrence type†
Locoregional	142 (28·6)	34 (37)	25 (27)	38 (19·9)	45 (37·5)
Regional nodal	122 (24·6)	14 (15)	32 (34)	48 (25·1)	28 (23·3)
Distant	194 (39·1)	43 (47)	37 (39)	67 (35·1)	47 (39·2)
Unknown	38 (7·7)	0 (0)	0 (0)	38 (19·9)	0 (0)
Additional surgery‡
Yes	13 (0·4)	13 (3·3)	0 (0)	0 (0)	0 (0)
n.r.	3167 (99·6)	385 (96·7)	1082 (100)	953 (100)	747 (100)
Radiotherapy
Yes	24 (0·8)	24 (6·0)	0 (0)	0 (0)	0 (0)
n.r.	3156 (99·2)	374 (94·0)	1082 (100)	953 (100)	747 (100)
Chemotherapy
Yes	12 (0·4)	12 (3·0)	0 (0)	0 (0)	0 (0)
n.r.	3168 (99·6)	386 (97·0)	1082 (100)	953 (100)	747 (100)
Novel therapy§
Yes	22 (0·7)	16 (4·0)	0 (0)	0 (0)	6 (0·8)
n.r.	3158 (99·3)	382 (96·0)	1082 (100)	953 (100)	741 (99·2)
Duration of follow‐up for survivors (months)*	61 (29–99)	94 (59–131)	33 (12–68)	87 (50–127)	57 (35–84)
Status
No evidence of disease	2736 (86·0)	318 (79·9)	984 (90·9)	786 (82·5)	648 (86·7)
Alive with disease	90 (2·8)	12 (3·0)	23 (2·1)	20 (2·1)	35 (4·7)
Died from disease	277 (8·7)	56 (14·1)	41 (3·8)	139 (14·6)	41 (5·5)
Died from other cause	75 (2·4)	12 (3·0)	34 (3·1)	8 (0·8)	21 (2·8)
n.r.	2 (0·1)	0 (0)	0 (0)	0 (0)	2 (0·3)

Values in parentheses are percentages unless indicated otherwise;

values are median (i.q.r.).

Defined as follows: locoregional recurrence only (for example in‐transit metastasis or satellites), regional nodal recurrence similar to sentinel node basin (with or without concurrent locoregional disease) and distant recurrence (with or without concurrent locoregional and/or regional nodal disease).

Includes resection of metastases or lymph node dissection.

Includes vaccines, targeted therapy and immunotherapy. EORTC, European Organization for Research and Treatment of Cancer; n.r., not reported.

Table 3 gives the results of the multivariable Cox model for recurrence including all nine candidate variables. After backwards selection and manual exclusion of Clark level (due to limited additional effect and current clinical practice), the final model for recurrence included three independent prognostic factors: ulceration, anatomical site and Breslow thickness (Table 4). The non‐linearity of Breslow thickness was highly significant (P < 0·001) and well represented by logarithmic transformation. The c‐index for the final recurrence model was 0·74 (95 per cent c.i. 0·71 to 0·76). In cross‐validation, the c‐index for a model based on three centres and applied to the centre that was left out ranged from 0·70 to 0·77. The additional prognostic value of mitotic rate (at least 1 mitosis/mm2 present in 112 of 151 observations), tested in a model with the linear predictor as an offset, was not significant (P = 0·678). The recurrence model was reasonably calibrated across the four centres in cross‐validation (Fig. S1, supporting information).

Table 3

Multivariable Cox analysis of recurrence

	Hazard ratio	P
Age (i.q.r. 67 versus 44 years)	1·06 (0·82, 1·36)	0·920
Sex
F	1·00 (reference)
M	1·20 (0·99, 1·45)	0·065
Breslow thickness (i.q.r. 3·0 versus 1·1 mm)	2·47 (1·94, 3·13)	< 0·001
Ulceration
No	1·00 (reference)
Yes	1·84 (1·50, 2·26)	< 0·001
Clark level		0·005
I–II	1·00 (reference)
III	1·59 (0·97, 2·61)
IV	1·68 (1·02, 2·75)
V	2·70 (1·51, 4·80)
Anatomical location		0·001
Arm	1·00 (reference)
Leg	1·38 (1·03, 1·87)
Trunk	1·54 (1·15, 2·07)
Head and neck	2·12 (1·45, 3·11)
Histology		0·336
SSM	1·00 (reference)
NM	1·18 (0·93, 1·49)
ALM	1·53 (0·94, 2·51)
LMM	1·25 (0·77, 2·03)
Other	0·89 (0·45, 1·79)
No. of SNs (i.q.r. 2 versus 1)	1·06 (0·87, 1·29)	0·800
Multiple SN fields
No	1·00 (reference)
Yes	1·15 (0·82, 1·62)	0·411

Values in parentheses are 95 per cent confidence intervals. SSM, superficial spreading melanoma; NM, nodular melanoma; ALM, acral lentiginous melanoma; LMM, lentigo maligna melanoma; SN, sentinel node.

Table 4

Final model for recurrence

	Hazard ratio	P
Breslow thickness (i.q.r. 3·0 versus 1·1 mm)	2·22 (1·97, 2·51)	< 0·001
Ulceration
No	1·00 (reference)
Yes	1·85 (1·52, 2·25)	< 0·001
Anatomical site
Arm	1·00 (reference)
Leg	1·35 (1·01, 1·81)	0·044
Trunk	1·55 (1·17, 2·05)	0·002
Head and neck	2·39 (1·66, 3·44)	< 0·001

Values in parentheses are 95 per cent confidence intervals.

The association between the linear predictors of recurrence and MSM was even stronger (calibration slope 1·10, 95 per cent c.i. 0·96 to 1·24). The c‐index for the calibrated MSM model was 0·76 (0·73 to 0·79). In cross‐validation, the c‐index for a calibrated model based on three centres applied to the centre that was left out ranged from 0·73 to 0·80. The calibrated model was reasonably calibrated across the four centres in cross‐validation (Fig. S2, supporting information). The performance of this calibrated MSM prediction model, based on the baseline hazard and the slope of the recurrence model, was similar to that of the independently developed MSM prediction model (c‐index 0·77, 0·74 to 0·80) (Table 5).

Table 5

Final model for melanoma‐specific mortality

	Hazard ratio	P
Breslow thickness (i.q.r. 3·0 versus 1·1 mm)	2·37 (2·03, 2·78)	< 0·001
Ulceration
No	1·00 (reference)
Yes	2·11 (1·62, 2·75)	< 0·001
Anatomical site
Arm	1·00 (reference)
Leg	0·97 (0·66, 1·44)	0·881
Trunk	1·70 (1·18, 2·44)	0·004
Head and neck	1·80 (1·07, 3·03)	0·028

Values in parentheses are 95 per cent confidence intervals.

A three‐item risk score was developed, assigning points to each prognostic factor based on the magnitude of association with recurrence. A nomogram to calculate the score and the risk of recurrence and MSM is presented in Fig. 1. The scores were divided into three classes based on the score distribution (each consisting of approximately one‐third of the cohort): low risk, score 0–6; intermediate risk, score 7–9; high risk, score 10 or more (Fig. 1). For recurrence, these risk classes correspond to the following probabilities: low risk, 1·6–8·2 per cent; intermediate risk, 11·0–18·0 per cent; high risk, 23·0 per cent or above. For MSM, these risk classes correspond to the following probabilities: low risk, 0·5–3·2 per cent; intermediate risk, 4·4–8·0 per cent; high risk, 11·0 per cent or more.

Figure 1

The curves refer to predicted recurrence or melanoma‐specific mortality (MSM) at 5 years. The histogram refers to the risk score distribution in the cohort; each bar represents the proportion of patients in the cohort that was assigned that specific score. The histogram was divided in tertiles: light pink bars, first tertile (low risk); medium pink bars, second tertile (intermediate risk); dark pink bars, third tertile (high risk). The nomogram incorporates three factors: ulceration, anatomical location and Breslow thickness. To calculate an individual's probability of 5‐year recurrence and MSM, values for the prognostic factors must be determined first (for example: ulceration; leg; Breslow thickness 2·5 mm). Second, for each value the corresponding points can be obtained by drawing a line from each value towards the points axis (in example: 2, 1 and 7 points respectively). Third, the points must be added up to obtain the total risk score (in example: risk score of 10). Finally, the 5‐year recurrence and MSM probability can be read by moving vertically from the x‐axis (total risk score) to the predicted risk curves and corresponding probabilities on the left y‐axis (for example: 23·0 per cent for recurrence and 11·0 per cent for MSM). The percentage of patients in the entire population (3180) that also had a total risk score of 10 can be determined from the histogram, as well as the corresponding percentage of patients on the right y‐axis (for example: 17·5 per cent)

Discussion

Patients with melanoma are staged according to the AJCC staging system, based on TNM criteria12. Within these stage groupings there is still marked prognostic heterogeneity, and several clinical prognostic tools have been developed to improve predictive accuracy25. None of these tools focuses specifically on outcomes in patients with negative SNs, and most predict only survival25, 26.

The present study developed and validated a nomogram to predict recurrence and MSM in patients with melanoma and negative SNs. Focusing on specifically these patients is important for several reasons. They comprise a large group with highly varying prognosis, who are generally offered regular surveillance examinations with the intent to detect early (locoregional) recurrence. Follow‐up strategies vary, but usually focus on regular clinical examination for 5–10 years15, 19. Some guidelines support a one‐off follow‐up visit with instructions for subsequent self‐examination after treatment for stage IA melanoma27. The recurrence rate for stage IA disease is reported to be 5 per cent28.

Besides personalized outcome prediction, the nomogram could be used to group patients. In the present study population, approximately one‐third of patients with a negative SN had an 8·2 per cent or less predicted probability of recurrence (risk score 6 or less). Surveillance strategies could be reduced in these patients, particularly as most recurrences are self‐detected, and less frequent follow‐up seems to have no effect on recurrence and self‐detection rates, and no adverse effects29, 30.

However, a 5‐year risk of relapse of 48 per cent has been reported for stage IIIA melanoma31. In the present study population, approximately one‐fifth had a 30 per cent or greater predicted recurrence probability (risk score 11 or more). Surveillance strategies could be intensified in these patients, or they could be considered for adjuvant therapy (trials). The present nomogram could aid in designing clinical trials by defining inclusion criteria, or help gain better equivalence between study arms.

In the current era of effective novel therapies in both the adjuvant and therapeutic setting it is highly relevant to focus on negative SN melanoma, as it is likely that most patients with negative SNs will not be offered adjuvant therapy before first recurrence. Mortality predictions in these patients might be partly affected in the present study, as those who developed recurrent disease after 2011 were eligible to receive effective therapy.

This study has important strengths, including its large size, widely available and easily ascertainable characteristics, multicentre composition, and outcomes that are of interest to both clinicians and patients. In multivariable analysis, ulceration, anatomical site and Breslow thickness proved to be significant independent prognostic factors, in concordance with previous reports10, 11, 13, 16, 32. Clark level is no longer part of the seventh AJCC staging edition for melanoma because it was shown not to be an independent prognostic factor when corrected for mitotic rate17. As its effect was marginal in the multivariable model for recurrence, Clark level was excluded manually. All patients were treated at multidisciplinary high‐volume European melanoma centres that applied similar international guideline criteria. This minimizes variability in the interpretation of results and, as generally a policy of centralized referral of patients with melanoma eligible for SLNB is recommended, the cohort is likely to be representative of the European melanoma population with a negative SN. Another strength of the nomogram is the model performance. Discrimination and calibration were good for both the recurrence model and the calibrated model for MSM. The performance of the calibrated model for MSM was comparable to the independently developed model for MSM, indicating the validity of the applied approach. Furthermore, the models were successful in cross‐validation and showed good agreement between prediction and actual observation. Validation of the nomogram is essential to avoid overfitting and to determine generalizability33. In the present study, the prediction models were validated using the recommended internal–external validation procedure. One centre at a time was left out to cross‐validate a model developed in the other centres; as this split was not random, it qualifies as external validation24. Previous prediction models did not focus on SN‐negative melanoma25, 26. The AJCC online prognostic calculator focused on localized melanoma but included both clinical and pathological stage I–II disease (thus also patients who did not undergo SLNB) and predicted only melanoma‐specific survival34. In addition, all tools but one predict survival (disease‐specific or overall)25, 35. The online Sunbelt predictor (MelanomaCalculator.com) included patients staged by SLNB (both positive and negative) and calculates overall survival, as well as DFS and locoregional recurrence‐free survival; however, only the methodology for predicting overall survival was published36.

This study also has several limitations. The first is the retrospective design, which has inherent biases. In addition, other prognostic factors such as regression or lymphatic invasion37, 38 could not be incorporated in the present models due to insufficient data. They could be incorporated in next‐generation nomograms. Another variable shown to have an independent prognostic effect is mitotic rate17, 32. The prognostic effect of mitotic rate was tested by introducing it as an offset term, but it was not significant. This study did not perform competing‐risk analysis, which has been done before26. Consequently the predictions are an overestimate of the actual risk, but, owing to relatively few competing events, this overestimation is expected to be limited. Presently, there is no online version of the nomogram, but it is hoped to have this available soon.

Fig. S1. Calibration plots for the recurrence model. The predicted probability is plotted on the x‐axis, the actual probability on the y‐axis. A plot along the 45° line would indicate perfect calibration in which the predicted probabilities were identical to the actual outcomes

Fig. S2. Calibration plots for the calibrated melanoma‐specific mortality model. The predicted probability is plotted on the x‐axis, the actual probability on the y‐axis. A plot along the 45° line would indicate perfect calibration in which the predicted probabilities were identical to the actual outcomes

Click here for additional data file.