Use of the Living Kidney Donor Profile Index in the Canadian Kidney Transplant Recipient Population: A Validation Study.

BACKGROUND: The Living Kidney Donor Profile Index (LKDPI) was derived in a cohort of kidney transplant recipients (KTR) from the United States to predict the risk of total graft failure. There are important differences in patient demographics, listing practices, access to transplantation, delivery of care, and posttransplant mortality in Canada as compared with the United States, and the generalizability of the LKDPI in the Canadian context is unknown.
OBJECTIVE: The purpose of this study was to externally validate the LKDPI in a large contemporary cohort of Canadian KTR.
DESIGN: Retrospective cohort validation study.
SETTING: Toronto General Hospital, University Health Network, Toronto, Ontario, Canada. PATIENTS: A total of 645 adult (≥18 years old) living donor KTR between January 1, 2006 and December 31, 2016 with follow-up until December 31, 2017 were included in the study. MEASUREMENTS: The predictive performance of the LKDPI was evaluated. The outcome of interest was total graft failure, defined as the need for chronic dialysis, retransplantation, or death with graft function.
METHODS: The Cox proportional hazards model was used to examine the relation between the LKDPI and total graft failure. The Cox proportional hazards model was also used for external validation and performance assessment of the model. Discrimination and calibration were used to assess model performance. Discrimination was assessed using Harrell's C statistic and calibration was assessed graphically, comparing observed versus predicted probabilities of total graft failure.
RESULTS: A total of 645 living donor KTR were included in the study. The median LKDPI score was 13 (interquartile range [IQR] = 1.1, 29.9). Higher LKDPI scores were associated with an increased risk of total graft failure (hazard ratio = 1.01; 95% confidence interval [CI] = 1.0-1.02; P = .02). Discrimination was poor (C statistic = 0.55; 95% CI = 0.48-0.61). Calibration was as good at 1-year posttransplant but suboptimal at 3- and 5-years posttransplant. LIMITATIONS: Limitations include a relatively small sample size, predicted probabilities for assessment of calibration only available for scores of 0 to 100, and some missing data handled by imputation.
CONCLUSIONS: In this external validation study, the predictive ability of the LKDPI was modest in a cohort of Canadian KTR. Validation of prediction models is an important step to assess performance in external populations. Potential recalibration of the LKDPI may be useful prior to clinical use in external cohorts.

What was known before

It is well established that there is a spectrum of quality among transplanted kidneys, and a number of donor, transplant, and recipient factors are known to affect graft survival after transplantation. The Living Kidney Donor Profile Index (LKDPI) uses a number of living donor/recipient characteristics to predict the probability of all-cause graft loss. However, the LKDPI was derived in a large US cohort of living donor kidney transplant recipients and the performance of the LKDPI in other populations is unknown.

What this adds

The LKDPI is a tool often used in the context of clinical decision-making, and therefore it is essential to assess the predictive performance of the LKDPI in an external cohort to determine the generalizability of this score to other kidney transplant recipient populations prior to clinical use. The results of our study demonstrate that in a large contemporary cohort of Canadian living donor kidney transplant recipients, the LKDPI was only modestly predictive. Much of the emphasis in prediction modeling has been placed on model development, and our study demonstrates the importance of also appropriately assessing model performance in external populations.

Introduction

The prevalence of patients with end-stage renal disease (ESRD) continues to increase resulting in an ongoing demand for life-saving renal replacement therapy.[1] Kidney transplantation is the best renal replacement therapy for patients with ESRD as it improves patient survival and quality of life as compared with chronic dialysis therapy.[2,3] Living donor kidney transplantation (LDKT) is the preferred treatment option compared with deceased donor kidney transplantation (DDKT), as it is associated with superior graft and patient survival.[4,5] In addition, LDKT may lead to a reduction in waiting time and allows blood type or crossmatch incompatible donor/recipient pairs the opportunity to receive compatible transplants through kidney paired donation programs.[6,7]

It is well established that there is a spectrum of quality among transplanted kidneys, and a number of donor, transplant, and recipient factors are known to affect graft survival after transplantation.[8,9] There has been an emergence of clinical prediction models, primarily derived in populations of deceased donor kidney transplant recipients (KTR), that integrate numerous clinical characteristics to predict the probability of graft failure. For example, Rao et al developed the Kidney Donor Risk Index (KDRI), which is a continuous numerical risk score that quantifies the risk of graft failure among deceased donor KTR based on 10 donor and 4 transplant-specific characteristics.[10] Subsequently, the Kidney Donor Profile Index (KDPI) was derived from the KDRI by removing transplant-specific parameters to assess donor kidney quality with donor-specific parameters alone, where an increasing KDRI/KDPI score is associated with an increased risk of graft failure.[4]

There was a need for a comparable risk score for living donor KTR, which (1) can be useful in selecting a living kidney donor where multiple candidates are available or (2) can be used to compare the quality of an approved living kidney donor with a deceased donor kidney for a potential recipient. In addition, a risk score for living donor KTR could also be useful in the setting of kidney paired donation, to determine which compatible living kidney donors would be acceptable for a particular recipient. Therefore, Massie et al developed a risk index for LDKT, the Living Kidney Donor Profile Index (LKDPI), where a number of living donor/recipient characteristics are used to predict the probability of all-cause graft loss.[11] Furthermore, the LKDPI allows for comparability between LDKT and DDKT by using the same scale as the KDPI.

External validation of a clinical prediction model prior to integration into practice is essential as it demonstrates that the model can be applied in a population other than the one from which it was derived.[12] The LKDPI was derived in a large US cohort of living donor KTR and the performance of the LKDPI in other populations is unknown. There are important differences in patient demographics, listing practices, access to transplantation, delivery of care, and posttransplant mortality in Canada as compared with the United States, and the generalizability of the LKDPI in the Canadian context is unknown.[13-15] Therefore, the purpose of this study is to externally validate the LKDPI in a contemporary cohort of Canadian living donor KTR.

Methods

Study Design and Participants

This is a retrospective cohort validation study of all adult (≥18 years old) living donor KTR at the Toronto General Hospital, University Health Network (TGH-UHN) between January 1, 2006 and December 31, 2016, with follow-up until December 31, 2017. We excluded all multiorgan transplant recipients, patients with a prior kidney transplant, and patients who received a kidney transplant outside of our institution. Patient follow-up started on the day of transplantation and continued until the conclusion of the observation period, death, graft failure, or loss to follow-up.

Data Sources

This study used data from the Comprehensive Renal Transplant Research Information System (CoReTRIS) at TGH-UHN. CoReTRIS is an in-center research database which houses an extensive set of recipient, donor, transplant, laboratory, outcome, and follow-up data on all KTR at TGH-UHN since January 1, 2000.[16]

Exposure and Outcome Assessment

The outcome of interest was total graft failure, defined as the need for chronic dialysis, retransplantation, or death with graft function. Total graft failure was chosen as the outcome of interest for this validation study as the LKDPI score was developed to predict total graft failure.[11] The LKDPI score was calculated using the following variables: donor age, sex, estimated glomerular filtration rate (eGFR), systolic blood pressure, body mass index (BMI), race, history of cigarette use, recipient sex, biological relationship between donor and recipient, ABO compatibility between donor and recipient, donor/recipient weight ratio, and donor/recipient HLA-B and HLA-DR mismatches.[11]

Statistical Analysis

Baseline characteristics were assessed using parametric and nonparametric methods as appropriate. Missing covariate data were handled using the following imputation method: For continuous variables, the median value for the covariate was used. For categorical variables, a value was randomly imputed based on the distribution of the nonmissing values for the specific covariate. The Kaplan-Meier product limit method was used to assess time to total graft failure. The Cox proportional hazards model was used to examine the relation between the LKDPI and total graft failure. The Cox proportional hazards model was also used for external validation and performance assessment of the model. Discrimination and calibration were used to assess model performance.[17] Discrimination indicates the proportion of times that a prediction model will correctly identify, within a pair of individuals, the individual that has the longest survival time. Harrell’s C statistic was used to assess discrimination of the LKDPI in the external validation cohort. Generally, a C statistic of 0.7 indicates acceptable discrimination, 0.8 indicates good discrimination and 0.9 indicates excellent discrimination.[18] Calibration refers to the agreement between predicted probabilities derived by the model and the actual observed probabilities in the external validation cohort. To generate predicted total graft survival probabilities for calibration assessment, the baseline hazard function from the original data is required, although this is infrequently reported.[19] In lieu of this, we used publically available total graft survival rates by KDPI score (0-100) based on data from the Organ Procurement and Transplantation Network from 2007 to 2017, which were used as accurate predicted probabilities.[20] These predicted probabilities for total graft survival were then graphically compared with observed probabilities in our external validation cohort. A model that calibrates well should have similar predicted and observed probabilities. Calibration was performed at 1-, 3-, and 5-years posttransplant time points. All statistical analyses were performed using Stata/MP4, version 12.1 (StataCorp, College Station, TX). The study was approved by the Research Ethics Board at TGH-UHN.

Results

Baseline Characteristics of the Cohort

There were 645 first-time adult living donor KTR included in the final cohort. Clinical characteristics of the cohort are shown in Table 1 (donor characteristics), Table 2 (recipient characteristics), and Supplemental Table 1 (donor and recipient characteristics including missing data). The mean donor age was 45.6 ± 12.1 years, 58.7% were female and less than 5% of donors were black. The mean predonation eGFR was 97.3 ± 14.7 mL/min/1.73 m2. The mean body mass index was 26.6 ± 4.2 kg/m2 and mean systolic blood pressure was 119 ± 13.5 mm Hg. Almost half of the living kidney donors were biologically related to their recipient and most donors were ABO compatible with their recipients. The mean recipient age was 46.9 ± 13.6 years and recipients were predominantly male. The mean recipient BMI was 27.1 ± 5.9 kg/m2. Glomerulonephritis was the most common cause of ESRD. The median time on dialysis prior to transplantation was 1.2 (interquartile range [IQR] = 0.0, 2.6) years and 26.7% received a preemptive transplant. Smoking history and coronary artery disease were present in 42.1 and 18.5% of recipients, respectively. In total, 64.5% of patients received a depleting agent as induction therapy (thymoglobulin) and tacrolimus was the most common calcineurin inhibitor prescribed at discharge.

Table 1.

Donor Characteristics of the External Validation Cohort.

Donor characteristics	Validation cohort (N = 645)
Age at donation, y, mean ± SD	45.6 ± 12.1
Donor race, %
Black	4.7
Nonblack	95.3
Donor sex, %
Male	41.3
Female	58.7
Donor eGFR (CKD-EPI), mL/min/1.73 m2, mean ± SD	97.3 ± 14.7
Donor BMI, kg/m2, mean ± SD	26.6 ± 4.2
Donor history of cigarette use, %
Yes	38.1
No	61.9
Donor systolic blood pressure, mm Hg, mean ± SD	119.1 ± 13.5
ABO incompatibility between donor and recipient, %
Yes	3.9
No	96.1
Biological relationship between donor and recipient, %
Related	47.6
Unrelated	52.4
Number of HLA B mismatches, median (IQR)	1 (1, 2)
Number of HLA DR mismatches, median (IQR)	1 (1, 2)
Donor/recipient weight ratio, median (IQR)	1.0 (0.8, 1.2)

Note. eGFR = estimated glomerular filtration rate; BMI = body mass index; CKD-EPI = Chronic Kidney Disease Epidemiology Collaboration; HLA = human leukocyte antigen; IQR = interquartile range.

Table 2.

Recipient Characteristics of the External Validation Cohort.

Recipient characteristics	Validation cohort (N = 645)
Recipient age at transplant, y, mean ± SD	46.9 ± 13.6
Recipient race, %
White	73.4
Nonwhite	26.6
Recipient sex, %
Male	56.0
Female	44.0
Recipient body mass index, kg/m2, mean ± SD	27.1 ± 5.9
Cause of end-stage renal disease, %
Glomerulonephritis	33.3
Diabetes mellitus	22.8
Polycystic kidney disease	14.9
Other	29.0
Time on dialysis, y, median (interquartile range)	1.2 (0.0, 2.6)
Preemptive transplant, %
Yes	26.7
No	73.3
Peak PRA, %
0	59.3
>0	40.7
Recipient history of smoking, %
Yes	42.1
No	57.9
Recipient history of coronary artery disease, %
Yes	18.5
No	81.6
Recipient history of cancer, %
Yes	6.7
No	93.3
Type of induction, %
Nondepleting agent	35.5
Depleting agent	64.5
Calcineurin inhibitor at discharge, %
Tacrolimus	85.3
Cyclosporine	14.7

Note. PRA = panel reactive antibody.

Relationship Between LKDPI and Total Graft Failure

The distribution of LKDPI values in the cohort is shown in Figure 1 and Table 3. The median LKDPI was 13.26 (IQR = 1.10, 29.91), with LKDPI values ranging from −67.2 to 90.7. Most living donor KTR had LKDPI values between 0 and 20 (38%) and 20 and 40 (27%). The median follow-up time was 5 years (IQR = 2.69, 7.81). Over follow-up, there were 52 deaths with graft function and 43 graft failures. There was a relationship between LKDPI and the risk of total graft failure, where every unit increase in the LKDPI, there was an increase in risk of total graft failure (hazard ratio = 1.01; 95% confidence interval [CI] = 1.00-1.02; P = .02). When stratified by subgroup of LKDPI, there was a trend toward an increase in the risk of total graft failure with increasing LKDPI; however, this did not meet statistical significance (Figure 2 and Table 4).

Figure 1.

Distribution of the Living Kidney Donor Profile Index (LKDPI) values in the external validation cohort.

Table 3.

Summary Statistics of the LKDPI Values for the External Validation Cohort.

LKDPI		Number of donors	Total graft failures (N)	Mean ± SD	Median (interquartile range)	Minimum	Maximum
Continuous		645	95	15.78 ± 21.65	13.26 (1.10, 29.91)	−67.23	90.72
Categorized	<0	145 (22.48%)	17	−11.10 ± 10.66	−8.61 (–15.73, –2.99)	−67.23	−0.01
	0-20	245 (37.98%)	31	9.62 ± 5.58	9.79 (5.15, 13.98)	0.01	19.96
	20-40	174 (26.98%)	34	29.34 ± 5.87	29.04 (24.11, 34.43)	20.02	39.99
	>40	81 (12.56%)	13	53.37 ± 12.03	49.23 (44.84, 59.36)	40.11	90.72

Note. LKDPI = Living Kidney Donor Profile Index.

Figure 2.

Cumulative incidence curves for total graft failure by LKDPI subgroups in the external validation cohort.

Note. LKDPI = Living Kidney Donor Profile Index.

Table 4.

Relationship Between LKDPI and Total Graft Failure in the External Validation Cohort.

Outcome	Exposure	LKDPI subgroup	Hazard ratio (95% confidence interval)	P value
Total graft failure	LKDPI	Continuous	1.01 (1.00-1.02)	.02
		≤0 vs 0-20	0.90 (0.50-1.62)	.72
		20-40 vs 0-20	1.69 (1.04-2.75)	.03
		>40 vs 0-20	1.57 (0.82-3.01)	.17

Note. LKDPI = Living Kidney Donor Profile Index.

Performance Assessment of the LKDPI in an External Cohort of Living Donor KTR

Performance of the model was assessed using both discrimination and calibration. In assessing discrimination, the C statistic for the model in our external cohort was 0.55 (95% CI = 0.59-0.61). With respect to calibration, we graphically compared predicted probabilities of total graft survival with observed probabilities of total graft survival in the external cohort at 1-, 3-, and 5-years posttransplant. These graphical comparisons are shown in Figure 3. Although predicted and observed probabilities were similar at the 1-year posttransplant time point, these probabilities were more discordant at 3- and 5-years posttransplant.

Figure 3.

Graphical assessment of calibration: predicted versus observed probabilities of total graft failure at 1-, 3-, and 5-years posttransplantation.

Note. LKDPI = Living Kidney Donor Profile Index.

Discussion

Similar to the large US cohort in which the LKDPI was derived, we demonstrate that an increasing LKDPI score is associated with an increase in the risk of total graft failure in a large cohort of Canadian KTR. However, the LKDPI is a tool often used in the context of clinical decision-making, and therefore it was essential to assess the predictive performance of the LKDPI in an external cohort to determine the generalizability of this score to other KTR populations prior to clinical use.[12] The results of our study demonstrate that in a large contemporary cohort of Canadian living donor KTR, the LKDPI was only modestly predictive.

Measures such as discrimination and calibration are typically used to assess the predictive performance of a model.[17,21] Discrimination is typically reported as a C statistic, which indicates how well a model can differentiate those with and without the event of interest. In the setting of a survival analysis, the C statistic represents the probability that the model will correctly identify the individual among a pair of patients with the longest total graft survival time.[21] In our external validation study, the C statistic of 0.55 indicates poor discrimination. Calibration can be assessed both graphically and statistically and refers to the agreement between predicted probabilities derived by a model and the observed probabilities of an event. A model that calibrates well is one where predicted and observed probabilities are similar.[21] In our study, predicted and observed probabilities of total graft survival were similar at 1-year posttransplantation across a broad range of LKDPI scores. However, the predicted and observed probabilities became more discordant as the time interval after transplantation increased. At 3-years posttransplant, the predicted probabilities derived from the model were lower than the observed total graft survival probabilities, and the discordance between predicted and observed probabilities was most notable where LKDPI scores were ≤5 or ≥60. At 5-years posttransplant, the predicted probabilities were consistently lower than the observed probabilities for total graft survival across all LKDPI subgroups.

Ideally, a prediction model will both discriminate and calibrate well. There may be several reasons why the LKDPI only had modest predictive performance in our Canadian cohort. First, the C statistic of the LKDPI in the original US cohort in which the model was derived was 0.59, and therefore it would be unrealistic to expect the LKDPI to have substantially better discrimination in an external cohort as compared with the original cohort. Second, there may be significant variation in the case-mix between the cohort of US KTR used for the derivation of the LKDPI and our Canadian cohort used for external validation. There are known differences in patient demographics, comorbid conditions, delivery of health services, and socioeconomic factors such as financial coverage for immunosuppressive medications between Canada and the United States.[13-15] These differences have been postulated to be important factors explaining the differences in transplant outcomes in various countries and may explain why the LKDPI may not be that generalizable to our cohort. Various studies have demonstrated differences in posttransplant graft survival and mortality,[13-15] which may account for the poor calibration seen at 5-years posttransplant where the predicted probabilities of total graft survival were consistently lower than the actual observed probabilities of total graft survival in our Canadian cohort. Both the US and Canadian populations had similar median LKDPI scores, reflecting similar quality of living donor kidneys, suggesting that the differences in total graft survival observed between cohorts may related to recipient differences. Third, from a statistical perspective, the number of total graft failure events in our external validation cohort was relatively small, and the sample size was reduced when the cohorts were stratified by LKDPI score to assess calibration. This may also be factor contributing to the modest predictive performance of the LKDPI in our cohort.

It is well known that there is a spectrum of quality among donor kidneys, and thus risk scores such as the LKDPI are useful in guiding clinicians in clinical decision-making.[22] The LKDPI is particularly advantageous over other clinical prediction models of graft survival as it was derived on the same scale as the KDPI.[23] However, prior to integration into clinical use, the performance of any clinical prediction model should be considered. Although the LKDPI is only modestly predictive in our cohort, it still may be clinically useful, particularly if the LKDPI allows for a better assessment of total graft failure as compared with the current state (ie, clinician opinion alone). Furthermore, when clinical prediction models perform suboptimally in external populations, there is an opportunity to revise the original model (ie, inclusion of potentially important new predictors) or recalibrate the model (ie, updating the intercept of the model) to improve generalizability.

Our study has several strengths. This is the first study, to our knowledge, that has validated the LKDPI in a large external Canadian cohort. Validation of the LKDPI in European and Australian cohorts has demonstrated poor discrimination as well; however, calibration of the LKDPI in these populations was not reported.[24,25] In external validation, calibration is particularly important as it allows for an assessment of the accuracy of the absolute risk estimates generated by a model. In the case of the LKDPI, without assessing calibration, it would be unknown whether a 10% risk of total graft failure at 3-years posttransplant associated with an LKDPI score of 20 would be accurate in an external cohort. Our study demonstrates that calibration in our cohort is good at 1-year posttransplant and suboptimal at 3- and 5-years posttransplantation. As such, the LKDPI may be only most useful to predict early total graft survival in external cohorts. The results of our study highlight the importance of external validation, assessing both discrimination and calibration, prior to clinical use. The limitations of our study must be acknowledged. First, the relatively small sample size might have affected the performance of the LKDPI in our cohort. In particular, although the performance of the LKDPI in our cohort was modest, it is possible that the model may perform better in certain subgroups (ie, in those with LKDPI scores > 40). However, due to the small number of patients in each subgroup, this could not be specifically tested. Second, this is a single center study and transplant outcomes may differ in other Canadian centers. Third, the baseline hazard was not available and therefore publically available predicted probabilities were used for assessment of calibration. Although these probabilities are accurate, they were only publicly available for scores of 0 to 100 and thus calibration could not be performed in circumstances where the LKDPI was <0. Fourth, our data set was not complete and missing data were handled using imputation and thus may have affected the robustness of our data. Finally, we chose to validate the LKDPI as it is commonly used in clinical practice; however, other predictive models of graft failure in living donor KTR exist. Tiong et al[26] and Akl et al[27] derived clinical prediction models for graft failure in living donor KTR using regression modeling and artificial neural networks. These models included between 8 and 21 variables. Interestingly, despite having better predictive performance as compared with the LKDPI (C statistics between 0.71 and 0.94), they are less widely used in clinical practice. Simple, easy to use, and parsimonious models are more likely to be integrated into clinical practice, as are models available in a Web-based or mobile application format.[28] Furthermore, models that include variables that are not easily available to the clinician at the time of desired use (ie, kidney biopsy findings) may also limit applicability in clinical practice. Further work is needed to develop easy to use, parsimonious, and highly predictive models in kidney transplantation.

In summary, the results of our external validation study demonstrate that the LKDPI only modestly predicts total graft survival in a large cohort of Canadian living donor KTR. Much of the emphasis in prediction modeling has been placed on model development, and our study demonstrates the importance of also appropriately assessing model performance in external populations. Suboptimal performance of prediction models in external cohorts is not uncommon and updating or recalibration of the LKDPI can be done to potentially improve the predictive performance. Finally, further study is needed to understand whether the implementation of clinical prediction models such as the LKDPI results in better decision-making than clinical judgment alone, and subsequent improvement in transplant outcomes.

Click here for additional data file.

Supplemental material, Supplementary_Table_1_3 for Use of the Living Kidney Donor Profile Index in the Canadian Kidney Transplant Recipient Population: A Validation Study by Mohamed Shantier, Yanhong Li, Monika Ashwin, Olsegun Famure and Sunita K. Singh in Canadian Journal of Kidney Health and Disease