An Intelligent Clinical Decision Support System for Predicting Acute Graft-versus-host Disease (aGvHD) following Allogeneic Hematopoietic Stem Cell Transplantation.

BACKGROUND: Acute graft-versus-host disease (aGvHD) is a complex and often multisystem disease that causes morbidity and mortality in 35% of patients receiving allogeneic hematopoietic stem cell transplantation (AHSCT).
OBJECTIVE: This study aimed to implement a Clinical Decision Support System (CDSS) for predicting aGvHD following AHSCT on the transplantation day.
MATERIAL AND METHODS: In this developmental study, the data of 182 patients with 31 attributes, which referred to Taleghani Hospital Tehran, Iran during 2009-2017, were analyzed by machine learning (ML) algorithms which included XGBClassifier, HistGradientBoostingClassifier, AdaBoostClassifier, and RandomForestClassifier. The criteria measurement used to evaluate these algorithms included accuracy, sensitivity, and specificity. Using the machine learning developed model, a CDSS was implemented. The performance of the CDSS was evaluated by Cohen's Kappa coefficient.
RESULTS: Of the 31 included variables, albumin, uric acid, C-reactive protein, donor age, platelet, lactate Dehydrogenase, and Hemoglobin were identified as the most important predictors. The two algorithms XGBClassifier and HistGradientBoostingClassifier with an average accuracy of 90.70%, sensitivity of 92.5%, and specificity of 89.13% were selected as the most appropriate ML models for predicting aGvHD. The agreement between CDSS prediction and patient outcome was 92%.
CONCLUSION: ML methods can reliably predict the likelihood of aGvHD at the time of transplantation. These methods can help us to limit the number of risk factors to those that have significant effects on the outcome. However, their performance is heavily dependent on selecting the appropriate methods and algorithms. The next generations of CDSS may use more and more machine learning approaches. Copyright: © Journal of Biomedical Physics and Engineering.

Introduction

Acute graft-versus-host disease (aGvHD) is a complex and often multisystem disease that causes morbidity and mortality in 35%-50% of patients receiving allogeneic hematopoietic stem cell transplantation (AHSCT) [ 1 ]. On the first 100 days after transplantation, donor T cells invade the host tissue and lead to dysfunction of the skin, gastrointestinal tract, and liver [ 1 - 4 ]. Given that it occurs at the stage of severe tissue damage, its diagnosis is late [ 5 ].

In recent years, biomarkers related to aGvHD have been considered as a tool in predicting the occurrence [ 5 ]. But the multiplicity of these biomarkers and the complexity of the various factors, contributing to the disease have made an accurate quick decision difficult. Besides, in previous studies [ 6 - 8 ]. The analyses performed on these biomarkers was univariate using classical statistical methods [ 9 - 11 ].

Since the 1960s, medical informatics experts have become interested in using clinical decision support systems (CDSS) to classify patient outcomes, reduce health-care costs, and alert physicians about the potential for dangerous medication interactions, resulting in the improvement of physicians’ diagnostic process, and provide diagnostic suggestions, and also increase safety and quality of patient care [ 12 - 15 ].

CDSS is defined as “a computer system designed to impact clinician decision-making about individual patients at the point in time that these decisions are made” [ 13 ]. CDSSs are divided into two categories of knowledge-based and non-knowledge-based [ 13 , 14 ]. In the knowledge-based type, the goal is to build a system that can simulate human thinking. These types of CDSSs use the knowledge as a rule or set of if-then rules in which they are specifically coded in clinical practice guidelines (CPG). Whereas, non-knowledge-based CDSSs use machine learning (ML) algorithms to extract knowledge [ 13 ].

Machine learning (ML) is a subset of artificial intelligence (AI) in which the algorithms, executing the prediction process extract the necessary knowledge from past experiences and/or find patterns in data [ 16 - 18 ]. ML is any process in which an algorithm is improved or “trained” by performing repetition on a training dataset to perform a task, usually a classification or identification [ 16 , 19 ]. The trained algorithms can then be evaluated by measuring its performance based on the test dataset [ 17 , 19 , 20 ].

There are several learning methods in ML, one of the most widely used and the popular of which is supervised learning. The goal of a supervised learning algorithm is to use the dataset to produce a model, taking a feature vector x as an input and outputting information, resulting in deduction if the label for this feature vector [ 20 ].

The two major types of supervised learning are classification and regression. Examples of classification are ensemble methods, K-nearest neighbors, support vector machine, decision trees, random forest, neural networks and so on. Regression examples are linear regression and logistic regression [ 17 , 19 - 21 ].

The ensemble is an ML concept in which the idea is to train multiple models using the same learning algorithm [ 22 ]. Ensemble algorithms are divided into two main types, including boosting and bagging. Ensemble methods include algorithms such as eXtreme Gradient Boosting classifier (XGBClassifier), AdaBoost classifier (AdaBoostClassifier), Histogram-based Gradient Boosting Classification Tree (HistGradientBoostingClassifier), and Random Forest classifier [ 20 ].

The XGBClassifier is a highly adaptable algorithm, working in most classifications. Boosting is a method, seeking to create a strong classifier based on weak classifiers. Weak and strong classification models mention to the correlation of outputs and actual class. By appending classifiers on top of each other iteratively, the next classifier can modify the errors of the previous one. This process is recurred until the training data set is accurately predicted [ 23 ].

The HistGradientBoostingClassifier has support for missing values. During training, the tree grower learns at each split point whether samples with missing values should go to the left or right child, based on the potential gain [ 19 ].

An AdaBoost classifier (AdaBoostClassifier) is one of the most popular algorithms for building robust classifiers with linear combinations of member classifiers. The member classifiers are chosen to minimize the errors in each iteration during the training process [ 24 ].

RandomForestClassifier synthesizes several randomized decision trees and gathers their predictions by averaging. In settings where the number of variables is much greater than the number of observations, this method has shown excellent performance [ 25 ].

Pre-occurrence prediction by these algorithms helps physicians to identify high-risk patients and reduce health care costs by performing time-consuming treatment interventions [ 26 ].

Previous studies have shown that neural network algorithms, support vector machine, naive bayes, K-nearest neighbors, regression, decision trees, and ensemble methods have been used to predict aGvHD [ 27 ]. Although in recent years the decision trees and ensemble methods have been given more attention for predicting aGvHD, there is no evidence that these algorithms are successfully used in the clinical setting [ 28 - 30 ]. Therefore, this study aimed to design, implement, and validate a clinical decision support system using ensemble methods to predict aGvHD following AHSCT on transplant day.

Material and Methods

Data Source, Study Roadmap, and Tools

In a developmental study, 31 variables [ 27 ] (which were classified into two groups: base-line and biomarker), which could potentially affect the transplantation outcome, were gathered on the day of transplantation from 190 patients who received AHSCT in Taleghani Hospital, Tehran, Iran, from 2009 to 2017. Then the CDSS was designed and implemented using Python programming language in four stages as pre-processing, learning, evaluation, and CDSS implementation as is shown in the below roadmap diagram (Figure 1).

Figure 1

Roadmap for building clinical decision support systems based on machine learning.

Pre-processing

Imputing missing value

In this phase, the raw data were imputed using the following two processes:

1- Records and variables with missing values (greater than 50%) were excluded from the dataset.

2- The missing values of continuous and discrete variables were replaced separately with mean and mode in each class, respectively.

Under-sampling

Under-sampling methods normalize the distribution of all classes by decreasing the number of majority class records in the imbalanced dataset [ 31 ]. An imbalanced class distribution will have one or more classes with few samples (the minority classes) and one or more classes with many samples (the majority classes). In this study, the RandomUnderSampler method was used to decrease the number of majority class records [ 32 ]. RandomUnderSampler is a fast and simple method to balance the patient’s dataset by randomly choosing a subset of data for the targeted classes.

Data Splitting

In this phase, patients’ datasets were divided into training and testing sets with a ratio of 70% and 30%, respectively.

Feature Scaling

In this phase, the training and test data sets were scaled separately using the normalizing method (Equation 1) [ 32 ]. The numerical values of the data set are between zero and one.

1) Xnormalized = (X-Xmin)/(Xmax-Xmin)

Feature Selection

In this phase, the Boruta algorithm, which is a type of wrapper method feature selection, was used to select the most important predictors of aGvHD prediction. This method, using RandomForestClassifier algorithm, identifies important features of the dataset as unbiased and stable [ 33 ].

Learning

Hyperparameters are parameters, governing the learning process, but they are not the part of the learning process. Besides, they have a great impact on the performance and results of modeling ML algorithms [ 34 ]. Adjustment of these hyperparameters is considered as an optimization problem and their search is usually done manually using methods such as randomized parameter optimization with k-fold cross-validation method (RandomizedSearchCV) [ 19 , 35 ]. In the present study, the RandomizedSearchCV method was used to optimize the hyperparameters of four ML algorithms, including XGBClassifier, HistGradientBoostingClassifier, AdaBoostClassifier, and RandomForestClassifier.

Evaluation

After modeling the ML algorithms, their performance was evaluated using the accuracy, sensitivity, specificity, F-measure, and AUC (area under the curve) criteria (Equations 2 to 5) [ 36 ].

2)

3)

4)

6) =

A ROC chart is defined by false positive rate (FPR) and true positive rate (TPR) as x and y axes, respectively, depicting relative trade-offs between true positive (TP) and false-positive (FP) [ 36 ].

Where TP is the number of actual patients, predicted correctly to have aGvHD. TN is the number of non-patients, predicted correctly not to have aGvHD. False-positive (FP) is the number of non-patients, predicted incorrectly to have aGvHD, and false-negative (FN) is the number of patients, predicted incorrectly not to have aGvHD [ 36 ].

CDSS

After selecting the most appropriate ML models, a CDSS was designed and implemented using Python programming language and MySQL database management system (Figure 1 Part D).

Then performance of CDSS was evaluated by calculating the agreement between CDSS prediction and the actual patient outcome after 100 days of transplantation, using Cohen’s Kappa coefficient and transplant data 30 patients, receiving AHSCT in 2018 [ 37 ].

Results

a. Patient Characteristics

Table 1 presents the most significant variables for predicting aGvHD.

Table 1

The dataset variables and their descriptions.

Type	Row	Variable	Description	Role
base-line	1	Patient Gender		Input
	2	Donor Gender		Input
	3	Donor-Patient Gender		Input
	4	Patient Blood group		Input
	5	Donor Blood group		Input
	6	Delivery	The process of giving birth for Donor.	Input
	7	Marital Status		Input
	8	Smoking		Input
	9	Blood group Compatibility	Donor and recipient have the same blood group antigens and plasma antibodies.	Input
	10	Donor recipient relationship	The relation between donor and patient gender including Related and Sibling.	Input
	11	Patient Age		Input
	12	Donor Age		Input
biomarker	1	Prophylaxis Regimen	Regimen use for the prevention of a specific disease.	Input
	2	Chemotherapy Regimen	Regimen 1-3: Myeloablative is an intensive conditioning regimen to destroy the bone marrow cells. Regimen 4: Reduced intensity conditioning that uses less chemotherapy and radiation than the Regimen 1-3.	Input
	3	Diagnosis		Input
	4	Complete Remission	Including: tests, physical exams, and scans show that all signs of cancer are gone.	Input
	5	Radiothrapypre Bone Marrow Transplantation	The treatment of disease with ionizing radiation.	Input
	6	White Blood Cells		Input
	7	Platelet count		Input
	8	lactate dehydrogenase (LDH)		Input
	9	cluster of differentiation 3 (CD3)		Input
	10	cluster of differentiation 34 (CD34)	The CD34 antigen identifies on a myeloid leukemia cell line.	Input
	11	mononuclear cell (MNC)		Input
	12	Diagnosis to Transplantation	The time between disease diagnosis and hematopoietic stem cell transplantation	Input
	13	Patient Body mass index		Input
	14	Donor Body mass index		Input
	15	Hemoglobin		Input
	16	Creatinine		Input
	17	Uric Acid		Input
	18	Albumin		Input
	19	C-Reactive Protein (CRP)		Input
		Acute graft-versus-host disease (aGvHD)		Target

b. Pre-processing

After discarding incomplete patient records, the patient dataset was reduced to 182 patients (71 case-patients diagnosed with aGvHD vs. 111 control patients who did not experience aGvHD). As a result of under-sampling, the final number of scaled (normalized) patient records was 142 patients (71 cases vs. 71 controls), of which 70% (99 patients) were selected for the training dataset and 30% (43 cases) for the testing dataset.

The results of feature selection showed that of the 31 included variables, albumin, uric acid, C-Reactive Protein, donor age, platelet, Lactate Dehydrogenase, and Hemoglobin were identified as the seven most important predictors of aGvHD (Table 2) of which, albumin had the highest importance.

Table 2

The most important predictors of acute graft-versus-host disease

Feature	Importance
Albumin	0.409
Uric Acid	0.151
C-Reactive Protein	0.148
Donor age	0.085
Platelet	0.081
Lactate Dehydrogenase	0.071
Hemoglobin	0.055

c. Predictive performance

The results of tuning the hyperparameters of ML algorithms are presented in Table 3.

Table 3

Results of optimized hyperparameters of machine learning algorithms

Classifier	Best F-measure %
*XGBClassifier	94
HistGradientBoostingClassifier	90
AdaBoostClassifier	90
RandomForestClassifier	95

eXtreme Gradient Boosting classifier

The evaluation results of ML models based on the test data set are shown in Figure 2 and Table 4. Based on the evaluation criteria including, accuracy, sensitivity, specificity, F-measure, and area under the curve (AUC), the two algorithms XGBClassifier (eXtreme Gradient Boosting Classifier) and HistGradientBoostingClassifier had the best performance. According to the mean of evaluation criteria, the XGBClassifier algorithm with 90.82 and the lowest number of false negative and false positive had the best performance (Table 4).

Figure 2

Results of classification report and Area under the curve (AUC) curve of machine learning models.

Table 4

Results of performance evaluation of machine learning models

Row	Classifier	Accuracy	Sensitivity	Specificity	F-measure	AUC	Mean
1	XGBClassifier	90.70	95.00	86.96	90.48	90.98	90.82
2	HistGradientBoostingClassifier	90.70	90.00	91.30	90.00	90.65	90.53
-	Average 1 and 2	90.70	92.50	89.13	90.24	90.82	-
3	AdaBoostClassifier	86.05	75.00	95.65	83.33	85.33	85.07
4	RandomForestClassifier	83.72	80.00	86.96	82.05	83.48	83.24

AUC: Area under the curve, XGBClassifier: eXtreme Gradient Boosting classifier

d. CDSS

Using the machine learning developed the model, a CDSS was designed and implemented, which is accessible via the https://agprcdss.ir/ (Figure 3). The agreement between CDSS prediction and the actual outcome that occurred within 100 days after AHSCT was 92%.

Figure 3

Graphical user interface of the clinical decision support system.

Discussion

In this study, we designed and implemented the AGPRC (Acute GvHD Prediction Transplant Day CDSS) for predicting the likelihood of aGvHD on transplantation day. Considering the most important aGvHD predictors and ML classification models, in the following lines, we have discussed some important aspects of this study.

I. The most important predictors for aGvHD

Biomarkers play a key role in predicting aGvHD as they help oncologists to identify patients who are at higher risk for aGvHD, and to select appropriate pre and post transplantation care plans for them. In this study, seven variables were identified as the most important factors associated with aGvHD on the transplantation day. These variables included albumin, uric acid, CRP, donor age, platelet, LDH, and hemoglobin.

In our study, the relative importance of albumin in predicting aGvHD was about 41%. Similarly, previous studies have also emphasized on the importance of the albumin level for predicting aGvHD [ 38 - 40 ], and low amounts of albumin alone and without dependence on other predictors affect overall mortality of aGvHD patients [ 41 ].

The second predictor for aGvHD is uric acid with the importance of 15.1%. In previous studies, this variable has been cited as a strong immunological risk signal. Low levels of this predictor, especially on the day of transplantation, increase the likelihood of aGvHD [ 42 , 43 ].

The third predictor for predicting aGvHD is CRP with the importance of 14.8%. High levels of this predictor in patients increase the risk of aGvHD, especially grade II to IV, asymptomatic death, and decreased overall survival [ 44 - 48 ].

The fourth predictor for aGvHD is the donor age with the importance of 8.5%. In studies related to AHSCT, donor age is considered as an important predictor [ 49 ], and one of the appropriate predictors that can be achieved easily and without cost.

The fifth predictor for aGvHD is platelets with the importance of 8.1%. Previous studies have emphasized that to prevent the likelihood of aGvHD in AHSCT patients, it is essential to maintain platelet counts above 10,000 mm3 [ 50 ].

The sixth predictor for aGvHD is LDH with the importance of 7.1%. Studies have shown that low levels of LDH and high levels of serum cyclosporine reduce the likelihood of aGvHD [ 51 ]. In some previous studies, this variable has been presented as one of the important predictors [ 10 , 52 ].

The seventh predictor for predicting aGvHD is hemoglobin with the importance of 5.5%. Previous studies have emphasized that to prevent the likelihood of aGvHD in AHSCT patients, maintaining a hemoglobin level above 8 to 9 g/dL is essential. Therefore, red blood cells and platelets are injected continuously in these patients [ 50 ].

According to the literature review, each one of previous studies has focused on the importance of a marker based on the diagnosis of aGvHD. Thus, the impact of the combination of these significant factors on aGvHD detection is reported for the first time in this study.

II. Selected machine learning models

After identifying the most important predictors, the algorithm modeling process was performed using the optimization of their hyperparameters. Based on the evaluation criteria of accuracy, sensitivity, specificity, AUC, F-measure, and the average of these criteria, XGBClassifier model and HistGradientBoostingClassifier had the best performance.

In previous studies, ML algorithms have been mainly used in laboratory settings, not in clinical practice. In the present study, the selected and tuned ML models were used as the inference engine of a CDSS to predict aGvHD in the transplantation unit of the target hospital.

In 2015, a study by Cocho et al, [ 53 ] aimed at using different ML algorithms to diagnose aGvHD by gene expression data, used support vector machine (SVM), shrinkage discriminant analysis (SDA), K-nearest neighbors (KNN) algorithms without tuning their hyper-parameters. The reported sensitivity, specificity and AUC were 100%, 92.9% and 99.5% for SVM, 92.9%, 92.9% and 95.9% for SDA and 92.9%, 92.9% and 92.9% for KNN, respectively. The ML models presented in this study had very good performance evaluation criteria, but there are three main criticisms for this study as follows: 1) this study aimed to diagnose aGvHD only based on gene expression data. 2) These ML models have not been tested in a clinical setting, and 3) the system cannot predict aGvHD before the patient goes through transplantation because the study was designed in such way, measured after transplantation.

In 2018, Arai et al, [ 28 ] conducted a study entitled “Predicting aGvHD following AHSCT using an ML algorithm” using the ADTree without the hyper-parameter optimization method. The reported AUC for grades 2-4 aGvHD was 61.6% and for grades 3-4 was 62.3%. This study aimed to develop ML models to accurately predict grades 2 to 4 of aGvHD, However, the performance of their models was poor. In contract, in our study, all performance measures of the selected ML models had values over 90%, which demonstrates a much better overall performance than Arai et al, study.

In 2018, Lee et al, [ 29 ] conducted a study entitled as “Predicting the absolute risk of aGvHD following AHSCT” using the ensemble method without optimizing the hyper-parameters of the employed algorithms. The reported AUC was in the range of 61.3% to 64% for these ensemble models. Despite the fact that in the present study, the models are also of the ensemble type, because of the hyperparameter tuning, the performance was much better compared to the study conducted by Lee et al.

In 2020, Tang et al, [ 54 ] conducted a study entitled “Predicting aGvHD using Machine Learning and Longitudinal Vital Signs Data from Electronic Health Records” using logistic regression without hyper-parameters optimization methods. The reported AUC for grades 2-4 aGvHD was 65.9%. This study, like a few other studies [ 28 , 29 , 53 ], was performed to diagnose aGvHD after transplantation. Compared to the ML models of the present study, the model proposed in the study of Tang et al, has a lower performance, and has not been used in a clinical setting.

Comparing the performance evaluation criteria of XGBClassifier and HistGradientBoostingClassifier with ML model presented in previous studies [ 28 , 29 , 53 , 54 ], it seems that the use of these ML models in CDSS to predict aGvHD in the process of modifying the care plan of patients who received AHSCT can be useful and effective. Thus, we designed and developed a CDSS and applied it in the transplantation unit of the target hospital to predict aGvHD on the day of transplantation.

III. CDSS performance evaluation

In terms of developing aGvHD, there was 92% agreement between the CDSS prediction outcome and the actual patient outcome that was measured 100 days after the AHSCT transplantation.

Given that the criteria of the average evaluation of the ML models used in this CDSS were 91%, it seems that this CDSS had acceptable performance.

Conclusion

According to the current results and previous research, it is obvious that training a model based on the aggregation of the most significant features achieves the better performance in comparison with generating a model concerning each important feature, separately.

In this study, seven variables were identified as the most important factors associated with aGvHD on the transplantation day. These variables included albumin, uric acid, CRP, donor age, platelet, LDH, and hemoglobin. Ensembled Machine learning methods can reliably predict the likelihood of aGvHD at the time of transplantation. These methods can help us to limit the number of risk factors to those that have the significant effects on the outcome. However, their performance is heavily dependent on selecting the appropriate methods and algorithms. Future studies should focus on determining the most appropriate aGvHD predictive models.