Elucidation of the Application of Blood Test Biomarkers to Predict Immune-Related Adverse Events in Atezolizumab-Treated NSCLC Patients Using Machine Learning Methods.

Background: Development of severe immune-related adverse events (irAEs) is a major predicament to stop treatment with immune checkpoint inhibitors, even though tumor progression is suppressed. However, no effective early phase biomarker has been established to predict irAE until now. Method: This study retrospectively used the data of four international, multi-center clinical trials to investigate the application of blood test biomarkers to predict irAEs in atezolizumab-treated advanced non-small cell lung cancer (NSCLC) patients. Seven machine learning methods were exploited to dissect the importance score of 21 blood test biomarkers after 1,000 simulations by the training cohort consisting of 80%, 70%, and 60% of the combined cohort with 1,320 eligible patients.
Results: XGBoost and LASSO exhibited the best performance in this study with relatively higher consistency between the training and test cohorts. The best area under the curve (AUC) was obtained by a 10-biomarker panel using the XGBoost method for the 8:2 training:test cohort ratio (training cohort AUC = 0.692, test cohort AUC = 0.681). This panel could be further narrowed down to a three-biomarker panel consisting of C-reactive protein (CRP), platelet-to-lymphocyte ratio (PLR), and thyroid-stimulating hormone (TSH) with a small median AUC difference using the XGBoost method [for the 8:2 training:test cohort ratio, training cohort AUC difference = -0.035 (p < 0.0001), and test cohort AUC difference = 0.001 (p=0.965)].
Conclusion: Blood test biomarkers currently do not have sufficient predictive power to predict irAE development in atezolizumab-treated advanced NSCLC patients. Nevertheless, biomarkers related to adaptive immunity and liver or thyroid dysfunction warrant further investigation.

Background

Immune checkpoint inhibitor (ICI) therapy has become a widely used first-line therapy for unresectable non-small cell lung cancer (NSCLC) patients. ICIs were developed against programmed cell death ligand 1 (PD-L1) on cancer cells, and the immune suppressive receptors programmed cell death 1 (PD-1) and cytotoxic T-lymphocyte-associated antigen 4 (CTLA-4) on cytotoxic T cells (1). Although ICI therapy can be effective, 5–10% of patients experience immune-related adverse events (irAEs), such as rashes and peripheral neuropathy, as soon as the next day after treatment starts (2–4). Over the course of treatment, some patients may develop more severe symptoms such as pneumonitis, pancreatitis, and vitiligo, which sometimes lead to death. Hence, prediction of severe irAEs before or during early treatment becomes indispensable. In this regard, several studies have been conducted to investigate the correlation between liquid biopsy biomarkers and irAEs (5–7). However, the concurrence of low incidence rate and limited cohort size in many studies restricts the extent of analysis to achieve statistical significance.

The real challenge of irAE prediction is finding the right biomarkers to indicate the immune landscape in a spatial and temporal manner (8). In contrast to therapies with known biological mechanisms or specificity towards certain organs or tissue, targeting of the immune system by ICIs seems to have more unpredictable AEs, particularly irAEs that could potentially affect any part of the body. Nevertheless, research on intrinsic and extrinsic irAE mediators are taking place (8). As such, we reasoned that the prediction of irAEs involves two parallel options: (1) monitoring of immune activity and (2) malfunction detection in vulnerable organs or tissue.

Blood test is regularly conducted before and during treatment and is feasible in almost any hospital. The regular blood test consists of two operations: the blood cell count test (BCT) and the blood biochemistry test (BBT). BCT provides a direct overview of the immune landscape based on the prevalence of immune cell populations. Our previous study established a BCTscore model as a valid predictive and prognostic biomarker for the early prediction of atezolizumab treatment outcomes (9). Because treatment outcome is often correlated to the onset of irAEs (10–12), and pretreatment blood cell count is reported to be associated with pembrolizumab-induced irAEs in patients with advanced NSCLC (13), we hypothesize that BCT biomarkers may also predict irAEs of atezolizumab-treated NSCLC patients. On the other hand, a comprehensive BBT provides a functional overview of various organs. Hence, we investigated the application of blood test parameters to predict irAEs in atezolizumab-treated NSCLC patients using data acquired from the four international, multicenter cohorts of FIR, BIRCH, POPLAR, and OAK.

Methods

Study Cohort

Pseudonymized individual participant data from the single-arm phase II studies FIR (NCT01846416) (14) and BIRCH (NCT02031458) (15) and the two-arm randomized controlled trials (RCTs) POPLAR phase II study (NCT01903993) (16) and OAK phase III study (NCT02008227) (17) were provided by Genentech Inc. and accessed through the secure Vivli online platform. Raw data were extracted and compared with the available published data to ensure accuracy. Secondary analysis of the trial data was deemed to be of negligible risk and was approved by the Institutional Review Board of the Second Affiliated Hospital of Zunyi Medical University [No. YXLL (KY-R)-2021-010]. Deidentified data were accessed according to Roche’s policy and process for Vivli. Data analyses were conducted from April 27 to November 30, 2021.

Definition of irAEs

irAEs are summarized using the National Cancer Institute (NCI) Common Terminology Criteria for Adverse Event (CTCAE) version 4.0 (18) by clinical study. The irAE data were confirmed from the Adverse Events of Special Interest (AESI) dataset according to the Council for International Organisations of Medical Sciences (CIOMS) form. The variable “AEGRP01F = Y” was selected to ensure that the irAE was associated with PD-L1 checkpoint blockade, as already defined by Khan and colleagues (3). Specifically, the CTCAE defines what symptoms constitute AEs and specifically, irAEs. On the other hand, the AESI was compiled by the Vivli platform from which our data were obtained. Incidences of irAE in all four cohorts were obtained under the header “AEGRP01F,” and “Y” stands for “Yes” in the patient records provided by the Vivli platform. We combined the four cohorts into one big cohort comprising 1,320 eligible atezolizumab-treated advanced NSCLC patients with irAE and pretreatment blood test records. After that, we summarized the number of patients with any grade irAE. Because our cohort was assembled from four cohorts, the numbers do not match those which are reported in each individual cohort. This strategy identified a collection of adverse events that had a putative immune-related etiology.

Machine Learning Methods

Because of the complexity of different parameters obtained from BCT and BBT, machine learning was used instead of conventional Cox regression model. Here, we applied seven machine learning methods (). The methods used in this study include the following: (1, 2) the Lasso (LASSO) or Elastic-Net Regularized Generalized Linear Model (GLM) (R package glmnet v.4.1.3) (19), (3) the Support Vector Machines model (SVM; R package e1071 v.1.7.9) (20), (4) the Recursive Partitioning and Regression Trees model, also known as Decision Tree model (DT; R package rpart v. 4.1.15) (21), (5) the Random Forest model (RF; R package randomForest v. 4.6.14) (22), (6) the eXtreme Gradient Boosting model (XGB; R package xgboost v. 0.4.2) (23), and (7) the Generalized Boosted Regression Models (GBMs; R package gbm v.2.1.8) (24). The function createDataPartition of caret (Classification and Regression Training) package v.6.0.89 (25) was used to create balanced splits of the data as training and test cohorts.

Analytic Procedures

The paradigm of this study is illustrated in . First, all four cohorts containing 2,316 advanced NSCLC patients were combined, of which 1,537 were eligible atezolizumab-treated advanced NSCLC patients with irAE. A total of 1,320 eligible atezolizumab-treated advanced NSCLC patients with irAE and their pretreatment blood test records were randomly separated into training and test cohorts under the criterion that each sample population contained 5% patients displaying any form of irAE. Next, blood test parameters with >10% missing values in the sample population were removed from analysis. Consequently, a total of 21 blood test parameters were fed into the machine learning models as primary classifiers. Binary outcomes of any irAE was applied to the prediction models. The sample populations for the training and test cohorts were randomly selected at the ratios of 6:4, 7:3, and 8:2 from the combined population of the four clinical trials for 1,000 times simulation. The blood test parameters were evaluated by the importance score generated by each method after each simulation for their performance. Model performance was evaluated by the area under curve (AUC) and corresponding 95% confidence interval (CI) of the receiver operating characteristic (ROC) curve. Analysis of variance (ANOVA) and Tukey honestly significant difference (Tukey HSD) tests were performed by R base package. Sensitivity, specificity, accuracy, and the Kappa statistic were calculated by ROCR package (v.1.0-11) (26) and interpreted as previously described (27).

Figure 1

Study overview. A total of 1,320 eligible NSCLC patients undergoing atezolizumab single-agent treatment is obtained from four international, multicenter clinical trials for this study.

Results

Blood Test Biomarkers That Indicate Adaptive Immunity and Liver Function Are Useful for irAE Prediction in Atezolizumab-Treated Advanced NSCLC Patients

Initially, we compared the median performance of all 21 blood test biomarkers after 1,000 simulations on the training cohorts by the seven machine learning methods (). We selected the top 10 biomarkers that stably displayed above median performance in all simulations (). Among these biomarkers were the neutrophil-to-lymphocyte ratio (NLR) and platelet-to-lymphocyte ratio (PLR). NLR and PLR indicate the host’s immune landscape and had been selected for our previous BCTscore model to predict survival benefit (9). Alternatively, red blood cell count (RBC), hematocrit (HCT), hemoglobin (HGB), albumin (ALB), and alkaline phosphatase (ALP) that indicate liver function also demonstrated good performance in irAE prediction. On the other hand, the BBT biomarkers of lactate dehydrogenase (LDH) and C-reactive protein (CRP) that indicate tissue damage and infection performed well in our first biomarker screening. Additionally, the BBT biomarker of thyroid-stimulating hormone (TSH) that indirectly suggest infection and cancer also demonstrated good performance in irAE prediction. Among the 10 biomarkers, the top 3 blood test biomarkers were PLR, CRP, and TSH (). Results showed that the 10-biomarker panel is most optimal for irAE prediction, where the AUC of the test cohort differed insignificantly from the training cohort at all three cohort ratios. However, the three-biomarker panel consisting of PLR, CRP, and TSH is sufficient, with small AUC difference from the 10-biomarker panel [for 8:2 training:test cohort ratio, the mean difference in AUC of the 10-biomarker panel vs. 3-biomarker panel of LASSO: training = −0.044 (p<0.0001), test = −0.026 (p<0.0001); and XGB: training = −0.035 (p<0.0001), test = 0.001 (p=0.965)] ().

Figure 2

Performance of the 10-biomarker panel evaluated by the seven machine learning methods of (A) DT, (B) GBM, (C) GLM, (D) LASSO, (E) RF, (F) SVM, and (G) XGB. Performance scores were computed by each machine learning method for 1,000 simulations of the training and test datasets at 8:2, 7:3, and 6:4 cohort ratios randomly selected from the combined cohort comprising 1,320 atezolizumab-treated and whisker plot shows the median (thick black line in the middle of the box), the interquartile range between 75% and 25% (upper and lower end of the box), and 1.5 multiplied by upper or lower interquartile range (whiskers), respectively. ns is P ≥ 0.05, *P < 0.05, **P < 0.01, *** P < 0.001, ****P < 0.0001.

Taken together, we conclude that blood test biomarkers indicating adaptive immunity and liver or thyroid dysfunction are useful for irAE prediction in atezolizumab-treated advanced NSCLC patients.

LASSO and XGB Exhibited the Best Performance in irAE Prediction in This Study

Because the seven machine learning methods are vastly different in terms of mathematical modeling and application, we tested which method is optimal for irAE prediction in atezolizumab-treated advanced NSCLC patients. Therefore, we compared the AUC distribution of our retrospective cohorts using the 21-, 10-, and 3-biomarker panels, respectively (). Furthermore, the combined cohort was randomly separated into training and test cohorts at different ratios to evaluate the consistency of irAE prediction in a retrospective manner. Results showed that increasing the training:test cohort ratio improves the predictive power of all seven machine learning methods. Furthermore, the 21-biomarker panel generally yields higher median AUC as compared to the 10- and 3-biomarker panels. Nevertheless, the seven machine learning methods displayed fundamental differences in irAE prediction performance. For instance, although RF consistently demonstrated median AUC >0.8 in the 1,000 simulations of the training cohorts comprising of 80%, 70%, and 60% of the combined cohort, its performance on the test cohort did not significantly differ from the other machine learning methods; the median AUC of all seven machine learning methods lay in the range of 0.517–0.581 throughout the 1,000 simulations of the test cohorts. Similarly, the DT and SVM models suffered from discrepancy between the training and test cohort prediction like the RF method. In contrast, the unsupervised, linear GLM and GBM models displayed consistent predictive power between training and test cohorts and yielded comparative median AUC to LASSO. However, the bigger variance in AUC distribution is less favorable for their application in real-life, prospective predictions.

Hence, LASSO and XGB showed the highest potential as optimal irAE prediction methods. These two methods gave the best AUC for the 10-biomarker panel (for 8:2 training:test cohort ratio, the mean AUC of the 10-biomarkers panel calculated by LASSO: training = 0.604, test = 0.642; and XGB: training = 0.692, test = 0.681) (). Other reasons for choosing these two methods include the following: (1) LASSO and XGB depicted good predictive power in both training and test cohorts at all three cohort ratios (); (2) they were minimally affected by reducing the number of biomarkers () as compared to the other methods (); and (3) these two methods are fundamentally different—LASSO uses a supervised, linear algorithm, whereas XGB uses a supervised, non-linear algorithm.

Figure 3

Best ROC curves of the 10-biomarker panel evaluated by the LASSO and XGB methods. The best ROC curves were obtained at 8:2 cohort ratio of the training and test datasets by the LASSO and XGB methods.

Table 1

Median AUC distribution of the three blood test biomarker panels.

	Cohort ratio	21-Biomarker		10-Biomarker		3-Biomarker
Method	(training:test)	Training	Test	Training	Test	Training	Test
LASSO	6:4	0.591 ± 0.027	0.574 ± 0.023	0.591 ± 0.024	0.578 ± 0.022	0.556 ± 0.021	0.553 ± 0.024
	7:3	0.594 ± 0.027	0.575 ± 0.027	0.594 ± 0.025	0.579 ± 0.027	0.557 ± 0.022	0.554 ± 0.028
	8:2	0.596 ± 0.030	0.578 ± 0.036	0.597 ± 0.029	0.582 ± 0.037	0.557 ± 0.026	0.556 ± 0.035
XGB	6:4	0.662 ± 0.089	0.563 ± 0.023	0.662 ± 0.080	0.564 ± 0.023	0.633 ± 0.062	0.561 ± 0.023
	7:3	0.670 ± 0.083	0.564 ± 0.028	0.664 ± 0.075	0.565 ± 0.028	0.639 ± 0.057	0.565 ± 0.027
	8:2	0.684 ± 0.079	0.566 ± 0.035	0.670 ± 0.070	0.567 ± 0.034	0.642 ± 0.054	0.571 ± 0.035

displays the best AUC obtained from LASSO and XGB among 1,000 simulations for the training and test cohorts, whereas in and , the AUC distribution of the 1,000 simulations are summarized. One can therefore conclude that the AUC is still unsatisfactory for real-life application. This conclusion is reinforced by the observed Kappa statistic (). The low Kappa statistic suggested that the accuracy of prediction is unacceptable for all three blood test biomarker panels. Hence, we concluded that blood test biomarkers do not have sufficient predictive power to predict irAE development in atezolizumab-treated advanced NSCLC patients.

Discussion

In this study, we analyzed 21 blood test biomarkers with comparison of the seven machine learning methods to identify the optimal biomarker panel and machine learning methods for irAE prediction in a combined cohort from four retrospective, multi-center clinical trials, involving advanced NSCLC patients treated with the anti-PD-L1 atezolizumab. Results showed that blood test biomarkers do not have sufficient predictive power to predict irAE development in atezolizumab-treated advanced NSCLC patients.

Better biomarkers are urgently needed. A literature review of the biological mechanisms of the best-performing biomarkers showed that the liver is critical for adaptive immunity (28–30) and is the primary synthetic site for CRP (31). Hence, although none of the biomarkers related to liver function made to the top three-biomarker panel, the inclusion of these biomarkers in the 10-biomarker panel increased median AUC of irAE prediction from that of the three-biomarker panel, with small but significant difference in the training cohort.

In the three-biomarker panel, PLR indisputably correlates with survival outcomes during ICI therapy (32). However, CRP and TSH are less studied. CRP has long been used as a universal biomarker for infection-induced inflammation (33). However, it has only been recently reported that different isoforms of CRP harness different biological pathways to trigger inflammation (31), while our clinical blood tests merely detect naive CRP. Nonetheless, the CRP/albumin ratio has been reported to be positively correlated with PLR and could serve as an independent risk factor for overall survival (OS) in advanced NSCLC patients (34, 35). Hence, although the biological mechanism remains elusive, CRP depicted good performance for irAE prediction in NSCLC, in consistence to previous findings in melanoma patients (36). On the other hand, even though the mechanism of TSH is also unclear, thyroid dysfunction is a prevalent irAE in NSCLC patients treated with the anti-PD1 antibody nivolumab (37) and thus is not surprising to perform well in this study.

Khan and colleagues found that genetic variation that is associated with thyroid autoimmunity interacts with biological pathways driving the systemic immune response to ICI (2). Another study demonstrated that activated CD4 memory T-cell abundance and TCR diversity are associated with severe irAE development regardless of the organ system involvement (38). Collectively, we deduce that biomarkers related to adaptive immunity and liver or thyroid dysfunction warrants further investigation.

Notably, there was insignificant difference in irAE prediction in the test cohorts for all seven machine learning methods, no matter how well these methods performed in the training cohorts. This observation suggested that shuffling patients or adjusting the training:test cohort ratios did not improve the irAE prediction model’s performance. Hence, the machine learning methods of RF, DT, and SVM were likely overfittings during training, thus performing poorly in the test cohorts because all patients differed between the training and test cohorts, except the fact that both cohorts contained 5% patients exhibiting any form of irAE. Additionally, 1,000 simulations of random selection of 1,320 patients into the training and test cohorts at different ratios ruled out the possibility of sample bias. Avoiding sample bias is particularly important because of the different patient eligibility criteria of the four clinical trials: specifically, except that the BIRCH and FIR trials contained a fraction of PD-L1-positive (PD-L1 ≥ 5%) advanced NSCLC patients with no prior chemotherapy, all trials recruited advanced NSCLC patients who underwent platinum-based therapy. Therefore, the failure of certain methods to predict irAE could not stem from accidental sample bias during simulation. Furthermore, it is noteworthy that the difference in median AUC predicted by the seven machine learning methods using the top 3-biomarker panel () narrowed as compared to the 10- and 21-biomarker panels (). Considering that these methods exploit disparate mathematical algorithms, it became explicit that the nature of algorithm did not affect the conclusion. Nevertheless, as pointed out earlier, overfitting seemed to be more prevalent in some algorithms than others. Moreover, some biomarkers with widely variable absolute detected range among individual patients may show inconsistent performance in some methods. For example, RBC showed a wide range of performance in the linear models of LASSO and GLM (). However, the fact that RBC performance was more consistent in the other linear models of SVM, DT, and GBM suggested that the divergence was method specific. In contrast, PLR is also inherently divergent like RBC, but its performance was much more consistent in all methods. Hence, even though blood test biomarkers may exhibit variable detected ranges, their irAE prediction performance is relatively stable. Therefore, caution should be taken when assessing the performance variance of certain biomarkers as a criterion for biomarker selection. In-depth analysis of the common detectable range and biological mechanism of each biomarker is highly recommended during panel construction.

Conclusion

Blood test biomarkers do not have sufficient predictive power to predict irAE development in atezolizumab-treated advanced NSCLC patients. Biomarkers related to adaptive immunity and liver or thyroid dysfunction warrant further investigation.

Data Availability Statement

The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found below: this publication is based on research using data from data contributors, Roche, that have been made available through Vivli, Inc. (Data Request ID: 5935; Lead Investigator: J-GZ). Vivli has not contributed to or approved, and is not in any way responsible for, the contents of this publication.

Ethics Statement

The studies involving human participants were reviewed and approved by the Institutional Review Board of the Second Affiliated Hospital of Zunyi Medical University [No. YXLL(KY-R)-2021-010]. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

Author Contributions

Conceptualization: JGZ, AHW, HM, UG, and SHJ. Methodology: JGZ, AHW, and FT. Validation: MH, HM, UG, and JGZ. Data curation: JGZ, AHW, GS, SSH, HM, SRC, and UG. Writing—original draft preparation: AHW, HW, and JGZ. Writing—review and editing: UG, HM, BF and MH. Visualization: JGZ and HW. Supervision: HM, UG, and JGZ. Project administration: JGZ, HM, and UG. Funding acquisition: JGZ, SHJ, and AHW. All authors contributed to thearticle and approved the submitted version.

Funding

This research was funded by the National Natural Science Foundation of China (Grant No. 82060475), the National Natural Science Foundation of Guizhou Province (Grant Nos. [2020]1Z062, ZK2021-YB435, and ZK2022-YB632), and Lian Yun Gang Shi Hui Lan Public Foundation (Grant No. HL-HS2020-92). AW Medical Company Limited received startup funding from the University Development Fund of University of Macau.

Conflict of Interest

AHW is a founder and shareholder of AW Medical Company Limited. MH reports collaborations with Merck Serono (advisory role, speakers’ bureau, honoraria, travel expenses, and research funding), MSD (advisory role, speakers’ bureau, honoraria, travel expenses, and research funding), AstraZeneca (research funding), Novartis (research funding), BMS (advisory role, honoraria, and speakers’ bureau), and Teva (travel expenses). UG and RF received support for presentation activities for Dr Sennewald Medizintechnik GmbH, have received support for investigator initiated clinical studies (IITs) from MSD and AstraZeneca, and contributed at Advisory Boards Meetings of AstraZeneca and Bristol-Myers Squibb.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.